From a9dc59546ba782593b74de5da315ad314411133b Mon Sep 17 00:00:00 2001 From: clawbot <35+clawbot@noreply.example.org> Date: Tue, 27 Oct 2026 19:28:09 +0100 Subject: [PATCH 1/9] New defaults for the request and byte limits (closes #161) The request limits and the hour and day byte limits now sit about twice above what 40 people busy with heavy web apps make together on one address: - requests: 6000 a minute, 100,000 an hour, 1,000,000 a day; - bytes: 20 GiB an hour, 200 GiB a day. The largest response drops from 5 GiB to 1 GiB. The minute byte limit of 4 GiB is sized to hold two of the largest responses with their requests. SPEC.md gives the new reasoning; README.md and the reference documents give the new figures. Model: opus-5-5 --- README.md | 8 +++---- SPEC.md | 30 ++++++++++++++++-------- docs/alerts.md | 10 ++++---- docs/gitea.md | 19 ++++++++------- docs/settings.md | 12 +++++----- internal/config/config.go | 12 +++++----- internal/config/config_test.go | 24 +++++++++---------- internal/smallwebwaf/smallwebwaf_test.go | 8 +++---- 8 files changed, 68 insertions(+), 55 deletions(-) diff --git a/README.md b/README.md index 4aecf31..712ad5a 100644 --- a/README.md +++ b/README.md @@ -17,10 +17,10 @@ reference documents under [`docs/`](docs/) are listed under "Documents" below. ## What it protects against - Scrapers that fetch every page of a site, all day. Rate limiting gives each - client 1000 requests a minute, 10,000 an hour and 50,000 a day. Byte limits - give it 10 GiB a minute, 20 GiB an hour and 50 GiB a day. A client over a - limit is refused and banned. The first ban lasts an hour. Each repeat within a - day makes the next ban three times as long. A ban that would last more than + client 6000 requests a minute, 100,000 an hour and 1,000,000 a day. Byte + limits give it 4 GiB a minute, 20 GiB an hour and 200 GiB a day. A client over + a limit is refused and banned. The first ban lasts an hour. Each repeat within + a day makes the next ban three times as long. A ban that would last more than seven days is permanent. - Scanners that probe for secrets, such as `/.env` or `/.git/config`. The default rule file bans such a client for seven days at its first probe. Its diff --git a/SPEC.md b/SPEC.md index 90b6c95..0b3d7b6 100644 --- a/SPEC.md +++ b/SPEC.md @@ -368,16 +368,23 @@ Conventions: - `SWWAF_ALLOW_NETS` skips the checks step 3 names, for monitoring or the owner's own networks. `SWWAF_RATE_LIMIT_EXEMPT_NETS` bypasses the request and byte limits only; the error burst, attack detection and bans still apply. -- Many clients share one address (mobile carriers, offices, Tor), and so share - its limits and its bans. `SWWAF_ALLOW_NETS` and `SWWAF_RATE_LIMIT_EXEMPT_NETS` - take known shared addresses out. +- Many people can share one address, as in an office, on a mobile carrier or + through Tor. They are one client, and share its limits and its bans. The + default limits are sized for about 40 busy people on one address. A known + shared address busier than that belongs in `SWWAF_ALLOW_NETS` or + `SWWAF_RATE_LIMIT_EXEMPT_NETS`. ### Request rate limits, per client -Breaking one bans the client, so the defaults sit several times above what one -busy person produces. A browser loading a heavy page makes a few hundred -requests, and a git clone a handful. A machine that talks to the app all day can -still pass the day limit. Its address belongs in `SWWAF_RATE_LIMIT_EXEMPT_NETS`. +Breaking one bans the client, so the defaults sit about twice above what 40 +people busy with heavy web apps make together on one address. Each such person +makes about 75 requests in a busy minute, and 1,250 an hour for ten hours a day. + +A client slower than those 40 people cannot be told from them by counting. Only +the other checks can catch an abuser that slow. They include the trap paths, the +rule files, the Core Rule Set and the blocklists. An abuser that trips none of +them is not caught. A machine that talks to the app all day can still pass the +day limit. Its address belongs in `SWWAF_RATE_LIMIT_EXEMPT_NETS`. `SWWAF_RATE_LIMIT_EXEMPT_PATHS` takes path prefixes out of the count, such as static assets or health checks; their bytes are not counted either. A prefix is @@ -390,9 +397,12 @@ matched the same way. ### Byte limits, per client -A response's bytes are counted when it ends (see step 15). So every default sits -above the largest request and answer together, and no single download breaks a -limit. +Each of the same 40 people moves about 256 MiB an hour, for ten hours a day. The +hour and day limits sit about twice above what they move together. A response's +bytes are counted when it ends (see step 15). So the minute limit holds two of +the largest responses, with their requests, ending in the same minute. No single +download then breaks a limit. An operator who raises `SWWAF_RESPONSE_MAX_BYTES` +raises the byte limits with it. ### Size and time limits diff --git a/docs/alerts.md b/docs/alerts.md index cbc4fe2..916912d 100644 --- a/docs/alerts.md +++ b/docs/alerts.md @@ -42,7 +42,7 @@ on one line: "asn": "", "as_name": "", "country": "", - "reason": "requests per minute over the limit of 1000", + "reason": "requests per minute over the limit of 6000", "detail": { "ban_expires": "2026-10-06T13:00:00.123Z", "cause": "limit", @@ -51,9 +51,9 @@ on one line: "as_name": "", "country": "", "kind": "requests", - "limit": 1000, + "limit": 6000, "window": "minute", - "count": 1001, + "count": 6001, "request": { "time": "2026-10-06T12:00:00.123456789Z", "method": "GET", @@ -62,7 +62,7 @@ on one line: "status": 403, "user_agent": "scraper/1.0" }, - "requests": 5210, + "requests": 8210, "refused": 0, "earlier_bans": { "limit": 0, @@ -89,7 +89,7 @@ on one line: - `asn`, `as_name` and `country`: the client's, as known when the alert was raised. They are empty when GeoJS had not answered yet. - `reason`: a short sentence, such as - `requests per minute over the limit of 1000` or `listed by a blocklist`. + `requests per minute over the limit of 6000` or `listed by a blocklist`. - `detail`: what is particular to the event: - for a ban, its `cause`, `ban_expires` and `notes` (the notes are listed under `bans.json` in [`state-files.md`](state-files.md)); diff --git a/docs/gitea.md b/docs/gitea.md index 312d778..df40899 100644 --- a/docs/gitea.md +++ b/docs/gitea.md @@ -11,8 +11,10 @@ checked again when gitea or the Core Rule Set changes version. - SSH access to gitea does not pass through `smallwebwaf` and is not protected by it. -- A clone is a response. It fits the defaults of 30 minutes and 5 GiB for all - but the largest repositories and slowest links. +- A clone is a response. It fits the defaults of 30 minutes and 1 GiB unless the + repository is large or the link slow. A repository whose clone is larger than + 1 GiB needs `SWWAF_RESPONSE_MAX_BYTES` raised, and the byte limits with it + ("Byte limits, per client" in `SPEC.md`). - A push is a request, and so is every other upload: - a git push; - an LFS object; @@ -30,12 +32,13 @@ checked again when gitea or the Core Rule Set changes version. request limits do most of the work there. - An Actions runner that reaches gitea through `smallwebwaf` sends requests all day. One older than version 0.4 (April 2026) asks for work every 2 seconds, - 43,200 requests a day, and reports a running job's log and state every second. - So it passes the day limit of 50,000 after about an hour and a quarter of - jobs. It is then banned, the job it is running fails, and each repeat within a - day makes the ban longer. Newer runners ask less often, but a busy one, or - several on one address, can still pass it. Put the runners' addresses in - `SWWAF_RATE_LIMIT_EXEMPT_NETS`. + 43,200 requests a day. While it runs a job, it reports the job's log and state + every second instead, two requests a second. So one runner busy all day sends + 172,800 requests, under the request limits. Six such runners on one address, + each busy all day, pass the day limit of 1,000,000 within two days; five stay + under it. Their address is then banned, their running jobs fail, and each + repeat within a day makes the ban longer. Newer runners ask for work less + often. Put the runners' addresses in `SWWAF_RATE_LIMIT_EXEMPT_NETS`. ## What the Core Rule Set lets through diff --git a/docs/settings.md b/docs/settings.md index 3ed2db3..ad41aee 100644 --- a/docs/settings.md +++ b/docs/settings.md @@ -99,16 +99,16 @@ that take `off`, the `_FILE` form, and what each group does and why. ## Request rate limits, per client -- `SWWAF_RATE_LIMIT_PER_MINUTE` (default `1000`), `SWWAF_RATE_LIMIT_PER_HOUR` - (default `10000`) and `SWWAF_RATE_LIMIT_PER_DAY` (default `50000`): the most - requests a client may make in each window. A count or `off`. +- `SWWAF_RATE_LIMIT_PER_MINUTE` (default `6000`), `SWWAF_RATE_LIMIT_PER_HOUR` + (default `100000`) and `SWWAF_RATE_LIMIT_PER_DAY` (default `1000000`): the + most requests a client may make in each window. A count or `off`. - `SWWAF_RATE_LIMIT_EXEMPT_PATHS` (default empty): path prefixes exempt from rate limiting and the byte limits, such as `/assets/`. ## Byte limits, per client -- `SWWAF_BYTES_LIMIT_PER_MINUTE` (default `10G`), `SWWAF_BYTES_LIMIT_PER_HOUR` - (default `20G`) and `SWWAF_BYTES_LIMIT_PER_DAY` (default `50G`): the most +- `SWWAF_BYTES_LIMIT_PER_MINUTE` (default `4G`), `SWWAF_BYTES_LIMIT_PER_HOUR` + (default `20G`) and `SWWAF_BYTES_LIMIT_PER_DAY` (default `200G`): the most bytes counted for a client in each window. A size or `off`. - `SWWAF_BYTES_COUNT` (default `both`): which body bytes the byte limits count: `response`, `request` or `both`. @@ -131,7 +131,7 @@ that take `off`, the `_FILE` form, and what each group does and why. to send its whole answer. A duration or `off`. - `SWWAF_REQUEST_MAX_BYTES` (default `100M`): the largest request body. A size or `off`. -- `SWWAF_RESPONSE_MAX_BYTES` (default `5G`): the largest response body. A size +- `SWWAF_RESPONSE_MAX_BYTES` (default `1G`): the largest response body. A size or `off`. ## Lookup of AS number and country diff --git a/internal/config/config.go b/internal/config/config.go index c86de80..0cc5def 100644 --- a/internal/config/config.go +++ b/internal/config/config.go @@ -502,17 +502,17 @@ func FromEnvironment(lookupEnv func(string) (string, bool)) (*Config, error) { UpstreamRequestTimeout: env.duration("SWWAF_UPSTREAM_REQUEST_TIMEOUT", "60s"), UpstreamResponseTimeout: env.duration("SWWAF_UPSTREAM_RESPONSE_TIMEOUT", "30m"), RequestMaxBytes: env.size("SWWAF_REQUEST_MAX_BYTES", "100M"), - ResponseMaxBytes: env.size("SWWAF_RESPONSE_MAX_BYTES", "5G"), + ResponseMaxBytes: env.size("SWWAF_RESPONSE_MAX_BYTES", "1G"), AllowNets: env.netblocks("SWWAF_ALLOW_NETS", ""), RateLimitExemptNets: env.netblocks("SWWAF_RATE_LIMIT_EXEMPT_NETS", ""), DenyNets: env.netblocks("SWWAF_DENY_NETS", ""), - RateLimitPerMinute: env.count("SWWAF_RATE_LIMIT_PER_MINUTE", "1000"), - RateLimitPerHour: env.count("SWWAF_RATE_LIMIT_PER_HOUR", "10000"), - RateLimitPerDay: env.count("SWWAF_RATE_LIMIT_PER_DAY", "50000"), + RateLimitPerMinute: env.count("SWWAF_RATE_LIMIT_PER_MINUTE", "6000"), + RateLimitPerHour: env.count("SWWAF_RATE_LIMIT_PER_HOUR", "100000"), + RateLimitPerDay: env.count("SWWAF_RATE_LIMIT_PER_DAY", "1000000"), RateLimitExemptPaths: env.pathPrefixes("SWWAF_RATE_LIMIT_EXEMPT_PATHS", ""), - BytesLimitPerMinute: env.size("SWWAF_BYTES_LIMIT_PER_MINUTE", "10G"), + BytesLimitPerMinute: env.size("SWWAF_BYTES_LIMIT_PER_MINUTE", "4G"), BytesLimitPerHour: env.size("SWWAF_BYTES_LIMIT_PER_HOUR", "20G"), - BytesLimitPerDay: env.size("SWWAF_BYTES_LIMIT_PER_DAY", "50G"), + BytesLimitPerDay: env.size("SWWAF_BYTES_LIMIT_PER_DAY", "200G"), BytesCount: env.bytesCount("SWWAF_BYTES_COUNT", "both"), LookupSource: env.lookupSource("SWWAF_LOOKUP_SOURCE", "geojs"), LookupDBPath: env.value("SWWAF_LOOKUP_DB_PATH", ""), diff --git a/internal/config/config_test.go b/internal/config/config_test.go index 9234df4..ff61910 100644 --- a/internal/config/config_test.go +++ b/internal/config/config_test.go @@ -261,10 +261,10 @@ func TestDefaults(t *testing.T) { UpstreamRequestTimeout: time.Minute, UpstreamResponseTimeout: 30 * time.Minute, RequestMaxBytes: 100 << 20, - ResponseMaxBytes: 5 << 30, - RateLimitPerMinute: 1000, - RateLimitPerHour: 10000, - RateLimitPerDay: 50000, + ResponseMaxBytes: 1 << 30, + RateLimitPerMinute: 6000, + RateLimitPerHour: 100000, + RateLimitPerDay: 1000000, BanResponse: 403, LimitBanDuration: time.Hour, LimitBanRepeatWindow: 24 * time.Hour, @@ -284,8 +284,8 @@ func TestDefaults(t *testing.T) { LookupSource: defaultLookupSource, LookupTimeout: time.Second, }) wantByteLimitSettings(t, cfg, config.Config{ - BytesLimitPerMinute: 10 << 30, BytesLimitPerHour: 20 << 30, - BytesLimitPerDay: 50 << 30, BytesCount: "both", + BytesLimitPerMinute: 4 << 30, BytesLimitPerHour: 20 << 30, + BytesLimitPerDay: 200 << 30, BytesCount: "both", }) if cfg.UpstreamURL.String() != "http://127.0.0.1:8081" { @@ -2371,17 +2371,17 @@ func TestLogsEachSettingWithItsValue(t *testing.T) { upstreamRequestTimeout: "60s", upstreamResponseTimeout: "30m", requestMaxBytes: "100M", - responseMaxBytes: "5G", + responseMaxBytes: "1G", allowNets: "", rateLimitExemptNets: "", denyNets: "", - rateLimitPerMinute: "1000", - rateLimitPerHour: "10000", - rateLimitPerDay: "50000", + rateLimitPerMinute: "6000", + rateLimitPerHour: "100000", + rateLimitPerDay: "1000000", rateLimitExemptPaths: "", - bytesLimitPerMinute: "10G", + bytesLimitPerMinute: "4G", bytesLimitPerHour: "20G", - bytesLimitPerDay: "50G", + bytesLimitPerDay: "200G", bytesCount: "both", lookupSource: defaultLookupSource, lookupDBPath: "", diff --git a/internal/smallwebwaf/smallwebwaf_test.go b/internal/smallwebwaf/smallwebwaf_test.go index 3fdf2a3..0aee935 100644 --- a/internal/smallwebwaf/smallwebwaf_test.go +++ b/internal/smallwebwaf/smallwebwaf_test.go @@ -1240,13 +1240,13 @@ func wantStartingLine(t *testing.T, line map[string]any, appURL, dir string) { "SWWAF_UPSTREAM_REQUEST_TIMEOUT": "60s", "SWWAF_UPSTREAM_RESPONSE_TIMEOUT": "30m", "SWWAF_REQUEST_MAX_BYTES": "100M", - "SWWAF_RESPONSE_MAX_BYTES": "5G", + "SWWAF_RESPONSE_MAX_BYTES": "1G", "SWWAF_ALLOW_NETS": "", rateLimitExemptNets: "", "SWWAF_DENY_NETS": "", - "SWWAF_RATE_LIMIT_PER_MINUTE": "1000", - "SWWAF_RATE_LIMIT_PER_HOUR": "10000", - rateLimitPerDay: "50000", + "SWWAF_RATE_LIMIT_PER_MINUTE": "6000", + "SWWAF_RATE_LIMIT_PER_HOUR": "100000", + rateLimitPerDay: "1000000", "SWWAF_RATE_LIMIT_EXEMPT_PATHS": "", "SWWAF_DENIED_COUNTRIES": "", "SWWAF_EXCLUSIVELY_ALLOWED_COUNTRIES": "", From fdabf4974bccc11671e49924d4c02d3bb85d0101 Mon Sep 17 00:00:00 2001 From: clawbot <35+clawbot@noreply.example.org> Date: Tue, 27 Oct 2026 21:00:16 +0100 Subject: [PATCH 2/9] Health check test uses addresses nothing can connect to (closes #158) TestHealthCheck closed the app's server, and later smallwebwaf, and expected connections to the freed ports to fail. Another test could take such a port in between, so the connection succeeded and the test failed. The failing app check now uses the multicast address 224.0.0.1, to which TCP cannot connect. The failing smallwebwaf check uses port 0, on which nothing listens. Neither result depends on other processes. The test now stops smallwebwaf after its last check, so that it exits before the test's temporary directories are removed. The code under test is unchanged. Model: opus-5-5 --- internal/smallwebwaf/healthcheck_test.go | 19 +++++++++++++------ 1 file changed, 13 insertions(+), 6 deletions(-) diff --git a/internal/smallwebwaf/healthcheck_test.go b/internal/smallwebwaf/healthcheck_test.go index 3089f22..a5ffd60 100644 --- a/internal/smallwebwaf/healthcheck_test.go +++ b/internal/smallwebwaf/healthcheck_test.go @@ -60,8 +60,19 @@ func TestHealthCheck(t *testing.T) { }, 0, "") } - app.Close() - wantHealthCheck(t, env, 1, "unhealthy: connect to the app: ") + // Another test can take a closed server's port at once, so the app's + // address here is a multicast address, to which TCP cannot connect. + wantHealthCheck(t, map[string]string{ + listenAddr: ":" + port, + upstreamURL: "http://224.0.0.1", + }, 1, "unhealthy: connect to the app: ") + + // Listening on port 0 takes any free port, so nothing listens on port 0 + // itself. + wantHealthCheck(t, map[string]string{listenAddr: ":0"}, 1, + "unhealthy: ask smallwebwaf: ") + wantHealthCheck(t, map[string]string{listenAddr: "8080"}, 1, + "unhealthy: invalid setting: SWWAF_LISTEN_ADDR: ") stop() @@ -70,10 +81,6 @@ func TestHealthCheck(t *testing.T) { case <-time.After(waitLimit): t.Fatal("still running after being told to stop") } - - wantHealthCheck(t, env, 1, "unhealthy: ask smallwebwaf: ") - wantHealthCheck(t, map[string]string{listenAddr: "8080"}, 1, - "unhealthy: invalid setting: SWWAF_LISTEN_ADDR: ") } func TestHealthCheckRefusesAnArgument(t *testing.T) { From 718a2c94b2f377d512177b3268f44b25317b88d9 Mon Sep 17 00:00:00 2001 From: clawbot <35+clawbot@noreply.example.org> Date: Wed, 4 Nov 2026 23:02:39 +0100 Subject: [PATCH 3/9] Default request limits per minute, hour and day, sized for API use (closes #176) The default request limits now suit programmatic use of an API, such as an automation fleet on another host, and still catch abusive bots: - per minute: 500, room for a fleet's busiest minute and a few first visits with an empty cache, and passed within seconds by a client firing as fast as it can; - per hour: 20000, passed by a client that keeps up 6 a second; - per day: 300000, passed by a scraper that keeps up 4 a second. The byte limits and ban lengths are unchanged. SPEC.md, README.md and docs/settings.md give the reasons; docs/gitea.md redoes the Actions runner sums. Judgement call: the docs/alerts.md ban example shows the minute limit. Model: opus-5-5 --- README.md | 19 ++++++--- SPEC.md | 50 ++++++++++++++++-------- docs/alerts.md | 18 ++++----- docs/gitea.md | 12 +++--- docs/settings.md | 18 +++++++-- internal/config/config.go | 6 +-- internal/config/config_test.go | 12 +++--- internal/smallwebwaf/smallwebwaf_test.go | 6 +-- 8 files changed, 90 insertions(+), 51 deletions(-) diff --git a/README.md b/README.md index 712ad5a..3caa55a 100644 --- a/README.md +++ b/README.md @@ -17,11 +17,20 @@ reference documents under [`docs/`](docs/) are listed under "Documents" below. ## What it protects against - Scrapers that fetch every page of a site, all day. Rate limiting gives each - client 6000 requests a minute, 100,000 an hour and 1,000,000 a day. Byte - limits give it 4 GiB a minute, 20 GiB an hour and 200 GiB a day. A client over - a limit is refused and banned. The first ban lasts an hour. Each repeat within - a day makes the next ban three times as long. A ban that would last more than - seven days is permanent. + client 500 requests a minute, 20,000 an hour and 300,000 a day. That leaves + room for programmatic use of an API, such as an automation fleet on another + host. A fleet of about 20 agents makes about 120 requests a minute at its + busiest. A gitea page opened with an empty cache loads about 40 files, so a + few such first visits fit in the minute limit too. The minute limit catches, + within seconds, a client firing requests as fast as it can. The hour limit + catches a client that keeps up more than about 5.5 a second for an hour. The + day limit catches a client that keeps up 4 a second around the clock. Counting + cannot catch a client that stays under all three limits, such as one that + keeps up 3 a second; only the protections below can. Byte limits give each + client 4 GiB a minute, 20 GiB an hour and 200 GiB a day. A client over a limit + is refused and banned. The first ban lasts an hour. Each repeat within a day + makes the next ban three times as long. A ban that would last more than seven + days is permanent. - Scanners that probe for secrets, such as `/.env` or `/.git/config`. The default rule file bans such a client for seven days at its first probe. Its next request during the ban makes the ban permanent. The default rule file diff --git a/SPEC.md b/SPEC.md index 0b3d7b6..127a35f 100644 --- a/SPEC.md +++ b/SPEC.md @@ -369,22 +369,38 @@ Conventions: owner's own networks. `SWWAF_RATE_LIMIT_EXEMPT_NETS` bypasses the request and byte limits only; the error burst, attack detection and bans still apply. - Many people can share one address, as in an office, on a mobile carrier or - through Tor. They are one client, and share its limits and its bans. The - default limits are sized for about 40 busy people on one address. A known - shared address busier than that belongs in `SWWAF_ALLOW_NETS` or + through Tor. They are one client, and share its limits and its bans. A known + shared address busier than the limits allow belongs in `SWWAF_ALLOW_NETS` or `SWWAF_RATE_LIMIT_EXEMPT_NETS`. ### Request rate limits, per client -Breaking one bans the client, so the defaults sit about twice above what 40 -people busy with heavy web apps make together on one address. Each such person -makes about 75 requests in a busy minute, and 1,250 an hour for ten hours a day. +Breaking one bans the client. The defaults suit programmatic use of an API, such +as an automation fleet on another host, and still stop abusive bots. A fleet of +20 agents, each making a request every 10 seconds at its busiest, makes 120 +requests a minute and 7,200 an hour. Working around the clock, it makes 172,800 +a day. It stays under all three limits. -A client slower than those 40 people cannot be told from them by counting. Only -the other checks can catch an abuser that slow. They include the trap paths, the -rule files, the Core Rule Set and the blocklists. An abuser that trips none of -them is not caught. A machine that talks to the app all day can still pass the -day limit. Its address belongs in `SWWAF_RATE_LIMIT_EXEMPT_NETS`. +- Per minute: 500, about 8 requests a second. It leaves room for the fleet's 120 + a minute and a few quick first visits with an empty cache. A gitea page opened + with an empty cache loads about 40 files. A client firing requests as fast as + it can passes the limit within seconds. +- Per hour: 20,000, about 5.5 requests a second for a whole hour. A client that + keeps up 6 a second makes 21,600 in an hour, and passes it, at 360 a minute, + under the minute limit. The fleet's 7,200 an hour is about a third of it. +- Per day: 300,000, about 3.5 requests a second around the clock. A scraper that + keeps up 4 a second makes 345,600 a day, and passes it. At 240 a minute and + 14,400 an hour, it stays under the other two limits. + +Counting cannot catch a client that stays under all three limits, such as one +that keeps up 3 requests a second. Only the other checks can catch an abuser +that slow. They include the trap paths, the rule files, the Core Rule Set and +the blocklists. An abuser that trips none of them is not caught. A machine that +keeps up more than 500 requests a minute passes the minute limit. One that keeps +up more than about 5.5 a second for an hour passes the hour limit. One that +keeps up more than about 3.5 a second all day passes the day limit, as two busy +Actions runners on one address do. Such a machine's address belongs in +`SWWAF_RATE_LIMIT_EXEMPT_NETS`. `SWWAF_RATE_LIMIT_EXEMPT_PATHS` takes path prefixes out of the count, such as static assets or health checks; their bytes are not counted either. A prefix is @@ -397,12 +413,12 @@ matched the same way. ### Byte limits, per client -Each of the same 40 people moves about 256 MiB an hour, for ten hours a day. The -hour and day limits sit about twice above what they move together. A response's -bytes are counted when it ends (see step 15). So the minute limit holds two of -the largest responses, with their requests, ending in the same minute. No single -download then breaks a limit. An operator who raises `SWWAF_RESPONSE_MAX_BYTES` -raises the byte limits with it. +The hour and day limits sit about twice above what 40 people busy with heavy web +apps move together on one address. Each moves about 256 MiB an hour, for ten +hours a day. A response's bytes are counted when it ends (see step 15). So the +minute limit holds two of the largest responses, with their requests, ending in +the same minute. No single download then breaks a limit. An operator who raises +`SWWAF_RESPONSE_MAX_BYTES` raises the byte limits with it. ### Size and time limits diff --git a/docs/alerts.md b/docs/alerts.md index 916912d..4d2bbfd 100644 --- a/docs/alerts.md +++ b/docs/alerts.md @@ -35,34 +35,34 @@ on one line: ```json { "instance": "fsn1app1/gitea", - "time": "2026-10-06T12:00:00.123461Z", + "time": "2026-10-06T12:00:50.123461Z", "event": "ban", "client": "203.0.113.9", "netblock": "203.0.113.9/32", "asn": "", "as_name": "", "country": "", - "reason": "requests per minute over the limit of 6000", + "reason": "requests per minute over the limit of 500", "detail": { - "ban_expires": "2026-10-06T13:00:00.123Z", + "ban_expires": "2026-10-06T13:00:50.123Z", "cause": "limit", "notes": { "asn": "", "as_name": "", "country": "", "kind": "requests", - "limit": 6000, + "limit": 500, "window": "minute", - "count": 6001, + "count": 501, "request": { - "time": "2026-10-06T12:00:00.123456789Z", + "time": "2026-10-06T12:00:50.123456789Z", "method": "GET", "host": "app.example", - "path": "/owner/repo/commits/branch/main?page=812", + "path": "/owner/repo/commits/branch/main?page=501", "status": 403, "user_agent": "scraper/1.0" }, - "requests": 8210, + "requests": 501, "refused": 0, "earlier_bans": { "limit": 0, @@ -89,7 +89,7 @@ on one line: - `asn`, `as_name` and `country`: the client's, as known when the alert was raised. They are empty when GeoJS had not answered yet. - `reason`: a short sentence, such as - `requests per minute over the limit of 6000` or `listed by a blocklist`. + `requests per minute over the limit of 500` or `listed by a blocklist`. - `detail`: what is particular to the event: - for a ban, its `cause`, `ban_expires` and `notes` (the notes are listed under `bans.json` in [`state-files.md`](state-files.md)); diff --git a/docs/gitea.md b/docs/gitea.md index df40899..1edf488 100644 --- a/docs/gitea.md +++ b/docs/gitea.md @@ -34,11 +34,13 @@ checked again when gitea or the Core Rule Set changes version. day. One older than version 0.4 (April 2026) asks for work every 2 seconds, 43,200 requests a day. While it runs a job, it reports the job's log and state every second instead, two requests a second. So one runner busy all day sends - 172,800 requests, under the request limits. Six such runners on one address, - each busy all day, pass the day limit of 1,000,000 within two days; five stay - under it. Their address is then banned, their running jobs fail, and each - repeat within a day makes the ban longer. Newer runners ask for work less - often. Put the runners' addresses in `SWWAF_RATE_LIMIT_EXEMPT_NETS`. + 120 requests a minute and 172,800 a day, under the request limits. Two such + runners on one address, each busy all day, make 345,600 and pass the day limit + of 300,000. Three busy at once make 21,600 an hour and pass the hour limit of + 20,000. Five busy at once make 600 a minute and pass the minute limit of 500. + Their address is then banned, their running jobs fail, and each repeat within + a day makes the ban longer. Newer runners ask for work less often. Put the + runners' addresses in `SWWAF_RATE_LIMIT_EXEMPT_NETS`. ## What the Core Rule Set lets through diff --git a/docs/settings.md b/docs/settings.md index ad41aee..83d995c 100644 --- a/docs/settings.md +++ b/docs/settings.md @@ -99,9 +99,21 @@ that take `off`, the `_FILE` form, and what each group does and why. ## Request rate limits, per client -- `SWWAF_RATE_LIMIT_PER_MINUTE` (default `6000`), `SWWAF_RATE_LIMIT_PER_HOUR` - (default `100000`) and `SWWAF_RATE_LIMIT_PER_DAY` (default `1000000`): the - most requests a client may make in each window. A count or `off`. +- `SWWAF_RATE_LIMIT_PER_MINUTE` (default `500`, about 8 a second): the most + requests a client may make in a minute. A count or `off`. The default catches, + within seconds, a client firing requests as fast as it can. It leaves room for + a fleet of about 20 agents at its busiest, about 120 requests a minute. It + also leaves room for a few quick first visits to gitea with an empty cache, + about 40 files each. +- `SWWAF_RATE_LIMIT_PER_HOUR` (default `20000`, about 5.5 a second): the most + requests a client may make in an hour. A count or `off`. The default leaves + room for an automation fleet at work. It catches a client that keeps up more + than about 5.5 a second for an hour. +- `SWWAF_RATE_LIMIT_PER_DAY` (default `300000`, about 3.5 a second): the most + requests a client may make in a day. A count or `off`. The default catches a + scraper that keeps up 4 a second around the clock. +- Counting cannot catch a client that stays under all three limits (see "Request + rate limits, per client" in [`SPEC.md`](../SPEC.md)). - `SWWAF_RATE_LIMIT_EXEMPT_PATHS` (default empty): path prefixes exempt from rate limiting and the byte limits, such as `/assets/`. diff --git a/internal/config/config.go b/internal/config/config.go index 0cc5def..5f01c82 100644 --- a/internal/config/config.go +++ b/internal/config/config.go @@ -506,9 +506,9 @@ func FromEnvironment(lookupEnv func(string) (string, bool)) (*Config, error) { AllowNets: env.netblocks("SWWAF_ALLOW_NETS", ""), RateLimitExemptNets: env.netblocks("SWWAF_RATE_LIMIT_EXEMPT_NETS", ""), DenyNets: env.netblocks("SWWAF_DENY_NETS", ""), - RateLimitPerMinute: env.count("SWWAF_RATE_LIMIT_PER_MINUTE", "6000"), - RateLimitPerHour: env.count("SWWAF_RATE_LIMIT_PER_HOUR", "100000"), - RateLimitPerDay: env.count("SWWAF_RATE_LIMIT_PER_DAY", "1000000"), + RateLimitPerMinute: env.count("SWWAF_RATE_LIMIT_PER_MINUTE", "500"), + RateLimitPerHour: env.count("SWWAF_RATE_LIMIT_PER_HOUR", "20000"), + RateLimitPerDay: env.count("SWWAF_RATE_LIMIT_PER_DAY", "300000"), RateLimitExemptPaths: env.pathPrefixes("SWWAF_RATE_LIMIT_EXEMPT_PATHS", ""), BytesLimitPerMinute: env.size("SWWAF_BYTES_LIMIT_PER_MINUTE", "4G"), BytesLimitPerHour: env.size("SWWAF_BYTES_LIMIT_PER_HOUR", "20G"), diff --git a/internal/config/config_test.go b/internal/config/config_test.go index ff61910..032a3e0 100644 --- a/internal/config/config_test.go +++ b/internal/config/config_test.go @@ -262,9 +262,9 @@ func TestDefaults(t *testing.T) { UpstreamResponseTimeout: 30 * time.Minute, RequestMaxBytes: 100 << 20, ResponseMaxBytes: 1 << 30, - RateLimitPerMinute: 6000, - RateLimitPerHour: 100000, - RateLimitPerDay: 1000000, + RateLimitPerMinute: 500, + RateLimitPerHour: 20000, + RateLimitPerDay: 300000, BanResponse: 403, LimitBanDuration: time.Hour, LimitBanRepeatWindow: 24 * time.Hour, @@ -2375,9 +2375,9 @@ func TestLogsEachSettingWithItsValue(t *testing.T) { allowNets: "", rateLimitExemptNets: "", denyNets: "", - rateLimitPerMinute: "6000", - rateLimitPerHour: "100000", - rateLimitPerDay: "1000000", + rateLimitPerMinute: "500", + rateLimitPerHour: "20000", + rateLimitPerDay: "300000", rateLimitExemptPaths: "", bytesLimitPerMinute: "4G", bytesLimitPerHour: "20G", diff --git a/internal/smallwebwaf/smallwebwaf_test.go b/internal/smallwebwaf/smallwebwaf_test.go index 0aee935..1d30bf4 100644 --- a/internal/smallwebwaf/smallwebwaf_test.go +++ b/internal/smallwebwaf/smallwebwaf_test.go @@ -1244,9 +1244,9 @@ func wantStartingLine(t *testing.T, line map[string]any, appURL, dir string) { "SWWAF_ALLOW_NETS": "", rateLimitExemptNets: "", "SWWAF_DENY_NETS": "", - "SWWAF_RATE_LIMIT_PER_MINUTE": "6000", - "SWWAF_RATE_LIMIT_PER_HOUR": "100000", - rateLimitPerDay: "1000000", + "SWWAF_RATE_LIMIT_PER_MINUTE": "500", + "SWWAF_RATE_LIMIT_PER_HOUR": "20000", + rateLimitPerDay: "300000", "SWWAF_RATE_LIMIT_EXEMPT_PATHS": "", "SWWAF_DENIED_COUNTRIES": "", "SWWAF_EXCLUSIVELY_ALLOWED_COUNTRIES": "", From 2eeea136e86d4af02177a1107206f9edc91787e1 Mon Sep 17 00:00:00 2001 From: clawbot <35+clawbot@noreply.example.org> Date: Thu, 5 Nov 2026 00:32:08 +0100 Subject: [PATCH 4/9] servewaf answers rproxy's question about each request and takes its report (closes #170) `smallwebwaf servewaf` runs today's checks on the visitor's request each question carries, answers rproxy, and passes nothing to an app. rproxy reports every request servewaf answered. A report counts an allowed request's bytes and writes servewaf's second log line. The table holds a copy of what a report needs, never the request, so a report cannot reach a request being handled. Judgement call: a question's body is read before any check; over 64 KiB gets 400. Judgement call: a report's bytes and durations have ceilings, so they cannot overflow. Judgement call: a `close` disposition for SWWAF_BAN_RESPONSE=close. Model: opus-5-5 --- README.md | 5 + SPEC.md | 211 ++- cmd/smallwebwaf/main.go | 5 +- docs/admin-endpoints.md | 5 + docs/alerts.md | 5 +- docs/metrics.md | 11 + docs/request-log.md | 63 + docs/settings.md | 20 +- internal/alerts/alerts.go | 15 +- internal/alerts/alerts_test.go | 4 +- internal/anomaly/anomaly.go | 3 + internal/metrics/metrics.go | 60 +- internal/proxy/admin.go | 9 + internal/proxy/anomaly_internal_test.go | 8 +- internal/proxy/bans.go | 9 +- internal/proxy/bans_test.go | 4 +- internal/proxy/coreruleset.go | 1 + internal/proxy/metrics_test.go | 2 +- internal/proxy/proxy.go | 85 +- internal/proxy/proxy_test.go | 12 +- internal/proxy/reputation.go | 1 + internal/proxy/request.go | 152 +- internal/proxy/servewaf.go | 464 ++++++ internal/proxy/servewaf_test.go | 1642 ++++++++++++++++++++++ internal/requestlog/requestlog.go | 17 +- internal/smallwebwaf/servewaf_test.go | 210 +++ internal/smallwebwaf/smallwebwaf.go | 29 +- internal/smallwebwaf/smallwebwaf_test.go | 10 +- 28 files changed, 2919 insertions(+), 143 deletions(-) create mode 100644 internal/proxy/servewaf.go create mode 100644 internal/proxy/servewaf_test.go create mode 100644 internal/smallwebwaf/servewaf_test.go diff --git a/README.md b/README.md index 3caa55a..504e4be 100644 --- a/README.md +++ b/README.md @@ -348,6 +348,9 @@ pinned. [`docs/settings.md`](docs/settings.md)). - The state files live in `/var/lib/smallwebwaf`. A volume mounted there keeps them across a deploy. +- `smallwebwaf servewaf` runs the same checks for the sites that ask it about + each request, and passes nothing to an app (see "`servewaf`" in + [`SPEC.md`](SPEC.md)). - `docker stop` stops both processes at once. `smallwebwaf` gives the requests in progress five seconds to finish, so the app must keep answering for five seconds after `SIGTERM`. @@ -360,6 +363,8 @@ what each does. ## TODO - Planned, as "Planned, not built" in [`SPEC.md`](SPEC.md) describes: + - `smallwebwaf rproxy`, the reverse proxy in front of each app that asks + `smallwebwaf servewaf` about each request; - sharing bans between the instances of several apps; - sending the log over RELP. diff --git a/SPEC.md b/SPEC.md index 127a35f..03b382a 100644 --- a/SPEC.md +++ b/SPEC.md @@ -3,7 +3,7 @@ `smallwebwaf` is a web application firewall and reverse proxy for one app. It runs inside the app's container, between traefik and the app. This document says what it must do and why, each requirement once. Everything in it is built, -except the last section, "Planned, not built". `README.md` says how to run it. +except what the last section lists. `README.md` says how to run it. `EVALUATION.md` says why no existing tool was adopted. The reference documents under `docs/` give the settings, the formats and the lists. This document links to them. @@ -47,14 +47,16 @@ to them. - `internal/smallwebwaf`: the process. It reads the settings, the rule files, the lookup database and the state files, listens, and serves until `SIGTERM` or `SIGINT`. Then it stops, writing the state files. Run as - `smallwebwaf healthcheck`, it is the image's health check instead. + `smallwebwaf servewaf`, it answers `rproxy` instead of passing requests to + an app (see "`servewaf`"). Run as `smallwebwaf healthcheck`, it is the + image's health check instead. - `internal/config`: reads the settings, the one place they are read. - `internal/proxy`: what happens to each request. It works out the client and runs the checks in `check`. It passes the request to the app with the standard library's `httputil.ReverseProxy`, within the timeouts and size limits, and answers its own endpoints in `answerAdmin`. Once the request has ended, it counts bytes, refusals and anomalies, and writes the log - line. + line. It also holds `servewaf`'s question, answer and report. - `internal/ratelimit`: the table of clients. It counts requests, bytes and refusals, and keeps each client's history. - `internal/bans`: the bans. It holds each netblock's bans with their notes, @@ -1022,6 +1024,205 @@ answers. costs). - A ban made through the endpoints starts at once. +## `servewaf` + +`smallwebwaf servewaf` runs the checks for `smallwebwaf rproxy`, a small reverse +proxy in front of each app. `rproxy` asks `servewaf` about each visitor's +request. `servewaf` answers `rproxy` and passes nothing to an app itself. One +`servewaf` serves many sites. + +This section is the agreement between the two. The header names, the fields and +the statuses below are exact. Each `SWWAF_` setting and check keeps the meaning +this document gives it, except as this section says. + +`servewaf` starts as `smallwebwaf` with no command does. It reads the settings, +the rule files, the lookup database and the state files, and listens at +`SWWAF_LISTEN_ADDR`. It stops in the same orderly way, writing the state files. +`docs/settings.md` lists the settings it ignores. + +### The question + +A request carrying `X-Smallwebwaf-Client` is `rproxy` asking about one visitor's +request. It is the visitor's request as the visitor sent it: + +- the method; +- the path and query as sent; +- `Host`; +- every header the visitor sent, including its own `X-Forwarded-For`; +- the first 64 KiB of the body, or all of it when shorter. + +A body whose length the visitor did not announce comes chunked. Any other body +comes with a `Content-Length` of the part it carries. + +`rproxy` sets these headers, replacing any the visitor sent, and adds no others: + +- `X-Smallwebwaf-Client`: the visitor's address, which `rproxy` has worked out. + `servewaf` takes it as the client, and groups an IPv6 address as "What happens + to a request" says. +- `X-Smallwebwaf-Peer`: the address of `rproxy`'s own TCP peer, for the log + line's `peer_ip`. +- `X-Smallwebwaf-Content-Length`: the body length the visitor announced. It is + absent when the visitor announced none. +- `X-Request-ID`: the request's id, which `rproxy` makes fresh for every request + when it arrives. `servewaf` takes it as given and never makes one. +- `X-Forwarded-Proto`: the scheme. + +Before any check, `servewaf` removes the three `X-Smallwebwaf-` headers. It also +sets `Content-Length` to the length the visitor announced, or removes it. So the +rule files and the Core Rule Set see the headers the visitor sent, with +`rproxy`'s `X-Request-ID` and `X-Forwarded-Proto`. + +`servewaf` refuses a question with `400` when: + +- `X-Smallwebwaf-Client` is not an address, or is an address with a zone, such + as `2001:db8::1%eth0`; +- `X-Request-ID` is absent, empty, over 128 bytes or not valid UTF-8; +- `X-Request-ID` is the id of a question `servewaf` is still handling, or of a + request it holds (see "The report"); +- the body is over 64 KiB. + +For a refused question, `servewaf` writes a process warning. It runs no check, +writes no request log line and counts nothing in the metrics. + +These do not refuse a question: + +- `X-Smallwebwaf-Peer` absent or not an address: the log line's `peer_ip` is + `invalid IP`. +- `X-Smallwebwaf-Content-Length` not a whole number, or negative: it counts as + absent. +- `X-Forwarded-Proto` absent: the log line's `scheme` is `http`. +- One of the five headers above sent more than once: `servewaf` uses its first + value. + +`servewaf` reads the whole body before any check. A question whose body stops +arriving gets no answer. `servewaf` closes the connection once the 60 seconds +below run out, or as soon as `rproxy` closes its side. It writes a process +warning, runs no check, writes no request log line and counts nothing in the +metrics. + +### The answer + +`servewaf` runs steps 3 to 12 of "What happens to a request", in that order. +Every answer but the last below carries `X-Smallwebwaf-Disposition`. + +- `X-Smallwebwaf-Disposition: allow`, status `200`, no body. `rproxy` passes the + request to the app. With `SWWAF_ADD_LOOKUP_HEADERS` the answer also carries + `X-Client-ASN` and `X-Client-Country`, which `rproxy` adds to the request it + passes on. A header whose value is unknown is left out. +- `X-Smallwebwaf-Disposition: deny`, with the status, headers and body the + visitor is to get. `rproxy` gives them to the visitor, less this header. The + status is one of: + - `SWWAF_BAN_RESPONSE`, for a refusal; + - `403`, for a `block` rule or the Core Rule Set; + - `404`, for a path under `/_smallwebwaf/` that passes the checks, so that + the app never sees that prefix. +- `X-Smallwebwaf-Disposition: close`, status `200`, no body. This takes the + place of a refusal while `SWWAF_BAN_RESPONSE` is `close`. `rproxy` closes the + visitor's connection without an answer. +- No such header, status `400`, for a question `servewaf` refuses. `rproxy` + treats this answer as it treats no answer at all. + +In `observe` mode a request that `enforce` mode would refuse is answered +`allow`, and its log line has `would_action`. A path under `/_smallwebwaf/` is +still answered `404`. + +The Core Rule Set inspects the body a question carries only while +`SWWAF_WAF_BODY_LIMIT` is set. It inspects at most 64 KiB less one byte. So it +does not inspect a JSON or XML body of 64 KiB or more, which may have been cut +short. + +### The report + +Once the visitor's answer has ended, `rproxy` sends `POST /_smallwebwaf/report` +for each request `servewaf` answered with `allow`, `deny` or `close`. It sends +none after a `400` or no answer. The report carries no `X-Smallwebwaf-Client`. +Its body is one JSON object with all of these fields, named as the request log's +fields of the same meaning: + +- `request_id`: the question's `X-Request-ID`; +- `status`: what the visitor got, `0` if nothing, otherwise from `100` to `599`; +- `request_bytes` and `response_bytes`: the body bytes each way, with what a + WebSocket carried each way, each a whole number from `0` to `1125899906842624` + (1 PiB); +- `duration_total`: milliseconds from the request's headers being read to the + end of the answer, from `0` to `1000000000000` (about 31 years); +- `duration_upstream_total`: milliseconds from the hand-over to the app to the + end of the answer, in the same range, and `0` for a request not passed to the + app. + +`servewaf` answers `400` and writes a process warning when the body: + +- is not one JSON object, such as `null`, or an object followed by other data; +- lacks one of these fields, or gives it as `null`; +- gives a value of the wrong type, or out of its range; +- is over 1 MiB. + +It answers `204` to any other report, also for a request it does not hold, which +it ignores. It ignores a field not listed here. `rproxy` does not act on the +answer. + +For a request it holds, `servewaf` writes the request's last log line (see "The +log"). For a request it allowed, it first: + +- counts the bytes as `SWWAF_BYTES_COUNT` says, toward the client's byte limits + at the percentages it gave that request; +- bans a client that passes a byte limit; +- adds the request with its status to the client's history, the anomaly counters + and the metrics. + +A denied request's counting is finished at its decision, so its report counts +nothing. + +`servewaf` holds each answered request until its report comes, up to 10,000 +requests. Past that, the request answered longest ago is dropped. A request +whose report never comes gets no last log line. If it was allowed, it is in +neither the client's history nor the metrics, and its bytes are not counted. + +`servewaf` holds only the decision's log line and what counting needs. It never +holds the request or response objects, and no `Authorization` or `Cookie`. A +report reaches only what is held, so it can never reach a question still being +handled. `servewaf` ignores a report naming such a question. + +### The log + +Each request gets three request log lines, joined by `request_id`: + +- `rproxy`'s short line; +- `servewaf`'s line at its decision; +- `servewaf`'s line when the report comes, with everything known: the decision + line's fields and the report's together. + +The repetition is intended. The field `servewaf_line` says which of `servewaf`'s +two lines a line is. `docs/request-log.md` gives the fields. + +### `servewaf`'s own endpoints + +A request without `X-Smallwebwaf-Client` is for `servewaf` itself, at its own +address on the private network. The internet must not be able to reach that +address. Its endpoints are: + +- the health check, the metrics, and the ban and client endpoints, as + `docs/admin-endpoints.md` lists them, with the same tokens; +- the report. + +Anything else gets `404`. No check runs on these requests, since their peers are +`rproxy` and the admin's tools. The report needs no token, so anything on the +private network can send one or ask a question. + +A request and its answer have 60 seconds together, from the request's headers. +`servewaf` closes a connection idle for 120 seconds. It takes a request line and +headers up to 1 MiB, Go's default, so that it never refuses a question for the +size of a visitor's headers. + +### Sites + +One `servewaf` keeps one table of clients, counters and bans for every site that +asks it. A client banned through one site is refused on all of them, and its +requests on all of them count toward one set of limits. Log lines, alerts raised +by a request and ban notes name the site by its host. + +`docs/metrics.md` says how its metrics count. + ## Deployment `smallwebwaf` runs inside the app's own container: the app's Dockerfile builds @@ -1187,9 +1388,9 @@ and `script/gitea-requests` checks those notes. ## Planned, not built -These two points are specified as later work, and nothing in the code does them -yet: +These points are specified as later work, and nothing in the code does them yet: +- Planned: `smallwebwaf rproxy`, the reverse proxy in front of each app. - Planned: sharing bans between the instances of several apps, by running one CrowdSec engine per host that every instance reports its bans to. It is reconsidered once real ban volumes are known. Until then each instance keeps diff --git a/cmd/smallwebwaf/main.go b/cmd/smallwebwaf/main.go index 108247c..9370da4 100644 --- a/cmd/smallwebwaf/main.go +++ b/cmd/smallwebwaf/main.go @@ -1,6 +1,7 @@ // Command smallwebwaf is a web application firewall for one app: it runs -// between traefik and the app, and passes requests through within its -// limits. Everything it does is in internal/smallwebwaf. +// between traefik and the app and passes requests through within its limits, +// or, as `smallwebwaf servewaf`, answers rproxy's questions about them. +// Everything it does is in internal/smallwebwaf. package main import ( diff --git a/docs/admin-endpoints.md b/docs/admin-endpoints.md index dde1d67..2f856bd 100644 --- a/docs/admin-endpoints.md +++ b/docs/admin-endpoints.md @@ -46,3 +46,8 @@ curl -H "Authorization: Bearer $TOKEN" -X DELETE \ curl -H "Authorization: Bearer $TOKEN" \ https://app.example/_smallwebwaf/clients/203.0.113.9 ``` + +## `servewaf` + +`smallwebwaf servewaf` answers these endpoints at its own address. No check runs +on those requests (see "`servewaf`" in [`SPEC.md`](../SPEC.md)). diff --git a/docs/alerts.md b/docs/alerts.md index 4d2bbfd..201a37e 100644 --- a/docs/alerts.md +++ b/docs/alerts.md @@ -78,6 +78,9 @@ on one line: - `instance`: `SWWAF_INSTANCE_NAME`. `time`: when the alert was raised. `event`: one of the events above. +- `host`: the site of the request that raised the alert. Only `servewaf` sends + this field, since it serves many sites, and not for `source_failure`, + `file_error` and `summary`. - `client`: the address whose request raised the alert. Empty for `source_failure`, `file_error` and `summary`. - `netblock`: empty for `source_failure`, `file_error` and `summary`; otherwise @@ -110,7 +113,7 @@ Slack and ntfy get a message with a title and a text. The title is the instance and the event, such as `fsn1app1/gitea: ban`. The text is the `reason`, then one line for each of these that the alert has: -- the client, the netblock and the country; +- the host, the client, the netblock and the country; - the file, the source, the error and the mode from `detail`; - the suppressed repeats, if any. diff --git a/docs/metrics.md b/docs/metrics.md index 87877e7..3bb111a 100644 --- a/docs/metrics.md +++ b/docs/metrics.md @@ -79,3 +79,14 @@ By `file`: `smallwebwaf_alerts_failed_total`, `smallwebwaf_alerts_suppressed_total` and `smallwebwaf_alerts_dropped_total`; - Go's own `go_` metrics and the process's `process_` metrics. + +## `servewaf` + +`smallwebwaf servewaf` counts a denied request at its decision. It counts an +allowed request when its report comes, with the status, bytes and times the +report gives. These are not counted: + +- a report itself; +- an allowed request whose report never comes; +- a question refused with `400`; +- a question whose body stopped arriving. diff --git a/docs/request-log.md b/docs/request-log.md index 10c67f3..1d584db 100644 --- a/docs/request-log.md +++ b/docs/request-log.md @@ -117,3 +117,66 @@ All in milliseconds: connection; - `duration_upstream_first_byte`: until the first byte of the answer; - `duration_upstream_total`: until the end of the answer. + +## `servewaf`'s lines + +`smallwebwaf servewaf` writes two lines for each question it answers, joined by +`request_id`. The field `servewaf_line` says which of the two a line is: + +- `decision`: the line it writes once it has answered the question; +- `report`: the line it writes when `rproxy`'s report comes, with everything + known. The repetition is intended. + +A request to its own endpoints writes one line, without `servewaf_line`. A +report writes no line but the `report` line of the request it names. + +These questions write no line: + +- a question refused with `400` (see "The question" under "`servewaf`" in + [`SPEC.md`](../SPEC.md)); +- a question whose body stopped arriving, which gets no answer. + +### The `decision` line + +It has the fields above, with these meanings where they differ: + +- `client_ip`: `X-Smallwebwaf-Client`. +- `peer_ip`: `X-Smallwebwaf-Peer`. +- `host`: the site the visitor asked for. +- `protocol`: the question's own, `HTTP/1.1`. +- `content_length`: `X-Smallwebwaf-Content-Length`. +- `request_id`: the question's `X-Request-ID`. +- `status`: what `servewaf` answered: `200` for `allow`, the status of `deny`, + or `0` for `close`, as the visitor is then sent nothing. +- `request_bytes`: how much of the question's body the Core Rule Set read. +- `response_bytes`: the body of `servewaf`'s answer. +- `action`: `forward` for `allow`, and `admin` for the `404` for a path under + `/_smallwebwaf/`. +- `aborted`: true when `rproxy` went away before the answer. +- `duration_total`: from `servewaf` having read the question's body to the line. +- `time`: when `servewaf` had read the question's body. + +The line leaves out `upstream_status`, `websocket` and the +`duration_upstream_...` fields, since `servewaf` passes nothing to the app. + +The bytes of an allowed request are counted when its report comes. So `counts` +holds the client's bytes without this request's. + +### The `report` line + +It has every field of the `decision` line, and `time` stays as the `decision` +line gives it. These fields say what the report says: + +- `status`: what the visitor got, `0` if nothing; +- `request_bytes` and `response_bytes`: the body bytes each way, with what a + WebSocket carried each way; +- `duration_total`: from the request's headers being read by `rproxy` to the end + of the answer; +- `duration_upstream_total`: from the hand-over to the app to the end of the + answer, for an allowed request. + +For an allowed request, the report's counting also changes these fields: + +- `counts`: the byte totals now include this request's; +- `limit_hit`, `offence` and `ban_expires`: set when the report's bytes broke a + byte limit. diff --git a/docs/settings.md b/docs/settings.md index 83d995c..a59e4df 100644 --- a/docs/settings.md +++ b/docs/settings.md @@ -339,4 +339,22 @@ A few limits are fixed rather than settings: - at most 100,000 DNS blocklist verdicts and 100,000 AbuseIPDB scores are kept; - at most 1,000 DNS blocklist queries are under way at once; - at most 20,000 anomaly counters are kept; -- each alert destination's queue holds at most 1,000 alerts. +- each alert destination's queue holds at most 1,000 alerts; +- `servewaf` holds at most 10,000 answered requests awaiting their reports. + +## `servewaf` + +`smallwebwaf servewaf` uses every setting here but these, which it ignores: + +- `SWWAF_UPSTREAM_URL` and `SWWAF_TRUSTED_PROXIES`; +- the five timeouts under "Size and time limits"; +- `SWWAF_CLIENT_REQUEST_HEADER_MAX_BYTES`, `SWWAF_REQUEST_MAX_BYTES` and + `SWWAF_RESPONSE_MAX_BYTES`. + +An invalid value of one of these still stops the start, and the line of settings +at start still shows them. + +`SWWAF_ADD_LOOKUP_HEADERS` puts the headers on the answer to `rproxy`, not on a +request to an app. `SWWAF_WAF_BODY_LIMIT` has an effect up to 64 KiB less one +byte, whatever its value. A question carries no more of a body than 64 KiB (see +"`servewaf`" in [`SPEC.md`](../SPEC.md)). diff --git a/internal/alerts/alerts.go b/internal/alerts/alerts.go index 7864637..422433c 100644 --- a/internal/alerts/alerts.go +++ b/internal/alerts/alerts.go @@ -150,13 +150,16 @@ type Params struct { // Alert is one alert, as the webhook is sent it and alerts.json holds it, // with the fields of "The JSON object" in docs/alerts.md. ASN, ASName and -// Country are, for a ban, the client's as the ban's notes give them. +// Country are, for a ban, the client's as the ban's notes give them. Host is +// the site of the request that raised the alert, while one servewaf serves +// many sites, and "" otherwise. // //nolint:tagliatelle // docs/alerts.md names the alert's fields in snake_case type Alert struct { Instance string `json:"instance"` Time time.Time `json:"time"` Event string `json:"event"` + Host string `json:"host,omitempty"` Client netip.Addr `json:"client"` Netblock netip.Prefix `json:"netblock"` ASN string `json:"asn"` @@ -899,12 +902,16 @@ func title(alert *Alert) string { } // text returns the text of alert in Slack and ntfy: its reason, then a -// line for each of its client, netblock and country, the file, source, -// error and mode its detail gives, and its suppressed repeats, that it -// has. +// line for each of its host, client, netblock and country, the file, +// source, error and mode its detail gives, and its suppressed repeats, that +// it has. func text(alert *Alert) string { lines := []string{alert.Reason} + if alert.Host != "" { + lines = append(lines, "host: "+alert.Host) + } + if alert.Client.IsValid() { lines = append(lines, "client: "+alert.Client.String()) } diff --git a/internal/alerts/alerts_test.go b/internal/alerts/alerts_test.go index 81d8b95..51afbb9 100644 --- a/internal/alerts/alerts_test.go +++ b/internal/alerts/alerts_test.go @@ -1284,9 +1284,11 @@ func anAlertForEachEvent() []eventMessage { Event: alerts.EventWAFBlock, Client: netip.MustParseAddr("192.0.2.1"), Netblock: netip.MustParsePrefix("192.0.2.1/32"), Reason: "refused by the Core Rule Set", + Host: "app.example", }, "fsn1app1/gitea: waf_block", "default shield", - "refused by the Core Rule Set\nclient: 192.0.2.1\nnetblock: 192.0.2.1/32", + "refused by the Core Rule Set\nhost: app.example\nclient: 192.0.2.1\n" + + "netblock: 192.0.2.1/32", }, { alerts.Alert{ diff --git a/internal/anomaly/anomaly.go b/internal/anomaly/anomaly.go index d30ae3e..3c4734d 100644 --- a/internal/anomaly/anomaly.go +++ b/internal/anomaly/anomaly.go @@ -103,6 +103,8 @@ type Counter struct { // Request is a request that has ended, as the counters count it. type Request struct { + // Host is the site the alert names, "" for none. + Host string // Client is the client's address, and ClientGroup the client it is // counted as: its IPv4 address, or the IPv6 netblock of // SWWAF_IPV6_GROUP_PREFIX its address is in. @@ -396,6 +398,7 @@ func alertFor(r Request, k key, o overThreshold) alerts.Alert { return alerts.Alert{ Event: alerts.EventAnomaly, + Host: r.Host, Client: r.Client, Netblock: k.netblock, ASN: r.ASN, diff --git a/internal/metrics/metrics.go b/internal/metrics/metrics.go index c7f7d96..d47dc95 100644 --- a/internal/metrics/metrics.go +++ b/internal/metrics/metrics.go @@ -403,28 +403,21 @@ func (m *Metrics) RequestEnded( line *requestlog.Line, limit string, duration, upstreamDuration time.Duration, ) { m.inFlight.Dec() + m.count(line, limit, duration, upstreamDuration) +} - class := statusClass(line.Status) - m.requests.WithLabelValues(class, line.Action).Inc() - m.requestBytes.WithLabelValues(class, line.Action).Add(float64(line.RequestBytes)) - m.responseBytes.WithLabelValues(class, line.Action).Add(float64(line.ResponseBytes)) - m.requestDuration.Observe(duration.Seconds()) +// RequestAllowed counts a question of rproxy's, answered allow, as no longer +// under way. RequestReported counts the request it was about. +func (m *Metrics) RequestAllowed() { + m.inFlight.Dec() +} - if upstreamDuration > 0 { - m.upstreamDuration.Observe(upstreamDuration.Seconds()) - } - - if limit != "" { - m.sizeAndTimeLimitHits.WithLabelValues(limit).Inc() - } - - if line.Country != "" { - m.countries.add(line.Country, line) - } - - if line.ASN != "" { - m.asns.add(line.ASN, line) - } +// RequestReported counts a request rproxy passed to the app, as RequestEnded +// does, from what its report says, which line holds. +func (m *Metrics) RequestReported( + line *requestlog.Line, duration, upstreamDuration time.Duration, +) { + m.count(line, "", duration, upstreamDuration) } // LimitHit counts a request that broke a rate limit, a byte limit or the @@ -493,6 +486,33 @@ func (m *Metrics) StateFileEditSetAside(name string) { m.stateFileEditsSetAside.WithLabelValues(name).Inc() } +// count counts a request that has ended, as RequestEnded describes. +func (m *Metrics) count( + line *requestlog.Line, limit string, duration, upstreamDuration time.Duration, +) { + class := statusClass(line.Status) + m.requests.WithLabelValues(class, line.Action).Inc() + m.requestBytes.WithLabelValues(class, line.Action).Add(float64(line.RequestBytes)) + m.responseBytes.WithLabelValues(class, line.Action).Add(float64(line.ResponseBytes)) + m.requestDuration.Observe(duration.Seconds()) + + if upstreamDuration > 0 { + m.upstreamDuration.Observe(upstreamDuration.Seconds()) + } + + if limit != "" { + m.sizeAndTimeLimitHits.WithLabelValues(limit).Inc() + } + + if line.Country != "" { + m.countries.add(line.Country, line) + } + + if line.ASN != "" { + m.asns.add(line.ASN, line) + } +} + // addReputationQueries adds the counter of the queries to source, a DNSBL // zone, or of the checks of clients with AbuseIPDB, which count tells. func (m *Metrics) addReputationQueries(source prometheus.Labels, count func() int) { diff --git a/internal/proxy/admin.go b/internal/proxy/admin.go index cbf86d6..f5f2a9e 100644 --- a/internal/proxy/admin.go +++ b/internal/proxy/admin.go @@ -41,6 +41,15 @@ var ( errNotAddress = errors.New("is not an address, such as 203.0.113.9") ) +// answerHealth answers a request for the health endpoint. +func (rq *request) answerHealth() { + rq.line.Action = requestlog.ActionAdmin + // Set here rather than left to Go's server, which would set it only + // after the log line has taken the response's headers. + rq.out.Header().Set("Content-Type", "text/plain; charset=utf-8") + _, _ = io.WriteString(rq.out, "ok\n") +} + // answerAdmin answers a request for smallwebwaf itself, under // /_smallwebwaf/, once it has passed the checks. Each endpoint needs a // token, sent as Authorization: Bearer : the metrics diff --git a/internal/proxy/anomaly_internal_test.go b/internal/proxy/anomaly_internal_test.go index af8034d..88fc46f 100644 --- a/internal/proxy/anomaly_internal_test.go +++ b/internal/proxy/anomaly_internal_test.go @@ -15,9 +15,11 @@ func TestWithEveryAnomalyThresholdOffARequestIsNotCounted(t *testing.T) { // anomaly counters, nor a clock, and the request no response: reading // any of them to count the request panics. rq := &request{ - h: &handler{config: &config.Config{LookupSource: "geojs"}}, - client: netip.MustParseAddr("203.0.113.9"), - lookedUp: true, + decision: decision{ + client: netip.MustParseAddr("203.0.113.9"), + lookedUp: true, + }, + h: &handler{config: &config.Config{LookupSource: "geojs"}}, } defer func() { diff --git a/internal/proxy/bans.go b/internal/proxy/bans.go index 5b1eedb..10f7fc2 100644 --- a/internal/proxy/bans.go +++ b/internal/proxy/bans.go @@ -323,6 +323,7 @@ func (rq *request) alertBan(ban bans.Ban) { rq.h.alerts.Raise(alerts.Alert{ Event: event, + Host: rq.site, Client: rq.client, Netblock: ban.Netblock, ASN: ban.Notes.ASN, @@ -339,11 +340,11 @@ func (rq *request) alertBan(ban bans.Ban) { func (rq *request) noted(now time.Time, status int) bans.Request { return bans.Request{ Time: now, - Method: rq.in.Method, - Host: rq.in.Host, - Path: rq.in.URL.RequestURI(), + Method: rq.line.Method, + Host: rq.line.Host, + Path: rq.requestURI, Status: status, - UserAgent: rq.in.UserAgent(), + UserAgent: rq.line.UserAgent, } } diff --git a/internal/proxy/bans_test.go b/internal/proxy/bans_test.go index d6379d9..aae674d 100644 --- a/internal/proxy/bans_test.go +++ b/internal/proxy/bans_test.go @@ -25,6 +25,8 @@ const ( userAgent = "ban-test/1.0" // permanent is the log line's ban_expires for a permanent ban. permanent = "permanent" + // banResponseClose is the SWWAF_BAN_RESPONSE that closes the connection. + banResponseClose = "close" ) func TestBrokenLimitBansTheClient(t *testing.T) { @@ -237,7 +239,7 @@ func TestBanResponseAnswersEveryRefusalButTheSizeLimits(t *testing.T) { {"", http.StatusForbidden}, {"403", http.StatusForbidden}, {"429", http.StatusTooManyRequests}, - {"close", 0}, + {banResponseClose, 0}, } { t.Run(banResponse+"="+tc.setting, func(t *testing.T) { t.Parallel() diff --git a/internal/proxy/coreruleset.go b/internal/proxy/coreruleset.go index 223c9f3..e5c4bf9 100644 --- a/internal/proxy/coreruleset.go +++ b/internal/proxy/coreruleset.go @@ -113,6 +113,7 @@ func (rq *request) alertWAFBlock(result waf.Result) { rq.h.alerts.Raise(alerts.Alert{ Event: alerts.EventWAFBlock, + Host: rq.site, Client: rq.client, Netblock: rq.h.clientGroup(rq.client), ASN: rq.line.ASN, diff --git a/internal/proxy/metrics_test.go b/internal/proxy/metrics_test.go index c291829..401c102 100644 --- a/internal/proxy/metrics_test.go +++ b/internal/proxy/metrics_test.go @@ -207,7 +207,7 @@ func TestMetricsCountLimitsAndBans(t *testing.T) { rateLimitPerMinute: "1", rateLimitExemptNets: scraper, denyNets: denied, - banResponse: "close", + banResponse: banResponseClose, limitBanDuration: "1h", maxBanDuration: "2h", }) diff --git a/internal/proxy/proxy.go b/internal/proxy/proxy.go index b575464..5eef404 100644 --- a/internal/proxy/proxy.go +++ b/internal/proxy/proxy.go @@ -1,6 +1,8 @@ -// Package proxy passes each request to the app and the app's answer back, -// unchanged, within the size and time limits, and writes one request log -// line for each request. +// Package proxy runs the checks on each request and writes its request log +// line. New passes each request it lets through to the app and the app's +// answer back, unchanged, within the size and time limits. NewServeWAF instead +// answers rproxy's question about each request, and takes rproxy's report once +// the visitor's answer has ended, which writes the request a second log line. package proxy import ( @@ -11,6 +13,8 @@ import ( "strings" "time" + lru "github.com/hashicorp/golang-lru/v2" + "sneak.berlin/go/smallwebwaf/internal/alerts" "sneak.berlin/go/smallwebwaf/internal/anomaly" "sneak.berlin/go/smallwebwaf/internal/bans" @@ -19,7 +23,6 @@ import ( "sneak.berlin/go/smallwebwaf/internal/metrics" "sneak.berlin/go/smallwebwaf/internal/ratelimit" "sneak.berlin/go/smallwebwaf/internal/reputation" - "sneak.berlin/go/smallwebwaf/internal/requestlog" "sneak.berlin/go/smallwebwaf/internal/rules" "sneak.berlin/go/smallwebwaf/internal/waf" ) @@ -53,7 +56,8 @@ const ClientsPath = "/_smallwebwaf/clients/" // Params are what New needs. type Params struct { Config *config.Config - // RequestLog receives one JSON line per request. + // RequestLog receives one JSON line per request, and from servewaf a second + // when the request's report comes. RequestLog io.Writer // ProcessLog receives the process's own messages. ProcessLog *slog.Logger @@ -109,6 +113,23 @@ type Server struct { // SWWAF_CLIENT_REQUEST_TIMEOUT while the headers arrive; the proxy // applies the timeouts and size limits from then on. func New(params Params) *Server { + h := newHandler(params) + + return h.server(&http.Server{ + Handler: h, + ReadHeaderTimeout: params.Config.ClientRequestTimeout, + // Off is an IdleTimeout of 0, which Go's server replaces with + // ReadTimeout: no limit, as long as ReadTimeout stays unset. + IdleTimeout: params.Config.ClientIdleTimeout, + // Go's server reads 4 KiB past MaxHeaderBytes before it + // refuses, so the limit a client meets is the setting. + MaxHeaderBytes: int(params.Config.ClientRequestHeaderMaxBytes - 4<<10), + }) +} + +// newHandler returns the handler with what every request shares, as the +// settings in params make it. +func newHandler(params Params) *handler { errorLog := slog.NewLogLogger(params.ProcessLog.Handler(), slog.LevelWarn) m := metrics.New(params.Config.MetricsTopN, params.Config.InstanceName) lists, dnsbl, abuseIPDB := newReputation(params, m) @@ -171,29 +192,7 @@ func New(params Params) *Server { m.AddBansAndClients(h.ledger, h.limiter, params.Now) m.AddRules(params.Rules) - return &Server{ - Server: &http.Server{ - Addr: params.Config.ListenAddr, - Handler: h, - ReadHeaderTimeout: params.Config.ClientRequestTimeout, - // Off is an IdleTimeout of 0, which Go's server replaces with - // ReadTimeout: no limit, as long as ReadTimeout stays unset. - IdleTimeout: params.Config.ClientIdleTimeout, - // Go's server reads 4 KiB past MaxHeaderBytes before it - // refuses, so the limit a client meets is the setting. - MaxHeaderBytes: int(params.Config.ClientRequestHeaderMaxBytes - 4<<10), - ErrorLog: errorLog, - }, - Ledger: h.ledger, - Limiter: h.limiter, - GeoJS: h.geojs, - Anomalies: h.anomalies, - LookupFile: h.lookupFile, - Lists: h.lists, - DNSBL: h.dnsbl, - AbuseIPDB: h.abuseIPDB, - Metrics: m, - } + return h } // newReputation returns the lists fetched from URLs, the DNSBL zones' @@ -274,6 +273,10 @@ type handler struct { rules *rules.Files coreRuleSet *waf.CoreRuleSet alerts *alerts.Queue + // awaitingReport holds, in servewaf, the decision on each answered + // question by its id until its report comes, and nil for a question still + // being handled. + awaitingReport *lru.Cache[string, *decision] } // newTransport returns what carries requests to the app. It never goes @@ -299,11 +302,7 @@ func (h *handler) ServeHTTP(w http.ResponseWriter, r *http.Request) { // The health endpoint is answered at once, before any check, so that // a health checker is never refused. It does not ask the app. if r.Method == http.MethodGet && r.URL.Path == HealthPath { - rq.line.Action = requestlog.ActionAdmin - // Set here rather than left to Go's server, which would set it only - // after the log line has taken the response's headers. - rq.out.Header().Set("Content-Type", "text/plain; charset=utf-8") - _, _ = io.WriteString(rq.out, "ok\n") + rq.answerHealth() return } @@ -339,3 +338,23 @@ func (h *handler) ServeHTTP(w http.ResponseWriter, r *http.Request) { rq.forward(r.Context()) } + +// server returns the Server of h, which listens at SWWAF_LISTEN_ADDR and +// logs through h, with what httpServer sets. +func (h *handler) server(httpServer *http.Server) *Server { + httpServer.Addr = h.config.ListenAddr + httpServer.ErrorLog = h.errorLog + + return &Server{ + Server: httpServer, + Ledger: h.ledger, + Limiter: h.limiter, + GeoJS: h.geojs, + Anomalies: h.anomalies, + LookupFile: h.lookupFile, + Lists: h.lists, + DNSBL: h.dnsbl, + AbuseIPDB: h.abuseIPDB, + Metrics: h.metrics, + } +} diff --git a/internal/proxy/proxy_test.go b/internal/proxy/proxy_test.go index 8507de6..6af3e76 100644 --- a/internal/proxy/proxy_test.go +++ b/internal/proxy/proxy_test.go @@ -284,6 +284,16 @@ func newProxy( ) (*proxy.Server, *output, *alerts.Queue) { t.Helper() + return newServerWith(t, proxy.New, appURL, geojsURL, now, env) +} + +// newServerWith is newProxy for the server build makes. +func newServerWith( + t *testing.T, build func(proxy.Params) *proxy.Server, + appURL, geojsURL string, now func() time.Time, env map[string]string, +) (*proxy.Server, *output, *alerts.Queue) { + t.Helper() + settings := map[string]string{ "SWWAF_UPSTREAM_URL": appURL, rulesDir: t.TempDir(), instanceName: "app", wafMode: off, @@ -334,7 +344,7 @@ func newProxy( } } - server := proxy.New(proxy.Params{ + server := build(proxy.Params{ Config: cfg, RequestLog: out, ProcessLog: processLog, diff --git a/internal/proxy/reputation.go b/internal/proxy/reputation.go index 0020c60..8ecaf5f 100644 --- a/internal/proxy/reputation.go +++ b/internal/proxy/reputation.go @@ -110,6 +110,7 @@ func (rq *request) noteHit(hit bans.ReputationHit, reason string) { rq.h.metrics.ReputationHit(hit.Source) rq.h.alerts.Raise(alerts.Alert{ Event: alerts.EventReputationHit, + Host: rq.site, Client: rq.client, Netblock: rq.h.clientGroup(rq.client), ASN: rq.line.ASN, diff --git a/internal/proxy/request.go b/internal/proxy/request.go index e9620d9..99c38d0 100644 --- a/internal/proxy/request.go +++ b/internal/proxy/request.go @@ -39,20 +39,20 @@ type refusal struct { limit string } -// request is one request on its way through smallwebwaf, from the moment -// its headers have been read to its log line. -type request struct { - h *handler - in *http.Request - // rc sets the deadlines of the connection to the client. - rc *http.ResponseController - out *responseWriter - body *requestBody // nil for a request without a body - line requestlog.Line - - client netip.Addr - peer netip.Addr - peerTrusted bool +// decision is what the checks note of a request: its log line, and what +// counting the request needs once it has ended. servewaf keeps a copy of it +// until the request's report comes. It holds no credentials, since the log line +// never does, and neither the request's objects nor their bodies. +type decision struct { + line requestlog.Line + client netip.Addr + // requestURI is the path with the query, as a ban's notes keep it. + requestURI string + // site is, in servewaf, the host the visitor asked for, which the + // alerts name, since one servewaf serves many sites. It is "" otherwise. + site string + // allowed is true for a question answered with DispositionAllow. + allowed bool // lookedUp is true once the client's AS number and country have been // looked up, whether or not an answer was there, and lookupAnswer is // what the lookup gave then, the zero Answer while GeoJS had given none. @@ -65,18 +65,37 @@ type request struct { counted bool limitPercent, bytesPercent percentage // attack is true for a request that matched a ban rule or asked for a - // trap path, ruleBlocked for one a block rule refused, wafBlocked for - // one the Core Rule Set refused, and tokenRefused for one refused for a - // missing or wrong token, each an offence its client's history counts. - attack, ruleBlocked, wafBlocked, tokenRefused bool + // trap path, ruleBlocked for one a block rule refused, and wafBlocked for + // one the Core Rule Set refused, each an offence its client's history + // counts. + attack, ruleBlocked, wafBlocked bool + // reputation is the reputation sources that list the client, for the + // notes of a ban the request makes. + reputation []bans.ReputationHit +} + +// request is one request on its way through smallwebwaf, from the moment +// its headers have been read to its log line. +type request struct { + decision + + h *handler + in *http.Request + // rc sets the deadlines of the connection to the client. + rc *http.ResponseController + out *responseWriter + body *requestBody // nil for a request without a body + + peer netip.Addr + peerTrusted bool + // tokenRefused is true for a request refused for a missing or wrong + // token, an offence its client's history counts. + tokenRefused bool // blocklisted is true once a blocklist is found to list the client, // dnsblListed once a DNSBL zone's verdict is, and abuseIPDBHit once // AbuseIPDB's score of it is a hit. blocklisted, dnsblListed, abuseIPDBHit bool - // reputation is the reputation sources that list the client, for the - // notes of a ban the request makes. - reputation []bans.ReputationHit - start time.Time + start time.Time // checked is when the checks were done, and upstreamStart when the // request was handed to the app. checked time.Time @@ -108,50 +127,62 @@ type request struct { answerStarted time.Time } -// newRequest starts handling r: it notes the time, counts the request as -// under way, works out the client, and starts the log line with what is -// known of the request. +// newRequest starts handling r: it works out the client from the TCP peer +// and X-Forwarded-For, and goes on as startRequest does. func (h *handler) newRequest(w http.ResponseWriter, r *http.Request) *request { + peer := peerAddress(r) + trusted := h.config.TrustedProxies + client := clientAddress(peer, r.Header.Values("X-Forwarded-For"), trusted) + + return h.startRequest(w, r, client, peer, isInside(peer, trusted)) +} + +// startRequest starts handling r from client, which came through peer: it +// notes the time, counts the request as under way, and starts the log line +// with what is known of the request. peerTrusted says whether to believe +// the request's id and scheme, which r's headers give. +func (h *handler) startRequest( + w http.ResponseWriter, r *http.Request, client, peer netip.Addr, peerTrusted bool, +) *request { h.metrics.RequestStarted() start := time.Now() - peer := peerAddress(r) - trusted := h.config.TrustedProxies - peerTrusted := isInside(peer, trusted) forwardedFor := r.Header.Values("X-Forwarded-For") - client := clientAddress(peer, forwardedFor, trusted) rq := &request{ + decision: decision{ + client: client, + requestURI: r.URL.RequestURI(), + line: requestlog.Line{ + Time: requestlog.FormatTime(start), + Instance: h.config.InstanceName, + ClientIP: client.String(), + Method: r.Method, + Scheme: scheme(r, peerTrusted), + Host: r.Host, + Path: rules.Path(r), + Query: r.URL.RawQuery, + Protocol: r.Proto, + Referer: r.Referer(), + UserAgent: r.UserAgent(), + RequestID: requestID(r, peerTrusted), + PeerIP: peer.String(), + ForwardedFor: strings.Join(forwardedFor, ", "), + ClientGroup: h.clientGroup(client).String(), + ContentType: r.Header.Get("Content-Type"), + RequestHeaders: requestHeaders(r, h.config.LogRequestHeaders), + HasAuthorization: len(r.Header.Values("Authorization")) > 0, + HasCookie: len(r.Header.Values("Cookie")) > 0, + Action: requestlog.ActionForward, + }, + }, h: h, in: r, rc: http.NewResponseController(w), out: &responseWriter{ResponseWriter: w}, - client: client, peer: peer, peerTrusted: peerTrusted, start: start, - line: requestlog.Line{ - Time: requestlog.FormatTime(start), - Instance: h.config.InstanceName, - ClientIP: client.String(), - Method: r.Method, - Scheme: scheme(r, peerTrusted), - Host: r.Host, - Path: rules.Path(r), - Query: r.URL.RawQuery, - Protocol: r.Proto, - Referer: r.Referer(), - UserAgent: r.UserAgent(), - RequestID: requestID(r, peerTrusted), - PeerIP: peer.String(), - ForwardedFor: strings.Join(forwardedFor, ", "), - ClientGroup: h.clientGroup(client).String(), - ContentType: r.Header.Get("Content-Type"), - RequestHeaders: requestHeaders(r, h.config.LogRequestHeaders), - HasAuthorization: len(r.Header.Values("Authorization")) > 0, - HasCookie: len(r.Header.Values("Cookie")) > 0, - Action: requestlog.ActionForward, - }, } // A length of -1 is a body whose length was not announced. @@ -569,10 +600,20 @@ func (rq *request) finish() { } // Counted before the log line is written, so that the metrics count - // every request whose line is out. - rq.h.metrics.RequestEnded(line, limit, duration, upstreamDuration) + // every request whose line is out. An allowed request is the exception: + // its report gives the status, bytes and times the metrics count. + if rq.allowed { + rq.h.metrics.RequestAllowed() + } else { + rq.h.metrics.RequestEnded(line, limit, duration, upstreamDuration) + } - err := requestlog.Write(rq.h.requestLog, line) + rq.writeLine() +} + +// writeLine writes the request's log line. +func (rq *request) writeLine() { + err := requestlog.Write(rq.h.requestLog, &rq.line) if err != nil { rq.h.processLog.Error("writing the request log failed", "error", err.Error()) } @@ -634,6 +675,7 @@ func (rq *request) countAnomalies() { answer, _ := rq.answerAtTheEnd() rq.h.anomalies.Count(rq.h.now(), anomaly.Request{ + Host: rq.site, Client: rq.client, ClientGroup: rq.h.clientGroup(rq.client), ASN: answer.ASN, diff --git a/internal/proxy/servewaf.go b/internal/proxy/servewaf.go new file mode 100644 index 0000000..9fe3923 --- /dev/null +++ b/internal/proxy/servewaf.go @@ -0,0 +1,464 @@ +package proxy + +import ( + "bytes" + "encoding/json" + "errors" + "fmt" + "io" + "net/http" + "net/netip" + "strconv" + "strings" + "time" + "unicode/utf8" + + lru "github.com/hashicorp/golang-lru/v2" + + "sneak.berlin/go/smallwebwaf/internal/requestlog" +) + +// The headers of rproxy's question and of servewaf's answer, as +// "servewaf" in SPEC.md gives them. +const ( + // ClientHeader marks a request as rproxy's question, and gives the + // visitor's address. + ClientHeader = "X-Smallwebwaf-Client" + // PeerHeader gives the address of rproxy's own TCP peer. + PeerHeader = "X-Smallwebwaf-Peer" + // ContentLengthHeader gives the body length the visitor announced. + ContentLengthHeader = "X-Smallwebwaf-Content-Length" + // DispositionHeader gives servewaf's answer. + DispositionHeader = "X-Smallwebwaf-Disposition" +) + +// The values of DispositionHeader. +const ( + // DispositionAllow tells rproxy to pass the request to the app. + DispositionAllow = "allow" + // DispositionDeny tells rproxy to give the visitor the answer's status, + // headers and body. + DispositionDeny = "deny" + // DispositionClose tells rproxy to close the visitor's connection + // without an answer, which SWWAF_BAN_RESPONSE asks for when it is close. + // servewaf closing the question's connection instead would read as + // servewaf being down, and rproxy would then let the request through. + DispositionClose = "close" +) + +// ReportPath is where rproxy sends the report of each request servewaf +// answered. +const ReportPath = "/_smallwebwaf/report" + +// QuestionBodyMaxBytes is the most of a visitor's body a question carries. +const QuestionBodyMaxBytes = 64 << 10 + +const ( + // maxAwaitingReport is the most answered requests held until their + // reports come. Past it, the request answered longest ago is dropped. + maxAwaitingReport = 10000 + // maxRequestIDBytes is the longest id a question may give. + maxRequestIDBytes = 128 + // reportBodyMaxBytes is the most of a report's body that is read. It is + // far above any report's. + reportBodyMaxBytes = 1 << 20 + // maxHTTPStatus is the highest status a report may give. + maxHTTPStatus = 599 + // maxReportedBytes is the most bytes a report may give for each way, 1 + // PiB, so that adding them up cannot overflow. + maxReportedBytes = 1 << 50 + // maxReportedMilliseconds is the longest duration a report may give, + // about 31 years, so that it fits a time.Duration. + maxReportedMilliseconds = 1e12 + // serveWAFTimeout bounds how long a request to servewaf takes to arrive, + // and its answer to leave, and serveWAFIdleTimeout how long a kept-open + // connection waits for its next request. + serveWAFTimeout = 60 * time.Second + serveWAFIdleTimeout = 120 * time.Second +) + +var ( + errReportFieldMissing = errors.New("a field is missing") + errReportValue = errors.New("a value is out of range") +) + +// Report is the JSON object rproxy posts to ReportPath once the visitor's +// answer has ended, its fields named as the request log's of the same +// meaning. RequestID is the question's id. Status is what the visitor got, 0 +// if nothing. The durations are in milliseconds. +// +//nolint:tagliatelle // docs/request-log.md names the fields in snake_case +type Report struct { + RequestID string `json:"request_id"` + Status int `json:"status"` + RequestBytes int64 `json:"request_bytes"` + ResponseBytes int64 `json:"response_bytes"` + DurationTotal float64 `json:"duration_total"` + DurationUpstreamTotal float64 `json:"duration_upstream_total"` +} + +// NewServeWAF returns the server `smallwebwaf servewaf` runs. It answers +// rproxy's questions with the checks New runs, takes rproxy's reports and +// answers its own endpoints. It passes nothing to an app. rproxy applies +// the size and time limits to the visitor, so their settings do not apply +// here. Go's default limit on a request's line and headers, 1 MiB, stays: +// it is far above rproxy's, and a question refused for its size would have +// rproxy let the request through. +func NewServeWAF(params Params) *Server { + cfg := *params.Config + cfg.RequestMaxBytes = 0 + cfg.ClientRequestTimeout = serveWAFTimeout + // The server's WriteTimeout does this. It sets a new deadline for each + // request, where a deadline a handler sets stays on a kept-open + // connection, and would run out under a later request's answer. + cfg.ClientResponseTimeout = 0 + // A question's body of QuestionBodyMaxBytes may be cut short, and the + // Core Rule Set does not read a JSON or XML body over its limit. + cfg.WAFBodyLimit = min(cfg.WAFBodyLimit, QuestionBodyMaxBytes-1) + params.Config = &cfg + + h := newHandler(params) + + awaitingReport, err := lru.New[string, *decision](maxAwaitingReport) + if err != nil { + panic(err) // New fails only for a size below one + } + + h.awaitingReport = awaitingReport + + return h.server(&http.Server{ + Handler: http.HandlerFunc(h.serveWAF), + ReadTimeout: serveWAFTimeout, + WriteTimeout: serveWAFTimeout, + IdleTimeout: serveWAFIdleTimeout, + }) +} + +// serveWAF handles one request to servewaf: a request carrying +// ClientHeader is rproxy's question, and any other is for servewaf itself. +func (h *handler) serveWAF(w http.ResponseWriter, r *http.Request) { + if len(r.Header.Values(ClientHeader)) > 0 { + h.answerQuestion(w, r) + + return + } + + h.answerOwnEndpoint(w, r) +} + +// answerQuestion runs the checks on the visitor's request a question +// carries, and answers rproxy what to do with it. A denied request's counting +// is finished here, and an allowed one's by its report. +func (h *handler) answerQuestion(w http.ResponseWriter, r *http.Request) { + client, body, ok := h.readQuestion(w, r) + if !ok { + return + } + + // One that cannot be read leaves the log line's peer_ip invalid. + peer, _ := netip.ParseAddr(r.Header.Get(PeerHeader)) + + rq := h.startRequest(w, asTheVisitorSentIt(r, body), client.Unmap(), peer.Unmap(), + true) + // The last deferred runs first, so that what is held has the finished log + // line. Go's server sends the answer only once this function has + // returned, so the report cannot come before it is held. + defer rq.holdForReport() + defer rq.finish() + + rq.site = r.Host + rq.line.ServeWAFLine = requestlog.ServeWAFLineDecision + refused := rq.check(r.Context()) + rq.checked = time.Now() + underOwnPrefix := strings.HasPrefix(r.URL.Path, adminPrefix) + + if refused == nil && !underOwnPrefix { + // In observe mode, what enforce mode would have refused counts here. + defer rq.countRefusal() + + rq.allow() + + return + } + + // The last deferred runs first, as in ServeHTTP. + defer rq.addToHistory() + defer rq.countAnomalies() + defer rq.countRefusal() + + if refused != nil { + rq.deny(*refused) + + return + } + + // The app never sees this prefix. servewaf's own endpoints are at its + // own address, so this answers as a path with no endpoint does. + rq.line.Action = requestlog.ActionAdmin + rq.out.Header().Set(DispositionHeader, DispositionDeny) + http.Error(rq.out, http.StatusText(http.StatusNotFound), http.StatusNotFound) +} + +// readQuestion returns the visitor's address and the body a question carries, +// and whether servewaf takes the question. It answers a question it does not +// take with 400 and no disposition, which rproxy treats as it treats no +// answer. The id of a question it takes is held from here on, without a +// decision yet, so that another question with that id is refused while this +// one is being handled. +// +// The body is read before any check, since Go's server does not send an answer +// until it has read the body. A question whose body stops arriving gets no +// answer, and is not looked at. +func (h *handler) readQuestion( + w http.ResponseWriter, r *http.Request, +) (netip.Addr, []byte, bool) { + client, err := netip.ParseAddr(r.Header.Get(ClientHeader)) + if err != nil || client.Zone() != "" { + h.refuseQuestion(w, ClientHeader+" is not an address without a zone") + + return netip.Addr{}, nil, false + } + + id := r.Header.Get(requestIDHeader) + if id == "" || len(id) > maxRequestIDBytes || !utf8.ValidString(id) { + h.refuseQuestion(w, requestIDHeader+" is not a valid id") + + return netip.Addr{}, nil, false + } + + body, err := io.ReadAll(http.MaxBytesReader(w, r.Body, QuestionBodyMaxBytes)) + + if _, tooLarge := errors.AsType[*http.MaxBytesError](err); tooLarge { + h.refuseQuestion(w, "the body is over 64 KiB") + + return netip.Addr{}, nil, false + } + + if err != nil { + h.processLog.Warn("a question's body stopped arriving", "error", err.Error()) + // This panic has Go's server close the connection without an answer, + // and log nothing. + panic(http.ErrAbortHandler) + } + + held, _ := h.awaitingReport.ContainsOrAdd(id, nil) + if held { + h.refuseQuestion(w, requestIDHeader+" is the id of a request servewaf holds") + + return netip.Addr{}, nil, false + } + + return client, body, true +} + +// refuseQuestion answers a question servewaf does not take, for reason. +func (h *handler) refuseQuestion(w http.ResponseWriter, reason string) { + h.processLog.Warn("a question is refused", "reason", reason) + http.Error(w, reason, http.StatusBadRequest) +} + +// asTheVisitorSentIt returns a copy of r, a question, as the checks and the +// log line are to see it: without the headers rproxy adds for servewaf, with +// body, which has been read of r, and with the length the visitor announced, +// as -1 when it announced none. The question's own length covers only the part +// of the body it carries. +func asTheVisitorSentIt(r *http.Request, body []byte) *http.Request { + sent := r.Clone(r.Context()) + announced := sent.Header.Get(ContentLengthHeader) + + sent.Header.Del(ClientHeader) + sent.Header.Del(PeerHeader) + sent.Header.Del(ContentLengthHeader) + sent.Header.Del("Content-Length") + + sent.ContentLength = -1 + + length, err := strconv.ParseInt(announced, 10, 64) + if err == nil && length >= 0 { + sent.ContentLength = length + sent.Header.Set("Content-Length", announced) + } + + sent.Body = http.NoBody + if len(body) > 0 { + sent.Body = io.NopCloser(bytes.NewReader(body)) + } + + return sent +} + +// allow answers a question with DispositionAllow and, while +// SWWAF_ADD_LOOKUP_HEADERS is set, the client's AS number and country. +func (rq *request) allow() { + rq.allowed = true + + header := rq.out.Header() + header.Set(DispositionHeader, DispositionAllow) + + if rq.h.config.AddLookupHeaders { + setLookupHeaders(header, rq.line.ASN, rq.line.Country) + } + + rq.out.WriteHeader(http.StatusOK) +} + +// holdForReport holds a copy of what the report's steps need, by the request's +// id. A report only ever reaches that copy, never the request being handled. +func (rq *request) holdForReport() { + held := rq.decision + rq.h.awaitingReport.Add(rq.line.RequestID, &held) +} + +// deny answers a question with what the visitor is to get for r. +func (rq *request) deny(r refusal) { + if r.status == 0 { + // SWWAF_BAN_RESPONSE is close. Written past rq.out, since the log line's + // status stays 0: the visitor is sent nothing. + rq.refused.CompareAndSwap(nil, &r) + rq.out.Header().Set(DispositionHeader, DispositionClose) + rq.out.ResponseWriter.WriteHeader(http.StatusOK) + + return + } + + rq.out.Header().Set(DispositionHeader, DispositionDeny) + rq.answer(r) +} + +// answerOwnEndpoint answers a request to servewaf's own address. No check +// runs on it: the address is on the private network, where the peers are +// rproxy and the admin's tools, not visitors. +func (h *handler) answerOwnEndpoint(w http.ResponseWriter, r *http.Request) { + if r.Method == http.MethodPost && r.URL.Path == ReportPath { + h.takeReport(w, r) + + return + } + + peer := peerAddress(r) + + rq := h.startRequest(w, r, peer, peer, false) + defer rq.finish() + + if r.Method == http.MethodGet && r.URL.Path == HealthPath { + rq.answerHealth() + + return + } + + rq.answerAdmin() +} + +// takeReport answers POST ReportPath: it finishes the request the report +// names, if it holds that request's decision, and ignores the report +// otherwise. +func (h *handler) takeReport(w http.ResponseWriter, r *http.Request) { + report, err := readReport(w, r) + if err != nil { + h.processLog.Warn("a report is refused", "error", err.Error()) + http.Error(w, "the body is not a report", http.StatusBadRequest) + + return + } + + held, found := h.awaitingReport.Peek(report.RequestID) + // Remove reports true to one report only, of several that name the id. + if found && held != nil && h.awaitingReport.Remove(report.RequestID) { + h.finishWith(held, report) + } + + w.WriteHeader(http.StatusNoContent) +} + +// readReport reads a report from r's body, which must be one JSON object with +// all of Report's fields, in the ranges SPEC.md gives. +func readReport(w http.ResponseWriter, r *http.Request) (Report, error) { + var report Report + + body, err := io.ReadAll(http.MaxBytesReader(w, r.Body, reportBodyMaxBytes)) + if err != nil { + return report, fmt.Errorf("read the body: %w", err) + } + + var fields map[string]json.RawMessage + + err = json.Unmarshal(body, &fields) + if err != nil { + return report, fmt.Errorf("decode the body: %w", err) + } + + for _, name := range []string{ + "request_id", "status", "request_bytes", "response_bytes", + "duration_total", "duration_upstream_total", + } { + value, there := fields[name] + if !there || string(value) == "null" { + return report, fmt.Errorf("%w: %s", errReportFieldMissing, name) + } + } + + err = json.Unmarshal(body, &report) + if err != nil { + return report, fmt.Errorf("decode the body: %w", err) + } + + return report, report.checkRanges() +} + +// checkRanges returns an error for a value of the report that cannot be true. +func (report *Report) checkRanges() error { + switch { + case report.Status != 0 && + (report.Status < http.StatusContinue || report.Status > maxHTTPStatus): + return fmt.Errorf("%w: status", errReportValue) + case report.RequestBytes < 0 || report.RequestBytes > maxReportedBytes: + return fmt.Errorf("%w: request_bytes", errReportValue) + case report.ResponseBytes < 0 || report.ResponseBytes > maxReportedBytes: + return fmt.Errorf("%w: response_bytes", errReportValue) + case report.DurationTotal < 0 || report.DurationTotal > maxReportedMilliseconds: + return fmt.Errorf("%w: duration_total", errReportValue) + case report.DurationUpstreamTotal < 0 || + report.DurationUpstreamTotal > maxReportedMilliseconds: + return fmt.Errorf("%w: duration_upstream_total", errReportValue) + } + + return nil +} + +// finishWith finishes a request with what its report says, and writes the +// request's last log line, which holds its decision's fields and the report's. +// For an allowed request it first takes the steps ServeHTTP takes once a +// request passed to the app has ended, with the status and bytes of the report +// in place of those it would have seen. A denied request's counting was +// finished at its decision. +func (h *handler) finishWith(held *decision, report Report) { + rq := &request{ + decision: *held, + h: h, + out: &responseWriter{status: report.Status, bytes: report.ResponseBytes}, + body: &requestBody{}, + } + rq.body.bytes.Store(report.RequestBytes) + + rq.line.ServeWAFLine = requestlog.ServeWAFLineReport + rq.line.Status = report.Status + rq.line.RequestBytes = report.RequestBytes + rq.line.ResponseBytes = report.ResponseBytes + rq.line.DurationTotal = report.DurationTotal + + if rq.allowed { + rq.line.DurationUpstreamTotal = &report.DurationUpstreamTotal + // addToHistory goes by this for whether the request reached the app. + rq.upstreamStart = time.Now() + + rq.countBytes() + rq.countAnomalies() + rq.addToHistory() + + rq.h.metrics.RequestReported(&rq.line, + time.Duration(report.DurationTotal*float64(time.Millisecond)), + time.Duration(report.DurationUpstreamTotal*float64(time.Millisecond))) + } + + rq.writeLine() +} diff --git a/internal/proxy/servewaf_test.go b/internal/proxy/servewaf_test.go new file mode 100644 index 0000000..b618493 --- /dev/null +++ b/internal/proxy/servewaf_test.go @@ -0,0 +1,1642 @@ +package proxy_test + +import ( + "context" + "crypto/rand" + "encoding/json" + "fmt" + "io" + "maps" + "math" + "net" + "net/http" + "net/netip" + "reflect" + "strconv" + "strings" + "testing" + "time" + + "sneak.berlin/go/smallwebwaf/internal/alerts" + "sneak.berlin/go/smallwebwaf/internal/bans" + "sneak.berlin/go/smallwebwaf/internal/proxy" + "sneak.berlin/go/smallwebwaf/internal/ratelimit" + "sneak.berlin/go/smallwebwaf/internal/requestlog" +) + +const ( + // siteA and siteB are two sites behind one servewaf. + siteA = "a.example" + siteB = "b.example" + // rproxyPeer is what the tests' questions give as rproxy's TCP peer. + rproxyPeer = "172.18.0.2" + // heldLimit is how many answered requests "servewaf" in SPEC.md says it + // holds. + heldLimit = 10000 + // reportedBytesLimit is the most bytes for each way SPEC.md lets a report + // give. + reportedBytesLimit = 1 << 50 + // reportBodyLimit is the longest body SPEC.md lets a report have. + reportBodyLimit = 1 << 20 + // anomalyClientPerMinute is the anomaly threshold these tests set. + anomalyClientPerMinute = "SWWAF_ANOMALY_CLIENT_REQUESTS_PER_MINUTE" +) + +// The fields of a report, as "servewaf" in SPEC.md names them. +const ( + reportID = "request_id" + reportStatus = "status" + reportRequestBytes = "request_bytes" + reportResponseBytes = "response_bytes" + reportDuration = "duration_total" + reportUpstreamDuration = "duration_upstream_total" +) + +// serveWAF is a servewaf running for one test, on a clock set to midnight. +type serveWAF struct { + t *testing.T + addr string + out *output + server *proxy.Server + clock *clock + queue *alerts.Queue +} + +// startServeWAF starts servewaf with the settings in env, clients' AS numbers +// and countries looked up at geojsURL. +func startServeWAF(t *testing.T, geojsURL string, env map[string]string) *serveWAF { + t.Helper() + + return startServeWAFWith(t, geojsURL, env, func(*proxy.Server) {}) +} + +// startServeWAFWith is startServeWAF with change applied to the server before +// it starts. +func startServeWAFWith( + t *testing.T, geojsURL string, env map[string]string, change func(*proxy.Server), +) *serveWAF { + t.Helper() + + clk := &clock{now: time.Date(2026, 10, 6, 0, 0, 0, 0, time.UTC)} + settings := map[string]string{ + alertWebhookURL: "https://alerts.example/servewaf", + instanceName: alertInstance, + } + maps.Copy(settings, env) + + server, out, queue := newServerWith(t, proxy.NewServeWAF, + "http://127.0.0.1:8081", geojsURL, clk.Now, settings) + change(server) + + listener, err := (&net.ListenConfig{}).Listen(t.Context(), "tcp", localhost+":0") + if err != nil { + t.Fatalf("listen: %v", err) + } + + go func() { + _ = server.Serve(listener) + }() + + t.Cleanup(func() { + _ = server.Close() + }) + + return &serveWAF{ + t: t, addr: listener.Addr().String(), out: out, server: server, + clock: clk, queue: queue, + } +} + +// questionWith makes rproxy's question about a request for path on host, from +// the visitor at from, carrying body and a new id, as rproxy makes for every +// request. +func (s *serveWAF) questionWith( + method, host, from, path string, body io.Reader, +) *http.Request { + s.t.Helper() + + req := newRequest(s.t, method, s.addr, path, body) + req.Host = host + req.Header.Set(proxy.ClientHeader, from) + req.Header.Set(proxy.PeerHeader, rproxyPeer) + req.Header.Set(requestIDHeader, rand.Text()) + + return req +} + +// question makes rproxy's question about a request for path without a body. +func (s *serveWAF) question(host, from, path string) *http.Request { + s.t.Helper() + + return s.questionWith(http.MethodGet, host, from, path, http.NoBody) +} + +// ask sends the question about a request for path on siteA from the visitor at +// from, and returns the answer. +func (s *serveWAF) ask(from, path string) answer { + s.t.Helper() + + return do(s.t, s.question(siteA, from, path)) +} + +// allowed asks about a request for / on siteA, checks that the answer is allow, +// and returns the request's id. +func (s *serveWAF) allowed(from string) string { + s.t.Helper() + + req := s.question(siteA, from, "/") + wantDisposition(s.t, do(s.t, req), http.StatusOK, proxy.DispositionAllow) + + return req.Header.Get(requestIDHeader) +} + +// report sends rproxy's report and returns the answer. +func (s *serveWAF) report(report proxy.Report) answer { + s.t.Helper() + + body, err := json.Marshal(report) + if err != nil { + s.t.Fatalf("encode the report: %v", err) + } + + return s.reportBody(string(body)) +} + +// reportBody sends body as rproxy's report and returns the answer. +func (s *serveWAF) reportBody(body string) answer { + s.t.Helper() + + return do(s.t, newRequest(s.t, http.MethodPost, s.addr, proxy.ReportPath, + strings.NewReader(body))) +} + +// passedOn asks about a request from the visitor at from on siteA, and reports +// it as answered 200 with bodyBytes and answerBytes. +func (s *serveWAF) passedOn(from string) { + s.t.Helper() + + got := s.report(proxy.Report{ + RequestID: s.allowed(from), + Status: http.StatusOK, + RequestBytes: bodyBytes, + ResponseBytes: answerBytes, + }) + wantStatus(s.t, got, http.StatusNoContent) +} + +// history returns what the table of clients holds of client's history. +func (s *serveWAF) history() ratelimit.History { + held, _ := s.server.Limiter.Client(netip.MustParsePrefix(client + "/32")) + + return held.History +} + +// metrics returns what the metrics endpoint answers, for a server started with +// the token as SWWAF_METRICS_TOKEN. +func (s *serveWAF) metrics() string { + s.t.Helper() + + req := newRequest(s.t, http.MethodGet, s.addr, proxy.MetricsPath, http.NoBody) + req.Header.Set("Authorization", bearer) + + got := do(s.t, req) + wantStatus(s.t, got, http.StatusOK) + + return string(got.body) +} + +// wantDisposition checks an answer's status and disposition. +func wantDisposition(t *testing.T, got answer, status int, disposition string) { + t.Helper() + wantStatus(t, got, status) + + if answered := got.header.Get(proxy.DispositionHeader); answered != disposition { + t.Errorf("answered with disposition %q, want %q", answered, disposition) + } +} + +// wantBody checks an answer's body. +func wantBody(t *testing.T, got answer, want string) { + t.Helper() + + if string(got.body) != want { + t.Errorf("answered with the body %q, want %q", got.body, want) + } +} + +// wantProcessMessages checks that msg is among the process's messages count +// times, each from the server's log at the level warnings are. +func wantProcessMessages(t *testing.T, s *serveWAF, msg string, count int) { + t.Helper() + + found := 0 + + for _, fields := range s.out.lines(t) { + if fields["msg"] == msg && fields["level"] == "WARN" { + found++ + } + } + + if found != count { + t.Errorf("%d process warnings %q, want %d; output:\n%s", found, msg, count, + s.out.text()) + } +} + +// wantNoRequestLines checks that no request log line has been written. +func wantNoRequestLines(t *testing.T, s *serveWAF) { + t.Helper() + + for _, fields := range s.out.lines(t) { + if fields["type"] == requestType { + t.Errorf("a request log line was written: %v", fields) + } + } +} + +func TestQuestionAllowedIsAnsweredAndLogged(t *testing.T) { + t.Parallel() + + s := startServeWAF(t, "", nil) + + req := s.question(siteA, client, "/page?q=1&token=visible") + req.Header.Set(requestIDHeader, "the-id") + req.Header.Set("X-Forwarded-Proto", "https") + req.Header.Set(proxy.ContentLengthHeader, "1234") + + got := do(t, req) + wantDisposition(t, got, http.StatusOK, proxy.DispositionAllow) + wantBody(t, got, "") + + if id := got.header.Get(requestIDHeader); id != "" { + t.Errorf("answered with the id %q, want none: rproxy makes the id", id) + } + + line := s.out.requestLine(t) + wantLine(t, line, http.StatusOK, requestlog.ActionForward) + + if line.ClientIP != client || line.PeerIP != rproxyPeer || line.Host != siteA || + line.Path != "/page" || line.Query != "q=1&token=visible" || + line.RequestID != "the-id" || line.Scheme != "https" || + line.ContentLength != 1234 || + line.ServeWAFLine != requestlog.ServeWAFLineDecision { + t.Errorf("log line is %+v", line.Line) + } +} + +func TestIPv6VisitorIsCountedInItsGroup(t *testing.T) { + t.Parallel() + + s := startServeWAF(t, "", map[string]string{rateLimitPerMinute: "1"}) + + s.allowed("2001:db8::1") + + got := s.ask("2001:db8::2", "/") + wantDisposition(t, got, http.StatusForbidden, proxy.DispositionDeny) +} + +func TestIPv4VisitorInIPv6FormIsTheIPv4Client(t *testing.T) { + t.Parallel() + + s := startServeWAF(t, "", map[string]string{denyNets: client}) + + req := s.question(siteA, "::ffff:"+client, "/") + req.Header.Set(proxy.PeerHeader, "::ffff:"+rproxyPeer) + + wantDisposition(t, do(t, req), http.StatusForbidden, proxy.DispositionDeny) + + line := s.out.requestLine(t) + if line.ClientIP != client || line.PeerIP != rproxyPeer { + t.Errorf("log line has client_ip %q and peer_ip %q", line.ClientIP, line.PeerIP) + } +} + +func TestMalformedQuestionGets400AndAWarningAndNoLine(t *testing.T) { + t.Parallel() + + for _, tc := range []struct { + name string + change func(*http.Request) + }{ + {"client not an address", func(req *http.Request) { + req.Header.Set(proxy.ClientHeader, "not an address") + }}, + {"client empty", func(req *http.Request) { + req.Header.Set(proxy.ClientHeader, "") + }}, + {"client address with a zone", func(req *http.Request) { + req.Header.Set(proxy.ClientHeader, "2001:db8::1%eth0") + }}, + {"id absent", func(req *http.Request) { + req.Header.Del(requestIDHeader) + }}, + {"id empty", func(req *http.Request) { + req.Header.Set(requestIDHeader, "") + }}, + {"id not UTF-8", func(req *http.Request) { + req.Header.Set(requestIDHeader, "not-utf-8-\xff") + }}, + {"id too long", func(req *http.Request) { + req.Header.Set(requestIDHeader, strings.Repeat("x", 129)) + }}, + } { + t.Run(tc.name, func(t *testing.T) { + t.Parallel() + + s := startServeWAF(t, "", map[string]string{metricsToken: token}) + + req := s.question(siteA, client, "/") + tc.change(req) + wantDisposition(t, do(t, req), http.StatusBadRequest, "") + + wantProcessMessages(t, s, "a question is refused", 1) + wantNoRequestLines(t, s) + + metrics := s.metrics() + if strings.Contains(metrics, `status_class="4xx"`) { + t.Errorf("a refused question is counted in the metrics:\n%s", metrics) + } + }) + } +} + +func TestLongestValidIDIsTaken(t *testing.T) { + t.Parallel() + + s := startServeWAF(t, "", nil) + + req := s.question(siteA, client, "/") + req.Header.Set(requestIDHeader, strings.Repeat("x", 128)) + wantDisposition(t, do(t, req), http.StatusOK, proxy.DispositionAllow) +} + +func TestQuestionWithABodyOver64KiBGets400(t *testing.T) { + t.Parallel() + + s := startServeWAF(t, "", nil) + + over := strings.NewReader(strings.Repeat("x", proxy.QuestionBodyMaxBytes+1)) + got := do(t, s.questionWith(http.MethodPost, siteA, client, "/", over)) + wantDisposition(t, got, http.StatusBadRequest, "") + wantProcessMessages(t, s, "a question is refused", 1) + wantNoRequestLines(t, s) + + whole := strings.NewReader(strings.Repeat("x", proxy.QuestionBodyMaxBytes)) + got = do(t, s.questionWith(http.MethodPost, siteA, client, "/", whole)) + wantDisposition(t, got, http.StatusOK, proxy.DispositionAllow) +} + +func TestQuestionWithAHeldIDIsRefused(t *testing.T) { + t.Parallel() + + s := startServeWAF(t, "", nil) + + id := s.allowed(client) + + again := s.question(siteA, client, "/") + again.Header.Set(requestIDHeader, id) + wantDisposition(t, do(t, again), http.StatusBadRequest, "") + wantProcessMessages(t, s, "a question is refused", 1) + + // A denied request's id is held as well. + denied := s.question(siteA, otherClient, proxy.HealthPath) + wantDisposition(t, do(t, denied), http.StatusNotFound, proxy.DispositionDeny) + + again = s.question(siteA, client, "/") + again.Header.Set(requestIDHeader, denied.Header.Get(requestIDHeader)) + wantDisposition(t, do(t, again), http.StatusBadRequest, "") + + // The report ends the holding, and the refused questions changed nothing. + got := s.report(proxy.Report{RequestID: id, ResponseBytes: answerBytes}) + wantStatus(t, got, http.StatusNoContent) + + if history := s.history(); history.ResponseBytes != answerBytes { + t.Errorf("history is %+v, want the first question's report", history) + } + + again = s.question(siteA, client, "/") + again.Header.Set(requestIDHeader, id) + wantDisposition(t, do(t, again), http.StatusOK, proxy.DispositionAllow) +} + +func TestReportNeverReachesAQuestionStillBeingHandled(t *testing.T) { + t.Parallel() + + geojsURL, asked, release := startHeldGeoJS(t) + s := startServeWAF(t, geojsURL, map[string]string{ + lookupTimeout: "1h", deniedCountries: "kp", bytesLimitPerMinute: byteLimit, + }) + + first := s.question(siteA, client, "/") + id := first.Header.Get(requestIDHeader) + shared := newClient(t) + answered := make(chan answer, 1) + + go func() { + res, err := shared.Do(first) + if err != nil { + t.Errorf("question: %v", err) + + answered <- answer{} + + return + } + + answered <- readAnswer(res) + }() + + // The lookup holds the question up. + waitUntil(func() bool { return len(asked()) > 0 }) + + // A report for a question not yet answered is ignored, and that question's + // id is taken. + got := s.report(proxy.Report{RequestID: id, ResponseBytes: 1000}) + wantStatus(t, got, http.StatusNoContent) + + // This would wait on the same lookup were the id not taken. + ctx, cancel := context.WithTimeout(t.Context(), waitLimit) + defer cancel() + + again := s.question(siteA, client, "/").WithContext(ctx) + again.Header.Set(requestIDHeader, id) + wantDisposition(t, do(t, again), http.StatusBadRequest, "") + + release() + + select { + case got := <-answered: + wantDisposition(t, got, http.StatusOK, proxy.DispositionAllow) + case <-time.After(waitLimit): + t.Fatal("the question got no answer") + } + + line := s.out.requestLine(t) + if line.ResponseBytes != 0 || line.ServeWAFLine != requestlog.ServeWAFLineDecision { + t.Errorf("the report reached the question being handled: %+v", line.Line) + } + + if held := s.server.Ledger.Snapshot(); len(held) != 0 { + t.Errorf("the ignored report made bans: %v", held) + } + + // The true report comes after, and is the one that counts. + got = s.report(proxy.Report{RequestID: id, ResponseBytes: answerBytes}) + wantStatus(t, got, http.StatusNoContent) + + if history := s.history(); history.ResponseBytes != answerBytes { + t.Errorf("history is %+v, want the true report's bytes", history) + } +} + +func TestPeerAbsentOrUnreadableIsAnsweredAndLoggedInvalid(t *testing.T) { + t.Parallel() + + for _, peer := range []string{"", "not an address"} { + t.Run(peer, func(t *testing.T) { + t.Parallel() + + s := startServeWAF(t, "", nil) + + req := s.question(siteA, client, "/") + req.Header.Del(proxy.PeerHeader) + + if peer != "" { + req.Header.Set(proxy.PeerHeader, peer) + } + + wantDisposition(t, do(t, req), http.StatusOK, proxy.DispositionAllow) + + if line := s.out.requestLine(t); line.PeerIP != "invalid IP" { + t.Errorf("log line has peer_ip %q, want invalid IP", line.PeerIP) + } + }) + } +} + +func TestContentLengthUnreadableOrNegativeCountsAsAbsent(t *testing.T) { + t.Parallel() + + for _, announced := range []string{"many", "-1", "1.5"} { + t.Run(announced, func(t *testing.T) { + t.Parallel() + + s := startServeWAF(t, "", map[string]string{ + rulesDir: writeRules(t, "any-length header:Content-Length block .\n"), + }) + + req := s.question(siteA, client, "/") + req.Header.Set(proxy.ContentLengthHeader, announced) + wantDisposition(t, do(t, req), http.StatusOK, proxy.DispositionAllow) + + if line := s.out.requestLine(t); line.ContentLength != 0 { + t.Errorf("log line has content_length %d, want none", line.ContentLength) + } + }) + } +} + +func TestHeaderSentTwiceUsesItsFirstValue(t *testing.T) { + t.Parallel() + + s := startServeWAF(t, "", nil) + + req := s.question(siteA, client, "/") + req.Header.Add(proxy.ClientHeader, otherClient) + req.Header.Add(proxy.PeerHeader, "172.18.0.3") + req.Header.Set(requestIDHeader, "first") + req.Header.Add(requestIDHeader, "second") + req.Header.Set(proxy.ContentLengthHeader, "1234") + req.Header.Add(proxy.ContentLengthHeader, "5678") + wantDisposition(t, do(t, req), http.StatusOK, proxy.DispositionAllow) + + line := s.out.requestLine(t) + if line.ClientIP != client || line.PeerIP != rproxyPeer || + line.RequestID != "first" || line.ContentLength != 1234 { + t.Errorf("log line is %+v", line.Line) + } +} + +func TestSchemeAbsentIsLoggedAsHTTP(t *testing.T) { + t.Parallel() + + s := startServeWAF(t, "", nil) + + s.allowed(client) + + if line := s.out.requestLine(t); line.Scheme != "http" { + t.Errorf("log line has scheme %q, want http", line.Scheme) + } +} + +// sendQuestionWithPartOfItsBody sends a question over conn whose Content-Length +// announces more than it sends. +func sendQuestionWithPartOfItsBody(t *testing.T, conn net.Conn) { + t.Helper() + + _, err := fmt.Fprintf(conn, + "POST / HTTP/1.1\r\nHost: %s\r\n%s: %s\r\n%s: an-id\r\n"+ + "Content-Length: 100\r\n\r\npart", + siteA, proxy.ClientHeader, client, requestIDHeader) + if err != nil { + t.Fatalf("send the question: %v", err) + } +} + +func TestQuestionWhoseBodyStopsArrivingGetsNoAnswer(t *testing.T) { + t.Parallel() + + // Stalling would take the server's 60 seconds otherwise. + const readTimeout = time.Second + + for _, tc := range []struct { + name string + stop func(*testing.T, net.Conn) + }{ + {"it stalls", func(*testing.T, net.Conn) {}}, + {"rproxy stops sending", func(t *testing.T, conn net.Conn) { + t.Helper() + + err := conn.(*net.TCPConn).CloseWrite() //nolint:forcetypeassert // dial's + if err != nil { + t.Fatalf("close the sending side: %v", err) + } + }}, + } { + t.Run(tc.name, func(t *testing.T) { + t.Parallel() + + s := startServeWAFWith(t, "", map[string]string{metricsToken: token}, + func(server *proxy.Server) { server.ReadTimeout = readTimeout }) + + conn := dial(t, s.addr) + sendQuestionWithPartOfItsBody(t, conn) + tc.stop(t, conn) + + err := conn.SetReadDeadline(time.Now().Add(waitLimit)) + if err != nil { + t.Fatalf("set the deadline: %v", err) + } + + sent, err := io.ReadAll(conn) + if err != nil || len(sent) != 0 { + t.Errorf("rproxy read %q (%v), want the connection closed unanswered", + sent, err) + } + + wantProcessMessages(t, s, "a question's body stopped arriving", 1) + wantNoRequestLines(t, s) + + // The id was not taken, and the metrics count nothing for it. + again := s.question(siteA, client, "/") + again.Header.Set(requestIDHeader, "an-id") + wantDisposition(t, do(t, again), http.StatusOK, proxy.DispositionAllow) + wantNoSeries(t, s.metrics(), + `smallwebwaf_size_and_time_limit_hits_total{instance="`+alertInstance+ + `",limit="SWWAF_CLIENT_REQUEST_TIMEOUT"}`) + }) + } +} + +func TestQuestionDeniedGetsTheBanResponse(t *testing.T) { + t.Parallel() + + for _, status := range []int{http.StatusForbidden, http.StatusTooManyRequests} { + t.Run(strconv.Itoa(status), func(t *testing.T) { + t.Parallel() + + s := startServeWAF(t, "", map[string]string{ + denyNets: client, banResponse: strconv.Itoa(status), + }) + + got := s.ask(client, "/") + wantDisposition(t, got, status, proxy.DispositionDeny) + wantBody(t, got, http.StatusText(status)+"\n") + wantLine(t, s.out.requestLine(t), status, requestlog.ActionDenied) + }) + } +} + +func TestBanResponseCloseIsAnAnswerOfItsOwn(t *testing.T) { + t.Parallel() + + s := startServeWAF(t, "", map[string]string{ + denyNets: client, banResponse: banResponseClose, metricsToken: token, + }) + + got := s.ask(client, "/") + wantDisposition(t, got, http.StatusOK, proxy.DispositionClose) + wantBody(t, got, "") + // The visitor is sent nothing. + wantLine(t, s.out.requestLine(t), 0, requestlog.ActionDenied) + wantMetric(t, s.metrics(), `smallwebwaf_requests_total{action="denied",instance="`+ + alertInstance+`",status_class="none"}`, 1) +} + +func TestBlockRuleIsDeniedWith403AndABanRuleWithTheBanResponse(t *testing.T) { + t.Parallel() + + s := startServeWAF(t, "", map[string]string{ + rulesDir: writeRules(t, testRules), + banResponse: "429", + }) + + got := s.ask(client, "/blocked") + wantDisposition(t, got, http.StatusForbidden, proxy.DispositionDeny) + wantLine(t, s.out.requestLine(t), http.StatusForbidden, + requestlog.ActionRuleBlocked) + + got = s.ask(client, "/.env") + wantDisposition(t, got, http.StatusTooManyRequests, proxy.DispositionDeny) + wantLine(t, s.out.requestLines(t, 2)[1], http.StatusTooManyRequests, + requestlog.ActionBanned) +} + +func TestQuestionUnderTheOwnPrefixIsDeniedWith404AfterTheChecks(t *testing.T) { + t.Parallel() + + s := startServeWAF(t, "", map[string]string{ + metricsToken: token, denyNets: otherClient, + }) + + for _, path := range []string{proxy.HealthPath, proxy.MetricsPath, proxy.ReportPath} { + got := s.ask(client, path) + wantDisposition(t, got, http.StatusNotFound, proxy.DispositionDeny) + } + + wantLine(t, s.out.requestLine(t), http.StatusNotFound, requestlog.ActionAdmin) + + got := s.ask(otherClient, proxy.MetricsPath) + wantDisposition(t, got, http.StatusForbidden, proxy.DispositionDeny) +} + +func TestObserveModeAllowsAndLogsWhatEnforceModeWouldHaveDone(t *testing.T) { + t.Parallel() + + s := startServeWAF(t, "", map[string]string{mode: observe, denyNets: client}) + + got := s.ask(client, "/") + wantDisposition(t, got, http.StatusOK, proxy.DispositionAllow) + + line := s.out.requestLine(t) + wantLine(t, line, http.StatusOK, requestlog.ActionForward) + wantWouldAction(t, line, requestlog.ActionDenied) +} + +func TestObserveModeRefusalAnsweredAllowCountsForTheErrorBurst(t *testing.T) { + t.Parallel() + + s := startServeWAF(t, "", map[string]string{ + mode: observe, rulesDir: writeRules(t, testRules), errorBurstThreshold: "1", + }) + + for range 2 { + got := s.ask(client, "/blocked") + wantDisposition(t, got, http.StatusOK, proxy.DispositionAllow) + } + + lines := s.out.requestLines(t, 2) + if lines[0].LimitHit != "" || lines[1].LimitHit != requestlog.LimitHitErrorBurst { + t.Errorf("log lines have limit_hit %q and %q, want none and %q", + lines[0].LimitHit, lines[1].LimitHit, requestlog.LimitHitErrorBurst) + } +} + +func TestDeniedQuestionCountsForTheErrorBurst(t *testing.T) { + t.Parallel() + + s := startServeWAF(t, "", map[string]string{ + rulesDir: writeRules(t, testRules), errorBurstThreshold: "1", + }) + + for range 2 { + got := s.ask(client, "/blocked") + wantDisposition(t, got, http.StatusForbidden, proxy.DispositionDeny) + } + + lines := s.out.requestLines(t, 2) + if lines[0].LimitHit != "" || lines[1].LimitHit != requestlog.LimitHitErrorBurst { + t.Errorf("log lines have limit_hit %q and %q, want none and %q", + lines[0].LimitHit, lines[1].LimitHit, requestlog.LimitHitErrorBurst) + } +} + +func TestAllowCarriesTheLookupHeadersWhileTheSettingIsSet(t *testing.T) { + t.Parallel() + + for _, tc := range []struct { + setting string + wantASN, wantCountry string + }{ + {"true", "AS64496", "DE"}, + {"false", "", ""}, + } { + t.Run(tc.setting, func(t *testing.T) { + t.Parallel() + + geojsURL, _ := startGeoJS(t) + s := startServeWAF(t, geojsURL, map[string]string{ + addLookupHeaders: tc.setting, lookupTimeout: "1h", deniedCountries: "kp", + }) + + got := s.ask(fromDE, "/") + wantDisposition(t, got, http.StatusOK, proxy.DispositionAllow) + + asn, country := got.header.Get("X-Client-Asn"), got.header.Get("X-Client-Country") + if asn != tc.wantASN || country != tc.wantCountry { + t.Errorf("answered with the AS number %q and the country %q, want %q and %q", + asn, country, tc.wantASN, tc.wantCountry) + } + }) + } +} + +func TestReportCountsTowardAByteLimitWhichThenBans(t *testing.T) { + t.Parallel() + + s := startServeWAF(t, "", map[string]string{bytesLimitPerMinute: byteLimit}) + + // The request's own question counts no bytes: it is the report that does. + id := s.allowed(client) + s.allowed(client) + + if held := s.server.Ledger.Snapshot(); len(held) != 0 { + t.Fatalf("banned before any report: %v", held) + } + + // 100 bytes are over the limit of 99. + got := s.report(proxy.Report{ + RequestID: id, Status: http.StatusBadGateway, + RequestBytes: bodyBytes, ResponseBytes: answerBytes, + }) + wantStatus(t, got, http.StatusNoContent) + + held := s.server.Ledger.Snapshot() + if len(held) != 1 || held[0].Cause != bans.CauseLimit || + held[0].Notes.Kind != ratelimit.KindBytes || + held[0].Notes.Request.Status != http.StatusBadGateway || + held[0].Notes.Request.Host != siteA || + held[0].Notes.Request.Path != "/" || + held[0].Notes.Request.Method != http.MethodGet { + t.Errorf("bans held are %+v", held) + } + + third := s.out.requestLines(t, 3)[2] + if third.ServeWAFLine != requestlog.ServeWAFLineReport || + third.LimitHit != minuteBytes || third.Offence != requestlog.OffenceLimit || + third.BanExpires == "" { + t.Errorf("the report's log line is %+v", third.Line) + } + + got = s.ask(client, "/") + wantDisposition(t, got, http.StatusForbidden, proxy.DispositionDeny) +} + +func TestReportCountsAtThePercentageItsQuestionWasGiven(t *testing.T) { + t.Parallel() + + geojsURL, _ := startGeoJS(t) + // 100 bytes pass half of a limit of 99 and are within a limit of 199. + s := startServeWAF(t, geojsURL, map[string]string{ + bytesLimitPerMinute: "199", countryBytesPercent: countryDEHalf, + lookupTimeout: "1h", + }) + + s.passedOn(fromKP) + wantDisposition(t, s.ask(fromKP, "/"), http.StatusOK, proxy.DispositionAllow) + + s.passedOn(fromDE) + wantDisposition(t, s.ask(fromDE, "/"), http.StatusForbidden, proxy.DispositionDeny) +} + +func TestReportAddsTheRequestToItsClientsHistoryAndTheAnomalyCounters(t *testing.T) { + t.Parallel() + + s := startServeWAF(t, "", map[string]string{anomalyClientPerMinute: "1000"}) + + id := s.allowed(client) + + if requests := s.history().Requests; requests != 0 { + t.Errorf("history holds %d requests before the report, want none", requests) + } + + if counters := s.server.Anomalies.Snapshot(); len(counters) != 0 { + t.Errorf("anomaly counters %v before the report, want none", counters) + } + + got := s.report(proxy.Report{ + RequestID: id, Status: http.StatusBadGateway, + RequestBytes: bodyBytes, ResponseBytes: answerBytes, + }) + wantStatus(t, got, http.StatusNoContent) + + history := s.history() + if history.Requests != 1 || history.Forwarded != 1 || + history.RequestBytes != bodyBytes || history.ResponseBytes != answerBytes || + history.Responses.Status5xx != 1 { + t.Errorf("history is %+v", history) + } + + if counters := s.server.Anomalies.Snapshot(); len(counters) != 1 { + t.Errorf("anomaly counters %v, want the client's", counters) + } +} + +func TestDeniedRequestIsFinishedAtItsDecision(t *testing.T) { + t.Parallel() + + s := startServeWAF(t, "", map[string]string{ + rateLimitPerMinute: "1", metricsToken: token, anomalyClientPerMinute: "1000", + }) + + s.allowed(client) + + denied := s.question(siteA, client, "/") + wantDisposition(t, do(t, denied), http.StatusForbidden, proxy.DispositionDeny) + + counters := s.server.Anomalies.Snapshot() + if len(counters) != 1 || counters[0].Minute.Current != 1 { + t.Errorf("anomaly counters %+v, want the denied request's", counters) + } + + history := s.history() + if history.Requests != 1 || history.Refused != 1 { + t.Errorf("history is %+v, want the refused request alone", history) + } + + const series = `smallwebwaf_requests_total{action="rate_limited",instance="` + + alertInstance + `",status_class="4xx"}` + + wantMetric(t, s.metrics(), series, 1) + + got := s.report(proxy.Report{ + RequestID: denied.Header.Get(requestIDHeader), Status: http.StatusForbidden, + ResponseBytes: 1000, DurationTotal: 2, + }) + wantStatus(t, got, http.StatusNoContent) + + if after := s.history(); after != history { + t.Errorf("the report changed the history to %+v, from %+v", after, history) + } + + counters = s.server.Anomalies.Snapshot() + if len(counters) != 1 || counters[0].Minute.Current != 1 || + counters[0].MinuteBytes.Current != 0 { + t.Errorf("the report changed the anomaly counters: %+v", counters) + } + + wantMetric(t, s.metrics(), series, 1) +} + +func TestMetricsCountAnAllowedRequestFromItsReport(t *testing.T) { + t.Parallel() + + s := startServeWAF(t, "", map[string]string{metricsToken: token}) + + id := s.allowed(client) + + const series = `{action="forward",instance="` + alertInstance + `",status_class="5xx"}` + + metrics := s.metrics() + wantNoSeries(t, metrics, "smallwebwaf_requests_total"+series) + wantMetric(t, metrics, `smallwebwaf_requests_in_flight{instance="`+alertInstance+`"}`, + 1) // the scrape itself + + got := s.report(proxy.Report{ + RequestID: id, Status: http.StatusBadGateway, + RequestBytes: bodyBytes, ResponseBytes: answerBytes, + DurationTotal: 5, DurationUpstreamTotal: 4, + }) + wantStatus(t, got, http.StatusNoContent) + + metrics = s.metrics() + wantMetric(t, metrics, "smallwebwaf_requests_total"+series, 1) + wantMetric(t, metrics, "smallwebwaf_request_bytes_total"+series, bodyBytes) + wantMetric(t, metrics, "smallwebwaf_response_bytes_total"+series, answerBytes) + wantMetric(t, metrics, `smallwebwaf_upstream_duration_seconds_sum{instance="`+ + alertInstance+`"}`, 0.004) + + // A scrape adds its own time to the request duration when it ends, so this + // check needs a server that has not been scraped yet. + unscraped := startServeWAF(t, "", map[string]string{metricsToken: token}) + + got = unscraped.report(proxy.Report{ + RequestID: unscraped.allowed(client), Status: http.StatusBadGateway, DurationTotal: 5, + }) + wantStatus(t, got, http.StatusNoContent) + wantMetric(t, unscraped.metrics(), + `smallwebwaf_request_duration_seconds_sum{instance="`+alertInstance+`"}`, 0.005) +} + +func TestReportForARequestNotHeldIsIgnored(t *testing.T) { + t.Parallel() + + s := startServeWAF(t, "", map[string]string{bytesLimitPerMinute: byteLimit}) + + got := s.report(proxy.Report{RequestID: "no-such-request", ResponseBytes: 1000}) + wantStatus(t, got, http.StatusNoContent) + + if clients := s.server.Limiter.Snapshot(); len(clients) != 0 { + t.Errorf("the table of clients holds %v", clients) + } + + if held := s.server.Ledger.Snapshot(); len(held) != 0 { + t.Errorf("bans held are %v", held) + } + + wantNoRequestLines(t, s) + wantProcessMessages(t, s, "a report is refused", 0) + + // A second report of the same request does not count it twice. + id := s.allowed(client) + for range 2 { + got = s.report(proxy.Report{RequestID: id, ResponseBytes: answerBytes}) + wantStatus(t, got, http.StatusNoContent) + } + + if counted := s.history().ResponseBytes; counted != answerBytes { + t.Errorf("history holds %d response bytes, want %d", counted, answerBytes) + } + + // The question's line and one for the first report, none for the second. + if lines := s.out.requestLines(t, 2); len(lines) != 2 { + t.Errorf("request log lines are %+v, want two", lines) + } +} + +// reportFields is the JSON of a report of the request id with a true value for +// every field, for a test to change and leave out. +func reportFields(id string) map[string]any { + return map[string]any{ + reportID: id, reportStatus: http.StatusOK, + reportRequestBytes: bodyBytes, reportResponseBytes: answerBytes, + reportDuration: 5, reportUpstreamDuration: 4, + } +} + +// encodedReport returns a report of id, with the value of each field in changed +// in place of its true one, and without the fields named in missing. +func encodedReport( + t *testing.T, id string, changed map[string]any, missing ...string, +) string { + t.Helper() + + fields := reportFields(id) + maps.Copy(fields, changed) + + for _, name := range missing { + delete(fields, name) + } + + encoded, err := json.Marshal(fields) + if err != nil { + t.Fatalf("encode the report: %v", err) + } + + return string(encoded) +} + +func TestReportThatIsNotOneObjectWithAllItsFieldsIsRefused(t *testing.T) { + t.Parallel() + + s := startServeWAF(t, "", nil) + + id := s.allowed(client) + whole := encodedReport(t, id, nil) + fields := reportFields(id) + bodies := make([]string, 0, 9+3*len(fields)) + bodies = append(bodies, + "", "not JSON", "null", "[]", `"text"`, "{}", + whole+" {}", + whole+whole, + `{"request_id":"`+id+`"}`) + + for name := range fields { + bodies = append(bodies, + encodedReport(t, id, nil, name), + encodedReport(t, id, map[string]any{name: nil}), + encodedReport(t, id, map[string]any{name: []int{1}})) + } + + for _, body := range bodies { + wantStatus(t, s.reportBody(body), http.StatusBadRequest) + } + + wantProcessMessages(t, s, "a report is refused", len(bodies)) + + // None of those finished the request, so its true report still counts. + wantStatus(t, s.reportBody(whole), http.StatusNoContent) + + if counted := s.history().ResponseBytes; counted != answerBytes { + t.Errorf("history holds %d response bytes, want %d", counted, answerBytes) + } +} + +func TestReportOver1MiBIsRefusedAndOneOfExactly1MiBIsTaken(t *testing.T) { + t.Parallel() + + s := startServeWAF(t, "", nil) + + id := s.allowed(client) + // The padding is a field of no meaning here, so only the size can refuse it. + padded := func(size int) string { + return encodedReport(t, id, map[string]any{"padding": strings.Repeat("x", size)}) + } + paddingBytes := reportBodyLimit - len(padded(0)) + + wantStatus(t, s.reportBody(padded(paddingBytes+1)), http.StatusBadRequest) + wantProcessMessages(t, s, "a report is refused", 1) + + if requests := s.history().Requests; requests != 0 { + t.Errorf("history holds %d requests, want the refused report not counted", requests) + } + + wantStatus(t, s.reportBody(padded(paddingBytes)), http.StatusNoContent) + + if requests := s.history().Requests; requests != 1 { + t.Errorf("history holds %d requests, want the report of 1 MiB counted", requests) + } +} + +func TestReportWithAValueOutOfRangeIsRefused(t *testing.T) { + t.Parallel() + + s := startServeWAF(t, "", nil) + + id := s.allowed(client) + cases := []struct { + field string + value any + }{ + {reportStatus, -5}, + {reportStatus, 99}, + {reportStatus, 600}, + {reportStatus, 99999}, + {reportStatus, 200.5}, + {reportStatus, "200"}, + {reportRequestBytes, -1}, + {reportRequestBytes, reportedBytesLimit + 1}, + {reportRequestBytes, int64(math.MaxInt64)}, + {reportRequestBytes, 1.5}, + {reportResponseBytes, -1}, + {reportResponseBytes, reportedBytesLimit + 1}, + {reportResponseBytes, int64(math.MaxInt64)}, + {reportDuration, -1}, + {reportDuration, 1e13}, + {reportDuration, "5"}, + {reportUpstreamDuration, -0.5}, + {reportUpstreamDuration, 1e13}, + } + + for _, tc := range cases { + body := encodedReport(t, id, map[string]any{tc.field: tc.value}) + wantStatus(t, s.reportBody(body), http.StatusBadRequest) + } + + wantProcessMessages(t, s, "a report is refused", len(cases)) + + if held := s.history(); held.Requests != 0 { + t.Errorf("a refused report counted the request: %+v", held) + } +} + +func TestReportValuesAtTheEndsOfTheirRangesAreTaken(t *testing.T) { + t.Parallel() + + s := startServeWAF(t, "", map[string]string{ + bytesLimitPerMinute: off, bytesLimitPerHour: off, bytesLimitPerDay: off, + }) + + for _, tc := range []struct { + field string + value any + }{ + {reportStatus, 0}, + {reportStatus, 100}, + {reportStatus, 599}, + {reportRequestBytes, 0}, + {reportRequestBytes, reportedBytesLimit}, + {reportResponseBytes, reportedBytesLimit}, + {reportDuration, 0}, + {reportDuration, 1e12}, + {reportUpstreamDuration, 1e12}, + } { + id := s.allowed(client) + body := encodedReport(t, id, map[string]any{tc.field: tc.value}) + wantStatus(t, s.reportBody(body), http.StatusNoContent) + } + + // A field of no meaning here is read as if it were not there. + id := s.allowed(client) + body := encodedReport(t, id, map[string]any{"some_later_field": 1}) + wantStatus(t, s.reportBody(body), http.StatusNoContent) + wantProcessMessages(t, s, "a report is refused", 0) +} + +func TestBytesAtTheirMostEachWayBanRatherThanOverflow(t *testing.T) { + t.Parallel() + + s := startServeWAF(t, "", map[string]string{bytesLimitPerMinute: byteLimit}) + + got := s.report(proxy.Report{ + RequestID: s.allowed(client), Status: http.StatusOK, + RequestBytes: reportedBytesLimit, ResponseBytes: reportedBytesLimit, + }) + wantStatus(t, got, http.StatusNoContent) + + if held := s.server.Ledger.Snapshot(); len(held) != 1 { + t.Errorf("bans held are %v, want the client's", held) + } +} + +// wantReportsFields checks that line has what the report in +// TestReportWritesTheLastLineWithTheDecisionsAndTheReportsFields says. +func wantReportsFields(t *testing.T, line logLine) { + t.Helper() + + if line.ServeWAFLine != requestlog.ServeWAFLineReport || + line.Status != http.StatusCreated || + line.RequestBytes != bodyBytes || line.ResponseBytes != answerBytes || + line.DurationTotal != 12.5 || line.DurationUpstreamTotal == nil || + *line.DurationUpstreamTotal != 10.25 { + t.Errorf("the report's line lacks the report's fields: %+v", line.Line) + } +} + +// wantSameDecision checks that the report's line is the decision's, but for the +// fields a report says and the byte totals its counting changes. +func wantSameDecision(t *testing.T, reported, decided logLine) { + t.Helper() + + same := reported.Line + same.ServeWAFLine = decided.ServeWAFLine + same.Status = decided.Status + same.RequestBytes = decided.RequestBytes + same.ResponseBytes = decided.ResponseBytes + same.DurationTotal = decided.DurationTotal + same.DurationUpstreamTotal = decided.DurationUpstreamTotal + same.Counts.MinuteBytes = decided.Counts.MinuteBytes + same.Counts.HourBytes = decided.Counts.HourBytes + same.Counts.DayBytes = decided.Counts.DayBytes + + if !reflect.DeepEqual(same, decided.Line) { + t.Errorf("the report's line lacks the decision's fields:\n%+v\n%+v", + reported.Line, decided.Line) + } +} + +// wantDecisionsFields checks what servewaf answered and noted on the decision's +// line, of a request with these fields. +func wantDecisionsFields(t *testing.T, line logLine, id, query, accept string) { + t.Helper() + + if line.ServeWAFLine != requestlog.ServeWAFLineDecision || + line.Status != http.StatusOK || line.ResponseBytes != 0 || + line.DurationUpstreamTotal != nil || line.DurationChecks == nil || + line.RequestID != id || line.Query != query || + !line.HasAuthorization || !line.HasCookie || + line.RequestHeaders["accept"] != accept || len(line.RuleIDs) != 1 || + line.Counts.Minute != 1 { + t.Errorf("the decision's line is %+v", line.Line) + } +} + +func TestReportWritesTheLastLineWithTheDecisionsAndTheReportsFields(t *testing.T) { + t.Parallel() + + s := startServeWAF(t, "", map[string]string{ + logRequestHeaders: "accept", + rulesDir: writeRules(t, "seen path log ^/page$\n"), + }) + + const ( + secret = "Bearer the-visitors-secret" + cookie = "session=the-visitors-cookie" + verbatim = "q=1&token=visible" + accept = "text/css" + ) + + req := s.question(siteA, client, "/page?"+verbatim) + req.Header.Set("Accept", accept) + req.Header.Set("Authorization", secret) + req.Header.Set("Cookie", cookie) + req.Header.Set("X-Forwarded-Proto", "https") + id := req.Header.Get(requestIDHeader) + + wantDisposition(t, do(t, req), http.StatusOK, proxy.DispositionAllow) + + got := s.report(proxy.Report{ + RequestID: id, Status: http.StatusCreated, + RequestBytes: bodyBytes, ResponseBytes: answerBytes, + DurationTotal: 12.5, DurationUpstreamTotal: 10.25, + }) + wantStatus(t, got, http.StatusNoContent) + + lines := s.out.requestLines(t, 2) + decided, reported := lines[0], lines[1] + + wantDecisionsFields(t, decided, id, verbatim, accept) + wantReportsFields(t, reported) + wantSameDecision(t, reported, decided) + + // The report's line has the request's bytes in the totals, which the + // decision could not know. + if decided.Counts.DayBytes != 0 || reported.Counts.DayBytes != 100 { + t.Errorf("the lines' day_bytes are %v and %v, want 0 and 100", + decided.Counts.DayBytes, reported.Counts.DayBytes) + } + + // Neither line holds a credential. + text := s.out.text() + if strings.Contains(text, "the-visitors-secret") || + strings.Contains(text, "the-visitors-cookie") { + t.Errorf("a line holds a credential:\n%s", text) + } +} + +func TestEveryDeniedRequestsReportGetsItsLastLine(t *testing.T) { + t.Parallel() + + for _, tc := range []struct { + name string + env map[string]string + path string + status int + action string + }{ + { + "deny", map[string]string{denyNets: client}, "/", http.StatusForbidden, + requestlog.ActionDenied, + }, + { + "close", map[string]string{denyNets: client, banResponse: banResponseClose}, "/", + 0, requestlog.ActionDenied, + }, + {"own prefix", nil, proxy.HealthPath, http.StatusNotFound, requestlog.ActionAdmin}, + } { + t.Run(tc.name, func(t *testing.T) { + t.Parallel() + + s := startServeWAF(t, "", tc.env) + + req := s.question(siteA, client, tc.path) + id := req.Header.Get(requestIDHeader) + do(t, req) + + got := s.report(proxy.Report{ + RequestID: id, Status: tc.status, ResponseBytes: 8, DurationTotal: 3, + }) + wantStatus(t, got, http.StatusNoContent) + + line := s.out.requestLines(t, 2)[1] + wantLine(t, line, tc.status, tc.action) + + if line.ServeWAFLine != requestlog.ServeWAFLineReport || + line.RequestID != id || line.ResponseBytes != 8 || line.DurationTotal != 3 || + line.DurationUpstreamTotal != nil { + t.Errorf("the report's line is %+v", line.Line) + } + }) + } +} + +func TestOldestAnsweredRequestIsDroppedPastTheBound(t *testing.T) { + t.Parallel() + + s := startServeWAF(t, "", map[string]string{ + rateLimitExemptNets: otherClient, + }) + + oldest := s.allowed(client) + next := s.allowed(client) + + // One connection for them all, as rproxy keeps one open. + shared := newClient(t) + + for range heldLimit - 1 { + res, err := shared.Do(s.question(siteA, otherClient, "/")) + if err != nil { + t.Fatalf("question: %v", err) + } + + wantDisposition(t, readAnswer(res), http.StatusOK, proxy.DispositionAllow) + } + + // The table is full again with next and those, without oldest. + got := s.report(proxy.Report{RequestID: oldest, ResponseBytes: answerBytes}) + wantStatus(t, got, http.StatusNoContent) + + if counted := s.history().Requests; counted != 0 { + t.Errorf("history holds %d requests, want the dropped one not counted", counted) + } + + got = s.report(proxy.Report{RequestID: next, ResponseBytes: answerBytes}) + wantStatus(t, got, http.StatusNoContent) + + if counted := s.history().Requests; counted != 1 { + t.Errorf("history holds %d requests, want the one still held", counted) + } +} + +func TestBanMadeThroughOneHostRefusesTheClientOnAnother(t *testing.T) { + t.Parallel() + + s := startServeWAF(t, "", map[string]string{rulesDir: writeRules(t, testRules)}) + + got := do(t, s.question(siteA, client, "/.env")) + wantDisposition(t, got, http.StatusForbidden, proxy.DispositionDeny) + + got = do(t, s.question(siteB, client, "/")) + wantDisposition(t, got, http.StatusForbidden, proxy.DispositionDeny) + + lines := s.out.requestLines(t, 2) + if lines[0].Host != siteA || lines[1].Host != siteB { + t.Errorf("log lines have the hosts %q and %q", lines[0].Host, lines[1].Host) + } + + held := s.server.Ledger.Snapshot() + if len(held) != 1 || held[0].Notes.Request.Host != siteA { + t.Errorf("bans held are %+v, want one whose notes name %s", held, siteA) + } + + waiting := s.queue.Snapshot().Waiting[alerts.DestinationWebhook] + if len(waiting) == 0 || waiting[0].Host != siteA { + t.Errorf("alerts waiting are %+v, want the ban's, naming %s", waiting, siteA) + } +} + +func TestRequestsOnEverySiteCountTowardOneSetOfLimits(t *testing.T) { + t.Parallel() + + s := startServeWAF(t, "", map[string]string{rateLimitPerMinute: "1"}) + + s.allowed(client) + + got := do(t, s.question(siteB, client, "/")) + wantDisposition(t, got, http.StatusForbidden, proxy.DispositionDeny) +} + +func TestOwnEndpointsAnswerOnlyWithoutTheClientHeader(t *testing.T) { + t.Parallel() + + // Were a check run on its requests, localhost's would be refused. + s := startServeWAF(t, "", map[string]string{ + metricsToken: token, adminToken: adminSecret, denyNets: localhost, + }) + + for _, tc := range []struct { + path, authorization string + want int + }{ + {proxy.HealthPath, "", http.StatusOK}, + {proxy.MetricsPath, bearer, http.StatusOK}, + {proxy.MetricsPath, "", http.StatusUnauthorized}, + {proxy.BansPath, adminBearer, http.StatusOK}, + {proxy.BansPath, "", http.StatusUnauthorized}, + {proxy.ClientsPath + client, adminBearer, http.StatusOK}, + {proxy.ReportPath, "", http.StatusNotFound}, + {"/_smallwebwaf/other", "", http.StatusNotFound}, + {"/", "", http.StatusNotFound}, + } { + req := newRequest(t, http.MethodGet, s.addr, tc.path, http.NoBody) + req.Header.Set("Authorization", tc.authorization) + wantDisposition(t, do(t, req), tc.want, "") + + // The same request carrying the header is a question about it. + req = s.question(siteA, client, tc.path) + req.Header.Set("Authorization", tc.authorization) + + if strings.HasPrefix(tc.path, "/_smallwebwaf/") { + wantDisposition(t, do(t, req), http.StatusNotFound, proxy.DispositionDeny) + } else { + wantDisposition(t, do(t, req), http.StatusOK, proxy.DispositionAllow) + } + } +} + +func TestOwnEndpointsLinesDoNotSayWhichOfTheTwoTheyAre(t *testing.T) { + t.Parallel() + + s := startServeWAF(t, "", nil) + + wantStatus(t, get(t, s.addr, proxy.HealthPath), http.StatusOK) + + line := s.out.requestLine(t) + if _, there := line.fields["servewaf_line"]; there || line.Path != proxy.HealthPath { + t.Errorf("the health check's log line is %+v", line.fields) + } +} + +func TestSettingsOfTheLimitsAndTheProxiesAreIgnored(t *testing.T) { + t.Parallel() + + s := startServeWAF(t, "", map[string]string{ + requestMaxBytes: "1", + clientHeaderMaxBytes: "5K", + clientRequestTimeout: off, + clientResponseTimeout: off, + trustedProxies: localhost, + }) + + req := s.questionWith(http.MethodPost, siteA, client, "/", + strings.NewReader("a body")) + req.Header.Set(proxy.ContentLengthHeader, "1073741824") + req.Header.Set(forwardedFor, otherClient) + req.Header.Set("X-Long", strings.Repeat("x", 40<<10)) + + got := do(t, req) + wantDisposition(t, got, http.StatusOK, proxy.DispositionAllow) + + line := s.out.requestLine(t) + if line.ClientIP != client || line.ContentLength != 1<<30 { + t.Errorf("log line has client_ip %q and content_length %d", + line.ClientIP, line.ContentLength) + } +} + +func TestShortClientResponseTimeoutIsIgnored(t *testing.T) { + t.Parallel() + + // The setting would leave no time to send any answer. + s := startServeWAF(t, "", map[string]string{ + clientResponseTimeout: time.Nanosecond.String(), denyNets: client, + }) + + wantDisposition(t, s.ask(client, "/"), http.StatusForbidden, proxy.DispositionDeny) + wantStatus(t, get(t, s.addr, proxy.HealthPath), http.StatusOK) +} + +func TestShortClientRequestTimeoutIsIgnored(t *testing.T) { + t.Parallel() + + // The setting would have run out while the Core Rule Set read the body. Go's + // server would then take the connection for gone, and the line say aborted. + s := startServeWAF(t, "", map[string]string{ + clientRequestTimeout: time.Nanosecond.String(), wafMode: block, wafBodyLimit: "1M", + }) + + req := s.questionWith(http.MethodPost, siteA, client, "/", strings.NewReader("a=1")) + req.Header.Set("Content-Type", formData) + wantDisposition(t, do(t, req), http.StatusOK, proxy.DispositionAllow) + + if line := s.out.requestLine(t); line.Aborted { + t.Errorf("log line says aborted: %+v", line.Line) + } +} + +func TestServerHasTheTimeLimitsSPECGives(t *testing.T) { + t.Parallel() + + s := startServeWAF(t, "", nil) + + if s.server.ReadTimeout != time.Minute || s.server.WriteTimeout != time.Minute || + s.server.IdleTimeout != 2*time.Minute { + t.Errorf("the read, write and idle limits are %v, %v and %v, want 1m, 1m and 2m", + s.server.ReadTimeout, s.server.WriteTimeout, s.server.IdleTimeout) + } +} + +func TestChecksSeeTheHeadersTheVisitorSent(t *testing.T) { + t.Parallel() + + s := startServeWAF(t, "", map[string]string{ + rulesDir: writeRules(t, ` +own-client header:X-Smallwebwaf-Client block . +own-peer header:X-Smallwebwaf-Peer block . +own-length header:X-Smallwebwaf-Content-Length block . +announced header:Content-Length block ^70000$ +carried header:Content-Length block ^6$ +`), + }) + + // What the question's own Content-Length says, 6, is not what the visitor + // announced. + for _, tc := range []struct { + announced string + want int + }{ + {"70000", http.StatusForbidden}, + {"", http.StatusOK}, + {"6000", http.StatusOK}, + } { + req := s.questionWith(http.MethodPost, siteA, client, "/", + strings.NewReader("a body")) + + if tc.announced != "" { + req.Header.Set(proxy.ContentLengthHeader, tc.announced) + } + + wantStatus(t, do(t, req), tc.want) + } +} + +func TestCoreRuleSetReadsTheBodyAQuestionCarries(t *testing.T) { + t.Parallel() + + s := startServeWAF(t, "", map[string]string{ + wafMode: block, wafBodyLimit: "1M", + }) + + ask := func(contentType, body string) answer { + req := s.questionWith(http.MethodPost, siteA, client, "/", + strings.NewReader(body)) + req.Header.Set("Content-Type", contentType) + req.Header.Set(proxy.ContentLengthHeader, "100000") + + return do(t, req) + } + + got := ask(formData, "id=1'%20OR%20'1'='1") + wantDisposition(t, got, http.StatusForbidden, proxy.DispositionDeny) + + // The first 64 KiB of a longer JSON body cannot be parsed, which is no + // attack, so the Core Rule Set does not read it. + cut := `{"text":"` + strings.Repeat("x", proxy.QuestionBodyMaxBytes) + got = ask("application/json", cut[:proxy.QuestionBodyMaxBytes]) + wantDisposition(t, got, http.StatusOK, proxy.DispositionAllow) +} + +func TestCoreRuleSetAlertNamesTheSite(t *testing.T) { + t.Parallel() + + s := startServeWAF(t, "", map[string]string{wafMode: block}) + + got := s.ask(client, sqlInjection) + wantDisposition(t, got, http.StatusForbidden, proxy.DispositionDeny) + + waiting := s.queue.Snapshot().Waiting[alerts.DestinationWebhook] + if len(waiting) != 1 || waiting[0].Event != alerts.EventWAFBlock || + waiting[0].Host != siteA { + t.Errorf("alerts waiting are %+v, want a waf_block naming %s", waiting, siteA) + } +} + +func TestReputationHitAlertNamesTheSite(t *testing.T) { + t.Parallel() + + s := startServeWAF(t, "", map[string]string{blocklistURLs: dropURL}) + loadLists(t, s.server, map[string][]string{dropURL: {client}}) + + wantDisposition(t, s.ask(client, "/"), http.StatusForbidden, proxy.DispositionDeny) + + waiting := s.queue.Snapshot().Waiting[alerts.DestinationWebhook] + if len(waiting) != 1 || waiting[0].Event != alerts.EventReputationHit || + waiting[0].Host != siteA { + t.Errorf("alerts waiting are %+v, want a reputation_hit naming %s", waiting, siteA) + } +} + +func TestAnomalyAlertNamesTheSite(t *testing.T) { + t.Parallel() + + s := startServeWAF(t, "", map[string]string{anomalyClientPerMinute: "1"}) + + for range 2 { + s.passedOn(client) + } + + waiting := s.queue.Snapshot().Waiting[alerts.DestinationWebhook] + if len(waiting) != 1 || waiting[0].Event != alerts.EventAnomaly || + waiting[0].Host != siteA { + t.Errorf("alerts waiting are %+v, want an anomaly naming %s", waiting, siteA) + } +} diff --git a/internal/requestlog/requestlog.go b/internal/requestlog/requestlog.go index 829f8a0..c05c0c2 100644 --- a/internal/requestlog/requestlog.go +++ b/internal/requestlog/requestlog.go @@ -1,6 +1,6 @@ // Package requestlog writes the lines smallwebwaf prints on stdout: one -// JSON object per request, marked "type":"request", and the process's own -// messages as JSON lines marked "type":"process". +// JSON object per request, or two from servewaf, marked "type":"request", and +// the process's own messages as JSON lines marked "type":"process". package requestlog import ( @@ -60,6 +60,16 @@ const OffenceLimit = "limit" // that broke the error burst. const LimitHitErrorBurst = "error_burst" +// The servewaf_line a request line names, which says which of servewaf's two +// lines about a question it is. +const ( + // ServeWAFLineDecision is servewaf's line at its decision. + ServeWAFLineDecision = "decision" + // ServeWAFLineReport is servewaf's line when rproxy's report comes, with + // the decision's fields and the report's together. + ServeWAFLineReport = "report" +) + // timeLayout is RFC 3339 with milliseconds. const timeLayout = "2006-01-02T15:04:05.000Z07:00" @@ -70,6 +80,9 @@ const timeLayout = "2006-01-02T15:04:05.000Z07:00" //nolint:tagliatelle // docs/request-log.md names the fields in snake_case type Line struct { Type string `json:"type"` + // ServeWAFLine is ServeWAFLineDecision or ServeWAFLineReport on servewaf's + // two lines about a question. + ServeWAFLine string `json:"servewaf_line,omitempty"` // The standard web log fields. Scheme is how the client reached // smallwebwaf, or the trusted proxy in front of it. diff --git a/internal/smallwebwaf/servewaf_test.go b/internal/smallwebwaf/servewaf_test.go new file mode 100644 index 0000000..6f6f5da --- /dev/null +++ b/internal/smallwebwaf/servewaf_test.go @@ -0,0 +1,210 @@ +package smallwebwaf_test + +import ( + "context" + "crypto/rand" + "io" + "net/http" + "os" + "os/exec" + "path/filepath" + "strings" + "syscall" + "testing" + "time" + + "sneak.berlin/go/smallwebwaf/internal/proxy" + "sneak.berlin/go/smallwebwaf/internal/smallwebwaf" +) + +// commandForMain, while set in the environment, has this test binary run Main +// with the command it gives, in place of the tests, so that a test can run +// Main in a process of its own. +const commandForMain = "SMALLWEBWAF_TEST_COMMAND" + +func TestMain(m *testing.M) { + command, set := os.LookupEnv(commandForMain) + if !set { + os.Exit(m.Run()) + } + + os.Args = strings.Fields("smallwebwaf " + command) + + os.Exit(smallwebwaf.Main(testVersion)) +} + +func TestMainAnswersQuestionsOnlyAsServeWAF(t *testing.T) { + t.Parallel() + + for _, tc := range []struct { + command, disposition string + }{ + {"servewaf", proxy.DispositionAllow}, + {"", ""}, + } { + t.Run("command "+tc.command, func(t *testing.T) { + t.Parallel() + + binary, err := os.Executable() + if err != nil { + t.Fatalf("find the test binary: %v", err) + } + + ctx, cancel := context.WithTimeout(t.Context(), 3*waitLimit) + defer cancel() + + out := &output{} + cmd := exec.CommandContext(ctx, binary) //nolint:gosec // this test binary + cmd.Stdout = out + + cmd.Env = append(os.Environ(), + commandForMain+"="+tc.command, + listenAddr+"="+localhost+":0", + upstreamURL+"="+startApp(t), + stateDir+"="+t.TempDir(), + rulesDir+"="+t.TempDir(), + lookupSource+"=off") + + err = cmd.Start() + if err != nil { + t.Fatalf("start: %v", err) + } + + t.Cleanup(func() { + _ = cmd.Process.Kill() + _ = cmd.Wait() + }) + + addr, _ := out.line(t, "msg", "starting")["address"].(string) + ask(t, addr, "/", http.StatusOK, tc.disposition) + + err = cmd.Process.Signal(syscall.SIGTERM) + if err != nil { + t.Fatalf("signal: %v", err) + } + + err = cmd.Wait() + if err != nil { + t.Errorf("exit: %v; output:\n%s", err, out.text()) + } + }) + } +} + +// ask sends servewaf at addr rproxy's question about a request for path from +// placed, checks the answer's status and disposition, and closes it. +func ask(t *testing.T, addr, path string, status int, disposition string) { + t.Helper() + + req, err := http.NewRequestWithContext(t.Context(), http.MethodGet, + "http://"+addr+path, http.NoBody) + if err != nil { + t.Fatalf("new request: %v", err) + } + + req.Header.Set(proxy.ClientHeader, placed) + req.Header.Set("X-Request-ID", rand.Text()) + + transport := &http.Transport{} + defer transport.CloseIdleConnections() + + res, err := (&http.Client{Transport: transport}).Do(req) + if err != nil { + t.Fatalf("question: %v", err) + } + + _, _ = io.Copy(io.Discard, res.Body) + _ = res.Body.Close() + + if res.StatusCode != status || + res.Header.Get(proxy.DispositionHeader) != disposition { + t.Errorf("answered %d with disposition %q, want %d with %q", res.StatusCode, + res.Header.Get(proxy.DispositionHeader), status, disposition) + } +} + +// runServeWAF runs `smallwebwaf servewaf` with the settings in env, has use +// send it requests at the address it listens on, stops it as SIGTERM does, and +// checks that it stopped in order. +func runServeWAF(t *testing.T, env map[string]string, use func(addr string)) { + t.Helper() + + ctx, stop := context.WithCancel(t.Context()) + out := &output{} + exited := make(chan int, 1) + + go func() { + exited <- runAs(ctx, true, env, out) + }() + + addr, _ := out.line(t, "msg", "starting")["address"].(string) + use(addr) + stop() + + select { + case status := <-exited: + if status != 0 { + t.Fatalf("exit status %d, want 0; output:\n%s", status, out.text()) + } + case <-time.After(waitLimit): + t.Fatal("still running after being told to stop") + } + + out.line(t, "msg", "stopped") +} + +func TestServeWAFKeepsItsBansAcrossARestartAndNeedsNoApp(t *testing.T) { + t.Parallel() + + env := map[string]string{ + listenAddr: localhost + ":0", + stateDir: t.TempDir(), + rulesDir: filepath.Join("..", "..", "share", "rules.d"), + } + + runServeWAF(t, env, func(addr string) { + ask(t, addr, "/", http.StatusOK, proxy.DispositionAllow) + // The default rule file bans a probe for this. + ask(t, addr, "/.env", http.StatusForbidden, proxy.DispositionDeny) + }) + + runServeWAF(t, env, func(addr string) { + ask(t, addr, "/", http.StatusForbidden, proxy.DispositionDeny) + }) +} + +func TestWithoutACommandTheClientHeaderIsNoQuestion(t *testing.T) { + t.Parallel() + + runUntilStopped(t, map[string]string{ + listenAddr: localhost + ":0", + upstreamURL: startApp(t), + stateDir: t.TempDir(), + rulesDir: t.TempDir(), + }, func(url string) { + req, err := http.NewRequestWithContext(t.Context(), http.MethodGet, url, + http.NoBody) + if err != nil { + t.Fatalf("new request: %v", err) + } + + req.Header.Set(proxy.ClientHeader, placed) + + transport := &http.Transport{} + defer transport.CloseIdleConnections() + + res, err := (&http.Client{Transport: transport}).Do(req) + if err != nil { + t.Fatalf("request: %v", err) + } + + body, err := io.ReadAll(res.Body) + _ = res.Body.Close() + + if err != nil || string(body) != greeting || + res.Header.Get(proxy.DispositionHeader) != "" { + t.Errorf("answered %q (%v) with disposition %q, want the app's greeting", + body, err, res.Header.Get(proxy.DispositionHeader)) + } + }) +} diff --git a/internal/smallwebwaf/smallwebwaf.go b/internal/smallwebwaf/smallwebwaf.go index ab43736..3545ef4 100644 --- a/internal/smallwebwaf/smallwebwaf.go +++ b/internal/smallwebwaf/smallwebwaf.go @@ -1,7 +1,8 @@ // Package smallwebwaf runs the smallwebwaf process: it reads the settings, // the rule files, the lookup database and the state files, serves requests // until it is told to stop, and then stops in an orderly way, writing the -// state files. +// state files. It serves them in front of an app, or, as `smallwebwaf +// servewaf`, by answering rproxy's questions about them. package smallwebwaf import ( @@ -45,11 +46,15 @@ type Params struct { LookupEnv func(string) (string, bool) // Stdout receives the request log and the process's own messages. Stdout io.Writer + // ServeWAF makes it `smallwebwaf servewaf`, which answers rproxy's + // questions, rather than the proxy in front of an app. + ServeWAF bool } // Main runs smallwebwaf until SIGTERM or SIGINT, and returns the -// process's exit status. Run as `smallwebwaf healthcheck`, it is the -// container's health check instead. +// process's exit status. Run as `smallwebwaf servewaf`, it answers +// rproxy's questions rather than pass requests to an app. Run as +// `smallwebwaf healthcheck`, it is the container's health check instead. func Main(version string) int { if len(os.Args) > 1 && os.Args[1] == "healthcheck" { return HealthCheck(context.Background(), os.Args[2:], os.LookupEnv, os.Stderr) @@ -63,6 +68,7 @@ func Main(version string) int { Version: version, LookupEnv: os.LookupEnv, Stdout: os.Stdout, + ServeWAF: len(os.Args) > 1 && os.Args[1] == "servewaf", }) } @@ -117,7 +123,12 @@ func Run(ctx context.Context, params Params) int { return 1 } - server, err := newServer(cfg, stdout, processLog, now, ruleFiles, alertQueue) + newProxy := proxy.New + if params.ServeWAF { + newProxy = proxy.NewServeWAF + } + + server, err := newServer(newProxy, cfg, stdout, processLog, now, ruleFiles, alertQueue) if err != nil { processLog.Error("cannot use the lookup database", "error", err.Error()) @@ -151,10 +162,12 @@ func Run(ctx context.Context, params Params) int { return serve(ctx, server, listener, files, ruleFiles, alertQueue, processLog) } -// newServer returns the server smallwebwaf runs, with the metrics of the -// alerts, after reading the lookup database while SWWAF_LOOKUP_SOURCE is -// file. A lookup database that cannot be read is an error. +// newServer returns the server smallwebwaf runs, which newProxy makes, with +// the metrics of the alerts, after reading the lookup database while +// SWWAF_LOOKUP_SOURCE is file. A lookup database that cannot be read is an +// error. func newServer( + newProxy func(proxy.Params) *proxy.Server, cfg *config.Config, stdout io.Writer, processLog *slog.Logger, now func() time.Time, ruleFiles *rules.Files, alertQueue *alerts.Queue, ) (*proxy.Server, error) { @@ -174,7 +187,7 @@ func newServer( } } - server := proxy.New(proxy.Params{ + server := newProxy(proxy.Params{ Config: cfg, RequestLog: stdout, ProcessLog: processLog, diff --git a/internal/smallwebwaf/smallwebwaf_test.go b/internal/smallwebwaf/smallwebwaf_test.go index 1d30bf4..75dc067 100644 --- a/internal/smallwebwaf/smallwebwaf_test.go +++ b/internal/smallwebwaf/smallwebwaf_test.go @@ -113,6 +113,13 @@ func (o *output) text() string { // returns its exit status. SWWAF_LOOKUP_SOURCE is off unless env sets it, // so that no test sends GeoJS its clients' addresses. func run(ctx context.Context, env map[string]string, out *output) int { + return runAs(ctx, false, env, out) +} + +// runAs is run as `smallwebwaf servewaf` while serveWAF is true. +func runAs( + ctx context.Context, serveWAF bool, env map[string]string, out *output, +) int { return smallwebwaf.Run(ctx, smallwebwaf.Params{ Version: testVersion, LookupEnv: func(name string) (string, bool) { @@ -123,7 +130,8 @@ func run(ctx context.Context, env map[string]string, out *output) int { return value, ok }, - Stdout: out, + Stdout: out, + ServeWAF: serveWAF, }) } From 4a4874ae2fa03becc31f944630bf41837b9cce1e Mon Sep 17 00:00:00 2001 From: clawbot <35+clawbot@noreply.example.org> Date: Thu, 5 Nov 2026 05:48:50 +0100 Subject: [PATCH 5/9] servewaf always answers rproxy well within its time limit (closes #182) A question waits on GeoJS at most 800ms, the new default of SWWAF_LOOKUP_TIMEOUT. In enforce mode it does not wait at all when it is over the header limits, on a fetched blocklist with deny, on the CrowdSec decision list, or over a whole rate limit. Not waiting never lowers a client's limit. Just before the Core Rule Set, servewaf refuses more than 50 cookies or 100 header fields with 431. Shared default changed: SWWAF_LOOKUP_TIMEOUT also applies to smallwebwaf with no command. Judgement call: SWWAF_REPUTATION_TIMEOUT stays 2s, since no request waits on the DNS blocklists or AbuseIPDB. Model: opus-5-5 --- SPEC.md | 70 +++- docs/metrics.md | 3 +- docs/request-log.md | 3 +- docs/settings.md | 13 +- internal/config/config.go | 2 +- internal/config/config_test.go | 6 +- internal/lookup/lookup.go | 16 +- internal/lookup/lookup_test.go | 37 +- internal/proxy/answertime_test.go | 600 +++++++++++++++++++++++++++ internal/proxy/biased.go | 16 +- internal/proxy/biased_test.go | 2 +- internal/proxy/countries_test.go | 6 +- internal/proxy/lookup.go | 22 +- internal/proxy/lookup_test.go | 6 +- internal/proxy/metrics_test.go | 4 +- internal/proxy/proxy_test.go | 3 + internal/proxy/request.go | 44 +- internal/proxy/servewaf.go | 63 +++ internal/proxy/servewaf_test.go | 2 +- internal/ratelimit/ratelimit.go | 24 ++ internal/ratelimit/ratelimit_test.go | 44 ++ internal/requestlog/requestlog.go | 6 +- 22 files changed, 937 insertions(+), 55 deletions(-) create mode 100644 internal/proxy/answertime_test.go diff --git a/SPEC.md b/SPEC.md index 03b382a..6843758 100644 --- a/SPEC.md +++ b/SPEC.md @@ -1102,8 +1102,9 @@ metrics. ### The answer -`servewaf` runs steps 3 to 12 of "What happens to a request", in that order. -Every answer but the last below carries `X-Smallwebwaf-Disposition`. +`servewaf` runs steps 3 to 12 of "What happens to a request", in that order, +with the header limits just before step 12 (see "Answering in time"). Every +answer but the last below carries `X-Smallwebwaf-Disposition`. - `X-Smallwebwaf-Disposition: allow`, status `200`, no body. `rproxy` passes the request to the app. With `SWWAF_ADD_LOOKUP_HEADERS` the answer also carries @@ -1114,6 +1115,7 @@ Every answer but the last below carries `X-Smallwebwaf-Disposition`. status is one of: - `SWWAF_BAN_RESPONSE`, for a refusal; - `403`, for a `block` rule or the Core Rule Set; + - `431`, for a request over the header limits; - `404`, for a path under `/_smallwebwaf/` that passes the checks, so that the app never sees that prefix. - `X-Smallwebwaf-Disposition: close`, status `200`, no body. This takes the @@ -1131,6 +1133,70 @@ The Core Rule Set inspects the body a question carries only while does not inspect a JSON or XML body of 64 KiB or more, which may have been cut short. +### Answering in time + +`rproxy` stops waiting for `servewaf` after `SWRP_TIMEOUT_MS`, by default 2 +seconds, and `SWRP_DEFAULT_DISPOSITION` then decides. So at the defaults, +`servewaf` answers a request well within that time, whatever the request and +whatever its outside sources do. + +GeoJS is the only outside source a question can wait on. A question waits only +when step 5 says so, and for at most `SWWAF_LOOKUP_TIMEOUT`, by default 800 +milliseconds. A client without an answer by then counts as unknown. No question +waits on these, which run in the background: + +- the DNS blocklists and AbuseIPDB; +- fetching the blocklists, the CrowdSec decision list and the file + `SWWAF_ASN_LIMIT_PERCENT_URL` names. + +In `enforce` mode, a question does not wait for GeoJS when it is a request: + +- over the header limits; +- from a client on a fetched blocklist, while `SWWAF_BLOCKLIST_ACTION` is + `deny`; +- from a client on the CrowdSec decision list; +- that would take its client over a rate limit at 100% of it, since a percentage + can only lower a limit. + +Such a request still goes through the steps in their order. Its client's AS +number and country stay unknown until GeoJS has answered. So while +`SWWAF_EXCLUSIVELY_ALLOWED_COUNTRIES` is set, that list can refuse it first. +`SWWAF_UNKNOWN_LIMIT_PERCENT` does not apply to it until then, so not waiting +never lowers its limit. + +The static lists and bans come before the lookup, so they never wait. These can +still wait for GeoJS: + +- a country list, which needs the answer; +- a rate limit a percentage has lowered, which needs the answer; +- a DNS blocklist or AbuseIPDB, while `SWWAF_REPUTATION_ACTION` is `deny`; +- a trap path; +- a rule file; +- the Core Rule Set. + +The header limits are fixed, with no setting. `servewaf` refuses a request over +either one: + +- more than 50 cookies. Each part of its `Cookie` headers between semicolons + counts as one, unless it is empty or only spaces and tabs. +- more than 100 header fields. A header sent twice counts twice. `Host` and + `Transfer-Encoding` do not count. `X-Request-ID` and `X-Forwarded-Proto` do. + +Real clients never reach these limits. The answer is `deny` with `431`, and the +log line's action is `too_large`. The refusal makes no ban and is no offence. +The rate limits count the request as usual, so a client that keeps sending such +requests breaks a limit. + +The limits apply on every path and in every `SWWAF_WAF_MODE`. A client in +`SWWAF_ALLOW_NETS` skips them, as it skips the Core Rule Set. In `observe` mode +such a request is answered `allow` with `would_action`, and the Core Rule Set +does not read it. + +The Core Rule Set runs each of its rules on each cookie, so thousands of small +cookies would keep it busy for over a second. Within the limits, one long cookie +or query string can still keep it busy for a fraction of a second. So enough +such requests at once can overload `servewaf` past `SWRP_TIMEOUT_MS`. + ### The report Once the visitor's answer has ended, `rproxy` sends `POST /_smallwebwaf/report` diff --git a/docs/metrics.md b/docs/metrics.md index 3bb111a..e2bff8c 100644 --- a/docs/metrics.md +++ b/docs/metrics.md @@ -17,7 +17,8 @@ label as `exported_instance` unless the scrape sets `honor_labels: true`. - `smallwebwaf_rate_limit_hits_total`, by `window` and by `kind` (`requests`, `bytes` or `refusals`); -- `smallwebwaf_size_and_time_limit_hits_total`, by `limit`, the setting passed; +- `smallwebwaf_size_and_time_limit_hits_total`, by `limit`, the setting passed. + `servewaf`'s header limits have no setting, so it leaves them out; - `smallwebwaf_offences_total`, by `kind`; - `smallwebwaf_bans_made_total`, by `cause`; - `smallwebwaf_active_bans` and `smallwebwaf_permanent_bans`. diff --git a/docs/request-log.md b/docs/request-log.md index 1d584db..1689319 100644 --- a/docs/request-log.md +++ b/docs/request-log.md @@ -67,7 +67,8 @@ Every line has: CrowdSec decision, each of which bans the client; - `country_denied`, `rate_limited`, `rule_blocked` and `waf_blocked`: refused by that check; - - `too_large` and `timed_out`: a size or time limit was passed; + - `too_large`: a size limit was passed, or a header limit in `servewaf`; + - `timed_out`: a time limit was passed; - `upstream_error`: the app could not be reached, or its answer broke off; - `admin`: answered at one of `smallwebwaf`'s own endpoints. - `would_action`: in `observe` mode, what `enforce` mode would have done with a diff --git a/docs/settings.md b/docs/settings.md index a59e4df..9c39e58 100644 --- a/docs/settings.md +++ b/docs/settings.md @@ -157,7 +157,7 @@ that take `off`, the `_FILE` form, and what each group does and why. - an anomaly threshold per AS number. - `SWWAF_LOOKUP_DB_PATH` (default empty): the IPinfo Lite database file, in its `.mmdb` form, for `SWWAF_LOOKUP_SOURCE=file`. -- `SWWAF_LOOKUP_TIMEOUT` (default `1s`): how long a request waits for its +- `SWWAF_LOOKUP_TIMEOUT` (default `800ms`): how long a request waits for its client's first answer from GeoJS, and how long a request to GeoJS may take. A duration; never `off`. - `SWWAF_ADD_LOOKUP_HEADERS` (default `false`): `true` passes the app the @@ -340,7 +340,10 @@ A few limits are fixed rather than settings: - at most 1,000 DNS blocklist queries are under way at once; - at most 20,000 anomaly counters are kept; - each alert destination's queue holds at most 1,000 alerts; -- `servewaf` holds at most 10,000 answered requests awaiting their reports. +- `servewaf` holds at most 10,000 answered requests awaiting their reports; +- `servewaf` refuses a request with more than 50 cookies or more than 100 header + fields (see "Answering in time" under "`servewaf`" in + [`SPEC.md`](../SPEC.md)). ## `servewaf` @@ -358,3 +361,9 @@ at start still shows them. request to an app. `SWWAF_WAF_BODY_LIMIT` has an effect up to 64 KiB less one byte, whatever its value. A question carries no more of a body than 64 KiB (see "`servewaf`" in [`SPEC.md`](../SPEC.md)). + +`SWWAF_LOOKUP_TIMEOUT` is the longest a question waits on an outside source. +Keep it well under `rproxy`'s `SWRP_TIMEOUT_MS`, as the default is. No question +waits for `SWWAF_REPUTATION_TIMEOUT`, since the DNS blocklists and AbuseIPDB are +asked in the background (see "Answering in time" under "`servewaf`" in +[`SPEC.md`](../SPEC.md)). diff --git a/internal/config/config.go b/internal/config/config.go index 5f01c82..914e3de 100644 --- a/internal/config/config.go +++ b/internal/config/config.go @@ -516,7 +516,7 @@ func FromEnvironment(lookupEnv func(string) (string, bool)) (*Config, error) { BytesCount: env.bytesCount("SWWAF_BYTES_COUNT", "both"), LookupSource: env.lookupSource("SWWAF_LOOKUP_SOURCE", "geojs"), LookupDBPath: env.value("SWWAF_LOOKUP_DB_PATH", ""), - LookupTimeout: env.durationNotOff("SWWAF_LOOKUP_TIMEOUT", "1s"), + LookupTimeout: env.durationNotOff("SWWAF_LOOKUP_TIMEOUT", "800ms"), AddLookupHeaders: env.boolean("SWWAF_ADD_LOOKUP_HEADERS", "false"), DeniedCountries: env.countries("SWWAF_DENIED_COUNTRIES", ""), ExclusivelyAllowedCountries: env.countries( diff --git a/internal/config/config_test.go b/internal/config/config_test.go index 032a3e0..67c3f33 100644 --- a/internal/config/config_test.go +++ b/internal/config/config_test.go @@ -281,7 +281,7 @@ func TestDefaults(t *testing.T) { RulesEnabled: true, }) wantLookupSettings(t, cfg, config.Config{ - LookupSource: defaultLookupSource, LookupTimeout: time.Second, + LookupSource: defaultLookupSource, LookupTimeout: 800 * time.Millisecond, }) wantByteLimitSettings(t, cfg, config.Config{ BytesLimitPerMinute: 4 << 30, BytesLimitPerHour: 20 << 30, @@ -1165,7 +1165,7 @@ func TestLookupSettingsAsSet(t *testing.T) { cfg = fromEnvironment(t, environment{lookupSource: off}) wantLookupSettings(t, cfg, config.Config{ - LookupSource: off, LookupTimeout: time.Second, + LookupSource: off, LookupTimeout: 800 * time.Millisecond, }) } @@ -2385,7 +2385,7 @@ func TestLogsEachSettingWithItsValue(t *testing.T) { bytesCount: "both", lookupSource: defaultLookupSource, lookupDBPath: "", - lookupTimeout: "1s", + lookupTimeout: "800ms", addLookupHeaders: "false", deniedCountries: "", allowedCountries: "", diff --git a/internal/lookup/lookup.go b/internal/lookup/lookup.go index cdd86cb..c483529 100644 --- a/internal/lookup/lookup.go +++ b/internal/lookup/lookup.go @@ -179,7 +179,7 @@ func New(params Params) *GeoJS { // own address for IPv4, and an address in the same place for an IPv6 // netblock. func (g *GeoJS) LookUp(ctx context.Context, client netip.Prefix) Answer { - answer, asked := g.answerOrWait(ctx, client) + answer, asked := g.answerOrWait(ctx, client, g.wait) if asked == nil { return answer } @@ -209,6 +209,14 @@ func (g *GeoJS) LookUp(ctx context.Context, client netip.Prefix) Answer { return answer } +// LookUpWithoutWaiting is LookUp for a request that does not need the answer, +// whatever Wait says: it returns the zero Answer at once when none is kept. +func (g *GeoJS) LookUpWithoutWaiting(ctx context.Context, client netip.Prefix) Answer { + answer, _ := g.answerOrWait(ctx, client, false) + + return answer +} + // Kept returns client's answer, if one is kept, without asking GeoJS. func (g *GeoJS) Kept(client netip.Prefix) (Answer, bool) { g.mu.Lock() @@ -263,9 +271,9 @@ func (g *GeoJS) Load(answers []Answer) { // answerOrWait returns client's kept answer if it has one. Otherwise it // puts the client among those waiting if there is room, has GeoJS asked // about them if it can be, and returns what to wait on for the answer, or -// nil when there is nothing to wait for. +// nil when there is nothing to wait for or mustWait is false. func (g *GeoJS) answerOrWait( - ctx context.Context, client netip.Prefix, + ctx context.Context, client netip.Prefix, mustWait bool, ) (Answer, <-chan struct{}) { g.mu.Lock() defer g.mu.Unlock() @@ -283,7 +291,7 @@ func (g *GeoJS) answerOrWait( g.ask(ctx) - if !g.wait { + if !mustWait { return Answer{}, nil // the answer is not needed before the request goes on } diff --git a/internal/lookup/lookup_test.go b/internal/lookup/lookup_test.go index e62d5a1..32d3497 100644 --- a/internal/lookup/lookup_test.go +++ b/internal/lookup/lookup_test.go @@ -72,6 +72,39 @@ func TestKeptAnswerIsUsedFor7DaysThenAskedAgain(t *testing.T) { }) } +func TestLookUpWithoutWaitingAsksGeoJSAndGoesOnAtOnce(t *testing.T) { + t.Parallel() + + synctest.Test(t, func(t *testing.T) { + geojs, clock, g := start() + client := netip.MustParsePrefix("203.0.113.9/32") + + // Wait is set, and GeoJS does not answer. + geojs.set(hanging) + + began := time.Now() + answer := g.LookUpWithoutWaiting(t.Context(), client) + + if waited := time.Since(began); waited != 0 || answer != (lookup.Answer{}) { + t.Errorf("got %+v after %s, want no answer at once", answer, waited) + } + + waitForRequests(t, geojs, 1) + wantAsked(t, geojs, 0, "203.0.113.9") + + // That request is abandoned, and has failed once this returns. GeoJS + // answers when it is asked again, and the answer is kept. + time.Sleep(timeout) + synctest.Wait() + geojs.set(answering) + waitForCountry(t, g, clock, client, germany) + + if got := g.LookUpWithoutWaiting(t.Context(), client).Country; got != germany { + t.Errorf("the kept answer gives the country %q, want %q", got, germany) + } + }) +} + func TestNewClientWaitsAtMostOneSecondThenCountsAsNotFound(t *testing.T) { t.Parallel() @@ -123,8 +156,8 @@ func TestRequestWaitsAsLongAsTheTimeoutSaysAndGeoJSIsAbandonedAfterIt(t *testing t.Parallel() synctest.Test(t, func(t *testing.T) { - // A timeout longer than the default second, and a GeoJS that does - // not answer. + // A timeout longer than the default, and a GeoJS that does not + // answer. const longerTimeout = 3 * time.Second m := metrics.New(1, "app") diff --git a/internal/proxy/answertime_test.go b/internal/proxy/answertime_test.go new file mode 100644 index 0000000..8520eae --- /dev/null +++ b/internal/proxy/answertime_test.go @@ -0,0 +1,600 @@ +package proxy_test + +import ( + "fmt" + "maps" + "net" + "net/http" + "net/http/httptest" + "net/netip" + "slices" + "strconv" + "strings" + "testing" + "testing/synctest" + "time" + + "sneak.berlin/go/smallwebwaf/internal/bans" + "sneak.berlin/go/smallwebwaf/internal/lookup" + "sneak.berlin/go/smallwebwaf/internal/proxy" + "sneak.berlin/go/smallwebwaf/internal/ratelimit" + "sneak.berlin/go/smallwebwaf/internal/reputation" + "sneak.berlin/go/smallwebwaf/internal/requestlog" +) + +const ( + // answerTimeLimit is the default of rproxy's SWRP_TIMEOUT_MS, the time + // servewaf has to answer. + answerTimeLimit = 2 * time.Second + // defaultLookupWait is the default of SWWAF_LOOKUP_TIMEOUT. + defaultLookupWait = 800 * time.Millisecond + // The header limits SPEC.md gives. + cookieLimit = 50 + headerFieldLimit = 100 + // cookieHeader is the header the cookie limit counts. + cookieHeader = "Cookie" + // tooLarge is how servewaf refuses a request over the header limits. + tooLarge = http.StatusRequestHeaderFieldsTooLarge + // silentAbuseIPDBURL is where AbuseIPDB never answers: at the stand-in + // that unansweredGeoJSURL names too. + silentAbuseIPDBURL = "unanswered://abuseipdb/api/v2/check" + // wholeLimit is the percentage of a rate limit that lowers it not at all. + wholeLimit = 100 + // clientsDecision is the CrowdSec decision list holding client alone. + clientsDecision = `[{"duration": "4h0m0s", "scenario": "crowdsecurity/ssh-bf", ` + + `"scope": "Ip", "type": "ban", "value": "` + client + `"}]` +) + +// cookies returns a Cookie header's value with count cookies. +func cookies(count int) string { + parts := make([]string, 0, count) + for i := range count { + parts = append(parts, "c"+strconv.Itoa(i)+"=1") + } + + return strings.Join(parts, "; ") +} + +// fillHeaders adds headers to a question until the checks see total header +// fields: all but ClientHeader and PeerHeader, which servewaf removes. Go's +// client sends nothing else but Host, which the limit does not count, once +// User-Agent is set. +func fillHeaders(req *http.Request, total int) { + req.Header.Set("User-Agent", "test") + + for i := len(req.Header) - 2; i < total; i++ { + req.Header.Set("X-Filler-"+strconv.Itoa(i), "1") + } +} + +func TestSixteenThousandCookiesAreRefusedWellWithinTheTimeLimit(t *testing.T) { + t.Parallel() + + s := startServeWAF(t, "", map[string]string{wafMode: block}) + req := s.question(siteA, client, "/") + req.Header.Set(cookieHeader, strings.Repeat("a;", 16000)) + + began := time.Now() + got := do(t, req) + + if took := time.Since(began); took >= answerTimeLimit/2 { + t.Errorf("answered after %s, want well within %s", took, answerTimeLimit) + } + + wantDisposition(t, got, tooLarge, proxy.DispositionDeny) + + line := s.out.requestLine(t) + wantLine(t, line, tooLarge, requestlog.ActionTooLarge) + + if line.DurationWAF != nil { + t.Errorf("the Core Rule Set read the request, for %f ms", *line.DurationWAF) + } +} + +func TestEachHeaderLimitRefusesARequestOverIt(t *testing.T) { + t.Parallel() + + repeated := make([]string, headerFieldLimit) + for i := range repeated { + repeated[i] = "1" + } + + for _, tc := range []struct { + name string + make func(*http.Request) + // refused is true for a request over a limit, and false for one at it. + refused bool + }{ + {"cookies at the limit", func(req *http.Request) { + req.Header.Set(cookieHeader, cookies(cookieLimit)) + }, false}, + {"cookies over the limit", func(req *http.Request) { + req.Header.Set(cookieHeader, cookies(cookieLimit+1)) + }, true}, + {"parts empty or only spaces and tabs are no cookies", func(req *http.Request) { + req.Header.Set(cookieHeader, cookies(cookieLimit)+"; \t;;") + }, false}, + {"parts of no-break spaces count", func(req *http.Request) { + req.Header.Set(cookieHeader, strings.Repeat(" ;", cookieLimit+1)) + }, true}, + {"cookies over the limit across two headers", func(req *http.Request) { + req.Header[cookieHeader] = []string{cookies(cookieLimit), "last=1"} + }, true}, + {"cookies without a value count", func(req *http.Request) { + req.Header.Set(cookieHeader, strings.Repeat("a;", cookieLimit+1)) + }, true}, + {"header fields at the limit", func(req *http.Request) { + fillHeaders(req, headerFieldLimit) + }, false}, + {"header fields over the limit", func(req *http.Request) { + fillHeaders(req, headerFieldLimit+1) + }, true}, + {"a header sent many times counts each time", func(req *http.Request) { + req.Header["X-Filler"] = repeated + }, true}, + } { + t.Run(tc.name, func(t *testing.T) { + t.Parallel() + + s := startServeWAF(t, "", map[string]string{wafMode: block}) + req := s.question(siteA, client, "/") + tc.make(req) + + got := do(t, req) + line := s.out.requestLine(t) + + if tc.refused { + wantDisposition(t, got, tooLarge, proxy.DispositionDeny) + wantLine(t, line, tooLarge, requestlog.ActionTooLarge) + + return + } + + wantDisposition(t, got, http.StatusOK, proxy.DispositionAllow) + wantLine(t, line, http.StatusOK, requestlog.ActionForward) + + if line.DurationWAF == nil { + t.Error("the Core Rule Set did not read a request within the limits") + } + }) + } +} + +func TestHeaderLimitsHoldWhateverTheCoreRuleSetSettingsSay(t *testing.T) { + t.Parallel() + + for _, env := range []map[string]string{ + {wafMode: off}, + {wafMode: "detect"}, + {wafMode: block, "SWWAF_WAF_EXEMPT_PATHS": "/"}, + } { + t.Run(fmt.Sprint(env), func(t *testing.T) { + t.Parallel() + + s := startServeWAF(t, "", env) + req := s.question(siteA, client, "/") + req.Header.Set(cookieHeader, cookies(cookieLimit+1)) + + wantDisposition(t, do(t, req), tooLarge, proxy.DispositionDeny) + }) + } +} + +func TestClientInTheAllowNetsSkipsTheHeaderLimits(t *testing.T) { + t.Parallel() + + s := startServeWAF(t, "", map[string]string{wafMode: block, allowNets: client}) + req := s.question(siteA, client, "/") + req.Header.Set(cookieHeader, cookies(cookieLimit+1)) + + wantDisposition(t, do(t, req), http.StatusOK, proxy.DispositionAllow) +} + +func TestObserveModeAllowsARequestOverTheHeaderLimitsUnread(t *testing.T) { + t.Parallel() + + s := startServeWAF(t, "", map[string]string{mode: observe, wafMode: block}) + req := s.question(siteA, client, "/") + req.Header.Set(cookieHeader, strings.Repeat("a;", 16000)) + + began := time.Now() + got := do(t, req) + + if took := time.Since(began); took >= answerTimeLimit/2 { + t.Errorf("answered after %s, want well within %s", took, answerTimeLimit) + } + + wantDisposition(t, got, http.StatusOK, proxy.DispositionAllow) + + line := s.out.requestLine(t) + if line.WouldAction != requestlog.ActionTooLarge || line.DurationWAF != nil { + t.Errorf("would_action %q and duration_waf %v, want %q and none", + line.WouldAction, line.DurationWAF, requestlog.ActionTooLarge) + } +} + +func TestRequestsOverTheHeaderLimitsCountTowardTheRateLimits(t *testing.T) { + t.Parallel() + + s := startServeWAF(t, "", map[string]string{rateLimitPerMinute: "1"}) + ask := func() answer { + req := s.question(siteA, client, "/") + req.Header.Set(cookieHeader, cookies(cookieLimit+1)) + + return do(t, req) + } + + wantDisposition(t, ask(), tooLarge, proxy.DispositionDeny) + wantDisposition(t, ask(), http.StatusForbidden, proxy.DispositionDeny) + + lines := s.out.requestLines(t, 2) + wantLine(t, lines[1], http.StatusForbidden, requestlog.ActionRateLimited) +} + +func TestTheHeaderLimitsComeAfterTheTrapPathsAndTheRuleFiles(t *testing.T) { + t.Parallel() + + for _, tc := range []struct { + name, path, action string + }{ + {"a trap path", "/wp-login.php", requestlog.ActionBanned}, + {"a block rule", "/blocked", requestlog.ActionRuleBlocked}, + } { + t.Run(tc.name, func(t *testing.T) { + t.Parallel() + + s := startServeWAF(t, "", map[string]string{ + trapPaths: trapPathList, rulesDir: writeRules(t, testRules), + }) + req := s.question(siteA, client, tc.path) + req.Header.Set(cookieHeader, cookies(cookieLimit+1)) + + wantDisposition(t, do(t, req), http.StatusForbidden, proxy.DispositionDeny) + wantLine(t, s.out.requestLine(t), http.StatusForbidden, tc.action) + }) + } +} + +func TestNotWaitingForGeoJSNeverLowersAClientsLimit(t *testing.T) { + t.Parallel() + + geojsURL, _ := startGeoJS(t) + s := startServeWAF(t, geojsURL, map[string]string{ + unknownLimitPercent: "0", lookupTimeout: "1h", + }) + req := s.question(siteA, fromDE, "/") + req.Header.Set(cookieHeader, cookies(cookieLimit+1)) + + wantDisposition(t, do(t, req), tooLarge, proxy.DispositionDeny) + + line := s.out.requestLine(t) + wantLine(t, line, tooLarge, requestlog.ActionTooLarge) + + if line.LimitPercent != nil { + t.Errorf("limit percentage %d, want none", *line.LimitPercent) + } + + // The client is under no ban, and GeoJS places it in a country. + wantDisposition(t, s.ask(fromDE, "/"), http.StatusOK, proxy.DispositionAllow) +} + +func TestAClientGeoJSPlacedInNoCountryCountsAsUnknownWithoutWaiting(t *testing.T) { + t.Parallel() + + geojsURL, _ := startGeoJS(t) + s := startServeWAF(t, geojsURL, map[string]string{unknownLimitPercent: "0"}) + s.server.GeoJS.Load([]lookup.Answer{{ + Client: netip.MustParsePrefix(client + "/32"), ASN: "AS64512", + Answered: s.clock.Now(), Used: s.clock.Now(), + }}) + + req := s.question(siteA, client, "/") + req.Header.Set(cookieHeader, cookies(cookieLimit+1)) + + wantDisposition(t, do(t, req), http.StatusForbidden, proxy.DispositionDeny) + wantLine(t, s.out.requestLine(t), http.StatusForbidden, requestlog.ActionRateLimited) +} + +func TestObserveModeWaitsForGeoJSForARequestOverTheHeaderLimits(t *testing.T) { + t.Parallel() + + geojsURL, _ := startGeoJS(t) + s := startServeWAF(t, geojsURL, map[string]string{ + mode: observe, addLookupHeaders: on, lookupTimeout: "1h", + }) + req := s.question(siteA, fromDE, "/") + req.Header.Set(cookieHeader, cookies(cookieLimit+1)) + + got := do(t, req) + wantDisposition(t, got, http.StatusOK, proxy.DispositionAllow) + + asn, country := got.header.Get("X-Client-Asn"), got.header.Get("X-Client-Country") + if asn != asnDE || country != "DE" { + t.Errorf("answered with the AS number %q and the country %q, want %q and DE", + asn, country, asnDE) + } +} + +func TestWithoutACommandTheHeaderLimitsDoNotApply(t *testing.T) { + t.Parallel() + + app := startApp(t, func(http.ResponseWriter, *http.Request) {}) + addr, out := startProxy(t, app.URL, nil) + + req := newRequest(t, http.MethodGet, addr, "/", http.NoBody) + req.Header.Set(cookieHeader, cookies(cookieLimit+1)) + fillHeaders(req, headerFieldLimit+1) + + wantStatus(t, do(t, req), http.StatusOK) + wantLine(t, out.requestLine(t), http.StatusOK, requestlog.ActionForward) +} + +// newServeWAFInBubble makes a servewaf for a synctest bubble: GeoJS never +// answers, a setting needs its answer, and everything is at its default but +// for env. No request to it may use the network. +func newServeWAFInBubble(t *testing.T, env map[string]string) (*proxy.Server, *output) { + t.Helper() + + settings := map[string]string{deniedCountries: "KP"} + maps.Copy(settings, env) + + server, out, _ := newServerWith(t, proxy.NewServeWAF, "http://app.invalid", + unansweredGeoJSURL, time.Now, settings) + + return server, out +} + +// askDirectly hands a question about a request from client straight to server, +// as change leaves it, and returns the answer's status and how long it took. +func askDirectly( + t *testing.T, server *proxy.Server, change func(*http.Request), +) (int, time.Duration) { + t.Helper() + + req := httptest.NewRequestWithContext(t.Context(), http.MethodGet, "/", http.NoBody) + req.RemoteAddr = net.JoinHostPort(rproxyPeer, "1234") + req.Header.Set(proxy.ClientHeader, client) + req.Header.Set(requestIDHeader, "the-id") + change(req) + + recorder := httptest.NewRecorder() + began := time.Now() + + server.Handler.ServeHTTP(recorder, req) + + return recorder.Code, time.Since(began) +} + +// banClient bans client at now. +func banClient(_ *testing.T, server *proxy.Server, now time.Time) { + server.Ledger.BanForLimit(netip.MustParsePrefix(client+"/32"), now, bans.Notes{}) +} + +// countClient counts one request from client at now for the rate limits. +func countClient(_ *testing.T, server *proxy.Server, now time.Time) { + server.Limiter.Count(netip.MustParsePrefix(client+"/32"), now, wholeLimit) +} + +// listClient puts client on the blocklist at dropURL. +func listClient(t *testing.T, server *proxy.Server, now time.Time) { + t.Helper() + + err := server.Lists.Load([]reputation.List{{ + URL: dropURL, Tried: now, Fetched: now, Lines: []string{client}, + }}) + if err != nil { + t.Fatalf("load the blocklist: %v", err) + } +} + +// decideOnClient puts client on the CrowdSec decision list. +func decideOnClient(t *testing.T, server *proxy.Server, now time.Time) { + t.Helper() + loadDecisions(t, server, now, clientsDecision) +} + +// leaveAsItIs prepares nothing. +func leaveAsItIs(*testing.T, *proxy.Server, time.Time) {} + +func TestARefusedRequestDoesNotWaitForASlowGeoJS(t *testing.T) { + t.Parallel() + + const forbidden = http.StatusForbidden + + for _, tc := range []struct { + name string + env map[string]string + // prepare gets the server ready, at now, before the question. + prepare func(t *testing.T, server *proxy.Server, now time.Time) + // cookie is the request's Cookie header. + cookie string + status int + action string + }{ + { + name: "SWWAF_DENY_NETS", env: map[string]string{denyNets: client}, + prepare: leaveAsItIs, status: forbidden, action: requestlog.ActionDenied, + }, + { + name: "a ban", + prepare: banClient, status: forbidden, action: requestlog.ActionBanned, + }, + { + name: "the rate limits", env: map[string]string{rateLimitPerMinute: "1"}, + prepare: countClient, status: forbidden, action: requestlog.ActionRateLimited, + }, + { + name: "the rate limits, with a biased threshold", + env: map[string]string{rateLimitPerMinute: "1", unknownLimitPercent: "50"}, + prepare: countClient, status: forbidden, action: requestlog.ActionRateLimited, + }, + { + name: "a blocklist", env: map[string]string{blocklistURLs: dropURL}, + prepare: listClient, status: forbidden, action: requestlog.ActionDenied, + }, + { + name: "the CrowdSec decision list", + env: map[string]string{crowdSecURL: lapi, crowdSecKey: bouncerKey}, + prepare: decideOnClient, status: forbidden, action: requestlog.ActionBanned, + }, + { + name: "the header limits", cookie: cookies(cookieLimit + 1), + prepare: leaveAsItIs, status: tooLarge, action: requestlog.ActionTooLarge, + }, + } { + t.Run(tc.name, func(t *testing.T) { + t.Parallel() + + // In a synctest bubble, as TestRequestWaitsAsLongAsTheLookupTimeoutSays + // says, so that a request that does not wait takes no time at all. + synctest.Test(t, func(t *testing.T) { + server, out := newServeWAFInBubble(t, tc.env) + tc.prepare(t, server, time.Now()) + + status, took := askDirectly(t, server, func(req *http.Request) { + req.Header.Set(cookieHeader, tc.cookie) + }) + + if status != tc.status || took != 0 { + t.Errorf("answered %d after %s, want %d at once; output:\n%s", + status, took, tc.status, out.text()) + } + + wantLine(t, out.requestLine(t), tc.status, tc.action) + + // GeoJS's request is abandoned before the bubble ends. + time.Sleep(defaultLookupWait) + }) + }) + } +} + +func TestAQuestionWaitsForASlowGeoJSOnlyAsLongAsTheDefaultSays(t *testing.T) { + t.Parallel() + + for _, tc := range []struct { + name string + env map[string]string + // prepare gets the server ready, at now, before the question. + prepare func(t *testing.T, server *proxy.Server, now time.Time) + status int + action string + }{ + {"nothing refuses it", nil, leaveAsItIs, http.StatusOK, requestlog.ActionForward}, + { + "a country list needs the answer", map[string]string{allowedCountries: "DE"}, + leaveAsItIs, http.StatusForbidden, requestlog.ActionCountryDenied, + }, + { + "a blocklist that only logs", + map[string]string{blocklistURLs: dropURL, blocklistAction: actionLog}, + listClient, http.StatusOK, requestlog.ActionForward, + }, + { + "a blocklist that only limits", + map[string]string{blocklistURLs: dropURL, blocklistAction: limitQuarter}, + listClient, http.StatusOK, requestlog.ActionForward, + }, + { + "over a rate limit on an exempt path", + map[string]string{rateLimitPerMinute: "1", rateLimitExemptPaths: "/"}, + countClient, http.StatusOK, requestlog.ActionForward, + }, + } { + t.Run(tc.name, func(t *testing.T) { + t.Parallel() + + synctest.Test(t, func(t *testing.T) { + server, out := newServeWAFInBubble(t, tc.env) + tc.prepare(t, server, time.Now()) + + status, took := askDirectly(t, server, func(*http.Request) {}) + + if status != tc.status || took != defaultLookupWait { + t.Errorf("answered %d after %s, want %d after %s; output:\n%s", + status, took, tc.status, defaultLookupWait, out.text()) + } + + wantLine(t, out.requestLine(t), tc.status, tc.action) + }) + }) + } +} + +func TestWithoutACommandARefusedRequestStillWaitsForGeoJS(t *testing.T) { + t.Parallel() + + synctest.Test(t, func(t *testing.T) { + server, out, _ := newProxy(t, "http://app.invalid", unansweredGeoJSURL, + time.Now, map[string]string{deniedCountries: "KP", blocklistURLs: dropURL}) + listClient(t, server, time.Now()) + + status, took := askDirectly(t, server, func(req *http.Request) { + req.RemoteAddr = net.JoinHostPort(client, "1234") + }) + + if status != http.StatusForbidden || took != defaultLookupWait { + t.Errorf("answered %d after %s, want %d after %s; output:\n%s", + status, took, http.StatusForbidden, defaultLookupWait, out.text()) + } + + wantLine(t, out.requestLine(t), http.StatusForbidden, requestlog.ActionDenied) + }) +} + +func TestServeWAFAnswersWithinTheTimeLimitWhileEveryOutsideSourceHangs(t *testing.T) { + t.Parallel() + + held, askedGeoJS, _ := startHeldGeoJS(t) + + // A resolver that takes every query and answers none. + resolver, err := (&net.ListenConfig{}).ListenPacket(t.Context(), "udp", localhost+":0") + if err != nil { + t.Fatalf("listen: %v", err) + } + + t.Cleanup(func() { _ = resolver.Close() }) + + server, out, _ := newServerWith(t, func(params proxy.Params) *proxy.Server { + params.AbuseIPDBURL = silentAbuseIPDBURL + + return proxy.NewServeWAF(params) + }, "http://app.invalid", held, time.Now, map[string]string{ + deniedCountries: "KP", wafMode: block, + dnsblZones: dnsblZone, dnsblResolver: resolver.LocalAddr().String(), + abuseIPDBKey: accountKey, + blocklistURLs: held + "/drop.txt", asnLimitPercentURL: held + "/asn.txt", + crowdSecURL: held, crowdSecKey: bouncerKey, + }) + + // AbuseIPDB is asked only about a client with an offence. + group := netip.MustParsePrefix(client + "/32") + server.Limiter.AddToHistory(group, time.Now(), ratelimit.Request{Attack: true}) + + go server.Lists.Run(t.Context()) + + status, took := askDirectly(t, server, func(req *http.Request) { + req.Header.Set("User-Agent", "test") + req.Header.Set("Accept", "*/*") + req.Host = siteA + }) + + if status != http.StatusOK || took >= answerTimeLimit { + t.Errorf("answered %d after %s, want %d within %s; output:\n%s", + status, took, http.StatusOK, answerTimeLimit, out.text()) + } + + // Each outside source was asked, and none has answered. The stand-in for + // GeoJS is the lists' too, which name no address. + if !slices.Contains(askedGeoJS(), client) { + t.Errorf("GeoJS was asked about %q, not %s", askedGeoJS(), client) + } + + if queries := server.DNSBL.Queries(dnsblZone); queries != 1 { + t.Errorf("%d queries to the DNS blocklist, want 1", queries) + } + + if checked := server.AbuseIPDB.Checked(); checked != 1 { + t.Errorf("%d checks with AbuseIPDB, want 1", checked) + } +} diff --git a/internal/proxy/biased.go b/internal/proxy/biased.go index 5370ff0..294cae9 100644 --- a/internal/proxy/biased.go +++ b/internal/proxy/biased.go @@ -34,19 +34,19 @@ func biasedThresholdsSet(cfg *config.Config) bool { // percentage SWWAF_ASN_LIMIT_PERCENT gives its AS number, the one the file // SWWAF_ASN_LIMIT_PERCENT_URL names gives it, the one // SWWAF_COUNTRY_LIMIT_PERCENT gives its country, for a client without a -// country, SWWAF_UNKNOWN_LIMIT_PERCENT, for a client a blocklist lists, -// the percentage of SWWAF_BLOCKLIST_ACTION while it is limit, and for a -// client a DNSBL zone's verdict lists, or whose AbuseIPDB score is a hit, -// the percentage of SWWAF_REPUTATION_ACTION while it is limit. For the -// byte limits, SWWAF_ASN_BYTES_PERCENT and SWWAF_COUNTRY_BYTES_PERCENT -// take the place of the first three for an AS number or a country they -// list. +// country, unless its request skipped the wait for GeoJS, +// SWWAF_UNKNOWN_LIMIT_PERCENT, for a client a blocklist lists, the +// percentage of SWWAF_BLOCKLIST_ACTION while it is limit, and for a client +// a DNSBL zone's verdict lists, or whose AbuseIPDB score is a hit, the +// percentage of SWWAF_REPUTATION_ACTION while it is limit. For the byte +// limits, SWWAF_ASN_BYTES_PERCENT and SWWAF_COUNTRY_BYTES_PERCENT take the +// place of the first three for an AS number or a country they list. func (rq *request) limitPercentages() (percentage, percentage) { cfg := rq.h.config asn, country := rq.line.ASN, rq.line.Country unknown := percentage{percent: whole} - if country == "" { + if country == "" && !rq.skippedLookupWait { unknown = percentage{cfg.UnknownLimitPercent, "SWWAF_UNKNOWN_LIMIT_PERCENT"} } diff --git a/internal/proxy/biased_test.go b/internal/proxy/biased_test.go index 03d4c33..cf1e5ab 100644 --- a/internal/proxy/biased_test.go +++ b/internal/proxy/biased_test.go @@ -284,7 +284,7 @@ func TestClientWithoutAnAnswerInTimeHasTheUnknownLimitPercent(t *testing.T) { server, out, _ := newProxy(t, "http://app.invalid", unansweredGeoJSURL, time.Now, map[string]string{unknownLimitPercent: "0"}) - // Once the second the request waits for its answer is up, the client + // Once the time the request waits for its answer is up, the client // counts as without a country, and its zero allowance refuses the // request before it reaches the app. serveFromDE(t, server, http.MethodGet, http.NoBody) diff --git a/internal/proxy/countries_test.go b/internal/proxy/countries_test.go index a5e558f..0725070 100644 --- a/internal/proxy/countries_test.go +++ b/internal/proxy/countries_test.go @@ -284,9 +284,9 @@ func TestExclusiveListRefusesAPrivateAddressUnlessAllowed(t *testing.T) { // each in an AS of its own, and no other address. It returns its URL, and // what returns the addresses it has been asked about. A test that needs // the stand-in to be asked or to answer sets SWWAF_LOOKUP_TIMEOUT to an -// hour, whether or not a request waits for the answer: on the default -// second, a hold-up of the test process can abandon the request to the -// stand-in, and leave the client unknown. +// hour, whether or not a request waits for the answer: on the default, a +// hold-up of the test process can abandon the request to the stand-in, and +// leave the client unknown. func startGeoJS(t *testing.T) (string, func() []string) { t.Helper() diff --git a/internal/proxy/lookup.go b/internal/proxy/lookup.go index 33190e4..1bf1fac 100644 --- a/internal/proxy/lookup.go +++ b/internal/proxy/lookup.go @@ -4,6 +4,7 @@ import ( "context" "net/http" "net/netip" + "time" "sneak.berlin/go/smallwebwaf/internal/lookup" ) @@ -23,17 +24,24 @@ const ( // SWWAF_LOOKUP_SOURCE is off or the client is on a private, loopback or // link-local address, which no lookup can place. The lookup database // answers at once. With GeoJS, while a setting needs the answer, such as a -// country list or a biased threshold, a new client's request waits for it. -// ctx is the request's own context. -func (rq *request) lookUp(ctx context.Context) { +// country list or a biased threshold, a new client's request waits for it, +// unless refusedWhateverTheAnswer reports true. ctx is the request's own +// context. +func (rq *request) lookUp(ctx context.Context, now time.Time) { if rq.h.config.LookupSource == "off" || !canBePlaced(rq.client) { return } - if rq.h.config.LookupSource == "file" { - rq.lookupAnswer = rq.h.lookupFile.LookUp(rq.h.clientGroup(rq.client)) - } else { - rq.lookupAnswer = rq.h.geojs.LookUp(ctx, rq.h.clientGroup(rq.client)) + client := rq.h.clientGroup(rq.client) + + switch { + case rq.h.config.LookupSource == "file": + rq.lookupAnswer = rq.h.lookupFile.LookUp(client) + case rq.h.answersQuestions() && rq.refusedWhateverTheAnswer(now): + rq.lookupAnswer = rq.h.geojs.LookUpWithoutWaiting(ctx, client) + rq.skippedLookupWait = rq.lookupAnswer.Answered.IsZero() + default: + rq.lookupAnswer = rq.h.geojs.LookUp(ctx, client) } rq.lookedUp = true diff --git a/internal/proxy/lookup_test.go b/internal/proxy/lookup_test.go index 695c657..9ea14eb 100644 --- a/internal/proxy/lookup_test.go +++ b/internal/proxy/lookup_test.go @@ -319,7 +319,7 @@ func TestLookupHeadersArePassedToTheAppAndTheClientsOwnRemoved(t *testing.T) { addr, out, _ := startProxyWithClock(t, app.URL, geojsURL, time.Now, map[string]string{ trustedProxies: trustLocalhost, lookupTimeout: "1h", - addLookupHeaders: "true", + addLookupHeaders: on, }) s := &sender{t: t, addr: addr, out: out} @@ -383,8 +383,8 @@ func TestRequestWaitsAsLongAsTheLookupTimeoutSays(t *testing.T) { // network, which would keep that clock from moving on: the request is // handed to the proxy's handler, and GeoJS is one that never answers. synctest.Test(t, func(t *testing.T) { - // Not the default second. The exclusive list needs the answer, and - // the app is never reached. + // Not the default. The exclusive list needs the answer, and the app + // is never reached. const timeout = 3 * time.Second server, out, _ := newProxy(t, "http://app.invalid", unansweredGeoJSURL, diff --git a/internal/proxy/metrics_test.go b/internal/proxy/metrics_test.go index 401c102..c4d1634 100644 --- a/internal/proxy/metrics_test.go +++ b/internal/proxy/metrics_test.go @@ -424,8 +424,8 @@ func TestMetricsCountGeoJSRequestsAndFailures(t *testing.T) { req.Header.Set(forwardedFor, fromDE) wantStatus(t, do(t, req), http.StatusOK) - // The client stops waiting for GeoJS after a second, so GeoJS's - // failure can come after its request has ended. + // The client stops waiting for GeoJS once SWWAF_LOOKUP_TIMEOUT is up, so + // GeoJS's failure can come after its request has ended. deadline := time.Now().Add(waitLimit) metrics := scrape(t, addr) diff --git a/internal/proxy/proxy_test.go b/internal/proxy/proxy_test.go index 6af3e76..0613933 100644 --- a/internal/proxy/proxy_test.go +++ b/internal/proxy/proxy_test.go @@ -91,6 +91,9 @@ const ( // off is the value that switches a setting off. const off = "off" +// on is the value that switches a setting that takes true or false on. +const on = "true" + // output collects what smallwebwaf writes on stdout. type output struct { mu sync.Mutex diff --git a/internal/proxy/request.go b/internal/proxy/request.go index 99c38d0..d5721e8 100644 --- a/internal/proxy/request.go +++ b/internal/proxy/request.go @@ -58,6 +58,10 @@ type decision struct { // what the lookup gave then, the zero Answer while GeoJS had given none. lookedUp bool lookupAnswer lookup.Answer + // skippedLookupWait is true for a request that did not wait for GeoJS and + // found no answer kept. SWWAF_UNKNOWN_LIMIT_PERCENT does not apply to it, + // so that not waiting never lowers its client's limits. + skippedLookupWait bool // counted is true for a request the rate limits counted, whose bytes // the byte limits count once it has ended. limitPercent and // bytesPercent are then its client's limit percentages for the rate @@ -222,12 +226,12 @@ func requestHeaders(r *http.Request, names []string) map[string]string { // is known, before anything reaches the app, and before its body is read, // but for the part the Core Rule Set reads. It returns nil to let the // request through. The checks of checkClient come first, answered with -// SWWAF_BAN_RESPONSE, or 403 for a block rule or the Core Rule Set, and -// then the size limit, so that a request the rate limits count is counted -// even when it is refused for its size. In observe mode a request -// checkClient refuses goes on to the size limit like any other. A size or -// time limit the Core Rule Set's reading of the body meets ends the -// request in either mode. ctx is the request's own context. +// SWWAF_BAN_RESPONSE, 403 for a block rule or the Core Rule Set, or 431 for +// servewaf's header limits, and then the size limit, so that a request the +// rate limits count is counted even when it is refused for its size. In +// observe mode a request checkClient refuses goes on to the size limit like +// any other. A size or time limit the Core Rule Set's reading of the body +// meets ends the request in either mode. ctx is the request's own context. func (rq *request) check(ctx context.Context) *refusal { action := rq.checkClient(ctx) @@ -243,6 +247,8 @@ func (rq *request) check(ctx context.Context) *refusal { rq.line.WouldAction = action case action == requestlog.ActionRuleBlocked || action == requestlog.ActionWAFBlocked: return &refusal{status: http.StatusForbidden, action: action} + case action == requestlog.ActionTooLarge: + return &refusal{status: http.StatusRequestHeaderFieldsTooLarge, action: action} default: return rq.banResponse(action) } @@ -271,9 +277,10 @@ func (rq *request) check(ctx context.Context) *refusal { // rate limits, unless the client is in SWWAF_RATE_LIMIT_EXEMPT_NETS or the // request's path is exempt under SWWAF_RATE_LIMIT_EXEMPT_PATHS, so that // every other request is counted, each of them by the client's limit -// percentages, then SWWAF_TRAP_PATHS, then the rule files, and last the -// Core Rule Set. A request exempt from the rate limits is exempt from the -// byte limits too. ctx is the request's own context. +// percentages, then SWWAF_TRAP_PATHS, then the rule files, then, in +// servewaf, the header limits, and last the Core Rule Set. A request exempt +// from the rate limits is exempt from the byte limits too. ctx is the +// request's own context. func (rq *request) checkClient(ctx context.Context) string { cfg := rq.h.config if isInside(rq.client, cfg.AllowNets) { @@ -290,7 +297,7 @@ func (rq *request) checkClient(ctx context.Context) string { return requestlog.ActionBanned } - rq.lookUp(ctx) + rq.lookUp(ctx, now) // The lookup can wait for GeoJS, so the checks after it read the time // again: a request counted at the time it arrived could be counted @@ -314,8 +321,7 @@ func (rq *request) checkClient(ctx context.Context) string { return requestlog.ActionDenied } - rq.counted = !isInside(rq.client, cfg.RateLimitExemptNets) && - !pathExempt(rq.in, cfg.RateLimitExemptPaths) + rq.counted = rq.countedByRateLimits() if rq.counted { rq.limitPercent, rq.bytesPercent = rq.limitPercentages() rq.line.LimitPercent, rq.line.LimitPercentSetting = rq.limitPercent.logged() @@ -335,9 +341,23 @@ func (rq *request) checkClient(ctx context.Context) string { return action } + if rq.overHeaderLimits() { + return requestlog.ActionTooLarge + } + return rq.checkCoreRuleSet() } +// countedByRateLimits reports whether the rate limits count the request: its +// client is not in SWWAF_RATE_LIMIT_EXEMPT_NETS, and its path is not exempt +// under SWWAF_RATE_LIMIT_EXEMPT_PATHS. +func (rq *request) countedByRateLimits() bool { + cfg := rq.h.config + + return !isInside(rq.client, cfg.RateLimitExemptNets) && + !pathExempt(rq.in, cfg.RateLimitExemptPaths) +} + // pathExempt reports whether r is exempt under prefixes, // SWWAF_RATE_LIMIT_EXEMPT_PATHS or SWWAF_WAF_EXEMPT_PATHS: whether its // path as sent, the path the app receives, not percent-decoded, starts diff --git a/internal/proxy/servewaf.go b/internal/proxy/servewaf.go index 9fe3923..61d7aa0 100644 --- a/internal/proxy/servewaf.go +++ b/internal/proxy/servewaf.go @@ -53,6 +53,13 @@ const ReportPath = "/_smallwebwaf/report" // QuestionBodyMaxBytes is the most of a visitor's body a question carries. const QuestionBodyMaxBytes = 64 << 10 +// The header limits: the most cookies and header fields a request may have +// when the checks reach the Core Rule Set. Real clients never reach them. +const ( + maxCookies = 50 + maxHeaderFields = 100 +) + const ( // maxAwaitingReport is the most answered requests held until their // reports come. Past it, the request answered longest ago is dropped. @@ -146,6 +153,62 @@ func (h *handler) serveWAF(w http.ResponseWriter, r *http.Request) { h.answerOwnEndpoint(w, r) } +// answersQuestions reports whether h is servewaf, which alone holds requests +// for their reports. +func (h *handler) answersQuestions() bool { + return h.awaitingReport != nil +} + +// refusedWhateverTheAnswer reports whether the request is one that does not wait +// for GeoJS, since the checks will refuse it at now whatever GeoJS answers about +// its client: it is over the header limits, a blocklist lists its client while +// SWWAF_BLOCKLIST_ACTION is deny, the CrowdSec decision list lists it, or it is +// over a whole rate limit. In observe mode, which refuses nothing, none is. +// rproxy lets a request through when servewaf answers too late. +func (rq *request) refusedWhateverTheAnswer(now time.Time) bool { + if rq.h.config.Observe { + return false + } + + lists := rq.h.lists + _, decided := lists.CrowdSecDecision(rq.client, now) + + return decided || rq.overHeaderLimits() || + (rq.h.config.BlocklistAction == deny && len(lists.ListedBy(rq.client)) > 0) || + (rq.countedByRateLimits() && + rq.h.limiter.OverARateLimit(rq.h.clientGroup(rq.client), now)) +} + +// overHeaderLimits reports whether servewaf refuses the request for its +// headers: it has more than maxHeaderFields fields, each value of a header sent +// twice being one, or more than maxCookies cookies, each part of a Cookie header +// between semicolons being one unless it is empty or only spaces and tabs. The +// Core Rule Set runs each of its rules on each cookie, so that thousands of small +// cookies keep it busy for over a second. It trims only spaces and control +// characters, of which a header value can hold only tabs, so a part of Unicode +// spaces is a cookie to it. +func (rq *request) overHeaderLimits() bool { + if !rq.h.answersQuestions() { + return false + } + + fields, cookies := 0, 0 + + for _, values := range rq.in.Header { + fields += len(values) + } + + for _, value := range rq.in.Header.Values("Cookie") { + for part := range strings.SplitSeq(value, ";") { + if strings.Trim(part, " \t") != "" { + cookies++ + } + } + } + + return fields > maxHeaderFields || cookies > maxCookies +} + // answerQuestion runs the checks on the visitor's request a question // carries, and answers rproxy what to do with it. A denied request's counting // is finished here, and an allowed one's by its report. diff --git a/internal/proxy/servewaf_test.go b/internal/proxy/servewaf_test.go index b618493..2081dd7 100644 --- a/internal/proxy/servewaf_test.go +++ b/internal/proxy/servewaf_test.go @@ -767,7 +767,7 @@ func TestAllowCarriesTheLookupHeadersWhileTheSettingIsSet(t *testing.T) { setting string wantASN, wantCountry string }{ - {"true", "AS64496", "DE"}, + {on, "AS64496", "DE"}, {"false", "", ""}, } { t.Run(tc.setting, func(t *testing.T) { diff --git a/internal/ratelimit/ratelimit.go b/internal/ratelimit/ratelimit.go index 2318102..73e35ee 100644 --- a/internal/ratelimit/ratelimit.go +++ b/internal/ratelimit/ratelimit.go @@ -228,6 +228,30 @@ func (l *Limiter) Count( return l.count(client, now, 1, 0, percent) } +// OverARateLimit reports whether one more request from client at now would +// take it over a whole rate limit. A client's percentage can only lower a +// limit, so that request is over the limit at any percentage. It counts +// nothing, and does not make the client the most recently seen. +func (l *Limiter) OverARateLimit(client netip.Prefix, now time.Time) bool { + l.mu.Lock() + defer l.mu.Unlock() + + c, seen := l.clients.Peek(client) + if !seen { + return false + } + + buckets := c.buckets() + + for i, w := range l.windows { + if w.limit > 0 && buckets[i].Add(now, w.length, 0)+1 > float64(w.limit) { + return true + } + } + + return false +} + // CountBytes counts bytes, those of a request from client that has ended, // at now, in every window, and returns the client's counts in each window. // It reports whether the bytes take the client over a byte limit, of which diff --git a/internal/ratelimit/ratelimit_test.go b/internal/ratelimit/ratelimit_test.go index 79f3b59..5791f48 100644 --- a/internal/ratelimit/ratelimit_test.go +++ b/internal/ratelimit/ratelimit_test.go @@ -26,6 +26,50 @@ const ( hour = "hour" ) +func TestOverARateLimitSaysWhatTheNextRequestWouldDoAndCountsNothing(t *testing.T) { + t.Parallel() + + client := netip.MustParsePrefix("203.0.113.9/32") + now := midnight() + limiter := ratelimit.New(ratelimit.Limits{PerMinute: limit, PerDay: limit + 1}, + tableSize) + + if limiter.OverARateLimit(client, now) { + t.Error("a client never seen is over a rate limit") + } + + if limiter.Len() != 0 { + t.Error("asking about a client never seen added it to the table") + } + + for range limit { + if limiter.OverARateLimit(client, now) { + t.Error("a client with room for another request is over a rate limit") + } + + wantCount(t, limiter, client, now, "") + } + + if !limiter.OverARateLimit(client, now) { + t.Error("a client at its limit is not over a rate limit") + } + + held, _ := limiter.Client(client) + if held.Minute.Current != limit { + t.Errorf("the client's minute holds %d requests, want %d", held.Minute.Current, limit) + } + + wantCount(t, limiter, client, now, minute) + + // A limit that is off is never gone over. + off := ratelimit.New(ratelimit.Limits{}, tableSize) + off.Count(client, now, whole) + + if off.OverARateLimit(client, now) { + t.Error("a client is over a rate limit that is off") + } +} + func TestEachWindowRefusesAtItsLimitAndLetsTheClientBack(t *testing.T) { t.Parallel() diff --git a/internal/requestlog/requestlog.go b/internal/requestlog/requestlog.go index c05c0c2..6ddf8e7 100644 --- a/internal/requestlog/requestlog.go +++ b/internal/requestlog/requestlog.go @@ -18,7 +18,8 @@ import ( const ( // ActionForward is a request passed to the app. ActionForward = "forward" - // ActionTooLarge is a request or response over its size limit. + // ActionTooLarge is a request or response over its size limit, or, in + // servewaf, a request over the header limits. ActionTooLarge = "too_large" // ActionTimedOut is a request or response that ran out of time. ActionTimedOut = "timed_out" @@ -139,7 +140,8 @@ type Line struct { // WouldAction is, in observe mode, the action enforce mode would have // taken with a request it would have refused: ActionDenied, // ActionBanned, ActionCountryDenied, ActionRateLimited, - // ActionRuleBlocked or ActionWAFBlocked. + // ActionRuleBlocked, ActionWAFBlocked, or, in servewaf, ActionTooLarge + // for the header limits. WouldAction string `json:"would_action,omitempty"` // LimitPercent and LimitPercentSetting are, for a request the rate // limits counted whose client a biased threshold gives a percentage of From 17c5dd2438e98de8f2fda7be52b5c9f8f42c3a13 Mon Sep 17 00:00:00 2001 From: clawbot <35+clawbot@noreply.example.org> Date: Thu, 5 Nov 2026 06:32:08 +0100 Subject: [PATCH 6/9] rproxy, the proxy in front of each app, asks servewaf and fails open (closes #171) `smallwebwaf rproxy` reads the first 64 KiB of each body, asks servewaf over HTTP/1.1, and passes what servewaf allows to the app within its limits. Any other answer goes to the visitor as servewaf gave it. SWRP_DEFAULT_DISPOSITION decides only when servewaf cannot be connected to, does not answer at all, or does not answer within SWRP_TIMEOUT_MS. A method SPEC.md does not list gets 405. Duplication: internal/rproxy copies passing-on code from internal/proxy until https://git.eeqj.de/sneak/smallwebwaf/issues/172 removes the originals. Rule suppressed: gosec G704 and G705; gosec at internal/smallwebwaf/rproxy_test.go:431; tagliatelle at internal/requestlog/requestlog.go:203; paralleltest on one test. Judgement call: an answer over 1 MiB, or with a status below 200, gets 502. Judgement call: every target not starting with one / gets 400. Model: opus-5-5 --- README.md | 21 +- SPEC.md | 265 +++- cmd/smallwebwaf/main.go | 3 +- docs/admin-endpoints.md | 6 + docs/request-log.md | 39 +- docs/settings.md | 44 +- internal/config/config.go | 10 +- internal/config/rproxy.go | 167 ++ internal/config/rproxy_test.go | 206 +++ internal/requestlog/requestlog.go | 46 +- internal/requestlog/requestlog_test.go | 22 + internal/rproxy/bodies.go | 211 +++ internal/rproxy/client.go | 122 ++ internal/rproxy/limits_test.go | 811 ++++++++++ internal/rproxy/question_test.go | 477 ++++++ internal/rproxy/request.go | 593 ++++++++ internal/rproxy/rproxy.go | 228 +++ internal/rproxy/rproxy_test.go | 1758 ++++++++++++++++++++++ internal/rproxy/servewaf.go | 266 ++++ internal/smallwebwaf/rproxy.go | 99 ++ internal/smallwebwaf/rproxy_test.go | 787 ++++++++++ internal/smallwebwaf/smallwebwaf.go | 18 +- internal/smallwebwaf/smallwebwaf_test.go | 43 +- 23 files changed, 6183 insertions(+), 59 deletions(-) create mode 100644 internal/config/rproxy.go create mode 100644 internal/config/rproxy_test.go create mode 100644 internal/rproxy/bodies.go create mode 100644 internal/rproxy/client.go create mode 100644 internal/rproxy/limits_test.go create mode 100644 internal/rproxy/question_test.go create mode 100644 internal/rproxy/request.go create mode 100644 internal/rproxy/rproxy.go create mode 100644 internal/rproxy/rproxy_test.go create mode 100644 internal/rproxy/servewaf.go create mode 100644 internal/smallwebwaf/rproxy.go create mode 100644 internal/smallwebwaf/rproxy_test.go diff --git a/README.md b/README.md index 504e4be..3bc03b1 100644 --- a/README.md +++ b/README.md @@ -10,9 +10,10 @@ refuses the ones that look like abuse. `smallwebwaf` is also a reverse proxy: it accepts requests on the app's behalf, passes them to the app, and passes the app's answers back. -It needs no setting. Every setting has a default chosen for a service on the -open internet. [`SPEC.md`](SPEC.md) says how `smallwebwaf` behaves and why. The -reference documents under [`docs/`](docs/) are listed under "Documents" below. +It needs no setting. Every `SWWAF_` setting has a default chosen for a service +on the open internet. [`SPEC.md`](SPEC.md) says how `smallwebwaf` behaves and +why. The reference documents under [`docs/`](docs/) are listed under "Documents" +below. ## What it protects against @@ -342,15 +343,21 @@ pinned. `127.0.0.1` and `::1` for forwarded headers, besides the private address ranges. - Both processes start with the container's environment variables, so every - setting of `smallwebwaf` starts with `SWWAF_`, and the app can read a token - given as one. A token given as a file that only the `smallwebwaf` user can - read is out of its reach (see "Settings given as files" in + setting of this `smallwebwaf` process starts with `SWWAF_`, and the app can + read a token given as one. A token given as a file that only the `smallwebwaf` + user can read is out of its reach (see "Settings given as files" in [`docs/settings.md`](docs/settings.md)). - The state files live in `/var/lib/smallwebwaf`. A volume mounted there keeps them across a deploy. - `smallwebwaf servewaf` runs the same checks for the sites that ask it about each request, and passes nothing to an app (see "`servewaf`" in [`SPEC.md`](SPEC.md)). +- `smallwebwaf rproxy` is the reverse proxy in front of one app that asks + `smallwebwaf servewaf` about each request. When `servewaf` cannot be connected + to, does not answer at all, or does not answer within `SWRP_TIMEOUT_MS`, + `SWRP_DEFAULT_DISPOSITION` decides. Its default, `allow`, keeps the site + answering. The settings of `rproxy` start with `SWRP_`, and it needs + `SWRP_WAF_BACKEND`, `servewaf`'s URL (see "`rproxy`" in [`SPEC.md`](SPEC.md)). - `docker stop` stops both processes at once. `smallwebwaf` gives the requests in progress five seconds to finish, so the app must keep answering for five seconds after `SIGTERM`. @@ -363,8 +370,6 @@ what each does. ## TODO - Planned, as "Planned, not built" in [`SPEC.md`](SPEC.md) describes: - - `smallwebwaf rproxy`, the reverse proxy in front of each app that asks - `smallwebwaf servewaf` about each request; - sharing bans between the instances of several apps; - sending the log over RELP. diff --git a/SPEC.md b/SPEC.md index 6843758..4980d1f 100644 --- a/SPEC.md +++ b/SPEC.md @@ -12,9 +12,9 @@ to them. - It protects one self-hosted app on the open internet from scrapers, scanners and other abusers. It does so from the app's first request, with no setting at - all. Every setting has a default chosen for a service facing the internet - in 2026. A setting that needs an alert destination, an account key or a token - stays off until the operator supplies it. + all. Every `SWWAF_` setting has a default chosen for a service facing the + internet in 2026. A setting that needs an alert destination, an account key or + a token stays off until the operator supplies it. - It limits each client's requests and bytes per minute, hour and day. It bounds the size and duration of every request and answer. It detects attacks. - It bans abusers: briefly at first, for seven days on a clear sign of attack, @@ -24,8 +24,9 @@ to them. reputation sources, and send alerts. - Everything is set by environment variables whose names start with `SWWAF_`, since `smallwebwaf` shares its container, and so its environment, with the - app. The only other inputs are files: the rule files, the state files, a - lookup database, and a setting given as a file. + app. `smallwebwaf rproxy` uses `SWRP_` in their place. The only other inputs + are files: the rule files, the state files, a lookup database, and a `SWWAF_` + setting given as a file. ## Non-goals @@ -48,8 +49,10 @@ to them. files, the lookup database and the state files, listens, and serves until `SIGTERM` or `SIGINT`. Then it stops, writing the state files. Run as `smallwebwaf servewaf`, it answers `rproxy` instead of passing requests to - an app (see "`servewaf`"). Run as `smallwebwaf healthcheck`, it is the - image's health check instead. + an app (see "`servewaf`"). Run as `smallwebwaf rproxy`, it reads only that + command's settings, and runs `internal/rproxy` until `SIGTERM` or `SIGINT` + (see "`rproxy`"). Run as `smallwebwaf healthcheck`, it is the image's + health check instead. - `internal/config`: reads the settings, the one place they are read. - `internal/proxy`: what happens to each request. It works out the client and runs the checks in `check`. It passes the request to the app with the @@ -57,6 +60,8 @@ to them. limits, and answers its own endpoints in `answerAdmin`. Once the request has ended, it counts bytes, refusals and anomalies, and writes the log line. It also holds `servewaf`'s question, answer and report. + - `internal/rproxy`: the reverse proxy in front of one app that asks + `servewaf` about each request. - `internal/ratelimit`: the table of clients. It counts requests, bytes and refusals, and keeps each client's history. - `internal/bans`: the bans. It holds each netblock's bans with their notes, @@ -291,17 +296,20 @@ Conventions: - `SWWAF_WAF_ANOMALY_THRESHOLD`, `SWWAF_WAF_BODY_LIMIT` and `SWWAF_ERROR_BURST_THRESHOLD`; - `SWWAF_ALERT_COOLDOWN` and `SWWAF_ALERT_MAX_PER_HOUR`; - - every anomaly threshold. + - every anomaly threshold; + - `SWRP_MAX_REQUEST_SIZE`, `SWRP_MAX_RESPONSE_SIZE`, + `SWRP_MAX_REQUEST_TIME_MS` and `SWRP_MAX_RESPONSE_TIME_MS`. No other setting can be switched off with `off`. - A list set to an empty value is an empty list, and replaces the default. -- Any setting can be read from a file instead, for secrets and long lists: - `SWWAF_ADMIN_TOKEN_FILE` names the file that holds `SWWAF_ADMIN_TOKEN`. The - file's contents are the value, less one newline at the end, and are checked as - the value itself would be. Setting both forms stops the start, and so does - naming a file that cannot be read. `SWWAF_LOG_REMOTE_TLS_CA_FILE` already - holds a file name, so it has no `_FILE` form. +- Any `SWWAF_` setting can be read from a file instead, for secrets and long + lists: `SWWAF_ADMIN_TOKEN_FILE` names the file that holds `SWWAF_ADMIN_TOKEN`. + The file's contents are the value, less one newline at the end, and are + checked as the value itself would be. Setting both forms stops the start, and + so does naming a file that cannot be read. `SWWAF_LOG_REMOTE_TLS_CA_FILE` + already holds a file name, so it has no `_FILE` form. No `SWRP_` setting has + one, since `rproxy` never touches disk. - Settings, files included, are read once at start. Changing one means restarting the container. The files `smallwebwaf` watches while it runs are its state files, its rule files and the lookup database. @@ -1051,14 +1059,27 @@ request. It is the visitor's request as the visitor sent it: - every header the visitor sent, including its own `X-Forwarded-For`; - the first 64 KiB of the body, or all of it when shorter. +Go's HTTP code changes some of the visitor's headers, such as: + +- Go's client does not send an empty `User-Agent`. +- Go's server adds `Cache-Control: no-cache` to a request with + `Pragma: no-cache` and no `Cache-Control`, before `rproxy` sees it. + +`rproxy` itself refuses any request it cannot send this way (see "Requests +`rproxy` refuses itself"). So a question always has: + +- a path starting with one `/`; +- a `Host` that is not empty and holds no IPv6 zone; +- at most one `User-Agent`. + A body whose length the visitor did not announce comes chunked. Any other body comes with a `Content-Length` of the part it carries. `rproxy` sets these headers, replacing any the visitor sent, and adds no others: -- `X-Smallwebwaf-Client`: the visitor's address, which `rproxy` has worked out. - `servewaf` takes it as the client, and groups an IPv6 address as "What happens - to a request" says. +- `X-Smallwebwaf-Client`: the visitor's address, which `rproxy` has worked out, + without any IPv6 zone. `servewaf` takes it as the client, and groups an IPv6 + address as "What happens to a request" says. - `X-Smallwebwaf-Peer`: the address of `rproxy`'s own TCP peer, for the log line's `peer_ip`. - `X-Smallwebwaf-Content-Length`: the body length the visitor announced. It is @@ -1122,7 +1143,8 @@ answer but the last below carries `X-Smallwebwaf-Disposition`. place of a refusal while `SWWAF_BAN_RESPONSE` is `close`. `rproxy` closes the visitor's connection without an answer. - No such header, status `400`, for a question `servewaf` refuses. `rproxy` - treats this answer as it treats no answer at all. + gives the visitor its status, headers and body, as it does for any answer + without a valid `X-Smallwebwaf-Disposition`. In `observe` mode a request that `enforce` mode would refuse is answered `allow`, and its log line has `would_action`. A path under `/_smallwebwaf/` is @@ -1201,9 +1223,14 @@ such requests at once can overload `servewaf` past `SWRP_TIMEOUT_MS`. Once the visitor's answer has ended, `rproxy` sends `POST /_smallwebwaf/report` for each request `servewaf` answered with `allow`, `deny` or `close`. It sends -none after a `400` or no answer. The report carries no `X-Smallwebwaf-Client`. -Its body is one JSON object with all of these fields, named as the request log's -fields of the same meaning: +none after: + +- an answer without a valid `X-Smallwebwaf-Disposition`, such as a `400`; +- an answer it cannot pass on (see "Without an answer from `servewaf`"); +- no answer. + +The report carries no `X-Smallwebwaf-Client`. Its body is one JSON object with +all of these fields, named as the request log's fields of the same meaning: - `request_id`: the question's `X-Request-ID`; - `status`: what the visitor got, `0` if nothing, otherwise from `100` to `599`; @@ -1289,6 +1316,201 @@ by a request and ban notes name the site by its host. `docs/metrics.md` says how its metrics count. +## `rproxy` + +`smallwebwaf rproxy` is the reverse proxy in front of one app. It asks +`servewaf` about each request, and passes on what `servewaf` allows. It keeps +nothing between requests and never touches disk. When `servewaf` cannot be +connected to, does not answer at all, or does not answer within +`SWRP_TIMEOUT_MS`, `SWRP_DEFAULT_DISPOSITION` decides. Its default, `allow`, +keeps the site answering. + +`rproxy` reads only its `SWRP_` settings, which `docs/settings.md` lists. One of +them is required: `SWRP_WAF_BACKEND`, `servewaf`'s URL. It has no metrics, no +rule files and no state files, and it writes its lines to stdout only. + +The image's run script and health check are those of `smallwebwaf` with no +command (see "Deployment"). A container that runs `rproxy` names its own command +and health check. + +### What `rproxy` does with a request + +1. `GET /_smallwebwaf/healthz` is answered `ok` at once, without asking + `servewaf`. +2. `rproxy` works out the visitor's address. It believes `X-Forwarded-For` only + from `SWRP_TRUSTED_PROXIES`, and reads it as step 2 of "What happens to a + request" says. It removes any IPv6 zone. +3. It makes a fresh request id, whatever id the request came with. `servewaf` + and the app get it as `X-Request-ID`, and `rproxy`'s log line gives it. +4. It refuses a request with a method it does not pass on, or one it cannot put + to `servewaf` as the visitor sent it (see "Requests `rproxy` refuses + itself"). +5. It reads up to the first 64 KiB of the body into memory. +6. It sends `servewaf` the question (see "The question"). It speaks HTTP/1.1 to + `servewaf`, whatever `servewaf` offers. Go's HTTP/2 client would refuse to + send a visitor's `Upgrade` header, and most `Connection` headers. The + question never carries the visitor's `Trailer`. +7. It does what the answer says (see "The answer"): + - `allow`: it passes the request to the app. + - `close`: it closes the visitor's connection without an answer. + - Any other answer: the visitor gets its status, headers and body, less + `X-Smallwebwaf-Disposition`. That holds for a `deny`, for an error such as + `servewaf`'s `400` or a `5xx`, and for an answer without a valid + `X-Smallwebwaf-Disposition`. +8. Once the answer has ended, it writes its log line. `docs/request-log.md` + gives the fields. An app's answer can end just after the line: Go's server + sends the closing chunk of a chunked answer, and may send all of an empty + one, only once `rproxy` has finished with the request. +9. It then sends the report, if `servewaf` answered `allow`, `deny` or `close` + (see "The report"). The visitor does not wait for it. A report that fails is + dropped, with a process warning. A status above `599` from the app goes to + the visitor as it is, and into the report as `599`, the highest `servewaf` + takes. + +### Requests `rproxy` refuses itself + +`rproxy` passes on only these methods: + +- `GET` +- `HEAD` +- `POST` +- `OPTIONS` +- `PUT` +- `DELETE` +- `PATCH` +- `QUERY` + +It answers any other method `405`, with no question and no report. The answer's +`Allow` header names these methods. + +`rproxy` answers `400`, with no question and no report, to a request with: + +- a target that does not start with `/`, such as `x:admin`, + `http://site.example/` or `site.example:443`; +- a target that starts with `//`; +- no `Host`, or an empty one; +- a `Host` holding an IPv6 zone: one that starts with `[` and holds a `%`, such + as `[fe80::1%25eth0]:80`; +- more than one `User-Agent`. + +Go's HTTP client could not send any of these as the visitor sent it. + +Go's HTTP server answers `OPTIONS *` itself, before `rproxy` sees it. + +`rproxy` closes the connection without an answer when it cannot read the first +64 KiB of the body. That happens for a malformed chunk, and for a visitor that +goes away or stops its sending side. There is no question and no report, and the +log line has status `0`. + +### Without an answer from `servewaf` + +`SWRP_DEFAULT_DISPOSITION` decides what is done with the request only when +`servewaf`: + +- cannot be connected to; +- does not answer at all; +- does not answer within `SWRP_TIMEOUT_MS`, which covers connecting, sending the + question and reading the whole answer. + +An answer whose body breaks off counts as no answer at all. + +`allow`, the default, passes the request to the app. `deny` answers `503`. +Either way `rproxy` writes a process warning, its log line has +`default_disposition`, and it sends no report. + +A visitor can make `servewaf` slow. A request that keeps `servewaf` busy past +`SWRP_TIMEOUT_MS`, like any request while `servewaf` is overloaded, gets +`SWRP_DEFAULT_DISPOSITION`. With `deny`, such a request gets `503` instead of +reaching the app. + +A path under `/_smallwebwaf/` that `allow` would pass on is answered `404` +instead, so that the app never sees that prefix. + +`rproxy` cannot pass on an answer whose body is over 1 MiB or whose status is +below `200`. The visitor gets `502` from `rproxy`, never the default. `rproxy` +writes a process warning, its log line has no `disposition`, and it sends no +report. + +A visitor that goes away while `servewaf` is being asked is no failure of +`servewaf`. `rproxy` closes the connection, with no warning, no default and no +report. Its log line has status `0` and no `disposition`. + +### Passing a request to the app + +`rproxy` sends the app at `SWRP_UPSTREAM_URL` the visitor's request: the part of +the body it holds, then the rest, streaming. The app's answer comes back the +same way. Apart from what Go's HTTP code changes, the request reaches the app +unchanged but for these headers: + +- `X-Forwarded-For`, with the address of `rproxy`'s TCP peer added, when that + peer is inside `SWRP_TRUSTED_PROXIES`. The peer's other forwarded headers pass + on. +- For a peer outside `SWRP_TRUSTED_PROXIES`, its forwarded headers are its own + claims and are replaced. `X-Forwarded-For` names the peer, `X-Forwarded-Host` + the host it asked for, and `X-Forwarded-Proto` plain `http`. The other headers + in which traefik tells an app about a visitor are removed. +- `X-Request-ID`, which carries the fresh id. +- `X-Client-ASN` and `X-Client-Country`, which carry what `servewaf` answered. A + header it did not give is left out. Any of the two the visitor sent is + removed. +- The headers that concern only the connection to `rproxy` never reach the app: + - `Connection`, and any header it names; + - `Keep-Alive`; + - `Proxy-Authenticate`; + - `Proxy-Authorization`; + - `Proxy-Connection`; + - `Te`, though one holding `trailers` goes on as `Te: trailers`; + - `Upgrade`. + +A chunked body's trailers reach the app, but not `servewaf`. + +A WebSocket upgrade passes through, with `Connection: Upgrade` and its +`Upgrade`. The report of a WebSocket comes when it closes, with what it carried +each way. + +### `rproxy`'s limits + +The limits work as "Size and time limits" says, with these settings in place of +those: + +- `SWRP_MAX_REQUEST_SIZE` for `SWWAF_REQUEST_MAX_BYTES`. +- `SWRP_MAX_RESPONSE_SIZE` for `SWWAF_RESPONSE_MAX_BYTES`. +- `SWRP_MAX_REQUEST_TIME_MS` for `SWWAF_CLIENT_REQUEST_TIMEOUT` and + `SWWAF_UPSTREAM_REQUEST_TIMEOUT`. +- `SWRP_MAX_RESPONSE_TIME_MS` for `SWWAF_CLIENT_RESPONSE_TIMEOUT` and + `SWWAF_UPSTREAM_RESPONSE_TIMEOUT`. + +The largest request line and headers is 32 KiB. A kept-open connection is closed +after 120 seconds idle. Neither has a setting. + +`SWRP_MAX_REQUEST_TIME_MS` bounds the request line and headers. From their end, +it then bounds the time until the app has been sent the whole request. The wait +for `servewaf` counts toward it. A visitor too slow to send the first 64 KiB is +answered `408`, with no question and no report. + +`SWRP_MAX_RESPONSE_TIME_MS` runs from the end of the request to the last byte of +the answer reaching the visitor. The request ends when the app has been sent all +of it, or when the app's answer arrives if that is sooner. An answer `rproxy` +gives itself has the whole time from when it starts. + +`rproxy` applies the request size limit only after `servewaf` has allowed the +request. So a request `servewaf` denies gets `servewaf`'s answer, whatever its +size. A request `servewaf` allows is answered `413` when its body is over the +limit: at once for a length it announced, otherwise when the limit is reached. +If part of the body has gone to the app by then, the request to the app is cut +off. + +### Starting and stopping + +An invalid setting, or an address it cannot listen on, stops the start with a +process line. The process lines give the host's name as `instance`, and only +those at `info` or more severe are written. + +On `SIGTERM` or `SIGINT`, `rproxy` stops taking requests and gives those in +progress five seconds to finish. It then waits for the reports on their way, +each for up to `SWRP_TIMEOUT_MS`. A report that comes due after that is dropped, +such as that of a WebSocket that closes later. + ## Deployment `smallwebwaf` runs inside the app's own container: the app's Dockerfile builds @@ -1456,7 +1678,6 @@ and `script/gitea-requests` checks those notes. These points are specified as later work, and nothing in the code does them yet: -- Planned: `smallwebwaf rproxy`, the reverse proxy in front of each app. - Planned: sharing bans between the instances of several apps, by running one CrowdSec engine per host that every instance reports its bans to. It is reconsidered once real ban volumes are known. Until then each instance keeps diff --git a/cmd/smallwebwaf/main.go b/cmd/smallwebwaf/main.go index 9370da4..090fb90 100644 --- a/cmd/smallwebwaf/main.go +++ b/cmd/smallwebwaf/main.go @@ -1,6 +1,7 @@ // Command smallwebwaf is a web application firewall for one app: it runs // between traefik and the app and passes requests through within its limits, -// or, as `smallwebwaf servewaf`, answers rproxy's questions about them. +// or, as `smallwebwaf servewaf`, answers rproxy's questions about them, or, as +// `smallwebwaf rproxy`, is the reverse proxy that asks those questions. // Everything it does is in internal/smallwebwaf. package main diff --git a/docs/admin-endpoints.md b/docs/admin-endpoints.md index 2f856bd..6e7e6f4 100644 --- a/docs/admin-endpoints.md +++ b/docs/admin-endpoints.md @@ -51,3 +51,9 @@ curl -H "Authorization: Bearer $TOKEN" \ `smallwebwaf servewaf` answers these endpoints at its own address. No check runs on those requests (see "`servewaf`" in [`SPEC.md`](../SPEC.md)). + +## `rproxy` + +`smallwebwaf rproxy` answers only the health check, at its own address. It asks +`servewaf` about any other request under `/_smallwebwaf/` (see "`rproxy`" in +[`SPEC.md`](../SPEC.md)). diff --git a/docs/request-log.md b/docs/request-log.md index 1689319..bccd93c 100644 --- a/docs/request-log.md +++ b/docs/request-log.md @@ -3,6 +3,7 @@ The fields of the JSON object `smallwebwaf` writes for each request. "Request log" in [`SPEC.md`](../SPEC.md) says when a line is written and how the log is sent to a syslog server. A field that does not apply to a request is left out. +`smallwebwaf rproxy` writes a shorter line, which "`rproxy`'s line" below gives. ## Example @@ -12,7 +13,7 @@ sent to a syslog server. A field that does not apply to a request is left out. ## Every line -Every line has: +Every line but `rproxy`'s has: - `type`: `request` (the process's own messages have `process`); - the standard web log fields; @@ -181,3 +182,39 @@ For an allowed request, the report's counting also changes these fields: - `counts`: the byte totals now include this request's; - `limit_hit`, `offence` and `ban_expires`: set when the report's bytes broke a byte limit. + +## `rproxy`'s line + +`smallwebwaf rproxy` writes one short line for each request. `request_id` joins +it to `servewaf`'s two. It has these fields and no others: + +- `type`: `request`. +- `time`: when the request arrived, RFC 3339 with milliseconds, in UTC. +- `request_id`: the id `rproxy` made for the request when it arrived. +- `client_ip`: the visitor's address. +- `host` and `method`. +- `path` and `query`: as the visitor sent them. +- `status`: what the visitor got, `0` if nothing. +- `request_bytes`: the bytes of the visitor's body that `rproxy` read, with what + a WebSocket carried from the visitor. +- `response_bytes`: the bytes of the answer's body, with what a WebSocket + carried from the app. +- `duration_request`: milliseconds from the request's headers being read until + the app had been sent the whole request, or the answer started if that was + sooner. It includes the wait for `servewaf`. +- `duration_response`: milliseconds from then until the end of the answer. +- `disposition`: what was done with the request: `allow`, `deny` or `close`. It + is `servewaf`'s answer, or the value of `SWRP_DEFAULT_DISPOSITION`. It is left + out when: + - `servewaf` was not asked, as for the health check; + - the visitor went away while `servewaf` was being asked; + - `servewaf`'s answer had no valid `X-Smallwebwaf-Disposition`; + - `rproxy` could not pass on `servewaf`'s answer, and answered `502`. +- `default_disposition`: `true` when `SWRP_DEFAULT_DISPOSITION` gave + `disposition`, and otherwise left out. + +For example: + +``` +{"type":"request","time":"2026-10-03T12:00:00.123Z","request_id":"7Q2NHZ4KJ3VXW5YB6R3MEFTD2A","client_ip":"203.0.113.9","host":"app.example","method":"GET","path":"/","query":"","status":200,"request_bytes":0,"response_bytes":5120,"duration_request":1.204,"duration_response":3.104,"disposition":"allow"} +``` diff --git a/docs/settings.md b/docs/settings.md index 9c39e58..9b5855d 100644 --- a/docs/settings.md +++ b/docs/settings.md @@ -343,11 +343,15 @@ A few limits are fixed rather than settings: - `servewaf` holds at most 10,000 answered requests awaiting their reports; - `servewaf` refuses a request with more than 50 cookies or more than 100 header fields (see "Answering in time" under "`servewaf`" in - [`SPEC.md`](../SPEC.md)). + [`SPEC.md`](../SPEC.md)); +- `rproxy` takes a request line and headers of at most 32 KiB, and closes a + kept-open connection idle for 120 seconds; +- `rproxy` reads at most 1 MiB of the body of `servewaf`'s answer, and answers + `502` to a longer one. ## `servewaf` -`smallwebwaf servewaf` uses every setting here but these, which it ignores: +`smallwebwaf servewaf` uses every `SWWAF_` setting but these, which it ignores: - `SWWAF_UPSTREAM_URL` and `SWWAF_TRUSTED_PROXIES`; - the five timeouts under "Size and time limits"; @@ -367,3 +371,39 @@ Keep it well under `rproxy`'s `SWRP_TIMEOUT_MS`, as the default is. No question waits for `SWWAF_REPUTATION_TIMEOUT`, since the DNS blocklists and AbuseIPDB are asked in the background (see "Answering in time" under "`servewaf`" in [`SPEC.md`](../SPEC.md)). + +## `rproxy` + +`smallwebwaf rproxy` reads the settings below and no others, and nothing else +reads them. They are read once at start, and an invalid value stops the start. +None can be given as a file, since `rproxy` never touches disk. "`rproxy`" in +[`SPEC.md`](../SPEC.md) says what they do. + +A time here is a whole number of milliseconds above zero, such as `2000`. + +- `SWRP_WAF_BACKEND` (required, no default): `servewaf`'s URL. An `http` or + `https` URL with a host and an optional port, and nothing more. `rproxy` + speaks HTTP/1.1 to it. +- `SWRP_TIMEOUT_MS` (default `2000`): how long `rproxy` waits for `servewaf`'s + answer, to a question and to a report. A time; never `off`. +- `SWRP_DEFAULT_DISPOSITION` (default `allow`): what is done with a request when + `servewaf` cannot be connected to, does not answer at all, or does not answer + within `SWRP_TIMEOUT_MS`. `allow` passes it to the app, and `deny` answers + `503`. It does not apply to an answer `servewaf` gives, such as an error. +- `SWRP_MAX_REQUEST_SIZE` (default `100M`): the largest request body. A size or + `off`. +- `SWRP_MAX_RESPONSE_SIZE` (default `1G`): the largest response body. A size or + `off`. +- `SWRP_MAX_REQUEST_TIME_MS` (default `60000`): how long the visitor may take to + send its request line and headers. It then also bounds, from the end of the + headers, the time until the app has been sent the whole request. A time or + `off`. +- `SWRP_MAX_RESPONSE_TIME_MS` (default `1800000`): how long the answer may take, + from the end of the request to its last byte reaching the visitor. A time or + `off`. +- `SWRP_LISTEN_ADDR` (default `:8080`): where `rproxy` listens, such as `:9000`. +- `SWRP_UPSTREAM_URL` (default `http://127.0.0.1:8081`): the app. An `http` or + `https` URL with a host and an optional port, and nothing more. +- `SWRP_TRUSTED_PROXIES` (default `10.0.0.0/8,172.16.0.0/12,192.168.0.0/16`): + the netblocks whose `X-Forwarded-For` is believed. Set but empty, it trusts + nothing. diff --git a/internal/config/config.go b/internal/config/config.go index 914e3de..926483c 100644 --- a/internal/config/config.go +++ b/internal/config/config.go @@ -1,7 +1,9 @@ -// Package config reads smallwebwaf's settings. Every setting is an -// environment variable whose name starts with SWWAF_, or a file such a -// variable names, every setting has a default, and this package is the -// one place they are read. +// Package config reads smallwebwaf's settings, and is the one place they are +// read. Every setting is an environment variable. The settings of +// `smallwebwaf` and `smallwebwaf servewaf` start with SWWAF_, every one of them +// has a default, and most can be given as a file instead. Those of `smallwebwaf +// rproxy` start with SWRP_. Each of them but SWRP_WAF_BACKEND has a default, and +// none can be given as a file, since rproxy never touches disk. package config import ( diff --git a/internal/config/rproxy.go b/internal/config/rproxy.go new file mode 100644 index 0000000..8e5f189 --- /dev/null +++ b/internal/config/rproxy.go @@ -0,0 +1,167 @@ +package config + +import ( + "errors" + "fmt" + "log/slog" + "math" + "net/netip" + "net/url" + "strconv" + "strings" + "time" +) + +// RProxy is the settings of `smallwebwaf rproxy`. A size or time limit of zero +// is off. +type RProxy struct { + // ListenAddr is where rproxy listens (SWRP_LISTEN_ADDR). + ListenAddr string + // UpstreamURL is the app (SWRP_UPSTREAM_URL). + UpstreamURL *url.URL + // TrustedProxies are the netblocks whose X-Forwarded-For is believed + // (SWRP_TRUSTED_PROXIES). + TrustedProxies []netip.Prefix + // WAFBackend is servewaf's URL (SWRP_WAF_BACKEND). + WAFBackend *url.URL + // Timeout is how long rproxy waits for servewaf's answer to a question, + // and to a report (SWRP_TIMEOUT_MS). It cannot be off. + Timeout time.Duration + // DefaultDisposition is what is done with a request when servewaf cannot be + // connected to, does not answer at all, or does not answer within Timeout: + // allow or deny (SWRP_DEFAULT_DISPOSITION). + DefaultDisposition string + // MaxRequestSize is the largest request body (SWRP_MAX_REQUEST_SIZE), and + // MaxResponseSize the largest response body (SWRP_MAX_RESPONSE_SIZE). + MaxRequestSize int64 + MaxResponseSize int64 + // MaxRequestTime bounds sending the whole request: the visitor's request + // line and headers, and then, from their end, its body on its way to the + // app (SWRP_MAX_REQUEST_TIME_MS). MaxResponseTime bounds the answer, from + // the end of the request to its last byte reaching the visitor + // (SWRP_MAX_RESPONSE_TIME_MS). + MaxRequestTime time.Duration + MaxResponseTime time.Duration + + // settings are the values read, as given or by default, for the log line + // at start. + settings []slog.Attr +} + +var ( + errNeedsWAFBackend = errors.New( + "is required: it is servewaf's URL, such as http://servewaf:8080") + errNotDisposition = errors.New("is not allow or deny") + errNotMilliseconds = errors.New( + "is not a whole number of milliseconds above zero, such as 2000") + errNotMillisecondsOrOff = errors.New( + "is not a whole number of milliseconds above zero, such as 60000, or off") +) + +// RProxyFromEnvironment reads rproxy's settings with lookupEnv, normally +// os.LookupEnv. A setting that is not set takes its default, and +// SWRP_WAF_BACKEND, which has none, must be set. A setting that is set but +// invalid is an error that names it. Since rproxy never touches disk, its +// settings have no _FILE form. +func RProxyFromEnvironment(lookupEnv func(string) (string, bool)) (*RProxy, error) { + env := &environment{lookupEnv: func(name string) (string, bool) { + if strings.HasSuffix(name, "_FILE") { + return "", false + } + + return lookupEnv(name) + }} + cfg := &RProxy{ + ListenAddr: env.address("SWRP_LISTEN_ADDR", defaultListenAddr), + UpstreamURL: env.appURL("SWRP_UPSTREAM_URL", defaultUpstreamURL), + TrustedProxies: env.netblocks("SWRP_TRUSTED_PROXIES", privateRanges), + WAFBackend: env.wafBackend("SWRP_WAF_BACKEND"), + Timeout: env.milliseconds("SWRP_TIMEOUT_MS", "2000"), + DefaultDisposition: env.disposition("SWRP_DEFAULT_DISPOSITION", "allow"), + MaxRequestSize: env.size("SWRP_MAX_REQUEST_SIZE", "100M"), + MaxResponseSize: env.size("SWRP_MAX_RESPONSE_SIZE", "1G"), + MaxRequestTime: env.millisecondsOrOff("SWRP_MAX_REQUEST_TIME_MS", "60000"), + MaxResponseTime: env.millisecondsOrOff("SWRP_MAX_RESPONSE_TIME_MS", "1800000"), + } + + if env.err != nil { + return nil, env.err + } + + cfg.settings = env.settings + + return cfg, nil +} + +// LogValue makes an RProxy log as each setting's name with the value it was +// given, or its default. +func (c *RProxy) LogValue() slog.Value { + return slog.GroupValue(c.settings...) +} + +// wafBackend reads the setting that is servewaf's URL, which must be set. +func (e *environment) wafBackend(name string) *url.URL { + value := e.value(name, "") + if value == "" { + e.check(name, errNeedsWAFBackend) + + return nil + } + + backend, err := parseUpstreamURL(value) + e.check(name, err) + + return backend +} + +// disposition reads the setting that is what is done with a request when servewaf +// cannot be connected to, does not answer at all, or does not answer in time: +// allow or deny. +func (e *environment) disposition(name, defaultValue string) string { + value := e.value(name, defaultValue) + if value != "allow" && value != "deny" { + e.check(name, fmt.Errorf("%q %w", value, errNotDisposition)) + } + + return value +} + +// milliseconds reads a setting that is a whole number of milliseconds above +// zero. +func (e *environment) milliseconds(name, defaultValue string) time.Duration { + value := e.value(name, defaultValue) + + duration, valid := parseMilliseconds(value) + if !valid { + e.check(name, fmt.Errorf("%q %w", value, errNotMilliseconds)) + } + + return duration +} + +// millisecondsOrOff reads a setting that is a whole number of milliseconds +// above zero, or off, which is 0. +func (e *environment) millisecondsOrOff(name, defaultValue string) time.Duration { + value := e.value(name, defaultValue) + if value == off { + return 0 + } + + duration, valid := parseMilliseconds(value) + if !valid { + e.check(name, fmt.Errorf("%q %w", value, errNotMillisecondsOrOff)) + } + + return duration +} + +// parseMilliseconds reads a whole number of milliseconds above zero, and +// reports whether it is one. +func parseMilliseconds(value string) (time.Duration, bool) { + n, err := strconv.ParseInt(value, 10, 64) + if err != nil || n <= 0 || n > math.MaxInt64/int64(time.Millisecond) { + return 0, false + } + + return time.Duration(n) * time.Millisecond, true +} diff --git a/internal/config/rproxy_test.go b/internal/config/rproxy_test.go new file mode 100644 index 0000000..8372292 --- /dev/null +++ b/internal/config/rproxy_test.go @@ -0,0 +1,206 @@ +package config_test + +import ( + "net/netip" + "os" + "path/filepath" + "slices" + "strings" + "testing" + "time" + + "sneak.berlin/go/smallwebwaf/internal/config" +) + +// rproxy's settings, by name, and their values in these tests. +const ( + wafBackend = "SWRP_WAF_BACKEND" + rproxyTimeout = "SWRP_TIMEOUT_MS" + rproxyDefault = "SWRP_DEFAULT_DISPOSITION" + rproxyListen = "SWRP_LISTEN_ADDR" + rproxyUpstream = "SWRP_UPSTREAM_URL" + rproxyTrusted = "SWRP_TRUSTED_PROXIES" + rproxyMaxRequest = "SWRP_MAX_REQUEST_SIZE" + rproxyMaxResponse = "SWRP_MAX_RESPONSE_SIZE" + rproxyRequestTime = "SWRP_MAX_REQUEST_TIME_MS" + rproxyAnswerTime = "SWRP_MAX_RESPONSE_TIME_MS" + backendURL = "http://servewaf:8080" + rproxyDefaultAddr = ":8080" + rproxyDefaultApp = "http://127.0.0.1:8081" + notASize = "a lot" +) + +// rproxyConfig reads rproxy's settings from env. +func rproxyConfig(env map[string]string) (*config.RProxy, error) { + return config.RProxyFromEnvironment(func(name string) (string, bool) { + value, ok := env[name] + + return value, ok + }) +} + +func TestRProxyDefaults(t *testing.T) { + t.Parallel() + + cfg, err := rproxyConfig(map[string]string{wafBackend: backendURL}) + if err != nil { + t.Fatalf("settings: %v", err) + } + + if cfg.ListenAddr != rproxyDefaultAddr || + cfg.UpstreamURL.String() != rproxyDefaultApp || + cfg.WAFBackend.String() != backendURL { + t.Errorf("addresses %q, %v, %v", cfg.ListenAddr, cfg.UpstreamURL, cfg.WAFBackend) + } + + if cfg.Timeout != 2*time.Second || cfg.DefaultDisposition != "allow" { + t.Errorf("timeout %v, default disposition %q", cfg.Timeout, cfg.DefaultDisposition) + } + + if cfg.MaxRequestSize != 100<<20 || cfg.MaxResponseSize != 1<<30 { + t.Errorf("sizes %d and %d", cfg.MaxRequestSize, cfg.MaxResponseSize) + } + + if cfg.MaxRequestTime != time.Minute || cfg.MaxResponseTime != 30*time.Minute { + t.Errorf("times %v and %v", cfg.MaxRequestTime, cfg.MaxResponseTime) + } + + want := []netip.Prefix{ + netip.MustParsePrefix("10.0.0.0/8"), + netip.MustParsePrefix("172.16.0.0/12"), + netip.MustParsePrefix("192.168.0.0/16"), + } + if !slices.Equal(cfg.TrustedProxies, want) { + t.Errorf("trusted proxies %v, want %v", cfg.TrustedProxies, want) + } +} + +func TestRProxySettingsAreRead(t *testing.T) { + t.Parallel() + + cfg, err := rproxyConfig(map[string]string{ + wafBackend: "https://waf.example:8443", + rproxyListen: ":9000", + rproxyUpstream: "http://app:3000", + rproxyTrusted: "192.0.2.0/24", + rproxyTimeout: "150", + rproxyDefault: "deny", + rproxyMaxRequest: "1K", + rproxyMaxResponse: off, + rproxyRequestTime: off, + rproxyAnswerTime: "250", + }) + if err != nil { + t.Fatalf("settings: %v", err) + } + + if cfg.ListenAddr != ":9000" || cfg.UpstreamURL.Host != "app:3000" || + cfg.WAFBackend.Host != "waf.example:8443" || + cfg.TrustedProxies[0] != netip.MustParsePrefix("192.0.2.0/24") { + t.Errorf("addresses %+v", cfg) + } + + if cfg.Timeout != 150*time.Millisecond || cfg.DefaultDisposition != "deny" || + cfg.MaxRequestSize != 1024 || cfg.MaxResponseSize != 0 || + cfg.MaxRequestTime != 0 || cfg.MaxResponseTime != 250*time.Millisecond { + t.Errorf("limits %+v", cfg) + } +} + +func TestRProxyInvalidSettingsAreRefused(t *testing.T) { + t.Parallel() + + for _, tc := range []struct { + name, value string + }{ + {wafBackend, ""}, + {wafBackend, "servewaf:8080"}, + {wafBackend, "http://servewaf:8080/path"}, + {rproxyTimeout, "0"}, + {rproxyTimeout, "-1"}, + {rproxyTimeout, off}, + {rproxyTimeout, "2s"}, + {rproxyTimeout, "1.25"}, + {rproxyTimeout, "99999999999999999"}, + {rproxyDefault, "maybe"}, + {rproxyListen, "8080"}, + {rproxyUpstream, "app"}, + {rproxyTrusted, "not a netblock"}, + {rproxyMaxRequest, notASize}, + {rproxyMaxResponse, notASize}, + {rproxyRequestTime, "0"}, + {rproxyRequestTime, "1m"}, + {rproxyAnswerTime, "0"}, + } { + t.Run(tc.name+"="+tc.value, func(t *testing.T) { + t.Parallel() + + _, err := rproxyConfig(map[string]string{wafBackend: backendURL, tc.name: tc.value}) + if err == nil || !strings.HasPrefix(err.Error(), tc.name+": ") { + t.Errorf("error %v, want one naming %s", err, tc.name) + } + }) + } +} + +func TestRProxyNeedsItsBackend(t *testing.T) { + t.Parallel() + + _, err := rproxyConfig(nil) + if err == nil || !strings.HasPrefix(err.Error(), wafBackend+": is required") { + t.Errorf("error %v, want one saying %s is required", err, wafBackend) + } +} + +func TestRProxyReadsNoFile(t *testing.T) { + t.Parallel() + + path := filepath.Join(t.TempDir(), "backend") + + err := os.WriteFile(path, []byte("http://from-a-file:8080\n"), 0o600) + if err != nil { + t.Fatalf("write: %v", err) + } + + _, err = rproxyConfig(map[string]string{wafBackend + "_FILE": path}) + if err == nil || !strings.HasPrefix(err.Error(), wafBackend+": is required") { + t.Errorf("error %v, want %s still required", err, wafBackend) + } + + cfg, err := rproxyConfig(map[string]string{ + wafBackend: backendURL, + rproxyTimeout + "_FILE": "/nonexistent/timeout", + }) + if err != nil { + t.Fatalf("settings: %v, with the file left unread", err) + } + + if cfg.Timeout != 2*time.Second { + t.Errorf("timeout %v, want the default", cfg.Timeout) + } +} + +func TestRProxySettingsAreLoggedByName(t *testing.T) { + t.Parallel() + + cfg, err := rproxyConfig(map[string]string{wafBackend: backendURL}) + if err != nil { + t.Fatalf("settings: %v", err) + } + + group := cfg.LogValue().Group() + + names := make([]string, 0, len(group)) + for _, attr := range group { + names = append(names, attr.Key) + } + + for _, name := range []string{ + rproxyListen, rproxyUpstream, rproxyTrusted, wafBackend, rproxyTimeout, rproxyDefault, + rproxyMaxRequest, rproxyMaxResponse, rproxyRequestTime, rproxyAnswerTime, + } { + if !slices.Contains(names, name) { + t.Errorf("%s is not logged, only %v", name, names) + } + } +} diff --git a/internal/requestlog/requestlog.go b/internal/requestlog/requestlog.go index 6ddf8e7..a4221cf 100644 --- a/internal/requestlog/requestlog.go +++ b/internal/requestlog/requestlog.go @@ -1,6 +1,7 @@ // Package requestlog writes the lines smallwebwaf prints on stdout: one -// JSON object per request, or two from servewaf, marked "type":"request", and -// the process's own messages as JSON lines marked "type":"process". +// JSON object per request, two from servewaf and a short one from rproxy, marked +// "type":"request", and the process's own messages as JSON lines marked +// "type":"process". package requestlog import ( @@ -195,10 +196,51 @@ type Line struct { DurationUpstreamTotal *float64 `json:"duration_upstream_total,omitempty"` } +// RProxyLine is rproxy's short line in the request log about one request. The +// field names, and their order, are those of docs/request-log.md. A field that +// does not apply to a request is left out of its line. +// +//nolint:tagliatelle // docs/request-log.md names the fields in snake_case +type RProxyLine struct { + Type string `json:"type"` + Time string `json:"time"` + RequestID string `json:"request_id"` + ClientIP string `json:"client_ip"` + Host string `json:"host"` + Method string `json:"method"` + Path string `json:"path"` + Query string `json:"query"` + Status int `json:"status"` + RequestBytes int64 `json:"request_bytes"` + ResponseBytes int64 `json:"response_bytes"` + // DurationRequest runs in milliseconds from the request's headers being read + // until the app has been sent the whole request, or the answer has started + // if that is sooner. DurationResponse runs from then until the answer ends. + DurationRequest float64 `json:"duration_request"` + DurationResponse float64 `json:"duration_response"` + // Disposition is the valid disposition servewaf answered or, while + // DefaultDisposition is true, the value of SWRP_DEFAULT_DISPOSITION. It is "" + // when neither gave one. + Disposition string `json:"disposition,omitempty"` + DefaultDisposition bool `json:"default_disposition,omitempty"` +} + // Write writes line to w as one JSON line marked "type":"request". func Write(w io.Writer, line *Line) error { line.Type = "request" + return writeJSONLine(w, line) +} + +// WriteRProxyLine writes line to w as one JSON line marked "type":"request". +func WriteRProxyLine(w io.Writer, line *RProxyLine) error { + line.Type = "request" + + return writeJSONLine(w, line) +} + +// writeJSONLine writes line to w as one JSON line. +func writeJSONLine(w io.Writer, line any) error { encoded, err := json.Marshal(line) if err != nil { return fmt.Errorf("encode the request log line: %w", err) diff --git a/internal/requestlog/requestlog_test.go b/internal/requestlog/requestlog_test.go index 9824b67..3f959f8 100644 --- a/internal/requestlog/requestlog_test.go +++ b/internal/requestlog/requestlog_test.go @@ -67,6 +67,28 @@ func TestWriteWritesOneJSONLineMarkedRequest(t *testing.T) { } } +func TestWriteRProxyLineWritesOneJSONLineWithOnlyItsFields(t *testing.T) { + t.Parallel() + + var out bytes.Buffer + + err := requestlog.WriteRProxyLine(&out, &requestlog.RProxyLine{ + RequestID: "an-id", + Disposition: "allow", + }) + if err != nil { + t.Fatalf("write: %v", err) + } + + const want = `{"type":"request","time":"","request_id":"an-id","client_ip":"",` + + `"host":"","method":"","path":"","query":"","status":0,"request_bytes":0,` + + `"response_bytes":0,"duration_request":0,"duration_response":0,` + + `"disposition":"allow"}` + "\n" + if out.String() != want { + t.Errorf("wrote %s, want %s", out.String(), want) + } +} + func TestProcessLinesAreMarkedProcessAndGiveTheInstance(t *testing.T) { t.Parallel() diff --git a/internal/rproxy/bodies.go b/internal/rproxy/bodies.go new file mode 100644 index 0000000..4bd20d1 --- /dev/null +++ b/internal/rproxy/bodies.go @@ -0,0 +1,211 @@ +package rproxy + +import ( + "bufio" + "errors" + "io" + "net" + "net/http" + "sync/atomic" +) + +// errResponseTooLarge ends an app's response body that is longer than +// SWRP_MAX_RESPONSE_SIZE. +var errResponseTooLarge = errors.New( + "the response body is over SWRP_MAX_RESPONSE_SIZE") + +// requestBody is the visitor's request body. rproxy first reads up to the first +// 64 KiB of it into held, and then, on its way to the app, the transport reads +// it on a goroutine of its own. +type requestBody struct { + // body is the visitor's body, ending in an *http.MaxBytesError past + // SWRP_MAX_REQUEST_SIZE. + body io.ReadCloser + rq *request + // held is the part read before the request goes to the app, which Read gives + // first. + held []byte + // waiting is true while a Read waits for the visitor to send more. + waiting atomic.Bool + // received is true once the visitor has sent the whole body. + received atomic.Bool + // bytes is how much of the body has been read. + bytes atomic.Int64 +} + +// Read reads from the visitor's body, after the part held, which has been +// counted already. +func (b *requestBody) Read(p []byte) (int, error) { + if len(b.held) > 0 { + n := copy(p, b.held) + b.held = b.held[n:] + + return n, nil + } + + b.waiting.Store(true) + n, err := b.body.Read(p) + b.waiting.Store(false) + b.bytes.Add(int64(n)) + + var tooLarge *http.MaxBytesError + + switch { + case errors.Is(err, io.EOF): + b.received.Store(true) + case errors.As(err, &tooLarge): + b.rq.refuse(http.StatusRequestEntityTooLarge) + } + + return n, err +} + +// Close closes the visitor's body. +func (b *requestBody) Close() error { + return b.body.Close() +} + +// responseBody is the app's response body on its way to the visitor. +type responseBody struct { + // body is the app's body, ending in an *http.MaxBytesError past + // SWRP_MAX_RESPONSE_SIZE. + body io.ReadCloser + rq *request +} + +// Read reads from the app's body. +func (b *responseBody) Read(p []byte) (int, error) { + n, err := b.body.Read(p) + if err == nil { + return n, nil + } + + var tooLarge *http.MaxBytesError + + switch { + case errors.Is(err, io.EOF): + b.rq.responseReceived() + case errors.As(err, &tooLarge): + b.rq.refuse(http.StatusBadGateway) + + return n, errResponseTooLarge + case b.rq.in.Context().Err() == nil: + // The answer broke off, not because the visitor went away. If a time + // limit cut it, that refusal came first and is the one kept. + b.rq.refuse(http.StatusBadGateway) + } + + return n, err +} + +// Close closes the app's body. +func (b *responseBody) Close() error { + return b.body.Close() +} + +// upgradedConn is the connection to the app once the app has switched +// protocols, as for a WebSocket. ReverseProxy writes to it what the visitor +// sends and reads from it what the app sends, on goroutines of its own, until +// the connection closes; it counts the bytes each way, for the report. +type upgradedConn struct { + io.ReadWriteCloser + + // fromApp is how many bytes the app has sent, and toApp how many the + // visitor has. + fromApp atomic.Int64 + toApp atomic.Int64 +} + +// Read reads what the app sends. +func (c *upgradedConn) Read(p []byte) (int, error) { + n, err := c.ReadWriteCloser.Read(p) + c.fromApp.Add(int64(n)) + + return n, err +} + +// Write sends the app what the visitor sent. +func (c *upgradedConn) Write(p []byte) (int, error) { + n, err := c.ReadWriteCloser.Write(p) + c.toApp.Add(int64(n)) + + return n, err +} + +// CloseWrite tells the app that the visitor sends no more, while what the app +// sends still passes. ReverseProxy calls it once the visitor has stopped +// sending, and closes the connection there if it is not supported. +func (c *upgradedConn) CloseWrite() error { + conn, ok := c.ReadWriteCloser.(interface{ CloseWrite() error }) + if !ok { + return http.ErrNotSupported + } + + return conn.CloseWrite() +} + +// limitBody returns body, cut off with an *http.MaxBytesError after maxBytes, or +// unchanged if maxBytes is zero, which is off. +func limitBody(body io.ReadCloser, maxBytes int64) io.ReadCloser { + if maxBytes == 0 { + return body + } + + // Without a ResponseWriter, MaxBytesReader only counts and cuts off. + return http.MaxBytesReader(nil, body, maxBytes) +} + +// responseWriter is the response to the visitor. It notes the status and size +// for the log line and the report. +type responseWriter struct { + http.ResponseWriter + + // status is the final status sent, or zero before one is. + status int + bytes int64 +} + +// WriteHeader sends the status and headers. An informational 1xx status is +// passed on and the final status still comes later. +func (w *responseWriter) WriteHeader(status int) { + if status >= http.StatusOK && w.status == 0 { + w.status = status + } + + w.ResponseWriter.WriteHeader(status) +} + +// Write sends part of the body. +func (w *responseWriter) Write(p []byte) (int, error) { + if w.status == 0 { + w.status = http.StatusOK + } + + n, err := w.ResponseWriter.Write(p) //nolint:gosec // the answer, as it was given + w.bytes += int64(n) + + return n, err +} + +// FlushError sends what has been written so far. http.ResponseController calls +// it, as ReverseProxy does after each write. +func (w *responseWriter) FlushError() error { + return http.NewResponseController(w.ResponseWriter).Flush() +} + +// Hijack takes over the connection to the visitor, which ReverseProxy does only +// to pass on the app's 101 Switching Protocols: that is then the status sent. +func (w *responseWriter) Hijack() (net.Conn, *bufio.ReadWriter, error) { + conn, buffered, err := http.NewResponseController(w.ResponseWriter).Hijack() + if err == nil { + w.status = http.StatusSwitchingProtocols + } + + return conn, buffered, err +} + +// Unwrap lets http.ResponseController reach net/http's own ResponseWriter for +// what responseWriter does not do itself. +func (w *responseWriter) Unwrap() http.ResponseWriter { + return w.ResponseWriter +} diff --git a/internal/rproxy/client.go b/internal/rproxy/client.go new file mode 100644 index 0000000..a79f3e9 --- /dev/null +++ b/internal/rproxy/client.go @@ -0,0 +1,122 @@ +package rproxy + +import ( + "net/http" + "net/netip" + "slices" + "strings" +) + +// The headers rproxy sets on the requests it passes to the app. Go writes every +// header name in this form, as it sends it and as it receives it, so +// X-Client-ASN arrives as X-Client-Asn, and Del removes a visitor's own whatever +// their case; header names are not case-sensitive. +const ( + requestIDHeader = "X-Request-ID" + asnHeader = "X-Client-Asn" + countryHeader = "X-Client-Country" +) + +// peerAddress is the address of the request's TCP peer, normally traefik, without +// any IPv6 zone, which servewaf refuses. +func peerAddress(r *http.Request) netip.Addr { + addrPort, err := netip.ParseAddrPort(r.RemoteAddr) + if err != nil { + return netip.Addr{} + } + + return addrPort.Addr().Unmap().WithZone("") +} + +// clientAddress works out who the visitor is. A peer outside the trusted +// proxies is the visitor, and what it says in X-Forwarded-For is ignored. For a +// peer inside them, X-Forwarded-For is read from the right, and the first +// address outside them is the visitor; if every address in it is inside, the +// leftmost is, and with no header, the peer. An entry that is not an address +// ends the reading, since nothing to its left can be believed. The visitor's +// address never has an IPv6 zone, which servewaf refuses. +func clientAddress( + peer netip.Addr, forwardedFor []string, trusted []netip.Prefix, +) netip.Addr { + client := peer + if !isInside(peer, trusted) { + return client + } + + entries := strings.Split(strings.Join(forwardedFor, ","), ",") + for _, entry := range slices.Backward(entries) { + addr, err := netip.ParseAddr(strings.TrimSpace(entry)) + if err != nil { + break + } + + client = addr.Unmap().WithZone("") + if !isInside(client, trusted) { + break + } + } + + return client +} + +// scheme is how the visitor reached traefik, as a trusted proxy says in +// X-Forwarded-Proto, or otherwise http, the only scheme rproxy serves. +func scheme(r *http.Request, peerTrusted bool) string { + proto := r.Header.Get("X-Forwarded-Proto") + if !peerTrusted || proto == "" { + return "http" + } + + return proto +} + +// isInside reports whether addr is in one of the netblocks. +func isInside(addr netip.Addr, netblocks []netip.Prefix) bool { + return slices.ContainsFunc(netblocks, func(netblock netip.Prefix) bool { + return netblock.Contains(addr) + }) +} + +// setForwardedHeaders sets the headers in which the app learns about the +// visitor, so that it sees what it would see from traefik directly. A trusted +// proxy's forwarded headers pass on, with the proxy's own address added to +// X-Forwarded-For. Those of any other peer are its own claims and are replaced: +// X-Forwarded-For names the peer, X-Forwarded-Host the host it asked for, and +// X-Forwarded-Proto plain http, which is how it reached rproxy. +func setForwardedHeaders(in, out *http.Request, peer netip.Addr, trusted bool) { + forwardedFor := peer.String() + + if trusted { + // ReverseProxy removes these from out before Rewrite. + for _, name := range []string{"Forwarded", "X-Forwarded-Host", "X-Forwarded-Proto"} { + values, ok := in.Header[name] + if ok { + out.Header[name] = values + } + } + + prior := in.Header.Values("X-Forwarded-For") + if len(prior) > 0 { + forwardedFor = strings.Join(prior, ", ") + ", " + forwardedFor + } + + out.Header.Set("X-Forwarded-For", forwardedFor) + + return + } + + // ReverseProxy has removed Forwarded and the three set below; these are the + // other headers in which traefik tells the app about the visitor and its + // request. + for _, name := range []string{ + "X-Forwarded-Port", "X-Forwarded-Server", "X-Forwarded-Uri", + "X-Forwarded-Method", "X-Forwarded-Prefix", "X-Forwarded-Tls-Client-Cert", + "X-Forwarded-Tls-Client-Cert-Info", "X-Real-Ip", + } { + out.Header.Del(name) + } + + out.Header.Set("X-Forwarded-For", forwardedFor) + out.Header.Set("X-Forwarded-Host", in.Host) + out.Header.Set("X-Forwarded-Proto", "http") +} diff --git a/internal/rproxy/limits_test.go b/internal/rproxy/limits_test.go new file mode 100644 index 0000000..638a085 --- /dev/null +++ b/internal/rproxy/limits_test.go @@ -0,0 +1,811 @@ +package rproxy_test + +import ( + "bytes" + "io" + "log/slog" + "net" + "net/http" + "net/http/httptest" + "strconv" + "strings" + "sync/atomic" + "testing" + "time" + + "sneak.berlin/go/smallwebwaf/internal/config" + "sneak.berlin/go/smallwebwaf/internal/proxy" + "sneak.berlin/go/smallwebwaf/internal/requestlog" + "sneak.berlin/go/smallwebwaf/internal/rproxy" +) + +// post is the start of a POST whose headers end with what follows it. +const post = "POST / HTTP/1.1\r\nHost: site.example\r\n" + +// announcedLength is the header that announces a body of size bytes, and the +// end of the headers. +func announcedLength(size int) string { + return "Content-Length: " + strconv.Itoa(size) + "\r\n\r\n" +} + +func TestServerHasTheFixedLimits(t *testing.T) { + t.Parallel() + + cfg, err := config.RProxyFromEnvironment(func(name string) (string, bool) { + if name == "SWRP_WAF_BACKEND" { + return nowhere, true + } + + return "", false + }) + if err != nil { + t.Fatalf("settings: %v", err) + } + + server := rproxy.New(rproxy.Params{ + Config: cfg, + RequestLog: io.Discard, + ProcessLog: requestlog.NewProcessLogger(io.Discard, "test", slog.LevelInfo), + }) + + if server.Addr != ":8080" || server.MaxHeaderBytes != 28<<10 || + server.IdleTimeout != 2*time.Minute || server.ReadHeaderTimeout != time.Minute { + t.Errorf("server listens on %q with header limit %d, idle time %s and "+ + "header timeout %s", server.Addr, server.MaxHeaderBytes, + server.IdleTimeout, server.ReadHeaderTimeout) + } +} + +func TestABodyAnnouncedOverItsLimitIsRefusedOnceServeWAFAllowsIt(t *testing.T) { + t.Parallel() + + for _, tc := range []struct { + name, limit string + size, asked int + }{ + {"with the limit within the first part", "1K", 1025, 1024}, + {"with the limit past the first part", "128K", 129 << 10, questionBodyMaxBytes}, + } { + t.Run(tc.name, func(t *testing.T) { + t.Parallel() + + servewaf := startServeWAF(t, allow) + app := startApp(t, func(http.ResponseWriter, *http.Request) { + t.Error("the app was asked") + }) + addr, _ := startRProxy(t, servewaf.URL, app.URL, map[string]string{ + maxRequestSize: tc.limit, + }) + + got := visit(t, newRequest(t, http.MethodPost, addr, "/", + bytes.NewReader(make([]byte, tc.size)))) + if got.status != http.StatusRequestEntityTooLarge { + t.Errorf("answered %+v, want 413", got) + } + + if asked := servewaf.question(t); len(asked.body) != tc.asked { + t.Errorf("the question carries %d bytes, want %d", len(asked.body), tc.asked) + } + + report := servewaf.report(t) + if report.Status != http.StatusRequestEntityTooLarge { + t.Errorf("report %+v, want status 413", report) + } + }) + } +} + +func TestADenyOutranksTheBodySizeLimit(t *testing.T) { + t.Parallel() + + servewaf := startServeWAF(t, deny) + app := startApp(t, greet) + addr, _ := startRProxy(t, servewaf.URL, app.URL, map[string]string{ + maxRequestSize: "1K", + }) + + got := visit(t, newRequest(t, http.MethodPost, addr, "/", + bytes.NewReader(make([]byte, 1025)))) + if got.status != http.StatusForbidden { + t.Errorf("answered %+v, want servewaf's 403", got) + } +} + +func TestABodyNotAnnouncedOverItsLimitIsRefused(t *testing.T) { + t.Parallel() + + for _, tc := range []struct { + name, limit string + size int + // mayReachApp is true when the part of the body the limit lets through + // goes on to the app before the limit is met. The app may still not have + // started on the request when it is cut off. + mayReachApp bool + }{ + {"within the first part", "1K", 2 << 10, false}, + {"past the first part", "128K", 129 << 10, true}, + } { + t.Run(tc.name, func(t *testing.T) { + t.Parallel() + + var asked atomic.Bool + + servewaf := startServeWAF(t, allow) + app := startApp(t, func(w http.ResponseWriter, r *http.Request) { + asked.Store(true) + greet(w, r) + }) + addr, _ := startRProxy(t, servewaf.URL, app.URL, map[string]string{ + maxRequestSize: tc.limit, + }) + + conn := dial(t, addr) + send(t, conn, post+"Transfer-Encoding: chunked\r\n\r\n"+ + strconv.FormatInt(int64(tc.size), 16)+"\r\n"+strings.Repeat("x", tc.size)+ + "\r\n0\r\n\r\n") + + if status := readStatus(t, conn); status != http.StatusRequestEntityTooLarge { + t.Errorf("status %d, want 413", status) + } + + if asked.Load() && !tc.mayReachApp { + t.Error("the app was asked") + } + }) + } +} + +func TestAResponseOverItsLimit(t *testing.T) { + t.Parallel() + + large := strings.Repeat("x", 2<<10) + + t.Run("announced, is refused", func(t *testing.T) { + t.Parallel() + + servewaf := startServeWAF(t, allow) + app := startApp(t, func(w http.ResponseWriter, _ *http.Request) { + w.Header().Set(contentLengthHeader, strconv.Itoa(len(large))) + _, _ = io.WriteString(w, large) + }) + addr, _ := startRProxy(t, servewaf.URL, app.URL, map[string]string{ + maxResponseSize: "1K", + }) + + got := get(t, addr, "/") + if got.status != http.StatusBadGateway { + t.Errorf("answered %d, want 502", got.status) + } + + if report := servewaf.report(t); report.Status != http.StatusBadGateway { + t.Errorf("report %+v, want status 502", report) + } + }) + + t.Run("not announced, is cut off", func(t *testing.T) { + t.Parallel() + + servewaf := startServeWAF(t, allow) + app := startApp(t, func(w http.ResponseWriter, _ *http.Request) { + _, _ = io.WriteString(w, large[:1]) + _ = http.NewResponseController(w).Flush() + _, _ = io.WriteString(w, large[1:]) + }) + addr, _ := startRProxy(t, servewaf.URL, app.URL, map[string]string{ + maxResponseSize: "1K", + }) + + conn := dial(t, addr) + send(t, conn, "GET / HTTP/1.1\r\nHost: site.example\r\n\r\n") + + text, err := io.ReadAll(conn) + if err != nil || bytes.Contains(text, []byte(large)) { + t.Errorf("read %d bytes (%v), want the answer cut off", len(text), err) + } + + report := servewaf.report(t) + if report.Status != http.StatusOK || report.ResponseBytes > 1<<10 { + t.Errorf("report %+v, want status 200 and at most 1K of the answer", report) + } + }) +} + +func TestAHeadWhoseAnswerAnnouncesMoreThanTheResponseLimitIsPassedOn(t *testing.T) { + t.Parallel() + + const announced = "2048" + + servewaf := startServeWAF(t, allow) + app := startApp(t, func(w http.ResponseWriter, _ *http.Request) { + w.Header().Set(contentLengthHeader, announced) + }) + addr, _ := startRProxy(t, servewaf.URL, app.URL, map[string]string{ + maxResponseSize: "1K", + }) + + got := visit(t, newRequest(t, http.MethodHead, addr, "/", http.NoBody)) + if got.status != http.StatusOK || got.header.Get(contentLengthHeader) != announced { + t.Errorf("answered %+v, want the app's 200 announcing %s bytes", got, announced) + } +} + +func TestASettingOfOffSetsNoLimit(t *testing.T) { + t.Parallel() + + for _, setting := range []string{ + maxRequestSize, maxResponseSize, maxRequestTime, maxResponseTime, + } { + t.Run(setting, func(t *testing.T) { + t.Parallel() + + servewaf := startServeWAF(t, allow) + app := startApp(t, greet) + addr, _ := startRProxy(t, servewaf.URL, app.URL, map[string]string{ + setting: "off", + }) + + // A body and an answer that a limit of zero would cut off, as it would the + // next answer, which rproxy gives itself. + wantGreeting(t, visit(t, newRequest(t, http.MethodPost, addr, "/", + strings.NewReader("a body")))) + + if got := get(t, addr, proxy.HealthPath); got.status != http.StatusOK { + t.Errorf("answered %+v, want rproxy's own 200", got) + } + }) + } +} + +func TestAVisitorTooSlowToSendTheFirstPartIsAnswered408(t *testing.T) { + t.Parallel() + + for _, tc := range []struct { + name, rest string + }{ + {"with its length announced", announcedLength(100) + "some of it"}, + {"with no length announced", "Transfer-Encoding: chunked\r\n\r\na\r\nsome of it\r\n"}, + } { + t.Run(tc.name, func(t *testing.T) { + t.Parallel() + + servewaf := startServeWAF(t, allow) + app := startApp(t, greet) + addr, out := startRProxy(t, servewaf.URL, app.URL, map[string]string{ + maxRequestTime: "500", + }) + + conn := dial(t, addr) + send(t, conn, post+tc.rest) + + if status := readStatus(t, conn); status != http.StatusRequestTimeout { + t.Errorf("status %d, want 408", status) + } + + line := out.requestLine(t) + if line["disposition"] != nil || line["request_bytes"] != 10.0 { + t.Errorf("line %v, want no disposition and 10 bytes", line) + } + + if len(servewaf.questions) != 0 { + t.Error("servewaf was asked") + } + + servewaf.wantNoReport(t) + }) + } +} + +func TestAVisitorThatGoesAwayBeforeTheFirstPartIsNotAsked(t *testing.T) { + t.Parallel() + + servewaf := startServeWAF(t, allow) + app := startApp(t, greet) + addr, out := startRProxy(t, servewaf.URL, app.URL, nil) + + conn := dial(t, addr) + send(t, conn, post+announcedLength(100)+"some of it") + _ = conn.Close() + + line := out.requestLine(t) + if line["status"] != 0.0 || line["disposition"] != nil { + t.Errorf("line %v, want no answer and no disposition", line) + } + + if len(servewaf.questions) != 0 { + t.Error("servewaf was asked") + } +} + +func TestABodyThatCannotBeReadClosesTheConnectionWithoutAnAnswer(t *testing.T) { + t.Parallel() + + for _, tc := range []struct { + name, request string + stopsSending bool + }{ + { + "a malformed chunk", post + "Transfer-Encoding: chunked\r\n\r\nnot a chunk\r\n", + false, + }, + {"a visitor that stops sending", post + announcedLength(100) + "some of it", true}, + } { + t.Run(tc.name, func(t *testing.T) { + t.Parallel() + + servewaf := startServeWAF(t, allow) + app := startApp(t, greet) + addr, out := startRProxy(t, servewaf.URL, app.URL, nil) + + conn := dial(t, addr) + send(t, conn, tc.request) + + if tc.stopsSending { + tcpConn, ok := conn.(*net.TCPConn) + if !ok { + t.Fatalf("the connection is a %T", conn) + } + + err := tcpConn.CloseWrite() + if err != nil { + t.Fatalf("stop sending: %v", err) + } + } + + text, err := io.ReadAll(conn) + if err != nil || len(text) != 0 { + t.Errorf("read %q (%v), want the connection closed without an answer", + text, err) + } + + line := out.requestLine(t) + if line["status"] != 0.0 || line["disposition"] != nil { + t.Errorf("line %v, want no answer and no disposition", line) + } + + if len(servewaf.questions) != 0 { + t.Error("servewaf was asked") + } + }) + } +} + +// deadlineRecorder notes the time limits rproxy sets on reading the visitor's body +// and on sending its answer. +type deadlineRecorder struct { + *httptest.ResponseRecorder + + readDeadline time.Time + writeDeadline time.Time +} + +// SetReadDeadline notes deadline. +func (w *deadlineRecorder) SetReadDeadline(deadline time.Time) error { + w.readDeadline = deadline + + return nil +} + +// SetWriteDeadline notes deadline. +func (w *deadlineRecorder) SetWriteDeadline(deadline time.Time) error { + w.writeDeadline = deadline + + return nil +} + +func TestABodyLeftUnreadIsReadOnlyUntilTheRequestTimeLimit(t *testing.T) { + t.Parallel() + + const ( + limit = 5 * time.Minute + // announced is over what Go's server reads of a body left unread, so that + // the app's own server answers without waiting for the rest. + announced = 100 * questionBodyMaxBytes + ) + + servewaf := startServeWAF(t, allow) + app := startApp(t, func(w http.ResponseWriter, _ *http.Request) { + w.WriteHeader(http.StatusNoContent) + }) + _, _, server := startServer(t, servewaf.URL, app.URL, map[string]string{ + maxRequestTime: strconv.FormatInt(limit.Milliseconds(), 10), + }) + + // The visitor sends the first part and a little more, and then nothing. + stalled, stop := io.Pipe() + + t.Cleanup(func() { _ = stop.Close() }) + + req := httptest.NewRequestWithContext(t.Context(), http.MethodPost, "/", + io.MultiReader(bytes.NewReader(make([]byte, questionBodyMaxBytes+1)), stalled)) + req.ContentLength = announced + req.Header.Set(contentLengthHeader, strconv.Itoa(announced)) + + w := &deadlineRecorder{ResponseRecorder: httptest.NewRecorder()} + + before := time.Now() + + server.Handler.ServeHTTP(w, req) + + if w.Code != http.StatusNoContent { + t.Errorf("status %d, want the app's 204", w.Code) + } + + if w.readDeadline.Before(before.Add(limit)) || + w.readDeadline.After(time.Now().Add(limit)) { + t.Errorf("the reading of the body ends at %s, want %s from the request", + w.readDeadline, limit) + } +} + +func TestAnAnswerRProxyGivesItselfHasTheResponseTimeToBeSent(t *testing.T) { + t.Parallel() + + const limit = 5 * time.Minute + + for _, tc := range []struct { + name, target string + status int + }{ + {"servewaf's deny", "/", http.StatusForbidden}, + {"rproxy's own 400", "x:admin", http.StatusBadRequest}, + } { + t.Run(tc.name, func(t *testing.T) { + t.Parallel() + + servewaf := startServeWAF(t, deny) + app := startApp(t, greet) + _, _, server := startServer(t, servewaf.URL, app.URL, map[string]string{ + maxResponseTime: strconv.FormatInt(limit.Milliseconds(), 10), + }) + + req := httptest.NewRequestWithContext(t.Context(), http.MethodGet, tc.target, + http.NoBody) + w := &deadlineRecorder{ResponseRecorder: httptest.NewRecorder()} + + before := time.Now() + + server.Handler.ServeHTTP(w, req) + + if w.Code != tc.status { + t.Errorf("status %d, want %d", w.Code, tc.status) + } + + if w.writeDeadline.Before(before.Add(limit)) || + w.writeDeadline.After(time.Now().Add(limit)) { + t.Errorf("the time limit on sending ends at %s, want %s from the answer", + w.writeDeadline, limit) + } + }) + } +} + +func TestAVisitorTooSlowToSendTheRestIsAnswered408(t *testing.T) { + t.Parallel() + + servewaf := startServeWAF(t, allow) + app := startApp(t, greet) + addr, _ := startRProxy(t, servewaf.URL, app.URL, map[string]string{ + maxRequestTime: "3000", + }) + + conn := dial(t, addr) + send(t, conn, post+announcedLength(2*questionBodyMaxBytes)+ + strings.Repeat("x", questionBodyMaxBytes+1)) + + if status := readStatus(t, conn); status != http.StatusRequestTimeout { + t.Errorf("status %d, want 408", status) + } + + if asked := servewaf.question(t); len(asked.body) != questionBodyMaxBytes { + t.Errorf("the question carries %d bytes", len(asked.body)) + } + + if report := servewaf.report(t); report.Status != http.StatusRequestTimeout { + t.Errorf("report %+v, want status 408", report) + } +} + +// startAppThatTakesNothing starts an app that accepts connections and never +// reads from them, and returns its URL. +func startAppThatTakesNothing(t *testing.T) string { + t.Helper() + + listener, err := (&net.ListenConfig{}).Listen(t.Context(), "tcp", localhost+":0") + if err != nil { + t.Fatalf("listen: %v", err) + } + + stopped := make(chan struct{}) + + t.Cleanup(func() { + close(stopped) + + _ = listener.Close() + }) + + go func() { + for { + conn, err := listener.Accept() + if err != nil { + return + } + + go func() { + <-stopped + + _ = conn.Close() + }() + } + }() + + return "http://" + listener.Addr().String() +} + +func TestAnAppTooSlowToTakeTheRequestIsAnswered504(t *testing.T) { + t.Parallel() + + // More than the connections hold while nobody reads, written a little at a + // time, so that rproxy soon waits for the app. + const ( + bodySize = 64 << 20 + writeSize = 32 << 10 + ) + + servewaf := startServeWAF(t, allow) + addr, _ := startRProxy(t, servewaf.URL, startAppThatTakesNothing(t), + map[string]string{maxRequestTime: "3000"}) + + conn := dial(t, addr) + send(t, conn, post+announcedLength(bodySize)) + + go func() { + part := make([]byte, writeSize) + + for range bodySize / writeSize { + _, err := conn.Write(part) + if err != nil { + return + } + } + }() + + if status := readStatus(t, conn); status != http.StatusGatewayTimeout { + t.Errorf("status %d, want 504", status) + } +} + +func TestAnAppTooSlowToAnswerIsAnswered504(t *testing.T) { + t.Parallel() + + release := make(chan struct{}) + servewaf := startServeWAF(t, allow) + app := startApp(t, func(http.ResponseWriter, *http.Request) { + <-release + }) + + t.Cleanup(func() { close(release) }) + + addr, _ := startRProxy(t, servewaf.URL, app.URL, map[string]string{ + maxResponseTime: "300", + }) + + got := get(t, addr, "/") + if got.status != http.StatusGatewayTimeout { + t.Errorf("answered %d, want 504", got.status) + } + + if report := servewaf.report(t); report.Status != http.StatusGatewayTimeout { + t.Errorf("report %+v, want status 504", report) + } +} + +func TestAnAppTooSlowToFinishItsAnswerIsCutOff(t *testing.T) { + t.Parallel() + + release := make(chan struct{}) + servewaf := startServeWAF(t, allow) + app := startApp(t, func(w http.ResponseWriter, _ *http.Request) { + _, _ = io.WriteString(w, "the start") + _ = http.NewResponseController(w).Flush() + + <-release + }) + + t.Cleanup(func() { close(release) }) + + addr, _ := startRProxy(t, servewaf.URL, app.URL, map[string]string{ + maxResponseTime: "300", + }) + + conn := dial(t, addr) + send(t, conn, "GET / HTTP/1.1\r\nHost: site.example\r\n\r\n") + + // dial's deadline ends this read with an error if the connection stays open. + _, err := io.ReadAll(conn) + if err != nil { + t.Errorf("read: %v, want the connection closed", err) + } + + if report := servewaf.report(t); report.Status != http.StatusOK { + t.Errorf("report %+v, want the status the visitor got", report) + } +} + +func TestAVisitorTooSlowToTakeTheAnswerIsCutOff(t *testing.T) { + t.Parallel() + + // More than the connections hold while nobody reads. + const ( + bodySize = 64 << 20 + writeSize = 32 << 10 + ) + + servewaf := startServeWAF(t, allow) + app := startApp(t, func(w http.ResponseWriter, _ *http.Request) { + part := make([]byte, writeSize) + + for range bodySize / writeSize { + _, err := w.Write(part) + if err != nil { + return + } + } + }) + addr, out := startRProxy(t, servewaf.URL, app.URL, map[string]string{ + maxResponseTime: "500", + }) + + // The visitor asks, and never reads the answer. + conn := dial(t, addr) + send(t, conn, "GET / HTTP/1.1\r\nHost: site.example\r\n\r\n") + + line := out.requestLine(t) + + responseBytes, _ := line["response_bytes"].(float64) + if line["status"] != float64(http.StatusOK) || responseBytes >= bodySize { + t.Errorf("line %v, want status 200 and the answer cut off", line) + } + + report := servewaf.report(t) + if report.Status != http.StatusOK || report.ResponseBytes >= bodySize { + t.Errorf("report %+v, want status 200 and the answer cut off", report) + } +} + +func TestAnAppThatCannotBeReachedIsAnswered502WithAReport(t *testing.T) { + t.Parallel() + + servewaf := startServeWAF(t, allow) + addr, _ := startRProxy(t, servewaf.URL, nowhere, nil) + + got := get(t, addr, "/") + if got.status != http.StatusBadGateway { + t.Errorf("answered %d, want 502", got.status) + } + + if report := servewaf.report(t); report.Status != http.StatusBadGateway { + t.Errorf("report %+v, want status 502", report) + } +} + +func TestAVisitorThatStopsSendingAfterTheAppHasAnsweredGetsTheAnswer(t *testing.T) { + t.Parallel() + + servewaf := startServeWAF(t, allow) + app := startApp(t, func(w http.ResponseWriter, _ *http.Request) { + _, _ = io.WriteString(w, greeting) + }) + addr, _ := startRProxy(t, servewaf.URL, app.URL, map[string]string{ + maxRequestTime: "3000", + }) + + // Far more announced than sent, so that Go's server answers for the app + // without waiting for the rest. + conn := dial(t, addr) + send(t, conn, post+announcedLength(100*questionBodyMaxBytes)+ + strings.Repeat("x", questionBodyMaxBytes)) + + if status := readStatus(t, conn); status != http.StatusOK { + t.Errorf("status %d, want the app's 200", status) + } +} + +func TestAVisitorThatStopsSendingWhileTheAppWorksGetsNoAnswer(t *testing.T) { + t.Parallel() + + working := make(chan struct{}) + servewaf := startServeWAF(t, allow) + app := startApp(t, func(_ http.ResponseWriter, r *http.Request) { + close(working) + <-r.Context().Done() + }) + addr, out := startRProxy(t, servewaf.URL, app.URL, nil) + + conn := dial(t, addr) + send(t, conn, "GET / HTTP/1.1\r\nHost: site.example\r\n\r\n") + + select { + case <-working: + case <-time.After(waitLimit): + t.Fatal("the app was not asked") + } + + tcpConn, ok := conn.(*net.TCPConn) + if !ok { + t.Fatalf("the connection is a %T", conn) + } + + err := tcpConn.CloseWrite() + if err != nil { + t.Fatalf("stop sending: %v", err) + } + + text, err := io.ReadAll(conn) + if err != nil || len(text) != 0 { + t.Errorf("read %q (%v), want the connection closed without an answer", text, err) + } + + if line := out.requestLine(t); line["status"] != 0.0 { + t.Errorf("line %v, want status 0", line) + } +} + +func TestTheRequestTimeLimitDoesNotCutAnAnswerOnceTheAppHasTheRequest(t *testing.T) { + t.Parallel() + + const answer = "the end of a long answer" + + servewaf := startServeWAF(t, allow) + app := startApp(t, func(w http.ResponseWriter, r *http.Request) { + _, _ = io.Copy(io.Discard, r.Body) + _ = http.NewResponseController(w).Flush() + + time.Sleep(2100 * time.Millisecond) + + _, _ = io.WriteString(w, answer) + }) + addr, _ := startRProxy(t, servewaf.URL, app.URL, map[string]string{ + maxRequestTime: "2000", + }) + + got := visit(t, newRequest(t, http.MethodPost, addr, "/", strings.NewReader("a body"))) + if got.status != http.StatusOK || got.body != answer { + t.Errorf("answered %d %q, want the app's whole answer", got.status, got.body) + } +} + +func TestAVisitorThatStopsSendingGetsAnAnswerRProxyGivesByTheDeadline(t *testing.T) { + t.Parallel() + + for _, tc := range []struct { + name, start string + status int + }{ + {"a deny", post, http.StatusForbidden}, + { + "the health check", + "GET " + proxy.HealthPath + " HTTP/1.1\r\nHost: site.example\r\n", + http.StatusOK, + }, + } { + t.Run(tc.name, func(t *testing.T) { + t.Parallel() + + servewaf := startServeWAF(t, deny) + app := startApp(t, greet) + addr, _ := startRProxy(t, servewaf.URL, app.URL, map[string]string{ + maxRequestTime: "500", + }) + + // Go's server waits for a little of the body left before it sends an answer, + // and the time for sending the request ends that wait. + conn := dial(t, addr) + send(t, conn, tc.start+announcedLength(questionBodyMaxBytes+100)+ + strings.Repeat("x", questionBodyMaxBytes)) + + if status := readStatus(t, conn); status != tc.status { + t.Errorf("status %d, want %d", status, tc.status) + } + }) + } +} diff --git a/internal/rproxy/question_test.go b/internal/rproxy/question_test.go new file mode 100644 index 0000000..0005f2a --- /dev/null +++ b/internal/rproxy/question_test.go @@ -0,0 +1,477 @@ +package rproxy_test + +import ( + "bytes" + "crypto/rand" + "io" + "maps" + "net/http" + "net/http/httptest" + "slices" + "strconv" + "strings" + "testing" + + "sneak.berlin/go/smallwebwaf/internal/proxy" +) + +// questionBodyMaxBytes is the most of a body a question carries. +const questionBodyMaxBytes = proxy.QuestionBodyMaxBytes + +// chunked is the encoding of a body whose length the visitor did not announce. +const chunked = "chunked" + +// pathGoReencodes is a path Go's client would decode and escape again, %2F turned +// into / and | into %7C, unless told to send it as it is. A test sets it as the +// visitor's URL.Opaque, so that the visitor sends it unchanged too. +const pathGoReencodes = "/some%2Fpath/with%20space|pipe" + +func TestQuestionIsTheVisitorsRequestWithRProxysHeaders(t *testing.T) { + t.Parallel() + + servewaf := startServeWAF(t, allow) + app := startApp(t, greet) + addr, _ := startRProxy(t, servewaf.URL, app.URL, nil) + + const target = pathGoReencodes + "?b=2&a=1&bad=%zz;x" + + req := newRequest(t, http.MethodDelete, addr, target, http.NoBody) + req.URL.Opaque = pathGoReencodes + req.Host = site + req.Header.Set("X-Test", "from the visitor") + req.Header.Set(forwardedForHeader, "198.51.100.1") + req.Header.Set(forwardedProtoHeader, "gopher") + req.Header.Set(requestIDHeader, "the-visitor's") + req.Header.Set(proxy.ClientHeader, "198.51.100.2") + req.Header.Set(proxy.PeerHeader, "198.51.100.3") + req.Header.Set(proxy.ContentLengthHeader, "12345") + // Go's client then sends no User-Agent, so that one in the question would be + // Go's own. + req.Header["User-Agent"] = nil + + wantGreeting(t, visit(t, req)) + + asked := servewaf.question(t) + if asked.method != http.MethodDelete || asked.target != target || + asked.host != site { + t.Errorf("asked about %s %s at %s", asked.method, asked.target, asked.host) + } + + if asked.header.Get(requestIDHeader) == "the-visitor's" { + t.Error("the question carries the visitor's id") + } + + // Exactly these: visit's connection has the visitor send Connection: close. + want := http.Header{} + want.Set("Connection", "close") + want.Set("X-Test", "from the visitor") + want.Set(forwardedForHeader, "198.51.100.1") + want.Set(forwardedProtoHeader, "http") + want.Set(requestIDHeader, asked.header.Get(requestIDHeader)) + want.Set(proxy.ClientHeader, localhost) + want.Set(proxy.PeerHeader, localhost) + + if !maps.EqualFunc(asked.header, want, slices.Equal) { + t.Errorf("the question's headers are %v, want %v", asked.header, want) + } +} + +func TestQuestionNamesTheVisitorATrustedProxySpeaksFor(t *testing.T) { + t.Parallel() + + const visitor = "203.0.113.9" + + for _, tc := range []struct { + name, trusted, client, proto, forwarded string + }{ + { + "from a trusted proxy", localhost + "/32", visitor, "https", + visitor + ", " + localhost, + }, + {"from a peer not trusted", "10.0.0.0/8", localhost, "http", localhost}, + } { + t.Run(tc.name, func(t *testing.T) { + t.Parallel() + + servewaf := startServeWAF(t, allow) + forwarded := make(chan http.Header, 1) + app := startApp(t, func(w http.ResponseWriter, r *http.Request) { + forwarded <- r.Header.Clone() + + greet(w, r) + }) + addr, out := startRProxy(t, servewaf.URL, app.URL, map[string]string{ + trustedProxies: tc.trusted, + }) + + req := newRequest(t, http.MethodGet, addr, "/", http.NoBody) + req.Header.Set(forwardedForHeader, visitor) + req.Header.Set(forwardedProtoHeader, "https") + wantGreeting(t, visit(t, req)) + + asked := servewaf.question(t) + if asked.header.Get(proxy.ClientHeader) != tc.client || + asked.header.Get(proxy.PeerHeader) != localhost || + asked.header.Get(forwardedForHeader) != visitor || + asked.header.Get(forwardedProtoHeader) != tc.proto { + t.Errorf("the question's headers are %v", asked.header) + } + + if line := out.requestLine(t); line["client_ip"] != tc.client { + t.Errorf("line %v, want client_ip %s", line, tc.client) + } + + if got := (<-forwarded).Get(forwardedForHeader); got != tc.forwarded { + t.Errorf("the app saw %s %q, want %q", forwardedForHeader, got, tc.forwarded) + } + }) + } +} + +// randomBytes returns n random bytes. +func randomBytes(n int) []byte { + body := make([]byte, n) + _, _ = rand.Read(body) // it never returns an error + + return body +} + +func TestFirst64KiBReachServeWAFAndTheWholeBodyTheApp(t *testing.T) { + t.Parallel() + + for _, size := range []int{ + 1, questionBodyMaxBytes - 1, questionBodyMaxBytes, questionBodyMaxBytes + 1, + 5 * questionBodyMaxBytes, + } { + for _, announced := range []bool{true, false} { + name := strconv.Itoa(size) + " bytes, length announced: " + + strconv.FormatBool(announced) + + t.Run(name, func(t *testing.T) { + t.Parallel() + + body := randomBytes(size) + servewaf := startServeWAF(t, allow) + received := make(chan []byte, 1) + app := startApp(t, func(w http.ResponseWriter, r *http.Request) { + whole, _ := io.ReadAll(r.Body) + received <- whole + + greet(w, r) + }) + addr, _ := startRProxy(t, servewaf.URL, app.URL, nil) + + req := newRequest(t, http.MethodPost, addr, "/", bytes.NewReader(body)) + if !announced { + // Go's client sends a body of unknown length in chunks. + req.ContentLength = -1 + } + + wantGreeting(t, visit(t, req)) + + asked := servewaf.question(t) + want := body[:min(size, questionBodyMaxBytes)] + wantQuestionBody(t, asked, want, size, announced) + + if !bytes.Equal(<-received, body) { + t.Error("the app did not get the whole body") + } + + if report := servewaf.report(t); report.RequestBytes != int64(size) { + t.Errorf("report %+v, of %d bytes", report, size) + } + }) + } + } +} + +// wantQuestionBody checks the body a question carries, want, and how it gives +// the length of the visitor's body, of size bytes. +func wantQuestionBody( + t *testing.T, asked question, want []byte, size int, announced bool, +) { + t.Helper() + + if !bytes.Equal(asked.body, want) { + t.Errorf("the question carries %d bytes, want the first %d", len(asked.body), + len(want)) + } + + values := asked.header.Values(proxy.ContentLengthHeader) + + if announced { + if !slices.Equal(values, []string{strconv.Itoa(size)}) || + asked.contentLength != int64(len(want)) { + t.Errorf("announced %v and length %d, want %d and %d", values, + asked.contentLength, size, len(want)) + } + + return + } + + if len(values) != 0 || !slices.Equal(asked.transferEncoding, []string{chunked}) { + t.Errorf("announced %v and encoded %v, want none and chunked", values, + asked.transferEncoding) + } +} + +func TestDenyNeedsOnlyTheFirstPartOfALargeBody(t *testing.T) { + t.Parallel() + + servewaf := startServeWAF(t, deny) + app := startApp(t, greet) + addr, _ := startRProxy(t, servewaf.URL, app.URL, nil) + + // Far more announced than sent. Go's server does not wait for that much, so + // the answer comes at once. + conn := dial(t, addr) + send(t, conn, post+announcedLength(100*questionBodyMaxBytes)+ + strings.Repeat("x", questionBodyMaxBytes)) + + if status := readStatus(t, conn); status != http.StatusForbidden { + t.Errorf("status %d, want servewaf's 403", status) + } + + if report := servewaf.report(t); report.RequestBytes != questionBodyMaxBytes { + t.Errorf("report %+v, want the first part's bytes", report) + } +} + +func TestAQuestionCarriesItsBodyWhateverTheMethod(t *testing.T) { + t.Parallel() + + for _, tc := range []struct { + name, framing, body string + encoding []string + }{ + { + chunked, "Transfer-Encoding: chunked\r\n\r\n4\r\nbody\r\n0\r\n\r\n", "body", + []string{chunked}, + }, + { + "chunked and empty", "Transfer-Encoding: chunked\r\n\r\n0\r\n\r\n", "", + []string{chunked}, + }, + {"with its length", announcedLength(4) + "body", "body", nil}, + } { + for _, method := range []string{ + http.MethodGet, http.MethodDelete, http.MethodPost, + } { + t.Run(method+" "+tc.name, func(t *testing.T) { + t.Parallel() + + servewaf := startServeWAF(t, deny) + addr, _ := startRProxy(t, servewaf.URL, nowhere, nil) + + conn := dial(t, addr) + send(t, conn, method+" /x HTTP/1.1\r\nHost: site.example\r\n"+tc.framing) + + if status := readStatus(t, conn); status != http.StatusForbidden { + t.Errorf("status %d, want servewaf's 403", status) + } + + asked := servewaf.question(t) + if string(asked.body) != tc.body || + !slices.Equal(asked.transferEncoding, tc.encoding) { + t.Errorf("the question carries %q, encoded %v, want %q, encoded %v", + asked.body, asked.transferEncoding, tc.body, tc.encoding) + } + }) + } + } +} + +func TestAMethodNotPassedOnIsAnswered405(t *testing.T) { + t.Parallel() + + for _, method := range []string{http.MethodTrace, "PROPFIND", "LOCK", "BREW"} { + t.Run(method, func(t *testing.T) { + t.Parallel() + + servewaf := startServeWAF(t, allow) + app := startApp(t, func(http.ResponseWriter, *http.Request) { + t.Error("the app was asked") + }) + addr, out := startRProxy(t, servewaf.URL, app.URL, nil) + + got := visit(t, newRequest(t, method, addr, "/", http.NoBody)) + if got.status != http.StatusMethodNotAllowed || + got.header.Get("Allow") != "GET, HEAD, POST, OPTIONS, PUT, DELETE, PATCH, QUERY" { + t.Errorf("answered %+v, want 405 naming the methods passed on", got) + } + + line := out.requestLine(t) + if line["status"] != float64(http.StatusMethodNotAllowed) || + line["disposition"] != nil || line["default_disposition"] != nil { + t.Errorf("line %v, want status 405 and no disposition", line) + } + + if len(servewaf.questions) != 0 { + t.Error("servewaf was asked") + } + + servewaf.wantNoReport(t) + }) + } +} + +func TestEachMethodPassedOnReachesServeWAFAndTheApp(t *testing.T) { + t.Parallel() + + for _, method := range []string{ + http.MethodGet, http.MethodHead, http.MethodPost, http.MethodOptions, + http.MethodPut, http.MethodDelete, http.MethodPatch, "QUERY", + } { + t.Run(method, func(t *testing.T) { + t.Parallel() + + reached := make(chan string, 1) + servewaf := startServeWAF(t, allow) + app := startApp(t, func(_ http.ResponseWriter, r *http.Request) { + reached <- r.Method + }) + addr, _ := startRProxy(t, servewaf.URL, app.URL, nil) + + got := visit(t, newRequest(t, method, addr, "/", http.NoBody)) + if got.status != http.StatusOK { + t.Errorf("answered %+v, want the app's 200", got) + } + + if asked := servewaf.question(t); asked.method != method { + t.Errorf("servewaf was asked about %s", asked.method) + } + + if atApp := <-reached; atApp != method { + t.Errorf("the app was asked %s", atApp) + } + }) + } +} + +func TestARequestThatCannotGoToServeWAFAsSentIsRefused(t *testing.T) { + t.Parallel() + + const host = "Host: site.example\r\n\r\n" + + for name, request := range map[string]string{ + "a target of a scheme": "GET x:admin HTTP/1.1\r\n" + host, + "a target of a URL": "GET http://site.example/ HTTP/1.1\r\n" + host, + "a host Go will not send": "GET http://a 0 && res.Body != http.NoBody && res.ContentLength > maxBytes { + rq.refuse(http.StatusBadGateway) + + return errResponseTooLarge + } + + // Go's server adds a Content-Type it guesses from the body to a response + // whose headers have no Content-Type key. The key with no value sends none, + // as the app sent none. + if res.Header.Get("Content-Type") == "" { + rq.out.Header()["Content-Type"] = nil + } + + res.Body = &responseBody{body: limitBody(res.Body, maxBytes), rq: rq} + rq.setResponseDeadline() + + return nil +} + +// answerError is ReverseProxy's ErrorHandler: the request could not be passed to +// the app, or the app's answer cannot be passed on. +func (rq *request) answerError(_ http.ResponseWriter, _ *http.Request, err error) { + if rq.upgraded != nil { + // The app switched protocols, but the switch was not passed on, and + // ReverseProxy leaves the app's connection open when it refuses it. + _ = rq.upgraded.Close() + } + + status := int(rq.refused.Load()) + if status == 0 { + if rq.in.Context().Err() != nil { + // The visitor went away or stopped its sending side. This panic has Go's + // server close the connection without an answer, and log nothing; the + // deferred finish still runs. + panic(http.ErrAbortHandler) + } + + rq.h.processLog.Warn("request to the app failed", "error", err.Error()) + + status = http.StatusBadGateway + } + + rq.answer(status) +} + +// answer sends rproxy's own answer, unless the response has already started. +func (rq *request) answer(status int) { + rq.refused.CompareAndSwap(0, int64(status)) + + if rq.out.status != 0 { + return // too late to answer: the connection can only be cut + } + + // A visitor found too slow is read no more; any other may go on sending until + // its time is up, so that Go's server can read the rest of the body and end + // the request cleanly. + deadline := rq.requestDeadline() + if status == http.StatusRequestTimeout { + deadline = time.Now() + } + + rq.prepareOwnAnswer(deadline) + + header := rq.out.Header() + header.Set("Content-Type", plainText) + header.Set("X-Content-Type-Options", "nosniff") + rq.send(status, []byte(http.StatusText(status)+"\n")) +} + +// send gives the visitor an answer rproxy holds whole: status, the headers set so +// far and body. Go's server would otherwise send so short an answer only once the +// handler has returned, after finish has written the log line and sent the report. +// Without Content-Length, the flush would send the body in chunks, the last of +// which would still wait for the handler. A HEAD's answer has no body, so any +// Content-Length it has stays as servewaf gave it. +func (rq *request) send(status int, body []byte) { + if rq.in.Method != http.MethodHead { + rq.out.Header().Set("Content-Length", strconv.Itoa(len(body))) + } + + rq.out.WriteHeader(status) + _, _ = rq.out.Write(body) + _ = rq.rc.Flush() +} + +// prepareOwnAnswer readies the connection for an answer rproxy gives itself: the +// reading of a body that has not arrived whole ends at deadline, and the answer +// has SWRP_MAX_RESPONSE_TIME_MS from now to reach the visitor, even when the app +// used that time up first. +func (rq *request) prepareOwnAnswer(deadline time.Time) { + rq.stopReadingBody(deadline) + rq.endRequest() + + limit := rq.h.config.MaxResponseTime + if limit > 0 { + _ = rq.rc.SetWriteDeadline(time.Now().Add(limit)) + } +} + +// refuse records status, unless an earlier refusal was, and ends the request to +// the app, if one was made. +func (rq *request) refuse(status int) { + rq.refused.CompareAndSwap(0, int64(status)) + + if rq.cancel != nil { + rq.cancel() + } +} + +// finish ends the request's time limits, writes its log line and, if servewaf +// answered the question, sends the report. ctx is the request's own context. +func (rq *request) finish(ctx context.Context) { + rq.stopTimers() + + if rq.refused.Load() == 0 { + rq.stopReadingBody(rq.requestDeadline()) + } + + requestEnd := rq.endRequest() + end := time.Now() + requestBytes, responseBytes := rq.bytesEachWay() + + err := requestlog.WriteRProxyLine(rq.h.requestLog, &requestlog.RProxyLine{ + Time: requestlog.FormatTime(rq.start), + RequestID: rq.id, + ClientIP: rq.client.String(), + Host: rq.in.Host, + Method: rq.in.Method, + Path: rules.Path(rq.in), + Query: rq.in.URL.RawQuery, + Status: rq.out.status, + RequestBytes: requestBytes, + ResponseBytes: responseBytes, + DurationRequest: requestlog.Milliseconds(requestEnd.Sub(rq.start)), + DurationResponse: requestlog.Milliseconds(end.Sub(requestEnd)), + Disposition: rq.disposition, + DefaultDisposition: rq.byDefault, + }) + if err != nil { + rq.h.processLog.Error("writing the request log failed", "error", err.Error()) + } + + if !rq.answered { + return + } + + report := &proxy.Report{ + RequestID: rq.id, + Status: min(rq.out.status, reportStatusMax), + RequestBytes: requestBytes, + ResponseBytes: responseBytes, + DurationTotal: requestlog.Milliseconds(end.Sub(rq.start)), + } + if !rq.upstreamStart.IsZero() { + report.DurationUpstreamTotal = requestlog.Milliseconds(end.Sub(rq.upstreamStart)) + } + + rq.h.sendReport(ctx, report) +} + +// bytesEachWay returns the body bytes of the request and of the answer. For an +// upgraded connection, such as a WebSocket, which has closed by then, what it +// carried from the visitor counts with the request's and what it carried from the +// app with the answer's. +func (rq *request) bytesEachWay() (int64, int64) { + var requestBytes int64 + if rq.body != nil { + requestBytes = rq.body.bytes.Load() + } + + responseBytes := rq.out.bytes + + if rq.upgraded != nil { + requestBytes += rq.upgraded.toApp.Load() + responseBytes += rq.upgraded.fromApp.Load() + } + + return requestBytes, responseBytes +} + +// requestDeadline is when the visitor must have sent its whole request, or zero +// when SWRP_MAX_REQUEST_TIME_MS is off. +func (rq *request) requestDeadline() time.Time { + limit := rq.h.config.MaxRequestTime + if limit == 0 { + return time.Time{} + } + + return rq.start.Add(limit) +} + +// stopReadingBody ends, at deadline, the reading of a visitor's body that has not +// arrived whole: Go's server then reads no more of it, and closes the connection +// after the answer. +func (rq *request) stopReadingBody(deadline time.Time) { + if rq.body == nil || rq.body.received.Load() { + return + } + + _ = rq.rc.SetReadDeadline(deadline) +} + +// startRequestTimer starts SWRP_MAX_REQUEST_TIME_MS, which runs from the +// request's headers being read until the app has been sent the whole request. +func (rq *request) startRequestTimer() { + deadline := rq.requestDeadline() + if deadline.IsZero() { + return + } + + rq.mu.Lock() + defer rq.mu.Unlock() + + rq.requestTimer = time.AfterFunc(time.Until(deadline), rq.requestTimedOut) +} + +// requestTimedOut is called when SWRP_MAX_REQUEST_TIME_MS runs out while the +// request is still on its way to the app. The answer names the side rproxy was +// waiting on at that moment: 408 when it was waiting for the visitor to send more +// of its body, 504 when it was waiting for the app to be reached or to take what +// it had. +func (rq *request) requestTimedOut() { + rq.mu.Lock() + defer rq.mu.Unlock() + + if rq.timersStopped { + return + } + + if rq.body == nil || !rq.body.waiting.Load() { + // Once the app has answered, the transport, ReverseProxy and Go's server + // each close the visitor's body, and closing it reads what is left of a + // body under 256 KiB without requestBody.Read, so waiting is false while + // that read waits for the visitor. That read ends now, before refuse: + // should rproxy still answer itself, answer then sets the deadline it + // gives its own answers. + if !rq.answerStarted.IsZero() { + rq.stopReadingBody(time.Now()) + } + + rq.refuse(http.StatusGatewayTimeout) + + return + } + + rq.refuse(http.StatusRequestTimeout) + // The transport gives up on the app only once its Read of the visitor's body + // returns, so that Read is ended now. The lock keeps this from reaching the + // connection after the request is handled. + _ = rq.rc.SetReadDeadline(time.Now()) +} + +// gotFirstResponseByte is called once the first byte of the app's answer has +// arrived. +func (rq *request) gotFirstResponseByte() { + rq.mu.Lock() + defer rq.mu.Unlock() + + rq.answerStarted = time.Now() +} + +// wroteRequest is called once the app has been sent the whole request: +// SWRP_MAX_REQUEST_TIME_MS ends and SWRP_MAX_RESPONSE_TIME_MS starts. +func (rq *request) wroteRequest(info httptrace.WroteRequestInfo) { + if info.Err != nil { + return // the transport gives up, or tries again + } + + requestEnd := rq.endRequest() + + rq.mu.Lock() + defer rq.mu.Unlock() + + if rq.timersStopped { + return + } + + stopTimer(rq.requestTimer) + + limit := rq.h.config.MaxResponseTime + if limit > 0 { + rq.responseTimer = time.AfterFunc(time.Until(requestEnd.Add(limit)), + rq.responseTimedOut) + } +} + +// responseTimedOut is called when SWRP_MAX_RESPONSE_TIME_MS runs out before the +// app has sent its whole answer. +func (rq *request) responseTimedOut() { + rq.mu.Lock() + defer rq.mu.Unlock() + + if !rq.timersStopped { + rq.refuse(http.StatusGatewayTimeout) + } +} + +// responseReceived is called once the app has sent its whole answer. An app may +// answer before the visitor has sent its whole body, as with a 404 it sends +// without reading it. The time limits then run on: until the transport's read of +// the body returns, Go's server cannot send the answer, and +// SWRP_MAX_REQUEST_TIME_MS is what ends that read for a visitor that stops +// sending. +func (rq *request) responseReceived() { + if rq.body != nil && !rq.body.received.Load() { + return + } + + rq.stopTimers() +} + +// setResponseDeadline sets SWRP_MAX_RESPONSE_TIME_MS on the connection to the +// visitor: the app's answer must reach the visitor within it of the end of the +// request. +func (rq *request) setResponseDeadline() { + requestEnd := rq.endRequest() + + limit := rq.h.config.MaxResponseTime + if limit > 0 { + _ = rq.rc.SetWriteDeadline(requestEnd.Add(limit)) + } +} + +// endRequest notes now as the end of the request, unless an end is noted, and +// returns the end noted. +func (rq *request) endRequest() time.Time { + rq.mu.Lock() + defer rq.mu.Unlock() + + if rq.requestEnd.IsZero() { + rq.requestEnd = time.Now() + } + + return rq.requestEnd +} + +// stopTimers stops the request's time limits and keeps any from starting later: +// the app's answer is complete, the connection upgraded, or the request handled. +func (rq *request) stopTimers() { + rq.mu.Lock() + defer rq.mu.Unlock() + + rq.timersStopped = true + stopTimer(rq.requestTimer) + stopTimer(rq.responseTimer) +} + +// stopTimer stops t, which is nil when its time limit is off. +func stopTimer(t *time.Timer) { + if t != nil { + t.Stop() + } +} diff --git a/internal/rproxy/rproxy.go b/internal/rproxy/rproxy.go new file mode 100644 index 0000000..e662104 --- /dev/null +++ b/internal/rproxy/rproxy.go @@ -0,0 +1,228 @@ +// Package rproxy is the reverse proxy in front of one app that asks servewaf +// about each request. It passes a request servewaf allows to the app and the +// app's answer back, within the size and time limits. Any other answer from +// servewaf goes to the visitor as servewaf gave it. SWRP_DEFAULT_DISPOSITION +// decides only when servewaf cannot be connected to, does not answer at all, or +// does not answer within SWRP_TIMEOUT_MS. rproxy itself refuses a request with a +// method it does not pass on, or one it cannot put to servewaf as the visitor +// sent it. It keeps nothing between requests and never touches disk. "rproxy" in +// SPEC.md says what it does. +package rproxy + +import ( + "context" + "io" + "log" + "log/slog" + "net/http" + "net/url" + "slices" + "strings" + "sync" + "time" + + "sneak.berlin/go/smallwebwaf/internal/config" + "sneak.berlin/go/smallwebwaf/internal/proxy" +) + +// How rproxy keeps connections to the app and to servewaf open between +// requests. +const ( + idleConns = 100 + idleConnTimeout = 90 * time.Second +) + +// The limits on a connection from a visitor that have no setting. +const ( + // requestHeaderMaxBytes is the largest request line and headers a visitor + // may send. + requestHeaderMaxBytes = 32 << 10 + // goServerExtraHeaderBytes is how far past MaxHeaderBytes Go's server reads + // before it refuses. + goServerExtraHeaderBytes = 4 << 10 + // idleTimeout is how long a kept-open connection may wait for its next + // request. + idleTimeout = 120 * time.Second +) + +// adminPrefix starts the path of every request for smallwebwaf itself, which +// never reaches the app. +const adminPrefix = "/_smallwebwaf/" + +// allowedMethods lists the methods rproxy passes on, as an Allow header gives +// them. It answers any other method 405 itself. +const allowedMethods = "GET, HEAD, POST, OPTIONS, PUT, DELETE, PATCH, QUERY" + +// Params are what New needs. +type Params struct { + Config *config.RProxy + // RequestLog receives one short JSON line per request. + RequestLog io.Writer + // ProcessLog receives the process's own messages. + ProcessLog *slog.Logger +} + +// Server is the server `smallwebwaf rproxy` runs. +type Server struct { + *http.Server + + h *handler +} + +// New returns the server `smallwebwaf rproxy` runs. Go's server itself refuses +// a request line and headers over requestHeaderMaxBytes, with 431, closes a +// connection idle for idleTimeout, and applies SWRP_MAX_REQUEST_TIME_MS while +// the headers arrive. The handler applies the time and size limits from then +// on. +func New(params Params) *Server { + backend := params.Config.WAFBackend + errorLog := slog.NewLogLogger(params.ProcessLog.Handler(), slog.LevelWarn) + h := &handler{ + config: params.Config, + requestLog: params.RequestLog, + processLog: params.ProcessLog, + errorLog: errorLog, + transport: newTransport(), + serveWAFTransport: newServeWAFTransport(), + reportURL: (&url.URL{ + Scheme: backend.Scheme, Host: backend.Host, Path: proxy.ReportPath, + }).String(), + } + + return &Server{ + Server: &http.Server{ + Addr: params.Config.ListenAddr, + Handler: h, + ErrorLog: errorLog, + ReadHeaderTimeout: params.Config.MaxRequestTime, + IdleTimeout: idleTimeout, + MaxHeaderBytes: requestHeaderMaxBytes - goServerExtraHeaderBytes, + }, + h: h, + } +} + +// WaitForReports returns once the reports being sent have gone or failed, +// which SWRP_TIMEOUT_MS bounds. Any report after this call is dropped: that of +// a WebSocket that closes while rproxy stops, for one. +func (s *Server) WaitForReports() { + s.h.reportsMu.Lock() + s.h.reportsStopped = true + s.h.reportsMu.Unlock() + + s.h.reports.Wait() +} + +// handler is the proxy. It holds what every request shares; what belongs to +// one request is in a request. +type handler struct { + config *config.RProxy + requestLog io.Writer + processLog *slog.Logger + errorLog *log.Logger + // transport carries requests to the app, and serveWAFTransport questions and + // reports to servewaf. + transport http.RoundTripper + serveWAFTransport http.RoundTripper + // reportURL is where the reports go. + reportURL string + // reports waits for the reports being sent. reportsMu guards starting one + // against reportsStopped, which is set once nothing is to wait for another. + reports sync.WaitGroup + reportsMu sync.Mutex + reportsStopped bool +} + +// newTransport returns what carries requests to the app. It never goes through +// a proxy named in the environment, and leaves answers compressed or not as they +// were sent. +func newTransport() *http.Transport { + return &http.Transport{ + MaxIdleConns: idleConns, + MaxIdleConnsPerHost: idleConns, + IdleConnTimeout: idleConnTimeout, + DisableCompression: true, + } +} + +// newServeWAFTransport returns what carries questions and reports to servewaf. +// It speaks HTTP/1.1 only. Go's HTTP/2 client refuses to send a visitor's Upgrade +// header, or a Connection header other than close or keep-alive, and that would +// read as servewaf giving no answer. +func newServeWAFTransport() *http.Transport { + transport := newTransport() + transport.Protocols = new(http.Protocols) + transport.Protocols.SetHTTP1(true) + + return transport +} + +// ServeHTTP handles one request: it answers the health check itself, refuses a +// request with a method it does not pass on or one it cannot put to servewaf as +// the visitor sent it, and for any other request reads the first part of the +// body, asks servewaf, and does what the answer says. Once the answer has ended, +// it writes the request's log line and sends its report. +func (h *handler) ServeHTTP(w http.ResponseWriter, r *http.Request) { + rq := h.newRequest(w, r) + defer rq.finish(r.Context()) + + if r.Method == http.MethodGet && r.URL.Path == proxy.HealthPath { + rq.answerHealth() + + return + } + + if !slices.Contains(strings.Split(allowedMethods, ", "), r.Method) { + rq.out.Header().Set("Allow", allowedMethods) + rq.answer(http.StatusMethodNotAllowed) + + return + } + + if !canBeAsked(r) { + rq.answer(http.StatusBadRequest) + + return + } + + if !rq.readFirstPartOfBody() { + return + } + + answer, err := rq.ask(r.Context()) + if err != nil { + rq.handleNoUsableAnswer(r.Context(), err) + + return + } + + rq.noteAnswer(answer) + + switch rq.disposition { + case proxy.DispositionAllow: + rq.passToApp(r.Context()) + case proxy.DispositionClose: + // This panic has Go's server close the connection without an answer, + // and log nothing; the deferred finish still runs. + panic(http.ErrAbortHandler) + default: // a deny, an error, or an answer without a valid disposition + rq.answerWith(answer) + } +} + +// passToApp passes the request on to the app, unless a limit or its path keeps +// it from the app. servewaf answers a path under adminPrefix itself, so such a +// request gets here only when SWRP_DEFAULT_DISPOSITION allowed it. ctx is the +// request's own context. +func (rq *request) passToApp(ctx context.Context) { + maxBytes := rq.h.config.MaxRequestSize + + switch { + case rq.refused.Load() != 0 || (maxBytes > 0 && rq.in.ContentLength > maxBytes): + rq.answer(http.StatusRequestEntityTooLarge) + case strings.HasPrefix(rq.in.URL.Path, adminPrefix): + rq.answer(http.StatusNotFound) + default: + rq.forward(ctx) + } +} diff --git a/internal/rproxy/rproxy_test.go b/internal/rproxy/rproxy_test.go new file mode 100644 index 0000000..2e65e00 --- /dev/null +++ b/internal/rproxy/rproxy_test.go @@ -0,0 +1,1758 @@ +package rproxy_test + +import ( + "bufio" + "bytes" + "encoding/json" + "errors" + "io" + "log/slog" + "maps" + "net" + "net/http" + "net/http/httptest" + "slices" + "strconv" + "strings" + "sync" + "testing" + "time" + + "sneak.berlin/go/smallwebwaf/internal/config" + "sneak.berlin/go/smallwebwaf/internal/proxy" + "sneak.berlin/go/smallwebwaf/internal/requestlog" + "sneak.berlin/go/smallwebwaf/internal/rproxy" +) + +const ( + // waitLimit bounds how long a test waits for what should happen. + waitLimit = 10 * time.Second + // pollInterval is how often a test looks for a log line. + pollInterval = 10 * time.Millisecond + // localhost is where every test server listens, and so the address rproxy + // sees each test's requests come from. + localhost = "127.0.0.1" + // site is the host the tests' requests ask for. + site = "site.example" + // greeting is what the tests' app answers by default. + greeting = "hello from the app" + // nowhere is a URL nothing listens at: port 9 is the discard service, which + // nothing on a test host runs. + nowhere = "http://127.0.0.1:9" +) + +// The settings the tests set. +const ( + timeoutSetting = "SWRP_TIMEOUT_MS" + defaultDisposition = "SWRP_DEFAULT_DISPOSITION" + trustedProxies = "SWRP_TRUSTED_PROXIES" + maxRequestSize = "SWRP_MAX_REQUEST_SIZE" + maxResponseSize = "SWRP_MAX_RESPONSE_SIZE" + maxRequestTime = "SWRP_MAX_REQUEST_TIME_MS" + maxResponseTime = "SWRP_MAX_RESPONSE_TIME_MS" +) + +// The headers the tests set and look at. +const ( + dispositionHeader = proxy.DispositionHeader + forwardedForHeader = "X-Forwarded-For" + forwardedProtoHeader = "X-Forwarded-Proto" + requestIDHeader = "X-Request-ID" + asnHeader = "X-Client-Asn" + countryHeader = "X-Client-Country" + contentLengthHeader = "Content-Length" + // fromServeWAFHeader is a header the stand-in for servewaf adds to a deny. + fromServeWAFHeader = "X-From-Servewaf" + // addedHeaderValue is the value of a header a test adds to an answer, to see it + // reach the visitor. + addedHeaderValue = "yes" +) + +// What the stand-in for servewaf answers. +const ( + dispositionAllow = proxy.DispositionAllow + dispositionDeny = proxy.DispositionDeny + dispositionClose = proxy.DispositionClose + // deniedBody is the body of a deny. + deniedBody = "refused by servewaf" +) + +// output collects what rproxy writes on stdout. +type output struct { + mu sync.Mutex + buf bytes.Buffer +} + +// Write adds lines rproxy writes. +func (o *output) Write(p []byte) (int, error) { + o.mu.Lock() + defer o.mu.Unlock() + + return o.buf.Write(p) +} + +// requestLine waits for the request log line of a test's one request. +func (o *output) requestLine(t *testing.T) map[string]any { + t.Helper() + + deadline := time.Now().Add(waitLimit) + for time.Now().Before(deadline) { + o.mu.Lock() + text := o.buf.String() + o.mu.Unlock() + + for line := range strings.Lines(text) { + var fields map[string]any + + err := json.Unmarshal([]byte(line), &fields) + if err != nil { + t.Fatalf("output line %q is not JSON: %v", line, err) + } + + if fields["type"] == "request" { + return fields + } + } + + time.Sleep(pollInterval) + } + + t.Fatal("no request log line") + + return nil +} + +// has reports whether what rproxy has written so far holds text. +func (o *output) has(text string) bool { + o.mu.Lock() + defer o.mu.Unlock() + + return strings.Contains(o.buf.String(), text) +} + +// waitFor waits until what rproxy has written holds text. +func (o *output) waitFor(t *testing.T, text string) { + t.Helper() + + deadline := time.Now().Add(waitLimit) + for time.Now().Before(deadline) { + if o.has(text) { + return + } + + time.Sleep(pollInterval) + } + + t.Fatalf("rproxy did not write %q", text) +} + +// heldLog is a request log whose Write waits until release is closed. +type heldLog struct { + release chan struct{} +} + +// Write waits until release is closed. +func (l heldLog) Write(p []byte) (int, error) { + <-l.release + + return len(p), nil +} + +// question is what the stand-in for servewaf received as a question. +type question struct { + method, target, host string + header http.Header + body []byte + contentLength int64 + transferEncoding []string +} + +// serveWAF is a stand-in for servewaf that answers each question as answer +// does, and keeps the questions and the reports it gets. +type serveWAF struct { + *httptest.Server + + questions chan question + reports chan proxy.Report +} + +// startServeWAF starts a stand-in for servewaf. +func startServeWAF(t *testing.T, answer http.HandlerFunc) *serveWAF { + t.Helper() + + s := &serveWAF{ + questions: make(chan question, 10), + reports: make(chan proxy.Report, 10), + } + s.Server = httptest.NewServer(http.HandlerFunc( + func(w http.ResponseWriter, r *http.Request) { + if r.Header.Get(proxy.ClientHeader) == "" { + s.takeReport(t, w, r) + + return + } + + body, err := io.ReadAll(r.Body) + if err != nil { + t.Errorf("read the question's body: %v", err) + } + + s.questions <- question{ + r.Method, r.RequestURI, r.Host, r.Header.Clone(), body, r.ContentLength, + r.TransferEncoding, + } + + answer(w, r) + })) + t.Cleanup(s.Close) + + return s +} + +// takeReport keeps a report and answers it as servewaf does. +func (s *serveWAF) takeReport(t *testing.T, w http.ResponseWriter, r *http.Request) { + t.Helper() + + var report proxy.Report + + err := json.NewDecoder(r.Body).Decode(&report) + if err != nil || r.Method != http.MethodPost || r.URL.Path != proxy.ReportPath { + t.Errorf("%s %s is no report: %v", r.Method, r.URL.Path, err) + } + + w.WriteHeader(http.StatusNoContent) + + s.reports <- report +} + +// wantNoReport checks that no report comes for a moment. +func (s *serveWAF) wantNoReport(t *testing.T) { + t.Helper() + + select { + case report := <-s.reports: + t.Errorf("report %+v, want none", report) + case <-time.After(10 * pollInterval): + } +} + +// report waits for a report. +func (s *serveWAF) report(t *testing.T) proxy.Report { + t.Helper() + + select { + case report := <-s.reports: + return report + case <-time.After(waitLimit): + t.Fatal("no report") + + return proxy.Report{} + } +} + +// question waits for a question. +func (s *serveWAF) question(t *testing.T) question { + t.Helper() + + select { + case asked := <-s.questions: + return asked + case <-time.After(waitLimit): + t.Fatal("no question") + + return question{} + } +} + +// startServeWAFAnsweringReports starts a stand-in for servewaf that answers each +// question as answer does and each report as report does, and returns its URL. +func startServeWAFAnsweringReports( + t *testing.T, answer, report http.HandlerFunc, +) string { + t.Helper() + + backend := httptest.NewServer(http.HandlerFunc( + func(w http.ResponseWriter, r *http.Request) { + if r.Header.Get(proxy.ClientHeader) != "" { + answer(w, r) + + return + } + + report(w, r) + })) + t.Cleanup(backend.Close) + + return backend.URL +} + +// allow answers a question with allow. +func allow(w http.ResponseWriter, _ *http.Request) { + w.Header().Set(dispositionHeader, dispositionAllow) + w.WriteHeader(http.StatusOK) +} + +// deny answers a question with deny, status 403 and a body of its own. +func deny(w http.ResponseWriter, _ *http.Request) { + w.Header().Set(dispositionHeader, dispositionDeny) + w.Header().Set(fromServeWAFHeader, addedHeaderValue) + w.WriteHeader(http.StatusForbidden) + _, _ = io.WriteString(w, deniedBody) +} + +// startApp starts app as the app rproxy passes requests to. +func startApp(t *testing.T, app http.HandlerFunc) *httptest.Server { + t.Helper() + + server := httptest.NewServer(app) + t.Cleanup(server.Close) + + return server +} + +// greet is an app that reads the request's body and answers greeting. +func greet(w http.ResponseWriter, r *http.Request) { + _, _ = io.Copy(io.Discard, r.Body) + _, _ = io.WriteString(w, greeting) +} + +// startRProxy starts rproxy with servewaf at backend and the app at appURL, and +// the settings in env on top of the defaults. It returns where rproxy listens +// and what it writes. +func startRProxy( + t *testing.T, backend, appURL string, env map[string]string, +) (string, *output) { + t.Helper() + + addr, out, _ := startServer(t, backend, appURL, env) + + return addr, out +} + +// startServer is startRProxy, and returns the server as well. +func startServer( + t *testing.T, backend, appURL string, env map[string]string, +) (string, *output, *rproxy.Server) { + t.Helper() + + out := &output{} + addr, server := startServerLogging(t, backend, appURL, env, out, out) + + return addr, out, server +} + +// startServerLogging is startServer with rproxy's request lines going to +// requestLog and its process lines to processLog. +func startServerLogging( + t *testing.T, backend, appURL string, env map[string]string, + requestLog, processLog io.Writer, +) (string, *rproxy.Server) { + t.Helper() + + settings := map[string]string{ + "SWRP_WAF_BACKEND": backend, "SWRP_UPSTREAM_URL": appURL, + } + maps.Copy(settings, env) + + cfg, err := config.RProxyFromEnvironment(func(name string) (string, bool) { + value, ok := settings[name] + + return value, ok + }) + if err != nil { + t.Fatalf("settings %v: %v", settings, err) + } + + server := rproxy.New(rproxy.Params{ + Config: cfg, + RequestLog: requestLog, + ProcessLog: requestlog.NewProcessLogger(processLog, "test", slog.LevelInfo), + }) + + listener, err := (&net.ListenConfig{}).Listen(t.Context(), "tcp", localhost+":0") + if err != nil { + t.Fatalf("listen: %v", err) + } + + go func() { + _ = server.Serve(listener) + }() + + t.Cleanup(func() { + _ = server.Close() + + server.WaitForReports() + }) + + return listener.Addr().String(), server +} + +// answered is the visitor's answer. +type answered struct { + status int + header http.Header + body string +} + +// visit sends req as a visitor, on a connection of its own, and returns the +// answer. +func visit(t *testing.T, req *http.Request) answered { + t.Helper() + + transport := &http.Transport{DisableKeepAlives: true, DisableCompression: true} + defer transport.CloseIdleConnections() + + res, err := transport.RoundTrip(req) + if err != nil { + t.Fatalf("request: %v", err) + } + + defer func() { + _ = res.Body.Close() + }() + + body, err := io.ReadAll(res.Body) + if err != nil { + t.Fatalf("read the answer: %v", err) + } + + return answered{res.StatusCode, res.Header, string(body)} +} + +// newRequest returns a request for target at addr. +func newRequest( + t *testing.T, method, addr, target string, body io.Reader, +) *http.Request { + t.Helper() + + req, err := http.NewRequestWithContext(t.Context(), method, "http://"+addr+target, + body) + if err != nil { + t.Fatalf("new request: %v", err) + } + + return req +} + +// get sends a GET for target at addr, and returns the answer. +func get(t *testing.T, addr, target string) answered { + t.Helper() + + return visit(t, newRequest(t, http.MethodGet, addr, target, http.NoBody)) +} + +// wantGreeting checks that the answer is the app's. +func wantGreeting(t *testing.T, got answered) { + t.Helper() + + if got.status != http.StatusOK || got.body != greeting { + t.Errorf("answered %d %q, want the app's greeting", got.status, got.body) + } +} + +// dial opens a connection to addr, closed as the test ends. +func dial(t *testing.T, addr string) net.Conn { + t.Helper() + + conn, err := (&net.Dialer{}).DialContext(t.Context(), "tcp", addr) + if err != nil { + t.Fatalf("dial: %v", err) + } + + t.Cleanup(func() { + _ = conn.Close() + }) + + err = conn.SetDeadline(time.Now().Add(waitLimit)) + if err != nil { + t.Fatalf("set the deadline: %v", err) + } + + return conn +} + +// send writes text to conn. +func send(t *testing.T, conn net.Conn, text string) { + t.Helper() + + _, err := io.WriteString(conn, text) + if err != nil { + t.Fatalf("send: %v", err) + } +} + +// readStatus reads the answer's status from conn. +func readStatus(t *testing.T, conn net.Conn) int { + t.Helper() + + res, err := http.ReadResponse(bufio.NewReader(conn), nil) + if err != nil { + t.Fatalf("read the answer: %v", err) + } + + _ = res.Body.Close() + + return res.StatusCode +} + +func TestPassesAnAllowedRequestToTheAppAndItsAnswerBack(t *testing.T) { + t.Parallel() + + servewaf := startServeWAF(t, func(w http.ResponseWriter, r *http.Request) { + w.Header().Set(asnHeader, "AS64496") + w.Header().Set(countryHeader, "DE") + allow(w, r) + }) + + seen := make(chan *http.Request, 1) + app := startApp(t, func(w http.ResponseWriter, r *http.Request) { + seen <- r.Clone(t.Context()) + + w.Header().Set("X-App", addedHeaderValue) + w.WriteHeader(http.StatusTeapot) + _, _ = io.WriteString(w, greeting) + }) + addr, _ := startRProxy(t, servewaf.URL, app.URL, nil) + + const target = pathGoReencodes + "?b=2&a=1&bad=%zz;x" + + req := newRequest(t, http.MethodGet, addr, target, http.NoBody) + req.URL.Opaque = pathGoReencodes + req.Host = site + req.Header.Set(asnHeader, "AS1") + req.Header.Set(countryHeader, "XX") + req.Header.Set(requestIDHeader, "the-visitor's") + + got := visit(t, req) + if got.status != http.StatusTeapot || got.body != greeting || + got.header.Get("X-App") != addedHeaderValue { + t.Errorf("answered %+v, want the app's answer", got) + } + + received := <-seen + asked := servewaf.question(t) + + if received.RequestURI != target || received.Host != site { + t.Errorf("the app saw %s for %s", received.RequestURI, received.Host) + } + + for name, want := range map[string]string{ + asnHeader: "AS64496", + countryHeader: "DE", + forwardedForHeader: localhost, + requestIDHeader: asked.header.Get(requestIDHeader), + } { + if received.Header.Get(name) != want { + t.Errorf("the app saw %s %q, want %q", name, received.Header.Get(name), want) + } + } +} + +func TestAPeerNotTrustedHasItsForwardedHeadersReplaced(t *testing.T) { + t.Parallel() + + removed := []string{ + "X-Forwarded-Port", "X-Forwarded-Server", "X-Forwarded-Uri", + "X-Forwarded-Method", "X-Forwarded-Prefix", "X-Forwarded-Tls-Client-Cert", + "X-Forwarded-Tls-Client-Cert-Info", "X-Real-Ip", + } + + servewaf := startServeWAF(t, allow) + seen := make(chan http.Header, 1) + app := startApp(t, func(w http.ResponseWriter, r *http.Request) { + seen <- r.Header.Clone() + + greet(w, r) + }) + addr, _ := startRProxy(t, servewaf.URL, app.URL, map[string]string{ + trustedProxies: "10.0.0.0/8", + }) + + req := newRequest(t, http.MethodGet, addr, "/", http.NoBody) + req.Host = site + req.Header.Set(forwardedForHeader, "198.51.100.1") + req.Header.Set("X-Forwarded-Host", "claimed.example") + req.Header.Set(forwardedProtoHeader, "https") + + for _, name := range removed { + req.Header.Set(name, "claimed") + } + + wantGreeting(t, visit(t, req)) + + header := <-seen + + for name, want := range map[string]string{ + forwardedForHeader: localhost, + "X-Forwarded-Host": site, + forwardedProtoHeader: "http", + } { + if got := header.Values(name); !slices.Equal(got, []string{want}) { + t.Errorf("the app saw %s %q, want %q", name, got, want) + } + } + + for _, name := range removed { + if got := header.Values(name); len(got) != 0 { + t.Errorf("the app saw %s %q, want none", name, got) + } + } +} + +func TestATrustedProxysForwardedHeadersPassOn(t *testing.T) { + t.Parallel() + + servewaf := startServeWAF(t, allow) + seen := make(chan http.Header, 1) + app := startApp(t, func(w http.ResponseWriter, r *http.Request) { + seen <- r.Header.Clone() + + greet(w, r) + }) + addr, _ := startRProxy(t, servewaf.URL, app.URL, map[string]string{ + trustedProxies: localhost + "/32", + }) + + sent := map[string]string{ + "Forwarded": "for=198.51.100.1", + "X-Forwarded-Host": "public.example", + forwardedProtoHeader: "https", + "X-Real-Ip": "198.51.100.1", + } + + req := newRequest(t, http.MethodGet, addr, "/", http.NoBody) + req.Host = site + + for name, value := range sent { + req.Header.Set(name, value) + } + + wantGreeting(t, visit(t, req)) + + header := <-seen + + for name, want := range sent { + if got := header.Values(name); !slices.Equal(got, []string{want}) { + t.Errorf("the app saw %s %q, want %q", name, got, want) + } + } +} + +func TestAnAnswerWithoutAContentTypeGetsNone(t *testing.T) { + t.Parallel() + + servewaf := startServeWAF(t, allow) + app := startApp(t, func(w http.ResponseWriter, _ *http.Request) { + w.Header()["Content-Type"] = nil + _, _ = io.WriteString(w, "") + }) + _, _, server := startServer(t, servewaf.URL, app.URL, nil) + + req := httptest.NewRequestWithContext(t.Context(), http.MethodGet, "/", http.NoBody) + w := httptest.NewRecorder() + + server.Handler.ServeHTTP(w, req) + + // Go's server guesses a type when the first part of the body goes before the + // headers are flushed, which is a matter of timing here. So this looks for what + // stops it: the key with no value. + values, there := w.Header()["Content-Type"] + if !there || len(values) != 0 { + t.Errorf("the answer's Content-Type is %q, there: %v, want the key with no value", + values, there) + } +} + +func TestRemovesTheVisitorsLookupHeadersWhenServeWAFGivesNone(t *testing.T) { + t.Parallel() + + for _, tc := range []struct { + name string + backend func(t *testing.T) string + }{ + {"servewaf allows", func(t *testing.T) string { + t.Helper() + + return startServeWAF(t, allow).URL + }}, + {"servewaf is down", func(*testing.T) string { return nowhere }}, + } { + t.Run(tc.name, func(t *testing.T) { + t.Parallel() + + seen := make(chan http.Header, 1) + app := startApp(t, func(w http.ResponseWriter, r *http.Request) { + seen <- r.Header.Clone() + + greet(w, r) + }) + addr, _ := startRProxy(t, tc.backend(t), app.URL, nil) + + req := newRequest(t, http.MethodGet, addr, "/", http.NoBody) + req.Header.Set(asnHeader, "AS1") + req.Header.Set(countryHeader, "XX") + wantGreeting(t, visit(t, req)) + + header := <-seen + if len(header.Values(asnHeader)) != 0 || len(header.Values(countryHeader)) != 0 { + t.Errorf("the app saw the visitor's lookup headers: %v", header) + } + }) + } +} + +func TestDenyGivesTheVisitorServeWAFsAnswer(t *testing.T) { + t.Parallel() + + servewaf := startServeWAF(t, deny) + app := startApp(t, func(http.ResponseWriter, *http.Request) { + t.Error("the app was asked") + }) + addr, out := startRProxy(t, servewaf.URL, app.URL, nil) + + got := get(t, addr, "/") + if got.status != http.StatusForbidden || got.body != deniedBody || + got.header.Get(fromServeWAFHeader) != addedHeaderValue { + t.Errorf("answered %+v, want servewaf's answer", got) + } + + if len(got.header.Values(dispositionHeader)) != 0 { + t.Errorf("the visitor got %s", dispositionHeader) + } + + report := servewaf.report(t) + if report.Status != http.StatusForbidden || + report.ResponseBytes != int64(len(deniedBody)) || + report.DurationUpstreamTotal != 0 { + t.Errorf("report %+v, of a request denied", report) + } + + line := out.requestLine(t) + if line["disposition"] != dispositionDeny || line["default_disposition"] != nil { + t.Errorf("line %v, want servewaf's deny", line) + } +} + +func TestADeniedHeadKeepsServeWAFsContentLength(t *testing.T) { + t.Parallel() + + servewaf := startServeWAF(t, deny) + app := startApp(t, greet) + addr, _ := startRProxy(t, servewaf.URL, app.URL, nil) + + got := visit(t, newRequest(t, http.MethodHead, addr, "/", http.NoBody)) + if got.status != http.StatusForbidden || + got.header.Get(contentLengthHeader) != strconv.Itoa(len(deniedBody)) { + t.Errorf("answered %+v, want servewaf's 403 and its length", got) + } +} + +func TestAnAnswerRProxyHoldsWholeIsSentBeforeItsLogLine(t *testing.T) { + t.Parallel() + + const host = " HTTP/1.1\r\nHost: site.example\r\n\r\n" + + for _, tc := range []struct { + name, request, body string + status int + }{ + {"a deny", "GET /" + host, deniedBody, http.StatusForbidden}, + {"rproxy's own answer", "GET x:admin" + host, "Bad Request\n", http.StatusBadRequest}, + {"the health check", "GET " + proxy.HealthPath + host, "ok\n", http.StatusOK}, + } { + t.Run(tc.name, func(t *testing.T) { + t.Parallel() + + log := heldLog{release: make(chan struct{})} + servewaf := startServeWAF(t, deny) + app := startApp(t, greet) + addr, _ := startServerLogging(t, servewaf.URL, app.URL, nil, log, io.Discard) + + t.Cleanup(func() { close(log.release) }) + + conn := dial(t, addr) + send(t, conn, tc.request) + + res, err := http.ReadResponse(bufio.NewReader(conn), nil) + if err != nil { + t.Fatalf("read the answer: %v", err) + } + + defer func() { + _ = res.Body.Close() + }() + + body, err := io.ReadAll(res.Body) + if err != nil || res.StatusCode != tc.status || string(body) != tc.body { + t.Errorf("answered %d %q (%v), want %d %q", res.StatusCode, body, err, + tc.status, tc.body) + } + }) + } +} + +func TestRProxysOwnAnswerIsPlainText(t *testing.T) { + t.Parallel() + + servewaf := startServeWAF(t, allow) + app := startApp(t, greet) + addr, _ := startRProxy(t, servewaf.URL, app.URL, nil) + + conn := dial(t, addr) + send(t, conn, "GET x:admin HTTP/1.1\r\nHost: site.example\r\n\r\n") + + res, err := http.ReadResponse(bufio.NewReader(conn), nil) + if err != nil { + t.Fatalf("read the answer: %v", err) + } + + _ = res.Body.Close() + + if res.Header.Get("Content-Type") != "text/plain; charset=utf-8" || + res.Header.Get("X-Content-Type-Options") != "nosniff" { + t.Errorf("the answer's headers are %v", res.Header) + } +} + +func TestCloseClosesTheConnectionWithoutAnAnswer(t *testing.T) { + t.Parallel() + + servewaf := startServeWAF(t, func(w http.ResponseWriter, _ *http.Request) { + w.Header().Set(dispositionHeader, dispositionClose) + w.WriteHeader(http.StatusOK) + }) + app := startApp(t, func(http.ResponseWriter, *http.Request) { + t.Error("the app was asked") + }) + addr, out := startRProxy(t, servewaf.URL, app.URL, nil) + + conn := dial(t, addr) + send(t, conn, "GET / HTTP/1.1\r\nHost: site.example\r\n\r\n") + + text, err := io.ReadAll(conn) + if err != nil || len(text) != 0 { + t.Errorf("read %q (%v), want the connection closed without an answer", + text, err) + } + + report := servewaf.report(t) + if report.Status != 0 || report.ResponseBytes != 0 { + t.Errorf("report %+v, of a visitor sent nothing", report) + } + + line := out.requestLine(t) + if line["disposition"] != dispositionClose || line["status"] != 0.0 { + t.Errorf("line %v, want close and status 0", line) + } +} + +func TestALineForARequestClosedWithoutAnAnswerHasNoNegativeTime(t *testing.T) { + t.Parallel() + + backend := startServeWAFAnsweringReports(t, + func(w http.ResponseWriter, _ *http.Request) { + w.Header().Set(dispositionHeader, dispositionClose) + }, + func(w http.ResponseWriter, _ *http.Request) { + w.WriteHeader(http.StatusNoContent) + }) + + // A negative time would be about a microsecond, which one request would seldom + // show. + for range 100 { + addr, out := startRProxy(t, backend, nowhere, nil) + + conn := dial(t, addr) + send(t, conn, "GET / HTTP/1.1\r\nHost: site.example\r\n\r\n") + + line := out.requestLine(t) + + duration, ok := line["duration_response"].(float64) + if !ok || duration < 0 { + t.Errorf("line %v, want a duration_response that is not negative", line) + } + } +} + +// answeringWith returns what starts a stand-in for servewaf that answers each +// question as answer does, and returns its URL. +func answeringWith(answer http.HandlerFunc) func(t *testing.T) string { + return func(t *testing.T) string { + t.Helper() + + return startServeWAF(t, answer).URL + } +} + +// noAnswers are the ways servewaf can leave the decision to the default. +func noAnswers() map[string]func(t *testing.T) string { + return map[string]func(t *testing.T) string{ + "no connection": func(*testing.T) string { return nowhere }, + "no answer at all": answeringWith(func(http.ResponseWriter, *http.Request) { + panic(http.ErrAbortHandler) + }), + "a slow start": answeringWith(func(_ http.ResponseWriter, r *http.Request) { + <-r.Context().Done() + }), + "headers followed by a stalled body": answeringWith( + func(w http.ResponseWriter, r *http.Request) { + w.Header().Set(dispositionHeader, dispositionDeny) + w.WriteHeader(http.StatusForbidden) + _ = http.NewResponseController(w).Flush() + + <-r.Context().Done() + }), + "an answer cut short": answeringWith( + func(w http.ResponseWriter, _ *http.Request) { + w.Header().Set(dispositionHeader, dispositionDeny) + w.Header().Set(contentLengthHeader, "100") + w.WriteHeader(http.StatusForbidden) + _, _ = io.WriteString(w, deniedBody) + // Go's server drops on this panic what it has not flushed. + _ = http.NewResponseController(w).Flush() + + panic(http.ErrAbortHandler) + }), + } +} + +func TestAnAnswerOtherThanAllowGoesToTheVisitorAsServeWAFGaveIt(t *testing.T) { + t.Parallel() + + for _, tc := range []struct { + name, disposition, body string + status int + }{ + {"a refusal", dispositionDeny, deniedBody, http.StatusForbidden}, + {"a refusal, 429", dispositionDeny, deniedBody, http.StatusTooManyRequests}, + {"servewaf's 400", "", "refused\n", http.StatusBadRequest}, + {"a 500", "", "broken\n", http.StatusInternalServerError}, + {"a 503 denied", dispositionDeny, "busy\n", http.StatusServiceUnavailable}, + {"no header", "", "who knows\n", http.StatusOK}, + {"an unknown disposition", "maybe", "who knows\n", http.StatusOK}, + } { + for _, setting := range []string{dispositionAllow, dispositionDeny} { + t.Run(tc.name+" with the default "+setting, func(t *testing.T) { + t.Parallel() + + servewaf := startServeWAF(t, func(w http.ResponseWriter, _ *http.Request) { + if tc.disposition != "" { + w.Header().Set(dispositionHeader, tc.disposition) + } + + w.Header().Set(fromServeWAFHeader, addedHeaderValue) + w.WriteHeader(tc.status) + _, _ = io.WriteString(w, tc.body) + }) + app := startApp(t, func(http.ResponseWriter, *http.Request) { + t.Error("the app was asked") + }) + addr, out := startRProxy(t, servewaf.URL, app.URL, map[string]string{ + defaultDisposition: setting, + }) + + got := get(t, addr, "/") + if got.status != tc.status || got.body != tc.body || + got.header.Get(fromServeWAFHeader) != addedHeaderValue { + t.Errorf("answered %+v, want servewaf's answer", got) + } + + if len(got.header.Values(dispositionHeader)) != 0 { + t.Errorf("the visitor got %s", dispositionHeader) + } + + line := out.requestLine(t) + if line["status"] != float64(tc.status) || line["default_disposition"] != nil { + t.Errorf("line %v, want status %d and no default", line, tc.status) + } + + if out.has("servewaf gave no answer") { + t.Error("rproxy warned that servewaf gave no answer") + } + }) + } + } +} + +func TestAnAnswerWithoutAValidDispositionGetsNoReport(t *testing.T) { + t.Parallel() + + servewaf := startServeWAF(t, func(w http.ResponseWriter, _ *http.Request) { + http.Error(w, "refused", http.StatusBadRequest) + }) + app := startApp(t, greet) + addr, out := startRProxy(t, servewaf.URL, app.URL, nil) + + if got := get(t, addr, "/"); got.status != http.StatusBadRequest { + t.Errorf("answered %+v, want servewaf's 400", got) + } + + if line := out.requestLine(t); line["disposition"] != nil { + t.Errorf("line %v, want no disposition", line) + } + + servewaf.wantNoReport(t) +} + +func TestAnAnswerWithoutAContentTypeGoesToTheVisitorWithNone(t *testing.T) { + t.Parallel() + + servewaf := startServeWAF(t, func(w http.ResponseWriter, _ *http.Request) { + w.Header()["Content-Type"] = nil + w.WriteHeader(http.StatusInternalServerError) + _, _ = io.WriteString(w, "broken") + }) + addr, _ := startRProxy(t, servewaf.URL, nowhere, nil) + + got := get(t, addr, "/") + if contentType, sent := got.header["Content-Type"]; sent { + t.Errorf("the visitor got the Content-Type %v", contentType) + } +} + +func TestAnAnswerOf1MiBGoesToTheVisitor(t *testing.T) { + t.Parallel() + + const size = 1 << 20 + + servewaf := startServeWAF(t, func(w http.ResponseWriter, _ *http.Request) { + w.WriteHeader(http.StatusInternalServerError) + _, _ = w.Write(make([]byte, size)) + }) + addr, _ := startRProxy(t, servewaf.URL, nowhere, nil) + + got := get(t, addr, "/") + if got.status != http.StatusInternalServerError || len(got.body) != size { + t.Errorf("answered %d with %d bytes, want servewaf's 500 with %d", + got.status, len(got.body), size) + } +} + +// startServeWAFWriting starts a stand-in for servewaf that reads each question and +// answers it with text, as it is, and returns its URL. It can send what Go's +// server would not. +func startServeWAFWriting(t *testing.T, text string) string { + t.Helper() + + listener, err := (&net.ListenConfig{}).Listen(t.Context(), "tcp", localhost+":0") + if err != nil { + t.Fatalf("listen: %v", err) + } + + t.Cleanup(func() { + _ = listener.Close() + }) + + go func() { + for { + conn, err := listener.Accept() + if err != nil { + return + } + + _ = conn.SetDeadline(time.Now().Add(waitLimit)) + _, _ = http.ReadRequest(bufio.NewReader(conn)) + _, _ = io.WriteString(conn, text) + _ = conn.Close() + } + }() + + return "http://" + listener.Addr().String() +} + +// overOneMiB answers a question with a body over the most rproxy reads, and with +// disposition unless it is "". +func overOneMiB(disposition string) http.HandlerFunc { + return func(w http.ResponseWriter, _ *http.Request) { + if disposition != "" { + w.Header().Set(dispositionHeader, disposition) + } + + w.WriteHeader(http.StatusForbidden) + _, _ = w.Write(make([]byte, 1<<20+1)) + } +} + +// switchAndHold answers a 101 and holds the connection open until rproxy closes +// it, or until long after any test has stopped waiting. +func switchAndHold(w http.ResponseWriter, _ *http.Request) { + conn, _, err := http.NewResponseController(w).Hijack() + if err != nil { + return + } + + defer func() { + _ = conn.Close() + }() + + _ = conn.SetDeadline(time.Now().Add(2 * waitLimit)) + _, _ = io.WriteString(conn, "HTTP/1.1 101 Switching Protocols\r\n"+ + "Connection: Upgrade\r\nUpgrade: websocket\r\n\r\n") + _, _ = io.Copy(io.Discard, conn) +} + +func TestAnAnswerThatCannotBePassedOnIsAnswered502(t *testing.T) { + t.Parallel() + + for name, backend := range map[string]func(t *testing.T) string{ + "a deny over 1 MiB": answeringWith(overOneMiB(dispositionDeny)), + "an allow over 1 MiB": answeringWith(overOneMiB(dispositionAllow)), + "no header over 1 MiB": answeringWith(overOneMiB("")), + "a status below 100": func(t *testing.T) string { + t.Helper() + + return startServeWAFWriting(t, "HTTP/1.1 099 Odd\r\nContent-Length: 0\r\n\r\n") + }, + "a 101 that holds its connection open": answeringWith(switchAndHold), + } { + for _, setting := range []string{dispositionAllow, dispositionDeny} { + t.Run(name+" with the default "+setting, func(t *testing.T) { + t.Parallel() + + app := startApp(t, func(http.ResponseWriter, *http.Request) { + t.Error("the app was asked") + }) + addr, out := startRProxy(t, backend(t), app.URL, map[string]string{ + defaultDisposition: setting, + }) + + if got := get(t, addr, "/"); got.status != http.StatusBadGateway { + t.Errorf("answered %+v, want 502", got) + } + + line := out.requestLine(t) + if line["status"] != float64(http.StatusBadGateway) || + line["disposition"] != nil || line["default_disposition"] != nil { + t.Errorf("line %v, want status 502 and no disposition", line) + } + + out.waitFor(t, "servewaf's answer is dropped") + }) + } + } +} + +func TestAnAnswerThatCannotBePassedOnGetsNoReport(t *testing.T) { + t.Parallel() + + servewaf := startServeWAF(t, overOneMiB(dispositionDeny)) + addr, out := startRProxy(t, servewaf.URL, nowhere, nil) + + if got := get(t, addr, "/"); got.status != http.StatusBadGateway { + t.Errorf("answered %+v, want 502", got) + } + + out.requestLine(t) + servewaf.wantNoReport(t) +} + +func TestWithoutAnAnswerTheDefaultDispositionDecides(t *testing.T) { + t.Parallel() + + for name, backend := range noAnswers() { + t.Run("fails open on "+name, func(t *testing.T) { + t.Parallel() + + app := startApp(t, greet) + addr, out := startRProxy(t, backend(t), app.URL, map[string]string{ + timeoutSetting: "200", + }) + + wantGreeting(t, get(t, addr, "/")) + + line := out.requestLine(t) + if line["disposition"] != dispositionAllow || + line["default_disposition"] != true { + t.Errorf("line %v, want allow by default", line) + } + }) + + t.Run("fails closed on "+name, func(t *testing.T) { + t.Parallel() + + app := startApp(t, func(http.ResponseWriter, *http.Request) { + t.Error("the app was asked") + }) + addr, out := startRProxy(t, backend(t), app.URL, map[string]string{ + timeoutSetting: "200", + defaultDisposition: dispositionDeny, + }) + + got := get(t, addr, "/") + if got.status != http.StatusServiceUnavailable { + t.Errorf("answered %+v, want 503", got) + } + + line := out.requestLine(t) + if line["disposition"] != dispositionDeny || + line["default_disposition"] != true { + t.Errorf("line %v, want deny by default", line) + } + }) + } +} + +func TestAVisitorThatGoesAwayWhileServeWAFIsAskedGetsNoDefault(t *testing.T) { + t.Parallel() + + for _, disposition := range []string{dispositionAllow, dispositionDeny} { + t.Run("with the default "+disposition, func(t *testing.T) { + t.Parallel() + + servewaf := startServeWAF(t, func(_ http.ResponseWriter, r *http.Request) { + <-r.Context().Done() + }) + app := startApp(t, func(http.ResponseWriter, *http.Request) { + t.Error("the app was asked") + }) + addr, out := startRProxy(t, servewaf.URL, app.URL, map[string]string{ + timeoutSetting: "60000", + defaultDisposition: disposition, + }) + + conn := dial(t, addr) + send(t, conn, "GET / HTTP/1.1\r\nHost: site.example\r\n\r\n") + servewaf.question(t) + + _ = conn.Close() + + line := out.requestLine(t) + if line["status"] != 0.0 || line["disposition"] != nil || + line["default_disposition"] != nil { + t.Errorf("line %v, want status 0 and no disposition", line) + } + + if out.has("servewaf gave no answer") { + t.Error("rproxy warned that servewaf gave no answer") + } + + servewaf.wantNoReport(t) + }) + } +} + +func TestRProxyWarnsWhenTheDefaultDecides(t *testing.T) { + t.Parallel() + + app := startApp(t, greet) + addr, out := startRProxy(t, nowhere, app.URL, nil) + + wantGreeting(t, get(t, addr, "/")) + out.waitFor(t, "servewaf gave no answer") +} + +func TestSendsNoReportWithoutAnAnswer(t *testing.T) { + t.Parallel() + + servewaf := startServeWAF(t, func(_ http.ResponseWriter, r *http.Request) { + <-r.Context().Done() + }) + app := startApp(t, greet) + addr, out := startRProxy(t, servewaf.URL, app.URL, map[string]string{ + timeoutSetting: "200", + }) + + wantGreeting(t, get(t, addr, "/")) + out.requestLine(t) + servewaf.wantNoReport(t) +} + +func TestAPathUnderOwnPrefixNeverReachesTheApp(t *testing.T) { + t.Parallel() + + app := startApp(t, func(http.ResponseWriter, *http.Request) { + t.Error("the app was asked") + }) + addr, _ := startRProxy(t, nowhere, app.URL, nil) + + got := get(t, addr, "/_smallwebwaf/metrics") + if got.status != http.StatusNotFound { + t.Errorf("answered %+v, want 404", got) + } +} + +func TestHealthCheckIsAnsweredWhileServeWAFIsDown(t *testing.T) { + t.Parallel() + + servewaf := startServeWAF(t, allow) + app := startApp(t, greet) + addr, out := startRProxy(t, nowhere, app.URL, nil) + + got := get(t, addr, proxy.HealthPath) + if got.status != http.StatusOK || got.body != "ok\n" { + t.Errorf("answered %+v, want ok", got) + } + + line := out.requestLine(t) + if line["path"] != proxy.HealthPath || line["disposition"] != nil { + t.Errorf("line %v, want one with no disposition", line) + } + + // With servewaf up too, it is not asked. + addr, _ = startRProxy(t, servewaf.URL, app.URL, nil) + get(t, addr, proxy.HealthPath) + + if len(servewaf.questions) != 0 { + t.Error("servewaf was asked about the health check") + } +} + +func TestTheVisitorDoesNotWaitForTheReport(t *testing.T) { + t.Parallel() + + release := make(chan struct{}) + backend := startServeWAFAnsweringReports(t, deny, + func(w http.ResponseWriter, r *http.Request) { + select { + case <-release: + case <-r.Context().Done(): + } + + w.WriteHeader(http.StatusNoContent) + }) + + app := startApp(t, greet) + addr, _ := startRProxy(t, backend, app.URL, map[string]string{ + timeoutSetting: "60000", + }) + + t.Cleanup(func() { close(release) }) + + // Go's server reads the next request on a connection only once the handler has + // returned, so the second answer comes only if the handler does not wait for + // the first report, which the stand-in leaves unanswered. + conn := dial(t, addr) + reader := bufio.NewReader(conn) + + for range 2 { + send(t, conn, "GET / HTTP/1.1\r\nHost: site.example\r\n\r\n") + + res, err := http.ReadResponse(reader, nil) + if err != nil { + t.Fatalf("read the answer: %v", err) + } + + _, _ = io.Copy(io.Discard, res.Body) + _ = res.Body.Close() + + if res.StatusCode != http.StatusForbidden { + t.Errorf("status %d, want servewaf's 403", res.StatusCode) + } + } +} + +func TestAReportServeWAFLeavesUnansweredIsDroppedAtTheTimeLimit(t *testing.T) { + t.Parallel() + + release := make(chan struct{}) + backend := startServeWAFAnsweringReports(t, allow, + func(w http.ResponseWriter, _ *http.Request) { + <-release + + w.WriteHeader(http.StatusNoContent) + }) + + app := startApp(t, greet) + addr, out := startRProxy(t, backend, app.URL, map[string]string{ + timeoutSetting: "1000", + }) + + t.Cleanup(func() { close(release) }) + + wantGreeting(t, get(t, addr, "/")) + out.waitFor(t, "a report is dropped") +} + +func TestAReportServeWAFRefusesIsDroppedWithAWarning(t *testing.T) { + t.Parallel() + + backend := startServeWAFAnsweringReports(t, allow, + func(w http.ResponseWriter, _ *http.Request) { + w.WriteHeader(http.StatusBadRequest) + }) + + app := startApp(t, greet) + addr, out := startRProxy(t, backend, app.URL, nil) + + wantGreeting(t, get(t, addr, "/")) + out.waitFor(t, "a report is dropped") +} + +func TestAReportAnswered101IsDroppedAtOnce(t *testing.T) { + t.Parallel() + + backend := startServeWAFAnsweringReports(t, allow, switchAndHold) + app := startApp(t, greet) + addr, out := startRProxy(t, backend, app.URL, map[string]string{ + timeoutSetting: "60000", + }) + + wantGreeting(t, get(t, addr, "/")) + out.waitFor(t, "a report is dropped") +} + +func TestAReportDueAfterWaitForReportsIsDropped(t *testing.T) { + t.Parallel() + + servewaf := startServeWAF(t, allow) + app := startApp(t, greet) + addr, out, server := startServer(t, servewaf.URL, app.URL, nil) + + server.WaitForReports() + + wantGreeting(t, get(t, addr, "/")) + out.requestLine(t) + servewaf.wantNoReport(t) +} + +func TestReportGivesWhatHappened(t *testing.T) { + t.Parallel() + + const body = "the visitor's body" + + servewaf := startServeWAF(t, allow) + app := startApp(t, greet) + addr, out := startRProxy(t, servewaf.URL, app.URL, nil) + + wantGreeting(t, visit(t, newRequest(t, http.MethodPost, addr, "/", + strings.NewReader(body)))) + + report := servewaf.report(t) + + if report.RequestID == "" || report.Status != http.StatusOK || + report.RequestBytes != int64(len(body)) || + report.ResponseBytes != int64(len(greeting)) || + report.DurationTotal <= 0 || report.DurationUpstreamTotal <= 0 || + report.DurationUpstreamTotal > report.DurationTotal { + t.Errorf("report %+v, of %d bytes up and %d back", report, len(body), len(greeting)) + } + + line := out.requestLine(t) + if line["request_id"] != report.RequestID { + t.Errorf("line %v, report %+v, want the same request id", line, report) + } +} + +func TestWaitForReportsWaitsForAReportBeingSent(t *testing.T) { + t.Parallel() + + reportGotThere := make(chan struct{}) + release := make(chan struct{}) + backend := httptest.NewServer(http.HandlerFunc( + func(w http.ResponseWriter, r *http.Request) { + if r.Header.Get(proxy.ClientHeader) != "" { + allow(w, r) + + return + } + + close(reportGotThere) + + select { + case <-release: + case <-r.Context().Done(): + } + + w.WriteHeader(http.StatusNoContent) + })) + t.Cleanup(backend.Close) + + app := startApp(t, greet) + addr, _, server := startServer(t, backend.URL, app.URL, map[string]string{ + timeoutSetting: "5000", + }) + + wantGreeting(t, get(t, addr, "/")) + + select { + case <-reportGotThere: + case <-time.After(waitLimit): + t.Fatal("no report") + } + + waited := make(chan struct{}) + + go func() { + server.WaitForReports() + close(waited) + }() + + select { + case <-waited: + t.Error("it did not wait for the report") + case <-time.After(10 * pollInterval): + } + + close(release) + + select { + case <-waited: + case <-time.After(waitLimit): + t.Error("it still waits, the report having been answered") + } +} + +func TestReportOfAWebSocketCountsWhatItCarried(t *testing.T) { + t.Parallel() + + servewaf := startServeWAF(t, allow) + app := startApp(t, echoAfterUpgrade) + addr, out := startRProxy(t, servewaf.URL, app.URL, map[string]string{ + maxRequestTime: "2000", + maxResponseTime: "2000", + }) + + conn := dial(t, addr) + send(t, conn, "GET /socket HTTP/1.1\r\nHost: app\r\n"+ + "Connection: Upgrade\r\nUpgrade: websocket\r\n\r\n") + + reader := bufio.NewReader(conn) + + res, err := http.ReadResponse(reader, nil) + if err != nil { + t.Fatalf("read the answer to the upgrade: %v", err) + } + + _ = res.Body.Close() + + if res.StatusCode != http.StatusSwitchingProtocols { + t.Fatalf("status %d, want %d", res.StatusCode, http.StatusSwitchingProtocols) + } + + // Past both time limits, the connection still carries what is sent. + time.Sleep(2100 * time.Millisecond) + + const message = "still here\n" + + send(t, conn, message) + + echoed, err := reader.ReadString('\n') + if err != nil || echoed != message { + t.Errorf("echo %q (%v), want %q", echoed, err, message) + } + + _ = conn.Close() + + report := servewaf.report(t) + if report.Status != http.StatusSwitchingProtocols || + report.RequestBytes != int64(len(message)) || + report.ResponseBytes != int64(len(message)) { + t.Errorf("report %+v, of %d bytes each way", report, len(message)) + } + + if line := out.requestLine(t); line["status"] != float64(res.StatusCode) { + t.Errorf("line %v, want status 101", line) + } +} + +func TestAWebSocketPassesOnThatTheVisitorStoppedSending(t *testing.T) { + t.Parallel() + + const goodbye = "sent after the visitor stopped sending\n" + + servewaf := startServeWAF(t, allow) + app := startApp(t, func(w http.ResponseWriter, _ *http.Request) { + conn, buffered, err := http.NewResponseController(w).Hijack() + if err != nil { + return + } + + defer func() { + _ = conn.Close() + }() + + _, _ = buffered.WriteString("HTTP/1.1 101 Switching Protocols\r\n" + + "Connection: Upgrade\r\nUpgrade: websocket\r\n\r\n") + _ = buffered.Flush() + + _, _ = io.Copy(io.Discard, buffered) + + _, _ = buffered.WriteString(goodbye) + _ = buffered.Flush() + }) + addr, _ := startRProxy(t, servewaf.URL, app.URL, nil) + + conn := dial(t, addr) + send(t, conn, "GET /socket HTTP/1.1\r\nHost: app\r\n"+ + "Connection: Upgrade\r\nUpgrade: websocket\r\n\r\n") + + reader := bufio.NewReader(conn) + + res, err := http.ReadResponse(reader, nil) + if err != nil { + t.Fatalf("read the answer to the upgrade: %v", err) + } + + _ = res.Body.Close() + + tcpConn, ok := conn.(*net.TCPConn) + if !ok { + t.Fatalf("the connection is a %T", conn) + } + + err = tcpConn.CloseWrite() + if err != nil { + t.Fatalf("stop sending: %v", err) + } + + text, err := io.ReadAll(reader) + if err != nil || string(text) != goodbye { + t.Errorf("read %q (%v), want %q", text, err, goodbye) + } +} + +// This test does not run in parallel. Go closes a connection nothing refers to at +// its next garbage collection, which other tests' allocations would soon start, and +// that would hide a connection left open. +// +//nolint:paralleltest // see above +func TestASwitchNotPassedOnClosesTheAppsConnection(t *testing.T) { + closed := make(chan error, 1) + servewaf := startServeWAF(t, allow) + app := startApp(t, func(w http.ResponseWriter, _ *http.Request) { + conn, buffered, err := http.NewResponseController(w).Hijack() + if err != nil { + closed <- err + + return + } + + defer func() { + _ = conn.Close() + }() + + // Not the protocol the visitor asked for, so ReverseProxy refuses the switch. + _, _ = buffered.WriteString("HTTP/1.1 101 Switching Protocols\r\n" + + "Connection: Upgrade\r\nUpgrade: another\r\n\r\n") + _ = buffered.Flush() + + _ = conn.SetReadDeadline(time.Now().Add(waitLimit)) + + _, err = buffered.ReadByte() + closed <- err + }) + addr, _ := startRProxy(t, servewaf.URL, app.URL, nil) + + conn := dial(t, addr) + send(t, conn, "GET /socket HTTP/1.1\r\nHost: app\r\n"+ + "Connection: Upgrade\r\nUpgrade: websocket\r\n\r\n") + + if status := readStatus(t, conn); status != http.StatusBadGateway { + t.Errorf("status %d, want 502", status) + } + + err := <-closed + if !errors.Is(err, io.EOF) { + t.Errorf("the app's read gave %v, want its connection closed", err) + } +} + +// echoAfterUpgrade is an app that switches protocols on request, and then sends +// back each line it receives. +func echoAfterUpgrade(w http.ResponseWriter, _ *http.Request) { + conn, buffered, err := http.NewResponseController(w).Hijack() + if err != nil { + return + } + + defer func() { + _ = conn.Close() + }() + + _, _ = buffered.WriteString("HTTP/1.1 101 Switching Protocols\r\n" + + "Connection: Upgrade\r\nUpgrade: websocket\r\n\r\n") + _ = buffered.Flush() + + for { + line, err := buffered.ReadString('\n') + if err != nil { + return + } + + _, _ = buffered.WriteString(line) + _ = buffered.Flush() + } +} + +func TestEachRequestGetsAFreshID(t *testing.T) { + t.Parallel() + + const fromTraefik = "an-id-from-traefik" + + servewaf := startServeWAF(t, allow) + app := startApp(t, greet) + addr, _ := startRProxy(t, servewaf.URL, app.URL, map[string]string{ + trustedProxies: localhost + "/32", + }) + + ids := make([]string, 0, 2) + + for range 2 { + req := newRequest(t, http.MethodGet, addr, "/", http.NoBody) + req.Header.Set(requestIDHeader, fromTraefik) + wantGreeting(t, visit(t, req)) + + ids = append(ids, servewaf.question(t).header.Get(requestIDHeader)) + } + + if slices.Contains(ids, fromTraefik) || ids[0] == "" || ids[0] == ids[1] { + t.Errorf("ids %v, want two fresh ones", ids) + } +} + +func TestLogLineGivesTheRequestAsSent(t *testing.T) { + t.Parallel() + + const ( + path = "/search%2Fall" + query = "q=secret&token=%zz" + body = "the visitor's body" + ) + + servewaf := startServeWAF(t, allow) + app := startApp(t, greet) + addr, out := startRProxy(t, servewaf.URL, app.URL, nil) + + req := newRequest(t, http.MethodPost, addr, path+"?"+query, strings.NewReader(body)) + req.Host = site + wantGreeting(t, visit(t, req)) + + line := out.requestLine(t) + asked := servewaf.question(t) + + for name, want := range map[string]any{ + "type": "request", + "request_id": asked.header.Get(requestIDHeader), + "client_ip": localhost, + "host": site, + "method": http.MethodPost, + "path": path, + "query": query, + "status": float64(http.StatusOK), + "request_bytes": float64(len(body)), + "response_bytes": float64(len(greeting)), + "disposition": dispositionAllow, + } { + if line[name] != want { + t.Errorf("%s is %v, want %v", name, line[name], want) + } + } + + if line["default_disposition"] != nil { + t.Errorf("default_disposition is %v, want it left out", line["default_disposition"]) + } + + for _, name := range []string{"duration_request", "duration_response"} { + duration, ok := line[name].(float64) + if !ok || duration <= 0 { + t.Errorf("%s is %v, want a time", name, line[name]) + } + } + + text, _ := line["time"].(string) + + _, err := time.Parse(time.RFC3339, text) + if err != nil { + t.Errorf("time: %v", err) + } +} diff --git a/internal/rproxy/servewaf.go b/internal/rproxy/servewaf.go new file mode 100644 index 0000000..3328e18 --- /dev/null +++ b/internal/rproxy/servewaf.go @@ -0,0 +1,266 @@ +package rproxy + +import ( + "bytes" + "context" + "encoding/json" + "errors" + "fmt" + "io" + "maps" + "net/http" + "strings" + + "sneak.berlin/go/smallwebwaf/internal/proxy" +) + +// answerBodyMaxBytes is the most of the body of servewaf's answer that is read. +// It is far above any answer's. +const answerBodyMaxBytes = 1 << 20 + +// reportAnswerMaxBytes is the most of the answer to a report that is read, +// which has no body when servewaf takes the report. +const reportAnswerMaxBytes = 4 << 10 + +var ( + errCannotPassOn = errors.New("the answer cannot be passed to the visitor") + errReportRefused = errors.New("the report was answered") +) + +// servewafAnswer is what servewaf answered to a question. +type servewafAnswer struct { + status int + header http.Header + body []byte +} + +// noteAnswer notes what servewaf answered. An answer without a valid disposition +// notes nothing, and no report is due for it: servewaf holds no decision to report +// on. +func (rq *request) noteAnswer(answer *servewafAnswer) { + disposition := answer.header.Get(proxy.DispositionHeader) + + switch disposition { + case proxy.DispositionAllow, proxy.DispositionDeny, proxy.DispositionClose: + rq.disposition = disposition + rq.answered = true + rq.asn = answer.header.Get(asnHeader) + rq.country = answer.header.Get(countryHeader) + } +} + +// handleNoUsableAnswer deals with a request whose question ended in err, which +// leaves no answer to act on. ctx is the request's own context. +func (rq *request) handleNoUsableAnswer(ctx context.Context, err error) { + switch { + case ctx.Err() != nil: + // The visitor went away while servewaf was being asked. servewaf has not + // failed, and nothing is noted. This panic has Go's server close the + // connection without an answer, and log nothing; the deferred finish still + // runs. + panic(http.ErrAbortHandler) + case errors.Is(err, errCannotPassOn): + rq.h.processLog.Warn("servewaf's answer is dropped", "error", err.Error()) + rq.answer(http.StatusBadGateway) + default: + // servewaf cannot be connected to, does not answer at all, or does not answer + // within SWRP_TIMEOUT_MS. A request that keeps servewaf busy past + // SWRP_TIMEOUT_MS, like any request while servewaf is overloaded, gets + // SWRP_DEFAULT_DISPOSITION here. deny closes this, at the cost of answering 503 + // whenever servewaf is slow. Making such requests hard to send is + // https://git.eeqj.de/sneak/smallwebwaf/issues/182. + rq.h.processLog.Warn("servewaf gave no answer", "error", err.Error()) + + rq.disposition = rq.h.config.DefaultDisposition + rq.byDefault = true + + if rq.disposition == proxy.DispositionAllow { + rq.passToApp(ctx) + } else { + rq.answer(http.StatusServiceUnavailable) + } + } +} + +// ask sends servewaf the question, and returns its answer, which must have come +// whole within SWRP_TIMEOUT_MS. It ends when ctx does, as when the visitor goes +// away. +func (rq *request) ask(ctx context.Context) (*servewafAnswer, error) { + ctx, cancel := context.WithTimeout(ctx, rq.h.config.Timeout) + defer cancel() + + question, err := rq.newQuestion(ctx) + if err != nil { + return nil, err + } + + res, err := rq.h.serveWAFTransport.RoundTrip(question) + if err != nil { + return nil, fmt.Errorf("send the question: %w", err) + } + + defer func() { + _ = res.Body.Close() + }() + + // The body of a 101 is the connection itself, and the time limit does not end + // a read of that. Go's client also takes a status below 100, which Go's server + // panics when asked to send. + if res.StatusCode < http.StatusOK { + return nil, fmt.Errorf("%w: its status is %d", errCannotPassOn, res.StatusCode) + } + + body, err := io.ReadAll(io.LimitReader(res.Body, answerBodyMaxBytes+1)) + if err != nil { + return nil, fmt.Errorf("read the answer: %w", err) + } + + if len(body) > answerBodyMaxBytes { + return nil, fmt.Errorf("%w: its body is over 1 MiB", errCannotPassOn) + } + + return &servewafAnswer{res.StatusCode, res.Header, body}, nil +} + +// canBeAsked reports whether Go's client can send servewaf r as the visitor sent +// it. It cannot send these as they came: +// - a target that does not start with /, or that starts with //; +// - an empty Host, for which it sends servewaf's own address; +// - a Host holding an IPv6 zone, which it removes; +// - more than one User-Agent, of which it sends the first; +// - an empty User-Agent, which it leaves out, the one case that still goes to +// servewaf. +func canBeAsked(r *http.Request) bool { + target, host := r.RequestURI, r.Host + hostHasZone := strings.HasPrefix(host, "[") && strings.Contains(host, "%") + + return strings.HasPrefix(target, "/") && !strings.HasPrefix(target, "//") && + host != "" && !hostHasZone && len(r.Header.Values("User-Agent")) <= 1 +} + +// newQuestion returns the question about the request, as "The question" in +// SPEC.md gives it: the visitor's request as the visitor sent it, with the part +// of the body it holds and rproxy's headers. +func (rq *request) newQuestion(ctx context.Context) (*http.Request, error) { + in := rq.in + + var held []byte + if rq.body != nil { + held = rq.body.held + } + + question, err := http.NewRequestWithContext( //nolint:gosec // a setting's URL + ctx, in.Method, rq.h.config.WAFBackend.String(), bytes.NewReader(held)) + if err != nil { + return nil, fmt.Errorf("make the question: %w", err) + } + + backend := question.URL + target := *in.URL + target.Scheme = backend.Scheme + target.Host = backend.Host + setPathAsSent(&target, in) + question.URL = &target + question.Host = in.Host + + // The length NewRequestWithContext set is that of what is held. This line makes + // a body whose length the visitor did not announce go chunked, as "The question" + // in SPEC.md requires. + question.TransferEncoding = in.TransferEncoding + + question.Header = in.Header.Clone() + question.Header.Set(proxy.ClientHeader, rq.client.String()) + question.Header.Set(proxy.PeerHeader, rq.peer.String()) + question.Header.Del(proxy.ContentLengthHeader) + + announced := in.Header.Get("Content-Length") + if announced != "" { + question.Header.Set(proxy.ContentLengthHeader, announced) + } + + question.Header.Set(requestIDHeader, rq.id) + question.Header.Set("X-Forwarded-Proto", scheme(in, rq.peerTrusted)) + + // Go's client adds a User-Agent of its own to a request without the key. + _, sent := question.Header["User-Agent"] + if !sent { + question.Header["User-Agent"] = nil + } + + return question, nil +} + +// answerWith gives the visitor servewaf's answer, less its disposition. +func (rq *request) answerWith(answer *servewafAnswer) { + rq.prepareOwnAnswer(rq.requestDeadline()) + + header := rq.out.Header() + maps.Copy(header, answer.header) + header.Del(proxy.DispositionHeader) + + // Go's server adds a Content-Type it guesses from the body to a response whose + // headers have no Content-Type key. The key with no value sends none. + if header.Get("Content-Type") == "" { + header["Content-Type"] = nil + } + + rq.send(answer.status, answer.body) +} + +// sendReport sends servewaf report on a goroutine of its own, so that the +// visitor does not wait for it. The report goes on after ctx, the request's own +// context, has ended. +func (h *handler) sendReport(ctx context.Context, report *proxy.Report) { + h.reportsMu.Lock() + defer h.reportsMu.Unlock() + + if h.reportsStopped { + return + } + + h.reports.Go(func() { + err := h.postReport(context.WithoutCancel(ctx), report) + if err != nil { + h.processLog.Warn("a report is dropped", "error", err.Error()) + } + }) +} + +// postReport posts report to servewaf, and waits for its answer for up to +// SWRP_TIMEOUT_MS. +func (h *handler) postReport(ctx context.Context, report *proxy.Report) error { + body, err := json.Marshal(report) + if err != nil { + return fmt.Errorf("encode the report: %w", err) + } + + ctx, cancel := context.WithTimeout(ctx, h.config.Timeout) + defer cancel() + + req, err := http.NewRequestWithContext( //nolint:gosec // a setting's URL + ctx, http.MethodPost, h.reportURL, bytes.NewReader(body)) + if err != nil { + return fmt.Errorf("make the request: %w", err) + } + + req.Header.Set("Content-Type", "application/json") + + res, err := h.serveWAFTransport.RoundTrip(req) + if err != nil { + return fmt.Errorf("send the report: %w", err) + } + + // The body of a 101 is the connection itself, and the time limit does not end + // a read of that. + if res.StatusCode >= http.StatusOK { + _, _ = io.Copy(io.Discard, io.LimitReader(res.Body, reportAnswerMaxBytes)) + } + + _ = res.Body.Close() + + if res.StatusCode != http.StatusNoContent { + return fmt.Errorf("%w %s", errReportRefused, res.Status) + } + + return nil +} diff --git a/internal/smallwebwaf/rproxy.go b/internal/smallwebwaf/rproxy.go new file mode 100644 index 0000000..0c95194 --- /dev/null +++ b/internal/smallwebwaf/rproxy.go @@ -0,0 +1,99 @@ +package smallwebwaf + +import ( + "context" + "errors" + "log/slog" + "net" + "net/http" + "os" + + "sneak.berlin/go/smallwebwaf/internal/config" + "sneak.berlin/go/smallwebwaf/internal/requestlog" + "sneak.berlin/go/smallwebwaf/internal/rproxy" +) + +// RunRProxy is `smallwebwaf rproxy`: it reads rproxy's settings, then serves +// requests until ctx is done. Then it gives the requests in progress +// shutdownTimeout to finish, and the reports being sent SWRP_TIMEOUT_MS. It +// returns the process's exit status, 1 when rproxy cannot start. Its process +// messages name the host's name as instance, and it writes those at info or +// more severe: rproxy has no setting for either. +func RunRProxy(ctx context.Context, params Params) int { + hostname, _ := os.Hostname() // "" when the host has no name to give + processLog := requestlog.NewProcessLogger(params.Stdout, hostname, slog.LevelInfo) + + cfg, err := config.RProxyFromEnvironment(params.LookupEnv) + if err != nil { + processLog.Error("invalid setting", "error", err.Error()) + + return 1 + } + + server := rproxy.New(rproxy.Params{ + Config: cfg, + RequestLog: params.Stdout, + ProcessLog: processLog, + }) + + listener, err := (&net.ListenConfig{}).Listen(ctx, "tcp", cfg.ListenAddr) + if err != nil { + processLog.Error("cannot listen on SWRP_LISTEN_ADDR", "error", err.Error()) + + return 1 + } + + processLog.Info("starting", + "version", params.Version, + "address", listener.Addr().String(), + "settings", cfg) + + return serveRProxy(ctx, server, listener, processLog) +} + +// serveRProxy serves requests on listener until ctx is done, and then stops as +// RunRProxy says. +func serveRProxy( + ctx context.Context, server *rproxy.Server, listener net.Listener, + processLog *slog.Logger, +) int { + served := make(chan error, 1) + + go func() { + served <- server.Serve(listener) + }() + + select { + case err := <-served: + processLog.Error("serving failed", "error", err.Error()) + + return 1 + case <-ctx.Done(): + } + + processLog.Info("stopping") + + shutdownCtx, cancel := context.WithTimeout(context.WithoutCancel(ctx), + shutdownTimeout) + defer cancel() + + err := server.Shutdown(shutdownCtx) + if err != nil { + processLog.Warn("requests still in progress were cut off", + "error", err.Error()) + + _ = server.Close() + } + + err = <-served + if !errors.Is(err, http.ErrServerClosed) { + processLog.Error("serving failed", "error", err.Error()) + + return 1 + } + + server.WaitForReports() + processLog.Info("stopped") + + return 0 +} diff --git a/internal/smallwebwaf/rproxy_test.go b/internal/smallwebwaf/rproxy_test.go new file mode 100644 index 0000000..4774c0f --- /dev/null +++ b/internal/smallwebwaf/rproxy_test.go @@ -0,0 +1,787 @@ +package smallwebwaf_test + +import ( + "bufio" + "context" + "encoding/pem" + "io" + "maps" + "net" + "net/http" + "net/http/httptest" + "os" + "os/exec" + "path/filepath" + "strconv" + "strings" + "sync/atomic" + "syscall" + "testing" + "time" + + "sneak.berlin/go/smallwebwaf/internal/proxy" + "sneak.berlin/go/smallwebwaf/internal/smallwebwaf" +) + +const ( + // wafBackend is the setting that names servewaf to rproxy. + wafBackend = "SWRP_WAF_BACKEND" + // rproxyListenAddr is the setting that says where rproxy listens. + rproxyListenAddr = "SWRP_LISTEN_ADDR" + // rproxyTimeout is the setting that says how long rproxy waits for servewaf. + rproxyTimeout = "SWRP_TIMEOUT_MS" + // bannedProbe is a path the default rule file bans a client for. + bannedProbe = "/.env" + // otherClient is a client the tests do not ban. + otherClient = "198.51.100.7" +) + +// runRProxy runs `smallwebwaf rproxy` with the settings in env until ctx is +// done, and returns its exit status. +func runRProxy(ctx context.Context, env map[string]string, out *output) int { + return smallwebwaf.RunRProxy(ctx, smallwebwaf.Params{ + Version: testVersion, + LookupEnv: func(name string) (string, bool) { + value, ok := env[name] + + return value, ok + }, + Stdout: out, + }) +} + +// startRProxy runs rproxy with servewaf at backend and the app at appURL, and +// returns its URL and what it writes. It believes X-Forwarded-For from the +// tests. It stops as the test ends, and that it stopped in order is checked. +func startRProxy(t *testing.T, backend, appURL string) (string, *output) { + t.Helper() + + ctx, stop := context.WithCancel(t.Context()) + out := &output{} + exited := make(chan int, 1) + + go func() { + exited <- runRProxy(ctx, map[string]string{ + rproxyListenAddr: localhost + ":0", + "SWRP_UPSTREAM_URL": appURL, + "SWRP_TRUSTED_PROXIES": localhost + "/32", + wafBackend: backend, + }, out) + }() + + t.Cleanup(func() { + stop() + + select { + case status := <-exited: + if status != 0 { + t.Errorf("rproxy's exit status %d, want 0; output:\n%s", status, out.text()) + } + case <-time.After(waitLimit): + t.Error("rproxy still running after being told to stop") + } + + out.line(t, "msg", "stopped") + }) + + addr, _ := out.line(t, "msg", "starting")["address"].(string) + + return "http://" + addr, out +} + +// startServeWAF runs servewaf with the settings in env, and returns its address +// and URL, what it writes, and what stops it, which checks that it stopped in +// order. +func startServeWAF(t *testing.T, env map[string]string) (string, *output, func()) { + t.Helper() + + ctx, stop := context.WithCancel(t.Context()) + out := &output{} + exited := make(chan int, 1) + + go func() { + exited <- runAs(ctx, true, env, out) + }() + + addr, _ := out.line(t, "msg", "starting")["address"].(string) + + return addr, out, func() { + stop() + + select { + case status := <-exited: + if status != 0 { + t.Fatalf("servewaf's exit status %d, want 0; output:\n%s", status, out.text()) + } + case <-time.After(waitLimit): + t.Fatal("servewaf still running after being told to stop") + } + } +} + +// serveWAFSettings are servewaf's settings with the default rule file, to listen +// at any free port. +func serveWAFSettings(t *testing.T) map[string]string { + t.Helper() + + return map[string]string{ + listenAddr: localhost + ":0", + stateDir: t.TempDir(), + rulesDir: filepath.Join("..", "..", "share", "rules.d"), + } +} + +func TestABanThroughOneSiteRefusesTheClientOnTheOther(t *testing.T) { + t.Parallel() + + addr, serveWAFOut, stop := startServeWAF(t, serveWAFSettings(t)) + defer stop() + + backend := "http://" + addr + first, firstOut := startRProxy(t, backend, startApp(t)) + second, _ := startRProxy(t, backend, startApp(t)) + + wantStatus(t, first+"/", placed, http.StatusOK) + wantStatus(t, second+"/", placed, http.StatusOK) + + // The default rule file bans a probe for this. + wantStatus(t, first+bannedProbe, placed, http.StatusForbidden) + wantStatus(t, second+"/", placed, http.StatusForbidden) + wantStatus(t, second+"/", otherClient, http.StatusOK) + + // One request's three lines carry one id, rproxy's with servewaf's answer. + line := firstOut.line(t, "path", bannedProbe) + if line["disposition"] != proxy.DispositionDeny || line["client_ip"] != placed { + t.Errorf("rproxy's line is %v", line) + } + + id, _ := line["request_id"].(string) + decision := serveWAFLine(t, serveWAFOut, id, "decision") + report := serveWAFLine(t, serveWAFOut, id, "report") + + if decision["host"] != line["host"] || decision["client_ip"] != placed || + report["status"] != float64(http.StatusForbidden) { + t.Errorf("servewaf's lines are %v and %v", decision, report) + } +} + +// serveWAFLine waits for servewaf's line with request id and servewaf_line kind. +func serveWAFLine(t *testing.T, out *output, id, kind string) map[string]any { + t.Helper() + + deadline := time.Now().Add(waitLimit) + for time.Now().Before(deadline) { + line := out.findLine(t, func(fields map[string]any) bool { + return fields["request_id"] == id && fields["servewaf_line"] == kind + }) + if line != nil { + return line + } + + time.Sleep(pollInterval) + } + + t.Fatalf("no %s line for %s in:\n%s", kind, id, out.text()) + + return nil +} + +func TestTheSiteAnswersWhileServeWAFIsStoppedAndStartedAgain(t *testing.T) { + t.Parallel() + + env := serveWAFSettings(t) + addr, _, stop := startServeWAF(t, env) + site, out := startRProxy(t, "http://"+addr, startApp(t)) + + // A visitor who keeps asking throughout, and is never refused. + visitorDone := make(chan struct{}) + visitorStop := make(chan struct{}) + + var refused atomic.Int64 + + go func() { + defer close(visitorDone) + + for { + select { + case <-visitorStop: + return + case <-time.After(pollInterval): + } + + if !answersNormally(t.Context(), site+"/", otherClient) { + refused.Add(1) + } + } + }() + + wantStatus(t, site+bannedProbe, placed, http.StatusForbidden) + wantStatus(t, site+"/", placed, http.StatusForbidden) + stop() + + // With servewaf down, the site answers, and rproxy says the default did it. + wantStatus(t, site+"/after-stop", placed, http.StatusOK) + + line := out.line(t, "path", "/after-stop") + if line["disposition"] != proxy.DispositionAllow || + line["default_disposition"] != true { + t.Errorf("rproxy's line is %v", line) + } + + // Back at its address, with its state files, it applies the ban again. + restarted := maps.Clone(env) + restarted[listenAddr] = addr + _, _, stop = startServeWAF(t, restarted) + + defer stop() + + wantStatus(t, site+"/", placed, http.StatusForbidden) + + close(visitorStop) + <-visitorDone + + if refused.Load() != 0 { + t.Errorf("the site refused its visitor %d times", refused.Load()) + } +} + +// answersNormally reports whether a request to url from the client at from, as +// X-Forwarded-For names it, gets the app's answer. +func answersNormally(ctx context.Context, url, from string) bool { + req, err := http.NewRequestWithContext(ctx, http.MethodGet, url, http.NoBody) + if err != nil { + return false + } + + req.Header.Set("X-Forwarded-For", from) + + transport := &http.Transport{} + defer transport.CloseIdleConnections() + + res, err := transport.RoundTrip(req) + if err != nil { + return false + } + + body, err := io.ReadAll(res.Body) + _ = res.Body.Close() + + return err == nil && res.StatusCode == http.StatusOK && string(body) == greeting +} + +// sendRaw sends request, as it is, to rproxy at site, and returns the status of +// the answer. +func sendRaw(t *testing.T, site, request string) int { + t.Helper() + + conn, err := (&net.Dialer{}).DialContext(t.Context(), "tcp", + strings.TrimPrefix(site, "http://")) + if err != nil { + t.Fatalf("dial: %v", err) + } + + defer func() { + _ = conn.Close() + }() + + err = conn.SetDeadline(time.Now().Add(waitLimit)) + if err != nil { + t.Fatalf("set the deadline: %v", err) + } + + _, err = io.WriteString(conn, request) + if err != nil { + t.Fatalf("send: %v", err) + } + + res, err := http.ReadResponse(bufio.NewReader(conn), nil) + if err != nil { + t.Fatalf("read the answer: %v", err) + } + + _ = res.Body.Close() + + return res.StatusCode +} + +func TestAChunkedBodyStaysABodyAtServeWAF(t *testing.T) { + t.Parallel() + + addr, serveWAFOut, stop := startServeWAF(t, serveWAFSettings(t)) + defer stop() + + site, _ := startRProxy(t, "http://"+addr, startApp(t)) + + // Read as a further request, this body would ban otherClient on every site. + const smuggledID = "smuggled-request" + + smuggled := "GET " + bannedProbe + " HTTP/1.1\r\nHost: site.example\r\n" + + proxy.ClientHeader + ": " + otherClient + "\r\n" + + "X-Request-ID: " + smuggledID + "\r\n\r\n" + + status := sendRaw(t, site, "POST /upload HTTP/1.1\r\nHost: site.example\r\n"+ + "X-Forwarded-For: "+placed+"\r\nTransfer-Encoding: chunked\r\n\r\n"+ + strconv.FormatInt(int64(len(smuggled)), 16)+"\r\n"+smuggled+"\r\n0\r\n\r\n") + if status != http.StatusOK { + t.Errorf("status %d, want the app's 200", status) + } + + // Read as a further request, it would act straight after the answer. + for range 20 { + wantStatus(t, site+"/", otherClient, http.StatusOK) + time.Sleep(pollInterval) + } + + line := serveWAFOut.findLine(t, func(fields map[string]any) bool { + return fields["request_id"] == smuggledID + }) + if line != nil { + t.Errorf("servewaf read the body as a request: %v", line) + } +} + +func TestATargetThatCannotGoToServeWAFIsRefusedForABannedClient(t *testing.T) { + t.Parallel() + + addr, _, stop := startServeWAF(t, serveWAFSettings(t)) + defer stop() + + site, out := startRProxy(t, "http://"+addr, startApp(t)) + + wantStatus(t, site+bannedProbe, placed, http.StatusForbidden) + + status := sendRaw(t, site, "GET x:admin HTTP/1.1\r\nHost: site.example\r\n"+ + "X-Forwarded-For: "+placed+"\r\n\r\n") + if status != http.StatusBadRequest { + t.Errorf("status %d, want 400", status) + } + + line := out.line(t, "path", "admin") + if line["disposition"] != nil || line["default_disposition"] != nil { + t.Errorf("rproxy's line is %v, want no disposition", line) + } +} + +func TestAClientWithAZoneIsTheClientWithout(t *testing.T) { + t.Parallel() + + const zoned = "2001:db8::1%eth0" + + addr, _, stop := startServeWAF(t, serveWAFSettings(t)) + defer stop() + + site, out := startRProxy(t, "http://"+addr, startApp(t)) + + wantStatus(t, site+bannedProbe, zoned, http.StatusForbidden) + wantStatus(t, site+"/", zoned, http.StatusForbidden) + wantStatus(t, site+"/", "2001:db8::1", http.StatusForbidden) + + line := out.line(t, "path", bannedProbe) + if line["client_ip"] != "2001:db8::1" || line["default_disposition"] != nil { + t.Errorf("rproxy's line is %v", line) + } +} + +func TestAStatusServeWAFDoesNotTakeIsReportedAs599(t *testing.T) { + t.Parallel() + + const status = 799 + + app := httptest.NewServer(http.HandlerFunc( + func(w http.ResponseWriter, _ *http.Request) { + w.WriteHeader(status) + })) + t.Cleanup(app.Close) + + addr, serveWAFOut, stop := startServeWAF(t, serveWAFSettings(t)) + defer stop() + + site, out := startRProxy(t, "http://"+addr, app.URL) + + wantStatus(t, site+"/", placed, status) + + line := out.line(t, "path", "/") + if line["status"] != float64(status) { + t.Errorf("rproxy's line is %v, want the status the visitor got", line) + } + + id, _ := line["request_id"].(string) + + report := serveWAFLine(t, serveWAFOut, id, "report") + if report["status"] != 599.0 { + t.Errorf("servewaf's line is %v, want status 599", report) + } +} + +// startRProxyProcess runs `smallwebwaf rproxy` in a process of its own, with the +// settings in env, each as NAME=value. It returns where rproxy listens, the +// process, and what rproxy writes. +func startRProxyProcess(t *testing.T, env ...string) (string, *exec.Cmd, *output) { + t.Helper() + + binary, err := os.Executable() + if err != nil { + t.Fatalf("find the test binary: %v", err) + } + + ctx, cancel := context.WithTimeout(t.Context(), 3*waitLimit) + t.Cleanup(cancel) + + out := &output{} + cmd := exec.CommandContext(ctx, binary) //nolint:gosec // this test binary + cmd.Stdout = out + + cmd.Env = append(os.Environ(), + commandForMain+"=rproxy", + "SWRP_LISTEN_ADDR="+localhost+":0") + cmd.Env = append(cmd.Env, env...) + + err = cmd.Start() + if err != nil { + t.Fatalf("start: %v", err) + } + + t.Cleanup(func() { + _ = cmd.Process.Kill() + _ = cmd.Wait() + }) + + addr, _ := out.line(t, "msg", "starting")["address"].(string) + + return addr, cmd, out +} + +func TestRProxyAsksAServeWAFThatSpeaksHTTP2OverHTTP1(t *testing.T) { + t.Parallel() + + protocols := make(chan string, 10) + backend := httptest.NewUnstartedServer(http.HandlerFunc( + func(w http.ResponseWriter, r *http.Request) { + if r.Header.Get(proxy.ClientHeader) != "" { + protocols <- r.Proto + + w.Header().Set(proxy.DispositionHeader, proxy.DispositionDeny) + w.WriteHeader(http.StatusForbidden) + } + })) + backend.EnableHTTP2 = true + backend.StartTLS() + t.Cleanup(backend.Close) + + // rproxy has no setting for the certificates it trusts, so it is told of this + // one as Go's client is told of any, in a process of its own. + certificates := filepath.Join(t.TempDir(), "servewaf.pem") + + err := os.WriteFile(certificates, pem.EncodeToMemory(&pem.Block{ + Type: "CERTIFICATE", Bytes: backend.Certificate().Raw, + }), 0o600) + if err != nil { + t.Fatalf("write the certificate: %v", err) + } + + addr, _, _ := startRProxyProcess(t, + wafBackend+"="+backend.URL, + "SWRP_UPSTREAM_URL="+startApp(t), + "SSL_CERT_FILE="+certificates) + + for name, headers := range map[string]string{ + "no such header": "", + "Upgrade": "Upgrade: anything\r\n", + "Connection": "Connection: anything\r\n", + "a WebSocket upgrade": "Connection: Upgrade\r\nUpgrade: websocket\r\n", + } { + status := sendRaw(t, addr, "GET / HTTP/1.1\r\nHost: site.example\r\n"+headers+"\r\n") + if status != http.StatusForbidden { + t.Errorf("with %s: status %d, want servewaf's 403", name, status) + } + + select { + case protocol := <-protocols: + if protocol != "HTTP/1.1" { + t.Errorf("with %s: asked over %s, want HTTP/1.1", name, protocol) + } + case <-time.After(waitLimit): + t.Errorf("with %s: servewaf was not asked", name) + } + } +} + +func TestRProxyProcessLinesNameTheHostAndComeAtInfo(t *testing.T) { + t.Parallel() + + hostname, err := os.Hostname() + if err != nil { + t.Fatalf("the host's name: %v", err) + } + + // Nothing listens at the backend's address; no request asks it. + _, out := startRProxy(t, "http://127.0.0.1:9", startApp(t)) + + line := out.line(t, "msg", "starting") + if line["instance"] != hostname || line["level"] != "INFO" { + t.Errorf("rproxy's line is %v, want instance %s at INFO", line, hostname) + } +} + +func TestRProxyStoppingWaitsForAReportBeingSent(t *testing.T) { + t.Parallel() + + reportGotThere := make(chan struct{}) + release := make(chan struct{}) + backend := httptest.NewServer(http.HandlerFunc( + func(w http.ResponseWriter, r *http.Request) { + if r.Header.Get(proxy.ClientHeader) != "" { + w.Header().Set(proxy.DispositionHeader, proxy.DispositionAllow) + + return + } + + close(reportGotThere) + <-release + w.WriteHeader(http.StatusNoContent) + })) + t.Cleanup(backend.Close) + + ctx, stop := context.WithCancel(t.Context()) + defer stop() + + env := map[string]string{ + rproxyListenAddr: localhost + ":0", + "SWRP_UPSTREAM_URL": startApp(t), + rproxyTimeout: "60000", + wafBackend: backend.URL, + } + out := &output{} + exited := make(chan int, 1) + + go func() { + exited <- runRProxy(ctx, env, out) + }() + + addr, _ := out.line(t, "msg", "starting")["address"].(string) + + wantStatus(t, "http://"+addr+"/", placed, http.StatusOK) + + select { + case <-reportGotThere: + case <-time.After(waitLimit): + t.Fatal("no report") + } + + stop() + + select { + case <-exited: + t.Error("rproxy stopped with a report on its way") + close(release) + + return + case <-time.After(10 * pollInterval): + } + + close(release) + + select { + case status := <-exited: + if status != 0 { + t.Errorf("rproxy's exit status %d, want 0; output:\n%s", status, out.text()) + } + case <-time.After(waitLimit): + t.Error("rproxy still running, the report having been answered") + } +} + +// sendAndKeepOpen sends request to rproxy at addr, and returns the connection. A +// read from it ends with an error once waitLimit is up. +func sendAndKeepOpen(t *testing.T, addr, request string) net.Conn { + t.Helper() + + conn, err := (&net.Dialer{}).DialContext(t.Context(), "tcp", addr) + if err != nil { + t.Fatalf("dial: %v", err) + } + + t.Cleanup(func() { + _ = conn.Close() + }) + + err = conn.SetDeadline(time.Now().Add(waitLimit)) + if err != nil { + t.Fatalf("set the deadline: %v", err) + } + + _, err = io.WriteString(conn, request) + if err != nil { + t.Fatalf("send: %v", err) + } + + return conn +} + +func TestRProxyStoppingCutsOffARequestAfterFiveSeconds(t *testing.T) { + t.Parallel() + + const ( + cutOffAfter = 5 * time.Second + upstreamSetting = "SWRP_UPSTREAM_URL" + ) + + asked := make(chan struct{}) + release := make(chan struct{}) + app := httptest.NewServer(http.HandlerFunc( + func(http.ResponseWriter, *http.Request) { + close(asked) + <-release + })) + t.Cleanup(app.Close) + t.Cleanup(func() { close(release) }) + + backend := httptest.NewServer(http.HandlerFunc( + func(w http.ResponseWriter, _ *http.Request) { + w.Header().Set(proxy.DispositionHeader, proxy.DispositionAllow) + })) + t.Cleanup(backend.Close) + + ctx, stop := context.WithCancel(t.Context()) + defer stop() + + env := map[string]string{ + rproxyListenAddr: localhost + ":0", + upstreamSetting: app.URL, + wafBackend: backend.URL, + } + out := &output{} + exited := make(chan int, 1) + + go func() { + exited <- runRProxy(ctx, env, out) + }() + + addr, _ := out.line(t, "msg", "starting")["address"].(string) + conn := sendAndKeepOpen(t, addr, "GET / HTTP/1.1\r\nHost: site.example\r\n\r\n") + + select { + case <-asked: + case <-time.After(waitLimit): + t.Fatal("the app was not asked") + } + + stopped := time.Now() + + stop() + + text, err := io.ReadAll(conn) + if err != nil || len(text) != 0 { + t.Errorf("read %q (%v), want the connection cut off", text, err) + } + + if elapsed := time.Since(stopped); elapsed < cutOffAfter { + t.Errorf("cut off after %s, want %s", elapsed, cutOffAfter) + } + + select { + case status := <-exited: + if status != 0 { + t.Errorf("rproxy's exit status %d, want 0; output:\n%s", status, out.text()) + } + case <-time.After(waitLimit): + t.Error("rproxy still running after the cut-off") + } + + out.line(t, "msg", "requests still in progress were cut off") +} + +func TestRProxyInvalidSettingStopsTheStart(t *testing.T) { + t.Parallel() + + for _, tc := range []struct { + name, setting string + env map[string]string + }{ + {"no backend", wafBackend, nil}, + { + "another invalid", rproxyTimeout, + map[string]string{wafBackend: "http://servewaf:8080", rproxyTimeout: "soon"}, + }, + } { + t.Run(tc.name, func(t *testing.T) { + t.Parallel() + + out := &output{} + + if status := runRProxy(t.Context(), tc.env, out); status != 1 { + t.Errorf("exit status %d, want 1", status) + } + + line := out.line(t, "msg", "invalid setting") + message, _ := line["error"].(string) + + if !strings.HasPrefix(message, tc.setting+": ") { + t.Errorf("start refused with %v, want it to name %s", line, tc.setting) + } + }) + } +} + +func TestRProxyAddressInUseStopsTheStart(t *testing.T) { + t.Parallel() + + taken, err := (&net.ListenConfig{}).Listen(t.Context(), "tcp", localhost+":0") + if err != nil { + t.Fatalf("listen: %v", err) + } + + defer func() { + _ = taken.Close() + }() + + out := &output{} + + status := runRProxy(t.Context(), map[string]string{ + wafBackend: "http://servewaf:8080", + rproxyListenAddr: taken.Addr().String(), + }, out) + if status != 1 { + t.Errorf("exit status %d, want 1", status) + } + + out.line(t, "msg", "cannot listen on SWRP_LISTEN_ADDR") +} + +func TestMainRunsRProxy(t *testing.T) { + t.Parallel() + + // Nothing listens there; the health check does not ask. + addr, cmd, out := startRProxyProcess(t, wafBackend+"=http://127.0.0.1:9") + + req, err := http.NewRequestWithContext(t.Context(), http.MethodGet, + "http://"+addr+proxy.HealthPath, http.NoBody) + if err != nil { + t.Fatalf("new request: %v", err) + } + + transport := &http.Transport{} + defer transport.CloseIdleConnections() + + res, err := transport.RoundTrip(req) + if err != nil { + t.Fatalf("health check: %v", err) + } + + body, err := io.ReadAll(res.Body) + _ = res.Body.Close() + + if err != nil || string(body) != "ok\n" { + t.Errorf("health check answered %q (%v)", body, err) + } + + err = cmd.Process.Signal(syscall.SIGTERM) + if err != nil { + t.Fatalf("signal: %v", err) + } + + err = cmd.Wait() + if err != nil { + t.Errorf("exit: %v; output:\n%s", err, out.text()) + } +} diff --git a/internal/smallwebwaf/smallwebwaf.go b/internal/smallwebwaf/smallwebwaf.go index 3545ef4..3e6809a 100644 --- a/internal/smallwebwaf/smallwebwaf.go +++ b/internal/smallwebwaf/smallwebwaf.go @@ -2,7 +2,9 @@ // the rule files, the lookup database and the state files, serves requests // until it is told to stop, and then stops in an orderly way, writing the // state files. It serves them in front of an app, or, as `smallwebwaf -// servewaf`, by answering rproxy's questions about them. +// servewaf`, by answering rproxy's questions about them. As `smallwebwaf +// rproxy`, it instead reads rproxy's settings alone, and runs the reverse proxy +// in front of one app that asks servewaf about each request. package smallwebwaf import ( @@ -54,7 +56,9 @@ type Params struct { // Main runs smallwebwaf until SIGTERM or SIGINT, and returns the // process's exit status. Run as `smallwebwaf servewaf`, it answers // rproxy's questions rather than pass requests to an app. Run as -// `smallwebwaf healthcheck`, it is the container's health check instead. +// `smallwebwaf rproxy`, it is the reverse proxy that asks those questions. +// Run as `smallwebwaf healthcheck`, it is the container's health check +// instead. func Main(version string) int { if len(os.Args) > 1 && os.Args[1] == "healthcheck" { return HealthCheck(context.Background(), os.Args[2:], os.LookupEnv, os.Stderr) @@ -64,12 +68,18 @@ func Main(version string) int { syscall.SIGTERM, os.Interrupt) defer stop() - return Run(ctx, Params{ + params := Params{ Version: version, LookupEnv: os.LookupEnv, Stdout: os.Stdout, ServeWAF: len(os.Args) > 1 && os.Args[1] == "servewaf", - }) + } + + if len(os.Args) > 1 && os.Args[1] == "rproxy" { + return RunRProxy(ctx, params) + } + + return Run(ctx, params) } // Run reads the settings, the rule files, the lookup database and the diff --git a/internal/smallwebwaf/smallwebwaf_test.go b/internal/smallwebwaf/smallwebwaf_test.go index 75dc067..610c957 100644 --- a/internal/smallwebwaf/smallwebwaf_test.go +++ b/internal/smallwebwaf/smallwebwaf_test.go @@ -76,21 +76,11 @@ func (o *output) line(t *testing.T, key, value string) map[string]any { deadline := time.Now().Add(waitLimit) for time.Now().Before(deadline) { - o.mu.Lock() - text := o.buf.String() - o.mu.Unlock() - - for line := range strings.Lines(text) { - var fields map[string]any - - err := json.Unmarshal([]byte(line), &fields) - if err != nil { - t.Fatalf("output line %q is not JSON: %v", line, err) - } - - if fields[key] == value { - return fields - } + fields := o.findLine(t, func(fields map[string]any) bool { + return fields[key] == value + }) + if fields != nil { + return fields } time.Sleep(pollInterval) @@ -101,6 +91,29 @@ func (o *output) line(t *testing.T, key, value string) map[string]any { return nil } +// findLine returns the first line written so far for which matches is true, or +// nil. +func (o *output) findLine( + t *testing.T, matches func(map[string]any) bool, +) map[string]any { + t.Helper() + + for line := range strings.Lines(o.text()) { + var fields map[string]any + + err := json.Unmarshal([]byte(line), &fields) + if err != nil { + t.Fatalf("output line %q is not JSON: %v", line, err) + } + + if matches(fields) { + return fields + } + } + + return nil +} + // text returns everything written so far. func (o *output) text() string { o.mu.Lock() From 344ce13c96da62e8630610d9f7dc9b67e52404c4 Mon Sep 17 00:00:00 2001 From: clawbot <35+clawbot@noreply.example.org> Date: Thu, 5 Nov 2026 13:15:05 +0100 Subject: [PATCH 7/9] rproxy and servewaf replace the one-process smallwebwaf (closes #172, closes #165) `smallwebwaf` with no command now prints the commands and exits with status 2. The image runs rproxy, or servewaf with SMALLWEBWAF_COMMAND=servewaf. The passing-on code, its settings and its log fields and metric go. internal/proxy keeps the checks, answers, reports and own endpoints; its tests run through the real rproxy. The documents describe the pair. Judgement call: clients.json keeps token_refused, since state files refuse unknown fields. Deviation: script/gitea-requests sets SWWAF_LOG_REQUEST_HEADERS, to label requests. Partially verified: make example-app and make gitea-requests never built the image, as Ubuntu's snapshot service answered 5xx. Model: opus-5-5 --- Dockerfile | 11 +- README.md | 352 ++++++---- SPEC.md | 803 ++++++++++++----------- cmd/smallwebwaf/main.go | 10 +- docs/admin-endpoints.md | 47 +- docs/alerts.md | 13 +- docs/gitea.md | 18 +- docs/image.md | 8 +- docs/metrics.md | 20 +- docs/request-log.md | 288 ++++---- docs/rules.md | 9 +- docs/settings.md | 109 ++- docs/state-files.md | 6 +- internal/config/config.go | 189 ++---- internal/config/config_test.go | 633 ++++++++---------- internal/config/rproxy.go | 41 +- internal/config/rproxy_test.go | 25 + internal/metrics/metrics.go | 61 +- internal/proxy/admin.go | 52 +- internal/proxy/admin_test.go | 158 +---- internal/proxy/alerts_test.go | 5 +- internal/proxy/anomaly_test.go | 1 + internal/proxy/answertime_test.go | 42 +- internal/proxy/bans.go | 37 +- internal/proxy/bans_test.go | 47 +- internal/proxy/biased_test.go | 27 +- internal/proxy/bodies.go | 239 ------- internal/proxy/bytelimits_test.go | 63 +- internal/proxy/client.go | 104 +-- internal/proxy/client_test.go | 165 ----- internal/proxy/coreruleset.go | 47 +- internal/proxy/coreruleset_test.go | 136 +--- internal/proxy/countries_test.go | 12 +- internal/proxy/crowdsec_test.go | 2 +- internal/proxy/errorburst_test.go | 63 +- internal/proxy/health_test.go | 26 +- internal/proxy/history_test.go | 27 +- internal/proxy/limits_test.go | 162 ----- internal/proxy/lookup.go | 15 +- internal/proxy/lookup_test.go | 15 +- internal/proxy/metrics_test.go | 208 ++---- internal/proxy/observe_test.go | 65 -- internal/proxy/passthrough_test.go | 556 ---------------- internal/proxy/proxy.go | 191 ++---- internal/proxy/proxy_test.go | 297 ++++++--- internal/proxy/ratelimits_test.go | 2 +- internal/proxy/reputation_test.go | 18 +- internal/proxy/request.go | 630 ++---------------- internal/proxy/request_internal_test.go | 55 -- internal/proxy/requestlog_test.go | 141 +--- internal/proxy/rulefiles_test.go | 4 +- internal/proxy/servewaf.go | 102 ++- internal/proxy/servewaf_test.go | 71 +- internal/proxy/staticlists_test.go | 21 +- internal/proxy/timeouts_test.go | 453 ------------- internal/ratelimit/history_test.go | 4 +- internal/ratelimit/ratelimit.go | 46 +- internal/requestlog/requestlog.go | 55 +- internal/rproxy/question_test.go | 46 ++ internal/smallwebwaf/healthcheck.go | 83 ++- internal/smallwebwaf/healthcheck_test.go | 113 +++- internal/smallwebwaf/rproxy_test.go | 4 +- internal/smallwebwaf/servewaf_test.go | 150 ++--- internal/smallwebwaf/smallwebwaf.go | 114 ++-- internal/smallwebwaf/smallwebwaf_test.go | 209 ++---- internal/state/state_test.go | 21 + internal/waf/waf.go | 3 +- script/example-app | 216 +++--- script/gitea-requests | 110 ++-- script/run | 36 +- share/smallwebwaf.run | 34 +- 71 files changed, 2653 insertions(+), 5493 deletions(-) delete mode 100644 internal/proxy/bodies.go delete mode 100644 internal/proxy/client_test.go delete mode 100644 internal/proxy/limits_test.go delete mode 100644 internal/proxy/passthrough_test.go delete mode 100644 internal/proxy/request_internal_test.go delete mode 100644 internal/proxy/timeouts_test.go diff --git a/Dockerfile b/Dockerfile index fc2c64c..7811799 100644 --- a/Dockerfile +++ b/Dockerfile @@ -117,8 +117,9 @@ RUN git checkout --quiet --detach b4b2c785308b1ce785b6155c7fe5f16879080193 \ && CGO_ENABLED=0 go build -trimpath -ldflags="-s -w" \ -o /usr/local/bin/runsvinit . -# The image an app's Dockerfile builds FROM, described under "Deployment" -# in SPEC.md. It is the last stage, so a plain `docker build .` builds it. +# The image, described under "Deployment" in SPEC.md. It runs rproxy unless +# SMALLWEBWAF_COMMAND says servewaf, and an app's Dockerfile can build FROM it. +# It is the last stage, so a plain `docker build .` builds it. # # ubuntu 26.04, 2026-09-27 FROM ubuntu@sha256:f144425ff09be612d6d9ad965196e9cdc23dae1f42110a8a11a3e9a8198759f7 @@ -197,11 +198,11 @@ RUN groupadd --system --gid 65532 smallwebwaf \ # The state files' directory, SWWAF_STATE_DIR by default, where a volume # is mounted to keep them across deploys. The run script gives it to the -# smallwebwaf user at each start. +# smallwebwaf user at each start of servewaf. RUN mkdir /var/lib/smallwebwaf -# The default rule file, in SWWAF_RULES_DIR by default, where an app's -# Dockerfile can copy rule files of its own beside it. +# The default rule file, in SWWAF_RULES_DIR by default, where a Dockerfile +# built FROM this image can copy rule files of its own beside it. COPY share/rules.d/00-default.rules /etc/smallwebwaf/rules.d/00-default.rules # runsvinit starts runit's runsvdir on /etc/service, where Ubuntu's sv diff --git a/README.md b/README.md index 3bc03b1..25298d5 100644 --- a/README.md +++ b/README.md @@ -1,19 +1,29 @@ # smallwebwaf `smallwebwaf` is an MIT-licensed web application firewall, written in Go by -[@sneak](https://sneak.berlin), for one self-hosted web app. It runs inside the -app's own container, in front of the app. It refuses the requests of scrapers, -scanners and other abusers before they reach the app. +[@sneak](https://sneak.berlin), for self-hosted web apps. It refuses the +requests of scrapers, scanners and other abusers before they reach an app. A web application firewall reads each request on its way to a web app and -refuses the ones that look like abuse. `smallwebwaf` is also a reverse proxy: it -accepts requests on the app's behalf, passes them to the app, and passes the -app's answers back. +refuses the ones that look like abuse. `smallwebwaf` is one binary with two +commands that do this together: -It needs no setting. Every `SWWAF_` setting has a default chosen for a service -on the open internet. [`SPEC.md`](SPEC.md) says how `smallwebwaf` behaves and -why. The reference documents under [`docs/`](docs/) are listed under "Documents" -below. +- `smallwebwaf rproxy` is a small reverse proxy in front of one app. It accepts + requests on the app's behalf and asks `servewaf` about each one. It passes the + allowed ones to the app, and the app's answers back. It keeps nothing, and + never touches disk. +- `smallwebwaf servewaf` decides. It runs the checks, and keeps the bans, + counters and history. One `servewaf` serves many sites. + +The site keeps answering while `servewaf` is down, slow or restarting. `rproxy` +waits at most 2 seconds for `servewaf`'s answer, and without one passes the +request to the app. + +`rproxy` requires one setting, `SWRP_WAF_BACKEND`, which says where `servewaf` +is. In a container of its own, it also needs `SWRP_UPSTREAM_URL`, which says +where the app is. Every other setting has a default chosen for a service on the +open internet. [`SPEC.md`](SPEC.md) says how `smallwebwaf` behaves and why. The +reference documents under [`docs/`](docs/) are listed under "Documents" below. ## What it protects against @@ -54,7 +64,7 @@ below. For the operator, it: -- writes one JSON log line for every request, saying what it did and why (see +- logs each request as JSON lines, saying what it did and why (see [`docs/request-log.md`](docs/request-log.md)); - sends alerts to a webhook, Slack or ntfy (see [`docs/alerts.md`](docs/alerts.md)); @@ -65,8 +75,8 @@ For the operator, it: Not built, and not planned: -- TLS and routing for several apps. Both stay with traefik, the reverse proxy in - front of the container. +- TLS and routing by host name. Both stay with traefik, the reverse proxy in + front of `rproxy`. - Browser challenges such as captchas. - A web console. - Defence against floods large enough to fill the host's network link. @@ -84,14 +94,68 @@ cd smallwebwaf make docker ``` -Push the image to a registry your hosts pull from. Then build the app's own -image on it. The app's `Dockerfile` starts `FROM` the `smallwebwaf` image, named -by digest, and adds: +Push the image to a registry your hosts pull from. + +The image runs `rproxy` by default. To run `servewaf` from the same image, set +the container's `SMALLWEBWAF_COMMAND` to `servewaf`. + +### The usual setup: three containers + +The usual setup is three containers on one private network: + +- the app; +- `rproxy`, in front of the app, which traefik routes to; +- `servewaf`, with a volume for its state files. + +```sh +docker network create site + +docker run --detach --name servewaf --network site \ + --env SMALLWEBWAF_COMMAND=servewaf \ + --volume servewaf-state:/var/lib/smallwebwaf \ + /smallwebwaf@sha256: + +docker run --detach --name app --network site \ + /app@sha256: + +docker run --detach --name rproxy --network site \ + --env SWRP_WAF_BACKEND=http://servewaf:8080 \ + --env SWRP_UPSTREAM_URL=http://app:8081 \ + /smallwebwaf@sha256: +``` + +Here the app listens on port 8081, at its container's address on that network. +Point traefik's labels at port 8080 of `rproxy` +(`traefik.http.services..loadbalancer.server.port=8080`). Publish no port +of the app or of `servewaf`. Anything on that network can reach the app around +`rproxy`, and can ask `servewaf` questions. + +With no setting, `servewaf` sends the address of every new visitor to GeoJS, a +web service, to find its country and network. `SWWAF_LOOKUP_SOURCE=off` sends +nothing. `SWWAF_LOOKUP_SOURCE=file`, with `SWWAF_LOOKUP_DB_PATH` naming a +database file on your own host, looks addresses up in that file (see "Lookup of +AS number and country" in [`SPEC.md`](SPEC.md)). + +### Many sites, one `servewaf` + +Run one `rproxy` for each app, all with `SWRP_WAF_BACKEND` naming the same +`servewaf`. It keeps one table of clients and bans for all of them: + +- A client banned through one site is refused on all of them. +- A client's requests on all of them count toward one set of limits. +- Log lines, alerts and ban notes name the site by its host. + +### `rproxy` in the app's container + +An app's image can also be built on the `smallwebwaf` image, so that `rproxy` +runs beside the app. `servewaf` still runs in a container of its own. The app's +`Dockerfile` starts `FROM` the `smallwebwaf` image, named by digest, and adds: - the app's binary; - the packages the app needs, from the nixpkgs in the image; - a user for the app; -- a `run` script that starts the app listening on `127.0.0.1:8081`. +- a `run` script that starts the app listening on `127.0.0.1:8081`, the default + of `SWRP_UPSTREAM_URL`. ```dockerfile # The smallwebwaf image, pinned by digest. @@ -126,38 +190,43 @@ main "$@" ``` Leave `ENTRYPOINT` and `USER` out of the app's `Dockerfile`. The image's own -entrypoint starts `smallwebwaf` and the app side by side, each as its own user. - -Deploy the app's image as you deploy any app. Point traefik's labels at port -8080 (`traefik.http.services..loadbalancer.server.port=8080`). Mount a -volume at `/var/lib/smallwebwaf`, so that bans and history survive a deploy. For -example, with no setting: - -```sh -docker run --detach --name app \ - --volume app-smallwebwaf:/var/lib/smallwebwaf \ - /app@sha256: -``` - -With no setting, the address of every new visitor is sent to GeoJS, a web -service, to find its country and network. `SWWAF_LOOKUP_SOURCE=off` sends -nothing. `SWWAF_LOOKUP_SOURCE=file`, with `SWWAF_LOOKUP_DB_PATH` naming a -database file on your own host, looks addresses up in that file (see "Lookup of -AS number and country" in [`SPEC.md`](SPEC.md)). +entrypoint starts `rproxy` and the app side by side, each as its own user. Run +the app's image with `SWRP_WAF_BACKEND` set. `deploy/example-app` is such an app. `make example-app` builds it and checks it -(see "Checking that it works" below). To run `smallwebwaf` outside the image -while working on it, see "Entrypoints" below. +(see "Checking that it works" below). To run `rproxy` and `servewaf` on the host +while working on them, see `script/run` under "Entrypoints" below. + +### When `servewaf` is down, slow or overloaded + +`SWRP_DEFAULT_DISPOSITION` decides what `rproxy` does with a request when +`servewaf`: + +- cannot be connected to; +- does not answer at all; +- does not answer within `SWRP_TIMEOUT_MS`, 2 seconds by default. + +Its default, `allow`, passes the request to the app without `servewaf`'s answer. +`deny` answers `503` instead. Either way the request gets no report, so its +bytes are not counted and it is not added to its client's history. When +`servewaf` does answer, a `deny` or an error goes to the visitor as `servewaf` +gave it, and `SWRP_DEFAULT_DISPOSITION` plays no part. ## What happens to a request -`smallwebwaf` handles each request in fifteen steps, in this order. Once a step -refuses a request, the steps after it are skipped, apart from step 15, and +A request goes through fifteen steps, in this order. Once `servewaf` refuses a +request, the steps after that are skipped, apart from the report of step 15, and nothing of the request reaches the app. "What happens to a request" in [`SPEC.md`](SPEC.md) gives each step in full, under the same numbers. +`rproxy` takes the first two: + 1. The health check, `GET /_smallwebwaf/healthz`, is answered at once. -2. The client is identified. +2. The visitor's address is worked out, and `servewaf` is asked about the + request. + +`servewaf` takes the checks: + 3. The static lists, `SWWAF_ALLOW_NETS` and `SWWAF_DENY_NETS`. 4. The bans. 5. The lookup of the client's AS number and country. @@ -167,31 +236,36 @@ nothing of the request reaches the app. "What happens to a request" in 9. The rate limits. 10. The trap paths. 11. The rule files. -12. The Core Rule Set. -13. `smallwebwaf`'s own endpoints, under `/_smallwebwaf/`. +12. The header limits, and then the Core Rule Set. +13. A path under `/_smallwebwaf/` is answered `404`, so that the app never sees + that prefix. + +`rproxy` takes the rest, with `servewaf` counting at the end: + 14. The request is passed to the app. -15. After the answer: the byte limits, the error burst and the client's history. +15. After the answer, `rproxy` sends `servewaf` a report. For an allowed + request, `servewaf` counts the byte limits and adds the request to the + client's history. A refused visitor is answered with `SWWAF_BAN_RESPONSE`, `403` by default (see "Ban lengths, response and scope" in [`SPEC.md`](SPEC.md)). A `block` rule or a -Core Rule Set match answers `403`. A missing or wrong token gets `401`. A -request over a size or time limit gets the status for that limit, such as `413` -for a body too large (see "Size and time limits" in [`SPEC.md`](SPEC.md)). +Core Rule Set match answers `403`. `rproxy` itself answers `405` to a method it +does not pass on, and the status for a size or time limit, such as `413` for a +body too large (see "`rproxy`" in [`SPEC.md`](SPEC.md)). ## Settings you are likely to touch -Each setting is an environment variable of the app's container, such as -`docker run -e SWWAF_MODE=observe ...`, read at start. +Each setting is an environment variable, read at start. Those of `servewaf` +start with `SWWAF_`, and those of `rproxy` with `SWRP_`. [`docs/settings.md`](docs/settings.md) lists every setting, its default and the values it takes. The ones most deployments touch: -- `SWWAF_MODE=observe`: `smallwebwaf` makes no ban of its own, and refuses a - request only for a size or time limit or a missing or wrong token. A request - it would have refused is passed to the app, and its log line says what - `enforce` mode would have done. For trying a configuration before enforcing - it. +- `SWWAF_MODE=observe`: `servewaf` refuses nothing and makes no ban of its own. + A request it would have refused is passed to the app, and its log line says + what `enforce` mode would have done. For trying a configuration before + enforcing it. The limits of `rproxy` still apply. - `SWWAF_ALLOW_NETS`: netblocks you trust, such as your monitoring. Their - clients are refused only for a size or time limit or a missing or wrong token. + clients skip `servewaf`'s checks. - `SWWAF_RATE_LIMIT_EXEMPT_NETS`: netblocks exempt from rate limiting and the byte limits, such as a machine that talks to the app all day. - `SWWAF_TRAP_PATHS`: paths only scanners ask for, such as @@ -201,39 +275,49 @@ values it takes. The ones most deployments touch: refuses your app's ordinary requests. The log line names the rules a request matched. Keep the six default ids in a list you give (see [`docs/settings.md`](docs/settings.md)). -- `SWWAF_REQUEST_MAX_BYTES`, `SWWAF_CLIENT_REQUEST_TIMEOUT` and - `SWWAF_UPSTREAM_REQUEST_TIMEOUT`: raise them for an app that takes uploads - over 100 MiB or slower than 60 seconds. +- `SWRP_MAX_REQUEST_SIZE` and `SWRP_MAX_REQUEST_TIME_MS`: raise them for an app + that takes uploads over 100 MiB or slower than 60 seconds. +- `SWRP_DEFAULT_DISPOSITION`: `allow` (the default) or `deny`, as above. +- `SWRP_TRUSTED_PROXIES`: the netblocks whose `X-Forwarded-For` `rproxy` + believes, by default the private address ranges. - `SWWAF_LOOKUP_SOURCE`: `geojs` (the default), `file` or `off`, as above. - `SWWAF_ADMIN_TOKEN` and `SWWAF_METRICS_TOKEN`: switch on the ban endpoints and - the metrics. Give them as files to keep them from the app. + the metrics, at `servewaf`'s own address. - `SWWAF_ALERT_WEBHOOK_URL`, `SWWAF_ALERT_SLACK_WEBHOOK_URL` or `SWWAF_ALERT_NTFY_URL`: where alerts go. -- `SWWAF_INSTANCE_NAME`: the name in every log line, alert and metric, such as - `fsn1app1/gitea`. +- `SWWAF_INSTANCE_NAME`: the name in every line `servewaf` logs, and in every + alert and metric, such as `fsn1/servewaf`. ## Checking that it works - `docker inspect --format '{{.State.Health.Status}}' ` says - `healthy` once `smallwebwaf` answers and the app accepts connections. A log - line with `"msg":"starting"` gives the version, the address and every setting - in effect, with secrets masked. -- A request to the site through traefik gets the app's answer, and its log line - has `"action":"forward"`. `https:///_smallwebwaf/healthz` answers `ok`. -- To see a refusal without banning yourself, run the container with - `SWWAF_MODE=observe` and ask for `/.env`. The request reaches the app. Its log - line has `"would_action":"banned"` and `"rule_ids":["env-file"]`. In `enforce` - mode the same request bans your address for seven days, and your next request - makes the ban permanent. Lift the ban by editing `bans.json` (see "Bans" in - [`SPEC.md`](SPEC.md)). -- With `SWWAF_METRICS_TOKEN` set, - `curl -H "Authorization: Bearer $TOKEN" https:///_smallwebwaf/metrics` - answers with the metrics. -- `make example-app` builds the image and the example app on it, runs the app, - and checks it. It needs network access. It checks: - - the health check; - - a request forwarded to the app; + `healthy` for each container of `smallwebwaf`: + - for `rproxy`, once it answers and the app accepts connections, whether or + not `servewaf` answers; + - for `servewaf`, once it answers. +- A log line with `"msg":"starting"` gives the version, the address and every + setting in effect, with secrets masked. +- A request to the site through traefik gets the app's answer. + `https:///_smallwebwaf/healthz` answers `ok`. `servewaf`'s log lines + about the request have `"action":"forward"`. +- To see a refusal without banning yourself, run `servewaf` with + `SWWAF_MODE=observe` and ask for `/.env`. The request reaches the app. + `servewaf`'s log lines have `"would_action":"banned"` and + `"rule_ids":["env-file"]`. In `enforce` mode the same request bans your + address for seven days, and your next request makes the ban permanent. Lift + the ban by editing `bans.json` (see "Bans" in [`SPEC.md`](SPEC.md)). +- With `SWWAF_METRICS_TOKEN` set, this answers with the metrics, from a host on + `servewaf`'s network: + `curl -H "Authorization: Bearer $TOKEN" http://servewaf:8080/_smallwebwaf/metrics`. +- `make example-app` builds the image and the example app on it, runs the app + with `rproxy` in its container and `servewaf` in another, and checks them. It + needs network access. It checks: + - the health checks; + - a request passed to the app; - a ban for a broken limit, and a ban for a probe made permanent; + - that only `servewaf`'s container changes the state directory's owner; + - with `servewaf` stopped, the app still answering through `rproxy`; + - with `servewaf` started again, bans applying again; - orderly stops with `sv stop` and `docker stop`; - bans surviving a new container on the same volume; - a setting given as a file. @@ -276,21 +360,26 @@ on the host. - `script/install-precommit`: installs that hook. `make hooks`. - `script/build`: builds `bin/smallwebwaf` with the Go on your host. `make build`. -- `script/run`: builds `bin/smallwebwaf` and runs it with the settings in your - environment, listening on port 8080 in front of an app at - `SWWAF_UPSTREAM_URL`. Its state files go in `bin/state` and its rule files are - those of `share/rules.d`, unless `SWWAF_STATE_DIR` and `SWWAF_RULES_DIR` say - otherwise. `make run`. -- `script/example-app`: builds the image and the example app on it, runs it with - a volume, and checks it as "Checking that it works" above describes. Then it - removes the containers, volumes and images. Needs network access. - `make example-app`. -- `script/gitea-requests`: builds the image and runs gitea 1.27.3 behind it at - the defaults. It sends through it a sample of gitea's ordinary use and the - requests the Core Rule Set is known to refuse, and prints what `smallwebwaf` - did with each. Needs network access, and a docker daemon that pushes to a - registry on `localhost` over plain HTTP. [`docs/gitea.md`](docs/gitea.md) - records the results. `make gitea-requests`. +- `script/run`: builds `bin/smallwebwaf` and runs `servewaf` and `rproxy` on the + host, with the settings in your environment. `make run`. Unless your settings + say otherwise: + - `rproxy` listens on port 8080, in front of an app at + `http://127.0.0.1:8081`; + - `servewaf` listens on port 8090; + - its state files go in `bin/state`; + - its rule files are those of `share/rules.d`. +- `script/example-app`: builds the image and the example app on it, runs them + with a volume, and checks them as "Checking that it works" above describes. + Then it removes the containers, the network, the volumes and the images. Needs + network access. `make example-app`. +- `script/gitea-requests`: builds the image and runs gitea 1.27.3 behind + `rproxy` and `servewaf`, three containers on one network. Every setting is at + its default, but for those that join the containers and the header that labels + each request. It sends through them a sample of gitea's ordinary use and the + requests the Core Rule Set is known to refuse, and prints what `servewaf` did + with each. Needs network access, and a docker daemon that pushes to a registry + on `localhost` over plain HTTP. [`docs/gitea.md`](docs/gitea.md) records the + results. `make gitea-requests`. ## Rationale @@ -303,16 +392,16 @@ Small self-hosted sites receive a great deal of traffic nobody asked for: Most of it comes from a small number of hosting networks and countries. A single-person operation has no abuse desk and no CDN contract. It needs -something small that can be put in front of one service and left alone. +something small that can be put in front of its services and left alone. The existing tools each solve part of this. Rule-based firewalls catch attack payloads but do not limit request rates. Rate limiters count requests but cannot tell a residential visitor from a rented server farm. The products that do most -of it want several containers, a database and a web console. None of them can -say "clients from these networks are banned after half as many requests as -anyone else". That is the useful thing to say when nearly all abuse comes from a -known list of AS numbers. [`EVALUATION.md`](EVALUATION.md) goes through the -candidates one by one. +of it want a database and a web console. None of them can say "clients from +these networks are banned after half as many requests as anyone else". That is +the useful thing to say when nearly all abuse comes from a known list of AS +numbers. [`EVALUATION.md`](EVALUATION.md) goes through the candidates one by +one. `smallwebwaf` fills that gap with: @@ -320,8 +409,8 @@ candidates one by one. - lower limits for the networks and countries abuse comes from; - bans that grow with repetition; - a log that is easy to search and ship; -- one binary in the app's own container, with no database and no required - setting. +- a site that keeps answering while the part that decides is down; +- one binary, with no database and one required setting. ## Design @@ -331,47 +420,36 @@ nixpkgs fixed at one release. An app's `Dockerfile` installs a package with [`docs/image.md`](docs/image.md) lists what the image holds and how each part is pinned. -- runit, a service supervisor, runs `smallwebwaf` and the app side by side, each - as its own user, and starts either again when it exits. The app cannot touch - the state files or the process of `smallwebwaf`. -- `smallwebwaf` listens on port 8080 and passes requests to the app on - `127.0.0.1:8081`. The app listens on `127.0.0.1` only, so nothing reaches it - except through `smallwebwaf`. `smallwebwaf`'s own endpoints are under - `/_smallwebwaf/`, on the same port. +- runit, a service supervisor, runs `smallwebwaf` and starts it again when it + exits. In an app's image built on the `smallwebwaf` image, it runs `rproxy` + and the app side by side, each as its own user. +- `rproxy` listens on port 8080 and passes requests to the app at + `SWRP_UPSTREAM_URL`. It answers the health check itself, and the refusals and + errors "`rproxy`" in [`SPEC.md`](SPEC.md) lists. +- `servewaf` listens on port 8080 of its own container. Nothing should route the + internet there. Its own endpoints, under `/_smallwebwaf/`, are for your admin + tools and your metrics collector. +- `rproxy` and `servewaf` use no token between them. Anything on their private + network can ask `servewaf` a question, or send it a report. - The app sees the headers traefik sent, with traefik's address added to - `X-Forwarded-For`, from a TCP peer on `127.0.0.1`. So the app must trust - `127.0.0.1` and `::1` for forwarded headers, besides the private address - ranges. -- Both processes start with the container's environment variables, so every - setting of this `smallwebwaf` process starts with `SWWAF_`, and the app can - read a token given as one. A token given as a file that only the `smallwebwaf` - user can read is out of its reach (see "Settings given as files" in - [`docs/settings.md`](docs/settings.md)). -- The state files live in `/var/lib/smallwebwaf`. A volume mounted there keeps - them across a deploy. -- `smallwebwaf servewaf` runs the same checks for the sites that ask it about - each request, and passes nothing to an app (see "`servewaf`" in - [`SPEC.md`](SPEC.md)). -- `smallwebwaf rproxy` is the reverse proxy in front of one app that asks - `smallwebwaf servewaf` about each request. When `servewaf` cannot be connected - to, does not answer at all, or does not answer within `SWRP_TIMEOUT_MS`, - `SWRP_DEFAULT_DISPOSITION` decides. Its default, `allow`, keeps the site - answering. The settings of `rproxy` start with `SWRP_`, and it needs - `SWRP_WAF_BACKEND`, `servewaf`'s URL (see "`rproxy`" in [`SPEC.md`](SPEC.md)). -- `docker stop` stops both processes at once. `smallwebwaf` gives the requests - in progress five seconds to finish, so the app must keep answering for five - seconds after `SIGTERM`. + `X-Forwarded-For`, from a TCP peer that is `rproxy`. So the app must trust + `rproxy`'s address for forwarded headers. With `rproxy` in the app's + container, that address is `127.0.0.1` or `::1`, which the private address + ranges leave out. +- The state files live in `/var/lib/smallwebwaf` in `servewaf`'s container. A + volume mounted there keeps them across a deploy. +- `docker stop` gives requests in progress five seconds to finish. With `rproxy` + in the app's container, the app must keep answering for five seconds after + `SIGTERM` for that to help them. -"Deployment" in [`SPEC.md`](SPEC.md) has the rest: the health check, what -happens when a process stops, moving `smallwebwaf` to another port, and the -forwarded headers. "Architecture" in [`SPEC.md`](SPEC.md) lists the packages and -what each does. +"Deployment" in [`SPEC.md`](SPEC.md) has the rest: the health checks, what +happens when a process stops, moving a port, and the forwarded headers. +"Architecture" in [`SPEC.md`](SPEC.md) lists the packages and what each does. ## TODO -- Planned, as "Planned, not built" in [`SPEC.md`](SPEC.md) describes: - - sharing bans between the instances of several apps; - - sending the log over RELP. +- Planned, as "Planned, not built" in [`SPEC.md`](SPEC.md) describes: sending + the log over RELP. ## Documents diff --git a/SPEC.md b/SPEC.md index 4980d1f..d2d6f01 100644 --- a/SPEC.md +++ b/SPEC.md @@ -1,20 +1,27 @@ # smallwebwaf specification -`smallwebwaf` is a web application firewall and reverse proxy for one app. It -runs inside the app's container, between traefik and the app. This document says -what it must do and why, each requirement once. Everything in it is built, -except what the last section lists. `README.md` says how to run it. +`smallwebwaf` is a web application firewall for self-hosted apps. It is one +binary, and two of its commands do the work: + +- `smallwebwaf rproxy` is a reverse proxy between traefik and one app. +- `smallwebwaf servewaf` decides what happens to each request, for many sites. + +This document says what it must do and why, each requirement once. Everything in +it is built, except what the last section lists. `README.md` says how to run it. `EVALUATION.md` says why no existing tool was adopted. The reference documents under `docs/` give the settings, the formats and the lists. This document links to them. ## Purpose -- It protects one self-hosted app on the open internet from scrapers, scanners - and other abusers. It does so from the app's first request, with no setting at - all. Every `SWWAF_` setting has a default chosen for a service facing the +- It protects self-hosted apps on the open internet from scrapers, scanners and + other abusers. It does so from an app's first request. `rproxy` requires only + the address of `servewaf`. In a container of its own it also needs the app's + address. Every other setting has a default chosen for a service facing the internet in 2026. A setting that needs an alert destination, an account key or a token stays off until the operator supplies it. +- It keeps a site answering while the part that decides is down, slow or + restarting (see "Without an answer from `servewaf`"). - It limits each client's requests and bytes per minute, hour and day. It bounds the size and duration of every request and answer. It detects attacks. - It bans abusers: briefly at first, for seven days on a clear sign of attack, @@ -22,46 +29,51 @@ to them. - It looks up each client's AS number and country. It can refuse whole countries and netblocks, lower its limits for listed AS numbers and countries, consult reputation sources, and send alerts. -- Everything is set by environment variables whose names start with `SWWAF_`, - since `smallwebwaf` shares its container, and so its environment, with the - app. `smallwebwaf rproxy` uses `SWRP_` in their place. The only other inputs - are files: the rule files, the state files, a lookup database, and a `SWWAF_` - setting given as a file. +- Everything is set by environment variables. The names of `servewaf`'s start + with `SWWAF_`, and those of `rproxy` with `SWRP_`. The only other inputs are + `servewaf`'s files: + - the rule files; + - the state files; + - a lookup database; + - a `SWWAF_` setting given as a file; + - the file `SWWAF_LOG_REMOTE_TLS_CA_FILE` names. ## Non-goals - TLS, certificates and routing by host name. Those are traefik's job. -- More than one app per `smallwebwaf`. Each app's image carries its own. +- More than one app behind one `rproxy`. Each app has its own. - Browser challenges such as captchas or proof of work. If wanted, chain Anubis - between `smallwebwaf` and the app. + between `rproxy` and the app. - Defence against floods that saturate the host's network link. That needs help upstream of the host. - A web console or a configuration file. ## Architecture -- One statically linked Go binary. It writes only to its state directory. It - ships in the image that is the base of the app's own image (see "Deployment"). +- One statically linked Go binary. Only `servewaf` writes to disk, and only to + its state directory. It ships in one image, which runs either command (see + "Deployment"). - No database of any kind. Nothing is read from disk while serving a request. - Packages: - `cmd/smallwebwaf`: the binary, which only calls `internal/smallwebwaf`. - - `internal/smallwebwaf`: the process. It reads the settings, the rule - files, the lookup database and the state files, listens, and serves until - `SIGTERM` or `SIGINT`. Then it stops, writing the state files. Run as - `smallwebwaf servewaf`, it answers `rproxy` instead of passing requests to - an app (see "`servewaf`"). Run as `smallwebwaf rproxy`, it reads only that - command's settings, and runs `internal/rproxy` until `SIGTERM` or `SIGINT` - (see "`rproxy`"). Run as `smallwebwaf healthcheck`, it is the image's - health check instead. + - `internal/smallwebwaf`: the process, as the command it is given. + `smallwebwaf servewaf` reads the settings, the rule files, the lookup + database and the state files, listens, and serves until `SIGTERM` or + `SIGINT`. Then it stops, writing the state files (see "`servewaf`"). + `smallwebwaf rproxy` reads only that command's settings, and runs + `internal/rproxy` until `SIGTERM` or `SIGINT` (see "`rproxy`"). + `smallwebwaf healthcheck` is the image's health check. With no command, or + one it does not know, the binary prints the commands and exits with + status 2. - `internal/config`: reads the settings, the one place they are read. - - `internal/proxy`: what happens to each request. It works out the client - and runs the checks in `check`. It passes the request to the app with the - standard library's `httputil.ReverseProxy`, within the timeouts and size - limits, and answers its own endpoints in `answerAdmin`. Once the request - has ended, it counts bytes, refusals and anomalies, and writes the log - line. It also holds `servewaf`'s question, answer and report. + - `internal/proxy`: `servewaf`'s server. It runs the checks in `check`, and + holds `servewaf`'s question, answer and report. It answers `servewaf`'s + own endpoints in `answerOwnEndpoint`. It counts bytes, refusals and + anomalies, and writes the log lines. - `internal/rproxy`: the reverse proxy in front of one app that asks - `servewaf` about each request. + `servewaf` about each request. It passes a request to the app with the + standard library's `httputil.ReverseProxy`, within the time and size + limits. - `internal/ratelimit`: the table of clients. It counts requests, bytes and refusals, and keeps each client's history. - `internal/bans`: the bans. It holds each netblock's bans with their notes, @@ -85,10 +97,10 @@ to them. sending them to the syslog server. - `internal/metrics`: the metrics, in the Prometheus text format. - `Dockerfile`: the lint and test phases, then the image. - `deploy/example-app` is an app built on the image, which - `script/example-app` checks. + `deploy/example-app` is an app built on the image, with `rproxy` in its + container, which `script/example-app` checks with `servewaf` in another. - Libraries, checked against the owner's Go dependency defaults: - - the standard library for the proxy, HTTP clients, DNS, logging + - the standard library for the reverse proxy, HTTP clients, DNS, logging (`log/slog`) and the state files (`encoding/json`, `os.Rename`); - `github.com/corazawaf/coraza/v3` and `github.com/corazawaf/coraza-coreruleset/v4` at `v4.25.0`, which carries @@ -106,28 +118,31 @@ to them. ## What happens to a request -The steps run in this order. The first step that produces a final answer ends -the checks; step 15 runs for every request but the health check. The size and -time limits apply to the whole exchange. Each step names the section that holds -its rules. +`rproxy` runs steps 1, 2 and 14, and sends the report of step 15. `servewaf` +runs steps 3 to 13, and does the counting of step 15. The first step that +produces a final answer ends the checks. Step 15 runs for each request +`servewaf` answers with `allow`, `deny` or `close`. `rproxy`'s size and time +limits apply to the whole exchange. Each step names the section that holds its +rules. -1. `GET /_smallwebwaf/healthz` is answered at once, before any check, so a - health checker never needs an exemption and is never refused. +1. `GET /_smallwebwaf/healthz` is answered at once by `rproxy`, before any + check, so a health checker never needs an exemption and is never refused. 2. The client is identified. - - If the TCP peer is inside `SWWAF_TRUSTED_PROXIES`, `X-Forwarded-For` is + - If the TCP peer is inside `SWRP_TRUSTED_PROXIES`, `X-Forwarded-For` is walked from the right, and the first address outside those netblocks is the client. If every address is inside them, the leftmost is, so that a - visitor on a private network is known by its own address. With no header, - the peer is the client. - - If the peer is outside `SWWAF_TRUSTED_PROXIES`, it is the client, and the + visitor on a private network is known by its own address. An entry that is + not an address ends the walk. With no header, the peer is the client. + - If the peer is outside `SWRP_TRUSTED_PROXIES`, it is the client, and the header is ignored. + - `rproxy` then sends `servewaf` the question (see "The question"). + `servewaf` takes the address the question gives as the client. - An IPv6 client is its group: the netblock of `SWWAF_IPV6_GROUP_PREFIX` around its address, a /64 by default, since one abuser usually controls a whole /64. A client is therefore one IPv4 address or one IPv6 group. -3. The static lists. A client in `SWWAF_ALLOW_NETS` skips steps 4 to 12 and the - byte limits and the error burst at step 15. It still needs a token at - step 13. A client in `SWWAF_DENY_NETS` is refused; a client in both lists is - let through. +3. The static lists. A client in `SWWAF_ALLOW_NETS` skips steps 4 to 12, the + byte limits and the error burst. A client in `SWWAF_DENY_NETS` is refused; a + client in both lists is let through. 4. Bans. A client under a ban is refused (see "Bans"). 5. The lookup, unless `SWWAF_LOOKUP_SOURCE` is `off` (see "Lookup of AS number and country"). The request goes on at once. It waits for the first GeoJS @@ -150,29 +165,36 @@ its rules. bans the client. 11. The rule files (see "Rule files"). Matching stops at the first rule that refuses. -12. The Core Rule Set (see "Attack detection"). A match is refused with `403` - while `SWWAF_WAF_MODE` is `block`, and only logged and alerted while it is +12. The header limits (see "Answering in time"), and then the Core Rule Set (see + "Attack detection"). A Core Rule Set match is refused with `403` while + `SWWAF_WAF_MODE` is `block`, and only logged and alerted while it is `detect`. -13. A request under `/_smallwebwaf/` is answered here (see "Admin endpoints"). -14. The request is forwarded to `SWWAF_UPSTREAM_URL`, streaming. - `X-Forwarded-For` is added, and `X-Request-ID` carries the request's id, - generated when traefik did not supply one. With `SWWAF_ADD_LOOKUP_HEADERS`, - `X-Client-ASN` and `X-Client-Country` are added for the app's own logs; a - header whose value is unknown is left out. Whatever that setting says, any - such header the client sent is removed. -15. After the response, or after the refusal: - - For a request forwarded to the app, the response bytes, the request bytes - or both are added to the byte counters, as `SWWAF_BYTES_COUNT` says. For a +13. A path under `/_smallwebwaf/` is answered `404`, so that the app never sees + that prefix. +14. `rproxy` passes the request to `SWRP_UPSTREAM_URL`, streaming (see "Passing + a request to the app"). With `SWWAF_ADD_LOOKUP_HEADERS`, `X-Client-ASN` and + `X-Client-Country` are added for the app's own logs; a header whose value is + unknown is left out. Whatever that setting says, any such header the client + sent is removed. +15. After the visitor's answer has ended, `rproxy` sends the report (see "The + report"). `servewaf` then counts: + - For a request it allowed, the response bytes, the request bytes or both + are added to the byte counters, as `SWWAF_BYTES_COUNT` says. For a WebSocket, what it carried each way counts once it closes. A client whose total passes a byte limit times the percentage has broken that limit and is banned. The response in progress is not cut off. - - The refusals the error burst counts (see "Attack detection") are counted. - - The client's history and the anomaly counters are updated, and the alert - thresholds are evaluated. + - For a request it allowed, the client's history and the anomaly counters + are updated, and the alert thresholds are evaluated. + + For a request it refused, `servewaf` adds nothing to the byte limits. It + updates the history and the anomaly counters at its decision. It counts the + refusals the error burst counts at the decision too (see "Attack + detection"). In `SWWAF_MODE=observe`, every refusal above is logged as what would have -happened, nothing is refused, and `smallwebwaf` makes no ban of its own. The -size and time limits and the token check still apply. +happened, nothing is refused, and `servewaf` makes no ban of its own. A path +under `/_smallwebwaf/` is still answered `404`, and `rproxy`'s size and time +limits still apply. ## Counting method @@ -202,12 +224,11 @@ address or one /64, so a permanent ban does not reach neighbours who did nothing. Requests refused under a ban are not counted against limits. They are counted in the client's history and in the ban's notes. -An offence is a request `smallwebwaf` holds against the client. It is one of: +An offence is a request `servewaf` holds against the client. It is one of: - a clear sign of attack; - a broken limit; -- a refusal by a rule file, by the Core Rule Set or for a missing or wrong - token. +- a refusal by a rule file or by the Core Rule Set. Offences are counted by kind in the client's history. The first two kinds lead to a ban, each by its own rule: @@ -261,20 +282,18 @@ no offence: - the country lists. Every ban carries notes with what an admin needs to decide whether to lift it, -drawn from what `smallwebwaf` already knows. No lookup is made for the notes -alone. `docs/state-files.md` lists the notes with the rest of a ban's entry. +drawn from what `servewaf` already knows. No lookup is made for the notes alone. +`docs/state-files.md` lists the notes with the rest of a ban's entry. An admin adds, keeps or lifts a ban by editing `bans.json` (see "Persistent state") or through the ban endpoints (see "Admin endpoints"). A ban an admin -makes has the cause `admin`; an admin who wants to keep a ban `smallwebwaf` made +makes has the cause `admin`; an admin who wants to keep a ban `servewaf` made sets its cause to `admin`. A ban lifted before it ends is kept, marked as lifted, and does not count toward a longer ban; it refuses nothing. Deleting its -entry from `bans.json` forgets it entirely. An admin whose own address is banned -lifts the ban by editing `bans.json`, since the endpoints refuse a banned -client's request like any other. +entry from `bans.json` forgets it entirely. A permanent ban does not run out. It ends when an admin lifts it, or when -`smallwebwaf` drops it from a full table, as below. When `smallwebwaf` holds +`servewaf` drops it from a full table, as below. When `servewaf` holds `SWWAF_MAX_BANS` bans of its own, it drops the earliest ban of one netblock: the netblock that has gone longest without a request. A past, active or permanent ban is dropped alike. A scanner whose ban was dropped that way is refused and @@ -285,13 +304,12 @@ table. ## Settings `docs/settings.md` lists every setting with its default and the values it takes, -in the groups below. It also gives the form each kind of value takes. +in the groups below. It also gives the form each kind of value takes. The groups +below are `servewaf`'s settings; "`rproxy`" gives the use of `rproxy`'s. Conventions: - These settings take `off`, which switches them off: - `SWWAF_LOOKUP_SOURCE` and `SWWAF_WAF_MODE`; - - the five timeouts under "Size and time limits"; - - `SWWAF_REQUEST_MAX_BYTES` and `SWWAF_RESPONSE_MAX_BYTES`; - the three rate limits and the three byte limits; - `SWWAF_WAF_ANOMALY_THRESHOLD`, `SWWAF_WAF_BODY_LIMIT` and `SWWAF_ERROR_BURST_THRESHOLD`; @@ -311,26 +329,22 @@ Conventions: already holds a file name, so it has no `_FILE` form. No `SWRP_` setting has one, since `rproxy` never touches disk. - Settings, files included, are read once at start. Changing one means - restarting the container. The files `smallwebwaf` watches while it runs are - its state files, its rule files and the lookup database. + restarting the container. The files `servewaf` watches while it runs are its + state files, its rule files and the lookup database. - An invalid setting stops the process at start with one log line, marked `"type":"process"` with `"msg":"invalid setting"`. The line names the variable - and what is wrong with the value; "The two processes" says what happens then. - At start the settings in effect are logged on one line; tokens, keys and the - paths of webhooks, which often carry a secret, are shown as `********`. + and what is wrong with the value; "Each container's processes" says what + happens then. At start the settings in effect are logged on one line; tokens, + keys and the paths of webhooks, which often carry a secret, are shown as + `********`. ### Core -- `SWWAF_INSTANCE_NAME` is in every log line, metric and alert, so that - instances can be told apart. The default, the container's host name, changes - at every redeploy; a name set by the operator survives it. +- `SWWAF_INSTANCE_NAME` is in each line `servewaf` logs, and in each metric and + alert, so that instances can be told apart. The default, the container's host + name, changes at every redeploy; a name set by the operator survives it. - `SWWAF_MODE=observe` lets a configuration be tried before it is enforced (see "What happens to a request"). -- `X-Forwarded-For` is believed only from `SWWAF_TRUSTED_PROXIES`, by default - the private address ranges, so a client that connects directly cannot claim - another address. Where other containers reach `smallwebwaf` directly, or - visitors on a private network reach traefik, they can name another address in - the header. Narrowing the setting to traefik's own network closes that gap. - `SWWAF_IPV6_GROUP_PREFIX` sets the group size. After a change, each IPv6 client starts afresh; existing bans keep refusing their netblocks until they end. `SWWAF_MAX_TRACKED_CLIENTS` and `SWWAF_MAX_BANS` set the bounds under @@ -340,14 +354,12 @@ Conventions: - `SWWAF_ADMIN_TOKEN` switches on the ban endpoints and `/_smallwebwaf/clients/`. The metrics are off until `SWWAF_METRICS_TOKEN` - is set, since without a token they would be open to anyone on the internet. - The two are separate, so the metrics collector, which holds the metrics token, - cannot manage bans. -- A token is a long random value, since the endpoints can be reached from the - internet. One shorter than 32 characters stops the start. The app starts with - the same environment variables as `smallwebwaf`, so it can read a token set as - an environment variable. The file form keeps a token out of the app's reach - (see "Tokens" under "Deployment"). + is set, since without a token they would be open to anything on `servewaf`'s + network. The two are separate, so the metrics collector, which holds the + metrics token, cannot manage bans. +- A token is a long random value. One shorter than 32 characters stops the + start. The file form keeps a token out of the container's environment (see + "Tokens" under "Deployment"). ### State files @@ -364,7 +376,7 @@ Conventions: at `debug`. `warn` and `error` hold back the line of settings at start. - `SWWAF_LOG_REQUEST_HEADERS` names the request headers to record. `Authorization`, `Cookie` and `Set-Cookie` values are never logged, even if - listed, only whether they were present. + listed. The log says only whether `Authorization` and `Cookie` were present. - `SWWAF_LOG_REMOTE_URL` names a syslog server (see "Request log"). ### Metrics @@ -426,51 +438,9 @@ matched the same way. The hour and day limits sit about twice above what 40 people busy with heavy web apps move together on one address. Each moves about 256 MiB an hour, for ten hours a day. A response's bytes are counted when it ends (see step 15). So the -minute limit holds two of the largest responses, with their requests, ending in -the same minute. No single download then breaks a limit. An operator who raises -`SWWAF_RESPONSE_MAX_BYTES` raises the byte limits with it. - -### Size and time limits - -These apply per request, in both directions. The client-facing timeouts apply -between the client and `smallwebwaf`, the app-facing ones between `smallwebwaf` -and the app, since a slow client and a slow app are separate problems. There is -one size limit for request bodies and one for response bodies. Bodies pass -through unchanged, so a limit on each side would bound the same bytes, and the -lower one would always decide. Bodies stream straight through, so a request body -reaches the app while the client is still sending it, except the part the Core -Rule Set reads while `SWWAF_WAF_BODY_LIMIT` is set. - -- `SWWAF_CLIENT_REQUEST_HEADER_MAX_BYTES` must be more than `4K` and cannot be - `off`. Go's HTTP server always has such a limit, and reads 4 KiB past the one - it is given before it refuses. -- `SWWAF_CLIENT_IDLE_TIMEOUT` is longer by default than the 90 seconds after - which traefik, by default, closes an unused connection. So traefik closes - first, and never sends a request on a connection `smallwebwaf` is closing. - -What the client gets when a limit is passed: - -- Go's HTTP server reads a request's line and headers before `smallwebwaf` sees - the request. It answers `431` to headers over - `SWWAF_CLIENT_REQUEST_HEADER_MAX_BYTES`, and a client too slow to send them - gets an error answer, mostly `400`, or none. Either way the server closes the - connection, nothing reaches the app, and the request log gets no line. -- A request that announces a body larger than its limit is refused before - anything reaches the app. -- A limit passed after the headers have been read, but before the response has - started, is answered by `smallwebwaf`: - - `413` for a request body that is too large; - - `408` for a client too slow to send its body; - - `502` for a response that is too large, or an app that cannot be reached; - - `504` for an app that is too slow. -- A request body streams through, so each side can hold up the other. A request - timeout can run out while the body is on its way. The answer is then `408` if - `smallwebwaf` was waiting for the client, and `504` if it was waiting for the - app. -- Once the response has started it can only be cut off, and the connection is - closed. -- A WebSocket connection leaves these limits behind once it is upgraded. It - stays open until either side closes it. +minute limit holds three of the largest responses, with their requests, ending +in the same minute. No single download then breaks a limit. An operator who +raises `SWRP_MAX_RESPONSE_SIZE` must raise the byte limits too. ### Lookup of AS number and country @@ -508,14 +478,14 @@ service that keeps its visitors' addresses on its own host. else treats it differently. - GeoJS publishes no rate limit, but its terms forbid "an excessive amount of API requests", judged by GeoJS alone, and let it block a caller. While GeoJS - is slow, down or refusing `smallwebwaf`, new clients count as unknown. GeoJS - is then left alone for a second, and twice as long after each further failure, - up to five minutes. Each failure raises a `source_failure` alert (see + is slow, down or refusing `servewaf`, new clients count as unknown. GeoJS is + then left alone for a second, and twice as long after each further failure, up + to five minutes. Each failure raises a `source_failure` alert (see "Alerting"). A service that cannot accept this uses the file. - The file is the free IPinfo Lite database (`ipinfo_lite.mmdb`), whose `asn`, `as_name` and `country_code` fields are read. The operator downloads it with a - free IPinfo account and mounts it read-only; `smallwebwaf` never fetches it - and holds no account or token for it. IPinfo updates it daily. Its licence is + free IPinfo account and mounts it read-only; `servewaf` never fetches it and + holds no account or token for it. IPinfo updates it daily. Its licence is Creative Commons Attribution-ShareAlike 4.0, and IPinfo asks for a link to it on the site that uses it, such as "IP address data is powered by IPinfo". - The file is read whole at start; one that is missing or unreadable stops the @@ -542,8 +512,8 @@ where it comes from. exclusive list already refuses every other country. A code on both lists stops the start with a message naming it. - A refused request is answered with `SWWAF_BAN_RESPONSE`, as soon as the - client's address has been looked up, before its body is read. It is logged - with the action `country_denied`, and counted in the metrics. + client's address has been looked up. It is logged with the action + `country_denied`, and counted in the metrics. ### Biased thresholds @@ -563,7 +533,7 @@ service's abuse comes from. `SWWAF_ASN_LIMIT_PERCENT` and `SWWAF_ASN_LIMIT_PERCENT_URL`; for a country, the one from `SWWAF_COUNTRY_LIMIT_PERCENT`. - `SWWAF_ASN_LIMIT_PERCENT_URL` names a text file of `AS:percent` lines, so one - abuse-source list can be shared by every `smallwebwaf` in the fleet. It is + abuse-source list can be shared by every `servewaf` in the fleet. It is fetched and kept like a blocklist (see "Reputation"). Where it and `SWWAF_ASN_LIMIT_PERCENT` both give a percentage, the lower applies. - A client's limit percentage is the lowest of those that apply to it: its AS @@ -616,16 +586,16 @@ sends it. probes these rules caught (see "Rule files"). A list you give replaces the default, so name these six ids in it to keep them off. -- `SWWAF_WAF_BODY_LIMIT` switches body inspection on, up to that size. Form data - and multipart bodies are read up to the size, and the rest streams on without - being held in memory. JSON and XML bodies are read only when no larger than - the size, since those cannot be read in part. Any other body is not read: the - Core Rule Set would read it as form data, where binary content such as a git - push trips rules written for text. A body the Core Rule Set cannot parse adds - 5 to the score, and so does a multipart body cut by the limit in the middle of - a part's header. The client has until `SWWAF_CLIENT_REQUEST_TIMEOUT` runs out - to send the part that is read. Body inspection suits apps whose forms carry no - code; `docs/gitea.md` records what it refuses in front of gitea. +- `SWWAF_WAF_BODY_LIMIT` switches body inspection on, up to that size. It takes + at most `64K`, since `servewaf` gets no more of a body than that (see "The + answer"). Form data and multipart bodies are read up to the size. JSON and XML + bodies are read only when no larger than the size, since those cannot be read + in part. Any other body is not read: the Core Rule Set would read it as form + data, where binary content such as a git push trips rules written for text. A + body the Core Rule Set cannot parse adds 5 to the score, and so does a + multipart body cut by the limit in the middle of a part's header. Body + inspection suits apps whose forms carry no code; `docs/gitea.md` records what + it refuses in front of gitea. - `SWWAF_TRAP_PATHS` names paths the app never serves and only scanners ask for, such as `/wp-login.php` in front of gitea. This is the short form of a `path` rule with the `ban` action, for deployments that mount no rule files. A trap @@ -637,14 +607,13 @@ sends it. kinds, and of no other: - a rule file match; - a trap path; - - a Core Rule Set match; - - a missing or wrong token. + - a Core Rule Set match. - More than this breaks a limit (see "Bans"). A client trying one attack or - token after another is refused many times a minute; a person rarely more - than a few times. Answers the app gives are not counted, since they do not - tell a scanner from an ordinary client. In front of gitea, registry and git - clients get `404` and `401` by design, hundreds of times a minute. + More than this breaks a limit (see "Bans"). A client trying one attack after + another is refused many times a minute; a person rarely more than a few + times. Answers the app gives are not counted, since they do not tell a + scanner from an ordinary client. In front of gitea, registry and git clients + get `404` and `401` by design, hundreds of times a minute. `smallwebwaf` changes the Core Rule Set 4.25.0 in six ways, since in front of gitea it would otherwise refuse ordinary requests. The changes hold in front of @@ -721,9 +690,9 @@ No source is on by default. A source acts on a list kept elsewhere; the defaults judge a client only by what it does here. AbuseIPDB and CrowdSec also need an account or an engine of the operator's own. The list best suited to be on by default, Spamhaus DROP, has terms, below, that forbid frequent fetching, and -Spamhaus may block an address that breaks them. Each `smallwebwaf` fetches its -own copy, so several apps on one host, sharing one address, would often fetch -too close together. +Spamhaus may block an address that breaks them. Each `servewaf` fetches its own +copy, so several of them on one host, sharing one address, would often fetch too +close together. - Blocklists (`SWWAF_BLOCKLIST_URLS`) are text files of addresses and netblocks, one to a line, as the Spamhaus DROP list is written. Anything after a `;` or @@ -762,7 +731,7 @@ too close together. banned (see "Bans"). Only decisions of the type `ban` on an address or a netblock are used; any other, such as a captcha or a country, is left out. A decision lasts for the `duration` it had left when the engine answered. This - is how a fleet-wide blocklist can arrive without `smallwebwaf` depending on + is how a fleet-wide blocklist can arrive without `servewaf` depending on CrowdSec. A fetch fails when: - the engine answers other than `200`, a redirect included; - the fetch does not finish within a minute; @@ -784,7 +753,7 @@ too close together. DNS blocklists made for mail servers list residential ranges, and would block ordinary visitors, so they should not be used. Spamhaus zones need their keyed query service, given as a zone name that contains the key. - `smallwebwaf` shows the key as `********` wherever it names the zone. Only + `servewaf` shows the key as `********` wherever it names the zone. Only `reputation.json` keeps the key. `SWWAF_DNSBL_RESOLVER` names a resolver, since public resolvers are refused by several list operators. @@ -852,23 +821,24 @@ addresses at low rates each, which the per-client limits do not see. A low - an AS number; - the whole service; - a named netblock in `SWWAF_WATCH_NETS`. -- Every request but the health check is counted, whatever was done with it, - clients in `SWWAF_ALLOW_NETS` and `SWWAF_RATE_LIMIT_EXEMPT_NETS` included, +- Every request `servewaf` answers with `allow`, `deny` or `close` is counted, + an allowed one when its report comes. That holds whatever was done with it, + and for clients in `SWWAF_ALLOW_NETS` and `SWWAF_RATE_LIMIT_EXEMPT_NETS`, since an exempt client misbehaving is worth knowing about. The counters are kept in memory and in `alerts.json`. ## Rule files -`smallwebwaf` reads every `*.rules` file in `SWWAF_RULES_DIR` and checks each +`servewaf` reads every `*.rules` file in `SWWAF_RULES_DIR` and checks each request against them. `docs/rules.md` gives the format, the targets and actions, and the default file. - The files are plain text meant to be edited by hand, so that a new scanning - pattern seen in the request log becomes a rule in one line. `smallwebwaf` - watches the directory: a file edited, added or removed there takes effect - while it runs. It reads the files again once the directory has had no change - for 2 seconds, so that a file still being written is read whole. A file whose - name starts with `.`, such as an editor's lock file, is not a rule file. + pattern seen in the request log becomes a rule in one line. `servewaf` watches + the directory: a file edited, added or removed there takes effect while it + runs. It reads the files again once the directory has had no change for 2 + seconds, so that a file still being written is read whole. A file whose name + starts with `.`, such as an editor's lock file, is not a rule file. - Files are read in name order (`00-default.rules` before `50-gitea.rules`), rules in line order. - A rule's regex is Go's RE2 syntax. It has no backreferences or lookaround; in @@ -876,11 +846,11 @@ and the default file. the proxy. - `path` and `uri` are matched both as received and once percent-decoded, so an encoded probe cannot slip past. A header sent more than once is matched with - its values joined by `, `; a header not sent is empty text. `Host` and - `Transfer-Encoding` cannot be targets, since Go's HTTP server takes them out - of every request; a rule naming one stops the start like any bad line. Request - bodies are not available to rule files; reading them is the Core Rule Set's - job. + its values joined by `, `; a header not sent is empty text. `header:Host` and + `header:Transfer-Encoding` cannot be targets, since Go's HTTP server takes + them out of every request; a rule naming one stops the start like any bad + line. The target `host` matches the request's host. Request bodies are not + available to rule files; reading them is the Core Rule Set's job. - `log` only notes the match in the request log. `block` refuses the request; `ban` makes the request a clear sign of attack (see "Bans"), and is for requests no real visitor sends. A visitor banned by mistake is let back in by @@ -901,21 +871,22 @@ and the default file. of gitea, `/.env` is a probe, while `///src/branch/main/.env.example` is a file in a repository that any visitor may open. Anchor the paths of your own `ban` rules at the site - root too. Anything app-specific, such as a rule banning WordPress probes in - front of gitea, belongs in a file the app's Dockerfile adds or the deployer - mounts. A directory mounted over `SWWAF_RULES_DIR` replaces the default file; - single files mounted into it add to it. -- Rules meant to be edited while `smallwebwaf` runs belong in a mounted - directory, not a mounted file. Many editors save a file by replacing it, and a - container keeps seeing the old one when the file alone is mounted. The lookup - database is mounted the same way, for the same reason. To run without rule - files, mount an empty directory or set `SWWAF_RULES_ENABLED=false`. + root too. Anything else, such as a rule banning WordPress probes in front of + gitea, belongs in a file the deployer adds to the image `servewaf` runs from, + or mounts. A rule has one target, so a `path` rule applies on every site + behind that `servewaf`. A directory mounted over `SWWAF_RULES_DIR` replaces + the default file; single files mounted into it add to it. +- Rules meant to be edited while `servewaf` runs belong in a mounted directory, + not a mounted file. Many editors save a file by replacing it, and a container + keeps seeing the old one when the file alone is mounted. The lookup database + is mounted the same way, for the same reason. To run without rule files, mount + an empty directory or set `SWWAF_RULES_ENABLED=false`. ## Persistent state All state lives in memory, and the files in `SWWAF_STATE_DIR` hold a copy of it, so an orderly stop loses nothing. The files are meant for people as well: an -admin can read or edit them at any time, and the running `smallwebwaf` takes the +admin can read or edit them at any time, and the running `servewaf` takes the edit in. `docs/state-files.md` lists the files, what each holds, and their sizes. @@ -937,8 +908,8 @@ sizes. held up. It goes to a temporary file in the same directory, which is synced and renamed over the real name, and the directory is synced. A crash at any point leaves either the old complete file or the new complete file, never a - partial one. Before writing a file, `smallwebwaf` takes in any edit made to it - on disk, so an admin's edit is never overwritten. + partial one. Before writing a file, `servewaf` takes in any edit made to it on + disk, so an admin's edit is never overwritten. - At start, the files are read back. Active bans go into an in-memory lookup by netblock, everything else into maps. Expired cache entries are discarded. A missing file means empty state, and is normal on first run. These stop the @@ -952,17 +923,17 @@ sizes. Since writes are atomic, a broken file can only come from a hand edit, which the editor should hear about. -- While running, `smallwebwaf` notices a file that is edited, replaced or added. - It tells its own writes from an admin's by comparing the file with what it - last read or wrote. An edit that parses is taken in at once: what the file - says replaces what `smallwebwaf` held for that file. An edit to `bans.json` - makes, lifts or forgets bans as "Bans" describes. A change `smallwebwaf` made - after the admin opened the file, such as a new ban, is lost when the admin - saves over it. Most editors warn when a file changed on disk while it was - open. A file that is removed is written again at its next write. -- An edit that does not parse never stops the running `smallwebwaf`. It keeps - the state it has and sets the edited file aside as `.bad`, so the edit - is kept. It writes the file again from memory, and names the file and the +- While running, `servewaf` notices a file that is edited, replaced or added. It + tells its own writes from an admin's by comparing the file with what it last + read or wrote. An edit that parses is taken in at once: what the file says + replaces what `servewaf` held for that file. An edit to `bans.json` makes, + lifts or forgets bans as "Bans" describes. A change `servewaf` made after the + admin opened the file, such as a new ban, is lost when the admin saves over + it. Most editors warn when a file changed on disk while it was open. A file + that is removed is written again at its next write. +- An edit that does not parse never stops the running `servewaf`. It keeps the + state it has and sets the edited file aside as `.bad`, so the edit is + kept. It writes the file again from memory, and names the file and the position of the error in the log and in one `file_error` alert. The admin mends the `.bad` file and moves it back. - A write that fails leaves the state correct in memory. It is logged, counted @@ -975,17 +946,18 @@ sizes. ## Request log -Every request `smallwebwaf` sees gets one JSON object on one line on stdout, -refused ones included. `docs/request-log.md` lists the fields. Bodies are never -logged, and no header but those listed there. Query strings are logged as -received; an app that carries secrets in query strings needs that fixed in the -app. The process's own messages (start, the settings, stop, errors) share the -stream as JSON lines with `"type":"process"`; request lines carry -`"type":"request"`. +`rproxy` and `servewaf` each write request lines on stdout, one JSON object to a +line, refused requests included. "The log" says which lines a request gets, and +`docs/request-log.md` lists the fields. Bodies are never logged, and no header +but those listed there. Query strings are logged as received; an app that +carries secrets in query strings needs that fixed in the app. Each process's own +messages (start, the settings, stop, errors) share its stream as JSON lines with +`"type":"process"`; request lines carry `"type":"request"`. -When `SWWAF_LOG_REMOTE_URL` is set, the same lines are also sent there as RFC -5424 syslog records. So a deployment can stop depending on docker's log handling -while `docker logs` keeps working. +When `SWWAF_LOG_REMOTE_URL` is set, every line `servewaf` writes on stdout is +also sent there as an RFC 5424 syslog record. So a deployment can stop depending +on docker's log handling while `docker logs` keeps working. `rproxy` writes its +lines to stdout only. - Over UDP each record is one datagram; a line too long for one datagram is dropped alone. Over TCP and TLS each record follows its length in bytes and a @@ -999,39 +971,40 @@ while `docker logs` keeps working. up a request and never stops stdout. When the buffer is full, the oldest line is dropped and counted in the metrics. A failed connection is tried again a second later, and twice as long after each further failure, up to a minute. -- As `smallwebwaf` stops, it spends at most two seconds sending the lines still +- As `servewaf` stops, it spends at most two seconds sending the lines still waiting, and drops the rest; stdout has carried them. ## Metrics endpoint -`GET /_smallwebwaf/metrics` answers in the Prometheus text format, for a request -carrying `SWWAF_METRICS_TOKEN`. `docs/metrics.md` lists the metrics. No metric -carries a client IP address as a label. Per-address questions are answered by -the request log and `GET /_smallwebwaf/clients/`. +`GET /_smallwebwaf/metrics`, at `servewaf`'s own address, answers in the +Prometheus text format, for a request carrying `SWWAF_METRICS_TOKEN`. +`docs/metrics.md` lists the metrics. No metric carries a client IP address as a +label. Per-address questions are answered by the request log and +`GET /_smallwebwaf/clients/`. ## Admin endpoints -A request whose path starts with `/_smallwebwaf/` is for `smallwebwaf` itself: -it answers it and never passes it to the app. The prefix carries the tool's -name, so it takes no path an app uses. Admins and the metrics collector reach -these endpoints through traefik, like any other request. -`docs/admin-endpoints.md` lists the endpoints, what each takes and what each -answers. +`servewaf`'s own endpoints are paths under `/_smallwebwaf/`, at its own address +on the private network. Admins and the metrics collector reach them there, and +nothing routes the internet to that address. The prefix carries the tool's name, +so it takes no path an app uses. `docs/admin-endpoints.md` lists the endpoints, +what each takes and what each answers. - `GET /_smallwebwaf/healthz` answers `200` with `ok` to anyone, without a - token, while `smallwebwaf` is running. It does not ask the app; the - container's health check does. + token, while `servewaf` is running. +- `POST /_smallwebwaf/report` takes `rproxy`'s report, without a token (see "The + report"). - Every other endpoint needs its token, sent as `Authorization: Bearer ` - (see "Tokens" under "Settings"). While a token is unset, the endpoints that - need it answer `404`, as does any other path under the prefix. -- Apart from the health check, these requests go through every check that any - other request goes through. They are answered at the point where another - request would be forwarded to the app: a banned or refused client stays - refused, and each request counts toward the client's limits. A missing or - wrong token is answered `401` (see "Attack detection" for what guessing - costs). + (see "Tokens" under "Settings"). A missing or wrong token is answered `401`. + While a token is unset, the endpoints that need it answer `404`, as does any + other path. +- No check runs on these requests. They count toward no limit, and no client's + history holds them. - A ban made through the endpoints starts at once. +`rproxy` answers `GET /_smallwebwaf/healthz` itself. It never passes a path +under the prefix to the app (see "Without an answer from `servewaf`"). + ## `servewaf` `smallwebwaf servewaf` runs the checks for `smallwebwaf rproxy`, a small reverse @@ -1040,13 +1013,12 @@ request. `servewaf` answers `rproxy` and passes nothing to an app itself. One `servewaf` serves many sites. This section is the agreement between the two. The header names, the fields and -the statuses below are exact. Each `SWWAF_` setting and check keeps the meaning -this document gives it, except as this section says. +the statuses below are exact. -`servewaf` starts as `smallwebwaf` with no command does. It reads the settings, -the rule files, the lookup database and the state files, and listens at -`SWWAF_LISTEN_ADDR`. It stops in the same orderly way, writing the state files. -`docs/settings.md` lists the settings it ignores. +`servewaf` reads the settings, the rule files, the lookup database and the state +files, and listens at `SWWAF_LISTEN_ADDR`. On `SIGTERM` or `SIGINT` it stops +taking requests and gives those in progress five seconds to finish. It then +writes the state files. ### The question @@ -1123,9 +1095,8 @@ metrics. ### The answer -`servewaf` runs steps 3 to 12 of "What happens to a request", in that order, -with the header limits just before step 12 (see "Answering in time"). Every -answer but the last below carries `X-Smallwebwaf-Disposition`. +`servewaf` runs steps 3 to 13 of "What happens to a request", in that order. +Every answer but the last below carries `X-Smallwebwaf-Disposition`. - `X-Smallwebwaf-Disposition: allow`, status `200`, no body. `rproxy` passes the request to the app. With `SWWAF_ADD_LOOKUP_HEADERS` the answer also carries @@ -1278,7 +1249,8 @@ handled. `servewaf` ignores a report naming such a question. ### The log -Each request gets three request log lines, joined by `request_id`: +A request `servewaf` answers with `allow`, `deny` or `close` gets three request +log lines, joined by `request_id`: - `rproxy`'s short line; - `servewaf`'s line at its decision; @@ -1286,7 +1258,9 @@ Each request gets three request log lines, joined by `request_id`: line's fields and the report's together. The repetition is intended. The field `servewaf_line` says which of `servewaf`'s -two lines a line is. `docs/request-log.md` gives the fields. +two lines a line is. A request `servewaf` is not asked about, such as the health +check, gets only `rproxy`'s line. A request without a report gets no third line. +`docs/request-log.md` gives the fields. ### `servewaf`'s own endpoints @@ -1295,7 +1269,7 @@ address on the private network. The internet must not be able to reach that address. Its endpoints are: - the health check, the metrics, and the ban and client endpoints, as - `docs/admin-endpoints.md` lists them, with the same tokens; + `docs/admin-endpoints.md` lists them, with their tokens; - the report. Anything else gets `404`. No check runs on these requests, since their peers are @@ -1327,11 +1301,8 @@ keeps the site answering. `rproxy` reads only its `SWRP_` settings, which `docs/settings.md` lists. One of them is required: `SWRP_WAF_BACKEND`, `servewaf`'s URL. It has no metrics, no -rule files and no state files, and it writes its lines to stdout only. - -The image's run script and health check are those of `smallwebwaf` with no -command (see "Deployment"). A container that runs `rproxy` names its own command -and health check. +rule files and no state files, and it writes its lines to stdout only. The image +runs it by default (see "Deployment"). ### What `rproxy` does with a request @@ -1470,35 +1441,50 @@ each way. ### `rproxy`'s limits -The limits work as "Size and time limits" says, with these settings in place of -those: +These limits apply per request, in both directions: -- `SWRP_MAX_REQUEST_SIZE` for `SWWAF_REQUEST_MAX_BYTES`. -- `SWRP_MAX_RESPONSE_SIZE` for `SWWAF_RESPONSE_MAX_BYTES`. -- `SWRP_MAX_REQUEST_TIME_MS` for `SWWAF_CLIENT_REQUEST_TIMEOUT` and - `SWWAF_UPSTREAM_REQUEST_TIMEOUT`. -- `SWRP_MAX_RESPONSE_TIME_MS` for `SWWAF_CLIENT_RESPONSE_TIMEOUT` and - `SWWAF_UPSTREAM_RESPONSE_TIMEOUT`. +- `SWRP_MAX_REQUEST_SIZE` is the largest request body. +- `SWRP_MAX_RESPONSE_SIZE` is the largest response body. +- `SWRP_MAX_REQUEST_TIME_MS` bounds the request line and headers. From their + end, it then bounds the time until the app has been sent the whole request. + The wait for `servewaf` counts toward it. +- `SWRP_MAX_RESPONSE_TIME_MS` runs from the end of the request to the last byte + of the answer reaching the visitor. The request ends when the app has been + sent all of it, or when the app's answer arrives if that is sooner. An answer + `rproxy` gives itself has the whole time from when it starts. The largest request line and headers is 32 KiB. A kept-open connection is closed -after 120 seconds idle. Neither has a setting. +after 120 seconds idle. Neither has a setting. traefik, by default, closes an +unused connection after 90 seconds. So traefik closes first, and never sends a +request on a connection `rproxy` is closing. -`SWRP_MAX_REQUEST_TIME_MS` bounds the request line and headers. From their end, -it then bounds the time until the app has been sent the whole request. The wait -for `servewaf` counts toward it. A visitor too slow to send the first 64 KiB is -answered `408`, with no question and no report. +What the visitor gets when a limit is passed: -`SWRP_MAX_RESPONSE_TIME_MS` runs from the end of the request to the last byte of -the answer reaching the visitor. The request ends when the app has been sent all -of it, or when the app's answer arrives if that is sooner. An answer `rproxy` -gives itself has the whole time from when it starts. - -`rproxy` applies the request size limit only after `servewaf` has allowed the -request. So a request `servewaf` denies gets `servewaf`'s answer, whatever its -size. A request `servewaf` allows is answered `413` when its body is over the -limit: at once for a length it announced, otherwise when the limit is reached. -If part of the body has gone to the app by then, the request to the app is cut -off. +- Go's HTTP server reads a request's line and headers before `rproxy` sees the + request. It answers `431` to headers over 32 KiB. A visitor too slow to send + them gets an error answer, mostly `400`, or none. Either way the server closes + the connection, nothing reaches the app, and the request log gets no line. +- A visitor too slow to send the first 64 KiB of its body is answered `408`, + with no question and no report. +- `rproxy` applies the request size limit only after `servewaf` has allowed the + request. So a request `servewaf` denies gets `servewaf`'s answer, whatever its + size. A request `servewaf` allows is answered `413` when its body is over the + limit: at once for a length it announced, otherwise when the limit is reached. + If part of the body has gone to the app by then, the request to the app is cut + off. +- A limit passed after that, but before the response has started, is answered by + `rproxy`: + - `408` for a visitor too slow to send the rest of its body; + - `502` for a response that is too large, or an app that cannot be reached; + - `504` for an app that is too slow. +- A request body streams through, so each side can hold up the other. + `SWRP_MAX_REQUEST_TIME_MS` can run out while the body is on its way. The + answer is then `408` if `rproxy` was waiting for the visitor, and `504` if it + was waiting for the app. +- Once the response has started it can only be cut off, and the connection is + closed. +- A WebSocket connection leaves these limits behind once it is upgraded. It + stays open until either side closes it. ### Starting and stopping @@ -1513,10 +1499,48 @@ such as that of a WebSocket that closes later. ## Deployment -`smallwebwaf` runs inside the app's own container: the app's Dockerfile builds -`FROM` the `smallwebwaf` image. Each app is one container. An app is hardened by -changing its base image. It can add packages from nixpkgs. `docs/image.md` lists -what the image holds and how each part is pinned. +One image runs either command. `SMALLWEBWAF_COMMAND`, in a container's +environment, chooses: `servewaf` runs `servewaf`, and with it unset the image +runs `rproxy`. `docs/image.md` lists what the image holds and how each part is +pinned. + +### The usual setup + +The usual setup is three containers on one private network: + +- the app; +- `rproxy`, which traefik routes to, with `SWRP_UPSTREAM_URL` naming the app and + `SWRP_WAF_BACKEND` naming `servewaf`; +- `servewaf`, with `SMALLWEBWAF_COMMAND=servewaf` and a volume for its state + files. + +Each further app adds its own container and its own `rproxy`, whose +`SWRP_WAF_BACKEND` names the same `servewaf` (see "Sites"). Anything on that +network can reach an app around `rproxy`, and can ask `servewaf` questions or +send it reports. So the network holds only what you trust. + +### `rproxy` in the app's container + +An app's Dockerfile can build `FROM` the `smallwebwaf` image instead, so that +`rproxy` runs beside the app in one container. The app is then hardened by +changing its base image, and can add packages from nixpkgs. `servewaf` still +runs in a container of its own, and the app's container sets `SWRP_WAF_BACKEND`. + +Beyond its `FROM` line, the app's Dockerfile adds: + +- its binary; +- any packages it needs, from nixpkgs, with `nix-env -iA nixpkgs.`; +- its runit service: a directory under `/etc/service` named after the app, never + `smallwebwaf`. Its `run` script starts the app with `chpst`, listening where + `SWRP_UPSTREAM_URL` points (see "Ports"). The app runs as a user of its own, + which the Dockerfile creates with `useradd`. That user is neither root nor + `smallwebwaf`, so that the app cannot touch the process of `rproxy`. + +It sets no `ENTRYPOINT` or `USER` of its own. The container must start +`runsvinit`, as root, so that it can start each service as its own user. +"Getting started" in `README.md` shows such a Dockerfile and `run` script. + +### The image - The image is Ubuntu's newest long-term support release, pinned by digest, and moves to the next LTS release when that ships. Its packages come from Ubuntu's @@ -1531,10 +1555,9 @@ what the image holds and how each part is pinned. of the check. The hashes are the amd64 archive's, so the image is built for amd64 only. - The Ubuntu image has no CA certificates. Without them Nix cannot download - packages and `smallwebwaf` cannot reach GeoJS, so `ca-certificates` is - installed by name. The snapshot itself is fetched over HTTPS with the CA - certificates of the Go image that builds `smallwebwaf`, itself pinned by - digest. + packages and `servewaf` cannot reach GeoJS, so `ca-certificates` is installed + by name. The snapshot itself is fetched over HTTPS with the CA certificates of + the Go image that builds `smallwebwaf`, itself pinned by digest. - nixpkgs comes from a release file on `releases.nixos.org`, named with its hash, not from a GitHub archive, whose bytes can change. Root uses Nix directly; no Nix daemon runs. `nix-bin` creates no build users, so @@ -1545,15 +1568,20 @@ what the image holds and how each part is pinned. from Ubuntu and nixpkgs come with a newer `smallwebwaf` image, which an app takes by changing the digest in its `FROM` line. The image moves to each new nixpkgs release, every six months. -- runit supervises the two processes, with `runsvinit` as the entrypoint, as the - owner's code style guide asks for service containers. No package carries - `runsvinit`, so the image builds it from its archived source, unchanged since - 2015, at a fixed commit hash, with a `go.mod` the build writes. `runsvinit` - starts runit's `runsvdir`, which runs the `run` script of each directory under - `/etc/service`. Ubuntu's `sv` looks for services there too. -- The image's `run` script for `smallwebwaf` makes `SWWAF_STATE_DIR` and every - file in it belong to the `smallwebwaf` user (uid and gid 65532). Then it - starts `smallwebwaf` as that user with runit's `chpst`. +- runit supervises a container's processes, with `runsvinit` as the entrypoint, + as the owner's code style guide asks for service containers. No package + carries `runsvinit`, so the image builds it from its archived source, + unchanged since 2015, at a fixed commit hash, with a `go.mod` the build + writes. `runsvinit` starts runit's `runsvdir`, which runs the `run` script of + each directory under `/etc/service`. Ubuntu's `sv` looks for services there + too. +- The image's `run` script is `/etc/service/smallwebwaf/run`. It starts + `servewaf` when `SMALLWEBWAF_COMMAND` is `servewaf`, and `rproxy` when it is + `rproxy`, unset or empty. It runs either as the `smallwebwaf` user (uid and + gid 65532), with runit's `chpst`. For `servewaf`, it first makes + `SWWAF_STATE_DIR` and every file in it belong to that user. For `rproxy` it + changes no owner. With any other value, it starts nothing, and stops with a + message naming `SMALLWEBWAF_COMMAND` and its value. Every `run` script, the app's included, is a bash script. It starts with `#!/usr/bin/env bash` and `set -euo pipefail`, and puts its code in a `main` @@ -1561,128 +1589,131 @@ function, called on its last line, as the owner's code style guide asks. Ubuntu ships bash. Each waits one second (`sleep 1`) before starting its program, so a process that stops at start is started again every second, not in a tight loop. -Beyond its `FROM` line, the app's Dockerfile adds: - -- its binary; -- any packages it needs, from nixpkgs, with `nix-env -iA nixpkgs.`; -- its runit service: a directory under `/etc/service` named after the app, never - `smallwebwaf`. Its `run` script starts the app with `chpst`, listening where - `SWWAF_UPSTREAM_URL` points (see "Ports"). The app runs as a user of its own, - which the Dockerfile creates with `useradd`. That user is neither root nor - `smallwebwaf`, so that the app cannot touch the state files or the process of - `smallwebwaf`. - -It sets no `ENTRYPOINT` or `USER` of its own. The container must start -`runsvinit`, as root, so that it can start each service as its own user. -"Getting started" in `README.md` shows such a Dockerfile and `run` script. - -### The two processes +### Each container's processes - `runsvinit`, `runsvdir` and each service's `runsv` run as root. Each `run` - script starts as root and hands over to its program's user with `chpst`. Both - services start with the container's environment variables and write to the + script starts as root and hands over to its program's user with `chpst`. Every + service starts with the container's environment variables and writes to the container's output, which `docker logs` shows. -- When `smallwebwaf` or the app exits, runit runs its `run` script again. The - other process keeps running, and the container does not stop. While - `smallwebwaf` is down, traefik answers `502`. While the app is down, - `smallwebwaf` answers `502`, which is not a client offence. -- Only faults found at start stop `smallwebwaf`: +- When a service exits, runit runs its `run` script again, and the container + does not stop. Beside an app, the other process keeps running. +- While `rproxy` is down, traefik answers `502`. While the app is down, `rproxy` + answers `502`, which is not a client offence. While `servewaf` is down, + `SWRP_DEFAULT_DISPOSITION` decides (see "Without an answer from `servewaf`"). +- Only faults found at start stop `servewaf`: - invalid configuration; - a lookup database that is missing or unreadable; - a rule file or state file that does not parse; - an unwritable `SWWAF_STATE_DIR`; - a `SWWAF_RULES_DIR` that does not exist or cannot be read; - a `SWWAF_LISTEN_ADDR` it cannot listen on. -- A `smallwebwaf` that stops at start logs its message at each try until the - cause is fixed; the health check fails meanwhile. Once running, a failing +- Only an invalid setting, or a `SWRP_LISTEN_ADDR` it cannot listen on, stops + `rproxy`. +- A process that stops at start logs its message at each try until the cause is + fixed; its container's health check fails meanwhile. Once running, a failing GeoJS, reputation source or alert destination, or a broken edit, never takes - the protected service down. -- `docker stop` sends `SIGTERM` to `runsvinit`, which has runit stop both - services at once, each with `SIGTERM`, and exits once they have stopped. - `smallwebwaf` stops taking requests, gives those in progress five seconds to - finish, and writes its state files. An app that exits at once on `SIGTERM` - ends those requests: the client gets `502`, or a response cut short. To let + `servewaf` down. +- `docker stop` sends `SIGTERM` to `runsvinit`, which has runit stop every + service at once, each with `SIGTERM`, and exits once they have stopped. + `rproxy` and `servewaf` then stop as "`rproxy`" and "`servewaf`" say. +- With `rproxy` beside the app, an app that exits at once on `SIGTERM` ends the + requests in progress: the visitor gets `502`, or a response cut short. To let them finish, the app must keep answering for five seconds after `SIGTERM`. - The container's root filesystem stays writable: runit writes each service's status into its directory under `/etc/service`. ### The health check -The image's `HEALTHCHECK` runs `smallwebwaf healthcheck`. It passes while -`smallwebwaf` answers `GET /_smallwebwaf/healthz` on `127.0.0.1`, at the port in -`SWWAF_LISTEN_ADDR`, and the app accepts connections at `SWWAF_UPSTREAM_URL`. So -the container shows as healthy only while both processes are up. traefik sends a -container no requests until it shows as healthy. So the check runs every second -from the container's start until it first passes, for up to a minute, and every -30 seconds after that. An app with a health check of its own can replace the -image's `HEALTHCHECK` with one that checks both. +The image's `HEALTHCHECK` runs `smallwebwaf healthcheck`. `SMALLWEBWAF_COMMAND` +says what it checks: + +- With `SMALLWEBWAF_COMMAND=servewaf`, it passes while `servewaf` answers + `GET /_smallwebwaf/healthz` on `127.0.0.1`, at the port in + `SWWAF_LISTEN_ADDR`. +- With `SMALLWEBWAF_COMMAND` unset, empty or `rproxy`, it passes while `rproxy` + answers that request on `127.0.0.1`, at the port in `SWRP_LISTEN_ADDR`, and + the app accepts connections at `SWRP_UPSTREAM_URL`. It does not ask + `servewaf`, so that the site stays in traefik's routes while `servewaf` is + down. +- With any other value, it fails with a message naming `SMALLWEBWAF_COMMAND` and + its value. + +traefik sends a container no requests until it shows as healthy. So the check +runs every second from the container's start until it first passes, for up to a +minute, and every 30 seconds after that. An app with a health check of its own +can replace the image's `HEALTHCHECK` with one that checks both. ### Ports -`smallwebwaf` listens on no port but the one in `SWWAF_LISTEN_ADDR`, 8080 by -default, which traefik routes to. Its own endpoints are on that listener, and it -forwards every other request to the app at `SWWAF_UPSTREAM_URL`, -`127.0.0.1:8081` by default. Another port may be set: the health check follows -the setting, traefik's port label -(`traefik.http.services..loadbalancer.server.port`) must name it, and the -app must leave it free. The address part of `SWWAF_LISTEN_ADDR` stays empty -(`:9000`, never `127.0.0.1:9000`), so that traefik reaches `smallwebwaf` on the -container's address and the health check on `127.0.0.1`. The app listens on that -loopback address only. An app that listens on every address can be reached -around `smallwebwaf` by anything that reaches the container. +`rproxy` listens on no port but the one in `SWRP_LISTEN_ADDR`, 8080 by default, +which traefik routes to. It passes requests to the app at `SWRP_UPSTREAM_URL`, +`http://127.0.0.1:8081` by default. Another port may be set: the health check +follows the setting, and traefik's port label +(`traefik.http.services..loadbalancer.server.port`) must name it. An app +beside `rproxy` must leave that port free. + +`servewaf` listens on no port but the one in `SWWAF_LISTEN_ADDR`, 8080 by +default. The questions, the reports and its own endpoints all come there. + +The address part of either setting stays empty (`:9000`, never +`127.0.0.1:9000`). That way the others reach the process on the container's +address, and the health check reaches it on `127.0.0.1`. + +An app beside `rproxy` listens on that loopback address only. An app that +listens on every address can be reached around `rproxy` by anything that reaches +the container. ### State -`smallwebwaf` keeps its state files in `SWWAF_STATE_DIR`, a directory of its own -beside the app's data directories, belonging to the `smallwebwaf` user. Without -a volume there, the files survive a restart of the container and are lost when a -deploy replaces it. A volume mounted at `/var/lib/smallwebwaf`, named or a host -directory, keeps them across deploys. A host directory mounted there needs no -change of owner, and files an earlier owner left in it can be read and replaced. -The volume is separate from the app's, holds a few tens of MiB at most with the -defaults, and needs no backup beyond the host's own. The image declares no -volume, since every app image built on it would inherit it. +`servewaf` keeps its state files in `SWWAF_STATE_DIR`, belonging to the +`smallwebwaf` user. Without a volume there, the files survive a restart of the +container and are lost when a deploy replaces it. A volume mounted at +`/var/lib/smallwebwaf`, named or a host directory, keeps them across deploys. A +host directory mounted there needs no change of owner, and files an earlier +owner left in it can be read and replaced. The volume holds a few tens of MiB at +most with the defaults, and needs no backup beyond the host's own. The image +declares no volume, since every app image built on it would inherit it. `rproxy` +keeps nothing, and needs no volume. ### Tokens A token given as a file (`SWWAF_ADMIN_TOKEN_FILE`, `SWWAF_METRICS_TOKEN_FILE`) -is out of the app's reach only while the `smallwebwaf` user alone can read the -file. The operator makes the file on the host, owned by uid 65532, the -`smallwebwaf` user, with mode `0400`, and mounts the directory that holds it -into the container read-only. The container sees the same owner and mode. -"Settings given as files" in `docs/settings.md` shows the commands. +stays out of the container's environment. The operator makes the file on the +host, owned by uid 65532, the `smallwebwaf` user, with mode `0400`, and mounts +the directory that holds it into `servewaf`'s container read-only. The container +sees the same owner and mode. "Settings given as files" in `docs/settings.md` +shows the commands. ### Forwarded headers -The app's TCP peer is `smallwebwaf` on `127.0.0.1`, and the `X-Forwarded-For` -the app receives ends with traefik's address, which `smallwebwaf` adds after the -visitor's. The app must therefore trust `127.0.0.1` and `::1` for forwarded -headers, besides the private address ranges traefik is on. The usual default for -a trusted-proxy setting, the private ranges alone, does not include loopback: an -app left at it ignores the header and sees every visitor as `127.0.0.1`. A list -given replaces that default, so the app's own trusted-proxy setting names them -all: `10.0.0.0/8,172.16.0.0/12,192.168.0.0/16,127.0.0.1/32,::1/128`. +The app's TCP peer is `rproxy`, and the `X-Forwarded-For` the app receives ends +with traefik's address, which `rproxy` adds after the visitor's. So the app must +trust `rproxy`'s address for forwarded headers, besides the private address +ranges traefik is on. + +- In a container of its own, `rproxy`'s address is on the private network. The + usual default for a trusted-proxy setting, the private ranges, covers it. +- Beside the app, `rproxy`'s address is `127.0.0.1` or `::1`. The private ranges + leave out loopback: an app left at them ignores the header and sees every + visitor as `127.0.0.1`. A list given replaces that default, so the app's own + trusted-proxy setting names them all: + `10.0.0.0/8,172.16.0.0/12,192.168.0.0/16,127.0.0.1/32,::1/128`. ### upaas -An app deployed by upaas, which rebuilds and restarts an app's Docker container -when its repository changes, takes this shape with no change to upaas. upaas -builds the app's Dockerfile and runs the one container as it runs any app. The -traefik labels route to port 8080, and any `SWWAF_` settings go with the app's -environment variables. The volume for `/var/lib/smallwebwaf` and the directory -of any token file go beside the app's own volumes. gitea is built on the image -like any other app; `docs/gitea.md` records what the defaults do in front of it, -and `script/gitea-requests` checks those notes. +upaas rebuilds and restarts an app's Docker container when its repository +changes. An app it deploys takes the shape of "`rproxy` in the app's container" +with no change to upaas. upaas builds the app's Dockerfile and runs the one +container as it runs any app. The traefik labels route to port 8080, and the +`SWRP_` settings go with the app's environment variables. `servewaf` runs in a +container of its own, on a network the app's container can reach, with its state +volume and the directory of any token file. `docs/gitea.md` records what the +defaults do in front of gitea, and `script/gitea-requests` checks those notes. ## Planned, not built -These points are specified as later work, and nothing in the code does them yet: +This point is specified as later work, and nothing in the code does it yet: -- Planned: sharing bans between the instances of several apps, by running one - CrowdSec engine per host that every instance reports its bans to. It is - reconsidered once real ban volumes are known. Until then each instance keeps - its own bans, and can read a CrowdSec engine's decisions as described under - "Reputation". - Planned: sending the log over RELP, the reliable syslog transport. It is revisited against the available Go packages. Until then the remote log goes over UDP, TCP or TLS as described under "Request log". diff --git a/cmd/smallwebwaf/main.go b/cmd/smallwebwaf/main.go index 090fb90..c7d998c 100644 --- a/cmd/smallwebwaf/main.go +++ b/cmd/smallwebwaf/main.go @@ -1,8 +1,8 @@ -// Command smallwebwaf is a web application firewall for one app: it runs -// between traefik and the app and passes requests through within its limits, -// or, as `smallwebwaf servewaf`, answers rproxy's questions about them, or, as -// `smallwebwaf rproxy`, is the reverse proxy that asks those questions. -// Everything it does is in internal/smallwebwaf. +// Command smallwebwaf is a web application firewall. `smallwebwaf rproxy` is +// the reverse proxy in front of one app, which asks servewaf about each request. +// `smallwebwaf servewaf` decides, for many sites, and keeps what lasts. +// `smallwebwaf healthcheck` is the container's health check. Everything it does +// is in internal/smallwebwaf. package main import ( diff --git a/docs/admin-endpoints.md b/docs/admin-endpoints.md index 6e7e6f4..ddb82ea 100644 --- a/docs/admin-endpoints.md +++ b/docs/admin-endpoints.md @@ -1,8 +1,8 @@ # Admin endpoints The endpoints under `/_smallwebwaf/`, what each takes and what each answers. -"Admin endpoints" in [`SPEC.md`](../SPEC.md) says which checks they go through -and how their tokens work. +`smallwebwaf servewaf` answers them at its own address, on the private network. +"Admin endpoints" in [`SPEC.md`](../SPEC.md) says how their tokens work. - `GET /_smallwebwaf/healthz`: `200` and `ok`, to anyone. - `GET /_smallwebwaf/metrics`: the metrics, for `SWWAF_METRICS_TOKEN` (see @@ -20,40 +20,41 @@ and how their tokens work. - one that names an IPv4-mapped netblock, or an address with a zone. - `DELETE /_smallwebwaf/bans/`: lifts every active ban covering the address ``. It answers `404` when none is active. -- `GET /_smallwebwaf/clients/`: what `smallwebwaf` knows of the client. - Under `client`, its counters and history as `clients.json` holds them, or - `null` when `clients.json` does not hold it. Under `bans`, every ban covering - the address, with its notes. +- `GET /_smallwebwaf/clients/`: what `servewaf` knows of the client. Under + `client`, its counters and history as `clients.json` holds them, or `null` + when `clients.json` does not hold it. Under `bans`, every ban covering the + address, with its notes. +- `POST /_smallwebwaf/report`: takes `rproxy`'s report, without a token. It + answers `204`, or `400` to a body that is no report (see "The report" in + [`SPEC.md`](../SPEC.md)). + +No check runs on these requests. A missing or wrong token is answered `401`. An +endpoint whose token is unset answers `404`, as does any other path. + +`smallwebwaf rproxy` answers only the health check of these, at its own address. +It asks `servewaf` about any other request under `/_smallwebwaf/`. One that +passes the checks is answered `404`, so that the app never sees that prefix (see +"`rproxy`" in [`SPEC.md`](../SPEC.md)). ## Examples -With `SWWAF_ADMIN_TOKEN` set, and the token in `$TOKEN`: +With `SWWAF_ADMIN_TOKEN` set, the token in `$TOKEN`, and `servewaf` reached as +`servewaf:8080` on its network: ```sh # Every ban. -curl -H "Authorization: Bearer $TOKEN" https://app.example/_smallwebwaf/bans +curl -H "Authorization: Bearer $TOKEN" http://servewaf:8080/_smallwebwaf/bans # Ban 203.0.113.0/24 for seven days. curl -H "Authorization: Bearer $TOKEN" \ --json '{"netblock": "203.0.113.0/24", "duration": "7d", "reason": "probes for logins"}' \ - https://app.example/_smallwebwaf/bans + http://servewaf:8080/_smallwebwaf/bans # Lift the bans on 203.0.113.9. curl -H "Authorization: Bearer $TOKEN" -X DELETE \ - https://app.example/_smallwebwaf/bans/203.0.113.9 + http://servewaf:8080/_smallwebwaf/bans/203.0.113.9 -# What smallwebwaf knows of 203.0.113.9. +# What servewaf knows of 203.0.113.9. curl -H "Authorization: Bearer $TOKEN" \ - https://app.example/_smallwebwaf/clients/203.0.113.9 + http://servewaf:8080/_smallwebwaf/clients/203.0.113.9 ``` - -## `servewaf` - -`smallwebwaf servewaf` answers these endpoints at its own address. No check runs -on those requests (see "`servewaf`" in [`SPEC.md`](../SPEC.md)). - -## `rproxy` - -`smallwebwaf rproxy` answers only the health check, at its own address. It asks -`servewaf` about any other request under `/_smallwebwaf/` (see "`rproxy`" in -[`SPEC.md`](../SPEC.md)). diff --git a/docs/alerts.md b/docs/alerts.md index 201a37e..093e0da 100644 --- a/docs/alerts.md +++ b/docs/alerts.md @@ -6,7 +6,7 @@ back or dropped. ## Events -- `ban`: a ban `smallwebwaf` made. +- `ban`: a ban `servewaf` made. - `permanent_ban`: a permanent ban it made, or an existing ban made permanent by a request. - `waf_block`: the Core Rule Set scored a request at or over the threshold. The @@ -34,9 +34,10 @@ on one line: ```json { - "instance": "fsn1app1/gitea", + "instance": "fsn1/servewaf", "time": "2026-10-06T12:00:50.123461Z", "event": "ban", + "host": "app.example", "client": "203.0.113.9", "netblock": "203.0.113.9/32", "asn": "", @@ -78,9 +79,9 @@ on one line: - `instance`: `SWWAF_INSTANCE_NAME`. `time`: when the alert was raised. `event`: one of the events above. -- `host`: the site of the request that raised the alert. Only `servewaf` sends - this field, since it serves many sites, and not for `source_failure`, - `file_error` and `summary`. +- `host`: the site of the request that raised the alert, since one `servewaf` + serves many sites. It is left out of `source_failure`, `file_error` and + `summary`. - `client`: the address whose request raised the alert. Empty for `source_failure`, `file_error` and `summary`. - `netblock`: empty for `source_failure`, `file_error` and `summary`; otherwise @@ -110,7 +111,7 @@ on one line: ## Slack and ntfy Slack and ntfy get a message with a title and a text. The title is the instance -and the event, such as `fsn1app1/gitea: ban`. The text is the `reason`, then one +and the event, such as `fsn1/servewaf: ban`. The text is the `reason`, then one line for each of these that the alert has: - the host, the client, the netblock and the country; diff --git a/docs/gitea.md b/docs/gitea.md index 1edf488..dd2d60c 100644 --- a/docs/gitea.md +++ b/docs/gitea.md @@ -2,10 +2,11 @@ These notes record what `smallwebwaf` does in front of gitea at the default settings. They are observations, not requirements; the requirements are in -[`SPEC.md`](../SPEC.md). `script/gitea-requests` runs gitea behind the image and -sends through it a sample of gitea's ordinary use and the requests named below. -It reports each request the Core Rule Set refuses, so that these notes can be -checked again when gitea or the Core Rule Set changes version. +[`SPEC.md`](../SPEC.md). `script/gitea-requests` runs gitea behind `rproxy` and +`servewaf`, and sends through them a sample of gitea's ordinary use and the +requests named below. It reports each request the Core Rule Set refuses, so that +these notes can be checked again when gitea or the Core Rule Set changes +version. ## What the defaults cover @@ -13,7 +14,7 @@ checked again when gitea or the Core Rule Set changes version. by it. - A clone is a response. It fits the defaults of 30 minutes and 1 GiB unless the repository is large or the link slow. A repository whose clone is larger than - 1 GiB needs `SWWAF_RESPONSE_MAX_BYTES` raised, and the byte limits with it + 1 GiB needs `SWRP_MAX_RESPONSE_SIZE` raised, and the byte limits with it ("Byte limits, per client" in `SPEC.md`). - A push is a request, and so is every other upload: - a git push; @@ -23,10 +24,9 @@ checked again when gitea or the Core Rule Set changes version. - a release attachment. At the defaults, an upload larger than 100 MiB or taking more than 60 - seconds is cut off ("Size and time limits" in `SPEC.md`). A gitea that takes - large uploads needs `SWWAF_REQUEST_MAX_BYTES`, - `SWWAF_CLIENT_REQUEST_TIMEOUT` and `SWWAF_UPSTREAM_REQUEST_TIMEOUT` raised - to fit. + seconds is cut off ("`rproxy`'s limits" in `SPEC.md`). A gitea that takes + large uploads needs `SWRP_MAX_REQUEST_SIZE` and `SWRP_MAX_REQUEST_TIME_MS` + raised to fit. - Archive download and blame or history pages are what scrapers hammer. The request limits do most of the work there. diff --git a/docs/image.md b/docs/image.md index d78d2b1..126532c 100644 --- a/docs/image.md +++ b/docs/image.md @@ -1,6 +1,7 @@ # What the image holds -The contents of the `smallwebwaf` image, and how each part is pinned. +The contents of the `smallwebwaf` image, and how each part is pinned. The image +runs `rproxy` by default, or `servewaf` with `SMALLWEBWAF_COMMAND=servewaf`. "Deployment" in [`SPEC.md`](../SPEC.md) says why each part is there and why it is pinned that way. @@ -18,8 +19,9 @@ is pinned that way. - The `smallwebwaf` binary, and the user `smallwebwaf` (uid and gid 65532). - The service directory `/etc/service/smallwebwaf`, with the `run` script `share/smallwebwaf.run`. -- `/var/lib/smallwebwaf` for the state files, and `/etc/smallwebwaf/rules.d` - with the default rule file `share/rules.d/00-default.rules`. +- `/var/lib/smallwebwaf` for `servewaf`'s state files, and + `/etc/smallwebwaf/rules.d` with the default rule file + `share/rules.d/00-default.rules`. - `EXPOSE 8080`, and the `HEALTHCHECK` that runs `smallwebwaf healthcheck`. ## Pins diff --git a/docs/metrics.md b/docs/metrics.md index e2bff8c..d6039be 100644 --- a/docs/metrics.md +++ b/docs/metrics.md @@ -1,8 +1,9 @@ # The metrics -What `GET /_smallwebwaf/metrics` answers with. "Metrics endpoint" in -[`SPEC.md`](../SPEC.md) describes the endpoint. Prometheus keeps the `instance` -label as `exported_instance` unless the scrape sets `honor_labels: true`. +What `GET /_smallwebwaf/metrics` answers with, at `servewaf`'s own address. +"Metrics endpoint" in [`SPEC.md`](../SPEC.md) describes the endpoint. Prometheus +keeps the `instance` label as `exported_instance` unless the scrape sets +`honor_labels: true`. `rproxy` has no metrics. ## Traffic @@ -11,14 +12,13 @@ label as `exported_instance` unless the scrape sets `honor_labels: true`. `action`; - `smallwebwaf_request_duration_seconds` and `smallwebwaf_upstream_duration_seconds`, as histograms; -- `smallwebwaf_requests_in_flight`. +- `smallwebwaf_requests_in_flight`: the questions and requests to its own + endpoints that `servewaf` is handling. ## Limits and bans - `smallwebwaf_rate_limit_hits_total`, by `window` and by `kind` (`requests`, `bytes` or `refusals`); -- `smallwebwaf_size_and_time_limit_hits_total`, by `limit`, the setting passed. - `servewaf`'s header limits have no setting, so it leaves them out; - `smallwebwaf_offences_total`, by `kind`; - `smallwebwaf_bans_made_total`, by `cause`; - `smallwebwaf_active_bans` and `smallwebwaf_permanent_bans`. @@ -81,11 +81,11 @@ By `file`: `smallwebwaf_alerts_dropped_total`; - Go's own `go_` metrics and the process's `process_` metrics. -## `servewaf` +## When a request is counted -`smallwebwaf servewaf` counts a denied request at its decision. It counts an -allowed request when its report comes, with the status, bytes and times the -report gives. These are not counted: +`servewaf` counts a denied request at its decision, and a request to its own +endpoints when it is answered. It counts an allowed request when its report +comes, with the status, bytes and times the report gives. These are not counted: - a report itself; - an allowed request whose report never comes; diff --git a/docs/request-log.md b/docs/request-log.md index bccd93c..0eb5411 100644 --- a/docs/request-log.md +++ b/docs/request-log.md @@ -1,124 +1,9 @@ # The request log's fields -The fields of the JSON object `smallwebwaf` writes for each request. "Request -log" in [`SPEC.md`](../SPEC.md) says when a line is written and how the log is -sent to a syslog server. A field that does not apply to a request is left out. -`smallwebwaf rproxy` writes a shorter line, which "`rproxy`'s line" below gives. - -## Example - -``` -{"type":"request","time":"2026-10-03T12:00:00.123Z","instance":"fsn1app1/gitea","client_ip":"203.0.113.9","method":"GET","scheme":"https","host":"app.example","path":"/","query":"","protocol":"HTTP/1.1","status":200,"request_bytes":0,"response_bytes":5120,"referer":"","user_agent":"curl/8.9.1","request_id":"7Q2NHZ4KJ3VXW5YB6R3MEFTD2A","peer_ip":"172.18.0.2","forwarded_for":"203.0.113.9","client_group":"203.0.113.9/32","asn":"AS64496","as_name":"Example Net","country":"DE","request_headers":{"accept":"*/*"},"response_content_type":"text/html; charset=utf-8","upstream_status":200,"action":"forward","counts":{"minute":1,"hour":12,"day":40,"minute_bytes":5120,"hour_bytes":61440,"day_bytes":204800},"duration_total":3.217,"duration_checks":0.041,"duration_upstream_connect":0.052,"duration_upstream_first_byte":2.874,"duration_upstream_total":3.104} -``` - -## Every line - -Every line but `rproxy`'s has: - -- `type`: `request` (the process's own messages have `process`); -- the standard web log fields; -- `request_id`, `peer_ip` and `client_group`; -- `asn`, `as_name` and `country`; -- `action` and `duration_total`. - -## Standard web log fields - -- `time`: when the request arrived, RFC 3339 with milliseconds, in UTC; -- `instance`: `SWWAF_INSTANCE_NAME`; -- `client_ip`, `method`, `scheme`, `host`, `path`, `query`, `protocol`; -- `status`: what the client was sent, `0` if nothing; -- `request_bytes` and `response_bytes`: body bytes each way; -- `referer` and `user_agent`. - -## Request detail - -- `request_id`: the `X-Request-ID` a trusted proxy sent, or a new random one of - 26 letters and digits. -- `peer_ip`: the TCP peer, normally traefik. -- `forwarded_for`: the `X-Forwarded-For` header as received. -- `client_group`: the client as rate limiting counts it, an IPv4 address as a - /32 or an IPv6 group. -- `asn`, `as_name` and `country`: the client's, as looked up. Each is empty - when: - - lookups are off; - - the client is not looked up; - - the source cannot place the client; - - GeoJS had not answered by the time the request went on. -- `content_type` and `content_length`: the request's own. -- `request_headers`: the headers listed in `SWWAF_LOG_REQUEST_HEADERS`, by name - in lower case. -- `has_authorization` and `has_cookie`: true when the request had those headers. - Their values are never logged. -- `websocket`: true when the app switched the connection to another protocol. - -## Response detail - -- `upstream_status`: what the app answered. It differs from `status` when - `smallwebwaf` answered itself. -- `response_content_type`, `cache_control` and `location`: the answer's headers. -- `aborted`: true when the client went away early. - -## Decision - -- `action`: what was done: - - `forward`: passed to the app; - - `denied`: refused for `SWWAF_DENY_NETS`, or for a reputation source with - the action `deny`; - - `banned`: refused under a ban, or for a `ban` rule, a trap path or a - CrowdSec decision, each of which bans the client; - - `country_denied`, `rate_limited`, `rule_blocked` and `waf_blocked`: - refused by that check; - - `too_large`: a size limit was passed, or a header limit in `servewaf`; - - `timed_out`: a time limit was passed; - - `upstream_error`: the app could not be reached, or its answer broke off; - - `admin`: answered at one of `smallwebwaf`'s own endpoints. -- `would_action`: in `observe` mode, what `enforce` mode would have done with a - request it would have refused. `action` then names what was done, normally - `forward`. -- `limit_percent` and `limit_percent_setting`: the client's percentage of the - rate limits and the setting that lowered it, when one did. `bytes_percent` and - `bytes_percent_setting` do the same for the byte limits. -- `counts`: the client's totals as rate limiting counts them, this request - included. It is left out when rate limiting did not count the request. It - holds: - - `minute`, `hour` and `day`: requests; - - `minute_bytes`, `hour_bytes` and `day_bytes`: bytes. - - A count can have a fraction (see "Counting method" in - [`SPEC.md`](../SPEC.md)). - -- `limit_hit`: the limit broken: - - `minute`, `hour` or `day` for a rate limit; - - `minute_bytes`, `hour_bytes` or `day_bytes` for a byte limit; - - `error_burst`. - - A request whose bytes broke a limit was passed on whole, so its `action` - stays `forward`. - -- `rule_ids`: the rule file rules that matched, in order, up to the one that - refused. -- `waf_score` and `waf_rule_ids`: the Core Rule Set's score and the rules that - matched, for a request it inspected. -- `reputation`: the sources the client is on, in this order: - - blocklist URLs; - - the CrowdSec decision list's URL; - - DNS blocklist zones; - - `abuseipdb`. -- `offence`: `limit`, for a request that broke a limit. -- `ban_expires`: when the client's ban ends, for a request that made one or was - refused under one: a time, or `permanent`. - -## Timings - -All in milliseconds: - -- `duration_total`: from the request's headers being read to its log line; -- `duration_checks`: to the end of the checks, before forwarding; -- `duration_waf`: the part of that the Core Rule Set took; -- `duration_upstream_connect`: from the hand-over to the app until there was a - connection; -- `duration_upstream_first_byte`: until the first byte of the answer; -- `duration_upstream_total`: until the end of the answer. +The fields of the JSON objects `rproxy` and `servewaf` write about each request. +"Request log" and "The log" in [`SPEC.md`](../SPEC.md) say when a line is +written and how the log is sent to a syslog server. A field that does not apply +to a request is left out. ## `servewaf`'s lines @@ -138,50 +23,151 @@ These questions write no line: [`SPEC.md`](../SPEC.md)); - a question whose body stopped arriving, which gets no answer. -### The `decision` line +A `report` line: -It has the fields above, with these meanings where they differ: +``` +{"type":"request","servewaf_line":"report","time":"2026-10-03T12:00:00.123Z","instance":"fsn1/servewaf","client_ip":"203.0.113.9","method":"GET","scheme":"https","host":"app.example","path":"/","query":"","protocol":"HTTP/1.1","status":200,"request_bytes":0,"response_bytes":5120,"referer":"","user_agent":"curl/8.9.1","request_id":"7Q2NHZ4KJ3VXW5YB6R3MEFTD2A","peer_ip":"172.18.0.2","forwarded_for":"203.0.113.9","client_group":"203.0.113.9/32","asn":"AS64496","as_name":"Example Net","country":"DE","request_headers":{"accept":"*/*"},"action":"forward","counts":{"minute":1,"hour":12,"day":40,"minute_bytes":5120,"hour_bytes":61440,"day_bytes":204800},"waf_score":0,"duration_total":3.217,"duration_checks":0.041,"duration_waf":0.029,"duration_upstream_total":3.104} +``` -- `client_ip`: `X-Smallwebwaf-Client`. -- `peer_ip`: `X-Smallwebwaf-Peer`. +### Every line + +Every one of these lines has: + +- `type`: `request` (the process's own messages have `process`); +- the standard web log fields; +- `request_id`, `peer_ip` and `client_group`; +- `asn`, `as_name` and `country`; +- `action` and `duration_total`. + +### Standard web log fields + +- `time`: when `servewaf` had read the question's body, RFC 3339 with + milliseconds, in UTC. The `report` line gives the same time as the `decision` + line. +- `instance`: `SWWAF_INSTANCE_NAME`. +- `client_ip`: the visitor's address, as `rproxy` worked it out. +- `method`, `path` and `query`: the visitor's request. +- `scheme`: how the visitor reached `rproxy` or the trusted proxy in front of + it. - `host`: the site the visitor asked for. - `protocol`: the question's own, `HTTP/1.1`. -- `content_length`: `X-Smallwebwaf-Content-Length`. -- `request_id`: the question's `X-Request-ID`. -- `status`: what `servewaf` answered: `200` for `allow`, the status of `deny`, - or `0` for `close`, as the visitor is then sent nothing. -- `request_bytes`: how much of the question's body the Core Rule Set read. -- `response_bytes`: the body of `servewaf`'s answer. -- `action`: `forward` for `allow`, and `admin` for the `404` for a path under - `/_smallwebwaf/`. +- `status`: + - on the `decision` line, what `servewaf` answered: `200` for `allow`, the + status of `deny`, or `0` for `close`, as the visitor is then sent nothing; + - on the `report` line, what the visitor got, `0` if nothing. +- `request_bytes`: + - on the `decision` line, how much of the question's body the Core Rule Set + read; + - on the `report` line, the bytes of the visitor's body that `rproxy` read, + with what a WebSocket carried from the visitor. +- `response_bytes`: + - on the `decision` line, the body of `servewaf`'s answer; + - on the `report` line, the bytes of the answer's body, with what a + WebSocket carried from the app. +- `referer` and `user_agent`. + +### Request detail + +- `request_id`: the id `rproxy` made for the request when it arrived. +- `peer_ip`: the address of `rproxy`'s TCP peer, normally traefik. +- `forwarded_for`: the `X-Forwarded-For` header the visitor's request came with. +- `client_group`: the client as rate limiting counts it, an IPv4 address as a + /32 or an IPv6 group. +- `asn`, `as_name` and `country`: the client's, as looked up. Each is empty + when: + - lookups are off; + - the client is not looked up; + - the source cannot place the client; + - GeoJS had not answered by the time `servewaf` answered. +- `content_type`: the visitor's `Content-Type`. +- `content_length`: the body length the visitor announced. +- `request_headers`: the headers listed in `SWWAF_LOG_REQUEST_HEADERS`, by name + in lower case. +- `has_authorization` and `has_cookie`: true when the request had those headers. + Their values are never logged. + +### Response detail + +- `response_content_type`: the type of `servewaf`'s own answer. - `aborted`: true when `rproxy` went away before the answer. -- `duration_total`: from `servewaf` having read the question's body to the line. -- `time`: when `servewaf` had read the question's body. -The line leaves out `upstream_status`, `websocket` and the -`duration_upstream_...` fields, since `servewaf` passes nothing to the app. +### Decision -The bytes of an allowed request are counted when its report comes. So `counts` -holds the client's bytes without this request's. +- `action`: what was done: + - `forward`: answered `allow`, so that `rproxy` passes it to the app; + - `denied`: refused for `SWWAF_DENY_NETS`, or for a reputation source with + the action `deny`; + - `banned`: refused under a ban, or for a `ban` rule, a trap path or a + CrowdSec decision, each of which bans the client; + - `country_denied`, `rate_limited`, `rule_blocked` and `waf_blocked`: + refused by that check; + - `too_large`: over a header limit; + - `admin`: answered at one of `servewaf`'s own endpoints, or `404` for a + path under `/_smallwebwaf/`. +- `would_action`: in `observe` mode, what `enforce` mode would have done with a + request it would have refused. `action` then names what was done, normally + `forward`. +- `limit_percent` and `limit_percent_setting`: the client's percentage of the + rate limits and the setting that lowered it, when one did. `bytes_percent` and + `bytes_percent_setting` do the same for the byte limits. +- `counts`: the client's totals as rate limiting counts them, this request + included. It is left out when rate limiting did not count the request. It + holds: + - `minute`, `hour` and `day`: requests; + - `minute_bytes`, `hour_bytes` and `day_bytes`: bytes. -### The `report` line + A count can have a fraction (see "Counting method" in + [`SPEC.md`](../SPEC.md)). The bytes of an allowed request are counted when + its report comes. So on the `decision` line, the bytes are without this + request's. -It has every field of the `decision` line, and `time` stays as the `decision` -line gives it. These fields say what the report says: +- `limit_hit`: the limit broken: + - `minute`, `hour` or `day` for a rate limit; + - `minute_bytes`, `hour_bytes` or `day_bytes` for a byte limit; + - `error_burst`. -- `status`: what the visitor got, `0` if nothing; -- `request_bytes` and `response_bytes`: the body bytes each way, with what a - WebSocket carried each way; -- `duration_total`: from the request's headers being read by `rproxy` to the end - of the answer; -- `duration_upstream_total`: from the hand-over to the app to the end of the - answer, for an allowed request. + A request whose bytes broke a limit was passed on whole, so its `action` + stays `forward`. Only its `report` line gives that limit. -For an allowed request, the report's counting also changes these fields: +- `rule_ids`: the rule file rules that matched, in order, up to the one that + refused. +- `waf_score` and `waf_rule_ids`: the Core Rule Set's score and the rules that + matched, for a request it inspected. +- `reputation`: the sources the client is on, in this order: + - blocklist URLs; + - the CrowdSec decision list's URL; + - DNS blocklist zones; + - `abuseipdb`. +- `offence`: `limit`, for a request that broke a limit. +- `ban_expires`: when the client's ban ends, for a request that made one or was + refused under one: a time, or `permanent`. -- `counts`: the byte totals now include this request's; -- `limit_hit`, `offence` and `ban_expires`: set when the report's bytes broke a - byte limit. +### Timings + +All in milliseconds: + +- `duration_total`: + - on the `decision` line, from `servewaf` having read the question's body to + the line; + - on the `report` line, from `rproxy` having read the request's headers to + the end of the answer. +- `duration_checks`: to the end of the checks; +- `duration_waf`: the part of that the Core Rule Set took; +- `duration_upstream_total`: on the `report` line of an allowed request, from + the hand-over to the app to the end of the answer. + +### Requests to its own endpoints + +The line of a request to `servewaf`'s own endpoints has these fields, with these +meanings where they differ: + +- `client_ip` and `peer_ip`: the request's TCP peer. +- `request_id`: a new random one of 26 letters and digits. +- `scheme`: `http`. +- `time`: when the request's headers had been read. +- `request_bytes`: the bytes read of a ban to add, and otherwise `0`. +- `action`: `admin`. +- `duration_total`: from the request's headers being read to the line. ## `rproxy`'s line diff --git a/docs/rules.md b/docs/rules.md index 1bd3fd3..b6f95f5 100644 --- a/docs/rules.md +++ b/docs/rules.md @@ -54,14 +54,19 @@ empty-agent user_agent log ^$ ## A file of your own -An app's `Dockerfile` adds rules of its own in a file named to sort after the -default, such as `50-gitea.rules`: +`servewaf` reads the rule files in its own container. Rules of your own go in a +file named to sort after the default, such as `50-gitea.rules`: ``` # 50-gitea.rules: WordPress probes, which gitea never serves wp-probe path ban (?i)^/(wp-login\.php|xmlrpc\.php|wp-admin/) ``` +A `Dockerfile` built on the `smallwebwaf` image, for `servewaf` to run from, +adds it: + ```dockerfile COPY 50-gitea.rules /etc/smallwebwaf/rules.d/50-gitea.rules ``` + +Or mount it there. A `path` rule applies on every site behind that `servewaf`. diff --git a/docs/settings.md b/docs/settings.md index 9b5855d..3997b25 100644 --- a/docs/settings.md +++ b/docs/settings.md @@ -4,6 +4,10 @@ Every setting, its default and the values it takes, in the groups of "Settings" in [`SPEC.md`](../SPEC.md). That section gives the conventions, the settings that take `off`, the `_FILE` form, and what each group does and why. +The settings that start with `SWWAF_` are `servewaf`'s. Those that start with +`SWRP_` are `rproxy`'s, and are under "`rproxy`" below. `SMALLWEBWAF_COMMAND` +chooses between the two (see "The image's command" below). + ## Forms of values - A duration is Go's syntax, with `d` for days: `90s`, `15m`, `24h`, `7d`. Zero @@ -24,26 +28,22 @@ that take `off`, the `_FILE` form, and what each group does and why. ## Core -- `SWWAF_UPSTREAM_URL` (default `http://127.0.0.1:8081`): the app. An `http` or - `https` URL with a host and an optional port, and nothing more. -- `SWWAF_LISTEN_ADDR` (default `:8080`): where `smallwebwaf` listens: `:` and a +- `SWWAF_LISTEN_ADDR` (default `:8080`): where `servewaf` listens: `:` and a port, such as `:9000`. Leave the host out (see "Ports" in [`SPEC.md`](../SPEC.md)). - `SWWAF_INSTANCE_NAME` (default: the container's host name): the `instance` - name in every log line, alert and metric, such as `fsn1app1/gitea`. Valid - UTF-8. No control character while `SWWAF_ALERT_NTFY_URL` is set. 1 to 48 - printable ASCII characters without a space while `SWWAF_LOG_REMOTE_URL` is set - and `SWWAF_LOG_REMOTE_APP_NAME` is not. + name in each line `servewaf` logs, and in each alert and metric, such as + `fsn1/servewaf`. Valid UTF-8. No control character while + `SWWAF_ALERT_NTFY_URL` is set. 1 to 48 printable ASCII characters without a + space while `SWWAF_LOG_REMOTE_URL` is set and `SWWAF_LOG_REMOTE_APP_NAME` is + not. - `SWWAF_MODE` (default `enforce`): `enforce` or `observe`. -- `SWWAF_TRUSTED_PROXIES` (default `10.0.0.0/8,172.16.0.0/12,192.168.0.0/16`): - the netblocks whose `X-Forwarded-For` is believed. Set but empty, it trusts - nothing. - `SWWAF_IPV6_GROUP_PREFIX` (default `64`): the prefix length of an IPv6 client's group, from 32 to 128. - `SWWAF_MAX_TRACKED_CLIENTS` (default `20000`): the most clients kept, a whole number above zero. -- `SWWAF_MAX_BANS` (default `5000`): how many of its own bans `smallwebwaf` - keeps, a whole number above zero. +- `SWWAF_MAX_BANS` (default `5000`): how many of its own bans `servewaf` keeps, + a whole number above zero. ## Tokens @@ -63,7 +63,7 @@ that take `off`, the `_FILE` form, and what each group does and why. ## Logging -- `SWWAF_LOG_LEVEL` (default `info`): the least severe of `smallwebwaf`'s own +- `SWWAF_LOG_LEVEL` (default `info`): the least severe of `servewaf`'s own messages that are written: `debug`, `info`, `warn` or `error`. - `SWWAF_LOG_REQUEST_HEADERS` (default `accept,accept-language,accept-encoding,content-type,origin,range`): the @@ -125,27 +125,6 @@ that take `off`, the `_FILE` form, and what each group does and why. - `SWWAF_BYTES_COUNT` (default `both`): which body bytes the byte limits count: `response`, `request` or `both`. -## Size and time limits - -- `SWWAF_CLIENT_REQUEST_TIMEOUT` (default `60s`): how long a client may take to - send its request line and headers, and then, from the end of the headers, its - body. A duration or `off`. -- `SWWAF_CLIENT_REQUEST_HEADER_MAX_BYTES` (default `32K`): the largest request - line and headers. A size over `4K`; never `off`. -- `SWWAF_CLIENT_IDLE_TIMEOUT` (default `120s`): how long a kept-open connection - may wait for its next request before it is closed. A duration or `off`. -- `SWWAF_CLIENT_RESPONSE_TIMEOUT` (default `30m`): how long the answer may take - to reach the client, from the end of the request to the last byte. A duration - or `off`. -- `SWWAF_UPSTREAM_REQUEST_TIMEOUT` (default `60s`): how long connecting to the - app and sending it the whole request may take. A duration or `off`. -- `SWWAF_UPSTREAM_RESPONSE_TIMEOUT` (default `30m`): how long the app may take - to send its whole answer. A duration or `off`. -- `SWWAF_REQUEST_MAX_BYTES` (default `100M`): the largest request body. A size - or `off`. -- `SWWAF_RESPONSE_MAX_BYTES` (default `1G`): the largest response body. A size - or `off`. - ## Lookup of AS number and country - `SWWAF_LOOKUP_SOURCE` (default `geojs`): `geojs`, `file` or `off`. `off` stops @@ -160,8 +139,9 @@ that take `off`, the `_FILE` form, and what each group does and why. - `SWWAF_LOOKUP_TIMEOUT` (default `800ms`): how long a request waits for its client's first answer from GeoJS, and how long a request to GeoJS may take. A duration; never `off`. -- `SWWAF_ADD_LOOKUP_HEADERS` (default `false`): `true` passes the app the - client's AS number in `X-Client-ASN` and its country in `X-Client-Country`. +- `SWWAF_ADD_LOOKUP_HEADERS` (default `false`): `true` has `rproxy` pass the app + the client's AS number in `X-Client-ASN` and its country in + `X-Client-Country`. ## Country lists @@ -206,7 +186,9 @@ that take `off`, the `_FILE` form, and what each group does and why. - `SWWAF_WAF_EXEMPT_PATHS` (default empty): path prefixes the Core Rule Set does not inspect. - `SWWAF_WAF_BODY_LIMIT` (default `off`): `off`, or how much of a request body - the Core Rule Set reads, a size of at most `1G`. + the Core Rule Set reads, a size of at most `64K`. A question carries no more + of a body than 64 KiB, and the Core Rule Set reads at most 64 KiB less one + byte of it, whatever the value (see "The answer" in [`SPEC.md`](../SPEC.md)). - `SWWAF_ERROR_BURST_THRESHOLD` (default `30`): the most refusals of one client in a minute (see "Attack detection" in `SPEC.md`). A count or `off`. @@ -327,6 +309,7 @@ Then mount its directory into the container read-only, and name the file: ```sh docker run \ -v /srv/app/tokens:/etc/smallwebwaf/tokens:ro \ + -e SMALLWEBWAF_COMMAND=servewaf \ -e SWWAF_METRICS_TOKEN_FILE=/etc/smallwebwaf/tokens/metrics \ ... ``` @@ -349,22 +332,7 @@ A few limits are fixed rather than settings: - `rproxy` reads at most 1 MiB of the body of `servewaf`'s answer, and answers `502` to a longer one. -## `servewaf` - -`smallwebwaf servewaf` uses every `SWWAF_` setting but these, which it ignores: - -- `SWWAF_UPSTREAM_URL` and `SWWAF_TRUSTED_PROXIES`; -- the five timeouts under "Size and time limits"; -- `SWWAF_CLIENT_REQUEST_HEADER_MAX_BYTES`, `SWWAF_REQUEST_MAX_BYTES` and - `SWWAF_RESPONSE_MAX_BYTES`. - -An invalid value of one of these still stops the start, and the line of settings -at start still shows them. - -`SWWAF_ADD_LOOKUP_HEADERS` puts the headers on the answer to `rproxy`, not on a -request to an app. `SWWAF_WAF_BODY_LIMIT` has an effect up to 64 KiB less one -byte, whatever its value. A question carries no more of a body than 64 KiB (see -"`servewaf`" in [`SPEC.md`](../SPEC.md)). +## `servewaf`'s answer and `rproxy`'s wait `SWWAF_LOOKUP_TIMEOUT` is the longest a question waits on an outside source. Keep it well under `rproxy`'s `SWRP_TIMEOUT_MS`, as the default is. No question @@ -374,10 +342,10 @@ asked in the background (see "Answering in time" under "`servewaf`" in ## `rproxy` -`smallwebwaf rproxy` reads the settings below and no others, and nothing else -reads them. They are read once at start, and an invalid value stops the start. -None can be given as a file, since `rproxy` never touches disk. "`rproxy`" in -[`SPEC.md`](../SPEC.md) says what they do. +`smallwebwaf rproxy` reads the settings below and no others. They are read once +at start, and an invalid value stops the start. None can be given as a file, +since `rproxy` never touches disk. "`rproxy`" in [`SPEC.md`](../SPEC.md) says +what they do. A time here is a whole number of milliseconds above zero, such as `2000`. @@ -406,4 +374,29 @@ A time here is a whole number of milliseconds above zero, such as `2000`. `https` URL with a host and an optional port, and nothing more. - `SWRP_TRUSTED_PROXIES` (default `10.0.0.0/8,172.16.0.0/12,192.168.0.0/16`): the netblocks whose `X-Forwarded-For` is believed. Set but empty, it trusts - nothing. + nothing. A client outside these netblocks cannot claim another address. Where + other containers reach `rproxy` directly, or visitors on a private network + reach traefik, they can name another address in the header. Narrowing the + setting to traefik's own network closes that gap. + +## The image's command + +`SMALLWEBWAF_COMMAND` (default `rproxy`) chooses what the image runs and what +its health check checks: `rproxy` or `servewaf` (see "Deployment" in +[`SPEC.md`](../SPEC.md)). It cannot be given as a file. + +## Where earlier settings went + +Nothing reads these names any more: + +- `SWWAF_UPSTREAM_URL` is `SWRP_UPSTREAM_URL`. +- `SWWAF_TRUSTED_PROXIES` is `SWRP_TRUSTED_PROXIES`. +- `SWWAF_REQUEST_MAX_BYTES` is `SWRP_MAX_REQUEST_SIZE`. +- `SWWAF_RESPONSE_MAX_BYTES` is `SWRP_MAX_RESPONSE_SIZE`. +- `SWWAF_CLIENT_REQUEST_TIMEOUT` and `SWWAF_UPSTREAM_REQUEST_TIMEOUT` are + `SWRP_MAX_REQUEST_TIME_MS`, in milliseconds. +- `SWWAF_CLIENT_RESPONSE_TIMEOUT` and `SWWAF_UPSTREAM_RESPONSE_TIMEOUT` are + `SWRP_MAX_RESPONSE_TIME_MS`, in milliseconds. +- `SWWAF_CLIENT_IDLE_TIMEOUT` and `SWWAF_CLIENT_REQUEST_HEADER_MAX_BYTES` have + no setting. `rproxy` closes a kept-open connection idle for 120 seconds, and + takes a request line and headers of at most 32 KiB. diff --git a/docs/state-files.md b/docs/state-files.md index eb8a4f5..509dc92 100644 --- a/docs/state-files.md +++ b/docs/state-files.md @@ -7,7 +7,7 @@ Each has a top-level `version` of 1. ## `bans.json` Active and past bans: every ban an admin made, and up to `SWWAF_MAX_BANS` of -those `smallwebwaf` made. Per entry: +those `servewaf` made. Per entry: - `netblock`, `start` and `expires` (`null` for a permanent ban); - `cause`: `attack`, `limit`, `admin` or `crowdsec`; @@ -63,8 +63,8 @@ since it was first seen: - total requests, and how many were forwarded and how many refused; - bytes in each direction; - responses by status class; -- offences by kind: `limit`, `attack`, `rule_blocked`, `waf_blocked` and - `token_refused`. +- offences by kind: `limit`, `attack`, `rule_blocked` and `waf_blocked`. A file + may also hold `token_refused`, which is kept, and which nothing adds to. ## `lookups.json` diff --git a/internal/config/config.go b/internal/config/config.go index 926483c..dbe8dd9 100644 --- a/internal/config/config.go +++ b/internal/config/config.go @@ -1,9 +1,10 @@ // Package config reads smallwebwaf's settings, and is the one place they are -// read. Every setting is an environment variable. The settings of -// `smallwebwaf` and `smallwebwaf servewaf` start with SWWAF_, every one of them -// has a default, and most can be given as a file instead. Those of `smallwebwaf -// rproxy` start with SWRP_. Each of them but SWRP_WAF_BACKEND has a default, and -// none can be given as a file, since rproxy never touches disk. +// read. Every setting is an environment variable. The settings of `smallwebwaf +// servewaf` start with SWWAF_, every one of them has a default, and most can be +// given as a file instead. Those of `smallwebwaf rproxy` start with SWRP_. Each +// of them but SWRP_WAF_BACKEND has a default, and none can be given as a file, +// since rproxy never touches disk. SMALLWEBWAF_COMMAND chooses which of the two +// the image runs. package config import ( @@ -30,13 +31,11 @@ import ( "sneak.berlin/go/smallwebwaf/internal/remotelog" ) -// Config is smallwebwaf's settings. A timeout, size or rate limit of zero -// is off. +// Config is the settings of `smallwebwaf servewaf`. A time or size limit of +// zero is off. type Config struct { - // ListenAddr is where smallwebwaf listens (SWWAF_LISTEN_ADDR). + // ListenAddr is where servewaf listens (SWWAF_LISTEN_ADDR). ListenAddr string - // UpstreamURL is the app (SWWAF_UPSTREAM_URL). - UpstreamURL *url.URL // InstanceName is the name every log line and alert gives as instance, // and every metric carries as its label instance (SWWAF_INSTANCE_NAME), // by default the host's name, which docker sets to the first 12 @@ -45,42 +44,14 @@ type Config struct { // Observe is true in observe mode, when SWWAF_MODE is observe rather // than enforce: a request that SWWAF_DENY_NETS, a ban, the country // lists, a rate limit, a rule or the Core Rule Set would refuse is - // passed to the app instead, and no ban is made. + // allowed instead, and no ban is made. Observe bool - // TrustedProxies are the netblocks whose X-Forwarded-For is - // believed (SWWAF_TRUSTED_PROXIES). - TrustedProxies []netip.Prefix // IPv6GroupPrefix is the length of the IPv6 netblock that is one client // (SWWAF_IPV6_GROUP_PREFIX), from 32 to 128. IPv6GroupPrefix int // MaxTrackedClients is the most clients the table of clients holds, in // memory and in clients.json (SWWAF_MAX_TRACKED_CLIENTS). MaxTrackedClients int - // ClientRequestTimeout bounds reading the whole request from the - // client (SWWAF_CLIENT_REQUEST_TIMEOUT). - ClientRequestTimeout time.Duration - // ClientRequestHeaderMaxBytes is the largest request line and headers - // a client may send (SWWAF_CLIENT_REQUEST_HEADER_MAX_BYTES). It is - // never off, and always more than 4K. - ClientRequestHeaderMaxBytes int64 - // ClientIdleTimeout bounds how long a kept-open client connection - // may wait for its next request (SWWAF_CLIENT_IDLE_TIMEOUT). - ClientIdleTimeout time.Duration - // ClientResponseTimeout bounds writing the whole response to the - // client (SWWAF_CLIENT_RESPONSE_TIMEOUT). - ClientResponseTimeout time.Duration - // UpstreamRequestTimeout bounds connecting to the app and writing - // the whole request to it (SWWAF_UPSTREAM_REQUEST_TIMEOUT). - UpstreamRequestTimeout time.Duration - // UpstreamResponseTimeout bounds reading the whole response from - // the app (SWWAF_UPSTREAM_RESPONSE_TIMEOUT). - UpstreamResponseTimeout time.Duration - // RequestMaxBytes is the largest request body - // (SWWAF_REQUEST_MAX_BYTES). - RequestMaxBytes int64 - // ResponseMaxBytes is the largest response body - // (SWWAF_RESPONSE_MAX_BYTES). - ResponseMaxBytes int64 // AllowNets are the netblocks whose clients skip every check // (SWWAF_ALLOW_NETS). RateLimitExemptNets are those whose clients the // rate limits neither count nor refuse (SWWAF_RATE_LIMIT_EXEMPT_NETS). @@ -260,9 +231,8 @@ type Config struct { // attack (SWWAF_TRAP_PATHS), each starting with / and without a ?. TrapPaths []string // ErrorBurstThreshold is the most requests of a client within a minute - // that smallwebwaf may refuse after a rule file or Core Rule Set match - // or for a missing or wrong token; one more breaks a limit - // (SWWAF_ERROR_BURST_THRESHOLD). 0 is off. + // that servewaf may refuse after a rule file or Core Rule Set match; one + // more breaks a limit (SWWAF_ERROR_BURST_THRESHOLD). 0 is off. ErrorBurstThreshold int64 // LogRemoteURL is where every line on stdout is also sent // (SWWAF_LOG_REMOTE_URL), nil while it is unset and nothing is sent. @@ -365,8 +335,8 @@ const ( // masked is what the log shows for a token that is set, and in place of // a secret in another setting. masked = "********" - // defaultListenAddr and defaultUpstreamURL are the defaults of - // SWWAF_LISTEN_ADDR and SWWAF_UPSTREAM_URL. + // defaultListenAddr is the default of SWWAF_LISTEN_ADDR and + // SWRP_LISTEN_ADDR, and defaultUpstreamURL that of SWRP_UPSTREAM_URL. defaultListenAddr = ":8080" defaultUpstreamURL = "http://127.0.0.1:8081" ) @@ -394,13 +364,14 @@ var ( errHeaderTakenOut = errors.New( "is taken out of every request by Go's HTTP server, so it can never " + "be logged") - errOnBothLists = errors.New("is in SWWAF_DENIED_COUNTRIES too") - errNotLookupSource = errors.New("is not geojs, file or off") - errNeedsLookups = errors.New("it needs each client looked up") - errNeedsDBPath = errors.New("it names the file to look clients up in") - errDBPathUnused = errors.New("only file reads it") - errNotOver4K = errors.New("is not a size of more than 4K, such as 32K") - errOver1G = errors.New("is more than 1G, the most Coraza reads") + errOnBothLists = errors.New("is in SWWAF_DENIED_COUNTRIES too") + errNotLookupSource = errors.New("is not geojs, file or off") + errNeedsLookups = errors.New("it needs each client looked up") + errNeedsDBPath = errors.New("it names the file to look clients up in") + errDBPathUnused = errors.New("only file reads it") + errOver64K = errors.New( + "is more than 64K, the most servewaf gets of a body") + errNotCommand = errors.New("is not rproxy or servewaf") errNotDurationAboveZero = errors.New( "is not a duration above zero, such as 1h or 7d") errNotNumberAboveZero = errors.New( @@ -489,38 +460,27 @@ var ( func FromEnvironment(lookupEnv func(string) (string, bool)) (*Config, error) { env := &environment{lookupEnv: lookupEnv} cfg := &Config{ - ListenAddr: env.address("SWWAF_LISTEN_ADDR", defaultListenAddr), - UpstreamURL: env.appURL("SWWAF_UPSTREAM_URL", defaultUpstreamURL), + ListenAddr: env.address("SWWAF_LISTEN_ADDR"), InstanceName: env.instanceName(), Observe: env.observe("SWWAF_MODE", "enforce"), - TrustedProxies: env.netblocks("SWWAF_TRUSTED_PROXIES", privateRanges), IPv6GroupPrefix: env.ipv6GroupPrefix("SWWAF_IPV6_GROUP_PREFIX", "64"), MaxTrackedClients: env.numberNotOff("SWWAF_MAX_TRACKED_CLIENTS", "20000"), - ClientRequestTimeout: env.duration("SWWAF_CLIENT_REQUEST_TIMEOUT", "60s"), - ClientRequestHeaderMaxBytes: env.headerSize( - "SWWAF_CLIENT_REQUEST_HEADER_MAX_BYTES", "32K"), - ClientIdleTimeout: env.duration("SWWAF_CLIENT_IDLE_TIMEOUT", "120s"), - ClientResponseTimeout: env.duration("SWWAF_CLIENT_RESPONSE_TIMEOUT", "30m"), - UpstreamRequestTimeout: env.duration("SWWAF_UPSTREAM_REQUEST_TIMEOUT", "60s"), - UpstreamResponseTimeout: env.duration("SWWAF_UPSTREAM_RESPONSE_TIMEOUT", "30m"), - RequestMaxBytes: env.size("SWWAF_REQUEST_MAX_BYTES", "100M"), - ResponseMaxBytes: env.size("SWWAF_RESPONSE_MAX_BYTES", "1G"), - AllowNets: env.netblocks("SWWAF_ALLOW_NETS", ""), - RateLimitExemptNets: env.netblocks("SWWAF_RATE_LIMIT_EXEMPT_NETS", ""), - DenyNets: env.netblocks("SWWAF_DENY_NETS", ""), - RateLimitPerMinute: env.count("SWWAF_RATE_LIMIT_PER_MINUTE", "500"), - RateLimitPerHour: env.count("SWWAF_RATE_LIMIT_PER_HOUR", "20000"), - RateLimitPerDay: env.count("SWWAF_RATE_LIMIT_PER_DAY", "300000"), - RateLimitExemptPaths: env.pathPrefixes("SWWAF_RATE_LIMIT_EXEMPT_PATHS", ""), - BytesLimitPerMinute: env.size("SWWAF_BYTES_LIMIT_PER_MINUTE", "4G"), - BytesLimitPerHour: env.size("SWWAF_BYTES_LIMIT_PER_HOUR", "20G"), - BytesLimitPerDay: env.size("SWWAF_BYTES_LIMIT_PER_DAY", "200G"), - BytesCount: env.bytesCount("SWWAF_BYTES_COUNT", "both"), - LookupSource: env.lookupSource("SWWAF_LOOKUP_SOURCE", "geojs"), - LookupDBPath: env.value("SWWAF_LOOKUP_DB_PATH", ""), - LookupTimeout: env.durationNotOff("SWWAF_LOOKUP_TIMEOUT", "800ms"), - AddLookupHeaders: env.boolean("SWWAF_ADD_LOOKUP_HEADERS", "false"), - DeniedCountries: env.countries("SWWAF_DENIED_COUNTRIES", ""), + AllowNets: env.netblocks("SWWAF_ALLOW_NETS", ""), + RateLimitExemptNets: env.netblocks("SWWAF_RATE_LIMIT_EXEMPT_NETS", ""), + DenyNets: env.netblocks("SWWAF_DENY_NETS", ""), + RateLimitPerMinute: env.count("SWWAF_RATE_LIMIT_PER_MINUTE", "500"), + RateLimitPerHour: env.count("SWWAF_RATE_LIMIT_PER_HOUR", "20000"), + RateLimitPerDay: env.count("SWWAF_RATE_LIMIT_PER_DAY", "300000"), + RateLimitExemptPaths: env.pathPrefixes("SWWAF_RATE_LIMIT_EXEMPT_PATHS", ""), + BytesLimitPerMinute: env.size("SWWAF_BYTES_LIMIT_PER_MINUTE", "4G"), + BytesLimitPerHour: env.size("SWWAF_BYTES_LIMIT_PER_HOUR", "20G"), + BytesLimitPerDay: env.size("SWWAF_BYTES_LIMIT_PER_DAY", "200G"), + BytesCount: env.bytesCount("SWWAF_BYTES_COUNT", "both"), + LookupSource: env.lookupSource("SWWAF_LOOKUP_SOURCE", "geojs"), + LookupDBPath: env.value("SWWAF_LOOKUP_DB_PATH", ""), + LookupTimeout: env.durationNotOff("SWWAF_LOOKUP_TIMEOUT", "800ms"), + AddLookupHeaders: env.boolean("SWWAF_ADD_LOOKUP_HEADERS", "false"), + DeniedCountries: env.countries("SWWAF_DENIED_COUNTRIES", ""), ExclusivelyAllowedCountries: env.countries( "SWWAF_EXCLUSIVELY_ALLOWED_COUNTRIES", ""), ASNLimitPercent: env.percents("SWWAF_ASN_LIMIT_PERCENT", ParseASN), @@ -612,22 +572,30 @@ func FromEnvironment(lookupEnv func(string) (string, bool)) (*Config, error) { return cfg, nil } -// ListenAddrAndUpstreamURL reads only SWWAF_LISTEN_ADDR and -// SWWAF_UPSTREAM_URL, either of which may be given as a file, as -// FromEnvironment does. The health check needs no other setting, so it -// reads no other, nor a file that another names. -func ListenAddrAndUpstreamURL( - lookupEnv func(string) (string, bool), -) (string, *url.URL, error) { - env := &environment{lookupEnv: lookupEnv} - listenAddr := env.address("SWWAF_LISTEN_ADDR", defaultListenAddr) - upstreamURL := env.appURL("SWWAF_UPSTREAM_URL", defaultUpstreamURL) +// Command reads only SMALLWEBWAF_COMMAND, the command the image runs and its +// health check checks: rproxy when it is unset or empty, or servewaf. It cannot +// be given as a file. Any other value is an error that names the setting. +func Command(lookupEnv func(string) (string, bool)) (string, error) { + command, _ := lookupEnv("SMALLWEBWAF_COMMAND") - if env.err != nil { - return "", nil, env.err + switch command { + case "", "rproxy": + return "rproxy", nil + case "servewaf": + return command, nil + default: + return "", fmt.Errorf("SMALLWEBWAF_COMMAND: %q %w", command, errNotCommand) } +} - return listenAddr, upstreamURL, nil +// ServeWAFListenAddr reads only SWWAF_LISTEN_ADDR, which may be given as a +// file, as FromEnvironment does. The health check in servewaf's container needs +// no other setting, so it reads no other, nor a file that another names. +func ServeWAFListenAddr(lookupEnv func(string) (string, bool)) (string, error) { + env := &environment{lookupEnv: lookupEnv} + listenAddr := env.address("SWWAF_LISTEN_ADDR") + + return listenAddr, env.err } // InstanceName reads only SWWAF_INSTANCE_NAME, which may be given as a @@ -709,9 +677,10 @@ func (e *environment) check(name string, err error) { } } -// address reads a setting that is an address to listen on. -func (e *environment) address(name, defaultValue string) string { - address, err := parseListenAddr(e.value(name, defaultValue)) +// address reads a setting that is an address to listen on, by default +// defaultListenAddr. +func (e *environment) address(name string) string { + address, err := parseListenAddr(e.value(name, defaultListenAddr)) e.check(name, err) return address @@ -779,15 +748,6 @@ func (e *environment) wafBodyLimit(name, defaultValue string) int64 { return limit } -// headerSize reads the setting that is the largest request line and -// headers. -func (e *environment) headerSize(name, defaultValue string) int64 { - size, err := parseHeaderSize(e.value(name, defaultValue)) - e.check(name, err) - - return size -} - // count reads a setting that is a number of requests. func (e *environment) count(name, defaultValue string) int64 { count, err := parseCount(e.value(name, defaultValue)) @@ -1434,30 +1394,17 @@ func parseSize(value string) (int64, error) { return n * unit, nil } -// parseHeaderSize reads the largest request line and headers: a size as -// parseSize reads it, but more than 4K and never off. Go's server reads 4K -// past the limit it is given before it refuses, so proxy.New gives it this -// size less 4K, which must leave a limit. -func parseHeaderSize(value string) (int64, error) { - size, err := parseSize(value) - if err != nil || size <= 4*kibibyte { - return 0, fmt.Errorf("%q %w", value, errNotOver4K) - } - - return size, nil -} - // parseWAFBodyLimit reads the most of a request body the Core Rule Set -// reads: a size as parseSize reads it, or off, but at most 1G, since -// Coraza, which runs the Core Rule Set, refuses to load with more. +// reads: a size as parseSize reads it, or off, but at most 64K, since servewaf +// gets no more of a body than that. func parseWAFBodyLimit(value string) (int64, error) { limit, err := parseSize(value) if err != nil { return 0, err } - if limit > gibibyte { - return 0, fmt.Errorf("%q %w", value, errOver1G) + if limit > 64*kibibyte { + return 0, fmt.Errorf("%q %w", value, errOver64K) } return limit, nil diff --git a/internal/config/config_test.go b/internal/config/config_test.go index 67c3f33..fa256cb 100644 --- a/internal/config/config_test.go +++ b/internal/config/config_test.go @@ -23,98 +23,88 @@ import ( // The settings, by name. const ( - listenAddr = "SWWAF_LISTEN_ADDR" - upstreamURL = "SWWAF_UPSTREAM_URL" - mode = "SWWAF_MODE" - trustedProxies = "SWWAF_TRUSTED_PROXIES" - ipv6GroupPrefix = "SWWAF_IPV6_GROUP_PREFIX" - maxTrackedClients = "SWWAF_MAX_TRACKED_CLIENTS" - clientRequestTimeout = "SWWAF_CLIENT_REQUEST_TIMEOUT" - clientHeaderMaxBytes = "SWWAF_CLIENT_REQUEST_HEADER_MAX_BYTES" - clientIdleTimeout = "SWWAF_CLIENT_IDLE_TIMEOUT" - clientResponseTimeout = "SWWAF_CLIENT_RESPONSE_TIMEOUT" - upstreamRequestTimeout = "SWWAF_UPSTREAM_REQUEST_TIMEOUT" - upstreamResponseTimeout = "SWWAF_UPSTREAM_RESPONSE_TIMEOUT" - requestMaxBytes = "SWWAF_REQUEST_MAX_BYTES" - responseMaxBytes = "SWWAF_RESPONSE_MAX_BYTES" - allowNets = "SWWAF_ALLOW_NETS" - rateLimitExemptNets = "SWWAF_RATE_LIMIT_EXEMPT_NETS" - denyNets = "SWWAF_DENY_NETS" - rateLimitPerMinute = "SWWAF_RATE_LIMIT_PER_MINUTE" - rateLimitPerHour = "SWWAF_RATE_LIMIT_PER_HOUR" - rateLimitPerDay = "SWWAF_RATE_LIMIT_PER_DAY" - rateLimitExemptPaths = "SWWAF_RATE_LIMIT_EXEMPT_PATHS" - bytesLimitPerMinute = "SWWAF_BYTES_LIMIT_PER_MINUTE" - bytesLimitPerHour = "SWWAF_BYTES_LIMIT_PER_HOUR" - bytesLimitPerDay = "SWWAF_BYTES_LIMIT_PER_DAY" - bytesCount = "SWWAF_BYTES_COUNT" - lookupSource = "SWWAF_LOOKUP_SOURCE" - lookupDBPath = "SWWAF_LOOKUP_DB_PATH" - lookupTimeout = "SWWAF_LOOKUP_TIMEOUT" - addLookupHeaders = "SWWAF_ADD_LOOKUP_HEADERS" - deniedCountries = "SWWAF_DENIED_COUNTRIES" - allowedCountries = "SWWAF_EXCLUSIVELY_ALLOWED_COUNTRIES" - asnLimitPercent = "SWWAF_ASN_LIMIT_PERCENT" - countryLimitPercent = "SWWAF_COUNTRY_LIMIT_PERCENT" - asnBytesPercent = "SWWAF_ASN_BYTES_PERCENT" - countryBytesPercent = "SWWAF_COUNTRY_BYTES_PERCENT" - unknownLimitPercent = "SWWAF_UNKNOWN_LIMIT_PERCENT" - asnLimitPercentURL = "SWWAF_ASN_LIMIT_PERCENT_URL" - blocklistURLs = "SWWAF_BLOCKLIST_URLS" - blocklistRefresh = "SWWAF_BLOCKLIST_REFRESH" - blocklistAction = "SWWAF_BLOCKLIST_ACTION" - dnsblZones = "SWWAF_DNSBL_ZONES" - dnsblResolver = "SWWAF_DNSBL_RESOLVER" - abuseIPDBKey = "SWWAF_ABUSEIPDB_KEY" - abuseIPDBMinScore = "SWWAF_ABUSEIPDB_MIN_SCORE" - abuseIPDBDailyBudget = "SWWAF_ABUSEIPDB_DAILY_BUDGET" - reputationAction = "SWWAF_REPUTATION_ACTION" - reputationCacheTTL = "SWWAF_REPUTATION_CACHE_TTL" - reputationTimeout = "SWWAF_REPUTATION_TIMEOUT" - crowdSecURL = "SWWAF_CROWDSEC_LAPI_URL" - crowdSecKey = "SWWAF_CROWDSEC_LAPI_KEY" - banResponse = "SWWAF_BAN_RESPONSE" - limitBanDuration = "SWWAF_LIMIT_BAN_DURATION" - limitBanRepeatWindow = "SWWAF_LIMIT_BAN_REPEAT_WINDOW" - maxBanDuration = "SWWAF_MAX_BAN_DURATION" - attackBanDuration = "SWWAF_ATTACK_BAN_DURATION" - maxBans = "SWWAF_MAX_BANS" - banScopeV4Prefix = "SWWAF_BAN_SCOPE_V4_PREFIX" - stateDir = "SWWAF_STATE_DIR" - stateWriteDelay = "SWWAF_STATE_WRITE_DELAY" - stateCounterInterval = "SWWAF_STATE_COUNTER_INTERVAL" - adminToken = "SWWAF_ADMIN_TOKEN" //nolint:gosec // the setting's name - metricsToken = "SWWAF_METRICS_TOKEN" //nolint:gosec // the setting's name - metricsTopN = "SWWAF_METRICS_TOP_N" - instanceName = "SWWAF_INSTANCE_NAME" - logRequestHeaders = "SWWAF_LOG_REQUEST_HEADERS" - logLevel = "SWWAF_LOG_LEVEL" - rulesDir = "SWWAF_RULES_DIR" - rulesEnabled = "SWWAF_RULES_ENABLED" - wafMode = "SWWAF_WAF_MODE" - wafParanoiaLevel = "SWWAF_WAF_PARANOIA_LEVEL" - wafAnomalyThreshold = "SWWAF_WAF_ANOMALY_THRESHOLD" - wafDisabledRules = "SWWAF_WAF_DISABLED_RULES" - wafExemptPaths = "SWWAF_WAF_EXEMPT_PATHS" - wafBodyLimit = "SWWAF_WAF_BODY_LIMIT" - trapPaths = "SWWAF_TRAP_PATHS" - errorBurstThreshold = "SWWAF_ERROR_BURST_THRESHOLD" - logRemoteURL = "SWWAF_LOG_REMOTE_URL" - logRemoteTLSCAFile = "SWWAF_LOG_REMOTE_TLS_CA_FILE" - logRemoteBuffer = "SWWAF_LOG_REMOTE_BUFFER" - logRemoteFacility = "SWWAF_LOG_REMOTE_FACILITY" - logRemoteAppName = "SWWAF_LOG_REMOTE_APP_NAME" - alertWebhookURL = "SWWAF_ALERT_WEBHOOK_URL" - alertWebhookHeaders = "SWWAF_ALERT_WEBHOOK_HEADERS" - alertSlackWebhookURL = "SWWAF_ALERT_SLACK_WEBHOOK_URL" - alertNtfyURL = "SWWAF_ALERT_NTFY_URL" - alertNtfyToken = "SWWAF_ALERT_NTFY_TOKEN" //nolint:gosec // the setting's name - alertEvents = "SWWAF_ALERT_EVENTS" - alertCooldown = "SWWAF_ALERT_COOLDOWN" - alertMaxPerHour = "SWWAF_ALERT_MAX_PER_HOUR" - anomalyNetV4Prefix = "SWWAF_ANOMALY_NET_V4_PREFIX" - anomalyNetV6Prefix = "SWWAF_ANOMALY_NET_V6_PREFIX" - watchNets = "SWWAF_WATCH_NETS" + listenAddr = "SWWAF_LISTEN_ADDR" + mode = "SWWAF_MODE" + ipv6GroupPrefix = "SWWAF_IPV6_GROUP_PREFIX" + maxTrackedClients = "SWWAF_MAX_TRACKED_CLIENTS" + allowNets = "SWWAF_ALLOW_NETS" + rateLimitExemptNets = "SWWAF_RATE_LIMIT_EXEMPT_NETS" + denyNets = "SWWAF_DENY_NETS" + rateLimitPerMinute = "SWWAF_RATE_LIMIT_PER_MINUTE" + rateLimitPerHour = "SWWAF_RATE_LIMIT_PER_HOUR" + rateLimitPerDay = "SWWAF_RATE_LIMIT_PER_DAY" + rateLimitExemptPaths = "SWWAF_RATE_LIMIT_EXEMPT_PATHS" + bytesLimitPerMinute = "SWWAF_BYTES_LIMIT_PER_MINUTE" + bytesLimitPerHour = "SWWAF_BYTES_LIMIT_PER_HOUR" + bytesLimitPerDay = "SWWAF_BYTES_LIMIT_PER_DAY" + bytesCount = "SWWAF_BYTES_COUNT" + lookupSource = "SWWAF_LOOKUP_SOURCE" + lookupDBPath = "SWWAF_LOOKUP_DB_PATH" + lookupTimeout = "SWWAF_LOOKUP_TIMEOUT" + addLookupHeaders = "SWWAF_ADD_LOOKUP_HEADERS" + deniedCountries = "SWWAF_DENIED_COUNTRIES" + allowedCountries = "SWWAF_EXCLUSIVELY_ALLOWED_COUNTRIES" + asnLimitPercent = "SWWAF_ASN_LIMIT_PERCENT" + countryLimitPercent = "SWWAF_COUNTRY_LIMIT_PERCENT" + asnBytesPercent = "SWWAF_ASN_BYTES_PERCENT" + countryBytesPercent = "SWWAF_COUNTRY_BYTES_PERCENT" + unknownLimitPercent = "SWWAF_UNKNOWN_LIMIT_PERCENT" + asnLimitPercentURL = "SWWAF_ASN_LIMIT_PERCENT_URL" + blocklistURLs = "SWWAF_BLOCKLIST_URLS" + blocklistRefresh = "SWWAF_BLOCKLIST_REFRESH" + blocklistAction = "SWWAF_BLOCKLIST_ACTION" + dnsblZones = "SWWAF_DNSBL_ZONES" + dnsblResolver = "SWWAF_DNSBL_RESOLVER" + abuseIPDBKey = "SWWAF_ABUSEIPDB_KEY" + abuseIPDBMinScore = "SWWAF_ABUSEIPDB_MIN_SCORE" + abuseIPDBDailyBudget = "SWWAF_ABUSEIPDB_DAILY_BUDGET" + reputationAction = "SWWAF_REPUTATION_ACTION" + reputationCacheTTL = "SWWAF_REPUTATION_CACHE_TTL" + reputationTimeout = "SWWAF_REPUTATION_TIMEOUT" + crowdSecURL = "SWWAF_CROWDSEC_LAPI_URL" + crowdSecKey = "SWWAF_CROWDSEC_LAPI_KEY" + banResponse = "SWWAF_BAN_RESPONSE" + limitBanDuration = "SWWAF_LIMIT_BAN_DURATION" + limitBanRepeatWindow = "SWWAF_LIMIT_BAN_REPEAT_WINDOW" + maxBanDuration = "SWWAF_MAX_BAN_DURATION" + attackBanDuration = "SWWAF_ATTACK_BAN_DURATION" + maxBans = "SWWAF_MAX_BANS" + banScopeV4Prefix = "SWWAF_BAN_SCOPE_V4_PREFIX" + stateDir = "SWWAF_STATE_DIR" + stateWriteDelay = "SWWAF_STATE_WRITE_DELAY" + stateCounterInterval = "SWWAF_STATE_COUNTER_INTERVAL" + adminToken = "SWWAF_ADMIN_TOKEN" //nolint:gosec // the setting's name + metricsToken = "SWWAF_METRICS_TOKEN" //nolint:gosec // the setting's name + metricsTopN = "SWWAF_METRICS_TOP_N" + instanceName = "SWWAF_INSTANCE_NAME" + logRequestHeaders = "SWWAF_LOG_REQUEST_HEADERS" + logLevel = "SWWAF_LOG_LEVEL" + rulesDir = "SWWAF_RULES_DIR" + rulesEnabled = "SWWAF_RULES_ENABLED" + wafMode = "SWWAF_WAF_MODE" + wafParanoiaLevel = "SWWAF_WAF_PARANOIA_LEVEL" + wafAnomalyThreshold = "SWWAF_WAF_ANOMALY_THRESHOLD" + wafDisabledRules = "SWWAF_WAF_DISABLED_RULES" + wafExemptPaths = "SWWAF_WAF_EXEMPT_PATHS" + wafBodyLimit = "SWWAF_WAF_BODY_LIMIT" + trapPaths = "SWWAF_TRAP_PATHS" + errorBurstThreshold = "SWWAF_ERROR_BURST_THRESHOLD" + logRemoteURL = "SWWAF_LOG_REMOTE_URL" + logRemoteTLSCAFile = "SWWAF_LOG_REMOTE_TLS_CA_FILE" + logRemoteBuffer = "SWWAF_LOG_REMOTE_BUFFER" + logRemoteFacility = "SWWAF_LOG_REMOTE_FACILITY" + logRemoteAppName = "SWWAF_LOG_REMOTE_APP_NAME" + alertWebhookURL = "SWWAF_ALERT_WEBHOOK_URL" + alertWebhookHeaders = "SWWAF_ALERT_WEBHOOK_HEADERS" + alertSlackWebhookURL = "SWWAF_ALERT_SLACK_WEBHOOK_URL" + alertNtfyURL = "SWWAF_ALERT_NTFY_URL" + alertNtfyToken = "SWWAF_ALERT_NTFY_TOKEN" //nolint:gosec // the setting's name + alertEvents = "SWWAF_ALERT_EVENTS" + alertCooldown = "SWWAF_ALERT_COOLDOWN" + alertMaxPerHour = "SWWAF_ALERT_MAX_PER_HOUR" + anomalyNetV4Prefix = "SWWAF_ANOMALY_NET_V4_PREFIX" + anomalyNetV6Prefix = "SWWAF_ANOMALY_NET_V6_PREFIX" + watchNets = "SWWAF_WATCH_NETS" ) // The anomaly thresholds: each of the prefixes below, which name a scope, @@ -252,33 +242,25 @@ func TestDefaults(t *testing.T) { cfg := fromEnvironment(t, environment{}) wantSettings(t, cfg, config.Config{ - ListenAddr: ":8080", - Observe: false, - ClientRequestTimeout: time.Minute, - ClientRequestHeaderMaxBytes: 32 << 10, - ClientIdleTimeout: 2 * time.Minute, - ClientResponseTimeout: 30 * time.Minute, - UpstreamRequestTimeout: time.Minute, - UpstreamResponseTimeout: 30 * time.Minute, - RequestMaxBytes: 100 << 20, - ResponseMaxBytes: 1 << 30, - RateLimitPerMinute: 500, - RateLimitPerHour: 20000, - RateLimitPerDay: 300000, - BanResponse: 403, - LimitBanDuration: time.Hour, - LimitBanRepeatWindow: 24 * time.Hour, - MaxBanDuration: 7 * 24 * time.Hour, - AttackBanDuration: 7 * 24 * time.Hour, - MaxBans: 5000, - BanScopeV4Prefix: 32, - StateDir: "/var/lib/smallwebwaf", - StateWriteDelay: 10 * time.Second, - StateCounterInterval: 15 * time.Minute, - MetricsToken: "", - MetricsTopN: 50, - RulesDir: "/etc/smallwebwaf/rules.d", - RulesEnabled: true, + ListenAddr: ":8080", + Observe: false, + RateLimitPerMinute: 500, + RateLimitPerHour: 20000, + RateLimitPerDay: 300000, + BanResponse: 403, + LimitBanDuration: time.Hour, + LimitBanRepeatWindow: 24 * time.Hour, + MaxBanDuration: 7 * 24 * time.Hour, + AttackBanDuration: 7 * 24 * time.Hour, + MaxBans: 5000, + BanScopeV4Prefix: 32, + StateDir: "/var/lib/smallwebwaf", + StateWriteDelay: 10 * time.Second, + StateCounterInterval: 15 * time.Minute, + MetricsToken: "", + MetricsTopN: 50, + RulesDir: "/etc/smallwebwaf/rules.d", + RulesEnabled: true, }) wantLookupSettings(t, cfg, config.Config{ LookupSource: defaultLookupSource, LookupTimeout: 800 * time.Millisecond, @@ -288,12 +270,6 @@ func TestDefaults(t *testing.T) { BytesLimitPerDay: 200 << 30, BytesCount: "both", }) - if cfg.UpstreamURL.String() != "http://127.0.0.1:8081" { - t.Errorf("%s is %s", upstreamURL, cfg.UpstreamURL) - } - - wantNetblocks(t, cfg.TrustedProxies, - "10.0.0.0/8", "172.16.0.0/12", "192.168.0.0/16") wantNetblocks(t, cfg.AllowNets) wantNetblocks(t, cfg.RateLimitExemptNets) wantNetblocks(t, cfg.DenyNets) @@ -322,8 +298,13 @@ func TestNoSingleRequestBreaksAByteLimitAtTheDefaults(t *testing.T) { cfg := fromEnvironment(t, environment{}) + rproxy, err := rproxyConfig(map[string]string{wafBackend: backendURL}) + if err != nil { + t.Fatalf("settings: %v", err) + } + // The largest request body and the largest response, both counted. - largest := cfg.RequestMaxBytes + cfg.ResponseMaxBytes + largest := rproxy.MaxRequestSize + rproxy.MaxResponseSize for _, limit := range []int64{ cfg.BytesLimitPerMinute, cfg.BytesLimitPerHour, cfg.BytesLimitPerDay, } { @@ -337,77 +318,54 @@ func TestValuesAsSet(t *testing.T) { t.Parallel() cfg := fromEnvironment(t, environment{ - listenAddr: "127.0.0.1:9000", - upstreamURL: "https://app.internal:8443/", - mode: "observe", - trustedProxies: " 192.0.2.1, 10.1.2.3/8 ,2001:db8::/32", - clientRequestTimeout: "90s", - clientHeaderMaxBytes: "8K", - clientIdleTimeout: "5m", - clientResponseTimeout: "7d", - upstreamRequestTimeout: "1h30m", - upstreamResponseTimeout: off, - requestMaxBytes: "512K", - responseMaxBytes: "1234", - allowNets: "192.0.2.7", - rateLimitExemptNets: "2001:db8::/48, 10.9.8.7", - denyNets: "198.51.100.0/24", - rateLimitPerMinute: "60", - rateLimitPerHour: "600", - rateLimitPerDay: "6000", - deniedCountries: "cn, RU,kp,Xk", - allowedCountries: "de", - banResponse: "429", - limitBanDuration: "15m", - limitBanRepeatWindow: "2d", - maxBanDuration: "30d", - attackBanDuration: "1d", - maxBans: "100", - banScopeV4Prefix: "24", - stateDir: "/srv/waf-state", - stateWriteDelay: "500ms", - stateCounterInterval: "1h", - metricsToken: token, - metricsTopN: "10", - rulesDir: "/srv/waf-rules", - rulesEnabled: "false", + listenAddr: "127.0.0.1:9000", + mode: "observe", + allowNets: "192.0.2.7", + rateLimitExemptNets: "2001:db8::/48, 10.9.8.7", + denyNets: "198.51.100.0/24", + rateLimitPerMinute: "60", + rateLimitPerHour: "600", + rateLimitPerDay: "6000", + deniedCountries: "cn, RU,kp,Xk", + allowedCountries: "de", + banResponse: "429", + limitBanDuration: "15m", + limitBanRepeatWindow: "2d", + maxBanDuration: "30d", + attackBanDuration: "1d", + maxBans: "100", + banScopeV4Prefix: "24", + stateDir: "/srv/waf-state", + stateWriteDelay: "500ms", + stateCounterInterval: "1h", + metricsToken: token, + metricsTopN: "10", + rulesDir: "/srv/waf-rules", + rulesEnabled: "false", }) wantSettings(t, cfg, config.Config{ - ListenAddr: "127.0.0.1:9000", - Observe: true, - ClientRequestTimeout: 90 * time.Second, - ClientRequestHeaderMaxBytes: 8 << 10, - ClientIdleTimeout: 5 * time.Minute, - ClientResponseTimeout: 7 * 24 * time.Hour, - UpstreamRequestTimeout: 90 * time.Minute, - UpstreamResponseTimeout: 0, - RequestMaxBytes: 512 << 10, - ResponseMaxBytes: 1234, - RateLimitPerMinute: 60, - RateLimitPerHour: 600, - RateLimitPerDay: 6000, - BanResponse: 429, - LimitBanDuration: 15 * time.Minute, - LimitBanRepeatWindow: 48 * time.Hour, - MaxBanDuration: 30 * 24 * time.Hour, - AttackBanDuration: 24 * time.Hour, - MaxBans: 100, - BanScopeV4Prefix: 24, - StateDir: "/srv/waf-state", - StateWriteDelay: 500 * time.Millisecond, - StateCounterInterval: time.Hour, - MetricsToken: token, - MetricsTopN: 10, - RulesDir: "/srv/waf-rules", - RulesEnabled: false, + ListenAddr: "127.0.0.1:9000", + Observe: true, + RateLimitPerMinute: 60, + RateLimitPerHour: 600, + RateLimitPerDay: 6000, + BanResponse: 429, + LimitBanDuration: 15 * time.Minute, + LimitBanRepeatWindow: 48 * time.Hour, + MaxBanDuration: 30 * 24 * time.Hour, + AttackBanDuration: 24 * time.Hour, + MaxBans: 100, + BanScopeV4Prefix: 24, + StateDir: "/srv/waf-state", + StateWriteDelay: 500 * time.Millisecond, + StateCounterInterval: time.Hour, + MetricsToken: token, + MetricsTopN: 10, + RulesDir: "/srv/waf-rules", + RulesEnabled: false, }) - if cfg.UpstreamURL.String() != "https://app.internal:8443/" { - t.Errorf("%s is %s", upstreamURL, cfg.UpstreamURL) - } - - wantNetblocks(t, cfg.TrustedProxies, "192.0.2.1/32", "10.0.0.0/8", "2001:db8::/32") wantNetblocks(t, cfg.AllowNets, "192.0.2.7/32") wantNetblocks(t, cfg.RateLimitExemptNets, "2001:db8::/48", "10.9.8.7/32") wantNetblocks(t, cfg.DenyNets, "198.51.100.0/24") @@ -581,13 +539,13 @@ func TestCoreRuleSetSettings(t *testing.T) { environment{ wafMode: config.WAFModeDetect, wafParanoiaLevel: "4", wafAnomalyThreshold: "10", wafDisabledRules: "942100, 920350", wafExemptPaths: "/api/, /static/", - wafBodyLimit: "128K", + wafBodyLimit: "32K", }, config.Config{ WAFMode: config.WAFModeDetect, WAFParanoiaLevel: 4, WAFAnomalyThreshold: 10, WAFDisabledRules: []int{942100, 920350}, WAFExemptPaths: []string{"/api/", "/static/"}, - WAFBodyLimit: 128 << 10, + WAFBodyLimit: 32 << 10, }, }, { @@ -615,12 +573,12 @@ func TestCoreRuleSetSettings(t *testing.T) { } } -func TestWAFBodyLimitOf1G(t *testing.T) { +func TestWAFBodyLimitOf64K(t *testing.T) { t.Parallel() - cfg := fromEnvironment(t, environment{wafBodyLimit: "1G"}) - if cfg.WAFBodyLimit != 1<<30 { - t.Errorf("1G read as %d", cfg.WAFBodyLimit) + cfg := fromEnvironment(t, environment{wafBodyLimit: "64K"}) + if cfg.WAFBodyLimit != 64<<10 { + t.Errorf("64K read as %d", cfg.WAFBodyLimit) } } @@ -669,10 +627,10 @@ func TestInvalidCoreRuleSetSettingStopsTheStart(t *testing.T) { wafBodyLimit, "128KB", `"128KB" is not a size such as 512K, 100M or 5G, or off`, }, - {wafBodyLimit, "2G", `"2G" is more than 1G, the most Coraza reads`}, + {wafBodyLimit, "65K", `"65K" is more than 64K, the most servewaf gets of a body`}, { - wafBodyLimit, "1073741825", - `"1073741825" is more than 1G, the most Coraza reads`, + wafBodyLimit, "65537", + `"65537" is more than 64K, the most servewaf gets of a body`, }, } { t.Run(tc.name+"="+tc.value, func(t *testing.T) { @@ -1967,45 +1925,15 @@ func TestSizesAndOff(t *testing.T) { t.Parallel() cfg := fromEnvironment(t, environment{ - requestMaxBytes: "3G", - responseMaxBytes: off, - clientRequestTimeout: off, - clientIdleTimeout: off, + bytesLimitPerMinute: "3G", + bytesLimitPerHour: off, + alertCooldown: off, }) - if cfg.RequestMaxBytes != 3<<30 || cfg.ResponseMaxBytes != 0 || - cfg.ClientRequestTimeout != 0 || cfg.ClientIdleTimeout != 0 { - t.Errorf("3G, off, off and off read as %d, %d, %s and %s", - cfg.RequestMaxBytes, cfg.ResponseMaxBytes, cfg.ClientRequestTimeout, - cfg.ClientIdleTimeout) - } -} - -func TestRequestHeaderMaxBytesJustOver4K(t *testing.T) { - t.Parallel() - - cfg := fromEnvironment(t, environment{clientHeaderMaxBytes: "4097"}) - if cfg.ClientRequestHeaderMaxBytes != 4097 { - t.Errorf("4097 read as %d", cfg.ClientRequestHeaderMaxBytes) - } -} - -func TestRequestHeaderMaxBytesRefusalNeverOffersOff(t *testing.T) { - t.Parallel() - - for _, value := range []string{"32KB", "0", "4K", off} { - t.Run(value, func(t *testing.T) { - t.Parallel() - - _, err := config.FromEnvironment( - environment{clientHeaderMaxBytes: value}.lookupEnv) - - want := clientHeaderMaxBytes + `: "` + value + - `" is not a size of more than 4K, such as 32K` - if err == nil || err.Error() != want { - t.Errorf("error %v, want %s", err, want) - } - }) + if cfg.BytesLimitPerMinute != 3<<30 || cfg.BytesLimitPerHour != 0 || + cfg.AlertCooldown != 0 { + t.Errorf("3G, off and off read as %d, %d and %s", + cfg.BytesLimitPerMinute, cfg.BytesLimitPerHour, cfg.AlertCooldown) } } @@ -2034,34 +1962,12 @@ func TestBanResponseCloseIsZero(t *testing.T) { } } -func TestTrustedProxiesSetButEmptyTrustNothing(t *testing.T) { - t.Parallel() - - cfg := fromEnvironment(t, environment{trustedProxies: ""}) - if len(cfg.TrustedProxies) != 0 { - t.Errorf("trusted proxies %v, want none", cfg.TrustedProxies) - } -} - func TestInvalidValueStopsTheStart(t *testing.T) { t.Parallel() wantStartStopped(t, []struct{ name, value string }{ {listenAddr, "8080"}, {listenAddr, ":http"}, {listenAddr, ":65536"}, - {upstreamURL, "127.0.0.1:8081"}, - {upstreamURL, "ftp://127.0.0.1:8081"}, - {upstreamURL, "http://"}, - {upstreamURL, "http://:8081"}, - {upstreamURL, "http://127.0.0.1:0"}, - {upstreamURL, "http://127.0.0.1:99999"}, - {upstreamURL, "http://127.0.0.1:8081/app"}, - {upstreamURL, "http://127.0.0.1:8081/?a=1"}, - {upstreamURL, "http://user:secret@127.0.0.1:8081"}, {mode, "Observe"}, {mode, "block"}, {mode, ""}, - {trustedProxies, "10.0.0.0/33"}, - {trustedProxies, "traefik"}, - {trustedProxies, "10.0.0.0/8,,192.168.0.0/16"}, - {trustedProxies, "fe80::1%eth0"}, {ipv6GroupPrefix, "31"}, {ipv6GroupPrefix, "129"}, {ipv6GroupPrefix, "/64"}, {ipv6GroupPrefix, off}, {ipv6GroupPrefix, ""}, {maxTrackedClients, "0"}, {maxTrackedClients, "-1"}, {maxTrackedClients, off}, @@ -2069,19 +1975,18 @@ func TestInvalidValueStopsTheStart(t *testing.T) { {allowNets, "192.0.2.0/24,monitoring"}, {rateLimitExemptNets, "2001:db8::/129"}, {denyNets, "198.51.100.0/24,"}, - {clientRequestTimeout, "60"}, {clientRequestTimeout, ""}, - {clientIdleTimeout, "0s"}, - {clientIdleTimeout, "2 minutes"}, - {clientResponseTimeout, "1y"}, - {upstreamRequestTimeout, "-1s"}, - {upstreamResponseTimeout, "0s"}, - {upstreamResponseTimeout, "1.5d"}, - {requestMaxBytes, "100MB"}, - {requestMaxBytes, "100m"}, - {requestMaxBytes, "1.5M"}, - {responseMaxBytes, "0"}, - {responseMaxBytes, "-5"}, - {responseMaxBytes, "99999999999G"}, + {alertCooldown, "60"}, {alertCooldown, ""}, + {alertCooldown, "0s"}, + {alertCooldown, "2 minutes"}, + {alertCooldown, "1y"}, + {alertCooldown, "-1s"}, + {alertCooldown, "1.5d"}, + {bytesLimitPerMinute, "100MB"}, + {bytesLimitPerMinute, "100m"}, + {bytesLimitPerMinute, "1.5M"}, + {bytesLimitPerHour, "0"}, + {bytesLimitPerHour, "-5"}, + {bytesLimitPerHour, "99999999999G"}, {rateLimitPerMinute, ""}, {rateLimitPerMinute, "1K"}, {rateLimitPerHour, "0"}, @@ -2246,10 +2151,10 @@ func TestSettingFromFileIsCheckedAsTheSettingItself(t *testing.T) { t.Parallel() _, err := config.FromEnvironment(environment{ - requestMaxBytes + "_FILE": writeFile(t, "lots\n"), + bytesLimitPerMinute + "_FILE": writeFile(t, "lots\n"), }.lookupEnv) - want := requestMaxBytes + `: "lots" is not a size such as 512K, 100M or 5G, or off` + want := bytesLimitPerMinute + `: "lots" is not a size such as 512K, 100M or 5G, or off` if err == nil || err.Error() != want { t.Errorf("error %v, want %s", err, want) } @@ -2354,99 +2259,89 @@ func writeFile(t *testing.T, contents string) string { func TestLogsEachSettingWithItsValue(t *testing.T) { t.Parallel() - cfg := fromEnvironment(t, environment{clientRequestTimeout: "45s"}) + cfg := fromEnvironment(t, environment{rateLimitPerMinute: "45"}) hostname, _ := os.Hostname() want := map[string]string{ - listenAddr: ":8080", - upstreamURL: "http://127.0.0.1:8081", - mode: "enforce", - trustedProxies: "10.0.0.0/8,172.16.0.0/12,192.168.0.0/16", - ipv6GroupPrefix: "64", - maxTrackedClients: "20000", - clientRequestTimeout: "45s", - clientHeaderMaxBytes: "32K", - clientIdleTimeout: "120s", - clientResponseTimeout: "30m", - upstreamRequestTimeout: "60s", - upstreamResponseTimeout: "30m", - requestMaxBytes: "100M", - responseMaxBytes: "1G", - allowNets: "", - rateLimitExemptNets: "", - denyNets: "", - rateLimitPerMinute: "500", - rateLimitPerHour: "20000", - rateLimitPerDay: "300000", - rateLimitExemptPaths: "", - bytesLimitPerMinute: "4G", - bytesLimitPerHour: "20G", - bytesLimitPerDay: "200G", - bytesCount: "both", - lookupSource: defaultLookupSource, - lookupDBPath: "", - lookupTimeout: "800ms", - addLookupHeaders: "false", - deniedCountries: "", - allowedCountries: "", - asnLimitPercent: "", - countryLimitPercent: "", - asnBytesPercent: "", - countryBytesPercent: "", - unknownLimitPercent: "100", - asnLimitPercentURL: "", - blocklistURLs: "", - blocklistRefresh: "24h", - blocklistAction: actionDeny, - dnsblZones: "", - dnsblResolver: "", - abuseIPDBKey: "", - abuseIPDBMinScore: "75", - abuseIPDBDailyBudget: "900", - reputationAction: "limit:25", - reputationCacheTTL: defaultReputationCacheTTL, - reputationTimeout: "2s", - crowdSecURL: "", - crowdSecKey: "", - banResponse: "403", - limitBanDuration: "1h", - limitBanRepeatWindow: "24h", - maxBanDuration: "7d", - attackBanDuration: "7d", - maxBans: "5000", - banScopeV4Prefix: "32", - stateDir: "/var/lib/smallwebwaf", - stateWriteDelay: "10s", - stateCounterInterval: "15m", - adminToken: "", - metricsToken: "", - metricsTopN: "50", - instanceName: hostname, - logRequestHeaders: defaultLogRequestHeaders, - logLevel: "info", - rulesDir: "/etc/smallwebwaf/rules.d", - rulesEnabled: "true", - wafMode: config.WAFModeBlock, - wafParanoiaLevel: "1", - wafAnomalyThreshold: "5", - wafDisabledRules: defaultWAFDisabledRules, - wafExemptPaths: "", - wafBodyLimit: off, - trapPaths: "", - errorBurstThreshold: "30", - logRemoteURL: "", - logRemoteTLSCAFile: "", - logRemoteBuffer: "10000", - logRemoteFacility: "local0", - logRemoteAppName: hostname, - alertWebhookURL: "", - alertWebhookHeaders: "", - alertSlackWebhookURL: "", - alertNtfyURL: "", - alertNtfyToken: "", - alertEvents: defaultAlertEvents, - alertCooldown: defaultAlertCooldown, - alertMaxPerHour: "60", + listenAddr: ":8080", + mode: "enforce", + ipv6GroupPrefix: "64", + maxTrackedClients: "20000", + allowNets: "", + rateLimitExemptNets: "", + denyNets: "", + rateLimitPerMinute: "45", + rateLimitPerHour: "20000", + rateLimitPerDay: "300000", + rateLimitExemptPaths: "", + bytesLimitPerMinute: "4G", + bytesLimitPerHour: "20G", + bytesLimitPerDay: "200G", + bytesCount: "both", + lookupSource: defaultLookupSource, + lookupDBPath: "", + lookupTimeout: "800ms", + addLookupHeaders: "false", + deniedCountries: "", + allowedCountries: "", + asnLimitPercent: "", + countryLimitPercent: "", + asnBytesPercent: "", + countryBytesPercent: "", + unknownLimitPercent: "100", + asnLimitPercentURL: "", + blocklistURLs: "", + blocklistRefresh: "24h", + blocklistAction: actionDeny, + dnsblZones: "", + dnsblResolver: "", + abuseIPDBKey: "", + abuseIPDBMinScore: "75", + abuseIPDBDailyBudget: "900", + reputationAction: "limit:25", + reputationCacheTTL: defaultReputationCacheTTL, + reputationTimeout: "2s", + crowdSecURL: "", + crowdSecKey: "", + banResponse: "403", + limitBanDuration: "1h", + limitBanRepeatWindow: "24h", + maxBanDuration: "7d", + attackBanDuration: "7d", + maxBans: "5000", + banScopeV4Prefix: "32", + stateDir: "/var/lib/smallwebwaf", + stateWriteDelay: "10s", + stateCounterInterval: "15m", + adminToken: "", + metricsToken: "", + metricsTopN: "50", + instanceName: hostname, + logRequestHeaders: defaultLogRequestHeaders, + logLevel: "info", + rulesDir: "/etc/smallwebwaf/rules.d", + rulesEnabled: "true", + wafMode: config.WAFModeBlock, + wafParanoiaLevel: "1", + wafAnomalyThreshold: "5", + wafDisabledRules: defaultWAFDisabledRules, + wafExemptPaths: "", + wafBodyLimit: off, + trapPaths: "", + errorBurstThreshold: "30", + logRemoteURL: "", + logRemoteTLSCAFile: "", + logRemoteBuffer: "10000", + logRemoteFacility: "local0", + logRemoteAppName: hostname, + alertWebhookURL: "", + alertWebhookHeaders: "", + alertSlackWebhookURL: "", + alertNtfyURL: "", + alertNtfyToken: "", + alertEvents: defaultAlertEvents, + alertCooldown: defaultAlertCooldown, + alertMaxPerHour: "60", } maps.Copy(want, loggedAnomalyDefaults()) @@ -2481,14 +2376,6 @@ func wantSettings(t *testing.T, got *config.Config, want config.Config) { if got.ListenAddr != want.ListenAddr || got.Observe != want.Observe || - got.ClientRequestTimeout != want.ClientRequestTimeout || - got.ClientRequestHeaderMaxBytes != want.ClientRequestHeaderMaxBytes || - got.ClientIdleTimeout != want.ClientIdleTimeout || - got.ClientResponseTimeout != want.ClientResponseTimeout || - got.UpstreamRequestTimeout != want.UpstreamRequestTimeout || - got.UpstreamResponseTimeout != want.UpstreamResponseTimeout || - got.RequestMaxBytes != want.RequestMaxBytes || - got.ResponseMaxBytes != want.ResponseMaxBytes || got.RateLimitPerMinute != want.RateLimitPerMinute || got.RateLimitPerHour != want.RateLimitPerHour || got.RateLimitPerDay != want.RateLimitPerDay { diff --git a/internal/config/rproxy.go b/internal/config/rproxy.go index 8e5f189..eb078ec 100644 --- a/internal/config/rproxy.go +++ b/internal/config/rproxy.go @@ -64,15 +64,9 @@ var ( // invalid is an error that names it. Since rproxy never touches disk, its // settings have no _FILE form. func RProxyFromEnvironment(lookupEnv func(string) (string, bool)) (*RProxy, error) { - env := &environment{lookupEnv: func(name string) (string, bool) { - if strings.HasSuffix(name, "_FILE") { - return "", false - } - - return lookupEnv(name) - }} + env := &environment{lookupEnv: withoutFileForms(lookupEnv)} cfg := &RProxy{ - ListenAddr: env.address("SWRP_LISTEN_ADDR", defaultListenAddr), + ListenAddr: env.address("SWRP_LISTEN_ADDR"), UpstreamURL: env.appURL("SWRP_UPSTREAM_URL", defaultUpstreamURL), TrustedProxies: env.netblocks("SWRP_TRUSTED_PROXIES", privateRanges), WAFBackend: env.wafBackend("SWRP_WAF_BACKEND"), @@ -93,6 +87,37 @@ func RProxyFromEnvironment(lookupEnv func(string) (string, bool)) (*RProxy, erro return cfg, nil } +// RProxyListenAddrAndUpstreamURL reads only SWRP_LISTEN_ADDR and +// SWRP_UPSTREAM_URL. The health check in rproxy's container needs no other +// setting, so it reads no other. +func RProxyListenAddrAndUpstreamURL( + lookupEnv func(string) (string, bool), +) (string, *url.URL, error) { + env := &environment{lookupEnv: withoutFileForms(lookupEnv)} + listenAddr := env.address("SWRP_LISTEN_ADDR") + upstreamURL := env.appURL("SWRP_UPSTREAM_URL", defaultUpstreamURL) + + if env.err != nil { + return "", nil, env.err + } + + return listenAddr, upstreamURL, nil +} + +// withoutFileForms returns lookupEnv finding no setting named with _FILE, since +// rproxy never touches disk. +func withoutFileForms( + lookupEnv func(string) (string, bool), +) func(string) (string, bool) { + return func(name string) (string, bool) { + if strings.HasSuffix(name, "_FILE") { + return "", false + } + + return lookupEnv(name) + } +} + // LogValue makes an RProxy log as each setting's name with the value it was // given, or its default. func (c *RProxy) LogValue() slog.Value { diff --git a/internal/config/rproxy_test.go b/internal/config/rproxy_test.go index 8372292..d7e74f4 100644 --- a/internal/config/rproxy_test.go +++ b/internal/config/rproxy_test.go @@ -125,7 +125,19 @@ func TestRProxyInvalidSettingsAreRefused(t *testing.T) { {rproxyDefault, "maybe"}, {rproxyListen, "8080"}, {rproxyUpstream, "app"}, + {rproxyUpstream, "127.0.0.1:8081"}, + {rproxyUpstream, "ftp://127.0.0.1:8081"}, + {rproxyUpstream, "http://"}, + {rproxyUpstream, "http://:8081"}, + {rproxyUpstream, "http://127.0.0.1:0"}, + {rproxyUpstream, "http://127.0.0.1:99999"}, + {rproxyUpstream, "http://127.0.0.1:8081/app"}, + {rproxyUpstream, "http://127.0.0.1:8081/?a=1"}, + {rproxyUpstream, "http://user:secret@127.0.0.1:8081"}, {rproxyTrusted, "not a netblock"}, + {rproxyTrusted, "10.0.0.0/33"}, + {rproxyTrusted, "10.0.0.0/8,,192.168.0.0/16"}, + {rproxyTrusted, "fe80::1%eth0"}, {rproxyMaxRequest, notASize}, {rproxyMaxResponse, notASize}, {rproxyRequestTime, "0"}, @@ -143,6 +155,19 @@ func TestRProxyInvalidSettingsAreRefused(t *testing.T) { } } +func TestRProxyTrustedProxiesSetButEmptyTrustNothing(t *testing.T) { + t.Parallel() + + cfg, err := rproxyConfig(map[string]string{wafBackend: backendURL, rproxyTrusted: ""}) + if err != nil { + t.Fatalf("settings: %v", err) + } + + if len(cfg.TrustedProxies) != 0 { + t.Errorf("trusted proxies %v, want none", cfg.TrustedProxies) + } +} + func TestRProxyNeedsItsBackend(t *testing.T) { t.Parallel() diff --git a/internal/metrics/metrics.go b/internal/metrics/metrics.go index d47dc95..1b14c0a 100644 --- a/internal/metrics/metrics.go +++ b/internal/metrics/metrics.go @@ -28,16 +28,15 @@ type Metrics struct { registry prometheus.Registerer handler http.Handler - inFlight prometheus.Gauge - requests *prometheus.CounterVec - requestBytes *prometheus.CounterVec - responseBytes *prometheus.CounterVec - requestDuration prometheus.Histogram - upstreamDuration prometheus.Histogram - rateLimitHits *prometheus.CounterVec - sizeAndTimeLimitHits *prometheus.CounterVec - offences *prometheus.CounterVec - wafMatches *prometheus.CounterVec + inFlight prometheus.Gauge + requests *prometheus.CounterVec + requestBytes *prometheus.CounterVec + responseBytes *prometheus.CounterVec + requestDuration prometheus.Histogram + upstreamDuration prometheus.Histogram + rateLimitHits *prometheus.CounterVec + offences *prometheus.CounterVec + wafMatches *prometheus.CounterVec // ruleMatches are made by AddRules, and reputationHits by // AddReputation. ruleMatches *prometheus.CounterVec @@ -100,9 +99,6 @@ func New(topN int, instanceName string) *Metrics { "Requests that broke a rate limit, a byte limit or the error burst, by "+ "its window and its kind, requests, bytes or refusals.", []string{"window", "kind"}), - sizeAndTimeLimitHits: counterVec("smallwebwaf_size_and_time_limit_hits_total", - "Requests that passed a size or time limit, by its setting.", - []string{"limit"}), offences: counterVec("smallwebwaf_offences_total", "Offences, by kind.", []string{"kind"}), wafMatches: counterVec("smallwebwaf_waf_matches_total", @@ -143,7 +139,7 @@ func New(topN int, instanceName string) *Metrics { collectors.NewProcessCollector(collectors.ProcessCollectorOpts{}), m.inFlight, m.requests, m.requestBytes, m.responseBytes, m.requestDuration, m.upstreamDuration, - m.rateLimitHits, m.sizeAndTimeLimitHits, m.offences, m.wafMatches, + m.rateLimitHits, m.offences, m.wafMatches, m.countries, m.asns, m.GeoJSRequests, m.GeoJSFailures, m.GeoJSUnanswered, m.stateFileWrites, m.stateFileWriteFailures, @@ -395,15 +391,11 @@ func (m *Metrics) RequestStarted() { m.inFlight.Inc() } -// RequestEnded counts a request that has ended, from its log line. limit -// is the setting whose size or time limit the request passed, "" if none. -// duration is how long the request took, and upstreamDuration how long it -// took from when it was passed to the app, zero if it was not. -func (m *Metrics) RequestEnded( - line *requestlog.Line, limit string, duration, upstreamDuration time.Duration, -) { +// RequestEnded counts a request that has ended without being passed to the app, +// from its log line. duration is how long the request took. +func (m *Metrics) RequestEnded(line *requestlog.Line, duration time.Duration) { m.inFlight.Dec() - m.count(line, limit, duration, upstreamDuration) + m.count(line, duration, 0) } // RequestAllowed counts a question of rproxy's, answered allow, as no longer @@ -412,12 +404,13 @@ func (m *Metrics) RequestAllowed() { m.inFlight.Dec() } -// RequestReported counts a request rproxy passed to the app, as RequestEnded -// does, from what its report says, which line holds. +// RequestReported counts a request rproxy passed to the app, from what its +// report says, which line holds: duration is how long the request took, and +// upstreamDuration how long it took from when it was passed to the app. func (m *Metrics) RequestReported( line *requestlog.Line, duration, upstreamDuration time.Duration, ) { - m.count(line, "", duration, upstreamDuration) + m.count(line, duration, upstreamDuration) } // LimitHit counts a request that broke a rate limit, a byte limit or the @@ -431,11 +424,10 @@ func (m *Metrics) LimitHit(hit ratelimit.Hit) { // them. func (m *Metrics) Offences(r ratelimit.Request) { for kind, committed := range map[string]bool{ - "limit": r.BrokeLimit, - "attack": r.Attack, - "rule_blocked": r.RuleBlocked, - "waf_blocked": r.WAFBlocked, - "token_refused": r.TokenRefused, + "limit": r.BrokeLimit, + "attack": r.Attack, + "rule_blocked": r.RuleBlocked, + "waf_blocked": r.WAFBlocked, } { if committed { m.offences.WithLabelValues(kind).Inc() @@ -486,9 +478,10 @@ func (m *Metrics) StateFileEditSetAside(name string) { m.stateFileEditsSetAside.WithLabelValues(name).Inc() } -// count counts a request that has ended, as RequestEnded describes. +// count counts a request that has ended, as RequestEnded and RequestReported +// describe. func (m *Metrics) count( - line *requestlog.Line, limit string, duration, upstreamDuration time.Duration, + line *requestlog.Line, duration, upstreamDuration time.Duration, ) { class := statusClass(line.Status) m.requests.WithLabelValues(class, line.Action).Inc() @@ -500,10 +493,6 @@ func (m *Metrics) count( m.upstreamDuration.Observe(upstreamDuration.Seconds()) } - if limit != "" { - m.sizeAndTimeLimitHits.WithLabelValues(limit).Inc() - } - if line.Country != "" { m.countries.add(line.Country, line) } diff --git a/internal/proxy/admin.go b/internal/proxy/admin.go index f5f2a9e..1332101 100644 --- a/internal/proxy/admin.go +++ b/internal/proxy/admin.go @@ -9,7 +9,6 @@ import ( "io" "net/http" "net/netip" - "os" "strings" "time" @@ -50,16 +49,13 @@ func (rq *request) answerHealth() { _, _ = io.WriteString(rq.out, "ok\n") } -// answerAdmin answers a request for smallwebwaf itself, under -// /_smallwebwaf/, once it has passed the checks. Each endpoint needs a -// token, sent as Authorization: Bearer : the metrics -// SWWAF_METRICS_TOKEN, the others SWWAF_ADMIN_TOKEN. A request without -// it is refused with 401, which counts toward the error burst. An -// endpoint whose token is unset answers 404, as any other request under -// /_smallwebwaf/ does. +// answerAdmin answers a request for one of servewaf's own endpoints. Each +// needs a token, sent as Authorization: Bearer : the metrics +// SWWAF_METRICS_TOKEN, the others SWWAF_ADMIN_TOKEN. A request without it is +// refused with 401. An endpoint whose token is unset answers 404, as any other +// path does. func (rq *request) answerAdmin() { rq.line.Action = requestlog.ActionAdmin - rq.startClientResponseTimeout() token, answer := rq.endpoint() @@ -67,12 +63,8 @@ func (rq *request) answerAdmin() { case token == "": http.Error(rq.out, http.StatusText(http.StatusNotFound), http.StatusNotFound) case !hasToken(rq.in, token): - rq.tokenRefused = true rq.out.Header().Set("WWW-Authenticate", "Bearer") - rq.answer(refusal{ - status: http.StatusUnauthorized, - action: requestlog.ActionAdmin, - }) + http.Error(rq.out, http.StatusText(http.StatusUnauthorized), http.StatusUnauthorized) default: answer() } @@ -129,26 +121,7 @@ type banToAdd struct { // an admin, from now for the duration the body gives, with its reason, // and answers with that ban. func (rq *request) addBan() { - // The body must arrive within SWWAF_CLIENT_REQUEST_TIMEOUT, as any - // other request's must. - rq.stopReadingBody(rq.clientRequestDeadline()) - toAdd, err := rq.readBanToAdd() - if refused := rq.refused.Load(); refused != nil { - rq.answer(*refused) // the body is over SWWAF_REQUEST_MAX_BYTES - - return - } - - if errors.Is(err, os.ErrDeadlineExceeded) { - rq.answer(refusal{ - status: http.StatusRequestTimeout, - action: requestlog.ActionTimedOut, - limit: "SWWAF_CLIENT_REQUEST_TIMEOUT", - }) - - return - } var ( netblock netip.Prefix @@ -177,12 +150,9 @@ func (rq *request) addBan() { // readBanToAdd reads the body of POST BansPath: a JSON object with // nothing but whitespace after it, in at most banBodyMaxBytes. func (rq *request) readBanToAdd() (banToAdd, error) { - var body io.ReadCloser = http.NoBody - if rq.body != nil { - body = rq.body - } + data, err := io.ReadAll(http.MaxBytesReader(nil, rq.in.Body, banBodyMaxBytes)) + rq.requestBytes = int64(len(data)) - data, err := io.ReadAll(http.MaxBytesReader(nil, body, banBodyMaxBytes)) if err != nil { return banToAdd{}, fmt.Errorf("%w: %w", errNotBanToAdd, err) } @@ -280,9 +250,9 @@ type clientAnswer struct { Bans []state.BanEntry `json:"bans"` } -// showClient answers GET ClientsPath with what smallwebwaf knows of -// the client: its counters, its history, which holds its country as last -// looked up and its offences, and its bans with their notes. +// showClient answers GET ClientsPath with what servewaf knows of the client: +// its counters, its history, which holds its country as last looked up and its +// offences, and its bans with their notes. func (rq *request) showClient() { addr, err := pathAddress(rq.in.URL.Path, ClientsPath) if err != nil { diff --git a/internal/proxy/admin_test.go b/internal/proxy/admin_test.go index 0a88484..480616f 100644 --- a/internal/proxy/admin_test.go +++ b/internal/proxy/admin_test.go @@ -5,7 +5,6 @@ import ( "net/http" "net/netip" "reflect" - "strconv" "strings" "testing" "time" @@ -23,8 +22,6 @@ const ( // how a request carries it. adminSecret = "fedcba9876543210fedcba9876543210" adminBearer = "Bearer " + adminSecret - // adminClient is the client the tests' admin sends its requests from. - adminClient = "192.0.2.10" // banOtherClient is the body of a request to ban otherClient for an // hour. banOtherClient = `{"netblock": "` + otherClient + `", "duration": "1h", ` + @@ -43,8 +40,7 @@ func TestAdminEndpointsAreOffWhileTheTokenIsUnset(t *testing.T) { // An empty token does not match the unset one either. for _, authorization := range []string{adminBearer, bearer, "Bearer ", ""} { for _, e := range adminEndpoints() { - s.adminRequest(adminClient, authorization, e.method, e.path, e.body, - http.StatusNotFound, requestlog.ActionAdmin) + s.adminRequest(authorization, e.method, e.path, e.body, http.StatusNotFound) } } @@ -71,8 +67,8 @@ func TestAdminEndpointsNeedTheAdminToken(t *testing.T) { for _, authorization := range []string{ "", bearer, "Bearer " + strings.ToUpper(adminSecret), "Basic " + adminSecret, } { - got := s.adminRequest(adminClient, authorization, e.method, e.path, e.body, - http.StatusUnauthorized, requestlog.ActionAdmin) + got := s.adminRequest(authorization, e.method, e.path, e.body, + http.StatusUnauthorized) if got.header.Get("WWW-Authenticate") != "Bearer" { t.Errorf("%s %s with %q was answered without WWW-Authenticate: Bearer", e.method, e.path, authorization) @@ -265,64 +261,12 @@ func TestBanToAddThatCannotBeReadIsRefused(t *testing.T) { } } -func TestBanToAddOverTheRequestSizeLimitIsRefused(t *testing.T) { - t.Parallel() - - s, _, server := startWithClock(t, "", map[string]string{ - adminToken: adminSecret, - requestMaxBytes: "16", - }) - - // Sent in a chunk, its length is not announced, so that it is found - // over SWWAF_REQUEST_MAX_BYTES only as it is read. - chunk := `{"netblock": "203.0.113.9", "duration": "1h"}` - s.adminRequest(adminClient, adminBearer+"\r\nTransfer-Encoding: chunked", - http.MethodPost, proxy.BansPath, - strconv.FormatInt(int64(len(chunk)), 16)+"\r\n"+chunk+"\r\n0\r\n\r\n", - http.StatusRequestEntityTooLarge, requestlog.ActionTooLarge) - - if held := server.Ledger.Snapshot(); len(held) != 0 { - t.Errorf("the ledger holds %+v, want no ban", held) - } -} - -func TestBanToAddSlowerThanTheClientRequestTimeoutIsRefused(t *testing.T) { - t.Parallel() - - s, _, server := startWithClock(t, "", map[string]string{ - adminToken: adminSecret, - metricsToken: token, - clientRequestTimeout: shortTimeoutSetting, - }) - - // The chunk announces 256 bytes and the rest of it never comes, so only - // the timeout ends the wait. A hold-up of the test process can only - // make the answer later, so the time is checked only for not being - // shorter than the timeout. - start := time.Now() - - s.adminRequest(adminClient, adminBearer+"\r\nTransfer-Encoding: chunked", - http.MethodPost, proxy.BansPath, "100\r\n"+`{"netblock": "203.0.113.9", `, - http.StatusRequestTimeout, requestlog.ActionTimedOut) - - if took := time.Since(start); took < shortTimeout { - t.Errorf("answered after %s, before the timeout of %s ran out", took, shortTimeout) - } - - wantLimitHits(t, s.addr, clientRequestTimeout, 1) - - if held := server.Ledger.Snapshot(); len(held) != 0 { - t.Errorf("the ledger holds %+v, want no ban", held) - } -} - func TestClientEndpointShowsTheClientAndItsBans(t *testing.T) { t.Parallel() s, clk, _ := startWithClock(t, "", map[string]string{ - adminToken: adminSecret, - rateLimitPerMinute: "2", - rateLimitExemptNets: adminClient, + adminToken: adminSecret, + rateLimitPerMinute: "2", }) start := clk.Now() @@ -371,61 +315,6 @@ func TestClientEndpointShowsTheClientAndItsBans(t *testing.T) { http.StatusBadRequest) } -func TestBannedClientIsRefusedAtTheEndpointsEvenWithTheToken(t *testing.T) { - t.Parallel() - - s, _, _ := startWithClock(t, "", map[string]string{adminToken: adminSecret}) - - s.admin(http.MethodPost, proxy.BansPath, banOtherClient, http.StatusOK) - - // otherClient cannot lift its own ban either. - for _, e := range adminEndpoints() { - s.adminRequest(otherClient, adminBearer, e.method, e.path, e.body, - http.StatusForbidden, requestlog.ActionBanned) - } -} - -func TestAdminRequestsCountTowardTheLimits(t *testing.T) { - t.Parallel() - - s, _, _ := startWithClock(t, "", map[string]string{ - adminToken: adminSecret, - rateLimitPerMinute: "2", - }) - - // A request refused for a missing token and one answered count toward - // the limit of two a minute, so the next breaks it. - s.adminRequest(client, "", http.MethodGet, proxy.BansPath, "", - http.StatusUnauthorized, requestlog.ActionAdmin) - s.adminRequest(client, adminBearer, http.MethodGet, proxy.BansPath, "", - http.StatusOK, requestlog.ActionAdmin) - s.adminRequest(client, adminBearer, http.MethodGet, proxy.BansPath, "", - http.StatusForbidden, requestlog.ActionRateLimited) -} - -func TestClientInAllowNetsSkipsTheChecksButNeedsTheToken(t *testing.T) { - t.Parallel() - - const allowed = "192.0.2.60" // in SWWAF_ALLOW_NETS - - s, clk, server := startWithClock(t, "", map[string]string{ - adminToken: adminSecret, - allowNets: allowed, - rateLimitPerMinute: "1", - }) - - // A ban on it refuses nothing, and its requests are not counted. - server.Ledger.BanForAdmin(netip.MustParsePrefix(allowed+"/32"), clk.Now(), - time.Time{}, "") - - for range 2 { - s.adminRequest(allowed, "", http.MethodGet, proxy.BansPath, "", - http.StatusUnauthorized, requestlog.ActionAdmin) - s.adminRequest(allowed, adminBearer, http.MethodGet, proxy.BansPath, "", - http.StatusOK, requestlog.ActionAdmin) - } -} - func TestAdminEndpointsNeedTheTokenInObserveMode(t *testing.T) { t.Parallel() @@ -435,8 +324,7 @@ func TestAdminEndpointsNeedTheTokenInObserveMode(t *testing.T) { }) for _, e := range adminEndpoints() { - s.adminRequest(adminClient, "", e.method, e.path, e.body, - http.StatusUnauthorized, requestlog.ActionAdmin) + s.adminRequest("", e.method, e.path, e.body, http.StatusUnauthorized) } if held := server.Ledger.Snapshot(); len(held) != 0 { @@ -461,39 +349,33 @@ func adminEndpoints() []adminEndpoint { } } -// admin sends a request with method for path, with body, from -// adminClient, with the admin token, and checks that it is answered with -// status, its log line's action admin. It returns the answer. +// admin sends a request with method for path, with body, to servewaf's own +// address, with the admin token, and checks that it is answered with status, its +// log line's action admin. It returns the answer. func (s *sender) admin(method, path, body string, status int) answer { s.t.Helper() - return s.adminRequest(adminClient, adminBearer, method, path, body, status, - requestlog.ActionAdmin) + return s.adminRequest(adminBearer, method, path, body, status) } -// adminRequest sends a request with method for path, with body, from the -// client at from, with authorization as its Authorization header unless -// it is "", and checks its answer's status and its log line's action, as -// request does. authorization may end in more header lines. A body that -// is not "" has its length announced, unless authorization names -// Transfer-Encoding. It returns the answer. +// adminRequest is admin with authorization as the request's Authorization header +// unless it is "". func (s *sender) adminRequest( - from, authorization, method, path, body string, status int, action string, + authorization, method, path, body string, status int, ) answer { s.t.Helper() - var header []string - + req := newRequest(s.t, method, s.serveWAFAddr, path, strings.NewReader(body)) if authorization != "" { - header = append(header, "Authorization: "+authorization) + req.Header.Set("Authorization", authorization) } - if body != "" && !strings.Contains(authorization, "Transfer-Encoding") { - header = append(header, "Content-Length: "+strconv.Itoa(len(body))) - } + got := do(s.t, req) + wantStatus(s.t, got, status) - _, got := s.requestWithBody(method, from, path, strings.Join(header, "\r\n"), - body, status, action) + line := s.out.endedLines(s.t, s.sent+1)[s.sent] + s.sent++ + wantLine(s.t, line, status, requestlog.ActionAdmin) return got } diff --git a/internal/proxy/alerts_test.go b/internal/proxy/alerts_test.go index 3caa81b..ed83044 100644 --- a/internal/proxy/alerts_test.go +++ b/internal/proxy/alerts_test.go @@ -224,7 +224,9 @@ func startAppWithAlerts( addr, out, server, queue := startProxyWithAlerts(t, app.URL, "", clk.Now, settings) - return &sender{t: t, addr: addr, out: out}, clk, server, queue + s := &sender{t: t, addr: addr, serveWAFAddr: server.Addr, out: out} + + return s, clk, server, queue } // banAlert returns the alert for event, raised by a request from client at @@ -237,6 +239,7 @@ func banAlert( Instance: alertInstance, Time: raised, Event: event, + Host: appHost, Client: netip.MustParseAddr(client), Netblock: ban.Netblock, Reason: ban.Reason, diff --git a/internal/proxy/anomaly_test.go b/internal/proxy/anomaly_test.go index f455b1f..36adc3f 100644 --- a/internal/proxy/anomaly_test.go +++ b/internal/proxy/anomaly_test.go @@ -117,6 +117,7 @@ func TestEachScopeAndWindowOverItsThresholdAlertsOncePerCooldown(t *testing.T) { Instance: alertInstance, Time: start, Event: alerts.EventAnomaly, + Host: appHost, Client: netip.MustParseAddr(client), Netblock: scope.netblock, ASN: asnDE, diff --git a/internal/proxy/answertime_test.go b/internal/proxy/answertime_test.go index 8520eae..cea0ea5 100644 --- a/internal/proxy/answertime_test.go +++ b/internal/proxy/answertime_test.go @@ -315,20 +315,6 @@ func TestObserveModeWaitsForGeoJSForARequestOverTheHeaderLimits(t *testing.T) { } } -func TestWithoutACommandTheHeaderLimitsDoNotApply(t *testing.T) { - t.Parallel() - - app := startApp(t, func(http.ResponseWriter, *http.Request) {}) - addr, out := startProxy(t, app.URL, nil) - - req := newRequest(t, http.MethodGet, addr, "/", http.NoBody) - req.Header.Set(cookieHeader, cookies(cookieLimit+1)) - fillHeaders(req, headerFieldLimit+1) - - wantStatus(t, do(t, req), http.StatusOK) - wantLine(t, out.requestLine(t), http.StatusOK, requestlog.ActionForward) -} - // newServeWAFInBubble makes a servewaf for a synctest bubble: GeoJS never // answers, a setting needs its answer, and everything is at its default but // for env. No request to it may use the network. @@ -338,8 +324,7 @@ func newServeWAFInBubble(t *testing.T, env map[string]string) (*proxy.Server, *o settings := map[string]string{deniedCountries: "KP"} maps.Copy(settings, env) - server, out, _ := newServerWith(t, proxy.NewServeWAF, "http://app.invalid", - unansweredGeoJSURL, time.Now, settings) + server, out, _ := newServeWAF(t, unansweredGeoJSURL, time.Now, settings) return server, out } @@ -521,27 +506,6 @@ func TestAQuestionWaitsForASlowGeoJSOnlyAsLongAsTheDefaultSays(t *testing.T) { } } -func TestWithoutACommandARefusedRequestStillWaitsForGeoJS(t *testing.T) { - t.Parallel() - - synctest.Test(t, func(t *testing.T) { - server, out, _ := newProxy(t, "http://app.invalid", unansweredGeoJSURL, - time.Now, map[string]string{deniedCountries: "KP", blocklistURLs: dropURL}) - listClient(t, server, time.Now()) - - status, took := askDirectly(t, server, func(req *http.Request) { - req.RemoteAddr = net.JoinHostPort(client, "1234") - }) - - if status != http.StatusForbidden || took != defaultLookupWait { - t.Errorf("answered %d after %s, want %d after %s; output:\n%s", - status, took, http.StatusForbidden, defaultLookupWait, out.text()) - } - - wantLine(t, out.requestLine(t), http.StatusForbidden, requestlog.ActionDenied) - }) -} - func TestServeWAFAnswersWithinTheTimeLimitWhileEveryOutsideSourceHangs(t *testing.T) { t.Parallel() @@ -555,11 +519,11 @@ func TestServeWAFAnswersWithinTheTimeLimitWhileEveryOutsideSourceHangs(t *testin t.Cleanup(func() { _ = resolver.Close() }) - server, out, _ := newServerWith(t, func(params proxy.Params) *proxy.Server { + server, out, _ := newServeWAFWith(t, func(params proxy.Params) *proxy.Server { params.AbuseIPDBURL = silentAbuseIPDBURL return proxy.NewServeWAF(params) - }, "http://app.invalid", held, time.Now, map[string]string{ + }, held, time.Now, map[string]string{ deniedCountries: "KP", wafMode: block, dnsblZones: dnsblZone, dnsblResolver: resolver.LocalAddr().String(), abuseIPDBKey: accountKey, diff --git a/internal/proxy/bans.go b/internal/proxy/bans.go index 10f7fc2..6b64168 100644 --- a/internal/proxy/bans.go +++ b/internal/proxy/bans.go @@ -58,13 +58,12 @@ func (rq *request) limitBroken(now time.Time) bool { } // countBytes counts the request's bytes, as countedBytes gives them, for -// the byte limits, once its response has ended, and notes the client's -// byte totals for the log line; its requests stay there as the rate limits -// counted them. Only a request passed to the app has them counted, and -// only one the rate limits counted; in observe mode, not one that enforce -// mode would have refused. Bytes that take the client over a byte limit, -// as its limit percentage for the byte limits lowers it, break it; the -// response was passed on whole. +// the byte limits, once its report has come, and notes the client's byte totals +// for the log line; its requests stay there as the rate limits counted them. +// Only an allowed request has them counted, and only one the rate limits +// counted; in observe mode, not one that enforce mode would have refused. Bytes +// that take the client over a byte limit, as its limit percentage for the byte +// limits lowers it, break it; the response was passed on whole. func (rq *request) countBytes() { if !rq.counted || rq.line.WouldAction != "" { return @@ -84,9 +83,8 @@ func (rq *request) countBytes() { } // countRefusal counts the request for the error burst once it has been -// answered, if smallwebwaf refused it after a rule file match, a trap path -// or a Core Rule Set match, or for a missing or wrong token, and in -// observe mode if enforce mode would have: more than +// answered, if servewaf refused it after a rule file match, a trap path or a +// Core Rule Set match, and in observe mode if enforce mode would have: more than // SWWAF_ERROR_BURST_THRESHOLD such refusals of the client within a minute // break a limit. A client in SWWAF_ALLOW_NETS, which the checks skip, is // not counted, and nothing is while the threshold is off. @@ -96,11 +94,7 @@ func (rq *request) countRefusal() { return } - // In observe mode, a request that enforce mode would have refused - // before it reached the endpoint has had no token refused there. - tokenRefused := rq.tokenRefused && rq.line.WouldAction == "" && - !isInside(rq.client, cfg.AllowNets) - if !rq.attack && !rq.ruleBlocked && !rq.wafBlocked && !tokenRefused { + if !rq.attack && !rq.ruleBlocked && !rq.wafBlocked { return } @@ -127,16 +121,9 @@ func (rq *request) countRefusal() { // countedBytes returns the request's bytes, once it has ended, as the // byte limits and the anomaly thresholds count them: the response's body -// bytes, the request's, or both, as SWWAF_BYTES_COUNT says. For an -// upgraded connection, such as a WebSocket, which has closed by then, what -// it carried from the app counts with the response's and what it carried -// from the client with the request's. +// bytes, the request's, or both, as SWWAF_BYTES_COUNT says. func (rq *request) countedBytes() int64 { - response, request := rq.out.bytes, rq.requestBytes() - if rq.upgraded != nil { - response += rq.upgraded.fromApp.Load() - request += rq.upgraded.toApp.Load() - } + response, request := rq.out.bytes, rq.requestBytes switch rq.h.config.BytesCount { case "response": @@ -151,7 +138,7 @@ func (rq *request) countedBytes() int64 { // banForLimit bans the client's netblock at now for a broken limit, the // one hit names, notes the offence for the log line and counts the hit in // the metrics. status is what the client was sent, or is sent: -// SWWAF_BAN_RESPONSE for a request over a rate limit, the app's answer for +// SWWAF_BAN_RESPONSE for a request over a rate limit, what the report says for // one whose bytes broke a byte limit, the refusal for one that broke the // error burst. The ban's notes give the client's limit percentage for a // rate limit or a byte limit; the error burst is not lowered. The ban sets diff --git a/internal/proxy/bans_test.go b/internal/proxy/bans_test.go index aae674d..164faa9 100644 --- a/internal/proxy/bans_test.go +++ b/internal/proxy/bans_test.go @@ -9,6 +9,7 @@ import ( "net/netip" "reflect" "slices" + "strings" "sync" "testing" "time" @@ -374,7 +375,7 @@ func (c *clock) advance(d time.Duration) { c.now = c.now.Add(d) } -// startWithClock starts smallwebwaf in front of an app that answers 200, +// startWithClock starts servewaf, and rproxy in front of an app that answers 200, // with the settings in env on top of trusting localhost's // X-Forwarded-For, clients' AS numbers and countries looked up at // geojsURL, and a clock set to midnight, the start of a bucket in every @@ -391,18 +392,19 @@ func startWithClock( addr, out, server := startProxyWithClock(t, app.URL, geojsURL, clk.Now, settings) - return &sender{t: t, addr: addr, out: out}, clk, server + return &sender{t: t, addr: addr, serveWAFAddr: server.Addr, out: out}, clk, server } -// sender sends requests to smallwebwaf one after another, each on a -// connection of its own, and checks each one's answer and log line. They -// must be the only requests smallwebwaf is sent, since the log lines are -// matched to them in order. +// sender sends requests to rproxy one after another, each on a connection of its +// own, and checks each one's answer and log line. serveWAFAddr is where servewaf's +// own endpoints are, which adminRequest asks. Every request servewaf gets must +// come through the sender, since the log lines are matched to them in order. type sender struct { - t *testing.T - addr string - out *output - sent int + t *testing.T + addr string + serveWAFAddr string + out *output + sent int } // get sends a GET request for / from the client at from. @@ -419,22 +421,19 @@ func (s *sender) get(from string, status int, action string) logLine { func (s *sender) request(from, path string, status int, action string) logLine { s.t.Helper() - line, _ := s.requestWithHeader(from, path, "", status, action) - - return line + return s.requestWithHeader(from, path, "", status, action) } // requestWithHeader is request with header, such as "Authorization: -// Bearer x", added to the request unless it is "". It returns the body of -// the answer too. +// Bearer x", added to the request unless it is "". func (s *sender) requestWithHeader( from, path, header string, status int, action string, -) (logLine, string) { +) logLine { s.t.Helper() - line, got := s.requestWithBody(http.MethodGet, from, path, header, "", status, action) + line, _ := s.requestWithBody(http.MethodGet, from, path, header, "", status, action) - return line, string(got.body) + return line } // requestWithBody is requestWithHeader for a request with method, whose @@ -450,10 +449,14 @@ func (s *sender) requestWithBody( header += "\r\n" } + // rproxy refuses a request with more than one. + if !strings.Contains(header, "User-Agent:") { + header += "User-Agent: " + userAgent + "\r\n" + } + conn := dial(s.t, s.addr) - send(s.t, conn, method+" "+path+" HTTP/1.1\r\nHost: "+appHost+ - "\r\nUser-Agent: "+userAgent+"\r\n"+forwardedFor+": "+from+"\r\n"+ - header+"\r\n"+body) + send(s.t, conn, method+" "+path+" HTTP/1.1\r\nHost: "+appHost+"\r\n"+ + forwardedFor+": "+from+"\r\n"+header+"\r\n"+body) err := conn.SetReadDeadline(time.Now().Add(waitLimit)) if err != nil { @@ -478,7 +481,7 @@ func (s *sender) requestWithBody( got.status, status) } - line := s.out.requestLines(s.t, s.sent+1)[s.sent] + line := s.out.endedLines(s.t, s.sent+1)[s.sent] s.sent++ wantLine(s.t, line, status, action) diff --git a/internal/proxy/biased_test.go b/internal/proxy/biased_test.go index cf1e5ab..821b1d2 100644 --- a/internal/proxy/biased_test.go +++ b/internal/proxy/biased_test.go @@ -4,7 +4,6 @@ import ( "fmt" "io" "maps" - "net" "net/http" "net/http/httptest" "net/netip" @@ -281,12 +280,12 @@ func TestClientWithoutAnAnswerInTimeHasTheUnknownLimitPercent(t *testing.T) { // In a synctest bubble, as TestRequestWaitsAsLongAsTheLookupTimeoutSays // says, with a GeoJS that never answers. synctest.Test(t, func(t *testing.T) { - server, out, _ := newProxy(t, "http://app.invalid", unansweredGeoJSURL, - time.Now, map[string]string{unknownLimitPercent: "0"}) + server, out, _ := newServeWAF(t, unansweredGeoJSURL, time.Now, + map[string]string{unknownLimitPercent: "0"}) // Once the time the request waits for its answer is up, the client // counts as without a country, and its zero allowance refuses the - // request before it reaches the app. + // request. serveFromDE(t, server, http.MethodGet, http.NoBody) line := out.requestLine(t) @@ -320,13 +319,8 @@ func TestRequestWaitsForItsLookupWhileABiasedThresholdIsSet(t *testing.T) { // In a synctest bubble, as TestRequestWaitsAsLongAsTheLookupTimeoutSays // says, with a GeoJS that never answers. synctest.Test(t, func(t *testing.T) { - // The request's body is over SWWAF_REQUEST_MAX_BYTES, so that it - // is refused after the checks, and never reaches the app. - server, out, _ := newProxy(t, "http://app.invalid", unansweredGeoJSURL, - time.Now, map[string]string{ - lookupTimeout: timeout.String(), requestMaxBytes: "1", - tc.setting: tc.value, - }) + server, out, _ := newServeWAF(t, unansweredGeoJSURL, time.Now, + map[string]string{lookupTimeout: timeout.String(), tc.setting: tc.value}) began := time.Now() serveFromDE(t, server, http.MethodPost, strings.NewReader("ab")) @@ -340,8 +334,7 @@ func TestRequestWaitsForItsLookupWhileABiasedThresholdIsSet(t *testing.T) { t.Errorf("the request waited %s for its answer, want %s", waited, want) } - wantLine(t, out.requestLine(t), http.StatusRequestEntityTooLarge, - requestlog.ActionTooLarge) + wantLine(t, out.requestLine(t), http.StatusOK, requestlog.ActionForward) // The bubble's clock stops once this function returns, so the // request to GeoJS, which a request that did not wait leaves @@ -467,13 +460,15 @@ func (s *sender) uploadFrom(from string) logLine { return line } -// serveFromDE hands a request from fromDE with method and body straight to -// server's handler, without the network, and returns once it is answered. +// serveFromDE hands a question about a request from fromDE with method and body +// straight to server's handler, without the network, and returns once it is +// answered. func serveFromDE(t *testing.T, server *proxy.Server, method string, body io.Reader) { t.Helper() req := httptest.NewRequestWithContext(t.Context(), method, "/", body) - req.RemoteAddr = net.JoinHostPort(fromDE, "1234") + req.Header.Set(proxy.ClientHeader, fromDE) + req.Header.Set(requestIDHeader, "the-id") server.Handler.ServeHTTP(httptest.NewRecorder(), req) } diff --git a/internal/proxy/bodies.go b/internal/proxy/bodies.go deleted file mode 100644 index 3180b6b..0000000 --- a/internal/proxy/bodies.go +++ /dev/null @@ -1,239 +0,0 @@ -package proxy - -import ( - "bufio" - "errors" - "io" - "net" - "net/http" - "sync/atomic" - - "sneak.berlin/go/smallwebwaf/internal/requestlog" -) - -// errResponseTooLarge ends an app's response body that is longer than -// SWWAF_RESPONSE_MAX_BYTES. -var errResponseTooLarge = errors.New( - "the response body is over SWWAF_RESPONSE_MAX_BYTES") - -// requestBody is the client's request body on its way to the app. The -// transport reads it on a goroutine of its own. -type requestBody struct { - // body is the client's body, ending in an *http.MaxBytesError past - // SWWAF_REQUEST_MAX_BYTES. - body io.ReadCloser - rq *request - // readByCoreRuleSet is what the Core Rule Set read of the body before - // the request went to the app, and Read gives first. - readByCoreRuleSet []byte - // waiting is true while a Read waits for the client to send more. - waiting atomic.Bool - // received is true once the client has sent the whole body. - received atomic.Bool - // bytes is how much of the body has been read. - bytes atomic.Int64 -} - -// Read reads from the client's body, after what the Core Rule Set read of -// it, which has been counted already. -func (b *requestBody) Read(p []byte) (int, error) { - if len(b.readByCoreRuleSet) > 0 { - n := copy(p, b.readByCoreRuleSet) - b.readByCoreRuleSet = b.readByCoreRuleSet[n:] - - return n, nil - } - - b.waiting.Store(true) - n, err := b.body.Read(p) - b.waiting.Store(false) - b.bytes.Add(int64(n)) - - var tooLarge *http.MaxBytesError - - switch { - case errors.Is(err, io.EOF): - b.received.Store(true) - b.rq.bodyReceived() - case errors.As(err, &tooLarge): - b.rq.refuse(refusal{ - status: http.StatusRequestEntityTooLarge, - action: requestlog.ActionTooLarge, - limit: "SWWAF_REQUEST_MAX_BYTES", - }) - } - - return n, err -} - -// Close closes the client's body. -func (b *requestBody) Close() error { - return b.body.Close() -} - -// responseBody is the app's response body on its way to the client. -type responseBody struct { - // body is the app's body, ending in an *http.MaxBytesError past - // SWWAF_RESPONSE_MAX_BYTES. - body io.ReadCloser - rq *request -} - -// Read reads from the app's body. -func (b *responseBody) Read(p []byte) (int, error) { - n, err := b.body.Read(p) - if err == nil { - return n, nil - } - - var tooLarge *http.MaxBytesError - - switch { - case errors.Is(err, io.EOF): - b.rq.responseReceived() - case errors.As(err, &tooLarge): - b.rq.refuse(refusal{ - status: http.StatusBadGateway, - action: requestlog.ActionTooLarge, - limit: "SWWAF_RESPONSE_MAX_BYTES", - }) - - return n, errResponseTooLarge - case b.rq.in.Context().Err() == nil: - // The answer broke off, not because the client went away. If a - // timeout cut it, that refusal came first and is the one kept. - b.rq.refuse(refusal{ - status: http.StatusBadGateway, - action: requestlog.ActionUpstreamError, - }) - } - - return n, err -} - -// Close closes the app's body. -func (b *responseBody) Close() error { - return b.body.Close() -} - -// upgradedConn is the connection to the app once the app has switched -// protocols, as for a WebSocket. ReverseProxy writes to it what the client -// sends and reads from it what the app sends, on goroutines of its own, -// until the connection closes; it counts the bytes each way, for the byte -// limits. -type upgradedConn struct { - io.ReadWriteCloser - - // fromApp is how many bytes the app has sent, and toApp how many the - // client has. - fromApp atomic.Int64 - toApp atomic.Int64 -} - -// Read reads what the app sends. -func (c *upgradedConn) Read(p []byte) (int, error) { - n, err := c.ReadWriteCloser.Read(p) - c.fromApp.Add(int64(n)) - - return n, err -} - -// Write sends the app what the client sent. -func (c *upgradedConn) Write(p []byte) (int, error) { - n, err := c.ReadWriteCloser.Write(p) - c.toApp.Add(int64(n)) - - return n, err -} - -// CloseWrite tells the app that the client sends no more, while what the -// app sends still passes. ReverseProxy calls it once the client has -// stopped sending, and closes the connection there if it is not supported. -func (c *upgradedConn) CloseWrite() error { - conn, ok := c.ReadWriteCloser.(interface{ CloseWrite() error }) - if !ok { - return http.ErrNotSupported - } - - return conn.CloseWrite() -} - -// limitBody returns body, cut off with an *http.MaxBytesError after -// maxBytes, or unchanged if maxBytes is zero, which is off. -func limitBody(body io.ReadCloser, maxBytes int64) io.ReadCloser { - if maxBytes == 0 { - return body - } - - // Without a ResponseWriter, MaxBytesReader only counts and cuts off. - return http.MaxBytesReader(nil, body, maxBytes) -} - -// responseWriter is the response to the client. It notes the status and -// size for the log line, and the first error writing to the client. -type responseWriter struct { - http.ResponseWriter - - // status is the final status sent, or zero before one is. - status int - bytes int64 - err error -} - -// WriteHeader sends the status and headers. An informational 1xx status -// is passed on and the final status still comes later. -func (w *responseWriter) WriteHeader(status int) { - if status >= http.StatusOK && w.status == 0 { - w.status = status - } - - w.ResponseWriter.WriteHeader(status) -} - -// Write sends part of the body. -func (w *responseWriter) Write(p []byte) (int, error) { - if w.status == 0 { - w.status = http.StatusOK - } - - n, err := w.ResponseWriter.Write(p) - w.bytes += int64(n) - w.noteError(err) - - return n, err -} - -// FlushError sends what has been written so far. -// http.ResponseController calls it, as ReverseProxy does after each -// write. -func (w *responseWriter) FlushError() error { - err := http.NewResponseController(w.ResponseWriter).Flush() - w.noteError(err) - - return err -} - -// Hijack takes over the connection to the client, which ReverseProxy does -// only to pass on the app's 101 Switching Protocols: that is then the -// status sent. -func (w *responseWriter) Hijack() (net.Conn, *bufio.ReadWriter, error) { - conn, buffered, err := http.NewResponseController(w.ResponseWriter).Hijack() - if err == nil { - w.status = http.StatusSwitchingProtocols - } - - return conn, buffered, err -} - -// Unwrap lets http.ResponseController reach net/http's own -// ResponseWriter for what responseWriter does not do itself. -func (w *responseWriter) Unwrap() http.ResponseWriter { - return w.ResponseWriter -} - -// noteError keeps the first error writing to the client. -func (w *responseWriter) noteError(err error) { - if w.err == nil { - w.err = err - } -} diff --git a/internal/proxy/bytelimits_test.go b/internal/proxy/bytelimits_test.go index a8f0077..327f233 100644 --- a/internal/proxy/bytelimits_test.go +++ b/internal/proxy/bytelimits_test.go @@ -47,8 +47,6 @@ const ( func TestEachByteLimitBansOnceTheResponseHasEnded(t *testing.T) { t.Parallel() - const scraper = "192.0.2.200" - for _, tc := range []struct { setting, window string // apart is the time between the two requests, which the window @@ -94,7 +92,7 @@ func TestEachByteLimitBansOnceTheResponseHasEnded(t *testing.T) { s.get(client, http.StatusForbidden, requestlog.ActionBanned) - wantMetric(t, s.scrape(scraper), `smallwebwaf_rate_limit_hits_total{`+ + wantMetric(t, s.scrape(), `smallwebwaf_rate_limit_hits_total{`+ `instance="`+alertInstance+`",kind="bytes",window="`+tc.window+`"}`, 1) }) } @@ -126,7 +124,7 @@ func TestBytesOfAnAnswerThatBreaksOffAreCounted(t *testing.T) { // The 70 bytes passed on before the app broke off are over the limit of // 50, and ban the client for an hour. line, got := s.requestWithBody(http.MethodGet, client, "/", "", "", http.StatusOK, - requestlog.ActionUpstreamError) + requestlog.ActionForward) if len(got.body) != answerBytes || line.LimitHit != minuteBytes || line.BanExpires != expires { t.Errorf("got %d bytes, and the log line has limit_hit %q and ban_expires %q, "+ @@ -168,39 +166,6 @@ func TestWebSocketBytesAreCountedOnceItCloses(t *testing.T) { } } -func TestWebSocketPassesTheAnswerAfterTheClientStopsSending(t *testing.T) { - t.Parallel() - - app := startApp(t, echoOnceTheClientStops) - addr, out := startProxy(t, app.URL, - map[string]string{trustedProxies: trustLocalhost}) - s := &sender{t: t, addr: addr, out: out} - - conn, reader := s.openWebSocket() - send(t, conn, uploadBody) - - // The client closes its sending side and waits for the answer, which the - // app sends only once it has seen the client stop. smallwebwaf passes the - // close on to the app through CloseWrite on upgradedConn; without that, - // it closes both connections, and the answer is lost. - tcp, ok := conn.(*net.TCPConn) - if !ok { - t.Fatalf("connection is a %T, want a *net.TCPConn", conn) - } - - err := tcp.CloseWrite() - if err != nil { - t.Fatalf("close the sending side: %v", err) - } - - got, err := io.ReadAll(reader) - if err != nil || string(got) != uploadBody { - t.Errorf("got %q (%v), want %q", got, err, uploadBody) - } - - s.closeWebSocket(conn) -} - func TestBytesCountSaysWhichBytesCount(t *testing.T) { t.Parallel() @@ -465,28 +430,6 @@ func answerAfterUpgrade(w http.ResponseWriter, _ *http.Request) { } } -// echoOnceTheClientStops is an app that switches protocols, as for a -// WebSocket, reads what the client sends until the client stops sending, -// and then sends it all back. -func echoOnceTheClientStops(w http.ResponseWriter, _ *http.Request) { - conn, buffered, err := http.NewResponseController(w).Hijack() - if err != nil { - return - } - - defer func() { - _ = conn.Close() - }() - - _, _ = buffered.WriteString("HTTP/1.1 101 Switching Protocols\r\n" + - "Connection: Upgrade\r\nUpgrade: websocket\r\n\r\n") - _ = buffered.Flush() - - received, _ := io.ReadAll(buffered) - _, _ = buffered.Write(received) - _ = buffered.Flush() -} - // webSocket opens a WebSocket from client to answerAfterUpgrade, sends a // line of bodyBytes on it, reads the answer, and closes it. It checks the // answer, and the log line as request does, and returns the log line. @@ -543,7 +486,7 @@ func (s *sender) closeWebSocket(conn net.Conn) logLine { _ = conn.Close() - line := s.out.requestLines(s.t, s.sent+1)[s.sent] + line := s.out.endedLines(s.t, s.sent+1)[s.sent] s.sent++ wantLine(s.t, line, http.StatusSwitchingProtocols, requestlog.ActionForward) diff --git a/internal/proxy/client.go b/internal/proxy/client.go index e725973..8a1da45 100644 --- a/internal/proxy/client.go +++ b/internal/proxy/client.go @@ -5,10 +5,9 @@ import ( "net/http" "net/netip" "slices" - "strings" ) -// peerAddress is the address of the request's TCP peer, normally traefik. +// peerAddress is the address of the request's TCP peer. func peerAddress(r *http.Request) netip.Addr { addrPort, err := netip.ParseAddrPort(r.RemoteAddr) if err != nil { @@ -18,58 +17,24 @@ func peerAddress(r *http.Request) netip.Addr { return addrPort.Addr().Unmap() } -// clientAddress works out who the client is. A peer outside the trusted -// proxies is the client, and what it says in X-Forwarded-For is ignored. -// For a peer inside them, X-Forwarded-For is read from the right, and the -// first address outside them is the client; if every address in it is -// inside, the leftmost is, and with no header, the peer. An entry that is -// not an address ends the reading, since nothing to its left can be -// believed. -func clientAddress( - peer netip.Addr, forwardedFor []string, trusted []netip.Prefix, -) netip.Addr { - client := peer - if !isInside(peer, trusted) { - return client - } - - entries := strings.Split(strings.Join(forwardedFor, ","), ",") - for _, entry := range slices.Backward(entries) { - addr, err := netip.ParseAddr(strings.TrimSpace(entry)) - if err != nil { - break - } - - client = addr.Unmap() - if !isInside(client, trusted) { - break - } - } - - return client -} - -// requestIDHeader carries the request's id, from traefik and to the app. +// requestIDHeader carries the request's id, from rproxy. const requestIDHeader = "X-Request-ID" -// requestID is the request's id: the one a trusted proxy sent, or a new -// random one. A peer outside the trusted proxies did not come through -// traefik, so the id it sends is its own claim, and is replaced. -func requestID(r *http.Request, peerTrusted bool) string { - id := r.Header.Get(requestIDHeader) - if !peerTrusted || id == "" { - id = rand.Text() +// requestID is the request's id: the one a question gives, or for any other +// request a new random one. +func requestID(r *http.Request, isQuestion bool) string { + if isQuestion { + return r.Header.Get(requestIDHeader) } - return id + return rand.Text() } -// scheme is how the client reached traefik, as a trusted proxy says in -// X-Forwarded-Proto, or otherwise http, the only scheme smallwebwaf -// serves. -func scheme(r *http.Request, peerTrusted bool) string { +// scheme is how the visitor reached rproxy, or the trusted proxy in front of +// it, as a question says in X-Forwarded-Proto, or otherwise http. +func scheme(r *http.Request, isQuestion bool) string { proto := r.Header.Get("X-Forwarded-Proto") - if !peerTrusted || proto == "" { + if !isQuestion || proto == "" { return "http" } @@ -95,48 +60,3 @@ func isInside(addr netip.Addr, netblocks []netip.Prefix) bool { return netblock.Contains(addr) }) } - -// setForwardedHeaders sets the headers in which the app learns about the -// client, so that it sees what it would see from traefik directly. A -// trusted proxy's forwarded headers pass on, with the proxy's own address -// added to X-Forwarded-For. Those of any other peer are its own claims and -// are replaced: X-Forwarded-For names the peer, X-Forwarded-Host the host -// it asked for, and X-Forwarded-Proto plain http, which is how it reached -// smallwebwaf. -func setForwardedHeaders(in, out *http.Request, peer netip.Addr, trusted bool) { - forwardedFor := peer.String() - - if trusted { - // ReverseProxy removes these from out before Rewrite. - for _, name := range []string{"Forwarded", "X-Forwarded-Host", "X-Forwarded-Proto"} { - values, ok := in.Header[name] - if ok { - out.Header[name] = values - } - } - - prior := in.Header.Values("X-Forwarded-For") - if len(prior) > 0 { - forwardedFor = strings.Join(prior, ", ") + ", " + forwardedFor - } - - out.Header.Set("X-Forwarded-For", forwardedFor) - - return - } - - // ReverseProxy has removed Forwarded and the three set below; these - // are the other headers in which traefik tells the app about the - // client and its request. - for _, name := range []string{ - "X-Forwarded-Port", "X-Forwarded-Server", "X-Forwarded-Uri", - "X-Forwarded-Method", "X-Forwarded-Prefix", "X-Forwarded-Tls-Client-Cert", - "X-Forwarded-Tls-Client-Cert-Info", "X-Real-Ip", - } { - out.Header.Del(name) - } - - out.Header.Set("X-Forwarded-For", forwardedFor) - out.Header.Set("X-Forwarded-Host", in.Host) - out.Header.Set("X-Forwarded-Proto", "http") -} diff --git a/internal/proxy/client_test.go b/internal/proxy/client_test.go deleted file mode 100644 index cf4ece5..0000000 --- a/internal/proxy/client_test.go +++ /dev/null @@ -1,165 +0,0 @@ -package proxy_test - -import ( - "encoding/json" - "net/http" - "testing" -) - -const ( - // trustLocalhost trusts the address every test connects from, and a - // network for proxies in front of it. - trustLocalhost = localhost + "/32,10.0.0.0/8" - // appHost is the host every test asks for. - appHost = "app.example" - // client is the client's address, as a proxy names it. - client = "203.0.113.9" - // forwardedFor is the header that lists the client and its proxies, - // and forwardedProto the one that gives the scheme the client used. - forwardedFor = "X-Forwarded-For" - forwardedProto = "X-Forwarded-Proto" - // secure is the scheme a client reached traefik with, and plain the - // one smallwebwaf serves. - secure = "https" - plain = "http" -) - -// appHeaders is what the app tells about the headers it received. -type appHeaders struct { - Host string `json:"host"` - ForwardedFor string `json:"forwardedFor"` - ForwardedHost string `json:"forwardedHost"` - ForwardedProto string `json:"forwardedProto"` - RealIP string `json:"realIp"` -} - -// clientAddressCase is a request and what smallwebwaf makes of it. -type clientAddressCase struct { - name string - env map[string]string - header http.Header - wantClient string - wantApp appHeaders -} - -func TestClientAddressAndForwardedHeaders(t *testing.T) { - t.Parallel() - - for _, tc := range clientAddressCases() { - t.Run(tc.name, func(t *testing.T) { - t.Parallel() - - got, line := requestWithHeaders(t, tc.env, tc.header) - - tc.wantApp.Host = appHost - if got != tc.wantApp { - t.Errorf("app received %+v, want %+v", got, tc.wantApp) - } - - if line.ClientIP != tc.wantClient || line.PeerIP != localhost { - t.Errorf("log line has client_ip %q and peer_ip %q, want %q and %q", - line.ClientIP, line.PeerIP, tc.wantClient, localhost) - } - }) - } -} - -// clientAddressCases are the requests TestClientAddressAndForwardedHeaders -// sends, from 127.0.0.1, which the default trusted proxies leave out. -func clientAddressCases() []clientAddressCase { - trusted := map[string]string{trustedProxies: trustLocalhost} - forged := http.Header{ - forwardedFor: {client}, - "X-Forwarded-Host": {"forged.example"}, - forwardedProto: {secure}, - "X-Real-Ip": {client}, - } - replaced := appHeaders{ - ForwardedFor: localhost, ForwardedHost: appHost, ForwardedProto: plain, - } - - return []clientAddressCase{{ - name: "a peer outside the trusted proxies is the client, " + - "and its forwarded headers are replaced", - header: forged, wantClient: localhost, wantApp: replaced, - }, { - name: "set but empty, the trusted proxies trust nothing", - env: map[string]string{trustedProxies: ""}, - header: forged, wantClient: localhost, wantApp: replaced, - }, { - name: "behind a trusted peer, the client is the first address " + - "outside the trusted proxies from the right", - env: trusted, - header: http.Header{ - forwardedFor: {"198.51.100.7, " + client + ", 10.0.0.2"}, - "X-Forwarded-Host": {appHost}, - forwardedProto: {secure}, - "X-Real-Ip": {client}, - }, - wantClient: client, - wantApp: appHeaders{ - ForwardedFor: "198.51.100.7, " + client + ", 10.0.0.2, " + localhost, - ForwardedHost: appHost, ForwardedProto: secure, RealIP: client, - }, - }, { - name: "when every address is a trusted proxy, the leftmost is the client", - env: trusted, - header: http.Header{forwardedFor: {"10.0.0.5, 10.0.0.2"}}, - wantClient: "10.0.0.5", - wantApp: appHeaders{ForwardedFor: "10.0.0.5, 10.0.0.2, " + localhost}, - }, { - name: "with no header, a trusted peer is the client", - env: trusted, - wantClient: localhost, - wantApp: appHeaders{ForwardedFor: localhost}, - }, { - name: "an entry that is not an address ends the reading", - env: trusted, - header: http.Header{forwardedFor: {client + ", unknown, 10.0.0.2"}}, - wantClient: "10.0.0.2", - wantApp: appHeaders{ - ForwardedFor: client + ", unknown, 10.0.0.2, " + localhost, - }, - }, { - name: "several header lines are read as one list", - env: trusted, - header: http.Header{forwardedFor: {"2001:db8::7", "10.0.0.2"}}, - wantClient: "2001:db8::7", - wantApp: appHeaders{ForwardedFor: "2001:db8::7, 10.0.0.2, " + localhost}, - }} -} - -// requestWithHeaders sends a request for appHost with header through -// smallwebwaf, with the settings in env, and returns the headers the app -// received and the request's log line. -func requestWithHeaders( - t *testing.T, env map[string]string, header http.Header, -) (appHeaders, logLine) { - t.Helper() - - app := startApp(t, func(w http.ResponseWriter, r *http.Request) { - _ = json.NewEncoder(w).Encode(appHeaders{ - Host: r.Host, - ForwardedFor: r.Header.Get(forwardedFor), - ForwardedHost: r.Header.Get("X-Forwarded-Host"), - ForwardedProto: r.Header.Get(forwardedProto), - RealIP: r.Header.Get("X-Real-IP"), - }) - }) - addr, out := startProxy(t, app.URL, env) - - req := newRequest(t, http.MethodGet, addr, "/", http.NoBody) - req.Host = appHost - req.Header = header.Clone() - - answered := do(t, req) - - var got appHeaders - - err := json.Unmarshal(answered.body, &got) - if err != nil { - t.Fatalf("decode the app's answer %q: %v", answered.body, err) - } - - return got, out.requestLine(t) -} diff --git a/internal/proxy/coreruleset.go b/internal/proxy/coreruleset.go index e5c4bf9..7fb81fc 100644 --- a/internal/proxy/coreruleset.go +++ b/internal/proxy/coreruleset.go @@ -1,9 +1,6 @@ package proxy import ( - "errors" - "net/http" - "os" "time" "sneak.berlin/go/smallwebwaf/internal/alerts" @@ -19,8 +16,7 @@ import ( // SWWAF_WAF_ANOMALY_THRESHOLD is a match: it raises the waf_block alert, // and in block mode refuses the request, which is an offence its client's // history counts, and so returns ActionWAFBlocked. It returns "" for a -// request it does not refuse, and for one whose body meets a size or time -// limit while the Core Rule Set reads it, which it notes nothing of. +// request it does not refuse. func (rq *request) checkCoreRuleSet() string { cfg := rq.h.config if cfg.WAFMode == config.WAFModeOff || pathExempt(rq.in, cfg.WAFExemptPaths) { @@ -29,10 +25,10 @@ func (rq *request) checkCoreRuleSet() string { start := time.Now() - result := rq.inspect() - if rq.refused.Load() != nil { - return "" // the refusal for that limit, which check returns - } + // A question's body has been read into memory, so that reading it here + // cannot fail. + result, read, _ := rq.h.coreRuleSet.Inspect(rq.in, rq.client, rq.in.Body) + rq.requestBytes = int64(len(read)) rq.line.DurationWAF = new(requestlog.Milliseconds(time.Since(start))) rq.line.WAFRuleIDs = result.RuleIDs @@ -58,39 +54,6 @@ func (rq *request) checkCoreRuleSet() string { return requestlog.ActionWAFBlocked } -// inspect runs the Core Rule Set on the request, which reads the part of -// its body it inspects within SWWAF_CLIENT_REQUEST_TIMEOUT, and keeps that -// part for the app. A client that runs out of time is refused with 408 -// here, and a body over SWWAF_REQUEST_MAX_BYTES with 413 as it is read; -// check returns the refusal. A body that breaks off for any other reason -// is passed on as far as it came, and the request to the app fails there, -// as it would have without the Core Rule Set. -func (rq *request) inspect() waf.Result { - if rq.body == nil { - // Nothing is read of no body, so nothing can go wrong reading it. - result, _, _ := rq.h.coreRuleSet.Inspect(rq.in, rq.client, http.NoBody) - - return result - } - - _ = rq.rc.SetReadDeadline(rq.clientRequestDeadline()) - result, read, err := rq.h.coreRuleSet.Inspect(rq.in, rq.client, rq.body) - // The timeouts that run while the request goes to the app take over. - _ = rq.rc.SetReadDeadline(time.Time{}) - - rq.body.readByCoreRuleSet = read - - if errors.Is(err, os.ErrDeadlineExceeded) { - rq.refuse(refusal{ - status: http.StatusRequestTimeout, - action: requestlog.ActionTimedOut, - limit: "SWWAF_CLIENT_REQUEST_TIMEOUT", - }) - } - - return result -} - // alertWAFBlock raises the waf_block alert for the request, which the Core // Rule Set scored at result, at or over SWWAF_WAF_ANOMALY_THRESHOLD. Its // detail gives the rule ids, the score, the method and the path with the diff --git a/internal/proxy/coreruleset_test.go b/internal/proxy/coreruleset_test.go index d371171..cc877bc 100644 --- a/internal/proxy/coreruleset_test.go +++ b/internal/proxy/coreruleset_test.go @@ -8,7 +8,6 @@ import ( "strconv" "strings" "testing" - "time" "sneak.berlin/go/smallwebwaf/internal/alerts" "sneak.berlin/go/smallwebwaf/internal/ratelimit" @@ -29,6 +28,13 @@ const ( // formData is the type of a form's body. const formData = "application/x-www-form-urlencoded" +// sizeLimit is SWWAF_WAF_BODY_LIMIT in the tests that read bodies, 1K as a +// setting. +const ( + sizeLimit = 1 << 10 + sizeLimitSetting = "1K" +) + // sqlInjection asks for / with an SQL injection in its query, which only // the Core Rule Set's rule 942100 matches, with a score of 5, the default // SWWAF_WAF_ANOMALY_THRESHOLD. @@ -96,7 +102,7 @@ func TestCoreRuleSetRefusesAttacksInBlockModeAndOnlyLogsThemInDetectMode(t *test } { s, _, _ := startWithClock(t, "", map[string]string{wafMode: tc.mode}) - line, _ := s.requestWithHeader(client, attack.path, attack.header, + line := s.requestWithHeader(client, attack.path, attack.header, tc.status, tc.action) wantWAF(t, line, &attack.score, attack.ruleIDs...) } @@ -299,7 +305,7 @@ func TestBodiesReachTheAppAsSentWhileBodiesAreRead(t *testing.T) { tc.name, len(got.body), got.status, len(tc.body)) } - line := out.requestLines(t, i+1)[i] + line := out.endedLines(t, i+1)[i] wantLine(t, line, http.StatusOK, requestlog.ActionForward) if line.RequestBytes != int64(len(tc.body)) { @@ -309,119 +315,6 @@ func TestBodiesReachTheAppAsSentWhileBodiesAreRead(t *testing.T) { } } -func TestFormBodyLongerThanTheLimitStreamsOnToTheApp(t *testing.T) { - t.Parallel() - - const ( - first = "a=" // and twice the limit of b's, then the rest - rest = 64 * sizeLimit - ) - - // past is closed once the app has received twice what the Core Rule - // Set reads, and got is the length of the whole body it received. - past := make(chan struct{}) - got := make(chan int64, 1) - app := startApp(t, func(_ http.ResponseWriter, r *http.Request) { - n, _ := io.CopyN(io.Discard, r.Body, 2*sizeLimit) - - close(past) - - m, _ := io.Copy(io.Discard, r.Body) - got <- n + m - }) - addr, out := startProxy(t, app.URL, map[string]string{ - wafMode: block, wafBodyLimit: sizeLimitSetting, - }) - - // The client sends the rest only once the app has received the first - // part: were smallwebwaf to hold the body until the end, it would - // never come. - body, sender := io.Pipe() - - go func() { - _, _ = io.WriteString(sender, first+strings.Repeat("b", 2*sizeLimit)) - - select { - case <-past: - case <-time.After(waitLimit): - t.Error("the app got no more than the Core Rule Set reads " + - "before the whole body was sent") - - _ = sender.CloseWithError(io.ErrUnexpectedEOF) - - return - } - - _, _ = io.WriteString(sender, strings.Repeat("b", rest)) - _ = sender.Close() - }() - - req := newRequest(t, http.MethodPost, addr, "/", body) - req.Header.Set("Content-Type", formData) - wantStatus(t, do(t, req), http.StatusOK) - - want := int64(len(first) + 2*sizeLimit + rest) - if n := <-got; n != want { - t.Errorf("the app got %d bytes, want %d", n, want) - } - - wantLine(t, out.requestLine(t), http.StatusOK, requestlog.ActionForward) -} - -func TestClientTooSlowToSendWhatTheCoreRuleSetReads(t *testing.T) { - t.Parallel() - - app := startApp(t, func(http.ResponseWriter, *http.Request) {}) - addr, out := startProxy(t, app.URL, map[string]string{ - wafMode: block, wafBodyLimit: sizeLimitSetting, - clientRequestTimeout: shortTimeoutSetting, metricsToken: token, - }) - - conn := dial(t, addr) - send(t, conn, "POST /comment HTTP/1.1\r\nHost: app\r\nContent-Type: "+formData+ - "\r\nContent-Length: 100\r\n\r\ncontent=the first bytes") - - wantStatus(t, readResponse(t, conn), http.StatusRequestTimeout) - - line := out.requestLine(t) - wantLine(t, line, http.StatusRequestTimeout, requestlog.ActionTimedOut) - wantNotSentToTheApp(t, line) - wantLimitHits(t, addr, clientRequestTimeout, 1) -} - -func TestBodyOverTheSizeLimitWhileTheCoreRuleSetReadsIt(t *testing.T) { - t.Parallel() - - app := startApp(t, func(http.ResponseWriter, *http.Request) {}) - addr, out := startProxy(t, app.URL, map[string]string{ - wafMode: block, wafBodyLimit: "4K", - requestMaxBytes: sizeLimitSetting, metricsToken: token, - }) - - // Sent in chunks, its length is not announced, and is found to be over - // the limit as the Core Rule Set reads it. - body := io.MultiReader(strings.NewReader("a=" + strings.Repeat("b", 2*sizeLimit))) - req := newRequest(t, http.MethodPost, addr, "/", body) - req.Header.Set("Content-Type", formData) - wantStatus(t, do(t, req), http.StatusRequestEntityTooLarge) - - line := out.requestLine(t) - wantLine(t, line, http.StatusRequestEntityTooLarge, requestlog.ActionTooLarge) - wantNotSentToTheApp(t, line) - wantLimitHits(t, addr, requestMaxBytes, 1) -} - -// wantNotSentToTheApp checks that the request of line was not sent to the -// app at all. -func wantNotSentToTheApp(t *testing.T, line logLine) { - t.Helper() - - _, sent := line.fields["duration_upstream_total"] - if sent { - t.Error("log line has duration_upstream_total, for a request sent to the app") - } -} - func TestResponsesAreNotInspected(t *testing.T) { t.Parallel() @@ -441,14 +334,12 @@ func TestResponsesAreNotInspected(t *testing.T) { t.Errorf("answered %d with %q, want 200 with the app's page", got.status, got.body) } - wantLine(t, out.requestLine(t), http.StatusOK, requestlog.ActionForward) + wantLine(t, out.endedLine(t), http.StatusOK, requestlog.ActionForward) } func TestCoreRuleSetRefusalIsAnOffenceAndCountsTowardTheErrorBurst(t *testing.T) { t.Parallel() - const scraper = "192.0.2.200" - s, _, server := startWithClock(t, "", map[string]string{ wafMode: block, errorBurstThreshold: "2", metricsToken: token, }) @@ -474,7 +365,7 @@ func TestCoreRuleSetRefusalIsAnOffenceAndCountsTowardTheErrorBurst(t *testing.T) t.Errorf("history counts the offences %+v, want %+v", offences, want) } - metrics := s.scrape(scraper) + metrics := s.scrape() wantMetric(t, metrics, `smallwebwaf_waf_matches_total{instance="app",mode="block",rule_id="942100"}`, 3) wantMetric(t, metrics, @@ -486,8 +377,6 @@ func TestCoreRuleSetRefusalIsAnOffenceAndCountsTowardTheErrorBurst(t *testing.T) func TestDetectModeMatchIsNoOffenceAndNotCountedTowardTheErrorBurst(t *testing.T) { t.Parallel() - const scraper = "192.0.2.200" - s, _, server := startWithClock(t, "", map[string]string{ wafMode: detect, errorBurstThreshold: "2", metricsToken: token, }) @@ -503,7 +392,7 @@ func TestDetectModeMatchIsNoOffenceAndNotCountedTowardTheErrorBurst(t *testing.T t.Errorf("history counts the offences %+v, want none", offences) } - metrics := s.scrape(scraper) + metrics := s.scrape() wantMetric(t, metrics, `smallwebwaf_waf_matches_total{instance="app",mode="detect",rule_id="942100"}`, 3) wantNoSeries(t, metrics, @@ -576,6 +465,7 @@ func TestCoreRuleSetMatchRaisesTheWAFBlockAlert(t *testing.T) { Instance: alertInstance, Time: clk.Now(), Event: alerts.EventWAFBlock, + Host: appHost, Client: netip.MustParseAddr(client), Netblock: netip.MustParsePrefix(client + "/32"), Reason: "scored by the Core Rule Set at or over SWWAF_WAF_ANOMALY_THRESHOLD", diff --git a/internal/proxy/countries_test.go b/internal/proxy/countries_test.go index 0725070..d148226 100644 --- a/internal/proxy/countries_test.go +++ b/internal/proxy/countries_test.go @@ -65,7 +65,7 @@ func TestCountryLists(t *testing.T) { req.Header.Set(forwardedFor, sent.client) got := do(t, req) - line := out.requestLines(t, i+1)[i] + line := out.endedLines(t, i+1)[i] if line.ASN != sent.asn || line.ASName != sent.asName || line.Country != sent.country { t.Errorf("log line has %q, %q and %q, want %q, %q and %q", @@ -90,7 +90,7 @@ func TestCountryLists(t *testing.T) { } } -func TestCountryRefusalComesBeforeTheBody(t *testing.T) { +func TestCountryRefusalKeepsARequestFromTheApp(t *testing.T) { t.Parallel() var calls atomic.Int32 @@ -109,13 +109,9 @@ func TestCountryRefusalComesBeforeTheBody(t *testing.T) { req.Header.Set(forwardedFor, fromKP) wantStatus(t, do(t, req), http.StatusForbidden) - line := out.requestLine(t) + line := out.endedLine(t) wantLine(t, line, http.StatusForbidden, requestlog.ActionCountryDenied) - if line.RequestBytes != 0 { - t.Errorf("log line has request_bytes %d, want 0", line.RequestBytes) - } - if calls.Load() != 0 { t.Errorf("the app was called %d times, want none", calls.Load()) } @@ -211,7 +207,7 @@ func TestPrivateAddressIsNeverLookedUp(t *testing.T) { wantStatus(t, do(t, req), http.StatusOK) - line := out.requestLines(t, i+1)[i] + line := out.endedLines(t, i+1)[i] wantLine(t, line, http.StatusOK, requestlog.ActionForward) for _, field := range []string{"asn", "as_name", "country"} { diff --git a/internal/proxy/crowdsec_test.go b/internal/proxy/crowdsec_test.go index 8868751..4c7add8 100644 --- a/internal/proxy/crowdsec_test.go +++ b/internal/proxy/crowdsec_test.go @@ -84,7 +84,7 @@ func TestClientTheCrowdSecDecisionListListsIsBannedUntilTheDecisionEnds(t *testi // The ban and the hit are counted, and the list has the metrics of any // list fetched from a URL. - metrics := s.scrape(unplaced) + metrics := s.scrape() labels := `{instance="` + alertInstance + `",source="` + decisionsURL + `"}` wantMetric(t, metrics, `smallwebwaf_bans_made_total{cause="crowdsec",`+ diff --git a/internal/proxy/errorburst_test.go b/internal/proxy/errorburst_test.go index bd8434c..57607d1 100644 --- a/internal/proxy/errorburst_test.go +++ b/internal/proxy/errorburst_test.go @@ -17,8 +17,8 @@ import ( const errorBurstThreshold = "SWWAF_ERROR_BURST_THRESHOLD" -// refused is a request the tests here send, which smallwebwaf refuses -// after a rule file match, or for a missing or wrong token. +// refused is a request the tests here send, which servewaf refuses after a rule +// file match. type refused int const ( @@ -27,11 +27,6 @@ const ( // banRule is one its ban rule refuses with 403, and bans the client // for. banRule - // noMetricsToken is one for the metrics without a token, and - // wrongAdminToken one for the bans with the metrics token, each - // refused with 401. - noMetricsToken - wrongAdminToken ) // send sends r from the client at from, checks its answer and log line as @@ -45,14 +40,6 @@ func (r refused) send(s *sender, from string) logLine { requestlog.ActionRuleBlocked) case banRule: return s.request(from, probePath, http.StatusForbidden, requestlog.ActionBanned) - case noMetricsToken: - return s.request(from, proxy.MetricsPath, http.StatusUnauthorized, - requestlog.ActionAdmin) - case wrongAdminToken: - line, _ := s.requestWithHeader(from, proxy.BansPath, "Authorization: "+bearer, - http.StatusUnauthorized, requestlog.ActionAdmin) - - return line } s.t.Fatalf("no request for the refusal %d", r) @@ -60,7 +47,7 @@ func (r refused) send(s *sender, from string) logLine { return logLine{} } -// startForErrorBurst is startWithClock with testRules, both tokens and +// startForErrorBurst is startWithClock with testRules, the metrics token and // SWWAF_ERROR_BURST_THRESHOLD at threshold, and the settings in env. func startForErrorBurst( t *testing.T, threshold string, env map[string]string, @@ -70,7 +57,6 @@ func startForErrorBurst( settings := map[string]string{ errorBurstThreshold: threshold, rulesDir: writeRules(t, testRules), - adminToken: adminSecret, metricsToken: token, } maps.Copy(settings, env) @@ -87,14 +73,7 @@ func TestErrorBurstBreaksAtOneOverTheThreshold(t *testing.T) { refusals []refused }{ {"block rule", []refused{blockRule, blockRule, blockRule, blockRule}}, - { - "missing or wrong token", - []refused{noMetricsToken, wrongAdminToken, noMetricsToken, wrongAdminToken}, - }, - { - "a mix ending in a ban rule", - []refused{blockRule, noMetricsToken, blockRule, banRule}, - }, + {"a ban rule last", []refused{blockRule, blockRule, blockRule, banRule}}, } { t.Run(tc.name, func(t *testing.T) { t.Parallel() @@ -130,13 +109,11 @@ func TestErrorBurstBreaksAtOneOverTheThreshold(t *testing.T) { func TestErrorBurstBanNotesHistoryAndMetrics(t *testing.T) { t.Parallel() - const scraper = "192.0.2.200" - s, clk, server := startForErrorBurst(t, "2", nil) start := clk.Now() blockRule.send(s, client) - wrongAdminToken.send(s, client) + blockRule.send(s, client) line := blockRule.send(s, client) expires := start.Add(time.Hour) @@ -173,19 +150,17 @@ func TestErrorBurstBanNotesHistoryAndMetrics(t *testing.T) { t.Fatalf("bans\n%+v\nwant\n%+v", got, want) } - wantOffences := ratelimit.Offences{Limit: 1, RuleBlocked: 2, TokenRefused: 1} + wantOffences := ratelimit.Offences{Limit: 1, RuleBlocked: 3} if offences := historyOf(t, server, client).Offences; offences != wantOffences { t.Errorf("history counts the offences %+v, want %+v", offences, wantOffences) } - metrics := s.scrape(scraper) + metrics := s.scrape() wantMetric(t, metrics, `smallwebwaf_rate_limit_hits_total{instance="app",`+ `kind="refusals",window="minute"}`, 1) wantMetric(t, metrics, `smallwebwaf_offences_total{instance="app",kind="limit"}`, 1) wantMetric(t, metrics, - `smallwebwaf_offences_total{instance="app",kind="rule_blocked"}`, 2) - wantMetric(t, metrics, - `smallwebwaf_offences_total{instance="app",kind="token_refused"}`, 1) + `smallwebwaf_offences_total{instance="app",kind="rule_blocked"}`, 3) wantMetric(t, metrics, `smallwebwaf_bans_made_total{cause="limit",instance="app"}`, 1) } @@ -291,19 +266,15 @@ func TestErrorBurstCountsEachClientTheChecksApplyTo(t *testing.T) { allowNets: allowed, rateLimitExemptNets: exempt, }) - // A client in SWWAF_ALLOW_NETS still needs the token, but is not - // counted. + // A client in SWWAF_ALLOW_NETS is not refused, and so not counted. for range 3 { - line := noMetricsToken.send(s, allowed) - if line.LimitHit != "" { - t.Errorf("log line has limit_hit %q, want none", line.LimitHit) - } + s.request(allowed, blockedPath, http.StatusOK, requestlog.ActionForward) } // One the rate limits do not apply to is. - noMetricsToken.send(s, exempt) + blockRule.send(s, exempt) - line := wrongAdminToken.send(s, exempt) + line := blockRule.send(s, exempt) if line.LimitHit != requestlog.LimitHitErrorBurst { t.Errorf("log line has limit_hit %q, want error_burst", line.LimitHit) } @@ -336,7 +307,6 @@ func TestObserveModeLogsAndAlertsTheErrorBurst(t *testing.T) { mode: observe, errorBurstThreshold: "1", rulesDir: writeRules(t, testRules), - adminToken: adminSecret, }) start := clk.Now() held := bans.Ban{ @@ -347,10 +317,11 @@ func TestObserveModeLogsAndAlertsTheErrorBurst(t *testing.T) { } server.Ledger.Load([]bans.Ban{held}) - // Under a ban, enforce mode would have refused these before the - // endpoint, so their tokens are not counted. + // Under a ban, enforce mode would have refused these before the rule, so + // they are not counted. for range 2 { - line := wrongAdminToken.send(s, otherClient) + line := s.request(otherClient, blockedPath, http.StatusOK, + requestlog.ActionForward) wantWouldAction(t, line, requestlog.ActionBanned) if line.LimitHit != "" { @@ -360,7 +331,7 @@ func TestObserveModeLogsAndAlertsTheErrorBurst(t *testing.T) { // The block rule's refusal, which enforce mode would have answered 403, // is the second of the client's, and would have banned it. - wrongAdminToken.send(s, client) + s.request(client, blockedPath, http.StatusOK, requestlog.ActionForward) line := s.request(client, blockedPath, http.StatusOK, requestlog.ActionForward) wantWouldAction(t, line, requestlog.ActionRuleBlocked) diff --git a/internal/proxy/health_test.go b/internal/proxy/health_test.go index e069d98..106c479 100644 --- a/internal/proxy/health_test.go +++ b/internal/proxy/health_test.go @@ -2,24 +2,18 @@ package proxy_test import ( "net/http" - "sync/atomic" "testing" "sneak.berlin/go/smallwebwaf/internal/proxy" "sneak.berlin/go/smallwebwaf/internal/requestlog" ) -func TestHealthEndpointIsAnsweredBeforeAnyCheck(t *testing.T) { +func TestHealthEndpointCountsTowardNoLimit(t *testing.T) { t.Parallel() - var calls atomic.Int32 - - app := startApp(t, func(http.ResponseWriter, *http.Request) { - calls.Add(1) - }) // With a limit of one request a minute, any request counted before // the last one would have it refused. - addr, out := startProxy(t, app.URL, map[string]string{rateLimitPerMinute: "1"}) + s, _, _ := startWithClock(t, "", map[string]string{rateLimitPerMinute: "1"}) const ( healthChecks = 3 @@ -27,8 +21,7 @@ func TestHealthEndpointIsAnsweredBeforeAnyCheck(t *testing.T) { ) for range healthChecks { - got := get(t, addr, proxy.HealthPath) - wantStatus(t, got, http.StatusOK) + got := s.adminRequest("", http.MethodGet, proxy.HealthPath, "", http.StatusOK) if string(got.body) != "ok\n" || got.header.Get("Content-Type") != contentType { t.Errorf("health endpoint answered %q with Content-Type %q, want ok "+ @@ -36,21 +29,12 @@ func TestHealthEndpointIsAnsweredBeforeAnyCheck(t *testing.T) { } } - wantStatus(t, get(t, addr, "/"), http.StatusOK) - - lines := out.requestLines(t, healthChecks+1) - for _, line := range lines[:healthChecks] { - wantLine(t, line, http.StatusOK, requestlog.ActionAdmin) + s.get(client, http.StatusOK, requestlog.ActionForward) + for _, line := range s.out.endedLines(t, healthChecks)[:healthChecks] { if line.ResponseContentType != contentType { t.Errorf("health check's log line has response_content_type %q, "+ "want %q", line.ResponseContentType, contentType) } } - - wantLine(t, lines[healthChecks], http.StatusOK, requestlog.ActionForward) - - if calls.Load() != 1 { - t.Errorf("the app was called %d times, want once", calls.Load()) - } } diff --git a/internal/proxy/history_test.go b/internal/proxy/history_test.go index ffaa9de..25ca614 100644 --- a/internal/proxy/history_test.go +++ b/internal/proxy/history_test.go @@ -86,7 +86,7 @@ func TestHistoryCountsTheBodiesEachWay(t *testing.T) { got := do(t, newRequest(t, http.MethodPost, addr, "/", strings.NewReader("abc"))) wantStatus(t, got, http.StatusOK) - out.requestLine(t) + out.endedLine(t) history := historyOf(t, server, localhost) if history.RequestBytes != 3 || history.ResponseBytes != 5 { @@ -95,38 +95,33 @@ func TestHistoryCountsTheBodiesEachWay(t *testing.T) { } } -func TestHealthEndpointIsNotInTheHistory(t *testing.T) { +func TestOwnEndpointsAreNotInTheHistory(t *testing.T) { t.Parallel() - app := startApp(t, func(http.ResponseWriter, *http.Request) {}) - addr, out, server := startProxyWithClock(t, app.URL, "", time.Now, nil) + s, _, server := startWithClock(t, "", map[string]string{metricsToken: token}) - wantStatus(t, get(t, addr, proxy.HealthPath), http.StatusOK) - out.requestLine(t) + s.adminRequest("", http.MethodGet, proxy.HealthPath, "", http.StatusOK) + s.adminRequest("", http.MethodGet, proxy.MetricsPath, "", http.StatusUnauthorized) + s.scrape() if clients := server.Limiter.Snapshot(); len(clients) != 0 { t.Errorf("the table holds %+v, want no client", clients) } } -func TestRequestForSmallwebwafIsRefusedOnlyWithoutTheToken(t *testing.T) { +func TestPathUnderTheOwnPrefixIsNeitherForwardedNorRefused(t *testing.T) { t.Parallel() app := startApp(t, func(http.ResponseWriter, *http.Request) {}) - addr, out, server := startProxyWithClock(t, app.URL, "", time.Now, - map[string]string{metricsToken: token}) + addr, out, server := startProxyWithClock(t, app.URL, "", time.Now, nil) - // The metrics and the 404 are neither forwarded nor refused; the 401 - // is refused. - scrape(t, addr) wantStatus(t, get(t, addr, "/_smallwebwaf/nothing"), http.StatusNotFound) - wantStatus(t, get(t, addr, proxy.MetricsPath), http.StatusUnauthorized) - out.requestLines(t, 3) + out.endedLine(t) history := historyOf(t, server, localhost) - if history.Requests != 3 || history.Forwarded != 0 || history.Refused != 1 { + if history.Requests != 1 || history.Forwarded != 0 || history.Refused != 0 { t.Errorf("history counts %d requests, %d forwarded and %d refused, "+ - "want 3, 0 and 1", history.Requests, history.Forwarded, history.Refused) + "want 1, 0 and 0", history.Requests, history.Forwarded, history.Refused) } } diff --git a/internal/proxy/limits_test.go b/internal/proxy/limits_test.go deleted file mode 100644 index 1f7efe3..0000000 --- a/internal/proxy/limits_test.go +++ /dev/null @@ -1,162 +0,0 @@ -package proxy_test - -import ( - "bytes" - "errors" - "io" - "net/http" - "strconv" - "sync/atomic" - "testing" - - "sneak.berlin/go/smallwebwaf/internal/requestlog" -) - -// sizeLimit is the size limit the tests set, 1K as a setting. -const ( - sizeLimit = 1 << 10 - sizeLimitSetting = "1K" -) - -func TestRequestBodyLimit(t *testing.T) { - t.Parallel() - - for _, tc := range []struct { - name string - size int - // announced sends the size in Content-Length; otherwise the body - // is sent in chunks with no length given. - announced bool - want int - action string - // refusedBeforeApp is a refusal before anything reaches the app. - // A body over the limit with no length given has already partly - // reached the app when it is refused. - refusedBeforeApp bool - }{ - {"announced, over the limit", 2 * sizeLimit, true, - http.StatusRequestEntityTooLarge, requestlog.ActionTooLarge, true}, - {"announced, at the limit", sizeLimit, true, - http.StatusOK, requestlog.ActionForward, false}, - {"not announced, over the limit", 4 * sizeLimit, false, - http.StatusRequestEntityTooLarge, requestlog.ActionTooLarge, false}, - {"not announced, at the limit", sizeLimit, false, - http.StatusOK, requestlog.ActionForward, false}, - } { - t.Run(tc.name, func(t *testing.T) { - t.Parallel() - - var calls atomic.Int32 - - app := startApp(t, func(_ http.ResponseWriter, r *http.Request) { - calls.Add(1) - - _, _ = io.Copy(io.Discard, r.Body) - }) - addr, out := startProxy(t, app.URL, map[string]string{ - requestMaxBytes: sizeLimitSetting, - metricsToken: token, - }) - - var body io.Reader = bytes.NewReader(make([]byte, tc.size)) - if !tc.announced { - body = io.MultiReader(body) // hides the length - } - - wantStatus(t, do(t, newRequest(t, http.MethodPost, addr, "/upload", body)), - tc.want) - wantLine(t, out.requestLine(t), tc.want, tc.action) - - hits := 0 - if tc.action == requestlog.ActionTooLarge { - hits = 1 - } - - wantLimitHits(t, addr, requestMaxBytes, hits) - - if tc.refusedBeforeApp && calls.Load() != 0 { - t.Errorf("the app was called %d times, want never", calls.Load()) - } - }) - } -} - -func TestResponseBodyLimit(t *testing.T) { - t.Parallel() - - for _, tc := range []struct { - name string - size int - // announced sends the size in Content-Length; otherwise the body - // is sent in chunks with no length given. - announced bool - want int - action string - // received is how much of a body the client gets, and cutOff - // whether the connection is then cut. - received int - cutOff bool - }{ - {"announced, over the limit", 2 * sizeLimit, true, http.StatusBadGateway, - requestlog.ActionTooLarge, len("Bad Gateway\n"), false}, - {"announced, at the limit", sizeLimit, true, http.StatusOK, - requestlog.ActionForward, sizeLimit, false}, - {"not announced, over the limit", 4 * sizeLimit, false, http.StatusOK, - requestlog.ActionTooLarge, sizeLimit, true}, - {"not announced, at the limit", sizeLimit, false, http.StatusOK, - requestlog.ActionForward, sizeLimit, false}, - } { - t.Run(tc.name, func(t *testing.T) { - t.Parallel() - - app := startApp(t, func(w http.ResponseWriter, _ *http.Request) { - answerWithSize(w, tc.size, tc.announced) - }) - addr, out := startProxy(t, app.URL, map[string]string{ - responseMaxBytes: sizeLimitSetting, - metricsToken: token, - }) - - got := get(t, addr, "/download") - wantStatus(t, got, tc.want) - - if len(got.body) != tc.received || - errors.Is(got.err, io.ErrUnexpectedEOF) != tc.cutOff { - t.Errorf("client got %d bytes (%v), want %d", - len(got.body), got.err, tc.received) - } - - line := out.requestLine(t) - wantLine(t, line, tc.want, tc.action) - - if line.UpstreamStatus != http.StatusOK { - t.Errorf("log line has upstream_status %d", line.UpstreamStatus) - } - - hits := 0 - if tc.action == requestlog.ActionTooLarge { - hits = 1 - } - - wantLimitHits(t, addr, responseMaxBytes, hits) - }) - } -} - -// answerWithSize answers with a body of size bytes, announced in -// Content-Length or sent in chunks with no length given. -func answerWithSize(w http.ResponseWriter, size int, announced bool) { - body := make([]byte, size) - if announced { - w.Header().Set("Content-Length", strconv.Itoa(size)) - _, _ = w.Write(body) - - return - } - - // Sending part of it before the end keeps Go's server from working - // out the length. - _, _ = w.Write(body[:size/2]) - _ = http.NewResponseController(w).Flush() - _, _ = w.Write(body[size/2:]) -} diff --git a/internal/proxy/lookup.go b/internal/proxy/lookup.go index 1bf1fac..a19f550 100644 --- a/internal/proxy/lookup.go +++ b/internal/proxy/lookup.go @@ -9,11 +9,10 @@ import ( "sneak.berlin/go/smallwebwaf/internal/lookup" ) -// The headers in which the app is passed the client's AS number and -// country while SWWAF_ADD_LOOKUP_HEADERS is set. Go writes every header -// name in this form, as it sends it and as it receives it, so X-Client-ASN -// arrives as X-Client-Asn, and Del removes a client's own whatever their -// case; header names are not case-sensitive. +// The headers in which an answer of allow gives rproxy the client's AS number +// and country, for the app, while SWWAF_ADD_LOOKUP_HEADERS is set. Go writes +// every header name in this form, as it sends it and as it receives it, so +// X-Client-ASN arrives as X-Client-Asn; header names are not case-sensitive. const ( asnHeader = "X-Client-Asn" countryHeader = "X-Client-Country" @@ -37,7 +36,7 @@ func (rq *request) lookUp(ctx context.Context, now time.Time) { switch { case rq.h.config.LookupSource == "file": rq.lookupAnswer = rq.h.lookupFile.LookUp(client) - case rq.h.answersQuestions() && rq.refusedWhateverTheAnswer(now): + case rq.refusedWhateverTheAnswer(now): rq.lookupAnswer = rq.h.geojs.LookUpWithoutWaiting(ctx, client) rq.skippedLookupWait = rq.lookupAnswer.Answered.IsZero() default: @@ -60,8 +59,8 @@ func (h *handler) addLookup(answer lookup.Answer) { answer.ASN, answer.ASName, answer.Country) } -// setLookupHeaders sets the headers in which the app is passed the -// client's AS number and country, leaving out one that is unknown. +// setLookupHeaders sets the headers that give the client's AS number and +// country, leaving out one that is unknown. func setLookupHeaders(header http.Header, asn, country string) { if asn != "" { header.Set(asnHeader, asn) diff --git a/internal/proxy/lookup_test.go b/internal/proxy/lookup_test.go index 9ea14eb..e45a9f4 100644 --- a/internal/proxy/lookup_test.go +++ b/internal/proxy/lookup_test.go @@ -1,9 +1,7 @@ package proxy_test import ( - "net" "net/http" - "net/http/httptest" "net/netip" "path/filepath" "slices" @@ -142,7 +140,7 @@ func TestRequestThatWaitedForItsLookupIsCountedWhenItGoesOn(t *testing.T) { t.Fatalf("first request: %v", err) } - wantLine(t, out.requestLines(t, 2)[1], http.StatusOK, requestlog.ActionForward) + wantLine(t, out.endedLines(t, 2)[1], http.StatusOK, requestlog.ActionForward) // The third request's count holds all three: the first was counted // when it went on, in the minute under way, and the second's count @@ -150,7 +148,7 @@ func TestRequestThatWaitedForItsLookupIsCountedWhenItGoesOn(t *testing.T) { clk.advance(500 * time.Millisecond) wantStatus(t, do(t, requestFromDE()), http.StatusOK) - if got := out.requestLines(t, 3)[2].Counts.Minute; got != 3 { + if got := out.endedLines(t, 3)[2].Counts.Minute; got != 3 { t.Errorf("the third request's count in the minute is %v, want 3", got) } } @@ -387,18 +385,15 @@ func TestRequestWaitsAsLongAsTheLookupTimeoutSays(t *testing.T) { // is never reached. const timeout = 3 * time.Second - server, out, _ := newProxy(t, "http://app.invalid", unansweredGeoJSURL, - time.Now, map[string]string{ + server, out, _ := newServeWAF(t, unansweredGeoJSURL, time.Now, + map[string]string{ lookupTimeout: timeout.String(), allowedCountries: "DE", }) - req := httptest.NewRequestWithContext(t.Context(), http.MethodGet, "/", - http.NoBody) - req.RemoteAddr = net.JoinHostPort(fromDE, "1234") began := time.Now() - server.Handler.ServeHTTP(httptest.NewRecorder(), req) + serveFromDE(t, server, http.MethodGet, http.NoBody) if waited := time.Since(began); waited != timeout { t.Errorf("the request waited %s for its answer, want %s", waited, timeout) diff --git a/internal/proxy/metrics_test.go b/internal/proxy/metrics_test.go index c4d1634..0f7c317 100644 --- a/internal/proxy/metrics_test.go +++ b/internal/proxy/metrics_test.go @@ -7,8 +7,6 @@ import ( "net/netip" "strconv" "strings" - "sync" - "sync/atomic" "testing" "time" @@ -30,41 +28,21 @@ const ( func TestMetricsAreOffWhileTheTokenIsUnset(t *testing.T) { t.Parallel() - var calls atomic.Int32 - - app := startApp(t, func(http.ResponseWriter, *http.Request) { - calls.Add(1) - }) - addr, out := startProxy(t, app.URL, nil) + s, _, _ := startWithClock(t, "", nil) // An empty token does not match the unset one either. - for i, authorization := range []string{bearer, "Bearer ", ""} { - req := newRequest(t, http.MethodGet, addr, proxy.MetricsPath, http.NoBody) - if authorization != "" { - req.Header.Set("Authorization", authorization) - } - - wantStatus(t, do(t, req), http.StatusNotFound) - wantLine(t, out.requestLines(t, i+1)[i], http.StatusNotFound, - requestlog.ActionAdmin) - } - - if calls.Load() != 0 { - t.Errorf("the app was called %d times, want never", calls.Load()) + for _, authorization := range []string{bearer, "Bearer ", ""} { + s.adminRequest(authorization, http.MethodGet, proxy.MetricsPath, "", + http.StatusNotFound) } } func TestMetricsNeedTheTokenAndOtherPathsAreNotFound(t *testing.T) { t.Parallel() - var calls atomic.Int32 + s, _, _ := startWithClock(t, "", map[string]string{metricsToken: token}) - app := startApp(t, func(http.ResponseWriter, *http.Request) { - calls.Add(1) - }) - addr, out := startProxy(t, app.URL, map[string]string{metricsToken: token}) - - for i, tc := range []struct { + for _, tc := range []struct { method, path, authorization string status int }{ @@ -81,14 +59,7 @@ func TestMetricsNeedTheTokenAndOtherPathsAreNotFound(t *testing.T) { {http.MethodGet, "/_smallwebwaf/bans", bearer, http.StatusNotFound}, {http.MethodPost, proxy.HealthPath, "", http.StatusNotFound}, } { - req := newRequest(t, tc.method, addr, tc.path, http.NoBody) - if tc.authorization != "" { - req.Header.Set("Authorization", tc.authorization) - } - - got := do(t, req) - wantStatus(t, got, tc.status) - wantLine(t, out.requestLines(t, i+1)[i], tc.status, requestlog.ActionAdmin) + got := s.adminRequest(tc.authorization, tc.method, tc.path, "", tc.status) if tc.status == http.StatusUnauthorized && got.header.Get("WWW-Authenticate") != "Bearer" { @@ -101,58 +72,28 @@ func TestMetricsNeedTheTokenAndOtherPathsAreNotFound(t *testing.T) { t.Errorf("the metrics are\n%s", got.body) } } - - if calls.Load() != 0 { - t.Errorf("the app was called %d times, want never", calls.Load()) - } -} - -func TestMetricsAreAskedForThroughTheChecks(t *testing.T) { - t.Parallel() - - s, _, _ := startWithClock(t, "", map[string]string{ - metricsToken: token, - rateLimitPerMinute: "1", - }) - - // Asking for the metrics counts toward the client's limit of one - // request a minute, so its next request breaks it, and bans it. A - // banned client is refused the metrics too. - s.scrape(client) - s.get(client, http.StatusForbidden, requestlog.ActionRateLimited) - s.requestWithHeader(client, proxy.MetricsPath, "Authorization: "+bearer, - http.StatusForbidden, requestlog.ActionBanned) } func TestMetricsCountTheTraffic(t *testing.T) { t.Parallel() - arrived, release := make(chan struct{}), make(chan struct{}) app := startApp(t, func(w http.ResponseWriter, r *http.Request) { _, _ = io.Copy(io.Discard, r.Body) - - if r.URL.Path == "/held" { - close(arrived) - <-release - } - _, _ = io.WriteString(w, "hello") }) - releaseApp := sync.OnceFunc(func() { close(release) }) - t.Cleanup(releaseApp) - - addr, out := startProxy(t, app.URL, map[string]string{metricsToken: token}) + addr, out, server := startProxyWithClock(t, app.URL, "", time.Now, + map[string]string{metricsToken: token}) got := do(t, newRequest(t, http.MethodPost, addr, "/", strings.NewReader("abc"))) wantStatus(t, got, http.StatusOK) wantStatus(t, get(t, addr, "/_smallwebwaf/nothing"), http.StatusNotFound) - out.requestLines(t, 2) + out.endedLines(t, 2) forward := `{action="forward",instance="app",status_class="2xx"}` notFound := `{action="admin",instance="app",status_class="4xx"}` // The request for the metrics is itself under way. - metrics := scrape(t, addr) + metrics := scrape(t, server.Addr) wantMetric(t, metrics, "smallwebwaf_requests_total"+forward, 1) wantMetric(t, metrics, "smallwebwaf_requests_total"+notFound, 1) wantMetric(t, metrics, "smallwebwaf_request_bytes_total"+forward, 3) @@ -166,50 +107,20 @@ func TestMetricsCountTheTraffic(t *testing.T) { wantMetric(t, metrics, `smallwebwaf_requests_in_flight{instance="app"}`, 1) metric(t, metrics, `go_goroutines{instance="app"}`) metric(t, metrics, `process_start_time_seconds{instance="app"}`) - - // A request the app holds is under way until it ends. - httpClient := newClient(t) - held := newRequest(t, http.MethodGet, addr, "/held", http.NoBody) - ended := make(chan error, 1) - - go func() { - res, err := httpClient.Do(held) - if err == nil { - err = readAnswer(res).err - } - - ended <- err - }() - - <-arrived - wantMetric(t, scrape(t, addr), `smallwebwaf_requests_in_flight{instance="app"}`, 2) - releaseApp() - - err := <-ended - if err != nil { - t.Fatalf("held request: %v", err) - } - - out.requestLines(t, 5) - wantMetric(t, scrape(t, addr), `smallwebwaf_requests_in_flight{instance="app"}`, 1) } func TestMetricsCountLimitsAndBans(t *testing.T) { t.Parallel() - const ( - scraper = "192.0.2.200" // in SWWAF_RATE_LIMIT_EXEMPT_NETS - denied = "192.0.2.50" // in SWWAF_DENY_NETS - ) + const denied = "192.0.2.50" // in SWWAF_DENY_NETS s, clk, _ := startWithClock(t, "", map[string]string{ - metricsToken: token, - rateLimitPerMinute: "1", - rateLimitExemptNets: scraper, - denyNets: denied, - banResponse: banResponseClose, - limitBanDuration: "1h", - maxBanDuration: "2h", + metricsToken: token, + rateLimitPerMinute: "1", + denyNets: denied, + banResponse: banResponseClose, + limitBanDuration: "1h", + maxBanDuration: "2h", }) // SWWAF_BAN_RESPONSE=close sends no status at all. @@ -219,7 +130,7 @@ func TestMetricsCountLimitsAndBans(t *testing.T) { s.get(client, http.StatusOK, requestlog.ActionForward) s.get(client, 0, requestlog.ActionRateLimited) - metrics := s.scrape(scraper) + metrics := s.scrape() wantMetric(t, metrics, `smallwebwaf_requests_total{action="denied",instance="app",status_class="none"}`, 1) wantMetric(t, metrics, `smallwebwaf_rate_limit_hits_total{instance="app",`+ @@ -230,38 +141,35 @@ func TestMetricsCountLimitsAndBans(t *testing.T) { wantMetric(t, metrics, `smallwebwaf_permanent_bans{instance="app"}`, 0) clk.advance(time.Hour) - wantMetric(t, s.scrape(scraper), `smallwebwaf_active_bans{instance="app"}`, 0) + wantMetric(t, s.scrape(), `smallwebwaf_active_bans{instance="app"}`, 0) // A limit broken again right after would ban for three hours, longer // than SWWAF_MAX_BAN_DURATION, so the ban is permanent. s.get(client, http.StatusOK, requestlog.ActionForward) s.get(client, 0, requestlog.ActionRateLimited) - metrics = s.scrape(scraper) + metrics = s.scrape() wantMetric(t, metrics, `smallwebwaf_rate_limit_hits_total{instance="app",`+ `kind="requests",window="minute"}`, 2) wantMetric(t, metrics, `smallwebwaf_offences_total{instance="app",kind="limit"}`, 2) wantMetric(t, metrics, `smallwebwaf_bans_made_total{cause="limit",instance="app"}`, 2) wantMetric(t, metrics, `smallwebwaf_active_bans{instance="app"}`, 1) wantMetric(t, metrics, `smallwebwaf_permanent_bans{instance="app"}`, 1) - // denied, client, and the scraper as of its earlier requests. - wantMetric(t, metrics, `smallwebwaf_tracked_clients{instance="app"}`, 3) + // denied and client. + wantMetric(t, metrics, `smallwebwaf_tracked_clients{instance="app"}`, 2) } func TestMetricsCountTheBansAnAdminMakes(t *testing.T) { t.Parallel() - const scraper = "192.0.2.200" // in SWWAF_RATE_LIMIT_EXEMPT_NETS - s, clk, server := startWithClock(t, "", map[string]string{ - metricsToken: token, - adminToken: adminSecret, - rateLimitExemptNets: scraper, + metricsToken: token, + adminToken: adminSecret, }) const admins = `smallwebwaf_bans_made_total{cause="admin",instance="app"}` - wantMetric(t, s.scrape(scraper), admins, 0) + wantMetric(t, s.scrape(), admins, 0) // As an admin's edit of bans.json that adds a ban is taken in. server.Ledger.LoadEdit([]bans.Ban{{ @@ -269,11 +177,11 @@ func TestMetricsCountTheBansAnAdminMakes(t *testing.T) { Start: clk.Now(), }}) - wantMetric(t, s.scrape(scraper), admins, 1) + wantMetric(t, s.scrape(), admins, 1) // And a ban made through the endpoint. s.admin(http.MethodPost, proxy.BansPath, banOtherClient, http.StatusOK) - wantMetric(t, s.scrape(scraper), admins, 2) + wantMetric(t, s.scrape(), admins, 2) } func TestMetricsByCountryKeepTheBusiestAndCountTheRestAsOther(t *testing.T) { @@ -312,7 +220,7 @@ func TestMetricsByCountryKeepTheBusiestAndCountTheRestAsOther(t *testing.T) { // Each is counted before the next is sent, so that the // countries are ranked in the order sent. lines++ - out.requestLines(t, lines) + out.endedLines(t, lines) } } @@ -321,7 +229,7 @@ func TestMetricsByCountryKeepTheBusiestAndCountTheRestAsOther(t *testing.T) { send(fromDE, 2, http.StatusOK) send(fromFR, 1, http.StatusOK) - metrics := scrape(t, addr) + metrics := scrape(t, server.Addr) lines++ wantMetric(t, metrics, @@ -349,7 +257,7 @@ func TestMetricsByCountryKeepTheBusiestAndCountTheRestAsOther(t *testing.T) { // from then on, and DE's is gone. send(fromFR, 3, http.StatusOK) - metrics = scrape(t, addr) + metrics = scrape(t, server.Addr) wantMetric(t, metrics, `smallwebwaf_country_requests_total{country="KP",instance="app"}`, 3) wantMetric(t, metrics, @@ -383,13 +291,12 @@ func TestMetricsByASNumberKeepTheBusiestAndCountTheRestAsOther(t *testing.T) { answer(fromDE, "AS64501"), answer(fromKP, "AS64502"), }) - // With one AS number of its own, the other is counted as other. The - // metrics are asked for from a private address, which has no AS number. + // With one AS number of its own, the other is counted as other. s.get(fromDE, http.StatusOK, requestlog.ActionForward) s.get(fromDE, http.StatusOK, requestlog.ActionForward) s.get(fromKP, http.StatusOK, requestlog.ActionForward) - metrics := s.scrape("10.0.0.9") + metrics := s.scrape() wantMetric(t, metrics, `smallwebwaf_asn_requests_total{asn="AS64501",instance="app"}`, 2) wantMetric(t, metrics, @@ -412,11 +319,12 @@ func TestMetricsCountGeoJSRequestsAndFailures(t *testing.T) { t.Cleanup(geojs.Close) app := startApp(t, func(http.ResponseWriter, *http.Request) {}) - addr, _ := startProxyWithGeoJS(t, app.URL, geojs.URL, map[string]string{ - trustedProxies: trustLocalhost, - metricsToken: token, - deniedCountries: "kp", - }) + addr, _, server := startProxyWithClock(t, app.URL, geojs.URL, time.Now, + map[string]string{ + trustedProxies: trustLocalhost, + metricsToken: token, + deniedCountries: "kp", + }) // GeoJS fails, so the client counts as coming from an unknown country, // which SWWAF_DENIED_COUNTRIES does not refuse. @@ -427,13 +335,13 @@ func TestMetricsCountGeoJSRequestsAndFailures(t *testing.T) { // The client stops waiting for GeoJS once SWWAF_LOOKUP_TIMEOUT is up, so // GeoJS's failure can come after its request has ended. deadline := time.Now().Add(waitLimit) - metrics := scrape(t, addr) + metrics := scrape(t, server.Addr) for metric(t, metrics, `smallwebwaf_geojs_failures_total{instance="app"}`) == 0 && time.Now().Before(deadline) { time.Sleep(pollInterval) - metrics = scrape(t, addr) + metrics = scrape(t, server.Addr) } wantMetric(t, metrics, `smallwebwaf_geojs_requests_total{instance="app"}`, 1) @@ -452,8 +360,8 @@ func keptAnswer(addr, country string) lookup.Answer { } } -// scrape asks smallwebwaf at addr for the metrics, with the token, and -// returns them. +// scrape asks servewaf at addr for the metrics, with the token, and returns +// them. func scrape(t *testing.T, addr string) string { t.Helper() @@ -468,15 +376,12 @@ func scrape(t *testing.T, addr string) string { return string(got.body) } -// scrape asks for the metrics, with the token, from the client at from, -// and returns them. -func (s *sender) scrape(from string) string { +// scrape asks for the metrics, with the token, and returns them. +func (s *sender) scrape() string { s.t.Helper() - _, metrics := s.requestWithHeader(from, proxy.MetricsPath, "Authorization: "+bearer, - http.StatusOK, requestlog.ActionAdmin) - - return metrics + return string(s.adminRequest(bearer, http.MethodGet, proxy.MetricsPath, "", + http.StatusOK).body) } // metric returns the value of series in metrics, which are in the @@ -524,22 +429,3 @@ func wantNoSeries(t *testing.T, metrics, series string) { t.Errorf("there is a series %s", series) } } - -// wantLimitHits checks that the metrics of smallwebwaf at addr count hits -// requests that passed the size or time limit of the setting limit, with -// no series for it when hits is 0. -func wantLimitHits(t *testing.T, addr, limit string, hits int) { - t.Helper() - - series := `smallwebwaf_size_and_time_limit_hits_total{instance="app",limit="` + - limit + `"}` - metrics := scrape(t, addr) - - if hits == 0 { - wantNoSeries(t, metrics, series) - - return - } - - wantMetric(t, metrics, series, float64(hits)) -} diff --git a/internal/proxy/observe_test.go b/internal/proxy/observe_test.go index 36a42bd..5aa8725 100644 --- a/internal/proxy/observe_test.go +++ b/internal/proxy/observe_test.go @@ -1,17 +1,13 @@ package proxy_test import ( - "bytes" - "io" "net/http" "net/netip" "reflect" - "sync/atomic" "testing" "time" "sneak.berlin/go/smallwebwaf/internal/bans" - "sneak.berlin/go/smallwebwaf/internal/proxy" "sneak.berlin/go/smallwebwaf/internal/requestlog" ) @@ -76,11 +72,6 @@ func TestObserveModeForwardsWhatEnforceModeRefuses(t *testing.T) { // Passed to the app, which answered it. line := s.get(sent.from, http.StatusOK, requestlog.ActionForward) wantWouldAction(t, line, sent.refusal) - - if line.UpstreamStatus != http.StatusOK { - t.Errorf("log line has upstream_status %d, want 200", - line.UpstreamStatus) - } } }) } @@ -133,62 +124,6 @@ func TestObserveModeMakesNoBanAndKeepsTheBansItHas(t *testing.T) { } } -func TestObserveModeKeepsTheSizeLimitsAndTheToken(t *testing.T) { - t.Parallel() - - const denied = "192.0.2.50" // in SWWAF_DENY_NETS - - var calls atomic.Int32 - - app := startApp(t, func(w http.ResponseWriter, _ *http.Request) { - calls.Add(1) - answerWithSize(w, 2*sizeLimit, true) - }) - addr, out := startProxy(t, app.URL, map[string]string{ - mode: observe, - trustedProxies: trustLocalhost, - denyNets: denied, - requestMaxBytes: sizeLimitSetting, - responseMaxBytes: sizeLimitSetting, - metricsToken: token, - }) - - // SWWAF_DENY_NETS would refuse each request; instead a size limit or - // the missing token does. - for i, tc := range []struct { - method, path string - body io.Reader - status int - action string - }{ - { - http.MethodPost, "/upload", bytes.NewReader(make([]byte, 2*sizeLimit)), - http.StatusRequestEntityTooLarge, requestlog.ActionTooLarge, - }, - { - http.MethodGet, "/download", http.NoBody, - http.StatusBadGateway, requestlog.ActionTooLarge, - }, - { - http.MethodGet, proxy.MetricsPath, http.NoBody, - http.StatusUnauthorized, requestlog.ActionAdmin, - }, - } { - req := newRequest(t, tc.method, addr, tc.path, tc.body) - req.Header.Set(forwardedFor, denied) - wantStatus(t, do(t, req), tc.status) - - line := out.requestLines(t, i+1)[i] - wantLine(t, line, tc.status, tc.action) - wantWouldAction(t, line, requestlog.ActionDenied) - } - - // The upload was refused before it reached the app. - if calls.Load() != 1 { - t.Errorf("the app was called %d times, want once", calls.Load()) - } -} - // wantWouldAction checks the request log line's would_action, and that a // line that should have none has no such field. func wantWouldAction(t *testing.T, line logLine, want string) { diff --git a/internal/proxy/passthrough_test.go b/internal/proxy/passthrough_test.go deleted file mode 100644 index abaf334..0000000 --- a/internal/proxy/passthrough_test.go +++ /dev/null @@ -1,556 +0,0 @@ -package proxy_test - -import ( - "bufio" - "bytes" - "errors" - "io" - "net/http" - "reflect" - "slices" - "strings" - "sync/atomic" - "testing" - "time" - - "sneak.berlin/go/smallwebwaf/internal/config" - "sneak.berlin/go/smallwebwaf/internal/proxy" - "sneak.berlin/go/smallwebwaf/internal/ratelimit" - "sneak.berlin/go/smallwebwaf/internal/requestlog" -) - -// A request target with an escaped slash and space in its path, and a -// query with a parameter ReverseProxy cannot parse. -const ( - rawPath = "/some%2Fpath/with%20space" - rawQuery = "b=2&a=1&bad=%zz;x" -) - -// chunkSize is the size of each part of a body a test sends in parts. -const chunkSize = 1 << 10 - -var errNotStreamed = errors.New("the first part never reached the app") - -// appSaw is what the app received. -type appSaw struct { - method string - target string - header http.Header - body []byte -} - -func TestPassesRequestAndAnswerUnchanged(t *testing.T) { - t.Parallel() - - requestBody := bytes.Repeat([]byte("request body "), 8000) - answerBody := bytes.Repeat([]byte("answer body "), 8000) - saw := make(chan appSaw, 1) - - app := startApp(t, func(w http.ResponseWriter, r *http.Request) { - body, _ := io.ReadAll(r.Body) - saw <- appSaw{r.Method, r.RequestURI, r.Header.Clone(), body} - - w.Header().Set("X-App", "yes") - w.Header().Add("Set-Cookie", "a=1") - w.Header().Add("Set-Cookie", "b=2") - w.WriteHeader(http.StatusTeapot) - _, _ = w.Write(answerBody) - }) - addr, out := startProxy(t, app.URL, nil) - - req := newRequest(t, http.MethodPatch, addr, rawPath+"?"+rawQuery, - bytes.NewReader(requestBody)) - req.Header.Add("X-Test", "one") - req.Header.Add("X-Test", "two") - req.Header.Set("User-Agent", "test-agent") - - got := do(t, req) - - wantAppSaw(t, <-saw, requestBody) - wantAnswer(t, got, answerBody) - - line := out.requestLine(t) - wantLine(t, line, http.StatusTeapot, requestlog.ActionForward) - wantRequestFields(t, line, addr, len(requestBody), len(answerBody)) -} - -func TestPassesThePathAsSent(t *testing.T) { - t.Parallel() - - for _, path := range []string{ - // An encoded slash and a character Go escapes: |, ^, and é sent - // as its UTF-8 bytes. - "/a%2Fb|c", "/feat%2Fx/file^1", "/x%2Fy/café", - // A path starting with //, which Go would send as a scheme and - // host if it were put in URL.Opaque. - "//a%2Fb", - } { - t.Run(path, func(t *testing.T) { - t.Parallel() - - saw := make(chan string, 1) - app := startApp(t, func(_ http.ResponseWriter, r *http.Request) { - saw <- r.RequestURI - }) - addr, out := startProxy(t, app.URL, nil) - - conn := dial(t, addr) - send(t, conn, "GET "+path+"?q=1 HTTP/1.1\r\nHost: "+appHost+"\r\n\r\n") - - got := readResponse(t, conn) - if got.status != http.StatusOK { - t.Fatalf("status %d, want %d", got.status, http.StatusOK) - } - - target := <-saw - if target != path+"?q=1" { - t.Errorf("app saw %s, want %s", target, path+"?q=1") - } - - line := out.requestLine(t) - if line.Path != path { - t.Errorf("log line has path %s, want %s", line.Path, path) - } - }) - } -} - -// wantAppSaw checks that the app received the test's request unchanged. -func wantAppSaw(t *testing.T, saw appSaw, body []byte) { - t.Helper() - - if saw.method != http.MethodPatch || saw.target != rawPath+"?"+rawQuery { - t.Errorf("app saw %s %s, want %s %s", saw.method, saw.target, - http.MethodPatch, rawPath+"?"+rawQuery) - } - - if !slices.Equal(saw.header.Values("X-Test"), []string{"one", "two"}) { - t.Errorf("app saw X-Test %q", saw.header.Values("X-Test")) - } - - if saw.header.Get("User-Agent") != "test-agent" { - t.Errorf("app saw User-Agent %q", saw.header.Get("User-Agent")) - } - - if !bytes.Equal(saw.body, body) { - t.Errorf("app saw a body of %d bytes, want the %d sent", - len(saw.body), len(body)) - } -} - -// wantAnswer checks that the client received the app's answer unchanged. -func wantAnswer(t *testing.T, got answer, body []byte) { - t.Helper() - - wantStatus(t, got, http.StatusTeapot) - - if got.header.Get("X-App") != "yes" { - t.Errorf("client got X-App %q", got.header.Get("X-App")) - } - - if !slices.Equal(got.header.Values("Set-Cookie"), []string{"a=1", "b=2"}) { - t.Errorf("client got Set-Cookie %q", got.header.Values("Set-Cookie")) - } - - if got.err != nil || !bytes.Equal(got.body, body) { - t.Errorf("client got %d bytes (%v), want the %d the app sent", - len(got.body), got.err, len(body)) - } -} - -// wantRequestFields checks the log line's fields about the request. Its -// time, its id and its timings are checked only for being there. -func wantRequestFields(t *testing.T, line logLine, host string, sent, received int) { - t.Helper() - - bytes := float64(sent + received) - - want := withTimings(line, requestlog.Line{ - Type: requestType, Time: line.Time, Instance: "app", - ClientIP: localhost, Method: http.MethodPatch, Scheme: plain, Host: host, - Path: rawPath, Query: rawQuery, Protocol: protocol, - Status: http.StatusTeapot, RequestBytes: int64(sent), - ResponseBytes: int64(received), UserAgent: "test-agent", - RequestID: line.RequestID, PeerIP: localhost, ClientGroup: localhost + "/32", - ContentLength: int64(sent), ResponseContentType: "text/plain; charset=utf-8", - UpstreamStatus: http.StatusTeapot, Action: requestlog.ActionForward, - Counts: ratelimit.Counts{ - Minute: 1, Hour: 1, Day: 1, - MinuteBytes: bytes, HourBytes: bytes, DayBytes: bytes, - }, - }) - if !reflect.DeepEqual(line.Line, want) { - t.Errorf("log line\n%+v\nwant\n%+v", line.Line, want) - } - - _, err := time.Parse(time.RFC3339, line.Time) - if err != nil || line.RequestID == "" || line.DurationTotal <= 0 || - line.DurationUpstreamTotal == nil || *line.DurationUpstreamTotal <= 0 { - t.Errorf("log line has time %q, request_id %q and durations %v and %v", - line.Time, line.RequestID, line.DurationTotal, - line.fields["duration_upstream_total"]) - } -} - -func TestAddsNoContentTypeTheAppDidNotSend(t *testing.T) { - t.Parallel() - - const typed = "text/plain" - - app := startApp(t, func(w http.ResponseWriter, r *http.Request) { - if r.URL.Path == "/typed" { - w.Header().Set("Content-Type", typed) - } else { - // The key with no value keeps Go's server from adding one. - w.Header()["Content-Type"] = nil - } - - _, _ = io.WriteString(w, "page") - }) - addr, _ := startProxy(t, app.URL, nil) - - got := get(t, addr, "/") - if got.header.Values("Content-Type") != nil { - t.Errorf("client got Content-Type %q, want none", - got.header.Get("Content-Type")) - } - - got = get(t, addr, "/typed") - if got.header.Get("Content-Type") != typed { - t.Errorf("client got Content-Type %q, want %q", - got.header.Get("Content-Type"), typed) - } -} - -func TestStreamsTheRequestBody(t *testing.T) { - t.Parallel() - - chunk := bytes.Repeat([]byte("x"), chunkSize) - firstArrived := make(chan struct{}) - - app := startApp(t, func(w http.ResponseWriter, r *http.Request) { - first := make([]byte, len(chunk)) - - _, err := io.ReadFull(r.Body, first) - if err != nil { - return - } - - close(firstArrived) - - rest, _ := io.ReadAll(r.Body) - _, _ = w.Write(rest) - }) - addr, _ := startProxy(t, app.URL, nil) - - body, writer := io.Pipe() - - go func() { - _, _ = writer.Write(chunk) - - select { - case <-firstArrived: - _, _ = writer.Write(chunk) - _ = writer.Close() - case <-time.After(waitLimit): - _ = writer.CloseWithError(errNotStreamed) - } - }() - - got := do(t, newRequest(t, http.MethodPost, addr, "/upload", body)) - if got.err != nil || !bytes.Equal(got.body, chunk) { - t.Errorf("app read %d bytes after the first part (%v), want %d", - len(got.body), got.err, len(chunk)) - } -} - -func TestStreamsTheAnswerBody(t *testing.T) { - t.Parallel() - - chunk := bytes.Repeat([]byte("y"), chunkSize) - firstArrived := make(chan struct{}) - - app := startApp(t, func(w http.ResponseWriter, _ *http.Request) { - _, _ = w.Write(chunk) - _ = http.NewResponseController(w).Flush() - - select { - case <-firstArrived: - _, _ = w.Write(chunk) - case <-time.After(waitLimit): - } - }) - addr, _ := startProxy(t, app.URL, nil) - - req := newRequest(t, http.MethodGet, addr, "/download", http.NoBody) - - res, err := newClient(t).Do(req) - if err != nil { - t.Fatalf("request: %v", err) - } - - first := make([]byte, len(chunk)) - _, err = io.ReadFull(res.Body, first) - - close(firstArrived) - - got := readAnswer(res) - if err != nil || got.err != nil || !bytes.Equal(got.body, chunk) { - t.Errorf("client read %d bytes after the first part (%v, %v), want %d", - len(got.body), err, got.err, len(chunk)) - } -} - -func TestUpgradedConnectionOutlastsTheTimeouts(t *testing.T) { - t.Parallel() - - app := startApp(t, echoAfterUpgrade) - addr, out := startProxy(t, app.URL, map[string]string{ - clientRequestTimeout: shortTimeoutSetting, - clientResponseTimeout: shortTimeoutSetting, - upstreamRequestTimeout: shortTimeoutSetting, - upstreamResponseTimeout: shortTimeoutSetting, - }) - - conn := dial(t, addr) - send(t, conn, "GET /socket HTTP/1.1\r\nHost: app\r\n"+ - "Connection: Upgrade\r\nUpgrade: websocket\r\n\r\n") - - reader := bufio.NewReader(conn) - - res, err := http.ReadResponse(reader, nil) - if err != nil { - t.Fatalf("read the answer to the upgrade: %v", err) - } - - answered := time.Now() - _ = res.Body.Close() - - if res.StatusCode != http.StatusSwitchingProtocols { - t.Fatalf("status %d, want %d", res.StatusCode, http.StatusSwitchingProtocols) - } - - // Every timeout started before the upgrade was answered, the response - // timeouts last, at the end of the request: wait until just past - // shortTimeout after the answer was read, then use the connection. - time.Sleep(time.Until(answered.Add(shortTimeout + 100*time.Millisecond))) - send(t, conn, "still here\n") - - echoed, err := reader.ReadString('\n') - if err != nil || echoed != "still here\n" { - t.Errorf("echo %q (%v), want %q", echoed, err, "still here\n") - } - - _ = conn.Close() - - wantLine(t, out.requestLine(t), http.StatusSwitchingProtocols, - requestlog.ActionForward) -} - -// echoAfterUpgrade is an app that switches protocols on request, and then -// sends back each line it receives. -func echoAfterUpgrade(w http.ResponseWriter, r *http.Request) { - if r.Header.Get("Upgrade") != "websocket" { - http.Error(w, "not an upgrade", http.StatusBadRequest) - - return - } - - conn, buffered, err := http.NewResponseController(w).Hijack() - if err != nil { - return - } - - defer func() { - _ = conn.Close() - }() - - _, _ = buffered.WriteString("HTTP/1.1 101 Switching Protocols\r\n" + - "Connection: Upgrade\r\nUpgrade: websocket\r\n\r\n") - _ = buffered.Flush() - - for { - line, err := buffered.ReadString('\n') - if err != nil { - return - } - - _, _ = buffered.WriteString(line) - _ = buffered.Flush() - } -} - -func TestAnswers502WhenTheAppSwitchesToAnotherProtocol(t *testing.T) { - t.Parallel() - - // answerAfterUpgrade switches to websocket whatever the client asked - // for, and ReverseProxy passes on a switch only to the protocol the - // client asked for, here "websocket, foo". - app := startApp(t, answerAfterUpgrade) - addr, out := startProxy(t, app.URL, nil) - - conn := dial(t, addr) - send(t, conn, "GET /ws HTTP/1.1\r\nHost: app\r\n"+ - "Connection: Upgrade\r\nUpgrade: websocket, foo\r\n\r\n") - - wantStatus(t, readResponse(t, conn), http.StatusBadGateway) - wantLine(t, out.requestLine(t), http.StatusBadGateway, requestlog.ActionUpstreamError) -} - -func TestServerHasTheDefaultLimits(t *testing.T) { - t.Parallel() - - cfg, err := config.FromEnvironment(func(string) (string, bool) { return "", false }) - if err != nil { - t.Fatalf("default settings: %v", err) - } - - server := proxy.New(proxy.Params{ - Config: cfg, - RequestLog: io.Discard, - ProcessLog: requestlog.NewProcessLogger(io.Discard, cfg.InstanceName, cfg.LogLevel), - }) - - if server.Addr != ":8080" || server.MaxHeaderBytes != 28<<10 || - server.IdleTimeout != 2*time.Minute || server.ReadHeaderTimeout != time.Minute { - t.Errorf("server listens on %q with header limit %d, idle time %s and "+ - "header timeout %s", server.Addr, server.MaxHeaderBytes, - server.IdleTimeout, server.ReadHeaderTimeout) - } -} - -func TestRefusesHeadersOverTheLimit(t *testing.T) { - t.Parallel() - - for _, tc := range []struct { - name string - env map[string]string - limit int - }{ - {"by default", nil, 32 << 10}, - {"as set", map[string]string{clientHeaderMaxBytes: "8K"}, 8 << 10}, - } { - t.Run(tc.name, func(t *testing.T) { - t.Parallel() - - var calls atomic.Int32 - - app := startApp(t, func(http.ResponseWriter, *http.Request) { - calls.Add(1) - }) - addr, _ := startProxy(t, app.URL, tc.env) - - // size counts every byte of the request: the request line, - // the headers and the blank line that ends them. - const ( - start = "GET / HTTP/1.1\r\nHost: app\r\nX-Large: " - end = "\r\n\r\n" - ) - - for _, sent := range []struct{ size, want int }{ - {tc.limit, http.StatusOK}, - {tc.limit + 1, http.StatusRequestHeaderFieldsTooLarge}, - } { - conn := dial(t, addr) - send(t, conn, - start+strings.Repeat("a", sent.size-len(start)-len(end))+end) - wantStatus(t, readResponse(t, conn), sent.want) - } - - if calls.Load() != 1 { - t.Errorf("the app was called %d times, want once", calls.Load()) - } - }) - } -} - -func TestAnswers502WhenTheAppCannotBeReached(t *testing.T) { - t.Parallel() - - // No test can listen on port 1: listening on port 0 gets one from 32768 up. - addr, out := startProxy(t, "http://"+localhost+":1", nil) - - wantStatus(t, get(t, addr, "/"), http.StatusBadGateway) - - line := out.requestLine(t) - wantLine(t, line, http.StatusBadGateway, requestlog.ActionUpstreamError) - - // There never was a connection to the app, nor an answer from it. - wantTimings(t, line, "duration_total", "duration_checks", - "duration_upstream_total") - - logged := slices.ContainsFunc(out.lines(t), func(line map[string]any) bool { - return line["type"] == "process" && line["msg"] == "request to the app failed" - }) - if !logged { - t.Errorf("no process line says the request to the app failed") - } -} - -func TestLogLevelHoldsBackNoRequestLine(t *testing.T) { - t.Parallel() - - // At error the warning that the request to the app failed is held back, - // and is written before the answer is. - addr, out := startProxy(t, "http://"+localhost+":1", map[string]string{ - "SWWAF_LOG_LEVEL": "error", - }) - - wantStatus(t, get(t, addr, "/"), http.StatusBadGateway) - wantLine(t, out.requestLine(t), http.StatusBadGateway, requestlog.ActionUpstreamError) - - for _, line := range out.lines(t) { - if line["type"] == "process" { - t.Errorf("process line %v, want none at error", line) - } - } -} - -func TestLogsAnAnswerThatBrokeOff(t *testing.T) { - t.Parallel() - - app := startApp(t, func(w http.ResponseWriter, _ *http.Request) { - _, _ = io.WriteString(w, "the first part") - _ = http.NewResponseController(w).Flush() - - panic(http.ErrAbortHandler) // drops the connection mid-answer - }) - addr, out := startProxy(t, app.URL, nil) - - got := get(t, addr, "/") - if string(got.body) != "the first part" || !errors.Is(got.err, io.ErrUnexpectedEOF) { - t.Errorf("client read %q (%v), want the first part cut off", got.body, got.err) - } - - wantLine(t, out.requestLine(t), http.StatusOK, requestlog.ActionUpstreamError) -} - -func TestLogsAClientThatWentAway(t *testing.T) { - t.Parallel() - - arrived := make(chan struct{}) - - app := startApp(t, func(_ http.ResponseWriter, r *http.Request) { - close(arrived) - <-r.Context().Done() - }) - addr, out := startProxy(t, app.URL, nil) - - conn := dial(t, addr) - send(t, conn, "GET /slow HTTP/1.1\r\nHost: app\r\n\r\n") - - select { - case <-arrived: - case <-time.After(waitLimit): - t.Fatal("the request never reached the app") - } - - _ = conn.Close() - - line := out.requestLine(t) - if !line.Aborted || line.Status != 0 || line.Action != requestlog.ActionForward { - t.Errorf("log line has aborted %v, status %d and action %q, "+ - "want true, 0 and %q", - line.Aborted, line.Status, line.Action, requestlog.ActionForward) - } -} diff --git a/internal/proxy/proxy.go b/internal/proxy/proxy.go index 5eef404..2d0cb6d 100644 --- a/internal/proxy/proxy.go +++ b/internal/proxy/proxy.go @@ -1,16 +1,15 @@ -// Package proxy runs the checks on each request and writes its request log -// line. New passes each request it lets through to the app and the app's -// answer back, unchanged, within the size and time limits. NewServeWAF instead -// answers rproxy's question about each request, and takes rproxy's report once -// the visitor's answer has ended, which writes the request a second log line. +// Package proxy is servewaf's server. It runs the checks on the request each of +// rproxy's questions carries, and answers rproxy what to do with that request. It +// takes rproxy's report once the visitor's answer has ended, which counts the +// request's bytes and writes its last log line. It answers servewaf's own +// endpoints: the health check, the metrics, the bans and the clients. It passes +// nothing to an app: internal/rproxy does that. package proxy import ( "io" - "log" "log/slog" "net/http" - "strings" "time" lru "github.com/hashicorp/golang-lru/v2" @@ -27,18 +26,12 @@ import ( "sneak.berlin/go/smallwebwaf/internal/waf" ) -// How smallwebwaf keeps connections to the app open between requests. -const ( - appIdleConns = 100 - appIdleConnTimeout = 90 * time.Second -) - -// adminPrefix starts the path of every request for smallwebwaf itself, -// which never reaches the app. +// adminPrefix starts the path of each of servewaf's own endpoints. The app never +// sees a path under it. const adminPrefix = "/_smallwebwaf/" -// HealthPath is smallwebwaf's health endpoint, which the container's -// health check asks. +// HealthPath is the health endpoint of servewaf and of rproxy, which the +// container's health check asks. const HealthPath = "/_smallwebwaf/healthz" // MetricsPath is where the metrics are, for a request that carries @@ -49,15 +42,15 @@ const MetricsPath = "/_smallwebwaf/metrics" // a client's address, lifts them, with SWWAF_ADMIN_TOKEN. const BansPath = "/_smallwebwaf/bans" -// ClientsPath is where an admin asks what smallwebwaf knows of a client, -// by the client's address after it, with SWWAF_ADMIN_TOKEN. +// ClientsPath is where an admin asks what servewaf knows of a client, by the +// client's address after it, with SWWAF_ADMIN_TOKEN. const ClientsPath = "/_smallwebwaf/clients/" -// Params are what New needs. +// Params are what NewServeWAF needs. type Params struct { Config *config.Config - // RequestLog receives one JSON line per request, and from servewaf a second - // when the request's report comes. + // RequestLog receives two JSON lines per question, one at its decision and + // one when its report comes, and one per request to servewaf's own endpoints. RequestLog io.Writer // ProcessLog receives the process's own messages. ProcessLog *slog.Logger @@ -77,7 +70,7 @@ type Params struct { // Rules are the rule files' rules, which each request is checked // against. Rules *rules.Files - // Alerts receive the alert for each ban the proxy makes or makes + // Alerts receive the alert for each ban servewaf makes or makes // permanent, for each count over an anomaly threshold, for each request // whose client a blocklist, the CrowdSec decision list, a DNSBL zone or // AbuseIPDB lists, for each request the Core Rule Set scores at or over @@ -87,11 +80,10 @@ type Params struct { Alerts *alerts.Queue } -// Server is the server smallwebwaf runs, with the parts of the proxy -// whose state the state files keep, the lookup database, nil unless -// SWWAF_LOOKUP_SOURCE is file, the lists fetched from URLs, which its Run -// fetches, the DNSBL zones' verdicts, AbuseIPDB's scores and checks -// spent, and the metrics. +// Server is the server servewaf runs, with the parts whose state the state +// files keep, the lookup database, nil unless SWWAF_LOOKUP_SOURCE is file, the +// lists fetched from URLs, which its Run fetches, the DNSBL zones' verdicts, +// AbuseIPDB's scores and checks spent, and the metrics. type Server struct { *http.Server @@ -106,41 +98,24 @@ type Server struct { Metrics *metrics.Metrics } -// New returns the server smallwebwaf runs: each request it reads passes -// through the proxy. Go's server itself refuses a request line and -// headers over SWWAF_CLIENT_REQUEST_HEADER_MAX_BYTES, with 431, closes a -// connection idle for SWWAF_CLIENT_IDLE_TIMEOUT, and applies -// SWWAF_CLIENT_REQUEST_TIMEOUT while the headers arrive; the proxy -// applies the timeouts and size limits from then on. -func New(params Params) *Server { - h := newHandler(params) - - return h.server(&http.Server{ - Handler: h, - ReadHeaderTimeout: params.Config.ClientRequestTimeout, - // Off is an IdleTimeout of 0, which Go's server replaces with - // ReadTimeout: no limit, as long as ReadTimeout stays unset. - IdleTimeout: params.Config.ClientIdleTimeout, - // Go's server reads 4 KiB past MaxHeaderBytes before it - // refuses, so the limit a client meets is the setting. - MaxHeaderBytes: int(params.Config.ClientRequestHeaderMaxBytes - 4<<10), - }) -} - // newHandler returns the handler with what every request shares, as the // settings in params make it. func newHandler(params Params) *handler { - errorLog := slog.NewLogLogger(params.ProcessLog.Handler(), slog.LevelWarn) m := metrics.New(params.Config.MetricsTopN, params.Config.InstanceName) lists, dnsbl, abuseIPDB := newReputation(params, m) + + awaitingReport, err := lru.New[string, *decision](maxAwaitingReport) + if err != nil { + panic(err) // New fails only for a size below one + } + h := &handler{ - config: params.Config, - requestLog: params.RequestLog, - processLog: params.ProcessLog, - errorLog: errorLog, - transport: newTransport(), - now: params.Now, - metrics: m, + config: params.Config, + requestLog: params.RequestLog, + processLog: params.ProcessLog, + now: params.Now, + metrics: m, + awaitingReport: awaitingReport, limiter: ratelimit.New(ratelimit.Limits{ PerMinute: params.Config.RateLimitPerMinute, PerHour: params.Config.RateLimitPerHour, @@ -232,7 +207,10 @@ func newReputation( // newCoreRuleSet returns the Core Rule Set at SWWAF_WAF_PARANOIA_LEVEL, // without the rules SWWAF_WAF_DISABLED_RULES switches off, reading bodies -// up to SWWAF_WAF_BODY_LIMIT, or nil while SWWAF_WAF_MODE is off. +// up to SWWAF_WAF_BODY_LIMIT, or nil while SWWAF_WAF_MODE is off. It reads at +// most QuestionBodyMaxBytes less one byte: a question's body of +// QuestionBodyMaxBytes may be cut short, and the Core Rule Set does not read a +// JSON or XML body over its limit. func newCoreRuleSet(cfg *config.Config) *waf.CoreRuleSet { if cfg.WAFMode == config.WAFModeOff { return nil @@ -240,26 +218,24 @@ func newCoreRuleSet(cfg *config.Config) *waf.CoreRuleSet { coreRuleSet, err := waf.New(waf.Params{ ParanoiaLevel: cfg.WAFParanoiaLevel, DisabledRules: cfg.WAFDisabledRules, - BodyLimit: cfg.WAFBodyLimit, + BodyLimit: min(cfg.WAFBodyLimit, QuestionBodyMaxBytes-1), }) if err != nil { // The Core Rule Set is built in, and the settings cannot break it: - // the paranoia level is from 1 to 4, the body limit at most 1G, and - // the id of no rule switches nothing off. + // the paranoia level is from 1 to 4, and the id of no rule switches + // nothing off. panic(err) } return coreRuleSet } -// handler is the proxy. It holds what every request shares; what belongs -// to one request is in a request. +// handler is servewaf's handler. It holds what every request shares; what +// belongs to one request is in a request. type handler struct { config *config.Config requestLog io.Writer processLog *slog.Logger - errorLog *log.Logger - transport http.RoundTripper now func() time.Time metrics *metrics.Metrics limiter *ratelimit.Limiter @@ -273,88 +249,7 @@ type handler struct { rules *rules.Files coreRuleSet *waf.CoreRuleSet alerts *alerts.Queue - // awaitingReport holds, in servewaf, the decision on each answered - // question by its id until its report comes, and nil for a question still - // being handled. + // awaitingReport holds the decision on each answered question by its id + // until its report comes, and nil for a question still being handled. awaitingReport *lru.Cache[string, *decision] } - -// newTransport returns what carries requests to the app. It never goes -// through a proxy named in the environment, and leaves the app's answers -// compressed or not as the app sent them. -func newTransport() *http.Transport { - return &http.Transport{ - MaxIdleConns: appIdleConns, - MaxIdleConnsPerHost: appIdleConns, - IdleConnTimeout: appIdleConnTimeout, - DisableCompression: true, - } -} - -// ServeHTTP handles one request: it works out the client, runs the -// checks, passes the request to the app and the answer back within the -// limits, or answers it itself if it is for smallwebwaf, and writes the -// request's log line. -func (h *handler) ServeHTTP(w http.ResponseWriter, r *http.Request) { - rq := h.newRequest(w, r) - defer rq.finish() - - // The health endpoint is answered at once, before any check, so that - // a health checker is never refused. It does not ask the app. - if r.Method == http.MethodGet && r.URL.Path == HealthPath { - rq.answerHealth() - - return - } - - // Once the request has ended, before its log line is written. The - // last deferred runs first: countRefusal before addToHistory, so that - // a broken error burst is in the client's history. - defer rq.addToHistory() - defer rq.countAnomalies() - defer rq.countRefusal() - - refused := rq.check(r.Context()) - rq.checked = time.Now() - - if refused != nil { - rq.answer(*refused) - - return - } - - // A request for smallwebwaf itself is answered where another would be - // passed to the app, so that it goes through every check first. - if strings.HasPrefix(r.URL.Path, adminPrefix) { - rq.answerAdmin() - - return - } - - // Once the response has ended, before the request is added to its - // client's history. Deferred, since ReverseProxy panics to end a - // response it cannot finish. - defer rq.countBytes() - - rq.forward(r.Context()) -} - -// server returns the Server of h, which listens at SWWAF_LISTEN_ADDR and -// logs through h, with what httpServer sets. -func (h *handler) server(httpServer *http.Server) *Server { - httpServer.Addr = h.config.ListenAddr - httpServer.ErrorLog = h.errorLog - - return &Server{ - Server: httpServer, - Ledger: h.ledger, - Limiter: h.limiter, - GeoJS: h.geojs, - Anomalies: h.anomalies, - LookupFile: h.lookupFile, - Lists: h.lists, - DNSBL: h.dnsbl, - AbuseIPDB: h.abuseIPDB, - Metrics: h.metrics, - } -} diff --git a/internal/proxy/proxy_test.go b/internal/proxy/proxy_test.go index 0613933..773794f 100644 --- a/internal/proxy/proxy_test.go +++ b/internal/proxy/proxy_test.go @@ -5,6 +5,7 @@ import ( "bytes" "encoding/json" "io" + "log/slog" "maps" "net" "net/http" @@ -19,73 +20,68 @@ import ( "sneak.berlin/go/smallwebwaf/internal/lookup" "sneak.berlin/go/smallwebwaf/internal/proxy" "sneak.berlin/go/smallwebwaf/internal/requestlog" + "sneak.berlin/go/smallwebwaf/internal/rproxy" "sneak.berlin/go/smallwebwaf/internal/rules" ) const ( - // shortTimeout is what a test sets a timeout to, to see it run out. - // It starts before the test has set up its case, such as an upgrade - // or the app's buffers filling, so it is as long as the hold-up of the - // test process that wantTimedOut allows: a shorter one can run out - // first on a busy host. - shortTimeout = waitLimit / 2 - // longTimeoutSetting is a timeout that does not run out in a test. - longTimeoutSetting = "1m" // waitLimit bounds how long a test waits for what should happen. waitLimit = 10 * time.Second // pollInterval is how often a test looks for a log line. pollInterval = 10 * time.Millisecond - // localhost is where every test server listens, and so the address - // smallwebwaf sees each test's requests come from. + // localhost is where every test server listens, and so the address rproxy + // sees each test's requests come from. localhost = "127.0.0.1" // requestType is the type that marks a request log line. requestType = "request" - // protocol is the protocol of every test's requests. + // protocol is the protocol of every question. protocol = "HTTP/1.1" + // trustLocalhost trusts the address every test connects from, and a + // network for proxies in front of it. + trustLocalhost = localhost + "/32,10.0.0.0/8" + // appHost is the host every test asks for. + appHost = "app.example" + // client is the client's address, as a proxy names it. + client = "203.0.113.9" + // forwardedFor is the header that lists the client and its proxies, + // and forwardedProto the one that gives the scheme the client used. + forwardedFor = "X-Forwarded-For" + forwardedProto = "X-Forwarded-Proto" + // secure is the scheme a client reached traefik with, and plain the one + // rproxy serves. + secure = "https" + plain = "http" ) -// shortTimeoutSetting is shortTimeout as a setting's value. -// -//nolint:gochecknoglobals // a constant cannot call String -var shortTimeoutSetting = shortTimeout.String() - // The settings the tests set. const ( - clientRequestTimeout = "SWWAF_CLIENT_REQUEST_TIMEOUT" - clientHeaderMaxBytes = "SWWAF_CLIENT_REQUEST_HEADER_MAX_BYTES" - clientIdleTimeout = "SWWAF_CLIENT_IDLE_TIMEOUT" - clientResponseTimeout = "SWWAF_CLIENT_RESPONSE_TIMEOUT" - upstreamRequestTimeout = "SWWAF_UPSTREAM_REQUEST_TIMEOUT" - upstreamResponseTimeout = "SWWAF_UPSTREAM_RESPONSE_TIMEOUT" - mode = "SWWAF_MODE" - requestMaxBytes = "SWWAF_REQUEST_MAX_BYTES" - responseMaxBytes = "SWWAF_RESPONSE_MAX_BYTES" - trustedProxies = "SWWAF_TRUSTED_PROXIES" - ipv6GroupPrefix = "SWWAF_IPV6_GROUP_PREFIX" - maxTrackedClients = "SWWAF_MAX_TRACKED_CLIENTS" - allowNets = "SWWAF_ALLOW_NETS" - rateLimitExemptNets = "SWWAF_RATE_LIMIT_EXEMPT_NETS" - denyNets = "SWWAF_DENY_NETS" - rateLimitPerMinute = "SWWAF_RATE_LIMIT_PER_MINUTE" - rateLimitPerDay = "SWWAF_RATE_LIMIT_PER_DAY" - rateLimitExemptPaths = "SWWAF_RATE_LIMIT_EXEMPT_PATHS" - lookupSource = "SWWAF_LOOKUP_SOURCE" - lookupDBPath = "SWWAF_LOOKUP_DB_PATH" - lookupTimeout = "SWWAF_LOOKUP_TIMEOUT" - addLookupHeaders = "SWWAF_ADD_LOOKUP_HEADERS" - deniedCountries = "SWWAF_DENIED_COUNTRIES" - allowedCountries = "SWWAF_EXCLUSIVELY_ALLOWED_COUNTRIES" - banResponse = "SWWAF_BAN_RESPONSE" - limitBanDuration = "SWWAF_LIMIT_BAN_DURATION" - limitBanRepeatWindow = "SWWAF_LIMIT_BAN_REPEAT_WINDOW" - maxBanDuration = "SWWAF_MAX_BAN_DURATION" - maxBans = "SWWAF_MAX_BANS" - banScopeV4Prefix = "SWWAF_BAN_SCOPE_V4_PREFIX" - instanceName = "SWWAF_INSTANCE_NAME" - logRequestHeaders = "SWWAF_LOG_REQUEST_HEADERS" - attackBanDuration = "SWWAF_ATTACK_BAN_DURATION" - rulesDir = "SWWAF_RULES_DIR" - wafMode = "SWWAF_WAF_MODE" + mode = "SWWAF_MODE" + trustedProxies = "SWRP_TRUSTED_PROXIES" + ipv6GroupPrefix = "SWWAF_IPV6_GROUP_PREFIX" + maxTrackedClients = "SWWAF_MAX_TRACKED_CLIENTS" + allowNets = "SWWAF_ALLOW_NETS" + rateLimitExemptNets = "SWWAF_RATE_LIMIT_EXEMPT_NETS" + denyNets = "SWWAF_DENY_NETS" + rateLimitPerMinute = "SWWAF_RATE_LIMIT_PER_MINUTE" + rateLimitPerDay = "SWWAF_RATE_LIMIT_PER_DAY" + rateLimitExemptPaths = "SWWAF_RATE_LIMIT_EXEMPT_PATHS" + lookupSource = "SWWAF_LOOKUP_SOURCE" + lookupDBPath = "SWWAF_LOOKUP_DB_PATH" + lookupTimeout = "SWWAF_LOOKUP_TIMEOUT" + addLookupHeaders = "SWWAF_ADD_LOOKUP_HEADERS" + deniedCountries = "SWWAF_DENIED_COUNTRIES" + allowedCountries = "SWWAF_EXCLUSIVELY_ALLOWED_COUNTRIES" + banResponse = "SWWAF_BAN_RESPONSE" + limitBanDuration = "SWWAF_LIMIT_BAN_DURATION" + limitBanRepeatWindow = "SWWAF_LIMIT_BAN_REPEAT_WINDOW" + maxBanDuration = "SWWAF_MAX_BAN_DURATION" + maxBans = "SWWAF_MAX_BANS" + banScopeV4Prefix = "SWWAF_BAN_SCOPE_V4_PREFIX" + instanceName = "SWWAF_INSTANCE_NAME" + logRequestHeaders = "SWWAF_LOG_REQUEST_HEADERS" + attackBanDuration = "SWWAF_ATTACK_BAN_DURATION" + rulesDir = "SWWAF_RULES_DIR" + wafMode = "SWWAF_WAF_MODE" ) // off is the value that switches a setting off. @@ -179,6 +175,73 @@ func (o *output) requestLine(t *testing.T) logLine { return o.requestLines(t, 1)[0] } +// endedLines waits for count requests that went through rproxy, or to servewaf's +// own endpoints, to have ended, and returns the last log line of each, in the +// order servewaf took them. A question's last line is the one its report writes. +func (o *output) endedLines(t *testing.T, count int) []logLine { + t.Helper() + + deadline := time.Now().Add(waitLimit) + for time.Now().Before(deadline) { + found := o.lastLines(t) + + ended := len(found) >= count + for _, line := range found[:min(count, len(found))] { + if line.ServeWAFLine == requestlog.ServeWAFLineDecision { + ended = false + } + } + + if ended { + return found + } + + time.Sleep(pollInterval) + } + + t.Fatalf("fewer than %d requests have ended after %s", count, waitLimit) + + return nil +} + +// lastLines returns the last request log line written so far of each request, +// in the order servewaf took them. +func (o *output) lastLines(t *testing.T) []logLine { + t.Helper() + + var found []logLine + + places := map[string]int{} + + for _, fields := range o.lines(t) { + if fields["type"] != requestType { + continue + } + + line := decodeLine(t, fields) + + place, seen := places[line.RequestID] + if seen { + found[place] = line + + continue + } + + places[line.RequestID] = len(found) + found = append(found, line) + } + + return found +} + +// endedLine waits for a test's one request to have ended, and returns its last +// log line. +func (o *output) endedLine(t *testing.T) logLine { + t.Helper() + + return o.endedLines(t, 1)[0] +} + // decodeLine reads a request log line's fields into a logLine. func decodeLine(t *testing.T, fields map[string]any) logLine { t.Helper() @@ -198,7 +261,7 @@ func decodeLine(t *testing.T, fields map[string]any) logLine { return line } -// startApp starts app as the app smallwebwaf passes requests to. +// startApp starts app as the app rproxy passes requests to. func startApp(t *testing.T, app http.HandlerFunc) *httptest.Server { t.Helper() @@ -208,9 +271,9 @@ func startApp(t *testing.T, app http.HandlerFunc) *httptest.Server { return server } -// startProxy starts smallwebwaf in front of the app at appURL, with the -// settings in env on top of the defaults, and returns where it listens and -// what it writes. +// startProxy starts servewaf, and rproxy in front of the app at appURL, each with +// the settings in env on top of the defaults, and returns where rproxy listens +// and what servewaf writes. func startProxy(t *testing.T, appURL string, env map[string]string) (string, *output) { t.Helper() @@ -230,10 +293,10 @@ func startProxyWithGeoJS( } // startProxyWithClock is startProxyWithGeoJS with requests counted and -// bans made by the time now tells, and returns the server as well. Unless -// env sets SWWAF_RULES_DIR, it is an empty directory, of no rules, and -// unless it sets SWWAF_INSTANCE_NAME, that is app, the label instance of -// every metric. +// bans made by the time now tells, and returns servewaf's server as well, whose +// Addr is where servewaf listens. Unless env sets SWWAF_RULES_DIR, it is an empty +// directory, of no rules, and unless it sets SWWAF_INSTANCE_NAME, that is app, +// the label instance of every metric. func startProxyWithClock( t *testing.T, appURL, geojsURL string, now func() time.Time, env map[string]string, @@ -246,14 +309,70 @@ func startProxyWithClock( } // startProxyWithAlerts is startProxyWithClock, and returns the queue of -// the alerts the proxy raises as well, as newProxy makes them. +// the alerts servewaf raises as well, as newServeWAF makes them. func startProxyWithAlerts( t *testing.T, appURL, geojsURL string, now func() time.Time, env map[string]string, ) (string, *output, *proxy.Server, *alerts.Queue) { t.Helper() - server, out, alertQueue := newProxy(t, appURL, geojsURL, now, env) + server, out, alertQueue := newServeWAF(t, geojsURL, now, env) + serve(t, server) + + return startRProxy(t, server.Addr, appURL, env), out, server, alertQueue +} + +// serve serves server on a port of its own until the test ends, and sets its +// Addr to where it listens. +func serve(t *testing.T, server *proxy.Server) { + t.Helper() + + listener, err := (&net.ListenConfig{}).Listen(t.Context(), "tcp", localhost+":0") + if err != nil { + t.Fatalf("listen: %v", err) + } + + server.Addr = listener.Addr().String() + + go func() { + _ = server.Serve(listener) + }() + + t.Cleanup(func() { + _ = server.Close() + }) +} + +// startRProxy starts rproxy in front of the app at appURL, asking the servewaf +// at serveWAFAddr, with the settings in env on top of the defaults, and returns +// where it listens. What it writes is dropped. SWRP_TIMEOUT_MS is a minute unless +// env sets it, so that on a busy host SWRP_DEFAULT_DISPOSITION does not decide. +func startRProxy( + t *testing.T, serveWAFAddr, appURL string, env map[string]string, +) string { + t.Helper() + + settings := map[string]string{ + "SWRP_WAF_BACKEND": "http://" + serveWAFAddr, + "SWRP_UPSTREAM_URL": appURL, + "SWRP_TIMEOUT_MS": "60000", + } + maps.Copy(settings, env) + + cfg, err := config.RProxyFromEnvironment(func(name string) (string, bool) { + value, ok := settings[name] + + return value, ok + }) + if err != nil { + t.Fatalf("settings %v: %v", settings, err) + } + + server := rproxy.New(rproxy.Params{ + Config: cfg, + RequestLog: io.Discard, + ProcessLog: requestlog.NewProcessLogger(io.Discard, "rproxy", slog.LevelInfo), + }) listener, err := (&net.ListenConfig{}).Listen(t.Context(), "tcp", localhost+":0") if err != nil { @@ -268,38 +387,35 @@ func startProxyWithAlerts( _ = server.Close() }) - return listener.Addr().String(), out, server, alertQueue + return listener.Addr().String() } -// newProxy makes the server startProxyWithClock starts, without starting -// it, and returns it, what it writes, and the queue of the alerts the -// proxy raises, as the settings in env make it. No alert is sent from the -// queue: they wait in it, for the test to look at. With no geojsURL, there -// is no stand-in for GeoJS to look clients up at, and SWWAF_LOOKUP_SOURCE -// is off unless env sets it. While it is file, the lookup database -// SWWAF_LOOKUP_DB_PATH names is read. Clients are checked with AbuseIPDB -// at abuseIPDBURL while env sets SWWAF_ABUSEIPDB_KEY. SWWAF_WAF_MODE is off -// unless env sets it, so that only the tests of the Core Rule Set have -// their requests inspected by it. -func newProxy( - t *testing.T, appURL, geojsURL string, now func() time.Time, - env map[string]string, +// newServeWAF makes servewaf's server, without starting it, and returns it, what +// it writes, and the queue of the alerts it raises, as the settings in env make +// it. No alert is sent from the queue: they wait in it, for the test to look at. +// With no geojsURL, there is no stand-in for GeoJS to look clients up at, and +// SWWAF_LOOKUP_SOURCE is off unless env sets it. While it is file, the lookup +// database SWWAF_LOOKUP_DB_PATH names is read. Clients are checked with AbuseIPDB +// at abuseIPDBURL while env sets SWWAF_ABUSEIPDB_KEY. SWWAF_WAF_MODE is off unless +// env sets it, so that only the tests of the Core Rule Set have their requests +// inspected by it. +func newServeWAF( + t *testing.T, geojsURL string, now func() time.Time, env map[string]string, ) (*proxy.Server, *output, *alerts.Queue) { t.Helper() - return newServerWith(t, proxy.New, appURL, geojsURL, now, env) + return newServeWAFWith(t, proxy.NewServeWAF, geojsURL, now, env) } -// newServerWith is newProxy for the server build makes. -func newServerWith( +// newServeWAFWith is newServeWAF for the server build makes. +func newServeWAFWith( t *testing.T, build func(proxy.Params) *proxy.Server, - appURL, geojsURL string, now func() time.Time, env map[string]string, + geojsURL string, now func() time.Time, env map[string]string, ) (*proxy.Server, *output, *alerts.Queue) { t.Helper() settings := map[string]string{ - "SWWAF_UPSTREAM_URL": appURL, rulesDir: t.TempDir(), instanceName: "app", - wafMode: off, + rulesDir: t.TempDir(), instanceName: "app", wafMode: off, } if geojsURL == "" { settings[lookupSource] = off @@ -391,7 +507,7 @@ func readAnswer(res *http.Response) answer { return answer{status: res.StatusCode, header: res.Header, body: body, err: err} } -// newRequest makes a request for path to smallwebwaf at addr. +// newRequest makes a request for path to addr, where rproxy or servewaf listens. func newRequest(t *testing.T, method, addr, path string, body io.Reader) *http.Request { t.Helper() @@ -415,15 +531,15 @@ func do(t *testing.T, req *http.Request) answer { return readAnswer(res) } -// get sends a GET request for path to smallwebwaf at addr. +// get sends a GET request for path to addr. func get(t *testing.T, addr, path string) answer { t.Helper() return do(t, newRequest(t, http.MethodGet, addr, path, http.NoBody)) } -// dial opens a connection to smallwebwaf at addr, for requests the HTTP -// client cannot make, such as one that stops sending halfway. +// dial opens a connection to addr, for requests the HTTP client cannot make, +// such as one that stops sending halfway. func dial(t *testing.T, addr string) net.Conn { t.Helper() @@ -484,14 +600,3 @@ func wantStatus(t *testing.T, got answer, status int) { t.Errorf("status %d, want %d", got.status, status) } } - -// wantTimedOut checks that what began at start ended once shortTimeout -// had run out, and not much later. -func wantTimedOut(t *testing.T, start time.Time) { - t.Helper() - - took := time.Since(start) - if took < shortTimeout || took > shortTimeout+waitLimit/2 { - t.Errorf("took %s, want %s", took, shortTimeout) - } -} diff --git a/internal/proxy/ratelimits_test.go b/internal/proxy/ratelimits_test.go index 1b55ffc..b2c3cda 100644 --- a/internal/proxy/ratelimits_test.go +++ b/internal/proxy/ratelimits_test.go @@ -51,7 +51,7 @@ func TestRateLimitRefusesBeforeTheApp(t *testing.T) { req.Header.Set(forwardedFor, sent.client) wantStatus(t, do(t, req), sent.want) - line := out.requestLines(t, i+1)[i] + line := out.endedLines(t, i+1)[i] if line.ClientIP != sent.logged { t.Errorf("log line has client_ip %q, want %q", line.ClientIP, sent.logged) } diff --git a/internal/proxy/reputation_test.go b/internal/proxy/reputation_test.go index d296a89..dae83bc 100644 --- a/internal/proxy/reputation_test.go +++ b/internal/proxy/reputation_test.go @@ -272,6 +272,7 @@ func TestEachBlocklistThatListsAClientRaisesAnAlertOncePerCooldownAndIsCounted( Instance: alertInstance, Time: clk.Now(), Event: alerts.EventReputationHit, + Host: appHost, Client: netip.MustParseAddr(fromDE), Netblock: netip.MustParsePrefix(fromDE + "/32"), ASN: asnDE, @@ -289,7 +290,7 @@ func TestEachBlocklistThatListsAClientRaisesAnAlertOncePerCooldownAndIsCounted( // Each list's hits, none of its fetches failed, and when its copy was // fetched, 0 for the one without. - metrics := s.scrape(unplaced) + metrics := s.scrape() fetched := float64(listsFetched().Unix()) for listURL, want := range map[string]struct{ hits, fetched float64 }{ @@ -513,6 +514,7 @@ func TestEachZoneThatListsAClientRaisesAnAlertOncePerCooldownAndIsCounted( Instance: alertInstance, Time: clk.Now(), Event: alerts.EventReputationHit, + Host: appHost, Client: netip.MustParseAddr(fromDE), Netblock: netip.MustParsePrefix(fromDE + "/32"), ASN: asnDE, @@ -530,7 +532,7 @@ func TestEachZoneThatListsAClientRaisesAnAlertOncePerCooldownAndIsCounted( // Each zone's hits, and its queries and their failures, none, since // every client had its verdicts. - metrics := s.scrape(unplaced) + metrics := s.scrape() for _, zone := range []string{dnsblZone, otherZone} { labels := `{instance="` + alertInstance + `",source="` + zone + `"}` @@ -564,7 +566,7 @@ func TestZoneKeyIsMaskedInTheLogTheAlertAndTheMetrics(t *testing.T) { masked) } - metrics := s.scrape(unplaced) + metrics := s.scrape() wantMetric(t, metrics, `smallwebwaf_reputation_hits_total{instance="`+ alertInstance+`",source="`+masked+`"}`, 1) @@ -737,10 +739,6 @@ func TestClientRefusedForAnOffenceIsCheckedWithAbuseIPDBAtItsNextRequest(t *test "a trap path", "/xmlrpc.php", http.StatusForbidden, requestlog.ActionBanned, ratelimit.Offences{Attack: 1}, }, - { - "a missing token", proxy.MetricsPath, http.StatusUnauthorized, - requestlog.ActionAdmin, ratelimit.Offences{TokenRefused: 1}, - }, } { t.Run(tc.name, func(t *testing.T) { t.Parallel() @@ -748,7 +746,7 @@ func TestClientRefusedForAnOffenceIsCheckedWithAbuseIPDBAtItsNextRequest(t *test s, clk, server := startWithClock(t, "", map[string]string{ abuseIPDBKey: accountKey, reputationAction: actionLog, rulesDir: writeRules(t, testRules), attackBanDuration: "1h", - trapPaths: trapPathList, metricsToken: token, wafMode: block, + trapPaths: trapPathList, wafMode: block, }) s.request(client, tc.path, tc.status, tc.action) @@ -862,7 +860,7 @@ func TestAbuseIPDBHitRaisesAnAlertWithTheScoreOncePerCooldownAndIsCounted( // The hits, and the checks, none, since no client committed an // offence, so that the whole budget is left. - metrics := s.scrape(unplaced) + metrics := s.scrape() labels := `{instance="` + alertInstance + `",source="` + abuseipdb + `"}` wantMetric(t, metrics, "smallwebwaf_reputation_hits_total"+labels, 2) @@ -889,7 +887,7 @@ func TestWithoutAnAbuseIPDBKeyNoClientIsCheckedNorAScoreUsed(t *testing.T) { wantReputation(t, s.get(fromDE, http.StatusOK, forward)) wantAbuseIPDBChecks(t, server, 0) - wantNoSeries(t, s.scrape(unplaced), `smallwebwaf_reputation_daily_budget_remaining{`+ + wantNoSeries(t, s.scrape(), `smallwebwaf_reputation_daily_budget_remaining{`+ `instance="`+alertInstance+`",source="`+abuseipdb+`"}`) } diff --git a/internal/proxy/request.go b/internal/proxy/request.go index d5721e8..210fa8a 100644 --- a/internal/proxy/request.go +++ b/internal/proxy/request.go @@ -2,17 +2,10 @@ package proxy import ( "context" - "errors" - "io" "net/http" - "net/http/httptrace" - "net/http/httputil" "net/netip" - "os" "slices" "strings" - "sync" - "sync/atomic" "time" "sneak.berlin/go/smallwebwaf/internal/anomaly" @@ -24,19 +17,11 @@ import ( "sneak.berlin/go/smallwebwaf/internal/rules" ) -// flushAfterEachWrite has ReverseProxy pass on each part of the app's -// answer as soon as it arrives. -const flushAfterEachWrite time.Duration = -1 - -// refusal is smallwebwaf refusing a request, or refusing to go on with it: -// the status the client is answered if the response has not started yet, -// 0 to close the connection without an answer, the action the log line -// names, and the setting whose size or time limit the request passed, if -// that is why. +// refusal is servewaf refusing a request: the status the visitor is to get, 0 +// to close its connection without an answer, and the action the log line names. type refusal struct { status int action string - limit string } // decision is what the checks note of a request: its log line, and what @@ -48,8 +33,8 @@ type decision struct { client netip.Addr // requestURI is the path with the query, as a ban's notes keep it. requestURI string - // site is, in servewaf, the host the visitor asked for, which the - // alerts name, since one servewaf serves many sites. It is "" otherwise. + // site is, for a question, the host the visitor asked for, which the + // alerts name, since one servewaf serves many sites. site string // allowed is true for a question answered with DispositionAllow. allowed bool @@ -78,75 +63,34 @@ type decision struct { reputation []bans.ReputationHit } -// request is one request on its way through smallwebwaf, from the moment -// its headers have been read to its log line. +// request is one request to servewaf, from the moment its headers have been +// read to its log line. type request struct { decision - h *handler - in *http.Request - // rc sets the deadlines of the connection to the client. - rc *http.ResponseController - out *responseWriter - body *requestBody // nil for a request without a body + h *handler + in *http.Request + out *responseWriter - peer netip.Addr - peerTrusted bool - // tokenRefused is true for a request refused for a missing or wrong - // token, an offence its client's history counts. - tokenRefused bool // blocklisted is true once a blocklist is found to list the client, // dnsblListed once a DNSBL zone's verdict is, and abuseIPDBHit once // AbuseIPDB's score of it is a hit. blocklisted, dnsblListed, abuseIPDBHit bool start time.Time - // checked is when the checks were done, and upstreamStart when the - // request was handed to the app. - checked time.Time - upstreamStart time.Time - // cancel ends the request to the app. - cancel context.CancelFunc - // refused is the first refusal, from whichever goroutine meets it. - refused atomic.Pointer[refusal] - // complete is true once the app's whole answer has been passed on. - complete bool - // upgraded is the connection to the app once the app has switched - // protocols, as for a WebSocket, and nil otherwise. - upgraded *upgradedConn - - // mu guards what follows. The timeouts run on goroutines of their - // own, and the transport starts and stops them, and notes the times - // below, from its own; once timersStopped is set, none of the timeouts - // acts any more. - mu sync.Mutex - timersStopped bool - clientRequestTimer *time.Timer - upstreamRequestTimer *time.Timer - upstreamResponseTimer *time.Timer - // connected is when there was a connection to the app, requestSent - // when the app had been sent the whole request, and answerStarted - // when the first byte of its answer arrived. - connected time.Time - requestSent time.Time - answerStarted time.Time -} - -// newRequest starts handling r: it works out the client from the TCP peer -// and X-Forwarded-For, and goes on as startRequest does. -func (h *handler) newRequest(w http.ResponseWriter, r *http.Request) *request { - peer := peerAddress(r) - trusted := h.config.TrustedProxies - client := clientAddress(peer, r.Header.Values("X-Forwarded-For"), trusted) - - return h.startRequest(w, r, client, peer, isInside(peer, trusted)) + // checked is when the checks were done. + checked time.Time + // refused is what the request was refused with, nil if it was not. + refused *refusal + // requestBytes is how many bytes of the request's body have been read. + requestBytes int64 } // startRequest starts handling r from client, which came through peer: it // notes the time, counts the request as under way, and starts the log line -// with what is known of the request. peerTrusted says whether to believe -// the request's id and scheme, which r's headers give. +// with what is known of the request. isQuestion says whether r is a question, +// whose headers give the request's id and scheme. func (h *handler) startRequest( - w http.ResponseWriter, r *http.Request, client, peer netip.Addr, peerTrusted bool, + w http.ResponseWriter, r *http.Request, client, peer netip.Addr, isQuestion bool, ) *request { h.metrics.RequestStarted() @@ -162,14 +106,14 @@ func (h *handler) startRequest( Instance: h.config.InstanceName, ClientIP: client.String(), Method: r.Method, - Scheme: scheme(r, peerTrusted), + Scheme: scheme(r, isQuestion), Host: r.Host, Path: rules.Path(r), Query: r.URL.RawQuery, Protocol: r.Proto, Referer: r.Referer(), UserAgent: r.UserAgent(), - RequestID: requestID(r, peerTrusted), + RequestID: requestID(r, isQuestion), PeerIP: peer.String(), ForwardedFor: strings.Join(forwardedFor, ", "), ClientGroup: h.clientGroup(client).String(), @@ -180,13 +124,10 @@ func (h *handler) startRequest( Action: requestlog.ActionForward, }, }, - h: h, - in: r, - rc: http.NewResponseController(w), - out: &responseWriter{ResponseWriter: w}, - peer: peer, - peerTrusted: peerTrusted, - start: start, + h: h, + in: r, + out: &responseWriter{ResponseWriter: w}, + start: start, } // A length of -1 is a body whose length was not announced. @@ -194,10 +135,6 @@ func (h *handler) startRequest( rq.line.ContentLength = r.ContentLength } - if r.Body != http.NoBody { - rq.body = &requestBody{body: limitBody(r.Body, h.config.RequestMaxBytes), rq: rq} - } - return rq } @@ -222,29 +159,21 @@ func requestHeaders(r *http.Request, names []string) map[string]string { return headers } -// check is the one place where a request can be refused once its client -// is known, before anything reaches the app, and before its body is read, -// but for the part the Core Rule Set reads. It returns nil to let the -// request through. The checks of checkClient come first, answered with -// SWWAF_BAN_RESPONSE, 403 for a block rule or the Core Rule Set, or 431 for -// servewaf's header limits, and then the size limit, so that a request the -// rate limits count is counted even when it is refused for its size. In -// observe mode a request checkClient refuses goes on to the size limit like -// any other. A size or time limit the Core Rule Set's reading of the body -// meets ends the request in either mode. ctx is the request's own context. +// check is the one place where a request can be refused. It returns nil to let +// the request through. The checks of checkClient decide: a request they refuse +// is answered with SWWAF_BAN_RESPONSE, 403 for a block rule or the Core Rule Set, +// or 431 for the header limits. In observe mode nothing is refused, and the log +// line names what enforce mode would have done. ctx is the request's own context. func (rq *request) check(ctx context.Context) *refusal { action := rq.checkClient(ctx) - refused := rq.refused.Load() - if refused != nil { - return refused - } - switch { case action == "": + return nil case rq.h.config.Observe: - // The log line names what enforce mode would have done. rq.line.WouldAction = action + + return nil case action == requestlog.ActionRuleBlocked || action == requestlog.ActionWAFBlocked: return &refusal{status: http.StatusForbidden, action: action} case action == requestlog.ActionTooLarge: @@ -252,17 +181,6 @@ func (rq *request) check(ctx context.Context) *refusal { default: return rq.banResponse(action) } - - maxBytes := rq.h.config.RequestMaxBytes - if maxBytes > 0 && rq.in.ContentLength > maxBytes { - return &refusal{ - status: http.StatusRequestEntityTooLarge, - action: requestlog.ActionTooLarge, - limit: "SWWAF_REQUEST_MAX_BYTES", - } - } - - return nil } // checkClient runs the checks on the request's client, and returns the @@ -277,10 +195,9 @@ func (rq *request) check(ctx context.Context) *refusal { // rate limits, unless the client is in SWWAF_RATE_LIMIT_EXEMPT_NETS or the // request's path is exempt under SWWAF_RATE_LIMIT_EXEMPT_PATHS, so that // every other request is counted, each of them by the client's limit -// percentages, then SWWAF_TRAP_PATHS, then the rule files, then, in -// servewaf, the header limits, and last the Core Rule Set. A request exempt -// from the rate limits is exempt from the byte limits too. ctx is the -// request's own context. +// percentages, then SWWAF_TRAP_PATHS, then the rule files, then the header +// limits, and last the Core Rule Set. A request exempt from the rate limits +// is exempt from the byte limits too. ctx is the request's own context. func (rq *request) checkClient(ctx context.Context) string { cfg := rq.h.config if isInside(rq.client, cfg.AllowNets) { @@ -381,251 +298,32 @@ func pathExempt(r *http.Request, prefixes []string) bool { }) } -// forward passes the request to the app and the app's answer back. ctx -// is the request's own context. -func (rq *request) forward(ctx context.Context) { - ctx, cancel := context.WithCancel(ctx) - defer cancel() - - rq.cancel = cancel - ctx = httptrace.WithClientTrace(ctx, &httptrace.ClientTrace{ - GotConn: rq.gotConn, - WroteRequest: rq.wroteRequest, - GotFirstResponseByte: rq.gotFirstResponseByte, - }) - - out := rq.in.WithContext(ctx) - if rq.body != nil { - out.Body = rq.body - } - - reverseProxy := &httputil.ReverseProxy{ - Rewrite: rq.rewrite, - Transport: rq.h.transport, - FlushInterval: flushAfterEachWrite, - ErrorLog: rq.h.errorLog, - ModifyResponse: rq.modifyResponse, - ErrorHandler: rq.answerError, - } - - rq.startRequestTimers() - rq.upstreamStart = time.Now() - reverseProxy.ServeHTTP(rq.out, out) -} - -// rewrite makes the request the app receives: the client's request, -// unchanged, sent to SWWAF_UPSTREAM_URL, with the forwarded headers and -// the request's id set, without any X-Client-ASN or X-Client-Country the -// client sent, whatever SWWAF_ADD_LOOKUP_HEADERS says, and, while it is -// set, with the client's AS number and country in them. -func (rq *request) rewrite(pr *httputil.ProxyRequest) { - upstream := rq.h.config.UpstreamURL - pr.Out.URL.Scheme = upstream.Scheme - pr.Out.URL.Host = upstream.Host - // ReverseProxy drops query parameters it cannot parse; the app gets - // the query as the client sent it. - pr.Out.URL.RawQuery = pr.In.URL.RawQuery - // ReverseProxy sends the path URL.EscapedPath gives, which, for a path - // holding a character Go escapes, such as |, is the whole path decoded - // and escaped again, %2F turned into /. Opaque is sent as it is, so the - // app gets the path as the client sent it. Left out are a target that - // is no path, such as *, and a path starting with //, which Go would - // send in Opaque as a scheme and host. - path := rules.Path(pr.In) - if strings.HasPrefix(path, "/") && !strings.HasPrefix(path, "//") { - pr.Out.URL.Opaque = path - } - - setForwardedHeaders(pr.In, pr.Out, rq.peer, rq.peerTrusted) - pr.Out.Header.Set(requestIDHeader, rq.line.RequestID) - pr.Out.Header.Del(asnHeader) - pr.Out.Header.Del(countryHeader) - - if rq.h.config.AddLookupHeaders { - setLookupHeaders(pr.Out.Header, rq.line.ASN, rq.line.Country) - } -} - -// modifyResponse looks at the app's answer before ReverseProxy passes it -// on. -func (rq *request) modifyResponse(res *http.Response) error { - rq.line.UpstreamStatus = res.StatusCode - - if res.StatusCode == http.StatusSwitchingProtocols { - // An upgraded connection, such as a WebSocket, is not cut by the - // timeouts. ReverseProxy writes this answer straight to the - // connection it takes over, not through rq.out, and then copies - // what passes each way through res.Body, the connection to the app. - // It may still refuse the switch, as when the app's Upgrade is not - // the client's, so the status is noted only once it has taken over - // the connection, by rq.out's Hijack. - rq.stopTimers() - rq.line.Websocket = true - - conn, ok := res.Body.(io.ReadWriteCloser) - if ok { - rq.upgraded = &upgradedConn{ReadWriteCloser: conn} - res.Body = rq.upgraded - } - - return nil - } - - maxBytes := rq.h.config.ResponseMaxBytes - if maxBytes > 0 && res.Body != http.NoBody && res.ContentLength > maxBytes { - rq.refuse(refusal{ - status: http.StatusBadGateway, - action: requestlog.ActionTooLarge, - limit: "SWWAF_RESPONSE_MAX_BYTES", - }) - - return errResponseTooLarge - } - - // Go's server adds a Content-Type it guesses from the body to a - // response whose headers have no Content-Type key. The key with no - // value sends none, as the app sent none. - if res.Header.Get("Content-Type") == "" { - rq.out.Header()["Content-Type"] = nil - } - - res.Body = &responseBody{body: limitBody(res.Body, maxBytes), rq: rq} - rq.startClientResponseTimeout() - - return nil -} - -// answerError is ReverseProxy's ErrorHandler: the request could not be -// passed to the app, or the app's answer cannot be passed on. -func (rq *request) answerError(_ http.ResponseWriter, _ *http.Request, err error) { - if rq.upgraded != nil { - // The app switched protocols, but the switch was not passed on, - // and ReverseProxy leaves the app's connection open when it - // refuses it. - _ = rq.upgraded.Close() - } - - refused := rq.refused.Load() - if refused == nil { - if rq.in.Context().Err() != nil { - return // the client has gone, and there is no one to answer - } - - rq.h.processLog.Warn("request to the app failed", "error", err.Error()) - - refused = &refusal{ - status: http.StatusBadGateway, - action: requestlog.ActionUpstreamError, - } - } - - rq.answer(*refused) -} - -// answer sends smallwebwaf's own answer, unless the response has already -// started, and records the refusal for the log line. -func (rq *request) answer(r refusal) { - rq.refused.CompareAndSwap(nil, &r) - - if rq.out.status != 0 { - return // too late to answer: the connection can only be cut - } - - if r.status == 0 { - // SWWAF_BAN_RESPONSE is close. This panic has Go's server close - // the connection without an answer, and log nothing; the log line - // is still written as the handler returns. - panic(http.ErrAbortHandler) - } - - // A client found too slow is read no more; any other may go on - // sending until its time is up, so that Go's server can read the - // rest of the body and end the request cleanly. - deadline := rq.clientRequestDeadline() - if r.status == http.StatusRequestTimeout { - deadline = time.Now() - } - - rq.stopReadingBody(deadline) - - timeout := rq.h.config.ClientResponseTimeout - if timeout > 0 { - _ = rq.rc.SetWriteDeadline(time.Now().Add(timeout)) - } - - http.Error(rq.out, http.StatusText(r.status), r.status) -} - -// refuse records r, unless an earlier refusal was, and ends the request -// to the app, if one was made: smallwebwaf reads the body of a request -// it answers itself too. -func (rq *request) refuse(r refusal) { - rq.refused.CompareAndSwap(nil, &r) - - if rq.cancel != nil { - rq.cancel() - } -} - -// finish ends the request's timeouts, counts it in the metrics and writes -// its log line. +// finish counts the request in the metrics and writes its log line. func (rq *request) finish() { - rq.stopTimers() - - refused := rq.refused.Load() - if refused == nil { - rq.stopReadingBody(rq.clientRequestDeadline()) - } - line := &rq.line line.Status = rq.out.status line.ResponseBytes = rq.out.bytes - header := rq.out.Header() - line.ResponseContentType = header.Get("Content-Type") - line.CacheControl = header.Get("Cache-Control") - line.Location = header.Get("Location") - line.RequestBytes = rq.requestBytes() - - // limit is the setting whose size or time limit the request passed. - var limit string + line.ResponseContentType = rq.out.Header().Get("Content-Type") + line.RequestBytes = rq.requestBytes switch { - case refused != nil: - line.Action = refused.action - limit = refused.limit - case errors.Is(rq.out.err, os.ErrDeadlineExceeded): - // The client took longer than SWWAF_CLIENT_RESPONSE_TIMEOUT to - // take the response. - line.Action = requestlog.ActionTimedOut - limit = "SWWAF_CLIENT_RESPONSE_TIMEOUT" - case !rq.complete && (rq.out.err != nil || rq.in.Context().Err() != nil): + case rq.refused != nil: + line.Action = rq.refused.action + case rq.out.err != nil || rq.in.Context().Err() != nil: line.Aborted = true } - now := time.Now() - duration := now.Sub(rq.start) + duration := time.Since(rq.start) line.DurationTotal = requestlog.Milliseconds(duration) line.DurationChecks = timing(rq.start, rq.checked) - var upstreamDuration time.Duration - - if !rq.upstreamStart.IsZero() { - upstreamDuration = now.Sub(rq.upstreamStart) - line.DurationUpstreamTotal = new(requestlog.Milliseconds(upstreamDuration)) - - rq.mu.Lock() - line.DurationUpstreamConnect = timing(rq.upstreamStart, rq.connected) - line.DurationUpstreamFirstByte = timing(rq.upstreamStart, rq.answerStarted) - rq.mu.Unlock() - } - // Counted before the log line is written, so that the metrics count // every request whose line is out. An allowed request is the exception: // its report gives the status, bytes and times the metrics count. if rq.allowed { rq.h.metrics.RequestAllowed() } else { - rq.h.metrics.RequestEnded(line, limit, duration, upstreamDuration) + rq.h.metrics.RequestEnded(line, duration) } rq.writeLine() @@ -657,18 +355,16 @@ func timing(start, end time.Time) *float64 { // before either was there, and one from GeoJS that comes later is added // when it comes. func (rq *request) addToHistory() { - forwarded := !rq.upstreamStart.IsZero() request := ratelimit.Request{ - Forwarded: forwarded, - Refused: !forwarded && rq.refused.Load() != nil, + Forwarded: rq.allowed, + Refused: rq.refused != nil, Status: rq.out.status, - RequestBytes: rq.requestBytes(), + RequestBytes: rq.requestBytes, ResponseBytes: rq.out.bytes, BrokeLimit: rq.line.Offence == requestlog.OffenceLimit, Attack: rq.attack, RuleBlocked: rq.ruleBlocked, WAFBlocked: rq.wafBlocked, - TokenRefused: rq.tokenRefused, } rq.h.limiter.AddToHistory(rq.h.clientGroup(rq.client), rq.h.now(), request) @@ -729,228 +425,38 @@ func (rq *request) answerAtTheEnd() (lookup.Answer, bool) { return rq.h.geojs.Kept(rq.h.clientGroup(rq.client)) } -// requestBytes is how many bytes of the request's body have been read. -func (rq *request) requestBytes() int64 { - if rq.body == nil { - return 0 - } +// responseWriter is servewaf's answer to a request. It notes the status and +// size for the log line, and the first error writing the answer. +type responseWriter struct { + http.ResponseWriter - return rq.body.bytes.Load() + // status is the status sent, or zero before one is. + status int + bytes int64 + err error } -// clientRequestDeadline is when the client must have sent its whole -// request, or zero when SWWAF_CLIENT_REQUEST_TIMEOUT is off. -func (rq *request) clientRequestDeadline() time.Time { - timeout := rq.h.config.ClientRequestTimeout - if timeout == 0 { - return time.Time{} +// WriteHeader sends the status and headers. +func (w *responseWriter) WriteHeader(status int) { + if w.status == 0 { + w.status = status } - return rq.start.Add(timeout) + w.ResponseWriter.WriteHeader(status) } -// stopReadingBody ends, at deadline, the reading of a client body that has -// not arrived whole: Go's server then reads no more of it, and closes the -// connection after the answer. -func (rq *request) stopReadingBody(deadline time.Time) { - if rq.body == nil || rq.body.received.Load() { - return +// Write sends part of the body. +func (w *responseWriter) Write(p []byte) (int, error) { + if w.status == 0 { + w.status = http.StatusOK } - _ = rq.rc.SetReadDeadline(deadline) -} + n, err := w.ResponseWriter.Write(p) + w.bytes += int64(n) -// startRequestTimers starts the timeouts that run while the request goes -// to the app: SWWAF_CLIENT_REQUEST_TIMEOUT until the client has sent its -// whole body, and SWWAF_UPSTREAM_REQUEST_TIMEOUT until the app has been -// sent the whole request. -func (rq *request) startRequestTimers() { - rq.mu.Lock() - defer rq.mu.Unlock() - - if rq.body != nil && rq.h.config.ClientRequestTimeout > 0 { - rq.clientRequestTimer = time.AfterFunc( - time.Until(rq.clientRequestDeadline()), func() { - rq.requestTimedOut("SWWAF_CLIENT_REQUEST_TIMEOUT") - }) + if w.err == nil { + w.err = err } - timeout := rq.h.config.UpstreamRequestTimeout - if timeout > 0 { - rq.upstreamRequestTimer = time.AfterFunc(timeout, func() { - rq.requestTimedOut("SWWAF_UPSTREAM_REQUEST_TIMEOUT") - }) - } -} - -// requestTimedOut is called when limit, SWWAF_CLIENT_REQUEST_TIMEOUT or -// SWWAF_UPSTREAM_REQUEST_TIMEOUT, runs out while the request is still on -// its way to the app. The answer names the side smallwebwaf was waiting -// on at that moment: 408 when it was waiting for the client to send more -// of its body, 504 when it was waiting for the app to be reached or to -// take what it had. -func (rq *request) requestTimedOut(limit string) { - rq.mu.Lock() - defer rq.mu.Unlock() - - if rq.timersStopped { - return - } - - if rq.body == nil || !rq.body.waiting.Load() { - // Once the app has answered, the transport, ReverseProxy and Go's - // server each close the client's body, and closing it reads what - // is left of a body under 256 KiB without requestBody.Read, so - // waiting is false while that read waits for the client. That - // read ends now, before refuse: should smallwebwaf still answer - // itself, answer then sets the deadline it gives its own answers. - if !rq.answerStarted.IsZero() { - rq.stopReadingBody(time.Now()) - } - - rq.refuse(refusal{ - status: http.StatusGatewayTimeout, - action: requestlog.ActionTimedOut, - limit: limit, - }) - - return - } - - rq.refuse(refusal{ - status: http.StatusRequestTimeout, - action: requestlog.ActionTimedOut, - limit: limit, - }) - // The transport gives up on the app only once its Read of the - // client's body returns, so that Read is ended now. The lock keeps - // this from reaching the connection after the request is handled. - _ = rq.rc.SetReadDeadline(time.Now()) -} - -// bodyReceived is called once the client has sent its whole body. -func (rq *request) bodyReceived() { - rq.mu.Lock() - defer rq.mu.Unlock() - - stopTimer(rq.clientRequestTimer) -} - -// gotConn is called once there is a connection to the app, a new one or -// one kept open from an earlier request. -func (rq *request) gotConn(httptrace.GotConnInfo) { - rq.mu.Lock() - defer rq.mu.Unlock() - - rq.connected = time.Now() -} - -// gotFirstResponseByte is called once the first byte of the app's answer -// has arrived. -func (rq *request) gotFirstResponseByte() { - rq.mu.Lock() - defer rq.mu.Unlock() - - rq.answerStarted = time.Now() -} - -// wroteRequest is called once the app has been sent the whole request: -// the request timeouts end and SWWAF_UPSTREAM_RESPONSE_TIMEOUT starts. -func (rq *request) wroteRequest(info httptrace.WroteRequestInfo) { - if info.Err != nil { - return // the transport gives up, or tries again - } - - rq.mu.Lock() - defer rq.mu.Unlock() - - if rq.timersStopped { - return - } - - stopTimer(rq.clientRequestTimer) - stopTimer(rq.upstreamRequestTimer) - rq.requestSent = time.Now() - - timeout := rq.h.config.UpstreamResponseTimeout - if timeout > 0 { - rq.upstreamResponseTimer = time.AfterFunc(timeout, rq.responseTimedOut) - } -} - -// responseTimedOut is called when SWWAF_UPSTREAM_RESPONSE_TIMEOUT runs out -// before the app has sent its whole answer. -func (rq *request) responseTimedOut() { - rq.mu.Lock() - defer rq.mu.Unlock() - - if !rq.timersStopped { - rq.refuse(refusal{ - status: http.StatusGatewayTimeout, - action: requestlog.ActionTimedOut, - limit: "SWWAF_UPSTREAM_RESPONSE_TIMEOUT", - }) - } -} - -// responseReceived is called once the app has sent its whole answer. An -// app may answer before the client has sent its whole body, as with a 404 -// it sends without reading it. The timeouts then run on: until the -// transport's read of the body returns, Go's server cannot send the -// answer, and SWWAF_CLIENT_REQUEST_TIMEOUT is what ends that read for a -// client that stops sending. -func (rq *request) responseReceived() { - rq.complete = true - - if rq.body != nil && !rq.body.received.Load() { - return - } - - rq.stopTimers() -} - -// startClientResponseTimeout sets SWWAF_CLIENT_RESPONSE_TIMEOUT on the -// connection to the client: the response must reach the client within it -// of the end of the request, or of now if the app answers before it has -// the whole request. -func (rq *request) startClientResponseTimeout() { - timeout := rq.h.config.ClientResponseTimeout - if timeout == 0 { - return - } - - from := rq.sentAt() - if from.IsZero() { - from = time.Now() - } - - _ = rq.rc.SetWriteDeadline(from.Add(timeout)) -} - -// sentAt is when the app had been sent the whole request, or zero. -func (rq *request) sentAt() time.Time { - rq.mu.Lock() - defer rq.mu.Unlock() - - return rq.requestSent -} - -// stopTimers stops the request's timeouts and keeps any from starting -// later: the app's answer is complete, the connection upgraded, or the -// request handled. -func (rq *request) stopTimers() { - rq.mu.Lock() - defer rq.mu.Unlock() - - rq.timersStopped = true - stopTimer(rq.clientRequestTimer) - stopTimer(rq.upstreamRequestTimer) - stopTimer(rq.upstreamResponseTimer) -} - -// stopTimer stops t, which is nil when its timeout is off. -func stopTimer(t *time.Timer) { - if t != nil { - t.Stop() - } + return n, err } diff --git a/internal/proxy/request_internal_test.go b/internal/proxy/request_internal_test.go deleted file mode 100644 index ce26481..0000000 --- a/internal/proxy/request_internal_test.go +++ /dev/null @@ -1,55 +0,0 @@ -package proxy - -import ( - "io" - "net/http" - "net/http/httptest" - "testing" - - "sneak.berlin/go/smallwebwaf/internal/config" -) - -func TestAnAnswerWithoutAContentTypeGetsNone(t *testing.T) { - t.Parallel() - - srv := httptest.NewServer(http.HandlerFunc( - func(w http.ResponseWriter, _ *http.Request) { - rq := &request{ - h: &handler{config: &config.Config{}}, - out: &responseWriter{ResponseWriter: w}, - } - app := &http.Response{ - StatusCode: http.StatusOK, - Header: http.Header{}, - Body: http.NoBody, - } - - err := rq.modifyResponse(app) - if err != nil { - t.Errorf("modifyResponse: %v", err) - } - - // The headers go out with this, the first part of the body, - // which Go's server would guess a Content-Type from. - _, _ = io.WriteString(w, "page") - })) - t.Cleanup(srv.Close) - - req, err := http.NewRequestWithContext(t.Context(), http.MethodGet, srv.URL, - http.NoBody) - if err != nil { - t.Fatalf("new request: %v", err) - } - - res, err := srv.Client().Do(req) - if err != nil { - t.Fatalf("get: %v", err) - } - - _ = res.Body.Close() - - if res.Header.Values("Content-Type") != nil { - t.Errorf("client got Content-Type %q, want none", - res.Header.Get("Content-Type")) - } -} diff --git a/internal/proxy/requestlog_test.go b/internal/proxy/requestlog_test.go index 75e20f9..8442465 100644 --- a/internal/proxy/requestlog_test.go +++ b/internal/proxy/requestlog_test.go @@ -2,7 +2,6 @@ package proxy_test import ( "io" - "maps" "math" "net/http" "reflect" @@ -42,9 +41,6 @@ func TestLogLineHasEachFieldWhereItApplies(t *testing.T) { return } - w.Header().Set("Content-Type", "text/html") - w.Header().Set("Cache-Control", "no-store") - w.Header().Set("Location", "/elsewhere") w.WriteHeader(http.StatusFound) _, _ = io.WriteString(w, "moved") }) @@ -70,7 +66,7 @@ func TestLogLineHasEachFieldWhereItApplies(t *testing.T) { // A request's log line can come after its answer: each is waited for // before the next request, so that the lines are in order. - full := out.requestLines(t, 1)[0] + full := out.endedLines(t, 1)[0] // This one comes from 127.0.0.1, which the rate limits do not count, // with a body of 4 bytes whose length it does not announce, so that its @@ -81,7 +77,7 @@ func TestLogLineHasEachFieldWhereItApplies(t *testing.T) { "Transfer-Encoding: chunked\r\n\r\n4\r\nbody\r\n0\r\n\r\n") wantStatus(t, readResponse(t, conn), http.StatusNoContent) - bare := out.requestLines(t, 2)[1] + bare := out.endedLines(t, 2)[1] wantFullLine(t, full) wantBareLine(t, bare) @@ -108,24 +104,23 @@ func wantFullLine(t *testing.T, line logLine) { headers := map[string]string{"accept": "text/html", "x-custom": "one, two"} want := withTimings(line, requestlog.Line{ - Type: requestType, Time: line.Time, Instance: instance, + Type: requestType, ServeWAFLine: requestlog.ServeWAFLineReport, + Time: line.Time, Instance: instance, ClientIP: ipv6Client, Method: http.MethodPost, Scheme: secure, Host: appHost, Path: "/full", Protocol: protocol, Status: http.StatusFound, RequestBytes: 3, ResponseBytes: 5, - RequestID: "from-traefik", PeerIP: localhost, + RequestID: line.RequestID, PeerIP: localhost, ForwardedFor: "198.51.100.7, " + ipv6Client, ClientGroup: ipv6Group, ContentType: "application/x-www-form-urlencoded", ContentLength: 3, RequestHeaders: headers, HasAuthorization: true, HasCookie: true, - ResponseContentType: "text/html", UpstreamStatus: http.StatusFound, - CacheControl: "no-store", Location: "/elsewhere", Action: requestlog.ActionForward, // Its 3 bytes in and 5 out, each way counted by default. Counts: ratelimit.Counts{ Minute: 1, Hour: 1, Day: 1, MinuteBytes: 8, HourBytes: 8, DayBytes: 8, }, }) - if !reflect.DeepEqual(line.Line, want) { - t.Errorf("log line\n%+v\nwant\n%+v", line.Line, want) + if !reflect.DeepEqual(line.Line, want) || line.RequestID == "from-traefik" { + t.Errorf("log line\n%+v\nwant\n%+v, with a new request id", line.Line, want) } } @@ -135,12 +130,13 @@ func wantBareLine(t *testing.T, line logLine) { t.Helper() want := withTimings(line, requestlog.Line{ - Type: requestType, Time: line.Time, Instance: instance, + Type: requestType, ServeWAFLine: requestlog.ServeWAFLineReport, + Time: line.Time, Instance: instance, ClientIP: localhost, Method: http.MethodPost, Scheme: plain, Host: appHost, Path: "/bare", Protocol: protocol, Status: http.StatusNoContent, RequestBytes: 4, RequestID: line.RequestID, PeerIP: localhost, ClientGroup: localhost + "/32", - UpstreamStatus: http.StatusNoContent, Action: requestlog.ActionForward, + Action: requestlog.ActionForward, }) if !reflect.DeepEqual(line.Line, want) || line.RequestID == "" { t.Errorf("log line\n%+v\nwant\n%+v, with a request id", line.Line, want) @@ -148,8 +144,7 @@ func wantBareLine(t *testing.T, line logLine) { for _, name := range []string{ "forwarded_for", "content_type", "content_length", "request_headers", - "has_authorization", "has_cookie", "websocket", "response_content_type", - "cache_control", "location", "counts", + "has_authorization", "has_cookie", "response_content_type", "counts", } { _, present := line.fields[name] if present { @@ -162,8 +157,6 @@ func wantBareLine(t *testing.T, line logLine) { func withTimings(line logLine, want requestlog.Line) requestlog.Line { want.DurationTotal = line.DurationTotal want.DurationChecks = line.DurationChecks - want.DurationUpstreamConnect = line.DurationUpstreamConnect - want.DurationUpstreamFirstByte = line.DurationUpstreamFirstByte want.DurationUpstreamTotal = line.DurationUpstreamTotal return want @@ -188,7 +181,7 @@ func TestHasAuthorizationAndHasCookieEachComeFromTheirOwnHeader(t *testing.T) { req.Header.Set(tc.header, "secret") wantStatus(t, do(t, req), http.StatusOK) - line := out.requestLine(t) + line := out.endedLine(t) _, otherPresent := line.fields[tc.other] if line.fields[tc.field] != true || otherPresent { @@ -199,81 +192,20 @@ func TestHasAuthorizationAndHasCookieEachComeFromTheirOwnHeader(t *testing.T) { } } -func TestRequestIDAndSchemeComeOnlyFromATrustedProxy(t *testing.T) { - t.Parallel() - - const sentID = "from-traefik" - - sent := http.Header{requestIDHeader: {sentID}, forwardedProto: {secure}} - trusted := map[string]string{trustedProxies: trustLocalhost} - - for _, tc := range []struct { - name string - env map[string]string - header http.Header - // wantID is the request id logged, "" for a new one. - wantID, wantScheme string - }{ - {"a trusted proxy's are kept", trusted, sent, sentID, secure}, - {"without them, the id is new and the scheme http", trusted, nil, "", plain}, - {"another peer's are replaced", nil, sent, "", plain}, - } { - t.Run(tc.name, func(t *testing.T) { - t.Parallel() - - received := make(chan string, 2) - app := startApp(t, func(_ http.ResponseWriter, r *http.Request) { - received <- r.Header.Get(requestIDHeader) - }) - addr, out := startProxy(t, app.URL, tc.env) - - // Two requests, so that two new ids can be told apart. - ids := make([]string, 0, 2) - - for i := range 2 { - req := newRequest(t, http.MethodGet, addr, "/", http.NoBody) - maps.Copy(req.Header, tc.header) - wantStatus(t, do(t, req), http.StatusOK) - - line := out.requestLines(t, i+1)[i] - ids = append(ids, line.RequestID) - - got := <-received - if line.RequestID != got || line.Scheme != tc.wantScheme { - t.Errorf("log line has request_id %q and scheme %q, and the "+ - "app received id %q; want the same id and scheme %q", - line.RequestID, line.Scheme, got, tc.wantScheme) - } - } - - switch { - case tc.wantID != "" && (ids[0] != tc.wantID || ids[1] != tc.wantID): - t.Errorf("request ids %q, want %q", ids, tc.wantID) - case tc.wantID == "" && (slices.Contains(ids, sentID) || - slices.Contains(ids, "") || ids[0] == ids[1]): - t.Errorf("request ids %q, want two new ones", ids) - } - }) - } -} - func TestTimingsAreInOrder(t *testing.T) { t.Parallel() const denied = "192.0.2.50" // in SWWAF_DENY_NETS app := startApp(t, func(w http.ResponseWriter, _ *http.Request) { - // The pauses set the times apart; a hold-up of the test only - // lengthens them. - time.Sleep(time.Millisecond) - w.WriteHeader(http.StatusOK) - _ = http.NewResponseController(w).Flush() - + // The pause sets the times apart; a hold-up of the test only lengthens + // it. time.Sleep(time.Millisecond) _, _ = io.WriteString(w, "done") }) - addr, out := startProxy(t, app.URL, map[string]string{ + + addr, out, server := startProxyWithClock(t, app.URL, "", time.Now, map[string]string{ trustedProxies: trustLocalhost, denyNets: denied, }) @@ -281,20 +213,19 @@ func TestTimingsAreInOrder(t *testing.T) { // Each log line is waited for before the next request, so that the // lines are in order. wantStatus(t, get(t, addr, "/"), http.StatusOK) - forwarded := out.requestLines(t, 1)[0] + forwarded := out.endedLines(t, 1)[0] req := newRequest(t, http.MethodGet, addr, "/", http.NoBody) req.Header.Set(forwardedFor, denied) wantStatus(t, do(t, req), http.StatusForbidden) - refused := out.requestLines(t, 2)[1] + refused := out.endedLines(t, 2)[1] - wantStatus(t, get(t, addr, proxy.HealthPath), http.StatusOK) - health := out.requestLines(t, 3)[2] + wantStatus(t, get(t, server.Addr, proxy.HealthPath), http.StatusOK) + health := out.endedLines(t, 3)[2] // A request passed to the app has every timing; one refused, none of // the app's; the health check, which runs no check, only the total. wantTimings(t, forwarded, "duration_total", "duration_checks", - "duration_upstream_connect", "duration_upstream_first_byte", "duration_upstream_total") wantTimings(t, refused, "duration_total", "duration_checks") wantTimings(t, health, "duration_total") @@ -307,17 +238,12 @@ func TestTimingsAreInOrder(t *testing.T) { // exact. total := microseconds(forwarded.DurationTotal) checks := microseconds(*forwarded.DurationChecks) - connect := microseconds(*forwarded.DurationUpstreamConnect) - firstByte := microseconds(*forwarded.DurationUpstreamFirstByte) upstream := microseconds(*forwarded.DurationUpstreamTotal) - // The checks end before the request is handed to the app, and the - // connection comes before the answer, which the app ends after a - // pause. - if checks+upstream > total || connect >= firstByte || firstByte >= upstream { - t.Errorf("timings in microseconds: total %d, checks %d, connect %d, "+ - "first byte %d, upstream total %d", total, checks, connect, firstByte, - upstream) + // The checks end before the request is handed to the app. + if checks+upstream > total { + t.Errorf("timings in microseconds: total %d, checks %d, upstream total %d", + total, checks, upstream) } if *refused.DurationChecks > refused.DurationTotal { @@ -350,22 +276,3 @@ func wantTimings(t *testing.T, line logLine, want ...string) { func microseconds(milliseconds float64) int64 { return int64(math.Round(milliseconds * 1000)) } - -func TestLogsAnUpgradedConnection(t *testing.T) { - t.Parallel() - - app := startApp(t, echoAfterUpgrade) - addr, out := startProxy(t, app.URL, nil) - - conn := dial(t, addr) - send(t, conn, "GET /socket HTTP/1.1\r\nHost: app\r\n"+ - "Connection: Upgrade\r\nUpgrade: websocket\r\n\r\n") - wantStatus(t, readResponse(t, conn), http.StatusSwitchingProtocols) - - _ = conn.Close() - - line := out.requestLine(t) - if line.fields["websocket"] != true { - t.Errorf("log line has websocket %v, want true", line.fields["websocket"]) - } -} diff --git a/internal/proxy/rulefiles_test.go b/internal/proxy/rulefiles_test.go index 20e74d0..0c8fe6c 100644 --- a/internal/proxy/rulefiles_test.go +++ b/internal/proxy/rulefiles_test.go @@ -185,8 +185,6 @@ func TestObserveModeLogsWhatTheRulesWouldDo(t *testing.T) { func TestMetricsCountRuleMatchesAndBansForAnAttack(t *testing.T) { t.Parallel() - const scraper = "192.0.2.200" - s, _, _ := startWithClock(t, "", map[string]string{ rulesDir: writeRules(t, testRules), metricsToken: token, @@ -196,7 +194,7 @@ func TestMetricsCountRuleMatchesAndBansForAnAttack(t *testing.T) { s.request(client, "/.env", http.StatusForbidden, requestlog.ActionBanned) s.get(client, http.StatusForbidden, requestlog.ActionBanned) - metrics := s.scrape(scraper) + metrics := s.scrape() wantMetric(t, metrics, `smallwebwaf_rule_matches_total{action="block",instance="app",rule_id="blocked"}`, 1) wantMetric(t, metrics, diff --git a/internal/proxy/servewaf.go b/internal/proxy/servewaf.go index 61d7aa0..3e49ad5 100644 --- a/internal/proxy/servewaf.go +++ b/internal/proxy/servewaf.go @@ -6,6 +6,7 @@ import ( "errors" "fmt" "io" + "log/slog" "net/http" "net/netip" "strconv" @@ -13,8 +14,6 @@ import ( "time" "unicode/utf8" - lru "github.com/hashicorp/golang-lru/v2" - "sneak.berlin/go/smallwebwaf/internal/requestlog" ) @@ -105,45 +104,38 @@ type Report struct { } // NewServeWAF returns the server `smallwebwaf servewaf` runs. It answers -// rproxy's questions with the checks New runs, takes rproxy's reports and -// answers its own endpoints. It passes nothing to an app. rproxy applies -// the size and time limits to the visitor, so their settings do not apply -// here. Go's default limit on a request's line and headers, 1 MiB, stays: -// it is far above rproxy's, and a question refused for its size would have -// rproxy let the request through. +// rproxy's questions, takes rproxy's reports and answers its own endpoints. It +// passes nothing to an app. rproxy applies the size and time limits to the +// visitor. Go's default limit on a request's line and headers, 1 MiB, stays: it +// is far above rproxy's, and a question refused for its size would have rproxy +// let the request through. func NewServeWAF(params Params) *Server { - cfg := *params.Config - cfg.RequestMaxBytes = 0 - cfg.ClientRequestTimeout = serveWAFTimeout - // The server's WriteTimeout does this. It sets a new deadline for each - // request, where a deadline a handler sets stays on a kept-open - // connection, and would run out under a later request's answer. - cfg.ClientResponseTimeout = 0 - // A question's body of QuestionBodyMaxBytes may be cut short, and the - // Core Rule Set does not read a JSON or XML body over its limit. - cfg.WAFBodyLimit = min(cfg.WAFBodyLimit, QuestionBodyMaxBytes-1) - params.Config = &cfg - h := newHandler(params) - awaitingReport, err := lru.New[string, *decision](maxAwaitingReport) - if err != nil { - panic(err) // New fails only for a size below one + return &Server{ + Server: &http.Server{ + Addr: params.Config.ListenAddr, + Handler: h, + ErrorLog: slog.NewLogLogger(params.ProcessLog.Handler(), slog.LevelWarn), + ReadTimeout: serveWAFTimeout, + WriteTimeout: serveWAFTimeout, + IdleTimeout: serveWAFIdleTimeout, + }, + Ledger: h.ledger, + Limiter: h.limiter, + GeoJS: h.geojs, + Anomalies: h.anomalies, + LookupFile: h.lookupFile, + Lists: h.lists, + DNSBL: h.dnsbl, + AbuseIPDB: h.abuseIPDB, + Metrics: h.metrics, } - - h.awaitingReport = awaitingReport - - return h.server(&http.Server{ - Handler: http.HandlerFunc(h.serveWAF), - ReadTimeout: serveWAFTimeout, - WriteTimeout: serveWAFTimeout, - IdleTimeout: serveWAFIdleTimeout, - }) } -// serveWAF handles one request to servewaf: a request carrying -// ClientHeader is rproxy's question, and any other is for servewaf itself. -func (h *handler) serveWAF(w http.ResponseWriter, r *http.Request) { +// ServeHTTP handles one request to servewaf: a request carrying ClientHeader is +// rproxy's question, and any other is for servewaf itself. +func (h *handler) ServeHTTP(w http.ResponseWriter, r *http.Request) { if len(r.Header.Values(ClientHeader)) > 0 { h.answerQuestion(w, r) @@ -153,12 +145,6 @@ func (h *handler) serveWAF(w http.ResponseWriter, r *http.Request) { h.answerOwnEndpoint(w, r) } -// answersQuestions reports whether h is servewaf, which alone holds requests -// for their reports. -func (h *handler) answersQuestions() bool { - return h.awaitingReport != nil -} - // refusedWhateverTheAnswer reports whether the request is one that does not wait // for GeoJS, since the checks will refuse it at now whatever GeoJS answers about // its client: it is over the header limits, a blocklist lists its client while @@ -188,10 +174,6 @@ func (rq *request) refusedWhateverTheAnswer(now time.Time) bool { // characters, of which a header value can hold only tabs, so a part of Unicode // spaces is a cookie to it. func (rq *request) overHeaderLimits() bool { - if !rq.h.answersQuestions() { - return false - } - fields, cookies := 0, 0 for _, values := range rq.in.Header { @@ -244,7 +226,8 @@ func (h *handler) answerQuestion(w http.ResponseWriter, r *http.Request) { return } - // The last deferred runs first, as in ServeHTTP. + // The last deferred runs first: countRefusal before addToHistory, so that a + // broken error burst is in the client's history. defer rq.addToHistory() defer rq.countAnomalies() defer rq.countRefusal() @@ -374,10 +357,11 @@ func (rq *request) holdForReport() { // deny answers a question with what the visitor is to get for r. func (rq *request) deny(r refusal) { + rq.refused = &r + if r.status == 0 { // SWWAF_BAN_RESPONSE is close. Written past rq.out, since the log line's // status stays 0: the visitor is sent nothing. - rq.refused.CompareAndSwap(nil, &r) rq.out.Header().Set(DispositionHeader, DispositionClose) rq.out.ResponseWriter.WriteHeader(http.StatusOK) @@ -385,7 +369,7 @@ func (rq *request) deny(r refusal) { } rq.out.Header().Set(DispositionHeader, DispositionDeny) - rq.answer(r) + http.Error(rq.out, http.StatusText(r.status), r.status) } // answerOwnEndpoint answers a request to servewaf's own address. No check @@ -425,7 +409,8 @@ func (h *handler) takeReport(w http.ResponseWriter, r *http.Request) { } held, found := h.awaitingReport.Peek(report.RequestID) - // Remove reports true to one report only, of several that name the id. + // Remove reports true to one report only, of several that name the id. A + // question still being handled is held as nil. if found && held != nil && h.awaitingReport.Remove(report.RequestID) { h.finishWith(held, report) } @@ -490,18 +475,17 @@ func (report *Report) checkRanges() error { // finishWith finishes a request with what its report says, and writes the // request's last log line, which holds its decision's fields and the report's. -// For an allowed request it first takes the steps ServeHTTP takes once a -// request passed to the app has ended, with the status and bytes of the report -// in place of those it would have seen. A denied request's counting was -// finished at its decision. +// For an allowed request it first counts the request's bytes, and adds it to its +// client's history, the anomaly counters and the metrics, with the status and +// bytes the report gives. A denied request's counting was finished at its +// decision. func (h *handler) finishWith(held *decision, report Report) { rq := &request{ - decision: *held, - h: h, - out: &responseWriter{status: report.Status, bytes: report.ResponseBytes}, - body: &requestBody{}, + decision: *held, + h: h, + out: &responseWriter{status: report.Status, bytes: report.ResponseBytes}, + requestBytes: report.RequestBytes, } - rq.body.bytes.Store(report.RequestBytes) rq.line.ServeWAFLine = requestlog.ServeWAFLineReport rq.line.Status = report.Status @@ -511,8 +495,6 @@ func (h *handler) finishWith(held *decision, report Report) { if rq.allowed { rq.line.DurationUpstreamTotal = &report.DurationUpstreamTotal - // addToHistory goes by this for whether the request reached the app. - rq.upstreamStart = time.Now() rq.countBytes() rq.countAnomalies() diff --git a/internal/proxy/servewaf_test.go b/internal/proxy/servewaf_test.go index 2081dd7..55bf113 100644 --- a/internal/proxy/servewaf_test.go +++ b/internal/proxy/servewaf_test.go @@ -84,26 +84,12 @@ func startServeWAFWith( } maps.Copy(settings, env) - server, out, queue := newServerWith(t, proxy.NewServeWAF, - "http://127.0.0.1:8081", geojsURL, clk.Now, settings) + server, out, queue := newServeWAF(t, geojsURL, clk.Now, settings) change(server) - - listener, err := (&net.ListenConfig{}).Listen(t.Context(), "tcp", localhost+":0") - if err != nil { - t.Fatalf("listen: %v", err) - } - - go func() { - _ = server.Serve(listener) - }() - - t.Cleanup(func() { - _ = server.Close() - }) + serve(t, server) return &serveWAF{ - t: t, addr: listener.Addr().String(), out: out, server: server, - clock: clk, queue: queue, + t: t, addr: server.Addr, out: out, server: server, clock: clk, queue: queue, } } @@ -626,13 +612,10 @@ func TestQuestionWhoseBodyStopsArrivingGetsNoAnswer(t *testing.T) { wantProcessMessages(t, s, "a question's body stopped arriving", 1) wantNoRequestLines(t, s) - // The id was not taken, and the metrics count nothing for it. + // The id was not taken. again := s.question(siteA, client, "/") again.Header.Set(requestIDHeader, "an-id") wantDisposition(t, do(t, again), http.StatusOK, proxy.DispositionAllow) - wantNoSeries(t, s.metrics(), - `smallwebwaf_size_and_time_limit_hits_total{instance="`+alertInstance+ - `",limit="SWWAF_CLIENT_REQUEST_TIMEOUT"}`) }) } } @@ -1466,16 +1449,10 @@ func TestOwnEndpointsLinesDoNotSayWhichOfTheTwoTheyAre(t *testing.T) { } } -func TestSettingsOfTheLimitsAndTheProxiesAreIgnored(t *testing.T) { +func TestALargeBodyAnnouncedAndLongHeadersAreNoReasonToRefuse(t *testing.T) { t.Parallel() - s := startServeWAF(t, "", map[string]string{ - requestMaxBytes: "1", - clientHeaderMaxBytes: "5K", - clientRequestTimeout: off, - clientResponseTimeout: off, - trustedProxies: localhost, - }) + s := startServeWAF(t, "", nil) req := s.questionWith(http.MethodPost, siteA, client, "/", strings.NewReader("a body")) @@ -1493,33 +1470,23 @@ func TestSettingsOfTheLimitsAndTheProxiesAreIgnored(t *testing.T) { } } -func TestShortClientResponseTimeoutIsIgnored(t *testing.T) { +func TestLogLevelHoldsBackNoRequestLine(t *testing.T) { t.Parallel() - // The setting would leave no time to send any answer. - s := startServeWAF(t, "", map[string]string{ - clientResponseTimeout: time.Nanosecond.String(), denyNets: client, - }) + s := startServeWAF(t, "", map[string]string{"SWWAF_LOG_LEVEL": "error"}) - wantDisposition(t, s.ask(client, "/"), http.StatusForbidden, proxy.DispositionDeny) - wantStatus(t, get(t, s.addr, proxy.HealthPath), http.StatusOK) -} + // A refused question would write a process warning, held back at error. + req := s.question(siteA, client, "/") + req.Header.Del(requestIDHeader) + wantStatus(t, do(t, req), http.StatusBadRequest) -func TestShortClientRequestTimeoutIsIgnored(t *testing.T) { - t.Parallel() + s.allowed(client) + wantLine(t, s.out.requestLine(t), http.StatusOK, requestlog.ActionForward) - // The setting would have run out while the Core Rule Set read the body. Go's - // server would then take the connection for gone, and the line say aborted. - s := startServeWAF(t, "", map[string]string{ - clientRequestTimeout: time.Nanosecond.String(), wafMode: block, wafBodyLimit: "1M", - }) - - req := s.questionWith(http.MethodPost, siteA, client, "/", strings.NewReader("a=1")) - req.Header.Set("Content-Type", formData) - wantDisposition(t, do(t, req), http.StatusOK, proxy.DispositionAllow) - - if line := s.out.requestLine(t); line.Aborted { - t.Errorf("log line says aborted: %+v", line.Line) + for _, line := range s.out.lines(t) { + if line["type"] == "process" { + t.Errorf("process line %v, want none at error", line) + } } } @@ -1573,7 +1540,7 @@ func TestCoreRuleSetReadsTheBodyAQuestionCarries(t *testing.T) { t.Parallel() s := startServeWAF(t, "", map[string]string{ - wafMode: block, wafBodyLimit: "1M", + wafMode: block, wafBodyLimit: "64K", }) ask := func(contentType, body string) answer { diff --git a/internal/proxy/staticlists_test.go b/internal/proxy/staticlists_test.go index ea7c67f..6f1712a 100644 --- a/internal/proxy/staticlists_test.go +++ b/internal/proxy/staticlists_test.go @@ -17,7 +17,7 @@ const ( unlistedAddr = "2001:db8::2" ) -func TestAllowNetsSkipEveryCheckButTheSizeLimit(t *testing.T) { +func TestAllowNetsSkipEveryCheck(t *testing.T) { t.Parallel() var calls atomic.Int32 @@ -34,7 +34,6 @@ func TestAllowNetsSkipEveryCheckButTheSizeLimit(t *testing.T) { denyNets: fromKP, deniedCountries: "kp", rateLimitPerMinute: "1", - requestMaxBytes: "1K", }) // Neither SWWAF_DENY_NETS, the country lists nor the limit of one @@ -49,14 +48,6 @@ func TestAllowNetsSkipEveryCheckButTheSizeLimit(t *testing.T) { t.Errorf("GeoJS was asked about %v, want nothing", asked()) } - // The size limit still applies. - body := strings.NewReader(strings.Repeat("a", 2<<10)) - req := newRequest(t, http.MethodPost, addr, "/", body) - req.Header.Set(forwardedFor, fromKP) - wantStatus(t, do(t, req), http.StatusRequestEntityTooLarge) - wantLine(t, out.requestLines(t, 3)[2], - http.StatusRequestEntityTooLarge, requestlog.ActionTooLarge) - if calls.Load() != 2 { t.Errorf("the app was called %d times, want 2", calls.Load()) } @@ -82,7 +73,7 @@ func TestRequestFromAllowNetsIsNotCounted(t *testing.T) { }) } -func TestDenyNetsRefuseBeforeTheLookupAndTheBody(t *testing.T) { +func TestDenyNetsRefuseBeforeTheLookup(t *testing.T) { t.Parallel() var calls atomic.Int32 @@ -101,13 +92,9 @@ func TestDenyNetsRefuseBeforeTheLookupAndTheBody(t *testing.T) { req.Header.Set(forwardedFor, fromDE) wantStatus(t, do(t, req), http.StatusForbidden) - line := out.requestLine(t) + line := out.endedLine(t) wantLine(t, line, http.StatusForbidden, requestlog.ActionDenied) - if line.RequestBytes != 0 { - t.Errorf("log line has request_bytes %d, want 0", line.RequestBytes) - } - if len(asked()) != 0 { t.Errorf("GeoJS was asked about %v, want nothing", asked()) } @@ -180,6 +167,6 @@ func wantAnswers(t *testing.T, addr string, out *output, requests []sentRequest) req := newRequest(t, http.MethodGet, addr, "/", http.NoBody) req.Header.Set(forwardedFor, sent.client) wantStatus(t, do(t, req), sent.status) - wantLine(t, out.requestLines(t, i+1)[i], sent.status, sent.action) + wantLine(t, out.endedLines(t, i+1)[i], sent.status, sent.action) } } diff --git a/internal/proxy/timeouts_test.go b/internal/proxy/timeouts_test.go deleted file mode 100644 index 639f86f..0000000 --- a/internal/proxy/timeouts_test.go +++ /dev/null @@ -1,453 +0,0 @@ -package proxy_test - -import ( - "errors" - "io" - "net" - "net/http" - "net/http/httptest" - "strconv" - "strings" - "sync" - "sync/atomic" - "testing" - "time" - - "sneak.berlin/go/smallwebwaf/internal/requestlog" -) - -// largeBodySize is more than the connections between the client, -// smallwebwaf and the app can hold while nobody reads, so that a sender -// soon waits. -const largeBodySize = 64 << 20 - -// writeSize is how much a test sender writes at a time. -const writeSize = 32 << 10 - -func TestRequestTimeouts(t *testing.T) { - t.Parallel() - - for _, tc := range []struct { - name string - // limit is the setting set to shortTimeout, which runs out; long - // is one set to longTimeoutSetting, which does not, or "". - limit, long string - // appTakesNothing has the app never read, while the client sends - // as fast as it can; otherwise the app reads, and the client - // stops sending halfway. - appTakesNothing bool - want int - }{ - { - name: "client request timeout, waiting on the client", - limit: clientRequestTimeout, - want: http.StatusRequestTimeout, - }, - { - name: "upstream request timeout, waiting on the client", - limit: upstreamRequestTimeout, - long: clientRequestTimeout, - want: http.StatusRequestTimeout, - }, - { - name: "upstream request timeout, waiting on the app", - limit: upstreamRequestTimeout, - appTakesNothing: true, - want: http.StatusGatewayTimeout, - }, - { - name: "client request timeout, waiting on the app", - limit: clientRequestTimeout, - long: upstreamRequestTimeout, - appTakesNothing: true, - want: http.StatusGatewayTimeout, - }, - } { - t.Run(tc.name, func(t *testing.T) { - t.Parallel() - - var ( - app *httptest.Server - appURL string - sendRequest func(*testing.T, string) net.Conn - appGotBody atomic.Bool - ) - - if tc.appTakesNothing { - appURL, sendRequest = startAppThatTakesNothing(t), sendLargeBody - } else { - app = startApp(t, func(_ http.ResponseWriter, r *http.Request) { - n, _ := io.Copy(io.Discard, r.Body) - appGotBody.Store(n > 0) - }) - appURL, sendRequest = app.URL, sendPartOfBody - } - - env := map[string]string{tc.limit: shortTimeoutSetting, metricsToken: token} - if tc.long != "" { - env[tc.long] = longTimeoutSetting - } - - addr, out := startProxy(t, appURL, env) - start := time.Now() - got := readResponse(t, sendRequest(t, addr)) - wantTimedOut(t, start) - - want := tc.want - - if app != nil { - // Close returns once the app has finished with the request. - app.Close() - - // Until some of the body has reached the app, smallwebwaf - // waits on the app, and SPEC.md asks for 504; the timeout - // runs out then only if the test process is held up. - if !appGotBody.Load() { - want = http.StatusGatewayTimeout - } - } - - wantStatus(t, got, want) - wantLine(t, out.requestLine(t), want, requestlog.ActionTimedOut) - wantLimitHits(t, addr, tc.limit, 1) - }) - } -} - -// startAppThatTakesNothing starts an app that accepts connections and -// never reads from them, and returns its URL. -func startAppThatTakesNothing(t *testing.T) string { - t.Helper() - - listener, err := (&net.ListenConfig{}).Listen(t.Context(), "tcp", localhost+":0") - if err != nil { - t.Fatalf("listen: %v", err) - } - - var ( - mu sync.Mutex - held []net.Conn - ) - - hold := func(conn net.Conn) { - mu.Lock() - defer mu.Unlock() - - held = append(held, conn) - } - - go func() { - for { - conn, err := listener.Accept() - if err != nil { - return - } - - hold(conn) - } - }() - - t.Cleanup(func() { - _ = listener.Close() - - mu.Lock() - defer mu.Unlock() - - for _, conn := range held { - _ = conn.Close() - } - }) - - return "http://" + listener.Addr().String() -} - -// sendPartOfBody sends a request that announces a large body, and only -// the first bytes of it. -func sendPartOfBody(t *testing.T, addr string) net.Conn { - t.Helper() - - conn := dial(t, addr) - send(t, conn, "POST /upload HTTP/1.1\r\nHost: app\r\nContent-Length: "+ - strconv.Itoa(largeBodySize)+"\r\n\r\nthe first bytes") - - return conn -} - -// sendLargeBody sends a request with a large body, as fast as smallwebwaf -// takes it, from a goroutine of its own. -func sendLargeBody(t *testing.T, addr string) net.Conn { - t.Helper() - - conn := dial(t, addr) - send(t, conn, "POST /upload HTTP/1.1\r\nHost: app\r\nContent-Length: "+ - strconv.Itoa(largeBodySize)+"\r\n\r\n") - - go func() { - chunk := make([]byte, writeSize) - for range largeBodySize / writeSize { - _, err := conn.Write(chunk) - if err != nil { - return - } - } - }() - - return conn -} - -func TestClientStopsSendingAfterTheAppHasAnswered(t *testing.T) { - t.Parallel() - - // answeredAt is when the app answered, or nil if the bytes the client - // sent never reached it. - var answeredAt atomic.Pointer[time.Time] - - // The app answers without reading the whole body, as an app does for - // a path it does not serve, and only once smallwebwaf waits for the - // rest of it: the app takes the bytes the client sent, and then waits - // half the timeout, far longer than smallwebwaf takes to go on to - // wait. - app := startApp(t, func(w http.ResponseWriter, r *http.Request) { - _, err := io.ReadFull(r.Body, make([]byte, len("the first bytes"))) - if err != nil { - return - } - - time.Sleep(shortTimeout / 2) - answeredAt.Store(new(time.Now())) - http.NotFound(w, r) - }) - addr, out := startProxy(t, app.URL, map[string]string{ - clientRequestTimeout: shortTimeoutSetting, - metricsToken: token, - }) - - start := time.Now() - got := readResponse(t, sendPartOfBody(t, addr)) - wantTimedOut(t, start) - - // Close returns once the app has finished with the request. - app.Close() - - // The app's answer came before the time ran out, so the client is - // sent that answer, not smallwebwaf's 408. Only a hold-up of the test - // process can make it come later, and SPEC.md then asks for 408, as - // smallwebwaf was waiting for the client, or for 504 if the bytes the - // client sent had not reached the app. - at := answeredAt.Load() - want := http.StatusNotFound - - switch { - case at == nil: - want = http.StatusGatewayTimeout - case at.Sub(start) >= shortTimeout: - want = http.StatusRequestTimeout - } - - wantStatus(t, got, want) - wantLine(t, out.requestLine(t), want, requestlog.ActionTimedOut) - wantLimitHits(t, addr, clientRequestTimeout, 1) -} - -func TestClientStopsSendingWithLittleOfItsBodyLeft(t *testing.T) { - t.Parallel() - - for _, tc := range []struct { - name string - // limit is the setting set to shortTimeout, which runs out, and - // clientTimeout what SWWAF_CLIENT_REQUEST_TIMEOUT is set to when it - // is not limit. - limit, clientTimeout string - }{ - { - name: "client request timeout", - limit: clientRequestTimeout, - }, - { - name: "upstream request timeout, client request timeout off", - limit: upstreamRequestTimeout, - clientTimeout: off, - }, - { - name: "upstream request timeout, client request timeout longer", - limit: upstreamRequestTimeout, - clientTimeout: longTimeoutSetting, - }, - } { - t.Run(tc.name, func(t *testing.T) { - t.Parallel() - - // appClosed is closed once smallwebwaf has closed its - // connection to the app, which it does once it has the app's - // answer. - appClosed := make(chan struct{}) - - // The app answers without reading the rest of the body, and - // says it closes the connection, only once smallwebwaf waits - // for the client to send more: it takes the bytes the client - // sent, and then waits half the timeout, far longer than - // smallwebwaf takes to go on to wait. - app := startApp(t, func(w http.ResponseWriter, _ *http.Request) { - conn, buf, err := http.NewResponseController(w).Hijack() - if err != nil { - return - } - - _, _ = io.ReadFull(buf.Reader, make([]byte, len("the first bytes"))) - - time.Sleep(shortTimeout / 2) - - _, _ = buf.WriteString("HTTP/1.1 404 Not Found\r\n" + - "Content-Length: 0\r\nConnection: close\r\n\r\n") - _ = buf.Flush() - _, _ = io.Copy(io.Discard, buf.Reader) - _ = conn.Close() - - close(appClosed) - }) - - env := map[string]string{tc.limit: shortTimeoutSetting, metricsToken: token} - if tc.clientTimeout != "" { - env[clientRequestTimeout] = tc.clientTimeout - } - - addr, out := startProxy(t, app.URL, env) - - // Less than 256 KiB of the body is left after the first bytes, - // little enough for Go's server in smallwebwaf to read the rest - // itself before it sends the app's answer. - start := time.Now() - conn := dial(t, addr) - send(t, conn, "POST /upload HTTP/1.1\r\nHost: app\r\nContent-Length: 200000\r\n\r\n"+ - "the first bytes") - - select { - case <-appClosed: - case <-time.After(waitLimit): - t.Fatal("smallwebwaf did not close its connection to the app") - } - - // More of the body, which can no longer go to the app, and then - // nothing. - send(t, conn, strings.Repeat("x", 8<<10)) - - got := readResponse(t, conn) - wantTimedOut(t, start) - wantStatus(t, got, http.StatusNotFound) - wantLine(t, out.requestLine(t), http.StatusNotFound, requestlog.ActionTimedOut) - wantLimitHits(t, addr, tc.limit, 1) - }) - } -} - -func TestAppTooSlowToAnswer(t *testing.T) { - t.Parallel() - - app := startApp(t, func(_ http.ResponseWriter, r *http.Request) { - <-r.Context().Done() - }) - addr, out := startProxy(t, app.URL, map[string]string{ - upstreamResponseTimeout: shortTimeoutSetting, - metricsToken: token, - }) - - start := time.Now() - - wantStatus(t, get(t, addr, "/slow"), http.StatusGatewayTimeout) - wantTimedOut(t, start) - - line := out.requestLine(t) - wantLine(t, line, http.StatusGatewayTimeout, requestlog.ActionTimedOut) - - _, answered := line.fields["upstream_status"] - if answered { - t.Errorf("log line has upstream_status %v for an app that never answered", - line.fields["upstream_status"]) - } - - wantLimitHits(t, addr, upstreamResponseTimeout, 1) -} - -func TestAppTooSlowToFinishItsAnswer(t *testing.T) { - t.Parallel() - - app := startApp(t, func(w http.ResponseWriter, r *http.Request) { - _, _ = io.WriteString(w, "the first part") - _ = http.NewResponseController(w).Flush() - - <-r.Context().Done() - }) - addr, out := startProxy(t, app.URL, map[string]string{ - upstreamResponseTimeout: shortTimeoutSetting, - }) - - start := time.Now() - got := get(t, addr, "/slow") - wantStatus(t, got, http.StatusOK) - - if string(got.body) != "the first part" || !errors.Is(got.err, io.ErrUnexpectedEOF) { - t.Errorf("client read %q (%v), want the first part cut off", got.body, got.err) - } - - wantTimedOut(t, start) - - line := out.requestLine(t) - wantLine(t, line, http.StatusOK, requestlog.ActionTimedOut) - - if line.UpstreamStatus != http.StatusOK { - t.Errorf("log line has upstream_status %d, want %d", - line.UpstreamStatus, http.StatusOK) - } -} - -func TestClientTooSlowToTakeTheAnswer(t *testing.T) { - t.Parallel() - - app := startApp(t, func(w http.ResponseWriter, _ *http.Request) { - chunk := make([]byte, writeSize) - for range largeBodySize / writeSize { - _, err := w.Write(chunk) - if err != nil { - return - } - } - }) - addr, out := startProxy(t, app.URL, map[string]string{ - clientResponseTimeout: shortTimeoutSetting, - metricsToken: token, - }) - - start := time.Now() - - // The client asks, and never reads the answer. - conn := dial(t, addr) - send(t, conn, "GET /large HTTP/1.1\r\nHost: app\r\n\r\n") - - line := out.requestLine(t) - wantTimedOut(t, start) - wantLine(t, line, http.StatusOK, requestlog.ActionTimedOut) - wantLimitHits(t, addr, clientResponseTimeout, 1) -} - -func TestClosesAnIdleConnection(t *testing.T) { - t.Parallel() - - app := startApp(t, func(http.ResponseWriter, *http.Request) {}) - addr, _ := startProxy(t, app.URL, map[string]string{ - clientIdleTimeout: shortTimeoutSetting, - }) - - // The idle time starts once the answer is sent, so after start. - start := time.Now() - conn := dial(t, addr) - send(t, conn, "GET / HTTP/1.1\r\nHost: app\r\n\r\n") - wantStatus(t, readResponse(t, conn), http.StatusOK) - - // The read deadline readResponse set still bounds this read. - _, err := conn.Read(make([]byte, 1)) - if !errors.Is(err, io.EOF) { - t.Fatalf("read on the idle connection: %v, want it closed", err) - } - - wantTimedOut(t, start) -} diff --git a/internal/ratelimit/history_test.go b/internal/ratelimit/history_test.go index bd6bf05..2cd83f5 100644 --- a/internal/ratelimit/history_test.go +++ b/internal/ratelimit/history_test.go @@ -23,8 +23,8 @@ func TestHistoryKeepsEveryRequest(t *testing.T) { {Forwarded: true, Status: 502, ResponseBytes: 12}, // Closed without an answer: refused, and no response. {Refused: true, Status: 0}, - // Answered 404 at smallwebwaf's own endpoints: neither forwarded - // nor refused. + // Answered 404 for a path under /_smallwebwaf/: neither forwarded nor + // refused. {Status: 404}, } { limiter.AddToHistory(client, start.Add(time.Duration(i)*time.Minute), r) diff --git a/internal/ratelimit/ratelimit.go b/internal/ratelimit/ratelimit.go index 73e35ee..49c8294 100644 --- a/internal/ratelimit/ratelimit.go +++ b/internal/ratelimit/ratelimit.go @@ -92,10 +92,9 @@ type History struct { ASName string `json:"as_name,omitempty"` Country string `json:"country,omitempty"` LookedUp time.Time `json:"looked_up,omitzero"` - // Requests are all the client's requests: Forwarded those passed to - // the app, Refused those refused before anything reached it, a 401 at - // smallwebwaf's own endpoints included, and neither the others - // smallwebwaf answered there. + // Requests are all the client's requests: Forwarded those servewaf allowed, + // Refused those it refused, and neither those for a path under + // /_smallwebwaf/. Requests int64 `json:"requests"` Forwarded int64 `json:"forwarded"` Refused int64 `json:"refused"` @@ -124,21 +123,20 @@ type Offences struct { // Limit is its requests that broke a rate limit, a byte limit or the // error burst, Attack those that were a clear sign of attack, a match // of a ban rule or a request for a trap path, RuleBlocked those a block - // rule refused, WAFBlocked those the Core Rule Set refused, and - // TokenRefused those refused for a missing or wrong token. - Limit int64 `json:"limit"` - Attack int64 `json:"attack"` - RuleBlocked int64 `json:"rule_blocked"` - WAFBlocked int64 `json:"waf_blocked"` - TokenRefused int64 `json:"token_refused"` + // rule refused, and WAFBlocked those the Core Rule Set refused. + Limit int64 `json:"limit"` + Attack int64 `json:"attack"` + RuleBlocked int64 `json:"rule_blocked"` + WAFBlocked int64 `json:"waf_blocked"` + // TokenRefused is what a clients.json written by an earlier version gives, + // which must still load. Nothing adds to it. + TokenRefused int64 `json:"token_refused,omitempty"` } // Request is what a client's history keeps of one of its requests. type Request struct { - // Forwarded is true for a request passed to the app, Refused for one - // refused before anything reached it, a 401 at smallwebwaf's own - // endpoints included. Both are false for any other request smallwebwaf - // answered there. + // Forwarded is true for a request servewaf allowed, and Refused for one it + // refused. Both are false for a request for a path under /_smallwebwaf/. Forwarded bool Refused bool // Status is what the client was sent, 0 if nothing was. @@ -149,14 +147,12 @@ type Request struct { ResponseBytes int64 // BrokeLimit is true for a request that broke a rate limit, a byte // limit or the error burst, Attack for one that matched a ban rule or - // asked for a trap path, RuleBlocked for one a block rule refused, - // WAFBlocked for one the Core Rule Set refused, and TokenRefused for - // one refused for a missing or wrong token. - BrokeLimit bool - Attack bool - RuleBlocked bool - WAFBlocked bool - TokenRefused bool + // asked for a trap path, RuleBlocked for one a block rule refused, and + // WAFBlocked for one the Core Rule Set refused. + BrokeLimit bool + Attack bool + RuleBlocked bool + WAFBlocked bool } // New returns a Limiter for limits, with no client counted yet, whose @@ -334,10 +330,6 @@ func (l *Limiter) AddToHistory(client netip.Prefix, now time.Time, r Request) { if r.WAFBlocked { h.Offences.WAFBlocked++ } - - if r.TokenRefused { - h.Offences.TokenRefused++ - } } // AddLookup gives client's history its AS number, AS name and country, as diff --git a/internal/requestlog/requestlog.go b/internal/requestlog/requestlog.go index a4221cf..c45e821 100644 --- a/internal/requestlog/requestlog.go +++ b/internal/requestlog/requestlog.go @@ -14,19 +14,13 @@ import ( "sneak.berlin/go/smallwebwaf/internal/ratelimit" ) -// The action a request line names: what smallwebwaf did with the -// request. +// The action a request line names: what servewaf did with the request. const ( - // ActionForward is a request passed to the app. + // ActionForward is a request servewaf allowed, which rproxy passes to the + // app. ActionForward = "forward" - // ActionTooLarge is a request or response over its size limit, or, in - // servewaf, a request over the header limits. + // ActionTooLarge is a request over the header limits. ActionTooLarge = "too_large" - // ActionTimedOut is a request or response that ran out of time. - ActionTimedOut = "timed_out" - // ActionUpstreamError is a request the app could not be reached - // for, or whose answer could not be passed on. - ActionUpstreamError = "upstream_error" // ActionRateLimited is a request refused because it took its client // over a rate limit, which bans the client. ActionRateLimited = "rate_limited" @@ -48,8 +42,8 @@ const ( ActionDenied = "denied" // ActionCountryDenied is a request refused for its client's country. ActionCountryDenied = "country_denied" - // ActionAdmin is a request smallwebwaf answered at one of its own - // endpoints, under /_smallwebwaf/. + // ActionAdmin is a request servewaf answered at one of its own endpoints, + // or a question about a path under /_smallwebwaf/. ActionAdmin = "admin" ) @@ -86,8 +80,8 @@ type Line struct { // two lines about a question. ServeWAFLine string `json:"servewaf_line,omitempty"` - // The standard web log fields. Scheme is how the client reached - // smallwebwaf, or the trusted proxy in front of it. + // The standard web log fields. Scheme is how the client reached rproxy, or + // the trusted proxy in front of it. Time string `json:"time"` Instance string `json:"instance"` ClientIP string `json:"client_ip"` @@ -103,8 +97,8 @@ type Line struct { Referer string `json:"referer"` UserAgent string `json:"user_agent"` - // Request detail. RequestID is the X-Request-ID a trusted proxy sent, - // or a new one, and is sent on to the app. ForwardedFor is the + // Request detail. RequestID is the id rproxy made for the request, or for a + // request to servewaf's own endpoints a new one. ForwardedFor is the // X-Forwarded-For header as received. ClientGroup is the netblock the // client is counted as. ASN, ASName and Country are the client's AS // number, AS name and country, as looked up. @@ -123,17 +117,10 @@ type Line struct { RequestHeaders map[string]string `json:"request_headers,omitempty"` HasAuthorization bool `json:"has_authorization,omitempty"` HasCookie bool `json:"has_cookie,omitempty"` - // Websocket is true when the connection was upgraded, as for a - // WebSocket. - Websocket bool `json:"websocket,omitempty"` - // Response detail, from the headers of the answer: the app's, as - // passed on, or those of smallwebwaf's own. Aborted is true when the - // client went away early. + // Response detail, of servewaf's own answer. Aborted is true when rproxy, or + // the admin's tool, went away before the answer. ResponseContentType string `json:"response_content_type,omitempty"` - UpstreamStatus int `json:"upstream_status,omitempty"` - CacheControl string `json:"cache_control,omitempty"` - Location string `json:"location,omitempty"` Aborted bool `json:"aborted,omitempty"` // The decision. @@ -183,17 +170,13 @@ type Line struct { // The timings, in milliseconds. DurationChecks is the time until the // checks were done, and DurationWAF the part of it the Core Rule Set - // took. DurationUpstreamConnect, DurationUpstreamFirstByte and - // DurationUpstreamTotal run from when the request was handed to the - // app: until there was a connection to it, until the first byte of its - // answer arrived, and until the end. Each but DurationTotal is nil for - // a request that did not get that far. - DurationTotal float64 `json:"duration_total"` - DurationChecks *float64 `json:"duration_checks,omitempty"` - DurationWAF *float64 `json:"duration_waf,omitempty"` - DurationUpstreamConnect *float64 `json:"duration_upstream_connect,omitempty"` - DurationUpstreamFirstByte *float64 `json:"duration_upstream_first_byte,omitempty"` - DurationUpstreamTotal *float64 `json:"duration_upstream_total,omitempty"` + // took. DurationUpstreamTotal is what the report of an allowed request gives, + // from the hand-over to the app until the end of its answer. Each but + // DurationTotal is nil for a request that did not get that far. + DurationTotal float64 `json:"duration_total"` + DurationChecks *float64 `json:"duration_checks,omitempty"` + DurationWAF *float64 `json:"duration_waf,omitempty"` + DurationUpstreamTotal *float64 `json:"duration_upstream_total,omitempty"` } // RProxyLine is rproxy's short line in the request log about one request. The diff --git a/internal/rproxy/question_test.go b/internal/rproxy/question_test.go index 0005f2a..16cda4e 100644 --- a/internal/rproxy/question_test.go +++ b/internal/rproxy/question_test.go @@ -128,6 +128,52 @@ func TestQuestionNamesTheVisitorATrustedProxySpeaksFor(t *testing.T) { } } +func TestQuestionNamesTheClientAsTheForwardedAddressesSay(t *testing.T) { + t.Parallel() + + for _, tc := range []struct { + name string + header []string + client string + }{ + { + "the first address outside the trusted proxies, from the right", + []string{"198.51.100.7, 203.0.113.9, 10.0.0.2"}, "203.0.113.9", + }, + { + "the leftmost, when every address is a trusted proxy", + []string{"10.0.0.5, 10.0.0.2"}, "10.0.0.5", + }, + {"the peer, with no header", nil, localhost}, + { + "the last address read, before an entry that is not an address", + []string{"203.0.113.9, unknown, 10.0.0.2"}, "10.0.0.2", + }, + { + "read as one list, with several header lines", + []string{"2001:db8::7", "10.0.0.2"}, "2001:db8::7", + }, + } { + t.Run(tc.name, func(t *testing.T) { + t.Parallel() + + servewaf := startServeWAF(t, allow) + app := startApp(t, greet) + addr, _ := startRProxy(t, servewaf.URL, app.URL, map[string]string{ + trustedProxies: localhost + "/32,10.0.0.0/8", + }) + + req := newRequest(t, http.MethodGet, addr, "/", http.NoBody) + req.Header[forwardedForHeader] = tc.header + wantGreeting(t, visit(t, req)) + + if got := servewaf.question(t).header.Get(proxy.ClientHeader); got != tc.client { + t.Errorf("the question names %s, want %s", got, tc.client) + } + }) + } +} + // randomBytes returns n random bytes. func randomBytes(n int) []byte { body := make([]byte, n) diff --git a/internal/smallwebwaf/healthcheck.go b/internal/smallwebwaf/healthcheck.go index cb5f9e7..c51df0b 100644 --- a/internal/smallwebwaf/healthcheck.go +++ b/internal/smallwebwaf/healthcheck.go @@ -17,13 +17,16 @@ import ( // healthCheckTimeout bounds the whole health check. const healthCheckTimeout = 5 * time.Second -var errHealthEndpoint = errors.New("smallwebwaf's health endpoint answered") +var errHealthEndpoint = errors.New("the health endpoint answered") -// HealthCheck is the container's health check. It returns 0 while -// smallwebwaf answers its health endpoint on 127.0.0.1, at the port in -// SWWAF_LISTEN_ADDR, and the app accepts connections at the address in -// SWWAF_UPSTREAM_URL. Otherwise it writes why to stderr and returns 1. -// It reads no other setting, nor a file that another names, so neither +// HealthCheck is the container's health check. In servewaf's container, where +// SMALLWEBWAF_COMMAND is servewaf, it returns 0 while servewaf answers its health +// endpoint on 127.0.0.1, at the port in SWWAF_LISTEN_ADDR. In rproxy's container, +// where SMALLWEBWAF_COMMAND is rproxy, unset or empty, it returns 0 while rproxy +// answers its health endpoint on 127.0.0.1, at the port in SWRP_LISTEN_ADDR, and +// the app accepts connections at the address in SWRP_UPSTREAM_URL. Otherwise, +// any other value of SMALLWEBWAF_COMMAND included, it writes why to stderr and +// returns 1. It reads no other setting, nor a file that another names, so neither // can fail it. // args are the arguments after `healthcheck`; it takes none, and given // one it names it on stderr and returns 1 without checking anything. @@ -38,7 +41,20 @@ func HealthCheck( return 1 } - err := healthCheck(ctx, lookupEnv) + ctx, cancel := context.WithTimeout(ctx, healthCheckTimeout) + defer cancel() + + command, err := config.Command(lookupEnv) + + switch { + case err != nil: + err = fmt.Errorf("invalid setting: %w", err) + case command == "servewaf": + err = checkServeWAF(ctx, lookupEnv) + default: + err = checkRProxyAndApp(ctx, lookupEnv) + } + if err != nil { _, _ = fmt.Fprintln(stderr, "unhealthy:", err) @@ -48,16 +64,48 @@ func HealthCheck( return 0 } -func healthCheck(ctx context.Context, lookupEnv func(string) (string, bool)) error { - ctx, cancel := context.WithTimeout(ctx, healthCheckTimeout) - defer cancel() - - listenAddr, upstreamURL, err := config.ListenAddrAndUpstreamURL(lookupEnv) +// checkServeWAF asks servewaf's health endpoint. +func checkServeWAF(ctx context.Context, lookupEnv func(string) (string, bool)) error { + listenAddr, err := config.ServeWAFListenAddr(lookupEnv) if err != nil { return fmt.Errorf("invalid setting: %w", err) } - // The settings have checked that the address has a port. + err = askHealthEndpoint(ctx, listenAddr) + if err != nil { + return fmt.Errorf("ask servewaf: %w", err) + } + + return nil +} + +// checkRProxyAndApp asks rproxy's health endpoint, and connects to the app. +func checkRProxyAndApp( + ctx context.Context, lookupEnv func(string) (string, bool), +) error { + listenAddr, upstreamURL, err := config.RProxyListenAddrAndUpstreamURL(lookupEnv) + if err != nil { + return fmt.Errorf("invalid setting: %w", err) + } + + err = askHealthEndpoint(ctx, listenAddr) + if err != nil { + return fmt.Errorf("ask rproxy: %w", err) + } + + conn, err := (&net.Dialer{}).DialContext(ctx, "tcp", appAddress(upstreamURL)) + if err != nil { + return fmt.Errorf("connect to the app: %w", err) + } + + _ = conn.Close() + + return nil +} + +// askHealthEndpoint asks the health endpoint on 127.0.0.1 at the port of +// listenAddr, which the settings have checked has one. +func askHealthEndpoint(ctx context.Context, listenAddr string) error { _, port, _ := net.SplitHostPort(listenAddr) health := "http://" + net.JoinHostPort("127.0.0.1", port) + proxy.HealthPath @@ -68,7 +116,7 @@ func healthCheck(ctx context.Context, lookupEnv func(string) (string, bool)) err res, err := http.DefaultClient.Do(req) if err != nil { - return fmt.Errorf("ask smallwebwaf: %w", err) + return fmt.Errorf("send the request: %w", err) } _ = res.Body.Close() @@ -77,13 +125,6 @@ func healthCheck(ctx context.Context, lookupEnv func(string) (string, bool)) err return fmt.Errorf("%w %s", errHealthEndpoint, res.Status) } - conn, err := (&net.Dialer{}).DialContext(ctx, "tcp", appAddress(upstreamURL)) - if err != nil { - return fmt.Errorf("connect to the app: %w", err) - } - - _ = conn.Close() - return nil } diff --git a/internal/smallwebwaf/healthcheck_test.go b/internal/smallwebwaf/healthcheck_test.go index a5ffd60..61f5d21 100644 --- a/internal/smallwebwaf/healthcheck_test.go +++ b/internal/smallwebwaf/healthcheck_test.go @@ -15,11 +15,16 @@ import ( "sneak.berlin/go/smallwebwaf/internal/smallwebwaf" ) -func TestHealthCheck(t *testing.T) { - t.Parallel() +const ( + // commandSetting says which command a container runs, and which the health + // check checks. + commandSetting = "SMALLWEBWAF_COMMAND" + // serveWAFCommand is the command that runs servewaf. + serveWAFCommand = "servewaf" +) - app := httptest.NewServer(http.NotFoundHandler()) - defer app.Close() +func TestHealthCheckInServeWAFsContainer(t *testing.T) { + t.Parallel() ctx, stop := context.WithCancel(t.Context()) defer stop() @@ -27,10 +32,9 @@ func TestHealthCheck(t *testing.T) { out := &output{} exited := make(chan int, 1) settings := map[string]string{ - listenAddr: localhost + ":0", - upstreamURL: app.URL, - stateDir: t.TempDir(), - rulesDir: t.TempDir(), + listenAddr: localhost + ":0", + stateDir: t.TempDir(), + rulesDir: t.TempDir(), } go func() { @@ -39,40 +43,36 @@ func TestHealthCheck(t *testing.T) { addr, _ := out.line(t, "msg", "starting")["address"].(string) _, port, _ := net.SplitHostPort(addr) + // The container's settings: an address to listen on with an empty // host part, which the health check asks at 127.0.0.1. - env := map[string]string{listenAddr: ":" + port, upstreamURL: app.URL} + wantHealthCheck(t, map[string]string{ + commandSetting: serveWAFCommand, listenAddr: ":" + port, + }, 0, "") - wantHealthCheck(t, env, 0, "") - - // The health check reads those two settings alone, here given as - // files: a removed or invalid token file, or an invalid value of - // another setting, does not fail it. + // The health check reads that setting alone, here given as a file: a removed + // or invalid token file, or an invalid value of another setting, does not + // fail it. for _, other := range []struct{ name, value string }{ {"SWWAF_METRICS_TOKEN_FILE", filepath.Join(t.TempDir(), "removed")}, {"SWWAF_METRICS_TOKEN_FILE", writeFile(t, "too short\n")}, {"SWWAF_MODE", "neither"}, } { wantHealthCheck(t, map[string]string{ - listenAddr + "_FILE": writeFile(t, ":"+port+"\n"), - upstreamURL + "_FILE": writeFile(t, app.URL+"\n"), - other.name: other.value, + commandSetting: serveWAFCommand, + listenAddr + "_FILE": writeFile(t, ":"+port+"\n"), + other.name: other.value, }, 0, "") } - // Another test can take a closed server's port at once, so the app's - // address here is a multicast address, to which TCP cannot connect. - wantHealthCheck(t, map[string]string{ - listenAddr: ":" + port, - upstreamURL: "http://224.0.0.1", - }, 1, "unhealthy: connect to the app: ") - // Listening on port 0 takes any free port, so nothing listens on port 0 // itself. - wantHealthCheck(t, map[string]string{listenAddr: ":0"}, 1, - "unhealthy: ask smallwebwaf: ") - wantHealthCheck(t, map[string]string{listenAddr: "8080"}, 1, - "unhealthy: invalid setting: SWWAF_LISTEN_ADDR: ") + wantHealthCheck(t, map[string]string{ + commandSetting: serveWAFCommand, listenAddr: ":0", + }, 1, "unhealthy: ask servewaf: ") + wantHealthCheck(t, map[string]string{ + commandSetting: serveWAFCommand, listenAddr: "8080", + }, 1, "unhealthy: invalid setting: SWWAF_LISTEN_ADDR: ") stop() @@ -83,6 +83,63 @@ func TestHealthCheck(t *testing.T) { } } +func TestHealthCheckInRProxysContainer(t *testing.T) { + t.Parallel() + + app := httptest.NewServer(http.NotFoundHandler()) + defer app.Close() + + // Nothing listens at the backend. The health check asks rproxy and the app, + // whether or not servewaf answers. + site, _ := startRProxy(t, "http://127.0.0.1:9", app.URL) + _, port, _ := net.SplitHostPort(strings.TrimPrefix(site, "http://")) + settings := func(more ...string) map[string]string { + env := map[string]string{rproxyListenAddr: ":" + port, rproxyUpstreamURL: app.URL} + for i := 0; i < len(more); i += 2 { + env[more[i]] = more[i+1] + } + + return env + } + + wantHealthCheck(t, settings(), 0, "") + wantHealthCheck(t, settings(commandSetting, ""), 0, "") + wantHealthCheck(t, settings(commandSetting, "rproxy"), 0, "") + + // SMALLWEBWAF_COMMAND cannot be given as a file. + wantHealthCheck(t, settings(commandSetting+"_FILE", writeFile(t, "servewaf\n")), + 0, "") + + // Another setting does not fail it, even an invalid one, and neither does + // SWRP_WAF_BACKEND being unset. + wantHealthCheck(t, settings("SWRP_TIMEOUT_MS", "soon"), 0, "") + + // Another test can take a closed server's port at once, so the app's + // address here is a multicast address, to which TCP cannot connect. + wantHealthCheck(t, settings(rproxyUpstreamURL, "http://224.0.0.1"), 1, + "unhealthy: connect to the app: ") + + // Listening on port 0 takes any free port, so nothing listens on port 0 + // itself. + wantHealthCheck(t, settings(rproxyListenAddr, ":0"), 1, "unhealthy: ask rproxy: ") + wantHealthCheck(t, settings(rproxyListenAddr, "8080"), 1, + "unhealthy: invalid setting: SWRP_LISTEN_ADDR: ") + + // No setting of rproxy's has a form given as a file. + wantHealthCheck(t, map[string]string{ + rproxyListenAddr + "_FILE": writeFile(t, ":"+port+"\n"), + rproxyUpstreamURL: app.URL, + }, 1, "unhealthy: ask rproxy: ") +} + +func TestHealthCheckFailsWithAnyOtherCommand(t *testing.T) { + t.Parallel() + + wantHealthCheck(t, map[string]string{commandSetting: "healthcheck"}, 1, + "unhealthy: invalid setting: SMALLWEBWAF_COMMAND: "+ + "\"healthcheck\" is not rproxy or servewaf\n") +} + func TestHealthCheckRefusesAnArgument(t *testing.T) { t.Parallel() diff --git a/internal/smallwebwaf/rproxy_test.go b/internal/smallwebwaf/rproxy_test.go index 4774c0f..6dca918 100644 --- a/internal/smallwebwaf/rproxy_test.go +++ b/internal/smallwebwaf/rproxy_test.go @@ -28,6 +28,8 @@ const ( wafBackend = "SWRP_WAF_BACKEND" // rproxyListenAddr is the setting that says where rproxy listens. rproxyListenAddr = "SWRP_LISTEN_ADDR" + // rproxyUpstreamURL is the setting that says where the app is. + rproxyUpstreamURL = "SWRP_UPSTREAM_URL" // rproxyTimeout is the setting that says how long rproxy waits for servewaf. rproxyTimeout = "SWRP_TIMEOUT_MS" // bannedProbe is a path the default rule file bans a client for. @@ -100,7 +102,7 @@ func startServeWAF(t *testing.T, env map[string]string) (string, *output, func() exited := make(chan int, 1) go func() { - exited <- runAs(ctx, true, env, out) + exited <- run(ctx, env, out) }() addr, _ := out.line(t, "msg", "starting")["address"].(string) diff --git a/internal/smallwebwaf/servewaf_test.go b/internal/smallwebwaf/servewaf_test.go index 6f6f5da..6259523 100644 --- a/internal/smallwebwaf/servewaf_test.go +++ b/internal/smallwebwaf/servewaf_test.go @@ -1,6 +1,7 @@ package smallwebwaf_test import ( + "bytes" "context" "crypto/rand" "io" @@ -33,59 +34,88 @@ func TestMain(m *testing.M) { os.Exit(smallwebwaf.Main(testVersion)) } -func TestMainAnswersQuestionsOnlyAsServeWAF(t *testing.T) { +func TestMainRunsServeWAF(t *testing.T) { t.Parallel() - for _, tc := range []struct { - command, disposition string - }{ - {"servewaf", proxy.DispositionAllow}, - {"", ""}, - } { - t.Run("command "+tc.command, func(t *testing.T) { + binary, err := os.Executable() + if err != nil { + t.Fatalf("find the test binary: %v", err) + } + + ctx, cancel := context.WithTimeout(t.Context(), 3*waitLimit) + defer cancel() + + out := &output{} + cmd := exec.CommandContext(ctx, binary) //nolint:gosec // this test binary + cmd.Stdout = out + + cmd.Env = append(os.Environ(), + commandForMain+"="+serveWAFCommand, + listenAddr+"="+localhost+":0", + stateDir+"="+t.TempDir(), + rulesDir+"="+t.TempDir(), + lookupSource+"=off") + + err = cmd.Start() + if err != nil { + t.Fatalf("start: %v", err) + } + + t.Cleanup(func() { + _ = cmd.Process.Kill() + _ = cmd.Wait() + }) + + addr, _ := out.line(t, "msg", "starting")["address"].(string) + ask(t, addr, "/", http.StatusOK, proxy.DispositionAllow) + + err = cmd.Process.Signal(syscall.SIGTERM) + if err != nil { + t.Fatalf("signal: %v", err) + } + + err = cmd.Wait() + if err != nil { + t.Errorf("exit: %v; output:\n%s", err, out.text()) + } +} + +func TestMainWithoutACommandItKnowsPrintsTheCommands(t *testing.T) { + t.Parallel() + + binary, err := os.Executable() + if err != nil { + t.Fatalf("find the test binary: %v", err) + } + + for _, command := range []string{"", "serve"} { + t.Run("command "+command, func(t *testing.T) { t.Parallel() - binary, err := os.Executable() - if err != nil { - t.Fatalf("find the test binary: %v", err) - } - - ctx, cancel := context.WithTimeout(t.Context(), 3*waitLimit) + ctx, cancel := context.WithTimeout(t.Context(), waitLimit) defer cancel() - out := &output{} - cmd := exec.CommandContext(ctx, binary) //nolint:gosec // this test binary - cmd.Stdout = out + var stdout, stderr bytes.Buffer - cmd.Env = append(os.Environ(), - commandForMain+"="+tc.command, - listenAddr+"="+localhost+":0", - upstreamURL+"="+startApp(t), - stateDir+"="+t.TempDir(), - rulesDir+"="+t.TempDir(), - lookupSource+"=off") + cmd := exec.CommandContext(ctx, binary) + cmd.Stdout, cmd.Stderr = &stdout, &stderr - err = cmd.Start() - if err != nil { - t.Fatalf("start: %v", err) + cmd.Env = append(os.Environ(), commandForMain+"="+command) + + err := cmd.Run() + + if code := cmd.ProcessState.ExitCode(); code != 2 { + t.Errorf("exit status %d (%v), want 2", code, err) } - t.Cleanup(func() { - _ = cmd.Process.Kill() - _ = cmd.Wait() - }) - - addr, _ := out.line(t, "msg", "starting")["address"].(string) - ask(t, addr, "/", http.StatusOK, tc.disposition) - - err = cmd.Process.Signal(syscall.SIGTERM) - if err != nil { - t.Fatalf("signal: %v", err) + for _, name := range []string{"rproxy", serveWAFCommand, "healthcheck"} { + if !strings.Contains(stderr.String(), name) { + t.Errorf("stderr %q does not name %s", stderr.String(), name) + } } - err = cmd.Wait() - if err != nil { - t.Errorf("exit: %v; output:\n%s", err, out.text()) + if stdout.Len() != 0 { + t.Errorf("stdout %q, want nothing", stdout.String()) } }) } @@ -134,7 +164,7 @@ func runServeWAF(t *testing.T, env map[string]string, use func(addr string)) { exited := make(chan int, 1) go func() { - exited <- runAs(ctx, true, env, out) + exited <- run(ctx, env, out) }() addr, _ := out.line(t, "msg", "starting")["address"].(string) @@ -172,39 +202,3 @@ func TestServeWAFKeepsItsBansAcrossARestartAndNeedsNoApp(t *testing.T) { ask(t, addr, "/", http.StatusForbidden, proxy.DispositionDeny) }) } - -func TestWithoutACommandTheClientHeaderIsNoQuestion(t *testing.T) { - t.Parallel() - - runUntilStopped(t, map[string]string{ - listenAddr: localhost + ":0", - upstreamURL: startApp(t), - stateDir: t.TempDir(), - rulesDir: t.TempDir(), - }, func(url string) { - req, err := http.NewRequestWithContext(t.Context(), http.MethodGet, url, - http.NoBody) - if err != nil { - t.Fatalf("new request: %v", err) - } - - req.Header.Set(proxy.ClientHeader, placed) - - transport := &http.Transport{} - defer transport.CloseIdleConnections() - - res, err := (&http.Client{Transport: transport}).Do(req) - if err != nil { - t.Fatalf("request: %v", err) - } - - body, err := io.ReadAll(res.Body) - _ = res.Body.Close() - - if err != nil || string(body) != greeting || - res.Header.Get(proxy.DispositionHeader) != "" { - t.Errorf("answered %q (%v) with disposition %q, want the app's greeting", - body, err, res.Header.Get(proxy.DispositionHeader)) - } - }) -} diff --git a/internal/smallwebwaf/smallwebwaf.go b/internal/smallwebwaf/smallwebwaf.go index 3e6809a..947cce1 100644 --- a/internal/smallwebwaf/smallwebwaf.go +++ b/internal/smallwebwaf/smallwebwaf.go @@ -1,15 +1,16 @@ -// Package smallwebwaf runs the smallwebwaf process: it reads the settings, -// the rule files, the lookup database and the state files, serves requests -// until it is told to stop, and then stops in an orderly way, writing the -// state files. It serves them in front of an app, or, as `smallwebwaf -// servewaf`, by answering rproxy's questions about them. As `smallwebwaf -// rproxy`, it instead reads rproxy's settings alone, and runs the reverse proxy -// in front of one app that asks servewaf about each request. +// Package smallwebwaf runs the smallwebwaf process, as the command it is given. +// `smallwebwaf servewaf` reads the settings, the rule files, the lookup database +// and the state files, answers rproxy's questions until it is told to stop, and +// then stops in an orderly way, writing the state files. `smallwebwaf rproxy` +// reads rproxy's settings alone, and runs the reverse proxy in front of one app +// that asks servewaf about each request. `smallwebwaf healthcheck` is the +// container's health check. package smallwebwaf import ( "context" "errors" + "fmt" "io" "log/slog" "net" @@ -35,12 +36,25 @@ import ( // runit and docker wait a little longer before they kill the process. const shutdownTimeout = 5 * time.Second -// remoteLogStopTimeout is how long, as smallwebwaf stops, the log lines -// still waiting are sent to SWWAF_LOG_REMOTE_URL before they are given -// up. stdout has carried them. +// remoteLogStopTimeout is how long, as servewaf stops, the log lines still +// waiting are sent to SWWAF_LOG_REMOTE_URL before they are given up. stdout has +// carried them. const remoteLogStopTimeout = 2 * time.Second -// Params are what Run needs from the process. +// usageStatus is the exit status with no command, or one smallwebwaf does not +// know. +const usageStatus = 2 + +// usage is what smallwebwaf prints then. +const usage = `usage: smallwebwaf + +commands: + rproxy the reverse proxy in front of one app + servewaf decides for rproxy, and keeps bans, counters and history + healthcheck the container's health check +` + +// Params are what RunServeWAF and RunRProxy need from the process. type Params struct { // Version is the version of the binary, set when it is built. Version string @@ -48,44 +62,47 @@ type Params struct { LookupEnv func(string) (string, bool) // Stdout receives the request log and the process's own messages. Stdout io.Writer - // ServeWAF makes it `smallwebwaf servewaf`, which answers rproxy's - // questions, rather than the proxy in front of an app. - ServeWAF bool } -// Main runs smallwebwaf until SIGTERM or SIGINT, and returns the -// process's exit status. Run as `smallwebwaf servewaf`, it answers -// rproxy's questions rather than pass requests to an app. Run as -// `smallwebwaf rproxy`, it is the reverse proxy that asks those questions. -// Run as `smallwebwaf healthcheck`, it is the container's health check -// instead. +// Main runs the command os.Args names, and returns the process's exit status. +// `smallwebwaf rproxy` and `smallwebwaf servewaf` run until SIGTERM or SIGINT. +// With no command, or one it does not know, it prints the commands and returns +// usageStatus. func Main(version string) int { - if len(os.Args) > 1 && os.Args[1] == "healthcheck" { - return HealthCheck(context.Background(), os.Args[2:], os.LookupEnv, os.Stderr) + args := os.Args[1:] + if len(args) == 0 { + return printUsage(os.Stderr) } ctx, stop := signal.NotifyContext(context.Background(), syscall.SIGTERM, os.Interrupt) defer stop() - params := Params{ - Version: version, - LookupEnv: os.LookupEnv, - Stdout: os.Stdout, - ServeWAF: len(os.Args) > 1 && os.Args[1] == "servewaf", - } + params := Params{Version: version, LookupEnv: os.LookupEnv, Stdout: os.Stdout} - if len(os.Args) > 1 && os.Args[1] == "rproxy" { + switch args[0] { + case "rproxy": return RunRProxy(ctx, params) + case "servewaf": + return RunServeWAF(ctx, params) + case "healthcheck": + return HealthCheck(ctx, args[1:], os.LookupEnv, os.Stderr) + default: + return printUsage(os.Stderr) } - - return Run(ctx, params) } -// Run reads the settings, the rule files, the lookup database and the -// state files, then serves requests until ctx is done. It returns the -// process's exit status, 1 when smallwebwaf cannot start. -func Run(ctx context.Context, params Params) int { +// printUsage writes the commands to w, and returns usageStatus. +func printUsage(w io.Writer) int { + _, _ = fmt.Fprint(w, usage) + + return usageStatus +} + +// RunServeWAF is `smallwebwaf servewaf`: it reads the settings, the rule files, +// the lookup database and the state files, then serves requests until ctx is +// done. It returns the process's exit status, 1 when servewaf cannot start. +func RunServeWAF(ctx context.Context, params Params) int { // Until the settings are read, the one message is an invalid setting's // error, which every SWWAF_LOG_LEVEL lets through. processLog := requestlog.NewProcessLogger(params.Stdout, @@ -133,12 +150,7 @@ func Run(ctx context.Context, params Params) int { return 1 } - newProxy := proxy.New - if params.ServeWAF { - newProxy = proxy.NewServeWAF - } - - server, err := newServer(newProxy, cfg, stdout, processLog, now, ruleFiles, alertQueue) + server, err := newServer(cfg, stdout, processLog, now, ruleFiles, alertQueue) if err != nil { processLog.Error("cannot use the lookup database", "error", err.Error()) @@ -172,12 +184,10 @@ func Run(ctx context.Context, params Params) int { return serve(ctx, server, listener, files, ruleFiles, alertQueue, processLog) } -// newServer returns the server smallwebwaf runs, which newProxy makes, with -// the metrics of the alerts, after reading the lookup database while -// SWWAF_LOOKUP_SOURCE is file. A lookup database that cannot be read is an -// error. +// newServer returns the server servewaf runs, with the metrics of the alerts, +// after reading the lookup database while SWWAF_LOOKUP_SOURCE is file. A lookup +// database that cannot be read is an error. func newServer( - newProxy func(proxy.Params) *proxy.Server, cfg *config.Config, stdout io.Writer, processLog *slog.Logger, now func() time.Time, ruleFiles *rules.Files, alertQueue *alerts.Queue, ) (*proxy.Server, error) { @@ -197,7 +207,7 @@ func newServer( } } - server := newProxy(proxy.Params{ + server := proxy.NewServeWAF(proxy.Params{ Config: cfg, RequestLog: stdout, ProcessLog: processLog, @@ -361,12 +371,10 @@ func serve( // Run and Watch have ended, so nothing else reads or writes the // files, and no alert is being sent, so that alerts.json keeps every - // alert not yet sent. Every request has ended too, but for two kinds - // that Go's server does not wait for: one cut off because Shutdown - // timed out, and one whose connection switched protocols, such as a - // WebSocket. Such a request adds to its client's history only as it - // ends, which can be after this write, and then that request is - // missing from clients.json. + // alert not yet sent. Every request has ended too, but for one cut off + // because Shutdown timed out, which adds to its client's history only as + // it ends. That can be after this write, and then that request is missing + // from clients.json. <-written <-watched <-rulesWatched diff --git a/internal/smallwebwaf/smallwebwaf_test.go b/internal/smallwebwaf/smallwebwaf_test.go index 610c957..44d560c 100644 --- a/internal/smallwebwaf/smallwebwaf_test.go +++ b/internal/smallwebwaf/smallwebwaf_test.go @@ -3,6 +3,7 @@ package smallwebwaf_test import ( "bytes" "context" + "crypto/rand" "encoding/json" "io" "net" @@ -19,6 +20,7 @@ import ( "time" "sneak.berlin/go/smallwebwaf/internal/lookup/lookuptest" + "sneak.berlin/go/smallwebwaf/internal/proxy" "sneak.berlin/go/smallwebwaf/internal/smallwebwaf" ) @@ -32,8 +34,6 @@ const ( // localhost is where the tests listen. localhost = "127.0.0.1" listenAddr = "SWWAF_LISTEN_ADDR" - upstreamURL = "SWWAF_UPSTREAM_URL" - trustedProxies = "SWWAF_TRUSTED_PROXIES" stateDir = "SWWAF_STATE_DIR" stateWriteDelay = "SWWAF_STATE_WRITE_DELAY" stateCounterInterval = "SWWAF_STATE_COUNTER_INTERVAL" @@ -122,18 +122,11 @@ func (o *output) text() string { return o.buf.String() } -// run runs smallwebwaf with the settings in env until ctx is done, and -// returns its exit status. SWWAF_LOOKUP_SOURCE is off unless env sets it, +// run runs `smallwebwaf servewaf` with the settings in env until ctx is done, +// and returns its exit status. SWWAF_LOOKUP_SOURCE is off unless env sets it, // so that no test sends GeoJS its clients' addresses. func run(ctx context.Context, env map[string]string, out *output) int { - return runAs(ctx, false, env, out) -} - -// runAs is run as `smallwebwaf servewaf` while serveWAF is true. -func runAs( - ctx context.Context, serveWAF bool, env map[string]string, out *output, -) int { - return smallwebwaf.Run(ctx, smallwebwaf.Params{ + return smallwebwaf.RunServeWAF(ctx, smallwebwaf.Params{ Version: testVersion, LookupEnv: func(name string) (string, bool) { value, ok := env[name] @@ -143,8 +136,7 @@ func runAs( return value, ok }, - Stdout: out, - ServeWAF: serveWAF, + Stdout: out, }) } @@ -154,8 +146,8 @@ func TestInvalidSettingStopsTheStart(t *testing.T) { out := &output{} status := run(t.Context(), map[string]string{ - "SWWAF_REQUEST_MAX_BYTES": "lots", - instanceName: instance, + "SWWAF_BYTES_LIMIT_PER_MINUTE": "lots", + instanceName: instance, }, out) if status != 1 { t.Errorf("exit status %d, want 1", status) @@ -166,7 +158,7 @@ func TestInvalidSettingStopsTheStart(t *testing.T) { if line["type"] != "process" || line["instance"] != instance || line["level"] != "ERROR" || - !strings.HasPrefix(message, "SWWAF_REQUEST_MAX_BYTES: ") { + !strings.HasPrefix(message, "SWWAF_BYTES_LIMIT_PER_MINUTE: ") { t.Errorf("start refused with %v", line) } } @@ -228,7 +220,6 @@ func TestAddressInUseStopsTheStart(t *testing.T) { func TestServesUntilToldToStop(t *testing.T) { t.Parallel() - appURL := startApp(t) dir := t.TempDir() ctx, stop := context.WithCancel(t.Context()) @@ -237,10 +228,9 @@ func TestServesUntilToldToStop(t *testing.T) { go func() { exited <- run(ctx, map[string]string{ - listenAddr: localhost + ":0", - upstreamURL: appURL, - stateDir: dir, - rulesDir: filepath.Join("..", "..", "share", "rules.d"), + listenAddr: localhost + ":0", + stateDir: dir, + rulesDir: filepath.Join("..", "..", "share", "rules.d"), }, out) }() @@ -250,10 +240,10 @@ func TestServesUntilToldToStop(t *testing.T) { } starting := out.line(t, "msg", "starting") - wantStartingLine(t, starting, appURL, dir) + wantStartingLine(t, starting, dir) addr, _ := starting["address"].(string) - wantGreeting(t, "http://"+addr+"/") + wantAllowed(t, "http://"+addr+"/") out.line(t, "type", "request") stop() @@ -325,7 +315,6 @@ func TestEveryLogLineAndMetricCarriesTheInstanceName(t *testing.T) { env := map[string]string{ listenAddr: localhost + ":0", - upstreamURL: startApp(t), stateDir: t.TempDir(), rulesDir: t.TempDir(), metricsToken: token, @@ -363,7 +352,6 @@ func TestStateKeptAcrossRestarts(t *testing.T) { env := map[string]string{ listenAddr: localhost + ":0", - upstreamURL: startApp(t), stateDir: t.TempDir(), rulesDir: t.TempDir(), rateLimitPerDay: "2", @@ -375,8 +363,8 @@ func TestStateKeptAcrossRestarts(t *testing.T) { // The two requests a day allows, and a stop. runUntilStopped(t, env, func(url string) { - wantGreeting(t, url) - wantGreeting(t, url) + wantAllowed(t, url) + wantAllowed(t, url) }) // After a restart the client has no fresh allowance: its third @@ -400,10 +388,8 @@ func TestBanRefusesItsNetblockAfterARestartWithAnotherScope(t *testing.T) { env := map[string]string{ listenAddr: localhost + ":0", - upstreamURL: startApp(t), stateDir: t.TempDir(), rulesDir: t.TempDir(), - trustedProxies: localhost + "/32", rateLimitPerDay: "1", scope: "24", } @@ -447,11 +433,9 @@ func TestBanAddedAndLiftedByEditingBansJSON(t *testing.T) { dir := t.TempDir() env := map[string]string{ - listenAddr: localhost + ":0", - upstreamURL: startApp(t), - stateDir: dir, - rulesDir: t.TempDir(), - trustedProxies: localhost + "/32", + listenAddr: localhost + ":0", + stateDir: dir, + rulesDir: t.TempDir(), // No write comes due in the test, so only the watch on the // directory can take the edits in. stateWriteDelay: "1h", @@ -472,12 +456,10 @@ func TestBanAddedAndLiftedThroughTheEndpointsKeptInBansJSON(t *testing.T) { dir := t.TempDir() env := map[string]string{ - listenAddr: localhost + ":0", - upstreamURL: startApp(t), - stateDir: dir, - rulesDir: t.TempDir(), - trustedProxies: localhost + "/32", - adminToken: adminSecret, + listenAddr: localhost + ":0", + stateDir: dir, + rulesDir: t.TempDir(), + adminToken: adminSecret, // Neither comes due in the test: the files are written as // smallwebwaf stops. stateWriteDelay: "1h", @@ -517,17 +499,16 @@ func TestRuleFileAddedWhileRunningTakesEffect(t *testing.T) { dir := t.TempDir() env := map[string]string{ - listenAddr: localhost + ":0", - upstreamURL: startApp(t), - stateDir: t.TempDir(), - rulesDir: dir, + listenAddr: localhost + ":0", + stateDir: t.TempDir(), + rulesDir: dir, // The requests sent until the rule takes effect must not break a // rate limit, whose ban would refuse them too. "SWWAF_RATE_LIMIT_PER_MINUTE": "off", } out := runUntilStopped(t, env, func(url string) { - wantGreeting(t, url) + wantAllowed(t, url) // Written once: each change would start the rule files' wait // again. A file written before smallwebwaf watches the directory is @@ -580,10 +561,8 @@ func TestLookupDatabaseReplacedWhileRunningTakesEffect(t *testing.T) { env := map[string]string{ listenAddr: localhost + ":0", - upstreamURL: startApp(t), stateDir: t.TempDir(), rulesDir: t.TempDir(), - trustedProxies: localhost + "/32", metricsToken: token, instanceName: instance, lookupSource: "file", @@ -679,10 +658,8 @@ func TestBlocklistTriesAndCopiesKeptInReputationJSONAcrossRestarts(t *testing.T) dir := t.TempDir() env := map[string]string{ listenAddr: localhost + ":0", - upstreamURL: startApp(t), stateDir: dir, rulesDir: t.TempDir(), - trustedProxies: localhost + "/32", metricsToken: token, instanceName: instance, "SWWAF_BLOCKLIST_URLS": torURL, @@ -763,10 +740,8 @@ func TestCrowdSecDecisionListKeptInReputationJSONAcrossARestart(t *testing.T) { dir := t.TempDir() env := map[string]string{ listenAddr: localhost + ":0", - upstreamURL: startApp(t), stateDir: dir, rulesDir: t.TempDir(), - trustedProxies: localhost + "/32", "SWWAF_CROWDSEC_LAPI_URL": engine.URL, "SWWAF_CROWDSEC_LAPI_KEY": bouncerKey, } @@ -864,14 +839,13 @@ func TestEveryLineIsAlsoSentToTheRemoteLogEndpoint(t *testing.T) { env := map[string]string{ listenAddr: localhost + ":0", - upstreamURL: startApp(t), stateDir: t.TempDir(), rulesDir: t.TempDir(), "SWWAF_LOG_REMOTE_URL": "syslog+tcp://" + endpoint.Addr().String(), } out := runUntilStopped(t, env, func(url string) { - wantGreeting(t, url) + wantAllowed(t, url) }) out.line(t, "type", "request") @@ -920,7 +894,6 @@ func TestStalledRemoteLogEndpointHoldsUpNoRequest(t *testing.T) { env := map[string]string{ listenAddr: localhost + ":0", - upstreamURL: startApp(t), stateDir: t.TempDir(), rulesDir: t.TempDir(), "SWWAF_LOG_REMOTE_URL": "syslog+tls://" + endpoint.Addr().String(), @@ -930,7 +903,7 @@ func TestStalledRemoteLogEndpointHoldsUpNoRequest(t *testing.T) { } out := runUntilStopped(t, env, func(url string) { - wantGreeting(t, url) + wantAllowed(t, url) // More than one line has been written, and the buffer holds the // last. @@ -980,7 +953,6 @@ func TestBanIsAlertedAndAnAlertNotSentIsKeptAcrossARestart(t *testing.T) { dir := t.TempDir() env := map[string]string{ listenAddr: localhost + ":0", - upstreamURL: startApp(t), stateDir: dir, rulesDir: rules, "SWWAF_ALERT_WEBHOOK_URL": webhook.url, @@ -1065,10 +1037,8 @@ func TestBanIsAlertedToSlackAndNtfy(t *testing.T) { slack, ntfy := startDestination(t), startDestination(t) env := map[string]string{ listenAddr: localhost + ":0", - upstreamURL: startApp(t), stateDir: t.TempDir(), rulesDir: t.TempDir(), - trustedProxies: localhost + "/32", rateLimitPerDay: "1", // The metrics are read from 127.0.0.1, which no limit counts. "SWWAF_ALLOW_NETS": localhost + "/32", @@ -1208,8 +1178,8 @@ func startApp(t *testing.T) string { return app.URL } -// runUntilStopped runs smallwebwaf with the settings in env, has use send -// it requests at url, then stops it as SIGTERM does, checks that it +// runUntilStopped runs `smallwebwaf servewaf` with the settings in env, has use +// send it requests at url, then stops it as SIGTERM does, checks that it // stopped in order, and returns its output. func runUntilStopped( t *testing.T, env map[string]string, use func(url string), @@ -1242,43 +1212,33 @@ func runUntilStopped( // wantStartingLine checks that the line at start gives the version and // every setting's value. -func wantStartingLine(t *testing.T, line map[string]any, appURL, dir string) { +func wantStartingLine(t *testing.T, line map[string]any, dir string) { t.Helper() settings, _ := line["settings"].(map[string]any) want := map[string]any{ - listenAddr: localhost + ":0", - upstreamURL: appURL, - stateDir: dir, - "SWWAF_MODE": "enforce", - stateWriteDelay: "10s", - stateCounterInterval: "15m", - trustedProxies: "10.0.0.0/8,172.16.0.0/12,192.168.0.0/16", - "SWWAF_CLIENT_REQUEST_TIMEOUT": "60s", - "SWWAF_CLIENT_REQUEST_HEADER_MAX_BYTES": "32K", - "SWWAF_CLIENT_IDLE_TIMEOUT": "120s", - "SWWAF_CLIENT_RESPONSE_TIMEOUT": "30m", - "SWWAF_UPSTREAM_REQUEST_TIMEOUT": "60s", - "SWWAF_UPSTREAM_RESPONSE_TIMEOUT": "30m", - "SWWAF_REQUEST_MAX_BYTES": "100M", - "SWWAF_RESPONSE_MAX_BYTES": "1G", - "SWWAF_ALLOW_NETS": "", - rateLimitExemptNets: "", - "SWWAF_DENY_NETS": "", - "SWWAF_RATE_LIMIT_PER_MINUTE": "500", - "SWWAF_RATE_LIMIT_PER_HOUR": "20000", - rateLimitPerDay: "300000", - "SWWAF_RATE_LIMIT_EXEMPT_PATHS": "", - "SWWAF_DENIED_COUNTRIES": "", - "SWWAF_EXCLUSIVELY_ALLOWED_COUNTRIES": "", - "SWWAF_BAN_RESPONSE": "403", - "SWWAF_LIMIT_BAN_DURATION": "1h", - "SWWAF_LIMIT_BAN_REPEAT_WINDOW": "24h", - "SWWAF_MAX_BAN_DURATION": "7d", - "SWWAF_ATTACK_BAN_DURATION": "7d", - "SWWAF_MAX_BANS": "5000", - "SWWAF_BAN_SCOPE_V4_PREFIX": "32", - "SWWAF_RULES_ENABLED": "true", + listenAddr: localhost + ":0", + stateDir: dir, + "SWWAF_MODE": "enforce", + stateWriteDelay: "10s", + stateCounterInterval: "15m", + "SWWAF_ALLOW_NETS": "", + rateLimitExemptNets: "", + "SWWAF_DENY_NETS": "", + "SWWAF_RATE_LIMIT_PER_MINUTE": "500", + "SWWAF_RATE_LIMIT_PER_HOUR": "20000", + rateLimitPerDay: "300000", + "SWWAF_RATE_LIMIT_EXEMPT_PATHS": "", + "SWWAF_DENIED_COUNTRIES": "", + "SWWAF_EXCLUSIVELY_ALLOWED_COUNTRIES": "", + "SWWAF_BAN_RESPONSE": "403", + "SWWAF_LIMIT_BAN_DURATION": "1h", + "SWWAF_LIMIT_BAN_REPEAT_WINDOW": "24h", + "SWWAF_MAX_BAN_DURATION": "7d", + "SWWAF_ATTACK_BAN_DURATION": "7d", + "SWWAF_MAX_BANS": "5000", + "SWWAF_BAN_SCOPE_V4_PREFIX": "32", + "SWWAF_RULES_ENABLED": "true", } for name, value := range want { @@ -1292,30 +1252,11 @@ func wantStartingLine(t *testing.T, line map[string]any, appURL, dir string) { } } -// wantGreeting checks that a request to url gets the app's answer. -func wantGreeting(t *testing.T, url string) { +// wantAllowed checks that servewaf at url allows a request from localhost. +func wantAllowed(t *testing.T, url string) { t.Helper() - req, err := http.NewRequestWithContext(t.Context(), http.MethodGet, url, - http.NoBody) - if err != nil { - t.Fatalf("new request: %v", err) - } - - transport := &http.Transport{} - defer transport.CloseIdleConnections() - - res, err := (&http.Client{Transport: transport}).Do(req) - if err != nil { - t.Fatalf("request: %v", err) - } - - body, err := io.ReadAll(res.Body) - _ = res.Body.Close() - - if err != nil || string(body) != greeting { - t.Errorf("got %q (%v), want the app's answer", body, err) - } + wantStatus(t, url, localhost, http.StatusOK) } // messages returns the message of each record in received, octet-counted @@ -1496,34 +1437,16 @@ func onlyBan(t *testing.T, dir string) map[string]any { return file.Bans[0] } -// wantRefused checks that a request to url is refused with 403, the -// default SWWAF_BAN_RESPONSE. +// wantRefused checks that servewaf at url refuses a request from localhost with +// 403, the default SWWAF_BAN_RESPONSE. func wantRefused(t *testing.T, url string) { t.Helper() - req, err := http.NewRequestWithContext(t.Context(), http.MethodGet, url, - http.NoBody) - if err != nil { - t.Fatalf("new request: %v", err) - } - - transport := &http.Transport{} - defer transport.CloseIdleConnections() - - res, err := (&http.Client{Transport: transport}).Do(req) - if err != nil { - t.Fatalf("request: %v", err) - } - - _ = res.Body.Close() - - if res.StatusCode != http.StatusForbidden { - t.Errorf("status %d, want %d", res.StatusCode, http.StatusForbidden) - } + wantStatus(t, url, localhost, http.StatusForbidden) } -// wantStatus checks that a request to url from the client at from, as -// X-Forwarded-For names it, is answered with status. +// wantStatus checks that a request to url from the client at from, as statusFrom +// sends it, is answered with status. func wantStatus(t *testing.T, url, from string, status int) { t.Helper() @@ -1690,8 +1613,10 @@ func (w *webhook) waitFor(t *testing.T, event string, answered bool) webhookPost } } -// statusFrom returns the status a request to url from the client at -// from, as X-Forwarded-For names it, is answered with. +// statusFrom returns the status a request to url from the client at from is +// answered with. The request serves rproxy and servewaf alike. rproxy reads the +// client from X-Forwarded-For, and replaces the other two headers. To servewaf, +// they make it a question about that client. func statusFrom(t *testing.T, url, from string) int { t.Helper() @@ -1702,6 +1627,8 @@ func statusFrom(t *testing.T, url, from string) int { } req.Header.Set("X-Forwarded-For", from) + req.Header.Set(proxy.ClientHeader, from) + req.Header.Set("X-Request-ID", rand.Text()) transport := &http.Transport{} defer transport.CloseIdleConnections() diff --git a/internal/state/state_test.go b/internal/state/state_test.go index 97bfccf..891723d 100644 --- a/internal/state/state_test.go +++ b/internal/state/state_test.go @@ -648,6 +648,27 @@ func TestClientWithRefusalsInTheMinuteWithoutTheirStartStopsTheStart(t *testing. `: entry 1 has no "minute_refusals.start"`) } +func TestClientsJSONFromAnEarlierVersionWithTokenRefusalsLoads(t *testing.T) { + t.Parallel() + + dir := t.TempDir() + + err := os.WriteFile(filepath.Join(dir, clientsJSON), + []byte(`{"version": 1, "clients": [{"client": "203.0.113.9/32", `+ + `"history": {"offences": {"token_refused": 2}}}]}`), 0o600) + if err != nil { + t.Fatalf("write %s: %v", clientsJSON, err) + } + + params := newParams(dir) + load(t, params) + + clients := params.Limiter.Snapshot() + if len(clients) != 1 || clients[0].History.Offences.TokenRefused != 2 { + t.Errorf("clients %+v, want one with its two refusals kept", clients) + } +} + func TestReputationJSONEntryWithoutAFieldItNeedsStopsTheStart(t *testing.T) { t.Parallel() diff --git a/internal/waf/waf.go b/internal/waf/waf.go index ad8a4e6..575b184 100644 --- a/internal/waf/waf.go +++ b/internal/waf/waf.go @@ -212,8 +212,7 @@ type Result struct { // method, its URL with the query, and its headers, the Cookie header // without the cookies in cookiesNotRead, and on body, r's body as the // caller has it, as readBody reads it. It returns what it found, what it -// read of body, which the app is still to be sent, and the error that -// ended the reading early, if one did. +// read of body, and the error that ended the reading early, if one did. func (c *CoreRuleSet) Inspect( r *http.Request, client netip.Addr, body io.Reader, ) (Result, []byte, error) { diff --git a/script/example-app b/script/example-app index c668d2a..da3eafd 100755 --- a/script/example-app +++ b/script/example-app @@ -1,19 +1,28 @@ #!/bin/sh # script/example-app: build the image, and on it the example app in -# deploy/example-app, then run the app's container with a volume for the -# state files and check that the health check passes, that a request is -# served through smallwebwaf, that a second one in a minute bans the -# client, that a probe for /.env bans another client, which its next -# request bans for good, that `sv stop` stops smallwebwaf in order, that -# `docker stop` stops the container without having to kill it, that a -# new container on the same volume still refuses the banned client, that -# a container with its state files on a new volume mounted at the -# directory SWWAF_STATE_DIR_FILE names becomes healthy, and that one whose -# SWWAF_STATE_DIR does not exist logs smallwebwaf's message about it. The -# containers run with SWWAF_LOOKUP_SOURCE=off, so that no address is sent -# to GeoJS. The containers, the volumes and both images are removed -# however the script ends. Building the app needs network access, for -# nixpkgs' binary cache. script/check does not run this. +# deploy/example-app, then run the app's container, with rproxy in it, and +# servewaf in a container of its own with a volume for the state files. It +# checks: +# - that the health checks pass; +# - that a request is served through rproxy, and a second one in a minute +# bans the client; +# - that a probe for /.env bans another client, which its next request bans +# for good; +# - that only servewaf's container changes the state directory's owner; +# - that the app still answers through rproxy with servewaf stopped, and +# that once servewaf has started again, bans apply again; +# - that `sv stop` stops rproxy in order, and `docker stop` stops each +# container without having to kill it; +# - that a new servewaf container on the same volume still refuses the banned +# client; +# - that a servewaf container with its state files on a new volume mounted at +# the directory SWWAF_STATE_DIR_FILE names becomes healthy; +# - that one whose SWWAF_STATE_DIR does not exist logs servewaf's message +# about it. +# servewaf runs with SWWAF_LOOKUP_SOURCE=off, so that no address is sent to +# GeoJS. The containers, the network, the volumes and both images are removed +# however the script ends. Building the app needs network access, for nixpkgs' +# binary cache. script/check does not run this. set -eu SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd -P)" @@ -24,7 +33,9 @@ ROOT="$(cd "$SCRIPT_DIR/.." && pwd -P)" NAME="$("$SCRIPT_DIR/projectname")-example-$$" IMAGE="$NAME-base" APP_IMAGE="$NAME-app" -CONTAINER="$NAME" +NETWORK="$NAME" +APP="$NAME-app-container" +SERVEWAF="$NAME-servewaf" VOLUME="$NAME-state" # The volumes of the container whose state directory SWWAF_STATE_DIR_FILE # names: its state files, and the file that names their directory. @@ -32,15 +43,18 @@ MOVED_VOLUME="$NAME-moved-state" SETTINGS_VOLUME="$NAME-settings" cleanup() { - docker rm --force "$CONTAINER" >/dev/null 2>&1 || true + docker rm --force "$APP" "$SERVEWAF" >/dev/null 2>&1 || true + docker network rm "$NETWORK" >/dev/null 2>&1 || true docker volume rm --force "$VOLUME" "$MOVED_VOLUME" "$SETTINGS_VOLUME" \ >/dev/null 2>&1 || true docker rmi --force "$APP_IMAGE" "$IMAGE" >/dev/null 2>&1 || true } fail() { - echo "example-app: $*; the container's output:" >&2 - docker logs "$CONTAINER" >&2 || true + echo "example-app: $*; the app container's output:" >&2 + docker logs "$APP" >&2 || true + echo "example-app: servewaf's output:" >&2 + docker logs "$SERVEWAF" >&2 || true exit 1 } @@ -57,52 +71,73 @@ wait_for() { done } +# healthy : docker says the container's health check passes. healthy() { - status="$(docker inspect --format '{{.State.Health.Status}}' "$CONTAINER")" + status="$(docker inspect --format '{{.State.Health.Status}}' "$1")" [ "$status" = healthy ] } -# logged ...: a line of the container's output holds every text, -# in any order. +# logged ...: a line of the container's output holds every +# text, in any order. logged() { - lines="$(docker logs "$CONTAINER" 2>&1)" + lines="$(docker logs "$1" 2>&1)" + shift for text in "$@"; do lines="$(printf '%s\n' "$lines" | grep -F "$text")" || return 1 done } -# start_container: run the app's container, with the state files on the -# volume, a rate limit of one request a minute and no client looked up, -# and wait until it is healthy. -start_container() { - docker run --detach --name "$CONTAINER" --publish 127.0.0.1::8080 \ +# start_servewaf: run servewaf from the image, with the state files on the +# volume, a rate limit of one request a minute and no client looked up, and wait +# until it is healthy. +start_servewaf() { + docker run --detach --name "$SERVEWAF" --network "$NETWORK" \ --volume "$VOLUME:/var/lib/smallwebwaf" \ + --env SMALLWEBWAF_COMMAND=servewaf \ --env SWWAF_RATE_LIMIT_PER_MINUTE=1 \ --env SWWAF_LOOKUP_SOURCE=off \ - "$APP_IMAGE" >/dev/null - wait_for "the health check did not pass" healthy - address="$(docker port "$CONTAINER" 8080/tcp)" + "$IMAGE" >/dev/null + wait_for "servewaf's health check did not pass" healthy "$SERVEWAF" } -# start_moved_container: run the app's container with the state files on -# a new volume at /srv/state, which SWWAF_STATE_DIR_FILE names in a file -# only the smallwebwaf user can read, as under "Settings given as files" -# in docs/settings.md, and wait until it is healthy. smallwebwaf can write the -# new volume only once the run script has given it to that user. -start_moved_container() { +# start_app: run the app's container, with rproxy in it, and wait until it is +# healthy. +start_app() { + docker run --detach --name "$APP" --network "$NETWORK" --publish 127.0.0.1::8080 \ + --env SWRP_WAF_BACKEND="http://$SERVEWAF:8080" \ + "$APP_IMAGE" >/dev/null + wait_for "the app's health check did not pass" healthy "$APP" + address="$(docker port "$APP" 8080/tcp)" +} + +# start_moved_servewaf: run servewaf with the state files on a new volume at +# /srv/state, which SWWAF_STATE_DIR_FILE names in a file only the smallwebwaf +# user can read, as under "Settings given as files" in docs/settings.md, and +# wait until it is healthy. servewaf can write the new volume only once the run +# script has given it to that user. +start_moved_servewaf() { docker volume create "$MOVED_VOLUME" >/dev/null docker volume create "$SETTINGS_VOLUME" >/dev/null docker run --rm --volume "$SETTINGS_VOLUME:/settings" \ --entrypoint bash "$IMAGE" -c 'echo /srv/state > /settings/state-dir && chown smallwebwaf:smallwebwaf /settings/state-dir && chmod 0400 /settings/state-dir' - docker run --detach --name "$CONTAINER" \ + docker run --detach --name "$SERVEWAF" --network "$NETWORK" \ --volume "$MOVED_VOLUME:/srv/state" \ --volume "$SETTINGS_VOLUME:/etc/smallwebwaf/settings:ro" \ + --env SMALLWEBWAF_COMMAND=servewaf \ --env SWWAF_STATE_DIR_FILE=/etc/smallwebwaf/settings/state-dir \ --env SWWAF_LOOKUP_SOURCE=off \ - "$APP_IMAGE" >/dev/null - wait_for "the health check did not pass" healthy + "$IMAGE" >/dev/null + wait_for "servewaf's health check did not pass" healthy "$SERVEWAF" +} + +# status_from : print the status a request for path from client, +# as X-Forwarded-For names it, gets. rproxy believes the header from docker's +# gateway, a private address. +status_from() { + curl --silent --output /dev/null --write-out '%{http_code}' \ + --max-time 10 --header "X-Forwarded-For: $1" "http://$address$2" || true } # refused: a request to the container gets 403, SWWAF_BAN_RESPONSE's @@ -113,13 +148,14 @@ refused() { [ "$code" = 403 ] } -# refused_from : a request for path from client, as -# X-Forwarded-For names it, gets 403. smallwebwaf believes the header -# from docker's gateway, a private address. +# refused_from : a request for path from client gets 403. refused_from() { - code="$(curl --silent --output /dev/null --write-out '%{http_code}' \ - --max-time 10 --header "X-Forwarded-For: $1" "http://$address$2")" || true - [ "$code" = 403 ] + [ "$(status_from "$1" "$2")" = 403 ] +} + +# owner_of_state_dir : print who owns /var/lib/smallwebwaf there. +owner_of_state_dir() { + docker exec "$1" stat --format '%U' /var/lib/smallwebwaf } main() { @@ -131,58 +167,88 @@ main() { docker build --no-cache --build-arg SMALLWEBWAF_IMAGE="$IMAGE" \ -t "$APP_IMAGE" deploy/example-app + docker network create "$NETWORK" >/dev/null docker volume create "$VOLUME" >/dev/null - start_container - echo "example-app: the health check passes" + start_servewaf + start_app + echo "example-app: the health checks pass" + + [ "$(owner_of_state_dir "$SERVEWAF")" = smallwebwaf ] || + fail "servewaf's container did not give the state directory to smallwebwaf" + [ "$(owner_of_state_dir "$APP")" = root ] || + fail "the app's container changed the state directory's owner" + echo "example-app: only servewaf's container changes the state directory's owner" page="$(curl --fail --silent --show-error --max-time 10 "http://$address/")" || fail "no answer on port 8080" [ "$page" = "hello from the example app" ] || fail "port 8080 answered $page" - wait_for "smallwebwaf logged no request it forwarded" logged '"action":"forward"' - echo "example-app: smallwebwaf passes a request to the app and its answer back" + wait_for "servewaf logged no request it allowed" \ + logged "$SERVEWAF" '"action":"forward"' '"servewaf_line":"report"' + echo "example-app: rproxy passes a request to the app and its answer back" refused || fail "a second request in a minute was not refused" - wait_for "smallwebwaf logged no ban" logged '"action":"rate_limited"' + wait_for "servewaf logged no ban" logged "$SERVEWAF" '"action":"rate_limited"' echo "example-app: a second request in a minute bans the client" refused_from 203.0.113.9 /.env || fail "a probe for /.env was not refused" - wait_for "smallwebwaf logged no ban for the probe" \ - logged '"action":"banned"' '"rule_ids":["env-file"]' + wait_for "servewaf logged no ban for the probe" \ + logged "$SERVEWAF" '"action":"banned"' '"rule_ids":["env-file"]' refused_from 203.0.113.9 / || fail "the client of the probe was let through" wait_for "the client's next request did not make its ban permanent" \ - logged '"ban_expires":"permanent"' + logged "$SERVEWAF" '"ban_expires":"permanent"' echo "example-app: a probe for /.env bans the client, its next request for good" - docker exec "$CONTAINER" sv stop smallwebwaf >/dev/null || - fail "sv stop smallwebwaf failed" - wait_for "smallwebwaf did not stop in order" logged '"msg":"stopped"' - echo "example-app: sv stop stops smallwebwaf in order" + docker exec "$SERVEWAF" sv stop smallwebwaf >/dev/null || + fail "sv stop smallwebwaf failed in servewaf's container" + wait_for "servewaf did not stop in order" logged "$SERVEWAF" '"msg":"stopped"' + [ "$(status_from 203.0.113.9 /)" = 200 ] || + fail "with servewaf stopped, the app did not answer through rproxy" + wait_for "rproxy logged no request SWRP_DEFAULT_DISPOSITION decided" \ + logged "$APP" '"default_disposition":true' + docker exec "$APP" /usr/local/bin/smallwebwaf healthcheck || + fail "with servewaf stopped, the app's health check failed" + echo "example-app: with servewaf stopped, the app answers through rproxy" - docker stop "$CONTAINER" >/dev/null - status="$(docker inspect --format '{{.State.ExitCode}}' "$CONTAINER")" - [ "$status" = 0 ] || fail "docker stop left exit status $status" - echo "example-app: docker stop stops the container in order" + docker exec "$SERVEWAF" sv start smallwebwaf >/dev/null || + fail "sv start smallwebwaf failed in servewaf's container" + wait_for "once servewaf had started again, the ban did not apply" \ + refused_from 203.0.113.9 / + echo "example-app: once servewaf has started again, bans apply again" - docker rm "$CONTAINER" >/dev/null - start_container - refused || fail "the new container let the banned client through" - wait_for "smallwebwaf logged no request refused under the ban" \ - logged '"action":"banned"' - echo "example-app: a new container on the same volume keeps the ban" + docker exec "$APP" sv stop smallwebwaf >/dev/null || + fail "sv stop smallwebwaf failed in the app's container" + wait_for "rproxy did not stop in order" logged "$APP" '"msg":"stopped"' + echo "example-app: sv stop stops rproxy in order" - docker rm --force "$CONTAINER" >/dev/null - start_moved_container + for container in "$APP" "$SERVEWAF"; do + docker stop "$container" >/dev/null + status="$(docker inspect --format '{{.State.ExitCode}}' "$container")" + [ "$status" = 0 ] || fail "docker stop left exit status $status" + done + echo "example-app: docker stop stops each container in order" + + docker rm "$APP" "$SERVEWAF" >/dev/null + start_servewaf + start_app + refused || fail "the new servewaf container let the banned client through" + wait_for "servewaf logged no request refused under the ban" \ + logged "$SERVEWAF" '"action":"banned"' + echo "example-app: a new servewaf container on the same volume keeps the ban" + + docker rm --force "$SERVEWAF" >/dev/null + start_moved_servewaf echo "example-app: the run script gives the state files' directory named" \ "by SWWAF_STATE_DIR_FILE to the smallwebwaf user" - docker rm --force "$CONTAINER" >/dev/null - docker run --detach --name "$CONTAINER" \ + docker rm --force "$SERVEWAF" >/dev/null + docker run --detach --name "$SERVEWAF" --network "$NETWORK" \ + --env SMALLWEBWAF_COMMAND=servewaf \ --env SWWAF_STATE_DIR=/srv/missing \ --env SWWAF_LOOKUP_SOURCE=off \ - "$APP_IMAGE" >/dev/null - wait_for "smallwebwaf did not report the missing state directory" \ - logged "SWWAF_STATE_DIR cannot be written" - echo "example-app: smallwebwaf reports a state directory that does not exist" + "$IMAGE" >/dev/null + wait_for "servewaf did not report the missing state directory" \ + logged "$SERVEWAF" "SWWAF_STATE_DIR cannot be written" + echo "example-app: servewaf reports a state directory that does not exist" } main "$@" diff --git a/script/gitea-requests b/script/gitea-requests index 798565e..ca6837e 100755 --- a/script/gitea-requests +++ b/script/gitea-requests @@ -1,13 +1,14 @@ #!/bin/sh -# script/gitea-requests: build the image and run gitea 1.27.3 behind it, -# with every setting of smallwebwaf at its default but SWWAF_UPSTREAM_URL, -# which names gitea's container. Then send through it the gitea requests -# that reading gitea's source found the Core Rule Set refuses, which -# docs/gitea.md records, and a short sample of gitea's ordinary use: -# signing in, pages of files and commits, a diff, the API, a push, clone -# and fetch over HTTP, and two container image pushes. Last, print each -# request smallwebwaf logged, with what it did with it and the ids of the -# Core Rule Set's rules that matched it. +# script/gitea-requests: build the image and run gitea 1.27.3 behind rproxy +# and servewaf, three containers on one network. Every setting is at its +# default but these: SWRP_UPSTREAM_URL names gitea's container, SWRP_WAF_BACKEND +# names servewaf's, and SWWAF_LOG_REQUEST_HEADERS adds the header that labels a +# request. Then send through rproxy the gitea requests that reading gitea's +# source found the Core Rule Set refuses, which docs/gitea.md records, and a +# short sample of gitea's ordinary use: signing in, pages of files and commits, +# a diff, the API, a push, clone and fetch over HTTP, and two container image +# pushes. Last, print each request servewaf logged, with what it did with it and +# the ids of the Core Rule Set's rules that matched it. # Every request comes from one client, docker's gateway, so more than # SWWAF_ERROR_BURST_THRESHOLD, 30, refusals in a minute would ban it, and # the requests after would show as banned. The containers, the network, @@ -29,7 +30,8 @@ GITEA_IMAGE="gitea/gitea@sha256:87a67ee09d3ae0d1df5fda5dcda3e2a1f9236a45b0a59025 NAME="$("$SCRIPT_DIR/projectname")-gitea-$$" IMAGE="$NAME-image" NETWORK="$NAME" -SMALLWEBWAF="$NAME-smallwebwaf" +RPROXY="$NAME-rproxy" +SERVEWAF="$NAME-servewaf" GITEA="$NAME-gitea" # gitea's one user, who owns the repository and the container images. @@ -37,16 +39,23 @@ PASSWORD="owner-password" BROWSER="Mozilla/5.0 (X11; Linux x86_64; rv:131.0) Gecko/20100101 Firefox/131.0" -# The files the script makes, and smallwebwaf's address once it runs, -# localhost and the port docker publishes it on: a host name, since the -# Core Rule Set takes a Host that is an IP address for a warning (920350). +# The header that labels a request in the report. rproxy makes every request's +# id itself, so a label cannot be the X-Request-ID. +LABEL_HEADER="X-Case" + +# The default of SWWAF_LOG_REQUEST_HEADERS, and the header above. +LOGGED_HEADERS="accept,accept-language,accept-encoding,content-type,origin,range,x-case" + +# The files the script makes, and rproxy's address once it runs, localhost and +# the port docker publishes it on: a host name, since the Core Rule Set takes a +# Host that is an IP address for a warning (920350). WORK="" address="" cleanup() { # So that a signal cannot stop it partway through. trap '' HUP INT TERM - docker rm --force --volumes "$SMALLWEBWAF" "$GITEA" >/dev/null 2>&1 || true + docker rm --force --volumes "$RPROXY" "$SERVEWAF" "$GITEA" >/dev/null 2>&1 || true docker network rm "$NETWORK" >/dev/null 2>&1 || true docker rmi --force "$IMAGE" >/dev/null 2>&1 || true if [ -n "$address" ]; then @@ -75,30 +84,30 @@ wait_for() { done } +# healthy : docker says the container's health check passes. healthy() { - status="$(docker inspect --format '{{.State.Health.Status}}' "$SMALLWEBWAF")" + status="$(docker inspect --format '{{.State.Health.Status}}' "$1")" [ "$status" = healthy ] } -# logged ...: a line of smallwebwaf's output holds every text, in -# any order. +# logged ...: a line of servewaf's output holds every text, in any order. logged() { - lines="$(docker logs "$SMALLWEBWAF" 2>&1)" + lines="$(docker logs "$SERVEWAF" 2>&1)" for text in "$@"; do lines="$(printf '%s\n' "$lines" | grep -F "$text")" || return 1 done } -# send