Country allow and deny lists, looked up through GeoJS (closes #44)
check / check (push) Successful in 2m18s
check / check (push) Successful in 2m18s
SWWAF_DENIED_COUNTRIES and SWWAF_EXCLUSIVELY_ALLOWED_COUNTRIES refuse a request with 403 before its body is read and before the rate limits count it, logged as country_denied; every log line gains country. The new internal/lookup asks GeoJS only while a list is set, one request at a time carrying up to 200 waiting clients, keeps answers 7 days (at most 100,000), and after a failure waits a second, doubling to five minutes. Private, loopback and link-local clients have no country and are never sent. Codes are checked with golang.org/x/text/language. Deviation from SPEC.md, per the issue: no SWWAF_LOOKUP_SOURCE or SWWAF_LOOKUP_TIMEOUT; 403, not SWWAF_BAN_RESPONSE. Deviation: GeoJS's country endpoint, not geo.json, since only the country is needed. Judgement call: an IPv6 /64 is asked about by its first address; at most 10,000 clients wait. Deviation: go.mod and go.sum hand-written; no make target tidies them. Model: opus-5-5
This commit is contained in:
@@ -12,15 +12,15 @@ state in memory and in JSON files you can read and edit, and writes a detailed
|
|||||||
JSON log line for every request.
|
JSON log line for every request.
|
||||||
|
|
||||||
Status: the first milestone is built
|
Status: the first milestone is built
|
||||||
(https://git.eeqj.de/sneak/smallwebwaf/issues/13), and the rate limits of the
|
(https://git.eeqj.de/sneak/smallwebwaf/issues/13), and the rate limits and
|
||||||
second (https://git.eeqj.de/sneak/smallwebwaf/issues/14). `smallwebwaf` passes
|
country lists of the second (https://git.eeqj.de/sneak/smallwebwaf/issues/14).
|
||||||
each request to the app and the app's answer back, unchanged, within its
|
`smallwebwaf` passes each request to the app and the app's answer back,
|
||||||
timeouts and size limits, works out each client's address, refuses a client that
|
unchanged, within its timeouts and size limits, works out each client's address,
|
||||||
sends too many requests, and writes a JSON log line for every request. The
|
refuses a client that sends too many requests or comes from a country you
|
||||||
country lists and the image an app builds on come with the rest of milestone 2,
|
refuse, and writes a JSON log line for every request. The image an app builds on
|
||||||
and the rest of the design after that, in the order of the build order in
|
comes with the rest of milestone 2, and the rest of the design after that, in
|
||||||
[`SPEC.md`](SPEC.md). The survey of existing tools that led to the design is in
|
the order of the build order in [`SPEC.md`](SPEC.md). The survey of existing
|
||||||
[`EVALUATION.md`](EVALUATION.md).
|
tools that led to the design is in [`EVALUATION.md`](EVALUATION.md).
|
||||||
|
|
||||||
## Getting started
|
## Getting started
|
||||||
|
|
||||||
@@ -72,6 +72,13 @@ gives those in progress five seconds to finish.
|
|||||||
much of it the window still covers. At most 20,000 clients are kept, the least
|
much of it the window still covers. At most 20,000 clients are kept, the least
|
||||||
recently seen dropped first, and only in memory: a restart starts every client
|
recently seen dropped first, and only in memory: a restart starts every client
|
||||||
afresh.
|
afresh.
|
||||||
|
- Refuses a request from a country you refuse with `403`, as soon as the
|
||||||
|
client's country is known and before its body is read; such a request is not
|
||||||
|
counted for the rate limits. While one of the country lists below is set, each
|
||||||
|
client's country is looked up through GeoJS (see "Country and AS number
|
||||||
|
lookup" below); with neither set, no visitor's address leaves the host. A
|
||||||
|
client on a private, loopback or link-local address has no country, and
|
||||||
|
neither list checks it.
|
||||||
- Writes a line in the request log for each request (see "Request log" below).
|
- Writes a line in the request log for each request (see "Request log" below).
|
||||||
|
|
||||||
## Settings
|
## Settings
|
||||||
@@ -102,20 +109,29 @@ it, and the effective settings are logged at start.
|
|||||||
requests a client may make in a minute, an hour and a day. The defaults are
|
requests a client may make in a minute, an hour and a day. The defaults are
|
||||||
several times what one busy person produces, since a browser loading a heavy
|
several times what one busy person produces, since a browser loading a heavy
|
||||||
page makes a few hundred requests and several people often share one address.
|
page makes a few hundred requests and several people often share one address.
|
||||||
|
- `SWWAF_DENIED_COUNTRIES` (default empty): countries whose clients are refused,
|
||||||
|
for example `cn,ru,kp`.
|
||||||
|
- `SWWAF_EXCLUSIVELY_ALLOWED_COUNTRIES` (default empty): when set, the only
|
||||||
|
countries whose clients get through, for example `us,de`. A client whose
|
||||||
|
country cannot be found is refused too, so that new clients are not let in
|
||||||
|
whenever GeoJS stops answering.
|
||||||
|
|
||||||
Durations are in Go's syntax, with `d` for days (`90s`, `15m`, `7d`). Sizes are
|
Durations are in Go's syntax, with `d` for days (`90s`, `15m`, `7d`). Sizes are
|
||||||
bytes, with an optional `K`, `M` or `G`, which are powers of 1024 (`1K` is 1024
|
bytes, with an optional `K`, `M` or `G`, which are powers of 1024 (`1K` is 1024
|
||||||
bytes). Rate limits are whole numbers of requests. Netblocks are in CIDR form,
|
bytes). Rate limits are whole numbers of requests. Netblocks are in CIDR form,
|
||||||
and a bare address stands for itself alone. `off` switches a timeout, a size
|
and a bare address stands for itself alone. Countries are two-letter ISO codes
|
||||||
limit or a rate limit off.
|
in either case (`de` and `DE` are the same); a code that is not a country code,
|
||||||
|
such as `nk` (North Korea is `kp`), stops the start, and so does a code on both
|
||||||
|
country lists. `off` switches a timeout, a size limit or a rate limit off.
|
||||||
|
|
||||||
Four limits are fixed rather than settings. The request line and headers may
|
Several limits are fixed rather than settings. The request line and headers may
|
||||||
take up to 32 KiB, above which the answer is `431` and nothing reaches the app.
|
take up to 32 KiB, above which the answer is `431` and nothing reaches the app.
|
||||||
A kept-open connection that sends nothing for 120 seconds is closed. That is
|
A kept-open connection that sends nothing for 120 seconds is closed. That is
|
||||||
longer than the 90 seconds after which traefik closes a connection it is not
|
longer than the 90 seconds after which traefik closes a connection it is not
|
||||||
using, so traefik never sends a request on a connection `smallwebwaf` is
|
using, so traefik never sends a request on a connection `smallwebwaf` is
|
||||||
closing. At most 20,000 clients are kept for the rate limits, and an IPv6 client
|
closing. At most 20,000 clients are kept for the rate limits, and an IPv6 client
|
||||||
is counted by its /64.
|
is counted by its /64. A new client waits at most a second for its country, and
|
||||||
|
at most 100,000 answers from GeoJS are kept, for 7 days each.
|
||||||
|
|
||||||
## Request log
|
## Request log
|
||||||
|
|
||||||
@@ -123,18 +139,23 @@ is counted by its /64.
|
|||||||
refused ones included:
|
refused ones included:
|
||||||
|
|
||||||
```
|
```
|
||||||
{"type":"request","time":"2026-10-03T12:00:00.123Z","client_ip":"203.0.113.9","peer_ip":"172.18.0.2","method":"GET","host":"app.example","path":"/","query":"","protocol":"HTTP/1.1","status":200,"upstream_status":200,"request_bytes":0,"response_bytes":5120,"referer":"","user_agent":"curl/8.9.1","action":"forward","duration_total":3.217,"duration_upstream_total":3.104}
|
{"type":"request","time":"2026-10-03T12:00:00.123Z","client_ip":"203.0.113.9","peer_ip":"172.18.0.2","country":"DE","method":"GET","host":"app.example","path":"/","query":"","protocol":"HTTP/1.1","status":200,"upstream_status":200,"request_bytes":0,"response_bytes":5120,"referer":"","user_agent":"curl/8.9.1","action":"forward","duration_total":3.217,"duration_upstream_total":3.104}
|
||||||
```
|
```
|
||||||
|
|
||||||
- `time` is when the request arrived, in UTC. `peer_ip` is the TCP peer,
|
- `time` is when the request arrived, in UTC. `peer_ip` is the TCP peer,
|
||||||
normally traefik. `path` and `query` are as the client sent them.
|
normally traefik. `path` and `query` are as the client sent them.
|
||||||
|
- `country` is the client's country as GeoJS places it, and empty when it is not
|
||||||
|
known: with neither country list set, for a client on a private, loopback or
|
||||||
|
link-local address, and when GeoJS cannot place the client or has not answered
|
||||||
|
in time.
|
||||||
- `status` is what the client was sent, `0` if nothing was; `upstream_status` is
|
- `status` is what the client was sent, `0` if nothing was; `upstream_status` is
|
||||||
what the app answered, and is left out when the app did not answer.
|
what the app answered, and is left out when the app did not answer.
|
||||||
- `request_bytes` and `response_bytes` count body bytes.
|
- `request_bytes` and `response_bytes` count body bytes.
|
||||||
- `action` is `forward` for a request passed to the app, `rate_limited` for one
|
- `action` is `forward` for a request passed to the app, `country_denied` for
|
||||||
refused for a rate limit, `too_large` for a request or response over its size
|
one refused for its client's country, `rate_limited` for one refused for a
|
||||||
limit, `timed_out` for one that ran out of time, and `upstream_error` when the
|
rate limit, `too_large` for a request or response over its size limit,
|
||||||
app could not be reached or its answer broke off.
|
`timed_out` for one that ran out of time, and `upstream_error` when the app
|
||||||
|
could not be reached or its answer broke off.
|
||||||
- `limit_hit` is there for a request refused for a rate limit, and names the
|
- `limit_hit` is there for a request refused for a rate limit, and names the
|
||||||
window whose limit it went over: `minute`, `hour` or `day`, the shortest if it
|
window whose limit it went over: `minute`, `hour` or `day`, the shortest if it
|
||||||
went over several.
|
went over several.
|
||||||
@@ -367,16 +388,29 @@ the metrics, failure behaviour and the build order.
|
|||||||
|
|
||||||
## Country and AS number lookup
|
## Country and AS number lookup
|
||||||
|
|
||||||
`smallwebwaf` looks up the AS number and country of every client, for the
|
So far `smallwebwaf` looks up only the country, only through GeoJS, and only
|
||||||
request log, the metrics and the ban notes, and for the country lists and biased
|
while `SWWAF_DENIED_COUNTRIES` or `SWWAF_EXCLUSIVELY_ALLOWED_COUNTRIES` is set:
|
||||||
limits when you set them. It works with no setup: by default it asks the free
|
then the address of every new visitor is sent to GeoJS, and with neither set,
|
||||||
GeoJS web service, which needs no account and no file. This means that, by
|
none is. An IPv6 visitor is asked about by the first address of its /64. A new
|
||||||
default, the address of every new visitor is sent to GeoJS. Each answer is kept
|
visitor waits at most a second for its answer, and without one counts as coming
|
||||||
in memory for seven days, and many addresses are asked about in one request;
|
from an unknown country until the answer arrives. The addresses waiting are
|
||||||
writing the answers to disk, so that they survive a restart, comes in milestone
|
asked about together, up to 200 in one request, one request at a time; at most
|
||||||
3 or later (see the build order in [`SPEC.md`](SPEC.md)). GeoJS publishes no
|
10,000 visitors wait, and one more counts as coming from an unknown country
|
||||||
rate limit but may block a caller it thinks asks too much; while it is not
|
until there is room. While GeoJS fails, visitors with a kept answer are
|
||||||
answering, new visitors count as coming from an unknown country, which
|
unaffected, new ones count as coming from an unknown country, and GeoJS is asked
|
||||||
|
again a second later, then twice as long after each failure in a row, up to five
|
||||||
|
minutes.
|
||||||
|
|
||||||
|
In the full design, `smallwebwaf` looks up the AS number and country of every
|
||||||
|
client, for the request log, the metrics and the ban notes, and for the country
|
||||||
|
lists and biased limits when you set them. It works with no setup: by default it
|
||||||
|
asks the free GeoJS web service, which needs no account and no file. This means
|
||||||
|
that, by default, the address of every new visitor is sent to GeoJS. Each answer
|
||||||
|
is kept in memory for seven days, and many addresses are asked about in one
|
||||||
|
request; writing the answers to disk, so that they survive a restart, comes in
|
||||||
|
milestone 3 or later (see the build order in [`SPEC.md`](SPEC.md)). GeoJS
|
||||||
|
publishes no rate limit but may block a caller it thinks asks too much; while it
|
||||||
|
is not answering, new visitors count as coming from an unknown country, which
|
||||||
`SWWAF_EXCLUSIVELY_ALLOWED_COUNTRIES` refuses.
|
`SWWAF_EXCLUSIVELY_ALLOWED_COUNTRIES` refuses.
|
||||||
|
|
||||||
To keep your visitors' addresses on your own host, set
|
To keep your visitors' addresses on your own host, set
|
||||||
@@ -413,16 +447,19 @@ refusal comes with `SWWAF_ALLOW_NETS` in milestone 3 or later.
|
|||||||
the checks, passes the request to the app and the answer back with the
|
the checks, passes the request to the app and the answer back with the
|
||||||
standard library's `httputil.ReverseProxy` within the timeouts and size
|
standard library's `httputil.ReverseProxy` within the timeouts and size
|
||||||
limits, and writes the request's log line. Its `check` method is where a
|
limits, and writes the request's log line. Its `check` method is where a
|
||||||
request is refused before anything reaches the app: for a rate limit, for an
|
request is refused before anything reaches the app: for the country lists, for
|
||||||
announced body over the size limit, and, with the rest of milestone 2, for the
|
a rate limit, and for an announced body over the size limit.
|
||||||
country lists.
|
- `internal/lookup`: looks up each client's country through GeoJS, and keeps the
|
||||||
|
answers.
|
||||||
- `internal/ratelimit`: counts each client's requests and tells when one takes
|
- `internal/ratelimit`: counts each client's requests and tells when one takes
|
||||||
it over a rate limit.
|
it over a rate limit.
|
||||||
- `internal/requestlog`: the lines on stdout: the request log line and the
|
- `internal/requestlog`: the lines on stdout: the request log line and the
|
||||||
process's own messages.
|
process's own messages.
|
||||||
|
|
||||||
Besides the Go standard library, `github.com/hashicorp/golang-lru/v2` keeps the
|
Besides the Go standard library, `github.com/hashicorp/golang-lru/v2` keeps the
|
||||||
table of clients to 20,000, dropping the least recently seen.
|
table of clients to 20,000 and the GeoJS answers to 100,000, dropping the least
|
||||||
|
recently seen, and `golang.org/x/text/language` knows which two-letter codes are
|
||||||
|
countries.
|
||||||
|
|
||||||
## Entrypoints
|
## Entrypoints
|
||||||
|
|
||||||
@@ -456,8 +493,9 @@ so that they run in minimal containers.
|
|||||||
|
|
||||||
## TODO
|
## TODO
|
||||||
|
|
||||||
- Milestone 2: the country lists and the image an app builds on
|
- Milestone 2: the image an app builds on
|
||||||
(https://git.eeqj.de/sneak/smallwebwaf/issues/14); its rate limits are built.
|
(https://git.eeqj.de/sneak/smallwebwaf/issues/14); its rate limits and country
|
||||||
|
lists are built.
|
||||||
- The rest of the design, in the order of the build order in
|
- The rest of the design, in the order of the build order in
|
||||||
[`SPEC.md`](SPEC.md).
|
[`SPEC.md`](SPEC.md).
|
||||||
|
|
||||||
|
|||||||
@@ -2,4 +2,7 @@ module sneak.berlin/go/smallwebwaf
|
|||||||
|
|
||||||
go 1.26.0
|
go 1.26.0
|
||||||
|
|
||||||
require github.com/hashicorp/golang-lru/v2 v2.0.7
|
require (
|
||||||
|
github.com/hashicorp/golang-lru/v2 v2.0.7
|
||||||
|
golang.org/x/text v0.42.0
|
||||||
|
)
|
||||||
|
|||||||
@@ -1,2 +1,4 @@
|
|||||||
github.com/hashicorp/golang-lru/v2 v2.0.7 h1:a+bsQ5rvGLjzHuww6tVxozPZFVghXaHOwFs4luLUK2k=
|
github.com/hashicorp/golang-lru/v2 v2.0.7 h1:a+bsQ5rvGLjzHuww6tVxozPZFVghXaHOwFs4luLUK2k=
|
||||||
github.com/hashicorp/golang-lru/v2 v2.0.7/go.mod h1:QeFd9opnmA6QUJc5vARoKUSoFhyfM2/ZepoAG6RGpeM=
|
github.com/hashicorp/golang-lru/v2 v2.0.7/go.mod h1:QeFd9opnmA6QUJc5vARoKUSoFhyfM2/ZepoAG6RGpeM=
|
||||||
|
golang.org/x/text v0.42.0 h1:JbOZXgfeCPU9gacVtYliJqOhD+zhrEqK4LfdpmlUZqI=
|
||||||
|
golang.org/x/text v0.42.0/go.mod h1:ojzP1Z+2QtioaF8DTtO8K5q7JWVVYwZKenzujK0Zd0E=
|
||||||
|
|||||||
@@ -11,9 +11,12 @@ import (
|
|||||||
"net"
|
"net"
|
||||||
"net/netip"
|
"net/netip"
|
||||||
"net/url"
|
"net/url"
|
||||||
|
"slices"
|
||||||
"strconv"
|
"strconv"
|
||||||
"strings"
|
"strings"
|
||||||
"time"
|
"time"
|
||||||
|
|
||||||
|
"golang.org/x/text/language"
|
||||||
)
|
)
|
||||||
|
|
||||||
// Config is smallwebwaf's settings. A timeout, size or rate limit of zero
|
// Config is smallwebwaf's settings. A timeout, size or rate limit of zero
|
||||||
@@ -51,6 +54,13 @@ type Config struct {
|
|||||||
RateLimitPerMinute int64
|
RateLimitPerMinute int64
|
||||||
RateLimitPerHour int64
|
RateLimitPerHour int64
|
||||||
RateLimitPerDay int64
|
RateLimitPerDay int64
|
||||||
|
// DeniedCountries are the countries whose clients are refused
|
||||||
|
// (SWWAF_DENIED_COUNTRIES). ExclusivelyAllowedCountries, when not
|
||||||
|
// empty, are the only countries whose clients are let through
|
||||||
|
// (SWWAF_EXCLUSIVELY_ALLOWED_COUNTRIES). Both hold two-letter codes in
|
||||||
|
// capitals, as GeoJS gives them.
|
||||||
|
DeniedCountries []string
|
||||||
|
ExclusivelyAllowedCountries []string
|
||||||
|
|
||||||
// settings are the values read, as given or by default, for the
|
// settings are the values read, as given or by default, for the
|
||||||
// log line at start.
|
// log line at start.
|
||||||
@@ -84,6 +94,9 @@ var (
|
|||||||
errNotUpstreamURL = errors.New(
|
errNotUpstreamURL = errors.New(
|
||||||
"is not a URL with only a scheme, a host and an optional port, " +
|
"is not a URL with only a scheme, a host and an optional port, " +
|
||||||
"such as http://127.0.0.1:8081")
|
"such as http://127.0.0.1:8081")
|
||||||
|
errNotCountry = errors.New(
|
||||||
|
"is not a two-letter country code such as de or kp")
|
||||||
|
errOnBothLists = errors.New("is in SWWAF_DENIED_COUNTRIES too")
|
||||||
)
|
)
|
||||||
|
|
||||||
// FromEnvironment reads the settings with lookupEnv, normally
|
// FromEnvironment reads the settings with lookupEnv, normally
|
||||||
@@ -104,6 +117,16 @@ func FromEnvironment(lookupEnv func(string) (string, bool)) (*Config, error) {
|
|||||||
RateLimitPerMinute: env.count("SWWAF_RATE_LIMIT_PER_MINUTE", "1000"),
|
RateLimitPerMinute: env.count("SWWAF_RATE_LIMIT_PER_MINUTE", "1000"),
|
||||||
RateLimitPerHour: env.count("SWWAF_RATE_LIMIT_PER_HOUR", "10000"),
|
RateLimitPerHour: env.count("SWWAF_RATE_LIMIT_PER_HOUR", "10000"),
|
||||||
RateLimitPerDay: env.count("SWWAF_RATE_LIMIT_PER_DAY", "50000"),
|
RateLimitPerDay: env.count("SWWAF_RATE_LIMIT_PER_DAY", "50000"),
|
||||||
|
DeniedCountries: env.countries("SWWAF_DENIED_COUNTRIES", ""),
|
||||||
|
ExclusivelyAllowedCountries: env.countries(
|
||||||
|
"SWWAF_EXCLUSIVELY_ALLOWED_COUNTRIES", ""),
|
||||||
|
}
|
||||||
|
|
||||||
|
for _, country := range cfg.ExclusivelyAllowedCountries {
|
||||||
|
if slices.Contains(cfg.DeniedCountries, country) {
|
||||||
|
env.check("SWWAF_EXCLUSIVELY_ALLOWED_COUNTRIES",
|
||||||
|
fmt.Errorf("%q %w", country, errOnBothLists))
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
if env.err != nil {
|
if env.err != nil {
|
||||||
@@ -201,6 +224,14 @@ func (e *environment) count(name, defaultValue string) int64 {
|
|||||||
return count
|
return count
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// countries reads a setting that is a list of countries.
|
||||||
|
func (e *environment) countries(name, defaultValue string) []string {
|
||||||
|
countries, err := parseCountries(e.value(name, defaultValue))
|
||||||
|
e.check(name, err)
|
||||||
|
|
||||||
|
return countries
|
||||||
|
}
|
||||||
|
|
||||||
// parseDuration reads a duration in Go's syntax, such as 90s or 15m, a
|
// parseDuration reads a duration in Go's syntax, such as 90s or 15m, a
|
||||||
// whole number of days such as 7d, or off.
|
// whole number of days such as 7d, or off.
|
||||||
func parseDuration(value string) (time.Duration, error) {
|
func parseDuration(value string) (time.Duration, error) {
|
||||||
@@ -348,6 +379,32 @@ func parseNetblock(value string) (netip.Prefix, error) {
|
|||||||
return netip.PrefixFrom(addr, addr.BitLen()), nil
|
return netip.PrefixFrom(addr, addr.BitLen()), nil
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// parseCountries reads a comma-separated list of two-letter ISO 3166-1
|
||||||
|
// country codes in either case, and returns them in capitals.
|
||||||
|
func parseCountries(value string) ([]string, error) {
|
||||||
|
items, err := parseList(value)
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
|
||||||
|
countries := make([]string, 0, len(items))
|
||||||
|
|
||||||
|
for _, item := range items {
|
||||||
|
// ParseRegion also takes three-letter and numeric codes, groups of
|
||||||
|
// countries such as eu, and codes replaced by another, such as uk
|
||||||
|
// by gb, which GeoJS never gives.
|
||||||
|
region, err := language.ParseRegion(item)
|
||||||
|
if err != nil || len(item) != 2 || !region.IsCountry() ||
|
||||||
|
region.Canonicalize() != region {
|
||||||
|
return nil, fmt.Errorf("%q %w", item, errNotCountry)
|
||||||
|
}
|
||||||
|
|
||||||
|
countries = append(countries, region.String())
|
||||||
|
}
|
||||||
|
|
||||||
|
return countries, nil
|
||||||
|
}
|
||||||
|
|
||||||
// parseListenAddr checks an address to listen on: an optional host and a
|
// parseListenAddr checks an address to listen on: an optional host and a
|
||||||
// port number.
|
// port number.
|
||||||
func parseListenAddr(value string) (string, error) {
|
func parseListenAddr(value string) (string, error) {
|
||||||
|
|||||||
@@ -28,6 +28,8 @@ const (
|
|||||||
rateLimitPerMinute = "SWWAF_RATE_LIMIT_PER_MINUTE"
|
rateLimitPerMinute = "SWWAF_RATE_LIMIT_PER_MINUTE"
|
||||||
rateLimitPerHour = "SWWAF_RATE_LIMIT_PER_HOUR"
|
rateLimitPerHour = "SWWAF_RATE_LIMIT_PER_HOUR"
|
||||||
rateLimitPerDay = "SWWAF_RATE_LIMIT_PER_DAY"
|
rateLimitPerDay = "SWWAF_RATE_LIMIT_PER_DAY"
|
||||||
|
deniedCountries = "SWWAF_DENIED_COUNTRIES"
|
||||||
|
allowedCountries = "SWWAF_EXCLUSIVELY_ALLOWED_COUNTRIES"
|
||||||
)
|
)
|
||||||
|
|
||||||
// off switches a timeout, a size limit or a rate limit off.
|
// off switches a timeout, a size limit or a rate limit off.
|
||||||
@@ -79,6 +81,8 @@ func TestDefaults(t *testing.T) {
|
|||||||
|
|
||||||
wantNetblocks(t, cfg.TrustedProxies,
|
wantNetblocks(t, cfg.TrustedProxies,
|
||||||
"10.0.0.0/8", "172.16.0.0/12", "192.168.0.0/16")
|
"10.0.0.0/8", "172.16.0.0/12", "192.168.0.0/16")
|
||||||
|
wantCountries(t, deniedCountries, cfg.DeniedCountries)
|
||||||
|
wantCountries(t, allowedCountries, cfg.ExclusivelyAllowedCountries)
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestValuesAsSet(t *testing.T) {
|
func TestValuesAsSet(t *testing.T) {
|
||||||
@@ -97,6 +101,8 @@ func TestValuesAsSet(t *testing.T) {
|
|||||||
rateLimitPerMinute: "60",
|
rateLimitPerMinute: "60",
|
||||||
rateLimitPerHour: "600",
|
rateLimitPerHour: "600",
|
||||||
rateLimitPerDay: "6000",
|
rateLimitPerDay: "6000",
|
||||||
|
deniedCountries: "cn, RU,kp,Xk",
|
||||||
|
allowedCountries: "de",
|
||||||
})
|
})
|
||||||
|
|
||||||
wantSettings(t, cfg, config.Config{
|
wantSettings(t, cfg, config.Config{
|
||||||
@@ -117,6 +123,25 @@ func TestValuesAsSet(t *testing.T) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
wantNetblocks(t, cfg.TrustedProxies, "192.0.2.1/32", "10.0.0.0/8", "2001:db8::/32")
|
wantNetblocks(t, cfg.TrustedProxies, "192.0.2.1/32", "10.0.0.0/8", "2001:db8::/32")
|
||||||
|
wantCountries(t, deniedCountries, cfg.DeniedCountries, "CN", "RU", "KP", "XK")
|
||||||
|
wantCountries(t, allowedCountries, cfg.ExclusivelyAllowedCountries, "DE")
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestCodeOnBothCountryListsStopsTheStart(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
_, err := config.FromEnvironment(environment{
|
||||||
|
deniedCountries: "cn,ru",
|
||||||
|
allowedCountries: "de,RU",
|
||||||
|
}.lookupEnv)
|
||||||
|
if err == nil {
|
||||||
|
t.Fatal("ru on both country lists was accepted")
|
||||||
|
}
|
||||||
|
|
||||||
|
if !strings.HasPrefix(err.Error(), allowedCountries+": ") ||
|
||||||
|
!strings.Contains(err.Error(), `"RU"`) {
|
||||||
|
t.Errorf("error %q does not name %s and RU", err, allowedCountries)
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestSizesAndOff(t *testing.T) {
|
func TestSizesAndOff(t *testing.T) {
|
||||||
@@ -198,6 +223,15 @@ func TestInvalidValueStopsTheStart(t *testing.T) {
|
|||||||
{rateLimitPerHour, "1.5"},
|
{rateLimitPerHour, "1.5"},
|
||||||
{rateLimitPerDay, "-1"},
|
{rateLimitPerDay, "-1"},
|
||||||
{rateLimitPerDay, "lots"},
|
{rateLimitPerDay, "lots"},
|
||||||
|
{deniedCountries, "nk"},
|
||||||
|
{deniedCountries, "kp,,ir"},
|
||||||
|
{deniedCountries, "prk"},
|
||||||
|
{deniedCountries, "408"},
|
||||||
|
{deniedCountries, "k"},
|
||||||
|
{deniedCountries, "eu"},
|
||||||
|
{allowedCountries, "uk"},
|
||||||
|
{allowedCountries, "zz"},
|
||||||
|
{allowedCountries, "de,germany"},
|
||||||
} {
|
} {
|
||||||
t.Run(tc.name+"="+tc.value, func(t *testing.T) {
|
t.Run(tc.name+"="+tc.value, func(t *testing.T) {
|
||||||
t.Parallel()
|
t.Parallel()
|
||||||
@@ -245,6 +279,8 @@ func TestLogsEachSettingWithItsValue(t *testing.T) {
|
|||||||
rateLimitPerMinute: "1000",
|
rateLimitPerMinute: "1000",
|
||||||
rateLimitPerHour: "10000",
|
rateLimitPerHour: "10000",
|
||||||
rateLimitPerDay: "50000",
|
rateLimitPerDay: "50000",
|
||||||
|
deniedCountries: "",
|
||||||
|
allowedCountries: "",
|
||||||
}
|
}
|
||||||
if !maps.Equal(line.Settings, want) {
|
if !maps.Equal(line.Settings, want) {
|
||||||
t.Errorf("logged settings\n%v\nwant\n%v", line.Settings, want)
|
t.Errorf("logged settings\n%v\nwant\n%v", line.Settings, want)
|
||||||
@@ -282,3 +318,12 @@ func wantNetblocks(t *testing.T, got []netip.Prefix, want ...string) {
|
|||||||
t.Errorf("netblocks %v, want %v", gotText, want)
|
t.Errorf("netblocks %v, want %v", gotText, want)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// wantCountries checks the list of countries the setting name gave.
|
||||||
|
func wantCountries(t *testing.T, name string, got []string, want ...string) {
|
||||||
|
t.Helper()
|
||||||
|
|
||||||
|
if !slices.Equal(got, want) {
|
||||||
|
t.Errorf("%s gave %v, want %v", name, got, want)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|||||||
@@ -0,0 +1,358 @@
|
|||||||
|
// Package lookup looks up each client's country through the GeoJS web
|
||||||
|
// service, and keeps the answers in memory, for at most 100,000 clients
|
||||||
|
// and for 7 days each.
|
||||||
|
package lookup
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"encoding/json"
|
||||||
|
"errors"
|
||||||
|
"fmt"
|
||||||
|
"io"
|
||||||
|
"log/slog"
|
||||||
|
"net/http"
|
||||||
|
"net/netip"
|
||||||
|
"strings"
|
||||||
|
"sync"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"github.com/hashicorp/golang-lru/v2/simplelru"
|
||||||
|
)
|
||||||
|
|
||||||
|
// URL is GeoJS's country endpoint. Asked about several addresses at once,
|
||||||
|
// comma separated in its ip parameter, it answers with a list.
|
||||||
|
const URL = "https://get.geojs.io/v1/ip/country.json"
|
||||||
|
|
||||||
|
const (
|
||||||
|
// keepFor is how long an answer is used instead of asking GeoJS again.
|
||||||
|
keepFor = 7 * 24 * time.Hour
|
||||||
|
// maxAnswers is how many answers are kept. Past it, the one used
|
||||||
|
// longest ago is dropped.
|
||||||
|
maxAnswers = 100000
|
||||||
|
// maxWaiting is how many clients may wait to be asked about. Past it,
|
||||||
|
// a new client counts as not found and is not asked about until there
|
||||||
|
// is room, so that a swarm of new addresses while GeoJS is down cannot
|
||||||
|
// fill the memory.
|
||||||
|
maxWaiting = 10000
|
||||||
|
// maxPerRequest is how many addresses one request to GeoJS asks about.
|
||||||
|
maxPerRequest = 200
|
||||||
|
// timeout is how long a new client waits for its answer, and how long
|
||||||
|
// a request to GeoJS may take before it is abandoned.
|
||||||
|
timeout = time.Second
|
||||||
|
// After a failure GeoJS is not asked again for a second, and for
|
||||||
|
// retryDelayFactor times as long after each further failure in a row,
|
||||||
|
// up to five minutes.
|
||||||
|
firstRetryDelay = time.Second
|
||||||
|
retryDelayFactor = 2
|
||||||
|
maxRetryDelay = 5 * time.Minute
|
||||||
|
// maxResponseBytes is the most of GeoJS's answer that is read.
|
||||||
|
maxResponseBytes = 1 << 20
|
||||||
|
)
|
||||||
|
|
||||||
|
var errStatus = errors.New("GeoJS answered")
|
||||||
|
|
||||||
|
// Params are what New needs.
|
||||||
|
type Params struct {
|
||||||
|
// URL is where GeoJS is asked, normally URL.
|
||||||
|
URL string
|
||||||
|
// Now tells the time, normally time.Now.
|
||||||
|
Now func() time.Time
|
||||||
|
// ProcessLog receives GeoJS's failures.
|
||||||
|
ProcessLog *slog.Logger
|
||||||
|
}
|
||||||
|
|
||||||
|
// GeoJS looks up clients' countries through GeoJS. At most one request
|
||||||
|
// to GeoJS is under way at a time, and it asks about every client waiting,
|
||||||
|
// up to maxPerRequest. It is safe for concurrent use.
|
||||||
|
type GeoJS struct {
|
||||||
|
url string
|
||||||
|
now func() time.Time
|
||||||
|
processLog *slog.Logger
|
||||||
|
|
||||||
|
mu sync.Mutex
|
||||||
|
answers *simplelru.LRU[netip.Prefix, answer]
|
||||||
|
// waiting are the clients without an answer: those to ask GeoJS about,
|
||||||
|
// and those it is being asked about.
|
||||||
|
waiting map[netip.Prefix]*wait
|
||||||
|
// asking is true while a request to GeoJS is under way.
|
||||||
|
asking bool
|
||||||
|
// retryDelay is how long GeoJS is left alone after its last failure,
|
||||||
|
// zero after an answer; retryAt is when it may be asked again.
|
||||||
|
retryDelay time.Duration
|
||||||
|
retryAt time.Time
|
||||||
|
}
|
||||||
|
|
||||||
|
// answer is what GeoJS said about a client: its country, "" when GeoJS
|
||||||
|
// cannot place it, and when GeoJS said so.
|
||||||
|
type answer struct {
|
||||||
|
country string
|
||||||
|
received time.Time
|
||||||
|
}
|
||||||
|
|
||||||
|
// wait is a client waiting for its answer.
|
||||||
|
type wait struct {
|
||||||
|
// asked is closed when the client gets its answer, and closed and
|
||||||
|
// replaced each time GeoJS fails before then.
|
||||||
|
asked chan struct{}
|
||||||
|
// late is true once the client has gone without an answer, for a
|
||||||
|
// whole timeout or because GeoJS failed: its requests no longer wait.
|
||||||
|
late bool
|
||||||
|
}
|
||||||
|
|
||||||
|
// New returns a GeoJS with no answer kept yet.
|
||||||
|
func New(params Params) *GeoJS {
|
||||||
|
answers, err := simplelru.NewLRU[netip.Prefix, answer](maxAnswers, nil)
|
||||||
|
if err != nil {
|
||||||
|
panic(err) // NewLRU fails only for a size below one
|
||||||
|
}
|
||||||
|
|
||||||
|
return &GeoJS{
|
||||||
|
url: params.URL,
|
||||||
|
now: params.Now,
|
||||||
|
processLog: params.ProcessLog,
|
||||||
|
answers: answers,
|
||||||
|
waiting: map[netip.Prefix]*wait{},
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Country returns the country GeoJS places client in, as a two-letter
|
||||||
|
// code in capitals, or "" when the country cannot be found: GeoJS cannot
|
||||||
|
// place the client, or has not answered in time. An answer is kept for 7
|
||||||
|
// days. Without one, a client waits up to timeout for it, unless it has
|
||||||
|
// gone without one before; until GeoJS answers, the client is asked about
|
||||||
|
// again in the background. ctx is the context of the client's request,
|
||||||
|
// and ends the wait when it ends.
|
||||||
|
//
|
||||||
|
// GeoJS is asked about the client's first address, which is the client's
|
||||||
|
// own address for IPv4, and an address in the same place for an IPv6 /64.
|
||||||
|
func (g *GeoJS) Country(ctx context.Context, client netip.Prefix) string {
|
||||||
|
country, asked := g.answerOrWait(ctx, client)
|
||||||
|
if asked == nil {
|
||||||
|
return country
|
||||||
|
}
|
||||||
|
|
||||||
|
timer := time.NewTimer(timeout)
|
||||||
|
defer timer.Stop()
|
||||||
|
|
||||||
|
select {
|
||||||
|
case <-asked:
|
||||||
|
case <-timer.C:
|
||||||
|
case <-ctx.Done():
|
||||||
|
}
|
||||||
|
|
||||||
|
g.mu.Lock()
|
||||||
|
defer g.mu.Unlock()
|
||||||
|
|
||||||
|
country, found := g.kept(client)
|
||||||
|
|
||||||
|
w, waiting := g.waiting[client]
|
||||||
|
if !found && waiting {
|
||||||
|
w.late = true
|
||||||
|
}
|
||||||
|
|
||||||
|
return country
|
||||||
|
}
|
||||||
|
|
||||||
|
// answerOrWait returns client's kept answer if it has one. Otherwise it
|
||||||
|
// puts the client among those waiting if there is room, has GeoJS asked
|
||||||
|
// about them if it can be, and returns what to wait on for the answer, or
|
||||||
|
// nil when there is nothing to wait for.
|
||||||
|
func (g *GeoJS) answerOrWait(
|
||||||
|
ctx context.Context, client netip.Prefix,
|
||||||
|
) (string, <-chan struct{}) {
|
||||||
|
g.mu.Lock()
|
||||||
|
defer g.mu.Unlock()
|
||||||
|
|
||||||
|
country, found := g.kept(client)
|
||||||
|
if found {
|
||||||
|
return country, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
w, waiting := g.waiting[client]
|
||||||
|
if !waiting && len(g.waiting) < maxWaiting {
|
||||||
|
w = &wait{asked: make(chan struct{})}
|
||||||
|
g.waiting[client] = w
|
||||||
|
}
|
||||||
|
|
||||||
|
g.ask(ctx)
|
||||||
|
|
||||||
|
if w == nil {
|
||||||
|
return "", nil // too many clients wait already
|
||||||
|
}
|
||||||
|
|
||||||
|
if !g.asking {
|
||||||
|
// GeoJS is left alone after a failure, so no answer can come.
|
||||||
|
w.late = true
|
||||||
|
}
|
||||||
|
|
||||||
|
if w.late {
|
||||||
|
return "", nil
|
||||||
|
}
|
||||||
|
|
||||||
|
return "", w.asked
|
||||||
|
}
|
||||||
|
|
||||||
|
// kept returns client's answer, if one was received less than keepFor
|
||||||
|
// ago.
|
||||||
|
func (g *GeoJS) kept(client netip.Prefix) (string, bool) {
|
||||||
|
kept, found := g.answers.Get(client)
|
||||||
|
if !found || g.now().Sub(kept.received) >= keepFor {
|
||||||
|
return "", false
|
||||||
|
}
|
||||||
|
|
||||||
|
return kept.country, true
|
||||||
|
}
|
||||||
|
|
||||||
|
// ask starts asking GeoJS about the waiting clients, unless a request to
|
||||||
|
// it is under way or it is left alone after a failure. The requests to
|
||||||
|
// GeoJS are for every client waiting, so they go on when the client's
|
||||||
|
// request whose ctx is given ends.
|
||||||
|
func (g *GeoJS) ask(ctx context.Context) {
|
||||||
|
if g.asking || g.now().Before(g.retryAt) {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
|
||||||
|
g.asking = true
|
||||||
|
|
||||||
|
go g.askAboutWaiting(context.WithoutCancel(ctx))
|
||||||
|
}
|
||||||
|
|
||||||
|
// askAboutWaiting asks GeoJS about the waiting clients, one request at a
|
||||||
|
// time, until none is left or GeoJS fails.
|
||||||
|
func (g *GeoJS) askAboutWaiting(ctx context.Context) {
|
||||||
|
for {
|
||||||
|
clients := g.nextClients()
|
||||||
|
if len(clients) == 0 {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
|
||||||
|
countries, err := g.request(ctx, clients)
|
||||||
|
if !g.keep(clients, countries, err) {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// nextClients returns up to maxPerRequest of the waiting clients. When
|
||||||
|
// none is waiting, it returns none and notes that no request to GeoJS is
|
||||||
|
// under way.
|
||||||
|
func (g *GeoJS) nextClients() []netip.Prefix {
|
||||||
|
g.mu.Lock()
|
||||||
|
defer g.mu.Unlock()
|
||||||
|
|
||||||
|
if len(g.waiting) == 0 {
|
||||||
|
g.asking = false
|
||||||
|
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
clients := make([]netip.Prefix, 0, min(len(g.waiting), maxPerRequest))
|
||||||
|
|
||||||
|
for client := range g.waiting {
|
||||||
|
if len(clients) == maxPerRequest {
|
||||||
|
break
|
||||||
|
}
|
||||||
|
|
||||||
|
clients = append(clients, client)
|
||||||
|
}
|
||||||
|
|
||||||
|
return clients
|
||||||
|
}
|
||||||
|
|
||||||
|
// keep notes how a request to GeoJS about clients ended, and reports
|
||||||
|
// whether GeoJS answered. Each client asked about gets its answer, with
|
||||||
|
// no country for one GeoJS gave none for. After a failure GeoJS is left
|
||||||
|
// alone for a while, and every waiting client stops waiting and is
|
||||||
|
// asked about once GeoJS is asked again.
|
||||||
|
func (g *GeoJS) keep(
|
||||||
|
clients []netip.Prefix, countries map[netip.Addr]string, err error,
|
||||||
|
) bool {
|
||||||
|
g.mu.Lock()
|
||||||
|
defer g.mu.Unlock()
|
||||||
|
|
||||||
|
now := g.now()
|
||||||
|
|
||||||
|
if err != nil {
|
||||||
|
g.retryDelay = min(max(retryDelayFactor*g.retryDelay, firstRetryDelay),
|
||||||
|
maxRetryDelay)
|
||||||
|
g.retryAt = now.Add(g.retryDelay)
|
||||||
|
g.asking = false
|
||||||
|
|
||||||
|
for _, w := range g.waiting {
|
||||||
|
close(w.asked)
|
||||||
|
|
||||||
|
w.asked = make(chan struct{})
|
||||||
|
w.late = true
|
||||||
|
}
|
||||||
|
|
||||||
|
g.processLog.Warn("asking GeoJS failed",
|
||||||
|
"error", err.Error(), "asking_again_in", g.retryDelay.String())
|
||||||
|
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
|
||||||
|
g.retryDelay = 0
|
||||||
|
|
||||||
|
for _, client := range clients {
|
||||||
|
g.answers.Add(client, answer{country: countries[client.Addr()], received: now})
|
||||||
|
close(g.waiting[client].asked)
|
||||||
|
delete(g.waiting, client)
|
||||||
|
}
|
||||||
|
|
||||||
|
return true
|
||||||
|
}
|
||||||
|
|
||||||
|
// request asks GeoJS about clients in one request, and returns the
|
||||||
|
// country it gave for each address it answered for.
|
||||||
|
func (g *GeoJS) request(
|
||||||
|
ctx context.Context, clients []netip.Prefix,
|
||||||
|
) (map[netip.Addr]string, error) {
|
||||||
|
addrs := make([]string, 0, len(clients))
|
||||||
|
|
||||||
|
for _, client := range clients {
|
||||||
|
addrs = append(addrs, client.Addr().String())
|
||||||
|
}
|
||||||
|
|
||||||
|
ctx, cancel := context.WithTimeout(ctx, timeout)
|
||||||
|
defer cancel()
|
||||||
|
|
||||||
|
req, err := http.NewRequestWithContext(ctx, http.MethodGet,
|
||||||
|
g.url+"?ip="+strings.Join(addrs, ","), http.NoBody)
|
||||||
|
if err != nil {
|
||||||
|
return nil, fmt.Errorf("make the request to GeoJS: %w", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
res, err := http.DefaultClient.Do(req)
|
||||||
|
if err != nil {
|
||||||
|
return nil, fmt.Errorf("ask GeoJS: %w", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
defer func() {
|
||||||
|
_ = res.Body.Close()
|
||||||
|
}()
|
||||||
|
|
||||||
|
if res.StatusCode != http.StatusOK {
|
||||||
|
return nil, fmt.Errorf("%w %s", errStatus, res.Status)
|
||||||
|
}
|
||||||
|
|
||||||
|
var answers []struct {
|
||||||
|
IP string `json:"ip"`
|
||||||
|
Country string `json:"country"`
|
||||||
|
}
|
||||||
|
|
||||||
|
err = json.NewDecoder(io.LimitReader(res.Body, maxResponseBytes)).Decode(&answers)
|
||||||
|
if err != nil {
|
||||||
|
return nil, fmt.Errorf("read GeoJS's answer: %w", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
countries := make(map[netip.Addr]string, len(answers))
|
||||||
|
|
||||||
|
for _, item := range answers {
|
||||||
|
addr, err := netip.ParseAddr(item.IP)
|
||||||
|
if err == nil {
|
||||||
|
countries[addr] = item.Country
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
return countries, nil
|
||||||
|
}
|
||||||
@@ -0,0 +1,411 @@
|
|||||||
|
package lookup_test
|
||||||
|
|
||||||
|
import (
|
||||||
|
"encoding/json"
|
||||||
|
"log/slog"
|
||||||
|
"net/http"
|
||||||
|
"net/http/httptest"
|
||||||
|
"net/netip"
|
||||||
|
"slices"
|
||||||
|
"strings"
|
||||||
|
"sync"
|
||||||
|
"testing"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"sneak.berlin/go/smallwebwaf/internal/lookup"
|
||||||
|
)
|
||||||
|
|
||||||
|
const (
|
||||||
|
// germany is where the stand-in for GeoJS places every address but
|
||||||
|
// unplaced.
|
||||||
|
germany = "DE"
|
||||||
|
// unplaced is the address it cannot place.
|
||||||
|
unplaced = "192.0.2.1"
|
||||||
|
// timeout is how long a new client waits for its answer.
|
||||||
|
timeout = time.Second
|
||||||
|
// waitLimit bounds how long a test waits for what should happen.
|
||||||
|
waitLimit = 10 * time.Second
|
||||||
|
// pollInterval is how often a test looks again.
|
||||||
|
pollInterval = 10 * time.Millisecond
|
||||||
|
// week is how long an answer is kept.
|
||||||
|
week = 7 * 24 * time.Hour
|
||||||
|
)
|
||||||
|
|
||||||
|
func TestKeptAnswerIsUsedFor7DaysThenAskedAgain(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
geojs, clock, g := start(t)
|
||||||
|
placed := netip.MustParsePrefix("203.0.113.9/32")
|
||||||
|
notPlaced := netip.MustParsePrefix(unplaced + "/32")
|
||||||
|
|
||||||
|
wantCountry(t, g, placed, germany)
|
||||||
|
wantCountry(t, g, notPlaced, "")
|
||||||
|
wantRequests(t, geojs, 2)
|
||||||
|
|
||||||
|
// An answer without a country is kept too.
|
||||||
|
clock.advance(week - time.Second)
|
||||||
|
wantCountry(t, g, placed, germany)
|
||||||
|
wantCountry(t, g, notPlaced, "")
|
||||||
|
wantRequests(t, geojs, 2)
|
||||||
|
|
||||||
|
clock.advance(time.Second)
|
||||||
|
wantCountry(t, g, placed, germany)
|
||||||
|
wantRequests(t, geojs, 3)
|
||||||
|
wantAsked(t, geojs, 2, "203.0.113.9")
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestNewClientWaitsAtMostOneSecondThenCountsAsNotFound(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
geojs, clock, g := start(t)
|
||||||
|
client := netip.MustParsePrefix("203.0.113.9/32")
|
||||||
|
|
||||||
|
// GeoJS does not answer: the request to it is abandoned.
|
||||||
|
geojs.set(hanging)
|
||||||
|
|
||||||
|
began := time.Now()
|
||||||
|
|
||||||
|
wantCountry(t, g, client, "")
|
||||||
|
|
||||||
|
took := time.Since(began)
|
||||||
|
if took < timeout || took > timeout+waitLimit/2 {
|
||||||
|
t.Errorf("waited %s for the answer, want %s", took, timeout)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Its next request does not wait.
|
||||||
|
began = time.Now()
|
||||||
|
|
||||||
|
wantCountry(t, g, client, "")
|
||||||
|
|
||||||
|
took = time.Since(began)
|
||||||
|
if took > timeout/2 {
|
||||||
|
t.Errorf("waited %s again, want no wait", took)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Once GeoJS answers, the client is asked about again in the
|
||||||
|
// background, and has its country.
|
||||||
|
geojs.set(answering)
|
||||||
|
|
||||||
|
deadline := time.Now().Add(waitLimit)
|
||||||
|
for g.Country(t.Context(), client) != germany {
|
||||||
|
if time.Now().After(deadline) {
|
||||||
|
t.Fatalf("no answer after %s", waitLimit)
|
||||||
|
}
|
||||||
|
|
||||||
|
clock.advance(time.Minute)
|
||||||
|
time.Sleep(pollInterval)
|
||||||
|
}
|
||||||
|
|
||||||
|
wantRequests(t, geojs, 2)
|
||||||
|
wantAsked(t, geojs, 1, "203.0.113.9")
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestWaitingClientsAreAskedAboutInOneRequest(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
geojs, clock, g := start(t)
|
||||||
|
|
||||||
|
// GeoJS fails, and is then left alone for a second, while three more
|
||||||
|
// clients come. An IPv6 client is a /64, and GeoJS is asked about its
|
||||||
|
// first address.
|
||||||
|
geojs.set(failing)
|
||||||
|
wantCountry(t, g, netip.MustParsePrefix("203.0.113.1/32"), "")
|
||||||
|
wantCountry(t, g, netip.MustParsePrefix("203.0.113.2/32"), "")
|
||||||
|
wantCountry(t, g, netip.MustParsePrefix("2001:db8:1:2::/64"), "")
|
||||||
|
wantRequests(t, geojs, 1)
|
||||||
|
|
||||||
|
geojs.set(answering)
|
||||||
|
clock.advance(time.Second)
|
||||||
|
wantCountry(t, g, netip.MustParsePrefix("203.0.113.3/32"), germany)
|
||||||
|
wantRequests(t, geojs, 2)
|
||||||
|
wantAsked(t, geojs, 1, "203.0.113.1", "203.0.113.2", "2001:db8:1:2::", "203.0.113.3")
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestKeptAnswersUnaffectedWhileGeoJSFailsAndAskedAgainWithBackoff(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
geojs, clock, g := start(t)
|
||||||
|
clients := newClients()
|
||||||
|
kept := clients()
|
||||||
|
|
||||||
|
wantCountry(t, g, kept, germany)
|
||||||
|
|
||||||
|
geojs.set(failing)
|
||||||
|
wantCountry(t, g, kept, germany)
|
||||||
|
wantRequests(t, geojs, 1)
|
||||||
|
|
||||||
|
// Each failure leaves GeoJS alone twice as long as the one before, up
|
||||||
|
// to five minutes. New clients meanwhile count as not found.
|
||||||
|
requests := 1
|
||||||
|
|
||||||
|
for _, delay := range []time.Duration{
|
||||||
|
time.Second, 2 * time.Second, 4 * time.Second, 8 * time.Second,
|
||||||
|
16 * time.Second, 32 * time.Second, 64 * time.Second, 128 * time.Second,
|
||||||
|
256 * time.Second, 5 * time.Minute, 5 * time.Minute,
|
||||||
|
} {
|
||||||
|
wantCountry(t, g, clients(), "")
|
||||||
|
|
||||||
|
requests++
|
||||||
|
wantRequests(t, geojs, requests)
|
||||||
|
|
||||||
|
clock.advance(delay - time.Millisecond)
|
||||||
|
wantCountry(t, g, clients(), "")
|
||||||
|
wantRequests(t, geojs, requests)
|
||||||
|
|
||||||
|
clock.advance(time.Millisecond)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Once GeoJS answers again, it is asked about every client waiting.
|
||||||
|
geojs.set(answering)
|
||||||
|
wantCountry(t, g, clients(), germany)
|
||||||
|
wantRequests(t, geojs, requests+1)
|
||||||
|
|
||||||
|
asked := waitForRequests(t, geojs, requests+1)
|
||||||
|
if len(asked[requests]) != 23 {
|
||||||
|
t.Errorf("GeoJS was asked about %d clients, want 23", len(asked[requests]))
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestAtMost200AddressesInOneRequest(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
geojs, clock, g := start(t)
|
||||||
|
clients := newClients()
|
||||||
|
first := clients()
|
||||||
|
|
||||||
|
// 201 clients wait while GeoJS is left alone after a failure.
|
||||||
|
geojs.set(failing)
|
||||||
|
wantCountry(t, g, first, "")
|
||||||
|
|
||||||
|
for range 200 {
|
||||||
|
wantCountry(t, g, clients(), "")
|
||||||
|
}
|
||||||
|
|
||||||
|
// The first one's next request has GeoJS asked again.
|
||||||
|
geojs.set(answering)
|
||||||
|
clock.advance(time.Second)
|
||||||
|
wantCountry(t, g, first, "")
|
||||||
|
|
||||||
|
asked := waitForRequests(t, geojs, 3)
|
||||||
|
if len(asked[1]) != 200 || len(asked[2]) != 1 {
|
||||||
|
t.Errorf("GeoJS was asked about %d and then %d clients, want 200 and 1",
|
||||||
|
len(asked[1]), len(asked[2]))
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestAtMost10000ClientsWait(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
geojs, clock, g := start(t)
|
||||||
|
clients := newClients()
|
||||||
|
first := clients()
|
||||||
|
|
||||||
|
// 10,000 clients wait while GeoJS is left alone after a failure, and
|
||||||
|
// one more cannot join them.
|
||||||
|
geojs.set(failing)
|
||||||
|
wantCountry(t, g, first, "")
|
||||||
|
|
||||||
|
for range 9999 {
|
||||||
|
wantCountry(t, g, clients(), "")
|
||||||
|
}
|
||||||
|
|
||||||
|
extra := clients()
|
||||||
|
wantCountry(t, g, extra, "")
|
||||||
|
|
||||||
|
// The first one's next request has GeoJS asked about the 10,000, 200
|
||||||
|
// at a time, and not about the one more.
|
||||||
|
geojs.set(answering)
|
||||||
|
clock.advance(time.Second)
|
||||||
|
wantCountry(t, g, first, "")
|
||||||
|
|
||||||
|
asked := waitForRequests(t, geojs, 51)
|
||||||
|
for i, request := range asked {
|
||||||
|
if slices.Contains(request, extra.Addr().String()) {
|
||||||
|
t.Errorf("request %d asked about %s", i, extra.Addr())
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// With room among those waiting, it is asked about.
|
||||||
|
wantCountry(t, g, extra, germany)
|
||||||
|
}
|
||||||
|
|
||||||
|
// How the stand-in for GeoJS answers.
|
||||||
|
const (
|
||||||
|
answering = iota
|
||||||
|
failing // with 503
|
||||||
|
hanging // not at all, until the request is abandoned
|
||||||
|
)
|
||||||
|
|
||||||
|
// standIn is a stand-in for GeoJS. It notes the addresses each request
|
||||||
|
// asks about.
|
||||||
|
type standIn struct {
|
||||||
|
server *httptest.Server
|
||||||
|
|
||||||
|
mu sync.Mutex
|
||||||
|
answers int
|
||||||
|
requests [][]string
|
||||||
|
}
|
||||||
|
|
||||||
|
// ServeHTTP answers a request about the addresses in its ip parameter.
|
||||||
|
func (s *standIn) ServeHTTP(w http.ResponseWriter, r *http.Request) {
|
||||||
|
addrs := strings.Split(r.URL.Query().Get("ip"), ",")
|
||||||
|
|
||||||
|
s.mu.Lock()
|
||||||
|
s.requests = append(s.requests, addrs)
|
||||||
|
answers := s.answers
|
||||||
|
s.mu.Unlock()
|
||||||
|
|
||||||
|
switch answers {
|
||||||
|
case failing:
|
||||||
|
w.WriteHeader(http.StatusServiceUnavailable)
|
||||||
|
|
||||||
|
return
|
||||||
|
case hanging:
|
||||||
|
<-r.Context().Done()
|
||||||
|
|
||||||
|
return
|
||||||
|
}
|
||||||
|
|
||||||
|
list := make([]map[string]string, 0, len(addrs))
|
||||||
|
|
||||||
|
for _, addr := range addrs {
|
||||||
|
country := germany
|
||||||
|
if addr == unplaced {
|
||||||
|
country = ""
|
||||||
|
}
|
||||||
|
|
||||||
|
list = append(list, map[string]string{"ip": addr, "country": country})
|
||||||
|
}
|
||||||
|
|
||||||
|
err := json.NewEncoder(w).Encode(list)
|
||||||
|
if err != nil {
|
||||||
|
http.Error(w, err.Error(), http.StatusInternalServerError)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// set sets how the stand-in answers.
|
||||||
|
func (s *standIn) set(answers int) {
|
||||||
|
s.mu.Lock()
|
||||||
|
defer s.mu.Unlock()
|
||||||
|
|
||||||
|
s.answers = answers
|
||||||
|
}
|
||||||
|
|
||||||
|
// asked returns the addresses each request has asked about so far.
|
||||||
|
func (s *standIn) asked() [][]string {
|
||||||
|
s.mu.Lock()
|
||||||
|
defer s.mu.Unlock()
|
||||||
|
|
||||||
|
return slices.Clone(s.requests)
|
||||||
|
}
|
||||||
|
|
||||||
|
// testClock is a clock the test sets.
|
||||||
|
type testClock struct {
|
||||||
|
mu sync.Mutex
|
||||||
|
now time.Time
|
||||||
|
}
|
||||||
|
|
||||||
|
// Now tells the time.
|
||||||
|
func (c *testClock) Now() time.Time {
|
||||||
|
c.mu.Lock()
|
||||||
|
defer c.mu.Unlock()
|
||||||
|
|
||||||
|
return c.now
|
||||||
|
}
|
||||||
|
|
||||||
|
// advance moves the clock on by d.
|
||||||
|
func (c *testClock) advance(d time.Duration) {
|
||||||
|
c.mu.Lock()
|
||||||
|
defer c.mu.Unlock()
|
||||||
|
|
||||||
|
c.now = c.now.Add(d)
|
||||||
|
}
|
||||||
|
|
||||||
|
// start starts a stand-in for GeoJS that answers, and returns it, a
|
||||||
|
// clock, and a GeoJS asking it by that clock.
|
||||||
|
func start(t *testing.T) (*standIn, *testClock, *lookup.GeoJS) {
|
||||||
|
t.Helper()
|
||||||
|
|
||||||
|
geojs := &standIn{}
|
||||||
|
geojs.server = httptest.NewServer(geojs)
|
||||||
|
t.Cleanup(geojs.server.Close)
|
||||||
|
|
||||||
|
clock := &testClock{now: time.Date(2026, 10, 4, 0, 0, 0, 0, time.UTC)}
|
||||||
|
g := lookup.New(lookup.Params{
|
||||||
|
URL: geojs.server.URL,
|
||||||
|
Now: clock.Now,
|
||||||
|
ProcessLog: slog.New(slog.DiscardHandler),
|
||||||
|
})
|
||||||
|
|
||||||
|
return geojs, clock, g
|
||||||
|
}
|
||||||
|
|
||||||
|
// newClients returns what returns a new IPv4 client each time it is
|
||||||
|
// called.
|
||||||
|
func newClients() func() netip.Prefix {
|
||||||
|
addr := netip.MustParseAddr("10.0.0.0")
|
||||||
|
|
||||||
|
return func() netip.Prefix {
|
||||||
|
addr = addr.Next()
|
||||||
|
|
||||||
|
return netip.PrefixFrom(addr, addr.BitLen())
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// wantCountry checks the country g gives client.
|
||||||
|
func wantCountry(t *testing.T, g *lookup.GeoJS, client netip.Prefix, want string) {
|
||||||
|
t.Helper()
|
||||||
|
|
||||||
|
got := g.Country(t.Context(), client)
|
||||||
|
if got != want {
|
||||||
|
t.Errorf("%s is in %q, want %q", client, got, want)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// wantRequests checks how many requests GeoJS has had.
|
||||||
|
func wantRequests(t *testing.T, geojs *standIn, want int) {
|
||||||
|
t.Helper()
|
||||||
|
|
||||||
|
got := len(geojs.asked())
|
||||||
|
if got != want {
|
||||||
|
t.Errorf("GeoJS had %d requests, want %d", got, want)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// wantAsked checks the addresses request i asked about, in any order.
|
||||||
|
func wantAsked(t *testing.T, geojs *standIn, i int, want ...string) {
|
||||||
|
t.Helper()
|
||||||
|
|
||||||
|
asked := geojs.asked()
|
||||||
|
if len(asked) <= i {
|
||||||
|
t.Fatalf("GeoJS had %d requests, want more than %d", len(asked), i)
|
||||||
|
}
|
||||||
|
|
||||||
|
got := slices.Sorted(slices.Values(asked[i]))
|
||||||
|
|
||||||
|
slices.Sort(want)
|
||||||
|
|
||||||
|
if !slices.Equal(got, want) {
|
||||||
|
t.Errorf("request %d asked about %v, want %v", i, got, want)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// waitForRequests waits for GeoJS to have had count requests, and returns
|
||||||
|
// the addresses each asked about.
|
||||||
|
func waitForRequests(t *testing.T, geojs *standIn, count int) [][]string {
|
||||||
|
t.Helper()
|
||||||
|
|
||||||
|
deadline := time.Now().Add(waitLimit)
|
||||||
|
for time.Now().Before(deadline) {
|
||||||
|
asked := geojs.asked()
|
||||||
|
if len(asked) >= count {
|
||||||
|
return asked
|
||||||
|
}
|
||||||
|
|
||||||
|
time.Sleep(pollInterval)
|
||||||
|
}
|
||||||
|
|
||||||
|
t.Fatalf("fewer than %d requests to GeoJS after %s", count, waitLimit)
|
||||||
|
|
||||||
|
return nil
|
||||||
|
}
|
||||||
@@ -0,0 +1,41 @@
|
|||||||
|
package proxy
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"net/netip"
|
||||||
|
"slices"
|
||||||
|
)
|
||||||
|
|
||||||
|
// countryDenied reports whether the country lists refuse the request.
|
||||||
|
// The client's country is looked up only while a list is set, and never
|
||||||
|
// for a client on a private, loopback or link-local address, which has
|
||||||
|
// no country and which neither list checks. A client whose country
|
||||||
|
// cannot be found is refused only by SWWAF_EXCLUSIVELY_ALLOWED_COUNTRIES.
|
||||||
|
// ctx is the request's own context.
|
||||||
|
func (rq *request) countryDenied(ctx context.Context) bool {
|
||||||
|
denied := rq.h.config.DeniedCountries
|
||||||
|
allowed := rq.h.config.ExclusivelyAllowedCountries
|
||||||
|
|
||||||
|
if len(denied) == 0 && len(allowed) == 0 {
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
|
||||||
|
if !hasCountry(rq.client) {
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
|
||||||
|
country := rq.h.geojs.Country(ctx, clientGroup(rq.client))
|
||||||
|
rq.line.Country = country
|
||||||
|
|
||||||
|
if slices.Contains(denied, country) {
|
||||||
|
return true
|
||||||
|
}
|
||||||
|
|
||||||
|
return len(allowed) > 0 && !slices.Contains(allowed, country)
|
||||||
|
}
|
||||||
|
|
||||||
|
// hasCountry reports whether addr can be placed in a country: private,
|
||||||
|
// loopback and link-local addresses cannot.
|
||||||
|
func hasCountry(addr netip.Addr) bool {
|
||||||
|
return !addr.IsPrivate() && !addr.IsLoopback() && !addr.IsLinkLocalUnicast()
|
||||||
|
}
|
||||||
@@ -0,0 +1,213 @@
|
|||||||
|
package proxy_test
|
||||||
|
|
||||||
|
import (
|
||||||
|
"encoding/json"
|
||||||
|
"maps"
|
||||||
|
"net/http"
|
||||||
|
"net/http/httptest"
|
||||||
|
"slices"
|
||||||
|
"strings"
|
||||||
|
"sync"
|
||||||
|
"sync/atomic"
|
||||||
|
"testing"
|
||||||
|
|
||||||
|
"sneak.berlin/go/smallwebwaf/internal/requestlog"
|
||||||
|
)
|
||||||
|
|
||||||
|
// The clients the stand-in for GeoJS knows about.
|
||||||
|
const (
|
||||||
|
// fromDE is placed in Germany.
|
||||||
|
fromDE = client
|
||||||
|
// fromKP is placed in North Korea.
|
||||||
|
fromKP = "198.51.100.7"
|
||||||
|
// unplaced cannot be placed in any country.
|
||||||
|
unplaced = "192.0.2.1"
|
||||||
|
)
|
||||||
|
|
||||||
|
func TestCountryLists(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
for _, tc := range []struct {
|
||||||
|
name string
|
||||||
|
env map[string]string
|
||||||
|
refused []string
|
||||||
|
}{
|
||||||
|
{"denied", map[string]string{deniedCountries: "kp"}, []string{fromKP}},
|
||||||
|
{
|
||||||
|
"exclusively allowed", map[string]string{allowedCountries: "DE"},
|
||||||
|
[]string{fromKP, unplaced},
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"both", map[string]string{deniedCountries: "kp", allowedCountries: "de,fr"},
|
||||||
|
[]string{fromKP, unplaced},
|
||||||
|
},
|
||||||
|
} {
|
||||||
|
t.Run(tc.name, func(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
var calls atomic.Int32
|
||||||
|
|
||||||
|
app := startApp(t, func(http.ResponseWriter, *http.Request) {
|
||||||
|
calls.Add(1)
|
||||||
|
})
|
||||||
|
geojsURL, _ := startGeoJS(t)
|
||||||
|
env := map[string]string{trustedProxies: trustLocalhost}
|
||||||
|
maps.Copy(env, tc.env)
|
||||||
|
addr, out := startProxyWithGeoJS(t, app.URL, geojsURL, env)
|
||||||
|
|
||||||
|
for i, sent := range []struct{ client, country string }{
|
||||||
|
{fromDE, "DE"}, {fromKP, "KP"}, {unplaced, ""},
|
||||||
|
} {
|
||||||
|
req := newRequest(t, http.MethodGet, addr, "/", http.NoBody)
|
||||||
|
req.Header.Set(forwardedFor, sent.client)
|
||||||
|
got := do(t, req)
|
||||||
|
|
||||||
|
line := out.requestLines(t, i+1)[i]
|
||||||
|
if line.Country != sent.country {
|
||||||
|
t.Errorf("log line has country %q, want %q", line.Country, sent.country)
|
||||||
|
}
|
||||||
|
|
||||||
|
if slices.Contains(tc.refused, sent.client) {
|
||||||
|
wantStatus(t, got, http.StatusForbidden)
|
||||||
|
wantLine(t, line, http.StatusForbidden, requestlog.ActionCountryDenied)
|
||||||
|
} else {
|
||||||
|
wantStatus(t, got, http.StatusOK)
|
||||||
|
wantLine(t, line, http.StatusOK, requestlog.ActionForward)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
if int(calls.Load()) != 3-len(tc.refused) {
|
||||||
|
t.Errorf("the app was called %d times, want %d",
|
||||||
|
calls.Load(), 3-len(tc.refused))
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestCountryRefusalComesBeforeTheBodyAndTheRateLimits(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
var calls atomic.Int32
|
||||||
|
|
||||||
|
app := startApp(t, func(http.ResponseWriter, *http.Request) {
|
||||||
|
calls.Add(1)
|
||||||
|
})
|
||||||
|
geojsURL, _ := startGeoJS(t)
|
||||||
|
addr, out := startProxyWithGeoJS(t, app.URL, geojsURL, map[string]string{
|
||||||
|
trustedProxies: trustLocalhost,
|
||||||
|
deniedCountries: "kp",
|
||||||
|
rateLimitPerMinute: "1",
|
||||||
|
})
|
||||||
|
|
||||||
|
// With a limit of one request a minute, the second request would be
|
||||||
|
// over it, were refused requests counted.
|
||||||
|
for i := range 2 {
|
||||||
|
req := newRequest(t, http.MethodPost, addr, "/", strings.NewReader("a body"))
|
||||||
|
req.Header.Set(forwardedFor, fromKP)
|
||||||
|
wantStatus(t, do(t, req), http.StatusForbidden)
|
||||||
|
|
||||||
|
line := out.requestLines(t, i+1)[i]
|
||||||
|
wantLine(t, line, http.StatusForbidden, requestlog.ActionCountryDenied)
|
||||||
|
|
||||||
|
if line.RequestBytes != 0 || line.LimitHit != "" {
|
||||||
|
t.Errorf("log line has request_bytes %d and limit_hit %q, want 0 and none",
|
||||||
|
line.RequestBytes, line.LimitHit)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
if calls.Load() != 0 {
|
||||||
|
t.Errorf("the app was called %d times, want none", calls.Load())
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestCountryNotLookedUpWithoutAListOrForAPrivateAddress(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
for _, tc := range []struct {
|
||||||
|
name string
|
||||||
|
env map[string]string
|
||||||
|
clients []string // "" sends no X-Forwarded-For: the client is 127.0.0.1
|
||||||
|
}{
|
||||||
|
{"no country list is set", nil, []string{fromKP, fromDE}},
|
||||||
|
{
|
||||||
|
"private, loopback and link-local addresses",
|
||||||
|
map[string]string{allowedCountries: "de"},
|
||||||
|
[]string{"10.0.0.5", "192.168.1.9", "fd00::5", "", "169.254.0.9", "fe80::9"},
|
||||||
|
},
|
||||||
|
} {
|
||||||
|
t.Run(tc.name, func(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
app := startApp(t, func(http.ResponseWriter, *http.Request) {})
|
||||||
|
geojsURL, asked := startGeoJS(t)
|
||||||
|
env := map[string]string{trustedProxies: trustLocalhost}
|
||||||
|
maps.Copy(env, tc.env)
|
||||||
|
addr, out := startProxyWithGeoJS(t, app.URL, geojsURL, env)
|
||||||
|
|
||||||
|
for i, sent := range tc.clients {
|
||||||
|
req := newRequest(t, http.MethodGet, addr, "/", http.NoBody)
|
||||||
|
if sent != "" {
|
||||||
|
req.Header.Set(forwardedFor, sent)
|
||||||
|
}
|
||||||
|
|
||||||
|
wantStatus(t, do(t, req), http.StatusOK)
|
||||||
|
|
||||||
|
line := out.requestLines(t, i+1)[i]
|
||||||
|
wantLine(t, line, http.StatusOK, requestlog.ActionForward)
|
||||||
|
|
||||||
|
country, present := line.fields["country"]
|
||||||
|
if !present || country != "" {
|
||||||
|
t.Errorf("log line for %q has country %v, want an empty one",
|
||||||
|
line.ClientIP, country)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
if len(asked()) != 0 {
|
||||||
|
t.Errorf("GeoJS was asked about %v, want nothing", asked())
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// startGeoJS starts a stand-in for GeoJS, which places fromDE and fromKP
|
||||||
|
// and no other address. It returns its URL, and what returns the
|
||||||
|
// addresses it has been asked about.
|
||||||
|
func startGeoJS(t *testing.T) (string, func() []string) {
|
||||||
|
t.Helper()
|
||||||
|
|
||||||
|
places := map[string]string{fromDE: "DE", fromKP: "KP"}
|
||||||
|
|
||||||
|
var asked struct {
|
||||||
|
mu sync.Mutex
|
||||||
|
addrs []string
|
||||||
|
}
|
||||||
|
|
||||||
|
geojs := httptest.NewServer(http.HandlerFunc(
|
||||||
|
func(w http.ResponseWriter, r *http.Request) {
|
||||||
|
addrs := strings.Split(r.URL.Query().Get("ip"), ",")
|
||||||
|
|
||||||
|
asked.mu.Lock()
|
||||||
|
asked.addrs = append(asked.addrs, addrs...)
|
||||||
|
asked.mu.Unlock()
|
||||||
|
|
||||||
|
answers := make([]map[string]string, 0, len(addrs))
|
||||||
|
for _, addr := range addrs {
|
||||||
|
answers = append(answers, map[string]string{
|
||||||
|
"ip": addr, "country": places[addr],
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
err := json.NewEncoder(w).Encode(answers)
|
||||||
|
if err != nil {
|
||||||
|
http.Error(w, err.Error(), http.StatusInternalServerError)
|
||||||
|
}
|
||||||
|
}))
|
||||||
|
t.Cleanup(geojs.Close)
|
||||||
|
|
||||||
|
return geojs.URL, func() []string {
|
||||||
|
asked.mu.Lock()
|
||||||
|
defer asked.mu.Unlock()
|
||||||
|
|
||||||
|
return slices.Clone(asked.addrs)
|
||||||
|
}
|
||||||
|
}
|
||||||
+11
-1
@@ -11,6 +11,7 @@ import (
|
|||||||
"time"
|
"time"
|
||||||
|
|
||||||
"sneak.berlin/go/smallwebwaf/internal/config"
|
"sneak.berlin/go/smallwebwaf/internal/config"
|
||||||
|
"sneak.berlin/go/smallwebwaf/internal/lookup"
|
||||||
"sneak.berlin/go/smallwebwaf/internal/ratelimit"
|
"sneak.berlin/go/smallwebwaf/internal/ratelimit"
|
||||||
)
|
)
|
||||||
|
|
||||||
@@ -40,6 +41,9 @@ type Params struct {
|
|||||||
RequestLog io.Writer
|
RequestLog io.Writer
|
||||||
// ProcessLog receives the process's own messages.
|
// ProcessLog receives the process's own messages.
|
||||||
ProcessLog *slog.Logger
|
ProcessLog *slog.Logger
|
||||||
|
// GeoJSURL is where clients' countries are looked up, normally
|
||||||
|
// lookup.URL. GeoJS is asked only while a country list is set.
|
||||||
|
GeoJSURL string
|
||||||
}
|
}
|
||||||
|
|
||||||
// New returns the server smallwebwaf runs: each request it reads passes
|
// New returns the server smallwebwaf runs: each request it reads passes
|
||||||
@@ -63,6 +67,11 @@ func New(params Params) *http.Server {
|
|||||||
PerHour: params.Config.RateLimitPerHour,
|
PerHour: params.Config.RateLimitPerHour,
|
||||||
PerDay: params.Config.RateLimitPerDay,
|
PerDay: params.Config.RateLimitPerDay,
|
||||||
}),
|
}),
|
||||||
|
geojs: lookup.New(lookup.Params{
|
||||||
|
URL: params.GeoJSURL,
|
||||||
|
Now: time.Now,
|
||||||
|
ProcessLog: params.ProcessLog,
|
||||||
|
}),
|
||||||
},
|
},
|
||||||
ReadHeaderTimeout: params.Config.ClientRequestTimeout,
|
ReadHeaderTimeout: params.Config.ClientRequestTimeout,
|
||||||
IdleTimeout: clientIdleTimeout,
|
IdleTimeout: clientIdleTimeout,
|
||||||
@@ -80,6 +89,7 @@ type handler struct {
|
|||||||
errorLog *log.Logger
|
errorLog *log.Logger
|
||||||
transport http.RoundTripper
|
transport http.RoundTripper
|
||||||
limiter *ratelimit.Limiter
|
limiter *ratelimit.Limiter
|
||||||
|
geojs *lookup.GeoJS
|
||||||
}
|
}
|
||||||
|
|
||||||
// newTransport returns what carries requests to the app. It never goes
|
// newTransport returns what carries requests to the app. It never goes
|
||||||
@@ -101,7 +111,7 @@ func (h *handler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
|
|||||||
rq := h.newRequest(w, r)
|
rq := h.newRequest(w, r)
|
||||||
defer rq.finish()
|
defer rq.finish()
|
||||||
|
|
||||||
refused := rq.check()
|
refused := rq.check(r.Context())
|
||||||
if refused != nil {
|
if refused != nil {
|
||||||
rq.answer(*refused)
|
rq.answer(*refused)
|
||||||
|
|
||||||
|
|||||||
@@ -45,6 +45,8 @@ const (
|
|||||||
responseMaxBytes = "SWWAF_RESPONSE_MAX_BYTES"
|
responseMaxBytes = "SWWAF_RESPONSE_MAX_BYTES"
|
||||||
trustedProxies = "SWWAF_TRUSTED_PROXIES"
|
trustedProxies = "SWWAF_TRUSTED_PROXIES"
|
||||||
rateLimitPerMinute = "SWWAF_RATE_LIMIT_PER_MINUTE"
|
rateLimitPerMinute = "SWWAF_RATE_LIMIT_PER_MINUTE"
|
||||||
|
deniedCountries = "SWWAF_DENIED_COUNTRIES"
|
||||||
|
allowedCountries = "SWWAF_EXCLUSIVELY_ALLOWED_COUNTRIES"
|
||||||
)
|
)
|
||||||
|
|
||||||
// output collects what smallwebwaf writes on stdout.
|
// output collects what smallwebwaf writes on stdout.
|
||||||
@@ -159,6 +161,16 @@ func startApp(t *testing.T, app http.HandlerFunc) *httptest.Server {
|
|||||||
func startProxy(t *testing.T, appURL string, env map[string]string) (string, *output) {
|
func startProxy(t *testing.T, appURL string, env map[string]string) (string, *output) {
|
||||||
t.Helper()
|
t.Helper()
|
||||||
|
|
||||||
|
return startProxyWithGeoJS(t, appURL, "", env)
|
||||||
|
}
|
||||||
|
|
||||||
|
// startProxyWithGeoJS is startProxy with clients' countries looked up at
|
||||||
|
// geojsURL.
|
||||||
|
func startProxyWithGeoJS(
|
||||||
|
t *testing.T, appURL, geojsURL string, env map[string]string,
|
||||||
|
) (string, *output) {
|
||||||
|
t.Helper()
|
||||||
|
|
||||||
settings := map[string]string{"SWWAF_UPSTREAM_URL": appURL}
|
settings := map[string]string{"SWWAF_UPSTREAM_URL": appURL}
|
||||||
maps.Copy(settings, env)
|
maps.Copy(settings, env)
|
||||||
|
|
||||||
@@ -176,6 +188,7 @@ func startProxy(t *testing.T, appURL string, env map[string]string) (string, *ou
|
|||||||
Config: cfg,
|
Config: cfg,
|
||||||
RequestLog: out,
|
RequestLog: out,
|
||||||
ProcessLog: requestlog.NewProcessLogger(out),
|
ProcessLog: requestlog.NewProcessLogger(out),
|
||||||
|
GeoJSURL: geojsURL,
|
||||||
})
|
})
|
||||||
|
|
||||||
listener, err := (&net.ListenConfig{}).Listen(t.Context(), "tcp", localhost+":0")
|
listener, err := (&net.ListenConfig{}).Listen(t.Context(), "tcp", localhost+":0")
|
||||||
|
|||||||
@@ -103,9 +103,18 @@ func (h *handler) newRequest(w http.ResponseWriter, r *http.Request) *request {
|
|||||||
|
|
||||||
// check is the one place where a request can be refused once its client
|
// check is the one place where a request can be refused once its client
|
||||||
// is known, before its body is read or anything reaches the app. It
|
// is known, before its body is read or anything reaches the app. It
|
||||||
// returns nil to let the request through. The rate limits come first, so
|
// returns nil to let the request through. The country lists come first,
|
||||||
// that every request is counted, one refused for its size too.
|
// and a request they refuse is not counted for the rate limits; then the
|
||||||
func (rq *request) check() *refusal {
|
// rate limits, so that every other request is counted, one refused for
|
||||||
|
// its size too. ctx is the request's own context.
|
||||||
|
func (rq *request) check(ctx context.Context) *refusal {
|
||||||
|
if rq.countryDenied(ctx) {
|
||||||
|
return &refusal{
|
||||||
|
status: http.StatusForbidden,
|
||||||
|
action: requestlog.ActionCountryDenied,
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
limitHit := rq.h.limiter.Count(clientGroup(rq.client), rq.start)
|
limitHit := rq.h.limiter.Count(clientGroup(rq.client), rq.start)
|
||||||
if limitHit != "" {
|
if limitHit != "" {
|
||||||
rq.line.LimitHit = limitHit
|
rq.line.LimitHit = limitHit
|
||||||
|
|||||||
@@ -26,6 +26,8 @@ const (
|
|||||||
// ActionRateLimited is a request refused because it took its client
|
// ActionRateLimited is a request refused because it took its client
|
||||||
// over a rate limit, or came while the client was over one.
|
// over a rate limit, or came while the client was over one.
|
||||||
ActionRateLimited = "rate_limited"
|
ActionRateLimited = "rate_limited"
|
||||||
|
// ActionCountryDenied is a request refused for its client's country.
|
||||||
|
ActionCountryDenied = "country_denied"
|
||||||
)
|
)
|
||||||
|
|
||||||
// timeLayout is RFC 3339 with milliseconds.
|
// timeLayout is RFC 3339 with milliseconds.
|
||||||
@@ -40,6 +42,7 @@ type Line struct {
|
|||||||
Time string `json:"time"`
|
Time string `json:"time"`
|
||||||
ClientIP string `json:"client_ip"`
|
ClientIP string `json:"client_ip"`
|
||||||
PeerIP string `json:"peer_ip"`
|
PeerIP string `json:"peer_ip"`
|
||||||
|
Country string `json:"country"`
|
||||||
Method string `json:"method"`
|
Method string `json:"method"`
|
||||||
Host string `json:"host"`
|
Host string `json:"host"`
|
||||||
Path string `json:"path"`
|
Path string `json:"path"`
|
||||||
|
|||||||
@@ -16,6 +16,7 @@ import (
|
|||||||
"time"
|
"time"
|
||||||
|
|
||||||
"sneak.berlin/go/smallwebwaf/internal/config"
|
"sneak.berlin/go/smallwebwaf/internal/config"
|
||||||
|
"sneak.berlin/go/smallwebwaf/internal/lookup"
|
||||||
"sneak.berlin/go/smallwebwaf/internal/proxy"
|
"sneak.berlin/go/smallwebwaf/internal/proxy"
|
||||||
"sneak.berlin/go/smallwebwaf/internal/requestlog"
|
"sneak.berlin/go/smallwebwaf/internal/requestlog"
|
||||||
)
|
)
|
||||||
@@ -73,6 +74,7 @@ func Run(ctx context.Context, params Params) int {
|
|||||||
Config: cfg,
|
Config: cfg,
|
||||||
RequestLog: params.Stdout,
|
RequestLog: params.Stdout,
|
||||||
ProcessLog: processLog,
|
ProcessLog: processLog,
|
||||||
|
GeoJSURL: lookup.URL,
|
||||||
})
|
})
|
||||||
|
|
||||||
processLog.Info("starting",
|
processLog.Info("starting",
|
||||||
|
|||||||
@@ -176,18 +176,20 @@ func wantStartingLine(t *testing.T, line map[string]any, appURL string) {
|
|||||||
|
|
||||||
settings, _ := line["settings"].(map[string]any)
|
settings, _ := line["settings"].(map[string]any)
|
||||||
want := map[string]any{
|
want := map[string]any{
|
||||||
listenAddr: localhost + ":0",
|
listenAddr: localhost + ":0",
|
||||||
"SWWAF_UPSTREAM_URL": appURL,
|
"SWWAF_UPSTREAM_URL": appURL,
|
||||||
"SWWAF_TRUSTED_PROXIES": "10.0.0.0/8,172.16.0.0/12,192.168.0.0/16",
|
"SWWAF_TRUSTED_PROXIES": "10.0.0.0/8,172.16.0.0/12,192.168.0.0/16",
|
||||||
"SWWAF_CLIENT_REQUEST_TIMEOUT": "60s",
|
"SWWAF_CLIENT_REQUEST_TIMEOUT": "60s",
|
||||||
"SWWAF_CLIENT_RESPONSE_TIMEOUT": "30m",
|
"SWWAF_CLIENT_RESPONSE_TIMEOUT": "30m",
|
||||||
"SWWAF_UPSTREAM_REQUEST_TIMEOUT": "60s",
|
"SWWAF_UPSTREAM_REQUEST_TIMEOUT": "60s",
|
||||||
"SWWAF_UPSTREAM_RESPONSE_TIMEOUT": "30m",
|
"SWWAF_UPSTREAM_RESPONSE_TIMEOUT": "30m",
|
||||||
"SWWAF_REQUEST_MAX_BYTES": "100M",
|
"SWWAF_REQUEST_MAX_BYTES": "100M",
|
||||||
"SWWAF_RESPONSE_MAX_BYTES": "5G",
|
"SWWAF_RESPONSE_MAX_BYTES": "5G",
|
||||||
"SWWAF_RATE_LIMIT_PER_MINUTE": "1000",
|
"SWWAF_RATE_LIMIT_PER_MINUTE": "1000",
|
||||||
"SWWAF_RATE_LIMIT_PER_HOUR": "10000",
|
"SWWAF_RATE_LIMIT_PER_HOUR": "10000",
|
||||||
"SWWAF_RATE_LIMIT_PER_DAY": "50000",
|
"SWWAF_RATE_LIMIT_PER_DAY": "50000",
|
||||||
|
"SWWAF_DENIED_COUNTRIES": "",
|
||||||
|
"SWWAF_EXCLUSIVELY_ALLOWED_COUNTRIES": "",
|
||||||
}
|
}
|
||||||
|
|
||||||
for name, value := range want {
|
for name, value := range want {
|
||||||
|
|||||||
Reference in New Issue
Block a user