Country allow and deny lists, looked up through GeoJS (closes #44)
check / check (push) Failing after 3s
check / check (push) Failing after 3s
SWWAF_DENIED_COUNTRIES and SWWAF_EXCLUSIVELY_ALLOWED_COUNTRIES refuse a request with 403 before its body is read or rate-limited, logged as country_denied. internal/lookup asks GeoJS only while a list is set, 200 clients per request, one at a time, keeping answers 7 days. Failures, a redirect or an answer leaving an address out included, are logged without addresses; GeoJS is then left alone a second, doubling to five minutes. Private, loopback and link-local clients are never sent. Deviation, per the issue: no SWWAF_LOOKUP_SOURCE or SWWAF_LOOKUP_TIMEOUT; 403, not SWWAF_BAN_RESPONSE. Deviation: GeoJS's country endpoint, not geo.json. Judgement call: an IPv6 /64 is asked about by its first address; at most 10,000 clients wait. Judgement call: config.go lists the ISO 3166-1 codes; no widely used library holds them. Model: opus-5-5
This commit was merged in pull request #54.
This commit is contained in:
@@ -12,15 +12,15 @@ state in memory and in JSON files you can read and edit, and writes a detailed
|
||||
JSON log line for every request.
|
||||
|
||||
Status: the first milestone is built
|
||||
(https://git.eeqj.de/sneak/smallwebwaf/issues/13), and the rate limits of the
|
||||
second (https://git.eeqj.de/sneak/smallwebwaf/issues/14). `smallwebwaf` passes
|
||||
each request to the app and the app's answer back, unchanged, within its
|
||||
timeouts and size limits, works out each client's address, refuses a client that
|
||||
sends too many requests, and writes a JSON log line for every request. The
|
||||
country lists and the image an app builds on come with the rest of milestone 2,
|
||||
and the rest of the design after that, in the order of the build order in
|
||||
[`SPEC.md`](SPEC.md). The survey of existing tools that led to the design is in
|
||||
[`EVALUATION.md`](EVALUATION.md).
|
||||
(https://git.eeqj.de/sneak/smallwebwaf/issues/13), and the rate limits and
|
||||
country lists of the second (https://git.eeqj.de/sneak/smallwebwaf/issues/14).
|
||||
`smallwebwaf` passes each request to the app and the app's answer back,
|
||||
unchanged, within its timeouts and size limits, works out each client's address,
|
||||
refuses a client that sends too many requests or comes from a country you
|
||||
refuse, and writes a JSON log line for every request. The image an app builds on
|
||||
comes with the rest of milestone 2, and the rest of the design after that, in
|
||||
the order of the build order in [`SPEC.md`](SPEC.md). The survey of existing
|
||||
tools that led to the design is in [`EVALUATION.md`](EVALUATION.md).
|
||||
|
||||
## Getting started
|
||||
|
||||
@@ -72,6 +72,13 @@ gives those in progress five seconds to finish.
|
||||
much of it the window still covers. At most 20,000 clients are kept, the least
|
||||
recently seen dropped first, and only in memory: a restart starts every client
|
||||
afresh.
|
||||
- Refuses a request from a country you refuse with `403`, as soon as the
|
||||
client's country is known and before its body is read; such a request is not
|
||||
counted for the rate limits. While one of the country lists below is set, each
|
||||
client's country is looked up through GeoJS (see "Country and AS number
|
||||
lookup" below); with neither set, no visitor's address leaves the host. A
|
||||
client on a private, loopback or link-local address has no country, and
|
||||
neither list checks it.
|
||||
- Writes a line in the request log for each request (see "Request log" below).
|
||||
|
||||
## Settings
|
||||
@@ -102,20 +109,30 @@ it, and the effective settings are logged at start.
|
||||
requests a client may make in a minute, an hour and a day. The defaults are
|
||||
several times what one busy person produces, since a browser loading a heavy
|
||||
page makes a few hundred requests and several people often share one address.
|
||||
- `SWWAF_DENIED_COUNTRIES` (default empty): countries whose clients are refused,
|
||||
for example `cn,ru,kp`.
|
||||
- `SWWAF_EXCLUSIVELY_ALLOWED_COUNTRIES` (default empty): when set, the only
|
||||
countries whose clients get through, for example `us,de`. A client whose
|
||||
country cannot be found is refused too, so that new clients are not let in
|
||||
whenever GeoJS stops answering.
|
||||
|
||||
Durations are in Go's syntax, with `d` for days (`90s`, `15m`, `7d`). Sizes are
|
||||
bytes, with an optional `K`, `M` or `G`, which are powers of 1024 (`1K` is 1024
|
||||
bytes). Rate limits are whole numbers of requests. Netblocks are in CIDR form,
|
||||
and a bare address stands for itself alone. `off` switches a timeout, a size
|
||||
limit or a rate limit off.
|
||||
and a bare address stands for itself alone. Countries are the two-letter codes
|
||||
ISO 3166-1 assigns today, and `xk` for Kosovo, in either case (`de` and `DE` are
|
||||
the same); any other code, such as `nk` (North Korea is `kp`) or the withdrawn
|
||||
`su`, stops the start, and so does a code on both country lists. `off` switches
|
||||
a timeout, a size limit or a rate limit off.
|
||||
|
||||
Four limits are fixed rather than settings. The request line and headers may
|
||||
Several limits are fixed rather than settings. The request line and headers may
|
||||
take up to 32 KiB, above which the answer is `431` and nothing reaches the app.
|
||||
A kept-open connection that sends nothing for 120 seconds is closed. That is
|
||||
longer than the 90 seconds after which traefik closes a connection it is not
|
||||
using, so traefik never sends a request on a connection `smallwebwaf` is
|
||||
closing. At most 20,000 clients are kept for the rate limits, and an IPv6 client
|
||||
is counted by its /64.
|
||||
is counted by its /64. A new client waits at most a second for its country, and
|
||||
at most 100,000 answers from GeoJS are kept, for 7 days each.
|
||||
|
||||
## Request log
|
||||
|
||||
@@ -123,18 +140,23 @@ is counted by its /64.
|
||||
refused ones included:
|
||||
|
||||
```
|
||||
{"type":"request","time":"2026-10-03T12:00:00.123Z","client_ip":"203.0.113.9","peer_ip":"172.18.0.2","method":"GET","host":"app.example","path":"/","query":"","protocol":"HTTP/1.1","status":200,"upstream_status":200,"request_bytes":0,"response_bytes":5120,"referer":"","user_agent":"curl/8.9.1","action":"forward","duration_total":3.217,"duration_upstream_total":3.104}
|
||||
{"type":"request","time":"2026-10-03T12:00:00.123Z","client_ip":"203.0.113.9","peer_ip":"172.18.0.2","country":"DE","method":"GET","host":"app.example","path":"/","query":"","protocol":"HTTP/1.1","status":200,"upstream_status":200,"request_bytes":0,"response_bytes":5120,"referer":"","user_agent":"curl/8.9.1","action":"forward","duration_total":3.217,"duration_upstream_total":3.104}
|
||||
```
|
||||
|
||||
- `time` is when the request arrived, in UTC. `peer_ip` is the TCP peer,
|
||||
normally traefik. `path` and `query` are as the client sent them.
|
||||
- `country` is the client's country as GeoJS places it, and empty when it is not
|
||||
known: with neither country list set, for a client on a private, loopback or
|
||||
link-local address, and when GeoJS cannot place the client or has not answered
|
||||
in time.
|
||||
- `status` is what the client was sent, `0` if nothing was; `upstream_status` is
|
||||
what the app answered, and is left out when the app did not answer.
|
||||
- `request_bytes` and `response_bytes` count body bytes.
|
||||
- `action` is `forward` for a request passed to the app, `rate_limited` for one
|
||||
refused for a rate limit, `too_large` for a request or response over its size
|
||||
limit, `timed_out` for one that ran out of time, and `upstream_error` when the
|
||||
app could not be reached or its answer broke off.
|
||||
- `action` is `forward` for a request passed to the app, `country_denied` for
|
||||
one refused for its client's country, `rate_limited` for one refused for a
|
||||
rate limit, `too_large` for a request or response over its size limit,
|
||||
`timed_out` for one that ran out of time, and `upstream_error` when the app
|
||||
could not be reached or its answer broke off.
|
||||
- `limit_hit` is there for a request refused for a rate limit, and names the
|
||||
window whose limit it went over: `minute`, `hour` or `day`, the shortest if it
|
||||
went over several.
|
||||
@@ -367,34 +389,49 @@ the metrics, failure behaviour and the build order.
|
||||
|
||||
## Country and AS number lookup
|
||||
|
||||
`smallwebwaf` looks up the AS number and country of every client, for the
|
||||
request log, the metrics and the ban notes, and for the country lists and biased
|
||||
limits when you set them. It works with no setup: by default it asks the free
|
||||
GeoJS web service, which needs no account and no file. This means that, by
|
||||
default, the address of every new visitor is sent to GeoJS. Each answer is kept
|
||||
in memory for seven days, and many addresses are asked about in one request;
|
||||
writing the answers to disk, so that they survive a restart, comes in milestone
|
||||
3 or later (see the build order in [`SPEC.md`](SPEC.md)). GeoJS publishes no
|
||||
rate limit but may block a caller it thinks asks too much; while it is not
|
||||
answering, new visitors count as coming from an unknown country, which
|
||||
So far `smallwebwaf` looks up only the country, only through GeoJS, and only
|
||||
while `SWWAF_DENIED_COUNTRIES` or `SWWAF_EXCLUSIVELY_ALLOWED_COUNTRIES` is set:
|
||||
then the address of every new visitor is sent to GeoJS, and with neither set,
|
||||
none is. An IPv6 visitor is asked about by the first address of its /64. A new
|
||||
visitor waits at most a second for its answer, and without one counts as coming
|
||||
from an unknown country until the answer arrives. The addresses waiting are
|
||||
asked about together, up to 200 in one request, one request at a time; at most
|
||||
10,000 visitors wait, and one more counts as coming from an unknown country
|
||||
until there is room. While GeoJS fails, visitors with a kept answer are
|
||||
unaffected and new ones count as coming from an unknown country. GeoJS is then
|
||||
left alone for a second, twice as long after each further failure up to five
|
||||
minutes, and asked again by the next request that needs it.
|
||||
|
||||
In the full design, `smallwebwaf` looks up the AS number and country of every
|
||||
client, for the request log, the metrics and the ban notes, and for the country
|
||||
lists and biased limits when you set them. It works with no setup: by default it
|
||||
asks the free GeoJS web service, which needs no account and no file. This means
|
||||
that, by default, the address of every new visitor is sent to GeoJS. Each answer
|
||||
is kept in memory for seven days, and many addresses are asked about in one
|
||||
request; writing the answers to disk, so that they survive a restart, comes in
|
||||
milestone 3 or later (see the build order in [`SPEC.md`](SPEC.md)). GeoJS
|
||||
publishes no rate limit but may block a caller it thinks asks too much; while it
|
||||
is not answering, new visitors count as coming from an unknown country, which
|
||||
`SWWAF_EXCLUSIVELY_ALLOWED_COUNTRIES` refuses.
|
||||
|
||||
To keep your visitors' addresses on your own host, set
|
||||
`SWWAF_LOOKUP_SOURCE=off`, or use the database file instead of GeoJS:
|
||||
`SWWAF_LOOKUP_SOURCE=file` reads the free IPinfo Lite database
|
||||
(`ipinfo_lite.mmdb`). You download it with your own IPinfo account, mount the
|
||||
directory that holds it into the container, point `SWWAF_LOOKUP_DB_PATH` at the
|
||||
file and refresh it when you choose; `smallwebwaf` never downloads it itself,
|
||||
and reads it again when you replace it. It has to be the directory rather than
|
||||
the file itself: docker does not show a single mounted file being replaced, so a
|
||||
refresh would go unseen. IPinfo releases it under the Creative Commons
|
||||
Attribution-ShareAlike 4.0 International License and asks for attribution, in
|
||||
its own words on https://ipinfo.io/lite: "The attribution requirements can be
|
||||
met by giving our service credit as your data source. Simply place a link to
|
||||
IPinfo on the website, application, or social media account that uses our data."
|
||||
Its example of such a credit is a link mentioning "IP address data is powered by
|
||||
IPinfo". A service that uses the database through `smallwebwaf` should carry
|
||||
that link.
|
||||
(`ipinfo_lite.mmdb`). `SWWAF_LOOKUP_SOURCE` comes in milestone 3 or later (see
|
||||
the build order in [`SPEC.md`](SPEC.md)); until then GeoJS is asked only while a
|
||||
country list is set. You download the database with your own IPinfo account,
|
||||
mount the directory that holds it into the container, point
|
||||
`SWWAF_LOOKUP_DB_PATH` at the file and refresh it when you choose; `smallwebwaf`
|
||||
never downloads it itself, and reads it again when you replace it. It has to be
|
||||
the directory rather than the file itself: docker does not show a single mounted
|
||||
file being replaced, so a refresh would go unseen. IPinfo releases it under the
|
||||
Creative Commons Attribution-ShareAlike 4.0 International License and asks for
|
||||
attribution, in its own words on https://ipinfo.io/lite: "The attribution
|
||||
requirements can be met by giving our service credit as your data source. Simply
|
||||
place a link to IPinfo on the website, application, or social media account that
|
||||
uses our data." Its example of such a credit is a link mentioning "IP address
|
||||
data is powered by IPinfo". A service that uses the database through
|
||||
`smallwebwaf` should carry that link.
|
||||
|
||||
Neither source can place a private address, so a client on one, such as a
|
||||
visitor on your local network, another container or your monitoring, has no
|
||||
@@ -413,16 +450,18 @@ refusal comes with `SWWAF_ALLOW_NETS` in milestone 3 or later.
|
||||
the checks, passes the request to the app and the answer back with the
|
||||
standard library's `httputil.ReverseProxy` within the timeouts and size
|
||||
limits, and writes the request's log line. Its `check` method is where a
|
||||
request is refused before anything reaches the app: for a rate limit, for an
|
||||
announced body over the size limit, and, with the rest of milestone 2, for the
|
||||
country lists.
|
||||
request is refused before anything reaches the app: for the country lists, for
|
||||
a rate limit, and for an announced body over the size limit.
|
||||
- `internal/lookup`: looks up each client's country through GeoJS, and keeps the
|
||||
answers.
|
||||
- `internal/ratelimit`: counts each client's requests and tells when one takes
|
||||
it over a rate limit.
|
||||
- `internal/requestlog`: the lines on stdout: the request log line and the
|
||||
process's own messages.
|
||||
|
||||
Besides the Go standard library, `github.com/hashicorp/golang-lru/v2` keeps the
|
||||
table of clients to 20,000, dropping the least recently seen.
|
||||
table of clients to 20,000 and the GeoJS answers to 100,000, dropping the least
|
||||
recently seen. The country codes are the list in `internal/config/config.go`.
|
||||
|
||||
## Entrypoints
|
||||
|
||||
@@ -456,8 +495,9 @@ so that they run in minimal containers.
|
||||
|
||||
## TODO
|
||||
|
||||
- Milestone 2: the country lists and the image an app builds on
|
||||
(https://git.eeqj.de/sneak/smallwebwaf/issues/14); its rate limits are built.
|
||||
- Milestone 2: the image an app builds on
|
||||
(https://git.eeqj.de/sneak/smallwebwaf/issues/14); its rate limits and country
|
||||
lists are built.
|
||||
- The rest of the design, in the order of the build order in
|
||||
[`SPEC.md`](SPEC.md).
|
||||
|
||||
|
||||
Reference in New Issue
Block a user