sneak/smallwebwaf
1
0
Fork 0
Code Issues 36 Pull Requests 2 Actions 22 Packages Projects Releases Wiki Activity

SPEC and README: bodies unread by default, one listener, GeoJS by default (closes #6)

Browse Source 
Address the third review of the spec update and sneak's two new rulings.
By default the Core Rule Set reads the URL, query string and headers but
no request body, since a code forge's bodies carry code it takes for
attacks; `WAF_BODY_LIMIT` switches body inspection on. `Content-Encoding`
is allowed only on unread bodies, and OAuth sign-in from local tools
passes. The default rule file bans common probes for secrets, version
control, backups and logs at the site root. The spec names the Core Rule
Set 4.25.0. One listener answers the sidecar's own endpoints under
`/_smallwebwaf/`, behind tokens. GeoJS is the default lookup source. Size
suffixes are powers of 1024.

Model: opus-5-5
This commit is contained in:
sneak
2026-09-28 18:50:00 +00:00
parent 2d15e7e746
commit 4020ff83c6
2 changed files with 312 additions and 199 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+35 -27
View File
@@ -55,15 +55,16 @@ goes through the candidates one by one.
counted by /64 by default. counted by /64 by default.
- Size and time limits on requests and responses, both between the client and - Size and time limits on requests and responses, both between the client and
`smallwebwaf` and between `smallwebwaf` and the app: by default a request may `smallwebwaf` and between `smallwebwaf` and the app: by default a request may
take 60 seconds and 100 MB, a response 30 minutes and 5 GB. take 60 seconds and 100 MiB, a response 30 minutes and 5 GiB.
- Rate limits per client on requests per minute, per hour and per day, and on - Rate limits per client on requests per minute, per hour and per day, and on
bytes per minute, per hour and per day, on by default and set well above what bytes per minute, per hour and per day, on by default and set well above what
real visitors need. real visitors need.
- Netblocks that bypass rate limiting, netblocks that bypass everything, and - Netblocks that bypass rate limiting, netblocks that bypass everything, and
netblocks that are always refused. netblocks that are always refused.
- AS number and country lookup for every client, off until you choose a source: - AS number and country lookup for every client, on by default through the free
the IPinfo Lite database file, which you download and mount, or the free GeoJS GeoJS web service, which is sent the address of every new visitor. The IPinfo
web service (see "Country and AS number lookup" below). Lite database file, which you download and mount, can be used instead, or
lookups switched off (see "Country and AS number lookup" below).
- Country lists: `DENIED_COUNTRIES` refuses every request from the countries - Country lists: `DENIED_COUNTRIES` refuses every request from the countries
listed, `EXCLUSIVELY_ALLOWED_COUNTRIES` every request from anywhere else. Such listed, `EXCLUSIVELY_ALLOWED_COUNTRIES` every request from anywhere else. Such
a request gets the answer a banned client gets as soon as the client's address a request gets the answer a banned client gets as soon as the client's address
@@ -78,7 +79,9 @@ goes through the candidates one by one.
scanning and penetration probes; easy to edit by hand, and picked up while scanning and penetration probes; easy to edit by hand, and picked up while
running; running;
- the OWASP Core Rule Set, run by the Coraza engine, refusing the requests - the OWASP Core Rule Set, run by the Coraza engine, refusing the requests
it flags; it flags; it reads the URL and headers of every request, and request
bodies only once you switch that on, since on a code forge they are full
of code it would take for attacks;
- trap paths, and a ban for a client that the rule files or the Core Rule - trap paths, and a ban for a client that the rule files or the Core Rule
Set refuse again and again. Set refuse again and again.
- Bans: - Bans:
@@ -102,14 +105,17 @@ goes through the candidates one by one.
- Request log: one JSON object per request on stdout with the usual web log - Request log: one JSON object per request on stdout with the usual web log
fields, the decision taken and why, AS number and country, and timings. fields, the decision taken and why, AS number and country, and timings.
Optionally also sent to a remote syslog server. Optionally also sent to a remote syslog server.
- Prometheus metrics on their own port. - Prometheus metrics, for a scraper that holds the metrics token.
- State (bans with their notes, each client's counters and history, the GeoJS - State (bans with their notes, each client's counters and history, the GeoJS
answers, the reputation cache, the alerting state) held in memory and kept in answers, the reputation cache, the alerting state) held in memory and kept in
readable JSON files, written regularly and at every stop, so a restart loses readable JSON files, written regularly and at every stop, so a restart loses
nothing. Edit a file, or add a rule file, and the running `smallwebwaf` picks nothing. Edit a file, or add a rule file, and the running `smallwebwaf` picks
up the change. Nothing is read from disk while serving a request. up the change. Nothing is read from disk while serving a request.
- A small admin endpoint for health checks, listing, adding and lifting bans, - Health checks, the metrics, and listing, adding and lifting bans or asking why
and asking why a given address was refused. a given address was refused, all on the one port every request uses: under
`/_smallwebwaf/` on the app's own address, through traefik like any other
request. The metrics need the metrics token, and the ban management the admin
token.
Not planned: TLS termination, routing for several apps, browser challenges Not planned: TLS termination, routing for several apps, browser challenges
(captcha or proof of work), a web console, or defence against floods large (captcha or proof of work), a web console, or defence against floods large
@@ -182,30 +188,32 @@ failure behaviour and the build order.
## Country and AS number lookup ## Country and AS number lookup
AS number and country lookups, and the country lists and biased limits that use `smallwebwaf` looks up the AS number and country of every client, for the
them, are off until you choose one of two sources with `LOOKUP_SOURCE`. request log, the metrics and the ban notes, and for the country lists and biased
limits when you set them. It works with no setup: by default it asks the free
`LOOKUP_SOURCE=file` reads the free IPinfo Lite database (`ipinfo_lite.mmdb`). GeoJS web service, which needs no account and no file. This means that, by
You download it with your own IPinfo account, mount the directory that holds it default, the address of every new visitor is sent to GeoJS. Each answer is kept
into the container, point `LOOKUP_DB_PATH` at the file and refresh it when you
choose; `smallwebwaf` never downloads it itself, and reads it again when you
replace it. It has to be the directory rather than the file itself: docker does
not show a single mounted file being replaced, so a refresh would go unseen.
IPinfo releases it under the Creative Commons Attribution-ShareAlike 4.0
International License and asks for attribution, in its own words on
https://ipinfo.io/lite: "The attribution requirements can be met by giving our
service credit as your data source. Simply place a link to IPinfo on the
website, application, or social media account that uses our data." Its example
of such a credit is a link mentioning "IP address data is powered by IPinfo". A
service that uses the database through `smallwebwaf` should carry that link.
`LOOKUP_SOURCE=geojs` asks the free GeoJS web service instead, with no account
and no file. Every new visitor's address is sent to GeoJS. Each answer is kept
for seven days, across restarts, and many addresses are asked about in one for seven days, across restarts, and many addresses are asked about in one
request. GeoJS publishes no rate limit but may block a caller it thinks asks too request. GeoJS publishes no rate limit but may block a caller it thinks asks too
much; while it is not answering, new visitors count as coming from an unknown much; while it is not answering, new visitors count as coming from an unknown
country, which `EXCLUSIVELY_ALLOWED_COUNTRIES` refuses. country, which `EXCLUSIVELY_ALLOWED_COUNTRIES` refuses.
To keep your visitors' addresses on your own host, set `LOOKUP_SOURCE=off`, or
use the database file instead of GeoJS: `LOOKUP_SOURCE=file` reads the free
IPinfo Lite database (`ipinfo_lite.mmdb`). You download it with your own IPinfo
account, mount the directory that holds it into the container, point
`LOOKUP_DB_PATH` at the file and refresh it when you choose; `smallwebwaf` never
downloads it itself, and reads it again when you replace it. It has to be the
directory rather than the file itself: docker does not show a single mounted
file being replaced, so a refresh would go unseen. IPinfo releases it under the
Creative Commons Attribution-ShareAlike 4.0 International License and asks for
attribution, in its own words on https://ipinfo.io/lite: "The attribution
requirements can be met by giving our service credit as your data source. Simply
place a link to IPinfo on the website, application, or social media account that
uses our data." Its example of such a credit is a link mentioning "IP address
data is powered by IPinfo". A service that uses the database through
`smallwebwaf` should carry that link.
Neither source can place a private address, so a client on one, such as a Neither source can place a private address, so a client on one, such as a
visitor on your local network, another container or your monitoring, has no visitor on your local network, another container or your monitoring, has no
country: `EXCLUSIVELY_ALLOWED_COUNTRIES` refuses it unless you list it in country: `EXCLUSIVELY_ALLOWED_COUNTRIES` refuses it unless you list it in
SPEC.md
+277 -172
View File
@@ -30,8 +30,9 @@ directory of hand-editable text files.
- Defence against traffic floods that saturate the host's network link. That - Defence against traffic floods that saturate the host's network link. That
needs help upstream of the host. needs help upstream of the host.
- A web UI or a configuration file. Settings are environment variables. Apart - A web UI or a configuration file. Settings are environment variables. Apart
from its own state files and the lookup database, the only files read are the from settings given as files (`NAME_FILE`, `LOG_REMOTE_TLS_CA_FILE`), its own
rule files, which hold one regex per line and nothing more elaborate. state files and the lookup database, the only files read are the rule files,
which hold one regex per line and nothing more elaborate.
- Sharing bans between sidecars in the first version: each sidecar keeps its - Sharing bans between sidecars in the first version: each sidecar keeps its
own. Running one CrowdSec engine per host, which every sidecar would report own. Running one CrowdSec engine per host, which every sidecar would report
its bans to, is reconsidered once real ban volumes are known. Reading a its bans to, is reconsidered once real ban volumes are known. Reading a
@@ -44,17 +45,16 @@ directory of hand-editable text files.
that volume (`/data`), so docker supplies an anonymous one when none is that volume (`/data`), so docker supplies an anonymous one when none is
mounted, and a sidecar started with only `UPSTREAM_URL` has somewhere to mounted, and a sidecar started with only `UPSTREAM_URL` has somewhere to
write. write.
- Three listeners, of which only the first is routed by traefik: - One listener, which traefik routes to. It forwards requests to the app, and it
- the proxy listener; answers the sidecar's own endpoints for health, metrics and ban management,
- a metrics listener serving Prometheus metrics, which can be switched off; under `/_smallwebwaf/`, which never reach the app (see "Admin endpoints").
- an admin listener for health and ban management.
- Components inside the process: - Components inside the process:
- Client identification: works out the real client IP from - Client identification: works out the real client IP from
`X-Forwarded-For`, trusting only the proxy netblocks in `TRUSTED_PROXIES`, `X-Forwarded-For`, trusting only the proxy netblocks in `TRUSTED_PROXIES`,
by default the private address ranges. by default the private address ranges.
- Lookup: AS number and country, from one of two sources the operator - Lookup: AS number and country, by default from the GeoJS web service,
chooses: a database file the operator supplies, held in memory, or the whose answers are kept for seven days, or instead from a database file the
GeoJS web service, whose answers are kept for seven days. operator supplies, held in memory.
- Reputation: static blocklists fetched on a schedule; DNSBL and reputation - Reputation: static blocklists fetched on a schedule; DNSBL and reputation
API queries made in the background and cached. API queries made in the background and cached.
- Counters: per-client request and byte counts per minute, hour and day, - Counters: per-client request and byte counts per minute, hour and day,
@@ -70,8 +70,7 @@ directory of hand-editable text files.
any kind is used. any kind is used.
- Request log: one JSON object per request on stdout, optionally also sent - Request log: one JSON object per request on stdout, optionally also sent
to a remote syslog server (see "Request log"). to a remote syslog server (see "Request log").
- Metrics: Prometheus counters and gauges on their own listener (see - Metrics: Prometheus counters and gauges (see "Metrics endpoint").
"Metrics endpoint").
- Alerting: a queue with de-duplication feeding webhook, Slack and ntfy - Alerting: a queue with de-duplication feeding webhook, Slack and ntfy
senders. senders.
- Proxy: the standard library's `net/http/httputil.ReverseProxy`, streaming - Proxy: the standard library's `net/http/httputil.ReverseProxy`, streaming
@@ -82,7 +81,12 @@ directory of hand-editable text files.
- standard library for the proxy, HTTP clients, DNS, logging (`log/slog`), - standard library for the proxy, HTTP clients, DNS, logging (`log/slog`),
state files (`encoding/json`, `os.Rename`); state files (`encoding/json`, `os.Rename`);
- `github.com/corazawaf/coraza/v3` and - `github.com/corazawaf/coraza/v3` and
`github.com/corazawaf/coraza-coreruleset` for attack detection; `github.com/corazawaf/coraza-coreruleset/v4` at `v4.25.0`, which carries
the Core Rule Set 4.25.0, for attack detection. The sidecar's changes to
the Core Rule Set and the default `WAF_DISABLED_RULES` (see "Configuration
surface", attack detection) are built for that version, since rule ids and
what each rule matches change between versions, and are checked again
before the version changes;
- `github.com/oschwald/maxminddb-golang/v2`, the current major version, to - `github.com/oschwald/maxminddb-golang/v2`, the current major version, to
read the lookup database; read the lookup database;
- `github.com/fsnotify/fsnotify` to notice files that are edited or added - `github.com/fsnotify/fsnotify` to notice files that are edited or added
@@ -99,6 +103,7 @@ Steps run in this order; the first step that produces a final answer ends
processing. The size and time limits (see "Configuration surface") apply to the processing. The size and time limits (see "Configuration surface") apply to the
whole exchange. whole exchange.
- A `GET /_smallwebwaf/healthz` is answered at once (see "Admin endpoints").
- Identify the client. - Identify the client.
- If the TCP peer is inside `TRUSTED_PROXIES`, walk `X-Forwarded-For` from - If the TCP peer is inside `TRUSTED_PROXIES`, walk `X-Forwarded-For` from
the right and take the first address not inside `TRUSTED_PROXIES`. If the right and take the first address not inside `TRUSTED_PROXIES`. If
@@ -112,17 +117,22 @@ whole exchange.
counting and banning, because one abuser usually controls a whole /64. A counting and banning, because one abuser usually controls a whole /64. A
client is therefore one IPv4 address or one IPv6 group, a /64 by default. client is therefore one IPv4 address or one IPv6 group, a /64 by default.
- Static lists. - Static lists.
- In `ALLOW_NETS`: skip every check below and forward. Still counted for - In `ALLOW_NETS`: skip every check below and forward, or answer a request
anomaly alerts. under `/_smallwebwaf/` (see "Admin endpoints"). Still counted for anomaly
alerts.
- In `DENY_NETS`: refuse. - In `DENY_NETS`: refuse.
- Ban ledger. An active ban on the client's netblock: refuse with - Ban ledger. An active ban on the client's netblock: refuse with
`BAN_RESPONSE`. If a clear sign of attack caused the ban, the ban becomes `BAN_RESPONSE`. If a clear sign of attack caused the ban, the ban becomes
permanent (see "Bans"). permanent (see "Bans").
- Look up the AS number and country, unless `LOOKUP_SOURCE` is `off`. With - Look up the AS number and country, unless `LOOKUP_SOURCE` is `off`. With
GeoJS, a client whose answer is not yet kept waits for it, up to GeoJS, the default, a request waits for its client's first answer, up to
`LOOKUP_TIMEOUT`. A client the lookup cannot place, which includes every `LOOKUP_TIMEOUT`, only when a setting needs it before the request goes on: the
private, loopback and link-local address, has an unknown country and AS country lists, the biased thresholds or `ADD_LOOKUP_HEADERS`. Otherwise the
number: the exclusive country list refuses it, the biased thresholds give it request goes on at once; the answer is added to the client's history and ban
notes when it comes, and a request that ends before then is logged without it.
A client the lookup cannot place, which includes every private, loopback and
link-local address, has an unknown country and AS number: the exclusive
country list refuses it, the biased thresholds give it
`UNKNOWN_LIMIT_PERCENT`, and nothing else treats it differently. `UNKNOWN_LIMIT_PERCENT`, and nothing else treats it differently.
- Country lists. A client whose country is in `DENIED_COUNTRIES`, or, when - Country lists. A client whose country is in `DENIED_COUNTRIES`, or, when
`EXCLUSIVELY_ALLOWED_COUNTRIES` is set, is not in it, is refused with `EXCLUSIVELY_ALLOWED_COUNTRIES` is set, is not in it, is refused with
@@ -145,13 +155,17 @@ whole exchange.
in file name order then line order. Each rule that matches takes its action: in file name order then line order. Each rule that matches takes its action:
only log, refuse with 403, or refuse and ban. Matching stops at the first rule only log, refuse with 403, or refuse and ban. Matching stops at the first rule
that refuses or bans. that refuses or bans.
- Core Rule Set inspection of the request: its headers, its URL, and its body up - Core Rule Set inspection of the request: its method, its URL with the query
to `WAF_BODY_LIMIT` when the body is form data, multipart, JSON or XML, the string, and its headers. By default no body is read. When `WAF_BODY_LIMIT` is
kinds the Core Rule Set can read. Any other body, such as a git push or a set to a size, a body is read up to that size if it is form data, multipart,
container image layer, reaches the app uninspected, and so does a JSON or XML JSON or XML, the kinds the Core Rule Set can read. Any other body, such as a
body larger than `WAF_BODY_LIMIT`, which cannot be read in part. Responses are git push or a container image layer, reaches the app uninspected, and so does
not inspected. In `block` mode, the default, a match at or over the anomaly a JSON or XML body larger than `WAF_BODY_LIMIT`, which cannot be read in part.
threshold is refused with 403; in `detect` mode it is only logged and alerted. Responses are not inspected. In `block` mode, the default, a match at or over
the anomaly threshold is refused with 403; in `detect` mode it is only logged
and alerted.
- A request under `/_smallwebwaf/` is answered by the sidecar here and goes no
further (see "Admin endpoints").
- Forward to `UPSTREAM_URL`, streaming. Add `X-Forwarded-For` and, if enabled, - Forward to `UPSTREAM_URL`, streaming. Add `X-Forwarded-For` and, if enabled,
`X-Client-ASN` and `X-Client-Country` for the app's own logs. `X-Client-ASN` and `X-Client-Country` for the app's own logs.
- After the response. - After the response.
@@ -159,7 +173,8 @@ whole exchange.
whose byte total passes a byte limit times the percentage has broken that whose byte total passes a byte limit times the percentage has broken that
limit and is banned. The response in progress is not cut off. limit and is banned. The response in progress is not cut off.
- Count the client's requests that the sidecar refused after a rule file or - Count the client's requests that the sidecar refused after a rule file or
Core Rule Set match; answers the app gave are not counted. More than Core Rule Set match, or for a missing or wrong token (see "Admin
endpoints"); answers the app gave are not counted. More than
`ERROR_BURST_THRESHOLD` of them within a minute breaks a limit and bans `ERROR_BURST_THRESHOLD` of them within a minute breaks a limit and bans
the client. the client.
- Update the client's history and the anomaly counters, and evaluate alert - Update the client's history and the anomaly counters, and evaluate alert
@@ -216,7 +231,8 @@ to a ban, each by its own rule:
next clear sign of attack bans it permanently at once. next clear sign of attack bans it permanently at once.
- A broken limit: a request over a request limit, a response that takes the - A broken limit: a request over a request limit, a response that takes the
client's byte total past a byte limit, or more than `ERROR_BURST_THRESHOLD` client's byte total past a byte limit, or more than `ERROR_BURST_THRESHOLD`
requests within a minute refused after a rule file or Core Rule Set match. requests within a minute refused after a rule file or Core Rule Set match, or
for a missing or wrong token.
- The first such ban, or one that comes more than `LIMIT_BAN_REPEAT_WINDOW` - The first such ban, or one that comes more than `LIMIT_BAN_REPEAT_WINDOW`
(default `24h`) after the last such ban ended, lasts `LIMIT_BAN_DURATION` (default `24h`) after the last such ban ended, lasts `LIMIT_BAN_DURATION`
(default `1h`). (default `1h`).
@@ -237,9 +253,10 @@ ban's notes.
Every ban carries notes with what an admin needs to decide whether to lift it Every ban carries notes with what an admin needs to decide whether to lift it
(see "Persistent state"). An admin can add or lift a ban at any time by editing (see "Persistent state"). An admin can add or lift a ban at any time by editing
`bans.json`, or through the admin listener when `ADMIN_TOKEN` is set. A ban `bans.json`, or through the ban endpoints when `ADMIN_TOKEN` is set (see "Admin
lifted before it ends is kept, marked as lifted, and does not count toward a endpoints"). A ban lifted before it ends is kept, marked as lifted, and does not
longer ban; deleting its entry from `bans.json` forgets it entirely. count toward a longer ban; deleting its entry from `bans.json` forgets it
entirely.
Clients of the AS numbers and countries listed in the biased thresholds, and Clients of the AS numbers and countries listed in the biased thresholds, and
clients listed by a reputation source whose action is `limit:<percent>`, get clients listed by a reputation source whose action is `limit:<percent>`, get
@@ -256,15 +273,15 @@ nothing in `bans.json`.
Conventions: lists are comma separated; netblocks are CIDR (a bare address means Conventions: lists are comma separated; netblocks are CIDR (a bare address means
/32 or /128); durations use Go syntax plus `d` for days (`90s`, `15m`, `24h`, /32 or /128); durations use Go syntax plus `d` for days (`90s`, `15m`, `24h`,
`7d`); byte sizes accept `K`, `M`, `G` suffixes. Countries are two-letter ISO `7d`); byte sizes accept `K`, `M`, `G` suffixes, which are powers of 1024: `1K`
is 1024 bytes, `1M` is `1024K` and `1G` is `1024M`. Countries are two-letter ISO
codes in either case (`de` and `DE` are the same); a code that is not a country codes in either case (`de` and `DE` are the same); a code that is not a country
code, such as `nk` (North Korea is `kp`), stops the start with a message naming code, such as `nk` (North Korea is `kp`), stops the start with a message naming
it. it.
- Only `UPSTREAM_URL` is required. Every other setting has a default chosen for - Only `UPSTREAM_URL` is required. Every other setting has a default chosen for
a service facing the internet in 2026, or stays off until the operator a service facing the internet in 2026, or stays off until the operator
supplies or chooses what it needs: an alert destination, an account key, a supplies what it needs: an alert destination, an account key, a token.
lookup source.
- Any limit or threshold can be switched off with the value `off`. - Any limit or threshold can be switched off with the value `off`.
- A list set to an empty value is an empty list, and replaces the default. - A list set to an empty value is an empty list, and replaces the default.
- Every variable may instead be given as `NAME_FILE` pointing at a file holding - Every variable may instead be given as `NAME_FILE` pointing at a file holding
@@ -280,11 +297,12 @@ The settings, by group:
- Core - Core
- `UPSTREAM_URL` (required): the application, for example - `UPSTREAM_URL` (required): the application, for example
`http://gitea:3000`. `http://gitea:3000`.
- `LISTEN_ADDR` (default `:8080`): proxy listener. - `LISTEN_ADDR` (default `:8080`): the one listener, for the requests the
- `ADMIN_LISTEN_ADDR` (default `127.0.0.1:9090`): admin listener. sidecar forwards and for its own endpoints (see "Admin endpoints").
- `ADMIN_TOKEN`: bearer token required for the ban management endpoints. - `ADMIN_TOKEN`: bearer token for the ban endpoints and
Unset by default, which switches those endpoints off; bans are then `/_smallwebwaf/clients/<ip>`, a long random value, since they can be
managed by editing `bans.json`. reached from the internet. Unset by default, which switches them off; bans
are then managed by editing `bans.json`.
- `INSTANCE_NAME` (default: the host name in `UPSTREAM_URL`, for example - `INSTANCE_NAME` (default: the host name in `UPSTREAM_URL`, for example
`gitea`): included in every log line, metric and alert. Set it, for `gitea`): included in every log line, metric and alert. Set it, for
example to `fsn1app1/gitea`, when several sidecars report to one place. example to `fsn1app1/gitea`, when several sidecars report to one place.
@@ -332,14 +350,11 @@ The settings, by group:
- `LOG_REMOTE_FACILITY` (default `local0`), `LOG_REMOTE_APP_NAME` (default - `LOG_REMOTE_FACILITY` (default `local0`), `LOG_REMOTE_APP_NAME` (default
`INSTANCE_NAME`): syslog header fields. `INSTANCE_NAME`): syslog header fields.
- Metrics - Metrics
- `METRICS_ENABLED` (default `true`). - `METRICS_TOKEN`: bearer token a scraper sends for `/_smallwebwaf/metrics`,
- `METRICS_LISTEN_ADDR` (default `:9100`): its own listener, so it can be a long random value. Unset by default, which switches the metrics off,
bound to a monitoring network without exposing the admin endpoints. since they would otherwise be open to anyone on the internet.
- `METRICS_PATH` (default `/metrics`).
- `METRICS_TOP_N` (default `50`): how many AS numbers and countries get - `METRICS_TOP_N` (default `50`): how many AS numbers and countries get
their own series; the rest are summed as `other`. their own series; the rest are summed as `other`.
- `METRICS_TOKEN`: optional bearer token; unset means no authentication,
which is the usual arrangement for a scraper on a private network.
- Static lists - Static lists
- `ALLOW_NETS`: bypass everything (monitoring, the owner's own networks). - `ALLOW_NETS`: bypass everything (monitoring, the owner's own networks).
- `RATE_LIMIT_EXEMPT_NETS`: bypass request and byte limits only; the error - `RATE_LIMIT_EXEMPT_NETS`: bypass request and byte limits only; the error
@@ -355,7 +370,7 @@ The settings, by group:
health checks). health checks).
- Byte limits, per client. A response's bytes are counted when it ends, so every - Byte limits, per client. A response's bytes are counted when it ends, so every
default sits above the largest response allowed (`CLIENT_RESPONSE_MAX_BYTES`, default sits above the largest response allowed (`CLIENT_RESPONSE_MAX_BYTES`,
5 GB) and no single download breaks one. 5 GiB) and no single download breaks one.
- `BYTES_LIMIT_PER_MINUTE` (default `10G`), `BYTES_LIMIT_PER_HOUR` (default - `BYTES_LIMIT_PER_MINUTE` (default `10G`), `BYTES_LIMIT_PER_HOUR` (default
`20G`), `BYTES_LIMIT_PER_DAY` (default `50G`). `20G`), `BYTES_LIMIT_PER_DAY` (default `50G`).
- `BYTES_COUNT` (default `both`): `response`, `request` or `both`. - `BYTES_COUNT` (default `both`): `response`, `request` or `both`.
@@ -397,9 +412,11 @@ The settings, by group:
started it can only be cut off, and the connection is closed. started it can only be cut off, and the connection is closed.
- A WebSocket connection leaves these limits behind once it is upgraded: it - A WebSocket connection leaves these limits behind once it is upgraded: it
stays open until either side closes it. stays open until either side closes it.
- Lookup of AS number and country (R7). Off by default: the file needs an - Lookup of AS number and country (R7). On by default through GeoJS, which needs
account only the operator can hold, and GeoJS is told every visitor's address. no account or file, so that country lookup works with no setup. The file is
- `LOOKUP_SOURCE` (default `off`): `off`, `file` or `geojs`. the alternative for a service that keeps its visitors' addresses on its own
host.
- `LOOKUP_SOURCE` (default `geojs`): `geojs`, `file` or `off`.
- `LOOKUP_DB_PATH`: the file for `file`, the IPinfo Lite database in its - `LOOKUP_DB_PATH`: the file for `file`, the IPinfo Lite database in its
`.mmdb` form (`ipinfo_lite.mmdb`), one file that carries both country and `.mmdb` form (`ipinfo_lite.mmdb`), one file that carries both country and
AS number; the sidecar reads its `asn`, `as_name` and `country_code` AS number; the sidecar reads its `asn`, `as_name` and `country_code`
@@ -418,20 +435,23 @@ The settings, by group:
`country_code`, `asn` and `organization_name` (the AS name). A `country_code`, `asn` and `organization_name` (the AS name). A
`country_code` of `null`, which GeoJS gives for some ranges, and an `asn` `country_code` of `null`, which GeoJS gives for some ranges, and an `asn`
of `64512`, which it gives when it knows none, count as unknown. GeoJS is of `64512`, which it gives when it knows none, count as unknown. GeoJS is
told the address of every new visitor, except private, loopback and told the address of every new visitor, whether or not a setting uses the
link-local addresses, which no source can place and which are never sent. answer, since the request log, the metrics, the client's history and ban
notes carry the AS number and country. Private, loopback and link-local
addresses, which no source can place, are never sent.
- Each GeoJS answer is kept for 7 days in memory and in `lookups.json`, - Each GeoJS answer is kept for 7 days in memory and in `lookups.json`,
apart from the table of clients, so it survives a restart and outlasts a apart from the table of clients, so it survives a restart and outlasts a
client dropped from that table; after 7 days the client's next request client dropped from that table; after 7 days the client's next request
asks again. Up to 100,000 answers are kept, about 15 MiB; when that many asks again. Up to 100,000 answers are kept, about 15 MiB; when that many
are held, the answer used longest ago goes first. are held, the answer used longest ago goes first.
- `LOOKUP_TIMEOUT` (default `1s`): how long a client with no kept answer - `LOOKUP_TIMEOUT` (default `1s`): how long a request waits for its client's
waits for one; GeoJS normally answers in a fraction of that. At most one first answer when a setting needs it (see "Data flow for one request");
request to GeoJS is under way at a time, the addresses that arrive GeoJS normally answers in a fraction of that. At most one request to GeoJS
meanwhile are asked about together in the next one, and a request that is under way at a time, the addresses that arrive meanwhile are asked
takes longer than `LOOKUP_TIMEOUT` is abandoned. A client whose answer has about together in the next one, and a request that takes longer than
not come in time counts as unknown until it comes: its later requests do `LOOKUP_TIMEOUT` is abandoned. A client whose answer has not come in time
not wait, and its address is asked about again in the background. counts as unknown until it comes: its later requests do not wait, and its
address is asked about again in the background.
- GeoJS publishes no rate limit, but its terms forbid "an excessive amount - GeoJS publishes no rate limit, but its terms forbid "an excessive amount
of API requests", judged by GeoJS alone, and let it block a caller. While of API requests", judged by GeoJS alone, and let it block a caller. While
GeoJS is slow, down or refusing the sidecar, clients with a kept answer GeoJS is slow, down or refusing the sidecar, clients with a kept answer
@@ -440,24 +460,25 @@ The settings, by group:
asking with backoff and sends one `source_failure` alert per cooldown. A asking with backoff and sends one `source_failure` alert per cooldown. A
service that cannot accept this uses the file. service that cannot accept this uses the file.
- One source at a time: `LOOKUP_SOURCE=file` without `LOOKUP_DB_PATH`, or - One source at a time: `LOOKUP_SOURCE=file` without `LOOKUP_DB_PATH`, or
`LOOKUP_DB_PATH` with any other `LOOKUP_SOURCE`, stops the start with a `LOOKUP_DB_PATH` with any other `LOOKUP_SOURCE`, the default `geojs`
message naming both. So does a setting that needs lookups while included, stops the start with a message naming both. So does a setting
`LOOKUP_SOURCE` is `off`: the country lists and the biased thresholds that needs lookups while `LOOKUP_SOURCE` is `off`: the country lists and
below, `ADD_LOOKUP_HEADERS`, and the per-AS-number anomaly thresholds. the biased thresholds below, `ADD_LOOKUP_HEADERS`, and the per-AS-number
anomaly thresholds.
- `ADD_LOOKUP_HEADERS` (default `false`): pass `X-Client-ASN` and - `ADD_LOOKUP_HEADERS` (default `false`): pass `X-Client-ASN` and
`X-Client-Country` to the app. `X-Client-Country` to the app.
- Country lists. Both are empty by default, since they need a lookup source. - Country lists. Both are empty by default: the defaults judge a client by what
Clients in `ALLOW_NETS` are not checked. it does, not by where it comes from. Clients in `ALLOW_NETS` are not checked.
- `DENIED_COUNTRIES`: for example `cn,ru,kp,ir,ua,by`. Every request from a - `DENIED_COUNTRIES`: for example `cn,ru,kp,ir,ua,by`. Every request from a
listed country is refused. listed country is refused.
- `EXCLUSIVELY_ALLOWED_COUNTRIES`: for example `us,de`. Every request from - `EXCLUSIVELY_ALLOWED_COUNTRIES`: for example `us,de`. Only clients placed
any other country is refused, and so is every request from a client the in a listed country get through. Every request from any other country is
lookup cannot place: one whose address is missing from the database, one refused, and so is every request from a client the lookup cannot place:
GeoJS has not answered for in time, and any client on a private address, one whose address is missing from the database, one GeoJS has not answered
such as a visitor on the local network, another container or internal for in time, and any client on a private address, such as a visitor on the
monitoring. Those that should reach the app go in `ALLOW_NETS`. A list local network, another container or internal monitoring. Those that should
that let unplaced clients through would let every new client in whenever reach the app go in `ALLOW_NETS`. A list that let unplaced clients through
GeoJS stops answering. would let every new client in whenever GeoJS stops answering.
- Both may be set. `DENIED_COUNTRIES` then adds nothing, since the exclusive - Both may be set. `DENIED_COUNTRIES` then adds nothing, since the exclusive
list already refuses every other country, and a code on both lists stops list already refuses every other country, and a code on both lists stops
the start with a message naming it. the start with a message naming it.
@@ -467,8 +488,9 @@ The settings, by group:
Core Rule Set, is logged with the action `country_denied`, and is counted Core Rule Set, is logged with the action `country_denied`, and is counted
in the metrics. It is a refusal, not a ban: it is not an offence and makes in the metrics. It is a refusal, not a ban: it is not an offence and makes
no ban record. no ban record.
- Biased thresholds (R8). The lists are empty by default: they need a lookup - Biased thresholds (R8). The lists are empty by default, for the same reason as
source, which only the operator can choose. the country lists; the request log and the metrics show which AS numbers and
countries a service's abuse comes from.
- `ASN_LIMIT_PERCENT`: for example `AS14061:50,AS16276:50,AS45102:25`. - `ASN_LIMIT_PERCENT`: for example `AS14061:50,AS16276:50,AS45102:25`.
Clients in a listed AS number get that percentage of every request and Clients in a listed AS number get that percentage of every request and
byte limit, so the rules in "Bans" ban them after fewer requests than byte limit, so the rules in "Bans" ban them after fewer requests than
@@ -491,7 +513,16 @@ The settings, by group:
- `ASN_LIMIT_PERCENT_URL`: optional URL of a text file of `AS:percent` - `ASN_LIMIT_PERCENT_URL`: optional URL of a text file of `AS:percent`
lines, so one abuse-source list can be shared by every sidecar in the lines, so one abuse-source list can be shared by every sidecar in the
fleet. It is fetched and refreshed like the blocklists. fleet. It is fetched and refreshed like the blocklists.
- Attack detection (R9) - Attack detection (R9). By default the Core Rule Set reads the method, the URL
with its query string, and the headers of every request, which is where
automated attacks show: injection in query strings, path traversal, attacks
carried in headers, scanners' user agents. It reads no request body. On a code
forge, the example deployment, bodies carry what people write and publish,
such as issue and comment text, wiki pages, files saved in the web editor and
package descriptions, and when these hold shell commands or code the Core Rule
Set takes them for attacks. The default gives up refusing an attack carried in
a body; the rule files, the limits and the bans still apply to the client that
sends it, and `WAF_BODY_LIMIT` switches body inspection on.
- `RULES_DIR` (default `/etc/smallwebwaf/rules.d`): directory of rule files, - `RULES_DIR` (default `/etc/smallwebwaf/rules.d`): directory of rule files,
read at start and again whenever a file in it changes; format under "Rule read at start and again whenever a file in it changes; format under "Rule
files". files".
@@ -501,55 +532,71 @@ The settings, by group:
- `WAF_PARANOIA_LEVEL` (default `1`), `WAF_ANOMALY_THRESHOLD` (default `5`): - `WAF_PARANOIA_LEVEL` (default `1`), `WAF_ANOMALY_THRESHOLD` (default `5`):
the Core Rule Set's own two tuning values, at the Core Rule Set's own the Core Rule Set's own two tuning values, at the Core Rule Set's own
defaults. defaults.
- The sidecar also changes the Core Rule Set in four ways that no setting - The sidecar also changes the Core Rule Set 4.25.0 in four ways that no
undoes, since in front of gitea each would otherwise refuse ordinary setting undoes, since in front of gitea each would otherwise refuse
requests: ordinary requests:
- PUT, PATCH and DELETE are allowed methods besides GET, HEAD, POST and - PUT, PATCH and DELETE are allowed methods besides GET, HEAD, POST and
OPTIONS; APIs, container image pushes and package uploads use them. OPTIONS; APIs, container image pushes and package uploads use them.
Other methods stay refused. Other methods stay refused.
- The request headers `Content-Encoding` and `Expect` are allowed: git - The request header `Expect` is allowed, since curl and git send it
compresses a fetch request over 1 KiB and says so in before some large uploads. `Content-Encoding` is allowed on a request
`Content-Encoding`, and curl and git send `Expect` before some large whose body the Core Rule Set does not read, which by default is every
uploads. The other headers the Core Rule Set refuses, such as `Proxy`, request: git compresses a fetch request over 1 KiB and says so in that
stay refused. header. On a body the Core Rule Set does read, the header stays
- Only a body the Core Rule Set can read is inspected (see "Data flow refused, since a compressed body cannot be inspected. The other
for one request"). It would read any other body as form data, where headers the Core Rule Set refuses, such as `Proxy`, stay refused.
binary content such as a git push trips rules written for text. - The `redirect_uri` parameter is not checked for a URL naming an IP
address or `localhost` (931100, 934110). Git Credential Manager,
git-credential-oauth and tea, which gitea registers for OAuth sign-in
out of the box, ask to be sent back to `http://127.0.0.1` on the
user's own machine, and the server never fetches that address.
- Responses are not inspected. A raw file from a repository, such as a - Responses are not inspected. A raw file from a repository, such as a
shell script, looks to the response rules like source code leaking shell script, looks to the response rules like source code leaking
from the server. from the server.
- `WAF_DISABLED_RULES` (default - `WAF_DISABLED_RULES` (default
`920340,920420,920440,920640,930130,930140,932180`): rule ids to switch `920340,920420,920440,920640,930130,930140`): rule ids to switch off when
off when an app trips a false positive. The default switches off the rules an app trips a false positive. The default switches off the rules that
that refuse a request for the type of its body, when it is missing or not refuse a request for the type of its body, when it is missing or not on
on the Core Rule Set's short list (920340, 920420, 920640); for its file the Core Rule Set's short list (920340, 920420, 920640); for its file
extension, such as `.sh` or `.sql` (920440); for a file or directory name extension, such as `.sh` or `.sql` (920440); and for a file or directory
in its path, such as `.git/`, `.gitignore`, `Dockerfile`, `package.json` name in its path, such as `.git/`, `.gitignore`, `Dockerfile`,
or an editor's settings directory (930130, 930140); and for the name of an `package.json` or an editor's settings directory (930130, 930140). In
uploaded file, such as `debug.log` or `config.yml` (932180). In front of a front of a code forge these refuse git over HTTP, container image and
code forge these refuse git over HTTP, container image and package package uploads, and views of ordinary files in a repository. At the site
uploads, views of ordinary files in a repository, and logs attached to root, where no app serves such files, the default rule file bans the
issues. The default rule file still bans probes for `.env` and `.git` at common probes these rules caught, such as `/.env`, `/.git/config`,
the site root (see "Rule files"). A list given replaces the default, so `/.aws/credentials`, `/.ssh/id_rsa`, `/.htpasswd` and `/wp-config.php.bak`
include them in it. (see "Rule files"). A list given replaces the default, so include them in
it.
- `WAF_EXEMPT_PATHS`: path prefixes not inspected. - `WAF_EXEMPT_PATHS`: path prefixes not inspected.
- `WAF_BODY_LIMIT` (default `128K`): bodies the Core Rule Set reads are - `WAF_BODY_LIMIT` (default `off`): `off` reads no request body. A size,
inspected up to this size and streamed beyond it without buffering, so such as `128K`, has the Core Rule Set read form data and multipart bodies
large uploads are not held in memory. up to that size, streaming the rest of a longer one on without holding it
in memory, and JSON and XML bodies no larger than it, since those cannot
be read in part. Any other body is not read, since the Core Rule Set would
read it as form data, where binary content such as a git push trips rules
written for text. Body inspection suits apps whose forms carry no code. In
front of gitea it refuses issue and comment text, wiki pages and files
saved in the web editor that hold shell commands or code (932125, 932235,
932250 and others), package descriptions that show code, PyPI uploads
(922130), and attachments named like `debug.log` or `config.yml` (932180),
until the rule ids the request log names are added to
`WAF_DISABLED_RULES`.
- `TRAP_PATHS`: paths the app never serves and only scanners ask for, for - `TRAP_PATHS`: paths the app never serves and only scanners ask for, for
example `/wp-login.php,/xmlrpc.php` in front of gitea. A request for one example `/wp-login.php,/xmlrpc.php` in front of gitea. A request for one
is a clear sign of attack. This is the env-var short form of a `path` rule is a clear sign of attack. This is the env-var short form of a `path` rule
with the `ban` action, for deployments that mount no rule files. with the `ban` action, for deployments that mount no rule files.
- `ERROR_BURST_THRESHOLD` (default `30`): requests per client per minute - `ERROR_BURST_THRESHOLD` (default `30`): requests per client per minute
that the sidecar refused after a rule file or Core Rule Set match; more that the sidecar refused after a rule file or Core Rule Set match, or for
than this breaks a limit (see "Bans"). A client trying one attack after a missing or wrong token; more than this breaks a limit (see "Bans"). A
another is refused many times a minute; a person rarely more than a few client trying one attack or token after another is refused many times a
times. Answers the app gives are not counted, since they do not tell a minute; a person rarely more than a few times. Answers the app gives are
scanner from an ordinary client: in front of gitea, container image and not counted, since they do not tell a scanner from an ordinary client: in
package clients are answered 404 by design, for each layer a push checks front of gitea, container image and package clients are answered 404 by
and each package a lookup asks about, often hundreds of times a minute, design, for each layer a push checks and each package a lookup asks about,
and git and registry clients are answered 401 at the start of every push, often hundreds of times a minute, and git and registry clients are
every fetch from a private repository and every image pull. answered 401 at the start of every push, every fetch from a private
repository and every image pull.
- Bans (R5), following the rules under "Bans" - Bans (R5), following the rules under "Bans"
- `ATTACK_BAN_DURATION` (default `7d`): the ban for a first clear sign of - `ATTACK_BAN_DURATION` (default `7d`): the ban for a first clear sign of
attack. attack.
@@ -565,14 +612,16 @@ The settings, by group:
traefik answers `502`, as it does whenever its backend drops a connection. traefik answers `502`, as it does whenever its backend drops a connection.
- `BAN_SCOPE_V4_PREFIX` (default `32`): widen to for example `24` to ban the - `BAN_SCOPE_V4_PREFIX` (default `32`): widen to for example `24` to ban the
surrounding netblock. surrounding netblock.
- Reputation (R6). No source is on by default. A DNS blocklist would be told the - Reputation (R6). No source is on by default, for the reason the country lists
address of every visitor. AbuseIPDB and CrowdSec need an account or an engine and the biased thresholds are empty: a source refuses the clients that a list
of the operator's own. A downloaded list reveals nothing about visitors, but kept elsewhere names, or lowers their limits, while the defaults judge a
the list most fit to be a default, Spamhaus DROP, must not be downloaded client only by what it does to this service. AbuseIPDB and CrowdSec also need
automatically more than once an hour, and Spamhaus may block an address that an account or an engine of the operator's own. Besides, the list best suited
downloads it too often. Each sidecar fetches its own copy, so on a host that to be on by default, Spamhaus DROP, must not be downloaded automatically more
runs several sidecars, which share one address, downloads would often come than once an hour, and Spamhaus may block an address that downloads it too
less than an hour apart. often. Each sidecar fetches its own copy, so on a host that runs several
sidecars, which share one address, downloads would often come less than an
hour apart.
- `BLOCKLIST_URLS`: text files of addresses and netblocks, one per line; - `BLOCKLIST_URLS`: text files of addresses and netblocks, one per line;
anything after a `;` or `#` on a line is ignored. For example the Spamhaus anything after a `;` or `#` on a line is ignored. For example the Spamhaus
DROP list, `https://www.spamhaus.org/drop/drop.txt`. Spamhaus's terms for DROP list, `https://www.spamhaus.org/drop/drop.txt`. Spamhaus's terms for
@@ -676,8 +725,8 @@ removed there takes effect while the sidecar runs.
so an encoded probe cannot slip past. so an encoded probe cannot slip past.
- `method`, `host`, `user_agent`, `referer`. - `method`, `host`, `user_agent`, `referer`.
- `header:<Name>`: any one request header. - `header:<Name>`: any one request header.
- Request bodies are not available to rule files; body inspection is the - Request bodies are not available to rule files; reading them is the Core
Core Rule Set's job. Rule Set's job, once `WAF_BODY_LIMIT` switches it on.
- `action`: - `action`:
- `log`: note the match in the request log and do nothing else. - `log`: note the match in the request log and do nothing else.
- `block`: refuse the request with 403. The client is not banned for it, but - `block`: refuse the request with 403. The client is not banned for it, but
@@ -712,27 +761,37 @@ removed there takes effect while the sidecar runs.
nothing is refused. nothing is refused.
- Clients in `ALLOW_NETS` are not checked. - Clients in `ALLOW_NETS` are not checked.
Example file: The default file the image ships:
``` ```
# 00-default.rules: probes no real visitor sends # 00-default.rules: probes no real visitor sends, anchored at the site root
# id target action regex # id target action regex
env-file path ban (?i)^/\.env(\.[a-z]+)?$ env-file path ban (?i)^/\.env(\.[a-z]+)?$
git-dir path ban ^/\.git/(config|HEAD|index)$ vcs-dir path ban (?i)^/\.(git|svn|hg|bzr)(/|$)
secrets-dir path ban (?i)^/\.(aws|ssh|docker|kube)/
secret-file path ban (?i)^/\.(htpasswd|htaccess|npmrc|netrc|pgpass|git-credentials|bash_history|DS_Store)$
editor-dir path ban (?i)^/\.(vscode|idea)/
backup-file path ban (?i)^/[^/]+\.(php(\.[a-z0-9]+|~)|sql(\.[a-z0-9]+)?)$
log-file path ban (?i)^/(debug|error|access)\.log$
compose-file path ban (?i)^/(docker-)?compose\.ya?ml$
php-shell path ban (?i)^/(shell|c99|r57|wso|alfa)\.php$ php-shell path ban (?i)^/(shell|c99|r57|wso|alfa)\.php$
scanner-agent user_agent ban (?i)\b(sqlmap|nikto|nuclei|masscan|zgrab|wpscan)\b scanner-agent user_agent ban (?i)\b(sqlmap|nikto|nuclei|masscan|zgrab|wpscan)\b
path-traversal uri block (\.\./){2,} path-traversal uri block (\.\./){2,}
empty-agent user_agent log ^$ empty-agent user_agent log ^$
``` ```
The image ships one default file of this kind. It is kept short and limited to It is kept short and limited to patterns that are wrong for every app, and its
patterns that are wrong for every app, and its path rules are anchored at the path rules are anchored at the site root: in front of gitea, `/.env` is a probe,
site root: in front of gitea, `/.env` is a probe, while while `/<owner>/<repo>/src/branch/main/.env.example` is a file in a repository
`/<owner>/<repo>/src/branch/main/.env.example` is a file in a repository that that any visitor or search crawler may open. Its path rules ban the common
any visitor or search crawler may open. Anything app-specific belongs in a file probes for secrets, version control directories, backups and logs. The Core Rule
the deployer mounts: a request for `/wp-login.php`, for example, is a clear sign Set's rules for file names and extensions (930130, 920440) refuse such names
of attack in front of gitea and an ordinary login in front of WordPress. anywhere in a path, which is why the default `WAF_DISABLED_RULES` switches them
off: a code forge serves such files deeper in its paths. Anything app-specific
belongs in a file the deployer mounts: a request for `/wp-login.php`, for
example, is a clear sign of attack in front of gitea and an ordinary login in
front of WordPress.
``` ```
# 50-gitea.rules: WordPress probes, which gitea never serves # 50-gitea.rules: WordPress probes, which gitea never serves
@@ -760,7 +819,7 @@ and the running sidecar takes the edit in.
each direction, how many requests were forwarded and how many refused, each direction, how many requests were forwarded and how many refused,
responses by status class, and offences by kind. Clients that were never responses by status class, and offences by kind. Clients that were never
banned are kept too, so every client's history survives a restart; banned are kept too, so every client's history survives a restart;
`GET /clients/<ip>` shows it. `GET /_smallwebwaf/clients/<ip>` shows it.
- `lookups.json`: GeoJS answers, one per client: the AS number, AS name and - `lookups.json`: GeoJS answers, one per client: the AS number, AS name and
country, when GeoJS was asked and when the answer was last used; up to country, when GeoJS was asked and when the answer was last used; up to
100,000, each for 7 days. 100,000, each for 7 days.
@@ -776,7 +835,8 @@ and the running sidecar takes the edit in.
- Ban notes. The notes on a ban hold what an admin needs to decide whether to - Ban notes. The notes on a ban hold what an admin needs to decide whether to
lift it, drawn from what the sidecar already knows; nothing is looked up to lift it, drawn from what the sidecar already knows; nothing is looked up to
fill them: fill them:
- the AS number, AS name and country (empty when lookups are off); - the AS number, AS name and country, added when the lookup answers (empty
when lookups are off);
- what was broken: the rule ids and target that matched, or the limit, its - what was broken: the rule ids and target that matched, or the limit, its
window, the count reached and the client's limit percentage with what set window, the count reached and the client's limit percentage with what set
it; and any reputation sources that listed the client; it; and any reputation sources that listed the client;
@@ -881,7 +941,8 @@ handling while `docker logs` keeps working.
redirects, `aborted` when the client went away early. redirects, `aborted` when the client went away early.
- Decision: `action` (`forward`, `banned`, `denied`, `country_denied`, - Decision: `action` (`forward`, `banned`, `denied`, `country_denied`,
`rate_limited`, `rule_blocked`, `waf_blocked`, `too_large`, `timed_out`, `rate_limited`, `rule_blocked`, `waf_blocked`, `too_large`, `timed_out`,
`upstream_error`), `would_action` in `observe` mode, `limit_percent` and which `upstream_error`, and `admin` for a request the sidecar answered at one of its
own endpoints), `would_action` in `observe` mode, `limit_percent` and which
rule set it, `counts` (the client's minute, hour and day request and byte rule set it, `counts` (the client's minute, hour and day request and byte
totals after this request), `limit_hit` (which window), `rule_ids` (rule file totals after this request), `limit_hit` (which window), `rule_ids` (rule file
rules that matched), `waf_rule_ids`, `waf_score`, `reputation` (sources that rules that matched), `waf_rule_ids`, `waf_score`, `reputation` (sources that
@@ -904,9 +965,10 @@ handling while `docker logs` keeps working.
## Metrics endpoint ## Metrics endpoint
Prometheus text format on `METRICS_LISTEN_ADDR` at `METRICS_PATH`, on by Prometheus text format at `/_smallwebwaf/metrics`, for a request carrying
default, switched off with `METRICS_ENABLED=false`. It has its own listener so `METRICS_TOKEN` (see "Admin endpoints"). While the token is unset, the metrics
it can be reached by a scraper without exposing ban management. are off. The token is separate from `ADMIN_TOKEN`, so a scraper that holds it
cannot manage bans.
- Traffic: requests and bytes in and out, by status class and `action`; request - Traffic: requests and bytes in and out, by status class and `action`; request
duration and upstream duration histograms; requests in flight. duration and upstream duration histograms; requests in flight.
@@ -928,17 +990,40 @@ it can be reached by a scraper without exposing ban management.
by destination; remote log lines sent, dropped and buffer depth; the standard by destination; remote log lines sent, dropped and buffer depth; the standard
Go runtime and process metrics. Go runtime and process metrics.
- No metric carries a client IP address as a label; per-address questions are - No metric carries a client IP address as a label; per-address questions are
answered by the request log and `GET /clients/<ip>`. answered by the request log and `GET /_smallwebwaf/clients/<ip>`.
## Admin listener ## Admin endpoints
- `GET /healthz`: for the container health check. The sidecar has one listener. A request whose path starts with `/_smallwebwaf/`
- `GET /bans`, `POST /bans` (client or netblock, duration, reason), is for the sidecar itself: it answers it and never passes it to the app. The
`DELETE /bans/<client>`: need `ADMIN_TOKEN`, and are switched off while it is prefix carries the tool's name, so it takes no path an app uses. Admins and
unset. Editing `bans.json` does the same without a token. scrapers reach these endpoints through traefik, like any other request.
- `GET /clients/<ip>`: current counters, history, lookup result, reputation,
offences, and bans with their notes; for answering "why was this address - `GET /_smallwebwaf/healthz`: answers `200` with `ok` to anyone, without a
refused" and "what has this netblock been doing". token, while the sidecar is running; it does not ask the app. It is answered
before any check, so a health checker never needs an exemption and is never
refused, for example by `EXCLUSIVELY_ALLOWED_COUNTRIES`.
- `GET /_smallwebwaf/metrics`: the metrics (see "Metrics endpoint"); needs
`METRICS_TOKEN`.
- `GET /_smallwebwaf/bans`, `POST /_smallwebwaf/bans` (client or netblock,
duration, reason), `DELETE /_smallwebwaf/bans/<client>`: need `ADMIN_TOKEN`.
Editing `bans.json` does the same without a token.
- `GET /_smallwebwaf/clients/<ip>`: needs `ADMIN_TOKEN`. Current counters,
history, lookup result, reputation, offences, and bans with their notes; for
answering "why was this address refused" and "what has this netblock been
doing".
- A token is sent as `Authorization: Bearer <token>`. While a token is unset,
the endpoints that need it answer `404`, as does any other path under the
prefix.
- Apart from the health check, these requests go through every check that any
other request goes through, and are answered at the point where another
request would be forwarded to the app: a banned or refused client stays
refused, and each request counts toward the client's limits. A missing or
wrong token is answered `401`, in `MODE=observe` too, and counts toward the
error burst, so a client guessing tokens is banned once it passes
`ERROR_BURST_THRESHOLD` guesses in a minute. A client in `ALLOW_NETS` skips
the checks but still needs the token.
- An admin whose own address is banned lifts the ban by editing `bans.json`.
## Alert webhook schema ## Alert webhook schema
@@ -995,9 +1080,10 @@ networks:
hostname, for example over a docker network the two share. hostname, for example over a docker network the two share.
- The application container leaves the traefik network, so the sidecar cannot be - The application container leaves the traefik network, so the sidecar cannot be
bypassed. bypassed.
- Traefik routes only port 8080. The metrics port is reached by the scraper over - Traefik routes port 8080, the sidecar's one listener. The sidecar's own
a docker network it shares with the sidecar; the admin port stays on loopback endpoints are reached through traefik like any other request, for example
inside the container and is used through `docker exec`. `https://git.example.invalid/_smallwebwaf/metrics` for a scraper that sends
`METRICS_TOKEN` (see "Admin endpoints").
- The state volume holds the JSON state files, a few tens of MiB at most with - The state volume holds the JSON state files, a few tens of MiB at most with
the defaults (see "Persistent state"); it needs no backup beyond whatever the the defaults (see "Persistent state"); it needs no backup beyond whatever the
host already does. host already does.
@@ -1009,27 +1095,35 @@ networks:
refused. A real visitor refused by mistake is let through with an exclusion refused. A real visitor refused by mistake is let through with an exclusion
(`WAF_DISABLED_RULES`, `WAF_EXEMPT_PATHS`, `RATE_LIMIT_EXEMPT_PATHS`) or an (`WAF_DISABLED_RULES`, `WAF_EXEMPT_PATHS`, `RATE_LIMIT_EXEMPT_PATHS`) or an
exemption (`RATE_LIMIT_EXEMPT_NETS`, `ALLOW_NETS`), and its ban is lifted in exemption (`RATE_LIMIT_EXEMPT_NETS`, `ALLOW_NETS`), and its ban is lifted in
`bans.json`. Additions to consider at any time: an alert destination, a lookup `bans.json`. Additions to consider at any time: an alert destination, country
source with country lists or biased thresholds, reputation sources, a remote lists or biased thresholds, reputation sources, the metrics and admin tokens,
log endpoint, and an app-specific rule file. a remote log endpoint, and an app-specific rule file.
- Notes specific to gitea: - Notes specific to gitea:
- A clone is a response, and fits the defaults of 30 minutes and 5 GB for - A clone is a response, and fits the defaults of 30 minutes and 5 GiB for
all but the largest repositories and slowest links. A push is a request: all but the largest repositories and slowest links. A push is a request:
at the defaults, one larger than 100 MB or taking more than 60 seconds to at the defaults, one larger than 100 MiB or taking more than 60 seconds to
upload is cut off, so a gitea that takes large pushes needs upload is cut off, so a gitea that takes large pushes needs
`CLIENT_REQUEST_MAX_BYTES`, `UPSTREAM_REQUEST_MAX_BYTES`, `CLIENT_REQUEST_MAX_BYTES`, `UPSTREAM_REQUEST_MAX_BYTES`,
`CLIENT_REQUEST_TIMEOUT` and `UPSTREAM_REQUEST_TIMEOUT` raised to fit. The `CLIENT_REQUEST_TIMEOUT` and `UPSTREAM_REQUEST_TIMEOUT` raised to fit. The
Core Rule Set does not read a pack upload, which streams through without Core Rule Set does not read a pack upload, which streams through without
being held in memory. being held in memory.
- With the sidecar's changes to the Core Rule Set and the default - At the defaults (see "Configuration surface", attack detection), the Core
`WAF_DISABLED_RULES` (see "Configuration surface", attack detection), Rule Set lets gitea's ordinary use through: browsing and views of files in
git's clone, fetch and push over HTTP, the API, container images, a repository, git's clone, fetch and push over HTTP, signing in, including
packages, and views of ordinary files pass the Core Rule Set. Requests with Git Credential Manager, git-credential-oauth or tea, the API, pushing
that carry text people write, such as issues, pull requests, searches, the and pulling container images and packages, and posting issues, pull
web editor and uploads, can still trip other rules, and so can a file requests, comments, wiki pages and files saved in the web editor, code
whose name the Core Rule Set takes for an attack, such as an `.xhtml` included, since no body is read. It can still refuse a request whose URL
page. Such a match refuses only that request, and the request log names reads to it as an attack: a search, or another query string, holding a
the rule in `waf_rule_ids` for `WAF_DISABLED_RULES`. shell command with its options or a system path (`ls -la`, `sed -i`,
`/bin/sh`), a command in backticks, script code (`fetch(`, `${VAR}`), HTML
(`<img src=`), an SQL statement (`SELECT * FROM users WHERE`), or a URL
naming an IP address or `localhost`; or a path such as a file name ending
in `~`, or an `.xhtml` file whose path holds a space. Such a refusal
answers only that request, with 403, and bans no one by itself: it counts
toward the error burst, which a person searching does not reach. The
request log names the rule in `waf_rule_ids`, which `WAF_DISABLED_RULES`
can switch off.
- Archive download and blame or history pages are what scrapers hammer; - Archive download and blame or history pages are what scrapers hammer;
request limits do most of the work there. request limits do most of the work there.
@@ -1081,8 +1175,9 @@ networks:
- A `ban` rule that matches a real visitor bans it for seven days, and the - A `ban` rule that matches a real visitor bans it for seven days, and the
visitor's next request during those days makes the ban permanent. So the visitor's next request during those days makes the ban permanent. So the
default rule file keeps to requests no real visitor sends, with its paths default rule file keeps to requests no real visitor sends, with its paths
anchored at the site root, where no app serves `.env` files or web shells. A anchored at the site root, where no app serves secrets, version control
visitor banned by mistake is let back in by lifting the ban in `bans.json`. directories, backups or web shells. A visitor banned by mistake is let back in
by lifting the ban in `bans.json`.
- IPv6 address rotation inside a /64: handled by grouping. - IPv6 address rotation inside a /64: handled by grouping.
- Widely distributed scrapers using thousands of addresses at low rates each: - Widely distributed scrapers using thousands of addresses at low rates each:
per-client limits do not see them. The AS number and netblock anomaly alerts per-client limits do not see them. The AS number and netblock anomaly alerts
@@ -1090,14 +1185,24 @@ networks:
that AS number is the response: it lowers the limits of each of its clients. that AS number is the response: it lowers the limits of each of its clients.
There is no shared budget for a whole AS number or country, which one abuser There is no shared budget for a whole AS number or country, which one abuser
could use up and so lock out everyone else there. could use up and so lock out everyone else there.
- Core Rule Set false positives against real apps (gitea's editor, API - Core Rule Set false positives against real apps (a search for code, and any
payloads): a match refuses only that request and bans no one by itself; the body once `WAF_BODY_LIMIT` is set): a match refuses only that request and bans
request log names the rule, and exclusions by rule id and path fix it. no one by itself; the request log names the rule, and exclusions by rule id
- GeoJS as the lookup source: every new visitor's address goes to a third party, and path fix it.
and a swarm of fresh addresses, when lookups peak, is when GeoJS may slow down - Attacks carried in request bodies: not refused by default, since on a code
or block the sidecar. Keeping answers for 7 days and asking about many forge bodies are full of code the Core Rule Set takes for attacks. What shows
addresses in one request keep the number of requests low; the file source has in URLs and headers is still refused, the client that sends an attack is still
neither risk. subject to the rule files, the limits and the bans, and `WAF_BODY_LIMIT`
switches body inspection on for apps whose forms carry no code.
- The admin endpoints can be reached from the internet: all but the health check
need a token and are off while it is unset, and a missing or wrong token
counts toward the error burst, so a client guessing tokens is soon banned.
Tokens are meant to be long random values.
- GeoJS, the default lookup source: every new visitor's address goes to a third
party, and a swarm of fresh addresses, when lookups peak, is when GeoJS may
slow down or block the sidecar. Keeping answers for 7 days and asking about
many addresses in one request keep the number of requests low; the file source
has neither risk.
- Slow-request attacks: `CLIENT_REQUEST_TIMEOUT` bounds how long a request may - Slow-request attacks: `CLIENT_REQUEST_TIMEOUT` bounds how long a request may
take to arrive, `CLIENT_REQUEST_HEADER_MAX_BYTES` how large its headers may take to arrive, `CLIENT_REQUEST_HEADER_MAX_BYTES` how large its headers may
be, and `CLIENT_IDLE_TIMEOUT` closes a kept-open connection that sends nothing be, and `CLIENT_IDLE_TIMEOUT` closes a kept-open connection that sends nothing