sneak/smallwebwaf
1
0 Fork 0
Code Issues 25 Pull Requests 7 Actions 82 Packages Projects Releases Wiki Activity

Rename discriminator to waf everywhere (closes #312)
check / check (push) Successful in 5m58s
Details

Browse Source 
The command is now `sww waf`, and the package is `internal/waf`. Types,
functions, files, tests, comments, docs, the Dockerfile, the runit
script, `script/`, compose examples, the log field `waf_line` and the
`X-From-Waf` test header follow. No behaviour changes, and no setting
changes name: rproxy already finds the waf through `SWRP_WAF_BACKEND`.

Done with one `sed -i` (Discriminator and DISCRIMINATOR to WAF,
discriminator to waf) over the files `git grep -i discriminator`
listed, as sneak allowed for this rename, then `git mv` for the
package and files. By hand afterwards: the header's spelling, which
the linter requires, and the padding after `waf` in `sww`'s usage
text.

Model: opus-5-5
This commit was merged in pull request #319.
This commit is contained in:
clawbot committed 2026-11-11 02:43:16 +01:00
1 parent ad36dd1775
commit f7e3190654
89 files changed
+1454 -1489
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

README.md
+106 -109
View File
@@ -11,16 +11,15 @@ refuses the ones that look like abuse. `smallwebwaf` is one program, `sww`,
shipped in one image, with two commands that do this together:
- `sww rproxy` is a small reverse proxy in front of one app. traefik routes to
it. It asks `discriminator` about each request, passes the allowed ones to the
app, and passes the app's answers back. It keeps nothing and never touches
disk.
- `sww discriminator` decides. It runs the checks and keeps the bans, the
counters and each client's history. One `discriminator` serves many sites: a
client banned through one site is refused on all of them, and its requests on
all of them count toward one set of limits.
it. It asks `waf` about each request, passes the allowed ones to the app, and
passes the app's answers back. It keeps nothing and never touches disk.
- `sww waf` decides. It runs the checks and keeps the bans, the counters and
each client's history. One `waf` serves many sites: a client banned through
one site is refused on all of them, and its requests on all of them count
toward one set of limits.
The site keeps answering while `discriminator` is down, slow or restarting:
without an answer in time, `rproxy` passes the request to the app.
The site keeps answering while `waf` is down, slow or restarting: without an
answer in time, `rproxy` passes the request to the app.
It protects against:
@@ -80,11 +79,11 @@ make docker
Push the image to a registry your hosts pull from, and name it there by digest.
The image runs `rproxy` by default. To run `discriminator` from the same image,
set the container's `SWW_COMMAND` to `discriminator`. The image's health check
follows the command: in `rproxy`'s container it passes once `rproxy` answers and
the app accepts connections, whether or not `discriminator` answers; in
`discriminator`'s container, once `discriminator` answers.
The image runs `rproxy` by default. To run `waf` from the same image, set the
container's `SWW_COMMAND` to `waf`. The image's health check follows the
command: in `rproxy`'s container it passes once `rproxy` answers and the app
accepts connections, whether or not `waf` answers; in `waf`'s container, once
`waf` answers.
### The usual setup: three containers
@@ -92,24 +91,24 @@ The usual setup is three containers on one private network:
- the app;
- `rproxy`, in front of the app, which traefik routes to: `SWRP_WAF_BACKEND`
names `discriminator` and `SWRP_UPSTREAM_URL` names the app;
- `discriminator`, with `SWW_COMMAND=discriminator` and a volume at
`/var/lib/sww` for its state files.
names `waf` and `SWRP_UPSTREAM_URL` names the app;
- `waf`, with `SWW_COMMAND=waf` and a volume at `/var/lib/sww` for its state
files.
Point traefik's labels at port 8080 of `rproxy`
(`traefik.http.services.<name>.loadbalancer.server.port=8080`). Publish no port
of the app or of `discriminator`. Anything on that network can reach the app
around `rproxy`, and can ask `discriminator` questions, so the network holds
only what you trust. Each further app adds its own container and its own
`rproxy`, whose `SWRP_WAF_BACKEND` names the same `discriminator`. The "Smallest
setup" example below shows this shape.
of the app or of `waf`. Anything on that network can reach the app around
`rproxy`, and can ask `waf` questions, so the network holds only what you trust.
Each further app adds its own container and its own `rproxy`, whose
`SWRP_WAF_BACKEND` names the same `waf`. The "Smallest setup" example below
shows this shape.
### `rproxy` in the app's container
An app's image can instead be built on the `smallwebwaf` image, so that `rproxy`
runs beside the app in one container. `discriminator` still runs in a container
of its own. The app's `Dockerfile` starts `FROM` the `smallwebwaf` image, named
by digest, and adds:
runs beside the app in one container. `waf` still runs in a container of its
own. The app's `Dockerfile` starts `FROM` the `smallwebwaf` image, named by
digest, and adds:
- the app's binary;
- the packages the app needs, from the nixpkgs in the image;
@@ -169,26 +168,25 @@ does.
- A log line with `"msg":"starting"` gives the version, the address and every
setting in effect, with secrets masked.
- A request to the site through traefik gets the app's answer.
`https://<site>/_sww/healthz` answers `ok`. `discriminator`'s log lines about
the request have `"action":"forward"`.
- To see a refusal without banning yourself, run `discriminator` with
`SWW_MODE=observe` and ask for `/.env`. The request reaches the app.
`discriminator`'s log lines have `"would_action":"temporarily_banned"` and
`"rule_ids":["env-file"]`. A ban you get in `enforce` mode is lifted as
"Clients and bans" below describes.
`https://<site>/_sww/healthz` answers `ok`. `waf`'s log lines about the
request have `"action":"forward"`.
- To see a refusal without banning yourself, run `waf` with `SWW_MODE=observe`
and ask for `/.env`. The request reaches the app. `waf`'s log lines have
`"would_action":"temporarily_banned"` and `"rule_ids":["env-file"]`. A ban you
get in `enforce` mode is lifted as "Clients and bans" below describes.
- With `SWW_METRICS_TOKEN` set, this answers with the metrics, from a host on
`discriminator`'s network:
`curl -H "Authorization: Bearer $TOKEN" http://discriminator:8080/_sww/metrics`.
`waf`'s network:
`curl -H "Authorization: Bearer $TOKEN" http://waf:8080/_sww/metrics`.
- `make example-app` builds the image and an app on it, runs them, and checks
the whole exchange (see "Entrypoints").
## Configuration
Every setting is an environment variable, read once at start; changing one means
restarting the container. `rproxy` reads only the `SWRP_` settings, and
`discriminator` only the `SWW_` ones. Every setting but `SWRP_WAF_BACKEND` has a
default, chosen for a service on the open internet. A setting that needs an
account, a key, a token or an alert destination stays off until you give one.
restarting the container. `rproxy` reads only the `SWRP_` settings, and `waf`
only the `SWW_` ones. Every setting but `SWRP_WAF_BACKEND` has a default, chosen
for a service on the open internet. A setting that needs an account, a key, a
token or an alert destination stays off until you give one.
- Any `SWW_` setting can be given as a file instead, for secrets and long lists:
`SWW_ADMIN_TOKEN_FILE` names the file that holds `SWW_ADMIN_TOKEN`. The file's
@@ -222,23 +220,23 @@ Every setting the code reads, with its default:
| Variable | Default | What it does |
| ---------------------------------------- | ---------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `SWW_COMMAND` | `rproxy` | Which command the image runs, and its health check checks: `rproxy` or `discriminator`. |
| `SWRP_WAF_BACKEND` | required | `discriminator`'s URL, such as `http://discriminator:8080`. |
| `SWW_COMMAND` | `rproxy` | Which command the image runs, and its health check checks: `rproxy` or `waf`. |
| `SWRP_WAF_BACKEND` | required | `waf`'s URL, such as `http://waf:8080`. |
| `SWRP_UPSTREAM_URL` | `http://127.0.0.1:8081` | The app's URL: a scheme, a host and an optional port, nothing more. |
| `SWRP_LISTEN_ADDR` | `:8080` | Where `rproxy` listens; leave the host part empty. |
| `SWRP_TRUSTED_PROXIES` | `10.0.0.0/8,172.16.0.0/12,192.168.0.0/16` | The netblocks whose `X-Forwarded-For` `rproxy` believes; set but empty, it believes none. |
| `SWRP_TIMEOUT_MS` | `2000` | How long `rproxy` waits for `discriminator`'s answer to a question and to a report, in milliseconds. |
| `SWRP_DEFAULT_DISPOSITION` | `allow` | What `rproxy` does with a request when `discriminator` cannot be reached or does not answer in time: `allow` passes it to the app, `deny` answers `503`. |
| `SWRP_TIMEOUT_MS` | `2000` | How long `rproxy` waits for `waf`'s answer to a question and to a report, in milliseconds. |
| `SWRP_DEFAULT_DISPOSITION` | `allow` | What `rproxy` does with a request when `waf` cannot be reached or does not answer in time: `allow` passes it to the app, `deny` answers `503`. |
| `SWRP_MAX_REQUEST_SIZE` | `100M` | The largest request body, a size or `off`; a larger one is answered `413`. |
| `SWRP_MAX_RESPONSE_SIZE` | `1G` | The largest response body, a size or `off`. The visitor gets `502` in place of a larger one, or it is cut off if it has already started. |
| `SWRP_MAX_REQUEST_TIME_MS` | `60000` | Two time limits, each this long, in milliseconds or `off`: one for the visitor to send its request line and headers, then one for the whole request to reach the app. |
| `SWRP_MAX_RESPONSE_TIME_MS` | `1800000` | How long the answer may take, from the end of the request to its last byte reaching the visitor, in milliseconds or `off`. |
| `SWW_LISTEN_ADDR` | `:8080` | Where `discriminator` listens; leave the host part empty. |
| `SWW_INSTANCE_NAME` | the container's host name | The name in every line `discriminator` logs, and in every alert and metric, such as `fsn1/discriminator`. |
| `SWW_LISTEN_ADDR` | `:8080` | Where `waf` listens; leave the host part empty. |
| `SWW_INSTANCE_NAME` | the container's host name | The name in every line `waf` logs, and in every alert and metric, such as `fsn1/waf`. |
| `SWW_MODE` | `enforce` | `observe` refuses nothing and makes no ban; each log line says what `enforce` would have done. |
| `SWW_IPV6_GROUP_PREFIX` | `64` | The prefix length of the IPv6 netblock that counts as one client, from 32 to 128. |
| `SWW_MAX_TRACKED_CLIENTS` | `20000` | The most clients kept, in memory and in `clients.json`; the least recently seen is dropped first. |
| `SWW_MAX_BANS` | `5000` | The most bans `discriminator` keeps of its own; bans an admin made do not count. |
| `SWW_MAX_BANS` | `5000` | The most bans `waf` keeps of its own; bans an admin made do not count. |
| `SWW_ALLOW_NETS` | empty | Netblocks whose clients skip every check, such as your monitoring. |
| `SWW_RATE_LIMIT_EXEMPT_NETS` | empty | Netblocks whose clients are exempt from rate limiting and the byte limits. |
| `SWW_DENY_NETS` | empty | Netblocks whose clients are refused. |
@@ -296,9 +294,9 @@ Every setting the code reads, with its default:
| `SWW_STATE_DIR` | `/var/lib/sww` | The directory of the state files, an absolute path. |
| `SWW_STATE_WRITE_DELAY` | `10s` | How long `bans.json` waits to be written after a ban is made, lifted or made permanent. |
| `SWW_STATE_COUNTER_INTERVAL` | `15m` | How often every state file is written. |
| `SWW_LOG_LEVEL` | `info` | The least severe of `discriminator`'s own messages that are written: `debug`, `info`, `warn` or `error`; request lines are always written. |
| `SWW_LOG_LEVEL` | `info` | The least severe of `waf`'s own messages that are written: `debug`, `info`, `warn` or `error`; request lines are always written. |
| `SWW_LOG_REQUEST_HEADERS` | `accept,accept-language,accept-encoding,content-type,origin,range` | The request headers whose values the request log records; `Authorization` and `Cookie` are never recorded. |
| `SWW_LOG_REMOTE_URL` | unset | A syslog server that also gets every line `discriminator` writes: `syslog+udp://`, `syslog+tcp://` or `syslog+tls://` with a host and a port. |
| `SWW_LOG_REMOTE_URL` | unset | A syslog server that also gets every line `waf` writes: `syslog+udp://`, `syslog+tcp://` or `syslog+tls://` with a host and a port. |
| `SWW_LOG_REMOTE_TLS_CA_FILE` | unset | A file of PEM certificates a `syslog+tls` server's certificate must chain to, instead of the host's own. |
| `SWW_LOG_REMOTE_BUFFER` | `10000` | The most lines held while they wait to be sent; past it, the oldest is dropped. |
| `SWW_LOG_REMOTE_FACILITY` | `local0` | The syslog facility of each record sent. |
@@ -351,8 +349,8 @@ own, where each reaches the others by service name. Each example publishes
### Smallest setup
One app, `rproxy` in front of it, and `discriminator` with a volume for its
state files, every setting at its default.
One app, `rproxy` in front of it, and `waf` with a volume for its state files,
every setting at its default.
```yaml
services:
@@ -363,20 +361,20 @@ services:
rproxy:
image: smallwebwaf
environment:
SWRP_WAF_BACKEND: http://discriminator:8080
SWRP_WAF_BACKEND: http://waf:8080
SWRP_UPSTREAM_URL: http://app:8081
ports:
- "8080:8080"
discriminator:
waf:
image: smallwebwaf
environment:
SWW_COMMAND: discriminator
SWW_COMMAND: waf
volumes:
- discriminator-state:/var/lib/sww
- waf-state:/var/lib/sww
volumes:
discriminator-state:
waf-state:
```
### Custom per-address limits
@@ -399,15 +397,15 @@ services:
rproxy:
image: smallwebwaf
environment:
SWRP_WAF_BACKEND: http://discriminator:8080
SWRP_WAF_BACKEND: http://waf:8080
SWRP_UPSTREAM_URL: http://app:8081
ports:
- "8080:8080"
discriminator:
waf:
image: smallwebwaf
environment:
SWW_COMMAND: discriminator
SWW_COMMAND: waf
# Monitoring skips every check.
SWW_ALLOW_NETS: 192.0.2.10
# The office and the automation host are never rate limited.
@@ -426,18 +424,18 @@ services:
SWW_BAN_RESPONSE: "429"
SWW_LIMIT_BAN_DURATION: 30m
volumes:
- discriminator-state:/var/lib/sww
- waf-state:/var/lib/sww
volumes:
discriminator-state:
waf-state:
```
### `rproxy` beside the app
The app is built on the `smallwebwaf` image, with `rproxy` inside its container.
`app/` holds the `Dockerfile` and `app.run` from "`rproxy` in the app's
container". `rproxy` answers `503` to every request while `discriminator` is
down, and `discriminator`:
container". `rproxy` answers `503` to every request while `waf` is down, and
`waf`:
- observes before it enforces;
- looks addresses up in a local database rather than GeoJS;
@@ -450,17 +448,17 @@ services:
# A Dockerfile built FROM the smallwebwaf image, with rproxy inside.
build: ./app
environment:
SWRP_WAF_BACKEND: http://discriminator:8080
# Answer 503 rather than pass a request on while discriminator is down.
SWRP_WAF_BACKEND: http://waf:8080
# Answer 503 rather than pass a request on while waf is down.
SWRP_DEFAULT_DISPOSITION: deny
ports:
- "8080:8080"
discriminator:
waf:
image: smallwebwaf
environment:
SWW_COMMAND: discriminator
SWW_INSTANCE_NAME: fsn1/discriminator
SWW_COMMAND: waf
SWW_INSTANCE_NAME: fsn1/waf
# Refuse nothing yet; log what enforce mode would have done.
SWW_MODE: observe
# Look addresses up in a local IPinfo Lite database, never GeoJS.
@@ -471,12 +469,12 @@ services:
SWW_ALERT_NTFY_TOKEN_FILE: /etc/sww/tokens/ntfy
SWW_ALERT_EVENTS: ban,permanent_ban,core_rule_set_match,source_failure,file_error
volumes:
- discriminator-state:/var/lib/sww
- waf-state:/var/lib/sww
- /srv/ipinfo:/var/lib/ipinfo:ro
- /srv/sww/tokens:/etc/sww/tokens:ro
volumes:
discriminator-state:
waf-state:
```
The database is mounted by its directory, not by file, so that a replaced
@@ -498,18 +496,18 @@ How it works inside. [`SPEC.md`](SPEC.md) gives every rule in full, and why.
### What happens to a request
A request goes through these steps, in this order. Once `discriminator` refuses
a request, the steps after that are skipped, apart from the report, and nothing
of the request reaches the app.
A request goes through these steps, in this order. Once `waf` refuses a request,
the steps after that are skipped, apart from the report, and nothing of the
request reaches the app.
`rproxy` takes the first two:
1. The health check, `GET /_sww/healthz`, is answered at once.
2. The visitor's address is worked out from `X-Forwarded-For`, believed only
from `SWRP_TRUSTED_PROXIES`. `rproxy` sends `discriminator` the request as
the visitor sent it, with the first 64 KiB of its body.
from `SWRP_TRUSTED_PROXIES`. `rproxy` sends `waf` the request as the visitor
sent it, with the first 64 KiB of its body.
`discriminator` takes the checks:
`waf` takes the checks:
3. The tag lists.
4. The static lists, `SWW_ALLOW_NETS` and `SWW_DENY_NETS`.
@@ -528,13 +526,13 @@ of the request reaches the app.
16. A path under `/_sww/` is answered `404`, so that the app never sees that
prefix.
`rproxy` takes the rest, with `discriminator` counting at the end:
`rproxy` takes the rest, with `waf` counting at the end:
17. The request is passed to the app, streaming.
18. After the answer, `rproxy` sends `discriminator` a report with the status
and the body bytes each way. For an allowed request, `discriminator` counts
the bytes `SWW_BYTES_COUNT` names toward the byte limits, bans the client if
they take it past one, and adds the request to the client's history.
18. After the answer, `rproxy` sends `waf` a report with the status and the body
bytes each way. For an allowed request, `waf` counts the bytes
`SWW_BYTES_COUNT` names toward the byte limits, bans the client if they take
it past one, and adds the request to the client's history.
A refused visitor is answered with `SWW_BAN_RESPONSE`. A `block` rule or a Core
Rule Set match answers `403` and bans nobody, since the Core Rule Set has false
@@ -560,8 +558,8 @@ status for a size or time limit, such as `413` for a body too large.
### Request tags
Each request gets tags from what `discriminator` knows of its client, written in
the `tags` field of its log line:
Each request gets tags from what `waf` knows of its client, written in the
`tags` field of its log line:
- `asn=<number>` and `country=<code>`, from the lookup;
- one tag per list of `SWW_TAG_LISTS` that holds the client, such as `torexit`
@@ -612,17 +610,17 @@ client's limits by tag. See "Request tags" in [`SPEC.md`](SPEC.md).
- `SWW_CORE_RULE_SET_EXCLUSIONS_FILE` switches single rules off for an app's own
fields or paths.
- Rule files are plain text, one rule to a line, edited by hand and read again
while `discriminator` runs. The image ships one file of probes no real visitor
sends (see [`docs/rules.md`](docs/rules.md)).
while `waf` runs. The image ships one file of probes no real visitor sends
(see [`docs/rules.md`](docs/rules.md)).
### State files
`discriminator` keeps everything in memory, and the five JSON files in
`SWW_STATE_DIR` hold a copy: `bans.json`, `clients.json`, `lookups.json`,
`reputation.json` and `alerts.json`. Each is written whole, to a temporary file
renamed over the real one, so a crash leaves a complete file. `discriminator`
notices a file edited on disk and takes the edit in; an edit that does not parse
is set aside as `<name>.bad`, and never stops the running process (see
`waf` keeps everything in memory, and the five JSON files in `SWW_STATE_DIR`
hold a copy: `bans.json`, `clients.json`, `lookups.json`, `reputation.json` and
`alerts.json`. Each is written whole, to a temporary file renamed over the real
one, so a crash leaves a complete file. `waf` notices a file edited on disk and
takes the edit in; an edit that does not parse is set aside as `<name>.bad`, and
never stops the running process (see
[`docs/state-files.md`](docs/state-files.md)).
### The image and its processes
@@ -632,20 +630,19 @@ is set aside as `<name>.bad`, and never stops the running process (see
`Dockerfile` installs a package with `nix-env -iA nixpkgs.<name>` and gets the
same package each time it is built (see [`docs/image.md`](docs/image.md)).
- runit, a service supervisor, runs `sww` as the `sww` user (uid and gid 65532)
and starts it again when it exits. The `run` script of `discriminator` gives
the state directory to the `sww` user at each start, so a volume needs no
change of owner.
- `rproxy` and `discriminator` use no token between them. `discriminator`'s own
endpoints, under `/_sww/`, are for your admin tools and your metrics
collector, at its own address on the private network; nothing routes the
internet there.
and starts it again when it exits. The `run` script of `waf` gives the state
directory to the `sww` user at each start, so a volume needs no change of
owner.
- `rproxy` and `waf` use no token between them. `waf`'s own endpoints, under
`/_sww/`, are for your admin tools and your metrics collector, at its own
address on the private network; nothing routes the internet there.
- `docker stop` gives requests in progress five seconds to finish. With `rproxy`
in the app's container, the app must keep answering for five seconds after
`SIGTERM` for that to help them.
- Only faults found at start stop a process: an invalid setting, a rule file or
state file that does not parse, a lookup database that cannot be read, or an
address it cannot listen on. Once running, a failing lookup service,
reputation source or alert destination never takes `discriminator` down.
reputation source or alert destination never takes `waf` down.
### Packages
@@ -653,8 +650,8 @@ is set aside as `<name>.bad`, and never stops the running process (see
- `internal/sww`: the process, as the command it is given, and the health check.
- `internal/config`: reads the settings, the one place they are read.
- `internal/rproxy`: the reverse proxy in front of one app.
- `internal/discriminator`: `discriminator`'s server, which runs the checks and
answers its own endpoints.
- `internal/waf`: `waf`'s server, which runs the checks and answers its own
endpoints.
- `internal/ratelimit`, `internal/bans`, `internal/rules`,
`internal/coreruleset`, `internal/lookup`, `internal/reputation`,
`internal/routewatch`, `internal/anomaly` and `internal/alerts`: one check or
@@ -734,25 +731,25 @@ on the host.
- `script/precommit`: run by the git pre-commit hook; runs `script/check`.
- `script/install-precommit`: installs that hook. `make hooks`.
- `script/build`: builds `bin/sww` with the Go on your host. `make build`.
- `script/run`: builds `bin/sww` and runs `discriminator` and `rproxy` on the
host, with the settings in your environment. `make run`. Unless your settings
say otherwise:
- `script/run`: builds `bin/sww` and runs `waf` and `rproxy` on the host, with
the settings in your environment. `make run`. Unless your settings say
otherwise:
- `rproxy` listens on port 8080, in front of an app at
`http://127.0.0.1:8081`;
- `discriminator` listens on port 8090;
- `waf` listens on port 8090;
- its state files go in `bin/state`;
- its rule files are those of `share/rules.d`.
- `script/example-app`: builds the image and the example app
(`deploy/example-app`) on it, runs the app with `rproxy` in its container and
`discriminator` in another, and checks them. Then it removes the containers,
the network, the volumes and the images. Needs network access.
`make example-app`. It checks:
`waf` in another, and checks them. Then it removes the containers, the
network, the volumes and the images. Needs network access. `make example-app`.
It checks:
- the health checks;
- a request passed to the app;
- a ban for a broken limit, and a ban for a probe made permanent;
- that only `discriminator`'s container changes the state directory's owner;
- with `discriminator` stopped, the app still answering through `rproxy`;
- with `discriminator` started again, bans applying again;
- that only `waf`'s container changes the state directory's owner;
- with `waf` stopped, the app still answering through `rproxy`;
- with `waf` started again, bans applying again;
- orderly stops with `sv stop` and `docker stop`;
- bans surviving a new container on the same volume;
- a setting given as a file.