smallwebwaf/README.md

# smallwebwaf

`smallwebwaf` is an MIT-licensed web application firewall, written in Go by
[@sneak](https://sneak.berlin), for self-hosted web apps. It refuses the
requests of scrapers, scanners and other abusers before they reach an app.

A web application firewall reads each request on its way to a web app and
refuses the ones that look like abuse. `smallwebwaf` is one binary with two
commands that do this together:

- `smallwebwaf rproxy` is a small reverse proxy in front of one app. It accepts
  requests on the app's behalf and asks `servewaf` about each one. It passes the
  allowed ones to the app, and the app's answers back. It keeps nothing, and
  never touches disk.
- `smallwebwaf servewaf` decides. It runs the checks, and keeps the bans,
  counters and history. One `servewaf` serves many sites.

The site keeps answering while `servewaf` is down, slow or restarting. `rproxy`
waits at most 2 seconds for `servewaf`'s answer, and without one passes the
request to the app.

`rproxy` requires one setting, `SWRP_WAF_BACKEND`, which says where `servewaf`
is. In a container of its own, it also needs `SWRP_UPSTREAM_URL`, which says
where the app is. Every other setting has a default chosen for a service on the
open internet. [`SPEC.md`](SPEC.md) says how `smallwebwaf` behaves and why. The
reference documents under [`docs/`](docs/) are listed under "Documents" below.

## What it protects against

- Scrapers that fetch every page of a site, all day. Rate limiting gives each
  client 500 requests a minute, 20,000 an hour and 300,000 a day. That leaves
  room for programmatic use of an API, such as an automation fleet on another
  host. A fleet of about 20 agents makes about 120 requests a minute at its
  busiest. A gitea page opened with an empty cache loads about 40 files, so a
  few such first visits fit in the minute limit too. The minute limit catches,
  within seconds, a client firing requests as fast as it can. The hour limit
  catches a client that keeps up more than about 5.5 a second for an hour. The
  day limit catches a client that keeps up 4 a second around the clock. Counting
  cannot catch a client that stays under all three limits, such as one that
  keeps up 3 a second; only the protections below can. Byte limits give each
  client 4 GiB a minute, 20 GiB an hour and 200 GiB a day. A client over a limit
  is refused and banned. The first ban lasts an hour. Each repeat within a day
  makes the next ban three times as long. A ban that would last more than seven
  days is permanent.
- Scanners that probe for secrets, such as `/.env` or `/.git/config`. The
  default rule file bans such a client for seven days at its first probe. Its
  next request during the ban makes the ban permanent. The default rule file
  does not cover probes for other apps, such as a WordPress login. A `ban` rule
  of your own, or an entry in `SWWAF_TRAP_PATHS`, bans those the same way.
- Attacks carried in a request's URL or headers, such as SQL injection and path
  traversal. The OWASP Core Rule Set is a published set of rules that recognise
  common web attacks. Coraza, a Go engine, runs those rules (see "Attack
  detection" in [`SPEC.md`](SPEC.md)). A request the rules take for an attack is
  refused.
- Abuse from networks and countries you do not want. You can:
    - refuse whole netblocks and countries;
    - give the clients of chosen networks and countries a fraction of every
      limit, so that they are banned sooner;
    - refuse or limit clients on a blocklist or a DNS blocklist, or with a high
      AbuseIPDB score;
    - ban clients named by your own CrowdSec engine.
- Slow and oversized requests. Every request and every answer has a time limit
  and a size limit.

For the operator, it:

- logs each request as JSON lines, saying what it did and why (see
  [`docs/request-log.md`](docs/request-log.md));
- sends alerts to a webhook, Slack or ntfy (see
  [`docs/alerts.md`](docs/alerts.md));
- serves Prometheus metrics (see [`docs/metrics.md`](docs/metrics.md));
- keeps its bans, counters and history in JSON files you can read and edit, so
  that a restart loses nothing and a ban can be lifted with an editor (see
  [`docs/state-files.md`](docs/state-files.md)).

Not built, and not planned:

- TLS and routing by host name. Both stay with traefik, the reverse proxy in
  front of `rproxy`.
- Browser challenges such as captchas.
- A web console.
- Defence against floods large enough to fill the host's network link.

## Getting started

Build the image from a clone. `make docker` runs the tests and the linter, then
builds the image, tagged `smallwebwaf`. It builds for amd64, on an amd64 host
only: the `Dockerfile` checks Ubuntu's package lists against the hashes of the
amd64 archive.

```sh
git clone https://git.eeqj.de/sneak/smallwebwaf.git
cd smallwebwaf
make docker
```

Push the image to a registry your hosts pull from.

The image runs `rproxy` by default. To run `servewaf` from the same image, set
the container's `SMALLWEBWAF_COMMAND` to `servewaf`.

### The usual setup: three containers

The usual setup is three containers on one private network:

- the app;
- `rproxy`, in front of the app, which traefik routes to;
- `servewaf`, with a volume for its state files.

```sh
docker network create site

docker run --detach --name servewaf --network site \
    --env SMALLWEBWAF_COMMAND=servewaf \
    --volume servewaf-state:/var/lib/smallwebwaf \
    <registry>/smallwebwaf@sha256:<digest>

docker run --detach --name app --network site \
    <registry>/app@sha256:<digest>

docker run --detach --name rproxy --network site \
    --env SWRP_WAF_BACKEND=http://servewaf:8080 \
    --env SWRP_UPSTREAM_URL=http://app:8081 \
    <registry>/smallwebwaf@sha256:<digest>
```

Here the app listens on port 8081, at its container's address on that network.
Point traefik's labels at port 8080 of `rproxy`
(`traefik.http.services.<name>.loadbalancer.server.port=8080`). Publish no port
of the app or of `servewaf`. Anything on that network can reach the app around
`rproxy`, and can ask `servewaf` questions.

With no setting, `servewaf` sends the address of every new visitor to GeoJS, a
web service, to find its country and network. `SWWAF_LOOKUP_SOURCE=off` sends
nothing. `SWWAF_LOOKUP_SOURCE=file`, with `SWWAF_LOOKUP_DB_PATH` naming a
database file on your own host, looks addresses up in that file (see "Lookup of
AS number and country" in [`SPEC.md`](SPEC.md)).

### Many sites, one `servewaf`

Run one `rproxy` for each app, all with `SWRP_WAF_BACKEND` naming the same
`servewaf`. It keeps one table of clients and bans for all of them:

- A client banned through one site is refused on all of them.
- A client's requests on all of them count toward one set of limits.
- Log lines, alerts and ban notes name the site by its host.

### `rproxy` in the app's container

An app's image can also be built on the `smallwebwaf` image, so that `rproxy`
runs beside the app. `servewaf` still runs in a container of its own. The app's
`Dockerfile` starts `FROM` the `smallwebwaf` image, named by digest, and adds:

- the app's binary;
- the packages the app needs, from the nixpkgs in the image;
- a user for the app;
- a `run` script that starts the app listening on `127.0.0.1:8081`, the default
  of `SWRP_UPSTREAM_URL`.

```dockerfile
# The smallwebwaf image, pinned by digest.
FROM <registry>/smallwebwaf@sha256:<digest>

# Packages the app needs, if any, from the nixpkgs in the image.
RUN nix-env -iA nixpkgs.git

# The app's binary, and a user of its own to run it.
COPY app /usr/local/bin/app
RUN useradd --system --no-create-home --shell /usr/sbin/nologin app

# The app's runit service.
COPY --chmod=755 app.run /etc/service/app/run
```

`app.run` sits beside the `Dockerfile`. `--listen` and `--trusted-proxies` stand
for the app's own options:

```bash
#!/usr/bin/env bash
set -euo pipefail

main() {
    sleep 1
    exec chpst -u app:app /usr/local/bin/app \
        --listen 127.0.0.1:8081 \
        --trusted-proxies 10.0.0.0/8,172.16.0.0/12,192.168.0.0/16,127.0.0.1/32,::1/128
}

main "$@"
```

Leave `ENTRYPOINT` and `USER` out of the app's `Dockerfile`. The image's own
entrypoint starts `rproxy` and the app side by side, each as its own user. Run
the app's image with `SWRP_WAF_BACKEND` set.

`deploy/example-app` is such an app. `make example-app` builds it and checks it
(see "Checking that it works" below). To run `rproxy` and `servewaf` on the host
while working on them, see `script/run` under "Entrypoints" below.

### When `servewaf` is down, slow or overloaded

`SWRP_DEFAULT_DISPOSITION` decides what `rproxy` does with a request when
`servewaf`:

- cannot be connected to;
- does not answer at all;
- does not answer within `SWRP_TIMEOUT_MS`, 2 seconds by default.

Its default, `allow`, passes the request to the app without `servewaf`'s answer.
`deny` answers `503` instead. Either way the request gets no report, so its
bytes are not counted and it is not added to its client's history. When
`servewaf` does answer, a `deny` or an error goes to the visitor as `servewaf`
gave it, and `SWRP_DEFAULT_DISPOSITION` plays no part.

## What happens to a request

A request goes through fifteen steps, in this order. Once `servewaf` refuses a
request, the steps after that are skipped, apart from the report of step 15, and
nothing of the request reaches the app. "What happens to a request" in
[`SPEC.md`](SPEC.md) gives each step in full, under the same numbers.

`rproxy` takes the first two:

1. The health check, `GET /_smallwebwaf/healthz`, is answered at once.
2. The visitor's address is worked out, and `servewaf` is asked about the
   request.

`servewaf` takes the checks:

3. The static lists, `SWWAF_ALLOW_NETS` and `SWWAF_DENY_NETS`.
4. The bans.
5. The lookup of the client's AS number and country.
6. The country lists.
7. The reputation sources.
8. The client's limit percentage.
9. The rate limits.
10. The trap paths.
11. The rule files.
12. The header limits, and then the Core Rule Set.
13. A path under `/_smallwebwaf/` is answered `404`, so that the app never sees
    that prefix.

`rproxy` takes the rest, with `servewaf` counting at the end:

14. The request is passed to the app.
15. After the answer, `rproxy` sends `servewaf` a report. For an allowed
    request, `servewaf` counts the byte limits and adds the request to the
    client's history.

A refused visitor is answered with `SWWAF_BAN_RESPONSE`, `403` by default (see
"Ban lengths, response and scope" in [`SPEC.md`](SPEC.md)). A `block` rule or a
Core Rule Set match answers `403`. `rproxy` itself answers `405` to a method it
does not pass on, and the status for a size or time limit, such as `413` for a
body too large (see "`rproxy`" in [`SPEC.md`](SPEC.md)).

## Settings you are likely to touch

Each setting is an environment variable, read at start. Those of `servewaf`
start with `SWWAF_`, and those of `rproxy` with `SWRP_`.
[`docs/settings.md`](docs/settings.md) lists every setting, its default and the
values it takes. The ones most deployments touch:

- `SWWAF_MODE=observe`: `servewaf` refuses nothing and makes no ban of its own.
  A request it would have refused is passed to the app, and its log line says
  what `enforce` mode would have done. For trying a configuration before
  enforcing it. The limits of `rproxy` still apply.
- `SWWAF_ALLOW_NETS`: netblocks you trust, such as your monitoring. Their
  clients skip `servewaf`'s checks.
- `SWWAF_RATE_LIMIT_EXEMPT_NETS`: netblocks exempt from rate limiting and the
  byte limits, such as a machine that talks to the app all day.
- `SWWAF_TRAP_PATHS`: paths only scanners ask for, such as
  `/wp-login.php,/xmlrpc.php` in front of a code forge. A request for one bans
  the client.
- `SWWAF_WAF_DISABLED_RULES`: Core Rule Set rule ids to switch off when a rule
  refuses your app's ordinary requests. The log line names the rules a request
  matched. Keep the six default ids in a list you give (see
  [`docs/settings.md`](docs/settings.md)).
- `SWRP_MAX_REQUEST_SIZE` and `SWRP_MAX_REQUEST_TIME_MS`: raise them for an app
  that takes uploads over 100 MiB or slower than 60 seconds.
- `SWRP_DEFAULT_DISPOSITION`: `allow` (the default) or `deny`, as above.
- `SWRP_TRUSTED_PROXIES`: the netblocks whose `X-Forwarded-For` `rproxy`
  believes, by default the private address ranges.
- `SWWAF_LOOKUP_SOURCE`: `geojs` (the default), `file` or `off`, as above.
- `SWWAF_ADMIN_TOKEN` and `SWWAF_METRICS_TOKEN`: switch on the ban endpoints and
  the metrics, at `servewaf`'s own address.
- `SWWAF_ALERT_WEBHOOK_URL`, `SWWAF_ALERT_SLACK_WEBHOOK_URL` or
  `SWWAF_ALERT_NTFY_URL`: where alerts go.
- `SWWAF_INSTANCE_NAME`: the name in every line `servewaf` logs, and in every
  alert and metric, such as `fsn1/servewaf`.

## Checking that it works

- `docker inspect --format '{{.State.Health.Status}}' <container>` says
  `healthy` for each container of `smallwebwaf`:
    - for `rproxy`, once it answers and the app accepts connections, whether or
      not `servewaf` answers;
    - for `servewaf`, once it answers.
- A log line with `"msg":"starting"` gives the version, the address and every
  setting in effect, with secrets masked.
- A request to the site through traefik gets the app's answer.
  `https://<site>/_smallwebwaf/healthz` answers `ok`. `servewaf`'s log lines
  about the request have `"action":"forward"`.
- To see a refusal without banning yourself, run `servewaf` with
  `SWWAF_MODE=observe` and ask for `/.env`. The request reaches the app.
  `servewaf`'s log lines have `"would_action":"banned"` and
  `"rule_ids":["env-file"]`. In `enforce` mode the same request bans your
  address for seven days, and your next request makes the ban permanent. Lift
  the ban by editing `bans.json` (see "Bans" in [`SPEC.md`](SPEC.md)).
- With `SWWAF_METRICS_TOKEN` set, this answers with the metrics, from a host on
  `servewaf`'s network:
  `curl -H "Authorization: Bearer $TOKEN" http://servewaf:8080/_smallwebwaf/metrics`.
- `make example-app` builds the image and the example app on it, runs the app
  with `rproxy` in its container and `servewaf` in another, and checks them. It
  needs network access. It checks:
    - the health checks;
    - a request passed to the app;
    - a ban for a broken limit, and a ban for a probe made permanent;
    - that only `servewaf`'s container changes the state directory's owner;
    - with `servewaf` stopped, the app still answering through `rproxy`;
    - with `servewaf` started again, bans applying again;
    - orderly stops with `sv stop` and `docker stop`;
    - bans surviving a new container on the same volume;
    - a setting given as a file.

## Entrypoints

This repository adheres to the
[Scripts to Rule Them All](https://github.com/github/scripts-to-rule-them-all)
standard. The scripts in `script/` are the entrypoints for working on it, and
the `Makefile` targets are thin shims that call them. The scripts are POSIX sh.
The tests and the linter run in Docker, as phases of the `Dockerfile`, and never
on the host.

- `script/bootstrap`: installs on the host what the other scripts need, with
  `make bootstrap`:
    - `make`, `git` and `curl`;
    - Go, for `gofmt`;
    - node, yarn and prettier.
- `script/setup`: readies a fresh clone: `script/bootstrap`, then
  `script/install-precommit`. `make setup`.
- `script/projectname`: prints `smallwebwaf`, which the other scripts tag their
  images with.
- `script/test`: checks that `go.mod` and `go.sum` are tidy, then runs the
  tests, as the `test` phase of the `Dockerfile`. `make test`.
- `script/tidy`: writes `go.mod` and `go.sum` as `go mod tidy` writes them, in
  the `tidy` stage of the `Dockerfile`. `make tidy`.
- `script/lint`: runs golangci-lint, as the `lint` phase of the `Dockerfile`.
  `make lint`.
- `script/fmt`: formats the Go code with `gofmt` and the Markdown with prettier.
  `make fmt`.
- `script/fmt-check`: checks the formatting and changes nothing.
  `make fmt-check`.
- `script/check`: runs `script/test`, `script/lint` and `script/fmt-check`.
  `make check`.
- `script/docker`: builds the image, whose build runs the tests and the linter
  first. `make docker`.
- `script/cibuild`: what CI runs: `script/bootstrap`, `script/check`, then the
  image build.
- `script/precommit`: run by the git pre-commit hook; runs `script/check`.
- `script/install-precommit`: installs that hook. `make hooks`.
- `script/build`: builds `bin/smallwebwaf` with the Go on your host.
  `make build`.
- `script/run`: builds `bin/smallwebwaf` and runs `servewaf` and `rproxy` on the
  host, with the settings in your environment. `make run`. Unless your settings
  say otherwise:
    - `rproxy` listens on port 8080, in front of an app at
      `http://127.0.0.1:8081`;
    - `servewaf` listens on port 8090;
    - its state files go in `bin/state`;
    - its rule files are those of `share/rules.d`.
- `script/example-app`: builds the image and the example app on it, runs them
  with a volume, and checks them as "Checking that it works" above describes.
  Then it removes the containers, the network, the volumes and the images. Needs
  network access. `make example-app`.
- `script/gitea-requests`: builds the image and runs gitea 1.27.3 behind
  `rproxy` and `servewaf`, three containers on one network. Every setting is at
  its default, but for those that join the containers and the header that labels
  each request. It sends through them a sample of gitea's ordinary use and the
  requests the Core Rule Set is known to refuse, and prints what `servewaf` did
  with each. Needs network access, and a docker daemon that pushes to a registry
  on `localhost` over plain HTTP. [`docs/gitea.md`](docs/gitea.md) records the
  results. `make gitea-requests`.

## Rationale

Small self-hosted sites receive a great deal of traffic nobody asked for:

- scrapers that ignore `robots.txt` and crawl every commit of every repository
  on a public git server;
- scanners walking through lists of WordPress and `.env` paths;
- credential-guessing bots.

Most of it comes from a small number of hosting networks and countries. A
single-person operation has no abuse desk and no CDN contract. It needs
something small that can be put in front of its services and left alone.

The existing tools each solve part of this. Rule-based firewalls catch attack
payloads but do not limit request rates. Rate limiters count requests but cannot
tell a residential visitor from a rented server farm. The products that do most
of it want a database and a web console. None of them can say "clients from
these networks are banned after half as many requests as anyone else". That is
the useful thing to say when nearly all abuse comes from a known list of AS
numbers. [`EVALUATION.md`](EVALUATION.md) goes through the candidates one by
one.

`smallwebwaf` fills that gap with:

- per-client limits that ban;
- lower limits for the networks and countries abuse comes from;
- bans that grow with repetition;
- a log that is easy to search and ship;
- a site that keeps answering while the part that decides is down;
- one binary, with no database and one required setting.

## Design

The `smallwebwaf` image is built on Ubuntu 26.04 LTS, pinned by digest, with
nixpkgs fixed at one release. An app's `Dockerfile` installs a package with
`nix-env -iA nixpkgs.<name>` and gets the same package each time it is built.
[`docs/image.md`](docs/image.md) lists what the image holds and how each part is
pinned.

- runit, a service supervisor, runs `smallwebwaf` and starts it again when it
  exits. In an app's image built on the `smallwebwaf` image, it runs `rproxy`
  and the app side by side, each as its own user.
- `rproxy` listens on port 8080 and passes requests to the app at
  `SWRP_UPSTREAM_URL`. It answers the health check itself, and the refusals and
  errors "`rproxy`" in [`SPEC.md`](SPEC.md) lists.
- `servewaf` listens on port 8080 of its own container. Nothing should route the
  internet there. Its own endpoints, under `/_smallwebwaf/`, are for your admin
  tools and your metrics collector.
- `rproxy` and `servewaf` use no token between them. Anything on their private
  network can ask `servewaf` a question, or send it a report.
- The app sees the headers traefik sent, with traefik's address added to
  `X-Forwarded-For`, from a TCP peer that is `rproxy`. So the app must trust
  `rproxy`'s address for forwarded headers. With `rproxy` in the app's
  container, that address is `127.0.0.1` or `::1`, which the private address
  ranges leave out.
- The state files live in `/var/lib/smallwebwaf` in `servewaf`'s container. A
  volume mounted there keeps them across a deploy.
- `docker stop` gives requests in progress five seconds to finish. With `rproxy`
  in the app's container, the app must keep answering for five seconds after
  `SIGTERM` for that to help them.

"Deployment" in [`SPEC.md`](SPEC.md) has the rest: the health checks, what
happens when a process stops, moving a port, and the forwarded headers.
"Architecture" in [`SPEC.md`](SPEC.md) lists the packages and what each does.

## TODO

- Planned, as "Planned, not built" in [`SPEC.md`](SPEC.md) describes: sending
  the log over RELP.

## Documents

- [`SPEC.md`](SPEC.md): how `smallwebwaf` behaves, and why.
- [`EVALUATION.md`](EVALUATION.md): what already exists, what each tool covers
  and misses, and why none was adopted.
- [`REPO_POLICIES.md`](REPO_POLICIES.md): the policies this repository follows.
- [`docs/settings.md`](docs/settings.md): every setting, its default and the
  values it takes.
- [`docs/rules.md`](docs/rules.md): the rule file format and the default file.
- [`docs/admin-endpoints.md`](docs/admin-endpoints.md): the endpoints under
  `/_smallwebwaf/`, with examples.
- [`docs/request-log.md`](docs/request-log.md): the request log's fields.
- [`docs/alerts.md`](docs/alerts.md): the alert events and what each destination
  gets.
- [`docs/metrics.md`](docs/metrics.md): the metrics.
- [`docs/state-files.md`](docs/state-files.md): the state files and what each
  holds.
- [`docs/image.md`](docs/image.md): what the image holds and how it is pinned.
- [`docs/gitea.md`](docs/gitea.md): `smallwebwaf` in front of gitea.

## License

MIT. See [`LICENSE`](LICENSE).

## Author

[@sneak](https://sneak.berlin)