smallwebwaf/README.md

# smallwebwaf

`smallwebwaf` is an MIT-licensed web application firewall, written in Go by
[@sneak](https://sneak.berlin), for one self-hosted web app. It runs inside the
app's own container, in front of the app. It refuses the requests of scrapers,
scanners and other abusers before they reach the app.

A web application firewall reads each request on its way to a web app and
refuses the ones that look like abuse. `smallwebwaf` is also a reverse proxy: it
accepts requests on the app's behalf, passes them to the app, and passes the
app's answers back. A request it does not refuse reaches the app as it came,
apart from a few headers (see "Design" below).

It needs no setting. Every setting has a default chosen for a service on the
open internet. [`SPEC.md`](SPEC.md) says how `smallwebwaf` behaves and why. The
reference documents under [`docs/`](docs/) are listed under "Documents" below.

## What it protects against

- Scrapers that fetch every page of a site, all day. Rate limiting gives each
  client 1000 requests a minute, 10,000 an hour and 50,000 a day. Byte limits
  give it 10 GiB a minute, 20 GiB an hour and 50 GiB a day. A client over a
  limit is refused and banned. The first ban lasts an hour. Each repeat within a
  day makes the next ban three times as long. A ban that would last more than
  seven days is permanent.
- Scanners that probe for secrets, such as `/.env` or `/.git/config`. The
  default rule file bans such a client for seven days at its first probe. Its
  next request during the ban makes the ban permanent. The default rule file
  does not cover probes for other apps, such as a WordPress login. A `ban` rule
  of your own, or an entry in `SWWAF_TRAP_PATHS`, bans those the same way.
- Attacks carried in a request's URL or headers, such as SQL injection and path
  traversal. The OWASP Core Rule Set is a published set of rules that recognise
  common web attacks. Coraza, a Go engine, runs those rules (step 11 under "What
  happens to a request" below). A request the rules take for an attack is
  refused.
- Abuse from networks and countries you do not want. You can:
    - refuse whole netblocks and countries;
    - give the clients of chosen networks and countries a fraction of every
      limit, so that they are banned sooner;
    - refuse or limit clients on a blocklist or a DNS blocklist, or with a high
      AbuseIPDB score;
    - ban clients named by your own CrowdSec engine.
- Slow and oversized requests. Every request and every answer has a time limit
  and a size limit.

For the operator, it:

- writes one JSON log line for every request, saying what it did and why (see
  [`docs/request-log.md`](docs/request-log.md));
- sends alerts to a webhook, Slack or ntfy (see
  [`docs/alerts.md`](docs/alerts.md));
- serves Prometheus metrics (see [`docs/metrics.md`](docs/metrics.md));
- keeps its bans, counters and history in JSON files you can read and edit, so
  that a restart loses nothing and a ban can be lifted with an editor (see
  [`docs/state-files.md`](docs/state-files.md)).

Not built, and not planned:

- TLS and routing for several apps. Both stay with traefik, the reverse proxy in
  front of the container.
- Browser challenges such as captchas.
- A web console.
- Defence against floods large enough to fill the host's network link.

## Getting started

Build the image from a clone. `make docker` runs the tests and the linter, then
builds the image, tagged `smallwebwaf`. It builds for amd64, on an amd64 host
only: the `Dockerfile` checks Ubuntu's package lists against the hashes of the
amd64 archive.

```sh
git clone https://git.eeqj.de/sneak/smallwebwaf.git
cd smallwebwaf
make docker
```

Push the image to a registry your hosts pull from. Then build the app's own
image on it. The app's `Dockerfile` starts `FROM` the `smallwebwaf` image, named
by digest, and adds:

- the app's binary;
- the packages the app needs, from the nixpkgs in the image;
- a user for the app;
- a `run` script that starts the app listening on `127.0.0.1:8081`.

```dockerfile
# The smallwebwaf image, pinned by digest.
FROM <registry>/smallwebwaf@sha256:<digest>

# Packages the app needs, if any, from the nixpkgs in the image.
RUN nix-env -iA nixpkgs.git

# The app's binary, and a user of its own to run it.
COPY app /usr/local/bin/app
RUN useradd --system --no-create-home --shell /usr/sbin/nologin app

# The app's runit service.
COPY --chmod=755 app.run /etc/service/app/run
```

`app.run` sits beside the `Dockerfile`. `--listen` and `--trusted-proxies` stand
for the app's own options:

```bash
#!/usr/bin/env bash
set -euo pipefail

main() {
    sleep 1
    exec chpst -u app:app /usr/local/bin/app \
        --listen 127.0.0.1:8081 \
        --trusted-proxies 10.0.0.0/8,172.16.0.0/12,192.168.0.0/16,127.0.0.1/32,::1/128
}

main "$@"
```

Leave `ENTRYPOINT` and `USER` out of the app's `Dockerfile`. The image's own
entrypoint starts `smallwebwaf` and the app side by side, each as its own user.

Deploy the app's image as you deploy any app. Point traefik's labels at port
8080 (`traefik.http.services.<name>.loadbalancer.server.port=8080`). Mount a
volume at `/var/lib/smallwebwaf`, so that bans and history survive a deploy. For
example, with no setting:

```sh
docker run --detach --name app \
    --volume app-smallwebwaf:/var/lib/smallwebwaf \
    <registry>/app@sha256:<digest>
```

With no setting, the address of every new visitor is sent to GeoJS, a web
service, to find its country and network. `SWWAF_LOOKUP_SOURCE=off` sends
nothing. `SWWAF_LOOKUP_SOURCE=file`, with `SWWAF_LOOKUP_DB_PATH` naming a
database file on your own host, looks addresses up in that file (see "Lookup of
AS number and country" in [`SPEC.md`](SPEC.md)).

`deploy/example-app` is such an app. `make example-app` builds it and checks it
(see "Checking that it works" below). To run `smallwebwaf` outside the image
while working on it, see "Entrypoints" below.

## What happens to a request

Two terms recur below. A clear sign of attack is a request that matches a rule
whose action is `ban`, or asks for a trap path (steps 9 and 10). An offence is a
request that counts against its client in its history. It is one of:

- a clear sign of attack;
- a broken limit;
- a refusal by a rule, by the Core Rule Set or for a missing or wrong token.

`smallwebwaf` handles each request in this order. Once a step refuses a request,
the steps after it are skipped, apart from step 13. Nothing of a refused request
reaches the app. "What happens to a request" in [`SPEC.md`](SPEC.md) gives each
step in full.

1. `GET /_smallwebwaf/healthz` is answered `200` with `ok` at once, before any
   check and without asking the app.
2. The client is worked out. A netblock is a range of addresses, written as an
   address and a prefix length: `203.0.113.0/24`. `SWWAF_TRUSTED_PROXIES` lists
   the netblocks traefik comes from, by default the private address ranges. For
   a TCP peer inside them, `X-Forwarded-For` is read from the right, and the
   first address outside those netblocks is the client. A peer outside them is
   the client itself. An IPv6 client is its group: the whole netblock around its
   address, a /64 by default; `SWWAF_IPV6_GROUP_PREFIX` sets its length.
3. A client in `SWWAF_ALLOW_NETS` skips steps 4 to 11, the byte limits and the
   error burst, but still needs a token at step 12. A client in
   `SWWAF_DENY_NETS` is refused.
4. A client under a ban is refused. If the ban was made for a clear sign of
   attack, this request makes it permanent.
5. The client's AS number and country are looked up, unless
   `SWWAF_LOOKUP_SOURCE` is `off`. An AS number names the network an address
   belongs to, such as `AS64496`.
6. The country lists are checked. A client from a country in
   `SWWAF_DENIED_COUNTRIES` is refused. When
   `SWWAF_EXCLUSIVELY_ALLOWED_COUNTRIES` is set, a client from any other country
   is refused, and so is a client without a country. Such a refusal is not an
   offence.
7. The reputation sources are checked: the blocklists, the CrowdSec decision
   list, the DNS blocklists and AbuseIPDB. A client on the CrowdSec decision
   list is banned until the decision ends. Any other source refuses the client,
   lowers its limits or only notes it, as `SWWAF_BLOCKLIST_ACTION` and
   `SWWAF_REPUTATION_ACTION` say. No request waits for a query.
8. Rate limiting counts the request, unless the client or the path is exempt. A
   request that takes the client over a limit is refused and bans the client.
9. A request for a path in `SWWAF_TRAP_PATHS` is refused and bans the client, as
   a clear sign of attack.
10. The rule files are checked. A `log` rule notes the match. A `block` rule
    refuses the request. A `ban` rule refuses it and bans the client, as a clear
    sign of attack.
11. The Core Rule Set inspects the request's method, URL and headers, and, once
    `SWWAF_WAF_BODY_LIMIT` is set, form, JSON and XML bodies. A request it takes
    for an attack is refused while `SWWAF_WAF_MODE` is `block`, the default, and
    let through while it is `detect`. It bans no one by itself.
12. A request under `/_smallwebwaf/` is answered here by `smallwebwaf`, after
    its token is checked. Any other request is passed to the app. The app's
    answer is streamed back within the size and time limits.
13. Once the answer has ended, a request passed to the app has its bytes counted
    for the byte limits. If a rule, a trap path, the Core Rule Set or a token
    check refused the request, the refusal is counted for the error burst. The
    client's history and the anomaly counters are updated. More than
    `SWWAF_ERROR_BURST_THRESHOLD` refusals of one client in a minute ban it.

In `observe` mode (`SWWAF_MODE=observe`), `smallwebwaf` makes no ban of its own.
It refuses a request only for a size or time limit, or for a missing or wrong
token. A request it would otherwise have refused is passed to the app, and its
log line says what `enforce` mode would have done. Observe mode is for trying a
configuration before enforcing it.

What a refused visitor sees:

- `SWWAF_BAN_RESPONSE`, by default `403` with the plain text `Forbidden`. It can
  be `429` instead, or `close`, which closes the connection, so that traefik
  answers `502`. It answers:
    - a ban or a broken rate limit;
    - a `ban` rule or a trap path;
    - `SWWAF_DENY_NETS` and the country lists;
    - a reputation source whose action is `deny`.
- `403` for a `block` rule and the Core Rule Set.
- For a size or time limit:
    - `431` for headers over their limit;
    - `413` for a body over its limit;
    - `408` for a client too slow to send its body;
    - `502` or `504` when the app passes a limit or cannot be reached.
- `401` from an endpoint of `smallwebwaf`'s own, for a request without its
  token.

## Settings you are likely to touch

Each setting is an environment variable of the app's container, such as
`docker run -e SWWAF_MODE=observe ...`, read at start.
[`docs/settings.md`](docs/settings.md) lists every setting, its default and the
values it takes. The ones most deployments touch:

- `SWWAF_MODE=observe`: log what would have been refused, refuse almost nothing,
  ban no one. For trying a configuration.
- `SWWAF_ALLOW_NETS`: netblocks you trust, such as your monitoring. Their
  clients are refused only for a size or time limit or a missing or wrong token.
- `SWWAF_RATE_LIMIT_EXEMPT_NETS`: netblocks exempt from rate limiting and the
  byte limits, such as a machine that talks to the app all day.
- `SWWAF_TRAP_PATHS`: paths only scanners ask for, such as
  `/wp-login.php,/xmlrpc.php` in front of a code forge. A request for one bans
  the client.
- `SWWAF_WAF_DISABLED_RULES`: Core Rule Set rule ids to switch off when a rule
  refuses your app's ordinary requests. The log line names the rules a request
  matched. Keep the six default ids in a list you give (see
  [`docs/settings.md`](docs/settings.md)).
- `SWWAF_REQUEST_MAX_BYTES`, `SWWAF_CLIENT_REQUEST_TIMEOUT` and
  `SWWAF_UPSTREAM_REQUEST_TIMEOUT`: raise them for an app that takes uploads
  over 100 MiB or slower than 60 seconds.
- `SWWAF_LOOKUP_SOURCE`: `geojs` (the default), `file` or `off`, as above.
- `SWWAF_ADMIN_TOKEN` and `SWWAF_METRICS_TOKEN`: switch on the ban endpoints and
  the metrics. Give them as files to keep them from the app.
- `SWWAF_ALERT_WEBHOOK_URL`, `SWWAF_ALERT_SLACK_WEBHOOK_URL` or
  `SWWAF_ALERT_NTFY_URL`: where alerts go.
- `SWWAF_INSTANCE_NAME`: the name in every log line, alert and metric, such as
  `fsn1app1/gitea`.

## Checking that it works

- `docker inspect --format '{{.State.Health.Status}}' <container>` says
  `healthy` once `smallwebwaf` answers and the app accepts connections. A log
  line with `"msg":"starting"` gives the version, the address and every setting
  in effect, with secrets masked.
- A request to the site through traefik gets the app's answer, and its log line
  has `"action":"forward"`. `https://<site>/_smallwebwaf/healthz` answers `ok`.
- To see a refusal without banning yourself, run the container with
  `SWWAF_MODE=observe` and ask for `/.env`. The request reaches the app. Its log
  line has `"would_action":"banned"` and `"rule_ids":["env-file"]`. In `enforce`
  mode the same request bans your address for seven days, and your next request
  makes the ban permanent. Lift the ban by editing `bans.json` (see "Bans" in
  [`SPEC.md`](SPEC.md)).
- With `SWWAF_METRICS_TOKEN` set,
  `curl -H "Authorization: Bearer $TOKEN" https://<site>/_smallwebwaf/metrics`
  answers with the metrics.
- `make example-app` builds the image and the example app on it, runs the app,
  and checks it. It needs network access. It checks:
    - the health check;
    - a request forwarded to the app;
    - a ban for a broken limit, and a ban for a probe made permanent;
    - orderly stops with `sv stop` and `docker stop`;
    - bans surviving a new container on the same volume;
    - a setting given as a file.

## Entrypoints

This repository adheres to the
[Scripts to Rule Them All](https://github.com/github/scripts-to-rule-them-all)
standard. The scripts in `script/` are the entrypoints for working on it, and
the `Makefile` targets are thin shims that call them. The scripts are POSIX sh.
The tests and the linter run in Docker, as phases of the `Dockerfile`, and never
on the host.

- `script/bootstrap`: installs on the host what the other scripts need, with
  `make bootstrap`:
    - `make`, `git` and `curl`;
    - Go, for `gofmt`;
    - node, yarn and prettier.
- `script/setup`: readies a fresh clone: `script/bootstrap`, then
  `script/install-precommit`. `make setup`.
- `script/projectname`: prints `smallwebwaf`, which the other scripts tag their
  images with.
- `script/test`: checks that `go.mod` and `go.sum` are tidy, then runs the
  tests, as the `test` phase of the `Dockerfile`. `make test`.
- `script/tidy`: writes `go.mod` and `go.sum` as `go mod tidy` writes them, in
  the `tidy` stage of the `Dockerfile`. `make tidy`.
- `script/lint`: runs golangci-lint, as the `lint` phase of the `Dockerfile`.
  `make lint`.
- `script/fmt`: formats the Go code with `gofmt` and the Markdown with prettier.
  `make fmt`.
- `script/fmt-check`: checks the formatting and changes nothing.
  `make fmt-check`.
- `script/check`: runs `script/test`, `script/lint` and `script/fmt-check`.
  `make check`.
- `script/docker`: builds the image, whose build runs the tests and the linter
  first. `make docker`.
- `script/cibuild`: what CI runs: `script/bootstrap`, `script/check`, then the
  image build.
- `script/precommit`: run by the git pre-commit hook; runs `script/check`.
- `script/install-precommit`: installs that hook. `make hooks`.
- `script/build`: builds `bin/smallwebwaf` with the Go on your host.
  `make build`.
- `script/run`: builds `bin/smallwebwaf` and runs it with the settings in your
  environment, listening on port 8080 in front of an app at
  `SWWAF_UPSTREAM_URL`. Its state files go in `bin/state` and its rule files are
  those of `share/rules.d`, unless `SWWAF_STATE_DIR` and `SWWAF_RULES_DIR` say
  otherwise. `make run`.
- `script/example-app`: builds the image and the example app on it, runs it with
  a volume, and checks it as "Checking that it works" above describes. Then it
  removes the containers, volumes and images. Needs network access.
  `make example-app`.
- `script/gitea-requests`: builds the image and runs gitea 1.27.3 behind it at
  the defaults. It sends through it a sample of gitea's ordinary use and the
  requests the Core Rule Set is known to refuse, and prints what `smallwebwaf`
  did with each. Needs network access, and a docker daemon that pushes to a
  registry on `localhost` over plain HTTP. [`docs/gitea.md`](docs/gitea.md)
  records the results. `make gitea-requests`.

## Rationale

Small self-hosted sites receive a great deal of traffic nobody asked for:

- scrapers that ignore `robots.txt` and crawl every commit of every repository
  on a public git server;
- scanners walking through lists of WordPress and `.env` paths;
- credential-guessing bots.

Most of it comes from a small number of hosting networks and countries. A
single-person operation has no abuse desk and no CDN contract. It needs
something small that can be put in front of one service and left alone.

The existing tools each solve part of this. Rule-based firewalls catch attack
payloads but do not limit request rates. Rate limiters count requests but cannot
tell a residential visitor from a rented server farm. The products that do most
of it want several containers, a database and a web console. None of them can
say "clients from these networks are banned after half as many requests as
anyone else". That is the useful thing to say when nearly all abuse comes from a
known list of AS numbers. [`EVALUATION.md`](EVALUATION.md) goes through the
candidates one by one.

`smallwebwaf` fills that gap with:

- per-client limits that ban;
- lower limits for the networks and countries abuse comes from;
- bans that grow with repetition;
- a log that is easy to search and ship;
- one binary in the app's own container, with no database and no required
  setting.

## Design

The `smallwebwaf` image is built on Ubuntu 26.04 LTS, pinned by digest, with
nixpkgs fixed at one release. An app's `Dockerfile` installs a package with
`nix-env -iA nixpkgs.<name>` and gets the same package each time it is built.
[`docs/image.md`](docs/image.md) lists what the image holds and how each part is
pinned.

- The container's first process is `runsvinit`, which starts runit, a service
  supervisor. runit runs the `run` script of each directory under
  `/etc/service`, and runs it again a second after its program exits.
  `smallwebwaf` runs as the `smallwebwaf` user (uid 65532) and the app as a user
  of its own. So the app cannot touch the state files or the process of
  `smallwebwaf`.
- `smallwebwaf` listens on port 8080, on every address. traefik sends the app's
  requests there. `smallwebwaf` passes them to the app on `127.0.0.1:8081`. The
  app listens on `127.0.0.1` only, so nothing reaches it except through
  `smallwebwaf`. The health check, the metrics and the ban management are on the
  same port, under `/_smallwebwaf/`. A request for that prefix never reaches the
  app.
- The app sees what it would see from traefik directly: the same `Host`, the
  same `X-Forwarded-Proto`, and `X-Forwarded-For` with traefik's address added
  at the end. The app's TCP peer is `smallwebwaf` on `127.0.0.1`. So the app
  must trust `127.0.0.1` and `::1` for forwarded headers, besides the private
  address ranges. An app that keeps its usual trusted proxies sees every visitor
  as `127.0.0.1`. Each request also carries its id in `X-Request-ID`. Any
  `X-Client-ASN` or `X-Client-Country` header a client sent is removed.
- Both processes start with the container's environment variables. That is why
  every setting of `smallwebwaf` starts with `SWWAF_`. A token given as an
  environment variable is readable by the app. A token given as a file that only
  the `smallwebwaf` user can read is not (see "Settings given as files" in
  [`docs/settings.md`](docs/settings.md)).
- The state files live in `/var/lib/smallwebwaf`. Without a volume there, they
  survive a restart of the container and are lost when a deploy replaces it. The
  `run` script gives that directory and every file in it to the `smallwebwaf`
  user at each start, so a host directory mounted there needs no change of
  owner.
- The image's health check passes while `smallwebwaf` answers
  `GET /_smallwebwaf/healthz` and the app accepts connections. traefik sends a
  container no requests until it is healthy.
- When `smallwebwaf` or the app exits, runit starts it again a second later. The
  other process keeps running. While either is down, the client gets `502`. A
  `smallwebwaf` that stops at start, as on an invalid setting, is started again
  every second until the cause is fixed, and logs its message each time.
- `docker stop` has runit send `SIGTERM` to both processes at once.
  `smallwebwaf` gives the requests in progress five seconds to finish and writes
  its state files. To let them finish, the app must keep answering for five
  seconds after `SIGTERM`.

"Deployment" in [`SPEC.md`](SPEC.md) has the rest: moving `smallwebwaf` to
another port, the health check's timing, and the forwarded headers.
"Architecture" in [`SPEC.md`](SPEC.md) lists the packages and what each does.

## TODO

- Planned, as "Planned, not built" in [`SPEC.md`](SPEC.md) describes:
    - sharing bans between the instances of several apps;
    - sending the log over RELP.

## Documents

- [`SPEC.md`](SPEC.md): how `smallwebwaf` behaves, and why.
- [`EVALUATION.md`](EVALUATION.md): what already exists, what each tool covers
  and misses, and why none was adopted.
- [`REPO_POLICIES.md`](REPO_POLICIES.md): the policies this repository follows.
- [`docs/settings.md`](docs/settings.md): every setting, its default and the
  values it takes.
- [`docs/rules.md`](docs/rules.md): the rule file format and the default file.
- [`docs/admin-endpoints.md`](docs/admin-endpoints.md): the endpoints under
  `/_smallwebwaf/`, with examples.
- [`docs/request-log.md`](docs/request-log.md): the request log's fields.
- [`docs/alerts.md`](docs/alerts.md): the alert events and what each destination
  gets.
- [`docs/metrics.md`](docs/metrics.md): the metrics.
- [`docs/state-files.md`](docs/state-files.md): the state files and what each
  holds.
- [`docs/image.md`](docs/image.md): what the image holds and how it is pinned.
- [`docs/gitea.md`](docs/gitea.md): `smallwebwaf` in front of gitea.

## License

MIT. See [`LICENSE`](LICENSE).

## Author

[@sneak](https://sneak.berlin)