README: blocks misbehaving scrapers, not normal ones (closes #387)

The opening now lists what bans a client with the default settings: the
rate and byte limits, a probe for a secret path, a security scanner's
user agent, and the refusal limit. A scraper that does none of these is
left alone.

Model: opus-5-5

README.md

@@ -2,7 +2,20 @@
 
 `smallwebwaf` is an MIT-licensed web application firewall, written in Go by
 [@sneak](https://sneak.berlin), for self-hosted web apps. It refuses the
-requests of scrapers, scanners and other abusers before they reach an app.
+requests of misbehaving scrapers, scanners and other abusers before they reach
+an app.
+
+With the default settings, a client is banned when it does any of these:
+
+- makes more than 120 requests in a minute, 1500 in an hour or 6000 in a day;
+- moves more than `50G` of request and response bodies in an hour, or `100G` in
+  a day;
+- asks for a secret or backup file, such as `/.env` or `/.git/config`;
+- sends a user agent that names a security scanner, such as `sqlmap` or `nikto`;
+- has more than 30 requests in a minute refused as probes or attacks.
+
+A scraper that does none of these is left alone. A scraper that waits 15 seconds
+between requests stays under the rate limits.
 
 ## What it does
 
@@ -10,11 +23,10 @@ A web application firewall reads each request on its way to a web app and
 refuses the ones that look like abuse. `smallwebwaf` is one program, `sww`,
 shipped in one image, with two commands that do this together:
 
-- `sww rproxy` is a small reverse proxy in front of one app. It sits behind the
-  reverse proxy that terminates TLS, such as traefik, nginx or Caddy. It asks
-  `waf` about each request, passes the allowed ones to the app, and passes the
-  app's answers back. It keeps nothing between requests and writes nothing to
-  disk.
+- `sww rproxy` is a small reverse proxy in front of one app. traefik routes to
+  it. It asks `waf` about each request, passes the allowed ones to the app, and
+  passes the app's answers back. It keeps nothing between requests and writes
+  nothing to disk.
 - `sww waf` decides. It runs the checks and keeps the bans, the counters and
   each client's history. One `waf` serves many sites: a client banned through
   one site is refused on all of them, and its requests on all of them count
@@ -58,8 +70,8 @@ For the operator, it:
 
 Not built, and not planned:
 
-- TLS and routing by host name. Both stay with the reverse proxy in front of
-  `rproxy`.
+- TLS and routing by host name. Both stay with traefik, the reverse proxy in
+  front of `rproxy`.
 - Browser challenges such as captchas.
 - A web console, or a configuration file.
 - Defence against floods large enough to fill the host's network link.
@@ -91,48 +103,20 @@ accepts connections, whether or not `waf` answers; in `waf`'s container, once
 The usual setup is three containers on one private network:
 
 - the app;
-- `rproxy`, in front of the app, which the reverse proxy that terminates TLS
-  forwards to: `SWW_RPROXY_WAF_BACKEND` names `waf` and
-  `SWW_RPROXY_UPSTREAM_URL` names the app;
+- `rproxy`, in front of the app, which traefik routes to:
+  `SWW_RPROXY_WAF_BACKEND` names `waf` and `SWW_RPROXY_UPSTREAM_URL` names the
+  app;
 - `waf`, with `SWW_COMMAND=waf` and a volume at `/var/lib/sww` for its state
   files.
 
-Publish no port of the app or of `waf`. Anything on that network can reach the
-app around `rproxy`, and can ask `waf` questions, so the network holds only what
-you trust. Each further app adds its own container and its own `rproxy`, whose
+Point traefik's labels at port 8080 of `rproxy`
+(`traefik.http.services.<name>.loadbalancer.server.port=8080`). Publish no port
+of the app or of `waf`. Anything on that network can reach the app around
+`rproxy`, and can ask `waf` questions, so the network holds only what you trust.
+Each further app adds its own container and its own `rproxy`, whose
 `SWW_RPROXY_WAF_BACKEND` names the same `waf`. The "Smallest setup" example
 below shows this shape.
 
-### The reverse proxy in front
-
-`rproxy` serves plain HTTP. It runs behind any reverse proxy that terminates
-TLS, such as traefik, nginx or Caddy. That proxy must:
-
-- forward each request to port 8080 of `rproxy` over plain HTTP;
-- keep the visitor's `Host` header;
-- add the visitor's address to `X-Forwarded-For`;
-- set `X-Forwarded-Proto` to the scheme the visitor used;
-- connect to `rproxy` from an address in `SWW_RPROXY_TRUSTED_PROXIES`, by
-  default `10.0.0.0/8`, `172.16.0.0/12` or `192.168.0.0/16`.
-
-From any other address, `rproxy` ignores the forwarded headers and takes the
-proxy for the visitor. Every visitor then shares one address, and one ban.
-
-traefik and Caddy keep the `Host` header and set both forwarded headers by
-default. traefik's Docker label
-`traefik.http.services.<name>.loadbalancer.server.port=8080` names the port, and
-Caddy's `reverse_proxy rproxy:8080` needs nothing more. nginx sets none of the
-headers by default, so its site names them:
-
-```nginx
-location / {
-    proxy_pass http://rproxy:8080;
-    proxy_set_header Host $host;
-    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
-    proxy_set_header X-Forwarded-Proto $scheme;
-}
-```
-
 ### `rproxy` in the app's container
 
 An app's image can instead be built on the `smallwebwaf` image, so that `rproxy`
@@ -183,13 +167,13 @@ entrypoint starts `rproxy` and the app side by side, each as its own user. Run
 the app's image with `SWW_RPROXY_WAF_BACKEND` set. `deploy/example-app` is such
 an app, and the "`rproxy` beside the app" example below runs one.
 
-The app sees the headers the reverse proxy in front sent, with that proxy's
-address added to `X-Forwarded-For`, from a TCP peer that is `rproxy`. So the app
-must trust `rproxy`'s address for forwarded headers. In a container of its own,
-`rproxy` is on the private network, which the usual default of a trusted-proxy
-setting covers. Beside the app, its address is `127.0.0.1` or `::1`, which the
-private ranges leave out, so the app's own setting must name them too, as
-`app.run` does.
+The app sees the headers traefik sent, with traefik's address added to
+`X-Forwarded-For`, from a TCP peer that is `rproxy`. So the app must trust
+`rproxy`'s address for forwarded headers. In a container of its own, `rproxy` is
+on the private network, which the usual default of a trusted-proxy setting
+covers. Beside the app, its address is `127.0.0.1` or `::1`, which the private
+ranges leave out, so the app's own setting must name them too, as `app.run`
+does.
 
 ### Checking that it works
 
@@ -197,9 +181,9 @@ private ranges leave out, so the app's own setting must name them too, as
   `healthy` for each container of `smallwebwaf`.
 - A log line with `"msg":"starting"` gives the version, the address and every
   setting in effect, with secrets masked.
-- A request to the site through the reverse proxy in front gets the app's
-  answer. `https://<site>/_sww/healthz` answers `ok`. `waf`'s log lines about
-  the request have `"action":"forward"`.
+- A request to the site through traefik gets the app's answer.
+  `https://<site>/_sww/healthz` answers `ok`. `waf`'s log lines about the
+  request have `"action":"forward"`.
 - To see a refusal without banning yourself, run `waf` with
   `SWW_WAF_MODE=observe` and ask for `/.env`. The request reaches the app.
   `waf`'s log lines have `"would_action":"temporarily_banned"` and

SPEC.md

@@ -3,8 +3,7 @@
 `smallwebwaf` is a web application firewall for self-hosted apps. It is one
 binary, `sww`, and two of its commands do the work:
 
-- `sww rproxy` is a reverse proxy between one app and the reverse proxy that
-  terminates TLS.
+- `sww rproxy` is a reverse proxy between traefik and one app.
 - `sww waf` decides what happens to each request, for many sites.
 
 This document says what it must do and why, each requirement once. Everything in
@@ -46,8 +45,7 @@ never as a time.
 
 ## Non-goals
 
-- TLS, certificates and routing by host name. Those are the job of the reverse
-  proxy in front of `rproxy`, such as traefik, nginx or Caddy.
+- TLS, certificates and routing by host name. Those are traefik's job.
 - More than one app behind one `rproxy`. Each app has its own.
 - Browser challenges such as captchas or proof of work. If wanted, chain Anubis
   between `rproxy` and the app.
@@ -1945,10 +1943,10 @@ leave out a cookie only when its name matches exactly, case included.
 ### Ban lengths, response and scope
 
 `SWW_WAF_BAN_RESPONSE` is `403` by default, which makes a ban easy to recognise
-when debugging; `429` and `close` are the alternatives. `close` does not leave
-the client unanswered: the reverse proxy in front answers `502`, as it does
-whenever its backend drops a connection. `SWW_WAF_BAN_SCOPE_V4_PREFIX` widens an
-IPv4 ban to the surrounding netblock. The four ban lengths are under "Bans".
+when debugging; `429` and `close` are the alternatives. Behind traefik, `close`
+does not leave the client unanswered: traefik answers `502`, as it does whenever
+its backend drops a connection. `SWW_WAF_BAN_SCOPE_V4_PREFIX` widens an IPv4 ban
+to the surrounding netblock. The four ban lengths are under "Bans".
 
 ### Reputation
 
@@ -2773,10 +2771,9 @@ its whole body. The visitor gets that answer at once, and the connection closes
 after it.
 
 The largest request line and headers is 32 KiB. A kept-open connection is closed
-after 120 seconds idle. Neither has a setting. The reverse proxy in front should
-close an unused connection sooner, so that it never sends a request on a
-connection `rproxy` is closing. traefik closes one after 90 seconds by default.
-nginx, by default, opens a new connection for each request.
+after 120 seconds idle. Neither has a setting. traefik, by default, closes an
+unused connection after 90 seconds. So traefik closes first, and never sends a
+request on a connection `rproxy` is closing.
 
 What the visitor gets when a limit is passed:
 
@@ -2827,16 +2824,14 @@ chooses: `waf` runs `waf`, and with it unset the image runs `rproxy`.
 The usual setup is three containers on one private network:
 
 - the app;
-- `rproxy`, which the reverse proxy that terminates TLS forwards to, with
-  `SWW_RPROXY_UPSTREAM_URL` naming the app and `SWW_RPROXY_WAF_BACKEND` naming
-  `waf`;
+- `rproxy`, which traefik routes to, with `SWW_RPROXY_UPSTREAM_URL` naming the
+  app and `SWW_RPROXY_WAF_BACKEND` naming `waf`;
 - `waf`, with `SWW_COMMAND=waf` and a volume for its state files.
 
 Each further app adds its own container and its own `rproxy`, whose
 `SWW_RPROXY_WAF_BACKEND` names the same `waf` (see "Sites"). Anything on that
 network can reach an app around `rproxy`, and can ask `waf` questions or send it
-reports. So the network holds only what you trust. "The reverse proxy in front"
-in `README.md` lists what that proxy must do.
+reports. So the network holds only what you trust.
 
 ### `rproxy` in the app's container
 
@@ -2908,10 +2903,10 @@ process that stops at start is started again every second, not in a tight loop.
   container's output, which `docker logs` shows.
 - When a service exits, runit runs its `run` script again, and the container
   does not stop. Beside an app, the other process keeps running.
-- While `rproxy` is down, the reverse proxy in front answers `502`. While the
-  app is down, `rproxy` answers `502`, which is not a client offence. While
-  `waf` is down, `SWW_RPROXY_WAF_BACKEND_NO_ANSWER_DISPOSITION` decides (see
-  "Without an answer from `waf`").
+- While `rproxy` is down, traefik answers `502`. While the app is down, `rproxy`
+  answers `502`, which is not a client offence. While `waf` is down,
+  `SWW_RPROXY_WAF_BACKEND_NO_ANSWER_DISPOSITION` decides (see "Without an answer
+  from `waf`").
 - Only faults found at start stop `waf`:
     - invalid configuration;
     - a lookup database that is missing or unreadable;
@@ -2944,26 +2939,23 @@ checks:
 - With `SWW_COMMAND` unset, empty or `rproxy`, it passes while `rproxy` answers
   that request on `127.0.0.1`, at the port in `SWW_RPROXY_LISTEN_ADDR`, and the
   app accepts connections at `SWW_RPROXY_UPSTREAM_URL`. It does not ask `waf`,
-  so that the site stays in the routes of the reverse proxy in front while `waf`
-  is down.
+  so that the site stays in traefik's routes while `waf` is down.
 - With any other value, it fails with a message naming `SWW_COMMAND` and its
   value.
 
-A reverse proxy that reads Docker's health status, as traefik does, sends a
-container no requests until it shows as healthy. So the check runs every second
-from the container's start until it first passes, for up to a minute, and every
-30 seconds after that. An app with a health check of its own can replace the
-image's `HEALTHCHECK` with one that checks both.
+traefik sends a container no requests until it shows as healthy. So the check
+runs every second from the container's start until it first passes, for up to a
+minute, and every 30 seconds after that. An app with a health check of its own
+can replace the image's `HEALTHCHECK` with one that checks both.
 
 ### Ports
 
 `rproxy` listens on no port but the one in `SWW_RPROXY_LISTEN_ADDR`, 8080 by
-default, which the reverse proxy in front forwards to. It passes requests to the
-app at `SWW_RPROXY_UPSTREAM_URL`, `http://127.0.0.1:8081` by default. Another
-port may be set: the health check follows the setting, and the reverse proxy in
-front must name it, as traefik's port label
-(`traefik.http.services.<name>.loadbalancer.server.port`) does. An app beside
-`rproxy` must leave that port free.
+default, which traefik routes to. It passes requests to the app at
+`SWW_RPROXY_UPSTREAM_URL`, `http://127.0.0.1:8081` by default. Another port may
+be set: the health check follows the setting, and traefik's port label
+(`traefik.http.services.<name>.loadbalancer.server.port`) must name it. An app
+beside `rproxy` must leave that port free.
 
 `waf` listens on no port but the one in `SWW_WAF_LISTEN_ADDR`, 8080 by default.
 The questions, the reports and its own endpoints all come there.
@@ -3004,9 +2996,9 @@ in `docs/settings.md` shows the commands.
 ### Forwarded headers
 
 The app's TCP peer is `rproxy`, and the `X-Forwarded-For` the app receives ends
-with the address of the reverse proxy in front, which `rproxy` adds after the
-visitor's. So the app must trust `rproxy`'s address for forwarded headers,
-besides the private address ranges that proxy is on.
+with traefik's address, which `rproxy` adds after the visitor's. So the app must
+trust `rproxy`'s address for forwarded headers, besides the private address
+ranges traefik is on.
 
 - In a container of its own, `rproxy`'s address is on the private network. The
   usual default for a trusted-proxy setting, the private ranges, covers it.

docs/request-log.md

@@ -68,8 +68,7 @@ Every one of these lines has:
 ### Request detail
 
 - `request_id`: the id `rproxy` made for the request when it arrived.
-- `peer_ip`: the address of `rproxy`'s TCP peer, normally the reverse proxy in
-  front of it.
+- `peer_ip`: the address of `rproxy`'s TCP peer, normally traefik.
 - `forwarded_for`: the `X-Forwarded-For` header the visitor's request came with.
 - `client_group`: the client as rate limiting counts it, an IPv4 address as a
   /32 or an IPv6 group.

docs/settings.md

@@ -478,9 +478,9 @@ start, and an invalid value stops the start. Any can be given as a file, as a
   `10.0.0.0/8,172.16.0.0/12,192.168.0.0/16`): the netblocks whose
   `X-Forwarded-For` is believed. Set but empty, it trusts nothing. A client
   outside these netblocks cannot claim another address. Where other containers
-  reach `rproxy` directly, or visitors on a private network reach the reverse
-  proxy in front, they can name another address in the header. Narrowing the
-  setting to that proxy's own network closes that gap.
+  reach `rproxy` directly, or visitors on a private network reach traefik, they
+  can name another address in the header. Narrowing the setting to traefik's own
+  network closes that gap.
 
 ## The image's command