Merge pull request 'Spec: SPEC.md and README.md to sneak's rulings, deploy model, milestones and Ubuntu base' (#31) from next into main

Reviewed-on: #31
2026-09-29 02:57:57 +02:00
parent a0d2c21346 b821897db8
commit fd77e76177
2 changed files with 1551 additions and 598 deletions
@@ -1,11 +1,14 @@
 # smallwebwaf
 `smallwebwaf` is a simple, fast, logging web application firewall for people who
-host their own services. It is one small container that sits between your
+host their own services. It runs inside the container of the one application it
-reverse proxy (traefik) and one application: traefik points at `smallwebwaf`,
+protects, between your reverse proxy (traefik) and the app: the app's Dockerfile
-and `smallwebwaf` points at the app. It is configured with environment
+builds `FROM` the `smallwebwaf` image, traefik sends the app's requests to
-variables, keeps its state in memory, and writes a detailed JSON log line for
+`smallwebwaf` on port 8080, and `smallwebwaf` passes them on to the app on
-every request.
+`127.0.0.1:8081`. It needs no setting, and protects the app from the first
 request with defaults chosen for a service on the open internet. It keeps its
 state in memory and in JSON files you can read and edit, and writes a detailed
 JSON log line for every request.
 Status: design stage. This repository currently holds the documents only; no
 code has been written. The design is in [`SPEC.md`](SPEC.md), and the survey of
@@ -17,73 +20,106 @@ Small self-hosted sites now receive a great deal of traffic nobody asked for:
 scrapers that ignore `robots.txt` and crawl every commit of every repository on
 a public git server, vulnerability scanners walking through lists of WordPress
 and `.env` paths, and credential-guessing bots. Most of it comes from a small
-number of hosting networks and countries. A single-person operation has no
+number of hosting networks and countries. A single-person operation has no abuse
-abuse desk and no CDN contract; it needs something small that can be put in
+desk and no CDN contract; it needs something small that can be put in front of
-front of one service and left alone.
+one service and left alone.
 The existing tools each solve part of this. Rule-based firewalls catch attack
-payloads but do not limit request rates. Rate limiters count requests but
+payloads but do not limit request rates. Rate limiters count requests but cannot
-cannot tell a residential visitor from a rented server farm. The products that
+tell a residential visitor from a rented server farm. The products that do most
-do most of it want several containers, a database and a web console. None of
+of it want several containers, a database and a web console. None of them can
-them can say "clients from these networks get half the normal allowance", which
+say "clients from these networks are banned after half as many requests as
-is the most useful thing to be able to say when nearly all abuse comes from a
+anyone else", which is the most useful thing to be able to say when nearly all
-known list of AS numbers. [`EVALUATION.md`](EVALUATION.md) goes through the
+abuse comes from a known list of AS numbers. [`EVALUATION.md`](EVALUATION.md)
-candidates one by one.
+goes through the candidates one by one.
 `smallwebwaf` is meant to fill that gap:
 - protect a service from misbehaving scrapers and scanners with per-client
  request and byte limits over a minute, an hour and a day;
- bias those limits against the countries and AS numbers that abuse commonly
+- lower those limits for the countries and AS numbers that abuse commonly comes
-  comes from, so their clients get a configured percentage of the normal
+  from, so their clients are banned after fewer requests than others;
-  allowance rather than an outright block;
+- ban abusers: briefly at first, longer each time they come back, and
- remember abusers, block them for a while, and ban the ones who keep coming
+  permanently when they keep at it; a scanner's first probe bans it for seven
-  back;
+  days;
 - log everything in a form that is easy to search and ship elsewhere;
- stay small enough to understand: one binary, one container, environment
+- stay small enough to understand: one binary in the app's own container,
-  variables, no database.
+  environment variables, no database, no required setting.
 ## Proposed features
- Reverse proxy for one upstream application, streaming in both directions,
+- Reverse proxy for one application, streaming in both directions, with
-  with WebSocket support. One `smallwebwaf` per app.
+  WebSocket support. One `smallwebwaf` per app, inside the app's own container.
- Real client address worked out from `X-Forwarded-For`, trusting only the
+- Internet-ready out of the box: no setting is required, and every setting has a
-  proxy networks you list. IPv6 clients are counted by /64.
+  default chosen for a service facing the internet in 2026. Every setting's name
  starts with `SWWAF_`, so that it cannot clash with the app's own.
 - Real client address worked out from `X-Forwarded-For`, trusting only the proxy
  networks you list, by default the private address ranges. IPv6 clients are
  counted by /64 by default.
 - Size and time limits on requests and responses, with the time limits both
  between the client and `smallwebwaf` and between `smallwebwaf` and the app: by
  default a request may take 60 seconds and 100 MiB, a response 30 minutes and 5
  GiB.
 - Rate limits per client on requests per minute, per hour and per day, and on
-  bytes per minute, per hour and per day.
+  bytes per minute, per hour and per day, on by default and set well above what
  real visitors need.
 - Netblocks that bypass rate limiting, netblocks that bypass everything, and
  netblocks that are always refused.
- AS number and country lookup for every client from a local database file.
+- AS number and country lookup for every client, on by default through the free
  GeoJS web service, which is sent the address of every new visitor. The IPinfo
  Lite database file, which you download and mount, can be used instead, or
  lookups switched off (see "Country and AS number lookup" below).
 - Country lists: `SWWAF_DENIED_COUNTRIES` refuses every request from the
  countries listed, `SWWAF_EXCLUSIVELY_ALLOWED_COUNTRIES` every request from
  anywhere else. Such a request gets the answer a banned client gets as soon as
  the client's address has been looked up, before its body is read and without
  the rule files or the Core Rule Set looking at it, and no ban is made.
 - Biased limits: listed AS numbers and countries get a percentage of every
-  limit, for example 50 percent for common abuse-source networks. Zero percent
+  limit, for example 50 percent for common abuse-source networks, so their
-  refuses outright.
+  clients are banned after fewer requests. Zero percent is a zero allowance: the
  first request breaks the limit and bans the client.
 - Attack detection:
    - a directory of plain text rule files, one regex per line, for catching
-      scanning and penetration probes; easy to edit by hand;
+      scanning and penetration probes; easy to edit by hand, and picked up while
-    - the OWASP Core Rule Set, run by the Coraza engine, in detect-only or
+      running;
-      blocking mode;
+    - the OWASP Core Rule Set, run by the Coraza engine, refusing the requests
-    - trap paths and bursts of error responses.
+      it flags; it reads the URL and headers of every request, and request
- Offences add up to a temporary block. Block lengths grow for repeat offenders
+      bodies only once you switch that on, since on a code forge they are full
-  (for example one hour, then a day, then a week) and end in a permanent ban.
+      of code it would take for attacks;
    - trap paths, and a ban for a client that the rule files or the Core Rule
      Set refuse again and again.
 - Bans:
    - a clear sign of attack, such as a probe for a `.env` file or a scanner's
      user agent, bans for seven days on the first request, and any further
      request during those days makes the ban permanent;
    - breaking a limit bans for an hour; breaking one again within a day of a
      ban ending triples the length, and a ban that would last longer than seven
      days is permanent instead;
    - every ban carries notes on why it was made, to help decide whether to lift
      it.
 - IP reputation: downloadable blocklists, DNS blocklists, AbuseIPDB, and an
  optional feed of decisions from a CrowdSec engine. Lookups happen in the
-  background and never delay a request.
+  background and never delay a request. None is on until you add it.
 - Alerts on attacks and bans to a generic webhook, Slack or ntfy, with a
  cooldown and an hourly cap so a wide attack cannot flood the channel.
- Anomaly alerts when requests or bytes per minute or hour cross a threshold,
+- Anomaly alerts when requests or bytes per minute or hour cross a threshold you
-  for a single client, its surrounding netblock, an AS number, a named netblock
+  set, for a single client, its surrounding netblock, an AS number, a named
-  or the whole service.
+  netblock or the whole service.
- Observe mode: log and alert on every decision while refusing nothing, for the
+- Observe mode: log and alert on every decision while refusing nothing.
  first days in front of a new service.
 - Request log: one JSON object per request on stdout with the usual web log
  fields, the decision taken and why, AS number and country, and timings.
-  Optionally also sent to a remote syslog or RELP endpoint.
+  Optionally also sent to a remote syslog server.
- Prometheus metrics on their own port.
+- Prometheus metrics, for a scraper that holds the metrics token.
- State (bans, offender history, hour and day counters, reputation cache) held
+- State (bans with their notes, each client's counters and history, the GeoJS
-  in memory and saved as readable, hand-editable JSON files, written atomically.
+  answers, the reputation cache, the alerting state) held in memory and kept in
-  Nothing is read from disk while serving a request.
+  readable JSON files, written regularly and at every stop, so a restart loses
- A small admin endpoint for health checks, listing, adding and lifting bans,
+  nothing. Edit a file, or add a rule file, and the running `smallwebwaf` picks
-  and asking why a given address was refused.
+  up the change. Nothing is read from disk while serving a request.
 - Health checks, the metrics, and listing, adding and lifting bans or asking why
  a given address was refused, all on the one port every request uses: under
  `/_smallwebwaf/` on the app's own address, through traefik like any other
  request. The metrics need the metrics token, and the ban management the admin
  token.
 Not planned: TLS termination, routing for several apps, browser challenges
 (captcha or proof of work), a web console, or defence against floods large
@@ -96,54 +132,122 @@ For each request `smallwebwaf`:
 - works out who the client really is;
 - lets it straight through if it is on the bypass list, refuses it if it is on
  the deny list or currently banned;
- looks up its AS number and country, and any cached reputation verdict;
+- looks up its AS number and country, and refuses it if that country is denied,
  or is not among the only ones allowed;
 - checks for a cached reputation verdict;
 - picks the client's limit percentage from those;
- checks the minute, hour and day request counters against the limits, and
+- checks the minute, hour and day request counters against the limits, and bans
-  answers 429 if one is exceeded;
+  the client if it breaks one;
- checks the request against the rule files and the Core Rule Set;
+- checks the request against the rule files and the Core Rule Set, and bans the
- forwards it to the app and streams the response back;
+  client at once for a clear sign of attack;
- counts the bytes, records any offence, bans the client if it has collected
+- forwards it to the app and streams the response back, within the size and time
-  enough of them, sends any alerts that are due, and writes the log line.
+  limits;
 - counts the bytes and any refusal by the rule files or the Core Rule Set, bans
  the client if it broke a limit, updates its history, sends any alerts that are
  due, and writes the log line.
-A minimal deployment beside an app in docker-compose:
+A minimal deployment is the app's own Dockerfile, built on the `smallwebwaf`
 image, with no setting. That image is built on Ubuntu 26.04 LTS, the newest
 long-term support release of Ubuntu, pinned by digest, and moves to the next one
 when it ships. It has nixpkgs installed, so the app adds the packages it needs
 from nixpkgs. Beyond its `FROM` line the app's Dockerfile adds the app's binary,
 any packages it needs, and the app's runit service, which starts the app as a
 user of its own, listening on `127.0.0.1:8081`:
-```yaml
+```dockerfile
-services:
+# The smallwebwaf image, pinned by digest.
-    app:
+FROM <registry>/smallwebwaf:<pinned digest>
        image: example/app
        networks: [internal]
-    waf:
+# Packages the app needs, if any, from the nixpkgs in the image.
-        image: <registry>/smallwebwaf:<pinned digest>
+RUN nix-env -iA nixpkgs.git
-        environment:
+
-            UPSTREAM_URL: http://app:3000
+# The app's binary, and a user of its own to run it.
-            TRUSTED_PROXIES: 172.18.0.0/16
+COPY app /usr/local/bin/app
-            MODE: observe
+RUN useradd --system --no-create-home --shell /usr/sbin/nologin app
-            RATE_LIMIT_PER_MINUTE: "120"
+
-            RATE_LIMIT_PER_HOUR: "2000"
+# The app's runit service.
-            RATE_LIMIT_PER_DAY: "10000"
+COPY --chmod=755 app.run /etc/service/app/run
            ASN_LIMIT_PERCENT: AS14061:50,AS16276:50
            COUNTRY_LIMIT_PERCENT: CN:25
            ALERT_NTFY_URL: https://ntfy.example.invalid/alerts
        volumes: [waf-state:/data]
        networks: [internal, traefik]
        labels:
            traefik.enable: "true"
            traefik.http.routers.app.rule: Host(`app.example.invalid`)
            traefik.http.services.app.loadbalancer.server.port: "8080"
 ```
-A rule file is one rule per line: a name, what to match against, what to do,
+with `app.run` beside the Dockerfile, where `--listen` and `--trusted-proxies`
-and a regex.
+stand for the app's own options:
-```
+```bash
-env-file       path        offence:3  (?i)/\.env(\.[a-z]+)?$
+#!/usr/bin/env bash
-scanner-agent  user_agent  ban        (?i)\b(sqlmap|nikto|nuclei|wpscan)\b
+set -euo pipefail
 sleep 1
 exec chpst -u app:app /usr/local/bin/app \
    --listen 127.0.0.1:8081 \
    --trusted-proxies 10.0.0.0/8,172.16.0.0/12,192.168.0.0/16,127.0.0.1/32,::1/128
 ```
-[`SPEC.md`](SPEC.md) has the full design: every environment variable, the rule
+- The image's entrypoint, `runsvinit`, has runit start `smallwebwaf` and the app
-file format, the state files, the log fields, the metrics, failure behaviour,
+  side by side, each as its own user, and start either again a second after it
-the build order, and the design questions still open for the owner.
+  exits. Leave out `ENTRYPOINT` and `USER` from the app's Dockerfile.
 - `nix-env -iA nixpkgs.<name>` installs a package from the nixpkgs in the image,
  and the app finds it on its `PATH`. That nixpkgs is fixed at one commit, so
  the same `smallwebwaf` image always gives the app the same packages; newer
  ones come with a newer `smallwebwaf` image.
 - Deploy it as you deploy any app, with traefik's labels on this one container
  pointing at port 8080. upaas needs no change for this.
 - The app has to trust `127.0.0.1` and `::1` for forwarded headers, besides the
  private address ranges, since the requests it gets now come from `smallwebwaf`
  on loopback. An app left at the usual default, the private ranges alone, sees
  every visitor as `127.0.0.1`.
 - Port 8080 is the only one the app must leave free: the health check, the
  metrics and ban management are all on it, under `/_smallwebwaf/`. The image's
  health check passes while `smallwebwaf` answers and the app accepts
  connections.
 - `smallwebwaf` keeps its state files in `/var/lib/smallwebwaf`. Mount a volume
  there to keep bans and client history when a deploy replaces the container;
  without one, it still starts.
 A rule file is one rule per line: a name, what to match against, what to do, and
 a regex.
 ```
 env-file       path        ban  (?i)^/\.env(\.[a-z]+)?$
 scanner-agent  user_agent  ban  (?i)\b(sqlmap|nikto|nuclei|wpscan)\b
 ```
 [`SPEC.md`](SPEC.md) has the full design: the deployment, every environment
 variable, the ban rules, the rule file format, the state files, the log fields,
 the metrics, failure behaviour and the build order.
 ## Country and AS number lookup
 `smallwebwaf` looks up the AS number and country of every client, for the
 request log, the metrics and the ban notes, and for the country lists and biased
 limits when you set them. It works with no setup: by default it asks the free
 GeoJS web service, which needs no account and no file. This means that, by
 default, the address of every new visitor is sent to GeoJS. Each answer is kept
 in memory for seven days, and many addresses are asked about in one request;
 writing the answers to disk, so that they survive a restart, comes in milestone
 3 or later. GeoJS publishes no rate limit but may block a caller it thinks asks
 too much; while it is not answering, new visitors count as coming from an
 unknown country, which `SWWAF_EXCLUSIVELY_ALLOWED_COUNTRIES` refuses.
 To keep your visitors' addresses on your own host, set
 `SWWAF_LOOKUP_SOURCE=off`, or use the database file instead of GeoJS:
 `SWWAF_LOOKUP_SOURCE=file` reads the free IPinfo Lite database
 (`ipinfo_lite.mmdb`). You download it with your own IPinfo account, mount the
 directory that holds it into the container, point `SWWAF_LOOKUP_DB_PATH` at the
 file and refresh it when you choose; `smallwebwaf` never downloads it itself,
 and reads it again when you replace it. It has to be the directory rather than
 the file itself: docker does not show a single mounted file being replaced, so a
 refresh would go unseen. IPinfo releases it under the Creative Commons
 Attribution-ShareAlike 4.0 International License and asks for attribution, in
 its own words on https://ipinfo.io/lite: "The attribution requirements can be
 met by giving our service credit as your data source. Simply place a link to
 IPinfo on the website, application, or social media account that uses our data."
 Its example of such a credit is a link mentioning "IP address data is powered by
 IPinfo". A service that uses the database through `smallwebwaf` should carry
 that link.
 Neither source can place a private address, so a client on one, such as a
 visitor on your local network, another container or your monitoring, has no
 country: `SWWAF_EXCLUSIVELY_ALLOWED_COUNTRIES` refuses it unless you list it in
 `SWWAF_ALLOW_NETS`. Such addresses are never sent to GeoJS.
 ## Documents