Add README, SPEC and tool evaluation, closes #1

Initial documents: what smallwebwaf is and why, the proposed feature list, the design spec with the rule file format and the open design questions, and the survey of existing tools. Model: fable-5-1
2026-09-21 07:42:45 +00:00
commit a0d2c21346
3 changed files with 1112 additions and 0 deletions
@@ -0,0 +1,152 @@
+# smallwebwaf
+
+`smallwebwaf` is a simple, fast, logging web application firewall for people who
+host their own services. It is one small container that sits between your
+reverse proxy (traefik) and one application: traefik points at `smallwebwaf`,
+and `smallwebwaf` points at the app. It is configured with environment
+variables, keeps its state in memory, and writes a detailed JSON log line for
+every request.
+
+Status: design stage. This repository currently holds the documents only; no
+code has been written. The design is in [`SPEC.md`](SPEC.md), and the survey of
+existing tools that led to it is in [`EVALUATION.md`](EVALUATION.md).
+
+## Why
+
+Small self-hosted sites now receive a great deal of traffic nobody asked for:
+scrapers that ignore `robots.txt` and crawl every commit of every repository on
+a public git server, vulnerability scanners walking through lists of WordPress
+and `.env` paths, and credential-guessing bots. Most of it comes from a small
+number of hosting networks and countries. A single-person operation has no
+abuse desk and no CDN contract; it needs something small that can be put in
+front of one service and left alone.
+
+The existing tools each solve part of this. Rule-based firewalls catch attack
+payloads but do not limit request rates. Rate limiters count requests but
+cannot tell a residential visitor from a rented server farm. The products that
+do most of it want several containers, a database and a web console. None of
+them can say "clients from these networks get half the normal allowance", which
+is the most useful thing to be able to say when nearly all abuse comes from a
+known list of AS numbers. [`EVALUATION.md`](EVALUATION.md) goes through the
+candidates one by one.
+
+`smallwebwaf` is meant to fill that gap:
+
+- protect a service from misbehaving scrapers and scanners with per-client
+  request and byte limits over a minute, an hour and a day;
+- bias those limits against the countries and AS numbers that abuse commonly
+  comes from, so their clients get a configured percentage of the normal
+  allowance rather than an outright block;
+- remember abusers, block them for a while, and ban the ones who keep coming
+  back;
+- log everything in a form that is easy to search and ship elsewhere;
+- stay small enough to understand: one binary, one container, environment
+  variables, no database.
+
+## Proposed features
+
+- Reverse proxy for one upstream application, streaming in both directions,
+  with WebSocket support. One `smallwebwaf` per app.
+- Real client address worked out from `X-Forwarded-For`, trusting only the
+  proxy networks you list. IPv6 clients are counted by /64.
+- Rate limits per client on requests per minute, per hour and per day, and on
+  bytes per minute, per hour and per day.
+- Netblocks that bypass rate limiting, netblocks that bypass everything, and
+  netblocks that are always refused.
+- AS number and country lookup for every client from a local database file.
+- Biased limits: listed AS numbers and countries get a percentage of every
+  limit, for example 50 percent for common abuse-source networks. Zero percent
+  refuses outright.
+- Attack detection:
+    - a directory of plain text rule files, one regex per line, for catching
+      scanning and penetration probes; easy to edit by hand;
+    - the OWASP Core Rule Set, run by the Coraza engine, in detect-only or
+      blocking mode;
+    - trap paths and bursts of error responses.
+- Offences add up to a temporary block. Block lengths grow for repeat offenders
+  (for example one hour, then a day, then a week) and end in a permanent ban.
+- IP reputation: downloadable blocklists, DNS blocklists, AbuseIPDB, and an
+  optional feed of decisions from a CrowdSec engine. Lookups happen in the
+  background and never delay a request.
+- Alerts on attacks and bans to a generic webhook, Slack or ntfy, with a
+  cooldown and an hourly cap so a wide attack cannot flood the channel.
+- Anomaly alerts when requests or bytes per minute or hour cross a threshold,
+  for a single client, its surrounding netblock, an AS number, a named netblock
+  or the whole service.
+- Observe mode: log and alert on every decision while refusing nothing, for the
+  first days in front of a new service.
+- Request log: one JSON object per request on stdout with the usual web log
+  fields, the decision taken and why, AS number and country, and timings.
+  Optionally also sent to a remote syslog or RELP endpoint.
+- Prometheus metrics on their own port.
+- State (bans, offender history, hour and day counters, reputation cache) held
+  in memory and saved as readable, hand-editable JSON files, written atomically.
+  Nothing is read from disk while serving a request.
+- A small admin endpoint for health checks, listing, adding and lifting bans,
+  and asking why a given address was refused.
+
+Not planned: TLS termination, routing for several apps, browser challenges
+(captcha or proof of work), a web console, or defence against floods large
+enough to fill the host's network link.
+
+## How it works, in short
+
+For each request `smallwebwaf`:
+
+- works out who the client really is;
+- lets it straight through if it is on the bypass list, refuses it if it is on
+  the deny list or currently banned;
+- looks up its AS number and country, and any cached reputation verdict;
+- picks the client's limit percentage from those;
+- checks the minute, hour and day request counters against the limits, and
+  answers 429 if one is exceeded;
+- checks the request against the rule files and the Core Rule Set;
+- forwards it to the app and streams the response back;
+- counts the bytes, records any offence, bans the client if it has collected
+  enough of them, sends any alerts that are due, and writes the log line.
+
+A minimal deployment beside an app in docker-compose:
+
+```yaml
+services:
+    app:
+        image: example/app
+        networks: [internal]
+
+    waf:
+        image: <registry>/smallwebwaf:<pinned digest>
+        environment:
+            UPSTREAM_URL: http://app:3000
+            TRUSTED_PROXIES: 172.18.0.0/16
+            MODE: observe
+            RATE_LIMIT_PER_MINUTE: "120"
+            RATE_LIMIT_PER_HOUR: "2000"
+            RATE_LIMIT_PER_DAY: "10000"
+            ASN_LIMIT_PERCENT: AS14061:50,AS16276:50
+            COUNTRY_LIMIT_PERCENT: CN:25
+            ALERT_NTFY_URL: https://ntfy.example.invalid/alerts
+        volumes: [waf-state:/data]
+        networks: [internal, traefik]
+        labels:
+            traefik.enable: "true"
+            traefik.http.routers.app.rule: Host(`app.example.invalid`)
+            traefik.http.services.app.loadbalancer.server.port: "8080"
+```
+
+A rule file is one rule per line: a name, what to match against, what to do,
+and a regex.
+
+```
+env-file       path        offence:3  (?i)/\.env(\.[a-z]+)?$
+scanner-agent  user_agent  ban        (?i)\b(sqlmap|nikto|nuclei|wpscan)\b
+```
+
+[`SPEC.md`](SPEC.md) has the full design: every environment variable, the rule
+file format, the state files, the log fields, the metrics, failure behaviour,
+the build order, and the design questions still open for the owner.
+
+## Documents
+
+- [`SPEC.md`](SPEC.md): the design.
+- [`EVALUATION.md`](EVALUATION.md): what already exists, what each tool covers
+  and misses, and why none was adopted.