README.md

@@ -3,9 +3,10 @@
 `smallwebwaf` is a simple, fast, logging web application firewall for people who
 host their own services. It is one small container that sits between your
 reverse proxy (traefik) and one application: traefik points at `smallwebwaf`,
-and `smallwebwaf` points at the app. It is configured with environment
-variables, keeps its state in memory, and writes a detailed JSON log line for
-every request.
+and `smallwebwaf` points at the app. It needs one setting, the address of the
+app, and protects the app from the first request with defaults chosen for a
+service on the open internet. It keeps its state in memory and in JSON files you
+can read and edit, and writes a detailed JSON log line for every request.
 
 Status: design stage. This repository currently holds the documents only; no
 code has been written. The design is in [`SPEC.md`](SPEC.md), and the survey of
@@ -17,71 +18,89 @@ Small self-hosted sites now receive a great deal of traffic nobody asked for:
 scrapers that ignore `robots.txt` and crawl every commit of every repository on
 a public git server, vulnerability scanners walking through lists of WordPress
 and `.env` paths, and credential-guessing bots. Most of it comes from a small
-number of hosting networks and countries. A single-person operation has no
-abuse desk and no CDN contract; it needs something small that can be put in
-front of one service and left alone.
+number of hosting networks and countries. A single-person operation has no abuse
+desk and no CDN contract; it needs something small that can be put in front of
+one service and left alone.
 
 The existing tools each solve part of this. Rule-based firewalls catch attack
-payloads but do not limit request rates. Rate limiters count requests but
-cannot tell a residential visitor from a rented server farm. The products that
-do most of it want several containers, a database and a web console. None of
-them can say "clients from these networks get half the normal allowance", which
-is the most useful thing to be able to say when nearly all abuse comes from a
-known list of AS numbers. [`EVALUATION.md`](EVALUATION.md) goes through the
-candidates one by one.
+payloads but do not limit request rates. Rate limiters count requests but cannot
+tell a residential visitor from a rented server farm. The products that do most
+of it want several containers, a database and a web console. None of them can
+say "clients from these networks are banned after half as many requests as
+anyone else", which is the most useful thing to be able to say when nearly all
+abuse comes from a known list of AS numbers. [`EVALUATION.md`](EVALUATION.md)
+goes through the candidates one by one.
 
 `smallwebwaf` is meant to fill that gap:
 
 - protect a service from misbehaving scrapers and scanners with per-client
   request and byte limits over a minute, an hour and a day;
-- bias those limits against the countries and AS numbers that abuse commonly
-  comes from, so their clients get a configured percentage of the normal
-  allowance rather than an outright block;
-- remember abusers, block them for a while, and ban the ones who keep coming
-  back;
+- lower those limits for the countries and AS numbers that abuse commonly comes
+  from, so their clients are banned after fewer requests than others;
+- ban abusers: briefly at first, longer each time they come back, and
+  permanently when they keep at it; a scanner's first probe bans it for seven
+  days;
 - log everything in a form that is easy to search and ship elsewhere;
 - stay small enough to understand: one binary, one container, environment
-  variables, no database.
+  variables, no database, one required setting.
 
 ## Proposed features
 
-- Reverse proxy for one upstream application, streaming in both directions,
-  with WebSocket support. One `smallwebwaf` per app.
-- Real client address worked out from `X-Forwarded-For`, trusting only the
-  proxy networks you list. IPv6 clients are counted by /64.
+- Reverse proxy for one upstream application, streaming in both directions, with
+  WebSocket support. One `smallwebwaf` per app.
+- Internet-ready out of the box: only `UPSTREAM_URL` must be set, and every
+  other setting has a default chosen for a service facing the internet in 2026.
+- Real client address worked out from `X-Forwarded-For`, trusting only the proxy
+  networks you list, by default the private address ranges. IPv6 clients are
+  counted by /64.
+- Size and time limits on requests and responses, both between the client and
+  `smallwebwaf` and between `smallwebwaf` and the app: by default a request may
+  take 60 seconds and 100 MB, a response 30 minutes and 5 GB.
 - Rate limits per client on requests per minute, per hour and per day, and on
-  bytes per minute, per hour and per day.
+  bytes per minute, per hour and per day, on by default and set well above what
+  real visitors need.
 - Netblocks that bypass rate limiting, netblocks that bypass everything, and
   netblocks that are always refused.
-- AS number and country lookup for every client from a local database file.
+- AS number and country lookup for every client from the IPinfo Lite database
+  file, which you download and mount (see "Lookup database" below).
 - Biased limits: listed AS numbers and countries get a percentage of every
-  limit, for example 50 percent for common abuse-source networks. Zero percent
-  refuses outright.
+  limit, for example 50 percent for common abuse-source networks, so their
+  clients are banned after fewer requests. Zero percent is a zero allowance: the
+  first request breaks the limit and bans the client.
 - Attack detection:
     - a directory of plain text rule files, one regex per line, for catching
-      scanning and penetration probes; easy to edit by hand;
-    - the OWASP Core Rule Set, run by the Coraza engine, in detect-only or
-      blocking mode;
+      scanning and penetration probes; easy to edit by hand, and picked up while
+      running;
+    - the OWASP Core Rule Set, run by the Coraza engine, refusing the requests
+      it flags;
     - trap paths and bursts of error responses.
-- Offences add up to a temporary block. Block lengths grow for repeat offenders
-  (for example one hour, then a day, then a week) and end in a permanent ban.
+- Bans:
+    - a clear sign of attack, such as a probe for a `.env` file or a scanner's
+      user agent, bans for seven days on the first request, and any further
+      request during those days makes the ban permanent;
+    - breaking a limit bans for an hour; breaking one again within a day of a
+      ban ending triples the length, and a ban that would last longer than seven
+      days is permanent instead;
+    - every ban carries notes on why it was made, to help decide whether to lift
+      it.
 - IP reputation: downloadable blocklists, DNS blocklists, AbuseIPDB, and an
   optional feed of decisions from a CrowdSec engine. Lookups happen in the
-  background and never delay a request.
+  background and never delay a request. None is on until you add it.
 - Alerts on attacks and bans to a generic webhook, Slack or ntfy, with a
   cooldown and an hourly cap so a wide attack cannot flood the channel.
-- Anomaly alerts when requests or bytes per minute or hour cross a threshold,
-  for a single client, its surrounding netblock, an AS number, a named netblock
-  or the whole service.
-- Observe mode: log and alert on every decision while refusing nothing, for the
-  first days in front of a new service.
+- Anomaly alerts when requests or bytes per minute or hour cross a threshold you
+  set, for a single client, its surrounding netblock, an AS number, a named
+  netblock or the whole service.
+- Observe mode: log and alert on every decision while refusing nothing.
 - Request log: one JSON object per request on stdout with the usual web log
   fields, the decision taken and why, AS number and country, and timings.
-  Optionally also sent to a remote syslog or RELP endpoint.
+  Optionally also sent to a remote syslog server.
 - Prometheus metrics on their own port.
-- State (bans, offender history, hour and day counters, reputation cache) held
-  in memory and saved as readable, hand-editable JSON files, written atomically.
-  Nothing is read from disk while serving a request.
+- State (bans with their notes, each client's counters and history, the
+  reputation cache) held in memory and kept in readable JSON files that always
+  hold a copy of it, so a restart loses nothing. Edit a file, or add a rule
+  file, and the running `smallwebwaf` picks up the change. Nothing is read from
+  disk while serving a request.
 - A small admin endpoint for health checks, listing, adding and lifting bans,
   and asking why a given address was refused.
 
@@ -98,14 +117,17 @@ For each request `smallwebwaf`:
   the deny list or currently banned;
 - looks up its AS number and country, and any cached reputation verdict;
 - picks the client's limit percentage from those;
-- checks the minute, hour and day request counters against the limits, and
-  answers 429 if one is exceeded;
-- checks the request against the rule files and the Core Rule Set;
-- forwards it to the app and streams the response back;
-- counts the bytes, records any offence, bans the client if it has collected
-  enough of them, sends any alerts that are due, and writes the log line.
+- checks the minute, hour and day request counters against the limits, and bans
+  the client if it breaks one;
+- checks the request against the rule files and the Core Rule Set, and bans the
+  client at once for a clear sign of attack;
+- forwards it to the app and streams the response back, within the size and time
+  limits;
+- counts the bytes and any error response, bans the client if it broke a limit,
+  updates its history, sends any alerts that are due, and writes the log line.
 
-A minimal deployment beside an app in docker-compose:
+A minimal deployment beside an app in docker-compose. `UPSTREAM_URL` is the only
+setting:
 
 ```yaml
 services:
@@ -117,33 +139,50 @@ services:
         image: <registry>/smallwebwaf:<pinned digest>
         environment:
             UPSTREAM_URL: http://app:3000
-            TRUSTED_PROXIES: 172.18.0.0/16
-            MODE: observe
-            RATE_LIMIT_PER_MINUTE: "120"
-            RATE_LIMIT_PER_HOUR: "2000"
-            RATE_LIMIT_PER_DAY: "10000"
-            ASN_LIMIT_PERCENT: AS14061:50,AS16276:50
-            COUNTRY_LIMIT_PERCENT: CN:25
-            ALERT_NTFY_URL: https://ntfy.example.invalid/alerts
         volumes: [waf-state:/data]
         networks: [internal, traefik]
         labels:
             traefik.enable: "true"
             traefik.http.routers.app.rule: Host(`app.example.invalid`)
             traefik.http.services.app.loadbalancer.server.port: "8080"
+
+volumes:
+    waf-state:
+
+networks:
+    internal:
+    traefik:
+        external: true
 ```
 
-A rule file is one rule per line: a name, what to match against, what to do,
-and a regex.
+The volume keeps bans and client history in a place you choose; without it
+`smallwebwaf` still starts, on a volume docker creates for it.
+
+A rule file is one rule per line: a name, what to match against, what to do, and
+a regex.
 
 ```
-env-file       path        offence:3  (?i)/\.env(\.[a-z]+)?$
+env-file       path        ban  (?i)/\.env(\.[a-z]+)?$
 scanner-agent  user_agent  ban  (?i)\b(sqlmap|nikto|nuclei|wpscan)\b
 ```
 
-[`SPEC.md`](SPEC.md) has the full design: every environment variable, the rule
-file format, the state files, the log fields, the metrics, failure behaviour,
-the build order, and the design questions still open for the owner.
+[`SPEC.md`](SPEC.md) has the full design: every environment variable, the ban
+rules, the rule file format, the state files, the log fields, the metrics,
+failure behaviour and the build order.
+
+## Lookup database
+
+AS number and country lookups, and the biased limits that use them, need the
+free IPinfo Lite database (`ipinfo_lite.mmdb`). You download it with your own
+IPinfo account, mount it into the container, point `LOOKUP_DB_PATH` at it and
+refresh it when you choose; `smallwebwaf` never downloads it itself. IPinfo
+releases it under the Creative Commons Attribution-ShareAlike 4.0 International
+License and asks for attribution, in its own words on https://ipinfo.io/lite:
+"The attribution requirements can be met by giving our service credit as your
+data source. Simply place a link to IPinfo on the website, application, or
+social media account that uses our data." Its example of such a credit is a link
+mentioning "IP address data is powered by IPinfo". A service that uses the
+database through `smallwebwaf` should carry that link.
 
 ## Documents