diff --git a/README.md b/README.md index 8071582..38bd9b9 100644 --- a/README.md +++ b/README.md @@ -3,9 +3,10 @@ `smallwebwaf` is a simple, fast, logging web application firewall for people who host their own services. It is one small container that sits between your reverse proxy (traefik) and one application: traefik points at `smallwebwaf`, -and `smallwebwaf` points at the app. It is configured with environment -variables, keeps its state in memory, and writes a detailed JSON log line for -every request. +and `smallwebwaf` points at the app. It needs one setting, the address of the +app, and protects the app from the first request with defaults chosen for a +service on the open internet. It keeps its state in memory and in JSON files you +can read and edit, and writes a detailed JSON log line for every request. Status: design stage. This repository currently holds the documents only; no code has been written. The design is in [`SPEC.md`](SPEC.md), and the survey of @@ -17,71 +18,89 @@ Small self-hosted sites now receive a great deal of traffic nobody asked for: scrapers that ignore `robots.txt` and crawl every commit of every repository on a public git server, vulnerability scanners walking through lists of WordPress and `.env` paths, and credential-guessing bots. Most of it comes from a small -number of hosting networks and countries. A single-person operation has no -abuse desk and no CDN contract; it needs something small that can be put in -front of one service and left alone. +number of hosting networks and countries. A single-person operation has no abuse +desk and no CDN contract; it needs something small that can be put in front of +one service and left alone. The existing tools each solve part of this. Rule-based firewalls catch attack -payloads but do not limit request rates. Rate limiters count requests but -cannot tell a residential visitor from a rented server farm. The products that -do most of it want several containers, a database and a web console. None of -them can say "clients from these networks get half the normal allowance", which -is the most useful thing to be able to say when nearly all abuse comes from a -known list of AS numbers. [`EVALUATION.md`](EVALUATION.md) goes through the -candidates one by one. +payloads but do not limit request rates. Rate limiters count requests but cannot +tell a residential visitor from a rented server farm. The products that do most +of it want several containers, a database and a web console. None of them can +say "clients from these networks are banned after half as many requests as +anyone else", which is the most useful thing to be able to say when nearly all +abuse comes from a known list of AS numbers. [`EVALUATION.md`](EVALUATION.md) +goes through the candidates one by one. `smallwebwaf` is meant to fill that gap: - protect a service from misbehaving scrapers and scanners with per-client request and byte limits over a minute, an hour and a day; -- bias those limits against the countries and AS numbers that abuse commonly - comes from, so their clients get a configured percentage of the normal - allowance rather than an outright block; -- remember abusers, block them for a while, and ban the ones who keep coming - back; +- lower those limits for the countries and AS numbers that abuse commonly comes + from, so their clients are banned after fewer requests than others; +- ban abusers: briefly at first, longer each time they come back, and + permanently when they keep at it; a scanner's first probe bans it for seven + days; - log everything in a form that is easy to search and ship elsewhere; - stay small enough to understand: one binary, one container, environment - variables, no database. + variables, no database, one required setting. ## Proposed features -- Reverse proxy for one upstream application, streaming in both directions, - with WebSocket support. One `smallwebwaf` per app. -- Real client address worked out from `X-Forwarded-For`, trusting only the - proxy networks you list. IPv6 clients are counted by /64. +- Reverse proxy for one upstream application, streaming in both directions, with + WebSocket support. One `smallwebwaf` per app. +- Internet-ready out of the box: only `UPSTREAM_URL` must be set, and every + other setting has a default chosen for a service facing the internet in 2026. +- Real client address worked out from `X-Forwarded-For`, trusting only the proxy + networks you list, by default the private address ranges. IPv6 clients are + counted by /64. +- Size and time limits on requests and responses, both between the client and + `smallwebwaf` and between `smallwebwaf` and the app: by default a request may + take 60 seconds and 100 MB, a response 30 minutes and 5 GB. - Rate limits per client on requests per minute, per hour and per day, and on - bytes per minute, per hour and per day. + bytes per minute, per hour and per day, on by default and set well above what + real visitors need. - Netblocks that bypass rate limiting, netblocks that bypass everything, and netblocks that are always refused. -- AS number and country lookup for every client from a local database file. +- AS number and country lookup for every client from the IPinfo Lite database + file, which you download and mount (see "Lookup database" below). - Biased limits: listed AS numbers and countries get a percentage of every - limit, for example 50 percent for common abuse-source networks. Zero percent - refuses outright. + limit, for example 50 percent for common abuse-source networks, so their + clients are banned after fewer requests. Zero percent is a zero allowance: the + first request breaks the limit and bans the client. - Attack detection: - a directory of plain text rule files, one regex per line, for catching - scanning and penetration probes; easy to edit by hand; - - the OWASP Core Rule Set, run by the Coraza engine, in detect-only or - blocking mode; + scanning and penetration probes; easy to edit by hand, and picked up while + running; + - the OWASP Core Rule Set, run by the Coraza engine, refusing the requests + it flags; - trap paths and bursts of error responses. -- Offences add up to a temporary block. Block lengths grow for repeat offenders - (for example one hour, then a day, then a week) and end in a permanent ban. +- Bans: + - a clear sign of attack, such as a probe for a `.env` file or a scanner's + user agent, bans for seven days on the first request, and any further + request during those days makes the ban permanent; + - breaking a limit bans for an hour; breaking one again within a day of a + ban ending triples the length, and a ban that would last longer than seven + days is permanent instead; + - every ban carries notes on why it was made, to help decide whether to lift + it. - IP reputation: downloadable blocklists, DNS blocklists, AbuseIPDB, and an optional feed of decisions from a CrowdSec engine. Lookups happen in the - background and never delay a request. + background and never delay a request. None is on until you add it. - Alerts on attacks and bans to a generic webhook, Slack or ntfy, with a cooldown and an hourly cap so a wide attack cannot flood the channel. -- Anomaly alerts when requests or bytes per minute or hour cross a threshold, - for a single client, its surrounding netblock, an AS number, a named netblock - or the whole service. -- Observe mode: log and alert on every decision while refusing nothing, for the - first days in front of a new service. +- Anomaly alerts when requests or bytes per minute or hour cross a threshold you + set, for a single client, its surrounding netblock, an AS number, a named + netblock or the whole service. +- Observe mode: log and alert on every decision while refusing nothing. - Request log: one JSON object per request on stdout with the usual web log fields, the decision taken and why, AS number and country, and timings. - Optionally also sent to a remote syslog or RELP endpoint. + Optionally also sent to a remote syslog server. - Prometheus metrics on their own port. -- State (bans, offender history, hour and day counters, reputation cache) held - in memory and saved as readable, hand-editable JSON files, written atomically. - Nothing is read from disk while serving a request. +- State (bans with their notes, each client's counters and history, the + reputation cache) held in memory and kept in readable JSON files that always + hold a copy of it, so a restart loses nothing. Edit a file, or add a rule + file, and the running `smallwebwaf` picks up the change. Nothing is read from + disk while serving a request. - A small admin endpoint for health checks, listing, adding and lifting bans, and asking why a given address was refused. @@ -98,14 +117,17 @@ For each request `smallwebwaf`: the deny list or currently banned; - looks up its AS number and country, and any cached reputation verdict; - picks the client's limit percentage from those; -- checks the minute, hour and day request counters against the limits, and - answers 429 if one is exceeded; -- checks the request against the rule files and the Core Rule Set; -- forwards it to the app and streams the response back; -- counts the bytes, records any offence, bans the client if it has collected - enough of them, sends any alerts that are due, and writes the log line. +- checks the minute, hour and day request counters against the limits, and bans + the client if it breaks one; +- checks the request against the rule files and the Core Rule Set, and bans the + client at once for a clear sign of attack; +- forwards it to the app and streams the response back, within the size and time + limits; +- counts the bytes and any error response, bans the client if it broke a limit, + updates its history, sends any alerts that are due, and writes the log line. -A minimal deployment beside an app in docker-compose: +A minimal deployment beside an app in docker-compose. `UPSTREAM_URL` is the only +setting: ```yaml services: @@ -117,33 +139,50 @@ services: image: /smallwebwaf: environment: UPSTREAM_URL: http://app:3000 - TRUSTED_PROXIES: 172.18.0.0/16 - MODE: observe - RATE_LIMIT_PER_MINUTE: "120" - RATE_LIMIT_PER_HOUR: "2000" - RATE_LIMIT_PER_DAY: "10000" - ASN_LIMIT_PERCENT: AS14061:50,AS16276:50 - COUNTRY_LIMIT_PERCENT: CN:25 - ALERT_NTFY_URL: https://ntfy.example.invalid/alerts volumes: [waf-state:/data] networks: [internal, traefik] labels: traefik.enable: "true" traefik.http.routers.app.rule: Host(`app.example.invalid`) traefik.http.services.app.loadbalancer.server.port: "8080" + +volumes: + waf-state: + +networks: + internal: + traefik: + external: true ``` -A rule file is one rule per line: a name, what to match against, what to do, -and a regex. +The volume keeps bans and client history in a place you choose; without it +`smallwebwaf` still starts, on a volume docker creates for it. + +A rule file is one rule per line: a name, what to match against, what to do, and +a regex. ``` -env-file path offence:3 (?i)/\.env(\.[a-z]+)?$ -scanner-agent user_agent ban (?i)\b(sqlmap|nikto|nuclei|wpscan)\b +env-file path ban (?i)/\.env(\.[a-z]+)?$ +scanner-agent user_agent ban (?i)\b(sqlmap|nikto|nuclei|wpscan)\b ``` -[`SPEC.md`](SPEC.md) has the full design: every environment variable, the rule -file format, the state files, the log fields, the metrics, failure behaviour, -the build order, and the design questions still open for the owner. +[`SPEC.md`](SPEC.md) has the full design: every environment variable, the ban +rules, the rule file format, the state files, the log fields, the metrics, +failure behaviour and the build order. + +## Lookup database + +AS number and country lookups, and the biased limits that use them, need the +free IPinfo Lite database (`ipinfo_lite.mmdb`). You download it with your own +IPinfo account, mount it into the container, point `LOOKUP_DB_PATH` at it and +refresh it when you choose; `smallwebwaf` never downloads it itself. IPinfo +releases it under the Creative Commons Attribution-ShareAlike 4.0 International +License and asks for attribution, in its own words on https://ipinfo.io/lite: +"The attribution requirements can be met by giving our service credit as your +data source. Simply place a link to IPinfo on the website, application, or +social media account that uses our data." Its example of such a credit is a link +mentioning "IP address data is powered by IPinfo". A service that uses the +database through `smallwebwaf` should carry that link. ## Documents diff --git a/SPEC.md b/SPEC.md index 2fb5ec6..492aca0 100644 --- a/SPEC.md +++ b/SPEC.md @@ -1,20 +1,25 @@ # smallwebwaf SPEC (draft): protective reverse-proxy sidecar -Status: second draft, with the owner's rulings on storage, logging and metrics -applied. Nothing has been built yet. `EVALUATION.md` beside this file explains why no existing tool was -chosen. Points that turn on an owner decision are collected under "Questions for -the owner" and are marked "(open)" where they appear. +Status: third draft, with the owner's rulings to date applied. Nothing has been +built yet. `EVALUATION.md` beside this file explains why no existing tool was +chosen. ## Purpose One small container that sits between traefik and one application container. The traefik router for the public hostname points at the sidecar; the sidecar forwards to the application. The sidecar limits request and byte rates per -client, detects attacks, blocks abusers for a time and bans repeat offenders, -consults IP reputation sources, looks up the AS number and country of each -client, scales its limits for listed AS numbers and countries, and sends alerts. -Everything is configured by environment variables, apart from the attack -detection rules, which are read from a directory of hand-editable text files. +client, bounds the size and duration of every request and response, detects +attacks, bans abusers (briefly at first, for seven days on a clear sign of +attack, permanently when they keep at it), consults IP reputation sources, looks +up the AS number and country of each client, lowers its limits for listed AS +numbers and countries, and sends alerts. + +It is meant to protect an app on the open internet from the first request with +one setting, `UPSTREAM_URL`: every other setting has a default chosen for a +service facing the internet in 2026. Everything is configured by environment +variables, apart from the attack detection rules, which are read from a +directory of hand-editable text files. ## Non-goals @@ -24,170 +29,254 @@ detection rules, which are read from a directory of hand-editable text files. the sidecar and the app. - Defence against traffic floods that saturate the host's network link. That needs help upstream of the host. -- A web UI or a configuration file. Settings are environment variables; the - only files read are the rule files, which hold one regex per line and nothing - more elaborate. -- Sharing ban state between sidecars in the first version (open, see - Questions). +- A web UI or a configuration file. Settings are environment variables. Apart + from its own state files and the lookup database, the only files read are the + rule files, which hold one regex per line and nothing more elaborate. +- Sharing bans between sidecars in the first version: each sidecar keeps its + own. Running one CrowdSec engine per host, which every sidecar would report + its bans to, is reconsidered once real ban volumes are known. Reading a + CrowdSec decision list is supported (see Reputation). ## Architecture - One statically linked Go binary in one container, running as a non-root user, - read-only root filesystem, one writable volume for state. + read-only root filesystem, one writable volume for state. The image declares + that volume (`/data`), so docker supplies an anonymous one when none is + mounted, and a sidecar started with only `UPSTREAM_URL` has somewhere to + write. - Three listeners, of which only the first is routed by traefik: - the proxy listener; - a metrics listener serving Prometheus metrics, which can be switched off; - an admin listener for health and ban management. - Components inside the process: - Client identification: works out the real client IP from - `X-Forwarded-For`, trusting only configured proxy netblocks. - - Lookup: AS number and country from local database files, held in memory. + `X-Forwarded-For`, trusting only the proxy netblocks in `TRUSTED_PROXIES`, + by default the private address ranges. + - Lookup: AS number and country from one database file the operator + supplies, held in memory. - Reputation: static blocklists fetched on a schedule; DNSBL and reputation API queries made in the background and cached. - Counters: per-client request and byte counts per minute, hour and day, - held in memory; the hour and day counts are written out so they survive a - restart. - - Attack detection: regex rules read at start from a directory of plain - text rule files (see "Rule files"), Coraza with the OWASP Core Rule Set, - and simple signals (requests for listed trap paths, bursts of error - responses from the app). - - Ban ledger: offences, active bans and ban history, held in memory and - written to disk as JSON (see "Persistent state"). No database of any kind - is used. + held in memory and written out with the rest of the state, so they survive + a restart. + - Attack detection: regex rules read from a directory of plain text rule + files and read again whenever the files change (see "Rule files"), Coraza + with the OWASP Core Rule Set, and simple signals (requests for listed trap + paths, bursts of error responses). + - Ban ledger: active and past bans with their notes, and each client's + history, held in memory and kept in JSON files that the sidecar watches + for an admin's edits (see "Bans" and "Persistent state"). No database of + any kind is used. - Request log: one JSON object per request on stdout, optionally also sent - to a remote syslog or RELP endpoint (see "Request log"). + to a remote syslog server (see "Request log"). - Metrics: Prometheus counters and gauges on their own listener (see "Metrics endpoint"). - Alerting: a queue with de-duplication feeding webhook, Slack and ntfy senders. - Proxy: the standard library's `net/http/httputil.ReverseProxy`, streaming - in both directions, with WebSocket upgrade support. + in both directions within the size and time limits, with WebSocket upgrade + support. - Proposed libraries (to be checked against the owner's Go dependency defaults before any are added): - standard library for the proxy, HTTP clients, DNS, logging (`log/slog`), state files (`encoding/json`, `os.Rename`); - `github.com/corazawaf/coraza/v3` and `github.com/corazawaf/coraza-coreruleset` for attack detection; - - `github.com/oschwald/maxminddb-golang` to read `.mmdb` lookup databases; + - `github.com/oschwald/maxminddb-golang/v2`, the current major version, to + read the lookup database; + - `github.com/fsnotify/fsnotify` to notice files that are edited or added + while the sidecar runs; - `github.com/prometheus/client_golang` for metrics; - - remote log sending: the standard library's `log/syslog` is frozen and - does not write the current syslog format (RFC 5424), and no widely used Go - RELP client was found, so the library for this is open (see Questions). + - remote log sending is written in this project: the standard library's + `log/syslog` is frozen and writes only the older syslog format, not the + current one (RFC 5424). RELP is not in the first release; it is revisited + against the available packages afterwards. ## Data flow for one request Steps run in this order; the first step that produces a final answer ends -processing. +processing. The size and time limits (see "Configuration surface") apply to the +whole exchange. - Identify the client. - If the TCP peer is inside `TRUSTED_PROXIES`, walk `X-Forwarded-For` from the right and take the first address not inside `TRUSTED_PROXIES`. Otherwise use the TCP peer address and ignore the header. - IPv6 clients are grouped by prefix (`IPV6_GROUP_PREFIX`, default 64) for - counting and banning, because one abuser usually controls a whole /64. + counting and banning, because one abuser usually controls a whole /64. A + client is therefore one IPv4 address or one IPv6 /64. - Static lists. - In `ALLOW_NETS`: skip every check below and forward. Still counted for anomaly alerts. - In `DENY_NETS`: refuse. -- Ban ledger. An active ban: refuse (`BAN_RESPONSE`). -- Lookup AS number and country. Unknown is a valid result and changes nothing. +- Ban ledger. An active ban on the client's netblock: refuse with + `BAN_RESPONSE`. If a clear sign of attack caused the ban, the ban becomes + permanent (see "Bans"). +- Lookup AS number and country, when a lookup database is configured. Unknown is + a valid result and changes nothing. - Reputation. - Address inside a fetched blocklist: apply `BLOCKLIST_ACTION`. - Cached DNSBL or reputation API result: apply `REPUTATION_ACTION`. - No cached result: queue a background query and carry on. A first request is never delayed by a reputation query. - Work out the client's limit percentage: the lowest of the percentages that - apply (AS number, country, reputation). 100 if none applies. 0 means refuse. - (Lowest-wins versus multiplying is open.) + apply (AS number, country, reputation), or 100 if none applies. - Request rate limits. Unless the client is in `RATE_LIMIT_EXEMPT_NETS`, check - the minute, hour and day counters against each configured limit times the - percentage. Over any limit: answer 429 with `Retry-After`, record an offence. -- Rule files. The request is checked against the rules loaded from - `RULES_DIR`, in file name order then line order. Each rule that matches takes - its action: record offences, refuse with 403, ban at once, or only log. - Matching stops at the first rule that refuses or bans. + the minute, hour and day counters against each limit times the percentage. A + request over any of them breaks that limit: it is refused with `BAN_RESPONSE` + and the client is banned (see "Bans"). +- Rule files. The request is checked against the rules loaded from `RULES_DIR`, + in file name order then line order. Each rule that matches takes its action: + only log, refuse with 403, or refuse and ban. Matching stops at the first rule + that refuses or bans. - Core Rule Set inspection of the request (headers, URL, and body up to - `WAF_BODY_LIMIT`). In `block` mode a match at or over the anomaly threshold is - refused with 403 and recorded as an offence; in `detect` mode it is only - logged and alerted. + `WAF_BODY_LIMIT`). In `block` mode, the default, a match at or over the + anomaly threshold is refused with 403; in `detect` mode it is only logged and + alerted. - Forward to `UPSTREAM_URL`, streaming. Add `X-Forwarded-For` and, if enabled, `X-Client-ASN` and `X-Client-Country` for the app's own logs. - After the response. - - Add response bytes (and request body bytes) to the byte counters. Once a - client's byte total passes a byte limit times the percentage, following - requests get 429 until the window moves on, and an offence is recorded. A - response already in progress is not cut off. - - Count 401, 403 and 404 responses from the app per client; passing - `ERROR_BURST_THRESHOLD` within a minute records an offence. - - Update anomaly counters and evaluate alert thresholds. -- Offences and bans. - - `BAN_AFTER_OFFENCES` offences within `OFFENCE_WINDOW` creates a ban. - - Ban length follows `BAN_DURATIONS` by the number of earlier bans for that - client within `BAN_HISTORY_WINDOW`: first ban the first value, second ban - the second value, and so on. - - After `PERMANENT_BAN_AFTER` bans within `BAN_HISTORY_WINDOW`, the ban has - no expiry. - - A request for a path in `TRAP_PATHS` counts as `TRAP_OFFENCE_WEIGHT` - offences at once. + - Add response bytes (and request body bytes) to the byte counters. A client + whose byte total passes a byte limit times the percentage has broken that + limit and is banned. The response in progress is not cut off. + - Count the client's requests answered with 401, 403 or 404, whether the app + gave that answer or the sidecar refused the request after a rule file or + Core Rule Set match. More than `ERROR_BURST_THRESHOLD` of them within a + minute breaks a limit and bans the client. + - Update the client's history and the anomaly counters, and evaluate alert + thresholds. ## Counting method - Each window (minute, hour, day) uses two adjacent fixed buckets per client, - with the previous bucket weighted by how much of it still overlaps the - sliding window. This costs a few integers per client per window and avoids - the burst at bucket boundaries that a single fixed bucket allows. -- Counters are read and updated in memory only. The hour and day counters are - written to disk on a timer and at shutdown and loaded again at start (see - "Persistent state"), so a restart does not hand every client a fresh daily - allowance. Minute counters are not written; a restart forgives at most one - minute of counting. Buckets whose time has passed are discarded on load. + with the previous bucket weighted by how much of it still overlaps the sliding + window. This costs a few integers per client per window and avoids the burst + at bucket boundaries that a single fixed bucket allows. +- Counters are read and updated in memory; a request never waits on the disk. + They are written to `clients.json` every `STATE_COUNTER_INTERVAL` and at + shutdown and loaded again at start (see "Persistent state"), so a restart does + not hand every client a fresh allowance. Buckets whose time has passed are + discarded on load. - Memory is bounded by `MAX_TRACKED_CLIENTS`; when full, the least recently seen - clients with no recent offences are dropped first. + client is dropped first, with its history and its past bans. Active bans are + never dropped; one whose client is no longer tracked is dropped when it ends. + +## Bans + +A ban refuses every request from a netblock, answering with `BAN_RESPONSE` +(`403` by default), until the ban ends; a permanent ban never ends. The netblock +a ban covers is the client's IPv4 address (or the wider IPv4 prefix that +`BAN_SCOPE_V4_PREFIX` sets) or its IPv6 /64, and never more, so a permanent ban +does not reach neighbours who did nothing. + +An offence is a request the sidecar holds against the client: one that carries a +clear sign of attack, breaks a limit, or is refused by a rule file or the Core +Rule Set. Offences are counted by kind in the client's history. Two kinds lead +to a ban, each by its own rule: + +- A clear sign of attack: a request that matches a rule file rule whose action + is `ban`, or asks for a path in `TRAP_PATHS`. Examples are a probe for a + `.env` file or a `.git` directory, and a known scanner's user agent. + - The first one bans the netblock for `ATTACK_BAN_DURATION` (default `7d`). + - Any further request from the netblock while that ban lasts shows it is + malicious: the ban becomes permanent. + - Once that ban has run out, the netblock is served like any other, but its + next clear sign of attack bans it permanently at once. +- A broken limit: a request over a request limit, a response that takes the + client's byte total past a byte limit, or more error responses than + `ERROR_BURST_THRESHOLD` within a minute. + - The first such ban, or one that comes more than `LIMIT_BAN_REPEAT_WINDOW` + (default `24h`) after the last such ban ended, lasts `LIMIT_BAN_DURATION` + (default `1h`). + - Breaking a limit again within that window makes the new ban three times as + long as the last one: one hour, then 3, 9, 27 and 81 hours. + - A ban that would be longer than `MAX_BAN_DURATION` (default `7d`) is + permanent instead; with the defaults, that is the sixth ban in a row. + +A Core Rule Set match refuses only the request it matched, without a ban, +because the Core Rule Set has false positives; a `block` rule does the same. +Those refusals count toward the error burst like any other 403, so a client that +keeps setting them off is banned under the second rule. Requests refused under a +ban are not counted against limits, but they are counted in the client's history +and in the ban's notes. + +Every ban carries notes with what an admin needs to decide whether to lift it +(see "Persistent state"). An admin can add or lift a ban at any time by editing +`bans.json`, or through the admin listener when `ADMIN_TOKEN` is set. A ban +lifted before it ends is kept, marked as lifted, and does not count toward a +longer ban; deleting its entry from `bans.json` forgets it entirely. + +Clients of listed AS numbers and countries, and clients with a poor reputation, +get lower limits (see "Biased thresholds"), so the same rules ban them after +fewer requests. The result is always a ban, first a temporary one and then, for +repeated abuse, a permanent one. ## Configuration surface Conventions: lists are comma separated; netblocks are CIDR (a bare address means /32 or /128); durations use Go syntax plus `d` for days (`90s`, `15m`, `24h`, -`7d`); byte sizes accept `K`, `M`, `G` suffixes; an unset limit means that -limit is off. Every variable may instead be given as `NAME_FILE` pointing at a -file holding the value, for secrets and long lists. Invalid configuration stops -the process at start with a message naming the variable. At start the effective -configuration is logged with secrets masked. +`7d`); byte sizes accept `K`, `M`, `G` suffixes. + +- Only `UPSTREAM_URL` is required. Every other setting has a default chosen for + a service facing the internet in 2026, or stays off until it is given + something only the operator can supply: an alert destination, an account key, + the lookup database. +- Any limit or threshold can be switched off with the value `off`. +- A list set to an empty value is an empty list, and replaces the default. +- Every variable may instead be given as `NAME_FILE` pointing at a file holding + the value, for secrets and long lists. +- Settings, including `NAME_FILE` files, are read once at start; changing one + means restarting the container. The files the sidecar watches while it runs + are its state files, its rule files and the lookup database. +- Invalid configuration stops the process at start with a message naming the + variable. At start the effective configuration is logged with secrets masked. + +The settings, by group: - Core - `UPSTREAM_URL` (required): the application, for example `http://gitea:3000`. - `LISTEN_ADDR` (default `:8080`): proxy listener. - `ADMIN_LISTEN_ADDR` (default `127.0.0.1:9090`): admin listener. - - `ADMIN_TOKEN`: bearer token required for ban management endpoints. - - `INSTANCE_NAME`: included in every log line, metric and alert, for example - `fsn1app1/gitea`. + - `ADMIN_TOKEN`: bearer token required for the ban management endpoints. + Unset by default, which switches those endpoints off; bans are then + managed by editing `bans.json`. + - `INSTANCE_NAME` (default: the host name in `UPSTREAM_URL`, for example + `gitea`): included in every log line, metric and alert. Set it, for + example to `fsn1app1/gitea`, when several sidecars report to one place. - `MODE` (default `enforce`): `enforce`, or `observe` to log and alert on - every decision while refusing nothing. Meant for the first days of a - rollout. - - `TRUSTED_PROXIES` (required): netblocks whose `X-Forwarded-For` is - believed, normally the docker network traefik reaches the sidecar from. + every decision while refusing nothing. + - `TRUSTED_PROXIES` (default `10.0.0.0/8,172.16.0.0/12,192.168.0.0/16`, the + private address ranges): netblocks whose `X-Forwarded-For` is believed, + normally where traefik reaches the sidecar from. The header is used only + when the TCP peer is inside this list; otherwise the peer's own address is + the client, so a client that connects directly cannot claim another + address. A list given replaces the default; set but empty, it trusts + nothing. - `IPV6_GROUP_PREFIX` (default `64`). - - `MAX_TRACKED_CLIENTS` (default `500000`). + - `MAX_TRACKED_CLIENTS` (default `50000`): clients held in memory and in + `clients.json` (see "Persistent state"). - Persistent state - - `STATE_DIR` (default `/data`): the JSON state files and downloaded lookup - databases. - - `STATE_WRITE_DELAY` (default `2s`): after a ban or offence changes, wait - this long before writing, so a burst of changes becomes one write. - - `STATE_COUNTER_INTERVAL` (default `60s`): how often the hour and day - counters and the reputation cache are written. + - `STATE_DIR` (default `/data`): the JSON state files. The image declares + `/data` as a volume, so it is writable even when nothing is mounted; a + named volume, as in the examples below, keeps the state in a place that + outlives the container. + - `STATE_WRITE_DELAY` (default `2s`): after a ban changes, wait this long + before writing `bans.json`, so a burst of changes becomes one write. + - `STATE_COUNTER_INTERVAL` (default `60s`): how often `clients.json` and + `reputation.json` are written. - Logging - The request log on stdout is always on and has no switch. - `LOG_LEVEL` (default `info`): for the process's own messages (start-up, fetch failures, state writes), which are JSON lines on stdout too, marked `"type":"process"`. It does not filter the request log. - `LOG_REQUEST_HEADERS` (default - `accept,accept-language,accept-encoding,content-type,origin,range`): - extra request headers to record. `Authorization`, `Cookie` and - `Set-Cookie` values are never logged, only whether they were present. + `accept,accept-language,accept-encoding,content-type,origin,range`): extra + request headers to record. `Authorization`, `Cookie` and `Set-Cookie` + values are never logged, only whether they were present. - `LOG_REMOTE_URL`: when set, every log line is also sent to this endpoint. Forms: `syslog+udp://host:514`, `syslog+tcp://host:514`, - `syslog+tls://host:6514`, `relp://host:2514`, `relp+tls://host:2514`. + `syslog+tls://host:6514`. - `LOG_REMOTE_TLS_CA_FILE`: optional CA certificate for the `+tls` forms. - `LOG_REMOTE_BUFFER` (default `10000`): lines held in memory while the endpoint is unreachable; when full the oldest are dropped and counted. @@ -204,34 +293,87 @@ configuration is logged with secrets masked. which is the usual arrangement for a scraper on a private network. - Static lists - `ALLOW_NETS`: bypass everything (monitoring, the owner's own networks). - - `RATE_LIMIT_EXEMPT_NETS`: bypass request and byte limits only; attack - detection and bans still apply. + - `RATE_LIMIT_EXEMPT_NETS`: bypass request and byte limits only; the error + burst threshold, attack detection and bans still apply. - `DENY_NETS`: always refused. -- Request rate limits, per client (R1, R2) - - `RATE_LIMIT_PER_MINUTE`, `RATE_LIMIT_PER_HOUR`, `RATE_LIMIT_PER_DAY`. +- Request rate limits, per client (R1, R2). Breaking one bans the client (see + "Bans"), so the defaults sit several times above what one busy person + produces: a browser loading a heavy page makes a few hundred requests, a git + clone a handful, and several people often share one address. + - `RATE_LIMIT_PER_MINUTE` (default `1000`), `RATE_LIMIT_PER_HOUR` (default + `10000`), `RATE_LIMIT_PER_DAY` (default `50000`). - `RATE_LIMIT_EXEMPT_PATHS`: path prefixes not counted (static assets, health checks). -- Byte limits, per client - - `BYTES_LIMIT_PER_MINUTE`, `BYTES_LIMIT_PER_HOUR`, `BYTES_LIMIT_PER_DAY`. +- Byte limits, per client. A response's bytes are counted when it ends, so every + default sits above the largest response allowed (`CLIENT_RESPONSE_MAX_BYTES`, + 5 GB) and no single download breaks one. + - `BYTES_LIMIT_PER_MINUTE` (default `10G`), `BYTES_LIMIT_PER_HOUR` (default + `20G`), `BYTES_LIMIT_PER_DAY` (default `50G`). - `BYTES_COUNT` (default `both`): `response`, `request` or `both`. +- Size and time limits, per request, in both directions. The client-facing + limits apply between the client and the sidecar, the app-facing ones between + the sidecar and the app. Bodies stream straight through, so a request body + reaches the app while the client is still sending it, and of two matching + limits the lower one acts first. + - `CLIENT_REQUEST_TIMEOUT` (default `60s`): how long a client may take to + send its whole request, headers and body. + - `CLIENT_REQUEST_MAX_BYTES` (default `100M`): the largest request body a + client may send. + - `CLIENT_RESPONSE_TIMEOUT` (default `30m`): how long the sidecar may take + to deliver one response to the client, from the end of the request to the + last byte. + - `CLIENT_RESPONSE_MAX_BYTES` (default `5G`): the largest response body sent + to a client. + - `UPSTREAM_REQUEST_TIMEOUT` (default `60s`): how long the sidecar may take + to connect to the app and send it the whole request. + - `UPSTREAM_REQUEST_MAX_BYTES` (default `100M`): the largest request body + sent to the app. + - `UPSTREAM_RESPONSE_TIMEOUT` (default `30m`): how long the app may take to + send one whole response, from the end of the request to the last byte. + - `UPSTREAM_RESPONSE_MAX_BYTES` (default `5G`): the largest response body + taken from the app. + - When a limit is passed before the response has started, the sidecar + answers itself: `413` for a request body that is too large, `408` for a + client that is too slow, `502` for a response that is too large, `504` for + an app that is too slow. A request that announces a body larger than its + limit is refused before anything reaches the app. Once the response has + started it can only be cut off, and the connection is closed. + - A WebSocket connection leaves these limits behind once it is upgraded: it + stays open until either side closes it. - Lookup of AS number and country (R7) - - `ASN_DB_PATH`, `COUNTRY_DB_PATH`: `.mmdb` files mounted into the - container; or - - `ASN_DB_URL`, `COUNTRY_DB_URL`: downloaded into `STATE_DIR` at start and - every `LOOKUP_DB_REFRESH` (default `7d`). One file may serve both. - - Known free sources in this format: IPinfo Lite (country and AS number in - one file, free account token), DB-IP Lite, MaxMind GeoLite2 (free - account). Choice is open. + - `LOOKUP_DB_PATH`: the IPinfo Lite database in its `.mmdb` form + (`ipinfo_lite.mmdb`), one file that carries both country and AS number; + the sidecar reads its `asn`, `as_name` and `country_code` fields. The + operator downloads it with a free IPinfo account and mounts it read-only. + The sidecar never fetches it and holds no account or token for it. + - Refreshing the file is the operator's business; IPinfo updates it daily. + The sidecar notices when the file is replaced and reads it again; a + replacement it cannot read is ignored, logged and alerted once, and the + file it has stays in use. Mount the directory that holds the file rather + than the file itself: docker does not show a single mounted file being + replaced on the host. + - Unset by default, which means no lookups. A setting that needs them (the + biased thresholds below, `ADD_LOOKUP_HEADERS`, the per-AS-number anomaly + thresholds) given without `LOOKUP_DB_PATH` stops the start with a message + naming both. A configured file that is missing or unreadable at start + stops the start too. - `ADD_LOOKUP_HEADERS` (default `false`): pass `X-Client-ASN` and `X-Client-Country` to the app. - - Missing or unreadable database: start anyway, treat every client as - unknown, log a warning and send one alert. -- Biased thresholds (R8) +- Biased thresholds (R8). The lists are empty by default: they need the lookup + database, which only the operator can supply. - `ASN_LIMIT_PERCENT`: for example `AS14061:50,AS16276:50,AS45102:25`. - Listed AS numbers get that percentage of every request and byte limit. - `0` refuses outright. + Clients in a listed AS number get that percentage of every request and + byte limit, so the rules in "Bans" ban them after fewer requests than + others. `0` is a zero allowance: the client's first request breaks a limit + and bans it. - `COUNTRY_LIMIT_PERCENT`: same form with ISO country codes, for example - `CN:25,RU:50`. + `CN:25,RU:50`. Each client from a listed country gets that percentage of + the normal per-client limits. + - No budget is shared by a whole country or AS number: one abuser could use + it up and lock out everyone else there, the denial of service this tool + exists to prevent. + - When more than one percentage applies to a client (its AS number, its + country, a reputation hit), the lowest applies. - `ASN_BYTES_PERCENT`, `COUNTRY_BYTES_PERCENT`: optional overrides applied to byte limits only, when the byte percentage should differ from the request percentage. @@ -239,43 +381,67 @@ configuration is logged with secrets masked. place. - `ASN_LIMIT_PERCENT_URL`: optional URL of a text file of `AS:percent` lines, so one abuse-source list can be shared by every sidecar in the - fleet. + fleet. It is fetched and refreshed like the blocklists. - Attack detection (R9) - - `RULES_DIR` (default `/etc/smallwebwaf/rules.d`): directory of rule files - read once at start; format under "Rule files". The image ships a default - file there. Mounting a directory over it replaces the defaults; mounting - single files into it adds to them. + - `RULES_DIR` (default `/etc/smallwebwaf/rules.d`): directory of rule files, + read at start and again whenever a file in it changes; format under "Rule + files". - `RULES_ENABLED` (default `true`): `false` skips rule files entirely. - - `WAF_MODE` (default `detect`): `off`, `detect`, `block`. + - `WAF_MODE` (default `block`): `off`, `detect` (log and alert only), or + `block`. - `WAF_PARANOIA_LEVEL` (default `1`), `WAF_ANOMALY_THRESHOLD` (default `5`): - the Core Rule Set's own two tuning values. - - `WAF_DISABLED_RULES`: rule ids to switch off when an app trips a false - positive. + the Core Rule Set's own two tuning values, at the Core Rule Set's own + defaults. + - `WAF_DISABLED_RULES` (default `920420,920440`): rule ids to switch off + when an app trips a false positive. The two in the default refuse requests + by content type and by file extension; in front of a code forge they would + refuse git's clone and push over HTTP and the display of source files such + as `.sh` or `.sql`. A list given replaces the default, so include them in + it. - `WAF_EXEMPT_PATHS`: path prefixes not inspected. - `WAF_BODY_LIMIT` (default `128K`): bodies are inspected up to this size and streamed beyond it without buffering, so large uploads and git pushes are not held in memory. - `TRAP_PATHS`: paths the app never serves and only scanners ask for, for - example `/.env,/wp-login.php,/.git/config`. `TRAP_OFFENCE_WEIGHT` - (default `3`). This is the env-var short form of a `path` rule with an - `offence` action, for deployments that mount no rule files. - - `ERROR_BURST_THRESHOLD` (default off): 401, 403 and 404 responses per - client per minute that count as an offence. -- Blocks and bans (R5) - - `BAN_AFTER_OFFENCES` (default `3`), `OFFENCE_WINDOW` (default `10m`). - - `BAN_DURATIONS` (default `1h,24h,7d`). - - `PERMANENT_BAN_AFTER` (default `4`; `0` never bans permanently), - `BAN_HISTORY_WINDOW` (default `90d`). + example `/wp-login.php,/xmlrpc.php` in front of gitea. A request for one + is a clear sign of attack. This is the env-var short form of a `path` rule + with the `ban` action, for deployments that mount no rule files. + - `ERROR_BURST_THRESHOLD` (default `30`): responses with status 401, 403 or + 404 per client per minute; more than this breaks a limit (see "Bans"). + Scanners walking lists of paths cause hundreds a minute; a person rarely + causes more than a few. +- Bans (R5), following the rules under "Bans" + - `ATTACK_BAN_DURATION` (default `7d`): the ban for a first clear sign of + attack. + - `LIMIT_BAN_DURATION` (default `1h`): the ban for a first broken limit. + - `LIMIT_BAN_REPEAT_WINDOW` (default `24h`): breaking a limit again within + this time after a ban for a broken limit ended makes the next ban three + times as long. + - `MAX_BAN_DURATION` (default `7d`): a ban that would be longer is permanent + instead. - `BAN_RESPONSE` (default `403`): `403`, `429`, or `close` to drop the - connection without an answer. + connection without an answer. `403` makes a ban easy to recognise when + debugging; `close` tells the client nothing. - `BAN_SCOPE_V4_PREFIX` (default `32`): widen to for example `24` to ban the surrounding netblock. -- Reputation (R6) - - `BLOCKLIST_URLS`: text files of addresses and netblocks, for example - Spamhaus DROP, FireHOL level 1, the Tor exit list. Refreshed every - `BLOCKLIST_REFRESH` (default `6h`); the last good copy is kept on failure. - - `BLOCKLIST_ACTION` (default `limit:25`): `deny`, `limit:`, or - `log`. +- Reputation (R6). No source is on by default. A DNS blocklist would be told the + address of every visitor. AbuseIPDB and CrowdSec need an account or an engine + of the operator's own. A downloaded list reveals nothing about visitors, but + each sidecar fetches its own copy, and the list most fit to be a default, + Spamhaus DROP, may be fetched at most once a day: a default would break that + on any host that runs several sidecars. + - `BLOCKLIST_URLS`: text files of addresses and netblocks, one per line; + anything after a `;` or `#` on a line is ignored. For example the Spamhaus + DROP list, `https://www.spamhaus.org/drop/drop.txt`; Spamhaus asks that it + be fetched at most once a day and that products using it credit The + Spamhaus Project. Refreshed every `BLOCKLIST_REFRESH` (default `24h`); the + last good copy is kept on failure and across restarts. + - `BLOCKLIST_ACTION` (default `deny`): `deny`, `limit:`, or `log`. + `deny` refuses every request from a listed address, which suits lists of + networks that send nothing legitimate, such as DROP. `limit:` + gives listed clients that percentage of every limit, so they are banned + after fewer requests; it suits a list of addresses shared with ordinary + visitors, such as Tor exits. - `DNSBL_ZONES`: for example `dnsbl.dronebl.org`. Zones meant for mail (lists of residential ranges) will block ordinary visitors and should not be used; Spamhaus zones need their keyed query service, given as a zone @@ -288,18 +454,20 @@ configuration is logged with secrets masked. so the budget is spent on suspects. - `CROWDSEC_LAPI_URL`, `CROWDSEC_LAPI_KEY`: optional. If a CrowdSec engine exists on the host, pull its decision list on a schedule and treat listed - addresses as banned. This is how a fleet-wide blocklist can arrive - without the sidecar depending on CrowdSec. + addresses as banned. This is how a fleet-wide blocklist can arrive without + the sidecar depending on CrowdSec. - `REPUTATION_ACTION` (default `limit:25`): `deny`, `limit:`, or - `log`, for DNSBL and API hits. + `log`, for DNSBL and API hits. Such verdicts are less certain than a + blocklist, so by default a listed client gets a quarter of every limit and + is banned after a quarter of the requests. - `REPUTATION_CACHE_TTL` (default `24h`), `REPUTATION_TIMEOUT` (default `2s`). - Alerting (R3) - - `ALERT_WEBHOOK_URL`: JSON POST; schema below. - `ALERT_WEBHOOK_HEADERS`: optional `Name:value` pairs for authentication. + - `ALERT_WEBHOOK_URL`: JSON POST; schema below. `ALERT_WEBHOOK_HEADERS`: + optional `Name:value` pairs for authentication. - `ALERT_SLACK_WEBHOOK_URL`: Slack incoming webhook, formatted message. - - `ALERT_NTFY_URL` (full topic URL), `ALERT_NTFY_TOKEN`: title, priority - and tags set from the event type. + - `ALERT_NTFY_URL` (full topic URL), `ALERT_NTFY_TOKEN`: title, priority and + tags set from the event type. - `ALERT_EVENTS` (default `ban,permanent_ban,waf_block,anomaly,reputation_hit,source_failure`): which event types are sent. @@ -308,14 +476,18 @@ configuration is logged with secrets masked. repeats is included in the next one. - `ALERT_MAX_PER_HOUR` (default `60`): beyond this, alerts are rolled into one summary per hour so a wide attack cannot flood the channel. -- Anomaly thresholds, alert only, nothing is blocked (R4) +- Anomaly thresholds, alert only, nothing is blocked (R4). None is set by + default, and a threshold left unset sends no alert: these only alert, an alert + needs a destination only the operator can supply, and what counts as unusual + depends on each service's normal traffic, which the metrics show. - Per client: `ANOMALY_CLIENT_REQUESTS_PER_MINUTE`, `ANOMALY_CLIENT_REQUESTS_PER_HOUR`, `ANOMALY_CLIENT_BYTES_PER_MINUTE`, `ANOMALY_CLIENT_BYTES_PER_HOUR`. - Per surrounding netblock (`ANOMALY_NET_V4_PREFIX` default `24`, `ANOMALY_NET_V6_PREFIX` default `48`): `ANOMALY_NET_REQUESTS_PER_MINUTE`, `..._PER_HOUR`, `ANOMALY_NET_BYTES_PER_MINUTE`, `..._PER_HOUR`. - - Per AS number: `ANOMALY_ASN_REQUESTS_PER_MINUTE`, `..._PER_HOUR`, + - Per AS number, which needs the lookup database: + `ANOMALY_ASN_REQUESTS_PER_MINUTE`, `..._PER_HOUR`, `ANOMALY_ASN_BYTES_PER_MINUTE`, `..._PER_HOUR`. - Whole service: `ANOMALY_TOTAL_REQUESTS_PER_MINUTE`, `..._PER_HOUR`, `ANOMALY_TOTAL_BYTES_PER_MINUTE`, `..._PER_HOUR`. @@ -329,10 +501,11 @@ configuration is logged with secrets masked. ## Rule files -A required feature: at start the sidecar reads every `*.rules` file in -`RULES_DIR` and checks each request against them. The files are plain text meant -to be edited by hand, so that a new scanning pattern seen in the request log can -be turned into a rule in one line. +A required feature: the sidecar reads every `*.rules` file in `RULES_DIR` and +checks each request against them. The files are plain text meant to be edited by +hand, so that a new scanning pattern seen in the request log can be turned into +a rule in one line. The sidecar watches the directory: a file edited, added or +removed there takes effect while the sidecar runs. - One rule per line, four fields separated by spaces or tabs; the fourth field runs to the end of the line: @@ -343,23 +516,24 @@ be turned into a rule in one line. - Blank lines and lines starting with `#` are ignored. - `id`: a short name of letters, digits, `-` and `_`, unique across all files. - It appears in the request log, metrics, alerts and offence history. + It appears in the request log, metrics, alerts and ban notes. - `target`: what the regex is matched against. - `path`: the URL path as received, before any decoding. - `query`: the raw query string. - - `uri`: path and query together, both as received and once - percent-decoded, so an encoded probe cannot slip past. + - `uri`: path and query together, both as received and once percent-decoded, + so an encoded probe cannot slip past. - `method`, `host`, `user_agent`, `referer`. - `header:`: any one request header. - Request bodies are not available to rule files; body inspection is the Core Rule Set's job. - `action`: - `log`: note the match in the request log and do nothing else. - - `offence` or `offence:`: record one or `n` offences and forward the - request. Enough offences within `OFFENCE_WINDOW` creates a ban. - - `block` or `block:`: refuse with 403 and record one or `n` offences. - - `ban`: refuse and ban the client at once, at whatever length its ban - history calls for. + - `block`: refuse the request with 403. The client is not banned for it, but + the refusal counts toward the error burst (see "Bans"). + - `ban`: the request is a clear sign of attack. Refuse it and ban the + client's netblock for seven days (`ATTACK_BAN_DURATION`); any further + request during those days, or a later clear sign of attack, makes the ban + permanent (see "Bans"). Use it only for requests no real visitor sends. - `regex`: Go regular expression syntax (RE2). It has no backreferences or lookaround, and in exchange matching time is linear in the input, so no rule can be made to stall the proxy. `(?i)` at the front makes a rule @@ -367,11 +541,19 @@ be turned into a rule in one line. anchor with `^` and `$` when that is not wanted. - Files are read in name order (`00-default.rules` before `50-gitea.rules`), rules in line order. -- Rules are compiled once at start and held in memory. A line that does not - parse, a regex that does not compile, or a duplicate id stops the process - with a message naming the file and line. A missing or empty directory is not - an error: the log says that no rules were loaded. Changing rules means - restarting the container; reloading while running is given up for now. +- Rules are compiled when their file is read and held in memory. At start, a + line that does not parse, a regex that does not compile, or a duplicate id + stops the process with a message naming the file and line. While running, the + same faults leave the rules as they were, including the earlier version of + that file, and the log and one alert name the file and line; once the file is + fixed, it is read again. A missing or empty directory is not an error: the log + says that no rules were loaded. +- The image ships a default file in `RULES_DIR`. Mounting a directory over it + replaces the defaults; mounting single files into it adds to them. Docker does + not show a single mounted file being replaced on the host, which is how many + editors save, so rules meant to be edited while the sidecar runs belong in a + mounted directory, with a copy of the default file if the defaults are to + stay. - In `MODE=observe` every action is logged as what would have happened and nothing is refused. - Clients in `ALLOW_NETS` are not checked. @@ -381,73 +563,112 @@ Example file: ``` # 00-default.rules: probes no real visitor sends -# id target action regex -env-file path offence:3 (?i)/\.env(\.[a-z]+)?$ -git-dir path offence:3 ^/\.git/(config|HEAD|index)$ -wp-probe path offence:3 (?i)^/(wp-login\.php|xmlrpc\.php|wp-admin/) -php-shell path block:3 (?i)/(shell|c99|r57|wso|alfa)\.php$ -path-traversal uri block (\.\./){2,} -scanner-agent user_agent ban (?i)\b(sqlmap|nikto|nuclei|masscan|zgrab|wpscan)\b -empty-agent user_agent log ^$ +# id target action regex +env-file path ban (?i)/\.env(\.[a-z]+)?$ +git-dir path ban ^/\.git/(config|HEAD|index)$ +php-shell path ban (?i)/(shell|c99|r57|wso|alfa)\.php$ +scanner-agent user_agent ban (?i)\b(sqlmap|nikto|nuclei|masscan|zgrab|wpscan)\b +path-traversal uri block (\.\./){2,} +empty-agent user_agent log ^$ ``` The image ships one default file of this kind. It is kept short and limited to -patterns that are wrong for every app; anything app-specific (for example -blocking `/wp-login.php` is harmless in front of gitea and fatal in front of -WordPress) belongs in a file the deployer mounts. +patterns that are wrong for every app. Anything app-specific belongs in a file +the deployer mounts: a request for `/wp-login.php`, for example, is a clear sign +of attack in front of gitea and an ordinary login in front of WordPress. + +``` +# 50-gitea.rules: WordPress probes, which gitea never serves +wp-probe path ban (?i)^/(wp-login\.php|xmlrpc\.php|wp-admin/) +``` ## Persistent state -All state lives in memory. No database is used. What must survive a restart is -written to `STATE_DIR` as JSON files; the files are read once at start and never -during a request. The whole state is expected to stay under 10 MiB, and reading -or writing even ten times that as JSON takes well under a second, so whole-file -rewrites are cheap enough to need nothing cleverer. +All state lives in memory, and the files in `STATE_DIR` hold a copy of all of +it, so an orderly stop loses nothing. No database is used, and nothing is read +from disk while serving a request. The files are meant for people as well: an +admin can read or edit them at any time, and the running sidecar takes the edit +in. - Files, each holding one kind of state: - - `bans.json`: active and past bans. Per entry: client or netblock, start, - expiry (`null` for permanent), reason, which ban this is for that client - (first, second, third), source (`local`, `admin`, `crowdsec`), and when - it was lifted, if it was. - - `offences.json`: offender history. Per entry: client, time, kind (`rate`, - `bytes`, `waf`, `trap`, `error_burst`), short detail. Entries older than - `BAN_HISTORY_WINDOW` are dropped at each write. - - `counters.json`: the hour and day request and byte counters per client. - - `reputation.json`: cached DNSBL and reputation API verdicts with the time - each was fetched. + - `bans.json`: every active and past ban. Per entry: the netblock, start, + expiry (`null` for permanent), what caused it (`attack`, `limit`, `admin` + or `crowdsec`), a short reason, when it was lifted if an admin lifted it, + and a `notes` field (below). + - `clients.json`: per client, its minute, hour and day counters and its + history since it was first seen: first and last time seen, the AS number, + AS name and country last looked up, total requests and bytes in each + direction, how many requests were forwarded and how many refused, + responses by status class, and offences by kind. Clients that were never + banned are kept too, so every client's history survives a restart; + `GET /clients/` shows it. + - `reputation.json`: the last good copy of each list fetched from a URL, and + cached DNSBL and reputation API verdicts, each with the time it was + fetched. +- Ban notes. The notes on a ban hold what an admin needs to decide whether to + lift it, drawn from what the sidecar already knows; nothing is looked up to + fill them: + - the AS number, AS name and country (empty without a lookup database); + - what was broken: the rule ids and target that matched, or the limit, its + window, the count reached and the client's limit percentage with what set + it; and any reputation sources that listed the client; + - the requests that caused the ban, up to the last ten: time, method, host, + path, status and user agent; + - how many requests counted toward the ban, and the time span over which + they came; + - the netblock's total requests since it was first seen, and the requests + refused under this ban so far, kept up to date while the ban lasts; + - how many earlier bans of each kind the netblock has had. +- Size: each tracked client takes about 2 KiB in `clients.json`, so at the + default `MAX_TRACKED_CLIENTS` of 50,000 the file can reach about 100 MiB; a + small service usually tracks a few thousand clients and a few MiB. The other + files stay small. Writing a whole file of that size takes about a second, in + the background, so rewriting whole files still needs nothing cleverer. - Format: indented JSON with a top-level `version` number, entries sorted by client address, times in RFC 3339 UTC, durations and sizes as plain numbers - with the unit in the field name. The aim is that a person can open - `bans.json` in an editor, find an address, and remove or add an entry. + with the unit in the field name. The aim is that a person can open `bans.json` + in an editor, find an address, and remove or add an entry. - Writing: - serialise from a snapshot taken under the lock, so requests are not held up while the file is written; - write to a temporary file in the same directory, sync it, rename it over the real name, sync the directory. A crash at any point leaves either the old complete file or the new complete file, never a partial one; - - `bans.json` and `offences.json` are written `STATE_WRITE_DELAY` after a - change; `counters.json` and `reputation.json` every - `STATE_COUNTER_INTERVAL`; all four on orderly shutdown (SIGTERM). -- Reading: - - at start only. Active bans go into an in-memory prefix lookup; everything - else into maps. Expired counters and cache entries are discarded. - - a missing file means empty state and is normal on first run. - - a file that does not parse, or has an unknown `version`, stops the - process with a message naming the file and position. Starting with empty - bans would silently forgive every repeat offender, and since writes are - atomic a broken file can only come from a hand edit, which the editor - should hear about. `counters.json` and `reputation.json` are the - exception: if unreadable they are logged, set aside with a `.bad` suffix, - and the process starts with them empty. -- Hand edits are made with the container stopped; a running process will - overwrite the file at its next write. Changes to a running process go through - the admin endpoints. Reloading files into a running process is given up for - now. -- `STATE_DIR` not writable at start: the process exits. A write that fails - while running: state stays correct in memory, the failure is logged, counted - in metrics, alerted once per cooldown, and retried at the next write. -- What a hard kill can lose: bans and offences from the last - `STATE_WRITE_DELAY`, counters from the last `STATE_COUNTER_INTERVAL`. Losing + - `bans.json` is written `STATE_WRITE_DELAY` after a change; `clients.json` + and `reputation.json` every `STATE_COUNTER_INTERVAL`; all three on orderly + shutdown (SIGTERM); + - before writing a file, the sidecar checks whether it changed on disk since + the sidecar last read or wrote it; if it did, the sidecar takes that edit + in first (below), so an admin's edit is never overwritten. +- Reading at start: + - Active bans go into an in-memory prefix lookup; everything else into maps. + Counter buckets and cache entries whose time has passed are discarded. + - A missing file means empty state and is normal on first run. + - A file that does not parse, or has an unknown `version`, stops the process + with a message naming the file and position. Starting with empty state + would silently forgive every repeat offender, and since writes are atomic + a broken file can only come from a hand edit, which the editor should hear + about. +- Edits while running: + - The sidecar watches `STATE_DIR` and notices a file that is edited, + replaced or added. It tells its own writes from an admin's by comparing + the file with what it last wrote. + - An edit that parses is taken in at once: what the file says replaces what + the sidecar held for that file. Removing a ban's entry from `bans.json` + lifts the ban; adding an entry bans. Changes the sidecar made after the + admin opened the file, such as a new ban, are lost when the admin saves + over them; most editors warn when a file changed on disk while it was + open. + - An edit that does not parse does not stop the running sidecar. It keeps + the state it has, renames the edited file to `.bad` (for example + `bans.json.bad`) so the edit is kept, writes the file again from memory, + and logs and alerts once with the file and the position of the error. The + admin fixes the `.bad` file and moves it back. +- `STATE_DIR` not writable at start: the process exits. A write that fails while + running: state stays correct in memory, the failure is logged, counted in + metrics, alerted once per cooldown, and retried at the next write. +- What a hard kill can lose: ban changes from the last `STATE_WRITE_DELAY`, and + counts, history and reputation from the last `STATE_COUNTER_INTERVAL`. Losing the volume loses ban history, not service. ## Request log @@ -464,18 +685,18 @@ handling while `docker logs` keeps working. passed to the app), `peer_ip` (the TCP peer, normally traefik), `forwarded_for` (the header as received), `client_group` (the /64 or configured prefix used for counting), `asn`, `as_name`, `country`, - `content_type`, `content_length`, the headers named in - `LOG_REQUEST_HEADERS`, `has_authorization` and `has_cookie` as booleans, - `websocket` when the connection was upgraded. + `content_type`, `content_length`, the headers named in `LOG_REQUEST_HEADERS`, + `has_authorization` and `has_cookie` as booleans, `websocket` when the + connection was upgraded. - Response detail: `response_content_type`, `upstream_status` (differs from `status` when the sidecar answered itself), `cache_control`, `location` on redirects, `aborted` when the client went away early. -- Decision: `action` (`forward`, `rate_limited`, `bytes_limited`, `banned`, - `denied`, `waf_blocked`, `upstream_error`), `would_action` in `observe` mode, - `limit_percent` and which rule set it, `counts` (the client's minute, hour - and day request and byte totals after this request), `limit_hit` (which - window), `rule_ids` (rule file rules that matched), `waf_rule_ids`, - `waf_score`, `reputation` (sources that listed the +- Decision: `action` (`forward`, `banned`, `denied`, `rate_limited`, + `rule_blocked`, `waf_blocked`, `too_large`, `timed_out`, `upstream_error`), + `would_action` in `observe` mode, `limit_percent` and which rule set it, + `counts` (the client's minute, hour and day request and byte totals after this + request), `limit_hit` (which window), `rule_ids` (rule file rules that + matched), `waf_rule_ids`, `waf_score`, `reputation` (sources that listed the client), `offence` when one was recorded, `ban_expires`. - Timings in milliseconds: `duration_total`, `duration_checks` (everything the sidecar did before forwarding), `duration_waf`, `duration_upstream_connect`, @@ -487,26 +708,23 @@ handling while `docker logs` keeps working. - Remote sending: - syslog forms send each line as the message of an RFC 5424 record, with octet-counted framing on TCP and TLS; - - RELP forms send the same record and wait for the receiver's - acknowledgement, so lines are not lost across a receiver restart. This is - the form to use with rsyslog; - sending happens on its own goroutine from a bounded buffer (`LOG_REMOTE_BUFFER`). An unreachable or slow endpoint never delays a request and never stops stdout; it reconnects with backoff, drops the - oldest lines when the buffer is full, and counts the drops in metrics. - UDP gives no delivery signal at all and is offered only for - compatibility. + oldest lines when the buffer is full, and counts the drops in metrics. UDP + gives no delivery signal at all and is offered only for compatibility. ## Metrics endpoint -Prometheus text format on `METRICS_LISTEN_ADDR` at `METRICS_PATH`, on by default, -switched off with `METRICS_ENABLED=false`. It has its own listener so it can be -reached by a scraper without exposing ban management. +Prometheus text format on `METRICS_LISTEN_ADDR` at `METRICS_PATH`, on by +default, switched off with `METRICS_ENABLED=false`. It has its own listener so +it can be reached by a scraper without exposing ban management. -- Traffic: requests and bytes in and out, by status class and `action`; - request duration and upstream duration histograms; requests in flight. -- Limits and bans: limit hits by window and kind (requests or bytes), offences - by kind, bans created by ordinal, permanent bans, active bans (gauge). +- Traffic: requests and bytes in and out, by status class and `action`; request + duration and upstream duration histograms; requests in flight. +- Limits and bans: limit hits by window and kind (requests or bytes), size and + time limit hits by limit, offences by kind, bans created by cause, permanent + bans, active bans (gauge). - Attack detection: rule file matches by rule id and action, and the number of rules loaded; Core Rule Set matches by mode and rule id (label limited to the rules that actually fired). @@ -515,10 +733,11 @@ reached by a scraper without exposing ban management. `other`, so the label set stays bounded; reputation queries, hits, failures and remaining daily budget by source; age of each blocklist and lookup database. -- Housekeeping: tracked clients (gauge), state file writes, write failures, - last successful write time and size per file; alerts sent, failed and - suppressed by destination; remote log lines sent, dropped and buffer depth; - the standard Go runtime and process metrics. +- Housekeeping: tracked clients (gauge), state file writes, write failures, last + successful write time and size per file; files read again after an edit, and + edits set aside because they did not parse; alerts sent, failed and suppressed + by destination; remote log lines sent, dropped and buffer depth; the standard + Go runtime and process metrics. - No metric carries a client IP address as a label; per-address questions are answered by the request log and `GET /clients/`. @@ -526,9 +745,11 @@ reached by a scraper without exposing ban management. - `GET /healthz`: for the container health check. - `GET /bans`, `POST /bans` (client or netblock, duration, reason), - `DELETE /bans/`: need `ADMIN_TOKEN`. -- `GET /clients/`: current counters, lookup result, reputation, offences; - for answering "why was this address refused". + `DELETE /bans/`: need `ADMIN_TOKEN`, and are switched off while it is + unset. Editing `bans.json` does the same without a token. +- `GET /clients/`: current counters, history, lookup result, reputation, + offences, and bans with their notes; for answering "why was this address + refused" and "what has this netblock been doing". ## Alert webhook schema @@ -538,13 +759,14 @@ One JSON object per alert: - `client`, `netblock`, `asn`, `as_name`, `country` - `reason`: short human-readable sentence - `detail`: event-specific fields, for example `window`, `count`, `limit`, - `limit_percent`, `rule_ids`, `path`, `ban_ordinal`, `ban_expires` + `limit_percent`, `rule_ids`, `path`, `ban_expires`, and for a ban its notes - `suppressed_repeats`: number of identical alerts held back by the cooldown ## Deployment as a sidecar -docker-compose shape, using gitea as the example; upaas deployments follow the -same shape if upaas can run a second container for a service (open): +docker-compose shape, using gitea as the example. The app carries no traefik +labels and publishes no ports; the sidecar carries the labels and points at the +app's hostname. `UPSTREAM_URL` is the only setting it needs: ```yaml services: @@ -559,93 +781,125 @@ services: user: "65532:65532" environment: UPSTREAM_URL: http://gitea:3000 - TRUSTED_PROXIES: 172.18.0.0/16 - INSTANCE_NAME: fsn1app1/gitea - MODE: observe - RATE_LIMIT_PER_MINUTE: "120" - RATE_LIMIT_PER_HOUR: "2000" - RATE_LIMIT_PER_DAY: "10000" - BYTES_LIMIT_PER_HOUR: 2G - ASN_DB_URL: https://example.invalid/asn-country.mmdb - ASN_LIMIT_PERCENT: AS14061:50,AS16276:50 - COUNTRY_LIMIT_PERCENT: CN:25 - WAF_MODE: detect - TRAP_PATHS: /.env,/wp-login.php,/.git/config - BLOCKLIST_URLS: https://www.spamhaus.org/drop/drop.txt - ALERT_NTFY_URL: https://ntfy.example.invalid/fleet-alerts - LOG_REMOTE_URL: relp://logs.example.invalid:2514 - METRICS_LISTEN_ADDR: :9100 volumes: - gitea-guard-state:/data - - ./50-gitea.rules:/etc/smallwebwaf/rules.d/50-gitea.rules:ro networks: [internal, traefik] labels: traefik.enable: "true" traefik.http.routers.gitea.rule: Host(`git.example.invalid`) traefik.http.services.gitea.loadbalancer.server.port: "8080" + +volumes: + gitea-guard-state: + +networks: + internal: + traefik: + external: true ``` -- The application container leaves the traefik network, so the sidecar cannot - be bypassed. -- Traefik routes only port 8080. The metrics port is reached by the scraper - over a docker network it shares with the sidecar; the admin port stays on - loopback inside the container and is used through `docker exec`. -- The state volume holds a few small JSON files; it needs no backup beyond - whatever the host already does. -- SSH access to gitea does not pass through the sidecar and is not protected - by it. -- Rollout per service: start in `MODE=observe` with `WAF_MODE=detect`, read a - few days of logs and alerts, set limits and rule exclusions from what real - traffic looks like, then switch to `enforce` and `block`. -- Notes specific to gitea: clones and pushes over HTTP are large and long, so - byte limits must be sized for a legitimate clone, the proxy sets no overall - response timeout, and `WAF_BODY_LIMIT` keeps pack uploads out of memory. - Archive download and blame or history pages are what scrapers hammer; request - limits do most of the work there. +- An app deployed by upaas takes the same shape, without the app's part of this + file. upaas deploys the app with no traefik labels; the sidecar runs beside it + from its own docker-compose file, carries the traefik labels, and points + `UPSTREAM_URL` at the app's hostname. The sidecar must be able to reach that + hostname, for example over a docker network the two share. +- The application container leaves the traefik network, so the sidecar cannot be + bypassed. +- Traefik routes only port 8080. The metrics port is reached by the scraper over + a docker network it shares with the sidecar; the admin port stays on loopback + inside the container and is used through `docker exec`. +- The state volume holds the JSON state files, a few MiB for a small service; it + needs no backup beyond whatever the host already does. +- SSH access to gitea does not pass through the sidecar and is not protected by + it. +- Rollout per service: point traefik at the sidecar with only `UPSTREAM_URL` + set. It protects the app from the first request, and nothing needs tuning + first. Afterwards the request log and the ban notes say why each client was + refused. A real visitor refused by mistake is let through with an exclusion + (`WAF_DISABLED_RULES`, `WAF_EXEMPT_PATHS`, `RATE_LIMIT_EXEMPT_PATHS`) or an + exemption (`RATE_LIMIT_EXEMPT_NETS`, `ALLOW_NETS`), and its ban is lifted in + `bans.json`. Additions to consider at any time: an alert destination, a lookup + database with biased thresholds, reputation sources, a remote log endpoint, + and an app-specific rule file. +- Notes specific to gitea: + - A clone is a response, and fits the defaults of 30 minutes and 5 GB for + all but the largest repositories and slowest links. A push is a request: + at the defaults, one larger than 100 MB or taking more than 60 seconds to + upload is cut off, so a gitea that takes large pushes needs + `CLIENT_REQUEST_MAX_BYTES`, `UPSTREAM_REQUEST_MAX_BYTES`, + `CLIENT_REQUEST_TIMEOUT` and `UPSTREAM_REQUEST_TIMEOUT` raised to fit. + `WAF_BODY_LIMIT` keeps pack uploads out of memory. + - The default `WAF_DISABLED_RULES` lets git's clone and push over HTTP, and + the display of source files, through the Core Rule Set. Pages where people + post code (issues, pull requests, the web editor) can still trip other + rules; such a match refuses only that request, and the request log names + the rule in `waf_rule_ids` for `WAF_DISABLED_RULES`. + - Archive download and blame or history pages are what scrapers hammer; + request limits do most of the work there. ## Failure behaviour -- Lookup database missing: clients are unknown, service continues, one alert. +- Lookup database: a configured file that is missing or unreadable at start + stops the start. A replacement that cannot be read while running is ignored: + the file already loaded stays in use, the problem is logged, one alert. - Reputation source down or over quota: no verdict, service continues, one `source_failure` alert per cooldown. - Alert destination down: retried with backoff from a bounded queue, oldest dropped first, drops counted in metrics. - Upstream down: 502 from the sidecar, not counted as client offences. -- Attack detection engine error on a request: request is forwarded, error - logged and counted. +- Attack detection engine error on a request: request is forwarded, error logged + and counted. - Remote log endpoint down: stdout continues, lines are buffered then dropped oldest first, drops counted in metrics. - State file write fails while running: memory stays authoritative, logged, counted, alerted, retried. -- In short: the only things that stop the process are invalid configuration, - a rule file that does not parse, an unwritable `STATE_DIR`, and an unparseable `bans.json` or `offences.json` - at start. A broken helper never takes the protected service down. +- A state file or rule file edited while running that does not parse: the + sidecar keeps running on what it has. The state file is set aside as + `.bad` and written again from memory; the rule file is left as it is. + Logged, one alert. +- In short: the only things that stop the process happen at start: invalid + configuration, a configured lookup database that is missing or unreadable, a + rule file or state file that does not parse, and an unwritable `STATE_DIR`. + Once running, a broken helper or a broken edit never takes the protected + service down. ## Risks the design has to handle -- Forged `X-Forwarded-For`: handled by `TRUSTED_PROXIES` being required and by - walking the header from the right. -- Many clients behind one address (mobile carriers, offices, Tor): rate limits - punish them collectively; `RATE_LIMIT_EXEMPT_NETS` and sensible day limits - are the remedy, and bans on such addresses should stay short. +- Forged `X-Forwarded-For`: handled by believing the header only from a peer + inside `TRUSTED_PROXIES` and by walking it from the right. The default trusts + every private address, which in the intended deployment means traefik; where + other containers can reach the sidecar directly, setting `TRUSTED_PROXIES` to + traefik's own network closes that gap. +- Many clients behind one address (mobile carriers, offices, Tor): they share + its limits and its bans. The default limits sit several times above what one + busy person produces, `RATE_LIMIT_EXEMPT_NETS` and `ALLOW_NETS` take known + shared addresses out, and the ban notes show an admin what happened before a + ban is lifted. +- A `ban` rule that matches a real visitor bans for seven days, so the default + rule file keeps to requests no real visitor sends. - IPv6 address rotation inside a /64: handled by grouping. - Widely distributed scrapers using thousands of addresses at low rates each: - per-client limits do not see them. The AS number and netblock anomaly - alerts reveal them, and `ASN_LIMIT_PERCENT` at a low value or `0` is the - response. A per-AS-number or per-country total limit is open. + per-client limits do not see them. The AS number and netblock anomaly alerts + reveal them once their thresholds are set, and a low `ASN_LIMIT_PERCENT` for + that AS number is the response: it lowers the limits of each of its clients. + There is no shared budget for a whole AS number or country, which one abuser + could use up and so lock out everyone else there. - Core Rule Set false positives against real apps (gitea's editor, API - payloads): `detect` by default, exclusions by rule id and path. -- Slow-request attacks: header read timeout and idle timeout on the listener. + payloads): a match refuses only that request and bans no one by itself; the + request log names the rule, and exclusions by rule id and path fix it. +- Slow-request attacks: `CLIENT_REQUEST_TIMEOUT` bounds how long a request may + take to arrive, and an idle timeout on the listener closes connections that + send nothing. - Alert floods: cooldown and hourly cap. ## Build order -- First: proxy, client identification, static lists, three-window request - limits, exemptions, `observe` mode, the full request log on stdout, the - metrics endpoint, health. -- Second: rule files, ban ledger with escalation and permanent bans, the JSON - state files, - admin endpoints, alerting to all three destinations, remote log sending. +- First: proxy with its size and time limits, client identification, static + lists, three-window request limits and the bans they lead to, the ban ledger + and the JSON state files with edits taken in while running, exemptions, + `observe` mode, the full request log on stdout, the metrics endpoint, health. +- Second: rule files, admin endpoints, alerting to all three destinations, + remote log sending. - Third: AS number and country lookup, biased thresholds, byte limits, anomaly thresholds. - Fourth: blocklists, DNSBL, AbuseIPDB, optional CrowdSec decision feed. @@ -653,43 +907,3 @@ services: bursts. - Each stage is usable on its own; the first three already cover the traffic problem the fleet has today. - -## Questions for the owner - -- Combining percentages. A client in a listed AS number (50 percent) and a - listed country (25 percent): should it get the lowest (25 percent of the - normal limit) or the product (12.5 percent)? Lowest is easier to predict when - reading a config; product punishes overlap harder. Recommendation: lowest. -- Meaning of "certain countries getting a percentage rate or byte limit". Read - here as: each client from that country gets that percentage of the normal - per-client limits. The other reading is a cap on the country as a whole, for - example all of one country together may use at most 20 percent of the - service's hourly byte budget. The second needs a configured total budget and - means one heavy client can lock out its compatriots. Recommendation: build - the per-client reading first; add whole-country and whole-AS-number caps only - if distributed scrapers make it necessary. -- Sharing bans across the fleet. As drafted, each sidecar remembers only its - own bans, so an abuser banned at gitea starts fresh at the next service. - Options: keep it per sidecar; or run one CrowdSec engine per host and have - every sidecar read its decision list (already supported above) and also - report its own bans to it. Recommendation: per sidecar for the first version, - decide on CrowdSec after seeing real ban volumes. -- Lookup database source. IPinfo Lite (one file with country and AS number, - free account, attribution required), DB-IP Lite (no account, attribution - required), or MaxMind GeoLite2 (free account, licence terms on - redistribution). Recommendation: IPinfo Lite, fetched once centrally and - served to the fleet from an internal URL so sidecars hold no account token. -- upaas. Whether upaas can deploy a second container beside an app and move - the traefik labels onto it was not examined. If it cannot, that is a feature - request for upaas, not something this sidecar can solve. -- Remote log library. Sending syslog in the current format (RFC 5424) is not - covered by Go's standard library, whose `log/syslog` package is frozen and - writes the older format, and no widely used Go RELP client was found. The - choices are a small third-party package for each, or writing both inside this - project; the syslog record is a few dozen lines, a RELP client with - acknowledgement handling is a few hundred. Recommendation: write the syslog - sender in-project, ship it first, and decide on RELP after looking at the - available packages; say if RELP must be in the first release. -- Ban response. `403` tells the abuser they were noticed; `close` wastes less - and tells them nothing. Recommendation: `403` by default for debuggability, - `close` available.