diff --git a/README.md b/README.md index 8071582..7e12d4d 100644 --- a/README.md +++ b/README.md @@ -3,9 +3,10 @@ `smallwebwaf` is a simple, fast, logging web application firewall for people who host their own services. It is one small container that sits between your reverse proxy (traefik) and one application: traefik points at `smallwebwaf`, -and `smallwebwaf` points at the app. It is configured with environment -variables, keeps its state in memory, and writes a detailed JSON log line for -every request. +and `smallwebwaf` points at the app. It needs one setting, the address of the +app, and protects the app from the first request with defaults chosen for a +service on the open internet. It keeps its state in memory and in JSON files you +can read and edit, and writes a detailed JSON log line for every request. Status: design stage. This repository currently holds the documents only; no code has been written. The design is in [`SPEC.md`](SPEC.md), and the survey of @@ -17,71 +18,96 @@ Small self-hosted sites now receive a great deal of traffic nobody asked for: scrapers that ignore `robots.txt` and crawl every commit of every repository on a public git server, vulnerability scanners walking through lists of WordPress and `.env` paths, and credential-guessing bots. Most of it comes from a small -number of hosting networks and countries. A single-person operation has no -abuse desk and no CDN contract; it needs something small that can be put in -front of one service and left alone. +number of hosting networks and countries. A single-person operation has no abuse +desk and no CDN contract; it needs something small that can be put in front of +one service and left alone. The existing tools each solve part of this. Rule-based firewalls catch attack -payloads but do not limit request rates. Rate limiters count requests but -cannot tell a residential visitor from a rented server farm. The products that -do most of it want several containers, a database and a web console. None of -them can say "clients from these networks get half the normal allowance", which -is the most useful thing to be able to say when nearly all abuse comes from a -known list of AS numbers. [`EVALUATION.md`](EVALUATION.md) goes through the -candidates one by one. +payloads but do not limit request rates. Rate limiters count requests but cannot +tell a residential visitor from a rented server farm. The products that do most +of it want several containers, a database and a web console. None of them can +say "clients from these networks are banned after half as many requests as +anyone else", which is the most useful thing to be able to say when nearly all +abuse comes from a known list of AS numbers. [`EVALUATION.md`](EVALUATION.md) +goes through the candidates one by one. `smallwebwaf` is meant to fill that gap: - protect a service from misbehaving scrapers and scanners with per-client request and byte limits over a minute, an hour and a day; -- bias those limits against the countries and AS numbers that abuse commonly - comes from, so their clients get a configured percentage of the normal - allowance rather than an outright block; -- remember abusers, block them for a while, and ban the ones who keep coming - back; +- lower those limits for the countries and AS numbers that abuse commonly comes + from, so their clients are banned after fewer requests than others; +- ban abusers: briefly at first, longer each time they come back, and + permanently when they keep at it; a scanner's first probe bans it for seven + days; - log everything in a form that is easy to search and ship elsewhere; - stay small enough to understand: one binary, one container, environment - variables, no database. + variables, no database, one required setting. ## Proposed features -- Reverse proxy for one upstream application, streaming in both directions, - with WebSocket support. One `smallwebwaf` per app. -- Real client address worked out from `X-Forwarded-For`, trusting only the - proxy networks you list. IPv6 clients are counted by /64. +- Reverse proxy for one upstream application, streaming in both directions, with + WebSocket support. One `smallwebwaf` per app. +- Internet-ready out of the box: only `UPSTREAM_URL` must be set, and every + other setting has a default chosen for a service facing the internet in 2026. +- Real client address worked out from `X-Forwarded-For`, trusting only the proxy + networks you list, by default the private address ranges. IPv6 clients are + counted by /64 by default. +- Size and time limits on requests and responses, both between the client and + `smallwebwaf` and between `smallwebwaf` and the app: by default a request may + take 60 seconds and 100 MB, a response 30 minutes and 5 GB. - Rate limits per client on requests per minute, per hour and per day, and on - bytes per minute, per hour and per day. + bytes per minute, per hour and per day, on by default and set well above what + real visitors need. - Netblocks that bypass rate limiting, netblocks that bypass everything, and netblocks that are always refused. -- AS number and country lookup for every client from a local database file. +- AS number and country lookup for every client, off until you choose a source: + the IPinfo Lite database file, which you download and mount, or the free GeoJS + web service (see "Country and AS number lookup" below). +- Country lists: `DENIED_COUNTRIES` refuses every request from the countries + listed, `EXCLUSIVELY_ALLOWED_COUNTRIES` every request from anywhere else. Such + a request gets the answer a banned client gets as soon as the client's address + has been looked up, before its body is read and without the rule files or the + Core Rule Set looking at it, and no ban is made. - Biased limits: listed AS numbers and countries get a percentage of every - limit, for example 50 percent for common abuse-source networks. Zero percent - refuses outright. + limit, for example 50 percent for common abuse-source networks, so their + clients are banned after fewer requests. Zero percent is a zero allowance: the + first request breaks the limit and bans the client. - Attack detection: - a directory of plain text rule files, one regex per line, for catching - scanning and penetration probes; easy to edit by hand; - - the OWASP Core Rule Set, run by the Coraza engine, in detect-only or - blocking mode; - - trap paths and bursts of error responses. -- Offences add up to a temporary block. Block lengths grow for repeat offenders - (for example one hour, then a day, then a week) and end in a permanent ban. + scanning and penetration probes; easy to edit by hand, and picked up while + running; + - the OWASP Core Rule Set, run by the Coraza engine, refusing the requests + it flags; + - trap paths, and a ban for a client that the rule files or the Core Rule + Set refuse again and again. +- Bans: + - a clear sign of attack, such as a probe for a `.env` file or a scanner's + user agent, bans for seven days on the first request, and any further + request during those days makes the ban permanent; + - breaking a limit bans for an hour; breaking one again within a day of a + ban ending triples the length, and a ban that would last longer than seven + days is permanent instead; + - every ban carries notes on why it was made, to help decide whether to lift + it. - IP reputation: downloadable blocklists, DNS blocklists, AbuseIPDB, and an optional feed of decisions from a CrowdSec engine. Lookups happen in the - background and never delay a request. + background and never delay a request. None is on until you add it. - Alerts on attacks and bans to a generic webhook, Slack or ntfy, with a cooldown and an hourly cap so a wide attack cannot flood the channel. -- Anomaly alerts when requests or bytes per minute or hour cross a threshold, - for a single client, its surrounding netblock, an AS number, a named netblock - or the whole service. -- Observe mode: log and alert on every decision while refusing nothing, for the - first days in front of a new service. +- Anomaly alerts when requests or bytes per minute or hour cross a threshold you + set, for a single client, its surrounding netblock, an AS number, a named + netblock or the whole service. +- Observe mode: log and alert on every decision while refusing nothing. - Request log: one JSON object per request on stdout with the usual web log fields, the decision taken and why, AS number and country, and timings. - Optionally also sent to a remote syslog or RELP endpoint. + Optionally also sent to a remote syslog server. - Prometheus metrics on their own port. -- State (bans, offender history, hour and day counters, reputation cache) held - in memory and saved as readable, hand-editable JSON files, written atomically. - Nothing is read from disk while serving a request. +- State (bans with their notes, each client's counters and history, the GeoJS + answers, the reputation cache, the alerting state) held in memory and kept in + readable JSON files, written regularly and at every stop, so a restart loses + nothing. Edit a file, or add a rule file, and the running `smallwebwaf` picks + up the change. Nothing is read from disk while serving a request. - A small admin endpoint for health checks, listing, adding and lifting bans, and asking why a given address was refused. @@ -96,16 +122,22 @@ For each request `smallwebwaf`: - works out who the client really is; - lets it straight through if it is on the bypass list, refuses it if it is on the deny list or currently banned; -- looks up its AS number and country, and any cached reputation verdict; +- looks up its AS number and country, and refuses it if that country is denied, + or is not among the only ones allowed; +- checks for a cached reputation verdict; - picks the client's limit percentage from those; -- checks the minute, hour and day request counters against the limits, and - answers 429 if one is exceeded; -- checks the request against the rule files and the Core Rule Set; -- forwards it to the app and streams the response back; -- counts the bytes, records any offence, bans the client if it has collected - enough of them, sends any alerts that are due, and writes the log line. +- checks the minute, hour and day request counters against the limits, and bans + the client if it breaks one; +- checks the request against the rule files and the Core Rule Set, and bans the + client at once for a clear sign of attack; +- forwards it to the app and streams the response back, within the size and time + limits; +- counts the bytes and any refusal by the rule files or the Core Rule Set, bans + the client if it broke a limit, updates its history, sends any alerts that are + due, and writes the log line. -A minimal deployment beside an app in docker-compose: +A minimal deployment beside an app in docker-compose. `UPSTREAM_URL` is the only +setting: ```yaml services: @@ -117,33 +149,67 @@ services: image: /smallwebwaf: environment: UPSTREAM_URL: http://app:3000 - TRUSTED_PROXIES: 172.18.0.0/16 - MODE: observe - RATE_LIMIT_PER_MINUTE: "120" - RATE_LIMIT_PER_HOUR: "2000" - RATE_LIMIT_PER_DAY: "10000" - ASN_LIMIT_PERCENT: AS14061:50,AS16276:50 - COUNTRY_LIMIT_PERCENT: CN:25 - ALERT_NTFY_URL: https://ntfy.example.invalid/alerts volumes: [waf-state:/data] networks: [internal, traefik] labels: traefik.enable: "true" traefik.http.routers.app.rule: Host(`app.example.invalid`) traefik.http.services.app.loadbalancer.server.port: "8080" + +volumes: + waf-state: + +networks: + internal: + traefik: + external: true ``` -A rule file is one rule per line: a name, what to match against, what to do, -and a regex. +The volume keeps bans and client history in a place you choose; without it +`smallwebwaf` still starts, on a volume docker creates for it. + +A rule file is one rule per line: a name, what to match against, what to do, and +a regex. ``` -env-file path offence:3 (?i)/\.env(\.[a-z]+)?$ -scanner-agent user_agent ban (?i)\b(sqlmap|nikto|nuclei|wpscan)\b +env-file path ban (?i)^/\.env(\.[a-z]+)?$ +scanner-agent user_agent ban (?i)\b(sqlmap|nikto|nuclei|wpscan)\b ``` -[`SPEC.md`](SPEC.md) has the full design: every environment variable, the rule -file format, the state files, the log fields, the metrics, failure behaviour, -the build order, and the design questions still open for the owner. +[`SPEC.md`](SPEC.md) has the full design: every environment variable, the ban +rules, the rule file format, the state files, the log fields, the metrics, +failure behaviour and the build order. + +## Country and AS number lookup + +AS number and country lookups, and the country lists and biased limits that use +them, are off until you choose one of two sources with `LOOKUP_SOURCE`. + +`LOOKUP_SOURCE=file` reads the free IPinfo Lite database (`ipinfo_lite.mmdb`). +You download it with your own IPinfo account, mount the directory that holds it +into the container, point `LOOKUP_DB_PATH` at the file and refresh it when you +choose; `smallwebwaf` never downloads it itself, and reads it again when you +replace it. It has to be the directory rather than the file itself: docker does +not show a single mounted file being replaced, so a refresh would go unseen. +IPinfo releases it under the Creative Commons Attribution-ShareAlike 4.0 +International License and asks for attribution, in its own words on +https://ipinfo.io/lite: "The attribution requirements can be met by giving our +service credit as your data source. Simply place a link to IPinfo on the +website, application, or social media account that uses our data." Its example +of such a credit is a link mentioning "IP address data is powered by IPinfo". A +service that uses the database through `smallwebwaf` should carry that link. + +`LOOKUP_SOURCE=geojs` asks the free GeoJS web service instead, with no account +and no file. Every new visitor's address is sent to GeoJS. Each answer is kept +for seven days, across restarts, and many addresses are asked about in one +request. GeoJS publishes no rate limit but may block a caller it thinks asks too +much; while it is not answering, new visitors count as coming from an unknown +country, which `EXCLUSIVELY_ALLOWED_COUNTRIES` refuses. + +Neither source can place a private address, so a client on one, such as a +visitor on your local network, another container or your monitoring, has no +country: `EXCLUSIVELY_ALLOWED_COUNTRIES` refuses it unless you list it in +`ALLOW_NETS`. Such addresses are never sent to GeoJS. ## Documents diff --git a/SPEC.md b/SPEC.md index 2fb5ec6..d94938b 100644 --- a/SPEC.md +++ b/SPEC.md @@ -1,20 +1,25 @@ # smallwebwaf SPEC (draft): protective reverse-proxy sidecar -Status: second draft, with the owner's rulings on storage, logging and metrics -applied. Nothing has been built yet. `EVALUATION.md` beside this file explains why no existing tool was -chosen. Points that turn on an owner decision are collected under "Questions for -the owner" and are marked "(open)" where they appear. +Status: third draft, with the owner's rulings to date applied. Nothing has been +built yet. `EVALUATION.md` beside this file explains why no existing tool was +chosen. ## Purpose One small container that sits between traefik and one application container. The traefik router for the public hostname points at the sidecar; the sidecar forwards to the application. The sidecar limits request and byte rates per -client, detects attacks, blocks abusers for a time and bans repeat offenders, -consults IP reputation sources, looks up the AS number and country of each -client, scales its limits for listed AS numbers and countries, and sends alerts. -Everything is configured by environment variables, apart from the attack -detection rules, which are read from a directory of hand-editable text files. +client, bounds the size and duration of every request and response, detects +attacks, bans abusers (briefly at first, for seven days on a clear sign of +attack, permanently when they keep at it), consults IP reputation sources, looks +up the AS number and country of each client, can refuse whole countries, lowers +its limits for listed AS numbers and countries, and sends alerts. + +It is meant to protect an app on the open internet from the first request with +one setting, `UPSTREAM_URL`: every other setting has a default chosen for a +service facing the internet in 2026. Everything is configured by environment +variables, apart from the attack detection rules, which are read from a +directory of hand-editable text files. ## Non-goals @@ -24,170 +29,303 @@ detection rules, which are read from a directory of hand-editable text files. the sidecar and the app. - Defence against traffic floods that saturate the host's network link. That needs help upstream of the host. -- A web UI or a configuration file. Settings are environment variables; the - only files read are the rule files, which hold one regex per line and nothing - more elaborate. -- Sharing ban state between sidecars in the first version (open, see - Questions). +- A web UI or a configuration file. Settings are environment variables. Apart + from its own state files and the lookup database, the only files read are the + rule files, which hold one regex per line and nothing more elaborate. +- Sharing bans between sidecars in the first version: each sidecar keeps its + own. Running one CrowdSec engine per host, which every sidecar would report + its bans to, is reconsidered once real ban volumes are known. Reading a + CrowdSec decision list is supported (see Reputation). ## Architecture - One statically linked Go binary in one container, running as a non-root user, - read-only root filesystem, one writable volume for state. + read-only root filesystem, one writable volume for state. The image declares + that volume (`/data`), so docker supplies an anonymous one when none is + mounted, and a sidecar started with only `UPSTREAM_URL` has somewhere to + write. - Three listeners, of which only the first is routed by traefik: - the proxy listener; - a metrics listener serving Prometheus metrics, which can be switched off; - an admin listener for health and ban management. - Components inside the process: - Client identification: works out the real client IP from - `X-Forwarded-For`, trusting only configured proxy netblocks. - - Lookup: AS number and country from local database files, held in memory. + `X-Forwarded-For`, trusting only the proxy netblocks in `TRUSTED_PROXIES`, + by default the private address ranges. + - Lookup: AS number and country, from one of two sources the operator + chooses: a database file the operator supplies, held in memory, or the + GeoJS web service, whose answers are kept for seven days. - Reputation: static blocklists fetched on a schedule; DNSBL and reputation API queries made in the background and cached. - Counters: per-client request and byte counts per minute, hour and day, - held in memory; the hour and day counts are written out so they survive a - restart. - - Attack detection: regex rules read at start from a directory of plain - text rule files (see "Rule files"), Coraza with the OWASP Core Rule Set, - and simple signals (requests for listed trap paths, bursts of error - responses from the app). - - Ban ledger: offences, active bans and ban history, held in memory and - written to disk as JSON (see "Persistent state"). No database of any kind - is used. + held in memory and written out with the rest of the state, so they survive + a restart. + - Attack detection: regex rules read from a directory of plain text rule + files and read again whenever the files change (see "Rule files"), Coraza + with the OWASP Core Rule Set, and simple signals (requests for listed trap + paths, bursts of requests refused by the rule files or the Core Rule Set). + - Ban ledger: active and past bans with their notes, and each client's + history, held in memory and kept in JSON files that the sidecar watches + for an admin's edits (see "Bans" and "Persistent state"). No database of + any kind is used. - Request log: one JSON object per request on stdout, optionally also sent - to a remote syslog or RELP endpoint (see "Request log"). + to a remote syslog server (see "Request log"). - Metrics: Prometheus counters and gauges on their own listener (see "Metrics endpoint"). - Alerting: a queue with de-duplication feeding webhook, Slack and ntfy senders. - Proxy: the standard library's `net/http/httputil.ReverseProxy`, streaming - in both directions, with WebSocket upgrade support. + in both directions within the size and time limits, with WebSocket upgrade + support. - Proposed libraries (to be checked against the owner's Go dependency defaults before any are added): - standard library for the proxy, HTTP clients, DNS, logging (`log/slog`), state files (`encoding/json`, `os.Rename`); - `github.com/corazawaf/coraza/v3` and `github.com/corazawaf/coraza-coreruleset` for attack detection; - - `github.com/oschwald/maxminddb-golang` to read `.mmdb` lookup databases; + - `github.com/oschwald/maxminddb-golang/v2`, the current major version, to + read the lookup database; + - `github.com/fsnotify/fsnotify` to notice files that are edited or added + while the sidecar runs; - `github.com/prometheus/client_golang` for metrics; - - remote log sending: the standard library's `log/syslog` is frozen and - does not write the current syslog format (RFC 5424), and no widely used Go - RELP client was found, so the library for this is open (see Questions). + - remote log sending is written in this project: the standard library's + `log/syslog` is frozen and writes only the older syslog format, not the + current one (RFC 5424). RELP is not in the first release; it is revisited + against the available packages afterwards. ## Data flow for one request Steps run in this order; the first step that produces a final answer ends -processing. +processing. The size and time limits (see "Configuration surface") apply to the +whole exchange. - Identify the client. - If the TCP peer is inside `TRUSTED_PROXIES`, walk `X-Forwarded-For` from - the right and take the first address not inside `TRUSTED_PROXIES`. - Otherwise use the TCP peer address and ignore the header. + the right and take the first address not inside `TRUSTED_PROXIES`. If + every address in the header is inside `TRUSTED_PROXIES`, take the + leftmost, so a visitor on a private network who comes through traefik is + known by its own address, not traefik's. If there is no header, as when + another container calls the sidecar directly, the TCP peer is the client. + - If the TCP peer is outside `TRUSTED_PROXIES`, it is the client, and the + header is ignored. - IPv6 clients are grouped by prefix (`IPV6_GROUP_PREFIX`, default 64) for - counting and banning, because one abuser usually controls a whole /64. + counting and banning, because one abuser usually controls a whole /64. A + client is therefore one IPv4 address or one IPv6 group, a /64 by default. - Static lists. - In `ALLOW_NETS`: skip every check below and forward. Still counted for anomaly alerts. - In `DENY_NETS`: refuse. -- Ban ledger. An active ban: refuse (`BAN_RESPONSE`). -- Lookup AS number and country. Unknown is a valid result and changes nothing. +- Ban ledger. An active ban on the client's netblock: refuse with + `BAN_RESPONSE`. If a clear sign of attack caused the ban, the ban becomes + permanent (see "Bans"). +- Look up the AS number and country, unless `LOOKUP_SOURCE` is `off`. With + GeoJS, a client whose answer is not yet kept waits for it, up to + `LOOKUP_TIMEOUT`. A client the lookup cannot place, which includes every + private, loopback and link-local address, has an unknown country and AS + number: the exclusive country list refuses it, the biased thresholds give it + `UNKNOWN_LIMIT_PERCENT`, and nothing else treats it differently. +- Country lists. A client whose country is in `DENIED_COUNTRIES`, or, when + `EXCLUSIVELY_ALLOWED_COUNTRIES` is set, is not in it, is refused with + `BAN_RESPONSE`. Every step up to here needs only the client's address, so the + request body has not been read yet, and the refusal skips everything below, + the rule files and the Core Rule Set included. It is counted in the metrics, + but it is not an offence and makes no ban. - Reputation. - Address inside a fetched blocklist: apply `BLOCKLIST_ACTION`. - Cached DNSBL or reputation API result: apply `REPUTATION_ACTION`. - No cached result: queue a background query and carry on. A first request is never delayed by a reputation query. - Work out the client's limit percentage: the lowest of the percentages that - apply (AS number, country, reputation). 100 if none applies. 0 means refuse. - (Lowest-wins versus multiplying is open.) + apply (AS number, country, reputation), or 100 if none applies. - Request rate limits. Unless the client is in `RATE_LIMIT_EXEMPT_NETS`, check - the minute, hour and day counters against each configured limit times the - percentage. Over any limit: answer 429 with `Retry-After`, record an offence. -- Rule files. The request is checked against the rules loaded from - `RULES_DIR`, in file name order then line order. Each rule that matches takes - its action: record offences, refuse with 403, ban at once, or only log. - Matching stops at the first rule that refuses or bans. -- Core Rule Set inspection of the request (headers, URL, and body up to - `WAF_BODY_LIMIT`). In `block` mode a match at or over the anomaly threshold is - refused with 403 and recorded as an offence; in `detect` mode it is only - logged and alerted. + the minute, hour and day counters against each limit times the percentage. A + request over any of them breaks that limit: it is refused with `BAN_RESPONSE` + and the client is banned (see "Bans"). +- Rule files. The request is checked against the rules loaded from `RULES_DIR`, + in file name order then line order. Each rule that matches takes its action: + only log, refuse with 403, or refuse and ban. Matching stops at the first rule + that refuses or bans. +- Core Rule Set inspection of the request: its headers, its URL, and its body up + to `WAF_BODY_LIMIT` when the body is form data, multipart, JSON or XML, the + kinds the Core Rule Set can read. Any other body, such as a git push or a + container image layer, reaches the app uninspected, and so does a JSON or XML + body larger than `WAF_BODY_LIMIT`, which cannot be read in part. Responses are + not inspected. In `block` mode, the default, a match at or over the anomaly + threshold is refused with 403; in `detect` mode it is only logged and alerted. - Forward to `UPSTREAM_URL`, streaming. Add `X-Forwarded-For` and, if enabled, `X-Client-ASN` and `X-Client-Country` for the app's own logs. - After the response. - - Add response bytes (and request body bytes) to the byte counters. Once a - client's byte total passes a byte limit times the percentage, following - requests get 429 until the window moves on, and an offence is recorded. A - response already in progress is not cut off. - - Count 401, 403 and 404 responses from the app per client; passing - `ERROR_BURST_THRESHOLD` within a minute records an offence. - - Update anomaly counters and evaluate alert thresholds. -- Offences and bans. - - `BAN_AFTER_OFFENCES` offences within `OFFENCE_WINDOW` creates a ban. - - Ban length follows `BAN_DURATIONS` by the number of earlier bans for that - client within `BAN_HISTORY_WINDOW`: first ban the first value, second ban - the second value, and so on. - - After `PERMANENT_BAN_AFTER` bans within `BAN_HISTORY_WINDOW`, the ban has - no expiry. - - A request for a path in `TRAP_PATHS` counts as `TRAP_OFFENCE_WEIGHT` - offences at once. + - Add response bytes (and request body bytes) to the byte counters. A client + whose byte total passes a byte limit times the percentage has broken that + limit and is banned. The response in progress is not cut off. + - Count the client's requests that the sidecar refused after a rule file or + Core Rule Set match; answers the app gave are not counted. More than + `ERROR_BURST_THRESHOLD` of them within a minute breaks a limit and bans + the client. + - Update the client's history and the anomaly counters, and evaluate alert + thresholds. ## Counting method - Each window (minute, hour, day) uses two adjacent fixed buckets per client, - with the previous bucket weighted by how much of it still overlaps the - sliding window. This costs a few integers per client per window and avoids - the burst at bucket boundaries that a single fixed bucket allows. -- Counters are read and updated in memory only. The hour and day counters are - written to disk on a timer and at shutdown and loaded again at start (see - "Persistent state"), so a restart does not hand every client a fresh daily - allowance. Minute counters are not written; a restart forgives at most one - minute of counting. Buckets whose time has passed are discarded on load. -- Memory is bounded by `MAX_TRACKED_CLIENTS`; when full, the least recently seen - clients with no recent offences are dropped first. + with the previous bucket weighted by how much of it still overlaps the sliding + window. This costs a few integers per client per window and avoids the burst + at bucket boundaries that a single fixed bucket allows. +- Counters are read and updated in memory; a request never waits on the disk. + They are written to `clients.json` every `STATE_COUNTER_INTERVAL` and at + shutdown and loaded again at start (see "Persistent state"), so a restart does + not hand every client a fresh allowance. Buckets whose time has passed are + discarded on load. +- Memory is bounded by `MAX_TRACKED_CLIENTS`, `MAX_BANS` and the number of GeoJS + answers kept (see "Configuration surface"). When the table of clients is full, + the least recently seen client is dropped first, with its history. When + `MAX_BANS` bans the sidecar made are held, the one that has gone longest + without a request from its netblock is dropped first, whether it is past, + active or permanent. Bans an admin made are never dropped, and there are as + many as the admin adds (see "Bans"). + +## Bans + +A ban refuses every request from a netblock, answering with `BAN_RESPONSE` +(`403` by default), until the ban ends. A permanent ban does not run out: it +ends when an admin lifts it or, if the sidecar made it, when a new ban is made +while `MAX_BANS` bans the sidecar made are held and it is the one that has gone +longest without a request. A scanner whose ban was dropped that way is refused +and banned again by its next probe. A ban an admin made, whose cause is `admin`, +is never dropped and does not count toward `MAX_BANS`, so such bans can never +fill the table, however many there are. An admin who wants to keep a ban the +sidecar made sets its cause to `admin`. + +The netblock a ban covers is the client: its IPv4 address, or its IPv6 group of +`IPV6_GROUP_PREFIX` (a /64 by default). `BAN_SCOPE_V4_PREFIX` can widen an IPv4 +ban to the surrounding netblock. At the defaults a ban covers one address or one +/64, so a permanent ban does not reach neighbours who did nothing. + +An offence is a request the sidecar holds against the client: one that carries a +clear sign of attack, breaks a limit, or is refused by a rule file or the Core +Rule Set. Offences are counted by kind in the client's history. Two kinds lead +to a ban, each by its own rule: + +- A clear sign of attack: a request that matches a rule file rule whose action + is `ban`, or asks for a path in `TRAP_PATHS`. Examples are a probe for a + `.env` file or a `.git` directory, and a known scanner's user agent. + - The first one bans the netblock for `ATTACK_BAN_DURATION` (default `7d`). + - Any further request from the netblock while that ban lasts shows it is + malicious: the ban becomes permanent. + - Once that ban has run out, the netblock is served like any other, but its + next clear sign of attack bans it permanently at once. +- A broken limit: a request over a request limit, a response that takes the + client's byte total past a byte limit, or more than `ERROR_BURST_THRESHOLD` + requests within a minute refused after a rule file or Core Rule Set match. + - The first such ban, or one that comes more than `LIMIT_BAN_REPEAT_WINDOW` + (default `24h`) after the last such ban ended, lasts `LIMIT_BAN_DURATION` + (default `1h`). + - Breaking a limit again within that window makes the new ban three times as + long as the last one: one hour, then 3, 9, 27 and 81 hours. + - A ban that would be longer than `MAX_BAN_DURATION` (default `7d`) is + permanent instead; with the defaults, that is the sixth ban in a row. + - Each such ban sets the client's counters for every limit back to zero, so + once it ends only new requests can break a limit again; the client's + history keeps its totals. + +A Core Rule Set match refuses only the request it matched, without a ban, +because the Core Rule Set has false positives; a `block` rule does the same. +Those refusals count toward the error burst, so a client that keeps setting them +off is banned under the second rule. Requests refused under a ban are not +counted against limits, but they are counted in the client's history and in the +ban's notes. + +Every ban carries notes with what an admin needs to decide whether to lift it +(see "Persistent state"). An admin can add or lift a ban at any time by editing +`bans.json`, or through the admin listener when `ADMIN_TOKEN` is set. A ban +lifted before it ends is kept, marked as lifted, and does not count toward a +longer ban; deleting its entry from `bans.json` forgets it entirely. + +Clients of the AS numbers and countries listed in the biased thresholds, and +clients listed by a reputation source whose action is `limit:`, get +lower limits (see "Biased thresholds"), so the same rules ban them after fewer +requests. For them the result is always a ban, first a temporary one and then, +for repeated abuse, a permanent one. + +Some refusals make no ban at all. `DENY_NETS`, a blocklist or reputation hit +whose action is `deny`, and the country lists (`DENIED_COUNTRIES`, +`EXCLUSIVELY_ALLOWED_COUNTRIES`) refuse each request they cover and record +nothing in `bans.json`. ## Configuration surface Conventions: lists are comma separated; netblocks are CIDR (a bare address means /32 or /128); durations use Go syntax plus `d` for days (`90s`, `15m`, `24h`, -`7d`); byte sizes accept `K`, `M`, `G` suffixes; an unset limit means that -limit is off. Every variable may instead be given as `NAME_FILE` pointing at a -file holding the value, for secrets and long lists. Invalid configuration stops -the process at start with a message naming the variable. At start the effective -configuration is logged with secrets masked. +`7d`); byte sizes accept `K`, `M`, `G` suffixes. Countries are two-letter ISO +codes in either case (`de` and `DE` are the same); a code that is not a country +code, such as `nk` (North Korea is `kp`), stops the start with a message naming +it. + +- Only `UPSTREAM_URL` is required. Every other setting has a default chosen for + a service facing the internet in 2026, or stays off until the operator + supplies or chooses what it needs: an alert destination, an account key, a + lookup source. +- Any limit or threshold can be switched off with the value `off`. +- A list set to an empty value is an empty list, and replaces the default. +- Every variable may instead be given as `NAME_FILE` pointing at a file holding + the value, for secrets and long lists. +- Settings, including `NAME_FILE` files, are read once at start; changing one + means restarting the container. The files the sidecar watches while it runs + are its state files, its rule files and the lookup database. +- Invalid configuration stops the process at start with a message naming the + variable. At start the effective configuration is logged with secrets masked. + +The settings, by group: - Core - `UPSTREAM_URL` (required): the application, for example `http://gitea:3000`. - `LISTEN_ADDR` (default `:8080`): proxy listener. - `ADMIN_LISTEN_ADDR` (default `127.0.0.1:9090`): admin listener. - - `ADMIN_TOKEN`: bearer token required for ban management endpoints. - - `INSTANCE_NAME`: included in every log line, metric and alert, for example - `fsn1app1/gitea`. + - `ADMIN_TOKEN`: bearer token required for the ban management endpoints. + Unset by default, which switches those endpoints off; bans are then + managed by editing `bans.json`. + - `INSTANCE_NAME` (default: the host name in `UPSTREAM_URL`, for example + `gitea`): included in every log line, metric and alert. Set it, for + example to `fsn1app1/gitea`, when several sidecars report to one place. - `MODE` (default `enforce`): `enforce`, or `observe` to log and alert on - every decision while refusing nothing. Meant for the first days of a - rollout. - - `TRUSTED_PROXIES` (required): netblocks whose `X-Forwarded-For` is - believed, normally the docker network traefik reaches the sidecar from. + every decision while refusing nothing. + - `TRUSTED_PROXIES` (default `10.0.0.0/8,172.16.0.0/12,192.168.0.0/16`, the + private address ranges): netblocks whose `X-Forwarded-For` is believed, + normally where traefik reaches the sidecar from. The header is used only + when the TCP peer is inside this list; otherwise the peer's own address is + the client, so a client that connects directly cannot claim another + address. A list given replaces the default; set but empty, it trusts + nothing. - `IPV6_GROUP_PREFIX` (default `64`). - - `MAX_TRACKED_CLIENTS` (default `500000`). + - `MAX_TRACKED_CLIENTS` (default `20000`): clients held in memory and in + `clients.json` (see "Persistent state"). + - `MAX_BANS` (default `5000`): bans the sidecar made, past, active and + permanent, held in memory and in `bans.json`. Bans an admin made are kept + besides these and never dropped (see "Bans"). - Persistent state - - `STATE_DIR` (default `/data`): the JSON state files and downloaded lookup - databases. - - `STATE_WRITE_DELAY` (default `2s`): after a ban or offence changes, wait - this long before writing, so a burst of changes becomes one write. - - `STATE_COUNTER_INTERVAL` (default `60s`): how often the hour and day - counters and the reputation cache are written. + - `STATE_DIR` (default `/data`): the JSON state files. The image declares + `/data` as a volume, so it is writable even when nothing is mounted; a + named volume, as in the examples below, keeps the state in a place that + outlives the container. + - `STATE_WRITE_DELAY` (default `10s`): after a ban is made, lifted or made + permanent, `bans.json` is written this long later with every change made + in between, so a burst of changes becomes one write. + - `STATE_COUNTER_INTERVAL` (default `15m`): how often `clients.json`, + `lookups.json`, `reputation.json` and `alerts.json` are written, and + `bans.json` when only the counts in its notes changed. - Logging - The request log on stdout is always on and has no switch. - `LOG_LEVEL` (default `info`): for the process's own messages (start-up, fetch failures, state writes), which are JSON lines on stdout too, marked `"type":"process"`. It does not filter the request log. - `LOG_REQUEST_HEADERS` (default - `accept,accept-language,accept-encoding,content-type,origin,range`): - extra request headers to record. `Authorization`, `Cookie` and - `Set-Cookie` values are never logged, only whether they were present. + `accept,accept-language,accept-encoding,content-type,origin,range`): extra + request headers to record. `Authorization`, `Cookie` and `Set-Cookie` + values are never logged, only whether they were present. - `LOG_REMOTE_URL`: when set, every log line is also sent to this endpoint. Forms: `syslog+udp://host:514`, `syslog+tcp://host:514`, - `syslog+tls://host:6514`, `relp://host:2514`, `relp+tls://host:2514`. + `syslog+tls://host:6514`. - `LOG_REMOTE_TLS_CA_FILE`: optional CA certificate for the `+tls` forms. - `LOG_REMOTE_BUFFER` (default `10000`): lines held in memory while the endpoint is unreachable; when full the oldest are dropped and counted. @@ -204,34 +342,147 @@ configuration is logged with secrets masked. which is the usual arrangement for a scraper on a private network. - Static lists - `ALLOW_NETS`: bypass everything (monitoring, the owner's own networks). - - `RATE_LIMIT_EXEMPT_NETS`: bypass request and byte limits only; attack - detection and bans still apply. + - `RATE_LIMIT_EXEMPT_NETS`: bypass request and byte limits only; the error + burst threshold, attack detection and bans still apply. - `DENY_NETS`: always refused. -- Request rate limits, per client (R1, R2) - - `RATE_LIMIT_PER_MINUTE`, `RATE_LIMIT_PER_HOUR`, `RATE_LIMIT_PER_DAY`. +- Request rate limits, per client (R1, R2). Breaking one bans the client (see + "Bans"), so the defaults sit several times above what one busy person + produces: a browser loading a heavy page makes a few hundred requests, a git + clone a handful, and several people often share one address. + - `RATE_LIMIT_PER_MINUTE` (default `1000`), `RATE_LIMIT_PER_HOUR` (default + `10000`), `RATE_LIMIT_PER_DAY` (default `50000`). - `RATE_LIMIT_EXEMPT_PATHS`: path prefixes not counted (static assets, health checks). -- Byte limits, per client - - `BYTES_LIMIT_PER_MINUTE`, `BYTES_LIMIT_PER_HOUR`, `BYTES_LIMIT_PER_DAY`. +- Byte limits, per client. A response's bytes are counted when it ends, so every + default sits above the largest response allowed (`CLIENT_RESPONSE_MAX_BYTES`, + 5 GB) and no single download breaks one. + - `BYTES_LIMIT_PER_MINUTE` (default `10G`), `BYTES_LIMIT_PER_HOUR` (default + `20G`), `BYTES_LIMIT_PER_DAY` (default `50G`). - `BYTES_COUNT` (default `both`): `response`, `request` or `both`. -- Lookup of AS number and country (R7) - - `ASN_DB_PATH`, `COUNTRY_DB_PATH`: `.mmdb` files mounted into the - container; or - - `ASN_DB_URL`, `COUNTRY_DB_URL`: downloaded into `STATE_DIR` at start and - every `LOOKUP_DB_REFRESH` (default `7d`). One file may serve both. - - Known free sources in this format: IPinfo Lite (country and AS number in - one file, free account token), DB-IP Lite, MaxMind GeoLite2 (free - account). Choice is open. +- Size and time limits, per request, in both directions. The client-facing + limits apply between the client and the sidecar, the app-facing ones between + the sidecar and the app. Bodies stream straight through, so a request body + reaches the app while the client is still sending it, and of two matching + limits the lower one acts first. + - `CLIENT_REQUEST_TIMEOUT` (default `60s`): how long a client may take to + send its whole request, headers and body. + - `CLIENT_REQUEST_MAX_BYTES` (default `100M`): the largest request body a + client may send. + - `CLIENT_REQUEST_HEADER_MAX_BYTES` (default `32K`): the largest request + line and headers a client may send. Over it, the sidecar answers `431` and + closes the connection, and nothing reaches the app. + - `CLIENT_IDLE_TIMEOUT` (default `120s`): how long a kept-open connection + may wait for its next request before the sidecar closes it. It is longer + than the 90 seconds after which traefik, by default, closes a connection + it is not using, so traefik closes first and never sends a request on a + connection the sidecar is closing. + - `CLIENT_RESPONSE_TIMEOUT` (default `30m`): how long the sidecar may take + to deliver one response to the client, from the end of the request to the + last byte. + - `CLIENT_RESPONSE_MAX_BYTES` (default `5G`): the largest response body sent + to a client. + - `UPSTREAM_REQUEST_TIMEOUT` (default `60s`): how long the sidecar may take + to connect to the app and send it the whole request. + - `UPSTREAM_REQUEST_MAX_BYTES` (default `100M`): the largest request body + sent to the app. + - `UPSTREAM_RESPONSE_TIMEOUT` (default `30m`): how long the app may take to + send one whole response, from the end of the request to the last byte. + - `UPSTREAM_RESPONSE_MAX_BYTES` (default `5G`): the largest response body + taken from the app. + - When a limit is passed before the response has started, the sidecar + answers itself: `413` for a request body that is too large, `408` for a + client that is too slow, `502` for a response that is too large, `504` for + an app that is too slow. A request that announces a body larger than its + limit is refused before anything reaches the app. Once the response has + started it can only be cut off, and the connection is closed. + - A WebSocket connection leaves these limits behind once it is upgraded: it + stays open until either side closes it. +- Lookup of AS number and country (R7). Off by default: the file needs an + account only the operator can hold, and GeoJS is told every visitor's address. + - `LOOKUP_SOURCE` (default `off`): `off`, `file` or `geojs`. + - `LOOKUP_DB_PATH`: the file for `file`, the IPinfo Lite database in its + `.mmdb` form (`ipinfo_lite.mmdb`), one file that carries both country and + AS number; the sidecar reads its `asn`, `as_name` and `country_code` + fields. The operator downloads it with a free IPinfo account and mounts it + read-only. The sidecar never fetches it and holds no account or token for + it. A file that is missing or unreadable at start stops the start. + - Refreshing the file is the operator's business; IPinfo updates it daily. + The sidecar notices when the file is replaced and reads it again; a + replacement it cannot read is ignored, logged and sent as one `file_error` + alert, and the file it has stays in use. Mount the directory that holds + the file rather than the file itself: docker does not show a single + mounted file being replaced on the host. + - `geojs` asks the free GeoJS web service, which needs no account or key, + about several addresses in one request + (`https://get.geojs.io/v1/ip/geo.json?ip=a,b,c`), and reads each answer's + `country_code`, `asn` and `organization_name` (the AS name). A + `country_code` of `null`, which GeoJS gives for some ranges, and an `asn` + of `64512`, which it gives when it knows none, count as unknown. GeoJS is + told the address of every new visitor, except private, loopback and + link-local addresses, which no source can place and which are never sent. + - Each GeoJS answer is kept for 7 days in memory and in `lookups.json`, + apart from the table of clients, so it survives a restart and outlasts a + client dropped from that table; after 7 days the client's next request + asks again. Up to 100,000 answers are kept, about 15 MiB; when that many + are held, the answer used longest ago goes first. + - `LOOKUP_TIMEOUT` (default `1s`): how long a client with no kept answer + waits for one; GeoJS normally answers in a fraction of that. At most one + request to GeoJS is under way at a time, the addresses that arrive + meanwhile are asked about together in the next one, and a request that + takes longer than `LOOKUP_TIMEOUT` is abandoned. A client whose answer has + not come in time counts as unknown until it comes: its later requests do + not wait, and its address is asked about again in the background. + - GeoJS publishes no rate limit, but its terms forbid "an excessive amount + of API requests", judged by GeoJS alone, and let it block a caller. While + GeoJS is slow, down or refusing the sidecar, clients with a kept answer + are unaffected and new clients count as unknown, so + `EXCLUSIVELY_ALLOWED_COUNTRIES`, when set, refuses them. The sidecar keeps + asking with backoff and sends one `source_failure` alert per cooldown. A + service that cannot accept this uses the file. + - One source at a time: `LOOKUP_SOURCE=file` without `LOOKUP_DB_PATH`, or + `LOOKUP_DB_PATH` with any other `LOOKUP_SOURCE`, stops the start with a + message naming both. So does a setting that needs lookups while + `LOOKUP_SOURCE` is `off`: the country lists and the biased thresholds + below, `ADD_LOOKUP_HEADERS`, and the per-AS-number anomaly thresholds. - `ADD_LOOKUP_HEADERS` (default `false`): pass `X-Client-ASN` and `X-Client-Country` to the app. - - Missing or unreadable database: start anyway, treat every client as - unknown, log a warning and send one alert. -- Biased thresholds (R8) +- Country lists. Both are empty by default, since they need a lookup source. + Clients in `ALLOW_NETS` are not checked. + - `DENIED_COUNTRIES`: for example `cn,ru,kp,ir,ua,by`. Every request from a + listed country is refused. + - `EXCLUSIVELY_ALLOWED_COUNTRIES`: for example `us,de`. Every request from + any other country is refused, and so is every request from a client the + lookup cannot place: one whose address is missing from the database, one + GeoJS has not answered for in time, and any client on a private address, + such as a visitor on the local network, another container or internal + monitoring. Those that should reach the app go in `ALLOW_NETS`. A list + that let unplaced clients through would let every new client in whenever + GeoJS stops answering. + - Both may be set. `DENIED_COUNTRIES` then adds nothing, since the exclusive + list already refuses every other country, and a code on both lists stops + the start with a message naming it. + - A refused request is answered with `BAN_RESPONSE`, as a banned client is, + as soon as the client's address has been looked up, before its body is + read. It skips the reputation checks, the limits, the rule files and the + Core Rule Set, is logged with the action `country_denied`, and is counted + in the metrics. It is a refusal, not a ban: it is not an offence and makes + no ban record. +- Biased thresholds (R8). The lists are empty by default: they need a lookup + source, which only the operator can choose. - `ASN_LIMIT_PERCENT`: for example `AS14061:50,AS16276:50,AS45102:25`. - Listed AS numbers get that percentage of every request and byte limit. - `0` refuses outright. + Clients in a listed AS number get that percentage of every request and + byte limit, so the rules in "Bans" ban them after fewer requests than + others. `0` is a zero allowance: the client's first request breaks a limit + and bans it. - `COUNTRY_LIMIT_PERCENT`: same form with ISO country codes, for example - `CN:25,RU:50`. + `CN:25,RU:50`. Each client from a listed country gets that percentage of + the normal per-client limits. + - No budget is shared by a whole country or AS number: one abuser could use + it up and lock out everyone else there, a denial of service nobody chose + and the one this tool exists to prevent. Refusing a whole country is left + to the operator's choice, through the country lists. + - When more than one percentage applies to a client (its AS number, its + country, a reputation hit), the lowest applies. - `ASN_BYTES_PERCENT`, `COUNTRY_BYTES_PERCENT`: optional overrides applied to byte limits only, when the byte percentage should differ from the request percentage. @@ -239,43 +490,104 @@ configuration is logged with secrets masked. place. - `ASN_LIMIT_PERCENT_URL`: optional URL of a text file of `AS:percent` lines, so one abuse-source list can be shared by every sidecar in the - fleet. + fleet. It is fetched and refreshed like the blocklists. - Attack detection (R9) - - `RULES_DIR` (default `/etc/smallwebwaf/rules.d`): directory of rule files - read once at start; format under "Rule files". The image ships a default - file there. Mounting a directory over it replaces the defaults; mounting - single files into it adds to them. + - `RULES_DIR` (default `/etc/smallwebwaf/rules.d`): directory of rule files, + read at start and again whenever a file in it changes; format under "Rule + files". - `RULES_ENABLED` (default `true`): `false` skips rule files entirely. - - `WAF_MODE` (default `detect`): `off`, `detect`, `block`. + - `WAF_MODE` (default `block`): `off`, `detect` (log and alert only), or + `block`. - `WAF_PARANOIA_LEVEL` (default `1`), `WAF_ANOMALY_THRESHOLD` (default `5`): - the Core Rule Set's own two tuning values. - - `WAF_DISABLED_RULES`: rule ids to switch off when an app trips a false - positive. + the Core Rule Set's own two tuning values, at the Core Rule Set's own + defaults. + - The sidecar also changes the Core Rule Set in four ways that no setting + undoes, since in front of gitea each would otherwise refuse ordinary + requests: + - PUT, PATCH and DELETE are allowed methods besides GET, HEAD, POST and + OPTIONS; APIs, container image pushes and package uploads use them. + Other methods stay refused. + - The request headers `Content-Encoding` and `Expect` are allowed: git + compresses a fetch request over 1 KiB and says so in + `Content-Encoding`, and curl and git send `Expect` before some large + uploads. The other headers the Core Rule Set refuses, such as `Proxy`, + stay refused. + - Only a body the Core Rule Set can read is inspected (see "Data flow + for one request"). It would read any other body as form data, where + binary content such as a git push trips rules written for text. + - Responses are not inspected. A raw file from a repository, such as a + shell script, looks to the response rules like source code leaking + from the server. + - `WAF_DISABLED_RULES` (default + `920340,920420,920440,920640,930130,930140,932180`): rule ids to switch + off when an app trips a false positive. The default switches off the rules + that refuse a request for the type of its body, when it is missing or not + on the Core Rule Set's short list (920340, 920420, 920640); for its file + extension, such as `.sh` or `.sql` (920440); for a file or directory name + in its path, such as `.git/`, `.gitignore`, `Dockerfile`, `package.json` + or an editor's settings directory (930130, 930140); and for the name of an + uploaded file, such as `debug.log` or `config.yml` (932180). In front of a + code forge these refuse git over HTTP, container image and package + uploads, views of ordinary files in a repository, and logs attached to + issues. The default rule file still bans probes for `.env` and `.git` at + the site root (see "Rule files"). A list given replaces the default, so + include them in it. - `WAF_EXEMPT_PATHS`: path prefixes not inspected. - - `WAF_BODY_LIMIT` (default `128K`): bodies are inspected up to this size - and streamed beyond it without buffering, so large uploads and git pushes - are not held in memory. + - `WAF_BODY_LIMIT` (default `128K`): bodies the Core Rule Set reads are + inspected up to this size and streamed beyond it without buffering, so + large uploads are not held in memory. - `TRAP_PATHS`: paths the app never serves and only scanners ask for, for - example `/.env,/wp-login.php,/.git/config`. `TRAP_OFFENCE_WEIGHT` - (default `3`). This is the env-var short form of a `path` rule with an - `offence` action, for deployments that mount no rule files. - - `ERROR_BURST_THRESHOLD` (default off): 401, 403 and 404 responses per - client per minute that count as an offence. -- Blocks and bans (R5) - - `BAN_AFTER_OFFENCES` (default `3`), `OFFENCE_WINDOW` (default `10m`). - - `BAN_DURATIONS` (default `1h,24h,7d`). - - `PERMANENT_BAN_AFTER` (default `4`; `0` never bans permanently), - `BAN_HISTORY_WINDOW` (default `90d`). + example `/wp-login.php,/xmlrpc.php` in front of gitea. A request for one + is a clear sign of attack. This is the env-var short form of a `path` rule + with the `ban` action, for deployments that mount no rule files. + - `ERROR_BURST_THRESHOLD` (default `30`): requests per client per minute + that the sidecar refused after a rule file or Core Rule Set match; more + than this breaks a limit (see "Bans"). A client trying one attack after + another is refused many times a minute; a person rarely more than a few + times. Answers the app gives are not counted, since they do not tell a + scanner from an ordinary client: in front of gitea, container image and + package clients are answered 404 by design, for each layer a push checks + and each package a lookup asks about, often hundreds of times a minute, + and git and registry clients are answered 401 at the start of every push, + every fetch from a private repository and every image pull. +- Bans (R5), following the rules under "Bans" + - `ATTACK_BAN_DURATION` (default `7d`): the ban for a first clear sign of + attack. + - `LIMIT_BAN_DURATION` (default `1h`): the ban for a first broken limit. + - `LIMIT_BAN_REPEAT_WINDOW` (default `24h`): breaking a limit again within + this time after a ban for a broken limit ended makes the next ban three + times as long. + - `MAX_BAN_DURATION` (default `7d`): a ban that would be longer is permanent + instead. - `BAN_RESPONSE` (default `403`): `403`, `429`, or `close` to drop the - connection without an answer. + connection without an answer. `403` makes a ban easy to recognise when + debugging. Behind traefik, `close` does not leave the client unanswered: + traefik answers `502`, as it does whenever its backend drops a connection. - `BAN_SCOPE_V4_PREFIX` (default `32`): widen to for example `24` to ban the surrounding netblock. -- Reputation (R6) - - `BLOCKLIST_URLS`: text files of addresses and netblocks, for example - Spamhaus DROP, FireHOL level 1, the Tor exit list. Refreshed every - `BLOCKLIST_REFRESH` (default `6h`); the last good copy is kept on failure. - - `BLOCKLIST_ACTION` (default `limit:25`): `deny`, `limit:`, or - `log`. +- Reputation (R6). No source is on by default. A DNS blocklist would be told the + address of every visitor. AbuseIPDB and CrowdSec need an account or an engine + of the operator's own. A downloaded list reveals nothing about visitors, but + the list most fit to be a default, Spamhaus DROP, must not be downloaded + automatically more than once an hour, and Spamhaus may block an address that + downloads it too often. Each sidecar fetches its own copy, so on a host that + runs several sidecars, which share one address, downloads would often come + less than an hour apart. + - `BLOCKLIST_URLS`: text files of addresses and netblocks, one per line; + anything after a `;` or `#` on a line is ignored. For example the Spamhaus + DROP list, `https://www.spamhaus.org/drop/drop.txt`. Spamhaus's terms for + it, on its DROP page: automated downloads must be at least one hour apart, + and once a day is more than enough in most cases; a product that uses it + must credit The Spamhaus Project and keep the list's date and copyright + lines with the data. Refreshed every `BLOCKLIST_REFRESH` (default `24h`); + the last good copy is kept whole, comment lines included, on failure and + across restarts. + - `BLOCKLIST_ACTION` (default `deny`): `deny`, `limit:`, or `log`. + `deny` refuses every request from a listed address, which suits lists of + networks that send nothing legitimate, such as DROP. `limit:` + gives listed clients that percentage of every limit, so they are banned + after fewer requests; it suits a list of addresses shared with ordinary + visitors, such as Tor exits. - `DNSBL_ZONES`: for example `dnsbl.dronebl.org`. Zones meant for mail (lists of residential ranges) will block ordinary visitors and should not be used; Spamhaus zones need their keyed query service, given as a zone @@ -288,34 +600,46 @@ configuration is logged with secrets masked. so the budget is spent on suspects. - `CROWDSEC_LAPI_URL`, `CROWDSEC_LAPI_KEY`: optional. If a CrowdSec engine exists on the host, pull its decision list on a schedule and treat listed - addresses as banned. This is how a fleet-wide blocklist can arrive - without the sidecar depending on CrowdSec. + addresses as banned. This is how a fleet-wide blocklist can arrive without + the sidecar depending on CrowdSec. The list is kept like a fetched + blocklist, and a listed client's request makes a ban with the cause + `crowdsec` that lasts as long as CrowdSec's decision, so a long list does + not fill `bans.json`. - `REPUTATION_ACTION` (default `limit:25`): `deny`, `limit:`, or - `log`, for DNSBL and API hits. + `log`, for DNSBL and API hits. Such verdicts are less certain than a + blocklist, so by default a listed client gets a quarter of every limit and + is banned after a quarter of the requests. - `REPUTATION_CACHE_TTL` (default `24h`), `REPUTATION_TIMEOUT` (default `2s`). - Alerting (R3) - - `ALERT_WEBHOOK_URL`: JSON POST; schema below. - `ALERT_WEBHOOK_HEADERS`: optional `Name:value` pairs for authentication. + - `ALERT_WEBHOOK_URL`: JSON POST; schema below. `ALERT_WEBHOOK_HEADERS`: + optional `Name:value` pairs for authentication. - `ALERT_SLACK_WEBHOOK_URL`: Slack incoming webhook, formatted message. - - `ALERT_NTFY_URL` (full topic URL), `ALERT_NTFY_TOKEN`: title, priority - and tags set from the event type. + - `ALERT_NTFY_URL` (full topic URL), `ALERT_NTFY_TOKEN`: title, priority and + tags set from the event type. - `ALERT_EVENTS` (default - `ban,permanent_ban,waf_block,anomaly,reputation_hit,source_failure`): - which event types are sent. + `ban,permanent_ban,waf_block,anomaly,reputation_hit,source_failure,file_error`): + which event types are sent. `source_failure` is a reputation source or + GeoJS failing or refusing the sidecar. `file_error` is a rule file or + state file edited while running that does not parse, a replacement lookup + database that cannot be read, or a state file that cannot be written. - `ALERT_COOLDOWN` (default `15m`): the same event type for the same client or netblock is not repeated within this time; a count of suppressed repeats is included in the next one. - `ALERT_MAX_PER_HOUR` (default `60`): beyond this, alerts are rolled into one summary per hour so a wide attack cannot flood the channel. -- Anomaly thresholds, alert only, nothing is blocked (R4) +- Anomaly thresholds, alert only, nothing is blocked (R4). None is set by + default, and a threshold left unset sends no alert: these only alert, an alert + needs a destination only the operator can supply, and what counts as unusual + depends on each service's normal traffic, which the metrics show. - Per client: `ANOMALY_CLIENT_REQUESTS_PER_MINUTE`, `ANOMALY_CLIENT_REQUESTS_PER_HOUR`, `ANOMALY_CLIENT_BYTES_PER_MINUTE`, `ANOMALY_CLIENT_BYTES_PER_HOUR`. - Per surrounding netblock (`ANOMALY_NET_V4_PREFIX` default `24`, `ANOMALY_NET_V6_PREFIX` default `48`): `ANOMALY_NET_REQUESTS_PER_MINUTE`, `..._PER_HOUR`, `ANOMALY_NET_BYTES_PER_MINUTE`, `..._PER_HOUR`. - - Per AS number: `ANOMALY_ASN_REQUESTS_PER_MINUTE`, `..._PER_HOUR`, + - Per AS number, which needs a lookup source: + `ANOMALY_ASN_REQUESTS_PER_MINUTE`, `..._PER_HOUR`, `ANOMALY_ASN_BYTES_PER_MINUTE`, `..._PER_HOUR`. - Whole service: `ANOMALY_TOTAL_REQUESTS_PER_MINUTE`, `..._PER_HOUR`, `ANOMALY_TOTAL_BYTES_PER_MINUTE`, `..._PER_HOUR`. @@ -329,10 +653,11 @@ configuration is logged with secrets masked. ## Rule files -A required feature: at start the sidecar reads every `*.rules` file in -`RULES_DIR` and checks each request against them. The files are plain text meant -to be edited by hand, so that a new scanning pattern seen in the request log can -be turned into a rule in one line. +A required feature: the sidecar reads every `*.rules` file in `RULES_DIR` and +checks each request against them. The files are plain text meant to be edited by +hand, so that a new scanning pattern seen in the request log can be turned into +a rule in one line. The sidecar watches the directory: a file edited, added or +removed there takes effect while the sidecar runs. - One rule per line, four fields separated by spaces or tabs; the fourth field runs to the end of the line: @@ -343,23 +668,26 @@ be turned into a rule in one line. - Blank lines and lines starting with `#` are ignored. - `id`: a short name of letters, digits, `-` and `_`, unique across all files. - It appears in the request log, metrics, alerts and offence history. + It appears in the request log, metrics, alerts and ban notes. - `target`: what the regex is matched against. - `path`: the URL path as received, before any decoding. - `query`: the raw query string. - - `uri`: path and query together, both as received and once - percent-decoded, so an encoded probe cannot slip past. + - `uri`: path and query together, both as received and once percent-decoded, + so an encoded probe cannot slip past. - `method`, `host`, `user_agent`, `referer`. - `header:`: any one request header. - Request bodies are not available to rule files; body inspection is the Core Rule Set's job. - `action`: - `log`: note the match in the request log and do nothing else. - - `offence` or `offence:`: record one or `n` offences and forward the - request. Enough offences within `OFFENCE_WINDOW` creates a ban. - - `block` or `block:`: refuse with 403 and record one or `n` offences. - - `ban`: refuse and ban the client at once, at whatever length its ban - history calls for. + - `block`: refuse the request with 403. The client is not banned for it, but + the refusal counts toward the error burst (see "Bans"). + - `ban`: the request is a clear sign of attack. Refuse it and ban the + client's netblock for seven days (`ATTACK_BAN_DURATION`); any further + request during those days, or a later clear sign of attack, makes the ban + permanent (see "Bans"). Use it only for requests no real visitor sends, + and anchor a path at the site root (`^/`): a file of the same name deeper + in a site can be ordinary content, such as a file in a gitea repository. - `regex`: Go regular expression syntax (RE2). It has no backreferences or lookaround, and in exchange matching time is linear in the input, so no rule can be made to stall the proxy. `(?i)` at the front makes a rule @@ -367,11 +695,19 @@ be turned into a rule in one line. anchor with `^` and `$` when that is not wanted. - Files are read in name order (`00-default.rules` before `50-gitea.rules`), rules in line order. -- Rules are compiled once at start and held in memory. A line that does not - parse, a regex that does not compile, or a duplicate id stops the process - with a message naming the file and line. A missing or empty directory is not - an error: the log says that no rules were loaded. Changing rules means - restarting the container; reloading while running is given up for now. +- Rules are compiled when their file is read and held in memory. At start, a + line that does not parse, a regex that does not compile, or a duplicate id + stops the process with a message naming the file and line. While running, the + same faults leave the rules as they were, including the earlier version of + that file, and the log and one `file_error` alert name the file and line; once + the file is fixed, it is read again. A missing or empty directory is not an + error: the log says that no rules were loaded. +- The image ships a default file in `RULES_DIR`. Mounting a directory over it + replaces the defaults; mounting single files into it adds to them. Docker does + not show a single mounted file being replaced on the host, which is how many + editors save, so rules meant to be edited while the sidecar runs belong in a + mounted directory, with a copy of the default file if the defaults are to + stay. - In `MODE=observe` every action is logged as what would have happened and nothing is refused. - Clients in `ALLOW_NETS` are not checked. @@ -381,74 +717,147 @@ Example file: ``` # 00-default.rules: probes no real visitor sends -# id target action regex -env-file path offence:3 (?i)/\.env(\.[a-z]+)?$ -git-dir path offence:3 ^/\.git/(config|HEAD|index)$ -wp-probe path offence:3 (?i)^/(wp-login\.php|xmlrpc\.php|wp-admin/) -php-shell path block:3 (?i)/(shell|c99|r57|wso|alfa)\.php$ -path-traversal uri block (\.\./){2,} -scanner-agent user_agent ban (?i)\b(sqlmap|nikto|nuclei|masscan|zgrab|wpscan)\b -empty-agent user_agent log ^$ +# id target action regex +env-file path ban (?i)^/\.env(\.[a-z]+)?$ +git-dir path ban ^/\.git/(config|HEAD|index)$ +php-shell path ban (?i)^/(shell|c99|r57|wso|alfa)\.php$ +scanner-agent user_agent ban (?i)\b(sqlmap|nikto|nuclei|masscan|zgrab|wpscan)\b +path-traversal uri block (\.\./){2,} +empty-agent user_agent log ^$ ``` The image ships one default file of this kind. It is kept short and limited to -patterns that are wrong for every app; anything app-specific (for example -blocking `/wp-login.php` is harmless in front of gitea and fatal in front of -WordPress) belongs in a file the deployer mounts. +patterns that are wrong for every app, and its path rules are anchored at the +site root: in front of gitea, `/.env` is a probe, while +`///src/branch/main/.env.example` is a file in a repository that +any visitor or search crawler may open. Anything app-specific belongs in a file +the deployer mounts: a request for `/wp-login.php`, for example, is a clear sign +of attack in front of gitea and an ordinary login in front of WordPress. + +``` +# 50-gitea.rules: WordPress probes, which gitea never serves +wp-probe path ban (?i)^/(wp-login\.php|xmlrpc\.php|wp-admin/) +``` ## Persistent state -All state lives in memory. No database is used. What must survive a restart is -written to `STATE_DIR` as JSON files; the files are read once at start and never -during a request. The whole state is expected to stay under 10 MiB, and reading -or writing even ten times that as JSON takes well under a second, so whole-file -rewrites are cheap enough to need nothing cleverer. +All state lives in memory, and the files in `STATE_DIR` hold a copy of all of +it, so an orderly stop loses nothing. The one exception is log lines still +waiting to be sent to `LOG_REMOTE_URL`, which stdout has already carried. No +database is used, and nothing is read from disk while serving a request. The +files are meant for people as well: an admin can read or edit them at any time, +and the running sidecar takes the edit in. - Files, each holding one kind of state: - - `bans.json`: active and past bans. Per entry: client or netblock, start, - expiry (`null` for permanent), reason, which ban this is for that client - (first, second, third), source (`local`, `admin`, `crowdsec`), and when - it was lifted, if it was. - - `offences.json`: offender history. Per entry: client, time, kind (`rate`, - `bytes`, `waf`, `trap`, `error_burst`), short detail. Entries older than - `BAN_HISTORY_WINDOW` are dropped at each write. - - `counters.json`: the hour and day request and byte counters per client. - - `reputation.json`: cached DNSBL and reputation API verdicts with the time - each was fetched. + - `bans.json`: active and past bans, up to `MAX_BANS` that the sidecar made + and every ban an admin made. Per entry: the netblock, start, expiry + (`null` for permanent), what caused it (`attack`, `limit`, `admin` or + `crowdsec`), a short reason, when it was lifted if an admin lifted it, and + a `notes` field (below). + - `clients.json`: per client, its minute, hour and day counters and its + history since it was first seen: first and last time seen, the AS number, + AS name and country last looked up and when, total requests and bytes in + each direction, how many requests were forwarded and how many refused, + responses by status class, and offences by kind. Clients that were never + banned are kept too, so every client's history survives a restart; + `GET /clients/` shows it. + - `lookups.json`: GeoJS answers, one per client: the AS number, AS name and + country, when GeoJS was asked and when the answer was last used; up to + 100,000, each for 7 days. + - `reputation.json`: the last good copy of each list fetched from a URL, + cached DNSBL and reputation API verdicts, each with the time it was + fetched, and the AbuseIPDB checks spent today, so a restart does not reset + the daily budget. + - `alerts.json`: the anomaly counters for surrounding netblocks, AS numbers, + named netblocks and the whole service; for each event type and client or + netblock, when it was last sent and how many repeats the cooldown has held + back since; the alerts sent this hour; and alerts still waiting to be + sent. +- Ban notes. The notes on a ban hold what an admin needs to decide whether to + lift it, drawn from what the sidecar already knows; nothing is looked up to + fill them: + - the AS number, AS name and country (empty when lookups are off); + - what was broken: the rule ids and target that matched, or the limit, its + window, the count reached and the client's limit percentage with what set + it; and any reputation sources that listed the client; + - the requests that caused the ban, up to the last ten: time, method, host, + path with its query string, status and user agent, each text cut to 256 + bytes; + - how many requests counted toward the ban, and the time span over which + they came; + - the netblock's total requests since it was first seen, and the requests + refused under this ban so far, kept up to date while the ban lasts; + - how many earlier bans of each kind the netblock has had. +- Size and disk writes. Each file is rewritten whole, so each has a bound: + - `clients.json` takes about 1 KiB per client. Clients are dropped only when + the table is full, so on a public service the file grows to the default + `MAX_TRACKED_CLIENTS` of 20,000, about 20 MiB. Written every 15 minutes, + that is under 2 GiB of disk writes a day. + - `bans.json` takes about 2 KiB per ban and at most about 8 KiB, since the + texts in the notes are cut short. At the default `MAX_BANS` of 5,000 it is + about 10 MiB, and never more than about 40 MiB, plus whatever bans an + admin made. It is written when a ban is made, lifted or made permanent, at + most once every 10 seconds, and otherwise with the 15-minute write, so its + writes follow the bans made: with a full file, a hundred new bans a day + come to about 1 GiB of disk writes. + - `lookups.json` takes about 150 bytes per answer, about 15 MiB when full. + Written every 15 minutes, that is under 1.5 GiB of disk writes a day. + - `reputation.json` and `alerts.json` are usually a few MiB or less. + - Writing a file of these sizes takes well under a second on an ordinary + disk, in the background, so rewriting whole files needs nothing cleverer. - Format: indented JSON with a top-level `version` number, entries sorted by client address, times in RFC 3339 UTC, durations and sizes as plain numbers - with the unit in the field name. The aim is that a person can open - `bans.json` in an editor, find an address, and remove or add an entry. + with the unit in the field name. The aim is that a person can open `bans.json` + in an editor, find an address, and remove or add an entry. `clients.json` and + `lookups.json` put each entry on one line instead, which halves their size and + lets `grep` show everything about one client. - Writing: - serialise from a snapshot taken under the lock, so requests are not held up while the file is written; - write to a temporary file in the same directory, sync it, rename it over the real name, sync the directory. A crash at any point leaves either the old complete file or the new complete file, never a partial one; - - `bans.json` and `offences.json` are written `STATE_WRITE_DELAY` after a - change; `counters.json` and `reputation.json` every - `STATE_COUNTER_INTERVAL`; all four on orderly shutdown (SIGTERM). -- Reading: - - at start only. Active bans go into an in-memory prefix lookup; everything - else into maps. Expired counters and cache entries are discarded. - - a missing file means empty state and is normal on first run. - - a file that does not parse, or has an unknown `version`, stops the - process with a message naming the file and position. Starting with empty - bans would silently forgive every repeat offender, and since writes are - atomic a broken file can only come from a hand edit, which the editor - should hear about. `counters.json` and `reputation.json` are the - exception: if unreadable they are logged, set aside with a `.bad` suffix, - and the process starts with them empty. -- Hand edits are made with the container stopped; a running process will - overwrite the file at its next write. Changes to a running process go through - the admin endpoints. Reloading files into a running process is given up for - now. -- `STATE_DIR` not writable at start: the process exits. A write that fails - while running: state stays correct in memory, the failure is logged, counted - in metrics, alerted once per cooldown, and retried at the next write. -- What a hard kill can lose: bans and offences from the last - `STATE_WRITE_DELAY`, counters from the last `STATE_COUNTER_INTERVAL`. Losing - the volume loses ban history, not service. + - `bans.json` is written `STATE_WRITE_DELAY` after a ban is made, lifted or + made permanent. Changes that only update the counts in its notes wait for + the write every `STATE_COUNTER_INTERVAL`, as all of `clients.json`, + `lookups.json`, `reputation.json` and `alerts.json` do. All five are + written on orderly shutdown (SIGTERM); + - before writing a file, the sidecar checks whether it changed on disk since + the sidecar last read or wrote it; if it did, the sidecar takes that edit + in first (below), so an admin's edit is never overwritten. +- Reading at start: + - Active bans go into an in-memory prefix lookup; everything else into maps. + Counter buckets and cache entries whose time has passed are discarded. + - A missing file means empty state and is normal on first run. + - A file that does not parse, or has an unknown `version`, stops the process + with a message naming the file and position. Starting with empty state + would silently forgive every repeat offender, and since writes are atomic + a broken file can only come from a hand edit, which the editor should hear + about. +- Edits while running: + - The sidecar watches `STATE_DIR` and notices a file that is edited, + replaced or added. It tells its own writes from an admin's by comparing + the file with what it last wrote. + - An edit that parses is taken in at once: what the file says replaces what + the sidecar held for that file. Removing a ban's entry from `bans.json` + lifts the ban; adding an entry bans. Changes the sidecar made after the + admin opened the file, such as a new ban, are lost when the admin saves + over them; most editors warn when a file changed on disk while it was + open. + - An edit that does not parse does not stop the running sidecar. It keeps + the state it has, renames the edited file to `.bad` (for example + `bans.json.bad`) so the edit is kept, writes the file again from memory, + logs the file and the position of the error, and sends them in one + `file_error` alert. The admin fixes the `.bad` file and moves it back. +- `STATE_DIR` not writable at start: the process exits. A write that fails while + running: state stays correct in memory, the failure is logged, counted in + metrics, sent as a `file_error` alert once per cooldown, and retried at the + next write. +- What a hard kill can lose: ban changes from the last `STATE_WRITE_DELAY` (10 + seconds), and everything else from the last `STATE_COUNTER_INTERVAL` (15 + minutes): counts and history, the counts in ban notes, GeoJS answers, + reputation verdicts and the AbuseIPDB count, anomaly counters and alert + cooldowns. Losing the volume loses ban history, not service. ## Request log @@ -464,19 +873,19 @@ handling while `docker logs` keeps working. passed to the app), `peer_ip` (the TCP peer, normally traefik), `forwarded_for` (the header as received), `client_group` (the /64 or configured prefix used for counting), `asn`, `as_name`, `country`, - `content_type`, `content_length`, the headers named in - `LOG_REQUEST_HEADERS`, `has_authorization` and `has_cookie` as booleans, - `websocket` when the connection was upgraded. + `content_type`, `content_length`, the headers named in `LOG_REQUEST_HEADERS`, + `has_authorization` and `has_cookie` as booleans, `websocket` when the + connection was upgraded. - Response detail: `response_content_type`, `upstream_status` (differs from `status` when the sidecar answered itself), `cache_control`, `location` on redirects, `aborted` when the client went away early. -- Decision: `action` (`forward`, `rate_limited`, `bytes_limited`, `banned`, - `denied`, `waf_blocked`, `upstream_error`), `would_action` in `observe` mode, - `limit_percent` and which rule set it, `counts` (the client's minute, hour - and day request and byte totals after this request), `limit_hit` (which - window), `rule_ids` (rule file rules that matched), `waf_rule_ids`, - `waf_score`, `reputation` (sources that listed the - client), `offence` when one was recorded, `ban_expires`. +- Decision: `action` (`forward`, `banned`, `denied`, `country_denied`, + `rate_limited`, `rule_blocked`, `waf_blocked`, `too_large`, `timed_out`, + `upstream_error`), `would_action` in `observe` mode, `limit_percent` and which + rule set it, `counts` (the client's minute, hour and day request and byte + totals after this request), `limit_hit` (which window), `rule_ids` (rule file + rules that matched), `waf_rule_ids`, `waf_score`, `reputation` (sources that + listed the client), `offence` when one was recorded, `ban_expires`. - Timings in milliseconds: `duration_total`, `duration_checks` (everything the sidecar did before forwarding), `duration_waf`, `duration_upstream_connect`, `duration_upstream_first_byte`, `duration_upstream_total`. @@ -487,38 +896,37 @@ handling while `docker logs` keeps working. - Remote sending: - syslog forms send each line as the message of an RFC 5424 record, with octet-counted framing on TCP and TLS; - - RELP forms send the same record and wait for the receiver's - acknowledgement, so lines are not lost across a receiver restart. This is - the form to use with rsyslog; - sending happens on its own goroutine from a bounded buffer (`LOG_REMOTE_BUFFER`). An unreachable or slow endpoint never delays a request and never stops stdout; it reconnects with backoff, drops the - oldest lines when the buffer is full, and counts the drops in metrics. - UDP gives no delivery signal at all and is offered only for - compatibility. + oldest lines when the buffer is full, and counts the drops in metrics. UDP + gives no delivery signal at all and is offered only for compatibility. ## Metrics endpoint -Prometheus text format on `METRICS_LISTEN_ADDR` at `METRICS_PATH`, on by default, -switched off with `METRICS_ENABLED=false`. It has its own listener so it can be -reached by a scraper without exposing ban management. +Prometheus text format on `METRICS_LISTEN_ADDR` at `METRICS_PATH`, on by +default, switched off with `METRICS_ENABLED=false`. It has its own listener so +it can be reached by a scraper without exposing ban management. -- Traffic: requests and bytes in and out, by status class and `action`; - request duration and upstream duration histograms; requests in flight. -- Limits and bans: limit hits by window and kind (requests or bytes), offences - by kind, bans created by ordinal, permanent bans, active bans (gauge). +- Traffic: requests and bytes in and out, by status class and `action`; request + duration and upstream duration histograms; requests in flight. +- Limits and bans: limit hits by window and kind (requests or bytes), size and + time limit hits by limit, offences by kind, bans created by cause, permanent + bans, active bans (gauge); requests refused by the country lists, by country. - Attack detection: rule file matches by rule id and action, and the number of rules loaded; Core Rule Set matches by mode and rule id (label limited to the rules that actually fired). - Lookup and reputation: requests and bytes by AS number and by country, limited to the `METRICS_TOP_N` (default `50`) busiest of each with the rest summed as `other`, so the label set stays bounded; reputation queries, hits, failures - and remaining daily budget by source; age of each blocklist and lookup - database. -- Housekeeping: tracked clients (gauge), state file writes, write failures, - last successful write time and size per file; alerts sent, failed and - suppressed by destination; remote log lines sent, dropped and buffer depth; - the standard Go runtime and process metrics. + and remaining daily budget by source; GeoJS requests and failures, and clients + that counted as unknown because it did not answer in time; age of each + blocklist and lookup database. +- Housekeeping: tracked clients (gauge), state file writes, write failures, last + successful write time and size per file; files read again after an edit, and + edits set aside because they did not parse; alerts sent, failed and suppressed + by destination; remote log lines sent, dropped and buffer depth; the standard + Go runtime and process metrics. - No metric carries a client IP address as a label; per-address questions are answered by the request log and `GET /clients/`. @@ -526,9 +934,11 @@ reached by a scraper without exposing ban management. - `GET /healthz`: for the container health check. - `GET /bans`, `POST /bans` (client or netblock, duration, reason), - `DELETE /bans/`: need `ADMIN_TOKEN`. -- `GET /clients/`: current counters, lookup result, reputation, offences; - for answering "why was this address refused". + `DELETE /bans/`: need `ADMIN_TOKEN`, and are switched off while it is + unset. Editing `bans.json` does the same without a token. +- `GET /clients/`: current counters, history, lookup result, reputation, + offences, and bans with their notes; for answering "why was this address + refused" and "what has this netblock been doing". ## Alert webhook schema @@ -538,13 +948,15 @@ One JSON object per alert: - `client`, `netblock`, `asn`, `as_name`, `country` - `reason`: short human-readable sentence - `detail`: event-specific fields, for example `window`, `count`, `limit`, - `limit_percent`, `rule_ids`, `path`, `ban_ordinal`, `ban_expires` + `limit_percent`, `rule_ids`, `path`, `ban_expires`, for a ban its notes, and + for `file_error` the file and the position of the error - `suppressed_repeats`: number of identical alerts held back by the cooldown ## Deployment as a sidecar -docker-compose shape, using gitea as the example; upaas deployments follow the -same shape if upaas can run a second container for a service (open): +docker-compose shape, using gitea as the example. The app carries no traefik +labels and publishes no ports; the sidecar carries the labels and points at the +app's hostname. `UPSTREAM_URL` is the only setting it needs: ```yaml services: @@ -559,137 +971,151 @@ services: user: "65532:65532" environment: UPSTREAM_URL: http://gitea:3000 - TRUSTED_PROXIES: 172.18.0.0/16 - INSTANCE_NAME: fsn1app1/gitea - MODE: observe - RATE_LIMIT_PER_MINUTE: "120" - RATE_LIMIT_PER_HOUR: "2000" - RATE_LIMIT_PER_DAY: "10000" - BYTES_LIMIT_PER_HOUR: 2G - ASN_DB_URL: https://example.invalid/asn-country.mmdb - ASN_LIMIT_PERCENT: AS14061:50,AS16276:50 - COUNTRY_LIMIT_PERCENT: CN:25 - WAF_MODE: detect - TRAP_PATHS: /.env,/wp-login.php,/.git/config - BLOCKLIST_URLS: https://www.spamhaus.org/drop/drop.txt - ALERT_NTFY_URL: https://ntfy.example.invalid/fleet-alerts - LOG_REMOTE_URL: relp://logs.example.invalid:2514 - METRICS_LISTEN_ADDR: :9100 volumes: - gitea-guard-state:/data - - ./50-gitea.rules:/etc/smallwebwaf/rules.d/50-gitea.rules:ro networks: [internal, traefik] labels: traefik.enable: "true" traefik.http.routers.gitea.rule: Host(`git.example.invalid`) traefik.http.services.gitea.loadbalancer.server.port: "8080" + +volumes: + gitea-guard-state: + +networks: + internal: + traefik: + external: true ``` -- The application container leaves the traefik network, so the sidecar cannot - be bypassed. -- Traefik routes only port 8080. The metrics port is reached by the scraper - over a docker network it shares with the sidecar; the admin port stays on - loopback inside the container and is used through `docker exec`. -- The state volume holds a few small JSON files; it needs no backup beyond - whatever the host already does. -- SSH access to gitea does not pass through the sidecar and is not protected - by it. -- Rollout per service: start in `MODE=observe` with `WAF_MODE=detect`, read a - few days of logs and alerts, set limits and rule exclusions from what real - traffic looks like, then switch to `enforce` and `block`. -- Notes specific to gitea: clones and pushes over HTTP are large and long, so - byte limits must be sized for a legitimate clone, the proxy sets no overall - response timeout, and `WAF_BODY_LIMIT` keeps pack uploads out of memory. - Archive download and blame or history pages are what scrapers hammer; request - limits do most of the work there. +- An app deployed by upaas takes the same shape, without the app's part of this + file. upaas deploys the app with no traefik labels; the sidecar runs beside it + from its own docker-compose file, carries the traefik labels, and points + `UPSTREAM_URL` at the app's hostname. The sidecar must be able to reach that + hostname, for example over a docker network the two share. +- The application container leaves the traefik network, so the sidecar cannot be + bypassed. +- Traefik routes only port 8080. The metrics port is reached by the scraper over + a docker network it shares with the sidecar; the admin port stays on loopback + inside the container and is used through `docker exec`. +- The state volume holds the JSON state files, a few tens of MiB at most with + the defaults (see "Persistent state"); it needs no backup beyond whatever the + host already does. +- SSH access to gitea does not pass through the sidecar and is not protected by + it. +- Rollout per service: point traefik at the sidecar with only `UPSTREAM_URL` + set. It protects the app from the first request, and nothing needs tuning + first. Afterwards the request log and the ban notes say why each client was + refused. A real visitor refused by mistake is let through with an exclusion + (`WAF_DISABLED_RULES`, `WAF_EXEMPT_PATHS`, `RATE_LIMIT_EXEMPT_PATHS`) or an + exemption (`RATE_LIMIT_EXEMPT_NETS`, `ALLOW_NETS`), and its ban is lifted in + `bans.json`. Additions to consider at any time: an alert destination, a lookup + source with country lists or biased thresholds, reputation sources, a remote + log endpoint, and an app-specific rule file. +- Notes specific to gitea: + - A clone is a response, and fits the defaults of 30 minutes and 5 GB for + all but the largest repositories and slowest links. A push is a request: + at the defaults, one larger than 100 MB or taking more than 60 seconds to + upload is cut off, so a gitea that takes large pushes needs + `CLIENT_REQUEST_MAX_BYTES`, `UPSTREAM_REQUEST_MAX_BYTES`, + `CLIENT_REQUEST_TIMEOUT` and `UPSTREAM_REQUEST_TIMEOUT` raised to fit. The + Core Rule Set does not read a pack upload, which streams through without + being held in memory. + - With the sidecar's changes to the Core Rule Set and the default + `WAF_DISABLED_RULES` (see "Configuration surface", attack detection), + git's clone, fetch and push over HTTP, the API, container images, + packages, and views of ordinary files pass the Core Rule Set. Requests + that carry text people write, such as issues, pull requests, searches, the + web editor and uploads, can still trip other rules, and so can a file + whose name the Core Rule Set takes for an attack, such as an `.xhtml` + page. Such a match refuses only that request, and the request log names + the rule in `waf_rule_ids` for `WAF_DISABLED_RULES`. + - Archive download and blame or history pages are what scrapers hammer; + request limits do most of the work there. ## Failure behaviour -- Lookup database missing: clients are unknown, service continues, one alert. +- Lookup database: a configured file that is missing or unreadable at start + stops the start. A replacement that cannot be read while running is ignored: + the file already loaded stays in use, the problem is logged, one `file_error` + alert. +- GeoJS slow, down or refusing the sidecar: clients with a kept answer are + unaffected, new clients count as unknown, the sidecar keeps asking with + backoff, one `source_failure` alert per cooldown. - Reputation source down or over quota: no verdict, service continues, one `source_failure` alert per cooldown. - Alert destination down: retried with backoff from a bounded queue, oldest dropped first, drops counted in metrics. - Upstream down: 502 from the sidecar, not counted as client offences. -- Attack detection engine error on a request: request is forwarded, error - logged and counted. +- Attack detection engine error on a request: request is forwarded, error logged + and counted. - Remote log endpoint down: stdout continues, lines are buffered then dropped oldest first, drops counted in metrics. - State file write fails while running: memory stays authoritative, logged, - counted, alerted, retried. -- In short: the only things that stop the process are invalid configuration, - a rule file that does not parse, an unwritable `STATE_DIR`, and an unparseable `bans.json` or `offences.json` - at start. A broken helper never takes the protected service down. + counted, one `file_error` alert per cooldown, retried. +- A state file or rule file edited while running that does not parse: the + sidecar keeps running on what it has. The state file is set aside as + `.bad` and written again from memory; the rule file is left as it is. + Logged, one `file_error` alert. +- In short: the only things that stop the process happen at start: invalid + configuration, a configured lookup database that is missing or unreadable, a + rule file or state file that does not parse, and an unwritable `STATE_DIR`. + Once running, a broken helper or a broken edit never takes the protected + service down. The nearest it comes is GeoJS failing while + `EXCLUSIVELY_ALLOWED_COUNTRIES` is set: new clients then cannot be placed, and + that list refuses them, as it refuses every client it cannot place. ## Risks the design has to handle -- Forged `X-Forwarded-For`: handled by `TRUSTED_PROXIES` being required and by - walking the header from the right. -- Many clients behind one address (mobile carriers, offices, Tor): rate limits - punish them collectively; `RATE_LIMIT_EXEMPT_NETS` and sensible day limits - are the remedy, and bans on such addresses should stay short. +- Forged `X-Forwarded-For`: handled by believing the header only from a peer + inside `TRUSTED_PROXIES` and by walking it from the right. The default trusts + every private address, which in the intended deployment means traefik; where + other containers can reach the sidecar directly, or visitors on a private + network reach traefik, they can name another address in the header, and + setting `TRUSTED_PROXIES` to traefik's own network closes that gap. +- Many clients behind one address (mobile carriers, offices, Tor): they share + its limits and its bans. The default limits sit several times above what one + busy person produces, `RATE_LIMIT_EXEMPT_NETS` and `ALLOW_NETS` take known + shared addresses out, and the ban notes show an admin what happened before a + ban is lifted. +- A `ban` rule that matches a real visitor bans it for seven days, and the + visitor's next request during those days makes the ban permanent. So the + default rule file keeps to requests no real visitor sends, with its paths + anchored at the site root, where no app serves `.env` files or web shells. A + visitor banned by mistake is let back in by lifting the ban in `bans.json`. - IPv6 address rotation inside a /64: handled by grouping. - Widely distributed scrapers using thousands of addresses at low rates each: - per-client limits do not see them. The AS number and netblock anomaly - alerts reveal them, and `ASN_LIMIT_PERCENT` at a low value or `0` is the - response. A per-AS-number or per-country total limit is open. + per-client limits do not see them. The AS number and netblock anomaly alerts + reveal them once their thresholds are set, and a low `ASN_LIMIT_PERCENT` for + that AS number is the response: it lowers the limits of each of its clients. + There is no shared budget for a whole AS number or country, which one abuser + could use up and so lock out everyone else there. - Core Rule Set false positives against real apps (gitea's editor, API - payloads): `detect` by default, exclusions by rule id and path. -- Slow-request attacks: header read timeout and idle timeout on the listener. + payloads): a match refuses only that request and bans no one by itself; the + request log names the rule, and exclusions by rule id and path fix it. +- GeoJS as the lookup source: every new visitor's address goes to a third party, + and a swarm of fresh addresses, when lookups peak, is when GeoJS may slow down + or block the sidecar. Keeping answers for 7 days and asking about many + addresses in one request keep the number of requests low; the file source has + neither risk. +- Slow-request attacks: `CLIENT_REQUEST_TIMEOUT` bounds how long a request may + take to arrive, `CLIENT_REQUEST_HEADER_MAX_BYTES` how large its headers may + be, and `CLIENT_IDLE_TIMEOUT` closes a kept-open connection that sends nothing + more. - Alert floods: cooldown and hourly cap. ## Build order -- First: proxy, client identification, static lists, three-window request - limits, exemptions, `observe` mode, the full request log on stdout, the - metrics endpoint, health. -- Second: rule files, ban ledger with escalation and permanent bans, the JSON - state files, - admin endpoints, alerting to all three destinations, remote log sending. -- Third: AS number and country lookup, biased thresholds, byte limits, anomaly - thresholds. +- First: proxy with its size and time limits, client identification, static + lists, three-window request limits and the bans they lead to, the ban ledger + and the JSON state files with edits taken in while running, exemptions, + `observe` mode, the full request log on stdout, the metrics endpoint, health. +- Second: rule files, admin endpoints, alerting to all three destinations, + remote log sending. +- Third: AS number and country lookup from the file or GeoJS, the country lists, + biased thresholds, byte limits, anomaly thresholds. - Fourth: blocklists, DNSBL, AbuseIPDB, optional CrowdSec decision feed. - Fifth: attack detection with Coraza and the Core Rule Set, trap paths, error bursts. - Each stage is usable on its own; the first three already cover the traffic problem the fleet has today. - -## Questions for the owner - -- Combining percentages. A client in a listed AS number (50 percent) and a - listed country (25 percent): should it get the lowest (25 percent of the - normal limit) or the product (12.5 percent)? Lowest is easier to predict when - reading a config; product punishes overlap harder. Recommendation: lowest. -- Meaning of "certain countries getting a percentage rate or byte limit". Read - here as: each client from that country gets that percentage of the normal - per-client limits. The other reading is a cap on the country as a whole, for - example all of one country together may use at most 20 percent of the - service's hourly byte budget. The second needs a configured total budget and - means one heavy client can lock out its compatriots. Recommendation: build - the per-client reading first; add whole-country and whole-AS-number caps only - if distributed scrapers make it necessary. -- Sharing bans across the fleet. As drafted, each sidecar remembers only its - own bans, so an abuser banned at gitea starts fresh at the next service. - Options: keep it per sidecar; or run one CrowdSec engine per host and have - every sidecar read its decision list (already supported above) and also - report its own bans to it. Recommendation: per sidecar for the first version, - decide on CrowdSec after seeing real ban volumes. -- Lookup database source. IPinfo Lite (one file with country and AS number, - free account, attribution required), DB-IP Lite (no account, attribution - required), or MaxMind GeoLite2 (free account, licence terms on - redistribution). Recommendation: IPinfo Lite, fetched once centrally and - served to the fleet from an internal URL so sidecars hold no account token. -- upaas. Whether upaas can deploy a second container beside an app and move - the traefik labels onto it was not examined. If it cannot, that is a feature - request for upaas, not something this sidecar can solve. -- Remote log library. Sending syslog in the current format (RFC 5424) is not - covered by Go's standard library, whose `log/syslog` package is frozen and - writes the older format, and no widely used Go RELP client was found. The - choices are a small third-party package for each, or writing both inside this - project; the syslog record is a few dozen lines, a RELP client with - acknowledgement handling is a few hundred. Recommendation: write the syslog - sender in-project, ship it first, and decide on RELP after looking at the - available packages; say if RELP must be in the first release. -- Ban response. `403` tells the abuser they were noticed; `close` wastes less - and tells them nothing. Recommendation: `403` by default for debuggability, - `close` available.