From 43c63faa04858528fac7da2672ac36b23c27e791 Mon Sep 17 00:00:00 2001 From: sneak Date: Wed, 23 Sep 2026 12:46:05 +0000 Subject: [PATCH 1/3] SPEC and README: owner rulings, issues 2 to 5, internet-ready defaults (closes #6) Rewrite `SPEC.md` and `README.md` to the owner's rulings. The seven answers to the spec's questions now stand where their topics live, and the questions section is gone. Bans follow the owner's model: seven days for a clear sign of attack and permanent on any further request, an hour for a broken limit, tripled on a repeat within a day, permanent past seven days. The state files hold all state, are watched, and take in an admin's edits while running. Every ban carries notes, and each client's history survives a restart. Size and time limits apply in both directions. Only `UPSTREAM_URL` is required: the Core Rule Set refuses what it flags and every limit has a default. Model: opus-5-5 --- README.md | 167 ++++++---- SPEC.md | 904 +++++++++++++++++++++++++++++++++--------------------- 2 files changed, 662 insertions(+), 409 deletions(-) diff --git a/README.md b/README.md index 8071582..38bd9b9 100644 --- a/README.md +++ b/README.md @@ -3,9 +3,10 @@ `smallwebwaf` is a simple, fast, logging web application firewall for people who host their own services. It is one small container that sits between your reverse proxy (traefik) and one application: traefik points at `smallwebwaf`, -and `smallwebwaf` points at the app. It is configured with environment -variables, keeps its state in memory, and writes a detailed JSON log line for -every request. +and `smallwebwaf` points at the app. It needs one setting, the address of the +app, and protects the app from the first request with defaults chosen for a +service on the open internet. It keeps its state in memory and in JSON files you +can read and edit, and writes a detailed JSON log line for every request. Status: design stage. This repository currently holds the documents only; no code has been written. The design is in [`SPEC.md`](SPEC.md), and the survey of @@ -17,71 +18,89 @@ Small self-hosted sites now receive a great deal of traffic nobody asked for: scrapers that ignore `robots.txt` and crawl every commit of every repository on a public git server, vulnerability scanners walking through lists of WordPress and `.env` paths, and credential-guessing bots. Most of it comes from a small -number of hosting networks and countries. A single-person operation has no -abuse desk and no CDN contract; it needs something small that can be put in -front of one service and left alone. +number of hosting networks and countries. A single-person operation has no abuse +desk and no CDN contract; it needs something small that can be put in front of +one service and left alone. The existing tools each solve part of this. Rule-based firewalls catch attack -payloads but do not limit request rates. Rate limiters count requests but -cannot tell a residential visitor from a rented server farm. The products that -do most of it want several containers, a database and a web console. None of -them can say "clients from these networks get half the normal allowance", which -is the most useful thing to be able to say when nearly all abuse comes from a -known list of AS numbers. [`EVALUATION.md`](EVALUATION.md) goes through the -candidates one by one. +payloads but do not limit request rates. Rate limiters count requests but cannot +tell a residential visitor from a rented server farm. The products that do most +of it want several containers, a database and a web console. None of them can +say "clients from these networks are banned after half as many requests as +anyone else", which is the most useful thing to be able to say when nearly all +abuse comes from a known list of AS numbers. [`EVALUATION.md`](EVALUATION.md) +goes through the candidates one by one. `smallwebwaf` is meant to fill that gap: - protect a service from misbehaving scrapers and scanners with per-client request and byte limits over a minute, an hour and a day; -- bias those limits against the countries and AS numbers that abuse commonly - comes from, so their clients get a configured percentage of the normal - allowance rather than an outright block; -- remember abusers, block them for a while, and ban the ones who keep coming - back; +- lower those limits for the countries and AS numbers that abuse commonly comes + from, so their clients are banned after fewer requests than others; +- ban abusers: briefly at first, longer each time they come back, and + permanently when they keep at it; a scanner's first probe bans it for seven + days; - log everything in a form that is easy to search and ship elsewhere; - stay small enough to understand: one binary, one container, environment - variables, no database. + variables, no database, one required setting. ## Proposed features -- Reverse proxy for one upstream application, streaming in both directions, - with WebSocket support. One `smallwebwaf` per app. -- Real client address worked out from `X-Forwarded-For`, trusting only the - proxy networks you list. IPv6 clients are counted by /64. +- Reverse proxy for one upstream application, streaming in both directions, with + WebSocket support. One `smallwebwaf` per app. +- Internet-ready out of the box: only `UPSTREAM_URL` must be set, and every + other setting has a default chosen for a service facing the internet in 2026. +- Real client address worked out from `X-Forwarded-For`, trusting only the proxy + networks you list, by default the private address ranges. IPv6 clients are + counted by /64. +- Size and time limits on requests and responses, both between the client and + `smallwebwaf` and between `smallwebwaf` and the app: by default a request may + take 60 seconds and 100 MB, a response 30 minutes and 5 GB. - Rate limits per client on requests per minute, per hour and per day, and on - bytes per minute, per hour and per day. + bytes per minute, per hour and per day, on by default and set well above what + real visitors need. - Netblocks that bypass rate limiting, netblocks that bypass everything, and netblocks that are always refused. -- AS number and country lookup for every client from a local database file. +- AS number and country lookup for every client from the IPinfo Lite database + file, which you download and mount (see "Lookup database" below). - Biased limits: listed AS numbers and countries get a percentage of every - limit, for example 50 percent for common abuse-source networks. Zero percent - refuses outright. + limit, for example 50 percent for common abuse-source networks, so their + clients are banned after fewer requests. Zero percent is a zero allowance: the + first request breaks the limit and bans the client. - Attack detection: - a directory of plain text rule files, one regex per line, for catching - scanning and penetration probes; easy to edit by hand; - - the OWASP Core Rule Set, run by the Coraza engine, in detect-only or - blocking mode; + scanning and penetration probes; easy to edit by hand, and picked up while + running; + - the OWASP Core Rule Set, run by the Coraza engine, refusing the requests + it flags; - trap paths and bursts of error responses. -- Offences add up to a temporary block. Block lengths grow for repeat offenders - (for example one hour, then a day, then a week) and end in a permanent ban. +- Bans: + - a clear sign of attack, such as a probe for a `.env` file or a scanner's + user agent, bans for seven days on the first request, and any further + request during those days makes the ban permanent; + - breaking a limit bans for an hour; breaking one again within a day of a + ban ending triples the length, and a ban that would last longer than seven + days is permanent instead; + - every ban carries notes on why it was made, to help decide whether to lift + it. - IP reputation: downloadable blocklists, DNS blocklists, AbuseIPDB, and an optional feed of decisions from a CrowdSec engine. Lookups happen in the - background and never delay a request. + background and never delay a request. None is on until you add it. - Alerts on attacks and bans to a generic webhook, Slack or ntfy, with a cooldown and an hourly cap so a wide attack cannot flood the channel. -- Anomaly alerts when requests or bytes per minute or hour cross a threshold, - for a single client, its surrounding netblock, an AS number, a named netblock - or the whole service. -- Observe mode: log and alert on every decision while refusing nothing, for the - first days in front of a new service. +- Anomaly alerts when requests or bytes per minute or hour cross a threshold you + set, for a single client, its surrounding netblock, an AS number, a named + netblock or the whole service. +- Observe mode: log and alert on every decision while refusing nothing. - Request log: one JSON object per request on stdout with the usual web log fields, the decision taken and why, AS number and country, and timings. - Optionally also sent to a remote syslog or RELP endpoint. + Optionally also sent to a remote syslog server. - Prometheus metrics on their own port. -- State (bans, offender history, hour and day counters, reputation cache) held - in memory and saved as readable, hand-editable JSON files, written atomically. - Nothing is read from disk while serving a request. +- State (bans with their notes, each client's counters and history, the + reputation cache) held in memory and kept in readable JSON files that always + hold a copy of it, so a restart loses nothing. Edit a file, or add a rule + file, and the running `smallwebwaf` picks up the change. Nothing is read from + disk while serving a request. - A small admin endpoint for health checks, listing, adding and lifting bans, and asking why a given address was refused. @@ -98,14 +117,17 @@ For each request `smallwebwaf`: the deny list or currently banned; - looks up its AS number and country, and any cached reputation verdict; - picks the client's limit percentage from those; -- checks the minute, hour and day request counters against the limits, and - answers 429 if one is exceeded; -- checks the request against the rule files and the Core Rule Set; -- forwards it to the app and streams the response back; -- counts the bytes, records any offence, bans the client if it has collected - enough of them, sends any alerts that are due, and writes the log line. +- checks the minute, hour and day request counters against the limits, and bans + the client if it breaks one; +- checks the request against the rule files and the Core Rule Set, and bans the + client at once for a clear sign of attack; +- forwards it to the app and streams the response back, within the size and time + limits; +- counts the bytes and any error response, bans the client if it broke a limit, + updates its history, sends any alerts that are due, and writes the log line. -A minimal deployment beside an app in docker-compose: +A minimal deployment beside an app in docker-compose. `UPSTREAM_URL` is the only +setting: ```yaml services: @@ -117,33 +139,50 @@ services: image: /smallwebwaf: environment: UPSTREAM_URL: http://app:3000 - TRUSTED_PROXIES: 172.18.0.0/16 - MODE: observe - RATE_LIMIT_PER_MINUTE: "120" - RATE_LIMIT_PER_HOUR: "2000" - RATE_LIMIT_PER_DAY: "10000" - ASN_LIMIT_PERCENT: AS14061:50,AS16276:50 - COUNTRY_LIMIT_PERCENT: CN:25 - ALERT_NTFY_URL: https://ntfy.example.invalid/alerts volumes: [waf-state:/data] networks: [internal, traefik] labels: traefik.enable: "true" traefik.http.routers.app.rule: Host(`app.example.invalid`) traefik.http.services.app.loadbalancer.server.port: "8080" + +volumes: + waf-state: + +networks: + internal: + traefik: + external: true ``` -A rule file is one rule per line: a name, what to match against, what to do, -and a regex. +The volume keeps bans and client history in a place you choose; without it +`smallwebwaf` still starts, on a volume docker creates for it. + +A rule file is one rule per line: a name, what to match against, what to do, and +a regex. ``` -env-file path offence:3 (?i)/\.env(\.[a-z]+)?$ -scanner-agent user_agent ban (?i)\b(sqlmap|nikto|nuclei|wpscan)\b +env-file path ban (?i)/\.env(\.[a-z]+)?$ +scanner-agent user_agent ban (?i)\b(sqlmap|nikto|nuclei|wpscan)\b ``` -[`SPEC.md`](SPEC.md) has the full design: every environment variable, the rule -file format, the state files, the log fields, the metrics, failure behaviour, -the build order, and the design questions still open for the owner. +[`SPEC.md`](SPEC.md) has the full design: every environment variable, the ban +rules, the rule file format, the state files, the log fields, the metrics, +failure behaviour and the build order. + +## Lookup database + +AS number and country lookups, and the biased limits that use them, need the +free IPinfo Lite database (`ipinfo_lite.mmdb`). You download it with your own +IPinfo account, mount it into the container, point `LOOKUP_DB_PATH` at it and +refresh it when you choose; `smallwebwaf` never downloads it itself. IPinfo +releases it under the Creative Commons Attribution-ShareAlike 4.0 International +License and asks for attribution, in its own words on https://ipinfo.io/lite: +"The attribution requirements can be met by giving our service credit as your +data source. Simply place a link to IPinfo on the website, application, or +social media account that uses our data." Its example of such a credit is a link +mentioning "IP address data is powered by IPinfo". A service that uses the +database through `smallwebwaf` should carry that link. ## Documents diff --git a/SPEC.md b/SPEC.md index 2fb5ec6..492aca0 100644 --- a/SPEC.md +++ b/SPEC.md @@ -1,20 +1,25 @@ # smallwebwaf SPEC (draft): protective reverse-proxy sidecar -Status: second draft, with the owner's rulings on storage, logging and metrics -applied. Nothing has been built yet. `EVALUATION.md` beside this file explains why no existing tool was -chosen. Points that turn on an owner decision are collected under "Questions for -the owner" and are marked "(open)" where they appear. +Status: third draft, with the owner's rulings to date applied. Nothing has been +built yet. `EVALUATION.md` beside this file explains why no existing tool was +chosen. ## Purpose One small container that sits between traefik and one application container. The traefik router for the public hostname points at the sidecar; the sidecar forwards to the application. The sidecar limits request and byte rates per -client, detects attacks, blocks abusers for a time and bans repeat offenders, -consults IP reputation sources, looks up the AS number and country of each -client, scales its limits for listed AS numbers and countries, and sends alerts. -Everything is configured by environment variables, apart from the attack -detection rules, which are read from a directory of hand-editable text files. +client, bounds the size and duration of every request and response, detects +attacks, bans abusers (briefly at first, for seven days on a clear sign of +attack, permanently when they keep at it), consults IP reputation sources, looks +up the AS number and country of each client, lowers its limits for listed AS +numbers and countries, and sends alerts. + +It is meant to protect an app on the open internet from the first request with +one setting, `UPSTREAM_URL`: every other setting has a default chosen for a +service facing the internet in 2026. Everything is configured by environment +variables, apart from the attack detection rules, which are read from a +directory of hand-editable text files. ## Non-goals @@ -24,170 +29,254 @@ detection rules, which are read from a directory of hand-editable text files. the sidecar and the app. - Defence against traffic floods that saturate the host's network link. That needs help upstream of the host. -- A web UI or a configuration file. Settings are environment variables; the - only files read are the rule files, which hold one regex per line and nothing - more elaborate. -- Sharing ban state between sidecars in the first version (open, see - Questions). +- A web UI or a configuration file. Settings are environment variables. Apart + from its own state files and the lookup database, the only files read are the + rule files, which hold one regex per line and nothing more elaborate. +- Sharing bans between sidecars in the first version: each sidecar keeps its + own. Running one CrowdSec engine per host, which every sidecar would report + its bans to, is reconsidered once real ban volumes are known. Reading a + CrowdSec decision list is supported (see Reputation). ## Architecture - One statically linked Go binary in one container, running as a non-root user, - read-only root filesystem, one writable volume for state. + read-only root filesystem, one writable volume for state. The image declares + that volume (`/data`), so docker supplies an anonymous one when none is + mounted, and a sidecar started with only `UPSTREAM_URL` has somewhere to + write. - Three listeners, of which only the first is routed by traefik: - the proxy listener; - a metrics listener serving Prometheus metrics, which can be switched off; - an admin listener for health and ban management. - Components inside the process: - Client identification: works out the real client IP from - `X-Forwarded-For`, trusting only configured proxy netblocks. - - Lookup: AS number and country from local database files, held in memory. + `X-Forwarded-For`, trusting only the proxy netblocks in `TRUSTED_PROXIES`, + by default the private address ranges. + - Lookup: AS number and country from one database file the operator + supplies, held in memory. - Reputation: static blocklists fetched on a schedule; DNSBL and reputation API queries made in the background and cached. - Counters: per-client request and byte counts per minute, hour and day, - held in memory; the hour and day counts are written out so they survive a - restart. - - Attack detection: regex rules read at start from a directory of plain - text rule files (see "Rule files"), Coraza with the OWASP Core Rule Set, - and simple signals (requests for listed trap paths, bursts of error - responses from the app). - - Ban ledger: offences, active bans and ban history, held in memory and - written to disk as JSON (see "Persistent state"). No database of any kind - is used. + held in memory and written out with the rest of the state, so they survive + a restart. + - Attack detection: regex rules read from a directory of plain text rule + files and read again whenever the files change (see "Rule files"), Coraza + with the OWASP Core Rule Set, and simple signals (requests for listed trap + paths, bursts of error responses). + - Ban ledger: active and past bans with their notes, and each client's + history, held in memory and kept in JSON files that the sidecar watches + for an admin's edits (see "Bans" and "Persistent state"). No database of + any kind is used. - Request log: one JSON object per request on stdout, optionally also sent - to a remote syslog or RELP endpoint (see "Request log"). + to a remote syslog server (see "Request log"). - Metrics: Prometheus counters and gauges on their own listener (see "Metrics endpoint"). - Alerting: a queue with de-duplication feeding webhook, Slack and ntfy senders. - Proxy: the standard library's `net/http/httputil.ReverseProxy`, streaming - in both directions, with WebSocket upgrade support. + in both directions within the size and time limits, with WebSocket upgrade + support. - Proposed libraries (to be checked against the owner's Go dependency defaults before any are added): - standard library for the proxy, HTTP clients, DNS, logging (`log/slog`), state files (`encoding/json`, `os.Rename`); - `github.com/corazawaf/coraza/v3` and `github.com/corazawaf/coraza-coreruleset` for attack detection; - - `github.com/oschwald/maxminddb-golang` to read `.mmdb` lookup databases; + - `github.com/oschwald/maxminddb-golang/v2`, the current major version, to + read the lookup database; + - `github.com/fsnotify/fsnotify` to notice files that are edited or added + while the sidecar runs; - `github.com/prometheus/client_golang` for metrics; - - remote log sending: the standard library's `log/syslog` is frozen and - does not write the current syslog format (RFC 5424), and no widely used Go - RELP client was found, so the library for this is open (see Questions). + - remote log sending is written in this project: the standard library's + `log/syslog` is frozen and writes only the older syslog format, not the + current one (RFC 5424). RELP is not in the first release; it is revisited + against the available packages afterwards. ## Data flow for one request Steps run in this order; the first step that produces a final answer ends -processing. +processing. The size and time limits (see "Configuration surface") apply to the +whole exchange. - Identify the client. - If the TCP peer is inside `TRUSTED_PROXIES`, walk `X-Forwarded-For` from the right and take the first address not inside `TRUSTED_PROXIES`. Otherwise use the TCP peer address and ignore the header. - IPv6 clients are grouped by prefix (`IPV6_GROUP_PREFIX`, default 64) for - counting and banning, because one abuser usually controls a whole /64. + counting and banning, because one abuser usually controls a whole /64. A + client is therefore one IPv4 address or one IPv6 /64. - Static lists. - In `ALLOW_NETS`: skip every check below and forward. Still counted for anomaly alerts. - In `DENY_NETS`: refuse. -- Ban ledger. An active ban: refuse (`BAN_RESPONSE`). -- Lookup AS number and country. Unknown is a valid result and changes nothing. +- Ban ledger. An active ban on the client's netblock: refuse with + `BAN_RESPONSE`. If a clear sign of attack caused the ban, the ban becomes + permanent (see "Bans"). +- Lookup AS number and country, when a lookup database is configured. Unknown is + a valid result and changes nothing. - Reputation. - Address inside a fetched blocklist: apply `BLOCKLIST_ACTION`. - Cached DNSBL or reputation API result: apply `REPUTATION_ACTION`. - No cached result: queue a background query and carry on. A first request is never delayed by a reputation query. - Work out the client's limit percentage: the lowest of the percentages that - apply (AS number, country, reputation). 100 if none applies. 0 means refuse. - (Lowest-wins versus multiplying is open.) + apply (AS number, country, reputation), or 100 if none applies. - Request rate limits. Unless the client is in `RATE_LIMIT_EXEMPT_NETS`, check - the minute, hour and day counters against each configured limit times the - percentage. Over any limit: answer 429 with `Retry-After`, record an offence. -- Rule files. The request is checked against the rules loaded from - `RULES_DIR`, in file name order then line order. Each rule that matches takes - its action: record offences, refuse with 403, ban at once, or only log. - Matching stops at the first rule that refuses or bans. + the minute, hour and day counters against each limit times the percentage. A + request over any of them breaks that limit: it is refused with `BAN_RESPONSE` + and the client is banned (see "Bans"). +- Rule files. The request is checked against the rules loaded from `RULES_DIR`, + in file name order then line order. Each rule that matches takes its action: + only log, refuse with 403, or refuse and ban. Matching stops at the first rule + that refuses or bans. - Core Rule Set inspection of the request (headers, URL, and body up to - `WAF_BODY_LIMIT`). In `block` mode a match at or over the anomaly threshold is - refused with 403 and recorded as an offence; in `detect` mode it is only - logged and alerted. + `WAF_BODY_LIMIT`). In `block` mode, the default, a match at or over the + anomaly threshold is refused with 403; in `detect` mode it is only logged and + alerted. - Forward to `UPSTREAM_URL`, streaming. Add `X-Forwarded-For` and, if enabled, `X-Client-ASN` and `X-Client-Country` for the app's own logs. - After the response. - - Add response bytes (and request body bytes) to the byte counters. Once a - client's byte total passes a byte limit times the percentage, following - requests get 429 until the window moves on, and an offence is recorded. A - response already in progress is not cut off. - - Count 401, 403 and 404 responses from the app per client; passing - `ERROR_BURST_THRESHOLD` within a minute records an offence. - - Update anomaly counters and evaluate alert thresholds. -- Offences and bans. - - `BAN_AFTER_OFFENCES` offences within `OFFENCE_WINDOW` creates a ban. - - Ban length follows `BAN_DURATIONS` by the number of earlier bans for that - client within `BAN_HISTORY_WINDOW`: first ban the first value, second ban - the second value, and so on. - - After `PERMANENT_BAN_AFTER` bans within `BAN_HISTORY_WINDOW`, the ban has - no expiry. - - A request for a path in `TRAP_PATHS` counts as `TRAP_OFFENCE_WEIGHT` - offences at once. + - Add response bytes (and request body bytes) to the byte counters. A client + whose byte total passes a byte limit times the percentage has broken that + limit and is banned. The response in progress is not cut off. + - Count the client's requests answered with 401, 403 or 404, whether the app + gave that answer or the sidecar refused the request after a rule file or + Core Rule Set match. More than `ERROR_BURST_THRESHOLD` of them within a + minute breaks a limit and bans the client. + - Update the client's history and the anomaly counters, and evaluate alert + thresholds. ## Counting method - Each window (minute, hour, day) uses two adjacent fixed buckets per client, - with the previous bucket weighted by how much of it still overlaps the - sliding window. This costs a few integers per client per window and avoids - the burst at bucket boundaries that a single fixed bucket allows. -- Counters are read and updated in memory only. The hour and day counters are - written to disk on a timer and at shutdown and loaded again at start (see - "Persistent state"), so a restart does not hand every client a fresh daily - allowance. Minute counters are not written; a restart forgives at most one - minute of counting. Buckets whose time has passed are discarded on load. + with the previous bucket weighted by how much of it still overlaps the sliding + window. This costs a few integers per client per window and avoids the burst + at bucket boundaries that a single fixed bucket allows. +- Counters are read and updated in memory; a request never waits on the disk. + They are written to `clients.json` every `STATE_COUNTER_INTERVAL` and at + shutdown and loaded again at start (see "Persistent state"), so a restart does + not hand every client a fresh allowance. Buckets whose time has passed are + discarded on load. - Memory is bounded by `MAX_TRACKED_CLIENTS`; when full, the least recently seen - clients with no recent offences are dropped first. + client is dropped first, with its history and its past bans. Active bans are + never dropped; one whose client is no longer tracked is dropped when it ends. + +## Bans + +A ban refuses every request from a netblock, answering with `BAN_RESPONSE` +(`403` by default), until the ban ends; a permanent ban never ends. The netblock +a ban covers is the client's IPv4 address (or the wider IPv4 prefix that +`BAN_SCOPE_V4_PREFIX` sets) or its IPv6 /64, and never more, so a permanent ban +does not reach neighbours who did nothing. + +An offence is a request the sidecar holds against the client: one that carries a +clear sign of attack, breaks a limit, or is refused by a rule file or the Core +Rule Set. Offences are counted by kind in the client's history. Two kinds lead +to a ban, each by its own rule: + +- A clear sign of attack: a request that matches a rule file rule whose action + is `ban`, or asks for a path in `TRAP_PATHS`. Examples are a probe for a + `.env` file or a `.git` directory, and a known scanner's user agent. + - The first one bans the netblock for `ATTACK_BAN_DURATION` (default `7d`). + - Any further request from the netblock while that ban lasts shows it is + malicious: the ban becomes permanent. + - Once that ban has run out, the netblock is served like any other, but its + next clear sign of attack bans it permanently at once. +- A broken limit: a request over a request limit, a response that takes the + client's byte total past a byte limit, or more error responses than + `ERROR_BURST_THRESHOLD` within a minute. + - The first such ban, or one that comes more than `LIMIT_BAN_REPEAT_WINDOW` + (default `24h`) after the last such ban ended, lasts `LIMIT_BAN_DURATION` + (default `1h`). + - Breaking a limit again within that window makes the new ban three times as + long as the last one: one hour, then 3, 9, 27 and 81 hours. + - A ban that would be longer than `MAX_BAN_DURATION` (default `7d`) is + permanent instead; with the defaults, that is the sixth ban in a row. + +A Core Rule Set match refuses only the request it matched, without a ban, +because the Core Rule Set has false positives; a `block` rule does the same. +Those refusals count toward the error burst like any other 403, so a client that +keeps setting them off is banned under the second rule. Requests refused under a +ban are not counted against limits, but they are counted in the client's history +and in the ban's notes. + +Every ban carries notes with what an admin needs to decide whether to lift it +(see "Persistent state"). An admin can add or lift a ban at any time by editing +`bans.json`, or through the admin listener when `ADMIN_TOKEN` is set. A ban +lifted before it ends is kept, marked as lifted, and does not count toward a +longer ban; deleting its entry from `bans.json` forgets it entirely. + +Clients of listed AS numbers and countries, and clients with a poor reputation, +get lower limits (see "Biased thresholds"), so the same rules ban them after +fewer requests. The result is always a ban, first a temporary one and then, for +repeated abuse, a permanent one. ## Configuration surface Conventions: lists are comma separated; netblocks are CIDR (a bare address means /32 or /128); durations use Go syntax plus `d` for days (`90s`, `15m`, `24h`, -`7d`); byte sizes accept `K`, `M`, `G` suffixes; an unset limit means that -limit is off. Every variable may instead be given as `NAME_FILE` pointing at a -file holding the value, for secrets and long lists. Invalid configuration stops -the process at start with a message naming the variable. At start the effective -configuration is logged with secrets masked. +`7d`); byte sizes accept `K`, `M`, `G` suffixes. + +- Only `UPSTREAM_URL` is required. Every other setting has a default chosen for + a service facing the internet in 2026, or stays off until it is given + something only the operator can supply: an alert destination, an account key, + the lookup database. +- Any limit or threshold can be switched off with the value `off`. +- A list set to an empty value is an empty list, and replaces the default. +- Every variable may instead be given as `NAME_FILE` pointing at a file holding + the value, for secrets and long lists. +- Settings, including `NAME_FILE` files, are read once at start; changing one + means restarting the container. The files the sidecar watches while it runs + are its state files, its rule files and the lookup database. +- Invalid configuration stops the process at start with a message naming the + variable. At start the effective configuration is logged with secrets masked. + +The settings, by group: - Core - `UPSTREAM_URL` (required): the application, for example `http://gitea:3000`. - `LISTEN_ADDR` (default `:8080`): proxy listener. - `ADMIN_LISTEN_ADDR` (default `127.0.0.1:9090`): admin listener. - - `ADMIN_TOKEN`: bearer token required for ban management endpoints. - - `INSTANCE_NAME`: included in every log line, metric and alert, for example - `fsn1app1/gitea`. + - `ADMIN_TOKEN`: bearer token required for the ban management endpoints. + Unset by default, which switches those endpoints off; bans are then + managed by editing `bans.json`. + - `INSTANCE_NAME` (default: the host name in `UPSTREAM_URL`, for example + `gitea`): included in every log line, metric and alert. Set it, for + example to `fsn1app1/gitea`, when several sidecars report to one place. - `MODE` (default `enforce`): `enforce`, or `observe` to log and alert on - every decision while refusing nothing. Meant for the first days of a - rollout. - - `TRUSTED_PROXIES` (required): netblocks whose `X-Forwarded-For` is - believed, normally the docker network traefik reaches the sidecar from. + every decision while refusing nothing. + - `TRUSTED_PROXIES` (default `10.0.0.0/8,172.16.0.0/12,192.168.0.0/16`, the + private address ranges): netblocks whose `X-Forwarded-For` is believed, + normally where traefik reaches the sidecar from. The header is used only + when the TCP peer is inside this list; otherwise the peer's own address is + the client, so a client that connects directly cannot claim another + address. A list given replaces the default; set but empty, it trusts + nothing. - `IPV6_GROUP_PREFIX` (default `64`). - - `MAX_TRACKED_CLIENTS` (default `500000`). + - `MAX_TRACKED_CLIENTS` (default `50000`): clients held in memory and in + `clients.json` (see "Persistent state"). - Persistent state - - `STATE_DIR` (default `/data`): the JSON state files and downloaded lookup - databases. - - `STATE_WRITE_DELAY` (default `2s`): after a ban or offence changes, wait - this long before writing, so a burst of changes becomes one write. - - `STATE_COUNTER_INTERVAL` (default `60s`): how often the hour and day - counters and the reputation cache are written. + - `STATE_DIR` (default `/data`): the JSON state files. The image declares + `/data` as a volume, so it is writable even when nothing is mounted; a + named volume, as in the examples below, keeps the state in a place that + outlives the container. + - `STATE_WRITE_DELAY` (default `2s`): after a ban changes, wait this long + before writing `bans.json`, so a burst of changes becomes one write. + - `STATE_COUNTER_INTERVAL` (default `60s`): how often `clients.json` and + `reputation.json` are written. - Logging - The request log on stdout is always on and has no switch. - `LOG_LEVEL` (default `info`): for the process's own messages (start-up, fetch failures, state writes), which are JSON lines on stdout too, marked `"type":"process"`. It does not filter the request log. - `LOG_REQUEST_HEADERS` (default - `accept,accept-language,accept-encoding,content-type,origin,range`): - extra request headers to record. `Authorization`, `Cookie` and - `Set-Cookie` values are never logged, only whether they were present. + `accept,accept-language,accept-encoding,content-type,origin,range`): extra + request headers to record. `Authorization`, `Cookie` and `Set-Cookie` + values are never logged, only whether they were present. - `LOG_REMOTE_URL`: when set, every log line is also sent to this endpoint. Forms: `syslog+udp://host:514`, `syslog+tcp://host:514`, - `syslog+tls://host:6514`, `relp://host:2514`, `relp+tls://host:2514`. + `syslog+tls://host:6514`. - `LOG_REMOTE_TLS_CA_FILE`: optional CA certificate for the `+tls` forms. - `LOG_REMOTE_BUFFER` (default `10000`): lines held in memory while the endpoint is unreachable; when full the oldest are dropped and counted. @@ -204,34 +293,87 @@ configuration is logged with secrets masked. which is the usual arrangement for a scraper on a private network. - Static lists - `ALLOW_NETS`: bypass everything (monitoring, the owner's own networks). - - `RATE_LIMIT_EXEMPT_NETS`: bypass request and byte limits only; attack - detection and bans still apply. + - `RATE_LIMIT_EXEMPT_NETS`: bypass request and byte limits only; the error + burst threshold, attack detection and bans still apply. - `DENY_NETS`: always refused. -- Request rate limits, per client (R1, R2) - - `RATE_LIMIT_PER_MINUTE`, `RATE_LIMIT_PER_HOUR`, `RATE_LIMIT_PER_DAY`. +- Request rate limits, per client (R1, R2). Breaking one bans the client (see + "Bans"), so the defaults sit several times above what one busy person + produces: a browser loading a heavy page makes a few hundred requests, a git + clone a handful, and several people often share one address. + - `RATE_LIMIT_PER_MINUTE` (default `1000`), `RATE_LIMIT_PER_HOUR` (default + `10000`), `RATE_LIMIT_PER_DAY` (default `50000`). - `RATE_LIMIT_EXEMPT_PATHS`: path prefixes not counted (static assets, health checks). -- Byte limits, per client - - `BYTES_LIMIT_PER_MINUTE`, `BYTES_LIMIT_PER_HOUR`, `BYTES_LIMIT_PER_DAY`. +- Byte limits, per client. A response's bytes are counted when it ends, so every + default sits above the largest response allowed (`CLIENT_RESPONSE_MAX_BYTES`, + 5 GB) and no single download breaks one. + - `BYTES_LIMIT_PER_MINUTE` (default `10G`), `BYTES_LIMIT_PER_HOUR` (default + `20G`), `BYTES_LIMIT_PER_DAY` (default `50G`). - `BYTES_COUNT` (default `both`): `response`, `request` or `both`. +- Size and time limits, per request, in both directions. The client-facing + limits apply between the client and the sidecar, the app-facing ones between + the sidecar and the app. Bodies stream straight through, so a request body + reaches the app while the client is still sending it, and of two matching + limits the lower one acts first. + - `CLIENT_REQUEST_TIMEOUT` (default `60s`): how long a client may take to + send its whole request, headers and body. + - `CLIENT_REQUEST_MAX_BYTES` (default `100M`): the largest request body a + client may send. + - `CLIENT_RESPONSE_TIMEOUT` (default `30m`): how long the sidecar may take + to deliver one response to the client, from the end of the request to the + last byte. + - `CLIENT_RESPONSE_MAX_BYTES` (default `5G`): the largest response body sent + to a client. + - `UPSTREAM_REQUEST_TIMEOUT` (default `60s`): how long the sidecar may take + to connect to the app and send it the whole request. + - `UPSTREAM_REQUEST_MAX_BYTES` (default `100M`): the largest request body + sent to the app. + - `UPSTREAM_RESPONSE_TIMEOUT` (default `30m`): how long the app may take to + send one whole response, from the end of the request to the last byte. + - `UPSTREAM_RESPONSE_MAX_BYTES` (default `5G`): the largest response body + taken from the app. + - When a limit is passed before the response has started, the sidecar + answers itself: `413` for a request body that is too large, `408` for a + client that is too slow, `502` for a response that is too large, `504` for + an app that is too slow. A request that announces a body larger than its + limit is refused before anything reaches the app. Once the response has + started it can only be cut off, and the connection is closed. + - A WebSocket connection leaves these limits behind once it is upgraded: it + stays open until either side closes it. - Lookup of AS number and country (R7) - - `ASN_DB_PATH`, `COUNTRY_DB_PATH`: `.mmdb` files mounted into the - container; or - - `ASN_DB_URL`, `COUNTRY_DB_URL`: downloaded into `STATE_DIR` at start and - every `LOOKUP_DB_REFRESH` (default `7d`). One file may serve both. - - Known free sources in this format: IPinfo Lite (country and AS number in - one file, free account token), DB-IP Lite, MaxMind GeoLite2 (free - account). Choice is open. + - `LOOKUP_DB_PATH`: the IPinfo Lite database in its `.mmdb` form + (`ipinfo_lite.mmdb`), one file that carries both country and AS number; + the sidecar reads its `asn`, `as_name` and `country_code` fields. The + operator downloads it with a free IPinfo account and mounts it read-only. + The sidecar never fetches it and holds no account or token for it. + - Refreshing the file is the operator's business; IPinfo updates it daily. + The sidecar notices when the file is replaced and reads it again; a + replacement it cannot read is ignored, logged and alerted once, and the + file it has stays in use. Mount the directory that holds the file rather + than the file itself: docker does not show a single mounted file being + replaced on the host. + - Unset by default, which means no lookups. A setting that needs them (the + biased thresholds below, `ADD_LOOKUP_HEADERS`, the per-AS-number anomaly + thresholds) given without `LOOKUP_DB_PATH` stops the start with a message + naming both. A configured file that is missing or unreadable at start + stops the start too. - `ADD_LOOKUP_HEADERS` (default `false`): pass `X-Client-ASN` and `X-Client-Country` to the app. - - Missing or unreadable database: start anyway, treat every client as - unknown, log a warning and send one alert. -- Biased thresholds (R8) +- Biased thresholds (R8). The lists are empty by default: they need the lookup + database, which only the operator can supply. - `ASN_LIMIT_PERCENT`: for example `AS14061:50,AS16276:50,AS45102:25`. - Listed AS numbers get that percentage of every request and byte limit. - `0` refuses outright. + Clients in a listed AS number get that percentage of every request and + byte limit, so the rules in "Bans" ban them after fewer requests than + others. `0` is a zero allowance: the client's first request breaks a limit + and bans it. - `COUNTRY_LIMIT_PERCENT`: same form with ISO country codes, for example - `CN:25,RU:50`. + `CN:25,RU:50`. Each client from a listed country gets that percentage of + the normal per-client limits. + - No budget is shared by a whole country or AS number: one abuser could use + it up and lock out everyone else there, the denial of service this tool + exists to prevent. + - When more than one percentage applies to a client (its AS number, its + country, a reputation hit), the lowest applies. - `ASN_BYTES_PERCENT`, `COUNTRY_BYTES_PERCENT`: optional overrides applied to byte limits only, when the byte percentage should differ from the request percentage. @@ -239,43 +381,67 @@ configuration is logged with secrets masked. place. - `ASN_LIMIT_PERCENT_URL`: optional URL of a text file of `AS:percent` lines, so one abuse-source list can be shared by every sidecar in the - fleet. + fleet. It is fetched and refreshed like the blocklists. - Attack detection (R9) - - `RULES_DIR` (default `/etc/smallwebwaf/rules.d`): directory of rule files - read once at start; format under "Rule files". The image ships a default - file there. Mounting a directory over it replaces the defaults; mounting - single files into it adds to them. + - `RULES_DIR` (default `/etc/smallwebwaf/rules.d`): directory of rule files, + read at start and again whenever a file in it changes; format under "Rule + files". - `RULES_ENABLED` (default `true`): `false` skips rule files entirely. - - `WAF_MODE` (default `detect`): `off`, `detect`, `block`. + - `WAF_MODE` (default `block`): `off`, `detect` (log and alert only), or + `block`. - `WAF_PARANOIA_LEVEL` (default `1`), `WAF_ANOMALY_THRESHOLD` (default `5`): - the Core Rule Set's own two tuning values. - - `WAF_DISABLED_RULES`: rule ids to switch off when an app trips a false - positive. + the Core Rule Set's own two tuning values, at the Core Rule Set's own + defaults. + - `WAF_DISABLED_RULES` (default `920420,920440`): rule ids to switch off + when an app trips a false positive. The two in the default refuse requests + by content type and by file extension; in front of a code forge they would + refuse git's clone and push over HTTP and the display of source files such + as `.sh` or `.sql`. A list given replaces the default, so include them in + it. - `WAF_EXEMPT_PATHS`: path prefixes not inspected. - `WAF_BODY_LIMIT` (default `128K`): bodies are inspected up to this size and streamed beyond it without buffering, so large uploads and git pushes are not held in memory. - `TRAP_PATHS`: paths the app never serves and only scanners ask for, for - example `/.env,/wp-login.php,/.git/config`. `TRAP_OFFENCE_WEIGHT` - (default `3`). This is the env-var short form of a `path` rule with an - `offence` action, for deployments that mount no rule files. - - `ERROR_BURST_THRESHOLD` (default off): 401, 403 and 404 responses per - client per minute that count as an offence. -- Blocks and bans (R5) - - `BAN_AFTER_OFFENCES` (default `3`), `OFFENCE_WINDOW` (default `10m`). - - `BAN_DURATIONS` (default `1h,24h,7d`). - - `PERMANENT_BAN_AFTER` (default `4`; `0` never bans permanently), - `BAN_HISTORY_WINDOW` (default `90d`). + example `/wp-login.php,/xmlrpc.php` in front of gitea. A request for one + is a clear sign of attack. This is the env-var short form of a `path` rule + with the `ban` action, for deployments that mount no rule files. + - `ERROR_BURST_THRESHOLD` (default `30`): responses with status 401, 403 or + 404 per client per minute; more than this breaks a limit (see "Bans"). + Scanners walking lists of paths cause hundreds a minute; a person rarely + causes more than a few. +- Bans (R5), following the rules under "Bans" + - `ATTACK_BAN_DURATION` (default `7d`): the ban for a first clear sign of + attack. + - `LIMIT_BAN_DURATION` (default `1h`): the ban for a first broken limit. + - `LIMIT_BAN_REPEAT_WINDOW` (default `24h`): breaking a limit again within + this time after a ban for a broken limit ended makes the next ban three + times as long. + - `MAX_BAN_DURATION` (default `7d`): a ban that would be longer is permanent + instead. - `BAN_RESPONSE` (default `403`): `403`, `429`, or `close` to drop the - connection without an answer. + connection without an answer. `403` makes a ban easy to recognise when + debugging; `close` tells the client nothing. - `BAN_SCOPE_V4_PREFIX` (default `32`): widen to for example `24` to ban the surrounding netblock. -- Reputation (R6) - - `BLOCKLIST_URLS`: text files of addresses and netblocks, for example - Spamhaus DROP, FireHOL level 1, the Tor exit list. Refreshed every - `BLOCKLIST_REFRESH` (default `6h`); the last good copy is kept on failure. - - `BLOCKLIST_ACTION` (default `limit:25`): `deny`, `limit:`, or - `log`. +- Reputation (R6). No source is on by default. A DNS blocklist would be told the + address of every visitor. AbuseIPDB and CrowdSec need an account or an engine + of the operator's own. A downloaded list reveals nothing about visitors, but + each sidecar fetches its own copy, and the list most fit to be a default, + Spamhaus DROP, may be fetched at most once a day: a default would break that + on any host that runs several sidecars. + - `BLOCKLIST_URLS`: text files of addresses and netblocks, one per line; + anything after a `;` or `#` on a line is ignored. For example the Spamhaus + DROP list, `https://www.spamhaus.org/drop/drop.txt`; Spamhaus asks that it + be fetched at most once a day and that products using it credit The + Spamhaus Project. Refreshed every `BLOCKLIST_REFRESH` (default `24h`); the + last good copy is kept on failure and across restarts. + - `BLOCKLIST_ACTION` (default `deny`): `deny`, `limit:`, or `log`. + `deny` refuses every request from a listed address, which suits lists of + networks that send nothing legitimate, such as DROP. `limit:` + gives listed clients that percentage of every limit, so they are banned + after fewer requests; it suits a list of addresses shared with ordinary + visitors, such as Tor exits. - `DNSBL_ZONES`: for example `dnsbl.dronebl.org`. Zones meant for mail (lists of residential ranges) will block ordinary visitors and should not be used; Spamhaus zones need their keyed query service, given as a zone @@ -288,18 +454,20 @@ configuration is logged with secrets masked. so the budget is spent on suspects. - `CROWDSEC_LAPI_URL`, `CROWDSEC_LAPI_KEY`: optional. If a CrowdSec engine exists on the host, pull its decision list on a schedule and treat listed - addresses as banned. This is how a fleet-wide blocklist can arrive - without the sidecar depending on CrowdSec. + addresses as banned. This is how a fleet-wide blocklist can arrive without + the sidecar depending on CrowdSec. - `REPUTATION_ACTION` (default `limit:25`): `deny`, `limit:`, or - `log`, for DNSBL and API hits. + `log`, for DNSBL and API hits. Such verdicts are less certain than a + blocklist, so by default a listed client gets a quarter of every limit and + is banned after a quarter of the requests. - `REPUTATION_CACHE_TTL` (default `24h`), `REPUTATION_TIMEOUT` (default `2s`). - Alerting (R3) - - `ALERT_WEBHOOK_URL`: JSON POST; schema below. - `ALERT_WEBHOOK_HEADERS`: optional `Name:value` pairs for authentication. + - `ALERT_WEBHOOK_URL`: JSON POST; schema below. `ALERT_WEBHOOK_HEADERS`: + optional `Name:value` pairs for authentication. - `ALERT_SLACK_WEBHOOK_URL`: Slack incoming webhook, formatted message. - - `ALERT_NTFY_URL` (full topic URL), `ALERT_NTFY_TOKEN`: title, priority - and tags set from the event type. + - `ALERT_NTFY_URL` (full topic URL), `ALERT_NTFY_TOKEN`: title, priority and + tags set from the event type. - `ALERT_EVENTS` (default `ban,permanent_ban,waf_block,anomaly,reputation_hit,source_failure`): which event types are sent. @@ -308,14 +476,18 @@ configuration is logged with secrets masked. repeats is included in the next one. - `ALERT_MAX_PER_HOUR` (default `60`): beyond this, alerts are rolled into one summary per hour so a wide attack cannot flood the channel. -- Anomaly thresholds, alert only, nothing is blocked (R4) +- Anomaly thresholds, alert only, nothing is blocked (R4). None is set by + default, and a threshold left unset sends no alert: these only alert, an alert + needs a destination only the operator can supply, and what counts as unusual + depends on each service's normal traffic, which the metrics show. - Per client: `ANOMALY_CLIENT_REQUESTS_PER_MINUTE`, `ANOMALY_CLIENT_REQUESTS_PER_HOUR`, `ANOMALY_CLIENT_BYTES_PER_MINUTE`, `ANOMALY_CLIENT_BYTES_PER_HOUR`. - Per surrounding netblock (`ANOMALY_NET_V4_PREFIX` default `24`, `ANOMALY_NET_V6_PREFIX` default `48`): `ANOMALY_NET_REQUESTS_PER_MINUTE`, `..._PER_HOUR`, `ANOMALY_NET_BYTES_PER_MINUTE`, `..._PER_HOUR`. - - Per AS number: `ANOMALY_ASN_REQUESTS_PER_MINUTE`, `..._PER_HOUR`, + - Per AS number, which needs the lookup database: + `ANOMALY_ASN_REQUESTS_PER_MINUTE`, `..._PER_HOUR`, `ANOMALY_ASN_BYTES_PER_MINUTE`, `..._PER_HOUR`. - Whole service: `ANOMALY_TOTAL_REQUESTS_PER_MINUTE`, `..._PER_HOUR`, `ANOMALY_TOTAL_BYTES_PER_MINUTE`, `..._PER_HOUR`. @@ -329,10 +501,11 @@ configuration is logged with secrets masked. ## Rule files -A required feature: at start the sidecar reads every `*.rules` file in -`RULES_DIR` and checks each request against them. The files are plain text meant -to be edited by hand, so that a new scanning pattern seen in the request log can -be turned into a rule in one line. +A required feature: the sidecar reads every `*.rules` file in `RULES_DIR` and +checks each request against them. The files are plain text meant to be edited by +hand, so that a new scanning pattern seen in the request log can be turned into +a rule in one line. The sidecar watches the directory: a file edited, added or +removed there takes effect while the sidecar runs. - One rule per line, four fields separated by spaces or tabs; the fourth field runs to the end of the line: @@ -343,23 +516,24 @@ be turned into a rule in one line. - Blank lines and lines starting with `#` are ignored. - `id`: a short name of letters, digits, `-` and `_`, unique across all files. - It appears in the request log, metrics, alerts and offence history. + It appears in the request log, metrics, alerts and ban notes. - `target`: what the regex is matched against. - `path`: the URL path as received, before any decoding. - `query`: the raw query string. - - `uri`: path and query together, both as received and once - percent-decoded, so an encoded probe cannot slip past. + - `uri`: path and query together, both as received and once percent-decoded, + so an encoded probe cannot slip past. - `method`, `host`, `user_agent`, `referer`. - `header:`: any one request header. - Request bodies are not available to rule files; body inspection is the Core Rule Set's job. - `action`: - `log`: note the match in the request log and do nothing else. - - `offence` or `offence:`: record one or `n` offences and forward the - request. Enough offences within `OFFENCE_WINDOW` creates a ban. - - `block` or `block:`: refuse with 403 and record one or `n` offences. - - `ban`: refuse and ban the client at once, at whatever length its ban - history calls for. + - `block`: refuse the request with 403. The client is not banned for it, but + the refusal counts toward the error burst (see "Bans"). + - `ban`: the request is a clear sign of attack. Refuse it and ban the + client's netblock for seven days (`ATTACK_BAN_DURATION`); any further + request during those days, or a later clear sign of attack, makes the ban + permanent (see "Bans"). Use it only for requests no real visitor sends. - `regex`: Go regular expression syntax (RE2). It has no backreferences or lookaround, and in exchange matching time is linear in the input, so no rule can be made to stall the proxy. `(?i)` at the front makes a rule @@ -367,11 +541,19 @@ be turned into a rule in one line. anchor with `^` and `$` when that is not wanted. - Files are read in name order (`00-default.rules` before `50-gitea.rules`), rules in line order. -- Rules are compiled once at start and held in memory. A line that does not - parse, a regex that does not compile, or a duplicate id stops the process - with a message naming the file and line. A missing or empty directory is not - an error: the log says that no rules were loaded. Changing rules means - restarting the container; reloading while running is given up for now. +- Rules are compiled when their file is read and held in memory. At start, a + line that does not parse, a regex that does not compile, or a duplicate id + stops the process with a message naming the file and line. While running, the + same faults leave the rules as they were, including the earlier version of + that file, and the log and one alert name the file and line; once the file is + fixed, it is read again. A missing or empty directory is not an error: the log + says that no rules were loaded. +- The image ships a default file in `RULES_DIR`. Mounting a directory over it + replaces the defaults; mounting single files into it adds to them. Docker does + not show a single mounted file being replaced on the host, which is how many + editors save, so rules meant to be edited while the sidecar runs belong in a + mounted directory, with a copy of the default file if the defaults are to + stay. - In `MODE=observe` every action is logged as what would have happened and nothing is refused. - Clients in `ALLOW_NETS` are not checked. @@ -381,73 +563,112 @@ Example file: ``` # 00-default.rules: probes no real visitor sends -# id target action regex -env-file path offence:3 (?i)/\.env(\.[a-z]+)?$ -git-dir path offence:3 ^/\.git/(config|HEAD|index)$ -wp-probe path offence:3 (?i)^/(wp-login\.php|xmlrpc\.php|wp-admin/) -php-shell path block:3 (?i)/(shell|c99|r57|wso|alfa)\.php$ -path-traversal uri block (\.\./){2,} -scanner-agent user_agent ban (?i)\b(sqlmap|nikto|nuclei|masscan|zgrab|wpscan)\b -empty-agent user_agent log ^$ +# id target action regex +env-file path ban (?i)/\.env(\.[a-z]+)?$ +git-dir path ban ^/\.git/(config|HEAD|index)$ +php-shell path ban (?i)/(shell|c99|r57|wso|alfa)\.php$ +scanner-agent user_agent ban (?i)\b(sqlmap|nikto|nuclei|masscan|zgrab|wpscan)\b +path-traversal uri block (\.\./){2,} +empty-agent user_agent log ^$ ``` The image ships one default file of this kind. It is kept short and limited to -patterns that are wrong for every app; anything app-specific (for example -blocking `/wp-login.php` is harmless in front of gitea and fatal in front of -WordPress) belongs in a file the deployer mounts. +patterns that are wrong for every app. Anything app-specific belongs in a file +the deployer mounts: a request for `/wp-login.php`, for example, is a clear sign +of attack in front of gitea and an ordinary login in front of WordPress. + +``` +# 50-gitea.rules: WordPress probes, which gitea never serves +wp-probe path ban (?i)^/(wp-login\.php|xmlrpc\.php|wp-admin/) +``` ## Persistent state -All state lives in memory. No database is used. What must survive a restart is -written to `STATE_DIR` as JSON files; the files are read once at start and never -during a request. The whole state is expected to stay under 10 MiB, and reading -or writing even ten times that as JSON takes well under a second, so whole-file -rewrites are cheap enough to need nothing cleverer. +All state lives in memory, and the files in `STATE_DIR` hold a copy of all of +it, so an orderly stop loses nothing. No database is used, and nothing is read +from disk while serving a request. The files are meant for people as well: an +admin can read or edit them at any time, and the running sidecar takes the edit +in. - Files, each holding one kind of state: - - `bans.json`: active and past bans. Per entry: client or netblock, start, - expiry (`null` for permanent), reason, which ban this is for that client - (first, second, third), source (`local`, `admin`, `crowdsec`), and when - it was lifted, if it was. - - `offences.json`: offender history. Per entry: client, time, kind (`rate`, - `bytes`, `waf`, `trap`, `error_burst`), short detail. Entries older than - `BAN_HISTORY_WINDOW` are dropped at each write. - - `counters.json`: the hour and day request and byte counters per client. - - `reputation.json`: cached DNSBL and reputation API verdicts with the time - each was fetched. + - `bans.json`: every active and past ban. Per entry: the netblock, start, + expiry (`null` for permanent), what caused it (`attack`, `limit`, `admin` + or `crowdsec`), a short reason, when it was lifted if an admin lifted it, + and a `notes` field (below). + - `clients.json`: per client, its minute, hour and day counters and its + history since it was first seen: first and last time seen, the AS number, + AS name and country last looked up, total requests and bytes in each + direction, how many requests were forwarded and how many refused, + responses by status class, and offences by kind. Clients that were never + banned are kept too, so every client's history survives a restart; + `GET /clients/` shows it. + - `reputation.json`: the last good copy of each list fetched from a URL, and + cached DNSBL and reputation API verdicts, each with the time it was + fetched. +- Ban notes. The notes on a ban hold what an admin needs to decide whether to + lift it, drawn from what the sidecar already knows; nothing is looked up to + fill them: + - the AS number, AS name and country (empty without a lookup database); + - what was broken: the rule ids and target that matched, or the limit, its + window, the count reached and the client's limit percentage with what set + it; and any reputation sources that listed the client; + - the requests that caused the ban, up to the last ten: time, method, host, + path, status and user agent; + - how many requests counted toward the ban, and the time span over which + they came; + - the netblock's total requests since it was first seen, and the requests + refused under this ban so far, kept up to date while the ban lasts; + - how many earlier bans of each kind the netblock has had. +- Size: each tracked client takes about 2 KiB in `clients.json`, so at the + default `MAX_TRACKED_CLIENTS` of 50,000 the file can reach about 100 MiB; a + small service usually tracks a few thousand clients and a few MiB. The other + files stay small. Writing a whole file of that size takes about a second, in + the background, so rewriting whole files still needs nothing cleverer. - Format: indented JSON with a top-level `version` number, entries sorted by client address, times in RFC 3339 UTC, durations and sizes as plain numbers - with the unit in the field name. The aim is that a person can open - `bans.json` in an editor, find an address, and remove or add an entry. + with the unit in the field name. The aim is that a person can open `bans.json` + in an editor, find an address, and remove or add an entry. - Writing: - serialise from a snapshot taken under the lock, so requests are not held up while the file is written; - write to a temporary file in the same directory, sync it, rename it over the real name, sync the directory. A crash at any point leaves either the old complete file or the new complete file, never a partial one; - - `bans.json` and `offences.json` are written `STATE_WRITE_DELAY` after a - change; `counters.json` and `reputation.json` every - `STATE_COUNTER_INTERVAL`; all four on orderly shutdown (SIGTERM). -- Reading: - - at start only. Active bans go into an in-memory prefix lookup; everything - else into maps. Expired counters and cache entries are discarded. - - a missing file means empty state and is normal on first run. - - a file that does not parse, or has an unknown `version`, stops the - process with a message naming the file and position. Starting with empty - bans would silently forgive every repeat offender, and since writes are - atomic a broken file can only come from a hand edit, which the editor - should hear about. `counters.json` and `reputation.json` are the - exception: if unreadable they are logged, set aside with a `.bad` suffix, - and the process starts with them empty. -- Hand edits are made with the container stopped; a running process will - overwrite the file at its next write. Changes to a running process go through - the admin endpoints. Reloading files into a running process is given up for - now. -- `STATE_DIR` not writable at start: the process exits. A write that fails - while running: state stays correct in memory, the failure is logged, counted - in metrics, alerted once per cooldown, and retried at the next write. -- What a hard kill can lose: bans and offences from the last - `STATE_WRITE_DELAY`, counters from the last `STATE_COUNTER_INTERVAL`. Losing + - `bans.json` is written `STATE_WRITE_DELAY` after a change; `clients.json` + and `reputation.json` every `STATE_COUNTER_INTERVAL`; all three on orderly + shutdown (SIGTERM); + - before writing a file, the sidecar checks whether it changed on disk since + the sidecar last read or wrote it; if it did, the sidecar takes that edit + in first (below), so an admin's edit is never overwritten. +- Reading at start: + - Active bans go into an in-memory prefix lookup; everything else into maps. + Counter buckets and cache entries whose time has passed are discarded. + - A missing file means empty state and is normal on first run. + - A file that does not parse, or has an unknown `version`, stops the process + with a message naming the file and position. Starting with empty state + would silently forgive every repeat offender, and since writes are atomic + a broken file can only come from a hand edit, which the editor should hear + about. +- Edits while running: + - The sidecar watches `STATE_DIR` and notices a file that is edited, + replaced or added. It tells its own writes from an admin's by comparing + the file with what it last wrote. + - An edit that parses is taken in at once: what the file says replaces what + the sidecar held for that file. Removing a ban's entry from `bans.json` + lifts the ban; adding an entry bans. Changes the sidecar made after the + admin opened the file, such as a new ban, are lost when the admin saves + over them; most editors warn when a file changed on disk while it was + open. + - An edit that does not parse does not stop the running sidecar. It keeps + the state it has, renames the edited file to `.bad` (for example + `bans.json.bad`) so the edit is kept, writes the file again from memory, + and logs and alerts once with the file and the position of the error. The + admin fixes the `.bad` file and moves it back. +- `STATE_DIR` not writable at start: the process exits. A write that fails while + running: state stays correct in memory, the failure is logged, counted in + metrics, alerted once per cooldown, and retried at the next write. +- What a hard kill can lose: ban changes from the last `STATE_WRITE_DELAY`, and + counts, history and reputation from the last `STATE_COUNTER_INTERVAL`. Losing the volume loses ban history, not service. ## Request log @@ -464,18 +685,18 @@ handling while `docker logs` keeps working. passed to the app), `peer_ip` (the TCP peer, normally traefik), `forwarded_for` (the header as received), `client_group` (the /64 or configured prefix used for counting), `asn`, `as_name`, `country`, - `content_type`, `content_length`, the headers named in - `LOG_REQUEST_HEADERS`, `has_authorization` and `has_cookie` as booleans, - `websocket` when the connection was upgraded. + `content_type`, `content_length`, the headers named in `LOG_REQUEST_HEADERS`, + `has_authorization` and `has_cookie` as booleans, `websocket` when the + connection was upgraded. - Response detail: `response_content_type`, `upstream_status` (differs from `status` when the sidecar answered itself), `cache_control`, `location` on redirects, `aborted` when the client went away early. -- Decision: `action` (`forward`, `rate_limited`, `bytes_limited`, `banned`, - `denied`, `waf_blocked`, `upstream_error`), `would_action` in `observe` mode, - `limit_percent` and which rule set it, `counts` (the client's minute, hour - and day request and byte totals after this request), `limit_hit` (which - window), `rule_ids` (rule file rules that matched), `waf_rule_ids`, - `waf_score`, `reputation` (sources that listed the +- Decision: `action` (`forward`, `banned`, `denied`, `rate_limited`, + `rule_blocked`, `waf_blocked`, `too_large`, `timed_out`, `upstream_error`), + `would_action` in `observe` mode, `limit_percent` and which rule set it, + `counts` (the client's minute, hour and day request and byte totals after this + request), `limit_hit` (which window), `rule_ids` (rule file rules that + matched), `waf_rule_ids`, `waf_score`, `reputation` (sources that listed the client), `offence` when one was recorded, `ban_expires`. - Timings in milliseconds: `duration_total`, `duration_checks` (everything the sidecar did before forwarding), `duration_waf`, `duration_upstream_connect`, @@ -487,26 +708,23 @@ handling while `docker logs` keeps working. - Remote sending: - syslog forms send each line as the message of an RFC 5424 record, with octet-counted framing on TCP and TLS; - - RELP forms send the same record and wait for the receiver's - acknowledgement, so lines are not lost across a receiver restart. This is - the form to use with rsyslog; - sending happens on its own goroutine from a bounded buffer (`LOG_REMOTE_BUFFER`). An unreachable or slow endpoint never delays a request and never stops stdout; it reconnects with backoff, drops the - oldest lines when the buffer is full, and counts the drops in metrics. - UDP gives no delivery signal at all and is offered only for - compatibility. + oldest lines when the buffer is full, and counts the drops in metrics. UDP + gives no delivery signal at all and is offered only for compatibility. ## Metrics endpoint -Prometheus text format on `METRICS_LISTEN_ADDR` at `METRICS_PATH`, on by default, -switched off with `METRICS_ENABLED=false`. It has its own listener so it can be -reached by a scraper without exposing ban management. +Prometheus text format on `METRICS_LISTEN_ADDR` at `METRICS_PATH`, on by +default, switched off with `METRICS_ENABLED=false`. It has its own listener so +it can be reached by a scraper without exposing ban management. -- Traffic: requests and bytes in and out, by status class and `action`; - request duration and upstream duration histograms; requests in flight. -- Limits and bans: limit hits by window and kind (requests or bytes), offences - by kind, bans created by ordinal, permanent bans, active bans (gauge). +- Traffic: requests and bytes in and out, by status class and `action`; request + duration and upstream duration histograms; requests in flight. +- Limits and bans: limit hits by window and kind (requests or bytes), size and + time limit hits by limit, offences by kind, bans created by cause, permanent + bans, active bans (gauge). - Attack detection: rule file matches by rule id and action, and the number of rules loaded; Core Rule Set matches by mode and rule id (label limited to the rules that actually fired). @@ -515,10 +733,11 @@ reached by a scraper without exposing ban management. `other`, so the label set stays bounded; reputation queries, hits, failures and remaining daily budget by source; age of each blocklist and lookup database. -- Housekeeping: tracked clients (gauge), state file writes, write failures, - last successful write time and size per file; alerts sent, failed and - suppressed by destination; remote log lines sent, dropped and buffer depth; - the standard Go runtime and process metrics. +- Housekeeping: tracked clients (gauge), state file writes, write failures, last + successful write time and size per file; files read again after an edit, and + edits set aside because they did not parse; alerts sent, failed and suppressed + by destination; remote log lines sent, dropped and buffer depth; the standard + Go runtime and process metrics. - No metric carries a client IP address as a label; per-address questions are answered by the request log and `GET /clients/`. @@ -526,9 +745,11 @@ reached by a scraper without exposing ban management. - `GET /healthz`: for the container health check. - `GET /bans`, `POST /bans` (client or netblock, duration, reason), - `DELETE /bans/`: need `ADMIN_TOKEN`. -- `GET /clients/`: current counters, lookup result, reputation, offences; - for answering "why was this address refused". + `DELETE /bans/`: need `ADMIN_TOKEN`, and are switched off while it is + unset. Editing `bans.json` does the same without a token. +- `GET /clients/`: current counters, history, lookup result, reputation, + offences, and bans with their notes; for answering "why was this address + refused" and "what has this netblock been doing". ## Alert webhook schema @@ -538,13 +759,14 @@ One JSON object per alert: - `client`, `netblock`, `asn`, `as_name`, `country` - `reason`: short human-readable sentence - `detail`: event-specific fields, for example `window`, `count`, `limit`, - `limit_percent`, `rule_ids`, `path`, `ban_ordinal`, `ban_expires` + `limit_percent`, `rule_ids`, `path`, `ban_expires`, and for a ban its notes - `suppressed_repeats`: number of identical alerts held back by the cooldown ## Deployment as a sidecar -docker-compose shape, using gitea as the example; upaas deployments follow the -same shape if upaas can run a second container for a service (open): +docker-compose shape, using gitea as the example. The app carries no traefik +labels and publishes no ports; the sidecar carries the labels and points at the +app's hostname. `UPSTREAM_URL` is the only setting it needs: ```yaml services: @@ -559,93 +781,125 @@ services: user: "65532:65532" environment: UPSTREAM_URL: http://gitea:3000 - TRUSTED_PROXIES: 172.18.0.0/16 - INSTANCE_NAME: fsn1app1/gitea - MODE: observe - RATE_LIMIT_PER_MINUTE: "120" - RATE_LIMIT_PER_HOUR: "2000" - RATE_LIMIT_PER_DAY: "10000" - BYTES_LIMIT_PER_HOUR: 2G - ASN_DB_URL: https://example.invalid/asn-country.mmdb - ASN_LIMIT_PERCENT: AS14061:50,AS16276:50 - COUNTRY_LIMIT_PERCENT: CN:25 - WAF_MODE: detect - TRAP_PATHS: /.env,/wp-login.php,/.git/config - BLOCKLIST_URLS: https://www.spamhaus.org/drop/drop.txt - ALERT_NTFY_URL: https://ntfy.example.invalid/fleet-alerts - LOG_REMOTE_URL: relp://logs.example.invalid:2514 - METRICS_LISTEN_ADDR: :9100 volumes: - gitea-guard-state:/data - - ./50-gitea.rules:/etc/smallwebwaf/rules.d/50-gitea.rules:ro networks: [internal, traefik] labels: traefik.enable: "true" traefik.http.routers.gitea.rule: Host(`git.example.invalid`) traefik.http.services.gitea.loadbalancer.server.port: "8080" + +volumes: + gitea-guard-state: + +networks: + internal: + traefik: + external: true ``` -- The application container leaves the traefik network, so the sidecar cannot - be bypassed. -- Traefik routes only port 8080. The metrics port is reached by the scraper - over a docker network it shares with the sidecar; the admin port stays on - loopback inside the container and is used through `docker exec`. -- The state volume holds a few small JSON files; it needs no backup beyond - whatever the host already does. -- SSH access to gitea does not pass through the sidecar and is not protected - by it. -- Rollout per service: start in `MODE=observe` with `WAF_MODE=detect`, read a - few days of logs and alerts, set limits and rule exclusions from what real - traffic looks like, then switch to `enforce` and `block`. -- Notes specific to gitea: clones and pushes over HTTP are large and long, so - byte limits must be sized for a legitimate clone, the proxy sets no overall - response timeout, and `WAF_BODY_LIMIT` keeps pack uploads out of memory. - Archive download and blame or history pages are what scrapers hammer; request - limits do most of the work there. +- An app deployed by upaas takes the same shape, without the app's part of this + file. upaas deploys the app with no traefik labels; the sidecar runs beside it + from its own docker-compose file, carries the traefik labels, and points + `UPSTREAM_URL` at the app's hostname. The sidecar must be able to reach that + hostname, for example over a docker network the two share. +- The application container leaves the traefik network, so the sidecar cannot be + bypassed. +- Traefik routes only port 8080. The metrics port is reached by the scraper over + a docker network it shares with the sidecar; the admin port stays on loopback + inside the container and is used through `docker exec`. +- The state volume holds the JSON state files, a few MiB for a small service; it + needs no backup beyond whatever the host already does. +- SSH access to gitea does not pass through the sidecar and is not protected by + it. +- Rollout per service: point traefik at the sidecar with only `UPSTREAM_URL` + set. It protects the app from the first request, and nothing needs tuning + first. Afterwards the request log and the ban notes say why each client was + refused. A real visitor refused by mistake is let through with an exclusion + (`WAF_DISABLED_RULES`, `WAF_EXEMPT_PATHS`, `RATE_LIMIT_EXEMPT_PATHS`) or an + exemption (`RATE_LIMIT_EXEMPT_NETS`, `ALLOW_NETS`), and its ban is lifted in + `bans.json`. Additions to consider at any time: an alert destination, a lookup + database with biased thresholds, reputation sources, a remote log endpoint, + and an app-specific rule file. +- Notes specific to gitea: + - A clone is a response, and fits the defaults of 30 minutes and 5 GB for + all but the largest repositories and slowest links. A push is a request: + at the defaults, one larger than 100 MB or taking more than 60 seconds to + upload is cut off, so a gitea that takes large pushes needs + `CLIENT_REQUEST_MAX_BYTES`, `UPSTREAM_REQUEST_MAX_BYTES`, + `CLIENT_REQUEST_TIMEOUT` and `UPSTREAM_REQUEST_TIMEOUT` raised to fit. + `WAF_BODY_LIMIT` keeps pack uploads out of memory. + - The default `WAF_DISABLED_RULES` lets git's clone and push over HTTP, and + the display of source files, through the Core Rule Set. Pages where people + post code (issues, pull requests, the web editor) can still trip other + rules; such a match refuses only that request, and the request log names + the rule in `waf_rule_ids` for `WAF_DISABLED_RULES`. + - Archive download and blame or history pages are what scrapers hammer; + request limits do most of the work there. ## Failure behaviour -- Lookup database missing: clients are unknown, service continues, one alert. +- Lookup database: a configured file that is missing or unreadable at start + stops the start. A replacement that cannot be read while running is ignored: + the file already loaded stays in use, the problem is logged, one alert. - Reputation source down or over quota: no verdict, service continues, one `source_failure` alert per cooldown. - Alert destination down: retried with backoff from a bounded queue, oldest dropped first, drops counted in metrics. - Upstream down: 502 from the sidecar, not counted as client offences. -- Attack detection engine error on a request: request is forwarded, error - logged and counted. +- Attack detection engine error on a request: request is forwarded, error logged + and counted. - Remote log endpoint down: stdout continues, lines are buffered then dropped oldest first, drops counted in metrics. - State file write fails while running: memory stays authoritative, logged, counted, alerted, retried. -- In short: the only things that stop the process are invalid configuration, - a rule file that does not parse, an unwritable `STATE_DIR`, and an unparseable `bans.json` or `offences.json` - at start. A broken helper never takes the protected service down. +- A state file or rule file edited while running that does not parse: the + sidecar keeps running on what it has. The state file is set aside as + `.bad` and written again from memory; the rule file is left as it is. + Logged, one alert. +- In short: the only things that stop the process happen at start: invalid + configuration, a configured lookup database that is missing or unreadable, a + rule file or state file that does not parse, and an unwritable `STATE_DIR`. + Once running, a broken helper or a broken edit never takes the protected + service down. ## Risks the design has to handle -- Forged `X-Forwarded-For`: handled by `TRUSTED_PROXIES` being required and by - walking the header from the right. -- Many clients behind one address (mobile carriers, offices, Tor): rate limits - punish them collectively; `RATE_LIMIT_EXEMPT_NETS` and sensible day limits - are the remedy, and bans on such addresses should stay short. +- Forged `X-Forwarded-For`: handled by believing the header only from a peer + inside `TRUSTED_PROXIES` and by walking it from the right. The default trusts + every private address, which in the intended deployment means traefik; where + other containers can reach the sidecar directly, setting `TRUSTED_PROXIES` to + traefik's own network closes that gap. +- Many clients behind one address (mobile carriers, offices, Tor): they share + its limits and its bans. The default limits sit several times above what one + busy person produces, `RATE_LIMIT_EXEMPT_NETS` and `ALLOW_NETS` take known + shared addresses out, and the ban notes show an admin what happened before a + ban is lifted. +- A `ban` rule that matches a real visitor bans for seven days, so the default + rule file keeps to requests no real visitor sends. - IPv6 address rotation inside a /64: handled by grouping. - Widely distributed scrapers using thousands of addresses at low rates each: - per-client limits do not see them. The AS number and netblock anomaly - alerts reveal them, and `ASN_LIMIT_PERCENT` at a low value or `0` is the - response. A per-AS-number or per-country total limit is open. + per-client limits do not see them. The AS number and netblock anomaly alerts + reveal them once their thresholds are set, and a low `ASN_LIMIT_PERCENT` for + that AS number is the response: it lowers the limits of each of its clients. + There is no shared budget for a whole AS number or country, which one abuser + could use up and so lock out everyone else there. - Core Rule Set false positives against real apps (gitea's editor, API - payloads): `detect` by default, exclusions by rule id and path. -- Slow-request attacks: header read timeout and idle timeout on the listener. + payloads): a match refuses only that request and bans no one by itself; the + request log names the rule, and exclusions by rule id and path fix it. +- Slow-request attacks: `CLIENT_REQUEST_TIMEOUT` bounds how long a request may + take to arrive, and an idle timeout on the listener closes connections that + send nothing. - Alert floods: cooldown and hourly cap. ## Build order -- First: proxy, client identification, static lists, three-window request - limits, exemptions, `observe` mode, the full request log on stdout, the - metrics endpoint, health. -- Second: rule files, ban ledger with escalation and permanent bans, the JSON - state files, - admin endpoints, alerting to all three destinations, remote log sending. +- First: proxy with its size and time limits, client identification, static + lists, three-window request limits and the bans they lead to, the ban ledger + and the JSON state files with edits taken in while running, exemptions, + `observe` mode, the full request log on stdout, the metrics endpoint, health. +- Second: rule files, admin endpoints, alerting to all three destinations, + remote log sending. - Third: AS number and country lookup, biased thresholds, byte limits, anomaly thresholds. - Fourth: blocklists, DNSBL, AbuseIPDB, optional CrowdSec decision feed. @@ -653,43 +907,3 @@ services: bursts. - Each stage is usable on its own; the first three already cover the traffic problem the fleet has today. - -## Questions for the owner - -- Combining percentages. A client in a listed AS number (50 percent) and a - listed country (25 percent): should it get the lowest (25 percent of the - normal limit) or the product (12.5 percent)? Lowest is easier to predict when - reading a config; product punishes overlap harder. Recommendation: lowest. -- Meaning of "certain countries getting a percentage rate or byte limit". Read - here as: each client from that country gets that percentage of the normal - per-client limits. The other reading is a cap on the country as a whole, for - example all of one country together may use at most 20 percent of the - service's hourly byte budget. The second needs a configured total budget and - means one heavy client can lock out its compatriots. Recommendation: build - the per-client reading first; add whole-country and whole-AS-number caps only - if distributed scrapers make it necessary. -- Sharing bans across the fleet. As drafted, each sidecar remembers only its - own bans, so an abuser banned at gitea starts fresh at the next service. - Options: keep it per sidecar; or run one CrowdSec engine per host and have - every sidecar read its decision list (already supported above) and also - report its own bans to it. Recommendation: per sidecar for the first version, - decide on CrowdSec after seeing real ban volumes. -- Lookup database source. IPinfo Lite (one file with country and AS number, - free account, attribution required), DB-IP Lite (no account, attribution - required), or MaxMind GeoLite2 (free account, licence terms on - redistribution). Recommendation: IPinfo Lite, fetched once centrally and - served to the fleet from an internal URL so sidecars hold no account token. -- upaas. Whether upaas can deploy a second container beside an app and move - the traefik labels onto it was not examined. If it cannot, that is a feature - request for upaas, not something this sidecar can solve. -- Remote log library. Sending syslog in the current format (RFC 5424) is not - covered by Go's standard library, whose `log/syslog` package is frozen and - writes the older format, and no widely used Go RELP client was found. The - choices are a small third-party package for each, or writing both inside this - project; the syslog record is a few dozen lines, a RELP client with - acknowledgement handling is a few hundred. Recommendation: write the syslog - sender in-project, ship it first, and decide on RELP after looking at the - available packages; say if RELP must be in the first release. -- Ban response. `403` tells the abuser they were noticed; `close` wastes less - and tells them nothing. Recommendation: `403` by default for debuggability, - `close` available. -- 2.54.0 From 8fc7539f1a0aeb5586a0d67288462c272741a716 Mon Sep 17 00:00:00 2001 From: sneak Date: Wed, 23 Sep 2026 13:37:34 +0000 Subject: [PATCH 2/3] SPEC and README: review fixes, country lists, GeoJS lookups (closes #6) Address the review of the spec update and fold in issues 8 and 11. Default ban rules are anchored at the site root. `bans.json` is bounded by `MAX_BANS` and rewritten only when a ban is made, lifted or made permanent. `clients.json` keeps one client per line, holds at most 20000 clients and is written every 15 minutes. Anomaly counters and alert state go to a new `alerts.json`, the AbuseIPDB count to `reputation.json`. 401 no longer counts toward the error burst. New settings: `DENIED_COUNTRIES`, `EXCLUSIVELY_ALLOWED_COUNTRIES`, `LOOKUP_SOURCE` (the IPinfo file or GeoJS), `LOOKUP_TIMEOUT`, `CLIENT_REQUEST_HEADER_MAX_BYTES` and `CLIENT_IDLE_TIMEOUT`. Model: opus-5-5 --- README.md | 64 +++++---- SPEC.md | 382 +++++++++++++++++++++++++++++++++++++----------------- 2 files changed, 304 insertions(+), 142 deletions(-) diff --git a/README.md b/README.md index 38bd9b9..973e8c6 100644 --- a/README.md +++ b/README.md @@ -52,7 +52,7 @@ goes through the candidates one by one. other setting has a default chosen for a service facing the internet in 2026. - Real client address worked out from `X-Forwarded-For`, trusting only the proxy networks you list, by default the private address ranges. IPv6 clients are - counted by /64. + counted by /64 by default. - Size and time limits on requests and responses, both between the client and `smallwebwaf` and between `smallwebwaf` and the app: by default a request may take 60 seconds and 100 MB, a response 30 minutes and 5 GB. @@ -61,8 +61,14 @@ goes through the candidates one by one. real visitors need. - Netblocks that bypass rate limiting, netblocks that bypass everything, and netblocks that are always refused. -- AS number and country lookup for every client from the IPinfo Lite database - file, which you download and mount (see "Lookup database" below). +- AS number and country lookup for every client, off until you choose a source: + the IPinfo Lite database file, which you download and mount, or the free GeoJS + web service (see "Country and AS number lookup" below). +- Country lists: `DENIED_COUNTRIES` refuses every request from the countries + listed, `EXCLUSIVELY_ALLOWED_COUNTRIES` every request from anywhere else. Such + a request gets the answer a banned client gets as soon as the client's address + has been looked up, before its body is read and without the rule files or the + Core Rule Set looking at it, and no ban is made. - Biased limits: listed AS numbers and countries get a percentage of every limit, for example 50 percent for common abuse-source networks, so their clients are banned after fewer requests. Zero percent is a zero allowance: the @@ -96,11 +102,11 @@ goes through the candidates one by one. fields, the decision taken and why, AS number and country, and timings. Optionally also sent to a remote syslog server. - Prometheus metrics on their own port. -- State (bans with their notes, each client's counters and history, the - reputation cache) held in memory and kept in readable JSON files that always - hold a copy of it, so a restart loses nothing. Edit a file, or add a rule - file, and the running `smallwebwaf` picks up the change. Nothing is read from - disk while serving a request. +- State (bans with their notes, each client's counters, history and lookup + answer, the reputation cache, the alerting state) held in memory and kept in + readable JSON files, written regularly and at every stop, so a restart loses + nothing. Edit a file, or add a rule file, and the running `smallwebwaf` picks + up the change. Nothing is read from disk while serving a request. - A small admin endpoint for health checks, listing, adding and lifting bans, and asking why a given address was refused. @@ -115,7 +121,9 @@ For each request `smallwebwaf`: - works out who the client really is; - lets it straight through if it is on the bypass list, refuses it if it is on the deny list or currently banned; -- looks up its AS number and country, and any cached reputation verdict; +- looks up its AS number and country, and refuses it if that country is denied, + or is not among the only ones allowed; +- checks for a cached reputation verdict; - picks the client's limit percentage from those; - checks the minute, hour and day request counters against the limits, and bans the client if it breaks one; @@ -162,7 +170,7 @@ A rule file is one rule per line: a name, what to match against, what to do, and a regex. ``` -env-file path ban (?i)/\.env(\.[a-z]+)?$ +env-file path ban (?i)^/\.env(\.[a-z]+)?$ scanner-agent user_agent ban (?i)\b(sqlmap|nikto|nuclei|wpscan)\b ``` @@ -170,19 +178,31 @@ scanner-agent user_agent ban (?i)\b(sqlmap|nikto|nuclei|wpscan)\b rules, the rule file format, the state files, the log fields, the metrics, failure behaviour and the build order. -## Lookup database +## Country and AS number lookup -AS number and country lookups, and the biased limits that use them, need the -free IPinfo Lite database (`ipinfo_lite.mmdb`). You download it with your own -IPinfo account, mount it into the container, point `LOOKUP_DB_PATH` at it and -refresh it when you choose; `smallwebwaf` never downloads it itself. IPinfo -releases it under the Creative Commons Attribution-ShareAlike 4.0 International -License and asks for attribution, in its own words on https://ipinfo.io/lite: -"The attribution requirements can be met by giving our service credit as your -data source. Simply place a link to IPinfo on the website, application, or -social media account that uses our data." Its example of such a credit is a link -mentioning "IP address data is powered by IPinfo". A service that uses the -database through `smallwebwaf` should carry that link. +AS number and country lookups, and the country lists and biased limits that use +them, are off until you choose one of two sources with `LOOKUP_SOURCE`. + +`LOOKUP_SOURCE=file` reads the free IPinfo Lite database (`ipinfo_lite.mmdb`). +You download it with your own IPinfo account, mount the directory that holds it +into the container, point `LOOKUP_DB_PATH` at the file and refresh it when you +choose; `smallwebwaf` never downloads it itself, and reads it again when you +replace it. It has to be the directory rather than the file itself: docker does +not show a single mounted file being replaced, so a refresh would go unseen. +IPinfo releases it under the Creative Commons Attribution-ShareAlike 4.0 +International License and asks for attribution, in its own words on +https://ipinfo.io/lite: "The attribution requirements can be met by giving our +service credit as your data source. Simply place a link to IPinfo on the +website, application, or social media account that uses our data." Its example +of such a credit is a link mentioning "IP address data is powered by IPinfo". A +service that uses the database through `smallwebwaf` should carry that link. + +`LOOKUP_SOURCE=geojs` asks the free GeoJS web service instead, with no account +and no file. Every new visitor's address is sent to GeoJS. Each answer is kept +for seven days, across restarts, and many addresses are asked about in one +request. GeoJS publishes no rate limit but may block a caller it thinks asks too +much; while it is not answering, new visitors count as coming from an unknown +country, which `EXCLUSIVELY_ALLOWED_COUNTRIES` refuses. ## Documents diff --git a/SPEC.md b/SPEC.md index 492aca0..2eb79c8 100644 --- a/SPEC.md +++ b/SPEC.md @@ -12,8 +12,8 @@ forwards to the application. The sidecar limits request and byte rates per client, bounds the size and duration of every request and response, detects attacks, bans abusers (briefly at first, for seven days on a clear sign of attack, permanently when they keep at it), consults IP reputation sources, looks -up the AS number and country of each client, lowers its limits for listed AS -numbers and countries, and sends alerts. +up the AS number and country of each client, can refuse whole countries, lowers +its limits for listed AS numbers and countries, and sends alerts. It is meant to protect an app on the open internet from the first request with one setting, `UPSTREAM_URL`: every other setting has a default chosen for a @@ -52,8 +52,9 @@ directory of hand-editable text files. - Client identification: works out the real client IP from `X-Forwarded-For`, trusting only the proxy netblocks in `TRUSTED_PROXIES`, by default the private address ranges. - - Lookup: AS number and country from one database file the operator - supplies, held in memory. + - Lookup: AS number and country, from one of two sources the operator + chooses: a database file the operator supplies, held in memory, or the + GeoJS web service, whose answers are kept for seven days. - Reputation: static blocklists fetched on a schedule; DNSBL and reputation API queries made in the background and cached. - Counters: per-client request and byte counts per minute, hour and day, @@ -104,7 +105,7 @@ whole exchange. Otherwise use the TCP peer address and ignore the header. - IPv6 clients are grouped by prefix (`IPV6_GROUP_PREFIX`, default 64) for counting and banning, because one abuser usually controls a whole /64. A - client is therefore one IPv4 address or one IPv6 /64. + client is therefore one IPv4 address or one IPv6 group, a /64 by default. - Static lists. - In `ALLOW_NETS`: skip every check below and forward. Still counted for anomaly alerts. @@ -112,8 +113,17 @@ whole exchange. - Ban ledger. An active ban on the client's netblock: refuse with `BAN_RESPONSE`. If a clear sign of attack caused the ban, the ban becomes permanent (see "Bans"). -- Lookup AS number and country, when a lookup database is configured. Unknown is - a valid result and changes nothing. +- Look up the AS number and country, unless `LOOKUP_SOURCE` is `off`. With + GeoJS, a client whose answer is not yet kept waits for it, up to + `LOOKUP_TIMEOUT`. A client the lookup cannot place has an unknown country and + AS number: the exclusive country list refuses it, the biased thresholds give + it `UNKNOWN_LIMIT_PERCENT`, and nothing else treats it differently. +- Country lists. A client whose country is in `DENIED_COUNTRIES`, or, when + `EXCLUSIVELY_ALLOWED_COUNTRIES` is set, is not in it, is refused with + `BAN_RESPONSE`. Every step up to here needs only the client's address, so the + request body has not been read yet, and the refusal skips everything below, + the rule files and the Core Rule Set included. It is counted in the metrics, + but it is not an offence and makes no ban. - Reputation. - Address inside a fetched blocklist: apply `BLOCKLIST_ACTION`. - Cached DNSBL or reputation API result: apply `REPUTATION_ACTION`. @@ -139,10 +149,10 @@ whole exchange. - Add response bytes (and request body bytes) to the byte counters. A client whose byte total passes a byte limit times the percentage has broken that limit and is banned. The response in progress is not cut off. - - Count the client's requests answered with 401, 403 or 404, whether the app - gave that answer or the sidecar refused the request after a rule file or - Core Rule Set match. More than `ERROR_BURST_THRESHOLD` of them within a - minute breaks a limit and bans the client. + - Count the client's requests answered with 403 or 404, whether the app gave + that answer or the sidecar refused the request after a rule file or Core + Rule Set match. More than `ERROR_BURST_THRESHOLD` of them within a minute + breaks a limit and bans the client. - Update the client's history and the anomaly counters, and evaluate alert thresholds. @@ -157,17 +167,25 @@ whole exchange. shutdown and loaded again at start (see "Persistent state"), so a restart does not hand every client a fresh allowance. Buckets whose time has passed are discarded on load. -- Memory is bounded by `MAX_TRACKED_CLIENTS`; when full, the least recently seen - client is dropped first, with its history and its past bans. Active bans are - never dropped; one whose client is no longer tracked is dropped when it ends. +- Memory is bounded by `MAX_TRACKED_CLIENTS` and `MAX_BANS`. When the table of + clients is full, the least recently seen client is dropped first, with its + history and its lookup answer. When `MAX_BANS` bans are held, the ban that has + gone longest without a request from its netblock is dropped first, whether it + is past, active or permanent (see "Bans"). ## Bans A ban refuses every request from a netblock, answering with `BAN_RESPONSE` -(`403` by default), until the ban ends; a permanent ban never ends. The netblock -a ban covers is the client's IPv4 address (or the wider IPv4 prefix that -`BAN_SCOPE_V4_PREFIX` sets) or its IPv6 /64, and never more, so a permanent ban -does not reach neighbours who did nothing. +(`403` by default), until the ban ends. A permanent ban does not run out: it +ends only when an admin lifts it, or when a new ban is made while `MAX_BANS` +bans are held and it is the ban that has gone longest without a request. A +scanner whose ban was dropped that way is refused and banned again by its next +probe. + +The netblock a ban covers is the client: its IPv4 address, or its IPv6 group of +`IPV6_GROUP_PREFIX` (a /64 by default). `BAN_SCOPE_V4_PREFIX` can widen an IPv4 +ban to the surrounding netblock. At the defaults a ban covers one address or one +/64, so a permanent ban does not reach neighbours who did nothing. An offence is a request the sidecar holds against the client: one that carries a clear sign of attack, breaks a limit, or is refused by a rule file or the Core @@ -206,21 +224,30 @@ Every ban carries notes with what an admin needs to decide whether to lift it lifted before it ends is kept, marked as lifted, and does not count toward a longer ban; deleting its entry from `bans.json` forgets it entirely. -Clients of listed AS numbers and countries, and clients with a poor reputation, -get lower limits (see "Biased thresholds"), so the same rules ban them after -fewer requests. The result is always a ban, first a temporary one and then, for -repeated abuse, a permanent one. +Clients of the AS numbers and countries listed in the biased thresholds, and +clients listed by a reputation source whose action is `limit:`, get +lower limits (see "Biased thresholds"), so the same rules ban them after fewer +requests. For them the result is always a ban, first a temporary one and then, +for repeated abuse, a permanent one. + +Some refusals make no ban at all. `DENY_NETS`, a blocklist or reputation hit +whose action is `deny`, and the country lists (`DENIED_COUNTRIES`, +`EXCLUSIVELY_ALLOWED_COUNTRIES`) refuse each request they cover and record +nothing in `bans.json`. ## Configuration surface Conventions: lists are comma separated; netblocks are CIDR (a bare address means /32 or /128); durations use Go syntax plus `d` for days (`90s`, `15m`, `24h`, -`7d`); byte sizes accept `K`, `M`, `G` suffixes. +`7d`); byte sizes accept `K`, `M`, `G` suffixes. Countries are two-letter ISO +codes in either case (`de` and `DE` are the same); a code that is not a country +code, such as `nk` (North Korea is `kp`), stops the start with a message naming +it. - Only `UPSTREAM_URL` is required. Every other setting has a default chosen for - a service facing the internet in 2026, or stays off until it is given - something only the operator can supply: an alert destination, an account key, - the lookup database. + a service facing the internet in 2026, or stays off until the operator + supplies or chooses what it needs: an alert destination, an account key, a + lookup source. - Any limit or threshold can be switched off with the value `off`. - A list set to an empty value is an empty list, and replaces the default. - Every variable may instead be given as `NAME_FILE` pointing at a file holding @@ -254,17 +281,21 @@ The settings, by group: address. A list given replaces the default; set but empty, it trusts nothing. - `IPV6_GROUP_PREFIX` (default `64`). - - `MAX_TRACKED_CLIENTS` (default `50000`): clients held in memory and in + - `MAX_TRACKED_CLIENTS` (default `20000`): clients held in memory and in `clients.json` (see "Persistent state"). + - `MAX_BANS` (default `5000`): bans held in memory and in `bans.json`, past, + active and permanent (see "Bans"). - Persistent state - `STATE_DIR` (default `/data`): the JSON state files. The image declares `/data` as a volume, so it is writable even when nothing is mounted; a named volume, as in the examples below, keeps the state in a place that outlives the container. - - `STATE_WRITE_DELAY` (default `2s`): after a ban changes, wait this long - before writing `bans.json`, so a burst of changes becomes one write. - - `STATE_COUNTER_INTERVAL` (default `60s`): how often `clients.json` and - `reputation.json` are written. + - `STATE_WRITE_DELAY` (default `10s`): after a ban is made, lifted or made + permanent, `bans.json` is written this long later with every change made + in between, so a burst of changes becomes one write. + - `STATE_COUNTER_INTERVAL` (default `15m`): how often `clients.json`, + `reputation.json` and `alerts.json` are written, and `bans.json` when only + the counts in its notes changed. - Logging - The request log on stdout is always on and has no switch. - `LOG_LEVEL` (default `info`): for the process's own messages (start-up, @@ -319,6 +350,14 @@ The settings, by group: send its whole request, headers and body. - `CLIENT_REQUEST_MAX_BYTES` (default `100M`): the largest request body a client may send. + - `CLIENT_REQUEST_HEADER_MAX_BYTES` (default `32K`): the largest request + line and headers a client may send. Over it, the sidecar answers `431` and + closes the connection, and nothing reaches the app. + - `CLIENT_IDLE_TIMEOUT` (default `120s`): how long a kept-open connection + may wait for its next request before the sidecar closes it. It is longer + than the 90 seconds after which traefik, by default, closes a connection + it is not using, so traefik closes first and never sends a request on a + connection the sidecar is closing. - `CLIENT_RESPONSE_TIMEOUT` (default `30m`): how long the sidecar may take to deliver one response to the client, from the end of the request to the last byte. @@ -340,27 +379,72 @@ The settings, by group: started it can only be cut off, and the connection is closed. - A WebSocket connection leaves these limits behind once it is upgraded: it stays open until either side closes it. -- Lookup of AS number and country (R7) - - `LOOKUP_DB_PATH`: the IPinfo Lite database in its `.mmdb` form - (`ipinfo_lite.mmdb`), one file that carries both country and AS number; - the sidecar reads its `asn`, `as_name` and `country_code` fields. The - operator downloads it with a free IPinfo account and mounts it read-only. - The sidecar never fetches it and holds no account or token for it. +- Lookup of AS number and country (R7). Off by default: the file needs an + account only the operator can hold, and GeoJS is told every visitor's address. + - `LOOKUP_SOURCE` (default `off`): `off`, `file` or `geojs`. + - `LOOKUP_DB_PATH`: the file for `file`, the IPinfo Lite database in its + `.mmdb` form (`ipinfo_lite.mmdb`), one file that carries both country and + AS number; the sidecar reads its `asn`, `as_name` and `country_code` + fields. The operator downloads it with a free IPinfo account and mounts it + read-only. The sidecar never fetches it and holds no account or token for + it. A file that is missing or unreadable at start stops the start. - Refreshing the file is the operator's business; IPinfo updates it daily. The sidecar notices when the file is replaced and reads it again; a - replacement it cannot read is ignored, logged and alerted once, and the - file it has stays in use. Mount the directory that holds the file rather - than the file itself: docker does not show a single mounted file being - replaced on the host. - - Unset by default, which means no lookups. A setting that needs them (the - biased thresholds below, `ADD_LOOKUP_HEADERS`, the per-AS-number anomaly - thresholds) given without `LOOKUP_DB_PATH` stops the start with a message - naming both. A configured file that is missing or unreadable at start - stops the start too. + replacement it cannot read is ignored, logged and sent as one `file_error` + alert, and the file it has stays in use. Mount the directory that holds + the file rather than the file itself: docker does not show a single + mounted file being replaced on the host. + - `geojs` asks the free GeoJS web service, which needs no account or key, + about several addresses in one request + (`https://get.geojs.io/v1/ip/geo.json?ip=a,b,c`), and reads each answer's + `country_code`, `asn` and `organization_name` (the AS name). A + `country_code` of `null`, which GeoJS gives for some ranges, and an `asn` + of `64512`, which it gives when it knows none, count as unknown. GeoJS is + told the address of every new visitor. + - Each GeoJS answer is kept for 7 days with the client's record, in memory + and in `clients.json`, so it survives a restart; after 7 days the client's + next request asks again. A client dropped from the table loses its answer. + - `LOOKUP_TIMEOUT` (default `1s`): how long a client with no kept answer + waits for one; GeoJS normally answers in a fraction of that. At most one + request to GeoJS is under way at a time, the addresses that arrive + meanwhile are asked about together in the next one, and a request that + takes longer than `LOOKUP_TIMEOUT` is abandoned. A client whose answer has + not come in time counts as unknown until it comes: its later requests do + not wait, and its address is asked about again in the background. + - GeoJS publishes no rate limit, but its terms forbid "an excessive amount + of API requests", judged by GeoJS alone, and let it block a caller. While + GeoJS is slow, down or refusing the sidecar, clients with a kept answer + are unaffected and new clients count as unknown, so + `EXCLUSIVELY_ALLOWED_COUNTRIES`, when set, refuses them. The sidecar keeps + asking with backoff and sends one `source_failure` alert per cooldown. A + service that cannot accept this uses the file. + - One source at a time: `LOOKUP_SOURCE=file` without `LOOKUP_DB_PATH`, or + `LOOKUP_DB_PATH` with any other `LOOKUP_SOURCE`, stops the start with a + message naming both. So does a setting that needs lookups while + `LOOKUP_SOURCE` is `off`: the country lists and the biased thresholds + below, `ADD_LOOKUP_HEADERS`, and the per-AS-number anomaly thresholds. - `ADD_LOOKUP_HEADERS` (default `false`): pass `X-Client-ASN` and `X-Client-Country` to the app. -- Biased thresholds (R8). The lists are empty by default: they need the lookup - database, which only the operator can supply. +- Country lists. Both are empty by default, since they need a lookup source. + Clients in `ALLOW_NETS` are not checked. + - `DENIED_COUNTRIES`: for example `cn,ru,kp,ir,ua,by`. Every request from a + listed country is refused. + - `EXCLUSIVELY_ALLOWED_COUNTRIES`: for example `us,de`. Every request from + any other country is refused, and so is every request from a client the + lookup cannot place, such as an address missing from the database or one + GeoJS has not answered for in time. A list that let those through would + let every new client in whenever GeoJS stops answering. + - Both may be set. `DENIED_COUNTRIES` then adds nothing, since the exclusive + list already refuses every other country, and a code on both lists stops + the start with a message naming it. + - A refused request is answered with `BAN_RESPONSE`, as a banned client is, + as soon as the client's address has been looked up, before its body is + read. It skips the reputation checks, the limits, the rule files and the + Core Rule Set, is logged with the action `country_denied`, and is counted + in the metrics. It is a refusal, not a ban: it is not an offence and makes + no ban record. +- Biased thresholds (R8). The lists are empty by default: they need a lookup + source, which only the operator can choose. - `ASN_LIMIT_PERCENT`: for example `AS14061:50,AS16276:50,AS45102:25`. Clients in a listed AS number get that percentage of every request and byte limit, so the rules in "Bans" ban them after fewer requests than @@ -370,8 +454,9 @@ The settings, by group: `CN:25,RU:50`. Each client from a listed country gets that percentage of the normal per-client limits. - No budget is shared by a whole country or AS number: one abuser could use - it up and lock out everyone else there, the denial of service this tool - exists to prevent. + it up and lock out everyone else there, a denial of service nobody chose + and the one this tool exists to prevent. Refusing a whole country is left + to the operator's choice, through the country lists. - When more than one percentage applies to a client (its AS number, its country, a reputation hit), the lowest applies. - `ASN_BYTES_PERCENT`, `COUNTRY_BYTES_PERCENT`: optional overrides applied @@ -406,10 +491,13 @@ The settings, by group: example `/wp-login.php,/xmlrpc.php` in front of gitea. A request for one is a clear sign of attack. This is the env-var short form of a `path` rule with the `ban` action, for deployments that mount no rule files. - - `ERROR_BURST_THRESHOLD` (default `30`): responses with status 401, 403 or - 404 per client per minute; more than this breaks a limit (see "Bans"). + - `ERROR_BURST_THRESHOLD` (default `30`): responses with status 403 or 404 + per client per minute; more than this breaks a limit (see "Bans"). Scanners walking lists of paths cause hundreds a minute; a person rarely - causes more than a few. + causes more than a few. 401 is not counted: git and container registry + clients send their first request without credentials and are answered 401 + at the start of every push, every fetch from a private repository and + every image pull. - Bans (R5), following the rules under "Bans" - `ATTACK_BAN_DURATION` (default `7d`): the ban for a first clear sign of attack. @@ -455,7 +543,10 @@ The settings, by group: - `CROWDSEC_LAPI_URL`, `CROWDSEC_LAPI_KEY`: optional. If a CrowdSec engine exists on the host, pull its decision list on a schedule and treat listed addresses as banned. This is how a fleet-wide blocklist can arrive without - the sidecar depending on CrowdSec. + the sidecar depending on CrowdSec. The list is kept like a fetched + blocklist, and a listed client's request makes a ban with the cause + `crowdsec` that lasts as long as CrowdSec's decision, so a long list does + not fill `bans.json`. - `REPUTATION_ACTION` (default `limit:25`): `deny`, `limit:`, or `log`, for DNSBL and API hits. Such verdicts are less certain than a blocklist, so by default a listed client gets a quarter of every limit and @@ -469,8 +560,11 @@ The settings, by group: - `ALERT_NTFY_URL` (full topic URL), `ALERT_NTFY_TOKEN`: title, priority and tags set from the event type. - `ALERT_EVENTS` (default - `ban,permanent_ban,waf_block,anomaly,reputation_hit,source_failure`): - which event types are sent. + `ban,permanent_ban,waf_block,anomaly,reputation_hit,source_failure,file_error`): + which event types are sent. `source_failure` is a reputation source or + GeoJS failing or refusing the sidecar. `file_error` is a rule file or + state file edited while running that does not parse, a replacement lookup + database that cannot be read, or a state file that cannot be written. - `ALERT_COOLDOWN` (default `15m`): the same event type for the same client or netblock is not repeated within this time; a count of suppressed repeats is included in the next one. @@ -486,7 +580,7 @@ The settings, by group: - Per surrounding netblock (`ANOMALY_NET_V4_PREFIX` default `24`, `ANOMALY_NET_V6_PREFIX` default `48`): `ANOMALY_NET_REQUESTS_PER_MINUTE`, `..._PER_HOUR`, `ANOMALY_NET_BYTES_PER_MINUTE`, `..._PER_HOUR`. - - Per AS number, which needs the lookup database: + - Per AS number, which needs a lookup source: `ANOMALY_ASN_REQUESTS_PER_MINUTE`, `..._PER_HOUR`, `ANOMALY_ASN_BYTES_PER_MINUTE`, `..._PER_HOUR`. - Whole service: `ANOMALY_TOTAL_REQUESTS_PER_MINUTE`, `..._PER_HOUR`, @@ -533,7 +627,9 @@ removed there takes effect while the sidecar runs. - `ban`: the request is a clear sign of attack. Refuse it and ban the client's netblock for seven days (`ATTACK_BAN_DURATION`); any further request during those days, or a later clear sign of attack, makes the ban - permanent (see "Bans"). Use it only for requests no real visitor sends. + permanent (see "Bans"). Use it only for requests no real visitor sends, + and anchor a path at the site root (`^/`): a file of the same name deeper + in a site can be ordinary content, such as a file in a gitea repository. - `regex`: Go regular expression syntax (RE2). It has no backreferences or lookaround, and in exchange matching time is linear in the input, so no rule can be made to stall the proxy. `(?i)` at the front makes a rule @@ -545,9 +641,9 @@ removed there takes effect while the sidecar runs. line that does not parse, a regex that does not compile, or a duplicate id stops the process with a message naming the file and line. While running, the same faults leave the rules as they were, including the earlier version of - that file, and the log and one alert name the file and line; once the file is - fixed, it is read again. A missing or empty directory is not an error: the log - says that no rules were loaded. + that file, and the log and one `file_error` alert name the file and line; once + the file is fixed, it is read again. A missing or empty directory is not an + error: the log says that no rules were loaded. - The image ships a default file in `RULES_DIR`. Mounting a directory over it replaces the defaults; mounting single files into it adds to them. Docker does not show a single mounted file being replaced on the host, which is how many @@ -564,16 +660,19 @@ Example file: # 00-default.rules: probes no real visitor sends # id target action regex -env-file path ban (?i)/\.env(\.[a-z]+)?$ +env-file path ban (?i)^/\.env(\.[a-z]+)?$ git-dir path ban ^/\.git/(config|HEAD|index)$ -php-shell path ban (?i)/(shell|c99|r57|wso|alfa)\.php$ +php-shell path ban (?i)^/(shell|c99|r57|wso|alfa)\.php$ scanner-agent user_agent ban (?i)\b(sqlmap|nikto|nuclei|masscan|zgrab|wpscan)\b path-traversal uri block (\.\./){2,} empty-agent user_agent log ^$ ``` The image ships one default file of this kind. It is kept short and limited to -patterns that are wrong for every app. Anything app-specific belongs in a file +patterns that are wrong for every app, and its path rules are anchored at the +site root: in front of gitea, `/.env` is a probe, while +`///src/branch/main/.env.example` is a file in a repository that +any visitor or search crawler may open. Anything app-specific belongs in a file the deployer mounts: a request for `/wp-login.php`, for example, is a clear sign of attack in front of gitea and an ordinary login in front of WordPress. @@ -585,57 +684,79 @@ wp-probe path ban (?i)^/(wp-login\.php|xmlrpc\.php|wp-admin/) ## Persistent state All state lives in memory, and the files in `STATE_DIR` hold a copy of all of -it, so an orderly stop loses nothing. No database is used, and nothing is read -from disk while serving a request. The files are meant for people as well: an -admin can read or edit them at any time, and the running sidecar takes the edit -in. +it, so an orderly stop loses nothing. The one exception is log lines still +waiting to be sent to `LOG_REMOTE_URL`, which stdout has already carried. No +database is used, and nothing is read from disk while serving a request. The +files are meant for people as well: an admin can read or edit them at any time, +and the running sidecar takes the edit in. - Files, each holding one kind of state: - - `bans.json`: every active and past ban. Per entry: the netblock, start, - expiry (`null` for permanent), what caused it (`attack`, `limit`, `admin` - or `crowdsec`), a short reason, when it was lifted if an admin lifted it, - and a `notes` field (below). + - `bans.json`: active and past bans, up to `MAX_BANS`. Per entry: the + netblock, start, expiry (`null` for permanent), what caused it (`attack`, + `limit`, `admin` or `crowdsec`), a short reason, when it was lifted if an + admin lifted it, and a `notes` field (below). - `clients.json`: per client, its minute, hour and day counters and its history since it was first seen: first and last time seen, the AS number, - AS name and country last looked up, total requests and bytes in each - direction, how many requests were forwarded and how many refused, - responses by status class, and offences by kind. Clients that were never - banned are kept too, so every client's history survives a restart; - `GET /clients/` shows it. - - `reputation.json`: the last good copy of each list fetched from a URL, and + AS name and country last looked up and when (with GeoJS, the answer kept + for 7 days), total requests and bytes in each direction, how many requests + were forwarded and how many refused, responses by status class, and + offences by kind. Clients that were never banned are kept too, so every + client's history survives a restart; `GET /clients/` shows it. + - `reputation.json`: the last good copy of each list fetched from a URL, cached DNSBL and reputation API verdicts, each with the time it was - fetched. + fetched, and the AbuseIPDB checks spent today, so a restart does not reset + the daily budget. + - `alerts.json`: the anomaly counters for surrounding netblocks, AS numbers, + named netblocks and the whole service; for each event type and client or + netblock, when it was last sent and how many repeats the cooldown has held + back since; the alerts sent this hour; and alerts still waiting to be + sent. - Ban notes. The notes on a ban hold what an admin needs to decide whether to lift it, drawn from what the sidecar already knows; nothing is looked up to fill them: - - the AS number, AS name and country (empty without a lookup database); + - the AS number, AS name and country (empty when lookups are off); - what was broken: the rule ids and target that matched, or the limit, its window, the count reached and the client's limit percentage with what set it; and any reputation sources that listed the client; - the requests that caused the ban, up to the last ten: time, method, host, - path, status and user agent; + path with its query string, status and user agent, each text cut to 256 + bytes; - how many requests counted toward the ban, and the time span over which they came; - the netblock's total requests since it was first seen, and the requests refused under this ban so far, kept up to date while the ban lasts; - how many earlier bans of each kind the netblock has had. -- Size: each tracked client takes about 2 KiB in `clients.json`, so at the - default `MAX_TRACKED_CLIENTS` of 50,000 the file can reach about 100 MiB; a - small service usually tracks a few thousand clients and a few MiB. The other - files stay small. Writing a whole file of that size takes about a second, in - the background, so rewriting whole files still needs nothing cleverer. +- Size and disk writes. Each file is rewritten whole, so each has a bound: + - `clients.json` takes about 1 KiB per client. Clients are dropped only when + the table is full, so on a public service the file grows to the default + `MAX_TRACKED_CLIENTS` of 20,000, about 20 MiB. Written every 15 minutes, + that is under 2 GiB of disk writes a day. + - `bans.json` takes about 2 KiB per ban and at most about 8 KiB, since the + texts in the notes are cut short. At the default `MAX_BANS` of 5,000 it is + about 10 MiB, and never more than about 40 MiB. It is written when a ban + is made, lifted or made permanent, at most once every 10 seconds, and + otherwise with the 15-minute write, so its writes follow the bans made: + with a full file, a hundred new bans a day come to about 1 GiB of disk + writes. + - `reputation.json` and `alerts.json` are usually a few MiB or less. + - Writing a file of these sizes takes well under a second on an ordinary + disk, in the background, so rewriting whole files needs nothing cleverer. - Format: indented JSON with a top-level `version` number, entries sorted by client address, times in RFC 3339 UTC, durations and sizes as plain numbers with the unit in the field name. The aim is that a person can open `bans.json` - in an editor, find an address, and remove or add an entry. + in an editor, find an address, and remove or add an entry. `clients.json` puts + each client on one line instead, which halves its size and lets `grep` show + everything about one client. - Writing: - serialise from a snapshot taken under the lock, so requests are not held up while the file is written; - write to a temporary file in the same directory, sync it, rename it over the real name, sync the directory. A crash at any point leaves either the old complete file or the new complete file, never a partial one; - - `bans.json` is written `STATE_WRITE_DELAY` after a change; `clients.json` - and `reputation.json` every `STATE_COUNTER_INTERVAL`; all three on orderly + - `bans.json` is written `STATE_WRITE_DELAY` after a ban is made, lifted or + made permanent. Changes that only update the counts in its notes wait for + the write every `STATE_COUNTER_INTERVAL`, as all of `clients.json`, + `reputation.json` and `alerts.json` do. All four are written on orderly shutdown (SIGTERM); - before writing a file, the sidecar checks whether it changed on disk since the sidecar last read or wrote it; if it did, the sidecar takes that edit @@ -662,14 +783,17 @@ in. - An edit that does not parse does not stop the running sidecar. It keeps the state it has, renames the edited file to `.bad` (for example `bans.json.bad`) so the edit is kept, writes the file again from memory, - and logs and alerts once with the file and the position of the error. The - admin fixes the `.bad` file and moves it back. + logs the file and the position of the error, and sends them in one + `file_error` alert. The admin fixes the `.bad` file and moves it back. - `STATE_DIR` not writable at start: the process exits. A write that fails while running: state stays correct in memory, the failure is logged, counted in - metrics, alerted once per cooldown, and retried at the next write. -- What a hard kill can lose: ban changes from the last `STATE_WRITE_DELAY`, and - counts, history and reputation from the last `STATE_COUNTER_INTERVAL`. Losing - the volume loses ban history, not service. + metrics, sent as a `file_error` alert once per cooldown, and retried at the + next write. +- What a hard kill can lose: ban changes from the last `STATE_WRITE_DELAY` (10 + seconds), and everything else from the last `STATE_COUNTER_INTERVAL` (15 + minutes): counts and history, the counts in ban notes, GeoJS answers, + reputation verdicts and the AbuseIPDB count, anomaly counters and alert + cooldowns. Losing the volume loses ban history, not service. ## Request log @@ -691,13 +815,13 @@ handling while `docker logs` keeps working. - Response detail: `response_content_type`, `upstream_status` (differs from `status` when the sidecar answered itself), `cache_control`, `location` on redirects, `aborted` when the client went away early. -- Decision: `action` (`forward`, `banned`, `denied`, `rate_limited`, - `rule_blocked`, `waf_blocked`, `too_large`, `timed_out`, `upstream_error`), - `would_action` in `observe` mode, `limit_percent` and which rule set it, - `counts` (the client's minute, hour and day request and byte totals after this - request), `limit_hit` (which window), `rule_ids` (rule file rules that - matched), `waf_rule_ids`, `waf_score`, `reputation` (sources that listed the - client), `offence` when one was recorded, `ban_expires`. +- Decision: `action` (`forward`, `banned`, `denied`, `country_denied`, + `rate_limited`, `rule_blocked`, `waf_blocked`, `too_large`, `timed_out`, + `upstream_error`), `would_action` in `observe` mode, `limit_percent` and which + rule set it, `counts` (the client's minute, hour and day request and byte + totals after this request), `limit_hit` (which window), `rule_ids` (rule file + rules that matched), `waf_rule_ids`, `waf_score`, `reputation` (sources that + listed the client), `offence` when one was recorded, `ban_expires`. - Timings in milliseconds: `duration_total`, `duration_checks` (everything the sidecar did before forwarding), `duration_waf`, `duration_upstream_connect`, `duration_upstream_first_byte`, `duration_upstream_total`. @@ -724,15 +848,16 @@ it can be reached by a scraper without exposing ban management. duration and upstream duration histograms; requests in flight. - Limits and bans: limit hits by window and kind (requests or bytes), size and time limit hits by limit, offences by kind, bans created by cause, permanent - bans, active bans (gauge). + bans, active bans (gauge); requests refused by the country lists, by country. - Attack detection: rule file matches by rule id and action, and the number of rules loaded; Core Rule Set matches by mode and rule id (label limited to the rules that actually fired). - Lookup and reputation: requests and bytes by AS number and by country, limited to the `METRICS_TOP_N` (default `50`) busiest of each with the rest summed as `other`, so the label set stays bounded; reputation queries, hits, failures - and remaining daily budget by source; age of each blocklist and lookup - database. + and remaining daily budget by source; GeoJS requests and failures, and clients + that counted as unknown because it did not answer in time; age of each + blocklist and lookup database. - Housekeeping: tracked clients (gauge), state file writes, write failures, last successful write time and size per file; files read again after an edit, and edits set aside because they did not parse; alerts sent, failed and suppressed @@ -759,7 +884,8 @@ One JSON object per alert: - `client`, `netblock`, `asn`, `as_name`, `country` - `reason`: short human-readable sentence - `detail`: event-specific fields, for example `window`, `count`, `limit`, - `limit_percent`, `rule_ids`, `path`, `ban_expires`, and for a ban its notes + `limit_percent`, `rule_ids`, `path`, `ban_expires`, for a ban its notes, and + for `file_error` the file and the position of the error - `suppressed_repeats`: number of identical alerts held back by the cooldown ## Deployment as a sidecar @@ -808,8 +934,9 @@ networks: - Traefik routes only port 8080. The metrics port is reached by the scraper over a docker network it shares with the sidecar; the admin port stays on loopback inside the container and is used through `docker exec`. -- The state volume holds the JSON state files, a few MiB for a small service; it - needs no backup beyond whatever the host already does. +- The state volume holds the JSON state files, a few tens of MiB at most with + the defaults (see "Persistent state"); it needs no backup beyond whatever the + host already does. - SSH access to gitea does not pass through the sidecar and is not protected by it. - Rollout per service: point traefik at the sidecar with only `UPSTREAM_URL` @@ -819,8 +946,8 @@ networks: (`WAF_DISABLED_RULES`, `WAF_EXEMPT_PATHS`, `RATE_LIMIT_EXEMPT_PATHS`) or an exemption (`RATE_LIMIT_EXEMPT_NETS`, `ALLOW_NETS`), and its ban is lifted in `bans.json`. Additions to consider at any time: an alert destination, a lookup - database with biased thresholds, reputation sources, a remote log endpoint, - and an app-specific rule file. + source with country lists or biased thresholds, reputation sources, a remote + log endpoint, and an app-specific rule file. - Notes specific to gitea: - A clone is a response, and fits the defaults of 30 minutes and 5 GB for all but the largest repositories and slowest links. A push is a request: @@ -841,7 +968,11 @@ networks: - Lookup database: a configured file that is missing or unreadable at start stops the start. A replacement that cannot be read while running is ignored: - the file already loaded stays in use, the problem is logged, one alert. + the file already loaded stays in use, the problem is logged, one `file_error` + alert. +- GeoJS slow, down or refusing the sidecar: clients with a kept answer are + unaffected, new clients count as unknown, the sidecar keeps asking with + backoff, one `source_failure` alert per cooldown. - Reputation source down or over quota: no verdict, service continues, one `source_failure` alert per cooldown. - Alert destination down: retried with backoff from a bounded queue, oldest @@ -852,16 +983,18 @@ networks: - Remote log endpoint down: stdout continues, lines are buffered then dropped oldest first, drops counted in metrics. - State file write fails while running: memory stays authoritative, logged, - counted, alerted, retried. + counted, one `file_error` alert per cooldown, retried. - A state file or rule file edited while running that does not parse: the sidecar keeps running on what it has. The state file is set aside as `.bad` and written again from memory; the rule file is left as it is. - Logged, one alert. + Logged, one `file_error` alert. - In short: the only things that stop the process happen at start: invalid configuration, a configured lookup database that is missing or unreadable, a rule file or state file that does not parse, and an unwritable `STATE_DIR`. Once running, a broken helper or a broken edit never takes the protected - service down. + service down. The nearest it comes is GeoJS failing while + `EXCLUSIVELY_ALLOWED_COUNTRIES` is set: new clients then cannot be placed, and + that list refuses them, as it refuses every client it cannot place. ## Risks the design has to handle @@ -875,8 +1008,11 @@ networks: busy person produces, `RATE_LIMIT_EXEMPT_NETS` and `ALLOW_NETS` take known shared addresses out, and the ban notes show an admin what happened before a ban is lifted. -- A `ban` rule that matches a real visitor bans for seven days, so the default - rule file keeps to requests no real visitor sends. +- A `ban` rule that matches a real visitor bans it for seven days, and the + visitor's next request during those days makes the ban permanent. So the + default rule file keeps to requests no real visitor sends, with its paths + anchored at the site root, where no app serves `.env` files or web shells. A + visitor banned by mistake is let back in by lifting the ban in `bans.json`. - IPv6 address rotation inside a /64: handled by grouping. - Widely distributed scrapers using thousands of addresses at low rates each: per-client limits do not see them. The AS number and netblock anomaly alerts @@ -887,9 +1023,15 @@ networks: - Core Rule Set false positives against real apps (gitea's editor, API payloads): a match refuses only that request and bans no one by itself; the request log names the rule, and exclusions by rule id and path fix it. +- GeoJS as the lookup source: every new visitor's address goes to a third party, + and a swarm of fresh addresses, when lookups peak, is when GeoJS may slow down + or block the sidecar. Keeping answers for 7 days and asking about many + addresses in one request keep the number of requests low; the file source has + neither risk. - Slow-request attacks: `CLIENT_REQUEST_TIMEOUT` bounds how long a request may - take to arrive, and an idle timeout on the listener closes connections that - send nothing. + take to arrive, `CLIENT_REQUEST_HEADER_MAX_BYTES` how large its headers may + be, and `CLIENT_IDLE_TIMEOUT` closes a kept-open connection that sends nothing + more. - Alert floods: cooldown and hourly cap. ## Build order @@ -900,8 +1042,8 @@ networks: `observe` mode, the full request log on stdout, the metrics endpoint, health. - Second: rule files, admin endpoints, alerting to all three destinations, remote log sending. -- Third: AS number and country lookup, biased thresholds, byte limits, anomaly - thresholds. +- Third: AS number and country lookup from the file or GeoJS, the country lists, + biased thresholds, byte limits, anomaly thresholds. - Fourth: blocklists, DNSBL, AbuseIPDB, optional CrowdSec decision feed. - Fifth: attack detection with Coraza and the Core Rule Set, trap paths, error bursts. -- 2.54.0 From 2d15e7e746db621d0b46dd00555f41648a3bfc16 Mon Sep 17 00:00:00 2001 From: sneak Date: Wed, 23 Sep 2026 14:36:32 +0000 Subject: [PATCH 3/3] SPEC and README: rule set changes, error bursts, admin bans, GeoJS answers (closes #6) Address the second review of the spec update. The Core Rule Set allows PUT, PATCH and DELETE and the headers git and curl send, inspects only bodies it can read, leaves responses alone, and by default switches off the rules that refuse a code forge's ordinary files, uploads and git traffic. The error burst counts only the sidecar's own refusals. Bans an admin made are never dropped. A limit ban resets the client's counters. GeoJS answers move to their own `lookups.json`. The spec now says which address is the client when every forwarded address is trusted, that the exclusive country list refuses private addresses, which are never sent to GeoJS, what `close` gives behind traefik, and Spamhaus's terms for DROP. Model: opus-5-5 --- README.md | 17 ++-- SPEC.md | 252 ++++++++++++++++++++++++++++++++++-------------------- 2 files changed, 173 insertions(+), 96 deletions(-) diff --git a/README.md b/README.md index 973e8c6..7e12d4d 100644 --- a/README.md +++ b/README.md @@ -79,7 +79,8 @@ goes through the candidates one by one. running; - the OWASP Core Rule Set, run by the Coraza engine, refusing the requests it flags; - - trap paths and bursts of error responses. + - trap paths, and a ban for a client that the rule files or the Core Rule + Set refuse again and again. - Bans: - a clear sign of attack, such as a probe for a `.env` file or a scanner's user agent, bans for seven days on the first request, and any further @@ -102,8 +103,8 @@ goes through the candidates one by one. fields, the decision taken and why, AS number and country, and timings. Optionally also sent to a remote syslog server. - Prometheus metrics on their own port. -- State (bans with their notes, each client's counters, history and lookup - answer, the reputation cache, the alerting state) held in memory and kept in +- State (bans with their notes, each client's counters and history, the GeoJS + answers, the reputation cache, the alerting state) held in memory and kept in readable JSON files, written regularly and at every stop, so a restart loses nothing. Edit a file, or add a rule file, and the running `smallwebwaf` picks up the change. Nothing is read from disk while serving a request. @@ -131,8 +132,9 @@ For each request `smallwebwaf`: client at once for a clear sign of attack; - forwards it to the app and streams the response back, within the size and time limits; -- counts the bytes and any error response, bans the client if it broke a limit, - updates its history, sends any alerts that are due, and writes the log line. +- counts the bytes and any refusal by the rule files or the Core Rule Set, bans + the client if it broke a limit, updates its history, sends any alerts that are + due, and writes the log line. A minimal deployment beside an app in docker-compose. `UPSTREAM_URL` is the only setting: @@ -204,6 +206,11 @@ request. GeoJS publishes no rate limit but may block a caller it thinks asks too much; while it is not answering, new visitors count as coming from an unknown country, which `EXCLUSIVELY_ALLOWED_COUNTRIES` refuses. +Neither source can place a private address, so a client on one, such as a +visitor on your local network, another container or your monitoring, has no +country: `EXCLUSIVELY_ALLOWED_COUNTRIES` refuses it unless you list it in +`ALLOW_NETS`. Such addresses are never sent to GeoJS. + ## Documents - [`SPEC.md`](SPEC.md): the design. diff --git a/SPEC.md b/SPEC.md index 2eb79c8..d94938b 100644 --- a/SPEC.md +++ b/SPEC.md @@ -63,7 +63,7 @@ directory of hand-editable text files. - Attack detection: regex rules read from a directory of plain text rule files and read again whenever the files change (see "Rule files"), Coraza with the OWASP Core Rule Set, and simple signals (requests for listed trap - paths, bursts of error responses). + paths, bursts of requests refused by the rule files or the Core Rule Set). - Ban ledger: active and past bans with their notes, and each client's history, held in memory and kept in JSON files that the sidecar watches for an admin's edits (see "Bans" and "Persistent state"). No database of @@ -101,8 +101,13 @@ whole exchange. - Identify the client. - If the TCP peer is inside `TRUSTED_PROXIES`, walk `X-Forwarded-For` from - the right and take the first address not inside `TRUSTED_PROXIES`. - Otherwise use the TCP peer address and ignore the header. + the right and take the first address not inside `TRUSTED_PROXIES`. If + every address in the header is inside `TRUSTED_PROXIES`, take the + leftmost, so a visitor on a private network who comes through traefik is + known by its own address, not traefik's. If there is no header, as when + another container calls the sidecar directly, the TCP peer is the client. + - If the TCP peer is outside `TRUSTED_PROXIES`, it is the client, and the + header is ignored. - IPv6 clients are grouped by prefix (`IPV6_GROUP_PREFIX`, default 64) for counting and banning, because one abuser usually controls a whole /64. A client is therefore one IPv4 address or one IPv6 group, a /64 by default. @@ -115,9 +120,10 @@ whole exchange. permanent (see "Bans"). - Look up the AS number and country, unless `LOOKUP_SOURCE` is `off`. With GeoJS, a client whose answer is not yet kept waits for it, up to - `LOOKUP_TIMEOUT`. A client the lookup cannot place has an unknown country and - AS number: the exclusive country list refuses it, the biased thresholds give - it `UNKNOWN_LIMIT_PERCENT`, and nothing else treats it differently. + `LOOKUP_TIMEOUT`. A client the lookup cannot place, which includes every + private, loopback and link-local address, has an unknown country and AS + number: the exclusive country list refuses it, the biased thresholds give it + `UNKNOWN_LIMIT_PERCENT`, and nothing else treats it differently. - Country lists. A client whose country is in `DENIED_COUNTRIES`, or, when `EXCLUSIVELY_ALLOWED_COUNTRIES` is set, is not in it, is refused with `BAN_RESPONSE`. Every step up to here needs only the client's address, so the @@ -139,20 +145,23 @@ whole exchange. in file name order then line order. Each rule that matches takes its action: only log, refuse with 403, or refuse and ban. Matching stops at the first rule that refuses or bans. -- Core Rule Set inspection of the request (headers, URL, and body up to - `WAF_BODY_LIMIT`). In `block` mode, the default, a match at or over the - anomaly threshold is refused with 403; in `detect` mode it is only logged and - alerted. +- Core Rule Set inspection of the request: its headers, its URL, and its body up + to `WAF_BODY_LIMIT` when the body is form data, multipart, JSON or XML, the + kinds the Core Rule Set can read. Any other body, such as a git push or a + container image layer, reaches the app uninspected, and so does a JSON or XML + body larger than `WAF_BODY_LIMIT`, which cannot be read in part. Responses are + not inspected. In `block` mode, the default, a match at or over the anomaly + threshold is refused with 403; in `detect` mode it is only logged and alerted. - Forward to `UPSTREAM_URL`, streaming. Add `X-Forwarded-For` and, if enabled, `X-Client-ASN` and `X-Client-Country` for the app's own logs. - After the response. - Add response bytes (and request body bytes) to the byte counters. A client whose byte total passes a byte limit times the percentage has broken that limit and is banned. The response in progress is not cut off. - - Count the client's requests answered with 403 or 404, whether the app gave - that answer or the sidecar refused the request after a rule file or Core - Rule Set match. More than `ERROR_BURST_THRESHOLD` of them within a minute - breaks a limit and bans the client. + - Count the client's requests that the sidecar refused after a rule file or + Core Rule Set match; answers the app gave are not counted. More than + `ERROR_BURST_THRESHOLD` of them within a minute breaks a limit and bans + the client. - Update the client's history and the anomaly counters, and evaluate alert thresholds. @@ -167,20 +176,25 @@ whole exchange. shutdown and loaded again at start (see "Persistent state"), so a restart does not hand every client a fresh allowance. Buckets whose time has passed are discarded on load. -- Memory is bounded by `MAX_TRACKED_CLIENTS` and `MAX_BANS`. When the table of - clients is full, the least recently seen client is dropped first, with its - history and its lookup answer. When `MAX_BANS` bans are held, the ban that has - gone longest without a request from its netblock is dropped first, whether it - is past, active or permanent (see "Bans"). +- Memory is bounded by `MAX_TRACKED_CLIENTS`, `MAX_BANS` and the number of GeoJS + answers kept (see "Configuration surface"). When the table of clients is full, + the least recently seen client is dropped first, with its history. When + `MAX_BANS` bans the sidecar made are held, the one that has gone longest + without a request from its netblock is dropped first, whether it is past, + active or permanent. Bans an admin made are never dropped, and there are as + many as the admin adds (see "Bans"). ## Bans A ban refuses every request from a netblock, answering with `BAN_RESPONSE` (`403` by default), until the ban ends. A permanent ban does not run out: it -ends only when an admin lifts it, or when a new ban is made while `MAX_BANS` -bans are held and it is the ban that has gone longest without a request. A -scanner whose ban was dropped that way is refused and banned again by its next -probe. +ends when an admin lifts it or, if the sidecar made it, when a new ban is made +while `MAX_BANS` bans the sidecar made are held and it is the one that has gone +longest without a request. A scanner whose ban was dropped that way is refused +and banned again by its next probe. A ban an admin made, whose cause is `admin`, +is never dropped and does not count toward `MAX_BANS`, so such bans can never +fill the table, however many there are. An admin who wants to keep a ban the +sidecar made sets its cause to `admin`. The netblock a ban covers is the client: its IPv4 address, or its IPv6 group of `IPV6_GROUP_PREFIX` (a /64 by default). `BAN_SCOPE_V4_PREFIX` can widen an IPv4 @@ -201,8 +215,8 @@ to a ban, each by its own rule: - Once that ban has run out, the netblock is served like any other, but its next clear sign of attack bans it permanently at once. - A broken limit: a request over a request limit, a response that takes the - client's byte total past a byte limit, or more error responses than - `ERROR_BURST_THRESHOLD` within a minute. + client's byte total past a byte limit, or more than `ERROR_BURST_THRESHOLD` + requests within a minute refused after a rule file or Core Rule Set match. - The first such ban, or one that comes more than `LIMIT_BAN_REPEAT_WINDOW` (default `24h`) after the last such ban ended, lasts `LIMIT_BAN_DURATION` (default `1h`). @@ -210,13 +224,16 @@ to a ban, each by its own rule: long as the last one: one hour, then 3, 9, 27 and 81 hours. - A ban that would be longer than `MAX_BAN_DURATION` (default `7d`) is permanent instead; with the defaults, that is the sixth ban in a row. + - Each such ban sets the client's counters for every limit back to zero, so + once it ends only new requests can break a limit again; the client's + history keeps its totals. A Core Rule Set match refuses only the request it matched, without a ban, because the Core Rule Set has false positives; a `block` rule does the same. -Those refusals count toward the error burst like any other 403, so a client that -keeps setting them off is banned under the second rule. Requests refused under a -ban are not counted against limits, but they are counted in the client's history -and in the ban's notes. +Those refusals count toward the error burst, so a client that keeps setting them +off is banned under the second rule. Requests refused under a ban are not +counted against limits, but they are counted in the client's history and in the +ban's notes. Every ban carries notes with what an admin needs to decide whether to lift it (see "Persistent state"). An admin can add or lift a ban at any time by editing @@ -283,8 +300,9 @@ The settings, by group: - `IPV6_GROUP_PREFIX` (default `64`). - `MAX_TRACKED_CLIENTS` (default `20000`): clients held in memory and in `clients.json` (see "Persistent state"). - - `MAX_BANS` (default `5000`): bans held in memory and in `bans.json`, past, - active and permanent (see "Bans"). + - `MAX_BANS` (default `5000`): bans the sidecar made, past, active and + permanent, held in memory and in `bans.json`. Bans an admin made are kept + besides these and never dropped (see "Bans"). - Persistent state - `STATE_DIR` (default `/data`): the JSON state files. The image declares `/data` as a volume, so it is writable even when nothing is mounted; a @@ -294,8 +312,8 @@ The settings, by group: permanent, `bans.json` is written this long later with every change made in between, so a burst of changes becomes one write. - `STATE_COUNTER_INTERVAL` (default `15m`): how often `clients.json`, - `reputation.json` and `alerts.json` are written, and `bans.json` when only - the counts in its notes changed. + `lookups.json`, `reputation.json` and `alerts.json` are written, and + `bans.json` when only the counts in its notes changed. - Logging - The request log on stdout is always on and has no switch. - `LOG_LEVEL` (default `info`): for the process's own messages (start-up, @@ -400,10 +418,13 @@ The settings, by group: `country_code`, `asn` and `organization_name` (the AS name). A `country_code` of `null`, which GeoJS gives for some ranges, and an `asn` of `64512`, which it gives when it knows none, count as unknown. GeoJS is - told the address of every new visitor. - - Each GeoJS answer is kept for 7 days with the client's record, in memory - and in `clients.json`, so it survives a restart; after 7 days the client's - next request asks again. A client dropped from the table loses its answer. + told the address of every new visitor, except private, loopback and + link-local addresses, which no source can place and which are never sent. + - Each GeoJS answer is kept for 7 days in memory and in `lookups.json`, + apart from the table of clients, so it survives a restart and outlasts a + client dropped from that table; after 7 days the client's next request + asks again. Up to 100,000 answers are kept, about 15 MiB; when that many + are held, the answer used longest ago goes first. - `LOOKUP_TIMEOUT` (default `1s`): how long a client with no kept answer waits for one; GeoJS normally answers in a fraction of that. At most one request to GeoJS is under way at a time, the addresses that arrive @@ -431,9 +452,12 @@ The settings, by group: listed country is refused. - `EXCLUSIVELY_ALLOWED_COUNTRIES`: for example `us,de`. Every request from any other country is refused, and so is every request from a client the - lookup cannot place, such as an address missing from the database or one - GeoJS has not answered for in time. A list that let those through would - let every new client in whenever GeoJS stops answering. + lookup cannot place: one whose address is missing from the database, one + GeoJS has not answered for in time, and any client on a private address, + such as a visitor on the local network, another container or internal + monitoring. Those that should reach the app go in `ALLOW_NETS`. A list + that let unplaced clients through would let every new client in whenever + GeoJS stops answering. - Both may be set. `DENIED_COUNTRIES` then adds nothing, since the exclusive list already refuses every other country, and a code on both lists stops the start with a message naming it. @@ -477,27 +501,55 @@ The settings, by group: - `WAF_PARANOIA_LEVEL` (default `1`), `WAF_ANOMALY_THRESHOLD` (default `5`): the Core Rule Set's own two tuning values, at the Core Rule Set's own defaults. - - `WAF_DISABLED_RULES` (default `920420,920440`): rule ids to switch off - when an app trips a false positive. The two in the default refuse requests - by content type and by file extension; in front of a code forge they would - refuse git's clone and push over HTTP and the display of source files such - as `.sh` or `.sql`. A list given replaces the default, so include them in - it. + - The sidecar also changes the Core Rule Set in four ways that no setting + undoes, since in front of gitea each would otherwise refuse ordinary + requests: + - PUT, PATCH and DELETE are allowed methods besides GET, HEAD, POST and + OPTIONS; APIs, container image pushes and package uploads use them. + Other methods stay refused. + - The request headers `Content-Encoding` and `Expect` are allowed: git + compresses a fetch request over 1 KiB and says so in + `Content-Encoding`, and curl and git send `Expect` before some large + uploads. The other headers the Core Rule Set refuses, such as `Proxy`, + stay refused. + - Only a body the Core Rule Set can read is inspected (see "Data flow + for one request"). It would read any other body as form data, where + binary content such as a git push trips rules written for text. + - Responses are not inspected. A raw file from a repository, such as a + shell script, looks to the response rules like source code leaking + from the server. + - `WAF_DISABLED_RULES` (default + `920340,920420,920440,920640,930130,930140,932180`): rule ids to switch + off when an app trips a false positive. The default switches off the rules + that refuse a request for the type of its body, when it is missing or not + on the Core Rule Set's short list (920340, 920420, 920640); for its file + extension, such as `.sh` or `.sql` (920440); for a file or directory name + in its path, such as `.git/`, `.gitignore`, `Dockerfile`, `package.json` + or an editor's settings directory (930130, 930140); and for the name of an + uploaded file, such as `debug.log` or `config.yml` (932180). In front of a + code forge these refuse git over HTTP, container image and package + uploads, views of ordinary files in a repository, and logs attached to + issues. The default rule file still bans probes for `.env` and `.git` at + the site root (see "Rule files"). A list given replaces the default, so + include them in it. - `WAF_EXEMPT_PATHS`: path prefixes not inspected. - - `WAF_BODY_LIMIT` (default `128K`): bodies are inspected up to this size - and streamed beyond it without buffering, so large uploads and git pushes - are not held in memory. + - `WAF_BODY_LIMIT` (default `128K`): bodies the Core Rule Set reads are + inspected up to this size and streamed beyond it without buffering, so + large uploads are not held in memory. - `TRAP_PATHS`: paths the app never serves and only scanners ask for, for example `/wp-login.php,/xmlrpc.php` in front of gitea. A request for one is a clear sign of attack. This is the env-var short form of a `path` rule with the `ban` action, for deployments that mount no rule files. - - `ERROR_BURST_THRESHOLD` (default `30`): responses with status 403 or 404 - per client per minute; more than this breaks a limit (see "Bans"). - Scanners walking lists of paths cause hundreds a minute; a person rarely - causes more than a few. 401 is not counted: git and container registry - clients send their first request without credentials and are answered 401 - at the start of every push, every fetch from a private repository and - every image pull. + - `ERROR_BURST_THRESHOLD` (default `30`): requests per client per minute + that the sidecar refused after a rule file or Core Rule Set match; more + than this breaks a limit (see "Bans"). A client trying one attack after + another is refused many times a minute; a person rarely more than a few + times. Answers the app gives are not counted, since they do not tell a + scanner from an ordinary client: in front of gitea, container image and + package clients are answered 404 by design, for each layer a push checks + and each package a lookup asks about, often hundreds of times a minute, + and git and registry clients are answered 401 at the start of every push, + every fetch from a private repository and every image pull. - Bans (R5), following the rules under "Bans" - `ATTACK_BAN_DURATION` (default `7d`): the ban for a first clear sign of attack. @@ -509,21 +561,27 @@ The settings, by group: instead. - `BAN_RESPONSE` (default `403`): `403`, `429`, or `close` to drop the connection without an answer. `403` makes a ban easy to recognise when - debugging; `close` tells the client nothing. + debugging. Behind traefik, `close` does not leave the client unanswered: + traefik answers `502`, as it does whenever its backend drops a connection. - `BAN_SCOPE_V4_PREFIX` (default `32`): widen to for example `24` to ban the surrounding netblock. - Reputation (R6). No source is on by default. A DNS blocklist would be told the address of every visitor. AbuseIPDB and CrowdSec need an account or an engine of the operator's own. A downloaded list reveals nothing about visitors, but - each sidecar fetches its own copy, and the list most fit to be a default, - Spamhaus DROP, may be fetched at most once a day: a default would break that - on any host that runs several sidecars. + the list most fit to be a default, Spamhaus DROP, must not be downloaded + automatically more than once an hour, and Spamhaus may block an address that + downloads it too often. Each sidecar fetches its own copy, so on a host that + runs several sidecars, which share one address, downloads would often come + less than an hour apart. - `BLOCKLIST_URLS`: text files of addresses and netblocks, one per line; anything after a `;` or `#` on a line is ignored. For example the Spamhaus - DROP list, `https://www.spamhaus.org/drop/drop.txt`; Spamhaus asks that it - be fetched at most once a day and that products using it credit The - Spamhaus Project. Refreshed every `BLOCKLIST_REFRESH` (default `24h`); the - last good copy is kept on failure and across restarts. + DROP list, `https://www.spamhaus.org/drop/drop.txt`. Spamhaus's terms for + it, on its DROP page: automated downloads must be at least one hour apart, + and once a day is more than enough in most cases; a product that uses it + must credit The Spamhaus Project and keep the list's date and copyright + lines with the data. Refreshed every `BLOCKLIST_REFRESH` (default `24h`); + the last good copy is kept whole, comment lines included, on failure and + across restarts. - `BLOCKLIST_ACTION` (default `deny`): `deny`, `limit:`, or `log`. `deny` refuses every request from a listed address, which suits lists of networks that send nothing legitimate, such as DROP. `limit:` @@ -691,17 +749,21 @@ files are meant for people as well: an admin can read or edit them at any time, and the running sidecar takes the edit in. - Files, each holding one kind of state: - - `bans.json`: active and past bans, up to `MAX_BANS`. Per entry: the - netblock, start, expiry (`null` for permanent), what caused it (`attack`, - `limit`, `admin` or `crowdsec`), a short reason, when it was lifted if an - admin lifted it, and a `notes` field (below). + - `bans.json`: active and past bans, up to `MAX_BANS` that the sidecar made + and every ban an admin made. Per entry: the netblock, start, expiry + (`null` for permanent), what caused it (`attack`, `limit`, `admin` or + `crowdsec`), a short reason, when it was lifted if an admin lifted it, and + a `notes` field (below). - `clients.json`: per client, its minute, hour and day counters and its history since it was first seen: first and last time seen, the AS number, - AS name and country last looked up and when (with GeoJS, the answer kept - for 7 days), total requests and bytes in each direction, how many requests - were forwarded and how many refused, responses by status class, and - offences by kind. Clients that were never banned are kept too, so every - client's history survives a restart; `GET /clients/` shows it. + AS name and country last looked up and when, total requests and bytes in + each direction, how many requests were forwarded and how many refused, + responses by status class, and offences by kind. Clients that were never + banned are kept too, so every client's history survives a restart; + `GET /clients/` shows it. + - `lookups.json`: GeoJS answers, one per client: the AS number, AS name and + country, when GeoJS was asked and when the answer was last used; up to + 100,000, each for 7 days. - `reputation.json`: the last good copy of each list fetched from a URL, cached DNSBL and reputation API verdicts, each with the time it was fetched, and the AbuseIPDB checks spent today, so a restart does not reset @@ -733,20 +795,22 @@ and the running sidecar takes the edit in. that is under 2 GiB of disk writes a day. - `bans.json` takes about 2 KiB per ban and at most about 8 KiB, since the texts in the notes are cut short. At the default `MAX_BANS` of 5,000 it is - about 10 MiB, and never more than about 40 MiB. It is written when a ban - is made, lifted or made permanent, at most once every 10 seconds, and - otherwise with the 15-minute write, so its writes follow the bans made: - with a full file, a hundred new bans a day come to about 1 GiB of disk - writes. + about 10 MiB, and never more than about 40 MiB, plus whatever bans an + admin made. It is written when a ban is made, lifted or made permanent, at + most once every 10 seconds, and otherwise with the 15-minute write, so its + writes follow the bans made: with a full file, a hundred new bans a day + come to about 1 GiB of disk writes. + - `lookups.json` takes about 150 bytes per answer, about 15 MiB when full. + Written every 15 minutes, that is under 1.5 GiB of disk writes a day. - `reputation.json` and `alerts.json` are usually a few MiB or less. - Writing a file of these sizes takes well under a second on an ordinary disk, in the background, so rewriting whole files needs nothing cleverer. - Format: indented JSON with a top-level `version` number, entries sorted by client address, times in RFC 3339 UTC, durations and sizes as plain numbers with the unit in the field name. The aim is that a person can open `bans.json` - in an editor, find an address, and remove or add an entry. `clients.json` puts - each client on one line instead, which halves its size and lets `grep` show - everything about one client. + in an editor, find an address, and remove or add an entry. `clients.json` and + `lookups.json` put each entry on one line instead, which halves their size and + lets `grep` show everything about one client. - Writing: - serialise from a snapshot taken under the lock, so requests are not held up while the file is written; @@ -756,8 +820,8 @@ and the running sidecar takes the edit in. - `bans.json` is written `STATE_WRITE_DELAY` after a ban is made, lifted or made permanent. Changes that only update the counts in its notes wait for the write every `STATE_COUNTER_INTERVAL`, as all of `clients.json`, - `reputation.json` and `alerts.json` do. All four are written on orderly - shutdown (SIGTERM); + `lookups.json`, `reputation.json` and `alerts.json` do. All five are + written on orderly shutdown (SIGTERM); - before writing a file, the sidecar checks whether it changed on disk since the sidecar last read or wrote it; if it did, the sidecar takes that edit in first (below), so an admin's edit is never overwritten. @@ -954,12 +1018,17 @@ networks: at the defaults, one larger than 100 MB or taking more than 60 seconds to upload is cut off, so a gitea that takes large pushes needs `CLIENT_REQUEST_MAX_BYTES`, `UPSTREAM_REQUEST_MAX_BYTES`, - `CLIENT_REQUEST_TIMEOUT` and `UPSTREAM_REQUEST_TIMEOUT` raised to fit. - `WAF_BODY_LIMIT` keeps pack uploads out of memory. - - The default `WAF_DISABLED_RULES` lets git's clone and push over HTTP, and - the display of source files, through the Core Rule Set. Pages where people - post code (issues, pull requests, the web editor) can still trip other - rules; such a match refuses only that request, and the request log names + `CLIENT_REQUEST_TIMEOUT` and `UPSTREAM_REQUEST_TIMEOUT` raised to fit. The + Core Rule Set does not read a pack upload, which streams through without + being held in memory. + - With the sidecar's changes to the Core Rule Set and the default + `WAF_DISABLED_RULES` (see "Configuration surface", attack detection), + git's clone, fetch and push over HTTP, the API, container images, + packages, and views of ordinary files pass the Core Rule Set. Requests + that carry text people write, such as issues, pull requests, searches, the + web editor and uploads, can still trip other rules, and so can a file + whose name the Core Rule Set takes for an attack, such as an `.xhtml` + page. Such a match refuses only that request, and the request log names the rule in `waf_rule_ids` for `WAF_DISABLED_RULES`. - Archive download and blame or history pages are what scrapers hammer; request limits do most of the work there. @@ -1001,8 +1070,9 @@ networks: - Forged `X-Forwarded-For`: handled by believing the header only from a peer inside `TRUSTED_PROXIES` and by walking it from the right. The default trusts every private address, which in the intended deployment means traefik; where - other containers can reach the sidecar directly, setting `TRUSTED_PROXIES` to - traefik's own network closes that gap. + other containers can reach the sidecar directly, or visitors on a private + network reach traefik, they can name another address in the header, and + setting `TRUSTED_PROXIES` to traefik's own network closes that gap. - Many clients behind one address (mobile carriers, offices, Tor): they share its limits and its bans. The default limits sit several times above what one busy person produces, `RATE_LIMIT_EXEMPT_NETS` and `ALLOW_NETS` take known -- 2.54.0