From 8fc7539f1a0aeb5586a0d67288462c272741a716 Mon Sep 17 00:00:00 2001 From: sneak Date: Wed, 23 Sep 2026 13:37:34 +0000 Subject: [PATCH] SPEC and README: review fixes, country lists, GeoJS lookups (closes #6) Address the review of the spec update and fold in issues 8 and 11. Default ban rules are anchored at the site root. `bans.json` is bounded by `MAX_BANS` and rewritten only when a ban is made, lifted or made permanent. `clients.json` keeps one client per line, holds at most 20000 clients and is written every 15 minutes. Anomaly counters and alert state go to a new `alerts.json`, the AbuseIPDB count to `reputation.json`. 401 no longer counts toward the error burst. New settings: `DENIED_COUNTRIES`, `EXCLUSIVELY_ALLOWED_COUNTRIES`, `LOOKUP_SOURCE` (the IPinfo file or GeoJS), `LOOKUP_TIMEOUT`, `CLIENT_REQUEST_HEADER_MAX_BYTES` and `CLIENT_IDLE_TIMEOUT`. Model: opus-5-5 --- README.md | 64 +++++---- SPEC.md | 382 +++++++++++++++++++++++++++++++++++++----------------- 2 files changed, 304 insertions(+), 142 deletions(-) diff --git a/README.md b/README.md index 38bd9b9..973e8c6 100644 --- a/README.md +++ b/README.md @@ -52,7 +52,7 @@ goes through the candidates one by one. other setting has a default chosen for a service facing the internet in 2026. - Real client address worked out from `X-Forwarded-For`, trusting only the proxy networks you list, by default the private address ranges. IPv6 clients are - counted by /64. + counted by /64 by default. - Size and time limits on requests and responses, both between the client and `smallwebwaf` and between `smallwebwaf` and the app: by default a request may take 60 seconds and 100 MB, a response 30 minutes and 5 GB. @@ -61,8 +61,14 @@ goes through the candidates one by one. real visitors need. - Netblocks that bypass rate limiting, netblocks that bypass everything, and netblocks that are always refused. -- AS number and country lookup for every client from the IPinfo Lite database - file, which you download and mount (see "Lookup database" below). +- AS number and country lookup for every client, off until you choose a source: + the IPinfo Lite database file, which you download and mount, or the free GeoJS + web service (see "Country and AS number lookup" below). +- Country lists: `DENIED_COUNTRIES` refuses every request from the countries + listed, `EXCLUSIVELY_ALLOWED_COUNTRIES` every request from anywhere else. Such + a request gets the answer a banned client gets as soon as the client's address + has been looked up, before its body is read and without the rule files or the + Core Rule Set looking at it, and no ban is made. - Biased limits: listed AS numbers and countries get a percentage of every limit, for example 50 percent for common abuse-source networks, so their clients are banned after fewer requests. Zero percent is a zero allowance: the @@ -96,11 +102,11 @@ goes through the candidates one by one. fields, the decision taken and why, AS number and country, and timings. Optionally also sent to a remote syslog server. - Prometheus metrics on their own port. -- State (bans with their notes, each client's counters and history, the - reputation cache) held in memory and kept in readable JSON files that always - hold a copy of it, so a restart loses nothing. Edit a file, or add a rule - file, and the running `smallwebwaf` picks up the change. Nothing is read from - disk while serving a request. +- State (bans with their notes, each client's counters, history and lookup + answer, the reputation cache, the alerting state) held in memory and kept in + readable JSON files, written regularly and at every stop, so a restart loses + nothing. Edit a file, or add a rule file, and the running `smallwebwaf` picks + up the change. Nothing is read from disk while serving a request. - A small admin endpoint for health checks, listing, adding and lifting bans, and asking why a given address was refused. @@ -115,7 +121,9 @@ For each request `smallwebwaf`: - works out who the client really is; - lets it straight through if it is on the bypass list, refuses it if it is on the deny list or currently banned; -- looks up its AS number and country, and any cached reputation verdict; +- looks up its AS number and country, and refuses it if that country is denied, + or is not among the only ones allowed; +- checks for a cached reputation verdict; - picks the client's limit percentage from those; - checks the minute, hour and day request counters against the limits, and bans the client if it breaks one; @@ -162,7 +170,7 @@ A rule file is one rule per line: a name, what to match against, what to do, and a regex. ``` -env-file path ban (?i)/\.env(\.[a-z]+)?$ +env-file path ban (?i)^/\.env(\.[a-z]+)?$ scanner-agent user_agent ban (?i)\b(sqlmap|nikto|nuclei|wpscan)\b ``` @@ -170,19 +178,31 @@ scanner-agent user_agent ban (?i)\b(sqlmap|nikto|nuclei|wpscan)\b rules, the rule file format, the state files, the log fields, the metrics, failure behaviour and the build order. -## Lookup database +## Country and AS number lookup -AS number and country lookups, and the biased limits that use them, need the -free IPinfo Lite database (`ipinfo_lite.mmdb`). You download it with your own -IPinfo account, mount it into the container, point `LOOKUP_DB_PATH` at it and -refresh it when you choose; `smallwebwaf` never downloads it itself. IPinfo -releases it under the Creative Commons Attribution-ShareAlike 4.0 International -License and asks for attribution, in its own words on https://ipinfo.io/lite: -"The attribution requirements can be met by giving our service credit as your -data source. Simply place a link to IPinfo on the website, application, or -social media account that uses our data." Its example of such a credit is a link -mentioning "IP address data is powered by IPinfo". A service that uses the -database through `smallwebwaf` should carry that link. +AS number and country lookups, and the country lists and biased limits that use +them, are off until you choose one of two sources with `LOOKUP_SOURCE`. + +`LOOKUP_SOURCE=file` reads the free IPinfo Lite database (`ipinfo_lite.mmdb`). +You download it with your own IPinfo account, mount the directory that holds it +into the container, point `LOOKUP_DB_PATH` at the file and refresh it when you +choose; `smallwebwaf` never downloads it itself, and reads it again when you +replace it. It has to be the directory rather than the file itself: docker does +not show a single mounted file being replaced, so a refresh would go unseen. +IPinfo releases it under the Creative Commons Attribution-ShareAlike 4.0 +International License and asks for attribution, in its own words on +https://ipinfo.io/lite: "The attribution requirements can be met by giving our +service credit as your data source. Simply place a link to IPinfo on the +website, application, or social media account that uses our data." Its example +of such a credit is a link mentioning "IP address data is powered by IPinfo". A +service that uses the database through `smallwebwaf` should carry that link. + +`LOOKUP_SOURCE=geojs` asks the free GeoJS web service instead, with no account +and no file. Every new visitor's address is sent to GeoJS. Each answer is kept +for seven days, across restarts, and many addresses are asked about in one +request. GeoJS publishes no rate limit but may block a caller it thinks asks too +much; while it is not answering, new visitors count as coming from an unknown +country, which `EXCLUSIVELY_ALLOWED_COUNTRIES` refuses. ## Documents diff --git a/SPEC.md b/SPEC.md index 492aca0..2eb79c8 100644 --- a/SPEC.md +++ b/SPEC.md @@ -12,8 +12,8 @@ forwards to the application. The sidecar limits request and byte rates per client, bounds the size and duration of every request and response, detects attacks, bans abusers (briefly at first, for seven days on a clear sign of attack, permanently when they keep at it), consults IP reputation sources, looks -up the AS number and country of each client, lowers its limits for listed AS -numbers and countries, and sends alerts. +up the AS number and country of each client, can refuse whole countries, lowers +its limits for listed AS numbers and countries, and sends alerts. It is meant to protect an app on the open internet from the first request with one setting, `UPSTREAM_URL`: every other setting has a default chosen for a @@ -52,8 +52,9 @@ directory of hand-editable text files. - Client identification: works out the real client IP from `X-Forwarded-For`, trusting only the proxy netblocks in `TRUSTED_PROXIES`, by default the private address ranges. - - Lookup: AS number and country from one database file the operator - supplies, held in memory. + - Lookup: AS number and country, from one of two sources the operator + chooses: a database file the operator supplies, held in memory, or the + GeoJS web service, whose answers are kept for seven days. - Reputation: static blocklists fetched on a schedule; DNSBL and reputation API queries made in the background and cached. - Counters: per-client request and byte counts per minute, hour and day, @@ -104,7 +105,7 @@ whole exchange. Otherwise use the TCP peer address and ignore the header. - IPv6 clients are grouped by prefix (`IPV6_GROUP_PREFIX`, default 64) for counting and banning, because one abuser usually controls a whole /64. A - client is therefore one IPv4 address or one IPv6 /64. + client is therefore one IPv4 address or one IPv6 group, a /64 by default. - Static lists. - In `ALLOW_NETS`: skip every check below and forward. Still counted for anomaly alerts. @@ -112,8 +113,17 @@ whole exchange. - Ban ledger. An active ban on the client's netblock: refuse with `BAN_RESPONSE`. If a clear sign of attack caused the ban, the ban becomes permanent (see "Bans"). -- Lookup AS number and country, when a lookup database is configured. Unknown is - a valid result and changes nothing. +- Look up the AS number and country, unless `LOOKUP_SOURCE` is `off`. With + GeoJS, a client whose answer is not yet kept waits for it, up to + `LOOKUP_TIMEOUT`. A client the lookup cannot place has an unknown country and + AS number: the exclusive country list refuses it, the biased thresholds give + it `UNKNOWN_LIMIT_PERCENT`, and nothing else treats it differently. +- Country lists. A client whose country is in `DENIED_COUNTRIES`, or, when + `EXCLUSIVELY_ALLOWED_COUNTRIES` is set, is not in it, is refused with + `BAN_RESPONSE`. Every step up to here needs only the client's address, so the + request body has not been read yet, and the refusal skips everything below, + the rule files and the Core Rule Set included. It is counted in the metrics, + but it is not an offence and makes no ban. - Reputation. - Address inside a fetched blocklist: apply `BLOCKLIST_ACTION`. - Cached DNSBL or reputation API result: apply `REPUTATION_ACTION`. @@ -139,10 +149,10 @@ whole exchange. - Add response bytes (and request body bytes) to the byte counters. A client whose byte total passes a byte limit times the percentage has broken that limit and is banned. The response in progress is not cut off. - - Count the client's requests answered with 401, 403 or 404, whether the app - gave that answer or the sidecar refused the request after a rule file or - Core Rule Set match. More than `ERROR_BURST_THRESHOLD` of them within a - minute breaks a limit and bans the client. + - Count the client's requests answered with 403 or 404, whether the app gave + that answer or the sidecar refused the request after a rule file or Core + Rule Set match. More than `ERROR_BURST_THRESHOLD` of them within a minute + breaks a limit and bans the client. - Update the client's history and the anomaly counters, and evaluate alert thresholds. @@ -157,17 +167,25 @@ whole exchange. shutdown and loaded again at start (see "Persistent state"), so a restart does not hand every client a fresh allowance. Buckets whose time has passed are discarded on load. -- Memory is bounded by `MAX_TRACKED_CLIENTS`; when full, the least recently seen - client is dropped first, with its history and its past bans. Active bans are - never dropped; one whose client is no longer tracked is dropped when it ends. +- Memory is bounded by `MAX_TRACKED_CLIENTS` and `MAX_BANS`. When the table of + clients is full, the least recently seen client is dropped first, with its + history and its lookup answer. When `MAX_BANS` bans are held, the ban that has + gone longest without a request from its netblock is dropped first, whether it + is past, active or permanent (see "Bans"). ## Bans A ban refuses every request from a netblock, answering with `BAN_RESPONSE` -(`403` by default), until the ban ends; a permanent ban never ends. The netblock -a ban covers is the client's IPv4 address (or the wider IPv4 prefix that -`BAN_SCOPE_V4_PREFIX` sets) or its IPv6 /64, and never more, so a permanent ban -does not reach neighbours who did nothing. +(`403` by default), until the ban ends. A permanent ban does not run out: it +ends only when an admin lifts it, or when a new ban is made while `MAX_BANS` +bans are held and it is the ban that has gone longest without a request. A +scanner whose ban was dropped that way is refused and banned again by its next +probe. + +The netblock a ban covers is the client: its IPv4 address, or its IPv6 group of +`IPV6_GROUP_PREFIX` (a /64 by default). `BAN_SCOPE_V4_PREFIX` can widen an IPv4 +ban to the surrounding netblock. At the defaults a ban covers one address or one +/64, so a permanent ban does not reach neighbours who did nothing. An offence is a request the sidecar holds against the client: one that carries a clear sign of attack, breaks a limit, or is refused by a rule file or the Core @@ -206,21 +224,30 @@ Every ban carries notes with what an admin needs to decide whether to lift it lifted before it ends is kept, marked as lifted, and does not count toward a longer ban; deleting its entry from `bans.json` forgets it entirely. -Clients of listed AS numbers and countries, and clients with a poor reputation, -get lower limits (see "Biased thresholds"), so the same rules ban them after -fewer requests. The result is always a ban, first a temporary one and then, for -repeated abuse, a permanent one. +Clients of the AS numbers and countries listed in the biased thresholds, and +clients listed by a reputation source whose action is `limit:`, get +lower limits (see "Biased thresholds"), so the same rules ban them after fewer +requests. For them the result is always a ban, first a temporary one and then, +for repeated abuse, a permanent one. + +Some refusals make no ban at all. `DENY_NETS`, a blocklist or reputation hit +whose action is `deny`, and the country lists (`DENIED_COUNTRIES`, +`EXCLUSIVELY_ALLOWED_COUNTRIES`) refuse each request they cover and record +nothing in `bans.json`. ## Configuration surface Conventions: lists are comma separated; netblocks are CIDR (a bare address means /32 or /128); durations use Go syntax plus `d` for days (`90s`, `15m`, `24h`, -`7d`); byte sizes accept `K`, `M`, `G` suffixes. +`7d`); byte sizes accept `K`, `M`, `G` suffixes. Countries are two-letter ISO +codes in either case (`de` and `DE` are the same); a code that is not a country +code, such as `nk` (North Korea is `kp`), stops the start with a message naming +it. - Only `UPSTREAM_URL` is required. Every other setting has a default chosen for - a service facing the internet in 2026, or stays off until it is given - something only the operator can supply: an alert destination, an account key, - the lookup database. + a service facing the internet in 2026, or stays off until the operator + supplies or chooses what it needs: an alert destination, an account key, a + lookup source. - Any limit or threshold can be switched off with the value `off`. - A list set to an empty value is an empty list, and replaces the default. - Every variable may instead be given as `NAME_FILE` pointing at a file holding @@ -254,17 +281,21 @@ The settings, by group: address. A list given replaces the default; set but empty, it trusts nothing. - `IPV6_GROUP_PREFIX` (default `64`). - - `MAX_TRACKED_CLIENTS` (default `50000`): clients held in memory and in + - `MAX_TRACKED_CLIENTS` (default `20000`): clients held in memory and in `clients.json` (see "Persistent state"). + - `MAX_BANS` (default `5000`): bans held in memory and in `bans.json`, past, + active and permanent (see "Bans"). - Persistent state - `STATE_DIR` (default `/data`): the JSON state files. The image declares `/data` as a volume, so it is writable even when nothing is mounted; a named volume, as in the examples below, keeps the state in a place that outlives the container. - - `STATE_WRITE_DELAY` (default `2s`): after a ban changes, wait this long - before writing `bans.json`, so a burst of changes becomes one write. - - `STATE_COUNTER_INTERVAL` (default `60s`): how often `clients.json` and - `reputation.json` are written. + - `STATE_WRITE_DELAY` (default `10s`): after a ban is made, lifted or made + permanent, `bans.json` is written this long later with every change made + in between, so a burst of changes becomes one write. + - `STATE_COUNTER_INTERVAL` (default `15m`): how often `clients.json`, + `reputation.json` and `alerts.json` are written, and `bans.json` when only + the counts in its notes changed. - Logging - The request log on stdout is always on and has no switch. - `LOG_LEVEL` (default `info`): for the process's own messages (start-up, @@ -319,6 +350,14 @@ The settings, by group: send its whole request, headers and body. - `CLIENT_REQUEST_MAX_BYTES` (default `100M`): the largest request body a client may send. + - `CLIENT_REQUEST_HEADER_MAX_BYTES` (default `32K`): the largest request + line and headers a client may send. Over it, the sidecar answers `431` and + closes the connection, and nothing reaches the app. + - `CLIENT_IDLE_TIMEOUT` (default `120s`): how long a kept-open connection + may wait for its next request before the sidecar closes it. It is longer + than the 90 seconds after which traefik, by default, closes a connection + it is not using, so traefik closes first and never sends a request on a + connection the sidecar is closing. - `CLIENT_RESPONSE_TIMEOUT` (default `30m`): how long the sidecar may take to deliver one response to the client, from the end of the request to the last byte. @@ -340,27 +379,72 @@ The settings, by group: started it can only be cut off, and the connection is closed. - A WebSocket connection leaves these limits behind once it is upgraded: it stays open until either side closes it. -- Lookup of AS number and country (R7) - - `LOOKUP_DB_PATH`: the IPinfo Lite database in its `.mmdb` form - (`ipinfo_lite.mmdb`), one file that carries both country and AS number; - the sidecar reads its `asn`, `as_name` and `country_code` fields. The - operator downloads it with a free IPinfo account and mounts it read-only. - The sidecar never fetches it and holds no account or token for it. +- Lookup of AS number and country (R7). Off by default: the file needs an + account only the operator can hold, and GeoJS is told every visitor's address. + - `LOOKUP_SOURCE` (default `off`): `off`, `file` or `geojs`. + - `LOOKUP_DB_PATH`: the file for `file`, the IPinfo Lite database in its + `.mmdb` form (`ipinfo_lite.mmdb`), one file that carries both country and + AS number; the sidecar reads its `asn`, `as_name` and `country_code` + fields. The operator downloads it with a free IPinfo account and mounts it + read-only. The sidecar never fetches it and holds no account or token for + it. A file that is missing or unreadable at start stops the start. - Refreshing the file is the operator's business; IPinfo updates it daily. The sidecar notices when the file is replaced and reads it again; a - replacement it cannot read is ignored, logged and alerted once, and the - file it has stays in use. Mount the directory that holds the file rather - than the file itself: docker does not show a single mounted file being - replaced on the host. - - Unset by default, which means no lookups. A setting that needs them (the - biased thresholds below, `ADD_LOOKUP_HEADERS`, the per-AS-number anomaly - thresholds) given without `LOOKUP_DB_PATH` stops the start with a message - naming both. A configured file that is missing or unreadable at start - stops the start too. + replacement it cannot read is ignored, logged and sent as one `file_error` + alert, and the file it has stays in use. Mount the directory that holds + the file rather than the file itself: docker does not show a single + mounted file being replaced on the host. + - `geojs` asks the free GeoJS web service, which needs no account or key, + about several addresses in one request + (`https://get.geojs.io/v1/ip/geo.json?ip=a,b,c`), and reads each answer's + `country_code`, `asn` and `organization_name` (the AS name). A + `country_code` of `null`, which GeoJS gives for some ranges, and an `asn` + of `64512`, which it gives when it knows none, count as unknown. GeoJS is + told the address of every new visitor. + - Each GeoJS answer is kept for 7 days with the client's record, in memory + and in `clients.json`, so it survives a restart; after 7 days the client's + next request asks again. A client dropped from the table loses its answer. + - `LOOKUP_TIMEOUT` (default `1s`): how long a client with no kept answer + waits for one; GeoJS normally answers in a fraction of that. At most one + request to GeoJS is under way at a time, the addresses that arrive + meanwhile are asked about together in the next one, and a request that + takes longer than `LOOKUP_TIMEOUT` is abandoned. A client whose answer has + not come in time counts as unknown until it comes: its later requests do + not wait, and its address is asked about again in the background. + - GeoJS publishes no rate limit, but its terms forbid "an excessive amount + of API requests", judged by GeoJS alone, and let it block a caller. While + GeoJS is slow, down or refusing the sidecar, clients with a kept answer + are unaffected and new clients count as unknown, so + `EXCLUSIVELY_ALLOWED_COUNTRIES`, when set, refuses them. The sidecar keeps + asking with backoff and sends one `source_failure` alert per cooldown. A + service that cannot accept this uses the file. + - One source at a time: `LOOKUP_SOURCE=file` without `LOOKUP_DB_PATH`, or + `LOOKUP_DB_PATH` with any other `LOOKUP_SOURCE`, stops the start with a + message naming both. So does a setting that needs lookups while + `LOOKUP_SOURCE` is `off`: the country lists and the biased thresholds + below, `ADD_LOOKUP_HEADERS`, and the per-AS-number anomaly thresholds. - `ADD_LOOKUP_HEADERS` (default `false`): pass `X-Client-ASN` and `X-Client-Country` to the app. -- Biased thresholds (R8). The lists are empty by default: they need the lookup - database, which only the operator can supply. +- Country lists. Both are empty by default, since they need a lookup source. + Clients in `ALLOW_NETS` are not checked. + - `DENIED_COUNTRIES`: for example `cn,ru,kp,ir,ua,by`. Every request from a + listed country is refused. + - `EXCLUSIVELY_ALLOWED_COUNTRIES`: for example `us,de`. Every request from + any other country is refused, and so is every request from a client the + lookup cannot place, such as an address missing from the database or one + GeoJS has not answered for in time. A list that let those through would + let every new client in whenever GeoJS stops answering. + - Both may be set. `DENIED_COUNTRIES` then adds nothing, since the exclusive + list already refuses every other country, and a code on both lists stops + the start with a message naming it. + - A refused request is answered with `BAN_RESPONSE`, as a banned client is, + as soon as the client's address has been looked up, before its body is + read. It skips the reputation checks, the limits, the rule files and the + Core Rule Set, is logged with the action `country_denied`, and is counted + in the metrics. It is a refusal, not a ban: it is not an offence and makes + no ban record. +- Biased thresholds (R8). The lists are empty by default: they need a lookup + source, which only the operator can choose. - `ASN_LIMIT_PERCENT`: for example `AS14061:50,AS16276:50,AS45102:25`. Clients in a listed AS number get that percentage of every request and byte limit, so the rules in "Bans" ban them after fewer requests than @@ -370,8 +454,9 @@ The settings, by group: `CN:25,RU:50`. Each client from a listed country gets that percentage of the normal per-client limits. - No budget is shared by a whole country or AS number: one abuser could use - it up and lock out everyone else there, the denial of service this tool - exists to prevent. + it up and lock out everyone else there, a denial of service nobody chose + and the one this tool exists to prevent. Refusing a whole country is left + to the operator's choice, through the country lists. - When more than one percentage applies to a client (its AS number, its country, a reputation hit), the lowest applies. - `ASN_BYTES_PERCENT`, `COUNTRY_BYTES_PERCENT`: optional overrides applied @@ -406,10 +491,13 @@ The settings, by group: example `/wp-login.php,/xmlrpc.php` in front of gitea. A request for one is a clear sign of attack. This is the env-var short form of a `path` rule with the `ban` action, for deployments that mount no rule files. - - `ERROR_BURST_THRESHOLD` (default `30`): responses with status 401, 403 or - 404 per client per minute; more than this breaks a limit (see "Bans"). + - `ERROR_BURST_THRESHOLD` (default `30`): responses with status 403 or 404 + per client per minute; more than this breaks a limit (see "Bans"). Scanners walking lists of paths cause hundreds a minute; a person rarely - causes more than a few. + causes more than a few. 401 is not counted: git and container registry + clients send their first request without credentials and are answered 401 + at the start of every push, every fetch from a private repository and + every image pull. - Bans (R5), following the rules under "Bans" - `ATTACK_BAN_DURATION` (default `7d`): the ban for a first clear sign of attack. @@ -455,7 +543,10 @@ The settings, by group: - `CROWDSEC_LAPI_URL`, `CROWDSEC_LAPI_KEY`: optional. If a CrowdSec engine exists on the host, pull its decision list on a schedule and treat listed addresses as banned. This is how a fleet-wide blocklist can arrive without - the sidecar depending on CrowdSec. + the sidecar depending on CrowdSec. The list is kept like a fetched + blocklist, and a listed client's request makes a ban with the cause + `crowdsec` that lasts as long as CrowdSec's decision, so a long list does + not fill `bans.json`. - `REPUTATION_ACTION` (default `limit:25`): `deny`, `limit:`, or `log`, for DNSBL and API hits. Such verdicts are less certain than a blocklist, so by default a listed client gets a quarter of every limit and @@ -469,8 +560,11 @@ The settings, by group: - `ALERT_NTFY_URL` (full topic URL), `ALERT_NTFY_TOKEN`: title, priority and tags set from the event type. - `ALERT_EVENTS` (default - `ban,permanent_ban,waf_block,anomaly,reputation_hit,source_failure`): - which event types are sent. + `ban,permanent_ban,waf_block,anomaly,reputation_hit,source_failure,file_error`): + which event types are sent. `source_failure` is a reputation source or + GeoJS failing or refusing the sidecar. `file_error` is a rule file or + state file edited while running that does not parse, a replacement lookup + database that cannot be read, or a state file that cannot be written. - `ALERT_COOLDOWN` (default `15m`): the same event type for the same client or netblock is not repeated within this time; a count of suppressed repeats is included in the next one. @@ -486,7 +580,7 @@ The settings, by group: - Per surrounding netblock (`ANOMALY_NET_V4_PREFIX` default `24`, `ANOMALY_NET_V6_PREFIX` default `48`): `ANOMALY_NET_REQUESTS_PER_MINUTE`, `..._PER_HOUR`, `ANOMALY_NET_BYTES_PER_MINUTE`, `..._PER_HOUR`. - - Per AS number, which needs the lookup database: + - Per AS number, which needs a lookup source: `ANOMALY_ASN_REQUESTS_PER_MINUTE`, `..._PER_HOUR`, `ANOMALY_ASN_BYTES_PER_MINUTE`, `..._PER_HOUR`. - Whole service: `ANOMALY_TOTAL_REQUESTS_PER_MINUTE`, `..._PER_HOUR`, @@ -533,7 +627,9 @@ removed there takes effect while the sidecar runs. - `ban`: the request is a clear sign of attack. Refuse it and ban the client's netblock for seven days (`ATTACK_BAN_DURATION`); any further request during those days, or a later clear sign of attack, makes the ban - permanent (see "Bans"). Use it only for requests no real visitor sends. + permanent (see "Bans"). Use it only for requests no real visitor sends, + and anchor a path at the site root (`^/`): a file of the same name deeper + in a site can be ordinary content, such as a file in a gitea repository. - `regex`: Go regular expression syntax (RE2). It has no backreferences or lookaround, and in exchange matching time is linear in the input, so no rule can be made to stall the proxy. `(?i)` at the front makes a rule @@ -545,9 +641,9 @@ removed there takes effect while the sidecar runs. line that does not parse, a regex that does not compile, or a duplicate id stops the process with a message naming the file and line. While running, the same faults leave the rules as they were, including the earlier version of - that file, and the log and one alert name the file and line; once the file is - fixed, it is read again. A missing or empty directory is not an error: the log - says that no rules were loaded. + that file, and the log and one `file_error` alert name the file and line; once + the file is fixed, it is read again. A missing or empty directory is not an + error: the log says that no rules were loaded. - The image ships a default file in `RULES_DIR`. Mounting a directory over it replaces the defaults; mounting single files into it adds to them. Docker does not show a single mounted file being replaced on the host, which is how many @@ -564,16 +660,19 @@ Example file: # 00-default.rules: probes no real visitor sends # id target action regex -env-file path ban (?i)/\.env(\.[a-z]+)?$ +env-file path ban (?i)^/\.env(\.[a-z]+)?$ git-dir path ban ^/\.git/(config|HEAD|index)$ -php-shell path ban (?i)/(shell|c99|r57|wso|alfa)\.php$ +php-shell path ban (?i)^/(shell|c99|r57|wso|alfa)\.php$ scanner-agent user_agent ban (?i)\b(sqlmap|nikto|nuclei|masscan|zgrab|wpscan)\b path-traversal uri block (\.\./){2,} empty-agent user_agent log ^$ ``` The image ships one default file of this kind. It is kept short and limited to -patterns that are wrong for every app. Anything app-specific belongs in a file +patterns that are wrong for every app, and its path rules are anchored at the +site root: in front of gitea, `/.env` is a probe, while +`///src/branch/main/.env.example` is a file in a repository that +any visitor or search crawler may open. Anything app-specific belongs in a file the deployer mounts: a request for `/wp-login.php`, for example, is a clear sign of attack in front of gitea and an ordinary login in front of WordPress. @@ -585,57 +684,79 @@ wp-probe path ban (?i)^/(wp-login\.php|xmlrpc\.php|wp-admin/) ## Persistent state All state lives in memory, and the files in `STATE_DIR` hold a copy of all of -it, so an orderly stop loses nothing. No database is used, and nothing is read -from disk while serving a request. The files are meant for people as well: an -admin can read or edit them at any time, and the running sidecar takes the edit -in. +it, so an orderly stop loses nothing. The one exception is log lines still +waiting to be sent to `LOG_REMOTE_URL`, which stdout has already carried. No +database is used, and nothing is read from disk while serving a request. The +files are meant for people as well: an admin can read or edit them at any time, +and the running sidecar takes the edit in. - Files, each holding one kind of state: - - `bans.json`: every active and past ban. Per entry: the netblock, start, - expiry (`null` for permanent), what caused it (`attack`, `limit`, `admin` - or `crowdsec`), a short reason, when it was lifted if an admin lifted it, - and a `notes` field (below). + - `bans.json`: active and past bans, up to `MAX_BANS`. Per entry: the + netblock, start, expiry (`null` for permanent), what caused it (`attack`, + `limit`, `admin` or `crowdsec`), a short reason, when it was lifted if an + admin lifted it, and a `notes` field (below). - `clients.json`: per client, its minute, hour and day counters and its history since it was first seen: first and last time seen, the AS number, - AS name and country last looked up, total requests and bytes in each - direction, how many requests were forwarded and how many refused, - responses by status class, and offences by kind. Clients that were never - banned are kept too, so every client's history survives a restart; - `GET /clients/` shows it. - - `reputation.json`: the last good copy of each list fetched from a URL, and + AS name and country last looked up and when (with GeoJS, the answer kept + for 7 days), total requests and bytes in each direction, how many requests + were forwarded and how many refused, responses by status class, and + offences by kind. Clients that were never banned are kept too, so every + client's history survives a restart; `GET /clients/` shows it. + - `reputation.json`: the last good copy of each list fetched from a URL, cached DNSBL and reputation API verdicts, each with the time it was - fetched. + fetched, and the AbuseIPDB checks spent today, so a restart does not reset + the daily budget. + - `alerts.json`: the anomaly counters for surrounding netblocks, AS numbers, + named netblocks and the whole service; for each event type and client or + netblock, when it was last sent and how many repeats the cooldown has held + back since; the alerts sent this hour; and alerts still waiting to be + sent. - Ban notes. The notes on a ban hold what an admin needs to decide whether to lift it, drawn from what the sidecar already knows; nothing is looked up to fill them: - - the AS number, AS name and country (empty without a lookup database); + - the AS number, AS name and country (empty when lookups are off); - what was broken: the rule ids and target that matched, or the limit, its window, the count reached and the client's limit percentage with what set it; and any reputation sources that listed the client; - the requests that caused the ban, up to the last ten: time, method, host, - path, status and user agent; + path with its query string, status and user agent, each text cut to 256 + bytes; - how many requests counted toward the ban, and the time span over which they came; - the netblock's total requests since it was first seen, and the requests refused under this ban so far, kept up to date while the ban lasts; - how many earlier bans of each kind the netblock has had. -- Size: each tracked client takes about 2 KiB in `clients.json`, so at the - default `MAX_TRACKED_CLIENTS` of 50,000 the file can reach about 100 MiB; a - small service usually tracks a few thousand clients and a few MiB. The other - files stay small. Writing a whole file of that size takes about a second, in - the background, so rewriting whole files still needs nothing cleverer. +- Size and disk writes. Each file is rewritten whole, so each has a bound: + - `clients.json` takes about 1 KiB per client. Clients are dropped only when + the table is full, so on a public service the file grows to the default + `MAX_TRACKED_CLIENTS` of 20,000, about 20 MiB. Written every 15 minutes, + that is under 2 GiB of disk writes a day. + - `bans.json` takes about 2 KiB per ban and at most about 8 KiB, since the + texts in the notes are cut short. At the default `MAX_BANS` of 5,000 it is + about 10 MiB, and never more than about 40 MiB. It is written when a ban + is made, lifted or made permanent, at most once every 10 seconds, and + otherwise with the 15-minute write, so its writes follow the bans made: + with a full file, a hundred new bans a day come to about 1 GiB of disk + writes. + - `reputation.json` and `alerts.json` are usually a few MiB or less. + - Writing a file of these sizes takes well under a second on an ordinary + disk, in the background, so rewriting whole files needs nothing cleverer. - Format: indented JSON with a top-level `version` number, entries sorted by client address, times in RFC 3339 UTC, durations and sizes as plain numbers with the unit in the field name. The aim is that a person can open `bans.json` - in an editor, find an address, and remove or add an entry. + in an editor, find an address, and remove or add an entry. `clients.json` puts + each client on one line instead, which halves its size and lets `grep` show + everything about one client. - Writing: - serialise from a snapshot taken under the lock, so requests are not held up while the file is written; - write to a temporary file in the same directory, sync it, rename it over the real name, sync the directory. A crash at any point leaves either the old complete file or the new complete file, never a partial one; - - `bans.json` is written `STATE_WRITE_DELAY` after a change; `clients.json` - and `reputation.json` every `STATE_COUNTER_INTERVAL`; all three on orderly + - `bans.json` is written `STATE_WRITE_DELAY` after a ban is made, lifted or + made permanent. Changes that only update the counts in its notes wait for + the write every `STATE_COUNTER_INTERVAL`, as all of `clients.json`, + `reputation.json` and `alerts.json` do. All four are written on orderly shutdown (SIGTERM); - before writing a file, the sidecar checks whether it changed on disk since the sidecar last read or wrote it; if it did, the sidecar takes that edit @@ -662,14 +783,17 @@ in. - An edit that does not parse does not stop the running sidecar. It keeps the state it has, renames the edited file to `.bad` (for example `bans.json.bad`) so the edit is kept, writes the file again from memory, - and logs and alerts once with the file and the position of the error. The - admin fixes the `.bad` file and moves it back. + logs the file and the position of the error, and sends them in one + `file_error` alert. The admin fixes the `.bad` file and moves it back. - `STATE_DIR` not writable at start: the process exits. A write that fails while running: state stays correct in memory, the failure is logged, counted in - metrics, alerted once per cooldown, and retried at the next write. -- What a hard kill can lose: ban changes from the last `STATE_WRITE_DELAY`, and - counts, history and reputation from the last `STATE_COUNTER_INTERVAL`. Losing - the volume loses ban history, not service. + metrics, sent as a `file_error` alert once per cooldown, and retried at the + next write. +- What a hard kill can lose: ban changes from the last `STATE_WRITE_DELAY` (10 + seconds), and everything else from the last `STATE_COUNTER_INTERVAL` (15 + minutes): counts and history, the counts in ban notes, GeoJS answers, + reputation verdicts and the AbuseIPDB count, anomaly counters and alert + cooldowns. Losing the volume loses ban history, not service. ## Request log @@ -691,13 +815,13 @@ handling while `docker logs` keeps working. - Response detail: `response_content_type`, `upstream_status` (differs from `status` when the sidecar answered itself), `cache_control`, `location` on redirects, `aborted` when the client went away early. -- Decision: `action` (`forward`, `banned`, `denied`, `rate_limited`, - `rule_blocked`, `waf_blocked`, `too_large`, `timed_out`, `upstream_error`), - `would_action` in `observe` mode, `limit_percent` and which rule set it, - `counts` (the client's minute, hour and day request and byte totals after this - request), `limit_hit` (which window), `rule_ids` (rule file rules that - matched), `waf_rule_ids`, `waf_score`, `reputation` (sources that listed the - client), `offence` when one was recorded, `ban_expires`. +- Decision: `action` (`forward`, `banned`, `denied`, `country_denied`, + `rate_limited`, `rule_blocked`, `waf_blocked`, `too_large`, `timed_out`, + `upstream_error`), `would_action` in `observe` mode, `limit_percent` and which + rule set it, `counts` (the client's minute, hour and day request and byte + totals after this request), `limit_hit` (which window), `rule_ids` (rule file + rules that matched), `waf_rule_ids`, `waf_score`, `reputation` (sources that + listed the client), `offence` when one was recorded, `ban_expires`. - Timings in milliseconds: `duration_total`, `duration_checks` (everything the sidecar did before forwarding), `duration_waf`, `duration_upstream_connect`, `duration_upstream_first_byte`, `duration_upstream_total`. @@ -724,15 +848,16 @@ it can be reached by a scraper without exposing ban management. duration and upstream duration histograms; requests in flight. - Limits and bans: limit hits by window and kind (requests or bytes), size and time limit hits by limit, offences by kind, bans created by cause, permanent - bans, active bans (gauge). + bans, active bans (gauge); requests refused by the country lists, by country. - Attack detection: rule file matches by rule id and action, and the number of rules loaded; Core Rule Set matches by mode and rule id (label limited to the rules that actually fired). - Lookup and reputation: requests and bytes by AS number and by country, limited to the `METRICS_TOP_N` (default `50`) busiest of each with the rest summed as `other`, so the label set stays bounded; reputation queries, hits, failures - and remaining daily budget by source; age of each blocklist and lookup - database. + and remaining daily budget by source; GeoJS requests and failures, and clients + that counted as unknown because it did not answer in time; age of each + blocklist and lookup database. - Housekeeping: tracked clients (gauge), state file writes, write failures, last successful write time and size per file; files read again after an edit, and edits set aside because they did not parse; alerts sent, failed and suppressed @@ -759,7 +884,8 @@ One JSON object per alert: - `client`, `netblock`, `asn`, `as_name`, `country` - `reason`: short human-readable sentence - `detail`: event-specific fields, for example `window`, `count`, `limit`, - `limit_percent`, `rule_ids`, `path`, `ban_expires`, and for a ban its notes + `limit_percent`, `rule_ids`, `path`, `ban_expires`, for a ban its notes, and + for `file_error` the file and the position of the error - `suppressed_repeats`: number of identical alerts held back by the cooldown ## Deployment as a sidecar @@ -808,8 +934,9 @@ networks: - Traefik routes only port 8080. The metrics port is reached by the scraper over a docker network it shares with the sidecar; the admin port stays on loopback inside the container and is used through `docker exec`. -- The state volume holds the JSON state files, a few MiB for a small service; it - needs no backup beyond whatever the host already does. +- The state volume holds the JSON state files, a few tens of MiB at most with + the defaults (see "Persistent state"); it needs no backup beyond whatever the + host already does. - SSH access to gitea does not pass through the sidecar and is not protected by it. - Rollout per service: point traefik at the sidecar with only `UPSTREAM_URL` @@ -819,8 +946,8 @@ networks: (`WAF_DISABLED_RULES`, `WAF_EXEMPT_PATHS`, `RATE_LIMIT_EXEMPT_PATHS`) or an exemption (`RATE_LIMIT_EXEMPT_NETS`, `ALLOW_NETS`), and its ban is lifted in `bans.json`. Additions to consider at any time: an alert destination, a lookup - database with biased thresholds, reputation sources, a remote log endpoint, - and an app-specific rule file. + source with country lists or biased thresholds, reputation sources, a remote + log endpoint, and an app-specific rule file. - Notes specific to gitea: - A clone is a response, and fits the defaults of 30 minutes and 5 GB for all but the largest repositories and slowest links. A push is a request: @@ -841,7 +968,11 @@ networks: - Lookup database: a configured file that is missing or unreadable at start stops the start. A replacement that cannot be read while running is ignored: - the file already loaded stays in use, the problem is logged, one alert. + the file already loaded stays in use, the problem is logged, one `file_error` + alert. +- GeoJS slow, down or refusing the sidecar: clients with a kept answer are + unaffected, new clients count as unknown, the sidecar keeps asking with + backoff, one `source_failure` alert per cooldown. - Reputation source down or over quota: no verdict, service continues, one `source_failure` alert per cooldown. - Alert destination down: retried with backoff from a bounded queue, oldest @@ -852,16 +983,18 @@ networks: - Remote log endpoint down: stdout continues, lines are buffered then dropped oldest first, drops counted in metrics. - State file write fails while running: memory stays authoritative, logged, - counted, alerted, retried. + counted, one `file_error` alert per cooldown, retried. - A state file or rule file edited while running that does not parse: the sidecar keeps running on what it has. The state file is set aside as `.bad` and written again from memory; the rule file is left as it is. - Logged, one alert. + Logged, one `file_error` alert. - In short: the only things that stop the process happen at start: invalid configuration, a configured lookup database that is missing or unreadable, a rule file or state file that does not parse, and an unwritable `STATE_DIR`. Once running, a broken helper or a broken edit never takes the protected - service down. + service down. The nearest it comes is GeoJS failing while + `EXCLUSIVELY_ALLOWED_COUNTRIES` is set: new clients then cannot be placed, and + that list refuses them, as it refuses every client it cannot place. ## Risks the design has to handle @@ -875,8 +1008,11 @@ networks: busy person produces, `RATE_LIMIT_EXEMPT_NETS` and `ALLOW_NETS` take known shared addresses out, and the ban notes show an admin what happened before a ban is lifted. -- A `ban` rule that matches a real visitor bans for seven days, so the default - rule file keeps to requests no real visitor sends. +- A `ban` rule that matches a real visitor bans it for seven days, and the + visitor's next request during those days makes the ban permanent. So the + default rule file keeps to requests no real visitor sends, with its paths + anchored at the site root, where no app serves `.env` files or web shells. A + visitor banned by mistake is let back in by lifting the ban in `bans.json`. - IPv6 address rotation inside a /64: handled by grouping. - Widely distributed scrapers using thousands of addresses at low rates each: per-client limits do not see them. The AS number and netblock anomaly alerts @@ -887,9 +1023,15 @@ networks: - Core Rule Set false positives against real apps (gitea's editor, API payloads): a match refuses only that request and bans no one by itself; the request log names the rule, and exclusions by rule id and path fix it. +- GeoJS as the lookup source: every new visitor's address goes to a third party, + and a swarm of fresh addresses, when lookups peak, is when GeoJS may slow down + or block the sidecar. Keeping answers for 7 days and asking about many + addresses in one request keep the number of requests low; the file source has + neither risk. - Slow-request attacks: `CLIENT_REQUEST_TIMEOUT` bounds how long a request may - take to arrive, and an idle timeout on the listener closes connections that - send nothing. + take to arrive, `CLIENT_REQUEST_HEADER_MAX_BYTES` how large its headers may + be, and `CLIENT_IDLE_TIMEOUT` closes a kept-open connection that sends nothing + more. - Alert floods: cooldown and hourly cap. ## Build order @@ -900,8 +1042,8 @@ networks: `observe` mode, the full request log on stdout, the metrics endpoint, health. - Second: rule files, admin endpoints, alerting to all three destinations, remote log sending. -- Third: AS number and country lookup, biased thresholds, byte limits, anomaly - thresholds. +- Third: AS number and country lookup from the file or GeoJS, the country lists, + biased thresholds, byte limits, anomaly thresholds. - Fourth: blocklists, DNSBL, AbuseIPDB, optional CrowdSec decision feed. - Fifth: attack detection with Coraza and the Core Rule Set, trap paths, error bursts.