README.md

@@ -52,7 +52,7 @@ goes through the candidates one by one.
   other setting has a default chosen for a service facing the internet in 2026.
 - Real client address worked out from `X-Forwarded-For`, trusting only the proxy
   networks you list, by default the private address ranges. IPv6 clients are
-  counted by /64.
+  counted by /64 by default.
 - Size and time limits on requests and responses, both between the client and
   `smallwebwaf` and between `smallwebwaf` and the app: by default a request may
   take 60 seconds and 100 MB, a response 30 minutes and 5 GB.
@@ -61,8 +61,14 @@ goes through the candidates one by one.
   real visitors need.
 - Netblocks that bypass rate limiting, netblocks that bypass everything, and
   netblocks that are always refused.
-- AS number and country lookup for every client from the IPinfo Lite database
+- AS number and country lookup for every client, off until you choose a source:
-  file, which you download and mount (see "Lookup database" below).
+  the IPinfo Lite database file, which you download and mount, or the free GeoJS
+  web service (see "Country and AS number lookup" below).
+- Country lists: `DENIED_COUNTRIES` refuses every request from the countries
+  listed, `EXCLUSIVELY_ALLOWED_COUNTRIES` every request from anywhere else. Such
+  a request gets the answer a banned client gets as soon as the client's address
+  has been looked up, before its body is read and without the rule files or the
+  Core Rule Set looking at it, and no ban is made.
 - Biased limits: listed AS numbers and countries get a percentage of every
   limit, for example 50 percent for common abuse-source networks, so their
   clients are banned after fewer requests. Zero percent is a zero allowance: the
@@ -96,11 +102,11 @@ goes through the candidates one by one.
   fields, the decision taken and why, AS number and country, and timings.
   Optionally also sent to a remote syslog server.
 - Prometheus metrics on their own port.
-- State (bans with their notes, each client's counters and history, the
+- State (bans with their notes, each client's counters, history and lookup
-  reputation cache) held in memory and kept in readable JSON files that always
+  answer, the reputation cache, the alerting state) held in memory and kept in
-  hold a copy of it, so a restart loses nothing. Edit a file, or add a rule
+  readable JSON files, written regularly and at every stop, so a restart loses
-  file, and the running `smallwebwaf` picks up the change. Nothing is read from
+  nothing. Edit a file, or add a rule file, and the running `smallwebwaf` picks
-  disk while serving a request.
+  up the change. Nothing is read from disk while serving a request.
 - A small admin endpoint for health checks, listing, adding and lifting bans,
   and asking why a given address was refused.
 
@@ -115,7 +121,9 @@ For each request `smallwebwaf`:
 - works out who the client really is;
 - lets it straight through if it is on the bypass list, refuses it if it is on
   the deny list or currently banned;
-- looks up its AS number and country, and any cached reputation verdict;
+- looks up its AS number and country, and refuses it if that country is denied,
+  or is not among the only ones allowed;
+- checks for a cached reputation verdict;
 - picks the client's limit percentage from those;
 - checks the minute, hour and day request counters against the limits, and bans
   the client if it breaks one;
@@ -162,7 +170,7 @@ A rule file is one rule per line: a name, what to match against, what to do, and
 a regex.
 
 ```
-env-file       path        ban  (?i)/\.env(\.[a-z]+)?$
+env-file       path        ban  (?i)^/\.env(\.[a-z]+)?$
 scanner-agent  user_agent  ban  (?i)\b(sqlmap|nikto|nuclei|wpscan)\b
 ```
 
@@ -170,19 +178,31 @@ scanner-agent  user_agent  ban  (?i)\b(sqlmap|nikto|nuclei|wpscan)\b
 rules, the rule file format, the state files, the log fields, the metrics,
 failure behaviour and the build order.
 
-## Lookup database
+## Country and AS number lookup
 
-AS number and country lookups, and the biased limits that use them, need the
+AS number and country lookups, and the country lists and biased limits that use
-free IPinfo Lite database (`ipinfo_lite.mmdb`). You download it with your own
+them, are off until you choose one of two sources with `LOOKUP_SOURCE`.
-IPinfo account, mount it into the container, point `LOOKUP_DB_PATH` at it and
+
-refresh it when you choose; `smallwebwaf` never downloads it itself. IPinfo
+`LOOKUP_SOURCE=file` reads the free IPinfo Lite database (`ipinfo_lite.mmdb`).
-releases it under the Creative Commons Attribution-ShareAlike 4.0 International
+You download it with your own IPinfo account, mount the directory that holds it
-License and asks for attribution, in its own words on https://ipinfo.io/lite:
+into the container, point `LOOKUP_DB_PATH` at the file and refresh it when you
-"The attribution requirements can be met by giving our service credit as your
+choose; `smallwebwaf` never downloads it itself, and reads it again when you
-data source. Simply place a link to IPinfo on the website, application, or
+replace it. It has to be the directory rather than the file itself: docker does
-social media account that uses our data." Its example of such a credit is a link
+not show a single mounted file being replaced, so a refresh would go unseen.
-mentioning "IP address data is powered by IPinfo". A service that uses the
+IPinfo releases it under the Creative Commons Attribution-ShareAlike 4.0
-database through `smallwebwaf` should carry that link.
+International License and asks for attribution, in its own words on
+https://ipinfo.io/lite: "The attribution requirements can be met by giving our
+service credit as your data source. Simply place a link to IPinfo on the
+website, application, or social media account that uses our data." Its example
+of such a credit is a link mentioning "IP address data is powered by IPinfo". A
+service that uses the database through `smallwebwaf` should carry that link.
+
+`LOOKUP_SOURCE=geojs` asks the free GeoJS web service instead, with no account
+and no file. Every new visitor's address is sent to GeoJS. Each answer is kept
+for seven days, across restarts, and many addresses are asked about in one
+request. GeoJS publishes no rate limit but may block a caller it thinks asks too
+much; while it is not answering, new visitors count as coming from an unknown
+country, which `EXCLUSIVELY_ALLOWED_COUNTRIES` refuses.
 
 ## Documents
 

SPEC.md

@@ -12,8 +12,8 @@ forwards to the application. The sidecar limits request and byte rates per
 client, bounds the size and duration of every request and response, detects
 attacks, bans abusers (briefly at first, for seven days on a clear sign of
 attack, permanently when they keep at it), consults IP reputation sources, looks
-up the AS number and country of each client, lowers its limits for listed AS
+up the AS number and country of each client, can refuse whole countries, lowers
-numbers and countries, and sends alerts.
+its limits for listed AS numbers and countries, and sends alerts.
 
 It is meant to protect an app on the open internet from the first request with
 one setting, `UPSTREAM_URL`: every other setting has a default chosen for a
@@ -52,8 +52,9 @@ directory of hand-editable text files.
     - Client identification: works out the real client IP from
       `X-Forwarded-For`, trusting only the proxy netblocks in `TRUSTED_PROXIES`,
       by default the private address ranges.
-    - Lookup: AS number and country from one database file the operator
+    - Lookup: AS number and country, from one of two sources the operator
-      supplies, held in memory.
+      chooses: a database file the operator supplies, held in memory, or the
+      GeoJS web service, whose answers are kept for seven days.
     - Reputation: static blocklists fetched on a schedule; DNSBL and reputation
       API queries made in the background and cached.
     - Counters: per-client request and byte counts per minute, hour and day,
@@ -104,7 +105,7 @@ whole exchange.
       Otherwise use the TCP peer address and ignore the header.
     - IPv6 clients are grouped by prefix (`IPV6_GROUP_PREFIX`, default 64) for
       counting and banning, because one abuser usually controls a whole /64. A
-      client is therefore one IPv4 address or one IPv6 /64.
+      client is therefore one IPv4 address or one IPv6 group, a /64 by default.
 - Static lists.
     - In `ALLOW_NETS`: skip every check below and forward. Still counted for
       anomaly alerts.
@@ -112,8 +113,17 @@ whole exchange.
 - Ban ledger. An active ban on the client's netblock: refuse with
   `BAN_RESPONSE`. If a clear sign of attack caused the ban, the ban becomes
   permanent (see "Bans").
-- Lookup AS number and country, when a lookup database is configured. Unknown is
+- Look up the AS number and country, unless `LOOKUP_SOURCE` is `off`. With
-  a valid result and changes nothing.
+  GeoJS, a client whose answer is not yet kept waits for it, up to
+  `LOOKUP_TIMEOUT`. A client the lookup cannot place has an unknown country and
+  AS number: the exclusive country list refuses it, the biased thresholds give
+  it `UNKNOWN_LIMIT_PERCENT`, and nothing else treats it differently.
+- Country lists. A client whose country is in `DENIED_COUNTRIES`, or, when
+  `EXCLUSIVELY_ALLOWED_COUNTRIES` is set, is not in it, is refused with
+  `BAN_RESPONSE`. Every step up to here needs only the client's address, so the
+  request body has not been read yet, and the refusal skips everything below,
+  the rule files and the Core Rule Set included. It is counted in the metrics,
+  but it is not an offence and makes no ban.
 - Reputation.
     - Address inside a fetched blocklist: apply `BLOCKLIST_ACTION`.
     - Cached DNSBL or reputation API result: apply `REPUTATION_ACTION`.
@@ -139,10 +149,10 @@ whole exchange.
     - Add response bytes (and request body bytes) to the byte counters. A client
       whose byte total passes a byte limit times the percentage has broken that
       limit and is banned. The response in progress is not cut off.
-    - Count the client's requests answered with 401, 403 or 404, whether the app
+    - Count the client's requests answered with 403 or 404, whether the app gave
-      gave that answer or the sidecar refused the request after a rule file or
+      that answer or the sidecar refused the request after a rule file or Core
-      Core Rule Set match. More than `ERROR_BURST_THRESHOLD` of them within a
+      Rule Set match. More than `ERROR_BURST_THRESHOLD` of them within a minute
-      minute breaks a limit and bans the client.
+      breaks a limit and bans the client.
     - Update the client's history and the anomaly counters, and evaluate alert
       thresholds.
 
@@ -157,17 +167,25 @@ whole exchange.
   shutdown and loaded again at start (see "Persistent state"), so a restart does
   not hand every client a fresh allowance. Buckets whose time has passed are
   discarded on load.
-- Memory is bounded by `MAX_TRACKED_CLIENTS`; when full, the least recently seen
+- Memory is bounded by `MAX_TRACKED_CLIENTS` and `MAX_BANS`. When the table of
-  client is dropped first, with its history and its past bans. Active bans are
+  clients is full, the least recently seen client is dropped first, with its
-  never dropped; one whose client is no longer tracked is dropped when it ends.
+  history and its lookup answer. When `MAX_BANS` bans are held, the ban that has
+  gone longest without a request from its netblock is dropped first, whether it
+  is past, active or permanent (see "Bans").
 
 ## Bans
 
 A ban refuses every request from a netblock, answering with `BAN_RESPONSE`
-(`403` by default), until the ban ends; a permanent ban never ends. The netblock
+(`403` by default), until the ban ends. A permanent ban does not run out: it
-a ban covers is the client's IPv4 address (or the wider IPv4 prefix that
+ends only when an admin lifts it, or when a new ban is made while `MAX_BANS`
-`BAN_SCOPE_V4_PREFIX` sets) or its IPv6 /64, and never more, so a permanent ban
+bans are held and it is the ban that has gone longest without a request. A
-does not reach neighbours who did nothing.
+scanner whose ban was dropped that way is refused and banned again by its next
+probe.
+
+The netblock a ban covers is the client: its IPv4 address, or its IPv6 group of
+`IPV6_GROUP_PREFIX` (a /64 by default). `BAN_SCOPE_V4_PREFIX` can widen an IPv4
+ban to the surrounding netblock. At the defaults a ban covers one address or one
+/64, so a permanent ban does not reach neighbours who did nothing.
 
 An offence is a request the sidecar holds against the client: one that carries a
 clear sign of attack, breaks a limit, or is refused by a rule file or the Core
@@ -206,21 +224,30 @@ Every ban carries notes with what an admin needs to decide whether to lift it
 lifted before it ends is kept, marked as lifted, and does not count toward a
 longer ban; deleting its entry from `bans.json` forgets it entirely.
 
-Clients of listed AS numbers and countries, and clients with a poor reputation,
+Clients of the AS numbers and countries listed in the biased thresholds, and
-get lower limits (see "Biased thresholds"), so the same rules ban them after
+clients listed by a reputation source whose action is `limit:<percent>`, get
-fewer requests. The result is always a ban, first a temporary one and then, for
+lower limits (see "Biased thresholds"), so the same rules ban them after fewer
-repeated abuse, a permanent one.
+requests. For them the result is always a ban, first a temporary one and then,
+for repeated abuse, a permanent one.
+
+Some refusals make no ban at all. `DENY_NETS`, a blocklist or reputation hit
+whose action is `deny`, and the country lists (`DENIED_COUNTRIES`,
+`EXCLUSIVELY_ALLOWED_COUNTRIES`) refuse each request they cover and record
+nothing in `bans.json`.
 
 ## Configuration surface
 
 Conventions: lists are comma separated; netblocks are CIDR (a bare address means
 /32 or /128); durations use Go syntax plus `d` for days (`90s`, `15m`, `24h`,
-`7d`); byte sizes accept `K`, `M`, `G` suffixes.
+`7d`); byte sizes accept `K`, `M`, `G` suffixes. Countries are two-letter ISO
+codes in either case (`de` and `DE` are the same); a code that is not a country
+code, such as `nk` (North Korea is `kp`), stops the start with a message naming
+it.
 
 - Only `UPSTREAM_URL` is required. Every other setting has a default chosen for
-  a service facing the internet in 2026, or stays off until it is given
+  a service facing the internet in 2026, or stays off until the operator
-  something only the operator can supply: an alert destination, an account key,
+  supplies or chooses what it needs: an alert destination, an account key, a
-  the lookup database.
+  lookup source.
 - Any limit or threshold can be switched off with the value `off`.
 - A list set to an empty value is an empty list, and replaces the default.
 - Every variable may instead be given as `NAME_FILE` pointing at a file holding
@@ -254,17 +281,21 @@ The settings, by group:
       address. A list given replaces the default; set but empty, it trusts
       nothing.
     - `IPV6_GROUP_PREFIX` (default `64`).
-    - `MAX_TRACKED_CLIENTS` (default `50000`): clients held in memory and in
+    - `MAX_TRACKED_CLIENTS` (default `20000`): clients held in memory and in
       `clients.json` (see "Persistent state").
+    - `MAX_BANS` (default `5000`): bans held in memory and in `bans.json`, past,
+      active and permanent (see "Bans").
 - Persistent state
     - `STATE_DIR` (default `/data`): the JSON state files. The image declares
       `/data` as a volume, so it is writable even when nothing is mounted; a
       named volume, as in the examples below, keeps the state in a place that
       outlives the container.
-    - `STATE_WRITE_DELAY` (default `2s`): after a ban changes, wait this long
+    - `STATE_WRITE_DELAY` (default `10s`): after a ban is made, lifted or made
-      before writing `bans.json`, so a burst of changes becomes one write.
+      permanent, `bans.json` is written this long later with every change made
-    - `STATE_COUNTER_INTERVAL` (default `60s`): how often `clients.json` and
+      in between, so a burst of changes becomes one write.
-      `reputation.json` are written.
+    - `STATE_COUNTER_INTERVAL` (default `15m`): how often `clients.json`,
+      `reputation.json` and `alerts.json` are written, and `bans.json` when only
+      the counts in its notes changed.
 - Logging
     - The request log on stdout is always on and has no switch.
     - `LOG_LEVEL` (default `info`): for the process's own messages (start-up,
@@ -319,6 +350,14 @@ The settings, by group:
       send its whole request, headers and body.
     - `CLIENT_REQUEST_MAX_BYTES` (default `100M`): the largest request body a
       client may send.
+    - `CLIENT_REQUEST_HEADER_MAX_BYTES` (default `32K`): the largest request
+      line and headers a client may send. Over it, the sidecar answers `431` and
+      closes the connection, and nothing reaches the app.
+    - `CLIENT_IDLE_TIMEOUT` (default `120s`): how long a kept-open connection
+      may wait for its next request before the sidecar closes it. It is longer
+      than the 90 seconds after which traefik, by default, closes a connection
+      it is not using, so traefik closes first and never sends a request on a
+      connection the sidecar is closing.
     - `CLIENT_RESPONSE_TIMEOUT` (default `30m`): how long the sidecar may take
       to deliver one response to the client, from the end of the request to the
       last byte.
@@ -340,27 +379,72 @@ The settings, by group:
       started it can only be cut off, and the connection is closed.
     - A WebSocket connection leaves these limits behind once it is upgraded: it
       stays open until either side closes it.
-- Lookup of AS number and country (R7)
+- Lookup of AS number and country (R7). Off by default: the file needs an
-    - `LOOKUP_DB_PATH`: the IPinfo Lite database in its `.mmdb` form
+  account only the operator can hold, and GeoJS is told every visitor's address.
-      (`ipinfo_lite.mmdb`), one file that carries both country and AS number;
+    - `LOOKUP_SOURCE` (default `off`): `off`, `file` or `geojs`.
-      the sidecar reads its `asn`, `as_name` and `country_code` fields. The
+    - `LOOKUP_DB_PATH`: the file for `file`, the IPinfo Lite database in its
-      operator downloads it with a free IPinfo account and mounts it read-only.
+      `.mmdb` form (`ipinfo_lite.mmdb`), one file that carries both country and
-      The sidecar never fetches it and holds no account or token for it.
+      AS number; the sidecar reads its `asn`, `as_name` and `country_code`
+      fields. The operator downloads it with a free IPinfo account and mounts it
+      read-only. The sidecar never fetches it and holds no account or token for
+      it. A file that is missing or unreadable at start stops the start.
     - Refreshing the file is the operator's business; IPinfo updates it daily.
       The sidecar notices when the file is replaced and reads it again; a
-      replacement it cannot read is ignored, logged and alerted once, and the
+      replacement it cannot read is ignored, logged and sent as one `file_error`
-      file it has stays in use. Mount the directory that holds the file rather
+      alert, and the file it has stays in use. Mount the directory that holds
-      than the file itself: docker does not show a single mounted file being
+      the file rather than the file itself: docker does not show a single
-      replaced on the host.
+      mounted file being replaced on the host.
-    - Unset by default, which means no lookups. A setting that needs them (the
+    - `geojs` asks the free GeoJS web service, which needs no account or key,
-      biased thresholds below, `ADD_LOOKUP_HEADERS`, the per-AS-number anomaly
+      about several addresses in one request
-      thresholds) given without `LOOKUP_DB_PATH` stops the start with a message
+      (`https://get.geojs.io/v1/ip/geo.json?ip=a,b,c`), and reads each answer's
-      naming both. A configured file that is missing or unreadable at start
+      `country_code`, `asn` and `organization_name` (the AS name). A
-      stops the start too.
+      `country_code` of `null`, which GeoJS gives for some ranges, and an `asn`
+      of `64512`, which it gives when it knows none, count as unknown. GeoJS is
+      told the address of every new visitor.
+    - Each GeoJS answer is kept for 7 days with the client's record, in memory
+      and in `clients.json`, so it survives a restart; after 7 days the client's
+      next request asks again. A client dropped from the table loses its answer.
+    - `LOOKUP_TIMEOUT` (default `1s`): how long a client with no kept answer
+      waits for one; GeoJS normally answers in a fraction of that. At most one
+      request to GeoJS is under way at a time, the addresses that arrive
+      meanwhile are asked about together in the next one, and a request that
+      takes longer than `LOOKUP_TIMEOUT` is abandoned. A client whose answer has
+      not come in time counts as unknown until it comes: its later requests do
+      not wait, and its address is asked about again in the background.
+    - GeoJS publishes no rate limit, but its terms forbid "an excessive amount
+      of API requests", judged by GeoJS alone, and let it block a caller. While
+      GeoJS is slow, down or refusing the sidecar, clients with a kept answer
+      are unaffected and new clients count as unknown, so
+      `EXCLUSIVELY_ALLOWED_COUNTRIES`, when set, refuses them. The sidecar keeps
+      asking with backoff and sends one `source_failure` alert per cooldown. A
+      service that cannot accept this uses the file.
+    - One source at a time: `LOOKUP_SOURCE=file` without `LOOKUP_DB_PATH`, or
+      `LOOKUP_DB_PATH` with any other `LOOKUP_SOURCE`, stops the start with a
+      message naming both. So does a setting that needs lookups while
+      `LOOKUP_SOURCE` is `off`: the country lists and the biased thresholds
+      below, `ADD_LOOKUP_HEADERS`, and the per-AS-number anomaly thresholds.
     - `ADD_LOOKUP_HEADERS` (default `false`): pass `X-Client-ASN` and
       `X-Client-Country` to the app.
-- Biased thresholds (R8). The lists are empty by default: they need the lookup
+- Country lists. Both are empty by default, since they need a lookup source.
-  database, which only the operator can supply.
+  Clients in `ALLOW_NETS` are not checked.
+    - `DENIED_COUNTRIES`: for example `cn,ru,kp,ir,ua,by`. Every request from a
+      listed country is refused.
+    - `EXCLUSIVELY_ALLOWED_COUNTRIES`: for example `us,de`. Every request from
+      any other country is refused, and so is every request from a client the
+      lookup cannot place, such as an address missing from the database or one
+      GeoJS has not answered for in time. A list that let those through would
+      let every new client in whenever GeoJS stops answering.
+    - Both may be set. `DENIED_COUNTRIES` then adds nothing, since the exclusive
+      list already refuses every other country, and a code on both lists stops
+      the start with a message naming it.
+    - A refused request is answered with `BAN_RESPONSE`, as a banned client is,
+      as soon as the client's address has been looked up, before its body is
+      read. It skips the reputation checks, the limits, the rule files and the
+      Core Rule Set, is logged with the action `country_denied`, and is counted
+      in the metrics. It is a refusal, not a ban: it is not an offence and makes
+      no ban record.
+- Biased thresholds (R8). The lists are empty by default: they need a lookup
+  source, which only the operator can choose.
     - `ASN_LIMIT_PERCENT`: for example `AS14061:50,AS16276:50,AS45102:25`.
       Clients in a listed AS number get that percentage of every request and
       byte limit, so the rules in "Bans" ban them after fewer requests than
@@ -370,8 +454,9 @@ The settings, by group:
       `CN:25,RU:50`. Each client from a listed country gets that percentage of
       the normal per-client limits.
     - No budget is shared by a whole country or AS number: one abuser could use
-      it up and lock out everyone else there, the denial of service this tool
+      it up and lock out everyone else there, a denial of service nobody chose
-      exists to prevent.
+      and the one this tool exists to prevent. Refusing a whole country is left
+      to the operator's choice, through the country lists.
     - When more than one percentage applies to a client (its AS number, its
       country, a reputation hit), the lowest applies.
     - `ASN_BYTES_PERCENT`, `COUNTRY_BYTES_PERCENT`: optional overrides applied
@@ -406,10 +491,13 @@ The settings, by group:
       example `/wp-login.php,/xmlrpc.php` in front of gitea. A request for one
       is a clear sign of attack. This is the env-var short form of a `path` rule
       with the `ban` action, for deployments that mount no rule files.
-    - `ERROR_BURST_THRESHOLD` (default `30`): responses with status 401, 403 or
+    - `ERROR_BURST_THRESHOLD` (default `30`): responses with status 403 or 404
-      404 per client per minute; more than this breaks a limit (see "Bans").
+      per client per minute; more than this breaks a limit (see "Bans").
       Scanners walking lists of paths cause hundreds a minute; a person rarely
-      causes more than a few.
+      causes more than a few. 401 is not counted: git and container registry
+      clients send their first request without credentials and are answered 401
+      at the start of every push, every fetch from a private repository and
+      every image pull.
 - Bans (R5), following the rules under "Bans"
     - `ATTACK_BAN_DURATION` (default `7d`): the ban for a first clear sign of
       attack.
@@ -455,7 +543,10 @@ The settings, by group:
     - `CROWDSEC_LAPI_URL`, `CROWDSEC_LAPI_KEY`: optional. If a CrowdSec engine
       exists on the host, pull its decision list on a schedule and treat listed
       addresses as banned. This is how a fleet-wide blocklist can arrive without
-      the sidecar depending on CrowdSec.
+      the sidecar depending on CrowdSec. The list is kept like a fetched
+      blocklist, and a listed client's request makes a ban with the cause
+      `crowdsec` that lasts as long as CrowdSec's decision, so a long list does
+      not fill `bans.json`.
     - `REPUTATION_ACTION` (default `limit:25`): `deny`, `limit:<percent>`, or
       `log`, for DNSBL and API hits. Such verdicts are less certain than a
       blocklist, so by default a listed client gets a quarter of every limit and
@@ -469,8 +560,11 @@ The settings, by group:
     - `ALERT_NTFY_URL` (full topic URL), `ALERT_NTFY_TOKEN`: title, priority and
       tags set from the event type.
     - `ALERT_EVENTS` (default
-      `ban,permanent_ban,waf_block,anomaly,reputation_hit,source_failure`):
+      `ban,permanent_ban,waf_block,anomaly,reputation_hit,source_failure,file_error`):
-      which event types are sent.
+      which event types are sent. `source_failure` is a reputation source or
+      GeoJS failing or refusing the sidecar. `file_error` is a rule file or
+      state file edited while running that does not parse, a replacement lookup
+      database that cannot be read, or a state file that cannot be written.
     - `ALERT_COOLDOWN` (default `15m`): the same event type for the same client
       or netblock is not repeated within this time; a count of suppressed
       repeats is included in the next one.
@@ -486,7 +580,7 @@ The settings, by group:
     - Per surrounding netblock (`ANOMALY_NET_V4_PREFIX` default `24`,
       `ANOMALY_NET_V6_PREFIX` default `48`): `ANOMALY_NET_REQUESTS_PER_MINUTE`,
       `..._PER_HOUR`, `ANOMALY_NET_BYTES_PER_MINUTE`, `..._PER_HOUR`.
-    - Per AS number, which needs the lookup database:
+    - Per AS number, which needs a lookup source:
       `ANOMALY_ASN_REQUESTS_PER_MINUTE`, `..._PER_HOUR`,
       `ANOMALY_ASN_BYTES_PER_MINUTE`, `..._PER_HOUR`.
     - Whole service: `ANOMALY_TOTAL_REQUESTS_PER_MINUTE`, `..._PER_HOUR`,
@@ -533,7 +627,9 @@ removed there takes effect while the sidecar runs.
     - `ban`: the request is a clear sign of attack. Refuse it and ban the
       client's netblock for seven days (`ATTACK_BAN_DURATION`); any further
       request during those days, or a later clear sign of attack, makes the ban
-      permanent (see "Bans"). Use it only for requests no real visitor sends.
+      permanent (see "Bans"). Use it only for requests no real visitor sends,
+      and anchor a path at the site root (`^/`): a file of the same name deeper
+      in a site can be ordinary content, such as a file in a gitea repository.
 - `regex`: Go regular expression syntax (RE2). It has no backreferences or
   lookaround, and in exchange matching time is linear in the input, so no rule
   can be made to stall the proxy. `(?i)` at the front makes a rule
@@ -545,9 +641,9 @@ removed there takes effect while the sidecar runs.
   line that does not parse, a regex that does not compile, or a duplicate id
   stops the process with a message naming the file and line. While running, the
   same faults leave the rules as they were, including the earlier version of
-  that file, and the log and one alert name the file and line; once the file is
+  that file, and the log and one `file_error` alert name the file and line; once
-  fixed, it is read again. A missing or empty directory is not an error: the log
+  the file is fixed, it is read again. A missing or empty directory is not an
-  says that no rules were loaded.
+  error: the log says that no rules were loaded.
 - The image ships a default file in `RULES_DIR`. Mounting a directory over it
   replaces the defaults; mounting single files into it adds to them. Docker does
   not show a single mounted file being replaced on the host, which is how many
@@ -564,16 +660,19 @@ Example file:
 # 00-default.rules: probes no real visitor sends
 
 # id             target      action  regex
-env-file         path        ban     (?i)/\.env(\.[a-z]+)?$
+env-file         path        ban     (?i)^/\.env(\.[a-z]+)?$
 git-dir          path        ban     ^/\.git/(config|HEAD|index)$
-php-shell        path        ban     (?i)/(shell|c99|r57|wso|alfa)\.php$
+php-shell        path        ban     (?i)^/(shell|c99|r57|wso|alfa)\.php$
 scanner-agent    user_agent  ban     (?i)\b(sqlmap|nikto|nuclei|masscan|zgrab|wpscan)\b
 path-traversal   uri         block   (\.\./){2,}
 empty-agent      user_agent  log     ^$
 ```
 
 The image ships one default file of this kind. It is kept short and limited to
-patterns that are wrong for every app. Anything app-specific belongs in a file
+patterns that are wrong for every app, and its path rules are anchored at the
+site root: in front of gitea, `/.env` is a probe, while
+`/<owner>/<repo>/src/branch/main/.env.example` is a file in a repository that
+any visitor or search crawler may open. Anything app-specific belongs in a file
 the deployer mounts: a request for `/wp-login.php`, for example, is a clear sign
 of attack in front of gitea and an ordinary login in front of WordPress.
 
@@ -585,57 +684,79 @@ wp-probe         path        ban     (?i)^/(wp-login\.php|xmlrpc\.php|wp-admin/)
 ## Persistent state
 
 All state lives in memory, and the files in `STATE_DIR` hold a copy of all of
-it, so an orderly stop loses nothing. No database is used, and nothing is read
+it, so an orderly stop loses nothing. The one exception is log lines still
-from disk while serving a request. The files are meant for people as well: an
+waiting to be sent to `LOG_REMOTE_URL`, which stdout has already carried. No
-admin can read or edit them at any time, and the running sidecar takes the edit
+database is used, and nothing is read from disk while serving a request. The
-in.
+files are meant for people as well: an admin can read or edit them at any time,
+and the running sidecar takes the edit in.
 
 - Files, each holding one kind of state:
-    - `bans.json`: every active and past ban. Per entry: the netblock, start,
+    - `bans.json`: active and past bans, up to `MAX_BANS`. Per entry: the
-      expiry (`null` for permanent), what caused it (`attack`, `limit`, `admin`
+      netblock, start, expiry (`null` for permanent), what caused it (`attack`,
-      or `crowdsec`), a short reason, when it was lifted if an admin lifted it,
+      `limit`, `admin` or `crowdsec`), a short reason, when it was lifted if an
-      and a `notes` field (below).
+      admin lifted it, and a `notes` field (below).
     - `clients.json`: per client, its minute, hour and day counters and its
       history since it was first seen: first and last time seen, the AS number,
-      AS name and country last looked up, total requests and bytes in each
+      AS name and country last looked up and when (with GeoJS, the answer kept
-      direction, how many requests were forwarded and how many refused,
+      for 7 days), total requests and bytes in each direction, how many requests
-      responses by status class, and offences by kind. Clients that were never
+      were forwarded and how many refused, responses by status class, and
-      banned are kept too, so every client's history survives a restart;
+      offences by kind. Clients that were never banned are kept too, so every
-      `GET /clients/<ip>` shows it.
+      client's history survives a restart; `GET /clients/<ip>` shows it.
-    - `reputation.json`: the last good copy of each list fetched from a URL, and
+    - `reputation.json`: the last good copy of each list fetched from a URL,
       cached DNSBL and reputation API verdicts, each with the time it was
-      fetched.
+      fetched, and the AbuseIPDB checks spent today, so a restart does not reset
+      the daily budget.
+    - `alerts.json`: the anomaly counters for surrounding netblocks, AS numbers,
+      named netblocks and the whole service; for each event type and client or
+      netblock, when it was last sent and how many repeats the cooldown has held
+      back since; the alerts sent this hour; and alerts still waiting to be
+      sent.
 - Ban notes. The notes on a ban hold what an admin needs to decide whether to
   lift it, drawn from what the sidecar already knows; nothing is looked up to
   fill them:
-    - the AS number, AS name and country (empty without a lookup database);
+    - the AS number, AS name and country (empty when lookups are off);
     - what was broken: the rule ids and target that matched, or the limit, its
       window, the count reached and the client's limit percentage with what set
       it; and any reputation sources that listed the client;
     - the requests that caused the ban, up to the last ten: time, method, host,
-      path, status and user agent;
+      path with its query string, status and user agent, each text cut to 256
+      bytes;
     - how many requests counted toward the ban, and the time span over which
       they came;
     - the netblock's total requests since it was first seen, and the requests
       refused under this ban so far, kept up to date while the ban lasts;
     - how many earlier bans of each kind the netblock has had.
-- Size: each tracked client takes about 2 KiB in `clients.json`, so at the
+- Size and disk writes. Each file is rewritten whole, so each has a bound:
-  default `MAX_TRACKED_CLIENTS` of 50,000 the file can reach about 100 MiB; a
+    - `clients.json` takes about 1 KiB per client. Clients are dropped only when
-  small service usually tracks a few thousand clients and a few MiB. The other
+      the table is full, so on a public service the file grows to the default
-  files stay small. Writing a whole file of that size takes about a second, in
+      `MAX_TRACKED_CLIENTS` of 20,000, about 20 MiB. Written every 15 minutes,
-  the background, so rewriting whole files still needs nothing cleverer.
+      that is under 2 GiB of disk writes a day.
+    - `bans.json` takes about 2 KiB per ban and at most about 8 KiB, since the
+      texts in the notes are cut short. At the default `MAX_BANS` of 5,000 it is
+      about 10 MiB, and never more than about 40 MiB. It is written when a ban
+      is made, lifted or made permanent, at most once every 10 seconds, and
+      otherwise with the 15-minute write, so its writes follow the bans made:
+      with a full file, a hundred new bans a day come to about 1 GiB of disk
+      writes.
+    - `reputation.json` and `alerts.json` are usually a few MiB or less.
+    - Writing a file of these sizes takes well under a second on an ordinary
+      disk, in the background, so rewriting whole files needs nothing cleverer.
 - Format: indented JSON with a top-level `version` number, entries sorted by
   client address, times in RFC 3339 UTC, durations and sizes as plain numbers
   with the unit in the field name. The aim is that a person can open `bans.json`
-  in an editor, find an address, and remove or add an entry.
+  in an editor, find an address, and remove or add an entry. `clients.json` puts
+  each client on one line instead, which halves its size and lets `grep` show
+  everything about one client.
 - Writing:
     - serialise from a snapshot taken under the lock, so requests are not held
       up while the file is written;
     - write to a temporary file in the same directory, sync it, rename it over
       the real name, sync the directory. A crash at any point leaves either the
       old complete file or the new complete file, never a partial one;
-    - `bans.json` is written `STATE_WRITE_DELAY` after a change; `clients.json`
+    - `bans.json` is written `STATE_WRITE_DELAY` after a ban is made, lifted or
-      and `reputation.json` every `STATE_COUNTER_INTERVAL`; all three on orderly
+      made permanent. Changes that only update the counts in its notes wait for
+      the write every `STATE_COUNTER_INTERVAL`, as all of `clients.json`,
+      `reputation.json` and `alerts.json` do. All four are written on orderly
       shutdown (SIGTERM);
     - before writing a file, the sidecar checks whether it changed on disk since
       the sidecar last read or wrote it; if it did, the sidecar takes that edit
@@ -662,14 +783,17 @@ in.
     - An edit that does not parse does not stop the running sidecar. It keeps
       the state it has, renames the edited file to `<name>.bad` (for example
       `bans.json.bad`) so the edit is kept, writes the file again from memory,
-      and logs and alerts once with the file and the position of the error. The
+      logs the file and the position of the error, and sends them in one
-      admin fixes the `.bad` file and moves it back.
+      `file_error` alert. The admin fixes the `.bad` file and moves it back.
 - `STATE_DIR` not writable at start: the process exits. A write that fails while
   running: state stays correct in memory, the failure is logged, counted in
-  metrics, alerted once per cooldown, and retried at the next write.
+  metrics, sent as a `file_error` alert once per cooldown, and retried at the
-- What a hard kill can lose: ban changes from the last `STATE_WRITE_DELAY`, and
+  next write.
-  counts, history and reputation from the last `STATE_COUNTER_INTERVAL`. Losing
+- What a hard kill can lose: ban changes from the last `STATE_WRITE_DELAY` (10
-  the volume loses ban history, not service.
+  seconds), and everything else from the last `STATE_COUNTER_INTERVAL` (15
+  minutes): counts and history, the counts in ban notes, GeoJS answers,
+  reputation verdicts and the AbuseIPDB count, anomaly counters and alert
+  cooldowns. Losing the volume loses ban history, not service.
 
 ## Request log
 
@@ -691,13 +815,13 @@ handling while `docker logs` keeps working.
 - Response detail: `response_content_type`, `upstream_status` (differs from
   `status` when the sidecar answered itself), `cache_control`, `location` on
   redirects, `aborted` when the client went away early.
-- Decision: `action` (`forward`, `banned`, `denied`, `rate_limited`,
+- Decision: `action` (`forward`, `banned`, `denied`, `country_denied`,
-  `rule_blocked`, `waf_blocked`, `too_large`, `timed_out`, `upstream_error`),
+  `rate_limited`, `rule_blocked`, `waf_blocked`, `too_large`, `timed_out`,
-  `would_action` in `observe` mode, `limit_percent` and which rule set it,
+  `upstream_error`), `would_action` in `observe` mode, `limit_percent` and which
-  `counts` (the client's minute, hour and day request and byte totals after this
+  rule set it, `counts` (the client's minute, hour and day request and byte
-  request), `limit_hit` (which window), `rule_ids` (rule file rules that
+  totals after this request), `limit_hit` (which window), `rule_ids` (rule file
-  matched), `waf_rule_ids`, `waf_score`, `reputation` (sources that listed the
+  rules that matched), `waf_rule_ids`, `waf_score`, `reputation` (sources that
-  client), `offence` when one was recorded, `ban_expires`.
+  listed the client), `offence` when one was recorded, `ban_expires`.
 - Timings in milliseconds: `duration_total`, `duration_checks` (everything the
   sidecar did before forwarding), `duration_waf`, `duration_upstream_connect`,
   `duration_upstream_first_byte`, `duration_upstream_total`.
@@ -724,15 +848,16 @@ it can be reached by a scraper without exposing ban management.
   duration and upstream duration histograms; requests in flight.
 - Limits and bans: limit hits by window and kind (requests or bytes), size and
   time limit hits by limit, offences by kind, bans created by cause, permanent
-  bans, active bans (gauge).
+  bans, active bans (gauge); requests refused by the country lists, by country.
 - Attack detection: rule file matches by rule id and action, and the number of
   rules loaded; Core Rule Set matches by mode and rule id (label limited to the
   rules that actually fired).
 - Lookup and reputation: requests and bytes by AS number and by country, limited
   to the `METRICS_TOP_N` (default `50`) busiest of each with the rest summed as
   `other`, so the label set stays bounded; reputation queries, hits, failures
-  and remaining daily budget by source; age of each blocklist and lookup
+  and remaining daily budget by source; GeoJS requests and failures, and clients
-  database.
+  that counted as unknown because it did not answer in time; age of each
+  blocklist and lookup database.
 - Housekeeping: tracked clients (gauge), state file writes, write failures, last
   successful write time and size per file; files read again after an edit, and
   edits set aside because they did not parse; alerts sent, failed and suppressed
@@ -759,7 +884,8 @@ One JSON object per alert:
 - `client`, `netblock`, `asn`, `as_name`, `country`
 - `reason`: short human-readable sentence
 - `detail`: event-specific fields, for example `window`, `count`, `limit`,
-  `limit_percent`, `rule_ids`, `path`, `ban_expires`, and for a ban its notes
+  `limit_percent`, `rule_ids`, `path`, `ban_expires`, for a ban its notes, and
+  for `file_error` the file and the position of the error
 - `suppressed_repeats`: number of identical alerts held back by the cooldown
 
 ## Deployment as a sidecar
@@ -808,8 +934,9 @@ networks:
 - Traefik routes only port 8080. The metrics port is reached by the scraper over
   a docker network it shares with the sidecar; the admin port stays on loopback
   inside the container and is used through `docker exec`.
-- The state volume holds the JSON state files, a few MiB for a small service; it
+- The state volume holds the JSON state files, a few tens of MiB at most with
-  needs no backup beyond whatever the host already does.
+  the defaults (see "Persistent state"); it needs no backup beyond whatever the
+  host already does.
 - SSH access to gitea does not pass through the sidecar and is not protected by
   it.
 - Rollout per service: point traefik at the sidecar with only `UPSTREAM_URL`
@@ -819,8 +946,8 @@ networks:
   (`WAF_DISABLED_RULES`, `WAF_EXEMPT_PATHS`, `RATE_LIMIT_EXEMPT_PATHS`) or an
   exemption (`RATE_LIMIT_EXEMPT_NETS`, `ALLOW_NETS`), and its ban is lifted in
   `bans.json`. Additions to consider at any time: an alert destination, a lookup
-  database with biased thresholds, reputation sources, a remote log endpoint,
+  source with country lists or biased thresholds, reputation sources, a remote
-  and an app-specific rule file.
+  log endpoint, and an app-specific rule file.
 - Notes specific to gitea:
     - A clone is a response, and fits the defaults of 30 minutes and 5 GB for
       all but the largest repositories and slowest links. A push is a request:
@@ -841,7 +968,11 @@ networks:
 
 - Lookup database: a configured file that is missing or unreadable at start
   stops the start. A replacement that cannot be read while running is ignored:
-  the file already loaded stays in use, the problem is logged, one alert.
+  the file already loaded stays in use, the problem is logged, one `file_error`
+  alert.
+- GeoJS slow, down or refusing the sidecar: clients with a kept answer are
+  unaffected, new clients count as unknown, the sidecar keeps asking with
+  backoff, one `source_failure` alert per cooldown.
 - Reputation source down or over quota: no verdict, service continues, one
   `source_failure` alert per cooldown.
 - Alert destination down: retried with backoff from a bounded queue, oldest
@@ -852,16 +983,18 @@ networks:
 - Remote log endpoint down: stdout continues, lines are buffered then dropped
   oldest first, drops counted in metrics.
 - State file write fails while running: memory stays authoritative, logged,
-  counted, alerted, retried.
+  counted, one `file_error` alert per cooldown, retried.
 - A state file or rule file edited while running that does not parse: the
   sidecar keeps running on what it has. The state file is set aside as
   `<name>.bad` and written again from memory; the rule file is left as it is.
-  Logged, one alert.
+  Logged, one `file_error` alert.
 - In short: the only things that stop the process happen at start: invalid
   configuration, a configured lookup database that is missing or unreadable, a
   rule file or state file that does not parse, and an unwritable `STATE_DIR`.
   Once running, a broken helper or a broken edit never takes the protected
-  service down.
+  service down. The nearest it comes is GeoJS failing while
+  `EXCLUSIVELY_ALLOWED_COUNTRIES` is set: new clients then cannot be placed, and
+  that list refuses them, as it refuses every client it cannot place.
 
 ## Risks the design has to handle
 
@@ -875,8 +1008,11 @@ networks:
   busy person produces, `RATE_LIMIT_EXEMPT_NETS` and `ALLOW_NETS` take known
   shared addresses out, and the ban notes show an admin what happened before a
   ban is lifted.
-- A `ban` rule that matches a real visitor bans for seven days, so the default
+- A `ban` rule that matches a real visitor bans it for seven days, and the
-  rule file keeps to requests no real visitor sends.
+  visitor's next request during those days makes the ban permanent. So the
+  default rule file keeps to requests no real visitor sends, with its paths
+  anchored at the site root, where no app serves `.env` files or web shells. A
+  visitor banned by mistake is let back in by lifting the ban in `bans.json`.
 - IPv6 address rotation inside a /64: handled by grouping.
 - Widely distributed scrapers using thousands of addresses at low rates each:
   per-client limits do not see them. The AS number and netblock anomaly alerts
@@ -887,9 +1023,15 @@ networks:
 - Core Rule Set false positives against real apps (gitea's editor, API
   payloads): a match refuses only that request and bans no one by itself; the
   request log names the rule, and exclusions by rule id and path fix it.
+- GeoJS as the lookup source: every new visitor's address goes to a third party,
+  and a swarm of fresh addresses, when lookups peak, is when GeoJS may slow down
+  or block the sidecar. Keeping answers for 7 days and asking about many
+  addresses in one request keep the number of requests low; the file source has
+  neither risk.
 - Slow-request attacks: `CLIENT_REQUEST_TIMEOUT` bounds how long a request may
-  take to arrive, and an idle timeout on the listener closes connections that
+  take to arrive, `CLIENT_REQUEST_HEADER_MAX_BYTES` how large its headers may
-  send nothing.
+  be, and `CLIENT_IDLE_TIMEOUT` closes a kept-open connection that sends nothing
+  more.
 - Alert floods: cooldown and hourly cap.
 
 ## Build order
@@ -900,8 +1042,8 @@ networks:
   `observe` mode, the full request log on stdout, the metrics endpoint, health.
 - Second: rule files, admin endpoints, alerting to all three destinations,
   remote log sending.
-- Third: AS number and country lookup, biased thresholds, byte limits, anomaly
+- Third: AS number and country lookup from the file or GeoJS, the country lists,
-  thresholds.
+  biased thresholds, byte limits, anomaly thresholds.
 - Fourth: blocklists, DNSBL, AbuseIPDB, optional CrowdSec decision feed.
 - Fifth: attack detection with Coraza and the Core Rule Set, trap paths, error
   bursts.