sneak/smallwebwaf
1
0
Fork 0
Code Issues 27 Pull Requests 1 Actions Packages Projects Releases Wiki Activity

SPEC and README: review fixes, country lists, GeoJS lookups (closes #6)

Browse Source 
Address the review of the spec update and fold in issues 8 and 11.
Default ban rules are anchored at the site root. `bans.json` is bounded
by `MAX_BANS` and rewritten only when a ban is made, lifted or made
permanent. `clients.json` keeps one client per line, holds at most 20000
clients and is written every 15 minutes. Anomaly counters and alert
state go to a new `alerts.json`, the AbuseIPDB count to
`reputation.json`. 401 no longer counts toward the error burst. New
settings: `DENIED_COUNTRIES`, `EXCLUSIVELY_ALLOWED_COUNTRIES`,
`LOOKUP_SOURCE` (the IPinfo file or GeoJS), `LOOKUP_TIMEOUT`,
`CLIENT_REQUEST_HEADER_MAX_BYTES` and `CLIENT_IDLE_TIMEOUT`.

Model: opus-5-5
This commit is contained in:
sneak
2026-09-23 13:37:34 +00:00
parent 43c63faa04
commit 8fc7539f1a
2 changed files with 304 additions and 142 deletions
Download Patch File Download Diff File Expand all files Collapse all files
SPEC.md
+262 -120
View File
@@ -12,8 +12,8 @@ forwards to the application. The sidecar limits request and byte rates per
client, bounds the size and duration of every request and response, detects
attacks, bans abusers (briefly at first, for seven days on a clear sign of
attack, permanently when they keep at it), consults IP reputation sources, looks
up the AS number and country of each client, lowers its limits for listed AS
numbers and countries, and sends alerts.
up the AS number and country of each client, can refuse whole countries, lowers
its limits for listed AS numbers and countries, and sends alerts.
It is meant to protect an app on the open internet from the first request with
one setting, `UPSTREAM_URL`: every other setting has a default chosen for a
@@ -52,8 +52,9 @@ directory of hand-editable text files.
- Client identification: works out the real client IP from
`X-Forwarded-For`, trusting only the proxy netblocks in `TRUSTED_PROXIES`,
by default the private address ranges.
- Lookup: AS number and country from one database file the operator
supplies, held in memory.
- Lookup: AS number and country, from one of two sources the operator
chooses: a database file the operator supplies, held in memory, or the
GeoJS web service, whose answers are kept for seven days.
- Reputation: static blocklists fetched on a schedule; DNSBL and reputation
API queries made in the background and cached.
- Counters: per-client request and byte counts per minute, hour and day,
@@ -104,7 +105,7 @@ whole exchange.
Otherwise use the TCP peer address and ignore the header.
- IPv6 clients are grouped by prefix (`IPV6_GROUP_PREFIX`, default 64) for
counting and banning, because one abuser usually controls a whole /64. A
client is therefore one IPv4 address or one IPv6 /64.
client is therefore one IPv4 address or one IPv6 group, a /64 by default.
- Static lists.
- In `ALLOW_NETS`: skip every check below and forward. Still counted for
anomaly alerts.
@@ -112,8 +113,17 @@ whole exchange.
- Ban ledger. An active ban on the client's netblock: refuse with
`BAN_RESPONSE`. If a clear sign of attack caused the ban, the ban becomes
permanent (see "Bans").
- Lookup AS number and country, when a lookup database is configured. Unknown is
a valid result and changes nothing.
- Look up the AS number and country, unless `LOOKUP_SOURCE` is `off`. With
GeoJS, a client whose answer is not yet kept waits for it, up to
`LOOKUP_TIMEOUT`. A client the lookup cannot place has an unknown country and
AS number: the exclusive country list refuses it, the biased thresholds give
it `UNKNOWN_LIMIT_PERCENT`, and nothing else treats it differently.
- Country lists. A client whose country is in `DENIED_COUNTRIES`, or, when
`EXCLUSIVELY_ALLOWED_COUNTRIES` is set, is not in it, is refused with
`BAN_RESPONSE`. Every step up to here needs only the client's address, so the
request body has not been read yet, and the refusal skips everything below,
the rule files and the Core Rule Set included. It is counted in the metrics,
but it is not an offence and makes no ban.
- Reputation.
- Address inside a fetched blocklist: apply `BLOCKLIST_ACTION`.
- Cached DNSBL or reputation API result: apply `REPUTATION_ACTION`.
@@ -139,10 +149,10 @@ whole exchange.
- Add response bytes (and request body bytes) to the byte counters. A client
whose byte total passes a byte limit times the percentage has broken that
limit and is banned. The response in progress is not cut off.
- Count the client's requests answered with 401, 403 or 404, whether the app
gave that answer or the sidecar refused the request after a rule file or
Core Rule Set match. More than `ERROR_BURST_THRESHOLD` of them within a
minute breaks a limit and bans the client.
- Count the client's requests answered with 403 or 404, whether the app gave
that answer or the sidecar refused the request after a rule file or Core
Rule Set match. More than `ERROR_BURST_THRESHOLD` of them within a minute
breaks a limit and bans the client.
- Update the client's history and the anomaly counters, and evaluate alert
thresholds.
@@ -157,17 +167,25 @@ whole exchange.
shutdown and loaded again at start (see "Persistent state"), so a restart does
not hand every client a fresh allowance. Buckets whose time has passed are
discarded on load.
- Memory is bounded by `MAX_TRACKED_CLIENTS`; when full, the least recently seen
client is dropped first, with its history and its past bans. Active bans are
never dropped; one whose client is no longer tracked is dropped when it ends.
- Memory is bounded by `MAX_TRACKED_CLIENTS` and `MAX_BANS`. When the table of
clients is full, the least recently seen client is dropped first, with its
history and its lookup answer. When `MAX_BANS` bans are held, the ban that has
gone longest without a request from its netblock is dropped first, whether it
is past, active or permanent (see "Bans").
## Bans
A ban refuses every request from a netblock, answering with `BAN_RESPONSE`
(`403` by default), until the ban ends; a permanent ban never ends. The netblock
a ban covers is the client's IPv4 address (or the wider IPv4 prefix that
`BAN_SCOPE_V4_PREFIX` sets) or its IPv6 /64, and never more, so a permanent ban
does not reach neighbours who did nothing.
(`403` by default), until the ban ends. A permanent ban does not run out: it
ends only when an admin lifts it, or when a new ban is made while `MAX_BANS`
bans are held and it is the ban that has gone longest without a request. A
scanner whose ban was dropped that way is refused and banned again by its next
probe.
The netblock a ban covers is the client: its IPv4 address, or its IPv6 group of
`IPV6_GROUP_PREFIX` (a /64 by default). `BAN_SCOPE_V4_PREFIX` can widen an IPv4
ban to the surrounding netblock. At the defaults a ban covers one address or one
/64, so a permanent ban does not reach neighbours who did nothing.
An offence is a request the sidecar holds against the client: one that carries a
clear sign of attack, breaks a limit, or is refused by a rule file or the Core
@@ -206,21 +224,30 @@ Every ban carries notes with what an admin needs to decide whether to lift it
lifted before it ends is kept, marked as lifted, and does not count toward a
longer ban; deleting its entry from `bans.json` forgets it entirely.
Clients of listed AS numbers and countries, and clients with a poor reputation,
get lower limits (see "Biased thresholds"), so the same rules ban them after
fewer requests. The result is always a ban, first a temporary one and then, for
repeated abuse, a permanent one.
Clients of the AS numbers and countries listed in the biased thresholds, and
clients listed by a reputation source whose action is `limit:<percent>`, get
lower limits (see "Biased thresholds"), so the same rules ban them after fewer
requests. For them the result is always a ban, first a temporary one and then,
for repeated abuse, a permanent one.
Some refusals make no ban at all. `DENY_NETS`, a blocklist or reputation hit
whose action is `deny`, and the country lists (`DENIED_COUNTRIES`,
`EXCLUSIVELY_ALLOWED_COUNTRIES`) refuse each request they cover and record
nothing in `bans.json`.
## Configuration surface
Conventions: lists are comma separated; netblocks are CIDR (a bare address means
/32 or /128); durations use Go syntax plus `d` for days (`90s`, `15m`, `24h`,
`7d`); byte sizes accept `K`, `M`, `G` suffixes.
`7d`); byte sizes accept `K`, `M`, `G` suffixes. Countries are two-letter ISO
codes in either case (`de` and `DE` are the same); a code that is not a country
code, such as `nk` (North Korea is `kp`), stops the start with a message naming
it.
- Only `UPSTREAM_URL` is required. Every other setting has a default chosen for
a service facing the internet in 2026, or stays off until it is given
something only the operator can supply: an alert destination, an account key,
the lookup database.
a service facing the internet in 2026, or stays off until the operator
supplies or chooses what it needs: an alert destination, an account key, a
lookup source.
- Any limit or threshold can be switched off with the value `off`.
- A list set to an empty value is an empty list, and replaces the default.
- Every variable may instead be given as `NAME_FILE` pointing at a file holding
@@ -254,17 +281,21 @@ The settings, by group:
address. A list given replaces the default; set but empty, it trusts
nothing.
- `IPV6_GROUP_PREFIX` (default `64`).
- `MAX_TRACKED_CLIENTS` (default `50000`): clients held in memory and in
- `MAX_TRACKED_CLIENTS` (default `20000`): clients held in memory and in
`clients.json` (see "Persistent state").
- `MAX_BANS` (default `5000`): bans held in memory and in `bans.json`, past,
active and permanent (see "Bans").
- Persistent state
- `STATE_DIR` (default `/data`): the JSON state files. The image declares
`/data` as a volume, so it is writable even when nothing is mounted; a
named volume, as in the examples below, keeps the state in a place that
outlives the container.
- `STATE_WRITE_DELAY` (default `2s`): after a ban changes, wait this long
before writing `bans.json`, so a burst of changes becomes one write.
- `STATE_COUNTER_INTERVAL` (default `60s`): how often `clients.json` and
`reputation.json` are written.
- `STATE_WRITE_DELAY` (default `10s`): after a ban is made, lifted or made
permanent, `bans.json` is written this long later with every change made
in between, so a burst of changes becomes one write.
- `STATE_COUNTER_INTERVAL` (default `15m`): how often `clients.json`,
`reputation.json` and `alerts.json` are written, and `bans.json` when only
the counts in its notes changed.
- Logging
- The request log on stdout is always on and has no switch.
- `LOG_LEVEL` (default `info`): for the process's own messages (start-up,
@@ -319,6 +350,14 @@ The settings, by group:
send its whole request, headers and body.
- `CLIENT_REQUEST_MAX_BYTES` (default `100M`): the largest request body a
client may send.
- `CLIENT_REQUEST_HEADER_MAX_BYTES` (default `32K`): the largest request
line and headers a client may send. Over it, the sidecar answers `431` and
closes the connection, and nothing reaches the app.
- `CLIENT_IDLE_TIMEOUT` (default `120s`): how long a kept-open connection
may wait for its next request before the sidecar closes it. It is longer
than the 90 seconds after which traefik, by default, closes a connection
it is not using, so traefik closes first and never sends a request on a
connection the sidecar is closing.
- `CLIENT_RESPONSE_TIMEOUT` (default `30m`): how long the sidecar may take
to deliver one response to the client, from the end of the request to the
last byte.
@@ -340,27 +379,72 @@ The settings, by group:
started it can only be cut off, and the connection is closed.
- A WebSocket connection leaves these limits behind once it is upgraded: it
stays open until either side closes it.
- Lookup of AS number and country (R7)
- `LOOKUP_DB_PATH`: the IPinfo Lite database in its `.mmdb` form
(`ipinfo_lite.mmdb`), one file that carries both country and AS number;
the sidecar reads its `asn`, `as_name` and `country_code` fields. The
operator downloads it with a free IPinfo account and mounts it read-only.
The sidecar never fetches it and holds no account or token for it.
- Lookup of AS number and country (R7). Off by default: the file needs an
account only the operator can hold, and GeoJS is told every visitor's address.
- `LOOKUP_SOURCE` (default `off`): `off`, `file` or `geojs`.
- `LOOKUP_DB_PATH`: the file for `file`, the IPinfo Lite database in its
`.mmdb` form (`ipinfo_lite.mmdb`), one file that carries both country and
AS number; the sidecar reads its `asn`, `as_name` and `country_code`
fields. The operator downloads it with a free IPinfo account and mounts it
read-only. The sidecar never fetches it and holds no account or token for
it. A file that is missing or unreadable at start stops the start.
- Refreshing the file is the operator's business; IPinfo updates it daily.
The sidecar notices when the file is replaced and reads it again; a
replacement it cannot read is ignored, logged and alerted once, and the
file it has stays in use. Mount the directory that holds the file rather
than the file itself: docker does not show a single mounted file being
replaced on the host.
- Unset by default, which means no lookups. A setting that needs them (the
biased thresholds below, `ADD_LOOKUP_HEADERS`, the per-AS-number anomaly
thresholds) given without `LOOKUP_DB_PATH` stops the start with a message
naming both. A configured file that is missing or unreadable at start
stops the start too.
replacement it cannot read is ignored, logged and sent as one `file_error`
alert, and the file it has stays in use. Mount the directory that holds
the file rather than the file itself: docker does not show a single
mounted file being replaced on the host.
- `geojs` asks the free GeoJS web service, which needs no account or key,
about several addresses in one request
(`https://get.geojs.io/v1/ip/geo.json?ip=a,b,c`), and reads each answer's
`country_code`, `asn` and `organization_name` (the AS name). A
`country_code` of `null`, which GeoJS gives for some ranges, and an `asn`
of `64512`, which it gives when it knows none, count as unknown. GeoJS is
told the address of every new visitor.
- Each GeoJS answer is kept for 7 days with the client's record, in memory
and in `clients.json`, so it survives a restart; after 7 days the client's
next request asks again. A client dropped from the table loses its answer.
- `LOOKUP_TIMEOUT` (default `1s`): how long a client with no kept answer
waits for one; GeoJS normally answers in a fraction of that. At most one
request to GeoJS is under way at a time, the addresses that arrive
meanwhile are asked about together in the next one, and a request that
takes longer than `LOOKUP_TIMEOUT` is abandoned. A client whose answer has
not come in time counts as unknown until it comes: its later requests do
not wait, and its address is asked about again in the background.
- GeoJS publishes no rate limit, but its terms forbid "an excessive amount
of API requests", judged by GeoJS alone, and let it block a caller. While
GeoJS is slow, down or refusing the sidecar, clients with a kept answer
are unaffected and new clients count as unknown, so
`EXCLUSIVELY_ALLOWED_COUNTRIES`, when set, refuses them. The sidecar keeps
asking with backoff and sends one `source_failure` alert per cooldown. A
service that cannot accept this uses the file.
- One source at a time: `LOOKUP_SOURCE=file` without `LOOKUP_DB_PATH`, or
`LOOKUP_DB_PATH` with any other `LOOKUP_SOURCE`, stops the start with a
message naming both. So does a setting that needs lookups while
`LOOKUP_SOURCE` is `off`: the country lists and the biased thresholds
below, `ADD_LOOKUP_HEADERS`, and the per-AS-number anomaly thresholds.
- `ADD_LOOKUP_HEADERS` (default `false`): pass `X-Client-ASN` and
`X-Client-Country` to the app.
- Biased thresholds (R8). The lists are empty by default: they need the lookup
database, which only the operator can supply.
- Country lists. Both are empty by default, since they need a lookup source.
Clients in `ALLOW_NETS` are not checked.
- `DENIED_COUNTRIES`: for example `cn,ru,kp,ir,ua,by`. Every request from a
listed country is refused.
- `EXCLUSIVELY_ALLOWED_COUNTRIES`: for example `us,de`. Every request from
any other country is refused, and so is every request from a client the
lookup cannot place, such as an address missing from the database or one
GeoJS has not answered for in time. A list that let those through would
let every new client in whenever GeoJS stops answering.
- Both may be set. `DENIED_COUNTRIES` then adds nothing, since the exclusive
list already refuses every other country, and a code on both lists stops
the start with a message naming it.
- A refused request is answered with `BAN_RESPONSE`, as a banned client is,
as soon as the client's address has been looked up, before its body is
read. It skips the reputation checks, the limits, the rule files and the
Core Rule Set, is logged with the action `country_denied`, and is counted
in the metrics. It is a refusal, not a ban: it is not an offence and makes
no ban record.
- Biased thresholds (R8). The lists are empty by default: they need a lookup
source, which only the operator can choose.
- `ASN_LIMIT_PERCENT`: for example `AS14061:50,AS16276:50,AS45102:25`.
Clients in a listed AS number get that percentage of every request and
byte limit, so the rules in "Bans" ban them after fewer requests than
@@ -370,8 +454,9 @@ The settings, by group:
`CN:25,RU:50`. Each client from a listed country gets that percentage of
the normal per-client limits.
- No budget is shared by a whole country or AS number: one abuser could use
it up and lock out everyone else there, the denial of service this tool
exists to prevent.
it up and lock out everyone else there, a denial of service nobody chose
and the one this tool exists to prevent. Refusing a whole country is left
to the operator's choice, through the country lists.
- When more than one percentage applies to a client (its AS number, its
country, a reputation hit), the lowest applies.
- `ASN_BYTES_PERCENT`, `COUNTRY_BYTES_PERCENT`: optional overrides applied
@@ -406,10 +491,13 @@ The settings, by group:
example `/wp-login.php,/xmlrpc.php` in front of gitea. A request for one
is a clear sign of attack. This is the env-var short form of a `path` rule
with the `ban` action, for deployments that mount no rule files.
- `ERROR_BURST_THRESHOLD` (default `30`): responses with status 401, 403 or
404 per client per minute; more than this breaks a limit (see "Bans").
- `ERROR_BURST_THRESHOLD` (default `30`): responses with status 403 or 404
per client per minute; more than this breaks a limit (see "Bans").
Scanners walking lists of paths cause hundreds a minute; a person rarely
causes more than a few.
causes more than a few. 401 is not counted: git and container registry
clients send their first request without credentials and are answered 401
at the start of every push, every fetch from a private repository and
every image pull.
- Bans (R5), following the rules under "Bans"
- `ATTACK_BAN_DURATION` (default `7d`): the ban for a first clear sign of
attack.
@@ -455,7 +543,10 @@ The settings, by group:
- `CROWDSEC_LAPI_URL`, `CROWDSEC_LAPI_KEY`: optional. If a CrowdSec engine
exists on the host, pull its decision list on a schedule and treat listed
addresses as banned. This is how a fleet-wide blocklist can arrive without
the sidecar depending on CrowdSec.
the sidecar depending on CrowdSec. The list is kept like a fetched
blocklist, and a listed client's request makes a ban with the cause
`crowdsec` that lasts as long as CrowdSec's decision, so a long list does
not fill `bans.json`.
- `REPUTATION_ACTION` (default `limit:25`): `deny`, `limit:<percent>`, or
`log`, for DNSBL and API hits. Such verdicts are less certain than a
blocklist, so by default a listed client gets a quarter of every limit and
@@ -469,8 +560,11 @@ The settings, by group:
- `ALERT_NTFY_URL` (full topic URL), `ALERT_NTFY_TOKEN`: title, priority and
tags set from the event type.
- `ALERT_EVENTS` (default
`ban,permanent_ban,waf_block,anomaly,reputation_hit,source_failure`):
which event types are sent.
`ban,permanent_ban,waf_block,anomaly,reputation_hit,source_failure,file_error`):
which event types are sent. `source_failure` is a reputation source or
GeoJS failing or refusing the sidecar. `file_error` is a rule file or
state file edited while running that does not parse, a replacement lookup
database that cannot be read, or a state file that cannot be written.
- `ALERT_COOLDOWN` (default `15m`): the same event type for the same client
or netblock is not repeated within this time; a count of suppressed
repeats is included in the next one.
@@ -486,7 +580,7 @@ The settings, by group:
- Per surrounding netblock (`ANOMALY_NET_V4_PREFIX` default `24`,
`ANOMALY_NET_V6_PREFIX` default `48`): `ANOMALY_NET_REQUESTS_PER_MINUTE`,
`..._PER_HOUR`, `ANOMALY_NET_BYTES_PER_MINUTE`, `..._PER_HOUR`.
- Per AS number, which needs the lookup database:
- Per AS number, which needs a lookup source:
`ANOMALY_ASN_REQUESTS_PER_MINUTE`, `..._PER_HOUR`,
`ANOMALY_ASN_BYTES_PER_MINUTE`, `..._PER_HOUR`.
- Whole service: `ANOMALY_TOTAL_REQUESTS_PER_MINUTE`, `..._PER_HOUR`,
@@ -533,7 +627,9 @@ removed there takes effect while the sidecar runs.
- `ban`: the request is a clear sign of attack. Refuse it and ban the
client's netblock for seven days (`ATTACK_BAN_DURATION`); any further
request during those days, or a later clear sign of attack, makes the ban
permanent (see "Bans"). Use it only for requests no real visitor sends.
permanent (see "Bans"). Use it only for requests no real visitor sends,
and anchor a path at the site root (`^/`): a file of the same name deeper
in a site can be ordinary content, such as a file in a gitea repository.
- `regex`: Go regular expression syntax (RE2). It has no backreferences or
lookaround, and in exchange matching time is linear in the input, so no rule
can be made to stall the proxy. `(?i)` at the front makes a rule
@@ -545,9 +641,9 @@ removed there takes effect while the sidecar runs.
line that does not parse, a regex that does not compile, or a duplicate id
stops the process with a message naming the file and line. While running, the
same faults leave the rules as they were, including the earlier version of
that file, and the log and one alert name the file and line; once the file is
fixed, it is read again. A missing or empty directory is not an error: the log
says that no rules were loaded.
that file, and the log and one `file_error` alert name the file and line; once
the file is fixed, it is read again. A missing or empty directory is not an
error: the log says that no rules were loaded.
- The image ships a default file in `RULES_DIR`. Mounting a directory over it
replaces the defaults; mounting single files into it adds to them. Docker does
not show a single mounted file being replaced on the host, which is how many
@@ -564,16 +660,19 @@ Example file:
# 00-default.rules: probes no real visitor sends
# id target action regex
env-file path ban (?i)/\.env(\.[a-z]+)?$
env-file path ban (?i)^/\.env(\.[a-z]+)?$
git-dir path ban ^/\.git/(config|HEAD|index)$
php-shell path ban (?i)/(shell|c99|r57|wso|alfa)\.php$
php-shell path ban (?i)^/(shell|c99|r57|wso|alfa)\.php$
scanner-agent user_agent ban (?i)\b(sqlmap|nikto|nuclei|masscan|zgrab|wpscan)\b
path-traversal uri block (\.\./){2,}
empty-agent user_agent log ^$
```
The image ships one default file of this kind. It is kept short and limited to
patterns that are wrong for every app. Anything app-specific belongs in a file
patterns that are wrong for every app, and its path rules are anchored at the
site root: in front of gitea, `/.env` is a probe, while
`/<owner>/<repo>/src/branch/main/.env.example` is a file in a repository that
any visitor or search crawler may open. Anything app-specific belongs in a file
the deployer mounts: a request for `/wp-login.php`, for example, is a clear sign
of attack in front of gitea and an ordinary login in front of WordPress.
@@ -585,57 +684,79 @@ wp-probe path ban (?i)^/(wp-login\.php|xmlrpc\.php|wp-admin/)
## Persistent state
All state lives in memory, and the files in `STATE_DIR` hold a copy of all of
it, so an orderly stop loses nothing. No database is used, and nothing is read
from disk while serving a request. The files are meant for people as well: an
admin can read or edit them at any time, and the running sidecar takes the edit
in.
it, so an orderly stop loses nothing. The one exception is log lines still
waiting to be sent to `LOG_REMOTE_URL`, which stdout has already carried. No
database is used, and nothing is read from disk while serving a request. The
files are meant for people as well: an admin can read or edit them at any time,
and the running sidecar takes the edit in.
- Files, each holding one kind of state:
- `bans.json`: every active and past ban. Per entry: the netblock, start,
expiry (`null` for permanent), what caused it (`attack`, `limit`, `admin`
or `crowdsec`), a short reason, when it was lifted if an admin lifted it,
and a `notes` field (below).
- `bans.json`: active and past bans, up to `MAX_BANS`. Per entry: the
netblock, start, expiry (`null` for permanent), what caused it (`attack`,
`limit`, `admin` or `crowdsec`), a short reason, when it was lifted if an
admin lifted it, and a `notes` field (below).
- `clients.json`: per client, its minute, hour and day counters and its
history since it was first seen: first and last time seen, the AS number,
AS name and country last looked up, total requests and bytes in each
direction, how many requests were forwarded and how many refused,
responses by status class, and offences by kind. Clients that were never
banned are kept too, so every client's history survives a restart;
`GET /clients/<ip>` shows it.
- `reputation.json`: the last good copy of each list fetched from a URL, and
AS name and country last looked up and when (with GeoJS, the answer kept
for 7 days), total requests and bytes in each direction, how many requests
were forwarded and how many refused, responses by status class, and
offences by kind. Clients that were never banned are kept too, so every
client's history survives a restart; `GET /clients/<ip>` shows it.
- `reputation.json`: the last good copy of each list fetched from a URL,
cached DNSBL and reputation API verdicts, each with the time it was
fetched.
fetched, and the AbuseIPDB checks spent today, so a restart does not reset
the daily budget.
- `alerts.json`: the anomaly counters for surrounding netblocks, AS numbers,
named netblocks and the whole service; for each event type and client or
netblock, when it was last sent and how many repeats the cooldown has held
back since; the alerts sent this hour; and alerts still waiting to be
sent.
- Ban notes. The notes on a ban hold what an admin needs to decide whether to
lift it, drawn from what the sidecar already knows; nothing is looked up to
fill them:
- the AS number, AS name and country (empty without a lookup database);
- the AS number, AS name and country (empty when lookups are off);
- what was broken: the rule ids and target that matched, or the limit, its
window, the count reached and the client's limit percentage with what set
it; and any reputation sources that listed the client;
- the requests that caused the ban, up to the last ten: time, method, host,
path, status and user agent;
path with its query string, status and user agent, each text cut to 256
bytes;
- how many requests counted toward the ban, and the time span over which
they came;
- the netblock's total requests since it was first seen, and the requests
refused under this ban so far, kept up to date while the ban lasts;
- how many earlier bans of each kind the netblock has had.
- Size: each tracked client takes about 2 KiB in `clients.json`, so at the
default `MAX_TRACKED_CLIENTS` of 50,000 the file can reach about 100 MiB; a
small service usually tracks a few thousand clients and a few MiB. The other
files stay small. Writing a whole file of that size takes about a second, in
the background, so rewriting whole files still needs nothing cleverer.
- Size and disk writes. Each file is rewritten whole, so each has a bound:
- `clients.json` takes about 1 KiB per client. Clients are dropped only when
the table is full, so on a public service the file grows to the default
`MAX_TRACKED_CLIENTS` of 20,000, about 20 MiB. Written every 15 minutes,
that is under 2 GiB of disk writes a day.
- `bans.json` takes about 2 KiB per ban and at most about 8 KiB, since the
texts in the notes are cut short. At the default `MAX_BANS` of 5,000 it is
about 10 MiB, and never more than about 40 MiB. It is written when a ban
is made, lifted or made permanent, at most once every 10 seconds, and
otherwise with the 15-minute write, so its writes follow the bans made:
with a full file, a hundred new bans a day come to about 1 GiB of disk
writes.
- `reputation.json` and `alerts.json` are usually a few MiB or less.
- Writing a file of these sizes takes well under a second on an ordinary
disk, in the background, so rewriting whole files needs nothing cleverer.
- Format: indented JSON with a top-level `version` number, entries sorted by
client address, times in RFC 3339 UTC, durations and sizes as plain numbers
with the unit in the field name. The aim is that a person can open `bans.json`
in an editor, find an address, and remove or add an entry.
in an editor, find an address, and remove or add an entry. `clients.json` puts
each client on one line instead, which halves its size and lets `grep` show
everything about one client.
- Writing:
- serialise from a snapshot taken under the lock, so requests are not held
up while the file is written;
- write to a temporary file in the same directory, sync it, rename it over
the real name, sync the directory. A crash at any point leaves either the
old complete file or the new complete file, never a partial one;
- `bans.json` is written `STATE_WRITE_DELAY` after a change; `clients.json`
and `reputation.json` every `STATE_COUNTER_INTERVAL`; all three on orderly
- `bans.json` is written `STATE_WRITE_DELAY` after a ban is made, lifted or
made permanent. Changes that only update the counts in its notes wait for
the write every `STATE_COUNTER_INTERVAL`, as all of `clients.json`,
`reputation.json` and `alerts.json` do. All four are written on orderly
shutdown (SIGTERM);
- before writing a file, the sidecar checks whether it changed on disk since
the sidecar last read or wrote it; if it did, the sidecar takes that edit
@@ -662,14 +783,17 @@ in.
- An edit that does not parse does not stop the running sidecar. It keeps
the state it has, renames the edited file to `<name>.bad` (for example
`bans.json.bad`) so the edit is kept, writes the file again from memory,
and logs and alerts once with the file and the position of the error. The
admin fixes the `.bad` file and moves it back.
logs the file and the position of the error, and sends them in one
`file_error` alert. The admin fixes the `.bad` file and moves it back.
- `STATE_DIR` not writable at start: the process exits. A write that fails while
running: state stays correct in memory, the failure is logged, counted in
metrics, alerted once per cooldown, and retried at the next write.
- What a hard kill can lose: ban changes from the last `STATE_WRITE_DELAY`, and
counts, history and reputation from the last `STATE_COUNTER_INTERVAL`. Losing
the volume loses ban history, not service.
metrics, sent as a `file_error` alert once per cooldown, and retried at the
next write.
- What a hard kill can lose: ban changes from the last `STATE_WRITE_DELAY` (10
seconds), and everything else from the last `STATE_COUNTER_INTERVAL` (15
minutes): counts and history, the counts in ban notes, GeoJS answers,
reputation verdicts and the AbuseIPDB count, anomaly counters and alert
cooldowns. Losing the volume loses ban history, not service.
## Request log
@@ -691,13 +815,13 @@ handling while `docker logs` keeps working.
- Response detail: `response_content_type`, `upstream_status` (differs from
`status` when the sidecar answered itself), `cache_control`, `location` on
redirects, `aborted` when the client went away early.
- Decision: `action` (`forward`, `banned`, `denied`, `rate_limited`,
`rule_blocked`, `waf_blocked`, `too_large`, `timed_out`, `upstream_error`),
`would_action` in `observe` mode, `limit_percent` and which rule set it,
`counts` (the client's minute, hour and day request and byte totals after this
request), `limit_hit` (which window), `rule_ids` (rule file rules that
matched), `waf_rule_ids`, `waf_score`, `reputation` (sources that listed the
client), `offence` when one was recorded, `ban_expires`.
- Decision: `action` (`forward`, `banned`, `denied`, `country_denied`,
`rate_limited`, `rule_blocked`, `waf_blocked`, `too_large`, `timed_out`,
`upstream_error`), `would_action` in `observe` mode, `limit_percent` and which
rule set it, `counts` (the client's minute, hour and day request and byte
totals after this request), `limit_hit` (which window), `rule_ids` (rule file
rules that matched), `waf_rule_ids`, `waf_score`, `reputation` (sources that
listed the client), `offence` when one was recorded, `ban_expires`.
- Timings in milliseconds: `duration_total`, `duration_checks` (everything the
sidecar did before forwarding), `duration_waf`, `duration_upstream_connect`,
`duration_upstream_first_byte`, `duration_upstream_total`.
@@ -724,15 +848,16 @@ it can be reached by a scraper without exposing ban management.
duration and upstream duration histograms; requests in flight.
- Limits and bans: limit hits by window and kind (requests or bytes), size and
time limit hits by limit, offences by kind, bans created by cause, permanent
bans, active bans (gauge).
bans, active bans (gauge); requests refused by the country lists, by country.
- Attack detection: rule file matches by rule id and action, and the number of
rules loaded; Core Rule Set matches by mode and rule id (label limited to the
rules that actually fired).
- Lookup and reputation: requests and bytes by AS number and by country, limited
to the `METRICS_TOP_N` (default `50`) busiest of each with the rest summed as
`other`, so the label set stays bounded; reputation queries, hits, failures
and remaining daily budget by source; age of each blocklist and lookup
database.
and remaining daily budget by source; GeoJS requests and failures, and clients
that counted as unknown because it did not answer in time; age of each
blocklist and lookup database.
- Housekeeping: tracked clients (gauge), state file writes, write failures, last
successful write time and size per file; files read again after an edit, and
edits set aside because they did not parse; alerts sent, failed and suppressed
@@ -759,7 +884,8 @@ One JSON object per alert:
- `client`, `netblock`, `asn`, `as_name`, `country`
- `reason`: short human-readable sentence
- `detail`: event-specific fields, for example `window`, `count`, `limit`,
`limit_percent`, `rule_ids`, `path`, `ban_expires`, and for a ban its notes
`limit_percent`, `rule_ids`, `path`, `ban_expires`, for a ban its notes, and
for `file_error` the file and the position of the error
- `suppressed_repeats`: number of identical alerts held back by the cooldown
## Deployment as a sidecar
@@ -808,8 +934,9 @@ networks:
- Traefik routes only port 8080. The metrics port is reached by the scraper over
a docker network it shares with the sidecar; the admin port stays on loopback
inside the container and is used through `docker exec`.
- The state volume holds the JSON state files, a few MiB for a small service; it
needs no backup beyond whatever the host already does.
- The state volume holds the JSON state files, a few tens of MiB at most with
the defaults (see "Persistent state"); it needs no backup beyond whatever the
host already does.
- SSH access to gitea does not pass through the sidecar and is not protected by
it.
- Rollout per service: point traefik at the sidecar with only `UPSTREAM_URL`
@@ -819,8 +946,8 @@ networks:
(`WAF_DISABLED_RULES`, `WAF_EXEMPT_PATHS`, `RATE_LIMIT_EXEMPT_PATHS`) or an
exemption (`RATE_LIMIT_EXEMPT_NETS`, `ALLOW_NETS`), and its ban is lifted in
`bans.json`. Additions to consider at any time: an alert destination, a lookup
database with biased thresholds, reputation sources, a remote log endpoint,
and an app-specific rule file.
source with country lists or biased thresholds, reputation sources, a remote
log endpoint, and an app-specific rule file.
- Notes specific to gitea:
- A clone is a response, and fits the defaults of 30 minutes and 5 GB for
all but the largest repositories and slowest links. A push is a request:
@@ -841,7 +968,11 @@ networks:
- Lookup database: a configured file that is missing or unreadable at start
stops the start. A replacement that cannot be read while running is ignored:
the file already loaded stays in use, the problem is logged, one alert.
the file already loaded stays in use, the problem is logged, one `file_error`
alert.
- GeoJS slow, down or refusing the sidecar: clients with a kept answer are
unaffected, new clients count as unknown, the sidecar keeps asking with
backoff, one `source_failure` alert per cooldown.
- Reputation source down or over quota: no verdict, service continues, one
`source_failure` alert per cooldown.
- Alert destination down: retried with backoff from a bounded queue, oldest
@@ -852,16 +983,18 @@ networks:
- Remote log endpoint down: stdout continues, lines are buffered then dropped
oldest first, drops counted in metrics.
- State file write fails while running: memory stays authoritative, logged,
counted, alerted, retried.
counted, one `file_error` alert per cooldown, retried.
- A state file or rule file edited while running that does not parse: the
sidecar keeps running on what it has. The state file is set aside as
`<name>.bad` and written again from memory; the rule file is left as it is.
Logged, one alert.
Logged, one `file_error` alert.
- In short: the only things that stop the process happen at start: invalid
configuration, a configured lookup database that is missing or unreadable, a
rule file or state file that does not parse, and an unwritable `STATE_DIR`.
Once running, a broken helper or a broken edit never takes the protected
service down.
service down. The nearest it comes is GeoJS failing while
`EXCLUSIVELY_ALLOWED_COUNTRIES` is set: new clients then cannot be placed, and
that list refuses them, as it refuses every client it cannot place.
## Risks the design has to handle
@@ -875,8 +1008,11 @@ networks:
busy person produces, `RATE_LIMIT_EXEMPT_NETS` and `ALLOW_NETS` take known
shared addresses out, and the ban notes show an admin what happened before a
ban is lifted.
- A `ban` rule that matches a real visitor bans for seven days, so the default
rule file keeps to requests no real visitor sends.
- A `ban` rule that matches a real visitor bans it for seven days, and the
visitor's next request during those days makes the ban permanent. So the
default rule file keeps to requests no real visitor sends, with its paths
anchored at the site root, where no app serves `.env` files or web shells. A
visitor banned by mistake is let back in by lifting the ban in `bans.json`.
- IPv6 address rotation inside a /64: handled by grouping.
- Widely distributed scrapers using thousands of addresses at low rates each:
per-client limits do not see them. The AS number and netblock anomaly alerts
@@ -887,9 +1023,15 @@ networks:
- Core Rule Set false positives against real apps (gitea's editor, API
payloads): a match refuses only that request and bans no one by itself; the
request log names the rule, and exclusions by rule id and path fix it.
- GeoJS as the lookup source: every new visitor's address goes to a third party,
and a swarm of fresh addresses, when lookups peak, is when GeoJS may slow down
or block the sidecar. Keeping answers for 7 days and asking about many
addresses in one request keep the number of requests low; the file source has
neither risk.
- Slow-request attacks: `CLIENT_REQUEST_TIMEOUT` bounds how long a request may
take to arrive, and an idle timeout on the listener closes connections that
send nothing.
take to arrive, `CLIENT_REQUEST_HEADER_MAX_BYTES` how large its headers may
be, and `CLIENT_IDLE_TIMEOUT` closes a kept-open connection that sends nothing
more.
- Alert floods: cooldown and hourly cap.
## Build order
@@ -900,8 +1042,8 @@ networks:
`observe` mode, the full request log on stdout, the metrics endpoint, health.
- Second: rule files, admin endpoints, alerting to all three destinations,
remote log sending.
- Third: AS number and country lookup, biased thresholds, byte limits, anomaly
thresholds.
- Third: AS number and country lookup from the file or GeoJS, the country lists,
biased thresholds, byte limits, anomaly thresholds.
- Fourth: blocklists, DNSBL, AbuseIPDB, optional CrowdSec decision feed.
- Fifth: attack detection with Coraza and the Core Rule Set, trap paths, error
bursts.