commit a0d2c21346d9e0bf1ee93eaa43cddd3bbd7e641d Author: sneak Date: Mon Sep 21 07:42:45 2026 +0000 Add README, SPEC and tool evaluation, closes #1 Initial documents: what smallwebwaf is and why, the proposed feature list, the design spec with the rule file format and the open design questions, and the survey of existing tools. Model: fable-5-1 diff --git a/EVALUATION.md b/EVALUATION.md new file mode 100644 index 0000000..f671f13 --- /dev/null +++ b/EVALUATION.md @@ -0,0 +1,265 @@ +# Evaluation: existing tools for an env-var-configured protective sidecar + +Date: 2026-09-21. Method: web survey of project documentation, repositories and +release pages. Nothing was installed or run. Claims marked "unverified" were not +confirmed against primary documentation. + +## What was asked for + +One container that sits between traefik and one application container, +configured only by environment variables, that provides: + +- R1: rate limits per minute, per hour and per day +- R2: netblocks that bypass rate limiting +- R3: alerts on attacks through webhook, Slack or ntfy +- R4: alerts on anomalies: requests or bytes per minute or hour, for given IPs + or netblocks, crossing a threshold +- R5: temporary blocks for abusers, permanent bans for repeat offenders +- R6: one or more RBLs or IP reputation APIs +- R7: AS number lookup +- R8: thresholds biased by AS number or country (for example, listed AS + numbers get 50 percent of the normal limit) +- R9: WAF-style attack detection and prevention +- R10: runs as a plain env-var-configured sidecar between traefik and one app + +## Verdict + +Nothing existing is suitable. No surveyed tool meets R8 (scaling limits by AS +number or country) or the bytes half of R4 at all, and the tools that come +closest on the rest fail R10: they need several containers, YAML or a web UI. +The recommendation is to build a small single-binary sidecar; `SPEC.md` in this +directory begins that spec. It reuses the mature parts of the surveyed field as +libraries and data sources (Coraza and the OWASP Core Rule Set for attack +detection, DNSBLs, AbuseIPDB, optionally a CrowdSec decision feed) instead of +reimplementing them. + +Closest existing options, and why each still falls short: + +- BunkerWeb is the closest single product. + - Meets: R1 (rates in requests per second, minute, hour or day), R2 + (`LIMIT_IGNORE_IP`, also by AS number and reverse DNS), R3 partly (webhook, + Slack, Discord, Matrix plugins, fired only on denied requests; ntfy only + through the generic webhook, payload format unverified), R5 partly + (`BAD_BEHAVIOR_BAN_TIME`, `0` means permanent; no escalation for repeat + offenders, the ban length is one fixed value), R6 (DNSBL plugin, external + blacklist URLs, optional CrowdSec), R7 partly (AS number used for + blacklist and whitelist decisions), R9 (ModSecurity with the Core Rule + Set, or Coraza plugin), env-var settings. + - Fails: R8 (AS number and country can only allow or deny, never scale a + limit), R4 (no volume or byte threshold alerts in the free edition; + reporting is a paid feature), R5 escalation, and R10 in spirit: since + 1.6 it needs a `bunkerweb` container plus a `bw-scheduler` container and + a database, or the all-in-one image that bundles nginx, scheduler, UI and + Redis in one container. It is designed to be the front door for many + sites, not a per-app sidecar. AGPL-3.0. + - Unverified: whether several rates (minute, hour, day) can be stacked on + the same URL pattern; the documented design is one rate per URL pattern. +- CrowdSec is the strongest detection and ban engine, but it is not a proxy. + - Meets: R5 fully (profiles with `duration_expr`, for example four hours + times the number of previous decisions), R3 (notification plugins for + Slack, generic HTTP, email and others; ntfy through the HTTP plugin), R6 + (community blocklist, further blocklists, reputation API), R7 (alerts are + enriched with AS number and country), R9 (AppSec component with virtual + patching and ModSecurity-syntax rules), R2 (allowlists). + - Partly: R1 and R8. Detection is by leaky-bucket scenarios over logs, and + a scenario can filter on AS number or country, so a stricter bucket for + listed AS numbers is possible, but each is a hand-written YAML scenario, + it reacts after the fact by banning, and it is not an inline limiter that + answers 429. + - Fails: R10 (needs the security engine container with persistent state, + log acquisition from traefik, a bouncer such as the traefik plugin, and + YAML for acquisition, profiles, scenarios and notifications), bytes half + of R4. +- CrowdSec plus traefik's own `rateLimit` middleware is the best combination + with no new code. It gives inline limiting (one window per middleware, keyed + by IP, with `sourceCriterion` exclusions), bans with escalation, alerts and + reputation. It still fails R8, the bytes half of R4, R1's three stacked + windows only by chaining three middlewares per router, and R10 entirely: the + configuration lives in traefik labels and CrowdSec YAML on each host, not in + one sidecar's environment. + +## Candidate by candidate + +- CrowdSec with the traefik bouncer plugin + (`maxlerebourg/crowdsec-bouncer-traefik-plugin`) + - What it is: a traefik middleware plugin that asks a CrowdSec local API + whether the client IP is banned, and can forward each request to the + CrowdSec AppSec component for a WAF verdict. Modes: query per request, + cached, streamed decision list (recommended), standalone against the + central API, or AppSec only. Supports captcha remediation. + - Covers: R5, R6, R9, R3 and R7 through the engine (see Verdict). + - Misses: the plugin itself does no rate limiting; R8; R4 bytes. + - Configuration: traefik static config to load the plugin, dynamic config + or labels for the middleware, CrowdSec YAML for everything else. + - Sidecar fit: no. It lives inside traefik, plus a separate engine + container. + - Maturity: widely used (about 900 stars, listed in the traefik plugin + catalog, documented by CrowdSec itself), actively maintained. Traefik + plugins run in an interpreter inside traefik, which costs some + per-request time. +- CrowdSec generic bouncers (nginx, Caddy, firewall) + - Same engine, different enforcement point. The firewall bouncer blocks at + nftables level on the host, which is cheap and covers every service on + the host at once; worth considering fleet-wide regardless of this + project. Not a sidecar, same misses as above. +- BunkerWeb 1.6.14 (`bunkerity/bunkerweb`) + - What it is: nginx with Lua plugins, ModSecurity and the Core Rule Set, + configured by settings that are passed as env vars to its scheduler + container. Documents running behind traefik with `USE_REAL_IP` and + `REAL_IP_FROM`. + - Coverage and misses: see Verdict. Also has bot challenges (cookie, + JavaScript, captcha), country allow and deny, Tor and user-agent lists. + - Sidecar fit: partial. Env vars yes; single small container no. + - Maturity: mature, frequent releases, commercial company behind it with a + paid edition; free plugins are AGPL-3.0. +- SafeLine (`chaitin/SafeLine`) + - What it is: nginx-derived (Tengine) reverse proxy with a proprietary + semantic detection engine and a web console. About 22,000 stars. + - Covers: R9 well, basic per-IP rate limiting with block duration, bot + challenge, authentication gate. + - Misses: R8, R7, R4; R1 has one window per rule. Country blocking, alert + messages, syslog forwarding and the better reputation database are paid + (Lite about 10 USD per month, Pro about 100 USD per month); the free + edition is capped at 10 applications. + - Configuration: web UI backed by PostgreSQL. No env-var configuration. + - Sidecar fit: no. Seven containers (postgres, management, detector, + tengine, and three helpers); the proxy container uses host networking. + The detection engine is closed source. + - Maturity: very active, vendor-driven. +- Coraza (`corazawaf/coraza`) and Coraza-based proxies + - What it is: a Go library that implements the ModSecurity rule language + and runs the OWASP Core Rule Set. OWASP project, actively maintained, the + successor path now that ModSecurity is in maintenance only. + - Packagings: `coraza-caddy` (Caddy module), `coraza-spoa` (HAProxy), + `coraza-proxy-wasm` (Envoy), a traefik WASM plugin, and + `coreruleset/coraza-crs-docker` (Caddy plus Coraza plus the Core Rule + Set, with env vars for backend address, engine mode and rule-set + tuning). + - Covers: R9 only. `coraza-crs-docker` fits R10 well: one container, env + vars, backend address. + - Misses: R1 to R8. ModSecurity-language rules can count requests per IP + in a persistent collection, but Coraza's support for persistent + collections is limited and this is not a practical rate limiter. + - Value here: the right library to embed for R9 in a purpose-built sidecar. +- ModSecurity Core Rule Set containers (`owasp/modsecurity-crs`) + - What it is: official images of Apache or nginx with ModSecurity and the + Core Rule Set, as a reverse proxy configured by env vars (`BACKEND`, + paranoia level, anomaly thresholds, many more). Current images carry + ModSecurity 2.9.x on Apache and Core Rule Set 4.x. + - Covers: R9, and R10 (single container, env vars, one backend). + - Misses: R1 to R8. + - Maturity: rule set is very actively maintained; the ModSecurity engine + itself is in maintenance under OWASP after Trustwave ended support in + 2024. +- Anubis (`TecharoHQ/anubis`), 1.27 current + - What it is: a single-binary reverse proxy that makes browsers solve a + proof-of-work challenge before passing them to `TARGET`. Aimed at + scrapers, which is likely a large share of unwanted traffic on a public + gitea. + - Covers: R10 well (one container, env vars for listener, target, + difficulty, cookies). Policy rules can match on path, user agent, + headers, IP ranges, and with the vendor's hosted data service also AS + number and country, and can weigh a request toward a harder challenge. + - Misses: R1, R3, R4, R5, R6, R9. Bot policy needs a YAML file, not env + vars. AS number and country matching depend on the vendor's hosted + service. Breaks non-browser clients unless paths are exempted; for + gitea, git-over-HTTP and API paths must be allowed through by rule. + - Maturity: very active, widely deployed on code forges since 2025. + - Value here: complementary. It can be chained (traefik, then the sidecar, + then Anubis, then the app) if challenge pages are wanted. +- go-away (`git.gammaspectra.live/git/go-away`) + - What it is: a single Go reverse proxy with a rule language over request + properties, several challenge types (including ones that need no + JavaScript), built-in DNSBL check (default `dnsbl.dronebl.org`), network + range lists, and metrics by network range, AS and user agent. + - Covers: R6 partly, R7 partly (through range lists rather than a lookup + database; unverified), bot filtering. + - Misses: R1, R3, R4, R5, R8, R9. YAML policy configuration. Small + maintainer base. +- fail2ban-style traefik plugins + - `tomMoulard/fail2ban`: watches request rate or response status per IP + inside traefik and bans for a fixed time. Covers a slice of R5 (fixed + temporary ban, no escalation, state lost on traefik restart) and R2 + (allowlist). Nothing else. + - `juitde/traefik-plugin-fail2ban`, fail2ban connector plugins, GeoBlock + plugins (country allow or deny through an external lookup API): same + shape, each one small feature, configured in traefik, state in memory. + - Sidecar fit: no; they live inside traefik. +- Traefik built-in middlewares + - `rateLimit` (one average-and-burst window per middleware, optional Redis + in traefik 3.x), `inFlightReq`, `ipAllowList`. No bans, alerts, + reputation or AS number awareness. +- caddy-waf (`fabriziosalmi/caddy-waf`), 0.4.x + - What it is: a Caddy module with regex rules and anomaly scoring, per-IP + and per-path rate limiting with one configurable window, IP and DNS + blacklists, Tor exit list fetch, country and AS number allow or deny + from MaxMind databases. + - Misses: R8 (allow or deny only), R3, R4, R5 (no documented ban state or + alerting), R1's three windows. Caddyfile configuration. One maintainer, + pre-1.0, AGPL-3.0. +- open-appsec (Check Point) + - Machine-learning WAF agent attached to nginx, Kong, Envoy or similar, + with a declarative policy file or the vendor's cloud console. Covers R9 + only; rate limiting and richer features are in paid tiers. Not a sidecar + in the required sense. +- iocaine + - Serves generated garbage pages to clients the fronting proxy classifies + as scrapers. Not a limiter, WAF or ban tool; out of scope except as a + curiosity for scraper traffic. +- Pangolin + - A tunnelled access platform that bundles traefik and optionally + CrowdSec. Replaces the ingress rather than adding a sidecar; out of + scope. + +## Requirement by requirement, across the field + +- R1 three windows: BunkerWeb (one rate per URL pattern; stacking unverified). + Everyone else offers one window per rule or none. +- R2 bypass netblocks: BunkerWeb, CrowdSec, traefik, fail2ban plugins. +- R3 alerts: CrowdSec (best), BunkerWeb (denied requests only), SafeLine (paid). +- R4 volume and byte anomalies: none. CrowdSec can approximate request-count + anomalies with a custom scenario; nothing handles bytes. +- R5 temporary then permanent: CrowdSec only. BunkerWeb has fixed or permanent, + not both by history. +- R6 reputation: CrowdSec, BunkerWeb, go-away, caddy-waf (static lists). +- R7 AS number lookup: CrowdSec (enrichment), BunkerWeb and caddy-waf (allow or + deny), Anubis (hosted service). +- R8 biased thresholds: none. +- R9 attack detection: Coraza or ModSecurity with the Core Rule Set (open, + standard), CrowdSec AppSec, SafeLine (closed engine), BunkerWeb (wraps the + first). +- R10 env-var sidecar: `owasp/modsecurity-crs`, `coraza-crs-docker`, Anubis. + None of these covers anything beyond its one job. + +## Not checked + +- No tool was installed, load-tested or run against real fleet traffic. +- Licence terms of reputation data sources for this use (AbuseIPDB free tier, + Spamhaus query policy, IPinfo Lite attribution) were not read in full. +- How upaas describes a second container for a deployed service was not + examined; no repo was opened for this task. +- Paid editions (BunkerWeb PRO, SafeLine Pro, CrowdSec paid blocklists) were + assessed from public feature lists only. +- The owner's Go dependency defaults file was not found at the documented path, + so library choices in `SPEC.md` are proposals to be checked against it. + +## Sources + +- https://docs.bunkerweb.io/latest/features/ +- https://docs.bunkerweb.io/latest/integrations/ +- https://github.com/bunkerity/bunkerweb-plugins +- https://github.com/maxlerebourg/crowdsec-bouncer-traefik-plugin +- https://docs.crowdsec.net/u/bouncers/traefik/ +- https://docs.crowdsec.net/docs/next/local_api/profiles/format/ +- https://docs.crowdsec.net/u/user_guides/waf_rp_howto/ +- https://www.crowdsec.net/blog/waf-traefik-crowdsec +- https://github.com/chaitin/SafeLine +- https://dev.to/carrie_luo1/differences-between-safeline-waf-free-and-safeline-waf-pro-2dak +- https://github.com/coreruleset/coraza-crs-docker +- https://github.com/coreruleset/modsecurity-crs-docker +- https://github.com/TecharoHQ/anubis +- https://git.gammaspectra.live/git/go-away +- https://github.com/fabriziosalmi/caddy-waf +- https://plugins.traefik.io/plugins/628c9ebcffc0cd18356a979f/fail2-ban +- https://github.com/juitde/traefik-plugin-fail2ban +- https://docs.openappsec.io/ diff --git a/README.md b/README.md new file mode 100644 index 0000000..8071582 --- /dev/null +++ b/README.md @@ -0,0 +1,152 @@ +# smallwebwaf + +`smallwebwaf` is a simple, fast, logging web application firewall for people who +host their own services. It is one small container that sits between your +reverse proxy (traefik) and one application: traefik points at `smallwebwaf`, +and `smallwebwaf` points at the app. It is configured with environment +variables, keeps its state in memory, and writes a detailed JSON log line for +every request. + +Status: design stage. This repository currently holds the documents only; no +code has been written. The design is in [`SPEC.md`](SPEC.md), and the survey of +existing tools that led to it is in [`EVALUATION.md`](EVALUATION.md). + +## Why + +Small self-hosted sites now receive a great deal of traffic nobody asked for: +scrapers that ignore `robots.txt` and crawl every commit of every repository on +a public git server, vulnerability scanners walking through lists of WordPress +and `.env` paths, and credential-guessing bots. Most of it comes from a small +number of hosting networks and countries. A single-person operation has no +abuse desk and no CDN contract; it needs something small that can be put in +front of one service and left alone. + +The existing tools each solve part of this. Rule-based firewalls catch attack +payloads but do not limit request rates. Rate limiters count requests but +cannot tell a residential visitor from a rented server farm. The products that +do most of it want several containers, a database and a web console. None of +them can say "clients from these networks get half the normal allowance", which +is the most useful thing to be able to say when nearly all abuse comes from a +known list of AS numbers. [`EVALUATION.md`](EVALUATION.md) goes through the +candidates one by one. + +`smallwebwaf` is meant to fill that gap: + +- protect a service from misbehaving scrapers and scanners with per-client + request and byte limits over a minute, an hour and a day; +- bias those limits against the countries and AS numbers that abuse commonly + comes from, so their clients get a configured percentage of the normal + allowance rather than an outright block; +- remember abusers, block them for a while, and ban the ones who keep coming + back; +- log everything in a form that is easy to search and ship elsewhere; +- stay small enough to understand: one binary, one container, environment + variables, no database. + +## Proposed features + +- Reverse proxy for one upstream application, streaming in both directions, + with WebSocket support. One `smallwebwaf` per app. +- Real client address worked out from `X-Forwarded-For`, trusting only the + proxy networks you list. IPv6 clients are counted by /64. +- Rate limits per client on requests per minute, per hour and per day, and on + bytes per minute, per hour and per day. +- Netblocks that bypass rate limiting, netblocks that bypass everything, and + netblocks that are always refused. +- AS number and country lookup for every client from a local database file. +- Biased limits: listed AS numbers and countries get a percentage of every + limit, for example 50 percent for common abuse-source networks. Zero percent + refuses outright. +- Attack detection: + - a directory of plain text rule files, one regex per line, for catching + scanning and penetration probes; easy to edit by hand; + - the OWASP Core Rule Set, run by the Coraza engine, in detect-only or + blocking mode; + - trap paths and bursts of error responses. +- Offences add up to a temporary block. Block lengths grow for repeat offenders + (for example one hour, then a day, then a week) and end in a permanent ban. +- IP reputation: downloadable blocklists, DNS blocklists, AbuseIPDB, and an + optional feed of decisions from a CrowdSec engine. Lookups happen in the + background and never delay a request. +- Alerts on attacks and bans to a generic webhook, Slack or ntfy, with a + cooldown and an hourly cap so a wide attack cannot flood the channel. +- Anomaly alerts when requests or bytes per minute or hour cross a threshold, + for a single client, its surrounding netblock, an AS number, a named netblock + or the whole service. +- Observe mode: log and alert on every decision while refusing nothing, for the + first days in front of a new service. +- Request log: one JSON object per request on stdout with the usual web log + fields, the decision taken and why, AS number and country, and timings. + Optionally also sent to a remote syslog or RELP endpoint. +- Prometheus metrics on their own port. +- State (bans, offender history, hour and day counters, reputation cache) held + in memory and saved as readable, hand-editable JSON files, written atomically. + Nothing is read from disk while serving a request. +- A small admin endpoint for health checks, listing, adding and lifting bans, + and asking why a given address was refused. + +Not planned: TLS termination, routing for several apps, browser challenges +(captcha or proof of work), a web console, or defence against floods large +enough to fill the host's network link. + +## How it works, in short + +For each request `smallwebwaf`: + +- works out who the client really is; +- lets it straight through if it is on the bypass list, refuses it if it is on + the deny list or currently banned; +- looks up its AS number and country, and any cached reputation verdict; +- picks the client's limit percentage from those; +- checks the minute, hour and day request counters against the limits, and + answers 429 if one is exceeded; +- checks the request against the rule files and the Core Rule Set; +- forwards it to the app and streams the response back; +- counts the bytes, records any offence, bans the client if it has collected + enough of them, sends any alerts that are due, and writes the log line. + +A minimal deployment beside an app in docker-compose: + +```yaml +services: + app: + image: example/app + networks: [internal] + + waf: + image: /smallwebwaf: + environment: + UPSTREAM_URL: http://app:3000 + TRUSTED_PROXIES: 172.18.0.0/16 + MODE: observe + RATE_LIMIT_PER_MINUTE: "120" + RATE_LIMIT_PER_HOUR: "2000" + RATE_LIMIT_PER_DAY: "10000" + ASN_LIMIT_PERCENT: AS14061:50,AS16276:50 + COUNTRY_LIMIT_PERCENT: CN:25 + ALERT_NTFY_URL: https://ntfy.example.invalid/alerts + volumes: [waf-state:/data] + networks: [internal, traefik] + labels: + traefik.enable: "true" + traefik.http.routers.app.rule: Host(`app.example.invalid`) + traefik.http.services.app.loadbalancer.server.port: "8080" +``` + +A rule file is one rule per line: a name, what to match against, what to do, +and a regex. + +``` +env-file path offence:3 (?i)/\.env(\.[a-z]+)?$ +scanner-agent user_agent ban (?i)\b(sqlmap|nikto|nuclei|wpscan)\b +``` + +[`SPEC.md`](SPEC.md) has the full design: every environment variable, the rule +file format, the state files, the log fields, the metrics, failure behaviour, +the build order, and the design questions still open for the owner. + +## Documents + +- [`SPEC.md`](SPEC.md): the design. +- [`EVALUATION.md`](EVALUATION.md): what already exists, what each tool covers + and misses, and why none was adopted. diff --git a/SPEC.md b/SPEC.md new file mode 100644 index 0000000..2fb5ec6 --- /dev/null +++ b/SPEC.md @@ -0,0 +1,695 @@ +# smallwebwaf SPEC (draft): protective reverse-proxy sidecar + +Status: second draft, with the owner's rulings on storage, logging and metrics +applied. Nothing has been built yet. `EVALUATION.md` beside this file explains why no existing tool was +chosen. Points that turn on an owner decision are collected under "Questions for +the owner" and are marked "(open)" where they appear. + +## Purpose + +One small container that sits between traefik and one application container. The +traefik router for the public hostname points at the sidecar; the sidecar +forwards to the application. The sidecar limits request and byte rates per +client, detects attacks, blocks abusers for a time and bans repeat offenders, +consults IP reputation sources, looks up the AS number and country of each +client, scales its limits for listed AS numbers and countries, and sends alerts. +Everything is configured by environment variables, apart from the attack +detection rules, which are read from a directory of hand-editable text files. + +## Non-goals + +- TLS termination, certificates, hostname routing: traefik's job. +- More than one upstream application per sidecar. Run one sidecar per app. +- Browser challenges (captcha, proof of work). If wanted, chain Anubis between + the sidecar and the app. +- Defence against traffic floods that saturate the host's network link. That + needs help upstream of the host. +- A web UI or a configuration file. Settings are environment variables; the + only files read are the rule files, which hold one regex per line and nothing + more elaborate. +- Sharing ban state between sidecars in the first version (open, see + Questions). + +## Architecture + +- One statically linked Go binary in one container, running as a non-root user, + read-only root filesystem, one writable volume for state. +- Three listeners, of which only the first is routed by traefik: + - the proxy listener; + - a metrics listener serving Prometheus metrics, which can be switched off; + - an admin listener for health and ban management. +- Components inside the process: + - Client identification: works out the real client IP from + `X-Forwarded-For`, trusting only configured proxy netblocks. + - Lookup: AS number and country from local database files, held in memory. + - Reputation: static blocklists fetched on a schedule; DNSBL and reputation + API queries made in the background and cached. + - Counters: per-client request and byte counts per minute, hour and day, + held in memory; the hour and day counts are written out so they survive a + restart. + - Attack detection: regex rules read at start from a directory of plain + text rule files (see "Rule files"), Coraza with the OWASP Core Rule Set, + and simple signals (requests for listed trap paths, bursts of error + responses from the app). + - Ban ledger: offences, active bans and ban history, held in memory and + written to disk as JSON (see "Persistent state"). No database of any kind + is used. + - Request log: one JSON object per request on stdout, optionally also sent + to a remote syslog or RELP endpoint (see "Request log"). + - Metrics: Prometheus counters and gauges on their own listener (see + "Metrics endpoint"). + - Alerting: a queue with de-duplication feeding webhook, Slack and ntfy + senders. + - Proxy: the standard library's `net/http/httputil.ReverseProxy`, streaming + in both directions, with WebSocket upgrade support. +- Proposed libraries (to be checked against the owner's Go dependency defaults + before any are added): + - standard library for the proxy, HTTP clients, DNS, logging (`log/slog`), + state files (`encoding/json`, `os.Rename`); + - `github.com/corazawaf/coraza/v3` and + `github.com/corazawaf/coraza-coreruleset` for attack detection; + - `github.com/oschwald/maxminddb-golang` to read `.mmdb` lookup databases; + - `github.com/prometheus/client_golang` for metrics; + - remote log sending: the standard library's `log/syslog` is frozen and + does not write the current syslog format (RFC 5424), and no widely used Go + RELP client was found, so the library for this is open (see Questions). + +## Data flow for one request + +Steps run in this order; the first step that produces a final answer ends +processing. + +- Identify the client. + - If the TCP peer is inside `TRUSTED_PROXIES`, walk `X-Forwarded-For` from + the right and take the first address not inside `TRUSTED_PROXIES`. + Otherwise use the TCP peer address and ignore the header. + - IPv6 clients are grouped by prefix (`IPV6_GROUP_PREFIX`, default 64) for + counting and banning, because one abuser usually controls a whole /64. +- Static lists. + - In `ALLOW_NETS`: skip every check below and forward. Still counted for + anomaly alerts. + - In `DENY_NETS`: refuse. +- Ban ledger. An active ban: refuse (`BAN_RESPONSE`). +- Lookup AS number and country. Unknown is a valid result and changes nothing. +- Reputation. + - Address inside a fetched blocklist: apply `BLOCKLIST_ACTION`. + - Cached DNSBL or reputation API result: apply `REPUTATION_ACTION`. + - No cached result: queue a background query and carry on. A first request + is never delayed by a reputation query. +- Work out the client's limit percentage: the lowest of the percentages that + apply (AS number, country, reputation). 100 if none applies. 0 means refuse. + (Lowest-wins versus multiplying is open.) +- Request rate limits. Unless the client is in `RATE_LIMIT_EXEMPT_NETS`, check + the minute, hour and day counters against each configured limit times the + percentage. Over any limit: answer 429 with `Retry-After`, record an offence. +- Rule files. The request is checked against the rules loaded from + `RULES_DIR`, in file name order then line order. Each rule that matches takes + its action: record offences, refuse with 403, ban at once, or only log. + Matching stops at the first rule that refuses or bans. +- Core Rule Set inspection of the request (headers, URL, and body up to + `WAF_BODY_LIMIT`). In `block` mode a match at or over the anomaly threshold is + refused with 403 and recorded as an offence; in `detect` mode it is only + logged and alerted. +- Forward to `UPSTREAM_URL`, streaming. Add `X-Forwarded-For` and, if enabled, + `X-Client-ASN` and `X-Client-Country` for the app's own logs. +- After the response. + - Add response bytes (and request body bytes) to the byte counters. Once a + client's byte total passes a byte limit times the percentage, following + requests get 429 until the window moves on, and an offence is recorded. A + response already in progress is not cut off. + - Count 401, 403 and 404 responses from the app per client; passing + `ERROR_BURST_THRESHOLD` within a minute records an offence. + - Update anomaly counters and evaluate alert thresholds. +- Offences and bans. + - `BAN_AFTER_OFFENCES` offences within `OFFENCE_WINDOW` creates a ban. + - Ban length follows `BAN_DURATIONS` by the number of earlier bans for that + client within `BAN_HISTORY_WINDOW`: first ban the first value, second ban + the second value, and so on. + - After `PERMANENT_BAN_AFTER` bans within `BAN_HISTORY_WINDOW`, the ban has + no expiry. + - A request for a path in `TRAP_PATHS` counts as `TRAP_OFFENCE_WEIGHT` + offences at once. + +## Counting method + +- Each window (minute, hour, day) uses two adjacent fixed buckets per client, + with the previous bucket weighted by how much of it still overlaps the + sliding window. This costs a few integers per client per window and avoids + the burst at bucket boundaries that a single fixed bucket allows. +- Counters are read and updated in memory only. The hour and day counters are + written to disk on a timer and at shutdown and loaded again at start (see + "Persistent state"), so a restart does not hand every client a fresh daily + allowance. Minute counters are not written; a restart forgives at most one + minute of counting. Buckets whose time has passed are discarded on load. +- Memory is bounded by `MAX_TRACKED_CLIENTS`; when full, the least recently seen + clients with no recent offences are dropped first. + +## Configuration surface + +Conventions: lists are comma separated; netblocks are CIDR (a bare address means +/32 or /128); durations use Go syntax plus `d` for days (`90s`, `15m`, `24h`, +`7d`); byte sizes accept `K`, `M`, `G` suffixes; an unset limit means that +limit is off. Every variable may instead be given as `NAME_FILE` pointing at a +file holding the value, for secrets and long lists. Invalid configuration stops +the process at start with a message naming the variable. At start the effective +configuration is logged with secrets masked. + +- Core + - `UPSTREAM_URL` (required): the application, for example + `http://gitea:3000`. + - `LISTEN_ADDR` (default `:8080`): proxy listener. + - `ADMIN_LISTEN_ADDR` (default `127.0.0.1:9090`): admin listener. + - `ADMIN_TOKEN`: bearer token required for ban management endpoints. + - `INSTANCE_NAME`: included in every log line, metric and alert, for example + `fsn1app1/gitea`. + - `MODE` (default `enforce`): `enforce`, or `observe` to log and alert on + every decision while refusing nothing. Meant for the first days of a + rollout. + - `TRUSTED_PROXIES` (required): netblocks whose `X-Forwarded-For` is + believed, normally the docker network traefik reaches the sidecar from. + - `IPV6_GROUP_PREFIX` (default `64`). + - `MAX_TRACKED_CLIENTS` (default `500000`). +- Persistent state + - `STATE_DIR` (default `/data`): the JSON state files and downloaded lookup + databases. + - `STATE_WRITE_DELAY` (default `2s`): after a ban or offence changes, wait + this long before writing, so a burst of changes becomes one write. + - `STATE_COUNTER_INTERVAL` (default `60s`): how often the hour and day + counters and the reputation cache are written. +- Logging + - The request log on stdout is always on and has no switch. + - `LOG_LEVEL` (default `info`): for the process's own messages (start-up, + fetch failures, state writes), which are JSON lines on stdout too, marked + `"type":"process"`. It does not filter the request log. + - `LOG_REQUEST_HEADERS` (default + `accept,accept-language,accept-encoding,content-type,origin,range`): + extra request headers to record. `Authorization`, `Cookie` and + `Set-Cookie` values are never logged, only whether they were present. + - `LOG_REMOTE_URL`: when set, every log line is also sent to this endpoint. + Forms: `syslog+udp://host:514`, `syslog+tcp://host:514`, + `syslog+tls://host:6514`, `relp://host:2514`, `relp+tls://host:2514`. + - `LOG_REMOTE_TLS_CA_FILE`: optional CA certificate for the `+tls` forms. + - `LOG_REMOTE_BUFFER` (default `10000`): lines held in memory while the + endpoint is unreachable; when full the oldest are dropped and counted. + - `LOG_REMOTE_FACILITY` (default `local0`), `LOG_REMOTE_APP_NAME` (default + `INSTANCE_NAME`): syslog header fields. +- Metrics + - `METRICS_ENABLED` (default `true`). + - `METRICS_LISTEN_ADDR` (default `:9100`): its own listener, so it can be + bound to a monitoring network without exposing the admin endpoints. + - `METRICS_PATH` (default `/metrics`). + - `METRICS_TOP_N` (default `50`): how many AS numbers and countries get + their own series; the rest are summed as `other`. + - `METRICS_TOKEN`: optional bearer token; unset means no authentication, + which is the usual arrangement for a scraper on a private network. +- Static lists + - `ALLOW_NETS`: bypass everything (monitoring, the owner's own networks). + - `RATE_LIMIT_EXEMPT_NETS`: bypass request and byte limits only; attack + detection and bans still apply. + - `DENY_NETS`: always refused. +- Request rate limits, per client (R1, R2) + - `RATE_LIMIT_PER_MINUTE`, `RATE_LIMIT_PER_HOUR`, `RATE_LIMIT_PER_DAY`. + - `RATE_LIMIT_EXEMPT_PATHS`: path prefixes not counted (static assets, + health checks). +- Byte limits, per client + - `BYTES_LIMIT_PER_MINUTE`, `BYTES_LIMIT_PER_HOUR`, `BYTES_LIMIT_PER_DAY`. + - `BYTES_COUNT` (default `both`): `response`, `request` or `both`. +- Lookup of AS number and country (R7) + - `ASN_DB_PATH`, `COUNTRY_DB_PATH`: `.mmdb` files mounted into the + container; or + - `ASN_DB_URL`, `COUNTRY_DB_URL`: downloaded into `STATE_DIR` at start and + every `LOOKUP_DB_REFRESH` (default `7d`). One file may serve both. + - Known free sources in this format: IPinfo Lite (country and AS number in + one file, free account token), DB-IP Lite, MaxMind GeoLite2 (free + account). Choice is open. + - `ADD_LOOKUP_HEADERS` (default `false`): pass `X-Client-ASN` and + `X-Client-Country` to the app. + - Missing or unreadable database: start anyway, treat every client as + unknown, log a warning and send one alert. +- Biased thresholds (R8) + - `ASN_LIMIT_PERCENT`: for example `AS14061:50,AS16276:50,AS45102:25`. + Listed AS numbers get that percentage of every request and byte limit. + `0` refuses outright. + - `COUNTRY_LIMIT_PERCENT`: same form with ISO country codes, for example + `CN:25,RU:50`. + - `ASN_BYTES_PERCENT`, `COUNTRY_BYTES_PERCENT`: optional overrides applied + to byte limits only, when the byte percentage should differ from the + request percentage. + - `UNKNOWN_LIMIT_PERCENT` (default `100`): for clients the lookup cannot + place. + - `ASN_LIMIT_PERCENT_URL`: optional URL of a text file of `AS:percent` + lines, so one abuse-source list can be shared by every sidecar in the + fleet. +- Attack detection (R9) + - `RULES_DIR` (default `/etc/smallwebwaf/rules.d`): directory of rule files + read once at start; format under "Rule files". The image ships a default + file there. Mounting a directory over it replaces the defaults; mounting + single files into it adds to them. + - `RULES_ENABLED` (default `true`): `false` skips rule files entirely. + - `WAF_MODE` (default `detect`): `off`, `detect`, `block`. + - `WAF_PARANOIA_LEVEL` (default `1`), `WAF_ANOMALY_THRESHOLD` (default `5`): + the Core Rule Set's own two tuning values. + - `WAF_DISABLED_RULES`: rule ids to switch off when an app trips a false + positive. + - `WAF_EXEMPT_PATHS`: path prefixes not inspected. + - `WAF_BODY_LIMIT` (default `128K`): bodies are inspected up to this size + and streamed beyond it without buffering, so large uploads and git pushes + are not held in memory. + - `TRAP_PATHS`: paths the app never serves and only scanners ask for, for + example `/.env,/wp-login.php,/.git/config`. `TRAP_OFFENCE_WEIGHT` + (default `3`). This is the env-var short form of a `path` rule with an + `offence` action, for deployments that mount no rule files. + - `ERROR_BURST_THRESHOLD` (default off): 401, 403 and 404 responses per + client per minute that count as an offence. +- Blocks and bans (R5) + - `BAN_AFTER_OFFENCES` (default `3`), `OFFENCE_WINDOW` (default `10m`). + - `BAN_DURATIONS` (default `1h,24h,7d`). + - `PERMANENT_BAN_AFTER` (default `4`; `0` never bans permanently), + `BAN_HISTORY_WINDOW` (default `90d`). + - `BAN_RESPONSE` (default `403`): `403`, `429`, or `close` to drop the + connection without an answer. + - `BAN_SCOPE_V4_PREFIX` (default `32`): widen to for example `24` to ban the + surrounding netblock. +- Reputation (R6) + - `BLOCKLIST_URLS`: text files of addresses and netblocks, for example + Spamhaus DROP, FireHOL level 1, the Tor exit list. Refreshed every + `BLOCKLIST_REFRESH` (default `6h`); the last good copy is kept on failure. + - `BLOCKLIST_ACTION` (default `limit:25`): `deny`, `limit:`, or + `log`. + - `DNSBL_ZONES`: for example `dnsbl.dronebl.org`. Zones meant for mail + (lists of residential ranges) will block ordinary visitors and should not + be used; Spamhaus zones need their keyed query service, given as a zone + name containing the key. + - `DNSBL_RESOLVER`: optional resolver address, since public resolvers are + refused by several list operators. + - `ABUSEIPDB_KEY`, `ABUSEIPDB_MIN_SCORE` (default `75`), + `ABUSEIPDB_DAILY_BUDGET` (default `900`; the free tier allows 1000 checks + a day). Only clients that have already committed one offence are queried, + so the budget is spent on suspects. + - `CROWDSEC_LAPI_URL`, `CROWDSEC_LAPI_KEY`: optional. If a CrowdSec engine + exists on the host, pull its decision list on a schedule and treat listed + addresses as banned. This is how a fleet-wide blocklist can arrive + without the sidecar depending on CrowdSec. + - `REPUTATION_ACTION` (default `limit:25`): `deny`, `limit:`, or + `log`, for DNSBL and API hits. + - `REPUTATION_CACHE_TTL` (default `24h`), `REPUTATION_TIMEOUT` (default + `2s`). +- Alerting (R3) + - `ALERT_WEBHOOK_URL`: JSON POST; schema below. + `ALERT_WEBHOOK_HEADERS`: optional `Name:value` pairs for authentication. + - `ALERT_SLACK_WEBHOOK_URL`: Slack incoming webhook, formatted message. + - `ALERT_NTFY_URL` (full topic URL), `ALERT_NTFY_TOKEN`: title, priority + and tags set from the event type. + - `ALERT_EVENTS` (default + `ban,permanent_ban,waf_block,anomaly,reputation_hit,source_failure`): + which event types are sent. + - `ALERT_COOLDOWN` (default `15m`): the same event type for the same client + or netblock is not repeated within this time; a count of suppressed + repeats is included in the next one. + - `ALERT_MAX_PER_HOUR` (default `60`): beyond this, alerts are rolled into + one summary per hour so a wide attack cannot flood the channel. +- Anomaly thresholds, alert only, nothing is blocked (R4) + - Per client: `ANOMALY_CLIENT_REQUESTS_PER_MINUTE`, + `ANOMALY_CLIENT_REQUESTS_PER_HOUR`, `ANOMALY_CLIENT_BYTES_PER_MINUTE`, + `ANOMALY_CLIENT_BYTES_PER_HOUR`. + - Per surrounding netblock (`ANOMALY_NET_V4_PREFIX` default `24`, + `ANOMALY_NET_V6_PREFIX` default `48`): `ANOMALY_NET_REQUESTS_PER_MINUTE`, + `..._PER_HOUR`, `ANOMALY_NET_BYTES_PER_MINUTE`, `..._PER_HOUR`. + - Per AS number: `ANOMALY_ASN_REQUESTS_PER_MINUTE`, `..._PER_HOUR`, + `ANOMALY_ASN_BYTES_PER_MINUTE`, `..._PER_HOUR`. + - Whole service: `ANOMALY_TOTAL_REQUESTS_PER_MINUTE`, `..._PER_HOUR`, + `ANOMALY_TOTAL_BYTES_PER_MINUTE`, `..._PER_HOUR`. + - Named netblocks: `WATCH_NETS`, for example + `office=203.0.113.0/24,scraper-x=198.51.100.0/22`, with + `WATCH_REQUESTS_PER_MINUTE`, `..._PER_HOUR`, `WATCH_BYTES_PER_MINUTE`, + `..._PER_HOUR` applied to each named block as a whole. + - Anomaly counting includes clients in `ALLOW_NETS` and + `RATE_LIMIT_EXEMPT_NETS`, since an exempt client misbehaving is worth + knowing about. + +## Rule files + +A required feature: at start the sidecar reads every `*.rules` file in +`RULES_DIR` and checks each request against them. The files are plain text meant +to be edited by hand, so that a new scanning pattern seen in the request log can +be turned into a rule in one line. + +- One rule per line, four fields separated by spaces or tabs; the fourth field + runs to the end of the line: + + ``` + + ``` + +- Blank lines and lines starting with `#` are ignored. +- `id`: a short name of letters, digits, `-` and `_`, unique across all files. + It appears in the request log, metrics, alerts and offence history. +- `target`: what the regex is matched against. + - `path`: the URL path as received, before any decoding. + - `query`: the raw query string. + - `uri`: path and query together, both as received and once + percent-decoded, so an encoded probe cannot slip past. + - `method`, `host`, `user_agent`, `referer`. + - `header:`: any one request header. + - Request bodies are not available to rule files; body inspection is the + Core Rule Set's job. +- `action`: + - `log`: note the match in the request log and do nothing else. + - `offence` or `offence:`: record one or `n` offences and forward the + request. Enough offences within `OFFENCE_WINDOW` creates a ban. + - `block` or `block:`: refuse with 403 and record one or `n` offences. + - `ban`: refuse and ban the client at once, at whatever length its ban + history calls for. +- `regex`: Go regular expression syntax (RE2). It has no backreferences or + lookaround, and in exchange matching time is linear in the input, so no rule + can be made to stall the proxy. `(?i)` at the front makes a rule + case-insensitive. A rule matches if the regex matches anywhere in the target; + anchor with `^` and `$` when that is not wanted. +- Files are read in name order (`00-default.rules` before `50-gitea.rules`), + rules in line order. +- Rules are compiled once at start and held in memory. A line that does not + parse, a regex that does not compile, or a duplicate id stops the process + with a message naming the file and line. A missing or empty directory is not + an error: the log says that no rules were loaded. Changing rules means + restarting the container; reloading while running is given up for now. +- In `MODE=observe` every action is logged as what would have happened and + nothing is refused. +- Clients in `ALLOW_NETS` are not checked. + +Example file: + +``` +# 00-default.rules: probes no real visitor sends + +# id target action regex +env-file path offence:3 (?i)/\.env(\.[a-z]+)?$ +git-dir path offence:3 ^/\.git/(config|HEAD|index)$ +wp-probe path offence:3 (?i)^/(wp-login\.php|xmlrpc\.php|wp-admin/) +php-shell path block:3 (?i)/(shell|c99|r57|wso|alfa)\.php$ +path-traversal uri block (\.\./){2,} +scanner-agent user_agent ban (?i)\b(sqlmap|nikto|nuclei|masscan|zgrab|wpscan)\b +empty-agent user_agent log ^$ +``` + +The image ships one default file of this kind. It is kept short and limited to +patterns that are wrong for every app; anything app-specific (for example +blocking `/wp-login.php` is harmless in front of gitea and fatal in front of +WordPress) belongs in a file the deployer mounts. + +## Persistent state + +All state lives in memory. No database is used. What must survive a restart is +written to `STATE_DIR` as JSON files; the files are read once at start and never +during a request. The whole state is expected to stay under 10 MiB, and reading +or writing even ten times that as JSON takes well under a second, so whole-file +rewrites are cheap enough to need nothing cleverer. + +- Files, each holding one kind of state: + - `bans.json`: active and past bans. Per entry: client or netblock, start, + expiry (`null` for permanent), reason, which ban this is for that client + (first, second, third), source (`local`, `admin`, `crowdsec`), and when + it was lifted, if it was. + - `offences.json`: offender history. Per entry: client, time, kind (`rate`, + `bytes`, `waf`, `trap`, `error_burst`), short detail. Entries older than + `BAN_HISTORY_WINDOW` are dropped at each write. + - `counters.json`: the hour and day request and byte counters per client. + - `reputation.json`: cached DNSBL and reputation API verdicts with the time + each was fetched. +- Format: indented JSON with a top-level `version` number, entries sorted by + client address, times in RFC 3339 UTC, durations and sizes as plain numbers + with the unit in the field name. The aim is that a person can open + `bans.json` in an editor, find an address, and remove or add an entry. +- Writing: + - serialise from a snapshot taken under the lock, so requests are not held + up while the file is written; + - write to a temporary file in the same directory, sync it, rename it over + the real name, sync the directory. A crash at any point leaves either the + old complete file or the new complete file, never a partial one; + - `bans.json` and `offences.json` are written `STATE_WRITE_DELAY` after a + change; `counters.json` and `reputation.json` every + `STATE_COUNTER_INTERVAL`; all four on orderly shutdown (SIGTERM). +- Reading: + - at start only. Active bans go into an in-memory prefix lookup; everything + else into maps. Expired counters and cache entries are discarded. + - a missing file means empty state and is normal on first run. + - a file that does not parse, or has an unknown `version`, stops the + process with a message naming the file and position. Starting with empty + bans would silently forgive every repeat offender, and since writes are + atomic a broken file can only come from a hand edit, which the editor + should hear about. `counters.json` and `reputation.json` are the + exception: if unreadable they are logged, set aside with a `.bad` suffix, + and the process starts with them empty. +- Hand edits are made with the container stopped; a running process will + overwrite the file at its next write. Changes to a running process go through + the admin endpoints. Reloading files into a running process is given up for + now. +- `STATE_DIR` not writable at start: the process exits. A write that fails + while running: state stays correct in memory, the failure is logged, counted + in metrics, alerted once per cooldown, and retried at the next write. +- What a hard kill can lose: bans and offences from the last + `STATE_WRITE_DELAY`, counters from the last `STATE_COUNTER_INTERVAL`. Losing + the volume loses ban history, not service. + +## Request log + +One JSON object per line on stdout for every request, including refused ones. +stdout is always on. When `LOG_REMOTE_URL` is set the same lines are also sent +to the remote endpoint, so a deployment can stop depending on docker's log +handling while `docker logs` keeps working. + +- Standard web log fields: `time` (RFC 3339 with milliseconds), `instance`, + `client_ip`, `method`, `scheme`, `host`, `path`, `query`, `protocol`, + `status`, `request_bytes`, `response_bytes`, `referer`, `user_agent`. +- Request detail: `request_id` (generated if traefik did not supply one, and + passed to the app), `peer_ip` (the TCP peer, normally traefik), + `forwarded_for` (the header as received), `client_group` (the /64 or + configured prefix used for counting), `asn`, `as_name`, `country`, + `content_type`, `content_length`, the headers named in + `LOG_REQUEST_HEADERS`, `has_authorization` and `has_cookie` as booleans, + `websocket` when the connection was upgraded. +- Response detail: `response_content_type`, `upstream_status` (differs from + `status` when the sidecar answered itself), `cache_control`, `location` on + redirects, `aborted` when the client went away early. +- Decision: `action` (`forward`, `rate_limited`, `bytes_limited`, `banned`, + `denied`, `waf_blocked`, `upstream_error`), `would_action` in `observe` mode, + `limit_percent` and which rule set it, `counts` (the client's minute, hour + and day request and byte totals after this request), `limit_hit` (which + window), `rule_ids` (rule file rules that matched), `waf_rule_ids`, + `waf_score`, `reputation` (sources that listed the + client), `offence` when one was recorded, `ban_expires`. +- Timings in milliseconds: `duration_total`, `duration_checks` (everything the + sidecar did before forwarding), `duration_waf`, `duration_upstream_connect`, + `duration_upstream_first_byte`, `duration_upstream_total`. +- Bodies are never logged. Query strings are logged as received; an app that + carries secrets in query strings needs that fixed in the app. +- The process's own messages share the stream as JSON lines with + `"type":"process"`; request lines carry `"type":"request"`. +- Remote sending: + - syslog forms send each line as the message of an RFC 5424 record, with + octet-counted framing on TCP and TLS; + - RELP forms send the same record and wait for the receiver's + acknowledgement, so lines are not lost across a receiver restart. This is + the form to use with rsyslog; + - sending happens on its own goroutine from a bounded buffer + (`LOG_REMOTE_BUFFER`). An unreachable or slow endpoint never delays a + request and never stops stdout; it reconnects with backoff, drops the + oldest lines when the buffer is full, and counts the drops in metrics. + UDP gives no delivery signal at all and is offered only for + compatibility. + +## Metrics endpoint + +Prometheus text format on `METRICS_LISTEN_ADDR` at `METRICS_PATH`, on by default, +switched off with `METRICS_ENABLED=false`. It has its own listener so it can be +reached by a scraper without exposing ban management. + +- Traffic: requests and bytes in and out, by status class and `action`; + request duration and upstream duration histograms; requests in flight. +- Limits and bans: limit hits by window and kind (requests or bytes), offences + by kind, bans created by ordinal, permanent bans, active bans (gauge). +- Attack detection: rule file matches by rule id and action, and the number of + rules loaded; Core Rule Set matches by mode and rule id (label limited to the + rules that actually fired). +- Lookup and reputation: requests and bytes by AS number and by country, limited + to the `METRICS_TOP_N` (default `50`) busiest of each with the rest summed as + `other`, so the label set stays bounded; reputation queries, hits, failures + and remaining daily budget by source; age of each blocklist and lookup + database. +- Housekeeping: tracked clients (gauge), state file writes, write failures, + last successful write time and size per file; alerts sent, failed and + suppressed by destination; remote log lines sent, dropped and buffer depth; + the standard Go runtime and process metrics. +- No metric carries a client IP address as a label; per-address questions are + answered by the request log and `GET /clients/`. + +## Admin listener + +- `GET /healthz`: for the container health check. +- `GET /bans`, `POST /bans` (client or netblock, duration, reason), + `DELETE /bans/`: need `ADMIN_TOKEN`. +- `GET /clients/`: current counters, lookup result, reputation, offences; + for answering "why was this address refused". + +## Alert webhook schema + +One JSON object per alert: + +- `instance`, `time`, `event` (one of the `ALERT_EVENTS` values) +- `client`, `netblock`, `asn`, `as_name`, `country` +- `reason`: short human-readable sentence +- `detail`: event-specific fields, for example `window`, `count`, `limit`, + `limit_percent`, `rule_ids`, `path`, `ban_ordinal`, `ban_expires` +- `suppressed_repeats`: number of identical alerts held back by the cooldown + +## Deployment as a sidecar + +docker-compose shape, using gitea as the example; upaas deployments follow the +same shape if upaas can run a second container for a service (open): + +```yaml +services: + gitea: + image: gitea/gitea:1 + # no traefik labels, no published ports + networks: [internal] + + gitea-guard: + image: /: + read_only: true + user: "65532:65532" + environment: + UPSTREAM_URL: http://gitea:3000 + TRUSTED_PROXIES: 172.18.0.0/16 + INSTANCE_NAME: fsn1app1/gitea + MODE: observe + RATE_LIMIT_PER_MINUTE: "120" + RATE_LIMIT_PER_HOUR: "2000" + RATE_LIMIT_PER_DAY: "10000" + BYTES_LIMIT_PER_HOUR: 2G + ASN_DB_URL: https://example.invalid/asn-country.mmdb + ASN_LIMIT_PERCENT: AS14061:50,AS16276:50 + COUNTRY_LIMIT_PERCENT: CN:25 + WAF_MODE: detect + TRAP_PATHS: /.env,/wp-login.php,/.git/config + BLOCKLIST_URLS: https://www.spamhaus.org/drop/drop.txt + ALERT_NTFY_URL: https://ntfy.example.invalid/fleet-alerts + LOG_REMOTE_URL: relp://logs.example.invalid:2514 + METRICS_LISTEN_ADDR: :9100 + volumes: + - gitea-guard-state:/data + - ./50-gitea.rules:/etc/smallwebwaf/rules.d/50-gitea.rules:ro + networks: [internal, traefik] + labels: + traefik.enable: "true" + traefik.http.routers.gitea.rule: Host(`git.example.invalid`) + traefik.http.services.gitea.loadbalancer.server.port: "8080" +``` + +- The application container leaves the traefik network, so the sidecar cannot + be bypassed. +- Traefik routes only port 8080. The metrics port is reached by the scraper + over a docker network it shares with the sidecar; the admin port stays on + loopback inside the container and is used through `docker exec`. +- The state volume holds a few small JSON files; it needs no backup beyond + whatever the host already does. +- SSH access to gitea does not pass through the sidecar and is not protected + by it. +- Rollout per service: start in `MODE=observe` with `WAF_MODE=detect`, read a + few days of logs and alerts, set limits and rule exclusions from what real + traffic looks like, then switch to `enforce` and `block`. +- Notes specific to gitea: clones and pushes over HTTP are large and long, so + byte limits must be sized for a legitimate clone, the proxy sets no overall + response timeout, and `WAF_BODY_LIMIT` keeps pack uploads out of memory. + Archive download and blame or history pages are what scrapers hammer; request + limits do most of the work there. + +## Failure behaviour + +- Lookup database missing: clients are unknown, service continues, one alert. +- Reputation source down or over quota: no verdict, service continues, one + `source_failure` alert per cooldown. +- Alert destination down: retried with backoff from a bounded queue, oldest + dropped first, drops counted in metrics. +- Upstream down: 502 from the sidecar, not counted as client offences. +- Attack detection engine error on a request: request is forwarded, error + logged and counted. +- Remote log endpoint down: stdout continues, lines are buffered then dropped + oldest first, drops counted in metrics. +- State file write fails while running: memory stays authoritative, logged, + counted, alerted, retried. +- In short: the only things that stop the process are invalid configuration, + a rule file that does not parse, an unwritable `STATE_DIR`, and an unparseable `bans.json` or `offences.json` + at start. A broken helper never takes the protected service down. + +## Risks the design has to handle + +- Forged `X-Forwarded-For`: handled by `TRUSTED_PROXIES` being required and by + walking the header from the right. +- Many clients behind one address (mobile carriers, offices, Tor): rate limits + punish them collectively; `RATE_LIMIT_EXEMPT_NETS` and sensible day limits + are the remedy, and bans on such addresses should stay short. +- IPv6 address rotation inside a /64: handled by grouping. +- Widely distributed scrapers using thousands of addresses at low rates each: + per-client limits do not see them. The AS number and netblock anomaly + alerts reveal them, and `ASN_LIMIT_PERCENT` at a low value or `0` is the + response. A per-AS-number or per-country total limit is open. +- Core Rule Set false positives against real apps (gitea's editor, API + payloads): `detect` by default, exclusions by rule id and path. +- Slow-request attacks: header read timeout and idle timeout on the listener. +- Alert floods: cooldown and hourly cap. + +## Build order + +- First: proxy, client identification, static lists, three-window request + limits, exemptions, `observe` mode, the full request log on stdout, the + metrics endpoint, health. +- Second: rule files, ban ledger with escalation and permanent bans, the JSON + state files, + admin endpoints, alerting to all three destinations, remote log sending. +- Third: AS number and country lookup, biased thresholds, byte limits, anomaly + thresholds. +- Fourth: blocklists, DNSBL, AbuseIPDB, optional CrowdSec decision feed. +- Fifth: attack detection with Coraza and the Core Rule Set, trap paths, error + bursts. +- Each stage is usable on its own; the first three already cover the traffic + problem the fleet has today. + +## Questions for the owner + +- Combining percentages. A client in a listed AS number (50 percent) and a + listed country (25 percent): should it get the lowest (25 percent of the + normal limit) or the product (12.5 percent)? Lowest is easier to predict when + reading a config; product punishes overlap harder. Recommendation: lowest. +- Meaning of "certain countries getting a percentage rate or byte limit". Read + here as: each client from that country gets that percentage of the normal + per-client limits. The other reading is a cap on the country as a whole, for + example all of one country together may use at most 20 percent of the + service's hourly byte budget. The second needs a configured total budget and + means one heavy client can lock out its compatriots. Recommendation: build + the per-client reading first; add whole-country and whole-AS-number caps only + if distributed scrapers make it necessary. +- Sharing bans across the fleet. As drafted, each sidecar remembers only its + own bans, so an abuser banned at gitea starts fresh at the next service. + Options: keep it per sidecar; or run one CrowdSec engine per host and have + every sidecar read its decision list (already supported above) and also + report its own bans to it. Recommendation: per sidecar for the first version, + decide on CrowdSec after seeing real ban volumes. +- Lookup database source. IPinfo Lite (one file with country and AS number, + free account, attribution required), DB-IP Lite (no account, attribution + required), or MaxMind GeoLite2 (free account, licence terms on + redistribution). Recommendation: IPinfo Lite, fetched once centrally and + served to the fleet from an internal URL so sidecars hold no account token. +- upaas. Whether upaas can deploy a second container beside an app and move + the traefik labels onto it was not examined. If it cannot, that is a feature + request for upaas, not something this sidecar can solve. +- Remote log library. Sending syslog in the current format (RFC 5424) is not + covered by Go's standard library, whose `log/syslog` package is frozen and + writes the older format, and no widely used Go RELP client was found. The + choices are a small third-party package for each, or writing both inside this + project; the syslog record is a few dozen lines, a RELP client with + acknowledgement handling is a few hundred. Recommendation: write the syslog + sender in-project, ship it first, and decide on RELP after looking at the + available packages; say if RELP must be in the first release. +- Ban response. `403` tells the abuser they were noticed; `close` wastes less + and tells them nothing. Recommendation: `403` by default for debuggability, + `close` available.