smallwebwaf/SPEC.md

smallwebwaf specification

smallwebwaf is a web application firewall for self-hosted apps. It is one binary, sww, and two of its commands do the work:

• sww rproxy is a reverse proxy between traefik and one app.
• sww discriminator decides what happens to each request, for many sites.

This document says what it must do and why, each requirement once. Everything in it is built, except what the last section lists. README.md says how to run it. EVALUATION.md says why no existing tool was adopted. The reference documents under docs/ give the settings, the formats and the lists. This document links to them.

Purpose

• It protects self-hosted apps on the open internet from scrapers, scanners and other abusers. It does so from an app's first request. rproxy requires only the address of discriminator. In a container of its own it also needs the app's address. Every other setting has a default chosen for a service facing the internet in 2026. A setting that needs an alert destination, an account key or a token stays off until the operator supplies it.
• It keeps a site answering while the part that decides is down, slow or restarting (see "Without an answer from discriminator").
• It limits each client's requests and bytes per minute, hour and day. It bounds the size and duration of every request and answer. It detects attacks.
• It bans abusers: briefly at first, for seven days on a clear sign of attack, and permanently when they keep at it.
• It looks up each client's AS number and country. It can refuse whole countries and netblocks, lower its limits for listed AS numbers and countries, consult reputation sources, and send alerts.
• Everything is set by environment variables. The names of discriminator's start with SWW_WAF_, and those of rproxy with SWW_RPROXY_. The only other inputs are files:
  • discriminator's rule files;
  • discriminator's state files;
  • discriminator's lookup database;
  • a SWW_WAF_ or SWW_RPROXY_ setting given as a file;
  • the file SWW_WAF_LOG_REMOTE_TLS_CA_FILE names;
  • the file SWW_WAF_CORE_RULE_SET_EXCLUSIONS_FILE names.

Non-goals

• TLS, certificates and routing by host name. Those are traefik's job.
• More than one app behind one rproxy. Each app has its own.
• Browser challenges such as captchas or proof of work. If wanted, chain Anubis between rproxy and the app.
• Defence against floods that saturate the host's network link. That needs help upstream of the host.
• A web console or a configuration file.

Architecture

• One statically linked Go binary. Only discriminator writes to disk, and only to its state directory. It ships in one image, which runs either command (see "Deployment").
• No database of any kind. Nothing is read from disk while serving a request.
• Packages:
  • cmd/sww: the binary, which only calls internal/sww.
  • internal/sww: the process, as the command it is given. sww discriminator reads the settings, the rule files, the lookup database and the state files, listens, and serves until SIGTERM or SIGINT. Then it stops, writing the state files (see "discriminator"). sww rproxy reads only that command's settings, and runs internal/rproxy until SIGTERM or SIGINT (see "rproxy"). sww healthcheck is the image's health check. With no command, or one it does not know, the binary prints the commands and exits with status 2.
  • internal/config: reads the settings, the one place they are read.
  • internal/discriminator: discriminator's server. It runs the checks in check, and holds discriminator's question, answer and report. It answers discriminator's own endpoints in answerOwnEndpoint. It counts bytes, refusals and anomalies, and writes the log lines.
  • internal/rproxy: the reverse proxy in front of one app that asks discriminator about each request. It passes a request to the app with the standard library's httputil.ReverseProxy, within the time and size limits.
  • internal/ratelimit: the table of clients. It counts requests, bytes and refusals, and keeps each client's history.
  • internal/bans: the bans. It holds each netblock's bans with their notes, decides how long a new ban lasts, and picks the ban to drop when the table of bans is full.
  • internal/rules: reads the rule files at start and again when they change, and tells which rules a request matches.
  • internal/waf: the Core Rule Set, with the eight changes listed under "Attack detection", as Coraza directives.
  • internal/lookup: looks clients up through GeoJS or in the lookup database.
  • internal/reputation: fetches the lists, asks the DNS blocklists and AbuseIPDB in the background, and keeps their answers.
  • internal/anomaly: the anomaly counters and their alerts.
  • internal/alerts: holds back repeats and alerts past the hourly limit, and sends the rest to the webhook, Slack and ntfy, each from a queue of its own.
  • internal/state: reads the state files at start, takes in edits while running, and writes the files when due and at the stop.
  • internal/requestlog and internal/remotelog: the lines on stdout, and sending them to the syslog server.
  • internal/metrics: the metrics, in the Prometheus text format.
  • Dockerfile: the lint and test phases, then the image. deploy/example-app is an app built on the image, with rproxy in its container, which script/example-app checks with discriminator in another.
• Libraries, checked against the owner's Go dependency defaults:
  • the standard library for the reverse proxy, HTTP clients, DNS, logging (log/slog) and the state files (encoding/json, os.Rename);
  • github.com/corazawaf/coraza/v3 and github.com/corazawaf/coraza-coreruleset/v4 at v4.25.0, which carries the Core Rule Set 4.25.0. The changes smallwebwaf makes to it, and the default SWW_WAF_CORE_RULE_SET_DISABLED_RULES, are built for that version. They are checked again before the version changes, since rule ids, and what each rule matches, change between versions;
  • github.com/hashicorp/golang-lru/v2 for the bounded tables;
  • github.com/oschwald/maxminddb-golang/v2 to read the lookup database;
  • github.com/fsnotify/fsnotify to notice edited files;
  • github.com/prometheus/client_golang for the metrics;
  • github.com/maxmind/mmdbwriter, in the tests, to write lookup databases;
  • remote log sending written in this project, since the standard library's log/syslog is frozen and writes only the older syslog format.

What happens to a request

rproxy runs steps 1, 2 and 14, and sends the report of step 15. discriminator runs steps 3 to 13, and does the counting of step 15. The first step that produces a final answer ends the checks. Step 15 runs for each request discriminator answers with allow, deny or close. rproxy's size and time limits apply to the whole exchange. Each step names the section that holds its rules.

1. GET /_sww/healthz is answered at once by rproxy, before any check, so a health checker never needs an exemption and is never refused.

2. The client is identified.

   • If the TCP peer is inside SWW_RPROXY_TRUSTED_PROXIES, X-Forwarded-For is walked from the right, and the first address outside those netblocks is the client. If every address is inside them, the leftmost is, so that a visitor on a private network is known by its own address. An entry that is not an address ends the walk. With no header, the peer is the client.
   • If the peer is outside SWW_RPROXY_TRUSTED_PROXIES, it is the client, and the header is ignored.
   • rproxy then sends discriminator the question (see "The question"). discriminator takes the address the question gives as the client.
   • An IPv6 client is its group: the netblock of SWW_WAF_IPV6_GROUP_PREFIX around its address, a /64 by default, since one abuser usually controls a whole /64. A client is therefore one IPv4 address or one IPv6 group.

3. The static lists. A client in SWW_WAF_ALLOW_NETS skips steps 4 to 12, the byte limits and the error burst. A client in SWW_WAF_DENY_NETS is refused; a client in both lists is let through.

4. Bans. A client under a ban is refused (see "Bans").

5. The lookup, unless SWW_WAF_LOOKUP_SOURCE is off (see "Lookup of AS number and country"). The request goes on at once. It waits for the first GeoJS answer, up to SWW_WAF_LOOKUP_TIMEOUT, only when one of these needs it:

   • a country list;
   • a biased threshold;
   • SWW_WAF_ADD_LOOKUP_HEADERS.

   A request that ends before the answer comes is logged without it.

6. The country lists (see "Country lists").

7. The reputation sources (see "Reputation"). A client on a fetched blocklist gets SWW_WAF_BLOCKLIST_ACTION. A client on the CrowdSec decision list is banned. A client listed by a DNS blocklist, or scored at least SWW_WAF_ABUSEIPDB_MIN_SCORE by AbuseIPDB, gets SWW_WAF_REPUTATION_ACTION.

8. The client's limit percentage (see "Biased thresholds").

9. Rate limiting (see "Request rate limits, per client"). A request over a limit times the percentage is refused and bans the client.

10. The trap paths (see "Attack detection"). A request for one is refused and bans the client.

11. The rule files (see "Rule files"). Matching stops at the first rule that refuses.

12. The header limits (see "Answering in time"), and then the Core Rule Set (see "Attack detection"). A Core Rule Set match is refused with 403 while SWW_WAF_CORE_RULE_SET_MODE is block, and only logged and alerted while it is detect.

13. A path under /_sww/ is answered 404, so that the app never sees that prefix.

14. rproxy passes the request to SWW_RPROXY_UPSTREAM_URL, streaming (see "Passing a request to the app"). With SWW_WAF_ADD_LOOKUP_HEADERS, X-Client-ASN and X-Client-Country are added for the app's own logs; a header whose value is unknown is left out. Whatever that setting says, any such header the client sent is removed.

15. After the visitor's answer has ended, rproxy sends the report (see "The report"). discriminator then counts:

    • For a request it allowed, the response bytes, the request bytes or both are added to the byte counters, as SWW_WAF_BYTES_COUNT says. For a WebSocket, what it carried each way counts once it closes. A client whose total passes a byte limit times the percentage has broken that limit and is banned. The response in progress is not cut off.
    • For a request it allowed, the client's history and the anomaly counters are updated, and the alert thresholds are evaluated.

    For a request it refused, discriminator adds nothing to the byte limits. It updates the history and the anomaly counters at its decision. It counts the refusals the error burst counts at the decision too (see "Attack detection").

In SWW_WAF_MODE=observe, every refusal above is logged as what would have happened, nothing is refused, and discriminator makes no ban of its own. A path under /_sww/ is still answered 404, and rproxy's size and time limits still apply.

Counting method

• Each window (minute, hour, day) uses two adjacent fixed buckets per client. The earlier bucket is weighted by how much of it still overlaps the sliding window. This costs a few integers per client per window and avoids the burst at bucket boundaries that a single fixed bucket allows.
• Counters are read and updated in memory. They are written to clients.json and loaded again at start (see "Persistent state"), so a restart does not hand every client a fresh allowance. Buckets whose time has passed are discarded on load.
• Memory is bounded. When the table of clients holds SWW_WAF_MAX_TRACKED_CLIENTS, the least recently seen client is dropped first, with its history. The bans are bounded by SWW_WAF_MAX_BANS (see "Bans"). The GeoJS answers, the reputation verdicts and scores, the DNS queries under way and the anomaly counters have fixed bounds, listed under "Fixed limits" in docs/settings.md. Past its bound, the GeoJS answer or anomaly counter used longest ago goes first, and the verdict or score fetched longest ago. Past the bound on DNS queries under way, a zone is asked at the client's next request.

Bans

Until a ban ends, every request from its netblock is refused with SWW_WAF_BAN_RESPONSE. The netblock is the client: its IPv4 address, widened to SWW_WAF_BAN_SCOPE_V4_PREFIX, or its IPv6 group. At the defaults a ban covers one address or one /64, so a permanent ban does not reach neighbours who did nothing. Requests refused under a ban are not counted against limits. They are counted in the client's history and in the ban's notes.

An offence is a request discriminator holds against the client. It is one of:

• a clear sign of attack;
• a broken limit;
• a refusal by a rule file or by the Core Rule Set.

Offences are counted by kind in the client's history. The first two kinds lead to a ban, each by its own rule:

• A clear sign of attack: a request that matches a rule whose action is ban, or asks for a path in SWW_WAF_TRAP_PATHS. Examples are a probe for a .env file or a .git directory, and a known scanner's user agent.

  • The first one bans the netblock for SWW_WAF_ATTACK_BAN_DURATION (default 7d), with the cause attack.
  • Any further request from the netblock while that ban lasts shows it is malicious: the ban becomes permanent. No other cause of ban becomes permanent this way.
  • Once that ban has run out, the netblock is served like any other. Its next clear sign of attack bans it permanently at once.

• A broken limit, one of:

  • a request over a rate limit;
  • a response that takes the client's byte total past a byte limit;
  • more than SWW_WAF_ERROR_BURST_THRESHOLD refusals within a minute.

  The bans for a broken limit:

  • The first such ban lasts SWW_WAF_LIMIT_BAN_DURATION (default 1h), with the cause limit. So does one that comes more than SWW_WAF_LIMIT_BAN_REPEAT_WINDOW (default 24h) after the last such ban ended.
  • Breaking a limit again within that window makes the new ban three times as long as the last one: one hour, then 3, 9, 27 and 81 hours.
  • A ban that would be longer than SWW_WAF_MAX_BAN_DURATION (default 7d) is permanent instead. At the defaults, that is the sixth ban in a row.
  • Each such ban sets the client's counters for every limit back to zero. Once it ends, only new requests can break a limit again. The client's history keeps its totals.

A Core Rule Set match refuses only the request it matched, with 403 and without a ban, because the Core Rule Set has false positives. A block rule does the same. Those refusals count toward the error burst, so a client that keeps setting them off is banned for a broken limit.

A client on the CrowdSec decision list is banned until the decision ends, with the cause crowdsec. Only a client that sends a request is banned, so a long list does not fill bans.json. Such a ban is never made permanent. It does not make the netblock's next ban for a broken limit longer. A ban lifted while the decision is in force is made again at the client's next request. To lift it for good, delete the decision in CrowdSec, then lift the ban once the list has been fetched again.

These refuse each request they cover, record nothing in bans.json, and count no offence:

• SWW_WAF_DENY_NETS;
• a blocklist or reputation hit whose action is deny;
• the country lists.

Every ban carries notes with what an admin needs to decide whether to lift it, drawn from what discriminator already knows. No lookup is made for the notes alone. docs/state-files.md lists the notes with the rest of a ban's entry.

An admin adds, keeps or lifts a ban by editing bans.json (see "Persistent state") or through the ban endpoints (see "Admin endpoints"). A ban an admin makes has the cause admin; an admin who wants to keep a ban discriminator made sets its cause to admin. A ban lifted before it ends is kept, marked as lifted, and does not count toward a longer ban; it refuses nothing. Deleting its entry from bans.json forgets it entirely.

A permanent ban does not run out. It ends when an admin lifts it, or when discriminator drops it from a full table, as below. When discriminator holds SWW_WAF_MAX_BANS bans of its own, it drops the earliest ban of one netblock: the netblock that has gone longest without a request. A past, active or permanent ban is dropped alike. A scanner whose ban was dropped that way is refused and banned again by its next probe. A ban whose cause is admin is never dropped and does not count toward SWW_WAF_MAX_BANS, so such bans can never fill the table.

Settings

docs/settings.md lists every setting with its default and the values it takes, in the groups below. It also gives the form each kind of value takes. The groups below are discriminator's settings; "rproxy" gives the use of rproxy's. Conventions:

• These settings take off, which switches them off:

  • SWW_WAF_LOOKUP_SOURCE and SWW_WAF_CORE_RULE_SET_MODE;
  • the three rate limits and the three byte limits;
  • SWW_WAF_CORE_RULE_SET_ANOMALY_THRESHOLD, SWW_WAF_CORE_RULE_SET_BODY_LIMIT and SWW_WAF_ERROR_BURST_THRESHOLD;
  • SWW_WAF_ALERT_COOLDOWN and SWW_WAF_ALERT_MAX_PER_HOUR;
  • every anomaly threshold;
  • SWW_RPROXY_MAX_REQUEST_SIZE, SWW_RPROXY_MAX_RESPONSE_SIZE, SWW_RPROXY_MAX_REQUEST_TIME_MS and SWW_RPROXY_MAX_RESPONSE_TIME_MS.

  No other setting can be switched off with off.

• A list set to an empty value is an empty list, and replaces the default.

• Any SWW_WAF_ or SWW_RPROXY_ setting can be read from a file instead, for secrets and long lists: SWW_WAF_ADMIN_TOKEN_FILE names the file that holds SWW_WAF_ADMIN_TOKEN. The file's contents are the value, less one newline at the end, and are checked as the value itself would be. Setting both forms stops the start, and so does naming a file that cannot be read. SWW_WAF_LOG_REMOTE_TLS_CA_FILE and SWW_WAF_CORE_RULE_SET_EXCLUSIONS_FILE already hold a file name, so they have no _FILE form. SWW_COMMAND cannot be given as a file.

• Settings, files included, are read once at start. Changing one means restarting the container. The files discriminator watches while it runs are its state files, its rule files and the lookup database.

• An invalid setting stops the process at start with one log line, marked "type":"process" with "msg":"invalid setting". The line names the variable and what is wrong with the value; "Each container's processes" says what happens then. At start the settings in effect are logged on one line; tokens, keys and the paths of webhooks, which often carry a secret, are shown as ********.

Core

• SWW_WAF_INSTANCE_NAME is in each line discriminator logs, and in each metric and alert, so that instances can be told apart. The default, the container's host name, changes at every redeploy; a name set by the operator survives it.
• SWW_WAF_MODE=observe lets a configuration be tried before it is enforced (see "What happens to a request").
• SWW_WAF_IPV6_GROUP_PREFIX sets the group size. After a change, each IPv6 client starts afresh; existing bans keep refusing their netblocks until they end. SWW_WAF_MAX_TRACKED_CLIENTS and SWW_WAF_MAX_BANS set the bounds under "Counting method" and "Bans".

Tokens

• SWW_WAF_ADMIN_TOKEN switches on the ban endpoints and /_sww/clients/<ip>. The metrics are off until SWW_WAF_METRICS_TOKEN is set, since without a token they would be open to anything on discriminator's network. The two are separate, so the metrics collector, which holds the metrics token, cannot manage bans.
• A token is a long random value. One shorter than 32 characters stops the start. The file form keeps a token out of the container's environment (see "Tokens" under "Deployment").

State files

• SWW_WAF_STATE_DIR names the directory of the state files (see "State" under "Deployment").
• bans.json is written SWW_WAF_STATE_WRITE_DELAY after a ban is made, lifted or made permanent, so a burst of changes becomes one write. Every file is written every SWW_WAF_STATE_COUNTER_INTERVAL, and at an orderly stop.

Logging

• The request log on stdout is always on and has no switch (see "Request log"). SWW_WAF_LOG_LEVEL filters only the process's own messages, and no message is at debug. warn and error hold back the line of settings at start.
• SWW_WAF_LOG_REQUEST_HEADERS names the request headers to record. Authorization, Cookie and Set-Cookie values are never logged, even if listed. The log says only whether Authorization and Cookie were present.
• SWW_WAF_LOG_REMOTE_URL names a syslog server (see "Request log").

Metrics

• SWW_WAF_METRICS_TOP_N is how many of the busiest AS numbers and countries get a series of their own. The rest are summed as other, so the label set stays bounded.

Static lists

• SWW_WAF_ALLOW_NETS skips the checks step 3 names, for monitoring or the owner's own networks. SWW_WAF_RATE_LIMIT_EXEMPT_NETS bypasses the request and byte limits only; the error burst, attack detection and bans still apply.
• Many people can share one address, as in an office, on a mobile carrier or through Tor. They are one client, and share its limits and its bans. A known shared address busier than the limits allow belongs in SWW_WAF_ALLOW_NETS or SWW_WAF_RATE_LIMIT_EXEMPT_NETS.

Request rate limits, per client

Breaking one bans the client. The defaults let people read a site and stop scrapers that send many requests slowly over hours or days. The hour limit is the main one.

• Per minute: 120, 2 requests a second. It leaves room for a person opening two pages at once with an empty cache. A web app's page opened with an empty cache loads 20 to 60 files, and a handful once cached.
• Per hour: 1,500. A person reading a site for a whole hour, a page every 20 seconds, makes about 900 requests. A scraper making one request every 2 seconds makes 1,800 an hour, more than the limit.
• Per day: 6,000, four hours at the hour limit. A scraper making one request every 10 seconds makes 8,640 a day, more than the limit.

Counting cannot catch a client that stays under all three limits, about 4 requests a minute around the clock. Only the other checks can catch an abuser that slow. They include the trap paths, the rule files, the Core Rule Set and the blocklists. An abuser that trips none of them is not caught.

Some legitimate clients pass the limits:

• A CI runner asks its code forge for work all day and can pass the limits.
• Everyone behind one shared address, such as an office, shares one client's limits.

Their addresses belong in SWW_WAF_RATE_LIMIT_EXEMPT_NETS.

SWW_WAF_RATE_LIMIT_EXEMPT_PATHS takes path prefixes out of the count, such as static assets or health checks; their bytes are not counted either. A prefix is matched exactly as sent, without the query, and is not percent-decoded: /assets/ matches /assets/app.js, and does not match /Assets/app.js, /%61ssets/app.js or /static/assets/app.js. There are no wildcards. A path holding .., a backslash or an encoded slash (%2F) is never exempt, since the app may read it as a path outside the prefix. SWW_WAF_CORE_RULE_SET_EXEMPT_PATHS is matched the same way.

Byte limits, per client

By default there is no minute limit, so a client can download in a burst at full speed. The hour limit defaults to 50 GiB and the day limit to 100 GiB. They allow several downloads of a few GiB each. A response's bytes are counted when it ends (see step 15).

Lookup of AS number and country

On by default through GeoJS, which needs no account or file, so that country lookup works with no setup. A local database file is the alternative for a service that keeps its visitors' addresses on its own host.

• Only one source is used. The start stops, naming both settings, when file is set without SWW_WAF_LOOKUP_DB_PATH, or SWW_WAF_LOOKUP_DB_PATH with any other source. The start also stops when the source is off and a setting that needs a lookup is set (docs/settings.md lists them).
• GeoJS is told the address of every new visitor, whether or not a setting uses the answer, since the request log, the metrics and the ban notes carry it. These addresses are never sent:
  • one in SWW_WAF_ALLOW_NETS or SWW_WAF_DENY_NETS;
  • one under a ban;
  • a private, loopback or link-local address, which no source can place.
• For an IPv6 visitor, GeoJS is asked about the first address of its group. Addresses are asked about together, up to 200 in one request (https://get.geojs.io/v1/ip/geo.json?ip=a,b,c), one request at a time, which keeps the number of requests to GeoJS low. Each answer's country_code, asn and organization_name are read. An answer without a country_code leaves the country unknown, and the asn 64512 leaves the AS number unknown.
• Each answer is kept for seven days, in memory and in lookups.json. Answers are kept apart from the table of clients, so that an answer outlasts a client dropped from that table. The answer reaches the client's history and ban notes when it comes.
• SWW_WAF_LOOKUP_TIMEOUT is how long a visitor's request waits for its client's first answer, when a setting needs it (see step 5). It also bounds each request to GeoJS. A client whose answer has not come in time counts as unknown until it comes, and its later requests do not wait. Its address is asked about again in the background.
• A client the lookup cannot place has an unknown country and AS number, which the country lists and the biased thresholds treat as described there. Nothing else treats it differently.
• GeoJS publishes no rate limit, but its terms forbid "an excessive amount of API requests", judged by GeoJS alone, and let it block a caller. While GeoJS is slow, down or refusing discriminator, new clients count as unknown. GeoJS is then left alone for a second, and twice as long after each further failure, up to five minutes. Each failure raises a source_failure alert (see "Alerting"). A service that cannot accept this uses the file.
• The file is the free IPinfo Lite database (ipinfo_lite.mmdb), whose asn, as_name and country_code fields are read. The operator downloads it with a free IPinfo account and mounts it read-only; discriminator never fetches it and holds no account or token for it. IPinfo updates it daily. Its licence is Creative Commons Attribution-ShareAlike 4.0, and IPinfo asks for a link to it on the site that uses it, such as "IP address data is powered by IPinfo".
• The file is read whole at start; one that is missing or unreadable stops the start. It is read again once it has gone 2 seconds without a change after being replaced. A replacement that cannot be read is logged, sent as one file_error alert and ignored. The database already loaded stays in use. A client whose address the file does not hold counts as unknown.

Country lists

Both are empty by default: the defaults judge a client by what it does, not by where it comes from.

• SWW_WAF_DENIED_COUNTRIES refuses every request from a listed country.
• SWW_WAF_EXCLUSIVELY_ALLOWED_COUNTRIES lets through only clients placed in a listed country. Every request from any other country is refused. So is every request from a client the lookup cannot place, such as internal monitoring or a visitor on a private address; those that should reach the app go in SWW_WAF_ALLOW_NETS. A list that let unplaced clients through would let every new client in whenever GeoJS stops answering. The cost: while GeoJS fails, this list refuses every new client. No other failure of GeoJS, a reputation source or an alert destination comes this close to taking the service down.
• Both may be set. SWW_WAF_DENIED_COUNTRIES then adds nothing, since the exclusive list already refuses every other country. A code on both lists stops the start with a message naming it.
• A refused request is answered with SWW_WAF_BAN_RESPONSE, as soon as the client's address has been looked up. It is logged with the action country_denied, and counted in the metrics.

Biased thresholds

The biased thresholds lower the limits of clients in listed AS numbers and countries. The lists are empty by default, for the same reason as the country lists. The request log and the metrics show which AS numbers and countries a service's abuse comes from.

• Each list gives AS numbers or countries a percentage, such as AS14061:50. Their clients get that percentage of every request and byte limit (SWW_WAF_ASN_LIMIT_PERCENT, SWW_WAF_COUNTRY_LIMIT_PERCENT). A lowered limit is rounded down. 0 is a zero allowance: the client's first request breaks a limit and bans it.
• SWW_WAF_ASN_BYTES_PERCENT and SWW_WAF_COUNTRY_BYTES_PERCENT set the byte limits' percentage alone, when it should differ from the request limits'. For an AS number they list, they replace its percentage from SWW_WAF_ASN_LIMIT_PERCENT and SWW_WAF_ASN_LIMIT_PERCENT_URL; for a country, the one from SWW_WAF_COUNTRY_LIMIT_PERCENT.
• SWW_WAF_ASN_LIMIT_PERCENT_URL names a text file of AS:percent lines, so one abuse-source list can be shared by every discriminator in the fleet. It is fetched and kept like a blocklist (see "Reputation"). Where it and SWW_WAF_ASN_LIMIT_PERCENT both give a percentage, the lower applies.
• A client's limit percentage is the lowest of those that apply to it: its AS number's, its country's, or a reputation hit's limit:<percent>. A client without a country gets SWW_WAF_UNKNOWN_LIMIT_PERCENT in place of a country's. The percentage is 100 if none applies. Clients given a lower percentage are banned after fewer requests, first for a while and then, for repeated abuse, permanently. The log line of each counted request, and the notes of each ban, say which setting lowered the limit.
• No budget is shared by a whole country or AS number. One abuser could use it up and lock out everyone else there: a denial of service nobody chose, and the one this tool exists to prevent. Refusing a whole country is left to the operator's choice, through the country lists.

Attack detection

By default the Core Rule Set reads the method, the URL with its query string, and the headers of every request. That is where automated attacks show: injection in query strings, path traversal, attacks carried in headers, scanners' user agents. It reads no request body. In front of a code forge, bodies carry what people write and publish, such as issue text and files saved in the web editor. The Core Rule Set takes the shell commands and code in them for attacks. The default gives up refusing an attack carried in a body. The rule files, the limits and the bans still apply to the client that sends it.

• The Core Rule Set skips a path under SWW_WAF_CORE_RULE_SET_EXEMPT_PATHS.

• SWW_WAF_CORE_RULE_SET_PARANOIA_LEVEL and SWW_WAF_CORE_RULE_SET_ANOMALY_THRESHOLD are the Core Rule Set's own two tuning values, at its own defaults. Each level up runs more rules, which find more attacks and take more ordinary requests for attacks. Each rule that matches adds its score to the request's, and a rule rated critical adds 5. A request whose score reaches the threshold is a match, so by default one critical rule is enough. With the threshold off, nothing is a match, and scores are still logged. A request with more than 1000 query parameters or body fields adds 5 to its score, since Coraza reads only the first 1000. For a match, in block and detect mode alike, the log line gives the rule ids and the score, and a waf_block alert is sent.

• SWW_WAF_CORE_RULE_SET_DISABLED_RULES switches off rule ids when an app trips a false positive. The default switches off six rules:

  • 920340, 920420 and 920640 refuse a request for the type of its body, when the type is missing or not on the Core Rule Set's short list;
  • 920440 refuses a request for its file extension, such as .sh or .sql;
  • 930130 and 930140 refuse a file or directory name in the path, such as .git/, .gitignore, Dockerfile, package.json or an editor's settings directory.

  In front of a code forge these refuse git over HTTP, container image and package uploads, and views of ordinary files in a repository. At the site root, where no app serves such files, the default rule file bans the common probes these rules caught (see "Rule files"). A list you give replaces the default, so name these six ids in it to keep them off.

• SWW_WAF_CORE_RULE_SET_EXCLUSIONS_FILE names a file of exclusions, written as Coraza directives. An exclusion switches one rule off for some fields or paths only, where SWW_WAF_CORE_RULE_SET_DISABLED_RULES switches it off everywhere. The file holds an app's own exclusions; the eighth change below holds gitea's.

  • discriminator loads the file after smallwebwaf's own setup rules and before the Core Rule Set's rules. The Core Rule Set loads its own REQUEST-900-EXCLUSION-RULES-BEFORE-CRS.conf at the same point.
  • So the file holds only SecAction and SecRule lines that exclude with ctl:ruleRemoveTargetById or ctl:ruleRemoveById. These run with each request, before the rules they exclude, those of phase 1 included.
  • A rule that SWW_WAF_CORE_RULE_SET_DISABLED_RULES switches off and the file names is simply off.
  • The start stops on a file that cannot be read or does not parse, whatever SWW_WAF_CORE_RULE_SET_MODE is.
  • It stops too on a file that names a number from 900000 to 900999, the ids of smallwebwaf's own rules, alone or in a range. Comment lines are not checked.
  • It stops too on a directive that removes or changes rules as the file loads, such as SecRuleRemoveById, since none of the Core Rule Set's rules is loaded yet.

• SWW_WAF_CORE_RULE_SET_BODY_LIMIT switches body inspection on, up to that size. It takes at most 64K, since discriminator gets no more of a body than that (see "The answer"). Form data and multipart bodies are read up to the size. JSON and XML bodies are read only when no larger than the size, since those cannot be read in part. Any other body is not read: the Core Rule Set would read it as form data, where binary content such as a git push trips rules written for text. A body the Core Rule Set cannot parse adds 5 to the score, and so does a multipart body cut by the limit in the middle of a part's header. Body inspection suits apps whose forms carry no code.

• SWW_WAF_TRAP_PATHS names paths the app never serves and only scanners ask for, such as /wp-login.php in front of an app that is not WordPress. This is the short form of a path rule with the ban action, for deployments that mount no rule files. A trap path is matched whole, without the query, against the path as sent and once percent-decoded: /wp-login.php matches /wp-login.php?x=1 and /%77p-login.php, and does not match /wp-login.php/ or /blog/wp-login.php.

• SWW_WAF_ERROR_BURST_THRESHOLD counts a client's refusals per minute of these kinds, and of no other:

  • a rule file match;
  • a trap path;
  • a Core Rule Set match.

  More than this breaks a limit (see "Bans"). A client trying one attack after another is refused many times a minute; a person rarely more than a few times. Answers the app gives are not counted, since they do not tell a scanner from an ordinary client. In front of a code forge, registry and git clients get 404 and 401 by design, hundreds of times a minute.

smallwebwaf changes the Core Rule Set 4.25.0 in eight ways, since in front of gitea it would otherwise refuse ordinary requests. The changes hold in front of every app, and no setting undoes them:

1. PUT, PATCH and DELETE are allowed methods besides GET, HEAD, POST and OPTIONS. APIs, container image pushes and package uploads use them. Other methods stay refused.

2. The request header Expect is allowed, since curl and git send it before some large uploads. Content-Encoding is allowed on a request whose body the Core Rule Set does not read, which by default is every request. git compresses a fetch request over 1 KiB and says so in that header. On a body the Core Rule Set does read, the header stays refused, since a compressed body cannot be inspected. The other headers the Core Rule Set refuses, such as Proxy, stay refused.

3. The redirect_uri parameter is not checked for a URL naming an IP address or localhost (931100, 934110). Git Credential Manager, git-credential-oauth and tea, which gitea registers for OAuth sign-in, ask to be sent back to http://127.0.0.1 on the user's own machine. The server never fetches that address.

4. Ten query parameters are not checked against the Core Rule Set's lists of system files (930120), shell paths (932160) and command names (932260):

   • path, files, skip-to and sub_path;
   • ref, sha, branch and workflow;
   • artifactName and redirect_to.

   gitea puts into them file paths, branch and workflow names, and the page to return to after signing in. It reads these values as names in a repository or in its own records, or as a page of its own site, never as a command. Any name on those lists can be an ordinary file or branch, such as .gitignore or docker-build. In these parameters, what only these three rules would catch reaches the app, such as the command |cat /etc/passwd. Path traversal, SQL and script injection, and PHP, Java and Node.js code are still refused there. Other parameters keep all three rules, except on the gitea paths the eighth change names. An app that uses one of these parameters as a file on the server, or passes it to a shell, must check the value itself. A rule file can refuse it instead.

5. Two cookies are left out, and Referer keeps fewer rules:

   • The Core Rule Set reads the request without the gitea_flash and redirect_to cookies. gitea puts a message naming what was just done into gitea_flash. It puts into redirect_to the page to return to after signing in. The Core Rule Set takes file names in them, such as package.json, for attacks. A browser sends such a cookie with every request until gitea replaces it. gitea cannot replace it while those requests are refused, so one refusal would keep the browser out of the whole site. gitea shows the message with any script removed and returns only to its own pages. Every other cookie is read in full. An app that reads a cookie of either name must check it itself.

   • Referer is not checked for a Unix command given without arguments (932340), for Java starting a process (944110) or for a Java class name (944130). Referer names the page a request comes from, on everything that page loads and on every link followed from it. The Core Rule Set takes some page addresses for attacks:

     • a search for env;
     • OpenJDK's Runtime.java, whose path holds both runtime and java.;
     • a file named java.lang.String.md.

     Every request from such a page would be refused. Referer keeps the other rules, except after some gitea searches (the eighth change). An app that passes Referer to a shell must check it itself.

6. Responses are not inspected. A raw file from a repository, such as a shell script, looks to the response rules like source code leaking from the server.

7. The from parameter of a container registry's upload path, /v2/<image>/blobs/uploads/, is not checked for command names (932260). When a push reuses a layer, from names the image the layer comes from. An image such as owner/python3-app would have that push refused.

8. On gitea's paths, some fields skip the rules that refused ordinary gitea requests there. They are the fields where gitea sends a search, a file name or a branch name. internal/waf/gitea.conf lists each rule and the request it refused. The fields are:

   • q on gitea's searches of code, of issues and pull requests, and of commits. A code search holds code, such as <script>, SELECT * FROM users WHERE or ../.

   • Referer of a page opened from such a search, for script and SQL injection and Java code (941110, 941120, 941160, 942100, 944100).

   • q on gitea's searches for a name, for the names of system files and commands (930120, 932120, 932260). Names such as .profile and python3-app were refused. gitea searches for the names of:

     • repositories;
     • users and organisations;
     • branches and tags;
     • packages.

     A user's or an organisation's page is any path of one segment, so on every such path, on any app, q skips these three rules, for example on /search.

     Some of these searches also skip the rule for Java class names (944130), which refused org.apache.commons.logging.

   • The path of these requests, which names a file:

     • a file's page;
     • its raw file;
     • its download;
     • its blame page;
     • its history;
     • its edit page;
     • its preview in the editor;
     • its delete page;
     • the API's call for its contents;
     • a release attachment's download.

     Names such as notes.txt~, process.env.d.ts and java.lang.String.md were refused.

   • A file name in a query parameter:

     • file of a commit graph;
     • path of the API's list of commits;
     • path of a diff's hidden lines;
     • body of the API's call for several files;
     • template of a new issue;
     • redirect_to of the sign-in page;
     • name of an attachment uploaded through the API.

   • A branch or tag name in a query parameter, for command names (932260). gitea sends one:

     • to delete or restore a branch;
     • to start a release from a tag;
     • to edit a branch protection rule;
     • to list a directory's last commits.

   • The Content-Range header of an artifact upload from actions/upload-artifact@v3 (920450).

   Elsewhere these fields keep every rule. The paths are those of gitea at the site root. An app other than gitea that serves one of them must check these fields itself.

The parameter and cookie names in these changes are matched without regard to case, as Coraza matches them.

Ban lengths, response and scope

SWW_WAF_BAN_RESPONSE is 403 by default, which makes a ban easy to recognise when debugging; 429 and close are the alternatives. Behind traefik, close does not leave the client unanswered: traefik answers 502, as it does whenever its backend drops a connection. SWW_WAF_BAN_SCOPE_V4_PREFIX widens an IPv4 ban to the surrounding netblock. The four ban lengths are under "Bans".

Reputation

No source is on by default. A source acts on a list kept elsewhere; the defaults judge a client only by what it does here. AbuseIPDB and CrowdSec also need an account or an engine of the operator's own. The list best suited to be on by default, Spamhaus DROP, has terms, below, that forbid frequent fetching, and Spamhaus may block an address that breaks them. Each discriminator fetches its own copy, so several of them on one host, sharing one address, would often fetch too close together.

• Blocklists (SWW_WAF_BLOCKLIST_URLS) are text files of addresses and netblocks, one to a line, as the Spamhaus DROP list is written. Anything after a ; or # on a line is ignored. An IPv4-mapped address such as ::ffff:192.0.2.0/120 is read as the IPv4 netblock it stands for. Spamhaus's terms for DROP, on its DROP page:

  • automated downloads must be at least one hour apart, and once a day is more than enough in most cases;
  • a product that uses it must credit The Spamhaus Project and keep the list's date and copyright lines with the data.

  So SWW_WAF_BLOCKLIST_REFRESH is never less than 1h, and the last good copy is kept whole, comment lines included, on failure and across restarts.

• Each list is fetched again SWW_WAF_BLOCKLIST_REFRESH after its last try, one list at a time. reputation.json keeps when each was last tried, so a restart does not fetch a list again before its time. A fetch fails when:

  • the server answers other than 200;
  • the fetch does not finish within a minute;
  • the list is longer than 16 MiB;
  • a line is not an address or netblock, or, for the AS number file, not AS<number>:<percent>.

  The log, the alerts, the ban notes and the metrics name each list by its URL, so a list's URL must not hold a secret.

• SWW_WAF_BLOCKLIST_ACTION is deny by default, which refuses every request from a listed address and suits lists of networks that send nothing legitimate, such as DROP. limit:<percent> gives listed clients that percentage of every request and byte limit, and suits a list of addresses shared with ordinary visitors, such as Tor exits. log only notes the hit in the log line and raises an alert.

• A CrowdSec engine on the host (SWW_WAF_CROWDSEC_LAPI_URL) has its decision list fetched every minute, with SWW_WAF_CROWDSEC_LAPI_KEY in the X-Api-Key header. The list is kept like a fetched blocklist, and a client on it is banned (see "Bans"). Only decisions of the type ban on an address or a netblock are used; any other, such as a captcha or a country, is left out. A decision lasts for the duration it had left when the engine answered. This is how a fleet-wide blocklist can arrive without discriminator depending on CrowdSec. A fetch fails when:

  • the engine answers other than 200, a redirect included;
  • the fetch does not finish within a minute;
  • the answer is longer than 16 MiB;
  • the answer is not a list of decisions.

• Each DNS blocklist (SWW_WAF_DNSBL_ZONES) is asked about a client's own address in the background, the first time the client is seen. No request waits for the answer. The name asked about is the one RFC 5782 gives: the numbers of an IPv4 address reversed, or the 32 hex digits of an IPv6 address reversed. So 192.0.2.99 is asked about as 99.2.0.192.dnsbl.dronebl.org. An answer in 127.0.0.0/8 lists the client, but one in 127.255.255.0/24 is Spamhaus refusing the query. A name that does not exist does not list the client. These are failures, after which the zone is not asked again for a minute:

  • an answer in 127.255.255.0/24;
  • an answer outside 127.0.0.0/8;
  • a resolver error;
  • no answer within SWW_WAF_REPUTATION_TIMEOUT.

  DNS blocklists made for mail servers list residential ranges, and would block ordinary visitors, so they should not be used. Spamhaus zones need their keyed query service, given as a zone name that contains the key. discriminator shows the key as ******** wherever it names the zone. Only reputation.json keeps the key. SWW_WAF_DNSBL_RESOLVER names a resolver, since public resolvers are refused by several list operators.

• AbuseIPDB (SWW_WAF_ABUSEIPDB_KEY) is asked, at its check endpoint, for the abuse confidence score of a client, from 0 to 100. Only a client that has already committed one offence is checked, so the daily budget (SWW_WAF_ABUSEIPDB_DAILY_BUDGET, under the free tier's 1000 checks a day) is spent on suspects. The check runs in the background at the client's next request that no ban refuses, so no request waits; the score applies to the client's later requests. For an IPv6 client, AbuseIPDB is asked about the address of the request that set off the check, and the score is used for the whole group. A score at or over SWW_WAF_ABUSEIPDB_MIN_SCORE is a hit. The budget is counted in UTC days and kept in reputation.json, so a restart does not make it whole again. The check that spends the last of it is logged and alerted, and no client is checked until the day ends. The key is sent in the Key header and nowhere else, and is never logged. A check fails, after which no client is checked for a minute, when:

  • AbuseIPDB answers other than 200;
  • the answer gives no score;
  • no answer comes within SWW_WAF_REPUTATION_TIMEOUT.

• SWW_WAF_REPUTATION_ACTION, for DNS blocklist and AbuseIPDB hits, is limit:25 by default. Such verdicts are less certain than a blocklist, so a listed client gets a quarter of every request and byte limit rather than a refusal.

• Verdicts and scores are kept for SWW_WAF_REPUTATION_CACHE_TTL, in memory and in reputation.json. A source that is down or over quota gives no verdict. The service continues. Each failure raises a source_failure alert (see "Alerting").

Alerting

• Alerts go to a webhook as JSON, and to Slack and ntfy as formatted messages. docs/alerts.md lists the events, the JSON object and the messages. SWW_WAF_ALERT_EVENTS chooses which of the seven event types are sent; the hourly summary is sent whatever it says. The bans an admin makes raise no alert. In observe mode, a request that would have made a ban still raises the ban's alert, with mode set to observe in its detail.
• A repeat is an alert with the same event, and the same netblock, source, file or anomaly scope, as one sent less than SWW_WAF_ALERT_COOLDOWN before. An anomaly scope is one of the things counted under "Anomaly thresholds". A repeat is held back and counted, and the next such alert sent gives the count.
• Past SWW_WAF_ALERT_MAX_PER_HOUR alerts in an hour, the rest are held back and counted by event, so a wide attack cannot flood the channel. Once the hour ends, one alert with the event summary gives those counts. It also counts the repeats that a cooldown held back and that no later alert reported.
• Each destination has a bounded queue (see "Fixed limits" in docs/settings.md), sent one at a time, so a slow destination holds up nothing. When a queue is full, its oldest alert is dropped and counted in the metrics. A 2xx answer takes an alert. A 4xx other than 408 and 429 refuses it, and it is dropped. Anything else is a failure: the alert is sent again a second later, and twice as long after each further failure, up to a minute. alerts.json keeps the queues and cooldowns across a restart.

Anomaly thresholds

These only alert; nothing is blocked. None is set by default, and a threshold left unset sends no alert. An alert needs a destination only the operator can supply, and what counts as unusual depends on each service's normal traffic, which the metrics show. They reveal a widely distributed scraper, thousands of addresses at low rates each, which the per-client limits do not see. A low SWW_WAF_ASN_LIMIT_PERCENT for its AS number is the response.

• Requests and bytes are counted per minute and per hour, for each of:
  • a client;
  • the surrounding netblock;
  • an AS number;
  • the whole service;
  • a named netblock in SWW_WAF_WATCH_NETS.
• Every request discriminator answers with allow, deny or close is counted, an allowed one when its report comes. That holds whatever was done with it, and for clients in SWW_WAF_ALLOW_NETS and SWW_WAF_RATE_LIMIT_EXEMPT_NETS, since an exempt client misbehaving is worth knowing about. The counters are kept in memory and in alerts.json.

Rule files

discriminator reads every *.rules file in SWW_WAF_RULES_DIR and checks each request against them. docs/rules.md gives the format, the targets and actions, and the default file.

• The files are plain text meant to be edited by hand, so that a new scanning pattern seen in the request log becomes a rule in one line. discriminator watches the directory: a file edited, added or removed there takes effect while it runs. It reads the files again once the directory has had no change for 2 seconds, so that a file still being written is read whole. A file whose name starts with ., such as an editor's lock file, is not a rule file.

• Files are read in name order (00-default.rules before 50-site.rules), rules in line order.

• A rule's regex is Go's RE2 syntax. It has no backreferences or lookaround; in exchange matching time is linear in the input, so no rule can be made to stall the proxy.

• path and uri are matched both as received and once percent-decoded, so an encoded probe cannot slip past. A header sent more than once is matched with its values joined by , ; a header not sent is empty text. header:Host and header:Transfer-Encoding cannot be targets, since Go's HTTP server takes them out of every request; a rule naming one stops the start like any bad line. The target host matches the request's host. Request bodies are not available to rule files; reading them is the Core Rule Set's job.

• log only notes the match in the request log. block refuses the request; ban makes the request a clear sign of attack (see "Bans"), and is for requests no real visitor sends. A visitor banned by mistake is let back in by lifting the ban.

• At start, these stop the process with a message naming the file and line:

  • a line that does not parse;
  • a regex that does not compile;
  • a duplicate id.

  So does a SWW_WAF_RULES_DIR that does not exist or cannot be read. While running, the same faults leave the rules as they were, and the log and one file_error alert name the file and line. Once the file is fixed, it is read again. An empty directory is not an error: the log says that no rules were loaded.

• The image ships a default file, kept short and limited to patterns that are wrong for every app, with its path rules anchored at the site root. In front of a code forge, /.env is a probe, while /<owner>/<repo>/src/branch/main/.env.example is a file in a repository that any visitor may open. Anchor the paths of your own ban rules at the site root too. A rule has one target, so a path rule applies on every site behind that discriminator. A directory mounted over SWW_WAF_RULES_DIR replaces the default file; single files mounted into it add to it.

• Rules meant to be edited while discriminator runs belong in a mounted directory, not a mounted file. Many editors save a file by replacing it, and a container keeps seeing the old one when the file alone is mounted. The lookup database is mounted the same way, for the same reason. To run without rule files, mount an empty directory or set SWW_WAF_RULES_ENABLED=false.

Persistent state

All state lives in memory, and the files in SWW_WAF_STATE_DIR hold a copy of it, so an orderly stop loses nothing. The files are meant for people as well: an admin can read or edit them at any time, and the running discriminator takes the edit in. docs/state-files.md lists the files, what each holds, and their sizes.

• Each file holds one kind of state, as JSON with:

  • a top-level version number;
  • entries sorted by client address;
  • times in RFC 3339 UTC;
  • durations and sizes as plain numbers, with the unit in the field name.

• bans.json, reputation.json and alerts.json are indented, so that a person can open bans.json in an editor, find an address, and remove or add an entry. clients.json and lookups.json put each entry on one line, which halves their size and lets grep show everything about one client.

• Clients that were never banned are kept too, so every client's history survives a restart.

• Each file has a bound, so it can be rewritten whole. Writing one takes well under a second, in the background, so rewriting whole files needs nothing cleverer.

• A write serialises from a snapshot taken under the lock, so requests are not held up. It goes to a temporary file in the same directory, which is synced and renamed over the real name, and the directory is synced. A crash at any point leaves either the old complete file or the new complete file, never a partial one. Before writing a file, discriminator takes in any edit made to it on disk, so an admin's edit is never overwritten.

• At start, the files are read back. Active bans go into an in-memory lookup by netblock, everything else into maps. Expired cache entries are discarded. A missing file means empty state, and is normal on first run. These stop the process with a message naming the file and position:

  • a file that does not parse, or has an unknown version;
  • an entry without a field it needs;
  • a ban whose cause is not one of the four;
  • a ban whose expires is not after its start.

  Starting with empty state would silently forgive every repeat offender. Since writes are atomic, a broken file can only come from a hand edit, which the editor should hear about.

• While running, discriminator notices a file that is edited, replaced or added. It tells its own writes from an admin's by comparing the file with what it last read or wrote. An edit that parses is taken in at once: what the file says replaces what discriminator held for that file. An edit to bans.json makes, lifts or forgets bans as "Bans" describes. A change discriminator made after the admin opened the file, such as a new ban, is lost when the admin saves over it. Most editors warn when a file changed on disk while it was open. A file that is removed is written again at its next write.

• An edit that does not parse never stops the running discriminator. It keeps the state it has and sets the edited file aside as <name>.bad, so the edit is kept. It writes the file again from memory, and names the file and the position of the error in the log and in one file_error alert. The admin mends the .bad file and moves it back.

• A write that fails leaves the state correct in memory. It is logged, counted in the metrics and sent as one file_error alert per cooldown, and retried at the next write.

• A hard kill loses the changes since the last write: ban changes from the last SWW_WAF_STATE_WRITE_DELAY, everything else from the last SWW_WAF_STATE_COUNTER_INTERVAL. Losing the volume loses ban history, not service.

Request log

rproxy and discriminator each write request lines on stdout, one JSON object to a line, refused requests included. "The log" says which lines a request gets, and docs/request-log.md lists the fields. Bodies are never logged, and no header but those listed there. Query strings are logged as received; an app that carries secrets in query strings needs that fixed in the app. Each process's own messages (start, the settings, stop, errors) share its stream as JSON lines with "type":"process"; request lines carry "type":"request".

When SWW_WAF_LOG_REMOTE_URL is set, every line discriminator writes on stdout is also sent there as an RFC 5424 syslog record. So a deployment can stop depending on docker's log handling while docker logs keeps working. rproxy writes its lines to stdout only.

• Over UDP each record is one datagram; a line too long for one datagram is dropped alone. Over TCP and TLS each record follows its length in bytes and a space. UDP gives no delivery signal at all and is offered only for compatibility.
• The record carries the facility from SWW_WAF_LOG_REMOTE_FACILITY, the severity informational, the time, the host's name and the app name from SWW_WAF_LOG_REMOTE_APP_NAME.
• Sending happens on its own goroutine from a buffer of SWW_WAF_LOG_REMOTE_BUFFER lines, so a server that is slow or down never holds up a request and never stops stdout. When the buffer is full, the oldest line is dropped and counted in the metrics. A failed connection is tried again a second later, and twice as long after each further failure, up to a minute.
• As discriminator stops, it spends at most two seconds sending the lines still waiting, and drops the rest; stdout has carried them.

Metrics endpoint

GET /_sww/metrics, at discriminator's own address, answers in the Prometheus text format, for a request carrying SWW_WAF_METRICS_TOKEN. docs/metrics.md lists the metrics. No metric carries a client IP address as a label. Per-address questions are answered by the request log and GET /_sww/clients/<ip>.

Admin endpoints

discriminator's own endpoints are paths under /_sww/, at its own address on the private network. Admins and the metrics collector reach them there, and nothing routes the internet to that address. The prefix carries the tool's name, so it takes no path an app uses. docs/admin-endpoints.md lists the endpoints, what each takes and what each answers.

• GET /_sww/healthz answers 200 with ok to anyone, without a token, while discriminator is running.
• POST /_sww/report takes rproxy's report, without a token (see "The report").
• Every other endpoint needs its token, sent as Authorization: Bearer <token> (see "Tokens" under "Settings"). A missing or wrong token is answered 401. While a token is unset, the endpoints that need it answer 404, as does any other path.
• No check runs on these requests. They count toward no limit, and no client's history holds them.
• A ban made through the endpoints starts at once.

rproxy answers GET /_sww/healthz itself. It never passes a path under the prefix to the app (see "Without an answer from discriminator").

discriminator

sww discriminator runs the checks for sww rproxy, a small reverse proxy in front of each app. rproxy asks discriminator about each visitor's request. discriminator answers rproxy and passes nothing to an app itself. One discriminator serves many sites.

This section is the agreement between the two. The header names, the fields and the statuses below are exact.

discriminator reads the settings, the rule files, the lookup database and the state files, and listens at SWW_WAF_LISTEN_ADDR. On SIGTERM or SIGINT it stops taking requests and gives those in progress five seconds to finish. It then writes the state files.

The question

A request carrying X-Sww-Client is rproxy asking about one visitor's request. It is the visitor's request as the visitor sent it:

• the method;
• the path and query as sent;
• Host;
• every header the visitor sent, including its own X-Forwarded-For;
• the first 64 KiB of the body, or all of it when shorter.

Go's HTTP code changes some of the visitor's headers, such as:

• Go's client does not send an empty User-Agent.
• Go's server adds Cache-Control: no-cache to a request with Pragma: no-cache and no Cache-Control, before rproxy sees it.

rproxy itself refuses any request it cannot send this way (see "Requests rproxy refuses itself"). So a question always has:

• a path starting with one /;
• a Host that is not empty and holds no IPv6 zone;
• at most one User-Agent.

A body whose length the visitor did not announce comes chunked. Any other body comes with a Content-Length of the part it carries.

rproxy sets these headers, replacing any the visitor sent, and adds no others:

• X-Sww-Client: the visitor's address, which rproxy has worked out, without any IPv6 zone. discriminator takes it as the client, and groups an IPv6 address as "What happens to a request" says.
• X-Sww-Peer: the address of rproxy's own TCP peer, for the log line's peer_ip.
• X-Sww-Content-Length: the body length the visitor announced. It is absent when the visitor announced none.
• X-Request-ID: the request's id, which rproxy makes fresh for every request when it arrives. discriminator takes it as given and never makes one.
• X-Forwarded-Proto: the scheme.

Before any check, discriminator removes the three X-Sww- headers. It also sets Content-Length to the length the visitor announced, or removes it. So the rule files and the Core Rule Set see the headers the visitor sent, with rproxy's X-Request-ID and X-Forwarded-Proto.

discriminator refuses a question with 400 when:

• X-Sww-Client is not an address, or is an address with a zone, such as 2001:db8::1%eth0;
• X-Request-ID is absent, empty, over 128 bytes or not valid UTF-8;
• X-Request-ID is the id of a question discriminator is still handling, or of a request it holds (see "The report");
• the body is over 64 KiB.

For a refused question, discriminator writes a process warning. It runs no check, writes no request log line and counts nothing in the metrics.

These do not refuse a question:

• X-Sww-Peer absent or not an address: the log line's peer_ip is invalid IP.
• X-Sww-Content-Length not a whole number, or negative: it counts as absent.
• X-Forwarded-Proto absent: the log line's scheme is http.
• One of the five headers above sent more than once: discriminator uses its first value.

discriminator reads the whole body before any check. A question whose body stops arriving gets no answer. discriminator closes the connection once the 60 seconds below run out, or as soon as rproxy closes its side. It writes a process warning, runs no check, writes no request log line and counts nothing in the metrics.

The answer

discriminator runs steps 3 to 13 of "What happens to a request", in that order. Every answer but the last below carries X-Sww-Disposition.

• X-Sww-Disposition: allow, status 200, no body. rproxy passes the request to the app. With SWW_WAF_ADD_LOOKUP_HEADERS the answer also carries X-Client-ASN and X-Client-Country, which rproxy adds to the request it passes on. A header whose value is unknown is left out.
• X-Sww-Disposition: deny, with the status, headers and body the visitor is to get. rproxy gives them to the visitor, less this header. The status is one of:
  • SWW_WAF_BAN_RESPONSE, for a refusal;
  • 403, for a block rule or the Core Rule Set;
  • 431, for a request over the header limits;
  • 404, for a path under /_sww/ that passes the checks, so that the app never sees that prefix.
• X-Sww-Disposition: close, status 200, no body. This takes the place of a refusal while SWW_WAF_BAN_RESPONSE is close. rproxy closes the visitor's connection without an answer.
• No such header, status 400, for a question discriminator refuses. rproxy gives the visitor its status, headers and body, as it does for any answer without a valid X-Sww-Disposition.

In observe mode a request that enforce mode would refuse is answered allow, and its log line has would_action. A path under /_sww/ is still answered 404.

The Core Rule Set inspects the body a question carries only while SWW_WAF_CORE_RULE_SET_BODY_LIMIT is set. It inspects at most 64 KiB less one byte. So it does not inspect a JSON or XML body of 64 KiB or more, which may have been cut short.

Answering in time

rproxy stops waiting for discriminator after SWW_RPROXY_TIMEOUT_MS, by default 2 seconds, and SWW_RPROXY_DEFAULT_DISPOSITION then decides. So at the defaults, discriminator answers a request well within that time, whatever the request and whatever its outside sources do.

GeoJS is the only outside source a question can wait on. A question waits only when step 5 says so, and for at most SWW_WAF_LOOKUP_TIMEOUT, by default 800 milliseconds. A client without an answer by then counts as unknown. No question waits on these, which run in the background:

• the DNS blocklists and AbuseIPDB;
• fetching the blocklists, the CrowdSec decision list and the file SWW_WAF_ASN_LIMIT_PERCENT_URL names.

In enforce mode, a question does not wait for GeoJS when it is a request:

• over the header limits;
• from a client on a fetched blocklist, while SWW_WAF_BLOCKLIST_ACTION is deny;
• from a client on the CrowdSec decision list;
• that would take its client over a rate limit at 100% of it, since a percentage can only lower a limit.

Such a request still goes through the steps in their order. Its client's AS number and country stay unknown until GeoJS has answered. So while SWW_WAF_EXCLUSIVELY_ALLOWED_COUNTRIES is set, that list can refuse it first. SWW_WAF_UNKNOWN_LIMIT_PERCENT does not apply to it until then, so not waiting never lowers its limit.

The static lists and bans come before the lookup, so they never wait. These can still wait for GeoJS:

• a country list, which needs the answer;
• a rate limit a percentage has lowered, which needs the answer;
• a DNS blocklist or AbuseIPDB, while SWW_WAF_REPUTATION_ACTION is deny;
• a trap path;
• a rule file;
• the Core Rule Set.

The header limits are fixed, with no setting. discriminator refuses a request over either one:

• more than 50 cookies. Each part of its Cookie headers between semicolons counts as one, unless it is empty or only spaces and tabs.
• more than 100 header fields. A header sent twice counts twice. Host and Transfer-Encoding do not count. X-Request-ID and X-Forwarded-Proto do.

Real clients never reach these limits. The answer is deny with 431, and the log line's action is too_large. The refusal makes no ban and is no offence. The rate limits count the request as usual, so a client that keeps sending such requests breaks a limit.

The limits apply on every path and in every SWW_WAF_CORE_RULE_SET_MODE. A client in SWW_WAF_ALLOW_NETS skips them, as it skips the Core Rule Set. In observe mode such a request is answered allow with would_action, and the Core Rule Set does not read it.

The Core Rule Set runs each of its rules on each cookie, so thousands of small cookies would keep it busy for over a second. Within the limits, one long cookie or query string can still keep it busy for a fraction of a second. So enough such requests at once can overload discriminator past SWW_RPROXY_TIMEOUT_MS.

The report

Once the visitor's answer has ended, rproxy sends POST /_sww/report for each request discriminator answered with allow, deny or close. It sends none after:

• an answer without a valid X-Sww-Disposition, such as a 400;
• an answer it cannot pass on (see "Without an answer from discriminator");
• no answer.

The report carries no X-Sww-Client. Its body is one JSON object with all of these fields, named as the request log's fields of the same meaning:

• request_id: the question's X-Request-ID;
• status: what the visitor got, 0 if nothing, otherwise from 100 to 599;
• request_bytes and response_bytes: the body bytes each way, with what a WebSocket carried each way, each a whole number from 0 to 1125899906842624 (1 PiB);
• duration_total: milliseconds from the request's headers being read to the end of the answer, from 0 to 1000000000000 (about 31 years);
• duration_upstream_total: milliseconds from the hand-over to the app to the end of the answer, in the same range, and 0 for a request not passed to the app.

discriminator answers 400 and writes a process warning when the body:

• is not one JSON object, such as null, or an object followed by other data;
• lacks one of these fields, or gives it as null;
• gives a value of the wrong type, or out of its range;
• is over 1 MiB.

It answers 204 to any other report, also for a request it does not hold, which it ignores. It ignores a field not listed here. rproxy does not act on the answer.

For a request it holds, discriminator writes the request's last log line (see "The log"). For a request it allowed, it first:

• counts the bytes as SWW_WAF_BYTES_COUNT says, toward the client's byte limits at the percentages it gave that request;
• bans a client that passes a byte limit;
• adds the request with its status to the client's history, the anomaly counters and the metrics.

A denied request's counting is finished at its decision, so its report counts nothing.

discriminator holds each answered request until its report comes, up to 10,000 requests. Past that, the request answered longest ago is dropped. A request whose report never comes gets no last log line. If it was allowed, it is in neither the client's history nor the metrics, and its bytes are not counted.

discriminator holds only the decision's log line and what counting needs. It never holds the request or response objects, and no Authorization or Cookie. A report reaches only what is held, so it can never reach a question still being handled. discriminator ignores a report naming such a question.

The log

A request discriminator answers with allow, deny or close gets three request log lines, joined by request_id:

• rproxy's short line;
• discriminator's line at its decision;
• discriminator's line when the report comes, with everything known: the decision line's fields and the report's together.

The repetition is intended. The field discriminator_line says which of discriminator's two lines a line is. A request discriminator is not asked about, such as the health check, gets only rproxy's line. A request without a report gets no third line. docs/request-log.md gives the fields.

discriminator's own endpoints

A request without X-Sww-Client is for discriminator itself, at its own address on the private network. The internet must not be able to reach that address. Its endpoints are:

• the health check, the metrics, and the ban and client endpoints, as docs/admin-endpoints.md lists them, with their tokens;
• the report.

Anything else gets 404. No check runs on these requests, since their peers are rproxy and the admin's tools. The report needs no token, so anything on the private network can send one or ask a question.

A request and its answer have 60 seconds together, from the request's headers. discriminator closes a connection idle for 120 seconds. It takes a request line and headers up to 1 MiB, Go's default, so that it never refuses a question for the size of a visitor's headers.

Sites

One discriminator keeps one table of clients, counters and bans for every site that asks it. A client banned through one site is refused on all of them, and its requests on all of them count toward one set of limits. Log lines, alerts raised by a request and ban notes name the site by its host.

docs/metrics.md says how its metrics count.

rproxy

sww rproxy is the reverse proxy in front of one app. It asks discriminator about each request, and passes on what discriminator allows. It keeps nothing between requests and never touches disk. When discriminator cannot be connected to, does not answer at all, or does not answer within SWW_RPROXY_TIMEOUT_MS, SWW_RPROXY_DEFAULT_DISPOSITION decides. Its default, allow, keeps the site answering.

rproxy reads only its SWW_RPROXY_ settings, which docs/settings.md lists. One of them is required: SWW_RPROXY_WAF_BACKEND, discriminator's URL. It has no metrics, no rule files and no state files, and it writes its lines to stdout only. The image runs it by default (see "Deployment").

What rproxy does with a request

1. GET /_sww/healthz is answered ok at once, without asking discriminator.
2. rproxy works out the visitor's address. It believes X-Forwarded-For only from SWW_RPROXY_TRUSTED_PROXIES, and reads it as step 2 of "What happens to a request" says. It removes any IPv6 zone.
3. It makes a fresh request id, whatever id the request came with. discriminator and the app get it as X-Request-ID, and rproxy's log line gives it.
4. It refuses a request with a method it does not pass on, or one it cannot put to discriminator as the visitor sent it (see "Requests rproxy refuses itself").
5. It reads up to the first 64 KiB of the body into memory.
6. It sends discriminator the question (see "The question"). It speaks HTTP/1.1 to discriminator, whatever discriminator offers. Go's HTTP/2 client would refuse to send a visitor's Upgrade header, and most Connection headers. The question never carries the visitor's Trailer.
7. It does what the answer says (see "The answer"):
   • allow: it passes the request to the app.
   • close: it closes the visitor's connection without an answer.
   • Any other answer: the visitor gets its status, headers and body, less X-Sww-Disposition. That holds for a deny, for an error such as discriminator's 400 or a 5xx, and for an answer without a valid X-Sww-Disposition.
8. Once the answer has ended, it writes its log line. docs/request-log.md gives the fields. An app's answer can end just after the line: Go's server sends the closing chunk of a chunked answer, and may send all of an empty one, only once rproxy has finished with the request.
9. It then sends the report, if discriminator answered allow, deny or close (see "The report"). The visitor does not wait for it. A report that fails is dropped, with a process warning. A status above 599 from the app goes to the visitor as it is, and into the report as 599, the highest discriminator takes.

Requests rproxy refuses itself

rproxy passes on only these methods:

• GET
• HEAD
• POST
• OPTIONS
• PUT
• DELETE
• PATCH
• QUERY

It answers any other method 405, with no question and no report. The answer's Allow header names these methods.

rproxy answers 400, with no question and no report, to a request with:

• a target that does not start with /, such as x:admin, http://site.example/ or site.example:443;
• a target that starts with //;
• no Host, or an empty one;
• a Host holding an IPv6 zone: one that starts with [ and holds a %, such as [fe80::1%25eth0]:80;
• more than one User-Agent.

Go's HTTP client could not send any of these as the visitor sent it.

Go's HTTP server answers OPTIONS * itself, before rproxy sees it.

rproxy closes the connection without an answer when it cannot read the first 64 KiB of the body. That happens for a malformed chunk, and for a visitor that goes away or stops its sending side. There is no question and no report, and the log line has status 0.

Without an answer from discriminator

SWW_RPROXY_DEFAULT_DISPOSITION decides what is done with the request only when discriminator:

• cannot be connected to;
• does not answer at all;
• does not answer within SWW_RPROXY_TIMEOUT_MS, which covers connecting, sending the question and reading the whole answer.

An answer whose body breaks off counts as no answer at all.

allow, the default, passes the request to the app. deny answers 503. Either way rproxy writes a process warning, its log line has default_disposition, and it sends no report.

A visitor can make discriminator slow. A request that keeps discriminator busy past SWW_RPROXY_TIMEOUT_MS, like any request while discriminator is overloaded, gets SWW_RPROXY_DEFAULT_DISPOSITION. With deny, such a request gets 503 instead of reaching the app.

A path under /_sww/ that allow would pass on is answered 404 instead, so that the app never sees that prefix.

rproxy cannot pass on an answer whose body is over 1 MiB or whose status is below 200. The visitor gets 502 from rproxy, never the default. rproxy writes a process warning, its log line has no disposition, and it sends no report.

A visitor that goes away while discriminator is being asked is no failure of discriminator. rproxy closes the connection, with no warning, no default and no report. Its log line has status 0 and no disposition.

Passing a request to the app

rproxy sends the app at SWW_RPROXY_UPSTREAM_URL the visitor's request: the part of the body it holds, then the rest, streaming. The app's answer comes back the same way. Apart from what Go's HTTP code changes, the request reaches the app unchanged but for these headers:

• X-Forwarded-For, with the address of rproxy's TCP peer added, when that peer is inside SWW_RPROXY_TRUSTED_PROXIES. The peer's other forwarded headers pass on.
• For a peer outside SWW_RPROXY_TRUSTED_PROXIES, its forwarded headers are its own claims and are replaced. X-Forwarded-For names the peer, X-Forwarded-Host the host it asked for, and X-Forwarded-Proto plain http. The other headers in which traefik tells an app about a visitor are removed.
• X-Request-ID, which carries the fresh id.
• X-Client-ASN and X-Client-Country, which carry what discriminator answered. A header it did not give is left out. Any of the two the visitor sent is removed.
• The headers that concern only the connection to rproxy never reach the app:
  • Connection, and any header it names;
  • Keep-Alive;
  • Proxy-Authenticate;
  • Proxy-Authorization;
  • Proxy-Connection;
  • Te, though one holding trailers goes on as Te: trailers;
  • Upgrade.

A chunked body's trailers reach the app, but not discriminator.

A WebSocket upgrade passes through, with Connection: Upgrade and its Upgrade. The report of a WebSocket comes when it closes, with what it carried each way.

rproxy's limits

These limits apply per request, in both directions:

• SWW_RPROXY_MAX_REQUEST_SIZE is the largest request body.
• SWW_RPROXY_MAX_RESPONSE_SIZE is the largest response body.
• SWW_RPROXY_MAX_REQUEST_TIME_MS bounds the request line and headers. From their end, it then bounds the time until the app has been sent the whole request. The wait for discriminator counts toward it.
• SWW_RPROXY_MAX_RESPONSE_TIME_MS runs from the end of the request to the last byte of the answer reaching the visitor. The request ends when the app has been sent all of it, or when the app's answer arrives if that is sooner. An answer rproxy gives itself has the whole time from when it starts.

The largest request line and headers is 32 KiB. A kept-open connection is closed after 120 seconds idle. Neither has a setting. traefik, by default, closes an unused connection after 90 seconds. So traefik closes first, and never sends a request on a connection rproxy is closing.

What the visitor gets when a limit is passed:

• Go's HTTP server reads a request's line and headers before rproxy sees the request. It answers 431 to headers over 32 KiB. A visitor too slow to send them gets an error answer, mostly 400, or none. Either way the server closes the connection, nothing reaches the app, and the request log gets no line.
• A visitor too slow to send the first 64 KiB of its body is answered 408, with no question and no report.
• rproxy applies the request size limit only after discriminator has allowed the request. So a request discriminator denies gets discriminator's answer, whatever its size. A request discriminator allows is answered 413 when its body is over the limit: at once for a length it announced, otherwise when the limit is reached. If part of the body has gone to the app by then, the request to the app is cut off.
• A limit passed after that, but before the response has started, is answered by rproxy:
  • 408 for a visitor too slow to send the rest of its body;
  • 502 for a response that is too large, or an app that cannot be reached;
  • 504 for an app that is too slow.
• A request body streams through, so each side can hold up the other. SWW_RPROXY_MAX_REQUEST_TIME_MS can run out while the body is on its way. The answer is then 408 if rproxy was waiting for the visitor, and 504 if it was waiting for the app.
• Once the response has started it can only be cut off, and the connection is closed.
• A WebSocket connection leaves these limits behind once it is upgraded. It stays open until either side closes it.

Starting and stopping

An invalid setting, or an address it cannot listen on, stops the start with a process line. The process lines give the host's name as instance, and only those at info or more severe are written.

On SIGTERM or SIGINT, rproxy stops taking requests and gives those in progress five seconds to finish. It then waits for the reports on their way, each for up to SWW_RPROXY_TIMEOUT_MS. A report that comes due after that is dropped, such as that of a WebSocket that closes later.

Deployment

One image runs either command. SWW_COMMAND, in a container's environment, chooses: discriminator runs discriminator, and with it unset the image runs rproxy. docs/image.md lists what the image holds and how each part is pinned.

The usual setup

The usual setup is three containers on one private network:

• the app;
• rproxy, which traefik routes to, with SWW_RPROXY_UPSTREAM_URL naming the app and SWW_RPROXY_WAF_BACKEND naming discriminator;
• discriminator, with SWW_COMMAND=discriminator and a volume for its state files.

Each further app adds its own container and its own rproxy, whose SWW_RPROXY_WAF_BACKEND names the same discriminator (see "Sites"). Anything on that network can reach an app around rproxy, and can ask discriminator questions or send it reports. So the network holds only what you trust.

rproxy in the app's container

An app's Dockerfile can build FROM the smallwebwaf image instead, so that rproxy runs beside the app in one container. The app is then hardened by changing its base image, and can add packages from nixpkgs. discriminator still runs in a container of its own, and the app's container sets SWW_RPROXY_WAF_BACKEND.

Beyond its FROM line, the app's Dockerfile adds:

• its binary;
• any packages it needs, from nixpkgs, with nix-env -iA nixpkgs.<name>;
• its runit service: a directory under /etc/service named after the app, never sww. Its run script starts the app with chpst, listening where SWW_RPROXY_UPSTREAM_URL points (see "Ports"). The app runs as a user of its own, which the Dockerfile creates with useradd. That user is neither root nor sww, so that the app cannot touch the process of rproxy.

It sets no ENTRYPOINT or USER of its own. The container must start runsvinit, as root, so that it can start each service as its own user. "Getting started" in README.md shows such a Dockerfile and run script.

The image

• The image is Ubuntu's newest long-term support release, pinned by digest, and moves to the next LTS release when that ships. Its packages come from Ubuntu's archive, whose packages change with every update.
• The Ubuntu image has no CA certificates. Without them Nix cannot download packages and discriminator cannot reach GeoJS, so ca-certificates is installed by name. apt fetches the packages over plain HTTP and checks each one against the archive's signed package lists, so the install needs no CA certificates.
• nixpkgs comes from a release file on releases.nixos.org, named with its hash, not from a GitHub archive, whose bytes can change. Root uses Nix directly; no Nix daemon runs. nix-bin creates no build users, so /etc/nix/nix.conf sets build-users-group =, which lets root build without them. Root's Nix profile is last on the PATH, so that no package hides the image's own commands; busybox's sv, for one, looks for services elsewhere. An app built on the image gets the same packages each time. Security fixes from Ubuntu and nixpkgs come with a newer smallwebwaf image, which an app takes by changing the digest in its FROM line. The image moves to each new nixpkgs release, every six months.
• runit supervises a container's processes, with runsvinit as the entrypoint, as the owner's code style guide asks for service containers. No package carries runsvinit, so the image builds it from its archived source, unchanged since 2015, at a fixed commit hash, with a go.mod the build writes. runsvinit starts runit's runsvdir, which runs the run script of each directory under /etc/service. Ubuntu's sv looks for services there too.
• The image's run script is /etc/service/sww/run. It starts discriminator when SWW_COMMAND is discriminator, and rproxy when it is rproxy, unset or empty. It runs either as the sww user (uid and gid 65532), with runit's chpst. For discriminator, it first makes SWW_WAF_STATE_DIR and every file in it belong to that user. For rproxy it changes no owner. With any other value, it starts nothing, and stops with a message naming SWW_COMMAND and its value.

Every run script, the app's included, is a bash script. It starts with #!/usr/bin/env bash and set -euo pipefail, and puts its code in a main function, called on its last line, as the owner's code style guide asks. Ubuntu ships bash. Each waits one second (sleep 1) before starting its program, so a process that stops at start is started again every second, not in a tight loop.

Each container's processes

• runsvinit, runsvdir and each service's runsv run as root. Each run script starts as root and hands over to its program's user with chpst. Every service starts with the container's environment variables and writes to the container's output, which docker logs shows.
• When a service exits, runit runs its run script again, and the container does not stop. Beside an app, the other process keeps running.
• While rproxy is down, traefik answers 502. While the app is down, rproxy answers 502, which is not a client offence. While discriminator is down, SWW_RPROXY_DEFAULT_DISPOSITION decides (see "Without an answer from discriminator").
• Only faults found at start stop discriminator:
  • invalid configuration;
  • a lookup database that is missing or unreadable;
  • a rule file or state file that does not parse;
  • an unwritable SWW_WAF_STATE_DIR;
  • a SWW_WAF_RULES_DIR that does not exist or cannot be read;
  • a SWW_WAF_LISTEN_ADDR it cannot listen on.
• Only an invalid setting, or a SWW_RPROXY_LISTEN_ADDR it cannot listen on, stops rproxy.
• A process that stops at start logs its message at each try until the cause is fixed; its container's health check fails meanwhile. Once running, a failing GeoJS, reputation source or alert destination, or a broken edit, never takes discriminator down.
• docker stop sends SIGTERM to runsvinit, which has runit stop every service at once, each with SIGTERM, and exits once they have stopped. rproxy and discriminator then stop as "rproxy" and "discriminator" say.
• With rproxy beside the app, an app that exits at once on SIGTERM ends the requests in progress: the visitor gets 502, or a response cut short. To let them finish, the app must keep answering for five seconds after SIGTERM.
• The container's root filesystem stays writable: runit writes each service's status into its directory under /etc/service.

The health check

The image's HEALTHCHECK runs sww healthcheck. SWW_COMMAND says what it checks:

• With SWW_COMMAND=discriminator, it passes while discriminator answers GET /_sww/healthz on 127.0.0.1, at the port in SWW_WAF_LISTEN_ADDR.
• With SWW_COMMAND unset, empty or rproxy, it passes while rproxy answers that request on 127.0.0.1, at the port in SWW_RPROXY_LISTEN_ADDR, and the app accepts connections at SWW_RPROXY_UPSTREAM_URL. It does not ask discriminator, so that the site stays in traefik's routes while discriminator is down.
• With any other value, it fails with a message naming SWW_COMMAND and its value.

traefik sends a container no requests until it shows as healthy. So the check runs every second from the container's start until it first passes, for up to a minute, and every 30 seconds after that. An app with a health check of its own can replace the image's HEALTHCHECK with one that checks both.

Ports

rproxy listens on no port but the one in SWW_RPROXY_LISTEN_ADDR, 8080 by default, which traefik routes to. It passes requests to the app at SWW_RPROXY_UPSTREAM_URL, http://127.0.0.1:8081 by default. Another port may be set: the health check follows the setting, and traefik's port label (traefik.http.services.<name>.loadbalancer.server.port) must name it. An app beside rproxy must leave that port free.

discriminator listens on no port but the one in SWW_WAF_LISTEN_ADDR, 8080 by default. The questions, the reports and its own endpoints all come there.

The address part of either setting stays empty (:9000, never 127.0.0.1:9000). That way the others reach the process on the container's address, and the health check reaches it on 127.0.0.1.

An app beside rproxy listens on that loopback address only. An app that listens on every address can be reached around rproxy by anything that reaches the container.

State

discriminator keeps its state files in SWW_WAF_STATE_DIR, belonging to the sww user. Without a volume there, the files survive a restart of the container and are lost when a deploy replaces it. A volume mounted at /var/lib/sww, named or a host directory, keeps them across deploys. A host directory mounted there needs no change of owner, and files an earlier owner left in it can be read and replaced. The volume holds a few tens of MiB at most with the defaults, and needs no backup beyond the host's own. The image declares no volume, since every app image built on it would inherit it. rproxy keeps nothing, and needs no volume.

Tokens

A token given as a file (SWW_WAF_ADMIN_TOKEN_FILE, SWW_WAF_METRICS_TOKEN_FILE) stays out of the container's environment. The operator makes the file on the host, owned by uid 65532, the sww user, with mode 0400, and mounts the directory that holds it into discriminator's container read-only. The container sees the same owner and mode. "Settings given as files" in docs/settings.md shows the commands.

Forwarded headers

The app's TCP peer is rproxy, and the X-Forwarded-For the app receives ends with traefik's address, which rproxy adds after the visitor's. So the app must trust rproxy's address for forwarded headers, besides the private address ranges traefik is on.

• In a container of its own, rproxy's address is on the private network. The usual default for a trusted-proxy setting, the private ranges, covers it.
• Beside the app, rproxy's address is 127.0.0.1 or ::1. The private ranges leave out loopback: an app left at them ignores the header and sees every visitor as 127.0.0.1. A list given replaces that default, so the app's own trusted-proxy setting names them all: 10.0.0.0/8,172.16.0.0/12,192.168.0.0/16,127.0.0.1/32,::1/128.

upaas

upaas rebuilds and restarts an app's Docker container when its repository changes. An app it deploys takes the shape of "rproxy in the app's container" with no change to upaas. upaas builds the app's Dockerfile and runs the one container as it runs any app. The traefik labels route to port 8080, and the SWW_RPROXY_ settings go with the app's environment variables. discriminator runs in a container of its own, on a network the app's container can reach, with its state volume and the directory of any token file.

Planned, not built

This point is specified as later work, and nothing in the code does it yet:

• Planned: sending the log over RELP, the reliable syslog transport. It is revisited against the available Go packages. Until then the remote log goes over UDP, TCP or TLS as described under "Request log".