smallwebwaf/SPEC.md

# smallwebwaf specification

`smallwebwaf` is a web application firewall for self-hosted apps. It is one
binary, `sww`, and two of its commands do the work:

- `sww rproxy` is a reverse proxy between traefik and one app.
- `sww discriminator` decides what happens to each request, for many sites.

This document says what it must do and why, each requirement once. Everything in
it is built, except what the last section lists. `README.md` says how to run it.
`EVALUATION.md` says why no existing tool was adopted. The reference documents
under `docs/` give the settings, the formats and the lists. This document links
to them.

## Purpose

- It protects self-hosted apps on the open internet from scrapers, scanners and
  other abusers. It does so from an app's first request. `rproxy` requires only
  the address of `discriminator`. In a container of its own it also needs the
  app's address. Every other setting has a default chosen for a service facing
  the internet in 2026. A setting that needs an alert destination, an account
  key or a token stays off until the operator supplies it.
- It keeps a site answering while the part that decides is down, slow or
  restarting (see "Without an answer from `discriminator`").
- It limits each client's requests and bytes per minute, hour and day. It bounds
  the size and duration of every request and answer. It detects attacks.
- It bans abusers: briefly at first, for seven days on a clear sign of attack,
  and permanently when they keep at it.
- It looks up each client's AS number and country. It can refuse whole countries
  and netblocks, lower its limits for listed AS numbers and countries, consult
  reputation sources, and send alerts.
- Everything is set by environment variables. The names of `discriminator`'s
  start with `SWW_`, and those of `rproxy` with `SWRP_`. The only other inputs
  are `discriminator`'s files:
    - the rule files;
    - the state files;
    - a lookup database;
    - a `SWW_` setting given as a file;
    - the file `SWW_LOG_REMOTE_TLS_CA_FILE` names;
    - the file `SWW_CORE_RULE_SET_EXCLUSIONS_FILE` names.

## Non-goals

- TLS, certificates and routing by host name. Those are traefik's job.
- More than one app behind one `rproxy`. Each app has its own.
- Browser challenges such as captchas or proof of work. If wanted, chain Anubis
  between `rproxy` and the app.
- Defence against floods that saturate the host's network link. That needs help
  upstream of the host.
- A web console or a configuration file.

## Architecture

- One statically linked Go binary. Only `discriminator` writes to disk, and only
  to its state directory. It ships in one image, which runs either command (see
  "Deployment").
- No database of any kind. Nothing is read from disk while serving a request.
- Packages:
    - `cmd/sww`: the binary, which only calls `internal/sww`.
    - `internal/sww`: the process, as the command it is given.
      `sww discriminator` reads the settings, the rule files, the lookup
      database and the state files, listens, and serves until `SIGTERM` or
      `SIGINT`. Then it stops, writing the state files (see "`discriminator`").
      `sww rproxy` reads only that command's settings, and runs
      `internal/rproxy` until `SIGTERM` or `SIGINT` (see "`rproxy`").
      `sww healthcheck` is the image's health check. With no command, or one it
      does not know, the binary prints the commands and exits with status 2.
    - `internal/config`: reads the settings, the one place they are read.
    - `internal/discriminator`: `discriminator`'s server. It runs the checks in
      `check`, and holds `discriminator`'s question, answer and report. It
      answers `discriminator`'s own endpoints in `answerOwnEndpoint`. It counts
      bytes, refusals and anomalies, and writes the log lines.
    - `internal/rproxy`: the reverse proxy in front of one app that asks
      `discriminator` about each request. It passes a request to the app with
      the standard library's `httputil.ReverseProxy`, within the time and size
      limits.
    - `internal/ratelimit`: the table of clients. It counts requests, bytes and
      refusals, and keeps each client's history.
    - `internal/bans`: the bans. It holds each netblock's bans with their notes,
      decides how long a new ban lasts, and picks the ban to drop when the table
      of bans is full.
    - `internal/rules`: reads the rule files at start and again when they
      change, and tells which rules a request matches.
    - `internal/waf`: the Core Rule Set, with the six changes and the apps'
      fixes listed under "Attack detection", as Coraza directives.
    - `internal/lookup`: looks clients up through GeoJS or in the lookup
      database.
    - `internal/reputation`: fetches the lists, asks the DNS blocklists and
      AbuseIPDB in the background, and keeps their answers.
    - `internal/anomaly`: the anomaly counters and their alerts.
    - `internal/alerts`: holds back repeats and alerts past the hourly limit,
      and sends the rest to the webhook, Slack and ntfy, each from a queue of
      its own.
    - `internal/state`: reads the state files at start, takes in edits while
      running, and writes the files when due and at the stop.
    - `internal/requestlog` and `internal/remotelog`: the lines on stdout, and
      sending them to the syslog server.
    - `internal/metrics`: the metrics, in the Prometheus text format.
    - `Dockerfile`: the lint and test phases, then the image.
      `deploy/example-app` is an app built on the image, with `rproxy` in its
      container, which `script/example-app` checks with `discriminator` in
      another.
- Libraries, checked against the owner's Go dependency defaults:
    - the standard library for the reverse proxy, HTTP clients, DNS, logging
      (`log/slog`) and the state files (`encoding/json`, `os.Rename`);
    - `github.com/corazawaf/coraza/v3` and
      `github.com/corazawaf/coraza-coreruleset/v4` at `v4.25.0`, which carries
      the Core Rule Set 4.25.0. The changes `smallwebwaf` makes to it, and the
      default `SWW_WAF_DISABLED_RULES`, are built for that version. They are
      checked again before the version changes, since rule ids, and what each
      rule matches, change between versions;
    - `github.com/hashicorp/golang-lru/v2` for the bounded tables;
    - `github.com/oschwald/maxminddb-golang/v2` to read the lookup database;
    - `github.com/fsnotify/fsnotify` to notice edited files;
    - `github.com/prometheus/client_golang` for the metrics;
    - `github.com/maxmind/mmdbwriter`, in the tests, to write lookup databases;
    - remote log sending written in this project, since the standard library's
      `log/syslog` is frozen and writes only the older syslog format.

## What happens to a request

`rproxy` runs steps 1, 2 and 14, and sends the report of step 15.
`discriminator` runs steps 3 to 13, and does the counting of step 15. The first
step that produces a final answer ends the checks. Step 15 runs for each request
`discriminator` answers with `allow`, `deny` or `close`. `rproxy`'s size and
time limits apply to the whole exchange. Each step names the section that holds
its rules.

1. `GET /_sww/healthz` is answered at once by `rproxy`, before any check, so a
   health checker never needs an exemption and is never refused.
2. The client is identified.
    - If the TCP peer is inside `SWRP_TRUSTED_PROXIES`, `X-Forwarded-For` is
      walked from the right, and the first address outside those netblocks is
      the client. If every address is inside them, the leftmost is, so that a
      visitor on a private network is known by its own address. An entry that is
      not an address ends the walk. With no header, the peer is the client.
    - If the peer is outside `SWRP_TRUSTED_PROXIES`, it is the client, and the
      header is ignored.
    - `rproxy` then sends `discriminator` the question (see "The question").
      `discriminator` takes the address the question gives as the client.
    - An IPv6 client is its group: the netblock of `SWW_IPV6_GROUP_PREFIX`
      around its address, a /64 by default, since one abuser usually controls a
      whole /64. A client is therefore one IPv4 address or one IPv6 group.
3. The static lists. A client in `SWW_ALLOW_NETS` skips steps 4 to 12, the byte
   limits and the error burst. A client in `SWW_DENY_NETS` is refused; a client
   in both lists is let through.
4. Bans. A client under a ban is refused (see "Bans").
5. The lookup, unless `SWW_LOOKUP_SOURCE` is `off` (see "Lookup of AS number and
   country"). The request goes on at once. It waits for the first GeoJS answer,
   up to `SWW_LOOKUP_TIMEOUT`, only when one of these needs it:
    - a country list;
    - a biased threshold;
    - `SWW_ADD_LOOKUP_HEADERS`.

    A request that ends before the answer comes is logged without it.

6. The country lists (see "Country lists").
7. The reputation sources (see "Reputation"). A client on a fetched blocklist
   gets `SWW_BLOCKLIST_ACTION`. A client on the CrowdSec decision list is
   banned. A client listed by a DNS blocklist, or scored at least
   `SWW_ABUSEIPDB_MIN_SCORE` by AbuseIPDB, gets `SWW_REPUTATION_ACTION`.
8. The client's limit percentage (see "Biased thresholds").
9. Rate limiting (see "Request rate limits, per client"). A request over a limit
   times the percentage is refused and bans the client.
10. The trap paths (see "Attack detection"). A request for one is refused and
    bans the client.
11. The rule files (see "Rule files"). Matching stops at the first rule that
    refuses.
12. The header limits (see "Answering in time"), and then the Core Rule Set (see
    "Attack detection"). A Core Rule Set match is refused with `403` while
    `SWW_WAF_MODE` is `block`, and only logged and alerted while it is `detect`.
13. A path under `/_sww/` is answered `404`, so that the app never sees that
    prefix.
14. `rproxy` passes the request to `SWRP_UPSTREAM_URL`, streaming (see "Passing
    a request to the app"). With `SWW_ADD_LOOKUP_HEADERS`, `X-Client-ASN` and
    `X-Client-Country` are added for the app's own logs; a header whose value is
    unknown is left out. Whatever that setting says, any such header the client
    sent is removed.
15. After the visitor's answer has ended, `rproxy` sends the report (see "The
    report"). `discriminator` then counts:
    - For a request it allowed, the response bytes, the request bytes or both
      are added to the byte counters, as `SWW_BYTES_COUNT` says. For a
      WebSocket, what it carried each way counts once it closes. A client whose
      total passes a byte limit times the percentage has broken that limit and
      is banned. The response in progress is not cut off.
    - For a request it allowed, the client's history and the anomaly counters
      are updated, and the alert thresholds are evaluated.

    For a request it refused, `discriminator` adds nothing to the byte limits.
    It updates the history and the anomaly counters at its decision. It counts
    the refusals the error burst counts at the decision too (see "Attack
    detection").

In `SWW_MODE=observe`, every refusal above is logged as what would have
happened, nothing is refused, and `discriminator` makes no ban of its own. A
path under `/_sww/` is still answered `404`, and `rproxy`'s size and time limits
still apply.

## Counting method

- Each window (minute, hour, day) uses two adjacent fixed buckets per client.
  The earlier bucket is weighted by how much of it still overlaps the sliding
  window. This costs a few integers per client per window and avoids the burst
  at bucket boundaries that a single fixed bucket allows.
- Counters are read and updated in memory. They are written to `clients.json`
  and loaded again at start (see "Persistent state"), so a restart does not hand
  every client a fresh allowance. Buckets whose time has passed are discarded on
  load.
- Memory is bounded. When the table of clients holds `SWW_MAX_TRACKED_CLIENTS`,
  the least recently seen client is dropped first, with its history. The bans
  are bounded by `SWW_MAX_BANS` (see "Bans"). The GeoJS answers, the reputation
  verdicts and scores, the DNS queries under way and the anomaly counters have
  fixed bounds, listed under "Fixed limits" in `docs/settings.md`. Past its
  bound, the GeoJS answer or anomaly counter used longest ago goes first, and
  the verdict or score fetched longest ago. Past the bound on DNS queries under
  way, a zone is asked at the client's next request.

## Bans

Until a ban ends, every request from its netblock is refused with
`SWW_BAN_RESPONSE`. The netblock is the client: its IPv4 address, widened to
`SWW_BAN_SCOPE_V4_PREFIX`, or its IPv6 group. At the defaults a ban covers one
address or one /64, so a permanent ban does not reach neighbours who did
nothing. Requests refused under a ban are not counted against limits. They are
counted in the client's history and in the ban's notes.

An offence is a request `discriminator` holds against the client. It is one of:

- a clear sign of attack;
- a broken limit;
- a refusal by a rule file or by the Core Rule Set.

Offences are counted by kind in the client's history. The first two kinds lead
to a ban, each by its own rule:

- A clear sign of attack: a request that matches a rule whose action is `ban`,
  or asks for a path in `SWW_TRAP_PATHS`. Examples are a probe for a `.env` file
  or a `.git` directory, and a known scanner's user agent.
    - The first one bans the netblock for `SWW_ATTACK_BAN_DURATION` (default
      `7d`), with the cause `attack`.
    - Any further request from the netblock while that ban lasts shows it is
      malicious: the ban becomes permanent. No other cause of ban becomes
      permanent this way.
    - Once that ban has run out, the netblock is served like any other. Its next
      clear sign of attack bans it permanently at once.
- A broken limit, one of:
    - a request over a rate limit;
    - a response that takes the client's byte total past a byte limit;
    - more than `SWW_ERROR_BURST_THRESHOLD` refusals within a minute.

    The bans for a broken limit:
    - The first such ban lasts `SWW_LIMIT_BAN_DURATION` (default `1h`), with the
      cause `limit`. So does one that comes more than
      `SWW_LIMIT_BAN_REPEAT_WINDOW` (default `24h`) after the last such ban
      ended.
    - Breaking a limit again within that window makes the new ban three times as
      long as the last one: one hour, then 3, 9, 27 and 81 hours.
    - A ban that would be longer than `SWW_MAX_BAN_DURATION` (default `7d`) is
      permanent instead. At the defaults, that is the sixth ban in a row.
    - Each such ban sets the client's counters for every limit back to zero.
      Once it ends, only new requests can break a limit again. The client's
      history keeps its totals.

A Core Rule Set match refuses only the request it matched, with `403` and
without a ban, because the Core Rule Set has false positives. A `block` rule
does the same. Those refusals count toward the error burst, so a client that
keeps setting them off is banned for a broken limit.

A client on the CrowdSec decision list is banned until the decision ends, with
the cause `crowdsec`. Only a client that sends a request is banned, so a long
list does not fill `bans.json`. Such a ban is never made permanent. It does not
make the netblock's next ban for a broken limit longer. A ban lifted while the
decision is in force is made again at the client's next request. To lift it for
good, delete the decision in CrowdSec, then lift the ban once the list has been
fetched again.

These refuse each request they cover, record nothing in `bans.json`, and count
no offence:

- `SWW_DENY_NETS`;
- a blocklist or reputation hit whose action is `deny`;
- the country lists.

Every ban carries notes with what an admin needs to decide whether to lift it,
drawn from what `discriminator` already knows. No lookup is made for the notes
alone. `docs/state-files.md` lists the notes with the rest of a ban's entry.

An admin adds, keeps or lifts a ban by editing `bans.json` (see "Persistent
state") or through the ban endpoints (see "Admin endpoints"). A ban an admin
makes has the cause `admin`; an admin who wants to keep a ban `discriminator`
made sets its cause to `admin`. A ban lifted before it ends is kept, marked as
lifted, and does not count toward a longer ban; it refuses nothing. Deleting its
entry from `bans.json` forgets it entirely.

A permanent ban does not run out. It ends when an admin lifts it, or when
`discriminator` drops it from a full table, as below. When `discriminator` holds
`SWW_MAX_BANS` bans of its own, it drops the earliest ban of one netblock: the
netblock that has gone longest without a request. A past, active or permanent
ban is dropped alike. A scanner whose ban was dropped that way is refused and
banned again by its next probe. A ban whose cause is `admin` is never dropped
and does not count toward `SWW_MAX_BANS`, so such bans can never fill the table.

## Settings

`docs/settings.md` lists every setting with its default and the values it takes,
in the groups below. It also gives the form each kind of value takes. The groups
below are `discriminator`'s settings; "`rproxy`" gives the use of `rproxy`'s.
Conventions:

- These settings take `off`, which switches them off:
    - `SWW_LOOKUP_SOURCE` and `SWW_WAF_MODE`;
    - the three rate limits and the three byte limits;
    - `SWW_WAF_ANOMALY_THRESHOLD`, `SWW_WAF_BODY_LIMIT` and
      `SWW_ERROR_BURST_THRESHOLD`;
    - `SWW_ALERT_COOLDOWN` and `SWW_ALERT_MAX_PER_HOUR`;
    - every anomaly threshold;
    - `SWRP_MAX_REQUEST_SIZE`, `SWRP_MAX_RESPONSE_SIZE`,
      `SWRP_MAX_REQUEST_TIME_MS` and `SWRP_MAX_RESPONSE_TIME_MS`.

    No other setting can be switched off with `off`.

- A list set to an empty value is an empty list, and replaces the default.
- Any `SWW_` setting can be read from a file instead, for secrets and long
  lists: `SWW_ADMIN_TOKEN_FILE` names the file that holds `SWW_ADMIN_TOKEN`. The
  file's contents are the value, less one newline at the end, and are checked as
  the value itself would be. Setting both forms stops the start, and so does
  naming a file that cannot be read. `SWW_LOG_REMOTE_TLS_CA_FILE` and
  `SWW_CORE_RULE_SET_EXCLUSIONS_FILE` already hold a file name, so they have no
  `_FILE` form. `SWW_COMMAND` cannot be given as a file. No `SWRP_` setting has
  one, since `rproxy` never touches disk.
- Settings, files included, are read once at start. Changing one means
  restarting the container. The files `discriminator` watches while it runs are
  its state files, its rule files and the lookup database.
- An invalid setting stops the process at start with one log line, marked
  `"type":"process"` with `"msg":"invalid setting"`. The line names the variable
  and what is wrong with the value; "Each container's processes" says what
  happens then. At start the settings in effect are logged on one line; tokens,
  keys and the paths of webhooks, which often carry a secret, are shown as
  `********`.

### Core

- `SWW_INSTANCE_NAME` is in each line `discriminator` logs, and in each metric
  and alert, so that instances can be told apart. The default, the container's
  host name, changes at every redeploy; a name set by the operator survives it.
- `SWW_MODE=observe` lets a configuration be tried before it is enforced (see
  "What happens to a request").
- `SWW_IPV6_GROUP_PREFIX` sets the group size. After a change, each IPv6 client
  starts afresh; existing bans keep refusing their netblocks until they end.
  `SWW_MAX_TRACKED_CLIENTS` and `SWW_MAX_BANS` set the bounds under "Counting
  method" and "Bans".

### Tokens

- `SWW_ADMIN_TOKEN` switches on the ban endpoints and `/_sww/clients/<ip>`. The
  metrics are off until `SWW_METRICS_TOKEN` is set, since without a token they
  would be open to anything on `discriminator`'s network. The two are separate,
  so the metrics collector, which holds the metrics token, cannot manage bans.
- A token is a long random value. One shorter than 32 characters stops the
  start. The file form keeps a token out of the container's environment (see
  "Tokens" under "Deployment").

### State files

- `SWW_STATE_DIR` names the directory of the state files (see "State" under
  "Deployment").
- `bans.json` is written `SWW_STATE_WRITE_DELAY` after a ban is made, lifted or
  made permanent, so a burst of changes becomes one write. Every file is written
  every `SWW_STATE_COUNTER_INTERVAL`, and at an orderly stop.

### Logging

- The request log on stdout is always on and has no switch (see "Request log").
  `SWW_LOG_LEVEL` filters only the process's own messages, and no message is at
  `debug`. `warn` and `error` hold back the line of settings at start.
- `SWW_LOG_REQUEST_HEADERS` names the request headers to record.
  `Authorization`, `Cookie` and `Set-Cookie` values are never logged, even if
  listed. The log says only whether `Authorization` and `Cookie` were present.
- `SWW_LOG_REMOTE_URL` names a syslog server (see "Request log").

### Metrics

- `SWW_METRICS_TOP_N` is how many of the busiest AS numbers and countries get a
  series of their own. The rest are summed as `other`, so the label set stays
  bounded.

### Static lists

- `SWW_ALLOW_NETS` skips the checks step 3 names, for monitoring or the owner's
  own networks. `SWW_RATE_LIMIT_EXEMPT_NETS` bypasses the request and byte
  limits only; the error burst, attack detection and bans still apply.
- Many people can share one address, as in an office, on a mobile carrier or
  through Tor. They are one client, and share its limits and its bans. A known
  shared address busier than the limits allow belongs in `SWW_ALLOW_NETS` or
  `SWW_RATE_LIMIT_EXEMPT_NETS`.

### Request rate limits, per client

Breaking one bans the client. The defaults let people read a site and stop
scrapers that send many requests slowly over hours or days. The hour limit is
the main one.

- Per minute: 120, 2 requests a second. It leaves room for a person opening two
  pages at once with an empty cache. A web app's page opened with an empty cache
  loads 20 to 60 files, and a handful once cached.
- Per hour: 1,500. A person reading a site for a whole hour, a page every 20
  seconds, makes about 900 requests. A scraper making one request every 2
  seconds makes 1,800 an hour, more than the limit.
- Per day: 6,000, four hours at the hour limit. A scraper making one request
  every 10 seconds makes 8,640 a day, more than the limit.

Counting cannot catch a client that stays under all three limits, about 4
requests a minute around the clock. Only the other checks can catch an abuser
that slow. They include the trap paths, the rule files, the Core Rule Set and
the blocklists. An abuser that trips none of them is not caught.

Some legitimate clients pass the limits:

- A CI runner asks its code forge for work all day and can pass the limits.
- Everyone behind one shared address, such as an office, shares one client's
  limits.

Their addresses belong in `SWW_RATE_LIMIT_EXEMPT_NETS`.

`SWW_RATE_LIMIT_EXEMPT_PATHS` takes path prefixes out of the count, such as
static assets or health checks; their bytes are not counted either. A prefix is
matched exactly as sent, without the query, and is not percent-decoded:
`/assets/` matches `/assets/app.js`, and does not match `/Assets/app.js`,
`/%61ssets/app.js` or `/static/assets/app.js`. There are no wildcards. A path
holding `..`, a backslash or an encoded slash (`%2F`) is never exempt, since the
app may read it as a path outside the prefix. `SWW_WAF_EXEMPT_PATHS` is matched
the same way.

### Byte limits, per client

By default there is no minute limit, so a client can download in a burst at full
speed. The hour limit defaults to 50 GiB and the day limit to 100 GiB. They
allow several downloads of a few GiB each. A response's bytes are counted when
it ends (see step 15).

### Lookup of AS number and country

On by default through GeoJS, which needs no account or file, so that country
lookup works with no setup. A local database file is the alternative for a
service that keeps its visitors' addresses on its own host.

- Only one source is used. The start stops, naming both settings, when `file` is
  set without `SWW_LOOKUP_DB_PATH`, or `SWW_LOOKUP_DB_PATH` with any other
  source. The start also stops when the source is `off` and a setting that needs
  a lookup is set (`docs/settings.md` lists them).
- GeoJS is told the address of every new visitor, whether or not a setting uses
  the answer, since the request log, the metrics and the ban notes carry it.
  These addresses are never sent:
    - one in `SWW_ALLOW_NETS` or `SWW_DENY_NETS`;
    - one under a ban;
    - a private, loopback or link-local address, which no source can place.
- For an IPv6 visitor, GeoJS is asked about the first address of its group.
  Addresses are asked about together, up to 200 in one request
  (`https://get.geojs.io/v1/ip/geo.json?ip=a,b,c`), one request at a time, which
  keeps the number of requests to GeoJS low. Each answer's `country_code`, `asn`
  and `organization_name` are read. An answer without a `country_code` leaves
  the country unknown, and the `asn` `64512` leaves the AS number unknown.
- Each answer is kept for seven days, in memory and in `lookups.json`. Answers
  are kept apart from the table of clients, so that an answer outlasts a client
  dropped from that table. The answer reaches the client's history and ban notes
  when it comes.
- `SWW_LOOKUP_TIMEOUT` is how long a visitor's request waits for its client's
  first answer, when a setting needs it (see step 5). It also bounds each
  request to GeoJS. A client whose answer has not come in time counts as unknown
  until it comes, and its later requests do not wait. Its address is asked about
  again in the background.
- A client the lookup cannot place has an unknown country and AS number, which
  the country lists and the biased thresholds treat as described there. Nothing
  else treats it differently.
- GeoJS publishes no rate limit, but its terms forbid "an excessive amount of
  API requests", judged by GeoJS alone, and let it block a caller. While GeoJS
  is slow, down or refusing `discriminator`, new clients count as unknown. GeoJS
  is then left alone for a second, and twice as long after each further failure,
  up to five minutes. Each failure raises a `source_failure` alert (see
  "Alerting"). A service that cannot accept this uses the file.
- The file is the free IPinfo Lite database (`ipinfo_lite.mmdb`), whose `asn`,
  `as_name` and `country_code` fields are read. The operator downloads it with a
  free IPinfo account and mounts it read-only; `discriminator` never fetches it
  and holds no account or token for it. IPinfo updates it daily. Its licence is
  Creative Commons Attribution-ShareAlike 4.0, and IPinfo asks for a link to it
  on the site that uses it, such as "IP address data is powered by IPinfo".
- The file is read whole at start; one that is missing or unreadable stops the
  start. It is read again once it has gone 2 seconds without a change after
  being replaced. A replacement that cannot be read is logged, sent as one
  `file_error` alert and ignored. The database already loaded stays in use. A
  client whose address the file does not hold counts as unknown.

### Country lists

Both are empty by default: the defaults judge a client by what it does, not by
where it comes from.

- `SWW_DENIED_COUNTRIES` refuses every request from a listed country.
- `SWW_EXCLUSIVELY_ALLOWED_COUNTRIES` lets through only clients placed in a
  listed country. Every request from any other country is refused. So is every
  request from a client the lookup cannot place, such as internal monitoring or
  a visitor on a private address; those that should reach the app go in
  `SWW_ALLOW_NETS`. A list that let unplaced clients through would let every new
  client in whenever GeoJS stops answering. The cost: while GeoJS fails, this
  list refuses every new client. No other failure of GeoJS, a reputation source
  or an alert destination comes this close to taking the service down.
- Both may be set. `SWW_DENIED_COUNTRIES` then adds nothing, since the exclusive
  list already refuses every other country. A code on both lists stops the start
  with a message naming it.
- A refused request is answered with `SWW_BAN_RESPONSE`, as soon as the client's
  address has been looked up. It is logged with the action `country_denied`, and
  counted in the metrics.

### Biased thresholds

The biased thresholds lower the limits of clients in listed AS numbers and
countries. The lists are empty by default, for the same reason as the country
lists. The request log and the metrics show which AS numbers and countries a
service's abuse comes from.

- Each list gives AS numbers or countries a percentage, such as `AS14061:50`.
  Their clients get that percentage of every request and byte limit
  (`SWW_ASN_LIMIT_PERCENT`, `SWW_COUNTRY_LIMIT_PERCENT`). A lowered limit is
  rounded down. `0` is a zero allowance: the client's first request breaks a
  limit and bans it.
- `SWW_ASN_BYTES_PERCENT` and `SWW_COUNTRY_BYTES_PERCENT` set the byte limits'
  percentage alone, when it should differ from the request limits'. For an AS
  number they list, they replace its percentage from `SWW_ASN_LIMIT_PERCENT` and
  `SWW_ASN_LIMIT_PERCENT_URL`; for a country, the one from
  `SWW_COUNTRY_LIMIT_PERCENT`.
- `SWW_ASN_LIMIT_PERCENT_URL` names a text file of `AS:percent` lines, so one
  abuse-source list can be shared by every `discriminator` in the fleet. It is
  fetched and kept like a blocklist (see "Reputation"). Where it and
  `SWW_ASN_LIMIT_PERCENT` both give a percentage, the lower applies.
- A client's limit percentage is the lowest of those that apply to it: its AS
  number's, its country's, or a reputation hit's `limit:<percent>`. A client
  without a country gets `SWW_UNKNOWN_LIMIT_PERCENT` in place of a country's.
  The percentage is 100 if none applies. Clients given a lower percentage are
  banned after fewer requests, first for a while and then, for repeated abuse,
  permanently. The log line of each counted request, and the notes of each ban,
  say which setting lowered the limit.
- No budget is shared by a whole country or AS number. One abuser could use it
  up and lock out everyone else there: a denial of service nobody chose, and the
  one this tool exists to prevent. Refusing a whole country is left to the
  operator's choice, through the country lists.

### Attack detection

By default the Core Rule Set reads the method, the URL with its query string,
and the headers of every request. That is where automated attacks show:
injection in query strings, path traversal, attacks carried in headers,
scanners' user agents. It reads no request body. In front of a code forge,
bodies carry what people write and publish, such as issue text and files saved
in the web editor. The Core Rule Set takes the shell commands and code in them
for attacks. The default gives up refusing an attack carried in a body. The rule
files, the limits and the bans still apply to the client that sends it.

- The Core Rule Set skips a path under `SWW_WAF_EXEMPT_PATHS`.
- `SWW_WAF_PARANOIA_LEVEL` and `SWW_WAF_ANOMALY_THRESHOLD` are the Core Rule
  Set's own two tuning values, at its own defaults. Each level up runs more
  rules, which find more attacks and take more ordinary requests for attacks.
  Each rule that matches adds its score to the request's, and a rule rated
  critical adds 5. A request whose score reaches the threshold is a match, so by
  default one critical rule is enough. With the threshold `off`, nothing is a
  match, and scores are still logged. A request with more than 1000 query
  parameters or body fields adds 5 to its score, since Coraza reads only the
  first 1000. For a match, in `block` and `detect` mode alike, the log line
  gives the rule ids and the score, and a `waf_block` alert is sent.
- `SWW_WAF_DISABLED_RULES` switches off rule ids when an app trips a false
  positive. The default switches off six rules:
    - 920340, 920420 and 920640 refuse a request for the type of its body, when
      the type is missing or not on the Core Rule Set's short list;
    - 920440 refuses a request for its file extension, such as `.sh` or `.sql`;
    - 930130 and 930140 refuse a file or directory name in the path, such as
      `.git/`, `.gitignore`, `Dockerfile`, `package.json` or an editor's
      settings directory.

    In front of a code forge these refuse git over HTTP, container image and
    package uploads, and views of ordinary files in a repository. At the site
    root, where no app serves such files, the default rule file bans the common
    probes these rules caught (see "Rule files"). A list you give replaces the
    default, so name these six ids in it to keep them off.

- `SWW_CORE_RULE_SET_EXCLUSIONS_FILE` names a file of exclusions, written as
  Coraza directives. An exclusion switches one rule off for some fields or paths
  only, where `SWW_WAF_DISABLED_RULES` switches it off everywhere. The file
  holds an app's own exclusions; gitea's fixes below hold gitea's.
    - `discriminator` loads the file after `smallwebwaf`'s own setup rules and
      before the Core Rule Set's rules. The Core Rule Set loads its own
      `REQUEST-900-EXCLUSION-RULES-BEFORE-CRS.conf` at the same point.
    - So the file holds only `SecAction` and `SecRule` lines that exclude with
      `ctl:ruleRemoveTargetById` or `ctl:ruleRemoveById`. These run with each
      request, before the rules they exclude, those of phase 1 included.
    - A rule that `SWW_WAF_DISABLED_RULES` switches off and the file names is
      simply off.
    - The start stops on a file that cannot be read or does not parse, whatever
      `SWW_WAF_MODE` is.
    - It stops too on a file that names a number from 900000 to 900999, the ids
      of `smallwebwaf`'s own rules, alone or in a range. Comment lines are not
      checked.
    - It stops too on a directive that removes or changes rules as the file
      loads, such as `SecRuleRemoveById`, since none of the Core Rule Set's
      rules is loaded yet.

- `SWW_WAF_BODY_LIMIT` switches body inspection on, up to that size. It takes at
  most `64K`, since `discriminator` gets no more of a body than that (see "The
  answer"). Form data and multipart bodies are read up to the size. JSON and XML
  bodies are read only when no larger than the size, since those cannot be read
  in part. Any other body is not read: the Core Rule Set would read it as form
  data, where binary content such as a git push trips rules written for text. A
  body the Core Rule Set cannot parse adds 5 to the score, and so does a
  multipart body cut by the limit in the middle of a part's header. Body
  inspection suits apps whose forms carry no code.
- `SWW_TRAP_PATHS` names paths the app never serves and only scanners ask for,
  such as `/wp-login.php` in front of an app that is not WordPress. This is the
  short form of a `path` rule with the `ban` action, for deployments that mount
  no rule files. A trap path is matched whole, without the query, against the
  path as sent and once percent-decoded: `/wp-login.php` matches
  `/wp-login.php?x=1` and `/%77p-login.php`, and does not match `/wp-login.php/`
  or `/blog/wp-login.php`.
- `SWW_ERROR_BURST_THRESHOLD` counts a client's refusals per minute of these
  kinds, and of no other:
    - a rule file match;
    - a trap path;
    - a Core Rule Set match.

    More than this breaks a limit (see "Bans"). A client trying one attack after
    another is refused many times a minute; a person rarely more than a few
    times. Answers the app gives are not counted, since they do not tell a
    scanner from an ordinary client. In front of a code forge, registry and git
    clients get `404` and `401` by design, hundreds of times a minute.

`smallwebwaf` changes the Core Rule Set 4.25.0 in six ways, since in front of
gitea it would otherwise refuse ordinary requests. The changes hold in front of
every app, and no setting undoes them:

1. `PUT`, `PATCH` and `DELETE` are allowed methods besides `GET`, `HEAD`, `POST`
   and `OPTIONS`. APIs, container image pushes and package uploads use them.
   Other methods stay refused.
2. The request header `Expect` is allowed, since curl and git send it before
   some large uploads. `Content-Encoding` is allowed on a request whose body the
   Core Rule Set does not read, which by default is every request. git
   compresses a fetch request over 1 KiB and says so in that header. On a body
   the Core Rule Set does read, the header stays refused, since a compressed
   body cannot be inspected. The other headers the Core Rule Set refuses, such
   as `Proxy`, stay refused.
3. The `redirect_uri` parameter is not checked for a URL naming an IP address or
   `localhost` (931100, 934110). Git Credential Manager, git-credential-oauth
   and tea, which gitea registers for OAuth sign-in, ask to be sent back to
   `http://127.0.0.1` on the user's own machine. The server never fetches that
   address.
4. `Referer` is not checked for a Unix command given without arguments (932340),
   for Java starting a process (944110) or for a Java class name (944130).
   `Referer` names the page a request comes from, on everything that page loads
   and on every link followed from it. The Core Rule Set takes some page
   addresses for attacks:
    - a search for `env`;
    - OpenJDK's `Runtime.java`, whose path holds both `runtime` and `java.`;
    - a file named `java.lang.String.md`.

    Every request from such a page would be refused. `Referer` keeps the other
    rules, except after some gitea searches (gitea's fixes). An app that passes
    `Referer` to a shell must check it itself.

5. Responses are not inspected. A raw file from a repository, such as a shell
   script, looks to the response rules like source code leaking from the server.
6. The `from` parameter of a container registry's upload path,
   `/v2/<image>/blobs/uploads/`, is not checked for command names (932260). When
   a push reuses a layer, `from` names the image the layer comes from. An image
   such as `owner/python3-app` would have that push refused.

Each supported app also has its own fixes, which let its ordinary requests
through. An app's fixes hold for the hostnames `SWW_APPS` lists the app for:

- `SWW_APPS` holds entries separated by semicolons. Each is a hostname, a colon,
  then one or more apps separated by commas, for example
  `git.example.com:gitea;code.example.com:gitea`.
- The request's `Host` is matched by exact name, without its port and without
  regard to case. There are no wildcards.
- A hostname not listed gets no app's fixes, so every rule checks it in full.
  Set empty, `SWW_APPS` lists no hostname.
- Unset, every app's fixes hold for every hostname. An app's fixes then weaken
  the check of any other app that serves the same paths or fields.
- Any of these mistakes stops the start:
    - an unknown app;
    - a hostname that is not a DNS name, such as `*.example.com`;
    - a hostname listed twice;
    - an app listed twice for one hostname;
    - an entry without a colon.

`gitea` is the only app supported. Each app's fixes are in one file, such as
`internal/waf/gitea.conf`, and one entry of `apps` in `internal/waf/waf.go`.
gitea's fixes are:

1. Ten query parameters are not checked against the Core Rule Set's lists of
   system files (930120), shell paths (932160) and command names (932260), on
   any path:
    - `path`, `files`, `skip-to` and `sub_path`;
    - `ref`, `sha`, `branch` and `workflow`;
    - `artifactName` and `redirect_to`.

    gitea puts into them file paths, branch and workflow names, and the page to
    return to after signing in. It reads these values as names in a repository
    or in its own records, or as a page of its own site, never as a command. Any
    name on those lists can be an ordinary file or branch, such as `.gitignore`
    or `docker-build`. In these parameters, what only these three rules would
    catch reaches the app, such as the command `|cat /etc/passwd`. Path
    traversal, SQL and script injection, and PHP, Java and Node.js code are
    still refused there. Other parameters keep all three rules, except on the
    gitea paths the third fix names.

2. The Core Rule Set reads the request without the `gitea_flash` and
   `redirect_to` cookies. gitea puts a message naming what was just done into
   `gitea_flash`. It puts into `redirect_to` the page to return to after signing
   in. The Core Rule Set takes file names in them, such as `package.json`, for
   attacks. A browser sends such a cookie with every request until gitea
   replaces it. gitea cannot replace it while those requests are refused, so one
   refusal would keep the browser out of the whole site. gitea shows the message
   with any script removed and returns only to its own pages. Every other cookie
   is read in full.
3. On gitea's paths, some fields skip the rules that refused ordinary gitea
   requests there. They are the fields where gitea sends a search, a file name
   or a branch name. `internal/waf/gitea.conf` lists each rule and the request
   it refused. The fields are:
    - `q` on gitea's searches of code, of issues and pull requests, and of
      commits. A code search holds code, such as `<script>`,
      `SELECT * FROM users WHERE` or `../`.
    - `Referer` of a page opened from such a search, for script and SQL
      injection and Java code (941110, 941120, 941160, 942100, 944100).
    - `q` on gitea's searches for a name, for the names of system files and
      commands (930120, 932120, 932260). Names such as `.profile` and
      `python3-app` were refused. gitea searches for the names of:
        - repositories;
        - users and organisations;
        - branches and tags;
        - packages.

        A user's or an organisation's page is any path of one segment, so on
        every such path `q` skips these three rules, for example on `/search`.

        Some of these searches also skip the rule for Java class names (944130),
        which refused `org.apache.commons.logging`.

    - The path of these requests, which names a file:
        - a file's page;
        - its raw file;
        - its download;
        - its blame page;
        - its history;
        - its edit page;
        - its preview in the editor;
        - its delete page;
        - the API's call for its contents;
        - a release attachment's download.

        Names such as `notes.txt~`, `process.env.d.ts` and `java.lang.String.md`
        were refused.

    - A file name in a query parameter:
        - `file` of a commit graph;
        - `path` of the API's list of commits;
        - `path` of a diff's hidden lines;
        - `body` of the API's call for several files;
        - `template` of a new issue;
        - `redirect_to` of the sign-in page;
        - `name` of an attachment uploaded through the API.
    - A branch or tag name in a query parameter, for command names (932260).
      gitea sends one:
        - to delete or restore a branch;
        - to start a release from a tag;
        - to edit a branch protection rule;
        - to list a directory's last commits.
    - The `Content-Range` header of an artifact upload from
      `actions/upload-artifact@v3` (920450).

    Elsewhere these fields keep every rule. The paths are those of gitea at the
    site root.

On a hostname that gets gitea's fixes, an app other than gitea must check these
fields and cookies itself. Listing each hostname's own apps in `SWW_APPS` keeps
the full check for the others.

The parameter and cookie names in these changes and fixes are matched without
regard to case, as Coraza matches them.

### Ban lengths, response and scope

`SWW_BAN_RESPONSE` is `403` by default, which makes a ban easy to recognise when
debugging; `429` and `close` are the alternatives. Behind traefik, `close` does
not leave the client unanswered: traefik answers `502`, as it does whenever its
backend drops a connection. `SWW_BAN_SCOPE_V4_PREFIX` widens an IPv4 ban to the
surrounding netblock. The four ban lengths are under "Bans".

### Reputation

No source is on by default. A source acts on a list kept elsewhere; the defaults
judge a client only by what it does here. AbuseIPDB and CrowdSec also need an
account or an engine of the operator's own. The list best suited to be on by
default, Spamhaus DROP, has terms, below, that forbid frequent fetching, and
Spamhaus may block an address that breaks them. Each `discriminator` fetches its
own copy, so several of them on one host, sharing one address, would often fetch
too close together.

- Blocklists (`SWW_BLOCKLIST_URLS`) are text files of addresses and netblocks,
  one to a line, as the Spamhaus DROP list is written. Anything after a `;` or
  `#` on a line is ignored. An IPv4-mapped address such as
  `::ffff:192.0.2.0/120` is read as the IPv4 netblock it stands for. Spamhaus's
  terms for DROP, on its DROP page:
    - automated downloads must be at least one hour apart, and once a day is
      more than enough in most cases;
    - a product that uses it must credit The Spamhaus Project and keep the
      list's date and copyright lines with the data.

    So `SWW_BLOCKLIST_REFRESH` is never less than `1h`, and the last good copy
    is kept whole, comment lines included, on failure and across restarts.

- Each list is fetched again `SWW_BLOCKLIST_REFRESH` after its last try, one
  list at a time. `reputation.json` keeps when each was last tried, so a restart
  does not fetch a list again before its time. A fetch fails when:
    - the server answers other than `200`;
    - the fetch does not finish within a minute;
    - the list is longer than 16 MiB;
    - a line is not an address or netblock, or, for the AS number file, not
      `AS<number>:<percent>`.

    The log, the alerts, the ban notes and the metrics name each list by its
    URL, so a list's URL must not hold a secret.

- `SWW_BLOCKLIST_ACTION` is `deny` by default, which refuses every request from
  a listed address and suits lists of networks that send nothing legitimate,
  such as DROP. `limit:<percent>` gives listed clients that percentage of every
  request and byte limit, and suits a list of addresses shared with ordinary
  visitors, such as Tor exits. `log` only notes the hit in the log line and
  raises an alert.
- A CrowdSec engine on the host (`SWW_CROWDSEC_LAPI_URL`) has its decision list
  fetched every minute, with `SWW_CROWDSEC_LAPI_KEY` in the `X-Api-Key` header.
  The list is kept like a fetched blocklist, and a client on it is banned (see
  "Bans"). Only decisions of the type `ban` on an address or a netblock are
  used; any other, such as a captcha or a country, is left out. A decision lasts
  for the `duration` it had left when the engine answered. This is how a
  fleet-wide blocklist can arrive without `discriminator` depending on CrowdSec.
  A fetch fails when:
    - the engine answers other than `200`, a redirect included;
    - the fetch does not finish within a minute;
    - the answer is longer than 16 MiB;
    - the answer is not a list of decisions.
- Each DNS blocklist (`SWW_DNSBL_ZONES`) is asked about a client's own address
  in the background, the first time the client is seen. No request waits for the
  answer. The name asked about is the one RFC 5782 gives: the numbers of an IPv4
  address reversed, or the 32 hex digits of an IPv6 address reversed. So
  `192.0.2.99` is asked about as `99.2.0.192.dnsbl.dronebl.org`. An answer in
  `127.0.0.0/8` lists the client, but one in `127.255.255.0/24` is Spamhaus
  refusing the query. A name that does not exist does not list the client. These
  are failures, after which the zone is not asked again for a minute:
    - an answer in `127.255.255.0/24`;
    - an answer outside `127.0.0.0/8`;
    - a resolver error;
    - no answer within `SWW_REPUTATION_TIMEOUT`.

    DNS blocklists made for mail servers list residential ranges, and would
    block ordinary visitors, so they should not be used. Spamhaus zones need
    their keyed query service, given as a zone name that contains the key.
    `discriminator` shows the key as `********` wherever it names the zone. Only
    `reputation.json` keeps the key. `SWW_DNSBL_RESOLVER` names a resolver,
    since public resolvers are refused by several list operators.

- AbuseIPDB (`SWW_ABUSEIPDB_KEY`) is asked, at its check endpoint, for the abuse
  confidence score of a client, from 0 to 100. Only a client that has already
  committed one offence is checked, so the daily budget
  (`SWW_ABUSEIPDB_DAILY_BUDGET`, under the free tier's 1000 checks a day) is
  spent on suspects. The check runs in the background at the client's next
  request that no ban refuses, so no request waits; the score applies to the
  client's later requests. For an IPv6 client, AbuseIPDB is asked about the
  address of the request that set off the check, and the score is used for the
  whole group. A score at or over `SWW_ABUSEIPDB_MIN_SCORE` is a hit. The budget
  is counted in UTC days and kept in `reputation.json`, so a restart does not
  make it whole again. The check that spends the last of it is logged and
  alerted, and no client is checked until the day ends. The key is sent in the
  `Key` header and nowhere else, and is never logged. A check fails, after which
  no client is checked for a minute, when:
    - AbuseIPDB answers other than `200`;
    - the answer gives no score;
    - no answer comes within `SWW_REPUTATION_TIMEOUT`.
- `SWW_REPUTATION_ACTION`, for DNS blocklist and AbuseIPDB hits, is `limit:25`
  by default. Such verdicts are less certain than a blocklist, so a listed
  client gets a quarter of every request and byte limit rather than a refusal.
- Verdicts and scores are kept for `SWW_REPUTATION_CACHE_TTL`, in memory and in
  `reputation.json`. A source that is down or over quota gives no verdict. The
  service continues. Each failure raises a `source_failure` alert (see
  "Alerting").

### Alerting

- Alerts go to a webhook as JSON, and to Slack and ntfy as formatted messages.
  `docs/alerts.md` lists the events, the JSON object and the messages.
  `SWW_ALERT_EVENTS` chooses which of the seven event types are sent; the hourly
  `summary` is sent whatever it says. The bans an admin makes raise no alert. In
  `observe` mode, a request that would have made a ban still raises the ban's
  alert, with `mode` set to `observe` in its `detail`.
- A repeat is an alert with the same event, and the same netblock, source, file
  or anomaly scope, as one sent less than `SWW_ALERT_COOLDOWN` before. An
  anomaly scope is one of the things counted under "Anomaly thresholds". A
  repeat is held back and counted, and the next such alert sent gives the count.
- Past `SWW_ALERT_MAX_PER_HOUR` alerts in an hour, the rest are held back and
  counted by event, so a wide attack cannot flood the channel. Once the hour
  ends, one alert with the event `summary` gives those counts. It also counts
  the repeats that a cooldown held back and that no later alert reported.
- Each destination has a bounded queue (see "Fixed limits" in
  `docs/settings.md`), sent one at a time, so a slow destination holds up
  nothing. When a queue is full, its oldest alert is dropped and counted in the
  metrics. A `2xx` answer takes an alert. A `4xx` other than `408` and `429`
  refuses it, and it is dropped. Anything else is a failure: the alert is sent
  again a second later, and twice as long after each further failure, up to a
  minute. `alerts.json` keeps the queues and cooldowns across a restart.

### Anomaly thresholds

These only alert; nothing is blocked. None is set by default, and a threshold
left unset sends no alert. An alert needs a destination only the operator can
supply, and what counts as unusual depends on each service's normal traffic,
which the metrics show. They reveal a widely distributed scraper, thousands of
addresses at low rates each, which the per-client limits do not see. A low
`SWW_ASN_LIMIT_PERCENT` for its AS number is the response.

- Requests and bytes are counted per minute and per hour, for each of:
    - a client;
    - the surrounding netblock;
    - an AS number;
    - the whole service;
    - a named netblock in `SWW_WATCH_NETS`.
- Every request `discriminator` answers with `allow`, `deny` or `close` is
  counted, an allowed one when its report comes. That holds whatever was done
  with it, and for clients in `SWW_ALLOW_NETS` and `SWW_RATE_LIMIT_EXEMPT_NETS`,
  since an exempt client misbehaving is worth knowing about. The counters are
  kept in memory and in `alerts.json`.

## Rule files

`discriminator` reads every `*.rules` file in `SWW_RULES_DIR` and checks each
request against them. `docs/rules.md` gives the format, the targets and actions,
and the default file.

- The files are plain text meant to be edited by hand, so that a new scanning
  pattern seen in the request log becomes a rule in one line. `discriminator`
  watches the directory: a file edited, added or removed there takes effect
  while it runs. It reads the files again once the directory has had no change
  for 2 seconds, so that a file still being written is read whole. A file whose
  name starts with `.`, such as an editor's lock file, is not a rule file.
- Files are read in name order (`00-default.rules` before `50-site.rules`),
  rules in line order.
- A rule's regex is Go's RE2 syntax. It has no backreferences or lookaround; in
  exchange matching time is linear in the input, so no rule can be made to stall
  the proxy.
- `path` and `uri` are matched both as received and once percent-decoded, so an
  encoded probe cannot slip past. A header sent more than once is matched with
  its values joined by `, `; a header not sent is empty text. `header:Host` and
  `header:Transfer-Encoding` cannot be targets, since Go's HTTP server takes
  them out of every request; a rule naming one stops the start like any bad
  line. The target `host` matches the request's host. Request bodies are not
  available to rule files; reading them is the Core Rule Set's job.
- `log` only notes the match in the request log. `block` refuses the request;
  `ban` makes the request a clear sign of attack (see "Bans"), and is for
  requests no real visitor sends. A visitor banned by mistake is let back in by
  lifting the ban.
- At start, these stop the process with a message naming the file and line:
    - a line that does not parse;
    - a regex that does not compile;
    - a duplicate id.

    So does a `SWW_RULES_DIR` that does not exist or cannot be read. While
    running, the same faults leave the rules as they were, and the log and one
    `file_error` alert name the file and line. Once the file is fixed, it is
    read again. An empty directory is not an error: the log says that no rules
    were loaded.

- The image ships a default file, kept short and limited to patterns that are
  wrong for every app, with its path rules anchored at the site root. In front
  of a code forge, `/.env` is a probe, while
  `/<owner>/<repo>/src/branch/main/.env.example` is a file in a repository that
  any visitor may open. Anchor the paths of your own `ban` rules at the site
  root too. A rule has one target, so a `path` rule applies on every site behind
  that `discriminator`. A directory mounted over `SWW_RULES_DIR` replaces the
  default file; single files mounted into it add to it.
- Rules meant to be edited while `discriminator` runs belong in a mounted
  directory, not a mounted file. Many editors save a file by replacing it, and a
  container keeps seeing the old one when the file alone is mounted. The lookup
  database is mounted the same way, for the same reason. To run without rule
  files, mount an empty directory or set `SWW_RULES_ENABLED=false`.

## Persistent state

All state lives in memory, and the files in `SWW_STATE_DIR` hold a copy of it,
so an orderly stop loses nothing. The files are meant for people as well: an
admin can read or edit them at any time, and the running `discriminator` takes
the edit in. `docs/state-files.md` lists the files, what each holds, and their
sizes.

- Each file holds one kind of state, as JSON with:
    - a top-level `version` number;
    - entries sorted by client address;
    - times in RFC 3339 UTC;
    - durations and sizes as plain numbers, with the unit in the field name.
- `bans.json`, `reputation.json` and `alerts.json` are indented, so that a
  person can open `bans.json` in an editor, find an address, and remove or add
  an entry. `clients.json` and `lookups.json` put each entry on one line, which
  halves their size and lets `grep` show everything about one client.
- Clients that were never banned are kept too, so every client's history
  survives a restart.
- Each file has a bound, so it can be rewritten whole. Writing one takes well
  under a second, in the background, so rewriting whole files needs nothing
  cleverer.
- A write serialises from a snapshot taken under the lock, so requests are not
  held up. It goes to a temporary file in the same directory, which is synced
  and renamed over the real name, and the directory is synced. A crash at any
  point leaves either the old complete file or the new complete file, never a
  partial one. Before writing a file, `discriminator` takes in any edit made to
  it on disk, so an admin's edit is never overwritten.
- At start, the files are read back. Active bans go into an in-memory lookup by
  netblock, everything else into maps. Expired cache entries are discarded. A
  missing file means empty state, and is normal on first run. These stop the
  process with a message naming the file and position:
    - a file that does not parse, or has an unknown `version`;
    - an entry without a field it needs;
    - a ban whose cause is not one of the four;
    - a ban whose `expires` is not after its `start`.

    Starting with empty state would silently forgive every repeat offender.
    Since writes are atomic, a broken file can only come from a hand edit, which
    the editor should hear about.

- While running, `discriminator` notices a file that is edited, replaced or
  added. It tells its own writes from an admin's by comparing the file with what
  it last read or wrote. An edit that parses is taken in at once: what the file
  says replaces what `discriminator` held for that file. An edit to `bans.json`
  makes, lifts or forgets bans as "Bans" describes. A change `discriminator`
  made after the admin opened the file, such as a new ban, is lost when the
  admin saves over it. Most editors warn when a file changed on disk while it
  was open. A file that is removed is written again at its next write.
- An edit that does not parse never stops the running `discriminator`. It keeps
  the state it has and sets the edited file aside as `<name>.bad`, so the edit
  is kept. It writes the file again from memory, and names the file and the
  position of the error in the log and in one `file_error` alert. The admin
  mends the `.bad` file and moves it back.
- A write that fails leaves the state correct in memory. It is logged, counted
  in the metrics and sent as one `file_error` alert per cooldown, and retried at
  the next write.
- A hard kill loses the changes since the last write: ban changes from the last
  `SWW_STATE_WRITE_DELAY`, everything else from the last
  `SWW_STATE_COUNTER_INTERVAL`. Losing the volume loses ban history, not
  service.

## Request log

`rproxy` and `discriminator` each write request lines on stdout, one JSON object
to a line, refused requests included. "The log" says which lines a request gets,
and `docs/request-log.md` lists the fields. Bodies are never logged, and no
header but those listed there. Query strings are logged as received; an app that
carries secrets in query strings needs that fixed in the app. Each process's own
messages (start, the settings, stop, errors) share its stream as JSON lines with
`"type":"process"`; request lines carry `"type":"request"`.

When `SWW_LOG_REMOTE_URL` is set, every line `discriminator` writes on stdout is
also sent there as an RFC 5424 syslog record. So a deployment can stop depending
on docker's log handling while `docker logs` keeps working. `rproxy` writes its
lines to stdout only.

- Over UDP each record is one datagram; a line too long for one datagram is
  dropped alone. Over TCP and TLS each record follows its length in bytes and a
  space. UDP gives no delivery signal at all and is offered only for
  compatibility.
- The record carries the facility from `SWW_LOG_REMOTE_FACILITY`, the severity
  informational, the time, the host's name and the app name from
  `SWW_LOG_REMOTE_APP_NAME`.
- Sending happens on its own goroutine from a buffer of `SWW_LOG_REMOTE_BUFFER`
  lines, so a server that is slow or down never holds up a request and never
  stops stdout. When the buffer is full, the oldest line is dropped and counted
  in the metrics. A failed connection is tried again a second later, and twice
  as long after each further failure, up to a minute.
- As `discriminator` stops, it spends at most two seconds sending the lines
  still waiting, and drops the rest; stdout has carried them.

## Metrics endpoint

`GET /_sww/metrics`, at `discriminator`'s own address, answers in the Prometheus
text format, for a request carrying `SWW_METRICS_TOKEN`. `docs/metrics.md` lists
the metrics. No metric carries a client IP address as a label. Per-address
questions are answered by the request log and `GET /_sww/clients/<ip>`.

## Admin endpoints

`discriminator`'s own endpoints are paths under `/_sww/`, at its own address on
the private network. Admins and the metrics collector reach them there, and
nothing routes the internet to that address. The prefix carries the tool's name,
so it takes no path an app uses. `docs/admin-endpoints.md` lists the endpoints,
what each takes and what each answers.

- `GET /_sww/healthz` answers `200` with `ok` to anyone, without a token, while
  `discriminator` is running.
- `POST /_sww/report` takes `rproxy`'s report, without a token (see "The
  report").
- Every other endpoint needs its token, sent as `Authorization: Bearer <token>`
  (see "Tokens" under "Settings"). A missing or wrong token is answered `401`.
  While a token is unset, the endpoints that need it answer `404`, as does any
  other path.
- No check runs on these requests. They count toward no limit, and no client's
  history holds them.
- A ban made through the endpoints starts at once.

`rproxy` answers `GET /_sww/healthz` itself. It never passes a path under the
prefix to the app (see "Without an answer from `discriminator`").

## `discriminator`

`sww discriminator` runs the checks for `sww rproxy`, a small reverse proxy in
front of each app. `rproxy` asks `discriminator` about each visitor's request.
`discriminator` answers `rproxy` and passes nothing to an app itself. One
`discriminator` serves many sites.

This section is the agreement between the two. The header names, the fields and
the statuses below are exact.

`discriminator` reads the settings, the rule files, the lookup database and the
state files, and listens at `SWW_LISTEN_ADDR`. On `SIGTERM` or `SIGINT` it stops
taking requests and gives those in progress five seconds to finish. It then
writes the state files.

### The question

A request carrying `X-Sww-Client` is `rproxy` asking about one visitor's
request. It is the visitor's request as the visitor sent it:

- the method;
- the path and query as sent;
- `Host`;
- every header the visitor sent, including its own `X-Forwarded-For`;
- the first 64 KiB of the body, or all of it when shorter.

Go's HTTP code changes some of the visitor's headers, such as:

- Go's client does not send an empty `User-Agent`.
- Go's server adds `Cache-Control: no-cache` to a request with
  `Pragma: no-cache` and no `Cache-Control`, before `rproxy` sees it.

`rproxy` itself refuses any request it cannot send this way (see "Requests
`rproxy` refuses itself"). So a question always has:

- a path starting with one `/`;
- a `Host` that is not empty and holds no IPv6 zone;
- at most one `User-Agent`.

A body whose length the visitor did not announce comes chunked. Any other body
comes with a `Content-Length` of the part it carries.

`rproxy` sets these headers, replacing any the visitor sent, and adds no others:

- `X-Sww-Client`: the visitor's address, which `rproxy` has worked out, without
  any IPv6 zone. `discriminator` takes it as the client, and groups an IPv6
  address as "What happens to a request" says.
- `X-Sww-Peer`: the address of `rproxy`'s own TCP peer, for the log line's
  `peer_ip`.
- `X-Sww-Content-Length`: the body length the visitor announced. It is absent
  when the visitor announced none.
- `X-Request-ID`: the request's id, which `rproxy` makes fresh for every request
  when it arrives. `discriminator` takes it as given and never makes one.
- `X-Forwarded-Proto`: the scheme.

Before any check, `discriminator` removes the three `X-Sww-` headers. It also
sets `Content-Length` to the length the visitor announced, or removes it. So the
rule files and the Core Rule Set see the headers the visitor sent, with
`rproxy`'s `X-Request-ID` and `X-Forwarded-Proto`.

`discriminator` refuses a question with `400` when:

- `X-Sww-Client` is not an address, or is an address with a zone, such as
  `2001:db8::1%eth0`;
- `X-Request-ID` is absent, empty, over 128 bytes or not valid UTF-8;
- `X-Request-ID` is the id of a question `discriminator` is still handling, or
  of a request it holds (see "The report");
- the body is over 64 KiB.

For a refused question, `discriminator` writes a process warning. It runs no
check, writes no request log line and counts nothing in the metrics.

These do not refuse a question:

- `X-Sww-Peer` absent or not an address: the log line's `peer_ip` is
  `invalid IP`.
- `X-Sww-Content-Length` not a whole number, or negative: it counts as absent.
- `X-Forwarded-Proto` absent: the log line's `scheme` is `http`.
- One of the five headers above sent more than once: `discriminator` uses its
  first value.

`discriminator` reads the whole body before any check. A question whose body
stops arriving gets no answer. `discriminator` closes the connection once the 60
seconds below run out, or as soon as `rproxy` closes its side. It writes a
process warning, runs no check, writes no request log line and counts nothing in
the metrics.

### The answer

`discriminator` runs steps 3 to 13 of "What happens to a request", in that
order. Every answer but the last below carries `X-Sww-Disposition`.

- `X-Sww-Disposition: allow`, status `200`, no body. `rproxy` passes the request
  to the app. With `SWW_ADD_LOOKUP_HEADERS` the answer also carries
  `X-Client-ASN` and `X-Client-Country`, which `rproxy` adds to the request it
  passes on. A header whose value is unknown is left out.
- `X-Sww-Disposition: deny`, with the status, headers and body the visitor is to
  get. `rproxy` gives them to the visitor, less this header. The status is one
  of:
    - `SWW_BAN_RESPONSE`, for a refusal;
    - `403`, for a `block` rule or the Core Rule Set;
    - `431`, for a request over the header limits;
    - `404`, for a path under `/_sww/` that passes the checks, so that the app
      never sees that prefix.
- `X-Sww-Disposition: close`, status `200`, no body. This takes the place of a
  refusal while `SWW_BAN_RESPONSE` is `close`. `rproxy` closes the visitor's
  connection without an answer.
- No such header, status `400`, for a question `discriminator` refuses. `rproxy`
  gives the visitor its status, headers and body, as it does for any answer
  without a valid `X-Sww-Disposition`.

In `observe` mode a request that `enforce` mode would refuse is answered
`allow`, and its log line has `would_action`. A path under `/_sww/` is still
answered `404`.

The Core Rule Set inspects the body a question carries only while
`SWW_WAF_BODY_LIMIT` is set. It inspects at most 64 KiB less one byte. So it
does not inspect a JSON or XML body of 64 KiB or more, which may have been cut
short.

### Answering in time

`rproxy` stops waiting for `discriminator` after `SWRP_TIMEOUT_MS`, by default 2
seconds, and `SWRP_DEFAULT_DISPOSITION` then decides. So at the defaults,
`discriminator` answers a request well within that time, whatever the request
and whatever its outside sources do.

GeoJS is the only outside source a question can wait on. A question waits only
when step 5 says so, and for at most `SWW_LOOKUP_TIMEOUT`, by default 800
milliseconds. A client without an answer by then counts as unknown. No question
waits on these, which run in the background:

- the DNS blocklists and AbuseIPDB;
- fetching the blocklists, the CrowdSec decision list and the file
  `SWW_ASN_LIMIT_PERCENT_URL` names.

In `enforce` mode, a question does not wait for GeoJS when it is a request:

- over the header limits;
- from a client on a fetched blocklist, while `SWW_BLOCKLIST_ACTION` is `deny`;
- from a client on the CrowdSec decision list;
- that would take its client over a rate limit at 100% of it, since a percentage
  can only lower a limit.

Such a request still goes through the steps in their order. Its client's AS
number and country stay unknown until GeoJS has answered. So while
`SWW_EXCLUSIVELY_ALLOWED_COUNTRIES` is set, that list can refuse it first.
`SWW_UNKNOWN_LIMIT_PERCENT` does not apply to it until then, so not waiting
never lowers its limit.

The static lists and bans come before the lookup, so they never wait. These can
still wait for GeoJS:

- a country list, which needs the answer;
- a rate limit a percentage has lowered, which needs the answer;
- a DNS blocklist or AbuseIPDB, while `SWW_REPUTATION_ACTION` is `deny`;
- a trap path;
- a rule file;
- the Core Rule Set.

The header limits are fixed, with no setting. `discriminator` refuses a request
over either one:

- more than 50 cookies. Each part of its `Cookie` headers between semicolons
  counts as one, unless it is empty or only spaces and tabs.
- more than 100 header fields. A header sent twice counts twice. `Host` and
  `Transfer-Encoding` do not count. `X-Request-ID` and `X-Forwarded-Proto` do.

Real clients never reach these limits. The answer is `deny` with `431`, and the
log line's action is `too_large`. The refusal makes no ban and is no offence.
The rate limits count the request as usual, so a client that keeps sending such
requests breaks a limit.

The limits apply on every path and in every `SWW_WAF_MODE`. A client in
`SWW_ALLOW_NETS` skips them, as it skips the Core Rule Set. In `observe` mode
such a request is answered `allow` with `would_action`, and the Core Rule Set
does not read it.

The Core Rule Set runs each of its rules on each cookie, so thousands of small
cookies would keep it busy for over a second. Within the limits, one long cookie
or query string can still keep it busy for a fraction of a second. So enough
such requests at once can overload `discriminator` past `SWRP_TIMEOUT_MS`.

### The report

Once the visitor's answer has ended, `rproxy` sends `POST /_sww/report` for each
request `discriminator` answered with `allow`, `deny` or `close`. It sends none
after:

- an answer without a valid `X-Sww-Disposition`, such as a `400`;
- an answer it cannot pass on (see "Without an answer from `discriminator`");
- no answer.

The report carries no `X-Sww-Client`. Its body is one JSON object with all of
these fields, named as the request log's fields of the same meaning:

- `request_id`: the question's `X-Request-ID`;
- `status`: what the visitor got, `0` if nothing, otherwise from `100` to `599`;
- `request_bytes` and `response_bytes`: the body bytes each way, with what a
  WebSocket carried each way, each a whole number from `0` to `1125899906842624`
  (1 PiB);
- `duration_total`: milliseconds from the request's headers being read to the
  end of the answer, from `0` to `1000000000000` (about 31 years);
- `duration_upstream_total`: milliseconds from the hand-over to the app to the
  end of the answer, in the same range, and `0` for a request not passed to the
  app.

`discriminator` answers `400` and writes a process warning when the body:

- is not one JSON object, such as `null`, or an object followed by other data;
- lacks one of these fields, or gives it as `null`;
- gives a value of the wrong type, or out of its range;
- is over 1 MiB.

It answers `204` to any other report, also for a request it does not hold, which
it ignores. It ignores a field not listed here. `rproxy` does not act on the
answer.

For a request it holds, `discriminator` writes the request's last log line (see
"The log"). For a request it allowed, it first:

- counts the bytes as `SWW_BYTES_COUNT` says, toward the client's byte limits at
  the percentages it gave that request;
- bans a client that passes a byte limit;
- adds the request with its status to the client's history, the anomaly counters
  and the metrics.

A denied request's counting is finished at its decision, so its report counts
nothing.

`discriminator` holds each answered request until its report comes, up to 10,000
requests. Past that, the request answered longest ago is dropped. A request
whose report never comes gets no last log line. If it was allowed, it is in
neither the client's history nor the metrics, and its bytes are not counted.

`discriminator` holds only the decision's log line and what counting needs. It
never holds the request or response objects, and no `Authorization` or `Cookie`.
A report reaches only what is held, so it can never reach a question still being
handled. `discriminator` ignores a report naming such a question.

### The log

A request `discriminator` answers with `allow`, `deny` or `close` gets three
request log lines, joined by `request_id`:

- `rproxy`'s short line;
- `discriminator`'s line at its decision;
- `discriminator`'s line when the report comes, with everything known: the
  decision line's fields and the report's together.

The repetition is intended. The field `discriminator_line` says which of
`discriminator`'s two lines a line is. A request `discriminator` is not asked
about, such as the health check, gets only `rproxy`'s line. A request without a
report gets no third line. `docs/request-log.md` gives the fields.

### `discriminator`'s own endpoints

A request without `X-Sww-Client` is for `discriminator` itself, at its own
address on the private network. The internet must not be able to reach that
address. Its endpoints are:

- the health check, the metrics, and the ban and client endpoints, as
  `docs/admin-endpoints.md` lists them, with their tokens;
- the report.

Anything else gets `404`. No check runs on these requests, since their peers are
`rproxy` and the admin's tools. The report needs no token, so anything on the
private network can send one or ask a question.

A request and its answer have 60 seconds together, from the request's headers.
`discriminator` closes a connection idle for 120 seconds. It takes a request
line and headers up to 1 MiB, Go's default, so that it never refuses a question
for the size of a visitor's headers.

### Sites

One `discriminator` keeps one table of clients, counters and bans for every site
that asks it. A client banned through one site is refused on all of them, and
its requests on all of them count toward one set of limits. Log lines, alerts
raised by a request and ban notes name the site by its host.

`docs/metrics.md` says how its metrics count.

## `rproxy`

`sww rproxy` is the reverse proxy in front of one app. It asks `discriminator`
about each request, and passes on what `discriminator` allows. It keeps nothing
between requests and never touches disk. When `discriminator` cannot be
connected to, does not answer at all, or does not answer within
`SWRP_TIMEOUT_MS`, `SWRP_DEFAULT_DISPOSITION` decides. Its default, `allow`,
keeps the site answering.

`rproxy` reads only its `SWRP_` settings, which `docs/settings.md` lists. One of
them is required: `SWRP_WAF_BACKEND`, `discriminator`'s URL. It has no metrics,
no rule files and no state files, and it writes its lines to stdout only. The
image runs it by default (see "Deployment").

### What `rproxy` does with a request

1. `GET /_sww/healthz` is answered `ok` at once, without asking `discriminator`.
2. `rproxy` works out the visitor's address. It believes `X-Forwarded-For` only
   from `SWRP_TRUSTED_PROXIES`, and reads it as step 2 of "What happens to a
   request" says. It removes any IPv6 zone.
3. It makes a fresh request id, whatever id the request came with.
   `discriminator` and the app get it as `X-Request-ID`, and `rproxy`'s log line
   gives it.
4. It refuses a request with a method it does not pass on, or one it cannot put
   to `discriminator` as the visitor sent it (see "Requests `rproxy` refuses
   itself").
5. It reads up to the first 64 KiB of the body into memory.
6. It sends `discriminator` the question (see "The question"). It speaks
   HTTP/1.1 to `discriminator`, whatever `discriminator` offers. Go's HTTP/2
   client would refuse to send a visitor's `Upgrade` header, and most
   `Connection` headers. The question never carries the visitor's `Trailer`.
7. It does what the answer says (see "The answer"):
    - `allow`: it passes the request to the app.
    - `close`: it closes the visitor's connection without an answer.
    - Any other answer: the visitor gets its status, headers and body, less
      `X-Sww-Disposition`. That holds for a `deny`, for an error such as
      `discriminator`'s `400` or a `5xx`, and for an answer without a valid
      `X-Sww-Disposition`.
8. Once the answer has ended, it writes its log line. `docs/request-log.md`
   gives the fields. An app's answer can end just after the line: Go's server
   sends the closing chunk of a chunked answer, and may send all of an empty
   one, only once `rproxy` has finished with the request.
9. It then sends the report, if `discriminator` answered `allow`, `deny` or
   `close` (see "The report"). The visitor does not wait for it. A report that
   fails is dropped, with a process warning. A status above `599` from the app
   goes to the visitor as it is, and into the report as `599`, the highest
   `discriminator` takes.

### Requests `rproxy` refuses itself

`rproxy` passes on only these methods:

- `GET`
- `HEAD`
- `POST`
- `OPTIONS`
- `PUT`
- `DELETE`
- `PATCH`
- `QUERY`

It answers any other method `405`, with no question and no report. The answer's
`Allow` header names these methods.

`rproxy` answers `400`, with no question and no report, to a request with:

- a target that does not start with `/`, such as `x:admin`,
  `http://site.example/` or `site.example:443`;
- a target that starts with `//`;
- no `Host`, or an empty one;
- a `Host` holding an IPv6 zone: one that starts with `[` and holds a `%`, such
  as `[fe80::1%25eth0]:80`;
- more than one `User-Agent`.

Go's HTTP client could not send any of these as the visitor sent it.

Go's HTTP server answers `OPTIONS *` itself, before `rproxy` sees it.

`rproxy` closes the connection without an answer when it cannot read the first
64 KiB of the body. That happens for a malformed chunk, and for a visitor that
goes away or stops its sending side. There is no question and no report, and the
log line has status `0`.

### Without an answer from `discriminator`

`SWRP_DEFAULT_DISPOSITION` decides what is done with the request only when
`discriminator`:

- cannot be connected to;
- does not answer at all;
- does not answer within `SWRP_TIMEOUT_MS`, which covers connecting, sending the
  question and reading the whole answer.

An answer whose body breaks off counts as no answer at all.

`allow`, the default, passes the request to the app. `deny` answers `503`.
Either way `rproxy` writes a process warning, its log line has
`default_disposition`, and it sends no report.

A visitor can make `discriminator` slow. A request that keeps `discriminator`
busy past `SWRP_TIMEOUT_MS`, like any request while `discriminator` is
overloaded, gets `SWRP_DEFAULT_DISPOSITION`. With `deny`, such a request gets
`503` instead of reaching the app.

A path under `/_sww/` that `allow` would pass on is answered `404` instead, so
that the app never sees that prefix.

`rproxy` cannot pass on an answer whose body is over 1 MiB or whose status is
below `200`. The visitor gets `502` from `rproxy`, never the default. `rproxy`
writes a process warning, its log line has no `disposition`, and it sends no
report.

A visitor that goes away while `discriminator` is being asked is no failure of
`discriminator`. `rproxy` closes the connection, with no warning, no default and
no report. Its log line has status `0` and no `disposition`.

### Passing a request to the app

`rproxy` sends the app at `SWRP_UPSTREAM_URL` the visitor's request: the part of
the body it holds, then the rest, streaming. The app's answer comes back the
same way. Apart from what Go's HTTP code changes, the request reaches the app
unchanged but for these headers:

- `X-Forwarded-For`, with the address of `rproxy`'s TCP peer added, when that
  peer is inside `SWRP_TRUSTED_PROXIES`. The peer's other forwarded headers pass
  on.
- For a peer outside `SWRP_TRUSTED_PROXIES`, its forwarded headers are its own
  claims and are replaced. `X-Forwarded-For` names the peer, `X-Forwarded-Host`
  the host it asked for, and `X-Forwarded-Proto` plain `http`. The other headers
  in which traefik tells an app about a visitor are removed.
- `X-Request-ID`, which carries the fresh id.
- `X-Client-ASN` and `X-Client-Country`, which carry what `discriminator`
  answered. A header it did not give is left out. Any of the two the visitor
  sent is removed.
- The headers that concern only the connection to `rproxy` never reach the app:
    - `Connection`, and any header it names;
    - `Keep-Alive`;
    - `Proxy-Authenticate`;
    - `Proxy-Authorization`;
    - `Proxy-Connection`;
    - `Te`, though one holding `trailers` goes on as `Te: trailers`;
    - `Upgrade`.

A chunked body's trailers reach the app, but not `discriminator`.

A WebSocket upgrade passes through, with `Connection: Upgrade` and its
`Upgrade`. The report of a WebSocket comes when it closes, with what it carried
each way.

### `rproxy`'s limits

These limits apply per request, in both directions:

- `SWRP_MAX_REQUEST_SIZE` is the largest request body.
- `SWRP_MAX_RESPONSE_SIZE` is the largest response body.
- `SWRP_MAX_REQUEST_TIME_MS` bounds the request line and headers. From their
  end, it then bounds the time until the app has been sent the whole request.
  The wait for `discriminator` counts toward it.
- `SWRP_MAX_RESPONSE_TIME_MS` runs from the end of the request to the last byte
  of the answer reaching the visitor. The request ends when the app has been
  sent all of it, or when the app's answer arrives if that is sooner. An answer
  `rproxy` gives itself has the whole time from when it starts.

The largest request line and headers is 32 KiB. A kept-open connection is closed
after 120 seconds idle. Neither has a setting. traefik, by default, closes an
unused connection after 90 seconds. So traefik closes first, and never sends a
request on a connection `rproxy` is closing.

What the visitor gets when a limit is passed:

- Go's HTTP server reads a request's line and headers before `rproxy` sees the
  request. It answers `431` to headers over 32 KiB. A visitor too slow to send
  them gets an error answer, mostly `400`, or none. Either way the server closes
  the connection, nothing reaches the app, and the request log gets no line.
- A visitor too slow to send the first 64 KiB of its body is answered `408`,
  with no question and no report.
- `rproxy` applies the request size limit only after `discriminator` has allowed
  the request. So a request `discriminator` denies gets `discriminator`'s
  answer, whatever its size. A request `discriminator` allows is answered `413`
  when its body is over the limit: at once for a length it announced, otherwise
  when the limit is reached. If part of the body has gone to the app by then,
  the request to the app is cut off.
- A limit passed after that, but before the response has started, is answered by
  `rproxy`:
    - `408` for a visitor too slow to send the rest of its body;
    - `502` for a response that is too large, or an app that cannot be reached;
    - `504` for an app that is too slow.
- A request body streams through, so each side can hold up the other.
  `SWRP_MAX_REQUEST_TIME_MS` can run out while the body is on its way. The
  answer is then `408` if `rproxy` was waiting for the visitor, and `504` if it
  was waiting for the app.
- Once the response has started it can only be cut off, and the connection is
  closed.
- A WebSocket connection leaves these limits behind once it is upgraded. It
  stays open until either side closes it.

### Starting and stopping

An invalid setting, or an address it cannot listen on, stops the start with a
process line. The process lines give the host's name as `instance`, and only
those at `info` or more severe are written.

On `SIGTERM` or `SIGINT`, `rproxy` stops taking requests and gives those in
progress five seconds to finish. It then waits for the reports on their way,
each for up to `SWRP_TIMEOUT_MS`. A report that comes due after that is dropped,
such as that of a WebSocket that closes later.

## Deployment

One image runs either command. `SWW_COMMAND`, in a container's environment,
chooses: `discriminator` runs `discriminator`, and with it unset the image runs
`rproxy`. `docs/image.md` lists what the image holds and how each part is
pinned.

### The usual setup

The usual setup is three containers on one private network:

- the app;
- `rproxy`, which traefik routes to, with `SWRP_UPSTREAM_URL` naming the app and
  `SWRP_WAF_BACKEND` naming `discriminator`;
- `discriminator`, with `SWW_COMMAND=discriminator` and a volume for its state
  files.

Each further app adds its own container and its own `rproxy`, whose
`SWRP_WAF_BACKEND` names the same `discriminator` (see "Sites"). Anything on
that network can reach an app around `rproxy`, and can ask `discriminator`
questions or send it reports. So the network holds only what you trust.

### `rproxy` in the app's container

An app's Dockerfile can build `FROM` the `smallwebwaf` image instead, so that
`rproxy` runs beside the app in one container. The app is then hardened by
changing its base image, and can add packages from nixpkgs. `discriminator`
still runs in a container of its own, and the app's container sets
`SWRP_WAF_BACKEND`.

Beyond its `FROM` line, the app's Dockerfile adds:

- its binary;
- any packages it needs, from nixpkgs, with `nix-env -iA nixpkgs.<name>`;
- its runit service: a directory under `/etc/service` named after the app, never
  `sww`. Its `run` script starts the app with `chpst`, listening where
  `SWRP_UPSTREAM_URL` points (see "Ports"). The app runs as a user of its own,
  which the Dockerfile creates with `useradd`. That user is neither root nor
  `sww`, so that the app cannot touch the process of `rproxy`.

It sets no `ENTRYPOINT` or `USER` of its own. The container must start
`runsvinit`, as root, so that it can start each service as its own user.
"Getting started" in `README.md` shows such a Dockerfile and `run` script.

### The image

- The image is Ubuntu's newest long-term support release, pinned by digest, and
  moves to the next LTS release when that ships. Its packages come from Ubuntu's
  archive, whose packages change with every update.
- The Ubuntu image has no CA certificates. Without them Nix cannot download
  packages and `discriminator` cannot reach GeoJS, so `ca-certificates` is
  installed by name. apt fetches the packages over plain HTTP and checks each
  one against the archive's signed package lists, so the install needs no CA
  certificates.
- nixpkgs comes from a release file on `releases.nixos.org`, named with its
  hash, not from a GitHub archive, whose bytes can change. Root uses Nix
  directly; no Nix daemon runs. `nix-bin` creates no build users, so
  `/etc/nix/nix.conf` sets `build-users-group =`, which lets root build without
  them. Root's Nix profile is last on the `PATH`, so that no package hides the
  image's own commands; busybox's `sv`, for one, looks for services elsewhere.
  An app built on the image gets the same packages each time. Security fixes
  from Ubuntu and nixpkgs come with a newer `smallwebwaf` image, which an app
  takes by changing the digest in its `FROM` line. The image moves to each new
  nixpkgs release, every six months.
- runit supervises a container's processes, with `runsvinit` as the entrypoint,
  as the owner's code style guide asks for service containers. No package
  carries `runsvinit`, so the image builds it from its archived source,
  unchanged since 2015, at a fixed commit hash, with a `go.mod` the build
  writes. `runsvinit` starts runit's `runsvdir`, which runs the `run` script of
  each directory under `/etc/service`. Ubuntu's `sv` looks for services there
  too.
- The image's `run` script is `/etc/service/sww/run`. It starts `discriminator`
  when `SWW_COMMAND` is `discriminator`, and `rproxy` when it is `rproxy`, unset
  or empty. It runs either as the `sww` user (uid and gid 65532), with runit's
  `chpst`. For `discriminator`, it first makes `SWW_STATE_DIR` and every file in
  it belong to that user. For `rproxy` it changes no owner. With any other
  value, it starts nothing, and stops with a message naming `SWW_COMMAND` and
  its value.

Every `run` script, the app's included, is a bash script. It starts with
`#!/usr/bin/env bash` and `set -euo pipefail`, and puts its code in a `main`
function, called on its last line, as the owner's code style guide asks. Ubuntu
ships bash. Each waits one second (`sleep 1`) before starting its program, so a
process that stops at start is started again every second, not in a tight loop.

### Each container's processes

- `runsvinit`, `runsvdir` and each service's `runsv` run as root. Each `run`
  script starts as root and hands over to its program's user with `chpst`. Every
  service starts with the container's environment variables and writes to the
  container's output, which `docker logs` shows.
- When a service exits, runit runs its `run` script again, and the container
  does not stop. Beside an app, the other process keeps running.
- While `rproxy` is down, traefik answers `502`. While the app is down, `rproxy`
  answers `502`, which is not a client offence. While `discriminator` is down,
  `SWRP_DEFAULT_DISPOSITION` decides (see "Without an answer from
  `discriminator`").
- Only faults found at start stop `discriminator`:
    - invalid configuration;
    - a lookup database that is missing or unreadable;
    - a rule file or state file that does not parse;
    - an unwritable `SWW_STATE_DIR`;
    - a `SWW_RULES_DIR` that does not exist or cannot be read;
    - a `SWW_LISTEN_ADDR` it cannot listen on.
- Only an invalid setting, or a `SWRP_LISTEN_ADDR` it cannot listen on, stops
  `rproxy`.
- A process that stops at start logs its message at each try until the cause is
  fixed; its container's health check fails meanwhile. Once running, a failing
  GeoJS, reputation source or alert destination, or a broken edit, never takes
  `discriminator` down.
- `docker stop` sends `SIGTERM` to `runsvinit`, which has runit stop every
  service at once, each with `SIGTERM`, and exits once they have stopped.
  `rproxy` and `discriminator` then stop as "`rproxy`" and "`discriminator`"
  say.
- With `rproxy` beside the app, an app that exits at once on `SIGTERM` ends the
  requests in progress: the visitor gets `502`, or a response cut short. To let
  them finish, the app must keep answering for five seconds after `SIGTERM`.
- The container's root filesystem stays writable: runit writes each service's
  status into its directory under `/etc/service`.

### The health check

The image's `HEALTHCHECK` runs `sww healthcheck`. `SWW_COMMAND` says what it
checks:

- With `SWW_COMMAND=discriminator`, it passes while `discriminator` answers
  `GET /_sww/healthz` on `127.0.0.1`, at the port in `SWW_LISTEN_ADDR`.
- With `SWW_COMMAND` unset, empty or `rproxy`, it passes while `rproxy` answers
  that request on `127.0.0.1`, at the port in `SWRP_LISTEN_ADDR`, and the app
  accepts connections at `SWRP_UPSTREAM_URL`. It does not ask `discriminator`,
  so that the site stays in traefik's routes while `discriminator` is down.
- With any other value, it fails with a message naming `SWW_COMMAND` and its
  value.

traefik sends a container no requests until it shows as healthy. So the check
runs every second from the container's start until it first passes, for up to a
minute, and every 30 seconds after that. An app with a health check of its own
can replace the image's `HEALTHCHECK` with one that checks both.

### Ports

`rproxy` listens on no port but the one in `SWRP_LISTEN_ADDR`, 8080 by default,
which traefik routes to. It passes requests to the app at `SWRP_UPSTREAM_URL`,
`http://127.0.0.1:8081` by default. Another port may be set: the health check
follows the setting, and traefik's port label
(`traefik.http.services.<name>.loadbalancer.server.port`) must name it. An app
beside `rproxy` must leave that port free.

`discriminator` listens on no port but the one in `SWW_LISTEN_ADDR`, 8080 by
default. The questions, the reports and its own endpoints all come there.

The address part of either setting stays empty (`:9000`, never
`127.0.0.1:9000`). That way the others reach the process on the container's
address, and the health check reaches it on `127.0.0.1`.

An app beside `rproxy` listens on that loopback address only. An app that
listens on every address can be reached around `rproxy` by anything that reaches
the container.

### State

`discriminator` keeps its state files in `SWW_STATE_DIR`, belonging to the `sww`
user. Without a volume there, the files survive a restart of the container and
are lost when a deploy replaces it. A volume mounted at `/var/lib/sww`, named or
a host directory, keeps them across deploys. A host directory mounted there
needs no change of owner, and files an earlier owner left in it can be read and
replaced. The volume holds a few tens of MiB at most with the defaults, and
needs no backup beyond the host's own. The image declares no volume, since every
app image built on it would inherit it. `rproxy` keeps nothing, and needs no
volume.

### Tokens

A token given as a file (`SWW_ADMIN_TOKEN_FILE`, `SWW_METRICS_TOKEN_FILE`) stays
out of the container's environment. The operator makes the file on the host,
owned by uid 65532, the `sww` user, with mode `0400`, and mounts the directory
that holds it into `discriminator`'s container read-only. The container sees the
same owner and mode. "Settings given as files" in `docs/settings.md` shows the
commands.

### Forwarded headers

The app's TCP peer is `rproxy`, and the `X-Forwarded-For` the app receives ends
with traefik's address, which `rproxy` adds after the visitor's. So the app must
trust `rproxy`'s address for forwarded headers, besides the private address
ranges traefik is on.

- In a container of its own, `rproxy`'s address is on the private network. The
  usual default for a trusted-proxy setting, the private ranges, covers it.
- Beside the app, `rproxy`'s address is `127.0.0.1` or `::1`. The private ranges
  leave out loopback: an app left at them ignores the header and sees every
  visitor as `127.0.0.1`. A list given replaces that default, so the app's own
  trusted-proxy setting names them all:
  `10.0.0.0/8,172.16.0.0/12,192.168.0.0/16,127.0.0.1/32,::1/128`.

### upaas

upaas rebuilds and restarts an app's Docker container when its repository
changes. An app it deploys takes the shape of "`rproxy` in the app's container"
with no change to upaas. upaas builds the app's Dockerfile and runs the one
container as it runs any app. The traefik labels route to port 8080, and the
`SWRP_` settings go with the app's environment variables. `discriminator` runs
in a container of its own, on a network the app's container can reach, with its
state volume and the directory of any token file.

## Planned, not built

This point is specified as later work, and nothing in the code does it yet:

- Planned: sending the log over RELP, the reliable syslog transport. It is
  revisited against the available Go packages. Until then the remote log goes
  over UDP, TCP or TLS as described under "Request log".