SPEC follows milestones 1 and 2: build order, two size limits, GeoJS answers in memory (closes #16)

SPEC.md now follows sneak's milestones. The build order starts with milestone 1 and milestone 2, each described briefly and linked, then the earlier stages less what the two milestones build; milestone 2 carries the container image, runit and the health check on /_smallwebwaf/healthz. The four body-size settings become SWWAF_REQUEST_MAX_BYTES and SWWAF_RESPONSE_MAX_BYTES, since smallwebwaf passes bodies through unchanged; the four timeouts stay. GeoJS answers are kept in memory, and writing them to lookups.json comes in milestone 3 or later, as he ruled. README.md loses the two sentences this made wrong.

Model: opus-5-5
This commit was merged in pull request #35.
This commit is contained in:
2026-09-29 02:10:34 +02:00
parent ba54ecb009
commit 7be4314f55
2 changed files with 69 additions and 44 deletions
+60 -37
View File
@@ -393,21 +393,22 @@ The settings, by group:
- `SWWAF_RATE_LIMIT_EXEMPT_PATHS`: path prefixes not counted (static assets,
health checks).
- Byte limits, per client. A response's bytes are counted when it ends, so every
default sits above the largest response allowed
(`SWWAF_CLIENT_RESPONSE_MAX_BYTES`, 5 GiB) and no single download breaks one.
default sits above the largest response allowed (`SWWAF_RESPONSE_MAX_BYTES`, 5
GiB) and no single download breaks one.
- `SWWAF_BYTES_LIMIT_PER_MINUTE` (default `10G`),
`SWWAF_BYTES_LIMIT_PER_HOUR` (default `20G`), `SWWAF_BYTES_LIMIT_PER_DAY`
(default `50G`).
- `SWWAF_BYTES_COUNT` (default `both`): `response`, `request` or `both`.
- Size and time limits, per request, in both directions. The client-facing
limits apply between the client and `smallwebwaf`, the app-facing ones between
`smallwebwaf` and the app. Bodies stream straight through, so a request body
reaches the app while the client is still sending it, and of two matching
limits the lower one acts first.
timeouts apply between the client and `smallwebwaf`, the app-facing ones
between `smallwebwaf` and the app, since a slow client and a slow app are
separate problems. There is one size limit for request bodies and one for
response bodies: `smallwebwaf` passes bodies through unchanged, so a limit on
each side would bound the same bytes and the lower one would always decide.
Bodies stream straight through, so a request body reaches the app while the
client is still sending it.
- `SWWAF_CLIENT_REQUEST_TIMEOUT` (default `60s`): how long a client may take
to send its whole request, headers and body.
- `SWWAF_CLIENT_REQUEST_MAX_BYTES` (default `100M`): the largest request
body a client may send.
- `SWWAF_CLIENT_REQUEST_HEADER_MAX_BYTES` (default `32K`): the largest
request line and headers a client may send. Over it, `smallwebwaf` answers
`431` and closes the connection, and nothing reaches the app.
@@ -419,17 +420,15 @@ The settings, by group:
- `SWWAF_CLIENT_RESPONSE_TIMEOUT` (default `30m`): how long `smallwebwaf`
may take to deliver one response to the client, from the end of the
request to the last byte.
- `SWWAF_CLIENT_RESPONSE_MAX_BYTES` (default `5G`): the largest response
body sent to a client.
- `SWWAF_UPSTREAM_REQUEST_TIMEOUT` (default `60s`): how long `smallwebwaf`
may take to connect to the app and send it the whole request.
- `SWWAF_UPSTREAM_REQUEST_MAX_BYTES` (default `100M`): the largest request
body sent to the app.
- `SWWAF_UPSTREAM_RESPONSE_TIMEOUT` (default `30m`): how long the app may
take to send one whole response, from the end of the request to the last
byte.
- `SWWAF_UPSTREAM_RESPONSE_MAX_BYTES` (default `5G`): the largest response
body taken from the app.
- `SWWAF_REQUEST_MAX_BYTES` (default `100M`): the largest request body, as
the client sends it and the app receives it.
- `SWWAF_RESPONSE_MAX_BYTES` (default `5G`): the largest response body, as
the app sends it and the client receives it.
- When a limit is passed before the response has started, `smallwebwaf`
answers itself: `413` for a request body that is too large, `408` for a
client that is too slow, `502` for a response that is too large, `504` for
@@ -465,11 +464,14 @@ The settings, by group:
setting uses the answer, since the request log, the metrics, the client's
history and ban notes carry the AS number and country. Private, loopback
and link-local addresses, which no source can place, are never sent.
- Each GeoJS answer is kept for 7 days in memory and in `lookups.json`,
apart from the table of clients, so it survives a restart and outlasts a
client dropped from that table; after 7 days the client's next request
asks again. Up to 100,000 answers are kept, about 15 MiB; when that many
are held, the answer used longest ago goes first.
- Each GeoJS answer is kept in memory for 7 days, apart from the table of
clients, so it outlasts a client dropped from that table; after 7 days the
client's next request asks again. Up to 100,000 answers are kept, about 15
MiB; when that many are held, the answer used longest ago goes first.
Writing the answers to `lookups.json` and reading them back at start, so
that they survive a restart, comes in milestone 3 or later
(https://git.eeqj.de/sneak/smallwebwaf/issues/17); until then a restart
loses them.
- `SWWAF_LOOKUP_TIMEOUT` (default `1s`): how long a request waits for its
client's first answer when a setting needs it (see "Data flow for one
request"); GeoJS normally answers in a fraction of that. At most one
@@ -909,7 +911,8 @@ and the running `smallwebwaf` takes the edit in.
`GET /_smallwebwaf/clients/<ip>` shows it.
- `lookups.json`: GeoJS answers, one per client: the AS number, AS name and
country, when GeoJS was asked and when the answer was last used; up to
100,000, each for 7 days.
100,000, each for 7 days. This file comes in milestone 3 or later; until
then the answers are kept in memory only.
- `reputation.json`: the last good copy of each list fetched from a URL,
cached DNSBL and reputation API verdicts, each with the time it was
fetched, and the AbuseIPDB checks spent today, so a restart does not reset
@@ -1291,10 +1294,10 @@ settings go with the app's environment variables, and the volume for
The client is answered `413` for a body that is too large, before anything
reaches gitea when the request announces its size, or `408` for one that
is too slow; the upload fails, and no one is banned for it. A gitea that
takes large uploads needs `SWWAF_CLIENT_REQUEST_MAX_BYTES`,
`SWWAF_UPSTREAM_REQUEST_MAX_BYTES`, `SWWAF_CLIENT_REQUEST_TIMEOUT` and
`SWWAF_UPSTREAM_REQUEST_TIMEOUT` raised to fit. The Core Rule Set does not
read an upload's body, which streams through without being held in memory.
takes large uploads needs `SWWAF_REQUEST_MAX_BYTES`,
`SWWAF_CLIENT_REQUEST_TIMEOUT` and `SWWAF_UPSTREAM_REQUEST_TIMEOUT` raised
to fit. The Core Rule Set does not read an upload's body, which streams
through without being held in memory.
- At the defaults (see "Configuration surface", attack detection), the Core
Rule Set lets gitea's ordinary use through, apart from the refusals in the
next note: browsing and views of files in a repository, with their
@@ -1478,16 +1481,36 @@ settings go with the app's environment variables, and the volume for
## Build order
- First: proxy with its size and time limits, client identification, static
lists, three-window request limits and the bans they lead to, the ban ledger
and the JSON state files with edits taken in while running, exemptions,
`observe` mode, the full request log on stdout, the metrics endpoint, health.
- Second: rule files, admin endpoints, alerting to all three destinations,
remote log sending.
- Third: AS number and country lookup from the file or GeoJS, the country lists,
biased thresholds, byte limits, anomaly thresholds.
- Fourth: blocklists, DNSBL, AbuseIPDB, optional CrowdSec decision feed.
- Fifth: attack detection with Coraza and the Core Rule Set, trap paths, error
bursts.
- Each stage is usable on its own; the first three already cover the traffic
problem the fleet has today.
- Milestone 1 (https://git.eeqj.de/sneak/smallwebwaf/issues/13): a pass-through
proxy with read and write timeouts, size limits and a request log. It
identifies the client and passes requests and answers through unchanged within
the four timeouts and the two size limits; the header size and the idle time
are fixed at their defaults, not settings. Each request gets a line in the
request log, with part of its fields. It writes nothing to disk.
- Milestone 2 (https://git.eeqj.de/sneak/smallwebwaf/issues/14): per-IP rate
limits and country allow and deny lists, ready for production.
- The request rate limits per client, over a minute, an hour and a day. A
client over one is refused with `429` until it is back under every limit;
bans come later.
- The country lists, with each client's country looked up through GeoJS,
only while a list is set. A client on a private, loopback or link-local
address has no country, and neither list checks it.
- Like milestone 1, it writes nothing to disk: the GeoJS answers and the
rate counters are kept in memory only, and a restart loses them.
- The container image described under "Deployment", with runit and the
container's health check. The health check calls `/_smallwebwaf/healthz`,
so milestone 2 answers that path, although the other admin endpoints come
later.
- After milestone 2, the rest of the design, in this order:
- static lists, the bans that broken request limits lead to, the ban ledger
and the JSON state files with edits taken in while running, exemptions,
`observe` mode, the rest of the request log's fields, the metrics
endpoint;
- rule files, the other admin endpoints, alerting to all three destinations,
remote log sending;
- AS number and country lookup for every client, from the file or GeoJS,
biased thresholds, byte limits, anomaly thresholds;
- blocklists, DNSBL, AbuseIPDB, optional CrowdSec decision feed;
- attack detection with Coraza and the Core Rule Set, trap paths, error
bursts.
- Each stage is usable on its own; milestone 2 is the first to go to production.