SPEC follows milestones 1 and 2: build order, two size limits, GeoJS answers in memory (closes #16)
SPEC.md now follows sneak's milestones. The build order starts with milestone 1 and milestone 2, each described briefly and linked, then the earlier stages less what the two milestones build; milestone 2 carries the container image, runit and the health check on /_smallwebwaf/healthz. The four body-size settings become SWWAF_REQUEST_MAX_BYTES and SWWAF_RESPONSE_MAX_BYTES, since smallwebwaf passes bodies through unchanged; the four timeouts stay. GeoJS answers are kept in memory, and writing them to lookups.json comes in milestone 3 or later, as he ruled. README.md loses the two sentences this made wrong. Model: opus-5-5
This commit was merged in pull request #35.
This commit is contained in:
@@ -393,21 +393,22 @@ The settings, by group:
|
||||
- `SWWAF_RATE_LIMIT_EXEMPT_PATHS`: path prefixes not counted (static assets,
|
||||
health checks).
|
||||
- Byte limits, per client. A response's bytes are counted when it ends, so every
|
||||
default sits above the largest response allowed
|
||||
(`SWWAF_CLIENT_RESPONSE_MAX_BYTES`, 5 GiB) and no single download breaks one.
|
||||
default sits above the largest response allowed (`SWWAF_RESPONSE_MAX_BYTES`, 5
|
||||
GiB) and no single download breaks one.
|
||||
- `SWWAF_BYTES_LIMIT_PER_MINUTE` (default `10G`),
|
||||
`SWWAF_BYTES_LIMIT_PER_HOUR` (default `20G`), `SWWAF_BYTES_LIMIT_PER_DAY`
|
||||
(default `50G`).
|
||||
- `SWWAF_BYTES_COUNT` (default `both`): `response`, `request` or `both`.
|
||||
- Size and time limits, per request, in both directions. The client-facing
|
||||
limits apply between the client and `smallwebwaf`, the app-facing ones between
|
||||
`smallwebwaf` and the app. Bodies stream straight through, so a request body
|
||||
reaches the app while the client is still sending it, and of two matching
|
||||
limits the lower one acts first.
|
||||
timeouts apply between the client and `smallwebwaf`, the app-facing ones
|
||||
between `smallwebwaf` and the app, since a slow client and a slow app are
|
||||
separate problems. There is one size limit for request bodies and one for
|
||||
response bodies: `smallwebwaf` passes bodies through unchanged, so a limit on
|
||||
each side would bound the same bytes and the lower one would always decide.
|
||||
Bodies stream straight through, so a request body reaches the app while the
|
||||
client is still sending it.
|
||||
- `SWWAF_CLIENT_REQUEST_TIMEOUT` (default `60s`): how long a client may take
|
||||
to send its whole request, headers and body.
|
||||
- `SWWAF_CLIENT_REQUEST_MAX_BYTES` (default `100M`): the largest request
|
||||
body a client may send.
|
||||
- `SWWAF_CLIENT_REQUEST_HEADER_MAX_BYTES` (default `32K`): the largest
|
||||
request line and headers a client may send. Over it, `smallwebwaf` answers
|
||||
`431` and closes the connection, and nothing reaches the app.
|
||||
@@ -419,17 +420,15 @@ The settings, by group:
|
||||
- `SWWAF_CLIENT_RESPONSE_TIMEOUT` (default `30m`): how long `smallwebwaf`
|
||||
may take to deliver one response to the client, from the end of the
|
||||
request to the last byte.
|
||||
- `SWWAF_CLIENT_RESPONSE_MAX_BYTES` (default `5G`): the largest response
|
||||
body sent to a client.
|
||||
- `SWWAF_UPSTREAM_REQUEST_TIMEOUT` (default `60s`): how long `smallwebwaf`
|
||||
may take to connect to the app and send it the whole request.
|
||||
- `SWWAF_UPSTREAM_REQUEST_MAX_BYTES` (default `100M`): the largest request
|
||||
body sent to the app.
|
||||
- `SWWAF_UPSTREAM_RESPONSE_TIMEOUT` (default `30m`): how long the app may
|
||||
take to send one whole response, from the end of the request to the last
|
||||
byte.
|
||||
- `SWWAF_UPSTREAM_RESPONSE_MAX_BYTES` (default `5G`): the largest response
|
||||
body taken from the app.
|
||||
- `SWWAF_REQUEST_MAX_BYTES` (default `100M`): the largest request body, as
|
||||
the client sends it and the app receives it.
|
||||
- `SWWAF_RESPONSE_MAX_BYTES` (default `5G`): the largest response body, as
|
||||
the app sends it and the client receives it.
|
||||
- When a limit is passed before the response has started, `smallwebwaf`
|
||||
answers itself: `413` for a request body that is too large, `408` for a
|
||||
client that is too slow, `502` for a response that is too large, `504` for
|
||||
@@ -465,11 +464,14 @@ The settings, by group:
|
||||
setting uses the answer, since the request log, the metrics, the client's
|
||||
history and ban notes carry the AS number and country. Private, loopback
|
||||
and link-local addresses, which no source can place, are never sent.
|
||||
- Each GeoJS answer is kept for 7 days in memory and in `lookups.json`,
|
||||
apart from the table of clients, so it survives a restart and outlasts a
|
||||
client dropped from that table; after 7 days the client's next request
|
||||
asks again. Up to 100,000 answers are kept, about 15 MiB; when that many
|
||||
are held, the answer used longest ago goes first.
|
||||
- Each GeoJS answer is kept in memory for 7 days, apart from the table of
|
||||
clients, so it outlasts a client dropped from that table; after 7 days the
|
||||
client's next request asks again. Up to 100,000 answers are kept, about 15
|
||||
MiB; when that many are held, the answer used longest ago goes first.
|
||||
Writing the answers to `lookups.json` and reading them back at start, so
|
||||
that they survive a restart, comes in milestone 3 or later
|
||||
(https://git.eeqj.de/sneak/smallwebwaf/issues/17); until then a restart
|
||||
loses them.
|
||||
- `SWWAF_LOOKUP_TIMEOUT` (default `1s`): how long a request waits for its
|
||||
client's first answer when a setting needs it (see "Data flow for one
|
||||
request"); GeoJS normally answers in a fraction of that. At most one
|
||||
@@ -909,7 +911,8 @@ and the running `smallwebwaf` takes the edit in.
|
||||
`GET /_smallwebwaf/clients/<ip>` shows it.
|
||||
- `lookups.json`: GeoJS answers, one per client: the AS number, AS name and
|
||||
country, when GeoJS was asked and when the answer was last used; up to
|
||||
100,000, each for 7 days.
|
||||
100,000, each for 7 days. This file comes in milestone 3 or later; until
|
||||
then the answers are kept in memory only.
|
||||
- `reputation.json`: the last good copy of each list fetched from a URL,
|
||||
cached DNSBL and reputation API verdicts, each with the time it was
|
||||
fetched, and the AbuseIPDB checks spent today, so a restart does not reset
|
||||
@@ -1291,10 +1294,10 @@ settings go with the app's environment variables, and the volume for
|
||||
The client is answered `413` for a body that is too large, before anything
|
||||
reaches gitea when the request announces its size, or `408` for one that
|
||||
is too slow; the upload fails, and no one is banned for it. A gitea that
|
||||
takes large uploads needs `SWWAF_CLIENT_REQUEST_MAX_BYTES`,
|
||||
`SWWAF_UPSTREAM_REQUEST_MAX_BYTES`, `SWWAF_CLIENT_REQUEST_TIMEOUT` and
|
||||
`SWWAF_UPSTREAM_REQUEST_TIMEOUT` raised to fit. The Core Rule Set does not
|
||||
read an upload's body, which streams through without being held in memory.
|
||||
takes large uploads needs `SWWAF_REQUEST_MAX_BYTES`,
|
||||
`SWWAF_CLIENT_REQUEST_TIMEOUT` and `SWWAF_UPSTREAM_REQUEST_TIMEOUT` raised
|
||||
to fit. The Core Rule Set does not read an upload's body, which streams
|
||||
through without being held in memory.
|
||||
- At the defaults (see "Configuration surface", attack detection), the Core
|
||||
Rule Set lets gitea's ordinary use through, apart from the refusals in the
|
||||
next note: browsing and views of files in a repository, with their
|
||||
@@ -1478,16 +1481,36 @@ settings go with the app's environment variables, and the volume for
|
||||
|
||||
## Build order
|
||||
|
||||
- First: proxy with its size and time limits, client identification, static
|
||||
lists, three-window request limits and the bans they lead to, the ban ledger
|
||||
and the JSON state files with edits taken in while running, exemptions,
|
||||
`observe` mode, the full request log on stdout, the metrics endpoint, health.
|
||||
- Second: rule files, admin endpoints, alerting to all three destinations,
|
||||
remote log sending.
|
||||
- Third: AS number and country lookup from the file or GeoJS, the country lists,
|
||||
biased thresholds, byte limits, anomaly thresholds.
|
||||
- Fourth: blocklists, DNSBL, AbuseIPDB, optional CrowdSec decision feed.
|
||||
- Fifth: attack detection with Coraza and the Core Rule Set, trap paths, error
|
||||
bursts.
|
||||
- Each stage is usable on its own; the first three already cover the traffic
|
||||
problem the fleet has today.
|
||||
- Milestone 1 (https://git.eeqj.de/sneak/smallwebwaf/issues/13): a pass-through
|
||||
proxy with read and write timeouts, size limits and a request log. It
|
||||
identifies the client and passes requests and answers through unchanged within
|
||||
the four timeouts and the two size limits; the header size and the idle time
|
||||
are fixed at their defaults, not settings. Each request gets a line in the
|
||||
request log, with part of its fields. It writes nothing to disk.
|
||||
- Milestone 2 (https://git.eeqj.de/sneak/smallwebwaf/issues/14): per-IP rate
|
||||
limits and country allow and deny lists, ready for production.
|
||||
- The request rate limits per client, over a minute, an hour and a day. A
|
||||
client over one is refused with `429` until it is back under every limit;
|
||||
bans come later.
|
||||
- The country lists, with each client's country looked up through GeoJS,
|
||||
only while a list is set. A client on a private, loopback or link-local
|
||||
address has no country, and neither list checks it.
|
||||
- Like milestone 1, it writes nothing to disk: the GeoJS answers and the
|
||||
rate counters are kept in memory only, and a restart loses them.
|
||||
- The container image described under "Deployment", with runit and the
|
||||
container's health check. The health check calls `/_smallwebwaf/healthz`,
|
||||
so milestone 2 answers that path, although the other admin endpoints come
|
||||
later.
|
||||
- After milestone 2, the rest of the design, in this order:
|
||||
- static lists, the bans that broken request limits lead to, the ban ledger
|
||||
and the JSON state files with edits taken in while running, exemptions,
|
||||
`observe` mode, the rest of the request log's fields, the metrics
|
||||
endpoint;
|
||||
- rule files, the other admin endpoints, alerting to all three destinations,
|
||||
remote log sending;
|
||||
- AS number and country lookup for every client, from the file or GeoJS,
|
||||
biased thresholds, byte limits, anomaly thresholds;
|
||||
- blocklists, DNSBL, AbuseIPDB, optional CrowdSec decision feed;
|
||||
- attack detection with Coraza and the Core Rule Set, trap paths, error
|
||||
bursts.
|
||||
- Each stage is usable on its own; milestone 2 is the first to go to production.
|
||||
|
||||
Reference in New Issue
Block a user