Core Rule Set reads request bodies up to SWWAF_WAF_BODY_LIMIT (closes #116)

SWWAF_WAF_BODY_LIMIT (default off, at most 1G) has the Core Rule Set read
form data and multipart up to the limit, the rest streaming on, and JSON
and XML (with +json, text/json and +xml) no larger than it. The part read
is held for the app. A size or time limit met while reading ends the
request. Content-Encoding is refused again on these kinds. A body Coraza
cannot parse, or a multipart body failing its strict checks, adds 5, but
not a multipart body reaching the limit. Coraza is built with
no_fs_access, so writes no file. Rule 900300 moves to phase 2.

Judgement call: Content-Encoding is refused on a JSON or XML body too
large to read, as SPEC.md allows.

Model: opus-5-5

README.md

@@ -64,11 +64,10 @@ cannot read, serves Prometheus metrics to a scraper that holds the metrics
 token, lets an admin who holds the admin token list, add and lift bans and ask
 what it knows of a client, and in `observe` mode passes on the requests it would
 refuse, logging what it would have done with them. It comes as the image the
-app's own image is built on. The Core Rule Set reads no request body yet:
-`SWWAF_WAF_BODY_LIMIT`, which switches that on, comes with
-https://git.eeqj.de/sneak/smallwebwaf/issues/116. The rest of the design comes
-after that, in the order of the build order in [`SPEC.md`](SPEC.md). The survey
-of existing tools that led to the design is in [`EVALUATION.md`](EVALUATION.md).
+app's own image is built on. The Core Rule Set reads a request's body too once
+`SWWAF_WAF_BODY_LIMIT` is set. The rest of the design comes next, in the order
+of the build order in [`SPEC.md`](SPEC.md). The survey of existing tools that
+led to the design is in [`EVALUATION.md`](EVALUATION.md).
 
 ## Getting started
 
@@ -98,7 +97,9 @@ in `bin/state` unless `SWWAF_STATE_DIR` is set, and the default rule file of
 
 - Passes each request to the app and the app's answer back unchanged: method,
   path, query, headers, body and status. Bodies stream through in both
-  directions and are never held whole in memory. A WebSocket, or any other
+  directions and are never held whole in memory, but for the part of a request
+  body the Core Rule Set reads, at most `SWWAF_WAF_BODY_LIMIT` and one byte
+  more, which is held until the app is sent it. A WebSocket, or any other
   upgraded connection, passes through, and the timeouts do not cut it.
 - Works out the client's address. A TCP peer outside `SWWAF_TRUSTED_PROXIES` is
   the client, and the forwarded headers it sends are replaced, not passed on.
@@ -203,26 +204,33 @@ in `bin/state` unless `SWWAF_STATE_DIR` is set, and the default rule file of
   A client in `SWWAF_ALLOW_NETS` is not checked.
 - Inspects each request with the OWASP Core Rule Set 4.25.0, run by Coraza,
   after the rule files, unless `SWWAF_WAF_MODE` is `off`: its method, its URL
-  with the query, and its headers, but no request body and no response. Each of
-  its rules that matches adds to the request's anomaly score, up to the paranoia
-  level `SWWAF_WAF_PARANOIA_LEVEL` sets. A request with more than 1000 query
-  parameters adds 5 (rule 900300), as a rule rated critical does, since Coraza
-  reads only the first 1000. A score at or over `SWWAF_WAF_ANOMALY_THRESHOLD`, 5
-  by default, is a match: in `block` mode, the default, the request is refused
-  with `403`, and in `detect` mode it goes on to the app. Either way its log
-  line names the rules and the score (see `waf_rule_ids` and `waf_score` in
-  "Request log" below), and it raises a `waf_block` alert. A refusal bans no one
-  by itself, since the Core Rule Set takes some ordinary requests for attacks,
-  but it is an offence the client's history counts, and it counts toward the
-  error burst; a match in `detect` mode is neither. A request a rule file
-  refuses, one for a path `SWWAF_WAF_EXEMPT_PATHS` exempts, and one from a
-  client in `SWWAF_ALLOW_NETS` are not inspected. `smallwebwaf` changes the Core
-  Rule Set in six ways, so that gitea's ordinary requests get through, and no
-  setting undoes them:
+  with the query, and its headers, and, once `SWWAF_WAF_BODY_LIMIT` is set, its
+  body when it is form data, multipart, JSON or XML, as that setting below
+  describes; no response is inspected. Each of its rules that matches adds to
+  the request's anomaly score, up to the paranoia level
+  `SWWAF_WAF_PARANOIA_LEVEL` sets. A request with more than 1000 query
+  parameters, or more than 1000 fields in a form data or JSON body it reads,
+  adds 5 (rule 900300), as a rule rated critical does, since Coraza reads only
+  the first 1000. A score at or over `SWWAF_WAF_ANOMALY_THRESHOLD`, 5 by
+  default, is a match: in `block` mode, the default, the request is refused with
+  `403`, and in `detect` mode it goes on to the app. Either way its log line
+  names the rules and the score (see `waf_rule_ids` and `waf_score` in "Request
+  log" below), and it raises a `waf_block` alert. A refusal bans no one by
+  itself, since the Core Rule Set takes some ordinary requests for attacks, but
+  it is an offence the client's history counts, and it counts toward the error
+  burst; a match in `detect` mode is neither. A request a rule file refuses, one
+  for a path `SWWAF_WAF_EXEMPT_PATHS` exempts, and one from a client in
+  `SWWAF_ALLOW_NETS` are not inspected. `smallwebwaf` changes the Core Rule Set
+  in six ways, so that gitea's ordinary requests get through, and no setting
+  undoes them:
     - `PUT`, `PATCH` and `DELETE` are allowed besides `GET`, `HEAD`, `POST` and
       `OPTIONS`; any other method stays refused.
     - The headers `Expect` and `Content-Encoding` are allowed; the others the
       Core Rule Set refuses, such as `Proxy` and `Content-Range`, stay refused.
+      Once `SWWAF_WAF_BODY_LIMIT` is set, `Content-Encoding` is refused again
+      (rule 920450) on form data, multipart, JSON and XML, the kinds of body the
+      Core Rule Set reads, since a compressed body cannot be inspected; so it is
+      on a JSON or XML body too large to be read.
     - The query parameter `redirect_uri` is not checked for a URL naming an IP
       address or `localhost` (rules 931100 and 934110), which Git Credential
       Manager, git-credential-oauth and tea ask to be sent back to.
@@ -234,7 +242,12 @@ in `bin/state` unless `SWWAF_STATE_DIR` is set, and the default rule file of
       refuse, such as `|cat /etc/passwd`; path traversal and SQL injection are
       still refused there. These names, and `redirect_uri` above, are matched
       without regard to case, as Coraza matches them, so `Path` or `PATH` is
-      treated as `path`.
+      treated as `path`. Once `SWWAF_WAF_BODY_LIMIT` is set, they are left out
+      in the same way among the fields of a form data or multipart body, which
+      Coraza holds with the query parameters, and gitea posts some of them in
+      its forms: `redirect_uri` when an OAuth sign-in is granted, and `ref` when
+      a workflow is run by hand. A field of a JSON body is named by its path,
+      such as `json.path`, and keeps these rules.
     - The cookies `gitea_flash` and `redirect_to` are not read, and `Referer` is
       not checked for a Unix command given without arguments (932340) or for
       Java starting a process (944110); it is checked by every other rule.
@@ -697,6 +710,29 @@ effective settings are logged at start, unless `SWWAF_LOG_LEVEL` is `warn` or
   Core Rule Set does not inspect, each starting with `/`, matched as
   `SWWAF_RATE_LIMIT_EXEMPT_PATHS` matches its own: a request whose path holds
   `..`, a backslash or an encoded slash is inspected whatever its prefix.
+- `SWWAF_WAF_BODY_LIMIT` (default `off`): `off` has the Core Rule Set read no
+  request body. A size, such as `128K`, at most `1G`, has it read a body of form
+  data or multipart up to that size, the rest of a longer one passing on to the
+  app as it arrives, without being held, and a JSON or XML body no larger than
+  that size, since those cannot be read in part. A body whose type ends in
+  `+json`, or is `text/json`, is JSON, and one whose type ends in `+xml` is XML.
+  Any other body reaches the app uninspected, and so does a larger JSON or XML
+  body: the Core Rule Set would read any other body as form data, where binary
+  content such as a git push trips rules written for text. A body it reads that
+  Coraza cannot parse (rule 900440), and a multipart body that fails Coraza's
+  strict checks (rule 900450), add 5 to the score, as a rule rated critical
+  does, since no rule reads what comes after the fault; a multipart body that
+  reaches the limit does not, since the limit can cut it inside a part's header.
+  The client has until `SWWAF_CLIENT_REQUEST_TIMEOUT` runs out to send the part
+  that is read, and a request whose body passes `SWWAF_REQUEST_MAX_BYTES` within
+  it is refused before anything reaches the app. Of a file in a multipart body,
+  Coraza counts the bytes and writes nothing. Body inspection suits apps whose
+  forms carry no code. In front of gitea it refuses issue and comment text, wiki
+  pages and files saved in the web editor that hold shell commands or code
+  (932125, 932235, 932250 and others), package descriptions that show code, PyPI
+  uploads (922130), and attachments named like `debug.log` or `config.yml`
+  (932180), until the rule ids the request log names are added to
+  `SWWAF_WAF_DISABLED_RULES`.
 - `SWWAF_TRAP_PATHS` (default empty): paths the app never serves and only
   scanners ask for, such as `/wp-login.php,/xmlrpc.php` in front of gitea; a
   request for one bans its client for a clear sign of attack, as a `ban` rule
@@ -985,13 +1021,13 @@ which every line has.
   from when the request's headers had been read to when its line is written, and
   `duration_checks` over the same start to when the checks were done; the health
   check runs none, and its line has no `duration_checks`. `duration_waf`, the
-  part of the checks the Core Rule Set took, is there with `waf_score`.
-  `duration_upstream_connect`, `duration_upstream_first_byte` and
-  `duration_upstream_total` are there for a request passed to the app, and run
-  from when it was handed to the app: until there was a connection to it, new or
-  kept open from an earlier request, until the first byte of its answer arrived,
-  and until the end. The first two are left out when that never happened, as for
-  an app that cannot be reached.
+  part of the checks the Core Rule Set took, reading the part of the body it
+  reads included, is there with `waf_score`. `duration_upstream_connect`,
+  `duration_upstream_first_byte` and `duration_upstream_total` are there for a
+  request passed to the app, and run from when it was handed to the app: until
+  there was a connection to it, new or kept open from an earlier request, until
+  the first byte of its answer arrived, and until the end. The first two are
+  left out when that never happened, as for an app that cannot be reached.
 
 No body is logged, and no header but those above. `smallwebwaf`'s own messages
 (start, the settings, stop, errors) share the stream as JSON lines marked
@@ -2214,15 +2250,17 @@ alerts, the metrics nor `reputation.json` hold it. Given as a file, with
   which bans the client, for a DNSBL zone's verdict, for AbuseIPDB's score, for
   a rate limit, which bans the client, for a trap path, which bans the client,
   for a `block` or `ban` rule, the latter banning the client, for a Core Rule
-  Set match in `block` mode, and for an announced body over the size limit; in
-  `observe` mode, only for the size limit, with what it would have refused for
-  noted in the log line. A request under `/_smallwebwaf/` that `check` lets
-  through is answered by `answerAdmin` instead of reaching the app. Once the
-  answer to a request passed to the app has ended, `countBytes` counts its bytes
-  for the byte limits, and once any request but the health check has ended,
-  `countRefusal` counts it for the error burst if it was refused after a rule
-  file match, a trap path or a Core Rule Set match or for its token, and
-  `countAnomalies` counts it for the anomaly thresholds.
+  Set match in `block` mode, for a body that passes the size limit or
+  `SWWAF_CLIENT_REQUEST_TIMEOUT` while the Core Rule Set reads it, and for an
+  announced body over the size limit; in `observe` mode, only for the last two,
+  with what it would have refused for noted in the log line. A request under
+  `/_smallwebwaf/` that `check` lets through is answered by `answerAdmin`
+  instead of reaching the app. Once the answer to a request passed to the app
+  has ended, `countBytes` counts its bytes for the byte limits, and once any
+  request but the health check has ended, `countRefusal` counts it for the error
+  burst if it was refused after a rule file match, a trap path or a Core Rule
+  Set match or for its token, and `countAnomalies` counts it for the anomaly
+  thresholds.
 - `internal/metrics`: the metrics, counted as the other parts tell it what
   happened, and served in the Prometheus text format.
 - `internal/bans`: the ban ledger: each netblock's bans with their notes, how
@@ -2231,8 +2269,8 @@ alerts, the metrics nor `reputation.json` hold it. Given as a file, with
 - `internal/rules`: reads the rule files at start and again as they change, and
   tells which of their rules a request matches.
 - `internal/waf`: the Core Rule Set with the six changes, as Coraza's own
-  directives, and what it finds in a request's method, URL and headers: the
-  rules that matched and the anomaly score.
+  directives, and what it finds in a request's method, URL, headers and the part
+  of its body it reads: the rules that matched and the anomaly score.
 - `internal/lookup`: looks up each client's AS number and country through GeoJS,
   keeps the answers, and hands each new one to the proxy, which adds it to the
   client's history and to the notes of its bans; or in the lookup database,