SPEC: requirements and reasons only, reference material in docs/

docs/settings.md is now the one settings reference; the spec keeps the
conventions, the off list and each group's requirements and reasons, and
links to it for the per-setting entries. The endpoint list, the ban
notes, the state file contents, the rule file format and the alert
object likewise live in docs/ and are linked from the spec. The package
list from the README's Design section joins "Architecture". The nine
docs files are byte-identical to those on the README branch. Package
comments that named "Alert webhook schema", "Request log" and "Metrics
endpoint" for their field lists now name the docs files.

Model: fable-5-1

docs/rules.md

@@ -0,0 +1,114 @@
+# Rule files
+
+`smallwebwaf` reads every `*.rules` file in `SWWAF_RULES_DIR`, in the order of
+their names, and checks each request against their rules in that order, unless
+the client is in `SWWAF_ALLOW_NETS`. The requirements the files meet are under
+"Rule files" in [`SPEC.md`](../SPEC.md). A file whose name starts with `.`, such
+as an editor's lock file, is not a rule file.
+
+## The format
+
+A rule is a line of four fields separated by spaces or tabs: an id, a target, an
+action and a regex. The regex runs to the end of the line. Blank lines and lines
+starting with `#` are ignored.
+
+```
+# id           target      action  regex
+env-file       path        ban     (?i)^/\.env(\.[a-z]+)?$
+scanner-agent  user_agent  ban     (?i)\b(sqlmap|nikto|nuclei|wpscan)\b
+```
+
+- The id is letters, digits, `-` and `_`. No two rules share one. The log line,
+  the metrics, the alerts and a ban's notes name the rule by it.
+- The target is what the regex is matched against:
+    - `path`: the path without the query;
+    - `uri`: the path and the query together;
+    - `query`: the query as sent;
+    - `method`, `host`, `user_agent` and `referer`;
+    - `header:<Name>`: any one request header but `Host` and
+      `Transfer-Encoding`, which Go's HTTP server takes out of every request.
+
+    `path` and `uri` are matched both as sent and once percent-decoded, so that
+    an encoded probe does not slip past. A header sent more than once is matched
+    with its values joined by `, `. A header not sent is empty text. No body is
+    read; reading bodies is the Core Rule Set's job, once `SWWAF_WAF_BODY_LIMIT`
+    switches it on.
+
+- The action is `log`, `block` or `ban`. `log` only notes the match in the log
+  line. `block` refuses the request with `403` and bans no one; the refusal
+  counts toward the error burst. `ban` refuses it and bans the client, as a
+  clear sign of attack. Matching stops at the first rule that refuses. Keep
+  `ban` for requests no real visitor sends. Anchor a path at the site root with
+  `^/`: a file of the same name deeper in a site can be ordinary content.
+- The regex is in Go's syntax, RE2. It has no backreferences or lookaround, and
+  takes time linear in the text it reads, so no rule can be made to stall the
+  proxy. It matches anywhere in the target unless anchored with `^` and `$`.
+  `(?i)` at its front makes it ignore case.
+
+## Errors
+
+These stop the start, with a message naming the file and line:
+
+- a line that is not a rule;
+- a bad `header:` name, or a rule for `Host` or `Transfer-Encoding`;
+- a regex that does not compile;
+- an id used twice.
+
+So does a `SWWAF_RULES_DIR` that does not exist. An empty directory is not an
+error: the log says that no rules were loaded.
+
+While it runs, `smallwebwaf` watches the directory. It reads the files again
+once the directory has had no change for 2 seconds, so that a file still being
+written is read whole. If the files then hold such an error, the rules stay as
+they were. The log and a `file_error` alert name the file and line. Once the
+file is fixed, it is read again.
+
+## The default file
+
+The image ships one file, `share/rules.d/00-default.rules`, installed as
+`/etc/smallwebwaf/rules.d/00-default.rules`:
+
+```
+# 00-default.rules: probes no real visitor sends, anchored at the site root
+
+# id             target      action  regex
+env-file         path        ban     (?i)^/\.env(\.[a-z]+)?$
+vcs-dir          path        ban     (?i)^/\.(git|svn|hg|bzr)(/|$)
+secrets-dir      path        ban     (?i)^/\.(aws|ssh|docker|kube)/
+secret-file      path        ban     (?i)^/\.(htpasswd|htaccess|npmrc|netrc|pgpass|git-credentials|bash_history|DS_Store)$
+editor-dir       path        ban     (?i)^/\.(vscode|idea)/
+backup-file      path        ban     (?i)^/[^/]+\.(php(\.[a-z0-9]+|~)|sql(\.[a-z0-9]+)?)$
+log-file         path        ban     (?i)^/(debug|error|access)\.log$
+compose-file     path        ban     (?i)^/(docker-)?compose\.ya?ml$
+php-shell        path        ban     (?i)^/(shell|c99|r57|wso|alfa)\.php$
+scanner-agent    user_agent  ban     (?i)\b(sqlmap|nikto|nuclei|masscan|zgrab|wpscan)\b
+path-traversal   uri         block   (\.\./){2,}
+empty-agent      user_agent  log     ^$
+```
+
+It is kept short and limited to patterns that are wrong for every app, with its
+path rules anchored at the site root. In front of gitea, `/.env` is a probe,
+while `/<owner>/<repo>/src/branch/main/.env.example` is a file in a repository
+that any visitor may open. Anything app-specific belongs in a file the app's
+`Dockerfile` adds or the deployer mounts. A request for `/wp-login.php`, for
+example, is a clear sign of attack in front of gitea and an ordinary login in
+front of WordPress. Such a rule goes in a file named to sort after the default:
+
+```
+# 50-gitea.rules: WordPress probes, which gitea never serves
+wp-probe         path        ban     (?i)^/(wp-login\.php|xmlrpc\.php|wp-admin/)
+```
+
+```dockerfile
+COPY 50-gitea.rules /etc/smallwebwaf/rules.d/50-gitea.rules
+```
+
+## Mounting rule files
+
+A directory mounted over `/etc/smallwebwaf/rules.d` replaces the default file;
+single files mounted into it add to it. Many editors save a file by replacing
+it, and a container keeps seeing the old one when the file alone is mounted. So
+rules to be edited while `smallwebwaf` runs belong in a mounted directory, with
+a copy of the default file if the defaults are to stay. The lookup database is
+mounted the same way, for the same reason. To run without rules, mount an empty
+directory or set `SWWAF_RULES_ENABLED=false`.