README rewritten from scratch so a newcomer can follow it (closes #154, closes #155)

The old README opened with project history and one list of a hundred
settings. The new one leads with what smallwebwaf is, what it protects
against, how to build and deploy it, how it sits in front of the app,
what happens to a request, the settings most deployments touch, how to
check it works and how to develop it. The reference material, every
setting, bans, rule files, the Core Rule Set, the admin endpoints, the
request log, alerts, metrics, state files, lookups, reputation sources,
the image and gitea, moves to files under docs/, each linked from the
README. Every statement was checked against the code on next.

Model: fable-5-1

docs/rules.md

@@ -0,0 +1,114 @@
+# Rule files
+
+`smallwebwaf` reads every `*.rules` file in `SWWAF_RULES_DIR`, in the order of
+their names, and checks each request against their rules in that order, unless
+the client is in `SWWAF_ALLOW_NETS`. The requirements the files meet are under
+"Rule files" in [`SPEC.md`](../SPEC.md). A file whose name starts with `.`, such
+as an editor's lock file, is not a rule file.
+
+## The format
+
+A rule is a line of four fields separated by spaces or tabs: an id, a target, an
+action and a regex. The regex runs to the end of the line. Blank lines and lines
+starting with `#` are ignored.
+
+```
+# id           target      action  regex
+env-file       path        ban     (?i)^/\.env(\.[a-z]+)?$
+scanner-agent  user_agent  ban     (?i)\b(sqlmap|nikto|nuclei|wpscan)\b
+```
+
+- The id is letters, digits, `-` and `_`. No two rules share one. The log line,
+  the metrics, the alerts and a ban's notes name the rule by it.
+- The target is what the regex is matched against:
+    - `path`: the path without the query;
+    - `uri`: the path and the query together;
+    - `query`: the query as sent;
+    - `method`, `host`, `user_agent` and `referer`;
+    - `header:<Name>`: any one request header but `Host` and
+      `Transfer-Encoding`, which Go's HTTP server takes out of every request.
+
+    `path` and `uri` are matched both as sent and once percent-decoded, so that
+    an encoded probe does not slip past. A header sent more than once is matched
+    with its values joined by `, `. A header not sent is empty text. No body is
+    read; reading bodies is the Core Rule Set's job, once `SWWAF_WAF_BODY_LIMIT`
+    switches it on.
+
+- The action is `log`, `block` or `ban`. `log` only notes the match in the log
+  line. `block` refuses the request with `403` and bans no one; the refusal
+  counts toward the error burst. `ban` refuses it and bans the client, as a
+  clear sign of attack. Matching stops at the first rule that refuses. Keep
+  `ban` for requests no real visitor sends. Anchor a path at the site root with
+  `^/`: a file of the same name deeper in a site can be ordinary content.
+- The regex is in Go's syntax, RE2. It has no backreferences or lookaround, and
+  takes time linear in the text it reads, so no rule can be made to stall the
+  proxy. It matches anywhere in the target unless anchored with `^` and `$`.
+  `(?i)` at its front makes it ignore case.
+
+## Errors
+
+These stop the start, with a message naming the file and line:
+
+- a line that is not a rule;
+- a bad `header:` name, or a rule for `Host` or `Transfer-Encoding`;
+- a regex that does not compile;
+- an id used twice.
+
+So does a `SWWAF_RULES_DIR` that does not exist. An empty directory is not an
+error: the log says that no rules were loaded.
+
+While it runs, `smallwebwaf` watches the directory. It reads the files again
+once the directory has had no change for 2 seconds, so that a file still being
+written is read whole. If the files then hold such an error, the rules stay as
+they were. The log and a `file_error` alert name the file and line. Once the
+file is fixed, it is read again.
+
+## The default file
+
+The image ships one file, `share/rules.d/00-default.rules`, installed as
+`/etc/smallwebwaf/rules.d/00-default.rules`:
+
+```
+# 00-default.rules: probes no real visitor sends, anchored at the site root
+
+# id             target      action  regex
+env-file         path        ban     (?i)^/\.env(\.[a-z]+)?$
+vcs-dir          path        ban     (?i)^/\.(git|svn|hg|bzr)(/|$)
+secrets-dir      path        ban     (?i)^/\.(aws|ssh|docker|kube)/
+secret-file      path        ban     (?i)^/\.(htpasswd|htaccess|npmrc|netrc|pgpass|git-credentials|bash_history|DS_Store)$
+editor-dir       path        ban     (?i)^/\.(vscode|idea)/
+backup-file      path        ban     (?i)^/[^/]+\.(php(\.[a-z0-9]+|~)|sql(\.[a-z0-9]+)?)$
+log-file         path        ban     (?i)^/(debug|error|access)\.log$
+compose-file     path        ban     (?i)^/(docker-)?compose\.ya?ml$
+php-shell        path        ban     (?i)^/(shell|c99|r57|wso|alfa)\.php$
+scanner-agent    user_agent  ban     (?i)\b(sqlmap|nikto|nuclei|masscan|zgrab|wpscan)\b
+path-traversal   uri         block   (\.\./){2,}
+empty-agent      user_agent  log     ^$
+```
+
+It is kept short and limited to patterns that are wrong for every app, with its
+path rules anchored at the site root. In front of gitea, `/.env` is a probe,
+while `/<owner>/<repo>/src/branch/main/.env.example` is a file in a repository
+that any visitor may open. Anything app-specific belongs in a file the app's
+`Dockerfile` adds or the deployer mounts. A request for `/wp-login.php`, for
+example, is a clear sign of attack in front of gitea and an ordinary login in
+front of WordPress. Such a rule goes in a file named to sort after the default:
+
+```
+# 50-gitea.rules: WordPress probes, which gitea never serves
+wp-probe         path        ban     (?i)^/(wp-login\.php|xmlrpc\.php|wp-admin/)
+```
+
+```dockerfile
+COPY 50-gitea.rules /etc/smallwebwaf/rules.d/50-gitea.rules
+```
+
+## Mounting rule files
+
+A directory mounted over `/etc/smallwebwaf/rules.d` replaces the default file;
+single files mounted into it add to it. Many editors save a file by replacing
+it, and a container keeps seeing the old one when the file alone is mounted. So
+rules to be edited while `smallwebwaf` runs belong in a mounted directory, with
+a copy of the default file if the defaults are to stay. The lookup database is
+mounted the same way, for the same reason. To run without rules, mount an empty
+directory or set `SWWAF_RULES_ENABLED=false`.