Send every log line to a syslog server as well (closes #28)

With SWWAF_LOG_REMOTE_URL set (syslog+udp, syslog+tcp or syslog+tls),
every line on stdout is also sent as the message of an RFC 5424 record,
octet-counted over TCP and TLS, from a bounded buffer that drops its
oldest line when full, so a slow or unreachable server holds up nothing.
Failed connections are retried with backoff; lines sent, dropped and
waiting are metrics. At a stop the lines still waiting get at most two
seconds. Standard library only: log/syslog writes only the older format.

Deviation: SWWAF_LOG_REMOTE_APP_NAME defaults to the host's name until
SWWAF_INSTANCE_NAME exists.

Model: opus-5-5

README.md

@@ -18,18 +18,20 @@ milestone 3: the static lists, the bans that broken rate limits lead to and the
 JSON state files with your edits taken in while it runs, which come next in the
 build order, `observe` mode, which comes a little later, and the metrics
 endpoint and the header size and the idle time as settings, which come last in
-it. `smallwebwaf` passes each request to the app and the app's answer back,
-unchanged, within its timeouts and size limits, works out each client's address,
-bans a client that sends too many requests, refuses a client that comes from a
-country you refuse or from a network you refuse, lets the networks you choose
-through, keeps its bans, each client's counters and history, and GeoJS's answers
-in JSON files across restarts, takes in your edits of those files while it runs,
-writes a JSON log line for every request, serves Prometheus metrics to a scraper
-that holds the metrics token, and in `observe` mode passes on the requests it
-would refuse, logging what it would have done with them. It comes as the image
-the app's own image is built on. The rest of the design comes after that, in the
-order of the build order in [`SPEC.md`](SPEC.md). The survey of existing tools
-that led to the design is in [`EVALUATION.md`](EVALUATION.md).
+it; and, from the stage after it, remote log sending. `smallwebwaf` passes each
+request to the app and the app's answer back, unchanged, within its timeouts and
+size limits, works out each client's address, bans a client that sends too many
+requests, refuses a client that comes from a country you refuse or from a
+network you refuse, lets the networks you choose through, keeps its bans, each
+client's counters and history, and GeoJS's answers in JSON files across
+restarts, takes in your edits of those files while it runs, writes a JSON log
+line for every request, sends its log lines to a syslog server too if you name
+one, serves Prometheus metrics to a scraper that holds the metrics token, and in
+`observe` mode passes on the requests it would refuse, logging what it would
+have done with them. It comes as the image the app's own image is built on. The
+rest of the design comes after that, in the order of the build order in
+[`SPEC.md`](SPEC.md). The survey of existing tools that led to the design is in
+[`EVALUATION.md`](EVALUATION.md).
 
 ## Getting started
 
@@ -144,6 +146,9 @@ in `bin/state` unless `SWWAF_STATE_DIR` is set.
   passed to the app: a banned client stays refused, and each counts toward the
   client's rate limits. None of them reaches the app.
 - Writes a line in the request log for each request (see "Request log" below).
+- Sends every line it writes on stdout to a syslog server as well, while
+  `SWWAF_LOG_REMOTE_URL` names one (see "Sending the log to a syslog server"
+  below).
 
 ## Settings
 
@@ -227,6 +232,23 @@ it, and the effective settings are logged at start.
   `********` in its place.
 - `SWWAF_METRICS_TOP_N` (default `50`): how many countries get series of their
   own in the metrics by country; the others are counted as `other`.
+- `SWWAF_LOG_REMOTE_URL` (default unset): a syslog server that every line on
+  stdout is also sent to, as `syslog+udp://`, `syslog+tcp://` or `syslog+tls://`
+  with a host and a port, such as `syslog+tls://logs.example:6514`. Unset or
+  empty, nothing is sent.
+- `SWWAF_LOG_REMOTE_TLS_CA_FILE` (default unset): a file of PEM certificates,
+  which the certificate of a `syslog+tls` server must chain to instead of the
+  host's own. A file that cannot be read or holds no certificate stops the
+  start.
+- `SWWAF_LOG_REMOTE_BUFFER` (default `10000`): the most lines held while they
+  wait to be sent.
+- `SWWAF_LOG_REMOTE_FACILITY` (default `local0`): the syslog facility the lines
+  are sent with: `kern`, `user`, `mail`, `daemon`, `auth`, `syslog`, `lpr`,
+  `news`, `uucp`, `cron`, `authpriv`, `ftp`, or `local0` to `local7`.
+- `SWWAF_LOG_REMOTE_APP_NAME` (default the host's name): the app name the lines
+  are sent with, 1 to 48 printable ASCII characters without a space. While
+  `SWWAF_LOG_REMOTE_URL` is set, a default that is not such a name stops the
+  start too.
 
 Durations are in Go's syntax, with `d` for days (`90s`, `15m`, `7d`). Sizes are
 bytes, with an optional `K`, `M` or `G`, which are powers of 1024 (`1K` is 1024
@@ -236,8 +258,8 @@ ISO 3166-1 assigns today, and `xk` for Kosovo, in either case (`de` and `DE` are
 the same); any other code, such as `nk` (North Korea is `kp`) or the withdrawn
 `su`, stops the start, and so does a code on both country lists. `off` switches
 a timeout, a size limit or a rate limit off;
-`SWWAF_CLIENT_REQUEST_HEADER_MAX_BYTES`, the ban settings, the state settings
-and `SWWAF_METRICS_TOP_N` cannot be off.
+`SWWAF_CLIENT_REQUEST_HEADER_MAX_BYTES`, the ban settings, the state settings,
+`SWWAF_METRICS_TOP_N` and `SWWAF_LOG_REMOTE_BUFFER` cannot be off.
 
 Several limits are fixed rather than settings. At most 20,000 clients are kept,
 with their counters and history, and an IPv6 client is counted by its /64. A new
@@ -297,6 +319,30 @@ which it answers `431`, headers slower than `SWWAF_CLIENT_REQUEST_TIMEOUT`,
 whose connection it closes without an answer, and requests it cannot read at
 all, which it answers itself, mostly with `400`.
 
+### Sending the log to a syslog server
+
+While `SWWAF_LOG_REMOTE_URL` is set, every line `smallwebwaf` writes on stdout,
+request lines and its own, is also sent to that syslog server, as the message of
+an RFC 5424 record: one record to a datagram over UDP, and over TCP and TLS each
+record after its length in bytes and a space. A record gives the facility
+`SWWAF_LOG_REMOTE_FACILITY` names, the severity informational, the time the line
+was written, in the same form as a request line's `time`, the host's name, and
+the app name `SWWAF_LOG_REMOTE_APP_NAME` gives. stdout is unchanged.
+
+The lines wait in a buffer of `SWWAF_LOG_REMOTE_BUFFER` lines and are sent from
+there, so a server that is slow or cannot be reached never holds up a request or
+stdout. When the buffer is full, its oldest line is dropped to make room. A line
+whose sending fails is dropped too, and the connection is made again at once. A
+failed attempt to connect is logged and followed by the next a second later,
+twice as long after each further failure up to a minute, and a second again once
+a connection is made. UDP gives no sign of what arrives, and over TCP and TLS a
+line sent on a connection the server has just closed can be lost before a
+failure shows; such a loss is not counted.
+
+As `smallwebwaf` stops, it sends the lines still waiting, on the connection open
+or a new one, for at most two seconds, and gives up the rest; stdout has carried
+them.
+
 ## State files
 
 `smallwebwaf` keeps its state in memory and a copy of it in three JSON files in
@@ -414,6 +460,11 @@ other request. No metric carries a client's address.
   `smallwebwaf_state_file_edits_taken_in_total`: your edits taken in, and
   `smallwebwaf_state_file_edits_set_aside_total`: those renamed to `<name>.bad`
   because they would stop the start.
+- While `SWWAF_LOG_REMOTE_URL` is set,
+  `smallwebwaf_remote_log_lines_sent_total`: the lines sent to it;
+  `smallwebwaf_remote_log_lines_dropped_total`: those dropped, from a full
+  buffer or because their sending failed; and
+  `smallwebwaf_remote_log_buffer_depth`: those waiting in the buffer.
 - Go's own `go_` metrics and the process's `process_` metrics.
 
 The requests Go's HTTP server ends before `smallwebwaf` sees them (see "Request
@@ -719,6 +770,10 @@ addresses are never sent to GeoJS.
   one while running, and writes them when they are due and at the stop.
 - `internal/requestlog`: the lines on stdout: the request log line and the
   process's own messages.
+- `internal/remotelog`: sends the lines on stdout to `SWWAF_LOG_REMOTE_URL`,
+  each as a syslog record, from a buffer of its own. It is written with the
+  standard library alone, whose `log/syslog` writes only the older syslog
+  format.
 - `Dockerfile`: the lint and test phases, then the image, whose last stage
   installs Ubuntu's packages, nixpkgs, `runsvinit` and `smallwebwaf`, with
   `share/smallwebwaf.run` as runit's `run` script for `smallwebwaf`.