Send every log line to a syslog server as well (closes #28)

With SWWAF_LOG_REMOTE_URL set (syslog+udp, syslog+tcp or syslog+tls),
every line on stdout is also sent as the message of an RFC 5424 record,
octet-counted over TCP and TLS, from a bounded buffer that drops its
oldest line when full, so a slow or unreachable server holds up nothing.
Failed connections are retried with backoff; lines sent, dropped and
waiting are metrics. At a stop the lines still waiting get at most two
seconds. SWWAF_LOG_REMOTE_APP_NAME defaults to SWWAF_INSTANCE_NAME; while
sending, an app name RFC 5424 does not allow stops the start. Standard
library only: log/syslog writes only the older format.

Model: opus-5-5

README.md

@@ -19,22 +19,23 @@ JSON state files with your edits taken in while it runs and the paths the rate
 limits do not count, which come next in the build order, `observe` mode and the
 rest of the request log's fields, which come a little later, and the metrics
 endpoint and the header size and the idle time as settings, which come last in
-it. So are the rule files, the first part of the stage after it, with the bans
-for a clear sign of attack. `smallwebwaf` passes each request to the app and the
-app's answer back, unchanged, within its timeouts and size limits, works out
-each client's address, bans a client that sends too many requests, not counting
-those for the paths you choose, refuses a client that comes from a country you
-refuse or from a network you refuse, lets the networks you choose through,
-checks each request against the rule files and bans a client whose request is a
-clear sign of attack, keeps its bans, each client's counters and history, and
-GeoJS's answers in JSON files across restarts, takes in your edits of those
-files and of the rule files while it runs, writes a JSON log line for every
-request, serves Prometheus metrics to a scraper that holds the metrics token,
-and in `observe` mode passes on the requests it would refuse, logging what it
-would have done with them. It comes as the image the app's own image is built
-on. The rest of the design comes after that, in the order of the build order in
-[`SPEC.md`](SPEC.md). The survey of existing tools that led to the design is in
-[`EVALUATION.md`](EVALUATION.md).
+it. So are two parts of the stage after it: the rule files, the first part, with
+the bans for a clear sign of attack, and remote log sending. `smallwebwaf`
+passes each request to the app and the app's answer back, unchanged, within its
+timeouts and size limits, works out each client's address, bans a client that
+sends too many requests, not counting those for the paths you choose, refuses a
+client that comes from a country you refuse or from a network you refuse, lets
+the networks you choose through, checks each request against the rule files and
+bans a client whose request is a clear sign of attack, keeps its bans, each
+client's counters and history, and GeoJS's answers in JSON files across
+restarts, takes in your edits of those files and of the rule files while it
+runs, writes a JSON log line for every request, sends its log lines to a syslog
+server too if you name one, serves Prometheus metrics to a scraper that holds
+the metrics token, and in `observe` mode passes on the requests it would refuse,
+logging what it would have done with them. It comes as the image the app's own
+image is built on. The rest of the design comes after that, in the order of the
+build order in [`SPEC.md`](SPEC.md). The survey of existing tools that led to
+the design is in [`EVALUATION.md`](EVALUATION.md).
 
 ## Getting started
 
@@ -171,6 +172,9 @@ in `bin/state` unless `SWWAF_STATE_DIR` is set, and the default rule file of
   passed to the app: a banned client stays refused, and each counts toward the
   client's rate limits. None of them reaches the app.
 - Writes a line in the request log for each request (see "Request log" below).
+- Sends every line it writes on stdout to a syslog server as well, while
+  `SWWAF_LOG_REMOTE_URL` names one (see "Sending the log to a syslog server"
+  below).
 
 ## Settings
 
@@ -288,6 +292,23 @@ it, and the effective settings are logged at start.
   rule files. A directory that does not exist stops the start.
 - `SWWAF_RULES_ENABLED` (default `true`): `false` reads no rule file, and checks
   no request against one.
+- `SWWAF_LOG_REMOTE_URL` (default unset): a syslog server that every line on
+  stdout is also sent to, as `syslog+udp://`, `syslog+tcp://` or `syslog+tls://`
+  with a host and a port, such as `syslog+tls://logs.example:6514`. Unset or
+  empty, nothing is sent.
+- `SWWAF_LOG_REMOTE_TLS_CA_FILE` (default unset): a file of PEM certificates,
+  which the certificate of a `syslog+tls` server must chain to instead of the
+  host's own. A file that cannot be read or holds no certificate stops the
+  start.
+- `SWWAF_LOG_REMOTE_BUFFER` (default `10000`): the most lines held while they
+  wait to be sent.
+- `SWWAF_LOG_REMOTE_FACILITY` (default `local0`): the syslog facility the lines
+  are sent with: `kern`, `user`, `mail`, `daemon`, `auth`, `syslog`, `lpr`,
+  `news`, `uucp`, `cron`, `authpriv`, `ftp`, or `local0` to `local7`.
+- `SWWAF_LOG_REMOTE_APP_NAME` (default `SWWAF_INSTANCE_NAME`): the app name the
+  lines are sent with, 1 to 48 printable ASCII characters without a space. While
+  `SWWAF_LOG_REMOTE_URL` is set, an `SWWAF_INSTANCE_NAME` that is not such a
+  name stops the start too, unless this setting gives one that is.
 
 Durations are in Go's syntax, with `d` for days (`90s`, `15m`, `7d`). Sizes are
 bytes, with an optional `K`, `M` or `G`, which are powers of 1024 (`1K` is 1024
@@ -297,8 +318,8 @@ ISO 3166-1 assigns today, and `xk` for Kosovo, in either case (`de` and `DE` are
 the same); any other code, such as `nk` (North Korea is `kp`) or the withdrawn
 `su`, stops the start, and so does a code on both country lists. `off` switches
 a timeout, a size limit or a rate limit off;
-`SWWAF_CLIENT_REQUEST_HEADER_MAX_BYTES`, the ban settings, the state settings
-and `SWWAF_METRICS_TOP_N` cannot be off.
+`SWWAF_CLIENT_REQUEST_HEADER_MAX_BYTES`, the ban settings, the state settings,
+`SWWAF_METRICS_TOP_N` and `SWWAF_LOG_REMOTE_BUFFER` cannot be off.
 
 Several limits are fixed rather than settings. At most 20,000 clients are kept,
 with their counters and history, and an IPv6 client is counted by its /64. A new
@@ -404,6 +425,32 @@ which it answers `431`, headers slower than `SWWAF_CLIENT_REQUEST_TIMEOUT`,
 whose connection it closes without an answer, and requests it cannot read at
 all, which it answers itself, mostly with `400`.
 
+### Sending the log to a syslog server
+
+While `SWWAF_LOG_REMOTE_URL` is set, every line `smallwebwaf` writes on stdout,
+request lines and its own, is also sent to that syslog server, as the message of
+an RFC 5424 record: one record to a datagram over UDP, and over TCP and TLS each
+record after its length in bytes and a space. A record gives the facility
+`SWWAF_LOG_REMOTE_FACILITY` names, the severity informational, the time the line
+was written, in the same form as a request line's `time`, the host's name, and
+the app name `SWWAF_LOG_REMOTE_APP_NAME` gives. stdout is unchanged.
+
+The lines wait in a buffer of `SWWAF_LOG_REMOTE_BUFFER` lines and are sent from
+there, so a server that is slow or cannot be reached never holds up a request or
+stdout. When the buffer is full, its oldest line is dropped to make room. A line
+whose sending fails is dropped too, and the connection closed. That failure,
+like a failed attempt to connect, is logged and followed by the next attempt to
+connect a second later, twice as long after each further failure up to a minute,
+and a second again after a connection that stayed up for a minute before it
+failed. A line too long for one UDP datagram is dropped alone, with no wait and
+nothing logged. UDP gives no sign of what arrives, and over TCP and TLS a line
+sent on a connection the server has just closed can be lost before a failure
+shows; such a loss is not counted.
+
+As `smallwebwaf` stops, it sends the lines still waiting, on the connection open
+or a new one, for at most two seconds, and gives up the rest; stdout has carried
+them.
+
 ## State files
 
 `smallwebwaf` keeps its state in memory and a copy of it in three JSON files in
@@ -604,6 +651,11 @@ other request. No metric carries a client's address.
   `smallwebwaf_state_file_edits_taken_in_total`: your edits taken in, and
   `smallwebwaf_state_file_edits_set_aside_total`: those renamed to `<name>.bad`
   because they would stop the start.
+- While `SWWAF_LOG_REMOTE_URL` is set,
+  `smallwebwaf_remote_log_lines_sent_total`: the lines sent to it;
+  `smallwebwaf_remote_log_lines_dropped_total`: those dropped, from a full
+  buffer or because their sending failed; and
+  `smallwebwaf_remote_log_buffer_depth`: those waiting in the buffer.
 - Go's own `go_` metrics and the process's `process_` metrics.
 
 The requests Go's HTTP server ends before `smallwebwaf` sees them (see "Request
@@ -913,6 +965,10 @@ addresses are never sent to GeoJS.
   one while running, and writes them when they are due and at the stop.
 - `internal/requestlog`: the lines on stdout: the request log line and the
   process's own messages.
+- `internal/remotelog`: sends the lines on stdout to `SWWAF_LOG_REMOTE_URL`,
+  each as a syslog record, from a buffer of its own. It is written with the
+  standard library alone, whose `log/syslog` writes only the older syslog
+  format.
 - `Dockerfile`: the lint and test phases, then the image, whose last stage
   installs Ubuntu's packages, nixpkgs, `runsvinit` and `smallwebwaf`, with
   `share/smallwebwaf.run` as runit's `run` script for `smallwebwaf` and