Serve Prometheus metrics behind SWWAF_METRICS_TOKEN (closes #23)

GET /_smallwebwaf/metrics answers in the Prometheus text format for a
request carrying SWWAF_METRICS_TOKEN, 401 without it and 404 while it is
unset. Every request under /_smallwebwaf/ but the health check now goes
through the checks and is answered where it would be forwarded, 404 for
any path but the metrics, so none reaches the app. In the client's
history a 401 counts as refused, the metrics and the 404s as neither.
SWWAF_METRICS_TOP_N bounds the series by country, the rest counted as
other.

Deviation: go.mod and go.sum written by hand, as go runs only through
make.
Deviation: no metrics yet for state files read again after an edit or
edits set aside; that work is not merged.

Model: opus-5-5

README.md

@@ -13,19 +13,20 @@ JSON log line for every request.
 
 Status: the first two milestones are built
 (https://git.eeqj.de/sneak/smallwebwaf/issues/13 and
-https://git.eeqj.de/sneak/smallwebwaf/issues/14), and so are four parts of
+https://git.eeqj.de/sneak/smallwebwaf/issues/14), and so are five parts of
 milestone 3: the static lists, the bans that broken rate limits lead to and the
-JSON state files, which come next in the build order, and the header size and
-the idle time as settings, which come last in it. `smallwebwaf` passes each
-request to the app and the app's answer back, unchanged, within its timeouts and
-size limits, works out each client's address, bans a client that sends too many
-requests, refuses a client that comes from a country you refuse or from a
-network you refuse, lets the networks you choose through, keeps its bans, each
-client's counters and history, and GeoJS's answers in JSON files across
-restarts, and writes a JSON log line for every request. It comes as the image
-the app's own image is built on. The rest of the design comes after that, in the
-order of the build order in [`SPEC.md`](SPEC.md). The survey of existing tools
-that led to the design is in [`EVALUATION.md`](EVALUATION.md).
+JSON state files, which come next in the build order, and the metrics endpoint
+and the header size and the idle time as settings, which come last in it.
+`smallwebwaf` passes each request to the app and the app's answer back,
+unchanged, within its timeouts and size limits, works out each client's address,
+bans a client that sends too many requests, refuses a client that comes from a
+country you refuse or from a network you refuse, lets the networks you choose
+through, keeps its bans, each client's counters and history, and GeoJS's answers
+in JSON files across restarts, writes a JSON log line for every request, and
+serves Prometheus metrics to a scraper that holds the metrics token. It comes as
+the image the app's own image is built on. The rest of the design comes after
+that, in the order of the build order in [`SPEC.md`](SPEC.md). The survey of
+existing tools that led to the design is in [`EVALUATION.md`](EVALUATION.md).
 
 ## Getting started
 
@@ -119,6 +120,14 @@ in `bin/state` unless `SWWAF_STATE_DIR` is set.
   limits; the country lists and bans still apply to it.
 - Answers `GET /_smallwebwaf/healthz` itself with `200` and `ok`, before any
   check and without asking the app, for the image's health check.
+- Answers `GET /_smallwebwaf/metrics` with its metrics (see "Metrics" below) for
+  a request that carries `SWWAF_METRICS_TOKEN` as
+  `Authorization: Bearer <token>`, and with `401` for one that does not. While
+  the token is unset the metrics answer `404`, as does any other request under
+  `/_smallwebwaf/`. Unlike the health check, such a request goes through every
+  check any other request goes through, and is answered where another would be
+  passed to the app: a banned client stays refused, and each counts toward the
+  client's rate limits. None of them reaches the app.
 - Writes a line in the request log for each request (see "Request log" below).
 
 ## Settings
@@ -194,6 +203,12 @@ it, and the effective settings are logged at start.
   `bans.json` is written, with every ban made in between.
 - `SWWAF_STATE_COUNTER_INTERVAL` (default `15m`): how often every state file is
   written.
+- `SWWAF_METRICS_TOKEN` (default unset): the token a scraper sends for the
+  metrics, a long random value. While it is unset the metrics are off; one
+  shorter than 32 characters stops the start. The settings logged at start show
+  `********` in its place.
+- `SWWAF_METRICS_TOP_N` (default `50`): how many countries get series of their
+  own in the metrics by country; the others are counted as `other`.
 
 Durations are in Go's syntax, with `d` for days (`90s`, `15m`, `7d`). Sizes are
 bytes, with an optional `K`, `M` or `G`, which are powers of 1024 (`1K` is 1024
@@ -203,8 +218,8 @@ ISO 3166-1 assigns today, and `xk` for Kosovo, in either case (`de` and `DE` are
 the same); any other code, such as `nk` (North Korea is `kp`) or the withdrawn
 `su`, stops the start, and so does a code on both country lists. `off` switches
 a timeout, a size limit or a rate limit off;
-`SWWAF_CLIENT_REQUEST_HEADER_MAX_BYTES`, the ban settings and the state settings
-cannot be off.
+`SWWAF_CLIENT_REQUEST_HEADER_MAX_BYTES`, the ban settings, the state settings
+and `SWWAF_METRICS_TOP_N` cannot be off.
 
 Several limits are fixed rather than settings. At most 20,000 clients are kept,
 with their counters and history, and an IPv6 client is counted by its /64. A new
@@ -268,10 +283,11 @@ entries by client address, with times in UTC.
   `expires` is `null`.
 - `clients.json`: each client's two buckets in the minute, the hour and the day,
   and its history: when it was first and last seen, its country as last looked
-  up and when, its requests, how many were forwarded and how many refused, the
-  body bytes in each direction, its responses by status class and its offences
-  by kind. Each client is on a line of its own, so `grep` shows everything about
-  one.
+  up and when, its requests, how many were forwarded and how many refused (one
+  `smallwebwaf` answered at its own endpoints is neither, unless it was refused
+  with `401` for a missing or wrong token), the body bytes in each direction,
+  its responses by status class and its offences by kind. Each client is on a
+  line of its own, so `grep` shows everything about one.
 - `lookups.json`: GeoJS's answers, one to a line, with when GeoJS gave each and
   when it was last used.
 
@@ -299,6 +315,50 @@ answer's `client`, `country`, which is `""` for a client GeoJS cannot place, or
 write: taking it in comes with https://git.eeqj.de/sneak/smallwebwaf/issues/68.
 The AS number and AS name come with their lookup.
 
+## Metrics
+
+`GET /_smallwebwaf/metrics` answers with the metrics in the Prometheus text
+format, for a scraper that sends `SWWAF_METRICS_TOKEN`, through traefik like any
+other request. No metric carries a client's address.
+
+- `smallwebwaf_requests_total`, `smallwebwaf_request_bytes_total` and
+  `smallwebwaf_response_bytes_total`: requests, and their body bytes each way,
+  by `status_class`, such as `2xx`, or `none` when nothing was sent, and by
+  `action`, as the request log names it.
+- `smallwebwaf_request_duration_seconds`: how long requests took, and
+  `smallwebwaf_upstream_duration_seconds`: how long those passed to the app took
+  from then on, as histograms; `smallwebwaf_requests_in_flight`: the requests
+  under way.
+- `smallwebwaf_rate_limit_hits_total` by `window`,
+  `smallwebwaf_size_and_time_limit_hits_total` by `limit`, the setting whose
+  limit was passed, `smallwebwaf_offences_total` by `kind`, and
+  `smallwebwaf_bans_made_total` by `cause`; `smallwebwaf_active_bans` and
+  `smallwebwaf_permanent_bans`.
+- `smallwebwaf_country_requests_total`,
+  `smallwebwaf_country_request_bytes_total`,
+  `smallwebwaf_country_response_bytes_total`, and
+  `smallwebwaf_country_list_refusals_total`, the requests the country lists
+  refused, by `country`, for the requests whose client's country is known. The
+  `SWWAF_METRICS_TOP_N` countries with the most requests since the start have
+  series of their own, and the others are counted as `other`. A country that
+  drops out of them loses its series, and its later requests count as `other`;
+  one that comes into them gets a series that counts from then on.
+- `smallwebwaf_geojs_requests_total`: the requests to GeoJS;
+  `smallwebwaf_geojs_failures_total`: those that failed, an answer that leaves
+  out an address asked about included; and `smallwebwaf_geojs_unanswered_total`:
+  the requests whose client counted as coming from an unknown country because
+  GeoJS had not answered about it in time.
+- `smallwebwaf_tracked_clients`: the clients in the table of clients.
+- `smallwebwaf_state_file_writes_total`,
+  `smallwebwaf_state_file_write_failures_total`,
+  `smallwebwaf_state_file_last_write_timestamp_seconds` and
+  `smallwebwaf_state_file_size_bytes`, by `file`.
+- Go's own `go_` metrics and the process's `process_` metrics.
+
+The requests Go's HTTP server ends before `smallwebwaf` sees them (see "Request
+log") are not counted. The metrics of the features still to come, such as the
+rule files, come with them.
+
 ## Why
 
 Small self-hosted sites now receive a great deal of traffic nobody asked for:
@@ -583,7 +643,10 @@ addresses are never sent to GeoJS.
   limits, and writes the request's log line. Its `check` method is where a
   request is refused before anything reaches the app: for `SWWAF_DENY_NETS`, for
   a ban, for the country lists, for a rate limit, which bans the client, and for
-  an announced body over the size limit.
+  an announced body over the size limit. A request under `/_smallwebwaf/` that
+  `check` lets through is answered by `answerAdmin` instead of reaching the app.
+- `internal/metrics`: the metrics, counted as the other parts tell it what
+  happened, and served in the Prometheus text format.
 - `internal/bans`: the ban ledger: each netblock's bans with their notes, how
   long a new ban lasts, and which ban is dropped when `SWWAF_MAX_BANS` are held.
 - `internal/lookup`: looks up each client's country through GeoJS, and keeps the
@@ -602,8 +665,9 @@ addresses are never sent to GeoJS.
 
 Besides the Go standard library, `github.com/hashicorp/golang-lru/v2` keeps the
 table of clients to 20,000, the GeoJS answers to 100,000 and the banned
-netblocks to `SWWAF_MAX_BANS`, dropping the least recently seen. The country
-codes are the list in `internal/config/config.go`.
+netblocks to `SWWAF_MAX_BANS`, dropping the least recently seen, and
+`github.com/prometheus/client_golang` keeps the metrics and serves them. The
+country codes are the list in `internal/config/config.go`.
 
 ## Entrypoints
 
@@ -647,9 +711,9 @@ so that they run in minimal containers.
 ## TODO
 
 - The rest of milestone 3, from taking in an admin's edits to the state files
-  (https://git.eeqj.de/sneak/smallwebwaf/issues/68) up to the metrics endpoint,
-  and the rest of the design, in the order of the build order in
-  [`SPEC.md`](SPEC.md).
+  (https://git.eeqj.de/sneak/smallwebwaf/issues/68) up to the rest of the
+  request log's fields, and the rest of the design, in the order of the build
+  order in [`SPEC.md`](SPEC.md).
 
 ## Documents