Serve Prometheus metrics behind SWWAF_METRICS_TOKEN (closes #23)

GET /_smallwebwaf/metrics answers in the Prometheus text format for a
request carrying SWWAF_METRICS_TOKEN, 401 without it and 404 while it is
unset. Every request under /_smallwebwaf/ but the health check now goes
through the checks and is answered where it would be forwarded, 404 for
any path but the metrics, so none reaches the app. SWWAF_METRICS_TOP_N
bounds the series by country, the rest counted as other.

Judgement call: a request answered at smallwebwaf's own endpoints is
neither forwarded nor refused in the client's history.
Deviation: go.mod and go.sum written by hand from the module proxy and
sum.golang.org, as go runs only through make.
Deviation: no metrics yet for state files read again after an edit or
edits set aside; that work is not merged.

Model: opus-5-5

README.md

@@ -13,19 +13,20 @@ JSON log line for every request.
 
 Status: the first two milestones are built
 (https://git.eeqj.de/sneak/smallwebwaf/issues/13 and
-https://git.eeqj.de/sneak/smallwebwaf/issues/14), and so are four parts of
+https://git.eeqj.de/sneak/smallwebwaf/issues/14), and so are five parts of
 milestone 3: the static lists, the bans that broken rate limits lead to and the
-JSON state files, which come next in the build order, and the header size and
-the idle time as settings, which come last in it. `smallwebwaf` passes each
-request to the app and the app's answer back, unchanged, within its timeouts and
-size limits, works out each client's address, bans a client that sends too many
-requests, refuses a client that comes from a country you refuse or from a
-network you refuse, lets the networks you choose through, keeps its bans, each
-client's counters and history, and GeoJS's answers in JSON files across
-restarts, and writes a JSON log line for every request. It comes as the image
-the app's own image is built on. The rest of the design comes after that, in the
-order of the build order in [`SPEC.md`](SPEC.md). The survey of existing tools
-that led to the design is in [`EVALUATION.md`](EVALUATION.md).
+JSON state files, which come next in the build order, and the metrics endpoint
+and the header size and the idle time as settings, which come last in it.
+`smallwebwaf` passes each request to the app and the app's answer back,
+unchanged, within its timeouts and size limits, works out each client's address,
+bans a client that sends too many requests, refuses a client that comes from a
+country you refuse or from a network you refuse, lets the networks you choose
+through, keeps its bans, each client's counters and history, and GeoJS's answers
+in JSON files across restarts, writes a JSON log line for every request, and
+serves Prometheus metrics to a scraper that holds the metrics token. It comes as
+the image the app's own image is built on. The rest of the design comes after
+that, in the order of the build order in [`SPEC.md`](SPEC.md). The survey of
+existing tools that led to the design is in [`EVALUATION.md`](EVALUATION.md).
 
 ## Getting started
 
@@ -119,6 +120,14 @@ in `bin/state` unless `SWWAF_STATE_DIR` is set.
   limits; the country lists and bans still apply to it.
 - Answers `GET /_smallwebwaf/healthz` itself with `200` and `ok`, before any
   check and without asking the app, for the image's health check.
+- Answers `GET /_smallwebwaf/metrics` with its metrics (see "Metrics" below) for
+  a request that carries `SWWAF_METRICS_TOKEN` as
+  `Authorization: Bearer <token>`, and with `401` for one that does not. While
+  the token is unset the metrics answer `404`, as does any other request under
+  `/_smallwebwaf/`. Unlike the health check, such a request goes through every
+  check any other request goes through, and is answered where another would be
+  passed to the app: a banned client stays refused, and each counts toward the
+  client's rate limits. None of them reaches the app.
 - Writes a line in the request log for each request (see "Request log" below).
 
 ## Settings
@@ -194,6 +203,12 @@ it, and the effective settings are logged at start.
   `bans.json` is written, with every ban made in between.
 - `SWWAF_STATE_COUNTER_INTERVAL` (default `15m`): how often every state file is
   written.
+- `SWWAF_METRICS_TOKEN` (default unset): the token a scraper sends for the
+  metrics, a long random value. While it is unset the metrics are off; one
+  shorter than 32 characters stops the start. The settings logged at start show
+  `********` in its place.
+- `SWWAF_METRICS_TOP_N` (default `50`): how many countries get series of their
+  own in the metrics by country; the others are counted as `other`.
 
 Durations are in Go's syntax, with `d` for days (`90s`, `15m`, `7d`). Sizes are
 bytes, with an optional `K`, `M` or `G`, which are powers of 1024 (`1K` is 1024
@@ -203,8 +218,8 @@ ISO 3166-1 assigns today, and `xk` for Kosovo, in either case (`de` and `DE` are
 the same); any other code, such as `nk` (North Korea is `kp`) or the withdrawn
 `su`, stops the start, and so does a code on both country lists. `off` switches
 a timeout, a size limit or a rate limit off;
-`SWWAF_CLIENT_REQUEST_HEADER_MAX_BYTES`, the ban settings and the state settings
-cannot be off.
+`SWWAF_CLIENT_REQUEST_HEADER_MAX_BYTES`, the ban settings, the state settings
+and `SWWAF_METRICS_TOP_N` cannot be off.
 
 Several limits are fixed rather than settings. At most 20,000 clients are kept,
 with their counters and history, and an IPv6 client is counted by its /64. A new
@@ -268,10 +283,10 @@ entries by client address, with times in UTC.
   `expires` is `null`.
 - `clients.json`: each client's two buckets in the minute, the hour and the day,
   and its history: when it was first and last seen, its country as last looked
-  up and when, its requests, how many were forwarded and how many refused, the
-  body bytes in each direction, its responses by status class and its offences
-  by kind. Each client is on a line of its own, so `grep` shows everything about
-  one.
+  up and when, its requests, how many were forwarded and how many refused (one
+  `smallwebwaf` answered at its own endpoints is neither), the body bytes in
+  each direction, its responses by status class and its offences by kind. Each
+  client is on a line of its own, so `grep` shows everything about one.
 - `lookups.json`: GeoJS's answers, one to a line, with when GeoJS gave each and
   when it was last used.
 
@@ -299,6 +314,48 @@ answer's `client`, `country`, which is `""` for a client GeoJS cannot place, or
 write: taking it in comes with https://git.eeqj.de/sneak/smallwebwaf/issues/68.
 The AS number and AS name come with their lookup.
 
+## Metrics
+
+`GET /_smallwebwaf/metrics` answers with the metrics in the Prometheus text
+format, for a scraper that sends `SWWAF_METRICS_TOKEN`, through traefik like any
+other request. No metric carries a client's address.
+
+- `smallwebwaf_requests_total`, `smallwebwaf_request_bytes_total` and
+  `smallwebwaf_response_bytes_total`: requests, and their body bytes each way,
+  by `status_class`, such as `2xx`, or `none` when nothing was sent, and by
+  `action`, as the request log names it.
+- `smallwebwaf_request_duration_seconds`: how long requests took, and
+  `smallwebwaf_upstream_duration_seconds`: how long those passed to the app took
+  from then on, as histograms; `smallwebwaf_requests_in_flight`: the requests
+  under way.
+- `smallwebwaf_rate_limit_hits_total` by `window`,
+  `smallwebwaf_size_and_time_limit_hits_total` by `limit`, the setting whose
+  limit was passed, `smallwebwaf_offences_total` by `kind`, and
+  `smallwebwaf_bans_made_total` by `cause`; `smallwebwaf_active_bans` and
+  `smallwebwaf_permanent_bans`.
+- `smallwebwaf_country_requests_total`,
+  `smallwebwaf_country_request_bytes_total`,
+  `smallwebwaf_country_response_bytes_total`, and
+  `smallwebwaf_country_list_refusals_total`, the requests the country lists
+  refused, by `country`, for the requests whose client's country is known. The
+  `SWWAF_METRICS_TOP_N` countries with the most requests since the start have
+  series of their own, and the others are counted as `other`. A country that
+  drops out of them loses its series, and its later requests count as `other`;
+  one that comes into them gets a series that counts from then on.
+- `smallwebwaf_geojs_requests_total`, `smallwebwaf_geojs_failures_total`, and
+  `smallwebwaf_geojs_unanswered_total`: the requests whose client counted as
+  coming from an unknown country because GeoJS had not answered in time.
+- `smallwebwaf_tracked_clients`: the clients in the table of clients.
+- `smallwebwaf_state_file_writes_total`,
+  `smallwebwaf_state_file_write_failures_total`,
+  `smallwebwaf_state_file_last_write_timestamp_seconds` and
+  `smallwebwaf_state_file_size_bytes`, by `file`.
+- Go's own `go_` metrics and the process's `process_` metrics.
+
+The requests Go's HTTP server ends before `smallwebwaf` sees them (see "Request
+log") are not counted. The metrics of the features still to come, such as the
+rule files, come with them.
+
 ## Why
 
 Small self-hosted sites now receive a great deal of traffic nobody asked for:
@@ -583,7 +640,10 @@ addresses are never sent to GeoJS.
   limits, and writes the request's log line. Its `check` method is where a
   request is refused before anything reaches the app: for `SWWAF_DENY_NETS`, for
   a ban, for the country lists, for a rate limit, which bans the client, and for
-  an announced body over the size limit.
+  an announced body over the size limit. A request under `/_smallwebwaf/` that
+  `check` lets through is answered by `answerAdmin` instead of reaching the app.
+- `internal/metrics`: the metrics, counted as the other parts tell it what
+  happened, and served in the Prometheus text format.
 - `internal/bans`: the ban ledger: each netblock's bans with their notes, how
   long a new ban lasts, and which ban is dropped when `SWWAF_MAX_BANS` are held.
 - `internal/lookup`: looks up each client's country through GeoJS, and keeps the
@@ -602,8 +662,9 @@ addresses are never sent to GeoJS.
 
 Besides the Go standard library, `github.com/hashicorp/golang-lru/v2` keeps the
 table of clients to 20,000, the GeoJS answers to 100,000 and the banned
-netblocks to `SWWAF_MAX_BANS`, dropping the least recently seen. The country
-codes are the list in `internal/config/config.go`.
+netblocks to `SWWAF_MAX_BANS`, dropping the least recently seen, and
+`github.com/prometheus/client_golang` keeps the metrics and serves them. The
+country codes are the list in `internal/config/config.go`.
 
 ## Entrypoints
 
@@ -647,9 +708,9 @@ so that they run in minimal containers.
 ## TODO
 
 - The rest of milestone 3, from taking in an admin's edits to the state files
-  (https://git.eeqj.de/sneak/smallwebwaf/issues/68) up to the metrics endpoint,
-  and the rest of the design, in the order of the build order in
-  [`SPEC.md`](SPEC.md).
+  (https://git.eeqj.de/sneak/smallwebwaf/issues/68) up to the rest of the
+  request log's fields, and the rest of the design, in the order of the build
+  order in [`SPEC.md`](SPEC.md).
 
 ## Documents