Look clients up in the IPinfo Lite file with SWWAF_LOOKUP_SOURCE=file (closes #22)

SWWAF_LOOKUP_SOURCE=file looks every client up in the file
SWWAF_LOOKUP_DB_PATH names, without GeoJS. file without the path, the
path with another source, or a file that cannot be read stops the start.
The file is read whole into memory, so overwriting it in place cannot
disturb a lookup, and read again 2 seconds after its last change; a
replacement that cannot be read is logged, counted and sent as a
file_error alert, and the old one stays in use. Metrics give when it
was read and the failed reads. Tests write their databases through
internal/lookup/lookuptest.

Deviation: go.mod and go.sum written by hand; go runs only through make.
Judgement call: the 2-second wait, as the rule files have.

Model: opus-5-5

README.md

@@ -23,27 +23,28 @@ and the idle time as settings, which come last in it. So are the four parts of
 the stage after it: the rule files, with the bans for a clear sign of attack,
 the other admin endpoints, alerts to all three destinations, a JSON webhook,
 Slack and ntfy, and remote log sending. So is the first part of the stage after
-that: the AS number and country of every client, looked up through GeoJS.
-`smallwebwaf` passes each request to the app and the app's answer back,
-unchanged, within its timeouts and size limits, works out each client's address,
-looks up its AS number and country unless you switch that off, bans a client
-that sends too many requests, not counting those for the paths you choose,
-refuses a client that comes from a country you refuse or from a network you
-refuse, lets the networks you choose through, checks each request against the
-rule files and bans a client whose request is a clear sign of attack, keeps its
-bans, each client's counters and history, and GeoJS's answers in JSON files
-across restarts, takes in your edits of those files, such as a ban you make,
-keep or lift, and of the rule files while it runs, writes a JSON log line for
-every request, sends its log lines to a syslog server too if you name one, sends
-an alert to a webhook, to Slack and to ntfy, each if you name one, for each ban
-it makes or makes permanent, for GeoJS failing and for a rule file or state file
-with an error, serves Prometheus metrics to a scraper that holds the metrics
-token, lets an admin who holds the admin token list, add and lift bans and ask
-what it knows of a client, and in `observe` mode passes on the requests it would
-refuse, logging what it would have done with them. It comes as the image the
-app's own image is built on. The rest of the design comes after that, in the
-order of the build order in [`SPEC.md`](SPEC.md). The survey of existing tools
-that led to the design is in [`EVALUATION.md`](EVALUATION.md).
+that: the AS number and country of every client, looked up through GeoJS or in
+the IPinfo Lite database file. `smallwebwaf` passes each request to the app and
+the app's answer back, unchanged, within its timeouts and size limits, works out
+each client's address, looks up its AS number and country unless you switch that
+off, bans a client that sends too many requests, not counting those for the
+paths you choose, refuses a client that comes from a country you refuse or from
+a network you refuse, lets the networks you choose through, checks each request
+against the rule files and bans a client whose request is a clear sign of
+attack, keeps its bans, each client's counters and history, and GeoJS's answers
+in JSON files across restarts, takes in your edits of those files, such as a ban
+you make, keep or lift, and of the rule files while it runs, writes a JSON log
+line for every request, sends its log lines to a syslog server too if you name
+one, sends an alert to a webhook, to Slack and to ntfy, each if you name one,
+for each ban it makes or makes permanent, for GeoJS failing, for a rule file or
+state file with an error and for a replacement of the lookup database it cannot
+read, serves Prometheus metrics to a scraper that holds the metrics token, lets
+an admin who holds the admin token list, add and lift bans and ask what it knows
+of a client, and in `observe` mode passes on the requests it would refuse,
+logging what it would have done with them. It comes as the image the app's own
+image is built on. The rest of the design comes after that, in the order of the
+build order in [`SPEC.md`](SPEC.md). The survey of existing tools that led to
+the design is in [`EVALUATION.md`](EVALUATION.md).
 
 ## Getting started
 
@@ -141,15 +142,16 @@ in `bin/state` unless `SWWAF_STATE_DIR` is set, and the default rule file of
   back to zero, and does not make the netblock's next ban for a broken limit
   longer. Its notes give the id and the target of the rule that matched in place
   of the limit.
-- Looks up the AS number and country of every client through GeoJS, after the
+- Looks up the AS number and country of every client through GeoJS, or in the
+  IPinfo Lite database file while `SWWAF_LOOKUP_SOURCE` is `file`, after the
   static lists and bans, unless `SWWAF_LOOKUP_SOURCE` is `off` (see "Country and
   AS number lookup" below), for the request log, the client's history, the notes
-  of its bans, their alerts and the metrics. A request waits for its client's
-  first answer only while a setting acts on it, a country list or
-  `SWWAF_ADD_LOOKUP_HEADERS`. Otherwise it goes on at once, and the answer
-  reaches the client's history and the notes of its bans when it comes, but not
-  the log lines of the requests that went on without it, nor the alerts already
-  raised for those bans.
+  of its bans, their alerts and the metrics. The file answers at once. With
+  GeoJS, a request waits for its client's first answer only while a setting acts
+  on it, a country list or `SWWAF_ADD_LOOKUP_HEADERS`. Otherwise it goes on at
+  once, and the answer reaches the client's history and the notes of its bans
+  when it comes, but not the log lines of the requests that went on without it,
+  nor the alerts already raised for those bans.
 - Refuses a request from a country you refuse with `SWWAF_BAN_RESPONSE`, as soon
   as the client's country is known and before its body is read; such a request
   is not counted for the rate limits. A client on a private, loopback or
@@ -200,12 +202,12 @@ in `bin/state` unless `SWWAF_STATE_DIR` is set, and the default rule file of
   `SWWAF_LOG_REMOTE_URL` names one (see "Sending the log to a syslog server"
   below).
 - Sends an alert for each ban it makes or makes permanent, for GeoJS failing,
-  and for a rule file or state file with an error, holding back repeats and,
-  past an hourly limit, rolling the rest into one summary, to each destination
-  you name: as a JSON object to the webhook `SWWAF_ALERT_WEBHOOK_URL` names, as
-  a message to the Slack incoming webhook `SWWAF_ALERT_SLACK_WEBHOOK_URL` names,
-  and as a message to the ntfy topic `SWWAF_ALERT_NTFY_URL` names (see "Alerts"
-  below).
+  for a rule file or state file with an error, and for a replacement of the
+  lookup database it cannot read, holding back repeats and, past an hourly
+  limit, rolling the rest into one summary, to each destination you name: as a
+  JSON object to the webhook `SWWAF_ALERT_WEBHOOK_URL` names, as a message to
+  the Slack incoming webhook `SWWAF_ALERT_SLACK_WEBHOOK_URL` names, and as a
+  message to the ntfy topic `SWWAF_ALERT_NTFY_URL` names (see "Alerts" below).
 
 ## Settings
 
@@ -278,14 +280,18 @@ effective settings are logged at start.
   other.
 - `SWWAF_LOOKUP_SOURCE` (default `geojs`): where each client's AS number and
   country are looked up: `geojs`, the GeoJS web service, which is then told the
-  address of every new visitor, or `off`, which looks up no client and sends no
-  address to GeoJS. `file`, for the IPinfo Lite database, comes with
-  https://git.eeqj.de/sneak/smallwebwaf/issues/22. With `off`, a country list
-  that is not empty, or `SWWAF_ADD_LOOKUP_HEADERS` set to `true`, stops the
-  start, with a message naming it and `SWWAF_LOOKUP_SOURCE`.
+  address of every new visitor, `file`, the IPinfo Lite database file
+  `SWWAF_LOOKUP_DB_PATH` names, or `off`, which looks up no client and sends no
+  address to GeoJS. With `off`, a country list that is not empty, or
+  `SWWAF_ADD_LOOKUP_HEADERS` set to `true`, stops the start, with a message
+  naming it and `SWWAF_LOOKUP_SOURCE`.
+- `SWWAF_LOOKUP_DB_PATH` (default empty): the IPinfo Lite database file, in its
+  `.mmdb` form, for `SWWAF_LOOKUP_SOURCE=file`. `file` without it, or it with
+  any other `SWWAF_LOOKUP_SOURCE`, the default included, stops the start, with a
+  message naming both.
 - `SWWAF_LOOKUP_TIMEOUT` (default `1s`): how long a request waits for its
-  client's first answer while a setting acts on it, and how long a request to
-  GeoJS may take before it is abandoned.
+  client's first answer from GeoJS while a setting acts on it, and how long a
+  request to GeoJS may take before it is abandoned.
 - `SWWAF_ADD_LOOKUP_HEADERS` (default `false`): `true` passes the app the
   client's AS number, such as `AS64496`, in `X-Client-ASN`, and its country in
   `X-Client-Country`, leaving out one that is unknown. A request then waits for
@@ -475,13 +481,14 @@ which every line has.
   `client_group` is the client as the rate limits count it: its IPv4 address as
   a /32, or the /64 of its IPv6 address.
 - `asn`, `as_name` and `country` are the client's AS number, such as `AS64496`,
-  the name of that AS, and its country, as GeoJS gives them. Each is empty when
-  `SWWAF_LOOKUP_SOURCE` is `off`, for a client in `SWWAF_ALLOW_NETS` or
-  `SWWAF_DENY_NETS`, for a client on a private, loopback or link-local address,
-  when GeoJS has not answered by the time the request went on, and for a request
-  whose client a ban covers, even when the answer is known. `asn` and `as_name`
-  are empty too when GeoJS knows no AS number for the client, which it gives as
-  64512, and `country` when GeoJS cannot place the client.
+  the name of that AS, and its country, as GeoJS or the lookup database gives
+  them. Each is empty when `SWWAF_LOOKUP_SOURCE` is `off`, for a client in
+  `SWWAF_ALLOW_NETS` or `SWWAF_DENY_NETS`, for a client on a private, loopback
+  or link-local address, when GeoJS has not answered by the time the request
+  went on, for a client whose address the lookup database does not hold, and for
+  a request whose client a ban covers, even when the answer is known. `asn` and
+  `as_name` are empty too when GeoJS knows no AS number for the client, which it
+  gives as 64512, and `country` when GeoJS cannot place the client.
 - `content_type` is the request's `Content-Type`, and `content_length` the
   length the request announced for its body, which is left out for none or zero.
 - `request_headers` are the request's headers that `SWWAF_LOG_REQUEST_HEADERS`
@@ -595,8 +602,9 @@ it, as below. An alert is for one of these events, and is sent when
   that a request made permanent.
 - `source_failure`: GeoJS failing or refusing `smallwebwaf`.
 - `file_error`: a rule file edited while it runs that has an error, an edit of a
-  state file set aside as `<name>.bad`, or a state file it could not write while
-  running.
+  state file set aside as `<name>.bad`, a state file it could not write while
+  running, or a replacement of the lookup database it could not read, which it
+  does not use.
 
 The bans you make, in `bans.json` or through the ban endpoints, raise no alert.
 In `observe` mode, a request that would have made a ban, or made one permanent,
@@ -752,11 +760,11 @@ entries by client address, but for the alerts waiting, with times in UTC.
   when you lifted it, and is left out until you do.
 - `clients.json`: each client's two buckets in the minute, the hour and the day,
   and its history: when it was first and last seen, its AS number, AS name and
-  country as last looked up and when GeoJS gave them, its requests, how many
-  were forwarded and how many refused (one `smallwebwaf` answered at its own
-  endpoints is neither, unless it was refused with `401` for a missing or wrong
-  token), the body bytes in each direction, its responses by status class and
-  its offences by kind. Each client is on a line of its own, so `grep` shows
+  country as last looked up and when the lookup gave them, its requests, how
+  many were forwarded and how many refused (one `smallwebwaf` answered at its
+  own endpoints is neither, unless it was refused with `401` for a missing or
+  wrong token), the body bytes in each direction, its responses by status class
+  and its offences by kind. Each client is on a line of its own, so `grep` shows
   everything about one.
 - `lookups.json`: GeoJS's answers, one to a line, each with the client's AS
   number, AS name and country, when GeoJS gave it and when it was last used.
@@ -977,6 +985,11 @@ scraped, and keeps this one as `exported_instance` unless the scrape sets
   the requests that needed their client's answer, for a country list or
   `SWWAF_ADD_LOOKUP_HEADERS`, and went on without it because GeoJS had not given
   it in time.
+- While `SWWAF_LOOKUP_SOURCE` is `file`,
+  `smallwebwaf_lookup_database_last_read_timestamp_seconds`: when the lookup
+  database in use was read; and
+  `smallwebwaf_lookup_database_read_failures_total`: the replacements of it that
+  could not be read.
 - `smallwebwaf_tracked_clients`: the clients in the table of clients.
 - `smallwebwaf_state_file_writes_total`,
   `smallwebwaf_state_file_write_failures_total`,
@@ -1316,21 +1329,27 @@ without an answer.
 
 To keep your visitors' addresses on your own host, set
 `SWWAF_LOOKUP_SOURCE=off`, or use the database file instead of GeoJS:
-`SWWAF_LOOKUP_SOURCE=file` reads the free IPinfo Lite database
-(`ipinfo_lite.mmdb`), and comes with
-https://git.eeqj.de/sneak/smallwebwaf/issues/22. You download the database with
-your own IPinfo account, mount the directory that holds it into the container,
-point `SWWAF_LOOKUP_DB_PATH` at the file and refresh it when you choose;
-`smallwebwaf` never downloads it itself, and reads it again when you replace it.
-It has to be the directory rather than the file itself: docker does not show a
-single mounted file being replaced, so a refresh would go unseen. IPinfo
-releases it under the Creative Commons Attribution-ShareAlike 4.0 International
-License and asks for attribution, in its own words on https://ipinfo.io/lite:
-"The attribution requirements can be met by giving our service credit as your
-data source. Simply place a link to IPinfo on the website, application, or
-social media account that uses our data." Its example of such a credit is a link
-mentioning "IP address data is powered by IPinfo". A service that uses the
-database through `smallwebwaf` should carry that link.
+`SWWAF_LOOKUP_SOURCE=file` looks every client up in the free IPinfo Lite
+database (`ipinfo_lite.mmdb`), at once, with no wait and nothing sent off the
+host. A client whose address it does not hold counts as coming from an unknown
+country. You download the database with your own IPinfo account, mount the
+directory that holds it into the container, point `SWWAF_LOOKUP_DB_PATH` at the
+file and refresh it when you choose; `smallwebwaf` never downloads it itself. It
+reads the whole file into memory at start, and a file that is missing or that it
+cannot read stops the start. It reads the file again once it has gone 2 seconds
+without a change after you replace it, so that a file still being copied in is
+read only once whole, and also 2 seconds after it starts watching, so that a
+file replaced while it started is not missed. A replacement it cannot read is
+logged and sent as a `file_error` alert, and the file read before stays in use.
+It has to be the directory rather than the file itself that you mount: docker
+does not show a single mounted file being replaced, so a refresh would go
+unseen. IPinfo releases it under the Creative Commons Attribution-ShareAlike 4.0
+International License and asks for attribution, in its own words on
+https://ipinfo.io/lite: "The attribution requirements can be met by giving our
+service credit as your data source. Simply place a link to IPinfo on the
+website, application, or social media account that uses our data." Its example
+of such a credit is a link mentioning "IP address data is powered by IPinfo". A
+service that uses the database through `smallwebwaf` should carry that link.
 
 Neither source can place a private address, so a client on one, such as a
 visitor on your local network, another container or your monitoring, has no
@@ -1341,10 +1360,10 @@ addresses are never sent to GeoJS.
 ## How the code is laid out
 
 - `cmd/smallwebwaf`: the binary, which only calls `internal/smallwebwaf`.
-- `internal/smallwebwaf`: the process: it reads the settings, the rule files and
-  the state files, listens, serves requests until `SIGTERM` or `SIGINT`, and
-  stops, writing the state files. Run as `smallwebwaf healthcheck`, it is the
-  image's health check instead.
+- `internal/smallwebwaf`: the process: it reads the settings, the rule files,
+  the lookup database and the state files, listens, serves requests until
+  `SIGTERM` or `SIGINT`, and stops, writing the state files. Run as
+  `smallwebwaf healthcheck`, it is the image's health check instead.
 - `internal/config`: reads the settings, the one place they are read.
 - `internal/proxy`: what happens to each request: it works out the client, runs
   the checks, passes the request to the app and the answer back with the
@@ -1366,7 +1385,9 @@ addresses are never sent to GeoJS.
   tells which of their rules a request matches.
 - `internal/lookup`: looks up each client's AS number and country through GeoJS,
   keeps the answers, and hands each new one to the proxy, which adds it to the
-  client's history and to the notes of its bans.
+  client's history and to the notes of its bans; or in the lookup database,
+  which it reads again when the file is replaced. `internal/lookup/lookuptest`
+  writes lookup databases for the tests.
 - `internal/ratelimit`: the table of clients: counts each client's requests,
   tells when one takes it over a rate limit, and keeps each client's history.
 - `internal/state`: reads the state files at start, takes in an admin's edit of
@@ -1393,7 +1414,10 @@ recently seen, and the banned netblocks in the order they were last seen, from
 which the ledger picks the ban to drop past `SWWAF_MAX_BANS`, and
 `github.com/prometheus/client_golang` keeps the metrics and serves them, and
 `github.com/fsnotify/fsnotify` tells `smallwebwaf` when a state file or a rule
-file is saved. The country codes are the list in `internal/config/config.go`.
+file is saved, or the lookup database replaced, and
+`github.com/oschwald/maxminddb-golang/v2` reads the lookup database, which the
+tests write with `github.com/maxmind/mmdbwriter`. The country codes are the list
+in `internal/config/config.go`.
 
 ## Entrypoints