Read SWWAF_IPV6_GROUP_PREFIX, SWWAF_MAX_TRACKED_CLIENTS and SWWAF_LOG_LEVEL (closes #112)

The IPv6 group that is one client, the size of the table of clients and
the level of the process's own lines become settings. clientGroup reads
the group length from them, so limits, bans, history, lookups, AbuseIPDB
scores and per-client anomaly counters all follow it; ratelimit.New
takes the table size; the process logger takes the level once the
settings are read, and request lines, written apart from it, are never
held back.

Judgement call: SWWAF_IPV6_GROUP_PREFIX accepts 32 to 128, the issue's example range.
Judgement call: the log level test picks a free port by listening and closing, since at warn no starting line gives the address.

Model: opus-5-5

README.md

@@ -116,11 +116,12 @@ in `bin/state` unless `SWWAF_STATE_DIR` is set, and the default rule file of
   `SWWAF_RATE_LIMIT_EXEMPT_PATHS`, as that setting below describes, is neither
   counted nor refused by the rate limits; the static lists, bans, the country
   lists and the rule files still apply to it. A client is one IPv4 address, or
-  one IPv6 /64, since one abuser usually holds a whole /64. Each window is
+  one IPv6 group, the netblock of `SWWAF_IPV6_GROUP_PREFIX` its address is in, a
+  /64 by default, since one abuser usually holds a whole /64. Each window is
   counted in two fixed buckets, the earlier one weighted by how much of it the
-  window still covers. At most 20,000 clients are kept, the least recently seen
-  dropped first, with their history, and a restart gives no client a fresh
-  allowance (see "State files" below).
+  window still covers. At most `SWWAF_MAX_TRACKED_CLIENTS` clients are kept,
+  20,000 by default, the least recently seen dropped first, with their history,
+  and a restart gives no client a fresh allowance (see "State files" below).
 - Counts each client's bytes over a minute, an hour and a day, in the same way:
   once a request passed to the app has ended, the body bytes of its answer, of
   the request, or of both, as `SWWAF_BYTES_COUNT` says. For a WebSocket, or any
@@ -153,8 +154,8 @@ in `bin/state` unless `SWWAF_STATE_DIR` is set, and the default rule file of
   broken again within a day of a ban ending bans for three times as long as that
   ban, so 1, 3, 9, 27 and 81 hours; a ban that would last longer than seven days
   is permanent instead. A ban covers the client's netblock: its IPv4 address, or
-  the netblock around it that `SWWAF_BAN_SCOPE_V4_PREFIX` sets, or its IPv6 /64.
-  While it lasts, every request from the netblock is refused with
+  the netblock around it that `SWWAF_BAN_SCOPE_V4_PREFIX` sets, or its IPv6
+  group. While it lasts, every request from the netblock is refused with
   `SWWAF_BAN_RESPONSE` after the static lists and before the country lists, so
   the client is not looked up, and is not counted for the rate limits. A ban
   sets the client's counters back to zero. Each ban carries notes for deciding
@@ -333,6 +334,18 @@ effective settings are logged at start.
 - `SWWAF_TRUSTED_PROXIES` (default `10.0.0.0/8,172.16.0.0/12,192.168.0.0/16`,
   the private address ranges): the netblocks whose `X-Forwarded-For` is
   believed. A list given replaces the default; set but empty, it trusts nothing.
+- `SWWAF_IPV6_GROUP_PREFIX` (default `64`): the length of an IPv6 client's
+  group, the netblock that is one client, from 32 to 128, since a shorter one
+  would make one client of the customers of several providers. The rate limits,
+  the byte limits, bans, the table of clients, the lookups, AbuseIPDB's scores
+  and the anomaly thresholds per client all take an IPv6 client as its group,
+  and `client_group` gives it. After it changes, each IPv6 client starts afresh:
+  what was kept of it under its earlier group, its counts, history, GeoJS answer
+  and AbuseIPDB score, is not used for it, while each ban keeps refusing its
+  netblock until it ends.
+- `SWWAF_MAX_TRACKED_CLIENTS` (default `20000`): the most clients kept in memory
+  and in `clients.json`, with their counters and history, a whole number above
+  zero. Past it, the least recently seen is dropped first.
 - `SWWAF_CLIENT_REQUEST_TIMEOUT` (default `60s`): how long a client may take to
   send its request line and headers, and then, from the end of the headers, its
   body.
@@ -510,7 +523,7 @@ effective settings are logged at start.
   kept, past, active and permanent. The bans you make or keep are kept besides.
 - `SWWAF_BAN_SCOPE_V4_PREFIX` (default `32`): the length of the netblock around
   an IPv4 client that a ban covers, such as `24` to ban the surrounding /24. An
-  IPv6 ban covers the client's /64.
+  IPv6 ban covers the client's group, as `SWWAF_IPV6_GROUP_PREFIX` sets it.
 - `SWWAF_STATE_DIR` (default `/var/lib/smallwebwaf`): the directory of the state
   files, an absolute path. A directory `smallwebwaf` cannot write stops the
   start.
@@ -525,6 +538,10 @@ effective settings are logged at start.
   (see "Request log" below). An entry naming `Host` or `Transfer-Encoding` stops
   the start, since Go's HTTP server takes both out of the request; the request's
   host is the field `host`.
+- `SWWAF_LOG_LEVEL` (default `info`): the least severe of `smallwebwaf`'s own
+  messages that are written, on stdout and to the syslog server: `debug`,
+  `info`, `warn` or `error`. No message is at `debug` yet, so it writes what
+  `info` does. It holds back no line of the request log.
 - `SWWAF_ADMIN_TOKEN` (default unset): the token an admin sends for the ban
   endpoints and `/_smallwebwaf/clients/<ip>` (see "Admin endpoints" below), a
   long random value. While it is unset they are off; one shorter than 32
@@ -637,7 +654,8 @@ Percentages are whole numbers from 0 to 100, and an entry of a list of them is
 an AS number or a country, `:` and a percentage; an AS number or a country
 listed twice in one of them stops the start. `off` switches a timeout, a size
 limit, a rate limit, a byte limit, an anomaly threshold, `SWWAF_ALERT_COOLDOWN`
-or `SWWAF_ALERT_MAX_PER_HOUR` off; `SWWAF_CLIENT_REQUEST_HEADER_MAX_BYTES`,
+or `SWWAF_ALERT_MAX_PER_HOUR` off; `SWWAF_IPV6_GROUP_PREFIX`,
+`SWWAF_MAX_TRACKED_CLIENTS`, `SWWAF_CLIENT_REQUEST_HEADER_MAX_BYTES`,
 `SWWAF_LOOKUP_TIMEOUT`, `SWWAF_UNKNOWN_LIMIT_PERCENT`,
 `SWWAF_BLOCKLIST_REFRESH`, `SWWAF_ABUSEIPDB_MIN_SCORE`,
 `SWWAF_ABUSEIPDB_DAILY_BUDGET`, `SWWAF_REPUTATION_CACHE_TTL`,
@@ -645,12 +663,10 @@ or `SWWAF_ALERT_MAX_PER_HOUR` off; `SWWAF_CLIENT_REQUEST_HEADER_MAX_BYTES`,
 `SWWAF_METRICS_TOP_N`, `SWWAF_LOG_REMOTE_BUFFER`, `SWWAF_ANOMALY_NET_V4_PREFIX`
 and `SWWAF_ANOMALY_NET_V6_PREFIX` cannot be off.
 
-Several limits are fixed rather than settings. At most 20,000 clients are kept,
-with their counters and history, and an IPv6 client is counted by its /64. At
-most 100,000 answers from GeoJS are kept, for 7 days each, at most 20,000
-anomaly counters, at most 100,000 verdicts of the DNSBL zones, with at most
-1,000 queries to them under way at once, and at most 100,000 scores of
-AbuseIPDB.
+Several limits are fixed rather than settings. At most 100,000 answers from
+GeoJS are kept, for 7 days each, at most 20,000 anomaly counters, at most
+100,000 verdicts of the DNSBL zones, with at most 1,000 queries to them under
+way at once, and at most 100,000 scores of AbuseIPDB.
 
 ### Settings given as files
 
@@ -707,7 +723,7 @@ which every line has.
 - `peer_ip` is the TCP peer, normally traefik. `forwarded_for` is the
   `X-Forwarded-For` header as received, several lines of it joined with `, `.
   `client_group` is the client as the rate limits count it: its IPv4 address as
-  a /32, or the /64 of its IPv6 address.
+  a /32, or its IPv6 group, as `SWWAF_IPV6_GROUP_PREFIX` sets it.
 - `asn`, `as_name` and `country` are the client's AS number, such as `AS64496`,
   the name of that AS, and its country, as GeoJS or the lookup database gives
   them. Each is empty when `SWWAF_LOOKUP_SOURCE` is `off`, for a client in
@@ -810,7 +826,8 @@ which every line has.
 
 No body is logged, and no header but those above. `smallwebwaf`'s own messages
 (start, the settings, stop, errors) share the stream as JSON lines marked
-`"type":"process"`, each with `instance` as a request's line has it.
+`"type":"process"`, each with `instance` as a request's line has it, and those
+less severe than `SWWAF_LOG_LEVEL` are not written.
 
 Go's HTTP server, on which `smallwebwaf` is built, reads a request's line and
 headers before `smallwebwaf` sees the request, and some requests end there,
@@ -1078,9 +1095,10 @@ with times in UTC.
   zone gave it, `fetched`; and under `abuseipdb` (see "AbuseIPDB" below), the
   `day`, in UTC, whose checks it counts, left out before the first, the checks
   `spent` that day, and under `scores`, each score of AbuseIPDB still in use:
-  the `client`, its IPv4 address as a /32 or its IPv6 /64, its `score`, and when
-  AbuseIPDB gave it, `fetched`. As the file is read, the lists the settings no
-  longer name, and the verdicts of the zones they no longer name, are dropped.
+  the `client`, its IPv4 address as a /32 or its IPv6 group, its `score`, and
+  when AbuseIPDB gave it, `fetched`. As the file is read, the lists the settings
+  no longer name, and the verdicts of the zones they no longer name, are
+  dropped.
 - `alerts.json`: the state of the alerts (see "Alerts" above), indented to be
   read: under `cooldowns`, for each event and netblock, with the `source` too
   for a `reputation_hit`, or event and `file` or `source`, or for an `anomaly`,
@@ -1379,7 +1397,7 @@ request that carries the token as `Authorization: Bearer <token>`:
   `reason`. `netblock` is a netblock such as `203.0.113.0/24`, or a client's
   address, which bans the netblock a ban on that client covers: its IPv4
   address, or the netblock around it that `SWWAF_BAN_SCOPE_V4_PREFIX` sets, or
-  its IPv6 /64. `duration` is a duration such as `1h` or `7d`, or `permanent`.
+  its IPv6 group. `duration` is a duration such as `1h` or `7d`, or `permanent`.
   The ban starts at once, its `cause` is `admin`, and it is made even while
   another ban on the netblock lasts. A body that is not such an object, has
   another field, has anything but whitespace after the object, or is longer than
@@ -1660,7 +1678,7 @@ setting uses the answer, unless you set `SWWAF_LOOKUP_SOURCE=off`. The only
 visitors it is not told about are those in `SWWAF_ALLOW_NETS` or
 `SWWAF_DENY_NETS`, those whose netblock a ban covers, and those on a private,
 loopback or link-local address. An IPv6 visitor is asked about by the first
-address of its /64. Each answer is kept for seven days, in memory and in
+address of its IPv6 group. Each answer is kept for seven days, in memory and in
 `lookups.json`, so that it survives a restart, and a visitor whose answer is
 kept is not asked about again.
 
@@ -1775,7 +1793,7 @@ request from it until the zone has answered. The name asked about is the one RFC
 `192.0.2.99` is asked about in `dnsbl.dronebl.org` as
 `99.2.0.192.dnsbl.dronebl.org`, or the 32 hex digits of an IPv6 address in
 reverse order, each followed by a dot. An IPv6 client is asked about by its own
-address, not by its /64.
+address, not by its IPv6 group.
 
 A zone that answers that the name does not exist, or has no address, does not
 list the client, and one that answers with an address in `127.0.0.0/8` lists it.
@@ -1821,8 +1839,8 @@ While `SWWAF_ABUSEIPDB_KEY` holds the key of an AbuseIPDB account, `smallwebwaf`
 asks AbuseIPDB's check endpoint, `https://api.abuseipdb.com/api/v2/check`, for
 the abuse confidence score of a client's own address, from 0 to 100. It is unset
 by default, for the reason no blocklist is named, and since AbuseIPDB needs an
-account. An IPv6 client, a /64, is checked by the address of the request that
-has it checked, and its score is used for the whole /64, whichever of its
+account. An IPv6 client, an IPv6 group, is checked by the address of the request
+that has it checked, and its score is used for the whole group, whichever of its
 addresses sends, so that one client costs at most one check every
 `SWWAF_REPUTATION_CACHE_TTL`.
 
@@ -1934,17 +1952,17 @@ given as files" above).
   checks.
 
 Besides the Go standard library, `github.com/hashicorp/golang-lru/v2` keeps the
-table of clients to 20,000 and the GeoJS answers to 100,000, dropping the least
-recently seen, the DNSBL zones' verdicts and AbuseIPDB's scores to 100,000 each,
-dropping the one fetched longest ago, the anomaly counters to 20,000, dropping
-the one counted least recently, and the banned netblocks in the order they were
-last seen, from which the ledger picks the ban to drop past `SWWAF_MAX_BANS`,
-and `github.com/prometheus/client_golang` keeps the metrics and serves them, and
-`github.com/fsnotify/fsnotify` tells `smallwebwaf` when a state file or a rule
-file is saved, or the lookup database replaced, and
-`github.com/oschwald/maxminddb-golang/v2` reads the lookup database, which the
-tests write with `github.com/maxmind/mmdbwriter`. The country codes are the list
-in `internal/config/config.go`.
+table of clients to `SWWAF_MAX_TRACKED_CLIENTS` and the GeoJS answers to
+100,000, dropping the least recently seen, the DNSBL zones' verdicts and
+AbuseIPDB's scores to 100,000 each, dropping the one fetched longest ago, the
+anomaly counters to 20,000, dropping the one counted least recently, and the
+banned netblocks in the order they were last seen, from which the ledger picks
+the ban to drop past `SWWAF_MAX_BANS`, and `github.com/prometheus/client_golang`
+keeps the metrics and serves them, and `github.com/fsnotify/fsnotify` tells
+`smallwebwaf` when a state file or a rule file is saved, or the lookup database
+replaced, and `github.com/oschwald/maxminddb-golang/v2` reads the lookup
+database, which the tests write with `github.com/maxmind/mmdbwriter`. The
+country codes are the list in `internal/config/config.go`.
 
 ## Entrypoints