Take in an admin's edits of the state files while running (closes #68)

smallwebwaf watches SWWAF_STATE_DIR with fsnotify and takes in a saved
edit of a state file in place of what it held. It tells its own writes
from an admin's by the SHA-256 of what it last read or wrote; each write
first takes in an edit made since. An edit that does not parse is
renamed to <name>.bad at the file's next write. Every ban on a netblock
is checked, and the next ban is worked out from the one that ended
last. Two metrics count the edits taken in and set aside. README.md
says how to add and lift a ban.

Judgement call: a broken edit is set aside at the next write, since an
editor's file can be read half written.

Model: opus-5-5

README.md

@@ -13,25 +13,21 @@ JSON log line for every request.
 
 Status: the first two milestones are built
 (https://git.eeqj.de/sneak/smallwebwaf/issues/13 and
-https://git.eeqj.de/sneak/smallwebwaf/issues/14), and so are eight parts of
-milestone 3: the static lists, the bans that broken rate limits lead to, the
-JSON state files with your edits taken in while it runs and the paths the rate
-limits do not count, which come next in the build order, `observe` mode and the
-rest of the request log's fields, which come a little later, and the metrics
-endpoint and the header size and the idle time as settings, which come last in
-it. `smallwebwaf` passes each request to the app and the app's answer back,
-unchanged, within its timeouts and size limits, works out each client's address,
-bans a client that sends too many requests, not counting those for the paths you
-choose, refuses a client that comes from a country you refuse or from a network
-you refuse, lets the networks you choose through, keeps its bans, each client's
-counters and history, and GeoJS's answers in JSON files across restarts, takes
-in your edits of those files while it runs, writes a JSON log line for every
-request, serves Prometheus metrics to a scraper that holds the metrics token,
-and in `observe` mode passes on the requests it would refuse, logging what it
-would have done with them. It comes as the image the app's own image is built
-on. The rest of the design comes after that, in the order of the build order in
-[`SPEC.md`](SPEC.md). The survey of existing tools that led to the design is in
-[`EVALUATION.md`](EVALUATION.md).
+https://git.eeqj.de/sneak/smallwebwaf/issues/14), and so are five parts of
+milestone 3: the static lists, the bans that broken rate limits lead to and the
+JSON state files with your edits taken in while it runs, which come next in the
+build order, and the metrics endpoint and the header size and the idle time as
+settings, which come last in it. `smallwebwaf` passes each request to the app
+and the app's answer back, unchanged, within its timeouts and size limits, works
+out each client's address, bans a client that sends too many requests, refuses a
+client that comes from a country you refuse or from a network you refuse, lets
+the networks you choose through, keeps its bans, each client's counters and
+history, and GeoJS's answers in JSON files across restarts, takes in your edits
+of those files while it runs, writes a JSON log line for every request, and
+serves Prometheus metrics to a scraper that holds the metrics token. It comes as
+the image the app's own image is built on. The rest of the design comes after
+that, in the order of the build order in [`SPEC.md`](SPEC.md). The survey of
+existing tools that led to the design is in [`EVALUATION.md`](EVALUATION.md).
 
 ## Getting started
 
@@ -68,9 +64,7 @@ in `bin/state` unless `SWWAF_STATE_DIR` is set.
   address outside `SWWAF_TRUSTED_PROXIES` is the client; if every address in it
   is inside, the leftmost is, and with no header the peer is. The app sees what
   it would see from traefik directly: the same `Host`, the same
-  `X-Forwarded-Proto`, and `X-Forwarded-For` with the peer added at the end. It
-  also gets the request's id in `X-Request-ID`, the same id as in the request's
-  log line (see `request_id` in "Request log" below).
+  `X-Forwarded-Proto`, and `X-Forwarded-For` with the peer added at the end.
 - Enforces the timeouts and the size limits below. A limit passed before the
   response has started gets `smallwebwaf`'s own answer: `408` for a client too
   slow to send its request, `413` for a request body that is too large, `504`
@@ -84,15 +78,12 @@ in `bin/state` unless `SWWAF_STATE_DIR` is set.
 - Counts each client's requests over a minute, an hour and a day. A request that
   takes the client over one of the rate limits below is refused with
   `SWWAF_BAN_RESPONSE`, `403` by default, before anything reaches the app, and
-  bans the client. A request whose path starts with one of
-  `SWWAF_RATE_LIMIT_EXEMPT_PATHS`, as that setting below describes, is neither
-  counted nor refused by the rate limits; the static lists, bans and the country
-  lists still apply to it. A client is one IPv4 address, or one IPv6 /64, since
-  one abuser usually holds a whole /64. Each window is counted in two fixed
-  buckets, the earlier one weighted by how much of it the window still covers.
-  At most 20,000 clients are kept, the least recently seen dropped first, with
-  their history, and a restart gives no client a fresh allowance (see "State
-  files" below).
+  bans the client. A client is one IPv4 address, or one IPv6 /64, since one
+  abuser usually holds a whole /64. Each window is counted in two fixed buckets,
+  the earlier one weighted by how much of it the window still covers. At most
+  20,000 clients are kept, the least recently seen dropped first, with their
+  history, and a restart gives no client a fresh allowance (see "State files"
+  below).
 - Bans a client that breaks a rate limit, as "Bans" in [`SPEC.md`](SPEC.md)
   describes: the first ban lasts an hour, and a limit broken again within a day
   of a ban ending bans for three times as long as that ban, so 1, 3, 9, 27 and
@@ -128,18 +119,6 @@ in `bin/state` unless `SWWAF_STATE_DIR` is set.
   `SWWAF_ALLOW_NETS` too is let through. A client in
   `SWWAF_RATE_LIMIT_EXEMPT_NETS` is neither counted nor refused by the rate
   limits; the country lists and bans still apply to it.
-- In `observe` mode, with `SWWAF_MODE=observe`, refuses none of the requests
-  that `SWWAF_DENY_NETS`, a ban, the country lists or a rate limit would refuse:
-  it passes them to the app, and their log lines name what `enforce` mode would
-  have done (see `would_action` in "Request log" below). The checks run, and
-  requests are counted, as in `enforce` mode, but a broken rate limit makes no
-  ban and does not set the client's counters back to zero, so each request over
-  the limit is logged as one that would be refused. The bans in `bans.json` are
-  kept, and refuse requests again when `smallwebwaf` next runs in `enforce`
-  mode, as long as they last. The timeouts and size limits still apply, since
-  they protect `smallwebwaf` and the app themselves, and a request for the
-  metrics without the token is still answered `401`. It is for trying a
-  configuration before enforcing it.
 - Answers `GET /_smallwebwaf/healthz` itself with `200` and `ok`, before any
   check and without asking the app, for the image's health check.
 - Answers `GET /_smallwebwaf/metrics` with its metrics (see "Metrics" below) for
@@ -161,14 +140,6 @@ it, and the effective settings are logged at start.
 - `SWWAF_LISTEN_ADDR` (default `:8080`): where `smallwebwaf` listens.
 - `SWWAF_UPSTREAM_URL` (default `http://127.0.0.1:8081`): the app, as `http` or
   `https`, a host and an optional port, and nothing more.
-- `SWWAF_INSTANCE_NAME` (default: the host's name, which docker sets to the
-  first 12 characters of the container's id unless the deployment names one):
-  the name each request log line gives as `instance`. Set it, for example to
-  `fsn1app1/gitea`, for a name that stays the same when a deploy replaces the
-  container, and that tells instances apart when several log to one place.
-- `SWWAF_MODE` (default `enforce`): `enforce`, or `observe` to pass on the
-  requests `smallwebwaf` would refuse and log what it would have done (see "What
-  it does so far" above).
 - `SWWAF_TRUSTED_PROXIES` (default `10.0.0.0/8,172.16.0.0/12,192.168.0.0/16`,
   the private address ranges): the netblocks whose `X-Forwarded-For` is
   believed. A list given replaces the default; set but empty, it trusts nothing.
@@ -204,20 +175,6 @@ it, and the effective settings are logged at start.
   requests a client may make in a minute, an hour and a day. The defaults are
   several times what one busy person produces, since a browser loading a heavy
   page makes a few hundred requests and several people often share one address.
-- `SWWAF_RATE_LIMIT_EXEMPT_PATHS` (default empty): path prefixes whose requests
-  the rate limits neither count nor refuse, such as `/assets/` for static
-  assets; each starts with `/`. A request whose path, percent-decoded, contains
-  `..` anywhere or a backslash, or whose path as sent holds an encoded slash
-  (`%2F` or `%2f`), is never exempt, since the app may act on it as a path
-  outside every prefix: `/assets/..%2Flogin` as `/login`. Any other request is
-  exempt when its path as sent, the path the app receives, before any query
-  string and not percent-decoded, starts with a prefix, character for character.
-  `/assets/` matches `/assets/app.js` and `/assets/`, but not `/assets`,
-  `/Assets/app.js`, `/%61ssets/app.js`, `/static/assets/app.js`,
-  `/static/../assets/app.js` or `/assets%2Fapp.js`. A character the client sends
-  percent-encoded, such as a space, is written percent-encoded in a prefix, as
-  in `/my%20files/`, and there are no wildcards: `*` is a character like any
-  other.
 - `SWWAF_DENIED_COUNTRIES` (default empty): countries whose clients are refused,
   for example `cn,ru,kp`.
 - `SWWAF_EXCLUSIVELY_ALLOWED_COUNTRIES` (default empty): when set, the only
@@ -247,13 +204,6 @@ it, and the effective settings are logged at start.
   `bans.json` is written, with every ban made in between.
 - `SWWAF_STATE_COUNTER_INTERVAL` (default `15m`): how often every state file is
   written.
-- `SWWAF_LOG_REQUEST_HEADERS` (default
-  `accept,accept-language,accept-encoding,content-type,origin,range`): the
-  request headers whose values the request log gives, in either case.
-  `Authorization`, `Cookie` and `Set-Cookie` are never logged, even when listed
-  (see "Request log" below). An entry naming `Host` or `Transfer-Encoding` stops
-  the start, since Go's HTTP server takes both out of the request; the request's
-  host is the field `host`.
 - `SWWAF_METRICS_TOKEN` (default unset): the token a scraper sends for the
   metrics, a long random value. While it is unset the metrics are off; one
   shorter than 32 characters stops the start. The settings logged at start show
@@ -283,42 +233,18 @@ GeoJS are kept, for 7 days each.
 refused ones included:
 
 ```
-{"type":"request","time":"2026-10-03T12:00:00.123Z","instance":"fsn1app1/gitea","client_ip":"203.0.113.9","method":"GET","scheme":"https","host":"app.example","path":"/","query":"","protocol":"HTTP/1.1","status":200,"request_bytes":0,"response_bytes":5120,"referer":"","user_agent":"curl/8.9.1","request_id":"7Q2NHZ4KJ3VXW5YB6R3MEFTD2A","peer_ip":"172.18.0.2","forwarded_for":"203.0.113.9","client_group":"203.0.113.9/32","country":"DE","request_headers":{"accept":"*/*"},"response_content_type":"text/html; charset=utf-8","upstream_status":200,"action":"forward","counts":{"minute":1,"hour":12,"day":40},"duration_total":3.217,"duration_checks":0.041,"duration_upstream_connect":0.052,"duration_upstream_first_byte":2.874,"duration_upstream_total":3.104}
+{"type":"request","time":"2026-10-03T12:00:00.123Z","client_ip":"203.0.113.9","peer_ip":"172.18.0.2","country":"DE","method":"GET","host":"app.example","path":"/","query":"","protocol":"HTTP/1.1","status":200,"upstream_status":200,"request_bytes":0,"response_bytes":5120,"referer":"","user_agent":"curl/8.9.1","action":"forward","duration_total":3.217,"duration_upstream_total":3.104}
 ```
 
-A field that does not apply to a request is left out of its line, apart from
-`type`, the fields from `time` to `user_agent`, `request_id`, `peer_ip`,
-`client_group`, `country`, `action` and `duration_total`, which every line has.
-
-- `time` is when the request arrived, in UTC. `instance` is
-  `SWWAF_INSTANCE_NAME`. `scheme` is the `X-Forwarded-Proto` a trusted proxy
-  sent, and otherwise `http`. `path` and `query` are as the client sent them.
-- `request_id` is the `X-Request-ID` a trusted proxy sent, or a new random one
-  of 26 letters and digits when it sent none, or when the peer is not a trusted
-  proxy. A request passed to the app takes it there in `X-Request-ID`.
-- `peer_ip` is the TCP peer, normally traefik. `forwarded_for` is the
-  `X-Forwarded-For` header as received, several lines of it joined with `, `.
-  `client_group` is the client as the rate limits count it: its IPv4 address as
-  a /32, or the /64 of its IPv6 address.
+- `time` is when the request arrived, in UTC. `peer_ip` is the TCP peer,
+  normally traefik. `path` and `query` are as the client sent them.
 - `country` is the client's country as GeoJS places it. It is empty with neither
   country list set, for a client in `SWWAF_ALLOW_NETS` or `SWWAF_DENY_NETS`, for
   a client on a private, loopback or link-local address, when GeoJS cannot place
-  the client or has not answered in time, and for a request whose client a ban
-  covers, even when the client's country is known.
-- `content_type` is the request's `Content-Type`, and `content_length` the
-  length the request announced for its body, which is left out for none or zero.
-- `request_headers` are the request's headers that `SWWAF_LOG_REQUEST_HEADERS`
-  names, by name in lower case, several lines of one joined with `, `.
-  `Authorization`, `Cookie` and `Set-Cookie` are never among them, whatever the
-  setting says: `has_authorization` and `has_cookie` are there instead, and
-  true, when the request has an `Authorization` or a `Cookie` header.
-- `websocket` is there, and true, when the app switched the connection to
-  another protocol, as it does for a WebSocket.
+  the client or has not answered in time, and for a request refused because a
+  ban covers its client, even when the client's country is known.
 - `status` is what the client was sent, `0` if nothing was; `upstream_status` is
   what the app answered, and is left out when the app did not answer.
-- `response_content_type`, `cache_control` and `location` are the
-  `Content-Type`, `Cache-Control` and `Location` headers of the answer: the
-  app's, as passed on, or those of `smallwebwaf`'s own answer.
 - `request_bytes` and `response_bytes` count body bytes.
 - `action` is `forward` for a request passed to the app, `denied` for one
   refused because its client is in `SWWAF_DENY_NETS`, `banned` for one refused
@@ -328,42 +254,16 @@ A field that does not apply to a request is left out of its line, apart from
   `timed_out` for one that ran out of time, `upstream_error` when the app could
   not be reached or its answer broke off, and `admin` for one `smallwebwaf`
   answered at its own endpoint.
-- `would_action` is there in `observe` mode for a request that
-  `SWWAF_DENY_NETS`, a ban, the country lists or a rate limit would have refused
-  in `enforce` mode, and names the action that refusal would have had: `denied`,
-  `banned`, `country_denied` or `rate_limited`. `action` then names what was
-  done: `forward` for a request passed to the app, and another action, such as
-  `too_large`, for one a size or time limit refused.
-- `counts` gives the client's requests in the minute, the hour and the day as
-  the rate limits count them, this request included: in each window, those in
-  the bucket under way and a share of those in the bucket before, so a count can
-  have a fraction. For a request that broke a limit, they are the counts that
-  broke it. It is left out for a request the rate limits do not count: the
-  health check, one from a client in `SWWAF_ALLOW_NETS` or
-  `SWWAF_RATE_LIMIT_EXEMPT_NETS`, one for a path that
-  `SWWAF_RATE_LIMIT_EXEMPT_PATHS` exempts, and one that `SWWAF_DENY_NETS`, a ban
-  or the country lists refuse, or would refuse in `observe` mode. The byte
-  totals come with the byte limits.
 - `limit_hit` is there for a request that broke a rate limit, and names the
   window whose limit it went over: `minute`, `hour` or `day`, the shortest if it
   went over several. `offence` is then `limit`.
 - `ban_expires` is there for a request that made a ban or was refused under one,
-  or in `observe` mode would have been refused under one, and gives when the ban
-  ends, in the same form as `time`, or `permanent`.
+  and gives when the ban ends, in the same form as `time`, or `permanent`.
 - `aborted` is there, and true, when the client went away early.
-- The timings are in milliseconds, to the microsecond. `duration_total` runs
-  from when the request's headers had been read to when its line is written, and
-  `duration_checks` over the same start to when the checks were done; the health
-  check runs none, and its line has no `duration_checks`.
-  `duration_upstream_connect`, `duration_upstream_first_byte` and
-  `duration_upstream_total` are there for a request passed to the app, and run
-  from when it was handed to the app: until there was a connection to it, new or
-  kept open from an earlier request, until the first byte of its answer arrived,
-  and until the end. The first two are left out when that never happened, as for
-  an app that cannot be reached.
+- `duration_total` and `duration_upstream_total` are in milliseconds.
 
-No body is logged, and no header but those above. `smallwebwaf`'s own messages
-(start, the settings, stop, errors) share the stream as JSON lines marked
+No body and no other header is logged. `smallwebwaf`'s own messages (start, the
+settings, stop, errors) share the stream as JSON lines marked
 `"type":"process"`.
 
 Go's HTTP server, on which `smallwebwaf` is built, reads a request's line and
@@ -718,16 +618,17 @@ the metrics, failure behaviour and the build order.
 So far `smallwebwaf` looks up only the country, only through GeoJS, and only
 while `SWWAF_DENIED_COUNTRIES` or `SWWAF_EXCLUSIVELY_ALLOWED_COUNTRIES` is set:
 then the address of every new visitor is sent to GeoJS, except a visitor in
-`SWWAF_ALLOW_NETS` or `SWWAF_DENY_NETS` and one whose netblock a ban covers, and
-with neither set, none is. An IPv6 visitor is asked about by the first address
-of its /64. A new visitor waits at most a second for its answer, and without one
-counts as coming from an unknown country until the answer arrives. The addresses
-waiting are asked about together, up to 200 in one request, one request at a
-time; at most 10,000 visitors wait, and one more counts as coming from an
-unknown country until there is room. While GeoJS fails, visitors with a kept
-answer are unaffected and new ones count as coming from an unknown country.
-GeoJS is then left alone for a second, twice as long after each further failure
-up to five minutes, and asked again by the next request that needs it.
+`SWWAF_ALLOW_NETS` or `SWWAF_DENY_NETS` and one refused because a ban covers its
+netblock, and with neither set, none is. An IPv6 visitor is asked about by the
+first address of its /64. A new visitor waits at most a second for its answer,
+and without one counts as coming from an unknown country until the answer
+arrives. The addresses waiting are asked about together, up to 200 in one
+request, one request at a time; at most 10,000 visitors wait, and one more
+counts as coming from an unknown country until there is room. While GeoJS fails,
+visitors with a kept answer are unaffected and new ones count as coming from an
+unknown country. GeoJS is then left alone for a second, twice as long after each
+further failure up to five minutes, and asked again by the next request that
+needs it.
 
 In the full design, `smallwebwaf` looks up the AS number and country of every
 client, for the request log, the metrics and the ban notes, and for the country
@@ -779,10 +680,8 @@ addresses are never sent to GeoJS.
   limits, and writes the request's log line. Its `check` method is where a
   request is refused before anything reaches the app: for `SWWAF_DENY_NETS`, for
   a ban, for the country lists, for a rate limit, which bans the client, and for
-  an announced body over the size limit; in `observe` mode, only for the size
-  limit, with what it would have refused for noted in the log line. A request
-  under `/_smallwebwaf/` that `check` lets through is answered by `answerAdmin`
-  instead of reaching the app.
+  an announced body over the size limit. A request under `/_smallwebwaf/` that
+  `check` lets through is answered by `answerAdmin` instead of reaching the app.
 - `internal/metrics`: the metrics, counted as the other parts tell it what
   happened, and served in the Prometheus text format.
 - `internal/bans`: the ban ledger: each netblock's bans with their notes, how
@@ -849,7 +748,8 @@ so that they run in minimal containers.
 
 ## TODO
 
-- The rest of the design, in the order of the build order in
+- The rest of milestone 3, from exemptions up to the rest of the request log's
+  fields, and the rest of the design, in the order of the build order in
   [`SPEC.md`](SPEC.md).
 
 ## Documents

internal/bans/bans.go

@@ -107,8 +107,8 @@ type Ledger struct {
 	changed chan struct{}
 
 	mu sync.Mutex
-	// netblocks holds each banned netblock's bans, oldest first. Check and
-	// Find make each netblock they find the most recently seen.
+	// netblocks holds each banned netblock's bans, oldest first. Check
+	// makes each netblock it finds the most recently seen.
 	netblocks *simplelru.LRU[netip.Prefix, *[]Ban]
 	// held is how many bans netblocks holds, at most rules.MaxBans.
 	held int
@@ -144,36 +144,34 @@ func (l *Ledger) Changed() <-chan struct{} {
 	return l.changed
 }
 
-// Check is called for a request from client, at now. It reports whether
-// a ban on a netblock client is in is active, and returns that ban, with
-// the request counted among those it refused.
+// Check is called for each request from client, at now. It reports
+// whether a ban on a netblock client is in is active, and returns that
+// ban, with the request counted among those it refused.
 func (l *Ledger) Check(client netip.Addr, now time.Time) (Ban, bool) {
 	l.mu.Lock()
 	defer l.mu.Unlock()
 
-	ban := l.active(client, now)
-	if ban == nil {
-		return Ban{}, false
+	lengths := l.v6Lengths
+	if client.Is4() {
+		lengths = l.v4Lengths
 	}
 
-	ban.Notes.Requests++
-	ban.Notes.Refused++
+	for _, length := range lengths {
+		bans, found := l.netblocks.Get(netip.PrefixFrom(client, length).Masked())
+		if !found {
+			continue
+		}
 
-	return *ban, true
-}
+		ban := activeBan(*bans, now)
+		if ban != nil {
+			ban.Notes.Requests++
+			ban.Notes.Refused++
 
-// Find is Check without counting the request among those the ban
-// refused: in observe mode a ban refuses nothing.
-func (l *Ledger) Find(client netip.Addr, now time.Time) (Ban, bool) {
-	l.mu.Lock()
-	defer l.mu.Unlock()
-
-	ban := l.active(client, now)
-	if ban == nil {
-		return Ban{}, false
+			return *ban, true
+		}
 	}
 
-	return *ban, true
+	return Ban{}, false
 }
 
 // activeBan returns the ban in bans, a netblock's bans oldest first, that
@@ -332,29 +330,6 @@ func (l *Ledger) Load(bans []Ban) {
 	}
 }
 
-// active returns the ban active at now on a netblock client is in, or
-// nil.
-func (l *Ledger) active(client netip.Addr, now time.Time) *Ban {
-	lengths := l.v6Lengths
-	if client.Is4() {
-		lengths = l.v4Lengths
-	}
-
-	for _, length := range lengths {
-		bans, found := l.netblocks.Get(netip.PrefixFrom(client, length).Masked())
-		if !found {
-			continue
-		}
-
-		ban := activeBan(*bans, now)
-		if ban != nil {
-			return ban
-		}
-	}
-
-	return nil
-}
-
 // add adds ban to its netblock's bans, after the last, and makes its
 // netblock the most recently seen. With MaxBans held, it drops one first.
 func (l *Ledger) add(ban Ban) {

internal/bans/bans_test.go

@@ -162,28 +162,6 @@ func TestCheckRefusesWhileTheBanLastsAndCountsTheRefusals(t *testing.T) {
 	}
 }
 
-func TestFindCountsNothing(t *testing.T) {
-	t.Parallel()
-
-	ledger := bans.New(defaultRules())
-	netblock := netip.MustParsePrefix("203.0.113.9/32")
-	ban := ledger.BanForLimit(netblock, midnight(), bans.Notes{Requests: 5})
-
-	got, banned := ledger.Find(netblock.Addr(), ban.Expires.Add(-time.Nanosecond))
-	if !banned || got != ban {
-		t.Errorf("find during the ban gives %+v and %t, want %+v", got, banned, ban)
-	}
-
-	_, banned = ledger.Find(netblock.Addr(), ban.Expires)
-	if banned {
-		t.Error("the ban did not end")
-	}
-
-	if notes := ledger.Bans(netblock)[0].Notes; notes != ban.Notes {
-		t.Errorf("the notes are %+v, want them unchanged, %+v", notes, ban.Notes)
-	}
-}
-
 func TestMaxBansDropsTheEarliestBanOfTheNetblockSeenLongestAgo(t *testing.T) {
 	t.Parallel()
 

internal/bans/snapshot_test.go

@@ -147,14 +147,8 @@ func TestPermanentBanStartedBeforeAnEndedOneRefuses(t *testing.T) {
 	ledger.Load([]bans.Ban{permanent, ended})
 
 	now := midnight().Add(2 * time.Hour)
-	client := netip.MustParseAddr("203.0.113.9")
 
-	ban, banned := ledger.Find(client, now)
-	if !banned || !ban.Permanent() {
-		t.Errorf("find gives %+v and %t, want the permanent ban", ban, banned)
-	}
-
-	ban, banned = ledger.Check(client, now)
+	ban, banned := ledger.Check(netip.MustParseAddr("203.0.113.9"), now)
 	if !banned || !ban.Permanent() {
 		t.Errorf("the client is refused: %t, under %+v, want under the permanent ban",
 			banned, ban)

internal/config/config.go

@@ -12,7 +12,6 @@ import (
 	"net/http"
 	"net/netip"
 	"net/url"
-	"os"
 	"path/filepath"
 	"slices"
 	"strconv"
@@ -28,15 +27,6 @@ type Config struct {
 	ListenAddr string
 	// UpstreamURL is the app (SWWAF_UPSTREAM_URL).
 	UpstreamURL *url.URL
-	// InstanceName is the name each request log line gives as instance
-	// (SWWAF_INSTANCE_NAME), by default the host's name, which docker sets
-	// to the first 12 characters of the container's id.
-	InstanceName string
-	// Observe is true in observe mode, when SWWAF_MODE is observe rather
-	// than enforce: a request that SWWAF_DENY_NETS, a ban, the country
-	// lists or a rate limit would refuse is passed to the app instead, and
-	// no ban is made.
-	Observe bool
 	// TrustedProxies are the netblocks whose X-Forwarded-For is
 	// believed (SWWAF_TRUSTED_PROXIES).
 	TrustedProxies []netip.Prefix
@@ -80,10 +70,6 @@ type Config struct {
 	RateLimitPerMinute int64
 	RateLimitPerHour   int64
 	RateLimitPerDay    int64
-	// RateLimitExemptPaths are the path prefixes whose requests the rate
-	// limits neither count nor refuse (SWWAF_RATE_LIMIT_EXEMPT_PATHS).
-	// Each starts with /.
-	RateLimitExemptPaths []string
 	// DeniedCountries are the countries whose clients are refused
 	// (SWWAF_DENIED_COUNTRIES). ExclusivelyAllowedCountries, when not
 	// empty, are the only countries whose clients are let through
@@ -118,9 +104,6 @@ type Config struct {
 	StateDir             string
 	StateWriteDelay      time.Duration
 	StateCounterInterval time.Duration
-	// LogRequestHeaders are the request headers whose values the request
-	// log gives, in lower case (SWWAF_LOG_REQUEST_HEADERS).
-	LogRequestHeaders []string
 	// MetricsToken is the bearer token a scraper sends for the metrics
 	// (SWWAF_METRICS_TOKEN), "" while it is unset and the metrics are off.
 	// MetricsTopN is how many countries get series of their own in the
@@ -167,11 +150,6 @@ var (
 			"such as http://127.0.0.1:8081")
 	errNotCountry = errors.New(
 		"is not a two-letter country code such as de or kp")
-	errNotHeaderName = errors.New(
-		"is not a header name such as accept-language")
-	errHeaderTakenOut = errors.New(
-		"is taken out of every request by Go's HTTP server, so it can never " +
-			"be logged")
 	errOnBothLists          = errors.New("is in SWWAF_DENIED_COUNTRIES too")
 	errNotOver4K            = errors.New("is not a size of more than 4K, such as 32K")
 	errNotDurationAboveZero = errors.New(
@@ -183,10 +161,7 @@ var (
 		"is not the length of an IPv4 netblock, from 0 to 32, such as 24")
 	errNotAbsolutePath = errors.New(
 		"is not an absolute path, such as /var/lib/smallwebwaf")
-	errShortToken    = errors.New("is shorter than 32 characters")
-	errNotMode       = errors.New("is not enforce or observe")
-	errNotPathPrefix = errors.New(
-		"is not a path prefix starting with /, such as /assets/")
+	errShortToken = errors.New("is shorter than 32 characters")
 )
 
 // FromEnvironment reads the settings with lookupEnv, normally
@@ -194,12 +169,9 @@ var (
 // that is set but invalid is an error that names it.
 func FromEnvironment(lookupEnv func(string) (string, bool)) (*Config, error) {
 	env := &environment{lookupEnv: lookupEnv}
-	hostname, _ := os.Hostname() // "" when the host has no name to give
 	cfg := &Config{
 		ListenAddr:           env.address("SWWAF_LISTEN_ADDR", ":8080"),
 		UpstreamURL:          env.appURL("SWWAF_UPSTREAM_URL", "http://127.0.0.1:8081"),
-		InstanceName:         env.value("SWWAF_INSTANCE_NAME", hostname),
-		Observe:              env.observe("SWWAF_MODE", "enforce"),
 		TrustedProxies:       env.netblocks("SWWAF_TRUSTED_PROXIES", privateRanges),
 		ClientRequestTimeout: env.duration("SWWAF_CLIENT_REQUEST_TIMEOUT", "60s"),
 		ClientRequestHeaderMaxBytes: env.headerSize(
@@ -216,7 +188,6 @@ func FromEnvironment(lookupEnv func(string) (string, bool)) (*Config, error) {
 		RateLimitPerMinute:      env.count("SWWAF_RATE_LIMIT_PER_MINUTE", "1000"),
 		RateLimitPerHour:        env.count("SWWAF_RATE_LIMIT_PER_HOUR", "10000"),
 		RateLimitPerDay:         env.count("SWWAF_RATE_LIMIT_PER_DAY", "50000"),
-		RateLimitExemptPaths:    env.pathPrefixes("SWWAF_RATE_LIMIT_EXEMPT_PATHS", ""),
 		DeniedCountries:         env.countries("SWWAF_DENIED_COUNTRIES", ""),
 		ExclusivelyAllowedCountries: env.countries(
 			"SWWAF_EXCLUSIVELY_ALLOWED_COUNTRIES", ""),
@@ -229,10 +200,8 @@ func FromEnvironment(lookupEnv func(string) (string, bool)) (*Config, error) {
 		StateDir:             env.absolutePath("SWWAF_STATE_DIR", "/var/lib/smallwebwaf"),
 		StateWriteDelay:      env.durationNotOff("SWWAF_STATE_WRITE_DELAY", "10s"),
 		StateCounterInterval: env.durationNotOff("SWWAF_STATE_COUNTER_INTERVAL", "15m"),
-		LogRequestHeaders: env.headerNames("SWWAF_LOG_REQUEST_HEADERS",
-			"accept,accept-language,accept-encoding,content-type,origin,range"),
-		MetricsToken: env.token("SWWAF_METRICS_TOKEN"),
-		MetricsTopN:  env.numberNotOff("SWWAF_METRICS_TOP_N", "50"),
+		MetricsToken:         env.token("SWWAF_METRICS_TOKEN"),
+		MetricsTopN:          env.numberNotOff("SWWAF_METRICS_TOP_N", "50"),
 	}
 
 	for _, country := range cfg.ExclusivelyAllowedCountries {
@@ -305,17 +274,6 @@ func (e *environment) appURL(name, defaultValue string) *url.URL {
 	return upstream
 }
 
-// observe reads the setting that is the mode, enforce or observe, and
-// reports whether it is observe.
-func (e *environment) observe(name, defaultValue string) bool {
-	mode := e.value(name, defaultValue)
-	if mode != "enforce" && mode != "observe" {
-		e.check(name, fmt.Errorf("%q %w", mode, errNotMode))
-	}
-
-	return mode == "observe"
-}
-
 // netblocks reads a setting that is a list of netblocks.
 func (e *environment) netblocks(name, defaultValue string) []netip.Prefix {
 	netblocks, err := parseNetblocks(e.value(name, defaultValue))
@@ -357,14 +315,6 @@ func (e *environment) count(name, defaultValue string) int64 {
 	return count
 }
 
-// pathPrefixes reads a setting that is a list of path prefixes.
-func (e *environment) pathPrefixes(name, defaultValue string) []string {
-	prefixes, err := parsePathPrefixes(e.value(name, defaultValue))
-	e.check(name, err)
-
-	return prefixes
-}
-
 // countries reads a setting that is a list of countries.
 func (e *environment) countries(name, defaultValue string) []string {
 	countries, err := parseCountries(e.value(name, defaultValue))
@@ -373,15 +323,6 @@ func (e *environment) countries(name, defaultValue string) []string {
 	return countries
 }
 
-// headerNames reads a setting that is a list of header names, and
-// returns them in lower case.
-func (e *environment) headerNames(name, defaultValue string) []string {
-	headers, err := parseHeaderNames(e.value(name, defaultValue))
-	e.check(name, err)
-
-	return headers
-}
-
 // durationNotOff reads a setting that is a duration and, unlike a
 // timeout, cannot be off.
 func (e *environment) durationNotOff(name, defaultValue string) time.Duration {
@@ -652,23 +593,6 @@ func parseNetblock(value string) (netip.Prefix, error) {
 	return netip.PrefixFrom(addr, addr.BitLen()), nil
 }
 
-// parsePathPrefixes reads a comma-separated list of path prefixes, each
-// starting with /.
-func parsePathPrefixes(value string) ([]string, error) {
-	prefixes, err := parseList(value)
-	if err != nil {
-		return nil, err
-	}
-
-	for _, prefix := range prefixes {
-		if !strings.HasPrefix(prefix, "/") {
-			return nil, fmt.Errorf("%q %w", prefix, errNotPathPrefix)
-		}
-	}
-
-	return prefixes, nil
-}
-
 // countryCodes are the two-letter codes ISO 3166-1 assigns today, and XK,
 // the code in common use for Kosovo. golang.org/x/text/language cannot
 // check them: it also takes withdrawn codes such as su, and reserved ones
@@ -725,44 +649,6 @@ func parseCountries(value string) ([]string, error) {
 	return countries, nil
 }
 
-// headerNameChars are the characters RFC 9110 allows in a header name:
-// letters, digits and these marks.
-const headerNameChars = "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz" +
-	"0123456789!#$%&'*+-.^_`|~"
-
-// parseHeaderNames reads a comma-separated list of header names in either
-// case, and returns them in lower case. Host and Transfer-Encoding are
-// refused: Go's HTTP server takes them out of the request's headers.
-func parseHeaderNames(value string) ([]string, error) {
-	items, err := parseList(value)
-	if err != nil {
-		return nil, err
-	}
-
-	headers := make([]string, 0, len(items))
-
-	for _, item := range items {
-		for _, char := range item {
-			if !strings.ContainsRune(headerNameChars, char) {
-				return nil, fmt.Errorf("%q %w", item, errNotHeaderName)
-			}
-		}
-
-		header := strings.ToLower(item)
-		switch header {
-		case "host":
-			return nil, fmt.Errorf("%q %w; the request's host is the field host",
-				item, errHeaderTakenOut)
-		case "transfer-encoding":
-			return nil, fmt.Errorf("%q %w", item, errHeaderTakenOut)
-		}
-
-		headers = append(headers, header)
-	}
-
-	return headers, nil
-}
-
 // parseListenAddr checks an address to listen on: an optional host and a
 // port number.
 func parseListenAddr(value string) (string, error) {

internal/config/config_test.go

@@ -6,7 +6,6 @@ import (
 	"log/slog"
 	"maps"
 	"net/netip"
-	"os"
 	"slices"
 	"strings"
 	"testing"
@@ -19,7 +18,6 @@ import (
 const (
 	listenAddr              = "SWWAF_LISTEN_ADDR"
 	upstreamURL             = "SWWAF_UPSTREAM_URL"
-	mode                    = "SWWAF_MODE"
 	trustedProxies          = "SWWAF_TRUSTED_PROXIES"
 	clientRequestTimeout    = "SWWAF_CLIENT_REQUEST_TIMEOUT"
 	clientHeaderMaxBytes    = "SWWAF_CLIENT_REQUEST_HEADER_MAX_BYTES"
@@ -35,7 +33,6 @@ const (
 	rateLimitPerMinute      = "SWWAF_RATE_LIMIT_PER_MINUTE"
 	rateLimitPerHour        = "SWWAF_RATE_LIMIT_PER_HOUR"
 	rateLimitPerDay         = "SWWAF_RATE_LIMIT_PER_DAY"
-	rateLimitExemptPaths    = "SWWAF_RATE_LIMIT_EXEMPT_PATHS"
 	deniedCountries         = "SWWAF_DENIED_COUNTRIES"
 	allowedCountries        = "SWWAF_EXCLUSIVELY_ALLOWED_COUNTRIES"
 	banResponse             = "SWWAF_BAN_RESPONSE"
@@ -49,14 +46,8 @@ const (
 	stateCounterInterval    = "SWWAF_STATE_COUNTER_INTERVAL"
 	metricsToken            = "SWWAF_METRICS_TOKEN" //nolint:gosec // the setting's name
 	metricsTopN             = "SWWAF_METRICS_TOP_N"
-	instanceName            = "SWWAF_INSTANCE_NAME"
-	logRequestHeaders       = "SWWAF_LOG_REQUEST_HEADERS"
 )
 
-// defaultLogRequestHeaders is the default of SWWAF_LOG_REQUEST_HEADERS.
-const defaultLogRequestHeaders = "accept,accept-language,accept-encoding," +
-	"content-type,origin,range"
-
 // token is a token of 32 characters, the shortest allowed.
 const token = "0123456789abcdef0123456789abcdef"
 
@@ -92,7 +83,6 @@ func TestDefaults(t *testing.T) {
 
 	wantSettings(t, cfg, config.Config{
 		ListenAddr:                  ":8080",
-		Observe:                     false,
 		ClientRequestTimeout:        time.Minute,
 		ClientRequestHeaderMaxBytes: 32 << 10,
 		ClientIdleTimeout:           2 * time.Minute,
@@ -128,22 +118,6 @@ func TestDefaults(t *testing.T) {
 	wantNetblocks(t, cfg.DenyNets)
 	wantCountries(t, deniedCountries, cfg.DeniedCountries)
 	wantCountries(t, allowedCountries, cfg.ExclusivelyAllowedCountries)
-
-	hostname, err := os.Hostname()
-	if err != nil || hostname == "" || cfg.InstanceName != hostname {
-		t.Errorf("%s is %q, want the host's name %q (%v)", instanceName,
-			cfg.InstanceName, hostname, err)
-	}
-
-	wantHeaders := strings.Split(defaultLogRequestHeaders, ",")
-	if !slices.Equal(cfg.LogRequestHeaders, wantHeaders) {
-		t.Errorf("%s gave %v, want %v", logRequestHeaders, cfg.LogRequestHeaders,
-			wantHeaders)
-	}
-
-	if len(cfg.RateLimitExemptPaths) != 0 {
-		t.Errorf("%s gave %v, want none", rateLimitExemptPaths, cfg.RateLimitExemptPaths)
-	}
 }
 
 func TestValuesAsSet(t *testing.T) {
@@ -152,7 +126,6 @@ func TestValuesAsSet(t *testing.T) {
 	cfg := fromEnvironment(t, environment{
 		listenAddr:              "127.0.0.1:9000",
 		upstreamURL:             "https://app.internal:8443/",
-		mode:                    "observe",
 		trustedProxies:          " 192.0.2.1, 10.1.2.3/8 ,2001:db8::/32",
 		clientRequestTimeout:    "90s",
 		clientHeaderMaxBytes:    "8K",
@@ -168,7 +141,6 @@ func TestValuesAsSet(t *testing.T) {
 		rateLimitPerMinute:      "60",
 		rateLimitPerHour:        "600",
 		rateLimitPerDay:         "6000",
-		rateLimitExemptPaths:    "/assets/, /favicon.ico",
 		deniedCountries:         "cn, RU,kp,Xk",
 		allowedCountries:        "de",
 		banResponse:             "429",
@@ -186,7 +158,6 @@ func TestValuesAsSet(t *testing.T) {
 
 	wantSettings(t, cfg, config.Config{
 		ListenAddr:                  "127.0.0.1:9000",
-		Observe:                     true,
 		ClientRequestTimeout:        90 * time.Second,
 		ClientRequestHeaderMaxBytes: 8 << 10,
 		ClientIdleTimeout:           5 * time.Minute,
@@ -221,39 +192,6 @@ func TestValuesAsSet(t *testing.T) {
 	wantNetblocks(t, cfg.DenyNets, "198.51.100.0/24")
 	wantCountries(t, deniedCountries, cfg.DeniedCountries, "CN", "RU", "KP", "XK")
 	wantCountries(t, allowedCountries, cfg.ExclusivelyAllowedCountries, "DE")
-
-	if !slices.Equal(cfg.RateLimitExemptPaths, []string{"/assets/", "/favicon.ico"}) {
-		t.Errorf("%s gave %v, want /assets/ and /favicon.ico",
-			rateLimitExemptPaths, cfg.RateLimitExemptPaths)
-	}
-}
-
-func TestPathPrefixNotStartingWithSlashStopsTheStart(t *testing.T) {
-	t.Parallel()
-
-	_, err := config.FromEnvironment(
-		environment{rateLimitExemptPaths: "/favicon.ico,assets/"}.lookupEnv)
-
-	want := rateLimitExemptPaths + `: "assets/" is not a path prefix ` +
-		`starting with /, such as /assets/`
-	if err == nil || err.Error() != want {
-		t.Errorf("error %v, want %s", err, want)
-	}
-}
-
-func TestInstanceNameAndLoggedHeadersAsSet(t *testing.T) {
-	t.Parallel()
-
-	cfg := fromEnvironment(t, environment{
-		instanceName:      "fsn1app1/gitea",
-		logRequestHeaders: " Accept , X-Custom",
-	})
-
-	if cfg.InstanceName != "fsn1app1/gitea" ||
-		!slices.Equal(cfg.LogRequestHeaders, []string{"accept", "x-custom"}) {
-		t.Errorf("%s is %q and %s gives %v", instanceName, cfg.InstanceName,
-			logRequestHeaders, cfg.LogRequestHeaders)
-	}
 }
 
 func TestCodeOnBothCountryListsStopsTheStart(t *testing.T) {
@@ -357,7 +295,9 @@ func TestInvalidValueStopsTheStart(t *testing.T) {
 	t.Parallel()
 
 	for _, tc := range []struct{ name, value string }{
-		{listenAddr, "8080"}, {listenAddr, ":http"}, {listenAddr, ":65536"},
+		{listenAddr, "8080"},
+		{listenAddr, ":http"},
+		{listenAddr, ":65536"},
 		{upstreamURL, "127.0.0.1:8081"},
 		{upstreamURL, "ftp://127.0.0.1:8081"},
 		{upstreamURL, "http://"},
@@ -367,7 +307,6 @@ func TestInvalidValueStopsTheStart(t *testing.T) {
 		{upstreamURL, "http://127.0.0.1:8081/app"},
 		{upstreamURL, "http://127.0.0.1:8081/?a=1"},
 		{upstreamURL, "http://user:secret@127.0.0.1:8081"},
-		{mode, "Observe"}, {mode, "block"}, {mode, ""},
 		{trustedProxies, "10.0.0.0/33"},
 		{trustedProxies, "traefik"},
 		{trustedProxies, "10.0.0.0/8,,192.168.0.0/16"},
@@ -375,7 +314,8 @@ func TestInvalidValueStopsTheStart(t *testing.T) {
 		{allowNets, "192.0.2.0/24,monitoring"},
 		{rateLimitExemptNets, "2001:db8::/129"},
 		{denyNets, "198.51.100.0/24,"},
-		{clientRequestTimeout, "60"}, {clientRequestTimeout, ""},
+		{clientRequestTimeout, "60"},
+		{clientRequestTimeout, ""},
 		{clientIdleTimeout, "0s"},
 		{clientIdleTimeout, "2 minutes"},
 		{clientResponseTimeout, "1y"},
@@ -392,8 +332,8 @@ func TestInvalidValueStopsTheStart(t *testing.T) {
 		{rateLimitPerMinute, "1K"},
 		{rateLimitPerHour, "0"},
 		{rateLimitPerHour, "1.5"},
-		{rateLimitPerDay, "-1"}, {rateLimitPerDay, "lots"},
-		{rateLimitExemptPaths, "/assets/,,/static/"},
+		{rateLimitPerDay, "-1"},
+		{rateLimitPerDay, "lots"},
 		{deniedCountries, "nk"},
 		{deniedCountries, "kp,,ir"},
 		{deniedCountries, "prk"},
@@ -416,10 +356,6 @@ func TestInvalidValueStopsTheStart(t *testing.T) {
 		{stateWriteDelay, off}, {stateWriteDelay, "0s"},
 		{stateCounterInterval, off}, {stateCounterInterval, "15"},
 		{metricsTopN, off}, {metricsTopN, "0"}, {metricsTopN, "-1"},
-		{logRequestHeaders, "accept,,origin"}, {logRequestHeaders, "accept;origin"},
-		{logRequestHeaders, "accept language"}, {logRequestHeaders, "x-foo:"},
-		{logRequestHeaders, "host"}, {logRequestHeaders, "accept,Host"},
-		{logRequestHeaders, "transfer-encoding"}, {logRequestHeaders, "TRANSFER-ENCODING"},
 	} {
 		t.Run(tc.name+"="+tc.value, func(t *testing.T) {
 			t.Parallel()
@@ -436,27 +372,6 @@ func TestInvalidValueStopsTheStart(t *testing.T) {
 	}
 }
 
-func TestHostOrTransferEncodingStopsTheStart(t *testing.T) {
-	t.Parallel()
-
-	// Only Host's message points to the field host.
-	for value, want := range map[string]string{
-		"Host": `"Host" is taken out of every request by Go's HTTP server, ` +
-			"so it can never be logged; the request's host is the field host",
-		"transfer-encoding": `"transfer-encoding" is taken out of every ` +
-			"request by Go's HTTP server, so it can never be logged",
-	} {
-		t.Run(value, func(t *testing.T) {
-			t.Parallel()
-
-			_, err := config.FromEnvironment(environment{logRequestHeaders: value}.lookupEnv)
-			if err == nil || err.Error() != logRequestHeaders+": "+want {
-				t.Errorf("error %v, want %s: %s", err, logRequestHeaders, want)
-			}
-		})
-	}
-}
-
 func TestShortTokenStopsTheStartWithoutShowingIt(t *testing.T) {
 	t.Parallel()
 
@@ -508,12 +423,9 @@ func TestLogsEachSettingWithItsValue(t *testing.T) {
 		t.Fatalf("decode %s: %v", out.Bytes(), err)
 	}
 
-	hostname, _ := os.Hostname()
-
 	want := map[string]string{
 		listenAddr:              ":8080",
 		upstreamURL:             "http://127.0.0.1:8081",
-		mode:                    "enforce",
 		trustedProxies:          "10.0.0.0/8,172.16.0.0/12,192.168.0.0/16",
 		clientRequestTimeout:    "45s",
 		clientHeaderMaxBytes:    "32K",
@@ -529,7 +441,6 @@ func TestLogsEachSettingWithItsValue(t *testing.T) {
 		rateLimitPerMinute:      "1000",
 		rateLimitPerHour:        "10000",
 		rateLimitPerDay:         "50000",
-		rateLimitExemptPaths:    "",
 		deniedCountries:         "",
 		allowedCountries:        "",
 		banResponse:             "403",
@@ -543,8 +454,6 @@ func TestLogsEachSettingWithItsValue(t *testing.T) {
 		stateCounterInterval:    "15m",
 		metricsToken:            "",
 		metricsTopN:             "50",
-		instanceName:            hostname,
-		logRequestHeaders:       defaultLogRequestHeaders,
 	}
 	if !maps.Equal(line.Settings, want) {
 		t.Errorf("logged settings\n%v\nwant\n%v", line.Settings, want)
@@ -556,7 +465,6 @@ func wantSettings(t *testing.T, got *config.Config, want config.Config) {
 	t.Helper()
 
 	if got.ListenAddr != want.ListenAddr ||
-		got.Observe != want.Observe ||
 		got.ClientRequestTimeout != want.ClientRequestTimeout ||
 		got.ClientRequestHeaderMaxBytes != want.ClientRequestHeaderMaxBytes ||
 		got.ClientIdleTimeout != want.ClientIdleTimeout ||

internal/proxy/bans.go

@@ -14,15 +14,10 @@ func (rq *request) banResponse(action string) *refusal {
 	return &refusal{status: rq.h.config.BanResponse, action: action}
 }
 
-// banned reports whether a ban on a netblock the client is in covers the
-// request at now, and notes for the log line when that ban ends.
+// banned reports whether a ban on a netblock the client is in refuses
+// the request at now, and notes for the log line when that ban ends.
 func (rq *request) banned(now time.Time) bool {
-	check := rq.h.ledger.Check
-	if rq.h.config.Observe {
-		check = rq.h.ledger.Find // in observe mode the ban refuses nothing
-	}
-
-	ban, banned := check(rq.client, now)
+	ban, banned := rq.h.ledger.Check(rq.client, now)
 	if banned {
 		rq.line.BanExpires = banExpires(ban)
 	}
@@ -30,28 +25,17 @@ func (rq *request) banned(now time.Time) bool {
 	return banned
 }
 
-// limitBroken counts the request for the rate limits at now, notes the
-// client's counts for the log line, and reports whether the request takes
-// the client over a limit. In enforce mode such a request bans the
-// client's netblock, and sets the client's counters back to zero; in
-// observe mode it does neither.
+// limitBroken counts the request for the rate limits at now, and reports
+// whether it takes the client over one. Such a request bans the client's
+// netblock, and sets the client's counters back to zero.
 func (rq *request) limitBroken(now time.Time) bool {
 	group := clientGroup(rq.client)
 
-	counts, hit, over := rq.h.limiter.Count(group, now)
-	rq.line.Counts = counts
-
+	hit, over := rq.h.limiter.Count(group, now)
 	if !over {
 		return false
 	}
 
-	rq.line.LimitHit = hit.Window
-	rq.line.Offence = requestlog.OffenceLimit
-
-	if rq.h.config.Observe {
-		return true
-	}
-
 	netblock := rq.netblock()
 	ban := rq.h.ledger.BanForLimit(netblock, now, bans.Notes{
 		Country: rq.line.Country,
@@ -70,6 +54,9 @@ func (rq *request) limitBroken(now time.Time) bool {
 		Requests: rq.h.limiter.Requests(netblock) + 1,
 	})
 	rq.h.limiter.Reset(group)
+
+	rq.line.LimitHit = hit.Window
+	rq.line.Offence = requestlog.OffenceLimit
 	rq.line.BanExpires = banExpires(ban)
 
 	return true

internal/proxy/client.go

@@ -1,7 +1,6 @@
 package proxy
 
 import (
-	"crypto/rand"
 	"net/http"
 	"net/netip"
 	"slices"
@@ -49,33 +48,6 @@ func clientAddress(
 	return client
 }
 
-// requestIDHeader carries the request's id, from traefik and to the app.
-const requestIDHeader = "X-Request-ID"
-
-// requestID is the request's id: the one a trusted proxy sent, or a new
-// random one. A peer outside the trusted proxies did not come through
-// traefik, so the id it sends is its own claim, and is replaced.
-func requestID(r *http.Request, peerTrusted bool) string {
-	id := r.Header.Get(requestIDHeader)
-	if !peerTrusted || id == "" {
-		id = rand.Text()
-	}
-
-	return id
-}
-
-// scheme is how the client reached traefik, as a trusted proxy says in
-// X-Forwarded-Proto, or otherwise http, the only scheme smallwebwaf
-// serves.
-func scheme(r *http.Request, peerTrusted bool) string {
-	proto := r.Header.Get("X-Forwarded-Proto")
-	if !peerTrusted || proto == "" {
-		return "http"
-	}
-
-	return proto
-}
-
 // ipv6GroupPrefix is the length of the IPv6 netblock that is one client.
 const ipv6GroupPrefix = 64
 

internal/proxy/client_test.go

@@ -14,14 +14,10 @@ const (
 	appHost = "app.example"
 	// client is the client's address, as a proxy names it.
 	client = "203.0.113.9"
-	// forwardedFor is the header that lists the client and its proxies,
-	// and forwardedProto the one that gives the scheme the client used.
-	forwardedFor   = "X-Forwarded-For"
-	forwardedProto = "X-Forwarded-Proto"
-	// secure is the scheme a client reached traefik with, and plain the
-	// one smallwebwaf serves.
+	// forwardedFor is the header that lists the client and its proxies.
+	forwardedFor = "X-Forwarded-For"
+	// secure is the scheme a client reached traefik with.
 	secure = "https"
-	plain  = "http"
 )
 
 // appHeaders is what the app tells about the headers it received.
@@ -69,13 +65,13 @@ func TestClientAddressAndForwardedHeaders(t *testing.T) {
 func clientAddressCases() []clientAddressCase {
 	trusted := map[string]string{trustedProxies: trustLocalhost}
 	forged := http.Header{
-		forwardedFor:       {client},
-		"X-Forwarded-Host": {"forged.example"},
-		forwardedProto:     {secure},
-		"X-Real-Ip":        {client},
+		forwardedFor:        {client},
+		"X-Forwarded-Host":  {"forged.example"},
+		"X-Forwarded-Proto": {secure},
+		"X-Real-Ip":         {client},
 	}
 	replaced := appHeaders{
-		ForwardedFor: localhost, ForwardedHost: appHost, ForwardedProto: plain,
+		ForwardedFor: localhost, ForwardedHost: appHost, ForwardedProto: "http",
 	}
 
 	return []clientAddressCase{{
@@ -91,10 +87,10 @@ func clientAddressCases() []clientAddressCase {
 			"outside the trusted proxies from the right",
 		env: trusted,
 		header: http.Header{
-			forwardedFor:       {"198.51.100.7, " + client + ", 10.0.0.2"},
-			"X-Forwarded-Host": {appHost},
-			forwardedProto:     {secure},
-			"X-Real-Ip":        {client},
+			forwardedFor:        {"198.51.100.7, " + client + ", 10.0.0.2"},
+			"X-Forwarded-Host":  {appHost},
+			"X-Forwarded-Proto": {secure},
+			"X-Real-Ip":         {client},
 		},
 		wantClient: client,
 		wantApp: appHeaders{
@@ -142,7 +138,7 @@ func requestWithHeaders(
 			Host:           r.Host,
 			ForwardedFor:   r.Header.Get(forwardedFor),
 			ForwardedHost:  r.Header.Get("X-Forwarded-Host"),
-			ForwardedProto: r.Header.Get(forwardedProto),
+			ForwardedProto: r.Header.Get("X-Forwarded-Proto"),
 			RealIP:         r.Header.Get("X-Real-IP"),
 		})
 	})

internal/proxy/health_test.go

@@ -21,18 +21,14 @@ func TestHealthEndpointIsAnsweredBeforeAnyCheck(t *testing.T) {
 	// the last one would have it refused.
 	addr, out := startProxy(t, app.URL, map[string]string{rateLimitPerMinute: "1"})
 
-	const (
-		healthChecks = 3
-		contentType  = "text/plain; charset=utf-8"
-	)
+	const healthChecks = 3
 
 	for range healthChecks {
 		got := get(t, addr, proxy.HealthPath)
 		wantStatus(t, got, http.StatusOK)
 
-		if string(got.body) != "ok\n" || got.header.Get("Content-Type") != contentType {
-			t.Errorf("health endpoint answered %q with Content-Type %q, want ok "+
-				"with %q", got.body, got.header.Get("Content-Type"), contentType)
+		if string(got.body) != "ok\n" {
+			t.Errorf("health endpoint answered %q, want ok", got.body)
 		}
 	}
 
@@ -41,11 +37,6 @@ func TestHealthEndpointIsAnsweredBeforeAnyCheck(t *testing.T) {
 	lines := out.requestLines(t, healthChecks+1)
 	for _, line := range lines[:healthChecks] {
 		wantLine(t, line, http.StatusOK, requestlog.ActionAdmin)
-
-		if line.ResponseContentType != contentType {
-			t.Errorf("health check's log line has response_content_type %q, "+
-				"want %q", line.ResponseContentType, contentType)
-		}
 	}
 
 	wantLine(t, lines[healthChecks], http.StatusOK, requestlog.ActionForward)

internal/proxy/observe_test.go

@@ -1,202 +0,0 @@
-package proxy_test
-
-import (
-	"bytes"
-	"io"
-	"net/http"
-	"net/netip"
-	"sync/atomic"
-	"testing"
-	"time"
-
-	"sneak.berlin/go/smallwebwaf/internal/bans"
-	"sneak.berlin/go/smallwebwaf/internal/proxy"
-	"sneak.berlin/go/smallwebwaf/internal/requestlog"
-)
-
-// observe is the value of SWWAF_MODE for observe mode.
-const observe = "observe"
-
-func TestObserveModeForwardsWhatEnforceModeRefuses(t *testing.T) {
-	t.Parallel()
-
-	const (
-		denied = "192.0.2.50" // in SWWAF_DENY_NETS
-		banned = otherClient  // under a ban read from bans.json
-	)
-
-	for _, tc := range []struct {
-		setting string // "" leaves SWWAF_MODE at its default
-		observe bool
-	}{
-		{"", false},
-		{"enforce", false},
-		{observe, true},
-	} {
-		t.Run(mode+"="+tc.setting, func(t *testing.T) {
-			t.Parallel()
-
-			geojsURL, _ := startGeoJS(t)
-			env := map[string]string{
-				rateLimitPerMinute: "1",
-				denyNets:           denied,
-				deniedCountries:    "kp",
-			}
-
-			if tc.setting != "" {
-				env[mode] = tc.setting
-			}
-
-			s, clk, server := startWithClock(t, geojsURL, env)
-			server.Ledger.Load([]bans.Ban{{
-				Netblock: netip.MustParsePrefix(banned + "/32"),
-				Start:    clk.Now(),
-				Expires:  clk.Now().Add(time.Hour),
-			}})
-
-			// fromDE's first request is within the limit of one a minute,
-			// and its second breaks it.
-			s.get(fromDE, http.StatusOK, requestlog.ActionForward)
-
-			for _, sent := range []struct{ from, refusal string }{
-				{denied, requestlog.ActionDenied},
-				{banned, requestlog.ActionBanned},
-				{fromKP, requestlog.ActionCountryDenied},
-				{fromDE, requestlog.ActionRateLimited},
-			} {
-				if !tc.observe {
-					line := s.get(sent.from, http.StatusForbidden, sent.refusal)
-					wantWouldAction(t, line, "")
-
-					continue
-				}
-
-				// Passed to the app, which answered it.
-				line := s.get(sent.from, http.StatusOK, requestlog.ActionForward)
-				wantWouldAction(t, line, sent.refusal)
-
-				if line.UpstreamStatus != http.StatusOK {
-					t.Errorf("log line has upstream_status %d, want 200",
-						line.UpstreamStatus)
-				}
-			}
-		})
-	}
-}
-
-func TestObserveModeMakesNoBanAndKeepsTheBansItHas(t *testing.T) {
-	t.Parallel()
-
-	s, clk, server := startWithClock(t, "", map[string]string{
-		mode:               observe,
-		rateLimitPerMinute: "1",
-	})
-	kept := bans.Ban{
-		Netblock: netip.MustParsePrefix(otherClient + "/32"),
-		Start:    clk.Now(),
-		Expires:  clk.Now().Add(time.Hour),
-	}
-	server.Ledger.Load([]bans.Ban{kept})
-
-	// No ban sets client's counters back to zero, so each request after
-	// the first breaks the limit of one a minute.
-	s.get(client, http.StatusOK, requestlog.ActionForward)
-
-	for range 2 {
-		line := s.get(client, http.StatusOK, requestlog.ActionForward)
-		wantWouldAction(t, line, requestlog.ActionRateLimited)
-
-		if line.LimitHit != minute || line.Offence != requestlog.OffenceLimit ||
-			line.BanExpires != "" {
-			t.Errorf("log line has limit_hit %q, offence %q and ban_expires %q, "+
-				"want minute, limit and none", line.LimitHit, line.Offence,
-				line.BanExpires)
-		}
-	}
-
-	// The ban read from bans.json refuses nothing, and so counts no
-	// refusal in its notes, but is kept.
-	line := s.get(otherClient, http.StatusOK, requestlog.ActionForward)
-	wantWouldAction(t, line, requestlog.ActionBanned)
-
-	if line.BanExpires != requestlog.FormatTime(kept.Expires) {
-		t.Errorf("log line has ban_expires %q, want %s", line.BanExpires,
-			requestlog.FormatTime(kept.Expires))
-	}
-
-	got := server.Ledger.Snapshot()
-	if len(got) != 1 || got[0] != kept {
-		t.Errorf("bans\n%+v\nwant only\n%+v", got, kept)
-	}
-}
-
-func TestObserveModeKeepsTheSizeLimitsAndTheToken(t *testing.T) {
-	t.Parallel()
-
-	const denied = "192.0.2.50" // in SWWAF_DENY_NETS
-
-	var calls atomic.Int32
-
-	app := startApp(t, func(w http.ResponseWriter, _ *http.Request) {
-		calls.Add(1)
-		answerWithSize(w, 2*sizeLimit, true)
-	})
-	addr, out := startProxy(t, app.URL, map[string]string{
-		mode:             observe,
-		trustedProxies:   trustLocalhost,
-		denyNets:         denied,
-		requestMaxBytes:  sizeLimitSetting,
-		responseMaxBytes: sizeLimitSetting,
-		metricsToken:     token,
-	})
-
-	// SWWAF_DENY_NETS would refuse each request; instead a size limit or
-	// the missing token does.
-	for i, tc := range []struct {
-		method, path string
-		body         io.Reader
-		status       int
-		action       string
-	}{
-		{
-			http.MethodPost, "/upload", bytes.NewReader(make([]byte, 2*sizeLimit)),
-			http.StatusRequestEntityTooLarge, requestlog.ActionTooLarge,
-		},
-		{
-			http.MethodGet, "/download", http.NoBody,
-			http.StatusBadGateway, requestlog.ActionTooLarge,
-		},
-		{
-			http.MethodGet, proxy.MetricsPath, http.NoBody,
-			http.StatusUnauthorized, requestlog.ActionAdmin,
-		},
-	} {
-		req := newRequest(t, tc.method, addr, tc.path, tc.body)
-		req.Header.Set(forwardedFor, denied)
-		wantStatus(t, do(t, req), tc.status)
-
-		line := out.requestLines(t, i+1)[i]
-		wantLine(t, line, tc.status, tc.action)
-		wantWouldAction(t, line, requestlog.ActionDenied)
-	}
-
-	// The upload was refused before it reached the app.
-	if calls.Load() != 1 {
-		t.Errorf("the app was called %d times, want once", calls.Load())
-	}
-}
-
-// wantWouldAction checks the request log line's would_action, and that a
-// line that should have none has no such field.
-func wantWouldAction(t *testing.T, line logLine, want string) {
-	t.Helper()
-
-	got, present := line.fields["would_action"]
-
-	switch {
-	case want == "" && present:
-		t.Errorf("log line has would_action %v, want none", got)
-	case want != "" && got != want:
-		t.Errorf("log line has would_action %v, want %s", got, want)
-	}
-}

internal/proxy/passthrough_test.go

@@ -6,8 +6,6 @@ import (
 	"errors"
 	"io"
 	"net/http"
-	"os"
-	"reflect"
 	"slices"
 	"strings"
 	"sync/atomic"
@@ -16,7 +14,6 @@ import (
 
 	"sneak.berlin/go/smallwebwaf/internal/config"
 	"sneak.berlin/go/smallwebwaf/internal/proxy"
-	"sneak.berlin/go/smallwebwaf/internal/ratelimit"
 	"sneak.berlin/go/smallwebwaf/internal/requestlog"
 )
 
@@ -118,34 +115,27 @@ func wantAnswer(t *testing.T, got answer, body []byte) {
 	}
 }
 
-// wantRequestFields checks the log line's fields about the request. Its
-// time, its id and its timings are checked only for being there.
+// wantRequestFields checks the log line's fields about the request.
 func wantRequestFields(t *testing.T, line logLine, host string, sent, received int) {
 	t.Helper()
 
-	hostname, _ := os.Hostname()
-
-	want := withTimings(line, requestlog.Line{
-		Type: requestType, Time: line.Time, Instance: hostname,
-		ClientIP: localhost, Method: http.MethodPatch, Scheme: plain, Host: host,
-		Path: rawPath, Query: rawQuery, Protocol: protocol,
-		Status: http.StatusTeapot, RequestBytes: int64(sent),
+	want := requestlog.Line{
+		Type: "request", Time: line.Time, ClientIP: localhost, PeerIP: localhost,
+		Method: http.MethodPatch, Host: host, Path: rawPath, Query: rawQuery,
+		Protocol: "HTTP/1.1", Status: http.StatusTeapot,
+		UpstreamStatus: http.StatusTeapot, RequestBytes: int64(sent),
 		ResponseBytes: int64(received), UserAgent: "test-agent",
-		RequestID: line.RequestID, PeerIP: localhost, ClientGroup: localhost + "/32",
-		ContentLength: int64(sent), ResponseContentType: "text/plain; charset=utf-8",
-		UpstreamStatus: http.StatusTeapot, Action: requestlog.ActionForward,
-		Counts: ratelimit.Counts{Minute: 1, Hour: 1, Day: 1},
-	})
-	if !reflect.DeepEqual(line.Line, want) {
+		Action: requestlog.ActionForward, DurationTotal: line.DurationTotal,
+		DurationUpstreamTotal: line.DurationUpstreamTotal,
+	}
+	if line.Line != want {
 		t.Errorf("log line\n%+v\nwant\n%+v", line.Line, want)
 	}
 
 	_, err := time.Parse(time.RFC3339, line.Time)
-	if err != nil || line.RequestID == "" || line.DurationTotal <= 0 ||
-		line.DurationUpstreamTotal == nil || *line.DurationUpstreamTotal <= 0 {
-		t.Errorf("log line has time %q, request_id %q and durations %v and %v",
-			line.Time, line.RequestID, line.DurationTotal,
-			line.fields["duration_upstream_total"])
+	if err != nil || line.DurationTotal <= 0 || line.DurationUpstreamTotal <= 0 {
+		t.Errorf("log line has time %q and durations %v and %v",
+			line.Time, line.DurationTotal, line.DurationUpstreamTotal)
 	}
 }
 
@@ -381,13 +371,8 @@ func TestAnswers502WhenTheAppCannotBeReached(t *testing.T) {
 	addr, out := startProxy(t, "http://"+localhost+":1", nil)
 
 	wantStatus(t, get(t, addr, "/"), http.StatusBadGateway)
-
-	line := out.requestLine(t)
-	wantLine(t, line, http.StatusBadGateway, requestlog.ActionUpstreamError)
-
-	// There never was a connection to the app, nor an answer from it.
-	wantTimings(t, line, "duration_total", "duration_checks",
-		"duration_upstream_total")
+	wantLine(t, out.requestLine(t), http.StatusBadGateway,
+		requestlog.ActionUpstreamError)
 
 	logged := slices.ContainsFunc(out.lines(t), func(line map[string]any) bool {
 		return line["type"] == "process" && line["msg"] == "request to the app failed"

internal/proxy/proxy.go

@@ -160,9 +160,6 @@ func (h *handler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
 	// a health checker is never refused. It does not ask the app.
 	if r.Method == http.MethodGet && r.URL.Path == HealthPath {
 		rq.line.Action = requestlog.ActionAdmin
-		// Set here rather than left to Go's server, which would set it only
-		// after the log line has taken the response's headers.
-		rq.out.Header().Set("Content-Type", "text/plain; charset=utf-8")
 		_, _ = io.WriteString(rq.out, "ok\n")
 
 		return
@@ -172,8 +169,6 @@ func (h *handler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
 	defer rq.addToHistory()
 
 	refused := rq.check(r.Context())
-	rq.checked = time.Now()
-
 	if refused != nil {
 		rq.answer(*refused)
 

internal/proxy/proxy_test.go

@@ -35,10 +35,6 @@ const (
 	// localhost is where every test server listens, and so the address
 	// smallwebwaf sees each test's requests come from.
 	localhost = "127.0.0.1"
-	// requestType is the type that marks a request log line.
-	requestType = "request"
-	// protocol is the protocol of every test's requests.
-	protocol = "HTTP/1.1"
 )
 
 // shortTimeoutSetting is shortTimeout as a setting's value.
@@ -54,7 +50,6 @@ const (
 	clientResponseTimeout   = "SWWAF_CLIENT_RESPONSE_TIMEOUT"
 	upstreamRequestTimeout  = "SWWAF_UPSTREAM_REQUEST_TIMEOUT"
 	upstreamResponseTimeout = "SWWAF_UPSTREAM_RESPONSE_TIMEOUT"
-	mode                    = "SWWAF_MODE"
 	requestMaxBytes         = "SWWAF_REQUEST_MAX_BYTES"
 	responseMaxBytes        = "SWWAF_RESPONSE_MAX_BYTES"
 	trustedProxies          = "SWWAF_TRUSTED_PROXIES"
@@ -63,7 +58,6 @@ const (
 	denyNets                = "SWWAF_DENY_NETS"
 	rateLimitPerMinute      = "SWWAF_RATE_LIMIT_PER_MINUTE"
 	rateLimitPerDay         = "SWWAF_RATE_LIMIT_PER_DAY"
-	rateLimitExemptPaths    = "SWWAF_RATE_LIMIT_EXEMPT_PATHS"
 	deniedCountries         = "SWWAF_DENIED_COUNTRIES"
 	allowedCountries        = "SWWAF_EXCLUSIVELY_ALLOWED_COUNTRIES"
 	banResponse             = "SWWAF_BAN_RESPONSE"
@@ -72,8 +66,6 @@ const (
 	maxBanDuration          = "SWWAF_MAX_BAN_DURATION"
 	maxBans                 = "SWWAF_MAX_BANS"
 	banScopeV4Prefix        = "SWWAF_BAN_SCOPE_V4_PREFIX"
-	instanceName            = "SWWAF_INSTANCE_NAME"
-	logRequestHeaders       = "SWWAF_LOG_REQUEST_HEADERS"
 )
 
 // output collects what smallwebwaf writes on stdout.
@@ -90,14 +82,6 @@ func (o *output) Write(p []byte) (int, error) {
 	return o.buf.Write(p)
 }
 
-// text returns everything written so far.
-func (o *output) text() string {
-	o.mu.Lock()
-	defer o.mu.Unlock()
-
-	return o.buf.String()
-}
-
 // lines returns every line written so far, decoded.
 func (o *output) lines(t *testing.T) []map[string]any {
 	t.Helper()
@@ -137,7 +121,7 @@ func (o *output) requestLines(t *testing.T, count int) []logLine {
 		var found []logLine
 
 		for _, fields := range o.lines(t) {
-			if fields["type"] == requestType {
+			if fields["type"] == "request" {
 				found = append(found, decodeLine(t, fields))
 			}
 		}

internal/proxy/ratelimits_test.go

@@ -5,8 +5,6 @@ import (
 	"sync/atomic"
 	"testing"
 
-	"sneak.berlin/go/smallwebwaf/internal/lookup"
-	"sneak.berlin/go/smallwebwaf/internal/ratelimit"
 	"sneak.berlin/go/smallwebwaf/internal/requestlog"
 )
 
@@ -71,89 +69,3 @@ func TestRateLimitRefusesBeforeTheApp(t *testing.T) {
 		t.Errorf("the app was called %d times, want 4", calls.Load())
 	}
 }
-
-func TestRateLimitExemptPathsAreNeitherCountedNorRefused(t *testing.T) {
-	t.Parallel()
-
-	const denied = "192.0.2.50" // in SWWAF_DENY_NETS
-
-	s, _, server := startWithClock(t, "", map[string]string{
-		rateLimitPerMinute:   "1",
-		rateLimitExemptPaths: "/assets/,/favicon.ico",
-		denyNets:             denied,
-		deniedCountries:      "kp",
-	})
-
-	// The answers are kept before the requests, so that none waits for
-	// GeoJS.
-	server.GeoJS.Load([]lookup.Answer{
-		keptAnswer(client, "DE"), keptAnswer(fromKP, "KP"),
-	})
-
-	// With a limit of one request a minute, the requests for paths under a
-	// prefix are not counted, so client's first request for / is within
-	// the limit; and once client has reached it, they are not refused.
-	s.request(client, "/assets/app.js", http.StatusOK, requestlog.ActionForward)
-	s.request(client, "/favicon.ico?v=2", http.StatusOK, requestlog.ActionForward)
-	s.get(client, http.StatusOK, requestlog.ActionForward)
-
-	line := s.request(client, "/assets/app.js", http.StatusOK, requestlog.ActionForward)
-	if line.LimitHit != "" || line.Counts != (ratelimit.Counts{}) {
-		t.Errorf("log line has limit_hit %q and counts %+v, want neither",
-			line.LimitHit, line.Counts)
-	}
-
-	// A path outside every prefix is counted: /assets is not under
-	// /assets/, and breaks the limit.
-	s.request(client, "/assets", http.StatusForbidden, requestlog.ActionRateLimited)
-
-	// A ban, SWWAF_DENY_NETS and the country lists still refuse a path
-	// under a prefix.
-	s.request(client, "/assets/app.js", http.StatusForbidden, requestlog.ActionBanned)
-	s.request(denied, "/assets/app.js", http.StatusForbidden, requestlog.ActionDenied)
-	s.request(fromKP, "/assets/app.js",
-		http.StatusForbidden, requestlog.ActionCountryDenied)
-}
-
-func TestRateLimitCountsPathsThatAreNotExempt(t *testing.T) {
-	t.Parallel()
-
-	for _, sent := range []string{
-		// A prefix matches only at the start of the path.
-		"/static/assets/app.js",
-		// A prefix matches the path as sent: a router that matches the
-		// path as received does not take /%61ssets/x for a path under
-		// /assets/.
-		"/%61ssets/x",
-		// .. once percent-decoded: an app may act on these as /login, the
-		// last as a path under /sneak/app/ or as /assets/x.
-		"/assets/../login",
-		"/assets/%2e%2e/login",
-		"/assets/..%2Flogin",
-		"/assets/..;/login",
-		"/sneak/app/src/branch/main/..%2F..%2F..%2F..%2F..%2F..%2Fassets/x",
-		// Not under /assets/ as sent: Go's router takes /assets%2Fx for one
-		// path segment, not a path under /assets/.
-		"/assets%2Fx",
-		"/assets%2fx",
-		// Under /assets/ as sent, but holding an encoded slash, in either
-		// case, or a backslash: never exempt, whatever the prefix.
-		"/assets/x%2Fy",
-		"/assets/x%2fy",
-		`/assets/x\y`,
-	} {
-		t.Run(sent, func(t *testing.T) {
-			t.Parallel()
-
-			s, _, _ := startWithClock(t, "", map[string]string{
-				rateLimitPerMinute:   "1",
-				rateLimitExemptPaths: "/assets/",
-			})
-
-			// Counted, the second request breaks the limit of one request
-			// a minute.
-			s.request(client, sent, http.StatusOK, requestlog.ActionForward)
-			s.request(client, sent, http.StatusForbidden, requestlog.ActionRateLimited)
-		})
-	}
-}

internal/proxy/request.go

@@ -7,10 +7,7 @@ import (
 	"net/http/httptrace"
 	"net/http/httputil"
 	"net/netip"
-	"net/url"
 	"os"
-	"slices"
-	"strings"
 	"sync"
 	"sync/atomic"
 	"time"
@@ -49,9 +46,7 @@ type request struct {
 	peer        netip.Addr
 	peerTrusted bool
 	start       time.Time
-	// checked is when the checks were done, and upstreamStart when the
-	// request was handed to the app.
-	checked       time.Time
+	// upstreamStart is when the request was handed to the app.
 	upstreamStart time.Time
 	// cancel ends the request to the app.
 	cancel context.CancelFunc
@@ -61,34 +56,26 @@ type request struct {
 	complete bool
 
 	// mu guards what follows. The timeouts run on goroutines of their
-	// own, and the transport starts and stops them, and notes the times
-	// below, from its own; once timersStopped is set, none of the timeouts
-	// acts any more.
+	// own, and the transport starts and stops them from its own; once
+	// timersStopped is set, none of them acts any more.
 	mu                    sync.Mutex
 	timersStopped         bool
 	clientRequestTimer    *time.Timer
 	upstreamRequestTimer  *time.Timer
 	upstreamResponseTimer *time.Timer
-	// connected is when there was a connection to the app, requestSent
-	// when the app had been sent the whole request, and answerStarted
-	// when the first byte of its answer arrived.
-	connected     time.Time
-	requestSent   time.Time
-	answerStarted time.Time
+	// requestSent is when the app had been sent the whole request.
+	requestSent time.Time
 }
 
 // newRequest starts handling r: it notes the time, counts the request as
-// under way, works out the client, and starts the log line with what is
-// known of the request.
+// under way, and works out the client.
 func (h *handler) newRequest(w http.ResponseWriter, r *http.Request) *request {
 	h.metrics.RequestStarted()
 
 	start := time.Now()
 	peer := peerAddress(r)
 	trusted := h.config.TrustedProxies
-	peerTrusted := isInside(peer, trusted)
-	forwardedFor := r.Header.Values("X-Forwarded-For")
-	client := clientAddress(peer, forwardedFor, trusted)
+	client := clientAddress(peer, r.Header.Values("X-Forwarded-For"), trusted)
 
 	rq := &request{
 		h:           h,
@@ -97,37 +84,22 @@ func (h *handler) newRequest(w http.ResponseWriter, r *http.Request) *request {
 		out:         &responseWriter{ResponseWriter: w},
 		client:      client,
 		peer:        peer,
-		peerTrusted: peerTrusted,
+		peerTrusted: isInside(peer, trusted),
 		start:       start,
 		line: requestlog.Line{
-			Time:             requestlog.FormatTime(start),
-			Instance:         h.config.InstanceName,
-			ClientIP:         client.String(),
-			Method:           r.Method,
-			Scheme:           scheme(r, peerTrusted),
-			Host:             r.Host,
-			Path:             r.URL.EscapedPath(),
-			Query:            r.URL.RawQuery,
-			Protocol:         r.Proto,
-			Referer:          r.Referer(),
-			UserAgent:        r.UserAgent(),
-			RequestID:        requestID(r, peerTrusted),
-			PeerIP:           peer.String(),
-			ForwardedFor:     strings.Join(forwardedFor, ", "),
-			ClientGroup:      clientGroup(client).String(),
-			ContentType:      r.Header.Get("Content-Type"),
-			RequestHeaders:   requestHeaders(r, h.config.LogRequestHeaders),
-			HasAuthorization: len(r.Header.Values("Authorization")) > 0,
-			HasCookie:        len(r.Header.Values("Cookie")) > 0,
-			Action:           requestlog.ActionForward,
+			Time:      requestlog.FormatTime(start),
+			ClientIP:  client.String(),
+			PeerIP:    peer.String(),
+			Method:    r.Method,
+			Host:      r.Host,
+			Path:      r.URL.EscapedPath(),
+			Query:     r.URL.RawQuery,
+			Protocol:  r.Proto,
+			Referer:   r.Referer(),
+			UserAgent: r.UserAgent(),
+			Action:    requestlog.ActionForward,
 		},
 	}
-
-	// A length of -1 is a body whose length was not announced.
-	if r.ContentLength > 0 {
-		rq.line.ContentLength = r.ContentLength
-	}
-
 	if r.Body != http.NoBody {
 		rq.body = &requestBody{body: limitBody(r.Body, h.config.RequestMaxBytes), rq: rq}
 	}
@@ -135,47 +107,40 @@ func (h *handler) newRequest(w http.ResponseWriter, r *http.Request) *request {
 	return rq
 }
 
-// requestHeaders returns the headers of r that names lists, by name in
-// lower case, each with its values joined by ", ". Authorization, Cookie
-// and Set-Cookie are never among them, whatever names says.
-func requestHeaders(r *http.Request, names []string) map[string]string {
-	headers := map[string]string{}
-
-	for _, name := range names {
-		switch name {
-		case "authorization", "cookie", "set-cookie":
-			continue
-		}
-
-		values := r.Header.Values(name)
-		if len(values) > 0 {
-			headers[name] = strings.Join(values, ", ")
-		}
-	}
-
-	return headers
-}
-
 // check is the one place where a request can be refused once its client
 // is known, before its body is read or anything reaches the app. It
-// returns nil to let the request through. The checks of checkClient come
-// first, answered with SWWAF_BAN_RESPONSE, and then the size limit, so
-// that a request the rate limits count is counted even when it is
-// refused for its size. In observe mode a request checkClient refuses
-// goes on to the size limit like any other. ctx is the request's own
-// context.
+// returns nil to let the request through. A client in SWWAF_ALLOW_NETS
+// skips every check but the size limit. For any other client,
+// SWWAF_DENY_NETS comes first, then a ban on its netblock, so that a
+// client either refuses is not looked up, and then the country lists; a
+// request any of them refuses is not counted for the rate limits. Then
+// come the rate limits, unless the client is in
+// SWWAF_RATE_LIMIT_EXEMPT_NETS, so that every other request is counted,
+// one refused for its size too. Every refusal but the size limit's is
+// answered with SWWAF_BAN_RESPONSE. ctx is the request's own context.
 func (rq *request) check(ctx context.Context) *refusal {
-	action := rq.checkClient(ctx)
-	if action != "" {
-		if !rq.h.config.Observe {
-			return rq.banResponse(action)
-		}
+	cfg := rq.h.config
+	allowed := isInside(rq.client, cfg.AllowNets)
+	exempt := isInside(rq.client, cfg.RateLimitExemptNets)
+	now := rq.h.now()
 
-		// The log line names what enforce mode would have done.
-		rq.line.WouldAction = action
+	if !allowed && isInside(rq.client, cfg.DenyNets) {
+		return rq.banResponse(requestlog.ActionDenied)
 	}
 
-	maxBytes := rq.h.config.RequestMaxBytes
+	if !allowed && rq.banned(now) {
+		return rq.banResponse(requestlog.ActionBanned)
+	}
+
+	if !allowed && rq.countryDenied(ctx) {
+		return rq.banResponse(requestlog.ActionCountryDenied)
+	}
+
+	if !allowed && !exempt && rq.limitBroken(now) {
+		return rq.banResponse(requestlog.ActionRateLimited)
+	}
+
+	maxBytes := cfg.RequestMaxBytes
 	if maxBytes > 0 && rq.in.ContentLength > maxBytes {
 		return &refusal{
 			status: http.StatusRequestEntityTooLarge,
@@ -187,69 +152,6 @@ func (rq *request) check(ctx context.Context) *refusal {
 	return nil
 }
 
-// checkClient runs the checks on the request's client, and returns the
-// action of the first that refuses the request, or "" when none does. A
-// client in SWWAF_ALLOW_NETS skips them. For any other client,
-// SWWAF_DENY_NETS comes first, then a ban on its netblock, so that a
-// client either refuses is not looked up, and then the country lists; a
-// request any of them refuses is not counted for the rate limits. Then
-// come the rate limits, unless the client is in
-// SWWAF_RATE_LIMIT_EXEMPT_NETS or the request's path is exempt under
-// SWWAF_RATE_LIMIT_EXEMPT_PATHS, so that every other request is counted.
-// ctx is the request's own context.
-func (rq *request) checkClient(ctx context.Context) string {
-	cfg := rq.h.config
-	if isInside(rq.client, cfg.AllowNets) {
-		return ""
-	}
-
-	now := rq.h.now()
-
-	if isInside(rq.client, cfg.DenyNets) {
-		return requestlog.ActionDenied
-	}
-
-	if rq.banned(now) {
-		return requestlog.ActionBanned
-	}
-
-	if rq.countryDenied(ctx) {
-		return requestlog.ActionCountryDenied
-	}
-
-	exempt := isInside(rq.client, cfg.RateLimitExemptNets) ||
-		pathExempt(rq.in.URL, cfg.RateLimitExemptPaths)
-	if !exempt && rq.limitBroken(now) {
-		return requestlog.ActionRateLimited
-	}
-
-	return ""
-}
-
-// pathExempt reports whether the rate limits leave out a request for u
-// because of SWWAF_RATE_LIMIT_EXEMPT_PATHS: whether its path as sent, the
-// path the app receives, not percent-decoded, starts with one of
-// prefixes, so that /%61ssets/x is not under /assets/ for an app whose
-// router matches the path as received. A request whose decoded path
-// contains .. anywhere or a backslash, or whose path as sent holds an
-// encoded slash (%2F or %2f), never is, since an app may act on it as a
-// path outside every prefix: /assets/..%2Flogin as /login, or /assets%2Fx
-// as one path segment, as Go's router does.
-func pathExempt(u *url.URL, prefixes []string) bool {
-	decoded := u.Path
-	// EscapedPath is the path as the app receives it, not decoded.
-	sent := u.EscapedPath()
-
-	if strings.Contains(decoded, "..") || strings.Contains(decoded, `\`) ||
-		strings.Contains(strings.ToLower(sent), "%2f") {
-		return false
-	}
-
-	return slices.ContainsFunc(prefixes, func(prefix string) bool {
-		return strings.HasPrefix(sent, prefix)
-	})
-}
-
 // forward passes the request to the app and the app's answer back. ctx
 // is the request's own context.
 func (rq *request) forward(ctx context.Context) {
@@ -258,9 +160,7 @@ func (rq *request) forward(ctx context.Context) {
 
 	rq.cancel = cancel
 	ctx = httptrace.WithClientTrace(ctx, &httptrace.ClientTrace{
-		GotConn:              rq.gotConn,
-		WroteRequest:         rq.wroteRequest,
-		GotFirstResponseByte: rq.gotFirstResponseByte,
+		WroteRequest: rq.wroteRequest,
 	})
 
 	out := rq.in.WithContext(ctx)
@@ -283,8 +183,7 @@ func (rq *request) forward(ctx context.Context) {
 }
 
 // rewrite makes the request the app receives: the client's request,
-// unchanged, sent to SWWAF_UPSTREAM_URL, with the forwarded headers and
-// the request's id set.
+// unchanged, sent to SWWAF_UPSTREAM_URL, with the forwarded headers set.
 func (rq *request) rewrite(pr *httputil.ProxyRequest) {
 	upstream := rq.h.config.UpstreamURL
 	pr.Out.URL.Scheme = upstream.Scheme
@@ -293,7 +192,6 @@ func (rq *request) rewrite(pr *httputil.ProxyRequest) {
 	// the query as the client sent it.
 	pr.Out.URL.RawQuery = pr.In.URL.RawQuery
 	setForwardedHeaders(pr.In, pr.Out, rq.peer, rq.peerTrusted)
-	pr.Out.Header.Set(requestIDHeader, rq.line.RequestID)
 }
 
 // modifyResponse looks at the app's answer before ReverseProxy passes it
@@ -307,7 +205,6 @@ func (rq *request) modifyResponse(res *http.Response) error {
 		// connection it takes over, not through rq.out.
 		rq.stopTimers()
 		rq.out.status = res.StatusCode
-		rq.line.Websocket = true
 
 		return nil
 	}
@@ -403,10 +300,6 @@ func (rq *request) finish() {
 	line := &rq.line
 	line.Status = rq.out.status
 	line.ResponseBytes = rq.out.bytes
-	header := rq.out.Header()
-	line.ResponseContentType = header.Get("Content-Type")
-	line.CacheControl = header.Get("Cache-Control")
-	line.Location = header.Get("Location")
 
 	if rq.body != nil {
 		line.RequestBytes = rq.body.bytes.Load()
@@ -431,18 +324,12 @@ func (rq *request) finish() {
 	now := time.Now()
 	duration := now.Sub(rq.start)
 	line.DurationTotal = requestlog.Milliseconds(duration)
-	line.DurationChecks = timing(rq.start, rq.checked)
 
 	var upstreamDuration time.Duration
 
 	if !rq.upstreamStart.IsZero() {
 		upstreamDuration = now.Sub(rq.upstreamStart)
-		line.DurationUpstreamTotal = new(requestlog.Milliseconds(upstreamDuration))
-
-		rq.mu.Lock()
-		line.DurationUpstreamConnect = timing(rq.upstreamStart, rq.connected)
-		line.DurationUpstreamFirstByte = timing(rq.upstreamStart, rq.answerStarted)
-		rq.mu.Unlock()
+		line.DurationUpstreamTotal = requestlog.Milliseconds(upstreamDuration)
 	}
 
 	// Counted before the log line is written, so that the metrics count
@@ -455,17 +342,6 @@ func (rq *request) finish() {
 	}
 }
 
-// timing is the time from start to end in milliseconds, for one of the
-// log line's timings, or nil when end is zero: what it times never
-// happened.
-func timing(start, end time.Time) *float64 {
-	if end.IsZero() {
-		return nil
-	}
-
-	return new(requestlog.Milliseconds(end.Sub(start)))
-}
-
 // addToHistory adds the request, which has ended, to its client's
 // history.
 func (rq *request) addToHistory() {
@@ -575,24 +451,6 @@ func (rq *request) bodyReceived() {
 	stopTimer(rq.clientRequestTimer)
 }
 
-// gotConn is called once there is a connection to the app, a new one or
-// one kept open from an earlier request.
-func (rq *request) gotConn(httptrace.GotConnInfo) {
-	rq.mu.Lock()
-	defer rq.mu.Unlock()
-
-	rq.connected = time.Now()
-}
-
-// gotFirstResponseByte is called once the first byte of the app's answer
-// has arrived.
-func (rq *request) gotFirstResponseByte() {
-	rq.mu.Lock()
-	defer rq.mu.Unlock()
-
-	rq.answerStarted = time.Now()
-}
-
 // wroteRequest is called once the app has been sent the whole request:
 // the request timeouts end and SWWAF_UPSTREAM_RESPONSE_TIMEOUT starts.
 func (rq *request) wroteRequest(info httptrace.WroteRequestInfo) {

internal/proxy/requestlog_test.go

@@ -1,368 +0,0 @@
-package proxy_test
-
-import (
-	"io"
-	"maps"
-	"math"
-	"net/http"
-	"reflect"
-	"slices"
-	"strings"
-	"testing"
-	"time"
-
-	"sneak.berlin/go/smallwebwaf/internal/proxy"
-	"sneak.berlin/go/smallwebwaf/internal/ratelimit"
-	"sneak.berlin/go/smallwebwaf/internal/requestlog"
-)
-
-const (
-	// requestIDHeader carries the request's id.
-	requestIDHeader = "X-Request-ID"
-	// instance is the SWWAF_INSTANCE_NAME a test sets.
-	instance = "fsn1app1/gitea"
-	// ipv6Client is a client on IPv6, and ipv6Group the netblock the rate
-	// limits count it as.
-	ipv6Client = "2001:db8::7"
-	ipv6Group  = "2001:db8::/64"
-)
-
-func TestLogLineHasEachFieldWhereItApplies(t *testing.T) {
-	t.Parallel()
-
-	received := make(chan string, 2) // the request ids the app received
-	app := startApp(t, func(w http.ResponseWriter, r *http.Request) {
-		received <- r.Header.Get(requestIDHeader)
-
-		_, _ = io.Copy(io.Discard, r.Body)
-
-		if r.URL.Path != "/full" {
-			w.WriteHeader(http.StatusNoContent)
-
-			return
-		}
-
-		w.Header().Set("Content-Type", "text/html")
-		w.Header().Set("Cache-Control", "no-store")
-		w.Header().Set("Location", "/elsewhere")
-		w.WriteHeader(http.StatusFound)
-		_, _ = io.WriteString(w, "moved")
-	})
-	addr, out := startProxy(t, app.URL, map[string]string{
-		trustedProxies:      trustLocalhost,
-		rateLimitExemptNets: localhost,
-		instanceName:        instance,
-		logRequestHeaders:   "Accept,x-custom,Authorization,cookie,SET-COOKIE",
-	})
-
-	// This request comes from ipv6Client through a trusted proxy, with a
-	// body and each header the log line looks at, and is answered with a
-	// redirect.
-	conn := dial(t, addr)
-	send(t, conn, "POST /full HTTP/1.1\r\nHost: "+appHost+"\r\n"+
-		forwardedFor+": 198.51.100.7, "+ipv6Client+"\r\n"+
-		forwardedProto+": "+secure+"\r\n"+requestIDHeader+": from-traefik\r\n"+
-		"Content-Type: application/x-www-form-urlencoded\r\nContent-Length: 3\r\n"+
-		"Accept: text/html\r\nX-Custom: one\r\nX-Custom: two\r\n"+
-		"Authorization: Bearer secret-token\r\nCookie: session=secret-cookie\r\n"+
-		"Set-Cookie: secret-set-cookie\r\n\r\na=b")
-	wantStatus(t, readResponse(t, conn), http.StatusFound)
-
-	// A request's log line can come after its answer: each is waited for
-	// before the next request, so that the lines are in order.
-	full := out.requestLines(t, 1)[0]
-
-	// This one comes from 127.0.0.1, which the rate limits do not count,
-	// with a body of 4 bytes whose length it does not announce, so that its
-	// request_bytes is not its content_length, and no header the log line
-	// looks at, and is answered with 204 and no header.
-	conn = dial(t, addr)
-	send(t, conn, "POST /bare HTTP/1.1\r\nHost: "+appHost+"\r\n"+
-		"Transfer-Encoding: chunked\r\n\r\n4\r\nbody\r\n0\r\n\r\n")
-	wantStatus(t, readResponse(t, conn), http.StatusNoContent)
-
-	bare := out.requestLines(t, 2)[1]
-
-	wantFullLine(t, full)
-	wantBareLine(t, bare)
-
-	for _, line := range []logLine{full, bare} {
-		got := <-received
-		if got != line.RequestID {
-			t.Errorf("the app received request id %q, the log line has %q",
-				got, line.RequestID)
-		}
-	}
-
-	if strings.Contains(out.text(), "secret") {
-		t.Errorf("a value of Authorization, Cookie or Set-Cookie is logged:\n%s",
-			out.text())
-	}
-}
-
-// wantFullLine checks the log line of the request with every header the
-// line looks at. Its timings are checked by TestTimingsAreInOrder.
-func wantFullLine(t *testing.T, line logLine) {
-	t.Helper()
-
-	headers := map[string]string{"accept": "text/html", "x-custom": "one, two"}
-
-	want := withTimings(line, requestlog.Line{
-		Type: requestType, Time: line.Time, Instance: instance,
-		ClientIP: ipv6Client, Method: http.MethodPost, Scheme: secure,
-		Host: appHost, Path: "/full", Protocol: protocol,
-		Status: http.StatusFound, RequestBytes: 3, ResponseBytes: 5,
-		RequestID: "from-traefik", PeerIP: localhost,
-		ForwardedFor: "198.51.100.7, " + ipv6Client, ClientGroup: ipv6Group,
-		ContentType: "application/x-www-form-urlencoded", ContentLength: 3,
-		RequestHeaders: headers, HasAuthorization: true, HasCookie: true,
-		ResponseContentType: "text/html", UpstreamStatus: http.StatusFound,
-		CacheControl: "no-store", Location: "/elsewhere",
-		Action: requestlog.ActionForward,
-		Counts: ratelimit.Counts{Minute: 1, Hour: 1, Day: 1},
-	})
-	if !reflect.DeepEqual(line.Line, want) {
-		t.Errorf("log line\n%+v\nwant\n%+v", line.Line, want)
-	}
-}
-
-// wantBareLine checks the log line of the request with none of them, and
-// that the fields that do not apply to it are left out.
-func wantBareLine(t *testing.T, line logLine) {
-	t.Helper()
-
-	want := withTimings(line, requestlog.Line{
-		Type: requestType, Time: line.Time, Instance: instance,
-		ClientIP: localhost, Method: http.MethodPost, Scheme: plain,
-		Host: appHost, Path: "/bare", Protocol: protocol,
-		Status: http.StatusNoContent, RequestBytes: 4, RequestID: line.RequestID,
-		PeerIP: localhost, ClientGroup: localhost + "/32",
-		UpstreamStatus: http.StatusNoContent, Action: requestlog.ActionForward,
-	})
-	if !reflect.DeepEqual(line.Line, want) || line.RequestID == "" {
-		t.Errorf("log line\n%+v\nwant\n%+v, with a request id", line.Line, want)
-	}
-
-	for _, name := range []string{
-		"forwarded_for", "content_type", "content_length", "request_headers",
-		"has_authorization", "has_cookie", "websocket", "response_content_type",
-		"cache_control", "location", "counts",
-	} {
-		_, present := line.fields[name]
-		if present {
-			t.Errorf("log line has %s, which does not apply", name)
-		}
-	}
-}
-
-// withTimings returns want with the timings of line.
-func withTimings(line logLine, want requestlog.Line) requestlog.Line {
-	want.DurationTotal = line.DurationTotal
-	want.DurationChecks = line.DurationChecks
-	want.DurationUpstreamConnect = line.DurationUpstreamConnect
-	want.DurationUpstreamFirstByte = line.DurationUpstreamFirstByte
-	want.DurationUpstreamTotal = line.DurationUpstreamTotal
-
-	return want
-}
-
-func TestHasAuthorizationAndHasCookieEachComeFromTheirOwnHeader(t *testing.T) {
-	t.Parallel()
-
-	const hasAuthorization, hasCookie = "has_authorization", "has_cookie"
-
-	for _, tc := range []struct{ header, field, other string }{
-		{"Authorization", hasAuthorization, hasCookie},
-		{"Cookie", hasCookie, hasAuthorization},
-	} {
-		t.Run("only "+tc.header, func(t *testing.T) {
-			t.Parallel()
-
-			app := startApp(t, func(http.ResponseWriter, *http.Request) {})
-			addr, out := startProxy(t, app.URL, nil)
-
-			req := newRequest(t, http.MethodGet, addr, "/", http.NoBody)
-			req.Header.Set(tc.header, "secret")
-			wantStatus(t, do(t, req), http.StatusOK)
-
-			line := out.requestLine(t)
-
-			_, otherPresent := line.fields[tc.other]
-			if line.fields[tc.field] != true || otherPresent {
-				t.Errorf("log line has %s %v and %s %v, want true and none",
-					tc.field, line.fields[tc.field], tc.other, line.fields[tc.other])
-			}
-		})
-	}
-}
-
-func TestRequestIDAndSchemeComeOnlyFromATrustedProxy(t *testing.T) {
-	t.Parallel()
-
-	const sentID = "from-traefik"
-
-	sent := http.Header{requestIDHeader: {sentID}, forwardedProto: {secure}}
-	trusted := map[string]string{trustedProxies: trustLocalhost}
-
-	for _, tc := range []struct {
-		name   string
-		env    map[string]string
-		header http.Header
-		// wantID is the request id logged, "" for a new one.
-		wantID, wantScheme string
-	}{
-		{"a trusted proxy's are kept", trusted, sent, sentID, secure},
-		{"without them, the id is new and the scheme http", trusted, nil, "", plain},
-		{"another peer's are replaced", nil, sent, "", plain},
-	} {
-		t.Run(tc.name, func(t *testing.T) {
-			t.Parallel()
-
-			received := make(chan string, 2)
-			app := startApp(t, func(_ http.ResponseWriter, r *http.Request) {
-				received <- r.Header.Get(requestIDHeader)
-			})
-			addr, out := startProxy(t, app.URL, tc.env)
-
-			// Two requests, so that two new ids can be told apart.
-			ids := make([]string, 0, 2)
-
-			for i := range 2 {
-				req := newRequest(t, http.MethodGet, addr, "/", http.NoBody)
-				maps.Copy(req.Header, tc.header)
-				wantStatus(t, do(t, req), http.StatusOK)
-
-				line := out.requestLines(t, i+1)[i]
-				ids = append(ids, line.RequestID)
-
-				got := <-received
-				if line.RequestID != got || line.Scheme != tc.wantScheme {
-					t.Errorf("log line has request_id %q and scheme %q, and the "+
-						"app received id %q; want the same id and scheme %q",
-						line.RequestID, line.Scheme, got, tc.wantScheme)
-				}
-			}
-
-			switch {
-			case tc.wantID != "" && (ids[0] != tc.wantID || ids[1] != tc.wantID):
-				t.Errorf("request ids %q, want %q", ids, tc.wantID)
-			case tc.wantID == "" && (slices.Contains(ids, sentID) ||
-				slices.Contains(ids, "") || ids[0] == ids[1]):
-				t.Errorf("request ids %q, want two new ones", ids)
-			}
-		})
-	}
-}
-
-func TestTimingsAreInOrder(t *testing.T) {
-	t.Parallel()
-
-	const denied = "192.0.2.50" // in SWWAF_DENY_NETS
-
-	app := startApp(t, func(w http.ResponseWriter, _ *http.Request) {
-		// The pauses set the times apart; a hold-up of the test only
-		// lengthens them.
-		time.Sleep(time.Millisecond)
-		w.WriteHeader(http.StatusOK)
-		_ = http.NewResponseController(w).Flush()
-
-		time.Sleep(time.Millisecond)
-
-		_, _ = io.WriteString(w, "done")
-	})
-	addr, out := startProxy(t, app.URL, map[string]string{
-		trustedProxies: trustLocalhost,
-		denyNets:       denied,
-	})
-
-	// Each log line is waited for before the next request, so that the
-	// lines are in order.
-	wantStatus(t, get(t, addr, "/"), http.StatusOK)
-	forwarded := out.requestLines(t, 1)[0]
-
-	req := newRequest(t, http.MethodGet, addr, "/", http.NoBody)
-	req.Header.Set(forwardedFor, denied)
-	wantStatus(t, do(t, req), http.StatusForbidden)
-	refused := out.requestLines(t, 2)[1]
-
-	wantStatus(t, get(t, addr, proxy.HealthPath), http.StatusOK)
-	health := out.requestLines(t, 3)[2]
-
-	// A request passed to the app has every timing; one refused, none of
-	// the app's; the health check, which runs no check, only the total.
-	wantTimings(t, forwarded, "duration_total", "duration_checks",
-		"duration_upstream_connect", "duration_upstream_first_byte",
-		"duration_upstream_total")
-	wantTimings(t, refused, "duration_total", "duration_checks")
-	wantTimings(t, health, "duration_total")
-
-	if t.Failed() {
-		return
-	}
-
-	// In whole microseconds, as they are logged, so that the sum below is
-	// exact.
-	total := microseconds(forwarded.DurationTotal)
-	checks := microseconds(*forwarded.DurationChecks)
-	connect := microseconds(*forwarded.DurationUpstreamConnect)
-	firstByte := microseconds(*forwarded.DurationUpstreamFirstByte)
-	upstream := microseconds(*forwarded.DurationUpstreamTotal)
-
-	// The checks end before the request is handed to the app, and the
-	// connection comes before the answer, which the app ends after a
-	// pause.
-	if checks+upstream > total || connect >= firstByte || firstByte >= upstream {
-		t.Errorf("timings in microseconds: total %d, checks %d, connect %d, "+
-			"first byte %d, upstream total %d", total, checks, connect, firstByte,
-			upstream)
-	}
-
-	if *refused.DurationChecks > refused.DurationTotal {
-		t.Errorf("refused request's checks took %v of %v milliseconds",
-			*refused.DurationChecks, refused.DurationTotal)
-	}
-}
-
-// wantTimings checks that the timings named are the only ones line has.
-func wantTimings(t *testing.T, line logLine, want ...string) {
-	t.Helper()
-
-	var got []string
-
-	for name := range line.fields {
-		if strings.HasPrefix(name, "duration_") {
-			got = append(got, name)
-		}
-	}
-
-	slices.Sort(got)
-	slices.Sort(want)
-
-	if !slices.Equal(got, want) {
-		t.Errorf("log line of %s has timings %v, want %v", line.Path, got, want)
-	}
-}
-
-// microseconds is a timing in whole microseconds.
-func microseconds(milliseconds float64) int64 {
-	return int64(math.Round(milliseconds * 1000))
-}
-
-func TestLogsAnUpgradedConnection(t *testing.T) {
-	t.Parallel()
-
-	app := startApp(t, echoAfterUpgrade)
-	addr, out := startProxy(t, app.URL, nil)
-
-	conn := dial(t, addr)
-	send(t, conn, "GET /socket HTTP/1.1\r\nHost: app\r\n"+
-		"Connection: Upgrade\r\nUpgrade: websocket\r\n\r\n")
-	wantStatus(t, readResponse(t, conn), http.StatusSwitchingProtocols)
-
-	_ = conn.Close()
-
-	line := out.requestLine(t)
-	if line.fields["websocket"] != true {
-		t.Errorf("log line has websocket %v, want true", line.fields["websocket"])
-	}
-}

internal/ratelimit/ratelimit.go

@@ -149,39 +149,26 @@ type Hit struct {
 	Requests float64
 }
 
-// Counts are a client's requests in the minute, the hour and the day that
-// end at a request, that request included.
-type Counts struct {
-	Minute float64 `json:"minute"`
-	Hour   float64 `json:"hour"`
-	Day    float64 `json:"day"`
-}
-
 // Count counts a request from client at now, in every window, whether or
-// not it is refused, and returns the client's requests in each window. It
-// reports whether the request takes the client over a limit, and the
-// window whose limit it goes over, the shortest if it is over several.
-func (l *Limiter) Count(client netip.Prefix, now time.Time) (Counts, Hit, bool) {
+// not it is refused. It reports whether the request takes the client over
+// a limit, and the window whose limit it goes over, the shortest if it is
+// over several.
+func (l *Limiter) Count(client netip.Prefix, now time.Time) (Hit, bool) {
 	l.mu.Lock()
 	defer l.mu.Unlock()
 
-	var (
-		requests [3]float64
-		hit      Hit
-	)
+	var hit Hit
 
 	for i, b := range l.get(client).buckets() {
 		w := l.windows[i]
 
-		requests[i] = b.add(now, w.length)
-		if hit.Window == "" && w.limit > 0 && requests[i] > float64(w.limit) {
-			hit = Hit{Window: w.name, Limit: w.limit, Requests: requests[i]}
+		requests := b.add(now, w.length)
+		if hit.Window == "" && w.limit > 0 && requests > float64(w.limit) {
+			hit = Hit{Window: w.name, Limit: w.limit, Requests: requests}
 		}
 	}
 
-	counts := Counts{Minute: requests[0], Hour: requests[1], Day: requests[2]}
-
-	return counts, hit, hit.Window != ""
+	return hit, hit.Window != ""
 }
 
 // Reset sets client's counts in every window back to zero. Its history

internal/ratelimit/ratelimit_test.go

@@ -62,14 +62,14 @@ func TestHitGivesTheLimitAndTheRequestsCounted(t *testing.T) {
 	start := midnight()
 
 	for range limit {
-		_, _, over := limiter.Count(client, start)
+		_, over := limiter.Count(client, start)
 		if over {
 			t.Fatal("a request within the limit is over it")
 		}
 	}
 
 	// Over both limits; the minute's is named, with the four requests.
-	_, hit, over := limiter.Count(client, start)
+	hit, over := limiter.Count(client, start)
 
 	want := ratelimit.Hit{Window: minute, Limit: limit, Requests: limit + 1}
 	if !over || hit != want {
@@ -78,29 +78,6 @@ func TestHitGivesTheLimitAndTheRequestsCounted(t *testing.T) {
 	}
 }
 
-func TestCountGivesTheRequestsInEachWindow(t *testing.T) {
-	t.Parallel()
-
-	limiter := ratelimit.New(ratelimit.Limits{})
-	client := netip.MustParsePrefix("203.0.113.9/32")
-	start := midnight()
-
-	for range 3 {
-		limiter.Count(client, start)
-	}
-
-	// A quarter into the next hour, the minute has only this request. The
-	// hour still covers three quarters of the bucket before, with its three
-	// requests, which count 2.25, and this one: 3.25. The day covers all
-	// four.
-	counts, _, _ := limiter.Count(client, start.Add(time.Hour+time.Hour/4))
-
-	want := ratelimit.Counts{Minute: 1, Hour: 3.25, Day: 4}
-	if counts != want {
-		t.Errorf("counts %+v, want %+v", counts, want)
-	}
-}
-
 func TestResetSetsTheCountsBackToZero(t *testing.T) {
 	t.Parallel()
 
@@ -261,7 +238,7 @@ func wantCount(
 ) {
 	t.Helper()
 
-	_, hit, _ := limiter.Count(client, now)
+	hit, _ := limiter.Count(client, now)
 	if hit.Window != want {
 		t.Errorf("request from %s at %s is over %q, want %q",
 			client, now.Format(time.RFC3339), hit.Window, want)

internal/requestlog/requestlog.go

@@ -9,8 +9,6 @@ import (
 	"io"
 	"log/slog"
 	"time"
-
-	"sneak.berlin/go/smallwebwaf/internal/ratelimit"
 )
 
 // The action a request line names: what smallwebwaf did with the
@@ -47,71 +45,28 @@ const OffenceLimit = "limit"
 // timeLayout is RFC 3339 with milliseconds.
 const timeLayout = "2006-01-02T15:04:05.000Z07:00"
 
-// Line is one request's line in the request log. The field names, and
-// their order, are those of the "Request log" section of SPEC.md. A field
-// that may not apply to a request is left out of its line when it does
-// not.
+// Line is one request's line in the request log. The field names are
+// those of the "Request log" section of SPEC.md.
 //
 //nolint:tagliatelle // SPEC.md's request log names its fields in snake_case
 type Line struct {
-	Type string `json:"type"`
-
-	// The standard web log fields. Scheme is how the client reached
-	// smallwebwaf, or the trusted proxy in front of it.
-	Time          string `json:"time"`
-	Instance      string `json:"instance"`
-	ClientIP      string `json:"client_ip"`
-	Method        string `json:"method"`
-	Scheme        string `json:"scheme"`
-	Host          string `json:"host"`
-	Path          string `json:"path"`
-	Query         string `json:"query"`
-	Protocol      string `json:"protocol"`
-	Status        int    `json:"status"`
-	RequestBytes  int64  `json:"request_bytes"`
-	ResponseBytes int64  `json:"response_bytes"`
-	Referer       string `json:"referer"`
-	UserAgent     string `json:"user_agent"`
-
-	// Request detail. RequestID is the X-Request-ID a trusted proxy sent,
-	// or a new one, and is sent on to the app. ForwardedFor is the
-	// X-Forwarded-For header as received. ClientGroup is the netblock the
-	// client is counted as.
-	RequestID    string `json:"request_id"`
-	PeerIP       string `json:"peer_ip"`
-	ForwardedFor string `json:"forwarded_for,omitempty"`
-	ClientGroup  string `json:"client_group"`
-	Country      string `json:"country"`
-	ContentType  string `json:"content_type,omitempty"`
-	// ContentLength is the length of its body the request announced.
-	ContentLength int64 `json:"content_length,omitempty"`
-	// RequestHeaders are the headers SWWAF_LOG_REQUEST_HEADERS names that
-	// the request carried, by name in lower case.
-	RequestHeaders   map[string]string `json:"request_headers,omitempty"`
-	HasAuthorization bool              `json:"has_authorization,omitempty"`
-	HasCookie        bool              `json:"has_cookie,omitempty"`
-	// Websocket is true when the connection was upgraded, as for a
-	// WebSocket.
-	Websocket bool `json:"websocket,omitempty"`
-
-	// Response detail, from the headers of the answer: the app's, as
-	// passed on, or those of smallwebwaf's own. Aborted is true when the
-	// client went away early.
-	ResponseContentType string `json:"response_content_type,omitempty"`
-	UpstreamStatus      int    `json:"upstream_status,omitempty"`
-	CacheControl        string `json:"cache_control,omitempty"`
-	Location            string `json:"location,omitempty"`
-	Aborted             bool   `json:"aborted,omitempty"`
-
-	// The decision.
-	Action string `json:"action"`
-	// WouldAction is, in observe mode, the action enforce mode would have
-	// taken with a request it would have refused: ActionDenied,
-	// ActionBanned, ActionCountryDenied or ActionRateLimited.
-	WouldAction string `json:"would_action,omitempty"`
-	// Counts are the client's requests as the rate limits counted them
-	// with this one, for a request they counted.
-	Counts ratelimit.Counts `json:"counts,omitzero"`
+	Type           string `json:"type"`
+	Time           string `json:"time"`
+	ClientIP       string `json:"client_ip"`
+	PeerIP         string `json:"peer_ip"`
+	Country        string `json:"country"`
+	Method         string `json:"method"`
+	Host           string `json:"host"`
+	Path           string `json:"path"`
+	Query          string `json:"query"`
+	Protocol       string `json:"protocol"`
+	Status         int    `json:"status"`
+	UpstreamStatus int    `json:"upstream_status,omitempty"`
+	RequestBytes   int64  `json:"request_bytes"`
+	ResponseBytes  int64  `json:"response_bytes"`
+	Referer        string `json:"referer"`
+	UserAgent      string `json:"user_agent"`
+	Action         string `json:"action"`
 	// LimitHit is the window whose rate limit the request went over:
 	// minute, hour or day.
 	LimitHit string `json:"limit_hit,omitempty"`
@@ -120,18 +75,11 @@ type Line struct {
 	// BanExpires is when the ban the request made, or was refused under,
 	// ends: a time, or "permanent".
 	BanExpires string `json:"ban_expires,omitempty"`
-
-	// The timings, in milliseconds. DurationChecks is the time until the
-	// checks were done. DurationUpstreamConnect, DurationUpstreamFirstByte
-	// and DurationUpstreamTotal run from when the request was handed to the
-	// app: until there was a connection to it, until the first byte of its
-	// answer arrived, and until the end. Each but DurationTotal is nil for
-	// a request that did not get that far.
-	DurationTotal             float64  `json:"duration_total"`
-	DurationChecks            *float64 `json:"duration_checks,omitempty"`
-	DurationUpstreamConnect   *float64 `json:"duration_upstream_connect,omitempty"`
-	DurationUpstreamFirstByte *float64 `json:"duration_upstream_first_byte,omitempty"`
-	DurationUpstreamTotal     *float64 `json:"duration_upstream_total,omitempty"`
+	// Aborted is true when the client went away early.
+	Aborted bool `json:"aborted,omitempty"`
+	// DurationTotal and DurationUpstreamTotal are in milliseconds.
+	DurationTotal         float64 `json:"duration_total"`
+	DurationUpstreamTotal float64 `json:"duration_upstream_total,omitempty"`
 }
 
 // Write writes line to w as one JSON line marked "type":"request".

internal/requestlog/requestlog_test.go

@@ -50,11 +50,7 @@ func TestWriteWritesOneJSONLineMarkedRequest(t *testing.T) {
 	}
 
 	unset := []string{
-		"forwarded_for", "content_type", "content_length", "request_headers",
-		"has_authorization", "has_cookie", "websocket", "response_content_type",
-		"upstream_status", "cache_control", "location", "aborted", "counts",
-		"limit_hit", "offence", "ban_expires", "duration_checks",
-		"duration_upstream_connect", "duration_upstream_first_byte",
+		"upstream_status", "limit_hit", "offence", "ban_expires", "aborted",
 		"duration_upstream_total",
 	}
 	for _, name := range unset {

internal/smallwebwaf/smallwebwaf_test.go

@@ -418,7 +418,6 @@ func wantStartingLine(t *testing.T, line map[string]any, appURL, dir string) {
 		listenAddr:                              localhost + ":0",
 		upstreamURL:                             appURL,
 		stateDir:                                dir,
-		"SWWAF_MODE":                            "enforce",
 		stateWriteDelay:                         "10s",
 		stateCounterInterval:                    "15m",
 		trustedProxies:                          "10.0.0.0/8,172.16.0.0/12,192.168.0.0/16",
@@ -436,7 +435,6 @@ func wantStartingLine(t *testing.T, line map[string]any, appURL, dir string) {
 		"SWWAF_RATE_LIMIT_PER_MINUTE":           "1000",
 		"SWWAF_RATE_LIMIT_PER_HOUR":             "10000",
 		rateLimitPerDay:                         "50000",
-		"SWWAF_RATE_LIMIT_EXEMPT_PATHS":         "",
 		"SWWAF_DENIED_COUNTRIES":                "",
 		"SWWAF_EXCLUSIVELY_ALLOWED_COUNTRIES":   "",
 		"SWWAF_BAN_RESPONSE":                    "403",

internal/state/state.go

@@ -222,7 +222,7 @@ func (f *Files) Watch(ctx context.Context) {
 		case event := <-watcher.Events:
 			switch name := filepath.Base(event.Name); name {
 			case bansJSON, clientsJSON, lookupsJSON:
-				f.fileChanged(name)
+				f.takeInEdit(name)
 			}
 		case err = <-watcher.Errors:
 			f.params.ProcessLog.Warn("watching the state files failed",
@@ -239,11 +239,10 @@ func (f *Files) logFailure(err error) {
 	}
 }
 
-// fileChanged takes in what the state file name holds, as Watch sees it
-// change, if that is an edit made since smallwebwaf last read or wrote
-// the file. A file that cannot be read or does not parse is left for its
-// next write.
-func (f *Files) fileChanged(name string) {
+// takeInEdit takes in an edit of the state file name and logs it, if the
+// file has changed since smallwebwaf last read or wrote it and parses. A
+// file that cannot be read or does not parse is left for its next write.
+func (f *Files) takeInEdit(name string) {
 	f.mu.Lock()
 	defer f.mu.Unlock()
 
@@ -252,17 +251,9 @@ func (f *Files) fileChanged(name string) {
 		return
 	}
 
-	_ = f.takeInEdit(name, data)
-}
-
-// takeInEdit takes in data, an edit of the state file name, as takeIn
-// does, and counts and logs it. Every edit taken in while smallwebwaf
-// runs, by Watch or by a write, is taken in here. An edit that does not
-// parse is neither counted nor logged, and takeIn's error returned.
-func (f *Files) takeInEdit(name string, data []byte) error {
-	_, err := f.takeIn(name, data)
+	_, err = f.takeIn(name, data)
 	if err != nil {
-		return err
+		return
 	}
 
 	// Counted before it is logged, so that the count is there once the
@@ -270,8 +261,6 @@ func (f *Files) takeInEdit(name string, data []byte) error {
 	f.params.Metrics.StateFileEditTakenIn(name)
 	f.params.ProcessLog.Info("took in an edit of a state file",
 		"file", filepath.Join(f.params.Dir, name))
-
-	return nil
 }
 
 // read takes in the state file name at start, and returns how many
@@ -368,8 +357,10 @@ func (f *Files) writeFile(name string) error {
 
 	data, changed, err := f.readChanged(name)
 	if err == nil && changed {
-		err = f.takeInEdit(name, data)
-		if err != nil {
+		_, err = f.takeIn(name, data)
+		if err == nil {
+			f.params.Metrics.StateFileEditTakenIn(name)
+		} else {
 			err = f.setAside(name, err)
 		}
 	}

internal/state/state_test.go

@@ -5,7 +5,6 @@ import (
 	"encoding/json"
 	"io/fs"
 	"log/slog"
-	"maps"
 	"net"
 	"net/http"
 	"net/http/httptest"
@@ -775,53 +774,6 @@ func TestEditsTakenInAreCountedInTheMetrics(t *testing.T) {
 		`smallwebwaf_state_file_edits_taken_in_total{file="bans.json"}`, 2)
 }
 
-func TestEditTakenInByAWriteIsLoggedAsWatchLogsIt(t *testing.T) {
-	t.Parallel()
-
-	dir := t.TempDir()
-	params := newParams(dir)
-	lines := logInto(&params)
-	files := load(t, params)
-
-	// An edit taken in by Watch, which is then stopped.
-	ctx, stop := context.WithCancel(t.Context())
-	stopped := make(chan struct{})
-
-	go func() {
-		files.Watch(ctx)
-		close(stopped)
-	}()
-
-	lines.waitFor(t, watching)
-	edit(t, dir, bansJSON, `{"version": 1, "bans": []}`)
-	byWatch := lines.waitFor(t, tookIn)
-
-	stop()
-	<-stopped
-
-	// An edit taken in by the write of its file. Nothing logs after the
-	// write, so the log is closed, and a write that does not log the edit
-	// fails the test at once instead of waiting for the line.
-	edit(t, dir, bansJSON, permanentBansJSON)
-
-	err := files.WriteAll()
-	if err != nil {
-		t.Fatalf("write: %v", err)
-	}
-
-	close(lines)
-
-	byWrite := lines.waitFor(t, tookIn)
-
-	// The two lines differ only in their time.
-	delete(byWatch, "time")
-	delete(byWrite, "time")
-
-	if !maps.Equal(byWrite, byWatch) {
-		t.Errorf("the write logged %v, where Watch logged %v", byWrite, byWatch)
-	}
-}
-
 func TestEditsSetAsideAreCountedInTheMetrics(t *testing.T) {
 	t.Parallel()
 
@@ -1005,9 +957,8 @@ func (l processLog) Write(line []byte) (int, error) {
 }
 
 // waitFor returns the next line of the process log whose message is msg,
-// passing over the lines before it, or nil if the log is closed first. It
-// waits as long as that takes, so that a slow test process cannot fail
-// the test.
+// passing over the lines before it. It waits as long as that takes, so
+// that a slow test process cannot fail the test.
 func (l processLog) waitFor(t *testing.T, msg string) map[string]any {
 	t.Helper()
 
@@ -1024,7 +975,7 @@ func (l processLog) waitFor(t *testing.T, msg string) map[string]any {
 		}
 	}
 
-	return nil
+	return nil // never reached: nothing closes the log
 }
 
 // wantTakenIn waits for the next edit taken in, and checks that it is of