Leave SWWAF_RATE_LIMIT_EXEMPT_PATHS out of the request rate limits (closes #77)

A request is neither counted nor refused by the request rate limits
when the path the app will act on starts with one of the
comma-separated prefixes in SWWAF_RATE_LIMIT_EXEMPT_PATHS. That path is
the request's path percent-decoded, with its . and .. segments and
repeated slashes resolved, so /assets/..%2Flogin is /login and is
counted. The static lists, bans and the country lists still apply. The
setting is empty by default, and a prefix that does not start with /
stops the start. README.md documents it.

Model: opus-5-5

README.md

@@ -13,21 +13,23 @@ JSON log line for every request.
 
 Status: the first two milestones are built
 (https://git.eeqj.de/sneak/smallwebwaf/issues/13 and
-https://git.eeqj.de/sneak/smallwebwaf/issues/14), and so are six parts of
+https://git.eeqj.de/sneak/smallwebwaf/issues/14), and so are seven parts of
 milestone 3: the static lists, the bans that broken rate limits lead to, the
 JSON state files and the paths the rate limits do not count, which come next in
-the build order, and the metrics endpoint and the header size and the idle time
+the build order, `observe` mode, which comes a little later, and the metrics
-as settings, which come last in it. `smallwebwaf` passes each request to the app
+endpoint and the header size and the idle time as settings, which come last in
-and the app's answer back, unchanged, within its timeouts and size limits, works
+it. `smallwebwaf` passes each request to the app and the app's answer back,
-out each client's address, bans a client that sends too many requests, not
+unchanged, within its timeouts and size limits, works out each client's address,
-counting those for the paths you choose, refuses a client that comes from a
+bans a client that sends too many requests, not counting those for the paths you
-country you refuse or from a network you refuse, lets the networks you choose
+choose, refuses a client that comes from a country you refuse or from a network
-through, keeps its bans, each client's counters and history, and GeoJS's answers
+you refuse, lets the networks you choose through, keeps its bans, each client's
-in JSON files across restarts, writes a JSON log line for every request, and
+counters and history, and GeoJS's answers in JSON files across restarts, writes
-serves Prometheus metrics to a scraper that holds the metrics token. It comes as
+a JSON log line for every request, serves Prometheus metrics to a scraper that
-the image the app's own image is built on. The rest of the design comes after
+holds the metrics token, and in `observe` mode passes on the requests it would
-that, in the order of the build order in [`SPEC.md`](SPEC.md). The survey of
+refuse, logging what it would have done with them. It comes as the image the
-existing tools that led to the design is in [`EVALUATION.md`](EVALUATION.md).
+app's own image is built on. The rest of the design comes after that, in the
+order of the build order in [`SPEC.md`](SPEC.md). The survey of existing tools
+that led to the design is in [`EVALUATION.md`](EVALUATION.md).
 
 ## Getting started
 
@@ -121,6 +123,18 @@ in `bin/state` unless `SWWAF_STATE_DIR` is set.
   `SWWAF_ALLOW_NETS` too is let through. A client in
   `SWWAF_RATE_LIMIT_EXEMPT_NETS` is neither counted nor refused by the rate
   limits; the country lists and bans still apply to it.
+- In `observe` mode, with `SWWAF_MODE=observe`, refuses none of the requests
+  that `SWWAF_DENY_NETS`, a ban, the country lists or a rate limit would refuse:
+  it passes them to the app, and their log lines name what `enforce` mode would
+  have done (see `would_action` in "Request log" below). The checks run, and
+  requests are counted, as in `enforce` mode, but a broken rate limit makes no
+  ban and does not set the client's counters back to zero, so each request over
+  the limit is logged as one that would be refused. The bans in `bans.json` are
+  kept, and refuse requests again when `smallwebwaf` next runs in `enforce`
+  mode, as long as they last. The timeouts and size limits still apply, since
+  they protect `smallwebwaf` and the app themselves, and a request for the
+  metrics without the token is still answered `401`. It is for trying a
+  configuration before enforcing it.
 - Answers `GET /_smallwebwaf/healthz` itself with `200` and `ok`, before any
   check and without asking the app, for the image's health check.
 - Answers `GET /_smallwebwaf/metrics` with its metrics (see "Metrics" below) for
@@ -142,6 +156,9 @@ it, and the effective settings are logged at start.
 - `SWWAF_LISTEN_ADDR` (default `:8080`): where `smallwebwaf` listens.
 - `SWWAF_UPSTREAM_URL` (default `http://127.0.0.1:8081`): the app, as `http` or
   `https`, a host and an optional port, and nothing more.
+- `SWWAF_MODE` (default `enforce`): `enforce`, or `observe` to pass on the
+  requests `smallwebwaf` would refuse and log what it would have done (see "What
+  it does so far" above).
 - `SWWAF_TRUSTED_PROXIES` (default `10.0.0.0/8,172.16.0.0/12,192.168.0.0/16`,
   the private address ranges): the netblocks whose `X-Forwarded-For` is
   believed. A list given replaces the default; set but empty, it trusts nothing.
@@ -179,11 +196,15 @@ it, and the effective settings are logged at start.
   page makes a few hundred requests and several people often share one address.
 - `SWWAF_RATE_LIMIT_EXEMPT_PATHS` (default empty): path prefixes whose requests
   the rate limits neither count nor refuse, such as `/assets/` for static
-  assets; each starts with `/`. A prefix is compared with the start of the path
+  assets; each starts with `/`. A prefix is compared, character for character,
-  as the client sent it, before any query string, as the request log's `path`
+  with the start of the path the app will act on: the request's path, before any
-  shows it, character for character: `/assets/` matches `/assets/app.js` and
+  query string, percent-decoded, with its `.` and `..` segments and repeated
-  `/assets/img/logo.png`, but not `/assets`, `/Assets/app.js` or
+  slashes resolved and without a trailing slash, which is not always what the
-  `/static/assets/app.js`. There are no wildcards: `*` is a character like any
+  request log's `path` shows. `/assets/` matches `/assets/app.js`,
+  `/assets//img/logo.png` and `/static/../assets/app.js`, but not `/assets/`
+  itself, `/assets`, `/Assets/app.js`, `/static/assets/app.js` or
+  `/assets/..%2Flogin`, which is `/login`. A prefix is written without
+  percent-encoding, and there are no wildcards: `*` is a character like any
   other.
 - `SWWAF_DENIED_COUNTRIES` (default empty): countries whose clients are refused,
   for example `cn,ru,kp`.
@@ -251,8 +272,8 @@ refused ones included:
 - `country` is the client's country as GeoJS places it. It is empty with neither
   country list set, for a client in `SWWAF_ALLOW_NETS` or `SWWAF_DENY_NETS`, for
   a client on a private, loopback or link-local address, when GeoJS cannot place
-  the client or has not answered in time, and for a request refused because a
+  the client or has not answered in time, and for a request whose client a ban
-  ban covers its client, even when the client's country is known.
+  covers, even when the client's country is known.
 - `status` is what the client was sent, `0` if nothing was; `upstream_status` is
   what the app answered, and is left out when the app did not answer.
 - `request_bytes` and `response_bytes` count body bytes.
@@ -264,11 +285,18 @@ refused ones included:
   `timed_out` for one that ran out of time, `upstream_error` when the app could
   not be reached or its answer broke off, and `admin` for one `smallwebwaf`
   answered at its own endpoint.
+- `would_action` is there in `observe` mode for a request that
+  `SWWAF_DENY_NETS`, a ban, the country lists or a rate limit would have refused
+  in `enforce` mode, and names the action that refusal would have had: `denied`,
+  `banned`, `country_denied` or `rate_limited`. `action` then names what was
+  done: `forward` for a request passed to the app, and another action, such as
+  `too_large`, for one a size or time limit refused.
 - `limit_hit` is there for a request that broke a rate limit, and names the
   window whose limit it went over: `minute`, `hour` or `day`, the shortest if it
   went over several. `offence` is then `limit`.
 - `ban_expires` is there for a request that made a ban or was refused under one,
-  and gives when the ban ends, in the same form as `time`, or `permanent`.
+  or in `observe` mode would have been refused under one, and gives when the ban
+  ends, in the same form as `time`, or `permanent`.
 - `aborted` is there, and true, when the client went away early.
 - `duration_total` and `duration_upstream_total` are in milliseconds.
 
@@ -592,17 +620,16 @@ the metrics, failure behaviour and the build order.
 So far `smallwebwaf` looks up only the country, only through GeoJS, and only
 while `SWWAF_DENIED_COUNTRIES` or `SWWAF_EXCLUSIVELY_ALLOWED_COUNTRIES` is set:
 then the address of every new visitor is sent to GeoJS, except a visitor in
-`SWWAF_ALLOW_NETS` or `SWWAF_DENY_NETS` and one refused because a ban covers its
+`SWWAF_ALLOW_NETS` or `SWWAF_DENY_NETS` and one whose netblock a ban covers, and
-netblock, and with neither set, none is. An IPv6 visitor is asked about by the
+with neither set, none is. An IPv6 visitor is asked about by the first address
-first address of its /64. A new visitor waits at most a second for its answer,
+of its /64. A new visitor waits at most a second for its answer, and without one
-and without one counts as coming from an unknown country until the answer
+counts as coming from an unknown country until the answer arrives. The addresses
-arrives. The addresses waiting are asked about together, up to 200 in one
+waiting are asked about together, up to 200 in one request, one request at a
-request, one request at a time; at most 10,000 visitors wait, and one more
+time; at most 10,000 visitors wait, and one more counts as coming from an
-counts as coming from an unknown country until there is room. While GeoJS fails,
+unknown country until there is room. While GeoJS fails, visitors with a kept
-visitors with a kept answer are unaffected and new ones count as coming from an
+answer are unaffected and new ones count as coming from an unknown country.
-unknown country. GeoJS is then left alone for a second, twice as long after each
+GeoJS is then left alone for a second, twice as long after each further failure
-further failure up to five minutes, and asked again by the next request that
+up to five minutes, and asked again by the next request that needs it.
-needs it.
 
 In the full design, `smallwebwaf` looks up the AS number and country of every
 client, for the request log, the metrics and the ban notes, and for the country
@@ -654,8 +681,10 @@ addresses are never sent to GeoJS.
   limits, and writes the request's log line. Its `check` method is where a
   request is refused before anything reaches the app: for `SWWAF_DENY_NETS`, for
   a ban, for the country lists, for a rate limit, which bans the client, and for
-  an announced body over the size limit. A request under `/_smallwebwaf/` that
+  an announced body over the size limit; in `observe` mode, only for the size
-  `check` lets through is answered by `answerAdmin` instead of reaching the app.
+  limit, with what it would have refused for noted in the log line. A request
+  under `/_smallwebwaf/` that `check` lets through is answered by `answerAdmin`
+  instead of reaching the app.
 - `internal/metrics`: the metrics, counted as the other parts tell it what
   happened, and served in the Prometheus text format.
 - `internal/bans`: the ban ledger: each netblock's bans with their notes, how
@@ -721,10 +750,10 @@ so that they run in minimal containers.
 
 ## TODO
 
-- The rest of milestone 3, from taking in an admin's edits to the state files
+- The rest of milestone 3: taking in an admin's edits to the state files
-  (https://git.eeqj.de/sneak/smallwebwaf/issues/68) up to the rest of the
+  (https://git.eeqj.de/sneak/smallwebwaf/issues/68), exemptions and the rest of
-  request log's fields, and the rest of the design, in the order of the build
+  the request log's fields; then the rest of the design, in the order of the
-  order in [`SPEC.md`](SPEC.md).
+  build order in [`SPEC.md`](SPEC.md).
 
 ## Documents
 

internal/bans/bans.go

@@ -107,8 +107,8 @@ type Ledger struct {
 	changed chan struct{}
 
 	mu sync.Mutex
-	// netblocks holds each banned netblock's bans, oldest first. Check
+	// netblocks holds each banned netblock's bans, oldest first. Check and
-	// makes each netblock it finds the most recently seen.
+	// Find make each netblock they find the most recently seen.
 	netblocks *simplelru.LRU[netip.Prefix, *[]Ban]
 	// held is how many bans netblocks holds, at most rules.MaxBans.
 	held int
@@ -144,38 +144,38 @@ func (l *Ledger) Changed() <-chan struct{} {
 	return l.changed
 }
 
-// Check is called for each request from client, at now. It reports
+// Check is called for a request from client, at now. It reports whether
-// whether a ban on a netblock client is in is active, and returns that
+// a ban on a netblock client is in is active, and returns that ban, with
-// ban, with the request counted among those it refused.
+// the request counted among those it refused.
 func (l *Ledger) Check(client netip.Addr, now time.Time) (Ban, bool) {
 	l.mu.Lock()
 	defer l.mu.Unlock()
 
-	lengths := l.v6Lengths
+	ban := l.active(client, now)
-	if client.Is4() {
+	if ban == nil {
-		lengths = l.v4Lengths
-	}
-
-	for _, length := range lengths {
-		bans, found := l.netblocks.Get(netip.PrefixFrom(client, length).Masked())
-		if !found {
-			continue
-		}
-
-		// A ban is made only once the one before has ended, so only the
-		// last can be active.
-		last := &(*bans)[len(*bans)-1]
-		if last.ActiveAt(now) {
-			last.Notes.Requests++
-			last.Notes.Refused++
-
-			return *last, true
-		}
-	}
-
 		return Ban{}, false
 	}
 
+	ban.Notes.Requests++
+	ban.Notes.Refused++
+
+	return *ban, true
+}
+
+// Find is Check without counting the request among those the ban
+// refused: in observe mode a ban refuses nothing.
+func (l *Ledger) Find(client netip.Addr, now time.Time) (Ban, bool) {
+	l.mu.Lock()
+	defer l.mu.Unlock()
+
+	ban := l.active(client, now)
+	if ban == nil {
+		return Ban{}, false
+	}
+
+	return *ban, true
+}
+
 // BanForLimit bans netblock at now for a broken limit, with notes, and
 // returns the ban. A first ban lasts LimitBanDuration. A ban made within
 // LimitBanRepeatWindow after the netblock's last ban ended lasts
@@ -304,6 +304,31 @@ func (l *Ledger) Load(bans []Ban) {
 	}
 }
 
+// active returns the ban active at now on a netblock client is in, or
+// nil.
+func (l *Ledger) active(client netip.Addr, now time.Time) *Ban {
+	lengths := l.v6Lengths
+	if client.Is4() {
+		lengths = l.v4Lengths
+	}
+
+	for _, length := range lengths {
+		bans, found := l.netblocks.Get(netip.PrefixFrom(client, length).Masked())
+		if !found {
+			continue
+		}
+
+		// A ban is made only once the one before has ended, so only the
+		// last can be active.
+		last := &(*bans)[len(*bans)-1]
+		if last.ActiveAt(now) {
+			return last
+		}
+	}
+
+	return nil
+}
+
 // add adds ban to its netblock's bans, after the last, and makes its
 // netblock the most recently seen. With MaxBans held, it drops one first.
 func (l *Ledger) add(ban Ban) {

internal/bans/bans_test.go

@@ -162,6 +162,28 @@ func TestCheckRefusesWhileTheBanLastsAndCountsTheRefusals(t *testing.T) {
 	}
 }
 
+func TestFindCountsNothing(t *testing.T) {
+	t.Parallel()
+
+	ledger := bans.New(defaultRules())
+	netblock := netip.MustParsePrefix("203.0.113.9/32")
+	ban := ledger.BanForLimit(netblock, midnight(), bans.Notes{Requests: 5})
+
+	got, banned := ledger.Find(netblock.Addr(), ban.Expires.Add(-time.Nanosecond))
+	if !banned || got != ban {
+		t.Errorf("find during the ban gives %+v and %t, want %+v", got, banned, ban)
+	}
+
+	_, banned = ledger.Find(netblock.Addr(), ban.Expires)
+	if banned {
+		t.Error("the ban did not end")
+	}
+
+	if notes := ledger.Bans(netblock)[0].Notes; notes != ban.Notes {
+		t.Errorf("the notes are %+v, want them unchanged, %+v", notes, ban.Notes)
+	}
+}
+
 func TestMaxBansDropsTheEarliestBanOfTheNetblockSeenLongestAgo(t *testing.T) {
 	t.Parallel()
 

internal/config/config.go

@@ -27,6 +27,11 @@ type Config struct {
 	ListenAddr string
 	// UpstreamURL is the app (SWWAF_UPSTREAM_URL).
 	UpstreamURL *url.URL
+	// Observe is true in observe mode, when SWWAF_MODE is observe rather
+	// than enforce: a request that SWWAF_DENY_NETS, a ban, the country
+	// lists or a rate limit would refuse is passed to the app instead, and
+	// no ban is made.
+	Observe bool
 	// TrustedProxies are the netblocks whose X-Forwarded-For is
 	// believed (SWWAF_TRUSTED_PROXIES).
 	TrustedProxies []netip.Prefix
@@ -166,6 +171,7 @@ var (
 	errNotAbsolutePath = errors.New(
 		"is not an absolute path, such as /var/lib/smallwebwaf")
 	errShortToken    = errors.New("is shorter than 32 characters")
+	errNotMode       = errors.New("is not enforce or observe")
 	errNotPathPrefix = errors.New(
 		"is not a path prefix starting with /, such as /assets/")
 )
@@ -178,6 +184,7 @@ func FromEnvironment(lookupEnv func(string) (string, bool)) (*Config, error) {
 	cfg := &Config{
 		ListenAddr:           env.address("SWWAF_LISTEN_ADDR", ":8080"),
 		UpstreamURL:          env.appURL("SWWAF_UPSTREAM_URL", "http://127.0.0.1:8081"),
+		Observe:              env.observe("SWWAF_MODE", "enforce"),
 		TrustedProxies:       env.netblocks("SWWAF_TRUSTED_PROXIES", privateRanges),
 		ClientRequestTimeout: env.duration("SWWAF_CLIENT_REQUEST_TIMEOUT", "60s"),
 		ClientRequestHeaderMaxBytes: env.headerSize(
@@ -281,6 +288,17 @@ func (e *environment) appURL(name, defaultValue string) *url.URL {
 	return upstream
 }
 
+// observe reads the setting that is the mode, enforce or observe, and
+// reports whether it is observe.
+func (e *environment) observe(name, defaultValue string) bool {
+	mode := e.value(name, defaultValue)
+	if mode != "enforce" && mode != "observe" {
+		e.check(name, fmt.Errorf("%q %w", mode, errNotMode))
+	}
+
+	return mode == "observe"
+}
+
 // netblocks reads a setting that is a list of netblocks.
 func (e *environment) netblocks(name, defaultValue string) []netip.Prefix {
 	netblocks, err := parseNetblocks(e.value(name, defaultValue))

internal/config/config_test.go

@@ -18,6 +18,7 @@ import (
 const (
 	listenAddr              = "SWWAF_LISTEN_ADDR"
 	upstreamURL             = "SWWAF_UPSTREAM_URL"
+	mode                    = "SWWAF_MODE"
 	trustedProxies          = "SWWAF_TRUSTED_PROXIES"
 	clientRequestTimeout    = "SWWAF_CLIENT_REQUEST_TIMEOUT"
 	clientHeaderMaxBytes    = "SWWAF_CLIENT_REQUEST_HEADER_MAX_BYTES"
@@ -84,6 +85,7 @@ func TestDefaults(t *testing.T) {
 
 	wantSettings(t, cfg, config.Config{
 		ListenAddr:                  ":8080",
+		Observe:                     false,
 		ClientRequestTimeout:        time.Minute,
 		ClientRequestHeaderMaxBytes: 32 << 10,
 		ClientIdleTimeout:           2 * time.Minute,
@@ -131,6 +133,7 @@ func TestValuesAsSet(t *testing.T) {
 	cfg := fromEnvironment(t, environment{
 		listenAddr:              "127.0.0.1:9000",
 		upstreamURL:             "https://app.internal:8443/",
+		mode:                    "observe",
 		trustedProxies:          " 192.0.2.1, 10.1.2.3/8 ,2001:db8::/32",
 		clientRequestTimeout:    "90s",
 		clientHeaderMaxBytes:    "8K",
@@ -164,6 +167,7 @@ func TestValuesAsSet(t *testing.T) {
 
 	wantSettings(t, cfg, config.Config{
 		ListenAddr:                  "127.0.0.1:9000",
+		Observe:                     true,
 		ClientRequestTimeout:        90 * time.Second,
 		ClientRequestHeaderMaxBytes: 8 << 10,
 		ClientIdleTimeout:           5 * time.Minute,
@@ -331,6 +335,7 @@ func TestInvalidValueStopsTheStart(t *testing.T) {
 		{upstreamURL, "http://127.0.0.1:8081/app"},
 		{upstreamURL, "http://127.0.0.1:8081/?a=1"},
 		{upstreamURL, "http://user:secret@127.0.0.1:8081"},
+		{mode, "Observe"}, {mode, "block"}, {mode, ""},
 		{trustedProxies, "10.0.0.0/33"},
 		{trustedProxies, "traefik"},
 		{trustedProxies, "10.0.0.0/8,,192.168.0.0/16"},
@@ -451,6 +456,7 @@ func TestLogsEachSettingWithItsValue(t *testing.T) {
 	want := map[string]string{
 		listenAddr:              ":8080",
 		upstreamURL:             "http://127.0.0.1:8081",
+		mode:                    "enforce",
 		trustedProxies:          "10.0.0.0/8,172.16.0.0/12,192.168.0.0/16",
 		clientRequestTimeout:    "45s",
 		clientHeaderMaxBytes:    "32K",
@@ -491,6 +497,7 @@ func wantSettings(t *testing.T, got *config.Config, want config.Config) {
 	t.Helper()
 
 	if got.ListenAddr != want.ListenAddr ||
+		got.Observe != want.Observe ||
 		got.ClientRequestTimeout != want.ClientRequestTimeout ||
 		got.ClientRequestHeaderMaxBytes != want.ClientRequestHeaderMaxBytes ||
 		got.ClientIdleTimeout != want.ClientIdleTimeout ||

internal/proxy/bans.go

@@ -14,10 +14,15 @@ func (rq *request) banResponse(action string) *refusal {
 	return &refusal{status: rq.h.config.BanResponse, action: action}
 }
 
-// banned reports whether a ban on a netblock the client is in refuses
+// banned reports whether a ban on a netblock the client is in covers the
-// the request at now, and notes for the log line when that ban ends.
+// request at now, and notes for the log line when that ban ends.
 func (rq *request) banned(now time.Time) bool {
-	ban, banned := rq.h.ledger.Check(rq.client, now)
+	check := rq.h.ledger.Check
+	if rq.h.config.Observe {
+		check = rq.h.ledger.Find // in observe mode the ban refuses nothing
+	}
+
+	ban, banned := check(rq.client, now)
 	if banned {
 		rq.line.BanExpires = banExpires(ban)
 	}
@@ -26,8 +31,9 @@ func (rq *request) banned(now time.Time) bool {
 }
 
 // limitBroken counts the request for the rate limits at now, and reports
-// whether it takes the client over one. Such a request bans the client's
+// whether it takes the client over one. In enforce mode such a request
-// netblock, and sets the client's counters back to zero.
+// bans the client's netblock, and sets the client's counters back to
+// zero; in observe mode it does neither.
 func (rq *request) limitBroken(now time.Time) bool {
 	group := clientGroup(rq.client)
 
@@ -36,6 +42,13 @@ func (rq *request) limitBroken(now time.Time) bool {
 		return false
 	}
 
+	rq.line.LimitHit = hit.Window
+	rq.line.Offence = requestlog.OffenceLimit
+
+	if rq.h.config.Observe {
+		return true
+	}
+
 	netblock := rq.netblock()
 	ban := rq.h.ledger.BanForLimit(netblock, now, bans.Notes{
 		Country: rq.line.Country,
@@ -54,9 +67,6 @@ func (rq *request) limitBroken(now time.Time) bool {
 		Requests: rq.h.limiter.Requests(netblock) + 1,
 	})
 	rq.h.limiter.Reset(group)
-
-	rq.line.LimitHit = hit.Window
-	rq.line.Offence = requestlog.OffenceLimit
 	rq.line.BanExpires = banExpires(ban)
 
 	return true

internal/proxy/observe_test.go

@@ -0,0 +1,202 @@
+package proxy_test
+
+import (
+	"bytes"
+	"io"
+	"net/http"
+	"net/netip"
+	"sync/atomic"
+	"testing"
+	"time"
+
+	"sneak.berlin/go/smallwebwaf/internal/bans"
+	"sneak.berlin/go/smallwebwaf/internal/proxy"
+	"sneak.berlin/go/smallwebwaf/internal/requestlog"
+)
+
+// observe is the value of SWWAF_MODE for observe mode.
+const observe = "observe"
+
+func TestObserveModeForwardsWhatEnforceModeRefuses(t *testing.T) {
+	t.Parallel()
+
+	const (
+		denied = "192.0.2.50" // in SWWAF_DENY_NETS
+		banned = otherClient  // under a ban read from bans.json
+	)
+
+	for _, tc := range []struct {
+		setting string // "" leaves SWWAF_MODE at its default
+		observe bool
+	}{
+		{"", false},
+		{"enforce", false},
+		{observe, true},
+	} {
+		t.Run(mode+"="+tc.setting, func(t *testing.T) {
+			t.Parallel()
+
+			geojsURL, _ := startGeoJS(t)
+			env := map[string]string{
+				rateLimitPerMinute: "1",
+				denyNets:           denied,
+				deniedCountries:    "kp",
+			}
+
+			if tc.setting != "" {
+				env[mode] = tc.setting
+			}
+
+			s, clk, server := startWithClock(t, geojsURL, env)
+			server.Ledger.Load([]bans.Ban{{
+				Netblock: netip.MustParsePrefix(banned + "/32"),
+				Start:    clk.Now(),
+				Expires:  clk.Now().Add(time.Hour),
+			}})
+
+			// fromDE's first request is within the limit of one a minute,
+			// and its second breaks it.
+			s.get(fromDE, http.StatusOK, requestlog.ActionForward)
+
+			for _, sent := range []struct{ from, refusal string }{
+				{denied, requestlog.ActionDenied},
+				{banned, requestlog.ActionBanned},
+				{fromKP, requestlog.ActionCountryDenied},
+				{fromDE, requestlog.ActionRateLimited},
+			} {
+				if !tc.observe {
+					line := s.get(sent.from, http.StatusForbidden, sent.refusal)
+					wantWouldAction(t, line, "")
+
+					continue
+				}
+
+				// Passed to the app, which answered it.
+				line := s.get(sent.from, http.StatusOK, requestlog.ActionForward)
+				wantWouldAction(t, line, sent.refusal)
+
+				if line.UpstreamStatus != http.StatusOK {
+					t.Errorf("log line has upstream_status %d, want 200",
+						line.UpstreamStatus)
+				}
+			}
+		})
+	}
+}
+
+func TestObserveModeMakesNoBanAndKeepsTheBansItHas(t *testing.T) {
+	t.Parallel()
+
+	s, clk, server := startWithClock(t, "", map[string]string{
+		mode:               observe,
+		rateLimitPerMinute: "1",
+	})
+	kept := bans.Ban{
+		Netblock: netip.MustParsePrefix(otherClient + "/32"),
+		Start:    clk.Now(),
+		Expires:  clk.Now().Add(time.Hour),
+	}
+	server.Ledger.Load([]bans.Ban{kept})
+
+	// No ban sets client's counters back to zero, so each request after
+	// the first breaks the limit of one a minute.
+	s.get(client, http.StatusOK, requestlog.ActionForward)
+
+	for range 2 {
+		line := s.get(client, http.StatusOK, requestlog.ActionForward)
+		wantWouldAction(t, line, requestlog.ActionRateLimited)
+
+		if line.LimitHit != minute || line.Offence != requestlog.OffenceLimit ||
+			line.BanExpires != "" {
+			t.Errorf("log line has limit_hit %q, offence %q and ban_expires %q, "+
+				"want minute, limit and none", line.LimitHit, line.Offence,
+				line.BanExpires)
+		}
+	}
+
+	// The ban read from bans.json refuses nothing, and so counts no
+	// refusal in its notes, but is kept.
+	line := s.get(otherClient, http.StatusOK, requestlog.ActionForward)
+	wantWouldAction(t, line, requestlog.ActionBanned)
+
+	if line.BanExpires != requestlog.FormatTime(kept.Expires) {
+		t.Errorf("log line has ban_expires %q, want %s", line.BanExpires,
+			requestlog.FormatTime(kept.Expires))
+	}
+
+	got := server.Ledger.Snapshot()
+	if len(got) != 1 || got[0] != kept {
+		t.Errorf("bans\n%+v\nwant only\n%+v", got, kept)
+	}
+}
+
+func TestObserveModeKeepsTheSizeLimitsAndTheToken(t *testing.T) {
+	t.Parallel()
+
+	const denied = "192.0.2.50" // in SWWAF_DENY_NETS
+
+	var calls atomic.Int32
+
+	app := startApp(t, func(w http.ResponseWriter, _ *http.Request) {
+		calls.Add(1)
+		answerWithSize(w, 2*sizeLimit, true)
+	})
+	addr, out := startProxy(t, app.URL, map[string]string{
+		mode:             observe,
+		trustedProxies:   trustLocalhost,
+		denyNets:         denied,
+		requestMaxBytes:  sizeLimitSetting,
+		responseMaxBytes: sizeLimitSetting,
+		metricsToken:     token,
+	})
+
+	// SWWAF_DENY_NETS would refuse each request; instead a size limit or
+	// the missing token does.
+	for i, tc := range []struct {
+		method, path string
+		body         io.Reader
+		status       int
+		action       string
+	}{
+		{
+			http.MethodPost, "/upload", bytes.NewReader(make([]byte, 2*sizeLimit)),
+			http.StatusRequestEntityTooLarge, requestlog.ActionTooLarge,
+		},
+		{
+			http.MethodGet, "/download", http.NoBody,
+			http.StatusBadGateway, requestlog.ActionTooLarge,
+		},
+		{
+			http.MethodGet, proxy.MetricsPath, http.NoBody,
+			http.StatusUnauthorized, requestlog.ActionAdmin,
+		},
+	} {
+		req := newRequest(t, tc.method, addr, tc.path, tc.body)
+		req.Header.Set(forwardedFor, denied)
+		wantStatus(t, do(t, req), tc.status)
+
+		line := out.requestLines(t, i+1)[i]
+		wantLine(t, line, tc.status, tc.action)
+		wantWouldAction(t, line, requestlog.ActionDenied)
+	}
+
+	// The upload was refused before it reached the app.
+	if calls.Load() != 1 {
+		t.Errorf("the app was called %d times, want once", calls.Load())
+	}
+}
+
+// wantWouldAction checks the request log line's would_action, and that a
+// line that should have none has no such field.
+func wantWouldAction(t *testing.T, line logLine, want string) {
+	t.Helper()
+
+	got, present := line.fields["would_action"]
+
+	switch {
+	case want == "" && present:
+		t.Errorf("log line has would_action %v, want none", got)
+	case want != "" && got != want:
+		t.Errorf("log line has would_action %v, want %s", got, want)
+	}
+}

internal/proxy/proxy_test.go

@@ -50,6 +50,7 @@ const (
 	clientResponseTimeout   = "SWWAF_CLIENT_RESPONSE_TIMEOUT"
 	upstreamRequestTimeout  = "SWWAF_UPSTREAM_REQUEST_TIMEOUT"
 	upstreamResponseTimeout = "SWWAF_UPSTREAM_RESPONSE_TIMEOUT"
+	mode                    = "SWWAF_MODE"
 	requestMaxBytes         = "SWWAF_REQUEST_MAX_BYTES"
 	responseMaxBytes        = "SWWAF_RESPONSE_MAX_BYTES"
 	trustedProxies          = "SWWAF_TRUSTED_PROXIES"

internal/proxy/ratelimits_test.go

@@ -5,6 +5,7 @@ import (
 	"sync/atomic"
 	"testing"
 
+	"sneak.berlin/go/smallwebwaf/internal/lookup"
 	"sneak.berlin/go/smallwebwaf/internal/requestlog"
 )
 
@@ -75,18 +76,28 @@ func TestRateLimitExemptPathsAreNeitherCountedNorRefused(t *testing.T) {
 
 	const denied = "192.0.2.50" // in SWWAF_DENY_NETS
 
-	s, _, _ := startWithClock(t, "", map[string]string{
+	s, _, server := startWithClock(t, "", map[string]string{
 		rateLimitPerMinute:   "1",
 		rateLimitExemptPaths: "/assets/,/favicon.ico",
 		denyNets:             denied,
+		deniedCountries:      "kp",
+	})
+
+	// The answers are kept before the requests, so that none waits for
+	// GeoJS.
+	server.GeoJS.Load([]lookup.Answer{
+		keptAnswer(client, "DE"), keptAnswer(fromKP, "KP"),
 	})
 
 	// With a limit of one request a minute, the requests for paths under a
 	// prefix are not counted, so client's first request for / is within
 	// the limit; and once client has reached it, they are not refused.
+	// The app acts on /static/../assets/app.js as /assets/app.js.
 	s.request(client, "/assets/app.js", http.StatusOK, requestlog.ActionForward)
 	s.request(client, "/favicon.ico?v=2", http.StatusOK, requestlog.ActionForward)
 	s.get(client, http.StatusOK, requestlog.ActionForward)
+	s.request(client, "/static/../assets/app.js",
+		http.StatusOK, requestlog.ActionForward)
 
 	line := s.request(client, "/assets/app.js", http.StatusOK, requestlog.ActionForward)
 	if line.LimitHit != "" {
@@ -97,7 +108,38 @@ func TestRateLimitExemptPathsAreNeitherCountedNorRefused(t *testing.T) {
 	// /assets/, and breaks the limit.
 	s.request(client, "/assets", http.StatusForbidden, requestlog.ActionRateLimited)
 
-	// A ban and SWWAF_DENY_NETS still refuse a path under a prefix.
+	// A ban, SWWAF_DENY_NETS and the country lists still refuse a path
+	// under a prefix.
 	s.request(client, "/assets/app.js", http.StatusForbidden, requestlog.ActionBanned)
 	s.request(denied, "/assets/app.js", http.StatusForbidden, requestlog.ActionDenied)
+	s.request(fromKP, "/assets/app.js",
+		http.StatusForbidden, requestlog.ActionCountryDenied)
+}
+
+func TestRateLimitCountsPathsOutsideEveryExemptPrefix(t *testing.T) {
+	t.Parallel()
+
+	// A prefix matches only at the start of the path, and the app acts on
+	// the last three paths as /login, once they are percent-decoded and
+	// their .. segments resolved.
+	for _, sent := range []string{
+		"/static/assets/app.js",
+		"/assets/../login",
+		"/assets/%2e%2e/login",
+		"/assets/..%2Flogin",
+	} {
+		t.Run(sent, func(t *testing.T) {
+			t.Parallel()
+
+			s, _, _ := startWithClock(t, "", map[string]string{
+				rateLimitPerMinute:   "1",
+				rateLimitExemptPaths: "/assets/",
+			})
+
+			// Counted, the second request breaks the limit of one request
+			// a minute.
+			s.request(client, sent, http.StatusOK, requestlog.ActionForward)
+			s.request(client, sent, http.StatusForbidden, requestlog.ActionRateLimited)
+		})
+	}
 }

internal/proxy/request.go

@@ -8,6 +8,7 @@ import (
 	"net/http/httputil"
 	"net/netip"
 	"os"
+	"path"
 	"slices"
 	"strings"
 	"sync"
@@ -111,41 +112,24 @@ func (h *handler) newRequest(w http.ResponseWriter, r *http.Request) *request {
 
 // check is the one place where a request can be refused once its client
 // is known, before its body is read or anything reaches the app. It
-// returns nil to let the request through. A client in SWWAF_ALLOW_NETS
+// returns nil to let the request through. The checks of checkClient come
-// skips every check but the size limit. For any other client,
+// first, answered with SWWAF_BAN_RESPONSE, and then the size limit, so
-// SWWAF_DENY_NETS comes first, then a ban on its netblock, so that a
+// that a request the rate limits count is counted even when it is
-// client either refuses is not looked up, and then the country lists; a
+// refused for its size. In observe mode a request checkClient refuses
-// request any of them refuses is not counted for the rate limits. Then
+// goes on to the size limit like any other. ctx is the request's own
-// come the rate limits, unless the client is in
+// context.
-// SWWAF_RATE_LIMIT_EXEMPT_NETS or the request's path, as the client sent
-// it and the log line shows it, starts with one of
-// SWWAF_RATE_LIMIT_EXEMPT_PATHS, so that every other request is counted,
-// one refused for its size too. Every refusal but the size limit's is
-// answered with SWWAF_BAN_RESPONSE. ctx is the request's own context.
 func (rq *request) check(ctx context.Context) *refusal {
-	cfg := rq.h.config
+	action := rq.checkClient(ctx)
-	allowed := isInside(rq.client, cfg.AllowNets)
+	if action != "" {
-	exempt := isInside(rq.client, cfg.RateLimitExemptNets) ||
+		if !rq.h.config.Observe {
-		startsWithAny(rq.in.URL.EscapedPath(), cfg.RateLimitExemptPaths)
+			return rq.banResponse(action)
-	now := rq.h.now()
-
-	if !allowed && isInside(rq.client, cfg.DenyNets) {
-		return rq.banResponse(requestlog.ActionDenied)
 		}
 
-	if !allowed && rq.banned(now) {
+		// The log line names what enforce mode would have done.
-		return rq.banResponse(requestlog.ActionBanned)
+		rq.line.WouldAction = action
 	}
 
-	if !allowed && rq.countryDenied(ctx) {
+	maxBytes := rq.h.config.RequestMaxBytes
-		return rq.banResponse(requestlog.ActionCountryDenied)
-	}
-
-	if !allowed && !exempt && rq.limitBroken(now) {
-		return rq.banResponse(requestlog.ActionRateLimited)
-	}
-
-	maxBytes := cfg.RequestMaxBytes
 	if maxBytes > 0 && rq.in.ContentLength > maxBytes {
 		return &refusal{
 			status: http.StatusRequestEntityTooLarge,
@@ -157,10 +141,54 @@ func (rq *request) check(ctx context.Context) *refusal {
 	return nil
 }
 
-// startsWithAny reports whether path starts with one of prefixes.
+// checkClient runs the checks on the request's client, and returns the
-func startsWithAny(path string, prefixes []string) bool {
+// action of the first that refuses the request, or "" when none does. A
+// client in SWWAF_ALLOW_NETS skips them. For any other client,
+// SWWAF_DENY_NETS comes first, then a ban on its netblock, so that a
+// client either refuses is not looked up, and then the country lists; a
+// request any of them refuses is not counted for the rate limits. Then
+// come the rate limits, unless the client is in
+// SWWAF_RATE_LIMIT_EXEMPT_NETS or the path the app will act on starts
+// with one of SWWAF_RATE_LIMIT_EXEMPT_PATHS, so that every other request
+// is counted. ctx is the request's own context.
+func (rq *request) checkClient(ctx context.Context) string {
+	cfg := rq.h.config
+	if isInside(rq.client, cfg.AllowNets) {
+		return ""
+	}
+
+	now := rq.h.now()
+
+	if isInside(rq.client, cfg.DenyNets) {
+		return requestlog.ActionDenied
+	}
+
+	if rq.banned(now) {
+		return requestlog.ActionBanned
+	}
+
+	if rq.countryDenied(ctx) {
+		return requestlog.ActionCountryDenied
+	}
+
+	// The prefixes are matched against the path the app will act on:
+	// URL.Path is the request's path percent-decoded, and path.Clean
+	// resolves its . and .. segments and repeated slashes, and drops a
+	// trailing slash. So /assets/..%2Flogin is /login, outside /assets/,
+	// whatever the log line's path shows.
+	exempt := isInside(rq.client, cfg.RateLimitExemptNets) ||
+		startsWithAny(path.Clean(rq.in.URL.Path), cfg.RateLimitExemptPaths)
+	if !exempt && rq.limitBroken(now) {
+		return requestlog.ActionRateLimited
+	}
+
+	return ""
+}
+
+// startsWithAny reports whether s starts with one of prefixes.
+func startsWithAny(s string, prefixes []string) bool {
 	return slices.ContainsFunc(prefixes, func(prefix string) bool {
-		return strings.HasPrefix(path, prefix)
+		return strings.HasPrefix(s, prefix)
 	})
 }
 

internal/requestlog/requestlog.go

@@ -67,6 +67,10 @@ type Line struct {
 	Referer        string `json:"referer"`
 	UserAgent      string `json:"user_agent"`
 	Action         string `json:"action"`
+	// WouldAction is, in observe mode, the action enforce mode would have
+	// taken with a request it would have refused: ActionDenied,
+	// ActionBanned, ActionCountryDenied or ActionRateLimited.
+	WouldAction string `json:"would_action,omitempty"`
 	// LimitHit is the window whose rate limit the request went over:
 	// minute, hour or day.
 	LimitHit string `json:"limit_hit,omitempty"`

internal/smallwebwaf/smallwebwaf_test.go

@@ -383,6 +383,7 @@ func wantStartingLine(t *testing.T, line map[string]any, appURL, dir string) {
 		listenAddr:                              localhost + ":0",
 		upstreamURL:                             appURL,
 		stateDir:                                dir,
+		"SWWAF_MODE":                            "enforce",
 		"SWWAF_STATE_WRITE_DELAY":               "10s",
 		"SWWAF_STATE_COUNTER_INTERVAL":          "15m",
 		"SWWAF_TRUSTED_PROXIES":                 "10.0.0.0/8,172.16.0.0/12,192.168.0.0/16",