Leave SWWAF_RATE_LIMIT_EXEMPT_PATHS out of the request rate limits (closes #77)

A request is neither counted nor refused by the request rate limits
when its path as sent, the path the app receives, not percent-decoded,
starts with one of the comma-separated prefixes in
SWWAF_RATE_LIMIT_EXEMPT_PATHS, so /%61ssets/x is not under /assets/. A
request whose decoded path contains .. or a backslash, or whose path as
sent holds an encoded slash, is never exempt, since an app may act on
it as a path outside every prefix, such as /assets/..%2Flogin as
/login. The static lists, bans and the country lists still apply. The
setting is empty by default, and a prefix that does not start with /
stops the start. README.md documents it.

Model: opus-5-5

README.md

@@ -13,21 +13,24 @@ JSON log line for every request.
 
 Status: the first two milestones are built
 (https://git.eeqj.de/sneak/smallwebwaf/issues/13 and
-https://git.eeqj.de/sneak/smallwebwaf/issues/14), and so are five parts of
-milestone 3: the static lists, the bans that broken rate limits lead to and the
-JSON state files with your edits taken in while it runs, which come next in the
-build order, and the metrics endpoint and the header size and the idle time as
-settings, which come last in it. `smallwebwaf` passes each request to the app
-and the app's answer back, unchanged, within its timeouts and size limits, works
-out each client's address, bans a client that sends too many requests, refuses a
-client that comes from a country you refuse or from a network you refuse, lets
-the networks you choose through, keeps its bans, each client's counters and
-history, and GeoJS's answers in JSON files across restarts, takes in your edits
-of those files while it runs, writes a JSON log line for every request, and
-serves Prometheus metrics to a scraper that holds the metrics token. It comes as
-the image the app's own image is built on. The rest of the design comes after
-that, in the order of the build order in [`SPEC.md`](SPEC.md). The survey of
-existing tools that led to the design is in [`EVALUATION.md`](EVALUATION.md).
+https://git.eeqj.de/sneak/smallwebwaf/issues/14), and so are seven parts of
+milestone 3: the static lists, the bans that broken rate limits lead to, the
+JSON state files with your edits taken in while it runs and the paths the rate
+limits do not count, which come next in the build order, `observe` mode, which
+comes a little later, and the metrics endpoint and the header size and the idle
+time as settings, which come last in it. `smallwebwaf` passes each request to
+the app and the app's answer back, unchanged, within its timeouts and size
+limits, works out each client's address, bans a client that sends too many
+requests, not counting those for the paths you choose, refuses a client that
+comes from a country you refuse or from a network you refuse, lets the networks
+you choose through, keeps its bans, each client's counters and history, and
+GeoJS's answers in JSON files across restarts, takes in your edits of those
+files while it runs, writes a JSON log line for every request, serves Prometheus
+metrics to a scraper that holds the metrics token, and in `observe` mode passes
+on the requests it would refuse, logging what it would have done with them. It
+comes as the image the app's own image is built on. The rest of the design comes
+after that, in the order of the build order in [`SPEC.md`](SPEC.md). The survey
+of existing tools that led to the design is in [`EVALUATION.md`](EVALUATION.md).
 
 ## Getting started
 
@@ -78,12 +81,15 @@ in `bin/state` unless `SWWAF_STATE_DIR` is set.
 - Counts each client's requests over a minute, an hour and a day. A request that
   takes the client over one of the rate limits below is refused with
   `SWWAF_BAN_RESPONSE`, `403` by default, before anything reaches the app, and
-  bans the client. A client is one IPv4 address, or one IPv6 /64, since one
-  abuser usually holds a whole /64. Each window is counted in two fixed buckets,
-  the earlier one weighted by how much of it the window still covers. At most
-  20,000 clients are kept, the least recently seen dropped first, with their
-  history, and a restart gives no client a fresh allowance (see "State files"
-  below).
+  bans the client. A request whose path starts with one of
+  `SWWAF_RATE_LIMIT_EXEMPT_PATHS`, as that setting below describes, is neither
+  counted nor refused by the rate limits; the static lists, bans and the country
+  lists still apply to it. A client is one IPv4 address, or one IPv6 /64, since
+  one abuser usually holds a whole /64. Each window is counted in two fixed
+  buckets, the earlier one weighted by how much of it the window still covers.
+  At most 20,000 clients are kept, the least recently seen dropped first, with
+  their history, and a restart gives no client a fresh allowance (see "State
+  files" below).
 - Bans a client that breaks a rate limit, as "Bans" in [`SPEC.md`](SPEC.md)
   describes: the first ban lasts an hour, and a limit broken again within a day
   of a ban ending bans for three times as long as that ban, so 1, 3, 9, 27 and
@@ -119,6 +125,18 @@ in `bin/state` unless `SWWAF_STATE_DIR` is set.
   `SWWAF_ALLOW_NETS` too is let through. A client in
   `SWWAF_RATE_LIMIT_EXEMPT_NETS` is neither counted nor refused by the rate
   limits; the country lists and bans still apply to it.
+- In `observe` mode, with `SWWAF_MODE=observe`, refuses none of the requests
+  that `SWWAF_DENY_NETS`, a ban, the country lists or a rate limit would refuse:
+  it passes them to the app, and their log lines name what `enforce` mode would
+  have done (see `would_action` in "Request log" below). The checks run, and
+  requests are counted, as in `enforce` mode, but a broken rate limit makes no
+  ban and does not set the client's counters back to zero, so each request over
+  the limit is logged as one that would be refused. The bans in `bans.json` are
+  kept, and refuse requests again when `smallwebwaf` next runs in `enforce`
+  mode, as long as they last. The timeouts and size limits still apply, since
+  they protect `smallwebwaf` and the app themselves, and a request for the
+  metrics without the token is still answered `401`. It is for trying a
+  configuration before enforcing it.
 - Answers `GET /_smallwebwaf/healthz` itself with `200` and `ok`, before any
   check and without asking the app, for the image's health check.
 - Answers `GET /_smallwebwaf/metrics` with its metrics (see "Metrics" below) for
@@ -140,6 +158,9 @@ it, and the effective settings are logged at start.
 - `SWWAF_LISTEN_ADDR` (default `:8080`): where `smallwebwaf` listens.
 - `SWWAF_UPSTREAM_URL` (default `http://127.0.0.1:8081`): the app, as `http` or
   `https`, a host and an optional port, and nothing more.
+- `SWWAF_MODE` (default `enforce`): `enforce`, or `observe` to pass on the
+  requests `smallwebwaf` would refuse and log what it would have done (see "What
+  it does so far" above).
 - `SWWAF_TRUSTED_PROXIES` (default `10.0.0.0/8,172.16.0.0/12,192.168.0.0/16`,
   the private address ranges): the netblocks whose `X-Forwarded-For` is
   believed. A list given replaces the default; set but empty, it trusts nothing.
@@ -175,6 +196,20 @@ it, and the effective settings are logged at start.
   requests a client may make in a minute, an hour and a day. The defaults are
   several times what one busy person produces, since a browser loading a heavy
   page makes a few hundred requests and several people often share one address.
+- `SWWAF_RATE_LIMIT_EXEMPT_PATHS` (default empty): path prefixes whose requests
+  the rate limits neither count nor refuse, such as `/assets/` for static
+  assets; each starts with `/`. A request whose path, percent-decoded, contains
+  `..` anywhere or a backslash, or whose path as sent holds an encoded slash
+  (`%2F` or `%2f`), is never exempt, since the app may act on it as a path
+  outside every prefix: `/assets/..%2Flogin` as `/login`. Any other request is
+  exempt when its path as sent, the path the app receives, before any query
+  string and not percent-decoded, starts with a prefix, character for character.
+  `/assets/` matches `/assets/app.js` and `/assets/`, but not `/assets`,
+  `/Assets/app.js`, `/%61ssets/app.js`, `/static/assets/app.js`,
+  `/static/../assets/app.js` or `/assets%2Fapp.js`. A character the client sends
+  percent-encoded, such as a space, is written percent-encoded in a prefix, as
+  in `/my%20files/`, and there are no wildcards: `*` is a character like any
+  other.
 - `SWWAF_DENIED_COUNTRIES` (default empty): countries whose clients are refused,
   for example `cn,ru,kp`.
 - `SWWAF_EXCLUSIVELY_ALLOWED_COUNTRIES` (default empty): when set, the only
@@ -241,8 +276,8 @@ refused ones included:
 - `country` is the client's country as GeoJS places it. It is empty with neither
   country list set, for a client in `SWWAF_ALLOW_NETS` or `SWWAF_DENY_NETS`, for
   a client on a private, loopback or link-local address, when GeoJS cannot place
-  the client or has not answered in time, and for a request refused because a
-  ban covers its client, even when the client's country is known.
+  the client or has not answered in time, and for a request whose client a ban
+  covers, even when the client's country is known.
 - `status` is what the client was sent, `0` if nothing was; `upstream_status` is
   what the app answered, and is left out when the app did not answer.
 - `request_bytes` and `response_bytes` count body bytes.
@@ -254,11 +289,18 @@ refused ones included:
   `timed_out` for one that ran out of time, `upstream_error` when the app could
   not be reached or its answer broke off, and `admin` for one `smallwebwaf`
   answered at its own endpoint.
+- `would_action` is there in `observe` mode for a request that
+  `SWWAF_DENY_NETS`, a ban, the country lists or a rate limit would have refused
+  in `enforce` mode, and names the action that refusal would have had: `denied`,
+  `banned`, `country_denied` or `rate_limited`. `action` then names what was
+  done: `forward` for a request passed to the app, and another action, such as
+  `too_large`, for one a size or time limit refused.
 - `limit_hit` is there for a request that broke a rate limit, and names the
   window whose limit it went over: `minute`, `hour` or `day`, the shortest if it
   went over several. `offence` is then `limit`.
 - `ban_expires` is there for a request that made a ban or was refused under one,
-  and gives when the ban ends, in the same form as `time`, or `permanent`.
+  or in `observe` mode would have been refused under one, and gives when the ban
+  ends, in the same form as `time`, or `permanent`.
 - `aborted` is there, and true, when the client went away early.
 - `duration_total` and `duration_upstream_total` are in milliseconds.
 
@@ -618,17 +660,16 @@ the metrics, failure behaviour and the build order.
 So far `smallwebwaf` looks up only the country, only through GeoJS, and only
 while `SWWAF_DENIED_COUNTRIES` or `SWWAF_EXCLUSIVELY_ALLOWED_COUNTRIES` is set:
 then the address of every new visitor is sent to GeoJS, except a visitor in
-`SWWAF_ALLOW_NETS` or `SWWAF_DENY_NETS` and one refused because a ban covers its
-netblock, and with neither set, none is. An IPv6 visitor is asked about by the
-first address of its /64. A new visitor waits at most a second for its answer,
-and without one counts as coming from an unknown country until the answer
-arrives. The addresses waiting are asked about together, up to 200 in one
-request, one request at a time; at most 10,000 visitors wait, and one more
-counts as coming from an unknown country until there is room. While GeoJS fails,
-visitors with a kept answer are unaffected and new ones count as coming from an
-unknown country. GeoJS is then left alone for a second, twice as long after each
-further failure up to five minutes, and asked again by the next request that
-needs it.
+`SWWAF_ALLOW_NETS` or `SWWAF_DENY_NETS` and one whose netblock a ban covers, and
+with neither set, none is. An IPv6 visitor is asked about by the first address
+of its /64. A new visitor waits at most a second for its answer, and without one
+counts as coming from an unknown country until the answer arrives. The addresses
+waiting are asked about together, up to 200 in one request, one request at a
+time; at most 10,000 visitors wait, and one more counts as coming from an
+unknown country until there is room. While GeoJS fails, visitors with a kept
+answer are unaffected and new ones count as coming from an unknown country.
+GeoJS is then left alone for a second, twice as long after each further failure
+up to five minutes, and asked again by the next request that needs it.
 
 In the full design, `smallwebwaf` looks up the AS number and country of every
 client, for the request log, the metrics and the ban notes, and for the country
@@ -680,8 +721,10 @@ addresses are never sent to GeoJS.
   limits, and writes the request's log line. Its `check` method is where a
   request is refused before anything reaches the app: for `SWWAF_DENY_NETS`, for
   a ban, for the country lists, for a rate limit, which bans the client, and for
-  an announced body over the size limit. A request under `/_smallwebwaf/` that
-  `check` lets through is answered by `answerAdmin` instead of reaching the app.
+  an announced body over the size limit; in `observe` mode, only for the size
+  limit, with what it would have refused for noted in the log line. A request
+  under `/_smallwebwaf/` that `check` lets through is answered by `answerAdmin`
+  instead of reaching the app.
 - `internal/metrics`: the metrics, counted as the other parts tell it what
   happened, and served in the Prometheus text format.
 - `internal/bans`: the ban ledger: each netblock's bans with their notes, how
@@ -748,9 +791,8 @@ so that they run in minimal containers.
 
 ## TODO
 
-- The rest of milestone 3, from exemptions up to the rest of the request log's
-  fields, and the rest of the design, in the order of the build order in
-  [`SPEC.md`](SPEC.md).
+- The rest of milestone 3: the rest of the request log's fields; then the rest
+  of the design, in the order of the build order in [`SPEC.md`](SPEC.md).
 
 ## Documents
 

go.sum

@@ -32,7 +32,6 @@ go.uber.org/goleak v1.3.0 h1:2K3zAYmnTNqV73imy9J1T3WC+gmCePx2hEGkimedGto=
 go.uber.org/goleak v1.3.0/go.mod h1:CoHD4mav9JJNrW/WLlf7HGZPjdw8EucARQHekz1X6bE=
 go.yaml.in/yaml/v2 v2.4.4 h1:tuyd0P+2Ont/d6e2rl3be67goVK4R6deVxCUX5vyPaQ=
 go.yaml.in/yaml/v2 v2.4.4/go.mod h1:gMZqIpDtDqOfM0uNfy0SkpRhvUryYH0Z6wdMYcacYXQ=
-golang.org/x/sys v0.13.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
 golang.org/x/sys v0.47.0 h1:o7XGOvZQCADBQQ4Y7VNq2dRWQR7JmOUW8Kxx4ZsNgWs=
 golang.org/x/sys v0.47.0/go.mod h1:4GL1E5IUh+htKOUEOaiffhrAeqysfVGipDYzABqnCmw=
 google.golang.org/protobuf v1.36.11 h1:fV6ZwhNocDyBLK0dj+fg8ektcVegBBuEolpbTQyBNVE=

internal/bans/bans.go

@@ -107,8 +107,8 @@ type Ledger struct {
 	changed chan struct{}
 
 	mu sync.Mutex
-	// netblocks holds each banned netblock's bans, oldest first. Check
-	// makes each netblock it finds the most recently seen.
+	// netblocks holds each banned netblock's bans, oldest first. Check and
+	// Find make each netblock they find the most recently seen.
 	netblocks *simplelru.LRU[netip.Prefix, *[]Ban]
 	// held is how many bans netblocks holds, at most rules.MaxBans.
 	held int
@@ -144,34 +144,36 @@ func (l *Ledger) Changed() <-chan struct{} {
 	return l.changed
 }
 
-// Check is called for each request from client, at now. It reports
-// whether a ban on a netblock client is in is active, and returns that
-// ban, with the request counted among those it refused.
+// Check is called for a request from client, at now. It reports whether
+// a ban on a netblock client is in is active, and returns that ban, with
+// the request counted among those it refused.
 func (l *Ledger) Check(client netip.Addr, now time.Time) (Ban, bool) {
 	l.mu.Lock()
 	defer l.mu.Unlock()
 
-	lengths := l.v6Lengths
-	if client.Is4() {
-		lengths = l.v4Lengths
+	ban := l.active(client, now)
+	if ban == nil {
+		return Ban{}, false
 	}
 
-	for _, length := range lengths {
-		bans, found := l.netblocks.Get(netip.PrefixFrom(client, length).Masked())
-		if !found {
-			continue
-		}
+	ban.Notes.Requests++
+	ban.Notes.Refused++
 
-		ban := activeBan(*bans, now)
-		if ban != nil {
-			ban.Notes.Requests++
-			ban.Notes.Refused++
+	return *ban, true
+}
 
-			return *ban, true
-		}
+// Find is Check without counting the request among those the ban
+// refused: in observe mode a ban refuses nothing.
+func (l *Ledger) Find(client netip.Addr, now time.Time) (Ban, bool) {
+	l.mu.Lock()
+	defer l.mu.Unlock()
+
+	ban := l.active(client, now)
+	if ban == nil {
+		return Ban{}, false
 	}
 
-	return Ban{}, false
+	return *ban, true
 }
 
 // activeBan returns the ban in bans, a netblock's bans oldest first, that
@@ -330,6 +332,29 @@ func (l *Ledger) Load(bans []Ban) {
 	}
 }
 
+// active returns the ban active at now on a netblock client is in, or
+// nil.
+func (l *Ledger) active(client netip.Addr, now time.Time) *Ban {
+	lengths := l.v6Lengths
+	if client.Is4() {
+		lengths = l.v4Lengths
+	}
+
+	for _, length := range lengths {
+		bans, found := l.netblocks.Get(netip.PrefixFrom(client, length).Masked())
+		if !found {
+			continue
+		}
+
+		ban := activeBan(*bans, now)
+		if ban != nil {
+			return ban
+		}
+	}
+
+	return nil
+}
+
 // add adds ban to its netblock's bans, after the last, and makes its
 // netblock the most recently seen. With MaxBans held, it drops one first.
 func (l *Ledger) add(ban Ban) {

internal/bans/bans_test.go

@@ -162,6 +162,28 @@ func TestCheckRefusesWhileTheBanLastsAndCountsTheRefusals(t *testing.T) {
 	}
 }
 
+func TestFindCountsNothing(t *testing.T) {
+	t.Parallel()
+
+	ledger := bans.New(defaultRules())
+	netblock := netip.MustParsePrefix("203.0.113.9/32")
+	ban := ledger.BanForLimit(netblock, midnight(), bans.Notes{Requests: 5})
+
+	got, banned := ledger.Find(netblock.Addr(), ban.Expires.Add(-time.Nanosecond))
+	if !banned || got != ban {
+		t.Errorf("find during the ban gives %+v and %t, want %+v", got, banned, ban)
+	}
+
+	_, banned = ledger.Find(netblock.Addr(), ban.Expires)
+	if banned {
+		t.Error("the ban did not end")
+	}
+
+	if notes := ledger.Bans(netblock)[0].Notes; notes != ban.Notes {
+		t.Errorf("the notes are %+v, want them unchanged, %+v", notes, ban.Notes)
+	}
+}
+
 func TestMaxBansDropsTheEarliestBanOfTheNetblockSeenLongestAgo(t *testing.T) {
 	t.Parallel()
 

internal/bans/snapshot_test.go

@@ -147,8 +147,14 @@ func TestPermanentBanStartedBeforeAnEndedOneRefuses(t *testing.T) {
 	ledger.Load([]bans.Ban{permanent, ended})
 
 	now := midnight().Add(2 * time.Hour)
+	client := netip.MustParseAddr("203.0.113.9")
 
-	ban, banned := ledger.Check(netip.MustParseAddr("203.0.113.9"), now)
+	ban, banned := ledger.Find(client, now)
+	if !banned || !ban.Permanent() {
+		t.Errorf("find gives %+v and %t, want the permanent ban", ban, banned)
+	}
+
+	ban, banned = ledger.Check(client, now)
 	if !banned || !ban.Permanent() {
 		t.Errorf("the client is refused: %t, under %+v, want under the permanent ban",
 			banned, ban)

internal/config/config.go

@@ -27,6 +27,11 @@ type Config struct {
 	ListenAddr string
 	// UpstreamURL is the app (SWWAF_UPSTREAM_URL).
 	UpstreamURL *url.URL
+	// Observe is true in observe mode, when SWWAF_MODE is observe rather
+	// than enforce: a request that SWWAF_DENY_NETS, a ban, the country
+	// lists or a rate limit would refuse is passed to the app instead, and
+	// no ban is made.
+	Observe bool
 	// TrustedProxies are the netblocks whose X-Forwarded-For is
 	// believed (SWWAF_TRUSTED_PROXIES).
 	TrustedProxies []netip.Prefix
@@ -70,6 +75,10 @@ type Config struct {
 	RateLimitPerMinute int64
 	RateLimitPerHour   int64
 	RateLimitPerDay    int64
+	// RateLimitExemptPaths are the path prefixes whose requests the rate
+	// limits neither count nor refuse (SWWAF_RATE_LIMIT_EXEMPT_PATHS).
+	// Each starts with /.
+	RateLimitExemptPaths []string
 	// DeniedCountries are the countries whose clients are refused
 	// (SWWAF_DENIED_COUNTRIES). ExclusivelyAllowedCountries, when not
 	// empty, are the only countries whose clients are let through
@@ -161,7 +170,10 @@ var (
 		"is not the length of an IPv4 netblock, from 0 to 32, such as 24")
 	errNotAbsolutePath = errors.New(
 		"is not an absolute path, such as /var/lib/smallwebwaf")
-	errShortToken = errors.New("is shorter than 32 characters")
+	errShortToken    = errors.New("is shorter than 32 characters")
+	errNotMode       = errors.New("is not enforce or observe")
+	errNotPathPrefix = errors.New(
+		"is not a path prefix starting with /, such as /assets/")
 )
 
 // FromEnvironment reads the settings with lookupEnv, normally
@@ -172,6 +184,7 @@ func FromEnvironment(lookupEnv func(string) (string, bool)) (*Config, error) {
 	cfg := &Config{
 		ListenAddr:           env.address("SWWAF_LISTEN_ADDR", ":8080"),
 		UpstreamURL:          env.appURL("SWWAF_UPSTREAM_URL", "http://127.0.0.1:8081"),
+		Observe:              env.observe("SWWAF_MODE", "enforce"),
 		TrustedProxies:       env.netblocks("SWWAF_TRUSTED_PROXIES", privateRanges),
 		ClientRequestTimeout: env.duration("SWWAF_CLIENT_REQUEST_TIMEOUT", "60s"),
 		ClientRequestHeaderMaxBytes: env.headerSize(
@@ -188,6 +201,7 @@ func FromEnvironment(lookupEnv func(string) (string, bool)) (*Config, error) {
 		RateLimitPerMinute:      env.count("SWWAF_RATE_LIMIT_PER_MINUTE", "1000"),
 		RateLimitPerHour:        env.count("SWWAF_RATE_LIMIT_PER_HOUR", "10000"),
 		RateLimitPerDay:         env.count("SWWAF_RATE_LIMIT_PER_DAY", "50000"),
+		RateLimitExemptPaths:    env.pathPrefixes("SWWAF_RATE_LIMIT_EXEMPT_PATHS", ""),
 		DeniedCountries:         env.countries("SWWAF_DENIED_COUNTRIES", ""),
 		ExclusivelyAllowedCountries: env.countries(
 			"SWWAF_EXCLUSIVELY_ALLOWED_COUNTRIES", ""),
@@ -274,6 +288,17 @@ func (e *environment) appURL(name, defaultValue string) *url.URL {
 	return upstream
 }
 
+// observe reads the setting that is the mode, enforce or observe, and
+// reports whether it is observe.
+func (e *environment) observe(name, defaultValue string) bool {
+	mode := e.value(name, defaultValue)
+	if mode != "enforce" && mode != "observe" {
+		e.check(name, fmt.Errorf("%q %w", mode, errNotMode))
+	}
+
+	return mode == "observe"
+}
+
 // netblocks reads a setting that is a list of netblocks.
 func (e *environment) netblocks(name, defaultValue string) []netip.Prefix {
 	netblocks, err := parseNetblocks(e.value(name, defaultValue))
@@ -315,6 +340,14 @@ func (e *environment) count(name, defaultValue string) int64 {
 	return count
 }
 
+// pathPrefixes reads a setting that is a list of path prefixes.
+func (e *environment) pathPrefixes(name, defaultValue string) []string {
+	prefixes, err := parsePathPrefixes(e.value(name, defaultValue))
+	e.check(name, err)
+
+	return prefixes
+}
+
 // countries reads a setting that is a list of countries.
 func (e *environment) countries(name, defaultValue string) []string {
 	countries, err := parseCountries(e.value(name, defaultValue))
@@ -593,6 +626,23 @@ func parseNetblock(value string) (netip.Prefix, error) {
 	return netip.PrefixFrom(addr, addr.BitLen()), nil
 }
 
+// parsePathPrefixes reads a comma-separated list of path prefixes, each
+// starting with /.
+func parsePathPrefixes(value string) ([]string, error) {
+	prefixes, err := parseList(value)
+	if err != nil {
+		return nil, err
+	}
+
+	for _, prefix := range prefixes {
+		if !strings.HasPrefix(prefix, "/") {
+			return nil, fmt.Errorf("%q %w", prefix, errNotPathPrefix)
+		}
+	}
+
+	return prefixes, nil
+}
+
 // countryCodes are the two-letter codes ISO 3166-1 assigns today, and XK,
 // the code in common use for Kosovo. golang.org/x/text/language cannot
 // check them: it also takes withdrawn codes such as su, and reserved ones

internal/config/config_test.go

@@ -18,6 +18,7 @@ import (
 const (
 	listenAddr              = "SWWAF_LISTEN_ADDR"
 	upstreamURL             = "SWWAF_UPSTREAM_URL"
+	mode                    = "SWWAF_MODE"
 	trustedProxies          = "SWWAF_TRUSTED_PROXIES"
 	clientRequestTimeout    = "SWWAF_CLIENT_REQUEST_TIMEOUT"
 	clientHeaderMaxBytes    = "SWWAF_CLIENT_REQUEST_HEADER_MAX_BYTES"
@@ -33,6 +34,7 @@ const (
 	rateLimitPerMinute      = "SWWAF_RATE_LIMIT_PER_MINUTE"
 	rateLimitPerHour        = "SWWAF_RATE_LIMIT_PER_HOUR"
 	rateLimitPerDay         = "SWWAF_RATE_LIMIT_PER_DAY"
+	rateLimitExemptPaths    = "SWWAF_RATE_LIMIT_EXEMPT_PATHS"
 	deniedCountries         = "SWWAF_DENIED_COUNTRIES"
 	allowedCountries        = "SWWAF_EXCLUSIVELY_ALLOWED_COUNTRIES"
 	banResponse             = "SWWAF_BAN_RESPONSE"
@@ -83,6 +85,7 @@ func TestDefaults(t *testing.T) {
 
 	wantSettings(t, cfg, config.Config{
 		ListenAddr:                  ":8080",
+		Observe:                     false,
 		ClientRequestTimeout:        time.Minute,
 		ClientRequestHeaderMaxBytes: 32 << 10,
 		ClientIdleTimeout:           2 * time.Minute,
@@ -118,6 +121,10 @@ func TestDefaults(t *testing.T) {
 	wantNetblocks(t, cfg.DenyNets)
 	wantCountries(t, deniedCountries, cfg.DeniedCountries)
 	wantCountries(t, allowedCountries, cfg.ExclusivelyAllowedCountries)
+
+	if len(cfg.RateLimitExemptPaths) != 0 {
+		t.Errorf("%s gave %v, want none", rateLimitExemptPaths, cfg.RateLimitExemptPaths)
+	}
 }
 
 func TestValuesAsSet(t *testing.T) {
@@ -126,6 +133,7 @@ func TestValuesAsSet(t *testing.T) {
 	cfg := fromEnvironment(t, environment{
 		listenAddr:              "127.0.0.1:9000",
 		upstreamURL:             "https://app.internal:8443/",
+		mode:                    "observe",
 		trustedProxies:          " 192.0.2.1, 10.1.2.3/8 ,2001:db8::/32",
 		clientRequestTimeout:    "90s",
 		clientHeaderMaxBytes:    "8K",
@@ -141,6 +149,7 @@ func TestValuesAsSet(t *testing.T) {
 		rateLimitPerMinute:      "60",
 		rateLimitPerHour:        "600",
 		rateLimitPerDay:         "6000",
+		rateLimitExemptPaths:    "/assets/, /favicon.ico",
 		deniedCountries:         "cn, RU,kp,Xk",
 		allowedCountries:        "de",
 		banResponse:             "429",
@@ -158,6 +167,7 @@ func TestValuesAsSet(t *testing.T) {
 
 	wantSettings(t, cfg, config.Config{
 		ListenAddr:                  "127.0.0.1:9000",
+		Observe:                     true,
 		ClientRequestTimeout:        90 * time.Second,
 		ClientRequestHeaderMaxBytes: 8 << 10,
 		ClientIdleTimeout:           5 * time.Minute,
@@ -192,6 +202,24 @@ func TestValuesAsSet(t *testing.T) {
 	wantNetblocks(t, cfg.DenyNets, "198.51.100.0/24")
 	wantCountries(t, deniedCountries, cfg.DeniedCountries, "CN", "RU", "KP", "XK")
 	wantCountries(t, allowedCountries, cfg.ExclusivelyAllowedCountries, "DE")
+
+	if !slices.Equal(cfg.RateLimitExemptPaths, []string{"/assets/", "/favicon.ico"}) {
+		t.Errorf("%s gave %v, want /assets/ and /favicon.ico",
+			rateLimitExemptPaths, cfg.RateLimitExemptPaths)
+	}
+}
+
+func TestPathPrefixNotStartingWithSlashStopsTheStart(t *testing.T) {
+	t.Parallel()
+
+	_, err := config.FromEnvironment(
+		environment{rateLimitExemptPaths: "/favicon.ico,assets/"}.lookupEnv)
+
+	want := rateLimitExemptPaths + `: "assets/" is not a path prefix ` +
+		`starting with /, such as /assets/`
+	if err == nil || err.Error() != want {
+		t.Errorf("error %v, want %s", err, want)
+	}
 }
 
 func TestCodeOnBothCountryListsStopsTheStart(t *testing.T) {
@@ -307,6 +335,7 @@ func TestInvalidValueStopsTheStart(t *testing.T) {
 		{upstreamURL, "http://127.0.0.1:8081/app"},
 		{upstreamURL, "http://127.0.0.1:8081/?a=1"},
 		{upstreamURL, "http://user:secret@127.0.0.1:8081"},
+		{mode, "Observe"}, {mode, "block"}, {mode, ""},
 		{trustedProxies, "10.0.0.0/33"},
 		{trustedProxies, "traefik"},
 		{trustedProxies, "10.0.0.0/8,,192.168.0.0/16"},
@@ -334,6 +363,7 @@ func TestInvalidValueStopsTheStart(t *testing.T) {
 		{rateLimitPerHour, "1.5"},
 		{rateLimitPerDay, "-1"},
 		{rateLimitPerDay, "lots"},
+		{rateLimitExemptPaths, "/assets/,,/static/"},
 		{deniedCountries, "nk"},
 		{deniedCountries, "kp,,ir"},
 		{deniedCountries, "prk"},
@@ -426,6 +456,7 @@ func TestLogsEachSettingWithItsValue(t *testing.T) {
 	want := map[string]string{
 		listenAddr:              ":8080",
 		upstreamURL:             "http://127.0.0.1:8081",
+		mode:                    "enforce",
 		trustedProxies:          "10.0.0.0/8,172.16.0.0/12,192.168.0.0/16",
 		clientRequestTimeout:    "45s",
 		clientHeaderMaxBytes:    "32K",
@@ -441,6 +472,7 @@ func TestLogsEachSettingWithItsValue(t *testing.T) {
 		rateLimitPerMinute:      "1000",
 		rateLimitPerHour:        "10000",
 		rateLimitPerDay:         "50000",
+		rateLimitExemptPaths:    "",
 		deniedCountries:         "",
 		allowedCountries:        "",
 		banResponse:             "403",
@@ -465,6 +497,7 @@ func wantSettings(t *testing.T, got *config.Config, want config.Config) {
 	t.Helper()
 
 	if got.ListenAddr != want.ListenAddr ||
+		got.Observe != want.Observe ||
 		got.ClientRequestTimeout != want.ClientRequestTimeout ||
 		got.ClientRequestHeaderMaxBytes != want.ClientRequestHeaderMaxBytes ||
 		got.ClientIdleTimeout != want.ClientIdleTimeout ||

internal/proxy/bans.go

@@ -14,10 +14,15 @@ func (rq *request) banResponse(action string) *refusal {
 	return &refusal{status: rq.h.config.BanResponse, action: action}
 }
 
-// banned reports whether a ban on a netblock the client is in refuses
-// the request at now, and notes for the log line when that ban ends.
+// banned reports whether a ban on a netblock the client is in covers the
+// request at now, and notes for the log line when that ban ends.
 func (rq *request) banned(now time.Time) bool {
-	ban, banned := rq.h.ledger.Check(rq.client, now)
+	check := rq.h.ledger.Check
+	if rq.h.config.Observe {
+		check = rq.h.ledger.Find // in observe mode the ban refuses nothing
+	}
+
+	ban, banned := check(rq.client, now)
 	if banned {
 		rq.line.BanExpires = banExpires(ban)
 	}
@@ -26,8 +31,9 @@ func (rq *request) banned(now time.Time) bool {
 }
 
 // limitBroken counts the request for the rate limits at now, and reports
-// whether it takes the client over one. Such a request bans the client's
-// netblock, and sets the client's counters back to zero.
+// whether it takes the client over one. In enforce mode such a request
+// bans the client's netblock, and sets the client's counters back to
+// zero; in observe mode it does neither.
 func (rq *request) limitBroken(now time.Time) bool {
 	group := clientGroup(rq.client)
 
@@ -36,6 +42,13 @@ func (rq *request) limitBroken(now time.Time) bool {
 		return false
 	}
 
+	rq.line.LimitHit = hit.Window
+	rq.line.Offence = requestlog.OffenceLimit
+
+	if rq.h.config.Observe {
+		return true
+	}
+
 	netblock := rq.netblock()
 	ban := rq.h.ledger.BanForLimit(netblock, now, bans.Notes{
 		Country: rq.line.Country,
@@ -54,9 +67,6 @@ func (rq *request) limitBroken(now time.Time) bool {
 		Requests: rq.h.limiter.Requests(netblock) + 1,
 	})
 	rq.h.limiter.Reset(group)
-
-	rq.line.LimitHit = hit.Window
-	rq.line.Offence = requestlog.OffenceLimit
 	rq.line.BanExpires = banExpires(ban)
 
 	return true

internal/proxy/observe_test.go

@@ -0,0 +1,202 @@
+package proxy_test
+
+import (
+	"bytes"
+	"io"
+	"net/http"
+	"net/netip"
+	"sync/atomic"
+	"testing"
+	"time"
+
+	"sneak.berlin/go/smallwebwaf/internal/bans"
+	"sneak.berlin/go/smallwebwaf/internal/proxy"
+	"sneak.berlin/go/smallwebwaf/internal/requestlog"
+)
+
+// observe is the value of SWWAF_MODE for observe mode.
+const observe = "observe"
+
+func TestObserveModeForwardsWhatEnforceModeRefuses(t *testing.T) {
+	t.Parallel()
+
+	const (
+		denied = "192.0.2.50" // in SWWAF_DENY_NETS
+		banned = otherClient  // under a ban read from bans.json
+	)
+
+	for _, tc := range []struct {
+		setting string // "" leaves SWWAF_MODE at its default
+		observe bool
+	}{
+		{"", false},
+		{"enforce", false},
+		{observe, true},
+	} {
+		t.Run(mode+"="+tc.setting, func(t *testing.T) {
+			t.Parallel()
+
+			geojsURL, _ := startGeoJS(t)
+			env := map[string]string{
+				rateLimitPerMinute: "1",
+				denyNets:           denied,
+				deniedCountries:    "kp",
+			}
+
+			if tc.setting != "" {
+				env[mode] = tc.setting
+			}
+
+			s, clk, server := startWithClock(t, geojsURL, env)
+			server.Ledger.Load([]bans.Ban{{
+				Netblock: netip.MustParsePrefix(banned + "/32"),
+				Start:    clk.Now(),
+				Expires:  clk.Now().Add(time.Hour),
+			}})
+
+			// fromDE's first request is within the limit of one a minute,
+			// and its second breaks it.
+			s.get(fromDE, http.StatusOK, requestlog.ActionForward)
+
+			for _, sent := range []struct{ from, refusal string }{
+				{denied, requestlog.ActionDenied},
+				{banned, requestlog.ActionBanned},
+				{fromKP, requestlog.ActionCountryDenied},
+				{fromDE, requestlog.ActionRateLimited},
+			} {
+				if !tc.observe {
+					line := s.get(sent.from, http.StatusForbidden, sent.refusal)
+					wantWouldAction(t, line, "")
+
+					continue
+				}
+
+				// Passed to the app, which answered it.
+				line := s.get(sent.from, http.StatusOK, requestlog.ActionForward)
+				wantWouldAction(t, line, sent.refusal)
+
+				if line.UpstreamStatus != http.StatusOK {
+					t.Errorf("log line has upstream_status %d, want 200",
+						line.UpstreamStatus)
+				}
+			}
+		})
+	}
+}
+
+func TestObserveModeMakesNoBanAndKeepsTheBansItHas(t *testing.T) {
+	t.Parallel()
+
+	s, clk, server := startWithClock(t, "", map[string]string{
+		mode:               observe,
+		rateLimitPerMinute: "1",
+	})
+	kept := bans.Ban{
+		Netblock: netip.MustParsePrefix(otherClient + "/32"),
+		Start:    clk.Now(),
+		Expires:  clk.Now().Add(time.Hour),
+	}
+	server.Ledger.Load([]bans.Ban{kept})
+
+	// No ban sets client's counters back to zero, so each request after
+	// the first breaks the limit of one a minute.
+	s.get(client, http.StatusOK, requestlog.ActionForward)
+
+	for range 2 {
+		line := s.get(client, http.StatusOK, requestlog.ActionForward)
+		wantWouldAction(t, line, requestlog.ActionRateLimited)
+
+		if line.LimitHit != minute || line.Offence != requestlog.OffenceLimit ||
+			line.BanExpires != "" {
+			t.Errorf("log line has limit_hit %q, offence %q and ban_expires %q, "+
+				"want minute, limit and none", line.LimitHit, line.Offence,
+				line.BanExpires)
+		}
+	}
+
+	// The ban read from bans.json refuses nothing, and so counts no
+	// refusal in its notes, but is kept.
+	line := s.get(otherClient, http.StatusOK, requestlog.ActionForward)
+	wantWouldAction(t, line, requestlog.ActionBanned)
+
+	if line.BanExpires != requestlog.FormatTime(kept.Expires) {
+		t.Errorf("log line has ban_expires %q, want %s", line.BanExpires,
+			requestlog.FormatTime(kept.Expires))
+	}
+
+	got := server.Ledger.Snapshot()
+	if len(got) != 1 || got[0] != kept {
+		t.Errorf("bans\n%+v\nwant only\n%+v", got, kept)
+	}
+}
+
+func TestObserveModeKeepsTheSizeLimitsAndTheToken(t *testing.T) {
+	t.Parallel()
+
+	const denied = "192.0.2.50" // in SWWAF_DENY_NETS
+
+	var calls atomic.Int32
+
+	app := startApp(t, func(w http.ResponseWriter, _ *http.Request) {
+		calls.Add(1)
+		answerWithSize(w, 2*sizeLimit, true)
+	})
+	addr, out := startProxy(t, app.URL, map[string]string{
+		mode:             observe,
+		trustedProxies:   trustLocalhost,
+		denyNets:         denied,
+		requestMaxBytes:  sizeLimitSetting,
+		responseMaxBytes: sizeLimitSetting,
+		metricsToken:     token,
+	})
+
+	// SWWAF_DENY_NETS would refuse each request; instead a size limit or
+	// the missing token does.
+	for i, tc := range []struct {
+		method, path string
+		body         io.Reader
+		status       int
+		action       string
+	}{
+		{
+			http.MethodPost, "/upload", bytes.NewReader(make([]byte, 2*sizeLimit)),
+			http.StatusRequestEntityTooLarge, requestlog.ActionTooLarge,
+		},
+		{
+			http.MethodGet, "/download", http.NoBody,
+			http.StatusBadGateway, requestlog.ActionTooLarge,
+		},
+		{
+			http.MethodGet, proxy.MetricsPath, http.NoBody,
+			http.StatusUnauthorized, requestlog.ActionAdmin,
+		},
+	} {
+		req := newRequest(t, tc.method, addr, tc.path, tc.body)
+		req.Header.Set(forwardedFor, denied)
+		wantStatus(t, do(t, req), tc.status)
+
+		line := out.requestLines(t, i+1)[i]
+		wantLine(t, line, tc.status, tc.action)
+		wantWouldAction(t, line, requestlog.ActionDenied)
+	}
+
+	// The upload was refused before it reached the app.
+	if calls.Load() != 1 {
+		t.Errorf("the app was called %d times, want once", calls.Load())
+	}
+}
+
+// wantWouldAction checks the request log line's would_action, and that a
+// line that should have none has no such field.
+func wantWouldAction(t *testing.T, line logLine, want string) {
+	t.Helper()
+
+	got, present := line.fields["would_action"]
+
+	switch {
+	case want == "" && present:
+		t.Errorf("log line has would_action %v, want none", got)
+	case want != "" && got != want:
+		t.Errorf("log line has would_action %v, want %s", got, want)
+	}
+}

internal/proxy/proxy_test.go

@@ -50,6 +50,7 @@ const (
 	clientResponseTimeout   = "SWWAF_CLIENT_RESPONSE_TIMEOUT"
 	upstreamRequestTimeout  = "SWWAF_UPSTREAM_REQUEST_TIMEOUT"
 	upstreamResponseTimeout = "SWWAF_UPSTREAM_RESPONSE_TIMEOUT"
+	mode                    = "SWWAF_MODE"
 	requestMaxBytes         = "SWWAF_REQUEST_MAX_BYTES"
 	responseMaxBytes        = "SWWAF_RESPONSE_MAX_BYTES"
 	trustedProxies          = "SWWAF_TRUSTED_PROXIES"
@@ -58,6 +59,7 @@ const (
 	denyNets                = "SWWAF_DENY_NETS"
 	rateLimitPerMinute      = "SWWAF_RATE_LIMIT_PER_MINUTE"
 	rateLimitPerDay         = "SWWAF_RATE_LIMIT_PER_DAY"
+	rateLimitExemptPaths    = "SWWAF_RATE_LIMIT_EXEMPT_PATHS"
 	deniedCountries         = "SWWAF_DENIED_COUNTRIES"
 	allowedCountries        = "SWWAF_EXCLUSIVELY_ALLOWED_COUNTRIES"
 	banResponse             = "SWWAF_BAN_RESPONSE"

internal/proxy/ratelimits_test.go

@@ -5,6 +5,7 @@ import (
 	"sync/atomic"
 	"testing"
 
+	"sneak.berlin/go/smallwebwaf/internal/lookup"
 	"sneak.berlin/go/smallwebwaf/internal/requestlog"
 )
 
@@ -69,3 +70,84 @@ func TestRateLimitRefusesBeforeTheApp(t *testing.T) {
 		t.Errorf("the app was called %d times, want 4", calls.Load())
 	}
 }
+
+func TestRateLimitExemptPathsAreNeitherCountedNorRefused(t *testing.T) {
+	t.Parallel()
+
+	const denied = "192.0.2.50" // in SWWAF_DENY_NETS
+
+	s, _, server := startWithClock(t, "", map[string]string{
+		rateLimitPerMinute:   "1",
+		rateLimitExemptPaths: "/assets/,/favicon.ico",
+		denyNets:             denied,
+		deniedCountries:      "kp",
+	})
+
+	// The answers are kept before the requests, so that none waits for
+	// GeoJS.
+	server.GeoJS.Load([]lookup.Answer{
+		keptAnswer(client, "DE"), keptAnswer(fromKP, "KP"),
+	})
+
+	// With a limit of one request a minute, the requests for paths under a
+	// prefix are not counted, so client's first request for / is within
+	// the limit; and once client has reached it, they are not refused.
+	s.request(client, "/assets/app.js", http.StatusOK, requestlog.ActionForward)
+	s.request(client, "/favicon.ico?v=2", http.StatusOK, requestlog.ActionForward)
+	s.get(client, http.StatusOK, requestlog.ActionForward)
+
+	line := s.request(client, "/assets/app.js", http.StatusOK, requestlog.ActionForward)
+	if line.LimitHit != "" {
+		t.Errorf("log line has limit_hit %q, want none", line.LimitHit)
+	}
+
+	// A path outside every prefix is counted: /assets is not under
+	// /assets/, and breaks the limit.
+	s.request(client, "/assets", http.StatusForbidden, requestlog.ActionRateLimited)
+
+	// A ban, SWWAF_DENY_NETS and the country lists still refuse a path
+	// under a prefix.
+	s.request(client, "/assets/app.js", http.StatusForbidden, requestlog.ActionBanned)
+	s.request(denied, "/assets/app.js", http.StatusForbidden, requestlog.ActionDenied)
+	s.request(fromKP, "/assets/app.js",
+		http.StatusForbidden, requestlog.ActionCountryDenied)
+}
+
+func TestRateLimitCountsPathsThatAreNotExempt(t *testing.T) {
+	t.Parallel()
+
+	for _, sent := range []string{
+		// A prefix matches only at the start of the path.
+		"/static/assets/app.js",
+		// A prefix matches the path as sent: a router that matches the
+		// path as received does not take /%61ssets/x for a path under
+		// /assets/.
+		"/%61ssets/x",
+		// .. once percent-decoded: an app may act on these as /login, the
+		// last as a path under /sneak/app/ or as /assets/x.
+		"/assets/../login",
+		"/assets/%2e%2e/login",
+		"/assets/..%2Flogin",
+		"/assets/..;/login",
+		"/sneak/app/src/branch/main/..%2F..%2F..%2F..%2F..%2F..%2Fassets/x",
+		// An encoded slash or a backslash: Go's router takes /assets%2Fx
+		// for one path segment, not a path under /assets/.
+		"/assets%2Fx",
+		"/assets%2fx",
+		`/assets/x\y`,
+	} {
+		t.Run(sent, func(t *testing.T) {
+			t.Parallel()
+
+			s, _, _ := startWithClock(t, "", map[string]string{
+				rateLimitPerMinute:   "1",
+				rateLimitExemptPaths: "/assets/",
+			})
+
+			// Counted, the second request breaks the limit of one request
+			// a minute.
+			s.request(client, sent, http.StatusOK, requestlog.ActionForward)
+			s.request(client, sent, http.StatusForbidden, requestlog.ActionRateLimited)
+		})
+	}
+}

internal/proxy/request.go

@@ -7,7 +7,10 @@ import (
 	"net/http/httptrace"
 	"net/http/httputil"
 	"net/netip"
+	"net/url"
 	"os"
+	"slices"
+	"strings"
 	"sync"
 	"sync/atomic"
 	"time"
@@ -109,38 +112,24 @@ func (h *handler) newRequest(w http.ResponseWriter, r *http.Request) *request {
 
 // check is the one place where a request can be refused once its client
 // is known, before its body is read or anything reaches the app. It
-// returns nil to let the request through. A client in SWWAF_ALLOW_NETS
-// skips every check but the size limit. For any other client,
-// SWWAF_DENY_NETS comes first, then a ban on its netblock, so that a
-// client either refuses is not looked up, and then the country lists; a
-// request any of them refuses is not counted for the rate limits. Then
-// come the rate limits, unless the client is in
-// SWWAF_RATE_LIMIT_EXEMPT_NETS, so that every other request is counted,
-// one refused for its size too. Every refusal but the size limit's is
-// answered with SWWAF_BAN_RESPONSE. ctx is the request's own context.
+// returns nil to let the request through. The checks of checkClient come
+// first, answered with SWWAF_BAN_RESPONSE, and then the size limit, so
+// that a request the rate limits count is counted even when it is
+// refused for its size. In observe mode a request checkClient refuses
+// goes on to the size limit like any other. ctx is the request's own
+// context.
 func (rq *request) check(ctx context.Context) *refusal {
-	cfg := rq.h.config
-	allowed := isInside(rq.client, cfg.AllowNets)
-	exempt := isInside(rq.client, cfg.RateLimitExemptNets)
-	now := rq.h.now()
+	action := rq.checkClient(ctx)
+	if action != "" {
+		if !rq.h.config.Observe {
+			return rq.banResponse(action)
+		}
 
-	if !allowed && isInside(rq.client, cfg.DenyNets) {
-		return rq.banResponse(requestlog.ActionDenied)
+		// The log line names what enforce mode would have done.
+		rq.line.WouldAction = action
 	}
 
-	if !allowed && rq.banned(now) {
-		return rq.banResponse(requestlog.ActionBanned)
-	}
-
-	if !allowed && rq.countryDenied(ctx) {
-		return rq.banResponse(requestlog.ActionCountryDenied)
-	}
-
-	if !allowed && !exempt && rq.limitBroken(now) {
-		return rq.banResponse(requestlog.ActionRateLimited)
-	}
-
-	maxBytes := cfg.RequestMaxBytes
+	maxBytes := rq.h.config.RequestMaxBytes
 	if maxBytes > 0 && rq.in.ContentLength > maxBytes {
 		return &refusal{
 			status: http.StatusRequestEntityTooLarge,
@@ -152,6 +141,69 @@ func (rq *request) check(ctx context.Context) *refusal {
 	return nil
 }
 
+// checkClient runs the checks on the request's client, and returns the
+// action of the first that refuses the request, or "" when none does. A
+// client in SWWAF_ALLOW_NETS skips them. For any other client,
+// SWWAF_DENY_NETS comes first, then a ban on its netblock, so that a
+// client either refuses is not looked up, and then the country lists; a
+// request any of them refuses is not counted for the rate limits. Then
+// come the rate limits, unless the client is in
+// SWWAF_RATE_LIMIT_EXEMPT_NETS or the request's path is exempt under
+// SWWAF_RATE_LIMIT_EXEMPT_PATHS, so that every other request is counted.
+// ctx is the request's own context.
+func (rq *request) checkClient(ctx context.Context) string {
+	cfg := rq.h.config
+	if isInside(rq.client, cfg.AllowNets) {
+		return ""
+	}
+
+	now := rq.h.now()
+
+	if isInside(rq.client, cfg.DenyNets) {
+		return requestlog.ActionDenied
+	}
+
+	if rq.banned(now) {
+		return requestlog.ActionBanned
+	}
+
+	if rq.countryDenied(ctx) {
+		return requestlog.ActionCountryDenied
+	}
+
+	exempt := isInside(rq.client, cfg.RateLimitExemptNets) ||
+		pathExempt(rq.in.URL, cfg.RateLimitExemptPaths)
+	if !exempt && rq.limitBroken(now) {
+		return requestlog.ActionRateLimited
+	}
+
+	return ""
+}
+
+// pathExempt reports whether the rate limits leave out a request for u
+// because of SWWAF_RATE_LIMIT_EXEMPT_PATHS: whether its path as sent, the
+// path the app receives, not percent-decoded, starts with one of
+// prefixes, so that /%61ssets/x is not under /assets/ for an app whose
+// router matches the path as received. A request whose decoded path
+// contains .. anywhere or a backslash, or whose path as sent holds an
+// encoded slash (%2F or %2f), never is, since an app may act on it as a
+// path outside every prefix: /assets/..%2Flogin as /login, or /assets%2Fx
+// as one path segment, as Go's router does.
+func pathExempt(u *url.URL, prefixes []string) bool {
+	decoded := u.Path
+	// EscapedPath is the path as the app receives it, not decoded.
+	sent := u.EscapedPath()
+
+	if strings.Contains(decoded, "..") || strings.Contains(decoded, `\`) ||
+		strings.Contains(strings.ToLower(sent), "%2f") {
+		return false
+	}
+
+	return slices.ContainsFunc(prefixes, func(prefix string) bool {
+		return strings.HasPrefix(sent, prefix)
+	})
+}
+
 // forward passes the request to the app and the app's answer back. ctx
 // is the request's own context.
 func (rq *request) forward(ctx context.Context) {

internal/requestlog/requestlog.go

@@ -67,6 +67,10 @@ type Line struct {
 	Referer        string `json:"referer"`
 	UserAgent      string `json:"user_agent"`
 	Action         string `json:"action"`
+	// WouldAction is, in observe mode, the action enforce mode would have
+	// taken with a request it would have refused: ActionDenied,
+	// ActionBanned, ActionCountryDenied or ActionRateLimited.
+	WouldAction string `json:"would_action,omitempty"`
 	// LimitHit is the window whose rate limit the request went over:
 	// minute, hour or day.
 	LimitHit string `json:"limit_hit,omitempty"`

internal/smallwebwaf/smallwebwaf_test.go

@@ -418,6 +418,7 @@ func wantStartingLine(t *testing.T, line map[string]any, appURL, dir string) {
 		listenAddr:                              localhost + ":0",
 		upstreamURL:                             appURL,
 		stateDir:                                dir,
+		"SWWAF_MODE":                            "enforce",
 		stateWriteDelay:                         "10s",
 		stateCounterInterval:                    "15m",
 		trustedProxies:                          "10.0.0.0/8,172.16.0.0/12,192.168.0.0/16",
@@ -435,6 +436,7 @@ func wantStartingLine(t *testing.T, line map[string]any, appURL, dir string) {
 		"SWWAF_RATE_LIMIT_PER_MINUTE":           "1000",
 		"SWWAF_RATE_LIMIT_PER_HOUR":             "10000",
 		rateLimitPerDay:                         "50000",
+		"SWWAF_RATE_LIMIT_EXEMPT_PATHS":         "",
 		"SWWAF_DENIED_COUNTRIES":                "",
 		"SWWAF_EXCLUSIVELY_ALLOWED_COUNTRIES":   "",
 		"SWWAF_BAN_RESPONSE":                    "403",

internal/state/state.go

@@ -222,7 +222,7 @@ func (f *Files) Watch(ctx context.Context) {
 		case event := <-watcher.Events:
 			switch name := filepath.Base(event.Name); name {
 			case bansJSON, clientsJSON, lookupsJSON:
-				f.takeInEdit(name)
+				f.fileChanged(name)
 			}
 		case err = <-watcher.Errors:
 			f.params.ProcessLog.Warn("watching the state files failed",
@@ -239,10 +239,11 @@ func (f *Files) logFailure(err error) {
 	}
 }
 
-// takeInEdit takes in an edit of the state file name and logs it, if the
-// file has changed since smallwebwaf last read or wrote it and parses. A
-// file that cannot be read or does not parse is left for its next write.
-func (f *Files) takeInEdit(name string) {
+// fileChanged takes in what the state file name holds, as Watch sees it
+// change, if that is an edit made since smallwebwaf last read or wrote
+// the file. A file that cannot be read or does not parse is left for its
+// next write.
+func (f *Files) fileChanged(name string) {
 	f.mu.Lock()
 	defer f.mu.Unlock()
 
@@ -251,9 +252,17 @@ func (f *Files) takeInEdit(name string) {
 		return
 	}
 
-	_, err = f.takeIn(name, data)
+	_ = f.takeInEdit(name, data)
+}
+
+// takeInEdit takes in data, an edit of the state file name, as takeIn
+// does, and counts and logs it. Every edit taken in while smallwebwaf
+// runs, by Watch or by a write, is taken in here. An edit that does not
+// parse is neither counted nor logged, and takeIn's error returned.
+func (f *Files) takeInEdit(name string, data []byte) error {
+	_, err := f.takeIn(name, data)
 	if err != nil {
-		return
+		return err
 	}
 
 	// Counted before it is logged, so that the count is there once the
@@ -261,6 +270,8 @@ func (f *Files) takeInEdit(name string) {
 	f.params.Metrics.StateFileEditTakenIn(name)
 	f.params.ProcessLog.Info("took in an edit of a state file",
 		"file", filepath.Join(f.params.Dir, name))
+
+	return nil
 }
 
 // read takes in the state file name at start, and returns how many
@@ -347,44 +358,33 @@ func (f *Files) takeIn(name string, data []byte) (int, error) {
 
 // writeFile writes the state file name from what smallwebwaf holds. An
 // edit made since smallwebwaf last read or wrote the file is taken in
-// first, so that it is not overwritten. An edit that does not parse is
-// renamed to name.bad, for the admin to mend, and logged with where in
-// the file the error is.
+// first, so that it is not overwritten, or set aside if it does not
+// parse. A file that cannot be read, or an edit that cannot be set
+// aside, is left as it is, and the write given up. Every write is counted
+// in the metrics, and one that fails or is given up as a failure.
 func (f *Files) writeFile(name string) error {
 	f.mu.Lock()
 	defer f.mu.Unlock()
 
-	path := filepath.Join(f.params.Dir, name)
-
 	data, changed, err := f.readChanged(name)
-	if err != nil {
-		return err
-	}
-
-	if changed {
-		_, err = f.takeIn(name, data)
-		if err == nil {
-			f.params.Metrics.StateFileEditTakenIn(name)
+	if err == nil && changed {
+		err = f.takeInEdit(name, data)
+		if err != nil {
+			err = f.setAside(name, err)
 		}
 	}
 
-	if err != nil {
-		renameErr := os.Rename(path, path+".bad")
-		if renameErr != nil {
-			return errors.Join(err, renameErr)
+	if err == nil {
+		data, err = f.encode(name)
+		if err != nil {
+			err = fmt.Errorf("encode %s: %w", name, err)
 		}
-
-		f.params.ProcessLog.Error("set aside an edit of a state file that does not parse",
-			"file", path+".bad", "error", err.Error())
-		f.params.Metrics.StateFileEditSetAside(name)
 	}
 
-	data, err = f.encode(name)
-	if err != nil {
-		return fmt.Errorf("encode %s: %w", name, err)
+	if err == nil {
+		err = write(f.params.Dir, name, data)
 	}
 
-	err = write(f.params.Dir, name, data)
 	if err == nil {
 		// The file holds data from here on, even if the directory sync
 		// fails, so that its next read does not take it for an admin's
@@ -398,6 +398,25 @@ func (f *Files) writeFile(name string) error {
 	return err
 }
 
+// setAside renames the state file name, an edit that does not parse with
+// parseErr, to name.bad, for the admin to mend, and logs it with where in
+// the file the error is. If the rename fails, the edit is left as it is,
+// and the error returned is parseErr joined with the rename's.
+func (f *Files) setAside(name string, parseErr error) error {
+	path := filepath.Join(f.params.Dir, name)
+
+	err := os.Rename(path, path+".bad")
+	if err != nil {
+		return errors.Join(parseErr, err)
+	}
+
+	f.params.ProcessLog.Error("set aside an edit of a state file that does not parse",
+		"file", path+".bad", "error", parseErr.Error())
+	f.params.Metrics.StateFileEditSetAside(name)
+
+	return nil
+}
+
 // encode returns the state file name as smallwebwaf writes it, from a
 // snapshot of the part that keeps that state.
 func (f *Files) encode(name string) ([]byte, error) {

internal/state/state_test.go

@@ -5,6 +5,7 @@ import (
 	"encoding/json"
 	"io/fs"
 	"log/slog"
+	"maps"
 	"net"
 	"net/http"
 	"net/http/httptest"
@@ -511,7 +512,8 @@ func TestFileThatCannotBeReadIsNotWrittenOver(t *testing.T) {
 
 	dir := t.TempDir()
 	path := filepath.Join(dir, bansJSON)
-	files := load(t, newParams(dir))
+	params := newParams(dir)
+	files := load(t, params)
 
 	// bans.json is a socket, which cannot be opened as a file, even by
 	// root, as the tests run in Docker, but which a rename could replace.
@@ -536,6 +538,38 @@ func TestFileThatCannotBeReadIsNotWrittenOver(t *testing.T) {
 	}
 
 	wantFiles(t, dir, bansJSON, clientsJSON, lookupsJSON)
+	wantWriteFailed(t, params, bansJSON)
+}
+
+func TestBrokenEditThatCannotBeSetAsideIsNotWrittenOver(t *testing.T) {
+	t.Parallel()
+
+	const broken = `{"version": 1, "bans": [`
+
+	dir := t.TempDir()
+	path := filepath.Join(dir, bansJSON)
+	params := newParams(dir)
+	files := load(t, params)
+
+	// A directory named bans.json.bad cannot be renamed over, so the
+	// broken edit cannot be set aside, and is left as it is.
+	edit(t, dir, bansJSON, broken)
+
+	err := os.Mkdir(path+".bad", 0o700)
+	if err != nil {
+		t.Fatalf("mkdir: %v", err)
+	}
+
+	err = files.WriteAll()
+	if err == nil {
+		t.Error("writing with bans.json.bad in the way did not fail")
+	}
+
+	if got := readFile(t, path); got != broken {
+		t.Errorf("bans.json holds\n%s\nwant the edit", got)
+	}
+
+	wantWriteFailed(t, params, bansJSON)
 }
 
 func TestEditOfEachFileTakenIn(t *testing.T) {
@@ -598,6 +632,39 @@ func TestOwnWritesAreNotTakenIn(t *testing.T) {
 	wantTakenIn(t, lines, dir, lookupsJSON)
 }
 
+func TestFileRenamedOverAStateFileTakenIn(t *testing.T) {
+	t.Parallel()
+
+	dir := t.TempDir()
+	path := filepath.Join(dir, bansJSON)
+	params := newParams(dir)
+	lines := logInto(&params)
+	files := load(t, params)
+
+	err := files.WriteAll()
+	if err != nil {
+		t.Fatalf("write: %v", err)
+	}
+
+	watch(t, files, lines)
+
+	// The admin mends bans.json.bad and moves it back, as editors that
+	// save by renaming do with a file of their own: nothing is written
+	// into bans.json itself. An edit of clients.json after it must be
+	// taken in second.
+	edit(t, dir, bansJSON+".bad", permanentBansJSON)
+
+	err = os.Rename(path+".bad", path)
+	if err != nil {
+		t.Fatalf("rename: %v", err)
+	}
+
+	edit(t, dir, clientsJSON, `{"version": 1, "clients": []}`)
+
+	wantTakenIn(t, lines, dir, bansJSON)
+	wantEqual(t, bansJSON, params.Ledger.Snapshot(), []bans.Ban{permanentBan()})
+}
+
 func TestBanAddedAndLiftedThroughBansJSON(t *testing.T) {
 	t.Parallel()
 
@@ -708,6 +775,53 @@ func TestEditsTakenInAreCountedInTheMetrics(t *testing.T) {
 		`smallwebwaf_state_file_edits_taken_in_total{file="bans.json"}`, 2)
 }
 
+func TestEditTakenInByAWriteIsLoggedAsWatchLogsIt(t *testing.T) {
+	t.Parallel()
+
+	dir := t.TempDir()
+	params := newParams(dir)
+	lines := logInto(&params)
+	files := load(t, params)
+
+	// An edit taken in by Watch, which is then stopped.
+	ctx, stop := context.WithCancel(t.Context())
+	stopped := make(chan struct{})
+
+	go func() {
+		files.Watch(ctx)
+		close(stopped)
+	}()
+
+	lines.waitFor(t, watching)
+	edit(t, dir, bansJSON, `{"version": 1, "bans": []}`)
+	byWatch := lines.waitFor(t, tookIn)
+
+	stop()
+	<-stopped
+
+	// An edit taken in by the write of its file. Nothing logs after the
+	// write, so the log is closed, and a write that does not log the edit
+	// fails the test at once instead of waiting for the line.
+	edit(t, dir, bansJSON, permanentBansJSON)
+
+	err := files.WriteAll()
+	if err != nil {
+		t.Fatalf("write: %v", err)
+	}
+
+	close(lines)
+
+	byWrite := lines.waitFor(t, tookIn)
+
+	// The two lines differ only in their time.
+	delete(byWatch, "time")
+	delete(byWrite, "time")
+
+	if !maps.Equal(byWrite, byWatch) {
+		t.Errorf("the write logged %v, where Watch logged %v", byWrite, byWatch)
+	}
+}
+
 func TestEditsSetAsideAreCountedInTheMetrics(t *testing.T) {
 	t.Parallel()
 
@@ -891,8 +1005,9 @@ func (l processLog) Write(line []byte) (int, error) {
 }
 
 // waitFor returns the next line of the process log whose message is msg,
-// passing over the lines before it. It waits as long as that takes, so
-// that a slow test process cannot fail the test.
+// passing over the lines before it, or nil if the log is closed first. It
+// waits as long as that takes, so that a slow test process cannot fail
+// the test.
 func (l processLog) waitFor(t *testing.T, msg string) map[string]any {
 	t.Helper()
 
@@ -909,7 +1024,7 @@ func (l processLog) waitFor(t *testing.T, msg string) map[string]any {
 		}
 	}
 
-	return nil // never reached: nothing closes the log
+	return nil
 }
 
 // wantTakenIn waits for the next edit taken in, and checks that it is of
@@ -1078,6 +1193,18 @@ func metric(t *testing.T, text, series string) float64 {
 	return 0
 }
 
+// wantWriteFailed checks that the metrics of params count one write of the
+// state file name, and that it failed.
+func wantWriteFailed(t *testing.T, params state.Params, name string) {
+	t.Helper()
+
+	got := scrape(t, params)
+	file := `{file="` + name + `"}`
+
+	wantMetric(t, got, "smallwebwaf_state_file_writes_total"+file, 1)
+	wantMetric(t, got, "smallwebwaf_state_file_write_failures_total"+file, 1)
+}
+
 // wantMetric checks the value of series in text, the metrics, as metric
 // reads it.
 func wantMetric(t *testing.T, text, series string, want float64) {