Compare commits
1
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
098bbe8878 |
No files matched your search
@@ -599,6 +599,11 @@ request reaches the app.
|
||||
16. A path under `/_sww/` is answered `404`, so that the app never sees that
|
||||
prefix.
|
||||
|
||||
Steps 12 to 14 only read the request. While the lookup waits for GeoJS, they run
|
||||
at the same time as steps 6 to 11. The first of the two sides to refuse the
|
||||
request ends the checks, and `waf` answers at once. Steps 12 to 14 have 500 ms
|
||||
together.
|
||||
|
||||
`rproxy` takes the rest, with `waf` counting at the end:
|
||||
|
||||
17. The request is passed to the app, streaming.
|
||||
@@ -611,9 +616,10 @@ A refused visitor is answered with `SWW_WAF_BAN_RESPONSE`. A `block` rule or a
|
||||
Core Rule Set match answers `403` and bans nobody, since the Core Rule Set has
|
||||
false positives; such refusals count toward `SWW_WAF_REFUSALS_LIMIT_PER_MINUTE`
|
||||
instead. A request over the header limits of step 3 is answered `414` for a
|
||||
request line over 8 KiB and `431` otherwise. It bans nobody either. `rproxy`
|
||||
itself answers `405` to a method it does not pass on, and the status for a size
|
||||
or time limit, such as `413` for a body too large.
|
||||
request line over 8 KiB and `431` otherwise. It bans nobody either. Nor does a
|
||||
request that steps 12 to 14 have not finished within 500 ms, which is answered
|
||||
`503`. `rproxy` itself answers `405` to a method it does not pass on, and the
|
||||
status for a size or time limit, such as `413` for a body too large.
|
||||
|
||||
### Clients and bans
|
||||
|
||||
|
||||
@@ -173,7 +173,8 @@ field gives that name. `refusal_limit` and `byte_limits` give theirs in
|
||||
`country=` tag.
|
||||
|
||||
Otherwise the request goes on at once. A request that ends before the answer
|
||||
comes is logged without it.
|
||||
comes is logged without it. While the request waits, steps 12 to 14 run at
|
||||
the same time as steps 6 to 11 (see "Answering in time").
|
||||
|
||||
7. The country list (`countries`; see "Country list").
|
||||
8. The reputation sources (see "Reputation"), in this order:
|
||||
@@ -2379,8 +2380,9 @@ runs no check, writes no request log line and counts nothing in the metrics.
|
||||
|
||||
### The answer
|
||||
|
||||
`waf` runs steps 3 to 16 of "What happens to a request", in that order. Every
|
||||
answer but the last below carries `X-Sww-Disposition`.
|
||||
`waf` runs steps 3 to 16 of "What happens to a request", in that order, apart
|
||||
from the steps that run at the same time (see "Answering in time"). Every answer
|
||||
but the last below carries `X-Sww-Disposition`.
|
||||
|
||||
- `X-Sww-Disposition: allow`, status `200`, no body. `rproxy` passes the request
|
||||
to the app. While `SWW_WAF_ADD_LOOKUP_HEADERS` is `true`, the answer also
|
||||
@@ -2392,6 +2394,8 @@ answer but the last below carries `X-Sww-Disposition`.
|
||||
- `SWW_WAF_BAN_RESPONSE`, for a refusal;
|
||||
- `403`, for a `block` rule or the Core Rule Set;
|
||||
- `414` or `431`, for a request over the header limits;
|
||||
- `503`, for a request that steps 12 to 14 have not finished within 500
|
||||
milliseconds;
|
||||
- `404`, for a path under `/_sww/` that passes the checks, so that the app
|
||||
never sees that prefix.
|
||||
- `X-Sww-Disposition: close`, status `200`, no body. This takes the place of a
|
||||
@@ -2412,9 +2416,15 @@ The Core Rule Set inspects the body a question carries only while
|
||||
|
||||
`rproxy` stops waiting for `waf` after `SWW_RPROXY_WAF_BACKEND_TIMEOUT_MS`, by
|
||||
default 2 seconds, and `SWW_RPROXY_WAF_BACKEND_NO_ANSWER_DISPOSITION` then
|
||||
decides. The longest a question waits on an outside source is
|
||||
`SWW_WAF_ASN_AND_COUNTRY_LOOKUP_TIMEOUT_MS`, by default 800 milliseconds, well
|
||||
within that time.
|
||||
decides. Two things in a decision take time:
|
||||
|
||||
- the wait for GeoJS, at most `SWW_WAF_ASN_AND_COUNTRY_LOOKUP_TIMEOUT_MS`, by
|
||||
default 800 milliseconds;
|
||||
- the trap paths, the rule files and the Core Rule Set, which have 500
|
||||
milliseconds.
|
||||
|
||||
They run at the same time, so `waf` answers within the longer of the two, well
|
||||
within `rproxy`'s wait.
|
||||
|
||||
GeoJS is the only outside source a question can wait on. A question waits only
|
||||
when step 6 says so, and for at most
|
||||
@@ -2446,10 +2456,38 @@ never wait. These can still wait for GeoJS:
|
||||
- a country list, which needs the answer;
|
||||
- a rate limit a percentage has lowered, which needs the answer;
|
||||
- a DNS blocklist or AbuseIPDB, while `SWW_WAF_DNSBL_ABUSEIPDB_ACTION` is
|
||||
`deny`;
|
||||
- a trap path;
|
||||
- a rule file;
|
||||
- the Core Rule Set.
|
||||
`deny`.
|
||||
|
||||
The trap paths, the rule files and the Core Rule Set, steps 12 to 14, run during
|
||||
that wait. Once a request has passed the bans, the lookup starts. While it waits
|
||||
for GeoJS, steps 12 to 14 run at the same time as steps 6 to 11. A lookup that
|
||||
needs no wait has its answer at once, and steps 7 to 11 then run before steps 12
|
||||
to 14 start.
|
||||
|
||||
- The first of the two sides to refuse the request ends the checks. `waf`
|
||||
answers at once, and does not wait for the other side. The lookup goes on, and
|
||||
its answer is kept.
|
||||
- What the other side finds leaves no trace. It counts nothing, bans nothing,
|
||||
raises no alert, counts no metric and adds nothing to the log line.
|
||||
- When both sides would refuse, the one that finishes first is answered and
|
||||
logged.
|
||||
- When neither refuses, step 15 runs.
|
||||
|
||||
Steps 12 to 14 only read the request. They have 500 milliseconds together, from
|
||||
their start. The limit is fixed, with no setting. A request they have not
|
||||
finished by then is refused:
|
||||
|
||||
- the answer is `deny`, with `503`;
|
||||
- the log line's action is `checks_timed_out`, and its `heuristic` names the
|
||||
step that had not finished;
|
||||
- the refusal makes no ban, is no offence and does not count toward the refusal
|
||||
limit;
|
||||
- it holds in every `SWW_WAF_CORE_RULE_SET_MODE`;
|
||||
- in `observe` mode the request is answered `allow` with `would_action`, like
|
||||
every refusal.
|
||||
|
||||
The limit is measured on the clock. While `waf`'s host is so busy that an
|
||||
ordinary request takes 500 milliseconds, that request is refused too.
|
||||
|
||||
The header limits are fixed, with no setting. A line's length is in bytes and
|
||||
does not count its line ending. `waf` refuses a request whose request line is
|
||||
@@ -2499,9 +2537,11 @@ body can add up to more than the body. From 1 KiB of JSON, Coraza makes at most
|
||||
|
||||
A request can reach all of these at once. At paranoia level 4, with no rule
|
||||
switched off, the Core Rule Set takes under a second on one core of a current
|
||||
server to inspect it. So `waf` answers it within the default 2 seconds, even
|
||||
after the default 800 milliseconds of waiting for GeoJS. Enough such requests at
|
||||
once can still overload `waf` past `SWW_RPROXY_WAF_BACKEND_TIMEOUT_MS`.
|
||||
server to inspect it. `waf` refuses such a request after 500 milliseconds.
|
||||
Coraza cannot stop an inspection part-way, so the inspection runs to its end
|
||||
with nobody waiting for it. These limits keep one request from holding a
|
||||
processor for more than about a second. Enough such requests at once can still
|
||||
overload `waf` past `SWW_RPROXY_WAF_BACKEND_TIMEOUT_MS`.
|
||||
|
||||
### The report
|
||||
|
||||
|
||||
@@ -105,6 +105,8 @@ Every one of these lines has:
|
||||
`blocked_by_core_rule_set`: refused by that check;
|
||||
- `request_line_too_long`: a request line over the header limits;
|
||||
- `request_header_too_large`: headers over the header limits;
|
||||
- `checks_timed_out`: refused with `503` because the trap paths, the rule
|
||||
files and the Core Rule Set had not finished within 500 ms;
|
||||
- `own_endpoint`: answered at one of `waf`'s own endpoints, or `404` for a
|
||||
path under `/_sww/`.
|
||||
- `would_action`: in `observe` mode, what `enforce` mode would have done with a
|
||||
@@ -134,6 +136,11 @@ Every one of these lines has:
|
||||
`SWW_WAF_CORE_RULE_SET_SCORE_THRESHOLD`;
|
||||
- `refusal_limit`: `window`, `count` and `limit`, of the refusal limit;
|
||||
- `byte_limits`: `window`, `count` and `limit`, the count in bytes.
|
||||
|
||||
For a request refused as `checks_timed_out`, `heuristic` is the one of
|
||||
`trap_paths`, `rule_files` and `core_rule_set` that had not finished, and
|
||||
its figure is `time_limit_ms`, 500.
|
||||
|
||||
- `after_answer_heuristic`: the heuristic that banned the client after `waf`
|
||||
answered the request, in `observe` mode too: `refusal_limit` or `byte_limits`.
|
||||
- `after_answer_heuristic_figures`: an object of the figures that heuristic
|
||||
|
||||
+3
-1
@@ -435,7 +435,9 @@ A few limits are fixed rather than settings:
|
||||
|
||||
`SWW_WAF_ASN_AND_COUNTRY_LOOKUP_TIMEOUT_MS` is the longest a question waits on
|
||||
an outside source. Keep it well under `rproxy`'s
|
||||
`SWW_RPROXY_WAF_BACKEND_TIMEOUT_MS`, as the default is. No question waits for
|
||||
`SWW_RPROXY_WAF_BACKEND_TIMEOUT_MS`, as the default is. The trap paths, the rule
|
||||
files and the Core Rule Set run during that wait and have 500 ms, so `waf`
|
||||
answers within the longer of the two times. No question waits for
|
||||
`SWW_WAF_DNSBL_ABUSEIPDB_TIMEOUT_MS`, since the DNS blocklists and AbuseIPDB are
|
||||
asked in the background (see "Answering in time" under "`waf`" in
|
||||
[`SPEC.md`](../SPEC.md)).
|
||||
|
||||
@@ -35,10 +35,11 @@ const (
|
||||
mostJSONRead = 1 << 10
|
||||
)
|
||||
|
||||
// The Core Rule Set has 1 second for one request. That is the default of
|
||||
// rproxy's SWW_RPROXY_WAF_BACKEND_TIMEOUT_MS, 2 seconds, less the default of
|
||||
// SWW_WAF_ASN_AND_COUNTRY_LOOKUP_TIMEOUT_MS, 800 milliseconds, and less 200
|
||||
// milliseconds for waf's other checks.
|
||||
// waf refuses a request the Core Rule Set has not inspected within 500
|
||||
// milliseconds, but Coraza cannot stop an inspection part-way, so it runs to
|
||||
// its end with nobody waiting for it. The test checks that an inspection ends
|
||||
// within about 1 second, so that a visitor cannot keep a processor busy for
|
||||
// longer with one request.
|
||||
//
|
||||
// A busy host slows all work, so the test does not compare an inspection with
|
||||
// 1 second. It compares it with referenceTime, timed on the same thread just
|
||||
|
||||
@@ -61,6 +61,12 @@ const (
|
||||
// ActionCountryDenied is a request refused because its client's country
|
||||
// is not in SWW_WAF_EXCLUSIVELY_ALLOWED_COUNTRIES.
|
||||
ActionCountryDenied = "country_denied"
|
||||
|
||||
// ActionChecksTimedOut is a request refused because the trap paths, the
|
||||
// rule files and the Core Rule Set had not finished with it within 500
|
||||
// milliseconds.
|
||||
ActionChecksTimedOut = "checks_timed_out"
|
||||
|
||||
// ActionOwnEndpoint is a request waf answered at one of its own
|
||||
// endpoints, or a question about a path under /_sww/.
|
||||
ActionOwnEndpoint = "own_endpoint"
|
||||
|
||||
@@ -19,6 +19,8 @@ const (
|
||||
alertMaxPerHour = "SWW_WAF_ALERT_MAX_PER_HOUR"
|
||||
// alertInstance is the instance every alert of these tests gives.
|
||||
alertInstance = "host1/blog"
|
||||
// alertWebhook is where the tests send alerts.
|
||||
alertWebhook = "https://alerts.example/sww"
|
||||
)
|
||||
|
||||
func TestBanForABrokenLimitRaisesABanAlertWithItsNotes(t *testing.T) {
|
||||
@@ -270,7 +272,7 @@ func startAppWithAlerts(
|
||||
clk := &clock{now: time.Date(2026, 10, 6, 0, 0, 0, 0, time.UTC)}
|
||||
settings := map[string]string{
|
||||
trustedProxies: trustLocalhost,
|
||||
alertWebhookURL: "https://alerts.example/sww",
|
||||
alertWebhookURL: alertWebhook,
|
||||
instanceName: alertInstance,
|
||||
}
|
||||
maps.Copy(settings, env)
|
||||
|
||||
@@ -0,0 +1,172 @@
|
||||
package waf
|
||||
|
||||
// The heuristics of besideLookup, and how they run beside the lookup.
|
||||
|
||||
import (
|
||||
"context"
|
||||
"sync/atomic"
|
||||
"time"
|
||||
|
||||
"sneak.berlin/go/smallwebwaf/internal/config"
|
||||
"sneak.berlin/go/smallwebwaf/internal/coreruleset"
|
||||
"sneak.berlin/go/smallwebwaf/internal/requestlog"
|
||||
"sneak.berlin/go/smallwebwaf/internal/rules"
|
||||
)
|
||||
|
||||
// besideLookupTimeLimit is how long the heuristics of besideLookup have for a
|
||||
// request, from when they start. They only read the request, and no request
|
||||
// should take them that long. A request they have not finished by then is
|
||||
// refused with 503, as ActionChecksTimedOut.
|
||||
const besideLookupTimeLimit = 500 * time.Millisecond
|
||||
|
||||
// besideLookupFindings is what the heuristics of besideLookup found of a
|
||||
// request, before any of it is applied to the request.
|
||||
type besideLookupFindings struct {
|
||||
// trapPath is the request's path when it is in SWW_WAF_TRAP_PATHS, and ""
|
||||
// otherwise.
|
||||
trapPath string
|
||||
// rules are the rules of the rule files the request matched.
|
||||
rules []rules.Rule
|
||||
// inspected is true when the Core Rule Set inspected the request.
|
||||
// inspection, read and inspectionTime are then what it found, what it read
|
||||
// of the body, and how long it took.
|
||||
inspected bool
|
||||
inspection coreruleset.Result
|
||||
read []byte
|
||||
inspectionTime time.Duration
|
||||
// refuses is true when one of the heuristics refuses the request, or would
|
||||
// in observe mode. Those after it did no work.
|
||||
refuses bool
|
||||
}
|
||||
|
||||
// runBesideLookup runs the heuristics of besideLookup while the lookup waits
|
||||
// for GeoJS, and returns the first verdict, or nil when none gives one.
|
||||
// lookedUp is the channel the lookup's result comes on, or nil once afterLookup
|
||||
// has let the request through.
|
||||
//
|
||||
// The first of the two to refuse the request ends the checks: waf answers at
|
||||
// once, and what the other found is dropped. besideLookup has
|
||||
// besideLookupTimeLimit; past it, the request is refused as
|
||||
// ActionChecksTimedOut. Once both have let the request through, the heuristics
|
||||
// of last run. ctx is the request's own context.
|
||||
func (rq *request) runBesideLookup(
|
||||
ctx context.Context, lookedUp <-chan lookupResult,
|
||||
) *verdict {
|
||||
var running atomic.Pointer[string]
|
||||
|
||||
running.Store(new(trapPaths{}.name()))
|
||||
|
||||
found := make(chan *besideLookupFindings, 1)
|
||||
|
||||
go func() { found <- rq.findBesideLookup(&running) }()
|
||||
|
||||
timeUp := time.NewTimer(rq.h.besideLookupLimit)
|
||||
defer timeUp.Stop()
|
||||
|
||||
var findings *besideLookupFindings
|
||||
|
||||
for findings == nil || lookedUp != nil {
|
||||
select {
|
||||
case result := <-lookedUp:
|
||||
lookedUp = nil
|
||||
|
||||
v := rq.afterLookUp(ctx, result)
|
||||
if v != nil {
|
||||
return v
|
||||
}
|
||||
case findings = <-found:
|
||||
found = nil
|
||||
|
||||
timeUp.Stop()
|
||||
|
||||
if findings.refuses {
|
||||
return rq.applyFindings(ctx, findings)
|
||||
}
|
||||
case <-timeUp.C:
|
||||
return rq.timedOut(*running.Load())
|
||||
}
|
||||
}
|
||||
|
||||
v := rq.applyFindings(ctx, findings)
|
||||
if v != nil {
|
||||
return v
|
||||
}
|
||||
|
||||
return rq.runHeuristics(ctx, last(), rq.h.now())
|
||||
}
|
||||
|
||||
// findBesideLookup does the work of the heuristics of besideLookup on the
|
||||
// request, in their order, until one refuses it, and returns what they found.
|
||||
// It changes nothing of the request or of what waf keeps, so that it can run
|
||||
// while the lookup and afterLookup do. running holds the name of the heuristic
|
||||
// at work, the first one's when findBesideLookup is called.
|
||||
func (rq *request) findBesideLookup(
|
||||
running *atomic.Pointer[string],
|
||||
) *besideLookupFindings {
|
||||
findings := &besideLookupFindings{}
|
||||
|
||||
findings.trapPath = rq.trapPath()
|
||||
if findings.trapPath != "" {
|
||||
findings.refuses = true
|
||||
|
||||
return findings
|
||||
}
|
||||
|
||||
running.Store(new(ruleFiles{}.name()))
|
||||
|
||||
findings.rules = rq.h.rules.Match(rq.in)
|
||||
if refusedByRule(findings.rules) {
|
||||
findings.refuses = true
|
||||
|
||||
return findings
|
||||
}
|
||||
|
||||
running.Store(new(coreRuleSet{}.name()))
|
||||
|
||||
cfg := rq.h.config
|
||||
if cfg.CoreRuleSetMode == config.CoreRuleSetModeOff ||
|
||||
pathExempt(rq.in, cfg.CoreRuleSetExemptPaths) {
|
||||
return findings
|
||||
}
|
||||
|
||||
start := time.Now()
|
||||
// A question's body has been read into memory, so that reading it here
|
||||
// cannot fail.
|
||||
findings.inspection, findings.read, _ = rq.h.coreRuleSet.Inspect(
|
||||
rq.in, rq.client, rq.in.Body,
|
||||
)
|
||||
findings.inspectionTime = time.Since(start)
|
||||
findings.inspected = true
|
||||
findings.refuses = rq.refusedByCoreRuleSet(findings.inspection)
|
||||
|
||||
return findings
|
||||
}
|
||||
|
||||
// applyFindings applies findings, what findBesideLookup found, to the
|
||||
// request: it runs the heuristics of besideLookup, which read them, at the
|
||||
// time it is called, and returns the first verdict, or nil when none gives
|
||||
// one.
|
||||
func (rq *request) applyFindings(
|
||||
ctx context.Context, findings *besideLookupFindings,
|
||||
) *verdict {
|
||||
rq.findings = findings
|
||||
|
||||
return rq.runHeuristics(ctx, besideLookup(), rq.h.now())
|
||||
}
|
||||
|
||||
// timedOut refuses the request for the heuristics of besideLookup not having
|
||||
// finished with it within besideLookupTimeLimit. running is the name of the
|
||||
// one at work, which the log line's heuristic gives. The refusal makes no ban
|
||||
// and is no offence.
|
||||
func (rq *request) timedOut(running string) *verdict {
|
||||
v := &verdict{
|
||||
action: requestlog.ActionChecksTimedOut,
|
||||
figures: map[string]any{
|
||||
"time_limit_ms": rq.h.besideLookupLimit.Milliseconds(),
|
||||
},
|
||||
}
|
||||
rq.line.Heuristic = running
|
||||
rq.line.HeuristicFigures = v.figures
|
||||
|
||||
return v
|
||||
}
|
||||
@@ -0,0 +1,426 @@
|
||||
package waf_test
|
||||
|
||||
import (
|
||||
"io"
|
||||
"maps"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"net/netip"
|
||||
"strconv"
|
||||
"strings"
|
||||
"sync/atomic"
|
||||
"testing"
|
||||
"testing/synctest"
|
||||
"time"
|
||||
|
||||
"github.com/prometheus/client_golang/prometheus/testutil"
|
||||
|
||||
"sneak.berlin/go/smallwebwaf/internal/alerts"
|
||||
"sneak.berlin/go/smallwebwaf/internal/coreruleset"
|
||||
"sneak.berlin/go/smallwebwaf/internal/ratelimit"
|
||||
"sneak.berlin/go/smallwebwaf/internal/requestlog"
|
||||
"sneak.berlin/go/smallwebwaf/internal/waf"
|
||||
)
|
||||
|
||||
// checksTimeLimit is how long the trap paths, the rule files and the Core Rule
|
||||
// Set have for a request, as SPEC.md gives it, and checksTimedOut the status of
|
||||
// a request they have not finished by then.
|
||||
const (
|
||||
checksTimeLimit = 500 * time.Millisecond
|
||||
checksTimedOut = http.StatusServiceUnavailable
|
||||
)
|
||||
|
||||
// heldCoreRuleSet is a stand-in for the Core Rule Set: each inspection takes
|
||||
// took, then finds result. asked counts the inspections.
|
||||
type heldCoreRuleSet struct {
|
||||
took time.Duration
|
||||
result coreruleset.Result
|
||||
asked atomic.Int64
|
||||
}
|
||||
|
||||
// Inspect inspects a request as the stand-in does, and reads all of body.
|
||||
func (c *heldCoreRuleSet) Inspect(
|
||||
_ *http.Request, _ netip.Addr, body io.Reader,
|
||||
) (coreruleset.Result, []byte, error) {
|
||||
c.asked.Add(1)
|
||||
time.Sleep(c.took)
|
||||
|
||||
read, err := io.ReadAll(body)
|
||||
|
||||
return c.result, read, err
|
||||
}
|
||||
|
||||
// coreRuleSetMatch returns what the stand-in finds of an attack: rule 942100,
|
||||
// with a score of 5, the default SWW_WAF_CORE_RULE_SET_SCORE_THRESHOLD.
|
||||
func coreRuleSetMatch() coreruleset.Result {
|
||||
return coreruleset.Result{RuleIDs: []int{942100}, Score: 5}
|
||||
}
|
||||
|
||||
// newWAFHoldingTheCoreRuleSet makes a waf, to be asked in a synctest bubble,
|
||||
// with crs in place of the Core Rule Set, in block mode. GeoJS never answers,
|
||||
// a setting needs its answer, alerts go to a webhook, and everything else is
|
||||
// at its default but for env. No request to it may use the network.
|
||||
func newWAFHoldingTheCoreRuleSet(
|
||||
t *testing.T, crs *heldCoreRuleSet, env map[string]string,
|
||||
) (*waf.Server, *output, *alerts.Queue) {
|
||||
t.Helper()
|
||||
|
||||
settings := map[string]string{
|
||||
tagDeny: denyKP, coreRuleSetMode: block,
|
||||
alertWebhookURL: alertWebhook,
|
||||
}
|
||||
maps.Copy(settings, env)
|
||||
|
||||
server, out, queue := newWAFWith(t, func(params waf.Params) *waf.Server {
|
||||
params.CoreRuleSet = crs
|
||||
|
||||
return waf.New(params)
|
||||
}, silentGeoJSURL, time.Now, settings)
|
||||
server.SetBesideLookupTimeLimit(waf.BesideLookupTimeLimit)
|
||||
|
||||
return server, out, queue
|
||||
}
|
||||
|
||||
// lookupWait is SWW_WAF_ASN_AND_COUNTRY_LOOKUP_TIMEOUT_MS for wait.
|
||||
func lookupWait(wait time.Duration) string {
|
||||
return strconv.FormatInt(wait.Milliseconds(), 10)
|
||||
}
|
||||
|
||||
// wantAnswer checks the status of an answer and how long it took.
|
||||
func wantAnswer(
|
||||
t *testing.T, status int, took time.Duration,
|
||||
wantStatus int, wantTook time.Duration, out *output,
|
||||
) {
|
||||
t.Helper()
|
||||
|
||||
if status != wantStatus || took != wantTook {
|
||||
t.Errorf(
|
||||
"answered %d after %s, want %d after %s; output:\n%s",
|
||||
status, took, wantStatus, wantTook, out.text(),
|
||||
)
|
||||
}
|
||||
}
|
||||
|
||||
// wantNoCoreRuleSetTrace checks that what the Core Rule Set found of a request
|
||||
// left no trace: line has none of its fields, and server has counted no match
|
||||
// of it and raised no alert for one.
|
||||
func wantNoCoreRuleSetTrace(
|
||||
t *testing.T, server *waf.Server, queue *alerts.Queue, line logLine,
|
||||
) {
|
||||
t.Helper()
|
||||
|
||||
if line.DurationCoreRuleSetMS != nil || line.CoreRuleSetScore != nil ||
|
||||
line.CoreRuleSetRuleIDs != nil {
|
||||
t.Errorf(
|
||||
"log line has Core Rule Set time %v, score %v and rule ids %v, want none",
|
||||
line.DurationCoreRuleSetMS,
|
||||
line.CoreRuleSetScore,
|
||||
line.CoreRuleSetRuleIDs,
|
||||
)
|
||||
}
|
||||
|
||||
recorder := httptest.NewRecorder()
|
||||
server.Metrics.ServeHTTP(
|
||||
recorder,
|
||||
httptest.NewRequestWithContext(
|
||||
t.Context(),
|
||||
http.MethodGet,
|
||||
waf.MetricsPath,
|
||||
http.NoBody,
|
||||
),
|
||||
)
|
||||
|
||||
if strings.Contains(
|
||||
recorder.Body.String(),
|
||||
"sww_core_rule_set_matches_total{",
|
||||
) {
|
||||
t.Errorf("a Core Rule Set match is counted")
|
||||
}
|
||||
|
||||
for _, alert := range queue.Snapshot().Waiting[alerts.DestinationWebhook] {
|
||||
if alert.Event == alerts.EventCoreRuleSetMatch {
|
||||
t.Errorf("alert raised: %+v", alert)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestTheLookupAndTheCoreRuleSetTakeTheLongerOfTheirTimesNotTheirSum(
|
||||
t *testing.T,
|
||||
) {
|
||||
t.Parallel()
|
||||
|
||||
inspection := checksTimeLimit - time.Millisecond
|
||||
|
||||
for _, tc := range []struct {
|
||||
name string
|
||||
// wait is how long the request waits for GeoJS.
|
||||
wait time.Duration
|
||||
}{
|
||||
{"the lookup waits longer", defaultLookupWait},
|
||||
{"the inspection takes longer", inspection / 2},
|
||||
} {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
crs := &heldCoreRuleSet{took: inspection}
|
||||
server, out, _ := newWAFHoldingTheCoreRuleSet(
|
||||
t,
|
||||
crs,
|
||||
map[string]string{
|
||||
asnAndCountryLookupTimeout: lookupWait(tc.wait),
|
||||
},
|
||||
)
|
||||
|
||||
synctest.Test(t, func(t *testing.T) {
|
||||
defer waitForGeoJSRequestsToBeAbandoned()
|
||||
|
||||
status, took := askDirectly(
|
||||
t,
|
||||
server,
|
||||
func(*http.Request) {},
|
||||
)
|
||||
|
||||
wantAnswer(t, status, took, http.StatusOK,
|
||||
max(tc.wait, inspection), out)
|
||||
wantLine(t, out.requestLine(t), http.StatusOK,
|
||||
requestlog.ActionForward)
|
||||
})
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// firstToRefuse is a case of
|
||||
// TestTheFirstOfTheLookupAndTheRequestChecksToRefuseIsAnswered: a request
|
||||
// under env, whose inspection by the Core Rule Set takes inspection, refused as
|
||||
// action after took.
|
||||
type firstToRefuse struct {
|
||||
name string
|
||||
env map[string]string
|
||||
inspection time.Duration
|
||||
took time.Duration
|
||||
action string
|
||||
}
|
||||
|
||||
// firstToRefuseCases are the cases of
|
||||
// TestTheFirstOfTheLookupAndTheRequestChecksToRefuseIsAnswered.
|
||||
func firstToRefuseCases() []firstToRefuse {
|
||||
return []firstToRefuse{
|
||||
{
|
||||
name: "a trap path, before GeoJS answers",
|
||||
env: map[string]string{trapPaths: "/", allowedCountries: "DE"},
|
||||
took: 0, action: requestlog.ActionTemporarilyBanned,
|
||||
},
|
||||
{
|
||||
name: "the Core Rule Set, before GeoJS answers",
|
||||
env: map[string]string{allowedCountries: "DE"},
|
||||
inspection: 100 * time.Millisecond,
|
||||
took: 100 * time.Millisecond, action: requestlog.ActionBlockedByCoreRuleSet,
|
||||
},
|
||||
{
|
||||
name: "the country list, while the Core Rule Set inspects",
|
||||
env: map[string]string{
|
||||
allowedCountries: "DE", asnAndCountryLookupTimeout: "100",
|
||||
},
|
||||
inspection: 400 * time.Millisecond,
|
||||
took: 100 * time.Millisecond, action: requestlog.ActionCountryDenied,
|
||||
},
|
||||
{
|
||||
name: "the rate limits, while the Core Rule Set inspects",
|
||||
env: map[string]string{
|
||||
rateLimitPerMinute: "2", unknownCountryLimitPercent: "50",
|
||||
asnAndCountryLookupTimeout: "100",
|
||||
},
|
||||
inspection: 400 * time.Millisecond,
|
||||
took: 100 * time.Millisecond, action: requestlog.ActionClientIsRateLimited,
|
||||
},
|
||||
}
|
||||
}
|
||||
|
||||
func TestTheFirstOfTheLookupAndTheRequestChecksToRefuseIsAnswered(
|
||||
t *testing.T,
|
||||
) {
|
||||
t.Parallel()
|
||||
|
||||
const forbidden = http.StatusForbidden
|
||||
|
||||
for _, tc := range firstToRefuseCases() {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
crs := &heldCoreRuleSet{
|
||||
took: tc.inspection,
|
||||
result: coreRuleSetMatch(),
|
||||
}
|
||||
server, out, queue := newWAFHoldingTheCoreRuleSet(t, crs, tc.env)
|
||||
|
||||
synctest.Test(t, func(t *testing.T) {
|
||||
defer waitForGeoJSRequestsToBeAbandoned()
|
||||
|
||||
// Under the rate limits of the last case, the client's next
|
||||
// request takes it over a limit only at the percentage of an
|
||||
// unknown country, so it waits for GeoJS.
|
||||
countClient(t, server, time.Now())
|
||||
|
||||
status, took := askDirectly(
|
||||
t,
|
||||
server,
|
||||
func(*http.Request) {},
|
||||
)
|
||||
|
||||
wantAnswer(t, status, took, forbidden, tc.took, out)
|
||||
|
||||
line := out.requestLine(t)
|
||||
wantLine(t, line, forbidden, tc.action)
|
||||
|
||||
if tc.action == requestlog.ActionBlockedByCoreRuleSet {
|
||||
return
|
||||
}
|
||||
|
||||
if tc.action != requestlog.ActionClientIsRateLimited &&
|
||||
line.Counts != (ratelimit.Counts{}) {
|
||||
t.Errorf(
|
||||
"the rate limits counted the request: %+v",
|
||||
line.Counts,
|
||||
)
|
||||
}
|
||||
|
||||
// What the Core Rule Set finds once the request is answered is
|
||||
// dropped.
|
||||
time.Sleep(tc.inspection)
|
||||
wantNoCoreRuleSetTrace(t, server, queue, line)
|
||||
})
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestARequestRefusedBeforeTheLookupStartsNoInspection(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
crs := &heldCoreRuleSet{took: time.Second}
|
||||
server, out, _ := newWAFHoldingTheCoreRuleSet(t, crs, nil)
|
||||
|
||||
synctest.Test(t, func(t *testing.T) {
|
||||
banClient(t, server, time.Now())
|
||||
|
||||
status, took := askDirectly(t, server, func(*http.Request) {})
|
||||
|
||||
wantAnswer(t, status, took, http.StatusForbidden, 0, out)
|
||||
|
||||
requests := testutil.ToFloat64(server.Metrics.GeoJSRequests)
|
||||
if asked := crs.asked.Load(); asked != 0 || requests != 0 {
|
||||
t.Errorf(
|
||||
"%d inspections and %v requests to GeoJS, want none",
|
||||
asked,
|
||||
requests,
|
||||
)
|
||||
}
|
||||
})
|
||||
}
|
||||
|
||||
func TestAClientInTheAllowNetsIsAllowedAtOnceAndNothingElseStarts(
|
||||
t *testing.T,
|
||||
) {
|
||||
t.Parallel()
|
||||
|
||||
crs := &heldCoreRuleSet{took: time.Second, result: coreRuleSetMatch()}
|
||||
server, out, _ := newWAFHoldingTheCoreRuleSet(t, crs,
|
||||
map[string]string{allowNets: client, allowedCountries: "DE"})
|
||||
|
||||
synctest.Test(t, func(t *testing.T) {
|
||||
status, took := askDirectly(t, server, func(*http.Request) {})
|
||||
|
||||
wantAnswer(t, status, took, http.StatusOK, 0, out)
|
||||
|
||||
requests := testutil.ToFloat64(server.Metrics.GeoJSRequests)
|
||||
if asked := crs.asked.Load(); asked != 0 || requests != 0 {
|
||||
t.Errorf(
|
||||
"%d inspections and %v requests to GeoJS, want none",
|
||||
asked,
|
||||
requests,
|
||||
)
|
||||
}
|
||||
})
|
||||
}
|
||||
|
||||
func TestARequestTheCoreRuleSetHasNotFinishedWithIn500MillisecondsIsRefused(
|
||||
t *testing.T,
|
||||
) {
|
||||
t.Parallel()
|
||||
|
||||
crs := &heldCoreRuleSet{took: checksTimeLimit + time.Second}
|
||||
server, out, _ := newWAFHoldingTheCoreRuleSet(t, crs,
|
||||
map[string]string{refusalsLimitPerMinute: "1"})
|
||||
|
||||
synctest.Test(t, func(t *testing.T) {
|
||||
defer waitForGeoJSRequestsToBeAbandoned()
|
||||
|
||||
// Were it counted toward the refusal limit, the second refusal would
|
||||
// ban the client, and the third request would be refused for the ban.
|
||||
for i := range 3 {
|
||||
status, took := askDirectly(t, server, func(req *http.Request) {
|
||||
req.Header.Set(requestIDHeader, "id-"+strconv.Itoa(i))
|
||||
})
|
||||
|
||||
wantAnswer(t, status, took, checksTimedOut, checksTimeLimit, out)
|
||||
|
||||
line := out.requestLines(t, i+1)[i]
|
||||
wantLine(
|
||||
t,
|
||||
line,
|
||||
checksTimedOut,
|
||||
requestlog.ActionChecksTimedOut,
|
||||
)
|
||||
wantHeuristic(t, line, "core_rule_set", `{"time_limit_ms": 500}`)
|
||||
|
||||
if line.Offence != "" || line.BanExpires != "" {
|
||||
t.Errorf(
|
||||
"log line has offence %q and ban_expires %q, want neither",
|
||||
line.Offence,
|
||||
line.BanExpires,
|
||||
)
|
||||
}
|
||||
}
|
||||
|
||||
if bans := server.Ledger.Snapshot(); len(bans) != 0 {
|
||||
t.Errorf("bans %+v, want none", bans)
|
||||
}
|
||||
|
||||
offences := historyOf(t, server, client).Offences
|
||||
if offences != (ratelimit.Offences{}) {
|
||||
t.Errorf("history counts the offences %+v, want none", offences)
|
||||
}
|
||||
|
||||
time.Sleep(crs.took)
|
||||
})
|
||||
}
|
||||
|
||||
func TestObserveModeAllowsARequestTheCoreRuleSetHasNotFinishedWith(
|
||||
t *testing.T,
|
||||
) {
|
||||
t.Parallel()
|
||||
|
||||
crs := &heldCoreRuleSet{took: checksTimeLimit + time.Second}
|
||||
server, out, _ := newWAFHoldingTheCoreRuleSet(t, crs,
|
||||
map[string]string{mode: observe})
|
||||
|
||||
synctest.Test(t, func(t *testing.T) {
|
||||
defer waitForGeoJSRequestsToBeAbandoned()
|
||||
|
||||
status, took := askDirectly(t, server, func(*http.Request) {})
|
||||
|
||||
wantAnswer(t, status, took, http.StatusOK, checksTimeLimit, out)
|
||||
|
||||
line := out.requestLine(t)
|
||||
wantLine(t, line, http.StatusOK, requestlog.ActionForward)
|
||||
|
||||
if line.WouldAction != requestlog.ActionChecksTimedOut {
|
||||
t.Errorf(
|
||||
"log line has would_action %q, want %q",
|
||||
line.WouldAction,
|
||||
requestlog.ActionChecksTimedOut,
|
||||
)
|
||||
}
|
||||
|
||||
time.Sleep(crs.took)
|
||||
})
|
||||
}
|
||||
+23
-15
@@ -13,10 +13,11 @@ import (
|
||||
"sneak.berlin/go/smallwebwaf/internal/requestlog"
|
||||
)
|
||||
|
||||
// coreRuleSet inspects the request with the Core Rule Set, unless
|
||||
// SWW_WAF_CORE_RULE_SET_MODE is off or SWW_WAF_CORE_RULE_SET_EXEMPT_PATHS
|
||||
// exempts its path, as pathExempt decides, and notes the rules it matched and
|
||||
// its score in the log line, and the rules in the metrics. A score at or over
|
||||
// coreRuleSet takes what the Core Rule Set found of the request, which
|
||||
// findBesideLookup inspected with it unless SWW_WAF_CORE_RULE_SET_MODE is off
|
||||
// or SWW_WAF_CORE_RULE_SET_EXEMPT_PATHS exempts its path, as pathExempt
|
||||
// decides. It notes the rules the request matched and its score in the log
|
||||
// line, and the rules in the metrics. A score at or over
|
||||
// SWW_WAF_CORE_RULE_SET_SCORE_THRESHOLD is a match: it adds the tag
|
||||
// exploitsignature, raises the core_rule_set_match alert, and in block mode
|
||||
// refuses the request with 403, which is an offence its client's history
|
||||
@@ -30,21 +31,17 @@ func (coreRuleSet) check(
|
||||
rq *request,
|
||||
_ time.Time,
|
||||
) *verdict {
|
||||
cfg := rq.h.config
|
||||
if cfg.CoreRuleSetMode == config.CoreRuleSetModeOff ||
|
||||
pathExempt(rq.in, cfg.CoreRuleSetExemptPaths) {
|
||||
findings := rq.findings
|
||||
if !findings.inspected {
|
||||
return nil
|
||||
}
|
||||
|
||||
start := time.Now()
|
||||
|
||||
// A question's body has been read into memory, so that reading it here
|
||||
// cannot fail.
|
||||
result, read, _ := rq.h.coreRuleSet.Inspect(rq.in, rq.client, rq.in.Body)
|
||||
rq.requestBytes = int64(len(read))
|
||||
cfg := rq.h.config
|
||||
result := findings.inspection
|
||||
rq.requestBytes = int64(len(findings.read))
|
||||
|
||||
rq.line.DurationCoreRuleSetMS = new(
|
||||
requestlog.Milliseconds(time.Since(start)),
|
||||
requestlog.Milliseconds(findings.inspectionTime),
|
||||
)
|
||||
rq.line.CoreRuleSetRuleIDs = result.RuleIDs
|
||||
rq.line.CoreRuleSetScore = &result.Score
|
||||
@@ -61,7 +58,7 @@ func (coreRuleSet) check(
|
||||
rq.addTag(config.ExploitSignatureTag)
|
||||
rq.alertCoreRuleSetMatch(result)
|
||||
|
||||
if cfg.CoreRuleSetMode == config.CoreRuleSetModeDetect {
|
||||
if !rq.refusedByCoreRuleSet(result) {
|
||||
return nil
|
||||
}
|
||||
|
||||
@@ -76,6 +73,17 @@ func (coreRuleSet) check(
|
||||
}
|
||||
}
|
||||
|
||||
// refusedByCoreRuleSet reports whether coreRuleSet refuses a request the Core
|
||||
// Rule Set found result of: whether the score is at or over
|
||||
// SWW_WAF_CORE_RULE_SET_SCORE_THRESHOLD in block mode.
|
||||
func (rq *request) refusedByCoreRuleSet(result coreruleset.Result) bool {
|
||||
cfg := rq.h.config
|
||||
threshold := cfg.CoreRuleSetScoreThreshold
|
||||
|
||||
return cfg.CoreRuleSetMode == config.CoreRuleSetModeBlock &&
|
||||
threshold != 0 && result.Score >= threshold
|
||||
}
|
||||
|
||||
// alertCoreRuleSetMatch raises the core_rule_set_match alert for the request,
|
||||
// which the Core Rule Set scored at result, at or over
|
||||
// SWW_WAF_CORE_RULE_SET_SCORE_THRESHOLD. Its detail gives the rule ids, the
|
||||
|
||||
@@ -0,0 +1,13 @@
|
||||
package waf
|
||||
|
||||
import "time"
|
||||
|
||||
// BesideLookupTimeLimit is besideLookupTimeLimit.
|
||||
const BesideLookupTimeLimit = besideLookupTimeLimit
|
||||
|
||||
// SetBesideLookupTimeLimit gives the heuristics of besideLookup limit on s in
|
||||
// place of besideLookupTimeLimit.
|
||||
func (s *Server) SetBesideLookupTimeLimit(limit time.Duration) {
|
||||
h := s.Handler.(*handler) //nolint:forcetypeassert // New makes it one
|
||||
h.besideLookupLimit = limit
|
||||
}
|
||||
@@ -54,8 +54,6 @@ func beforeLookup() []heuristic {
|
||||
}
|
||||
|
||||
// afterLookup are the heuristics that run once the lookup has been made.
|
||||
// tag_deny runs twice: before the rate limits, on the tags known by then, and
|
||||
// last, on the tag rule_files and core_rule_set can add.
|
||||
func afterLookup() []heuristic {
|
||||
return []heuristic{
|
||||
countries{},
|
||||
@@ -64,9 +62,26 @@ func afterLookup() []heuristic {
|
||||
abuseIPDB{},
|
||||
tagDeny{},
|
||||
rateLimits{},
|
||||
}
|
||||
}
|
||||
|
||||
// besideLookup are the heuristics that only read the request. Their work runs
|
||||
// beside the lookup and afterLookup, in findBesideLookup, which changes nothing
|
||||
// of the request. Their check then applies what that work found, and runs only
|
||||
// once waf takes it.
|
||||
func besideLookup() []heuristic {
|
||||
return []heuristic{
|
||||
trapPaths{},
|
||||
ruleFiles{},
|
||||
coreRuleSet{},
|
||||
}
|
||||
}
|
||||
|
||||
// last are the heuristics that run once both lists above have let the request
|
||||
// through. tag_deny runs a second time, on the tag rule_files and
|
||||
// core_rule_set can add.
|
||||
func last() []heuristic {
|
||||
return []heuristic{
|
||||
tagDeny{},
|
||||
}
|
||||
}
|
||||
@@ -95,7 +110,7 @@ func (rq *request) runAfterAnswer(ctx context.Context) {
|
||||
rq.runBanningHeuristics(ctx, afterAnswer())
|
||||
}
|
||||
|
||||
// runHeuristics runs list, beforeLookup or afterLookup, on the request at now,
|
||||
// runHeuristics runs list, one of those checkClient runs, on the request at now,
|
||||
// and returns the first verdict, or nil when none gives one. The log line's
|
||||
// heuristic and heuristic_figures name the heuristic that gave it, and give its
|
||||
// figures. ctx is the request's own context.
|
||||
|
||||
+66
-22
@@ -21,36 +21,80 @@ const (
|
||||
countryHeader = "X-Client-Country"
|
||||
)
|
||||
|
||||
// lookUp looks up the client's AS number and country, in the lookup
|
||||
// database or through GeoJS, and notes them for the log line and as the
|
||||
// request's asn= and country= tags, unless
|
||||
// SWW_WAF_LOOKUP_SOURCE is off or the client is on a private, loopback or
|
||||
// link-local address, which no lookup can place. The lookup database
|
||||
// answers at once. With GeoJS, while a setting needs the answer, such as a
|
||||
// country list or a biased threshold, a new client's request waits for it up
|
||||
// to SWW_WAF_ASN_AND_COUNTRY_LOOKUP_TIMEOUT_MS, unless refusedWhateverTheAnswer
|
||||
// reports true. ctx is the request's own context.
|
||||
func (rq *request) lookUp(ctx context.Context, now time.Time) {
|
||||
// lookupResult is what the lookup gave for a request: whether its client was
|
||||
// looked up, the answer, and whether the request skipped the wait for GeoJS.
|
||||
type lookupResult struct {
|
||||
lookedUp, skippedWait bool
|
||||
answer geojs.Answer
|
||||
}
|
||||
|
||||
// startLookUp starts looking up the client's AS number and country, in the
|
||||
// lookup database or through GeoJS, unless SWW_WAF_LOOKUP_SOURCE is off or the
|
||||
// client is on a private, loopback or link-local address, which no lookup can
|
||||
// place. It returns the channel the result comes on. A result that needs no
|
||||
// wait is on it when startLookUp returns: the lookup database's, an answer
|
||||
// GeoJS gave earlier, or none. With GeoJS, while a setting needs the answer,
|
||||
// such as a country list or a biased threshold, a new client's request waits
|
||||
// for it up to SWW_WAF_ASN_AND_COUNTRY_LOOKUP_TIMEOUT_MS, unless
|
||||
// refusedWhateverTheAnswer reports true. The wait goes on to its end even once
|
||||
// the request has been answered, so that it ends as it would have for a
|
||||
// request that waited. startLookUp changes nothing of the request. ctx is the
|
||||
// request's own context.
|
||||
func (rq *request) startLookUp(
|
||||
ctx context.Context, now time.Time,
|
||||
) <-chan lookupResult {
|
||||
result := make(chan lookupResult, 1)
|
||||
|
||||
if rq.h.config.LookupSource == "off" || !canBePlaced(rq.client) {
|
||||
return
|
||||
result <- lookupResult{}
|
||||
|
||||
return result
|
||||
}
|
||||
|
||||
client := rq.h.clientGroup(rq.client)
|
||||
|
||||
switch {
|
||||
case rq.h.config.LookupSource == "file":
|
||||
rq.lookupAnswer = rq.h.lookupFile.LookUp(client)
|
||||
case rq.refusedWhateverTheAnswer(now):
|
||||
rq.lookupAnswer = rq.h.geojs.LookUpWithoutWaiting(ctx, client)
|
||||
rq.skippedLookupWait = rq.lookupAnswer.Answered.IsZero()
|
||||
case !rq.h.waitForGeoJS:
|
||||
rq.lookupAnswer = rq.h.geojs.LookUpWithoutWaiting(ctx, client)
|
||||
default:
|
||||
rq.lookupAnswer = rq.h.geojs.LookUp(ctx, client,
|
||||
rq.h.config.ASNAndCountryLookupTimeout)
|
||||
if rq.h.config.LookupSource == "file" {
|
||||
result <- lookupResult{
|
||||
lookedUp: true, answer: rq.h.lookupFile.LookUp(client),
|
||||
}
|
||||
|
||||
return result
|
||||
}
|
||||
|
||||
answer := rq.h.geojs.LookUpWithoutWaiting(ctx, client)
|
||||
refused := rq.refusedWhateverTheAnswer(now)
|
||||
|
||||
if !answer.Answered.IsZero() || refused || !rq.h.waitForGeoJS {
|
||||
result <- lookupResult{
|
||||
lookedUp: true,
|
||||
skippedWait: refused && answer.Answered.IsZero(),
|
||||
answer: answer,
|
||||
}
|
||||
|
||||
return result
|
||||
}
|
||||
|
||||
go func() {
|
||||
result <- lookupResult{
|
||||
lookedUp: true,
|
||||
answer: rq.h.geojs.LookUp(context.WithoutCancel(ctx), client,
|
||||
rq.h.config.ASNAndCountryLookupTimeout),
|
||||
}
|
||||
}()
|
||||
|
||||
return result
|
||||
}
|
||||
|
||||
// noteLookup notes result for the log line and as the request's asn= and
|
||||
// country= tags.
|
||||
func (rq *request) noteLookup(result lookupResult) {
|
||||
if !result.lookedUp {
|
||||
return
|
||||
}
|
||||
|
||||
rq.lookedUp = true
|
||||
rq.lookupAnswer = result.answer
|
||||
rq.skippedLookupWait = result.skippedWait
|
||||
rq.line.ASN = rq.lookupAnswer.ASN
|
||||
rq.line.ASName = rq.lookupAnswer.ASName
|
||||
rq.line.Country = rq.lookupAnswer.Country
|
||||
|
||||
@@ -190,7 +190,7 @@ func TestASNumberAndNameInTheLogLineTheHistoryTheBanNotesAndTheAlert(
|
||||
clk.Now,
|
||||
map[string]string{
|
||||
trustedProxies: trustLocalhost,
|
||||
alertWebhookURL: "https://alerts.example/sww",
|
||||
alertWebhookURL: alertWebhook,
|
||||
rateLimitPerMinute: "1",
|
||||
},
|
||||
)
|
||||
|
||||
+15
-2
@@ -8,6 +8,7 @@ import (
|
||||
"io"
|
||||
"log/slog"
|
||||
"net/http"
|
||||
"net/netip"
|
||||
"time"
|
||||
|
||||
lru "github.com/hashicorp/golang-lru/v2"
|
||||
@@ -66,7 +67,7 @@ type Params struct {
|
||||
LookupFile *lookup.File
|
||||
// CoreRuleSet is what NewCoreRuleSet returns: what inspects requests
|
||||
// while SWW_WAF_CORE_RULE_SET_MODE is not off, and nil while it is.
|
||||
CoreRuleSet *coreruleset.CoreRuleSet
|
||||
CoreRuleSet CoreRuleSetInspector
|
||||
// Now tells the time by which requests are counted for the rate
|
||||
// limits, bans are made and run out, and GeoJS's answers are kept,
|
||||
// normally time.Now in UTC, the time the state files give.
|
||||
@@ -84,6 +85,14 @@ type Params struct {
|
||||
Alerts *alerts.Queue
|
||||
}
|
||||
|
||||
// CoreRuleSetInspector inspects a request with the Core Rule Set, as
|
||||
// coreruleset.CoreRuleSet's Inspect says. Tests give waf a stand-in for it.
|
||||
type CoreRuleSetInspector interface {
|
||||
Inspect(
|
||||
r *http.Request, client netip.Addr, body io.Reader,
|
||||
) (coreruleset.Result, []byte, error)
|
||||
}
|
||||
|
||||
// Server is the server waf runs, with the parts whose state the state
|
||||
// files keep, the lookup database, nil unless SWW_WAF_LOOKUP_SOURCE is file, the
|
||||
// lists fetched from URLs, which its Run fetches, the DNSBL zones' verdicts,
|
||||
@@ -158,6 +167,8 @@ func newHandler(params Params) *handler {
|
||||
rules: params.Rules,
|
||||
coreRuleSet: params.CoreRuleSet,
|
||||
alerts: params.Alerts,
|
||||
|
||||
besideLookupLimit: besideLookupTimeLimit,
|
||||
}
|
||||
if params.Config.RouteWatchURL != "" {
|
||||
h.routeWatch = routewatch.New(routewatch.Params{
|
||||
@@ -276,11 +287,13 @@ type handler struct {
|
||||
abuseIPDB *reputation.AbuseIPDB
|
||||
routeWatch *routewatch.Client
|
||||
rules *rules.Files
|
||||
coreRuleSet *coreruleset.CoreRuleSet
|
||||
coreRuleSet CoreRuleSetInspector
|
||||
alerts *alerts.Queue
|
||||
// waitForGeoJS is true when a setting needs each request's answer from
|
||||
// GeoJS before the request goes on.
|
||||
waitForGeoJS bool
|
||||
// besideLookupLimit is besideLookupTimeLimit. Only tests change it.
|
||||
besideLookupLimit time.Duration
|
||||
// awaitingReport holds the decision on each answered question by its id
|
||||
// until its report comes, and nil for a question still being handled.
|
||||
awaitingReport *lru.Cache[string, *decision]
|
||||
|
||||
@@ -512,6 +512,11 @@ func newWAFWith(
|
||||
Alerts: alertQueue,
|
||||
})
|
||||
|
||||
// The race detector and a busy host can slow the Core Rule Set's
|
||||
// inspections past the time limit. The tests of the limit set it back, and
|
||||
// hold the Core Rule Set with a stand-in.
|
||||
server.SetBesideLookupTimeLimit(time.Hour)
|
||||
|
||||
return server, out, alertQueue
|
||||
}
|
||||
|
||||
|
||||
+43
-9
@@ -81,6 +81,9 @@ type request struct {
|
||||
checked time.Time
|
||||
// refused is what the request was refused with, nil if it was not.
|
||||
refused *refusal
|
||||
// findings is what the heuristics of besideLookup found of the request,
|
||||
// once waf has taken it, and nil before.
|
||||
findings *besideLookupFindings
|
||||
// requestBytes is how many bytes of the request's body have been read.
|
||||
requestBytes int64
|
||||
}
|
||||
@@ -177,9 +180,9 @@ func requestHeaders(r *http.Request, names []string) map[string]string {
|
||||
// check is the one place where a request can be refused. It returns nil to let
|
||||
// the request through. The heuristics of checkClient decide: a request they
|
||||
// refuse is answered with SWW_WAF_BAN_RESPONSE, 403 for a block rule or the Core
|
||||
// Rule Set, or 414 or 431 for the header limits. In observe mode nothing is refused,
|
||||
// and the log line names what enforce mode would have done. ctx is the
|
||||
// request's own context.
|
||||
// Rule Set, 503 for heuristics that ran out of time, or 414 or 431 for the
|
||||
// header limits. In observe mode nothing is refused, and the log line names
|
||||
// what enforce mode would have done. ctx is the request's own context.
|
||||
func (rq *request) check(ctx context.Context) *refusal {
|
||||
v := rq.checkClient(ctx)
|
||||
if v == nil {
|
||||
@@ -196,6 +199,11 @@ func (rq *request) check(ctx context.Context) *refusal {
|
||||
case action == requestlog.ActionBlockedByBlockRule ||
|
||||
action == requestlog.ActionBlockedByCoreRuleSet:
|
||||
return &refusal{status: http.StatusForbidden, action: action}
|
||||
case action == requestlog.ActionChecksTimedOut:
|
||||
return &refusal{
|
||||
status: http.StatusServiceUnavailable,
|
||||
action: action,
|
||||
}
|
||||
case action == requestlog.ActionRequestLineTooLong:
|
||||
return &refusal{status: http.StatusRequestURITooLong, action: action}
|
||||
case action == requestlog.ActionRequestHeaderTooLarge:
|
||||
@@ -208,11 +216,16 @@ func (rq *request) check(ctx context.Context) *refusal {
|
||||
}
|
||||
}
|
||||
|
||||
// checkClient runs the heuristics of beforeLookup on the request, then looks
|
||||
// up the client's AS number and country, then runs those of afterLookup. It
|
||||
// returns the first verdict, or nil when none gives one. A client in
|
||||
// SWW_WAF_ALLOW_NETS skips them all, and is not looked up. ctx is the request's
|
||||
// own context.
|
||||
// checkClient runs the heuristics on the request, and returns the first
|
||||
// verdict, or nil when none gives one. A client in SWW_WAF_ALLOW_NETS skips them
|
||||
// all, and is not looked up. ctx is the request's own context.
|
||||
//
|
||||
// The heuristics of beforeLookup run first. Then the client's AS number and
|
||||
// country are looked up, and afterLookup runs once the lookup has its result.
|
||||
// While the lookup waits for GeoJS, besideLookup runs too, as runBesideLookup
|
||||
// says. A lookup that needs no wait is done, and afterLookup with it, before
|
||||
// besideLookup starts, so that besideLookup does no work for a request
|
||||
// afterLookup refuses.
|
||||
func (rq *request) checkClient(ctx context.Context) *verdict {
|
||||
if isInside(rq.client, rq.h.config.AllowNets) {
|
||||
return nil
|
||||
@@ -225,7 +238,28 @@ func (rq *request) checkClient(ctx context.Context) *verdict {
|
||||
return v
|
||||
}
|
||||
|
||||
rq.lookUp(ctx, now)
|
||||
lookedUp := rq.startLookUp(ctx, now)
|
||||
|
||||
select {
|
||||
case result := <-lookedUp:
|
||||
v := rq.afterLookUp(ctx, result)
|
||||
if v != nil {
|
||||
return v
|
||||
}
|
||||
|
||||
return rq.runBesideLookup(ctx, nil)
|
||||
default:
|
||||
return rq.runBesideLookup(ctx, lookedUp)
|
||||
}
|
||||
}
|
||||
|
||||
// afterLookUp notes result, the lookup's, and runs the heuristics of
|
||||
// afterLookup. It returns the first verdict, or nil when none gives one. ctx is
|
||||
// the request's own context.
|
||||
func (rq *request) afterLookUp(
|
||||
ctx context.Context, result lookupResult,
|
||||
) *verdict {
|
||||
rq.noteLookup(result)
|
||||
|
||||
// The lookup can wait for GeoJS, so the checks after it read the time
|
||||
// again: a request counted at the time it arrived could be counted
|
||||
|
||||
@@ -13,8 +13,8 @@ import (
|
||||
"sneak.berlin/go/smallwebwaf/internal/rules"
|
||||
)
|
||||
|
||||
// ruleFiles checks the request against the rules of the rule files at now,
|
||||
// notes the ids of those it matches in the log line, adds the tag
|
||||
// ruleFiles takes the rules of the rule files the request matched, which
|
||||
// findBesideLookup found, notes their ids in the log line, adds the tag
|
||||
// exploitsignature when it matches any, and refuses the request
|
||||
// when the last rule it matches refuses it: with ActionBlockedByBlockRule for a
|
||||
// block rule, and for a ban rule with the action for the ban it makes. A ban
|
||||
@@ -31,7 +31,7 @@ func (ruleFiles) check(
|
||||
rq *request,
|
||||
now time.Time,
|
||||
) *verdict {
|
||||
matched := rq.h.rules.Match(rq.in)
|
||||
matched := rq.findings.rules
|
||||
|
||||
for _, rule := range matched {
|
||||
rq.line.RuleIDs = append(rq.line.RuleIDs, rule.ID)
|
||||
@@ -70,3 +70,15 @@ func (ruleFiles) check(
|
||||
return nil
|
||||
}
|
||||
}
|
||||
|
||||
// refusedByRule reports whether ruleFiles refuses a request that matched
|
||||
// matched: whether the last of them is a block rule or a ban rule.
|
||||
func refusedByRule(matched []rules.Rule) bool {
|
||||
if len(matched) == 0 {
|
||||
return false
|
||||
}
|
||||
|
||||
action := matched[len(matched)-1].Action
|
||||
|
||||
return action == rules.ActionBlock || action == rules.ActionBan
|
||||
}
|
||||
@@ -11,9 +11,8 @@ import (
|
||||
"sneak.berlin/go/smallwebwaf/internal/rules"
|
||||
)
|
||||
|
||||
// trapPaths refuses a request for a path in SWW_WAF_TRAP_PATHS: its path as a
|
||||
// path rule sees it, without the query, as sent or once percent-decoded, is one
|
||||
// of them. Such a request is a clear sign of attack, as a ban rule's match is:
|
||||
// trapPaths refuses a request for a path in SWW_WAF_TRAP_PATHS, as trapPath
|
||||
// finds it. Such a request is a clear sign of attack, as a ban rule's match is:
|
||||
// it bans the client's netblock, or in observe mode raises the alert for the
|
||||
// ban it would have made. The ban's notes give the path as sent, and so does
|
||||
// its figure.
|
||||
@@ -26,9 +25,8 @@ func (trapPaths) check(
|
||||
rq *request,
|
||||
now time.Time,
|
||||
) *verdict {
|
||||
path := rules.Path(rq.in)
|
||||
if !slices.Contains(rq.h.config.TrapPaths, path) &&
|
||||
!slices.Contains(rq.h.config.TrapPaths, rules.DecodeOnce(path)) {
|
||||
path := rq.findings.trapPath
|
||||
if path == "" {
|
||||
return nil
|
||||
}
|
||||
|
||||
@@ -39,3 +37,16 @@ func (trapPaths) check(
|
||||
figures: map[string]any{"path": path},
|
||||
}
|
||||
}
|
||||
|
||||
// trapPath returns the request's path as sent, when its path as a path rule
|
||||
// sees it, without the query, as sent or once percent-decoded, is in
|
||||
// SWW_WAF_TRAP_PATHS, and "" otherwise.
|
||||
func (rq *request) trapPath() string {
|
||||
path := rules.Path(rq.in)
|
||||
if !slices.Contains(rq.h.config.TrapPaths, path) &&
|
||||
!slices.Contains(rq.h.config.TrapPaths, rules.DecodeOnce(path)) {
|
||||
return ""
|
||||
}
|
||||
|
||||
return path
|
||||
}
|
||||
Reference in new issue
Block a user