DNS blocklists asked in the background, verdicts kept (closes #104)
check / check (push) Waiting to run
check / check (push) Waiting to run
Zones in SWWAF_DNSBL_ZONES are asked about each client (RFC 5782 names) in the background, through the host's resolver or SWWAF_DNSBL_RESOLVER; no request waits. Verdicts last SWWAF_REPUTATION_CACHE_TTL and are kept in reputation.json, at most 100,000. After the blocklists, SWWAF_REPUTATION_ACTION (limit:25) denies, limits or logs a listed client; the log line names the zones, each raises reputation_hit, with metrics by zone. A failed, timed-out or refused query gives no verdict, raises source_failure, and pauses the zone a minute. Judgement call: answers in 127.255.255.0/24 or outside 127.0.0.0/8 are failures. Judgement call: the minute's pause after a failure; at most 1,000 queries at once. Rule suppressed: paralleltest on the DNSBL tests (Go's resolver shares state across synctest bubbles), funlen on the test of every logged setting. Model: opus-5-5
This commit is contained in:
+22
-15
@@ -28,16 +28,18 @@ func biasedThresholdsSet(cfg *config.Config) bool {
|
||||
|
||||
// limitPercentages returns the client's limit percentages, for the rate
|
||||
// limits and for the byte limits, by its AS number and country as looked
|
||||
// up, each "" when unknown, and the blocklists that list it. Each is the
|
||||
// lowest of those the settings give it, the first of them in the order
|
||||
// below when several are lowest: the percentage SWWAF_ASN_LIMIT_PERCENT
|
||||
// gives its AS number, the one the file SWWAF_ASN_LIMIT_PERCENT_URL names
|
||||
// gives it, the one SWWAF_COUNTRY_LIMIT_PERCENT gives its country, for a
|
||||
// client without a country, SWWAF_UNKNOWN_LIMIT_PERCENT, and for a client
|
||||
// a blocklist lists, the percentage of SWWAF_BLOCKLIST_ACTION while it is
|
||||
// limit. For the byte limits, SWWAF_ASN_BYTES_PERCENT and
|
||||
// SWWAF_COUNTRY_BYTES_PERCENT take the place of the first three for an AS
|
||||
// number or a country they list.
|
||||
// up, each "" when unknown, and the blocklists and DNSBL zones that list
|
||||
// it. Each is the lowest of those the settings give it, the first of them
|
||||
// in the order below when several are lowest: the percentage
|
||||
// SWWAF_ASN_LIMIT_PERCENT gives its AS number, the one the file
|
||||
// SWWAF_ASN_LIMIT_PERCENT_URL names gives it, the one
|
||||
// SWWAF_COUNTRY_LIMIT_PERCENT gives its country, for a client without a
|
||||
// country, SWWAF_UNKNOWN_LIMIT_PERCENT, for a client a blocklist lists,
|
||||
// the percentage of SWWAF_BLOCKLIST_ACTION while it is limit, and for a
|
||||
// client a DNSBL zone's verdict lists, the percentage of
|
||||
// SWWAF_REPUTATION_ACTION while it is limit. For the byte limits,
|
||||
// SWWAF_ASN_BYTES_PERCENT and SWWAF_COUNTRY_BYTES_PERCENT take the place
|
||||
// of the first three for an AS number or a country they list.
|
||||
func (rq *request) limitPercentages() (percentage, percentage) {
|
||||
cfg := rq.h.config
|
||||
asn, country := rq.line.ASN, rq.line.Country
|
||||
@@ -52,9 +54,14 @@ func (rq *request) limitPercentages() (percentage, percentage) {
|
||||
fetched = percentage{percent, "SWWAF_ASN_LIMIT_PERCENT_URL"}
|
||||
}
|
||||
|
||||
listed := percentage{percent: whole}
|
||||
if len(rq.line.Reputation) > 0 && cfg.BlocklistAction == "limit" {
|
||||
listed = percentage{cfg.BlocklistLimitPercent, "SWWAF_BLOCKLIST_ACTION"}
|
||||
blocklisted := percentage{percent: whole}
|
||||
if rq.blocklisted && cfg.BlocklistAction == "limit" {
|
||||
blocklisted = percentage{cfg.BlocklistLimitPercent, "SWWAF_BLOCKLIST_ACTION"}
|
||||
}
|
||||
|
||||
dnsblListed := percentage{percent: whole}
|
||||
if rq.dnsblListed && cfg.ReputationAction == "limit" {
|
||||
dnsblListed = percentage{cfg.ReputationLimitPercent, "SWWAF_REPUTATION_ACTION"}
|
||||
}
|
||||
|
||||
asnRequests := lowest(given(cfg.ASNLimitPercent, asn, "SWWAF_ASN_LIMIT_PERCENT"),
|
||||
@@ -71,8 +78,8 @@ func (rq *request) limitPercentages() (percentage, percentage) {
|
||||
countryBytes = given(cfg.CountryBytesPercent, country, "SWWAF_COUNTRY_BYTES_PERCENT")
|
||||
}
|
||||
|
||||
return lowest(asnRequests, countryRequests, unknown, listed),
|
||||
lowest(asnBytes, countryBytes, unknown, listed)
|
||||
return lowest(asnRequests, countryRequests, unknown, blocklisted, dnsblListed),
|
||||
lowest(asnBytes, countryBytes, unknown, blocklisted, dnsblListed)
|
||||
}
|
||||
|
||||
// given returns the percentage percents, the setting named setting, gives
|
||||
|
||||
+31
-11
@@ -71,15 +71,15 @@ type Params struct {
|
||||
Rules *rules.Files
|
||||
// Alerts receive the alert for each ban the proxy makes or makes
|
||||
// permanent, for each count over an anomaly threshold, for each request
|
||||
// whose client a blocklist lists, and for GeoJS failing or a fetch of a
|
||||
// list failing.
|
||||
// whose client a blocklist or a DNSBL zone lists, and for GeoJS failing,
|
||||
// a fetch of a list failing or a query to a DNSBL zone failing.
|
||||
Alerts *alerts.Queue
|
||||
}
|
||||
|
||||
// Server is the server smallwebwaf runs, with the parts of the proxy
|
||||
// whose state the state files keep, the lookup database, nil unless
|
||||
// SWWAF_LOOKUP_SOURCE is file, the lists fetched from URLs, which its Run
|
||||
// fetches, and the metrics.
|
||||
// fetches, the DNSBL zones' verdicts, and the metrics.
|
||||
type Server struct {
|
||||
*http.Server
|
||||
|
||||
@@ -89,6 +89,7 @@ type Server struct {
|
||||
Anomalies *anomaly.Counters
|
||||
LookupFile *lookup.File
|
||||
Lists *reputation.Lists
|
||||
DNSBL *reputation.DNSBL
|
||||
Metrics *metrics.Metrics
|
||||
}
|
||||
|
||||
@@ -101,6 +102,7 @@ type Server struct {
|
||||
func New(params Params) *Server {
|
||||
errorLog := slog.NewLogLogger(params.ProcessLog.Handler(), slog.LevelWarn)
|
||||
m := metrics.New(params.Config.MetricsTopN, params.Config.InstanceName)
|
||||
lists, dnsbl := newReputation(params)
|
||||
h := &handler{
|
||||
config: params.Config,
|
||||
requestLog: params.RequestLog,
|
||||
@@ -136,13 +138,10 @@ func New(params Params) *Server {
|
||||
Alerts: params.Alerts,
|
||||
}),
|
||||
lookupFile: params.LookupFile,
|
||||
lists: reputation.New(reputation.Params{
|
||||
BlocklistURLs: params.Config.BlocklistURLs, Refresh: params.Config.BlocklistRefresh,
|
||||
ASNLimitPercentURL: params.Config.ASNLimitPercentURL, Now: params.Now,
|
||||
ProcessLog: params.ProcessLog, Alerts: params.Alerts,
|
||||
}),
|
||||
rules: params.Rules,
|
||||
alerts: params.Alerts,
|
||||
lists: lists,
|
||||
dnsbl: dnsbl,
|
||||
rules: params.Rules,
|
||||
alerts: params.Alerts,
|
||||
}
|
||||
h.geojs = lookup.New(lookup.Params{
|
||||
URL: params.GeoJSURL,
|
||||
@@ -160,7 +159,7 @@ func New(params Params) *Server {
|
||||
})
|
||||
m.AddBansAndClients(h.ledger, h.limiter, params.Now)
|
||||
m.AddRules(params.Rules)
|
||||
m.AddReputation(h.lists)
|
||||
m.AddReputation(h.lists, h.dnsbl)
|
||||
|
||||
return &Server{
|
||||
Server: &http.Server{
|
||||
@@ -181,10 +180,30 @@ func New(params Params) *Server {
|
||||
Anomalies: h.anomalies,
|
||||
LookupFile: h.lookupFile,
|
||||
Lists: h.lists,
|
||||
DNSBL: h.dnsbl,
|
||||
Metrics: m,
|
||||
}
|
||||
}
|
||||
|
||||
// newReputation returns the lists fetched from URLs and the DNSBL zones'
|
||||
// verdicts, as the settings in params name them, with none fetched or
|
||||
// asked for yet.
|
||||
func newReputation(params Params) (*reputation.Lists, *reputation.DNSBL) {
|
||||
cfg := params.Config
|
||||
lists := reputation.New(reputation.Params{
|
||||
BlocklistURLs: cfg.BlocklistURLs, Refresh: cfg.BlocklistRefresh,
|
||||
ASNLimitPercentURL: cfg.ASNLimitPercentURL, Now: params.Now,
|
||||
ProcessLog: params.ProcessLog, Alerts: params.Alerts,
|
||||
})
|
||||
dnsbl := reputation.NewDNSBL(reputation.DNSBLParams{
|
||||
Zones: cfg.DNSBLZones, Resolver: cfg.DNSBLResolver, CacheTTL: cfg.ReputationCacheTTL,
|
||||
Timeout: cfg.ReputationTimeout, Now: params.Now, ProcessLog: params.ProcessLog,
|
||||
Alerts: params.Alerts,
|
||||
})
|
||||
|
||||
return lists, dnsbl
|
||||
}
|
||||
|
||||
// handler is the proxy. It holds what every request shares; what belongs
|
||||
// to one request is in a request.
|
||||
type handler struct {
|
||||
@@ -201,6 +220,7 @@ type handler struct {
|
||||
anomalies *anomaly.Counters
|
||||
lookupFile *lookup.File
|
||||
lists *reputation.Lists
|
||||
dnsbl *reputation.DNSBL
|
||||
rules *rules.Files
|
||||
alerts *alerts.Queue
|
||||
}
|
||||
|
||||
@@ -1,21 +1,47 @@
|
||||
package proxy
|
||||
|
||||
import (
|
||||
"context"
|
||||
|
||||
"sneak.berlin/go/smallwebwaf/internal/alerts"
|
||||
)
|
||||
|
||||
// blocklistDenied notes in the log line the URLs of the blocklists that
|
||||
// list the client, counts each of them in the metrics and raises a
|
||||
// reputation_hit alert for it, and reports whether SWWAF_BLOCKLIST_ACTION,
|
||||
// being deny, refuses the request. Being limit, it lowers the client's
|
||||
// limits instead (see limitPercentages), and being log, it does nothing
|
||||
// more.
|
||||
// blocklistDenied notes the blocklists that list the client, as
|
||||
// noteListed does, and reports whether SWWAF_BLOCKLIST_ACTION, being deny,
|
||||
// refuses the request. Being limit, it lowers the client's limits instead
|
||||
// (see limitPercentages), and being log, it does nothing more.
|
||||
func (rq *request) blocklistDenied() bool {
|
||||
listedBy := rq.h.lists.ListedBy(rq.client)
|
||||
rq.line.Reputation = listedBy
|
||||
rq.blocklisted = len(listedBy) > 0
|
||||
rq.noteListed(listedBy, "listed by a blocklist")
|
||||
|
||||
for _, listURL := range listedBy {
|
||||
rq.h.metrics.ReputationHit(listURL)
|
||||
return rq.blocklisted && rq.h.config.BlocklistAction == "deny"
|
||||
}
|
||||
|
||||
// dnsblDenied notes the DNSBL zones whose verdict lists the client, as
|
||||
// noteListed does, and reports whether SWWAF_REPUTATION_ACTION, being
|
||||
// deny, refuses the request. Being limit, it lowers the client's limits
|
||||
// instead (see limitPercentages), and being log, it does nothing more. A
|
||||
// zone without a verdict on the client is asked about it in the
|
||||
// background, and the request does not wait for the answer. ctx is the
|
||||
// request's own context.
|
||||
func (rq *request) dnsblDenied(ctx context.Context) bool {
|
||||
listedBy := rq.h.dnsbl.ListedBy(ctx, rq.client)
|
||||
rq.dnsblListed = len(listedBy) > 0
|
||||
rq.noteListed(listedBy, "listed by a DNSBL zone")
|
||||
|
||||
return rq.dnsblListed && rq.h.config.ReputationAction == "deny"
|
||||
}
|
||||
|
||||
// noteListed adds sources, the URLs of the blocklists or the DNSBL zones
|
||||
// that list the client, to the log line's reputation, counts each of them
|
||||
// in the metrics, and raises a reputation_hit alert, with reason, for
|
||||
// each.
|
||||
func (rq *request) noteListed(sources []string, reason string) {
|
||||
rq.line.Reputation = append(rq.line.Reputation, sources...)
|
||||
|
||||
for _, source := range sources {
|
||||
rq.h.metrics.ReputationHit(source)
|
||||
rq.h.alerts.Raise(alerts.Alert{
|
||||
Event: alerts.EventReputationHit,
|
||||
Client: rq.client,
|
||||
@@ -23,10 +49,8 @@ func (rq *request) blocklistDenied() bool {
|
||||
ASN: rq.line.ASN,
|
||||
ASName: rq.line.ASName,
|
||||
Country: rq.line.Country,
|
||||
Reason: "listed by a blocklist",
|
||||
Detail: map[string]any{"source": listURL},
|
||||
Reason: reason,
|
||||
Detail: map[string]any{"source": source},
|
||||
})
|
||||
}
|
||||
|
||||
return len(listedBy) > 0 && rq.h.config.BlocklistAction == "deny"
|
||||
}
|
||||
|
||||
@@ -21,11 +21,14 @@ const (
|
||||
asnLimitPercentURL = "SWWAF_ASN_LIMIT_PERCENT_URL"
|
||||
)
|
||||
|
||||
// The actions of SWWAF_BLOCKLIST_ACTION but limit, which has a
|
||||
// percentage.
|
||||
// The actions of SWWAF_BLOCKLIST_ACTION and SWWAF_REPUTATION_ACTION:
|
||||
// limitHalf gives a listed client half of every limit, and limitQuarter a
|
||||
// quarter.
|
||||
const (
|
||||
actionDeny = "deny"
|
||||
actionLog = "log"
|
||||
actionDeny = "deny"
|
||||
actionLog = "log"
|
||||
limitHalf = "limit:50"
|
||||
limitQuarter = "limit:25"
|
||||
)
|
||||
|
||||
// The lists these tests name, which are never fetched: each test puts in
|
||||
@@ -55,7 +58,7 @@ func TestEachBlocklistActionForAListedAddressAndAListedNetblock(t *testing.T) {
|
||||
},
|
||||
{
|
||||
// Half of 4 requests a minute: the third breaks the limit.
|
||||
"limit:50", []int{http.StatusOK, http.StatusOK, http.StatusForbidden},
|
||||
limitHalf, []int{http.StatusOK, http.StatusOK, http.StatusForbidden},
|
||||
[]string{forward, forward, requestlog.ActionRateLimited},
|
||||
"50 from " + blocklistAction,
|
||||
},
|
||||
@@ -151,10 +154,10 @@ func TestBlocklistLimitTakesPartInTheLowestPercentageOfEveryLimit(t *testing.T)
|
||||
// percentText gives them, and limitHit its limit_hit.
|
||||
want, limitHit string
|
||||
}{
|
||||
{"limit:50", asnDEQuarter, "25 from " + asnLimitPercent, minuteBytes},
|
||||
{"limit:25", asnDEHalf, "25 from " + blocklistAction, minuteBytes},
|
||||
{limitHalf, asnDEQuarter, "25 from " + asnLimitPercent, minuteBytes},
|
||||
{limitQuarter, asnDEHalf, "25 from " + blocklistAction, minuteBytes},
|
||||
// The AS number's, the first of two alike.
|
||||
{"limit:25", asnDEQuarter, "25 from " + asnLimitPercent, minuteBytes},
|
||||
{limitQuarter, asnDEQuarter, "25 from " + asnLimitPercent, minuteBytes},
|
||||
{actionLog, asnDE + ":100", none, ""},
|
||||
} {
|
||||
t.Run(tc.action+" "+tc.asnPercent, func(t *testing.T) {
|
||||
@@ -299,11 +302,298 @@ func TestEachBlocklistThatListsAClientRaisesAnAlertOncePerCooldownAndIsCounted(
|
||||
}
|
||||
}
|
||||
|
||||
// The DNSBL settings.
|
||||
const (
|
||||
dnsblZones = "SWWAF_DNSBL_ZONES"
|
||||
dnsblResolver = "SWWAF_DNSBL_RESOLVER"
|
||||
reputationAction = "SWWAF_REPUTATION_ACTION"
|
||||
)
|
||||
|
||||
// The DNSBL zones these tests name, which are never asked about the
|
||||
// clients the tests send requests from: each test puts in the verdicts it
|
||||
// needs, as reputation.json would at start. A query a test does start is
|
||||
// sent to noResolver, where nothing listens, so that none leaves the host.
|
||||
const (
|
||||
dnsblZone = "dnsbl.example"
|
||||
otherZone = "other.example"
|
||||
noResolver = "127.0.0.1:9"
|
||||
)
|
||||
|
||||
func TestEachReputationActionForAClientADNSBLZoneLists(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
forward, denied := requestlog.ActionForward, requestlog.ActionDenied
|
||||
|
||||
for _, tc := range []struct {
|
||||
action string
|
||||
// statuses and actions are those of a listed client's three
|
||||
// requests, and percent their limit_percent, as percentText gives it.
|
||||
statuses []int
|
||||
actions []string
|
||||
percent string
|
||||
}{
|
||||
{
|
||||
actionDeny, []int{http.StatusForbidden, http.StatusForbidden, http.StatusForbidden},
|
||||
[]string{denied, denied, denied}, none,
|
||||
},
|
||||
{
|
||||
// Half of 4 requests a minute: the third breaks the limit.
|
||||
limitHalf, []int{http.StatusOK, http.StatusOK, http.StatusForbidden},
|
||||
[]string{forward, forward, requestlog.ActionRateLimited},
|
||||
"50 from " + reputationAction,
|
||||
},
|
||||
{
|
||||
actionLog, []int{http.StatusOK, http.StatusOK, http.StatusOK},
|
||||
[]string{forward, forward, forward}, none,
|
||||
},
|
||||
} {
|
||||
t.Run(tc.action, func(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
s, server, _ := startWithLookups(t, map[string]string{
|
||||
rateLimitPerMinute: fourAMinute, dnsblZones: dnsblZone + "," + otherZone,
|
||||
dnsblResolver: noResolver, reputationAction: tc.action,
|
||||
})
|
||||
listedBy := map[string][]string{
|
||||
fromDE: {dnsblZone, otherZone}, fromKP: {otherZone}, unplaced: nil,
|
||||
}
|
||||
loadVerdicts(server, listedBy)
|
||||
|
||||
for _, from := range []string{fromDE, fromKP} {
|
||||
for i := range 3 {
|
||||
line := s.get(from, tc.statuses[i], tc.actions[i])
|
||||
// In the order SWWAF_DNSBL_ZONES names them.
|
||||
wantReputation(t, line, listedBy[from]...)
|
||||
wantPercent(t, "limit_percent", line.LimitPercent,
|
||||
line.LimitPercentSetting, tc.percent)
|
||||
|
||||
// A request refused for the verdict is not counted.
|
||||
counted := line.fields["counts"] != nil
|
||||
if counted != (tc.actions[i] != denied) {
|
||||
t.Errorf("request from %s counted %t, logged %s", from, counted,
|
||||
tc.actions[i])
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// A client no zone lists has the whole limit.
|
||||
for range 3 {
|
||||
line := s.get(unplaced, http.StatusOK, forward)
|
||||
wantReputation(t, line)
|
||||
wantPercent(t, "limit_percent", line.LimitPercent, line.LimitPercentSetting,
|
||||
none)
|
||||
}
|
||||
|
||||
// A refusal for the verdict makes no ban, and every client had its
|
||||
// verdicts, so no zone was asked.
|
||||
if held := server.Ledger.Snapshot(); tc.action == actionDeny && len(held) != 0 {
|
||||
t.Errorf("bans %+v, want none", held)
|
||||
}
|
||||
|
||||
if queries := server.DNSBL.Queries(dnsblZone); queries != 0 {
|
||||
t.Errorf("%d queries, want none", queries)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestDNSBLZonesComeAfterTheBlocklistsAndSkipAllowNets(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
s, server, queue := startWithLookups(t, map[string]string{
|
||||
blocklistURLs: dropURL, dnsblZones: dnsblZone, dnsblResolver: noResolver,
|
||||
reputationAction: actionDeny, allowNets: fromDE,
|
||||
})
|
||||
loadLists(t, server, map[string][]string{dropURL: {fromKP}})
|
||||
loadVerdicts(server, map[string][]string{fromKP: {dnsblZone}, fromDE: {dnsblZone}})
|
||||
|
||||
// The blocklist refuses fromKP before its verdict is looked at, and
|
||||
// fromDE, in SWWAF_ALLOW_NETS, is not checked at all: neither is noted
|
||||
// for the zone, nor alerted, nor asked about.
|
||||
wantReputation(t, s.get(fromKP, http.StatusForbidden, requestlog.ActionDenied),
|
||||
dropURL)
|
||||
wantReputation(t, s.get(fromDE, http.StatusOK, requestlog.ActionForward))
|
||||
|
||||
waiting := queue.Snapshot().Waiting[alerts.DestinationWebhook]
|
||||
if len(waiting) != 1 || waiting[0].Detail["source"] != dropURL {
|
||||
t.Errorf("alerts waiting %+v, want the blocklist's reputation_hit alone", waiting)
|
||||
}
|
||||
|
||||
if queries := server.DNSBL.Queries(dnsblZone); queries != 0 {
|
||||
t.Errorf("%d queries, want none", queries)
|
||||
}
|
||||
}
|
||||
|
||||
func TestObserveModeForwardsAClientADNSBLZoneDeniesAndAlertsIt(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
s, server, queue := startWithLookups(t, map[string]string{
|
||||
dnsblZones: dnsblZone, dnsblResolver: noResolver, reputationAction: actionDeny,
|
||||
mode: observe,
|
||||
})
|
||||
loadVerdicts(server, map[string][]string{fromDE: {dnsblZone}})
|
||||
|
||||
line := s.get(fromDE, http.StatusOK, requestlog.ActionForward)
|
||||
wantWouldAction(t, line, requestlog.ActionDenied)
|
||||
wantReputation(t, line, dnsblZone)
|
||||
|
||||
waiting := queue.Snapshot().Waiting[alerts.DestinationWebhook]
|
||||
if len(waiting) != 1 || waiting[0].Event != alerts.EventReputationHit {
|
||||
t.Errorf("alerts waiting %+v, want a reputation_hit alert", waiting)
|
||||
}
|
||||
}
|
||||
|
||||
func TestReputationLimitTakesPartInTheLowestPercentageOfEveryLimit(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
for _, tc := range []struct {
|
||||
blocklistAction, reputationAction string
|
||||
// want is the request's limit_percent and bytes_percent, as
|
||||
// percentText gives them.
|
||||
want string
|
||||
}{
|
||||
{limitHalf, limitQuarter, "25 from " + reputationAction},
|
||||
{limitQuarter, limitHalf, "25 from " + blocklistAction},
|
||||
// The blocklist's, the first of two alike.
|
||||
{limitQuarter, limitQuarter, "25 from " + blocklistAction},
|
||||
{actionLog, limitQuarter, "25 from " + reputationAction},
|
||||
{actionLog, actionLog, none},
|
||||
} {
|
||||
t.Run(tc.blocklistAction+" "+tc.reputationAction, func(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
s, server, _ := startWithLookups(t, map[string]string{
|
||||
blocklistURLs: dropURL, blocklistAction: tc.blocklistAction,
|
||||
dnsblZones: dnsblZone, dnsblResolver: noResolver,
|
||||
reputationAction: tc.reputationAction,
|
||||
})
|
||||
loadLists(t, server, map[string][]string{dropURL: {fromDE}})
|
||||
loadVerdicts(server, map[string][]string{fromDE: {dnsblZone}})
|
||||
|
||||
// Named by the blocklist, then by the zone.
|
||||
line := s.get(fromDE, http.StatusOK, requestlog.ActionForward)
|
||||
wantReputation(t, line, dropURL, dnsblZone)
|
||||
wantPercent(t, "limit_percent", line.LimitPercent, line.LimitPercentSetting,
|
||||
tc.want)
|
||||
wantPercent(t, "bytes_percent", line.BytesPercent, line.BytesPercentSetting,
|
||||
tc.want)
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestEachZoneThatListsAClientRaisesAnAlertOncePerCooldownAndIsCounted(
|
||||
t *testing.T,
|
||||
) {
|
||||
t.Parallel()
|
||||
|
||||
s, clk, server, queue := startWithLookupsAndClock(t, map[string]string{
|
||||
dnsblZones: dnsblZone + "," + otherZone, dnsblResolver: noResolver,
|
||||
reputationAction: actionLog, metricsToken: token,
|
||||
})
|
||||
loadVerdicts(server, map[string][]string{
|
||||
fromDE: {dnsblZone, otherZone}, unplaced: nil,
|
||||
})
|
||||
|
||||
// The second request's alerts are repeats, which the cooldown holds
|
||||
// back.
|
||||
for range 2 {
|
||||
wantReputation(t, s.get(fromDE, http.StatusOK, requestlog.ActionForward),
|
||||
dnsblZone, otherZone)
|
||||
}
|
||||
|
||||
hit := func(zone string) alerts.Alert {
|
||||
return alerts.Alert{
|
||||
Instance: alertInstance,
|
||||
Time: clk.Now(),
|
||||
Event: alerts.EventReputationHit,
|
||||
Client: netip.MustParseAddr(fromDE),
|
||||
Netblock: netip.MustParsePrefix(fromDE + "/32"),
|
||||
ASN: asnDE,
|
||||
ASName: asNameDE,
|
||||
Country: "DE",
|
||||
Reason: "listed by a DNSBL zone",
|
||||
Detail: map[string]any{"source": zone},
|
||||
}
|
||||
}
|
||||
wantAlerts(t, queue, hit(dnsblZone), hit(otherZone))
|
||||
|
||||
if queue.Suppressed() != 2 {
|
||||
t.Errorf("%d alerts held back, want the second request's 2", queue.Suppressed())
|
||||
}
|
||||
|
||||
// Each zone's hits, and its queries and their failures, none, since
|
||||
// every client had its verdicts.
|
||||
metrics := s.scrape(unplaced)
|
||||
|
||||
for _, zone := range []string{dnsblZone, otherZone} {
|
||||
labels := `{instance="` + alertInstance + `",source="` + zone + `"}`
|
||||
|
||||
wantMetric(t, metrics, "smallwebwaf_reputation_hits_total"+labels, 2)
|
||||
wantMetric(t, metrics, "smallwebwaf_reputation_queries_total"+labels, 0)
|
||||
wantMetric(t, metrics, "smallwebwaf_reputation_failures_total"+labels, 0)
|
||||
}
|
||||
}
|
||||
|
||||
func TestRequestFromAClientWithoutAVerdictHasTheZoneAskedAboutIt(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
s, server, _ := startWithLookups(t, map[string]string{
|
||||
dnsblZones: dnsblZone + "," + otherZone, dnsblResolver: noResolver,
|
||||
})
|
||||
server.DNSBL.Load([]reputation.Verdict{{
|
||||
Zone: otherZone, Client: netip.MustParseAddr(fromDE), Listed: true,
|
||||
Fetched: verdictsFetched(),
|
||||
}})
|
||||
|
||||
// The verdict of the other zone is used, and dnsbl.example, which has
|
||||
// none, is asked about the client in the background, once: the second
|
||||
// request finds the query under way, or the zone left alone after it
|
||||
// failed, since nothing answers at noResolver.
|
||||
for range 2 {
|
||||
wantReputation(t, s.get(fromDE, http.StatusOK, requestlog.ActionForward), otherZone)
|
||||
}
|
||||
|
||||
if queries := server.DNSBL.Queries(dnsblZone); queries != 1 {
|
||||
t.Errorf("%d queries to %s, want 1", queries, dnsblZone)
|
||||
}
|
||||
|
||||
if queries := server.DNSBL.Queries(otherZone); queries != 0 {
|
||||
t.Errorf("%d queries to %s, want none", queries, otherZone)
|
||||
}
|
||||
}
|
||||
|
||||
// listsFetched is when loadLists has the copies fetched.
|
||||
func listsFetched() time.Time {
|
||||
return time.Date(2026, 10, 5, 0, 0, 0, 0, time.UTC)
|
||||
}
|
||||
|
||||
// verdictsFetched is when loadVerdicts has the verdicts fetched: half a
|
||||
// day before the time the tests' clock is set to, so that they are in use
|
||||
// until half a day later.
|
||||
func verdictsFetched() time.Time {
|
||||
return time.Date(2026, 10, 5, 12, 0, 0, 0, time.UTC)
|
||||
}
|
||||
|
||||
// loadVerdicts puts into server's DNSBL, for each client listedBy names,
|
||||
// a verdict of each zone SWWAF_DNSBL_ZONES names, fetched at
|
||||
// verdictsFetched, as reputation.json would at start: one that lists the
|
||||
// client from each zone listedBy gives for it, and one that does not from
|
||||
// each other zone.
|
||||
func loadVerdicts(server *proxy.Server, listedBy map[string][]string) {
|
||||
verdicts := make([]reputation.Verdict, 0, len(listedBy)*len(server.DNSBL.Zones()))
|
||||
|
||||
for client, zones := range listedBy {
|
||||
for _, zone := range server.DNSBL.Zones() {
|
||||
verdicts = append(verdicts, reputation.Verdict{
|
||||
Zone: zone, Client: netip.MustParseAddr(client),
|
||||
Listed: slices.Contains(zones, zone), Fetched: verdictsFetched(),
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
server.DNSBL.Load(verdicts)
|
||||
}
|
||||
|
||||
// loadLists puts copies of lists into server's lists, by URL, each with
|
||||
// its lines, fetched at listsFetched, as reputation.json would at start.
|
||||
func loadLists(t *testing.T, server *proxy.Server, copies map[string][]string) {
|
||||
|
||||
@@ -63,7 +63,10 @@ type request struct {
|
||||
// limits and for the byte limits.
|
||||
counted bool
|
||||
limitPercent, bytesPercent percentage
|
||||
start time.Time
|
||||
// blocklisted is true once a blocklist is found to list the client,
|
||||
// and dnsblListed once a DNSBL zone's verdict is.
|
||||
blocklisted, dnsblListed bool
|
||||
start time.Time
|
||||
// checked is when the checks were done, and upstreamStart when the
|
||||
// request was handed to the app.
|
||||
checked time.Time
|
||||
@@ -213,14 +216,14 @@ func (rq *request) check(ctx context.Context) *refusal {
|
||||
// client in SWWAF_ALLOW_NETS skips them, and is not looked up. For any
|
||||
// other client, SWWAF_DENY_NETS comes first, then a ban on its netblock,
|
||||
// so that a client either refuses is not looked up, then the lookup of
|
||||
// its AS number and country, then the country lists, and then the
|
||||
// blocklists; a request any of them refuses is not counted for the rate
|
||||
// limits. Then come the rate limits, unless the client is in
|
||||
// SWWAF_RATE_LIMIT_EXEMPT_NETS or the request's path is exempt under
|
||||
// SWWAF_RATE_LIMIT_EXEMPT_PATHS, so that every other request is counted,
|
||||
// each of them by the client's limit percentages, and last the rule
|
||||
// files. A request exempt from the rate limits is exempt from the byte
|
||||
// limits too. ctx is the request's own context.
|
||||
// its AS number and country, then the country lists, then the blocklists,
|
||||
// and then the DNSBL zones' verdicts; a request any of them refuses is not
|
||||
// counted for the rate limits. Then come the rate limits, unless the
|
||||
// client is in SWWAF_RATE_LIMIT_EXEMPT_NETS or the request's path is
|
||||
// exempt under SWWAF_RATE_LIMIT_EXEMPT_PATHS, so that every other request
|
||||
// is counted, each of them by the client's limit percentages, and last the
|
||||
// rule files. A request exempt from the rate limits is exempt from the
|
||||
// byte limits too. ctx is the request's own context.
|
||||
func (rq *request) checkClient(ctx context.Context) string {
|
||||
cfg := rq.h.config
|
||||
if isInside(rq.client, cfg.AllowNets) {
|
||||
@@ -247,6 +250,10 @@ func (rq *request) checkClient(ctx context.Context) string {
|
||||
return requestlog.ActionDenied
|
||||
}
|
||||
|
||||
if rq.dnsblDenied(ctx) {
|
||||
return requestlog.ActionDenied
|
||||
}
|
||||
|
||||
rq.counted = !isInside(rq.client, cfg.RateLimitExemptNets) &&
|
||||
!pathExempt(rq.in.URL, cfg.RateLimitExemptPaths)
|
||||
if rq.counted {
|
||||
|
||||
Reference in New Issue
Block a user