Compare commits
1
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
4bc8d9edb4 |
@@ -30,7 +30,7 @@ unusual traffic that refuse nothing. So are the first three parts of the stage
|
|||||||
after that: the blocklists you name by URL, which it fetches and keeps, with a
|
after that: the blocklists you name by URL, which it fetches and keeps, with a
|
||||||
file of AS numbers' percentages fetched the same way, the DNS blocklists (DNSBL
|
file of AS numbers' percentages fetched the same way, the DNS blocklists (DNSBL
|
||||||
zones), which it asks about each client in the background, and AbuseIPDB, which
|
zones), which it asks about each client in the background, and AbuseIPDB, which
|
||||||
it asks in the background about each client that has committed an offence.
|
it asks in the background about each client that has broken a limit.
|
||||||
`smallwebwaf` passes each request to the app and the app's answer back,
|
`smallwebwaf` passes each request to the app and the app's answer back,
|
||||||
unchanged, within its timeouts and size limits, works out each client's address,
|
unchanged, within its timeouts and size limits, works out each client's address,
|
||||||
looks up its AS number and country unless you switch that off, bans a client
|
looks up its AS number and country unless you switch that off, bans a client
|
||||||
@@ -225,16 +225,15 @@ in `bin/state` unless `SWWAF_STATE_DIR` is set, and the default rule file of
|
|||||||
make no ban. With `log`, nothing more is done. Whatever the action, the
|
make no ban. With `log`, nothing more is done. Whatever the action, the
|
||||||
request's log line names the zones, and each raises an alert. A client a
|
request's log line names the zones, and each raises an alert. A client a
|
||||||
blocklist refuses is not checked.
|
blocklist refuses is not checked.
|
||||||
- Checks the client with AbuseIPDB while `SWWAF_ABUSEIPDB_KEY` is set, after the
|
- Checks the client's own address with AbuseIPDB while `SWWAF_ABUSEIPDB_KEY` is
|
||||||
DNSBL zones and before the rate limits (see "AbuseIPDB" below), by the scores
|
set, after the DNSBL zones and before the rate limits (see "AbuseIPDB" below),
|
||||||
it keeps. Only a client whose history counts an offence is checked, so far one
|
by the scores it keeps. Only a client whose history counts an offence is
|
||||||
that has broken a rate limit or a byte limit, matched a ban rule, or had a
|
checked, so far one that has broken a rate limit or a byte limit, and only in
|
||||||
request refused by a block rule, and only in the background, so that no
|
the background, so that no request waits for AbuseIPDB. A score at or over
|
||||||
request waits for AbuseIPDB. A score at or over `SWWAF_ABUSEIPDB_MIN_SCORE` is
|
`SWWAF_ABUSEIPDB_MIN_SCORE` is a hit, and `SWWAF_REPUTATION_ACTION` does with
|
||||||
a hit, and `SWWAF_REPUTATION_ACTION` does with its client what it does with
|
its client what it does with one a DNSBL zone's verdict lists. The request's
|
||||||
one a DNSBL zone's verdict lists. The request's log line names AbuseIPDB, and
|
log line names AbuseIPDB, and it raises an alert. A client a blocklist or a
|
||||||
it raises an alert. A client a blocklist or a DNSBL zone refuses is not
|
DNSBL zone refuses is not checked.
|
||||||
checked.
|
|
||||||
- Checks the client's own address against the static lists, the three netblock
|
- Checks the client's own address against the static lists, the three netblock
|
||||||
settings below, before anything else, its lookup included. A client in
|
settings below, before anything else, its lookup included. A client in
|
||||||
`SWWAF_ALLOW_NETS` skips bans, the country lists, the blocklists, the DNSBL
|
`SWWAF_ALLOW_NETS` skips bans, the country lists, the blocklists, the DNSBL
|
||||||
@@ -1078,9 +1077,9 @@ with times in UTC.
|
|||||||
zone gave it, `fetched`; and under `abuseipdb` (see "AbuseIPDB" below), the
|
zone gave it, `fetched`; and under `abuseipdb` (see "AbuseIPDB" below), the
|
||||||
`day`, in UTC, whose checks it counts, left out before the first, the checks
|
`day`, in UTC, whose checks it counts, left out before the first, the checks
|
||||||
`spent` that day, and under `scores`, each score of AbuseIPDB still in use:
|
`spent` that day, and under `scores`, each score of AbuseIPDB still in use:
|
||||||
the `client`, its IPv4 address as a /32 or its IPv6 /64, its `score`, and when
|
the `client`'s address, its `score`, and when AbuseIPDB gave it, `fetched`. As
|
||||||
AbuseIPDB gave it, `fetched`. As the file is read, the lists the settings no
|
the file is read, the lists the settings no longer name, and the verdicts of
|
||||||
longer name, and the verdicts of the zones they no longer name, are dropped.
|
the zones they no longer name, are dropped.
|
||||||
- `alerts.json`: the state of the alerts (see "Alerts" above), indented to be
|
- `alerts.json`: the state of the alerts (see "Alerts" above), indented to be
|
||||||
read: under `cooldowns`, for each event and netblock, with the `source` too
|
read: under `cooldowns`, for each event and netblock, with the `source` too
|
||||||
for a `reputation_hit`, or event and `file` or `source`, or for an `anomaly`,
|
for a `reputation_hit`, or event and `file` or `source`, or for an `anomaly`,
|
||||||
@@ -1821,19 +1820,15 @@ While `SWWAF_ABUSEIPDB_KEY` holds the key of an AbuseIPDB account, `smallwebwaf`
|
|||||||
asks AbuseIPDB's check endpoint, `https://api.abuseipdb.com/api/v2/check`, for
|
asks AbuseIPDB's check endpoint, `https://api.abuseipdb.com/api/v2/check`, for
|
||||||
the abuse confidence score of a client's own address, from 0 to 100. It is unset
|
the abuse confidence score of a client's own address, from 0 to 100. It is unset
|
||||||
by default, for the reason no blocklist is named, and since AbuseIPDB needs an
|
by default, for the reason no blocklist is named, and since AbuseIPDB needs an
|
||||||
account. An IPv6 client, a /64, is checked by the address of the request that
|
account. An IPv6 client is checked by its own address, not by its /64.
|
||||||
has it checked, and its score is used for the whole /64, whichever of its
|
|
||||||
addresses sends, so that one client costs at most one check every
|
|
||||||
`SWWAF_REPUTATION_CACHE_TTL`.
|
|
||||||
|
|
||||||
Only a client whose history counts an offence is checked, so that the checks are
|
Only a client whose history counts an offence is checked, so that the checks are
|
||||||
spent on suspects: so far, one that has broken a rate limit or a byte limit,
|
spent on suspects: so far, one that has broken a rate limit or a byte limit. A
|
||||||
matched a ban rule, or had a request refused by a block rule. A client dropped
|
client dropped from the table of clients loses its history, and with it its
|
||||||
from the table of clients loses its history, and with it its offences. A client
|
offences. A client is checked in the background, at its first request after its
|
||||||
is checked in the background, at its first request after its offence that
|
offence that reaches the check: no request waits, a request refused under its
|
||||||
reaches the check: no request waits, a request refused under its ban is not
|
ban is not checked, and the request that has it checked, and any other from it
|
||||||
checked, and the request that has it checked, and any other from it before the
|
before the answer comes, goes on as from a client without a score.
|
||||||
answer comes, goes on as from a client without a score.
|
|
||||||
|
|
||||||
A score at or over `SWWAF_ABUSEIPDB_MIN_SCORE`, 75 by default, is a hit, and
|
A score at or over `SWWAF_ABUSEIPDB_MIN_SCORE`, 75 by default, is a hit, and
|
||||||
`SWWAF_REPUTATION_ACTION` says what is done with its client, as for a DNSBL
|
`SWWAF_REPUTATION_ACTION` says what is done with its client, as for a DNSBL
|
||||||
@@ -1906,8 +1901,8 @@ given as files" above).
|
|||||||
file gives an AS number; asks the DNSBL zones about clients in the background,
|
file gives an AS number; asks the DNSBL zones about clients in the background,
|
||||||
through the standard library's resolver, keeps their verdicts, and tells which
|
through the standard library's resolver, keeps their verdicts, and tells which
|
||||||
zones' verdicts list an address; and checks clients with AbuseIPDB in the
|
zones' verdicts list an address; and checks clients with AbuseIPDB in the
|
||||||
background, keeps their scores and the checks spent today, and tells whether a
|
background, keeps their scores and the checks spent today, and tells whether
|
||||||
client's score is a hit.
|
an address's score is a hit.
|
||||||
- `internal/ratelimit`: the table of clients: counts each client's requests and
|
- `internal/ratelimit`: the table of clients: counts each client's requests and
|
||||||
bytes, tells when they take it over a rate limit or a byte limit, and keeps
|
bytes, tells when they take it over a rate limit or a byte limit, and keeps
|
||||||
each client's history.
|
each client's history.
|
||||||
|
|||||||
@@ -43,20 +43,18 @@ func (rq *request) dnsblDenied(ctx context.Context) bool {
|
|||||||
// its score of the client is a hit, and reports whether
|
// its score of the client is a hit, and reports whether
|
||||||
// SWWAF_REPUTATION_ACTION, being deny, refuses the request, as dnsblDenied
|
// SWWAF_REPUTATION_ACTION, being deny, refuses the request, as dnsblDenied
|
||||||
// does for a zone. While SWWAF_ABUSEIPDB_KEY is unset it does nothing. A
|
// does for a zone. While SWWAF_ABUSEIPDB_KEY is unset it does nothing. A
|
||||||
// client without a score is checked in the background, by the request's
|
// client without a score is checked in the background if its history
|
||||||
// address, if its history counts an offence, and the request does not
|
// counts an offence, and the request does not wait for the answer. ctx is
|
||||||
// wait for the answer. The score is then used for each address of the
|
// the request's own context.
|
||||||
// client. ctx is the request's own context.
|
|
||||||
func (rq *request) abuseIPDBDenied(ctx context.Context) bool {
|
func (rq *request) abuseIPDBDenied(ctx context.Context) bool {
|
||||||
if rq.h.config.AbuseIPDBKey == "" {
|
if rq.h.config.AbuseIPDBKey == "" {
|
||||||
return false
|
return false
|
||||||
}
|
}
|
||||||
|
|
||||||
client := clientGroup(rq.client)
|
held, _ := rq.h.limiter.Client(clientGroup(rq.client))
|
||||||
held, _ := rq.h.limiter.Client(client)
|
|
||||||
offender := held.History.Offences != ratelimit.Offences{}
|
offender := held.History.Offences != ratelimit.Offences{}
|
||||||
|
|
||||||
score, hit := rq.h.abuseIPDB.Hit(ctx, client, rq.client, offender)
|
score, hit := rq.h.abuseIPDB.Hit(ctx, rq.client, offender)
|
||||||
if !hit {
|
if !hit {
|
||||||
return false
|
return false
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -1,20 +1,17 @@
|
|||||||
package proxy_test
|
package proxy_test
|
||||||
|
|
||||||
import (
|
import (
|
||||||
"fmt"
|
|
||||||
"io"
|
"io"
|
||||||
"maps"
|
"maps"
|
||||||
"net/http"
|
"net/http"
|
||||||
"net/netip"
|
"net/netip"
|
||||||
"slices"
|
"slices"
|
||||||
"strconv"
|
|
||||||
"strings"
|
"strings"
|
||||||
"testing"
|
"testing"
|
||||||
"time"
|
"time"
|
||||||
|
|
||||||
"sneak.berlin/go/smallwebwaf/internal/alerts"
|
"sneak.berlin/go/smallwebwaf/internal/alerts"
|
||||||
"sneak.berlin/go/smallwebwaf/internal/proxy"
|
"sneak.berlin/go/smallwebwaf/internal/proxy"
|
||||||
"sneak.berlin/go/smallwebwaf/internal/ratelimit"
|
|
||||||
"sneak.berlin/go/smallwebwaf/internal/reputation"
|
"sneak.berlin/go/smallwebwaf/internal/reputation"
|
||||||
"sneak.berlin/go/smallwebwaf/internal/requestlog"
|
"sneak.berlin/go/smallwebwaf/internal/requestlog"
|
||||||
)
|
)
|
||||||
@@ -664,87 +661,6 @@ func TestOnlyAClientThatHasCommittedAnOffenceIsCheckedWithAbuseIPDB(t *testing.T
|
|||||||
wantAbuseIPDBChecks(t, server, 1)
|
wantAbuseIPDBChecks(t, server, 1)
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestIPv6ClientCostsOneAbuseIPDBCheckWhicheverOfItsAddressesSends(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
forward := requestlog.ActionForward
|
|
||||||
|
|
||||||
// 15 addresses of 2001:db8:1:2::/64, one client, each in a part of it
|
|
||||||
// of its own.
|
|
||||||
var addresses []string
|
|
||||||
for i := 1; i < 16; i++ {
|
|
||||||
addresses = append(addresses, fmt.Sprintf("2001:db8:1:2:%x::9", i<<12))
|
|
||||||
}
|
|
||||||
|
|
||||||
s, clk, server := startWithClock(t, "", map[string]string{
|
|
||||||
abuseIPDBKey: accountKey, reputationAction: actionLog,
|
|
||||||
rateLimitPerMinute: strconv.Itoa(len(addresses)),
|
|
||||||
})
|
|
||||||
|
|
||||||
// The client breaks the rate limit from its first address, which bans
|
|
||||||
// it for an hour.
|
|
||||||
for range addresses {
|
|
||||||
s.get(addresses[0], http.StatusOK, forward)
|
|
||||||
}
|
|
||||||
|
|
||||||
s.get(addresses[0], http.StatusForbidden, requestlog.ActionRateLimited)
|
|
||||||
clk.advance(time.Hour)
|
|
||||||
|
|
||||||
// Once the ban has ended, which set its counters back to zero, a
|
|
||||||
// request from each of its addresses has it checked once.
|
|
||||||
for _, address := range addresses {
|
|
||||||
s.get(address, http.StatusOK, forward)
|
|
||||||
}
|
|
||||||
|
|
||||||
wantAbuseIPDBChecks(t, server, 1)
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestClientARuleRefusedIsCheckedWithAbuseIPDBAtItsNextRequest(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
for _, tc := range []struct {
|
|
||||||
name string
|
|
||||||
// path is what the client asks for, status and action what that
|
|
||||||
// request is answered and logged with, and want the offences its
|
|
||||||
// history then counts.
|
|
||||||
path string
|
|
||||||
status int
|
|
||||||
action string
|
|
||||||
want ratelimit.Offences
|
|
||||||
}{
|
|
||||||
{
|
|
||||||
"a block rule", "/blocked", http.StatusForbidden, requestlog.ActionRuleBlocked,
|
|
||||||
ratelimit.Offences{RuleBlocked: 1},
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"a ban rule", "/.env", http.StatusForbidden, requestlog.ActionBanned,
|
|
||||||
ratelimit.Offences{Attack: 1},
|
|
||||||
},
|
|
||||||
} {
|
|
||||||
t.Run(tc.name, func(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
s, clk, server := startWithClock(t, "", map[string]string{
|
|
||||||
abuseIPDBKey: accountKey, reputationAction: actionLog,
|
|
||||||
rulesDir: writeRules(t, testRules), attackBanDuration: "1h",
|
|
||||||
})
|
|
||||||
|
|
||||||
s.request(client, tc.path, tc.status, tc.action)
|
|
||||||
wantAbuseIPDBChecks(t, server, 0)
|
|
||||||
|
|
||||||
if got := historyOf(t, server, client).Offences; got != tc.want {
|
|
||||||
t.Errorf("history counts the offences %+v, want %+v", got, tc.want)
|
|
||||||
}
|
|
||||||
|
|
||||||
// Its next request, once a ban rule's ban has ended, has it
|
|
||||||
// checked.
|
|
||||||
clk.advance(time.Hour)
|
|
||||||
s.get(client, http.StatusOK, requestlog.ActionForward)
|
|
||||||
wantAbuseIPDBChecks(t, server, 1)
|
|
||||||
})
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestEachReputationActionForAClientAbuseIPDBScoresAtOrOverTheMinimum(t *testing.T) {
|
func TestEachReputationActionForAClientAbuseIPDBScoresAtOrOverTheMinimum(t *testing.T) {
|
||||||
t.Parallel()
|
t.Parallel()
|
||||||
|
|
||||||
@@ -904,14 +820,12 @@ func loadVerdicts(server *proxy.Server, listedBy map[string][]string) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
// loadScores puts into server's AbuseIPDB the score scores gives each
|
// loadScores puts into server's AbuseIPDB the score scores gives each
|
||||||
// client, an IPv4 address, fetched at verdictsFetched, as reputation.json
|
// client, fetched at verdictsFetched, as reputation.json would at start.
|
||||||
// would at start.
|
|
||||||
func loadScores(server *proxy.Server, scores map[string]int64) {
|
func loadScores(server *proxy.Server, scores map[string]int64) {
|
||||||
kept := make([]reputation.Score, 0, len(scores))
|
kept := make([]reputation.Score, 0, len(scores))
|
||||||
for client, score := range scores {
|
for client, score := range scores {
|
||||||
kept = append(kept, reputation.Score{
|
kept = append(kept, reputation.Score{
|
||||||
Client: netip.MustParsePrefix(client + "/32"), Score: score,
|
Client: netip.MustParseAddr(client), Score: score, Fetched: verdictsFetched(),
|
||||||
Fetched: verdictsFetched(),
|
|
||||||
})
|
})
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
@@ -63,10 +63,6 @@ type request struct {
|
|||||||
// limits and for the byte limits.
|
// limits and for the byte limits.
|
||||||
counted bool
|
counted bool
|
||||||
limitPercent, bytesPercent percentage
|
limitPercent, bytesPercent percentage
|
||||||
// attack is true for a request that matched a ban rule, and
|
|
||||||
// ruleBlocked for one a block rule refused, each an offence its
|
|
||||||
// client's history counts.
|
|
||||||
attack, ruleBlocked bool
|
|
||||||
// blocklisted is true once a blocklist is found to list the client,
|
// blocklisted is true once a blocklist is found to list the client,
|
||||||
// dnsblListed once a DNSBL zone's verdict is, and abuseIPDBHit once
|
// dnsblListed once a DNSBL zone's verdict is, and abuseIPDBHit once
|
||||||
// AbuseIPDB's score of it is a hit.
|
// AbuseIPDB's score of it is a hit.
|
||||||
@@ -546,8 +542,6 @@ func (rq *request) addToHistory() {
|
|||||||
RequestBytes: rq.requestBytes(),
|
RequestBytes: rq.requestBytes(),
|
||||||
ResponseBytes: rq.out.bytes,
|
ResponseBytes: rq.out.bytes,
|
||||||
BrokeLimit: rq.line.Offence == requestlog.OffenceLimit,
|
BrokeLimit: rq.line.Offence == requestlog.OffenceLimit,
|
||||||
Attack: rq.attack,
|
|
||||||
RuleBlocked: rq.ruleBlocked,
|
|
||||||
})
|
})
|
||||||
|
|
||||||
answer, found := rq.answerAtTheEnd()
|
answer, found := rq.answerAtTheEnd()
|
||||||
|
|||||||
@@ -12,8 +12,7 @@ import (
|
|||||||
// action of the rule that refuses it, ActionRuleBlocked for a block rule
|
// action of the rule that refuses it, ActionRuleBlocked for a block rule
|
||||||
// and ActionBanned for a ban rule, or "" when none does. A ban rule bans
|
// and ActionBanned for a ban rule, or "" when none does. A ban rule bans
|
||||||
// the client's netblock for a clear sign of attack, or in observe mode
|
// the client's netblock for a clear sign of attack, or in observe mode
|
||||||
// raises the alert for the ban it would have made. Either rule's match
|
// raises the alert for the ban it would have made.
|
||||||
// is noted as an offence, for the client's history.
|
|
||||||
func (rq *request) checkRules(now time.Time) string {
|
func (rq *request) checkRules(now time.Time) string {
|
||||||
matched := rq.h.rules.Match(rq.in)
|
matched := rq.h.rules.Match(rq.in)
|
||||||
|
|
||||||
@@ -29,11 +28,8 @@ func (rq *request) checkRules(now time.Time) string {
|
|||||||
// Only the last rule matched can refuse the request.
|
// Only the last rule matched can refuse the request.
|
||||||
switch last := matched[len(matched)-1]; last.Action {
|
switch last := matched[len(matched)-1]; last.Action {
|
||||||
case rules.ActionBlock:
|
case rules.ActionBlock:
|
||||||
rq.ruleBlocked = true
|
|
||||||
|
|
||||||
return requestlog.ActionRuleBlocked
|
return requestlog.ActionRuleBlocked
|
||||||
case rules.ActionBan:
|
case rules.ActionBan:
|
||||||
rq.attack = true
|
|
||||||
rq.banForAttack(now, last)
|
rq.banForAttack(now, last)
|
||||||
|
|
||||||
return requestlog.ActionBanned
|
return requestlog.ActionBanned
|
||||||
|
|||||||
@@ -118,12 +118,8 @@ type Responses struct {
|
|||||||
|
|
||||||
// Offences are a client's offences, by kind.
|
// Offences are a client's offences, by kind.
|
||||||
type Offences struct {
|
type Offences struct {
|
||||||
// Limit is its requests that broke a rate limit or a byte limit,
|
// Limit is its requests that broke a rate limit or a byte limit.
|
||||||
// Attack those that matched a ban rule, a clear sign of attack, and
|
Limit int64 `json:"limit"`
|
||||||
// RuleBlocked those a block rule refused.
|
|
||||||
Limit int64 `json:"limit"`
|
|
||||||
Attack int64 `json:"attack"`
|
|
||||||
RuleBlocked int64 `json:"rule_blocked"`
|
|
||||||
}
|
}
|
||||||
|
|
||||||
// Request is what a client's history keeps of one of its requests.
|
// Request is what a client's history keeps of one of its requests.
|
||||||
@@ -141,11 +137,8 @@ type Request struct {
|
|||||||
RequestBytes int64
|
RequestBytes int64
|
||||||
ResponseBytes int64
|
ResponseBytes int64
|
||||||
// BrokeLimit is true for a request that broke a rate limit or a byte
|
// BrokeLimit is true for a request that broke a rate limit or a byte
|
||||||
// limit, Attack for one that matched a ban rule, and RuleBlocked for
|
// limit.
|
||||||
// one a block rule refused.
|
BrokeLimit bool
|
||||||
BrokeLimit bool
|
|
||||||
Attack bool
|
|
||||||
RuleBlocked bool
|
|
||||||
}
|
}
|
||||||
|
|
||||||
// New returns a Limiter for limits, with no client counted yet.
|
// New returns a Limiter for limits, with no client counted yet.
|
||||||
@@ -265,14 +258,6 @@ func (l *Limiter) AddToHistory(client netip.Prefix, now time.Time, r Request) {
|
|||||||
if r.BrokeLimit {
|
if r.BrokeLimit {
|
||||||
h.Offences.Limit++
|
h.Offences.Limit++
|
||||||
}
|
}
|
||||||
|
|
||||||
if r.Attack {
|
|
||||||
h.Offences.Attack++
|
|
||||||
}
|
|
||||||
|
|
||||||
if r.RuleBlocked {
|
|
||||||
h.Offences.RuleBlocked++
|
|
||||||
}
|
|
||||||
}
|
}
|
||||||
|
|
||||||
// AddLookup gives client's history its AS number, AS name and country, as
|
// AddLookup gives client's history its AS number, AS name and country, as
|
||||||
|
|||||||
@@ -38,12 +38,12 @@ var (
|
|||||||
)
|
)
|
||||||
|
|
||||||
// Score is what AbuseIPDB said about a client, as reputation.json holds
|
// Score is what AbuseIPDB said about a client, as reputation.json holds
|
||||||
// it: the client, an IPv4 address or an IPv6 group, its abuse confidence
|
// it: the client's address, its abuse confidence score, from 0 to 100,
|
||||||
// score, from 0 to 100, and when AbuseIPDB answered.
|
// and when AbuseIPDB answered.
|
||||||
type Score struct {
|
type Score struct {
|
||||||
Client netip.Prefix `json:"client"`
|
Client netip.Addr `json:"client"`
|
||||||
Score int64 `json:"score"`
|
Score int64 `json:"score"`
|
||||||
Fetched time.Time `json:"fetched"`
|
Fetched time.Time `json:"fetched"`
|
||||||
}
|
}
|
||||||
|
|
||||||
// Checks are what reputation.json keeps of the checks of clients with
|
// Checks are what reputation.json keeps of the checks of clients with
|
||||||
@@ -89,9 +89,9 @@ type AbuseIPDB struct {
|
|||||||
mu sync.Mutex
|
mu sync.Mutex
|
||||||
// scores are by client. Each is added as it is fetched and never moved
|
// scores are by client. Each is added as it is fetched and never moved
|
||||||
// up, so that the one fetched longest ago is the first dropped.
|
// up, so that the one fetched longest ago is the first dropped.
|
||||||
scores *simplelru.LRU[netip.Prefix, Score]
|
scores *simplelru.LRU[netip.Addr, Score]
|
||||||
// checking are the clients whose check is under way.
|
// checking are the clients whose check is under way.
|
||||||
checking map[netip.Prefix]bool
|
checking map[netip.Addr]bool
|
||||||
// day is the day, in UTC, of the checks spent counts.
|
// day is the day, in UTC, of the checks spent counts.
|
||||||
day time.Time
|
day time.Time
|
||||||
spent int
|
spent int
|
||||||
@@ -105,7 +105,7 @@ type AbuseIPDB struct {
|
|||||||
|
|
||||||
// NewAbuseIPDB returns an AbuseIPDB with no score yet, and no check spent.
|
// NewAbuseIPDB returns an AbuseIPDB with no score yet, and no check spent.
|
||||||
func NewAbuseIPDB(params AbuseIPDBParams) *AbuseIPDB {
|
func NewAbuseIPDB(params AbuseIPDBParams) *AbuseIPDB {
|
||||||
scores, err := simplelru.NewLRU[netip.Prefix, Score](maxVerdicts, nil)
|
scores, err := simplelru.NewLRU[netip.Addr, Score](maxVerdicts, nil)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
panic(err) // NewLRU fails only for a size below one
|
panic(err) // NewLRU fails only for a size below one
|
||||||
}
|
}
|
||||||
@@ -114,29 +114,27 @@ func NewAbuseIPDB(params AbuseIPDBParams) *AbuseIPDB {
|
|||||||
params: params,
|
params: params,
|
||||||
httpClient: &http.Client{},
|
httpClient: &http.Client{},
|
||||||
scores: scores,
|
scores: scores,
|
||||||
checking: map[netip.Prefix]bool{},
|
checking: map[netip.Addr]bool{},
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// Hit returns AbuseIPDB's score of client, an IPv4 address or an IPv6
|
// Hit returns AbuseIPDB's score of addr, a client's address, and whether
|
||||||
// group, and whether it is a hit: MinScore or more. A score is used until
|
// it is a hit: MinScore or more. A score is used until CacheTTL has passed
|
||||||
// CacheTTL has passed since it was fetched, whichever of the client's
|
// since it was fetched. A client without one is checked in the
|
||||||
// addresses its request comes from. A client without one is checked in
|
// background if offender, if it has committed an offence, unless its
|
||||||
// the background, by addr, the address its request came from, if
|
// check is under way, a check failed less than failureDelay ago, or the
|
||||||
// offender, if it has committed an offence, unless its check is under
|
// day's checks have used up DailyBudget; Hit never waits for a check. The
|
||||||
// way, a check failed less than failureDelay ago, or the day's checks
|
// check that uses the budget up is logged and raised as a source_failure
|
||||||
// have used up DailyBudget; Hit never waits for a check. The check that
|
// alert. ctx is the context of the client's request, and a check goes on
|
||||||
// uses the budget up is logged and raised as a source_failure alert. ctx
|
// after the request ends.
|
||||||
// is the context of the client's request, and a check goes on after the
|
|
||||||
// request ends.
|
|
||||||
func (a *AbuseIPDB) Hit(
|
func (a *AbuseIPDB) Hit(
|
||||||
ctx context.Context, client netip.Prefix, addr netip.Addr, offender bool,
|
ctx context.Context, addr netip.Addr, offender bool,
|
||||||
) (int64, bool) {
|
) (int64, bool) {
|
||||||
a.mu.Lock()
|
a.mu.Lock()
|
||||||
|
|
||||||
now := a.params.Now()
|
now := a.params.Now()
|
||||||
|
|
||||||
kept, found := a.scores.Peek(client)
|
kept, found := a.scores.Peek(addr)
|
||||||
if found && now.Sub(kept.Fetched) < a.params.CacheTTL {
|
if found && now.Sub(kept.Fetched) < a.params.CacheTTL {
|
||||||
a.mu.Unlock()
|
a.mu.Unlock()
|
||||||
|
|
||||||
@@ -147,14 +145,14 @@ func (a *AbuseIPDB) Hit(
|
|||||||
a.day, a.spent = today, 0
|
a.day, a.spent = today, 0
|
||||||
}
|
}
|
||||||
|
|
||||||
check := offender && !a.checking[client] && !now.Before(a.retryAt) &&
|
check := offender && !a.checking[addr] && !now.Before(a.retryAt) &&
|
||||||
a.spent < a.params.DailyBudget
|
a.spent < a.params.DailyBudget
|
||||||
if check {
|
if check {
|
||||||
a.checking[client] = true
|
a.checking[addr] = true
|
||||||
a.checks++
|
a.checks++
|
||||||
a.spent++
|
a.spent++
|
||||||
|
|
||||||
go a.check(context.WithoutCancel(ctx), client, addr)
|
go a.check(context.WithoutCancel(ctx), addr)
|
||||||
}
|
}
|
||||||
|
|
||||||
usedUp := check && a.spent == a.params.DailyBudget
|
usedUp := check && a.spent == a.params.DailyBudget
|
||||||
@@ -241,20 +239,20 @@ func (a *AbuseIPDB) Load(checks Checks) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// check checks client with AbuseIPDB by addr, one of its addresses, keeps
|
// check checks addr with AbuseIPDB, keeps the score, and notes the check
|
||||||
// the score as client's, and notes the check as no longer under way. A
|
// as no longer under way. A check that fails gives no score: it is
|
||||||
// check that fails gives no score: it is counted, logged and raised as a
|
// counted, logged and raised as a source_failure alert, and no client is
|
||||||
// source_failure alert, and no client is checked for failureDelay.
|
// checked for failureDelay.
|
||||||
func (a *AbuseIPDB) check(ctx context.Context, client netip.Prefix, addr netip.Addr) {
|
func (a *AbuseIPDB) check(ctx context.Context, addr netip.Addr) {
|
||||||
score, err := a.ask(ctx, addr)
|
score, err := a.ask(ctx, addr)
|
||||||
now := a.params.Now()
|
now := a.params.Now()
|
||||||
|
|
||||||
a.mu.Lock()
|
a.mu.Lock()
|
||||||
|
|
||||||
delete(a.checking, client)
|
delete(a.checking, addr)
|
||||||
|
|
||||||
if err == nil {
|
if err == nil {
|
||||||
a.scores.Add(client, Score{Client: client, Score: score, Fetched: now})
|
a.scores.Add(addr, Score{Client: addr, Score: score, Fetched: now})
|
||||||
} else {
|
} else {
|
||||||
a.failures++
|
a.failures++
|
||||||
a.retryAt = now.Add(failureDelay)
|
a.retryAt = now.Add(failureDelay)
|
||||||
|
|||||||
@@ -59,38 +59,6 @@ func TestOnlyAnOffenderWithoutAScoreIsChecked(t *testing.T) {
|
|||||||
})
|
})
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestIPv6ClientIsCheckedOnceAndItsScoreUsedForEachOfItsAddresses(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
synctest.Test(t, func(t *testing.T) {
|
|
||||||
// 15 addresses of 2001:db8:1:2::/64, one client, each in a part of
|
|
||||||
// it of its own.
|
|
||||||
var addresses []string
|
|
||||||
for i := 1; i < 16; i++ {
|
|
||||||
addresses = append(addresses, fmt.Sprintf("2001:db8:1:2:%x::9", i<<12))
|
|
||||||
}
|
|
||||||
|
|
||||||
abuseIPDB := &abuseIPDBStandIn{scores: map[string]int64{addresses[0]: 100}}
|
|
||||||
checker := newAbuseIPDB(abuseIPDB, abuseIPDBParams())
|
|
||||||
|
|
||||||
// A request from each has the client checked once, by the first.
|
|
||||||
for _, address := range addresses {
|
|
||||||
hitFrom(t, checker, address, true)
|
|
||||||
}
|
|
||||||
|
|
||||||
synctest.Wait()
|
|
||||||
wantChecked(t, abuseIPDB, addresses[0])
|
|
||||||
|
|
||||||
// Its score is the whole client's.
|
|
||||||
for _, address := range addresses {
|
|
||||||
wantScore(t, checker, address, true, 100, true)
|
|
||||||
}
|
|
||||||
|
|
||||||
synctest.Wait()
|
|
||||||
wantChecked(t, abuseIPDB, addresses[0])
|
|
||||||
})
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestScoreAtOrOverTheMinimumIsAHit(t *testing.T) {
|
func TestScoreAtOrOverTheMinimumIsAHit(t *testing.T) {
|
||||||
t.Parallel()
|
t.Parallel()
|
||||||
|
|
||||||
@@ -101,7 +69,7 @@ func TestScoreAtOrOverTheMinimumIsAHit(t *testing.T) {
|
|||||||
checker := newAbuseIPDB(&abuseIPDBStandIn{scores: scores}, p)
|
checker := newAbuseIPDB(&abuseIPDBStandIn{scores: scores}, p)
|
||||||
|
|
||||||
for client := range scores {
|
for client := range scores {
|
||||||
hitFrom(t, checker, client, true)
|
checker.Hit(t.Context(), netip.MustParseAddr(client), true)
|
||||||
}
|
}
|
||||||
|
|
||||||
synctest.Wait()
|
synctest.Wait()
|
||||||
@@ -119,7 +87,7 @@ func TestScoreUsedUntilTheCacheTTLHasPassedSinceItWasFetched(t *testing.T) {
|
|||||||
abuseIPDB := &abuseIPDBStandIn{scores: map[string]int64{suspect: 100}}
|
abuseIPDB := &abuseIPDBStandIn{scores: map[string]int64{suspect: 100}}
|
||||||
checker := newAbuseIPDB(abuseIPDB, abuseIPDBParams())
|
checker := newAbuseIPDB(abuseIPDB, abuseIPDBParams())
|
||||||
|
|
||||||
hitFrom(t, checker, suspect, true)
|
checker.Hit(t.Context(), netip.MustParseAddr(suspect), true)
|
||||||
synctest.Wait()
|
synctest.Wait()
|
||||||
|
|
||||||
// AbuseIPDB gives another score from now on, but the one kept is
|
// AbuseIPDB gives another score from now on, but the one kept is
|
||||||
@@ -162,7 +130,7 @@ func TestDailyBudgetKeptAcrossARestartAndWholeAgainAsTheDayEnds(t *testing.T) {
|
|||||||
|
|
||||||
clients := []string{suspect, "192.0.2.2", "192.0.2.3", unchecked}
|
clients := []string{suspect, "192.0.2.2", "192.0.2.3", unchecked}
|
||||||
for _, client := range clients {
|
for _, client := range clients {
|
||||||
hitFrom(t, checker, client, true)
|
checker.Hit(t.Context(), netip.MustParseAddr(client), true)
|
||||||
}
|
}
|
||||||
|
|
||||||
synctest.Wait()
|
synctest.Wait()
|
||||||
@@ -384,10 +352,10 @@ func TestMetricsCountTheChecksTheFailuresAndTheBudgetLeft(t *testing.T) {
|
|||||||
m.AddAbuseIPDB(checker)
|
m.AddAbuseIPDB(checker)
|
||||||
|
|
||||||
// One check that AbuseIPDB answers, and one that fails.
|
// One check that AbuseIPDB answers, and one that fails.
|
||||||
hitFrom(t, checker, suspect, true)
|
checker.Hit(t.Context(), netip.MustParseAddr(suspect), true)
|
||||||
synctest.Wait()
|
synctest.Wait()
|
||||||
abuseIPDB.answerWith(http.StatusInternalServerError, "")
|
abuseIPDB.answerWith(http.StatusInternalServerError, "")
|
||||||
hitFrom(t, checker, other, true)
|
checker.Hit(t.Context(), netip.MustParseAddr(other), true)
|
||||||
synctest.Wait()
|
synctest.Wait()
|
||||||
|
|
||||||
scraped := scrapeMetrics(t, m)
|
scraped := scrapeMetrics(t, m)
|
||||||
@@ -414,15 +382,13 @@ func TestScoreFetchedATTLAgoIsNeitherUsedNorKept(t *testing.T) {
|
|||||||
p.Now = func() time.Time { return now }
|
p.Now = func() time.Time { return now }
|
||||||
checker := reputation.NewAbuseIPDB(p)
|
checker := reputation.NewAbuseIPDB(p)
|
||||||
|
|
||||||
// The last score still in use, and one, of other's /64, fetched a TTL
|
// The last score still in use, and one fetched a TTL ago.
|
||||||
// ago.
|
|
||||||
inUse := reputation.Score{
|
inUse := reputation.Score{
|
||||||
Client: netip.MustParsePrefix(suspect + "/32"), Score: 100,
|
Client: netip.MustParseAddr(suspect), Score: 100,
|
||||||
Fetched: now.Add(-cacheTTL + time.Nanosecond),
|
Fetched: now.Add(-cacheTTL + time.Nanosecond),
|
||||||
}
|
}
|
||||||
stale := reputation.Score{
|
stale := reputation.Score{
|
||||||
Client: netip.MustParsePrefix("2001:db8::/64"), Score: 100,
|
Client: netip.MustParseAddr(other), Score: 100, Fetched: now.Add(-cacheTTL),
|
||||||
Fetched: now.Add(-cacheTTL),
|
|
||||||
}
|
}
|
||||||
|
|
||||||
checker.Load(reputation.Checks{Scores: []reputation.Score{stale, inUse}})
|
checker.Load(reputation.Checks{Scores: []reputation.Score{stale, inUse}})
|
||||||
@@ -451,13 +417,12 @@ func TestAtMost100000ScoresKeptTheOneFetchedLongestAgoDroppedFirst(t *testing.T)
|
|||||||
|
|
||||||
scores := make([]reputation.Score, 0, count)
|
scores := make([]reputation.Score, 0, count)
|
||||||
|
|
||||||
addr := netip.MustParseAddr("198.18.0.0")
|
client := netip.MustParseAddr("198.18.0.0")
|
||||||
for i := range count {
|
for i := range count {
|
||||||
scores = append(scores, reputation.Score{
|
scores = append(scores, reputation.Score{
|
||||||
Client: netip.PrefixFrom(addr, 32),
|
Client: client, Fetched: now.Add(-time.Duration(i) * time.Millisecond),
|
||||||
Fetched: now.Add(-time.Duration(i) * time.Millisecond),
|
|
||||||
})
|
})
|
||||||
addr = addr.Next()
|
client = client.Next()
|
||||||
}
|
}
|
||||||
|
|
||||||
checker.Load(reputation.Checks{Scores: scores})
|
checker.Load(reputation.Checks{Scores: scores})
|
||||||
@@ -572,31 +537,13 @@ func wantScore(
|
|||||||
) {
|
) {
|
||||||
t.Helper()
|
t.Helper()
|
||||||
|
|
||||||
gotScore, gotHit := hitFrom(t, checker, client, offender)
|
gotScore, gotHit := checker.Hit(t.Context(), netip.MustParseAddr(client), offender)
|
||||||
if gotScore != score || gotHit != hit {
|
if gotScore != score || gotHit != hit {
|
||||||
t.Errorf("%s has the score %d, a hit %t, want %d, %t", client, gotScore, gotHit,
|
t.Errorf("%s has the score %d, a hit %t, want %d, %t", client, gotScore, gotHit,
|
||||||
score, hit)
|
score, hit)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// hitFrom is checker's Hit for a request from address, offender or not.
|
|
||||||
// Its client is address for an IPv4 address, and its /64 for an IPv6 one,
|
|
||||||
// as smallwebwaf counts clients.
|
|
||||||
func hitFrom(
|
|
||||||
t *testing.T, checker *reputation.AbuseIPDB, address string, offender bool,
|
|
||||||
) (int64, bool) {
|
|
||||||
t.Helper()
|
|
||||||
|
|
||||||
addr := netip.MustParseAddr(address)
|
|
||||||
|
|
||||||
client := netip.PrefixFrom(addr, addr.BitLen())
|
|
||||||
if addr.Is6() {
|
|
||||||
client = netip.PrefixFrom(addr, 64).Masked()
|
|
||||||
}
|
|
||||||
|
|
||||||
return checker.Hit(t.Context(), client, addr, offender)
|
|
||||||
}
|
|
||||||
|
|
||||||
// wantChecked checks the clients the stand-in was asked about, in any
|
// wantChecked checks the clients the stand-in was asked about, in any
|
||||||
// order.
|
// order.
|
||||||
func wantChecked(t *testing.T, abuseIPDB *abuseIPDBStandIn, want ...string) {
|
func wantChecked(t *testing.T, abuseIPDB *abuseIPDBStandIn, want ...string) {
|
||||||
|
|||||||
@@ -252,12 +252,12 @@ const filledReputationJSON = `{
|
|||||||
"spent": 3,
|
"spent": 3,
|
||||||
"scores": [
|
"scores": [
|
||||||
{
|
{
|
||||||
"client": "203.0.113.9/32",
|
"client": "203.0.113.9",
|
||||||
"score": 100,
|
"score": 100,
|
||||||
"fetched": "2026-10-05T23:00:00Z"
|
"fetched": "2026-10-05T23:00:00Z"
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"client": "2001:db8::/64",
|
"client": "2001:db8::1",
|
||||||
"score": 0,
|
"score": 0,
|
||||||
"fetched": "2026-10-05T22:00:00Z"
|
"fetched": "2026-10-05T22:00:00Z"
|
||||||
}
|
}
|
||||||
@@ -710,7 +710,7 @@ func TestReputationJSONScoreWithoutAFieldItNeedsStopsTheStart(t *testing.T) {
|
|||||||
// score and fetched make a score.
|
// score and fetched make a score.
|
||||||
const (
|
const (
|
||||||
scores = `{"version": 1, "abuseipdb": {"scores": [`
|
scores = `{"version": 1, "abuseipdb": {"scores": [`
|
||||||
client = `"client": "198.51.100.7/32", `
|
client = `"client": "198.51.100.7", `
|
||||||
score = `"score": 0, `
|
score = `"score": 0, `
|
||||||
fetched = `"fetched": "2026-10-06T00:00:00Z"`
|
fetched = `"fetched": "2026-10-06T00:00:00Z"`
|
||||||
ends = `}]}}`
|
ends = `}]}}`
|
||||||
@@ -1250,7 +1250,7 @@ func TestEditOfEachFileTakenIn(t *testing.T) {
|
|||||||
`"lines": ["198.51.100.7"]}], "verdicts": [{"zone": "`+dnsblZone+`", `+
|
`"lines": ["198.51.100.7"]}], "verdicts": [{"zone": "`+dnsblZone+`", `+
|
||||||
`"client": "198.51.100.7", "listed": true, "fetched": "2026-10-06T00:00:00Z"}], `+
|
`"client": "198.51.100.7", "listed": true, "fetched": "2026-10-06T00:00:00Z"}], `+
|
||||||
`"abuseipdb": {"day": "2026-10-06T00:00:00Z", "spent": 9, "scores": [`+
|
`"abuseipdb": {"day": "2026-10-06T00:00:00Z", "spent": 9, "scores": [`+
|
||||||
`{"client": "198.51.100.7/32", "score": 80, "fetched": "2026-10-06T00:00:00Z"}]}}`)
|
`{"client": "198.51.100.7", "score": 80, "fetched": "2026-10-06T00:00:00Z"}]}}`)
|
||||||
wantTakenIn(t, lines, dir, reputationJSON)
|
wantTakenIn(t, lines, dir, reputationJSON)
|
||||||
|
|
||||||
listedBy := params.Lists.ListedBy(client.Addr())
|
listedBy := params.Lists.ListedBy(client.Addr())
|
||||||
@@ -1265,7 +1265,7 @@ func TestEditOfEachFileTakenIn(t *testing.T) {
|
|||||||
|
|
||||||
checks := reputation.Checks{
|
checks := reputation.Checks{
|
||||||
Day: midnight(), Spent: 9,
|
Day: midnight(), Spent: 9,
|
||||||
Scores: []reputation.Score{{Client: client, Score: 80, Fetched: midnight()}},
|
Scores: []reputation.Score{{Client: client.Addr(), Score: 80, Fetched: midnight()}},
|
||||||
}
|
}
|
||||||
if got := params.AbuseIPDB.Snapshot(); !reflect.DeepEqual(got, checks) {
|
if got := params.AbuseIPDB.Snapshot(); !reflect.DeepEqual(got, checks) {
|
||||||
t.Errorf("%s taken in as\n%+v\nwant\n%+v", reputationJSON, got, checks)
|
t.Errorf("%s taken in as\n%+v\nwant\n%+v", reputationJSON, got, checks)
|
||||||
@@ -1778,8 +1778,8 @@ func fill(params state.Params) {
|
|||||||
{Zone: dnsblZone, Client: client.Addr(), Listed: true, Fetched: now.Add(-time.Hour)},
|
{Zone: dnsblZone, Client: client.Addr(), Listed: true, Fetched: now.Add(-time.Hour)},
|
||||||
})
|
})
|
||||||
params.AbuseIPDB.Load(reputation.Checks{Day: now, Spent: 3, Scores: []reputation.Score{
|
params.AbuseIPDB.Load(reputation.Checks{Day: now, Spent: 3, Scores: []reputation.Score{
|
||||||
{Client: netip.MustParsePrefix("2001:db8::/64"), Fetched: now.Add(-2 * time.Hour)},
|
{Client: netip.MustParseAddr("2001:db8::1"), Fetched: now.Add(-2 * time.Hour)},
|
||||||
{Client: client, Score: 100, Fetched: now.Add(-time.Hour)},
|
{Client: client.Addr(), Score: 100, Fetched: now.Add(-time.Hour)},
|
||||||
}})
|
}})
|
||||||
|
|
||||||
// An alert waiting, a repeat of it the cooldown holds back, another
|
// An alert waiting, a repeat of it the cooldown holds back, another
|
||||||
|
|||||||
Reference in New Issue
Block a user