Compare commits

..
Author SHA1 Message Date
clawbot 89f2918d25 bans: one ledger method makes a request's ban, one place decides observe mode (closes #444)
check / check (push) Waiting to run
The ledger's BanForLimit, WouldBanForLimit, BanForAttack and
WouldBanForAttack become Ban and WouldBan, which take the cause and the
notes in one bans.Grounds. The waf's banForLimit and banForAttack fill in
what differs and call banClient, the one function that tests observe
mode on the way to a ban. Behaviour is unchanged; tests change only
where they called the four methods.

Model: opus-5-5
2026-11-19 17:39:32 +00:00
clawbot 805c64a256 config: give the fixed-value settings named types (closes #427)
check / check (push) Waiting to run
BytesCounted, LookupSource, BlocklistAction, DNSBLAbuseIPDBAction and
CoreRuleSetMode become named string types declared in config, with
constants for their values. Every comparison in waf and sww uses the
constants, and the private duplicates fileSource and deny go. The two
action settings share ReputationAction. The values and the strings
written to logs, metrics and alerts are unchanged.

Model: opus-5-5
2026-11-19 18:32:29 +01:00
clawbot 967ccdf75e sww: build the Core Rule Set from the settings (closes #434)
check / check (push) Waiting to run
The body of waf.NewCoreRuleSet moves into newCoreRuleSet in
internal/sww, beside the providers of the other parts; waf no longer
exports it. The waf test helper builds the Core Rule Set with
coreruleset.New while the mode is not off.

Model: opus-5-5
2026-11-19 18:15:33 +01:00
36 changed files with 680 additions and 452 deletions

No files matched your search

+30 -31
View File
@@ -817,8 +817,8 @@ The supported apps are:
- `vaultwarden`;
- `wordpress`.
Each app's fixes are in one file named for the app, such as
`internal/coreruleset/apps/gitea.conf`. gitea's fixes are:
Each app's fixes are in one file, such as `internal/coreruleset/gitea.conf`, and
one entry of `apps` in `internal/coreruleset/coreruleset.go`. gitea's fixes are:
1. Ten parameters are not checked against the Core Rule Set's lists of system
files (930120), shell paths (932160) and command names (932260), on any path:
@@ -851,8 +851,8 @@ Each app's fixes are in one file named for the app, such as
3. On gitea's paths, some fields skip the rules that refused ordinary gitea
requests there. They are the fields where gitea sends a search, a file's name
or text, a commit message, a branch name or a repository's name.
`internal/coreruleset/apps/gitea.conf` lists each rule and the request it
refused. The fields are:
`internal/coreruleset/gitea.conf` lists each rule and the request it refused.
The fields are:
- `q` on gitea's searches of code, of issues and pull requests, and of
commits. A code search holds code, such as `<script>`,
`SELECT * FROM users WHERE` or `../`. The searches of issues and pull
@@ -1037,7 +1037,7 @@ Each app's fixes are in one file named for the app, such as
- The `Content-Range` header of an artifact upload from
`actions/upload-artifact@v3` (920450).
- The passwords in a body, for each rule that random passwords set off. The
numbers are in `internal/coreruleset/apps/gitea.conf`.
numbers are in `internal/coreruleset/gitea.conf`.
- `password` on these paths:
- `/user/login`, `/user/sign_up`, `/user/activate`;
- `/user/link_account_signin` and `/user/link_account_signup`;
@@ -1069,8 +1069,8 @@ Each app's fixes are in one file named for the app, such as
such as `owner/python3-app` would have that push refused.
WordPress's fixes: on WordPress's paths, some fields skip the rules that refused
ordinary WordPress requests there. `internal/coreruleset/apps/wordpress.conf`
lists each rule and the request it refused. The REST API has four addresses:
ordinary WordPress requests there. `internal/coreruleset/wordpress.conf` lists
each rule and the request it refused. The REST API has four addresses:
- `/wp-json/...`
- `/index.php/wp-json/...`
@@ -1162,9 +1162,9 @@ Elsewhere WordPress's fixes do not apply. The paths are those of WordPress at
the site root.
Drupal's fixes: on Drupal's paths, some fields skip the rules that refused
ordinary Drupal requests there. `internal/coreruleset/apps/drupal.conf` lists
each rule and the request it refused. These query fields skip the lists of
system files and command names (930120, 932120, 932250, 932260):
ordinary Drupal requests there. `internal/coreruleset/drupal.conf` lists each
rule and the request it refused. These query fields skip the lists of system
files and command names (930120, 932120, 932250, 932260):
- On every page under `/admin`: `title`, `name`, `user`, `filename`, `filemime`,
`info`, `subject`, `author_name`, `search`, `q` and `destination`.
@@ -1254,10 +1254,10 @@ first path segment of two or three lowercase letters, optionally followed by a
hyphen and more letters, so `/api` counts too.
MediaWiki's fixes: on MediaWiki's paths, some fields skip the rules that refused
ordinary MediaWiki requests there. `internal/coreruleset/apps/mediawiki.conf`
lists each rule and the request it refused. These fields skip the lists of
system files and command names (930120, 932120, 932250, 932260), in the query or
the body:
ordinary MediaWiki requests there. `internal/coreruleset/mediawiki.conf` lists
each rule and the request it refused. These fields skip the lists of system
files and command names (930120, 932120, 932250, 932260), in the query or the
body:
- On MediaWiki's pages, which are `/index.php` and the paths under `/index.php/`
and `/wiki/`:
@@ -1363,9 +1363,9 @@ Elsewhere these fields keep every rule. The paths are those of MediaWiki at the
site root or under `/w/`.
Mastodon's fixes: on Mastodon's paths, some fields skip the rules that refused
ordinary Mastodon requests there. `internal/coreruleset/apps/mastodon.conf`
lists each rule and the request it refused. These fields skip the lists of
system files and command names (930120, 932120, 932250, 932260), in the query:
ordinary Mastodon requests there. `internal/coreruleset/mastodon.conf` lists
each rule and the request it refused. These fields skip the lists of system
files and command names (930120, 932120, 932250, 932260), in the query:
- On the API's search, accounts and admin accounts, under `/api/v1/` and
`/api/v2/`, the searches and names `q`, `acct`, `username`, `display_name`,
@@ -1526,7 +1526,7 @@ every rule.
Vaultwarden's fixes: on Vaultwarden's paths, some fields skip the rules that
refused ordinary Vaultwarden requests there.
`internal/coreruleset/apps/vaultwarden.conf` lists each rule and the request it
`internal/coreruleset/vaultwarden.conf` lists each rule and the request it
refused. The apps encrypt vault items, folders, collections and sends before
sending them, so those fields skip no rules. The fields below hold plain text.
@@ -1577,8 +1577,8 @@ has a parameter of the same name.
`token` in the body of `/admin`, the admin token typed on the admin page, skips
each rule that random passwords set off. The numbers are in
`internal/coreruleset/apps/vaultwarden.conf`. It applies only when the query has
no `token` parameter. `' OR '1'='1`, `../../etc/passwd`, `; cat /etc/passwd` and
`internal/coreruleset/vaultwarden.conf`. It applies only when the query has no
`token` parameter. `' OR '1'='1`, `../../etc/passwd`, `; cat /etc/passwd` and
`<img src=x onerror=x()>` sent as the token reach Vaultwarden;
`<script>alert(1)</script>` and `javascript:alert(1)` are still refused.
@@ -1600,7 +1600,7 @@ Elsewhere these fields keep every rule.
Nextcloud's fixes: on Nextcloud's paths, some fields skip the rules that refused
ordinary Nextcloud requests there, and its WebDAV methods are allowed.
`internal/coreruleset/apps/nextcloud.conf` lists each rule and the request it
`internal/coreruleset/nextcloud.conf` lists each rule and the request it
refused.
The sync client and the Files app use WebDAV on these paths:
@@ -1610,8 +1610,7 @@ The sync client and the Files app use WebDAV on these paths:
- `/remote.php/webdav/`;
- `/public.php/webdav/`.
These methods are allowed there, besides `PUT`, `PATCH` and `DELETE`, which
every app gets:
These methods are allowed there, besides those of the first change:
- `PROPFIND` and `PROPPATCH`;
- `MKCOL`;
@@ -1755,7 +1754,7 @@ gets gitea's fixes too. Forgejo's own fixes hold only what Forgejo sends and
gitea does not.
On Forgejo's paths, some fields skip the rules that refused ordinary Forgejo
requests there. `internal/coreruleset/apps/forgejo.conf` lists each rule and the
requests there. `internal/coreruleset/forgejo.conf` lists each rule and the
request it refused.
`q` skips rules on some of Forgejo's searches. The searches are:
@@ -1801,10 +1800,10 @@ The Core Rule Set reads the request without the `flash` cookie, in which Forgejo
puts what gitea puts in `gitea_flash` (gitea's second fix).
Ghost's fixes: on Ghost's paths, some fields skip the rules that refused
ordinary Ghost requests there. `internal/coreruleset/apps/ghost.conf` lists each
rule and the request it refused. Ghost's admin and integrations call the Admin
API under `/ghost/api/admin/`. Themes and Ghost's scripts call the Content API
under `/ghost/api/content/`, and members call `/members/api/`. The admin calls
ordinary Ghost requests there. `internal/coreruleset/ghost.conf` lists each rule
and the request it refused. Ghost's admin and integrations call the Admin API
under `/ghost/api/admin/`. Themes and Ghost's scripts call the Content API under
`/ghost/api/content/`, and members call `/members/api/`. The admin calls
ActivityPub, a service of its own, under `/.ghost/activitypub/`.
Eight rules refused words that name commands and files, such as `python3`,
@@ -1942,9 +1941,9 @@ That name holds `.profile`, a file on the list of system files, so the name
skips 930120 when a profile is saved.
The passwords skip each rule that random passwords set off. The numbers are in
`internal/coreruleset/apps/ghost.conf`. They are `password` when signing in, and
the passwords sent when creating the first staff user, accepting an invitation,
and resetting or changing a password. `' OR '1'='1`, `../../etc/passwd`,
`internal/coreruleset/ghost.conf`. They are `password` when signing in, and the
passwords sent when creating the first staff user, accepting an invitation, and
resetting or changing a password. `' OR '1'='1`, `../../etc/passwd`,
`; cat /etc/passwd` and `<img src=x onerror=x()>` sent as a password reach
Ghost; `<script>alert(1)</script>` and `javascript:alert(1)` are still refused.
+8 -4
View File
@@ -54,7 +54,11 @@ func TestBlacklistLeavesANetblockWithAnActiveBanAsItIs(t *testing.T) {
now := midnight()
scores := map[netip.Prefix]int64{netblock: 100}
limitBan, _ := ledger.BanForLimit(netblock, now, bans.Notes{})
limitBan, _ := ledger.Ban(
netblock,
now,
bans.Grounds{Cause: bans.CauseLimit},
)
if made := ledger.BanForAbuseIPDB(now, now.Add(30*day), scores); made != 0 {
t.Errorf("made %d bans over a ban for a limit, want none", made)
@@ -119,7 +123,7 @@ func TestEndedBlacklistBansAreForgottenAtTheNextFetch(t *testing.T) {
expires := now.Add(30 * day)
ledger.BanForAbuseIPDB(now, expires, map[netip.Prefix]int64{listed: 100})
ledger.BanForLimit(limited, now, bans.Notes{})
ledger.Ban(limited, now, bans.Grounds{Cause: bans.CauseLimit})
// listed is off the blacklist when it is fetched after the ban's end.
if made := ledger.BanForAbuseIPDB(expires, expires.Add(30*day), nil); made != 0 {
@@ -144,7 +148,7 @@ func TestBlacklistBansDoNotCountTowardMaxBans(t *testing.T) {
scores[netip.PrefixFrom(netip.AddrFrom4([4]byte{198, 51, 100, byte(i)}), 32)] = 100
}
ledger.BanForLimit(limited, now, bans.Notes{})
ledger.Ban(limited, now, bans.Grounds{Cause: bans.CauseLimit})
if made := ledger.BanForAbuseIPDB(now, now.Add(30*day), scores); made != 3 {
t.Errorf("made %d bans for 3 netblocks, want 3", made)
@@ -154,7 +158,7 @@ func TestBlacklistBansDoNotCountTowardMaxBans(t *testing.T) {
// The ban for an attack takes the place of the one for a limit, and no
// blacklist ban goes.
ledger.BanForAttack(attacker, now, bans.Notes{})
ledger.Ban(attacker, now, bans.Grounds{Cause: bans.CauseAttack})
want := map[netip.Prefix]int{limited: 0, attacker: 1}
for netblock := range scores {
+64 -23
View File
@@ -61,7 +61,7 @@ func TestAdminsBansAreNeverDroppedAndDoNotCountTowardMaxBans(t *testing.T) {
wantBans(t, ledger, map[netip.Prefix]int{adminsOnly: 2, both: 2})
// A new ban drops the ban sww made, and only that one.
ledger.BanForLimit(second, midnight(), bans.Notes{})
ledger.Ban(second, midnight(), bans.Grounds{Cause: bans.CauseLimit})
wantBans(
t,
ledger,
@@ -77,7 +77,7 @@ func TestAdminsBansAreNeverDroppedAndDoNotCountTowardMaxBans(t *testing.T) {
}
// And the next drops that one.
ledger.BanForLimit(third, midnight(), bans.Notes{})
ledger.Ban(third, midnight(), bans.Grounds{Cause: bans.CauseLimit})
wantBans(
t,
ledger,
@@ -98,20 +98,28 @@ func TestEarlierBansCountABanDroppedAfterAnAdminsBan(t *testing.T) {
// twice, each time after its last ban has ended.
ledger.BanForAdmin(netblock, midnight(), midnight().Add(time.Minute),
"probes for logins")
first, _ := ledger.BanForLimit(
first, _ := ledger.Ban(
netblock,
midnight().Add(time.Minute),
bans.Notes{},
bans.Grounds{Cause: bans.CauseLimit},
)
second, _ := ledger.Ban(
netblock,
first.Expires,
bans.Grounds{Cause: bans.CauseLimit},
)
second, _ := ledger.BanForLimit(netblock, first.Expires, bans.Notes{})
// A ban on another netblock fills the ledger, and the netblock's first
// ban for a limit, its earliest that is not an admin's, goes.
ledger.BanForLimit(other, second.Start, bans.Notes{})
ledger.Ban(other, second.Start, bans.Grounds{Cause: bans.CauseLimit})
wantBans(t, ledger, map[netip.Prefix]int{netblock: 2, other: 1})
// The netblock's next ban still counts it.
third, _ := ledger.BanForLimit(netblock, second.Expires, bans.Notes{})
third, _ := ledger.Ban(
netblock,
second.Expires,
bans.Grounds{Cause: bans.CauseLimit},
)
if third.Notes.EarlierBans != (bans.EarlierBans{Limit: 2, Admin: 1}) {
t.Errorf("the third ban for a limit has earlier bans %+v, "+
"want 2 for a limit and 1 an admin's", third.Notes.EarlierBans)
@@ -127,15 +135,27 @@ func TestEarlierBansCountABanAnAdminKeepsOnce(t *testing.T) {
// The netblock breaks a limit twice, the second time after its first
// ban has ended, and the second ban's notes count the first. An admin
// then keeps the first ban.
first, _ := ledger.BanForLimit(netblock, midnight(), bans.Notes{})
second, _ := ledger.BanForLimit(netblock, first.Expires, bans.Notes{})
first, _ := ledger.Ban(
netblock,
midnight(),
bans.Grounds{Cause: bans.CauseLimit},
)
second, _ := ledger.Ban(
netblock,
first.Expires,
bans.Grounds{Cause: bans.CauseLimit},
)
kept := first
kept.Cause = bans.CauseAdmin
ledger.LoadEdit([]bans.Ban{kept, second})
// The netblock's next ban counts the kept ban once, as an admin's.
third, _ := ledger.BanForLimit(netblock, second.Expires, bans.Notes{})
third, _ := ledger.Ban(
netblock,
second.Expires,
bans.Grounds{Cause: bans.CauseLimit},
)
if third.Notes.EarlierBans != (bans.EarlierBans{Limit: 1, Admin: 1}) {
t.Errorf("the third ban for a limit has earlier bans %+v, "+
"want 1 for a limit and 1 an admin's", third.Notes.EarlierBans)
@@ -147,20 +167,37 @@ func TestReasonOfTheBansSwwMakes(t *testing.T) {
ledger := bans.New(defaultParams())
limit, _ := ledger.BanForLimit(
limit, _ := ledger.Ban(
netip.MustParsePrefix("203.0.113.1/32"),
midnight(),
bans.Notes{Kind: "requests", Limit: 1000, Window: "minute"},
bans.Grounds{
Cause: bans.CauseLimit,
Notes: bans.Notes{
Kind: "requests",
Limit: 1000,
Window: "minute",
},
},
)
byteLimit, _ := ledger.BanForLimit(
byteLimit, _ := ledger.Ban(
netip.MustParsePrefix("203.0.113.3/32"),
midnight(),
bans.Notes{Kind: "bytes", Limit: 10 << 30, Window: "hour"},
bans.Grounds{
Cause: bans.CauseLimit,
Notes: bans.Notes{
Kind: "bytes",
Limit: 10 << 30,
Window: "hour",
},
},
)
attack, _ := ledger.BanForAttack(
attack, _ := ledger.Ban(
netip.MustParsePrefix("203.0.113.2/32"),
midnight(),
bans.Notes{RuleID: "git-dir", Target: "path"},
bans.Grounds{
Cause: bans.CauseAttack,
Notes: bans.Notes{RuleID: "git-dir", Target: "path"},
},
)
for _, tc := range []struct{ got, want string }{
@@ -200,7 +237,7 @@ func TestLiftedBanForALimitRefusesNothingAndMakesNoBanLonger(t *testing.T) {
t.Error("the lifted ban refuses")
}
ban, _ := ledger.BanForLimit(netblock, now, bans.Notes{})
ban, _ := ledger.Ban(netblock, now, bans.Grounds{Cause: bans.CauseLimit})
if ban.Expires.Sub(ban.Start) != time.Hour ||
ban.Notes.EarlierBans != (bans.EarlierBans{Limit: 1}) {
t.Errorf(
@@ -251,7 +288,11 @@ func TestLiftedBanForAnAttackRefusesNothingAndMakesNoBanLonger(
}
// The next clear sign of attack bans for seven days, as a first does.
ban, _ := ledger.BanForAttack(netblock, now, bans.Notes{})
ban, _ := ledger.Ban(
netblock,
now,
bans.Grounds{Cause: bans.CauseAttack},
)
if ban.Expires.Sub(ban.Start) != 7*day {
t.Errorf(
"the next ban for an attack ends at %s, want seven days on",
@@ -264,10 +305,10 @@ func TestLoadEditCountsTheBansAnAdminMade(t *testing.T) {
t.Parallel()
ledger := bans.New(defaultParams())
made, _ := ledger.BanForLimit(
made, _ := ledger.Ban(
netip.MustParsePrefix("203.0.113.1/32"),
midnight(),
bans.Notes{},
bans.Grounds{Cause: bans.CauseLimit},
)
atStart := bans.Ban{
Netblock: netip.MustParsePrefix("203.0.113.2/32"),
@@ -311,7 +352,7 @@ func TestAdminsBanIsMadeWhileAnotherLasts(t *testing.T) {
ledger := bans.New(defaultParams())
// An hour's ban for a broken limit.
ledger.BanForLimit(netblock, midnight(), bans.Notes{})
ledger.Ban(netblock, midnight(), bans.Grounds{Cause: bans.CauseLimit})
wantChanged(t, ledger, true)
// A minute later an admin bans the netblock for good, named by an
@@ -431,10 +472,10 @@ func TestPermanentBanOnAWiderNetblockIsReturnedOverATemporaryBan(
ledger := bans.New(defaultParams())
// The temporary ban is made first, so that its netblock is looked at first.
ledger.BanForLimit(
ledger.Ban(
netip.MustParsePrefix("203.0.113.9/32"),
midnight(),
bans.Notes{},
bans.Grounds{Cause: bans.CauseLimit},
)
ledger.BanForAdmin(
netip.MustParsePrefix("203.0.113.0/24"),
+54 -82
View File
@@ -300,81 +300,50 @@ func activeBan(bans []Ban, now time.Time) *Ban {
return newest
}
// BanForLimit bans netblock at now for a broken limit, with notes, and
// returns the ban, and true. A first ban lasts LimitBanDuration. A ban
// made within LimitBanRepeatWindow after the netblock's ban that ended
// last, other than one for a clear sign of attack or a lifted one, lasts
// repeatFactor times as long as that one. A ban that would be longer
// than LimitBanPermanentAbove is permanent instead. If a ban on netblock is still
// active, as when two of its requests break a limit at once, that ban is
// returned with false, and no other is made. The ledger fills in the
// notes' Refused and EarlierBans itself, and gives the ban the reason
// "<Kind> per <Window> over the limit of <Limit>", from the notes, such
// as "requests per minute over the limit of 1000".
func (l *Ledger) BanForLimit(
netblock netip.Prefix, now time.Time, notes Notes,
) (Ban, bool) {
return l.ban(
netblock,
now,
CauseLimit,
limitReason(notes),
notes,
true,
)
// Grounds are why a request makes a ban: its cause, CauseLimit or
// CauseAttack, and its notes.
type Grounds struct {
Cause string
Notes Notes
}
// WouldBanForLimit returns what BanForLimit would, without making the ban:
// what observe mode would have done.
func (l *Ledger) WouldBanForLimit(
netblock netip.Prefix, now time.Time, notes Notes,
// Ban bans netblock at now on grounds, and returns the ban, and true. If
// a ban on netblock is still active, as when two of its requests break a
// limit at once, that ban is returned with false, and no other is made.
// The ledger fills in the notes' Refused and EarlierBans itself.
//
// A ban for a broken limit:
// - lasts LimitBanDuration the first time;
// - lasts repeatFactor times as long as the netblock's ban that ended
// last, when made within LimitBanRepeatWindow after it, unless that
// ban was for a clear sign of attack or was lifted;
// - is permanent instead when it would be longer than
// LimitBanPermanentAbove;
// - has the reason "<Kind> per <Window> over the limit of <Limit>", from
// the notes, such as "requests per minute over the limit of 1000".
//
// A ban for a clear sign of attack:
// - lasts AttackBanDuration the first time;
// - is permanent once the netblock has had one that was not lifted;
// - has the reason "matched the rule <RuleID>", or "asked for the trap
// path <TrapPath>".
func (l *Ledger) Ban(
netblock netip.Prefix, now time.Time, grounds Grounds,
) (Ban, bool) {
return l.ban(
netblock,
now,
CauseLimit,
limitReason(notes),
notes,
false,
)
return l.ban(netblock, now, grounds, true)
}
// BanForAttack bans netblock at now for a clear sign of attack, with
// notes, and returns the ban, and whether it made it, as BanForLimit
// does. A first ban lasts AttackBanDuration; once the netblock has had
// one that was not lifted, the next is permanent. Its reason is "matched
// the rule <RuleID>", or "asked for the trap path <TrapPath>".
func (l *Ledger) BanForAttack(
netblock netip.Prefix, now time.Time, notes Notes,
// WouldBan returns what Ban would, without making the ban: what observe
// mode would have done.
func (l *Ledger) WouldBan(
netblock netip.Prefix, now time.Time, grounds Grounds,
) (Ban, bool) {
return l.ban(
netblock,
now,
CauseAttack,
attackReason(notes),
notes,
true,
)
}
// WouldBanForAttack returns what BanForAttack would, without making the
// ban: what observe mode would have done.
func (l *Ledger) WouldBanForAttack(
netblock netip.Prefix, now time.Time, notes Notes,
) (Ban, bool) {
return l.ban(
netblock,
now,
CauseAttack,
attackReason(notes),
notes,
false,
)
return l.ban(netblock, now, grounds, false)
}
// WouldBePermanent reports whether a ban on netblock for cause, CauseLimit
// or CauseAttack, made at now would be permanent, as BanForLimit or
// BanForAttack would make it. It works out nothing else of the ban.
// or CauseAttack, made at now would be permanent, as Ban would make it. It
// works out nothing else of the ban.
func (l *Ledger) WouldBePermanent(
netblock netip.Prefix, now time.Time, cause string,
) bool {
@@ -411,10 +380,9 @@ func attackReason(notes Notes) string {
// BanForAdmin bans netblock at now for an admin, with reason, until
// expires, or for good when expires is zero, and returns the ban, whose
// cause is CauseAdmin. Unlike BanForLimit and BanForAttack, it makes the
// ban even while another on netblock is active, since the admin asked
// for this one. The ledger fills in the notes' EarlierBans, and counts
// the ban among those made.
// cause is CauseAdmin. Unlike Ban, it makes the ban even while another
// on netblock is active, since the admin asked for this one. The ledger
// fills in the notes' EarlierBans, and counts the ban among those made.
func (l *Ledger) BanForAdmin(
netblock netip.Prefix, now, expires time.Time, reason string,
) Ban {
@@ -759,15 +727,13 @@ func (l *Ledger) forgetEndedAbuseIPDBBans(now time.Time) bool {
return forgot
}
// ban bans netblock at now for cause, CauseLimit or CauseAttack, with
// reason and notes, as BanForLimit and BanForAttack describe, and returns
// the ban, and whether it made it. Unless keep is true, the ban is not
// made, only returned: it is the ban that would have been made.
// ban bans netblock at now on grounds, as Ban describes, and returns the
// ban, and whether it made it. Unless keep is true, the ban is not made,
// only returned: it is the ban that would have been made.
func (l *Ledger) ban(
netblock netip.Prefix,
now time.Time,
cause, reason string,
notes Notes,
grounds Grounds,
keep bool,
) (Ban, bool) {
l.mu.Lock()
@@ -776,6 +742,8 @@ func (l *Ledger) ban(
// held are the netblock's bans, none of them active.
var held []Ban
notes := grounds.Notes
bans, found := l.netblocks.Get(netblock)
if found {
active := activeBan(*bans, now)
@@ -791,15 +759,19 @@ func (l *Ledger) ban(
ban := Ban{
Netblock: netblock,
Start: now,
Cause: cause,
Reason: reason,
Cause: grounds.Cause,
Notes: notes,
}
if cause == CauseAttack {
ban.Expires = l.attackExpiry(held, now)
} else {
switch grounds.Cause {
case CauseLimit:
ban.Reason = limitReason(notes)
ban.Expires = l.limitExpiry(held, now)
case CauseAttack:
ban.Reason = attackReason(notes)
ban.Expires = l.attackExpiry(held, now)
default:
panic("bans: a request's ban has the cause " + grounds.Cause)
}
if !keep {
@@ -807,7 +779,7 @@ func (l *Ledger) ban(
}
l.add(ban)
l.made[cause]++
l.made[grounds.Cause]++
l.markChanged()
return ban, true
+121 -51
View File
@@ -22,7 +22,11 @@ func TestRepeatsTripleUntilPermanent(t *testing.T) {
// Each ban is followed by another as soon as it ends: 1, 3, 9, 27 and
// 81 hours.
for i, hours := range []int{1, 3, 9, 27, 81} {
ban, _ := ledger.BanForLimit(netblock, now, bans.Notes{})
ban, _ := ledger.Ban(
netblock,
now,
bans.Grounds{Cause: bans.CauseLimit},
)
length := time.Duration(hours) * time.Hour
if !ban.Expires.Equal(now.Add(length)) ||
@@ -42,7 +46,7 @@ func TestRepeatsTripleUntilPermanent(t *testing.T) {
// The sixth would last 243 hours, more than seven days: it is
// permanent, and never ends.
ban, _ := ledger.BanForLimit(netblock, now, bans.Notes{})
ban, _ := ledger.Ban(netblock, now, bans.Grounds{Cause: bans.CauseLimit})
if !ban.Permanent() {
t.Fatalf("sixth ban ends at %s, want a permanent one", ban.Expires)
}
@@ -71,15 +75,15 @@ func TestRepeatWindowRunsOut(t *testing.T) {
ledger := bans.New(defaultParams())
netblock := netip.MustParsePrefix("203.0.113.9/32")
first, _ := ledger.BanForLimit(
first, _ := ledger.Ban(
netblock,
midnight(),
bans.Notes{},
bans.Grounds{Cause: bans.CauseLimit},
)
second, _ := ledger.BanForLimit(
second, _ := ledger.Ban(
netblock,
first.Expires.Add(tc.gap),
bans.Notes{},
bans.Grounds{Cause: bans.CauseLimit},
)
if second.Expires.Sub(second.Start) != tc.want ||
@@ -104,10 +108,10 @@ func TestFirstBanLongerThanLimitBanPermanentAboveIsPermanent(t *testing.T) {
params.LimitBanDuration = params.LimitBanPermanentAbove + time.Hour
ledger := bans.New(params)
ban, _ := ledger.BanForLimit(
ban, _ := ledger.Ban(
netip.MustParsePrefix("203.0.113.9/32"),
midnight(),
bans.Notes{},
bans.Grounds{Cause: bans.CauseLimit},
)
if !ban.Permanent() {
t.Errorf("first ban ends at %s, want a permanent one", ban.Expires)
@@ -127,7 +131,11 @@ func TestLimitBanPermanentAboveSetFarOffDoesNotOverflow(t *testing.T) {
now := midnight()
for i := range 14 {
ban, _ := ledger.BanForLimit(netblock, now, bans.Notes{})
ban, _ := ledger.Ban(
netblock,
now,
bans.Grounds{Cause: bans.CauseLimit},
)
if !ban.Expires.After(ban.Start) {
t.Fatalf(
"ban %d starts at %s and ends at %s",
@@ -140,7 +148,7 @@ func TestLimitBanPermanentAboveSetFarOffDoesNotOverflow(t *testing.T) {
now = ban.Expires
}
ban, _ := ledger.BanForLimit(netblock, now, bans.Notes{})
ban, _ := ledger.Ban(netblock, now, bans.Grounds{Cause: bans.CauseLimit})
if !ban.Permanent() {
t.Errorf("15th ban ends at %s, want a permanent one", ban.Expires)
}
@@ -152,15 +160,19 @@ func TestBrokenLimitDuringABanMakesNoOther(t *testing.T) {
ledger := bans.New(defaultParams())
netblock := netip.MustParsePrefix("203.0.113.9/32")
first, made := ledger.BanForLimit(netblock, midnight(), bans.Notes{})
first, made := ledger.Ban(
netblock,
midnight(),
bans.Grounds{Cause: bans.CauseLimit},
)
if !made {
t.Error("the first ban was not made")
}
again, made := ledger.BanForLimit(
again, made := ledger.Ban(
netblock,
midnight().Add(time.Minute),
bans.Notes{},
bans.Grounds{Cause: bans.CauseLimit},
)
if made || !reflect.DeepEqual(again, first) ||
@@ -175,10 +187,10 @@ func TestBrokenLimitDuringABanMakesNoOther(t *testing.T) {
)
}
again, made = ledger.BanForAttack(
again, made = ledger.Ban(
netblock,
midnight().Add(time.Minute),
bans.Notes{},
bans.Grounds{Cause: bans.CauseAttack},
)
if made || !reflect.DeepEqual(again, first) {
t.Errorf(
@@ -195,10 +207,10 @@ func TestCheckRefusesWhileTheBanLastsAndCountsTheRefusals(t *testing.T) {
ledger := bans.New(defaultParams())
netblock := netip.MustParsePrefix("203.0.113.9/32")
ban, _ := ledger.BanForLimit(
ban, _ := ledger.Ban(
netblock,
midnight(),
bans.Notes{Requests: 5},
bans.Grounds{Cause: bans.CauseLimit, Notes: bans.Notes{Requests: 5}},
)
for range 3 {
@@ -237,10 +249,10 @@ func TestFindCountsNothing(t *testing.T) {
ledger := bans.New(defaultParams())
netblock := netip.MustParsePrefix("203.0.113.9/32")
ban, _ := ledger.BanForLimit(
ban, _ := ledger.Ban(
netblock,
midnight(),
bans.Notes{Requests: 5},
bans.Grounds{Cause: bans.CauseLimit, Notes: bans.Notes{Requests: 5}},
)
got, banned, _ := ledger.Find(
@@ -287,24 +299,24 @@ func TestMaxBansDropsTheEarliestBanOfTheNetblockSeenLongestAgo(
d := netip.MustParsePrefix("2001:db8::/64")
now := midnight()
first, _ := ledger.BanForLimit(a, now, bans.Notes{})
ledger.BanForLimit(b, now, bans.Notes{})
ledger.BanForLimit(c, now, bans.Notes{})
first, _ := ledger.Ban(a, now, bans.Grounds{Cause: bans.CauseLimit})
ledger.Ban(b, now, bans.Grounds{Cause: bans.CauseLimit})
ledger.Ban(c, now, bans.Grounds{Cause: bans.CauseLimit})
// A request from a makes b the netblock seen longest ago, and its ban
// goes to make room for d's.
ledger.Check(a.Addr(), now)
ledger.BanForLimit(d, now, bans.Notes{})
ledger.Ban(d, now, bans.Grounds{Cause: bans.CauseLimit})
wantBans(t, ledger, map[netip.Prefix]int{a: 1, b: 0, c: 1, d: 1})
// a is banned again once its ban has ended; c, seen longest ago, goes.
ledger.BanForLimit(a, first.Expires, bans.Notes{})
ledger.Ban(a, first.Expires, bans.Grounds{Cause: bans.CauseLimit})
wantBans(t, ledger, map[netip.Prefix]int{a: 2, c: 0, d: 1})
// With d seen since, a is seen longest ago, and its earlier ban goes
// first.
ledger.Check(d.Addr(), first.Expires)
ledger.BanForLimit(b, first.Expires, bans.Notes{})
ledger.Ban(b, first.Expires, bans.Grounds{Cause: bans.CauseLimit})
wantBans(t, ledger, map[netip.Prefix]int{a: 1, b: 1, d: 1})
if !ledger.Bans(a)[0].Start.Equal(first.Expires) {
@@ -325,8 +337,16 @@ func TestFullLedgerDropsTheEarlierBanOfTheNetblockBannedAgain(t *testing.T) {
ledger := bans.New(params)
netblock := netip.MustParsePrefix("203.0.113.9/32")
first, _ := ledger.BanForLimit(netblock, midnight(), bans.Notes{})
second, _ := ledger.BanForLimit(netblock, first.Expires, bans.Notes{})
first, _ := ledger.Ban(
netblock,
midnight(),
bans.Grounds{Cause: bans.CauseLimit},
)
second, _ := ledger.Ban(
netblock,
first.Expires,
bans.Grounds{Cause: bans.CauseLimit},
)
held := ledger.Bans(netblock)
if len(held) != 1 || !reflect.DeepEqual(held[0], second) ||
@@ -343,7 +363,11 @@ func TestRequestDuringAnAttackBanMakesItPermanent(t *testing.T) {
netblock := netip.MustParsePrefix("203.0.113.9/32")
notes := bans.Notes{RuleID: "env-file", Target: "path"}
ban, _ := ledger.BanForAttack(netblock, midnight(), notes)
ban, _ := ledger.Ban(
netblock,
midnight(),
bans.Grounds{Cause: bans.CauseAttack, Notes: notes},
)
if !ban.Expires.Equal(midnight().Add(7*day)) ||
ban.Cause != bans.CauseAttack ||
ban.Notes.RuleID != "env-file" ||
@@ -418,8 +442,16 @@ func TestAttackAfterAnAttackBanHasEndedBansPermanently(t *testing.T) {
netblock := netip.MustParsePrefix("203.0.113.9/32")
// A ban for a broken limit before does not count.
first, _ := ledger.BanForLimit(netblock, midnight(), bans.Notes{})
second, _ := ledger.BanForAttack(netblock, first.Expires, bans.Notes{})
first, _ := ledger.Ban(
netblock,
midnight(),
bans.Grounds{Cause: bans.CauseLimit},
)
second, _ := ledger.Ban(
netblock,
first.Expires,
bans.Grounds{Cause: bans.CauseAttack},
)
if second.Expires.Sub(second.Start) != 7*day {
t.Fatalf("the first ban for an attack lasts %s, want 7 days",
@@ -435,10 +467,10 @@ func TestAttackAfterAnAttackBanHasEndedBansPermanently(t *testing.T) {
// Its notes show the earlier ban for an attack that makes it permanent,
// beside the one for a limit.
third, _ := ledger.BanForAttack(
third, _ := ledger.Ban(
netblock,
second.Expires.Add(30*day),
bans.Notes{},
bans.Grounds{Cause: bans.CauseAttack},
)
if !third.Permanent() ||
third.Notes.EarlierBans != (bans.EarlierBans{Limit: 1, Attack: 1}) {
@@ -452,14 +484,18 @@ func TestWouldBanGivesTheBanWithoutMakingIt(t *testing.T) {
ledger := bans.New(defaultParams())
netblock := netip.MustParsePrefix("203.0.113.9/32")
first, _ := ledger.BanForLimit(netblock, midnight(), bans.Notes{})
first, _ := ledger.Ban(
netblock,
midnight(),
bans.Grounds{Cause: bans.CauseLimit},
)
wantChanged(t, ledger, true)
// While the first ban lasts, none would be made.
during, would := ledger.WouldBanForAttack(
during, would := ledger.WouldBan(
netblock,
midnight(),
bans.Notes{},
bans.Grounds{Cause: bans.CauseAttack},
)
if would || !reflect.DeepEqual(during, first) {
t.Errorf(
@@ -472,13 +508,19 @@ func TestWouldBanGivesTheBanWithoutMakingIt(t *testing.T) {
// As it ends, a clear sign of attack would ban for seven days, and a
// limit broken again for three hours, but neither is made.
limitNotes := bans.Notes{Kind: "requests", Limit: 1, Window: "minute"}
attack, wouldAttack := ledger.WouldBanForAttack(netblock, first.Expires,
bans.Notes{RuleID: "git-dir"})
limit, wouldLimit := ledger.WouldBanForLimit(
limitGrounds := bans.Grounds{
Cause: bans.CauseLimit,
Notes: bans.Notes{Kind: "requests", Limit: 1, Window: "minute"},
}
attack, wouldAttack := ledger.WouldBan(netblock, first.Expires,
bans.Grounds{
Cause: bans.CauseAttack,
Notes: bans.Notes{RuleID: "git-dir"},
})
limit, wouldLimit := ledger.WouldBan(
netblock,
first.Expires,
limitNotes,
limitGrounds,
)
if !wouldAttack || !attack.Expires.Equal(first.Expires.Add(7*day)) ||
@@ -505,7 +547,7 @@ func TestWouldBanGivesTheBanWithoutMakingIt(t *testing.T) {
wantChanged(t, ledger, false)
// The ban made is the one that would have been.
made, _ := ledger.BanForLimit(netblock, first.Expires, limitNotes)
made, _ := ledger.Ban(netblock, first.Expires, limitGrounds)
if !reflect.DeepEqual(made, limit) {
t.Errorf("the ban made is %+v, want %+v", made, limit)
}
@@ -526,7 +568,11 @@ func TestWouldBePermanentAnswersAsTheBanWouldBeMade(t *testing.T) {
t.Fatalf("ban %d for a limit would be permanent", i+1)
}
ban, _ := ledger.BanForLimit(netblock, now, bans.Notes{})
ban, _ := ledger.Ban(
netblock,
now,
bans.Grounds{Cause: bans.CauseLimit},
)
now = ban.Expires
}
@@ -539,7 +585,11 @@ func TestWouldBePermanentAnswersAsTheBanWouldBeMade(t *testing.T) {
}
// Once a first ban for an attack has ended, the next would be permanent.
attack, _ := ledger.BanForAttack(netblock, now, bans.Notes{})
attack, _ := ledger.Ban(
netblock,
now,
bans.Grounds{Cause: bans.CauseAttack},
)
if !ledger.WouldBePermanent(netblock, attack.Expires, bans.CauseAttack) {
t.Error("a second ban for an attack would not be permanent")
}
@@ -553,8 +603,16 @@ func TestAttackBanDoesNotLengthenTheNextBanForALimit(t *testing.T) {
// Three times the seven days would be permanent; a limit broken as the
// ban for an attack ends bans for an hour, as a first broken limit does.
attack, _ := ledger.BanForAttack(netblock, midnight(), bans.Notes{})
limit, _ := ledger.BanForLimit(netblock, attack.Expires, bans.Notes{})
attack, _ := ledger.Ban(
netblock,
midnight(),
bans.Grounds{Cause: bans.CauseAttack},
)
limit, _ := ledger.Ban(
netblock,
attack.Expires,
bans.Grounds{Cause: bans.CauseLimit},
)
if limit.Expires.Sub(limit.Start) != time.Hour ||
limit.Cause != bans.CauseLimit {
@@ -578,10 +636,13 @@ func TestRequestTextsAreCutTo256Bytes(t *testing.T) {
Time: midnight(), Method: long, Host: long, Path: long, Status: 403, UserAgent: long,
}
ban, _ := ledger.BanForLimit(
ban, _ := ledger.Ban(
netblock,
midnight(),
bans.Notes{Request: request},
bans.Grounds{
Cause: bans.CauseLimit,
Notes: bans.Notes{Request: request},
},
)
cut := long[:256]
@@ -607,11 +668,20 @@ func TestLookupFillsTheNotesOfTheNetblocksBansWithoutOne(t *testing.T) {
// A ban made with the client's lookup, one made before it came, after
// the first ended, and one on another netblock.
ledger.BanForLimit(netblock, midnight(), bans.Notes{
ASN: "AS64497", ASName: "Other Net", Country: "FR",
ledger.Ban(netblock, midnight(), bans.Grounds{
Cause: bans.CauseLimit,
Notes: bans.Notes{
ASN: "AS64497",
ASName: "Other Net",
Country: "FR",
},
})
ledger.BanForLimit(netblock, midnight().Add(time.Hour), bans.Notes{})
ledger.BanForLimit(other, midnight(), bans.Notes{})
ledger.Ban(
netblock,
midnight().Add(time.Hour),
bans.Grounds{Cause: bans.CauseLimit},
)
ledger.Ban(other, midnight(), bans.Grounds{Cause: bans.CauseLimit})
ledger.AddLookup(netblock, "AS64496", "Example Net", "DE")
+36 -21
View File
@@ -19,25 +19,29 @@ func TestChangedAfterABanIsMade(t *testing.T) {
wantChanged(t, ledger, false)
ledger.BanForLimit(netblock, midnight(), bans.Notes{})
ledger.Ban(netblock, midnight(), bans.Grounds{Cause: bans.CauseLimit})
wantChanged(t, ledger, true)
// A limit broken during the ban makes no other, and a refusal changes
// only the counts in the notes, which wait for the interval's write.
ledger.BanForLimit(netblock, midnight().Add(time.Minute), bans.Notes{})
ledger.Ban(
netblock,
midnight().Add(time.Minute),
bans.Grounds{Cause: bans.CauseLimit},
)
ledger.Check(netblock.Addr(), midnight().Add(time.Minute))
wantChanged(t, ledger, false)
// Two bans before the value is read leave one.
ledger.BanForLimit(
ledger.Ban(
netip.MustParsePrefix("203.0.113.10/32"),
midnight(),
bans.Notes{},
bans.Grounds{Cause: bans.CauseLimit},
)
ledger.BanForLimit(
ledger.Ban(
netip.MustParsePrefix("203.0.113.11/32"),
midnight(),
bans.Notes{},
bans.Grounds{Cause: bans.CauseLimit},
)
wantChanged(t, ledger, true)
wantChanged(t, ledger, false)
@@ -51,10 +55,14 @@ func TestSnapshotListsEveryBanByNetblock(t *testing.T) {
high := netip.MustParsePrefix("203.0.113.10/32")
low := netip.MustParsePrefix("203.0.113.9/32")
first, _ := ledger.BanForLimit(v6, midnight(), bans.Notes{})
ledger.BanForLimit(high, midnight(), bans.Notes{})
ledger.BanForLimit(low, midnight(), bans.Notes{})
ledger.BanForLimit(v6, first.Expires, bans.Notes{})
first, _ := ledger.Ban(
v6,
midnight(),
bans.Grounds{Cause: bans.CauseLimit},
)
ledger.Ban(high, midnight(), bans.Grounds{Cause: bans.CauseLimit})
ledger.Ban(low, midnight(), bans.Grounds{Cause: bans.CauseLimit})
ledger.Ban(v6, first.Expires, bans.Grounds{Cause: bans.CauseLimit})
snapshot := ledger.Snapshot()
@@ -80,7 +88,10 @@ func TestLoadedBansCarryOn(t *testing.T) {
before := bans.New(defaultParams())
netblock := netip.MustParsePrefix("203.0.113.9/32")
ban, _ := before.BanForLimit(netblock, midnight(), bans.Notes{Limit: 1})
ban, _ := before.Ban(netblock, midnight(), bans.Grounds{
Cause: bans.CauseLimit,
Notes: bans.Notes{Limit: 1},
})
// Loaded into a new ledger, as across a restart, the ban still refuses
// while it lasts, and once it has ended a broken limit bans for three
@@ -96,7 +107,11 @@ func TestLoadedBansCarryOn(t *testing.T) {
t.Error("the loaded ban does not refuse")
}
again, _ := after.BanForLimit(netblock, ban.Expires, bans.Notes{})
again, _ := after.Ban(
netblock,
ban.Expires,
bans.Grounds{Cause: bans.CauseLimit},
)
if again.Expires.Sub(again.Start) != 3*time.Hour ||
again.Notes.EarlierBans != (bans.EarlierBans{Limit: 1}) {
t.Errorf(
@@ -124,10 +139,10 @@ func TestLoadedBanRefusesEveryClientInItsNetblock(t *testing.T) {
Start: midnight(),
},
})
ledger.BanForLimit(
ledger.Ban(
netip.MustParsePrefix("198.51.100.7/32"),
midnight(),
bans.Notes{},
bans.Grounds{Cause: bans.CauseLimit},
)
for client, want := range map[string]bool{
@@ -200,7 +215,7 @@ func TestPermanentBanStartedBeforeAnEndedOneRefuses(t *testing.T) {
}
// A limit broken now makes no shorter ban over the permanent one.
ban, _ = ledger.BanForLimit(netblock, now, bans.Notes{})
ban, _ = ledger.Ban(netblock, now, bans.Grounds{Cause: bans.CauseLimit})
if !ban.Permanent() || len(ledger.Bans(netblock)) != 2 {
t.Errorf(
"a broken limit returned %+v and left the netblock %d bans, "+
@@ -236,10 +251,10 @@ func TestNextBanWorkedOutFromTheBanThatEndedLast(t *testing.T) {
// Once both have ended, a limit broken within the repeat window bans
// for three times the 9 hours, and the notes count the two bans
// before the 9-hour one and it, for a limit, and the admin's.
ban, _ := ledger.BanForLimit(
ban, _ := ledger.Ban(
netblock,
nineHours.Expires.Add(time.Hour),
bans.Notes{},
bans.Grounds{Cause: bans.CauseLimit},
)
if ban.Expires.Sub(ban.Start) != 27*time.Hour ||
ban.Notes.EarlierBans != (bans.EarlierBans{Limit: 3, Admin: 1}) {
@@ -313,15 +328,15 @@ func TestLoadReplacesTheBansHeld(t *testing.T) {
}
// The ledger holds one ban, so it makes two more without dropping any.
first, _ := ledger.BanForLimit(
first, _ := ledger.Ban(
netip.MustParsePrefix("198.51.100.7/32"),
midnight(),
bans.Notes{},
bans.Grounds{Cause: bans.CauseLimit},
)
second, _ := ledger.BanForLimit(
second, _ := ledger.Ban(
netip.MustParsePrefix("198.51.100.8/32"),
midnight(),
bans.Notes{},
bans.Grounds{Cause: bans.CauseLimit},
)
want := []bans.Ban{first, second, kept}
+93 -32
View File
@@ -75,7 +75,7 @@ type WAF struct {
BytesLimitPerMinute int64
BytesLimitPerHour int64
BytesLimitPerDay int64
BytesCounted string
BytesCounted BytesCounted
// LookupSource is where each client's AS number and country are looked
// up (SWW_WAF_LOOKUP_SOURCE): geojs, file, or off for nowhere.
// LookupDBPath is the lookup database, the IPinfo Lite file looked up in
@@ -86,7 +86,7 @@ type WAF struct {
// AddLookupHeaders is true when the app is passed the client's AS number
// and country in headers (SWW_WAF_ADD_LOOKUP_HEADERS), by default unless
// LookupSource is off.
LookupSource string
LookupSource LookupSource
LookupDBPath string
ASNAndCountryLookupTimeout time.Duration
AddLookupHeaders bool
@@ -131,7 +131,7 @@ type WAF struct {
// BlocklistLimitPercent is the percentage of every limit it gets.
BlocklistURLs []string
BlocklistRefresh time.Duration
BlocklistAction string
BlocklistAction ReputationAction
BlocklistLimitPercent int64
// DNSBLZones are the DNSBL zones clients are asked about
// (SWW_WAF_DNSBL_ZONES), through DNSBLResolver (SWW_WAF_DNSBL_RESOLVER), or
@@ -154,7 +154,7 @@ type WAF struct {
AbuseIPDBKey string
AbuseIPDBMinScore int64
AbuseIPDBDailyBudget int
DNSBLAbuseIPDBAction string
DNSBLAbuseIPDBAction ReputationAction
DNSBLAbuseIPDBLimitPercent int64
IPReputationCacheTTL time.Duration
DNSBLAbuseIPDBTimeout time.Duration
@@ -237,7 +237,7 @@ type WAF struct {
// reads none.
// CoreRuleSetExclusionsFile is the path of the file of its exclusions
// (SWW_WAF_CORE_RULE_SET_EXCLUSIONS_FILE), "" while there is none.
CoreRuleSetMode string
CoreRuleSetMode CoreRuleSetMode
CoreRuleSetParanoiaLevel int
CoreRuleSetScoreThreshold int
CoreRuleSetDisabledRules []int
@@ -316,20 +316,66 @@ type WAF struct {
settings []slog.Attr
}
// BytesCounted is which body bytes count toward the byte limits and the
// byte anomaly thresholds, a value of SWW_WAF_BYTES_COUNTED.
type BytesCounted string
// The values of SWW_WAF_BYTES_COUNTED.
const (
// BytesCountedResponse counts the response's body bytes.
BytesCountedResponse BytesCounted = "response"
// BytesCountedRequest counts the request's body bytes.
BytesCountedRequest BytesCounted = "request"
// BytesCountedBoth counts the body bytes of both.
BytesCountedBoth BytesCounted = "both"
)
// LookupSource is where each client's AS number and country are looked up,
// a value of SWW_WAF_LOOKUP_SOURCE.
type LookupSource string
// The values of SWW_WAF_LOOKUP_SOURCE.
const (
// LookupSourceGeoJS asks GeoJS.
LookupSourceGeoJS LookupSource = "geojs"
// LookupSourceFile looks clients up in the lookup database, the file
// SWW_WAF_LOOKUP_DB_PATH names.
LookupSourceFile LookupSource = "file"
// LookupSourceOff looks no client up.
LookupSourceOff LookupSource = "off"
)
// ReputationAction is what is done with a client a reputation source
// lists, a value of SWW_WAF_BLOCKLIST_ACTION and of
// SWW_WAF_DNSBL_ABUSEIPDB_ACTION.
type ReputationAction string
// The values of SWW_WAF_BLOCKLIST_ACTION and SWW_WAF_DNSBL_ABUSEIPDB_ACTION.
const (
// ReputationActionDeny refuses the client's requests.
ReputationActionDeny ReputationAction = "deny"
// ReputationActionLimit gives the client a percentage of every limit,
// written limit:<percent> in the setting.
ReputationActionLimit ReputationAction = "limit"
// ReputationActionLog only notes the hit in the log line and raises an
// alert.
ReputationActionLog ReputationAction = "log"
)
// CoreRuleSetMode is what the Core Rule Set does, a value of
// SWW_WAF_CORE_RULE_SET_MODE.
type CoreRuleSetMode string
// The values of SWW_WAF_CORE_RULE_SET_MODE.
const (
// CoreRuleSetModeOff runs no request through the Core Rule Set.
CoreRuleSetModeOff = off
CoreRuleSetModeOff CoreRuleSetMode = "off"
// CoreRuleSetModeDetect logs and alerts a match, and refuses nothing.
CoreRuleSetModeDetect = "detect"
CoreRuleSetModeDetect CoreRuleSetMode = "detect"
// CoreRuleSetModeBlock refuses a match with 403.
CoreRuleSetModeBlock = "block"
CoreRuleSetModeBlock CoreRuleSetMode = "block"
)
// fileSource is the SWW_WAF_LOOKUP_SOURCE that looks clients up in the
// lookup database, the file SWW_WAF_LOOKUP_DB_PATH names.
const fileSource = "file"
const (
day = 24 * time.Hour
ipv4Bits = 32
@@ -558,10 +604,13 @@ func WAFFromEnvironment(v *viper.Viper) (*WAF, error) {
),
BytesLimitPerHour: env.size("SWW_WAF_BYTES_LIMIT_PER_HOUR", "50G"),
BytesLimitPerDay: env.size("SWW_WAF_BYTES_LIMIT_PER_DAY", "100G"),
BytesCounted: env.bytesCounted("SWW_WAF_BYTES_COUNTED", "both"),
BytesCounted: env.bytesCounted(
"SWW_WAF_BYTES_COUNTED",
BytesCountedBoth,
),
LookupSource: env.lookupSource(
"SWW_WAF_LOOKUP_SOURCE",
"geojs",
LookupSourceGeoJS,
),
LookupDBPath: env.value("SWW_WAF_LOOKUP_DB_PATH", ""),
ASNAndCountryLookupTimeout: env.milliseconds(
@@ -758,9 +807,9 @@ func WAFFromEnvironment(v *viper.Viper) (*WAF, error) {
cfg.InstanceName, cfg.LogRemoteURL != nil)
// The app gets the headers by default, unless no client is looked up.
cfg.AddLookupHeaders = env.boolean("SWW_WAF_ADD_LOOKUP_HEADERS",
strconv.FormatBool(cfg.LookupSource != off))
strconv.FormatBool(cfg.LookupSource != LookupSourceOff))
cfg.BlocklistAction, cfg.BlocklistLimitPercent = env.action(
"SWW_WAF_BLOCKLIST_ACTION", "deny")
"SWW_WAF_BLOCKLIST_ACTION", string(ReputationActionDeny))
cfg.DNSBLAbuseIPDBAction, cfg.DNSBLAbuseIPDBLimitPercent = env.action(
"SWW_WAF_DNSBL_ABUSEIPDB_ACTION", "limit:25")
@@ -858,9 +907,12 @@ func (e *environment) count(name, defaultValue string) int64 {
// bytesCounted reads the setting that is which body bytes count toward the
// byte limits and the byte anomaly thresholds: response, request or both.
func (e *environment) bytesCounted(name, defaultValue string) string {
value := e.value(name, defaultValue)
if value != "response" && value != "request" && value != "both" {
func (e *environment) bytesCounted(
name string, defaultValue BytesCounted,
) BytesCounted {
value := BytesCounted(e.value(name, string(defaultValue)))
if value != BytesCountedResponse && value != BytesCountedRequest &&
value != BytesCountedBoth {
e.check(name, fmt.Errorf("%q %w", value, errNotBytesCounted))
}
@@ -905,8 +957,10 @@ func (e *environment) trapPaths(name string) []string {
// coreRuleSetMode reads the setting that is what the Core Rule Set does: off,
// detect or block.
func (e *environment) coreRuleSetMode(name, defaultValue string) string {
mode := e.value(name, defaultValue)
func (e *environment) coreRuleSetMode(
name string, defaultValue CoreRuleSetMode,
) CoreRuleSetMode {
mode := CoreRuleSetMode(e.value(name, string(defaultValue)))
if mode != CoreRuleSetModeOff && mode != CoreRuleSetModeDetect &&
mode != CoreRuleSetModeBlock {
e.check(name, fmt.Errorf("%q %w", mode, errNotCoreRuleSetMode))
@@ -1056,20 +1110,24 @@ func (e *environment) refresh(name, defaultValue string) time.Duration {
// action reads a setting that is what is done with a client a blocklist
// or a DNSBL zone lists: deny, log, or limit:<percent>, which it returns
// as limit and the percentage.
func (e *environment) action(name, defaultValue string) (string, int64) {
value := e.value(name, defaultValue)
if value == "deny" || value == "log" {
func (e *environment) action(
name, defaultValue string,
) (ReputationAction, int64) {
value := ReputationAction(e.value(name, defaultValue))
if value == ReputationActionDeny || value == ReputationActionLog {
return value, 0
}
percentText, isLimit := strings.CutPrefix(value, "limit:")
percentText, isLimit := strings.CutPrefix(
string(value), string(ReputationActionLimit)+":",
)
percent, err := ParsePercent(percentText)
if !isLimit || err != nil {
e.check(name, fmt.Errorf("%q %w", value, errNotAction))
}
return "limit", percent
return ReputationActionLimit, percent
}
// zones reads the setting that is the list of DNSBL zones. It is empty by
@@ -1120,9 +1178,12 @@ func (e *environment) routeWatchURL(name string) string {
// lookupSource reads the setting that is where clients are looked up:
// geojs, file, or off.
func (e *environment) lookupSource(name, defaultValue string) string {
source := e.value(name, defaultValue)
if source != "geojs" && source != fileSource && source != off {
func (e *environment) lookupSource(
name string, defaultValue LookupSource,
) LookupSource {
source := LookupSource(e.value(name, string(defaultValue)))
if source != LookupSourceGeoJS && source != LookupSourceFile &&
source != LookupSourceOff {
e.check(name, fmt.Errorf("%q %w", source, errNotLookupSource))
}
@@ -1134,12 +1195,12 @@ func (e *environment) lookupSource(name, defaultValue string) string {
// one source at a time.
func (e *environment) checkLookupDBPath(cfg *WAF) {
switch {
case cfg.LookupSource == fileSource && cfg.LookupDBPath == "":
case cfg.LookupSource == LookupSourceFile && cfg.LookupDBPath == "":
e.check("SWW_WAF_LOOKUP_SOURCE", fmt.Errorf(
"is file while SWW_WAF_LOOKUP_DB_PATH is unset; %w",
errNeedsDBPath,
))
case cfg.LookupSource != fileSource && cfg.LookupDBPath != "":
case cfg.LookupSource != LookupSourceFile && cfg.LookupDBPath != "":
e.check("SWW_WAF_LOOKUP_DB_PATH", fmt.Errorf(
"is set while SWW_WAF_LOOKUP_SOURCE is %s; %w",
cfg.LookupSource,
@@ -1164,7 +1225,7 @@ func (e *environment) checkCountriesAndLookups(cfg *WAF) {
}
}
if cfg.LookupSource != off {
if cfg.LookupSource != LookupSourceOff {
return
}
+6 -5
View File
@@ -614,7 +614,7 @@ func TestCoreRuleSetSettings(t *testing.T) {
},
{
environment{
coreRuleSetMode: config.CoreRuleSetModeDetect,
coreRuleSetMode: string(config.CoreRuleSetModeDetect),
coreRuleSetParanoiaLevel: "4",
coreRuleSetScoreThreshold: "10",
coreRuleSetDisabledRules: "942100, 920350",
@@ -2033,8 +2033,9 @@ func TestReputationSettingsAsSet(t *testing.T) {
}
for _, tc := range []struct {
value, action string
percent int64
value string
action config.ReputationAction
percent int64
}{
{actionDeny, actionDeny, 0},
{actionLog, actionLog, 0},
@@ -2167,7 +2168,7 @@ func TestDNSBLSettingsAsSet(t *testing.T) {
// want are the zones, resolver, action and percent config.WAF gives.
want []string
wantResolver string
wantAction string
wantAction config.ReputationAction
wantPercent int64
}{
{
@@ -3049,7 +3050,7 @@ func TestLogsEachSettingWithItsValue(t *testing.T) {
logLevel: "info",
ruleFilesDir: "/etc/sww/rules.d",
ruleFilesEnabled: "true",
coreRuleSetMode: config.CoreRuleSetModeBlock,
coreRuleSetMode: string(config.CoreRuleSetModeBlock),
coreRuleSetParanoiaLevel: "1",
coreRuleSetScoreThreshold: "5",
coreRuleSetDisabledRules: defaultCoreRuleSetDisabledRules,
+65 -43
View File
@@ -12,10 +12,11 @@
package coreruleset
import (
"embed"
_ "embed"
"errors"
"fmt"
"io"
"maps"
"net"
"net/http"
"net/netip"
@@ -34,7 +35,7 @@ import (
// directives are the Core Rule Set as sww runs it, with the
// paranoia level for %d, bodyDirectives for the first %s while
// SWW_WAF_CORE_RULE_SET_BODY_INSPECTION_SIZE is set, the exclusions file for
// the second, and the fixes of every app in appFixes for the third.
// the second, and the directives of every app in apps for the third.
// Each rule sww adds has an id from 900000 to 900999, the ids the
// Core Rule Set keeps for the rules that set it up, which
// SWW_WAF_CORE_RULE_SET_DISABLED_RULES and the exclusions file refuse, so that no
@@ -53,12 +54,13 @@ Include @crs-setup.conf.example
SecAction "id:900000,phase:1,pass,nolog,\
setvar:tx.blocking_paranoia_level=%d"
# PUT, PATCH and DELETE are allowed besides GET, HEAD, POST and OPTIONS.
# The first change: PUT, PATCH and DELETE are allowed besides GET, HEAD,
# POST and OPTIONS.
SecAction "id:900200,phase:1,pass,nolog,\
setvar:'tx.allowed_methods=GET HEAD POST OPTIONS PUT PATCH DELETE'"
# Expect and Content-Encoding are taken off the Core Rule Set's list of the
# headers it refuses. Content-Encoding goes back on it for a
# The second: Expect and Content-Encoding are taken off the Core Rule Set's
# list of the headers it refuses. Content-Encoding goes back on it for a
# body the Core Rule Set reads (900260 in bodyDirectives).
SecAction "id:900250,phase:1,pass,nolog,\
setvar:'tx.restricted_headers_basic=/proxy/ /lock-token/ /content-range/ \
@@ -84,17 +86,18 @@ SecRule ARGUMENTS_LIMIT_REACHED "@eq 1" "id:900300,phase:2,pass,\
# The apps' fixes come before the rules for the same reason.
%s
# Only the rules for requests are loaded, and no response is inspected.
# The fifth: only the rules for requests are loaded, and no response is
# inspected.
Include @owasp_crs/REQUEST-*.conf
# redirect_uri is not checked for a URL naming an IP address or
# The third: redirect_uri is not checked for a URL naming an IP address or
# localhost. Coraza matches a parameter name here without regard to case.
# ARGS holds the fields of a form data or multipart body Coraza reads as
# well as the query parameters, so a field of this name is left out too.
SecRuleUpdateTargetById 931100 "!ARGS:redirect_uri"
SecRuleUpdateTargetById 934110 "!ARGS:redirect_uri"
# Referer is not checked for a Unix command without arguments
# The fourth: Referer is not checked for a Unix command without arguments
# or for Java starting a process.
SecRuleUpdateTargetById 932340 "!REQUEST_HEADERS:Referer"
SecRuleUpdateTargetById 944110 "!REQUEST_HEADERS:Referer"
@@ -119,9 +122,9 @@ SecRule REQUEST_HEADERS:Content-Type \
"@rx ^(?:application|text)/(?:[a-z0-9.-]+[+])?json" \
"id:900420,phase:1,pass,nolog,t:none,t:lowercase,ctl:requestBodyProcessor=JSON"
# Content-Encoding goes back on the list of the headers the Core Rule Set
# refuses (900250 in directives) for a body of a kind it reads, since a
# compressed body cannot be inspected.
# The rest of the second change: Content-Encoding is refused again on a
# body of a kind the Core Rule Set reads, since a compressed body cannot be
# inspected.
SecRule REQBODY_PROCESSOR "@rx ^(?:URLENCODED|MULTIPART|JSON|XML)$" \
"id:900260,phase:1,pass,nolog,\
setvar:'tx.restricted_headers_basic=%%{tx.restricted_headers_basic} \
@@ -142,19 +145,15 @@ SecRule MULTIPART_STRICT_ERROR "!@eq 0" "id:900450,phase:2,pass,\
severity:'CRITICAL',setvar:'tx.inbound_anomaly_score_pl1=+5'"
`
// appFixes holds the fixes of each app whose ordinary requests the Core
// Rule Set would refuse, in a file named for the app by the name
// SWW_WAF_APPS gives it, such as apps/gitea.conf. The fixes are Coraza
// exclusions, loaded before the Core Rule Set's rules. Their first rule
// skips to the marker they end with, unless Inspect sets the transaction
// variable named for the app, such as app_gitea. The skip reaches only
// phase 1, so a rule of a later phase must check that variable itself.
//
//go:embed apps/*.conf
var appFixes embed.FS
// app is what an app's file in appFixes cannot say about it.
// app is an app whose ordinary requests the Core Rule Set would refuse,
// with the fixes that let them pass.
type app struct {
// directives are Coraza exclusions, loaded before the Core Rule Set's
// rules. Their first rule skips to the marker they end with, unless
// Inspect sets the transaction variable named for the app, such as
// app_gitea. The skip reaches only phase 1, so a rule of a later phase
// must check that variable itself.
directives string
// cookies are the cookies the Core Rule Set reads a request without.
cookies []string
// forkOf names the app this one is a fork of, "" for none. A hostname
@@ -162,34 +161,60 @@ type app struct {
forkOf string
}
// apps are the apps that need more than their file in appFixes, by the
// name SWW_WAF_APPS gives each. flash is Forgejo's name for gitea_flash.
//go:embed gitea.conf
var giteaDirectives string
//go:embed wordpress.conf
var wordpressDirectives string
//go:embed drupal.conf
var drupalDirectives string
//go:embed mediawiki.conf
var mediawikiDirectives string
//go:embed mastodon.conf
var mastodonDirectives string
//go:embed vaultwarden.conf
var vaultwardenDirectives string
//go:embed nextcloud.conf
var nextcloudDirectives string
//go:embed forgejo.conf
var forgejoDirectives string
//go:embed ghost.conf
var ghostDirectives string
// apps are the apps whose fixes sww has, by the name SWW_WAF_APPS gives each.
// flash is Forgejo's name for gitea_flash.
//
//nolint:gochecknoglobals // a constant cannot be a map
var apps = map[string]app{
"drupal": {directives: drupalDirectives},
"forgejo": {
cookies: []string{"flash"},
forkOf: "gitea",
directives: forgejoDirectives,
cookies: []string{"flash"},
forkOf: "gitea",
},
"ghost": {directives: ghostDirectives},
"gitea": {
cookies: []string{"gitea_flash", "redirect_to"},
directives: giteaDirectives,
cookies: []string{"gitea_flash", "redirect_to"},
},
"mastodon": {directives: mastodonDirectives},
"mediawiki": {directives: mediawikiDirectives},
"nextcloud": {directives: nextcloudDirectives},
"vaultwarden": {directives: vaultwardenDirectives},
"wordpress": {directives: wordpressDirectives},
}
// AppNames returns the names of the apps whose fixes sww has, in
// alphabetical order.
func AppNames() []string {
// An embedded directory always reads.
files, _ := appFixes.ReadDir("apps")
names := make([]string, len(files))
for i, file := range files {
names[i] = strings.TrimSuffix(file.Name(), ".conf")
}
slices.Sort(names)
return names
return slices.Sorted(maps.Keys(apps))
}
// Params are what New needs.
@@ -348,11 +373,8 @@ func New(params Params) (*CoreRuleSet, error) {
}
var fixes strings.Builder
for _, name := range AppNames() {
// An embedded file always reads.
file, _ := appFixes.ReadFile("apps/" + name + ".conf")
fixes.WriteString(string(file) + "\n")
fixes.WriteString(apps[name].directives + "\n")
}
text := fmt.Sprintf(directives, params.ParanoiaLevel, body, exclusions,
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
+3 -3
View File
@@ -17,7 +17,7 @@ const (
)
// giteaSearches are searches of the pages gitea and Forgejo share whose q
// skips the lists of system files and command names, in gitea's fixes.
// skips the lists of system files and command names, in the eighth change.
// gitea serves its admin pages under /-/admin/ and Forgejo under /admin/.
//
//nolint:gochecknoglobals // a constant cannot be a list
@@ -222,8 +222,8 @@ func TestGiteaSearchExclusionsAreNarrow(t *testing.T) {
}
// Under /admin/, where Forgejo serves its admin pages, only the hard
// values are left out: Drupal's fixes let them through in q on every path
// there.
// values are left out: the tenth change lets them through in q on every
// path there, for Drupal.
for _, target := range []string{
"/admin/packages/delete?q=",
"/admin/packages?other=",
File renamed without changes.
File renamed without changes.
@@ -37,10 +37,9 @@
SecRule &TX:app_nextcloud "@eq 0" \
"id:900090,phase:1,pass,nolog,skipAfter:END_NEXTCLOUD"
# The sync client and the Files app use these WebDAV methods besides PUT,
# PATCH and DELETE, which sww allows for every app, on /remote.php/dav/ and
# /public.php/dav/ and on the older /remote.php/webdav/ and /public.php/webdav/.
# 911100 refused each:
# The sync client and the Files app use these WebDAV methods besides those
# the first change allows, on /remote.php/dav/ and /public.php/dav/ and on the
# older /remote.php/webdav/ and /public.php/webdav/. 911100 refused each:
# PROPFIND, PROPPATCH: listing a folder, marking a file a favourite
# MKCOL: making a folder, and starting a chunked upload
# COPY, MOVE: copying, moving and renaming, and ending a chunked upload
File renamed without changes.
+30 -18
View File
@@ -1284,18 +1284,18 @@ func TestBansWrittenOnceBansWriteDelayAfterABan(t *testing.T) {
// A second ban, made while the first waits to be written, puts the
// write off no further, and is written with it.
first, _ := params.Ledger.BanForLimit(
first, _ := params.Ledger.Ban(
netip.MustParsePrefix("203.0.113.9/32"),
midnight(),
bans.Notes{},
bans.Grounds{Cause: bans.CauseLimit},
)
time.Sleep(5 * time.Second)
second, _ := params.Ledger.BanForLimit(
second, _ := params.Ledger.Ban(
netip.MustParsePrefix("203.0.113.10/32"),
midnight(),
bans.Notes{},
bans.Grounds{Cause: bans.CauseLimit},
)
time.Sleep(5*time.Second - time.Nanosecond)
@@ -1358,16 +1358,16 @@ func TestEditJustBeforeAScheduledWriteSurvivesIt(t *testing.T) {
run(t, load(t, params).Run)
// A ban, and bans.json written with it.
params.Ledger.BanForLimit(netip.MustParsePrefix("203.0.113.9/32"),
midnight(), bans.Notes{})
params.Ledger.Ban(netip.MustParsePrefix("203.0.113.9/32"),
midnight(), bans.Grounds{Cause: bans.CauseLimit})
time.Sleep(params.BansWriteDelay)
synctest.Wait()
// A second ban is to be written BansWriteDelay later. Just before
// then, an admin saves bans.json with the first ban lifted and
// another added.
params.Ledger.BanForLimit(netip.MustParsePrefix("203.0.113.10/32"),
midnight(), bans.Notes{})
params.Ledger.Ban(netip.MustParsePrefix("203.0.113.10/32"),
midnight(), bans.Grounds{Cause: bans.CauseLimit})
time.Sleep(params.BansWriteDelay - time.Nanosecond)
synctest.Wait()
edit(t, dir, bansJSON, permanentBansJSON)
@@ -1483,10 +1483,10 @@ func TestFailedWriteLeavesTheFileAsItWas(t *testing.T) {
t.Fatalf("mkdir: %v", err)
}
params.Ledger.BanForLimit(
params.Ledger.Ban(
netip.MustParsePrefix("203.0.113.9/32"),
midnight(),
bans.Notes{},
bans.Grounds{Cause: bans.CauseLimit},
)
params.Limiter.Count(
netip.MustParsePrefix("203.0.113.9/32"),
@@ -1556,10 +1556,10 @@ func TestWritesAreCountedInTheMetrics(t *testing.T) {
t.Fatalf("mkdir: %v", err)
}
params.Ledger.BanForLimit(
params.Ledger.Ban(
netip.MustParsePrefix("203.0.113.9/32"),
midnight(),
bans.Notes{},
bans.Grounds{Cause: bans.CauseLimit},
)
err = files.WriteAll()
@@ -2135,7 +2135,11 @@ func TestBanLiftedByAnEditWhileRunning(t *testing.T) {
// The ban that liftedBansJSON lifts, before it is lifted.
netblock := netip.MustParsePrefix(liftedClient + "/32")
params.Ledger.BanForLimit(netblock, midnight(), bans.Notes{})
params.Ledger.Ban(
netblock,
midnight(),
bans.Grounds{Cause: bans.CauseLimit},
)
_, banned, _ := params.Ledger.Find(netblock.Addr(), afterLifting())
if !banned {
@@ -2510,11 +2514,15 @@ func fill(params state.Params) {
client := netip.MustParsePrefix("203.0.113.9/32")
params.Ledger.Load([]bans.Ban{permanentBan()})
params.Ledger.BanForLimit(client, now, bans.Notes{
ASN: asn, ASName: asName, Country: "DE", Limit: 1,
params.Ledger.Ban(client, now, bans.Grounds{
Cause: bans.CauseLimit,
Notes: bans.Notes{ASN: asn, ASName: asName, Country: "DE", Limit: 1},
})
params.Ledger.BanForAttack(netip.MustParsePrefix("192.0.2.1/32"), now,
bans.Notes{RuleID: "env-file", Target: "path"})
params.Ledger.Ban(netip.MustParsePrefix("192.0.2.1/32"), now,
bans.Grounds{
Cause: bans.CauseAttack,
Notes: bans.Notes{RuleID: "env-file", Target: "path"},
})
for _, c := range []string{"2001:db8::/64", "203.0.113.9/32", "192.0.2.1/32"} {
params.Limiter.Count(netip.MustParsePrefix(c), now, whole)
@@ -2700,7 +2708,11 @@ func wantLiftedBanKept(
}
// Were the lifted ban counted, the next would last three hours.
ban, _ := ledger.BanForLimit(netblock, afterLifting(), bans.Notes{})
ban, _ := ledger.Ban(
netblock,
afterLifting(),
bans.Grounds{Cause: bans.CauseLimit},
)
if ban.Expires.Sub(ban.Start) != time.Hour {
t.Errorf(
"the next ban lasts %s, want 1h",
+3 -1
View File
@@ -660,7 +660,9 @@ func TestCoreRuleSetExclusionsFileThatCannotBeUsedStopsTheStartWhileTheCoreRuleS
status := run(t.Context(), map[string]string{
listenAddr: localhost + ":0", stateDir: t.TempDir(), ruleFilesDir: t.TempDir(),
"SWW_WAF_CORE_RULE_SET_MODE": config.CoreRuleSetModeOff,
"SWW_WAF_CORE_RULE_SET_MODE": string(
config.CoreRuleSetModeOff,
),
"SWW_WAF_CORE_RULE_SET_EXCLUSIONS_FILE": path,
}, out)
if status != 1 {
+29 -4
View File
@@ -2,6 +2,7 @@ package sww
import (
"context"
"fmt"
"io"
"log/slog"
"net"
@@ -175,11 +176,35 @@ func newRuleFiles(
return ruleFiles, nil
}
// newCoreRuleSet returns the Core Rule Set, as waf.NewCoreRuleSet does.
// newCoreRuleSet returns the Core Rule Set at SWW_WAF_CORE_RULE_SET_PARANOIA_LEVEL,
// without the rules SWW_WAF_CORE_RULE_SET_DISABLED_RULES switches off, reading bodies
// up to SWW_WAF_CORE_RULE_SET_BODY_INSPECTION_SIZE, with the exclusions in
// SWW_WAF_CORE_RULE_SET_EXCLUSIONS_FILE and the fixes of the apps in SWW_WAF_APPS,
// or nil while SWW_WAF_CORE_RULE_SET_MODE is off. An exclusions file that
// coreruleset.New refuses is an error in every mode. While the mode is off, a
// set file is loaded with the Core Rule Set only to check it.
func newCoreRuleSet(cfg *config.WAF) (*coreruleset.CoreRuleSet, error) {
crs, err := waf.NewCoreRuleSet(cfg)
if cfg.CoreRuleSetMode == config.CoreRuleSetModeOff &&
cfg.CoreRuleSetExclusionsFile == "" {
return nil, nil //nolint:nilnil // off is no Core Rule Set, and no error
}
crs, err := coreruleset.New(coreruleset.Params{
ParanoiaLevel: cfg.CoreRuleSetParanoiaLevel,
DisabledRules: cfg.CoreRuleSetDisabledRules,
BodyInspectionSize: cfg.CoreRuleSetBodyInspectionSize,
ExclusionsFile: cfg.CoreRuleSetExclusionsFile,
Apps: cfg.Apps,
})
if err != nil {
return nil, startError{"cannot use the Core Rule Set", err}
return nil, startError{
"cannot use the Core Rule Set",
fmt.Errorf("SWW_WAF_CORE_RULE_SET_EXCLUSIONS_FILE: %w", err),
}
}
if cfg.CoreRuleSetMode == config.CoreRuleSetModeOff {
return nil, nil //nolint:nilnil // off is no Core Rule Set, and no error
}
return crs, nil
@@ -192,7 +217,7 @@ func newLookupFile(
lc fx.Lifecycle, cfg *config.WAF, processLog *slog.Logger,
alertQueue *alerts.Queue,
) (*lookup.File, error) {
if cfg.LookupSource != "file" {
if cfg.LookupSource != config.LookupSourceFile {
return nil, nil //nolint:nilnil // no lookup database, and no error
}
+1 -1
View File
@@ -54,7 +54,7 @@ func (abuseIPDB) check(
"scored by AbuseIPDB at or over SWW_WAF_ABUSEIPDB_MIN_SCORE",
)
if rq.h.config.DNSBLAbuseIPDBAction != deny {
if rq.h.config.DNSBLAbuseIPDBAction != config.ReputationActionDeny {
return nil
}
+2 -2
View File
@@ -37,10 +37,10 @@ func TestAdminEndpointsAreOffWhileTheTokenIsUnset(t *testing.T) {
"",
map[string]string{metricsToken: token},
)
server.Ledger.BanForLimit(
server.Ledger.Ban(
netip.MustParsePrefix(otherClient+"/32"),
clk.Now(),
bans.Notes{},
bans.Grounds{Cause: bans.CauseLimit},
)
before := server.Ledger.Snapshot()
+3 -5
View File
@@ -130,11 +130,9 @@ func TestObserveModeRaisesTheBanAlertsItWouldHave(t *testing.T) {
// A ban for a clear sign of attack, which a request under it would make
// permanent.
group := netip.MustParsePrefix(ipv6Group)
attackBan, _ := server.Ledger.BanForAttack(
group,
start,
bans.Notes{RuleID: "probe"},
)
attackBan, _ := server.Ledger.Ban(group, start, bans.Grounds{
Cause: bans.CauseAttack, Notes: bans.Notes{RuleID: "probe"},
})
// The third request breaks the limit, and so does the fourth, within the
// cooldown, which raises nothing. The probe is a clear sign of attack.
+2 -2
View File
@@ -686,10 +686,10 @@ func askDirectly(
// banClient bans client at now.
func banClient(_ *testing.T, server *waf.Server, now time.Time) {
server.Ledger.BanForLimit(
server.Ledger.Ban(
netip.MustParsePrefix(client+"/32"),
now,
bans.Notes{},
bans.Grounds{Cause: bans.CauseLimit},
)
}
+77 -57
View File
@@ -86,13 +86,14 @@ func (rq *request) countedBytes() int64 {
response, request := rq.out.bytes, rq.requestBytes
switch rq.h.config.BytesCounted {
case "response":
case config.BytesCountedResponse:
return response
case "request":
case config.BytesCountedRequest:
return request
default: // both
return response + request
case config.BytesCountedBoth:
}
return response + request
}
// banForLimit bans the client's netblock at now for a broken limit, the
@@ -102,9 +103,8 @@ func (rq *request) countedBytes() int64 {
// one whose bytes broke a byte limit, the refusal for one that broke the
// refusal limit. The ban's notes give the client's limit percentage for a
// rate limit or a byte limit; the refusal limit is not lowered. The ban sets
// the client's counters back to zero. In observe mode it makes no ban and
// sets nothing back, and raises the alert for the ban it would have made,
// if that alert would be sent.
// the client's counters back to zero. In observe mode, banClient says what
// happens instead.
func (rq *request) banForLimit(
now time.Time,
hit ratelimit.Hit,
@@ -122,23 +122,11 @@ func (rq *request) banForLimit(
rq.line.Offence = requestlog.OffenceLimit
rq.h.metrics.LimitHit(hit)
netblock := banNetblock(rq.h.config, rq.client)
if rq.h.config.Observe &&
!rq.wouldAlertBan(netblock, now, bans.CauseLimit) {
return
}
notes := bans.Notes{
ASN: rq.line.ASN,
ASName: rq.line.ASName,
Country: rq.line.Country,
Kind: hit.Kind,
Limit: hit.Limit,
Window: hit.Window,
Count: hit.Count,
Reputation: rq.reputation,
Request: rq.noted(now, status),
Requests: rq.netblockRequests(netblock),
Kind: hit.Kind,
Limit: hit.Limit,
Window: hit.Window,
Count: hit.Count,
}
switch hit.Kind {
@@ -148,47 +136,54 @@ func (rq *request) banForLimit(
notes.LimitPercent, notes.LimitPercentSetting = rq.bytesLimitPercent.logged()
}
if rq.h.config.Observe {
ban, wouldBan := rq.h.ledger.WouldBanForLimit(netblock, now, notes)
if wouldBan {
rq.alertBan(ban)
}
return
}
ban, made := rq.h.ledger.BanForLimit(netblock, now, notes)
rq.h.limiter.Reset(clientGroup(rq.h.config, rq.client))
rq.line.BanExpires = banExpires(ban)
if made {
rq.alertBan(ban)
}
rq.banClient(
now,
status,
bans.Grounds{Cause: bans.CauseLimit, Notes: notes},
)
}
// banForAttack bans the client's netblock at now for a clear sign of
// attack, which notes name: the ban rule that matched, or the trap path
// asked for. It fills in the rest of the notes. In observe mode it makes
// no ban, and raises the alert for the ban it would have made, if that
// alert would be sent. It returns the action for that ban.
// asked for. It returns the action for that ban. In observe mode, banClient
// says what happens instead.
func (rq *request) banForAttack(now time.Time, notes bans.Notes) string {
netblock := banNetblock(rq.h.config, rq.client)
if rq.h.config.Observe &&
!rq.wouldAlertBan(netblock, now, bans.CauseAttack) {
return banAction(
rq.h.ledger.WouldBePermanent(netblock, now, bans.CauseAttack),
)
}
return rq.banClient(
now,
rq.h.config.BanResponse,
bans.Grounds{Cause: bans.CauseAttack, Notes: notes},
)
}
notes.ASN = rq.line.ASN
notes.ASName = rq.line.ASName
notes.Country = rq.line.Country
notes.Reputation = rq.reputation
notes.Request = rq.noted(now, rq.h.config.BanResponse)
notes.Requests = rq.netblockRequests(netblock)
// banClient bans the client's netblock at now on grounds, and returns the
// action for that ban. It adds to the notes what the request tells of the
// client, with status, what the client was sent. A ban for a broken limit
// sets the client's counters back to zero.
//
// In observe mode it makes no ban and sets nothing back. It raises the
// alert for the ban it would have made, if that alert would be sent.
func (rq *request) banClient(
now time.Time,
status int,
grounds bans.Grounds,
) string {
netblock := banNetblock(rq.h.config, rq.client)
if rq.h.config.Observe {
ban, wouldBan := rq.h.ledger.WouldBanForAttack(netblock, now, notes)
if !rq.wouldAlertBan(netblock, now, grounds.Cause) {
return banAction(
rq.h.ledger.WouldBePermanent(netblock, now, grounds.Cause),
)
}
grounds.Notes = rq.withClientNotes(
grounds.Notes,
now,
status,
netblock,
)
ban, wouldBan := rq.h.ledger.WouldBan(netblock, now, grounds)
if wouldBan {
rq.alertBan(ban)
}
@@ -196,7 +191,13 @@ func (rq *request) banForAttack(now time.Time, notes bans.Notes) string {
return banAction(ban.Permanent())
}
ban, made := rq.h.ledger.BanForAttack(netblock, now, notes)
grounds.Notes = rq.withClientNotes(grounds.Notes, now, status, netblock)
ban, made := rq.h.ledger.Ban(netblock, now, grounds)
if grounds.Cause == bans.CauseLimit {
rq.h.limiter.Reset(clientGroup(rq.h.config, rq.client))
}
rq.line.BanExpires = banExpires(ban)
if made {
@@ -206,6 +207,25 @@ func (rq *request) banForAttack(now time.Time, notes bans.Notes) string {
return banAction(ban.Permanent())
}
// withClientNotes is notes with what the request tells of the client
// added: its AS, country and reputation, the request at now with status,
// and how many requests netblock, the client's, has sent.
func (rq *request) withClientNotes(
notes bans.Notes,
now time.Time,
status int,
netblock netip.Prefix,
) bans.Notes {
notes.ASN = rq.line.ASN
notes.ASName = rq.line.ASName
notes.Country = rq.line.Country
notes.Reputation = rq.reputation
notes.Request = rq.noted(now, status)
notes.Requests = rq.netblockRequests(netblock)
return notes
}
// wouldAlertBan reports whether the alert for a ban on netblock for cause
// made at now would be sent. In observe mode the ban the request would
// have made is worked out only then, at most once per
+3 -2
View File
@@ -63,7 +63,8 @@ func (rq *request) limitPercentages() (percentage, percentage) {
}
blocklisted := percentage{percent: whole}
if rq.blocklisted && cfg.BlocklistAction == "limit" {
if rq.blocklisted &&
cfg.BlocklistAction == config.ReputationActionLimit {
blocklisted = percentage{
cfg.BlocklistLimitPercent,
"SWW_WAF_BLOCKLIST_ACTION",
@@ -72,7 +73,7 @@ func (rq *request) limitPercentages() (percentage, percentage) {
dnsblAbuseIPDBListed := percentage{percent: whole}
if (rq.dnsblListed || rq.abuseIPDBHit) &&
cfg.DNSBLAbuseIPDBAction == "limit" {
cfg.DNSBLAbuseIPDBAction == config.ReputationActionLimit {
dnsblAbuseIPDBListed = percentage{
cfg.DNSBLAbuseIPDBLimitPercent,
"SWW_WAF_DNSBL_ABUSEIPDB_ACTION",
+3 -1
View File
@@ -7,6 +7,7 @@ import (
"context"
"time"
"sneak.berlin/go/smallwebwaf/internal/config"
"sneak.berlin/go/smallwebwaf/internal/requestlog"
)
@@ -41,5 +42,6 @@ func (blocklists) check(
// client, given listedBy, the blocklists that list it: it does when one lists
// it and SWW_WAF_BLOCKLIST_ACTION is deny.
func (rq *request) refusedByBlocklists(listedBy []string) bool {
return len(listedBy) > 0 && rq.h.config.BlocklistAction == deny
return len(listedBy) > 0 && rq.h.config.BlocklistAction ==
config.ReputationActionDeny
}
+2 -2
View File
@@ -47,7 +47,7 @@ func (coreRuleSet) check(
rq.line.CoreRuleSetScore = &result.Score
for _, id := range result.RuleIDs {
rq.h.metrics.CoreRuleSetMatched(cfg.CoreRuleSetMode, id)
rq.h.metrics.CoreRuleSetMatched(string(cfg.CoreRuleSetMode), id)
}
threshold := cfg.CoreRuleSetScoreThreshold
@@ -99,7 +99,7 @@ func (rq *request) alertCoreRuleSetMatch(result coreruleset.Result) {
switch {
case rq.h.config.CoreRuleSetMode == config.CoreRuleSetModeDetect:
detail["mode"] = config.CoreRuleSetModeDetect
detail["mode"] = string(config.CoreRuleSetModeDetect)
case rq.h.config.Observe:
detail["mode"] = "observe"
}
+2 -1
View File
@@ -32,7 +32,8 @@ func (dnsbl) check(ctx context.Context, rq *request, _ time.Time) *verdict {
rq.addTag(config.DNSBLTagPrefix + strings.ToLower(zone))
}
if !rq.dnsblListed || rq.h.config.DNSBLAbuseIPDBAction != deny {
if !rq.dnsblListed ||
rq.h.config.DNSBLAbuseIPDBAction != config.ReputationActionDeny {
return nil
}
+3 -2
View File
@@ -49,7 +49,8 @@ func (rq *request) startLookUp(
) <-chan lookupResult {
result := make(chan lookupResult, 1)
if rq.h.config.LookupSource == "off" || !canBePlaced(rq.client) {
if rq.h.config.LookupSource == config.LookupSourceOff ||
!canBePlaced(rq.client) {
result <- lookupResult{}
return result
@@ -57,7 +58,7 @@ func (rq *request) startLookUp(
client := clientGroup(rq.h.config, rq.client)
if rq.h.config.LookupSource == "file" {
if rq.h.config.LookupSource == config.LookupSourceFile {
result <- lookupResult{
lookedUp: true, answer: rq.h.lookupFile.LookUp(client),
}
+2 -37
View File
@@ -6,7 +6,6 @@ package waf
import (
"cmp"
"fmt"
"io"
"log/slog"
"net/http"
@@ -83,8 +82,8 @@ type Params struct {
// LookupFile is the lookup database they are looked up in while
// SWW_WAF_LOOKUP_SOURCE is file, and nil otherwise.
LookupFile *lookup.File
// CoreRuleSet is what NewCoreRuleSet returns: what inspects requests
// while SWW_WAF_CORE_RULE_SET_MODE is not off, and nil while it is.
// CoreRuleSet is what inspects requests while
// SWW_WAF_CORE_RULE_SET_MODE is not off, and nil while it is.
CoreRuleSet CoreRuleSetInspector
// Now tells the time by which requests are counted for the rate
// limits, bans are made and run out, and GeoJS's answers are kept,
@@ -174,40 +173,6 @@ func newHandler(params Params) *handler {
}
}
// NewCoreRuleSet returns the Core Rule Set at SWW_WAF_CORE_RULE_SET_PARANOIA_LEVEL,
// without the rules SWW_WAF_CORE_RULE_SET_DISABLED_RULES switches off, reading bodies
// up to SWW_WAF_CORE_RULE_SET_BODY_INSPECTION_SIZE, with the exclusions in
// SWW_WAF_CORE_RULE_SET_EXCLUSIONS_FILE and the fixes of the apps in SWW_WAF_APPS,
// or nil while SWW_WAF_CORE_RULE_SET_MODE is off. An exclusions file that
// coreruleset.New refuses is an error in every mode. While the mode is off, a
// set file is loaded with the Core Rule Set only to check it.
func NewCoreRuleSet(cfg *config.WAF) (*coreruleset.CoreRuleSet, error) {
if cfg.CoreRuleSetMode == config.CoreRuleSetModeOff &&
cfg.CoreRuleSetExclusionsFile == "" {
return nil, nil //nolint:nilnil // off is no Core Rule Set, and no error
}
coreRuleSet, err := coreruleset.New(coreruleset.Params{
ParanoiaLevel: cfg.CoreRuleSetParanoiaLevel,
DisabledRules: cfg.CoreRuleSetDisabledRules,
BodyInspectionSize: cfg.CoreRuleSetBodyInspectionSize,
ExclusionsFile: cfg.CoreRuleSetExclusionsFile,
Apps: cfg.Apps,
})
if err != nil {
return nil, fmt.Errorf(
"SWW_WAF_CORE_RULE_SET_EXCLUSIONS_FILE: %w",
err,
)
}
if cfg.CoreRuleSetMode == config.CoreRuleSetModeOff {
return nil, nil //nolint:nilnil // off is no Core Rule Set, and no error
}
return coreRuleSet, nil
}
// handler is waf's handler. It holds what every request shares; what
// belongs to one request is in a request.
type handler struct {
+34 -13
View File
@@ -20,6 +20,7 @@ import (
"sneak.berlin/go/smallwebwaf/internal/bans"
"sneak.berlin/go/smallwebwaf/internal/config"
"sneak.berlin/go/smallwebwaf/internal/config/configtest"
"sneak.berlin/go/smallwebwaf/internal/coreruleset"
"sneak.berlin/go/smallwebwaf/internal/geojs"
"sneak.berlin/go/smallwebwaf/internal/lookup"
"sneak.berlin/go/smallwebwaf/internal/metrics"
@@ -504,24 +505,20 @@ func newWAFWith(
}
}
coreRuleSet, err := waf.NewCoreRuleSet(cfg)
if err != nil {
t.Fatalf("Core Rule Set: %v", err)
}
params := waf.Params{
Config: cfg,
RequestLog: out,
ProcessLog: processLog,
LookupFile: lookupFile,
CoreRuleSet: coreRuleSet,
Now: now,
Rules: ruleFiles,
Alerts: alertQueue,
Config: cfg,
RequestLog: out,
ProcessLog: processLog,
LookupFile: lookupFile,
Now: now,
Rules: ruleFiles,
Alerts: alertQueue,
// The tests of the limit set it back, and hold the Core Rule Set
// with a stand-in.
BesideLookupTimeLimit: time.Hour,
}
addCoreRuleSet(t, &params)
addCounters(&params)
addOutsideSources(&params, geojsURL, abuseIPDBCheckURL)
@@ -530,6 +527,30 @@ func newWAFWith(
return server, out, alertQueue
}
// addCoreRuleSet adds to params the Core Rule Set, with the settings for it,
// while SWW_WAF_CORE_RULE_SET_MODE is not off.
func addCoreRuleSet(t *testing.T, params *waf.Params) {
t.Helper()
cfg := params.Config
if cfg.CoreRuleSetMode == config.CoreRuleSetModeOff {
return
}
coreRuleSet, err := coreruleset.New(coreruleset.Params{
ParanoiaLevel: cfg.CoreRuleSetParanoiaLevel,
DisabledRules: cfg.CoreRuleSetDisabledRules,
BodyInspectionSize: cfg.CoreRuleSetBodyInspectionSize,
ExclusionsFile: cfg.CoreRuleSetExclusionsFile,
Apps: cfg.Apps,
})
if err != nil {
t.Fatalf("Core Rule Set: %v", err)
}
params.CoreRuleSet = coreRuleSet
}
// addCounters adds to params the metrics, the table of clients, the ledger
// and the anomaly counters, as sww builds them from params' settings. The
// metrics read the ledger, the table of clients and the rules.
-4
View File
@@ -7,10 +7,6 @@ import (
"sneak.berlin/go/smallwebwaf/internal/bans"
)
// deny is the SWW_WAF_BLOCKLIST_ACTION and the SWW_WAF_DNSBL_ABUSEIPDB_ACTION that
// refuses the requests of a client a source lists.
const deny = "deny"
// noteListed notes each of sources, the URLs of the blocklists or the
// DNSBL zones, their keys masked, that list the client, as noteHit does,
// with reason.
+1 -1
View File
@@ -421,7 +421,7 @@ func (rq *request) answerAtTheEnd() (geojs.Answer, bool) {
return geojs.Answer{}, false
}
if rq.h.config.LookupSource == "file" {
if rq.h.config.LookupSource == config.LookupSourceFile {
return rq.lookupAnswer, true
}