Compare commits

..
Author SHA1 Message Date
sneak 145b80cc08 waf: end the asks to GeoJS, the DNSBL zones, AbuseIPDB and routewatch at the stop (closes #439)
check / check (push) Failing after 2m4s
Each of the four keeps the cancel function of every ask it started, and a
wait group of their goroutines. Its new method (GeoJS.StopAsking,
DNSBL.StopAsking, AbuseIPDB.StopChecking, routewatch Client.StopAsking)
refuses further asks, ends those under way, and returns once their
goroutines have. An ask ended that way is no failure: no alert, no failure
count, no wait before the next. Each provider in internal/sww adds the
method as a stop hook, so the last write of the state files follows it.

Model: opus-5-5
2026-11-19 16:46:01 +00:00
21 changed files with 481 additions and 148 deletions

No files matched your search

+30 -31
View File
@@ -817,8 +817,8 @@ The supported apps are:
- `vaultwarden`; - `vaultwarden`;
- `wordpress`. - `wordpress`.
Each app's fixes are in one file named for the app, such as Each app's fixes are in one file, such as `internal/coreruleset/gitea.conf`, and
`internal/coreruleset/apps/gitea.conf`. gitea's fixes are: one entry of `apps` in `internal/coreruleset/coreruleset.go`. gitea's fixes are:
1. Ten parameters are not checked against the Core Rule Set's lists of system 1. Ten parameters are not checked against the Core Rule Set's lists of system
files (930120), shell paths (932160) and command names (932260), on any path: files (930120), shell paths (932160) and command names (932260), on any path:
@@ -851,8 +851,8 @@ Each app's fixes are in one file named for the app, such as
3. On gitea's paths, some fields skip the rules that refused ordinary gitea 3. On gitea's paths, some fields skip the rules that refused ordinary gitea
requests there. They are the fields where gitea sends a search, a file's name requests there. They are the fields where gitea sends a search, a file's name
or text, a commit message, a branch name or a repository's name. or text, a commit message, a branch name or a repository's name.
`internal/coreruleset/apps/gitea.conf` lists each rule and the request it `internal/coreruleset/gitea.conf` lists each rule and the request it refused.
refused. The fields are: The fields are:
- `q` on gitea's searches of code, of issues and pull requests, and of - `q` on gitea's searches of code, of issues and pull requests, and of
commits. A code search holds code, such as `<script>`, commits. A code search holds code, such as `<script>`,
`SELECT * FROM users WHERE` or `../`. The searches of issues and pull `SELECT * FROM users WHERE` or `../`. The searches of issues and pull
@@ -1037,7 +1037,7 @@ Each app's fixes are in one file named for the app, such as
- The `Content-Range` header of an artifact upload from - The `Content-Range` header of an artifact upload from
`actions/upload-artifact@v3` (920450). `actions/upload-artifact@v3` (920450).
- The passwords in a body, for each rule that random passwords set off. The - The passwords in a body, for each rule that random passwords set off. The
numbers are in `internal/coreruleset/apps/gitea.conf`. numbers are in `internal/coreruleset/gitea.conf`.
- `password` on these paths: - `password` on these paths:
- `/user/login`, `/user/sign_up`, `/user/activate`; - `/user/login`, `/user/sign_up`, `/user/activate`;
- `/user/link_account_signin` and `/user/link_account_signup`; - `/user/link_account_signin` and `/user/link_account_signup`;
@@ -1069,8 +1069,8 @@ Each app's fixes are in one file named for the app, such as
such as `owner/python3-app` would have that push refused. such as `owner/python3-app` would have that push refused.
WordPress's fixes: on WordPress's paths, some fields skip the rules that refused WordPress's fixes: on WordPress's paths, some fields skip the rules that refused
ordinary WordPress requests there. `internal/coreruleset/apps/wordpress.conf` ordinary WordPress requests there. `internal/coreruleset/wordpress.conf` lists
lists each rule and the request it refused. The REST API has four addresses: each rule and the request it refused. The REST API has four addresses:
- `/wp-json/...` - `/wp-json/...`
- `/index.php/wp-json/...` - `/index.php/wp-json/...`
@@ -1162,9 +1162,9 @@ Elsewhere WordPress's fixes do not apply. The paths are those of WordPress at
the site root. the site root.
Drupal's fixes: on Drupal's paths, some fields skip the rules that refused Drupal's fixes: on Drupal's paths, some fields skip the rules that refused
ordinary Drupal requests there. `internal/coreruleset/apps/drupal.conf` lists ordinary Drupal requests there. `internal/coreruleset/drupal.conf` lists each
each rule and the request it refused. These query fields skip the lists of rule and the request it refused. These query fields skip the lists of system
system files and command names (930120, 932120, 932250, 932260): files and command names (930120, 932120, 932250, 932260):
- On every page under `/admin`: `title`, `name`, `user`, `filename`, `filemime`, - On every page under `/admin`: `title`, `name`, `user`, `filename`, `filemime`,
`info`, `subject`, `author_name`, `search`, `q` and `destination`. `info`, `subject`, `author_name`, `search`, `q` and `destination`.
@@ -1254,10 +1254,10 @@ first path segment of two or three lowercase letters, optionally followed by a
hyphen and more letters, so `/api` counts too. hyphen and more letters, so `/api` counts too.
MediaWiki's fixes: on MediaWiki's paths, some fields skip the rules that refused MediaWiki's fixes: on MediaWiki's paths, some fields skip the rules that refused
ordinary MediaWiki requests there. `internal/coreruleset/apps/mediawiki.conf` ordinary MediaWiki requests there. `internal/coreruleset/mediawiki.conf` lists
lists each rule and the request it refused. These fields skip the lists of each rule and the request it refused. These fields skip the lists of system
system files and command names (930120, 932120, 932250, 932260), in the query or files and command names (930120, 932120, 932250, 932260), in the query or the
the body: body:
- On MediaWiki's pages, which are `/index.php` and the paths under `/index.php/` - On MediaWiki's pages, which are `/index.php` and the paths under `/index.php/`
and `/wiki/`: and `/wiki/`:
@@ -1363,9 +1363,9 @@ Elsewhere these fields keep every rule. The paths are those of MediaWiki at the
site root or under `/w/`. site root or under `/w/`.
Mastodon's fixes: on Mastodon's paths, some fields skip the rules that refused Mastodon's fixes: on Mastodon's paths, some fields skip the rules that refused
ordinary Mastodon requests there. `internal/coreruleset/apps/mastodon.conf` ordinary Mastodon requests there. `internal/coreruleset/mastodon.conf` lists
lists each rule and the request it refused. These fields skip the lists of each rule and the request it refused. These fields skip the lists of system
system files and command names (930120, 932120, 932250, 932260), in the query: files and command names (930120, 932120, 932250, 932260), in the query:
- On the API's search, accounts and admin accounts, under `/api/v1/` and - On the API's search, accounts and admin accounts, under `/api/v1/` and
`/api/v2/`, the searches and names `q`, `acct`, `username`, `display_name`, `/api/v2/`, the searches and names `q`, `acct`, `username`, `display_name`,
@@ -1526,7 +1526,7 @@ every rule.
Vaultwarden's fixes: on Vaultwarden's paths, some fields skip the rules that Vaultwarden's fixes: on Vaultwarden's paths, some fields skip the rules that
refused ordinary Vaultwarden requests there. refused ordinary Vaultwarden requests there.
`internal/coreruleset/apps/vaultwarden.conf` lists each rule and the request it `internal/coreruleset/vaultwarden.conf` lists each rule and the request it
refused. The apps encrypt vault items, folders, collections and sends before refused. The apps encrypt vault items, folders, collections and sends before
sending them, so those fields skip no rules. The fields below hold plain text. sending them, so those fields skip no rules. The fields below hold plain text.
@@ -1577,8 +1577,8 @@ has a parameter of the same name.
`token` in the body of `/admin`, the admin token typed on the admin page, skips `token` in the body of `/admin`, the admin token typed on the admin page, skips
each rule that random passwords set off. The numbers are in each rule that random passwords set off. The numbers are in
`internal/coreruleset/apps/vaultwarden.conf`. It applies only when the query has `internal/coreruleset/vaultwarden.conf`. It applies only when the query has no
no `token` parameter. `' OR '1'='1`, `../../etc/passwd`, `; cat /etc/passwd` and `token` parameter. `' OR '1'='1`, `../../etc/passwd`, `; cat /etc/passwd` and
`<img src=x onerror=x()>` sent as the token reach Vaultwarden; `<img src=x onerror=x()>` sent as the token reach Vaultwarden;
`<script>alert(1)</script>` and `javascript:alert(1)` are still refused. `<script>alert(1)</script>` and `javascript:alert(1)` are still refused.
@@ -1600,7 +1600,7 @@ Elsewhere these fields keep every rule.
Nextcloud's fixes: on Nextcloud's paths, some fields skip the rules that refused Nextcloud's fixes: on Nextcloud's paths, some fields skip the rules that refused
ordinary Nextcloud requests there, and its WebDAV methods are allowed. ordinary Nextcloud requests there, and its WebDAV methods are allowed.
`internal/coreruleset/apps/nextcloud.conf` lists each rule and the request it `internal/coreruleset/nextcloud.conf` lists each rule and the request it
refused. refused.
The sync client and the Files app use WebDAV on these paths: The sync client and the Files app use WebDAV on these paths:
@@ -1610,8 +1610,7 @@ The sync client and the Files app use WebDAV on these paths:
- `/remote.php/webdav/`; - `/remote.php/webdav/`;
- `/public.php/webdav/`. - `/public.php/webdav/`.
These methods are allowed there, besides `PUT`, `PATCH` and `DELETE`, which These methods are allowed there, besides those of the first change:
every app gets:
- `PROPFIND` and `PROPPATCH`; - `PROPFIND` and `PROPPATCH`;
- `MKCOL`; - `MKCOL`;
@@ -1755,7 +1754,7 @@ gets gitea's fixes too. Forgejo's own fixes hold only what Forgejo sends and
gitea does not. gitea does not.
On Forgejo's paths, some fields skip the rules that refused ordinary Forgejo On Forgejo's paths, some fields skip the rules that refused ordinary Forgejo
requests there. `internal/coreruleset/apps/forgejo.conf` lists each rule and the requests there. `internal/coreruleset/forgejo.conf` lists each rule and the
request it refused. request it refused.
`q` skips rules on some of Forgejo's searches. The searches are: `q` skips rules on some of Forgejo's searches. The searches are:
@@ -1801,10 +1800,10 @@ The Core Rule Set reads the request without the `flash` cookie, in which Forgejo
puts what gitea puts in `gitea_flash` (gitea's second fix). puts what gitea puts in `gitea_flash` (gitea's second fix).
Ghost's fixes: on Ghost's paths, some fields skip the rules that refused Ghost's fixes: on Ghost's paths, some fields skip the rules that refused
ordinary Ghost requests there. `internal/coreruleset/apps/ghost.conf` lists each ordinary Ghost requests there. `internal/coreruleset/ghost.conf` lists each rule
rule and the request it refused. Ghost's admin and integrations call the Admin and the request it refused. Ghost's admin and integrations call the Admin API
API under `/ghost/api/admin/`. Themes and Ghost's scripts call the Content API under `/ghost/api/admin/`. Themes and Ghost's scripts call the Content API under
under `/ghost/api/content/`, and members call `/members/api/`. The admin calls `/ghost/api/content/`, and members call `/members/api/`. The admin calls
ActivityPub, a service of its own, under `/.ghost/activitypub/`. ActivityPub, a service of its own, under `/.ghost/activitypub/`.
Eight rules refused words that name commands and files, such as `python3`, Eight rules refused words that name commands and files, such as `python3`,
@@ -1942,9 +1941,9 @@ That name holds `.profile`, a file on the list of system files, so the name
skips 930120 when a profile is saved. skips 930120 when a profile is saved.
The passwords skip each rule that random passwords set off. The numbers are in The passwords skip each rule that random passwords set off. The numbers are in
`internal/coreruleset/apps/ghost.conf`. They are `password` when signing in, and `internal/coreruleset/ghost.conf`. They are `password` when signing in, and the
the passwords sent when creating the first staff user, accepting an invitation, passwords sent when creating the first staff user, accepting an invitation, and
and resetting or changing a password. `' OR '1'='1`, `../../etc/passwd`, resetting or changing a password. `' OR '1'='1`, `../../etc/passwd`,
`; cat /etc/passwd` and `<img src=x onerror=x()>` sent as a password reach `; cat /etc/passwd` and `<img src=x onerror=x()>` sent as a password reach
Ghost; `<script>alert(1)</script>` and `javascript:alert(1)` are still refused. Ghost; `<script>alert(1)</script>` and `javascript:alert(1)` are still refused.
+65 -43
View File
@@ -12,10 +12,11 @@
package coreruleset package coreruleset
import ( import (
"embed" _ "embed"
"errors" "errors"
"fmt" "fmt"
"io" "io"
"maps"
"net" "net"
"net/http" "net/http"
"net/netip" "net/netip"
@@ -34,7 +35,7 @@ import (
// directives are the Core Rule Set as sww runs it, with the // directives are the Core Rule Set as sww runs it, with the
// paranoia level for %d, bodyDirectives for the first %s while // paranoia level for %d, bodyDirectives for the first %s while
// SWW_WAF_CORE_RULE_SET_BODY_INSPECTION_SIZE is set, the exclusions file for // SWW_WAF_CORE_RULE_SET_BODY_INSPECTION_SIZE is set, the exclusions file for
// the second, and the fixes of every app in appFixes for the third. // the second, and the directives of every app in apps for the third.
// Each rule sww adds has an id from 900000 to 900999, the ids the // Each rule sww adds has an id from 900000 to 900999, the ids the
// Core Rule Set keeps for the rules that set it up, which // Core Rule Set keeps for the rules that set it up, which
// SWW_WAF_CORE_RULE_SET_DISABLED_RULES and the exclusions file refuse, so that no // SWW_WAF_CORE_RULE_SET_DISABLED_RULES and the exclusions file refuse, so that no
@@ -53,12 +54,13 @@ Include @crs-setup.conf.example
SecAction "id:900000,phase:1,pass,nolog,\ SecAction "id:900000,phase:1,pass,nolog,\
setvar:tx.blocking_paranoia_level=%d" setvar:tx.blocking_paranoia_level=%d"
# PUT, PATCH and DELETE are allowed besides GET, HEAD, POST and OPTIONS. # The first change: PUT, PATCH and DELETE are allowed besides GET, HEAD,
# POST and OPTIONS.
SecAction "id:900200,phase:1,pass,nolog,\ SecAction "id:900200,phase:1,pass,nolog,\
setvar:'tx.allowed_methods=GET HEAD POST OPTIONS PUT PATCH DELETE'" setvar:'tx.allowed_methods=GET HEAD POST OPTIONS PUT PATCH DELETE'"
# Expect and Content-Encoding are taken off the Core Rule Set's list of the # The second: Expect and Content-Encoding are taken off the Core Rule Set's
# headers it refuses. Content-Encoding goes back on it for a # list of the headers it refuses. Content-Encoding goes back on it for a
# body the Core Rule Set reads (900260 in bodyDirectives). # body the Core Rule Set reads (900260 in bodyDirectives).
SecAction "id:900250,phase:1,pass,nolog,\ SecAction "id:900250,phase:1,pass,nolog,\
setvar:'tx.restricted_headers_basic=/proxy/ /lock-token/ /content-range/ \ setvar:'tx.restricted_headers_basic=/proxy/ /lock-token/ /content-range/ \
@@ -84,17 +86,18 @@ SecRule ARGUMENTS_LIMIT_REACHED "@eq 1" "id:900300,phase:2,pass,\
# The apps' fixes come before the rules for the same reason. # The apps' fixes come before the rules for the same reason.
%s %s
# Only the rules for requests are loaded, and no response is inspected. # The fifth: only the rules for requests are loaded, and no response is
# inspected.
Include @owasp_crs/REQUEST-*.conf Include @owasp_crs/REQUEST-*.conf
# redirect_uri is not checked for a URL naming an IP address or # The third: redirect_uri is not checked for a URL naming an IP address or
# localhost. Coraza matches a parameter name here without regard to case. # localhost. Coraza matches a parameter name here without regard to case.
# ARGS holds the fields of a form data or multipart body Coraza reads as # ARGS holds the fields of a form data or multipart body Coraza reads as
# well as the query parameters, so a field of this name is left out too. # well as the query parameters, so a field of this name is left out too.
SecRuleUpdateTargetById 931100 "!ARGS:redirect_uri" SecRuleUpdateTargetById 931100 "!ARGS:redirect_uri"
SecRuleUpdateTargetById 934110 "!ARGS:redirect_uri" SecRuleUpdateTargetById 934110 "!ARGS:redirect_uri"
# Referer is not checked for a Unix command without arguments # The fourth: Referer is not checked for a Unix command without arguments
# or for Java starting a process. # or for Java starting a process.
SecRuleUpdateTargetById 932340 "!REQUEST_HEADERS:Referer" SecRuleUpdateTargetById 932340 "!REQUEST_HEADERS:Referer"
SecRuleUpdateTargetById 944110 "!REQUEST_HEADERS:Referer" SecRuleUpdateTargetById 944110 "!REQUEST_HEADERS:Referer"
@@ -119,9 +122,9 @@ SecRule REQUEST_HEADERS:Content-Type \
"@rx ^(?:application|text)/(?:[a-z0-9.-]+[+])?json" \ "@rx ^(?:application|text)/(?:[a-z0-9.-]+[+])?json" \
"id:900420,phase:1,pass,nolog,t:none,t:lowercase,ctl:requestBodyProcessor=JSON" "id:900420,phase:1,pass,nolog,t:none,t:lowercase,ctl:requestBodyProcessor=JSON"
# Content-Encoding goes back on the list of the headers the Core Rule Set # The rest of the second change: Content-Encoding is refused again on a
# refuses (900250 in directives) for a body of a kind it reads, since a # body of a kind the Core Rule Set reads, since a compressed body cannot be
# compressed body cannot be inspected. # inspected.
SecRule REQBODY_PROCESSOR "@rx ^(?:URLENCODED|MULTIPART|JSON|XML)$" \ SecRule REQBODY_PROCESSOR "@rx ^(?:URLENCODED|MULTIPART|JSON|XML)$" \
"id:900260,phase:1,pass,nolog,\ "id:900260,phase:1,pass,nolog,\
setvar:'tx.restricted_headers_basic=%%{tx.restricted_headers_basic} \ setvar:'tx.restricted_headers_basic=%%{tx.restricted_headers_basic} \
@@ -142,19 +145,15 @@ SecRule MULTIPART_STRICT_ERROR "!@eq 0" "id:900450,phase:2,pass,\
severity:'CRITICAL',setvar:'tx.inbound_anomaly_score_pl1=+5'" severity:'CRITICAL',setvar:'tx.inbound_anomaly_score_pl1=+5'"
` `
// appFixes holds the fixes of each app whose ordinary requests the Core // app is an app whose ordinary requests the Core Rule Set would refuse,
// Rule Set would refuse, in a file named for the app by the name // with the fixes that let them pass.
// SWW_WAF_APPS gives it, such as apps/gitea.conf. The fixes are Coraza
// exclusions, loaded before the Core Rule Set's rules. Their first rule
// skips to the marker they end with, unless Inspect sets the transaction
// variable named for the app, such as app_gitea. The skip reaches only
// phase 1, so a rule of a later phase must check that variable itself.
//
//go:embed apps/*.conf
var appFixes embed.FS
// app is what an app's file in appFixes cannot say about it.
type app struct { type app struct {
// directives are Coraza exclusions, loaded before the Core Rule Set's
// rules. Their first rule skips to the marker they end with, unless
// Inspect sets the transaction variable named for the app, such as
// app_gitea. The skip reaches only phase 1, so a rule of a later phase
// must check that variable itself.
directives string
// cookies are the cookies the Core Rule Set reads a request without. // cookies are the cookies the Core Rule Set reads a request without.
cookies []string cookies []string
// forkOf names the app this one is a fork of, "" for none. A hostname // forkOf names the app this one is a fork of, "" for none. A hostname
@@ -162,34 +161,60 @@ type app struct {
forkOf string forkOf string
} }
// apps are the apps that need more than their file in appFixes, by the //go:embed gitea.conf
// name SWW_WAF_APPS gives each. flash is Forgejo's name for gitea_flash. var giteaDirectives string
//go:embed wordpress.conf
var wordpressDirectives string
//go:embed drupal.conf
var drupalDirectives string
//go:embed mediawiki.conf
var mediawikiDirectives string
//go:embed mastodon.conf
var mastodonDirectives string
//go:embed vaultwarden.conf
var vaultwardenDirectives string
//go:embed nextcloud.conf
var nextcloudDirectives string
//go:embed forgejo.conf
var forgejoDirectives string
//go:embed ghost.conf
var ghostDirectives string
// apps are the apps whose fixes sww has, by the name SWW_WAF_APPS gives each.
// flash is Forgejo's name for gitea_flash.
// //
//nolint:gochecknoglobals // a constant cannot be a map //nolint:gochecknoglobals // a constant cannot be a map
var apps = map[string]app{ var apps = map[string]app{
"drupal": {directives: drupalDirectives},
"forgejo": { "forgejo": {
cookies: []string{"flash"}, directives: forgejoDirectives,
forkOf: "gitea", cookies: []string{"flash"},
forkOf: "gitea",
}, },
"ghost": {directives: ghostDirectives},
"gitea": { "gitea": {
cookies: []string{"gitea_flash", "redirect_to"}, directives: giteaDirectives,
cookies: []string{"gitea_flash", "redirect_to"},
}, },
"mastodon": {directives: mastodonDirectives},
"mediawiki": {directives: mediawikiDirectives},
"nextcloud": {directives: nextcloudDirectives},
"vaultwarden": {directives: vaultwardenDirectives},
"wordpress": {directives: wordpressDirectives},
} }
// AppNames returns the names of the apps whose fixes sww has, in // AppNames returns the names of the apps whose fixes sww has, in
// alphabetical order. // alphabetical order.
func AppNames() []string { func AppNames() []string {
// An embedded directory always reads. return slices.Sorted(maps.Keys(apps))
files, _ := appFixes.ReadDir("apps")
names := make([]string, len(files))
for i, file := range files {
names[i] = strings.TrimSuffix(file.Name(), ".conf")
}
slices.Sort(names)
return names
} }
// Params are what New needs. // Params are what New needs.
@@ -348,11 +373,8 @@ func New(params Params) (*CoreRuleSet, error) {
} }
var fixes strings.Builder var fixes strings.Builder
for _, name := range AppNames() { for _, name := range AppNames() {
// An embedded file always reads. fixes.WriteString(apps[name].directives + "\n")
file, _ := appFixes.ReadFile("apps/" + name + ".conf")
fixes.WriteString(string(file) + "\n")
} }
text := fmt.Sprintf(directives, params.ParanoiaLevel, body, exclusions, text := fmt.Sprintf(directives, params.ParanoiaLevel, body, exclusions,
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
+3 -3
View File
@@ -17,7 +17,7 @@ const (
) )
// giteaSearches are searches of the pages gitea and Forgejo share whose q // giteaSearches are searches of the pages gitea and Forgejo share whose q
// skips the lists of system files and command names, in gitea's fixes. // skips the lists of system files and command names, in the eighth change.
// gitea serves its admin pages under /-/admin/ and Forgejo under /admin/. // gitea serves its admin pages under /-/admin/ and Forgejo under /admin/.
// //
//nolint:gochecknoglobals // a constant cannot be a list //nolint:gochecknoglobals // a constant cannot be a list
@@ -222,8 +222,8 @@ func TestGiteaSearchExclusionsAreNarrow(t *testing.T) {
} }
// Under /admin/, where Forgejo serves its admin pages, only the hard // Under /admin/, where Forgejo serves its admin pages, only the hard
// values are left out: Drupal's fixes let them through in q on every path // values are left out: the tenth change lets them through in q on every
// there. // path there, for Drupal.
for _, target := range []string{ for _, target := range []string{
"/admin/packages/delete?q=", "/admin/packages/delete?q=",
"/admin/packages?other=", "/admin/packages?other=",
File renamed without changes.
File renamed without changes.
@@ -37,10 +37,9 @@
SecRule &TX:app_nextcloud "@eq 0" \ SecRule &TX:app_nextcloud "@eq 0" \
"id:900090,phase:1,pass,nolog,skipAfter:END_NEXTCLOUD" "id:900090,phase:1,pass,nolog,skipAfter:END_NEXTCLOUD"
# The sync client and the Files app use these WebDAV methods besides PUT, # The sync client and the Files app use these WebDAV methods besides those
# PATCH and DELETE, which sww allows for every app, on /remote.php/dav/ and # the first change allows, on /remote.php/dav/ and /public.php/dav/ and on the
# /public.php/dav/ and on the older /remote.php/webdav/ and /public.php/webdav/. # older /remote.php/webdav/ and /public.php/webdav/. 911100 refused each:
# 911100 refused each:
# PROPFIND, PROPPATCH: listing a folder, marking a file a favourite # PROPFIND, PROPPATCH: listing a folder, marking a file a favourite
# MKCOL: making a folder, and starting a chunked upload # MKCOL: making a folder, and starting a chunked upload
# COPY, MOVE: copying, moving and renaming, and ending a chunked upload # COPY, MOVE: copying, moving and renaming, and ending a chunked upload
File renamed without changes.
+47 -14
View File
@@ -99,8 +99,13 @@ type GeoJS struct {
// waiting are the clients without an answer: those to ask GeoJS about, // waiting are the clients without an answer: those to ask GeoJS about,
// and those it is being asked about. // and those it is being asked about.
waiting map[netip.Prefix]*wait waiting map[netip.Prefix]*wait
// asking is true while a request to GeoJS is under way. // asking is true while a request to GeoJS is under way, and endAsking
asking bool // ends it. stopped is true once StopAsking has been called, and
// askGoroutines waits for the goroutine that makes the requests.
asking bool
endAsking context.CancelFunc
stopped bool
askGoroutines sync.WaitGroup
// retryDelay is how long GeoJS is left alone after its last failure, // retryDelay is how long GeoJS is left alone after its last failure,
// zero after an answer; retryAt is when it may be asked again. // zero after an answer; retryAt is when it may be asked again.
retryDelay time.Duration retryDelay time.Duration
@@ -210,6 +215,23 @@ func (g *GeoJS) LookUpWithoutWaiting(
return answer return answer
} }
// StopAsking ends the request to GeoJS under way, and returns once the
// goroutine that made it has. A request it ends is no failure: Failed is not
// called, and GeoJS is not left alone after it. After StopAsking, GeoJS is
// asked nothing.
func (g *GeoJS) StopAsking() {
g.mu.Lock()
g.stopped = true
if g.asking {
g.endAsking()
}
g.mu.Unlock()
g.askGoroutines.Wait()
}
// Kept returns client's answer, if one is kept, without asking GeoJS. // Kept returns client's answer, if one is kept, without asking GeoJS.
func (g *GeoJS) Kept(client netip.Prefix) (Answer, bool) { func (g *GeoJS) Kept(client netip.Prefix) (Answer, bool) {
g.mu.Lock() g.mu.Lock()
@@ -294,7 +316,8 @@ func (g *GeoJS) answerOrWait(
} }
if !g.asking { if !g.asking {
// GeoJS is left alone after a failure, so no answer can come. // GeoJS is left alone after a failure, or asked nothing after
// StopAsking, so no answer can come.
w.late = true w.late = true
} }
@@ -323,22 +346,28 @@ func (g *GeoJS) kept(client netip.Prefix) (Answer, bool) {
} }
// ask starts asking GeoJS about the waiting clients, unless a request to // ask starts asking GeoJS about the waiting clients, unless a request to
// it is under way or it is left alone after a failure. The requests to // it is under way, it is left alone after a failure, or StopAsking has
// GeoJS are for every client waiting, so they go on when the caller whose // been called. The requests to GeoJS are for every client waiting, so they
// ctx is given stops waiting. // go on when the caller whose ctx is given stops waiting, until StopAsking.
func (g *GeoJS) ask(ctx context.Context) { func (g *GeoJS) ask(ctx context.Context) {
if g.asking || g.now().Before(g.retryAt) { if g.asking || g.stopped || g.now().Before(g.retryAt) {
return return
} }
g.asking = true askCtx, end := context.WithCancel(context.WithoutCancel(ctx))
g.asking, g.endAsking = true, end
go g.askAboutWaiting(context.WithoutCancel(ctx)) g.askGoroutines.Go(func() {
defer end()
g.askAboutWaiting(askCtx)
})
} }
// askAboutWaiting asks GeoJS about the waiting clients, one request at a // askAboutWaiting asks GeoJS about the waiting clients, one request at a
// time, until none is left or GeoJS fails. Each answer kept is given to // time, until none is left, GeoJS fails, or StopAsking ends ctx. Each
// Answered, outside the lock, since Answered takes locks of its own. // answer kept is given to Answered, outside the lock, since Answered takes
// locks of its own.
func (g *GeoJS) askAboutWaiting(ctx context.Context) { func (g *GeoJS) askAboutWaiting(ctx context.Context) {
for { for {
clients := g.nextClients() clients := g.nextClients()
@@ -347,6 +376,10 @@ func (g *GeoJS) askAboutWaiting(ctx context.Context) {
} }
given, err := g.request(ctx, clients) given, err := g.request(ctx, clients)
if err != nil && ctx.Err() != nil {
continue // ended by StopAsking, which is no failure
}
kept, answered := g.keep(clients, given, err) kept, answered := g.keep(clients, given, err)
if g.answered != nil { if g.answered != nil {
@@ -362,13 +395,13 @@ func (g *GeoJS) askAboutWaiting(ctx context.Context) {
} }
// nextClients returns up to maxPerRequest of the waiting clients. When // nextClients returns up to maxPerRequest of the waiting clients. When
// none is waiting, it returns none and notes that no request to GeoJS is // none is waiting, or StopAsking has been called, it returns none and
// under way. // notes that no request to GeoJS is under way.
func (g *GeoJS) nextClients() []netip.Prefix { func (g *GeoJS) nextClients() []netip.Prefix {
g.mu.Lock() g.mu.Lock()
defer g.mu.Unlock() defer g.mu.Unlock()
if len(g.waiting) == 0 { if len(g.waiting) == 0 || g.stopped {
g.asking = false g.asking = false
return nil return nil
+38
View File
@@ -318,6 +318,44 @@ func TestCallerWhoseContextEndsEarlyDoesNotStopTheRequest(t *testing.T) {
}) })
} }
func TestStopAskingEndsTheRequestUnderWayAsNoFailureAndAsksNoMore(
t *testing.T,
) {
t.Parallel()
synctest.Test(t, func(t *testing.T) {
service, _, g, reports := startWithReports()
clients := newClients()
service.set(hanging)
g.LookUpWithoutWaiting(t.Context(), clients())
waitForRequests(t, service, 1)
// The request is ended, not waited out, and is no failure.
began := time.Now()
g.StopAsking()
if waited := time.Since(began); waited != 0 {
t.Errorf("StopAsking returned after %s, want at once", waited)
}
reports.wantFailures(t)
// A new client is then not asked about, and its lookup does not
// wait.
service.set(answering)
wantCountry(t, g, clients(), "")
if waited := time.Since(began); waited != 0 {
t.Errorf("the lookup returned after %s, want at once", waited)
}
waitForRequests(t, service, 1)
reports.wantFailures(t)
})
}
func TestRequestOver10SecondsIsAFailure(t *testing.T) { func TestRequestOver10SecondsIsAFailure(t *testing.T) {
t.Parallel() t.Parallel()
+48 -16
View File
@@ -110,8 +110,12 @@ type AbuseIPDB struct {
// scores are by client. Each is added as it is fetched and never moved // scores are by client. Each is added as it is fetched and never moved
// up, so that the one fetched longest ago is the first dropped. // up, so that the one fetched longest ago is the first dropped.
scores *simplelru.LRU[netip.Prefix, Score] scores *simplelru.LRU[netip.Prefix, Score]
// checking are the clients whose check is under way. // checking are the clients whose check is under way, each with the
checking map[netip.Prefix]bool // function that ends it. stopped is true once StopChecking has been
// called, and checkGoroutines waits for the goroutine of each check.
checking map[netip.Prefix]context.CancelFunc
stopped bool
checkGoroutines sync.WaitGroup
// day is the day, in UTC, of the checks spent counts. // day is the day, in UTC, of the checks spent counts.
day civil.Date day civil.Date
spent int spent int
@@ -134,7 +138,7 @@ func NewAbuseIPDB(params AbuseIPDBParams) *AbuseIPDB {
params: params, params: params,
httpClient: &http.Client{}, httpClient: &http.Client{},
scores: scores, scores: scores,
checking: map[netip.Prefix]bool{}, checking: map[netip.Prefix]context.CancelFunc{},
} }
} }
@@ -144,11 +148,11 @@ func NewAbuseIPDB(params AbuseIPDBParams) *AbuseIPDB {
// addresses its request comes from. A client without one is checked in // addresses its request comes from. A client without one is checked in
// the background, by addr, the address its request came from, if // the background, by addr, the address its request came from, if
// offender, if it has committed an offence, unless its check is under // offender, if it has committed an offence, unless its check is under
// way, a check failed less than failureDelay ago, or the day's checks // way, a check failed less than failureDelay ago, the day's checks have
// have used up DailyBudget; Hit never waits for a check. The check that // used up DailyBudget, or StopChecking has been called; Hit never waits
// uses the budget up is logged and raised as a source_failure alert. ctx // for a check. The check that uses the budget up is logged and raised as a
// is the context of the client's request, and a check goes on after the // source_failure alert. ctx is the context of the client's request, and a
// request ends. // check goes on after the request ends, until StopChecking.
func (a *AbuseIPDB) Hit( func (a *AbuseIPDB) Hit(
ctx context.Context, client netip.Prefix, addr netip.Addr, offender bool, ctx context.Context, client netip.Prefix, addr netip.Addr, offender bool,
) (int64, bool) { ) (int64, bool) {
@@ -167,14 +171,21 @@ func (a *AbuseIPDB) Hit(
a.day, a.spent = today, 0 a.day, a.spent = today, 0
} }
check := offender && !a.checking[client] && !now.Before(a.retryAt) && _, checking := a.checking[client]
a.spent < a.params.DailyBudget
check := offender && !checking && !a.stopped &&
!now.Before(a.retryAt) && a.spent < a.params.DailyBudget
if check { if check {
a.checking[client] = true checkCtx, end := context.WithCancel(context.WithoutCancel(ctx))
a.checking[client] = end
a.checks++ a.checks++
a.spent++ a.spent++
go a.check(context.WithoutCancel(ctx), client, addr) a.checkGoroutines.Go(func() {
defer end()
a.check(checkCtx, client, addr)
})
} }
usedUp := check && a.spent == a.params.DailyBudget usedUp := check && a.spent == a.params.DailyBudget
@@ -189,6 +200,24 @@ func (a *AbuseIPDB) Hit(
return 0, false return 0, false
} }
// StopChecking ends the checks under way, and returns once their goroutines
// have. A check it ends gives no score and is no failure: it is neither
// counted, logged nor raised as an alert, and checks are not left alone
// after it. The ended check stays spent from the day's budget. After
// StopChecking, Hit checks no client.
func (a *AbuseIPDB) StopChecking() {
a.mu.Lock()
a.stopped = true
for _, end := range a.checking {
end()
}
a.mu.Unlock()
a.checkGoroutines.Wait()
}
// Checked returns how many checks were made. // Checked returns how many checks were made.
func (a *AbuseIPDB) Checked() int { func (a *AbuseIPDB) Checked() int {
a.mu.Lock() a.mu.Lock()
@@ -315,7 +344,8 @@ func (a *AbuseIPDB) RunBlacklist(ctx context.Context) {
// check checks client with AbuseIPDB by addr, one of its addresses, keeps // check checks client with AbuseIPDB by addr, one of its addresses, keeps
// the score as client's, and notes the check as no longer under way. A // the score as client's, and notes the check as no longer under way. A
// check that fails gives no score: it is counted, logged and raised as a // check that fails gives no score: it is counted, logged and raised as a
// source_failure alert, and no client is checked for failureDelay. // source_failure alert, and no client is checked for failureDelay. A check
// ended as ctx ends, by StopChecking, is no failure.
func (a *AbuseIPDB) check( func (a *AbuseIPDB) check(
ctx context.Context, ctx context.Context,
client netip.Prefix, client netip.Prefix,
@@ -323,24 +353,26 @@ func (a *AbuseIPDB) check(
) { ) {
score, err := a.ask(ctx, addr) score, err := a.ask(ctx, addr)
now := a.params.Now() now := a.params.Now()
isFailure := err != nil && ctx.Err() == nil
a.mu.Lock() a.mu.Lock()
delete(a.checking, client) delete(a.checking, client)
if err == nil { switch {
case err == nil:
a.scores.Add( a.scores.Add(
client, client,
Score{Client: client, Score: score, Fetched: now}, Score{Client: client, Score: score, Fetched: now},
) )
} else { case isFailure:
a.failures++ a.failures++
a.retryAt = now.Add(failureDelay) a.retryAt = now.Add(failureDelay)
} }
a.mu.Unlock() a.mu.Unlock()
if err != nil { if isFailure {
a.alert("checking a client with AbuseIPDB failed", err) a.alert("checking a client with AbuseIPDB failed", err)
} }
} }
+37
View File
@@ -394,6 +394,43 @@ func TestCheckNotAnsweredWithinTheTimeoutFailsAndHitNeverWaits(
}) })
} }
func TestStopCheckingEndsTheCheckUnderWayAsNoFailureAndChecksNoMore(
t *testing.T,
) {
t.Parallel()
synctest.Test(t, func(t *testing.T) {
queue := newQueue()
p := abuseIPDBParams()
p.Alerts = queue
abuseIPDB := &abuseIPDBStandIn{hanging: true}
checker := newAbuseIPDB(abuseIPDB, p)
wantScore(t, checker, suspect, true, 0, false)
synctest.Wait()
// The check is ended, not waited out, and is no failure.
began := time.Now()
checker.StopChecking()
if waited := time.Since(began); waited != 0 {
t.Errorf("StopChecking returned after %s, want at once", waited)
}
wantFailures(t, checker, 0)
if got := waiting(queue); len(got) != 0 {
t.Errorf("alerts waiting %+v, want none", got)
}
// An offender without a score is then not checked.
wantScore(t, checker, other, true, 0, false)
synctest.Wait()
wantChecked(t, abuseIPDB, suspect)
})
}
func TestKeyIsSentInTheKeyHeaderAndNeverShown(t *testing.T) { func TestKeyIsSentInTheKeyHeaderAndNeverShown(t *testing.T) {
t.Parallel() t.Parallel()
+44 -13
View File
@@ -82,8 +82,12 @@ type DNSBL struct {
// verdicts are by query. Each is added as it is fetched and never moved // verdicts are by query. Each is added as it is fetched and never moved
// up, so that the one fetched longest ago is the first dropped. // up, so that the one fetched longest ago is the first dropped.
verdicts *simplelru.LRU[query, Verdict] verdicts *simplelru.LRU[query, Verdict]
// asking are the queries under way. // asking are the queries under way, each with the function that ends
asking map[query]bool // it. stopped is true once StopAsking has been called, and
// queryGoroutines waits for the goroutine of each query.
asking map[query]context.CancelFunc
stopped bool
queryGoroutines sync.WaitGroup
// queries and failures count, by zone, the queries made and those that // queries and failures count, by zone, the queries made and those that
// failed, and retryAt is when a zone whose last query failed may be // failed, and retryAt is when a zone whose last query failed may be
// asked again. // asked again.
@@ -120,7 +124,7 @@ func NewDNSBL(params DNSBLParams) *DNSBL {
params: params, params: params,
resolver: resolver, resolver: resolver,
verdicts: verdicts, verdicts: verdicts,
asking: map[query]bool{}, asking: map[query]context.CancelFunc{},
queries: map[string]int{}, queries: map[string]int{},
failures: map[string]int{}, failures: map[string]int{},
retryAt: map[string]time.Time{}, retryAt: map[string]time.Time{},
@@ -138,9 +142,10 @@ func (d *DNSBL) Zones() []string {
// log, the alerts and the metrics. A verdict is used until CacheTTL has // log, the alerts and the metrics. A verdict is used until CacheTTL has
// passed since it was fetched. Each zone without one is asked about addr // passed since it was fetched. Each zone without one is asked about addr
// in the background, unless a query about addr to it is under way, the // in the background, unless a query about addr to it is under way, the
// zone is left alone after a failure, or maxQueries are under way; // zone is left alone after a failure, maxQueries are under way, or
// ListedBy never waits for a query. ctx is the context of the client's // StopAsking has been called; ListedBy never waits for a query. ctx is the
// request, and a query goes on after the request ends. // context of the client's request, and a query goes on after the request
// ends, until StopAsking.
func (d *DNSBL) ListedBy(ctx context.Context, addr netip.Addr) []string { func (d *DNSBL) ListedBy(ctx context.Context, addr netip.Addr) []string {
d.mu.Lock() d.mu.Lock()
defer d.mu.Unlock() defer d.mu.Unlock()
@@ -153,23 +158,47 @@ func (d *DNSBL) ListedBy(ctx context.Context, addr netip.Addr) []string {
q := query{zone: zone, client: addr} q := query{zone: zone, client: addr}
kept, found := d.verdicts.Peek(q) kept, found := d.verdicts.Peek(q)
_, asking := d.asking[q]
switch { switch {
case found && now.Sub(kept.Fetched) < d.params.CacheTTL: case found && now.Sub(kept.Fetched) < d.params.CacheTTL:
if kept.Listed { if kept.Listed {
listedBy = append(listedBy, config.MaskZoneKey(zone)) listedBy = append(listedBy, config.MaskZoneKey(zone))
} }
case !d.asking[q] && !now.Before(d.retryAt[zone]) && len(d.asking) < maxQueries: case !asking && !d.stopped && !now.Before(d.retryAt[zone]) &&
d.asking[q] = true len(d.asking) < maxQueries:
queryCtx, end := context.WithCancel(context.WithoutCancel(ctx))
d.asking[q] = end
d.queries[zone]++ d.queries[zone]++
go d.ask(context.WithoutCancel(ctx), q) d.queryGoroutines.Go(func() {
defer end()
d.ask(queryCtx, q)
})
} }
} }
return listedBy return listedBy
} }
// StopAsking ends the queries under way, and returns once their
// goroutines have. A query it ends gives no verdict and is no failure: it
// is neither counted, logged nor raised as an alert, and the zone is not
// left alone after it. After StopAsking, ListedBy asks no zone.
func (d *DNSBL) StopAsking() {
d.mu.Lock()
d.stopped = true
for _, end := range d.asking {
end()
}
d.mu.Unlock()
d.queryGoroutines.Wait()
}
// Queries returns how many queries were made to zone. // Queries returns how many queries were made to zone.
func (d *DNSBL) Queries(zone string) int { func (d *DNSBL) Queries(zone string) int {
d.mu.Lock() d.mu.Lock()
@@ -238,27 +267,29 @@ func (d *DNSBL) Load(verdicts []Verdict) {
// query as no longer under way. A query that fails gives no verdict: it // query as no longer under way. A query that fails gives no verdict: it
// is counted, logged and raised as a source_failure alert, which show the // is counted, logged and raised as a source_failure alert, which show the
// zone with its key masked, and the zone is not asked again for // zone with its key masked, and the zone is not asked again for
// failureDelay. // failureDelay. A query ended as ctx ends, by StopAsking, is no failure.
func (d *DNSBL) ask(ctx context.Context, q query) { func (d *DNSBL) ask(ctx context.Context, q query) {
listed, err := d.lookUp(ctx, q) listed, err := d.lookUp(ctx, q)
now := d.params.Now() now := d.params.Now()
isFailure := err != nil && ctx.Err() == nil
d.mu.Lock() d.mu.Lock()
delete(d.asking, q) delete(d.asking, q)
if err == nil { switch {
case err == nil:
d.verdicts.Add(q, Verdict{ d.verdicts.Add(q, Verdict{
Zone: q.zone, Client: q.client, Listed: listed, Fetched: now, Zone: q.zone, Client: q.client, Listed: listed, Fetched: now,
}) })
} else { case isFailure:
d.failures[q.zone]++ d.failures[q.zone]++
d.retryAt[q.zone] = now.Add(failureDelay) d.retryAt[q.zone] = now.Add(failureDelay)
} }
d.mu.Unlock() d.mu.Unlock()
if err != nil { if isFailure {
const failed = "asking a DNSBL zone failed" const failed = "asking a DNSBL zone failed"
shown := config.MaskZoneKey(q.zone) shown := config.MaskZoneKey(q.zone)
+39
View File
@@ -210,6 +210,45 @@ func TestQueryNotAnsweredWithinTheTimeoutFails(t *testing.T) {
}) })
} }
//
//nolint:paralleltest // one at a time, as the comment at the top of this file says
func TestStopAskingEndsTheQueryUnderWayAsNoFailureAndAsksNoMore(
t *testing.T,
) {
synctest.Test(t, func(t *testing.T) {
queue := newQueue()
p := dnsblParams(zone)
p.Alerts = queue
resolver := &resolverStandIn{hanging: true}
dnsbl := newDNSBL(resolver, p)
wantZones(t, dnsbl, listed)
synctest.Wait()
// The query is ended, not waited out, and is no failure.
began := time.Now()
dnsbl.StopAsking()
if waited := time.Since(began); waited != 0 {
t.Errorf("StopAsking returned after %s, want at once", waited)
}
wantQueries(t, dnsbl, 1, 0)
if got := waiting(queue); len(got) != 0 {
t.Errorf("alerts waiting %+v, want none", got)
}
// A client without a verdict is then asked about in no zone.
wantZones(t, dnsbl, unlisted)
synctest.Wait()
wantQueries(t, dnsbl, 1, 0)
wantAsked(t, resolver, listedName)
waitForTheResolver()
})
}
// //
//nolint:paralleltest // one at a time, as the comment at the top of this file says //nolint:paralleltest // one at a time, as the comment at the top of this file says
func TestZoneThatFailsOrRefusesGivesNoVerdictAndIsLeftAloneForAMinute( func TestZoneThatFailsOrRefusesGivesNoVerdictAndIsLeftAloneForAMinute(
+43 -12
View File
@@ -99,8 +99,12 @@ type Client struct {
// answers are by client. Each is added as it comes and never moved up, // answers are by client. Each is added as it comes and never moved up,
// so that the one that came longest ago is the first dropped. // so that the one that came longest ago is the first dropped.
answers *simplelru.LRU[netip.Prefix, Answer] answers *simplelru.LRU[netip.Prefix, Answer]
// asking are the clients whose query is under way. // asking are the clients whose query is under way, each with the
asking map[netip.Prefix]bool // function that ends it. stopped is true once StopAsking has been
// called, and queryGoroutines waits for the goroutine of each query.
asking map[netip.Prefix]context.CancelFunc
stopped bool
queryGoroutines sync.WaitGroup
// retryAt is when routewatch may be asked again after a query failed. // retryAt is when routewatch may be asked again after a query failed.
retryAt time.Time retryAt time.Time
} }
@@ -116,7 +120,7 @@ func New(params Params) *Client {
params: params, params: params,
httpClient: &http.Client{}, httpClient: &http.Client{},
answers: answers, answers: answers,
asking: map[netip.Prefix]bool{}, asking: map[netip.Prefix]context.CancelFunc{},
} }
} }
@@ -124,8 +128,9 @@ func New(params Params) *Client {
// IPv6 group, or why it has no route. It never waits for a query. A client // IPv6 group, or why it has no route. It never waits for a query. A client
// without an answer is asked about in the background, by addr, the address its // without an answer is asked about in the background, by addr, the address its
// request came from, unless its query is under way, a query failed less than // request came from, unless its query is under way, a query failed less than
// failureKept ago, or maxQueries are under way. ctx is the context of the // failureKept ago, maxQueries are under way, or StopAsking has been called. ctx
// client's request, and a query goes on after the request ends. // is the context of the client's request, and a query goes on after the request
// ends, until StopAsking.
func (c *Client) Route( func (c *Client) Route(
ctx context.Context, client netip.Prefix, addr netip.Addr, ctx context.Context, client netip.Prefix, addr netip.Addr,
) (Route, Missing) { ) (Route, Missing) {
@@ -147,15 +152,38 @@ func (c *Client) Route(
return Route{}, RequestFailed return Route{}, RequestFailed
} }
if !c.asking[client] && len(c.asking) < maxQueries { _, asking := c.asking[client]
c.asking[client] = true if !asking && !c.stopped && len(c.asking) < maxQueries {
queryCtx, end := context.WithCancel(context.WithoutCancel(ctx))
c.asking[client] = end
go c.ask(context.WithoutCancel(ctx), client, addr) c.queryGoroutines.Go(func() {
defer end()
c.ask(queryCtx, client, addr)
})
} }
return Route{}, NoAnswerYet return Route{}, NoAnswerYet
} }
// StopAsking ends the queries under way, and returns once their goroutines
// have. A query it ends gives no answer and is no failure: it is not logged,
// and routewatch is not left alone after it. After StopAsking, Route asks
// routewatch nothing.
func (c *Client) StopAsking() {
c.mu.Lock()
c.stopped = true
for _, end := range c.asking {
end()
}
c.mu.Unlock()
c.queryGoroutines.Wait()
}
// Snapshot returns every answer still in use, sorted by client, as // Snapshot returns every answer still in use, sorted by client, as
// routewatch.json lists them. // routewatch.json lists them.
func (c *Client) Snapshot() []Answer { func (c *Client) Snapshot() []Answer {
@@ -205,7 +233,8 @@ func (c *Client) Load(answers []Answer) {
// ask asks routewatch about addr, keeps the answer as client's, and notes the // ask asks routewatch about addr, keeps the answer as client's, and notes the
// query as no longer under way. A query that fails gives no answer, and // query as no longer under way. A query that fails gives no answer, and
// routewatch is not asked again for failureKept. Only the first failure of // routewatch is not asked again for failureKept. Only the first failure of
// those under way is logged. // those under way is logged. A query ended as ctx ends, by StopAsking, is no
// failure.
func (c *Client) ask( func (c *Client) ask(
ctx context.Context, ctx context.Context,
client netip.Prefix, client netip.Prefix,
@@ -213,18 +242,20 @@ func (c *Client) ask(
) { ) {
route, err := c.query(ctx, addr) route, err := c.query(ctx, addr)
now := c.params.Now() now := c.params.Now()
isFailure := err != nil && ctx.Err() == nil
c.mu.Lock() c.mu.Lock()
delete(c.asking, client) delete(c.asking, client)
firstFailure := err != nil && !now.Before(c.retryAt) firstFailure := isFailure && !now.Before(c.retryAt)
if err == nil { switch {
case err == nil:
c.answers.Add(client, Answer{ c.answers.Add(client, Answer{
Client: client, Netblock: route.Netblock, ASN: route.ASN, Answered: now, Client: client, Netblock: route.Netblock, ASN: route.ASN, Answered: now,
}) })
} else { case isFailure:
c.retryAt = now.Add(failureKept) c.retryAt = now.Add(failureKept)
} }
+48
View File
@@ -361,6 +361,54 @@ func TestHangingRoutewatchNeverDelaysAndFailsTheQuery(t *testing.T) {
}) })
} }
func TestStopAskingEndsTheQueryUnderWayAsNoFailureAndAsksNoMore(
t *testing.T,
) {
t.Parallel()
// In a synctest bubble, as TestHangingRoutewatchNeverDelaysAndFailsTheQuery
// says.
synctest.Test(t, func(t *testing.T) {
f := &fixture{t: t, log: &syncBuffer{}}
f.client = routewatch.New(routewatch.Params{
URL: silentURL, Now: time.Now,
ProcessLog: slog.New(slog.NewTextHandler(f.log, nil)),
})
f.route()
synctest.Wait()
// The query is ended, not waited out.
began := time.Now()
f.client.StopAsking()
if waited := time.Since(began); waited != 0 {
t.Errorf("StopAsking returned after %s, want at once", waited)
}
// A query started now would fail within waitLimit. None is, and the
// one ended is no failure: nothing is logged, and routewatch is not
// left alone.
for range 2 {
if _, missing := f.route(); missing != routewatch.NoAnswerYet {
t.Errorf(
"Route gave %q, want %q",
missing,
routewatch.NoAnswerYet,
)
}
time.Sleep(waitLimit)
synctest.Wait()
}
if f.log.String() != "" {
t.Errorf("process log %q, want nothing", f.log.String())
}
})
}
func TestRepeatedRequestsMakeOneQueryAndTheAnswerExpires(t *testing.T) { func TestRepeatedRequestsMakeOneQueryAndTheAnswerExpires(t *testing.T) {
t.Parallel() t.Parallel()
+36 -12
View File
@@ -117,7 +117,8 @@ func RunWAF(ctx context.Context, params Params) int {
"settings", cfg) "settings", cfg)
// The last write follows the stop of every hook, and so the end of every // The last write follows the stop of every hook, and so the end of every
// loop that changes the state or raises an alert. A hook of the state // loop that changes the state or raises an alert, and of every ask of
// GeoJS, the DNSBL zones, AbuseIPDB and routewatch. A hook of the state
// files would stop before the loops of the parts built ahead of them. // files would stop before the loops of the parts built ahead of them.
return run(ctx, app, failed, processLog, func() error { return run(ctx, app, failed, processLog, func() error {
err := files.WriteAll() err := files.WriteAll()
@@ -272,11 +273,13 @@ func newLists(
return lists return lists
} }
// newDNSBL returns what asks the DNSBL zones, none asked yet. // newDNSBL returns what asks the DNSBL zones, none asked yet. At the stop it
// ends the queries under way, as reputation.DNSBL.StopAsking says.
func newDNSBL( func newDNSBL(
cfg *config.WAF, processLog *slog.Logger, alertQueue *alerts.Queue, lc fx.Lifecycle, cfg *config.WAF, processLog *slog.Logger,
alertQueue *alerts.Queue,
) *reputation.DNSBL { ) *reputation.DNSBL {
return reputation.NewDNSBL(reputation.DNSBLParams{ dnsbl := reputation.NewDNSBL(reputation.DNSBLParams{
Zones: cfg.DNSBLZones, Zones: cfg.DNSBLZones,
Resolver: cfg.DNSBLResolver, Resolver: cfg.DNSBLResolver,
CacheTTL: cfg.IPReputationCacheTTL, CacheTTL: cfg.IPReputationCacheTTL,
@@ -285,12 +288,17 @@ func newDNSBL(
ProcessLog: processLog, ProcessLog: processLog,
Alerts: alertQueue, Alerts: alertQueue,
}) })
lc.Append(fx.StopHook(dnsbl.StopAsking))
return dnsbl
} }
// newAbuseIPDB returns what checks clients with AbuseIPDB while // newAbuseIPDB returns what checks clients with AbuseIPDB while
// SWW_WAF_ABUSEIPDB_KEY is set, none checked yet. Until waf stops, it // SWW_WAF_ABUSEIPDB_KEY is set, none checked yet. Until waf stops, it
// fetches AbuseIPDB's blacklist as it is due, and bans the clients on it in // fetches AbuseIPDB's blacklist as it is due, and bans the clients on it in
// ledger, as waf.BanAbuseIPDBBlacklist says. // ledger, as waf.BanAbuseIPDBBlacklist says. At the stop it ends the checks
// under way, as reputation.AbuseIPDB.StopChecking says.
func newAbuseIPDB( func newAbuseIPDB(
lc fx.Lifecycle, cfg *config.WAF, processLog *slog.Logger, lc fx.Lifecycle, cfg *config.WAF, processLog *slog.Logger,
alertQueue *alerts.Queue, ledger *bans.Ledger, alertQueue *alerts.Queue, ledger *bans.Ledger,
@@ -316,24 +324,30 @@ func newAbuseIPDB(
}) })
lc.Append(background(abuseIPDB.RunBlacklist)) lc.Append(background(abuseIPDB.RunBlacklist))
lc.Append(fx.StopHook(abuseIPDB.StopChecking))
return abuseIPDB return abuseIPDB
} }
// newRouteWatch returns what asks routewatch about clients, or nil while // newRouteWatch returns what asks routewatch about clients, or nil while
// SWW_WAF_ROUTEWATCH_URL is unset. // SWW_WAF_ROUTEWATCH_URL is unset. At the stop it ends the queries under
// way, as routewatch.Client.StopAsking says.
func newRouteWatch( func newRouteWatch(
cfg *config.WAF, processLog *slog.Logger, lc fx.Lifecycle, cfg *config.WAF, processLog *slog.Logger,
) *routewatch.Client { ) *routewatch.Client {
if cfg.RouteWatchURL == "" { if cfg.RouteWatchURL == "" {
return nil return nil
} }
return routewatch.New(routewatch.Params{ routeWatch := routewatch.New(routewatch.Params{
URL: cfg.RouteWatchURL, URL: cfg.RouteWatchURL,
Now: utcNow, Now: utcNow,
ProcessLog: processLog, ProcessLog: processLog,
}) })
lc.Append(fx.StopHook(routeWatch.StopAsking))
return routeWatch
} }
// newMetrics returns the metrics, with those read from the other parts as // newMetrics returns the metrics, with those read from the other parts as
@@ -371,12 +385,18 @@ func newMetrics(
// newGeoJS returns what looks clients up at GeoJS, none looked up yet. Each // newGeoJS returns what looks clients up at GeoJS, none looked up yet. Each
// answer goes to the limiter and the ledger, as waf.AddLookup says, and each // answer goes to the limiter and the ledger, as waf.AddLookup says, and each
// failure is handled as waf.GeoJSFailed says. // failure is handled as waf.GeoJSFailed says. At the stop it ends the request
// under way, as geojs.GeoJS.StopAsking says.
func newGeoJS( func newGeoJS(
cfg *config.WAF, m *metrics.Metrics, limiter *ratelimit.Limiter, lc fx.Lifecycle,
ledger *bans.Ledger, alertQueue *alerts.Queue, processLog *slog.Logger, cfg *config.WAF,
m *metrics.Metrics,
limiter *ratelimit.Limiter,
ledger *bans.Ledger,
alertQueue *alerts.Queue,
processLog *slog.Logger,
) *geojs.GeoJS { ) *geojs.GeoJS {
return geojs.New(geojs.Params{ geoJS := geojs.New(geojs.Params{
URL: geojs.URL, URL: geojs.URL,
Answered: func(answer geojs.Answer) { Answered: func(answer geojs.Answer) {
waf.AddLookup(cfg, limiter, ledger, answer) waf.AddLookup(cfg, limiter, ledger, answer)
@@ -388,6 +408,10 @@ func newGeoJS(
}, },
WentWithoutAnswer: m.RequestsWithoutGeoJSAnswer.Inc, WentWithoutAnswer: m.RequestsWithoutGeoJSAnswer.Inc,
}) })
lc.Append(fx.StopHook(geoJS.StopAsking))
return geoJS
} }
// newWAFServer returns the server waf runs, with its parts and the time // newWAFServer returns the server waf runs, with its parts and the time