From 56f93d4f743ff047f2b92d75c0ea157b3f286821 Mon Sep 17 00:00:00 2001 From: sneak Date: Mon, 16 Nov 2026 23:18:27 +0000 Subject: [PATCH] geojs: a visitor's wait no longer cuts off the request to GeoJS (closes #404) The GeoJS lookup and its cache move from internal/lookup to internal/geojs, which imports no other package of this repository. waf hands it functions to count requests, report failures and count lookups that went without an answer; waf keeps the log line, the source_failure alert and the metrics. The caller now passes its wait to each lookup. waf passes SWW_WAF_ASN_AND_COUNTRY_LOOKUP_TIMEOUT_MS. A request to GeoJS has its own fixed limit of 10 seconds, and only passing it or an error from GeoJS is a failure. An answer that comes after the caller stopped waiting is kept. internal/lookup keeps the lookup database and returns geojs.Answer. Model: opus-5-5 --- README.md | 8 +- SPEC.md | 35 +- docs/settings.md | 5 +- internal/{lookup => geojs}/export_test.go | 2 +- internal/{lookup/lookup.go => geojs/geojs.go} | 141 ++-- .../lookup_test.go => geojs/geojs_test.go} | 698 ++++++++++-------- internal/{lookup => geojs}/snapshot_test.go | 22 +- internal/lookup/file.go | 8 +- internal/lookup/file_internal_test.go | 3 +- internal/metrics/metrics.go | 3 +- internal/state/state.go | 8 +- internal/state/state_test.go | 10 +- internal/sww/sww.go | 3 +- internal/waf/answertime_test.go | 13 +- internal/waf/biased_test.go | 4 + internal/waf/lookup.go | 32 +- internal/waf/lookup_test.go | 21 +- internal/waf/metrics_test.go | 64 +- internal/waf/proxy.go | 37 +- internal/waf/ratelimits_test.go | 4 +- internal/waf/request.go | 8 +- 21 files changed, 645 insertions(+), 484 deletions(-) rename internal/{lookup => geojs}/export_test.go (93%) rename internal/{lookup/lookup.go => geojs/geojs.go} (75%) rename internal/{lookup/lookup_test.go => geojs/geojs_test.go} (60%) rename internal/{lookup => geojs}/snapshot_test.go (85%) diff --git a/README.md b/README.md index eb4ed0b..6cf5164 100644 --- a/README.md +++ b/README.md @@ -329,7 +329,7 @@ Every setting the code reads, with its default: | `SWW_WAF_BAN_SCOPE_V4_PREFIX` | `32` | The prefix length of the netblock a ban on an IPv4 client covers, from 0 to 32. | | `SWW_WAF_LOOKUP_SOURCE` | `geojs` | Where each client's AS number and country come from: `geojs` (a web service), `file` (a local database) or `off`. | | `SWW_WAF_LOOKUP_DB_PATH` | empty | The IPinfo Lite database file, in its `.mmdb` form, read while the source is `file`. | -| `SWW_WAF_ASN_AND_COUNTRY_LOOKUP_TIMEOUT_MS` | `800` | How long a request waits for its client's first GeoJS answer when a setting needs it, and how long a request to GeoJS may take, in milliseconds. | +| `SWW_WAF_ASN_AND_COUNTRY_LOOKUP_TIMEOUT_MS` | `800` | How long a request waits for its client's first GeoJS answer when a setting needs it, in milliseconds. The request to GeoJS goes on, and a later answer is kept. | | `SWW_WAF_ADD_LOOKUP_HEADERS` | `true`, or `false` while the lookup source is `off` | `true` passes the app the client's AS number and country in `X-Client-ASN` and `X-Client-Country`. | | `SWW_WAF_EXCLUSIVELY_ALLOWED_COUNTRIES` | empty | When set, the only countries whose clients get through; a client the lookup cannot place is refused too. | | `SWW_WAF_TAG_LISTS` | empty | Address lists that each add a tag to their clients' requests, as `name=URL` pairs, such as `torexit=https://example.com/tor.txt`. | @@ -749,9 +749,9 @@ stops the running process (see [`docs/state-files.md`](docs/state-files.md)). - `internal/waf`: `waf`'s server, which runs the checks and answers its own endpoints. - `internal/ratelimit`, `internal/bans`, `internal/rules`, - `internal/coreruleset`, `internal/lookup`, `internal/reputation`, - `internal/routewatch`, `internal/anomaly` and `internal/alerts`: one check or - source each. + `internal/coreruleset`, `internal/geojs`, `internal/lookup`, + `internal/reputation`, `internal/routewatch`, `internal/anomaly` and + `internal/alerts`: one check or source each. - `internal/state`, `internal/requestlog`, `internal/remotelog` and `internal/metrics`: the state and cache files, the log lines, the syslog sender and the metrics. diff --git a/SPEC.md b/SPEC.md index 8a2a71f..a86a50f 100644 --- a/SPEC.md +++ b/SPEC.md @@ -87,8 +87,10 @@ never as a time. change, and tells which rules a request matches. - `internal/coreruleset`: the Core Rule Set, with the five changes and the apps' fixes listed under "Attack detection", as Coraza directives. - - `internal/lookup`: looks clients up through GeoJS or in the lookup - database. + - `internal/geojs`: looks clients up through GeoJS and keeps the answers. It + imports no other package of this repository: `waf` gives it what it needs + to count and report failures. + - `internal/lookup`: looks clients up in the lookup database. - `internal/reputation`: fetches the lists, asks the DNS blocklists and AbuseIPDB in the background, and keeps their answers. - `internal/routewatch`: asks a routewatch instance about clients in the @@ -507,19 +509,25 @@ service that keeps its visitors' addresses on its own host. dropped from that table. The answer reaches the client's history and ban notes when it comes. - `SWW_WAF_ASN_AND_COUNTRY_LOOKUP_TIMEOUT_MS` is how long a visitor's request - waits for its client's first answer, when a setting needs it (see step 6). It - also bounds each request to GeoJS. A client whose answer has not come in time - counts as unknown until it comes, and its later requests do not wait. Its - address is asked about again in the background. + waits for its client's first answer, when a setting needs it (see step 6). A + client whose answer has not come in time counts as unknown until it comes, and + its later requests do not wait. +- A request to GeoJS may take up to 10 seconds, however long visitors wait. It + goes on after the visitor's request ends, and its answer is kept. Say GeoJS + answers after 5 seconds. The visitor's request is decided after 800 + milliseconds, without a country. A request from the same client 6 seconds + later gets the answer. +- Requests from one client that come together share one request to GeoJS. - A client the lookup cannot place has an unknown country and AS number, which the country list and the biased thresholds treat as described there. Nothing else treats it differently. - GeoJS publishes no rate limit, but its terms forbid "an excessive amount of - API requests", judged by GeoJS alone, and let it block a caller. While GeoJS - is slow, down or refusing `waf`, new clients count as unknown. GeoJS is then - left alone for a second, and twice as long after each further failure, up to - five minutes. Each failure raises a `source_failure` alert (see "Alerting"). A - service that cannot accept this uses the file. + API requests", judged by GeoJS alone, and let it block a caller. A request to + GeoJS fails when it takes over 10 seconds or GeoJS answers with an error. + While GeoJS is slow, down or refusing `waf`, new clients count as unknown. + After a failure GeoJS is left alone for a second, and twice as long after each + further failure, up to five minutes. Each failure raises a `source_failure` + alert (see "Alerting"). A service that cannot accept this uses the file. - The file is the free IPinfo Lite database (`ipinfo_lite.mmdb`), whose `asn`, `as_name` and `country_code` fields are read. The operator downloads it with a free IPinfo account and mounts it read-only; `waf` never fetches it and holds @@ -2411,8 +2419,9 @@ within that time. GeoJS is the only outside source a question can wait on. A question waits only when step 6 says so, and for at most `SWW_WAF_ASN_AND_COUNTRY_LOOKUP_TIMEOUT_MS`, by default 800 milliseconds. A -client without an answer by then counts as unknown. No question waits on these, -which run in the background: +client without an answer by then counts as unknown. The setting bounds only the +question's wait: the request to GeoJS goes on, and its answer is kept for later +questions. No question waits on these, which run in the background: - the DNS blocklists and AbuseIPDB; - fetching the blocklists, the tag lists and the file diff --git a/docs/settings.md b/docs/settings.md index aae3bf5..51cbb2e 100644 --- a/docs/settings.md +++ b/docs/settings.md @@ -159,8 +159,9 @@ cut off; the client is banned. - `SWW_WAF_LOOKUP_DB_PATH` (default empty): the IPinfo Lite database file, in its `.mmdb` form, for `SWW_WAF_LOOKUP_SOURCE=file`. - `SWW_WAF_ASN_AND_COUNTRY_LOOKUP_TIMEOUT_MS` (default `800`): how long a - request waits for its client's first answer from GeoJS, and how long a request - to GeoJS may take. A time in milliseconds; never `off`. + request waits for its client's first answer from GeoJS. A time in + milliseconds; never `off`. It does not cut off the request to GeoJS, whose + answer is kept when it comes later. - `SWW_WAF_ADD_LOOKUP_HEADERS` (default `true`): `true` has `rproxy` pass the app the client's AS number in `X-Client-ASN` and its country in `X-Client-Country`. The default is `false` while `SWW_WAF_LOOKUP_SOURCE` is diff --git a/internal/lookup/export_test.go b/internal/geojs/export_test.go similarity index 93% rename from internal/lookup/export_test.go rename to internal/geojs/export_test.go index 3276a6c..360ff2d 100644 --- a/internal/lookup/export_test.go +++ b/internal/geojs/export_test.go @@ -1,4 +1,4 @@ -package lookup +package geojs import "net/http" diff --git a/internal/lookup/lookup.go b/internal/geojs/geojs.go similarity index 75% rename from internal/lookup/lookup.go rename to internal/geojs/geojs.go index d5d5db0..962bc51 100644 --- a/internal/lookup/lookup.go +++ b/internal/geojs/geojs.go @@ -1,9 +1,9 @@ -// Package lookup looks up each client's AS number and country, through -// the GeoJS web service or in the lookup database, the IPinfo Lite file -// SWW_WAF_LOOKUP_DB_PATH names. GeoJS's answers are kept in memory, for at -// most 100,000 clients and for 7 days each, and are written to -// lookups.json and read from it by the state package. -package lookup +// Package geojs looks up clients' AS numbers and countries through the GeoJS +// web service. Its answers are kept in memory, for at most 100,000 clients and +// for 7 days each. A caller waits for an answer as long as it says. A request to +// GeoJS goes on after its callers stop waiting, and its answer is kept. The +// package imports no other package of this repository. +package geojs import ( "context" @@ -11,7 +11,6 @@ import ( "errors" "fmt" "io" - "log/slog" "net/http" "net/netip" "slices" @@ -21,8 +20,6 @@ import ( "time" "github.com/hashicorp/golang-lru/v2/simplelru" - "sneak.berlin/go/smallwebwaf/internal/alerts" - "sneak.berlin/go/smallwebwaf/internal/metrics" ) // URL is GeoJS's endpoint for an address's place and network. Asked about @@ -43,6 +40,9 @@ const ( maxWaiting = 10000 // maxPerRequest is how many addresses one request to GeoJS asks about. maxPerRequest = 200 + // requestTimeout is how long a request to GeoJS may take before it is + // abandoned, which is a failure. It is apart from how long callers wait. + requestTimeout = 10 * time.Second // unknownASN is the AS number GeoJS gives when it knows none. unknownASN = 64512 // After a failure GeoJS is not asked again for a second, and for @@ -64,39 +64,32 @@ var ( type Params struct { // URL is where GeoJS is asked, normally URL. URL string - // Timeout is how long a request waits for its client's first answer, - // and how long a request to GeoJS may take before it is abandoned - // (SWW_WAF_ASN_AND_COUNTRY_LOOKUP_TIMEOUT_MS). - Timeout time.Duration - // Wait is true when a setting needs each request's answer before the - // request goes on. Otherwise no request waits for one. - Wait bool // Answered, unless nil, is given each answer GeoJS gives, once it is // kept. Answered func(Answer) // Now tells the time, normally time.Now in UTC. Now func() time.Time - // ProcessLog receives GeoJS's failures. - ProcessLog *slog.Logger - // Metrics count the requests to GeoJS, those that failed, and the - // clients that go without an answer. - Metrics *metrics.Metrics - // Alerts receive a source_failure alert each time GeoJS fails. - Alerts *alerts.Queue + // Asked is called for each request to GeoJS. + Asked func() + // Failed is called for each failure of GeoJS, with what went wrong and + // how long GeoJS is left alone after it. The error never names the + // addresses asked about. + Failed func(err error, askingAgainIn time.Duration) + // WentWithoutAnswer is called each time a caller that waits for an + // answer goes on without one. + WentWithoutAnswer func() } // GeoJS looks up clients' AS numbers and countries through GeoJS. At most // one request to GeoJS is under way at a time, and it asks about every // client waiting, up to maxPerRequest. It is safe for concurrent use. type GeoJS struct { - url string - timeout time.Duration - wait bool - answered func(Answer) - now func() time.Time - processLog *slog.Logger - metrics *metrics.Metrics - alerts *alerts.Queue + url string + answered func(Answer) + now func() time.Time + asked func() + failed func(err error, askingAgainIn time.Duration) + wentWithoutAnswer func() // httpClient follows no redirect, so that visitors' addresses go to // GeoJS alone: a redirect is a failure. httpClient *http.Client @@ -114,12 +107,10 @@ type GeoJS struct { retryAt time.Time } -// Answer is what GeoJS or the lookup database said about a client: its AS -// number, such as AS64496, and the AS's name, both "" when the source knows -// no AS number for it; its country, "" when the source cannot place it; -// when the source said so; and, for GeoJS's answers, which lookups.json -// holds, when the answer was last used. The zero Answer is that of a -// client with no answer. +// Answer is what GeoJS said about a client: its AS number, such as AS64496, +// and the AS's name, both "" when GeoJS knows no AS number for it; its +// country, "" when GeoJS cannot place it; when GeoJS said so; and when the +// answer was last used. The zero Answer is that of a client with no answer. // //nolint:tagliatelle // the state files use snake_case, as the request log does type Answer struct { @@ -137,7 +128,7 @@ type wait struct { // replaced each time GeoJS fails before then. asked chan struct{} // late is true once the client has gone without an answer, for a - // whole timeout or because GeoJS failed: its requests no longer wait. + // whole wait or because GeoJS failed: its callers no longer wait. late bool } @@ -149,14 +140,12 @@ func New(params Params) *GeoJS { } return &GeoJS{ - url: params.URL, - timeout: params.Timeout, - wait: params.Wait, - answered: params.Answered, - now: params.Now, - processLog: params.ProcessLog, - metrics: params.Metrics, - alerts: params.Alerts, + url: params.URL, + answered: params.Answered, + now: params.Now, + asked: params.Asked, + failed: params.Failed, + wentWithoutAnswer: params.WentWithoutAnswer, httpClient: &http.Client{ CheckRedirect: func(*http.Request, []*http.Request) error { return http.ErrUseLastResponse @@ -170,21 +159,22 @@ func New(params Params) *GeoJS { // LookUp returns the answer GeoJS gave about client, with its country as // a two-letter code in capitals, or the zero Answer when there is none // yet. An answer is kept for 7 days. Without one, the client is asked -// about in the background, and, while Wait is set, the request waits up -// to Timeout for the answer, unless the client has gone without one -// before. ctx is the context of the client's request, and ends the wait -// when it ends. +// about in the background, and LookUp waits up to wait for the answer, +// unless the client has gone without one before. The wait ends early when +// ctx ends. Neither ends the request to GeoJS. // // GeoJS is asked about the client's first address, which is the client's // own address for IPv4, and an address in the same place for an IPv6 // netblock. -func (g *GeoJS) LookUp(ctx context.Context, client netip.Prefix) Answer { - answer, asked := g.answerOrWait(ctx, client, g.wait) +func (g *GeoJS) LookUp( + ctx context.Context, client netip.Prefix, wait time.Duration, +) Answer { + answer, asked := g.answerOrWait(ctx, client, true) if asked == nil { return answer } - timer := time.NewTimer(g.timeout) + timer := time.NewTimer(wait) defer timer.Stop() select { @@ -198,7 +188,7 @@ func (g *GeoJS) LookUp(ctx context.Context, client netip.Prefix) Answer { answer, found := g.kept(client) if !found { - g.metrics.RequestsWithoutGeoJSAnswer.Inc() + g.wentWithoutAnswer() } w, waiting := g.waiting[client] @@ -209,8 +199,8 @@ func (g *GeoJS) LookUp(ctx context.Context, client netip.Prefix) Answer { return answer } -// LookUpWithoutWaiting is LookUp for a request that does not need the answer, -// whatever Wait says: it returns the zero Answer at once when none is kept. +// LookUpWithoutWaiting is LookUp for a caller that does not need the answer: +// it returns the zero Answer at once when none is kept. func (g *GeoJS) LookUpWithoutWaiting( ctx context.Context, client netip.Prefix, @@ -228,8 +218,7 @@ func (g *GeoJS) Kept(client netip.Prefix) (Answer, bool) { return g.kept(client) } -// Snapshot returns every answer kept, sorted by client, as lookups.json -// lists them. +// Snapshot returns every answer kept, sorted by client. func (g *GeoJS) Snapshot() []Answer { g.mu.Lock() @@ -247,9 +236,9 @@ func (g *GeoJS) Snapshot() []Answer { return answers } -// Load keeps answers read from lookups.json, in place of the answers it -// keeps, in the order they were last used, so that the one used longest -// ago is dropped first. Answers GeoJS gave keepFor ago or more are +// Load keeps answers, such as those of an earlier Snapshot, in place of the +// answers it keeps, in the order they were last used, so that the one used +// longest ago is dropped first. Answers GeoJS gave keepFor ago or more are // dropped. func (g *GeoJS) Load(answers []Answer) { answers = slices.Clone(answers) @@ -295,11 +284,11 @@ func (g *GeoJS) answerOrWait( g.ask(ctx) if !mustWait { - return Answer{}, nil // the answer is not needed before the request goes on + return Answer{}, nil // the answer is not needed before the caller goes on } if w == nil { - g.metrics.RequestsWithoutGeoJSAnswer.Inc() + g.wentWithoutAnswer() return Answer{}, nil // too many clients wait already } @@ -310,7 +299,7 @@ func (g *GeoJS) answerOrWait( } if w.late { - g.metrics.RequestsWithoutGeoJSAnswer.Inc() + g.wentWithoutAnswer() return Answer{}, nil } @@ -335,8 +324,8 @@ func (g *GeoJS) kept(client netip.Prefix) (Answer, bool) { // ask starts asking GeoJS about the waiting clients, unless a request to // it is under way or it is left alone after a failure. The requests to -// GeoJS are for every client waiting, so they go on when the client's -// request whose ctx is given ends. +// GeoJS are for every client waiting, so they go on when the caller whose +// ctx is given stops waiting. func (g *GeoJS) ask(ctx context.Context) { if g.asking || g.now().Before(g.retryAt) { return @@ -441,8 +430,6 @@ func (g *GeoJS) keep( } if err != nil { - g.metrics.GeoJSFailures.Inc() - g.retryDelay = min( max(retryDelayFactor*g.retryDelay, firstRetryDelay), maxRetryDelay, @@ -457,16 +444,7 @@ func (g *GeoJS) keep( w.late = true } - g.processLog.Warn("asking GeoJS failed", - "error", err.Error(), "asking_again_in", g.retryDelay.String()) - g.alerts.Raise(alerts.Alert{ - Event: alerts.EventSourceFailure, - Reason: "asking GeoJS failed", - Detail: map[string]any{ - "source": "geojs", "error": err.Error(), - "asking_again_in": g.retryDelay.String(), - }, - }) + g.failed(err, g.retryDelay) return kept, false } @@ -479,7 +457,8 @@ func (g *GeoJS) keep( // request asks GeoJS about clients in one request, and returns the answer // for each address GeoJS's answer names: its AS number and the AS's name, // both "" for the AS number 64512, which GeoJS gives when it knows none, -// and its country, in capitals. +// and its country, in capitals. The request is abandoned after +// requestTimeout. func (g *GeoJS) request( ctx context.Context, clients []netip.Prefix, ) (map[netip.Addr]Answer, error) { @@ -489,7 +468,7 @@ func (g *GeoJS) request( addrs = append(addrs, client.Addr().String()) } - ctx, cancel := context.WithTimeout(ctx, g.timeout) + ctx, cancel := context.WithTimeout(ctx, requestTimeout) defer cancel() req, err := http.NewRequestWithContext( @@ -504,7 +483,7 @@ func (g *GeoJS) request( req.URL.RawQuery = "ip=" + strings.Join(addrs, ",") - g.metrics.GeoJSRequests.Inc() + g.asked() res, err := g.httpClient.Do(req) if err != nil { diff --git a/internal/lookup/lookup_test.go b/internal/geojs/geojs_test.go similarity index 60% rename from internal/lookup/lookup_test.go rename to internal/geojs/geojs_test.go index 4f67d02..a793f6b 100644 --- a/internal/lookup/lookup_test.go +++ b/internal/geojs/geojs_test.go @@ -1,13 +1,11 @@ -package lookup_test +package geojs_test import ( + "context" "encoding/json" - "log/slog" "net/http" "net/http/httptest" "net/netip" - "net/url" - "reflect" "slices" "strings" "sync" @@ -15,10 +13,7 @@ import ( "testing/synctest" "time" - "github.com/prometheus/client_golang/prometheus/testutil" - "sneak.berlin/go/smallwebwaf/internal/alerts" - "sneak.berlin/go/smallwebwaf/internal/lookup" - "sneak.berlin/go/smallwebwaf/internal/metrics" + "sneak.berlin/go/smallwebwaf/internal/geojs" ) const ( @@ -34,8 +29,14 @@ const ( // leftOut is the address it leaves out of its answer when // answeringWithoutLeftOut. leftOut = "203.0.113.7" - // timeout is how long a new client waits for its answer. - timeout = time.Second + // wait is how long a lookup waits for a new client's answer. + wait = time.Second + // requestTimeout is how long a request to GeoJS may take before it is + // abandoned. + requestTimeout = 10 * time.Second + // lateAnswer is how long the stand-in takes to answer when + // answeringLate. + lateAnswer = 5 * time.Second // week is how long an answer is kept. week = 7 * 24 * time.Hour ) @@ -51,24 +52,24 @@ func TestKeptAnswerIsUsedFor7DaysThenAskedAgain(t *testing.T) { t.Parallel() synctest.Test(t, func(t *testing.T) { - geojs, clock, g := start() + service, clock, g := start() placed := netip.MustParsePrefix("203.0.113.9/32") notPlaced := netip.MustParsePrefix(unplaced + "/32") wantCountry(t, g, placed, germany) wantCountry(t, g, notPlaced, "") - wantRequests(t, geojs, 2) + wantRequests(t, service, 2) // An answer without a country is kept too. clock.advance(week - time.Second) wantCountry(t, g, placed, germany) wantCountry(t, g, notPlaced, "") - wantRequests(t, geojs, 2) + wantRequests(t, service, 2) clock.advance(time.Second) wantCountry(t, g, placed, germany) - wantRequests(t, geojs, 3) - wantAsked(t, geojs, 2, "203.0.113.9") + wantRequests(t, service, 3) + wantAsked(t, service, 2, "203.0.113.9") }) } @@ -76,17 +77,17 @@ func TestLookUpWithoutWaitingAsksGeoJSAndGoesOnAtOnce(t *testing.T) { t.Parallel() synctest.Test(t, func(t *testing.T) { - geojs, clock, g := start() + service, clock, g := start() client := netip.MustParsePrefix("203.0.113.9/32") - // Wait is set, and GeoJS does not answer. - geojs.set(hanging) + // GeoJS does not answer. + service.set(hanging) began := time.Now() answer := g.LookUpWithoutWaiting(t.Context(), client) if waited := time.Since(began); waited != 0 || - answer != (lookup.Answer{}) { + answer != (geojs.Answer{}) { t.Errorf( "got %+v after %s, want no answer at once", answer, @@ -94,14 +95,14 @@ func TestLookUpWithoutWaitingAsksGeoJSAndGoesOnAtOnce(t *testing.T) { ) } - waitForRequests(t, geojs, 1) - wantAsked(t, geojs, 0, "203.0.113.9") + waitForRequests(t, service, 1) + wantAsked(t, service, 0, "203.0.113.9") // That request is abandoned, and has failed once this returns. GeoJS // answers when it is asked again, and the answer is kept. - time.Sleep(timeout) + time.Sleep(requestTimeout) synctest.Wait() - geojs.set(answering) + service.set(answering) waitForCountry(t, g, clock, client, germany) if got := g.LookUpWithoutWaiting(t.Context(), client).Country; got != germany { @@ -114,39 +115,45 @@ func TestLookUpWithoutWaitingAsksGeoJSAndGoesOnAtOnce(t *testing.T) { }) } -func TestNewClientWaitsAtMostOneSecondThenCountsAsNotFound(t *testing.T) { +func TestNewClientWaitsAsLongAsItsCallerSaysThenCountsAsNotFound( + t *testing.T, +) { t.Parallel() synctest.Test(t, func(t *testing.T) { - geojs, clock, g := start() + service, clock, g := start() client := netip.MustParsePrefix("203.0.113.9/32") // The client comes while GeoJS is asked about an earlier client, which // it answers most of a second later. It is then asked about the client - // and does not answer: that request is abandoned a second after it + // and does not answer: that request is abandoned ten seconds after it // began, well after the client's wait is over. - geojs.set(answeringSlowly) + service.set(answeringSlowly) var earlier sync.WaitGroup - earlier.Go( - func() { g.LookUp(t.Context(), netip.MustParsePrefix("203.0.113.1/32")) }, - ) + earlier.Go(func() { + g.LookUp( + t.Context(), + netip.MustParsePrefix("203.0.113.1/32"), + wait, + ) + }) defer earlier.Wait() - waitForRequests(t, geojs, 1) - geojs.set(hanging) + waitForRequests(t, service, 1) + service.set(hanging) began := time.Now() wantCountry(t, g, client, "") took := time.Since(began) - if took != timeout { - t.Errorf("waited %s for the answer, want %s", took, timeout) + if took != wait { + t.Errorf("waited %s for the answer, want %s", took, wait) } - // Its next request does not wait. + // Its next lookup does not wait. began = time.Now() wantCountry(t, g, client, "") @@ -158,66 +165,212 @@ func TestNewClientWaitsAtMostOneSecondThenCountsAsNotFound(t *testing.T) { // Once GeoJS answers, the client is asked about again in the // background, and has its country. - geojs.set(answering) + service.set(answering) waitForCountry(t, g, clock, client, germany) }) } -func TestRequestWaitsAsLongAsTheTimeoutSaysAndGeoJSIsAbandonedAfterIt( +func TestAnswerAfterTheWaitIsKeptAndIsNoFailure(t *testing.T) { + t.Parallel() + + synctest.Test(t, func(t *testing.T) { + service, _, g, reports := startWithReports() + client := netip.MustParsePrefix("203.0.113.9/32") + + // GeoJS answers after the lookup's wait is over. + service.set(answeringLate) + wantCountry(t, g, client, "") + + time.Sleep(lateAnswer - wait) + synctest.Wait() + + // The answer is kept, and the next lookup returns it without asking + // GeoJS again. + answer, found := g.Kept(client) + if !found || answer.Country != germany { + t.Errorf( + "kept %+v, %t; want the answer in %s", + answer, + found, + germany, + ) + } + + wantCountry(t, g, client, germany) + wantRequests(t, service, 1) + reports.wantFailures(t) + + // GeoJS is not left alone: a new client is asked about at once. + service.set(answering) + wantCountry(t, g, netip.MustParsePrefix("203.0.113.10/32"), germany) + wantRequests(t, service, 2) + }) +} + +func TestAnswerAfter5SecondsIsReadFromTheCache6SecondsAfterTheFirstLookUp( t *testing.T, ) { t.Parallel() synctest.Test(t, func(t *testing.T) { - // A timeout longer than the default, and a GeoJS that does not - // answer. - const longerTimeout = 3 * time.Second + const shortWait = 800 * time.Millisecond - m := metrics.New(1, "app") - g := lookup.New(lookup.Params{ - URL: lookup.URL, - Timeout: longerTimeout, - Wait: true, - Now: time.Now, - ProcessLog: slog.New(slog.DiscardHandler), - Metrics: m, - Alerts: alerts.New(alerts.Params{}), - }) - g.SetTransport(&standIn{answers: hanging}) + service, _, g := start() + client := netip.MustParsePrefix("203.0.113.9/32") - var ( - request sync.WaitGroup - waited time.Duration - ) + service.set(answeringLate) - request.Go(func() { - began := time.Now() + began := time.Now() - wantCountry(t, g, netip.MustParsePrefix("203.0.113.9/32"), "") - - waited = time.Since(began) - }) - - // A moment before the timeout runs out, GeoJS is still being asked: - // the request to it has not failed. - time.Sleep(longerTimeout - time.Millisecond) - synctest.Wait() - wantFailures(t, m, 0) - - // As it runs out, the client's request goes on, and the request to - // GeoJS is abandoned, which counts as a failure. - request.Wait() - synctest.Wait() - - if waited != longerTimeout { + answer := g.LookUp(t.Context(), client, shortWait) + if took := time.Since(began); took != shortWait || + answer != (geojs.Answer{}) { t.Errorf( - "waited %s for the answer, want %s", - waited, - longerTimeout, + "got %+v after %s, want no answer after %s", + answer, + took, + shortWait, ) } - wantFailures(t, m, 1) + time.Sleep(6*time.Second - shortWait) + + began = time.Now() + + answer = g.LookUp(t.Context(), client, shortWait) + if took := time.Since(began); took != 0 || + answer.Country != germany { + t.Errorf( + "got %+v after %s, want the answer in %s at once", + answer, + took, + germany, + ) + } + + wantRequests(t, service, 1) + }) +} + +func TestTenLookUpsAtOnceOfOneClientCauseOneRequest(t *testing.T) { + t.Parallel() + + synctest.Test(t, func(t *testing.T) { + service, _, g := start() + client := netip.MustParsePrefix("203.0.113.9/32") + + service.set(answeringSlowly) + + var lookUps sync.WaitGroup + + for range 10 { + lookUps.Go(func() { wantCountry(t, g, client, germany) }) + } + + lookUps.Wait() + wantRequests(t, service, 1) + wantAsked(t, service, 0, "203.0.113.9") + }) +} + +func TestCallerWhoseContextEndsEarlyDoesNotStopTheRequest(t *testing.T) { + t.Parallel() + + synctest.Test(t, func(t *testing.T) { + const early = 100 * time.Millisecond + + service, _, g, reports := startWithReports() + client := netip.MustParsePrefix("203.0.113.9/32") + + service.set(answeringLate) + + ctx, cancel := context.WithTimeout(t.Context(), early) + defer cancel() + + began := time.Now() + + answer := g.LookUp(ctx, client, wait) + if took := time.Since(began); took != early || + answer != (geojs.Answer{}) { + t.Errorf( + "got %+v after %s, want no answer after %s", + answer, + took, + early, + ) + } + + time.Sleep(lateAnswer) + synctest.Wait() + + answer, found := g.Kept(client) + if !found || answer.Country != germany { + t.Errorf( + "kept %+v, %t; want the answer in %s", + answer, + found, + germany, + ) + } + + wantRequests(t, service, 1) + reports.wantFailures(t) + }) +} + +func TestRequestOver10SecondsIsAFailure(t *testing.T) { + t.Parallel() + + synctest.Test(t, func(t *testing.T) { + service, _, g, reports := startWithReports() + + // GeoJS does not answer. + service.set(hanging) + wantCountry(t, g, netip.MustParsePrefix("203.0.113.9/32"), "") + + // A moment before 10 seconds are up, the request has not failed. + time.Sleep(requestTimeout - wait - time.Millisecond) + synctest.Wait() + reports.wantFailures(t) + + // As they run out, it is abandoned, which is a failure. The error + // does not name the address asked about. + time.Sleep(time.Millisecond) + synctest.Wait() + reports.wantFailures(t, failure{ + err: "ask GeoJS: context deadline exceeded", + askingAgainIn: time.Second, + }) + + // GeoJS is then left alone, so a new client is not asked about. + wantCountry(t, g, netip.MustParsePrefix("203.0.113.10/32"), "") + wantRequests(t, service, 1) + }) +} + +func TestFailureIsReportedWithHowLongGeoJSIsLeftAlone(t *testing.T) { + t.Parallel() + + synctest.Test(t, func(t *testing.T) { + service, clock, g, reports := startWithReports() + clients := newClients() + + service.set(failing) + wantCountry(t, g, clients(), "") + + clock.advance(time.Second) + wantCountry(t, g, clients(), "") + + reports.wantFailures(t, + failure{ + err: "GeoJS answered 503 Service Unavailable", + askingAgainIn: time.Second, + }, + failure{ + err: "GeoJS answered 503 Service Unavailable", + askingAgainIn: 2 * time.Second, + }, + ) }) } @@ -238,32 +391,32 @@ func TestAddressLeftOutOfAnAnswerIsAskedAboutAgain(t *testing.T) { t.Parallel() synctest.Test(t, func(t *testing.T) { - geojs, clock, g := start() + service, clock, g := start() other := netip.MustParsePrefix("203.0.113.1/32") client := netip.MustParsePrefix(leftOut + "/32") // GeoJS fails, and is left alone for a second while the client // comes too, so that the next request asks about both. - geojs.set(failing) + service.set(failing) wantCountry(t, g, other, "") wantCountry(t, g, client, "") - geojs.set(tc.answers) + service.set(tc.answers) clock.advance(time.Second) wantCountry(t, g, other, "") - waitForRequests(t, geojs, 2) + waitForRequests(t, service, 2) // The answer counts as a failure, and the client is asked about // again, with the other client only if the answer left it out too. - geojs.set(answering) + service.set(answering) waitForCountry(t, g, clock, client, germany) wantCountry(t, g, other, germany) - wantRequests(t, geojs, 3) + wantRequests(t, service, 3) if tc.named { - wantAsked(t, geojs, 2, leftOut) + wantAsked(t, service, 2, leftOut) } else { - wantAsked(t, geojs, 2, leftOut, "203.0.113.1") + wantAsked(t, service, 2, leftOut, "203.0.113.1") } }) }) @@ -274,11 +427,11 @@ func TestRedirectCountsAsFailure(t *testing.T) { t.Parallel() synctest.Test(t, func(t *testing.T) { - geojs, _, g := start() - geojs.set(redirecting) + service, _, g := start() + service.set(redirecting) wantCountry(t, g, netip.MustParsePrefix("203.0.113.9/32"), "") - wantRequests(t, geojs, 1) + wantRequests(t, service, 1) }) } @@ -286,8 +439,8 @@ func TestCountryIsKeptInCapitals(t *testing.T) { t.Parallel() synctest.Test(t, func(t *testing.T) { - geojs, _, g := start() - geojs.set(answeringInLowerCase) + service, _, g := start() + service.set(answeringInLowerCase) wantCountry(t, g, netip.MustParsePrefix("203.0.113.9/32"), germany) }) @@ -304,14 +457,14 @@ func TestAnswerHoldsTheASNumberTheASNameAndTheCountry(t *testing.T) { // For the client it cannot place, GeoJS gives the AS number 64512 // and the AS name Unknown, which count as unknown. - for client, want := range map[netip.Prefix]lookup.Answer{ + for client, want := range map[netip.Prefix]geojs.Answer{ placed: { Client: placed, ASN: asn, ASName: asName, Country: germany, Answered: now, Used: now, }, notPlaced: {Client: notPlaced, Answered: now, Used: now}, } { - got := g.LookUp(t.Context(), client) + got := g.LookUp(t.Context(), client, wait) if got != want { t.Errorf("answer for %s\n%+v\nwant\n%+v", client, got, want) } @@ -319,146 +472,54 @@ func TestAnswerHoldsTheASNumberTheASNameAndTheCountry(t *testing.T) { }) } -func TestWithoutWaitTheRequestGoesOnAtOnceAndTheAnswerIsGivenWhenItComes( - t *testing.T, -) { +func TestEachAnswerIsGivenToAnsweredOnceKept(t *testing.T) { t.Parallel() synctest.Test(t, func(t *testing.T) { var ( mu sync.Mutex - given []lookup.Answer + given []geojs.Answer ) - geojs := &standIn{answers: answeringSlowly} + service := &standIn{answers: answeringSlowly} clock := newClock() - m := metrics.New(1, "app") - g := lookup.New(lookup.Params{ - URL: lookup.URL, - Timeout: timeout, - Answered: func(answer lookup.Answer) { + g, reports := newGeoJS(service, geojs.Params{ + Answered: func(answer geojs.Answer) { mu.Lock() defer mu.Unlock() given = append(given, answer) }, - Now: clock.Now, - ProcessLog: slog.New(slog.DiscardHandler), - Metrics: m, - Alerts: alerts.New(alerts.Params{}), + Now: clock.Now, }) - g.SetTransport(geojs) client := netip.MustParsePrefix("203.0.113.9/32") - // The request goes on at once, without an answer, and GeoJS is asked - // about the client, which it answers most of a second later. - began := time.Now() - - got := g.LookUp(t.Context(), client) - if took := time.Since(began); took != 0 || got != (lookup.Answer{}) { - t.Errorf( - "waited %s for %+v, want no wait and no answer", - took, - got, - ) + // GeoJS answers about the client most of a second later. + if got := g.LookUpWithoutWaiting(t.Context(), client); got != (geojs.Answer{}) { + t.Errorf("got %+v, want no answer", got) } - waitForRequests(t, geojs, 1) - wantAsked(t, geojs, 0, "203.0.113.9") + waitForRequests(t, service, 1) + wantAsked(t, service, 0, "203.0.113.9") - time.Sleep(timeout) + time.Sleep(wait) synctest.Wait() now := clock.Now() - want := lookup.Answer{ + want := geojs.Answer{ Client: client, ASN: asn, ASName: asName, Country: germany, Answered: now, Used: now, } mu.Lock() - if !slices.Equal(given, []lookup.Answer{want}) { + if !slices.Equal(given, []geojs.Answer{want}) { t.Errorf("answers given %+v, want only %+v", given, want) } mu.Unlock() wantCountry(t, g, client, germany) - wantRequestsWithoutGeoJSAnswer(t, m, 0) - }) -} - -func TestFailureIsLoggedWithoutTheAddressesAskedAbout(t *testing.T) { - t.Parallel() - - synctest.Test(t, func(t *testing.T) { - var log strings.Builder - - // GeoJS does not answer, so the request to it is abandoned, and fails. - geojs := &standIn{answers: hanging} - g := lookup.New(lookup.Params{ - URL: lookup.URL, - Timeout: timeout, - Wait: true, - Now: time.Now, - ProcessLog: slog.New(slog.NewTextHandler(&log, nil)), - Metrics: metrics.New(1, "app"), - Alerts: alerts.New(alerts.Params{}), - }) - g.SetTransport(geojs) - - wantCountry(t, g, netip.MustParsePrefix("203.0.113.9/32"), "") - synctest.Wait() - - logged := log.String() - if !strings.Contains(logged, "asking GeoJS failed") || - strings.Contains(logged, "203.0.113.9") { - t.Errorf( - "logged %q, want the failure without the address asked about", - logged, - ) - } - }) -} - -func TestFailureRaisesASourceFailureAlertOncePerCooldown(t *testing.T) { - t.Parallel() - - synctest.Test(t, func(t *testing.T) { - geojs, clock, g, queue := startWithAlerts() - clients := newClients() - - geojs.set(failing) - - wantCountry(t, g, clients(), "") - - want := alerts.Alert{ - Time: clock.Now(), - Event: alerts.EventSourceFailure, - Reason: "asking GeoJS failed", - Detail: map[string]any{ - "source": "geojs", - "error": "GeoJS answered 503 Service Unavailable", - "asking_again_in": "1s", - }, - } - - // The next failure, a second later, is a repeat within the - // cooldown. - clock.advance(time.Second) - wantCountry(t, g, clients(), "") - wantRequests(t, geojs, 2) - - waiting := queue.Snapshot().Waiting[alerts.DestinationWebhook] - if len(waiting) != 1 || !reflect.DeepEqual(waiting[0], want) { - t.Errorf("alerts waiting %+v, want only %+v", waiting, want) - } - - if queue.Suppressed() != 1 { - t.Errorf( - "%d alerts held back, want the repeat", - queue.Suppressed(), - ) - } + reports.wantWithoutAnswer(t, 0) }) } @@ -466,24 +527,24 @@ func TestWaitingClientsAreAskedAboutInOneRequest(t *testing.T) { t.Parallel() synctest.Test(t, func(t *testing.T) { - geojs, clock, g := start() + service, clock, g := start() // GeoJS fails, and is then left alone for a second, while three more // clients come. An IPv6 client is a /64, and GeoJS is asked about its // first address. - geojs.set(failing) + service.set(failing) wantCountry(t, g, netip.MustParsePrefix("203.0.113.1/32"), "") wantCountry(t, g, netip.MustParsePrefix("203.0.113.2/32"), "") wantCountry(t, g, netip.MustParsePrefix("2001:db8:1:2::/64"), "") - wantRequests(t, geojs, 1) + wantRequests(t, service, 1) - geojs.set(answering) + service.set(answering) clock.advance(time.Second) wantCountry(t, g, netip.MustParsePrefix("203.0.113.3/32"), germany) - wantRequests(t, geojs, 2) + wantRequests(t, service, 2) wantAsked( t, - geojs, + service, 1, "203.0.113.1", "203.0.113.2", @@ -499,15 +560,15 @@ func TestKeptAnswersUnaffectedWhileGeoJSFailsAndAskedAgainWithBackoff( t.Parallel() synctest.Test(t, func(t *testing.T) { - geojs, clock, g := start() + service, clock, g := start() clients := newClients() kept := clients() wantCountry(t, g, kept, germany) - geojs.set(failing) + service.set(failing) wantCountry(t, g, kept, germany) - wantRequests(t, geojs, 1) + wantRequests(t, service, 1) // Each failure leaves GeoJS alone twice as long as the one before, up // to five minutes. New clients meanwhile count as not found, and the @@ -523,22 +584,22 @@ func TestKeptAnswersUnaffectedWhileGeoJSFailsAndAskedAgainWithBackoff( wantCountry(t, g, clients(), "") requests++ - wantRequests(t, geojs, requests) + wantRequests(t, service, requests) clock.advance(delay - time.Millisecond) wantCountry(t, g, clients(), "") wantCountry(t, g, kept, germany) - wantRequests(t, geojs, requests) + wantRequests(t, service, requests) clock.advance(time.Millisecond) } // Once GeoJS answers again, it is asked about every client waiting. - geojs.set(answering) + service.set(answering) wantCountry(t, g, clients(), germany) - wantRequests(t, geojs, requests+1) + wantRequests(t, service, requests+1) - asked := waitForRequests(t, geojs, requests+1) + asked := waitForRequests(t, service, requests+1) if len(asked[requests]) != 23 { t.Errorf( "GeoJS was asked about %d clients, want 23", @@ -552,24 +613,24 @@ func TestAtMost200AddressesInOneRequest(t *testing.T) { t.Parallel() synctest.Test(t, func(t *testing.T) { - geojs, clock, g := start() + service, clock, g := start() clients := newClients() first := clients() // 201 clients wait while GeoJS is left alone after a failure. - geojs.set(failing) + service.set(failing) wantCountry(t, g, first, "") for range 200 { wantCountry(t, g, clients(), "") } - // The first one's next request has GeoJS asked again. - geojs.set(answering) + // The first one's next lookup has GeoJS asked again. + service.set(answering) clock.advance(time.Second) wantCountry(t, g, first, "") - asked := waitForRequests(t, geojs, 3) + asked := waitForRequests(t, service, 3) if len(asked[1]) != 200 || len(asked[2]) != 1 { t.Errorf( "GeoJS was asked about %d and then %d clients, want 200 and 1", @@ -584,13 +645,13 @@ func TestAtMost10000ClientsWait(t *testing.T) { t.Parallel() synctest.Test(t, func(t *testing.T) { - geojs, clock, g := start() + service, clock, g := start() clients := newClients() first := clients() // 10,000 clients wait while GeoJS is left alone after a failure, and // one more cannot join them. - geojs.set(failing) + service.set(failing) wantCountry(t, g, first, "") for range 9999 { @@ -600,13 +661,13 @@ func TestAtMost10000ClientsWait(t *testing.T) { extra := clients() wantCountry(t, g, extra, "") - // The first one's next request has GeoJS asked about the 10,000, 200 + // The first one's next lookup has GeoJS asked about the 10,000, 200 // at a time, and not about the one more. - geojs.set(answering) + service.set(answering) clock.advance(time.Second) wantCountry(t, g, first, "") - asked := waitForRequests(t, geojs, 51) + asked := waitForRequests(t, service, 51) for i, request := range asked { if slices.Contains(request, extra.Addr().String()) { t.Errorf("request %d asked about %s", i, extra.Addr()) @@ -622,24 +683,15 @@ func TestClientsWithoutAnAnswerAreCounted(t *testing.T) { t.Parallel() synctest.Test(t, func(t *testing.T) { - m := metrics.New(1, "app") - g := lookup.New(lookup.Params{ - URL: lookup.URL, - Timeout: timeout, - Wait: true, - Now: time.Now, - ProcessLog: slog.New(slog.DiscardHandler), - Metrics: m, - Alerts: alerts.New(alerts.Params{}), - }) - g.SetTransport(&standIn{answers: failing}) - + service, _, g, reports := startWithReports() clients := newClients() + service.set(failing) + // GeoJS fails, so the first client goes without an answer, and GeoJS // is left alone for a second, which does not pass in this test. wantCountry(t, g, clients(), "") - wantRequestsWithoutGeoJSAnswer(t, m, 1) + reports.wantWithoutAnswer(t, 1) // Meanwhile each new client goes without one at once, while there is // room for it among the 10,000 that may wait. @@ -647,11 +699,11 @@ func TestClientsWithoutAnAnswerAreCounted(t *testing.T) { wantCountry(t, g, clients(), "") } - wantRequestsWithoutGeoJSAnswer(t, m, 10000) + reports.wantWithoutAnswer(t, 10000) // One more, for which there is no room, goes without one too. wantCountry(t, g, clients(), "") - wantRequestsWithoutGeoJSAnswer(t, m, 10001) + reports.wantWithoutAnswer(t, 10001) }) } @@ -659,6 +711,7 @@ func TestClientsWithoutAnAnswerAreCounted(t *testing.T) { const ( answering = iota answeringSlowly // most of a second later + answeringLate // lateAnswer later answeringInLowerCase // with each country in lower case answeringWithoutLeftOut // with a list that leaves leftOut out answeringEmptyList // with [] @@ -712,9 +765,9 @@ func (s *standIn) ServeHTTP(w http.ResponseWriter, r *http.Request) { http.Redirect(w, r, "/", http.StatusFound) return - case answeringSlowly: + case answeringSlowly, answeringLate: select { - case <-time.After(timeout * 4 / 5): + case <-time.After(delay(answers)): case <-r.Context().Done(): return } @@ -759,6 +812,16 @@ func (s *standIn) ServeHTTP(w http.ResponseWriter, r *http.Request) { } } +// delay returns how long the stand-in takes to answer when answering +// slowly or late. +func delay(answers int) time.Duration { + if answers == answeringLate { + return lateAnswer + } + + return wait * 4 / 5 +} + // set sets how the stand-in answers. func (s *standIn) set(answers int) { s.mu.Lock() @@ -775,6 +838,46 @@ func (s *standIn) asked() [][]string { return slices.Clone(s.requests) } +// reports are what a GeoJS reported through Failed and WentWithoutAnswer. +type reports struct { + mu sync.Mutex + failures []failure + withoutAnswer int +} + +// failure is one report of Failed: the error, and how long GeoJS is left +// alone. +type failure struct { + err string + askingAgainIn time.Duration +} + +// wantFailures checks the failures reported so far. +func (r *reports) wantFailures(t *testing.T, want ...failure) { + t.Helper() + + r.mu.Lock() + defer r.mu.Unlock() + + if !slices.Equal(r.failures, want) { + t.Errorf("failures %+v, want %+v", r.failures, want) + } +} + +// wantWithoutAnswer checks how many times a caller has gone without an +// answer. +func (r *reports) wantWithoutAnswer(t *testing.T, want int) { + t.Helper() + + r.mu.Lock() + defer r.mu.Unlock() + + if r.withoutAnswer != want { + t.Errorf("%d lookups went without an answer, want %d", + r.withoutAnswer, want) + } +} + // testClock is a clock the test sets. GeoJS tells the time by it, while // waits run on the bubble's clock. type testClock struct { @@ -799,38 +902,47 @@ func (c *testClock) advance(d time.Duration) { } // start returns a stand-in for GeoJS that answers, a clock, and a GeoJS -// asking the stand-in by that clock, for which a request waits for its -// client's first answer. -func start() (*standIn, *testClock, *lookup.GeoJS) { - geojs, clock, g, _ := startWithAlerts() +// asking the stand-in by that clock. +func start() (*standIn, *testClock, *geojs.GeoJS) { + service, clock, g, _ := startWithReports() - return geojs, clock, g + return service, clock, g } -// startWithAlerts is start, and returns the queue of the alerts GeoJS -// raises as well, for a webhook that is never sent them, with the default -// cooldown, by the same clock. -func startWithAlerts() (*standIn, *testClock, *lookup.GeoJS, *alerts.Queue) { - geojs := &standIn{} +// startWithReports is start, and returns what the GeoJS reports as well. +func startWithReports() (*standIn, *testClock, *geojs.GeoJS, *reports) { + service := &standIn{} clock := newClock() - queue := alerts.New(alerts.Params{ - WebhookURL: &url.URL{Scheme: "https", Host: "alerts.example"}, - Events: alerts.Events(), - Cooldown: 15 * time.Minute, - Now: clock.Now, - }) - g := lookup.New(lookup.Params{ - URL: lookup.URL, - Timeout: timeout, - Wait: true, - Now: clock.Now, - ProcessLog: slog.New(slog.DiscardHandler), - Metrics: metrics.New(1, "app"), - Alerts: queue, - }) - g.SetTransport(geojs) + g, reports := newGeoJS(service, geojs.Params{Now: clock.Now}) - return geojs, clock, g, queue + return service, clock, g, reports +} + +// newGeoJS returns a GeoJS that asks service, made with params and with +// what it reports noted in the reports returned. +func newGeoJS( + service *standIn, params geojs.Params, +) (*geojs.GeoJS, *reports) { + r := &reports{} + params.URL = geojs.URL + params.Asked = func() {} + params.Failed = func(err error, askingAgainIn time.Duration) { + r.mu.Lock() + defer r.mu.Unlock() + + r.failures = append(r.failures, failure{err.Error(), askingAgainIn}) + } + params.WentWithoutAnswer = func() { + r.mu.Lock() + defer r.mu.Unlock() + + r.withoutAnswer++ + } + + g := geojs.New(params) + g.SetTransport(service) + + return g, r } // newClock returns a clock set to the start of a day. @@ -850,36 +962,37 @@ func newClients() func() netip.Prefix { } } -// wantCountry checks the country g gives client. +// wantCountry checks the country g gives client, waiting up to wait for +// it. func wantCountry( t *testing.T, - g *lookup.GeoJS, + g *geojs.GeoJS, client netip.Prefix, want string, ) { t.Helper() - got := g.LookUp(t.Context(), client).Country + got := g.LookUp(t.Context(), client, wait).Country if got != want { t.Errorf("%s is in %q, want %q", client, got, want) } } // wantRequests checks how many requests GeoJS has had. -func wantRequests(t *testing.T, geojs *standIn, want int) { +func wantRequests(t *testing.T, service *standIn, want int) { t.Helper() - got := len(geojs.asked()) + got := len(service.asked()) if got != want { t.Errorf("GeoJS had %d requests, want %d", got, want) } } // wantAsked checks the addresses request i asked about, in any order. -func wantAsked(t *testing.T, geojs *standIn, i int, want ...string) { +func wantAsked(t *testing.T, service *standIn, i int, want ...string) { t.Helper() - asked := geojs.asked() + asked := service.asked() if len(asked) <= i { t.Fatalf("GeoJS had %d requests, want more than %d", len(asked), i) } @@ -893,40 +1006,15 @@ func wantAsked(t *testing.T, geojs *standIn, i int, want ...string) { } } -// wantRequestsWithoutGeoJSAnswer checks how many requests m counts as having -// gone without an answer from GeoJS. -func wantRequestsWithoutGeoJSAnswer( - t *testing.T, - m *metrics.Metrics, - want float64, -) { - t.Helper() - - got := testutil.ToFloat64(m.RequestsWithoutGeoJSAnswer) - if got != want { - t.Errorf("%v requests went without an answer, want %v", got, want) - } -} - -// wantFailures checks how many requests to GeoJS m counts as failed. -func wantFailures(t *testing.T, m *metrics.Metrics, want float64) { - t.Helper() - - got := testutil.ToFloat64(m.GeoJSFailures) - if got != want { - t.Errorf("%v requests to GeoJS failed, want %v", got, want) - } -} - // waitForRequests waits until g has done all it can before time passes, // checks that GeoJS has had count requests, and returns the addresses each // asked about. -func waitForRequests(t *testing.T, geojs *standIn, count int) [][]string { +func waitForRequests(t *testing.T, service *standIn, count int) [][]string { t.Helper() synctest.Wait() - asked := geojs.asked() + asked := service.asked() if len(asked) != count { t.Fatalf("GeoJS had %d requests, want %d", len(asked), count) } @@ -936,18 +1024,18 @@ func waitForRequests(t *testing.T, geojs *standIn, count int) [][]string { // waitForCountry lets a request to GeoJS under way be abandoned, and moves // the clock on a minute, so that GeoJS may be asked again after a failure. -// It then checks that client's next request does not wait but has it asked +// It then checks that client's next lookup does not wait but has it asked // about again in the background, after which g gives it the country want. func waitForCountry( t *testing.T, - g *lookup.GeoJS, + g *geojs.GeoJS, clock *testClock, client netip.Prefix, want string, ) { t.Helper() - time.Sleep(timeout) + time.Sleep(requestTimeout) clock.advance(time.Minute) wantCountry(t, g, client, "") synctest.Wait() diff --git a/internal/lookup/snapshot_test.go b/internal/geojs/snapshot_test.go similarity index 85% rename from internal/lookup/snapshot_test.go rename to internal/geojs/snapshot_test.go index af854a3..27c71bc 100644 --- a/internal/lookup/snapshot_test.go +++ b/internal/geojs/snapshot_test.go @@ -1,4 +1,4 @@ -package lookup_test +package geojs_test import ( "net/netip" @@ -7,7 +7,7 @@ import ( "testing/synctest" "time" - "sneak.berlin/go/smallwebwaf/internal/lookup" + "sneak.berlin/go/smallwebwaf/internal/geojs" ) func TestSnapshotHoldsEachAnswerAndWhenItWasLastUsed(t *testing.T) { @@ -25,7 +25,7 @@ func TestSnapshotHoldsEachAnswerAndWhenItWasLastUsed(t *testing.T) { clock.advance(time.Hour) wantCountry(t, g, placed, germany) - want := []lookup.Answer{ + want := []geojs.Answer{ {Client: notPlaced, Answered: asked, Used: asked}, { Client: placed, ASN: asn, ASName: asName, Country: germany, @@ -41,24 +41,24 @@ func TestSnapshotHoldsEachAnswerAndWhenItWasLastUsed(t *testing.T) { func TestLoadedAnswersAreKeptFor7DaysFromWhenGeoJSGaveThem(t *testing.T) { t.Parallel() - geojs, clock, g := start() + service, clock, g := start() now := clock.Now() - kept := lookup.Answer{ + kept := geojs.Answer{ Client: netip.MustParsePrefix("203.0.113.9/32"), Country: "FR", Answered: now.Add(-week + time.Second), Used: now.Add(-time.Hour), } - stale := lookup.Answer{ + stale := geojs.Answer{ Client: netip.MustParsePrefix("203.0.113.10/32"), Country: "FR", Answered: now.Add(-week), Used: now.Add(-time.Hour), } - g.Load([]lookup.Answer{kept, stale}) + g.Load([]geojs.Answer{kept, stale}) - if got := g.Snapshot(); !slices.Equal(got, []lookup.Answer{kept}) { + if got := g.Snapshot(); !slices.Equal(got, []geojs.Answer{kept}) { t.Errorf( "kept %+v, want only the answer GeoJS gave less than 7 days ago", got, @@ -66,7 +66,7 @@ func TestLoadedAnswersAreKeptFor7DaysFromWhenGeoJSGaveThem(t *testing.T) { } wantCountry(t, g, kept.Client, "FR") - wantRequests(t, geojs, 0) + wantRequests(t, service, 0) } func TestLoadDropsTheAnswerUsedLongestAgoFirst(t *testing.T) { @@ -80,11 +80,11 @@ func TestLoadDropsTheAnswerUsedLongestAgoFirst(t *testing.T) { // lookups.json lists the answers by client. Here each was last used a // second before the one listed before it, so the last listed is the // one used longest ago, and the one dropped. - answers := make([]lookup.Answer, maxAnswers+1) + answers := make([]geojs.Answer, maxAnswers+1) addr := netip.MustParseAddr("10.0.0.0") for i := range answers { - answers[i] = lookup.Answer{ + answers[i] = geojs.Answer{ Client: netip.PrefixFrom(addr, addr.BitLen()), Country: germany, Answered: now, diff --git a/internal/lookup/file.go b/internal/lookup/file.go index 74292d1..1c028f5 100644 --- a/internal/lookup/file.go +++ b/internal/lookup/file.go @@ -1,3 +1,6 @@ +// Package lookup looks up each client's AS number and country in the lookup +// database, the IPinfo Lite file SWW_WAF_LOOKUP_DB_PATH names. Its answers +// take the form of GeoJS's, so that waf handles both alike. package lookup import ( @@ -14,6 +17,7 @@ import ( "github.com/oschwald/maxminddb-golang/v2" "sneak.berlin/go/smallwebwaf/internal/alerts" + "sneak.berlin/go/smallwebwaf/internal/geojs" ) // quietTime is how long the lookup database must go without a change @@ -77,7 +81,7 @@ func OpenFile(params FileParams) (*File, error) { // each "" when the database does not give it, as for an address missing // from it. The database is asked about the client's first address, as // GeoJS is. -func (f *File) LookUp(client netip.Prefix) Answer { +func (f *File) LookUp(client netip.Prefix) geojs.Answer { f.mu.Lock() reader := f.reader f.mu.Unlock() @@ -91,7 +95,7 @@ func (f *File) LookUp(client netip.Prefix) Answer { found = record{} } - return Answer{ + return geojs.Answer{ Client: client, ASN: found.ASN, ASName: found.ASName, diff --git a/internal/lookup/file_internal_test.go b/internal/lookup/file_internal_test.go index c51c3bd..a49c68b 100644 --- a/internal/lookup/file_internal_test.go +++ b/internal/lookup/file_internal_test.go @@ -16,6 +16,7 @@ import ( "github.com/maxmind/mmdbwriter/mmdbtype" "sneak.berlin/go/smallwebwaf/internal/alerts" + "sneak.berlin/go/smallwebwaf/internal/geojs" "sneak.berlin/go/smallwebwaf/internal/lookup/lookuptest" ) @@ -68,7 +69,7 @@ func TestFilePlacesClientsAndCountsAnAddressMissingFromItAsUnknown( prefix := netip.MustParsePrefix(client) got := f.LookUp(prefix) - if got != (Answer{ + if got != (geojs.Answer{ Client: prefix, ASN: want.ASN, ASName: want.ASName, Country: want.Country, Answered: now, }) { diff --git a/internal/metrics/metrics.go b/internal/metrics/metrics.go index 24e92f3..2f787e6 100644 --- a/internal/metrics/metrics.go +++ b/internal/metrics/metrics.go @@ -274,8 +274,7 @@ func (m *Metrics) AddRemoteLog(remote *remotelog.Sender) { // AddLookupFile adds the metrics of the lookup database, read as the // metrics are asked for: when the file in use was read, which lastRead // returns, and the replacements of it that could not be read, which -// readFailures returns. The lookup package's File, which has both, cannot -// be named here: that package counts GeoJS's requests in these metrics. +// readFailures returns. func (m *Metrics) AddLookupFile( lastRead func() time.Time, readFailures func() int, diff --git a/internal/state/state.go b/internal/state/state.go index 9e9d4f4..9a04cc9 100644 --- a/internal/state/state.go +++ b/internal/state/state.go @@ -46,7 +46,7 @@ import ( "sneak.berlin/go/smallwebwaf/internal/alerts" "sneak.berlin/go/smallwebwaf/internal/anomaly" "sneak.berlin/go/smallwebwaf/internal/bans" - "sneak.berlin/go/smallwebwaf/internal/lookup" + "sneak.berlin/go/smallwebwaf/internal/geojs" "sneak.berlin/go/smallwebwaf/internal/metrics" "sneak.berlin/go/smallwebwaf/internal/ratelimit" "sneak.berlin/go/smallwebwaf/internal/reputation" @@ -135,7 +135,7 @@ type Params struct { // sww runs. Ledger *bans.Ledger Limiter *ratelimit.Limiter - GeoJS *lookup.GeoJS + GeoJS *geojs.GeoJS Lists *reputation.Lists DNSBL *reputation.DNSBL AbuseIPDB *reputation.AbuseIPDB @@ -195,8 +195,8 @@ type clientsFile struct { // lookupsFile is lookups.json, with each answer on a line of its own. type lookupsFile struct { - Version int `json:"version"` - Lookups []lookup.Answer `json:"lookups"` + Version int `json:"version"` + Lookups []geojs.Answer `json:"lookups"` } // abuseIPDBChecksSpentFile is abuseipdb-checks-spent.json, indented for an diff --git a/internal/state/state_test.go b/internal/state/state_test.go index 6aea5ff..2ee13d9 100644 --- a/internal/state/state_test.go +++ b/internal/state/state_test.go @@ -26,7 +26,7 @@ import ( "sneak.berlin/go/smallwebwaf/internal/alerts" "sneak.berlin/go/smallwebwaf/internal/anomaly" "sneak.berlin/go/smallwebwaf/internal/bans" - "sneak.berlin/go/smallwebwaf/internal/lookup" + "sneak.berlin/go/smallwebwaf/internal/geojs" "sneak.berlin/go/smallwebwaf/internal/metrics" "sneak.berlin/go/smallwebwaf/internal/ratelimit" "sneak.berlin/go/smallwebwaf/internal/reputation" @@ -1700,7 +1700,7 @@ func TestEditOfEachFileTakenIn(t *testing.T) { t, lookupsJSON, params.GeoJS.Snapshot(), - []lookup.Answer{ + []geojs.Answer{ {Client: client, Country: "FR", Answered: midnight()}, }, ) @@ -2415,9 +2415,7 @@ func newParams(dir string) state.Params { MaxBans: 5000, }), Limiter: ratelimit.New(ratelimit.Limits{}, 20000), - GeoJS: lookup.New(lookup.Params{ - Now: midnight, ProcessLog: discard, Metrics: m, - }), + GeoJS: geojs.New(geojs.Params{Now: midnight}), Lists: reputation.New(reputation.Params{ BlocklistURLs: []string{ blocklistURL, @@ -2494,7 +2492,7 @@ func fill(params state.Params) { }) params.Limiter.AddLookup(client, now.Add(-time.Hour), asn, asName, "DE") - params.GeoJS.Load([]lookup.Answer{ + params.GeoJS.Load([]geojs.Answer{ { Client: client, ASN: asn, ASName: asName, Country: "DE", Answered: now.Add(-time.Hour), Used: now, diff --git a/internal/sww/sww.go b/internal/sww/sww.go index 3da2b60..7befcde 100644 --- a/internal/sww/sww.go +++ b/internal/sww/sww.go @@ -25,6 +25,7 @@ import ( "sneak.berlin/go/smallwebwaf/internal/alerts" "sneak.berlin/go/smallwebwaf/internal/config" "sneak.berlin/go/smallwebwaf/internal/coreruleset" + "sneak.berlin/go/smallwebwaf/internal/geojs" "sneak.berlin/go/smallwebwaf/internal/lookup" "sneak.berlin/go/smallwebwaf/internal/remotelog" "sneak.berlin/go/smallwebwaf/internal/reputation" @@ -270,7 +271,7 @@ func newServer( Config: cfg, RequestLog: stdout, ProcessLog: processLog, - GeoJSURL: lookup.URL, + GeoJSURL: geojs.URL, AbuseIPDBURL: reputation.AbuseIPDBURL, LookupFile: lookupFile, CoreRuleSet: crs, diff --git a/internal/waf/answertime_test.go b/internal/waf/answertime_test.go index 408e9e6..3e184a0 100644 --- a/internal/waf/answertime_test.go +++ b/internal/waf/answertime_test.go @@ -20,7 +20,7 @@ import ( "github.com/prometheus/client_golang/prometheus/testutil" "sneak.berlin/go/smallwebwaf/internal/bans" - "sneak.berlin/go/smallwebwaf/internal/lookup" + "sneak.berlin/go/smallwebwaf/internal/geojs" "sneak.berlin/go/smallwebwaf/internal/lookup/lookuptest" "sneak.berlin/go/smallwebwaf/internal/ratelimit" "sneak.berlin/go/smallwebwaf/internal/reputation" @@ -591,7 +591,7 @@ func TestAClientGeoJSPlacedInNoCountryCountsAsUnknownWithoutWaiting( s := startWAF(t, geojsURL, map[string]string{ unknownCountryLimitPercent: "0", rateLimitPerMinute: "1", }) - s.server.GeoJS.Load([]lookup.Answer{{ + s.server.GeoJS.Load([]geojs.Answer{{ Client: netip.MustParsePrefix(client + "/32"), ASN: "AS64512", Answered: s.clock.Now(), Used: s.clock.Now(), }}) @@ -770,6 +770,8 @@ func TestARefusedRequestDoesNotWaitForASlowGeoJS(t *testing.T) { // TestRequestWaitsAsLongAsTheASNAndCountryLookupTimeoutSays says, // so that a request that does not wait takes no time at all. synctest.Test(t, func(t *testing.T) { + defer waitForGeoJSRequestsToBeAbandoned() + server, out := newWAFInBubble(t, tc.env) tc.prepare(t, server, time.Now()) @@ -838,6 +840,8 @@ func TestAQuestionWaitsForASlowGeoJSOnlyAsLongAsTheDefaultSays( t.Parallel() synctest.Test(t, func(t *testing.T) { + defer waitForGeoJSRequestsToBeAbandoned() + server, out := newWAFInBubble(t, tc.env) tc.prepare(t, server, time.Now()) @@ -950,10 +954,11 @@ func TestWAFAnswersWithinTheTimeLimitWhileEveryOutsideSourceHangs( t.Errorf("%d checks with AbuseIPDB, want 1", checked) } - // The query is answered, and the check with AbuseIPDB abandoned, - // before the bubble ends. + // The query is answered, and the check with AbuseIPDB and the request + // to GeoJS abandoned, before the bubble ends. releaseResolver() time.Sleep(defaultDNSBLAbuseIPDBWait) + waitForGeoJSRequestsToBeAbandoned() }) } diff --git a/internal/waf/biased_test.go b/internal/waf/biased_test.go index 17dadf7..0835a4f 100644 --- a/internal/waf/biased_test.go +++ b/internal/waf/biased_test.go @@ -339,6 +339,8 @@ func TestClientWithoutAnAnswerInTimeHasTheUnknownCountryLimitPercent( // TestRequestWaitsAsLongAsTheASNAndCountryLookupTimeoutSays says, with // a GeoJS that never answers. synctest.Test(t, func(t *testing.T) { + defer waitForGeoJSRequestsToBeAbandoned() + server, out, _ := newWAF(t, silentGeoJSURL, time.Now, map[string]string{unknownCountryLimitPercent: "0"}) @@ -390,6 +392,8 @@ func TestRequestWaitsForItsLookupWhileABiasedThresholdIsSet(t *testing.T) { // with a GeoJS that never answers. The headers, which need the // answer too, are off. synctest.Test(t, func(t *testing.T) { + defer waitForGeoJSRequestsToBeAbandoned() + server, out, _ := newWAF(t, silentGeoJSURL, time.Now, map[string]string{ asnAndCountryLookupTimeout: strconv.FormatInt( diff --git a/internal/waf/lookup.go b/internal/waf/lookup.go index 8627cd2..886201a 100644 --- a/internal/waf/lookup.go +++ b/internal/waf/lookup.go @@ -7,8 +7,9 @@ import ( "strings" "time" + "sneak.berlin/go/smallwebwaf/internal/alerts" "sneak.berlin/go/smallwebwaf/internal/config" - "sneak.berlin/go/smallwebwaf/internal/lookup" + "sneak.berlin/go/smallwebwaf/internal/geojs" ) // The headers in which an answer of allow gives rproxy the client's AS number @@ -26,9 +27,9 @@ const ( // SWW_WAF_LOOKUP_SOURCE is off or the client is on a private, loopback or // link-local address, which no lookup can place. The lookup database // answers at once. With GeoJS, while a setting needs the answer, such as a -// country list or a biased threshold, a new client's request waits for it, -// unless refusedWhateverTheAnswer reports true. ctx is the request's own -// context. +// country list or a biased threshold, a new client's request waits for it up +// to SWW_WAF_ASN_AND_COUNTRY_LOOKUP_TIMEOUT_MS, unless refusedWhateverTheAnswer +// reports true. ctx is the request's own context. func (rq *request) lookUp(ctx context.Context, now time.Time) { if rq.h.config.LookupSource == "off" || !canBePlaced(rq.client) { return @@ -42,8 +43,11 @@ func (rq *request) lookUp(ctx context.Context, now time.Time) { case rq.refusedWhateverTheAnswer(now): rq.lookupAnswer = rq.h.geojs.LookUpWithoutWaiting(ctx, client) rq.skippedLookupWait = rq.lookupAnswer.Answered.IsZero() + case !rq.h.waitForGeoJS: + rq.lookupAnswer = rq.h.geojs.LookUpWithoutWaiting(ctx, client) default: - rq.lookupAnswer = rq.h.geojs.LookUp(ctx, client) + rq.lookupAnswer = rq.h.geojs.LookUp(ctx, client, + rq.h.config.ASNAndCountryLookupTimeout) } rq.lookedUp = true @@ -83,13 +87,29 @@ func countryTag(country string) string { // addLookup adds answer, an answer about a client from the lookup // database or GeoJS, to the client's history, and to the notes of the bans // on its netblock that have no AS number, AS name or country yet. -func (h *handler) addLookup(answer lookup.Answer) { +func (h *handler) addLookup(answer geojs.Answer) { h.limiter.AddLookup(answer.Client, answer.Answered, answer.ASN, answer.ASName, answer.Country) h.ledger.AddLookup(h.netblock(answer.Client.Addr()), answer.ASN, answer.ASName, answer.Country) } +// geojsFailed logs a failure of GeoJS, raises a source_failure alert for it +// and counts it. askingAgainIn is how long GeoJS is left alone after it. +func (h *handler) geojsFailed(err error, askingAgainIn time.Duration) { + h.processLog.Warn("asking GeoJS failed", + "error", err.Error(), "asking_again_in", askingAgainIn.String()) + h.alerts.Raise(alerts.Alert{ + Event: alerts.EventSourceFailure, + Reason: "asking GeoJS failed", + Detail: map[string]any{ + "source": "geojs", "error": err.Error(), + "asking_again_in": askingAgainIn.String(), + }, + }) + h.metrics.GeoJSFailures.Inc() +} + // setLookupHeaders sets the headers that give the client's AS number and // country, leaving out one that is unknown. func setLookupHeaders(header http.Header, asn, country string) { diff --git a/internal/waf/lookup_test.go b/internal/waf/lookup_test.go index 407c87c..d6db8c0 100644 --- a/internal/waf/lookup_test.go +++ b/internal/waf/lookup_test.go @@ -12,7 +12,7 @@ import ( "time" "sneak.berlin/go/smallwebwaf/internal/alerts" - "sneak.berlin/go/smallwebwaf/internal/lookup" + "sneak.berlin/go/smallwebwaf/internal/geojs" "sneak.berlin/go/smallwebwaf/internal/lookup/lookuptest" "sneak.berlin/go/smallwebwaf/internal/requestlog" ) @@ -138,7 +138,7 @@ func TestRequestThatWaitedForItsLookupIsCountedWhenItGoesOn(t *testing.T) { // of lookups.json would keep it, while the first request still waits // for GeoJS. The second request goes on at once, and is counted. clk.advance(2500 * time.Millisecond) - server.GeoJS.Load([]lookup.Answer{{ + server.GeoJS.Load([]geojs.Answer{{ Client: netip.MustParsePrefix( fromDE + "/32", ), ASN: asnDE, ASName: asNameDE, @@ -199,7 +199,7 @@ func TestASNumberAndNameInTheLogLineTheHistoryTheBanNotesAndTheAlert( // The answer is kept before the requests, so GeoJS is not asked, and // gives no answer of its own. netblock := netip.MustParsePrefix(fromDE + "/32") - server.GeoJS.Load([]lookup.Answer{{ + server.GeoJS.Load([]geojs.Answer{{ Client: netblock, ASN: asnDE, ASName: asNameDE, Country: "DE", Answered: clk.Now(), Used: clk.Now(), }}) @@ -249,7 +249,7 @@ func TestLookupSourceOffLooksNoClientUp(t *testing.T) { ) // Even an answer kept from before is not used. - server.GeoJS.Load([]lookup.Answer{{ + server.GeoJS.Load([]geojs.Answer{{ Client: netip.MustParsePrefix( fromDE + "/32", ), ASN: asnDE, ASName: asNameDE, @@ -450,6 +450,8 @@ func TestRequestWaitsAsLongAsTheASNAndCountryLookupTimeoutSays( // network, which would keep that clock from moving on: the request is // handed to waf's handler, and GeoJS is one that never answers. synctest.Test(t, func(t *testing.T) { + defer waitForGeoJSRequestsToBeAbandoned() + // Not the default. The exclusive list needs the answer, and the app // is never reached. const timeout = 3 * time.Second @@ -495,6 +497,17 @@ func TestMain(m *testing.M) { m.Run() } +// geojsRequestTimeout is how long a request to GeoJS may take before it is +// abandoned. +const geojsRequestTimeout = 10 * time.Second + +// waitForGeoJSRequestsToBeAbandoned sleeps until every request to the GeoJS at +// silentGeoJSURL made so far has been abandoned. A synctest bubble that asks +// it defers this: the bubble cannot end while a request waits on it. +func waitForGeoJSRequestsToBeAbandoned() { + time.Sleep(geojsRequestTimeout) +} + // silentGeoJS is the GeoJS at silentGeoJSURL. type silentGeoJS struct{} diff --git a/internal/waf/metrics_test.go b/internal/waf/metrics_test.go index bf96455..dac7175 100644 --- a/internal/waf/metrics_test.go +++ b/internal/waf/metrics_test.go @@ -1,6 +1,7 @@ package waf_test import ( + "fmt" "io" "net/http" "net/http/httptest" @@ -10,8 +11,9 @@ import ( "testing" "time" + "sneak.berlin/go/smallwebwaf/internal/alerts" "sneak.berlin/go/smallwebwaf/internal/bans" - "sneak.berlin/go/smallwebwaf/internal/lookup" + "sneak.berlin/go/smallwebwaf/internal/geojs" "sneak.berlin/go/smallwebwaf/internal/requestlog" "sneak.berlin/go/smallwebwaf/internal/waf" ) @@ -248,7 +250,7 @@ func TestMetricsByCountryKeepTheBusiestAndCountTheRestAsOther(t *testing.T) { // The answers are kept before the requests, so that none waits for // GeoJS. - server.GeoJS.Load([]lookup.Answer{ + server.GeoJS.Load([]geojs.Answer{ keptAnswer(fromKP, "KP"), keptAnswer(fromDE, "DE"), keptAnswer(fromFR, "FR"), }) @@ -329,13 +331,13 @@ func TestMetricsByASNumberKeepTheBusiestAndCountTheRestAsOther( // The answers are kept before the requests, so that GeoJS gives none // of its own. Each client is in an AS of its own. - answer := func(addr, asn string) lookup.Answer { - return lookup.Answer{ + answer := func(addr, asn string) geojs.Answer { + return geojs.Answer{ Client: netip.MustParsePrefix(addr + "/32"), ASN: asn, Answered: clk.Now(), Used: clk.Now(), } } - server.GeoJS.Load([]lookup.Answer{ + server.GeoJS.Load([]geojs.Answer{ answer(fromDE, "AS64501"), answer(fromKP, "AS64502"), }) @@ -357,21 +359,23 @@ func TestMetricsByASNumberKeepTheBusiestAndCountTheRestAsOther( `sww_asn_requests_total{asn="AS64502",instance="app"}`) } -func TestMetricsCountGeoJSRequestsAndFailures(t *testing.T) { +func TestGeoJSFailureIsCountedLoggedAndAlerted(t *testing.T) { t.Parallel() - geojs := httptest.NewServer(http.HandlerFunc( + failingGeoJS := httptest.NewServer(http.HandlerFunc( func(w http.ResponseWriter, _ *http.Request) { w.WriteHeader(http.StatusServiceUnavailable) })) - t.Cleanup(geojs.Close) + t.Cleanup(failingGeoJS.Close) app := startApp(t, func(http.ResponseWriter, *http.Request) {}) - addr, _, server := startProxyWithClock(t, app.URL, geojs.URL, time.Now, + addr, out, server, queue := startProxyWithAlerts(t, app.URL, + failingGeoJS.URL, time.Now, map[string]string{ - trustedProxies: trustLocalhost, - metricsToken: token, - tagDeny: denyKP, + trustedProxies: trustLocalhost, + metricsToken: token, + tagDeny: denyKP, + alertWebhookURL: "https://alerts.example/waf", }) // GeoJS fails, so the client has no country= tag, and SWW_WAF_TAG_DENY does @@ -401,14 +405,46 @@ func TestMetricsCountGeoJSRequestsAndFailures(t *testing.T) { `sww_requests_without_geojs_answer_total{instance="app"}`, 1, ) + + // The process log's line names the failure but not the address asked + // about. + logged := "" + + for line := range strings.Lines(out.text()) { + if strings.Contains(line, "asking GeoJS failed") { + logged = line + } + } + + if logged == "" || strings.Contains(logged, fromDE) { + t.Errorf( + "logged %q, want the failure without the address asked about", + logged, + ) + } + + // fmt prints a map sorted by its keys. + const want = "map[asking_again_in:1s " + + "error:GeoJS answered 503 Service Unavailable source:geojs]" + + waiting := queue.Snapshot().Waiting[alerts.DestinationWebhook] + if len(waiting) != 1 || waiting[0].Event != alerts.EventSourceFailure || + waiting[0].Reason != "asking GeoJS failed" || + fmt.Sprint(waiting[0].Detail) != want { + t.Errorf( + "alerts waiting %+v, want one source_failure alert with %v", + waiting, + want, + ) + } } // keptAnswer returns GeoJS's answer that the client at addr is in // country, given now. -func keptAnswer(addr, country string) lookup.Answer { +func keptAnswer(addr, country string) geojs.Answer { now := time.Now() - return lookup.Answer{ + return geojs.Answer{ Client: netip.MustParsePrefix(addr + "/32"), Country: country, Answered: now, Used: now, } diff --git a/internal/waf/proxy.go b/internal/waf/proxy.go index c79153d..63bae67 100644 --- a/internal/waf/proxy.go +++ b/internal/waf/proxy.go @@ -17,6 +17,7 @@ import ( "sneak.berlin/go/smallwebwaf/internal/bans" "sneak.berlin/go/smallwebwaf/internal/config" "sneak.berlin/go/smallwebwaf/internal/coreruleset" + "sneak.berlin/go/smallwebwaf/internal/geojs" "sneak.berlin/go/smallwebwaf/internal/lookup" "sneak.berlin/go/smallwebwaf/internal/metrics" "sneak.berlin/go/smallwebwaf/internal/ratelimit" @@ -55,7 +56,7 @@ type Params struct { // ProcessLog receives the process's own messages. ProcessLog *slog.Logger // GeoJSURL is where clients' AS numbers and countries are looked up - // while SWW_WAF_LOOKUP_SOURCE is geojs, normally lookup.URL. + // while SWW_WAF_LOOKUP_SOURCE is geojs, normally geojs.URL. GeoJSURL string // AbuseIPDBURL is where clients are checked with AbuseIPDB while // SWW_WAF_ABUSEIPDB_KEY is set, normally reputation.AbuseIPDBURL. @@ -93,7 +94,7 @@ type Server struct { Ledger *bans.Ledger Limiter *ratelimit.Limiter - GeoJS *lookup.GeoJS + GeoJS *geojs.GeoJS Anomalies *anomaly.Counters LookupFile *lookup.File Lists *reputation.Lists @@ -164,20 +165,19 @@ func newHandler(params Params) *handler { }) } - h.geojs = lookup.New(lookup.Params{ - URL: params.GeoJSURL, - Timeout: params.Config.ASNAndCountryLookupTimeout, - // The country list, the headers, the tag settings that name an asn= - // or country= tag and the biased thresholds act on the answer before - // the request goes on. - Wait: len(params.Config.ExclusivelyAllowedCountries) > 0 || - params.Config.AddLookupHeaders || params.Config.TagsNeedLookup() || - biasedThresholdsSet(params.Config), - Answered: h.addLookup, - Now: params.Now, - ProcessLog: params.ProcessLog, - Metrics: m, - Alerts: params.Alerts, + // The country list, the headers, the tag settings that name an asn= or + // country= tag and the biased thresholds act on the answer before the + // request goes on. + h.waitForGeoJS = len(params.Config.ExclusivelyAllowedCountries) > 0 || + params.Config.AddLookupHeaders || params.Config.TagsNeedLookup() || + biasedThresholdsSet(params.Config) + h.geojs = geojs.New(geojs.Params{ + URL: params.GeoJSURL, + Answered: h.addLookup, + Now: params.Now, + Asked: m.GeoJSRequests.Inc, + Failed: h.geojsFailed, + WentWithoutAnswer: m.RequestsWithoutGeoJSAnswer.Inc, }) m.AddBansAndClients(h.ledger, h.limiter, params.Now) m.AddRules(params.Rules) @@ -268,7 +268,7 @@ type handler struct { metrics *metrics.Metrics limiter *ratelimit.Limiter ledger *bans.Ledger - geojs *lookup.GeoJS + geojs *geojs.GeoJS anomalies *anomaly.Counters lookupFile *lookup.File lists *reputation.Lists @@ -278,6 +278,9 @@ type handler struct { rules *rules.Files coreRuleSet *coreruleset.CoreRuleSet alerts *alerts.Queue + // waitForGeoJS is true when a setting needs each request's answer from + // GeoJS before the request goes on. + waitForGeoJS bool // awaitingReport holds the decision on each answered question by its id // until its report comes, and nil for a question still being handled. awaitingReport *lru.Cache[string, *decision] diff --git a/internal/waf/ratelimits_test.go b/internal/waf/ratelimits_test.go index 47aca6f..962668a 100644 --- a/internal/waf/ratelimits_test.go +++ b/internal/waf/ratelimits_test.go @@ -5,7 +5,7 @@ import ( "sync/atomic" "testing" - "sneak.berlin/go/smallwebwaf/internal/lookup" + "sneak.berlin/go/smallwebwaf/internal/geojs" "sneak.berlin/go/smallwebwaf/internal/ratelimit" "sneak.berlin/go/smallwebwaf/internal/requestlog" ) @@ -162,7 +162,7 @@ func TestRateLimitExemptPathsAreNeitherCountedNorRefused(t *testing.T) { // The answers are kept before the requests, so that none waits for // GeoJS. - server.GeoJS.Load([]lookup.Answer{ + server.GeoJS.Load([]geojs.Answer{ keptAnswer(client, "DE"), keptAnswer(fromKP, "KP"), }) diff --git a/internal/waf/request.go b/internal/waf/request.go index c945d32..2d28141 100644 --- a/internal/waf/request.go +++ b/internal/waf/request.go @@ -11,7 +11,7 @@ import ( "sneak.berlin/go/smallwebwaf/internal/anomaly" "sneak.berlin/go/smallwebwaf/internal/bans" "sneak.berlin/go/smallwebwaf/internal/config" - "sneak.berlin/go/smallwebwaf/internal/lookup" + "sneak.berlin/go/smallwebwaf/internal/geojs" "sneak.berlin/go/smallwebwaf/internal/ratelimit" "sneak.berlin/go/smallwebwaf/internal/requestlog" "sneak.berlin/go/smallwebwaf/internal/rules" @@ -42,7 +42,7 @@ type decision struct { // looked up, whether or not an answer was there, and lookupAnswer is // what the lookup gave then, the zero Answer while GeoJS had given none. lookedUp bool - lookupAnswer lookup.Answer + lookupAnswer geojs.Answer // skippedLookupWait is true for a request that did not wait for GeoJS and // found no answer kept. SWW_WAF_UNKNOWN_COUNTRY_LIMIT_PERCENT does not apply to it, // so that not waiting never lowers its client's limits. @@ -382,9 +382,9 @@ func anomalyThresholdsSet(cfg *config.WAF) bool { // answer about it as the request ends, and whether there is one: the // lookup database's, which was there at once, or the one GeoJS has given // by then, which a request does not wait for unless a setting needs it. -func (rq *request) answerAtTheEnd() (lookup.Answer, bool) { +func (rq *request) answerAtTheEnd() (geojs.Answer, bool) { if !rq.lookedUp { - return lookup.Answer{}, false + return geojs.Answer{}, false } if rq.h.config.LookupSource == "file" { -- 2.54.0