GeoJS lookup as a package of its own: a visitor's wait no longer cuts off the request to GeoJS #404

Closed
opened 2026-11-16 23:54:40 +01:00 by clawbot · 1 comment
Collaborator

Split from #401. First of three. The second, on deciding in parallel, builds on this one.

What sneak asked for

Reworded from his comments on the closed pull request (#403 (comment), #403 (comment), #403 (comment), #403 (comment)):

  • The GeoJS query and its cache are one unit that handles everything below without waf seeing it.
  • The caller's deadline is separate from the unit's own time limit for a request to GeoJS. The caller's deadline is configurable.
  • A lookup is fire and forget: a caller that stops waiting never stops the request to GeoJS.
  • An answer that comes after the caller stopped waiting is still kept, for at least 24 hours. His example: GeoJS takes 5 seconds to answer. The visitor's request is decided within its deadline, without a country. A request 6 seconds later reads the answer from the cache.
  • Lookups under way are tracked by address. 10 requests at once from one uncached address cause one request to GeoJS, not 10.

Where the code is today

All in internal/lookup/lookup.go.

  • Already as asked:
    • one entry per client waiting for an answer (waiting);
    • one request to GeoJS at a time, for up to 200 addresses;
    • answers kept for 7 days, in memory and in lookups.json;
    • the request to GeoJS goes on when the visitor's request ends (context.WithoutCancel in ask).
  • Not as asked:
    • request cuts the request to GeoJS off at Params.Timeout. That is the same value a visitor's request waits, SWW_WAF_ASN_AND_COUNTRY_LOOKUP_TIMEOUT_MS. An answer later than that is thrown away and counted as a failure of GeoJS. GeoJS is then left alone for up to 5 minutes and a source_failure alert is raised.
    • The package imports internal/alerts and internal/metrics, and shares its directory with the lookup database (file.go).

To build

  • Move the GeoJS lookup and its cache into a package of its own, internal/geojs. It imports no other package of this repository. The lookup database stays in internal/lookup.
  • waf hands the new package what it needs to report a failure and to count (functions or a small interface). The alerts, the metrics and their names stay as they are.
  • The caller says on each lookup how long it waits. waf passes SWW_WAF_ASN_AND_COUNTRY_LOOKUP_TIMEOUT_MS. Its default stays 800 ms.
  • A request to GeoJS gets a time limit of its own inside the package: 10 seconds, fixed, with no new setting. Only passing that limit, or an error from GeoJS, is a failure of GeoJS.
  • Everything else behaves as now:
    • up to 200 addresses in one request, one request at a time;
    • answers kept 7 days, at most 100,000 of them, and at most 10,000 clients waiting;
    • GeoJS left alone after a failure, for a second and then twice as long each time, up to 5 minutes;
    • lookups.json read at start and written as now;
    • which requests wait at all, and that a client which already went a whole wait without an answer does not wait again.
  • Docs say that the setting bounds only the visitor's wait, and no longer the request to GeoJS:
    • SPEC.md: "Lookup of AS number and country", "Answering in time", and the package list under "Architecture";
    • docs/settings.md;
    • the settings table in README.md.

Done when

Each point has a test.

  • GeoJS answers after the caller's wait has passed. The caller got no answer within its wait. The answer is kept. The next lookup of that address returns it without a new request to GeoJS. No failure is counted, no alert is raised and GeoJS is not left alone.
  • sneak's example holds. GeoJS takes 5 seconds. A lookup that waits 800 ms returns without an answer after 800 ms. A lookup of the same address 6 seconds after the first returns the answer from the cache.
  • 10 lookups at once of one uncached address cause one request to GeoJS.
  • A caller whose context ends early does not stop the request to GeoJS.
  • A request to GeoJS that takes over 10 seconds is a failure, handled as a failure is now.
  • TestRequestWaitsAsLongAsTheTimeoutSaysAndGeoJSIsAbandonedAfterIt in internal/lookup/lookup_test.go asserts the opposite of the first point. It is changed to match.
  • internal/geojs imports no package of this repository.
  • make check passes.

Readings taken

Each is mine, not sneak's ruling, and each is cheap to reverse.

  • He wrote "its own Go module". This builds a package inside this repository that depends on nothing else in it, as #401 (comment) read it. Moving it to a repository of its own later moves the directory as it is. That step needs sneak to create the repository.
  • 10 seconds for a request to GeoJS. His example needs more than 5. Reversing it is one constant.
  • The cache stays at 7 days, which meets "at least 24 hours".
  • The default wait stays 800 ms. The closed pull request lowered it to 500 ms to make room for the Core Rule Set. Once the two waits overlap, that room is not needed.

Model: fable-5-2

Split from https://git.eeqj.de/sneak/smallwebwaf/issues/401. First of three. The second, on deciding in parallel, builds on this one. ## What sneak asked for Reworded from his comments on the closed pull request (https://git.eeqj.de/sneak/smallwebwaf/pulls/403#issuecomment-182171, https://git.eeqj.de/sneak/smallwebwaf/pulls/403#issuecomment-182183, https://git.eeqj.de/sneak/smallwebwaf/pulls/403#issuecomment-182187, https://git.eeqj.de/sneak/smallwebwaf/pulls/403#issuecomment-182190): - The GeoJS query and its cache are one unit that handles everything below without `waf` seeing it. - The caller's deadline is separate from the unit's own time limit for a request to GeoJS. The caller's deadline is configurable. - A lookup is fire and forget: a caller that stops waiting never stops the request to GeoJS. - An answer that comes after the caller stopped waiting is still kept, for at least 24 hours. His example: GeoJS takes 5 seconds to answer. The visitor's request is decided within its deadline, without a country. A request 6 seconds later reads the answer from the cache. - Lookups under way are tracked by address. 10 requests at once from one uncached address cause one request to GeoJS, not 10. ## Where the code is today All in `internal/lookup/lookup.go`. - Already as asked: - one entry per client waiting for an answer (`waiting`); - one request to GeoJS at a time, for up to 200 addresses; - answers kept for 7 days, in memory and in `lookups.json`; - the request to GeoJS goes on when the visitor's request ends (`context.WithoutCancel` in `ask`). - Not as asked: - `request` cuts the request to GeoJS off at `Params.Timeout`. That is the same value a visitor's request waits, `SWW_WAF_ASN_AND_COUNTRY_LOOKUP_TIMEOUT_MS`. An answer later than that is thrown away and counted as a failure of GeoJS. GeoJS is then left alone for up to 5 minutes and a `source_failure` alert is raised. - The package imports `internal/alerts` and `internal/metrics`, and shares its directory with the lookup database (`file.go`). ## To build - Move the GeoJS lookup and its cache into a package of its own, `internal/geojs`. It imports no other package of this repository. The lookup database stays in `internal/lookup`. - `waf` hands the new package what it needs to report a failure and to count (functions or a small interface). The alerts, the metrics and their names stay as they are. - The caller says on each lookup how long it waits. `waf` passes `SWW_WAF_ASN_AND_COUNTRY_LOOKUP_TIMEOUT_MS`. Its default stays 800 ms. - A request to GeoJS gets a time limit of its own inside the package: 10 seconds, fixed, with no new setting. Only passing that limit, or an error from GeoJS, is a failure of GeoJS. - Everything else behaves as now: - up to 200 addresses in one request, one request at a time; - answers kept 7 days, at most 100,000 of them, and at most 10,000 clients waiting; - GeoJS left alone after a failure, for a second and then twice as long each time, up to 5 minutes; - `lookups.json` read at start and written as now; - which requests wait at all, and that a client which already went a whole wait without an answer does not wait again. - Docs say that the setting bounds only the visitor's wait, and no longer the request to GeoJS: - `SPEC.md`: "Lookup of AS number and country", "Answering in time", and the package list under "Architecture"; - `docs/settings.md`; - the settings table in `README.md`. ## Done when Each point has a test. - GeoJS answers after the caller's wait has passed. The caller got no answer within its wait. The answer is kept. The next lookup of that address returns it without a new request to GeoJS. No failure is counted, no alert is raised and GeoJS is not left alone. - sneak's example holds. GeoJS takes 5 seconds. A lookup that waits 800 ms returns without an answer after 800 ms. A lookup of the same address 6 seconds after the first returns the answer from the cache. - 10 lookups at once of one uncached address cause one request to GeoJS. - A caller whose context ends early does not stop the request to GeoJS. - A request to GeoJS that takes over 10 seconds is a failure, handled as a failure is now. - `TestRequestWaitsAsLongAsTheTimeoutSaysAndGeoJSIsAbandonedAfterIt` in `internal/lookup/lookup_test.go` asserts the opposite of the first point. It is changed to match. - `internal/geojs` imports no package of this repository. - `make check` passes. ## Readings taken Each is mine, not sneak's ruling, and each is cheap to reverse. - He wrote "its own Go module". This builds a package inside this repository that depends on nothing else in it, as https://git.eeqj.de/sneak/smallwebwaf/issues/401#issuecomment-182199 read it. Moving it to a repository of its own later moves the directory as it is. That step needs sneak to create the repository. - 10 seconds for a request to GeoJS. His example needs more than 5. Reversing it is one constant. - The cache stays at 7 days, which meets "at least 24 hours". - The default wait stays 800 ms. The closed pull request lowered it to 500 ms to make room for the Core Rule Set. Once the two waits overlap, that room is not needed. Model: fable-5-2
Author
Collaborator

Built in #407, branch issue-404-geojs-package. The GeoJS lookup and its cache are now in internal/geojs. Each lookup takes the caller's wait. A request to GeoJS has its own 10-second limit, and an answer that comes after the wait is kept.

  • Judgement call: internal/lookup imports internal/geojs for geojs.Answer.
  • No test enforces the import rule for internal/geojs, since that would be a check of the tree rather than of behaviour.

Model: opus-5-5

Built in https://git.eeqj.de/sneak/smallwebwaf/pulls/407, branch `issue-404-geojs-package`. The GeoJS lookup and its cache are now in `internal/geojs`. Each lookup takes the caller's wait. A request to GeoJS has its own 10-second limit, and an answer that comes after the wait is kept. - Judgement call: `internal/lookup` imports `internal/geojs` for `geojs.Answer`. - No test enforces the import rule for `internal/geojs`, since that would be a check of the tree rather than of behaviour. Model: opus-5-5
Sign in to join this conversation.
1 Participants
Notifications
Due Date
No due date set.
Reference: sneak/smallwebwaf#404