Author SHA1 Message Date
sneak fe392db4fc Add best-effort geolocation to the IP lookup from geojs and DB-IP City Lite (closes #239)
check / check (push) Successful in 4m19s
Each result gains `geo`. One request to geojs carries every address, 1 second in all. On a timeout, an error or an answer with no country, city or coordinates, the lookup reads DB-IP City Lite, downloaded monthly into the state directory. geojs answers stay in memory (250,000, least recently used dropped) and in a new `cache.db` table for 7 days.

Judgement call: a reserved address goes to neither source, and its `geo` is null.
Judgement call: `updated` for DB-IP is when the copy was downloaded.
Deviation: the cache is SQLite, by sneak's words.

Model: opus-5-5
2026-11-10 14:31:47 +00:00
2 changed files with 36 additions and 48 deletions

No files matched your search

+9 -11
View File
@@ -143,8 +143,9 @@ and origin AS can have two kept routes, one or none.
AS path, a next hop and a time. The next hop is the router the peer sends the AS path, a next hop and a time. The next hop is the router the peer sends the
prefix's traffic to. RouteWatch records the peer itself as the next hop. The prefix's traffic to. RouteWatch records the peer itself as the next hop. The
page also lists the prefix's origin changes, newest first. An origin change is page also lists the prefix's origin changes, newest first. An origin change is
recorded when an AS gains its first peer for the prefix, or loses its last. an AS starting to announce the prefix, when its first peer carries it, or
The page shows at most the newest 5,000. Each row shows: stopping, when its last peer no longer does. The page shows at most the newest
5,000. Each row shows:
- the time - the time
- announced or withdrawn - announced or withdrawn
- the origin AS - the origin AS
@@ -501,13 +502,11 @@ How RouteWatch keeps the answers:
there. there.
- It writes new answers every 5 seconds to `cache.db` in the state directory - It writes new answers every 5 seconds to `cache.db` in the state directory
(see Configuration), and loads them from there at startup. (see Configuration), and loads them from there at startup.
- Each DNS record carries a TTL, the time for which it may be kept. An answer - An answer expires after the TTL of its records, the shortest one when there
expires after the shortest TTL of its records. RouteWatch keeps it for at are several. RouteWatch keeps it for at least 60 seconds and at most 24 hours.
least 60 seconds and at most 24 hours. - When an address has no PTR names, that answer expires after the SOA record's
- When an address has no PTR names, the answer's SOA record says how long that minimum field, or the record's own TTL when that is shorter. RouteWatch keeps
answer may be kept. The answer expires after the SOA record's minimum field, it for at most 1 hour.
or the record's own TTL when that is shorter. RouteWatch keeps it for at most
1 hour.
- An answer with no PTR names and no SOA record is not kept. Neither is one - An answer with no PTR names and no SOA record is not kept. Neither is one
whose SOA record has a TTL or minimum field of 0. whose SOA record has a TTL or minimum field of 0.
- A failed lookup is not kept, so the next lookup of the address asks again. - A failed lookup is not kept, so the next lookup of the address asks again.
@@ -521,8 +520,7 @@ An IP lookup finds where each address is, on a best-effort basis. It uses two
sources: sources:
- [geojs](https://www.geojs.io/) is asked first. One request carries every - [geojs](https://www.geojs.io/) is asked first. One request carries every
address of the lookup that the cache does not hold, and it has 1 second to address of the lookup, and it has 1 second to answer.
answer.
- [DB-IP City Lite](https://db-ip.com/db/download/ip-to-city-lite) answers when - [DB-IP City Lite](https://db-ip.com/db/download/ip-to-city-lite) answers when
geojs has no answer for the address. It is a database file that RouteWatch geojs has no answer for the address. It is a database file that RouteWatch
reads itself. reads itself.
+27 -37
View File
@@ -22,46 +22,37 @@ formatting of the Go code and the markdown. Then it builds the image.
The image sets memory ceilings for a 5 GiB container (README "Memory"). README The image sets memory ceilings for a 5 GiB container (README "Memory"). README
"Running under upaas" says how to run it. "Running under upaas" says how to run it.
Since 2026-10-27, `next` has run on the live feed five times, from three hours A five-day run of `1c65656` on the live feed, from 2026-10-27 to 2026-11-01,
to five days. The runs found the handlers dropping updates for five causes, and found three problems:
the WHOIS fetcher asking a registry that had refused it. Each is fixed on
`next`, and its Completed Steps entry below names its issue. A 21-hour run of
`742e759`, which has every fix but one, dropped no update except after the feed
stalled. That cause, https://git.eeqj.de/sneak/routewatch/issues/229, is fixed
on `next`.
A 24-hour run of `e58c663` on the live feed began at 09:19 UTC on 2026-11-10 - The prefix handler dropped updates while a peer's session went down
(https://git.eeqj.de/sneak/routewatch/issues/232). It reads whether any handler (https://git.eeqj.de/sneak/routewatch/issues/216).
drops an update, the size of the `origin_changes` table on disk, and the - The AS handler dropped updates when the feed burst
process's peak memory against 5 GiB. The seven IP lookup additions of (https://git.eeqj.de/sneak/routewatch/issues/217).
https://git.eeqj.de/sneak/routewatch/issues/203 landed after it began, so it - The WHOIS fetcher kept asking a registry that had refused it
does not exercise them. (https://git.eeqj.de/sneak/routewatch/issues/218).
All three are fixed on `next`. So is
https://git.eeqj.de/sneak/routewatch/issues/222, which the third fix brought up.
A run of `26d939a` on the live feed showed the prefix handler still dropping
updates while the home page's cards were worked out.
https://git.eeqj.de/sneak/routewatch/issues/226 fixed that on `next`.
https://git.eeqj.de/sneak/routewatch/issues/225, also on `next`, takes the peers
whose sessions go down together out of the table in one walk. A 24-hour run of
`9bd0e91` on the live feed is checking whether any handler still drops updates.
The order of the project's remaining work is kept on The order of the project's remaining work is kept on
https://git.eeqj.de/sneak/routewatch/issues/86. https://git.eeqj.de/sneak/routewatch/issues/86.
# Next Step # Next Step
Post the outcome of the 24-hour run of `e58c663` on Post the outcome of the 24-hour run of `9bd0e91` on
https://git.eeqj.de/sneak/routewatch/issues/232. The run lasts 24 hours, or https://git.eeqj.de/sneak/routewatch/issues/226: whether any handler still drops
until the feed has stalled or reconnected once, up to 72 hours. The outcome updates, and the process's peak memory against 5 GiB. File an issue for each
says: problem the run finds.
- whether any handler dropped an update, and the prefix queue's high-water mark
- the held-back count
- the size of the `origin_changes` table on disk
- the process's peak memory against 5 GiB
File an issue for each problem the run finds.
# Completed Steps # Completed Steps
- 2026-11-10: `README.md` and this file are read whole against the rules of
https://git.eeqj.de/sneak/routewatch/issues/157, and every statement is
checked against the code on `next`. Four README sentences, on origin changes,
PTR lookups and geolocation, are rewritten. Status and Next Step describe the
live-feed run in progress (closes
https://git.eeqj.de/sneak/routewatch/issues/251).
- 2026-11-10: each result of the IP lookup has `geo`, where the address is. It - 2026-11-10: each result of the IP lookup has `geo`, where the address is. It
comes from geojs, with DB-IP's City Lite database as the fallback. The geojs comes from geojs, with DB-IP's City Lite database as the fallback. The geojs
answers stay in memory and in `cache.db` for 7 days (closes answers stay in memory and in `cache.db` for 7 days (closes
@@ -69,17 +60,16 @@ File an issue for each problem the run finds.
- 2026-11-10: the IP lookup checks the `icloud_private_relay` list of Apple's - 2026-11-10: the IP lookup checks the `icloud_private_relay` list of Apple's
iCloud Private Relay networks, downloaded once a day. Each result has iCloud Private Relay networks, downloaded once a day. Each result has
`flags.is_relay` (closes https://git.eeqj.de/sneak/routewatch/issues/237). `flags.is_relay` (closes https://git.eeqj.de/sneak/routewatch/issues/237).
- 2026-11-10: the IP lookup checks two ProtonVPN lists from a public mirror of - 2026-11-10: the IP lookup checks the `protonvpn` list of ProtonVPN's /24
Proton's server list. `protonvpn` holds the /24 networks of ProtonVPN's networks and the `protonvpn_entry` list of its servers' IPv4 and IPv6
servers, and `protonvpn_entry` holds the servers' IPv4 and IPv6 addresses addresses, from a public mirror of Proton's server list (closes
(closes https://git.eeqj.de/sneak/routewatch/issues/236). https://git.eeqj.de/sneak/routewatch/issues/236).
- 2026-11-10: the IP lookup finds PTR names over DNS-over-HTTPS and caches them - 2026-11-10: the IP lookup finds PTR names over DNS-over-HTTPS and caches them
for the record's TTL, in memory and in `cache.db` (closes for the record's TTL, in memory and in `cache.db` (closes
https://git.eeqj.de/sneak/routewatch/issues/238). https://git.eeqj.de/sneak/routewatch/issues/238).
- 2026-11-10: the tests no longer fail when a busy host stalls for several - 2026-11-10: the tests no longer fail when a busy host stalls for several
seconds. Most waits for something that should happen are now 15 seconds. Tests seconds. Most such waits are now 15 seconds. The test of the stream's idle
that depend on a short idle timeout or a request limit run on a fake clock timeout runs on a fake clock (closes #240)
(closes https://git.eeqj.de/sneak/routewatch/issues/240)
- 2026-11-10: the IP lookup checks the public VPN server lists of Mullvad, - 2026-11-10: the IP lookup checks the public VPN server lists of Mullvad,
NordVPN, Private Internet Access and IVPN, and X4BNet's community list. Each NordVPN, Private Internet Access and IVPN, and X4BNet's community list. Each
result has `flags.is_vpn` and `vpn_providers` (closes result has `flags.is_vpn` and `vpn_providers` (closes