Add best-effort geolocation to the IP lookup from geojs and DB-IP City Lite (closes #239)
check / check (push) Successful in 3m58s
check / check (push) Successful in 3m58s
Each result gains `geo`. One request to geojs carries every address, 1 second in all. On a timeout, an error or an answer with no country, city or coordinates, the lookup reads DB-IP City Lite, downloaded monthly into the state directory. geojs answers stay in memory (250,000, least recently used dropped) and in a new `cache.db` table for 7 days. Judgement call: a reserved address goes to neither source, and its `geo` is null. Judgement call: `updated` for DB-IP is when the copy was downloaded. Deviation: the cache is SQLite, by sneak's words. Model: opus-5-5
This commit is contained in:
sneak committed
1 parent
b2553f044d
commit
c8ae355787
16 files changed
+2372
-22
No files matched your search
@@ -63,8 +63,8 @@ docker run --memory=5g --memory-swap=5g -p 8080:8080 \
|
||||
-v routewatch:/var/lib/berlin.sneak.app.routewatch routewatch
|
||||
```
|
||||
|
||||
The memory limit is explained in Memory. The volume holds the database and the
|
||||
copies of the IP lists.
|
||||
The memory limit is explained in Memory. The volume holds the database, the
|
||||
copies of the IP lists and the geolocation files.
|
||||
|
||||
To run the binary on the host instead, build it with `make build`. The build
|
||||
needs Go and a C compiler, because the SQLite library is written in C:
|
||||
@@ -293,6 +293,7 @@ address. Every entry has the same fields:
|
||||
an empty list when it is on none
|
||||
- `ip_lists`: the public IP lists the address is on (see IP lists), or an empty
|
||||
list when it is on none
|
||||
- `geo`: where the address is (see Geolocation), or null when no source answers
|
||||
|
||||
Every other field is about the prefix in the routing table that holds the
|
||||
address. Each is null when no prefix in the routing table holds the address:
|
||||
@@ -509,6 +510,72 @@ How RouteWatch keeps the answers:
|
||||
- It keeps answers for at most 250,000 addresses, and keeps no answer for a new
|
||||
address while it holds that many.
|
||||
|
||||
## Geolocation
|
||||
|
||||
An IP lookup finds where each address is, on a best-effort basis. It uses two
|
||||
sources:
|
||||
|
||||
- [geojs](https://www.geojs.io/) is asked first. One request carries every
|
||||
address of the lookup, and it has 1 second to answer.
|
||||
- [DB-IP City Lite](https://db-ip.com/db/download/ip-to-city-lite) answers when
|
||||
geojs has no answer for the address. It is a database file that RouteWatch
|
||||
reads itself.
|
||||
|
||||
geojs has no answer when its request fails or runs out of time. An answer with
|
||||
no country, city or coordinates also counts as no answer, and RouteWatch does
|
||||
not cache it.
|
||||
|
||||
The `geo` object of a result has these fields:
|
||||
|
||||
- `country_code`: the two-letter country code
|
||||
- `country`: the country's name in English
|
||||
- `continent_code`: the two-letter continent code
|
||||
- `region`: the region or state
|
||||
- `city`: the city
|
||||
- `latitude` and `longitude`: the location in degrees
|
||||
- `accuracy_radius_km`: how far from that location the address may be, in km
|
||||
- `time_zone`: the time zone's name, such as `Europe/Berlin`
|
||||
- `source`: the source of the answer, `geojs` or `dbip`
|
||||
- `updated`: when the answer was fetched from geojs, or when the database copy
|
||||
was downloaded
|
||||
|
||||
A field the source does not know is null. DB-IP has no `accuracy_radius_km` and
|
||||
no `time_zone`, so both are null in its answers. `geo` itself is null when
|
||||
neither source answers. A reserved address, one whose `is_bogon` flag is true,
|
||||
goes to neither source, so its `geo` is null.
|
||||
|
||||
What goes outside: geojs receives each address that the cache does not hold. For
|
||||
`/ip/` and the home page, that address is the visitor's own. A lookup answered
|
||||
from the cache sends nothing. DB-IP receives only the monthly request for its
|
||||
file.
|
||||
|
||||
How RouteWatch keeps the answers of geojs:
|
||||
|
||||
- It holds at most 250,000 addresses in memory, and drops the least recently
|
||||
used first. That takes about 75 MB of the Go heap.
|
||||
- It writes new answers every 5 seconds to `cache.db` in the state directory.
|
||||
- It loads the newest 250,000 answers from `cache.db` at startup.
|
||||
- An address not in memory is read from `cache.db` before geojs is asked.
|
||||
- An answer expires 7 days after its fetch. Every 10 minutes RouteWatch deletes
|
||||
expired answers from memory and from `cache.db`.
|
||||
|
||||
Answers from the database are not cached, because reading the file is local.
|
||||
|
||||
How RouteWatch keeps the database:
|
||||
|
||||
- It keeps one copy in the state directory, named `dbip-city-lite.mmdb`.
|
||||
- It downloads the current month's file at the first start, and then once a
|
||||
month. Until the first download finishes, only geojs answers.
|
||||
- A download fails when it takes over 10 minutes or unpacks to over 200 MiB.
|
||||
- A new file replaces the old copy only once it opens. After a failed download
|
||||
RouteWatch keeps the old copy and tries again an hour later.
|
||||
- The file is about 130 MB. RouteWatch maps it into memory, outside the Go heap.
|
||||
The kernel loads the parts that are read, and can drop them again when memory
|
||||
is short.
|
||||
|
||||
The home page's footer says "IP Geolocation by DB-IP" with a link to
|
||||
[db-ip.com](https://db-ip.com).
|
||||
|
||||
## Configuration
|
||||
|
||||
RouteWatch reads its settings from environment variables. It takes no
|
||||
@@ -560,7 +627,9 @@ are not read.
|
||||
The state directory holds:
|
||||
|
||||
- `db.sqlite`: the database
|
||||
- `cache.db`: the answers of PTR lookups
|
||||
- `cache.db`: the answers of PTR lookups and of geojs
|
||||
- `dbip-city-lite.mmdb`: the copy of DB-IP's City Lite database (see
|
||||
Geolocation)
|
||||
- `iplists`: the IP list copies
|
||||
|
||||
It is at:
|
||||
@@ -585,8 +654,9 @@ a memory limit of 5 GiB and swap disabled:
|
||||
docker run --memory=5g --memory-swap=5g ...
|
||||
```
|
||||
|
||||
The 5 GiB leaves room above the ceilings for spikes and for the kernel's cache
|
||||
of the database file.
|
||||
The 5 GiB leaves room above the ceilings for spikes, for the kernel's cache of
|
||||
the database file, and for the mapped copy of DB-IP's database (see
|
||||
Geolocation).
|
||||
|
||||
The ceilings add up to 3.7 GiB:
|
||||
|
||||
@@ -722,6 +792,8 @@ The parts:
|
||||
address.
|
||||
- The PTR cache looks up the PTR names of addresses and keeps the answers (see
|
||||
PTR lookups).
|
||||
- The geolocation part finds where addresses are, keeps the answers of geojs and
|
||||
downloads DB-IP's database (see Geolocation).
|
||||
- The database maintainer checkpoints the database every 5 seconds: it folds the
|
||||
write-ahead log, where SQLite first writes changes, into the database file. It
|
||||
reclaims free space every 10 minutes. Every hour it refreshes the statistics
|
||||
@@ -831,7 +903,8 @@ On `SIGTERM` or `SIGINT` the daemon stops its parts in this order:
|
||||
4. the database maintainer
|
||||
5. the IP list downloads
|
||||
6. the HTTP server
|
||||
7. the PTR cache, which writes what it holds
|
||||
7. the geolocation part, which writes the geojs answers it holds
|
||||
8. the PTR cache, which writes what it holds
|
||||
|
||||
Last, the daemon checkpoints and closes the database. It gives all this 8
|
||||
seconds and then exits with status 0. A stop that is still running after 9
|
||||
@@ -854,6 +927,8 @@ What a stop can lose:
|
||||
SQLite rolls back any write the exit cut off.
|
||||
- When the 8 seconds run out, the PTR answers not yet in `cache.db` are lost.
|
||||
RouteWatch asks the resolvers for them again the next time they are needed.
|
||||
- When the 8 seconds run out, the geojs answers not yet in `cache.db` are lost.
|
||||
RouteWatch asks geojs for them again the next time they are needed.
|
||||
|
||||
### Code layout
|
||||
|
||||
@@ -870,6 +945,7 @@ routewatch/
|
||||
│ ├── cachedb/ # cache.db, the caches of outside answers
|
||||
│ ├── config/ # Settings from the environment
|
||||
│ ├── database/ # SQLite storage
|
||||
│ ├── geo/ # Geolocation: geojs, DB-IP City Lite, the cache
|
||||
│ ├── iplist/ # Public IP lists: downloads and lookups
|
||||
│ ├── logger/ # Structured logging
|
||||
│ ├── metrics/ # Feed and route update figures
|
||||
@@ -922,11 +998,16 @@ announcements(id PRIMARY KEY, prefix_id, peer_asn, origin_asn, path,
|
||||
|
||||
The indexes are in `internal/database/schema.sql`.
|
||||
|
||||
`cache.db` is a separate file, with one table:
|
||||
`cache.db` is a separate file, with two tables:
|
||||
|
||||
```sql
|
||||
-- The answers of PTR lookups. names is empty for an address with no PTR names.
|
||||
hostnames(address PRIMARY KEY, names, expires_at)
|
||||
|
||||
-- The answers of geojs. A part geojs does not know is NULL.
|
||||
geolocations(address PRIMARY KEY, country_code, country, continent_code,
|
||||
region, city, latitude, longitude, accuracy_radius_km, time_zone,
|
||||
fetched_at)
|
||||
```
|
||||
|
||||
## Development
|
||||
|
||||
Reference in new issue
Block a user