Compare commits
8
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
b1bd1d97cf | ||
|
|
760ba4746b | ||
|
|
dcb3935643 | ||
|
|
6de44ac054 | ||
|
|
cca402c73d | ||
|
|
2c3f6f71dc | ||
|
|
b2553f044d | ||
|
|
0cbd9cd943 |
No files matched your search
+1
-1
@@ -105,7 +105,7 @@ RUN apt-get update && apt-get install -y --no-install-recommends \
|
|||||||
ARG CACHEBUST
|
ARG CACHEBUST
|
||||||
|
|
||||||
# Create non-root user
|
# Create non-root user
|
||||||
RUN useradd -r -u 1000 -m routewatch
|
RUN useradd -u 1000 -m routewatch
|
||||||
|
|
||||||
RUN mkdir -p /var/lib/berlin.sneak.app.routewatch && chown routewatch:routewatch /var/lib/berlin.sneak.app.routewatch
|
RUN mkdir -p /var/lib/berlin.sneak.app.routewatch && chown routewatch:routewatch /var/lib/berlin.sneak.app.routewatch
|
||||||
|
|
||||||
|
|||||||
@@ -63,8 +63,8 @@ docker run --memory=5g --memory-swap=5g -p 8080:8080 \
|
|||||||
-v routewatch:/var/lib/berlin.sneak.app.routewatch routewatch
|
-v routewatch:/var/lib/berlin.sneak.app.routewatch routewatch
|
||||||
```
|
```
|
||||||
|
|
||||||
The memory limit is explained in Memory. The volume holds the database and the
|
The memory limit is explained in Memory. The volume holds the database, the
|
||||||
copies of the IP lists.
|
copies of the IP lists and the geolocation files.
|
||||||
|
|
||||||
To run the binary on the host instead, build it with `make build`. The build
|
To run the binary on the host instead, build it with `make build`. The build
|
||||||
needs Go and a C compiler, because the SQLite library is written in C:
|
needs Go and a C compiler, because the SQLite library is written in C:
|
||||||
@@ -143,9 +143,9 @@ and origin AS can have two kept routes, one or none.
|
|||||||
AS path, a next hop and a time. The next hop is the router the peer sends the
|
AS path, a next hop and a time. The next hop is the router the peer sends the
|
||||||
prefix's traffic to. RouteWatch records the peer itself as the next hop. The
|
prefix's traffic to. RouteWatch records the peer itself as the next hop. The
|
||||||
page also lists the prefix's origin changes, newest first. An origin change is
|
page also lists the prefix's origin changes, newest first. An origin change is
|
||||||
an AS starting to announce the prefix, when its first peer carries it, or
|
recorded when the first peer starts to carry the prefix with an origin AS, or
|
||||||
stopping, when its last peer no longer does. The page shows at most the newest
|
when the last peer stops. The page shows at most the newest 5,000. Each row
|
||||||
5,000. Each row shows:
|
shows:
|
||||||
- the time
|
- the time
|
||||||
- announced or withdrawn
|
- announced or withdrawn
|
||||||
- the origin AS
|
- the origin AS
|
||||||
@@ -197,8 +197,9 @@ also carries an `@meta` object:
|
|||||||
- `api_version`: `1`
|
- `api_version`: `1`
|
||||||
- `execution_time_ms`: the number of milliseconds the request took
|
- `execution_time_ms`: the number of milliseconds the request took
|
||||||
|
|
||||||
Every value with a unit is a number, and the unit ends the field name. Examples
|
Every value with a unit is a number. The unit ends the field name, as in `_ms`,
|
||||||
are `_ms`, `_seconds`, `_bytes`, `_per_sec` and `_per_min`.
|
`_seconds`, `_bytes`, `_per_sec` and `_per_min`. `latitude` and `longitude` are
|
||||||
|
in degrees, and their names do not say so.
|
||||||
|
|
||||||
The paths:
|
The paths:
|
||||||
|
|
||||||
@@ -287,12 +288,14 @@ address. Every entry has the same fields:
|
|||||||
- `is_tor`: true when the address is on the `tor_exits` list
|
- `is_tor`: true when the address is on the `tor_exits` list
|
||||||
- `is_vpn`: true when the address is on any list of VPN servers (see IP
|
- `is_vpn`: true when the address is on any list of VPN servers (see IP
|
||||||
lists)
|
lists)
|
||||||
|
- `is_relay`: true when the address is on the `icloud_private_relay` list
|
||||||
- `is_bogon`: true when the address is reserved, so that no network should
|
- `is_bogon`: true when the address is reserved, so that no network should
|
||||||
announce it on the internet
|
announce it on the internet
|
||||||
- `vpn_providers`: the names of the lists of VPN servers the address is on, or
|
- `vpn_providers`: the names of the lists of VPN servers the address is on, or
|
||||||
an empty list when it is on none
|
an empty list when it is on none
|
||||||
- `ip_lists`: the public IP lists the address is on (see IP lists), or an empty
|
- `ip_lists`: the public IP lists the address is on (see IP lists), or an empty
|
||||||
list when it is on none
|
list when it is on none
|
||||||
|
- `geo`: where the address is (see Geolocation), or null when no source answers
|
||||||
|
|
||||||
Every other field is about the prefix in the routing table that holds the
|
Every other field is about the prefix in the routing table that holds the
|
||||||
address. Each is null when no prefix in the routing table holds the address:
|
address. Each is null when no prefix in the routing table holds the address:
|
||||||
@@ -421,6 +424,9 @@ The lists:
|
|||||||
last 30 days. From
|
last 30 days. From
|
||||||
[FireHOL](https://iplists.firehol.org/?ipset=firehol_level3), downloaded every
|
[FireHOL](https://iplists.firehol.org/?ipset=firehol_level3), downloaded every
|
||||||
3 hours.
|
3 hours.
|
||||||
|
- `icloud_private_relay`: the networks where iCloud Private Relay traffic
|
||||||
|
leaves. From [Apple](https://mask-api.icloud.com/egress-ip-ranges.csv),
|
||||||
|
downloaded once a day.
|
||||||
- `ivpn`: VPN servers run by IVPN. From
|
- `ivpn`: VPN servers run by IVPN. From
|
||||||
[IVPN](https://api.ivpn.net/v5/servers.json), downloaded once a day.
|
[IVPN](https://api.ivpn.net/v5/servers.json), downloaded once a day.
|
||||||
- `mullvad`: VPN servers run by Mullvad. From
|
- `mullvad`: VPN servers run by Mullvad. From
|
||||||
@@ -431,6 +437,14 @@ The lists:
|
|||||||
- `pia`: VPN servers run by Private Internet Access. From
|
- `pia`: VPN servers run by Private Internet Access. From
|
||||||
[Private Internet Access](https://serverlist.piaservers.net/vpninfo/servers/v6),
|
[Private Internet Access](https://serverlist.piaservers.net/vpninfo/servers/v6),
|
||||||
downloaded once a day.
|
downloaded once a day.
|
||||||
|
- `protonvpn`: the /24 networks that hold ProtonVPN servers. The whole /24 is
|
||||||
|
listed because the ProtonVPN-IPs README says traffic leaves from unpublished
|
||||||
|
addresses in it. From the public
|
||||||
|
[ProtonVPN-IPs](https://github.com/tn3w/ProtonVPN-IPs) mirror of Proton's
|
||||||
|
server list, downloaded once a day.
|
||||||
|
- `protonvpn_entry`: the IPv4 and IPv6 addresses of ProtonVPN servers. From the
|
||||||
|
same [ProtonVPN-IPs](https://github.com/tn3w/ProtonVPN-IPs) mirror, downloaded
|
||||||
|
once a day.
|
||||||
- `spamhaus_drop` and `spamhaus_dropv6`: IPv4 and IPv6 networks hijacked or
|
- `spamhaus_drop` and `spamhaus_dropv6`: IPv4 and IPv6 networks hijacked or
|
||||||
leased by spammers or cyber criminals, according to Spamhaus. From
|
leased by spammers or cyber criminals, according to Spamhaus. From
|
||||||
[Spamhaus DROP](https://www.spamhaus.org/blocklists/do-not-route-or-peer/),
|
[Spamhaus DROP](https://www.spamhaus.org/blocklists/do-not-route-or-peer/),
|
||||||
@@ -489,11 +503,13 @@ How RouteWatch keeps the answers:
|
|||||||
there.
|
there.
|
||||||
- It writes new answers every 5 seconds to `cache.db` in the state directory
|
- It writes new answers every 5 seconds to `cache.db` in the state directory
|
||||||
(see Configuration), and loads them from there at startup.
|
(see Configuration), and loads them from there at startup.
|
||||||
- An answer expires after the TTL of its records, the shortest one when there
|
- Each DNS record carries a TTL, the time for which it may be kept. An answer
|
||||||
are several. RouteWatch keeps it for at least 60 seconds and at most 24 hours.
|
expires after the shortest TTL of its records. RouteWatch keeps it for at
|
||||||
- When an address has no PTR names, that answer expires after the SOA record's
|
least 60 seconds and at most 24 hours.
|
||||||
minimum field, or the record's own TTL when that is shorter. RouteWatch keeps
|
- When an address has no PTR names, the answer's SOA record says how long that
|
||||||
it for at most 1 hour.
|
answer may be kept. The answer expires after the SOA record's minimum field,
|
||||||
|
or the record's own TTL when that is shorter. RouteWatch keeps it for at most
|
||||||
|
1 hour.
|
||||||
- An answer with no PTR names and no SOA record is not kept. Neither is one
|
- An answer with no PTR names and no SOA record is not kept. Neither is one
|
||||||
whose SOA record has a TTL or minimum field of 0.
|
whose SOA record has a TTL or minimum field of 0.
|
||||||
- A failed lookup is not kept, so the next lookup of the address asks again.
|
- A failed lookup is not kept, so the next lookup of the address asks again.
|
||||||
@@ -501,6 +517,73 @@ How RouteWatch keeps the answers:
|
|||||||
- It keeps answers for at most 250,000 addresses, and keeps no answer for a new
|
- It keeps answers for at most 250,000 addresses, and keeps no answer for a new
|
||||||
address while it holds that many.
|
address while it holds that many.
|
||||||
|
|
||||||
|
## Geolocation
|
||||||
|
|
||||||
|
An IP lookup finds where each address is, on a best-effort basis. It uses two
|
||||||
|
sources:
|
||||||
|
|
||||||
|
- [geojs](https://www.geojs.io/) is asked first. One request carries every
|
||||||
|
address of the lookup that the cache does not hold. geojs has 1 second to
|
||||||
|
answer.
|
||||||
|
- [DB-IP City Lite](https://db-ip.com/db/download/ip-to-city-lite) answers when
|
||||||
|
geojs has no answer for the address. It is a database file that RouteWatch
|
||||||
|
reads itself.
|
||||||
|
|
||||||
|
geojs has no answer when its request fails or runs out of time. An answer with
|
||||||
|
no country, city or coordinates also counts as no answer, and RouteWatch does
|
||||||
|
not cache it.
|
||||||
|
|
||||||
|
The `geo` object of a result has these fields:
|
||||||
|
|
||||||
|
- `country_code`: the two-letter country code
|
||||||
|
- `country`: the country's name in English
|
||||||
|
- `continent_code`: the two-letter continent code
|
||||||
|
- `region`: the region or state
|
||||||
|
- `city`: the city
|
||||||
|
- `latitude` and `longitude`: the location in degrees
|
||||||
|
- `accuracy_radius_km`: how far from that location the address may be, in km
|
||||||
|
- `time_zone`: the time zone's name, such as `Europe/Berlin`
|
||||||
|
- `source`: the source of the answer, `geojs` or `dbip`
|
||||||
|
- `updated`: when the answer was fetched from geojs, or when the database copy
|
||||||
|
was downloaded
|
||||||
|
|
||||||
|
A field the source does not know is null. DB-IP has no `accuracy_radius_km` and
|
||||||
|
no `time_zone`, so both are null in its answers. `geo` itself is null when
|
||||||
|
neither source answers. A reserved address, one whose `is_bogon` flag is true,
|
||||||
|
goes to neither source, so its `geo` is null.
|
||||||
|
|
||||||
|
What goes outside: geojs receives each address that the cache does not hold. For
|
||||||
|
`/ip/` and the home page, that address is the visitor's own. A lookup answered
|
||||||
|
from the cache sends nothing. DB-IP receives only the monthly request for its
|
||||||
|
file.
|
||||||
|
|
||||||
|
How RouteWatch keeps the answers of geojs:
|
||||||
|
|
||||||
|
- It holds at most 250,000 addresses in memory, and drops the least recently
|
||||||
|
used first. That takes about 100 MB of the Go heap.
|
||||||
|
- It writes new answers every 5 seconds to `cache.db` in the state directory.
|
||||||
|
- It loads the newest 250,000 answers from `cache.db` at startup.
|
||||||
|
- An address not in memory is read from `cache.db` before geojs is asked.
|
||||||
|
- An answer expires 7 days after its fetch. Every 10 minutes RouteWatch deletes
|
||||||
|
expired answers from memory and from `cache.db`.
|
||||||
|
|
||||||
|
Answers from the database are not cached, because reading the file is local.
|
||||||
|
|
||||||
|
How RouteWatch keeps the database:
|
||||||
|
|
||||||
|
- It keeps one copy in the state directory, named `dbip-city-lite.mmdb`.
|
||||||
|
- It downloads the current month's file at the first start, and then once a
|
||||||
|
month. Until the first download finishes, only geojs answers.
|
||||||
|
- A download fails when it takes over 10 minutes or unpacks to over 200 MiB.
|
||||||
|
- A new file replaces the old copy only once it opens. After a failed download
|
||||||
|
RouteWatch keeps the old copy and tries again an hour later.
|
||||||
|
- The file is about 130 MB. RouteWatch maps it into memory, outside the Go heap.
|
||||||
|
The kernel loads the parts that are read, and can drop them again when memory
|
||||||
|
is short.
|
||||||
|
|
||||||
|
The home page's footer says "IP Geolocation by DB-IP" with a link to
|
||||||
|
[db-ip.com](https://db-ip.com).
|
||||||
|
|
||||||
## Configuration
|
## Configuration
|
||||||
|
|
||||||
RouteWatch reads its settings from environment variables. It takes no
|
RouteWatch reads its settings from environment variables. It takes no
|
||||||
@@ -552,7 +635,9 @@ are not read.
|
|||||||
The state directory holds:
|
The state directory holds:
|
||||||
|
|
||||||
- `db.sqlite`: the database
|
- `db.sqlite`: the database
|
||||||
- `cache.db`: the answers of PTR lookups
|
- `cache.db`: the answers of PTR lookups and of geojs
|
||||||
|
- `dbip-city-lite.mmdb`: the copy of DB-IP's City Lite database (see
|
||||||
|
Geolocation)
|
||||||
- `iplists`: the IP list copies
|
- `iplists`: the IP list copies
|
||||||
|
|
||||||
It is at:
|
It is at:
|
||||||
@@ -577,8 +662,9 @@ a memory limit of 5 GiB and swap disabled:
|
|||||||
docker run --memory=5g --memory-swap=5g ...
|
docker run --memory=5g --memory-swap=5g ...
|
||||||
```
|
```
|
||||||
|
|
||||||
The 5 GiB leaves room above the ceilings for spikes and for the kernel's cache
|
The 5 GiB leaves room above the ceilings for spikes, for the kernel's cache of
|
||||||
of the database file.
|
the database file, and for the mapped copy of DB-IP's database (see
|
||||||
|
Geolocation).
|
||||||
|
|
||||||
The ceilings add up to 3.7 GiB:
|
The ceilings add up to 3.7 GiB:
|
||||||
|
|
||||||
@@ -598,6 +684,13 @@ The ceilings add up to 3.7 GiB:
|
|||||||
limit caps the SQLite library.
|
limit caps the SQLite library.
|
||||||
- About 0.2 GiB for everything else in the runtime.
|
- About 0.2 GiB for everything else in the runtime.
|
||||||
|
|
||||||
|
The `icloud_private_relay` list has about 285,000 networks. Neighbouring
|
||||||
|
networks merge, which leaves about 6,900 address ranges. Loaded, the list adds
|
||||||
|
about 0.3 MiB to the Go heap and under 6 MiB to the process's resident memory.
|
||||||
|
Parsing the file allocates about 94 MiB in about 0.2 seconds. The next garbage
|
||||||
|
collection frees all of it but the 0.3 MiB above. Parsing happens at startup and
|
||||||
|
whenever a download brings a new version.
|
||||||
|
|
||||||
The SQLite library gets its memory from glibc, the C library. glibc keeps freed
|
The SQLite library gets its memory from glibc, the C library. glibc keeps freed
|
||||||
memory in pools called arenas, and hands it back to the kernel slowly. It
|
memory in pools called arenas, and hands it back to the kernel slowly. It
|
||||||
creates up to eight arenas per processor core. So on a host with many cores the
|
creates up to eight arenas per processor core. So on a host with many cores the
|
||||||
@@ -613,8 +706,10 @@ What happens at each limit:
|
|||||||
allocating more.
|
allocating more.
|
||||||
- The SQLite hard heap limit: a database statement fails with an out-of-memory
|
- The SQLite hard heap limit: a database statement fails with an out-of-memory
|
||||||
error, which the handler logs. The prefix handler writes the routing table's
|
error, which the handler logs. The prefix handler writes the routing table's
|
||||||
changes again in a later write. Everything else in the failed write is
|
changes and its origin changes again in a later write. Everything else in a
|
||||||
dropped. The process keeps running.
|
handler's failed write is dropped. The PTR cache and the geolocation part
|
||||||
|
write their answers to `cache.db` again in a later write. The process keeps
|
||||||
|
running.
|
||||||
- The handler queues: the handlers are the four parts that take the feed's
|
- The handler queues: the handlers are the four parts that take the feed's
|
||||||
messages (see Design). The prefix handler's queue holds at most 100,000
|
messages (see Design). The prefix handler's queue holds at most 100,000
|
||||||
messages, about 100 MiB. Each of the other three holds at most 20,000. When a
|
messages, about 100 MiB. Each of the other three holds at most 20,000. When a
|
||||||
@@ -714,6 +809,8 @@ The parts:
|
|||||||
address.
|
address.
|
||||||
- The PTR cache looks up the PTR names of addresses and keeps the answers (see
|
- The PTR cache looks up the PTR names of addresses and keeps the answers (see
|
||||||
PTR lookups).
|
PTR lookups).
|
||||||
|
- The geolocation part finds where addresses are, keeps the answers of geojs and
|
||||||
|
downloads DB-IP's database (see Geolocation).
|
||||||
- The database maintainer checkpoints the database every 5 seconds: it folds the
|
- The database maintainer checkpoints the database every 5 seconds: it folds the
|
||||||
write-ahead log, where SQLite first writes changes, into the database file. It
|
write-ahead log, where SQLite first writes changes, into the database file. It
|
||||||
reclaims free space every 10 minutes. Every hour it refreshes the statistics
|
reclaims free space every 10 minutes. Every hour it refreshes the statistics
|
||||||
@@ -823,7 +920,8 @@ On `SIGTERM` or `SIGINT` the daemon stops its parts in this order:
|
|||||||
4. the database maintainer
|
4. the database maintainer
|
||||||
5. the IP list downloads
|
5. the IP list downloads
|
||||||
6. the HTTP server
|
6. the HTTP server
|
||||||
7. the PTR cache, which writes what it holds
|
7. the geolocation part, which writes the geojs answers it holds
|
||||||
|
8. the PTR cache, which writes what it holds
|
||||||
|
|
||||||
Last, the daemon checkpoints and closes the database. It gives all this 8
|
Last, the daemon checkpoints and closes the database. It gives all this 8
|
||||||
seconds and then exits with status 0. A stop that is still running after 9
|
seconds and then exits with status 0. A stop that is still running after 9
|
||||||
@@ -846,6 +944,8 @@ What a stop can lose:
|
|||||||
SQLite rolls back any write the exit cut off.
|
SQLite rolls back any write the exit cut off.
|
||||||
- When the 8 seconds run out, the PTR answers not yet in `cache.db` are lost.
|
- When the 8 seconds run out, the PTR answers not yet in `cache.db` are lost.
|
||||||
RouteWatch asks the resolvers for them again the next time they are needed.
|
RouteWatch asks the resolvers for them again the next time they are needed.
|
||||||
|
- When the 8 seconds run out, the geojs answers not yet in `cache.db` are lost.
|
||||||
|
RouteWatch asks geojs for them again the next time they are needed.
|
||||||
|
|
||||||
### Code layout
|
### Code layout
|
||||||
|
|
||||||
@@ -862,6 +962,7 @@ routewatch/
|
|||||||
│ ├── cachedb/ # cache.db, the caches of outside answers
|
│ ├── cachedb/ # cache.db, the caches of outside answers
|
||||||
│ ├── config/ # Settings from the environment
|
│ ├── config/ # Settings from the environment
|
||||||
│ ├── database/ # SQLite storage
|
│ ├── database/ # SQLite storage
|
||||||
|
│ ├── geo/ # Geolocation: geojs, DB-IP City Lite, the cache
|
||||||
│ ├── iplist/ # Public IP lists: downloads and lookups
|
│ ├── iplist/ # Public IP lists: downloads and lookups
|
||||||
│ ├── logger/ # Structured logging
|
│ ├── logger/ # Structured logging
|
||||||
│ ├── metrics/ # Feed and route update figures
|
│ ├── metrics/ # Feed and route update figures
|
||||||
@@ -914,11 +1015,16 @@ announcements(id PRIMARY KEY, prefix_id, peer_asn, origin_asn, path,
|
|||||||
|
|
||||||
The indexes are in `internal/database/schema.sql`.
|
The indexes are in `internal/database/schema.sql`.
|
||||||
|
|
||||||
`cache.db` is a separate file, with one table:
|
`cache.db` is a separate file, with two tables:
|
||||||
|
|
||||||
```sql
|
```sql
|
||||||
-- The answers of PTR lookups. names is empty for an address with no PTR names.
|
-- The answers of PTR lookups. names is empty for an address with no PTR names.
|
||||||
hostnames(address PRIMARY KEY, names, expires_at)
|
hostnames(address PRIMARY KEY, names, expires_at)
|
||||||
|
|
||||||
|
-- The answers of geojs. A part geojs does not know is NULL.
|
||||||
|
geolocations(address PRIMARY KEY, country_code, country, continent_code,
|
||||||
|
region, city, latitude, longitude, accuracy_radius_km, time_zone,
|
||||||
|
fetched_at)
|
||||||
```
|
```
|
||||||
|
|
||||||
## Development
|
## Development
|
||||||
@@ -970,8 +1076,8 @@ calls one of them. The scripts:
|
|||||||
- `script/projectname`: print the project name, `routewatch`, which
|
- `script/projectname`: print the project name, `routewatch`, which
|
||||||
`script/docker` uses as the image tag.
|
`script/docker` uses as the image tag.
|
||||||
- `script/test`: build the `test` phase of the `Dockerfile` alone. It runs
|
- `script/test`: build the `test` phase of the `Dockerfile` alone. It runs
|
||||||
`go test -short -count=1 -timeout 90s -race -cover ./...`, and reruns it with
|
`go test -short -count=1 -timeout 90s -race -cover ./...`. On failure it runs
|
||||||
`-v` on failure.
|
the tests again with `-v` and without `-cover`.
|
||||||
- `script/lint`: build the `lint` phase of the `Dockerfile` alone. It runs
|
- `script/lint`: build the `lint` phase of the `Dockerfile` alone. It runs
|
||||||
`golangci-lint` v2.14.0 with `.golangci.yml`.
|
`golangci-lint` v2.14.0 with `.golangci.yml`.
|
||||||
- `script/fmt`: format the Go code with `go fmt` and the markdown with
|
- `script/fmt`: format the Go code with `go fmt` and the markdown with
|
||||||
|
|||||||
@@ -22,43 +22,76 @@ formatting of the Go code and the markdown. Then it builds the image.
|
|||||||
The image sets memory ceilings for a 5 GiB container (README "Memory"). README
|
The image sets memory ceilings for a 5 GiB container (README "Memory"). README
|
||||||
"Running under upaas" says how to run it.
|
"Running under upaas" says how to run it.
|
||||||
|
|
||||||
A five-day run of `1c65656` on the live feed, from 2026-10-27 to 2026-11-01,
|
Five runs of `next` on the live feed have ended since 2026-10-27, from three
|
||||||
found three problems:
|
hours to five days long. The runs found the handlers dropping updates for five
|
||||||
|
causes, and the WHOIS fetcher asking a registry that had refused it. Each is
|
||||||
|
fixed on `next`, and its Completed Steps entry below names its issue. A 21-hour
|
||||||
|
run of `742e759`, which has every fix but one, dropped no update except after
|
||||||
|
the feed stalled. That cause, https://git.eeqj.de/sneak/routewatch/issues/229,
|
||||||
|
is fixed on `next`.
|
||||||
|
|
||||||
- The prefix handler dropped updates while a peer's session went down
|
A run of `e58c663` on the live feed began at 09:19 UTC on 2026-11-10
|
||||||
(https://git.eeqj.de/sneak/routewatch/issues/216).
|
(https://git.eeqj.de/sneak/routewatch/issues/232). It is to confirm that fix.
|
||||||
- The AS handler dropped updates when the feed burst
|
Next Step lists what its outcome reports. The seven IP lookup additions of
|
||||||
(https://git.eeqj.de/sneak/routewatch/issues/217).
|
https://git.eeqj.de/sneak/routewatch/issues/203 landed after it began, so it
|
||||||
- The WHOIS fetcher kept asking a registry that had refused it
|
does not exercise them.
|
||||||
(https://git.eeqj.de/sneak/routewatch/issues/218).
|
|
||||||
|
|
||||||
All three are fixed on `next`. So is
|
|
||||||
https://git.eeqj.de/sneak/routewatch/issues/222, which the third fix brought up.
|
|
||||||
A run of `26d939a` on the live feed showed the prefix handler still dropping
|
|
||||||
updates while the home page's cards were worked out.
|
|
||||||
https://git.eeqj.de/sneak/routewatch/issues/226 fixed that on `next`.
|
|
||||||
https://git.eeqj.de/sneak/routewatch/issues/225, also on `next`, takes the peers
|
|
||||||
whose sessions go down together out of the table in one walk. A 24-hour run of
|
|
||||||
`9bd0e91` on the live feed is checking whether any handler still drops updates.
|
|
||||||
|
|
||||||
The order of the project's remaining work is kept on
|
The order of the project's remaining work is kept on
|
||||||
https://git.eeqj.de/sneak/routewatch/issues/86.
|
https://git.eeqj.de/sneak/routewatch/issues/86.
|
||||||
|
|
||||||
# Next Step
|
# Next Step
|
||||||
|
|
||||||
Post the outcome of the 24-hour run of `9bd0e91` on
|
Post the outcome of the run of `e58c663` on
|
||||||
https://git.eeqj.de/sneak/routewatch/issues/226: whether any handler still drops
|
https://git.eeqj.de/sneak/routewatch/issues/232. The run lasts at least 24
|
||||||
updates, and the process's peak memory against 5 GiB. File an issue for each
|
hours. If the feed has not stalled or reconnected by then, the run goes on until
|
||||||
problem the run finds.
|
it does, up to 72 hours in all. The outcome says:
|
||||||
|
|
||||||
|
- whether the feed stalled or reconnected
|
||||||
|
- whether any handler dropped an update, and the most messages the prefix
|
||||||
|
handler's queue held at once
|
||||||
|
- the most messages the prefix handler held back at once
|
||||||
|
- the size of the `origin_changes` table on disk
|
||||||
|
- the process's peak memory against 5 GiB
|
||||||
|
|
||||||
|
File an issue for each problem the run finds.
|
||||||
|
|
||||||
# Completed Steps
|
# Completed Steps
|
||||||
|
|
||||||
|
- 2026-11-11: two more tests no longer fail on a busy host. The test of the home
|
||||||
|
page's card refresh runs on a fake clock. The geo test in which geojs never
|
||||||
|
answers counts each request as it is sent, not when it reaches the server
|
||||||
|
(closes https://git.eeqj.de/sneak/routewatch/issues/256).
|
||||||
|
- 2026-11-10: `github.com/mattn/go-sqlite3` is at v1.14.53, so building on the
|
||||||
|
host no longer prints its two C compiler warnings (closes
|
||||||
|
https://git.eeqj.de/sneak/routewatch/issues/245).
|
||||||
|
- 2026-11-10: `README.md` and this file are read whole against the rules of
|
||||||
|
https://git.eeqj.de/sneak/routewatch/issues/157, and every statement except
|
||||||
|
measured figures and what outside services do is checked against the code on
|
||||||
|
`next`. Four README sentences failed the prose rules. Five disagreed with the
|
||||||
|
code. Each is rewritten. Status and Next Step describe the live-feed run in
|
||||||
|
progress (closes https://git.eeqj.de/sneak/routewatch/issues/251).
|
||||||
|
- 2026-11-10: the image creates the `routewatch` user as a regular user with UID
|
||||||
|
1000, so the build no longer warns about the system UID range (closes
|
||||||
|
https://git.eeqj.de/sneak/routewatch/issues/248).
|
||||||
|
- 2026-11-10: each result of the IP lookup has `geo`, where the address is. It
|
||||||
|
comes from geojs, with DB-IP's City Lite database as the fallback. The geojs
|
||||||
|
answers stay in memory and in `cache.db` for 7 days (closes
|
||||||
|
https://git.eeqj.de/sneak/routewatch/issues/239).
|
||||||
|
- 2026-11-10: the IP lookup checks the `icloud_private_relay` list of Apple's
|
||||||
|
iCloud Private Relay networks, downloaded once a day. Each result has
|
||||||
|
`flags.is_relay` (closes https://git.eeqj.de/sneak/routewatch/issues/237).
|
||||||
|
- 2026-11-10: the IP lookup checks two ProtonVPN lists from a public mirror of
|
||||||
|
Proton's server list. `protonvpn` holds the /24 networks of ProtonVPN's
|
||||||
|
servers, and `protonvpn_entry` holds the servers' IPv4 and IPv6 addresses
|
||||||
|
(closes https://git.eeqj.de/sneak/routewatch/issues/236).
|
||||||
- 2026-11-10: the IP lookup finds PTR names over DNS-over-HTTPS and caches them
|
- 2026-11-10: the IP lookup finds PTR names over DNS-over-HTTPS and caches them
|
||||||
for the record's TTL, in memory and in `cache.db` (closes
|
in memory and in `cache.db` for the record's TTL. A TTL is the time for which
|
||||||
|
a DNS record may be kept (closes
|
||||||
https://git.eeqj.de/sneak/routewatch/issues/238).
|
https://git.eeqj.de/sneak/routewatch/issues/238).
|
||||||
- 2026-11-10: the tests no longer fail when a busy host stalls for several
|
- 2026-11-10: the tests no longer fail when a busy host stalls for several
|
||||||
seconds. Most such waits are now 15 seconds. The test of the stream's idle
|
seconds. Most waits for something that should happen are now 15 seconds. Tests
|
||||||
timeout runs on a fake clock (closes #240)
|
that depend on a short idle timeout or a request limit run on a fake clock
|
||||||
|
(closes https://git.eeqj.de/sneak/routewatch/issues/240)
|
||||||
- 2026-11-10: the IP lookup checks the public VPN server lists of Mullvad,
|
- 2026-11-10: the IP lookup checks the public VPN server lists of Mullvad,
|
||||||
NordVPN, Private Internet Access and IVPN, and X4BNet's community list. Each
|
NordVPN, Private Internet Access and IVPN, and X4BNet's community list. Each
|
||||||
result has `flags.is_vpn` and `vpn_providers` (closes
|
result has `flags.is_vpn` and `vpn_providers` (closes
|
||||||
|
|||||||
@@ -8,8 +8,11 @@ require (
|
|||||||
github.com/go-chi/chi/v5 v5.2.2
|
github.com/go-chi/chi/v5 v5.2.2
|
||||||
github.com/google/btree v1.1.3
|
github.com/google/btree v1.1.3
|
||||||
github.com/google/uuid v1.6.0
|
github.com/google/uuid v1.6.0
|
||||||
github.com/mattn/go-sqlite3 v1.14.29
|
github.com/hashicorp/golang-lru/v2 v2.0.7
|
||||||
|
github.com/mattn/go-sqlite3 v1.14.53
|
||||||
|
github.com/maxmind/mmdbwriter v1.2.0
|
||||||
github.com/miekg/dns v1.1.73
|
github.com/miekg/dns v1.1.73
|
||||||
|
github.com/oschwald/maxminddb-golang/v2 v2.9.0
|
||||||
github.com/rcrowley/go-metrics v0.0.0-20250401214520-65e299d6c5c9
|
github.com/rcrowley/go-metrics v0.0.0-20250401214520-65e299d6c5c9
|
||||||
go.uber.org/fx v1.24.0
|
go.uber.org/fx v1.24.0
|
||||||
go4.org/netipx v0.0.0-20260823151212-3075585bcbeb
|
go4.org/netipx v0.0.0-20260823151212-3075585bcbeb
|
||||||
@@ -22,5 +25,5 @@ require (
|
|||||||
go.uber.org/multierr v1.10.0 // indirect
|
go.uber.org/multierr v1.10.0 // indirect
|
||||||
go.uber.org/zap v1.26.0 // indirect
|
go.uber.org/zap v1.26.0 // indirect
|
||||||
golang.org/x/net v0.57.0 // indirect
|
golang.org/x/net v0.57.0 // indirect
|
||||||
golang.org/x/sys v0.47.0 // indirect
|
golang.org/x/sys v0.49.0 // indirect
|
||||||
)
|
)
|
||||||
@@ -1,7 +1,5 @@
|
|||||||
github.com/bits-and-blooms/bitset v1.26.0 h1:WPvcPZbP1yuKofmjs2SYIFO1EjzIGhCP9ducElLsh5Q=
|
github.com/bits-and-blooms/bitset v1.26.0 h1:WPvcPZbP1yuKofmjs2SYIFO1EjzIGhCP9ducElLsh5Q=
|
||||||
github.com/bits-and-blooms/bitset v1.26.0/go.mod h1:7hO7Gc7Pp1vODcmWvKMRA9BNmbv6a/7QIWpPxHddWR8=
|
github.com/bits-and-blooms/bitset v1.26.0/go.mod h1:7hO7Gc7Pp1vODcmWvKMRA9BNmbv6a/7QIWpPxHddWR8=
|
||||||
github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c=
|
|
||||||
github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
|
|
||||||
github.com/dustin/go-humanize v1.0.1 h1:GzkhY7T5VNhEkwH0PVJgjz+fX1rhBrR7pRT3mDkpeCY=
|
github.com/dustin/go-humanize v1.0.1 h1:GzkhY7T5VNhEkwH0PVJgjz+fX1rhBrR7pRT3mDkpeCY=
|
||||||
github.com/dustin/go-humanize v1.0.1/go.mod h1:Mu1zIs6XwVuF/gI1OepvI0qD18qycQx+mFykh5fBlto=
|
github.com/dustin/go-humanize v1.0.1/go.mod h1:Mu1zIs6XwVuF/gI1OepvI0qD18qycQx+mFykh5fBlto=
|
||||||
github.com/go-chi/chi/v5 v5.2.2 h1:CMwsvRVTbXVytCk1Wd72Zy1LAsAh9GxMmSNWLHCG618=
|
github.com/go-chi/chi/v5 v5.2.2 h1:CMwsvRVTbXVytCk1Wd72Zy1LAsAh9GxMmSNWLHCG618=
|
||||||
@@ -10,16 +8,20 @@ github.com/google/btree v1.1.3 h1:CVpQJjYgC4VbzxeGVHfvZrv1ctoYCAI8vbl07Fcxlyg=
|
|||||||
github.com/google/btree v1.1.3/go.mod h1:qOPhT0dTNdNzV6Z/lhRX0YXUafgPLFUh+gZMl761Gm4=
|
github.com/google/btree v1.1.3/go.mod h1:qOPhT0dTNdNzV6Z/lhRX0YXUafgPLFUh+gZMl761Gm4=
|
||||||
github.com/google/uuid v1.6.0 h1:NIvaJDMOsjHA8n1jAhLSgzrAzy1Hgr+hNrb57e+94F0=
|
github.com/google/uuid v1.6.0 h1:NIvaJDMOsjHA8n1jAhLSgzrAzy1Hgr+hNrb57e+94F0=
|
||||||
github.com/google/uuid v1.6.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo=
|
github.com/google/uuid v1.6.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo=
|
||||||
github.com/mattn/go-sqlite3 v1.14.29 h1:1O6nRLJKvsi1H2Sj0Hzdfojwt8GiGKm+LOfLaBFaouQ=
|
github.com/hashicorp/golang-lru/v2 v2.0.7 h1:a+bsQ5rvGLjzHuww6tVxozPZFVghXaHOwFs4luLUK2k=
|
||||||
github.com/mattn/go-sqlite3 v1.14.29/go.mod h1:Uh1q+B4BYcTPb+yiD3kU8Ct7aC0hY9fxUwlHK0RXw+Y=
|
github.com/hashicorp/golang-lru/v2 v2.0.7/go.mod h1:QeFd9opnmA6QUJc5vARoKUSoFhyfM2/ZepoAG6RGpeM=
|
||||||
|
github.com/mattn/go-sqlite3 v1.14.53 h1:mnTM0owzUMrKuLo+0w51NUiB0ir2Fo7kFXlPt/l3BGs=
|
||||||
|
github.com/mattn/go-sqlite3 v1.14.53/go.mod h1:6JTjA44L93a0QCyJef5YvlPoKXntQPjzWv5gtm9sB6w=
|
||||||
|
github.com/maxmind/mmdbwriter v1.2.0 h1:hyvDopImmgvle3aR8AaddxXnT0iQH2KWJX3vNfkwzYM=
|
||||||
|
github.com/maxmind/mmdbwriter v1.2.0/go.mod h1:EQmKHhk2y9DRVvyNxwCLKC5FrkXZLx4snc5OlLY5XLE=
|
||||||
github.com/miekg/dns v1.1.73 h1:uhT8nJxmTrPJYClxVxTCX+CVn6qnzSiybRk72Z6DgrE=
|
github.com/miekg/dns v1.1.73 h1:uhT8nJxmTrPJYClxVxTCX+CVn6qnzSiybRk72Z6DgrE=
|
||||||
github.com/miekg/dns v1.1.73/go.mod h1:RW2Obtfd5NZHvOFe3zYG0W8koWOQtAzyHaLo8vASBuQ=
|
github.com/miekg/dns v1.1.73/go.mod h1:RW2Obtfd5NZHvOFe3zYG0W8koWOQtAzyHaLo8vASBuQ=
|
||||||
github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM=
|
github.com/oschwald/maxminddb-golang/v2 v2.9.0 h1:YpIl3WoA9oIuNvqNaO/aD1N/7Ci2wOqoKH5GWo53H5Y=
|
||||||
github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
|
github.com/oschwald/maxminddb-golang/v2 v2.9.0/go.mod h1:3TxexMBcR0ss3SbyYvnqfsDzDjN4o1KCg/nTsaMAivU=
|
||||||
github.com/rcrowley/go-metrics v0.0.0-20250401214520-65e299d6c5c9 h1:bsUq1dX0N8AOIL7EB/X911+m4EHsnWEHeJ0c+3TTBrg=
|
github.com/rcrowley/go-metrics v0.0.0-20250401214520-65e299d6c5c9 h1:bsUq1dX0N8AOIL7EB/X911+m4EHsnWEHeJ0c+3TTBrg=
|
||||||
github.com/rcrowley/go-metrics v0.0.0-20250401214520-65e299d6c5c9/go.mod h1:bCqnVzQkZxMG4s8nGwiZ5l3QUCyqpo9Y+/ZMZ9VjZe4=
|
github.com/rcrowley/go-metrics v0.0.0-20250401214520-65e299d6c5c9/go.mod h1:bCqnVzQkZxMG4s8nGwiZ5l3QUCyqpo9Y+/ZMZ9VjZe4=
|
||||||
github.com/stretchr/testify v1.8.1 h1:w7B6lhMri9wdJUVmEZPGGhZzrYTPvgJArz7wNPgYKsk=
|
github.com/stretchr/testify v1.12.1 h1:EuwCh5fleGS7H32xRwO3wRGT7DxrDhLAT6FF8MpWDWE=
|
||||||
github.com/stretchr/testify v1.8.1/go.mod h1:w2LPCIKwWwSfY2zedu0+kehJoqGctiVI29o6fzry7u4=
|
github.com/stretchr/testify v1.12.1/go.mod h1:MDEgiDPPsNp5cuIrHPPCyornHKgEVbtFUmoNlxoYthg=
|
||||||
go.uber.org/dig v1.19.0 h1:BACLhebsYdpQ7IROQ1AGPjrXcP5dF80U3gKoFzbaq/4=
|
go.uber.org/dig v1.19.0 h1:BACLhebsYdpQ7IROQ1AGPjrXcP5dF80U3gKoFzbaq/4=
|
||||||
go.uber.org/dig v1.19.0/go.mod h1:Us0rSJiThwCv2GteUN0Q7OKvU7n5J4dxZ9JKUXozFdE=
|
go.uber.org/dig v1.19.0/go.mod h1:Us0rSJiThwCv2GteUN0Q7OKvU7n5J4dxZ9JKUXozFdE=
|
||||||
go.uber.org/fx v1.24.0 h1:wE8mruvpg2kiiL1Vqd0CC+tr0/24XIB10Iwp2lLWzkg=
|
go.uber.org/fx v1.24.0 h1:wE8mruvpg2kiiL1Vqd0CC+tr0/24XIB10Iwp2lLWzkg=
|
||||||
@@ -30,14 +32,16 @@ go.uber.org/multierr v1.10.0 h1:S0h4aNzvfcFsC3dRF1jLoaov7oRaKqRGC/pUEJ2yvPQ=
|
|||||||
go.uber.org/multierr v1.10.0/go.mod h1:20+QtiLqy0Nd6FdQB9TLXag12DsQkrbs3htMFfDN80Y=
|
go.uber.org/multierr v1.10.0/go.mod h1:20+QtiLqy0Nd6FdQB9TLXag12DsQkrbs3htMFfDN80Y=
|
||||||
go.uber.org/zap v1.26.0 h1:sI7k6L95XOKS281NhVKOFCUNIvv9e0w4BF8N3u+tCRo=
|
go.uber.org/zap v1.26.0 h1:sI7k6L95XOKS281NhVKOFCUNIvv9e0w4BF8N3u+tCRo=
|
||||||
go.uber.org/zap v1.26.0/go.mod h1:dtElttAiwGvoJ/vj4IwHBS/gXsEu/pZ50mUIRWuG0so=
|
go.uber.org/zap v1.26.0/go.mod h1:dtElttAiwGvoJ/vj4IwHBS/gXsEu/pZ50mUIRWuG0so=
|
||||||
|
go.yaml.in/yaml/v3 v3.0.5 h1:N6y/pJk8buWs9NY5ERU2HSMfm+IuD/OtfdAnq6kESPw=
|
||||||
|
go.yaml.in/yaml/v3 v3.0.5/go.mod h1:HVTZu1O7/Vkt2N+BFy8Zza+lnLsABggaTM2ZpNIGuKg=
|
||||||
go4.org/netipx v0.0.0-20260823151212-3075585bcbeb h1:XBM4hvfwGAttkkiTIFfeigdfcL1xIfdKXqFdgiHGtDs=
|
go4.org/netipx v0.0.0-20260823151212-3075585bcbeb h1:XBM4hvfwGAttkkiTIFfeigdfcL1xIfdKXqFdgiHGtDs=
|
||||||
go4.org/netipx v0.0.0-20260823151212-3075585bcbeb/go.mod h1:PLyyIXexvUFg3Owu6p/WfdlivPbZJsZdgWZlrGope/Y=
|
go4.org/netipx v0.0.0-20260823151212-3075585bcbeb/go.mod h1:PLyyIXexvUFg3Owu6p/WfdlivPbZJsZdgWZlrGope/Y=
|
||||||
golang.org/x/net v0.57.0 h1:K5+3DljvIuDG9/Jv9rvyMywYNFCQ9RSUY6OOTTkT+tE=
|
golang.org/x/net v0.57.0 h1:K5+3DljvIuDG9/Jv9rvyMywYNFCQ9RSUY6OOTTkT+tE=
|
||||||
golang.org/x/net v0.57.0/go.mod h1:KpXc8iv+r3XplLAG/f7Jsf9RPszJzdR0f58q9vGOuEU=
|
golang.org/x/net v0.57.0/go.mod h1:KpXc8iv+r3XplLAG/f7Jsf9RPszJzdR0f58q9vGOuEU=
|
||||||
golang.org/x/sync v0.24.0 h1:iqiDrGoDgyuvAKrhLQthUEzU/R378N78gXbBjxed7oA=
|
golang.org/x/sync v0.24.0 h1:iqiDrGoDgyuvAKrhLQthUEzU/R378N78gXbBjxed7oA=
|
||||||
golang.org/x/sync v0.24.0/go.mod h1:sUUOizhqBxiL6pEWpqNLUiaJn1ShEbZ6BBqskPbjZm0=
|
golang.org/x/sync v0.24.0/go.mod h1:sUUOizhqBxiL6pEWpqNLUiaJn1ShEbZ6BBqskPbjZm0=
|
||||||
golang.org/x/sys v0.47.0 h1:o7XGOvZQCADBQQ4Y7VNq2dRWQR7JmOUW8Kxx4ZsNgWs=
|
golang.org/x/sys v0.49.0 h1:XbzkgYJHdqh/8m2Uu0W/dQv8nktxx4BFHp1M0gROTXA=
|
||||||
golang.org/x/sys v0.47.0/go.mod h1:4GL1E5IUh+htKOUEOaiffhrAeqysfVGipDYzABqnCmw=
|
golang.org/x/sys v0.49.0/go.mod h1:hNLxWAXmnKAxqDtdwIYC4bM9oQPEecfsnNMuSxOs3og=
|
||||||
golang.org/x/term v0.45.0 h1:NwWyBmoJCbfTHpxrWoZ9C6/VxOf7ic219I8xZZFdrf0=
|
golang.org/x/term v0.45.0 h1:NwWyBmoJCbfTHpxrWoZ9C6/VxOf7ic219I8xZZFdrf0=
|
||||||
golang.org/x/term v0.45.0/go.mod h1:9aqxs0blBcrm/n0L9QW0aRVD+ktan8ssZromtqJC43w=
|
golang.org/x/term v0.45.0/go.mod h1:9aqxs0blBcrm/n0L9QW0aRVD+ktan8ssZromtqJC43w=
|
||||||
golang.org/x/text v0.43.0 h1:1QivrlhwAsnMOcqgOdl9my7s9OyZl8O5/UGUTJVXoWc=
|
golang.org/x/text v0.43.0 h1:1QivrlhwAsnMOcqgOdl9my7s9OyZl8O5/UGUTJVXoWc=
|
||||||
|
|||||||
@@ -6,3 +6,19 @@ CREATE TABLE IF NOT EXISTS hostnames (
|
|||||||
names TEXT NOT NULL,
|
names TEXT NOT NULL,
|
||||||
expires_at INTEGER NOT NULL
|
expires_at INTEGER NOT NULL
|
||||||
);
|
);
|
||||||
|
|
||||||
|
-- geolocations holds the answers of geojs. A part geojs does not know is NULL.
|
||||||
|
-- fetched_at is in seconds since 1970.
|
||||||
|
CREATE TABLE IF NOT EXISTS geolocations (
|
||||||
|
address TEXT PRIMARY KEY,
|
||||||
|
country_code TEXT,
|
||||||
|
country TEXT,
|
||||||
|
continent_code TEXT,
|
||||||
|
region TEXT,
|
||||||
|
city TEXT,
|
||||||
|
latitude REAL,
|
||||||
|
longitude REAL,
|
||||||
|
accuracy_radius_km INTEGER,
|
||||||
|
time_zone TEXT,
|
||||||
|
fetched_at INTEGER NOT NULL
|
||||||
|
);
|
||||||
@@ -0,0 +1,253 @@
|
|||||||
|
package geo
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"database/sql"
|
||||||
|
"errors"
|
||||||
|
"fmt"
|
||||||
|
"maps"
|
||||||
|
"time"
|
||||||
|
)
|
||||||
|
|
||||||
|
const (
|
||||||
|
// entryLifetime is how long a geojs answer is kept after its fetch.
|
||||||
|
entryLifetime = 7 * 24 * time.Hour
|
||||||
|
|
||||||
|
// flushInterval is how often new answers are written to cache.db.
|
||||||
|
flushInterval = 5 * time.Second
|
||||||
|
|
||||||
|
// sweepInterval is how often expired answers are deleted.
|
||||||
|
sweepInterval = 10 * time.Minute
|
||||||
|
|
||||||
|
// rowColumns are the columns of the geolocations table in the order
|
||||||
|
// scanRow reads them.
|
||||||
|
rowColumns = "address, country_code, country, continent_code, region, " +
|
||||||
|
"city, latitude, longitude, accuracy_radius_km, time_zone, fetched_at"
|
||||||
|
)
|
||||||
|
|
||||||
|
// rowScanner is the Scan method of *sql.Row and *sql.Rows.
|
||||||
|
type rowScanner interface {
|
||||||
|
Scan(dest ...any) error
|
||||||
|
}
|
||||||
|
|
||||||
|
// cached returns the current answer held for address, from memory, then from
|
||||||
|
// the answers waiting to be written, then from cache.db. It returns nil when
|
||||||
|
// there is none.
|
||||||
|
func (l *Locator) cached(ctx context.Context, address string) *Location {
|
||||||
|
found, ok := l.entries.Get(address)
|
||||||
|
if ok {
|
||||||
|
if !l.expired(found) {
|
||||||
|
return found
|
||||||
|
}
|
||||||
|
|
||||||
|
l.entries.Remove(address)
|
||||||
|
}
|
||||||
|
|
||||||
|
l.pendingMu.Lock()
|
||||||
|
waiting := l.pending[address]
|
||||||
|
l.pendingMu.Unlock()
|
||||||
|
|
||||||
|
if waiting != nil && !l.expired(waiting) {
|
||||||
|
l.entries.Add(address, waiting)
|
||||||
|
|
||||||
|
return waiting
|
||||||
|
}
|
||||||
|
|
||||||
|
stored, err := l.readRow(ctx, address)
|
||||||
|
if err != nil {
|
||||||
|
l.logger.Warn("Failed to read a geolocation cache row",
|
||||||
|
"address", address, "error", err)
|
||||||
|
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
if stored != nil {
|
||||||
|
l.entries.Add(address, stored)
|
||||||
|
}
|
||||||
|
|
||||||
|
return stored
|
||||||
|
}
|
||||||
|
|
||||||
|
// expired says whether the answer is at the end of its lifetime.
|
||||||
|
func (l *Locator) expired(answer *Location) bool {
|
||||||
|
return !l.now().Before(answer.Updated.Add(entryLifetime))
|
||||||
|
}
|
||||||
|
|
||||||
|
// cutoff returns the fetch time at or before which an answer has expired.
|
||||||
|
func (l *Locator) cutoff() int64 {
|
||||||
|
return l.now().Add(-entryLifetime).Unix()
|
||||||
|
}
|
||||||
|
|
||||||
|
// store keeps an answer in memory and queues it for cache.db. While cache.db
|
||||||
|
// cannot be written, the queue stops growing at maxEntries answers.
|
||||||
|
func (l *Locator) store(address string, answer *Location) {
|
||||||
|
l.entries.Add(address, answer)
|
||||||
|
|
||||||
|
l.pendingMu.Lock()
|
||||||
|
defer l.pendingMu.Unlock()
|
||||||
|
|
||||||
|
if len(l.pending) < l.maxEntries {
|
||||||
|
l.pending[address] = answer
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// keepCache writes new answers and deletes expired ones until ctx ends.
|
||||||
|
func (l *Locator) keepCache(ctx context.Context) {
|
||||||
|
flushes := time.NewTicker(flushInterval)
|
||||||
|
defer flushes.Stop()
|
||||||
|
|
||||||
|
sweeps := time.NewTicker(sweepInterval)
|
||||||
|
defer sweeps.Stop()
|
||||||
|
|
||||||
|
for {
|
||||||
|
select {
|
||||||
|
case <-ctx.Done():
|
||||||
|
l.flush(context.WithoutCancel(ctx))
|
||||||
|
|
||||||
|
return
|
||||||
|
case <-flushes.C:
|
||||||
|
l.flush(ctx)
|
||||||
|
case <-sweeps.C:
|
||||||
|
l.sweep(ctx)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// load reads the answers that have not expired into memory, at most maxEntries
|
||||||
|
// of them, the newest last so that they are the last to be dropped.
|
||||||
|
func (l *Locator) load(ctx context.Context) error {
|
||||||
|
rows, err := l.db.QueryContext(ctx,
|
||||||
|
"SELECT "+rowColumns+" FROM (SELECT "+rowColumns+" FROM geolocations "+
|
||||||
|
"WHERE fetched_at > ? ORDER BY fetched_at DESC LIMIT ?) "+
|
||||||
|
"ORDER BY fetched_at",
|
||||||
|
l.cutoff(), l.maxEntries)
|
||||||
|
if err != nil {
|
||||||
|
return fmt.Errorf("reading the geolocation cache: %w", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
defer func() { _ = rows.Close() }()
|
||||||
|
|
||||||
|
for rows.Next() {
|
||||||
|
address, answer, err := scanRow(rows)
|
||||||
|
if err != nil {
|
||||||
|
return fmt.Errorf("reading a geolocation cache row: %w", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
l.entries.Add(address, answer)
|
||||||
|
}
|
||||||
|
|
||||||
|
err = rows.Err()
|
||||||
|
if err != nil {
|
||||||
|
return fmt.Errorf("reading the geolocation cache: %w", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// readRow returns the current answer for address in cache.db, or nil when it
|
||||||
|
// has none.
|
||||||
|
func (l *Locator) readRow(
|
||||||
|
ctx context.Context, address string,
|
||||||
|
) (*Location, error) {
|
||||||
|
row := l.db.QueryRowContext(ctx,
|
||||||
|
"SELECT "+rowColumns+" FROM geolocations "+
|
||||||
|
"WHERE address = ? AND fetched_at > ?",
|
||||||
|
address, l.cutoff())
|
||||||
|
|
||||||
|
_, answer, err := scanRow(row)
|
||||||
|
if errors.Is(err, sql.ErrNoRows) {
|
||||||
|
return nil, nil //nolint:nilnil // no row is not a failure
|
||||||
|
}
|
||||||
|
|
||||||
|
return answer, err
|
||||||
|
}
|
||||||
|
|
||||||
|
// scanRow reads one row of the geolocations table.
|
||||||
|
func scanRow(row rowScanner) (string, *Location, error) {
|
||||||
|
var (
|
||||||
|
address string
|
||||||
|
fetched int64
|
||||||
|
answer Location
|
||||||
|
)
|
||||||
|
|
||||||
|
err := row.Scan(&address, &answer.CountryCode, &answer.Country,
|
||||||
|
&answer.ContinentCode, &answer.Region, &answer.City, &answer.Latitude,
|
||||||
|
&answer.Longitude, &answer.AccuracyRadiusKm, &answer.TimeZone, &fetched)
|
||||||
|
if err != nil {
|
||||||
|
return "", nil, err
|
||||||
|
}
|
||||||
|
|
||||||
|
answer.Source = sourceGeoJS
|
||||||
|
answer.Updated = time.Unix(fetched, 0).UTC()
|
||||||
|
|
||||||
|
return address, &answer, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// flush writes the pending answers to cache.db. Those it cannot write stay
|
||||||
|
// pending, unless a newer answer for the address has arrived meanwhile.
|
||||||
|
func (l *Locator) flush(ctx context.Context) {
|
||||||
|
l.pendingMu.Lock()
|
||||||
|
batch := l.pending
|
||||||
|
l.pending = make(map[string]*Location)
|
||||||
|
l.pendingMu.Unlock()
|
||||||
|
|
||||||
|
if len(batch) == 0 {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
|
||||||
|
err := l.writeRows(ctx, batch)
|
||||||
|
if err == nil {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
|
||||||
|
l.logger.Error("Failed to write the geolocation cache", "error", err)
|
||||||
|
|
||||||
|
l.pendingMu.Lock()
|
||||||
|
defer l.pendingMu.Unlock()
|
||||||
|
|
||||||
|
maps.Insert(batch, maps.All(l.pending))
|
||||||
|
l.pending = batch
|
||||||
|
}
|
||||||
|
|
||||||
|
// writeRows writes batch to cache.db in one transaction.
|
||||||
|
func (l *Locator) writeRows(
|
||||||
|
ctx context.Context, batch map[string]*Location,
|
||||||
|
) error {
|
||||||
|
tx, err := l.db.BeginTx(ctx, nil)
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
|
||||||
|
defer func() { _ = tx.Rollback() }()
|
||||||
|
|
||||||
|
for address, answer := range batch {
|
||||||
|
_, err = tx.ExecContext(ctx,
|
||||||
|
"INSERT OR REPLACE INTO geolocations ("+rowColumns+") "+
|
||||||
|
"VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)",
|
||||||
|
address, answer.CountryCode, answer.Country, answer.ContinentCode,
|
||||||
|
answer.Region, answer.City, answer.Latitude, answer.Longitude,
|
||||||
|
answer.AccuracyRadiusKm, answer.TimeZone, answer.Updated.Unix())
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
return tx.Commit()
|
||||||
|
}
|
||||||
|
|
||||||
|
// sweep deletes the expired answers from memory and cache.db.
|
||||||
|
func (l *Locator) sweep(ctx context.Context) {
|
||||||
|
for _, address := range l.entries.Keys() {
|
||||||
|
answer, ok := l.entries.Peek(address)
|
||||||
|
if ok && l.expired(answer) {
|
||||||
|
l.entries.Remove(address)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
_, err := l.db.ExecContext(ctx,
|
||||||
|
"DELETE FROM geolocations WHERE fetched_at <= ?", l.cutoff())
|
||||||
|
if err != nil {
|
||||||
|
l.logger.Error("Failed to delete expired geolocation cache rows",
|
||||||
|
"error", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,349 @@
|
|||||||
|
package geo
|
||||||
|
|
||||||
|
import (
|
||||||
|
"compress/gzip"
|
||||||
|
"context"
|
||||||
|
"errors"
|
||||||
|
"fmt"
|
||||||
|
"io"
|
||||||
|
"io/fs"
|
||||||
|
"net/http"
|
||||||
|
"net/netip"
|
||||||
|
"os"
|
||||||
|
"path/filepath"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"github.com/oschwald/maxminddb-golang/v2"
|
||||||
|
)
|
||||||
|
|
||||||
|
const (
|
||||||
|
// DatabaseDownloadFormat is the URL of DB-IP's City Lite database, with %s
|
||||||
|
// for the year and month.
|
||||||
|
DatabaseDownloadFormat = "https://download.db-ip.com/free/" +
|
||||||
|
"dbip-city-lite-%s.mmdb.gz"
|
||||||
|
|
||||||
|
// databaseMonthFormat is the year and month in the URL.
|
||||||
|
databaseMonthFormat = "2006-01"
|
||||||
|
|
||||||
|
// databaseFileName is the copy of the database in the state directory.
|
||||||
|
databaseFileName = "dbip-city-lite.mmdb"
|
||||||
|
|
||||||
|
// databaseDownloadSuffix ends the name of the file a download is written
|
||||||
|
// to, until it has opened.
|
||||||
|
databaseDownloadSuffix = ".download"
|
||||||
|
|
||||||
|
// maxDatabaseSize is the largest database accepted once unpacked, in
|
||||||
|
// bytes.
|
||||||
|
maxDatabaseSize = 200 << 20
|
||||||
|
|
||||||
|
// databaseDownloadTimeout limits one download.
|
||||||
|
databaseDownloadTimeout = 10 * time.Minute
|
||||||
|
|
||||||
|
// databaseCheckInterval is how often RouteWatch checks whether a new
|
||||||
|
// month's database is due. A new month's file appears some time on the
|
||||||
|
// first of the month, so a failed download is tried again this much later.
|
||||||
|
databaseCheckInterval = time.Hour
|
||||||
|
)
|
||||||
|
|
||||||
|
var (
|
||||||
|
errDatabaseStatus = errors.New("unexpected HTTP status from DB-IP")
|
||||||
|
errDatabaseTooLong = errors.New("database is over the size limit")
|
||||||
|
)
|
||||||
|
|
||||||
|
// dbipName is the part of a database record that holds an English name.
|
||||||
|
type dbipName struct {
|
||||||
|
Names struct {
|
||||||
|
English string `maxminddb:"en"`
|
||||||
|
} `maxminddb:"names"`
|
||||||
|
}
|
||||||
|
|
||||||
|
// dbipRecord is the part of a database record RouteWatch reads.
|
||||||
|
type dbipRecord struct {
|
||||||
|
City dbipName `maxminddb:"city"`
|
||||||
|
Continent struct {
|
||||||
|
Code string `maxminddb:"code"`
|
||||||
|
} `maxminddb:"continent"`
|
||||||
|
Country struct {
|
||||||
|
dbipName
|
||||||
|
|
||||||
|
ISOCode string `maxminddb:"iso_code"`
|
||||||
|
} `maxminddb:"country"`
|
||||||
|
Location struct {
|
||||||
|
Latitude *float64 `maxminddb:"latitude"`
|
||||||
|
Longitude *float64 `maxminddb:"longitude"`
|
||||||
|
} `maxminddb:"location"`
|
||||||
|
Subdivisions []dbipName `maxminddb:"subdivisions"`
|
||||||
|
}
|
||||||
|
|
||||||
|
// location returns the record as a Location fetched at the time given. The
|
||||||
|
// first subdivision is the region. The database has no accuracy radius and no
|
||||||
|
// time zone.
|
||||||
|
func (r *dbipRecord) location(fetched time.Time) *Location {
|
||||||
|
found := &Location{
|
||||||
|
CountryCode: nilIfEmpty(r.Country.ISOCode),
|
||||||
|
Country: nilIfEmpty(r.Country.Names.English),
|
||||||
|
ContinentCode: nilIfEmpty(r.Continent.Code),
|
||||||
|
City: nilIfEmpty(r.City.Names.English),
|
||||||
|
Latitude: r.Location.Latitude,
|
||||||
|
Longitude: r.Location.Longitude,
|
||||||
|
Source: sourceDBIP,
|
||||||
|
Updated: fetched,
|
||||||
|
}
|
||||||
|
|
||||||
|
if len(r.Subdivisions) > 0 {
|
||||||
|
found.Region = nilIfEmpty(r.Subdivisions[0].Names.English)
|
||||||
|
}
|
||||||
|
|
||||||
|
return found
|
||||||
|
}
|
||||||
|
|
||||||
|
// nilIfEmpty returns a pointer to text, or nil when text is empty.
|
||||||
|
func nilIfEmpty(text string) *string {
|
||||||
|
if text == "" {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
return &text
|
||||||
|
}
|
||||||
|
|
||||||
|
// fromDatabase returns where the copy of the database places addr. It returns
|
||||||
|
// nil when there is no copy yet or the copy has no record for addr. It also
|
||||||
|
// returns nil, and logs the error, when reading the copy fails.
|
||||||
|
func (l *Locator) fromDatabase(addr netip.Addr) *Location {
|
||||||
|
l.databaseMu.RLock()
|
||||||
|
defer l.databaseMu.RUnlock()
|
||||||
|
|
||||||
|
if l.database == nil {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
result := l.database.Lookup(addr)
|
||||||
|
|
||||||
|
err := result.Err()
|
||||||
|
if err != nil {
|
||||||
|
l.logger.Error("Failed to look up an address in the database",
|
||||||
|
"address", addr, "error", err)
|
||||||
|
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
if !result.Found() {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
var record dbipRecord
|
||||||
|
|
||||||
|
err = result.Decode(&record)
|
||||||
|
if err != nil {
|
||||||
|
l.logger.Error("Failed to read a database record",
|
||||||
|
"address", addr, "error", err)
|
||||||
|
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
return record.location(l.databaseFetched)
|
||||||
|
}
|
||||||
|
|
||||||
|
// keepDatabaseCurrent downloads the database when it is due, at once and then
|
||||||
|
// every databaseCheckInterval, until ctx ends.
|
||||||
|
func (l *Locator) keepDatabaseCurrent(ctx context.Context) {
|
||||||
|
l.downloadDatabaseIfDue(ctx)
|
||||||
|
|
||||||
|
checks := time.NewTicker(databaseCheckInterval)
|
||||||
|
defer checks.Stop()
|
||||||
|
|
||||||
|
for {
|
||||||
|
select {
|
||||||
|
case <-ctx.Done():
|
||||||
|
return
|
||||||
|
case <-checks.C:
|
||||||
|
l.downloadDatabaseIfDue(ctx)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// downloadDatabaseIfDue downloads this month's database, unless the copy in
|
||||||
|
// use was downloaded this month. A failure leaves the copy as it is.
|
||||||
|
func (l *Locator) downloadDatabaseIfDue(ctx context.Context) {
|
||||||
|
now := l.now().UTC()
|
||||||
|
|
||||||
|
if !l.databaseDue(now) {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
|
||||||
|
err := l.downloadDatabase(ctx, now)
|
||||||
|
if err != nil && ctx.Err() == nil {
|
||||||
|
l.logger.Error("Database download failed; keeping the previous copy",
|
||||||
|
"error", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// databaseDue says whether there is no open copy or it is from an earlier
|
||||||
|
// month than now.
|
||||||
|
func (l *Locator) databaseDue(now time.Time) bool {
|
||||||
|
l.databaseMu.RLock()
|
||||||
|
defer l.databaseMu.RUnlock()
|
||||||
|
|
||||||
|
return l.database == nil ||
|
||||||
|
l.databaseFetched.Format(databaseMonthFormat) !=
|
||||||
|
now.Format(databaseMonthFormat)
|
||||||
|
}
|
||||||
|
|
||||||
|
// downloadDatabase fetches the database of now's month. The file replaces the
|
||||||
|
// copy, on disk and in use, only once it has opened.
|
||||||
|
func (l *Locator) downloadDatabase(ctx context.Context, now time.Time) error {
|
||||||
|
ctx, cancel := context.WithTimeout(ctx, databaseDownloadTimeout)
|
||||||
|
defer cancel()
|
||||||
|
|
||||||
|
url := fmt.Sprintf(l.databaseURLFormat, now.Format(databaseMonthFormat))
|
||||||
|
|
||||||
|
request, err := http.NewRequestWithContext(ctx, http.MethodGet, url, nil)
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
|
||||||
|
request.Header.Set(userAgentHeader, userAgent)
|
||||||
|
|
||||||
|
response, err := l.client.Do(request)
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
|
||||||
|
defer func() { _ = response.Body.Close() }()
|
||||||
|
|
||||||
|
if response.StatusCode != http.StatusOK {
|
||||||
|
return fmt.Errorf("%w: %s", errDatabaseStatus, response.Status)
|
||||||
|
}
|
||||||
|
|
||||||
|
return l.replaceDatabase(response.Body, now)
|
||||||
|
}
|
||||||
|
|
||||||
|
// replaceDatabase unpacks the gzip stream body beside the copy, then makes the
|
||||||
|
// result the copy once it has opened. Whatever fails, the file it wrote is
|
||||||
|
// removed and the copy stays.
|
||||||
|
func (l *Locator) replaceDatabase(body io.Reader, now time.Time) error {
|
||||||
|
root, err := os.OpenRoot(l.dir)
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
|
||||||
|
defer func() { _ = root.Close() }()
|
||||||
|
|
||||||
|
downloadName := databaseFileName + databaseDownloadSuffix
|
||||||
|
|
||||||
|
err = unpackDatabase(root, downloadName, body, l.maxDatabaseBytes)
|
||||||
|
if err != nil {
|
||||||
|
_ = root.Remove(downloadName)
|
||||||
|
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
|
||||||
|
opened, err := maxminddb.Open(filepath.Join(l.dir, downloadName))
|
||||||
|
if err != nil {
|
||||||
|
_ = root.Remove(downloadName)
|
||||||
|
|
||||||
|
return fmt.Errorf("opening the downloaded database: %w", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
err = root.Rename(downloadName, databaseFileName)
|
||||||
|
if err != nil {
|
||||||
|
_ = opened.Close()
|
||||||
|
_ = root.Remove(downloadName)
|
||||||
|
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
|
||||||
|
l.useDatabase(opened, now.Truncate(time.Second))
|
||||||
|
l.logger.Info("Downloaded a new copy of the database")
|
||||||
|
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// unpackDatabase writes the gzip stream body to the file name in root, up to
|
||||||
|
// limit bytes once unpacked.
|
||||||
|
func unpackDatabase(
|
||||||
|
root *os.Root, name string, body io.Reader, limit int64,
|
||||||
|
) error {
|
||||||
|
unpacked, err := gzip.NewReader(body)
|
||||||
|
if err != nil {
|
||||||
|
return fmt.Errorf("unpacking the database: %w", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
f, err := root.Create(name)
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
|
||||||
|
defer func() { _ = f.Close() }()
|
||||||
|
|
||||||
|
written, err := io.Copy(f, io.LimitReader(unpacked, limit+1))
|
||||||
|
if err != nil {
|
||||||
|
return fmt.Errorf("unpacking the database: %w", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
if written > limit {
|
||||||
|
return errDatabaseTooLong
|
||||||
|
}
|
||||||
|
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// openDatabaseCopy opens the copy left by an earlier run, if there is one. A
|
||||||
|
// copy that does not open is treated as no copy, so it is downloaded again.
|
||||||
|
func (l *Locator) openDatabaseCopy() {
|
||||||
|
path := filepath.Join(l.dir, databaseFileName)
|
||||||
|
|
||||||
|
info, err := os.Stat(path)
|
||||||
|
if errors.Is(err, fs.ErrNotExist) {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
|
||||||
|
if err != nil {
|
||||||
|
l.logger.Error("Failed to read the database copy", "error", err)
|
||||||
|
|
||||||
|
return
|
||||||
|
}
|
||||||
|
|
||||||
|
opened, err := maxminddb.Open(path)
|
||||||
|
if err != nil {
|
||||||
|
l.logger.Error("Failed to open the database copy", "error", err)
|
||||||
|
|
||||||
|
return
|
||||||
|
}
|
||||||
|
|
||||||
|
l.useDatabase(opened, info.ModTime().UTC().Truncate(time.Second))
|
||||||
|
}
|
||||||
|
|
||||||
|
// useDatabase makes opened the copy in use and closes the one it replaces.
|
||||||
|
func (l *Locator) useDatabase(opened *maxminddb.Reader, fetched time.Time) {
|
||||||
|
l.databaseMu.Lock()
|
||||||
|
defer l.databaseMu.Unlock()
|
||||||
|
|
||||||
|
if l.database != nil {
|
||||||
|
err := l.database.Close()
|
||||||
|
if err != nil {
|
||||||
|
l.logger.Error("Failed to close the old database copy", "error", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
l.database = opened
|
||||||
|
l.databaseFetched = fetched
|
||||||
|
}
|
||||||
|
|
||||||
|
// closeDatabase closes the copy in use, after which no source can answer from
|
||||||
|
// it.
|
||||||
|
func (l *Locator) closeDatabase() {
|
||||||
|
l.databaseMu.Lock()
|
||||||
|
defer l.databaseMu.Unlock()
|
||||||
|
|
||||||
|
if l.database == nil {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
|
||||||
|
err := l.database.Close()
|
||||||
|
if err != nil {
|
||||||
|
l.logger.Error("Failed to close the database copy", "error", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
l.database = nil
|
||||||
|
}
|
||||||
@@ -0,0 +1,151 @@
|
|||||||
|
package geo
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"encoding/json"
|
||||||
|
"errors"
|
||||||
|
"fmt"
|
||||||
|
"io"
|
||||||
|
"math"
|
||||||
|
"net/http"
|
||||||
|
"net/netip"
|
||||||
|
"strconv"
|
||||||
|
"strings"
|
||||||
|
"time"
|
||||||
|
)
|
||||||
|
|
||||||
|
const (
|
||||||
|
// GeoJSEndpoint is the URL of the geojs geo endpoint.
|
||||||
|
GeoJSEndpoint = "https://get.geojs.io/v1/ip/geo.json"
|
||||||
|
|
||||||
|
// geojsTimeLimit limits one request to geojs.
|
||||||
|
geojsTimeLimit = time.Second
|
||||||
|
|
||||||
|
// geojsAddressSeparator separates the addresses of one request.
|
||||||
|
geojsAddressSeparator = ","
|
||||||
|
|
||||||
|
// geojsUnknown is what geojs sends for a value it does not know.
|
||||||
|
geojsUnknown = "nil"
|
||||||
|
|
||||||
|
// maxGeoJSAnswerBytes is far more than an answer for the most addresses
|
||||||
|
// an IP lookup can ask about.
|
||||||
|
maxGeoJSAnswerBytes = 1 << 20
|
||||||
|
)
|
||||||
|
|
||||||
|
var errGeoJSStatus = errors.New("unexpected HTTP status from geojs")
|
||||||
|
|
||||||
|
// geojsAnswer is one address's entry in an answer from geojs.
|
||||||
|
//
|
||||||
|
//nolint:tagliatelle // the keys geojs sends
|
||||||
|
type geojsAnswer struct {
|
||||||
|
IP string `json:"ip"`
|
||||||
|
CountryCode string `json:"country_code"`
|
||||||
|
Country string `json:"country"`
|
||||||
|
ContinentCode string `json:"continent_code"`
|
||||||
|
Region string `json:"region"`
|
||||||
|
City string `json:"city"`
|
||||||
|
Latitude string `json:"latitude"`
|
||||||
|
Longitude string `json:"longitude"`
|
||||||
|
Accuracy *int `json:"accuracy"`
|
||||||
|
TimeZone string `json:"timezone"`
|
||||||
|
}
|
||||||
|
|
||||||
|
// askGeoJS asks geojs about addresses in one request, and returns its answers
|
||||||
|
// by address. An address geojs left out, or gave no country, city or
|
||||||
|
// coordinates for, has no entry.
|
||||||
|
func (l *Locator) askGeoJS(
|
||||||
|
ctx context.Context, addresses []string,
|
||||||
|
) (map[string]*Location, error) {
|
||||||
|
ctx, cancel := context.WithTimeout(ctx, l.timeLimit)
|
||||||
|
defer cancel()
|
||||||
|
|
||||||
|
// A comma and a colon are legal in a query, and the escaped forms are
|
||||||
|
// not what geojs documents.
|
||||||
|
url := l.geojsURL + "?ip=" + strings.Join(addresses, geojsAddressSeparator)
|
||||||
|
|
||||||
|
request, err := http.NewRequestWithContext(ctx, http.MethodGet, url, nil)
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
|
||||||
|
request.Header.Set(userAgentHeader, userAgent)
|
||||||
|
|
||||||
|
response, err := l.client.Do(request)
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
|
||||||
|
defer func() { _ = response.Body.Close() }()
|
||||||
|
|
||||||
|
if response.StatusCode != http.StatusOK {
|
||||||
|
return nil, fmt.Errorf("%w: %d", errGeoJSStatus, response.StatusCode)
|
||||||
|
}
|
||||||
|
|
||||||
|
var parsed []geojsAnswer
|
||||||
|
|
||||||
|
err = json.NewDecoder(io.LimitReader(response.Body, maxGeoJSAnswerBytes)).
|
||||||
|
Decode(&parsed)
|
||||||
|
if err != nil {
|
||||||
|
return nil, fmt.Errorf("reading the geojs answer: %w", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
fetched := l.now().UTC().Truncate(time.Second)
|
||||||
|
answers := make(map[string]*Location, len(parsed))
|
||||||
|
|
||||||
|
for _, answer := range parsed {
|
||||||
|
addr, err := netip.ParseAddr(answer.IP)
|
||||||
|
if err != nil {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
|
||||||
|
location := answer.location(fetched)
|
||||||
|
if hasLocation(location) {
|
||||||
|
answers[plainAddress(addr)] = location
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
return answers, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// hasLocation says whether found has a country, a city or a coordinate.
|
||||||
|
func hasLocation(found *Location) bool {
|
||||||
|
return found.CountryCode != nil || found.Country != nil ||
|
||||||
|
found.City != nil || found.Latitude != nil || found.Longitude != nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// location returns the answer as a Location fetched at the time given.
|
||||||
|
func (a geojsAnswer) location(fetched time.Time) *Location {
|
||||||
|
return &Location{
|
||||||
|
CountryCode: knownText(a.CountryCode),
|
||||||
|
Country: knownText(a.Country),
|
||||||
|
ContinentCode: knownText(a.ContinentCode),
|
||||||
|
Region: knownText(a.Region),
|
||||||
|
City: knownText(a.City),
|
||||||
|
Latitude: knownNumber(a.Latitude),
|
||||||
|
Longitude: knownNumber(a.Longitude),
|
||||||
|
AccuracyRadiusKm: a.Accuracy,
|
||||||
|
TimeZone: knownText(a.TimeZone),
|
||||||
|
Source: sourceGeoJS,
|
||||||
|
Updated: fetched,
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// knownText returns a pointer to text, or nil when geojs does not know it.
|
||||||
|
func knownText(text string) *string {
|
||||||
|
if text == "" || text == geojsUnknown {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
return &text
|
||||||
|
}
|
||||||
|
|
||||||
|
// knownNumber returns text as a number, or nil when geojs does not know it. NaN
|
||||||
|
// and infinity parse, but JSON cannot carry them.
|
||||||
|
func knownNumber(text string) *float64 {
|
||||||
|
number, err := strconv.ParseFloat(text, 64)
|
||||||
|
if err != nil || math.IsNaN(number) || math.IsInf(number, 0) {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
return &number
|
||||||
|
}
|
||||||
@@ -0,0 +1,232 @@
|
|||||||
|
// Package geo finds where IP addresses are. It asks geojs.io, and keeps those
|
||||||
|
// answers in memory and in cache.db. When geojs.io gives no location for an
|
||||||
|
// address, it reads a copy of DB-IP's City Lite database, which it downloads
|
||||||
|
// once a month.
|
||||||
|
package geo
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"database/sql"
|
||||||
|
"fmt"
|
||||||
|
"log/slog"
|
||||||
|
"net/http"
|
||||||
|
"net/netip"
|
||||||
|
"sync"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
lru "github.com/hashicorp/golang-lru/v2"
|
||||||
|
"github.com/oschwald/maxminddb-golang/v2"
|
||||||
|
)
|
||||||
|
|
||||||
|
// The values of Location.Source.
|
||||||
|
const (
|
||||||
|
sourceGeoJS = "geojs"
|
||||||
|
sourceDBIP = "dbip"
|
||||||
|
)
|
||||||
|
|
||||||
|
const (
|
||||||
|
// maxEntries limits how many geojs answers are held in memory.
|
||||||
|
maxEntries = 250_000
|
||||||
|
|
||||||
|
// userAgentHeader and userAgent name routewatch to geojs and DB-IP.
|
||||||
|
userAgentHeader = "User-Agent"
|
||||||
|
userAgent = "routewatch (+https://git.eeqj.de/sneak/routewatch)"
|
||||||
|
)
|
||||||
|
|
||||||
|
// Params is what New needs.
|
||||||
|
type Params struct {
|
||||||
|
// DB is the open cache.db.
|
||||||
|
DB *sql.DB
|
||||||
|
|
||||||
|
// Dir is the directory that holds the copy of the database.
|
||||||
|
Dir string
|
||||||
|
|
||||||
|
// GeoJSURL is the URL of the geojs geo endpoint, GeoJSEndpoint in use.
|
||||||
|
GeoJSURL string
|
||||||
|
|
||||||
|
// DatabaseURLFormat is the URL of the database with %s for the year and
|
||||||
|
// month, DatabaseDownloadFormat in use.
|
||||||
|
DatabaseURLFormat string
|
||||||
|
|
||||||
|
Logger *slog.Logger
|
||||||
|
}
|
||||||
|
|
||||||
|
// Location is where an address is. A part the source does not know is nil.
|
||||||
|
//
|
||||||
|
//nolint:tagliatelle // snake_case keys, as the Go style guide asks
|
||||||
|
type Location struct {
|
||||||
|
CountryCode *string `json:"country_code"`
|
||||||
|
Country *string `json:"country"`
|
||||||
|
ContinentCode *string `json:"continent_code"`
|
||||||
|
Region *string `json:"region"`
|
||||||
|
City *string `json:"city"`
|
||||||
|
Latitude *float64 `json:"latitude"`
|
||||||
|
Longitude *float64 `json:"longitude"`
|
||||||
|
AccuracyRadiusKm *int `json:"accuracy_radius_km"`
|
||||||
|
TimeZone *string `json:"time_zone"`
|
||||||
|
|
||||||
|
// Source is geojs or dbip.
|
||||||
|
Source string `json:"source"`
|
||||||
|
|
||||||
|
// Updated is when the value was fetched: the geojs answer, or the copy of
|
||||||
|
// the database.
|
||||||
|
Updated time.Time `json:"updated"`
|
||||||
|
}
|
||||||
|
|
||||||
|
// Locator answers where addresses are. Start loads its cache and starts its
|
||||||
|
// goroutines.
|
||||||
|
type Locator struct {
|
||||||
|
db *sql.DB
|
||||||
|
dir string
|
||||||
|
geojsURL string
|
||||||
|
databaseURLFormat string
|
||||||
|
timeLimit time.Duration
|
||||||
|
maxEntries int
|
||||||
|
maxDatabaseBytes int64
|
||||||
|
logger *slog.Logger
|
||||||
|
client *http.Client
|
||||||
|
now func() time.Time
|
||||||
|
|
||||||
|
// entries holds geojs answers by address, and drops the least recently
|
||||||
|
// used first.
|
||||||
|
entries *lru.Cache[string, *Location]
|
||||||
|
|
||||||
|
// pending holds the answers that cache.db does not have yet.
|
||||||
|
pendingMu sync.Mutex
|
||||||
|
pending map[string]*Location
|
||||||
|
|
||||||
|
// database is the open copy, nil until one has opened, and databaseFetched
|
||||||
|
// is when that copy was downloaded. A lookup holds the read lock, so that
|
||||||
|
// a new copy never closes the old one under it.
|
||||||
|
databaseMu sync.RWMutex
|
||||||
|
database *maxminddb.Reader
|
||||||
|
databaseFetched time.Time
|
||||||
|
|
||||||
|
// stop ends the goroutines Start runs, and done is closed when they have
|
||||||
|
// returned.
|
||||||
|
stop context.CancelFunc
|
||||||
|
done chan struct{}
|
||||||
|
}
|
||||||
|
|
||||||
|
// New returns a Locator with an empty cache. Start loads it.
|
||||||
|
func New(params Params) (*Locator, error) {
|
||||||
|
entries, err := lru.New[string, *Location](maxEntries)
|
||||||
|
if err != nil {
|
||||||
|
return nil, fmt.Errorf("creating the geolocation cache: %w", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
return &Locator{
|
||||||
|
db: params.DB,
|
||||||
|
dir: params.Dir,
|
||||||
|
geojsURL: params.GeoJSURL,
|
||||||
|
databaseURLFormat: params.DatabaseURLFormat,
|
||||||
|
timeLimit: geojsTimeLimit,
|
||||||
|
maxEntries: maxEntries,
|
||||||
|
maxDatabaseBytes: maxDatabaseSize,
|
||||||
|
logger: params.Logger.With("component", "geo"),
|
||||||
|
client: &http.Client{},
|
||||||
|
now: time.Now,
|
||||||
|
entries: entries,
|
||||||
|
pending: make(map[string]*Location),
|
||||||
|
done: make(chan struct{}),
|
||||||
|
}, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// Lookup returns where each of addrs is, in the same order. The entry for an
|
||||||
|
// address neither source answers for is nil. Addresses with no current cached
|
||||||
|
// answer go to geojs in one request. The returned Locations are shared with
|
||||||
|
// the cache and must not be changed.
|
||||||
|
func (l *Locator) Lookup(ctx context.Context, addrs []netip.Addr) []*Location {
|
||||||
|
locations := make([]*Location, len(addrs))
|
||||||
|
|
||||||
|
var unanswered []string
|
||||||
|
|
||||||
|
for i, addr := range addrs {
|
||||||
|
address := plainAddress(addr)
|
||||||
|
|
||||||
|
locations[i] = l.cached(ctx, address)
|
||||||
|
if locations[i] == nil {
|
||||||
|
unanswered = append(unanswered, address)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
if len(unanswered) == 0 {
|
||||||
|
return locations
|
||||||
|
}
|
||||||
|
|
||||||
|
answers, err := l.askGeoJS(ctx, unanswered)
|
||||||
|
if err != nil {
|
||||||
|
l.logger.Warn("geojs lookup failed; using the database", "error", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
for i, addr := range addrs {
|
||||||
|
if locations[i] != nil {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
|
||||||
|
address := plainAddress(addr)
|
||||||
|
|
||||||
|
answer, ok := answers[address]
|
||||||
|
if ok {
|
||||||
|
l.store(address, answer)
|
||||||
|
locations[i] = answer
|
||||||
|
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
|
||||||
|
locations[i] = l.fromDatabase(addr)
|
||||||
|
}
|
||||||
|
|
||||||
|
return locations
|
||||||
|
}
|
||||||
|
|
||||||
|
// plainAddress returns addr as the requests to geojs and the cache name it: an
|
||||||
|
// IPv4 address mapped into IPv6 as plain IPv4, and with no zone. A client
|
||||||
|
// chooses the zone, so it must not reach the request.
|
||||||
|
func plainAddress(addr netip.Addr) string {
|
||||||
|
return addr.Unmap().WithZone("").String()
|
||||||
|
}
|
||||||
|
|
||||||
|
// Start loads the cached answers that have not expired and opens the copy of
|
||||||
|
// the database, if there is one. It then starts the goroutines that write new
|
||||||
|
// answers, delete expired ones and download the database. They run until Stop:
|
||||||
|
// the end of ctx does not stop them.
|
||||||
|
func (l *Locator) Start(ctx context.Context) error {
|
||||||
|
err := l.load(ctx)
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
|
||||||
|
l.logger.Info("Geolocation cache loaded", "entries", l.entries.Len())
|
||||||
|
|
||||||
|
l.openDatabaseCopy()
|
||||||
|
|
||||||
|
ctx, l.stop = context.WithCancel(context.WithoutCancel(ctx))
|
||||||
|
|
||||||
|
go func() {
|
||||||
|
defer close(l.done)
|
||||||
|
defer l.closeDatabase()
|
||||||
|
|
||||||
|
var workers sync.WaitGroup
|
||||||
|
|
||||||
|
workers.Go(func() { l.keepCache(ctx) })
|
||||||
|
workers.Go(func() { l.keepDatabaseCurrent(ctx) })
|
||||||
|
workers.Wait()
|
||||||
|
}()
|
||||||
|
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// Stop cancels a download in progress, writes the answers cache.db does not
|
||||||
|
// have yet, and waits until the goroutines have returned or ctx ends.
|
||||||
|
func (l *Locator) Stop(ctx context.Context) {
|
||||||
|
l.stop()
|
||||||
|
|
||||||
|
select {
|
||||||
|
case <-l.done:
|
||||||
|
l.logger.Info("Geolocation stopped")
|
||||||
|
case <-ctx.Done():
|
||||||
|
l.logger.Warn("Stop deadline passed before the geolocation cache " +
|
||||||
|
"was written")
|
||||||
|
}
|
||||||
|
}
|
||||||
File diff suppressed because it is too large.
Load diff
@@ -21,7 +21,7 @@ const (
|
|||||||
downloadTimeout = time.Minute
|
downloadTimeout = time.Minute
|
||||||
|
|
||||||
// maxDownloadSize is the size of the largest list file accepted, in
|
// maxDownloadSize is the size of the largest list file accepted, in
|
||||||
// bytes. The largest list, FireHOL abusers, is about 2.5 MiB.
|
// bytes. The largest list, iCloud Private Relay, is about 12 MiB.
|
||||||
maxDownloadSize = 200 << 20
|
maxDownloadSize = 200 << 20
|
||||||
)
|
)
|
||||||
|
|
||||||
|
|||||||
@@ -51,6 +51,16 @@ const (
|
|||||||
// FormatIVPNJSON is IVPN's JSON server list. The hosts of its wireguard
|
// FormatIVPNJSON is IVPN's JSON server list. The hosts of its wireguard
|
||||||
// and openvpn gateways have their addresses in host and v2ray.
|
// and openvpn gateways have their addresses in host and v2ray.
|
||||||
FormatIVPNJSON
|
FormatIVPNJSON
|
||||||
|
|
||||||
|
// FormatProtonVPN has one address or network per line, with comment lines
|
||||||
|
// that start with #. Unlike FormatLines, a file with no addresses or
|
||||||
|
// networks fails.
|
||||||
|
FormatProtonVPN
|
||||||
|
|
||||||
|
// FormatICloudPrivateRelayCSV is Apple's CSV file of iCloud Private Relay
|
||||||
|
// networks. Each line starts with a network, followed by its country,
|
||||||
|
// region and city.
|
||||||
|
FormatICloudPrivateRelayCSV
|
||||||
)
|
)
|
||||||
|
|
||||||
// List is one public IP list.
|
// List is one public IP list.
|
||||||
|
|||||||
@@ -70,6 +70,31 @@ const (
|
|||||||
"config":{"api":{"ips":["192.0.2.73"]}}}`
|
"config":{"api":{"ips":["192.0.2.73"]}}}`
|
||||||
)
|
)
|
||||||
|
|
||||||
|
// protonVPNBody is a small file in the format of the ProtonVPN-IPs list.
|
||||||
|
const protonVPNBody = "#\n# protonvpn_ip_ranges.txt\n" +
|
||||||
|
"# https://github.com/tn3w/ProtonVPN-IPs\n#\n" +
|
||||||
|
"# The /24 block of every advertised IPv4.\n#\n" +
|
||||||
|
"192.0.2.0/24\n198.51.100.0/24\n"
|
||||||
|
|
||||||
|
// protonVPNEntryBody is a small file in the format of the ProtonVPN-IPs list
|
||||||
|
// of server addresses.
|
||||||
|
const protonVPNEntryBody = "#\n# protonvpn_entry_ips.txt\n" +
|
||||||
|
"# https://github.com/tn3w/ProtonVPN-IPs\n#\n" +
|
||||||
|
"# Entry IPs clients connect to, resolved from ProtonVPN hostnames.\n#\n" +
|
||||||
|
"192.0.2.80\n2001:db8::80\n"
|
||||||
|
|
||||||
|
// iCloudPrivateRelayAddr is an address on iCloudPrivateRelayBody.
|
||||||
|
const iCloudPrivateRelayAddr = "2001:db8:1::5"
|
||||||
|
|
||||||
|
// iCloudPrivateRelayBody is a small file in the format of Apple's iCloud
|
||||||
|
// Private Relay file.
|
||||||
|
const iCloudPrivateRelayBody = "192.0.2.0/31,GB,GB-EN,London,\n" +
|
||||||
|
"198.51.100.32/27,US,US-CA,Los Angeles,\n" +
|
||||||
|
"2001:db8:1::/64,US,US-AK,Anchorage,\n"
|
||||||
|
|
||||||
|
// emptyFileCase names the case of a wrong document that is an empty file.
|
||||||
|
const emptyFileCase = "an empty file"
|
||||||
|
|
||||||
// errorPage is a file that is no list, as a publisher's error page is.
|
// errorPage is a file that is no list, as a publisher's error page is.
|
||||||
const errorPage = "<!DOCTYPE html>\n<html></html>\n"
|
const errorPage = "<!DOCTYPE html>\n<html></html>\n"
|
||||||
|
|
||||||
@@ -230,6 +255,15 @@ func TestLookupFindsListedAddresses(t *testing.T) {
|
|||||||
},
|
},
|
||||||
unlisted: []string{"192.0.2.12", "2001:db8::2"},
|
unlisted: []string{"192.0.2.12", "2001:db8::2"},
|
||||||
},
|
},
|
||||||
|
{
|
||||||
|
name: "iCloud Private Relay networks",
|
||||||
|
format: FormatICloudPrivateRelayCSV,
|
||||||
|
body: iCloudPrivateRelayBody,
|
||||||
|
listed: []string{
|
||||||
|
"192.0.2.1", "198.51.100.63", iCloudPrivateRelayAddr,
|
||||||
|
},
|
||||||
|
unlisted: []string{"192.0.2.2", "198.51.100.64", "2001:db8:2::1"},
|
||||||
|
},
|
||||||
})
|
})
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -267,6 +301,20 @@ func TestLookupFindsVPNServers(t *testing.T) {
|
|||||||
listed: []string{"192.0.2.70", "192.0.2.71", "192.0.2.72"},
|
listed: []string{"192.0.2.70", "192.0.2.71", "192.0.2.72"},
|
||||||
unlisted: []string{"192.0.2.73", "2001:db8::70"},
|
unlisted: []string{"192.0.2.73", "2001:db8::70"},
|
||||||
},
|
},
|
||||||
|
{
|
||||||
|
name: "ProtonVPN networks",
|
||||||
|
format: FormatProtonVPN,
|
||||||
|
body: protonVPNBody,
|
||||||
|
listed: []string{"192.0.2.0", "192.0.2.255", "198.51.100.7"},
|
||||||
|
unlisted: []string{"192.0.3.1", "2001:db8::90"},
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "ProtonVPN server addresses",
|
||||||
|
format: FormatProtonVPN,
|
||||||
|
body: protonVPNEntryBody,
|
||||||
|
listed: []string{"192.0.2.80", "2001:db8::80"},
|
||||||
|
unlisted: []string{"192.0.2.81", "2001:db8::81"},
|
||||||
|
},
|
||||||
})
|
})
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -458,12 +506,12 @@ func TestWrongVPNDocumentFailsToParse(t *testing.T) {
|
|||||||
|
|
||||||
for _, f := range formats {
|
for _, f := range formats {
|
||||||
bodies := map[string]string{
|
bodies := map[string]string{
|
||||||
"not JSON": errorPage,
|
"not JSON": errorPage,
|
||||||
"other JSON": `{"status":"ok","data":{"count":1}}`,
|
"other JSON": `{"status":"ok","data":{"count":1}}`,
|
||||||
"no addresses": f.empty,
|
"no addresses": f.empty,
|
||||||
"bad address": f.badAddress,
|
"bad address": f.badAddress,
|
||||||
"cut off": f.good[:len(f.good)/2],
|
"cut off": f.good[:len(f.good)/2],
|
||||||
"an empty file": "",
|
emptyFileCase: "",
|
||||||
}
|
}
|
||||||
|
|
||||||
for name, body := range bodies {
|
for name, body := range bodies {
|
||||||
@@ -479,6 +527,59 @@ func TestWrongVPNDocumentFailsToParse(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// TestWrongProtonVPNDocumentFailsToParse parses documents that are not a
|
||||||
|
// ProtonVPN-IPs list. Each fails, so none can replace a good copy.
|
||||||
|
func TestWrongProtonVPNDocumentFailsToParse(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
bodies := map[string]string{
|
||||||
|
"error page": errorPage,
|
||||||
|
"JSON": `{"status":"ok"}`,
|
||||||
|
"comments only": "#\n# protonvpn_ip_ranges.txt\n#\n",
|
||||||
|
"bad network": "192.0.2.0/24\nexample\n",
|
||||||
|
emptyFileCase: "",
|
||||||
|
}
|
||||||
|
|
||||||
|
for name, body := range bodies {
|
||||||
|
t.Run(name, func(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
addresses, err := parse(strings.NewReader(body), FormatProtonVPN)
|
||||||
|
if err == nil {
|
||||||
|
t.Errorf("parse returned %v and no error", addresses.Ranges())
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestWrongICloudPrivateRelayDocumentFailsToParse parses documents that are
|
||||||
|
// not Apple's iCloud Private Relay file. Each fails, so none can replace a
|
||||||
|
// good copy.
|
||||||
|
func TestWrongICloudPrivateRelayDocumentFailsToParse(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
bodies := map[string]string{
|
||||||
|
"error page": errorPage,
|
||||||
|
"JSON": `{"status":"ok","count":1}`,
|
||||||
|
"no other field": "192.0.2.0/24\n",
|
||||||
|
"bad network": "192.0.2.0/24,GB,GB-EN,London,\nexample,GB,GB-EN,London,\n",
|
||||||
|
"an address": "192.0.2.1,GB,GB-EN,London,\n",
|
||||||
|
emptyFileCase: "",
|
||||||
|
}
|
||||||
|
|
||||||
|
for name, body := range bodies {
|
||||||
|
t.Run(name, func(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
addresses, err := parse(strings.NewReader(body),
|
||||||
|
FormatICloudPrivateRelayCSV)
|
||||||
|
if err == nil {
|
||||||
|
t.Errorf("parse returned %v and no error", addresses.Ranges())
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
// TestLookupMarksVPNLists looks up an address on a VPN list and on another
|
// TestLookupMarksVPNLists looks up an address on a VPN list and on another
|
||||||
// list. Only the VPN list's listing says it is one.
|
// list. Only the VPN list's listing says it is one.
|
||||||
func TestLookupMarksVPNLists(t *testing.T) {
|
func TestLookupMarksVPNLists(t *testing.T) {
|
||||||
@@ -523,7 +624,10 @@ func TestPublicListsMarkTheVPNLists(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
want := []string{"ivpn", "mullvad", "nordvpn", "pia", "x4bnet", "x4bnet_v6"}
|
want := []string{
|
||||||
|
"ivpn", "mullvad", "nordvpn", "pia", "protonvpn", "protonvpn_entry",
|
||||||
|
"x4bnet", "x4bnet_v6",
|
||||||
|
}
|
||||||
if !slices.Equal(got, want) {
|
if !slices.Equal(got, want) {
|
||||||
t.Errorf("VPN lists %v, want %v", got, want)
|
t.Errorf("VPN lists %v, want %v", got, want)
|
||||||
}
|
}
|
||||||
@@ -830,15 +934,33 @@ func TestStopCancelsADownloadInProgress(t *testing.T) {
|
|||||||
func TestTorExitsListHoldsIPv6Exits(t *testing.T) {
|
func TestTorExitsListHoldsIPv6Exits(t *testing.T) {
|
||||||
t.Parallel()
|
t.Parallel()
|
||||||
|
|
||||||
|
checkPublicListHolds(t, TorExitsList, onionooBody, onionooIPv6ExitAddr)
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestICloudPrivateRelayListHoldsItsNetworks downloads a file in Apple's
|
||||||
|
// format as the public iCloud Private Relay list, which is what an address's
|
||||||
|
// is_relay answer comes from.
|
||||||
|
func TestICloudPrivateRelayListHoldsItsNetworks(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
checkPublicListHolds(t, ICloudPrivateRelayList, iCloudPrivateRelayBody,
|
||||||
|
iCloudPrivateRelayAddr)
|
||||||
|
}
|
||||||
|
|
||||||
|
// checkPublicListHolds downloads body as the public list called name and
|
||||||
|
// checks that addr has one listing, which names that list.
|
||||||
|
func checkPublicListHolds(t *testing.T, name, body, addr string) {
|
||||||
|
t.Helper()
|
||||||
|
|
||||||
index := slices.IndexFunc(Lists(), func(list List) bool {
|
index := slices.IndexFunc(Lists(), func(list List) bool {
|
||||||
return list.Name == TorExitsList
|
return list.Name == name
|
||||||
})
|
})
|
||||||
if index < 0 {
|
if index < 0 {
|
||||||
t.Fatalf("no list named %s", TorExitsList)
|
t.Fatalf("no list named %s", name)
|
||||||
}
|
}
|
||||||
|
|
||||||
list := Lists()[index]
|
list := Lists()[index]
|
||||||
_, list.URL = newPublisher(t, onionooBody)
|
_, list.URL = newPublisher(t, body)
|
||||||
b := newTestIPLists(t.TempDir(), list)
|
b := newTestIPLists(t.TempDir(), list)
|
||||||
|
|
||||||
err := b.download(t.Context(), list)
|
err := b.download(t.Context(), list)
|
||||||
@@ -846,9 +968,9 @@ func TestTorExitsListHoldsIPv6Exits(t *testing.T) {
|
|||||||
t.Fatalf("download: %v", err)
|
t.Fatalf("download: %v", err)
|
||||||
}
|
}
|
||||||
|
|
||||||
listings := lookup(t, b, onionooIPv6ExitAddr)
|
listings := lookup(t, b, addr)
|
||||||
if len(listings) != 1 || listings[0].Name != TorExitsList {
|
if len(listings) != 1 || listings[0].Name != name {
|
||||||
t.Errorf("listings %+v, want one for %s", listings, TorExitsList)
|
t.Errorf("listings %+v, want one for %s", listings, name)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
@@ -8,6 +8,13 @@ import (
|
|||||||
// TorExitsList is the name of the list of Tor exit nodes.
|
// TorExitsList is the name of the list of Tor exit nodes.
|
||||||
const TorExitsList = "tor_exits"
|
const TorExitsList = "tor_exits"
|
||||||
|
|
||||||
|
// ICloudPrivateRelayList is the name of the list of iCloud Private Relay's
|
||||||
|
// networks.
|
||||||
|
const ICloudPrivateRelayList = "icloud_private_relay"
|
||||||
|
|
||||||
|
// protonVPNIPsURL is where the ProtonVPN-IPs mirror publishes its files.
|
||||||
|
const protonVPNIPsURL = "https://raw.githubusercontent.com/tn3w/ProtonVPN-IPs/master/"
|
||||||
|
|
||||||
// How often each list is downloaded. No list is downloaded more often than
|
// How often each list is downloaded. No list is downloaded more often than
|
||||||
// its publisher updates it or allows.
|
// its publisher updates it or allows.
|
||||||
const (
|
const (
|
||||||
@@ -27,6 +34,9 @@ const (
|
|||||||
fireHOLAbusersInterval = 5 * time.Hour
|
fireHOLAbusersInterval = 5 * time.Hour
|
||||||
fireHOLLevel3Interval = 3 * time.Hour
|
fireHOLLevel3Interval = 3 * time.Hour
|
||||||
|
|
||||||
|
// Apple's iCloud Private Relay file is 12 MB, so once a day.
|
||||||
|
iCloudPrivateRelayInterval = 24 * time.Hour
|
||||||
|
|
||||||
// Spamhaus asks that DROP be downloaded at most once a day:
|
// Spamhaus asks that DROP be downloaded at most once a day:
|
||||||
// https://www.spamhaus.org/blocklists/do-not-route-or-peer/
|
// https://www.spamhaus.org/blocklists/do-not-route-or-peer/
|
||||||
spamhausDROPInterval = 24 * time.Hour
|
spamhausDROPInterval = 24 * time.Hour
|
||||||
@@ -81,6 +91,13 @@ func nonVPNLists() []List {
|
|||||||
Interval: fireHOLLevel3Interval,
|
Interval: fireHOLLevel3Interval,
|
||||||
Format: FormatLines,
|
Format: FormatLines,
|
||||||
},
|
},
|
||||||
|
{
|
||||||
|
Name: ICloudPrivateRelayList,
|
||||||
|
Description: "In a network where iCloud Private Relay traffic leaves",
|
||||||
|
URL: "https://mask-api.icloud.com/egress-ip-ranges.csv",
|
||||||
|
Interval: iCloudPrivateRelayInterval,
|
||||||
|
Format: FormatICloudPrivateRelayCSV,
|
||||||
|
},
|
||||||
{
|
{
|
||||||
Name: "spamhaus_drop",
|
Name: "spamhaus_drop",
|
||||||
Description: "In an IPv4 network that Spamhaus says spammers " +
|
Description: "In an IPv4 network that Spamhaus says spammers " +
|
||||||
@@ -146,6 +163,28 @@ func vpnLists() []List {
|
|||||||
Format: FormatPIAJSON,
|
Format: FormatPIAJSON,
|
||||||
VPN: true,
|
VPN: true,
|
||||||
},
|
},
|
||||||
|
{
|
||||||
|
// Proton's own server list needs a logged-in session, so this is
|
||||||
|
// the public ProtonVPN-IPs mirror of it. Proton publishes only
|
||||||
|
// the addresses clients connect to. The mirror's README says
|
||||||
|
// traffic leaves from others in the same /24, so this file of
|
||||||
|
// /24 networks is the one it says to use for detecting exits.
|
||||||
|
Name: "protonvpn",
|
||||||
|
Description: "In a /24 network that holds a server of the " +
|
||||||
|
"ProtonVPN service",
|
||||||
|
URL: protonVPNIPsURL + "protonvpn_ip_ranges.txt",
|
||||||
|
Interval: vpnListInterval,
|
||||||
|
Format: FormatProtonVPN,
|
||||||
|
VPN: true,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
Name: "protonvpn_entry",
|
||||||
|
Description: "A server of the ProtonVPN service",
|
||||||
|
URL: protonVPNIPsURL + "protonvpn_entry_ips.txt",
|
||||||
|
Interval: vpnListInterval,
|
||||||
|
Format: FormatProtonVPN,
|
||||||
|
VPN: true,
|
||||||
|
},
|
||||||
{
|
{
|
||||||
Name: "x4bnet",
|
Name: "x4bnet",
|
||||||
Description: "In an IPv4 network of a VPN service, according to " +
|
Description: "In an IPv4 network of a VPN service, according to " +
|
||||||
|
|||||||
@@ -25,6 +25,10 @@ const (
|
|||||||
// dshieldFieldSeparator separates the fields of a DShield line.
|
// dshieldFieldSeparator separates the fields of a DShield line.
|
||||||
dshieldFieldSeparator = "\t"
|
dshieldFieldSeparator = "\t"
|
||||||
|
|
||||||
|
// iCloudPrivateRelayFieldSeparator separates the fields of a line of
|
||||||
|
// Apple's iCloud Private Relay file.
|
||||||
|
iCloudPrivateRelayFieldSeparator = ","
|
||||||
|
|
||||||
// spamhausMetadataType is the type of the line that ends a Spamhaus
|
// spamhausMetadataType is the type of the line that ends a Spamhaus
|
||||||
// file. That line holds the file's metadata and no network.
|
// file. That line holds the file's metadata and no network.
|
||||||
spamhausMetadataType = "metadata"
|
spamhausMetadataType = "metadata"
|
||||||
@@ -58,6 +62,10 @@ func parse(r io.Reader, format Format) (*netipx.IPSet, error) {
|
|||||||
return parsePIA(r)
|
return parsePIA(r)
|
||||||
case FormatIVPNJSON:
|
case FormatIVPNJSON:
|
||||||
return parseIVPN(r)
|
return parseIVPN(r)
|
||||||
|
case FormatProtonVPN:
|
||||||
|
return parseLinesNotEmpty(r, addAddressOrNetwork)
|
||||||
|
case FormatICloudPrivateRelayCSV:
|
||||||
|
return parseLinesNotEmpty(r, addICloudPrivateRelayNetwork)
|
||||||
default:
|
default:
|
||||||
return nil, errUnknownFormat
|
return nil, errUnknownFormat
|
||||||
}
|
}
|
||||||
@@ -329,6 +337,41 @@ func parseIVPN(r io.Reader) (*netipx.IPSet, error) {
|
|||||||
return addressSet(texts)
|
return addressSet(texts)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// parseLinesNotEmpty is parseLines for a list that always holds addresses, so
|
||||||
|
// a file with none fails.
|
||||||
|
func parseLinesNotEmpty(
|
||||||
|
r io.Reader, addLine func(*netipx.IPSetBuilder, string) error,
|
||||||
|
) (*netipx.IPSet, error) {
|
||||||
|
addresses, err := parseLines(r, addLine)
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
|
||||||
|
if len(addresses.Ranges()) == 0 {
|
||||||
|
return nil, errNoAddresses
|
||||||
|
}
|
||||||
|
|
||||||
|
return addresses, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// addICloudPrivateRelayNetwork adds the network at the start of a line of
|
||||||
|
// Apple's iCloud Private Relay file to builder.
|
||||||
|
func addICloudPrivateRelayNetwork(builder *netipx.IPSetBuilder, line string) error {
|
||||||
|
field, _, found := strings.Cut(line, iCloudPrivateRelayFieldSeparator)
|
||||||
|
if !found {
|
||||||
|
return errNoNetwork
|
||||||
|
}
|
||||||
|
|
||||||
|
network, err := netip.ParsePrefix(field)
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
|
||||||
|
builder.AddPrefix(network)
|
||||||
|
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
// readJSON reads r to its end and decodes it, as one JSON document, into v.
|
// readJSON reads r to its end and decodes it, as one JSON document, into v.
|
||||||
func readJSON(r io.Reader, v any) error {
|
func readJSON(r io.Reader, v any) error {
|
||||||
data, err := io.ReadAll(r)
|
data, err := io.ReadAll(r)
|
||||||
|
|||||||
@@ -19,6 +19,7 @@ import (
|
|||||||
"git.eeqj.de/sneak/routewatch/internal/cachedb"
|
"git.eeqj.de/sneak/routewatch/internal/cachedb"
|
||||||
"git.eeqj.de/sneak/routewatch/internal/config"
|
"git.eeqj.de/sneak/routewatch/internal/config"
|
||||||
"git.eeqj.de/sneak/routewatch/internal/database"
|
"git.eeqj.de/sneak/routewatch/internal/database"
|
||||||
|
"git.eeqj.de/sneak/routewatch/internal/geo"
|
||||||
"git.eeqj.de/sneak/routewatch/internal/iplist"
|
"git.eeqj.de/sneak/routewatch/internal/iplist"
|
||||||
"git.eeqj.de/sneak/routewatch/internal/logger"
|
"git.eeqj.de/sneak/routewatch/internal/logger"
|
||||||
"git.eeqj.de/sneak/routewatch/internal/metrics"
|
"git.eeqj.de/sneak/routewatch/internal/metrics"
|
||||||
@@ -67,6 +68,7 @@ type RouteWatch struct {
|
|||||||
ipLists *iplist.IPLists
|
ipLists *iplist.IPLists
|
||||||
cacheDB *sql.DB
|
cacheDB *sql.DB
|
||||||
ptrCache *ptr.Cache
|
ptrCache *ptr.Cache
|
||||||
|
geoLocator *geo.Locator
|
||||||
}
|
}
|
||||||
|
|
||||||
// ipListsDirName is the directory in the state directory that holds the IP
|
// ipListsDirName is the directory in the state directory that holds the IP
|
||||||
@@ -160,6 +162,11 @@ func (rw *RouteWatch) Run(ctx context.Context) error {
|
|||||||
return err
|
return err
|
||||||
}
|
}
|
||||||
|
|
||||||
|
err = rw.startGeoLocator(ctx)
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
|
||||||
// Start HTTP server
|
// Start HTTP server
|
||||||
err = rw.server.Start(ctx)
|
err = rw.server.Start(ctx)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
@@ -178,8 +185,9 @@ func (rw *RouteWatch) Run(ctx context.Context) error {
|
|||||||
|
|
||||||
// Shutdown stops all services in order: the streamer, the handlers, which
|
// Shutdown stops all services in order: the streamer, the handlers, which
|
||||||
// write what they hold, the WHOIS fetcher, the database maintainer, the
|
// write what they hold, the WHOIS fetcher, the database maintainer, the
|
||||||
// IP list downloads, the HTTP server and the PTR cache, and last the database,
|
// IP list downloads, the HTTP server, the geolocation cache and the PTR cache,
|
||||||
// which it checkpoints and closes. Each stage waits at most until ctx ends, so
|
// and last the database, which it checkpoints and closes. Each stage waits at
|
||||||
|
// most until ctx ends, so
|
||||||
// Shutdown returns soon after: a handler logs what it could not write by
|
// Shutdown returns soon after: a handler logs what it could not write by
|
||||||
// then, and the HTTP server closes the connections of requests still
|
// then, and the HTTP server closes the connections of requests still
|
||||||
// running.
|
// running.
|
||||||
@@ -239,6 +247,10 @@ func (rw *RouteWatch) Shutdown(ctx context.Context) {
|
|||||||
"error", err)
|
"error", err)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
if rw.geoLocator != nil {
|
||||||
|
rw.geoLocator.Stop(ctx)
|
||||||
|
}
|
||||||
|
|
||||||
rw.stopPTRCache(ctx)
|
rw.stopPTRCache(ctx)
|
||||||
|
|
||||||
rw.closeDatabase(ctx)
|
rw.closeDatabase(ctx)
|
||||||
@@ -283,6 +295,32 @@ func (rw *RouteWatch) startPTRCache(ctx context.Context) error {
|
|||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// startGeoLocator loads the geolocation cache from cache.db, which
|
||||||
|
// startPTRCache has opened, and gives the server the locator. The HTTP handlers
|
||||||
|
// read it without a lock, so it is set before they run.
|
||||||
|
func (rw *RouteWatch) startGeoLocator(ctx context.Context) error {
|
||||||
|
locator, err := geo.New(geo.Params{
|
||||||
|
DB: rw.cacheDB,
|
||||||
|
Dir: rw.config.StateDir,
|
||||||
|
GeoJSURL: geo.GeoJSEndpoint,
|
||||||
|
DatabaseURLFormat: geo.DatabaseDownloadFormat,
|
||||||
|
Logger: rw.logger.Logger,
|
||||||
|
})
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
|
||||||
|
err = locator.Start(ctx)
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
|
||||||
|
rw.geoLocator = locator
|
||||||
|
rw.server.SetGeoLocator(locator)
|
||||||
|
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
// stopPTRCache stops the PTR cache, which writes the answers cache.db does not
|
// stopPTRCache stops the PTR cache, which writes the answers cache.db does not
|
||||||
// have yet, and closes cache.db, unless ctx has ended and a write may still be
|
// have yet, and closes cache.db, unless ctx has ended and a write may still be
|
||||||
// running.
|
// running.
|
||||||
|
|||||||
@@ -17,6 +17,7 @@ import (
|
|||||||
|
|
||||||
"git.eeqj.de/sneak/routewatch/internal/bogon"
|
"git.eeqj.de/sneak/routewatch/internal/bogon"
|
||||||
"git.eeqj.de/sneak/routewatch/internal/database"
|
"git.eeqj.de/sneak/routewatch/internal/database"
|
||||||
|
"git.eeqj.de/sneak/routewatch/internal/geo"
|
||||||
"git.eeqj.de/sneak/routewatch/internal/iplist"
|
"git.eeqj.de/sneak/routewatch/internal/iplist"
|
||||||
"git.eeqj.de/sneak/routewatch/internal/metrics"
|
"git.eeqj.de/sneak/routewatch/internal/metrics"
|
||||||
"git.eeqj.de/sneak/routewatch/internal/ristypes"
|
"git.eeqj.de/sneak/routewatch/internal/ristypes"
|
||||||
@@ -764,9 +765,9 @@ type IPLookupResponse struct {
|
|||||||
}
|
}
|
||||||
|
|
||||||
// IPLookupResult is the answer for one address: its PTR names, its flags, its
|
// IPLookupResult is the answer for one address: its PTR names, its flags, its
|
||||||
// live route and that route's AS, and the IP lists that contain it. Every
|
// live route and that route's AS, the IP lists that contain it, and where it
|
||||||
// result has every field. The fields of IPInfo are null when the address has
|
// is. Every result has every field. The fields of IPInfo are null when the
|
||||||
// no live route.
|
// address has no live route.
|
||||||
//
|
//
|
||||||
//nolint:tagliatelle // snake_case keys, as the Go style guide asks
|
//nolint:tagliatelle // snake_case keys, as the Go style guide asks
|
||||||
type IPLookupResult struct {
|
type IPLookupResult struct {
|
||||||
@@ -784,6 +785,9 @@ type IPLookupResult struct {
|
|||||||
|
|
||||||
// IPLists has one entry for each IP list the address is on.
|
// IPLists has one entry for each IP list the address is on.
|
||||||
IPLists []iplist.Listing `json:"ip_lists"`
|
IPLists []iplist.Listing `json:"ip_lists"`
|
||||||
|
|
||||||
|
// Geo is where the address is. It is nil when no source answers.
|
||||||
|
Geo *geo.Location `json:"geo"`
|
||||||
}
|
}
|
||||||
|
|
||||||
// IPLookupFlags are the yes-or-no facts about an address. Every result has
|
// IPLookupFlags are the yes-or-no facts about an address. Every result has
|
||||||
@@ -797,6 +801,9 @@ type IPLookupFlags struct {
|
|||||||
// IsVPN says that the address is on a list of VPN servers.
|
// IsVPN says that the address is on a list of VPN servers.
|
||||||
IsVPN bool `json:"is_vpn"`
|
IsVPN bool `json:"is_vpn"`
|
||||||
|
|
||||||
|
// IsRelay says that the address is on the iCloud Private Relay list.
|
||||||
|
IsRelay bool `json:"is_relay"`
|
||||||
|
|
||||||
// IsBogon says that the address is reserved, as package bogon tells.
|
// IsBogon says that the address is reserved, as package bogon tells.
|
||||||
IsBogon bool `json:"is_bogon"`
|
IsBogon bool `json:"is_bogon"`
|
||||||
}
|
}
|
||||||
@@ -839,6 +846,8 @@ func (s *Server) handleIPInfo() http.HandlerFunc {
|
|||||||
return
|
return
|
||||||
}
|
}
|
||||||
|
|
||||||
|
locations := s.geoLocations(ctx, ipsToLookup)
|
||||||
|
|
||||||
// Track ASNs that need WHOIS refresh
|
// Track ASNs that need WHOIS refresh
|
||||||
refreshASNs := make(map[int]bool)
|
refreshASNs := make(map[int]bool)
|
||||||
|
|
||||||
@@ -874,7 +883,9 @@ func (s *Server) handleIPInfo() http.HandlerFunc {
|
|||||||
return
|
return
|
||||||
}
|
}
|
||||||
|
|
||||||
response.Results = append(response.Results, s.ipLookupResult(ip, ptrs, ipInfo))
|
result := s.ipLookupResult(ip, ptrs, ipInfo)
|
||||||
|
result.Geo = locations[ip]
|
||||||
|
response.Results = append(response.Results, result)
|
||||||
|
|
||||||
if ipInfo.NeedsWHOISRefresh {
|
if ipInfo.NeedsWHOISRefresh {
|
||||||
refreshASNs[*ipInfo.ASN] = true
|
refreshASNs[*ipInfo.ASN] = true
|
||||||
@@ -1025,10 +1036,52 @@ func ipLookupFlags(ip string, listings []iplist.Listing) IPLookupFlags {
|
|||||||
IsVPN: slices.ContainsFunc(listings, func(l iplist.Listing) bool {
|
IsVPN: slices.ContainsFunc(listings, func(l iplist.Listing) bool {
|
||||||
return l.VPN
|
return l.VPN
|
||||||
}),
|
}),
|
||||||
|
IsRelay: slices.ContainsFunc(listings, func(l iplist.Listing) bool {
|
||||||
|
return l.Name == iplist.ICloudPrivateRelayList
|
||||||
|
}),
|
||||||
IsBogon: isBogon(ip),
|
IsBogon: isBogon(ip),
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// geoLocations returns where ips are, by address. An address that does not
|
||||||
|
// parse has no entry. Nor has a reserved one, which is sent to no outside
|
||||||
|
// source and has no location.
|
||||||
|
func (s *Server) geoLocations(
|
||||||
|
ctx context.Context, ips []string,
|
||||||
|
) map[string]*geo.Location {
|
||||||
|
if s.geoLocator == nil {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
var (
|
||||||
|
kept []string
|
||||||
|
addrs []netip.Addr
|
||||||
|
)
|
||||||
|
|
||||||
|
for _, ip := range ips {
|
||||||
|
addr, err := netip.ParseAddr(ip)
|
||||||
|
if err != nil || isBogon(ip) {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
|
||||||
|
kept = append(kept, ip)
|
||||||
|
addrs = append(addrs, addr)
|
||||||
|
}
|
||||||
|
|
||||||
|
if len(addrs) == 0 {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
answers := s.geoLocator.Lookup(ctx, addrs)
|
||||||
|
locations := make(map[string]*geo.Location, len(kept))
|
||||||
|
|
||||||
|
for i, ip := range kept {
|
||||||
|
locations[ip] = answers[i]
|
||||||
|
}
|
||||||
|
|
||||||
|
return locations
|
||||||
|
}
|
||||||
|
|
||||||
// isBogon says whether the address ip is reserved, as package bogon tells.
|
// isBogon says whether the address ip is reserved, as package bogon tells.
|
||||||
func isBogon(ip string) bool {
|
func isBogon(ip string) bool {
|
||||||
addr, err := netip.ParseAddr(ip)
|
addr, err := netip.ParseAddr(ip)
|
||||||
|
|||||||
@@ -15,6 +15,7 @@ import (
|
|||||||
"slices"
|
"slices"
|
||||||
"strconv"
|
"strconv"
|
||||||
"strings"
|
"strings"
|
||||||
|
"sync"
|
||||||
"sync/atomic"
|
"sync/atomic"
|
||||||
"testing"
|
"testing"
|
||||||
"testing/synctest"
|
"testing/synctest"
|
||||||
@@ -22,6 +23,7 @@ import (
|
|||||||
|
|
||||||
"git.eeqj.de/sneak/routewatch/internal/config"
|
"git.eeqj.de/sneak/routewatch/internal/config"
|
||||||
"git.eeqj.de/sneak/routewatch/internal/database"
|
"git.eeqj.de/sneak/routewatch/internal/database"
|
||||||
|
"git.eeqj.de/sneak/routewatch/internal/geo"
|
||||||
"git.eeqj.de/sneak/routewatch/internal/iplist"
|
"git.eeqj.de/sneak/routewatch/internal/iplist"
|
||||||
"git.eeqj.de/sneak/routewatch/internal/logger"
|
"git.eeqj.de/sneak/routewatch/internal/logger"
|
||||||
"git.eeqj.de/sneak/routewatch/internal/metrics"
|
"git.eeqj.de/sneak/routewatch/internal/metrics"
|
||||||
@@ -1504,6 +1506,10 @@ const listedUnroutedAddr = "203.0.113.1"
|
|||||||
// torExitAddr is the address that fakeIPLists lists on the Tor exit list.
|
// torExitAddr is the address that fakeIPLists lists on the Tor exit list.
|
||||||
const torExitAddr = "203.0.113.2"
|
const torExitAddr = "203.0.113.2"
|
||||||
|
|
||||||
|
// relayAddr is the address that fakeIPLists lists on the iCloud Private Relay
|
||||||
|
// list.
|
||||||
|
const relayAddr = "203.0.113.5"
|
||||||
|
|
||||||
// The addresses that fakeIPLists lists on one and on two VPN lists, and the
|
// The addresses that fakeIPLists lists on one and on two VPN lists, and the
|
||||||
// names of those lists.
|
// names of those lists.
|
||||||
const (
|
const (
|
||||||
@@ -1513,10 +1519,11 @@ const (
|
|||||||
otherTestVPN = "other_test_vpn"
|
otherTestVPN = "other_test_vpn"
|
||||||
)
|
)
|
||||||
|
|
||||||
// fakeIPLists has four lists. The test list holds addrInTestPrefixV4,
|
// fakeIPLists has five lists. The test list holds addrInTestPrefixV4,
|
||||||
// listedUnroutedAddr and twoVPNListsAddr. The Tor exit list holds torExitAddr.
|
// listedUnroutedAddr and twoVPNListsAddr. The Tor exit list holds torExitAddr.
|
||||||
// The test VPN list holds oneVPNListAddr and twoVPNListsAddr, and the other
|
// The iCloud Private Relay list holds relayAddr. The test VPN list holds
|
||||||
// test VPN list holds twoVPNListsAddr.
|
// oneVPNListAddr and twoVPNListsAddr, and the other test VPN list holds
|
||||||
|
// twoVPNListsAddr.
|
||||||
type fakeIPLists struct{}
|
type fakeIPLists struct{}
|
||||||
|
|
||||||
func testListing() iplist.Listing {
|
func testListing() iplist.Listing {
|
||||||
@@ -1537,6 +1544,10 @@ func (fakeIPLists) Lookup(addr netip.Addr) []iplist.Listing {
|
|||||||
return []iplist.Listing{{Name: iplist.TorExitsList}}
|
return []iplist.Listing{{Name: iplist.TorExitsList}}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
if addr == netip.MustParseAddr(relayAddr) {
|
||||||
|
return []iplist.Listing{{Name: iplist.ICloudPrivateRelayList}}
|
||||||
|
}
|
||||||
|
|
||||||
if addr == netip.MustParseAddr(oneVPNListAddr) {
|
if addr == netip.MustParseAddr(oneVPNListAddr) {
|
||||||
return []iplist.Listing{{Name: testVPNList, VPN: true}}
|
return []iplist.Listing{{Name: testVPNList, VPN: true}}
|
||||||
}
|
}
|
||||||
@@ -1569,20 +1580,22 @@ func TestIPLookupFlags(t *testing.T) {
|
|||||||
name string
|
name string
|
||||||
addr string
|
addr string
|
||||||
tor bool
|
tor bool
|
||||||
|
relay bool
|
||||||
bogon bool
|
bogon bool
|
||||||
}{
|
}{
|
||||||
{"on the Tor exit list", torExitAddr, true, true},
|
{"on the Tor exit list", torExitAddr, true, false, true},
|
||||||
{"on another list", listedUnroutedAddr, false, true},
|
{"on the iCloud Private Relay list", relayAddr, false, true, true},
|
||||||
{"routed IPv4", addrInTestPrefixV4, false, true},
|
{"on another list", listedUnroutedAddr, false, false, true},
|
||||||
{"routed IPv6", addrInTestPrefixV6, false, true},
|
{"routed IPv4", addrInTestPrefixV4, false, false, true},
|
||||||
{"private IPv4", "10.0.0.1", false, true},
|
{"routed IPv6", addrInTestPrefixV6, false, false, true},
|
||||||
{"loopback IPv4", "127.0.0.1", false, true},
|
{"private IPv4", "10.0.0.1", false, false, true},
|
||||||
{"public IPv4", "1.1.1.1", false, false},
|
{"loopback IPv4", "127.0.0.1", false, false, true},
|
||||||
{"private IPv6", "fd00::1", false, true},
|
{"public IPv4", "1.1.1.1", false, false, false},
|
||||||
{"loopback IPv6", "::1", false, true},
|
{"private IPv6", "fd00::1", false, false, true},
|
||||||
{"public IPv6", "2606:4700:4700::1111", false, false},
|
{"loopback IPv6", "::1", false, false, true},
|
||||||
{"public IPv4 in IPv6 form", "::ffff:1.1.1.1", false, false},
|
{"public IPv6", "2606:4700:4700::1111", false, false, false},
|
||||||
{"private IPv4 in IPv6 form", "::ffff:10.0.0.1", false, true},
|
{"public IPv4 in IPv6 form", "::ffff:1.1.1.1", false, false, false},
|
||||||
|
{"private IPv4 in IPv6 form", "::ffff:10.0.0.1", false, false, true},
|
||||||
}
|
}
|
||||||
|
|
||||||
for _, tt := range tests {
|
for _, tt := range tests {
|
||||||
@@ -1604,7 +1617,8 @@ func TestIPLookupFlags(t *testing.T) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
want := map[string]bool{
|
want := map[string]bool{
|
||||||
"is_tor": tt.tor, "is_vpn": false, "is_bogon": tt.bogon,
|
"is_tor": tt.tor, "is_vpn": false, "is_relay": tt.relay,
|
||||||
|
"is_bogon": tt.bogon,
|
||||||
}
|
}
|
||||||
if !maps.Equal(got, want) {
|
if !maps.Equal(got, want) {
|
||||||
t.Errorf("%s: flags %v, want %v", path, got, want)
|
t.Errorf("%s: flags %v, want %v", path, got, want)
|
||||||
@@ -1613,6 +1627,131 @@ func TestIPLookupFlags(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// fakeGeoLocator answers every address with location, and keeps the addresses
|
||||||
|
// of each call. It is safe for the concurrent subtests that share it.
|
||||||
|
type fakeGeoLocator struct {
|
||||||
|
location *geo.Location
|
||||||
|
|
||||||
|
mu sync.Mutex
|
||||||
|
calls [][]netip.Addr
|
||||||
|
}
|
||||||
|
|
||||||
|
func (f *fakeGeoLocator) Lookup(
|
||||||
|
_ context.Context, addrs []netip.Addr,
|
||||||
|
) []*geo.Location {
|
||||||
|
f.mu.Lock()
|
||||||
|
defer f.mu.Unlock()
|
||||||
|
|
||||||
|
f.calls = append(f.calls, addrs)
|
||||||
|
|
||||||
|
answers := make([]*geo.Location, len(addrs))
|
||||||
|
for i := range addrs {
|
||||||
|
answers[i] = f.location
|
||||||
|
}
|
||||||
|
|
||||||
|
return answers
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestIPLookupGivesTheGeoObject looks up two public addresses. They go to the
|
||||||
|
// locator in one call, and each result has the geo object under the keys the
|
||||||
|
// API documents. These addresses are public, because a reserved one is not
|
||||||
|
// looked up.
|
||||||
|
func TestIPLookupGivesTheGeoObject(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
s := newServerWithTestRoutes(t)
|
||||||
|
useFakeResolver(s, &fakeResolver{})
|
||||||
|
|
||||||
|
locator := &fakeGeoLocator{location: &geo.Location{
|
||||||
|
CountryCode: new("DE"),
|
||||||
|
City: new("Berlin"),
|
||||||
|
Latitude: new(52.5),
|
||||||
|
Source: "geojs",
|
||||||
|
Updated: testRouteTime(0),
|
||||||
|
}}
|
||||||
|
s.SetGeoLocator(locator)
|
||||||
|
|
||||||
|
path := "/api/v1/ip/1.1.1.1,8.8.8.8"
|
||||||
|
|
||||||
|
results := resultFields(t, get(t, s, path).Body.Bytes())
|
||||||
|
if len(results) != 2 {
|
||||||
|
t.Fatalf("%s: %d results, want 2", path, len(results))
|
||||||
|
}
|
||||||
|
|
||||||
|
if len(locator.calls) != 1 || len(locator.calls[0]) != 2 {
|
||||||
|
t.Errorf("locator calls %v, want one with both addresses", locator.calls)
|
||||||
|
}
|
||||||
|
|
||||||
|
var got map[string]json.RawMessage
|
||||||
|
|
||||||
|
err := json.Unmarshal(results[0]["geo"], &got)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("decoding geo %s: %v", results[0]["geo"], err)
|
||||||
|
}
|
||||||
|
|
||||||
|
wantKeys := []string{
|
||||||
|
"accuracy_radius_km", "city", "continent_code", "country",
|
||||||
|
"country_code", "latitude", "longitude", "region", "source",
|
||||||
|
"time_zone", "updated",
|
||||||
|
}
|
||||||
|
if keys := slices.Sorted(maps.Keys(got)); !slices.Equal(keys, wantKeys) {
|
||||||
|
t.Errorf("geo keys %v, want %v", keys, wantKeys)
|
||||||
|
}
|
||||||
|
|
||||||
|
if string(got["city"]) != `"Berlin"` || string(got["latitude"]) != "52.5" ||
|
||||||
|
string(got["region"]) != jsonNull {
|
||||||
|
t.Errorf("wrong geo %s", results[0]["geo"])
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestIPLookupOfAReservedAddressHasNoGeo looks up a reserved address alone and
|
||||||
|
// beside a public one. The reserved one never reaches the locator, and its geo
|
||||||
|
// is null.
|
||||||
|
func TestIPLookupOfAReservedAddressHasNoGeo(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
s := newServerWithTestRoutes(t)
|
||||||
|
useFakeResolver(s, &fakeResolver{})
|
||||||
|
|
||||||
|
locator := &fakeGeoLocator{location: &geo.Location{Source: "geojs"}}
|
||||||
|
s.SetGeoLocator(locator)
|
||||||
|
|
||||||
|
results := resultFields(t, get(t, s, "/api/v1/ip/"+addrInTestPrefixV4).Body.Bytes())
|
||||||
|
if len(results) != 1 || string(results[0]["geo"]) != jsonNull {
|
||||||
|
t.Fatalf("results %v, want one with geo null", results)
|
||||||
|
}
|
||||||
|
|
||||||
|
if len(locator.calls) != 0 {
|
||||||
|
t.Errorf("locator calls %v, want none", locator.calls)
|
||||||
|
}
|
||||||
|
|
||||||
|
results = resultFields(t,
|
||||||
|
get(t, s, "/api/v1/ip/"+addrInTestPrefixV4+",1.1.1.1").Body.Bytes())
|
||||||
|
if len(results) != 2 || string(results[0]["geo"]) != jsonNull ||
|
||||||
|
string(results[1]["geo"]) == jsonNull {
|
||||||
|
t.Errorf("results %v, want geo only for the public address", results)
|
||||||
|
}
|
||||||
|
|
||||||
|
if len(locator.calls) != 1 || len(locator.calls[0]) != 1 {
|
||||||
|
t.Errorf("locator calls %v, want one with one address", locator.calls)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestIPLookupGeoIsNullWhenNoSourceAnswers checks an address the locator has
|
||||||
|
// no answer for.
|
||||||
|
func TestIPLookupGeoIsNullWhenNoSourceAnswers(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
s := newServerWithTestRoutes(t)
|
||||||
|
useFakeResolver(s, &fakeResolver{})
|
||||||
|
s.SetGeoLocator(&fakeGeoLocator{})
|
||||||
|
|
||||||
|
results := resultFields(t, get(t, s, "/api/v1/ip/1.1.1.1").Body.Bytes())
|
||||||
|
if len(results) != 1 || string(results[0]["geo"]) != jsonNull {
|
||||||
|
t.Errorf("results %v, want one with geo null", results)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
// TestIPLookupReportsVPNProviders looks up addresses on no list, on a list
|
// TestIPLookupReportsVPNProviders looks up addresses on no list, on a list
|
||||||
// that is not a VPN list, on one VPN list and on two. is_vpn says whether any
|
// that is not a VPN list, on one VPN list and on two. is_vpn says whether any
|
||||||
// VPN list holds the address, and vpn_providers names only those lists. An
|
// VPN list holds the address, and vpn_providers names only those lists. An
|
||||||
|
|||||||
@@ -275,49 +275,66 @@ func TestPagesServeTheCopyOfTheCards(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// TestKeepHomeCardsFreshWorksThemOutEveryInterval runs keepHomeCardsFresh with
|
// TestKeepHomeCardsFreshWorksThemOutEveryInterval runs keepHomeCardsFresh. It
|
||||||
// a short interval. It works the cards out at once, again after the table
|
// works the cards out at once, again an interval after the table changes, and
|
||||||
// changes, and stops when its context ends.
|
// stops when its context ends. It runs in a synctest bubble, whose clock is
|
||||||
|
// fake, so a stall on the host cannot fail it.
|
||||||
func TestKeepHomeCardsFreshWorksThemOutEveryInterval(t *testing.T) {
|
func TestKeepHomeCardsFreshWorksThemOutEveryInterval(t *testing.T) {
|
||||||
t.Parallel()
|
t.Parallel()
|
||||||
|
|
||||||
const (
|
const interval = time.Minute
|
||||||
interval = 10 * time.Millisecond
|
|
||||||
maxWait = 15 * time.Second
|
|
||||||
)
|
|
||||||
|
|
||||||
table := emptyTable()
|
table := emptyTable()
|
||||||
announce(table, "1.0.0.0/24", twoPrefixASN, testPeer(1), testRouteTime(0))
|
announce(table, "1.0.0.0/24", twoPrefixASN, testPeer(1), testRouteTime(0))
|
||||||
|
|
||||||
s := newServerWithTable(t, table)
|
s := newServerWithTable(t, table)
|
||||||
|
|
||||||
ctx, cancel := context.WithCancel(t.Context())
|
// newestRouteOrigin returns the origin AS of the newest route on the
|
||||||
stopped := make(chan struct{})
|
// cards, or 0 when there is none.
|
||||||
|
newestRouteOrigin := func() int {
|
||||||
go func() {
|
|
||||||
s.keepHomeCardsFresh(ctx, interval)
|
|
||||||
close(stopped)
|
|
||||||
}()
|
|
||||||
|
|
||||||
newestRouteASNIs := func(asn int) bool {
|
|
||||||
cards := s.cards.Load()
|
cards := s.cards.Load()
|
||||||
|
if cards == nil || cards.NewestRoute == nil {
|
||||||
|
return 0
|
||||||
|
}
|
||||||
|
|
||||||
return cards != nil && cards.NewestRoute != nil &&
|
return cards.NewestRoute.OriginASN
|
||||||
cards.NewestRoute.OriginASN == asn
|
|
||||||
}
|
}
|
||||||
|
|
||||||
waitUntil(t, maxWait, func() bool { return newestRouteASNIs(twoPrefixASN) })
|
synctest.Test(t, func(t *testing.T) {
|
||||||
|
ctx, cancel := context.WithCancel(t.Context())
|
||||||
|
stopped := make(chan struct{})
|
||||||
|
|
||||||
announce(table, "8.8.8.0/24", newestRouteASN, testPeer(2), testRouteTime(1))
|
go func() {
|
||||||
waitUntil(t, maxWait, func() bool { return newestRouteASNIs(newestRouteASN) })
|
s.keepHomeCardsFresh(ctx, interval)
|
||||||
|
close(stopped)
|
||||||
|
}()
|
||||||
|
|
||||||
cancel()
|
synctest.Wait()
|
||||||
|
|
||||||
select {
|
if got := newestRouteOrigin(); got != twoPrefixASN {
|
||||||
case <-stopped:
|
t.Fatalf("at the start, the newest route's AS is %d, want %d",
|
||||||
case <-time.After(maxWait):
|
got, twoPrefixASN)
|
||||||
t.Fatal("keepHomeCardsFresh did not stop when its context ended")
|
}
|
||||||
}
|
|
||||||
|
announce(table, "8.8.8.0/24", newestRouteASN, testPeer(2),
|
||||||
|
testRouteTime(1))
|
||||||
|
time.Sleep(interval)
|
||||||
|
synctest.Wait()
|
||||||
|
|
||||||
|
if got := newestRouteOrigin(); got != newestRouteASN {
|
||||||
|
t.Fatalf("an interval after the table changed, the newest "+
|
||||||
|
"route's AS is %d, want %d", got, newestRouteASN)
|
||||||
|
}
|
||||||
|
|
||||||
|
cancel()
|
||||||
|
synctest.Wait()
|
||||||
|
|
||||||
|
select {
|
||||||
|
case <-stopped:
|
||||||
|
default:
|
||||||
|
t.Fatal("keepHomeCardsFresh did not stop when its context ended")
|
||||||
|
}
|
||||||
|
})
|
||||||
}
|
}
|
||||||
|
|
||||||
// TestStartedServerWorksOutTheCards starts a server and waits until its home
|
// TestStartedServerWorksOutTheCards starts a server and waits until its home
|
||||||
|
|||||||
@@ -24,6 +24,7 @@ import (
|
|||||||
|
|
||||||
"git.eeqj.de/sneak/routewatch/internal/config"
|
"git.eeqj.de/sneak/routewatch/internal/config"
|
||||||
"git.eeqj.de/sneak/routewatch/internal/database"
|
"git.eeqj.de/sneak/routewatch/internal/database"
|
||||||
|
"git.eeqj.de/sneak/routewatch/internal/geo"
|
||||||
"git.eeqj.de/sneak/routewatch/internal/iplist"
|
"git.eeqj.de/sneak/routewatch/internal/iplist"
|
||||||
"git.eeqj.de/sneak/routewatch/internal/logger"
|
"git.eeqj.de/sneak/routewatch/internal/logger"
|
||||||
"git.eeqj.de/sneak/routewatch/internal/routetable"
|
"git.eeqj.de/sneak/routewatch/internal/routetable"
|
||||||
@@ -63,6 +64,12 @@ type PTRResolver interface {
|
|||||||
LookupAddr(ctx context.Context, addr string) ([]string, error)
|
LookupAddr(ctx context.Context, addr string) ([]string, error)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// GeoLocator finds where addresses are, as *geo.Locator does. It returns one
|
||||||
|
// entry for each address, nil when no source answers.
|
||||||
|
type GeoLocator interface {
|
||||||
|
Lookup(ctx context.Context, addrs []netip.Addr) []*geo.Location
|
||||||
|
}
|
||||||
|
|
||||||
// Server provides HTTP endpoints for status monitoring
|
// Server provides HTTP endpoints for status monitoring
|
||||||
type Server struct {
|
type Server struct {
|
||||||
router *chi.Mux
|
router *chi.Mux
|
||||||
@@ -86,6 +93,10 @@ type Server struct {
|
|||||||
// ptrResolver looks up the IP lookup's PTR names. SetPTRResolver sets it.
|
// ptrResolver looks up the IP lookup's PTR names. SetPTRResolver sets it.
|
||||||
ptrResolver PTRResolver
|
ptrResolver PTRResolver
|
||||||
|
|
||||||
|
// geoLocator finds where the IP lookup's addresses are. SetGeoLocator
|
||||||
|
// sets it.
|
||||||
|
geoLocator GeoLocator
|
||||||
|
|
||||||
// cards is the copy of the home page's cards that the pages serve, nil
|
// cards is the copy of the home page's cards that the pages serve, nil
|
||||||
// until the server first works them out
|
// until the server first works them out
|
||||||
cards atomic.Pointer[homeCards]
|
cards atomic.Pointer[homeCards]
|
||||||
@@ -209,6 +220,12 @@ func (s *Server) SetPTRResolver(resolver PTRResolver) {
|
|||||||
s.ptrResolver = resolver
|
s.ptrResolver = resolver
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// SetGeoLocator sets what finds where the IP lookup's addresses are. Call it
|
||||||
|
// before Start: the handlers read it without a lock.
|
||||||
|
func (s *Server) SetGeoLocator(locator GeoLocator) {
|
||||||
|
s.geoLocator = locator
|
||||||
|
}
|
||||||
|
|
||||||
// SetASNFetcher sets the ASN WHOIS fetcher for on-demand lookups. Call it
|
// SetASNFetcher sets the ASN WHOIS fetcher for on-demand lookups. Call it
|
||||||
// before Start: the handlers read the fetcher without a lock.
|
// before Start: the handlers read the fetcher without a lock.
|
||||||
func (s *Server) SetASNFetcher(fetcher ASNFetcher) {
|
func (s *Server) SetASNFetcher(fetcher ASNFetcher) {
|
||||||
|
|||||||
@@ -544,6 +544,8 @@
|
|||||||
<span>{{appLicense}}</span>
|
<span>{{appLicense}}</span>
|
||||||
<span class="separator">|</span>
|
<span class="separator">|</span>
|
||||||
<span><a href="{{appGitCommitURL}}">{{appGitRevision}}</a></span>
|
<span><a href="{{appGitCommitURL}}">{{appGitRevision}}</a></span>
|
||||||
|
<span class="separator">|</span>
|
||||||
|
<span>IP Geolocation by <a href="https://db-ip.com">DB-IP</a></span>
|
||||||
</footer>
|
</footer>
|
||||||
|
|
||||||
<script nonce="{{.CSPNonce}}">
|
<script nonce="{{.CSPNonce}}">
|
||||||
|
|||||||
@@ -67,6 +67,23 @@ func TestFooterLicense(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// TestHomePageCreditsDBIP expects the home page's footer to carry the credit
|
||||||
|
// to DB-IP.
|
||||||
|
func TestHomePageCreditsDBIP(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
const attribution = `IP Geolocation by <a href="https://db-ip.com">DB-IP</a>`
|
||||||
|
|
||||||
|
_, footer, found := strings.Cut(renderPage(t, templates.IndexTemplate()), "<footer")
|
||||||
|
if !found {
|
||||||
|
t.Fatal("the page has no footer")
|
||||||
|
}
|
||||||
|
|
||||||
|
if !strings.Contains(footer, attribution) {
|
||||||
|
t.Errorf("footer does not credit DB-IP: %q", footer)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
// TestFreshnessIndicator renders the home and status pages and expects each to
|
// TestFreshnessIndicator renders the home and status pages and expects each to
|
||||||
// have, right after the status element, the element where the page's script
|
// have, right after the status element, the element where the page's script
|
||||||
// shows its spinner or "stale".
|
// shows its spinner or "stale".
|
||||||
|
|||||||
Reference in new issue
Block a user