Compare commits
7
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
 sneak
|
cddae7fd68 | ||
 clawbot
|
760ba4746b | ||
|
clawbot
|
dcb3935643 | ||
|
clawbot
|
6de44ac054 | ||
|
clawbot
|
cca402c73d | ||
|
clawbot
|
2c3f6f71dc | ||
|
clawbot
|
b2553f044d |
No files matched your search
+1
-1
@@ -105,7 +105,7 @@ RUN apt-get update && apt-get install -y --no-install-recommends \
|
||||
ARG CACHEBUST
|
||||
|
||||
# Create non-root user
|
||||
RUN useradd -r -u 1000 -m routewatch
|
||||
RUN useradd -u 1000 -m routewatch
|
||||
|
||||
RUN mkdir -p /var/lib/berlin.sneak.app.routewatch && chown routewatch:routewatch /var/lib/berlin.sneak.app.routewatch
|
||||
|
||||
|
||||
@@ -63,8 +63,8 @@ docker run --memory=5g --memory-swap=5g -p 8080:8080 \
|
||||
-v routewatch:/var/lib/berlin.sneak.app.routewatch routewatch
|
||||
```
|
||||
|
||||
The memory limit is explained in Memory. The volume holds the database and the
|
||||
copies of the IP lists.
|
||||
The memory limit is explained in Memory. The volume holds the database, the
|
||||
copies of the IP lists and the geolocation files.
|
||||
|
||||
To run the binary on the host instead, build it with `make build`. The build
|
||||
needs Go and a C compiler, because the SQLite library is written in C:
|
||||
@@ -143,9 +143,9 @@ and origin AS can have two kept routes, one or none.
|
||||
AS path, a next hop and a time. The next hop is the router the peer sends the
|
||||
prefix's traffic to. RouteWatch records the peer itself as the next hop. The
|
||||
page also lists the prefix's origin changes, newest first. An origin change is
|
||||
an AS starting to announce the prefix, when its first peer carries it, or
|
||||
stopping, when its last peer no longer does. The page shows at most the newest
|
||||
5,000. Each row shows:
|
||||
recorded when the first peer starts to carry the prefix with an origin AS, or
|
||||
when the last peer stops. The page shows at most the newest 5,000. Each row
|
||||
shows:
|
||||
- the time
|
||||
- announced or withdrawn
|
||||
- the origin AS
|
||||
@@ -197,8 +197,9 @@ also carries an `@meta` object:
|
||||
- `api_version`: `1`
|
||||
- `execution_time_ms`: the number of milliseconds the request took
|
||||
|
||||
Every value with a unit is a number, and the unit ends the field name. Examples
|
||||
are `_ms`, `_seconds`, `_bytes`, `_per_sec` and `_per_min`.
|
||||
Every value with a unit is a number. The unit ends the field name, as in `_ms`,
|
||||
`_seconds`, `_bytes`, `_per_sec` and `_per_min`. `latitude` and `longitude` are
|
||||
in degrees, and their names do not say so.
|
||||
|
||||
The paths:
|
||||
|
||||
@@ -287,12 +288,14 @@ address. Every entry has the same fields:
|
||||
- `is_tor`: true when the address is on the `tor_exits` list
|
||||
- `is_vpn`: true when the address is on any list of VPN servers (see IP
|
||||
lists)
|
||||
- `is_relay`: true when the address is on the `icloud_private_relay` list
|
||||
- `is_bogon`: true when the address is reserved, so that no network should
|
||||
announce it on the internet
|
||||
- `vpn_providers`: the names of the lists of VPN servers the address is on, or
|
||||
an empty list when it is on none
|
||||
- `ip_lists`: the public IP lists the address is on (see IP lists), or an empty
|
||||
list when it is on none
|
||||
- `geo`: where the address is (see Geolocation), or null when no source answers
|
||||
|
||||
Every other field is about the prefix in the routing table that holds the
|
||||
address. Each is null when no prefix in the routing table holds the address:
|
||||
@@ -421,6 +424,9 @@ The lists:
|
||||
last 30 days. From
|
||||
[FireHOL](https://iplists.firehol.org/?ipset=firehol_level3), downloaded every
|
||||
3 hours.
|
||||
- `icloud_private_relay`: the networks where iCloud Private Relay traffic
|
||||
leaves. From [Apple](https://mask-api.icloud.com/egress-ip-ranges.csv),
|
||||
downloaded once a day.
|
||||
- `ivpn`: VPN servers run by IVPN. From
|
||||
[IVPN](https://api.ivpn.net/v5/servers.json), downloaded once a day.
|
||||
- `mullvad`: VPN servers run by Mullvad. From
|
||||
@@ -431,6 +437,14 @@ The lists:
|
||||
- `pia`: VPN servers run by Private Internet Access. From
|
||||
[Private Internet Access](https://serverlist.piaservers.net/vpninfo/servers/v6),
|
||||
downloaded once a day.
|
||||
- `protonvpn`: the /24 networks that hold ProtonVPN servers. The whole /24 is
|
||||
listed because the ProtonVPN-IPs README says traffic leaves from unpublished
|
||||
addresses in it. From the public
|
||||
[ProtonVPN-IPs](https://github.com/tn3w/ProtonVPN-IPs) mirror of Proton's
|
||||
server list, downloaded once a day.
|
||||
- `protonvpn_entry`: the IPv4 and IPv6 addresses of ProtonVPN servers. From the
|
||||
same [ProtonVPN-IPs](https://github.com/tn3w/ProtonVPN-IPs) mirror, downloaded
|
||||
once a day.
|
||||
- `spamhaus_drop` and `spamhaus_dropv6`: IPv4 and IPv6 networks hijacked or
|
||||
leased by spammers or cyber criminals, according to Spamhaus. From
|
||||
[Spamhaus DROP](https://www.spamhaus.org/blocklists/do-not-route-or-peer/),
|
||||
@@ -489,11 +503,13 @@ How RouteWatch keeps the answers:
|
||||
there.
|
||||
- It writes new answers every 5 seconds to `cache.db` in the state directory
|
||||
(see Configuration), and loads them from there at startup.
|
||||
- An answer expires after the TTL of its records, the shortest one when there
|
||||
are several. RouteWatch keeps it for at least 60 seconds and at most 24 hours.
|
||||
- When an address has no PTR names, that answer expires after the SOA record's
|
||||
minimum field, or the record's own TTL when that is shorter. RouteWatch keeps
|
||||
it for at most 1 hour.
|
||||
- Each DNS record carries a TTL, the time for which it may be kept. An answer
|
||||
expires after the shortest TTL of its records. RouteWatch keeps it for at
|
||||
least 60 seconds and at most 24 hours.
|
||||
- When an address has no PTR names, the answer's SOA record says how long that
|
||||
answer may be kept. The answer expires after the SOA record's minimum field,
|
||||
or the record's own TTL when that is shorter. RouteWatch keeps it for at most
|
||||
1 hour.
|
||||
- An answer with no PTR names and no SOA record is not kept. Neither is one
|
||||
whose SOA record has a TTL or minimum field of 0.
|
||||
- A failed lookup is not kept, so the next lookup of the address asks again.
|
||||
@@ -501,6 +517,73 @@ How RouteWatch keeps the answers:
|
||||
- It keeps answers for at most 250,000 addresses, and keeps no answer for a new
|
||||
address while it holds that many.
|
||||
|
||||
## Geolocation
|
||||
|
||||
An IP lookup finds where each address is, on a best-effort basis. It uses two
|
||||
sources:
|
||||
|
||||
- [geojs](https://www.geojs.io/) is asked first. One request carries every
|
||||
address of the lookup that the cache does not hold. geojs has 1 second to
|
||||
answer.
|
||||
- [DB-IP City Lite](https://db-ip.com/db/download/ip-to-city-lite) answers when
|
||||
geojs has no answer for the address. It is a database file that RouteWatch
|
||||
reads itself.
|
||||
|
||||
geojs has no answer when its request fails or runs out of time. An answer with
|
||||
no country, city or coordinates also counts as no answer, and RouteWatch does
|
||||
not cache it.
|
||||
|
||||
The `geo` object of a result has these fields:
|
||||
|
||||
- `country_code`: the two-letter country code
|
||||
- `country`: the country's name in English
|
||||
- `continent_code`: the two-letter continent code
|
||||
- `region`: the region or state
|
||||
- `city`: the city
|
||||
- `latitude` and `longitude`: the location in degrees
|
||||
- `accuracy_radius_km`: how far from that location the address may be, in km
|
||||
- `time_zone`: the time zone's name, such as `Europe/Berlin`
|
||||
- `source`: the source of the answer, `geojs` or `dbip`
|
||||
- `updated`: when the answer was fetched from geojs, or when the database copy
|
||||
was downloaded
|
||||
|
||||
A field the source does not know is null. DB-IP has no `accuracy_radius_km` and
|
||||
no `time_zone`, so both are null in its answers. `geo` itself is null when
|
||||
neither source answers. A reserved address, one whose `is_bogon` flag is true,
|
||||
goes to neither source, so its `geo` is null.
|
||||
|
||||
What goes outside: geojs receives each address that the cache does not hold. For
|
||||
`/ip/` and the home page, that address is the visitor's own. A lookup answered
|
||||
from the cache sends nothing. DB-IP receives only the monthly request for its
|
||||
file.
|
||||
|
||||
How RouteWatch keeps the answers of geojs:
|
||||
|
||||
- It holds at most 250,000 addresses in memory, and drops the least recently
|
||||
used first. That takes about 100 MB of the Go heap.
|
||||
- It writes new answers every 5 seconds to `cache.db` in the state directory.
|
||||
- It loads the newest 250,000 answers from `cache.db` at startup.
|
||||
- An address not in memory is read from `cache.db` before geojs is asked.
|
||||
- An answer expires 7 days after its fetch. Every 10 minutes RouteWatch deletes
|
||||
expired answers from memory and from `cache.db`.
|
||||
|
||||
Answers from the database are not cached, because reading the file is local.
|
||||
|
||||
How RouteWatch keeps the database:
|
||||
|
||||
- It keeps one copy in the state directory, named `dbip-city-lite.mmdb`.
|
||||
- It downloads the current month's file at the first start, and then once a
|
||||
month. Until the first download finishes, only geojs answers.
|
||||
- A download fails when it takes over 10 minutes or unpacks to over 200 MiB.
|
||||
- A new file replaces the old copy only once it opens. After a failed download
|
||||
RouteWatch keeps the old copy and tries again an hour later.
|
||||
- The file is about 130 MB. RouteWatch maps it into memory, outside the Go heap.
|
||||
The kernel loads the parts that are read, and can drop them again when memory
|
||||
is short.
|
||||
|
||||
The home page's footer says "IP Geolocation by DB-IP" with a link to
|
||||
[db-ip.com](https://db-ip.com).
|
||||
|
||||
## Configuration
|
||||
|
||||
RouteWatch reads its settings from environment variables. It takes no
|
||||
@@ -552,7 +635,9 @@ are not read.
|
||||
The state directory holds:
|
||||
|
||||
- `db.sqlite`: the database
|
||||
- `cache.db`: the answers of PTR lookups
|
||||
- `cache.db`: the answers of PTR lookups and of geojs
|
||||
- `dbip-city-lite.mmdb`: the copy of DB-IP's City Lite database (see
|
||||
Geolocation)
|
||||
- `iplists`: the IP list copies
|
||||
|
||||
It is at:
|
||||
@@ -577,8 +662,9 @@ a memory limit of 5 GiB and swap disabled:
|
||||
docker run --memory=5g --memory-swap=5g ...
|
||||
```
|
||||
|
||||
The 5 GiB leaves room above the ceilings for spikes and for the kernel's cache
|
||||
of the database file.
|
||||
The 5 GiB leaves room above the ceilings for spikes, for the kernel's cache of
|
||||
the database file, and for the mapped copy of DB-IP's database (see
|
||||
Geolocation).
|
||||
|
||||
The ceilings add up to 3.7 GiB:
|
||||
|
||||
@@ -598,6 +684,13 @@ The ceilings add up to 3.7 GiB:
|
||||
limit caps the SQLite library.
|
||||
- About 0.2 GiB for everything else in the runtime.
|
||||
|
||||
The `icloud_private_relay` list has about 285,000 networks. Neighbouring
|
||||
networks merge, which leaves about 6,900 address ranges. Loaded, the list adds
|
||||
about 0.3 MiB to the Go heap and under 6 MiB to the process's resident memory.
|
||||
Parsing the file allocates about 94 MiB in about 0.2 seconds. The next garbage
|
||||
collection frees all of it but the 0.3 MiB above. Parsing happens at startup and
|
||||
whenever a download brings a new version.
|
||||
|
||||
The SQLite library gets its memory from glibc, the C library. glibc keeps freed
|
||||
memory in pools called arenas, and hands it back to the kernel slowly. It
|
||||
creates up to eight arenas per processor core. So on a host with many cores the
|
||||
@@ -613,8 +706,10 @@ What happens at each limit:
|
||||
allocating more.
|
||||
- The SQLite hard heap limit: a database statement fails with an out-of-memory
|
||||
error, which the handler logs. The prefix handler writes the routing table's
|
||||
changes again in a later write. Everything else in the failed write is
|
||||
dropped. The process keeps running.
|
||||
changes and its origin changes again in a later write. Everything else in a
|
||||
handler's failed write is dropped. The PTR cache and the geolocation part
|
||||
write their answers to `cache.db` again in a later write. The process keeps
|
||||
running.
|
||||
- The handler queues: the handlers are the four parts that take the feed's
|
||||
messages (see Design). The prefix handler's queue holds at most 100,000
|
||||
messages, about 100 MiB. Each of the other three holds at most 20,000. When a
|
||||
@@ -714,6 +809,8 @@ The parts:
|
||||
address.
|
||||
- The PTR cache looks up the PTR names of addresses and keeps the answers (see
|
||||
PTR lookups).
|
||||
- The geolocation part finds where addresses are, keeps the answers of geojs and
|
||||
downloads DB-IP's database (see Geolocation).
|
||||
- The database maintainer checkpoints the database every 5 seconds: it folds the
|
||||
write-ahead log, where SQLite first writes changes, into the database file. It
|
||||
reclaims free space every 10 minutes. Every hour it refreshes the statistics
|
||||
@@ -823,7 +920,8 @@ On `SIGTERM` or `SIGINT` the daemon stops its parts in this order:
|
||||
4. the database maintainer
|
||||
5. the IP list downloads
|
||||
6. the HTTP server
|
||||
7. the PTR cache, which writes what it holds
|
||||
7. the geolocation part, which writes the geojs answers it holds
|
||||
8. the PTR cache, which writes what it holds
|
||||
|
||||
Last, the daemon checkpoints and closes the database. It gives all this 8
|
||||
seconds and then exits with status 0. A stop that is still running after 9
|
||||
@@ -846,6 +944,8 @@ What a stop can lose:
|
||||
SQLite rolls back any write the exit cut off.
|
||||
- When the 8 seconds run out, the PTR answers not yet in `cache.db` are lost.
|
||||
RouteWatch asks the resolvers for them again the next time they are needed.
|
||||
- When the 8 seconds run out, the geojs answers not yet in `cache.db` are lost.
|
||||
RouteWatch asks geojs for them again the next time they are needed.
|
||||
|
||||
### Code layout
|
||||
|
||||
@@ -862,6 +962,7 @@ routewatch/
|
||||
│ ├── cachedb/ # cache.db, the caches of outside answers
|
||||
│ ├── config/ # Settings from the environment
|
||||
│ ├── database/ # SQLite storage
|
||||
│ ├── geo/ # Geolocation: geojs, DB-IP City Lite, the cache
|
||||
│ ├── iplist/ # Public IP lists: downloads and lookups
|
||||
│ ├── logger/ # Structured logging
|
||||
│ ├── metrics/ # Feed and route update figures
|
||||
@@ -914,11 +1015,16 @@ announcements(id PRIMARY KEY, prefix_id, peer_asn, origin_asn, path,
|
||||
|
||||
The indexes are in `internal/database/schema.sql`.
|
||||
|
||||
`cache.db` is a separate file, with one table:
|
||||
`cache.db` is a separate file, with two tables:
|
||||
|
||||
```sql
|
||||
-- The answers of PTR lookups. names is empty for an address with no PTR names.
|
||||
hostnames(address PRIMARY KEY, names, expires_at)
|
||||
|
||||
-- The answers of geojs. A part geojs does not know is NULL.
|
||||
geolocations(address PRIMARY KEY, country_code, country, continent_code,
|
||||
region, city, latitude, longitude, accuracy_radius_km, time_zone,
|
||||
fetched_at)
|
||||
```
|
||||
|
||||
## Development
|
||||
@@ -970,8 +1076,8 @@ calls one of them. The scripts:
|
||||
- `script/projectname`: print the project name, `routewatch`, which
|
||||
`script/docker` uses as the image tag.
|
||||
- `script/test`: build the `test` phase of the `Dockerfile` alone. It runs
|
||||
`go test -short -count=1 -timeout 90s -race -cover ./...`, and reruns it with
|
||||
`-v` on failure.
|
||||
`go test -short -count=1 -timeout 90s -race -cover ./...`. On failure it runs
|
||||
the tests again with `-v` and without `-cover`.
|
||||
- `script/lint`: build the `lint` phase of the `Dockerfile` alone. It runs
|
||||
`golangci-lint` v2.14.0 with `.golangci.yml`.
|
||||
- `script/fmt`: format the Go code with `go fmt` and the markdown with
|
||||
@@ -994,8 +1100,7 @@ calls one of them. The scripts:
|
||||
|
||||
## TODO
|
||||
|
||||
`TODO.md` holds the working list: the next step, the steps done and the steps
|
||||
planned.
|
||||
`TODO.md` holds the working list: the next step and the steps planned.
|
||||
|
||||
## License
|
||||
|
||||
|
||||
@@ -10,452 +10,24 @@
|
||||
|
||||
# Status
|
||||
|
||||
RouteWatch is pre-1.0, with no git tags. `main` last took `next` on 2026-10-27,
|
||||
through https://git.eeqj.de/sneak/routewatch/pulls/166.
|
||||
https://git.eeqj.de/sneak/routewatch/pulls/202 carries the commits since then
|
||||
into `main`. It is labelled `merge-ready` and waits for sneak.
|
||||
|
||||
The linter and the tests are phases of the Docker build. The Gitea workflow runs
|
||||
`script/cibuild` on every push. That script runs the two phases and checks the
|
||||
formatting of the Go code and the markdown. Then it builds the image.
|
||||
|
||||
The image sets memory ceilings for a 5 GiB container (README "Memory"). README
|
||||
"Running under upaas" says how to run it.
|
||||
|
||||
A five-day run of `1c65656` on the live feed, from 2026-10-27 to 2026-11-01,
|
||||
found three problems:
|
||||
|
||||
- The prefix handler dropped updates while a peer's session went down
|
||||
(https://git.eeqj.de/sneak/routewatch/issues/216).
|
||||
- The AS handler dropped updates when the feed burst
|
||||
(https://git.eeqj.de/sneak/routewatch/issues/217).
|
||||
- The WHOIS fetcher kept asking a registry that had refused it
|
||||
(https://git.eeqj.de/sneak/routewatch/issues/218).
|
||||
|
||||
All three are fixed on `next`. So is
|
||||
https://git.eeqj.de/sneak/routewatch/issues/222, which the third fix brought up.
|
||||
A run of `26d939a` on the live feed showed the prefix handler still dropping
|
||||
updates while the home page's cards were worked out.
|
||||
https://git.eeqj.de/sneak/routewatch/issues/226 fixed that on `next`.
|
||||
https://git.eeqj.de/sneak/routewatch/issues/225, also on `next`, takes the peers
|
||||
whose sessions go down together out of the table in one walk. A 24-hour run of
|
||||
`9bd0e91` on the live feed is checking whether any handler still drops updates.
|
||||
|
||||
The order of the project's remaining work is kept on
|
||||
RouteWatch is pre-1.0. The order of the remaining work is kept on
|
||||
https://git.eeqj.de/sneak/routewatch/issues/86.
|
||||
|
||||
# Next Step
|
||||
|
||||
Post the outcome of the 24-hour run of `9bd0e91` on
|
||||
https://git.eeqj.de/sneak/routewatch/issues/226: whether any handler still drops
|
||||
updates, and the process's peak memory against 5 GiB. File an issue for each
|
||||
problem the run finds.
|
||||
Post the outcome of the run of `e58c663` on the live feed on
|
||||
https://git.eeqj.de/sneak/routewatch/issues/232. The run began at 09:19 UTC on
|
||||
2026-11-10 and lasts at least 24 hours. If the feed has not stalled or
|
||||
reconnected by then, the run goes on until it does, up to 72 hours in all. The
|
||||
outcome says:
|
||||
|
||||
# Completed Steps
|
||||
- whether the feed stalled or reconnected
|
||||
- whether any handler dropped an update, and the most messages the prefix
|
||||
handler's queue held at once
|
||||
- the most messages the prefix handler held back at once
|
||||
- the size of the `origin_changes` table on disk
|
||||
- the process's peak memory against 5 GiB
|
||||
|
||||
- 2026-11-10: the IP lookup finds PTR names over DNS-over-HTTPS and caches them
|
||||
for the record's TTL, in memory and in `cache.db` (closes
|
||||
https://git.eeqj.de/sneak/routewatch/issues/238).
|
||||
- 2026-11-10: the tests no longer fail when a busy host stalls for several
|
||||
seconds. Most such waits are now 15 seconds. The test of the stream's idle
|
||||
timeout runs on a fake clock (closes #240)
|
||||
- 2026-11-10: the IP lookup checks the public VPN server lists of Mullvad,
|
||||
NordVPN, Private Internet Access and IVPN, and X4BNet's community list. Each
|
||||
result has `flags.is_vpn` and `vpn_providers` (closes
|
||||
https://git.eeqj.de/sneak/routewatch/issues/235).
|
||||
- 2026-11-10: `tor_exits` comes from Onionoo, so it holds IPv6 exits and
|
||||
`is_tor` is true for them (closes
|
||||
https://git.eeqj.de/sneak/routewatch/issues/234).
|
||||
- 2026-11-10: each result of the IP lookup has a `flags` object with `is_tor`
|
||||
and `is_bogon`. The public IP lists are named `ip_lists` in the answer,
|
||||
`internal/iplist` in the code and `iplists` in the state directory. A list
|
||||
file may be up to 200 MiB (closes
|
||||
https://git.eeqj.de/sneak/routewatch/issues/233).
|
||||
- 2026-11-05: the prefix page lists the prefix's origin changes, newest first.
|
||||
The routing table records each change, and the prefix handler writes them to a
|
||||
new `origin_changes` table. The database maintainer prunes each prefix to its
|
||||
newest 500 rows or its rows from the last 72 hours, whichever keeps more. The
|
||||
page shows at most the newest 5,000. Each row shows:
|
||||
- the time
|
||||
- announced or withdrawn
|
||||
- the origin AS
|
||||
- the peer count
|
||||
|
||||
A row's peer count is how many peers carry the prefix just after the change,
|
||||
with any origin AS (closes #161).
|
||||
|
||||
- 2026-11-03: the streamer drops an update for a handler only when that
|
||||
handler's queue is full. It used to drop a random share of the updates once
|
||||
the queue was half full. The prefix handler's queue now holds 100,000
|
||||
messages, about 100 MiB. It used to hold 20,000. While the routing table's
|
||||
`Overview` walks every prefix, the prefix handler applies updates more slowly.
|
||||
In a run on the live feed, the feed paused during such a walk and then sent
|
||||
its backlog. The streamer then dropped about 10,000 of the prefix handler's
|
||||
updates (closes #229)
|
||||
- 2026-11-02: `README.md` is rewritten from scratch, sentence by sentence,
|
||||
against the rules of https://git.eeqj.de/sneak/routewatch/issues/157. Every
|
||||
statement was checked against the code on `next` (closes #157)
|
||||
- 2026-11-02: `TODO.md` is rewritten from scratch, sentence by sentence, against
|
||||
the rules of https://git.eeqj.de/sneak/routewatch/issues/157. Every Completed
|
||||
Steps entry keeps its date, its facts and its issue. Status and Next Step
|
||||
describe the repo as the tracker shows it (part of #157)
|
||||
- 2026-11-02: when the sessions of several peers go down, the prefix handler
|
||||
takes the peers out of the routing table together. The table's `PeersDown`
|
||||
takes several peers out of every prefix in one walk. One walk runs at a time.
|
||||
The next one takes every peer whose session went down since the previous one
|
||||
began. Each peer used to have a walk of its own. A peer's messages stayed held
|
||||
back while the walks queued ahead of its own ran. The handler holds back at
|
||||
most 150,000 messages, about 150 MiB. It used to hold back at most 20,000. The
|
||||
status page shows how many messages the handler holds back (closes #225)
|
||||
- 2026-11-02: working out the home page's cards no longer stops the prefix
|
||||
handler. The walk over the routing table holds the table's lock for 10,000
|
||||
prefixes at a time, then lets the lock go for as long again, as `PeerDown`
|
||||
does. It used to take the lock back at once. The prefix handler then applied
|
||||
about one update per 10,000 prefixes walked. During a walk, its queue could
|
||||
fill, and the streamer then dropped updates for it (closes #226)
|
||||
- 2026-11-02: A skipped WHOIS lookup now makes the fetcher wait longer before
|
||||
the next one, as a failed lookup does. A lookup is skipped when its RIR is
|
||||
paused. An RIR is one of the five regional Internet registries. The longest
|
||||
wait is one minute. The status page does not count a skip as a failure. A skip
|
||||
used to leave the wait as it was. Each lookup asks IANA which RIR holds the AS
|
||||
before it finds that RIR paused. So while most stale ASes belonged to a paused
|
||||
RIR, the fetcher could ask IANA every second (closes #222)
|
||||
- 2026-11-02: A peer's session going down no longer stops the prefix handler.
|
||||
The handler runs the routing table's `PeerDown` on a goroutine of its own, one
|
||||
peer at a time. Meanwhile it goes on taking messages. `PeerDown` holds the
|
||||
table's lock for 10,000 prefixes at a time, then lets the lock go for as long
|
||||
again. The handler holds back the down peer's later messages until that peer's
|
||||
`PeerDown` returns. Then the handler applies them in order, one at a time, and
|
||||
takes new messages in between. It holds back at most 20,000 messages. At that
|
||||
number, `HandleMessage` waits until one of them is applied. When `Stop` runs
|
||||
past its deadline, it logs how many held-back messages were not applied
|
||||
(closes #216)
|
||||
- 2026-11-02: RouteWatch pauses an RIR's or NIC.br's WHOIS server for an hour
|
||||
after the server refuses a query. NIC.br is Brazil's registry. A refused or
|
||||
reset connection is a refusal. So is an answer with an `%ERROR:201:` or
|
||||
`%ERROR:202:` line. Lookups at the other servers go on. An AS whose RIR is
|
||||
paused is looked up after the pause. While NIC.br is paused, a Brazilian AS
|
||||
gets the details from LACNIC's answer. IANA is never paused, because a pause
|
||||
of IANA would stop every lookup. A refusal by IANA is an ordinary failure.
|
||||
NIC.br is also paused for an hour after it says RouteWatch has passed its
|
||||
query limit. Before, the fetcher asked a server again within seconds of a
|
||||
refusal (closes #218)
|
||||
- 2026-11-02: The AS handler never waits for a database write. Its batch holds
|
||||
one entry per AS, with the earliest and the latest time of its updates. So the
|
||||
batch no longer grows with the feed's rate. It used to hold one entry for each
|
||||
AS of each path. At 300,000 entries it waited for the write in progress. In a
|
||||
burst of the feed its queue then filled, and the streamer dropped messages for
|
||||
it (closes #217)
|
||||
- 2026-10-27: The tests of `internal/routewatch` no longer fail on a loaded
|
||||
host. Five of them gave `HandleMessage` two seconds to apply a full prefix
|
||||
batch. Under the race detector that can take longer. A test now waits up to 15
|
||||
seconds for each step it expects. The tests of the handlers' limits now give
|
||||
the handlers small limits. So the package no longer runs past its 90-second
|
||||
limit (closes #213, #215)
|
||||
- 2026-10-27: The server works out the home page's cards every minute until it
|
||||
stops. It used to work them out only once, at startup. The end of the context
|
||||
given to `Start` stopped the refreshes (closes #211)
|
||||
- 2026-10-27: README "Rationale" says why RouteWatch exists. RouteWatch answers
|
||||
our other services' questions about IP addresses, from its own routing table
|
||||
and the WHOIS details it keeps. The section still says what RIS Live does not
|
||||
give (closes #204)
|
||||
- 2026-10-27: Every JSON answer carries `@meta`. A panic on a JSON route answers
|
||||
500 with the JSON error body. `/status.json` and the health check answer 405
|
||||
with the JSON error body to any method they do not take. The 500 and the 405
|
||||
used to have an empty body. The home page's IP lookup shows the whole JSON
|
||||
answer, `@meta` included. It used to show only the answer's `data`, or only
|
||||
the message of an error answer. A test checks `@meta` on every JSON route, in
|
||||
its successes and its errors (closes #201)
|
||||
- 2026-10-27: Every value with a unit in a JSON answer is a number. The unit is
|
||||
at the end of the field name. `@meta.execution_time` is now
|
||||
`execution_time_ms`. In the stats, `uptime` is now `uptime_seconds`,
|
||||
`connection_duration` is now `connection_duration_seconds`, and `memory_usage`
|
||||
is now `memory_usage_bytes`. `whois_stats.current_interval` is now
|
||||
`whois_stats.current_interval_seconds`. They used to be strings such as "12
|
||||
ms", "1h18m34s" or "1.1 GB". `connection_duration_seconds` is null while the
|
||||
feed is not connected. `API_POLICIES.md` is vendored from `sneak/prompts`, and
|
||||
`REPO_POLICIES.md` is re-vendored (closes #199)
|
||||
- 2026-10-27: Every result of the IP lookup has the same fields. A field without
|
||||
a value is null. It used to be left out. An address with no route in the
|
||||
routing table has every route and AS field set to null. `ptr` is an empty list
|
||||
when there are no PTR names. The new field `num_origins` gives the number of
|
||||
origin ASes of the netblock, and `origin_asns` gives which they are (closes
|
||||
#200)
|
||||
- 2026-10-27: With `TRUSTED_PROXIES` unset, the server trusts reverse proxies at
|
||||
the private addresses of RFC 1918. `/ip/`, the home page's `curl` card and the
|
||||
request log then give the client's address from `X-Forwarded-For`. They used
|
||||
to give the proxy's address. A list in `TRUSTED_PROXIES` replaces the default.
|
||||
`TRUSTED_PROXIES` set but empty trusts no proxy (closes #198)
|
||||
- 2026-10-27: The home page has cards with figures of the routing table. One
|
||||
card shows the newest and the oldest kept route. One shows the newest and the
|
||||
oldest updated AS. One shows the AS with the most prefixes. One shows the ASes
|
||||
with a single prefix. One shows fifteen well-known networks. Two cards count
|
||||
the bogon announcements and the prefixes with more than one origin AS. They
|
||||
link to the new pages `/bogons` and `/multiple-origins`, which list them. A
|
||||
new package, `internal/bogon`, tells which prefixes and AS numbers are
|
||||
reserved. The server works the figures out once a minute (closes #154)
|
||||
- 2026-10-27: The per-peer route tables are gone, with the code that wrote,
|
||||
counted and read them. The prefix handler applies each update to the routing
|
||||
table in memory. It writes the table's changes every second. It also stores
|
||||
each new prefix with its first-seen time. The image's memory limits fit a full
|
||||
routing table of about 1 GB. `GOMEMLIMIT` is 2.5 GiB. The writing connection's
|
||||
page cache is 512 MiB. SQLite's soft and hard heap limits are 768 MiB and 1
|
||||
GiB (closes #122)
|
||||
- 2026-10-27: The pages, the API and the stats read every route from the routing
|
||||
table in memory. For a prefix with several origin ASes, the IP lookup gives
|
||||
the one most peers carry. Its `last_seen` is the newest kept route's time. The
|
||||
prefix page and API list every peer that carries the prefix. Only the kept
|
||||
routes show an AS path, a next hop and a time. The stats count the origins
|
||||
whose kept routes were all withdrawn, in `origins_without_kept_route` (part of
|
||||
#122)
|
||||
- 2026-10-27: The prefix, AS and peer handlers make a timed write on every tick
|
||||
of their timers. The prefix handler's timer ticks every second. The other two
|
||||
tick every two seconds. They used to skip every other tick. A full batch's
|
||||
write restarts the timer. So the next timed write comes one interval after
|
||||
that write began (closes #192)
|
||||
- 2026-10-27: Once the stop deadline has passed, the AS, peer and peering
|
||||
handlers start no write either. Before, a full batch or a timer tick could
|
||||
wait while a write ran past the deadline. Either could then start one more
|
||||
write. The flush goroutines of the prefix, AS and peer handlers now run the
|
||||
same code (closes #189)
|
||||
- 2026-10-27: Once the stop deadline has passed, the prefix handler starts no
|
||||
flush. Before, a batch that filled during a late write could still be written
|
||||
after the deadline (closes #188)
|
||||
- 2026-10-27: The prefix handler applies each update to the routing table in
|
||||
memory. It writes the table's changes to the database every second. A write
|
||||
takes at most 20,000 changed pairs of prefix and origin AS. The rest follow in
|
||||
the next writes. A failed write is written again later. Startup reads the
|
||||
table before the streamer connects. The pages still read the per-peer route
|
||||
tables (part of #122)
|
||||
- 2026-10-27: The database keeps a copy of the whole routing table in a new
|
||||
`routes` table, one row per prefix and origin AS. The peers' numbers are in a
|
||||
new `route_peers` table. One function writes the pairs that changed in memory.
|
||||
Another reads the whole table back into memory at startup. Nothing uses them
|
||||
yet (part of #122)
|
||||
- 2026-10-27: A new package, `internal/routetable`, holds the whole routing
|
||||
table in memory. For each prefix and origin AS it keeps the peers that carry
|
||||
it, one bit per peer. It also keeps the two newest of their routes. Nothing
|
||||
uses it yet (part of #122)
|
||||
- 2026-10-27: Each result of the IP lookup lists the public blocklists the
|
||||
address is on, in `blocklists`. The lists are DShield, Feodo Tracker, two
|
||||
FireHOL lists, two Spamhaus DROP lists and the Tor exit list. RouteWatch
|
||||
downloads each list on its own schedule and answers from copies in memory. The
|
||||
copies are also kept in the state directory's `blocklists` directory. IP
|
||||
lookup answers carry `Cache-Control`. An address with no route now has a
|
||||
result too, without route fields. So the IP lookup no longer answers 404
|
||||
(closes #158)
|
||||
- 2026-10-27: The home page has an IP information API card. It shows the `curl`
|
||||
command that looks up the visitor's own address at `/ip/{addr}`. It also shows
|
||||
that command's live answer (closes #156)
|
||||
- 2026-10-27: On the AS page, a click on the heading of the IPv4 or the IPv6
|
||||
prefix list folds or unfolds the list. Both lists start open (closes #155)
|
||||
- 2026-10-27: The AS page lists its IPv4 and its IPv6 prefixes by mask length,
|
||||
shortest first. Prefixes of one mask length are listed by address. They used
|
||||
to be sorted by address first, then by mask length (closes #153)
|
||||
- 2026-10-27: The connection status on `/` and `/status` gets a spinner. It
|
||||
turns one frame each time the stats show more messages than the time before.
|
||||
After three refreshes with no new messages, the page shows "stale" in red
|
||||
instead. Next to it is the time since the last fresh answer (closes #152)
|
||||
- 2026-10-27: The WHOIS details of a Brazilian AS keep their accented letters.
|
||||
LACNIC's answer for such an AS is NIC.br's answer with each accented letter
|
||||
replaced. NIC.br is Brazil's registry. The details now come from NIC.br's own
|
||||
server, `whois.registro.br`. When that query fails, the details from LACNIC's
|
||||
answer are kept and a warning is logged. RouteWatch stops asking NIC.br for an
|
||||
hour after NIC.br says RouteWatch is over its query limit (closes #174)
|
||||
- 2026-10-27: `make asupdate` stops with an error when the download of `as.csv`
|
||||
breaks off. The generator used to retry the failed read forever. A record that
|
||||
does not parse is still logged and skipped (closes #177)
|
||||
- 2026-10-27: `make asupdate` runs the new `script/asupdate`. It replaces the AS
|
||||
list built into RouteWatch only when the generator succeeds and finds at least
|
||||
one AS. A failed download used to replace the list with an empty file. The
|
||||
generator reads `as.csv` from `ipverse/as-metadata`, the source's new name
|
||||
(closes #173)
|
||||
- 2026-10-27: The WHOIS details of an AS at RIPE, APNIC or AFRINIC come only
|
||||
from its `aut-num` object and the organisation object it names. RIPE and APNIC
|
||||
start their answer with the `as-block` object of the AS's block. Its dates
|
||||
were stored as the AS's (closes #168)
|
||||
- 2026-10-27: AS names keep their accented letters. The AS list built into
|
||||
RouteWatch is rebuilt from today's `as.csv`, which added a country code
|
||||
column. That file no longer lists AS 0 or the documentation AS numbers. A
|
||||
WHOIS answer that is not valid UTF-8 is decoded as Latin-1 before it is parsed
|
||||
and stored. RIPE and APNIC send Latin-1 (closes #165)
|
||||
- 2026-10-27: The Age column on the AS and prefix pages reads in words, such as
|
||||
"5 minutes ago" or "2 weeks ago". It used to print Go durations such as
|
||||
"360h0m0s ago". An age under a minute reads "now" or in seconds. An age over a
|
||||
month reads in months or years instead of a date (closes #163)
|
||||
- 2026-10-26: A write to a stored AS, peering or peer keeps the earlier of the
|
||||
two first-seen times. It keeps the later of the two last-seen times. An old
|
||||
update from a collector catching up on a backlog no longer moves last seen
|
||||
back (closes #162)
|
||||
- 2026-10-26: The status page shows "Route Churn/min". That is the number of
|
||||
prefix announcements and withdrawals from all peers in the last full clock
|
||||
minute. `/api/v1/stats` reports it as `route_churn_per_min` in `stream`. It
|
||||
replaces the moving average `route_churn_per_sec`. The messages and updates
|
||||
per second on `/` and `/status` are whole numbers (closes #160)
|
||||
- 2026-10-26: The WHOIS fetcher backs off more slowly. A failure makes its wait
|
||||
1.5 times as long, up to one minute instead of five. Its queries to RIPE,
|
||||
APNIC and AFRINIC leave out person and role objects. RIPE counts those objects
|
||||
against a daily limit per address. A refused query is logged with the RIR's
|
||||
error line (closes #159)
|
||||
- 2026-10-26: The home page's AS name search works. It lists the ASes whose
|
||||
handle or description holds the text, as links to their pages. The new
|
||||
`/api/v1/search/as` answers it. The search ignores case. When the database has
|
||||
no record of an AS, its page takes the handle and description from the AS list
|
||||
built into RouteWatch. It still lists the AS's prefixes (closes #151)
|
||||
- 2026-10-19: README "HTTP Endpoints" lists every route with what it answers.
|
||||
README "Database Schema" matches `schema.sql`. The rest of the README matches
|
||||
the code. This file is brought up to date (closes #72)
|
||||
- 2026-10-19: Route writes are cheaper. The live route and prefix tables lose
|
||||
their random UUID keys. The prefix tables lose their duplicate index. A route
|
||||
update sets only the columns that change. Every write goes through one
|
||||
connection with a page cache of 1 GiB (README "Memory"). A benchmark writes
|
||||
the prefix handler's largest batch faster than the feed sends it (closes #96)
|
||||
- 2026-10-19: The checkpoint of the WAL (SQLite's write-ahead log), the
|
||||
incremental vacuum and `ANALYZE` take the database lock that every batch write
|
||||
holds. So a vacuum no longer fails with "database is locked". An `ANALYZE` no
|
||||
longer makes a batch write lose its updates. `ANALYZE` runs as
|
||||
`PRAGMA optimize` with an analysis limit (closes #94)
|
||||
- 2026-10-19: Shutdown stops the streamer first. Each handler then writes
|
||||
everything left on its queue before it stops. So the updates waiting there are
|
||||
no longer lost (closes #69)
|
||||
- 2026-10-19: A path that ends in an AS set of several members has origin AS 0,
|
||||
which means no known origin. It used to have the set's last member as its
|
||||
origin. The set adds no peerings. The pages show such a route as having no
|
||||
known origin (closes #75)
|
||||
- 2026-10-19: `bgp_peer_count` drops to 0 when the last peer goes down (closes
|
||||
#92)
|
||||
- 2026-10-19: `bgp_peer_count` counts every peer the feed has heard from. It
|
||||
used to count only the peers whose session opened since startup. It drops a
|
||||
peer whose session goes down (closes #74)
|
||||
- 2026-10-19: The shared files are re-vendored from `sneak/prompts` at
|
||||
`0653579`. The linter is `golangci-lint` v2.14.0 with its configuration. Lint
|
||||
and test are `Dockerfile` phases that keep Go's and `golangci-lint`'s caches.
|
||||
`make fmt` also formats the markdown with `prettier`. The agent guidance moves
|
||||
to `AGENTS.md`. The code is fixed for what the new linter configuration finds
|
||||
(closes #52)
|
||||
- 2026-10-19: The streamer reads feed lines up to 1 MiB. A line over 64 KiB used
|
||||
to end the connection (closes #76)
|
||||
- 2026-10-19: A peer's live routes are removed when its BGP session goes down.
|
||||
`STATE` messages are never dropped when a queue is full. One that finds its
|
||||
queue full waits for room ahead of the handler's other messages (closes #56)
|
||||
- 2026-10-19: The prefix-length pages pick their sample with a bounded number of
|
||||
index lookups. They used to read every route of that length. They list each
|
||||
prefix once (closes #66)
|
||||
- 2026-10-19: A request past its time limit answers 504 instead of 408.
|
||||
`/api/v1/stats` answers 504 instead of 500 when its read runs out of time
|
||||
(closes #70)
|
||||
- 2026-10-19: The AS, peer and prefix handlers write each batch outside the lock
|
||||
their message handling needs. So messages keep coming off their queues during
|
||||
a write. Meanwhile the next batch grows to at most ten batch sizes (closes
|
||||
#65)
|
||||
- 2026-10-19: A live route is identified by prefix and peer. So an announcement
|
||||
with a new origin AS replaces the peer's route. A withdrawal removes it
|
||||
whatever its origin AS (closes #55)
|
||||
- 2026-10-14: The streamer reconnects when RIS Live ends the stream or sends
|
||||
nothing for 60 seconds. The health check reports the feed disconnected
|
||||
meanwhile (closes #53)
|
||||
- 2026-10-14: The prefix page takes the names of the origin ASes from the prefix
|
||||
query. It used to read every route of each origin AS (closes #67)
|
||||
- 2026-10-14: The daemon exits with status 1 when its web server cannot listen
|
||||
on its port. It used to run on without a web server (closes #73)
|
||||
- 2026-10-14: The prefix pages parse the prefix before looking it up. One that
|
||||
does not parse answers 400. Any valid way of writing a prefix finds its routes
|
||||
(closes #71)
|
||||
- 2026-10-14: The IP lookup ignores routes shorter than /8 for IPv4 and /16 for
|
||||
IPv6. So a default route is no longer named as an address's owner (closes #57)
|
||||
- 2026-10-14: A prefix batch keeps each peer's last update for a prefix. It used
|
||||
to keep one update per prefix across all peers (closes #54)
|
||||
- 2026-10-14: `reconnect_count` in `/api/v1/stats` counts every reconnect
|
||||
(closes #68)
|
||||
- 2026-10-03: `/api/v1/stats` serves the prefix distribution from memory. The
|
||||
distribution is seeded at startup and adjusted on every live-route write. So a
|
||||
request no longer reads every live route (closes #30)
|
||||
- 2026-10-03: A new database is created with `auto_vacuum` set to incremental.
|
||||
The setting goes in the connection string, so it is set before the file is
|
||||
first written. Each periodic incremental vacuum now returns up to 1000 free
|
||||
pages. The late `PRAGMA auto_vacuum` in `Initialize`, which SQLite ignored, is
|
||||
gone (closes #43)
|
||||
- 2026-10-03: Looking up an IP address no longer reads every IPv6 route. Both
|
||||
families find the most specific live route with at most 33 or 129 lookups on
|
||||
the prefix index. The IPv4 range columns are gone. Prefixes from the feed are
|
||||
stored in one text form. So an IPv6 withdrawal, which the feed sends
|
||||
uncompressed, now removes its route (closes #48)
|
||||
- 2026-10-02: A plain `docker build .` stamps the commit's tag or short commit
|
||||
into the page footer. It used to stamp `unknown`. The value comes from
|
||||
`git describe --tags --always`. `.dockerignore` sends `.git` without
|
||||
`.git/config`. A `VERSION` build argument takes precedence. `make build`
|
||||
stamps the same value (closes #46)
|
||||
- 2026-09-29: The entrypoint creates the data directory if it is missing. It
|
||||
stops the start if a step fails. README "Running under upaas" no longer asks
|
||||
for the host directory to be created first (closes #42)
|
||||
- 2026-09-29: `.dockerignore` keeps `.git`, local build output, local databases
|
||||
and `.env` out of the Docker build context. So they stay out of the source
|
||||
archive in the image (closes #39)
|
||||
- 2026-09-29: The README's first line names the MIT licence and the author. The
|
||||
License section says MIT and links `LICENSE`. An Author section is added. This
|
||||
file is brought up to date (closes #38)
|
||||
- 2026-09-29: `LICENSE` is added, with the MIT licence (closes #1)
|
||||
- 2026-09-28: Stopping the daemon while the feed is flowing no longer panics
|
||||
with "send on closed channel". The read loop checks for a stop just before
|
||||
handing a message to the handler queues. A second `Stop` no longer closes the
|
||||
queues again (closes #34)
|
||||
- 2026-09-28: `docker stop` no longer kills the daemon 2 seconds after the stop
|
||||
signal. The entrypoint switches to the `routewatch` user with `setpriv`
|
||||
instead of `runuser`. So the daemon receives the signal itself. It gets the
|
||||
whole wait `docker stop` allows, up to its own 60-second limit (closes #33)
|
||||
- 2026-09-28: RouteWatch is ready to run under upaas. A set but invalid `PORT`,
|
||||
`XDG_DATA_HOME` or `MALLOC_ARENA_MAX` stops the start. The health check
|
||||
follows `PORT`. README gains a "Running under upaas" section (closes #31)
|
||||
- 2026-09-22: The database statistics are kept in memory. The counts are seeded
|
||||
at startup and adjusted on every write. The oldest and newest route times come
|
||||
from a lookup at each end of an index. `/api/v1/stats` no longer scans the
|
||||
tables (closes #27)
|
||||
- 2026-09-21: Batch writes take the write lock when their transaction begins
|
||||
(`_txlock=immediate`). So they wait out a WAL checkpoint instead of failing
|
||||
with "database is locked" (closes #25)
|
||||
- 2026-09-21: `MALLOC_ARENA_MAX=2` in the image caps the number of glibc malloc
|
||||
arenas. So memory outside the Go runtime no longer grows with the core count
|
||||
(closes #23)
|
||||
- 2026-09-21: The image sets `GOMEMLIMIT=1536MiB`. README "Memory" gives the
|
||||
memory budget and the 5 GiB container limit (closes #13)
|
||||
- 2026-09-21: The four handler queues hold at most 20,000 messages each, down
|
||||
from 100,000 (closes #11)
|
||||
- 2026-09-21: Two goroutine leaks are fixed. The stats handlers leaked after a
|
||||
timeout. The streamer's tickers leaked on every reconnect (closes #12)
|
||||
- 2026-09-21: Parsed RIS messages no longer keep the unused `Community` and
|
||||
`Raw` fields (closes #9)
|
||||
- 2026-09-21: `.editorconfig` is added. A Gitea workflow runs `script/cibuild`
|
||||
on every push (closes #14)
|
||||
- 2026-09-21: The peering handler's AS-path map holds at most 500,000 paths.
|
||||
Every 30 seconds it is swapped for an empty one instead of copied (closes #10)
|
||||
- 2026-09-21: SQLite's memory is bounded across the whole connection pool. Each
|
||||
connection has a 64 MiB page cache. The heap limits are 1 GiB soft and 1.5 GiB
|
||||
hard (closes #8)
|
||||
- 2026-09-21: The Docker build runs the format check, the linter and the tests.
|
||||
Linting runs in a separate stage on a `golangci-lint` image pinned by digest
|
||||
(closes #5)
|
||||
- 2026-09-21: `make test` skips the live feed test, with `-short`. So
|
||||
`make check` no longer depends on the network (closes #2)
|
||||
- 2026-07-07: The scripts-to-rule-them-all layout is adopted. `script/` holds
|
||||
the entrypoints. The `Makefile` targets `check`, `test`, `lint`, `fmt` and
|
||||
`fmt-check` call them. So do `bootstrap`, `setup`, `docker` and `hooks`.
|
||||
README gains an "Entrypoints" section
|
||||
- 2026-02-22: The repo is brought in line with the repo policies. The required
|
||||
policy files are added. `.gitignore` is updated. The `Makefile` gains
|
||||
`fmt-check`, `check`, `docker` and `hooks` targets. The Go code is formatted
|
||||
with `gofmt`. The work was on the branch `repo-policies-compliance`, unmerged
|
||||
when this entry was written
|
||||
- 2026-01-01: The WAL is checkpointed every 5 seconds, and at startup with a log
|
||||
line. Checkpoints run in `TRUNCATE` mode. This fixed slow queries
|
||||
- 2025-12-31: The pages get a navigation bar and a home page with a search. The
|
||||
status page shows the oldest and newest route times. The WHOIS stats query
|
||||
handles `NULL`. The container runs as the `routewatch` user with the right
|
||||
state directory
|
||||
- 2025-12-30: HTTP requests are logged as structured log lines. The HTTP
|
||||
timeouts are longer. Prefix URLs with a slash in them route correctly. The
|
||||
status page gets more metrics and a footer
|
||||
- 2025-12-29: The `Dockerfile` builds in stages and puts a source archive in the
|
||||
image. SQLite reclaims space with incremental vacuum, which does not block
|
||||
- 2025-07-27: The first commit: RouteWatch, a BGP stream monitor
|
||||
File an issue for each problem the run finds.
|
||||
|
||||
# Future Steps
|
||||
@@ -8,8 +8,11 @@ require (
|
||||
github.com/go-chi/chi/v5 v5.2.2
|
||||
github.com/google/btree v1.1.3
|
||||
github.com/google/uuid v1.6.0
|
||||
github.com/mattn/go-sqlite3 v1.14.29
|
||||
github.com/hashicorp/golang-lru/v2 v2.0.7
|
||||
github.com/mattn/go-sqlite3 v1.14.53
|
||||
github.com/maxmind/mmdbwriter v1.2.0
|
||||
github.com/miekg/dns v1.1.73
|
||||
github.com/oschwald/maxminddb-golang/v2 v2.9.0
|
||||
github.com/rcrowley/go-metrics v0.0.0-20250401214520-65e299d6c5c9
|
||||
go.uber.org/fx v1.24.0
|
||||
go4.org/netipx v0.0.0-20260823151212-3075585bcbeb
|
||||
@@ -22,5 +25,5 @@ require (
|
||||
go.uber.org/multierr v1.10.0 // indirect
|
||||
go.uber.org/zap v1.26.0 // indirect
|
||||
golang.org/x/net v0.57.0 // indirect
|
||||
golang.org/x/sys v0.47.0 // indirect
|
||||
golang.org/x/sys v0.49.0 // indirect
|
||||
)
|
||||
@@ -1,7 +1,5 @@
|
||||
github.com/bits-and-blooms/bitset v1.26.0 h1:WPvcPZbP1yuKofmjs2SYIFO1EjzIGhCP9ducElLsh5Q=
|
||||
github.com/bits-and-blooms/bitset v1.26.0/go.mod h1:7hO7Gc7Pp1vODcmWvKMRA9BNmbv6a/7QIWpPxHddWR8=
|
||||
github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c=
|
||||
github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
|
||||
github.com/dustin/go-humanize v1.0.1 h1:GzkhY7T5VNhEkwH0PVJgjz+fX1rhBrR7pRT3mDkpeCY=
|
||||
github.com/dustin/go-humanize v1.0.1/go.mod h1:Mu1zIs6XwVuF/gI1OepvI0qD18qycQx+mFykh5fBlto=
|
||||
github.com/go-chi/chi/v5 v5.2.2 h1:CMwsvRVTbXVytCk1Wd72Zy1LAsAh9GxMmSNWLHCG618=
|
||||
@@ -10,16 +8,20 @@ github.com/google/btree v1.1.3 h1:CVpQJjYgC4VbzxeGVHfvZrv1ctoYCAI8vbl07Fcxlyg=
|
||||
github.com/google/btree v1.1.3/go.mod h1:qOPhT0dTNdNzV6Z/lhRX0YXUafgPLFUh+gZMl761Gm4=
|
||||
github.com/google/uuid v1.6.0 h1:NIvaJDMOsjHA8n1jAhLSgzrAzy1Hgr+hNrb57e+94F0=
|
||||
github.com/google/uuid v1.6.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo=
|
||||
github.com/mattn/go-sqlite3 v1.14.29 h1:1O6nRLJKvsi1H2Sj0Hzdfojwt8GiGKm+LOfLaBFaouQ=
|
||||
github.com/mattn/go-sqlite3 v1.14.29/go.mod h1:Uh1q+B4BYcTPb+yiD3kU8Ct7aC0hY9fxUwlHK0RXw+Y=
|
||||
github.com/hashicorp/golang-lru/v2 v2.0.7 h1:a+bsQ5rvGLjzHuww6tVxozPZFVghXaHOwFs4luLUK2k=
|
||||
github.com/hashicorp/golang-lru/v2 v2.0.7/go.mod h1:QeFd9opnmA6QUJc5vARoKUSoFhyfM2/ZepoAG6RGpeM=
|
||||
github.com/mattn/go-sqlite3 v1.14.53 h1:mnTM0owzUMrKuLo+0w51NUiB0ir2Fo7kFXlPt/l3BGs=
|
||||
github.com/mattn/go-sqlite3 v1.14.53/go.mod h1:6JTjA44L93a0QCyJef5YvlPoKXntQPjzWv5gtm9sB6w=
|
||||
github.com/maxmind/mmdbwriter v1.2.0 h1:hyvDopImmgvle3aR8AaddxXnT0iQH2KWJX3vNfkwzYM=
|
||||
github.com/maxmind/mmdbwriter v1.2.0/go.mod h1:EQmKHhk2y9DRVvyNxwCLKC5FrkXZLx4snc5OlLY5XLE=
|
||||
github.com/miekg/dns v1.1.73 h1:uhT8nJxmTrPJYClxVxTCX+CVn6qnzSiybRk72Z6DgrE=
|
||||
github.com/miekg/dns v1.1.73/go.mod h1:RW2Obtfd5NZHvOFe3zYG0W8koWOQtAzyHaLo8vASBuQ=
|
||||
github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM=
|
||||
github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
|
||||
github.com/oschwald/maxminddb-golang/v2 v2.9.0 h1:YpIl3WoA9oIuNvqNaO/aD1N/7Ci2wOqoKH5GWo53H5Y=
|
||||
github.com/oschwald/maxminddb-golang/v2 v2.9.0/go.mod h1:3TxexMBcR0ss3SbyYvnqfsDzDjN4o1KCg/nTsaMAivU=
|
||||
github.com/rcrowley/go-metrics v0.0.0-20250401214520-65e299d6c5c9 h1:bsUq1dX0N8AOIL7EB/X911+m4EHsnWEHeJ0c+3TTBrg=
|
||||
github.com/rcrowley/go-metrics v0.0.0-20250401214520-65e299d6c5c9/go.mod h1:bCqnVzQkZxMG4s8nGwiZ5l3QUCyqpo9Y+/ZMZ9VjZe4=
|
||||
github.com/stretchr/testify v1.8.1 h1:w7B6lhMri9wdJUVmEZPGGhZzrYTPvgJArz7wNPgYKsk=
|
||||
github.com/stretchr/testify v1.8.1/go.mod h1:w2LPCIKwWwSfY2zedu0+kehJoqGctiVI29o6fzry7u4=
|
||||
github.com/stretchr/testify v1.12.1 h1:EuwCh5fleGS7H32xRwO3wRGT7DxrDhLAT6FF8MpWDWE=
|
||||
github.com/stretchr/testify v1.12.1/go.mod h1:MDEgiDPPsNp5cuIrHPPCyornHKgEVbtFUmoNlxoYthg=
|
||||
go.uber.org/dig v1.19.0 h1:BACLhebsYdpQ7IROQ1AGPjrXcP5dF80U3gKoFzbaq/4=
|
||||
go.uber.org/dig v1.19.0/go.mod h1:Us0rSJiThwCv2GteUN0Q7OKvU7n5J4dxZ9JKUXozFdE=
|
||||
go.uber.org/fx v1.24.0 h1:wE8mruvpg2kiiL1Vqd0CC+tr0/24XIB10Iwp2lLWzkg=
|
||||
@@ -30,14 +32,16 @@ go.uber.org/multierr v1.10.0 h1:S0h4aNzvfcFsC3dRF1jLoaov7oRaKqRGC/pUEJ2yvPQ=
|
||||
go.uber.org/multierr v1.10.0/go.mod h1:20+QtiLqy0Nd6FdQB9TLXag12DsQkrbs3htMFfDN80Y=
|
||||
go.uber.org/zap v1.26.0 h1:sI7k6L95XOKS281NhVKOFCUNIvv9e0w4BF8N3u+tCRo=
|
||||
go.uber.org/zap v1.26.0/go.mod h1:dtElttAiwGvoJ/vj4IwHBS/gXsEu/pZ50mUIRWuG0so=
|
||||
go.yaml.in/yaml/v3 v3.0.5 h1:N6y/pJk8buWs9NY5ERU2HSMfm+IuD/OtfdAnq6kESPw=
|
||||
go.yaml.in/yaml/v3 v3.0.5/go.mod h1:HVTZu1O7/Vkt2N+BFy8Zza+lnLsABggaTM2ZpNIGuKg=
|
||||
go4.org/netipx v0.0.0-20260823151212-3075585bcbeb h1:XBM4hvfwGAttkkiTIFfeigdfcL1xIfdKXqFdgiHGtDs=
|
||||
go4.org/netipx v0.0.0-20260823151212-3075585bcbeb/go.mod h1:PLyyIXexvUFg3Owu6p/WfdlivPbZJsZdgWZlrGope/Y=
|
||||
golang.org/x/net v0.57.0 h1:K5+3DljvIuDG9/Jv9rvyMywYNFCQ9RSUY6OOTTkT+tE=
|
||||
golang.org/x/net v0.57.0/go.mod h1:KpXc8iv+r3XplLAG/f7Jsf9RPszJzdR0f58q9vGOuEU=
|
||||
golang.org/x/sync v0.24.0 h1:iqiDrGoDgyuvAKrhLQthUEzU/R378N78gXbBjxed7oA=
|
||||
golang.org/x/sync v0.24.0/go.mod h1:sUUOizhqBxiL6pEWpqNLUiaJn1ShEbZ6BBqskPbjZm0=
|
||||
golang.org/x/sys v0.47.0 h1:o7XGOvZQCADBQQ4Y7VNq2dRWQR7JmOUW8Kxx4ZsNgWs=
|
||||
golang.org/x/sys v0.47.0/go.mod h1:4GL1E5IUh+htKOUEOaiffhrAeqysfVGipDYzABqnCmw=
|
||||
golang.org/x/sys v0.49.0 h1:XbzkgYJHdqh/8m2Uu0W/dQv8nktxx4BFHp1M0gROTXA=
|
||||
golang.org/x/sys v0.49.0/go.mod h1:hNLxWAXmnKAxqDtdwIYC4bM9oQPEecfsnNMuSxOs3og=
|
||||
golang.org/x/term v0.45.0 h1:NwWyBmoJCbfTHpxrWoZ9C6/VxOf7ic219I8xZZFdrf0=
|
||||
golang.org/x/term v0.45.0/go.mod h1:9aqxs0blBcrm/n0L9QW0aRVD+ktan8ssZromtqJC43w=
|
||||
golang.org/x/text v0.43.0 h1:1QivrlhwAsnMOcqgOdl9my7s9OyZl8O5/UGUTJVXoWc=
|
||||
|
||||
@@ -6,3 +6,19 @@ CREATE TABLE IF NOT EXISTS hostnames (
|
||||
names TEXT NOT NULL,
|
||||
expires_at INTEGER NOT NULL
|
||||
);
|
||||
|
||||
-- geolocations holds the answers of geojs. A part geojs does not know is NULL.
|
||||
-- fetched_at is in seconds since 1970.
|
||||
CREATE TABLE IF NOT EXISTS geolocations (
|
||||
address TEXT PRIMARY KEY,
|
||||
country_code TEXT,
|
||||
country TEXT,
|
||||
continent_code TEXT,
|
||||
region TEXT,
|
||||
city TEXT,
|
||||
latitude REAL,
|
||||
longitude REAL,
|
||||
accuracy_radius_km INTEGER,
|
||||
time_zone TEXT,
|
||||
fetched_at INTEGER NOT NULL
|
||||
);
|
||||
@@ -0,0 +1,253 @@
|
||||
package geo
|
||||
|
||||
import (
|
||||
"context"
|
||||
"database/sql"
|
||||
"errors"
|
||||
"fmt"
|
||||
"maps"
|
||||
"time"
|
||||
)
|
||||
|
||||
const (
|
||||
// entryLifetime is how long a geojs answer is kept after its fetch.
|
||||
entryLifetime = 7 * 24 * time.Hour
|
||||
|
||||
// flushInterval is how often new answers are written to cache.db.
|
||||
flushInterval = 5 * time.Second
|
||||
|
||||
// sweepInterval is how often expired answers are deleted.
|
||||
sweepInterval = 10 * time.Minute
|
||||
|
||||
// rowColumns are the columns of the geolocations table in the order
|
||||
// scanRow reads them.
|
||||
rowColumns = "address, country_code, country, continent_code, region, " +
|
||||
"city, latitude, longitude, accuracy_radius_km, time_zone, fetched_at"
|
||||
)
|
||||
|
||||
// rowScanner is the Scan method of *sql.Row and *sql.Rows.
|
||||
type rowScanner interface {
|
||||
Scan(dest ...any) error
|
||||
}
|
||||
|
||||
// cached returns the current answer held for address, from memory, then from
|
||||
// the answers waiting to be written, then from cache.db. It returns nil when
|
||||
// there is none.
|
||||
func (l *Locator) cached(ctx context.Context, address string) *Location {
|
||||
found, ok := l.entries.Get(address)
|
||||
if ok {
|
||||
if !l.expired(found) {
|
||||
return found
|
||||
}
|
||||
|
||||
l.entries.Remove(address)
|
||||
}
|
||||
|
||||
l.pendingMu.Lock()
|
||||
waiting := l.pending[address]
|
||||
l.pendingMu.Unlock()
|
||||
|
||||
if waiting != nil && !l.expired(waiting) {
|
||||
l.entries.Add(address, waiting)
|
||||
|
||||
return waiting
|
||||
}
|
||||
|
||||
stored, err := l.readRow(ctx, address)
|
||||
if err != nil {
|
||||
l.logger.Warn("Failed to read a geolocation cache row",
|
||||
"address", address, "error", err)
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
if stored != nil {
|
||||
l.entries.Add(address, stored)
|
||||
}
|
||||
|
||||
return stored
|
||||
}
|
||||
|
||||
// expired says whether the answer is at the end of its lifetime.
|
||||
func (l *Locator) expired(answer *Location) bool {
|
||||
return !l.now().Before(answer.Updated.Add(entryLifetime))
|
||||
}
|
||||
|
||||
// cutoff returns the fetch time at or before which an answer has expired.
|
||||
func (l *Locator) cutoff() int64 {
|
||||
return l.now().Add(-entryLifetime).Unix()
|
||||
}
|
||||
|
||||
// store keeps an answer in memory and queues it for cache.db. While cache.db
|
||||
// cannot be written, the queue stops growing at maxEntries answers.
|
||||
func (l *Locator) store(address string, answer *Location) {
|
||||
l.entries.Add(address, answer)
|
||||
|
||||
l.pendingMu.Lock()
|
||||
defer l.pendingMu.Unlock()
|
||||
|
||||
if len(l.pending) < l.maxEntries {
|
||||
l.pending[address] = answer
|
||||
}
|
||||
}
|
||||
|
||||
// keepCache writes new answers and deletes expired ones until ctx ends.
|
||||
func (l *Locator) keepCache(ctx context.Context) {
|
||||
flushes := time.NewTicker(flushInterval)
|
||||
defer flushes.Stop()
|
||||
|
||||
sweeps := time.NewTicker(sweepInterval)
|
||||
defer sweeps.Stop()
|
||||
|
||||
for {
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
l.flush(context.WithoutCancel(ctx))
|
||||
|
||||
return
|
||||
case <-flushes.C:
|
||||
l.flush(ctx)
|
||||
case <-sweeps.C:
|
||||
l.sweep(ctx)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// load reads the answers that have not expired into memory, at most maxEntries
|
||||
// of them, the newest last so that they are the last to be dropped.
|
||||
func (l *Locator) load(ctx context.Context) error {
|
||||
rows, err := l.db.QueryContext(ctx,
|
||||
"SELECT "+rowColumns+" FROM (SELECT "+rowColumns+" FROM geolocations "+
|
||||
"WHERE fetched_at > ? ORDER BY fetched_at DESC LIMIT ?) "+
|
||||
"ORDER BY fetched_at",
|
||||
l.cutoff(), l.maxEntries)
|
||||
if err != nil {
|
||||
return fmt.Errorf("reading the geolocation cache: %w", err)
|
||||
}
|
||||
|
||||
defer func() { _ = rows.Close() }()
|
||||
|
||||
for rows.Next() {
|
||||
address, answer, err := scanRow(rows)
|
||||
if err != nil {
|
||||
return fmt.Errorf("reading a geolocation cache row: %w", err)
|
||||
}
|
||||
|
||||
l.entries.Add(address, answer)
|
||||
}
|
||||
|
||||
err = rows.Err()
|
||||
if err != nil {
|
||||
return fmt.Errorf("reading the geolocation cache: %w", err)
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
// readRow returns the current answer for address in cache.db, or nil when it
|
||||
// has none.
|
||||
func (l *Locator) readRow(
|
||||
ctx context.Context, address string,
|
||||
) (*Location, error) {
|
||||
row := l.db.QueryRowContext(ctx,
|
||||
"SELECT "+rowColumns+" FROM geolocations "+
|
||||
"WHERE address = ? AND fetched_at > ?",
|
||||
address, l.cutoff())
|
||||
|
||||
_, answer, err := scanRow(row)
|
||||
if errors.Is(err, sql.ErrNoRows) {
|
||||
return nil, nil //nolint:nilnil // no row is not a failure
|
||||
}
|
||||
|
||||
return answer, err
|
||||
}
|
||||
|
||||
// scanRow reads one row of the geolocations table.
|
||||
func scanRow(row rowScanner) (string, *Location, error) {
|
||||
var (
|
||||
address string
|
||||
fetched int64
|
||||
answer Location
|
||||
)
|
||||
|
||||
err := row.Scan(&address, &answer.CountryCode, &answer.Country,
|
||||
&answer.ContinentCode, &answer.Region, &answer.City, &answer.Latitude,
|
||||
&answer.Longitude, &answer.AccuracyRadiusKm, &answer.TimeZone, &fetched)
|
||||
if err != nil {
|
||||
return "", nil, err
|
||||
}
|
||||
|
||||
answer.Source = sourceGeoJS
|
||||
answer.Updated = time.Unix(fetched, 0).UTC()
|
||||
|
||||
return address, &answer, nil
|
||||
}
|
||||
|
||||
// flush writes the pending answers to cache.db. Those it cannot write stay
|
||||
// pending, unless a newer answer for the address has arrived meanwhile.
|
||||
func (l *Locator) flush(ctx context.Context) {
|
||||
l.pendingMu.Lock()
|
||||
batch := l.pending
|
||||
l.pending = make(map[string]*Location)
|
||||
l.pendingMu.Unlock()
|
||||
|
||||
if len(batch) == 0 {
|
||||
return
|
||||
}
|
||||
|
||||
err := l.writeRows(ctx, batch)
|
||||
if err == nil {
|
||||
return
|
||||
}
|
||||
|
||||
l.logger.Error("Failed to write the geolocation cache", "error", err)
|
||||
|
||||
l.pendingMu.Lock()
|
||||
defer l.pendingMu.Unlock()
|
||||
|
||||
maps.Insert(batch, maps.All(l.pending))
|
||||
l.pending = batch
|
||||
}
|
||||
|
||||
// writeRows writes batch to cache.db in one transaction.
|
||||
func (l *Locator) writeRows(
|
||||
ctx context.Context, batch map[string]*Location,
|
||||
) error {
|
||||
tx, err := l.db.BeginTx(ctx, nil)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
defer func() { _ = tx.Rollback() }()
|
||||
|
||||
for address, answer := range batch {
|
||||
_, err = tx.ExecContext(ctx,
|
||||
"INSERT OR REPLACE INTO geolocations ("+rowColumns+") "+
|
||||
"VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)",
|
||||
address, answer.CountryCode, answer.Country, answer.ContinentCode,
|
||||
answer.Region, answer.City, answer.Latitude, answer.Longitude,
|
||||
answer.AccuracyRadiusKm, answer.TimeZone, answer.Updated.Unix())
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
|
||||
return tx.Commit()
|
||||
}
|
||||
|
||||
// sweep deletes the expired answers from memory and cache.db.
|
||||
func (l *Locator) sweep(ctx context.Context) {
|
||||
for _, address := range l.entries.Keys() {
|
||||
answer, ok := l.entries.Peek(address)
|
||||
if ok && l.expired(answer) {
|
||||
l.entries.Remove(address)
|
||||
}
|
||||
}
|
||||
|
||||
_, err := l.db.ExecContext(ctx,
|
||||
"DELETE FROM geolocations WHERE fetched_at <= ?", l.cutoff())
|
||||
if err != nil {
|
||||
l.logger.Error("Failed to delete expired geolocation cache rows",
|
||||
"error", err)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,349 @@
|
||||
package geo
|
||||
|
||||
import (
|
||||
"compress/gzip"
|
||||
"context"
|
||||
"errors"
|
||||
"fmt"
|
||||
"io"
|
||||
"io/fs"
|
||||
"net/http"
|
||||
"net/netip"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"time"
|
||||
|
||||
"github.com/oschwald/maxminddb-golang/v2"
|
||||
)
|
||||
|
||||
const (
|
||||
// DatabaseDownloadFormat is the URL of DB-IP's City Lite database, with %s
|
||||
// for the year and month.
|
||||
DatabaseDownloadFormat = "https://download.db-ip.com/free/" +
|
||||
"dbip-city-lite-%s.mmdb.gz"
|
||||
|
||||
// databaseMonthFormat is the year and month in the URL.
|
||||
databaseMonthFormat = "2006-01"
|
||||
|
||||
// databaseFileName is the copy of the database in the state directory.
|
||||
databaseFileName = "dbip-city-lite.mmdb"
|
||||
|
||||
// databaseDownloadSuffix ends the name of the file a download is written
|
||||
// to, until it has opened.
|
||||
databaseDownloadSuffix = ".download"
|
||||
|
||||
// maxDatabaseSize is the largest database accepted once unpacked, in
|
||||
// bytes.
|
||||
maxDatabaseSize = 200 << 20
|
||||
|
||||
// databaseDownloadTimeout limits one download.
|
||||
databaseDownloadTimeout = 10 * time.Minute
|
||||
|
||||
// databaseCheckInterval is how often RouteWatch checks whether a new
|
||||
// month's database is due. A new month's file appears some time on the
|
||||
// first of the month, so a failed download is tried again this much later.
|
||||
databaseCheckInterval = time.Hour
|
||||
)
|
||||
|
||||
var (
|
||||
errDatabaseStatus = errors.New("unexpected HTTP status from DB-IP")
|
||||
errDatabaseTooLong = errors.New("database is over the size limit")
|
||||
)
|
||||
|
||||
// dbipName is the part of a database record that holds an English name.
|
||||
type dbipName struct {
|
||||
Names struct {
|
||||
English string `maxminddb:"en"`
|
||||
} `maxminddb:"names"`
|
||||
}
|
||||
|
||||
// dbipRecord is the part of a database record RouteWatch reads.
|
||||
type dbipRecord struct {
|
||||
City dbipName `maxminddb:"city"`
|
||||
Continent struct {
|
||||
Code string `maxminddb:"code"`
|
||||
} `maxminddb:"continent"`
|
||||
Country struct {
|
||||
dbipName
|
||||
|
||||
ISOCode string `maxminddb:"iso_code"`
|
||||
} `maxminddb:"country"`
|
||||
Location struct {
|
||||
Latitude *float64 `maxminddb:"latitude"`
|
||||
Longitude *float64 `maxminddb:"longitude"`
|
||||
} `maxminddb:"location"`
|
||||
Subdivisions []dbipName `maxminddb:"subdivisions"`
|
||||
}
|
||||
|
||||
// location returns the record as a Location fetched at the time given. The
|
||||
// first subdivision is the region. The database has no accuracy radius and no
|
||||
// time zone.
|
||||
func (r *dbipRecord) location(fetched time.Time) *Location {
|
||||
found := &Location{
|
||||
CountryCode: nilIfEmpty(r.Country.ISOCode),
|
||||
Country: nilIfEmpty(r.Country.Names.English),
|
||||
ContinentCode: nilIfEmpty(r.Continent.Code),
|
||||
City: nilIfEmpty(r.City.Names.English),
|
||||
Latitude: r.Location.Latitude,
|
||||
Longitude: r.Location.Longitude,
|
||||
Source: sourceDBIP,
|
||||
Updated: fetched,
|
||||
}
|
||||
|
||||
if len(r.Subdivisions) > 0 {
|
||||
found.Region = nilIfEmpty(r.Subdivisions[0].Names.English)
|
||||
}
|
||||
|
||||
return found
|
||||
}
|
||||
|
||||
// nilIfEmpty returns a pointer to text, or nil when text is empty.
|
||||
func nilIfEmpty(text string) *string {
|
||||
if text == "" {
|
||||
return nil
|
||||
}
|
||||
|
||||
return &text
|
||||
}
|
||||
|
||||
// fromDatabase returns where the copy of the database places addr. It returns
|
||||
// nil when there is no copy yet or the copy has no record for addr. It also
|
||||
// returns nil, and logs the error, when reading the copy fails.
|
||||
func (l *Locator) fromDatabase(addr netip.Addr) *Location {
|
||||
l.databaseMu.RLock()
|
||||
defer l.databaseMu.RUnlock()
|
||||
|
||||
if l.database == nil {
|
||||
return nil
|
||||
}
|
||||
|
||||
result := l.database.Lookup(addr)
|
||||
|
||||
err := result.Err()
|
||||
if err != nil {
|
||||
l.logger.Error("Failed to look up an address in the database",
|
||||
"address", addr, "error", err)
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
if !result.Found() {
|
||||
return nil
|
||||
}
|
||||
|
||||
var record dbipRecord
|
||||
|
||||
err = result.Decode(&record)
|
||||
if err != nil {
|
||||
l.logger.Error("Failed to read a database record",
|
||||
"address", addr, "error", err)
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
return record.location(l.databaseFetched)
|
||||
}
|
||||
|
||||
// keepDatabaseCurrent downloads the database when it is due, at once and then
|
||||
// every databaseCheckInterval, until ctx ends.
|
||||
func (l *Locator) keepDatabaseCurrent(ctx context.Context) {
|
||||
l.downloadDatabaseIfDue(ctx)
|
||||
|
||||
checks := time.NewTicker(databaseCheckInterval)
|
||||
defer checks.Stop()
|
||||
|
||||
for {
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
return
|
||||
case <-checks.C:
|
||||
l.downloadDatabaseIfDue(ctx)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// downloadDatabaseIfDue downloads this month's database, unless the copy in
|
||||
// use was downloaded this month. A failure leaves the copy as it is.
|
||||
func (l *Locator) downloadDatabaseIfDue(ctx context.Context) {
|
||||
now := l.now().UTC()
|
||||
|
||||
if !l.databaseDue(now) {
|
||||
return
|
||||
}
|
||||
|
||||
err := l.downloadDatabase(ctx, now)
|
||||
if err != nil && ctx.Err() == nil {
|
||||
l.logger.Error("Database download failed; keeping the previous copy",
|
||||
"error", err)
|
||||
}
|
||||
}
|
||||
|
||||
// databaseDue says whether there is no open copy or it is from an earlier
|
||||
// month than now.
|
||||
func (l *Locator) databaseDue(now time.Time) bool {
|
||||
l.databaseMu.RLock()
|
||||
defer l.databaseMu.RUnlock()
|
||||
|
||||
return l.database == nil ||
|
||||
l.databaseFetched.Format(databaseMonthFormat) !=
|
||||
now.Format(databaseMonthFormat)
|
||||
}
|
||||
|
||||
// downloadDatabase fetches the database of now's month. The file replaces the
|
||||
// copy, on disk and in use, only once it has opened.
|
||||
func (l *Locator) downloadDatabase(ctx context.Context, now time.Time) error {
|
||||
ctx, cancel := context.WithTimeout(ctx, databaseDownloadTimeout)
|
||||
defer cancel()
|
||||
|
||||
url := fmt.Sprintf(l.databaseURLFormat, now.Format(databaseMonthFormat))
|
||||
|
||||
request, err := http.NewRequestWithContext(ctx, http.MethodGet, url, nil)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
request.Header.Set(userAgentHeader, userAgent)
|
||||
|
||||
response, err := l.client.Do(request)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
defer func() { _ = response.Body.Close() }()
|
||||
|
||||
if response.StatusCode != http.StatusOK {
|
||||
return fmt.Errorf("%w: %s", errDatabaseStatus, response.Status)
|
||||
}
|
||||
|
||||
return l.replaceDatabase(response.Body, now)
|
||||
}
|
||||
|
||||
// replaceDatabase unpacks the gzip stream body beside the copy, then makes the
|
||||
// result the copy once it has opened. Whatever fails, the file it wrote is
|
||||
// removed and the copy stays.
|
||||
func (l *Locator) replaceDatabase(body io.Reader, now time.Time) error {
|
||||
root, err := os.OpenRoot(l.dir)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
defer func() { _ = root.Close() }()
|
||||
|
||||
downloadName := databaseFileName + databaseDownloadSuffix
|
||||
|
||||
err = unpackDatabase(root, downloadName, body, l.maxDatabaseBytes)
|
||||
if err != nil {
|
||||
_ = root.Remove(downloadName)
|
||||
|
||||
return err
|
||||
}
|
||||
|
||||
opened, err := maxminddb.Open(filepath.Join(l.dir, downloadName))
|
||||
if err != nil {
|
||||
_ = root.Remove(downloadName)
|
||||
|
||||
return fmt.Errorf("opening the downloaded database: %w", err)
|
||||
}
|
||||
|
||||
err = root.Rename(downloadName, databaseFileName)
|
||||
if err != nil {
|
||||
_ = opened.Close()
|
||||
_ = root.Remove(downloadName)
|
||||
|
||||
return err
|
||||
}
|
||||
|
||||
l.useDatabase(opened, now.Truncate(time.Second))
|
||||
l.logger.Info("Downloaded a new copy of the database")
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
// unpackDatabase writes the gzip stream body to the file name in root, up to
|
||||
// limit bytes once unpacked.
|
||||
func unpackDatabase(
|
||||
root *os.Root, name string, body io.Reader, limit int64,
|
||||
) error {
|
||||
unpacked, err := gzip.NewReader(body)
|
||||
if err != nil {
|
||||
return fmt.Errorf("unpacking the database: %w", err)
|
||||
}
|
||||
|
||||
f, err := root.Create(name)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
defer func() { _ = f.Close() }()
|
||||
|
||||
written, err := io.Copy(f, io.LimitReader(unpacked, limit+1))
|
||||
if err != nil {
|
||||
return fmt.Errorf("unpacking the database: %w", err)
|
||||
}
|
||||
|
||||
if written > limit {
|
||||
return errDatabaseTooLong
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
// openDatabaseCopy opens the copy left by an earlier run, if there is one. A
|
||||
// copy that does not open is treated as no copy, so it is downloaded again.
|
||||
func (l *Locator) openDatabaseCopy() {
|
||||
path := filepath.Join(l.dir, databaseFileName)
|
||||
|
||||
info, err := os.Stat(path)
|
||||
if errors.Is(err, fs.ErrNotExist) {
|
||||
return
|
||||
}
|
||||
|
||||
if err != nil {
|
||||
l.logger.Error("Failed to read the database copy", "error", err)
|
||||
|
||||
return
|
||||
}
|
||||
|
||||
opened, err := maxminddb.Open(path)
|
||||
if err != nil {
|
||||
l.logger.Error("Failed to open the database copy", "error", err)
|
||||
|
||||
return
|
||||
}
|
||||
|
||||
l.useDatabase(opened, info.ModTime().UTC().Truncate(time.Second))
|
||||
}
|
||||
|
||||
// useDatabase makes opened the copy in use and closes the one it replaces.
|
||||
func (l *Locator) useDatabase(opened *maxminddb.Reader, fetched time.Time) {
|
||||
l.databaseMu.Lock()
|
||||
defer l.databaseMu.Unlock()
|
||||
|
||||
if l.database != nil {
|
||||
err := l.database.Close()
|
||||
if err != nil {
|
||||
l.logger.Error("Failed to close the old database copy", "error", err)
|
||||
}
|
||||
}
|
||||
|
||||
l.database = opened
|
||||
l.databaseFetched = fetched
|
||||
}
|
||||
|
||||
// closeDatabase closes the copy in use, after which no source can answer from
|
||||
// it.
|
||||
func (l *Locator) closeDatabase() {
|
||||
l.databaseMu.Lock()
|
||||
defer l.databaseMu.Unlock()
|
||||
|
||||
if l.database == nil {
|
||||
return
|
||||
}
|
||||
|
||||
err := l.database.Close()
|
||||
if err != nil {
|
||||
l.logger.Error("Failed to close the database copy", "error", err)
|
||||
}
|
||||
|
||||
l.database = nil
|
||||
}
|
||||
@@ -0,0 +1,151 @@
|
||||
package geo
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"io"
|
||||
"math"
|
||||
"net/http"
|
||||
"net/netip"
|
||||
"strconv"
|
||||
"strings"
|
||||
"time"
|
||||
)
|
||||
|
||||
const (
|
||||
// GeoJSEndpoint is the URL of the geojs geo endpoint.
|
||||
GeoJSEndpoint = "https://get.geojs.io/v1/ip/geo.json"
|
||||
|
||||
// geojsTimeLimit limits one request to geojs.
|
||||
geojsTimeLimit = time.Second
|
||||
|
||||
// geojsAddressSeparator separates the addresses of one request.
|
||||
geojsAddressSeparator = ","
|
||||
|
||||
// geojsUnknown is what geojs sends for a value it does not know.
|
||||
geojsUnknown = "nil"
|
||||
|
||||
// maxGeoJSAnswerBytes is far more than an answer for the most addresses
|
||||
// an IP lookup can ask about.
|
||||
maxGeoJSAnswerBytes = 1 << 20
|
||||
)
|
||||
|
||||
var errGeoJSStatus = errors.New("unexpected HTTP status from geojs")
|
||||
|
||||
// geojsAnswer is one address's entry in an answer from geojs.
|
||||
//
|
||||
//nolint:tagliatelle // the keys geojs sends
|
||||
type geojsAnswer struct {
|
||||
IP string `json:"ip"`
|
||||
CountryCode string `json:"country_code"`
|
||||
Country string `json:"country"`
|
||||
ContinentCode string `json:"continent_code"`
|
||||
Region string `json:"region"`
|
||||
City string `json:"city"`
|
||||
Latitude string `json:"latitude"`
|
||||
Longitude string `json:"longitude"`
|
||||
Accuracy *int `json:"accuracy"`
|
||||
TimeZone string `json:"timezone"`
|
||||
}
|
||||
|
||||
// askGeoJS asks geojs about addresses in one request, and returns its answers
|
||||
// by address. An address geojs left out, or gave no country, city or
|
||||
// coordinates for, has no entry.
|
||||
func (l *Locator) askGeoJS(
|
||||
ctx context.Context, addresses []string,
|
||||
) (map[string]*Location, error) {
|
||||
ctx, cancel := context.WithTimeout(ctx, l.timeLimit)
|
||||
defer cancel()
|
||||
|
||||
// A comma and a colon are legal in a query, and the escaped forms are
|
||||
// not what geojs documents.
|
||||
url := l.geojsURL + "?ip=" + strings.Join(addresses, geojsAddressSeparator)
|
||||
|
||||
request, err := http.NewRequestWithContext(ctx, http.MethodGet, url, nil)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
request.Header.Set(userAgentHeader, userAgent)
|
||||
|
||||
response, err := l.client.Do(request)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
defer func() { _ = response.Body.Close() }()
|
||||
|
||||
if response.StatusCode != http.StatusOK {
|
||||
return nil, fmt.Errorf("%w: %d", errGeoJSStatus, response.StatusCode)
|
||||
}
|
||||
|
||||
var parsed []geojsAnswer
|
||||
|
||||
err = json.NewDecoder(io.LimitReader(response.Body, maxGeoJSAnswerBytes)).
|
||||
Decode(&parsed)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("reading the geojs answer: %w", err)
|
||||
}
|
||||
|
||||
fetched := l.now().UTC().Truncate(time.Second)
|
||||
answers := make(map[string]*Location, len(parsed))
|
||||
|
||||
for _, answer := range parsed {
|
||||
addr, err := netip.ParseAddr(answer.IP)
|
||||
if err != nil {
|
||||
continue
|
||||
}
|
||||
|
||||
location := answer.location(fetched)
|
||||
if hasLocation(location) {
|
||||
answers[plainAddress(addr)] = location
|
||||
}
|
||||
}
|
||||
|
||||
return answers, nil
|
||||
}
|
||||
|
||||
// hasLocation says whether found has a country, a city or a coordinate.
|
||||
func hasLocation(found *Location) bool {
|
||||
return found.CountryCode != nil || found.Country != nil ||
|
||||
found.City != nil || found.Latitude != nil || found.Longitude != nil
|
||||
}
|
||||
|
||||
// location returns the answer as a Location fetched at the time given.
|
||||
func (a geojsAnswer) location(fetched time.Time) *Location {
|
||||
return &Location{
|
||||
CountryCode: knownText(a.CountryCode),
|
||||
Country: knownText(a.Country),
|
||||
ContinentCode: knownText(a.ContinentCode),
|
||||
Region: knownText(a.Region),
|
||||
City: knownText(a.City),
|
||||
Latitude: knownNumber(a.Latitude),
|
||||
Longitude: knownNumber(a.Longitude),
|
||||
AccuracyRadiusKm: a.Accuracy,
|
||||
TimeZone: knownText(a.TimeZone),
|
||||
Source: sourceGeoJS,
|
||||
Updated: fetched,
|
||||
}
|
||||
}
|
||||
|
||||
// knownText returns a pointer to text, or nil when geojs does not know it.
|
||||
func knownText(text string) *string {
|
||||
if text == "" || text == geojsUnknown {
|
||||
return nil
|
||||
}
|
||||
|
||||
return &text
|
||||
}
|
||||
|
||||
// knownNumber returns text as a number, or nil when geojs does not know it. NaN
|
||||
// and infinity parse, but JSON cannot carry them.
|
||||
func knownNumber(text string) *float64 {
|
||||
number, err := strconv.ParseFloat(text, 64)
|
||||
if err != nil || math.IsNaN(number) || math.IsInf(number, 0) {
|
||||
return nil
|
||||
}
|
||||
|
||||
return &number
|
||||
}
|
||||
@@ -0,0 +1,232 @@
|
||||
// Package geo finds where IP addresses are. It asks geojs.io, and keeps those
|
||||
// answers in memory and in cache.db. When geojs.io gives no location for an
|
||||
// address, it reads a copy of DB-IP's City Lite database, which it downloads
|
||||
// once a month.
|
||||
package geo
|
||||
|
||||
import (
|
||||
"context"
|
||||
"database/sql"
|
||||
"fmt"
|
||||
"log/slog"
|
||||
"net/http"
|
||||
"net/netip"
|
||||
"sync"
|
||||
"time"
|
||||
|
||||
lru "github.com/hashicorp/golang-lru/v2"
|
||||
"github.com/oschwald/maxminddb-golang/v2"
|
||||
)
|
||||
|
||||
// The values of Location.Source.
|
||||
const (
|
||||
sourceGeoJS = "geojs"
|
||||
sourceDBIP = "dbip"
|
||||
)
|
||||
|
||||
const (
|
||||
// maxEntries limits how many geojs answers are held in memory.
|
||||
maxEntries = 250_000
|
||||
|
||||
// userAgentHeader and userAgent name routewatch to geojs and DB-IP.
|
||||
userAgentHeader = "User-Agent"
|
||||
userAgent = "routewatch (+https://git.eeqj.de/sneak/routewatch)"
|
||||
)
|
||||
|
||||
// Params is what New needs.
|
||||
type Params struct {
|
||||
// DB is the open cache.db.
|
||||
DB *sql.DB
|
||||
|
||||
// Dir is the directory that holds the copy of the database.
|
||||
Dir string
|
||||
|
||||
// GeoJSURL is the URL of the geojs geo endpoint, GeoJSEndpoint in use.
|
||||
GeoJSURL string
|
||||
|
||||
// DatabaseURLFormat is the URL of the database with %s for the year and
|
||||
// month, DatabaseDownloadFormat in use.
|
||||
DatabaseURLFormat string
|
||||
|
||||
Logger *slog.Logger
|
||||
}
|
||||
|
||||
// Location is where an address is. A part the source does not know is nil.
|
||||
//
|
||||
//nolint:tagliatelle // snake_case keys, as the Go style guide asks
|
||||
type Location struct {
|
||||
CountryCode *string `json:"country_code"`
|
||||
Country *string `json:"country"`
|
||||
ContinentCode *string `json:"continent_code"`
|
||||
Region *string `json:"region"`
|
||||
City *string `json:"city"`
|
||||
Latitude *float64 `json:"latitude"`
|
||||
Longitude *float64 `json:"longitude"`
|
||||
AccuracyRadiusKm *int `json:"accuracy_radius_km"`
|
||||
TimeZone *string `json:"time_zone"`
|
||||
|
||||
// Source is geojs or dbip.
|
||||
Source string `json:"source"`
|
||||
|
||||
// Updated is when the value was fetched: the geojs answer, or the copy of
|
||||
// the database.
|
||||
Updated time.Time `json:"updated"`
|
||||
}
|
||||
|
||||
// Locator answers where addresses are. Start loads its cache and starts its
|
||||
// goroutines.
|
||||
type Locator struct {
|
||||
db *sql.DB
|
||||
dir string
|
||||
geojsURL string
|
||||
databaseURLFormat string
|
||||
timeLimit time.Duration
|
||||
maxEntries int
|
||||
maxDatabaseBytes int64
|
||||
logger *slog.Logger
|
||||
client *http.Client
|
||||
now func() time.Time
|
||||
|
||||
// entries holds geojs answers by address, and drops the least recently
|
||||
// used first.
|
||||
entries *lru.Cache[string, *Location]
|
||||
|
||||
// pending holds the answers that cache.db does not have yet.
|
||||
pendingMu sync.Mutex
|
||||
pending map[string]*Location
|
||||
|
||||
// database is the open copy, nil until one has opened, and databaseFetched
|
||||
// is when that copy was downloaded. A lookup holds the read lock, so that
|
||||
// a new copy never closes the old one under it.
|
||||
databaseMu sync.RWMutex
|
||||
database *maxminddb.Reader
|
||||
databaseFetched time.Time
|
||||
|
||||
// stop ends the goroutines Start runs, and done is closed when they have
|
||||
// returned.
|
||||
stop context.CancelFunc
|
||||
done chan struct{}
|
||||
}
|
||||
|
||||
// New returns a Locator with an empty cache. Start loads it.
|
||||
func New(params Params) (*Locator, error) {
|
||||
entries, err := lru.New[string, *Location](maxEntries)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("creating the geolocation cache: %w", err)
|
||||
}
|
||||
|
||||
return &Locator{
|
||||
db: params.DB,
|
||||
dir: params.Dir,
|
||||
geojsURL: params.GeoJSURL,
|
||||
databaseURLFormat: params.DatabaseURLFormat,
|
||||
timeLimit: geojsTimeLimit,
|
||||
maxEntries: maxEntries,
|
||||
maxDatabaseBytes: maxDatabaseSize,
|
||||
logger: params.Logger.With("component", "geo"),
|
||||
client: &http.Client{},
|
||||
now: time.Now,
|
||||
entries: entries,
|
||||
pending: make(map[string]*Location),
|
||||
done: make(chan struct{}),
|
||||
}, nil
|
||||
}
|
||||
|
||||
// Lookup returns where each of addrs is, in the same order. The entry for an
|
||||
// address neither source answers for is nil. Addresses with no current cached
|
||||
// answer go to geojs in one request. The returned Locations are shared with
|
||||
// the cache and must not be changed.
|
||||
func (l *Locator) Lookup(ctx context.Context, addrs []netip.Addr) []*Location {
|
||||
locations := make([]*Location, len(addrs))
|
||||
|
||||
var unanswered []string
|
||||
|
||||
for i, addr := range addrs {
|
||||
address := plainAddress(addr)
|
||||
|
||||
locations[i] = l.cached(ctx, address)
|
||||
if locations[i] == nil {
|
||||
unanswered = append(unanswered, address)
|
||||
}
|
||||
}
|
||||
|
||||
if len(unanswered) == 0 {
|
||||
return locations
|
||||
}
|
||||
|
||||
answers, err := l.askGeoJS(ctx, unanswered)
|
||||
if err != nil {
|
||||
l.logger.Warn("geojs lookup failed; using the database", "error", err)
|
||||
}
|
||||
|
||||
for i, addr := range addrs {
|
||||
if locations[i] != nil {
|
||||
continue
|
||||
}
|
||||
|
||||
address := plainAddress(addr)
|
||||
|
||||
answer, ok := answers[address]
|
||||
if ok {
|
||||
l.store(address, answer)
|
||||
locations[i] = answer
|
||||
|
||||
continue
|
||||
}
|
||||
|
||||
locations[i] = l.fromDatabase(addr)
|
||||
}
|
||||
|
||||
return locations
|
||||
}
|
||||
|
||||
// plainAddress returns addr as the requests to geojs and the cache name it: an
|
||||
// IPv4 address mapped into IPv6 as plain IPv4, and with no zone. A client
|
||||
// chooses the zone, so it must not reach the request.
|
||||
func plainAddress(addr netip.Addr) string {
|
||||
return addr.Unmap().WithZone("").String()
|
||||
}
|
||||
|
||||
// Start loads the cached answers that have not expired and opens the copy of
|
||||
// the database, if there is one. It then starts the goroutines that write new
|
||||
// answers, delete expired ones and download the database. They run until Stop:
|
||||
// the end of ctx does not stop them.
|
||||
func (l *Locator) Start(ctx context.Context) error {
|
||||
err := l.load(ctx)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
l.logger.Info("Geolocation cache loaded", "entries", l.entries.Len())
|
||||
|
||||
l.openDatabaseCopy()
|
||||
|
||||
ctx, l.stop = context.WithCancel(context.WithoutCancel(ctx))
|
||||
|
||||
go func() {
|
||||
defer close(l.done)
|
||||
defer l.closeDatabase()
|
||||
|
||||
var workers sync.WaitGroup
|
||||
|
||||
workers.Go(func() { l.keepCache(ctx) })
|
||||
workers.Go(func() { l.keepDatabaseCurrent(ctx) })
|
||||
workers.Wait()
|
||||
}()
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
// Stop cancels a download in progress, writes the answers cache.db does not
|
||||
// have yet, and waits until the goroutines have returned or ctx ends.
|
||||
func (l *Locator) Stop(ctx context.Context) {
|
||||
l.stop()
|
||||
|
||||
select {
|
||||
case <-l.done:
|
||||
l.logger.Info("Geolocation stopped")
|
||||
case <-ctx.Done():
|
||||
l.logger.Warn("Stop deadline passed before the geolocation cache " +
|
||||
"was written")
|
||||
}
|
||||
}
|
||||
File diff suppressed because it is too large.
Load diff
@@ -21,7 +21,7 @@ const (
|
||||
downloadTimeout = time.Minute
|
||||
|
||||
// maxDownloadSize is the size of the largest list file accepted, in
|
||||
// bytes. The largest list, FireHOL abusers, is about 2.5 MiB.
|
||||
// bytes. The largest list, iCloud Private Relay, is about 12 MiB.
|
||||
maxDownloadSize = 200 << 20
|
||||
)
|
||||
|
||||
|
||||
@@ -51,6 +51,16 @@ const (
|
||||
// FormatIVPNJSON is IVPN's JSON server list. The hosts of its wireguard
|
||||
// and openvpn gateways have their addresses in host and v2ray.
|
||||
FormatIVPNJSON
|
||||
|
||||
// FormatProtonVPN has one address or network per line, with comment lines
|
||||
// that start with #. Unlike FormatLines, a file with no addresses or
|
||||
// networks fails.
|
||||
FormatProtonVPN
|
||||
|
||||
// FormatICloudPrivateRelayCSV is Apple's CSV file of iCloud Private Relay
|
||||
// networks. Each line starts with a network, followed by its country,
|
||||
// region and city.
|
||||
FormatICloudPrivateRelayCSV
|
||||
)
|
||||
|
||||
// List is one public IP list.
|
||||
|
||||
@@ -70,6 +70,31 @@ const (
|
||||
"config":{"api":{"ips":["192.0.2.73"]}}}`
|
||||
)
|
||||
|
||||
// protonVPNBody is a small file in the format of the ProtonVPN-IPs list.
|
||||
const protonVPNBody = "#\n# protonvpn_ip_ranges.txt\n" +
|
||||
"# https://github.com/tn3w/ProtonVPN-IPs\n#\n" +
|
||||
"# The /24 block of every advertised IPv4.\n#\n" +
|
||||
"192.0.2.0/24\n198.51.100.0/24\n"
|
||||
|
||||
// protonVPNEntryBody is a small file in the format of the ProtonVPN-IPs list
|
||||
// of server addresses.
|
||||
const protonVPNEntryBody = "#\n# protonvpn_entry_ips.txt\n" +
|
||||
"# https://github.com/tn3w/ProtonVPN-IPs\n#\n" +
|
||||
"# Entry IPs clients connect to, resolved from ProtonVPN hostnames.\n#\n" +
|
||||
"192.0.2.80\n2001:db8::80\n"
|
||||
|
||||
// iCloudPrivateRelayAddr is an address on iCloudPrivateRelayBody.
|
||||
const iCloudPrivateRelayAddr = "2001:db8:1::5"
|
||||
|
||||
// iCloudPrivateRelayBody is a small file in the format of Apple's iCloud
|
||||
// Private Relay file.
|
||||
const iCloudPrivateRelayBody = "192.0.2.0/31,GB,GB-EN,London,\n" +
|
||||
"198.51.100.32/27,US,US-CA,Los Angeles,\n" +
|
||||
"2001:db8:1::/64,US,US-AK,Anchorage,\n"
|
||||
|
||||
// emptyFileCase names the case of a wrong document that is an empty file.
|
||||
const emptyFileCase = "an empty file"
|
||||
|
||||
// errorPage is a file that is no list, as a publisher's error page is.
|
||||
const errorPage = "<!DOCTYPE html>\n<html></html>\n"
|
||||
|
||||
@@ -230,6 +255,15 @@ func TestLookupFindsListedAddresses(t *testing.T) {
|
||||
},
|
||||
unlisted: []string{"192.0.2.12", "2001:db8::2"},
|
||||
},
|
||||
{
|
||||
name: "iCloud Private Relay networks",
|
||||
format: FormatICloudPrivateRelayCSV,
|
||||
body: iCloudPrivateRelayBody,
|
||||
listed: []string{
|
||||
"192.0.2.1", "198.51.100.63", iCloudPrivateRelayAddr,
|
||||
},
|
||||
unlisted: []string{"192.0.2.2", "198.51.100.64", "2001:db8:2::1"},
|
||||
},
|
||||
})
|
||||
}
|
||||
|
||||
@@ -267,6 +301,20 @@ func TestLookupFindsVPNServers(t *testing.T) {
|
||||
listed: []string{"192.0.2.70", "192.0.2.71", "192.0.2.72"},
|
||||
unlisted: []string{"192.0.2.73", "2001:db8::70"},
|
||||
},
|
||||
{
|
||||
name: "ProtonVPN networks",
|
||||
format: FormatProtonVPN,
|
||||
body: protonVPNBody,
|
||||
listed: []string{"192.0.2.0", "192.0.2.255", "198.51.100.7"},
|
||||
unlisted: []string{"192.0.3.1", "2001:db8::90"},
|
||||
},
|
||||
{
|
||||
name: "ProtonVPN server addresses",
|
||||
format: FormatProtonVPN,
|
||||
body: protonVPNEntryBody,
|
||||
listed: []string{"192.0.2.80", "2001:db8::80"},
|
||||
unlisted: []string{"192.0.2.81", "2001:db8::81"},
|
||||
},
|
||||
})
|
||||
}
|
||||
|
||||
@@ -458,12 +506,12 @@ func TestWrongVPNDocumentFailsToParse(t *testing.T) {
|
||||
|
||||
for _, f := range formats {
|
||||
bodies := map[string]string{
|
||||
"not JSON": errorPage,
|
||||
"other JSON": `{"status":"ok","data":{"count":1}}`,
|
||||
"no addresses": f.empty,
|
||||
"bad address": f.badAddress,
|
||||
"cut off": f.good[:len(f.good)/2],
|
||||
"an empty file": "",
|
||||
"not JSON": errorPage,
|
||||
"other JSON": `{"status":"ok","data":{"count":1}}`,
|
||||
"no addresses": f.empty,
|
||||
"bad address": f.badAddress,
|
||||
"cut off": f.good[:len(f.good)/2],
|
||||
emptyFileCase: "",
|
||||
}
|
||||
|
||||
for name, body := range bodies {
|
||||
@@ -479,6 +527,59 @@ func TestWrongVPNDocumentFailsToParse(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
// TestWrongProtonVPNDocumentFailsToParse parses documents that are not a
|
||||
// ProtonVPN-IPs list. Each fails, so none can replace a good copy.
|
||||
func TestWrongProtonVPNDocumentFailsToParse(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
bodies := map[string]string{
|
||||
"error page": errorPage,
|
||||
"JSON": `{"status":"ok"}`,
|
||||
"comments only": "#\n# protonvpn_ip_ranges.txt\n#\n",
|
||||
"bad network": "192.0.2.0/24\nexample\n",
|
||||
emptyFileCase: "",
|
||||
}
|
||||
|
||||
for name, body := range bodies {
|
||||
t.Run(name, func(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
addresses, err := parse(strings.NewReader(body), FormatProtonVPN)
|
||||
if err == nil {
|
||||
t.Errorf("parse returned %v and no error", addresses.Ranges())
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// TestWrongICloudPrivateRelayDocumentFailsToParse parses documents that are
|
||||
// not Apple's iCloud Private Relay file. Each fails, so none can replace a
|
||||
// good copy.
|
||||
func TestWrongICloudPrivateRelayDocumentFailsToParse(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
bodies := map[string]string{
|
||||
"error page": errorPage,
|
||||
"JSON": `{"status":"ok","count":1}`,
|
||||
"no other field": "192.0.2.0/24\n",
|
||||
"bad network": "192.0.2.0/24,GB,GB-EN,London,\nexample,GB,GB-EN,London,\n",
|
||||
"an address": "192.0.2.1,GB,GB-EN,London,\n",
|
||||
emptyFileCase: "",
|
||||
}
|
||||
|
||||
for name, body := range bodies {
|
||||
t.Run(name, func(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
addresses, err := parse(strings.NewReader(body),
|
||||
FormatICloudPrivateRelayCSV)
|
||||
if err == nil {
|
||||
t.Errorf("parse returned %v and no error", addresses.Ranges())
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// TestLookupMarksVPNLists looks up an address on a VPN list and on another
|
||||
// list. Only the VPN list's listing says it is one.
|
||||
func TestLookupMarksVPNLists(t *testing.T) {
|
||||
@@ -523,7 +624,10 @@ func TestPublicListsMarkTheVPNLists(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
want := []string{"ivpn", "mullvad", "nordvpn", "pia", "x4bnet", "x4bnet_v6"}
|
||||
want := []string{
|
||||
"ivpn", "mullvad", "nordvpn", "pia", "protonvpn", "protonvpn_entry",
|
||||
"x4bnet", "x4bnet_v6",
|
||||
}
|
||||
if !slices.Equal(got, want) {
|
||||
t.Errorf("VPN lists %v, want %v", got, want)
|
||||
}
|
||||
@@ -830,15 +934,33 @@ func TestStopCancelsADownloadInProgress(t *testing.T) {
|
||||
func TestTorExitsListHoldsIPv6Exits(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
checkPublicListHolds(t, TorExitsList, onionooBody, onionooIPv6ExitAddr)
|
||||
}
|
||||
|
||||
// TestICloudPrivateRelayListHoldsItsNetworks downloads a file in Apple's
|
||||
// format as the public iCloud Private Relay list, which is what an address's
|
||||
// is_relay answer comes from.
|
||||
func TestICloudPrivateRelayListHoldsItsNetworks(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
checkPublicListHolds(t, ICloudPrivateRelayList, iCloudPrivateRelayBody,
|
||||
iCloudPrivateRelayAddr)
|
||||
}
|
||||
|
||||
// checkPublicListHolds downloads body as the public list called name and
|
||||
// checks that addr has one listing, which names that list.
|
||||
func checkPublicListHolds(t *testing.T, name, body, addr string) {
|
||||
t.Helper()
|
||||
|
||||
index := slices.IndexFunc(Lists(), func(list List) bool {
|
||||
return list.Name == TorExitsList
|
||||
return list.Name == name
|
||||
})
|
||||
if index < 0 {
|
||||
t.Fatalf("no list named %s", TorExitsList)
|
||||
t.Fatalf("no list named %s", name)
|
||||
}
|
||||
|
||||
list := Lists()[index]
|
||||
_, list.URL = newPublisher(t, onionooBody)
|
||||
_, list.URL = newPublisher(t, body)
|
||||
b := newTestIPLists(t.TempDir(), list)
|
||||
|
||||
err := b.download(t.Context(), list)
|
||||
@@ -846,9 +968,9 @@ func TestTorExitsListHoldsIPv6Exits(t *testing.T) {
|
||||
t.Fatalf("download: %v", err)
|
||||
}
|
||||
|
||||
listings := lookup(t, b, onionooIPv6ExitAddr)
|
||||
if len(listings) != 1 || listings[0].Name != TorExitsList {
|
||||
t.Errorf("listings %+v, want one for %s", listings, TorExitsList)
|
||||
listings := lookup(t, b, addr)
|
||||
if len(listings) != 1 || listings[0].Name != name {
|
||||
t.Errorf("listings %+v, want one for %s", listings, name)
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@@ -8,6 +8,13 @@ import (
|
||||
// TorExitsList is the name of the list of Tor exit nodes.
|
||||
const TorExitsList = "tor_exits"
|
||||
|
||||
// ICloudPrivateRelayList is the name of the list of iCloud Private Relay's
|
||||
// networks.
|
||||
const ICloudPrivateRelayList = "icloud_private_relay"
|
||||
|
||||
// protonVPNIPsURL is where the ProtonVPN-IPs mirror publishes its files.
|
||||
const protonVPNIPsURL = "https://raw.githubusercontent.com/tn3w/ProtonVPN-IPs/master/"
|
||||
|
||||
// How often each list is downloaded. No list is downloaded more often than
|
||||
// its publisher updates it or allows.
|
||||
const (
|
||||
@@ -27,6 +34,9 @@ const (
|
||||
fireHOLAbusersInterval = 5 * time.Hour
|
||||
fireHOLLevel3Interval = 3 * time.Hour
|
||||
|
||||
// Apple's iCloud Private Relay file is 12 MB, so once a day.
|
||||
iCloudPrivateRelayInterval = 24 * time.Hour
|
||||
|
||||
// Spamhaus asks that DROP be downloaded at most once a day:
|
||||
// https://www.spamhaus.org/blocklists/do-not-route-or-peer/
|
||||
spamhausDROPInterval = 24 * time.Hour
|
||||
@@ -81,6 +91,13 @@ func nonVPNLists() []List {
|
||||
Interval: fireHOLLevel3Interval,
|
||||
Format: FormatLines,
|
||||
},
|
||||
{
|
||||
Name: ICloudPrivateRelayList,
|
||||
Description: "In a network where iCloud Private Relay traffic leaves",
|
||||
URL: "https://mask-api.icloud.com/egress-ip-ranges.csv",
|
||||
Interval: iCloudPrivateRelayInterval,
|
||||
Format: FormatICloudPrivateRelayCSV,
|
||||
},
|
||||
{
|
||||
Name: "spamhaus_drop",
|
||||
Description: "In an IPv4 network that Spamhaus says spammers " +
|
||||
@@ -146,6 +163,28 @@ func vpnLists() []List {
|
||||
Format: FormatPIAJSON,
|
||||
VPN: true,
|
||||
},
|
||||
{
|
||||
// Proton's own server list needs a logged-in session, so this is
|
||||
// the public ProtonVPN-IPs mirror of it. Proton publishes only
|
||||
// the addresses clients connect to. The mirror's README says
|
||||
// traffic leaves from others in the same /24, so this file of
|
||||
// /24 networks is the one it says to use for detecting exits.
|
||||
Name: "protonvpn",
|
||||
Description: "In a /24 network that holds a server of the " +
|
||||
"ProtonVPN service",
|
||||
URL: protonVPNIPsURL + "protonvpn_ip_ranges.txt",
|
||||
Interval: vpnListInterval,
|
||||
Format: FormatProtonVPN,
|
||||
VPN: true,
|
||||
},
|
||||
{
|
||||
Name: "protonvpn_entry",
|
||||
Description: "A server of the ProtonVPN service",
|
||||
URL: protonVPNIPsURL + "protonvpn_entry_ips.txt",
|
||||
Interval: vpnListInterval,
|
||||
Format: FormatProtonVPN,
|
||||
VPN: true,
|
||||
},
|
||||
{
|
||||
Name: "x4bnet",
|
||||
Description: "In an IPv4 network of a VPN service, according to " +
|
||||
|
||||
@@ -25,6 +25,10 @@ const (
|
||||
// dshieldFieldSeparator separates the fields of a DShield line.
|
||||
dshieldFieldSeparator = "\t"
|
||||
|
||||
// iCloudPrivateRelayFieldSeparator separates the fields of a line of
|
||||
// Apple's iCloud Private Relay file.
|
||||
iCloudPrivateRelayFieldSeparator = ","
|
||||
|
||||
// spamhausMetadataType is the type of the line that ends a Spamhaus
|
||||
// file. That line holds the file's metadata and no network.
|
||||
spamhausMetadataType = "metadata"
|
||||
@@ -58,6 +62,10 @@ func parse(r io.Reader, format Format) (*netipx.IPSet, error) {
|
||||
return parsePIA(r)
|
||||
case FormatIVPNJSON:
|
||||
return parseIVPN(r)
|
||||
case FormatProtonVPN:
|
||||
return parseLinesNotEmpty(r, addAddressOrNetwork)
|
||||
case FormatICloudPrivateRelayCSV:
|
||||
return parseLinesNotEmpty(r, addICloudPrivateRelayNetwork)
|
||||
default:
|
||||
return nil, errUnknownFormat
|
||||
}
|
||||
@@ -329,6 +337,41 @@ func parseIVPN(r io.Reader) (*netipx.IPSet, error) {
|
||||
return addressSet(texts)
|
||||
}
|
||||
|
||||
// parseLinesNotEmpty is parseLines for a list that always holds addresses, so
|
||||
// a file with none fails.
|
||||
func parseLinesNotEmpty(
|
||||
r io.Reader, addLine func(*netipx.IPSetBuilder, string) error,
|
||||
) (*netipx.IPSet, error) {
|
||||
addresses, err := parseLines(r, addLine)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
if len(addresses.Ranges()) == 0 {
|
||||
return nil, errNoAddresses
|
||||
}
|
||||
|
||||
return addresses, nil
|
||||
}
|
||||
|
||||
// addICloudPrivateRelayNetwork adds the network at the start of a line of
|
||||
// Apple's iCloud Private Relay file to builder.
|
||||
func addICloudPrivateRelayNetwork(builder *netipx.IPSetBuilder, line string) error {
|
||||
field, _, found := strings.Cut(line, iCloudPrivateRelayFieldSeparator)
|
||||
if !found {
|
||||
return errNoNetwork
|
||||
}
|
||||
|
||||
network, err := netip.ParsePrefix(field)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
builder.AddPrefix(network)
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
// readJSON reads r to its end and decodes it, as one JSON document, into v.
|
||||
func readJSON(r io.Reader, v any) error {
|
||||
data, err := io.ReadAll(r)
|
||||
|
||||
@@ -19,6 +19,7 @@ import (
|
||||
"git.eeqj.de/sneak/routewatch/internal/cachedb"
|
||||
"git.eeqj.de/sneak/routewatch/internal/config"
|
||||
"git.eeqj.de/sneak/routewatch/internal/database"
|
||||
"git.eeqj.de/sneak/routewatch/internal/geo"
|
||||
"git.eeqj.de/sneak/routewatch/internal/iplist"
|
||||
"git.eeqj.de/sneak/routewatch/internal/logger"
|
||||
"git.eeqj.de/sneak/routewatch/internal/metrics"
|
||||
@@ -67,6 +68,7 @@ type RouteWatch struct {
|
||||
ipLists *iplist.IPLists
|
||||
cacheDB *sql.DB
|
||||
ptrCache *ptr.Cache
|
||||
geoLocator *geo.Locator
|
||||
}
|
||||
|
||||
// ipListsDirName is the directory in the state directory that holds the IP
|
||||
@@ -160,6 +162,11 @@ func (rw *RouteWatch) Run(ctx context.Context) error {
|
||||
return err
|
||||
}
|
||||
|
||||
err = rw.startGeoLocator(ctx)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
// Start HTTP server
|
||||
err = rw.server.Start(ctx)
|
||||
if err != nil {
|
||||
@@ -178,8 +185,9 @@ func (rw *RouteWatch) Run(ctx context.Context) error {
|
||||
|
||||
// Shutdown stops all services in order: the streamer, the handlers, which
|
||||
// write what they hold, the WHOIS fetcher, the database maintainer, the
|
||||
// IP list downloads, the HTTP server and the PTR cache, and last the database,
|
||||
// which it checkpoints and closes. Each stage waits at most until ctx ends, so
|
||||
// IP list downloads, the HTTP server, the geolocation cache and the PTR cache,
|
||||
// and last the database, which it checkpoints and closes. Each stage waits at
|
||||
// most until ctx ends, so
|
||||
// Shutdown returns soon after: a handler logs what it could not write by
|
||||
// then, and the HTTP server closes the connections of requests still
|
||||
// running.
|
||||
@@ -239,6 +247,10 @@ func (rw *RouteWatch) Shutdown(ctx context.Context) {
|
||||
"error", err)
|
||||
}
|
||||
|
||||
if rw.geoLocator != nil {
|
||||
rw.geoLocator.Stop(ctx)
|
||||
}
|
||||
|
||||
rw.stopPTRCache(ctx)
|
||||
|
||||
rw.closeDatabase(ctx)
|
||||
@@ -283,6 +295,32 @@ func (rw *RouteWatch) startPTRCache(ctx context.Context) error {
|
||||
return nil
|
||||
}
|
||||
|
||||
// startGeoLocator loads the geolocation cache from cache.db, which
|
||||
// startPTRCache has opened, and gives the server the locator. The HTTP handlers
|
||||
// read it without a lock, so it is set before they run.
|
||||
func (rw *RouteWatch) startGeoLocator(ctx context.Context) error {
|
||||
locator, err := geo.New(geo.Params{
|
||||
DB: rw.cacheDB,
|
||||
Dir: rw.config.StateDir,
|
||||
GeoJSURL: geo.GeoJSEndpoint,
|
||||
DatabaseURLFormat: geo.DatabaseDownloadFormat,
|
||||
Logger: rw.logger.Logger,
|
||||
})
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
err = locator.Start(ctx)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
rw.geoLocator = locator
|
||||
rw.server.SetGeoLocator(locator)
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
// stopPTRCache stops the PTR cache, which writes the answers cache.db does not
|
||||
// have yet, and closes cache.db, unless ctx has ended and a write may still be
|
||||
// running.
|
||||
|
||||
@@ -17,6 +17,7 @@ import (
|
||||
|
||||
"git.eeqj.de/sneak/routewatch/internal/bogon"
|
||||
"git.eeqj.de/sneak/routewatch/internal/database"
|
||||
"git.eeqj.de/sneak/routewatch/internal/geo"
|
||||
"git.eeqj.de/sneak/routewatch/internal/iplist"
|
||||
"git.eeqj.de/sneak/routewatch/internal/metrics"
|
||||
"git.eeqj.de/sneak/routewatch/internal/ristypes"
|
||||
@@ -764,9 +765,9 @@ type IPLookupResponse struct {
|
||||
}
|
||||
|
||||
// IPLookupResult is the answer for one address: its PTR names, its flags, its
|
||||
// live route and that route's AS, and the IP lists that contain it. Every
|
||||
// result has every field. The fields of IPInfo are null when the address has
|
||||
// no live route.
|
||||
// live route and that route's AS, the IP lists that contain it, and where it
|
||||
// is. Every result has every field. The fields of IPInfo are null when the
|
||||
// address has no live route.
|
||||
//
|
||||
//nolint:tagliatelle // snake_case keys, as the Go style guide asks
|
||||
type IPLookupResult struct {
|
||||
@@ -784,6 +785,9 @@ type IPLookupResult struct {
|
||||
|
||||
// IPLists has one entry for each IP list the address is on.
|
||||
IPLists []iplist.Listing `json:"ip_lists"`
|
||||
|
||||
// Geo is where the address is. It is nil when no source answers.
|
||||
Geo *geo.Location `json:"geo"`
|
||||
}
|
||||
|
||||
// IPLookupFlags are the yes-or-no facts about an address. Every result has
|
||||
@@ -797,6 +801,9 @@ type IPLookupFlags struct {
|
||||
// IsVPN says that the address is on a list of VPN servers.
|
||||
IsVPN bool `json:"is_vpn"`
|
||||
|
||||
// IsRelay says that the address is on the iCloud Private Relay list.
|
||||
IsRelay bool `json:"is_relay"`
|
||||
|
||||
// IsBogon says that the address is reserved, as package bogon tells.
|
||||
IsBogon bool `json:"is_bogon"`
|
||||
}
|
||||
@@ -839,6 +846,8 @@ func (s *Server) handleIPInfo() http.HandlerFunc {
|
||||
return
|
||||
}
|
||||
|
||||
locations := s.geoLocations(ctx, ipsToLookup)
|
||||
|
||||
// Track ASNs that need WHOIS refresh
|
||||
refreshASNs := make(map[int]bool)
|
||||
|
||||
@@ -874,7 +883,9 @@ func (s *Server) handleIPInfo() http.HandlerFunc {
|
||||
return
|
||||
}
|
||||
|
||||
response.Results = append(response.Results, s.ipLookupResult(ip, ptrs, ipInfo))
|
||||
result := s.ipLookupResult(ip, ptrs, ipInfo)
|
||||
result.Geo = locations[ip]
|
||||
response.Results = append(response.Results, result)
|
||||
|
||||
if ipInfo.NeedsWHOISRefresh {
|
||||
refreshASNs[*ipInfo.ASN] = true
|
||||
@@ -1025,10 +1036,52 @@ func ipLookupFlags(ip string, listings []iplist.Listing) IPLookupFlags {
|
||||
IsVPN: slices.ContainsFunc(listings, func(l iplist.Listing) bool {
|
||||
return l.VPN
|
||||
}),
|
||||
IsRelay: slices.ContainsFunc(listings, func(l iplist.Listing) bool {
|
||||
return l.Name == iplist.ICloudPrivateRelayList
|
||||
}),
|
||||
IsBogon: isBogon(ip),
|
||||
}
|
||||
}
|
||||
|
||||
// geoLocations returns where ips are, by address. An address that does not
|
||||
// parse has no entry. Nor has a reserved one, which is sent to no outside
|
||||
// source and has no location.
|
||||
func (s *Server) geoLocations(
|
||||
ctx context.Context, ips []string,
|
||||
) map[string]*geo.Location {
|
||||
if s.geoLocator == nil {
|
||||
return nil
|
||||
}
|
||||
|
||||
var (
|
||||
kept []string
|
||||
addrs []netip.Addr
|
||||
)
|
||||
|
||||
for _, ip := range ips {
|
||||
addr, err := netip.ParseAddr(ip)
|
||||
if err != nil || isBogon(ip) {
|
||||
continue
|
||||
}
|
||||
|
||||
kept = append(kept, ip)
|
||||
addrs = append(addrs, addr)
|
||||
}
|
||||
|
||||
if len(addrs) == 0 {
|
||||
return nil
|
||||
}
|
||||
|
||||
answers := s.geoLocator.Lookup(ctx, addrs)
|
||||
locations := make(map[string]*geo.Location, len(kept))
|
||||
|
||||
for i, ip := range kept {
|
||||
locations[ip] = answers[i]
|
||||
}
|
||||
|
||||
return locations
|
||||
}
|
||||
|
||||
// isBogon says whether the address ip is reserved, as package bogon tells.
|
||||
func isBogon(ip string) bool {
|
||||
addr, err := netip.ParseAddr(ip)
|
||||
|
||||
@@ -15,6 +15,7 @@ import (
|
||||
"slices"
|
||||
"strconv"
|
||||
"strings"
|
||||
"sync"
|
||||
"sync/atomic"
|
||||
"testing"
|
||||
"testing/synctest"
|
||||
@@ -22,6 +23,7 @@ import (
|
||||
|
||||
"git.eeqj.de/sneak/routewatch/internal/config"
|
||||
"git.eeqj.de/sneak/routewatch/internal/database"
|
||||
"git.eeqj.de/sneak/routewatch/internal/geo"
|
||||
"git.eeqj.de/sneak/routewatch/internal/iplist"
|
||||
"git.eeqj.de/sneak/routewatch/internal/logger"
|
||||
"git.eeqj.de/sneak/routewatch/internal/metrics"
|
||||
@@ -1504,6 +1506,10 @@ const listedUnroutedAddr = "203.0.113.1"
|
||||
// torExitAddr is the address that fakeIPLists lists on the Tor exit list.
|
||||
const torExitAddr = "203.0.113.2"
|
||||
|
||||
// relayAddr is the address that fakeIPLists lists on the iCloud Private Relay
|
||||
// list.
|
||||
const relayAddr = "203.0.113.5"
|
||||
|
||||
// The addresses that fakeIPLists lists on one and on two VPN lists, and the
|
||||
// names of those lists.
|
||||
const (
|
||||
@@ -1513,10 +1519,11 @@ const (
|
||||
otherTestVPN = "other_test_vpn"
|
||||
)
|
||||
|
||||
// fakeIPLists has four lists. The test list holds addrInTestPrefixV4,
|
||||
// fakeIPLists has five lists. The test list holds addrInTestPrefixV4,
|
||||
// listedUnroutedAddr and twoVPNListsAddr. The Tor exit list holds torExitAddr.
|
||||
// The test VPN list holds oneVPNListAddr and twoVPNListsAddr, and the other
|
||||
// test VPN list holds twoVPNListsAddr.
|
||||
// The iCloud Private Relay list holds relayAddr. The test VPN list holds
|
||||
// oneVPNListAddr and twoVPNListsAddr, and the other test VPN list holds
|
||||
// twoVPNListsAddr.
|
||||
type fakeIPLists struct{}
|
||||
|
||||
func testListing() iplist.Listing {
|
||||
@@ -1537,6 +1544,10 @@ func (fakeIPLists) Lookup(addr netip.Addr) []iplist.Listing {
|
||||
return []iplist.Listing{{Name: iplist.TorExitsList}}
|
||||
}
|
||||
|
||||
if addr == netip.MustParseAddr(relayAddr) {
|
||||
return []iplist.Listing{{Name: iplist.ICloudPrivateRelayList}}
|
||||
}
|
||||
|
||||
if addr == netip.MustParseAddr(oneVPNListAddr) {
|
||||
return []iplist.Listing{{Name: testVPNList, VPN: true}}
|
||||
}
|
||||
@@ -1569,20 +1580,22 @@ func TestIPLookupFlags(t *testing.T) {
|
||||
name string
|
||||
addr string
|
||||
tor bool
|
||||
relay bool
|
||||
bogon bool
|
||||
}{
|
||||
{"on the Tor exit list", torExitAddr, true, true},
|
||||
{"on another list", listedUnroutedAddr, false, true},
|
||||
{"routed IPv4", addrInTestPrefixV4, false, true},
|
||||
{"routed IPv6", addrInTestPrefixV6, false, true},
|
||||
{"private IPv4", "10.0.0.1", false, true},
|
||||
{"loopback IPv4", "127.0.0.1", false, true},
|
||||
{"public IPv4", "1.1.1.1", false, false},
|
||||
{"private IPv6", "fd00::1", false, true},
|
||||
{"loopback IPv6", "::1", false, true},
|
||||
{"public IPv6", "2606:4700:4700::1111", false, false},
|
||||
{"public IPv4 in IPv6 form", "::ffff:1.1.1.1", false, false},
|
||||
{"private IPv4 in IPv6 form", "::ffff:10.0.0.1", false, true},
|
||||
{"on the Tor exit list", torExitAddr, true, false, true},
|
||||
{"on the iCloud Private Relay list", relayAddr, false, true, true},
|
||||
{"on another list", listedUnroutedAddr, false, false, true},
|
||||
{"routed IPv4", addrInTestPrefixV4, false, false, true},
|
||||
{"routed IPv6", addrInTestPrefixV6, false, false, true},
|
||||
{"private IPv4", "10.0.0.1", false, false, true},
|
||||
{"loopback IPv4", "127.0.0.1", false, false, true},
|
||||
{"public IPv4", "1.1.1.1", false, false, false},
|
||||
{"private IPv6", "fd00::1", false, false, true},
|
||||
{"loopback IPv6", "::1", false, false, true},
|
||||
{"public IPv6", "2606:4700:4700::1111", false, false, false},
|
||||
{"public IPv4 in IPv6 form", "::ffff:1.1.1.1", false, false, false},
|
||||
{"private IPv4 in IPv6 form", "::ffff:10.0.0.1", false, false, true},
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
@@ -1604,7 +1617,8 @@ func TestIPLookupFlags(t *testing.T) {
|
||||
}
|
||||
|
||||
want := map[string]bool{
|
||||
"is_tor": tt.tor, "is_vpn": false, "is_bogon": tt.bogon,
|
||||
"is_tor": tt.tor, "is_vpn": false, "is_relay": tt.relay,
|
||||
"is_bogon": tt.bogon,
|
||||
}
|
||||
if !maps.Equal(got, want) {
|
||||
t.Errorf("%s: flags %v, want %v", path, got, want)
|
||||
@@ -1613,6 +1627,131 @@ func TestIPLookupFlags(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
// fakeGeoLocator answers every address with location, and keeps the addresses
|
||||
// of each call. It is safe for the concurrent subtests that share it.
|
||||
type fakeGeoLocator struct {
|
||||
location *geo.Location
|
||||
|
||||
mu sync.Mutex
|
||||
calls [][]netip.Addr
|
||||
}
|
||||
|
||||
func (f *fakeGeoLocator) Lookup(
|
||||
_ context.Context, addrs []netip.Addr,
|
||||
) []*geo.Location {
|
||||
f.mu.Lock()
|
||||
defer f.mu.Unlock()
|
||||
|
||||
f.calls = append(f.calls, addrs)
|
||||
|
||||
answers := make([]*geo.Location, len(addrs))
|
||||
for i := range addrs {
|
||||
answers[i] = f.location
|
||||
}
|
||||
|
||||
return answers
|
||||
}
|
||||
|
||||
// TestIPLookupGivesTheGeoObject looks up two public addresses. They go to the
|
||||
// locator in one call, and each result has the geo object under the keys the
|
||||
// API documents. These addresses are public, because a reserved one is not
|
||||
// looked up.
|
||||
func TestIPLookupGivesTheGeoObject(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
s := newServerWithTestRoutes(t)
|
||||
useFakeResolver(s, &fakeResolver{})
|
||||
|
||||
locator := &fakeGeoLocator{location: &geo.Location{
|
||||
CountryCode: new("DE"),
|
||||
City: new("Berlin"),
|
||||
Latitude: new(52.5),
|
||||
Source: "geojs",
|
||||
Updated: testRouteTime(0),
|
||||
}}
|
||||
s.SetGeoLocator(locator)
|
||||
|
||||
path := "/api/v1/ip/1.1.1.1,8.8.8.8"
|
||||
|
||||
results := resultFields(t, get(t, s, path).Body.Bytes())
|
||||
if len(results) != 2 {
|
||||
t.Fatalf("%s: %d results, want 2", path, len(results))
|
||||
}
|
||||
|
||||
if len(locator.calls) != 1 || len(locator.calls[0]) != 2 {
|
||||
t.Errorf("locator calls %v, want one with both addresses", locator.calls)
|
||||
}
|
||||
|
||||
var got map[string]json.RawMessage
|
||||
|
||||
err := json.Unmarshal(results[0]["geo"], &got)
|
||||
if err != nil {
|
||||
t.Fatalf("decoding geo %s: %v", results[0]["geo"], err)
|
||||
}
|
||||
|
||||
wantKeys := []string{
|
||||
"accuracy_radius_km", "city", "continent_code", "country",
|
||||
"country_code", "latitude", "longitude", "region", "source",
|
||||
"time_zone", "updated",
|
||||
}
|
||||
if keys := slices.Sorted(maps.Keys(got)); !slices.Equal(keys, wantKeys) {
|
||||
t.Errorf("geo keys %v, want %v", keys, wantKeys)
|
||||
}
|
||||
|
||||
if string(got["city"]) != `"Berlin"` || string(got["latitude"]) != "52.5" ||
|
||||
string(got["region"]) != jsonNull {
|
||||
t.Errorf("wrong geo %s", results[0]["geo"])
|
||||
}
|
||||
}
|
||||
|
||||
// TestIPLookupOfAReservedAddressHasNoGeo looks up a reserved address alone and
|
||||
// beside a public one. The reserved one never reaches the locator, and its geo
|
||||
// is null.
|
||||
func TestIPLookupOfAReservedAddressHasNoGeo(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
s := newServerWithTestRoutes(t)
|
||||
useFakeResolver(s, &fakeResolver{})
|
||||
|
||||
locator := &fakeGeoLocator{location: &geo.Location{Source: "geojs"}}
|
||||
s.SetGeoLocator(locator)
|
||||
|
||||
results := resultFields(t, get(t, s, "/api/v1/ip/"+addrInTestPrefixV4).Body.Bytes())
|
||||
if len(results) != 1 || string(results[0]["geo"]) != jsonNull {
|
||||
t.Fatalf("results %v, want one with geo null", results)
|
||||
}
|
||||
|
||||
if len(locator.calls) != 0 {
|
||||
t.Errorf("locator calls %v, want none", locator.calls)
|
||||
}
|
||||
|
||||
results = resultFields(t,
|
||||
get(t, s, "/api/v1/ip/"+addrInTestPrefixV4+",1.1.1.1").Body.Bytes())
|
||||
if len(results) != 2 || string(results[0]["geo"]) != jsonNull ||
|
||||
string(results[1]["geo"]) == jsonNull {
|
||||
t.Errorf("results %v, want geo only for the public address", results)
|
||||
}
|
||||
|
||||
if len(locator.calls) != 1 || len(locator.calls[0]) != 1 {
|
||||
t.Errorf("locator calls %v, want one with one address", locator.calls)
|
||||
}
|
||||
}
|
||||
|
||||
// TestIPLookupGeoIsNullWhenNoSourceAnswers checks an address the locator has
|
||||
// no answer for.
|
||||
func TestIPLookupGeoIsNullWhenNoSourceAnswers(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
s := newServerWithTestRoutes(t)
|
||||
useFakeResolver(s, &fakeResolver{})
|
||||
s.SetGeoLocator(&fakeGeoLocator{})
|
||||
|
||||
results := resultFields(t, get(t, s, "/api/v1/ip/1.1.1.1").Body.Bytes())
|
||||
if len(results) != 1 || string(results[0]["geo"]) != jsonNull {
|
||||
t.Errorf("results %v, want one with geo null", results)
|
||||
}
|
||||
}
|
||||
|
||||
// TestIPLookupReportsVPNProviders looks up addresses on no list, on a list
|
||||
// that is not a VPN list, on one VPN list and on two. is_vpn says whether any
|
||||
// VPN list holds the address, and vpn_providers names only those lists. An
|
||||
|
||||
@@ -24,6 +24,7 @@ import (
|
||||
|
||||
"git.eeqj.de/sneak/routewatch/internal/config"
|
||||
"git.eeqj.de/sneak/routewatch/internal/database"
|
||||
"git.eeqj.de/sneak/routewatch/internal/geo"
|
||||
"git.eeqj.de/sneak/routewatch/internal/iplist"
|
||||
"git.eeqj.de/sneak/routewatch/internal/logger"
|
||||
"git.eeqj.de/sneak/routewatch/internal/routetable"
|
||||
@@ -63,6 +64,12 @@ type PTRResolver interface {
|
||||
LookupAddr(ctx context.Context, addr string) ([]string, error)
|
||||
}
|
||||
|
||||
// GeoLocator finds where addresses are, as *geo.Locator does. It returns one
|
||||
// entry for each address, nil when no source answers.
|
||||
type GeoLocator interface {
|
||||
Lookup(ctx context.Context, addrs []netip.Addr) []*geo.Location
|
||||
}
|
||||
|
||||
// Server provides HTTP endpoints for status monitoring
|
||||
type Server struct {
|
||||
router *chi.Mux
|
||||
@@ -86,6 +93,10 @@ type Server struct {
|
||||
// ptrResolver looks up the IP lookup's PTR names. SetPTRResolver sets it.
|
||||
ptrResolver PTRResolver
|
||||
|
||||
// geoLocator finds where the IP lookup's addresses are. SetGeoLocator
|
||||
// sets it.
|
||||
geoLocator GeoLocator
|
||||
|
||||
// cards is the copy of the home page's cards that the pages serve, nil
|
||||
// until the server first works them out
|
||||
cards atomic.Pointer[homeCards]
|
||||
@@ -209,6 +220,12 @@ func (s *Server) SetPTRResolver(resolver PTRResolver) {
|
||||
s.ptrResolver = resolver
|
||||
}
|
||||
|
||||
// SetGeoLocator sets what finds where the IP lookup's addresses are. Call it
|
||||
// before Start: the handlers read it without a lock.
|
||||
func (s *Server) SetGeoLocator(locator GeoLocator) {
|
||||
s.geoLocator = locator
|
||||
}
|
||||
|
||||
// SetASNFetcher sets the ASN WHOIS fetcher for on-demand lookups. Call it
|
||||
// before Start: the handlers read the fetcher without a lock.
|
||||
func (s *Server) SetASNFetcher(fetcher ASNFetcher) {
|
||||
|
||||
@@ -544,6 +544,8 @@
|
||||
<span>{{appLicense}}</span>
|
||||
<span class="separator">|</span>
|
||||
<span><a href="{{appGitCommitURL}}">{{appGitRevision}}</a></span>
|
||||
<span class="separator">|</span>
|
||||
<span>IP Geolocation by <a href="https://db-ip.com">DB-IP</a></span>
|
||||
</footer>
|
||||
|
||||
<script nonce="{{.CSPNonce}}">
|
||||
|
||||
@@ -67,6 +67,23 @@ func TestFooterLicense(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
// TestHomePageCreditsDBIP expects the home page's footer to carry the credit
|
||||
// to DB-IP.
|
||||
func TestHomePageCreditsDBIP(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
const attribution = `IP Geolocation by <a href="https://db-ip.com">DB-IP</a>`
|
||||
|
||||
_, footer, found := strings.Cut(renderPage(t, templates.IndexTemplate()), "<footer")
|
||||
if !found {
|
||||
t.Fatal("the page has no footer")
|
||||
}
|
||||
|
||||
if !strings.Contains(footer, attribution) {
|
||||
t.Errorf("footer does not credit DB-IP: %q", footer)
|
||||
}
|
||||
}
|
||||
|
||||
// TestFreshnessIndicator renders the home and status pages and expects each to
|
||||
// have, right after the status element, the element where the page's script
|
||||
// shows its spinner or "stale".
|
||||
|
||||
Reference in new issue
Block a user