Compare commits
12
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
 clawbot
|
5111697885 | ||
|
clawbot
|
caed69249c | ||
|
clawbot
|
760ba4746b | ||
|
clawbot
|
dcb3935643 | ||
|
clawbot
|
6de44ac054 | ||
|
clawbot
|
cca402c73d | ||
|
clawbot
|
2c3f6f71dc | ||
|
clawbot
|
b2553f044d | ||
|
clawbot
|
0cbd9cd943 | ||
|
clawbot
|
641798bb22 | ||
|
clawbot
|
28ef0275ba | ||
|
clawbot
|
3204b7ee4b |
No files matched your search
+3
-2
@@ -93,11 +93,12 @@
|
||||
**/*.dylib
|
||||
|
||||
# Local databases. The image carries a source archive of the whole build
|
||||
# context, so these would otherwise ship inside it. The daemon's database
|
||||
# is db.sqlite, with its -wal and -shm files while it runs.
|
||||
# context, so these would otherwise ship inside it. The daemon's databases
|
||||
# are db.sqlite and cache.db, with their -wal and -shm files while it runs.
|
||||
**/*.db
|
||||
**/*.db-journal
|
||||
**/*.db-wal
|
||||
**/*.db-shm
|
||||
**/*.sqlite
|
||||
**/*.sqlite-wal
|
||||
**/*.sqlite-shm
|
||||
|
||||
+3
-1
@@ -68,10 +68,12 @@ go.work
|
||||
go.work.sum
|
||||
|
||||
# Local databases, temporary files and debug output. The daemon's
|
||||
# database is db.sqlite, with its -wal and -shm files while it runs.
|
||||
# databases are db.sqlite and cache.db, with their -wal and -shm files
|
||||
# while it runs.
|
||||
*.db
|
||||
*.db-journal
|
||||
*.db-wal
|
||||
*.db-shm
|
||||
*.sqlite
|
||||
*.sqlite-wal
|
||||
*.sqlite-shm
|
||||
|
||||
+1
-1
@@ -105,7 +105,7 @@ RUN apt-get update && apt-get install -y --no-install-recommends \
|
||||
ARG CACHEBUST
|
||||
|
||||
# Create non-root user
|
||||
RUN useradd -r -u 1000 -m routewatch
|
||||
RUN useradd -u 1000 -m routewatch
|
||||
|
||||
RUN mkdir -p /var/lib/berlin.sneak.app.routewatch && chown routewatch:routewatch /var/lib/berlin.sneak.app.routewatch
|
||||
|
||||
|
||||
@@ -63,8 +63,8 @@ docker run --memory=5g --memory-swap=5g -p 8080:8080 \
|
||||
-v routewatch:/var/lib/berlin.sneak.app.routewatch routewatch
|
||||
```
|
||||
|
||||
The memory limit is explained in Memory. The volume holds the database and the
|
||||
copies of the IP lists.
|
||||
The memory limit is explained in Memory. The volume holds the database, the
|
||||
copies of the IP lists and the geolocation files.
|
||||
|
||||
To run the binary on the host instead, build it with `make build`. The build
|
||||
needs Go and a C compiler, because the SQLite library is written in C:
|
||||
@@ -143,9 +143,9 @@ and origin AS can have two kept routes, one or none.
|
||||
AS path, a next hop and a time. The next hop is the router the peer sends the
|
||||
prefix's traffic to. RouteWatch records the peer itself as the next hop. The
|
||||
page also lists the prefix's origin changes, newest first. An origin change is
|
||||
an AS starting to announce the prefix, when its first peer carries it, or
|
||||
stopping, when its last peer no longer does. The page shows at most the newest
|
||||
5,000. Each row shows:
|
||||
recorded when the first peer starts to carry the prefix with an origin AS, or
|
||||
when the last peer stops. The page shows at most the newest 5,000. Each row
|
||||
shows:
|
||||
- the time
|
||||
- announced or withdrawn
|
||||
- the origin AS
|
||||
@@ -197,8 +197,9 @@ also carries an `@meta` object:
|
||||
- `api_version`: `1`
|
||||
- `execution_time_ms`: the number of milliseconds the request took
|
||||
|
||||
Every value with a unit is a number, and the unit ends the field name. Examples
|
||||
are `_ms`, `_seconds`, `_bytes`, `_per_sec` and `_per_min`.
|
||||
Every value with a unit is a number. The unit ends the field name, as in `_ms`,
|
||||
`_seconds`, `_bytes`, `_per_sec` and `_per_min`. `latitude` and `longitude` are
|
||||
in degrees, and their names do not say so.
|
||||
|
||||
The paths:
|
||||
|
||||
@@ -281,14 +282,20 @@ answer holds `query`, the text looked up, and `results`, with one entry for each
|
||||
address. Every entry has the same fields:
|
||||
|
||||
- `ip`: the address
|
||||
- `ptr`: the address's PTR names, the host names that DNS gives for it, or an
|
||||
empty list when it has none
|
||||
- `ptr`: the address's PTR names, the host names that DNS gives for it (see PTR
|
||||
lookups), or an empty list when it has none
|
||||
- `flags`: yes-or-no facts about the address:
|
||||
- `is_tor`: true when the address is on the `tor_exits` list
|
||||
- `is_vpn`: true when the address is on any list of VPN servers (see IP
|
||||
lists)
|
||||
- `is_relay`: true when the address is on the `icloud_private_relay` list
|
||||
- `is_bogon`: true when the address is reserved, so that no network should
|
||||
announce it on the internet
|
||||
- `vpn_providers`: the names of the lists of VPN servers the address is on, or
|
||||
an empty list when it is on none
|
||||
- `ip_lists`: the public IP lists the address is on (see IP lists), or an empty
|
||||
list when it is on none
|
||||
- `geo`: where the address is (see Geolocation), or null when no source answers
|
||||
|
||||
Every other field is about the prefix in the routing table that holds the
|
||||
address. Each is null when no prefix in the routing table holds the address:
|
||||
@@ -330,9 +337,6 @@ address, such as one that resolves to a private address, is reported as
|
||||
- `{ip}` comes to more than 10 addresses once its names are resolved
|
||||
- nothing in `{ip}` is an IP address or a host name with a public address
|
||||
|
||||
Each PTR lookup gets 2 seconds. An address whose lookup takes longer is answered
|
||||
without PTR names.
|
||||
|
||||
A successful answer carries `Cache-Control` with a `max-age` of 15 minutes, the
|
||||
shortest IP list interval. The answer for the caller's own address (`/ip/`) is
|
||||
`private`, and the others are `public`.
|
||||
@@ -420,6 +424,27 @@ The lists:
|
||||
last 30 days. From
|
||||
[FireHOL](https://iplists.firehol.org/?ipset=firehol_level3), downloaded every
|
||||
3 hours.
|
||||
- `icloud_private_relay`: the networks where iCloud Private Relay traffic
|
||||
leaves. From [Apple](https://mask-api.icloud.com/egress-ip-ranges.csv),
|
||||
downloaded once a day.
|
||||
- `ivpn`: VPN servers run by IVPN. From
|
||||
[IVPN](https://api.ivpn.net/v5/servers.json), downloaded once a day.
|
||||
- `mullvad`: VPN servers run by Mullvad. From
|
||||
[Mullvad](https://api.mullvad.net/www/relays/all/), downloaded once a day.
|
||||
- `nordvpn`: VPN servers run by NordVPN. From
|
||||
[NordVPN](https://api.nordvpn.com/v1/servers?limit=0&fields[servers.station]&fields[servers.ipv6_station]&fields[servers.hostname]),
|
||||
downloaded once a day.
|
||||
- `pia`: VPN servers run by Private Internet Access. From
|
||||
[Private Internet Access](https://serverlist.piaservers.net/vpninfo/servers/v6),
|
||||
downloaded once a day.
|
||||
- `protonvpn`: the /24 networks that hold ProtonVPN servers. The whole /24 is
|
||||
listed because the ProtonVPN-IPs README says traffic leaves from unpublished
|
||||
addresses in it. From the public
|
||||
[ProtonVPN-IPs](https://github.com/tn3w/ProtonVPN-IPs) mirror of Proton's
|
||||
server list, downloaded once a day.
|
||||
- `protonvpn_entry`: the IPv4 and IPv6 addresses of ProtonVPN servers. From the
|
||||
same [ProtonVPN-IPs](https://github.com/tn3w/ProtonVPN-IPs) mirror, downloaded
|
||||
once a day.
|
||||
- `spamhaus_drop` and `spamhaus_dropv6`: IPv4 and IPv6 networks hijacked or
|
||||
leased by spammers or cyber criminals, according to Spamhaus. From
|
||||
[Spamhaus DROP](https://www.spamhaus.org/blocklists/do-not-route-or-peer/),
|
||||
@@ -428,6 +453,9 @@ The lists:
|
||||
the Tor network leaves. From the Tor Project's
|
||||
[Onionoo](https://metrics.torproject.org/onionoo.html) service, downloaded
|
||||
every hour.
|
||||
- `x4bnet` and `x4bnet_v6`: IPv4 and IPv6 networks of VPN services, according to
|
||||
a community list that names no service. From
|
||||
[X4BNet](https://github.com/X4BNet/lists_vpn), downloaded once a day.
|
||||
|
||||
No list is downloaded more often than its publisher updates it or allows. Each
|
||||
can be downloaded whole without an account or a key.
|
||||
@@ -448,6 +476,114 @@ How RouteWatch keeps the lists:
|
||||
- When a download fails, or its file does not parse, RouteWatch keeps the
|
||||
previous copy and logs the failure. It tries again one interval later.
|
||||
|
||||
## PTR lookups
|
||||
|
||||
An IP lookup finds each address's PTR names over DNS-over-HTTPS. It asks these
|
||||
public resolvers in order, until one answers:
|
||||
|
||||
- Google: `https://dns.google/dns-query`
|
||||
- Cloudflare: `https://cloudflare-dns.com/dns-query`
|
||||
- AdGuard: `https://unfiltered.adguard-dns.com/dns-query`
|
||||
- Quad9: `https://dns.quad9.net/dns-query`
|
||||
|
||||
One address's lookup gets 2 seconds across all four. Each resolver gets an equal
|
||||
share of the time left, so one that does not answer leaves time for the next.
|
||||
When none answers in time, the address's `ptr` is an empty list.
|
||||
|
||||
A reserved address, one whose `is_bogon` flag is true, is not sent to the
|
||||
resolvers. Its `ptr` is an empty list.
|
||||
|
||||
What goes outside: each resolver asked receives the address being looked up. For
|
||||
`/ip/` and the home page, that address is the visitor's own. A lookup answered
|
||||
from the cache sends nothing.
|
||||
|
||||
How RouteWatch keeps the answers:
|
||||
|
||||
- It holds answers in memory. An answer that has not expired is served from
|
||||
there.
|
||||
- It writes new answers every 5 seconds to `cache.db` in the state directory
|
||||
(see Configuration), and loads them from there at startup.
|
||||
- Each DNS record carries a TTL, the time for which it may be kept. An answer
|
||||
expires after the shortest TTL of its records. RouteWatch keeps it for at
|
||||
least 60 seconds and at most 24 hours.
|
||||
- When an address has no PTR names, the answer's SOA record says how long that
|
||||
answer may be kept. The answer expires after the SOA record's minimum field,
|
||||
or the record's own TTL when that is shorter. RouteWatch keeps it for at most
|
||||
1 hour.
|
||||
- An answer with no PTR names and no SOA record is not kept. Neither is one
|
||||
whose SOA record has a TTL or minimum field of 0.
|
||||
- A failed lookup is not kept, so the next lookup of the address asks again.
|
||||
- Every 10 minutes it deletes expired answers from memory and from `cache.db`.
|
||||
- It keeps answers for at most 250,000 addresses, and keeps no answer for a new
|
||||
address while it holds that many.
|
||||
|
||||
## Geolocation
|
||||
|
||||
An IP lookup finds where each address is, on a best-effort basis. It uses two
|
||||
sources:
|
||||
|
||||
- [geojs](https://www.geojs.io/) is asked first. One request carries every
|
||||
address of the lookup that the cache does not hold. geojs has 1 second to
|
||||
answer.
|
||||
- [DB-IP City Lite](https://db-ip.com/db/download/ip-to-city-lite) answers when
|
||||
geojs has no answer for the address. It is a database file that RouteWatch
|
||||
reads itself.
|
||||
|
||||
geojs has no answer when its request fails or runs out of time. An answer with
|
||||
no country, city or coordinates also counts as no answer, and RouteWatch does
|
||||
not cache it.
|
||||
|
||||
The `geo` object of a result has these fields:
|
||||
|
||||
- `country_code`: the two-letter country code
|
||||
- `country`: the country's name in English
|
||||
- `continent_code`: the two-letter continent code
|
||||
- `region`: the region or state
|
||||
- `city`: the city
|
||||
- `latitude` and `longitude`: the location in degrees
|
||||
- `accuracy_radius_km`: how far from that location the address may be, in km
|
||||
- `time_zone`: the time zone's name, such as `Europe/Berlin`
|
||||
- `source`: the source of the answer, `geojs` or `dbip`
|
||||
- `updated`: when the answer was fetched from geojs, or when the database copy
|
||||
was downloaded
|
||||
|
||||
A field the source does not know is null. DB-IP has no `accuracy_radius_km` and
|
||||
no `time_zone`, so both are null in its answers. `geo` itself is null when
|
||||
neither source answers. A reserved address, one whose `is_bogon` flag is true,
|
||||
goes to neither source, so its `geo` is null.
|
||||
|
||||
What goes outside: geojs receives each address that the cache does not hold. For
|
||||
`/ip/` and the home page, that address is the visitor's own. A lookup answered
|
||||
from the cache sends nothing. DB-IP receives only the monthly request for its
|
||||
file.
|
||||
|
||||
How RouteWatch keeps the answers of geojs:
|
||||
|
||||
- It holds at most 250,000 addresses in memory, and drops the least recently
|
||||
used first. That takes about 100 MB of the Go heap.
|
||||
- It writes new answers every 5 seconds to `cache.db` in the state directory.
|
||||
- It loads the newest 250,000 answers from `cache.db` at startup.
|
||||
- An address not in memory is read from `cache.db` before geojs is asked.
|
||||
- An answer expires 7 days after its fetch. Every 10 minutes RouteWatch deletes
|
||||
expired answers from memory and from `cache.db`.
|
||||
|
||||
Answers from the database are not cached, because reading the file is local.
|
||||
|
||||
How RouteWatch keeps the database:
|
||||
|
||||
- It keeps one copy in the state directory, named `dbip-city-lite.mmdb`.
|
||||
- It downloads the current month's file at the first start, and then once a
|
||||
month. Until the first download finishes, only geojs answers.
|
||||
- A download fails when it takes over 10 minutes or unpacks to over 200 MiB.
|
||||
- A new file replaces the old copy only once it opens. After a failed download
|
||||
RouteWatch keeps the old copy and tries again an hour later.
|
||||
- The file is about 130 MB. RouteWatch maps it into memory, outside the Go heap.
|
||||
The kernel loads the parts that are read, and can drop them again when memory
|
||||
is short.
|
||||
|
||||
The home page's footer says "IP Geolocation by DB-IP" with a link to
|
||||
[db-ip.com](https://db-ip.com).
|
||||
|
||||
## Configuration
|
||||
|
||||
RouteWatch reads its settings from environment variables. It takes no
|
||||
@@ -496,7 +632,15 @@ by the client. For a request from any other address, they use the address the
|
||||
request came from, whatever headers it carries. `X-Real-IP` and `True-Client-IP`
|
||||
are not read.
|
||||
|
||||
The state directory holds the database, `db.sqlite`, and the IP list copies:
|
||||
The state directory holds:
|
||||
|
||||
- `db.sqlite`: the database
|
||||
- `cache.db`: the answers of PTR lookups and of geojs
|
||||
- `dbip-city-lite.mmdb`: the copy of DB-IP's City Lite database (see
|
||||
Geolocation)
|
||||
- `iplists`: the IP list copies
|
||||
|
||||
It is at:
|
||||
|
||||
- macOS: `~/Library/Application Support/berlin.sneak.app.routewatch/`
|
||||
- Linux and the BSDs, as root: `/var/lib/berlin.sneak.app.routewatch/`
|
||||
@@ -518,8 +662,9 @@ a memory limit of 5 GiB and swap disabled:
|
||||
docker run --memory=5g --memory-swap=5g ...
|
||||
```
|
||||
|
||||
The 5 GiB leaves room above the ceilings for spikes and for the kernel's cache
|
||||
of the database file.
|
||||
The 5 GiB leaves room above the ceilings for spikes, for the kernel's cache of
|
||||
the database file, and for the mapped copy of DB-IP's database (see
|
||||
Geolocation).
|
||||
|
||||
The ceilings add up to 3.7 GiB:
|
||||
|
||||
@@ -539,6 +684,13 @@ The ceilings add up to 3.7 GiB:
|
||||
limit caps the SQLite library.
|
||||
- About 0.2 GiB for everything else in the runtime.
|
||||
|
||||
The `icloud_private_relay` list has about 285,000 networks. Neighbouring
|
||||
networks merge, which leaves about 6,900 address ranges. Loaded, the list adds
|
||||
about 0.3 MiB to the Go heap and under 6 MiB to the process's resident memory.
|
||||
Parsing the file allocates about 94 MiB in about 0.2 seconds. The next garbage
|
||||
collection frees all of it but the 0.3 MiB above. Parsing happens at startup and
|
||||
whenever a download brings a new version.
|
||||
|
||||
The SQLite library gets its memory from glibc, the C library. glibc keeps freed
|
||||
memory in pools called arenas, and hands it back to the kernel slowly. It
|
||||
creates up to eight arenas per processor core. So on a host with many cores the
|
||||
@@ -554,8 +706,10 @@ What happens at each limit:
|
||||
allocating more.
|
||||
- The SQLite hard heap limit: a database statement fails with an out-of-memory
|
||||
error, which the handler logs. The prefix handler writes the routing table's
|
||||
changes again in a later write. Everything else in the failed write is
|
||||
dropped. The process keeps running.
|
||||
changes and its origin changes again in a later write. Everything else in a
|
||||
handler's failed write is dropped. The PTR cache and the geolocation part
|
||||
write their answers to `cache.db` again in a later write. The process keeps
|
||||
running.
|
||||
- The handler queues: the handlers are the four parts that take the feed's
|
||||
messages (see Design). The prefix handler's queue holds at most 100,000
|
||||
messages, about 100 MiB. Each of the other three holds at most 20,000. When a
|
||||
@@ -653,6 +807,10 @@ The parts:
|
||||
with old or missing details asks the fetcher for it at once, without the wait.
|
||||
- The IP lists part downloads the IP lists and answers which lists hold an
|
||||
address.
|
||||
- The PTR cache looks up the PTR names of addresses and keeps the answers (see
|
||||
PTR lookups).
|
||||
- The geolocation part finds where addresses are, keeps the answers of geojs and
|
||||
downloads DB-IP's database (see Geolocation).
|
||||
- The database maintainer checkpoints the database every 5 seconds: it folds the
|
||||
write-ahead log, where SQLite first writes changes, into the database file. It
|
||||
reclaims free space every 10 minutes. Every hour it refreshes the statistics
|
||||
@@ -762,6 +920,8 @@ On `SIGTERM` or `SIGINT` the daemon stops its parts in this order:
|
||||
4. the database maintainer
|
||||
5. the IP list downloads
|
||||
6. the HTTP server
|
||||
7. the geolocation part, which writes the geojs answers it holds
|
||||
8. the PTR cache, which writes what it holds
|
||||
|
||||
Last, the daemon checkpoints and closes the database. It gives all this 8
|
||||
seconds and then exits with status 0. A stop that is still running after 9
|
||||
@@ -782,6 +942,10 @@ What a stop can lose:
|
||||
- When the 8 seconds run out, the streamer and the handlers log what they could
|
||||
not hand over or write. The database is then not closed. When it next opens,
|
||||
SQLite rolls back any write the exit cut off.
|
||||
- When the 8 seconds run out, the PTR answers not yet in `cache.db` are lost.
|
||||
RouteWatch asks the resolvers for them again the next time they are needed.
|
||||
- When the 8 seconds run out, the geojs answers not yet in `cache.db` are lost.
|
||||
RouteWatch asks geojs for them again the next time they are needed.
|
||||
|
||||
### Code layout
|
||||
|
||||
@@ -795,11 +959,14 @@ routewatch/
|
||||
│ └── message-examples.json # Examples of RIS Live messages
|
||||
├── internal/
|
||||
│ ├── bogon/ # Reserved prefixes and AS numbers
|
||||
│ ├── cachedb/ # cache.db, the caches of outside answers
|
||||
│ ├── config/ # Settings from the environment
|
||||
│ ├── database/ # SQLite storage
|
||||
│ ├── geo/ # Geolocation: geojs, DB-IP City Lite, the cache
|
||||
│ ├── iplist/ # Public IP lists: downloads and lookups
|
||||
│ ├── logger/ # Structured logging
|
||||
│ ├── metrics/ # Feed and route update figures
|
||||
│ ├── ptr/ # PTR lookups over DNS-over-HTTPS, and their cache
|
||||
│ ├── ristypes/ # The RIS Live message types
|
||||
│ ├── routetable/ # The whole routing table in memory
|
||||
│ ├── routewatch/ # The daemon: startup, handlers, WHOIS fetcher
|
||||
@@ -848,6 +1015,18 @@ announcements(id PRIMARY KEY, prefix_id, peer_asn, origin_asn, path,
|
||||
|
||||
The indexes are in `internal/database/schema.sql`.
|
||||
|
||||
`cache.db` is a separate file, with two tables:
|
||||
|
||||
```sql
|
||||
-- The answers of PTR lookups. names is empty for an address with no PTR names.
|
||||
hostnames(address PRIMARY KEY, names, expires_at)
|
||||
|
||||
-- The answers of geojs. A part geojs does not know is NULL.
|
||||
geolocations(address PRIMARY KEY, country_code, country, continent_code,
|
||||
region, city, latitude, longitude, accuracy_radius_km, time_zone,
|
||||
fetched_at)
|
||||
```
|
||||
|
||||
## Development
|
||||
|
||||
The `make` targets:
|
||||
@@ -897,8 +1076,8 @@ calls one of them. The scripts:
|
||||
- `script/projectname`: print the project name, `routewatch`, which
|
||||
`script/docker` uses as the image tag.
|
||||
- `script/test`: build the `test` phase of the `Dockerfile` alone. It runs
|
||||
`go test -short -count=1 -timeout 90s -race -cover ./...`, and reruns it with
|
||||
`-v` on failure.
|
||||
`go test -short -count=1 -timeout 90s -race -cover ./...`. On failure it runs
|
||||
the tests again with `-v` and without `-cover`.
|
||||
- `script/lint`: build the `lint` phase of the `Dockerfile` alone. It runs
|
||||
`golangci-lint` v2.14.0 with `.golangci.yml`.
|
||||
- `script/fmt`: format the Go code with `go fmt` and the markdown with
|
||||
@@ -921,8 +1100,7 @@ calls one of them. The scripts:
|
||||
|
||||
## TODO
|
||||
|
||||
`TODO.md` holds the working list: the next step, the steps done and the steps
|
||||
planned.
|
||||
`TODO.md` holds the working list: the next step and the steps planned.
|
||||
|
||||
## License
|
||||
|
||||
|
||||
@@ -2,7 +2,7 @@
|
||||
|
||||
- branch (from `main`)
|
||||
- do the work in Next Step
|
||||
- move Next Step to the top of Completed Steps
|
||||
- delete Next Step from `TODO.md`
|
||||
- move the top item of Future Steps into Next Step
|
||||
- commit (`TODO.md` changes in the same commit as the work)
|
||||
- merge to `main` if the branch is not protected, otherwise open a PR
|
||||
@@ -10,442 +10,31 @@
|
||||
|
||||
# Status
|
||||
|
||||
RouteWatch is pre-1.0, with no git tags. `main` last took `next` on 2026-10-27,
|
||||
through https://git.eeqj.de/sneak/routewatch/pulls/166.
|
||||
https://git.eeqj.de/sneak/routewatch/pulls/202 carries the commits since then
|
||||
into `main`. It is labelled `merge-ready` and waits for sneak.
|
||||
|
||||
The linter and the tests are phases of the Docker build. The Gitea workflow runs
|
||||
`script/cibuild` on every push. That script runs the two phases and checks the
|
||||
formatting of the Go code and the markdown. Then it builds the image.
|
||||
|
||||
The image sets memory ceilings for a 5 GiB container (README "Memory"). README
|
||||
"Running under upaas" says how to run it.
|
||||
|
||||
A five-day run of `1c65656` on the live feed, from 2026-10-27 to 2026-11-01,
|
||||
found three problems:
|
||||
|
||||
- The prefix handler dropped updates while a peer's session went down
|
||||
(https://git.eeqj.de/sneak/routewatch/issues/216).
|
||||
- The AS handler dropped updates when the feed burst
|
||||
(https://git.eeqj.de/sneak/routewatch/issues/217).
|
||||
- The WHOIS fetcher kept asking a registry that had refused it
|
||||
(https://git.eeqj.de/sneak/routewatch/issues/218).
|
||||
|
||||
All three are fixed on `next`. So is
|
||||
https://git.eeqj.de/sneak/routewatch/issues/222, which the third fix brought up.
|
||||
A run of `26d939a` on the live feed showed the prefix handler still dropping
|
||||
updates while the home page's cards were worked out.
|
||||
https://git.eeqj.de/sneak/routewatch/issues/226 fixed that on `next`.
|
||||
https://git.eeqj.de/sneak/routewatch/issues/225, also on `next`, takes the peers
|
||||
whose sessions go down together out of the table in one walk. A 24-hour run of
|
||||
`9bd0e91` on the live feed is checking whether any handler still drops updates.
|
||||
|
||||
The order of the project's remaining work is kept on
|
||||
RouteWatch is pre-1.0. The order of the remaining work is kept on
|
||||
https://git.eeqj.de/sneak/routewatch/issues/86.
|
||||
|
||||
# Next Step
|
||||
|
||||
Post the outcome of the 24-hour run of `9bd0e91` on
|
||||
https://git.eeqj.de/sneak/routewatch/issues/226: whether any handler still drops
|
||||
updates, and the process's peak memory against 5 GiB. File an issue for each
|
||||
problem the run finds.
|
||||
Post the outcome of the run of `e58c663` on the live feed on
|
||||
https://git.eeqj.de/sneak/routewatch/issues/232. Work it out from the run's
|
||||
data:
|
||||
|
||||
# Completed Steps
|
||||
- the log of the stopped container `routewatch-run-e58c663`, read with
|
||||
`docker logs`
|
||||
- the database in the Docker volume `routewatch-run-e58c663`
|
||||
|
||||
- 2026-11-10: `tor_exits` comes from Onionoo, so it holds IPv6 exits and
|
||||
`is_tor` is true for them (closes
|
||||
https://git.eeqj.de/sneak/routewatch/issues/234).
|
||||
- 2026-11-10: each result of the IP lookup has a `flags` object with `is_tor`
|
||||
and `is_bogon`. The public IP lists are named `ip_lists` in the answer,
|
||||
`internal/iplist` in the code and `iplists` in the state directory. A list
|
||||
file may be up to 200 MiB (closes
|
||||
https://git.eeqj.de/sneak/routewatch/issues/233).
|
||||
- 2026-11-05: the prefix page lists the prefix's origin changes, newest first.
|
||||
The routing table records each change, and the prefix handler writes them to a
|
||||
new `origin_changes` table. The database maintainer prunes each prefix to its
|
||||
newest 500 rows or its rows from the last 72 hours, whichever keeps more. The
|
||||
page shows at most the newest 5,000. Each row shows:
|
||||
- the time
|
||||
- announced or withdrawn
|
||||
- the origin AS
|
||||
- the peer count
|
||||
Keep the container until the outcome is posted. The outcome says:
|
||||
|
||||
A row's peer count is how many peers carry the prefix just after the change,
|
||||
with any origin AS (closes #161).
|
||||
- whether the feed stalled or reconnected
|
||||
- whether any handler dropped an update, and the most messages the prefix
|
||||
handler's queue held at once
|
||||
- the most messages the prefix handler held back at once
|
||||
- the size of the `origin_changes` table on disk
|
||||
- the process's peak memory against 5 GiB
|
||||
|
||||
- 2026-11-03: the streamer drops an update for a handler only when that
|
||||
handler's queue is full. It used to drop a random share of the updates once
|
||||
the queue was half full. The prefix handler's queue now holds 100,000
|
||||
messages, about 100 MiB. It used to hold 20,000. While the routing table's
|
||||
`Overview` walks every prefix, the prefix handler applies updates more slowly.
|
||||
In a run on the live feed, the feed paused during such a walk and then sent
|
||||
its backlog. The streamer then dropped about 10,000 of the prefix handler's
|
||||
updates (closes #229)
|
||||
- 2026-11-02: `README.md` is rewritten from scratch, sentence by sentence,
|
||||
against the rules of https://git.eeqj.de/sneak/routewatch/issues/157. Every
|
||||
statement was checked against the code on `next` (closes #157)
|
||||
- 2026-11-02: `TODO.md` is rewritten from scratch, sentence by sentence, against
|
||||
the rules of https://git.eeqj.de/sneak/routewatch/issues/157. Every Completed
|
||||
Steps entry keeps its date, its facts and its issue. Status and Next Step
|
||||
describe the repo as the tracker shows it (part of #157)
|
||||
- 2026-11-02: when the sessions of several peers go down, the prefix handler
|
||||
takes the peers out of the routing table together. The table's `PeersDown`
|
||||
takes several peers out of every prefix in one walk. One walk runs at a time.
|
||||
The next one takes every peer whose session went down since the previous one
|
||||
began. Each peer used to have a walk of its own. A peer's messages stayed held
|
||||
back while the walks queued ahead of its own ran. The handler holds back at
|
||||
most 150,000 messages, about 150 MiB. It used to hold back at most 20,000. The
|
||||
status page shows how many messages the handler holds back (closes #225)
|
||||
- 2026-11-02: working out the home page's cards no longer stops the prefix
|
||||
handler. The walk over the routing table holds the table's lock for 10,000
|
||||
prefixes at a time, then lets the lock go for as long again, as `PeerDown`
|
||||
does. It used to take the lock back at once. The prefix handler then applied
|
||||
about one update per 10,000 prefixes walked. During a walk, its queue could
|
||||
fill, and the streamer then dropped updates for it (closes #226)
|
||||
- 2026-11-02: A skipped WHOIS lookup now makes the fetcher wait longer before
|
||||
the next one, as a failed lookup does. A lookup is skipped when its RIR is
|
||||
paused. An RIR is one of the five regional Internet registries. The longest
|
||||
wait is one minute. The status page does not count a skip as a failure. A skip
|
||||
used to leave the wait as it was. Each lookup asks IANA which RIR holds the AS
|
||||
before it finds that RIR paused. So while most stale ASes belonged to a paused
|
||||
RIR, the fetcher could ask IANA every second (closes #222)
|
||||
- 2026-11-02: A peer's session going down no longer stops the prefix handler.
|
||||
The handler runs the routing table's `PeerDown` on a goroutine of its own, one
|
||||
peer at a time. Meanwhile it goes on taking messages. `PeerDown` holds the
|
||||
table's lock for 10,000 prefixes at a time, then lets the lock go for as long
|
||||
again. The handler holds back the down peer's later messages until that peer's
|
||||
`PeerDown` returns. Then the handler applies them in order, one at a time, and
|
||||
takes new messages in between. It holds back at most 20,000 messages. At that
|
||||
number, `HandleMessage` waits until one of them is applied. When `Stop` runs
|
||||
past its deadline, it logs how many held-back messages were not applied
|
||||
(closes #216)
|
||||
- 2026-11-02: RouteWatch pauses an RIR's or NIC.br's WHOIS server for an hour
|
||||
after the server refuses a query. NIC.br is Brazil's registry. A refused or
|
||||
reset connection is a refusal. So is an answer with an `%ERROR:201:` or
|
||||
`%ERROR:202:` line. Lookups at the other servers go on. An AS whose RIR is
|
||||
paused is looked up after the pause. While NIC.br is paused, a Brazilian AS
|
||||
gets the details from LACNIC's answer. IANA is never paused, because a pause
|
||||
of IANA would stop every lookup. A refusal by IANA is an ordinary failure.
|
||||
NIC.br is also paused for an hour after it says RouteWatch has passed its
|
||||
query limit. Before, the fetcher asked a server again within seconds of a
|
||||
refusal (closes #218)
|
||||
- 2026-11-02: The AS handler never waits for a database write. Its batch holds
|
||||
one entry per AS, with the earliest and the latest time of its updates. So the
|
||||
batch no longer grows with the feed's rate. It used to hold one entry for each
|
||||
AS of each path. At 300,000 entries it waited for the write in progress. In a
|
||||
burst of the feed its queue then filled, and the streamer dropped messages for
|
||||
it (closes #217)
|
||||
- 2026-10-27: The tests of `internal/routewatch` no longer fail on a loaded
|
||||
host. Five of them gave `HandleMessage` two seconds to apply a full prefix
|
||||
batch. Under the race detector that can take longer. A test now waits up to 15
|
||||
seconds for each step it expects. The tests of the handlers' limits now give
|
||||
the handlers small limits. So the package no longer runs past its 90-second
|
||||
limit (closes #213, #215)
|
||||
- 2026-10-27: The server works out the home page's cards every minute until it
|
||||
stops. It used to work them out only once, at startup. The end of the context
|
||||
given to `Start` stopped the refreshes (closes #211)
|
||||
- 2026-10-27: README "Rationale" says why RouteWatch exists. RouteWatch answers
|
||||
our other services' questions about IP addresses, from its own routing table
|
||||
and the WHOIS details it keeps. The section still says what RIS Live does not
|
||||
give (closes #204)
|
||||
- 2026-10-27: Every JSON answer carries `@meta`. A panic on a JSON route answers
|
||||
500 with the JSON error body. `/status.json` and the health check answer 405
|
||||
with the JSON error body to any method they do not take. The 500 and the 405
|
||||
used to have an empty body. The home page's IP lookup shows the whole JSON
|
||||
answer, `@meta` included. It used to show only the answer's `data`, or only
|
||||
the message of an error answer. A test checks `@meta` on every JSON route, in
|
||||
its successes and its errors (closes #201)
|
||||
- 2026-10-27: Every value with a unit in a JSON answer is a number. The unit is
|
||||
at the end of the field name. `@meta.execution_time` is now
|
||||
`execution_time_ms`. In the stats, `uptime` is now `uptime_seconds`,
|
||||
`connection_duration` is now `connection_duration_seconds`, and `memory_usage`
|
||||
is now `memory_usage_bytes`. `whois_stats.current_interval` is now
|
||||
`whois_stats.current_interval_seconds`. They used to be strings such as "12
|
||||
ms", "1h18m34s" or "1.1 GB". `connection_duration_seconds` is null while the
|
||||
feed is not connected. `API_POLICIES.md` is vendored from `sneak/prompts`, and
|
||||
`REPO_POLICIES.md` is re-vendored (closes #199)
|
||||
- 2026-10-27: Every result of the IP lookup has the same fields. A field without
|
||||
a value is null. It used to be left out. An address with no route in the
|
||||
routing table has every route and AS field set to null. `ptr` is an empty list
|
||||
when there are no PTR names. The new field `num_origins` gives the number of
|
||||
origin ASes of the netblock, and `origin_asns` gives which they are (closes
|
||||
#200)
|
||||
- 2026-10-27: With `TRUSTED_PROXIES` unset, the server trusts reverse proxies at
|
||||
the private addresses of RFC 1918. `/ip/`, the home page's `curl` card and the
|
||||
request log then give the client's address from `X-Forwarded-For`. They used
|
||||
to give the proxy's address. A list in `TRUSTED_PROXIES` replaces the default.
|
||||
`TRUSTED_PROXIES` set but empty trusts no proxy (closes #198)
|
||||
- 2026-10-27: The home page has cards with figures of the routing table. One
|
||||
card shows the newest and the oldest kept route. One shows the newest and the
|
||||
oldest updated AS. One shows the AS with the most prefixes. One shows the ASes
|
||||
with a single prefix. One shows fifteen well-known networks. Two cards count
|
||||
the bogon announcements and the prefixes with more than one origin AS. They
|
||||
link to the new pages `/bogons` and `/multiple-origins`, which list them. A
|
||||
new package, `internal/bogon`, tells which prefixes and AS numbers are
|
||||
reserved. The server works the figures out once a minute (closes #154)
|
||||
- 2026-10-27: The per-peer route tables are gone, with the code that wrote,
|
||||
counted and read them. The prefix handler applies each update to the routing
|
||||
table in memory. It writes the table's changes every second. It also stores
|
||||
each new prefix with its first-seen time. The image's memory limits fit a full
|
||||
routing table of about 1 GB. `GOMEMLIMIT` is 2.5 GiB. The writing connection's
|
||||
page cache is 512 MiB. SQLite's soft and hard heap limits are 768 MiB and 1
|
||||
GiB (closes #122)
|
||||
- 2026-10-27: The pages, the API and the stats read every route from the routing
|
||||
table in memory. For a prefix with several origin ASes, the IP lookup gives
|
||||
the one most peers carry. Its `last_seen` is the newest kept route's time. The
|
||||
prefix page and API list every peer that carries the prefix. Only the kept
|
||||
routes show an AS path, a next hop and a time. The stats count the origins
|
||||
whose kept routes were all withdrawn, in `origins_without_kept_route` (part of
|
||||
#122)
|
||||
- 2026-10-27: The prefix, AS and peer handlers make a timed write on every tick
|
||||
of their timers. The prefix handler's timer ticks every second. The other two
|
||||
tick every two seconds. They used to skip every other tick. A full batch's
|
||||
write restarts the timer. So the next timed write comes one interval after
|
||||
that write began (closes #192)
|
||||
- 2026-10-27: Once the stop deadline has passed, the AS, peer and peering
|
||||
handlers start no write either. Before, a full batch or a timer tick could
|
||||
wait while a write ran past the deadline. Either could then start one more
|
||||
write. The flush goroutines of the prefix, AS and peer handlers now run the
|
||||
same code (closes #189)
|
||||
- 2026-10-27: Once the stop deadline has passed, the prefix handler starts no
|
||||
flush. Before, a batch that filled during a late write could still be written
|
||||
after the deadline (closes #188)
|
||||
- 2026-10-27: The prefix handler applies each update to the routing table in
|
||||
memory. It writes the table's changes to the database every second. A write
|
||||
takes at most 20,000 changed pairs of prefix and origin AS. The rest follow in
|
||||
the next writes. A failed write is written again later. Startup reads the
|
||||
table before the streamer connects. The pages still read the per-peer route
|
||||
tables (part of #122)
|
||||
- 2026-10-27: The database keeps a copy of the whole routing table in a new
|
||||
`routes` table, one row per prefix and origin AS. The peers' numbers are in a
|
||||
new `route_peers` table. One function writes the pairs that changed in memory.
|
||||
Another reads the whole table back into memory at startup. Nothing uses them
|
||||
yet (part of #122)
|
||||
- 2026-10-27: A new package, `internal/routetable`, holds the whole routing
|
||||
table in memory. For each prefix and origin AS it keeps the peers that carry
|
||||
it, one bit per peer. It also keeps the two newest of their routes. Nothing
|
||||
uses it yet (part of #122)
|
||||
- 2026-10-27: Each result of the IP lookup lists the public blocklists the
|
||||
address is on, in `blocklists`. The lists are DShield, Feodo Tracker, two
|
||||
FireHOL lists, two Spamhaus DROP lists and the Tor exit list. RouteWatch
|
||||
downloads each list on its own schedule and answers from copies in memory. The
|
||||
copies are also kept in the state directory's `blocklists` directory. IP
|
||||
lookup answers carry `Cache-Control`. An address with no route now has a
|
||||
result too, without route fields. So the IP lookup no longer answers 404
|
||||
(closes #158)
|
||||
- 2026-10-27: The home page has an IP information API card. It shows the `curl`
|
||||
command that looks up the visitor's own address at `/ip/{addr}`. It also shows
|
||||
that command's live answer (closes #156)
|
||||
- 2026-10-27: On the AS page, a click on the heading of the IPv4 or the IPv6
|
||||
prefix list folds or unfolds the list. Both lists start open (closes #155)
|
||||
- 2026-10-27: The AS page lists its IPv4 and its IPv6 prefixes by mask length,
|
||||
shortest first. Prefixes of one mask length are listed by address. They used
|
||||
to be sorted by address first, then by mask length (closes #153)
|
||||
- 2026-10-27: The connection status on `/` and `/status` gets a spinner. It
|
||||
turns one frame each time the stats show more messages than the time before.
|
||||
After three refreshes with no new messages, the page shows "stale" in red
|
||||
instead. Next to it is the time since the last fresh answer (closes #152)
|
||||
- 2026-10-27: The WHOIS details of a Brazilian AS keep their accented letters.
|
||||
LACNIC's answer for such an AS is NIC.br's answer with each accented letter
|
||||
replaced. NIC.br is Brazil's registry. The details now come from NIC.br's own
|
||||
server, `whois.registro.br`. When that query fails, the details from LACNIC's
|
||||
answer are kept and a warning is logged. RouteWatch stops asking NIC.br for an
|
||||
hour after NIC.br says RouteWatch is over its query limit (closes #174)
|
||||
- 2026-10-27: `make asupdate` stops with an error when the download of `as.csv`
|
||||
breaks off. The generator used to retry the failed read forever. A record that
|
||||
does not parse is still logged and skipped (closes #177)
|
||||
- 2026-10-27: `make asupdate` runs the new `script/asupdate`. It replaces the AS
|
||||
list built into RouteWatch only when the generator succeeds and finds at least
|
||||
one AS. A failed download used to replace the list with an empty file. The
|
||||
generator reads `as.csv` from `ipverse/as-metadata`, the source's new name
|
||||
(closes #173)
|
||||
- 2026-10-27: The WHOIS details of an AS at RIPE, APNIC or AFRINIC come only
|
||||
from its `aut-num` object and the organisation object it names. RIPE and APNIC
|
||||
start their answer with the `as-block` object of the AS's block. Its dates
|
||||
were stored as the AS's (closes #168)
|
||||
- 2026-10-27: AS names keep their accented letters. The AS list built into
|
||||
RouteWatch is rebuilt from today's `as.csv`, which added a country code
|
||||
column. That file no longer lists AS 0 or the documentation AS numbers. A
|
||||
WHOIS answer that is not valid UTF-8 is decoded as Latin-1 before it is parsed
|
||||
and stored. RIPE and APNIC send Latin-1 (closes #165)
|
||||
- 2026-10-27: The Age column on the AS and prefix pages reads in words, such as
|
||||
"5 minutes ago" or "2 weeks ago". It used to print Go durations such as
|
||||
"360h0m0s ago". An age under a minute reads "now" or in seconds. An age over a
|
||||
month reads in months or years instead of a date (closes #163)
|
||||
- 2026-10-26: A write to a stored AS, peering or peer keeps the earlier of the
|
||||
two first-seen times. It keeps the later of the two last-seen times. An old
|
||||
update from a collector catching up on a backlog no longer moves last seen
|
||||
back (closes #162)
|
||||
- 2026-10-26: The status page shows "Route Churn/min". That is the number of
|
||||
prefix announcements and withdrawals from all peers in the last full clock
|
||||
minute. `/api/v1/stats` reports it as `route_churn_per_min` in `stream`. It
|
||||
replaces the moving average `route_churn_per_sec`. The messages and updates
|
||||
per second on `/` and `/status` are whole numbers (closes #160)
|
||||
- 2026-10-26: The WHOIS fetcher backs off more slowly. A failure makes its wait
|
||||
1.5 times as long, up to one minute instead of five. Its queries to RIPE,
|
||||
APNIC and AFRINIC leave out person and role objects. RIPE counts those objects
|
||||
against a daily limit per address. A refused query is logged with the RIR's
|
||||
error line (closes #159)
|
||||
- 2026-10-26: The home page's AS name search works. It lists the ASes whose
|
||||
handle or description holds the text, as links to their pages. The new
|
||||
`/api/v1/search/as` answers it. The search ignores case. When the database has
|
||||
no record of an AS, its page takes the handle and description from the AS list
|
||||
built into RouteWatch. It still lists the AS's prefixes (closes #151)
|
||||
- 2026-10-19: README "HTTP Endpoints" lists every route with what it answers.
|
||||
README "Database Schema" matches `schema.sql`. The rest of the README matches
|
||||
the code. This file is brought up to date (closes #72)
|
||||
- 2026-10-19: Route writes are cheaper. The live route and prefix tables lose
|
||||
their random UUID keys. The prefix tables lose their duplicate index. A route
|
||||
update sets only the columns that change. Every write goes through one
|
||||
connection with a page cache of 1 GiB (README "Memory"). A benchmark writes
|
||||
the prefix handler's largest batch faster than the feed sends it (closes #96)
|
||||
- 2026-10-19: The checkpoint of the WAL (SQLite's write-ahead log), the
|
||||
incremental vacuum and `ANALYZE` take the database lock that every batch write
|
||||
holds. So a vacuum no longer fails with "database is locked". An `ANALYZE` no
|
||||
longer makes a batch write lose its updates. `ANALYZE` runs as
|
||||
`PRAGMA optimize` with an analysis limit (closes #94)
|
||||
- 2026-10-19: Shutdown stops the streamer first. Each handler then writes
|
||||
everything left on its queue before it stops. So the updates waiting there are
|
||||
no longer lost (closes #69)
|
||||
- 2026-10-19: A path that ends in an AS set of several members has origin AS 0,
|
||||
which means no known origin. It used to have the set's last member as its
|
||||
origin. The set adds no peerings. The pages show such a route as having no
|
||||
known origin (closes #75)
|
||||
- 2026-10-19: `bgp_peer_count` drops to 0 when the last peer goes down (closes
|
||||
#92)
|
||||
- 2026-10-19: `bgp_peer_count` counts every peer the feed has heard from. It
|
||||
used to count only the peers whose session opened since startup. It drops a
|
||||
peer whose session goes down (closes #74)
|
||||
- 2026-10-19: The shared files are re-vendored from `sneak/prompts` at
|
||||
`0653579`. The linter is `golangci-lint` v2.14.0 with its configuration. Lint
|
||||
and test are `Dockerfile` phases that keep Go's and `golangci-lint`'s caches.
|
||||
`make fmt` also formats the markdown with `prettier`. The agent guidance moves
|
||||
to `AGENTS.md`. The code is fixed for what the new linter configuration finds
|
||||
(closes #52)
|
||||
- 2026-10-19: The streamer reads feed lines up to 1 MiB. A line over 64 KiB used
|
||||
to end the connection (closes #76)
|
||||
- 2026-10-19: A peer's live routes are removed when its BGP session goes down.
|
||||
`STATE` messages are never dropped when a queue is full. One that finds its
|
||||
queue full waits for room ahead of the handler's other messages (closes #56)
|
||||
- 2026-10-19: The prefix-length pages pick their sample with a bounded number of
|
||||
index lookups. They used to read every route of that length. They list each
|
||||
prefix once (closes #66)
|
||||
- 2026-10-19: A request past its time limit answers 504 instead of 408.
|
||||
`/api/v1/stats` answers 504 instead of 500 when its read runs out of time
|
||||
(closes #70)
|
||||
- 2026-10-19: The AS, peer and prefix handlers write each batch outside the lock
|
||||
their message handling needs. So messages keep coming off their queues during
|
||||
a write. Meanwhile the next batch grows to at most ten batch sizes (closes
|
||||
#65)
|
||||
- 2026-10-19: A live route is identified by prefix and peer. So an announcement
|
||||
with a new origin AS replaces the peer's route. A withdrawal removes it
|
||||
whatever its origin AS (closes #55)
|
||||
- 2026-10-14: The streamer reconnects when RIS Live ends the stream or sends
|
||||
nothing for 60 seconds. The health check reports the feed disconnected
|
||||
meanwhile (closes #53)
|
||||
- 2026-10-14: The prefix page takes the names of the origin ASes from the prefix
|
||||
query. It used to read every route of each origin AS (closes #67)
|
||||
- 2026-10-14: The daemon exits with status 1 when its web server cannot listen
|
||||
on its port. It used to run on without a web server (closes #73)
|
||||
- 2026-10-14: The prefix pages parse the prefix before looking it up. One that
|
||||
does not parse answers 400. Any valid way of writing a prefix finds its routes
|
||||
(closes #71)
|
||||
- 2026-10-14: The IP lookup ignores routes shorter than /8 for IPv4 and /16 for
|
||||
IPv6. So a default route is no longer named as an address's owner (closes #57)
|
||||
- 2026-10-14: A prefix batch keeps each peer's last update for a prefix. It used
|
||||
to keep one update per prefix across all peers (closes #54)
|
||||
- 2026-10-14: `reconnect_count` in `/api/v1/stats` counts every reconnect
|
||||
(closes #68)
|
||||
- 2026-10-03: `/api/v1/stats` serves the prefix distribution from memory. The
|
||||
distribution is seeded at startup and adjusted on every live-route write. So a
|
||||
request no longer reads every live route (closes #30)
|
||||
- 2026-10-03: A new database is created with `auto_vacuum` set to incremental.
|
||||
The setting goes in the connection string, so it is set before the file is
|
||||
first written. Each periodic incremental vacuum now returns up to 1000 free
|
||||
pages. The late `PRAGMA auto_vacuum` in `Initialize`, which SQLite ignored, is
|
||||
gone (closes #43)
|
||||
- 2026-10-03: Looking up an IP address no longer reads every IPv6 route. Both
|
||||
families find the most specific live route with at most 33 or 129 lookups on
|
||||
the prefix index. The IPv4 range columns are gone. Prefixes from the feed are
|
||||
stored in one text form. So an IPv6 withdrawal, which the feed sends
|
||||
uncompressed, now removes its route (closes #48)
|
||||
- 2026-10-02: A plain `docker build .` stamps the commit's tag or short commit
|
||||
into the page footer. It used to stamp `unknown`. The value comes from
|
||||
`git describe --tags --always`. `.dockerignore` sends `.git` without
|
||||
`.git/config`. A `VERSION` build argument takes precedence. `make build`
|
||||
stamps the same value (closes #46)
|
||||
- 2026-09-29: The entrypoint creates the data directory if it is missing. It
|
||||
stops the start if a step fails. README "Running under upaas" no longer asks
|
||||
for the host directory to be created first (closes #42)
|
||||
- 2026-09-29: `.dockerignore` keeps `.git`, local build output, local databases
|
||||
and `.env` out of the Docker build context. So they stay out of the source
|
||||
archive in the image (closes #39)
|
||||
- 2026-09-29: The README's first line names the MIT licence and the author. The
|
||||
License section says MIT and links `LICENSE`. An Author section is added. This
|
||||
file is brought up to date (closes #38)
|
||||
- 2026-09-29: `LICENSE` is added, with the MIT licence (closes #1)
|
||||
- 2026-09-28: Stopping the daemon while the feed is flowing no longer panics
|
||||
with "send on closed channel". The read loop checks for a stop just before
|
||||
handing a message to the handler queues. A second `Stop` no longer closes the
|
||||
queues again (closes #34)
|
||||
- 2026-09-28: `docker stop` no longer kills the daemon 2 seconds after the stop
|
||||
signal. The entrypoint switches to the `routewatch` user with `setpriv`
|
||||
instead of `runuser`. So the daemon receives the signal itself. It gets the
|
||||
whole wait `docker stop` allows, up to its own 60-second limit (closes #33)
|
||||
- 2026-09-28: RouteWatch is ready to run under upaas. A set but invalid `PORT`,
|
||||
`XDG_DATA_HOME` or `MALLOC_ARENA_MAX` stops the start. The health check
|
||||
follows `PORT`. README gains a "Running under upaas" section (closes #31)
|
||||
- 2026-09-22: The database statistics are kept in memory. The counts are seeded
|
||||
at startup and adjusted on every write. The oldest and newest route times come
|
||||
from a lookup at each end of an index. `/api/v1/stats` no longer scans the
|
||||
tables (closes #27)
|
||||
- 2026-09-21: Batch writes take the write lock when their transaction begins
|
||||
(`_txlock=immediate`). So they wait out a WAL checkpoint instead of failing
|
||||
with "database is locked" (closes #25)
|
||||
- 2026-09-21: `MALLOC_ARENA_MAX=2` in the image caps the number of glibc malloc
|
||||
arenas. So memory outside the Go runtime no longer grows with the core count
|
||||
(closes #23)
|
||||
- 2026-09-21: The image sets `GOMEMLIMIT=1536MiB`. README "Memory" gives the
|
||||
memory budget and the 5 GiB container limit (closes #13)
|
||||
- 2026-09-21: The four handler queues hold at most 20,000 messages each, down
|
||||
from 100,000 (closes #11)
|
||||
- 2026-09-21: Two goroutine leaks are fixed. The stats handlers leaked after a
|
||||
timeout. The streamer's tickers leaked on every reconnect (closes #12)
|
||||
- 2026-09-21: Parsed RIS messages no longer keep the unused `Community` and
|
||||
`Raw` fields (closes #9)
|
||||
- 2026-09-21: `.editorconfig` is added. A Gitea workflow runs `script/cibuild`
|
||||
on every push (closes #14)
|
||||
- 2026-09-21: The peering handler's AS-path map holds at most 500,000 paths.
|
||||
Every 30 seconds it is swapped for an empty one instead of copied (closes #10)
|
||||
- 2026-09-21: SQLite's memory is bounded across the whole connection pool. Each
|
||||
connection has a 64 MiB page cache. The heap limits are 1 GiB soft and 1.5 GiB
|
||||
hard (closes #8)
|
||||
- 2026-09-21: The Docker build runs the format check, the linter and the tests.
|
||||
Linting runs in a separate stage on a `golangci-lint` image pinned by digest
|
||||
(closes #5)
|
||||
- 2026-09-21: `make test` skips the live feed test, with `-short`. So
|
||||
`make check` no longer depends on the network (closes #2)
|
||||
- 2026-07-07: The scripts-to-rule-them-all layout is adopted. `script/` holds
|
||||
the entrypoints. The `Makefile` targets `check`, `test`, `lint`, `fmt` and
|
||||
`fmt-check` call them. So do `bootstrap`, `setup`, `docker` and `hooks`.
|
||||
README gains an "Entrypoints" section
|
||||
- 2026-02-22: The repo is brought in line with the repo policies. The required
|
||||
policy files are added. `.gitignore` is updated. The `Makefile` gains
|
||||
`fmt-check`, `check`, `docker` and `hooks` targets. The Go code is formatted
|
||||
with `gofmt`. The work was on the branch `repo-policies-compliance`, unmerged
|
||||
when this entry was written
|
||||
- 2026-01-01: The WAL is checkpointed every 5 seconds, and at startup with a log
|
||||
line. Checkpoints run in `TRUNCATE` mode. This fixed slow queries
|
||||
- 2025-12-31: The pages get a navigation bar and a home page with a search. The
|
||||
status page shows the oldest and newest route times. The WHOIS stats query
|
||||
handles `NULL`. The container runs as the `routewatch` user with the right
|
||||
state directory
|
||||
- 2025-12-30: HTTP requests are logged as structured log lines. The HTTP
|
||||
timeouts are longer. Prefix URLs with a slash in them route correctly. The
|
||||
status page gets more metrics and a footer
|
||||
- 2025-12-29: The `Dockerfile` builds in stages and puts a source archive in the
|
||||
image. SQLite reclaims space with incremental vacuum, which does not block
|
||||
- 2025-07-27: The first commit: RouteWatch, a BGP stream monitor
|
||||
The run's data does not show the most messages the prefix handler held back at
|
||||
once, or the process's peak memory.
|
||||
|
||||
File an issue for each problem the run found.
|
||||
|
||||
# Future Steps
|
||||
@@ -12,7 +12,7 @@ import (
|
||||
|
||||
// parseTimeLimit is how long a test waits for parseCSV to return before it
|
||||
// fails.
|
||||
const parseTimeLimit = 2 * time.Second
|
||||
const parseTimeLimit = 15 * time.Second
|
||||
|
||||
// TestParseCSVReadsTodaysLayout parses an as.csv with the country code column
|
||||
// that the source added after the description. It expects each AS with its
|
||||
|
||||
@@ -8,11 +8,15 @@ require (
|
||||
github.com/go-chi/chi/v5 v5.2.2
|
||||
github.com/google/btree v1.1.3
|
||||
github.com/google/uuid v1.6.0
|
||||
github.com/mattn/go-sqlite3 v1.14.29
|
||||
github.com/hashicorp/golang-lru/v2 v2.0.7
|
||||
github.com/mattn/go-sqlite3 v1.14.53
|
||||
github.com/maxmind/mmdbwriter v1.2.0
|
||||
github.com/miekg/dns v1.1.73
|
||||
github.com/oschwald/maxminddb-golang/v2 v2.9.0
|
||||
github.com/rcrowley/go-metrics v0.0.0-20250401214520-65e299d6c5c9
|
||||
go.uber.org/fx v1.24.0
|
||||
go4.org/netipx v0.0.0-20260823151212-3075585bcbeb
|
||||
golang.org/x/term v0.33.0
|
||||
golang.org/x/term v0.45.0
|
||||
golang.org/x/text v0.43.0
|
||||
)
|
||||
|
||||
@@ -20,5 +24,6 @@ require (
|
||||
go.uber.org/dig v1.19.0 // indirect
|
||||
go.uber.org/multierr v1.10.0 // indirect
|
||||
go.uber.org/zap v1.26.0 // indirect
|
||||
golang.org/x/sys v0.34.0 // indirect
|
||||
golang.org/x/net v0.57.0 // indirect
|
||||
golang.org/x/sys v0.49.0 // indirect
|
||||
)
|
||||
@@ -1,7 +1,5 @@
|
||||
github.com/bits-and-blooms/bitset v1.26.0 h1:WPvcPZbP1yuKofmjs2SYIFO1EjzIGhCP9ducElLsh5Q=
|
||||
github.com/bits-and-blooms/bitset v1.26.0/go.mod h1:7hO7Gc7Pp1vODcmWvKMRA9BNmbv6a/7QIWpPxHddWR8=
|
||||
github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c=
|
||||
github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
|
||||
github.com/dustin/go-humanize v1.0.1 h1:GzkhY7T5VNhEkwH0PVJgjz+fX1rhBrR7pRT3mDkpeCY=
|
||||
github.com/dustin/go-humanize v1.0.1/go.mod h1:Mu1zIs6XwVuF/gI1OepvI0qD18qycQx+mFykh5fBlto=
|
||||
github.com/go-chi/chi/v5 v5.2.2 h1:CMwsvRVTbXVytCk1Wd72Zy1LAsAh9GxMmSNWLHCG618=
|
||||
@@ -10,14 +8,20 @@ github.com/google/btree v1.1.3 h1:CVpQJjYgC4VbzxeGVHfvZrv1ctoYCAI8vbl07Fcxlyg=
|
||||
github.com/google/btree v1.1.3/go.mod h1:qOPhT0dTNdNzV6Z/lhRX0YXUafgPLFUh+gZMl761Gm4=
|
||||
github.com/google/uuid v1.6.0 h1:NIvaJDMOsjHA8n1jAhLSgzrAzy1Hgr+hNrb57e+94F0=
|
||||
github.com/google/uuid v1.6.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo=
|
||||
github.com/mattn/go-sqlite3 v1.14.29 h1:1O6nRLJKvsi1H2Sj0Hzdfojwt8GiGKm+LOfLaBFaouQ=
|
||||
github.com/mattn/go-sqlite3 v1.14.29/go.mod h1:Uh1q+B4BYcTPb+yiD3kU8Ct7aC0hY9fxUwlHK0RXw+Y=
|
||||
github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM=
|
||||
github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
|
||||
github.com/hashicorp/golang-lru/v2 v2.0.7 h1:a+bsQ5rvGLjzHuww6tVxozPZFVghXaHOwFs4luLUK2k=
|
||||
github.com/hashicorp/golang-lru/v2 v2.0.7/go.mod h1:QeFd9opnmA6QUJc5vARoKUSoFhyfM2/ZepoAG6RGpeM=
|
||||
github.com/mattn/go-sqlite3 v1.14.53 h1:mnTM0owzUMrKuLo+0w51NUiB0ir2Fo7kFXlPt/l3BGs=
|
||||
github.com/mattn/go-sqlite3 v1.14.53/go.mod h1:6JTjA44L93a0QCyJef5YvlPoKXntQPjzWv5gtm9sB6w=
|
||||
github.com/maxmind/mmdbwriter v1.2.0 h1:hyvDopImmgvle3aR8AaddxXnT0iQH2KWJX3vNfkwzYM=
|
||||
github.com/maxmind/mmdbwriter v1.2.0/go.mod h1:EQmKHhk2y9DRVvyNxwCLKC5FrkXZLx4snc5OlLY5XLE=
|
||||
github.com/miekg/dns v1.1.73 h1:uhT8nJxmTrPJYClxVxTCX+CVn6qnzSiybRk72Z6DgrE=
|
||||
github.com/miekg/dns v1.1.73/go.mod h1:RW2Obtfd5NZHvOFe3zYG0W8koWOQtAzyHaLo8vASBuQ=
|
||||
github.com/oschwald/maxminddb-golang/v2 v2.9.0 h1:YpIl3WoA9oIuNvqNaO/aD1N/7Ci2wOqoKH5GWo53H5Y=
|
||||
github.com/oschwald/maxminddb-golang/v2 v2.9.0/go.mod h1:3TxexMBcR0ss3SbyYvnqfsDzDjN4o1KCg/nTsaMAivU=
|
||||
github.com/rcrowley/go-metrics v0.0.0-20250401214520-65e299d6c5c9 h1:bsUq1dX0N8AOIL7EB/X911+m4EHsnWEHeJ0c+3TTBrg=
|
||||
github.com/rcrowley/go-metrics v0.0.0-20250401214520-65e299d6c5c9/go.mod h1:bCqnVzQkZxMG4s8nGwiZ5l3QUCyqpo9Y+/ZMZ9VjZe4=
|
||||
github.com/stretchr/testify v1.8.1 h1:w7B6lhMri9wdJUVmEZPGGhZzrYTPvgJArz7wNPgYKsk=
|
||||
github.com/stretchr/testify v1.8.1/go.mod h1:w2LPCIKwWwSfY2zedu0+kehJoqGctiVI29o6fzry7u4=
|
||||
github.com/stretchr/testify v1.12.1 h1:EuwCh5fleGS7H32xRwO3wRGT7DxrDhLAT6FF8MpWDWE=
|
||||
github.com/stretchr/testify v1.12.1/go.mod h1:MDEgiDPPsNp5cuIrHPPCyornHKgEVbtFUmoNlxoYthg=
|
||||
go.uber.org/dig v1.19.0 h1:BACLhebsYdpQ7IROQ1AGPjrXcP5dF80U3gKoFzbaq/4=
|
||||
go.uber.org/dig v1.19.0/go.mod h1:Us0rSJiThwCv2GteUN0Q7OKvU7n5J4dxZ9JKUXozFdE=
|
||||
go.uber.org/fx v1.24.0 h1:wE8mruvpg2kiiL1Vqd0CC+tr0/24XIB10Iwp2lLWzkg=
|
||||
@@ -28,12 +32,18 @@ go.uber.org/multierr v1.10.0 h1:S0h4aNzvfcFsC3dRF1jLoaov7oRaKqRGC/pUEJ2yvPQ=
|
||||
go.uber.org/multierr v1.10.0/go.mod h1:20+QtiLqy0Nd6FdQB9TLXag12DsQkrbs3htMFfDN80Y=
|
||||
go.uber.org/zap v1.26.0 h1:sI7k6L95XOKS281NhVKOFCUNIvv9e0w4BF8N3u+tCRo=
|
||||
go.uber.org/zap v1.26.0/go.mod h1:dtElttAiwGvoJ/vj4IwHBS/gXsEu/pZ50mUIRWuG0so=
|
||||
go.yaml.in/yaml/v3 v3.0.5 h1:N6y/pJk8buWs9NY5ERU2HSMfm+IuD/OtfdAnq6kESPw=
|
||||
go.yaml.in/yaml/v3 v3.0.5/go.mod h1:HVTZu1O7/Vkt2N+BFy8Zza+lnLsABggaTM2ZpNIGuKg=
|
||||
go4.org/netipx v0.0.0-20260823151212-3075585bcbeb h1:XBM4hvfwGAttkkiTIFfeigdfcL1xIfdKXqFdgiHGtDs=
|
||||
go4.org/netipx v0.0.0-20260823151212-3075585bcbeb/go.mod h1:PLyyIXexvUFg3Owu6p/WfdlivPbZJsZdgWZlrGope/Y=
|
||||
golang.org/x/sys v0.34.0 h1:H5Y5sJ2L2JRdyv7ROF1he/lPdvFsd0mJHFw2ThKHxLA=
|
||||
golang.org/x/sys v0.34.0/go.mod h1:BJP2sWEmIv4KK5OTEluFJCKSidICx8ciO85XgH3Ak8k=
|
||||
golang.org/x/term v0.33.0 h1:NuFncQrRcaRvVmgRkvM3j/F00gWIAlcmlB8ACEKmGIg=
|
||||
golang.org/x/term v0.33.0/go.mod h1:s18+ql9tYWp1IfpV9DmCtQDDSRBUjKaw9M1eAv5UeF0=
|
||||
golang.org/x/net v0.57.0 h1:K5+3DljvIuDG9/Jv9rvyMywYNFCQ9RSUY6OOTTkT+tE=
|
||||
golang.org/x/net v0.57.0/go.mod h1:KpXc8iv+r3XplLAG/f7Jsf9RPszJzdR0f58q9vGOuEU=
|
||||
golang.org/x/sync v0.24.0 h1:iqiDrGoDgyuvAKrhLQthUEzU/R378N78gXbBjxed7oA=
|
||||
golang.org/x/sync v0.24.0/go.mod h1:sUUOizhqBxiL6pEWpqNLUiaJn1ShEbZ6BBqskPbjZm0=
|
||||
golang.org/x/sys v0.49.0 h1:XbzkgYJHdqh/8m2Uu0W/dQv8nktxx4BFHp1M0gROTXA=
|
||||
golang.org/x/sys v0.49.0/go.mod h1:hNLxWAXmnKAxqDtdwIYC4bM9oQPEecfsnNMuSxOs3og=
|
||||
golang.org/x/term v0.45.0 h1:NwWyBmoJCbfTHpxrWoZ9C6/VxOf7ic219I8xZZFdrf0=
|
||||
golang.org/x/term v0.45.0/go.mod h1:9aqxs0blBcrm/n0L9QW0aRVD+ktan8ssZromtqJC43w=
|
||||
golang.org/x/text v0.43.0 h1:1QivrlhwAsnMOcqgOdl9my7s9OyZl8O5/UGUTJVXoWc=
|
||||
golang.org/x/text v0.43.0/go.mod h1:hbSIYA/amXcRXTFSZqh8tWHEoqRvvbKPJ2uz/KU+a+s=
|
||||
gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA=
|
||||
|
||||
+11
-2
@@ -1,5 +1,5 @@
|
||||
// Package bogon tells whether a prefix or an AS number is reserved: set aside
|
||||
// for a special purpose, so that no network should announce it on the
|
||||
// Package bogon tells whether a prefix, an address or an AS number is reserved:
|
||||
// set aside for a special purpose, so that no network should announce it on the
|
||||
// internet. A route is a bogon announcement when its prefix is reserved or its
|
||||
// origin AS is.
|
||||
//
|
||||
@@ -42,6 +42,15 @@ func Prefix(prefix netip.Prefix) (string, bool) {
|
||||
return holder.name + " (" + holder.prefix.String() + ")", true
|
||||
}
|
||||
|
||||
// Addr returns why addr is reserved, and true, when it lies inside a reserved
|
||||
// block, as Prefix decides. An IPv4 address written in IPv6 form counts as the
|
||||
// IPv4 address it holds. It returns "" and false when addr is not reserved.
|
||||
func Addr(addr netip.Addr) (string, bool) {
|
||||
addr = addr.Unmap()
|
||||
|
||||
return Prefix(netip.PrefixFrom(addr, addr.BitLen()))
|
||||
}
|
||||
|
||||
// ASN returns why asn is reserved, and true, when it is. It returns "" and
|
||||
// false when asn is not reserved.
|
||||
func ASN(asn int) (string, bool) {
|
||||
|
||||
@@ -8,6 +8,9 @@ import (
|
||||
"git.eeqj.de/sneak/routewatch/internal/bogon"
|
||||
)
|
||||
|
||||
// privateUseTen is why a prefix or an address inside 10.0.0.0/8 is reserved.
|
||||
const privateUseTen = "Private-Use (10.0.0.0/8)"
|
||||
|
||||
func TestPrefix(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
@@ -16,8 +19,8 @@ func TestPrefix(t *testing.T) {
|
||||
// want is why the prefix is reserved, or "" when it is not.
|
||||
want string
|
||||
}{
|
||||
{"10.1.2.0/24", "Private-Use (10.0.0.0/8)"},
|
||||
{"10.0.0.0/8", "Private-Use (10.0.0.0/8)"},
|
||||
{"10.1.2.0/24", privateUseTen},
|
||||
{"10.0.0.0/8", privateUseTen},
|
||||
{"192.0.2.0/24", "Documentation (TEST-NET-1) (192.0.2.0/24)"},
|
||||
{"2001:db8:1::/48", "Documentation (2001:db8::/32)"},
|
||||
{"fd00::/8", "Unique-Local (fc00::/7)"},
|
||||
@@ -56,6 +59,39 @@ func TestPrefix(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestAddr(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
tests := []struct {
|
||||
addr string
|
||||
// want is why the address is reserved, or "" when it is not.
|
||||
want string
|
||||
}{
|
||||
{"10.0.0.1", privateUseTen},
|
||||
{"fd00::1", "Unique-Local (fc00::/7)"},
|
||||
// An IPv4 address written in IPv6 form counts as the IPv4 address.
|
||||
{"::ffff:10.0.0.1", privateUseTen},
|
||||
{"::ffff:8.8.8.8", ""},
|
||||
// An address that may be announced, inside a reserved block.
|
||||
{"192.0.0.9", ""},
|
||||
// Addresses anyone may announce.
|
||||
{"8.8.8.8", ""},
|
||||
{"2001:4860:4860::8888", ""},
|
||||
}
|
||||
|
||||
for _, tc := range tests {
|
||||
t.Run(tc.addr, func(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
reason, reserved := bogon.Addr(netip.MustParseAddr(tc.addr))
|
||||
if reason != tc.want || reserved != (tc.want != "") {
|
||||
t.Errorf("Addr(%s) = %q, %v; want %q", tc.addr, reason,
|
||||
reserved, tc.want)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestASN(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
|
||||
@@ -0,0 +1,51 @@
|
||||
// Package cachedb opens cache.db, the SQLite file in the state directory that
|
||||
// holds RouteWatch's caches of answers from outside services. It is separate
|
||||
// from the routes database so that a cache write never queues behind a write
|
||||
// to that database.
|
||||
package cachedb
|
||||
|
||||
import (
|
||||
"context"
|
||||
"database/sql"
|
||||
_ "embed" // for schema.sql
|
||||
"fmt"
|
||||
"path/filepath"
|
||||
|
||||
_ "github.com/mattn/go-sqlite3" // CGO SQLite driver
|
||||
)
|
||||
|
||||
const (
|
||||
// fileName is the database's file in the state directory.
|
||||
fileName = "cache.db"
|
||||
|
||||
// busyTimeoutMs is how long a write waits for another write to finish.
|
||||
busyTimeoutMs = 5000
|
||||
)
|
||||
|
||||
//go:embed schema.sql
|
||||
var schema string
|
||||
|
||||
// Open opens cache.db in stateDir, creating the file and its tables when they
|
||||
// do not exist. The caller closes the database.
|
||||
func Open(ctx context.Context, stateDir string) (*sql.DB, error) {
|
||||
dsn := fmt.Sprintf(
|
||||
"file:%s?_journal_mode=WAL&_synchronous=NORMAL&_busy_timeout=%d"+
|
||||
"&_txlock=immediate",
|
||||
filepath.Join(stateDir, fileName),
|
||||
busyTimeoutMs,
|
||||
)
|
||||
|
||||
db, err := sql.Open("sqlite3", dsn)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("opening the cache database: %w", err)
|
||||
}
|
||||
|
||||
_, err = db.ExecContext(ctx, schema)
|
||||
if err != nil {
|
||||
_ = db.Close()
|
||||
|
||||
return nil, fmt.Errorf("creating the cache database's tables: %w", err)
|
||||
}
|
||||
|
||||
return db, nil
|
||||
}
|
||||
@@ -0,0 +1,64 @@
|
||||
package cachedb
|
||||
|
||||
import (
|
||||
"os"
|
||||
"path/filepath"
|
||||
"testing"
|
||||
)
|
||||
|
||||
// TestOpenCreatesTheFileInWALMode opens a new database, and then opens it
|
||||
// again, as a restart does.
|
||||
func TestOpenCreatesTheFileInWALMode(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
dir := t.TempDir()
|
||||
|
||||
for range 2 {
|
||||
db, err := Open(t.Context(), dir)
|
||||
if err != nil {
|
||||
t.Fatalf("Open: %v", err)
|
||||
}
|
||||
|
||||
var mode string
|
||||
|
||||
err = db.QueryRowContext(t.Context(), "PRAGMA journal_mode").Scan(&mode)
|
||||
if err != nil {
|
||||
t.Fatalf("reading the journal mode: %v", err)
|
||||
}
|
||||
|
||||
if mode != "wal" {
|
||||
t.Errorf("journal mode %q, want wal", mode)
|
||||
}
|
||||
|
||||
var rows int
|
||||
|
||||
err = db.QueryRowContext(t.Context(),
|
||||
"SELECT COUNT(*) FROM hostnames").Scan(&rows)
|
||||
if err != nil {
|
||||
t.Fatalf("reading the hostnames table: %v", err)
|
||||
}
|
||||
|
||||
err = db.Close()
|
||||
if err != nil {
|
||||
t.Fatalf("Close: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
_, err := os.Stat(filepath.Join(dir, fileName))
|
||||
if err != nil {
|
||||
t.Errorf("the file is not in the state directory: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
// TestOpenFailsWhenTheDirectoryDoesNotExist checks that a state directory that
|
||||
// is not there is reported, not ignored.
|
||||
func TestOpenFailsWhenTheDirectoryDoesNotExist(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
db, err := Open(t.Context(), filepath.Join(t.TempDir(), "missing"))
|
||||
if err == nil {
|
||||
_ = db.Close()
|
||||
|
||||
t.Fatal("Open succeeded in a directory that does not exist")
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,24 @@
|
||||
-- hostnames holds the PTR names of addresses. names has the names separated by
|
||||
-- spaces, and is empty for an address with no PTR names. expires_at is in
|
||||
-- seconds since 1970.
|
||||
CREATE TABLE IF NOT EXISTS hostnames (
|
||||
address TEXT PRIMARY KEY,
|
||||
names TEXT NOT NULL,
|
||||
expires_at INTEGER NOT NULL
|
||||
);
|
||||
|
||||
-- geolocations holds the answers of geojs. A part geojs does not know is NULL.
|
||||
-- fetched_at is in seconds since 1970.
|
||||
CREATE TABLE IF NOT EXISTS geolocations (
|
||||
address TEXT PRIMARY KEY,
|
||||
country_code TEXT,
|
||||
country TEXT,
|
||||
continent_code TEXT,
|
||||
region TEXT,
|
||||
city TEXT,
|
||||
latitude REAL,
|
||||
longitude REAL,
|
||||
accuracy_radius_km INTEGER,
|
||||
time_zone TEXT,
|
||||
fetched_at INTEGER NOT NULL
|
||||
);
|
||||
@@ -72,17 +72,34 @@ const (
|
||||
// lockTestPrefix is the prefix the batch writes in the checkpoint tests
|
||||
// store.
|
||||
lockTestPrefix = "192.0.2.0/24"
|
||||
// longBusyTimeoutMs is the busy timeout setLongBusyTimeout gives the
|
||||
// writing connection. A checkpoint that waits for an open read takes this
|
||||
// long.
|
||||
longBusyTimeoutMs = 30000
|
||||
// readerCheckpointLimit is how long a checkpoint, and a batch write
|
||||
// waiting for it, may take while a reader is open: well under the busy
|
||||
// timeout, which a checkpoint that waits for the reader takes.
|
||||
readerCheckpointLimit = time.Second
|
||||
// waiting for it, may take while a reader is open: well under
|
||||
// longBusyTimeoutMs, with room for a stall on a busy host.
|
||||
readerCheckpointLimit = 15 * time.Second
|
||||
// truncateCheckpointLimit is how long a checkpoint of a WAL file over
|
||||
// sqliteWALSizeLimitBytes may take to give up while a reader is open: its
|
||||
// wait for the reader plus a second, well under the busy timeout.
|
||||
// wait for the reader plus readerCheckpointLimit.
|
||||
truncateCheckpointLimit = truncateBusyTimeoutMs*time.Millisecond +
|
||||
time.Second
|
||||
readerCheckpointLimit
|
||||
)
|
||||
|
||||
// setLongBusyTimeout gives the writing connection of db longBusyTimeoutMs, so
|
||||
// that a checkpoint that waits for an open read takes far longer than a stall
|
||||
// on a busy host.
|
||||
func setLongBusyTimeout(ctx context.Context, t *testing.T, db *Database) {
|
||||
t.Helper()
|
||||
|
||||
_, err := db.writer.ExecContext(ctx,
|
||||
fmt.Sprintf("PRAGMA busy_timeout=%d", longBusyTimeoutMs))
|
||||
if err != nil {
|
||||
t.Fatalf("failed to set the busy timeout: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
// TestConnectionPoolPragmas holds several pooled connections open at once and
|
||||
// checks each one carries the per-connection settings from the DSN, plus the
|
||||
// process-wide heap limits, and sees the new file with auto_vacuum
|
||||
@@ -579,6 +596,8 @@ func TestCheckpointDoesNotWaitForReader(t *testing.T) {
|
||||
ctx := context.Background()
|
||||
ts := time.Now().UTC()
|
||||
|
||||
setLongBusyTimeout(ctx, t, db)
|
||||
|
||||
err = db.UpdatePrefixesBatch(map[string]time.Time{"198.51.100.0/24": ts})
|
||||
if err != nil {
|
||||
t.Fatalf("failed to store the prefix the reader reads: %v", err)
|
||||
@@ -690,6 +709,7 @@ func TestCheckpointEmptiesWALOnceNoReadIsOpen(t *testing.T) {
|
||||
}
|
||||
|
||||
fillWALPastLimit(ctx, t, db)
|
||||
setLongBusyTimeout(ctx, t, db)
|
||||
|
||||
started := time.Now()
|
||||
|
||||
|
||||
@@ -0,0 +1,253 @@
|
||||
package geo
|
||||
|
||||
import (
|
||||
"context"
|
||||
"database/sql"
|
||||
"errors"
|
||||
"fmt"
|
||||
"maps"
|
||||
"time"
|
||||
)
|
||||
|
||||
const (
|
||||
// entryLifetime is how long a geojs answer is kept after its fetch.
|
||||
entryLifetime = 7 * 24 * time.Hour
|
||||
|
||||
// flushInterval is how often new answers are written to cache.db.
|
||||
flushInterval = 5 * time.Second
|
||||
|
||||
// sweepInterval is how often expired answers are deleted.
|
||||
sweepInterval = 10 * time.Minute
|
||||
|
||||
// rowColumns are the columns of the geolocations table in the order
|
||||
// scanRow reads them.
|
||||
rowColumns = "address, country_code, country, continent_code, region, " +
|
||||
"city, latitude, longitude, accuracy_radius_km, time_zone, fetched_at"
|
||||
)
|
||||
|
||||
// rowScanner is the Scan method of *sql.Row and *sql.Rows.
|
||||
type rowScanner interface {
|
||||
Scan(dest ...any) error
|
||||
}
|
||||
|
||||
// cached returns the current answer held for address, from memory, then from
|
||||
// the answers waiting to be written, then from cache.db. It returns nil when
|
||||
// there is none.
|
||||
func (l *Locator) cached(ctx context.Context, address string) *Location {
|
||||
found, ok := l.entries.Get(address)
|
||||
if ok {
|
||||
if !l.expired(found) {
|
||||
return found
|
||||
}
|
||||
|
||||
l.entries.Remove(address)
|
||||
}
|
||||
|
||||
l.pendingMu.Lock()
|
||||
waiting := l.pending[address]
|
||||
l.pendingMu.Unlock()
|
||||
|
||||
if waiting != nil && !l.expired(waiting) {
|
||||
l.entries.Add(address, waiting)
|
||||
|
||||
return waiting
|
||||
}
|
||||
|
||||
stored, err := l.readRow(ctx, address)
|
||||
if err != nil {
|
||||
l.logger.Warn("Failed to read a geolocation cache row",
|
||||
"address", address, "error", err)
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
if stored != nil {
|
||||
l.entries.Add(address, stored)
|
||||
}
|
||||
|
||||
return stored
|
||||
}
|
||||
|
||||
// expired says whether the answer is at the end of its lifetime.
|
||||
func (l *Locator) expired(answer *Location) bool {
|
||||
return !l.now().Before(answer.Updated.Add(entryLifetime))
|
||||
}
|
||||
|
||||
// cutoff returns the fetch time at or before which an answer has expired.
|
||||
func (l *Locator) cutoff() int64 {
|
||||
return l.now().Add(-entryLifetime).Unix()
|
||||
}
|
||||
|
||||
// store keeps an answer in memory and queues it for cache.db. While cache.db
|
||||
// cannot be written, the queue stops growing at maxEntries answers.
|
||||
func (l *Locator) store(address string, answer *Location) {
|
||||
l.entries.Add(address, answer)
|
||||
|
||||
l.pendingMu.Lock()
|
||||
defer l.pendingMu.Unlock()
|
||||
|
||||
if len(l.pending) < l.maxEntries {
|
||||
l.pending[address] = answer
|
||||
}
|
||||
}
|
||||
|
||||
// keepCache writes new answers and deletes expired ones until ctx ends.
|
||||
func (l *Locator) keepCache(ctx context.Context) {
|
||||
flushes := time.NewTicker(flushInterval)
|
||||
defer flushes.Stop()
|
||||
|
||||
sweeps := time.NewTicker(sweepInterval)
|
||||
defer sweeps.Stop()
|
||||
|
||||
for {
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
l.flush(context.WithoutCancel(ctx))
|
||||
|
||||
return
|
||||
case <-flushes.C:
|
||||
l.flush(ctx)
|
||||
case <-sweeps.C:
|
||||
l.sweep(ctx)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// load reads the answers that have not expired into memory, at most maxEntries
|
||||
// of them, the newest last so that they are the last to be dropped.
|
||||
func (l *Locator) load(ctx context.Context) error {
|
||||
rows, err := l.db.QueryContext(ctx,
|
||||
"SELECT "+rowColumns+" FROM (SELECT "+rowColumns+" FROM geolocations "+
|
||||
"WHERE fetched_at > ? ORDER BY fetched_at DESC LIMIT ?) "+
|
||||
"ORDER BY fetched_at",
|
||||
l.cutoff(), l.maxEntries)
|
||||
if err != nil {
|
||||
return fmt.Errorf("reading the geolocation cache: %w", err)
|
||||
}
|
||||
|
||||
defer func() { _ = rows.Close() }()
|
||||
|
||||
for rows.Next() {
|
||||
address, answer, err := scanRow(rows)
|
||||
if err != nil {
|
||||
return fmt.Errorf("reading a geolocation cache row: %w", err)
|
||||
}
|
||||
|
||||
l.entries.Add(address, answer)
|
||||
}
|
||||
|
||||
err = rows.Err()
|
||||
if err != nil {
|
||||
return fmt.Errorf("reading the geolocation cache: %w", err)
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
// readRow returns the current answer for address in cache.db, or nil when it
|
||||
// has none.
|
||||
func (l *Locator) readRow(
|
||||
ctx context.Context, address string,
|
||||
) (*Location, error) {
|
||||
row := l.db.QueryRowContext(ctx,
|
||||
"SELECT "+rowColumns+" FROM geolocations "+
|
||||
"WHERE address = ? AND fetched_at > ?",
|
||||
address, l.cutoff())
|
||||
|
||||
_, answer, err := scanRow(row)
|
||||
if errors.Is(err, sql.ErrNoRows) {
|
||||
return nil, nil //nolint:nilnil // no row is not a failure
|
||||
}
|
||||
|
||||
return answer, err
|
||||
}
|
||||
|
||||
// scanRow reads one row of the geolocations table.
|
||||
func scanRow(row rowScanner) (string, *Location, error) {
|
||||
var (
|
||||
address string
|
||||
fetched int64
|
||||
answer Location
|
||||
)
|
||||
|
||||
err := row.Scan(&address, &answer.CountryCode, &answer.Country,
|
||||
&answer.ContinentCode, &answer.Region, &answer.City, &answer.Latitude,
|
||||
&answer.Longitude, &answer.AccuracyRadiusKm, &answer.TimeZone, &fetched)
|
||||
if err != nil {
|
||||
return "", nil, err
|
||||
}
|
||||
|
||||
answer.Source = sourceGeoJS
|
||||
answer.Updated = time.Unix(fetched, 0).UTC()
|
||||
|
||||
return address, &answer, nil
|
||||
}
|
||||
|
||||
// flush writes the pending answers to cache.db. Those it cannot write stay
|
||||
// pending, unless a newer answer for the address has arrived meanwhile.
|
||||
func (l *Locator) flush(ctx context.Context) {
|
||||
l.pendingMu.Lock()
|
||||
batch := l.pending
|
||||
l.pending = make(map[string]*Location)
|
||||
l.pendingMu.Unlock()
|
||||
|
||||
if len(batch) == 0 {
|
||||
return
|
||||
}
|
||||
|
||||
err := l.writeRows(ctx, batch)
|
||||
if err == nil {
|
||||
return
|
||||
}
|
||||
|
||||
l.logger.Error("Failed to write the geolocation cache", "error", err)
|
||||
|
||||
l.pendingMu.Lock()
|
||||
defer l.pendingMu.Unlock()
|
||||
|
||||
maps.Insert(batch, maps.All(l.pending))
|
||||
l.pending = batch
|
||||
}
|
||||
|
||||
// writeRows writes batch to cache.db in one transaction.
|
||||
func (l *Locator) writeRows(
|
||||
ctx context.Context, batch map[string]*Location,
|
||||
) error {
|
||||
tx, err := l.db.BeginTx(ctx, nil)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
defer func() { _ = tx.Rollback() }()
|
||||
|
||||
for address, answer := range batch {
|
||||
_, err = tx.ExecContext(ctx,
|
||||
"INSERT OR REPLACE INTO geolocations ("+rowColumns+") "+
|
||||
"VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)",
|
||||
address, answer.CountryCode, answer.Country, answer.ContinentCode,
|
||||
answer.Region, answer.City, answer.Latitude, answer.Longitude,
|
||||
answer.AccuracyRadiusKm, answer.TimeZone, answer.Updated.Unix())
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
|
||||
return tx.Commit()
|
||||
}
|
||||
|
||||
// sweep deletes the expired answers from memory and cache.db.
|
||||
func (l *Locator) sweep(ctx context.Context) {
|
||||
for _, address := range l.entries.Keys() {
|
||||
answer, ok := l.entries.Peek(address)
|
||||
if ok && l.expired(answer) {
|
||||
l.entries.Remove(address)
|
||||
}
|
||||
}
|
||||
|
||||
_, err := l.db.ExecContext(ctx,
|
||||
"DELETE FROM geolocations WHERE fetched_at <= ?", l.cutoff())
|
||||
if err != nil {
|
||||
l.logger.Error("Failed to delete expired geolocation cache rows",
|
||||
"error", err)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,349 @@
|
||||
package geo
|
||||
|
||||
import (
|
||||
"compress/gzip"
|
||||
"context"
|
||||
"errors"
|
||||
"fmt"
|
||||
"io"
|
||||
"io/fs"
|
||||
"net/http"
|
||||
"net/netip"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"time"
|
||||
|
||||
"github.com/oschwald/maxminddb-golang/v2"
|
||||
)
|
||||
|
||||
const (
|
||||
// DatabaseDownloadFormat is the URL of DB-IP's City Lite database, with %s
|
||||
// for the year and month.
|
||||
DatabaseDownloadFormat = "https://download.db-ip.com/free/" +
|
||||
"dbip-city-lite-%s.mmdb.gz"
|
||||
|
||||
// databaseMonthFormat is the year and month in the URL.
|
||||
databaseMonthFormat = "2006-01"
|
||||
|
||||
// databaseFileName is the copy of the database in the state directory.
|
||||
databaseFileName = "dbip-city-lite.mmdb"
|
||||
|
||||
// databaseDownloadSuffix ends the name of the file a download is written
|
||||
// to, until it has opened.
|
||||
databaseDownloadSuffix = ".download"
|
||||
|
||||
// maxDatabaseSize is the largest database accepted once unpacked, in
|
||||
// bytes.
|
||||
maxDatabaseSize = 200 << 20
|
||||
|
||||
// databaseDownloadTimeout limits one download.
|
||||
databaseDownloadTimeout = 10 * time.Minute
|
||||
|
||||
// databaseCheckInterval is how often RouteWatch checks whether a new
|
||||
// month's database is due. A new month's file appears some time on the
|
||||
// first of the month, so a failed download is tried again this much later.
|
||||
databaseCheckInterval = time.Hour
|
||||
)
|
||||
|
||||
var (
|
||||
errDatabaseStatus = errors.New("unexpected HTTP status from DB-IP")
|
||||
errDatabaseTooLong = errors.New("database is over the size limit")
|
||||
)
|
||||
|
||||
// dbipName is the part of a database record that holds an English name.
|
||||
type dbipName struct {
|
||||
Names struct {
|
||||
English string `maxminddb:"en"`
|
||||
} `maxminddb:"names"`
|
||||
}
|
||||
|
||||
// dbipRecord is the part of a database record RouteWatch reads.
|
||||
type dbipRecord struct {
|
||||
City dbipName `maxminddb:"city"`
|
||||
Continent struct {
|
||||
Code string `maxminddb:"code"`
|
||||
} `maxminddb:"continent"`
|
||||
Country struct {
|
||||
dbipName
|
||||
|
||||
ISOCode string `maxminddb:"iso_code"`
|
||||
} `maxminddb:"country"`
|
||||
Location struct {
|
||||
Latitude *float64 `maxminddb:"latitude"`
|
||||
Longitude *float64 `maxminddb:"longitude"`
|
||||
} `maxminddb:"location"`
|
||||
Subdivisions []dbipName `maxminddb:"subdivisions"`
|
||||
}
|
||||
|
||||
// location returns the record as a Location fetched at the time given. The
|
||||
// first subdivision is the region. The database has no accuracy radius and no
|
||||
// time zone.
|
||||
func (r *dbipRecord) location(fetched time.Time) *Location {
|
||||
found := &Location{
|
||||
CountryCode: nilIfEmpty(r.Country.ISOCode),
|
||||
Country: nilIfEmpty(r.Country.Names.English),
|
||||
ContinentCode: nilIfEmpty(r.Continent.Code),
|
||||
City: nilIfEmpty(r.City.Names.English),
|
||||
Latitude: r.Location.Latitude,
|
||||
Longitude: r.Location.Longitude,
|
||||
Source: sourceDBIP,
|
||||
Updated: fetched,
|
||||
}
|
||||
|
||||
if len(r.Subdivisions) > 0 {
|
||||
found.Region = nilIfEmpty(r.Subdivisions[0].Names.English)
|
||||
}
|
||||
|
||||
return found
|
||||
}
|
||||
|
||||
// nilIfEmpty returns a pointer to text, or nil when text is empty.
|
||||
func nilIfEmpty(text string) *string {
|
||||
if text == "" {
|
||||
return nil
|
||||
}
|
||||
|
||||
return &text
|
||||
}
|
||||
|
||||
// fromDatabase returns where the copy of the database places addr. It returns
|
||||
// nil when there is no copy yet or the copy has no record for addr. It also
|
||||
// returns nil, and logs the error, when reading the copy fails.
|
||||
func (l *Locator) fromDatabase(addr netip.Addr) *Location {
|
||||
l.databaseMu.RLock()
|
||||
defer l.databaseMu.RUnlock()
|
||||
|
||||
if l.database == nil {
|
||||
return nil
|
||||
}
|
||||
|
||||
result := l.database.Lookup(addr)
|
||||
|
||||
err := result.Err()
|
||||
if err != nil {
|
||||
l.logger.Error("Failed to look up an address in the database",
|
||||
"address", addr, "error", err)
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
if !result.Found() {
|
||||
return nil
|
||||
}
|
||||
|
||||
var record dbipRecord
|
||||
|
||||
err = result.Decode(&record)
|
||||
if err != nil {
|
||||
l.logger.Error("Failed to read a database record",
|
||||
"address", addr, "error", err)
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
return record.location(l.databaseFetched)
|
||||
}
|
||||
|
||||
// keepDatabaseCurrent downloads the database when it is due, at once and then
|
||||
// every databaseCheckInterval, until ctx ends.
|
||||
func (l *Locator) keepDatabaseCurrent(ctx context.Context) {
|
||||
l.downloadDatabaseIfDue(ctx)
|
||||
|
||||
checks := time.NewTicker(databaseCheckInterval)
|
||||
defer checks.Stop()
|
||||
|
||||
for {
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
return
|
||||
case <-checks.C:
|
||||
l.downloadDatabaseIfDue(ctx)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// downloadDatabaseIfDue downloads this month's database, unless the copy in
|
||||
// use was downloaded this month. A failure leaves the copy as it is.
|
||||
func (l *Locator) downloadDatabaseIfDue(ctx context.Context) {
|
||||
now := l.now().UTC()
|
||||
|
||||
if !l.databaseDue(now) {
|
||||
return
|
||||
}
|
||||
|
||||
err := l.downloadDatabase(ctx, now)
|
||||
if err != nil && ctx.Err() == nil {
|
||||
l.logger.Error("Database download failed; keeping the previous copy",
|
||||
"error", err)
|
||||
}
|
||||
}
|
||||
|
||||
// databaseDue says whether there is no open copy or it is from an earlier
|
||||
// month than now.
|
||||
func (l *Locator) databaseDue(now time.Time) bool {
|
||||
l.databaseMu.RLock()
|
||||
defer l.databaseMu.RUnlock()
|
||||
|
||||
return l.database == nil ||
|
||||
l.databaseFetched.Format(databaseMonthFormat) !=
|
||||
now.Format(databaseMonthFormat)
|
||||
}
|
||||
|
||||
// downloadDatabase fetches the database of now's month. The file replaces the
|
||||
// copy, on disk and in use, only once it has opened.
|
||||
func (l *Locator) downloadDatabase(ctx context.Context, now time.Time) error {
|
||||
ctx, cancel := context.WithTimeout(ctx, databaseDownloadTimeout)
|
||||
defer cancel()
|
||||
|
||||
url := fmt.Sprintf(l.databaseURLFormat, now.Format(databaseMonthFormat))
|
||||
|
||||
request, err := http.NewRequestWithContext(ctx, http.MethodGet, url, nil)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
request.Header.Set(userAgentHeader, userAgent)
|
||||
|
||||
response, err := l.client.Do(request)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
defer func() { _ = response.Body.Close() }()
|
||||
|
||||
if response.StatusCode != http.StatusOK {
|
||||
return fmt.Errorf("%w: %s", errDatabaseStatus, response.Status)
|
||||
}
|
||||
|
||||
return l.replaceDatabase(response.Body, now)
|
||||
}
|
||||
|
||||
// replaceDatabase unpacks the gzip stream body beside the copy, then makes the
|
||||
// result the copy once it has opened. Whatever fails, the file it wrote is
|
||||
// removed and the copy stays.
|
||||
func (l *Locator) replaceDatabase(body io.Reader, now time.Time) error {
|
||||
root, err := os.OpenRoot(l.dir)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
defer func() { _ = root.Close() }()
|
||||
|
||||
downloadName := databaseFileName + databaseDownloadSuffix
|
||||
|
||||
err = unpackDatabase(root, downloadName, body, l.maxDatabaseBytes)
|
||||
if err != nil {
|
||||
_ = root.Remove(downloadName)
|
||||
|
||||
return err
|
||||
}
|
||||
|
||||
opened, err := maxminddb.Open(filepath.Join(l.dir, downloadName))
|
||||
if err != nil {
|
||||
_ = root.Remove(downloadName)
|
||||
|
||||
return fmt.Errorf("opening the downloaded database: %w", err)
|
||||
}
|
||||
|
||||
err = root.Rename(downloadName, databaseFileName)
|
||||
if err != nil {
|
||||
_ = opened.Close()
|
||||
_ = root.Remove(downloadName)
|
||||
|
||||
return err
|
||||
}
|
||||
|
||||
l.useDatabase(opened, now.Truncate(time.Second))
|
||||
l.logger.Info("Downloaded a new copy of the database")
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
// unpackDatabase writes the gzip stream body to the file name in root, up to
|
||||
// limit bytes once unpacked.
|
||||
func unpackDatabase(
|
||||
root *os.Root, name string, body io.Reader, limit int64,
|
||||
) error {
|
||||
unpacked, err := gzip.NewReader(body)
|
||||
if err != nil {
|
||||
return fmt.Errorf("unpacking the database: %w", err)
|
||||
}
|
||||
|
||||
f, err := root.Create(name)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
defer func() { _ = f.Close() }()
|
||||
|
||||
written, err := io.Copy(f, io.LimitReader(unpacked, limit+1))
|
||||
if err != nil {
|
||||
return fmt.Errorf("unpacking the database: %w", err)
|
||||
}
|
||||
|
||||
if written > limit {
|
||||
return errDatabaseTooLong
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
// openDatabaseCopy opens the copy left by an earlier run, if there is one. A
|
||||
// copy that does not open is treated as no copy, so it is downloaded again.
|
||||
func (l *Locator) openDatabaseCopy() {
|
||||
path := filepath.Join(l.dir, databaseFileName)
|
||||
|
||||
info, err := os.Stat(path)
|
||||
if errors.Is(err, fs.ErrNotExist) {
|
||||
return
|
||||
}
|
||||
|
||||
if err != nil {
|
||||
l.logger.Error("Failed to read the database copy", "error", err)
|
||||
|
||||
return
|
||||
}
|
||||
|
||||
opened, err := maxminddb.Open(path)
|
||||
if err != nil {
|
||||
l.logger.Error("Failed to open the database copy", "error", err)
|
||||
|
||||
return
|
||||
}
|
||||
|
||||
l.useDatabase(opened, info.ModTime().UTC().Truncate(time.Second))
|
||||
}
|
||||
|
||||
// useDatabase makes opened the copy in use and closes the one it replaces.
|
||||
func (l *Locator) useDatabase(opened *maxminddb.Reader, fetched time.Time) {
|
||||
l.databaseMu.Lock()
|
||||
defer l.databaseMu.Unlock()
|
||||
|
||||
if l.database != nil {
|
||||
err := l.database.Close()
|
||||
if err != nil {
|
||||
l.logger.Error("Failed to close the old database copy", "error", err)
|
||||
}
|
||||
}
|
||||
|
||||
l.database = opened
|
||||
l.databaseFetched = fetched
|
||||
}
|
||||
|
||||
// closeDatabase closes the copy in use, after which no source can answer from
|
||||
// it.
|
||||
func (l *Locator) closeDatabase() {
|
||||
l.databaseMu.Lock()
|
||||
defer l.databaseMu.Unlock()
|
||||
|
||||
if l.database == nil {
|
||||
return
|
||||
}
|
||||
|
||||
err := l.database.Close()
|
||||
if err != nil {
|
||||
l.logger.Error("Failed to close the database copy", "error", err)
|
||||
}
|
||||
|
||||
l.database = nil
|
||||
}
|
||||
@@ -0,0 +1,151 @@
|
||||
package geo
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"io"
|
||||
"math"
|
||||
"net/http"
|
||||
"net/netip"
|
||||
"strconv"
|
||||
"strings"
|
||||
"time"
|
||||
)
|
||||
|
||||
const (
|
||||
// GeoJSEndpoint is the URL of the geojs geo endpoint.
|
||||
GeoJSEndpoint = "https://get.geojs.io/v1/ip/geo.json"
|
||||
|
||||
// geojsTimeLimit limits one request to geojs.
|
||||
geojsTimeLimit = time.Second
|
||||
|
||||
// geojsAddressSeparator separates the addresses of one request.
|
||||
geojsAddressSeparator = ","
|
||||
|
||||
// geojsUnknown is what geojs sends for a value it does not know.
|
||||
geojsUnknown = "nil"
|
||||
|
||||
// maxGeoJSAnswerBytes is far more than an answer for the most addresses
|
||||
// an IP lookup can ask about.
|
||||
maxGeoJSAnswerBytes = 1 << 20
|
||||
)
|
||||
|
||||
var errGeoJSStatus = errors.New("unexpected HTTP status from geojs")
|
||||
|
||||
// geojsAnswer is one address's entry in an answer from geojs.
|
||||
//
|
||||
//nolint:tagliatelle // the keys geojs sends
|
||||
type geojsAnswer struct {
|
||||
IP string `json:"ip"`
|
||||
CountryCode string `json:"country_code"`
|
||||
Country string `json:"country"`
|
||||
ContinentCode string `json:"continent_code"`
|
||||
Region string `json:"region"`
|
||||
City string `json:"city"`
|
||||
Latitude string `json:"latitude"`
|
||||
Longitude string `json:"longitude"`
|
||||
Accuracy *int `json:"accuracy"`
|
||||
TimeZone string `json:"timezone"`
|
||||
}
|
||||
|
||||
// askGeoJS asks geojs about addresses in one request, and returns its answers
|
||||
// by address. An address geojs left out, or gave no country, city or
|
||||
// coordinates for, has no entry.
|
||||
func (l *Locator) askGeoJS(
|
||||
ctx context.Context, addresses []string,
|
||||
) (map[string]*Location, error) {
|
||||
ctx, cancel := context.WithTimeout(ctx, l.timeLimit)
|
||||
defer cancel()
|
||||
|
||||
// A comma and a colon are legal in a query, and the escaped forms are
|
||||
// not what geojs documents.
|
||||
url := l.geojsURL + "?ip=" + strings.Join(addresses, geojsAddressSeparator)
|
||||
|
||||
request, err := http.NewRequestWithContext(ctx, http.MethodGet, url, nil)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
request.Header.Set(userAgentHeader, userAgent)
|
||||
|
||||
response, err := l.client.Do(request)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
defer func() { _ = response.Body.Close() }()
|
||||
|
||||
if response.StatusCode != http.StatusOK {
|
||||
return nil, fmt.Errorf("%w: %d", errGeoJSStatus, response.StatusCode)
|
||||
}
|
||||
|
||||
var parsed []geojsAnswer
|
||||
|
||||
err = json.NewDecoder(io.LimitReader(response.Body, maxGeoJSAnswerBytes)).
|
||||
Decode(&parsed)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("reading the geojs answer: %w", err)
|
||||
}
|
||||
|
||||
fetched := l.now().UTC().Truncate(time.Second)
|
||||
answers := make(map[string]*Location, len(parsed))
|
||||
|
||||
for _, answer := range parsed {
|
||||
addr, err := netip.ParseAddr(answer.IP)
|
||||
if err != nil {
|
||||
continue
|
||||
}
|
||||
|
||||
location := answer.location(fetched)
|
||||
if hasLocation(location) {
|
||||
answers[plainAddress(addr)] = location
|
||||
}
|
||||
}
|
||||
|
||||
return answers, nil
|
||||
}
|
||||
|
||||
// hasLocation says whether found has a country, a city or a coordinate.
|
||||
func hasLocation(found *Location) bool {
|
||||
return found.CountryCode != nil || found.Country != nil ||
|
||||
found.City != nil || found.Latitude != nil || found.Longitude != nil
|
||||
}
|
||||
|
||||
// location returns the answer as a Location fetched at the time given.
|
||||
func (a geojsAnswer) location(fetched time.Time) *Location {
|
||||
return &Location{
|
||||
CountryCode: knownText(a.CountryCode),
|
||||
Country: knownText(a.Country),
|
||||
ContinentCode: knownText(a.ContinentCode),
|
||||
Region: knownText(a.Region),
|
||||
City: knownText(a.City),
|
||||
Latitude: knownNumber(a.Latitude),
|
||||
Longitude: knownNumber(a.Longitude),
|
||||
AccuracyRadiusKm: a.Accuracy,
|
||||
TimeZone: knownText(a.TimeZone),
|
||||
Source: sourceGeoJS,
|
||||
Updated: fetched,
|
||||
}
|
||||
}
|
||||
|
||||
// knownText returns a pointer to text, or nil when geojs does not know it.
|
||||
func knownText(text string) *string {
|
||||
if text == "" || text == geojsUnknown {
|
||||
return nil
|
||||
}
|
||||
|
||||
return &text
|
||||
}
|
||||
|
||||
// knownNumber returns text as a number, or nil when geojs does not know it. NaN
|
||||
// and infinity parse, but JSON cannot carry them.
|
||||
func knownNumber(text string) *float64 {
|
||||
number, err := strconv.ParseFloat(text, 64)
|
||||
if err != nil || math.IsNaN(number) || math.IsInf(number, 0) {
|
||||
return nil
|
||||
}
|
||||
|
||||
return &number
|
||||
}
|
||||
@@ -0,0 +1,232 @@
|
||||
// Package geo finds where IP addresses are. It asks geojs.io, and keeps those
|
||||
// answers in memory and in cache.db. When geojs.io gives no location for an
|
||||
// address, it reads a copy of DB-IP's City Lite database, which it downloads
|
||||
// once a month.
|
||||
package geo
|
||||
|
||||
import (
|
||||
"context"
|
||||
"database/sql"
|
||||
"fmt"
|
||||
"log/slog"
|
||||
"net/http"
|
||||
"net/netip"
|
||||
"sync"
|
||||
"time"
|
||||
|
||||
lru "github.com/hashicorp/golang-lru/v2"
|
||||
"github.com/oschwald/maxminddb-golang/v2"
|
||||
)
|
||||
|
||||
// The values of Location.Source.
|
||||
const (
|
||||
sourceGeoJS = "geojs"
|
||||
sourceDBIP = "dbip"
|
||||
)
|
||||
|
||||
const (
|
||||
// maxEntries limits how many geojs answers are held in memory.
|
||||
maxEntries = 250_000
|
||||
|
||||
// userAgentHeader and userAgent name routewatch to geojs and DB-IP.
|
||||
userAgentHeader = "User-Agent"
|
||||
userAgent = "routewatch (+https://git.eeqj.de/sneak/routewatch)"
|
||||
)
|
||||
|
||||
// Params is what New needs.
|
||||
type Params struct {
|
||||
// DB is the open cache.db.
|
||||
DB *sql.DB
|
||||
|
||||
// Dir is the directory that holds the copy of the database.
|
||||
Dir string
|
||||
|
||||
// GeoJSURL is the URL of the geojs geo endpoint, GeoJSEndpoint in use.
|
||||
GeoJSURL string
|
||||
|
||||
// DatabaseURLFormat is the URL of the database with %s for the year and
|
||||
// month, DatabaseDownloadFormat in use.
|
||||
DatabaseURLFormat string
|
||||
|
||||
Logger *slog.Logger
|
||||
}
|
||||
|
||||
// Location is where an address is. A part the source does not know is nil.
|
||||
//
|
||||
//nolint:tagliatelle // snake_case keys, as the Go style guide asks
|
||||
type Location struct {
|
||||
CountryCode *string `json:"country_code"`
|
||||
Country *string `json:"country"`
|
||||
ContinentCode *string `json:"continent_code"`
|
||||
Region *string `json:"region"`
|
||||
City *string `json:"city"`
|
||||
Latitude *float64 `json:"latitude"`
|
||||
Longitude *float64 `json:"longitude"`
|
||||
AccuracyRadiusKm *int `json:"accuracy_radius_km"`
|
||||
TimeZone *string `json:"time_zone"`
|
||||
|
||||
// Source is geojs or dbip.
|
||||
Source string `json:"source"`
|
||||
|
||||
// Updated is when the value was fetched: the geojs answer, or the copy of
|
||||
// the database.
|
||||
Updated time.Time `json:"updated"`
|
||||
}
|
||||
|
||||
// Locator answers where addresses are. Start loads its cache and starts its
|
||||
// goroutines.
|
||||
type Locator struct {
|
||||
db *sql.DB
|
||||
dir string
|
||||
geojsURL string
|
||||
databaseURLFormat string
|
||||
timeLimit time.Duration
|
||||
maxEntries int
|
||||
maxDatabaseBytes int64
|
||||
logger *slog.Logger
|
||||
client *http.Client
|
||||
now func() time.Time
|
||||
|
||||
// entries holds geojs answers by address, and drops the least recently
|
||||
// used first.
|
||||
entries *lru.Cache[string, *Location]
|
||||
|
||||
// pending holds the answers that cache.db does not have yet.
|
||||
pendingMu sync.Mutex
|
||||
pending map[string]*Location
|
||||
|
||||
// database is the open copy, nil until one has opened, and databaseFetched
|
||||
// is when that copy was downloaded. A lookup holds the read lock, so that
|
||||
// a new copy never closes the old one under it.
|
||||
databaseMu sync.RWMutex
|
||||
database *maxminddb.Reader
|
||||
databaseFetched time.Time
|
||||
|
||||
// stop ends the goroutines Start runs, and done is closed when they have
|
||||
// returned.
|
||||
stop context.CancelFunc
|
||||
done chan struct{}
|
||||
}
|
||||
|
||||
// New returns a Locator with an empty cache. Start loads it.
|
||||
func New(params Params) (*Locator, error) {
|
||||
entries, err := lru.New[string, *Location](maxEntries)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("creating the geolocation cache: %w", err)
|
||||
}
|
||||
|
||||
return &Locator{
|
||||
db: params.DB,
|
||||
dir: params.Dir,
|
||||
geojsURL: params.GeoJSURL,
|
||||
databaseURLFormat: params.DatabaseURLFormat,
|
||||
timeLimit: geojsTimeLimit,
|
||||
maxEntries: maxEntries,
|
||||
maxDatabaseBytes: maxDatabaseSize,
|
||||
logger: params.Logger.With("component", "geo"),
|
||||
client: &http.Client{},
|
||||
now: time.Now,
|
||||
entries: entries,
|
||||
pending: make(map[string]*Location),
|
||||
done: make(chan struct{}),
|
||||
}, nil
|
||||
}
|
||||
|
||||
// Lookup returns where each of addrs is, in the same order. The entry for an
|
||||
// address neither source answers for is nil. Addresses with no current cached
|
||||
// answer go to geojs in one request. The returned Locations are shared with
|
||||
// the cache and must not be changed.
|
||||
func (l *Locator) Lookup(ctx context.Context, addrs []netip.Addr) []*Location {
|
||||
locations := make([]*Location, len(addrs))
|
||||
|
||||
var unanswered []string
|
||||
|
||||
for i, addr := range addrs {
|
||||
address := plainAddress(addr)
|
||||
|
||||
locations[i] = l.cached(ctx, address)
|
||||
if locations[i] == nil {
|
||||
unanswered = append(unanswered, address)
|
||||
}
|
||||
}
|
||||
|
||||
if len(unanswered) == 0 {
|
||||
return locations
|
||||
}
|
||||
|
||||
answers, err := l.askGeoJS(ctx, unanswered)
|
||||
if err != nil {
|
||||
l.logger.Warn("geojs lookup failed; using the database", "error", err)
|
||||
}
|
||||
|
||||
for i, addr := range addrs {
|
||||
if locations[i] != nil {
|
||||
continue
|
||||
}
|
||||
|
||||
address := plainAddress(addr)
|
||||
|
||||
answer, ok := answers[address]
|
||||
if ok {
|
||||
l.store(address, answer)
|
||||
locations[i] = answer
|
||||
|
||||
continue
|
||||
}
|
||||
|
||||
locations[i] = l.fromDatabase(addr)
|
||||
}
|
||||
|
||||
return locations
|
||||
}
|
||||
|
||||
// plainAddress returns addr as the requests to geojs and the cache name it: an
|
||||
// IPv4 address mapped into IPv6 as plain IPv4, and with no zone. A client
|
||||
// chooses the zone, so it must not reach the request.
|
||||
func plainAddress(addr netip.Addr) string {
|
||||
return addr.Unmap().WithZone("").String()
|
||||
}
|
||||
|
||||
// Start loads the cached answers that have not expired and opens the copy of
|
||||
// the database, if there is one. It then starts the goroutines that write new
|
||||
// answers, delete expired ones and download the database. They run until Stop:
|
||||
// the end of ctx does not stop them.
|
||||
func (l *Locator) Start(ctx context.Context) error {
|
||||
err := l.load(ctx)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
l.logger.Info("Geolocation cache loaded", "entries", l.entries.Len())
|
||||
|
||||
l.openDatabaseCopy()
|
||||
|
||||
ctx, l.stop = context.WithCancel(context.WithoutCancel(ctx))
|
||||
|
||||
go func() {
|
||||
defer close(l.done)
|
||||
defer l.closeDatabase()
|
||||
|
||||
var workers sync.WaitGroup
|
||||
|
||||
workers.Go(func() { l.keepCache(ctx) })
|
||||
workers.Go(func() { l.keepDatabaseCurrent(ctx) })
|
||||
workers.Wait()
|
||||
}()
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
// Stop cancels a download in progress, writes the answers cache.db does not
|
||||
// have yet, and waits until the goroutines have returned or ctx ends.
|
||||
func (l *Locator) Stop(ctx context.Context) {
|
||||
l.stop()
|
||||
|
||||
select {
|
||||
case <-l.done:
|
||||
l.logger.Info("Geolocation stopped")
|
||||
case <-ctx.Done():
|
||||
l.logger.Warn("Stop deadline passed before the geolocation cache " +
|
||||
"was written")
|
||||
}
|
||||
}
|
||||
File diff suppressed because it is too large.
Load diff
@@ -21,7 +21,7 @@ const (
|
||||
downloadTimeout = time.Minute
|
||||
|
||||
// maxDownloadSize is the size of the largest list file accepted, in
|
||||
// bytes. The largest list, FireHOL abusers, is about 2.5 MiB.
|
||||
// bytes. The largest list, iCloud Private Relay, is about 12 MiB.
|
||||
maxDownloadSize = 200 << 20
|
||||
)
|
||||
|
||||
|
||||
@@ -34,6 +34,33 @@ const (
|
||||
// addresses in two fields: exit_addresses, which has IPv4 only, and
|
||||
// or_addresses, which has IPv4 and IPv6, each with a port.
|
||||
FormatOnionooJSON
|
||||
|
||||
// FormatMullvadJSON is Mullvad's JSON list of relays. A relay's addresses
|
||||
// are in ipv4_addr_in, ipv6_addr_in and ipv4_v2ray.
|
||||
FormatMullvadJSON
|
||||
|
||||
// FormatNordVPNJSON is NordVPN's JSON list of servers. A server's
|
||||
// addresses are in station and ipv6_station.
|
||||
FormatNordVPNJSON
|
||||
|
||||
// FormatPIAJSON is Private Internet Access's JSON server list, then a
|
||||
// blank line and a signature. Each region's servers has lists of servers
|
||||
// by protocol, and each server has its address in ip.
|
||||
FormatPIAJSON
|
||||
|
||||
// FormatIVPNJSON is IVPN's JSON server list. The hosts of its wireguard
|
||||
// and openvpn gateways have their addresses in host and v2ray.
|
||||
FormatIVPNJSON
|
||||
|
||||
// FormatProtonVPN has one address or network per line, with comment lines
|
||||
// that start with #. Unlike FormatLines, a file with no addresses or
|
||||
// networks fails.
|
||||
FormatProtonVPN
|
||||
|
||||
// FormatICloudPrivateRelayCSV is Apple's CSV file of iCloud Private Relay
|
||||
// networks. Each line starts with a network, followed by its country,
|
||||
// region and city.
|
||||
FormatICloudPrivateRelayCSV
|
||||
)
|
||||
|
||||
// List is one public IP list.
|
||||
@@ -53,6 +80,9 @@ type List struct {
|
||||
|
||||
// Format is how the list's file is written.
|
||||
Format Format
|
||||
|
||||
// VPN says that the list holds servers of VPN services.
|
||||
VPN bool
|
||||
}
|
||||
|
||||
// Listing says that a list contains an address.
|
||||
@@ -63,6 +93,10 @@ type Listing struct {
|
||||
// Updated is when routewatch last downloaded the list, or last heard from
|
||||
// its publisher that its copy was still current.
|
||||
Updated time.Time `json:"updated"`
|
||||
|
||||
// VPN says that the list holds servers of VPN services. A lookup answer
|
||||
// reports it through is_vpn and vpn_providers.
|
||||
VPN bool `json:"-"`
|
||||
}
|
||||
|
||||
// listCopy is a list's last good copy in memory.
|
||||
@@ -129,6 +163,7 @@ func (b *IPLists) Lookup(addr netip.Addr) []Listing {
|
||||
Name: list.Name,
|
||||
Description: list.Description,
|
||||
Updated: c.updated,
|
||||
VPN: list.VPN,
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
@@ -43,6 +43,61 @@ const (
|
||||
],"bridges":[]}`
|
||||
)
|
||||
|
||||
// Small documents in the VPN services' formats. PIA's and IVPN's have no
|
||||
// public IPv6 addresses, and IVPN's holds addresses that are not its servers,
|
||||
// which must not be read.
|
||||
const (
|
||||
mullvadBody = `[
|
||||
{"hostname":"a-wg-001","ipv4_addr_in":"192.0.2.40","ipv6_addr_in":"2001:db8::40"},
|
||||
{"hostname":"a-br-001","ipv4_addr_in":"192.0.2.41","ipv6_addr_in":null,
|
||||
"ipv4_v2ray":"192.0.2.42"}
|
||||
]`
|
||||
nordVPNBody = `[
|
||||
{"hostname":"a1.example.com","station":"192.0.2.50","ipv6_station":""},
|
||||
{"hostname":"a2.example.com","station":"192.0.2.51","ipv6_station":"2001:db8::51"}
|
||||
]`
|
||||
piaBody = `{"groups":{"wg":[{"name":"wireguard","ports":[1337]}]},"regions":[
|
||||
{"id":"a","servers":{
|
||||
"wg":[{"ip":"192.0.2.60","cn":"a1"}],
|
||||
"ikev2":[{"ip":"192.0.2.61","cn":"a1"}]}},
|
||||
{"id":"b","servers":{"meta":[{"ip":"192.0.2.62","cn":"b1","van":true}]}}
|
||||
]}` + "\n\nc2lnbmF0dXJl\nbGluZSB0d28=\n"
|
||||
ivpnBody = `{"wireguard":[{"gateway":"a.wg.example.com","hosts":[
|
||||
{"hostname":"a1","host":"192.0.2.70","v2ray":"192.0.2.71",
|
||||
"ipv6":{"local_ip":"2001:db8::70/96"}}]}],
|
||||
"openvpn":[{"gateway":"a.gw.example.com","hosts":[
|
||||
{"hostname":"a1","host":"192.0.2.72"}]}],
|
||||
"config":{"api":{"ips":["192.0.2.73"]}}}`
|
||||
)
|
||||
|
||||
// protonVPNBody is a small file in the format of the ProtonVPN-IPs list.
|
||||
const protonVPNBody = "#\n# protonvpn_ip_ranges.txt\n" +
|
||||
"# https://github.com/tn3w/ProtonVPN-IPs\n#\n" +
|
||||
"# The /24 block of every advertised IPv4.\n#\n" +
|
||||
"192.0.2.0/24\n198.51.100.0/24\n"
|
||||
|
||||
// protonVPNEntryBody is a small file in the format of the ProtonVPN-IPs list
|
||||
// of server addresses.
|
||||
const protonVPNEntryBody = "#\n# protonvpn_entry_ips.txt\n" +
|
||||
"# https://github.com/tn3w/ProtonVPN-IPs\n#\n" +
|
||||
"# Entry IPs clients connect to, resolved from ProtonVPN hostnames.\n#\n" +
|
||||
"192.0.2.80\n2001:db8::80\n"
|
||||
|
||||
// iCloudPrivateRelayAddr is an address on iCloudPrivateRelayBody.
|
||||
const iCloudPrivateRelayAddr = "2001:db8:1::5"
|
||||
|
||||
// iCloudPrivateRelayBody is a small file in the format of Apple's iCloud
|
||||
// Private Relay file.
|
||||
const iCloudPrivateRelayBody = "192.0.2.0/31,GB,GB-EN,London,\n" +
|
||||
"198.51.100.32/27,US,US-CA,Los Angeles,\n" +
|
||||
"2001:db8:1::/64,US,US-AK,Anchorage,\n"
|
||||
|
||||
// emptyFileCase names the case of a wrong document that is an empty file.
|
||||
const emptyFileCase = "an empty file"
|
||||
|
||||
// errorPage is a file that is no list, as a publisher's error page is.
|
||||
const errorPage = "<!DOCTYPE html>\n<html></html>\n"
|
||||
|
||||
// testETag is the ETag a publisher sends with a version of the test list.
|
||||
const testETag = `"v1"`
|
||||
|
||||
@@ -158,13 +213,7 @@ func lookup(t *testing.T, b *IPLists, addr string) []Listing {
|
||||
func TestLookupFindsListedAddresses(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
tests := []struct {
|
||||
name string
|
||||
format Format
|
||||
body string
|
||||
listed []string
|
||||
unlisted []string
|
||||
}{
|
||||
runLookupCases(t, []lookupCase{
|
||||
{
|
||||
name: "single addresses",
|
||||
format: FormatLines,
|
||||
@@ -206,9 +255,83 @@ func TestLookupFindsListedAddresses(t *testing.T) {
|
||||
},
|
||||
unlisted: []string{"192.0.2.12", "2001:db8::2"},
|
||||
},
|
||||
}
|
||||
{
|
||||
name: "iCloud Private Relay networks",
|
||||
format: FormatICloudPrivateRelayCSV,
|
||||
body: iCloudPrivateRelayBody,
|
||||
listed: []string{
|
||||
"192.0.2.1", "198.51.100.63", iCloudPrivateRelayAddr,
|
||||
},
|
||||
unlisted: []string{"192.0.2.2", "198.51.100.64", "2001:db8:2::1"},
|
||||
},
|
||||
})
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
// TestLookupFindsVPNServers downloads a list in each VPN service's format and
|
||||
// looks up addresses on it and off it.
|
||||
func TestLookupFindsVPNServers(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
runLookupCases(t, []lookupCase{
|
||||
{
|
||||
name: "Mullvad relays",
|
||||
format: FormatMullvadJSON,
|
||||
body: mullvadBody,
|
||||
listed: []string{"192.0.2.40", "192.0.2.41", "192.0.2.42", "2001:db8::40"},
|
||||
unlisted: []string{"192.0.2.43", "2001:db8::41"},
|
||||
},
|
||||
{
|
||||
name: "NordVPN servers",
|
||||
format: FormatNordVPNJSON,
|
||||
body: nordVPNBody,
|
||||
listed: []string{"192.0.2.50", "192.0.2.51", "2001:db8::51"},
|
||||
unlisted: []string{"192.0.2.52", "2001:db8::50"},
|
||||
},
|
||||
{
|
||||
name: "PIA servers, with the signature after the JSON",
|
||||
format: FormatPIAJSON,
|
||||
body: piaBody,
|
||||
listed: []string{"192.0.2.60", "192.0.2.61", "192.0.2.62"},
|
||||
unlisted: []string{"192.0.2.63", "2001:db8::60"},
|
||||
},
|
||||
{
|
||||
name: "IVPN hosts",
|
||||
format: FormatIVPNJSON,
|
||||
body: ivpnBody,
|
||||
listed: []string{"192.0.2.70", "192.0.2.71", "192.0.2.72"},
|
||||
unlisted: []string{"192.0.2.73", "2001:db8::70"},
|
||||
},
|
||||
{
|
||||
name: "ProtonVPN networks",
|
||||
format: FormatProtonVPN,
|
||||
body: protonVPNBody,
|
||||
listed: []string{"192.0.2.0", "192.0.2.255", "198.51.100.7"},
|
||||
unlisted: []string{"192.0.3.1", "2001:db8::90"},
|
||||
},
|
||||
{
|
||||
name: "ProtonVPN server addresses",
|
||||
format: FormatProtonVPN,
|
||||
body: protonVPNEntryBody,
|
||||
listed: []string{"192.0.2.80", "2001:db8::80"},
|
||||
unlisted: []string{"192.0.2.81", "2001:db8::81"},
|
||||
},
|
||||
})
|
||||
}
|
||||
|
||||
// lookupCase is a list file, the addresses it holds and some it does not.
|
||||
type lookupCase struct {
|
||||
name string
|
||||
format Format
|
||||
body string
|
||||
listed []string
|
||||
unlisted []string
|
||||
}
|
||||
|
||||
// runLookupCases downloads each case's list file and checks its addresses.
|
||||
func runLookupCases(t *testing.T, cases []lookupCase) {
|
||||
t.Helper()
|
||||
|
||||
for _, tt := range cases {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
@@ -260,7 +383,7 @@ func TestFailedDownloadKeepsThePreviousCopy(t *testing.T) {
|
||||
body string
|
||||
}{
|
||||
{"server error", http.StatusInternalServerError, "unavailable"},
|
||||
{"not a list", http.StatusOK, "<!DOCTYPE html>\n<html></html>\n"},
|
||||
{"not a list", http.StatusOK, errorPage},
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
@@ -306,7 +429,7 @@ func TestFileThatIsNotAnOnionooDocumentKeepsThePreviousCopy(t *testing.T) {
|
||||
name string
|
||||
body string
|
||||
}{
|
||||
{"not JSON", "<!DOCTYPE html>\n<html></html>\n"},
|
||||
{"not JSON", errorPage},
|
||||
{"other JSON", `{"status":"ok","data":{"relays_count":1}}`},
|
||||
{"JSON array", `[{"or_addresses":["192.0.2.1:443"]}]`},
|
||||
{"no relays", `{"relays":[],"bridges":[]}`},
|
||||
@@ -348,6 +471,168 @@ func TestFileThatIsNotAnOnionooDocumentKeepsThePreviousCopy(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
// TestWrongVPNDocumentFailsToParse parses documents that are not a VPN
|
||||
// service's list. Each fails, so none can replace a good copy.
|
||||
func TestWrongVPNDocumentFailsToParse(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
formats := []struct {
|
||||
name string
|
||||
format Format
|
||||
good string
|
||||
empty string
|
||||
badAddress string
|
||||
}{
|
||||
{
|
||||
"Mullvad", FormatMullvadJSON, mullvadBody,
|
||||
`[{"hostname":"a"}]`, `[{"ipv4_addr_in":"example"}]`,
|
||||
},
|
||||
{
|
||||
"NordVPN", FormatNordVPNJSON, nordVPNBody,
|
||||
`[{"hostname":"a","station":"","ipv6_station":""}]`,
|
||||
`[{"station":"example"}]`,
|
||||
},
|
||||
{
|
||||
"PIA", FormatPIAJSON, piaBody,
|
||||
`{"regions":[{"id":"a","servers":{}}]}`,
|
||||
`{"regions":[{"servers":{"wg":[{"ip":"example"}]}}]}`,
|
||||
},
|
||||
{
|
||||
"IVPN", FormatIVPNJSON, ivpnBody,
|
||||
`{"wireguard":[{"hosts":[]}],"openvpn":[]}`,
|
||||
`{"wireguard":[{"hosts":[{"host":"example"}]}]}`,
|
||||
},
|
||||
}
|
||||
|
||||
for _, f := range formats {
|
||||
bodies := map[string]string{
|
||||
"not JSON": errorPage,
|
||||
"other JSON": `{"status":"ok","data":{"count":1}}`,
|
||||
"no addresses": f.empty,
|
||||
"bad address": f.badAddress,
|
||||
"cut off": f.good[:len(f.good)/2],
|
||||
emptyFileCase: "",
|
||||
}
|
||||
|
||||
for name, body := range bodies {
|
||||
t.Run(f.name+" "+name, func(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
addresses, err := parse(strings.NewReader(body), f.format)
|
||||
if err == nil {
|
||||
t.Errorf("parse returned %v and no error", addresses.Ranges())
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// TestWrongProtonVPNDocumentFailsToParse parses documents that are not a
|
||||
// ProtonVPN-IPs list. Each fails, so none can replace a good copy.
|
||||
func TestWrongProtonVPNDocumentFailsToParse(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
bodies := map[string]string{
|
||||
"error page": errorPage,
|
||||
"JSON": `{"status":"ok"}`,
|
||||
"comments only": "#\n# protonvpn_ip_ranges.txt\n#\n",
|
||||
"bad network": "192.0.2.0/24\nexample\n",
|
||||
emptyFileCase: "",
|
||||
}
|
||||
|
||||
for name, body := range bodies {
|
||||
t.Run(name, func(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
addresses, err := parse(strings.NewReader(body), FormatProtonVPN)
|
||||
if err == nil {
|
||||
t.Errorf("parse returned %v and no error", addresses.Ranges())
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// TestWrongICloudPrivateRelayDocumentFailsToParse parses documents that are
|
||||
// not Apple's iCloud Private Relay file. Each fails, so none can replace a
|
||||
// good copy.
|
||||
func TestWrongICloudPrivateRelayDocumentFailsToParse(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
bodies := map[string]string{
|
||||
"error page": errorPage,
|
||||
"JSON": `{"status":"ok","count":1}`,
|
||||
"no other field": "192.0.2.0/24\n",
|
||||
"bad network": "192.0.2.0/24,GB,GB-EN,London,\nexample,GB,GB-EN,London,\n",
|
||||
"an address": "192.0.2.1,GB,GB-EN,London,\n",
|
||||
emptyFileCase: "",
|
||||
}
|
||||
|
||||
for name, body := range bodies {
|
||||
t.Run(name, func(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
addresses, err := parse(strings.NewReader(body),
|
||||
FormatICloudPrivateRelayCSV)
|
||||
if err == nil {
|
||||
t.Errorf("parse returned %v and no error", addresses.Ranges())
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// TestLookupMarksVPNLists looks up an address on a VPN list and on another
|
||||
// list. Only the VPN list's listing says it is one.
|
||||
func TestLookupMarksVPNLists(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
_, url := newPublisher(t, goodListBody)
|
||||
vpnList := testList(url, FormatLines)
|
||||
vpnList.Name = "test_vpn_list"
|
||||
vpnList.VPN = true
|
||||
otherList := testList(url, FormatLines)
|
||||
|
||||
b := New(Params{
|
||||
Dir: t.TempDir(),
|
||||
Lists: []List{vpnList, otherList},
|
||||
Logger: slog.New(slog.DiscardHandler),
|
||||
})
|
||||
|
||||
for _, list := range b.lists {
|
||||
err := b.download(t.Context(), list)
|
||||
if err != nil {
|
||||
t.Fatalf("download of %s: %v", list.Name, err)
|
||||
}
|
||||
}
|
||||
|
||||
listings := lookup(t, b, listedAddr)
|
||||
if len(listings) != 2 || !listings[0].VPN || listings[1].VPN {
|
||||
t.Errorf("listings %+v, want the first of two marked as a VPN list",
|
||||
listings)
|
||||
}
|
||||
}
|
||||
|
||||
// TestPublicListsMarkTheVPNLists checks which public lists are VPN lists,
|
||||
// since their names are what vpn_providers reports.
|
||||
func TestPublicListsMarkTheVPNLists(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
var got []string
|
||||
|
||||
for _, list := range Lists() {
|
||||
if list.VPN {
|
||||
got = append(got, list.Name)
|
||||
}
|
||||
}
|
||||
|
||||
want := []string{
|
||||
"ivpn", "mullvad", "nordvpn", "pia", "protonvpn", "protonvpn_entry",
|
||||
"x4bnet", "x4bnet_v6",
|
||||
}
|
||||
if !slices.Equal(got, want) {
|
||||
t.Errorf("VPN lists %v, want %v", got, want)
|
||||
}
|
||||
}
|
||||
|
||||
// checkGoodCopyOnDisk checks that dir holds the good version of the test list
|
||||
// and no part of a failed download.
|
||||
func checkGoodCopyOnDisk(t *testing.T, dir string) {
|
||||
@@ -465,9 +750,13 @@ func TestRestartLoadsCopiesFromDisk(t *testing.T) {
|
||||
second.Start(t.Context())
|
||||
second.Stop(t.Context())
|
||||
|
||||
// The download takes its time after writing the file, and the restart
|
||||
// reads the file's time. A busy host can put seconds between the two.
|
||||
const maxUpdatedDifference = 15 * time.Second
|
||||
|
||||
listings := lookup(t, second, listedAddr)
|
||||
if len(listings) != 1 ||
|
||||
listings[0].Updated.Sub(downloaded).Abs() > time.Second {
|
||||
listings[0].Updated.Sub(downloaded).Abs() > maxUpdatedDifference {
|
||||
t.Errorf("after the restart, listings %+v, want one updated at %s",
|
||||
listings, downloaded)
|
||||
}
|
||||
@@ -645,15 +934,33 @@ func TestStopCancelsADownloadInProgress(t *testing.T) {
|
||||
func TestTorExitsListHoldsIPv6Exits(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
checkPublicListHolds(t, TorExitsList, onionooBody, onionooIPv6ExitAddr)
|
||||
}
|
||||
|
||||
// TestICloudPrivateRelayListHoldsItsNetworks downloads a file in Apple's
|
||||
// format as the public iCloud Private Relay list, which is what an address's
|
||||
// is_relay answer comes from.
|
||||
func TestICloudPrivateRelayListHoldsItsNetworks(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
checkPublicListHolds(t, ICloudPrivateRelayList, iCloudPrivateRelayBody,
|
||||
iCloudPrivateRelayAddr)
|
||||
}
|
||||
|
||||
// checkPublicListHolds downloads body as the public list called name and
|
||||
// checks that addr has one listing, which names that list.
|
||||
func checkPublicListHolds(t *testing.T, name, body, addr string) {
|
||||
t.Helper()
|
||||
|
||||
index := slices.IndexFunc(Lists(), func(list List) bool {
|
||||
return list.Name == TorExitsList
|
||||
return list.Name == name
|
||||
})
|
||||
if index < 0 {
|
||||
t.Fatalf("no list named %s", TorExitsList)
|
||||
t.Fatalf("no list named %s", name)
|
||||
}
|
||||
|
||||
list := Lists()[index]
|
||||
_, list.URL = newPublisher(t, onionooBody)
|
||||
_, list.URL = newPublisher(t, body)
|
||||
b := newTestIPLists(t.TempDir(), list)
|
||||
|
||||
err := b.download(t.Context(), list)
|
||||
@@ -661,9 +968,9 @@ func TestTorExitsListHoldsIPv6Exits(t *testing.T) {
|
||||
t.Fatalf("download: %v", err)
|
||||
}
|
||||
|
||||
listings := lookup(t, b, onionooIPv6ExitAddr)
|
||||
if len(listings) != 1 || listings[0].Name != TorExitsList {
|
||||
t.Errorf("listings %+v, want one for %s", listings, TorExitsList)
|
||||
listings := lookup(t, b, addr)
|
||||
if len(listings) != 1 || listings[0].Name != name {
|
||||
t.Errorf("listings %+v, want one for %s", listings, name)
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
+112
-1
@@ -1,10 +1,20 @@
|
||||
package iplist
|
||||
|
||||
import "time"
|
||||
import (
|
||||
"slices"
|
||||
"time"
|
||||
)
|
||||
|
||||
// TorExitsList is the name of the list of Tor exit nodes.
|
||||
const TorExitsList = "tor_exits"
|
||||
|
||||
// ICloudPrivateRelayList is the name of the list of iCloud Private Relay's
|
||||
// networks.
|
||||
const ICloudPrivateRelayList = "icloud_private_relay"
|
||||
|
||||
// protonVPNIPsURL is where the ProtonVPN-IPs mirror publishes its files.
|
||||
const protonVPNIPsURL = "https://raw.githubusercontent.com/tn3w/ProtonVPN-IPs/master/"
|
||||
|
||||
// How often each list is downloaded. No list is downloaded more often than
|
||||
// its publisher updates it or allows.
|
||||
const (
|
||||
@@ -24,6 +34,9 @@ const (
|
||||
fireHOLAbusersInterval = 5 * time.Hour
|
||||
fireHOLLevel3Interval = 3 * time.Hour
|
||||
|
||||
// Apple's iCloud Private Relay file is 12 MB, so once a day.
|
||||
iCloudPrivateRelayInterval = 24 * time.Hour
|
||||
|
||||
// Spamhaus asks that DROP be downloaded at most once a day:
|
||||
// https://www.spamhaus.org/blocklists/do-not-route-or-peer/
|
||||
spamhausDROPInterval = 24 * time.Hour
|
||||
@@ -32,12 +45,20 @@ const (
|
||||
// Onionoo sets no download limit:
|
||||
// https://metrics.torproject.org/onionoo.html
|
||||
torExitsInterval = time.Hour
|
||||
|
||||
// A VPN service's servers change slowly, so once a day is enough.
|
||||
vpnListInterval = 24 * time.Hour
|
||||
)
|
||||
|
||||
// Lists returns the public IP lists routewatch downloads. Each can be
|
||||
// downloaded whole without an account or a key. Spamhaus merged its EDROP
|
||||
// list into DROP, so DROP covers both.
|
||||
func Lists() []List {
|
||||
return slices.Concat(nonVPNLists(), vpnLists())
|
||||
}
|
||||
|
||||
// nonVPNLists returns the public lists that are not lists of VPN servers.
|
||||
func nonVPNLists() []List {
|
||||
return []List{
|
||||
{
|
||||
Name: "dshield",
|
||||
@@ -70,6 +91,13 @@ func Lists() []List {
|
||||
Interval: fireHOLLevel3Interval,
|
||||
Format: FormatLines,
|
||||
},
|
||||
{
|
||||
Name: ICloudPrivateRelayList,
|
||||
Description: "In a network where iCloud Private Relay traffic leaves",
|
||||
URL: "https://mask-api.icloud.com/egress-ip-ranges.csv",
|
||||
Interval: iCloudPrivateRelayInterval,
|
||||
Format: FormatICloudPrivateRelayCSV,
|
||||
},
|
||||
{
|
||||
Name: "spamhaus_drop",
|
||||
Description: "In an IPv4 network that Spamhaus says spammers " +
|
||||
@@ -96,3 +124,86 @@ func Lists() []List {
|
||||
},
|
||||
}
|
||||
}
|
||||
|
||||
// vpnLists returns the public lists of VPN servers. Their names are what
|
||||
// vpn_providers reports.
|
||||
func vpnLists() []List {
|
||||
return []List{
|
||||
{
|
||||
Name: "ivpn",
|
||||
Description: "A server of the IVPN service",
|
||||
URL: "https://api.ivpn.net/v5/servers.json",
|
||||
Interval: vpnListInterval,
|
||||
Format: FormatIVPNJSON,
|
||||
VPN: true,
|
||||
},
|
||||
{
|
||||
Name: "mullvad",
|
||||
Description: "A server of the Mullvad VPN service",
|
||||
URL: "https://api.mullvad.net/www/relays/all/",
|
||||
Interval: vpnListInterval,
|
||||
Format: FormatMullvadJSON,
|
||||
VPN: true,
|
||||
},
|
||||
{
|
||||
Name: "nordvpn",
|
||||
Description: "A server of the NordVPN service",
|
||||
URL: "https://api.nordvpn.com/v1/servers?limit=0" +
|
||||
"&fields[servers.station]&fields[servers.ipv6_station]" +
|
||||
"&fields[servers.hostname]",
|
||||
Interval: vpnListInterval,
|
||||
Format: FormatNordVPNJSON,
|
||||
VPN: true,
|
||||
},
|
||||
{
|
||||
Name: "pia",
|
||||
Description: "A server of the Private Internet Access VPN service",
|
||||
URL: "https://serverlist.piaservers.net/vpninfo/servers/v6",
|
||||
Interval: vpnListInterval,
|
||||
Format: FormatPIAJSON,
|
||||
VPN: true,
|
||||
},
|
||||
{
|
||||
// Proton's own server list needs a logged-in session, so this is
|
||||
// the public ProtonVPN-IPs mirror of it. Proton publishes only
|
||||
// the addresses clients connect to. The mirror's README says
|
||||
// traffic leaves from others in the same /24, so this file of
|
||||
// /24 networks is the one it says to use for detecting exits.
|
||||
Name: "protonvpn",
|
||||
Description: "In a /24 network that holds a server of the " +
|
||||
"ProtonVPN service",
|
||||
URL: protonVPNIPsURL + "protonvpn_ip_ranges.txt",
|
||||
Interval: vpnListInterval,
|
||||
Format: FormatProtonVPN,
|
||||
VPN: true,
|
||||
},
|
||||
{
|
||||
Name: "protonvpn_entry",
|
||||
Description: "A server of the ProtonVPN service",
|
||||
URL: protonVPNIPsURL + "protonvpn_entry_ips.txt",
|
||||
Interval: vpnListInterval,
|
||||
Format: FormatProtonVPN,
|
||||
VPN: true,
|
||||
},
|
||||
{
|
||||
Name: "x4bnet",
|
||||
Description: "In an IPv4 network of a VPN service, according to " +
|
||||
"X4BNet's community list",
|
||||
URL: "https://raw.githubusercontent.com/X4BNet/lists_vpn/main/" +
|
||||
"output/vpn/ipv4.txt",
|
||||
Interval: vpnListInterval,
|
||||
Format: FormatLines,
|
||||
VPN: true,
|
||||
},
|
||||
{
|
||||
Name: "x4bnet_v6",
|
||||
Description: "In an IPv6 network of a VPN service, according to " +
|
||||
"X4BNet's community list",
|
||||
URL: "https://raw.githubusercontent.com/X4BNet/lists_vpn/main/" +
|
||||
"output/vpn/ipv6.txt",
|
||||
Interval: vpnListInterval,
|
||||
Format: FormatLines,
|
||||
VPN: true,
|
||||
},
|
||||
}
|
||||
}
|
||||
+201
-7
@@ -8,6 +8,7 @@ import (
|
||||
"io"
|
||||
"net"
|
||||
"net/netip"
|
||||
"slices"
|
||||
"strings"
|
||||
|
||||
"go4.org/netipx"
|
||||
@@ -24,6 +25,10 @@ const (
|
||||
// dshieldFieldSeparator separates the fields of a DShield line.
|
||||
dshieldFieldSeparator = "\t"
|
||||
|
||||
// iCloudPrivateRelayFieldSeparator separates the fields of a line of
|
||||
// Apple's iCloud Private Relay file.
|
||||
iCloudPrivateRelayFieldSeparator = ","
|
||||
|
||||
// spamhausMetadataType is the type of the line that ends a Spamhaus
|
||||
// file. That line holds the file's metadata and no network.
|
||||
spamhausMetadataType = "metadata"
|
||||
@@ -33,7 +38,7 @@ var (
|
||||
errUnknownFormat = errors.New("unknown list format")
|
||||
errNoNetwork = errors.New("no network on the line")
|
||||
errBadRange = errors.New("first and last address make no range")
|
||||
errNoAddresses = errors.New("no relay addresses in the document")
|
||||
errNoAddresses = errors.New("no addresses in the document")
|
||||
)
|
||||
|
||||
// parse reads a list file written in format and returns the addresses it
|
||||
@@ -49,6 +54,18 @@ func parse(r io.Reader, format Format) (*netipx.IPSet, error) {
|
||||
return parseLines(r, addDShieldNetwork)
|
||||
case FormatOnionooJSON:
|
||||
return parseOnionoo(r)
|
||||
case FormatMullvadJSON:
|
||||
return parseMullvad(r)
|
||||
case FormatNordVPNJSON:
|
||||
return parseNordVPN(r)
|
||||
case FormatPIAJSON:
|
||||
return parsePIA(r)
|
||||
case FormatIVPNJSON:
|
||||
return parseIVPN(r)
|
||||
case FormatProtonVPN:
|
||||
return parseLinesNotEmpty(r, addAddressOrNetwork)
|
||||
case FormatICloudPrivateRelayCSV:
|
||||
return parseLinesNotEmpty(r, addICloudPrivateRelayNetwork)
|
||||
default:
|
||||
return nil, errUnknownFormat
|
||||
}
|
||||
@@ -168,11 +185,6 @@ func addDShieldNetwork(builder *netipx.IPSetBuilder, line string) error {
|
||||
// fields of the relays in an Onionoo details document. A document with no
|
||||
// addresses fails, because the Tor network always has relays.
|
||||
func parseOnionoo(r io.Reader) (*netipx.IPSet, error) {
|
||||
data, err := io.ReadAll(r)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
//nolint:tagliatelle // Onionoo's own field names
|
||||
var details struct {
|
||||
Relays []struct {
|
||||
@@ -181,7 +193,7 @@ func parseOnionoo(r io.Reader) (*netipx.IPSet, error) {
|
||||
} `json:"relays"`
|
||||
}
|
||||
|
||||
err = json.Unmarshal(data, &details)
|
||||
err := readJSON(r, &details)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
@@ -216,6 +228,188 @@ func parseOnionoo(r io.Reader) (*netipx.IPSet, error) {
|
||||
return addresses, nil
|
||||
}
|
||||
|
||||
// parseMullvad returns the addresses of the relays in a Mullvad document. A
|
||||
// relay with no IPv6 address has null for it, and only bridges have
|
||||
// ipv4_v2ray.
|
||||
func parseMullvad(r io.Reader) (*netipx.IPSet, error) {
|
||||
//nolint:tagliatelle // Mullvad's own field names
|
||||
var relays []struct {
|
||||
IPv4 string `json:"ipv4_addr_in"`
|
||||
IPv6 string `json:"ipv6_addr_in"`
|
||||
IPv4V2Ray string `json:"ipv4_v2ray"`
|
||||
}
|
||||
|
||||
err := readJSON(r, &relays)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
var texts []string
|
||||
|
||||
for _, relay := range relays {
|
||||
texts = append(texts, relay.IPv4, relay.IPv6, relay.IPv4V2Ray)
|
||||
}
|
||||
|
||||
return addressSet(texts)
|
||||
}
|
||||
|
||||
// parseNordVPN returns the addresses of the servers in a NordVPN document. A
|
||||
// server with no IPv6 address has an empty ipv6_station.
|
||||
func parseNordVPN(r io.Reader) (*netipx.IPSet, error) {
|
||||
//nolint:tagliatelle // NordVPN's own field names
|
||||
var servers []struct {
|
||||
Station string `json:"station"`
|
||||
IPv6Station string `json:"ipv6_station"`
|
||||
}
|
||||
|
||||
err := readJSON(r, &servers)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
var texts []string
|
||||
|
||||
for _, server := range servers {
|
||||
texts = append(texts, server.Station, server.IPv6Station)
|
||||
}
|
||||
|
||||
return addressSet(texts)
|
||||
}
|
||||
|
||||
// parsePIA returns the addresses of the servers in a Private Internet Access
|
||||
// document. json.Decoder stops at the end of the JSON, so the signature after
|
||||
// it is not read.
|
||||
func parsePIA(r io.Reader) (*netipx.IPSet, error) {
|
||||
var list struct {
|
||||
Regions []struct {
|
||||
Servers map[string][]struct {
|
||||
IP string `json:"ip"`
|
||||
} `json:"servers"`
|
||||
} `json:"regions"`
|
||||
}
|
||||
|
||||
err := json.NewDecoder(r).Decode(&list)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
var texts []string
|
||||
|
||||
for _, region := range list.Regions {
|
||||
for _, servers := range region.Servers {
|
||||
for _, server := range servers {
|
||||
texts = append(texts, server.IP)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
return addressSet(texts)
|
||||
}
|
||||
|
||||
// parseIVPN returns the addresses of the hosts in an IVPN document. A host's
|
||||
// ipv6 object holds only an address inside the VPN, so it is not read.
|
||||
func parseIVPN(r io.Reader) (*netipx.IPSet, error) {
|
||||
type gateway struct {
|
||||
Hosts []struct {
|
||||
Host string `json:"host"`
|
||||
V2Ray string `json:"v2ray"`
|
||||
} `json:"hosts"`
|
||||
}
|
||||
|
||||
var list struct {
|
||||
WireGuard []gateway `json:"wireguard"`
|
||||
OpenVPN []gateway `json:"openvpn"`
|
||||
}
|
||||
|
||||
err := readJSON(r, &list)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
var texts []string
|
||||
|
||||
for _, gw := range slices.Concat(list.WireGuard, list.OpenVPN) {
|
||||
for _, host := range gw.Hosts {
|
||||
texts = append(texts, host.Host, host.V2Ray)
|
||||
}
|
||||
}
|
||||
|
||||
return addressSet(texts)
|
||||
}
|
||||
|
||||
// parseLinesNotEmpty is parseLines for a list that always holds addresses, so
|
||||
// a file with none fails.
|
||||
func parseLinesNotEmpty(
|
||||
r io.Reader, addLine func(*netipx.IPSetBuilder, string) error,
|
||||
) (*netipx.IPSet, error) {
|
||||
addresses, err := parseLines(r, addLine)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
if len(addresses.Ranges()) == 0 {
|
||||
return nil, errNoAddresses
|
||||
}
|
||||
|
||||
return addresses, nil
|
||||
}
|
||||
|
||||
// addICloudPrivateRelayNetwork adds the network at the start of a line of
|
||||
// Apple's iCloud Private Relay file to builder.
|
||||
func addICloudPrivateRelayNetwork(builder *netipx.IPSetBuilder, line string) error {
|
||||
field, _, found := strings.Cut(line, iCloudPrivateRelayFieldSeparator)
|
||||
if !found {
|
||||
return errNoNetwork
|
||||
}
|
||||
|
||||
network, err := netip.ParsePrefix(field)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
builder.AddPrefix(network)
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
// readJSON reads r to its end and decodes it, as one JSON document, into v.
|
||||
func readJSON(r io.Reader, v any) error {
|
||||
data, err := io.ReadAll(r)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
return json.Unmarshal(data, v)
|
||||
}
|
||||
|
||||
// addressSet returns the set of the addresses in texts, skipping empty ones.
|
||||
// A list with no addresses fails, because every VPN service has servers.
|
||||
func addressSet(texts []string) (*netipx.IPSet, error) {
|
||||
var builder netipx.IPSetBuilder
|
||||
|
||||
for _, text := range texts {
|
||||
if text == "" {
|
||||
continue
|
||||
}
|
||||
|
||||
err := addAddress(&builder, text)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
}
|
||||
|
||||
addresses, err := builder.IPSet()
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
if len(addresses.Ranges()) == 0 {
|
||||
return nil, errNoAddresses
|
||||
}
|
||||
|
||||
return addresses, nil
|
||||
}
|
||||
|
||||
// addAddress adds an address, such as 192.0.2.1, to builder.
|
||||
func addAddress(builder *netipx.IPSetBuilder, text string) error {
|
||||
addr, err := netip.ParseAddr(text)
|
||||
|
||||
@@ -0,0 +1,324 @@
|
||||
// Package ptr looks up the PTR names of addresses over DNS-over-HTTPS. It
|
||||
// keeps answers in memory for as long as DNS allows, and in cache.db so that
|
||||
// the answers survive a restart.
|
||||
package ptr
|
||||
|
||||
import (
|
||||
"context"
|
||||
"database/sql"
|
||||
"fmt"
|
||||
"log/slog"
|
||||
"maps"
|
||||
"net/http"
|
||||
"net/netip"
|
||||
"slices"
|
||||
"strings"
|
||||
"sync"
|
||||
"time"
|
||||
|
||||
"git.eeqj.de/sneak/routewatch/internal/bogon"
|
||||
"github.com/miekg/dns"
|
||||
)
|
||||
|
||||
const (
|
||||
// flushInterval is how often new answers are written to cache.db.
|
||||
flushInterval = 5 * time.Second
|
||||
|
||||
// sweepInterval is how often expired answers are deleted.
|
||||
sweepInterval = 10 * time.Minute
|
||||
|
||||
// nameSeparator separates the names in a row of the hostnames table.
|
||||
nameSeparator = " "
|
||||
|
||||
// maxAddresses limits how many addresses the cache holds, so that a public
|
||||
// lookup cannot grow memory and cache.db without bound.
|
||||
maxAddresses = 250_000
|
||||
)
|
||||
|
||||
// Params is what New needs.
|
||||
type Params struct {
|
||||
// DB is the open cache.db.
|
||||
DB *sql.DB
|
||||
|
||||
// Resolvers are the URLs of the DNS-over-HTTPS resolvers to ask, in order.
|
||||
Resolvers []string
|
||||
|
||||
Logger *slog.Logger
|
||||
}
|
||||
|
||||
// entry is one address's cached answer.
|
||||
type entry struct {
|
||||
// names are the address's PTR names, nil when it has none.
|
||||
names []string
|
||||
expires time.Time
|
||||
}
|
||||
|
||||
// Cache answers PTR lookups from memory, and asks the resolvers for an address
|
||||
// it holds no current answer for. Start loads it from cache.db.
|
||||
type Cache struct {
|
||||
db *sql.DB
|
||||
resolvers []string
|
||||
timeLimit time.Duration
|
||||
maxEntries int
|
||||
logger *slog.Logger
|
||||
client *http.Client
|
||||
now func() time.Time
|
||||
|
||||
// entries holds the kept answers by address, at most maxEntries of them.
|
||||
// pending holds those that cache.db does not have yet.
|
||||
mu sync.RWMutex
|
||||
entries map[string]entry
|
||||
pending map[string]entry
|
||||
|
||||
// stop ends the goroutine Start runs, and done is closed when it returns.
|
||||
stop context.CancelFunc
|
||||
done chan struct{}
|
||||
}
|
||||
|
||||
// New returns an empty Cache. Start loads it.
|
||||
func New(params Params) *Cache {
|
||||
return &Cache{
|
||||
db: params.DB,
|
||||
resolvers: params.Resolvers,
|
||||
timeLimit: LookupTimeout,
|
||||
maxEntries: maxAddresses,
|
||||
logger: params.Logger.With("component", "ptr"),
|
||||
client: &http.Client{},
|
||||
now: time.Now,
|
||||
entries: make(map[string]entry),
|
||||
pending: make(map[string]entry),
|
||||
done: make(chan struct{}),
|
||||
}
|
||||
}
|
||||
|
||||
// LookupAddr returns the PTR names of addr, fully qualified, or none when it
|
||||
// has none. A reserved address has none, and is not sent to the resolvers. A
|
||||
// lookup takes at most LookupTimeout. A cached answer makes no outside request.
|
||||
// It returns an error when no resolver answered, and nothing is cached then.
|
||||
func (c *Cache) LookupAddr(ctx context.Context, addr string) ([]string, error) {
|
||||
parsed, err := netip.ParseAddr(addr)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("parsing the address: %w", err)
|
||||
}
|
||||
|
||||
_, reserved := bogon.Addr(parsed)
|
||||
if reserved {
|
||||
return []string{}, nil
|
||||
}
|
||||
|
||||
address := parsed.Unmap().String()
|
||||
|
||||
cached, ok := c.current(address)
|
||||
if ok {
|
||||
return slices.Clone(cached.names), nil
|
||||
}
|
||||
|
||||
reverseName, err := dns.ReverseAddr(address)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("building the reverse name: %w", err)
|
||||
}
|
||||
|
||||
result, err := c.lookup(ctx, reverseName)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
if result.ttl > 0 {
|
||||
c.store(address, entry{names: result.names, expires: c.now().Add(result.ttl)})
|
||||
}
|
||||
|
||||
return slices.Clone(result.names), nil
|
||||
}
|
||||
|
||||
// Start loads the answers that have not expired from cache.db. It then starts
|
||||
// the goroutine that writes new answers there and deletes expired ones. The
|
||||
// goroutine runs until Stop: the end of ctx does not stop it.
|
||||
func (c *Cache) Start(ctx context.Context) error {
|
||||
err := c.load(ctx)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
c.logger.Info("PTR cache loaded", "entries", len(c.entries))
|
||||
|
||||
ctx, c.stop = context.WithCancel(context.WithoutCancel(ctx))
|
||||
|
||||
go c.run(ctx)
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
// Stop ends the goroutine, which first writes the answers cache.db does not
|
||||
// have yet, and waits until it has returned or ctx ends.
|
||||
func (c *Cache) Stop(ctx context.Context) {
|
||||
c.stop()
|
||||
|
||||
select {
|
||||
case <-c.done:
|
||||
c.logger.Info("PTR cache stopped")
|
||||
case <-ctx.Done():
|
||||
c.logger.Warn("Stop deadline passed before the PTR cache was written")
|
||||
}
|
||||
}
|
||||
|
||||
// current returns the answer held for address, unless it has expired.
|
||||
func (c *Cache) current(address string) (entry, bool) {
|
||||
c.mu.RLock()
|
||||
defer c.mu.RUnlock()
|
||||
|
||||
found, ok := c.entries[address]
|
||||
if !ok || !c.now().Before(found.expires) {
|
||||
return entry{}, false
|
||||
}
|
||||
|
||||
return found, true
|
||||
}
|
||||
|
||||
// store keeps an answer in memory and queues it for cache.db. Once entries
|
||||
// holds maxEntries addresses it keeps no new address. It still replaces the
|
||||
// answer of an address already held.
|
||||
func (c *Cache) store(address string, answer entry) {
|
||||
c.mu.Lock()
|
||||
defer c.mu.Unlock()
|
||||
|
||||
_, held := c.entries[address]
|
||||
if !held && len(c.entries) >= c.maxEntries {
|
||||
return
|
||||
}
|
||||
|
||||
c.entries[address] = answer
|
||||
c.pending[address] = answer
|
||||
}
|
||||
|
||||
// run writes new answers and deletes expired ones until ctx ends.
|
||||
func (c *Cache) run(ctx context.Context) {
|
||||
defer close(c.done)
|
||||
|
||||
flushes := time.NewTicker(flushInterval)
|
||||
defer flushes.Stop()
|
||||
|
||||
sweeps := time.NewTicker(sweepInterval)
|
||||
defer sweeps.Stop()
|
||||
|
||||
for {
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
c.flush(context.WithoutCancel(ctx))
|
||||
|
||||
return
|
||||
case <-flushes.C:
|
||||
c.flush(ctx)
|
||||
case <-sweeps.C:
|
||||
c.sweep(ctx)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// load reads the rows that have not expired into memory.
|
||||
func (c *Cache) load(ctx context.Context) error {
|
||||
rows, err := c.db.QueryContext(ctx,
|
||||
"SELECT address, names, expires_at FROM hostnames WHERE expires_at > ?",
|
||||
c.now().Unix())
|
||||
if err != nil {
|
||||
return fmt.Errorf("reading the PTR cache: %w", err)
|
||||
}
|
||||
|
||||
defer func() { _ = rows.Close() }()
|
||||
|
||||
for rows.Next() {
|
||||
var (
|
||||
address, names string
|
||||
expires int64
|
||||
)
|
||||
|
||||
err = rows.Scan(&address, &names, &expires)
|
||||
if err != nil {
|
||||
return fmt.Errorf("reading a PTR cache row: %w", err)
|
||||
}
|
||||
|
||||
c.entries[address] = entry{
|
||||
names: strings.Fields(names),
|
||||
expires: time.Unix(expires, 0).UTC(),
|
||||
}
|
||||
}
|
||||
|
||||
err = rows.Err()
|
||||
if err != nil {
|
||||
return fmt.Errorf("reading the PTR cache: %w", err)
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
// flush writes the pending answers to cache.db. Those it cannot write stay
|
||||
// pending, unless a newer answer for the address has arrived meanwhile.
|
||||
func (c *Cache) flush(ctx context.Context) {
|
||||
c.mu.Lock()
|
||||
batch := c.pending
|
||||
c.pending = make(map[string]entry)
|
||||
c.mu.Unlock()
|
||||
|
||||
if len(batch) == 0 {
|
||||
return
|
||||
}
|
||||
|
||||
err := c.writeRows(ctx, batch)
|
||||
if err == nil {
|
||||
return
|
||||
}
|
||||
|
||||
c.logger.Error("Failed to write the PTR cache", "error", err)
|
||||
|
||||
c.mu.Lock()
|
||||
defer c.mu.Unlock()
|
||||
|
||||
for address, answer := range batch {
|
||||
_, newer := c.pending[address]
|
||||
if !newer {
|
||||
c.pending[address] = answer
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// writeRows writes batch to cache.db in one transaction.
|
||||
func (c *Cache) writeRows(ctx context.Context, batch map[string]entry) error {
|
||||
tx, err := c.db.BeginTx(ctx, nil)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
defer func() { _ = tx.Rollback() }()
|
||||
|
||||
for address, answer := range batch {
|
||||
_, err = tx.ExecContext(ctx,
|
||||
"INSERT OR REPLACE INTO hostnames (address, names, expires_at) "+
|
||||
"VALUES (?, ?, ?)",
|
||||
address,
|
||||
strings.Join(answer.names, nameSeparator),
|
||||
answer.expires.Unix())
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
|
||||
return tx.Commit()
|
||||
}
|
||||
|
||||
// sweep deletes the expired answers from memory and cache.db.
|
||||
func (c *Cache) sweep(ctx context.Context) {
|
||||
now := c.now()
|
||||
expired := func(_ string, answer entry) bool {
|
||||
return !now.Before(answer.expires)
|
||||
}
|
||||
|
||||
c.mu.Lock()
|
||||
maps.DeleteFunc(c.entries, expired)
|
||||
maps.DeleteFunc(c.pending, expired)
|
||||
c.mu.Unlock()
|
||||
|
||||
_, err := c.db.ExecContext(ctx,
|
||||
"DELETE FROM hostnames WHERE expires_at <= ?", now.Unix())
|
||||
if err != nil {
|
||||
c.logger.Error("Failed to delete expired PTR cache rows", "error", err)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,951 @@
|
||||
package ptr
|
||||
|
||||
import (
|
||||
"context"
|
||||
"database/sql"
|
||||
"errors"
|
||||
"io"
|
||||
"log/slog"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"slices"
|
||||
"strconv"
|
||||
"sync"
|
||||
"testing"
|
||||
"testing/synctest"
|
||||
"time"
|
||||
|
||||
"git.eeqj.de/sneak/routewatch/internal/cachedb"
|
||||
"github.com/miekg/dns"
|
||||
)
|
||||
|
||||
// These addresses are public, because a reserved address is not sent to the
|
||||
// resolvers. That rules out the documentation ranges.
|
||||
const (
|
||||
testAddress = "8.8.4.4"
|
||||
testReverse = "4.4.8.8.in-addr.arpa."
|
||||
mappedAddress = "::ffff:" + testAddress
|
||||
otherAddress = "8.8.8.8"
|
||||
testV6Address = "2606:4700:4700::1111"
|
||||
testV6Reverse = "1.1.1.1.0.0.0.0.0.0.0.0.0.0.0.0." +
|
||||
"0.0.0.0.0.0.7.4.0.0.7.4.6.0.6.2.ip6.arpa."
|
||||
)
|
||||
|
||||
const (
|
||||
testName = "host.example.com."
|
||||
otherName = "other.example.com."
|
||||
|
||||
// testTimeLimit replaces LookupTimeout in the tests that use a real
|
||||
// server. A stall on a busy host cannot use it up, and no test waits for
|
||||
// it.
|
||||
testTimeLimit = 2 * time.Minute
|
||||
)
|
||||
|
||||
// classlessOwner is the name that holds the PTR records of a CNAME chain.
|
||||
const classlessOwner = "4.0-25.4.8.8.in-addr.arpa."
|
||||
|
||||
// fixedTime returns the time a test's clock starts at.
|
||||
func fixedTime() time.Time {
|
||||
return time.Date(2026, time.January, 2, 3, 4, 5, 0, time.UTC)
|
||||
}
|
||||
|
||||
// fakeResolver is a DNS-over-HTTPS server. It answers each query with reply's
|
||||
// result, and counts its requests.
|
||||
type fakeResolver struct {
|
||||
server *httptest.Server
|
||||
|
||||
mu sync.Mutex
|
||||
requests int
|
||||
reply func(http.ResponseWriter, *http.Request, *dns.Msg)
|
||||
}
|
||||
|
||||
// newFakeResolver starts a fake resolver that answers with reply.
|
||||
func newFakeResolver(
|
||||
t *testing.T,
|
||||
reply func(http.ResponseWriter, *http.Request, *dns.Msg),
|
||||
) *fakeResolver {
|
||||
t.Helper()
|
||||
|
||||
resolver := &fakeResolver{reply: reply}
|
||||
resolver.server = httptest.NewServer(http.HandlerFunc(resolver.serve))
|
||||
t.Cleanup(resolver.server.Close)
|
||||
|
||||
return resolver
|
||||
}
|
||||
|
||||
// serve checks that the request is RFC 8484 wire format and passes the
|
||||
// question on to the current reply.
|
||||
func (f *fakeResolver) serve(w http.ResponseWriter, r *http.Request) {
|
||||
f.mu.Lock()
|
||||
f.requests++
|
||||
reply := f.reply
|
||||
f.mu.Unlock()
|
||||
|
||||
body, err := io.ReadAll(r.Body)
|
||||
if err != nil {
|
||||
http.Error(w, err.Error(), http.StatusBadRequest)
|
||||
|
||||
return
|
||||
}
|
||||
|
||||
var query dns.Msg
|
||||
|
||||
err = query.Unpack(body)
|
||||
if err != nil || r.Header.Get("Content-Type") != dnsMessageType ||
|
||||
r.Header.Get("Accept") != dnsMessageType || r.Method != http.MethodPost {
|
||||
http.Error(w, "not an RFC 8484 request", http.StatusBadRequest)
|
||||
|
||||
return
|
||||
}
|
||||
|
||||
reply(w, r, &query)
|
||||
}
|
||||
|
||||
func (f *fakeResolver) setReply(
|
||||
reply func(http.ResponseWriter, *http.Request, *dns.Msg),
|
||||
) {
|
||||
f.mu.Lock()
|
||||
defer f.mu.Unlock()
|
||||
|
||||
f.reply = reply
|
||||
}
|
||||
|
||||
func (f *fakeResolver) requestCount() int {
|
||||
f.mu.Lock()
|
||||
defer f.mu.Unlock()
|
||||
|
||||
return f.requests
|
||||
}
|
||||
|
||||
// writeReply packs reply and writes it as the answer to query.
|
||||
func writeReply(w http.ResponseWriter, query, reply *dns.Msg) {
|
||||
reply.SetReply(query)
|
||||
|
||||
wire, err := reply.Pack()
|
||||
if err != nil {
|
||||
http.Error(w, err.Error(), http.StatusInternalServerError)
|
||||
|
||||
return
|
||||
}
|
||||
|
||||
w.Header().Set("Content-Type", dnsMessageType)
|
||||
_, _ = w.Write(wire)
|
||||
}
|
||||
|
||||
// ptrReply answers with one PTR record for each name, each with the TTL ttl.
|
||||
func ptrReply(
|
||||
ttl uint32, names ...string,
|
||||
) func(http.ResponseWriter, *http.Request, *dns.Msg) {
|
||||
return func(w http.ResponseWriter, _ *http.Request, query *dns.Msg) {
|
||||
reply := new(dns.Msg)
|
||||
|
||||
for _, name := range names {
|
||||
reply.Answer = append(reply.Answer, &dns.PTR{
|
||||
Hdr: dns.RR_Header{
|
||||
Name: query.Question[0].Name, Rrtype: dns.TypePTR,
|
||||
Class: dns.ClassINET, Ttl: ttl,
|
||||
},
|
||||
Ptr: name,
|
||||
})
|
||||
}
|
||||
|
||||
writeReply(w, query, reply)
|
||||
}
|
||||
}
|
||||
|
||||
// rcodeReply answers with the response code and nothing else.
|
||||
func rcodeReply(rcode int) func(http.ResponseWriter, *http.Request, *dns.Msg) {
|
||||
return func(w http.ResponseWriter, _ *http.Request, query *dns.Msg) {
|
||||
reply := new(dns.Msg)
|
||||
reply.SetRcode(query, rcode)
|
||||
|
||||
wire, err := reply.Pack()
|
||||
if err != nil {
|
||||
http.Error(w, err.Error(), http.StatusInternalServerError)
|
||||
|
||||
return
|
||||
}
|
||||
|
||||
_, _ = w.Write(wire)
|
||||
}
|
||||
}
|
||||
|
||||
// soaReply answers with the response code and a SOA record with these TTLs.
|
||||
func soaReply(
|
||||
rcode int, soaTTL, minimum uint32,
|
||||
) func(http.ResponseWriter, *http.Request, *dns.Msg) {
|
||||
return func(w http.ResponseWriter, _ *http.Request, query *dns.Msg) {
|
||||
reply := new(dns.Msg)
|
||||
reply.SetRcode(query, rcode)
|
||||
reply.Ns = []dns.RR{&dns.SOA{
|
||||
Hdr: dns.RR_Header{
|
||||
Name: "arpa.", Rrtype: dns.TypeSOA, Class: dns.ClassINET, Ttl: soaTTL,
|
||||
},
|
||||
Ns: "ns.example.com.", Mbox: "hostmaster.example.com.", Minttl: minimum,
|
||||
}}
|
||||
|
||||
wire, err := reply.Pack()
|
||||
if err != nil {
|
||||
http.Error(w, err.Error(), http.StatusInternalServerError)
|
||||
|
||||
return
|
||||
}
|
||||
|
||||
_, _ = w.Write(wire)
|
||||
}
|
||||
}
|
||||
|
||||
// hangingReply never answers: it waits until the client gives up.
|
||||
func hangingReply(_ http.ResponseWriter, r *http.Request, _ *dns.Msg) {
|
||||
<-r.Context().Done()
|
||||
}
|
||||
|
||||
// statusReply answers with an HTTP status.
|
||||
func statusReply(status int) func(http.ResponseWriter, *http.Request, *dns.Msg) {
|
||||
return func(w http.ResponseWriter, _ *http.Request, _ *dns.Msg) {
|
||||
w.WriteHeader(status)
|
||||
}
|
||||
}
|
||||
|
||||
// openTestDB opens a cache.db in dir.
|
||||
func openTestDB(t *testing.T, dir string) *sql.DB {
|
||||
t.Helper()
|
||||
|
||||
db, err := cachedb.Open(t.Context(), dir)
|
||||
if err != nil {
|
||||
t.Fatalf("opening the cache database: %v", err)
|
||||
}
|
||||
|
||||
t.Cleanup(func() { _ = db.Close() })
|
||||
|
||||
return db
|
||||
}
|
||||
|
||||
// testClock is a clock a test moves by hand.
|
||||
type testClock struct {
|
||||
mu sync.Mutex
|
||||
now time.Time
|
||||
}
|
||||
|
||||
func (c *testClock) time() time.Time {
|
||||
c.mu.Lock()
|
||||
defer c.mu.Unlock()
|
||||
|
||||
return c.now
|
||||
}
|
||||
|
||||
func (c *testClock) advance(d time.Duration) {
|
||||
c.mu.Lock()
|
||||
defer c.mu.Unlock()
|
||||
|
||||
c.now = c.now.Add(d)
|
||||
}
|
||||
|
||||
// newTestCache returns a Cache that asks the resolvers in order, reads the
|
||||
// time from a clock the test holds, has testTimeLimit, and is not started.
|
||||
func newTestCache(
|
||||
t *testing.T, db *sql.DB, resolvers ...*fakeResolver,
|
||||
) (*Cache, *testClock) {
|
||||
t.Helper()
|
||||
|
||||
urls := make([]string, 0, len(resolvers))
|
||||
for _, resolver := range resolvers {
|
||||
urls = append(urls, resolver.server.URL)
|
||||
}
|
||||
|
||||
cache := New(Params{DB: db, Resolvers: urls, Logger: slog.Default()})
|
||||
clock := &testClock{now: fixedTime()}
|
||||
cache.now = clock.time
|
||||
cache.timeLimit = testTimeLimit
|
||||
|
||||
return cache, clock
|
||||
}
|
||||
|
||||
// handlerTransport hands each request to the fake resolver for its host, with
|
||||
// no network between them. A test in a synctest bubble needs this: the bubble's
|
||||
// clock stands still while one of its goroutines waits on a real connection. As
|
||||
// over a network, a request whose time is up is not sent.
|
||||
type handlerTransport map[string]*fakeResolver
|
||||
|
||||
func (h handlerTransport) RoundTrip(r *http.Request) (*http.Response, error) {
|
||||
recorder := httptest.NewRecorder()
|
||||
|
||||
if r.Context().Err() == nil {
|
||||
h[r.URL.Host].serve(recorder, r)
|
||||
}
|
||||
|
||||
err := r.Context().Err()
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
return recorder.Result(), nil
|
||||
}
|
||||
|
||||
// newBubbleCache returns a Cache with the real time limit, for a test in a
|
||||
// synctest bubble. It has one fake resolver for each reply, in order, reached
|
||||
// through a handlerTransport.
|
||||
func newBubbleCache(
|
||||
replies ...func(http.ResponseWriter, *http.Request, *dns.Msg),
|
||||
) (*Cache, []*fakeResolver) {
|
||||
transport := make(handlerTransport, len(replies))
|
||||
resolvers := make([]*fakeResolver, 0, len(replies))
|
||||
urls := make([]string, 0, len(replies))
|
||||
|
||||
for i, reply := range replies {
|
||||
host := "resolver" + strconv.Itoa(i) + ".example.com"
|
||||
resolver := &fakeResolver{reply: reply}
|
||||
|
||||
transport[host] = resolver
|
||||
resolvers = append(resolvers, resolver)
|
||||
urls = append(urls, "https://"+host+"/dns-query")
|
||||
}
|
||||
|
||||
cache := New(Params{Resolvers: urls, Logger: slog.Default()})
|
||||
cache.client = &http.Client{Transport: transport}
|
||||
|
||||
return cache, resolvers
|
||||
}
|
||||
|
||||
// lookupOK looks up address and fails the test on an error.
|
||||
func lookupOK(t *testing.T, cache *Cache, address string) []string {
|
||||
t.Helper()
|
||||
|
||||
names, err := cache.LookupAddr(t.Context(), address)
|
||||
if err != nil {
|
||||
t.Fatalf("LookupAddr(%s): %v", address, err)
|
||||
}
|
||||
|
||||
return names
|
||||
}
|
||||
|
||||
func countRows(t *testing.T, db *sql.DB) int {
|
||||
t.Helper()
|
||||
|
||||
var rows int
|
||||
|
||||
err := db.QueryRowContext(t.Context(),
|
||||
"SELECT COUNT(*) FROM hostnames").Scan(&rows)
|
||||
if err != nil {
|
||||
t.Fatalf("counting rows: %v", err)
|
||||
}
|
||||
|
||||
return rows
|
||||
}
|
||||
|
||||
func TestLookupAsksForTheReverseName(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
tests := []struct {
|
||||
address string
|
||||
want string
|
||||
}{
|
||||
{testAddress, testReverse},
|
||||
{mappedAddress, testReverse},
|
||||
{testV6Address, testV6Reverse},
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
var asked string
|
||||
|
||||
resolver := newFakeResolver(t,
|
||||
func(w http.ResponseWriter, r *http.Request, query *dns.Msg) {
|
||||
asked = query.Question[0].Name
|
||||
if query.Question[0].Qtype != dns.TypePTR {
|
||||
t.Errorf("query type %d, want PTR", query.Question[0].Qtype)
|
||||
}
|
||||
|
||||
ptrReply(300, testName)(w, r, query)
|
||||
})
|
||||
cache, _ := newTestCache(t, openTestDB(t, t.TempDir()), resolver)
|
||||
|
||||
names := lookupOK(t, cache, tt.address)
|
||||
if !slices.Equal(names, []string{testName}) {
|
||||
t.Errorf("%s: names %v, want %s", tt.address, names, testName)
|
||||
}
|
||||
|
||||
if asked != tt.want {
|
||||
t.Errorf("%s: asked for %s, want %s", tt.address, asked, tt.want)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// TestOnlyPublicAddressesAreSentToTheResolvers gives every resolver an answer
|
||||
// to send, which a reserved address must not get.
|
||||
func TestOnlyPublicAddressesAreSentToTheResolvers(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
tests := []struct {
|
||||
name string
|
||||
address string
|
||||
want []string
|
||||
requests int
|
||||
}{
|
||||
{"private IPv4", "10.0.0.1", []string{}, 0},
|
||||
{"documentation IPv4", "192.0.2.1", []string{}, 0},
|
||||
{"private IPv4 in IPv6 form", "::ffff:10.0.0.1", []string{}, 0},
|
||||
{"unique-local IPv6", "fd00::1", []string{}, 0},
|
||||
{"documentation IPv6", "2001:db8::1", []string{}, 0},
|
||||
{"public IPv4", testAddress, []string{testName}, 1},
|
||||
{"public IPv4 in IPv6 form", mappedAddress, []string{testName}, 1},
|
||||
{"public IPv6", testV6Address, []string{testName}, 1},
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
resolver := newFakeResolver(t, ptrReply(300, testName))
|
||||
cache, _ := newTestCache(t, openTestDB(t, t.TempDir()), resolver)
|
||||
|
||||
names := lookupOK(t, cache, tt.address)
|
||||
if names == nil || !slices.Equal(names, tt.want) {
|
||||
t.Errorf("%s: names %#v, want %#v", tt.name, names, tt.want)
|
||||
}
|
||||
|
||||
if got := resolver.requestCount(); got != tt.requests {
|
||||
t.Errorf("%s: %d requests, want %d", tt.name, got, tt.requests)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// TestResolversAreTriedInOrderUntilOneAnswers has the first two resolvers fail,
|
||||
// in different ways, and the third answer. The fourth is not asked.
|
||||
func TestResolversAreTriedInOrderUntilOneAnswers(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
var (
|
||||
mu sync.Mutex
|
||||
order []int
|
||||
)
|
||||
|
||||
asked := func(
|
||||
index int, reply func(http.ResponseWriter, *http.Request, *dns.Msg),
|
||||
) func(http.ResponseWriter, *http.Request, *dns.Msg) {
|
||||
return func(w http.ResponseWriter, r *http.Request, query *dns.Msg) {
|
||||
mu.Lock()
|
||||
|
||||
order = append(order, index)
|
||||
|
||||
mu.Unlock()
|
||||
|
||||
reply(w, r, query)
|
||||
}
|
||||
}
|
||||
|
||||
resolvers := []*fakeResolver{
|
||||
newFakeResolver(t, asked(0, statusReply(http.StatusServiceUnavailable))),
|
||||
newFakeResolver(t, asked(1, rcodeReply(dns.RcodeServerFailure))),
|
||||
newFakeResolver(t, asked(2, ptrReply(300, testName))),
|
||||
newFakeResolver(t, asked(3, ptrReply(300, otherName))),
|
||||
}
|
||||
cache, _ := newTestCache(t, openTestDB(t, t.TempDir()), resolvers...)
|
||||
|
||||
names := lookupOK(t, cache, testAddress)
|
||||
if !slices.Equal(names, []string{testName}) {
|
||||
t.Errorf("names %v, want %s", names, testName)
|
||||
}
|
||||
|
||||
if !slices.Equal(order, []int{0, 1, 2}) {
|
||||
t.Errorf("resolvers asked %v, want [0 1 2]", order)
|
||||
}
|
||||
}
|
||||
|
||||
// TestResolverThatNeverAnswersLeavesTimeForTheNext would fail if the first
|
||||
// resolver could take the whole time limit. The first gets half of it, the
|
||||
// share of one resolver in two. The test runs in a synctest bubble, whose clock
|
||||
// is fake, so a stall on the host uses none of the limit.
|
||||
func TestResolverThatNeverAnswersLeavesTimeForTheNext(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
synctest.Test(t, func(t *testing.T) {
|
||||
cache, _ := newBubbleCache(hangingReply, ptrReply(300, testName))
|
||||
|
||||
start := time.Now()
|
||||
names := lookupOK(t, cache, testAddress)
|
||||
elapsed := time.Since(start)
|
||||
|
||||
if !slices.Equal(names, []string{testName}) {
|
||||
t.Errorf("names %v, want %s", names, testName)
|
||||
}
|
||||
|
||||
if elapsed != time.Second {
|
||||
t.Errorf("took %s, want 1s", elapsed)
|
||||
}
|
||||
})
|
||||
}
|
||||
|
||||
// TestLookupStopsAtTheTimeLimit has no resolver answer. The lookup fails when
|
||||
// 2 seconds have passed, having asked every resolver. The test runs in a
|
||||
// synctest bubble, whose clock is fake, so a stall on the host uses none of the
|
||||
// limit.
|
||||
func TestLookupStopsAtTheTimeLimit(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
synctest.Test(t, func(t *testing.T) {
|
||||
cache, resolvers := newBubbleCache(
|
||||
hangingReply, hangingReply, hangingReply, hangingReply)
|
||||
|
||||
start := time.Now()
|
||||
_, err := cache.LookupAddr(t.Context(), testAddress)
|
||||
elapsed := time.Since(start)
|
||||
|
||||
if !errors.Is(err, context.DeadlineExceeded) {
|
||||
t.Errorf("error %v, want %v", err, context.DeadlineExceeded)
|
||||
}
|
||||
|
||||
if elapsed != 2*time.Second {
|
||||
t.Errorf("took %s, want 2s", elapsed)
|
||||
}
|
||||
|
||||
for i, resolver := range resolvers {
|
||||
if resolver.requestCount() != 1 {
|
||||
t.Errorf("resolver %d asked %d times, want once",
|
||||
i, resolver.requestCount())
|
||||
}
|
||||
}
|
||||
})
|
||||
}
|
||||
|
||||
// TestAnswerIsServedFromMemoryUntilItExpires makes a second lookup within the
|
||||
// TTL, which asks nobody, and a third after it, which asks again.
|
||||
func TestAnswerIsServedFromMemoryUntilItExpires(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
resolver := newFakeResolver(t, ptrReply(300, testName))
|
||||
cache, clock := newTestCache(t, openTestDB(t, t.TempDir()), resolver)
|
||||
|
||||
lookupOK(t, cache, testAddress)
|
||||
|
||||
names := lookupOK(t, cache, testAddress)
|
||||
if !slices.Equal(names, []string{testName}) {
|
||||
t.Errorf("cached names %v, want %s", names, testName)
|
||||
}
|
||||
|
||||
if got := resolver.requestCount(); got != 1 {
|
||||
t.Errorf("%d requests after the second lookup, want 1", got)
|
||||
}
|
||||
|
||||
clock.advance(300 * time.Second)
|
||||
lookupOK(t, cache, testAddress)
|
||||
|
||||
if got := resolver.requestCount(); got != 2 {
|
||||
t.Errorf("%d requests after the TTL passed, want 2", got)
|
||||
}
|
||||
}
|
||||
|
||||
// TestAnAddressHasOneCacheEntryHoweverItIsWritten looks up an IPv4 address in
|
||||
// its IPv4-mapped IPv6 form, then in its usual form.
|
||||
func TestAnAddressHasOneCacheEntryHoweverItIsWritten(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
resolver := newFakeResolver(t, ptrReply(300, testName))
|
||||
cache, _ := newTestCache(t, openTestDB(t, t.TempDir()), resolver)
|
||||
|
||||
lookupOK(t, cache, mappedAddress)
|
||||
lookupOK(t, cache, testAddress)
|
||||
|
||||
if got := resolver.requestCount(); got != 1 {
|
||||
t.Errorf("%d requests, want 1", got)
|
||||
}
|
||||
}
|
||||
|
||||
// TestCallerCannotChangeACachedAnswer changes the list each lookup returned, as
|
||||
// the server does when it removes the trailing dots. The second lookup is the
|
||||
// first answered from the cache. The third shows that the cached list did not
|
||||
// change with the list the second returned.
|
||||
func TestCallerCannotChangeACachedAnswer(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
resolver := newFakeResolver(t, ptrReply(300, testName))
|
||||
cache, _ := newTestCache(t, openTestDB(t, t.TempDir()), resolver)
|
||||
|
||||
for range 3 {
|
||||
names := lookupOK(t, cache, testAddress)
|
||||
if !slices.Equal(names, []string{testName}) {
|
||||
t.Errorf("names %v, want %s", names, testName)
|
||||
}
|
||||
|
||||
names[0] = "changed"
|
||||
}
|
||||
}
|
||||
|
||||
func TestTTLIsClamped(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
tests := []struct {
|
||||
name string
|
||||
ttl uint32
|
||||
want time.Duration
|
||||
}{
|
||||
{"below the minimum", 1, time.Minute},
|
||||
{"zero", 0, time.Minute},
|
||||
{"at the minimum", 60, time.Minute},
|
||||
{"in range", 300, 300 * time.Second},
|
||||
{"at the maximum", 86400, 24 * time.Hour},
|
||||
{"above the maximum", 604800, 24 * time.Hour},
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
resolver := newFakeResolver(t, ptrReply(tt.ttl, testName))
|
||||
cache, _ := newTestCache(t, openTestDB(t, t.TempDir()), resolver)
|
||||
|
||||
lookupOK(t, cache, testAddress)
|
||||
|
||||
got := cache.entries[testAddress].expires.Sub(fixedTime())
|
||||
if got != tt.want {
|
||||
t.Errorf("%s: kept for %s, want %s", tt.name, got, tt.want)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// TestSmallestTTLOfTheChainCounts answers with a CNAME record and two PTR
|
||||
// records, whose TTLs all differ.
|
||||
func TestSmallestTTLOfTheChainCounts(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
reply := func(w http.ResponseWriter, _ *http.Request, query *dns.Msg) {
|
||||
owner := query.Question[0].Name
|
||||
answer := new(dns.Msg)
|
||||
answer.Answer = []dns.RR{
|
||||
&dns.CNAME{
|
||||
Hdr: dns.RR_Header{
|
||||
Name: owner, Rrtype: dns.TypeCNAME, Class: dns.ClassINET, Ttl: 200,
|
||||
},
|
||||
Target: classlessOwner,
|
||||
},
|
||||
&dns.PTR{
|
||||
Hdr: dns.RR_Header{
|
||||
Name: classlessOwner, Rrtype: dns.TypePTR,
|
||||
Class: dns.ClassINET, Ttl: 900,
|
||||
},
|
||||
Ptr: testName,
|
||||
},
|
||||
&dns.PTR{
|
||||
Hdr: dns.RR_Header{
|
||||
Name: classlessOwner, Rrtype: dns.TypePTR,
|
||||
Class: dns.ClassINET, Ttl: 600,
|
||||
},
|
||||
Ptr: otherName,
|
||||
},
|
||||
}
|
||||
|
||||
writeReply(w, query, answer)
|
||||
}
|
||||
|
||||
cache, _ := newTestCache(t, openTestDB(t, t.TempDir()), newFakeResolver(t, reply))
|
||||
|
||||
names := lookupOK(t, cache, testAddress)
|
||||
if !slices.Equal(names, []string{testName, otherName}) {
|
||||
t.Errorf("names %v, want both PTR names", names)
|
||||
}
|
||||
|
||||
got := cache.entries[testAddress].expires.Sub(fixedTime())
|
||||
if got != 200*time.Second {
|
||||
t.Errorf("kept for %s, want 200s", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestNegativeTTL(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
tests := []struct {
|
||||
name string
|
||||
rcode int
|
||||
soaTTL uint32
|
||||
minimum uint32
|
||||
want time.Duration
|
||||
}{
|
||||
{"name does not exist", dns.RcodeNameError, 3600, 300, 300 * time.Second},
|
||||
{"no PTR record", dns.RcodeSuccess, 3600, 300, 300 * time.Second},
|
||||
{"limited to an hour", dns.RcodeNameError, 86400, 7200, time.Hour},
|
||||
{"SOA's own TTL is smaller", dns.RcodeNameError, 100, 300, 100 * time.Second},
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
resolver := newFakeResolver(t, soaReply(tt.rcode, tt.soaTTL, tt.minimum))
|
||||
cache, _ := newTestCache(t, openTestDB(t, t.TempDir()), resolver)
|
||||
|
||||
names := lookupOK(t, cache, testAddress)
|
||||
if len(names) != 0 {
|
||||
t.Errorf("%s: names %v, want none", tt.name, names)
|
||||
}
|
||||
|
||||
got := cache.entries[testAddress].expires.Sub(fixedTime())
|
||||
if got != tt.want {
|
||||
t.Errorf("%s: kept for %s, want %s", tt.name, got, tt.want)
|
||||
}
|
||||
|
||||
lookupOK(t, cache, testAddress)
|
||||
|
||||
if count := resolver.requestCount(); count != 1 {
|
||||
t.Errorf("%s: %d requests, want 1", tt.name, count)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// TestAnswerWithoutNamesOrSOAIsNotCached has nothing to say how long the
|
||||
// answer may be kept.
|
||||
func TestAnswerWithoutNamesOrSOAIsNotCached(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
resolver := newFakeResolver(t, rcodeReply(dns.RcodeNameError))
|
||||
cache, _ := newTestCache(t, openTestDB(t, t.TempDir()), resolver)
|
||||
|
||||
lookupOK(t, cache, testAddress)
|
||||
lookupOK(t, cache, testAddress)
|
||||
|
||||
if got := resolver.requestCount(); got != 2 {
|
||||
t.Errorf("%d requests, want 2", got)
|
||||
}
|
||||
}
|
||||
|
||||
// TestFailedQueryIsNotCached has the only resolver fail, and then recover.
|
||||
func TestFailedQueryIsNotCached(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
resolver := newFakeResolver(t, rcodeReply(dns.RcodeServerFailure))
|
||||
cache, _ := newTestCache(t, openTestDB(t, t.TempDir()), resolver)
|
||||
|
||||
_, err := cache.LookupAddr(t.Context(), testAddress)
|
||||
if !errors.Is(err, errBadResponse) {
|
||||
t.Fatalf("error %v, want %v", err, errBadResponse)
|
||||
}
|
||||
|
||||
if len(cache.entries)+len(cache.pending) != 0 {
|
||||
t.Error("a failed query left an entry in the cache")
|
||||
}
|
||||
|
||||
resolver.setReply(ptrReply(300, testName))
|
||||
|
||||
names := lookupOK(t, cache, testAddress)
|
||||
if !slices.Equal(names, []string{testName}) {
|
||||
t.Errorf("names %v, want %s", names, testName)
|
||||
}
|
||||
}
|
||||
|
||||
// TestNamesThatAreNotHostNamesAreDropped answers with a name that holds markup
|
||||
// beside a good one, as a PTR record can.
|
||||
func TestNamesThatAreNotHostNamesAreDropped(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
resolver := newFakeResolver(t,
|
||||
ptrReply(300, "bad<script>.example.com.", testName, "."))
|
||||
cache, _ := newTestCache(t, openTestDB(t, t.TempDir()), resolver)
|
||||
|
||||
names := lookupOK(t, cache, testAddress)
|
||||
if !slices.Equal(names, []string{testName}) {
|
||||
t.Errorf("names %v, want only %s", names, testName)
|
||||
}
|
||||
}
|
||||
|
||||
// TestCacheSurvivesARestart stops a cache that holds a positive and a negative
|
||||
// answer, and starts a new one on the reopened file. It makes no request.
|
||||
func TestCacheSurvivesARestart(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
dir := t.TempDir()
|
||||
db := openTestDB(t, dir)
|
||||
|
||||
resolver := newFakeResolver(t,
|
||||
func(w http.ResponseWriter, r *http.Request, query *dns.Msg) {
|
||||
if query.Question[0].Name == testReverse {
|
||||
ptrReply(300, testName, otherName)(w, r, query)
|
||||
|
||||
return
|
||||
}
|
||||
|
||||
soaReply(dns.RcodeNameError, 3600, 300)(w, r, query)
|
||||
})
|
||||
|
||||
cache, _ := newTestCache(t, db, resolver)
|
||||
|
||||
err := cache.Start(t.Context())
|
||||
if err != nil {
|
||||
t.Fatalf("Start: %v", err)
|
||||
}
|
||||
|
||||
lookupOK(t, cache, testAddress)
|
||||
lookupOK(t, cache, otherAddress)
|
||||
cache.Stop(t.Context())
|
||||
|
||||
err = db.Close()
|
||||
if err != nil {
|
||||
t.Fatalf("closing the database: %v", err)
|
||||
}
|
||||
|
||||
restarted, _ := newTestCache(t, openTestDB(t, dir), resolver)
|
||||
|
||||
err = restarted.Start(t.Context())
|
||||
if err != nil {
|
||||
t.Fatalf("Start after the restart: %v", err)
|
||||
}
|
||||
|
||||
defer restarted.Stop(t.Context())
|
||||
|
||||
requestsBefore := resolver.requestCount()
|
||||
|
||||
names := lookupOK(t, restarted, testAddress)
|
||||
if !slices.Equal(names, []string{testName, otherName}) {
|
||||
t.Errorf("names after restart %v", names)
|
||||
}
|
||||
|
||||
if names := lookupOK(t, restarted, otherAddress); len(names) != 0 {
|
||||
t.Errorf("names of the address with none after restart: %v", names)
|
||||
}
|
||||
|
||||
if got := resolver.requestCount(); got != requestsBefore {
|
||||
t.Errorf("%d requests after the restart, want none",
|
||||
got-requestsBefore)
|
||||
}
|
||||
}
|
||||
|
||||
// TestSweepDeletesExpiredRows also checks that a restart does not load a row
|
||||
// that has expired since it was written.
|
||||
func TestSweepDeletesExpiredRows(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
db := openTestDB(t, t.TempDir())
|
||||
resolver := newFakeResolver(t,
|
||||
func(w http.ResponseWriter, r *http.Request, query *dns.Msg) {
|
||||
ttl := uint32(300)
|
||||
if query.Question[0].Name == testReverse {
|
||||
ttl = 60
|
||||
}
|
||||
|
||||
ptrReply(ttl, testName)(w, r, query)
|
||||
})
|
||||
cache, clock := newTestCache(t, db, resolver)
|
||||
|
||||
lookupOK(t, cache, testAddress)
|
||||
lookupOK(t, cache, otherAddress)
|
||||
cache.flush(t.Context())
|
||||
|
||||
if got := countRows(t, db); got != 2 {
|
||||
t.Fatalf("%d rows before the sweep, want 2", got)
|
||||
}
|
||||
|
||||
clock.advance(90 * time.Second)
|
||||
|
||||
loaded, loadedClock := newTestCache(t, db, resolver)
|
||||
loadedClock.advance(90 * time.Second)
|
||||
|
||||
err := loaded.load(t.Context())
|
||||
if err != nil {
|
||||
t.Fatalf("load: %v", err)
|
||||
}
|
||||
|
||||
if len(loaded.entries) != 1 {
|
||||
t.Errorf("loaded %d entries, want the 1 not expired", len(loaded.entries))
|
||||
}
|
||||
|
||||
cache.sweep(t.Context())
|
||||
|
||||
if got := countRows(t, db); got != 1 {
|
||||
t.Errorf("%d rows after the sweep, want 1", got)
|
||||
}
|
||||
|
||||
if len(cache.entries) != 1 {
|
||||
t.Errorf("%d entries in memory after the sweep, want 1", len(cache.entries))
|
||||
}
|
||||
}
|
||||
|
||||
// TestFlushKeepsAnswersItCouldNotWrite makes one write fail by hiding the
|
||||
// table. The next write, once the table is back, stores the answer.
|
||||
func TestFlushKeepsAnswersItCouldNotWrite(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
db := openTestDB(t, t.TempDir())
|
||||
resolver := newFakeResolver(t, ptrReply(300, testName))
|
||||
cache, _ := newTestCache(t, db, resolver)
|
||||
|
||||
lookupOK(t, cache, testAddress)
|
||||
|
||||
_, err := db.ExecContext(t.Context(), "ALTER TABLE hostnames RENAME TO hidden")
|
||||
if err != nil {
|
||||
t.Fatalf("renaming the table: %v", err)
|
||||
}
|
||||
|
||||
cache.flush(t.Context())
|
||||
|
||||
_, err = db.ExecContext(t.Context(), "ALTER TABLE hidden RENAME TO hostnames")
|
||||
if err != nil {
|
||||
t.Fatalf("renaming the table back: %v", err)
|
||||
}
|
||||
|
||||
if got := countRows(t, db); got != 0 {
|
||||
t.Fatalf("%d rows after the failed write, want 0", got)
|
||||
}
|
||||
|
||||
cache.flush(t.Context())
|
||||
|
||||
if got := countRows(t, db); got != 1 {
|
||||
t.Errorf("%d rows after the second write, want 1", got)
|
||||
}
|
||||
}
|
||||
|
||||
// TestNewAddressIsNotKeptAtTheLimit fills a cache that holds one address. A
|
||||
// lookup of a second address still returns its answer, and the cache keeps
|
||||
// nothing for it until the sweep has deleted the first address's expired
|
||||
// answer.
|
||||
func TestNewAddressIsNotKeptAtTheLimit(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
db := openTestDB(t, t.TempDir())
|
||||
resolver := newFakeResolver(t, ptrReply(300, testName))
|
||||
cache, clock := newTestCache(t, db, resolver)
|
||||
cache.maxEntries = 1
|
||||
|
||||
lookupOK(t, cache, testAddress)
|
||||
|
||||
names := lookupOK(t, cache, otherAddress)
|
||||
if !slices.Equal(names, []string{testName}) {
|
||||
t.Errorf("names %v at the limit, want %s", names, testName)
|
||||
}
|
||||
|
||||
_, inEntries := cache.entries[otherAddress]
|
||||
_, inPending := cache.pending[otherAddress]
|
||||
|
||||
if inEntries || inPending {
|
||||
t.Errorf("a new address was kept at the limit: in entries %t, in pending %t",
|
||||
inEntries, inPending)
|
||||
}
|
||||
|
||||
cache.flush(t.Context())
|
||||
|
||||
if got := countRows(t, db); got != 1 {
|
||||
t.Errorf("%d rows at the limit, want 1", got)
|
||||
}
|
||||
|
||||
clock.advance(300 * time.Second)
|
||||
cache.sweep(t.Context())
|
||||
lookupOK(t, cache, otherAddress)
|
||||
|
||||
_, inEntries = cache.entries[otherAddress]
|
||||
if !inEntries {
|
||||
t.Error("a new address was not kept after the sweep made room")
|
||||
}
|
||||
}
|
||||
|
||||
// TestHeldAddressIsReplacedAtTheLimit lets the answer of the only address a
|
||||
// cache holds expire. The new answer replaces it in memory and is queued for
|
||||
// cache.db.
|
||||
func TestHeldAddressIsReplacedAtTheLimit(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
resolver := newFakeResolver(t, ptrReply(300, testName))
|
||||
cache, clock := newTestCache(t, openTestDB(t, t.TempDir()), resolver)
|
||||
cache.maxEntries = 1
|
||||
|
||||
lookupOK(t, cache, testAddress)
|
||||
clock.advance(300 * time.Second)
|
||||
resolver.setReply(ptrReply(300, otherName))
|
||||
lookupOK(t, cache, testAddress)
|
||||
|
||||
want := []string{otherName}
|
||||
|
||||
if got := cache.entries[testAddress].names; !slices.Equal(got, want) {
|
||||
t.Errorf("entries holds %v, want %v", got, want)
|
||||
}
|
||||
|
||||
if got := cache.pending[testAddress].names; !slices.Equal(got, want) {
|
||||
t.Errorf("pending holds %v, want %v", got, want)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,200 @@
|
||||
package ptr
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"errors"
|
||||
"fmt"
|
||||
"io"
|
||||
"math"
|
||||
"net/http"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"github.com/miekg/dns"
|
||||
)
|
||||
|
||||
const (
|
||||
// LookupTimeout limits one address's lookup, every resolver included.
|
||||
LookupTimeout = 2 * time.Second
|
||||
|
||||
// minTTL and maxTTL limit how long PTR names are kept.
|
||||
minTTL = time.Minute
|
||||
maxTTL = 24 * time.Hour
|
||||
|
||||
// maxNegativeTTL limits how long an address with no PTR names is
|
||||
// remembered as having none.
|
||||
maxNegativeTTL = time.Hour
|
||||
|
||||
// dnsMessageType is the content type of RFC 8484 requests and answers.
|
||||
dnsMessageType = "application/dns-message"
|
||||
|
||||
// maxMessageBytes is the largest possible DNS message.
|
||||
maxMessageBytes = 65535
|
||||
)
|
||||
|
||||
var (
|
||||
errBadStatus = errors.New("unexpected HTTP status")
|
||||
errBadResponse = errors.New("resolver reported an error")
|
||||
)
|
||||
|
||||
// Resolvers returns the URLs of the public DNS-over-HTTPS resolvers, in the
|
||||
// order they are tried.
|
||||
func Resolvers() []string {
|
||||
return []string{
|
||||
"https://dns.google/dns-query",
|
||||
"https://cloudflare-dns.com/dns-query",
|
||||
"https://unfiltered.adguard-dns.com/dns-query",
|
||||
"https://dns.quad9.net/dns-query",
|
||||
}
|
||||
}
|
||||
|
||||
// answer is a resolver's answer for one address.
|
||||
type answer struct {
|
||||
// names are the address's PTR names, nil when it has none.
|
||||
names []string
|
||||
|
||||
// ttl is how long the answer may be kept. Zero means it is not kept.
|
||||
ttl time.Duration
|
||||
}
|
||||
|
||||
// lookup asks the resolvers in turn for the PTR names of the reverse name
|
||||
// until one answers. It fails when none has answered within the time limit.
|
||||
func (c *Cache) lookup(ctx context.Context, reverseName string) (answer, error) {
|
||||
ctx, cancel := context.WithTimeout(ctx, c.timeLimit)
|
||||
defer cancel()
|
||||
|
||||
query := new(dns.Msg).SetQuestion(reverseName, dns.TypePTR)
|
||||
query.Id = 0 // RFC 8484 asks for 0, so that equal queries can be cached.
|
||||
|
||||
wire, err := query.Pack()
|
||||
if err != nil {
|
||||
return answer{}, fmt.Errorf("packing the query: %w", err)
|
||||
}
|
||||
|
||||
var failures []error
|
||||
|
||||
for i, resolver := range c.resolvers {
|
||||
// Each resolver gets an equal share of the time left, so that one
|
||||
// that never answers leaves time for the next.
|
||||
deadline, _ := ctx.Deadline()
|
||||
share := time.Until(deadline) / time.Duration(len(c.resolvers)-i)
|
||||
|
||||
result, err := c.ask(ctx, resolver, wire, share)
|
||||
if err == nil {
|
||||
return result, nil
|
||||
}
|
||||
|
||||
failures = append(failures, fmt.Errorf("%s: %w", resolver, err))
|
||||
}
|
||||
|
||||
return answer{}, errors.Join(failures...)
|
||||
}
|
||||
|
||||
// ask sends the query to one resolver and waits at most timeout for its answer.
|
||||
func (c *Cache) ask(
|
||||
ctx context.Context, resolver string, query []byte, timeout time.Duration,
|
||||
) (answer, error) {
|
||||
ctx, cancel := context.WithTimeout(ctx, timeout)
|
||||
defer cancel()
|
||||
|
||||
request, err := http.NewRequestWithContext(
|
||||
ctx, http.MethodPost, resolver, bytes.NewReader(query))
|
||||
if err != nil {
|
||||
return answer{}, err
|
||||
}
|
||||
|
||||
request.Header.Set("Content-Type", dnsMessageType)
|
||||
request.Header.Set("Accept", dnsMessageType)
|
||||
|
||||
response, err := c.client.Do(request)
|
||||
if err != nil {
|
||||
return answer{}, err
|
||||
}
|
||||
|
||||
defer func() { _ = response.Body.Close() }()
|
||||
|
||||
if response.StatusCode != http.StatusOK {
|
||||
return answer{}, fmt.Errorf("%w: %d", errBadStatus, response.StatusCode)
|
||||
}
|
||||
|
||||
body, err := io.ReadAll(io.LimitReader(response.Body, maxMessageBytes))
|
||||
if err != nil {
|
||||
return answer{}, err
|
||||
}
|
||||
|
||||
var reply dns.Msg
|
||||
|
||||
err = reply.Unpack(body)
|
||||
if err != nil {
|
||||
return answer{}, err
|
||||
}
|
||||
|
||||
return answerFrom(&reply)
|
||||
}
|
||||
|
||||
// answerFrom reads a resolver's reply. A name that does not exist and a name
|
||||
// with no PTR records are answers. Every other response code is a failure.
|
||||
func answerFrom(reply *dns.Msg) (answer, error) {
|
||||
if reply.Rcode != dns.RcodeSuccess && reply.Rcode != dns.RcodeNameError {
|
||||
return answer{}, fmt.Errorf("%w: %s",
|
||||
errBadResponse, dns.RcodeToString[reply.Rcode])
|
||||
}
|
||||
|
||||
var names []string
|
||||
|
||||
seconds := uint32(math.MaxUint32)
|
||||
|
||||
for _, record := range reply.Answer {
|
||||
// The chain to a PTR record can hold CNAME records, and the answer
|
||||
// is good only as long as every record in it.
|
||||
seconds = min(seconds, record.Header().Ttl)
|
||||
|
||||
ptr, ok := record.(*dns.PTR)
|
||||
if ok && isHostName(ptr.Ptr) {
|
||||
names = append(names, ptr.Ptr)
|
||||
}
|
||||
}
|
||||
|
||||
if len(names) == 0 {
|
||||
return answer{ttl: negativeTTL(reply.Ns)}, nil
|
||||
}
|
||||
|
||||
ttl := time.Duration(seconds) * time.Second
|
||||
|
||||
return answer{names: names, ttl: min(max(ttl, minTTL), maxTTL)}, nil
|
||||
}
|
||||
|
||||
// negativeTTL returns how long a reply with no PTR names may be kept: the
|
||||
// smaller of the SOA record's own TTL and its minimum field (RFC 2308), at most
|
||||
// maxNegativeTTL. Without a SOA record it is zero.
|
||||
func negativeTTL(authority []dns.RR) time.Duration {
|
||||
for _, record := range authority {
|
||||
soa, ok := record.(*dns.SOA)
|
||||
if ok {
|
||||
ttl := time.Duration(min(soa.Hdr.Ttl, soa.Minttl)) * time.Second
|
||||
|
||||
return min(ttl, maxNegativeTTL)
|
||||
}
|
||||
}
|
||||
|
||||
return 0
|
||||
}
|
||||
|
||||
// isHostName reports whether name is a host name. A PTR record can hold any
|
||||
// bytes, and the names go into web pages, so anything else is dropped, as
|
||||
// net.Resolver.LookupAddr drops it. The root name is not a host name.
|
||||
func isHostName(name string) bool {
|
||||
_, ok := dns.IsDomainName(name)
|
||||
|
||||
return ok && name != "." && !strings.ContainsFunc(name, notInHostName)
|
||||
}
|
||||
|
||||
// notInHostName reports whether r cannot be part of a host name.
|
||||
func notInHostName(r rune) bool {
|
||||
return !strings.ContainsRune(hostNameCharacters, r)
|
||||
}
|
||||
|
||||
// hostNameCharacters are the characters of a host name, in presentation format.
|
||||
const hostNameCharacters = "abcdefghijklmnopqrstuvwxyz" +
|
||||
"ABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789-_."
|
||||
+101
-2
@@ -9,17 +9,21 @@ package routewatch
|
||||
|
||||
import (
|
||||
"context"
|
||||
"database/sql"
|
||||
"errors"
|
||||
"fmt"
|
||||
"path/filepath"
|
||||
"sync"
|
||||
"time"
|
||||
|
||||
"git.eeqj.de/sneak/routewatch/internal/cachedb"
|
||||
"git.eeqj.de/sneak/routewatch/internal/config"
|
||||
"git.eeqj.de/sneak/routewatch/internal/database"
|
||||
"git.eeqj.de/sneak/routewatch/internal/geo"
|
||||
"git.eeqj.de/sneak/routewatch/internal/iplist"
|
||||
"git.eeqj.de/sneak/routewatch/internal/logger"
|
||||
"git.eeqj.de/sneak/routewatch/internal/metrics"
|
||||
"git.eeqj.de/sneak/routewatch/internal/ptr"
|
||||
"git.eeqj.de/sneak/routewatch/internal/routetable"
|
||||
"git.eeqj.de/sneak/routewatch/internal/server"
|
||||
"git.eeqj.de/sneak/routewatch/internal/streamer"
|
||||
@@ -62,6 +66,9 @@ type RouteWatch struct {
|
||||
asnFetcher *ASNFetcher
|
||||
dbMaintainer *DBMaintainer
|
||||
ipLists *iplist.IPLists
|
||||
cacheDB *sql.DB
|
||||
ptrCache *ptr.Cache
|
||||
geoLocator *geo.Locator
|
||||
}
|
||||
|
||||
// ipListsDirName is the directory in the state directory that holds the IP
|
||||
@@ -150,6 +157,16 @@ func (rw *RouteWatch) Run(ctx context.Context) error {
|
||||
rw.ipLists = ipLists
|
||||
rw.server.SetIPLists(ipLists)
|
||||
|
||||
err = rw.startPTRCache(ctx)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
err = rw.startGeoLocator(ctx)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
// Start HTTP server
|
||||
err = rw.server.Start(ctx)
|
||||
if err != nil {
|
||||
@@ -168,8 +185,9 @@ func (rw *RouteWatch) Run(ctx context.Context) error {
|
||||
|
||||
// Shutdown stops all services in order: the streamer, the handlers, which
|
||||
// write what they hold, the WHOIS fetcher, the database maintainer, the
|
||||
// IP list downloads and the HTTP server, and last the database, which it
|
||||
// checkpoints and closes. Each stage waits at most until ctx ends, so
|
||||
// IP list downloads, the HTTP server, the geolocation cache and the PTR cache,
|
||||
// and last the database, which it checkpoints and closes. Each stage waits at
|
||||
// most until ctx ends, so
|
||||
// Shutdown returns soon after: a handler logs what it could not write by
|
||||
// then, and the HTTP server closes the connections of requests still
|
||||
// running.
|
||||
@@ -229,6 +247,12 @@ func (rw *RouteWatch) Shutdown(ctx context.Context) {
|
||||
"error", err)
|
||||
}
|
||||
|
||||
if rw.geoLocator != nil {
|
||||
rw.geoLocator.Stop(ctx)
|
||||
}
|
||||
|
||||
rw.stopPTRCache(ctx)
|
||||
|
||||
rw.closeDatabase(ctx)
|
||||
|
||||
// Log final metrics
|
||||
@@ -242,6 +266,81 @@ func (rw *RouteWatch) Shutdown(ctx context.Context) {
|
||||
)
|
||||
}
|
||||
|
||||
// startPTRCache opens cache.db, loads the PTR cache from it and gives the
|
||||
// server the cache. The HTTP handlers read it without a lock, so it is set
|
||||
// before they run.
|
||||
func (rw *RouteWatch) startPTRCache(ctx context.Context) error {
|
||||
cacheDB, err := cachedb.Open(ctx, rw.config.StateDir)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
ptrCache := ptr.New(ptr.Params{
|
||||
DB: cacheDB,
|
||||
Resolvers: ptr.Resolvers(),
|
||||
Logger: rw.logger.Logger,
|
||||
})
|
||||
|
||||
err = ptrCache.Start(ctx)
|
||||
if err != nil {
|
||||
_ = cacheDB.Close()
|
||||
|
||||
return err
|
||||
}
|
||||
|
||||
rw.cacheDB = cacheDB
|
||||
rw.ptrCache = ptrCache
|
||||
rw.server.SetPTRResolver(ptrCache)
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
// startGeoLocator loads the geolocation cache from cache.db, which
|
||||
// startPTRCache has opened, and gives the server the locator. The HTTP handlers
|
||||
// read it without a lock, so it is set before they run.
|
||||
func (rw *RouteWatch) startGeoLocator(ctx context.Context) error {
|
||||
locator, err := geo.New(geo.Params{
|
||||
DB: rw.cacheDB,
|
||||
Dir: rw.config.StateDir,
|
||||
GeoJSURL: geo.GeoJSEndpoint,
|
||||
DatabaseURLFormat: geo.DatabaseDownloadFormat,
|
||||
Logger: rw.logger.Logger,
|
||||
})
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
err = locator.Start(ctx)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
rw.geoLocator = locator
|
||||
rw.server.SetGeoLocator(locator)
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
// stopPTRCache stops the PTR cache, which writes the answers cache.db does not
|
||||
// have yet, and closes cache.db, unless ctx has ended and a write may still be
|
||||
// running.
|
||||
func (rw *RouteWatch) stopPTRCache(ctx context.Context) {
|
||||
if rw.ptrCache == nil {
|
||||
return
|
||||
}
|
||||
|
||||
rw.ptrCache.Stop(ctx)
|
||||
|
||||
if ctx.Err() != nil {
|
||||
return
|
||||
}
|
||||
|
||||
err := rw.cacheDB.Close()
|
||||
if err != nil {
|
||||
rw.logger.Error("Failed to close the cache database", "error", err)
|
||||
}
|
||||
}
|
||||
|
||||
// closeDatabase checkpoints and closes the database, unless ctx has ended: a
|
||||
// write may then still be running, and both would wait for it. SQLite rolls
|
||||
// back a write cut off by the exit when the database next opens.
|
||||
|
||||
@@ -8,6 +8,8 @@ import (
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"net/netip"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"slices"
|
||||
"strconv"
|
||||
"strings"
|
||||
@@ -783,3 +785,74 @@ func TestShutdownCheckpointsAndClosesTheDatabaseLast(t *testing.T) {
|
||||
t.Errorf("Shutdown made the calls %q, want %q", store.calls, want)
|
||||
}
|
||||
}
|
||||
|
||||
// TestPTRCacheAnswersTheIPLookup starts the PTR cache and the HTTP server as
|
||||
// Run does, and looks up a reserved address, which asks no resolver. Without
|
||||
// the PTR cache the server has no PTR resolver, and every IP lookup answers
|
||||
// 500.
|
||||
func TestPTRCacheAnswersTheIPLookup(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
store := newMockStore()
|
||||
log := logger.New()
|
||||
port := freePort(t)
|
||||
cfg := &config.Config{Port: port, StateDir: t.TempDir()}
|
||||
s := streamer.New(log, metrics.New())
|
||||
|
||||
// A started server reads its routing table in the background, so it
|
||||
// needs one.
|
||||
rw := New(Dependencies{
|
||||
DB: store,
|
||||
Streamer: s,
|
||||
Server: server.New(store, routetable.New(routetable.Params{}), s, log,
|
||||
cfg),
|
||||
Logger: log,
|
||||
Config: cfg,
|
||||
})
|
||||
|
||||
err := rw.startPTRCache(t.Context())
|
||||
if err != nil {
|
||||
t.Fatalf("failed to start the PTR cache: %v", err)
|
||||
}
|
||||
|
||||
defer rw.stopPTRCache(t.Context())
|
||||
|
||||
err = rw.server.Start(t.Context())
|
||||
if err != nil {
|
||||
t.Fatalf("failed to start the HTTP server: %v", err)
|
||||
}
|
||||
|
||||
defer func() { _ = rw.server.Stop(t.Context()) }()
|
||||
|
||||
req, err := http.NewRequestWithContext(t.Context(), http.MethodGet,
|
||||
"http://"+net.JoinHostPort("127.0.0.1", strconv.Itoa(port))+
|
||||
"/api/v1/ip/10.0.0.1", nil)
|
||||
if err != nil {
|
||||
t.Fatalf("failed to build the request: %v", err)
|
||||
}
|
||||
|
||||
resp, err := http.DefaultClient.Do(req)
|
||||
if err != nil {
|
||||
t.Fatalf("the IP lookup failed: %v", err)
|
||||
}
|
||||
|
||||
defer func() { _ = resp.Body.Close() }()
|
||||
|
||||
body, err := io.ReadAll(resp.Body)
|
||||
if err != nil {
|
||||
t.Fatalf("failed to read the answer: %v", err)
|
||||
}
|
||||
|
||||
if resp.StatusCode != http.StatusOK {
|
||||
t.Fatalf("status %d, want %d; body %s", resp.StatusCode, http.StatusOK, body)
|
||||
}
|
||||
|
||||
if !strings.Contains(string(body), `"ptr":[]`) {
|
||||
t.Errorf("answer %s, want an empty ptr", body)
|
||||
}
|
||||
|
||||
_, err = os.Stat(filepath.Join(cfg.StateDir, "cache.db"))
|
||||
if err != nil {
|
||||
t.Errorf("cache.db is not in the state directory: %v", err)
|
||||
}
|
||||
}
|
||||
+101
-25
@@ -17,6 +17,7 @@ import (
|
||||
|
||||
"git.eeqj.de/sneak/routewatch/internal/bogon"
|
||||
"git.eeqj.de/sneak/routewatch/internal/database"
|
||||
"git.eeqj.de/sneak/routewatch/internal/geo"
|
||||
"git.eeqj.de/sneak/routewatch/internal/iplist"
|
||||
"git.eeqj.de/sneak/routewatch/internal/metrics"
|
||||
"git.eeqj.de/sneak/routewatch/internal/ristypes"
|
||||
@@ -39,12 +40,9 @@ const (
|
||||
const (
|
||||
// maxLookupTargets is the most addresses and host names an IP lookup may
|
||||
// list, and the most addresses it may have once the names are resolved:
|
||||
// each address costs two database reads and a PTR lookup.
|
||||
// each address costs two database reads and a PTR lookup. Ten PTR lookups
|
||||
// that run to ptr.LookupTimeout fit in the request's 30-second limit.
|
||||
maxLookupTargets = 10
|
||||
|
||||
// ptrLookupTimeout is each PTR lookup's own time limit, so that ten
|
||||
// addresses whose PTR lookups never answer fit in the 30-second limit.
|
||||
ptrLookupTimeout = 2 * time.Second
|
||||
)
|
||||
|
||||
// The Cache-Control header of an IP lookup answer, such as
|
||||
@@ -767,9 +765,9 @@ type IPLookupResponse struct {
|
||||
}
|
||||
|
||||
// IPLookupResult is the answer for one address: its PTR names, its flags, its
|
||||
// live route and that route's AS, and the IP lists that contain it. Every
|
||||
// result has every field. The fields of IPInfo are null when the address has
|
||||
// no live route.
|
||||
// live route and that route's AS, the IP lists that contain it, and where it
|
||||
// is. Every result has every field. The fields of IPInfo are null when the
|
||||
// address has no live route.
|
||||
//
|
||||
//nolint:tagliatelle // snake_case keys, as the Go style guide asks
|
||||
type IPLookupResult struct {
|
||||
@@ -781,8 +779,15 @@ type IPLookupResult struct {
|
||||
|
||||
Flags IPLookupFlags `json:"flags"`
|
||||
|
||||
// VPNProviders names the VPN lists the address is on. It is empty, never
|
||||
// nil, when there are none.
|
||||
VPNProviders []string `json:"vpn_providers"`
|
||||
|
||||
// IPLists has one entry for each IP list the address is on.
|
||||
IPLists []iplist.Listing `json:"ip_lists"`
|
||||
|
||||
// Geo is where the address is. It is nil when no source answers.
|
||||
Geo *geo.Location `json:"geo"`
|
||||
}
|
||||
|
||||
// IPLookupFlags are the yes-or-no facts about an address. Every result has
|
||||
@@ -793,6 +798,12 @@ type IPLookupFlags struct {
|
||||
// IsTor says that the address is on the Tor exit list.
|
||||
IsTor bool `json:"is_tor"`
|
||||
|
||||
// IsVPN says that the address is on a list of VPN servers.
|
||||
IsVPN bool `json:"is_vpn"`
|
||||
|
||||
// IsRelay says that the address is on the iCloud Private Relay list.
|
||||
IsRelay bool `json:"is_relay"`
|
||||
|
||||
// IsBogon says that the address is reserved, as package bogon tells.
|
||||
IsBogon bool `json:"is_bogon"`
|
||||
}
|
||||
@@ -835,6 +846,8 @@ func (s *Server) handleIPInfo() http.HandlerFunc {
|
||||
return
|
||||
}
|
||||
|
||||
locations := s.geoLocations(ctx, ipsToLookup)
|
||||
|
||||
// Track ASNs that need WHOIS refresh
|
||||
refreshASNs := make(map[int]bool)
|
||||
|
||||
@@ -870,7 +883,9 @@ func (s *Server) handleIPInfo() http.HandlerFunc {
|
||||
return
|
||||
}
|
||||
|
||||
response.Results = append(response.Results, s.ipLookupResult(ip, ptrs, ipInfo))
|
||||
result := s.ipLookupResult(ip, ptrs, ipInfo)
|
||||
result.Geo = locations[ip]
|
||||
response.Results = append(response.Results, result)
|
||||
|
||||
if ipInfo.NeedsWHOISRefresh {
|
||||
refreshASNs[*ipInfo.ASN] = true
|
||||
@@ -977,14 +992,30 @@ func (s *Server) ipLookupResult(ip string, ptrs []string, info IPInfo) IPLookupR
|
||||
listings := s.ipListListings(ip)
|
||||
|
||||
return IPLookupResult{
|
||||
IP: ip,
|
||||
PTR: ptrs,
|
||||
IPInfo: info,
|
||||
Flags: ipLookupFlags(ip, listings),
|
||||
IPLists: listings,
|
||||
IP: ip,
|
||||
PTR: ptrs,
|
||||
IPInfo: info,
|
||||
Flags: ipLookupFlags(ip, listings),
|
||||
VPNProviders: vpnProviders(listings),
|
||||
IPLists: listings,
|
||||
}
|
||||
}
|
||||
|
||||
// vpnProviders returns the names of the VPN lists among listings. It returns
|
||||
// an empty list, never nil, so that an answer's vpn_providers is [] rather
|
||||
// than null.
|
||||
func vpnProviders(listings []iplist.Listing) []string {
|
||||
names := []string{}
|
||||
|
||||
for _, listing := range listings {
|
||||
if listing.VPN {
|
||||
names = append(names, listing.Name)
|
||||
}
|
||||
}
|
||||
|
||||
return names
|
||||
}
|
||||
|
||||
// ipListListings returns the IP lists that contain ip. It returns an empty
|
||||
// list, never nil, so that an answer's ip_lists is [] rather than null.
|
||||
func (s *Server) ipListListings(ip string) []iplist.Listing {
|
||||
@@ -1002,20 +1033,63 @@ func ipLookupFlags(ip string, listings []iplist.Listing) IPLookupFlags {
|
||||
IsTor: slices.ContainsFunc(listings, func(l iplist.Listing) bool {
|
||||
return l.Name == iplist.TorExitsList
|
||||
}),
|
||||
IsVPN: slices.ContainsFunc(listings, func(l iplist.Listing) bool {
|
||||
return l.VPN
|
||||
}),
|
||||
IsRelay: slices.ContainsFunc(listings, func(l iplist.Listing) bool {
|
||||
return l.Name == iplist.ICloudPrivateRelayList
|
||||
}),
|
||||
IsBogon: isBogon(ip),
|
||||
}
|
||||
}
|
||||
|
||||
// isBogon says whether the address ip is reserved. An IPv4 address written in
|
||||
// IPv6 form counts as the IPv4 address it holds.
|
||||
// geoLocations returns where ips are, by address. An address that does not
|
||||
// parse has no entry. Nor has a reserved one, which is sent to no outside
|
||||
// source and has no location.
|
||||
func (s *Server) geoLocations(
|
||||
ctx context.Context, ips []string,
|
||||
) map[string]*geo.Location {
|
||||
if s.geoLocator == nil {
|
||||
return nil
|
||||
}
|
||||
|
||||
var (
|
||||
kept []string
|
||||
addrs []netip.Addr
|
||||
)
|
||||
|
||||
for _, ip := range ips {
|
||||
addr, err := netip.ParseAddr(ip)
|
||||
if err != nil || isBogon(ip) {
|
||||
continue
|
||||
}
|
||||
|
||||
kept = append(kept, ip)
|
||||
addrs = append(addrs, addr)
|
||||
}
|
||||
|
||||
if len(addrs) == 0 {
|
||||
return nil
|
||||
}
|
||||
|
||||
answers := s.geoLocator.Lookup(ctx, addrs)
|
||||
locations := make(map[string]*geo.Location, len(kept))
|
||||
|
||||
for i, ip := range kept {
|
||||
locations[ip] = answers[i]
|
||||
}
|
||||
|
||||
return locations
|
||||
}
|
||||
|
||||
// isBogon says whether the address ip is reserved, as package bogon tells.
|
||||
func isBogon(ip string) bool {
|
||||
addr, err := netip.ParseAddr(ip)
|
||||
if err != nil {
|
||||
return false
|
||||
}
|
||||
|
||||
addr = addr.Unmap()
|
||||
_, reserved := bogon.Prefix(netip.PrefixFrom(addr, addr.BitLen()))
|
||||
_, reserved := bogon.Addr(addr)
|
||||
|
||||
return reserved
|
||||
}
|
||||
@@ -1194,15 +1268,17 @@ func publicAddresses(addrs []string) []string {
|
||||
}
|
||||
|
||||
// lookupPTRNames returns the PTR records of ip without their trailing dots. It
|
||||
// returns an empty list, never nil, when there are none, when the lookup fails
|
||||
// and when it runs past ptrLookupTimeout, so that an answer's ptr is [] rather
|
||||
// than null.
|
||||
// returns an empty list, never nil, when there are none and when the lookup
|
||||
// fails, so that an answer's ptr is [] rather than null.
|
||||
func (s *Server) lookupPTRNames(ctx context.Context, ip string) []string {
|
||||
ctx, cancel := context.WithTimeout(ctx, ptrLookupTimeout)
|
||||
defer cancel()
|
||||
ptrs, err := s.ptrResolver.LookupAddr(ctx, ip)
|
||||
if err != nil {
|
||||
s.logger.Warn("PTR lookup failed", "ip", ip, "error", err)
|
||||
|
||||
ptrs, err := s.resolver.LookupAddr(ctx, ip)
|
||||
if err != nil || ptrs == nil {
|
||||
return []string{}
|
||||
}
|
||||
|
||||
if ptrs == nil {
|
||||
return []string{}
|
||||
}
|
||||
|
||||
|
||||
@@ -15,15 +15,19 @@ import (
|
||||
"slices"
|
||||
"strconv"
|
||||
"strings"
|
||||
"sync"
|
||||
"sync/atomic"
|
||||
"testing"
|
||||
"testing/synctest"
|
||||
"time"
|
||||
|
||||
"git.eeqj.de/sneak/routewatch/internal/config"
|
||||
"git.eeqj.de/sneak/routewatch/internal/database"
|
||||
"git.eeqj.de/sneak/routewatch/internal/geo"
|
||||
"git.eeqj.de/sneak/routewatch/internal/iplist"
|
||||
"git.eeqj.de/sneak/routewatch/internal/logger"
|
||||
"git.eeqj.de/sneak/routewatch/internal/metrics"
|
||||
"git.eeqj.de/sneak/routewatch/internal/ptr"
|
||||
"git.eeqj.de/sneak/routewatch/internal/ristypes"
|
||||
"git.eeqj.de/sneak/routewatch/internal/routetable"
|
||||
"git.eeqj.de/sneak/routewatch/internal/streamer"
|
||||
@@ -173,6 +177,9 @@ type tableStats struct {
|
||||
// carried 203.0.113.0/24, the two with kept routes withdrew it, so its origin
|
||||
// keeps no route. The oldest and newest routes are those of the other two
|
||||
// prefixes.
|
||||
//
|
||||
// The requests run in a synctest bubble, whose clock is fake, so a stall on
|
||||
// the host does not use up the handlers' 4-second limit.
|
||||
func TestStatsHandlersAnswerFromMemory(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
@@ -218,7 +225,11 @@ func TestStatsHandlersAnswerFromMemory(t *testing.T) {
|
||||
|
||||
for name, handler := range handlers {
|
||||
rec := httptest.NewRecorder()
|
||||
handler(rec, httptest.NewRequestWithContext(t.Context(), http.MethodGet, "/", nil))
|
||||
|
||||
synctest.Test(t, func(t *testing.T) {
|
||||
handler(rec,
|
||||
httptest.NewRequestWithContext(t.Context(), http.MethodGet, "/", nil))
|
||||
})
|
||||
|
||||
if rec.Code != http.StatusOK {
|
||||
t.Errorf("%s: status %d, want %d; body %s",
|
||||
@@ -253,15 +264,20 @@ func TestStatsHandlersAnswerFromMemory(t *testing.T) {
|
||||
// TestStatsAnswerCarriesChurnPerMinute checks that the stream object of the
|
||||
// /api/v1/stats answer holds route_churn_per_min as a whole number, and no
|
||||
// longer route_churn_per_sec, the moving average it replaces
|
||||
// (https://git.eeqj.de/sneak/routewatch/issues/160).
|
||||
// (https://git.eeqj.de/sneak/routewatch/issues/160). The request runs in a
|
||||
// synctest bubble, whose clock is fake, so a stall on the host does not use up
|
||||
// the handler's 4-second limit.
|
||||
func TestStatsAnswerCarriesChurnPerMinute(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
s := newServerWithTestRoutes(t)
|
||||
|
||||
rec := httptest.NewRecorder()
|
||||
s.handleStats()(rec,
|
||||
httptest.NewRequestWithContext(t.Context(), http.MethodGet, statsPath, nil))
|
||||
|
||||
synctest.Test(t, func(t *testing.T) {
|
||||
s.handleStats()(rec,
|
||||
httptest.NewRequestWithContext(t.Context(), http.MethodGet, statsPath, nil))
|
||||
})
|
||||
|
||||
if rec.Code != http.StatusOK {
|
||||
t.Fatalf("status %d, want %d; body %s", rec.Code, http.StatusOK, rec.Body)
|
||||
@@ -309,18 +325,26 @@ func (heldBackHandler) HeldBackCount() int { return heldBackMess
|
||||
|
||||
// TestStatsAnswerGivesHowManyMessagesAHandlerHoldsBack registers a handler
|
||||
// that holds back messages. Its figures in /api/v1/stats, which the status
|
||||
// page shows, must give how many.
|
||||
// page shows, must give how many. The request runs in a synctest bubble, whose
|
||||
// clock is fake, so a stall on the host does not use up the handler's 4-second
|
||||
// limit.
|
||||
func TestStatsAnswerGivesHowManyMessagesAHandlerHoldsBack(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
s := newServerWithTestRoutes(t)
|
||||
s.streamer.RegisterHandler(heldBackHandler{})
|
||||
|
||||
var answer []byte
|
||||
|
||||
synctest.Test(t, func(t *testing.T) {
|
||||
answer = get(t, s, statsPath).Body.Bytes()
|
||||
})
|
||||
|
||||
var body struct {
|
||||
Data StatsResponse `json:"data"`
|
||||
}
|
||||
|
||||
err := json.Unmarshal(get(t, s, statsPath).Body.Bytes(), &body)
|
||||
err := json.Unmarshal(answer, &body)
|
||||
if err != nil {
|
||||
t.Fatalf("decoding the answer: %v", err)
|
||||
}
|
||||
@@ -364,7 +388,9 @@ type unitStats struct {
|
||||
// names. They used to be strings with the unit in them, such as "1h18m34s" or
|
||||
// "1.1 GB" (https://git.eeqj.de/sneak/routewatch/issues/199). The stream has
|
||||
// never connected, so the uptime is 0. The memory use in /api/v1/stats is the
|
||||
// same number of bytes as its gc.heap_alloc_bytes.
|
||||
// same number of bytes as its gc.heap_alloc_bytes. The requests run in a
|
||||
// synctest bubble, whose clock is fake, so a stall on the host does not use up
|
||||
// the handlers' 4-second limit.
|
||||
func TestStatsAnswersGiveUnitsInFieldNames(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
@@ -374,7 +400,11 @@ func TestStatsAnswersGiveUnitsInFieldNames(t *testing.T) {
|
||||
s.SetASNFetcher(fixedIntervalASNFetcher{interval: fetchInterval})
|
||||
|
||||
for _, path := range []string{statusJSONPath, statsPath} {
|
||||
answer := get(t, s, path).Body.Bytes()
|
||||
var answer []byte
|
||||
|
||||
synctest.Test(t, func(t *testing.T) {
|
||||
answer = get(t, s, path).Body.Bytes()
|
||||
})
|
||||
|
||||
var body struct {
|
||||
Data unitStats `json:"data"`
|
||||
@@ -415,82 +445,89 @@ func TestStatsAnswersGiveUnitsInFieldNames(t *testing.T) {
|
||||
// connection_duration_seconds of /api/v1/stats is null while the stream is not
|
||||
// connected, and a whole number of seconds once it is. It used to be the
|
||||
// string "disconnected", or a Go duration such as "1h18m34s"
|
||||
// (https://git.eeqj.de/sneak/routewatch/issues/199).
|
||||
// (https://git.eeqj.de/sneak/routewatch/issues/199). The test runs in a
|
||||
// synctest bubble, whose clock is fake, so a stall on the host does not use up
|
||||
// the handler's 4-second limit.
|
||||
func TestStatsAnswerGivesConnectionDurationInSeconds(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
s := newServerWithTestRoutes(t)
|
||||
|
||||
connectionDuration := func() json.RawMessage {
|
||||
answer := get(t, s, statsPath).Body.Bytes()
|
||||
synctest.Test(t, func(t *testing.T) {
|
||||
connectionDuration := func() json.RawMessage {
|
||||
answer := get(t, s, statsPath).Body.Bytes()
|
||||
|
||||
var body struct {
|
||||
Data map[string]json.RawMessage `json:"data"`
|
||||
var body struct {
|
||||
Data map[string]json.RawMessage `json:"data"`
|
||||
}
|
||||
|
||||
err := json.Unmarshal(answer, &body)
|
||||
if err != nil {
|
||||
t.Fatalf("decoding the answer: %v; answer %s", err, answer)
|
||||
}
|
||||
|
||||
duration, ok := body.Data["connection_duration_seconds"]
|
||||
if !ok {
|
||||
t.Fatalf("no connection_duration_seconds: %s", answer)
|
||||
}
|
||||
|
||||
return duration
|
||||
}
|
||||
|
||||
err := json.Unmarshal(answer, &body)
|
||||
if err != nil {
|
||||
t.Fatalf("decoding the answer: %v; answer %s", err, answer)
|
||||
disconnected := connectionDuration()
|
||||
if string(disconnected) != "null" {
|
||||
t.Errorf("disconnected: connection_duration_seconds = %s, want null",
|
||||
disconnected)
|
||||
}
|
||||
|
||||
duration, ok := body.Data["connection_duration_seconds"]
|
||||
if !ok {
|
||||
t.Fatalf("no connection_duration_seconds: %s", answer)
|
||||
s.streamer.GetMetricsTracker().SetConnected(true)
|
||||
|
||||
var seconds int64
|
||||
|
||||
connected := connectionDuration()
|
||||
|
||||
err := json.Unmarshal(connected, &seconds)
|
||||
if err != nil || seconds != 0 {
|
||||
t.Errorf("just connected: connection_duration_seconds = %s, want 0",
|
||||
connected)
|
||||
}
|
||||
|
||||
return duration
|
||||
}
|
||||
|
||||
disconnected := connectionDuration()
|
||||
if string(disconnected) != "null" {
|
||||
t.Errorf("disconnected: connection_duration_seconds = %s, want null",
|
||||
disconnected)
|
||||
}
|
||||
|
||||
s.streamer.GetMetricsTracker().SetConnected(true)
|
||||
|
||||
var seconds int64
|
||||
|
||||
connected := connectionDuration()
|
||||
|
||||
err := json.Unmarshal(connected, &seconds)
|
||||
if err != nil || seconds != 0 {
|
||||
t.Errorf("just connected: connection_duration_seconds = %s, want 0",
|
||||
connected)
|
||||
}
|
||||
})
|
||||
}
|
||||
|
||||
// TestStreamDurationsCountSeconds checks that the uptime and the connection
|
||||
// duration of a stream that connected 90 seconds ago are both 90. A stream
|
||||
// that has never connected has an uptime of 0, and one that is not connected
|
||||
// has no connection duration.
|
||||
// has no connection duration. The test runs in a synctest bubble, whose clock
|
||||
// is fake, so a stall on the host does not add to the 90 seconds.
|
||||
func TestStreamDurationsCountSeconds(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
const connectedFor = 90 * time.Second
|
||||
synctest.Test(t, func(t *testing.T) {
|
||||
const connectedFor = 90 * time.Second
|
||||
|
||||
connectedSince := time.Now().Add(-connectedFor)
|
||||
want := int64(connectedFor / time.Second)
|
||||
connectedSince := time.Now().Add(-connectedFor)
|
||||
want := int64(connectedFor / time.Second)
|
||||
|
||||
if got := uptimeSeconds(connectedSince); got != want {
|
||||
t.Errorf("uptimeSeconds = %d, want %d", got, want)
|
||||
}
|
||||
if got := uptimeSeconds(connectedSince); got != want {
|
||||
t.Errorf("uptimeSeconds = %d, want %d", got, want)
|
||||
}
|
||||
|
||||
if got := uptimeSeconds(time.Time{}); got != 0 {
|
||||
t.Errorf("uptimeSeconds before any connection = %d, want 0", got)
|
||||
}
|
||||
if got := uptimeSeconds(time.Time{}); got != 0 {
|
||||
t.Errorf("uptimeSeconds before any connection = %d, want 0", got)
|
||||
}
|
||||
|
||||
switch got := connectionDurationSeconds(true, connectedSince); {
|
||||
case got == nil:
|
||||
t.Errorf("connectionDurationSeconds = nil, want %d", want)
|
||||
case *got != want:
|
||||
t.Errorf("connectionDurationSeconds = %d, want %d", *got, want)
|
||||
}
|
||||
switch got := connectionDurationSeconds(true, connectedSince); {
|
||||
case got == nil:
|
||||
t.Errorf("connectionDurationSeconds = nil, want %d", want)
|
||||
case *got != want:
|
||||
t.Errorf("connectionDurationSeconds = %d, want %d", *got, want)
|
||||
}
|
||||
|
||||
if got := connectionDurationSeconds(false, connectedSince); got != nil {
|
||||
t.Errorf("connectionDurationSeconds while not connected = %d, want nil",
|
||||
*got)
|
||||
}
|
||||
if got := connectionDurationSeconds(false, connectedSince); got != nil {
|
||||
t.Errorf("connectionDurationSeconds while not connected = %d, want nil",
|
||||
*got)
|
||||
}
|
||||
})
|
||||
}
|
||||
|
||||
// newTestTable returns a routing table that holds a route for testPrefixV4,
|
||||
@@ -523,8 +560,11 @@ func newServerWithTable(t *testing.T, table *routetable.Table) *Server {
|
||||
|
||||
t.Cleanup(func() { _ = db.Close() })
|
||||
|
||||
return New(db, table, streamer.New(logger.New(), metrics.New()), logger.New(),
|
||||
s := New(db, table, streamer.New(logger.New(), metrics.New()), logger.New(),
|
||||
&config.Config{})
|
||||
useFakeResolver(s, &fakeResolver{})
|
||||
|
||||
return s
|
||||
}
|
||||
|
||||
// TestPrefixDetailHandlersParsePrefix checks that both prefix pages find a
|
||||
@@ -1134,7 +1174,9 @@ func (failingDB) GetPrefixFirstSeen(
|
||||
// with the driver's error text. It also sends a prefix that does not parse.
|
||||
// No answer may carry the driver's or the prefix parser's text. Before the fix
|
||||
// the answers sent it to the client
|
||||
// (https://git.eeqj.de/sneak/routewatch/issues/108).
|
||||
// (https://git.eeqj.de/sneak/routewatch/issues/108). The requests run in a
|
||||
// synctest bubble, whose clock is fake, so a stall on the host does not use up
|
||||
// the stats handlers' 4-second limit.
|
||||
func TestJSONRoutesDoNotSendErrorDetails(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
@@ -1155,8 +1197,11 @@ func TestJSONRoutesDoNotSendErrorDetails(t *testing.T) {
|
||||
|
||||
for _, tt := range tests {
|
||||
rec := httptest.NewRecorder()
|
||||
s.router.ServeHTTP(rec, httptest.NewRequestWithContext(
|
||||
t.Context(), http.MethodGet, tt.path, nil))
|
||||
|
||||
synctest.Test(t, func(t *testing.T) {
|
||||
s.router.ServeHTTP(rec, httptest.NewRequestWithContext(
|
||||
t.Context(), http.MethodGet, tt.path, nil))
|
||||
})
|
||||
|
||||
if rec.Code != tt.wantCode {
|
||||
t.Errorf("%s: status %d, want %d; body %s",
|
||||
@@ -1213,7 +1258,7 @@ func TestIPLookupAnswers500WhenDatabaseReadFails(t *testing.T) {
|
||||
|
||||
s := New(failingIPLookupDB{}, newTestTable(),
|
||||
streamer.New(logger.New(), metrics.New()), logger.New(), &config.Config{})
|
||||
s.resolver = &fakeResolver{}
|
||||
useFakeResolver(s, &fakeResolver{})
|
||||
|
||||
tests := []struct {
|
||||
path string
|
||||
@@ -1249,16 +1294,19 @@ func TestIPLookupAnswers500WhenDatabaseReadFails(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
// errNoSuchHost is what fakeResolver answers a lookup it has no answer for
|
||||
// with.
|
||||
// errNoSuchHost is what fakeResolver answers a host name lookup it has no
|
||||
// addresses for with.
|
||||
var errNoSuchHost = errors.New("no such host")
|
||||
|
||||
// fakeResolver answers a host name lookup with the addresses hosts lists for
|
||||
// the name, and a PTR lookup with no names or, when ptrNeverAnswers is set,
|
||||
// not at all: it waits until the lookup's context ends. hostLookups counts the
|
||||
// host name lookups.
|
||||
// the name. It answers a PTR lookup with the names ptrs lists for the address,
|
||||
// and with no names and no error, as the real one does, when ptrs lists none.
|
||||
// When ptrNeverAnswers is set, a PTR lookup fails after ptr.LookupTimeout, as
|
||||
// the real one does when no resolver answers. hostLookups counts the host name
|
||||
// lookups.
|
||||
type fakeResolver struct {
|
||||
hosts map[string][]string
|
||||
ptrs map[string][]string
|
||||
ptrNeverAnswers bool
|
||||
|
||||
hostLookups atomic.Int64
|
||||
@@ -1275,14 +1323,23 @@ func (r *fakeResolver) LookupHost(_ context.Context, host string) ([]string, err
|
||||
return addrs, nil
|
||||
}
|
||||
|
||||
func (r *fakeResolver) LookupAddr(ctx context.Context, _ string) ([]string, error) {
|
||||
func (r *fakeResolver) LookupAddr(ctx context.Context, addr string) ([]string, error) {
|
||||
if r.ptrNeverAnswers {
|
||||
ctx, cancel := context.WithTimeout(ctx, ptr.LookupTimeout)
|
||||
defer cancel()
|
||||
|
||||
<-ctx.Done()
|
||||
|
||||
return nil, ctx.Err()
|
||||
}
|
||||
|
||||
return nil, errNoSuchHost
|
||||
return r.ptrs[addr], nil
|
||||
}
|
||||
|
||||
// useFakeResolver makes r answer the server's host name and PTR lookups.
|
||||
func useFakeResolver(s *Server, r *fakeResolver) {
|
||||
s.resolver = r
|
||||
s.ptrResolver = r
|
||||
}
|
||||
|
||||
// countingIPLookupDB embeds database.Store (left nil) and answers the IP
|
||||
@@ -1353,7 +1410,7 @@ func TestIPLookupRefusesTooManyAddresses(t *testing.T) {
|
||||
s := New(db, newTestTable(), streamer.New(logger.New(), metrics.New()),
|
||||
logger.New(), &config.Config{})
|
||||
resolver := &fakeResolver{hosts: hosts}
|
||||
s.resolver = resolver
|
||||
useFakeResolver(s, resolver)
|
||||
|
||||
rec := httptest.NewRecorder()
|
||||
s.router.ServeHTTP(rec, httptest.NewRequestWithContext(
|
||||
@@ -1388,17 +1445,21 @@ const addrInTestPrefixV4 = "192.0.2.1"
|
||||
|
||||
// TestIPLookupKeepsRouteWhenPTRLookupDoesNotAnswer looks up a routed address
|
||||
// with a resolver that never answers its PTR lookup. Once the PTR lookup's own
|
||||
// limit passes, the lookup answers 200 with the route. Before the fix the PTR
|
||||
// lookup ran until the request's time limit, and the lookup answered 504
|
||||
// without the route (https://git.eeqj.de/sneak/routewatch/issues/110).
|
||||
// limit passes, the lookup answers 200 with the route and an empty ptr. Before
|
||||
// the fix the PTR lookup ran until the request's time limit, and the lookup
|
||||
// answered 504 without the route
|
||||
// (https://git.eeqj.de/sneak/routewatch/issues/110).
|
||||
func TestIPLookupKeepsRouteWhenPTRLookupDoesNotAnswer(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
s := newServerWithTestRoutes(t)
|
||||
s.resolver = &fakeResolver{ptrNeverAnswers: true}
|
||||
useFakeResolver(s, &fakeResolver{ptrNeverAnswers: true})
|
||||
|
||||
// The request's time limit passes after the PTR lookup's.
|
||||
ctx, cancel := context.WithTimeout(t.Context(), 2*ptrLookupTimeout)
|
||||
// The request's time limit passes well after the PTR lookup's, so that a
|
||||
// stall on a busy host does not use up the difference.
|
||||
const requestTimeLimit = ptr.LookupTimeout + 15*time.Second
|
||||
|
||||
ctx, cancel := context.WithTimeout(t.Context(), requestTimeLimit)
|
||||
defer cancel()
|
||||
|
||||
rec := httptest.NewRecorder()
|
||||
@@ -1422,6 +1483,10 @@ func TestIPLookupKeepsRouteWhenPTRLookupDoesNotAnswer(t *testing.T) {
|
||||
*body.Data.Results[0].Netblock != testPrefixV4 {
|
||||
t.Errorf("answer %s, want one result in %s", rec.Body, testPrefixV4)
|
||||
}
|
||||
|
||||
if !strings.Contains(rec.Body.String(), `"ptr":[]`) {
|
||||
t.Errorf("answer %s, want an empty ptr list", rec.Body)
|
||||
}
|
||||
}
|
||||
|
||||
// addrInTestPrefixV6 is an address in testPrefixV6, which
|
||||
@@ -1441,8 +1506,24 @@ const listedUnroutedAddr = "203.0.113.1"
|
||||
// torExitAddr is the address that fakeIPLists lists on the Tor exit list.
|
||||
const torExitAddr = "203.0.113.2"
|
||||
|
||||
// fakeIPLists has two lists. The test list holds addrInTestPrefixV4 and
|
||||
// listedUnroutedAddr, and the Tor exit list holds torExitAddr.
|
||||
// relayAddr is the address that fakeIPLists lists on the iCloud Private Relay
|
||||
// list.
|
||||
const relayAddr = "203.0.113.5"
|
||||
|
||||
// The addresses that fakeIPLists lists on one and on two VPN lists, and the
|
||||
// names of those lists.
|
||||
const (
|
||||
oneVPNListAddr = "203.0.113.3"
|
||||
twoVPNListsAddr = "203.0.113.4"
|
||||
testVPNList = "test_vpn"
|
||||
otherTestVPN = "other_test_vpn"
|
||||
)
|
||||
|
||||
// fakeIPLists has five lists. The test list holds addrInTestPrefixV4,
|
||||
// listedUnroutedAddr and twoVPNListsAddr. The Tor exit list holds torExitAddr.
|
||||
// The iCloud Private Relay list holds relayAddr. The test VPN list holds
|
||||
// oneVPNListAddr and twoVPNListsAddr, and the other test VPN list holds
|
||||
// twoVPNListsAddr.
|
||||
type fakeIPLists struct{}
|
||||
|
||||
func testListing() iplist.Listing {
|
||||
@@ -1463,6 +1544,22 @@ func (fakeIPLists) Lookup(addr netip.Addr) []iplist.Listing {
|
||||
return []iplist.Listing{{Name: iplist.TorExitsList}}
|
||||
}
|
||||
|
||||
if addr == netip.MustParseAddr(relayAddr) {
|
||||
return []iplist.Listing{{Name: iplist.ICloudPrivateRelayList}}
|
||||
}
|
||||
|
||||
if addr == netip.MustParseAddr(oneVPNListAddr) {
|
||||
return []iplist.Listing{{Name: testVPNList, VPN: true}}
|
||||
}
|
||||
|
||||
if addr == netip.MustParseAddr(twoVPNListsAddr) {
|
||||
return []iplist.Listing{
|
||||
{Name: otherTestVPN, VPN: true},
|
||||
testListing(),
|
||||
{Name: testVPNList, VPN: true},
|
||||
}
|
||||
}
|
||||
|
||||
return []iplist.Listing{}
|
||||
}
|
||||
|
||||
@@ -1471,32 +1568,34 @@ func (fakeIPLists) ShortestInterval() time.Duration {
|
||||
}
|
||||
|
||||
// TestIPLookupFlags looks up addresses with and without a route. Every result
|
||||
// has both flags, under the JSON keys the API documents.
|
||||
// has every flag, under the JSON keys the API documents.
|
||||
func TestIPLookupFlags(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
s := newServerWithTestRoutes(t)
|
||||
s.resolver = &fakeResolver{}
|
||||
useFakeResolver(s, &fakeResolver{})
|
||||
s.SetIPLists(fakeIPLists{})
|
||||
|
||||
tests := []struct {
|
||||
name string
|
||||
addr string
|
||||
tor bool
|
||||
relay bool
|
||||
bogon bool
|
||||
}{
|
||||
{"on the Tor exit list", torExitAddr, true, true},
|
||||
{"on another list", listedUnroutedAddr, false, true},
|
||||
{"routed IPv4", addrInTestPrefixV4, false, true},
|
||||
{"routed IPv6", addrInTestPrefixV6, false, true},
|
||||
{"private IPv4", "10.0.0.1", false, true},
|
||||
{"loopback IPv4", "127.0.0.1", false, true},
|
||||
{"public IPv4", "1.1.1.1", false, false},
|
||||
{"private IPv6", "fd00::1", false, true},
|
||||
{"loopback IPv6", "::1", false, true},
|
||||
{"public IPv6", "2606:4700:4700::1111", false, false},
|
||||
{"public IPv4 in IPv6 form", "::ffff:1.1.1.1", false, false},
|
||||
{"private IPv4 in IPv6 form", "::ffff:10.0.0.1", false, true},
|
||||
{"on the Tor exit list", torExitAddr, true, false, true},
|
||||
{"on the iCloud Private Relay list", relayAddr, false, true, true},
|
||||
{"on another list", listedUnroutedAddr, false, false, true},
|
||||
{"routed IPv4", addrInTestPrefixV4, false, false, true},
|
||||
{"routed IPv6", addrInTestPrefixV6, false, false, true},
|
||||
{"private IPv4", "10.0.0.1", false, false, true},
|
||||
{"loopback IPv4", "127.0.0.1", false, false, true},
|
||||
{"public IPv4", "1.1.1.1", false, false, false},
|
||||
{"private IPv6", "fd00::1", false, false, true},
|
||||
{"loopback IPv6", "::1", false, false, true},
|
||||
{"public IPv6", "2606:4700:4700::1111", false, false, false},
|
||||
{"public IPv4 in IPv6 form", "::ffff:1.1.1.1", false, false, false},
|
||||
{"private IPv4 in IPv6 form", "::ffff:10.0.0.1", false, false, true},
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
@@ -1517,7 +1616,10 @@ func TestIPLookupFlags(t *testing.T) {
|
||||
t.Fatalf("%s: decoding flags %s: %v", path, results[0]["flags"], err)
|
||||
}
|
||||
|
||||
want := map[string]bool{"is_tor": tt.tor, "is_bogon": tt.bogon}
|
||||
want := map[string]bool{
|
||||
"is_tor": tt.tor, "is_vpn": false, "is_relay": tt.relay,
|
||||
"is_bogon": tt.bogon,
|
||||
}
|
||||
if !maps.Equal(got, want) {
|
||||
t.Errorf("%s: flags %v, want %v", path, got, want)
|
||||
}
|
||||
@@ -1525,6 +1627,186 @@ func TestIPLookupFlags(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
// fakeGeoLocator answers every address with location, and keeps the addresses
|
||||
// of each call. It is safe for the concurrent subtests that share it.
|
||||
type fakeGeoLocator struct {
|
||||
location *geo.Location
|
||||
|
||||
mu sync.Mutex
|
||||
calls [][]netip.Addr
|
||||
}
|
||||
|
||||
func (f *fakeGeoLocator) Lookup(
|
||||
_ context.Context, addrs []netip.Addr,
|
||||
) []*geo.Location {
|
||||
f.mu.Lock()
|
||||
defer f.mu.Unlock()
|
||||
|
||||
f.calls = append(f.calls, addrs)
|
||||
|
||||
answers := make([]*geo.Location, len(addrs))
|
||||
for i := range addrs {
|
||||
answers[i] = f.location
|
||||
}
|
||||
|
||||
return answers
|
||||
}
|
||||
|
||||
// TestIPLookupGivesTheGeoObject looks up two public addresses. They go to the
|
||||
// locator in one call, and each result has the geo object under the keys the
|
||||
// API documents. These addresses are public, because a reserved one is not
|
||||
// looked up.
|
||||
func TestIPLookupGivesTheGeoObject(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
s := newServerWithTestRoutes(t)
|
||||
useFakeResolver(s, &fakeResolver{})
|
||||
|
||||
locator := &fakeGeoLocator{location: &geo.Location{
|
||||
CountryCode: new("DE"),
|
||||
City: new("Berlin"),
|
||||
Latitude: new(52.5),
|
||||
Source: "geojs",
|
||||
Updated: testRouteTime(0),
|
||||
}}
|
||||
s.SetGeoLocator(locator)
|
||||
|
||||
path := "/api/v1/ip/1.1.1.1,8.8.8.8"
|
||||
|
||||
results := resultFields(t, get(t, s, path).Body.Bytes())
|
||||
if len(results) != 2 {
|
||||
t.Fatalf("%s: %d results, want 2", path, len(results))
|
||||
}
|
||||
|
||||
if len(locator.calls) != 1 || len(locator.calls[0]) != 2 {
|
||||
t.Errorf("locator calls %v, want one with both addresses", locator.calls)
|
||||
}
|
||||
|
||||
var got map[string]json.RawMessage
|
||||
|
||||
err := json.Unmarshal(results[0]["geo"], &got)
|
||||
if err != nil {
|
||||
t.Fatalf("decoding geo %s: %v", results[0]["geo"], err)
|
||||
}
|
||||
|
||||
wantKeys := []string{
|
||||
"accuracy_radius_km", "city", "continent_code", "country",
|
||||
"country_code", "latitude", "longitude", "region", "source",
|
||||
"time_zone", "updated",
|
||||
}
|
||||
if keys := slices.Sorted(maps.Keys(got)); !slices.Equal(keys, wantKeys) {
|
||||
t.Errorf("geo keys %v, want %v", keys, wantKeys)
|
||||
}
|
||||
|
||||
if string(got["city"]) != `"Berlin"` || string(got["latitude"]) != "52.5" ||
|
||||
string(got["region"]) != jsonNull {
|
||||
t.Errorf("wrong geo %s", results[0]["geo"])
|
||||
}
|
||||
}
|
||||
|
||||
// TestIPLookupOfAReservedAddressHasNoGeo looks up a reserved address alone and
|
||||
// beside a public one. The reserved one never reaches the locator, and its geo
|
||||
// is null.
|
||||
func TestIPLookupOfAReservedAddressHasNoGeo(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
s := newServerWithTestRoutes(t)
|
||||
useFakeResolver(s, &fakeResolver{})
|
||||
|
||||
locator := &fakeGeoLocator{location: &geo.Location{Source: "geojs"}}
|
||||
s.SetGeoLocator(locator)
|
||||
|
||||
results := resultFields(t, get(t, s, "/api/v1/ip/"+addrInTestPrefixV4).Body.Bytes())
|
||||
if len(results) != 1 || string(results[0]["geo"]) != jsonNull {
|
||||
t.Fatalf("results %v, want one with geo null", results)
|
||||
}
|
||||
|
||||
if len(locator.calls) != 0 {
|
||||
t.Errorf("locator calls %v, want none", locator.calls)
|
||||
}
|
||||
|
||||
results = resultFields(t,
|
||||
get(t, s, "/api/v1/ip/"+addrInTestPrefixV4+",1.1.1.1").Body.Bytes())
|
||||
if len(results) != 2 || string(results[0]["geo"]) != jsonNull ||
|
||||
string(results[1]["geo"]) == jsonNull {
|
||||
t.Errorf("results %v, want geo only for the public address", results)
|
||||
}
|
||||
|
||||
if len(locator.calls) != 1 || len(locator.calls[0]) != 1 {
|
||||
t.Errorf("locator calls %v, want one with one address", locator.calls)
|
||||
}
|
||||
}
|
||||
|
||||
// TestIPLookupGeoIsNullWhenNoSourceAnswers checks an address the locator has
|
||||
// no answer for.
|
||||
func TestIPLookupGeoIsNullWhenNoSourceAnswers(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
s := newServerWithTestRoutes(t)
|
||||
useFakeResolver(s, &fakeResolver{})
|
||||
s.SetGeoLocator(&fakeGeoLocator{})
|
||||
|
||||
results := resultFields(t, get(t, s, "/api/v1/ip/1.1.1.1").Body.Bytes())
|
||||
if len(results) != 1 || string(results[0]["geo"]) != jsonNull {
|
||||
t.Errorf("results %v, want one with geo null", results)
|
||||
}
|
||||
}
|
||||
|
||||
// TestIPLookupReportsVPNProviders looks up addresses on no list, on a list
|
||||
// that is not a VPN list, on one VPN list and on two. is_vpn says whether any
|
||||
// VPN list holds the address, and vpn_providers names only those lists. An
|
||||
// address on none has [], not null.
|
||||
func TestIPLookupReportsVPNProviders(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
s := newServerWithTestRoutes(t)
|
||||
s.resolver = &fakeResolver{}
|
||||
s.SetIPLists(fakeIPLists{})
|
||||
|
||||
tests := []struct {
|
||||
name string
|
||||
addr string
|
||||
isVPN string
|
||||
providers string
|
||||
}{
|
||||
{"on no list", "1.1.1.1", "false", `[]`},
|
||||
{"on a list that is not a VPN list", listedUnroutedAddr, "false", `[]`},
|
||||
{"on one VPN list", oneVPNListAddr, "true", `["` + testVPNList + `"]`},
|
||||
{
|
||||
"on two VPN lists", twoVPNListsAddr, "true",
|
||||
`["` + otherTestVPN + `","` + testVPNList + `"]`,
|
||||
},
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
path := "/api/v1/ip/" + tt.addr
|
||||
|
||||
results := resultFields(t, get(t, s, path).Body.Bytes())
|
||||
if len(results) != 1 {
|
||||
t.Fatalf("%s: %d results, want 1", path, len(results))
|
||||
}
|
||||
|
||||
var flags map[string]json.RawMessage
|
||||
|
||||
err := json.Unmarshal(results[0]["flags"], &flags)
|
||||
if err != nil {
|
||||
t.Fatalf("%s: decoding flags %s: %v", path, results[0]["flags"], err)
|
||||
}
|
||||
|
||||
if got := string(flags["is_vpn"]); got != tt.isVPN {
|
||||
t.Errorf("%s: is_vpn %s, want %s", path, got, tt.isVPN)
|
||||
}
|
||||
|
||||
if got := string(results[0]["vpn_providers"]); got != tt.providers {
|
||||
t.Errorf("%s: vpn_providers %s, want %s", path, got, tt.providers)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// TestIPLookupReportsIPLists looks up two routed addresses, one of them on an
|
||||
// IP list. The listed address's result names the list, says what it means and
|
||||
// when it was updated. The other's result has an empty list.
|
||||
@@ -1532,7 +1814,7 @@ func TestIPLookupReportsIPLists(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
s := newServerWithTestRoutes(t)
|
||||
s.resolver = &fakeResolver{}
|
||||
useFakeResolver(s, &fakeResolver{})
|
||||
s.SetIPLists(fakeIPLists{})
|
||||
|
||||
rec := httptest.NewRecorder()
|
||||
@@ -1609,7 +1891,7 @@ func TestIPLookupReportsIPListsOfAnAddressWithNoRoute(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
s := newServerWithTestRoutes(t)
|
||||
s.resolver = &fakeResolver{}
|
||||
useFakeResolver(s, &fakeResolver{})
|
||||
s.SetIPLists(fakeIPLists{})
|
||||
|
||||
// routeFields are the fields of a result that come from the address's
|
||||
@@ -1687,9 +1969,9 @@ func TestIPLookupResultsHaveTheSameFields(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
s := newServerWithTestRoutes(t)
|
||||
s.resolver = &fakeResolver{hosts: map[string][]string{
|
||||
useFakeResolver(s, &fakeResolver{hosts: map[string][]string{
|
||||
twoAddressHostName: {addrInTestPrefixV4, listedUnroutedAddr},
|
||||
}}
|
||||
}})
|
||||
s.SetIPLists(fakeIPLists{})
|
||||
|
||||
// want is the fields of the first result.
|
||||
@@ -1722,7 +2004,7 @@ func TestIPLookupOfAnAddressWithoutPTRNamesOrStoredNetblock(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
s := newServerWithTestRoutes(t)
|
||||
s.resolver = &fakeResolver{}
|
||||
useFakeResolver(s, &fakeResolver{})
|
||||
|
||||
path := "/api/v1/ip/" + addrInTestPrefixV4
|
||||
|
||||
@@ -1744,6 +2026,32 @@ func TestIPLookupOfAnAddressWithoutPTRNamesOrStoredNetblock(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
// TestIPLookupGivesPTRNamesWithoutTrailingDots looks up an address whose PTR
|
||||
// name the resolver gives fully qualified. The result's ptr holds the name
|
||||
// without its trailing dot.
|
||||
func TestIPLookupGivesPTRNamesWithoutTrailingDots(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
s := newServerWithTestRoutes(t)
|
||||
useFakeResolver(s, &fakeResolver{ptrs: map[string][]string{
|
||||
addrInTestPrefixV4: {"host.example.com."},
|
||||
}})
|
||||
|
||||
path := "/api/v1/ip/" + addrInTestPrefixV4
|
||||
|
||||
results := resultFields(t, get(t, s, path).Body.Bytes())
|
||||
if len(results) != 1 {
|
||||
t.Fatalf("%s: %d results, want 1", path, len(results))
|
||||
}
|
||||
|
||||
got := string(results[0]["ptr"])
|
||||
|
||||
want := `["host.example.com"]`
|
||||
if got != want {
|
||||
t.Errorf("%s: ptr is %s, want %s", path, got, want)
|
||||
}
|
||||
}
|
||||
|
||||
// TestIPLookupAnswersMayBeCached checks the Cache-Control header of the IP
|
||||
// lookup answers. Caches may keep them for the IP lists' shortest interval,
|
||||
// and only the caller's own browser may keep the answer about the caller's
|
||||
@@ -1752,7 +2060,7 @@ func TestIPLookupAnswersMayBeCached(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
s := newServerWithTestRoutes(t)
|
||||
s.resolver = &fakeResolver{}
|
||||
useFakeResolver(s, &fakeResolver{})
|
||||
s.SetIPLists(fakeIPLists{})
|
||||
|
||||
tests := []struct {
|
||||
@@ -1803,10 +2111,10 @@ func TestIPLookupHidesPrivateAddresses(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
s := newServerWithTestRoutes(t)
|
||||
s.resolver = &fakeResolver{hosts: map[string][]string{
|
||||
useFakeResolver(s, &fakeResolver{hosts: map[string][]string{
|
||||
internalHostName: {internalHostAddr},
|
||||
"mixed.test": {internalHostAddr, addrInTestPrefixV4},
|
||||
}}
|
||||
}})
|
||||
|
||||
notPublic := internalHostName + ": " + hostNotPublicMessage
|
||||
|
||||
@@ -2287,9 +2595,13 @@ func settledGoroutineCount() int {
|
||||
return prev
|
||||
}
|
||||
|
||||
// waitForGoroutines waits until the goroutine count drops to target or below.
|
||||
// waitForGoroutines waits up to 15 seconds for the goroutine count to drop to
|
||||
// target or below.
|
||||
func waitForGoroutines(target int) bool {
|
||||
for range 100 {
|
||||
const maxWait = 15 * time.Second
|
||||
|
||||
deadline := time.Now().Add(maxWait)
|
||||
for time.Now().Before(deadline) {
|
||||
if runtime.NumGoroutine() <= target {
|
||||
return true
|
||||
}
|
||||
|
||||
@@ -85,7 +85,7 @@ func TestIPLookupGivesTheOriginMostPeersCarry(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
s := newServerWithTable(t, newTwoOriginTable())
|
||||
s.resolver = &fakeResolver{}
|
||||
useFakeResolver(s, &fakeResolver{})
|
||||
|
||||
firstSeen := testRouteTime(-1)
|
||||
|
||||
|
||||
@@ -76,7 +76,9 @@ func asLink(asn int) string {
|
||||
}
|
||||
|
||||
// TestHomePageCardsShowTheTableFigures fills a routing table and the asns
|
||||
// table, works out the cards, and checks that each card shows its figure.
|
||||
// table, works out the cards, and checks that each card shows its figure. The
|
||||
// cards are worked out in a synctest bubble, whose clock is fake, so a stall on
|
||||
// the host does not use up the 4-second limit on their database read.
|
||||
func TestHomePageCardsShowTheTableFigures(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
@@ -95,7 +97,9 @@ func TestHomePageCardsShowTheTableFigures(t *testing.T) {
|
||||
t.Fatalf("GetOrCreateASNBatch: %v", err)
|
||||
}
|
||||
|
||||
s.refreshHomeCards(t.Context())
|
||||
synctest.Test(t, func(t *testing.T) {
|
||||
s.refreshHomeCards(t.Context())
|
||||
})
|
||||
|
||||
page := get(t, s, "/").Body.String()
|
||||
|
||||
@@ -271,49 +275,66 @@ func TestPagesServeTheCopyOfTheCards(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
// TestKeepHomeCardsFreshWorksThemOutEveryInterval runs keepHomeCardsFresh with
|
||||
// a short interval. It works the cards out at once, again after the table
|
||||
// changes, and stops when its context ends.
|
||||
// TestKeepHomeCardsFreshWorksThemOutEveryInterval runs keepHomeCardsFresh. It
|
||||
// works the cards out at once, again an interval after the table changes, and
|
||||
// stops when its context ends. It runs in a synctest bubble, whose clock is
|
||||
// fake, so a stall on the host cannot fail it.
|
||||
func TestKeepHomeCardsFreshWorksThemOutEveryInterval(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
const (
|
||||
interval = 10 * time.Millisecond
|
||||
maxWait = 5 * time.Second
|
||||
)
|
||||
const interval = time.Minute
|
||||
|
||||
table := emptyTable()
|
||||
announce(table, "1.0.0.0/24", twoPrefixASN, testPeer(1), testRouteTime(0))
|
||||
|
||||
s := newServerWithTable(t, table)
|
||||
|
||||
ctx, cancel := context.WithCancel(t.Context())
|
||||
stopped := make(chan struct{})
|
||||
|
||||
go func() {
|
||||
s.keepHomeCardsFresh(ctx, interval)
|
||||
close(stopped)
|
||||
}()
|
||||
|
||||
newestRouteASNIs := func(asn int) bool {
|
||||
// newestRouteOrigin returns the origin AS of the newest route on the
|
||||
// cards, or 0 when there is none.
|
||||
newestRouteOrigin := func() int {
|
||||
cards := s.cards.Load()
|
||||
if cards == nil || cards.NewestRoute == nil {
|
||||
return 0
|
||||
}
|
||||
|
||||
return cards != nil && cards.NewestRoute != nil &&
|
||||
cards.NewestRoute.OriginASN == asn
|
||||
return cards.NewestRoute.OriginASN
|
||||
}
|
||||
|
||||
waitUntil(t, maxWait, func() bool { return newestRouteASNIs(twoPrefixASN) })
|
||||
synctest.Test(t, func(t *testing.T) {
|
||||
ctx, cancel := context.WithCancel(t.Context())
|
||||
stopped := make(chan struct{})
|
||||
|
||||
announce(table, "8.8.8.0/24", newestRouteASN, testPeer(2), testRouteTime(1))
|
||||
waitUntil(t, maxWait, func() bool { return newestRouteASNIs(newestRouteASN) })
|
||||
go func() {
|
||||
s.keepHomeCardsFresh(ctx, interval)
|
||||
close(stopped)
|
||||
}()
|
||||
|
||||
cancel()
|
||||
synctest.Wait()
|
||||
|
||||
select {
|
||||
case <-stopped:
|
||||
case <-time.After(maxWait):
|
||||
t.Fatal("keepHomeCardsFresh did not stop when its context ended")
|
||||
}
|
||||
if got := newestRouteOrigin(); got != twoPrefixASN {
|
||||
t.Fatalf("at the start, the newest route's AS is %d, want %d",
|
||||
got, twoPrefixASN)
|
||||
}
|
||||
|
||||
announce(table, "8.8.8.0/24", newestRouteASN, testPeer(2),
|
||||
testRouteTime(1))
|
||||
time.Sleep(interval)
|
||||
synctest.Wait()
|
||||
|
||||
if got := newestRouteOrigin(); got != newestRouteASN {
|
||||
t.Fatalf("an interval after the table changed, the newest "+
|
||||
"route's AS is %d, want %d", got, newestRouteASN)
|
||||
}
|
||||
|
||||
cancel()
|
||||
synctest.Wait()
|
||||
|
||||
select {
|
||||
case <-stopped:
|
||||
default:
|
||||
t.Fatal("keepHomeCardsFresh did not stop when its context ended")
|
||||
}
|
||||
})
|
||||
}
|
||||
|
||||
// TestStartedServerWorksOutTheCards starts a server and waits until its home
|
||||
@@ -321,7 +342,7 @@ func TestKeepHomeCardsFreshWorksThemOutEveryInterval(t *testing.T) {
|
||||
func TestStartedServerWorksOutTheCards(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
const maxWait = 5 * time.Second
|
||||
const maxWait = 15 * time.Second
|
||||
|
||||
s := newServerWithTable(t, newCardsTable())
|
||||
|
||||
|
||||
@@ -14,6 +14,7 @@ import (
|
||||
"regexp"
|
||||
"strings"
|
||||
"testing"
|
||||
"testing/synctest"
|
||||
"time"
|
||||
|
||||
"git.eeqj.de/sneak/routewatch/internal/config"
|
||||
@@ -114,7 +115,7 @@ func TestJSONAnswersPutNoUnitInAString(t *testing.T) {
|
||||
`^([0-9]+(\.[0-9]+)? ?(ns|µs|ms|s|m|h|B|kB|MB|GB|TB))+$`)
|
||||
|
||||
s := newServerWithTestRoutes(t)
|
||||
s.resolver = &fakeResolver{}
|
||||
useFakeResolver(s, &fakeResolver{})
|
||||
s.SetASNFetcher(fixedIntervalASNFetcher{interval: time.Second})
|
||||
|
||||
paths := []string{
|
||||
@@ -163,6 +164,10 @@ func TestJSONAnswersPutNoUnitInAString(t *testing.T) {
|
||||
// router and the middleware write. Before the fix a panic in a handler, and a
|
||||
// method /status.json or the health check does not take, answered with an
|
||||
// empty body (https://git.eeqj.de/sneak/routewatch/issues/201).
|
||||
//
|
||||
// The requests run in a synctest bubble, whose clock is fake, so a stall on
|
||||
// the host does not use up the 2-second limit of the health check or the
|
||||
// 4-second limit of the stats handlers.
|
||||
func TestJSONAnswersCarryMeta(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
@@ -174,13 +179,17 @@ func TestJSONAnswersCarryMeta(t *testing.T) {
|
||||
)
|
||||
|
||||
s := newServerWithTestRoutes(t)
|
||||
s.resolver = &fakeResolver{}
|
||||
useFakeResolver(s, &fakeResolver{})
|
||||
s.streamer.GetMetricsTracker().SetConnected(true)
|
||||
s.router.Get(writesNothingPath, JSONValidationMiddleware(http.HandlerFunc(
|
||||
func(http.ResponseWriter, *http.Request) {})).ServeHTTP)
|
||||
s.router.Get(panicsPath, JSONValidationMiddleware(http.HandlerFunc(
|
||||
func(http.ResponseWriter, *http.Request) { panic(errDiskIO) })).ServeHTTP)
|
||||
|
||||
failing := New(failingDB{}, newTestTable(),
|
||||
streamer.New(logger.New(), metrics.New()), logger.New(), &config.Config{})
|
||||
useFakeResolver(failing, &fakeResolver{})
|
||||
|
||||
tests := []struct {
|
||||
method string
|
||||
path string
|
||||
@@ -216,42 +225,40 @@ func TestJSONAnswersCarryMeta(t *testing.T) {
|
||||
{http.MethodGet, panicsPath, http.StatusInternalServerError},
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
checkAnswerCarriesMeta(t, s, httptest.NewRequestWithContext(
|
||||
t.Context(), tt.method, tt.path, nil), tt.wantCode)
|
||||
}
|
||||
synctest.Test(t, func(t *testing.T) {
|
||||
for _, tt := range tests {
|
||||
checkAnswerCarriesMeta(t, s, httptest.NewRequestWithContext(
|
||||
t.Context(), tt.method, tt.path, nil), tt.wantCode)
|
||||
}
|
||||
|
||||
// A request whose time limit has already passed answers 504. The handlers
|
||||
// write the answers of the first five routes, and the JSON check on the
|
||||
// route writes the last.
|
||||
pastTimeLimit, cancel := context.WithDeadline(t.Context(), time.Now())
|
||||
defer cancel()
|
||||
// A request whose time limit has already passed answers 504. The
|
||||
// handlers write the answers of the first five routes, and the JSON
|
||||
// check on the route writes the last.
|
||||
pastTimeLimit, cancel := context.WithDeadline(t.Context(), time.Now())
|
||||
defer cancel()
|
||||
|
||||
for _, path := range []string{
|
||||
statusJSONPath, statsPath, "/ip/" + addrInTestPrefixV4,
|
||||
"/api/v1/ip/" + addrInTestPrefixV4, asJSONPath, writesNothingPath,
|
||||
} {
|
||||
checkAnswerCarriesMeta(t, s, httptest.NewRequestWithContext(
|
||||
pastTimeLimit, http.MethodGet, path, nil), http.StatusGatewayTimeout)
|
||||
}
|
||||
for _, path := range []string{
|
||||
statusJSONPath, statsPath, "/ip/" + addrInTestPrefixV4,
|
||||
"/api/v1/ip/" + addrInTestPrefixV4, asJSONPath, writesNothingPath,
|
||||
} {
|
||||
checkAnswerCarriesMeta(t, s, httptest.NewRequestWithContext(
|
||||
pastTimeLimit, http.MethodGet, path, nil), http.StatusGatewayTimeout)
|
||||
}
|
||||
|
||||
// With a store whose reads fail, the health check answers 503, and the
|
||||
// handlers of the other routes 500.
|
||||
failing := New(failingDB{}, newTestTable(),
|
||||
streamer.New(logger.New(), metrics.New()), logger.New(), &config.Config{})
|
||||
failing.resolver = &fakeResolver{}
|
||||
|
||||
checkAnswerCarriesMeta(t, failing, httptest.NewRequestWithContext(
|
||||
t.Context(), http.MethodGet, healthCheckPath, nil),
|
||||
http.StatusServiceUnavailable)
|
||||
|
||||
for _, path := range []string{
|
||||
statusJSONPath, statsPath, asJSONPath,
|
||||
"/api/v1/ip/" + addrInTestPrefixV4, "/ip/" + addrInTestPrefixV4,
|
||||
} {
|
||||
// With a store whose reads fail, the health check answers 503, and
|
||||
// the handlers of the other routes 500.
|
||||
checkAnswerCarriesMeta(t, failing, httptest.NewRequestWithContext(
|
||||
t.Context(), http.MethodGet, path, nil), http.StatusInternalServerError)
|
||||
}
|
||||
t.Context(), http.MethodGet, healthCheckPath, nil),
|
||||
http.StatusServiceUnavailable)
|
||||
|
||||
for _, path := range []string{
|
||||
statusJSONPath, statsPath, asJSONPath,
|
||||
"/api/v1/ip/" + addrInTestPrefixV4, "/ip/" + addrInTestPrefixV4,
|
||||
} {
|
||||
checkAnswerCarriesMeta(t, failing, httptest.NewRequestWithContext(
|
||||
t.Context(), http.MethodGet, path, nil), http.StatusInternalServerError)
|
||||
}
|
||||
})
|
||||
}
|
||||
|
||||
// checkAnswerCarriesMeta serves req through the server's router, and checks
|
||||
@@ -514,7 +521,7 @@ func newServerWithConfig(
|
||||
|
||||
s := New(db, newTestTable(), streamer.New(logger.New(), metrics.New()),
|
||||
&logger.Logger{Logger: slog.New(slog.NewJSONHandler(requestLog, nil))}, cfg)
|
||||
s.resolver = &fakeResolver{}
|
||||
useFakeResolver(s, &fakeResolver{})
|
||||
|
||||
return s
|
||||
}
|
||||
|
||||
@@ -5,6 +5,7 @@ import (
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"testing"
|
||||
"testing/synctest"
|
||||
)
|
||||
|
||||
// TestRoutesAnswerHEAD sends HEAD for a page and JSON routes through the
|
||||
@@ -12,15 +13,23 @@ import (
|
||||
// lookup of the caller's own address, are in the list because chi's GetHead
|
||||
// does not reach them. Before the fix every route answered HEAD with 405
|
||||
// (https://git.eeqj.de/sneak/routewatch/issues/111).
|
||||
//
|
||||
// The requests run in a synctest bubble, whose clock is fake, so a stall on
|
||||
// the host does not use up the stats handlers' 4-second limit. The resolver
|
||||
// is fake because no time limit would cut off a real DNS lookup in a bubble.
|
||||
func TestRoutesAnswerHEAD(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
s := newServerWithTestRoutes(t)
|
||||
s.resolver = &fakeResolver{}
|
||||
|
||||
for _, path := range []string{"/", statusJSONPath, statsPath, "/ip", "/ip/"} {
|
||||
rec := httptest.NewRecorder()
|
||||
s.router.ServeHTTP(rec, httptest.NewRequestWithContext(
|
||||
t.Context(), http.MethodHead, path, nil))
|
||||
|
||||
synctest.Test(t, func(t *testing.T) {
|
||||
s.router.ServeHTTP(rec, httptest.NewRequestWithContext(
|
||||
t.Context(), http.MethodHead, path, nil))
|
||||
})
|
||||
|
||||
if rec.Code != http.StatusOK {
|
||||
t.Errorf("HEAD %s: status %d, want %d; body %s",
|
||||
|
||||
@@ -24,6 +24,7 @@ import (
|
||||
|
||||
"git.eeqj.de/sneak/routewatch/internal/config"
|
||||
"git.eeqj.de/sneak/routewatch/internal/database"
|
||||
"git.eeqj.de/sneak/routewatch/internal/geo"
|
||||
"git.eeqj.de/sneak/routewatch/internal/iplist"
|
||||
"git.eeqj.de/sneak/routewatch/internal/logger"
|
||||
"git.eeqj.de/sneak/routewatch/internal/routetable"
|
||||
@@ -52,13 +53,23 @@ type IPLists interface {
|
||||
ShortestInterval() time.Duration
|
||||
}
|
||||
|
||||
// dnsResolver looks up the addresses of host names and the names of addresses
|
||||
// in DNS, as *net.Resolver does.
|
||||
// dnsResolver looks up the addresses of host names in DNS, as *net.Resolver
|
||||
// does.
|
||||
type dnsResolver interface {
|
||||
LookupHost(ctx context.Context, host string) ([]string, error)
|
||||
}
|
||||
|
||||
// PTRResolver looks up the PTR names of addresses, as *ptr.Cache does.
|
||||
type PTRResolver interface {
|
||||
LookupAddr(ctx context.Context, addr string) ([]string, error)
|
||||
}
|
||||
|
||||
// GeoLocator finds where addresses are, as *geo.Locator does. It returns one
|
||||
// entry for each address, nil when no source answers.
|
||||
type GeoLocator interface {
|
||||
Lookup(ctx context.Context, addrs []netip.Addr) []*geo.Location
|
||||
}
|
||||
|
||||
// Server provides HTTP endpoints for status monitoring
|
||||
type Server struct {
|
||||
router *chi.Mux
|
||||
@@ -75,10 +86,17 @@ type Server struct {
|
||||
// RealIPMiddleware believes
|
||||
trustedProxies []netip.Prefix
|
||||
|
||||
// resolver makes the IP lookup's DNS lookups: net.DefaultResolver, or a
|
||||
// resolver looks up the IP lookup's host names: net.DefaultResolver, or a
|
||||
// fake in tests
|
||||
resolver dnsResolver
|
||||
|
||||
// ptrResolver looks up the IP lookup's PTR names. SetPTRResolver sets it.
|
||||
ptrResolver PTRResolver
|
||||
|
||||
// geoLocator finds where the IP lookup's addresses are. SetGeoLocator
|
||||
// sets it.
|
||||
geoLocator GeoLocator
|
||||
|
||||
// cards is the copy of the home page's cards that the pages serve, nil
|
||||
// until the server first works them out
|
||||
cards atomic.Pointer[homeCards]
|
||||
@@ -196,6 +214,18 @@ func (s *Server) Stop(ctx context.Context) error {
|
||||
return nil
|
||||
}
|
||||
|
||||
// SetPTRResolver sets what looks up the IP lookup's PTR names. Call it before
|
||||
// Start: the handlers read it without a lock.
|
||||
func (s *Server) SetPTRResolver(resolver PTRResolver) {
|
||||
s.ptrResolver = resolver
|
||||
}
|
||||
|
||||
// SetGeoLocator sets what finds where the IP lookup's addresses are. Call it
|
||||
// before Start: the handlers read it without a lock.
|
||||
func (s *Server) SetGeoLocator(locator GeoLocator) {
|
||||
s.geoLocator = locator
|
||||
}
|
||||
|
||||
// SetASNFetcher sets the ASN WHOIS fetcher for on-demand lookups. Call it
|
||||
// before Start: the handlers read the fetcher without a lock.
|
||||
func (s *Server) SetASNFetcher(fetcher ASNFetcher) {
|
||||
|
||||
@@ -22,10 +22,10 @@ func TestStopClosesTheConnectionOfARequestStillRunning(t *testing.T) {
|
||||
stopDeadline = 100 * time.Millisecond
|
||||
// waitTimeout is how long the test waits for the request to reach
|
||||
// the handler, and then for the client to see its connection closed.
|
||||
waitTimeout = 2 * time.Second
|
||||
waitTimeout = 15 * time.Second
|
||||
// readHeaderTimeout bounds the test server's read of the request
|
||||
// header, as every http.Server must.
|
||||
readHeaderTimeout = time.Second
|
||||
readHeaderTimeout = waitTimeout
|
||||
)
|
||||
|
||||
release := make(chan struct{})
|
||||
|
||||
@@ -14,6 +14,7 @@ import (
|
||||
"strings"
|
||||
"sync/atomic"
|
||||
"testing"
|
||||
"testing/synctest"
|
||||
"time"
|
||||
|
||||
"git.eeqj.de/sneak/routewatch/internal/logger"
|
||||
@@ -398,7 +399,7 @@ func TestStopQueuesTheHeldStateMessages(t *testing.T) {
|
||||
const (
|
||||
held = 2
|
||||
lines = held + 1
|
||||
maxWait = 5 * time.Second
|
||||
maxWait = 15 * time.Second
|
||||
)
|
||||
|
||||
srv := httptest.NewServer(http.HandlerFunc(
|
||||
@@ -478,15 +479,15 @@ func TestStopQueuesTheHeldStateMessages(t *testing.T) {
|
||||
// TestStopReturnsByItsDeadline stops a streamer whose handler has a full queue
|
||||
// and a STATE message held for it, and a worker that stays busy until the test
|
||||
// ends, as one does whose handler waits for a database write. Stop must return
|
||||
// within its deadline plus a small margin instead of waiting for room on the
|
||||
// queue and for the worker, and log as not handed over the held message and
|
||||
// every message on the queue.
|
||||
// within its deadline plus a margin instead of waiting for room on the queue
|
||||
// and for the worker, and log as not handed over the held message and every
|
||||
// message on the queue.
|
||||
func TestStopReturnsByItsDeadline(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
const (
|
||||
deadline = 100 * time.Millisecond
|
||||
margin = 2 * time.Second
|
||||
margin = 15 * time.Second
|
||||
)
|
||||
|
||||
var logs bytes.Buffer
|
||||
@@ -577,7 +578,7 @@ func TestStopBeforeMessageReachesQueues(t *testing.T) {
|
||||
func TestReconnectWhenStreamEnds(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
endResponse := func(http.ResponseWriter, *http.Request) {}
|
||||
endResponse := pipeTransport{write: func(context.Context, io.Writer) {}}
|
||||
checkReconnect(t, endResponse, streamIdleTimeout)
|
||||
}
|
||||
|
||||
@@ -589,9 +590,8 @@ func TestReconnectWhenStreamGoesIdle(t *testing.T) {
|
||||
|
||||
const idleTimeout = 50 * time.Millisecond
|
||||
|
||||
holdResponse := func(w http.ResponseWriter, r *http.Request) {
|
||||
flush(w)
|
||||
<-r.Context().Done()
|
||||
holdResponse := pipeTransport{
|
||||
write: func(ctx context.Context, _ io.Writer) { <-ctx.Done() },
|
||||
}
|
||||
checkReconnect(t, holdResponse, idleTimeout)
|
||||
}
|
||||
@@ -604,16 +604,51 @@ func TestReconnectWhenServerNeverAnswers(t *testing.T) {
|
||||
|
||||
const idleTimeout = 50 * time.Millisecond
|
||||
|
||||
neverAnswer := func(_ http.ResponseWriter, r *http.Request) {
|
||||
<-r.Context().Done()
|
||||
}
|
||||
neverAnswer := roundTripFunc(
|
||||
func(req *http.Request) (*http.Response, error) {
|
||||
<-req.Context().Done()
|
||||
|
||||
return nil, req.Context().Err()
|
||||
},
|
||||
)
|
||||
checkReconnect(t, neverAnswer, idleTimeout)
|
||||
}
|
||||
|
||||
// TestStreamNotDroppedWhileLinesArrive has the server send lines well within
|
||||
// the idle timeout for several times the timeout, then end the response. Each
|
||||
// line must start the idle timeout over, so the stream lasts until the server
|
||||
// ends it instead of being dropped as idle.
|
||||
// pipeTransport answers every request with a response whose body write fills.
|
||||
// The body fails when the request's context ends, as a connection's does.
|
||||
// write gets that context, so that it can end when the request does.
|
||||
type pipeTransport struct {
|
||||
write func(context.Context, io.Writer)
|
||||
}
|
||||
|
||||
func (p pipeTransport) RoundTrip(req *http.Request) (*http.Response, error) {
|
||||
body, w := io.Pipe()
|
||||
|
||||
context.AfterFunc(req.Context(), func() {
|
||||
_ = w.CloseWithError(req.Context().Err())
|
||||
})
|
||||
|
||||
go func(ctx context.Context) {
|
||||
p.write(ctx, w)
|
||||
|
||||
_ = w.Close()
|
||||
}(req.Context())
|
||||
|
||||
return &http.Response{StatusCode: http.StatusOK, Body: body}, nil
|
||||
}
|
||||
|
||||
// roundTripFunc makes a function an http.RoundTripper.
|
||||
type roundTripFunc func(*http.Request) (*http.Response, error)
|
||||
|
||||
func (f roundTripFunc) RoundTrip(req *http.Request) (*http.Response, error) {
|
||||
return f(req)
|
||||
}
|
||||
|
||||
// TestStreamNotDroppedWhileLinesArrive has a response send lines well within
|
||||
// the idle timeout for several times the timeout, then end. Each line must
|
||||
// start the idle timeout over, so the stream lasts until the response ends
|
||||
// instead of being dropped as idle. It runs in a synctest bubble, whose clock
|
||||
// is fake, so a stall on the host does not use up the timeout.
|
||||
func TestStreamNotDroppedWhileLinesArrive(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
@@ -623,26 +658,35 @@ func TestStreamNotDroppedWhileLinesArrive(t *testing.T) {
|
||||
sendFor = 5 * idleTimeout
|
||||
)
|
||||
|
||||
srv := httptest.NewServer(http.HandlerFunc(
|
||||
func(w http.ResponseWriter, _ *http.Request) {
|
||||
// The streamer is made outside the bubble. Its metrics start a goroutine
|
||||
// that never returns, and a bubble waits for each of its goroutines to
|
||||
// return.
|
||||
s := New(logger.New(), metrics.New())
|
||||
s.idleTimeout = idleTimeout
|
||||
s.client = &http.Client{Transport: pipeTransport{
|
||||
write: func(ctx context.Context, w io.Writer) {
|
||||
deadline := time.Now().Add(sendFor)
|
||||
for time.Now().Before(deadline) {
|
||||
_, _ = io.WriteString(w, updateLine)
|
||||
|
||||
flush(w)
|
||||
time.Sleep(lineInterval)
|
||||
// The wait ends with the stream. A bubble's clock stops when
|
||||
// the test function returns, and a goroutine then still
|
||||
// asleep panics the test binary.
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
return
|
||||
case <-time.After(lineInterval):
|
||||
}
|
||||
}
|
||||
}))
|
||||
defer srv.Close()
|
||||
},
|
||||
}}
|
||||
|
||||
s := New(logger.New(), metrics.New())
|
||||
s.url = srv.URL
|
||||
s.idleTimeout = idleTimeout
|
||||
|
||||
err := s.stream(context.Background())
|
||||
if !errors.Is(err, errStreamEnded) {
|
||||
t.Fatalf("stream returned %v, want %v", err, errStreamEnded)
|
||||
}
|
||||
synctest.Test(t, func(t *testing.T) {
|
||||
err := s.stream(context.Background())
|
||||
if !errors.Is(err, errStreamEnded) {
|
||||
t.Fatalf("stream returned %v, want %v", err, errStreamEnded)
|
||||
}
|
||||
})
|
||||
}
|
||||
|
||||
// TestStreamReadsLongLine has the server send one UPDATE line of about
|
||||
@@ -706,7 +750,7 @@ func TestHandlerStatsReadWhileMessagesAreDropped(t *testing.T) {
|
||||
const (
|
||||
lines = 5
|
||||
wantDropped = lines - 2
|
||||
maxWait = 5 * time.Second
|
||||
maxWait = 15 * time.Second
|
||||
)
|
||||
|
||||
srv := httptest.NewServer(http.HandlerFunc(
|
||||
@@ -820,61 +864,70 @@ func streamLines(t *testing.T, s *Streamer, lines string) {
|
||||
// line and then stays open. The streamer must come back for a second
|
||||
// response, must not be marked connected when it does, and must handle the
|
||||
// UPDATE in it.
|
||||
//
|
||||
// It runs in a synctest bubble, whose clock is fake, so a stall on the host
|
||||
// does not use up the idle timeout. The fake server gets the requests in the
|
||||
// order the streamer makes them, because no real connection carries them.
|
||||
func checkReconnect(
|
||||
t *testing.T, firstResponse http.HandlerFunc, idleTimeout time.Duration,
|
||||
t *testing.T, firstResponse http.RoundTripper, idleTimeout time.Duration,
|
||||
) {
|
||||
t.Helper()
|
||||
|
||||
const maxWait = 5 * time.Second
|
||||
const maxWait = 15 * time.Second
|
||||
|
||||
// The tracker is made outside the bubble. It starts a goroutine that never
|
||||
// returns, and a bubble waits for each of its goroutines to return.
|
||||
tracker := metrics.New()
|
||||
connectedAtReconnect := make(chan bool, 1)
|
||||
|
||||
var requests atomic.Int32
|
||||
synctest.Test(t, func(t *testing.T) {
|
||||
connectedAtReconnect := make(chan bool, 1)
|
||||
laterResponse := pipeTransport{
|
||||
write: func(ctx context.Context, w io.Writer) {
|
||||
_, _ = io.WriteString(w, updateLine)
|
||||
|
||||
srv := httptest.NewServer(http.HandlerFunc(
|
||||
func(w http.ResponseWriter, r *http.Request) {
|
||||
if requests.Add(1) == 1 {
|
||||
firstResponse(w, r)
|
||||
<-ctx.Done()
|
||||
},
|
||||
}
|
||||
|
||||
return
|
||||
}
|
||||
var requests atomic.Int32
|
||||
|
||||
select {
|
||||
case connectedAtReconnect <- tracker.IsConnected():
|
||||
default:
|
||||
}
|
||||
s := New(logger.New(), tracker)
|
||||
s.client = &http.Client{Transport: roundTripFunc(
|
||||
func(req *http.Request) (*http.Response, error) {
|
||||
if requests.Add(1) == 1 {
|
||||
return firstResponse.RoundTrip(req)
|
||||
}
|
||||
|
||||
_, _ = io.WriteString(w, updateLine)
|
||||
select {
|
||||
case connectedAtReconnect <- tracker.IsConnected():
|
||||
default:
|
||||
}
|
||||
|
||||
flush(w)
|
||||
<-r.Context().Done()
|
||||
}))
|
||||
defer srv.Close()
|
||||
return laterResponse.RoundTrip(req)
|
||||
},
|
||||
)}
|
||||
s.idleTimeout = idleTimeout
|
||||
s.minBackoff = time.Millisecond
|
||||
handler := updateHandler{handled: make(chan struct{}, 1)}
|
||||
s.RegisterHandler(handler)
|
||||
|
||||
s := New(logger.New(), tracker)
|
||||
s.url = srv.URL
|
||||
s.idleTimeout = idleTimeout
|
||||
s.minBackoff = time.Millisecond
|
||||
handler := updateHandler{handled: make(chan struct{}, 1)}
|
||||
s.RegisterHandler(handler)
|
||||
err := s.Start(t.Context())
|
||||
if err != nil {
|
||||
t.Fatalf("Start returned error: %v", err)
|
||||
}
|
||||
defer s.Stop(t.Context())
|
||||
|
||||
err := s.Start(t.Context())
|
||||
if err != nil {
|
||||
t.Fatalf("Start returned error: %v", err)
|
||||
}
|
||||
defer s.Stop(t.Context())
|
||||
select {
|
||||
case <-handler.handled:
|
||||
case <-time.After(maxWait):
|
||||
t.Fatalf("no message from a second response handled within %s", maxWait)
|
||||
}
|
||||
|
||||
select {
|
||||
case <-handler.handled:
|
||||
case <-time.After(maxWait):
|
||||
t.Fatalf("no message from a second response handled within %s", maxWait)
|
||||
}
|
||||
|
||||
if <-connectedAtReconnect {
|
||||
t.Error("streamer was still marked connected when it came back " +
|
||||
"for a second response")
|
||||
}
|
||||
if <-connectedAtReconnect {
|
||||
t.Error("streamer was still marked connected when it came back " +
|
||||
"for a second response")
|
||||
}
|
||||
})
|
||||
}
|
||||
|
||||
// settledGoroutineCount lets transient goroutines finish, then reports the
|
||||
@@ -896,9 +949,13 @@ func settledGoroutineCount() int {
|
||||
return prev
|
||||
}
|
||||
|
||||
// waitForGoroutines waits until the goroutine count drops to target or below.
|
||||
// waitForGoroutines waits up to 15 seconds for the goroutine count to drop to
|
||||
// target or below.
|
||||
func waitForGoroutines(target int) bool {
|
||||
for range 100 {
|
||||
const maxWait = 15 * time.Second
|
||||
|
||||
deadline := time.Now().Add(maxWait)
|
||||
for time.Now().Before(deadline) {
|
||||
if runtime.NumGoroutine() <= target {
|
||||
return true
|
||||
}
|
||||
|
||||
@@ -544,6 +544,8 @@
|
||||
<span>{{appLicense}}</span>
|
||||
<span class="separator">|</span>
|
||||
<span><a href="{{appGitCommitURL}}">{{appGitRevision}}</a></span>
|
||||
<span class="separator">|</span>
|
||||
<span>IP Geolocation by <a href="https://db-ip.com">DB-IP</a></span>
|
||||
</footer>
|
||||
|
||||
<script nonce="{{.CSPNonce}}">
|
||||
|
||||
@@ -67,6 +67,23 @@ func TestFooterLicense(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
// TestHomePageCreditsDBIP expects the home page's footer to carry the credit
|
||||
// to DB-IP.
|
||||
func TestHomePageCreditsDBIP(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
const attribution = `IP Geolocation by <a href="https://db-ip.com">DB-IP</a>`
|
||||
|
||||
_, footer, found := strings.Cut(renderPage(t, templates.IndexTemplate()), "<footer")
|
||||
if !found {
|
||||
t.Fatal("the page has no footer")
|
||||
}
|
||||
|
||||
if !strings.Contains(footer, attribution) {
|
||||
t.Errorf("footer does not credit DB-IP: %q", footer)
|
||||
}
|
||||
}
|
||||
|
||||
// TestFreshnessIndicator renders the home and status pages and expects each to
|
||||
// have, right after the status element, the element where the page's script
|
||||
// shows its spinner or "stale".
|
||||
|
||||
Reference in new issue
Block a user