Compare commits
1
Commits
main
..
dcf2e8aa3d
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
dcf2e8aa3d |
+6
-12
@@ -121,11 +121,10 @@ RUN chown -R routewatch:routewatch /app
|
|||||||
|
|
||||||
ENV XDG_DATA_HOME=/var/lib
|
ENV XDG_DATA_HOME=/var/lib
|
||||||
|
|
||||||
# Cap the Go heap at 2.5 GiB so the runtime collects harder before the
|
# Cap the Go heap at 1.5 GiB so the runtime collects harder before the
|
||||||
# container's memory limit is reached. A full routing table takes about 1 GB
|
# container's memory limit is reached. setpriv in the entrypoint preserves this
|
||||||
# of it (README "Memory"). setpriv in the entrypoint preserves this the way it
|
# the way it does XDG_DATA_HOME above.
|
||||||
# does XDG_DATA_HOME above.
|
ENV GOMEMLIMIT=1536MiB
|
||||||
ENV GOMEMLIMIT=2560MiB
|
|
||||||
|
|
||||||
# Cap glibc's malloc arenas. The SQLite C library allocates and frees millions
|
# Cap glibc's malloc arenas. The SQLite C library allocates and frees millions
|
||||||
# of small page-cache chunks from many threads; glibc otherwise creates up to
|
# of small page-cache chunks from many threads; glibc otherwise creates up to
|
||||||
@@ -140,13 +139,8 @@ EXPOSE 8080
|
|||||||
|
|
||||||
COPY ./entrypoint.sh /entrypoint.sh
|
COPY ./entrypoint.sh /entrypoint.sh
|
||||||
|
|
||||||
# Health check using the health endpoint, on the port PORT names. During the
|
# Health check using the health endpoint, on the port PORT names
|
||||||
# start period, Docker 25 and later checks every 5 seconds, and the first pass
|
HEALTHCHECK --interval=30s --timeout=5s --start-period=10s --retries=3 \
|
||||||
# makes the container healthy. After it, Docker checks every 30 seconds. The
|
|
||||||
# start period covers the 60 seconds after which upaas reads the health
|
|
||||||
# (README "Running under upaas"). Startup reads the routing table first,
|
|
||||||
# which takes about 25 seconds when the table is full.
|
|
||||||
HEALTHCHECK --interval=30s --timeout=5s --start-period=60s --retries=3 \
|
|
||||||
CMD curl -sf "http://localhost:${PORT:-8080}/.well-known/healthcheck.json" || exit 1
|
CMD curl -sf "http://localhost:${PORT:-8080}/.well-known/healthcheck.json" || exit 1
|
||||||
|
|
||||||
ENTRYPOINT ["/bin/bash", "/entrypoint.sh" ]
|
ENTRYPOINT ["/bin/bash", "/entrypoint.sh" ]
|
||||||
|
|||||||
@@ -52,6 +52,7 @@ clean:
|
|||||||
run: build
|
run: build
|
||||||
DEBUG=routewatch ./bin/routewatch 2>&1 | tee log.txt
|
DEBUG=routewatch ./bin/routewatch 2>&1 | tee log.txt
|
||||||
|
|
||||||
# Rebuild the AS list built into RouteWatch from as.csv (needs the network).
|
|
||||||
asupdate:
|
asupdate:
|
||||||
@script/asupdate
|
@echo "Updating AS info data..."
|
||||||
|
@go run cmd/asinfo-gen/main.go | gzip > pkg/asinfo/asdata.json.gz.tmp && \
|
||||||
|
mv pkg/asinfo/asdata.json.gz.tmp pkg/asinfo/asdata.json.gz
|
||||||
|
|||||||
@@ -1,27 +1,25 @@
|
|||||||
# RouteWatch
|
# RouteWatch
|
||||||
|
|
||||||
RouteWatch is an MIT-licensed Go daemon by @sneak that monitors the BGP routing
|
RouteWatch is an MIT-licensed Go daemon by @sneak that monitors the BGP routing
|
||||||
table in real time. It works like this:
|
table in real time: it streams BGP UPDATE messages from the RIPE RIS Live
|
||||||
|
service, maintains a live routing table in SQLite, and provides HTTP APIs for
|
||||||
- It streams BGP UPDATE messages from the RIPE RIS Live service.
|
querying routing information.
|
||||||
- It keeps the whole routing table in memory, with a copy in SQLite.
|
|
||||||
- It serves web pages and a JSON API for looking routes up.
|
|
||||||
|
|
||||||
## Rationale
|
## Rationale
|
||||||
|
|
||||||
RIPE RIS Live streams the BGP updates that RIPE's route collectors receive, as
|
RIPE RIS Live streams the BGP updates that RIPE's route collectors receive, as
|
||||||
they happen, from the moment a client connects. It reports changes, not the
|
they happen, from the moment a client connects. It reports changes, not the
|
||||||
routing table they add up to, so it cannot say which AS an IP address is routed
|
routing table they add up to, so it cannot say which AS an IP address is routed
|
||||||
to now or which routes a prefix has. RouteWatch applies the feed's updates to a
|
to now or which routes a prefix has. RouteWatch applies the feed's updates to
|
||||||
routing table that it keeps in memory, with a copy in its own SQLite database.
|
live routing tables in its own SQLite database and answers IP-to-AS lookups and
|
||||||
It answers IP-to-AS lookups and prefix pages from that table.
|
prefix pages from them.
|
||||||
|
|
||||||
The feed sends no table when a client connects. So the routing table holds only
|
The feed sends no table when a client connects, so these tables hold only the
|
||||||
the routes announced since RouteWatch first started. An IP-to-AS lookup finds
|
routes the feed has announced since RouteWatch first started: an IP-to-AS lookup
|
||||||
nothing for an address whose routes have not been announced since then. The
|
finds nothing for an address whose routes have not been announced since then.
|
||||||
routing table also misses updates RouteWatch drops when it falls behind, and
|
The tables also miss the updates RouteWatch drops when it falls behind under
|
||||||
updates sent while it is stopped. So a lookup or a prefix page can show a route
|
load and those sent while it is stopped, so a lookup or a prefix page can show a
|
||||||
the feed has withdrawn.
|
route the feed has withdrawn.
|
||||||
|
|
||||||
## Features
|
## Features
|
||||||
|
|
||||||
@@ -29,8 +27,6 @@ the feed has withdrawn.
|
|||||||
- Maintains live IPv4 and IPv6 routing tables
|
- Maintains live IPv4 and IPv6 routing tables
|
||||||
- Tracks AS peering relationships
|
- Tracks AS peering relationships
|
||||||
- HTTP API for IP-to-AS lookups, prefix details, and AS information
|
- HTTP API for IP-to-AS lookups, prefix details, and AS information
|
||||||
- IP lookups report the public blocklists an address is on, from copies kept in
|
|
||||||
memory and on disk
|
|
||||||
- Automatic reconnection with exponential backoff
|
- Automatic reconnection with exponential backoff
|
||||||
- Batched database writes for high performance
|
- Batched database writes for high performance
|
||||||
- Backpressure handling to prevent memory exhaustion
|
- Backpressure handling to prevent memory exhaustion
|
||||||
@@ -58,24 +54,12 @@ DEBUG=routewatch ./routewatch
|
|||||||
|
|
||||||
Every route answers `HEAD` as it answers `GET`, without the body.
|
Every route answers `HEAD` as it answers `GET`, without the body.
|
||||||
|
|
||||||
The pages and the API read routes from the routing table that RouteWatch keeps
|
|
||||||
in memory. For each prefix and origin AS, the table keeps the two newest routes
|
|
||||||
in full, with their AS paths and times. These are the kept routes. The table
|
|
||||||
keeps only the address of every other peer that carries the prefix. When both
|
|
||||||
kept routes of an origin AS are withdrawn, that origin has no kept route until
|
|
||||||
one of its peers sends an update.
|
|
||||||
|
|
||||||
### Web Interface
|
### Web Interface
|
||||||
|
|
||||||
- `GET /` - Home page (HTML): headline figures, lookups by AS number and by IP
|
- `GET /` - Home page (HTML): headline figures, lookups by AS number and by IP
|
||||||
address, and a search of AS names. A card shows the `curl` command that looks
|
address, and a search of AS names
|
||||||
up the visitor's own address at `/ip/{addr}`, and that command's live answer.
|
|
||||||
- `GET /status` - HTML status dashboard, which reads `/api/v1/stats`
|
- `GET /status` - HTML status dashboard, which reads `/api/v1/stats`
|
||||||
- `GET /status.json` - JSON statistics: feed, memory, database and route
|
- `GET /status.json` - JSON statistics: feed, memory, database and route figures
|
||||||
figures. The route figures come from the routing table. `oldest_route` and
|
|
||||||
`newest_route` are the times of the oldest and the newest kept routes.
|
|
||||||
`origins_without_kept_route` counts, across all prefixes, the origin ASes that
|
|
||||||
have no kept route.
|
|
||||||
- `GET /.well-known/healthcheck.json` - Health check (JSON): 200 when the
|
- `GET /.well-known/healthcheck.json` - Health check (JSON): 200 when the
|
||||||
database answers and the RIS Live feed is connected, otherwise 503;
|
database answers and the RIS Live feed is connected, otherwise 503;
|
||||||
`data.status` is `ok` or `error` and `data.checks` says which check failed
|
`data.status` is `ok` or `error` and `data.checks` says which check failed
|
||||||
@@ -84,11 +68,10 @@ one of its peers sends an update.
|
|||||||
description come from the AS list built into RouteWatch. The page then shows
|
description come from the AS list built into RouteWatch. The page then shows
|
||||||
no first-seen date.
|
no first-seen date.
|
||||||
- `GET /prefix/{prefix}/{length}` - Prefix detail page (HTML), such as
|
- `GET /prefix/{prefix}/{length}` - Prefix detail page (HTML), such as
|
||||||
`/prefix/192.0.2.0/24`: its origin ASes and every peer that carries it. Only
|
`/prefix/192.0.2.0/24`: every live route for the prefix, one per peer, and its
|
||||||
the kept routes show an AS path, a next hop and a time.
|
origin ASes
|
||||||
- `GET /prefixlength/{length}` - Up to 500 IPv4 prefixes with that mask length,
|
- `GET /prefixlength/{length}` - Up to 500 IPv4 prefixes with that mask length,
|
||||||
picked at random (HTML). Each shows the origin AS that most peers carry it
|
picked at random, each with its newest route (HTML)
|
||||||
with, and the age of its newest kept route.
|
|
||||||
- `GET /prefixlength6/{length}` - The same for IPv6
|
- `GET /prefixlength6/{length}` - The same for IPv6
|
||||||
- `GET /ip/{addr}` - The same JSON answer as `/api/v1/ip/{ip}`
|
- `GET /ip/{addr}` - The same JSON answer as `/api/v1/ip/{ip}`
|
||||||
- `GET /ip/` - The same, for the caller's own address: the address the request
|
- `GET /ip/` - The same, for the caller's own address: the address the request
|
||||||
@@ -101,36 +84,24 @@ one of its peers sends an update.
|
|||||||
collector, stream and handler queue statistics. `stream.route_churn_per_min`
|
collector, stream and handler queue statistics. `stream.route_churn_per_min`
|
||||||
is the number of prefix announcements and withdrawals, from all peers, in the
|
is the number of prefix announcements and withdrawals, from all peers, in the
|
||||||
last full clock minute.
|
last full clock minute.
|
||||||
- `GET /api/v1/ip/{ip}` - Looks up each address. `{ip}` is an IP address, a host
|
- `GET /api/v1/ip/{ip}` - Looks up each address: its most specific live route
|
||||||
name, or a comma-separated list of at most 10 of these. `results` has one
|
(`netblock`), that route's origin AS with its WHOIS details, and the address's
|
||||||
entry for each address, with:
|
PTR names. `{ip}` is an IP address, a host name, or a comma-separated list of
|
||||||
- `ip` and `ptr`: the address and its PTR names
|
at most 10 of these. Of the addresses a host name resolves to, only public
|
||||||
- `netblock` and the fields beside it: the most specific prefix that holds
|
ones are looked up: a name with none, such as one that resolves to a private
|
||||||
the address, and its origin AS with that AS's WHOIS details. For a prefix
|
address, is reported as `host name has no public address`, without its
|
||||||
with several origin ASes, `asn` is the one that most peers carry.
|
addresses. `results` has one entry per address with a live route and `errors`
|
||||||
`last_seen` is the time of the prefix's newest kept route.
|
one per address or name that failed. The answer is 400 when `{ip}` lists more
|
||||||
- `blocklists`: the public blocklists the address is on (see Blocklists)
|
than 10, when it comes to more than 10 addresses once its names are resolved,
|
||||||
|
or when nothing in it is an IP address or a host name with a public address;
|
||||||
An address with no live route has a result without the route and AS fields.
|
and 404 when none of the addresses has a live route. A PTR lookup gets 2
|
||||||
Of the addresses a host name resolves to, only public ones are looked up. A
|
seconds; an address whose lookup takes longer is answered without PTR names.
|
||||||
name with none, such as one that resolves to a private address, is reported
|
|
||||||
as `host name has no public address`, without its addresses. `errors` has
|
|
||||||
one entry for each host name that failed. The answer is 400 when:
|
|
||||||
- `{ip}` lists more than 10
|
|
||||||
- `{ip}` comes to more than 10 addresses once its names are resolved
|
|
||||||
- nothing in `{ip}` is an IP address or a host name with a public address
|
|
||||||
|
|
||||||
A PTR lookup gets 2 seconds. An address whose lookup takes longer is
|
|
||||||
answered without PTR names.
|
|
||||||
|
|
||||||
- `GET /api/v1/as/{asn}` - The AS and the prefixes it originates, IPv4 and IPv6
|
- `GET /api/v1/as/{asn}` - The AS and the prefixes it originates, IPv4 and IPv6
|
||||||
apart, each with the time of its newest kept route. A prefix that keeps no
|
apart, each with when its newest route was updated. When the database has no
|
||||||
route with the AS has no `last_updated`. When the database has no record of
|
record of the AS, its handle and description come from the AS list built into
|
||||||
the AS, its handle and description come from the AS list built into
|
|
||||||
RouteWatch. The answer then has no `first_seen` or `last_seen`.
|
RouteWatch. The answer then has no `first_seen` or `last_seen`.
|
||||||
- `GET /api/v1/prefix/{prefix}/{length}` - One route for each peer that carries
|
- `GET /api/v1/prefix/{prefix}/{length}` - Every live route for the prefix, one
|
||||||
the prefix, and the same routes grouped by origin AS in `origins`. Only the
|
per peer, and the same routes grouped by origin AS in `origins`
|
||||||
kept routes have `as_path`, `next_hop` and `last_updated`.
|
|
||||||
- `GET /api/v1/search/as?q={text}` - Finds the ASes whose handle or description
|
- `GET /api/v1/search/as?q={text}` - Finds the ASes whose handle or description
|
||||||
contains `{text}`, ignoring case. `data` holds:
|
contains `{text}`, ignoring case. `data` holds:
|
||||||
- `query`: `{text}` without its leading and trailing spaces
|
- `query`: `{text}` without its leading and trailing spaces
|
||||||
@@ -139,10 +110,9 @@ one of its peers sends an update.
|
|||||||
`handle` and `description`
|
`handle` and `description`
|
||||||
|
|
||||||
The search reads the AS list built into RouteWatch, not the database. The
|
The search reads the AS list built into RouteWatch, not the database. The
|
||||||
list comes from
|
list comes from [ipverse/asn-info](https://github.com/ipverse/asn-info). A
|
||||||
[ipverse/as-metadata](https://github.com/ipverse/as-metadata). A match can
|
match can therefore be an AS that the database has no record of. A `q` that
|
||||||
therefore be an AS that the database has no record of. A `q` that is
|
is missing, empty or only spaces answers 400.
|
||||||
missing, empty or only spaces answers 400.
|
|
||||||
|
|
||||||
A JSON answer is `{"status": "ok", "data": ...}`, or for an error
|
A JSON answer is `{"status": "ok", "data": ...}`, or for an error
|
||||||
`{"status": "error", "error": {"msg": ..., "code": ...}}`, and carries an
|
`{"status": "error", "error": {"msg": ..., "code": ...}}`, and carries an
|
||||||
@@ -153,16 +123,16 @@ took. The health check is the exception: its 503 answer is also
|
|||||||
A request that runs past its time limit answers 504: the limit is 30 seconds,
|
A request that runs past its time limit answers 504: the limit is 30 seconds,
|
||||||
and 4 seconds for the database reads behind `/status.json` and `/api/v1/stats`.
|
and 4 seconds for the database reads behind `/status.json` and `/api/v1/stats`.
|
||||||
An AS number, prefix or mask length that does not parse answers 400, as does a
|
An AS number, prefix or mask length that does not parse answers 400, as does a
|
||||||
mask length out of range for its IP version. An AS answers 404 when it
|
mask length out of range for its IP version. An AS answers 404 when it is not in
|
||||||
originates no prefix and has no record in the database or in the AS list built
|
the AS list built into RouteWatch and the database has neither a record of it
|
||||||
into RouteWatch. AS 0 always answers 404, because it means no known origin. The
|
nor a live route it originates. AS 0 always answers 404, because it means no
|
||||||
AS name search never lists it. A prefix with no live route answers 404 too. The
|
known origin. The AS name search never lists it. A prefix with no live route
|
||||||
HTML pages give these errors as plain text. Under `/api/v1` and `/ip/`, a path
|
answers 404 too. The HTML pages give these errors as plain text. Under `/api/v1`
|
||||||
no route matches answers 404 with the message `Not found`, and `POST`, `PUT`,
|
and `/ip/`, a path no route matches answers 404 with the message `Not found`,
|
||||||
`PATCH`, `DELETE`, `OPTIONS`, `TRACE` or `CONNECT` answers 405 with
|
and `POST`, `PUT`, `PATCH`, `DELETE`, `OPTIONS`, `TRACE` or `CONNECT` answers
|
||||||
`Method not allowed`. A 500 answer says only `Internal server error`, and a host
|
405 with `Method not allowed`. A 500 answer says only `Internal server error`,
|
||||||
name that does not resolve only `could not resolve host name`; the detail goes
|
and a host name that does not resolve only `could not resolve host name`; the
|
||||||
to the log.
|
detail goes to the log.
|
||||||
|
|
||||||
Every answer carries `Strict-Transport-Security`, `Content-Security-Policy`,
|
Every answer carries `Strict-Transport-Security`, `Content-Security-Policy`,
|
||||||
`X-Frame-Options`, `X-Content-Type-Options`, `Referrer-Policy` and
|
`X-Frame-Options`, `X-Content-Type-Options`, `Referrer-Policy` and
|
||||||
@@ -178,66 +148,6 @@ in a prefix's `origins`) means the route has no known origin: its AS path ends
|
|||||||
in an AS set of several members, any of which may have originated it. The pages
|
in an AS set of several members, any of which may have originated it. The pages
|
||||||
show such a route as having no known origin.
|
show such a route as having no known origin.
|
||||||
|
|
||||||
## Blocklists
|
|
||||||
|
|
||||||
The IP lookup reports which public blocklists list each address. Each result has
|
|
||||||
a `blocklists` list with one entry for each list the address is on:
|
|
||||||
|
|
||||||
- `name`: the list's name, from the lists below
|
|
||||||
- `description`: what being on the list means
|
|
||||||
- `updated`: when RouteWatch last downloaded the list, or last heard from its
|
|
||||||
publisher that its copy was current
|
|
||||||
|
|
||||||
An address on no list has an empty `blocklists`. An address with no live route
|
|
||||||
still has a result with its `blocklists`.
|
|
||||||
|
|
||||||
The lists:
|
|
||||||
|
|
||||||
- `dshield`: the 20 networks that DShield saw scan the most targets in the last
|
|
||||||
three days. From [DShield](https://isc.sans.edu/feeds_doc.html), downloaded
|
|
||||||
every hour.
|
|
||||||
- `feodo_tracker`: botnet command-and-control servers. From abuse.ch
|
|
||||||
[Feodo Tracker](https://feodotracker.abuse.ch/blocklist/), downloaded every 15
|
|
||||||
minutes.
|
|
||||||
- `firehol_abusers_30d`: addresses reported for comment, form or forum spam in
|
|
||||||
the last 30 days. From
|
|
||||||
[FireHOL](https://iplists.firehol.org/?ipset=firehol_abusers_30d), downloaded
|
|
||||||
every 5 hours.
|
|
||||||
- `firehol_level3`: addresses reported for attacks, spyware or viruses in the
|
|
||||||
last 30 days. From
|
|
||||||
[FireHOL](https://iplists.firehol.org/?ipset=firehol_level3), downloaded every
|
|
||||||
3 hours.
|
|
||||||
- `spamhaus_drop` and `spamhaus_dropv6`: IPv4 and IPv6 networks that Spamhaus
|
|
||||||
says spammers or cyber criminals have hijacked or lease. From
|
|
||||||
[Spamhaus DROP](https://www.spamhaus.org/blocklists/do-not-route-or-peer/),
|
|
||||||
downloaded once a day. Spamhaus merged its EDROP list into DROP.
|
|
||||||
- `tor_exits`: Tor exit nodes, where traffic from the Tor network leaves. From
|
|
||||||
the [Tor Project](https://check.torproject.org/torbulkexitlist), downloaded
|
|
||||||
every hour.
|
|
||||||
|
|
||||||
No list is downloaded more often than its publisher updates it or allows. Each
|
|
||||||
can be downloaded whole without an account or a key.
|
|
||||||
|
|
||||||
How RouteWatch keeps the lists:
|
|
||||||
|
|
||||||
- It holds each list in memory, so a lookup makes no outside request.
|
|
||||||
- It keeps a copy of each list in the `blocklists` directory of the state
|
|
||||||
directory (see Configuration). Each copy is a file named after its list.
|
|
||||||
- Beside each copy, a file ending in `.validators` keeps the `Last-Modified` and
|
|
||||||
`ETag` the publisher sent with that copy.
|
|
||||||
- At startup it loads each list from its copy. It downloads a list at once when
|
|
||||||
the copy is older than the list's interval, or when there is no copy.
|
|
||||||
- A download sends the copy's `Last-Modified` as `If-Modified-Since` and its
|
|
||||||
`ETag` as `If-None-Match`. The publisher then sends the list only when it has
|
|
||||||
changed. Without either, the download asks for the whole list.
|
|
||||||
- A download fails when it takes over a minute or its file is over 16 MiB.
|
|
||||||
- When a download fails, or its file does not parse, RouteWatch keeps the
|
|
||||||
previous copy and logs the failure. It tries again one interval later.
|
|
||||||
|
|
||||||
A successful IP lookup answer carries `Cache-Control` with a `max-age` of 15
|
|
||||||
minutes, the shortest list interval. The answer for the caller's own address
|
|
||||||
(`/ip/`) is `private`, and the others are `public`.
|
|
||||||
|
|
||||||
## Code Structure
|
## Code Structure
|
||||||
|
|
||||||
```
|
```
|
||||||
@@ -247,11 +157,9 @@ routewatch/
|
|||||||
│ ├── asinfo-gen/ # Utility to generate AS info data
|
│ ├── asinfo-gen/ # Utility to generate AS info data
|
||||||
│ └── streamdumper/ # Debug utility for raw stream output
|
│ └── streamdumper/ # Debug utility for raw stream output
|
||||||
├── internal/
|
├── internal/
|
||||||
│ ├── blocklist/ # Public IP blocklists: downloads and lookups
|
|
||||||
│ ├── routewatch/ # Core application logic
|
│ ├── routewatch/ # Core application logic
|
||||||
│ ├── server/ # HTTP server and handlers
|
│ ├── server/ # HTTP server and handlers
|
||||||
│ ├── database/ # SQLite storage layer
|
│ ├── database/ # SQLite storage layer
|
||||||
│ ├── routetable/ # The whole routing table in memory
|
|
||||||
│ ├── streamer/ # RIPE RIS Live client
|
│ ├── streamer/ # RIPE RIS Live client
|
||||||
│ ├── ristypes/ # BGP message data structures
|
│ ├── ristypes/ # BGP message data structures
|
||||||
│ ├── logger/ # Structured logging wrapper
|
│ ├── logger/ # Structured logging wrapper
|
||||||
@@ -268,36 +176,31 @@ routewatch/
|
|||||||
|
|
||||||
### Component Relationships
|
### Component Relationships
|
||||||
|
|
||||||
`internal/routewatch/app.go` builds and starts these parts:
|
|
||||||
|
|
||||||
```
|
```
|
||||||
┌──────────────┐ ┌──────────────┐ ┌──────────────┐
|
┌─────────────────────────────────────────────────────────────────┐
|
||||||
│ Streamer │───▶│ Handlers │───▶│ Database │
|
│ RouteWatch │
|
||||||
│ │ │ │ │ │
|
│ (internal/routewatch/app.go - main orchestrator) │
|
||||||
│ RIS Live │ │ - ASHandler │ │ SQLite with │
|
├─────────────────────────────────────────────────────────────────┤
|
||||||
│ HTTP stream │ │ - PeerHandler│ │ WAL mode │
|
│ │
|
||||||
│ client │ │ - PrefixHdlr │ │ │
|
│ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ │
|
||||||
│ │ │ - PeeringHdlr│ │ Tables: │
|
│ │ Streamer │───▶│ Handlers │───▶│ Database │ │
|
||||||
└──────────────┘ └──────┬───────┘ │ - asns │
|
│ │ │ │ │ │ │ │
|
||||||
│ │ - prefixes │
|
│ │ RIS Live │ │ - ASHandler │ │ SQLite with │ │
|
||||||
▼ │ - routes │
|
│ │ HTTP stream │ │ - PeerHandler│ │ WAL mode │ │
|
||||||
┌──────────────┐ │ - route_peers│
|
│ │ client │ │ - PrefixHdlr │ │ │ │
|
||||||
│ Routing table│───▶│ - peerings │
|
│ │ │ │ - PeeringHdlr│ │ Tables: │ │
|
||||||
│ in memory │ │ - bgp_peers │
|
│ └──────────────┘ └──────────────┘ │ - asns │ │
|
||||||
└──────────────┘ └──────────────┘
|
│ │ - prefixes │ │
|
||||||
▲ ▲
|
│ ┌──────────────┐ ┌──────────────┐ │ - live_routes│ │
|
||||||
┌──────────────┐ ┌──────┴───────┐ │
|
│ │ Server │───▶│ Handlers │───▶│ - peerings │ │
|
||||||
│ Server │───▶│ Handlers │───────────┘
|
│ │ │ │ │ │ - bgp_peers │ │
|
||||||
│ │ │ │
|
│ │ Chi router │ │ Status, API │ └──────────────┘ │
|
||||||
│ Chi router │ │ Status, API │
|
│ │ port 8080 │ │ AS, Prefix │ │
|
||||||
│ port 8080 │ │ AS, Prefix │
|
│ └──────────────┘ └──────────────┘ │
|
||||||
└──────────────┘ └──────────────┘
|
│ │
|
||||||
|
└─────────────────────────────────────────────────────────────────┘
|
||||||
```
|
```
|
||||||
|
|
||||||
The prefix handler keeps the routing table in memory current. It writes the
|
|
||||||
table's changes to the database every second. The web handlers read routes from
|
|
||||||
the routing table, and everything else from the database.
|
|
||||||
|
|
||||||
### Execution Flow
|
### Execution Flow
|
||||||
|
|
||||||
1. **Startup** (`cmd/routewatch/main.go` → `internal/routewatch/cli.go`)
|
1. **Startup** (`cmd/routewatch/main.go` → `internal/routewatch/cli.go`)
|
||||||
@@ -306,17 +209,13 @@ the routing table, and everything else from the database.
|
|||||||
|
|
||||||
2. **Initialization** (`internal/routewatch/app.go`)
|
2. **Initialization** (`internal/routewatch/app.go`)
|
||||||
- Database created with SQLite schema (WAL mode, page caches as in "Memory")
|
- Database created with SQLite schema (WAL mode, page caches as in "Memory")
|
||||||
- The routing table read from the `routes` table into memory, before the
|
|
||||||
streamer connects. A full table takes about 25 seconds to read.
|
|
||||||
- Message handlers registered with the streamer
|
- Message handlers registered with the streamer
|
||||||
- HTTP server started on configured port
|
- HTTP server started on configured port
|
||||||
|
|
||||||
3. **Message Processing Pipeline**
|
3. **Message Processing Pipeline**
|
||||||
|
|
||||||
```
|
```
|
||||||
RIS Live Stream → JSON Parser → Message Dispatcher → Handler Queues
|
RIS Live Stream → JSON Parser → Message Dispatcher → Handler Queues → Batch Writers → SQLite
|
||||||
├→ Batch Writers → SQLite
|
|
||||||
└→ PrefixHandler → Routing Table in Memory → SQLite, every second
|
|
||||||
```
|
```
|
||||||
|
|
||||||
- Streamer connects to `ris-live.ripe.net` via HTTP
|
- Streamer connects to `ris-live.ripe.net` via HTTP
|
||||||
@@ -331,41 +230,21 @@ the routing table, and everything else from the database.
|
|||||||
4. **Handler Details**
|
4. **Handler Details**
|
||||||
- **ASHandler**: Tracks all ASNs seen in AS paths
|
- **ASHandler**: Tracks all ASNs seen in AS paths
|
||||||
- **PeerHandler**: Records BGP peer information
|
- **PeerHandler**: Records BGP peer information
|
||||||
- **PrefixHandler**: Keeps the routing table in memory current. It applies
|
- **PrefixHandler**: Maintains live routing table (upserts on announcement,
|
||||||
each update to the table as it handles it:
|
deletes on withdrawal or on an announcement whose AS path is empty or
|
||||||
- An announcement gives the peer a route to the prefix.
|
holds AS 0 anywhere, deletes all of a peer's routes when its BGP session
|
||||||
- A withdrawal removes the peer's route. So does an announcement whose
|
goes down). Routes are kept per peer IP, so when one peer IP has sessions
|
||||||
AS path is empty or holds AS 0 anywhere.
|
on two collectors, a session going down on one also removes the routes the
|
||||||
- A peer whose BGP session goes down leaves every prefix.
|
other still carries, until they are announced again.
|
||||||
|
|
||||||
Every second it writes the changed prefix and origin AS pairs to the
|
|
||||||
`routes` table. A write takes at most 20,000 pairs, and the rest follow
|
|
||||||
in the next writes. It also stores each new prefix with the time it was
|
|
||||||
first seen. Peers are known by their IP address. A peer IP can have
|
|
||||||
sessions on two collectors. A session going down on one then also
|
|
||||||
removes the routes the other still carries, until they are announced
|
|
||||||
again.
|
|
||||||
|
|
||||||
- **PeeringHandler**: Extracts AS peering relationships from AS paths
|
- **PeeringHandler**: Extracts AS peering relationships from AS paths
|
||||||
|
|
||||||
5. **HTTP Request Flow**
|
5. **HTTP Request Flow**
|
||||||
```
|
```
|
||||||
Request → Chi Router → Middleware (timeout, logging) → Handler → Routing Table or Database → Response
|
Request → Chi Router → Middleware (timeout, logging) → Handler → Database Query → Response
|
||||||
```
|
```
|
||||||
|
|
||||||
### Key Design Patterns
|
### Key Design Patterns
|
||||||
|
|
||||||
- **Routing Table in Memory**: `internal/routetable` holds the whole routing
|
|
||||||
table. For each prefix and origin AS it keeps:
|
|
||||||
- every peer whose current route to the prefix has that origin AS, one bit
|
|
||||||
per peer;
|
|
||||||
- the newest two of those routes in full, with AS path and time.
|
|
||||||
|
|
||||||
Of the other routes, only the peer is kept. The prefix handler writes the
|
|
||||||
changed pairs to the `routes` table every second, one row per prefix and
|
|
||||||
origin AS. Startup reads that table back into memory before the streamer
|
|
||||||
connects. A crash loses only the changes not yet written.
|
|
||||||
|
|
||||||
- **Batched Writes**: The handlers write the feed's updates to the database in
|
- **Batched Writes**: The handlers write the feed's updates to the database in
|
||||||
batches for performance
|
batches for performance
|
||||||
- **Backpressure**: Probabilistic message dropping when queues exceed 50%
|
- **Backpressure**: Probabilistic message dropping when queues exceed 50%
|
||||||
@@ -380,10 +259,11 @@ the routing table, and everything else from the database.
|
|||||||
with exponential backoff (5s-320s). The backoff resets after a stream that
|
with exponential backoff (5s-320s). The backoff resets after a stream that
|
||||||
lasted more than 30s. A stream with no line for 60s is dropped and
|
lasted more than 30s. A stream with no line for 60s is dropped and
|
||||||
reconnected.
|
reconnected.
|
||||||
- **IP Lookup**: the routing table in memory finds the most specific prefix that
|
- **IP Lookup**: the most specific live route is found by looking up the
|
||||||
holds the address. It looks up the address's prefix at each mask length,
|
address's prefix at each mask length, longest first, in the prefix index (at
|
||||||
longest first: at most 25 lookups for IPv4 and 113 for IPv6. Prefixes shorter
|
most 25 lookups for IPv4, 113 for IPv6); prefixes are stored in the text form
|
||||||
than /8 for IPv4 or /16 for IPv6, such as a default route, are not used.
|
Go's `net/netip` prints. Routes shorter than /8 for IPv4 or /16 for IPv6, such
|
||||||
|
as a default route, are not used.
|
||||||
|
|
||||||
### Database Schema
|
### Database Schema
|
||||||
|
|
||||||
@@ -397,14 +277,13 @@ asns(asn PRIMARY KEY, handle, description,
|
|||||||
prefixes_v4(prefix PRIMARY KEY, first_seen) WITHOUT ROWID
|
prefixes_v4(prefix PRIMARY KEY, first_seen) WITHOUT ROWID
|
||||||
prefixes_v6(prefix PRIMARY KEY, first_seen) WITHOUT ROWID
|
prefixes_v6(prefix PRIMARY KEY, first_seen) WITHOUT ROWID
|
||||||
|
|
||||||
-- The routing table, one row per prefix and origin AS that some peer
|
-- Live routing tables (one per IP version), one route per peer and prefix
|
||||||
-- carries: the peer set, and the newest two routes in full
|
live_routes_v4(id INTEGER PRIMARY KEY, prefix, mask_length, origin_asn,
|
||||||
routes(prefix, origin_asn, peers,
|
peer_ip, as_path, next_hop, last_updated,
|
||||||
newest_peer, newest_as_path, newest_time,
|
UNIQUE(peer_ip, prefix))
|
||||||
older_peer, older_as_path, older_time,
|
live_routes_v6(id INTEGER PRIMARY KEY, prefix, mask_length, origin_asn,
|
||||||
PRIMARY KEY (prefix, origin_asn)) WITHOUT ROWID
|
peer_ip, as_path, next_hop, last_updated,
|
||||||
-- The peers of the routes table, each with its number in the peer sets
|
UNIQUE(peer_ip, prefix))
|
||||||
route_peers(number PRIMARY KEY, address UNIQUE)
|
|
||||||
|
|
||||||
-- Relationship tracking
|
-- Relationship tracking
|
||||||
peerings(id PRIMARY KEY, as_a, as_b, first_seen, last_seen,
|
peerings(id PRIMARY KEY, as_a, as_b, first_seen, last_seen,
|
||||||
@@ -428,7 +307,7 @@ Configuration is handled via environment variables and OS-specific paths:
|
|||||||
| `PORT` | `8080` | HTTP server port, a whole number from 1 to 65535 |
|
| `PORT` | `8080` | HTTP server port, a whole number from 1 to 65535 |
|
||||||
| `DEBUG` | (empty) | Set to `routewatch` for debug logging |
|
| `DEBUG` | (empty) | Set to `routewatch` for debug logging |
|
||||||
| `XDG_DATA_HOME` | `/var/lib` (in the Docker image) | Base of the state directory; must be an absolute path |
|
| `XDG_DATA_HOME` | `/var/lib` (in the Docker image) | Base of the state directory; must be an absolute path |
|
||||||
| `GOMEMLIMIT` | `2560MiB` (in the Docker image) | Go soft memory limit; see Memory |
|
| `GOMEMLIMIT` | `1536MiB` (in the Docker image) | Go soft memory limit; see Memory |
|
||||||
| `MALLOC_ARENA_MAX` | `2` (in the Docker image) | glibc malloc arena cap, a positive whole number; see Memory |
|
| `MALLOC_ARENA_MAX` | `2` (in the Docker image) | glibc malloc arena cap, a positive whole number; see Memory |
|
||||||
| `TRUSTED_PROXIES` | (empty) | Addresses and CIDR ranges of trusted proxies; see below |
|
| `TRUSTED_PROXIES` | (empty) | Addresses and CIDR ranges of trusted proxies; see below |
|
||||||
|
|
||||||
@@ -444,7 +323,7 @@ client sent. For a request from any other address they use the address the
|
|||||||
request came from, whatever forwarding headers it carries. `X-Real-IP` and
|
request came from, whatever forwarding headers it carries. `X-Real-IP` and
|
||||||
`True-Client-IP` are not read.
|
`True-Client-IP` are not read.
|
||||||
|
|
||||||
State directory (the database and the blocklist copies):
|
State directory (database location):
|
||||||
|
|
||||||
- macOS: `~/Library/Application Support/berlin.sneak.app.routewatch/`
|
- macOS: `~/Library/Application Support/berlin.sneak.app.routewatch/`
|
||||||
- Linux: `/var/lib/berlin.sneak.app.routewatch/` when running as root, otherwise
|
- Linux: `/var/lib/berlin.sneak.app.routewatch/` when running as root, otherwise
|
||||||
@@ -454,37 +333,28 @@ State directory (the database and the blocklist copies):
|
|||||||
|
|
||||||
## Memory
|
## Memory
|
||||||
|
|
||||||
The daemon keeps the whole routing table in memory. The image sets ceilings that
|
The daemon holds a live routing table, so its memory grows with the size of the
|
||||||
keep it inside a 5 GiB container.
|
data it tracks. The image sets ceilings that keep it inside a 5 GiB container.
|
||||||
|
|
||||||
Budget:
|
Budget:
|
||||||
|
|
||||||
- Go heap: a 2.5 GiB soft limit (`GOMEMLIMIT=2560MiB`, set in the image).
|
- Go heap: a 1.5 GiB soft limit (`GOMEMLIMIT=1536MiB`, set in the image).
|
||||||
- The routing table takes about 700 bytes per prefix and origin AS. A full
|
- SQLite: 1 GiB of page cache for the one connection every write uses, which
|
||||||
table has about 1.4 million of these pairs, so it takes about 1 GB.
|
keeps the pages the route writes touch from one batch to the next, and 16 MiB
|
||||||
- The rest of the heap takes up to about 0.5 GiB. Most of it is the
|
for each of the 9 reading connections, whose cache SQLite empties whenever
|
||||||
handlers' batches and queues, described below.
|
another connection has written: about 1.15 GiB in all. A 1.5 GiB soft heap
|
||||||
- The remaining 1 GiB is room for the garbage collector. With much less, it
|
limit sits above the caches, so it does not shrink them, and a 2 GiB hard heap
|
||||||
would run almost all the time.
|
limit caps the C library.
|
||||||
- SQLite: 656 MiB of page cache.
|
- glibc allocator: the SQLite C library runs on glibc `malloc`, which frees
|
||||||
- The connection that does every write has 512 MiB. It keeps the pages the
|
page-cache chunks back to per-arena free lists rather than to the kernel, so
|
||||||
writes touch from one write to the next. With a full routing table, the
|
process RSS tracks the high-water mark of those arenas, not SQLite's live
|
||||||
`routes` table takes about 450 MB, so its pages fit.
|
heap. glibc creates up to eight arenas per core, so on a many-core host the
|
||||||
- Each of the 9 reading connections has 16 MiB. SQLite empties a reading
|
retained memory — and thus RSS — grows with the core count. The image sets
|
||||||
connection's cache whenever another connection has written.
|
`MALLOC_ARENA_MAX=2` to bound it; the two-arena cap costs nothing here because
|
||||||
- A 768 MiB soft heap limit sits above the caches, so it does not shrink
|
database writes are already serialized.
|
||||||
them. A 1 GiB hard heap limit caps the C library.
|
|
||||||
- glibc allocator: the SQLite C library allocates memory with glibc `malloc`.
|
|
||||||
- glibc keeps freed page-cache chunks in per-arena free lists instead of
|
|
||||||
returning them to the kernel. So the process's RSS follows the high-water
|
|
||||||
mark of those arenas, not SQLite's live heap.
|
|
||||||
- glibc creates up to eight arenas per core, so on a many-core host RSS
|
|
||||||
grows with the core count.
|
|
||||||
- The image sets `MALLOC_ARENA_MAX=2` to bound it. Two arenas cost nothing
|
|
||||||
here, because database writes already run one at a time.
|
|
||||||
- About 0.2 GiB for everything else in the runtime.
|
- About 0.2 GiB for everything else in the runtime.
|
||||||
|
|
||||||
The ceilings add up to 3.7 GiB: the Go heap's 2.5 GiB, SQLite's 1 GiB and the
|
The ceilings add up to 3.7 GiB: the Go heap's 1.5 GiB, SQLite's 2 GiB and the
|
||||||
0.2 GiB for the rest.
|
0.2 GiB for the rest.
|
||||||
|
|
||||||
Run the container with a memory limit of 5 GiB and swap disabled:
|
Run the container with a memory limit of 5 GiB and swap disabled:
|
||||||
@@ -496,36 +366,38 @@ docker run --memory=5g --memory-swap=5g ...
|
|||||||
or the equivalent in your deployment tool. This leaves headroom above the
|
or the equivalent in your deployment tool. This leaves headroom above the
|
||||||
ceilings for spikes and the kernel page cache.
|
ceilings for spikes and the kernel page cache.
|
||||||
|
|
||||||
To replace the image's Go soft limit, set `GOMEMLIMIT` in the environment, for
|
Override the Go soft limit by setting `GOMEMLIMIT` in the environment (for
|
||||||
example `-e GOMEMLIMIT=3GiB`. A limit close to the routing table's size makes
|
example `-e GOMEMLIMIT=1GiB`); this replaces the image default.
|
||||||
the garbage collector run almost all the time. `MALLOC_ARENA_MAX` can be
|
`MALLOC_ARENA_MAX` can be overridden the same way, but raising it lets RSS climb
|
||||||
overridden the same way, but raising it lets RSS climb again on a many-core
|
again on a many-core host.
|
||||||
host.
|
|
||||||
|
|
||||||
What happens at each limit:
|
What happens at each limit:
|
||||||
|
|
||||||
- Go soft limit: as the heap approaches `GOMEMLIMIT`, the runtime runs garbage
|
- Go soft limit: as the heap approaches `GOMEMLIMIT`, the runtime runs garbage
|
||||||
collection more aggressively rather than growing further.
|
collection more aggressively rather than growing further.
|
||||||
- SQLite: at the 768 MiB soft heap limit it recycles its page cache instead of
|
- SQLite: at the 1.5 GiB soft heap limit it recycles its page cache instead of
|
||||||
allocating more. At the 1 GiB hard heap limit a statement fails with an
|
allocating more; at the 2 GiB hard heap limit a statement fails with an
|
||||||
out-of-memory error, and the handler logs it. The prefix handler writes the
|
out-of-memory error, and the handler logs it and drops that batch. The process
|
||||||
routing table's changes again on a later write. The other writes drop their
|
keeps running.
|
||||||
batch. The process keeps running.
|
|
||||||
- Handler queues: each of the four handler queues holds at most 20,000 messages.
|
- Handler queues: each of the four handler queues holds at most 20,000 messages.
|
||||||
When a queue fills, the streamer drops messages instead of blocking, except a
|
When a queue fills, the streamer drops messages instead of blocking, except a
|
||||||
`STATE` message, which it holds, in feed order, until there is room: a few
|
`STATE` message, which it holds, in feed order, until there is room: a few
|
||||||
hundred bytes each, about ten a second.
|
hundred bytes each, about ten a second.
|
||||||
- Handler batches: while the AS, peer or prefix handler writes a batch to the
|
- Handler batches: while the AS, peer or prefix handler writes a batch to the
|
||||||
database, the updates that arrive collect in its next batch. That batch stops
|
database, the updates that arrive collect in its next batch, which stops at
|
||||||
at ten times the batch size:
|
ten times the batch size: 300,000 updates for the AS handler
|
||||||
- 300,000 updates for the AS handler (`asnBatchMaxSize`, about 10 MiB);
|
(`asnBatchMaxSize`, about 10 MiB), 100,000 for the peer handler
|
||||||
- 100,000 for the peer handler (`peerBatchMaxSize`, about 10 MiB);
|
(`peerBatchMaxSize`, about 10 MiB) and 250,000 for the prefix handler
|
||||||
- 250,000 for the prefix handler (`prefixBatchMaxSize`, about 15 MiB).
|
(`prefixBatchMaxSize`, about 60 MiB). At that size the handler takes nothing
|
||||||
|
more off its queue until the write ends, so the queue fills and the streamer
|
||||||
At that size the handler takes nothing more off its queue until the write
|
drops messages as above. When the delete of a peer's live routes after its
|
||||||
ends. The queue then fills, and the streamer drops messages as above. With
|
session went down fails, the prefix handler keeps the updates the peer sent
|
||||||
the batches being written, the three handlers' batches take at most about
|
after the down for the next write, which retries the delete first; on top of
|
||||||
0.1 GiB of the Go heap.
|
its 250,000 it keeps at most the last 25,000 of them (`prefixBatchSize`, about
|
||||||
|
6 MiB) and drops and logs the older ones, so while the delete keeps failing
|
||||||
|
the other peers' updates still come off the queue. With the batches being
|
||||||
|
written, the three handlers' batches take at most about 0.25 GiB of the Go
|
||||||
|
heap's 1.5 GiB.
|
||||||
|
|
||||||
With `DEBUG=routewatch` the daemon logs a `System stats` line every 60 seconds
|
With `DEBUG=routewatch` the daemon logs a `System stats` line every 60 seconds
|
||||||
with the goroutine count and Go memory figures.
|
with the goroutine count and Go memory figures.
|
||||||
@@ -547,25 +419,25 @@ What the [upaas](https://git.eeqj.de/sneak/upaas) app needs:
|
|||||||
- Health check: the image's `HEALTHCHECK` requests
|
- Health check: the image's `HEALTHCHECK` requests
|
||||||
`/.well-known/healthcheck.json` on the container port. upaas reads the
|
`/.well-known/healthcheck.json` on the container port. upaas reads the
|
||||||
container's health 60 seconds after a deploy and fails the deploy unless it is
|
container's health 60 seconds after a deploy and fails the deploy unless it is
|
||||||
`healthy`. For the first 60 seconds, Docker 25 and later runs the check every
|
`healthy`.
|
||||||
5 seconds, and the first pass makes the container healthy. Startup first reads
|
|
||||||
the routing table, which takes about 25 seconds when the table is full.
|
|
||||||
- Stop grace: the container needs 9 seconds between SIGTERM and SIGKILL. upaas
|
- Stop grace: the container needs 9 seconds between SIGTERM and SIGKILL. upaas
|
||||||
gives 10 (`stopTimeoutSeconds` in its `internal/docker/client.go`), as
|
gives 10 (`stopTimeoutSeconds` in its `internal/docker/client.go`), as
|
||||||
`docker stop` does by default. The daemon allows 8 seconds for the handlers to
|
`docker stop` does by default. The daemon allows 8 seconds for the handlers to
|
||||||
write what they hold and for the requests in progress to finish. Then it stops
|
write what they hold and for the requests in progress to finish, then stops
|
||||||
the HTTP server, closes the database and exits with status 0. A longer grace
|
the HTTP server, closes the database and exits with status 0. A longer grace
|
||||||
changes nothing. What is not written after 8 seconds is lost, and the streamer
|
changes nothing. What is not written after 8 seconds is lost, and the streamer
|
||||||
and each handler log how much.
|
and each handler log how much. When the prefix handler has fallen behind the
|
||||||
- The prefix handler's last write takes at most 20,000 of the routing
|
feed, a stop loses far more: its batch is full at 250,000 updates and the
|
||||||
table's changed pairs. A full-table peer's session going down changes
|
streamer's worker for it waits for room, so the streamer's stop takes the
|
||||||
about a million pairs, which take about a minute to write. A stop in that
|
whole 8 seconds and every handler then skips its last write. Lost then are the
|
||||||
minute leaves the rest unwritten.
|
messages left on the four handlers' queues, what the AS, peer and peering
|
||||||
- A lost withdrawal or peer down leaves its routes in the `routes` table,
|
handlers hold, the prefix handler's batch, and the write it has in progress,
|
||||||
and the next start reads them back. They stay until the peer sends another
|
up to another 250,000 updates, which take about 20 seconds and are cut off by
|
||||||
update for the prefix, or its session goes down.
|
the exit; withdrawals and peer downs are among them. A route withdrawn in them
|
||||||
- When the 8 seconds run out the database is not closed. When the database
|
stays in the live route table until its peer announces or withdraws the prefix
|
||||||
next opens, SQLite rolls back any write the exit cut off.
|
again or its session goes down. When the 8 seconds run out the database is not
|
||||||
|
closed, and SQLite rolls back a write the exit cut off when the database next
|
||||||
|
opens.
|
||||||
|
|
||||||
## Development
|
## Development
|
||||||
|
|
||||||
@@ -613,11 +485,6 @@ provide:
|
|||||||
or `go.sum`) and `go fmt` (writes), then `script/check`
|
or `go.sum`) and `go fmt` (writes), then `script/check`
|
||||||
- `script/install-precommit` — install the git pre-commit hook that runs
|
- `script/install-precommit` — install the git pre-commit hook that runs
|
||||||
`script/precommit`
|
`script/precommit`
|
||||||
- `script/asupdate` — rebuild the AS list built into RouteWatch,
|
|
||||||
`pkg/asinfo/asdata.json.gz`, from the `as.csv` file of
|
|
||||||
[ipverse/as-metadata](https://github.com/ipverse/as-metadata). It needs the
|
|
||||||
network. It runs `cmd/asinfo-gen`, which fails when it finds no AS. When
|
|
||||||
`cmd/asinfo-gen` fails, the old list stays and the script exits non-zero.
|
|
||||||
|
|
||||||
Tests and the linter run only in Docker, as the `test` and `lint` phases of the
|
Tests and the linter run only in Docker, as the `test` and `lint` phases of the
|
||||||
`Dockerfile`; the image cannot be built unless both pass. Every `docker build`
|
`Dockerfile`; the image cannot be built unless both pass. Every `docker build`
|
||||||
|
|||||||
@@ -25,106 +25,14 @@ are measured by a benchmark, not yet on the live feed.
|
|||||||
|
|
||||||
# Next Step
|
# Next Step
|
||||||
|
|
||||||
Run `next` on the live feed, as README "Running under upaas" describes, until
|
Run `next` on the live feed, as README "Running under upaas" describes, for the
|
||||||
the routing table stops growing. Post its size and memory figures on
|
final check https://git.eeqj.de/sneak/routewatch/issues/96 names: whether the
|
||||||
https://git.eeqj.de/sneak/routewatch/issues/122. The same run makes the final
|
prefix handler still drops updates after the first hour. If it does, a new issue
|
||||||
check https://git.eeqj.de/sneak/routewatch/issues/96 names: whether the prefix
|
|
||||||
handler still drops updates after the first hour. If it does, a new issue
|
|
||||||
follows with the numbers. Then an audit of the code at the commit that run
|
follows with the numbers. Then an audit of the code at the commit that run
|
||||||
tested.
|
tested.
|
||||||
|
|
||||||
# Completed Steps
|
# Completed Steps
|
||||||
|
|
||||||
- 2026-10-27: the per-peer route tables are gone, with the code that wrote,
|
|
||||||
counted and read them. The prefix handler applies each update to the routing
|
|
||||||
table in memory and writes the table's changes every second. It also stores
|
|
||||||
each new prefix with its first-seen time. The image's memory limits are set
|
|
||||||
for a full routing table of about 1 GB: `GOMEMLIMIT` is 2.5 GiB, the writing
|
|
||||||
connection's page cache 512 MiB, and SQLite's heap limits 768 MiB and 1 GiB
|
|
||||||
(closes #122)
|
|
||||||
- 2026-10-27: the pages, the API and the stats read every route from the routing
|
|
||||||
table in memory. For a prefix with several origin ASes, the IP lookup gives
|
|
||||||
the one most peers carry. Its `last_seen` is the newest kept route's time. The
|
|
||||||
prefix page and API list every peer that carries the prefix, with an AS path,
|
|
||||||
next hop and time only for the kept routes. The stats count the origins whose
|
|
||||||
kept routes were all withdrawn, in `origins_without_kept_route` (part of #122)
|
|
||||||
- 2026-10-27: the prefix, AS and peer handlers make a timed write on every tick
|
|
||||||
of their timers, every 1, 2 and 2 seconds. They used to skip every other tick.
|
|
||||||
A full batch's write restarts the timer, so the next timed write comes one
|
|
||||||
interval after that write began (closes #192)
|
|
||||||
- 2026-10-27: once the stop deadline has passed, the AS, peer and peering
|
|
||||||
handlers start no write either. A full batch or a tick of the timer that
|
|
||||||
waited while a write ran past the deadline could still start one. The flush
|
|
||||||
goroutines of the prefix, AS and peer handlers now run the same code (closes
|
|
||||||
#189)
|
|
||||||
- 2026-10-27: once the stop deadline has passed, the prefix handler starts no
|
|
||||||
flush. A batch that filled while a write ran past the deadline could still be
|
|
||||||
written after it (closes #188)
|
|
||||||
- 2026-10-27: the prefix handler applies each update to the routing table in
|
|
||||||
memory, and writes the table's changes to the database every second. A write
|
|
||||||
takes at most 20,000 changed pairs, and the rest follow in the next writes. A
|
|
||||||
failed write is written again later. Startup reads the table before the
|
|
||||||
streamer connects. The pages still read the per-peer route tables (part of
|
|
||||||
#122)
|
|
||||||
- 2026-10-27: the database keeps a copy of the whole routing table in a new
|
|
||||||
`routes` table, one row per prefix and origin AS. The peers' numbers are in a
|
|
||||||
new `route_peers` table. One function writes the pairs that changed in memory,
|
|
||||||
and another reads the whole table back into memory at startup. Nothing uses
|
|
||||||
them yet (part of #122)
|
|
||||||
- 2026-10-27: a new package, `internal/routetable`, holds the whole routing
|
|
||||||
table in memory. For each prefix and origin AS it keeps the peers that carry
|
|
||||||
it, one bit per peer, and the two newest of their routes. Nothing uses it yet
|
|
||||||
(part of #122)
|
|
||||||
- 2026-10-27: each result of the IP lookup lists the public blocklists the
|
|
||||||
address is on, in `blocklists`. The lists are DShield, Feodo Tracker, two
|
|
||||||
FireHOL lists, Spamhaus DROP and the Tor exit list. RouteWatch downloads each
|
|
||||||
list on its own schedule and answers from copies in memory. The copies are
|
|
||||||
also kept in the state directory's `blocklists` directory. IP lookup answers
|
|
||||||
carry `Cache-Control`. An address with no live route now has a result too,
|
|
||||||
without route fields, so the IP lookup no longer answers 404 (closes #158)
|
|
||||||
- 2026-10-27: the home page has an IP information API card. It shows the `curl`
|
|
||||||
command that looks up the visitor's own address at `/ip/{addr}`, and that
|
|
||||||
command's live answer (closes #156)
|
|
||||||
- 2026-10-27: on the AS page, a click on the heading of the IPv4 or the IPv6
|
|
||||||
prefix list folds or unfolds the list. Both lists start open (closes #155)
|
|
||||||
- 2026-10-27: the AS page lists its IPv4 and its IPv6 prefixes by mask length,
|
|
||||||
shortest first, and prefixes of one mask length by address. They used to be
|
|
||||||
listed by address (closes #153)
|
|
||||||
- 2026-10-27: on `/` and `/status`, a spinner beside the status turns one frame
|
|
||||||
each time the stats show more messages than the time before. A page that has
|
|
||||||
not seen more messages for three refreshes shows "stale" and the time since
|
|
||||||
then instead, in red (closes #152)
|
|
||||||
- 2026-10-27: the WHOIS details of a Brazilian AS keep their accented letters.
|
|
||||||
LACNIC's answer for such an AS comes from NIC.br, Brazil's registry, with each
|
|
||||||
accented letter replaced. The details now come from NIC.br's own server,
|
|
||||||
`whois.registro.br`. When that query fails, the details from LACNIC's answer
|
|
||||||
are kept and a warning is logged. After NIC.br answers that RouteWatch is over
|
|
||||||
its query limit, RouteWatch stops asking NIC.br for an hour (closes #174)
|
|
||||||
- 2026-10-27: `make asupdate` stops with an error when the download of `as.csv`
|
|
||||||
breaks off. The generator used to retry the failed read forever. A record that
|
|
||||||
does not parse is still logged and skipped (closes #177)
|
|
||||||
- 2026-10-27: `make asupdate` runs the new `script/asupdate`. It replaces the AS
|
|
||||||
list built into RouteWatch only when the generator succeeds and finds at least
|
|
||||||
one AS. A failed download used to replace the list with an empty file. The
|
|
||||||
generator reads `as.csv` from `ipverse/as-metadata`, the source's new name
|
|
||||||
(closes #173)
|
|
||||||
- 2026-10-27: the WHOIS details of an AS at RIPE, APNIC or AFRINIC come only
|
|
||||||
from its `aut-num` object and the organisation object it names. RIPE and APNIC
|
|
||||||
start their answer with the `as-block` object of the AS's block. Its dates
|
|
||||||
were stored as the AS's (closes #168)
|
|
||||||
- 2026-10-27: AS names keep their accented letters. The AS list built into
|
|
||||||
RouteWatch is rebuilt from today's `as.csv`, which added a country code
|
|
||||||
column. That file no longer lists AS 0 or the documentation AS numbers. A
|
|
||||||
WHOIS answer that is not valid UTF-8 is decoded as Latin-1, which RIPE and
|
|
||||||
APNIC send, before it is parsed and stored (closes #165)
|
|
||||||
- 2026-10-27: the Age column on the AS and prefix pages reads in words, such as
|
|
||||||
"5 minutes ago" or "2 weeks ago", instead of Go durations such as "360h0m0s
|
|
||||||
ago". An age under a minute reads in seconds, and one over a month in months
|
|
||||||
or years instead of a date (closes #163)
|
|
||||||
- 2026-10-26: a write to a stored AS, peering or peer keeps the earlier of the
|
|
||||||
two first-seen times and the later of the two last-seen times. An old update
|
|
||||||
from a collector catching up on a backlog no longer moves last seen back
|
|
||||||
(closes #162)
|
|
||||||
- 2026-10-26: the status page shows "Route Churn/min", the number of prefix
|
- 2026-10-26: the status page shows "Route Churn/min", the number of prefix
|
||||||
announcements and withdrawals, from all peers, in the last full clock minute.
|
announcements and withdrawals, from all peers, in the last full clock minute.
|
||||||
`/api/v1/stats` reports it as `route_churn_per_min` in `stream`, which
|
`/api/v1/stats` reports it as `route_churn_per_min` in `stream`, which
|
||||||
@@ -286,4 +194,4 @@ tested.
|
|||||||
|
|
||||||
# Future Steps
|
# Future Steps
|
||||||
|
|
||||||
None.
|
None open.
|
||||||
|
|||||||
+28
-60
@@ -6,7 +6,6 @@ import (
|
|||||||
"encoding/csv"
|
"encoding/csv"
|
||||||
"encoding/json"
|
"encoding/json"
|
||||||
"errors"
|
"errors"
|
||||||
"fmt"
|
|
||||||
"io"
|
"io"
|
||||||
"log"
|
"log"
|
||||||
"net/http"
|
"net/http"
|
||||||
@@ -15,21 +14,7 @@ import (
|
|||||||
"strings"
|
"strings"
|
||||||
)
|
)
|
||||||
|
|
||||||
const asCsvURL = "https://github.com/ipverse/as-metadata/raw/refs/heads/master/as.csv"
|
const asCsvURL = "https://github.com/ipverse/asn-info/raw/refs/heads/master/as.csv"
|
||||||
|
|
||||||
// The first columns of as.csv, the ones the generator reads. It reads past
|
|
||||||
// the columns after them, such as the country code.
|
|
||||||
const (
|
|
||||||
asnColumn = "asn"
|
|
||||||
handleColumn = "handle"
|
|
||||||
descriptionColumn = "description"
|
|
||||||
columnsRead = 3
|
|
||||||
)
|
|
||||||
|
|
||||||
var (
|
|
||||||
errUnexpectedHeader = errors.New("unexpected CSV header")
|
|
||||||
errNoASes = errors.New("no AS in the CSV")
|
|
||||||
)
|
|
||||||
|
|
||||||
// ASInfo represents information about an Autonomous System
|
// ASInfo represents information about an Autonomous System
|
||||||
type ASInfo struct {
|
type ASInfo struct {
|
||||||
@@ -60,66 +45,40 @@ func main() {
|
|||||||
log.Fatalf("Failed to fetch CSV: HTTP %d", resp.StatusCode)
|
log.Fatalf("Failed to fetch CSV: HTTP %d", resp.StatusCode)
|
||||||
}
|
}
|
||||||
|
|
||||||
asInfos, err := parseCSV(resp.Body)
|
// Parse CSV
|
||||||
|
reader := csv.NewReader(resp.Body)
|
||||||
// Closed here rather than deferred: log.Fatalf exits without running
|
|
||||||
// deferred calls.
|
|
||||||
_ = resp.Body.Close()
|
|
||||||
|
|
||||||
if err != nil {
|
|
||||||
log.Fatalf("Failed to parse CSV: %v", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
log.Printf("Parsed %d AS records", len(asInfos))
|
|
||||||
|
|
||||||
// Convert to JSON and output to stdout
|
|
||||||
encoder := json.NewEncoder(os.Stdout)
|
|
||||||
encoder.SetIndent("", "\t")
|
|
||||||
|
|
||||||
err = encoder.Encode(asInfos)
|
|
||||||
if err != nil {
|
|
||||||
log.Fatalf("Failed to encode JSON: %v", err)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// parseCSV reads as.csv from r and returns its ASes, with the handle and the
|
|
||||||
// description as the file spells them. A header whose first columns are not
|
|
||||||
// the AS number, the handle and the description is an error. A record that
|
|
||||||
// does not parse is logged and skipped. A read that fails, such as on a broken
|
|
||||||
// download, is an error. A file left with no AS is an error.
|
|
||||||
func parseCSV(r io.Reader) ([]ASInfo, error) {
|
|
||||||
reader := csv.NewReader(r)
|
|
||||||
|
|
||||||
|
// Read header
|
||||||
header, err := reader.Read()
|
header, err := reader.Read()
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return nil, fmt.Errorf("read CSV header: %w", err)
|
log.Fatalf("Failed to read CSV header: %v", err)
|
||||||
}
|
}
|
||||||
|
|
||||||
if len(header) < columnsRead || header[0] != asnColumn ||
|
if len(header) != 3 || header[0] != "asn" || header[1] != "handle" ||
|
||||||
header[1] != handleColumn || header[2] != descriptionColumn {
|
header[2] != "description" {
|
||||||
return nil, fmt.Errorf("%w: %v", errUnexpectedHeader, header)
|
log.Fatalf("Unexpected CSV header: %v", header)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// Read all records
|
||||||
var asInfos []ASInfo
|
var asInfos []ASInfo
|
||||||
|
|
||||||
for {
|
for {
|
||||||
// The reader returns an error for a record with a different number
|
|
||||||
// of fields than the header.
|
|
||||||
record, err := reader.Read()
|
record, err := reader.Read()
|
||||||
if errors.Is(err, io.EOF) {
|
if errors.Is(err, io.EOF) {
|
||||||
break
|
break
|
||||||
}
|
}
|
||||||
|
|
||||||
if _, ok := errors.AsType[*csv.ParseError](err); ok {
|
if err != nil {
|
||||||
log.Printf("Error reading CSV record: %v", err)
|
log.Printf("Error reading CSV record: %v", err)
|
||||||
|
|
||||||
continue
|
continue
|
||||||
}
|
}
|
||||||
|
|
||||||
// Any other error comes from the download, such as a broken
|
const expectedFields = 3
|
||||||
// connection. Every later read returns it again.
|
if len(record) != expectedFields {
|
||||||
if err != nil {
|
log.Printf("Skipping invalid record: %v", record)
|
||||||
return nil, fmt.Errorf("read CSV record: %w", err)
|
|
||||||
|
continue
|
||||||
}
|
}
|
||||||
|
|
||||||
asn, err := strconv.Atoi(record[0])
|
asn, err := strconv.Atoi(record[0])
|
||||||
@@ -136,9 +95,18 @@ func parseCSV(r io.Reader) ([]ASInfo, error) {
|
|||||||
})
|
})
|
||||||
}
|
}
|
||||||
|
|
||||||
if len(asInfos) == 0 {
|
// Closed here rather than deferred: log.Fatalf exits without running
|
||||||
return nil, errNoASes
|
// deferred calls.
|
||||||
}
|
_ = resp.Body.Close()
|
||||||
|
|
||||||
return asInfos, nil
|
log.Printf("Parsed %d AS records", len(asInfos))
|
||||||
|
|
||||||
|
// Convert to JSON and output to stdout
|
||||||
|
encoder := json.NewEncoder(os.Stdout)
|
||||||
|
encoder.SetIndent("", "\t")
|
||||||
|
|
||||||
|
err = encoder.Encode(asInfos)
|
||||||
|
if err != nil {
|
||||||
|
log.Fatalf("Failed to encode JSON: %v", err)
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -1,142 +0,0 @@
|
|||||||
package main
|
|
||||||
|
|
||||||
import (
|
|
||||||
"errors"
|
|
||||||
"io"
|
|
||||||
"slices"
|
|
||||||
"strings"
|
|
||||||
"testing"
|
|
||||||
"testing/iotest"
|
|
||||||
"time"
|
|
||||||
)
|
|
||||||
|
|
||||||
// parseTimeLimit is how long a test waits for parseCSV to return before it
|
|
||||||
// fails.
|
|
||||||
const parseTimeLimit = 2 * time.Second
|
|
||||||
|
|
||||||
// TestParseCSVReadsTodaysLayout parses an as.csv with the country code column
|
|
||||||
// that the source added after the description. It expects each AS with its
|
|
||||||
// handle and description spelled exactly as in the file, accents included.
|
|
||||||
func TestParseCSVReadsTodaysLayout(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
const asCSV = `asn,handle,description,country-code
|
|
||||||
1,LVLT-1,Level 3 Parent LLC,US
|
|
||||||
1251,FUNDACAO-AMPARO-PESQUISA,FUNDAÇÃO DE AMPARO À PESQUISA DO ESTADO SÃO PAULO,BR
|
|
||||||
`
|
|
||||||
|
|
||||||
got, err := parseCSV(strings.NewReader(asCSV))
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("parseCSV: %v", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
want := []ASInfo{
|
|
||||||
{ASN: 1, Handle: "LVLT-1", Description: "Level 3 Parent LLC"},
|
|
||||||
{
|
|
||||||
ASN: 1251,
|
|
||||||
Handle: "FUNDACAO-AMPARO-PESQUISA",
|
|
||||||
Description: "FUNDAÇÃO DE AMPARO À PESQUISA DO ESTADO " +
|
|
||||||
"SÃO PAULO",
|
|
||||||
},
|
|
||||||
}
|
|
||||||
|
|
||||||
if !slices.Equal(got, want) {
|
|
||||||
t.Errorf("parseCSV = %+v, want %+v", got, want)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestParseCSVRejectsAnotherHeader parses files whose first columns are not
|
|
||||||
// the AS number, the handle and the description. It expects an error rather
|
|
||||||
// than a list read from the wrong columns.
|
|
||||||
func TestParseCSVRejectsAnotherHeader(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
for _, asCSV := range []string{
|
|
||||||
"asn,handle\n1,LVLT-1\n",
|
|
||||||
"asn,description,handle\n1,Level 3 Parent LLC,LVLT-1\n",
|
|
||||||
} {
|
|
||||||
_, err := parseCSV(strings.NewReader(asCSV))
|
|
||||||
if err == nil {
|
|
||||||
t.Errorf("parseCSV(%q): no error", asCSV)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestParseCSVRejectsAFileWithNoASes parses files that hold no AS, one with
|
|
||||||
// only the header and one whose only record has no valid AS number. It
|
|
||||||
// expects errNoASes, so that make asupdate keeps the old list.
|
|
||||||
func TestParseCSVRejectsAFileWithNoASes(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
for _, asCSV := range []string{
|
|
||||||
"asn,handle,description,country-code\n",
|
|
||||||
"asn,handle,description,country-code\nAS1,LVLT-1,Level 3,US\n",
|
|
||||||
} {
|
|
||||||
_, err := parseCSV(strings.NewReader(asCSV))
|
|
||||||
if !errors.Is(err, errNoASes) {
|
|
||||||
t.Errorf("parseCSV(%q) = %v, want %v", asCSV, err, errNoASes)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestParseCSVSkipsARecordThatDoesNotParse parses an as.csv with a record
|
|
||||||
// that has too few fields between two valid records. It expects both valid
|
|
||||||
// ASes and no error, so that one bad record does not stop make asupdate.
|
|
||||||
func TestParseCSVSkipsARecordThatDoesNotParse(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
const asCSV = "asn,handle,description,country-code\n" +
|
|
||||||
"1,LVLT-1,Level 3 Parent LLC,US\n" +
|
|
||||||
"2,UDEL-DCN\n" +
|
|
||||||
"3,MIT-GATEWAYS,Massachusetts Institute of Technology,US\n"
|
|
||||||
|
|
||||||
got, err := parseCSV(strings.NewReader(asCSV))
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("parseCSV: %v", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
want := []ASInfo{
|
|
||||||
{ASN: 1, Handle: "LVLT-1", Description: "Level 3 Parent LLC"},
|
|
||||||
{
|
|
||||||
ASN: 3,
|
|
||||||
Handle: "MIT-GATEWAYS",
|
|
||||||
Description: "Massachusetts Institute of Technology",
|
|
||||||
},
|
|
||||||
}
|
|
||||||
|
|
||||||
if !slices.Equal(got, want) {
|
|
||||||
t.Errorf("parseCSV = %+v, want %+v", got, want)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestParseCSVStopsWhenTheDownloadBreaksOff parses an as.csv whose download
|
|
||||||
// breaks off after the first record. From then on every read fails, as on a
|
|
||||||
// broken connection. It expects parseCSV to return that error, so that make
|
|
||||||
// asupdate keeps the old list.
|
|
||||||
func TestParseCSVStopsWhenTheDownloadBreaksOff(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
const asCSV = "asn,handle,description,country-code\n" +
|
|
||||||
"1,LVLT-1,Level 3 Parent LLC,US\n"
|
|
||||||
|
|
||||||
download := io.MultiReader(
|
|
||||||
strings.NewReader(asCSV), iotest.ErrReader(io.ErrUnexpectedEOF))
|
|
||||||
|
|
||||||
// parseCSV runs on its own goroutine, so that a parseCSV that never
|
|
||||||
// returns fails the test instead of hanging it.
|
|
||||||
parsed := make(chan error, 1)
|
|
||||||
|
|
||||||
go func() {
|
|
||||||
_, err := parseCSV(download)
|
|
||||||
parsed <- err
|
|
||||||
}()
|
|
||||||
|
|
||||||
select {
|
|
||||||
case err := <-parsed:
|
|
||||||
if !errors.Is(err, io.ErrUnexpectedEOF) {
|
|
||||||
t.Errorf("parseCSV = %v, want %v", err, io.ErrUnexpectedEOF)
|
|
||||||
}
|
|
||||||
case <-time.After(parseTimeLimit):
|
|
||||||
t.Fatalf("parseCSV did not return within %v", parseTimeLimit)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
@@ -3,17 +3,13 @@ module git.eeqj.de/sneak/routewatch
|
|||||||
go 1.27.1
|
go 1.27.1
|
||||||
|
|
||||||
require (
|
require (
|
||||||
github.com/bits-and-blooms/bitset v1.26.0
|
|
||||||
github.com/dustin/go-humanize v1.0.1
|
github.com/dustin/go-humanize v1.0.1
|
||||||
github.com/go-chi/chi/v5 v5.2.2
|
github.com/go-chi/chi/v5 v5.2.2
|
||||||
github.com/google/btree v1.1.3
|
|
||||||
github.com/google/uuid v1.6.0
|
github.com/google/uuid v1.6.0
|
||||||
github.com/mattn/go-sqlite3 v1.14.29
|
github.com/mattn/go-sqlite3 v1.14.29
|
||||||
github.com/rcrowley/go-metrics v0.0.0-20250401214520-65e299d6c5c9
|
github.com/rcrowley/go-metrics v0.0.0-20250401214520-65e299d6c5c9
|
||||||
go.uber.org/fx v1.24.0
|
go.uber.org/fx v1.24.0
|
||||||
go4.org/netipx v0.0.0-20260823151212-3075585bcbeb
|
|
||||||
golang.org/x/term v0.33.0
|
golang.org/x/term v0.33.0
|
||||||
golang.org/x/text v0.43.0
|
|
||||||
)
|
)
|
||||||
|
|
||||||
require (
|
require (
|
||||||
|
|||||||
@@ -1,13 +1,9 @@
|
|||||||
github.com/bits-and-blooms/bitset v1.26.0 h1:WPvcPZbP1yuKofmjs2SYIFO1EjzIGhCP9ducElLsh5Q=
|
|
||||||
github.com/bits-and-blooms/bitset v1.26.0/go.mod h1:7hO7Gc7Pp1vODcmWvKMRA9BNmbv6a/7QIWpPxHddWR8=
|
|
||||||
github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c=
|
github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c=
|
||||||
github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
|
github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
|
||||||
github.com/dustin/go-humanize v1.0.1 h1:GzkhY7T5VNhEkwH0PVJgjz+fX1rhBrR7pRT3mDkpeCY=
|
github.com/dustin/go-humanize v1.0.1 h1:GzkhY7T5VNhEkwH0PVJgjz+fX1rhBrR7pRT3mDkpeCY=
|
||||||
github.com/dustin/go-humanize v1.0.1/go.mod h1:Mu1zIs6XwVuF/gI1OepvI0qD18qycQx+mFykh5fBlto=
|
github.com/dustin/go-humanize v1.0.1/go.mod h1:Mu1zIs6XwVuF/gI1OepvI0qD18qycQx+mFykh5fBlto=
|
||||||
github.com/go-chi/chi/v5 v5.2.2 h1:CMwsvRVTbXVytCk1Wd72Zy1LAsAh9GxMmSNWLHCG618=
|
github.com/go-chi/chi/v5 v5.2.2 h1:CMwsvRVTbXVytCk1Wd72Zy1LAsAh9GxMmSNWLHCG618=
|
||||||
github.com/go-chi/chi/v5 v5.2.2/go.mod h1:L2yAIGWB3H+phAw1NxKwWM+7eUH/lU8pOMm5hHcoops=
|
github.com/go-chi/chi/v5 v5.2.2/go.mod h1:L2yAIGWB3H+phAw1NxKwWM+7eUH/lU8pOMm5hHcoops=
|
||||||
github.com/google/btree v1.1.3 h1:CVpQJjYgC4VbzxeGVHfvZrv1ctoYCAI8vbl07Fcxlyg=
|
|
||||||
github.com/google/btree v1.1.3/go.mod h1:qOPhT0dTNdNzV6Z/lhRX0YXUafgPLFUh+gZMl761Gm4=
|
|
||||||
github.com/google/uuid v1.6.0 h1:NIvaJDMOsjHA8n1jAhLSgzrAzy1Hgr+hNrb57e+94F0=
|
github.com/google/uuid v1.6.0 h1:NIvaJDMOsjHA8n1jAhLSgzrAzy1Hgr+hNrb57e+94F0=
|
||||||
github.com/google/uuid v1.6.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo=
|
github.com/google/uuid v1.6.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo=
|
||||||
github.com/mattn/go-sqlite3 v1.14.29 h1:1O6nRLJKvsi1H2Sj0Hzdfojwt8GiGKm+LOfLaBFaouQ=
|
github.com/mattn/go-sqlite3 v1.14.29 h1:1O6nRLJKvsi1H2Sj0Hzdfojwt8GiGKm+LOfLaBFaouQ=
|
||||||
@@ -28,13 +24,9 @@ go.uber.org/multierr v1.10.0 h1:S0h4aNzvfcFsC3dRF1jLoaov7oRaKqRGC/pUEJ2yvPQ=
|
|||||||
go.uber.org/multierr v1.10.0/go.mod h1:20+QtiLqy0Nd6FdQB9TLXag12DsQkrbs3htMFfDN80Y=
|
go.uber.org/multierr v1.10.0/go.mod h1:20+QtiLqy0Nd6FdQB9TLXag12DsQkrbs3htMFfDN80Y=
|
||||||
go.uber.org/zap v1.26.0 h1:sI7k6L95XOKS281NhVKOFCUNIvv9e0w4BF8N3u+tCRo=
|
go.uber.org/zap v1.26.0 h1:sI7k6L95XOKS281NhVKOFCUNIvv9e0w4BF8N3u+tCRo=
|
||||||
go.uber.org/zap v1.26.0/go.mod h1:dtElttAiwGvoJ/vj4IwHBS/gXsEu/pZ50mUIRWuG0so=
|
go.uber.org/zap v1.26.0/go.mod h1:dtElttAiwGvoJ/vj4IwHBS/gXsEu/pZ50mUIRWuG0so=
|
||||||
go4.org/netipx v0.0.0-20260823151212-3075585bcbeb h1:XBM4hvfwGAttkkiTIFfeigdfcL1xIfdKXqFdgiHGtDs=
|
|
||||||
go4.org/netipx v0.0.0-20260823151212-3075585bcbeb/go.mod h1:PLyyIXexvUFg3Owu6p/WfdlivPbZJsZdgWZlrGope/Y=
|
|
||||||
golang.org/x/sys v0.34.0 h1:H5Y5sJ2L2JRdyv7ROF1he/lPdvFsd0mJHFw2ThKHxLA=
|
golang.org/x/sys v0.34.0 h1:H5Y5sJ2L2JRdyv7ROF1he/lPdvFsd0mJHFw2ThKHxLA=
|
||||||
golang.org/x/sys v0.34.0/go.mod h1:BJP2sWEmIv4KK5OTEluFJCKSidICx8ciO85XgH3Ak8k=
|
golang.org/x/sys v0.34.0/go.mod h1:BJP2sWEmIv4KK5OTEluFJCKSidICx8ciO85XgH3Ak8k=
|
||||||
golang.org/x/term v0.33.0 h1:NuFncQrRcaRvVmgRkvM3j/F00gWIAlcmlB8ACEKmGIg=
|
golang.org/x/term v0.33.0 h1:NuFncQrRcaRvVmgRkvM3j/F00gWIAlcmlB8ACEKmGIg=
|
||||||
golang.org/x/term v0.33.0/go.mod h1:s18+ql9tYWp1IfpV9DmCtQDDSRBUjKaw9M1eAv5UeF0=
|
golang.org/x/term v0.33.0/go.mod h1:s18+ql9tYWp1IfpV9DmCtQDDSRBUjKaw9M1eAv5UeF0=
|
||||||
golang.org/x/text v0.43.0 h1:1QivrlhwAsnMOcqgOdl9my7s9OyZl8O5/UGUTJVXoWc=
|
|
||||||
golang.org/x/text v0.43.0/go.mod h1:hbSIYA/amXcRXTFSZqh8tWHEoqRvvbKPJ2uz/KU+a+s=
|
|
||||||
gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA=
|
gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA=
|
||||||
gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
|
gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
|
||||||
|
|||||||
@@ -1,184 +0,0 @@
|
|||||||
// Package blocklist downloads public IP blocklists on a schedule, keeps the
|
|
||||||
// last good copy of each on disk and in memory, and answers which lists
|
|
||||||
// contain an address.
|
|
||||||
package blocklist
|
|
||||||
|
|
||||||
import (
|
|
||||||
"context"
|
|
||||||
"log/slog"
|
|
||||||
"net/netip"
|
|
||||||
"sync"
|
|
||||||
"time"
|
|
||||||
|
|
||||||
"go4.org/netipx"
|
|
||||||
)
|
|
||||||
|
|
||||||
// Format is how a list's file is written.
|
|
||||||
type Format int
|
|
||||||
|
|
||||||
const (
|
|
||||||
// FormatLines has one address or network per line, such as 192.0.2.1 or
|
|
||||||
// 192.0.2.0/24. Lines that start with # are comments.
|
|
||||||
FormatLines Format = iota
|
|
||||||
|
|
||||||
// FormatSpamhausJSON has one JSON object per line, with a network in its
|
|
||||||
// cidr field. The last line holds the file's metadata and no network.
|
|
||||||
FormatSpamhausJSON
|
|
||||||
|
|
||||||
// FormatDShield has one network per line, as tab-separated fields that
|
|
||||||
// start with the network's first and last address. Lines that start
|
|
||||||
// with # are comments.
|
|
||||||
FormatDShield
|
|
||||||
)
|
|
||||||
|
|
||||||
// List is one public blocklist.
|
|
||||||
type List struct {
|
|
||||||
// Name names the list in lookup answers, and names its copy on disk.
|
|
||||||
Name string
|
|
||||||
|
|
||||||
// Description says in one line what being on the list means.
|
|
||||||
Description string
|
|
||||||
|
|
||||||
// URL is where the publisher serves the whole list.
|
|
||||||
URL string
|
|
||||||
|
|
||||||
// Interval is how long a copy is used before the list is downloaded
|
|
||||||
// again.
|
|
||||||
Interval time.Duration
|
|
||||||
|
|
||||||
// Format is how the list's file is written.
|
|
||||||
Format Format
|
|
||||||
}
|
|
||||||
|
|
||||||
// Listing says that a list contains an address.
|
|
||||||
type Listing struct {
|
|
||||||
Name string `json:"name"`
|
|
||||||
Description string `json:"description"`
|
|
||||||
|
|
||||||
// Updated is when routewatch last downloaded the list, or last heard from
|
|
||||||
// its publisher that its copy was still current.
|
|
||||||
Updated time.Time `json:"updated"`
|
|
||||||
}
|
|
||||||
|
|
||||||
// listCopy is a list's last good copy in memory.
|
|
||||||
type listCopy struct {
|
|
||||||
addresses *netipx.IPSet
|
|
||||||
updated time.Time
|
|
||||||
|
|
||||||
// validators identify the version of the list this copy holds.
|
|
||||||
validators validators
|
|
||||||
}
|
|
||||||
|
|
||||||
// Params is what New needs.
|
|
||||||
type Params struct {
|
|
||||||
// Dir is the directory that holds the copies on disk.
|
|
||||||
Dir string
|
|
||||||
|
|
||||||
// Lists are the lists to download and look addresses up in.
|
|
||||||
Lists []List
|
|
||||||
|
|
||||||
Logger *slog.Logger
|
|
||||||
}
|
|
||||||
|
|
||||||
// Blocklists keeps a copy of each list and answers which lists contain an
|
|
||||||
// address. Start loads the copies from disk and starts downloading the lists.
|
|
||||||
type Blocklists struct {
|
|
||||||
dir string
|
|
||||||
lists []List
|
|
||||||
logger *slog.Logger
|
|
||||||
|
|
||||||
// copies holds each list's last good copy, by list name. A list with no
|
|
||||||
// good copy yet has none.
|
|
||||||
mu sync.RWMutex
|
|
||||||
copies map[string]listCopy
|
|
||||||
|
|
||||||
// stop ends the goroutine Start runs, and done is closed when it returns.
|
|
||||||
stop context.CancelFunc
|
|
||||||
done chan struct{}
|
|
||||||
}
|
|
||||||
|
|
||||||
// New returns a Blocklists with no copies. Start loads them.
|
|
||||||
func New(params Params) *Blocklists {
|
|
||||||
return &Blocklists{
|
|
||||||
dir: params.Dir,
|
|
||||||
lists: params.Lists,
|
|
||||||
logger: params.Logger.With("component", "blocklists"),
|
|
||||||
copies: make(map[string]listCopy),
|
|
||||||
done: make(chan struct{}),
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// Lookup returns a Listing for each list whose copy contains addr, in the
|
|
||||||
// order of the lists given to New. It makes no outside request. When no list
|
|
||||||
// contains addr, it returns an empty list, not nil.
|
|
||||||
func (b *Blocklists) Lookup(addr netip.Addr) []Listing {
|
|
||||||
b.mu.RLock()
|
|
||||||
defer b.mu.RUnlock()
|
|
||||||
|
|
||||||
listings := []Listing{}
|
|
||||||
|
|
||||||
for _, list := range b.lists {
|
|
||||||
c, ok := b.copies[list.Name]
|
|
||||||
if ok && c.addresses.Contains(addr.Unmap()) {
|
|
||||||
listings = append(listings, Listing{
|
|
||||||
Name: list.Name,
|
|
||||||
Description: list.Description,
|
|
||||||
Updated: c.updated,
|
|
||||||
})
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
return listings
|
|
||||||
}
|
|
||||||
|
|
||||||
// ShortestInterval returns the shortest Interval of the lists. That is the
|
|
||||||
// soonest a copy can change, and so how long a lookup answer stays current.
|
|
||||||
func (b *Blocklists) ShortestInterval() time.Duration {
|
|
||||||
var shortest time.Duration
|
|
||||||
|
|
||||||
for i, list := range b.lists {
|
|
||||||
if i == 0 || list.Interval < shortest {
|
|
||||||
shortest = list.Interval
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
return shortest
|
|
||||||
}
|
|
||||||
|
|
||||||
// updated returns when the named list's copy was last downloaded or found
|
|
||||||
// current, or the zero time when the list has no copy.
|
|
||||||
func (b *Blocklists) updated(name string) time.Time {
|
|
||||||
b.mu.RLock()
|
|
||||||
defer b.mu.RUnlock()
|
|
||||||
|
|
||||||
return b.copies[name].updated
|
|
||||||
}
|
|
||||||
|
|
||||||
// copyValidators returns the validators of the named list's copy. They are
|
|
||||||
// empty when the list has no copy.
|
|
||||||
func (b *Blocklists) copyValidators(name string) validators {
|
|
||||||
b.mu.RLock()
|
|
||||||
defer b.mu.RUnlock()
|
|
||||||
|
|
||||||
return b.copies[name].validators
|
|
||||||
}
|
|
||||||
|
|
||||||
// setCopy makes c the named list's copy.
|
|
||||||
func (b *Blocklists) setCopy(name string, c listCopy) {
|
|
||||||
b.mu.Lock()
|
|
||||||
defer b.mu.Unlock()
|
|
||||||
|
|
||||||
b.copies[name] = c
|
|
||||||
}
|
|
||||||
|
|
||||||
// setUpdated records that the named list's copy was found current at t.
|
|
||||||
func (b *Blocklists) setUpdated(name string, t time.Time) {
|
|
||||||
b.mu.Lock()
|
|
||||||
defer b.mu.Unlock()
|
|
||||||
|
|
||||||
c, ok := b.copies[name]
|
|
||||||
if ok {
|
|
||||||
c.updated = t
|
|
||||||
b.copies[name] = c
|
|
||||||
}
|
|
||||||
}
|
|
||||||
@@ -1,494 +0,0 @@
|
|||||||
package blocklist
|
|
||||||
|
|
||||||
import (
|
|
||||||
"context"
|
|
||||||
"errors"
|
|
||||||
"io"
|
|
||||||
"io/fs"
|
|
||||||
"log/slog"
|
|
||||||
"net/http"
|
|
||||||
"net/http/httptest"
|
|
||||||
"net/netip"
|
|
||||||
"os"
|
|
||||||
"path/filepath"
|
|
||||||
"strings"
|
|
||||||
"sync"
|
|
||||||
"testing"
|
|
||||||
"time"
|
|
||||||
)
|
|
||||||
|
|
||||||
// The test list's name and description, the address its good version lists,
|
|
||||||
// and the address a newer version lists instead.
|
|
||||||
const (
|
|
||||||
testListName = "test_list"
|
|
||||||
testListDescription = "On the test list"
|
|
||||||
listedAddr = "192.0.2.1"
|
|
||||||
goodListBody = "# A test list\n" + listedAddr + "\n"
|
|
||||||
newListedAddr = "192.0.2.2"
|
|
||||||
newListBody = "# A test list\n" + newListedAddr + "\n"
|
|
||||||
)
|
|
||||||
|
|
||||||
// testETag is the ETag a publisher sends with a version of the test list.
|
|
||||||
const testETag = `"v1"`
|
|
||||||
|
|
||||||
// maxWait is how long a test waits for the download goroutine.
|
|
||||||
const maxWait = 10 * time.Second
|
|
||||||
|
|
||||||
// publisher stands in for a list's publisher. It answers every request with
|
|
||||||
// its status and body, and keeps the headers of each request. A 200 answer
|
|
||||||
// carries lastModified and etag, when set, as Last-Modified and ETag. As
|
|
||||||
// from a real publisher, a request whose If-None-Match or If-Modified-Since
|
|
||||||
// matches them is answered "not modified".
|
|
||||||
type publisher struct {
|
|
||||||
mu sync.Mutex
|
|
||||||
status int
|
|
||||||
body string
|
|
||||||
lastModified time.Time
|
|
||||||
etag string
|
|
||||||
headers []http.Header
|
|
||||||
}
|
|
||||||
|
|
||||||
// newPublisher starts a publisher that answers 200 with body, and returns it
|
|
||||||
// and its URL.
|
|
||||||
func newPublisher(t *testing.T, body string) (*publisher, string) {
|
|
||||||
t.Helper()
|
|
||||||
|
|
||||||
p := &publisher{status: http.StatusOK, body: body}
|
|
||||||
srv := httptest.NewServer(p)
|
|
||||||
t.Cleanup(srv.Close)
|
|
||||||
|
|
||||||
return p, srv.URL
|
|
||||||
}
|
|
||||||
|
|
||||||
func (p *publisher) ServeHTTP(w http.ResponseWriter, r *http.Request) {
|
|
||||||
p.mu.Lock()
|
|
||||||
defer p.mu.Unlock()
|
|
||||||
|
|
||||||
p.headers = append(p.headers, r.Header.Clone())
|
|
||||||
|
|
||||||
if p.status != http.StatusOK {
|
|
||||||
w.WriteHeader(p.status)
|
|
||||||
_, _ = io.WriteString(w, p.body)
|
|
||||||
|
|
||||||
return
|
|
||||||
}
|
|
||||||
|
|
||||||
if p.etag != "" {
|
|
||||||
w.Header().Set(etagHeader, p.etag)
|
|
||||||
}
|
|
||||||
|
|
||||||
// ServeContent sends Last-Modified unless lastModified is zero, and
|
|
||||||
// answers "not modified" when the request's conditions match.
|
|
||||||
http.ServeContent(w, r, testListName, p.lastModified,
|
|
||||||
strings.NewReader(p.body))
|
|
||||||
}
|
|
||||||
|
|
||||||
// answer makes the publisher answer later requests with status and body.
|
|
||||||
func (p *publisher) answer(status int, body string) {
|
|
||||||
p.mu.Lock()
|
|
||||||
defer p.mu.Unlock()
|
|
||||||
|
|
||||||
p.status, p.body = status, body
|
|
||||||
}
|
|
||||||
|
|
||||||
// publish makes the publisher answer later requests with a version of the
|
|
||||||
// list: body, with lastModified and etag. A zero lastModified or an empty
|
|
||||||
// etag is not sent.
|
|
||||||
func (p *publisher) publish(body string, lastModified time.Time, etag string) {
|
|
||||||
p.mu.Lock()
|
|
||||||
defer p.mu.Unlock()
|
|
||||||
|
|
||||||
p.status, p.body = http.StatusOK, body
|
|
||||||
p.lastModified, p.etag = lastModified, etag
|
|
||||||
}
|
|
||||||
|
|
||||||
// requests returns the headers of the requests the publisher has had.
|
|
||||||
func (p *publisher) requests() []http.Header {
|
|
||||||
p.mu.Lock()
|
|
||||||
defer p.mu.Unlock()
|
|
||||||
|
|
||||||
return append([]http.Header(nil), p.headers...)
|
|
||||||
}
|
|
||||||
|
|
||||||
// testList returns a list served at url, written in format.
|
|
||||||
func testList(url string, format Format) List {
|
|
||||||
return List{
|
|
||||||
Name: testListName,
|
|
||||||
Description: testListDescription,
|
|
||||||
URL: url,
|
|
||||||
Interval: time.Hour,
|
|
||||||
Format: format,
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
func newTestBlocklists(dir string, list List) *Blocklists {
|
|
||||||
return New(Params{
|
|
||||||
Dir: dir,
|
|
||||||
Lists: []List{list},
|
|
||||||
Logger: slog.New(slog.DiscardHandler),
|
|
||||||
})
|
|
||||||
}
|
|
||||||
|
|
||||||
// lookup looks addr up in b.
|
|
||||||
func lookup(t *testing.T, b *Blocklists, addr string) []Listing {
|
|
||||||
t.Helper()
|
|
||||||
|
|
||||||
return b.Lookup(netip.MustParseAddr(addr))
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestLookupFindsListedAddresses downloads a list in each format and looks
|
|
||||||
// up addresses on it, inside its networks, and off it, in IPv4 and IPv6.
|
|
||||||
// A listed address has one listing that names the list; any other has an
|
|
||||||
// empty list of listings.
|
|
||||||
func TestLookupFindsListedAddresses(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
tests := []struct {
|
|
||||||
name string
|
|
||||||
format Format
|
|
||||||
body string
|
|
||||||
listed []string
|
|
||||||
unlisted []string
|
|
||||||
}{
|
|
||||||
{
|
|
||||||
name: "single addresses",
|
|
||||||
format: FormatLines,
|
|
||||||
body: "# comment\n192.0.2.1\n\n2001:db8::1\n",
|
|
||||||
listed: []string{"192.0.2.1", "::ffff:192.0.2.1", "2001:db8::1"},
|
|
||||||
unlisted: []string{"192.0.2.2", "2001:db8::2"},
|
|
||||||
},
|
|
||||||
{
|
|
||||||
name: "networks",
|
|
||||||
format: FormatLines,
|
|
||||||
body: "# comment\n198.51.100.0/24\n2001:db8:1::/48\n",
|
|
||||||
listed: []string{"198.51.100.0", "198.51.100.255", "2001:db8:1::5"},
|
|
||||||
unlisted: []string{"198.51.101.0", "2001:db8:2::1"},
|
|
||||||
},
|
|
||||||
{
|
|
||||||
name: "Spamhaus networks",
|
|
||||||
format: FormatSpamhausJSON,
|
|
||||||
body: `{"cidr":"203.0.113.0/25","sblid":"SBL1","rir":"arin"}` + "\n" +
|
|
||||||
`{"cidr":"2001:db8:3::/48","sblid":"SBL2","rir":"ripencc"}` + "\n" +
|
|
||||||
`{"type":"metadata","timestamp":1793074442,"records":2}` + "\n",
|
|
||||||
listed: []string{"203.0.113.1", "2001:db8:3::1"},
|
|
||||||
unlisted: []string{"203.0.113.200", "2001:db8:4::1"},
|
|
||||||
},
|
|
||||||
{
|
|
||||||
name: "DShield networks",
|
|
||||||
format: FormatDShield,
|
|
||||||
body: "# DShield.org Recommended Block List\n" +
|
|
||||||
"192.0.2.0\t192.0.2.255\t24\t327\tEXAMPLE\tUS\tNone\n",
|
|
||||||
listed: []string{"192.0.2.0", "192.0.2.77"},
|
|
||||||
unlisted: []string{"192.0.3.1", "2001:db8::1"},
|
|
||||||
},
|
|
||||||
}
|
|
||||||
|
|
||||||
for _, tt := range tests {
|
|
||||||
t.Run(tt.name, func(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
_, url := newPublisher(t, tt.body)
|
|
||||||
b := newTestBlocklists(t.TempDir(), testList(url, tt.format))
|
|
||||||
|
|
||||||
err := b.download(t.Context(), b.lists[0])
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("download: %v", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
for _, addr := range tt.listed {
|
|
||||||
listings := lookup(t, b, addr)
|
|
||||||
if len(listings) != 1 || listings[0].Name != testListName ||
|
|
||||||
listings[0].Description != testListDescription ||
|
|
||||||
listings[0].Updated.IsZero() {
|
|
||||||
t.Errorf("%s: listings %+v, want one for %s", addr,
|
|
||||||
listings, testListName)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
for _, addr := range tt.unlisted {
|
|
||||||
listings := lookup(t, b, addr)
|
|
||||||
if listings == nil || len(listings) != 0 {
|
|
||||||
t.Errorf("%s: listings %#v, want an empty list", addr, listings)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
})
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestFailedDownloadKeepsThePreviousCopy downloads a good version of a list,
|
|
||||||
// then one that fails. The good copy stays in memory and on disk, and no
|
|
||||||
// part of the failed download is left beside it.
|
|
||||||
func TestFailedDownloadKeepsThePreviousCopy(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
tests := []struct {
|
|
||||||
name string
|
|
||||||
status int
|
|
||||||
body string
|
|
||||||
}{
|
|
||||||
{"server error", http.StatusInternalServerError, "unavailable"},
|
|
||||||
{"not a list", http.StatusOK, "<!DOCTYPE html>\n<html></html>\n"},
|
|
||||||
{
|
|
||||||
"over the size limit", http.StatusOK,
|
|
||||||
strings.Repeat(listedAddr+"\n", maxDownloadSize/len(listedAddr)),
|
|
||||||
},
|
|
||||||
}
|
|
||||||
|
|
||||||
for _, tt := range tests {
|
|
||||||
t.Run(tt.name, func(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
dir := t.TempDir()
|
|
||||||
p, url := newPublisher(t, goodListBody)
|
|
||||||
b := newTestBlocklists(dir, testList(url, FormatLines))
|
|
||||||
|
|
||||||
err := b.download(t.Context(), b.lists[0])
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("first download: %v", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
before := lookup(t, b, listedAddr)
|
|
||||||
|
|
||||||
p.answer(tt.status, tt.body)
|
|
||||||
|
|
||||||
err = b.download(t.Context(), b.lists[0])
|
|
||||||
if err == nil {
|
|
||||||
t.Fatal("second download returned no error")
|
|
||||||
}
|
|
||||||
|
|
||||||
after := lookup(t, b, listedAddr)
|
|
||||||
if len(after) != 1 || len(before) != 1 ||
|
|
||||||
!after[0].Updated.Equal(before[0].Updated) {
|
|
||||||
t.Errorf("listings %+v after the failure, want %+v", after, before)
|
|
||||||
}
|
|
||||||
|
|
||||||
copies := os.DirFS(dir)
|
|
||||||
|
|
||||||
onDisk, err := fs.ReadFile(copies, testListName)
|
|
||||||
if err != nil || string(onDisk) != goodListBody {
|
|
||||||
t.Errorf("copy on disk %q (%v), want %q", onDisk, err, goodListBody)
|
|
||||||
}
|
|
||||||
|
|
||||||
_, err = fs.Stat(copies, testListName+downloadSuffix)
|
|
||||||
if !errors.Is(err, fs.ErrNotExist) {
|
|
||||||
t.Errorf("the failed download's file is still there (%v)", err)
|
|
||||||
}
|
|
||||||
})
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestRestartLoadsCopiesFromDisk downloads a list, then starts a new
|
|
||||||
// Blocklists on the same directory, as a restart does. The new one answers
|
|
||||||
// from the copy on disk at once, and does not download the list again while
|
|
||||||
// the copy is younger than the list's interval.
|
|
||||||
func TestRestartLoadsCopiesFromDisk(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
dir := t.TempDir()
|
|
||||||
p, url := newPublisher(t, goodListBody)
|
|
||||||
list := testList(url, FormatLines)
|
|
||||||
|
|
||||||
first := newTestBlocklists(dir, list)
|
|
||||||
|
|
||||||
err := first.download(t.Context(), list)
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("download: %v", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
downloaded := lookup(t, first, listedAddr)[0].Updated
|
|
||||||
|
|
||||||
second := newTestBlocklists(dir, list)
|
|
||||||
second.Start(t.Context())
|
|
||||||
second.Stop(t.Context())
|
|
||||||
|
|
||||||
listings := lookup(t, second, listedAddr)
|
|
||||||
if len(listings) != 1 ||
|
|
||||||
listings[0].Updated.Sub(downloaded).Abs() > time.Second {
|
|
||||||
t.Errorf("after the restart, listings %+v, want one updated at %s",
|
|
||||||
listings, downloaded)
|
|
||||||
}
|
|
||||||
|
|
||||||
if got := len(p.requests()); got != 1 {
|
|
||||||
t.Errorf("the publisher had %d requests, want 1", got)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestStartRefreshesAnOldCopy starts on a copy older than its list's
|
|
||||||
// interval. The list is downloaded at once, with routewatch's User-Agent and
|
|
||||||
// the Last-Modified and ETag the publisher sent with the copy, kept on disk.
|
|
||||||
// The publisher answers that the copy is still current, so the copy is kept
|
|
||||||
// and marked as updated now.
|
|
||||||
func TestStartRefreshesAnOldCopy(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
dir := t.TempDir()
|
|
||||||
p, url := newPublisher(t, goodListBody)
|
|
||||||
list := testList(url, FormatLines)
|
|
||||||
|
|
||||||
lastModified := time.Now().Add(-3 * list.Interval).UTC().Truncate(time.Second)
|
|
||||||
p.publish(goodListBody, lastModified, testETag)
|
|
||||||
|
|
||||||
err := newTestBlocklists(dir, list).download(t.Context(), list)
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("download: %v", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
old := time.Now().Add(-2 * list.Interval).UTC().Truncate(time.Second)
|
|
||||||
|
|
||||||
err = os.Chtimes(filepath.Join(dir, testListName), old, old)
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("Chtimes: %v", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
b := newTestBlocklists(dir, list)
|
|
||||||
b.Start(t.Context())
|
|
||||||
|
|
||||||
deadline := time.Now().Add(maxWait)
|
|
||||||
for !lookup(t, b, listedAddr)[0].Updated.After(old) {
|
|
||||||
if time.Now().After(deadline) {
|
|
||||||
t.Fatalf("the copy was not marked updated within %s", maxWait)
|
|
||||||
}
|
|
||||||
|
|
||||||
time.Sleep(time.Millisecond)
|
|
||||||
}
|
|
||||||
|
|
||||||
b.Stop(t.Context())
|
|
||||||
|
|
||||||
requests := p.requests()
|
|
||||||
last := requests[len(requests)-1]
|
|
||||||
|
|
||||||
want := lastModified.Format(http.TimeFormat)
|
|
||||||
if got := last.Get(ifModifiedSinceHeader); got != want {
|
|
||||||
t.Errorf("If-Modified-Since %q, want %q", got, want)
|
|
||||||
}
|
|
||||||
|
|
||||||
if got := last.Get(ifNoneMatchHeader); got != testETag {
|
|
||||||
t.Errorf("If-None-Match %q, want %q", got, testETag)
|
|
||||||
}
|
|
||||||
|
|
||||||
if got := last.Get(userAgentHeader); got != userAgent {
|
|
||||||
t.Errorf("User-Agent %q, want %q", got, userAgent)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestNewVersionOlderThanTheLastDownloadReplacesTheCopy downloads a list.
|
|
||||||
// The publisher then puts out a new version whose Last-Modified is earlier
|
|
||||||
// than that download, as when a cache in front of the publisher passes the
|
|
||||||
// version on late. The next download sends the Last-Modified of the copy,
|
|
||||||
// not the time of the download, so it gets the new version.
|
|
||||||
func TestNewVersionOlderThanTheLastDownloadReplacesTheCopy(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
p, url := newPublisher(t, goodListBody)
|
|
||||||
list := testList(url, FormatLines)
|
|
||||||
b := newTestBlocklists(t.TempDir(), list)
|
|
||||||
|
|
||||||
firstVersion := time.Now().Add(-2 * list.Interval).UTC().Truncate(time.Second)
|
|
||||||
p.publish(goodListBody, firstVersion, "")
|
|
||||||
|
|
||||||
err := b.download(t.Context(), list)
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("first download: %v", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
p.publish(newListBody, firstVersion.Add(list.Interval), "")
|
|
||||||
|
|
||||||
err = b.download(t.Context(), list)
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("second download: %v", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
if listings := lookup(t, b, newListedAddr); len(listings) != 1 {
|
|
||||||
t.Errorf("%s: listings %+v, want one from the new version",
|
|
||||||
newListedAddr, listings)
|
|
||||||
}
|
|
||||||
|
|
||||||
want := firstVersion.Format(http.TimeFormat)
|
|
||||||
if got := p.requests()[1].Get(ifModifiedSinceHeader); got != want {
|
|
||||||
t.Errorf("If-Modified-Since %q, want %q", got, want)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestDownloadWithoutLastModifiedOrETagSendsNoCondition downloads a list
|
|
||||||
// twice from a publisher that sends neither Last-Modified nor ETag. Neither
|
|
||||||
// request carries If-Modified-Since or If-None-Match, so the second download
|
|
||||||
// gets the whole list again.
|
|
||||||
func TestDownloadWithoutLastModifiedOrETagSendsNoCondition(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
p, url := newPublisher(t, goodListBody)
|
|
||||||
list := testList(url, FormatLines)
|
|
||||||
b := newTestBlocklists(t.TempDir(), list)
|
|
||||||
|
|
||||||
err := b.download(t.Context(), list)
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("first download: %v", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
err = b.download(t.Context(), list)
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("second download: %v", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
for i, request := range p.requests() {
|
|
||||||
if request.Get(ifModifiedSinceHeader) != "" ||
|
|
||||||
request.Get(ifNoneMatchHeader) != "" {
|
|
||||||
t.Errorf("request %d has a condition: %v", i, request)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestStopCancelsADownloadInProgress stops while the publisher has not
|
|
||||||
// answered. Stop cancels the download and returns once the download
|
|
||||||
// goroutine has stopped.
|
|
||||||
func TestStopCancelsADownloadInProgress(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
arrived := make(chan struct{})
|
|
||||||
|
|
||||||
var once sync.Once
|
|
||||||
|
|
||||||
srv := httptest.NewServer(http.HandlerFunc(
|
|
||||||
func(_ http.ResponseWriter, r *http.Request) {
|
|
||||||
once.Do(func() { close(arrived) })
|
|
||||||
<-r.Context().Done()
|
|
||||||
}))
|
|
||||||
t.Cleanup(srv.Close)
|
|
||||||
|
|
||||||
b := newTestBlocklists(t.TempDir(), testList(srv.URL, FormatLines))
|
|
||||||
b.Start(t.Context())
|
|
||||||
|
|
||||||
select {
|
|
||||||
case <-arrived:
|
|
||||||
case <-time.After(maxWait):
|
|
||||||
t.Fatalf("no download began within %s", maxWait)
|
|
||||||
}
|
|
||||||
|
|
||||||
ctx, cancel := context.WithTimeout(t.Context(), maxWait)
|
|
||||||
defer cancel()
|
|
||||||
|
|
||||||
b.Stop(ctx)
|
|
||||||
|
|
||||||
select {
|
|
||||||
case <-b.done:
|
|
||||||
default:
|
|
||||||
t.Error("Stop returned before the download goroutine stopped")
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestPublicListsAreComplete checks that every public list has what a
|
|
||||||
// download and a lookup answer need, and a unique name.
|
|
||||||
func TestPublicListsAreComplete(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
names := make(map[string]bool)
|
|
||||||
|
|
||||||
for _, list := range Lists() {
|
|
||||||
if list.Name == "" || list.Description == "" || list.URL == "" ||
|
|
||||||
list.Interval <= 0 || names[list.Name] {
|
|
||||||
t.Errorf("list %+v is incomplete or has a name used before", list)
|
|
||||||
}
|
|
||||||
|
|
||||||
names[list.Name] = true
|
|
||||||
}
|
|
||||||
}
|
|
||||||
@@ -1,379 +0,0 @@
|
|||||||
package blocklist
|
|
||||||
|
|
||||||
import (
|
|
||||||
"context"
|
|
||||||
"encoding/json"
|
|
||||||
"errors"
|
|
||||||
"fmt"
|
|
||||||
"io"
|
|
||||||
"io/fs"
|
|
||||||
"net/http"
|
|
||||||
"os"
|
|
||||||
"slices"
|
|
||||||
"time"
|
|
||||||
|
|
||||||
"go4.org/netipx"
|
|
||||||
)
|
|
||||||
|
|
||||||
// Limits of one download.
|
|
||||||
const (
|
|
||||||
// downloadTimeout is how long one download may take.
|
|
||||||
downloadTimeout = time.Minute
|
|
||||||
|
|
||||||
// maxDownloadSize is the size of the largest list file accepted, in
|
|
||||||
// bytes. The largest list, FireHOL abusers, is about 2.5 MiB.
|
|
||||||
maxDownloadSize = 16 << 20
|
|
||||||
)
|
|
||||||
|
|
||||||
// The request headers a download sets.
|
|
||||||
const (
|
|
||||||
userAgentHeader = "User-Agent"
|
|
||||||
ifModifiedSinceHeader = "If-Modified-Since"
|
|
||||||
ifNoneMatchHeader = "If-None-Match"
|
|
||||||
|
|
||||||
// userAgent names routewatch to the publishers.
|
|
||||||
userAgent = "routewatch (+https://git.eeqj.de/sneak/routewatch)"
|
|
||||||
)
|
|
||||||
|
|
||||||
// The response headers that identify the version of a list a publisher sent.
|
|
||||||
const (
|
|
||||||
lastModifiedHeader = "Last-Modified"
|
|
||||||
etagHeader = "ETag"
|
|
||||||
)
|
|
||||||
|
|
||||||
const (
|
|
||||||
// downloadSuffix ends the name of the file a download is written to,
|
|
||||||
// beside the list's copy, until it has parsed.
|
|
||||||
downloadSuffix = ".download"
|
|
||||||
|
|
||||||
// validatorsSuffix ends the name of the file beside a list's copy that
|
|
||||||
// keeps the copy's validators.
|
|
||||||
validatorsSuffix = ".validators"
|
|
||||||
|
|
||||||
// dirPermissions are those of the directory that holds the copies.
|
|
||||||
dirPermissions = 0o750
|
|
||||||
|
|
||||||
// validatorsPermissions are those of the file that keeps a copy's
|
|
||||||
// validators.
|
|
||||||
validatorsPermissions = 0o600
|
|
||||||
)
|
|
||||||
|
|
||||||
// validators are the Last-Modified and ETag headers a publisher sent with a
|
|
||||||
// version of a list. A download sends them back, as If-Modified-Since and
|
|
||||||
// If-None-Match, so that the publisher sends the list only when it has
|
|
||||||
// changed. They are kept as the publisher wrote them.
|
|
||||||
//
|
|
||||||
//nolint:tagliatelle // snake_case keys, as the Go style guide asks
|
|
||||||
type validators struct {
|
|
||||||
LastModified string `json:"last_modified,omitempty"`
|
|
||||||
ETag string `json:"etag,omitempty"`
|
|
||||||
}
|
|
||||||
|
|
||||||
var (
|
|
||||||
errUnexpectedStatus = errors.New("unexpected HTTP status")
|
|
||||||
errTooLarge = errors.New("list is over the size limit")
|
|
||||||
)
|
|
||||||
|
|
||||||
// Start loads each list's copy from disk. It then starts the goroutine that
|
|
||||||
// downloads each list once its copy is older than its Interval, and a list
|
|
||||||
// with no copy at once. The goroutine runs until Stop: the end of ctx does
|
|
||||||
// not stop it.
|
|
||||||
func (b *Blocklists) Start(ctx context.Context) {
|
|
||||||
err := os.MkdirAll(b.dir, dirPermissions)
|
|
||||||
if err != nil {
|
|
||||||
b.logger.Error("Failed to create the blocklists directory",
|
|
||||||
"dir", b.dir, "error", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
nextDownload := make([]time.Time, len(b.lists))
|
|
||||||
|
|
||||||
for i, list := range b.lists {
|
|
||||||
loadErr := b.loadCopy(list)
|
|
||||||
if loadErr != nil && !errors.Is(loadErr, fs.ErrNotExist) {
|
|
||||||
b.logger.Error("Failed to load a blocklist copy",
|
|
||||||
"list", list.Name, "error", loadErr)
|
|
||||||
}
|
|
||||||
|
|
||||||
// A list with no copy has the zero time, so it is due at once.
|
|
||||||
nextDownload[i] = b.updated(list.Name).Add(list.Interval)
|
|
||||||
}
|
|
||||||
|
|
||||||
ctx, b.stop = context.WithCancel(context.WithoutCancel(ctx))
|
|
||||||
|
|
||||||
go func() {
|
|
||||||
b.run(ctx, nextDownload)
|
|
||||||
}()
|
|
||||||
|
|
||||||
b.logger.Info("Blocklist downloads started", "dir", b.dir)
|
|
||||||
}
|
|
||||||
|
|
||||||
// Stop stops the downloads, cancelling one in progress, and waits until the
|
|
||||||
// goroutine has returned or ctx ends. The copies in memory stay, so Lookup
|
|
||||||
// still answers.
|
|
||||||
func (b *Blocklists) Stop(ctx context.Context) {
|
|
||||||
b.stop()
|
|
||||||
|
|
||||||
select {
|
|
||||||
case <-b.done:
|
|
||||||
b.logger.Info("Blocklist downloads stopped")
|
|
||||||
case <-ctx.Done():
|
|
||||||
b.logger.Warn("Stop deadline passed before the blocklist downloads " +
|
|
||||||
"stopped")
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// run downloads each list when nextDownload says, until ctx ends. After each
|
|
||||||
// try, whether it worked or not, the list's next download is an Interval
|
|
||||||
// later.
|
|
||||||
func (b *Blocklists) run(ctx context.Context, nextDownload []time.Time) {
|
|
||||||
defer close(b.done)
|
|
||||||
|
|
||||||
if len(nextDownload) == 0 {
|
|
||||||
return
|
|
||||||
}
|
|
||||||
|
|
||||||
for {
|
|
||||||
soonest := slices.MinFunc(nextDownload, time.Time.Compare)
|
|
||||||
timer := time.NewTimer(time.Until(soonest))
|
|
||||||
|
|
||||||
select {
|
|
||||||
case <-ctx.Done():
|
|
||||||
timer.Stop()
|
|
||||||
|
|
||||||
return
|
|
||||||
case <-timer.C:
|
|
||||||
}
|
|
||||||
|
|
||||||
for i, list := range b.lists {
|
|
||||||
if time.Now().Before(nextDownload[i]) {
|
|
||||||
continue
|
|
||||||
}
|
|
||||||
|
|
||||||
err := b.download(ctx, list)
|
|
||||||
if ctx.Err() != nil {
|
|
||||||
return
|
|
||||||
}
|
|
||||||
|
|
||||||
if err != nil {
|
|
||||||
b.logger.Error("Blocklist download failed; keeping the "+
|
|
||||||
"previous copy", "list", list.Name, "error", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
nextDownload[i] = time.Now().Add(list.Interval)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// download asks the publisher for list, sending the validators of the list's
|
|
||||||
// copy when it has them. A new version that parses becomes the list's copy,
|
|
||||||
// on disk and in memory. An answer that the copy is still current marks the
|
|
||||||
// copy as updated now. Anything else leaves the copy as it is and returns an
|
|
||||||
// error.
|
|
||||||
func (b *Blocklists) download(ctx context.Context, list List) error {
|
|
||||||
ctx, cancel := context.WithTimeout(ctx, downloadTimeout)
|
|
||||||
defer cancel()
|
|
||||||
|
|
||||||
req, err := http.NewRequestWithContext(ctx, http.MethodGet, list.URL, nil)
|
|
||||||
if err != nil {
|
|
||||||
return err
|
|
||||||
}
|
|
||||||
|
|
||||||
req.Header.Set(userAgentHeader, userAgent)
|
|
||||||
|
|
||||||
held := b.copyValidators(list.Name)
|
|
||||||
if held.LastModified != "" {
|
|
||||||
req.Header.Set(ifModifiedSinceHeader, held.LastModified)
|
|
||||||
}
|
|
||||||
|
|
||||||
if held.ETag != "" {
|
|
||||||
req.Header.Set(ifNoneMatchHeader, held.ETag)
|
|
||||||
}
|
|
||||||
|
|
||||||
resp, err := http.DefaultClient.Do(req)
|
|
||||||
if err != nil {
|
|
||||||
return err
|
|
||||||
}
|
|
||||||
|
|
||||||
defer func() { _ = resp.Body.Close() }()
|
|
||||||
|
|
||||||
switch resp.StatusCode {
|
|
||||||
case http.StatusOK:
|
|
||||||
return b.replaceCopy(list, resp.Body, validators{
|
|
||||||
LastModified: resp.Header.Get(lastModifiedHeader),
|
|
||||||
ETag: resp.Header.Get(etagHeader),
|
|
||||||
})
|
|
||||||
case http.StatusNotModified:
|
|
||||||
return b.markCurrent(list)
|
|
||||||
default:
|
|
||||||
return fmt.Errorf("%w: %s", errUnexpectedStatus, resp.Status)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// replaceCopy writes a new version of list, read from body, beside the list's
|
|
||||||
// copy. Only when it parses does it replace the copy, on disk and in memory.
|
|
||||||
// sent are the validators the publisher sent with the new version.
|
|
||||||
func (b *Blocklists) replaceCopy(list List, body io.Reader, sent validators) error {
|
|
||||||
root, err := os.OpenRoot(b.dir)
|
|
||||||
if err != nil {
|
|
||||||
return err
|
|
||||||
}
|
|
||||||
|
|
||||||
defer func() { _ = root.Close() }()
|
|
||||||
|
|
||||||
downloadName := list.Name + downloadSuffix
|
|
||||||
|
|
||||||
addresses, err := writeAndParse(root, downloadName, body, list.Format)
|
|
||||||
if err != nil {
|
|
||||||
_ = root.Remove(downloadName)
|
|
||||||
|
|
||||||
return err
|
|
||||||
}
|
|
||||||
|
|
||||||
// The old copy's validators are removed before the copy is replaced, so
|
|
||||||
// that they are never kept with the new copy.
|
|
||||||
err = root.Remove(list.Name + validatorsSuffix)
|
|
||||||
if err != nil && !errors.Is(err, fs.ErrNotExist) {
|
|
||||||
return err
|
|
||||||
}
|
|
||||||
|
|
||||||
err = root.Rename(downloadName, list.Name)
|
|
||||||
if err != nil {
|
|
||||||
return err
|
|
||||||
}
|
|
||||||
|
|
||||||
b.setCopy(list.Name, listCopy{
|
|
||||||
addresses: addresses, updated: time.Now().UTC(), validators: sent,
|
|
||||||
})
|
|
||||||
b.logger.Info("Downloaded a new version of a blocklist", "list", list.Name)
|
|
||||||
|
|
||||||
// Without this file, the first download after a restart asks for the
|
|
||||||
// whole list.
|
|
||||||
err = writeValidators(root, list.Name, sent)
|
|
||||||
if err != nil {
|
|
||||||
b.logger.Error("Failed to keep the validators of a blocklist copy",
|
|
||||||
"list", list.Name, "error", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
return nil
|
|
||||||
}
|
|
||||||
|
|
||||||
// writeAndParse writes body to the file name in root, up to maxDownloadSize
|
|
||||||
// bytes, and parses the file as format.
|
|
||||||
func writeAndParse(
|
|
||||||
root *os.Root, name string, body io.Reader, format Format,
|
|
||||||
) (*netipx.IPSet, error) {
|
|
||||||
f, err := root.Create(name)
|
|
||||||
if err != nil {
|
|
||||||
return nil, err
|
|
||||||
}
|
|
||||||
|
|
||||||
defer func() { _ = f.Close() }()
|
|
||||||
|
|
||||||
written, err := io.Copy(f, io.LimitReader(body, maxDownloadSize+1))
|
|
||||||
if err != nil {
|
|
||||||
return nil, err
|
|
||||||
}
|
|
||||||
|
|
||||||
if written > maxDownloadSize {
|
|
||||||
return nil, errTooLarge
|
|
||||||
}
|
|
||||||
|
|
||||||
_, err = f.Seek(0, io.SeekStart)
|
|
||||||
if err != nil {
|
|
||||||
return nil, err
|
|
||||||
}
|
|
||||||
|
|
||||||
return parse(f, format)
|
|
||||||
}
|
|
||||||
|
|
||||||
// markCurrent records that the publisher says the list's copy is still
|
|
||||||
// current: the copy, on disk and in memory, is marked as updated now.
|
|
||||||
func (b *Blocklists) markCurrent(list List) error {
|
|
||||||
root, err := os.OpenRoot(b.dir)
|
|
||||||
if err != nil {
|
|
||||||
return err
|
|
||||||
}
|
|
||||||
|
|
||||||
defer func() { _ = root.Close() }()
|
|
||||||
|
|
||||||
now := time.Now().UTC()
|
|
||||||
|
|
||||||
err = root.Chtimes(list.Name, now, now)
|
|
||||||
if err != nil {
|
|
||||||
return err
|
|
||||||
}
|
|
||||||
|
|
||||||
b.setUpdated(list.Name, now)
|
|
||||||
b.logger.Debug("Blocklist copy is still current", "list", list.Name)
|
|
||||||
|
|
||||||
return nil
|
|
||||||
}
|
|
||||||
|
|
||||||
// loadCopy makes the list's copy on disk, with the validators kept beside it,
|
|
||||||
// its copy in memory. The file's modification time is when the copy was
|
|
||||||
// updated.
|
|
||||||
func (b *Blocklists) loadCopy(list List) error {
|
|
||||||
root, err := os.OpenRoot(b.dir)
|
|
||||||
if err != nil {
|
|
||||||
return err
|
|
||||||
}
|
|
||||||
|
|
||||||
defer func() { _ = root.Close() }()
|
|
||||||
|
|
||||||
f, err := root.Open(list.Name)
|
|
||||||
if err != nil {
|
|
||||||
return err
|
|
||||||
}
|
|
||||||
|
|
||||||
defer func() { _ = f.Close() }()
|
|
||||||
|
|
||||||
info, err := f.Stat()
|
|
||||||
if err != nil {
|
|
||||||
return err
|
|
||||||
}
|
|
||||||
|
|
||||||
addresses, err := parse(f, list.Format)
|
|
||||||
if err != nil {
|
|
||||||
return err
|
|
||||||
}
|
|
||||||
|
|
||||||
held, err := readValidators(root, list.Name)
|
|
||||||
if err != nil {
|
|
||||||
return err
|
|
||||||
}
|
|
||||||
|
|
||||||
b.setCopy(list.Name, listCopy{
|
|
||||||
addresses: addresses, updated: info.ModTime().UTC(), validators: held,
|
|
||||||
})
|
|
||||||
|
|
||||||
return nil
|
|
||||||
}
|
|
||||||
|
|
||||||
// writeValidators keeps held in the file beside the named list's copy.
|
|
||||||
func writeValidators(root *os.Root, name string, held validators) error {
|
|
||||||
data, err := json.Marshal(held)
|
|
||||||
if err != nil {
|
|
||||||
return err
|
|
||||||
}
|
|
||||||
|
|
||||||
return root.WriteFile(name+validatorsSuffix, data, validatorsPermissions)
|
|
||||||
}
|
|
||||||
|
|
||||||
// readValidators returns the validators kept beside the named list's copy.
|
|
||||||
// They are empty when there is no such file.
|
|
||||||
func readValidators(root *os.Root, name string) (validators, error) {
|
|
||||||
var held validators
|
|
||||||
|
|
||||||
data, err := root.ReadFile(name + validatorsSuffix)
|
|
||||||
if errors.Is(err, fs.ErrNotExist) {
|
|
||||||
return held, nil
|
|
||||||
}
|
|
||||||
|
|
||||||
if err != nil {
|
|
||||||
return held, err
|
|
||||||
}
|
|
||||||
|
|
||||||
err = json.Unmarshal(data, &held)
|
|
||||||
|
|
||||||
return held, err
|
|
||||||
}
|
|
||||||
@@ -1,93 +0,0 @@
|
|||||||
package blocklist
|
|
||||||
|
|
||||||
import "time"
|
|
||||||
|
|
||||||
// How often each list is downloaded. No list is downloaded more often than
|
|
||||||
// its publisher updates it or allows.
|
|
||||||
const (
|
|
||||||
// DShield asks that its feeds be downloaded at most once an hour:
|
|
||||||
// https://isc.sans.edu/feeds_doc.html
|
|
||||||
dshieldInterval = time.Hour
|
|
||||||
|
|
||||||
// abuse.ch makes the list every 5 minutes and recommends a download at
|
|
||||||
// least every 15: https://feodotracker.abuse.ch/blocklist/
|
|
||||||
feodoTrackerInterval = 15 * time.Minute
|
|
||||||
|
|
||||||
// FireHOL's page for each list gives how often the list changes on
|
|
||||||
// average: about every 5 hours for abusers and every 3 hours for level
|
|
||||||
// 3. FireHOL sets no download limit.
|
|
||||||
// https://iplists.firehol.org/?ipset=firehol_abusers_30d
|
|
||||||
// https://iplists.firehol.org/?ipset=firehol_level3
|
|
||||||
fireHOLAbusersInterval = 5 * time.Hour
|
|
||||||
fireHOLLevel3Interval = 3 * time.Hour
|
|
||||||
|
|
||||||
// Spamhaus asks that DROP be downloaded at most once a day:
|
|
||||||
// https://www.spamhaus.org/blocklists/do-not-route-or-peer/
|
|
||||||
spamhausDROPInterval = 24 * time.Hour
|
|
||||||
|
|
||||||
// Tor publishes a new exit list every hour and sets no download limit:
|
|
||||||
// https://collector.torproject.org/recent/exit-lists/
|
|
||||||
torExitsInterval = time.Hour
|
|
||||||
)
|
|
||||||
|
|
||||||
// Lists returns the public blocklists routewatch downloads. Each can be
|
|
||||||
// downloaded whole without an account or a key. Spamhaus merged its EDROP
|
|
||||||
// list into DROP, so DROP covers both.
|
|
||||||
func Lists() []List {
|
|
||||||
return []List{
|
|
||||||
{
|
|
||||||
Name: "dshield",
|
|
||||||
Description: "In one of the 20 networks that DShield saw scan " +
|
|
||||||
"the most targets in the last three days",
|
|
||||||
URL: "https://feeds.dshield.org/block.txt",
|
|
||||||
Interval: dshieldInterval,
|
|
||||||
Format: FormatDShield,
|
|
||||||
},
|
|
||||||
{
|
|
||||||
Name: "feodo_tracker",
|
|
||||||
Description: "A botnet command-and-control server",
|
|
||||||
URL: "https://feodotracker.abuse.ch/downloads/ipblocklist.txt",
|
|
||||||
Interval: feodoTrackerInterval,
|
|
||||||
Format: FormatLines,
|
|
||||||
},
|
|
||||||
{
|
|
||||||
Name: "firehol_abusers_30d",
|
|
||||||
Description: "Reported for comment, form or forum spam in the " +
|
|
||||||
"last 30 days",
|
|
||||||
URL: "https://iplists.firehol.org/files/firehol_abusers_30d.netset",
|
|
||||||
Interval: fireHOLAbusersInterval,
|
|
||||||
Format: FormatLines,
|
|
||||||
},
|
|
||||||
{
|
|
||||||
Name: "firehol_level3",
|
|
||||||
Description: "Reported for attacks, spyware or viruses in the " +
|
|
||||||
"last 30 days",
|
|
||||||
URL: "https://iplists.firehol.org/files/firehol_level3.netset",
|
|
||||||
Interval: fireHOLLevel3Interval,
|
|
||||||
Format: FormatLines,
|
|
||||||
},
|
|
||||||
{
|
|
||||||
Name: "spamhaus_drop",
|
|
||||||
Description: "In an IPv4 network that Spamhaus says spammers " +
|
|
||||||
"or cyber criminals have hijacked or lease",
|
|
||||||
URL: "https://www.spamhaus.org/drop/drop_v4.json",
|
|
||||||
Interval: spamhausDROPInterval,
|
|
||||||
Format: FormatSpamhausJSON,
|
|
||||||
},
|
|
||||||
{
|
|
||||||
Name: "spamhaus_dropv6",
|
|
||||||
Description: "In an IPv6 network that Spamhaus says spammers " +
|
|
||||||
"or cyber criminals have hijacked or lease",
|
|
||||||
URL: "https://www.spamhaus.org/drop/drop_v6.json",
|
|
||||||
Interval: spamhausDROPInterval,
|
|
||||||
Format: FormatSpamhausJSON,
|
|
||||||
},
|
|
||||||
{
|
|
||||||
Name: "tor_exits",
|
|
||||||
Description: "A Tor exit node, where traffic from the Tor network leaves",
|
|
||||||
URL: "https://check.torproject.org/torbulkexitlist",
|
|
||||||
Interval: torExitsInterval,
|
|
||||||
Format: FormatLines,
|
|
||||||
},
|
|
||||||
}
|
|
||||||
}
|
|
||||||
@@ -1,158 +0,0 @@
|
|||||||
package blocklist
|
|
||||||
|
|
||||||
import (
|
|
||||||
"bufio"
|
|
||||||
"encoding/json"
|
|
||||||
"errors"
|
|
||||||
"fmt"
|
|
||||||
"io"
|
|
||||||
"net/netip"
|
|
||||||
"strings"
|
|
||||||
|
|
||||||
"go4.org/netipx"
|
|
||||||
)
|
|
||||||
|
|
||||||
const (
|
|
||||||
// commentPrefix starts a comment line in a list file.
|
|
||||||
commentPrefix = "#"
|
|
||||||
|
|
||||||
// networkSeparator separates the address and the mask length of a
|
|
||||||
// network, as in 192.0.2.0/24.
|
|
||||||
networkSeparator = "/"
|
|
||||||
|
|
||||||
// dshieldFieldSeparator separates the fields of a DShield line.
|
|
||||||
dshieldFieldSeparator = "\t"
|
|
||||||
|
|
||||||
// spamhausMetadataType is the type of the line that ends a Spamhaus
|
|
||||||
// file. That line holds the file's metadata and no network.
|
|
||||||
spamhausMetadataType = "metadata"
|
|
||||||
)
|
|
||||||
|
|
||||||
var (
|
|
||||||
errUnknownFormat = errors.New("unknown list format")
|
|
||||||
errNoNetwork = errors.New("no network on the line")
|
|
||||||
errBadRange = errors.New("first and last address make no range")
|
|
||||||
)
|
|
||||||
|
|
||||||
// parse reads a list file written in format and returns the addresses it
|
|
||||||
// holds. A line it cannot read fails the whole file, so a file that is not
|
|
||||||
// the list, such as an error page, never replaces a good copy.
|
|
||||||
func parse(r io.Reader, format Format) (*netipx.IPSet, error) {
|
|
||||||
var builder netipx.IPSetBuilder
|
|
||||||
|
|
||||||
scanner := bufio.NewScanner(r)
|
|
||||||
|
|
||||||
for lineNumber := 1; scanner.Scan(); lineNumber++ {
|
|
||||||
line := strings.TrimSpace(scanner.Text())
|
|
||||||
if line == "" || strings.HasPrefix(line, commentPrefix) {
|
|
||||||
continue
|
|
||||||
}
|
|
||||||
|
|
||||||
err := addLine(&builder, line, format)
|
|
||||||
if err != nil {
|
|
||||||
return nil, fmt.Errorf("line %d: %w", lineNumber, err)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
err := scanner.Err()
|
|
||||||
if err != nil {
|
|
||||||
return nil, err
|
|
||||||
}
|
|
||||||
|
|
||||||
return builder.IPSet()
|
|
||||||
}
|
|
||||||
|
|
||||||
// addLine adds the address or network on one line of a list file to builder.
|
|
||||||
func addLine(builder *netipx.IPSetBuilder, line string, format Format) error {
|
|
||||||
switch format {
|
|
||||||
case FormatLines:
|
|
||||||
return addAddressOrNetwork(builder, line)
|
|
||||||
case FormatSpamhausJSON:
|
|
||||||
return addSpamhausNetwork(builder, line)
|
|
||||||
case FormatDShield:
|
|
||||||
return addDShieldNetwork(builder, line)
|
|
||||||
default:
|
|
||||||
return errUnknownFormat
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// addAddressOrNetwork adds an address, such as 192.0.2.1, or a network, such
|
|
||||||
// as 192.0.2.0/24, to builder.
|
|
||||||
func addAddressOrNetwork(builder *netipx.IPSetBuilder, text string) error {
|
|
||||||
if strings.Contains(text, networkSeparator) {
|
|
||||||
network, err := netip.ParsePrefix(text)
|
|
||||||
if err != nil {
|
|
||||||
return err
|
|
||||||
}
|
|
||||||
|
|
||||||
builder.AddPrefix(network)
|
|
||||||
|
|
||||||
return nil
|
|
||||||
}
|
|
||||||
|
|
||||||
addr, err := netip.ParseAddr(text)
|
|
||||||
if err != nil {
|
|
||||||
return err
|
|
||||||
}
|
|
||||||
|
|
||||||
builder.Add(addr)
|
|
||||||
|
|
||||||
return nil
|
|
||||||
}
|
|
||||||
|
|
||||||
// addSpamhausNetwork adds the network on a line of a Spamhaus file to
|
|
||||||
// builder. The metadata line adds nothing.
|
|
||||||
func addSpamhausNetwork(builder *netipx.IPSetBuilder, line string) error {
|
|
||||||
var entry struct {
|
|
||||||
CIDR string `json:"cidr"`
|
|
||||||
Type string `json:"type"`
|
|
||||||
}
|
|
||||||
|
|
||||||
err := json.Unmarshal([]byte(line), &entry)
|
|
||||||
if err != nil {
|
|
||||||
return err
|
|
||||||
}
|
|
||||||
|
|
||||||
if entry.Type == spamhausMetadataType {
|
|
||||||
return nil
|
|
||||||
}
|
|
||||||
|
|
||||||
network, err := netip.ParsePrefix(entry.CIDR)
|
|
||||||
if err != nil {
|
|
||||||
return err
|
|
||||||
}
|
|
||||||
|
|
||||||
builder.AddPrefix(network)
|
|
||||||
|
|
||||||
return nil
|
|
||||||
}
|
|
||||||
|
|
||||||
// addDShieldNetwork adds the network on a line of a DShield file to builder.
|
|
||||||
// The line's first two fields are the network's first and last address.
|
|
||||||
func addDShieldNetwork(builder *netipx.IPSetBuilder, line string) error {
|
|
||||||
firstField, rest, found := strings.Cut(line, dshieldFieldSeparator)
|
|
||||||
if !found {
|
|
||||||
return errNoNetwork
|
|
||||||
}
|
|
||||||
|
|
||||||
lastField, _, _ := strings.Cut(rest, dshieldFieldSeparator)
|
|
||||||
|
|
||||||
first, err := netip.ParseAddr(firstField)
|
|
||||||
if err != nil {
|
|
||||||
return err
|
|
||||||
}
|
|
||||||
|
|
||||||
last, err := netip.ParseAddr(lastField)
|
|
||||||
if err != nil {
|
|
||||||
return err
|
|
||||||
}
|
|
||||||
|
|
||||||
network := netipx.IPRangeFrom(first, last)
|
|
||||||
if !network.IsValid() {
|
|
||||||
return errBadRange
|
|
||||||
}
|
|
||||||
|
|
||||||
builder.AddRange(network)
|
|
||||||
|
|
||||||
return nil
|
|
||||||
}
|
|
||||||
+100
-9
@@ -6,10 +6,11 @@ import (
|
|||||||
"sync"
|
"sync"
|
||||||
)
|
)
|
||||||
|
|
||||||
// liveCounts holds the running row counts that the stats endpoints report.
|
// liveCounts holds the running row counts and the prefix distribution that the
|
||||||
// They are seeded once at startup from the tables and then adjusted on every
|
// stats endpoints report. They are seeded once at startup from the tables and
|
||||||
// write, so a stats read serves them from memory instead of running a query
|
// then adjusted on every write, so a stats read serves them from memory instead
|
||||||
// over the tables. The COUNT(*) scans (issue 27) grew with the database until
|
// of running a query over the tables. The COUNT(*) scans (issue 27) and then
|
||||||
|
// the prefix distribution query (issue 30) each grew with the database until
|
||||||
// they took the whole request timeout and made /api/v1/stats return 500.
|
// they took the whole request timeout and made /api/v1/stats return 500.
|
||||||
//
|
//
|
||||||
// A single mutex guards all fields so the stats reader takes a consistent
|
// A single mutex guards all fields so the stats reader takes a consistent
|
||||||
@@ -22,11 +23,20 @@ type liveCounts struct {
|
|||||||
prefixesV6 int
|
prefixesV6 int
|
||||||
peerings int
|
peerings int
|
||||||
peers int
|
peers int
|
||||||
|
routesV4 int
|
||||||
|
routesV6 int
|
||||||
|
// The prefix distribution: for each mask length, the number of distinct
|
||||||
|
// prefixes that have at least one live route.
|
||||||
|
distributionV4 [ipv4Bits + 1]int
|
||||||
|
distributionV6 [ipv6Bits + 1]int
|
||||||
}
|
}
|
||||||
|
|
||||||
// seed sets every count to the value read from the tables at startup. It runs
|
// seed sets every count to the value read from the tables at startup. It runs
|
||||||
// before any writer, so it needs no coordination with the adjust methods.
|
// before any writer, so it needs no coordination with the adjust methods.
|
||||||
func (c *liveCounts) seed(asns, prefixesV4, prefixesV6, peerings, peers int) {
|
func (c *liveCounts) seed(
|
||||||
|
asns, prefixesV4, prefixesV6, peerings, peers, routesV4, routesV6 int,
|
||||||
|
distributionV4, distributionV6 []PrefixDistribution,
|
||||||
|
) {
|
||||||
c.mu.Lock()
|
c.mu.Lock()
|
||||||
defer c.mu.Unlock()
|
defer c.mu.Unlock()
|
||||||
|
|
||||||
@@ -35,6 +45,16 @@ func (c *liveCounts) seed(asns, prefixesV4, prefixesV6, peerings, peers int) {
|
|||||||
c.prefixesV6 = prefixesV6
|
c.prefixesV6 = prefixesV6
|
||||||
c.peerings = peerings
|
c.peerings = peerings
|
||||||
c.peers = peers
|
c.peers = peers
|
||||||
|
c.routesV4 = routesV4
|
||||||
|
c.routesV6 = routesV6
|
||||||
|
|
||||||
|
for _, entry := range distributionV4 {
|
||||||
|
addAtMaskLength(c.distributionV4[:], entry.MaskLength, entry.Count)
|
||||||
|
}
|
||||||
|
|
||||||
|
for _, entry := range distributionV6 {
|
||||||
|
addAtMaskLength(c.distributionV6[:], entry.MaskLength, entry.Count)
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// addASNs adds n to the ASN count.
|
// addASNs adds n to the ASN count.
|
||||||
@@ -66,7 +86,41 @@ func (c *liveCounts) addPeers(n int) {
|
|||||||
c.mu.Unlock()
|
c.mu.Unlock()
|
||||||
}
|
}
|
||||||
|
|
||||||
// fill copies the counts into a Stats, including the derived total, under a
|
// addRoutes adds to the IPv4 and IPv6 live-route counts. Deletions pass
|
||||||
|
// negative values.
|
||||||
|
func (c *liveCounts) addRoutes(v4, v6 int) {
|
||||||
|
c.mu.Lock()
|
||||||
|
c.routesV4 += v4
|
||||||
|
c.routesV6 += v6
|
||||||
|
c.mu.Unlock()
|
||||||
|
}
|
||||||
|
|
||||||
|
// addToDistribution adds n to the IPv4 and IPv6 prefix distributions once for
|
||||||
|
// each listed mask length. A write lists the mask lengths of the prefixes it
|
||||||
|
// gave their first live route with n = 1, and of the prefixes it left with no
|
||||||
|
// live route with n = -1.
|
||||||
|
func (c *liveCounts) addToDistribution(maskLengthsV4, maskLengthsV6 []int, n int) {
|
||||||
|
c.mu.Lock()
|
||||||
|
defer c.mu.Unlock()
|
||||||
|
|
||||||
|
for _, maskLength := range maskLengthsV4 {
|
||||||
|
addAtMaskLength(c.distributionV4[:], maskLength, n)
|
||||||
|
}
|
||||||
|
|
||||||
|
for _, maskLength := range maskLengthsV6 {
|
||||||
|
addAtMaskLength(c.distributionV6[:], maskLength, n)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// addAtMaskLength adds n to counts[maskLength]. A mask length the array has no
|
||||||
|
// entry for is ignored, so a malformed route cannot crash the daemon.
|
||||||
|
func addAtMaskLength(counts []int, maskLength, n int) {
|
||||||
|
if maskLength >= 0 && maskLength < len(counts) {
|
||||||
|
counts[maskLength] += n
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// fill copies the counts into a Stats, including the derived totals, under a
|
||||||
// single read lock so the reader sees one consistent snapshot.
|
// single read lock so the reader sees one consistent snapshot.
|
||||||
func (c *liveCounts) fill(s *Stats) {
|
func (c *liveCounts) fill(s *Stats) {
|
||||||
c.mu.RLock()
|
c.mu.RLock()
|
||||||
@@ -78,6 +132,26 @@ func (c *liveCounts) fill(s *Stats) {
|
|||||||
s.Prefixes = c.prefixesV4 + c.prefixesV6
|
s.Prefixes = c.prefixesV4 + c.prefixesV6
|
||||||
s.Peerings = c.peerings
|
s.Peerings = c.peerings
|
||||||
s.Peers = c.peers
|
s.Peers = c.peers
|
||||||
|
s.IPv4Routes = c.routesV4
|
||||||
|
s.IPv6Routes = c.routesV6
|
||||||
|
s.LiveRoutes = c.routesV4 + c.routesV6
|
||||||
|
s.IPv4PrefixDistribution = distributionList(c.distributionV4[:])
|
||||||
|
s.IPv6PrefixDistribution = distributionList(c.distributionV6[:])
|
||||||
|
}
|
||||||
|
|
||||||
|
// distributionList lists the mask lengths that have at least one prefix, in
|
||||||
|
// ascending order, the way the distribution query returns them.
|
||||||
|
func distributionList(counts []int) []PrefixDistribution {
|
||||||
|
var list []PrefixDistribution
|
||||||
|
|
||||||
|
for maskLength, count := range counts {
|
||||||
|
if count > 0 {
|
||||||
|
list = append(list,
|
||||||
|
PrefixDistribution{MaskLength: maskLength, Count: count})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
return list
|
||||||
}
|
}
|
||||||
|
|
||||||
// countRows returns the number of rows in the named table. It is used only at
|
// countRows returns the number of rows in the named table. It is used only at
|
||||||
@@ -93,8 +167,9 @@ func (d *Database) countRows(ctx context.Context, table string) (int, error) {
|
|||||||
return n, nil
|
return n, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
// seedCounts reads the current row counts from the tables into the in-memory
|
// seedCounts reads the current row counts and prefix distribution from the
|
||||||
// counters. It runs once at startup, before the streamer begins writing.
|
// tables into the in-memory counters. It runs once at startup, before the
|
||||||
|
// streamer begins writing.
|
||||||
func (d *Database) seedCounts(ctx context.Context) error {
|
func (d *Database) seedCounts(ctx context.Context) error {
|
||||||
asns, err := d.countRows(ctx, "asns")
|
asns, err := d.countRows(ctx, "asns")
|
||||||
if err != nil {
|
if err != nil {
|
||||||
@@ -121,7 +196,23 @@ func (d *Database) seedCounts(ctx context.Context) error {
|
|||||||
return err
|
return err
|
||||||
}
|
}
|
||||||
|
|
||||||
d.counts.seed(asns, prefixesV4, prefixesV6, peerings, peers)
|
routesV4, err := d.countRows(ctx, tableLiveRoutesV4)
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
|
||||||
|
routesV6, err := d.countRows(ctx, tableLiveRoutesV6)
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
|
||||||
|
distributionV4, distributionV6, err := d.GetPrefixDistributionContext(ctx)
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
|
||||||
|
d.counts.seed(asns, prefixesV4, prefixesV6, peerings, peers, routesV4, routesV6,
|
||||||
|
distributionV4, distributionV6)
|
||||||
|
|
||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -3,23 +3,62 @@ package database
|
|||||||
import (
|
import (
|
||||||
"context"
|
"context"
|
||||||
"fmt"
|
"fmt"
|
||||||
|
"slices"
|
||||||
"sync"
|
"sync"
|
||||||
"testing"
|
"testing"
|
||||||
"time"
|
"time"
|
||||||
|
|
||||||
"git.eeqj.de/sneak/routewatch/internal/config"
|
"git.eeqj.de/sneak/routewatch/internal/config"
|
||||||
"git.eeqj.de/sneak/routewatch/internal/logger"
|
"git.eeqj.de/sneak/routewatch/internal/logger"
|
||||||
|
"github.com/google/uuid"
|
||||||
)
|
)
|
||||||
|
|
||||||
// testV4PeerIP is the peer the IPv4 rows in these tests are learned from.
|
// testV4PeerIP is the peer the IPv4 routes in these tests are learned from.
|
||||||
const testV4PeerIP = "192.0.2.1"
|
const testV4PeerIP = "192.0.2.1"
|
||||||
|
|
||||||
// testV6Prefix is the IPv6 prefix of these tests.
|
// testV6Prefix is the prefix of the IPv6 routes in these tests.
|
||||||
const testV6Prefix = "2001:db8::/32"
|
const testV6Prefix = "2001:db8::/32"
|
||||||
|
|
||||||
|
// mkV4Route builds an IPv4 live route with its mask length taken from the
|
||||||
|
// prefix.
|
||||||
|
func mkV4Route(t *testing.T, prefix string, asn int, ts time.Time) *LiveRoute {
|
||||||
|
t.Helper()
|
||||||
|
|
||||||
|
maskLength, err := prefixMaskLength(prefix)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("prefixMaskLength(%s): %v", prefix, err)
|
||||||
|
}
|
||||||
|
|
||||||
|
return &LiveRoute{
|
||||||
|
Prefix: prefix,
|
||||||
|
MaskLength: maskLength,
|
||||||
|
IPVersion: ipVersionV4,
|
||||||
|
OriginASN: asn,
|
||||||
|
PeerIP: testV4PeerIP,
|
||||||
|
ASPath: []int{asn},
|
||||||
|
NextHop: "192.0.2.254",
|
||||||
|
LastUpdated: ts,
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// mkV6Route builds an IPv6 live route.
|
||||||
|
func mkV6Route(prefix string, asn int, ts time.Time) *LiveRoute {
|
||||||
|
return &LiveRoute{
|
||||||
|
Prefix: prefix,
|
||||||
|
MaskLength: 32,
|
||||||
|
IPVersion: ipVersionV6,
|
||||||
|
OriginASN: asn,
|
||||||
|
PeerIP: "2001:db8::1",
|
||||||
|
ASPath: []int{asn},
|
||||||
|
NextHop: "2001:db8::ffff",
|
||||||
|
LastUpdated: ts,
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
// TestLiveCountsTrackWritesInRealtime checks that the stats counts start at
|
// TestLiveCountsTrackWritesInRealtime checks that the stats counts start at
|
||||||
// zero and reflect each write the moment it commits (no recompute, no timer).
|
// zero, reflect each write the moment it commits (no recompute, no timer), do
|
||||||
// These counts are what /api/v1/stats reports; before this change the
|
// not move when a route is merely re-announced, and drop when a route is
|
||||||
|
// deleted. These counts are what /api/v1/stats reports; before this change the
|
||||||
// endpoint recomputed them with a COUNT(*) over each table on every request.
|
// endpoint recomputed them with a COUNT(*) over each table on every request.
|
||||||
func TestLiveCountsTrackWritesInRealtime(t *testing.T) {
|
func TestLiveCountsTrackWritesInRealtime(t *testing.T) {
|
||||||
t.Parallel()
|
t.Parallel()
|
||||||
@@ -40,7 +79,7 @@ func TestLiveCountsTrackWritesInRealtime(t *testing.T) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
if empty.ASNs != 0 || empty.Prefixes != 0 || empty.Peerings != 0 ||
|
if empty.ASNs != 0 || empty.Prefixes != 0 || empty.Peerings != 0 ||
|
||||||
empty.Peers != 0 {
|
empty.Peers != 0 || empty.LiveRoutes != 0 {
|
||||||
t.Fatalf("empty database counts nonzero: %+v", empty)
|
t.Fatalf("empty database counts nonzero: %+v", empty)
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -48,6 +87,17 @@ func TestLiveCountsTrackWritesInRealtime(t *testing.T) {
|
|||||||
|
|
||||||
writeASNsPrefixesAndPeers(t, db, ts)
|
writeASNsPrefixesAndPeers(t, db, ts)
|
||||||
|
|
||||||
|
routes := []*LiveRoute{
|
||||||
|
mkV4Route(t, "198.51.100.0/24", 64500, ts),
|
||||||
|
mkV4Route(t, "203.0.113.0/24", 64501, ts.Add(time.Minute)),
|
||||||
|
mkV6Route(testV6Prefix, 64502, ts.Add(2*time.Minute)),
|
||||||
|
}
|
||||||
|
|
||||||
|
err = db.UpsertLiveRouteBatch(routes)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("UpsertLiveRouteBatch: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
stats, err := db.GetStatsContext(ctx)
|
stats, err := db.GetStatsContext(ctx)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatalf("GetStatsContext: %v", err)
|
t.Fatalf("GetStatsContext: %v", err)
|
||||||
@@ -55,11 +105,46 @@ func TestLiveCountsTrackWritesInRealtime(t *testing.T) {
|
|||||||
|
|
||||||
assertCounts(t, "after inserts", stats, wantCounts{
|
assertCounts(t, "after inserts", stats, wantCounts{
|
||||||
asns: 2, prefixes: 2, peerings: 1, peers: 1,
|
asns: 2, prefixes: 2, peerings: 1, peers: 1,
|
||||||
|
ipv4Routes: 2, ipv6Routes: 1, liveRoutes: 3,
|
||||||
|
})
|
||||||
|
|
||||||
|
// Re-announcing the same routes is an update, not an insert: counts hold.
|
||||||
|
err = db.UpsertLiveRouteBatch(routes)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("UpsertLiveRouteBatch (re-announce): %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
stats, err = db.GetStatsContext(ctx)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("GetStatsContext: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
assertCounts(t, "after re-announce", stats, wantCounts{
|
||||||
|
asns: 2, prefixes: 2, peerings: 1, peers: 1,
|
||||||
|
ipv4Routes: 2, ipv6Routes: 1, liveRoutes: 3,
|
||||||
|
})
|
||||||
|
|
||||||
|
// A withdrawal removes one route.
|
||||||
|
err = db.DeleteLiveRouteBatch([]LiveRouteDeletion{
|
||||||
|
{Prefix: "203.0.113.0/24", PeerIP: testV4PeerIP, IPVersion: ipVersionV4},
|
||||||
|
})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("DeleteLiveRouteBatch: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
stats, err = db.GetStatsContext(ctx)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("GetStatsContext: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
assertCounts(t, "after delete", stats, wantCounts{
|
||||||
|
asns: 2, prefixes: 2, peerings: 1, peers: 1,
|
||||||
|
ipv4Routes: 1, ipv6Routes: 1, liveRoutes: 2,
|
||||||
})
|
})
|
||||||
}
|
}
|
||||||
|
|
||||||
// writeASNsPrefixesAndPeers writes two ASNs, two prefixes, one peer and one
|
// writeASNsPrefixesAndPeers writes two ASNs, two prefixes, one peer and one
|
||||||
// peering: the rows the stats count.
|
// peering: the rows besides live routes that the stats count.
|
||||||
func writeASNsPrefixesAndPeers(t *testing.T, db *Database, ts time.Time) {
|
func writeASNsPrefixesAndPeers(t *testing.T, db *Database, ts time.Time) {
|
||||||
t.Helper()
|
t.Helper()
|
||||||
|
|
||||||
@@ -95,8 +180,8 @@ func writeASNsPrefixesAndPeers(t *testing.T, db *Database, ts time.Time) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
// TestLiveCountsSeededFromDatabaseAtStartup writes rows, reopens the same
|
// TestLiveCountsSeededFromDatabaseAtStartup writes rows, reopens the same
|
||||||
// database file, and checks the counts come back from the seed scan rather
|
// database file, and checks the counts and the prefix distribution come back
|
||||||
// than starting at zero.
|
// from the seed scan rather than starting at zero.
|
||||||
func TestLiveCountsSeededFromDatabaseAtStartup(t *testing.T) {
|
func TestLiveCountsSeededFromDatabaseAtStartup(t *testing.T) {
|
||||||
t.Parallel()
|
t.Parallel()
|
||||||
|
|
||||||
@@ -107,7 +192,23 @@ func TestLiveCountsSeededFromDatabaseAtStartup(t *testing.T) {
|
|||||||
t.Fatalf("failed to create database: %v", err)
|
t.Fatalf("failed to create database: %v", err)
|
||||||
}
|
}
|
||||||
|
|
||||||
writeASNsPrefixesAndPeers(t, db, time.Date(2026, 1, 2, 3, 4, 5, 0, time.UTC))
|
ts := time.Date(2026, 1, 2, 3, 4, 5, 0, time.UTC)
|
||||||
|
|
||||||
|
seen := SeenTimes{First: ts, Last: ts}
|
||||||
|
|
||||||
|
err = db.GetOrCreateASNBatch(
|
||||||
|
map[int]SeenTimes{64500: seen, 64501: seen, 64502: seen})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("GetOrCreateASNBatch: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
err = db.UpsertLiveRouteBatch([]*LiveRoute{
|
||||||
|
mkV4Route(t, "198.51.100.0/24", 64500, ts),
|
||||||
|
mkV6Route(testV6Prefix, 64502, ts),
|
||||||
|
})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("UpsertLiveRouteBatch: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
err = db.Close()
|
err = db.Close()
|
||||||
if err != nil {
|
if err != nil {
|
||||||
@@ -125,9 +226,489 @@ func TestLiveCountsSeededFromDatabaseAtStartup(t *testing.T) {
|
|||||||
t.Fatalf("GetStatsContext after reopen: %v", err)
|
t.Fatalf("GetStatsContext after reopen: %v", err)
|
||||||
}
|
}
|
||||||
|
|
||||||
assertCounts(t, "after reopen", stats, wantCounts{
|
if stats.ASNs != 3 {
|
||||||
asns: 2, prefixes: 2, peerings: 1, peers: 1,
|
t.Errorf("seeded ASNs = %d, want 3", stats.ASNs)
|
||||||
|
}
|
||||||
|
|
||||||
|
if stats.IPv4Routes != 1 || stats.IPv6Routes != 1 || stats.LiveRoutes != 2 {
|
||||||
|
t.Errorf("seeded routes = (v4 %d, v6 %d, total %d), want (1, 1, 2)",
|
||||||
|
stats.IPv4Routes, stats.IPv6Routes, stats.LiveRoutes)
|
||||||
|
}
|
||||||
|
|
||||||
|
assertDistribution(t, "seeded IPv4 distribution", stats.IPv4PrefixDistribution,
|
||||||
|
[]PrefixDistribution{{MaskLength: 24, Count: 1}})
|
||||||
|
assertDistribution(t, "seeded IPv6 distribution", stats.IPv6PrefixDistribution,
|
||||||
|
[]PrefixDistribution{{MaskLength: 32, Count: 1}})
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestPrefixDistributionTracksWrites checks that the prefix distribution the
|
||||||
|
// stats read reports stays exact across each kind of live-route write, and that
|
||||||
|
// after every step it equals what the distribution query reads from the route
|
||||||
|
// tables. The steps run once through the batch methods the prefix handler uses
|
||||||
|
// and once through the single-route methods.
|
||||||
|
func TestPrefixDistributionTracksWrites(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
steps := prefixDistributionSteps(t)
|
||||||
|
|
||||||
|
for _, batch := range []bool{true, false} {
|
||||||
|
name := "single-route writes"
|
||||||
|
if batch {
|
||||||
|
name = "batch writes"
|
||||||
|
}
|
||||||
|
|
||||||
|
t.Run(name, func(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
db, err := New(&config.Config{StateDir: t.TempDir()}, logger.New())
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("failed to create database: %v", err)
|
||||||
|
}
|
||||||
|
defer func() { _ = db.Close() }()
|
||||||
|
|
||||||
|
ctx := context.Background()
|
||||||
|
|
||||||
|
for _, step := range steps {
|
||||||
|
err := announceRoutes(db, batch, step.announce)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("%s: announce: %v", step.name, err)
|
||||||
|
}
|
||||||
|
|
||||||
|
err = withdrawRoutes(db, batch, step.withdraw)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("%s: withdraw: %v", step.name, err)
|
||||||
|
}
|
||||||
|
|
||||||
|
stats, err := db.GetStatsContext(ctx)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("%s: GetStatsContext: %v", step.name, err)
|
||||||
|
}
|
||||||
|
|
||||||
|
queryV4, queryV6, err := db.GetPrefixDistributionContext(ctx)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("%s: GetPrefixDistributionContext: %v", step.name, err)
|
||||||
|
}
|
||||||
|
|
||||||
|
assertDistribution(t, step.name+": IPv4 distribution",
|
||||||
|
stats.IPv4PrefixDistribution, step.wantV4)
|
||||||
|
assertDistribution(t, step.name+": IPv6 distribution",
|
||||||
|
stats.IPv6PrefixDistribution, step.wantV6)
|
||||||
|
assertDistribution(t, step.name+": IPv4 distribution query",
|
||||||
|
queryV4, step.wantV4)
|
||||||
|
assertDistribution(t, step.name+": IPv6 distribution query",
|
||||||
|
queryV6, step.wantV6)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// prefixDistributionStep is one step of TestPrefixDistributionTracksWrites:
|
||||||
|
// the routes it announces, then the routes it withdraws, then the
|
||||||
|
// distributions both the stats and the distribution query must report.
|
||||||
|
type prefixDistributionStep struct {
|
||||||
|
name string
|
||||||
|
announce []*LiveRoute
|
||||||
|
withdraw []*LiveRoute
|
||||||
|
wantV4 []PrefixDistribution
|
||||||
|
wantV6 []PrefixDistribution
|
||||||
|
}
|
||||||
|
|
||||||
|
// prefixDistributionSteps returns the steps of
|
||||||
|
// TestPrefixDistributionTracksWrites, in the order they run.
|
||||||
|
func prefixDistributionSteps(t *testing.T) []prefixDistributionStep {
|
||||||
|
t.Helper()
|
||||||
|
|
||||||
|
ts := time.Date(2026, 1, 2, 3, 4, 5, 0, time.UTC)
|
||||||
|
shared := mkV4Route(t, "198.51.100.0/24", 64500, ts)
|
||||||
|
sharedSecondPeer := mkV4Route(t, "198.51.100.0/24", 64500, ts)
|
||||||
|
sharedSecondPeer.PeerIP = "192.0.2.2"
|
||||||
|
other := mkV4Route(t, "203.0.113.0/24", 64501, ts)
|
||||||
|
wide := mkV4Route(t, "172.16.0.0/16", 64502, ts)
|
||||||
|
v6 := mkV6Route(testV6Prefix, 64503, ts)
|
||||||
|
v6SecondPeer := mkV6Route(testV6Prefix, 64503, ts)
|
||||||
|
v6SecondPeer.PeerIP = "2001:db8::2"
|
||||||
|
|
||||||
|
all := []PrefixDistribution{{MaskLength: 16, Count: 1}, {MaskLength: 24, Count: 2}}
|
||||||
|
v6Only := []PrefixDistribution{{MaskLength: 32, Count: 1}}
|
||||||
|
|
||||||
|
return []prefixDistributionStep{
|
||||||
|
{
|
||||||
|
name: "new routes",
|
||||||
|
announce: []*LiveRoute{shared, other, wide, v6},
|
||||||
|
wantV4: all, wantV6: v6Only,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "re-announcement",
|
||||||
|
announce: []*LiveRoute{shared, other, wide, v6},
|
||||||
|
wantV4: all, wantV6: v6Only,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "second peer announces a prefix that has a live route",
|
||||||
|
announce: []*LiveRoute{sharedSecondPeer, v6SecondPeer},
|
||||||
|
wantV4: all, wantV6: v6Only,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "withdrawal of a route that is not the last for its prefix",
|
||||||
|
withdraw: []*LiveRoute{shared, v6},
|
||||||
|
wantV4: all, wantV6: v6Only,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "withdrawal of the last route for a prefix",
|
||||||
|
withdraw: []*LiveRoute{sharedSecondPeer, v6SecondPeer},
|
||||||
|
wantV4: []PrefixDistribution{
|
||||||
|
{MaskLength: 16, Count: 1}, {MaskLength: 24, Count: 1},
|
||||||
|
},
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "withdrawal of every remaining route",
|
||||||
|
withdraw: []*LiveRoute{other, wide},
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "two peers announce a new prefix together",
|
||||||
|
announce: []*LiveRoute{shared, sharedSecondPeer, v6, v6SecondPeer},
|
||||||
|
wantV4: []PrefixDistribution{{MaskLength: 24, Count: 1}},
|
||||||
|
wantV6: v6Only,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "both routes for a prefix withdrawn together",
|
||||||
|
withdraw: []*LiveRoute{shared, sharedSecondPeer, v6, v6SecondPeer},
|
||||||
|
},
|
||||||
|
// The feed often withdraws a route that is not live. That must not take
|
||||||
|
// the prefix out of the distribution. A count wrongly taken below zero is
|
||||||
|
// left out of the answer, so the next step shows it: its announcement
|
||||||
|
// at the same mask length would then not be counted.
|
||||||
|
{
|
||||||
|
name: "withdrawal of routes that are not live",
|
||||||
|
withdraw: []*LiveRoute{other, v6},
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "announcement after a withdrawal of routes that are not live",
|
||||||
|
announce: []*LiveRoute{other, v6},
|
||||||
|
wantV4: []PrefixDistribution{{MaskLength: 24, Count: 1}},
|
||||||
|
wantV6: v6Only,
|
||||||
|
},
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// announceRoutes writes routes in one UpsertLiveRouteBatch, or with one
|
||||||
|
// UpsertLiveRoute each.
|
||||||
|
func announceRoutes(db *Database, batch bool, routes []*LiveRoute) error {
|
||||||
|
if batch {
|
||||||
|
return db.UpsertLiveRouteBatch(routes)
|
||||||
|
}
|
||||||
|
|
||||||
|
for _, route := range routes {
|
||||||
|
err := db.UpsertLiveRoute(route)
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// withdrawRoutes removes routes in one DeleteLiveRouteBatch, or with one
|
||||||
|
// DeleteLiveRoute each.
|
||||||
|
func withdrawRoutes(db *Database, batch bool, routes []*LiveRoute) error {
|
||||||
|
if !batch {
|
||||||
|
for _, route := range routes {
|
||||||
|
err := db.DeleteLiveRoute(route.Prefix, route.PeerIP)
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
deletions := make([]LiveRouteDeletion, 0, len(routes))
|
||||||
|
for _, route := range routes {
|
||||||
|
deletions = append(deletions, LiveRouteDeletion{
|
||||||
|
Prefix: route.Prefix, PeerIP: route.PeerIP, IPVersion: route.IPVersion,
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
return db.DeleteLiveRouteBatch(deletions)
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestDeleteLiveRoutesOfPeer gives two peers IPv4 and IPv6 live routes, some
|
||||||
|
// for the same prefixes, and deletes all of one peer's routes. The other peer's
|
||||||
|
// routes stay, and the route counts and prefix distribution held in memory
|
||||||
|
// equal what is read from the route tables.
|
||||||
|
func TestDeleteLiveRoutesOfPeer(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
const (
|
||||||
|
peerIP = "192.0.2.1"
|
||||||
|
otherPeerIP = "192.0.2.2"
|
||||||
|
sharedV4 = "198.51.100.0/24"
|
||||||
|
sharedV6 = "2001:db8::/32"
|
||||||
|
)
|
||||||
|
|
||||||
|
db, err := New(&config.Config{StateDir: t.TempDir()}, logger.New())
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("failed to create database: %v", err)
|
||||||
|
}
|
||||||
|
defer func() { _ = db.Close() }()
|
||||||
|
|
||||||
|
ts := time.Date(2026, 1, 2, 3, 4, 5, 0, time.UTC)
|
||||||
|
ofPeer := func(peer string, route *LiveRoute) *LiveRoute {
|
||||||
|
route.PeerIP = peer
|
||||||
|
|
||||||
|
return route
|
||||||
|
}
|
||||||
|
onlyV6 := mkV6Route("2001:db8:1::/48", 64503, ts)
|
||||||
|
onlyV6.MaskLength = 48
|
||||||
|
|
||||||
|
err = db.UpsertLiveRouteBatch([]*LiveRoute{
|
||||||
|
ofPeer(peerIP, mkV4Route(t, sharedV4, 64500, ts)),
|
||||||
|
ofPeer(otherPeerIP, mkV4Route(t, sharedV4, 64500, ts)),
|
||||||
|
ofPeer(peerIP, mkV4Route(t, "203.0.113.0/24", 64501, ts)),
|
||||||
|
ofPeer(peerIP, mkV6Route(sharedV6, 64502, ts)),
|
||||||
|
ofPeer(otherPeerIP, mkV6Route(sharedV6, 64502, ts)),
|
||||||
|
ofPeer(peerIP, onlyV6),
|
||||||
})
|
})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("UpsertLiveRouteBatch: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
// peerIP has two IPv4 and two IPv6 routes; a second delete finds none.
|
||||||
|
for _, want := range []int{4, 0} {
|
||||||
|
deleted, err := db.DeleteLiveRoutesOfPeer(peerIP)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("DeleteLiveRoutesOfPeer: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
if deleted != want {
|
||||||
|
t.Errorf("DeleteLiveRoutesOfPeer deleted %d routes, want %d",
|
||||||
|
deleted, want)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
for _, prefix := range []string{sharedV4, sharedV6} {
|
||||||
|
routes, _, err := db.GetPrefixDetailsContext(context.Background(), prefix)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("GetPrefixDetailsContext(%s): %v", prefix, err)
|
||||||
|
}
|
||||||
|
|
||||||
|
if len(routes) != 1 || routes[0].PeerIP != otherPeerIP {
|
||||||
|
t.Errorf("live routes for %s = %+v, want only the one from %s",
|
||||||
|
prefix, routes, otherPeerIP)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
checkOneLiveRoutePerIPVersion(t, db)
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestDeleteLiveRoutesOfPeerDeletesInChunks gives a peer one IPv4 route more
|
||||||
|
// than a chunk holds, two of them for prefixes another peer also has. One chunk
|
||||||
|
// deletes peerRouteChunkSize of the peer's routes and commits, leaving the last
|
||||||
|
// one. The routes are written again, and DeleteLiveRoutesOfPeer deletes all of
|
||||||
|
// them, a chunk and then the last one. After each step the route count and
|
||||||
|
// prefix distribution held in memory and those read from the route table are
|
||||||
|
// the ones wanted.
|
||||||
|
func TestDeleteLiveRoutesOfPeerDeletesInChunks(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
const (
|
||||||
|
peerIP = "192.0.2.1"
|
||||||
|
otherPeerIP = "192.0.2.2"
|
||||||
|
// How many of the peer's prefixes the other peer also has, the first
|
||||||
|
// ones in prefix order.
|
||||||
|
sharedPrefixes = 2
|
||||||
|
)
|
||||||
|
|
||||||
|
db, err := New(&config.Config{StateDir: t.TempDir()}, logger.New())
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("failed to create database: %v", err)
|
||||||
|
}
|
||||||
|
defer func() { _ = db.Close() }()
|
||||||
|
|
||||||
|
ts := time.Date(2026, 1, 2, 3, 4, 5, 0, time.UTC)
|
||||||
|
|
||||||
|
peerRoutes := make([]*LiveRoute, 0, peerRouteChunkSize+1)
|
||||||
|
otherPeerRoutes := make([]*LiveRoute, 0, sharedPrefixes)
|
||||||
|
|
||||||
|
for i := range peerRouteChunkSize + 1 {
|
||||||
|
prefix := fmt.Sprintf("10.%d.%d.0/24", i/256, i%256)
|
||||||
|
|
||||||
|
route := mkV4Route(t, prefix, 64500, ts)
|
||||||
|
route.PeerIP = peerIP
|
||||||
|
peerRoutes = append(peerRoutes, route)
|
||||||
|
|
||||||
|
if i < sharedPrefixes {
|
||||||
|
route := mkV4Route(t, prefix, 64500, ts)
|
||||||
|
route.PeerIP = otherPeerIP
|
||||||
|
otherPeerRoutes = append(otherPeerRoutes, route)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
err = db.UpsertLiveRouteBatch(slices.Concat(peerRoutes, otherPeerRoutes))
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("UpsertLiveRouteBatch: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
deletedV4, deletedV6, err := db.deletePeerRouteChunk(peerIP)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("deletePeerRouteChunk: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
if deletedV4 != peerRouteChunkSize || deletedV6 != 0 {
|
||||||
|
t.Errorf("deletePeerRouteChunk deleted (%d, %d) routes, want (%d, 0)",
|
||||||
|
deletedV4, deletedV6, peerRouteChunkSize)
|
||||||
|
}
|
||||||
|
|
||||||
|
// The shared prefixes went with the chunk, so each route left has a
|
||||||
|
// prefix of its own.
|
||||||
|
checkV4RoutesOfOwnPrefixes(t, db, "after one chunk", 1+sharedPrefixes)
|
||||||
|
|
||||||
|
err = db.UpsertLiveRouteBatch(peerRoutes)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("UpsertLiveRouteBatch: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
deleted, err := db.DeleteLiveRoutesOfPeer(peerIP)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("DeleteLiveRoutesOfPeer: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
if deleted != len(peerRoutes) {
|
||||||
|
t.Errorf("DeleteLiveRoutesOfPeer deleted %d routes, want %d",
|
||||||
|
deleted, len(peerRoutes))
|
||||||
|
}
|
||||||
|
|
||||||
|
checkV4RoutesOfOwnPrefixes(t, db, "after DeleteLiveRoutesOfPeer",
|
||||||
|
sharedPrefixes)
|
||||||
|
}
|
||||||
|
|
||||||
|
// checkV4RoutesOfOwnPrefixes checks that the database has wantRoutes IPv4 live
|
||||||
|
// routes, each for a /24 no other live route has, both as counted in memory and
|
||||||
|
// as read from the route table. when says which step of the test checks.
|
||||||
|
func checkV4RoutesOfOwnPrefixes(
|
||||||
|
t *testing.T, db *Database, when string, wantRoutes int,
|
||||||
|
) {
|
||||||
|
t.Helper()
|
||||||
|
|
||||||
|
ctx := context.Background()
|
||||||
|
|
||||||
|
stats, err := db.GetStatsContext(ctx)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("GetStatsContext: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
tableRoutes, _, err := db.GetLiveRouteCountsContext(ctx)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("GetLiveRouteCountsContext: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
if stats.IPv4Routes != wantRoutes || tableRoutes != wantRoutes {
|
||||||
|
t.Errorf("%s: IPv4 live routes: %d in memory, %d in the table, want %d",
|
||||||
|
when, stats.IPv4Routes, tableRoutes, wantRoutes)
|
||||||
|
}
|
||||||
|
|
||||||
|
queryDistribution, _, err := db.GetPrefixDistributionContext(ctx)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("GetPrefixDistributionContext: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
want := []PrefixDistribution{{MaskLength: 24, Count: wantRoutes}}
|
||||||
|
assertDistribution(t, when+": IPv4 distribution",
|
||||||
|
stats.IPv4PrefixDistribution, want)
|
||||||
|
assertDistribution(t, when+": IPv4 distribution query",
|
||||||
|
queryDistribution, want)
|
||||||
|
}
|
||||||
|
|
||||||
|
// checkOneLiveRoutePerIPVersion checks that the database has one IPv4 live
|
||||||
|
// route, for a /24, and one IPv6 live route, for a /32, both as counted in
|
||||||
|
// memory and as read from the route tables.
|
||||||
|
func checkOneLiveRoutePerIPVersion(t *testing.T, db *Database) {
|
||||||
|
t.Helper()
|
||||||
|
|
||||||
|
ctx := context.Background()
|
||||||
|
|
||||||
|
stats, err := db.GetStatsContext(ctx)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("GetStatsContext: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
tableV4, tableV6, err := db.GetLiveRouteCountsContext(ctx)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("GetLiveRouteCountsContext: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
if stats.IPv4Routes != 1 || tableV4 != 1 || stats.IPv6Routes != 1 ||
|
||||||
|
tableV6 != 1 {
|
||||||
|
t.Errorf("live routes (IPv4, IPv6): (%d, %d) in memory, (%d, %d) in "+
|
||||||
|
"the tables, want (1, 1)",
|
||||||
|
stats.IPv4Routes, stats.IPv6Routes, tableV4, tableV6)
|
||||||
|
}
|
||||||
|
|
||||||
|
queryV4, queryV6, err := db.GetPrefixDistributionContext(ctx)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("GetPrefixDistributionContext: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
wantV4 := []PrefixDistribution{{MaskLength: 24, Count: 1}}
|
||||||
|
wantV6 := []PrefixDistribution{{MaskLength: 32, Count: 1}}
|
||||||
|
|
||||||
|
assertDistribution(t, "IPv4 distribution", stats.IPv4PrefixDistribution, wantV4)
|
||||||
|
assertDistribution(t, "IPv6 distribution", stats.IPv6PrefixDistribution, wantV6)
|
||||||
|
assertDistribution(t, "IPv4 distribution query", queryV4, wantV4)
|
||||||
|
assertDistribution(t, "IPv6 distribution query", queryV6, wantV6)
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestStatsRouteTimestamps checks the oldest/newest route timestamps are read
|
||||||
|
// from the right rows across both tables and parse into time.Time. The old
|
||||||
|
// MIN/MAX union query read its result into *time.Time, which the driver could
|
||||||
|
// not parse, so it logged a warning every call and left both timestamps nil.
|
||||||
|
func TestStatsRouteTimestamps(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
cfg := &config.Config{StateDir: t.TempDir()}
|
||||||
|
|
||||||
|
db, err := New(cfg, logger.New())
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("failed to create database: %v", err)
|
||||||
|
}
|
||||||
|
defer func() { _ = db.Close() }()
|
||||||
|
|
||||||
|
ctx := context.Background()
|
||||||
|
|
||||||
|
empty, err := db.GetStatsContext(ctx)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("GetStatsContext on empty database: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
if empty.OldestRoute != nil || empty.NewestRoute != nil {
|
||||||
|
t.Fatalf("empty database timestamps = (%v, %v), want (nil, nil)",
|
||||||
|
empty.OldestRoute, empty.NewestRoute)
|
||||||
|
}
|
||||||
|
|
||||||
|
base := time.Date(2026, 1, 2, 3, 4, 5, 0, time.UTC)
|
||||||
|
oldest := base
|
||||||
|
newest := base.Add(2 * time.Minute)
|
||||||
|
|
||||||
|
err = db.UpsertLiveRouteBatch([]*LiveRoute{
|
||||||
|
mkV4Route(t, "198.51.100.0/24", 64500, base.Add(time.Minute)),
|
||||||
|
mkV4Route(t, "203.0.113.0/24", 64501, oldest),
|
||||||
|
mkV6Route(testV6Prefix, 64502, newest),
|
||||||
|
})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("UpsertLiveRouteBatch: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
stats, err := db.GetStatsContext(ctx)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("GetStatsContext: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
if stats.OldestRoute == nil || !stats.OldestRoute.Equal(oldest) {
|
||||||
|
t.Errorf("OldestRoute = %v, want %v", stats.OldestRoute, oldest)
|
||||||
|
}
|
||||||
|
|
||||||
|
if stats.NewestRoute == nil || !stats.NewestRoute.Equal(newest) {
|
||||||
|
t.Errorf("NewestRoute = %v, want %v", stats.NewestRoute, newest)
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// TestLiveCountsConcurrentReadWrite runs writers and stats readers at once so
|
// TestLiveCountsConcurrentReadWrite runs writers and stats readers at once so
|
||||||
@@ -145,10 +726,7 @@ func TestLiveCountsConcurrentReadWrite(t *testing.T) {
|
|||||||
|
|
||||||
ts := time.Date(2026, 1, 2, 3, 4, 5, 0, time.UTC)
|
ts := time.Date(2026, 1, 2, 3, 4, 5, 0, time.UTC)
|
||||||
|
|
||||||
const (
|
const writers = 4
|
||||||
writers = 4
|
|
||||||
prefixesPerWrite = 25
|
|
||||||
)
|
|
||||||
|
|
||||||
var wg sync.WaitGroup
|
var wg sync.WaitGroup
|
||||||
wg.Add(writers)
|
wg.Add(writers)
|
||||||
@@ -157,12 +735,14 @@ func TestLiveCountsConcurrentReadWrite(t *testing.T) {
|
|||||||
go func(base int) {
|
go func(base int) {
|
||||||
defer wg.Done()
|
defer wg.Done()
|
||||||
|
|
||||||
for i := range prefixesPerWrite {
|
for i := range 25 {
|
||||||
prefix := fmt.Sprintf("10.%d.%d.0/24", base, i)
|
asn := 65000 + base*100 + i
|
||||||
|
route := mkV6Route(testV6Prefix, asn, ts)
|
||||||
|
route.PeerIP = "2001:db8::" + uuid.NewString()
|
||||||
|
|
||||||
err := db.UpdatePrefixesBatch(map[string]time.Time{prefix: ts})
|
err := db.UpsertLiveRoute(route)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
t.Errorf("UpdatePrefixesBatch: %v", err)
|
t.Errorf("UpsertLiveRoute: %v", err)
|
||||||
|
|
||||||
return
|
return
|
||||||
}
|
}
|
||||||
@@ -202,16 +782,22 @@ func TestLiveCountsConcurrentReadWrite(t *testing.T) {
|
|||||||
t.Fatalf("GetStatsContext: %v", err)
|
t.Fatalf("GetStatsContext: %v", err)
|
||||||
}
|
}
|
||||||
|
|
||||||
if want := writers * prefixesPerWrite; stats.IPv4Prefixes != want {
|
if want := writers * 25; stats.IPv6Routes != want {
|
||||||
t.Errorf("IPv4Prefixes = %d, want %d", stats.IPv4Prefixes, want)
|
t.Errorf("IPv6Routes = %d, want %d", stats.IPv6Routes, want)
|
||||||
}
|
}
|
||||||
|
// Every writer announced the same prefix, so it counts once.
|
||||||
|
assertDistribution(t, "IPv6 distribution", stats.IPv6PrefixDistribution,
|
||||||
|
[]PrefixDistribution{{MaskLength: 32, Count: 1}})
|
||||||
}
|
}
|
||||||
|
|
||||||
type wantCounts struct {
|
type wantCounts struct {
|
||||||
asns int
|
asns int
|
||||||
prefixes int
|
prefixes int
|
||||||
peerings int
|
peerings int
|
||||||
peers int
|
peers int
|
||||||
|
ipv4Routes int
|
||||||
|
ipv6Routes int
|
||||||
|
liveRoutes int
|
||||||
}
|
}
|
||||||
|
|
||||||
func assertCounts(t *testing.T, when string, got Stats, want wantCounts) {
|
func assertCounts(t *testing.T, when string, got Stats, want wantCounts) {
|
||||||
@@ -232,4 +818,24 @@ func assertCounts(t *testing.T, when string, got Stats, want wantCounts) {
|
|||||||
if got.Peers != want.peers {
|
if got.Peers != want.peers {
|
||||||
t.Errorf("%s: Peers = %d, want %d", when, got.Peers, want.peers)
|
t.Errorf("%s: Peers = %d, want %d", when, got.Peers, want.peers)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
if got.IPv4Routes != want.ipv4Routes {
|
||||||
|
t.Errorf("%s: IPv4Routes = %d, want %d", when, got.IPv4Routes, want.ipv4Routes)
|
||||||
|
}
|
||||||
|
|
||||||
|
if got.IPv6Routes != want.ipv6Routes {
|
||||||
|
t.Errorf("%s: IPv6Routes = %d, want %d", when, got.IPv6Routes, want.ipv6Routes)
|
||||||
|
}
|
||||||
|
|
||||||
|
if got.LiveRoutes != want.liveRoutes {
|
||||||
|
t.Errorf("%s: LiveRoutes = %d, want %d", when, got.LiveRoutes, want.liveRoutes)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func assertDistribution(t *testing.T, what string, got, want []PrefixDistribution) {
|
||||||
|
t.Helper()
|
||||||
|
|
||||||
|
if !slices.Equal(got, want) {
|
||||||
|
t.Errorf("%s = %v, want %v", what, got, want)
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
+1559
-139
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
@@ -1,31 +1,34 @@
|
|||||||
// Package database provides SQLite storage for BGP routing data: ASNs,
|
// Package database provides SQLite storage for BGP routing data including ASNs,
|
||||||
// prefixes, announcements, peerings and BGP peers. Its routes table holds a
|
// prefixes, announcements, peerings, and live route tables.
|
||||||
// copy of the whole routing table that internal/routetable keeps in memory.
|
|
||||||
package database
|
package database
|
||||||
|
|
||||||
import (
|
import (
|
||||||
"context"
|
"context"
|
||||||
"net/netip"
|
|
||||||
"time"
|
"time"
|
||||||
|
|
||||||
"git.eeqj.de/sneak/routewatch/internal/routetable"
|
|
||||||
)
|
)
|
||||||
|
|
||||||
// Stats contains database statistics: counts of ASNs, prefixes, peerings and
|
// Stats contains database statistics including counts for ASNs, prefixes,
|
||||||
// peers, and the file size.
|
// peerings, peers, and live routes, as well as file size and prefix distribution data.
|
||||||
type Stats struct {
|
type Stats struct {
|
||||||
ASNs int
|
ASNs int
|
||||||
Prefixes int
|
Prefixes int
|
||||||
IPv4Prefixes int
|
IPv4Prefixes int
|
||||||
IPv6Prefixes int
|
IPv6Prefixes int
|
||||||
Peerings int
|
Peerings int
|
||||||
Peers int
|
Peers int
|
||||||
FileSizeBytes int64
|
FileSizeBytes int64
|
||||||
|
LiveRoutes int
|
||||||
|
IPv4Routes int
|
||||||
|
IPv6Routes int
|
||||||
|
OldestRoute *time.Time
|
||||||
|
NewestRoute *time.Time
|
||||||
|
IPv4PrefixDistribution []PrefixDistribution
|
||||||
|
IPv6PrefixDistribution []PrefixDistribution
|
||||||
}
|
}
|
||||||
|
|
||||||
// Store defines the interface for database operations. It provides methods for
|
// Store defines the interface for database operations. It provides methods for
|
||||||
// managing ASNs, prefixes, announcements, peerings, BGP peers, and the copy of
|
// managing ASNs, prefixes, announcements, peerings, BGP peers, and live routes.
|
||||||
// the routing table. Implementations must be safe for concurrent use.
|
// Implementations must be safe for concurrent use.
|
||||||
//
|
//
|
||||||
//nolint:interfacebloat // the one seam handlers and tests depend on
|
//nolint:interfacebloat // the one seam handlers and tests depend on
|
||||||
type Store interface {
|
type Store interface {
|
||||||
@@ -51,9 +54,22 @@ type Store interface {
|
|||||||
UpdatePeer(peerIP string, peerASN int, messageType string, timestamp time.Time) error
|
UpdatePeer(peerIP string, peerASN int, messageType string, timestamp time.Time) error
|
||||||
UpdatePeerBatch(peers map[string]PeerUpdate) error
|
UpdatePeerBatch(peers map[string]PeerUpdate) error
|
||||||
|
|
||||||
// Routing table operations: the database's copy of the table in memory
|
// Live route operations
|
||||||
WriteRouteChanges(changes []routetable.Change, newPeers map[string]uint) error
|
UpsertLiveRoute(route *LiveRoute) error
|
||||||
ReadRouteTable(ctx context.Context) (*routetable.Table, error)
|
UpsertLiveRouteBatch(routes []*LiveRoute) error
|
||||||
|
DeleteLiveRoute(prefix, peerIP string) error
|
||||||
|
DeleteLiveRouteBatch(deletions []LiveRouteDeletion) error
|
||||||
|
DeleteLiveRoutesOfPeer(peerIP string) (deleted int, err error)
|
||||||
|
GetPrefixDistribution() (
|
||||||
|
ipv4 []PrefixDistribution, ipv6 []PrefixDistribution, err error)
|
||||||
|
GetPrefixDistributionContext(ctx context.Context) (
|
||||||
|
ipv4 []PrefixDistribution, ipv6 []PrefixDistribution, err error)
|
||||||
|
GetLiveRouteCounts() (ipv4Count, ipv6Count int, err error)
|
||||||
|
GetLiveRouteCountsContext(ctx context.Context) (ipv4Count, ipv6Count int, err error)
|
||||||
|
|
||||||
|
// IP lookup operations
|
||||||
|
GetIPInfo(ip string) (*IPInfo, error)
|
||||||
|
GetIPInfoContext(ctx context.Context, ip string) (*IPInfo, error)
|
||||||
|
|
||||||
// ASN WHOIS operations
|
// ASN WHOIS operations
|
||||||
GetNextStaleASN(ctx context.Context, staleThreshold time.Duration) (int, error)
|
GetNextStaleASN(ctx context.Context, staleThreshold time.Duration) (int, error)
|
||||||
@@ -61,11 +77,18 @@ type Store interface {
|
|||||||
GetWHOISStats(ctx context.Context, staleThreshold time.Duration) (*WHOISStats, error)
|
GetWHOISStats(ctx context.Context, staleThreshold time.Duration) (*WHOISStats, error)
|
||||||
|
|
||||||
// AS and prefix detail operations
|
// AS and prefix detail operations
|
||||||
GetAS(ctx context.Context, asn int) (*ASN, bool, error)
|
GetASDetails(asn int) (*ASN, []LiveRoute, error)
|
||||||
GetASWHOIS(ctx context.Context, asn int) (*ASN, error)
|
GetASDetailsContext(ctx context.Context, asn int) (*ASN, []LiveRoute, error)
|
||||||
GetPrefixFirstSeen(ctx context.Context, prefix netip.Prefix) (time.Time, error)
|
|
||||||
GetASPeers(asn int) ([]ASPeer, error)
|
GetASPeers(asn int) ([]ASPeer, error)
|
||||||
GetASPeersContext(ctx context.Context, asn int) ([]ASPeer, error)
|
GetASPeersContext(ctx context.Context, asn int) ([]ASPeer, error)
|
||||||
|
GetPrefixDetails(prefix string) ([]LiveRoute, map[int]ASN, error)
|
||||||
|
GetPrefixDetailsContext(
|
||||||
|
ctx context.Context, prefix string,
|
||||||
|
) ([]LiveRoute, map[int]ASN, error)
|
||||||
|
GetRandomPrefixesByLength(maskLength, ipVersion, limit int) ([]LiveRoute, error)
|
||||||
|
GetRandomPrefixesByLengthContext(
|
||||||
|
ctx context.Context, maskLength, ipVersion, limit int,
|
||||||
|
) ([]LiveRoute, error)
|
||||||
|
|
||||||
// Lifecycle
|
// Lifecycle
|
||||||
Close() error
|
Close() error
|
||||||
|
|||||||
@@ -66,10 +66,67 @@ type ASNPeering struct {
|
|||||||
LastSeen time.Time
|
LastSeen time.Time
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// LiveRoute represents a route in the live routing table
|
||||||
|
//
|
||||||
|
//nolint:tagliatelle // field names are the existing JSON API
|
||||||
|
type LiveRoute struct {
|
||||||
|
Prefix string `json:"prefix"`
|
||||||
|
MaskLength int `json:"mask_length"`
|
||||||
|
IPVersion int `json:"ip_version"`
|
||||||
|
OriginASN int `json:"origin_asn"`
|
||||||
|
PeerIP string `json:"peer_ip"`
|
||||||
|
ASPath []int `json:"as_path"`
|
||||||
|
NextHop string `json:"next_hop"`
|
||||||
|
LastUpdated time.Time `json:"last_updated"`
|
||||||
|
}
|
||||||
|
|
||||||
|
// PrefixDistribution represents the distribution of prefixes by mask length
|
||||||
|
//
|
||||||
|
//nolint:tagliatelle // field names are the existing JSON API
|
||||||
|
type PrefixDistribution struct {
|
||||||
|
MaskLength int `json:"mask_length"`
|
||||||
|
Count int `json:"count"`
|
||||||
|
}
|
||||||
|
|
||||||
|
// IPInfo represents comprehensive IP information for the /ip endpoint
|
||||||
|
//
|
||||||
|
//nolint:tagliatelle // field names are the existing JSON API
|
||||||
|
type IPInfo struct {
|
||||||
|
IP string `json:"ip"`
|
||||||
|
PTR []string `json:"ptr,omitempty"`
|
||||||
|
Netblock string `json:"netblock"`
|
||||||
|
MaskLength int `json:"mask_length"`
|
||||||
|
IPVersion int `json:"ip_version"`
|
||||||
|
NumPeers int `json:"num_peers"`
|
||||||
|
// AS information
|
||||||
|
ASN int `json:"asn"`
|
||||||
|
ASName string `json:"as_name,omitempty"`
|
||||||
|
Handle string `json:"handle,omitempty"`
|
||||||
|
Description string `json:"description,omitempty"`
|
||||||
|
OrgName string `json:"org_name,omitempty"`
|
||||||
|
OrgID string `json:"org_id,omitempty"`
|
||||||
|
Address string `json:"address,omitempty"`
|
||||||
|
CountryCode string `json:"country_code,omitempty"`
|
||||||
|
AbuseEmail string `json:"abuse_email,omitempty"`
|
||||||
|
RIR string `json:"rir,omitempty"`
|
||||||
|
// Timestamps
|
||||||
|
FirstSeen time.Time `json:"first_seen"`
|
||||||
|
LastSeen time.Time `json:"last_seen"`
|
||||||
|
// Indicates if WHOIS data needs refresh (not serialized)
|
||||||
|
NeedsWHOISRefresh bool `json:"-"`
|
||||||
|
}
|
||||||
|
|
||||||
|
// LiveRouteDeletion represents parameters for deleting a live route
|
||||||
|
type LiveRouteDeletion struct {
|
||||||
|
Prefix string
|
||||||
|
PeerIP string
|
||||||
|
IPVersion int
|
||||||
|
}
|
||||||
|
|
||||||
// SeenTimes is the earliest and the latest time one batch of updates saw an AS,
|
// SeenTimes is the earliest and the latest time one batch of updates saw an AS,
|
||||||
// a peer or a peering. A row the batch adds is stored first seen at First and
|
// a peer or a peering. A row the batch adds is stored first seen at First and
|
||||||
// last seen at Last. A row already stored takes First as its first_seen only
|
// last seen at Last; a row already stored keeps its first_seen and has its
|
||||||
// when First is earlier, and Last as its last_seen only when Last is later.
|
// last_seen set to Last.
|
||||||
type SeenTimes struct {
|
type SeenTimes struct {
|
||||||
First time.Time
|
First time.Time
|
||||||
Last time.Time
|
Last time.Time
|
||||||
|
|||||||
@@ -1,339 +0,0 @@
|
|||||||
package database
|
|
||||||
|
|
||||||
import (
|
|
||||||
"context"
|
|
||||||
"database/sql"
|
|
||||||
"encoding/json"
|
|
||||||
"errors"
|
|
||||||
"fmt"
|
|
||||||
"net/netip"
|
|
||||||
"time"
|
|
||||||
|
|
||||||
"github.com/bits-and-blooms/bitset"
|
|
||||||
|
|
||||||
"git.eeqj.de/sneak/routewatch/internal/routetable"
|
|
||||||
)
|
|
||||||
|
|
||||||
// keptRoutesPerRow is how many kept routes a row of the routes table has
|
|
||||||
// columns for: the newest and the older one.
|
|
||||||
const keptRoutesPerRow = 2
|
|
||||||
|
|
||||||
// errUnknownRoutePeer is returned for a peer that route_peers lacks: on a
|
|
||||||
// write, the peer of a kept route; on a read, a peer number in a row.
|
|
||||||
var errUnknownRoutePeer = errors.New("peer not in route_peers")
|
|
||||||
|
|
||||||
// The statements of the routes and route_peers tables (schema.sql).
|
|
||||||
const (
|
|
||||||
insertRoutePeerSQL = `INSERT INTO route_peers (number, address)
|
|
||||||
VALUES (?, ?)`
|
|
||||||
routePeerNumberSQL = `SELECT number FROM route_peers WHERE address = ?`
|
|
||||||
routePeersSQL = `SELECT number, address FROM route_peers`
|
|
||||||
|
|
||||||
upsertRouteSQL = `INSERT INTO routes (prefix, origin_asn, peers,
|
|
||||||
newest_peer, newest_as_path, newest_time,
|
|
||||||
older_peer, older_as_path, older_time)
|
|
||||||
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?)
|
|
||||||
ON CONFLICT (prefix, origin_asn) DO UPDATE SET peers = excluded.peers,
|
|
||||||
newest_peer = excluded.newest_peer,
|
|
||||||
newest_as_path = excluded.newest_as_path,
|
|
||||||
newest_time = excluded.newest_time,
|
|
||||||
older_peer = excluded.older_peer,
|
|
||||||
older_as_path = excluded.older_as_path,
|
|
||||||
older_time = excluded.older_time`
|
|
||||||
deleteRouteSQL = `DELETE FROM routes WHERE prefix = ? AND origin_asn = ?`
|
|
||||||
routesSQL = `SELECT prefix, origin_asn, peers,
|
|
||||||
newest_peer, newest_as_path, newest_time,
|
|
||||||
older_peer, older_as_path, older_time FROM routes`
|
|
||||||
)
|
|
||||||
|
|
||||||
// WriteRouteChanges writes, in one transaction, what one call of the routing
|
|
||||||
// table's TakeChanges returned. It stores each new peer with its number, writes
|
|
||||||
// the row of each changed prefix and origin AS pair, and deletes the row of
|
|
||||||
// each removed one. Route times are stored to the second, as the feed gives
|
|
||||||
// them.
|
|
||||||
//
|
|
||||||
// When it fails, it writes nothing. The caller then writes these changes and
|
|
||||||
// peers again, because later changes can name the peers it did not store.
|
|
||||||
func (d *Database) WriteRouteChanges(
|
|
||||||
changes []routetable.Change, newPeers map[string]uint,
|
|
||||||
) error {
|
|
||||||
if len(changes) == 0 && len(newPeers) == 0 {
|
|
||||||
return nil
|
|
||||||
}
|
|
||||||
|
|
||||||
d.lock("WriteRouteChanges")
|
|
||||||
defer d.unlock()
|
|
||||||
|
|
||||||
tx, err := d.beginTx()
|
|
||||||
if err != nil {
|
|
||||||
return fmt.Errorf("failed to begin transaction: %w", err)
|
|
||||||
}
|
|
||||||
defer d.rollback(tx)
|
|
||||||
|
|
||||||
ctx := context.Background()
|
|
||||||
|
|
||||||
// The new peers first, so that the rows can look up their numbers.
|
|
||||||
for address, number := range newPeers {
|
|
||||||
_, err := tx.ExecContext(ctx, insertRoutePeerSQL, number, address)
|
|
||||||
if err != nil {
|
|
||||||
return fmt.Errorf("failed to store peer %s: %w", address, err)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
err = writeRouteRows(ctx, tx, changes)
|
|
||||||
if err != nil {
|
|
||||||
return err
|
|
||||||
}
|
|
||||||
|
|
||||||
err = tx.Commit()
|
|
||||||
if err != nil {
|
|
||||||
return fmt.Errorf("failed to commit transaction: %w", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
return nil
|
|
||||||
}
|
|
||||||
|
|
||||||
// writeRouteRows upserts the row of each change in tx, or deletes it for a
|
|
||||||
// removed pair.
|
|
||||||
func writeRouteRows(
|
|
||||||
ctx context.Context, tx *loggingTx, changes []routetable.Change,
|
|
||||||
) error {
|
|
||||||
upsert, err := tx.PrepareContext(ctx, upsertRouteSQL)
|
|
||||||
if err != nil {
|
|
||||||
return fmt.Errorf("failed to prepare route upsert: %w", err)
|
|
||||||
}
|
|
||||||
defer func() { _ = upsert.Close() }()
|
|
||||||
|
|
||||||
del, err := tx.PrepareContext(ctx, deleteRouteSQL)
|
|
||||||
if err != nil {
|
|
||||||
return fmt.Errorf("failed to prepare route delete: %w", err)
|
|
||||||
}
|
|
||||||
defer func() { _ = del.Close() }()
|
|
||||||
|
|
||||||
peerNumber, err := tx.PrepareContext(ctx, routePeerNumberSQL)
|
|
||||||
if err != nil {
|
|
||||||
return fmt.Errorf("failed to prepare peer number lookup: %w", err)
|
|
||||||
}
|
|
||||||
defer func() { _ = peerNumber.Close() }()
|
|
||||||
|
|
||||||
for _, change := range changes {
|
|
||||||
prefix := change.Prefix.String()
|
|
||||||
|
|
||||||
if change.Removed {
|
|
||||||
_, err := del.ExecContext(ctx, prefix, change.OriginASN)
|
|
||||||
if err != nil {
|
|
||||||
return fmt.Errorf("failed to delete route %s: %w", prefix, err)
|
|
||||||
}
|
|
||||||
|
|
||||||
continue
|
|
||||||
}
|
|
||||||
|
|
||||||
row, err := routeRow(ctx, peerNumber, change)
|
|
||||||
if err != nil {
|
|
||||||
return fmt.Errorf("failed to write route %s: %w", prefix, err)
|
|
||||||
}
|
|
||||||
|
|
||||||
_, err = upsert.ExecContext(ctx, row...)
|
|
||||||
if err != nil {
|
|
||||||
return fmt.Errorf("failed to write route %s: %w", prefix, err)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
return nil
|
|
||||||
}
|
|
||||||
|
|
||||||
// routeRow returns the values of the routes row of change, in the order of
|
|
||||||
// upsertRouteSQL. peerNumber looks up a peer's number by its address. The
|
|
||||||
// table keeps at most keptRoutesPerRow routes of a pair.
|
|
||||||
func routeRow(
|
|
||||||
ctx context.Context, peerNumber *sql.Stmt, change routetable.Change,
|
|
||||||
) ([]any, error) {
|
|
||||||
peers, err := change.Peers.MarshalBinary()
|
|
||||||
if err != nil {
|
|
||||||
return nil, err
|
|
||||||
}
|
|
||||||
|
|
||||||
row := []any{change.Prefix.String(), change.OriginASN, peers}
|
|
||||||
|
|
||||||
for i := range keptRoutesPerRow {
|
|
||||||
if i >= len(change.Routes) {
|
|
||||||
row = append(row, nil, nil, nil)
|
|
||||||
|
|
||||||
continue
|
|
||||||
}
|
|
||||||
|
|
||||||
r := change.Routes[i]
|
|
||||||
|
|
||||||
var number uint
|
|
||||||
|
|
||||||
err := peerNumber.QueryRowContext(ctx, r.Peer).Scan(&number)
|
|
||||||
if errors.Is(err, sql.ErrNoRows) {
|
|
||||||
return nil, fmt.Errorf("%w: %s", errUnknownRoutePeer, r.Peer)
|
|
||||||
}
|
|
||||||
|
|
||||||
if err != nil {
|
|
||||||
return nil, err
|
|
||||||
}
|
|
||||||
|
|
||||||
path, err := json.Marshal(r.ASPath)
|
|
||||||
if err != nil {
|
|
||||||
return nil, err
|
|
||||||
}
|
|
||||||
|
|
||||||
row = append(row, number, string(path), r.Time.Unix())
|
|
||||||
}
|
|
||||||
|
|
||||||
return row, nil
|
|
||||||
}
|
|
||||||
|
|
||||||
// ReadRouteTable returns the routing table that the routes and route_peers
|
|
||||||
// tables hold, as startup reads it. It reads the peers first, so that the
|
|
||||||
// table keeps their numbers, then loads each row into the table as it reads
|
|
||||||
// it. The two reads are separate, so nothing may write the tables meanwhile.
|
|
||||||
// It fails on a row that names a peer number route_peers lacks.
|
|
||||||
func (d *Database) ReadRouteTable(ctx context.Context) (*routetable.Table, error) {
|
|
||||||
peerNumbers, err := d.readRoutePeers(ctx)
|
|
||||||
if err != nil {
|
|
||||||
return nil, err
|
|
||||||
}
|
|
||||||
|
|
||||||
addresses := make(map[uint]string, len(peerNumbers))
|
|
||||||
numbers := new(bitset.BitSet)
|
|
||||||
|
|
||||||
for address, number := range peerNumbers {
|
|
||||||
addresses[number] = address
|
|
||||||
numbers.Set(number)
|
|
||||||
}
|
|
||||||
|
|
||||||
table := routetable.New(routetable.Params{PeerNumbers: peerNumbers})
|
|
||||||
|
|
||||||
rows, err := d.db.QueryContext(ctx, routesSQL)
|
|
||||||
if err != nil {
|
|
||||||
return nil, fmt.Errorf("failed to read routes: %w", err)
|
|
||||||
}
|
|
||||||
defer func() { _ = rows.Close() }()
|
|
||||||
|
|
||||||
for rows.Next() {
|
|
||||||
change, err := scanRouteRow(rows, addresses, numbers)
|
|
||||||
if err != nil {
|
|
||||||
return nil, err
|
|
||||||
}
|
|
||||||
|
|
||||||
table.Load(change)
|
|
||||||
}
|
|
||||||
|
|
||||||
err = rows.Err()
|
|
||||||
if err != nil {
|
|
||||||
return nil, fmt.Errorf("failed to read routes: %w", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
return table, nil
|
|
||||||
}
|
|
||||||
|
|
||||||
// readRoutePeers returns the number of each peer in route_peers, by address.
|
|
||||||
func (d *Database) readRoutePeers(ctx context.Context) (map[string]uint, error) {
|
|
||||||
rows, err := d.db.QueryContext(ctx, routePeersSQL)
|
|
||||||
if err != nil {
|
|
||||||
return nil, fmt.Errorf("failed to read route peers: %w", err)
|
|
||||||
}
|
|
||||||
defer func() { _ = rows.Close() }()
|
|
||||||
|
|
||||||
peerNumbers := make(map[string]uint)
|
|
||||||
|
|
||||||
for rows.Next() {
|
|
||||||
var (
|
|
||||||
number uint
|
|
||||||
address string
|
|
||||||
)
|
|
||||||
|
|
||||||
err := rows.Scan(&number, &address)
|
|
||||||
if err != nil {
|
|
||||||
return nil, fmt.Errorf("failed to read a route peer: %w", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
peerNumbers[address] = number
|
|
||||||
}
|
|
||||||
|
|
||||||
err = rows.Err()
|
|
||||||
if err != nil {
|
|
||||||
return nil, fmt.Errorf("failed to read route peers: %w", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
return peerNumbers, nil
|
|
||||||
}
|
|
||||||
|
|
||||||
// keptRouteColumns are the three columns of one kept route in a routes row.
|
|
||||||
type keptRouteColumns struct {
|
|
||||||
peer sql.Null[uint]
|
|
||||||
asPath sql.NullString
|
|
||||||
time sql.NullInt64
|
|
||||||
}
|
|
||||||
|
|
||||||
// scanRouteRow returns the content of the routes row rows is at, with each kept
|
|
||||||
// route's peer named by its address in addresses. It fails when the row names a
|
|
||||||
// peer number that addresses lacks. numbers holds the numbers of addresses as a
|
|
||||||
// set, against which a row's whole peer set is checked at once.
|
|
||||||
func scanRouteRow(
|
|
||||||
rows *sql.Rows, addresses map[uint]string, numbers *bitset.BitSet,
|
|
||||||
) (routetable.Change, error) {
|
|
||||||
var (
|
|
||||||
change routetable.Change
|
|
||||||
prefix string
|
|
||||||
peers []byte
|
|
||||||
kept [keptRoutesPerRow]keptRouteColumns
|
|
||||||
)
|
|
||||||
|
|
||||||
err := rows.Scan(&prefix, &change.OriginASN, &peers,
|
|
||||||
&kept[0].peer, &kept[0].asPath, &kept[0].time,
|
|
||||||
&kept[1].peer, &kept[1].asPath, &kept[1].time)
|
|
||||||
if err != nil {
|
|
||||||
return change, fmt.Errorf("failed to read a route: %w", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
change.Prefix, err = netip.ParsePrefix(prefix)
|
|
||||||
if err != nil {
|
|
||||||
return change, fmt.Errorf("failed to read route %s: %w", prefix, err)
|
|
||||||
}
|
|
||||||
|
|
||||||
change.Peers = new(bitset.BitSet)
|
|
||||||
|
|
||||||
err = change.Peers.UnmarshalBinary(peers)
|
|
||||||
if err != nil {
|
|
||||||
return change, fmt.Errorf("failed to read the peers of route %s: %w",
|
|
||||||
prefix, err)
|
|
||||||
}
|
|
||||||
|
|
||||||
if !numbers.IsSuperSet(change.Peers) {
|
|
||||||
unknown, _ := change.Peers.Difference(numbers).NextSet(0)
|
|
||||||
|
|
||||||
return change, fmt.Errorf("%w: number %d in the peers of route %s",
|
|
||||||
errUnknownRoutePeer, unknown, prefix)
|
|
||||||
}
|
|
||||||
|
|
||||||
for _, columns := range kept {
|
|
||||||
if !columns.peer.Valid {
|
|
||||||
continue
|
|
||||||
}
|
|
||||||
|
|
||||||
address, known := addresses[columns.peer.V]
|
|
||||||
if !known {
|
|
||||||
return change, fmt.Errorf("%w: number %d in route %s",
|
|
||||||
errUnknownRoutePeer, columns.peer.V, prefix)
|
|
||||||
}
|
|
||||||
|
|
||||||
r := routetable.Route{
|
|
||||||
Peer: address,
|
|
||||||
Time: time.Unix(columns.time.Int64, 0).UTC(),
|
|
||||||
}
|
|
||||||
|
|
||||||
err := json.Unmarshal([]byte(columns.asPath.String), &r.ASPath)
|
|
||||||
if err != nil {
|
|
||||||
return change, fmt.Errorf("failed to read an AS path of route %s: %w",
|
|
||||||
prefix, err)
|
|
||||||
}
|
|
||||||
|
|
||||||
change.Routes = append(change.Routes, r)
|
|
||||||
}
|
|
||||||
|
|
||||||
return change, nil
|
|
||||||
}
|
|
||||||
@@ -1,527 +0,0 @@
|
|||||||
package database
|
|
||||||
|
|
||||||
import (
|
|
||||||
"errors"
|
|
||||||
"fmt"
|
|
||||||
"math"
|
|
||||||
"math/rand/v2"
|
|
||||||
"net/netip"
|
|
||||||
"os"
|
|
||||||
"reflect"
|
|
||||||
"slices"
|
|
||||||
"testing"
|
|
||||||
"time"
|
|
||||||
|
|
||||||
"github.com/bits-and-blooms/bitset"
|
|
||||||
|
|
||||||
"git.eeqj.de/sneak/routewatch/internal/config"
|
|
||||||
"git.eeqj.de/sneak/routewatch/internal/logger"
|
|
||||||
"git.eeqj.de/sneak/routewatch/internal/routetable"
|
|
||||||
)
|
|
||||||
|
|
||||||
// Peers, origin ASes and prefixes of the routes table tests.
|
|
||||||
const (
|
|
||||||
peerA = "192.0.2.1"
|
|
||||||
peerB = "192.0.2.2"
|
|
||||||
peerC = "2001:db8::3"
|
|
||||||
peerD = "2001:db8::4"
|
|
||||||
peerE = "2001:db8::5"
|
|
||||||
|
|
||||||
asA = 64500
|
|
||||||
asB = 64501
|
|
||||||
// transitAS is the first AS of every test path, the peer's own.
|
|
||||||
transitAS = 64496
|
|
||||||
|
|
||||||
tablePrefixV4 = "10.1.2.0/24"
|
|
||||||
tablePrefixShortV4 = "10.9.0.0/16"
|
|
||||||
tablePrefixV6 = "2001:db8:1::/48"
|
|
||||||
)
|
|
||||||
|
|
||||||
// tableTime returns the time second seconds after the start of every routes
|
|
||||||
// table test.
|
|
||||||
func tableTime(second int) time.Time {
|
|
||||||
return time.Date(2026, 1, 2, 3, 4, 5, 0, time.UTC).
|
|
||||||
Add(time.Duration(second) * time.Second)
|
|
||||||
}
|
|
||||||
|
|
||||||
// announceTo has peer announce prefix with origin AS asn second seconds after
|
|
||||||
// the start.
|
|
||||||
func announceTo(
|
|
||||||
table *routetable.Table, prefix string, asn int, peer string, second int,
|
|
||||||
) {
|
|
||||||
table.Announce(netip.MustParsePrefix(prefix), asn, routetable.Route{
|
|
||||||
Peer: peer, ASPath: []int{transitAS, asn}, Time: tableTime(second),
|
|
||||||
})
|
|
||||||
}
|
|
||||||
|
|
||||||
// newRoutesTestDatabase returns a new database in a temporary directory.
|
|
||||||
func newRoutesTestDatabase(t *testing.T) *Database {
|
|
||||||
t.Helper()
|
|
||||||
|
|
||||||
db, err := New(&config.Config{StateDir: t.TempDir()}, logger.New())
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("failed to create database: %v", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
t.Cleanup(func() { _ = db.Close() })
|
|
||||||
|
|
||||||
return db
|
|
||||||
}
|
|
||||||
|
|
||||||
// writeTableChanges writes what table changed since the last call.
|
|
||||||
func writeTableChanges(t *testing.T, db *Database, table *routetable.Table) {
|
|
||||||
t.Helper()
|
|
||||||
|
|
||||||
changes, newPeers := table.TakeChanges(math.MaxInt)
|
|
||||||
|
|
||||||
err := db.WriteRouteChanges(changes, newPeers)
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("WriteRouteChanges: %v", err)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestReadRouteTableGivesBackWhatWasWritten(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
db := newRoutesTestDatabase(t)
|
|
||||||
source := routetable.New(routetable.Params{})
|
|
||||||
|
|
||||||
// Two origins of one prefix: the routes of peerC and peerB are kept for
|
|
||||||
// asA, peerA keeps only its bit.
|
|
||||||
announceTo(source, tablePrefixV4, asA, peerA, 10)
|
|
||||||
announceTo(source, tablePrefixV4, asA, peerB, 20)
|
|
||||||
announceTo(source, tablePrefixV4, asA, peerC, 30)
|
|
||||||
announceTo(source, tablePrefixV4, asB, peerD, 40)
|
|
||||||
// Two routes with the same time.
|
|
||||||
announceTo(source, tablePrefixV6, asA, peerA, 50)
|
|
||||||
announceTo(source, tablePrefixV6, asA, peerC, 50)
|
|
||||||
// An origin whose kept routes were withdrawn while peerA carries it.
|
|
||||||
announceTo(source, tablePrefixShortV4, asB, peerA, 10)
|
|
||||||
announceTo(source, tablePrefixShortV4, asB, peerB, 20)
|
|
||||||
announceTo(source, tablePrefixShortV4, asB, peerC, 30)
|
|
||||||
source.Withdraw(netip.MustParsePrefix(tablePrefixShortV4), peerB)
|
|
||||||
source.Withdraw(netip.MustParsePrefix(tablePrefixShortV4), peerC)
|
|
||||||
|
|
||||||
writeTableChanges(t, db, source)
|
|
||||||
checkReadTable(t, db, source)
|
|
||||||
|
|
||||||
// A peer moves to another origin, one goes down and a new one comes.
|
|
||||||
announceTo(source, tablePrefixV4, asB, peerA, 60)
|
|
||||||
source.PeerDown(peerC)
|
|
||||||
announceTo(source, tablePrefixV6, asB, peerE, 70)
|
|
||||||
|
|
||||||
writeTableChanges(t, db, source)
|
|
||||||
checkReadTable(t, db, source)
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestReadRouteTableGivesATableThatTakesLaterUpdates(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
db := newRoutesTestDatabase(t)
|
|
||||||
source := routetable.New(routetable.Params{})
|
|
||||||
|
|
||||||
// peerA and peerB carry tablePrefixV4, peerB and peerC tablePrefixV6.
|
|
||||||
announceTo(source, tablePrefixV4, asA, peerA, 10)
|
|
||||||
announceTo(source, tablePrefixV4, asA, peerB, 20)
|
|
||||||
announceTo(source, tablePrefixV6, asA, peerB, 30)
|
|
||||||
announceTo(source, tablePrefixV6, asA, peerC, 40)
|
|
||||||
|
|
||||||
writeTableChanges(t, db, source)
|
|
||||||
read := checkReadTable(t, db, source)
|
|
||||||
|
|
||||||
// The same updates to both tables: a loaded peer withdraws a prefix,
|
|
||||||
// another one goes down, and a new peer comes.
|
|
||||||
for _, table := range []*routetable.Table{source, read} {
|
|
||||||
table.Withdraw(netip.MustParsePrefix(tablePrefixV4), peerA)
|
|
||||||
table.PeerDown(peerB)
|
|
||||||
announceTo(table, tablePrefixShortV4, asB, peerD, 50)
|
|
||||||
}
|
|
||||||
|
|
||||||
checkSameReads(t, read, source)
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestReadRouteTableFailsOnAPeerWithoutANumber(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
db := newRoutesTestDatabase(t)
|
|
||||||
table := routetable.New(routetable.Params{})
|
|
||||||
|
|
||||||
// Only the routes of peerB and peerC are kept, so only the peer set names
|
|
||||||
// peerA.
|
|
||||||
announceTo(table, tablePrefixV4, asA, peerA, 10)
|
|
||||||
announceTo(table, tablePrefixV4, asA, peerB, 20)
|
|
||||||
announceTo(table, tablePrefixV4, asA, peerC, 30)
|
|
||||||
writeTableChanges(t, db, table)
|
|
||||||
|
|
||||||
_, err := db.db.ExecContext(t.Context(),
|
|
||||||
"DELETE FROM route_peers WHERE address = ?", peerA)
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("deleting a route peer: %v", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
_, err = db.ReadRouteTable(t.Context())
|
|
||||||
if !errors.Is(err, errUnknownRoutePeer) {
|
|
||||||
t.Fatalf("ReadRouteTable = %v, want %v", err, errUnknownRoutePeer)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// checkReadTable checks that the table ReadRouteTable reads answers every read
|
|
||||||
// as want does, and that reading it recorded no change. It returns that table.
|
|
||||||
func checkReadTable(
|
|
||||||
t *testing.T, db *Database, want *routetable.Table,
|
|
||||||
) *routetable.Table {
|
|
||||||
t.Helper()
|
|
||||||
|
|
||||||
got, err := db.ReadRouteTable(t.Context())
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("ReadRouteTable: %v", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
checkSameReads(t, got, want)
|
|
||||||
|
|
||||||
changes, newPeers := got.TakeChanges(math.MaxInt)
|
|
||||||
if len(changes) != 0 || len(newPeers) != 0 {
|
|
||||||
t.Errorf("the read recorded changes %+v and new peers %v, want none",
|
|
||||||
changes, newPeers)
|
|
||||||
}
|
|
||||||
|
|
||||||
return got
|
|
||||||
}
|
|
||||||
|
|
||||||
// checkSameReads checks that got answers every read as want does, for the
|
|
||||||
// prefixes and the origin ASes the tests use.
|
|
||||||
func checkSameReads(t *testing.T, got, want *routetable.Table) {
|
|
||||||
t.Helper()
|
|
||||||
|
|
||||||
if !reflect.DeepEqual(got.Counts(), want.Counts()) {
|
|
||||||
t.Errorf("Counts() = %+v, want %+v", got.Counts(), want.Counts())
|
|
||||||
}
|
|
||||||
|
|
||||||
for _, text := range []string{
|
|
||||||
tablePrefixV4, tablePrefixShortV4, tablePrefixV6,
|
|
||||||
} {
|
|
||||||
prefix := netip.MustParsePrefix(text)
|
|
||||||
|
|
||||||
if !reflect.DeepEqual(got.Origins(prefix), want.Origins(prefix)) {
|
|
||||||
t.Errorf("Origins(%s) = %+v, want %+v",
|
|
||||||
prefix, got.Origins(prefix), want.Origins(prefix))
|
|
||||||
}
|
|
||||||
|
|
||||||
gotSummary, gotFound := got.Lookup(prefix.Addr())
|
|
||||||
wantSummary, wantFound := want.Lookup(prefix.Addr())
|
|
||||||
|
|
||||||
if gotFound != wantFound || !reflect.DeepEqual(gotSummary, wantSummary) {
|
|
||||||
t.Errorf("Lookup(%s) = %+v, %v; want %+v, %v", prefix.Addr(),
|
|
||||||
gotSummary, gotFound, wantSummary, wantFound)
|
|
||||||
}
|
|
||||||
|
|
||||||
ipVersion := ipVersionV6
|
|
||||||
if prefix.Addr().Is4() {
|
|
||||||
ipVersion = ipVersionV4
|
|
||||||
}
|
|
||||||
|
|
||||||
gotSample := sortedSample(got, ipVersion, prefix.Bits())
|
|
||||||
wantSample := sortedSample(want, ipVersion, prefix.Bits())
|
|
||||||
|
|
||||||
if !reflect.DeepEqual(gotSample, wantSample) {
|
|
||||||
t.Errorf("SamplePrefixes(%d, %d) = %+v, want %+v",
|
|
||||||
ipVersion, prefix.Bits(), gotSample, wantSample)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
for _, asn := range []int{asA, asB} {
|
|
||||||
if !reflect.DeepEqual(got.OriginatedPrefixes(asn),
|
|
||||||
want.OriginatedPrefixes(asn)) {
|
|
||||||
t.Errorf("OriginatedPrefixes(%d) = %+v, want %+v", asn,
|
|
||||||
got.OriginatedPrefixes(asn), want.OriginatedPrefixes(asn))
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// sortedSample returns every prefix of the IP version and mask length in
|
|
||||||
// table, in prefix order.
|
|
||||||
func sortedSample(
|
|
||||||
table *routetable.Table, ipVersion, maskLength int,
|
|
||||||
) []routetable.PrefixSummary {
|
|
||||||
// More than the tests have of any mask length.
|
|
||||||
const all = 100
|
|
||||||
|
|
||||||
sample := table.SamplePrefixes(ipVersion, maskLength, all)
|
|
||||||
slices.SortFunc(sample, func(a, b routetable.PrefixSummary) int {
|
|
||||||
return a.Prefix.Compare(b.Prefix)
|
|
||||||
})
|
|
||||||
|
|
||||||
return sample
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestWriteRouteChangesDeletesTheRowOfARemovedPair(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
db := newRoutesTestDatabase(t)
|
|
||||||
table := routetable.New(routetable.Params{})
|
|
||||||
|
|
||||||
announceTo(table, tablePrefixV4, asA, peerA, 10)
|
|
||||||
announceTo(table, tablePrefixV4, asB, peerB, 20)
|
|
||||||
writeTableChanges(t, db, table)
|
|
||||||
|
|
||||||
// The prefix's only peer with asA withdraws it.
|
|
||||||
table.Withdraw(netip.MustParsePrefix(tablePrefixV4), peerA)
|
|
||||||
writeTableChanges(t, db, table)
|
|
||||||
|
|
||||||
var origins []int
|
|
||||||
|
|
||||||
rows, err := db.db.QueryContext(t.Context(),
|
|
||||||
"SELECT origin_asn FROM routes WHERE prefix = ?", tablePrefixV4)
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("reading the routes: %v", err)
|
|
||||||
}
|
|
||||||
defer func() { _ = rows.Close() }()
|
|
||||||
|
|
||||||
for rows.Next() {
|
|
||||||
var origin int
|
|
||||||
|
|
||||||
err := rows.Scan(&origin)
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("reading a route: %v", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
origins = append(origins, origin)
|
|
||||||
}
|
|
||||||
|
|
||||||
err = rows.Err()
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("reading the routes: %v", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
if !slices.Equal(origins, []int{asB}) {
|
|
||||||
t.Errorf("rows of %s have origins %v, want only %d",
|
|
||||||
tablePrefixV4, origins, asB)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// routePeer is one row of route_peers.
|
|
||||||
type routePeer struct {
|
|
||||||
number uint
|
|
||||||
address string
|
|
||||||
}
|
|
||||||
|
|
||||||
// readRoutePeerRows returns the rows of route_peers in number order.
|
|
||||||
func readRoutePeerRows(t *testing.T, db *Database) []routePeer {
|
|
||||||
t.Helper()
|
|
||||||
|
|
||||||
rows, err := db.db.QueryContext(t.Context(),
|
|
||||||
"SELECT number, address FROM route_peers ORDER BY number")
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("reading the route peers: %v", err)
|
|
||||||
}
|
|
||||||
defer func() { _ = rows.Close() }()
|
|
||||||
|
|
||||||
var peers []routePeer
|
|
||||||
|
|
||||||
for rows.Next() {
|
|
||||||
var peer routePeer
|
|
||||||
|
|
||||||
err := rows.Scan(&peer.number, &peer.address)
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("reading a route peer: %v", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
peers = append(peers, peer)
|
|
||||||
}
|
|
||||||
|
|
||||||
err = rows.Err()
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("reading the route peers: %v", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
return peers
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestWriteRouteChangesStoresNewPeers(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
db := newRoutesTestDatabase(t)
|
|
||||||
table := routetable.New(routetable.Params{})
|
|
||||||
|
|
||||||
announceTo(table, tablePrefixV4, asA, peerA, 10)
|
|
||||||
announceTo(table, tablePrefixV6, asA, peerC, 20)
|
|
||||||
writeTableChanges(t, db, table)
|
|
||||||
|
|
||||||
// peerA is stored already; peerB is new.
|
|
||||||
announceTo(table, tablePrefixV4, asA, peerA, 30)
|
|
||||||
announceTo(table, tablePrefixV4, asA, peerB, 40)
|
|
||||||
writeTableChanges(t, db, table)
|
|
||||||
|
|
||||||
want := []routePeer{{0, peerA}, {1, peerC}, {2, peerB}}
|
|
||||||
|
|
||||||
got := readRoutePeerRows(t, db)
|
|
||||||
if !slices.Equal(got, want) {
|
|
||||||
t.Errorf("route_peers = %+v, want %+v", got, want)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestWriteRouteChangesWritesNothingWhenItFails(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
db := newRoutesTestDatabase(t)
|
|
||||||
table := routetable.New(routetable.Params{})
|
|
||||||
|
|
||||||
announceTo(table, tablePrefixV4, asA, peerA, 10)
|
|
||||||
announceTo(table, tablePrefixV6, asA, peerB, 20)
|
|
||||||
|
|
||||||
// peerB's number is missing, so its kept route cannot be written.
|
|
||||||
changes, newPeers := table.TakeChanges(math.MaxInt)
|
|
||||||
delete(newPeers, peerB)
|
|
||||||
|
|
||||||
err := db.WriteRouteChanges(changes, newPeers)
|
|
||||||
if !errors.Is(err, errUnknownRoutePeer) {
|
|
||||||
t.Fatalf("WriteRouteChanges = %v, want %v", err, errUnknownRoutePeer)
|
|
||||||
}
|
|
||||||
|
|
||||||
peers := readRoutePeerRows(t, db)
|
|
||||||
if len(peers) != 0 {
|
|
||||||
t.Errorf("route_peers = %+v after the failed write, want none", peers)
|
|
||||||
}
|
|
||||||
|
|
||||||
var routes int
|
|
||||||
|
|
||||||
err = db.db.QueryRowContext(t.Context(),
|
|
||||||
"SELECT COUNT(*) FROM routes").Scan(&routes)
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("counting the routes: %v", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
if routes != 0 {
|
|
||||||
t.Errorf("routes holds %d rows after the failed write, want none", routes)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// Parameters for BenchmarkReadRouteTable.
|
|
||||||
const (
|
|
||||||
// benchTablePairs is how many prefix and origin AS pairs the table holds:
|
|
||||||
// one origin AS for each prefix. Every benchTableV6Every-th prefix is an
|
|
||||||
// IPv6 /48, the others IPv4 /24s.
|
|
||||||
benchTablePairs = 200000
|
|
||||||
benchTableV6Every = 5
|
|
||||||
// benchTablePeers is how many peers there are, about as many as RIS has.
|
|
||||||
// benchTablePeersPerPair is how many of them, picked at random, carry each
|
|
||||||
// pair: RIS's 460 million routes over 1.4 million pairs.
|
|
||||||
benchTablePeers = 1400
|
|
||||||
benchTablePeersPerPair = 330
|
|
||||||
// benchTableBatch is how many changes each write holds.
|
|
||||||
benchTableBatch = 10000
|
|
||||||
)
|
|
||||||
|
|
||||||
// BenchmarkReadRouteTable times the read of a routing table of 200,000 prefix
|
|
||||||
// and origin AS pairs, each carried by 330 of 1,400 peers and keeping two
|
|
||||||
// routes. It writes that table first. It reports the bytes of the database
|
|
||||||
// file per pair. It is run by hand, not by make test, for example in the test
|
|
||||||
// image:
|
|
||||||
//
|
|
||||||
// go test -run '^$' -bench ReadRouteTable -benchtime 5x ./internal/database/
|
|
||||||
func BenchmarkReadRouteTable(b *testing.B) {
|
|
||||||
db, err := New(&config.Config{StateDir: b.TempDir()}, logger.New())
|
|
||||||
if err != nil {
|
|
||||||
b.Fatalf("failed to create database: %v", err)
|
|
||||||
}
|
|
||||||
defer func() { _ = db.Close() }()
|
|
||||||
|
|
||||||
//nolint:gosec // The benchmark needs the same table every run.
|
|
||||||
rng := rand.New(rand.NewPCG(1, 2))
|
|
||||||
|
|
||||||
newPeers := make(map[string]uint, benchTablePeers)
|
|
||||||
for number := range uint(benchTablePeers) {
|
|
||||||
newPeers[benchPeerAddress(number)] = number
|
|
||||||
}
|
|
||||||
|
|
||||||
for first := 0; first < benchTablePairs; first += benchTableBatch {
|
|
||||||
changes := make([]routetable.Change, 0, benchTableBatch)
|
|
||||||
for pair := first; pair < first+benchTableBatch; pair++ {
|
|
||||||
changes = append(changes, benchTableChange(rng, pair))
|
|
||||||
}
|
|
||||||
|
|
||||||
err := db.WriteRouteChanges(changes, newPeers)
|
|
||||||
if err != nil {
|
|
||||||
b.Fatalf("WriteRouteChanges: %v", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
newPeers = nil
|
|
||||||
}
|
|
||||||
|
|
||||||
// The WAL is copied into the database file, whose size is reported.
|
|
||||||
err = db.Checkpoint(b.Context())
|
|
||||||
if err != nil {
|
|
||||||
b.Fatalf("Checkpoint: %v", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
file, err := os.Stat(db.path)
|
|
||||||
if err != nil {
|
|
||||||
b.Fatalf("reading the database file's size: %v", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
for b.Loop() {
|
|
||||||
table, err := db.ReadRouteTable(b.Context())
|
|
||||||
if err != nil {
|
|
||||||
b.Fatalf("ReadRouteTable: %v", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
counts := table.Counts()
|
|
||||||
if counts.IPv4Prefixes+counts.IPv6Prefixes != benchTablePairs {
|
|
||||||
b.Fatalf("read %d prefixes, want %d",
|
|
||||||
counts.IPv4Prefixes+counts.IPv6Prefixes, benchTablePairs)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// After the loop: b.Loop drops the metrics reported before it starts.
|
|
||||||
b.ReportMetric(float64(file.Size())/benchTablePairs, "file-bytes/pair")
|
|
||||||
}
|
|
||||||
|
|
||||||
// benchPeerAddress returns the address of the peer numbered number.
|
|
||||||
func benchPeerAddress(number uint) string {
|
|
||||||
const addressesPerBlock = 256
|
|
||||||
|
|
||||||
return fmt.Sprintf("198.18.%d.%d",
|
|
||||||
number/addressesPerBlock, number%addressesPerBlock)
|
|
||||||
}
|
|
||||||
|
|
||||||
// benchTableChange returns the content of the pair numbered pair: its prefix,
|
|
||||||
// its origin AS, benchTablePeersPerPair peers picked at random, and the routes
|
|
||||||
// of two of them, each with an AS path of four ASes.
|
|
||||||
func benchTableChange(rng *rand.Rand, pair int) routetable.Change {
|
|
||||||
const (
|
|
||||||
firstOrigin = 64500
|
|
||||||
transitASes = 50000
|
|
||||||
olderSeconds = 10
|
|
||||||
)
|
|
||||||
|
|
||||||
text := fmt.Sprintf("%d.%d.%d.0/24", 1+pair>>16, (pair>>8)%256, pair%256)
|
|
||||||
if pair%benchTableV6Every == 0 {
|
|
||||||
text = fmt.Sprintf("2001:%x:%x::/48", pair>>16, pair%65536)
|
|
||||||
}
|
|
||||||
|
|
||||||
prefix := netip.MustParsePrefix(text)
|
|
||||||
origin := firstOrigin + pair%transitASes
|
|
||||||
peers := bitset.New(benchTablePeers)
|
|
||||||
|
|
||||||
for peers.Count() < benchTablePeersPerPair {
|
|
||||||
peers.Set(rng.UintN(benchTablePeers))
|
|
||||||
}
|
|
||||||
|
|
||||||
change := routetable.Change{Prefix: prefix, OriginASN: origin, Peers: peers}
|
|
||||||
|
|
||||||
// The routes of the first two peers, newest first.
|
|
||||||
for i, number := range slices.Collect(peers.EachSet())[:keptRoutesPerRow] {
|
|
||||||
change.Routes = append(change.Routes, routetable.Route{
|
|
||||||
Peer: benchPeerAddress(number),
|
|
||||||
ASPath: []int{
|
|
||||||
firstOrigin + rng.IntN(transitASes),
|
|
||||||
firstOrigin + rng.IntN(transitASes),
|
|
||||||
firstOrigin + rng.IntN(transitASes),
|
|
||||||
origin,
|
|
||||||
},
|
|
||||||
Time: tableTime(pair - i*olderSeconds),
|
|
||||||
})
|
|
||||||
}
|
|
||||||
|
|
||||||
return change
|
|
||||||
}
|
|
||||||
@@ -89,31 +89,61 @@ CREATE INDEX IF NOT EXISTS idx_bgp_peers_asn ON bgp_peers(peer_asn);
|
|||||||
CREATE INDEX IF NOT EXISTS idx_bgp_peers_last_seen ON bgp_peers(last_seen);
|
CREATE INDEX IF NOT EXISTS idx_bgp_peers_last_seen ON bgp_peers(last_seen);
|
||||||
CREATE INDEX IF NOT EXISTS idx_bgp_peers_ip ON bgp_peers(peer_ip);
|
CREATE INDEX IF NOT EXISTS idx_bgp_peers_ip ON bgp_peers(peer_ip);
|
||||||
|
|
||||||
-- The whole routing table, as internal/routetable keeps it in memory: one row
|
-- IPv4 routing table maintained by PrefixHandler. A route write changes the
|
||||||
-- for each prefix and origin AS that some peer carries. Startup reads every
|
-- table and its entry in every index on a column the write sets, so the table
|
||||||
-- row, and the rows of the pairs that change are written back.
|
-- has only the indexes that the queries named with them use.
|
||||||
CREATE TABLE IF NOT EXISTS routes (
|
CREATE TABLE IF NOT EXISTS live_routes_v4 (
|
||||||
prefix TEXT NOT NULL, -- in the text form Go's net/netip prints
|
-- The row id, which the prefix-length pages' sample draws through the
|
||||||
|
-- mask_length index. Declared as a column, it stays the same for as long
|
||||||
|
-- as the route is live, also through a VACUUM.
|
||||||
|
id INTEGER PRIMARY KEY,
|
||||||
|
prefix TEXT NOT NULL,
|
||||||
|
mask_length INTEGER NOT NULL, -- CIDR mask length (0-32)
|
||||||
origin_asn INTEGER NOT NULL,
|
origin_asn INTEGER NOT NULL,
|
||||||
-- The peers whose current route to the prefix has this origin AS: a set
|
peer_ip TEXT NOT NULL,
|
||||||
-- of their numbers in route_peers, as the bitset package's MarshalBinary
|
as_path TEXT NOT NULL, -- JSON array
|
||||||
-- writes it.
|
next_hop TEXT NOT NULL,
|
||||||
peers BLOB NOT NULL,
|
last_updated DATETIME NOT NULL,
|
||||||
-- The newest of those routes and the one before it, the two kept in full.
|
-- A peer has at most one route for a prefix. This key finds the route a
|
||||||
-- Each has its peer's number in route_peers, its AS path as a JSON array,
|
-- write updates or deletes and, peer_ip coming first, all of a peer's
|
||||||
-- and its time in seconds since 1970, UTC. A route that is not kept has
|
-- routes when its BGP session goes down.
|
||||||
-- NULL in its three columns.
|
UNIQUE(peer_ip, prefix)
|
||||||
newest_peer INTEGER,
|
|
||||||
newest_as_path TEXT,
|
|
||||||
newest_time INTEGER,
|
|
||||||
older_peer INTEGER,
|
|
||||||
older_as_path TEXT,
|
|
||||||
older_time INTEGER,
|
|
||||||
PRIMARY KEY (prefix, origin_asn)
|
|
||||||
) WITHOUT ROWID;
|
|
||||||
|
|
||||||
-- The peers of the routes table, each with the number it has there.
|
|
||||||
CREATE TABLE IF NOT EXISTS route_peers (
|
|
||||||
number INTEGER PRIMARY KEY,
|
|
||||||
address TEXT NOT NULL UNIQUE
|
|
||||||
);
|
);
|
||||||
|
|
||||||
|
-- IPv6 routing table maintained by PrefixHandler, as live_routes_v4.
|
||||||
|
CREATE TABLE IF NOT EXISTS live_routes_v6 (
|
||||||
|
id INTEGER PRIMARY KEY,
|
||||||
|
prefix TEXT NOT NULL,
|
||||||
|
mask_length INTEGER NOT NULL, -- CIDR mask length (0-128)
|
||||||
|
origin_asn INTEGER NOT NULL,
|
||||||
|
peer_ip TEXT NOT NULL,
|
||||||
|
as_path TEXT NOT NULL, -- JSON array
|
||||||
|
next_hop TEXT NOT NULL,
|
||||||
|
last_updated DATETIME NOT NULL,
|
||||||
|
UNIQUE(peer_ip, prefix)
|
||||||
|
);
|
||||||
|
|
||||||
|
-- Indexes for both live route tables, and the queries that use each.
|
||||||
|
--
|
||||||
|
-- prefix, last_updated: a prefix's routes, for the prefix page and its API
|
||||||
|
-- answer, the IP lookup and its peer count, and the check whether a prefix
|
||||||
|
-- still has a live route; and a prefix's newest route for the prefix-length
|
||||||
|
-- pages and the AS page, in one search whatever the number of peers.
|
||||||
|
CREATE INDEX IF NOT EXISTS idx_live_routes_v4_prefix_last_updated ON live_routes_v4(prefix, last_updated);
|
||||||
|
CREATE INDEX IF NOT EXISTS idx_live_routes_v6_prefix_last_updated ON live_routes_v6(prefix, last_updated);
|
||||||
|
-- mask_length: the prefix-length pages' random sample, the row id range of a
|
||||||
|
-- mask length and the first route of it at or after a row id.
|
||||||
|
CREATE INDEX IF NOT EXISTS idx_live_routes_v4_mask_length ON live_routes_v4(mask_length);
|
||||||
|
CREATE INDEX IF NOT EXISTS idx_live_routes_v6_mask_length ON live_routes_v6(mask_length);
|
||||||
|
-- mask_length, prefix: the prefix-length pages' list of a mask length's
|
||||||
|
-- prefixes in text order, and the prefix distribution read at startup.
|
||||||
|
CREATE INDEX IF NOT EXISTS idx_live_routes_v4_mask_prefix ON live_routes_v4(mask_length, prefix);
|
||||||
|
CREATE INDEX IF NOT EXISTS idx_live_routes_v6_mask_prefix ON live_routes_v6(mask_length, prefix);
|
||||||
|
-- origin_asn, prefix: the prefixes an AS originates, each in one search
|
||||||
|
-- whatever the number of peers, for the AS page and its API answer.
|
||||||
|
CREATE INDEX IF NOT EXISTS idx_live_routes_v4_origin_prefix ON live_routes_v4(origin_asn, prefix);
|
||||||
|
CREATE INDEX IF NOT EXISTS idx_live_routes_v6_origin_prefix ON live_routes_v6(origin_asn, prefix);
|
||||||
|
-- last_updated: the oldest and newest route times in /api/v1/stats and
|
||||||
|
-- /status.json.
|
||||||
|
CREATE INDEX IF NOT EXISTS idx_live_routes_v4_last_updated ON live_routes_v4(last_updated);
|
||||||
|
CREATE INDEX IF NOT EXISTS idx_live_routes_v6_last_updated ON live_routes_v6(last_updated);
|
||||||
@@ -1,6 +1,8 @@
|
|||||||
package database
|
package database
|
||||||
|
|
||||||
import (
|
import (
|
||||||
|
"fmt"
|
||||||
|
"net"
|
||||||
"strings"
|
"strings"
|
||||||
|
|
||||||
"github.com/google/uuid"
|
"github.com/google/uuid"
|
||||||
@@ -18,3 +20,15 @@ func detectIPVersion(prefix string) int {
|
|||||||
|
|
||||||
return ipVersionV4
|
return ipVersionV4
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// prefixMaskLength returns the mask length of a prefix such as 192.0.2.0/24.
|
||||||
|
func prefixMaskLength(prefix string) (int, error) {
|
||||||
|
_, network, err := net.ParseCIDR(prefix)
|
||||||
|
if err != nil {
|
||||||
|
return 0, fmt.Errorf("invalid prefix %s: %w", prefix, err)
|
||||||
|
}
|
||||||
|
|
||||||
|
maskLength, _ := network.Mask.Size()
|
||||||
|
|
||||||
|
return maskLength, nil
|
||||||
|
}
|
||||||
|
|||||||
@@ -1,155 +0,0 @@
|
|||||||
package routetable_test
|
|
||||||
|
|
||||||
import (
|
|
||||||
"fmt"
|
|
||||||
"math"
|
|
||||||
"math/rand/v2"
|
|
||||||
"net/netip"
|
|
||||||
"runtime"
|
|
||||||
"testing"
|
|
||||||
"time"
|
|
||||||
|
|
||||||
"git.eeqj.de/sneak/routewatch/internal/routetable"
|
|
||||||
)
|
|
||||||
|
|
||||||
// Parameters for BenchmarkApplyMixedBatch.
|
|
||||||
const (
|
|
||||||
// benchPeers is how many peers there are, about as many as RIS has.
|
|
||||||
benchPeers = 1400
|
|
||||||
// benchPrefixes is how many IPv4 /24 prefixes the table holds, and
|
|
||||||
// benchPeersPerPrefix how many peers, picked at random, carry each.
|
|
||||||
benchPrefixes = 100000
|
|
||||||
benchPeersPerPrefix = 20
|
|
||||||
// benchBatchSize is how many updates the timed batch holds, as many as
|
|
||||||
// the prefix handler's largest batch (prefixBatchMaxSize in
|
|
||||||
// internal/routewatch).
|
|
||||||
benchBatchSize = 250000
|
|
||||||
// In the timed batch every benchWithdrawalEvery-th update is a
|
|
||||||
// withdrawal, and every benchNewOriginEvery-th announcement has an
|
|
||||||
// origin AS other than the prefix's.
|
|
||||||
benchWithdrawalEvery = 5
|
|
||||||
benchNewOriginEvery = 7
|
|
||||||
// benchOrigins is how many origin ASes the prefixes are spread over,
|
|
||||||
// from benchFirstASN on: 20 prefixes each, about as many as an origin AS
|
|
||||||
// has on average in the full table.
|
|
||||||
benchOrigins = 5000
|
|
||||||
benchFirstASN = 64500
|
|
||||||
// benchUpdatesPerSecond is how many updates share one time.
|
|
||||||
benchUpdatesPerSecond = 1000
|
|
||||||
benchMaskLength = 24
|
|
||||||
)
|
|
||||||
|
|
||||||
// benchUpdate is one update of the benchmark.
|
|
||||||
type benchUpdate struct {
|
|
||||||
prefix netip.Prefix
|
|
||||||
origin int
|
|
||||||
route routetable.Route
|
|
||||||
withdraw bool
|
|
||||||
}
|
|
||||||
|
|
||||||
// BenchmarkApplyMixedBatch times one batch of 250,000 updates of routes picked
|
|
||||||
// at random: announcements, withdrawals and announcements with a new origin AS.
|
|
||||||
// The table it applies them to holds 100,000 IPv4 prefixes from 5,000 origin
|
|
||||||
// ASes, each carried by 20 of 1,400 peers. It reports the heap bytes that table
|
|
||||||
// takes per prefix and origin AS pair. It is run by hand, not by make test, for
|
|
||||||
// example in the test image:
|
|
||||||
//
|
|
||||||
// go test -run '^$' -bench MixedBatch -benchtime 10x ./internal/routetable/
|
|
||||||
func BenchmarkApplyMixedBatch(b *testing.B) {
|
|
||||||
//nolint:gosec // The benchmark needs the same updates every run.
|
|
||||||
rng := rand.New(rand.NewPCG(1, 2))
|
|
||||||
peers := make([]string, benchPeers)
|
|
||||||
|
|
||||||
for i := range peers {
|
|
||||||
peers[i] = fmt.Sprintf("198.18.%d.%d", i/256, i%256)
|
|
||||||
}
|
|
||||||
|
|
||||||
before := heapBytes()
|
|
||||||
table := routetable.New(routetable.Params{})
|
|
||||||
updates := 0
|
|
||||||
|
|
||||||
for prefix := range benchPrefixes {
|
|
||||||
for range benchPeersPerPrefix {
|
|
||||||
peer := peers[rng.IntN(benchPeers)]
|
|
||||||
apply(table, benchAnnouncement(rng, prefix, peer, updates, false))
|
|
||||||
updates++
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// The record of changes is handed over every second while routewatch
|
|
||||||
// runs, so it is not part of the table's size.
|
|
||||||
table.TakeChanges(math.MaxInt)
|
|
||||||
|
|
||||||
bytesPerPair := float64(heapBytes()-before) / benchPrefixes
|
|
||||||
|
|
||||||
batch := make([]benchUpdate, 0, benchBatchSize)
|
|
||||||
for i := range benchBatchSize {
|
|
||||||
update := benchAnnouncement(rng, rng.IntN(benchPrefixes),
|
|
||||||
peers[rng.IntN(benchPeers)], updates+i, i%benchNewOriginEvery == 0)
|
|
||||||
update.withdraw = i%benchWithdrawalEvery == 0
|
|
||||||
batch = append(batch, update)
|
|
||||||
}
|
|
||||||
|
|
||||||
for b.Loop() {
|
|
||||||
for _, update := range batch {
|
|
||||||
apply(table, update)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// After the loop: b.Loop drops the metrics reported before it starts.
|
|
||||||
b.ReportMetric(bytesPerPair, "bytes/pair")
|
|
||||||
}
|
|
||||||
|
|
||||||
// benchAnnouncement returns the announcement of the prefix numbered prefix, a
|
|
||||||
// /24 from 1.0.0.0/24 on, by peer. It is the updates-th update of the
|
|
||||||
// benchmark, and its AS path holds four ASes from the peer's AS to the origin
|
|
||||||
// AS. newOrigin gives it an origin AS other than the prefix's.
|
|
||||||
func benchAnnouncement(
|
|
||||||
rng *rand.Rand, prefix int, peer string, updates int, newOrigin bool,
|
|
||||||
) benchUpdate {
|
|
||||||
origin := benchFirstASN + prefix%benchOrigins
|
|
||||||
if newOrigin {
|
|
||||||
origin += benchOrigins
|
|
||||||
}
|
|
||||||
|
|
||||||
start := time.Date(2026, 1, 2, 3, 4, 5, 0, time.UTC)
|
|
||||||
|
|
||||||
return benchUpdate{
|
|
||||||
prefix: netip.MustParsePrefix(fmt.Sprintf("%d.%d.%d.0/%d",
|
|
||||||
1+prefix>>16, (prefix>>8)%256, prefix%256, benchMaskLength)),
|
|
||||||
origin: origin,
|
|
||||||
route: routetable.Route{
|
|
||||||
Peer: peer,
|
|
||||||
ASPath: []int{
|
|
||||||
benchFirstASN + rng.IntN(benchOrigins),
|
|
||||||
benchFirstASN + rng.IntN(benchOrigins),
|
|
||||||
benchFirstASN + rng.IntN(benchOrigins),
|
|
||||||
origin,
|
|
||||||
},
|
|
||||||
Time: start.Add(time.Duration(updates/benchUpdatesPerSecond) *
|
|
||||||
time.Second),
|
|
||||||
},
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// apply applies update to table.
|
|
||||||
func apply(table *routetable.Table, update benchUpdate) {
|
|
||||||
if update.withdraw {
|
|
||||||
table.Withdraw(update.prefix, update.route.Peer)
|
|
||||||
|
|
||||||
return
|
|
||||||
}
|
|
||||||
|
|
||||||
table.Announce(update.prefix, update.origin, update.route)
|
|
||||||
}
|
|
||||||
|
|
||||||
// heapBytes returns the bytes of the live heap objects, after a collection.
|
|
||||||
func heapBytes() uint64 {
|
|
||||||
runtime.GC()
|
|
||||||
|
|
||||||
var stats runtime.MemStats
|
|
||||||
|
|
||||||
runtime.ReadMemStats(&stats)
|
|
||||||
|
|
||||||
return stats.HeapAlloc
|
|
||||||
}
|
|
||||||
@@ -1,233 +0,0 @@
|
|||||||
package routetable
|
|
||||||
|
|
||||||
import (
|
|
||||||
"net/netip"
|
|
||||||
"slices"
|
|
||||||
"time"
|
|
||||||
|
|
||||||
"github.com/bits-and-blooms/bitset"
|
|
||||||
)
|
|
||||||
|
|
||||||
// PrefixSummary is what the table answers about one prefix.
|
|
||||||
type PrefixSummary struct {
|
|
||||||
Prefix netip.Prefix
|
|
||||||
// OriginASN is the origin AS that the most peers carry the prefix with.
|
|
||||||
// When several have as many peers, it is the lowest AS number.
|
|
||||||
OriginASN int
|
|
||||||
// Peers is how many peers carry the prefix.
|
|
||||||
Peers int
|
|
||||||
// Newest is the time of the newest kept route of the prefix, or nil when
|
|
||||||
// it keeps none.
|
|
||||||
Newest *time.Time
|
|
||||||
}
|
|
||||||
|
|
||||||
// OriginatedPrefix is one prefix an AS originates.
|
|
||||||
type OriginatedPrefix struct {
|
|
||||||
Prefix netip.Prefix
|
|
||||||
// Newest is the time of the newest kept route of the prefix with that
|
|
||||||
// origin AS, or nil when it keeps none.
|
|
||||||
Newest *time.Time
|
|
||||||
}
|
|
||||||
|
|
||||||
// Counts are the table's figures for the stats.
|
|
||||||
type Counts struct {
|
|
||||||
// Pairs counts the prefix and origin AS pairs that some peer carries.
|
|
||||||
Pairs int
|
|
||||||
// Peers counts the peers that have a number: those given in
|
|
||||||
// Params.PeerNumbers and those numbered since.
|
|
||||||
Peers int
|
|
||||||
IPv4Prefixes int
|
|
||||||
IPv6Prefixes int
|
|
||||||
// IPv4PrefixesByLength[n] and IPv6PrefixesByLength[n] are how many
|
|
||||||
// prefixes of mask length n some peer carries.
|
|
||||||
IPv4PrefixesByLength []int
|
|
||||||
IPv6PrefixesByLength []int
|
|
||||||
// IPv4Routes and IPv6Routes count peer and prefix pairs: each peer that
|
|
||||||
// carries a prefix counts once for it.
|
|
||||||
IPv4Routes int
|
|
||||||
IPv6Routes int
|
|
||||||
// OriginsWithoutKeptRoute counts the prefix and origin AS pairs that peers
|
|
||||||
// carry but whose kept routes were all withdrawn.
|
|
||||||
OriginsWithoutKeptRoute int
|
|
||||||
// OldestKeptRoute and NewestKeptRoute are the times of the oldest and the
|
|
||||||
// newest kept routes, or nil when the table keeps none.
|
|
||||||
OldestKeptRoute *time.Time
|
|
||||||
NewestKeptRoute *time.Time
|
|
||||||
}
|
|
||||||
|
|
||||||
// Lookup returns the most specific prefix that holds addr and that some peer
|
|
||||||
// carries. It ignores prefixes shorter than /8 for IPv4 and /16 for IPv6. It
|
|
||||||
// makes one map lookup per mask length, and reports false when no prefix
|
|
||||||
// holds addr.
|
|
||||||
func (t *Table) Lookup(addr netip.Addr) (PrefixSummary, bool) {
|
|
||||||
t.mu.RLock()
|
|
||||||
defer t.mu.RUnlock()
|
|
||||||
|
|
||||||
// An IPv4 address written in IPv6 form (::ffff:192.0.2.1) is looked up as
|
|
||||||
// IPv4.
|
|
||||||
addr = addr.Unmap()
|
|
||||||
f := t.familyOf(addr)
|
|
||||||
|
|
||||||
for bits := addr.BitLen(); bits >= f.minLookupLength; bits-- {
|
|
||||||
prefix := netip.PrefixFrom(addr, bits).Masked()
|
|
||||||
|
|
||||||
origins, found := f.byLength[bits][prefix]
|
|
||||||
if found {
|
|
||||||
return summarize(prefix, origins), true
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
return PrefixSummary{}, false
|
|
||||||
}
|
|
||||||
|
|
||||||
// OriginatedPrefixes returns the prefixes AS asn originates, in prefix order.
|
|
||||||
func (t *Table) OriginatedPrefixes(asn int) []OriginatedPrefix {
|
|
||||||
t.mu.RLock()
|
|
||||||
defer t.mu.RUnlock()
|
|
||||||
|
|
||||||
prefixes := make([]OriginatedPrefix, 0, len(t.byOrigin[asn]))
|
|
||||||
for prefix, o := range t.byOrigin[asn] {
|
|
||||||
prefixes = append(prefixes,
|
|
||||||
OriginatedPrefix{Prefix: prefix, Newest: o.newestTime()})
|
|
||||||
}
|
|
||||||
|
|
||||||
slices.SortFunc(prefixes, func(a, b OriginatedPrefix) int {
|
|
||||||
return a.Prefix.Compare(b.Prefix)
|
|
||||||
})
|
|
||||||
|
|
||||||
return prefixes
|
|
||||||
}
|
|
||||||
|
|
||||||
// Origins returns each origin AS of prefix, in AS number order, with its peers
|
|
||||||
// and its kept routes. It returns nil when no peer carries the prefix.
|
|
||||||
func (t *Table) Origins(prefix netip.Prefix) []Origin {
|
|
||||||
t.mu.RLock()
|
|
||||||
defer t.mu.RUnlock()
|
|
||||||
|
|
||||||
if !prefix.IsValid() {
|
|
||||||
return nil
|
|
||||||
}
|
|
||||||
|
|
||||||
var origins []Origin
|
|
||||||
|
|
||||||
for _, o := range t.origins(prefix.Masked()) {
|
|
||||||
origins = append(origins, Origin{
|
|
||||||
ASN: o.asn,
|
|
||||||
Peers: t.addresses(&o.peers),
|
|
||||||
Routes: t.exportRoutes(o.routes),
|
|
||||||
})
|
|
||||||
}
|
|
||||||
|
|
||||||
return origins
|
|
||||||
}
|
|
||||||
|
|
||||||
// SamplePrefixes returns up to limit of the prefixes of IP version ipVersion
|
|
||||||
// (4 or 6) and mask length maskLength. It reads only the prefixes it returns.
|
|
||||||
// Go starts each walk over a map at a random place, so each call returns
|
|
||||||
// another sample.
|
|
||||||
func (t *Table) SamplePrefixes(ipVersion, maskLength, limit int) []PrefixSummary {
|
|
||||||
t.mu.RLock()
|
|
||||||
defer t.mu.RUnlock()
|
|
||||||
|
|
||||||
var f *family
|
|
||||||
|
|
||||||
switch ipVersion {
|
|
||||||
case ipVersion4:
|
|
||||||
f = &t.ipv4
|
|
||||||
case ipVersion6:
|
|
||||||
f = &t.ipv6
|
|
||||||
default:
|
|
||||||
return nil
|
|
||||||
}
|
|
||||||
|
|
||||||
if maskLength < 0 || maskLength >= len(f.byLength) || limit <= 0 {
|
|
||||||
return nil
|
|
||||||
}
|
|
||||||
|
|
||||||
prefixes := f.byLength[maskLength]
|
|
||||||
sample := make([]PrefixSummary, 0, min(limit, len(prefixes)))
|
|
||||||
|
|
||||||
for prefix, origins := range prefixes {
|
|
||||||
if len(sample) == limit {
|
|
||||||
break
|
|
||||||
}
|
|
||||||
|
|
||||||
sample = append(sample, summarize(prefix, origins))
|
|
||||||
}
|
|
||||||
|
|
||||||
return sample
|
|
||||||
}
|
|
||||||
|
|
||||||
// Counts returns the table's figures for the stats. The writes keep them up to
|
|
||||||
// date, so this reads no prefix.
|
|
||||||
func (t *Table) Counts() Counts {
|
|
||||||
t.mu.RLock()
|
|
||||||
defer t.mu.RUnlock()
|
|
||||||
|
|
||||||
counts := Counts{
|
|
||||||
Peers: len(t.peerNumbers),
|
|
||||||
IPv4PrefixesByLength: t.ipv4.prefixesByLength(),
|
|
||||||
IPv6PrefixesByLength: t.ipv6.prefixesByLength(),
|
|
||||||
IPv4Routes: t.ipv4.routes,
|
|
||||||
IPv6Routes: t.ipv6.routes,
|
|
||||||
OriginsWithoutKeptRoute: t.originsWithoutKeptRoute,
|
|
||||||
}
|
|
||||||
|
|
||||||
for _, n := range counts.IPv4PrefixesByLength {
|
|
||||||
counts.IPv4Prefixes += n
|
|
||||||
}
|
|
||||||
|
|
||||||
for _, n := range counts.IPv6PrefixesByLength {
|
|
||||||
counts.IPv6Prefixes += n
|
|
||||||
}
|
|
||||||
|
|
||||||
for _, prefixes := range t.byOrigin {
|
|
||||||
counts.Pairs += len(prefixes)
|
|
||||||
}
|
|
||||||
|
|
||||||
oldest, found := t.keptTimes.Min()
|
|
||||||
if found {
|
|
||||||
counts.OldestKeptRoute = &oldest.time
|
|
||||||
}
|
|
||||||
|
|
||||||
newest, found := t.keptTimes.Max()
|
|
||||||
if found {
|
|
||||||
counts.NewestKeptRoute = &newest.time
|
|
||||||
}
|
|
||||||
|
|
||||||
return counts
|
|
||||||
}
|
|
||||||
|
|
||||||
// addresses returns the addresses of the peers in peers, in number order.
|
|
||||||
func (t *Table) addresses(peers *bitset.BitSet) []string {
|
|
||||||
addresses := make([]string, 0, peers.Count())
|
|
||||||
for number := range peers.EachSet() {
|
|
||||||
addresses = append(addresses, t.peerAddresses[number])
|
|
||||||
}
|
|
||||||
|
|
||||||
return addresses
|
|
||||||
}
|
|
||||||
|
|
||||||
// summarize returns the summary of prefix, whose origins are in AS number
|
|
||||||
// order.
|
|
||||||
func summarize(prefix netip.Prefix, origins []*origin) PrefixSummary {
|
|
||||||
summary := PrefixSummary{Prefix: prefix}
|
|
||||||
mostPeers := 0
|
|
||||||
|
|
||||||
for _, o := range origins {
|
|
||||||
peers := int(o.peers.Count())
|
|
||||||
summary.Peers += peers
|
|
||||||
|
|
||||||
if peers > mostPeers {
|
|
||||||
mostPeers = peers
|
|
||||||
summary.OriginASN = o.asn
|
|
||||||
}
|
|
||||||
|
|
||||||
newest := o.newestTime()
|
|
||||||
if newest != nil && (summary.Newest == nil || newest.After(*summary.Newest)) {
|
|
||||||
summary.Newest = newest
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
return summary
|
|
||||||
}
|
|
||||||
@@ -1,265 +0,0 @@
|
|||||||
package routetable_test
|
|
||||||
|
|
||||||
import (
|
|
||||||
"net/netip"
|
|
||||||
"reflect"
|
|
||||||
"testing"
|
|
||||||
|
|
||||||
"git.eeqj.de/sneak/routewatch/internal/routetable"
|
|
||||||
)
|
|
||||||
|
|
||||||
// announceAll has each listed peer announce prefix with origin AS asn, the
|
|
||||||
// peer at index i i*10 seconds after start.
|
|
||||||
func announceAll(
|
|
||||||
table *routetable.Table, prefix string, asn, start int, peers ...string,
|
|
||||||
) {
|
|
||||||
const secondsBetweenPeers = 10
|
|
||||||
|
|
||||||
for i, peer := range peers {
|
|
||||||
table.Announce(netip.MustParsePrefix(prefix), asn,
|
|
||||||
routeOf(peer, asn, start+i*secondsBetweenPeers))
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestLookup(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
table := routetable.New(routetable.Params{})
|
|
||||||
announceAll(table, "10.0.0.0/8", asA, 10, peerA)
|
|
||||||
// Three peers with two origins: asB has more of them.
|
|
||||||
announceAll(table, "10.1.0.0/16", asA, 10, peerA)
|
|
||||||
announceAll(table, "10.1.0.0/16", asB, 20, peerB, peerC)
|
|
||||||
// Two origins with one peer each: the lower AS number wins.
|
|
||||||
announceAll(table, prefixV4, asB, 20, peerB)
|
|
||||||
announceAll(table, prefixV4, asA, 10, peerA)
|
|
||||||
// An origin whose kept routes were withdrawn: no newest time.
|
|
||||||
announceAll(table, "10.9.0.0/16", asA, 10, peerA, peerB, peerC)
|
|
||||||
table.Withdraw(netip.MustParsePrefix("10.9.0.0/16"), peerB)
|
|
||||||
table.Withdraw(netip.MustParsePrefix("10.9.0.0/16"), peerC)
|
|
||||||
announceAll(table, "2001:db8::/32", asA, 10, peerA)
|
|
||||||
announceAll(table, prefixV6, asB, 20, peerB)
|
|
||||||
announceAll(table, "3000::/16", asA, 10, peerA)
|
|
||||||
// Shorter than the lookup uses.
|
|
||||||
announceAll(table, "192.0.0.0/7", asA, 10, peerA)
|
|
||||||
announceAll(table, "2000::/15", asA, 10, peerA)
|
|
||||||
|
|
||||||
tests := []struct {
|
|
||||||
addr string
|
|
||||||
// want is the summary's prefix, or empty when none is found.
|
|
||||||
want string
|
|
||||||
originASN int
|
|
||||||
peers int
|
|
||||||
newest int // in seconds after the start; 0 for none
|
|
||||||
}{
|
|
||||||
{"10.1.2.3", prefixV4, asA, 2, 20},
|
|
||||||
{"::ffff:10.1.2.3", prefixV4, asA, 2, 20},
|
|
||||||
{"10.1.3.4", "10.1.0.0/16", asB, 3, 30},
|
|
||||||
{"10.2.0.1", "10.0.0.0/8", asA, 1, 10},
|
|
||||||
{"10.9.0.1", "10.9.0.0/16", asA, 1, 0},
|
|
||||||
{"192.0.2.1", "", 0, 0, 0},
|
|
||||||
{"2001:db8:1::5", prefixV6, asB, 1, 20},
|
|
||||||
{"2001:db8:2::1", "2001:db8::/32", asA, 1, 10},
|
|
||||||
{"3000::1", "3000::/16", asA, 1, 10},
|
|
||||||
{"2001:db9::1", "", 0, 0, 0},
|
|
||||||
}
|
|
||||||
|
|
||||||
for _, tc := range tests {
|
|
||||||
t.Run(tc.addr, func(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
got, found := table.Lookup(netip.MustParseAddr(tc.addr))
|
|
||||||
if tc.want == "" {
|
|
||||||
if found {
|
|
||||||
t.Errorf("Lookup(%s) = %+v, want none", tc.addr, got)
|
|
||||||
}
|
|
||||||
|
|
||||||
return
|
|
||||||
}
|
|
||||||
|
|
||||||
want := routetable.PrefixSummary{
|
|
||||||
Prefix: netip.MustParsePrefix(tc.want),
|
|
||||||
OriginASN: tc.originASN,
|
|
||||||
Peers: tc.peers,
|
|
||||||
}
|
|
||||||
if tc.newest != 0 {
|
|
||||||
want.Newest = timeAt(tc.newest)
|
|
||||||
}
|
|
||||||
|
|
||||||
if !found || !reflect.DeepEqual(got, want) {
|
|
||||||
t.Errorf("Lookup(%s) = %+v, %v; want %+v", tc.addr, got, found, want)
|
|
||||||
}
|
|
||||||
})
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestOriginatedPrefixes(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
table := routetable.New(routetable.Params{})
|
|
||||||
announceAll(table, "2001:db8::/32", asA, 30, peerA)
|
|
||||||
announceAll(table, prefixV4, asA, 10, peerA)
|
|
||||||
// asB's newer route for the same prefix does not count for asA.
|
|
||||||
announceAll(table, prefixV4, asB, 40, peerB)
|
|
||||||
announceAll(table, "10.1.0.0/16", asA, 10, peerA, peerB)
|
|
||||||
announceAll(table, "10.9.0.0/16", asA, 10, peerA, peerB, peerC)
|
|
||||||
table.Withdraw(netip.MustParsePrefix("10.9.0.0/16"), peerB)
|
|
||||||
table.Withdraw(netip.MustParsePrefix("10.9.0.0/16"), peerC)
|
|
||||||
|
|
||||||
want := []routetable.OriginatedPrefix{
|
|
||||||
{Prefix: netip.MustParsePrefix("10.1.0.0/16"), Newest: timeAt(20)},
|
|
||||||
{Prefix: netip.MustParsePrefix(prefixV4), Newest: timeAt(10)},
|
|
||||||
{Prefix: netip.MustParsePrefix("10.9.0.0/16")},
|
|
||||||
{Prefix: netip.MustParsePrefix("2001:db8::/32"), Newest: timeAt(30)},
|
|
||||||
}
|
|
||||||
|
|
||||||
got := table.OriginatedPrefixes(asA)
|
|
||||||
if !reflect.DeepEqual(got, want) {
|
|
||||||
t.Errorf("OriginatedPrefixes(%d) = %+v, want %+v", asA, got, want)
|
|
||||||
}
|
|
||||||
|
|
||||||
got = table.OriginatedPrefixes(asC)
|
|
||||||
if len(got) != 0 {
|
|
||||||
t.Errorf("OriginatedPrefixes(%d) = %+v, want none", asC, got)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestSamplePrefixes(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
table := routetable.New(routetable.Params{})
|
|
||||||
|
|
||||||
announced := []string{
|
|
||||||
"10.0.0.0/24", "10.0.1.0/24", "10.0.2.0/24", "10.0.3.0/24",
|
|
||||||
"10.0.4.0/24", "10.0.0.0/16",
|
|
||||||
"2001:db8::/48", "2001:db8:1::/48", "2001:db8:2::/48",
|
|
||||||
}
|
|
||||||
for _, prefix := range announced {
|
|
||||||
announceAll(table, prefix, asA, 10, peerA)
|
|
||||||
}
|
|
||||||
|
|
||||||
tests := []struct {
|
|
||||||
name string
|
|
||||||
ipVersion, maskLength, limit int
|
|
||||||
want int
|
|
||||||
}{
|
|
||||||
{"fewer than there are", 4, 24, 3, 3},
|
|
||||||
{"more than there are", 4, 24, 10, 5},
|
|
||||||
{"IPv6", 6, 48, 10, 3},
|
|
||||||
{"a mask length with none", 4, 20, 10, 0},
|
|
||||||
{"no such IP version", 5, 24, 10, 0},
|
|
||||||
{"a mask length too long", 4, 33, 10, 0},
|
|
||||||
{"a negative mask length", 4, -1, 10, 0},
|
|
||||||
{"a limit of zero", 4, 24, 0, 0},
|
|
||||||
}
|
|
||||||
|
|
||||||
for _, tc := range tests {
|
|
||||||
t.Run(tc.name, func(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
sample := table.SamplePrefixes(tc.ipVersion, tc.maskLength, tc.limit)
|
|
||||||
if len(sample) != tc.want {
|
|
||||||
t.Fatalf("SamplePrefixes(%d, %d, %d) returned %d prefixes, want %d",
|
|
||||||
tc.ipVersion, tc.maskLength, tc.limit, len(sample), tc.want)
|
|
||||||
}
|
|
||||||
|
|
||||||
seen := make(map[netip.Prefix]bool)
|
|
||||||
|
|
||||||
for _, got := range sample {
|
|
||||||
want := routetable.PrefixSummary{
|
|
||||||
Prefix: got.Prefix, OriginASN: asA, Peers: 1, Newest: timeAt(10),
|
|
||||||
}
|
|
||||||
if seen[got.Prefix] || got.Prefix.Bits() != tc.maskLength ||
|
|
||||||
!reflect.DeepEqual(got, want) {
|
|
||||||
t.Errorf("SamplePrefixes(%d, %d, %d) returned %+v",
|
|
||||||
tc.ipVersion, tc.maskLength, tc.limit, got)
|
|
||||||
}
|
|
||||||
|
|
||||||
seen[got.Prefix] = true
|
|
||||||
}
|
|
||||||
})
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestCounts(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
table := routetable.New(routetable.Params{})
|
|
||||||
checkCounts(t, table, routetable.Counts{
|
|
||||||
IPv4PrefixesByLength: byLength(ipv4Bits, nil),
|
|
||||||
IPv6PrefixesByLength: byLength(ipv6Bits, nil),
|
|
||||||
})
|
|
||||||
|
|
||||||
announceAll(table, prefixV4, asA, 10, peerA, peerB)
|
|
||||||
announceAll(table, "10.2.0.0/16", asA, 15, peerA)
|
|
||||||
announceAll(table, "2001:db8::/32", asA, 5, peerA)
|
|
||||||
announceAll(table, "2001:db8::/32", asA, 25, peerC)
|
|
||||||
checkCounts(t, table, routetable.Counts{
|
|
||||||
Pairs: 3,
|
|
||||||
Peers: 3,
|
|
||||||
IPv4Prefixes: 2,
|
|
||||||
IPv6Prefixes: 1,
|
|
||||||
IPv4PrefixesByLength: byLength(ipv4Bits, map[int]int{16: 1, 24: 1}),
|
|
||||||
IPv6PrefixesByLength: byLength(ipv6Bits, map[int]int{32: 1}),
|
|
||||||
IPv4Routes: 3,
|
|
||||||
IPv6Routes: 2,
|
|
||||||
OldestKeptRoute: timeAt(5),
|
|
||||||
NewestKeptRoute: timeAt(25),
|
|
||||||
})
|
|
||||||
|
|
||||||
// peerC's route drops peerA's from the kept ones, and peerA leaves the
|
|
||||||
// IPv6 prefix.
|
|
||||||
announceAll(table, prefixV4, asA, 30, peerC)
|
|
||||||
table.Withdraw(netip.MustParsePrefix("2001:db8::/32"), peerA)
|
|
||||||
checkCounts(t, table, routetable.Counts{
|
|
||||||
Pairs: 3,
|
|
||||||
Peers: 3,
|
|
||||||
IPv4Prefixes: 2,
|
|
||||||
IPv6Prefixes: 1,
|
|
||||||
IPv4PrefixesByLength: byLength(ipv4Bits, map[int]int{16: 1, 24: 1}),
|
|
||||||
IPv6PrefixesByLength: byLength(ipv6Bits, map[int]int{32: 1}),
|
|
||||||
IPv4Routes: 4,
|
|
||||||
IPv6Routes: 1,
|
|
||||||
OldestKeptRoute: timeAt(15),
|
|
||||||
NewestKeptRoute: timeAt(30),
|
|
||||||
})
|
|
||||||
|
|
||||||
// peerC was the last peer of the IPv6 prefix. It keeps its number.
|
|
||||||
table.PeerDown(peerC)
|
|
||||||
checkCounts(t, table, routetable.Counts{
|
|
||||||
Pairs: 2,
|
|
||||||
Peers: 3,
|
|
||||||
IPv4Prefixes: 2,
|
|
||||||
IPv4PrefixesByLength: byLength(ipv4Bits, map[int]int{16: 1, 24: 1}),
|
|
||||||
IPv6PrefixesByLength: byLength(ipv6Bits, nil),
|
|
||||||
IPv4Routes: 3,
|
|
||||||
OldestKeptRoute: timeAt(15),
|
|
||||||
NewestKeptRoute: timeAt(20),
|
|
||||||
})
|
|
||||||
}
|
|
||||||
|
|
||||||
// The address lengths of the two IP versions.
|
|
||||||
const (
|
|
||||||
ipv4Bits = 32
|
|
||||||
ipv6Bits = 128
|
|
||||||
)
|
|
||||||
|
|
||||||
// byLength returns the prefix counts by mask length, for addresses of bits
|
|
||||||
// bits, with counts[n] prefixes of mask length n and none of any other.
|
|
||||||
func byLength(bits int, counts map[int]int) []int {
|
|
||||||
all := make([]int, bits+1)
|
|
||||||
for n, count := range counts {
|
|
||||||
all[n] = count
|
|
||||||
}
|
|
||||||
|
|
||||||
return all
|
|
||||||
}
|
|
||||||
|
|
||||||
// checkCounts checks that the table's counts are want.
|
|
||||||
func checkCounts(t *testing.T, table *routetable.Table, want routetable.Counts) {
|
|
||||||
t.Helper()
|
|
||||||
|
|
||||||
got := table.Counts()
|
|
||||||
if !reflect.DeepEqual(got, want) {
|
|
||||||
t.Errorf("Counts() = %+v, want %+v", got, want)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
@@ -1,677 +0,0 @@
|
|||||||
// Package routetable holds the whole routing table in memory. For each prefix
|
|
||||||
// and origin AS, it keeps the peers whose current route to the prefix has that
|
|
||||||
// origin AS. It also keeps the two newest of those routes in full. It applies
|
|
||||||
// announcements, withdrawals and peers' sessions going down. It answers the
|
|
||||||
// route reads that the pages and the API need.
|
|
||||||
package routetable
|
|
||||||
|
|
||||||
import (
|
|
||||||
"cmp"
|
|
||||||
"maps"
|
|
||||||
"net/netip"
|
|
||||||
"slices"
|
|
||||||
"sync"
|
|
||||||
"time"
|
|
||||||
|
|
||||||
"github.com/bits-and-blooms/bitset"
|
|
||||||
"github.com/google/btree"
|
|
||||||
)
|
|
||||||
|
|
||||||
const (
|
|
||||||
// keptRoutesPerOrigin is how many routes of each prefix and origin AS are
|
|
||||||
// kept in full: the newest ones. Every other peer that carries the prefix
|
|
||||||
// with that origin AS keeps only its bit in the peer set.
|
|
||||||
keptRoutesPerOrigin = 2
|
|
||||||
|
|
||||||
// keptTimesDegree is the degree of the B-tree that holds the times of the
|
|
||||||
// kept routes in order: each of its nodes holds up to 63 times.
|
|
||||||
keptTimesDegree = 32
|
|
||||||
|
|
||||||
ipv4Bits = 32
|
|
||||||
ipv6Bits = 128
|
|
||||||
|
|
||||||
// The address lookup ignores prefixes shorter than these mask lengths. A
|
|
||||||
// default route or a route this short only says where a peer sends traffic
|
|
||||||
// it has no better route for; it does not say who holds the address.
|
|
||||||
minLookupMaskLengthV4 = 8
|
|
||||||
minLookupMaskLengthV6 = 16
|
|
||||||
|
|
||||||
// The IP versions SamplePrefixes takes.
|
|
||||||
ipVersion4 = 4
|
|
||||||
ipVersion6 = 6
|
|
||||||
)
|
|
||||||
|
|
||||||
// Route is one peer's route to one prefix. The next hop is the peer itself, so
|
|
||||||
// it is not stored.
|
|
||||||
type Route struct {
|
|
||||||
// Peer is the peer's address.
|
|
||||||
Peer string
|
|
||||||
ASPath []int
|
|
||||||
// Time is when the peer announced the route.
|
|
||||||
Time time.Time
|
|
||||||
}
|
|
||||||
|
|
||||||
// Origin is one origin AS of a prefix: the peers whose current route to the
|
|
||||||
// prefix has that origin AS, and the newest of those routes in full.
|
|
||||||
type Origin struct {
|
|
||||||
// ASN is the origin AS. 0 means no known origin: the path ends in an AS
|
|
||||||
// set of several members.
|
|
||||||
ASN int
|
|
||||||
// Peers are the peers' addresses, in the order of their numbers: the
|
|
||||||
// order they were first seen in.
|
|
||||||
Peers []string
|
|
||||||
// Routes are the kept routes, newest first: at most two, and none when
|
|
||||||
// both were withdrawn while other peers still carry the prefix.
|
|
||||||
Routes []Route
|
|
||||||
}
|
|
||||||
|
|
||||||
// Change is the current content of one prefix and origin AS pair that changed:
|
|
||||||
// what a database row for the pair holds.
|
|
||||||
type Change struct {
|
|
||||||
Prefix netip.Prefix
|
|
||||||
OriginASN int
|
|
||||||
// Removed says that no peer carries the prefix with this origin AS any
|
|
||||||
// more. Peers and Routes are then nil.
|
|
||||||
Removed bool
|
|
||||||
// Peers holds the number of each peer that carries the prefix with this
|
|
||||||
// origin AS.
|
|
||||||
Peers *bitset.BitSet
|
|
||||||
// Routes are the kept routes, newest first.
|
|
||||||
Routes []Route
|
|
||||||
}
|
|
||||||
|
|
||||||
// Params configures New.
|
|
||||||
type Params struct {
|
|
||||||
// PeerNumbers holds the number each peer was given before, by address. A
|
|
||||||
// peer not in it gets the next number above all of them the first time it
|
|
||||||
// announces a route.
|
|
||||||
PeerNumbers map[string]uint
|
|
||||||
}
|
|
||||||
|
|
||||||
// Table is the routing table. Its methods are safe for concurrent use.
|
|
||||||
type Table struct {
|
|
||||||
mu sync.RWMutex
|
|
||||||
|
|
||||||
ipv4 family
|
|
||||||
ipv6 family
|
|
||||||
// byOrigin holds, for each origin AS, the prefixes it originates.
|
|
||||||
byOrigin map[int]map[netip.Prefix]*origin
|
|
||||||
|
|
||||||
// peerNumbers and peerAddresses give each peer's number by its address
|
|
||||||
// and its address by its number. nextPeerNumber is the number the next
|
|
||||||
// new peer gets.
|
|
||||||
peerNumbers map[string]uint
|
|
||||||
peerAddresses map[uint]string
|
|
||||||
nextPeerNumber uint
|
|
||||||
// peerPrefixes[n] counts the prefixes the peer numbered n carries, for
|
|
||||||
// each n below nextPeerNumber. It is a slice, not a map, because Load
|
|
||||||
// counts once for every peer of every pair it loads: with a map, reading
|
|
||||||
// a full table at startup took several times as long.
|
|
||||||
peerPrefixes []int
|
|
||||||
|
|
||||||
// newPeers and changed are what TakeChanges hands over: the peers
|
|
||||||
// numbered since it last ran, and the prefix and origin AS pairs changed
|
|
||||||
// since it last took them.
|
|
||||||
newPeers map[string]uint
|
|
||||||
changed map[pair]struct{}
|
|
||||||
|
|
||||||
// keptTimes counts the kept routes at each time, in time order, so its
|
|
||||||
// first and last items are the oldest and newest kept routes.
|
|
||||||
keptTimes *btree.BTreeG[keptTime]
|
|
||||||
// originsWithoutKeptRoute counts the origins that peers carry but that
|
|
||||||
// keep no route.
|
|
||||||
originsWithoutKeptRoute int
|
|
||||||
}
|
|
||||||
|
|
||||||
// New returns an empty table that numbers new peers on from
|
|
||||||
// params.PeerNumbers.
|
|
||||||
func New(params Params) *Table {
|
|
||||||
t := &Table{
|
|
||||||
ipv4: newFamily(ipv4Bits, minLookupMaskLengthV4),
|
|
||||||
ipv6: newFamily(ipv6Bits, minLookupMaskLengthV6),
|
|
||||||
byOrigin: make(map[int]map[netip.Prefix]*origin),
|
|
||||||
peerNumbers: make(map[string]uint, len(params.PeerNumbers)),
|
|
||||||
peerAddresses: make(map[uint]string, len(params.PeerNumbers)),
|
|
||||||
newPeers: make(map[string]uint),
|
|
||||||
changed: make(map[pair]struct{}),
|
|
||||||
keptTimes: btree.NewG(keptTimesDegree, func(a, b keptTime) bool {
|
|
||||||
return a.time.Before(b.time)
|
|
||||||
}),
|
|
||||||
}
|
|
||||||
|
|
||||||
for address, number := range params.PeerNumbers {
|
|
||||||
t.peerNumbers[address] = number
|
|
||||||
t.peerAddresses[number] = address
|
|
||||||
t.nextPeerNumber = max(t.nextPeerNumber, number+1)
|
|
||||||
}
|
|
||||||
|
|
||||||
t.peerPrefixes = make([]int, t.nextPeerNumber)
|
|
||||||
|
|
||||||
return t
|
|
||||||
}
|
|
||||||
|
|
||||||
// Announce applies an announcement: the peer's current route to prefix is now
|
|
||||||
// r, with origin AS originASN. A peer that carried the prefix with another
|
|
||||||
// origin AS leaves that origin's peer set, and its route there is dropped. r
|
|
||||||
// is kept in full when it is one of the two newest routes of its prefix and
|
|
||||||
// origin AS; otherwise only the peer's bit is.
|
|
||||||
//
|
|
||||||
// An announcement whose path is empty or holds AS 0 is a withdrawal, and the
|
|
||||||
// caller passes it to Withdraw instead.
|
|
||||||
func (t *Table) Announce(prefix netip.Prefix, originASN int, r Route) {
|
|
||||||
t.mu.Lock()
|
|
||||||
defer t.mu.Unlock()
|
|
||||||
|
|
||||||
if !prefix.IsValid() {
|
|
||||||
return
|
|
||||||
}
|
|
||||||
|
|
||||||
prefix = prefix.Masked()
|
|
||||||
peer := t.numberPeer(r.Peer)
|
|
||||||
|
|
||||||
current := originWithPeer(t.origins(prefix), peer)
|
|
||||||
if current != nil && current.asn != originASN {
|
|
||||||
t.leave(prefix, current, peer)
|
|
||||||
}
|
|
||||||
|
|
||||||
t.join(prefix, t.findOrAddOrigin(prefix, originASN), route{
|
|
||||||
peer: peer,
|
|
||||||
path: slices.Clone(r.ASPath),
|
|
||||||
time: r.Time,
|
|
||||||
})
|
|
||||||
}
|
|
||||||
|
|
||||||
// Withdraw applies a withdrawal: the peer no longer carries prefix. It leaves
|
|
||||||
// the prefix's peer set, and its kept route, if any, is dropped.
|
|
||||||
func (t *Table) Withdraw(prefix netip.Prefix, peer string) {
|
|
||||||
t.mu.Lock()
|
|
||||||
defer t.mu.Unlock()
|
|
||||||
|
|
||||||
number, known := t.peerNumbers[peer]
|
|
||||||
if !known || !prefix.IsValid() {
|
|
||||||
return
|
|
||||||
}
|
|
||||||
|
|
||||||
prefix = prefix.Masked()
|
|
||||||
|
|
||||||
o := originWithPeer(t.origins(prefix), number)
|
|
||||||
if o != nil {
|
|
||||||
t.leave(prefix, o, number)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// PeerDown applies a peer's session going down: the peer leaves every peer
|
|
||||||
// set, and every route of it is dropped. When the peer carries any prefix, it
|
|
||||||
// visits every prefix in the table.
|
|
||||||
func (t *Table) PeerDown(peer string) {
|
|
||||||
t.mu.Lock()
|
|
||||||
defer t.mu.Unlock()
|
|
||||||
|
|
||||||
number, known := t.peerNumbers[peer]
|
|
||||||
if !known || t.peerPrefixes[number] == 0 {
|
|
||||||
return
|
|
||||||
}
|
|
||||||
|
|
||||||
for _, f := range []*family{&t.ipv4, &t.ipv6} {
|
|
||||||
for _, prefixes := range f.byLength {
|
|
||||||
for prefix, origins := range prefixes {
|
|
||||||
o := originWithPeer(origins, number)
|
|
||||||
if o != nil {
|
|
||||||
t.leave(prefix, o, number)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// Load puts one prefix and origin AS pair that the database holds into the
|
|
||||||
// table, as startup does with each row it reads. The change's peers join the
|
|
||||||
// pair's peer set, and its routes are kept as an announcement's would be.
|
|
||||||
// Unlike an update, Load records no change, so TakeChanges does not hand the
|
|
||||||
// pair back to be written again. Loading a pair twice changes nothing.
|
|
||||||
//
|
|
||||||
// The caller makes sure that every peer change names, in its peer set or in
|
|
||||||
// its routes, has a number in Params.PeerNumbers.
|
|
||||||
func (t *Table) Load(change Change) {
|
|
||||||
t.mu.Lock()
|
|
||||||
defer t.mu.Unlock()
|
|
||||||
|
|
||||||
// A removed pair has no peers, and a pair without peers is not in the
|
|
||||||
// table.
|
|
||||||
if change.Peers == nil || change.Peers.None() || !change.Prefix.IsValid() {
|
|
||||||
return
|
|
||||||
}
|
|
||||||
|
|
||||||
prefix := change.Prefix.Masked()
|
|
||||||
o := t.findOrAddOrigin(prefix, change.OriginASN)
|
|
||||||
wasWithoutKeptRoute := o.withoutKeptRoute()
|
|
||||||
|
|
||||||
// The peers not yet in the peer set join it in one union, which takes
|
|
||||||
// less time than setting their bits one at a time.
|
|
||||||
joining := change.Peers.Difference(&o.peers)
|
|
||||||
o.peers.InPlaceUnion(joining)
|
|
||||||
t.familyOf(prefix.Addr()).routes += int(joining.Count())
|
|
||||||
|
|
||||||
for peer := range joining.EachSet() {
|
|
||||||
t.peerPrefixes[peer]++
|
|
||||||
}
|
|
||||||
|
|
||||||
// Oldest first, as they were announced, so that two routes with the same
|
|
||||||
// time are kept in the order they had.
|
|
||||||
for _, r := range slices.Backward(change.Routes) {
|
|
||||||
peer := t.peerNumbers[r.Peer]
|
|
||||||
|
|
||||||
t.dropKeptRoute(o, peer)
|
|
||||||
t.keepIfNewest(o, route{
|
|
||||||
peer: peer,
|
|
||||||
path: slices.Clone(r.ASPath),
|
|
||||||
time: r.Time,
|
|
||||||
})
|
|
||||||
}
|
|
||||||
|
|
||||||
t.countOriginWithoutKeptRoute(o, wasWithoutKeptRoute)
|
|
||||||
}
|
|
||||||
|
|
||||||
// TakeChanges returns at most maxChanges of the prefix and origin AS pairs that
|
|
||||||
// changed since they were last taken, in prefix order, each with its current
|
|
||||||
// content. The other changed pairs wait for a later call. It also returns every
|
|
||||||
// peer numbered since it last ran, by address, so a pair taken later names only
|
|
||||||
// peers already returned. It then forgets what it returned.
|
|
||||||
func (t *Table) TakeChanges(maxChanges int) ([]Change, map[string]uint) {
|
|
||||||
t.mu.Lock()
|
|
||||||
defer t.mu.Unlock()
|
|
||||||
|
|
||||||
changes := make([]Change, 0, min(maxChanges, len(t.changed)))
|
|
||||||
for key := range t.changed {
|
|
||||||
if len(changes) == maxChanges {
|
|
||||||
break
|
|
||||||
}
|
|
||||||
|
|
||||||
changes = append(changes, t.currentContent(key))
|
|
||||||
delete(t.changed, key)
|
|
||||||
}
|
|
||||||
|
|
||||||
// A map keeps the memory of its largest size, so an emptied one is
|
|
||||||
// replaced.
|
|
||||||
if len(t.changed) == 0 {
|
|
||||||
t.changed = make(map[pair]struct{})
|
|
||||||
}
|
|
||||||
|
|
||||||
slices.SortFunc(changes, func(a, b Change) int {
|
|
||||||
return cmp.Or(a.Prefix.Compare(b.Prefix),
|
|
||||||
cmp.Compare(a.OriginASN, b.OriginASN))
|
|
||||||
})
|
|
||||||
|
|
||||||
newPeers := t.newPeers
|
|
||||||
t.newPeers = make(map[string]uint)
|
|
||||||
|
|
||||||
return changes, newPeers
|
|
||||||
}
|
|
||||||
|
|
||||||
// PutBackChanges records again what one TakeChanges call returned, for a
|
|
||||||
// caller that could not write it. The next TakeChanges returns those peers
|
|
||||||
// again, and those pairs wait to be taken again like any changed pair. Each is
|
|
||||||
// then taken with its content at that time, so a later change of a pair
|
|
||||||
// replaces the earlier.
|
|
||||||
func (t *Table) PutBackChanges(changes []Change, newPeers map[string]uint) {
|
|
||||||
t.mu.Lock()
|
|
||||||
defer t.mu.Unlock()
|
|
||||||
|
|
||||||
for _, change := range changes {
|
|
||||||
t.changed[pair{prefix: change.Prefix, asn: change.OriginASN}] = struct{}{}
|
|
||||||
}
|
|
||||||
|
|
||||||
maps.Copy(t.newPeers, newPeers)
|
|
||||||
}
|
|
||||||
|
|
||||||
// ChangeCount returns how many changed prefix and origin AS pairs wait for
|
|
||||||
// TakeChanges.
|
|
||||||
func (t *Table) ChangeCount() int {
|
|
||||||
t.mu.RLock()
|
|
||||||
defer t.mu.RUnlock()
|
|
||||||
|
|
||||||
return len(t.changed)
|
|
||||||
}
|
|
||||||
|
|
||||||
// numberPeer returns the peer's number, and gives it the next one the first
|
|
||||||
// time the table sees it.
|
|
||||||
func (t *Table) numberPeer(address string) uint {
|
|
||||||
number, known := t.peerNumbers[address]
|
|
||||||
if known {
|
|
||||||
return number
|
|
||||||
}
|
|
||||||
|
|
||||||
number = t.nextPeerNumber
|
|
||||||
t.nextPeerNumber++
|
|
||||||
t.peerNumbers[address] = number
|
|
||||||
t.peerAddresses[number] = address
|
|
||||||
t.peerPrefixes = append(t.peerPrefixes, 0)
|
|
||||||
t.newPeers[address] = number
|
|
||||||
|
|
||||||
return number
|
|
||||||
}
|
|
||||||
|
|
||||||
// familyOf returns the part of the table for addr's IP version.
|
|
||||||
func (t *Table) familyOf(addr netip.Addr) *family {
|
|
||||||
if addr.Is4() {
|
|
||||||
return &t.ipv4
|
|
||||||
}
|
|
||||||
|
|
||||||
return &t.ipv6
|
|
||||||
}
|
|
||||||
|
|
||||||
// prefixesOfLength returns the prefixes of the IP version and the mask length
|
|
||||||
// of prefix, a valid masked prefix.
|
|
||||||
func (t *Table) prefixesOfLength(prefix netip.Prefix) map[netip.Prefix][]*origin {
|
|
||||||
return t.familyOf(prefix.Addr()).byLength[prefix.Bits()]
|
|
||||||
}
|
|
||||||
|
|
||||||
// origins returns the origins of prefix, a valid masked prefix.
|
|
||||||
func (t *Table) origins(prefix netip.Prefix) []*origin {
|
|
||||||
return t.prefixesOfLength(prefix)[prefix]
|
|
||||||
}
|
|
||||||
|
|
||||||
// findOrAddOrigin returns the origin of prefix with AS asn, added with no
|
|
||||||
// peers when the prefix has none with that AS.
|
|
||||||
func (t *Table) findOrAddOrigin(prefix netip.Prefix, asn int) *origin {
|
|
||||||
prefixes := t.prefixesOfLength(prefix)
|
|
||||||
origins := prefixes[prefix]
|
|
||||||
|
|
||||||
i, found := slices.BinarySearchFunc(origins, asn,
|
|
||||||
func(o *origin, asn int) int { return cmp.Compare(o.asn, asn) })
|
|
||||||
if found {
|
|
||||||
return origins[i]
|
|
||||||
}
|
|
||||||
|
|
||||||
o := &origin{asn: asn}
|
|
||||||
prefixes[prefix] = slices.Insert(origins, i, o)
|
|
||||||
|
|
||||||
if t.byOrigin[asn] == nil {
|
|
||||||
t.byOrigin[asn] = make(map[netip.Prefix]*origin)
|
|
||||||
}
|
|
||||||
|
|
||||||
t.byOrigin[asn][prefix] = o
|
|
||||||
|
|
||||||
return o
|
|
||||||
}
|
|
||||||
|
|
||||||
// removeOrigin removes o, which no peer carries any more, from prefix, and the
|
|
||||||
// prefix from the table when it has no origin left.
|
|
||||||
func (t *Table) removeOrigin(prefix netip.Prefix, o *origin) {
|
|
||||||
prefixes := t.prefixesOfLength(prefix)
|
|
||||||
|
|
||||||
origins := slices.DeleteFunc(prefixes[prefix],
|
|
||||||
func(other *origin) bool { return other == o })
|
|
||||||
if len(origins) == 0 {
|
|
||||||
delete(prefixes, prefix)
|
|
||||||
} else {
|
|
||||||
prefixes[prefix] = origins
|
|
||||||
}
|
|
||||||
|
|
||||||
delete(t.byOrigin[o.asn], prefix)
|
|
||||||
|
|
||||||
if len(t.byOrigin[o.asn]) == 0 {
|
|
||||||
delete(t.byOrigin, o.asn)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// join puts the peer of r in the peer set of o, an origin of prefix, and keeps
|
|
||||||
// r in full when it is one of the two newest routes of o. The peer's earlier
|
|
||||||
// route, if o keeps one, is replaced: it is no longer the peer's current route.
|
|
||||||
// The pair is recorded as changed only when its peer set or kept routes change.
|
|
||||||
func (t *Table) join(prefix netip.Prefix, o *origin, r route) {
|
|
||||||
if slices.ContainsFunc(o.routes, r.equal) {
|
|
||||||
// The peer announced its kept route again.
|
|
||||||
return
|
|
||||||
}
|
|
||||||
|
|
||||||
wasWithoutKeptRoute := o.withoutKeptRoute()
|
|
||||||
|
|
||||||
joined := !o.peers.Test(r.peer)
|
|
||||||
if joined {
|
|
||||||
o.peers.Set(r.peer)
|
|
||||||
|
|
||||||
t.familyOf(prefix.Addr()).routes++
|
|
||||||
t.peerPrefixes[r.peer]++
|
|
||||||
}
|
|
||||||
|
|
||||||
dropped := t.dropKeptRoute(o, r.peer)
|
|
||||||
kept := t.keepIfNewest(o, r)
|
|
||||||
|
|
||||||
if joined || dropped || kept {
|
|
||||||
t.recordChange(prefix, o, wasWithoutKeptRoute)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// leave takes peer out of the peer set of o, an origin of prefix, and drops
|
|
||||||
// its kept route, if any.
|
|
||||||
func (t *Table) leave(prefix netip.Prefix, o *origin, peer uint) {
|
|
||||||
wasWithoutKeptRoute := o.withoutKeptRoute()
|
|
||||||
|
|
||||||
o.peers.Clear(peer)
|
|
||||||
t.dropKeptRoute(o, peer)
|
|
||||||
|
|
||||||
t.familyOf(prefix.Addr()).routes--
|
|
||||||
t.peerPrefixes[peer]--
|
|
||||||
|
|
||||||
t.recordChange(prefix, o, wasWithoutKeptRoute)
|
|
||||||
}
|
|
||||||
|
|
||||||
// keepIfNewest keeps r in o unless both kept routes of o are newer, and says
|
|
||||||
// whether it kept r. When o already keeps two routes, the older one is
|
|
||||||
// dropped, and its peer keeps only its bit.
|
|
||||||
func (t *Table) keepIfNewest(o *origin, r route) bool {
|
|
||||||
// The kept routes stay newest first: r goes before the first one that is
|
|
||||||
// not newer than it.
|
|
||||||
i := 0
|
|
||||||
for i < len(o.routes) && o.routes[i].time.After(r.time) {
|
|
||||||
i++
|
|
||||||
}
|
|
||||||
|
|
||||||
if i == keptRoutesPerOrigin {
|
|
||||||
return false
|
|
||||||
}
|
|
||||||
|
|
||||||
if len(o.routes) == keptRoutesPerOrigin {
|
|
||||||
t.dropKeptRoute(o, o.routes[keptRoutesPerOrigin-1].peer)
|
|
||||||
}
|
|
||||||
|
|
||||||
o.routes = slices.Insert(o.routes, i, r)
|
|
||||||
t.countKeptTime(r.time, 1)
|
|
||||||
|
|
||||||
return true
|
|
||||||
}
|
|
||||||
|
|
||||||
// dropKeptRoute drops the peer's route from the kept routes of o, if o keeps
|
|
||||||
// one, and says whether it did.
|
|
||||||
func (t *Table) dropKeptRoute(o *origin, peer uint) bool {
|
|
||||||
i := slices.IndexFunc(o.routes, func(r route) bool { return r.peer == peer })
|
|
||||||
if i < 0 {
|
|
||||||
return false
|
|
||||||
}
|
|
||||||
|
|
||||||
t.countKeptTime(o.routes[i].time, -1)
|
|
||||||
o.routes = slices.Delete(o.routes, i, i+1)
|
|
||||||
|
|
||||||
return true
|
|
||||||
}
|
|
||||||
|
|
||||||
// recordChange records that the pair of prefix and the AS of o changed, and
|
|
||||||
// brings the count of origins without a kept route up to date.
|
|
||||||
// wasWithoutKeptRoute says whether o was such an origin before the change. An
|
|
||||||
// origin that no peer carries any more is removed.
|
|
||||||
func (t *Table) recordChange(
|
|
||||||
prefix netip.Prefix, o *origin, wasWithoutKeptRoute bool,
|
|
||||||
) {
|
|
||||||
t.changed[pair{prefix: prefix, asn: o.asn}] = struct{}{}
|
|
||||||
|
|
||||||
t.countOriginWithoutKeptRoute(o, wasWithoutKeptRoute)
|
|
||||||
|
|
||||||
if o.peers.None() {
|
|
||||||
t.removeOrigin(prefix, o)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// countOriginWithoutKeptRoute brings the count of origins without a kept route
|
|
||||||
// up to date after a change to o. wasWithoutKeptRoute says whether o was such
|
|
||||||
// an origin before the change.
|
|
||||||
func (t *Table) countOriginWithoutKeptRoute(o *origin, wasWithoutKeptRoute bool) {
|
|
||||||
if wasWithoutKeptRoute {
|
|
||||||
t.originsWithoutKeptRoute--
|
|
||||||
}
|
|
||||||
|
|
||||||
if o.withoutKeptRoute() {
|
|
||||||
t.originsWithoutKeptRoute++
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// countKeptTime adds delta to the number of kept routes with time at.
|
|
||||||
func (t *Table) countKeptTime(at time.Time, delta int) {
|
|
||||||
count, _ := t.keptTimes.Get(keptTime{time: at})
|
|
||||||
count.time = at
|
|
||||||
count.routes += delta
|
|
||||||
|
|
||||||
if count.routes == 0 {
|
|
||||||
t.keptTimes.Delete(count)
|
|
||||||
|
|
||||||
return
|
|
||||||
}
|
|
||||||
|
|
||||||
t.keptTimes.ReplaceOrInsert(count)
|
|
||||||
}
|
|
||||||
|
|
||||||
// currentContent returns what the table holds for the pair now.
|
|
||||||
func (t *Table) currentContent(key pair) Change {
|
|
||||||
o := t.byOrigin[key.asn][key.prefix]
|
|
||||||
if o == nil {
|
|
||||||
return Change{Prefix: key.prefix, OriginASN: key.asn, Removed: true}
|
|
||||||
}
|
|
||||||
|
|
||||||
return Change{
|
|
||||||
Prefix: key.prefix,
|
|
||||||
OriginASN: key.asn,
|
|
||||||
Peers: o.peers.Clone(),
|
|
||||||
Routes: t.exportRoutes(o.routes),
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// exportRoutes returns copies of routes that name each peer by its address, or
|
|
||||||
// nil when there are none.
|
|
||||||
func (t *Table) exportRoutes(routes []route) []Route {
|
|
||||||
if len(routes) == 0 {
|
|
||||||
return nil
|
|
||||||
}
|
|
||||||
|
|
||||||
exported := make([]Route, 0, len(routes))
|
|
||||||
for _, r := range routes {
|
|
||||||
exported = append(exported, Route{
|
|
||||||
Peer: t.peerAddresses[r.peer],
|
|
||||||
ASPath: slices.Clone(r.path),
|
|
||||||
Time: r.time,
|
|
||||||
})
|
|
||||||
}
|
|
||||||
|
|
||||||
return exported
|
|
||||||
}
|
|
||||||
|
|
||||||
// family is the part of the table for one IP version.
|
|
||||||
type family struct {
|
|
||||||
// byLength[n] holds the prefixes of mask length n that some peer carries,
|
|
||||||
// each with its origins in AS number order.
|
|
||||||
byLength []map[netip.Prefix][]*origin
|
|
||||||
// minLookupLength is the shortest mask length the address lookup uses.
|
|
||||||
minLookupLength int
|
|
||||||
// routes counts the peer and prefix pairs: each peer that carries a
|
|
||||||
// prefix counts once for it.
|
|
||||||
routes int
|
|
||||||
}
|
|
||||||
|
|
||||||
// newFamily returns an empty family for addresses of bits bits.
|
|
||||||
func newFamily(bits, minLookupLength int) family {
|
|
||||||
f := family{
|
|
||||||
byLength: make([]map[netip.Prefix][]*origin, bits+1),
|
|
||||||
minLookupLength: minLookupLength,
|
|
||||||
}
|
|
||||||
|
|
||||||
for n := range f.byLength {
|
|
||||||
f.byLength[n] = make(map[netip.Prefix][]*origin)
|
|
||||||
}
|
|
||||||
|
|
||||||
return f
|
|
||||||
}
|
|
||||||
|
|
||||||
// prefixesByLength returns how many prefixes of each mask length f holds.
|
|
||||||
func (f *family) prefixesByLength() []int {
|
|
||||||
counts := make([]int, len(f.byLength))
|
|
||||||
for n, prefixes := range f.byLength {
|
|
||||||
counts[n] = len(prefixes)
|
|
||||||
}
|
|
||||||
|
|
||||||
return counts
|
|
||||||
}
|
|
||||||
|
|
||||||
// origin is what the table holds for one prefix and origin AS.
|
|
||||||
type origin struct {
|
|
||||||
asn int
|
|
||||||
// peers holds the number of each peer whose current route to the prefix
|
|
||||||
// has this origin AS.
|
|
||||||
peers bitset.BitSet
|
|
||||||
// routes are the newest of those routes, newest first, at most
|
|
||||||
// keptRoutesPerOrigin.
|
|
||||||
routes []route
|
|
||||||
}
|
|
||||||
|
|
||||||
// withoutKeptRoute says whether peers carry o but o keeps no route.
|
|
||||||
func (o *origin) withoutKeptRoute() bool {
|
|
||||||
return o.peers.Any() && len(o.routes) == 0
|
|
||||||
}
|
|
||||||
|
|
||||||
// newestTime returns the time of the newest kept route of o, or nil when o
|
|
||||||
// keeps none.
|
|
||||||
func (o *origin) newestTime() *time.Time {
|
|
||||||
if len(o.routes) == 0 {
|
|
||||||
return nil
|
|
||||||
}
|
|
||||||
|
|
||||||
newest := o.routes[0].time
|
|
||||||
|
|
||||||
return &newest
|
|
||||||
}
|
|
||||||
|
|
||||||
// originWithPeer returns the origin whose peer set holds peer, or nil when
|
|
||||||
// none does. A peer carries a prefix with one origin AS at most.
|
|
||||||
func originWithPeer(origins []*origin, peer uint) *origin {
|
|
||||||
for _, o := range origins {
|
|
||||||
if o.peers.Test(peer) {
|
|
||||||
return o
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
return nil
|
|
||||||
}
|
|
||||||
|
|
||||||
// route is a kept route. peer is the peer's number.
|
|
||||||
type route struct {
|
|
||||||
peer uint
|
|
||||||
path []int
|
|
||||||
time time.Time
|
|
||||||
}
|
|
||||||
|
|
||||||
// equal says whether r and other are the same peer's route with the same path
|
|
||||||
// and time.
|
|
||||||
func (r route) equal(other route) bool {
|
|
||||||
return r.peer == other.peer && r.time.Equal(other.time) &&
|
|
||||||
slices.Equal(r.path, other.path)
|
|
||||||
}
|
|
||||||
|
|
||||||
// pair names one prefix and origin AS.
|
|
||||||
type pair struct {
|
|
||||||
prefix netip.Prefix
|
|
||||||
asn int
|
|
||||||
}
|
|
||||||
|
|
||||||
// keptTime counts the kept routes with one time.
|
|
||||||
type keptTime struct {
|
|
||||||
time time.Time
|
|
||||||
routes int
|
|
||||||
}
|
|
||||||
@@ -1,667 +0,0 @@
|
|||||||
package routetable_test
|
|
||||||
|
|
||||||
import (
|
|
||||||
"math"
|
|
||||||
"net/netip"
|
|
||||||
"reflect"
|
|
||||||
"slices"
|
|
||||||
"sync"
|
|
||||||
"testing"
|
|
||||||
"time"
|
|
||||||
|
|
||||||
"github.com/bits-and-blooms/bitset"
|
|
||||||
|
|
||||||
"git.eeqj.de/sneak/routewatch/internal/routetable"
|
|
||||||
)
|
|
||||||
|
|
||||||
const (
|
|
||||||
peerA = "192.0.2.1"
|
|
||||||
peerB = "192.0.2.2"
|
|
||||||
peerC = "2001:db8::3"
|
|
||||||
peerD = "2001:db8::4"
|
|
||||||
|
|
||||||
asA = 64500
|
|
||||||
asB = 64501
|
|
||||||
asC = 64502
|
|
||||||
// transitAS is the first AS of every test path, the peer's own.
|
|
||||||
transitAS = 64496
|
|
||||||
|
|
||||||
prefixV4 = "10.1.2.0/24"
|
|
||||||
prefixV6 = "2001:db8:1::/48"
|
|
||||||
)
|
|
||||||
|
|
||||||
// at returns the time second seconds after the start of every test.
|
|
||||||
func at(second int) time.Time {
|
|
||||||
return time.Date(2026, 1, 2, 3, 4, 5, 0, time.UTC).
|
|
||||||
Add(time.Duration(second) * time.Second)
|
|
||||||
}
|
|
||||||
|
|
||||||
// timeAt returns at(second) for a field that may be nil.
|
|
||||||
func timeAt(second int) *time.Time {
|
|
||||||
t := at(second)
|
|
||||||
|
|
||||||
return &t
|
|
||||||
}
|
|
||||||
|
|
||||||
// routeOf returns the route peer announces with origin AS asn second seconds
|
|
||||||
// after the start.
|
|
||||||
func routeOf(peer string, asn, second int) routetable.Route {
|
|
||||||
return routetable.Route{
|
|
||||||
Peer: peer, ASPath: []int{transitAS, asn}, Time: at(second),
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// step is one update a test applies to a table for one prefix.
|
|
||||||
type step func(table *routetable.Table, prefix netip.Prefix)
|
|
||||||
|
|
||||||
// announce is the step of peer announcing prefix with origin AS asn second
|
|
||||||
// seconds after the start.
|
|
||||||
func announce(peer string, asn, second int) step {
|
|
||||||
return func(table *routetable.Table, prefix netip.Prefix) {
|
|
||||||
table.Announce(prefix, asn, routeOf(peer, asn, second))
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// withdraw is the step of peer withdrawing prefix.
|
|
||||||
func withdraw(peer string) step {
|
|
||||||
return func(table *routetable.Table, prefix netip.Prefix) {
|
|
||||||
table.Withdraw(prefix, peer)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// peerDown is the step of peer's session going down.
|
|
||||||
func peerDown(peer string) step {
|
|
||||||
return func(table *routetable.Table, _ netip.Prefix) {
|
|
||||||
table.PeerDown(peer)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// rule is one case of the rule that decides what the table holds for a prefix:
|
|
||||||
// the steps applied to an empty table, and the prefix's origins after them.
|
|
||||||
// The peers are numbered in the order they first announce, which is the order
|
|
||||||
// Origin.Peers lists them in.
|
|
||||||
type rule struct {
|
|
||||||
name string
|
|
||||||
steps []step
|
|
||||||
want []routetable.Origin
|
|
||||||
}
|
|
||||||
|
|
||||||
// checkRules runs each case for an IPv4 and an IPv6 prefix.
|
|
||||||
func checkRules(t *testing.T, rules []rule) {
|
|
||||||
t.Helper()
|
|
||||||
|
|
||||||
for _, tc := range rules {
|
|
||||||
for _, text := range []string{prefixV4, prefixV6} {
|
|
||||||
t.Run(tc.name+" "+text, func(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
prefix := netip.MustParsePrefix(text)
|
|
||||||
table := routetable.New(routetable.Params{})
|
|
||||||
|
|
||||||
for _, apply := range tc.steps {
|
|
||||||
apply(table, prefix)
|
|
||||||
}
|
|
||||||
|
|
||||||
got := table.Origins(prefix)
|
|
||||||
if !reflect.DeepEqual(got, tc.want) {
|
|
||||||
t.Errorf("Origins(%s) = %+v, want %+v", prefix, got, tc.want)
|
|
||||||
}
|
|
||||||
})
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestAnnouncementsKeepTheTwoNewestRoutes(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
checkRules(t, []rule{
|
|
||||||
{
|
|
||||||
name: "an announcement is kept",
|
|
||||||
steps: []step{announce(peerA, asA, 10)},
|
|
||||||
want: []routetable.Origin{{ASN: asA, Peers: []string{peerA},
|
|
||||||
Routes: []routetable.Route{routeOf(peerA, asA, 10)}}},
|
|
||||||
},
|
|
||||||
{
|
|
||||||
name: "a lagging peer's older route is not kept",
|
|
||||||
steps: []step{announce(peerA, asA, 20), announce(peerB, asA, 30),
|
|
||||||
announce(peerC, asA, 10)},
|
|
||||||
want: []routetable.Origin{{ASN: asA,
|
|
||||||
Peers: []string{peerA, peerB, peerC}, Routes: []routetable.Route{
|
|
||||||
routeOf(peerB, asA, 30), routeOf(peerA, asA, 20),
|
|
||||||
}}},
|
|
||||||
},
|
|
||||||
{
|
|
||||||
name: "a third newer route drops the older kept one, whose peer " +
|
|
||||||
"keeps its bit",
|
|
||||||
steps: []step{announce(peerA, asA, 10), announce(peerB, asA, 20),
|
|
||||||
announce(peerC, asA, 30)},
|
|
||||||
want: []routetable.Origin{{ASN: asA,
|
|
||||||
Peers: []string{peerA, peerB, peerC}, Routes: []routetable.Route{
|
|
||||||
routeOf(peerC, asA, 30), routeOf(peerB, asA, 20),
|
|
||||||
}}},
|
|
||||||
},
|
|
||||||
{
|
|
||||||
name: "a route as old as the older kept one replaces it",
|
|
||||||
steps: []step{announce(peerA, asA, 10), announce(peerB, asA, 20),
|
|
||||||
announce(peerC, asA, 10)},
|
|
||||||
want: []routetable.Origin{{ASN: asA,
|
|
||||||
Peers: []string{peerA, peerB, peerC}, Routes: []routetable.Route{
|
|
||||||
routeOf(peerB, asA, 20), routeOf(peerC, asA, 10),
|
|
||||||
}}},
|
|
||||||
},
|
|
||||||
{
|
|
||||||
name: "a peer's new route replaces its kept one",
|
|
||||||
steps: []step{announce(peerA, asA, 10), announce(peerB, asA, 20),
|
|
||||||
announce(peerA, asA, 30)},
|
|
||||||
want: []routetable.Origin{{ASN: asA, Peers: []string{peerA, peerB},
|
|
||||||
Routes: []routetable.Route{
|
|
||||||
routeOf(peerA, asA, 30), routeOf(peerB, asA, 20),
|
|
||||||
}}},
|
|
||||||
},
|
|
||||||
{
|
|
||||||
name: "a peer's new route replaces its kept one even when older",
|
|
||||||
steps: []step{announce(peerA, asA, 10), announce(peerB, asA, 20),
|
|
||||||
announce(peerB, asA, 5)},
|
|
||||||
want: []routetable.Origin{{ASN: asA, Peers: []string{peerA, peerB},
|
|
||||||
Routes: []routetable.Route{
|
|
||||||
routeOf(peerA, asA, 10), routeOf(peerB, asA, 5),
|
|
||||||
}}},
|
|
||||||
},
|
|
||||||
})
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestAPeerCarriesAPrefixWithOneOrigin(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
checkRules(t, []rule{
|
|
||||||
{
|
|
||||||
name: "a peer moving to a new origin leaves the old one",
|
|
||||||
steps: []step{announce(peerA, asA, 10), announce(peerB, asA, 20),
|
|
||||||
announce(peerA, asB, 30)},
|
|
||||||
want: []routetable.Origin{
|
|
||||||
{ASN: asA, Peers: []string{peerB},
|
|
||||||
Routes: []routetable.Route{routeOf(peerB, asA, 20)}},
|
|
||||||
{ASN: asB, Peers: []string{peerA},
|
|
||||||
Routes: []routetable.Route{routeOf(peerA, asB, 30)}},
|
|
||||||
},
|
|
||||||
},
|
|
||||||
{
|
|
||||||
name: "a peer whose route is not kept moves to a new origin",
|
|
||||||
steps: []step{announce(peerA, asA, 10), announce(peerB, asA, 20),
|
|
||||||
announce(peerC, asA, 30), announce(peerA, asB, 40)},
|
|
||||||
want: []routetable.Origin{
|
|
||||||
{ASN: asA, Peers: []string{peerB, peerC},
|
|
||||||
Routes: []routetable.Route{
|
|
||||||
routeOf(peerC, asA, 30), routeOf(peerB, asA, 20),
|
|
||||||
}},
|
|
||||||
{ASN: asB, Peers: []string{peerA},
|
|
||||||
Routes: []routetable.Route{routeOf(peerA, asB, 40)}},
|
|
||||||
},
|
|
||||||
},
|
|
||||||
{
|
|
||||||
name: "origins are listed in AS number order",
|
|
||||||
steps: []step{announce(peerA, asC, 10), announce(peerB, asA, 20),
|
|
||||||
announce(peerC, asB, 30)},
|
|
||||||
want: []routetable.Origin{
|
|
||||||
{ASN: asA, Peers: []string{peerB},
|
|
||||||
Routes: []routetable.Route{routeOf(peerB, asA, 20)}},
|
|
||||||
{ASN: asB, Peers: []string{peerC},
|
|
||||||
Routes: []routetable.Route{routeOf(peerC, asB, 30)}},
|
|
||||||
{ASN: asC, Peers: []string{peerA},
|
|
||||||
Routes: []routetable.Route{routeOf(peerA, asC, 10)}},
|
|
||||||
},
|
|
||||||
},
|
|
||||||
})
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestWithdrawalsAndPeerDowns(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
// threePeers is peerA, peerB and peerC announcing in turn; the routes of
|
|
||||||
// peerC and peerB are kept.
|
|
||||||
threePeers := []step{announce(peerA, asA, 10), announce(peerB, asA, 20),
|
|
||||||
announce(peerC, asA, 30)}
|
|
||||||
|
|
||||||
checkRules(t, []rule{
|
|
||||||
{
|
|
||||||
name: "a withdrawal drops the peer's kept route",
|
|
||||||
steps: append(slices.Clone(threePeers), withdraw(peerC)),
|
|
||||||
want: []routetable.Origin{{ASN: asA, Peers: []string{peerA, peerB},
|
|
||||||
Routes: []routetable.Route{routeOf(peerB, asA, 20)}}},
|
|
||||||
},
|
|
||||||
{
|
|
||||||
name: "a withdrawal of a route that is not kept leaves the kept ones",
|
|
||||||
steps: append(slices.Clone(threePeers), withdraw(peerA)),
|
|
||||||
want: []routetable.Origin{{ASN: asA, Peers: []string{peerB, peerC},
|
|
||||||
Routes: []routetable.Route{
|
|
||||||
routeOf(peerC, asA, 30), routeOf(peerB, asA, 20),
|
|
||||||
}}},
|
|
||||||
},
|
|
||||||
{
|
|
||||||
name: "an origin whose kept routes are withdrawn keeps its other " +
|
|
||||||
"peers",
|
|
||||||
steps: append(slices.Clone(threePeers), withdraw(peerB),
|
|
||||||
withdraw(peerC)),
|
|
||||||
want: []routetable.Origin{{ASN: asA, Peers: []string{peerA}}},
|
|
||||||
},
|
|
||||||
{
|
|
||||||
name: "an origin without kept routes keeps the next update",
|
|
||||||
steps: append(slices.Clone(threePeers), withdraw(peerB),
|
|
||||||
withdraw(peerC), announce(peerA, asA, 5)),
|
|
||||||
want: []routetable.Origin{{ASN: asA, Peers: []string{peerA},
|
|
||||||
Routes: []routetable.Route{routeOf(peerA, asA, 5)}}},
|
|
||||||
},
|
|
||||||
{
|
|
||||||
name: "an origin no peer carries is removed",
|
|
||||||
steps: []step{announce(peerA, asA, 10), announce(peerB, asB, 20),
|
|
||||||
withdraw(peerA)},
|
|
||||||
want: []routetable.Origin{{ASN: asB, Peers: []string{peerB},
|
|
||||||
Routes: []routetable.Route{routeOf(peerB, asB, 20)}}},
|
|
||||||
},
|
|
||||||
{
|
|
||||||
name: "a prefix no peer carries is removed",
|
|
||||||
steps: []step{announce(peerA, asA, 10), withdraw(peerA)},
|
|
||||||
want: nil,
|
|
||||||
},
|
|
||||||
{
|
|
||||||
name: "a withdrawal from a peer that does not carry it changes nothing",
|
|
||||||
steps: []step{announce(peerA, asA, 10), withdraw(peerB)},
|
|
||||||
want: []routetable.Origin{{ASN: asA, Peers: []string{peerA},
|
|
||||||
Routes: []routetable.Route{routeOf(peerA, asA, 10)}}},
|
|
||||||
},
|
|
||||||
{
|
|
||||||
name: "a peer going down leaves the prefix",
|
|
||||||
steps: []step{announce(peerA, asA, 10), announce(peerB, asA, 20),
|
|
||||||
peerDown(peerB)},
|
|
||||||
want: []routetable.Origin{{ASN: asA, Peers: []string{peerA},
|
|
||||||
Routes: []routetable.Route{routeOf(peerA, asA, 10)}}},
|
|
||||||
},
|
|
||||||
{
|
|
||||||
name: "a peer going down after it withdrew changes nothing",
|
|
||||||
steps: []step{announce(peerA, asA, 10), announce(peerB, asA, 20),
|
|
||||||
withdraw(peerB), peerDown(peerB)},
|
|
||||||
want: []routetable.Origin{{ASN: asA, Peers: []string{peerA},
|
|
||||||
Routes: []routetable.Route{routeOf(peerA, asA, 10)}}},
|
|
||||||
},
|
|
||||||
})
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestPeerDownLeavesEveryPrefix(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
table := routetable.New(routetable.Params{})
|
|
||||||
prefixes := []netip.Prefix{
|
|
||||||
netip.MustParsePrefix(prefixV4),
|
|
||||||
netip.MustParsePrefix("10.2.0.0/16"),
|
|
||||||
netip.MustParsePrefix(prefixV6),
|
|
||||||
}
|
|
||||||
|
|
||||||
// peerA carries every prefix with origin asA, peerB the first one with
|
|
||||||
// origin asB.
|
|
||||||
for _, prefix := range prefixes {
|
|
||||||
table.Announce(prefix, asA, routeOf(peerA, asA, 10))
|
|
||||||
}
|
|
||||||
|
|
||||||
table.Announce(prefixes[0], asB, routeOf(peerB, asB, 20))
|
|
||||||
table.PeerDown(peerA)
|
|
||||||
|
|
||||||
want := []routetable.Origin{{ASN: asB, Peers: []string{peerB},
|
|
||||||
Routes: []routetable.Route{routeOf(peerB, asB, 20)}}}
|
|
||||||
|
|
||||||
got := table.Origins(prefixes[0])
|
|
||||||
if !reflect.DeepEqual(got, want) {
|
|
||||||
t.Errorf("Origins(%s) = %+v, want %+v", prefixes[0], got, want)
|
|
||||||
}
|
|
||||||
|
|
||||||
for _, prefix := range prefixes[1:] {
|
|
||||||
got := table.Origins(prefix)
|
|
||||||
if got != nil {
|
|
||||||
t.Errorf("Origins(%s) = %+v after the only peer went down, want nil",
|
|
||||||
prefix, got)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
counts := table.Counts()
|
|
||||||
if counts.IPv4Prefixes != 1 || counts.IPv6Prefixes != 0 ||
|
|
||||||
counts.IPv4Routes != 1 || counts.IPv6Routes != 0 {
|
|
||||||
t.Errorf("Counts() = %+v, want one IPv4 prefix and route", counts)
|
|
||||||
}
|
|
||||||
|
|
||||||
if len(table.OriginatedPrefixes(asA)) != 0 {
|
|
||||||
t.Errorf("OriginatedPrefixes(%d) = %+v, want none",
|
|
||||||
asA, table.OriginatedPrefixes(asA))
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestCountsOriginsWithoutKeptRoute(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
prefix := netip.MustParsePrefix(prefixV4)
|
|
||||||
table := routetable.New(routetable.Params{})
|
|
||||||
|
|
||||||
steps := []struct {
|
|
||||||
apply step
|
|
||||||
want int
|
|
||||||
}{
|
|
||||||
{announce(peerA, asA, 10), 0},
|
|
||||||
{announce(peerB, asA, 20), 0},
|
|
||||||
{announce(peerC, asA, 30), 0},
|
|
||||||
{withdraw(peerC), 0},
|
|
||||||
{withdraw(peerB), 1},
|
|
||||||
// An update from the last peer is kept.
|
|
||||||
{announce(peerA, asA, 40), 0},
|
|
||||||
{withdraw(peerA), 0},
|
|
||||||
// Once more without kept routes, then without peers: the origin
|
|
||||||
// is removed.
|
|
||||||
{announce(peerA, asA, 10), 0},
|
|
||||||
{announce(peerB, asA, 20), 0},
|
|
||||||
{announce(peerC, asA, 30), 0},
|
|
||||||
{withdraw(peerB), 0},
|
|
||||||
{withdraw(peerC), 1},
|
|
||||||
{peerDown(peerA), 0},
|
|
||||||
}
|
|
||||||
|
|
||||||
for i, s := range steps {
|
|
||||||
s.apply(table, prefix)
|
|
||||||
|
|
||||||
got := table.Counts().OriginsWithoutKeptRoute
|
|
||||||
if got != s.want {
|
|
||||||
t.Errorf("after step %d, OriginsWithoutKeptRoute = %d, want %d",
|
|
||||||
i, got, s.want)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// change is a routetable.Change with its peer set as a list of peer numbers.
|
|
||||||
type change struct {
|
|
||||||
Prefix string
|
|
||||||
OriginASN int
|
|
||||||
Removed bool
|
|
||||||
Peers []uint
|
|
||||||
Routes []routetable.Route
|
|
||||||
}
|
|
||||||
|
|
||||||
// peerNumbers returns the numbers in set, or nil for a nil set.
|
|
||||||
func peerNumbers(set *bitset.BitSet) []uint {
|
|
||||||
if set == nil {
|
|
||||||
return nil
|
|
||||||
}
|
|
||||||
|
|
||||||
return slices.Collect(set.EachSet())
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestTakeChanges(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
// peerA was numbered 5 before; the next new peer is 6.
|
|
||||||
table := routetable.New(routetable.Params{
|
|
||||||
PeerNumbers: map[string]uint{peerA: 5},
|
|
||||||
})
|
|
||||||
v4 := netip.MustParsePrefix(prefixV4)
|
|
||||||
v6 := netip.MustParsePrefix(prefixV6)
|
|
||||||
|
|
||||||
table.Announce(v6, asA, routeOf(peerA, asA, 10))
|
|
||||||
table.Announce(v4, asA, routeOf(peerA, asA, 10))
|
|
||||||
table.Announce(v4, asA, routeOf(peerB, asA, 20))
|
|
||||||
|
|
||||||
checkChanges(t, table, []change{
|
|
||||||
{Prefix: prefixV4, OriginASN: asA, Peers: []uint{5, 6},
|
|
||||||
Routes: []routetable.Route{
|
|
||||||
routeOf(peerB, asA, 20), routeOf(peerA, asA, 10),
|
|
||||||
}},
|
|
||||||
{Prefix: prefixV6, OriginASN: asA, Peers: []uint{5},
|
|
||||||
Routes: []routetable.Route{routeOf(peerA, asA, 10)}},
|
|
||||||
}, map[string]uint{peerB: 6})
|
|
||||||
|
|
||||||
// TakeChanges cleared the record.
|
|
||||||
checkChanges(t, table, []change{}, map[string]uint{})
|
|
||||||
|
|
||||||
// peerB moves to asB; peerA withdraws, which removes the pair of v4 and
|
|
||||||
// asA.
|
|
||||||
table.Announce(v4, asB, routeOf(peerB, asB, 30))
|
|
||||||
table.Withdraw(v4, peerA)
|
|
||||||
|
|
||||||
checkChanges(t, table, []change{
|
|
||||||
{Prefix: prefixV4, OriginASN: asA, Removed: true},
|
|
||||||
{Prefix: prefixV4, OriginASN: asB, Peers: []uint{6},
|
|
||||||
Routes: []routetable.Route{routeOf(peerB, asB, 30)}},
|
|
||||||
}, map[string]uint{})
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestTakeChangesHoldsOnlyPairsThatChanged(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
table := routetable.New(routetable.Params{})
|
|
||||||
v4 := netip.MustParsePrefix(prefixV4)
|
|
||||||
v6 := netip.MustParsePrefix(prefixV6)
|
|
||||||
|
|
||||||
// peerA, peerB and peerC are numbered 0, 1 and 2. The routes of peerC and
|
|
||||||
// peerB to v4 are kept; peerA keeps only its bit.
|
|
||||||
announceAll(table, prefixV4, asA, 10, peerA, peerB, peerC)
|
|
||||||
table.Announce(v6, asA, routeOf(peerB, asA, 10))
|
|
||||||
table.TakeChanges(math.MaxInt)
|
|
||||||
|
|
||||||
// peerA announces again, with a time older than both kept routes.
|
|
||||||
table.Announce(v4, asA, routeOf(peerA, asA, 15))
|
|
||||||
checkChanges(t, table, []change{}, map[string]uint{})
|
|
||||||
|
|
||||||
// peerC announces its kept route again.
|
|
||||||
table.Announce(v4, asA, routeOf(peerC, asA, 30))
|
|
||||||
checkChanges(t, table, []change{}, map[string]uint{})
|
|
||||||
|
|
||||||
// peerC announces the same path with a newer time, so its kept route
|
|
||||||
// changes.
|
|
||||||
table.Announce(v4, asA, routeOf(peerC, asA, 40))
|
|
||||||
checkChanges(t, table, []change{
|
|
||||||
{Prefix: prefixV4, OriginASN: asA, Peers: []uint{0, 1, 2},
|
|
||||||
Routes: []routetable.Route{
|
|
||||||
routeOf(peerC, asA, 40), routeOf(peerB, asA, 20),
|
|
||||||
}},
|
|
||||||
}, map[string]uint{})
|
|
||||||
|
|
||||||
// peerA withdraws v6, which it does not carry.
|
|
||||||
table.Withdraw(v6, peerA)
|
|
||||||
checkChanges(t, table, []change{}, map[string]uint{})
|
|
||||||
|
|
||||||
// peerA going down changes v4, the one prefix it carries.
|
|
||||||
table.PeerDown(peerA)
|
|
||||||
checkChanges(t, table, []change{
|
|
||||||
{Prefix: prefixV4, OriginASN: asA, Peers: []uint{1, 2},
|
|
||||||
Routes: []routetable.Route{
|
|
||||||
routeOf(peerC, asA, 40), routeOf(peerB, asA, 20),
|
|
||||||
}},
|
|
||||||
}, map[string]uint{})
|
|
||||||
|
|
||||||
// peerA, which carries nothing now, goes down again.
|
|
||||||
table.PeerDown(peerA)
|
|
||||||
checkChanges(t, table, []change{}, map[string]uint{})
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestTakeChangesTakesAtMostMaxChanges changes three prefix and origin AS
|
|
||||||
// pairs, then takes the changes two at a time. The first call must return two
|
|
||||||
// pairs and both new peers, and the second the third pair and no peer.
|
|
||||||
func TestTakeChangesTakesAtMostMaxChanges(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
const maxChanges = 2
|
|
||||||
|
|
||||||
table := routetable.New(routetable.Params{})
|
|
||||||
v4 := netip.MustParsePrefix(prefixV4)
|
|
||||||
v6 := netip.MustParsePrefix(prefixV6)
|
|
||||||
|
|
||||||
table.Announce(v4, asA, routeOf(peerA, asA, 10))
|
|
||||||
table.Announce(v4, asB, routeOf(peerB, asB, 10))
|
|
||||||
table.Announce(v6, asA, routeOf(peerA, asA, 10))
|
|
||||||
|
|
||||||
first, firstPeers := table.TakeChanges(maxChanges)
|
|
||||||
second, secondPeers := table.TakeChanges(maxChanges)
|
|
||||||
|
|
||||||
if len(first) != maxChanges || len(second) != 1 {
|
|
||||||
t.Fatalf("TakeChanges(%d) returned %d and then %d pairs, want %d and 1",
|
|
||||||
maxChanges, len(first), len(second), maxChanges)
|
|
||||||
}
|
|
||||||
|
|
||||||
wantPeers := map[string]uint{peerA: 0, peerB: 1}
|
|
||||||
if !reflect.DeepEqual(firstPeers, wantPeers) || len(secondPeers) != 0 {
|
|
||||||
t.Errorf("TakeChanges(%d) returned the new peers %v and then %v, "+
|
|
||||||
"want %v and none", maxChanges, firstPeers, secondPeers, wantPeers)
|
|
||||||
}
|
|
||||||
|
|
||||||
for _, c := range first {
|
|
||||||
if c.Prefix == second[0].Prefix && c.OriginASN == second[0].OriginASN {
|
|
||||||
t.Errorf("both calls returned %s with AS%d", c.Prefix, c.OriginASN)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestPutBackChanges takes a table's changes and puts them back, as a caller
|
|
||||||
// does whose write failed. The next TakeChanges must return them again, with
|
|
||||||
// each pair's content then, and with the changes since.
|
|
||||||
func TestPutBackChanges(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
table := routetable.New(routetable.Params{})
|
|
||||||
v4 := netip.MustParsePrefix(prefixV4)
|
|
||||||
v6 := netip.MustParsePrefix(prefixV6)
|
|
||||||
|
|
||||||
table.Announce(v4, asA, routeOf(peerA, asA, 10))
|
|
||||||
table.Announce(v6, asA, routeOf(peerA, asA, 10))
|
|
||||||
table.PutBackChanges(table.TakeChanges(math.MaxInt))
|
|
||||||
|
|
||||||
if got := table.ChangeCount(); got != 2 {
|
|
||||||
t.Errorf("ChangeCount() = %d after the changes were put back, want 2",
|
|
||||||
got)
|
|
||||||
}
|
|
||||||
|
|
||||||
// peerA withdraws v4, so v4's earlier content must not come back; peerB
|
|
||||||
// announces v6, and is a new peer as well.
|
|
||||||
table.Withdraw(v4, peerA)
|
|
||||||
table.Announce(v6, asA, routeOf(peerB, asA, 20))
|
|
||||||
|
|
||||||
checkChanges(t, table, []change{
|
|
||||||
{Prefix: prefixV4, OriginASN: asA, Removed: true},
|
|
||||||
{Prefix: prefixV6, OriginASN: asA, Peers: []uint{0, 1},
|
|
||||||
Routes: []routetable.Route{
|
|
||||||
routeOf(peerB, asA, 20), routeOf(peerA, asA, 10),
|
|
||||||
}},
|
|
||||||
}, map[string]uint{peerA: 0, peerB: 1})
|
|
||||||
|
|
||||||
if got := table.ChangeCount(); got != 0 {
|
|
||||||
t.Errorf("ChangeCount() = %d after TakeChanges, want 0", got)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestLoad(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
const otherV4 = "10.9.0.0/16"
|
|
||||||
|
|
||||||
v4 := netip.MustParsePrefix(prefixV4)
|
|
||||||
v6 := netip.MustParsePrefix(prefixV6)
|
|
||||||
other := netip.MustParsePrefix(otherV4)
|
|
||||||
|
|
||||||
// The routes of peerC and peerB to v4 are kept; peerA keeps only its bit.
|
|
||||||
// The two routes to v6 have the same time. The routes of other were
|
|
||||||
// withdrawn while peerA still carries it.
|
|
||||||
source := routetable.New(routetable.Params{})
|
|
||||||
announceAll(source, prefixV4, asA, 10, peerA, peerB, peerC)
|
|
||||||
announceAll(source, prefixV4, asB, 40, peerD)
|
|
||||||
source.Announce(v6, asA, routeOf(peerA, asA, 50))
|
|
||||||
source.Announce(v6, asA, routeOf(peerC, asA, 50))
|
|
||||||
announceAll(source, otherV4, asB, 10, peerA, peerB, peerC)
|
|
||||||
source.Withdraw(other, peerB)
|
|
||||||
source.Withdraw(other, peerC)
|
|
||||||
|
|
||||||
changes, peers := source.TakeChanges(math.MaxInt)
|
|
||||||
loaded := routetable.New(routetable.Params{PeerNumbers: peers})
|
|
||||||
|
|
||||||
// Loading a pair twice changes nothing.
|
|
||||||
for range 2 {
|
|
||||||
for _, change := range changes {
|
|
||||||
loaded.Load(change)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
for _, prefix := range []netip.Prefix{v4, v6, other} {
|
|
||||||
got, want := loaded.Origins(prefix), source.Origins(prefix)
|
|
||||||
if !reflect.DeepEqual(got, want) {
|
|
||||||
t.Errorf("Origins(%s) = %+v after the load, want %+v", prefix, got, want)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
if !reflect.DeepEqual(loaded.Counts(), source.Counts()) {
|
|
||||||
t.Errorf("Counts() = %+v after the load, want %+v",
|
|
||||||
loaded.Counts(), source.Counts())
|
|
||||||
}
|
|
||||||
|
|
||||||
checkChanges(t, loaded, []change{}, map[string]uint{})
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestConcurrentUse has goroutines update one table while others read it. The
|
|
||||||
// race detector, which make test runs, fails it when a method uses the table
|
|
||||||
// without its lock.
|
|
||||||
func TestConcurrentUse(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
const (
|
|
||||||
// rounds is how many times each goroutine calls its method.
|
|
||||||
rounds = 1000
|
|
||||||
ipVersion = 4
|
|
||||||
sampleSize = 10
|
|
||||||
)
|
|
||||||
|
|
||||||
table := routetable.New(routetable.Params{})
|
|
||||||
prefix := netip.MustParsePrefix(prefixV4)
|
|
||||||
|
|
||||||
uses := []func(round int){
|
|
||||||
func(round int) { table.Announce(prefix, asA, routeOf(peerA, asA, round)) },
|
|
||||||
func(round int) { table.Announce(prefix, asB, routeOf(peerB, asB, round)) },
|
|
||||||
func(int) { table.Withdraw(prefix, peerA) },
|
|
||||||
func(int) { table.PeerDown(peerB) },
|
|
||||||
func(int) { table.PutBackChanges(table.TakeChanges(math.MaxInt)) },
|
|
||||||
func(int) { table.ChangeCount() },
|
|
||||||
func(int) { table.Lookup(prefix.Addr()) },
|
|
||||||
func(int) { table.Origins(prefix) },
|
|
||||||
func(int) { table.OriginatedPrefixes(asA) },
|
|
||||||
func(int) { table.SamplePrefixes(ipVersion, prefix.Bits(), sampleSize) },
|
|
||||||
func(int) { table.Counts() },
|
|
||||||
}
|
|
||||||
|
|
||||||
var wg sync.WaitGroup
|
|
||||||
for _, use := range uses {
|
|
||||||
wg.Go(func() {
|
|
||||||
for round := range rounds {
|
|
||||||
use(round)
|
|
||||||
}
|
|
||||||
})
|
|
||||||
}
|
|
||||||
|
|
||||||
wg.Wait()
|
|
||||||
}
|
|
||||||
|
|
||||||
// checkChanges checks that TakeChanges, taking every change, returns want and
|
|
||||||
// wantPeers.
|
|
||||||
func checkChanges(
|
|
||||||
t *testing.T, table *routetable.Table, want []change,
|
|
||||||
wantPeers map[string]uint,
|
|
||||||
) {
|
|
||||||
t.Helper()
|
|
||||||
|
|
||||||
changes, newPeers := table.TakeChanges(math.MaxInt)
|
|
||||||
|
|
||||||
got := make([]change, 0, len(changes))
|
|
||||||
for _, c := range changes {
|
|
||||||
got = append(got, change{
|
|
||||||
Prefix: c.Prefix.String(), OriginASN: c.OriginASN, Removed: c.Removed,
|
|
||||||
Peers: peerNumbers(c.Peers), Routes: c.Routes,
|
|
||||||
})
|
|
||||||
}
|
|
||||||
|
|
||||||
if !reflect.DeepEqual(got, want) {
|
|
||||||
t.Errorf("TakeChanges() changes = %+v, want %+v", got, want)
|
|
||||||
}
|
|
||||||
|
|
||||||
if !reflect.DeepEqual(newPeers, wantPeers) {
|
|
||||||
t.Errorf("TakeChanges() new peers = %v, want %v", newPeers, wantPeers)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
+11
-67
@@ -2,25 +2,19 @@
|
|||||||
// running instance of the application and contains pointers to its core
|
// running instance of the application and contains pointers to its core
|
||||||
// dependencies, and is responsible for initialization. It also contains the
|
// dependencies, and is responsible for initialization. It also contains the
|
||||||
// ASN WHOIS fetcher for background updates and the database maintainer for
|
// ASN WHOIS fetcher for background updates and the database maintainer for
|
||||||
// background maintenance tasks. The package also holds the handlers that
|
// background maintenance tasks.
|
||||||
// write the feed's updates to the database in batches, and the flush loop
|
|
||||||
// that the AS, peer and prefix handlers share.
|
|
||||||
package routewatch
|
package routewatch
|
||||||
|
|
||||||
import (
|
import (
|
||||||
"context"
|
"context"
|
||||||
"errors"
|
"errors"
|
||||||
"fmt"
|
|
||||||
"path/filepath"
|
|
||||||
"sync"
|
"sync"
|
||||||
"time"
|
"time"
|
||||||
|
|
||||||
"git.eeqj.de/sneak/routewatch/internal/blocklist"
|
|
||||||
"git.eeqj.de/sneak/routewatch/internal/config"
|
"git.eeqj.de/sneak/routewatch/internal/config"
|
||||||
"git.eeqj.de/sneak/routewatch/internal/database"
|
"git.eeqj.de/sneak/routewatch/internal/database"
|
||||||
"git.eeqj.de/sneak/routewatch/internal/logger"
|
"git.eeqj.de/sneak/routewatch/internal/logger"
|
||||||
"git.eeqj.de/sneak/routewatch/internal/metrics"
|
"git.eeqj.de/sneak/routewatch/internal/metrics"
|
||||||
"git.eeqj.de/sneak/routewatch/internal/routetable"
|
|
||||||
"git.eeqj.de/sneak/routewatch/internal/server"
|
"git.eeqj.de/sneak/routewatch/internal/server"
|
||||||
"git.eeqj.de/sneak/routewatch/internal/streamer"
|
"git.eeqj.de/sneak/routewatch/internal/streamer"
|
||||||
|
|
||||||
@@ -31,13 +25,11 @@ import (
|
|||||||
// are disabled, since only the batched handlers exist.
|
// are disabled, since only the batched handlers exist.
|
||||||
var errNonBatchedNotImplemented = errors.New("non-batched handlers not implemented")
|
var errNonBatchedNotImplemented = errors.New("non-batched handlers not implemented")
|
||||||
|
|
||||||
// Dependencies contains all dependencies for RouteWatch. Table is the routing
|
// Dependencies contains all dependencies for RouteWatch
|
||||||
// table read from the database at startup, which the server reads too.
|
|
||||||
type Dependencies struct {
|
type Dependencies struct {
|
||||||
fx.In
|
fx.In
|
||||||
|
|
||||||
DB database.Store
|
DB database.Store
|
||||||
Table *routetable.Table
|
|
||||||
Streamer *streamer.Streamer
|
Streamer *streamer.Streamer
|
||||||
Server *server.Server
|
Server *server.Server
|
||||||
Logger *logger.Logger
|
Logger *logger.Logger
|
||||||
@@ -47,7 +39,6 @@ type Dependencies struct {
|
|||||||
// RouteWatch represents the main application instance
|
// RouteWatch represents the main application instance
|
||||||
type RouteWatch struct {
|
type RouteWatch struct {
|
||||||
db database.Store
|
db database.Store
|
||||||
table *routetable.Table
|
|
||||||
streamer *streamer.Streamer
|
streamer *streamer.Streamer
|
||||||
server *server.Server
|
server *server.Server
|
||||||
logger *logger.Logger
|
logger *logger.Logger
|
||||||
@@ -61,18 +52,12 @@ type RouteWatch struct {
|
|||||||
peeringHandler *PeeringHandler
|
peeringHandler *PeeringHandler
|
||||||
asnFetcher *ASNFetcher
|
asnFetcher *ASNFetcher
|
||||||
dbMaintainer *DBMaintainer
|
dbMaintainer *DBMaintainer
|
||||||
blocklists *blocklist.Blocklists
|
|
||||||
}
|
}
|
||||||
|
|
||||||
// blocklistsDirName is the directory in the state directory that holds the
|
|
||||||
// blocklist copies.
|
|
||||||
const blocklistsDirName = "blocklists"
|
|
||||||
|
|
||||||
// New creates a new RouteWatch instance
|
// New creates a new RouteWatch instance
|
||||||
func New(deps Dependencies) *RouteWatch {
|
func New(deps Dependencies) *RouteWatch {
|
||||||
rw := &RouteWatch{
|
rw := &RouteWatch{
|
||||||
db: deps.DB,
|
db: deps.DB,
|
||||||
table: deps.Table,
|
|
||||||
streamer: deps.Streamer,
|
streamer: deps.Streamer,
|
||||||
server: deps.Server,
|
server: deps.Server,
|
||||||
logger: deps.Logger,
|
logger: deps.Logger,
|
||||||
@@ -100,7 +85,6 @@ func (rw *RouteWatch) Run(ctx context.Context) error {
|
|||||||
// Register database handler to process BGP UPDATE messages
|
// Register database handler to process BGP UPDATE messages
|
||||||
if rw.config.EnableBatchedDatabaseWrites {
|
if rw.config.EnableBatchedDatabaseWrites {
|
||||||
rw.logger.Info("Using batched database handlers for improved performance")
|
rw.logger.Info("Using batched database handlers for improved performance")
|
||||||
|
|
||||||
// ASHandler maintains the asns table
|
// ASHandler maintains the asns table
|
||||||
rw.dbHandler = NewASHandler(rw.db, rw.logger)
|
rw.dbHandler = NewASHandler(rw.db, rw.logger)
|
||||||
rw.streamer.RegisterHandler(rw.dbHandler)
|
rw.streamer.RegisterHandler(rw.dbHandler)
|
||||||
@@ -109,10 +93,8 @@ func (rw *RouteWatch) Run(ctx context.Context) error {
|
|||||||
rw.peerHandler = NewPeerHandler(rw.db, rw.logger)
|
rw.peerHandler = NewPeerHandler(rw.db, rw.logger)
|
||||||
rw.streamer.RegisterHandler(rw.peerHandler)
|
rw.streamer.RegisterHandler(rw.peerHandler)
|
||||||
|
|
||||||
// PrefixHandler keeps the routing table in memory current, and writes
|
// PrefixHandler maintains the prefixes and live_routes tables
|
||||||
// its changes to the routes table and new prefixes to the prefix
|
rw.prefixHandler = NewPrefixHandler(rw.db, rw.logger)
|
||||||
// tables
|
|
||||||
rw.prefixHandler = NewPrefixHandler(rw.db, rw.logger, rw.table)
|
|
||||||
rw.prefixHandler.SetMetricsTracker(rw.streamer.GetMetricsTracker())
|
rw.prefixHandler.SetMetricsTracker(rw.streamer.GetMetricsTracker())
|
||||||
rw.streamer.RegisterHandler(rw.prefixHandler)
|
rw.streamer.RegisterHandler(rw.prefixHandler)
|
||||||
|
|
||||||
@@ -126,6 +108,8 @@ func (rw *RouteWatch) Run(ctx context.Context) error {
|
|||||||
return errNonBatchedNotImplemented
|
return errNonBatchedNotImplemented
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// No longer need routing table handler - PrefixHandler maintains live_routes table
|
||||||
|
|
||||||
// Start streaming
|
// Start streaming
|
||||||
err := rw.streamer.Start(ctx)
|
err := rw.streamer.Start(ctx)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
@@ -138,18 +122,6 @@ func (rw *RouteWatch) Run(ctx context.Context) error {
|
|||||||
rw.asnFetcher.Start(ctx)
|
rw.asnFetcher.Start(ctx)
|
||||||
rw.server.SetASNFetcher(rw.asnFetcher)
|
rw.server.SetASNFetcher(rw.asnFetcher)
|
||||||
|
|
||||||
// Load the blocklist copies from disk and start downloading the lists.
|
|
||||||
// The HTTP handlers read the server's blocklists without a lock, so they
|
|
||||||
// are set before the handlers run.
|
|
||||||
blocklists := blocklist.New(blocklist.Params{
|
|
||||||
Dir: filepath.Join(rw.config.StateDir, blocklistsDirName),
|
|
||||||
Lists: blocklist.Lists(),
|
|
||||||
Logger: rw.logger.Logger,
|
|
||||||
})
|
|
||||||
blocklists.Start(ctx)
|
|
||||||
rw.blocklists = blocklists
|
|
||||||
rw.server.SetBlocklists(blocklists)
|
|
||||||
|
|
||||||
// Start HTTP server
|
// Start HTTP server
|
||||||
err = rw.server.Start(ctx)
|
err = rw.server.Start(ctx)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
@@ -167,12 +139,11 @@ func (rw *RouteWatch) Run(ctx context.Context) error {
|
|||||||
}
|
}
|
||||||
|
|
||||||
// Shutdown stops all services in order: the streamer, the handlers, which
|
// Shutdown stops all services in order: the streamer, the handlers, which
|
||||||
// write what they hold, the WHOIS fetcher, the database maintainer, the
|
// write what they hold, the WHOIS fetcher, the database maintainer and the
|
||||||
// blocklist downloads and the HTTP server, and last the database, which it
|
// HTTP server, and last the database, which it checkpoints and closes. Each
|
||||||
// checkpoints and closes. Each stage waits at most until ctx ends, so
|
// stage waits at most until ctx ends, so Shutdown returns soon after: a
|
||||||
// Shutdown returns soon after: a handler logs what it could not write by
|
// handler logs what it could not write by then, and the HTTP server closes
|
||||||
// then, and the HTTP server closes the connections of requests still
|
// the connections of requests still running.
|
||||||
// running.
|
|
||||||
func (rw *RouteWatch) Shutdown(ctx context.Context) {
|
func (rw *RouteWatch) Shutdown(ctx context.Context) {
|
||||||
rw.mu.Lock()
|
rw.mu.Lock()
|
||||||
if rw.shutdown {
|
if rw.shutdown {
|
||||||
@@ -219,10 +190,6 @@ func (rw *RouteWatch) Shutdown(ctx context.Context) {
|
|||||||
rw.dbMaintainer.Stop(ctx)
|
rw.dbMaintainer.Stop(ctx)
|
||||||
}
|
}
|
||||||
|
|
||||||
if rw.blocklists != nil {
|
|
||||||
rw.blocklists.Stop(ctx)
|
|
||||||
}
|
|
||||||
|
|
||||||
err := rw.server.Stop(ctx)
|
err := rw.server.Stop(ctx)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
rw.logger.Error("Stopped the HTTP server before every request finished",
|
rw.logger.Error("Stopped the HTTP server before every request finished",
|
||||||
@@ -263,28 +230,6 @@ func (rw *RouteWatch) closeDatabase(ctx context.Context) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// readRouteTable reads the routing table from the database and logs how many
|
|
||||||
// prefix and origin AS pairs and peers it holds and how long the read took.
|
|
||||||
// fx calls it once, to build the server and RouteWatch, before the streamer
|
|
||||||
// connects, so the prefix handler applies the feed's updates to all of it. No
|
|
||||||
// start timeout limits the read: fx builds them before it runs a start hook.
|
|
||||||
func readRouteTable(
|
|
||||||
db database.Store, logger *logger.Logger,
|
|
||||||
) (*routetable.Table, error) {
|
|
||||||
start := time.Now()
|
|
||||||
|
|
||||||
table, err := db.ReadRouteTable(context.Background())
|
|
||||||
if err != nil {
|
|
||||||
return nil, fmt.Errorf("failed to read the routing table: %w", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
counts := table.Counts()
|
|
||||||
logger.Info("Read the routing table", "pairs", counts.Pairs,
|
|
||||||
"peers", counts.Peers, "duration_ms", time.Since(start).Milliseconds())
|
|
||||||
|
|
||||||
return table, nil
|
|
||||||
}
|
|
||||||
|
|
||||||
// getModule provides all fx dependencies
|
// getModule provides all fx dependencies
|
||||||
func getModule() []fx.Option {
|
func getModule() []fx.Option {
|
||||||
return []fx.Option{
|
return []fx.Option{
|
||||||
@@ -296,7 +241,6 @@ func getModule() []fx.Option {
|
|||||||
database.New,
|
database.New,
|
||||||
fx.As(new(database.Store)),
|
fx.As(new(database.Store)),
|
||||||
),
|
),
|
||||||
readRouteTable,
|
|
||||||
streamer.New,
|
streamer.New,
|
||||||
server.New,
|
server.New,
|
||||||
New,
|
New,
|
||||||
|
|||||||
@@ -7,7 +7,6 @@ import (
|
|||||||
"net"
|
"net"
|
||||||
"net/http"
|
"net/http"
|
||||||
"net/http/httptest"
|
"net/http/httptest"
|
||||||
"net/netip"
|
|
||||||
"slices"
|
"slices"
|
||||||
"strconv"
|
"strconv"
|
||||||
"strings"
|
"strings"
|
||||||
@@ -20,7 +19,6 @@ import (
|
|||||||
"git.eeqj.de/sneak/routewatch/internal/logger"
|
"git.eeqj.de/sneak/routewatch/internal/logger"
|
||||||
"git.eeqj.de/sneak/routewatch/internal/metrics"
|
"git.eeqj.de/sneak/routewatch/internal/metrics"
|
||||||
"git.eeqj.de/sneak/routewatch/internal/ristypes"
|
"git.eeqj.de/sneak/routewatch/internal/ristypes"
|
||||||
"git.eeqj.de/sneak/routewatch/internal/routetable"
|
|
||||||
"git.eeqj.de/sneak/routewatch/internal/server"
|
"git.eeqj.de/sneak/routewatch/internal/server"
|
||||||
"git.eeqj.de/sneak/routewatch/internal/streamer"
|
"git.eeqj.de/sneak/routewatch/internal/streamer"
|
||||||
)
|
)
|
||||||
@@ -33,15 +31,14 @@ type mockStore struct {
|
|||||||
ASNCount int
|
ASNCount int
|
||||||
PrefixCount int
|
PrefixCount int
|
||||||
PeeringCount int
|
PeeringCount int
|
||||||
|
RouteCount int
|
||||||
WithdrawalCount int
|
WithdrawalCount int
|
||||||
|
|
||||||
// Track unique items
|
// Track unique items
|
||||||
ASNs map[int]*database.ASN
|
ASNs map[int]*database.ASN
|
||||||
Prefixes map[string]*database.Prefix
|
Prefixes map[string]*database.Prefix
|
||||||
Peerings map[string]bool // key is "from_to"
|
Peerings map[string]bool // key is "from_to"
|
||||||
// TablePrefixes holds the prefixes that the written changes of the
|
Routes map[string]bool // key is "prefix_origin_peer"
|
||||||
// routing table leave some peer carrying.
|
|
||||||
TablePrefixes map[string]bool
|
|
||||||
|
|
||||||
// Track IP versions
|
// Track IP versions
|
||||||
IPv4Prefixes int
|
IPv4Prefixes int
|
||||||
@@ -51,10 +48,10 @@ type mockStore struct {
|
|||||||
// newMockStore creates a new mock store
|
// newMockStore creates a new mock store
|
||||||
func newMockStore() *mockStore {
|
func newMockStore() *mockStore {
|
||||||
return &mockStore{
|
return &mockStore{
|
||||||
ASNs: make(map[int]*database.ASN),
|
ASNs: make(map[int]*database.ASN),
|
||||||
Prefixes: make(map[string]*database.Prefix),
|
Prefixes: make(map[string]*database.Prefix),
|
||||||
Peerings: make(map[string]bool),
|
Peerings: make(map[string]bool),
|
||||||
TablePrefixes: make(map[string]bool),
|
Routes: make(map[string]bool),
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -104,13 +101,6 @@ func (m *mockStore) UpdatePrefixesBatch(prefixes map[string]time.Time) error {
|
|||||||
IPVersion: ipVersion,
|
IPVersion: ipVersion,
|
||||||
FirstSeen: timestamp,
|
FirstSeen: timestamp,
|
||||||
}
|
}
|
||||||
m.PrefixCount++
|
|
||||||
|
|
||||||
if ipVersion == ipVersionV4 {
|
|
||||||
m.IPv4Prefixes++
|
|
||||||
} else {
|
|
||||||
m.IPv6Prefixes++
|
|
||||||
}
|
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -211,30 +201,92 @@ func (m *mockStore) GetStatsContext(_ context.Context) (database.Stats, error) {
|
|||||||
return m.GetStats()
|
return m.GetStats()
|
||||||
}
|
}
|
||||||
|
|
||||||
// GetAS returns the stored AS, or only its number when none is stored.
|
// UpsertLiveRoute mock implementation
|
||||||
func (m *mockStore) GetAS(_ context.Context, asn int) (*database.ASN, bool, error) {
|
func (m *mockStore) UpsertLiveRoute(_ *database.LiveRoute) error {
|
||||||
|
// Simple mock - just return nil
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// DeleteLiveRoute mock implementation
|
||||||
|
func (m *mockStore) DeleteLiveRoute(_, _ string) error {
|
||||||
|
// Simple mock - just return nil
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// GetPrefixDistribution mock implementation
|
||||||
|
func (m *mockStore) GetPrefixDistribution() (
|
||||||
|
[]database.PrefixDistribution, []database.PrefixDistribution, error,
|
||||||
|
) {
|
||||||
|
// Return empty distributions for now
|
||||||
|
return nil, nil, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// GetPrefixDistributionContext mock implementation with context support
|
||||||
|
func (m *mockStore) GetPrefixDistributionContext(_ context.Context) (
|
||||||
|
[]database.PrefixDistribution, []database.PrefixDistribution, error,
|
||||||
|
) {
|
||||||
|
return m.GetPrefixDistribution()
|
||||||
|
}
|
||||||
|
|
||||||
|
// GetLiveRouteCounts mock implementation
|
||||||
|
func (m *mockStore) GetLiveRouteCounts() (int, int, error) {
|
||||||
|
// Return mock counts
|
||||||
|
return m.RouteCount / 2, m.RouteCount / 2, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// GetLiveRouteCountsContext mock implementation with context support
|
||||||
|
func (m *mockStore) GetLiveRouteCountsContext(_ context.Context) (int, int, error) {
|
||||||
|
return m.GetLiveRouteCounts()
|
||||||
|
}
|
||||||
|
|
||||||
|
// GetASDetails mock implementation
|
||||||
|
func (m *mockStore) GetASDetails(asn int) (*database.ASN, []database.LiveRoute, error) {
|
||||||
m.mu.Lock()
|
m.mu.Lock()
|
||||||
defer m.mu.Unlock()
|
defer m.mu.Unlock()
|
||||||
|
|
||||||
|
// Check if ASN exists
|
||||||
if asnInfo, exists := m.ASNs[asn]; exists {
|
if asnInfo, exists := m.ASNs[asn]; exists {
|
||||||
return asnInfo, true, nil
|
// Return empty prefixes for now
|
||||||
|
return asnInfo, []database.LiveRoute{}, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
return &database.ASN{ASN: asn}, false, nil
|
return nil, nil, database.ErrNoRoute
|
||||||
}
|
}
|
||||||
|
|
||||||
// GetASWHOIS returns the stored AS, or only its number when none is stored.
|
// GetASDetailsContext mock implementation with context support
|
||||||
func (m *mockStore) GetASWHOIS(ctx context.Context, asn int) (*database.ASN, error) {
|
func (m *mockStore) GetASDetailsContext(
|
||||||
asnInfo, _, err := m.GetAS(ctx, asn)
|
_ context.Context, asn int,
|
||||||
|
) (*database.ASN, []database.LiveRoute, error) {
|
||||||
return asnInfo, err
|
return m.GetASDetails(asn)
|
||||||
}
|
}
|
||||||
|
|
||||||
// GetPrefixFirstSeen returns the zero time: the mock keeps no prefix times.
|
// GetPrefixDetails mock implementation
|
||||||
func (m *mockStore) GetPrefixFirstSeen(
|
func (m *mockStore) GetPrefixDetails(
|
||||||
_ context.Context, _ netip.Prefix,
|
_ string,
|
||||||
) (time.Time, error) {
|
) ([]database.LiveRoute, map[int]database.ASN, error) {
|
||||||
return time.Time{}, nil
|
// Return empty routes for now
|
||||||
|
return []database.LiveRoute{}, map[int]database.ASN{}, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// GetPrefixDetailsContext mock implementation with context support
|
||||||
|
func (m *mockStore) GetPrefixDetailsContext(
|
||||||
|
_ context.Context, prefix string,
|
||||||
|
) ([]database.LiveRoute, map[int]database.ASN, error) {
|
||||||
|
return m.GetPrefixDetails(prefix)
|
||||||
|
}
|
||||||
|
|
||||||
|
func (m *mockStore) GetRandomPrefixesByLength(
|
||||||
|
_, _, _ int,
|
||||||
|
) ([]database.LiveRoute, error) {
|
||||||
|
// Return empty routes for now
|
||||||
|
return []database.LiveRoute{}, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// GetRandomPrefixesByLengthContext mock implementation with context support
|
||||||
|
func (m *mockStore) GetRandomPrefixesByLengthContext(
|
||||||
|
_ context.Context, maskLength, ipVersion, limit int,
|
||||||
|
) ([]database.LiveRoute, error) {
|
||||||
|
return m.GetRandomPrefixesByLength(maskLength, ipVersion, limit)
|
||||||
}
|
}
|
||||||
|
|
||||||
// GetASPeers mock implementation
|
// GetASPeers mock implementation
|
||||||
@@ -250,6 +302,32 @@ func (m *mockStore) GetASPeersContext(
|
|||||||
return m.GetASPeers(asn)
|
return m.GetASPeers(asn)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// GetIPInfo mock implementation
|
||||||
|
func (m *mockStore) GetIPInfo(ip string) (*database.IPInfo, error) {
|
||||||
|
return m.GetIPInfoContext(context.Background(), ip)
|
||||||
|
}
|
||||||
|
|
||||||
|
// GetIPInfoContext mock implementation with context support
|
||||||
|
func (m *mockStore) GetIPInfoContext(
|
||||||
|
_ context.Context, ip string,
|
||||||
|
) (*database.IPInfo, error) {
|
||||||
|
now := time.Now()
|
||||||
|
|
||||||
|
return &database.IPInfo{
|
||||||
|
IP: ip,
|
||||||
|
Netblock: "8.8.8.0/24",
|
||||||
|
MaskLength: 24,
|
||||||
|
IPVersion: 4,
|
||||||
|
NumPeers: 3,
|
||||||
|
ASN: 15169,
|
||||||
|
Handle: "GOOGLE",
|
||||||
|
Description: "Google LLC",
|
||||||
|
CountryCode: "US",
|
||||||
|
FirstSeen: now.Add(-24 * time.Hour),
|
||||||
|
LastSeen: now,
|
||||||
|
}, nil
|
||||||
|
}
|
||||||
|
|
||||||
// GetNextStaleASN mock implementation
|
// GetNextStaleASN mock implementation
|
||||||
func (m *mockStore) GetNextStaleASN(_ context.Context, _ time.Duration) (int, error) {
|
func (m *mockStore) GetNextStaleASN(_ context.Context, _ time.Duration) (int, error) {
|
||||||
return 0, database.ErrNoStaleASN
|
return 0, database.ErrNoStaleASN
|
||||||
@@ -277,28 +355,44 @@ func (m *mockStore) GetWHOISStats(
|
|||||||
}, nil
|
}, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
// WriteRouteChanges records the prefixes the changes leave some peer carrying,
|
// UpsertLiveRouteBatch mock implementation
|
||||||
// whatever their origin AS.
|
func (m *mockStore) UpsertLiveRouteBatch(routes []*database.LiveRoute) error {
|
||||||
func (m *mockStore) WriteRouteChanges(
|
|
||||||
changes []routetable.Change, _ map[string]uint,
|
|
||||||
) error {
|
|
||||||
m.mu.Lock()
|
m.mu.Lock()
|
||||||
defer m.mu.Unlock()
|
defer m.mu.Unlock()
|
||||||
|
|
||||||
for _, change := range changes {
|
for _, route := range routes {
|
||||||
if change.Removed {
|
// Track prefix
|
||||||
delete(m.TablePrefixes, change.Prefix.String())
|
if _, exists := m.Prefixes[route.Prefix]; !exists {
|
||||||
} else {
|
m.Prefixes[route.Prefix] = &database.Prefix{
|
||||||
m.TablePrefixes[change.Prefix.String()] = true
|
Prefix: route.Prefix,
|
||||||
|
IPVersion: route.IPVersion,
|
||||||
|
FirstSeen: route.LastUpdated,
|
||||||
|
}
|
||||||
|
m.PrefixCount++
|
||||||
|
|
||||||
|
if route.IPVersion == 4 {
|
||||||
|
m.IPv4Prefixes++
|
||||||
|
} else {
|
||||||
|
m.IPv6Prefixes++
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
m.RouteCount++
|
||||||
}
|
}
|
||||||
|
|
||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
|
|
||||||
// ReadRouteTable returns an empty routing table.
|
// DeleteLiveRouteBatch mock implementation
|
||||||
func (m *mockStore) ReadRouteTable(_ context.Context) (*routetable.Table, error) {
|
func (m *mockStore) DeleteLiveRouteBatch(_ []database.LiveRouteDeletion) error {
|
||||||
return routetable.New(routetable.Params{}), nil
|
// Simple mock - just return nil
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// DeleteLiveRoutesOfPeer mock implementation
|
||||||
|
func (m *mockStore) DeleteLiveRoutesOfPeer(_ string) (int, error) {
|
||||||
|
// Simple mock - deletes nothing
|
||||||
|
return 0, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
// GetOrCreateASNBatch mock implementation
|
// GetOrCreateASNBatch mock implementation
|
||||||
@@ -372,22 +466,19 @@ func TestRouteWatchLiveFeed(t *testing.T) {
|
|||||||
// Create streamer
|
// Create streamer
|
||||||
s := streamer.New(logger, metricsTracker)
|
s := streamer.New(logger, metricsTracker)
|
||||||
|
|
||||||
// The state directory holds the blocklist copies Run downloads.
|
// Create test config with empty state dir (no snapshot loading)
|
||||||
cfg := &config.Config{
|
cfg := &config.Config{
|
||||||
StateDir: t.TempDir(),
|
StateDir: "",
|
||||||
MaxRuntime: 5 * time.Second,
|
MaxRuntime: 5 * time.Second,
|
||||||
EnableBatchedDatabaseWrites: true,
|
EnableBatchedDatabaseWrites: true,
|
||||||
}
|
}
|
||||||
|
|
||||||
table := routetable.New(routetable.Params{})
|
|
||||||
|
|
||||||
// Create server
|
// Create server
|
||||||
srv := server.New(mockDB, table, s, logger, cfg)
|
srv := server.New(mockDB, s, logger, cfg)
|
||||||
|
|
||||||
// Create RouteWatch with 5 second limit
|
// Create RouteWatch with 5 second limit
|
||||||
deps := Dependencies{
|
deps := Dependencies{
|
||||||
DB: mockDB,
|
DB: mockDB,
|
||||||
Table: table,
|
|
||||||
Streamer: s,
|
Streamer: s,
|
||||||
Server: srv,
|
Server: srv,
|
||||||
Logger: logger,
|
Logger: logger,
|
||||||
@@ -475,8 +566,7 @@ func announcementFeed(prefixes []string) string {
|
|||||||
// handler's worker waits on the first one until the server sees the streamer
|
// handler's worker waits on the first one until the server sees the streamer
|
||||||
// close the connection, which only Shutdown does, so all three are still in
|
// close the connection, which only Shutdown does, so all three are still in
|
||||||
// the streamer when Shutdown starts. Each prefix must be in the database when
|
// the streamer when Shutdown starts. Each prefix must be in the database when
|
||||||
// Shutdown returns, both in the prefix tables and in the routing table's
|
// Shutdown returns.
|
||||||
// writes.
|
|
||||||
func TestShutdownWritesTheUpdatesQueuedInTheStreamer(t *testing.T) {
|
func TestShutdownWritesTheUpdatesQueuedInTheStreamer(t *testing.T) {
|
||||||
t.Parallel()
|
t.Parallel()
|
||||||
|
|
||||||
@@ -506,11 +596,11 @@ func TestShutdownWritesTheUpdatesQueuedInTheStreamer(t *testing.T) {
|
|||||||
rw := New(Dependencies{
|
rw := New(Dependencies{
|
||||||
DB: store,
|
DB: store,
|
||||||
Streamer: s,
|
Streamer: s,
|
||||||
Server: server.New(store, nil, s, log, cfg),
|
Server: server.New(store, s, log, cfg),
|
||||||
Logger: log,
|
Logger: log,
|
||||||
Config: cfg,
|
Config: cfg,
|
||||||
})
|
})
|
||||||
rw.prefixHandler = NewPrefixHandler(store, log, routetable.New(routetable.Params{}))
|
rw.prefixHandler = NewPrefixHandler(store, log)
|
||||||
s.RegisterHandler(gatedPrefixHandler{
|
s.RegisterHandler(gatedPrefixHandler{
|
||||||
PrefixHandler: rw.prefixHandler, release: release,
|
PrefixHandler: rw.prefixHandler, release: release,
|
||||||
})
|
})
|
||||||
@@ -550,11 +640,9 @@ func TestShutdownWritesTheUpdatesQueuedInTheStreamer(t *testing.T) {
|
|||||||
defer store.mu.Unlock()
|
defer store.mu.Unlock()
|
||||||
|
|
||||||
for _, prefix := range prefixes {
|
for _, prefix := range prefixes {
|
||||||
_, inPrefixTables := store.Prefixes[prefix]
|
if _, ok := store.Prefixes[prefix]; !ok {
|
||||||
if !inPrefixTables || !store.TablePrefixes[prefix] {
|
t.Errorf("%s was queued in the streamer when Shutdown started "+
|
||||||
t.Errorf("%s was queued in the streamer when Shutdown started; "+
|
"but is not in the database", prefix)
|
||||||
"written to the prefix tables %t, in the routing table's "+
|
|
||||||
"writes %t", prefix, inPrefixTables, store.TablePrefixes[prefix])
|
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -608,7 +696,7 @@ func TestShutdownReturnsByItsDeadlineWhileAWriteIsBlocked(t *testing.T) {
|
|||||||
rw := New(Dependencies{
|
rw := New(Dependencies{
|
||||||
DB: store,
|
DB: store,
|
||||||
Streamer: s,
|
Streamer: s,
|
||||||
Server: server.New(store, nil, s, log, cfg),
|
Server: server.New(store, s, log, cfg),
|
||||||
Logger: log,
|
Logger: log,
|
||||||
Config: cfg,
|
Config: cfg,
|
||||||
})
|
})
|
||||||
@@ -722,7 +810,7 @@ func TestShutdownCheckpointsAndClosesTheDatabaseLast(t *testing.T) {
|
|||||||
rw := New(Dependencies{
|
rw := New(Dependencies{
|
||||||
DB: store,
|
DB: store,
|
||||||
Streamer: s,
|
Streamer: s,
|
||||||
Server: server.New(store, nil, s, log, cfg),
|
Server: server.New(store, s, log, cfg),
|
||||||
Logger: log,
|
Logger: log,
|
||||||
Config: cfg,
|
Config: cfg,
|
||||||
})
|
})
|
||||||
|
|||||||
@@ -52,6 +52,7 @@ type ASHandler struct {
|
|||||||
// &mu.
|
// &mu.
|
||||||
batchTaken sync.Cond
|
batchTaken sync.Cond
|
||||||
batchFullCh chan struct{} // wakes the flush goroutine when batch is full
|
batchFullCh chan struct{} // wakes the flush goroutine when batch is full
|
||||||
|
lastFlush time.Time // used only by the flush goroutine
|
||||||
// stopCh hands Stop's context to the flush goroutine. It holds one, so
|
// stopCh hands Stop's context to the flush goroutine. It holds one, so
|
||||||
// Stop does not wait for a write in progress to hand it over.
|
// Stop does not wait for a write in progress to hand it over.
|
||||||
stopCh chan context.Context
|
stopCh chan context.Context
|
||||||
@@ -69,27 +70,19 @@ type asnOp struct {
|
|||||||
// Stop when finished to ensure all pending operations are flushed and the
|
// Stop when finished to ensure all pending operations are flushed and the
|
||||||
// background goroutine is terminated.
|
// background goroutine is terminated.
|
||||||
func NewASHandler(db database.Store, logger *logger.Logger) *ASHandler {
|
func NewASHandler(db database.Store, logger *logger.Logger) *ASHandler {
|
||||||
return newASHandler(db, logger, asnBatchTimeout)
|
|
||||||
}
|
|
||||||
|
|
||||||
// newASHandler is NewASHandler with the flush timer's interval, batchTimeout,
|
|
||||||
// in place of asnBatchTimeout. A test that the timer must not disturb gives an
|
|
||||||
// interval longer than the test runs.
|
|
||||||
func newASHandler(
|
|
||||||
db database.Store, logger *logger.Logger, batchTimeout time.Duration,
|
|
||||||
) *ASHandler {
|
|
||||||
h := &ASHandler{
|
h := &ASHandler{
|
||||||
db: db,
|
db: db,
|
||||||
logger: logger,
|
logger: logger,
|
||||||
batch: make([]asnOp, 0, asnBatchSize),
|
batch: make([]asnOp, 0, asnBatchSize),
|
||||||
batchFullCh: make(chan struct{}, 1),
|
batchFullCh: make(chan struct{}, 1),
|
||||||
|
lastFlush: time.Now(),
|
||||||
stopCh: make(chan context.Context, 1),
|
stopCh: make(chan context.Context, 1),
|
||||||
done: make(chan struct{}),
|
done: make(chan struct{}),
|
||||||
}
|
}
|
||||||
h.batchTaken.L = &h.mu
|
h.batchTaken.L = &h.mu
|
||||||
|
|
||||||
// Start the flush timer goroutine
|
// Start the flush timer goroutine
|
||||||
go flushLoop(batchTimeout, h.batchFullCh, h.stopCh, h.done, h.flushBatch)
|
go h.flushLoop()
|
||||||
|
|
||||||
return h
|
return h
|
||||||
}
|
}
|
||||||
@@ -182,6 +175,33 @@ func (h *ASHandler) Stop(ctx context.Context) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// flushLoop runs in a goroutine, the only one that writes the batch: it
|
||||||
|
// flushes when the batch is full, periodically, and once more on Stop.
|
||||||
|
func (h *ASHandler) flushLoop() {
|
||||||
|
defer close(h.done)
|
||||||
|
|
||||||
|
ticker := time.NewTicker(asnBatchTimeout)
|
||||||
|
defer ticker.Stop()
|
||||||
|
|
||||||
|
for {
|
||||||
|
select {
|
||||||
|
case <-ticker.C:
|
||||||
|
if time.Since(h.lastFlush) >= asnBatchTimeout {
|
||||||
|
h.flushBatch()
|
||||||
|
}
|
||||||
|
case <-h.batchFullCh:
|
||||||
|
h.flushBatch()
|
||||||
|
case ctx := <-h.stopCh:
|
||||||
|
// Final flush, unless the stop deadline has passed
|
||||||
|
if ctx.Err() == nil {
|
||||||
|
h.flushBatch()
|
||||||
|
}
|
||||||
|
|
||||||
|
return
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
// takeBatch returns the pending operations and leaves an empty batch in their
|
// takeBatch returns the pending operations and leaves an empty batch in their
|
||||||
// place for HandleMessage to fill. They count in writing until flushBatch has
|
// place for HandleMessage to fill. They count in writing until flushBatch has
|
||||||
// written them.
|
// written them.
|
||||||
@@ -229,4 +249,6 @@ func (h *ASHandler) flushBatch() {
|
|||||||
h.mu.Lock()
|
h.mu.Lock()
|
||||||
h.writing = 0
|
h.writing = 0
|
||||||
h.mu.Unlock()
|
h.mu.Unlock()
|
||||||
|
|
||||||
|
h.lastFlush = time.Now()
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -29,9 +29,9 @@ func TestASHandlerStoresTheEarliestAndLatestTimesOfANewAS(t *testing.T) {
|
|||||||
h.HandleMessage(second)
|
h.HandleMessage(second)
|
||||||
h.flushBatch()
|
h.flushBatch()
|
||||||
|
|
||||||
asn, _, err := db.GetAS(t.Context(), testASNA)
|
asn, _, err := db.GetASDetails(testASNA)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatalf("GetAS(%d): %v", testASNA, err)
|
t.Fatalf("GetASDetails(%d): %v", testASNA, err)
|
||||||
}
|
}
|
||||||
|
|
||||||
if !asn.FirstSeen.Equal(first.ParsedTimestamp) {
|
if !asn.FirstSeen.Equal(first.ParsedTimestamp) {
|
||||||
|
|||||||
@@ -68,12 +68,10 @@ type ASNFetcher struct {
|
|||||||
|
|
||||||
// NewASNFetcher creates a new ASN fetcher.
|
// NewASNFetcher creates a new ASN fetcher.
|
||||||
func NewASNFetcher(db database.Store, logger *slog.Logger) *ASNFetcher {
|
func NewASNFetcher(db database.Store, logger *slog.Logger) *ASNFetcher {
|
||||||
logger = logger.With("component", "asn_fetcher")
|
|
||||||
|
|
||||||
return &ASNFetcher{
|
return &ASNFetcher{
|
||||||
db: db,
|
db: db,
|
||||||
whoisClient: whois.NewClient(logger),
|
whoisClient: whois.NewClient(),
|
||||||
logger: logger,
|
logger: logger.With("component", "asn_fetcher"),
|
||||||
immediateQueue: make(chan int, immediateQueueSize),
|
immediateQueue: make(chan int, immediateQueueSize),
|
||||||
stopCh: make(chan struct{}),
|
stopCh: make(chan struct{}),
|
||||||
done: make(chan struct{}),
|
done: make(chan struct{}),
|
||||||
|
|||||||
@@ -1,62 +0,0 @@
|
|||||||
package routewatch
|
|
||||||
|
|
||||||
import (
|
|
||||||
"context"
|
|
||||||
"time"
|
|
||||||
)
|
|
||||||
|
|
||||||
// flushLoop is the flush goroutine of the prefix, AS and peer handlers. It is
|
|
||||||
// the only goroutine that writes a handler's batch. flushBatch takes the batch
|
|
||||||
// and writes it.
|
|
||||||
//
|
|
||||||
// flushLoop calls flushBatch on each tick of a timer, every batchTimeout. It
|
|
||||||
// also calls it at once when batchFullCh wakes it. A full batch's write
|
|
||||||
// restarts the timer, so the next tick comes batchTimeout after that write
|
|
||||||
// began. When stopCh hands it Stop's context, it calls flushBatch once more,
|
|
||||||
// unless the stop deadline has passed. Then it closes done and returns.
|
|
||||||
func flushLoop(
|
|
||||||
batchTimeout time.Duration,
|
|
||||||
batchFullCh <-chan struct{},
|
|
||||||
stopCh <-chan context.Context,
|
|
||||||
done chan<- struct{},
|
|
||||||
flushBatch func(),
|
|
||||||
) {
|
|
||||||
defer close(done)
|
|
||||||
|
|
||||||
ticker := time.NewTicker(batchTimeout)
|
|
||||||
defer ticker.Stop()
|
|
||||||
|
|
||||||
for {
|
|
||||||
// Stop goes first. A write can run past the stop deadline while a full
|
|
||||||
// batch's wake-up or a tick waits beside Stop's context. The select
|
|
||||||
// below picks one of them at random, but past the deadline no flush
|
|
||||||
// may start.
|
|
||||||
select {
|
|
||||||
case ctx := <-stopCh:
|
|
||||||
// Final flush, unless the stop deadline has passed
|
|
||||||
if ctx.Err() == nil {
|
|
||||||
flushBatch()
|
|
||||||
}
|
|
||||||
|
|
||||||
return
|
|
||||||
default:
|
|
||||||
}
|
|
||||||
|
|
||||||
select {
|
|
||||||
case <-ticker.C:
|
|
||||||
flushBatch()
|
|
||||||
case <-batchFullCh:
|
|
||||||
// Reset also drops a tick that waits in ticker.C, so no timed
|
|
||||||
// write follows this one at once.
|
|
||||||
ticker.Reset(batchTimeout)
|
|
||||||
flushBatch()
|
|
||||||
case ctx := <-stopCh:
|
|
||||||
// Final flush, unless the stop deadline has passed
|
|
||||||
if ctx.Err() == nil {
|
|
||||||
flushBatch()
|
|
||||||
}
|
|
||||||
|
|
||||||
return
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
@@ -4,18 +4,15 @@ import (
|
|||||||
"bytes"
|
"bytes"
|
||||||
"context"
|
"context"
|
||||||
"encoding/json"
|
"encoding/json"
|
||||||
"fmt"
|
|
||||||
"log/slog"
|
"log/slog"
|
||||||
"slices"
|
"slices"
|
||||||
"sync"
|
"sync"
|
||||||
"testing"
|
"testing"
|
||||||
"testing/synctest"
|
|
||||||
"time"
|
"time"
|
||||||
|
|
||||||
"git.eeqj.de/sneak/routewatch/internal/database"
|
"git.eeqj.de/sneak/routewatch/internal/database"
|
||||||
"git.eeqj.de/sneak/routewatch/internal/logger"
|
"git.eeqj.de/sneak/routewatch/internal/logger"
|
||||||
"git.eeqj.de/sneak/routewatch/internal/ristypes"
|
"git.eeqj.de/sneak/routewatch/internal/ristypes"
|
||||||
"git.eeqj.de/sneak/routewatch/internal/routetable"
|
|
||||||
)
|
)
|
||||||
|
|
||||||
const (
|
const (
|
||||||
@@ -28,62 +25,10 @@ const (
|
|||||||
stillWaitingAfter = 100 * time.Millisecond
|
stillWaitingAfter = 100 * time.Millisecond
|
||||||
)
|
)
|
||||||
|
|
||||||
// TestFlushLoopWritesOnEveryTick runs flushLoop in a synctest bubble, whose
|
|
||||||
// clock is fake: time.Sleep there moves the clock on and returns at once. Each
|
|
||||||
// write takes a tenth of the interval. Two timed writes must start one interval
|
|
||||||
// apart. A full batch's write, which starts as the second timed write ends,
|
|
||||||
// must be followed by a timed write one interval after it began.
|
|
||||||
func TestFlushLoopWritesOnEveryTick(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
synctest.Test(t, func(t *testing.T) {
|
|
||||||
const (
|
|
||||||
interval = time.Second
|
|
||||||
writeTime = 100 * time.Millisecond
|
|
||||||
)
|
|
||||||
|
|
||||||
writeStarts := make(chan time.Time)
|
|
||||||
batchFullCh := make(chan struct{}, 1)
|
|
||||||
stopCh := make(chan context.Context, 1)
|
|
||||||
done := make(chan struct{})
|
|
||||||
|
|
||||||
go func() {
|
|
||||||
flushLoop(interval, batchFullCh, stopCh, done, func() {
|
|
||||||
writeStarts <- time.Now()
|
|
||||||
|
|
||||||
time.Sleep(writeTime)
|
|
||||||
})
|
|
||||||
}()
|
|
||||||
|
|
||||||
first := <-writeStarts
|
|
||||||
second := <-writeStarts
|
|
||||||
|
|
||||||
if gap := second.Sub(first); gap != interval {
|
|
||||||
t.Errorf("the second timed write started %s after the first, "+
|
|
||||||
"want %s", gap, interval)
|
|
||||||
}
|
|
||||||
|
|
||||||
batchFullCh <- struct{}{}
|
|
||||||
|
|
||||||
full := <-writeStarts
|
|
||||||
timed := <-writeStarts
|
|
||||||
|
|
||||||
if gap := timed.Sub(full); gap != interval {
|
|
||||||
t.Errorf("the timed write after a full batch's started %s after "+
|
|
||||||
"it, want %s", gap, interval)
|
|
||||||
}
|
|
||||||
|
|
||||||
stopCh <- t.Context()
|
|
||||||
|
|
||||||
<-writeStarts // Stop's last write
|
|
||||||
<-done
|
|
||||||
})
|
|
||||||
}
|
|
||||||
|
|
||||||
// blockingStore is a mockStore whose batch writes for the AS, peer and prefix
|
// blockingStore is a mockStore whose batch writes for the AS, peer and prefix
|
||||||
// handlers and peering write wait until the test calls releaseWrites.
|
// handlers, delete of a peer's routes and peering write wait until the test
|
||||||
// writeStarted is closed when the first of them starts. It records what each
|
// calls releaseWrites. writeStarted is closed when the first of them starts. It
|
||||||
// batch write was given.
|
// records what each batch write was given.
|
||||||
type blockingStore struct {
|
type blockingStore struct {
|
||||||
*mockStore
|
*mockStore
|
||||||
|
|
||||||
@@ -109,13 +54,13 @@ func newBlockingStore() *blockingStore {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func (s *blockingStore) UpdatePrefixesBatch(prefixes map[string]time.Time) error {
|
func (s *blockingStore) UpsertLiveRouteBatch(routes []*database.LiveRoute) error {
|
||||||
s.waitForRelease()
|
s.waitForRelease()
|
||||||
s.mu.Lock()
|
s.mu.Lock()
|
||||||
defer s.mu.Unlock()
|
defer s.mu.Unlock()
|
||||||
|
|
||||||
for prefix := range prefixes {
|
for _, route := range routes {
|
||||||
s.prefixes[prefix] = true
|
s.prefixes[route.Prefix] = true
|
||||||
}
|
}
|
||||||
|
|
||||||
return nil
|
return nil
|
||||||
@@ -145,6 +90,12 @@ func (s *blockingStore) UpdatePeerBatch(peers map[string]database.PeerUpdate) er
|
|||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func (s *blockingStore) DeleteLiveRoutesOfPeer(peerIP string) (int, error) {
|
||||||
|
s.waitForRelease()
|
||||||
|
|
||||||
|
return s.mockStore.DeleteLiveRoutesOfPeer(peerIP)
|
||||||
|
}
|
||||||
|
|
||||||
func (s *blockingStore) RecordPeering(
|
func (s *blockingStore) RecordPeering(
|
||||||
asA, asB int, seen database.SeenTimes,
|
asA, asB int, seen database.SeenTimes,
|
||||||
) error {
|
) error {
|
||||||
@@ -168,10 +119,10 @@ func (s *blockingStore) releaseWrites() {
|
|||||||
//
|
//
|
||||||
//nolint:tagliatelle // field names are the keys the handlers log
|
//nolint:tagliatelle // field names are the keys the handlers log
|
||||||
type stopLog struct {
|
type stopLog struct {
|
||||||
Msg string `json:"msg"`
|
Msg string `json:"msg"`
|
||||||
Updates int `json:"updates"`
|
Updates int `json:"updates"`
|
||||||
TableChanges int `json:"table_changes"`
|
PeersDown int `json:"peers_down"`
|
||||||
Paths int `json:"paths"`
|
Paths int `json:"paths"`
|
||||||
}
|
}
|
||||||
|
|
||||||
// newErrorLogger returns a logger that writes only errors, as JSON lines, to
|
// newErrorLogger returns a logger that writes only errors, as JSON lines, to
|
||||||
@@ -215,10 +166,9 @@ func returnsInTime(f func()) bool {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// startBlockedWrite hands h the messages in fill and waits until h starts to
|
// startBlockedWrite hands h the messages in fill, which fill its batch so that
|
||||||
// write them to store. The prefix, AS and peer handlers start when fill fills
|
// the handler writes it to store, and waits until that write starts. store
|
||||||
// their batch, and the peering handler on its timer. store holds the write
|
// holds it until the test calls releaseWrites.
|
||||||
// until the test calls releaseWrites.
|
|
||||||
func startBlockedWrite(
|
func startBlockedWrite(
|
||||||
t *testing.T,
|
t *testing.T,
|
||||||
store *blockingStore,
|
store *blockingStore,
|
||||||
@@ -241,7 +191,7 @@ func startBlockedWrite(
|
|||||||
select {
|
select {
|
||||||
case <-store.writeStarted:
|
case <-store.writeStarted:
|
||||||
case <-time.After(handleMessageTimeout):
|
case <-time.After(handleMessageTimeout):
|
||||||
t.Fatal("the handler did not start to write the messages")
|
t.Fatal("the handler did not write its full batch")
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -321,7 +271,7 @@ func handleAtMaxBatchSize(
|
|||||||
}
|
}
|
||||||
|
|
||||||
// stopWhileAWriteIsBlocked starts a blocked write with the messages in fill,
|
// stopWhileAWriteIsBlocked starts a blocked write with the messages in fill,
|
||||||
// hands h the messages in extra, and stops h with a deadline that passes while
|
// hands h the message extra, and stops h with a deadline that passes while
|
||||||
// store holds the write. Stop must return all the same. Then it lets the write
|
// store holds the write. Stop must return all the same. Then it lets the write
|
||||||
// finish and waits until done, closed when h's flush goroutine returns. Past
|
// finish and waits until done, closed when h's flush goroutine returns. Past
|
||||||
// the deadline that goroutine skips its last write, so extra is not written,
|
// the deadline that goroutine skips its last write, so extra is not written,
|
||||||
@@ -335,7 +285,7 @@ func stopWhileAWriteIsBlocked(
|
|||||||
},
|
},
|
||||||
done <-chan struct{},
|
done <-chan struct{},
|
||||||
fill []*ristypes.RISMessage,
|
fill []*ristypes.RISMessage,
|
||||||
extra ...*ristypes.RISMessage,
|
extra *ristypes.RISMessage,
|
||||||
) {
|
) {
|
||||||
t.Helper()
|
t.Helper()
|
||||||
|
|
||||||
@@ -344,10 +294,7 @@ func stopWhileAWriteIsBlocked(
|
|||||||
defer store.releaseWrites()
|
defer store.releaseWrites()
|
||||||
|
|
||||||
startBlockedWrite(t, store, h, fill)
|
startBlockedWrite(t, store, h, fill)
|
||||||
|
h.HandleMessage(extra)
|
||||||
for _, msg := range extra {
|
|
||||||
h.HandleMessage(msg)
|
|
||||||
}
|
|
||||||
|
|
||||||
ctx, cancel := context.WithTimeout(t.Context(), stopDeadline)
|
ctx, cancel := context.WithTimeout(t.Context(), stopDeadline)
|
||||||
defer cancel()
|
defer cancel()
|
||||||
@@ -366,11 +313,6 @@ func stopWhileAWriteIsBlocked(
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// TestPrefixHandlerSkipsItsLastWriteAfterTheStopDeadline also checks the
|
|
||||||
// routing table's changes: the flush whose prefix write is held wrote those of
|
|
||||||
// the fill, so Stop must log the one of the extra prefix as not written. The
|
|
||||||
// flush timer must not fire. A flush it started before the fill would be the
|
|
||||||
// one held, and it would not have written the fill's change.
|
|
||||||
func TestPrefixHandlerSkipsItsLastWriteAfterTheStopDeadline(t *testing.T) {
|
func TestPrefixHandlerSkipsItsLastWriteAfterTheStopDeadline(t *testing.T) {
|
||||||
t.Parallel()
|
t.Parallel()
|
||||||
|
|
||||||
@@ -379,11 +321,12 @@ func TestPrefixHandlerSkipsItsLastWriteAfterTheStopDeadline(t *testing.T) {
|
|||||||
var logs bytes.Buffer
|
var logs bytes.Buffer
|
||||||
|
|
||||||
store := newBlockingStore()
|
store := newBlockingStore()
|
||||||
h := newPrefixHandler(store, newErrorLogger(&logs),
|
h := NewPrefixHandler(store, newErrorLogger(&logs))
|
||||||
routetable.New(routetable.Params{}), longerThanAnyTest)
|
|
||||||
fill := slices.Repeat([]string{testV4Prefix}, prefixBatchSize)
|
fill := slices.Repeat([]string{testV4Prefix}, prefixBatchSize)
|
||||||
|
// The peer's session goes down first, so the write held is the delete of
|
||||||
|
// its routes, and Stop must log that peer down too.
|
||||||
stopWhileAWriteIsBlocked(t, store, h, h.done,
|
stopWhileAWriteIsBlocked(t, store, h, h.done,
|
||||||
[]*ristypes.RISMessage{prefixAnnouncement(fill)},
|
[]*ristypes.RISMessage{sessionDown(), prefixAnnouncement(fill)},
|
||||||
prefixAnnouncement([]string{extraPrefix}))
|
prefixAnnouncement([]string{extraPrefix}))
|
||||||
|
|
||||||
store.mu.Lock()
|
store.mu.Lock()
|
||||||
@@ -394,75 +337,9 @@ func TestPrefixHandlerSkipsItsLastWriteAfterTheStopDeadline(t *testing.T) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
logged := lastLogged(t, &logs)
|
logged := lastLogged(t, &logs)
|
||||||
if logged.Updates != prefixBatchSize+1 || logged.TableChanges != 1 {
|
if logged.Updates != prefixBatchSize+1 || logged.PeersDown != 1 {
|
||||||
t.Errorf("Stop logged %d updates and %d table changes not written, "+
|
t.Errorf("Stop logged %d updates and %d peer downs not written, "+
|
||||||
"want %d and 1", logged.Updates, logged.TableChanges,
|
"want %d and 1", logged.Updates, logged.PeersDown, prefixBatchSize+1)
|
||||||
prefixBatchSize+1)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestPrefixHandlerSkipsAFullBatchAfterTheStopDeadline hands the prefix handler
|
|
||||||
// a full batch while a write is held, so the batch's wake-up of the flush
|
|
||||||
// goroutine waits. The stop deadline passes meanwhile. When the write ends, the
|
|
||||||
// flush goroutine must stop instead of writing the batch.
|
|
||||||
func TestPrefixHandlerSkipsAFullBatchAfterTheStopDeadline(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
const extraPrefix = "198.51.100.0/24"
|
|
||||||
|
|
||||||
store := newBlockingStore()
|
|
||||||
h := NewPrefixHandler(store, logger.New(), routetable.New(routetable.Params{}))
|
|
||||||
fill := slices.Repeat([]string{testV4Prefix}, prefixBatchSize)
|
|
||||||
full := slices.Repeat([]string{extraPrefix}, prefixBatchSize)
|
|
||||||
stopWhileAWriteIsBlocked(t, store, h, h.done,
|
|
||||||
[]*ristypes.RISMessage{prefixAnnouncement(fill)},
|
|
||||||
prefixAnnouncement(full))
|
|
||||||
|
|
||||||
store.mu.Lock()
|
|
||||||
defer store.mu.Unlock()
|
|
||||||
|
|
||||||
if store.prefixes[extraPrefix] {
|
|
||||||
t.Errorf("%s was written after the stop deadline", extraPrefix)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// tableWriteBlockingStore is a blockingStore whose writes of the routing
|
|
||||||
// table's changes also wait until the test calls releaseWrites.
|
|
||||||
type tableWriteBlockingStore struct {
|
|
||||||
*blockingStore
|
|
||||||
}
|
|
||||||
|
|
||||||
func (s tableWriteBlockingStore) WriteRouteChanges(
|
|
||||||
changes []routetable.Change, newPeers map[string]uint,
|
|
||||||
) error {
|
|
||||||
s.waitForRelease()
|
|
||||||
|
|
||||||
return s.mockStore.WriteRouteChanges(changes, newPeers)
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestPrefixHandlerStopCountsTheTableChangesOfAWriteInProgress holds the write
|
|
||||||
// of the routing table's changes that a full batch starts, then hands the
|
|
||||||
// prefix handler one more prefix and stops it with a deadline that passes
|
|
||||||
// meanwhile. Stop must log as not written both the change the held write took
|
|
||||||
// and the one since.
|
|
||||||
func TestPrefixHandlerStopCountsTheTableChangesOfAWriteInProgress(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
const extraPrefix = "198.51.100.0/24"
|
|
||||||
|
|
||||||
var logs bytes.Buffer
|
|
||||||
|
|
||||||
blocking := newBlockingStore()
|
|
||||||
h := NewPrefixHandler(tableWriteBlockingStore{blockingStore: blocking},
|
|
||||||
newErrorLogger(&logs), routetable.New(routetable.Params{}))
|
|
||||||
fill := slices.Repeat([]string{testV4Prefix}, prefixBatchSize)
|
|
||||||
stopWhileAWriteIsBlocked(t, blocking, h, h.done,
|
|
||||||
[]*ristypes.RISMessage{prefixAnnouncement(fill)},
|
|
||||||
prefixAnnouncement([]string{extraPrefix}))
|
|
||||||
|
|
||||||
if logged := lastLogged(t, &logs); logged.TableChanges != 2 {
|
|
||||||
t.Errorf("Stop logged %d table changes not written, want 2",
|
|
||||||
logged.TableChanges)
|
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -476,7 +353,7 @@ func TestASHandlerSkipsItsLastWriteAfterTheStopDeadline(t *testing.T) {
|
|||||||
var logs bytes.Buffer
|
var logs bytes.Buffer
|
||||||
|
|
||||||
store := newBlockingStore()
|
store := newBlockingStore()
|
||||||
h := newASHandler(store, newErrorLogger(&logs), longerThanAnyTest)
|
h := NewASHandler(store, newErrorLogger(&logs))
|
||||||
fill := ristypes.ASPath{Sequence: slices.Repeat([]int{testASNA}, asnBatchSize)}
|
fill := ristypes.ASPath{Sequence: slices.Repeat([]int{testASNA}, asnBatchSize)}
|
||||||
stopWhileAWriteIsBlocked(t, store, h, h.done,
|
stopWhileAWriteIsBlocked(t, store, h, h.done,
|
||||||
[]*ristypes.RISMessage{{Path: fill}},
|
[]*ristypes.RISMessage{{Path: fill}},
|
||||||
@@ -495,29 +372,6 @@ func TestASHandlerSkipsItsLastWriteAfterTheStopDeadline(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// TestASHandlerSkipsAFullBatchAfterTheStopDeadline hands the AS handler a full
|
|
||||||
// batch while a write is held, so the batch's wake-up of the flush goroutine
|
|
||||||
// waits. The stop deadline passes meanwhile. When the write ends, the flush
|
|
||||||
// goroutine must stop instead of writing the batch.
|
|
||||||
func TestASHandlerSkipsAFullBatchAfterTheStopDeadline(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
store := newBlockingStore()
|
|
||||||
h := newASHandler(store, logger.New(), longerThanAnyTest)
|
|
||||||
fill := ristypes.ASPath{Sequence: slices.Repeat([]int{testASNA}, asnBatchSize)}
|
|
||||||
full := ristypes.ASPath{Sequence: slices.Repeat([]int{testASNB}, asnBatchSize)}
|
|
||||||
stopWhileAWriteIsBlocked(t, store, h, h.done,
|
|
||||||
[]*ristypes.RISMessage{{Path: fill}},
|
|
||||||
&ristypes.RISMessage{Path: full})
|
|
||||||
|
|
||||||
store.mu.Lock()
|
|
||||||
defer store.mu.Unlock()
|
|
||||||
|
|
||||||
if store.asns[testASNB] {
|
|
||||||
t.Errorf("AS%d was written after the stop deadline", testASNB)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestPeerHandlerSkipsItsLastWriteAfterTheStopDeadline(t *testing.T) {
|
func TestPeerHandlerSkipsItsLastWriteAfterTheStopDeadline(t *testing.T) {
|
||||||
t.Parallel()
|
t.Parallel()
|
||||||
|
|
||||||
@@ -526,7 +380,7 @@ func TestPeerHandlerSkipsItsLastWriteAfterTheStopDeadline(t *testing.T) {
|
|||||||
var logs bytes.Buffer
|
var logs bytes.Buffer
|
||||||
|
|
||||||
store := newBlockingStore()
|
store := newBlockingStore()
|
||||||
h := newPeerHandler(store, newErrorLogger(&logs), longerThanAnyTest)
|
h := NewPeerHandler(store, newErrorLogger(&logs))
|
||||||
stopWhileAWriteIsBlocked(t, store, h, h.done,
|
stopWhileAWriteIsBlocked(t, store, h, h.done,
|
||||||
slices.Repeat([]*ristypes.RISMessage{{Peer: testPeerIP}}, peerBatchSize),
|
slices.Repeat([]*ristypes.RISMessage{{Peer: testPeerIP}}, peerBatchSize),
|
||||||
&ristypes.RISMessage{Peer: otherPeerIP})
|
&ristypes.RISMessage{Peer: otherPeerIP})
|
||||||
@@ -544,30 +398,6 @@ func TestPeerHandlerSkipsItsLastWriteAfterTheStopDeadline(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// TestPeerHandlerSkipsAFullBatchAfterTheStopDeadline hands the peer handler a
|
|
||||||
// full batch while a write is held, so the batch's wake-up of the flush
|
|
||||||
// goroutine waits. The stop deadline passes meanwhile. When the write ends, the
|
|
||||||
// flush goroutine must stop instead of writing the batch.
|
|
||||||
func TestPeerHandlerSkipsAFullBatchAfterTheStopDeadline(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
const otherPeerIP = "2001:db8:ffff::2"
|
|
||||||
|
|
||||||
store := newBlockingStore()
|
|
||||||
h := newPeerHandler(store, logger.New(), longerThanAnyTest)
|
|
||||||
stopWhileAWriteIsBlocked(t, store, h, h.done,
|
|
||||||
slices.Repeat([]*ristypes.RISMessage{{Peer: testPeerIP}}, peerBatchSize),
|
|
||||||
slices.Repeat([]*ristypes.RISMessage{{Peer: otherPeerIP}},
|
|
||||||
peerBatchSize)...)
|
|
||||||
|
|
||||||
store.mu.Lock()
|
|
||||||
defer store.mu.Unlock()
|
|
||||||
|
|
||||||
if store.peers[otherPeerIP] {
|
|
||||||
t.Errorf("peer %s was written after the stop deadline", otherPeerIP)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestPeeringHandlerStopCountsTheWriteInProgress hands a running peering
|
// TestPeeringHandlerStopCountsTheWriteInProgress hands a running peering
|
||||||
// handler one AS path and stops it with a deadline that passes while store
|
// handler one AS path and stops it with a deadline that passes while store
|
||||||
// holds Stop's last write, of that path's peering. Stop must log that path as
|
// holds Stop's last write, of that path's peering. Stop must log that path as
|
||||||
@@ -607,35 +437,6 @@ func TestPeeringHandlerStopCountsTheWriteInProgress(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// TestPeeringHandlerStartsNoWriteAfterTheStopDeadline has the peering handler's
|
|
||||||
// timer start a write that store holds, and hands the handler one more AS path
|
|
||||||
// meanwhile. The stop deadline passes, and a tick of the timer waits. When the
|
|
||||||
// write ends, the processing goroutine must stop instead of writing the path.
|
|
||||||
func TestPeeringHandlerStartsNoWriteAfterTheStopDeadline(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
// The timer fires many times while Stop waits for its deadline.
|
|
||||||
const processInterval = 10 * time.Millisecond
|
|
||||||
|
|
||||||
store := newBlockingStore()
|
|
||||||
h := newPeeringHandler(store, logger.New(), processInterval)
|
|
||||||
stopWhileAWriteIsBlocked(t, store, h, h.done,
|
|
||||||
[]*ristypes.RISMessage{
|
|
||||||
{Path: ristypes.ASPath{Sequence: []int{testASNA, testASNB}}},
|
|
||||||
},
|
|
||||||
&ristypes.RISMessage{
|
|
||||||
Path: ristypes.ASPath{Sequence: []int{testASNC, testASND}},
|
|
||||||
})
|
|
||||||
|
|
||||||
store.mockStore.mu.Lock()
|
|
||||||
defer store.mockStore.mu.Unlock()
|
|
||||||
|
|
||||||
if store.Peerings[fmt.Sprintf("%d_%d", testASNC, testASND)] {
|
|
||||||
t.Errorf("the peering of AS%d and AS%d was written after the stop "+
|
|
||||||
"deadline", testASNC, testASND)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// prefixAnnouncement returns a message in which testPeerIP announces prefixes
|
// prefixAnnouncement returns a message in which testPeerIP announces prefixes
|
||||||
// with origin AS testASNA.
|
// with origin AS testASNA.
|
||||||
func prefixAnnouncement(prefixes []string) *ristypes.RISMessage {
|
func prefixAnnouncement(prefixes []string) *ristypes.RISMessage {
|
||||||
@@ -654,7 +455,7 @@ func TestPrefixHandlerHandlesMessagesWhileAWriteIsBlocked(t *testing.T) {
|
|||||||
const extraPrefix = "198.51.100.0/24"
|
const extraPrefix = "198.51.100.0/24"
|
||||||
|
|
||||||
store := newBlockingStore()
|
store := newBlockingStore()
|
||||||
h := NewPrefixHandler(store, logger.New(), routetable.New(routetable.Params{}))
|
h := NewPrefixHandler(store, logger.New())
|
||||||
fill := slices.Repeat([]string{testV4Prefix}, prefixBatchSize)
|
fill := slices.Repeat([]string{testV4Prefix}, prefixBatchSize)
|
||||||
handleWhileAWriteIsBlocked(t, store, h,
|
handleWhileAWriteIsBlocked(t, store, h,
|
||||||
[]*ristypes.RISMessage{prefixAnnouncement(fill)},
|
[]*ristypes.RISMessage{prefixAnnouncement(fill)},
|
||||||
@@ -673,7 +474,7 @@ func TestASHandlerHandlesMessagesWhileAWriteIsBlocked(t *testing.T) {
|
|||||||
t.Parallel()
|
t.Parallel()
|
||||||
|
|
||||||
store := newBlockingStore()
|
store := newBlockingStore()
|
||||||
h := newASHandler(store, logger.New(), longerThanAnyTest)
|
h := NewASHandler(store, logger.New())
|
||||||
fill := ristypes.ASPath{Sequence: slices.Repeat([]int{testASNA}, asnBatchSize)}
|
fill := ristypes.ASPath{Sequence: slices.Repeat([]int{testASNA}, asnBatchSize)}
|
||||||
handleWhileAWriteIsBlocked(t, store, h,
|
handleWhileAWriteIsBlocked(t, store, h,
|
||||||
[]*ristypes.RISMessage{{Path: fill}},
|
[]*ristypes.RISMessage{{Path: fill}},
|
||||||
@@ -694,7 +495,7 @@ func TestPeerHandlerHandlesMessagesWhileAWriteIsBlocked(t *testing.T) {
|
|||||||
const otherPeerIP = "2001:db8:ffff::2"
|
const otherPeerIP = "2001:db8:ffff::2"
|
||||||
|
|
||||||
store := newBlockingStore()
|
store := newBlockingStore()
|
||||||
h := newPeerHandler(store, logger.New(), longerThanAnyTest)
|
h := NewPeerHandler(store, logger.New())
|
||||||
handleWhileAWriteIsBlocked(t, store, h,
|
handleWhileAWriteIsBlocked(t, store, h,
|
||||||
slices.Repeat([]*ristypes.RISMessage{{Peer: testPeerIP}}, peerBatchSize),
|
slices.Repeat([]*ristypes.RISMessage{{Peer: testPeerIP}}, peerBatchSize),
|
||||||
&ristypes.RISMessage{Peer: otherPeerIP})
|
&ristypes.RISMessage{Peer: otherPeerIP})
|
||||||
@@ -714,7 +515,7 @@ func TestPrefixHandlerWaitsAtTheMaximumBatchSize(t *testing.T) {
|
|||||||
const extraPrefix = "198.51.100.0/24"
|
const extraPrefix = "198.51.100.0/24"
|
||||||
|
|
||||||
store := newBlockingStore()
|
store := newBlockingStore()
|
||||||
h := NewPrefixHandler(store, logger.New(), routetable.New(routetable.Params{}))
|
h := NewPrefixHandler(store, logger.New())
|
||||||
fill := slices.Repeat([]string{testV4Prefix}, prefixBatchSize)
|
fill := slices.Repeat([]string{testV4Prefix}, prefixBatchSize)
|
||||||
toMax := slices.Repeat([]string{testV4Prefix}, prefixBatchMaxSize)
|
toMax := slices.Repeat([]string{testV4Prefix}, prefixBatchMaxSize)
|
||||||
handleAtMaxBatchSize(t, store, h,
|
handleAtMaxBatchSize(t, store, h,
|
||||||
@@ -735,7 +536,7 @@ func TestASHandlerWaitsAtTheMaximumBatchSize(t *testing.T) {
|
|||||||
t.Parallel()
|
t.Parallel()
|
||||||
|
|
||||||
store := newBlockingStore()
|
store := newBlockingStore()
|
||||||
h := newASHandler(store, logger.New(), longerThanAnyTest)
|
h := NewASHandler(store, logger.New())
|
||||||
fill := ristypes.ASPath{Sequence: slices.Repeat([]int{testASNA}, asnBatchSize)}
|
fill := ristypes.ASPath{Sequence: slices.Repeat([]int{testASNA}, asnBatchSize)}
|
||||||
toMax := ristypes.ASPath{Sequence: slices.Repeat([]int{testASNA}, asnBatchMaxSize)}
|
toMax := ristypes.ASPath{Sequence: slices.Repeat([]int{testASNA}, asnBatchMaxSize)}
|
||||||
handleAtMaxBatchSize(t, store, h,
|
handleAtMaxBatchSize(t, store, h,
|
||||||
@@ -752,17 +553,13 @@ func TestASHandlerWaitsAtTheMaximumBatchSize(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// TestPeerHandlerWaitsAtTheMaximumBatchSize must not have its flush timer fire.
|
|
||||||
// A timer flush during the fill would take only part of it. The batch would
|
|
||||||
// then reach its maximum size before the test's last message, and the test
|
|
||||||
// would hang.
|
|
||||||
func TestPeerHandlerWaitsAtTheMaximumBatchSize(t *testing.T) {
|
func TestPeerHandlerWaitsAtTheMaximumBatchSize(t *testing.T) {
|
||||||
t.Parallel()
|
t.Parallel()
|
||||||
|
|
||||||
const otherPeerIP = "2001:db8:ffff::2"
|
const otherPeerIP = "2001:db8:ffff::2"
|
||||||
|
|
||||||
store := newBlockingStore()
|
store := newBlockingStore()
|
||||||
h := newPeerHandler(store, logger.New(), longerThanAnyTest)
|
h := NewPeerHandler(store, logger.New())
|
||||||
message := []*ristypes.RISMessage{{Peer: testPeerIP}}
|
message := []*ristypes.RISMessage{{Peer: testPeerIP}}
|
||||||
handleAtMaxBatchSize(t, store, h,
|
handleAtMaxBatchSize(t, store, h,
|
||||||
slices.Repeat(message, peerBatchSize),
|
slices.Repeat(message, peerBatchSize),
|
||||||
@@ -795,10 +592,7 @@ func TestHandlersWakeTheFlushGoroutineWhenTheBatchIsFull(t *testing.T) {
|
|||||||
name: "prefix",
|
name: "prefix",
|
||||||
batchSize: prefixBatchSize,
|
batchSize: prefixBatchSize,
|
||||||
handle: func(n int) chan struct{} {
|
handle: func(n int) chan struct{} {
|
||||||
h := &PrefixHandler{
|
h := &PrefixHandler{batchFullCh: make(chan struct{}, 1)}
|
||||||
batchFullCh: make(chan struct{}, 1),
|
|
||||||
table: routetable.New(routetable.Params{}),
|
|
||||||
}
|
|
||||||
h.HandleMessage(prefixAnnouncement(
|
h.HandleMessage(prefixAnnouncement(
|
||||||
slices.Repeat([]string{testV4Prefix}, n)))
|
slices.Repeat([]string{testV4Prefix}, n)))
|
||||||
|
|
||||||
|
|||||||
@@ -56,6 +56,7 @@ type PeerHandler struct {
|
|||||||
// is &mu.
|
// is &mu.
|
||||||
batchTaken sync.Cond
|
batchTaken sync.Cond
|
||||||
batchFullCh chan struct{} // wakes the flush goroutine when peerBatch is full
|
batchFullCh chan struct{} // wakes the flush goroutine when peerBatch is full
|
||||||
|
lastFlush time.Time // used only by the flush goroutine
|
||||||
// stopCh hands Stop's context to the flush goroutine. It holds one, so
|
// stopCh hands Stop's context to the flush goroutine. It holds one, so
|
||||||
// Stop does not wait for a write in progress to hand it over.
|
// Stop does not wait for a write in progress to hand it over.
|
||||||
stopCh chan context.Context
|
stopCh chan context.Context
|
||||||
@@ -74,27 +75,19 @@ type peerUpdate struct {
|
|||||||
// periodically flushes accumulated peer updates to the database. The handler
|
// periodically flushes accumulated peer updates to the database. The handler
|
||||||
// should be stopped by calling Stop when it is no longer needed.
|
// should be stopped by calling Stop when it is no longer needed.
|
||||||
func NewPeerHandler(db database.Store, logger *logger.Logger) *PeerHandler {
|
func NewPeerHandler(db database.Store, logger *logger.Logger) *PeerHandler {
|
||||||
return newPeerHandler(db, logger, peerBatchTimeout)
|
|
||||||
}
|
|
||||||
|
|
||||||
// newPeerHandler is NewPeerHandler with the flush timer's interval,
|
|
||||||
// batchTimeout, in place of peerBatchTimeout. A test that the timer must not
|
|
||||||
// disturb gives an interval longer than the test runs.
|
|
||||||
func newPeerHandler(
|
|
||||||
db database.Store, logger *logger.Logger, batchTimeout time.Duration,
|
|
||||||
) *PeerHandler {
|
|
||||||
h := &PeerHandler{
|
h := &PeerHandler{
|
||||||
db: db,
|
db: db,
|
||||||
logger: logger,
|
logger: logger,
|
||||||
peerBatch: make([]peerUpdate, 0, peerBatchSize),
|
peerBatch: make([]peerUpdate, 0, peerBatchSize),
|
||||||
batchFullCh: make(chan struct{}, 1),
|
batchFullCh: make(chan struct{}, 1),
|
||||||
|
lastFlush: time.Now(),
|
||||||
stopCh: make(chan context.Context, 1),
|
stopCh: make(chan context.Context, 1),
|
||||||
done: make(chan struct{}),
|
done: make(chan struct{}),
|
||||||
}
|
}
|
||||||
h.batchTaken.L = &h.mu
|
h.batchTaken.L = &h.mu
|
||||||
|
|
||||||
// Start the flush timer goroutine
|
// Start the flush timer goroutine
|
||||||
go flushLoop(batchTimeout, h.batchFullCh, h.stopCh, h.done, h.flushBatch)
|
go h.flushLoop()
|
||||||
|
|
||||||
return h
|
return h
|
||||||
}
|
}
|
||||||
@@ -180,6 +173,33 @@ func (h *PeerHandler) Stop(ctx context.Context) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// flushLoop runs in a goroutine, the only one that writes the batch: it
|
||||||
|
// flushes when the batch is full, periodically, and once more on Stop.
|
||||||
|
func (h *PeerHandler) flushLoop() {
|
||||||
|
defer close(h.done)
|
||||||
|
|
||||||
|
ticker := time.NewTicker(peerBatchTimeout)
|
||||||
|
defer ticker.Stop()
|
||||||
|
|
||||||
|
for {
|
||||||
|
select {
|
||||||
|
case <-ticker.C:
|
||||||
|
if time.Since(h.lastFlush) >= peerBatchTimeout {
|
||||||
|
h.flushBatch()
|
||||||
|
}
|
||||||
|
case <-h.batchFullCh:
|
||||||
|
h.flushBatch()
|
||||||
|
case ctx := <-h.stopCh:
|
||||||
|
// Final flush, unless the stop deadline has passed
|
||||||
|
if ctx.Err() == nil {
|
||||||
|
h.flushBatch()
|
||||||
|
}
|
||||||
|
|
||||||
|
return
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
// takeBatch returns the pending updates and leaves an empty batch in their
|
// takeBatch returns the pending updates and leaves an empty batch in their
|
||||||
// place for HandleMessage to fill. They count in writing until flushBatch has
|
// place for HandleMessage to fill. They count in writing until flushBatch has
|
||||||
// written them.
|
// written them.
|
||||||
@@ -234,4 +254,6 @@ func (h *PeerHandler) flushBatch() {
|
|||||||
h.mu.Lock()
|
h.mu.Lock()
|
||||||
h.writing = 0
|
h.writing = 0
|
||||||
h.mu.Unlock()
|
h.mu.Unlock()
|
||||||
|
|
||||||
|
h.lastFlush = time.Now()
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -62,15 +62,6 @@ type PeeringHandler struct {
|
|||||||
// periodically processes accumulated AS paths into peering records. The
|
// periodically processes accumulated AS paths into peering records. The
|
||||||
// handler begins processing immediately upon creation.
|
// handler begins processing immediately upon creation.
|
||||||
func NewPeeringHandler(db database.Store, logger *logger.Logger) *PeeringHandler {
|
func NewPeeringHandler(db database.Store, logger *logger.Logger) *PeeringHandler {
|
||||||
return newPeeringHandler(db, logger, peeringProcessInterval)
|
|
||||||
}
|
|
||||||
|
|
||||||
// newPeeringHandler is NewPeeringHandler with the processing timer's interval,
|
|
||||||
// processInterval, in place of peeringProcessInterval. A test gives a short
|
|
||||||
// one to have the timer start a write.
|
|
||||||
func newPeeringHandler(
|
|
||||||
db database.Store, logger *logger.Logger, processInterval time.Duration,
|
|
||||||
) *PeeringHandler {
|
|
||||||
h := &PeeringHandler{
|
h := &PeeringHandler{
|
||||||
db: db,
|
db: db,
|
||||||
logger: logger,
|
logger: logger,
|
||||||
@@ -80,7 +71,7 @@ func newPeeringHandler(
|
|||||||
}
|
}
|
||||||
|
|
||||||
// Start the periodic processing goroutine
|
// Start the periodic processing goroutine
|
||||||
go h.processLoop(processInterval)
|
go h.processLoop()
|
||||||
|
|
||||||
return h
|
return h
|
||||||
}
|
}
|
||||||
@@ -172,29 +163,15 @@ func (h *PeeringHandler) Stop(ctx context.Context) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// processLoop processes AS paths into peerings every processInterval, and once
|
// processLoop runs periodically to process AS paths into peerings, and once
|
||||||
// more on Stop.
|
// more on Stop.
|
||||||
func (h *PeeringHandler) processLoop(processInterval time.Duration) {
|
func (h *PeeringHandler) processLoop() {
|
||||||
defer close(h.done)
|
defer close(h.done)
|
||||||
|
|
||||||
ticker := time.NewTicker(processInterval)
|
ticker := time.NewTicker(peeringProcessInterval)
|
||||||
defer ticker.Stop()
|
defer ticker.Stop()
|
||||||
|
|
||||||
for {
|
for {
|
||||||
// Stop goes first. A write can run past the stop deadline while a
|
|
||||||
// tick waits beside Stop's context. The select below picks one of
|
|
||||||
// them at random, but past the deadline no write may start.
|
|
||||||
select {
|
|
||||||
case ctx := <-h.stopCh:
|
|
||||||
// Final processing, unless the stop deadline has passed
|
|
||||||
if ctx.Err() == nil {
|
|
||||||
h.processPeerings()
|
|
||||||
}
|
|
||||||
|
|
||||||
return
|
|
||||||
default:
|
|
||||||
}
|
|
||||||
|
|
||||||
select {
|
select {
|
||||||
case <-ticker.C:
|
case <-ticker.C:
|
||||||
h.processPeerings()
|
h.processPeerings()
|
||||||
|
|||||||
@@ -2,8 +2,10 @@ package routewatch
|
|||||||
|
|
||||||
import (
|
import (
|
||||||
"context"
|
"context"
|
||||||
|
"net"
|
||||||
"net/netip"
|
"net/netip"
|
||||||
"slices"
|
"slices"
|
||||||
|
"strings"
|
||||||
"sync"
|
"sync"
|
||||||
"time"
|
"time"
|
||||||
|
|
||||||
@@ -11,7 +13,6 @@ import (
|
|||||||
"git.eeqj.de/sneak/routewatch/internal/logger"
|
"git.eeqj.de/sneak/routewatch/internal/logger"
|
||||||
"git.eeqj.de/sneak/routewatch/internal/metrics"
|
"git.eeqj.de/sneak/routewatch/internal/metrics"
|
||||||
"git.eeqj.de/sneak/routewatch/internal/ristypes"
|
"git.eeqj.de/sneak/routewatch/internal/ristypes"
|
||||||
"git.eeqj.de/sneak/routewatch/internal/routetable"
|
|
||||||
)
|
)
|
||||||
|
|
||||||
const (
|
const (
|
||||||
@@ -26,71 +27,64 @@ const (
|
|||||||
prefixBatchSize = 25000
|
prefixBatchSize = 25000
|
||||||
|
|
||||||
// prefixBatchMaxSize is the most prefix updates the batch holds while the
|
// prefixBatchMaxSize is the most prefix updates the batch holds while the
|
||||||
// flush goroutine writes the previous one: ten batches, about 15 MiB
|
// flush goroutine writes the previous one: ten batches, about 60 MiB
|
||||||
// (README "Memory"). At this size HandleMessage waits until the flush
|
// (README "Memory"). At this size HandleMessage waits until the flush
|
||||||
// goroutine takes the batch.
|
// goroutine takes the batch. On top of these, the handler holds at most
|
||||||
|
// prefixBatchSize updates, about 6 MiB, that a flush kept for the peers
|
||||||
|
// whose route delete failed (held).
|
||||||
prefixBatchMaxSize = 250000
|
prefixBatchMaxSize = 250000
|
||||||
|
|
||||||
// prefixBatchTimeout is the maximum time to wait before flushing a batch
|
// prefixBatchTimeout is the maximum time to wait before flushing a batch
|
||||||
// DO NOT reduce this timeout - larger batches are more efficient
|
// DO NOT reduce this timeout - larger batches are more efficient
|
||||||
prefixBatchTimeout = 1 * time.Second
|
prefixBatchTimeout = 1 * time.Second
|
||||||
|
|
||||||
// maxTableChangesPerWrite is the most changed prefix and origin AS pairs
|
// IP version constants
|
||||||
// of the routing table that one flush writes; the rest wait for the next
|
ipv4Version = 4
|
||||||
// flushes. The change of one pair takes about 430 bytes of memory and 15
|
ipv6Version = 6
|
||||||
// microseconds to write. So a write holds about 9 MB and takes about a
|
|
||||||
// third of the one-second flush interval, which leaves the rest to the
|
|
||||||
// batch. The feed peaks at about 5,000 updates a second, each changing at
|
|
||||||
// most one pair. So the million pairs that a full-table peer's session
|
|
||||||
// going down changes are written in about a minute.
|
|
||||||
maxTableChangesPerWrite = 20000
|
|
||||||
)
|
)
|
||||||
|
|
||||||
// PrefixHandler keeps the routing table in memory current from the feed. It
|
// PrefixHandler tracks BGP prefixes and maintains a live routing table in the
|
||||||
// applies each update to the table as it handles it. An announcement gives
|
// database. Routes are added on announcement and deleted on withdrawal or on an
|
||||||
// the peer a route to the prefix. A withdrawal removes the peer's route, as
|
// announcement whose AS path is empty or holds AS 0 anywhere, and all routes of
|
||||||
// does an announcement whose AS path is empty or holds AS 0 anywhere. A peer
|
// a peer are deleted when its BGP session goes down.
|
||||||
// whose BGP session goes down leaves every prefix.
|
|
||||||
//
|
|
||||||
// Each flush writes the table's changes to the database. It also stores each
|
|
||||||
// prefix the updates name with the time it was first seen.
|
|
||||||
type PrefixHandler struct {
|
type PrefixHandler struct {
|
||||||
db database.Store
|
db database.Store
|
||||||
logger *logger.Logger
|
logger *logger.Logger
|
||||||
metrics *metrics.Tracker
|
metrics *metrics.Tracker
|
||||||
table *routetable.Table
|
|
||||||
|
|
||||||
// Batching. mu guards batch, writing and writingTableChanges only: the
|
// Batching. mu guards batch, held, peersDown, writing and deleting only:
|
||||||
// flush goroutine takes them under mu and writes them after releasing
|
// the flush goroutine takes them under mu and writes them after releasing
|
||||||
// mu, so HandleMessage waits for a database write only while batch is at
|
// mu, so HandleMessage waits for a database write only while batch is at
|
||||||
// prefixBatchMaxSize.
|
// prefixBatchMaxSize.
|
||||||
mu sync.Mutex
|
mu sync.Mutex
|
||||||
batch []seenPrefix
|
batch []prefixUpdate
|
||||||
|
// held holds the updates that the peers whose route delete failed in the
|
||||||
|
// last flush sent after their session went down, for the next flush to
|
||||||
|
// write after it retries the delete. They are not in batch, so they do
|
||||||
|
// not count toward a full batch: a delete that keeps failing is retried
|
||||||
|
// at the pace of the flush timer.
|
||||||
|
held []prefixUpdate
|
||||||
|
// peersDown holds, each once, the peers whose BGP session went down
|
||||||
|
// since the batch was taken, and those whose route delete failed in the
|
||||||
|
// last flush.
|
||||||
|
peersDown []string
|
||||||
// writing is how many updates the flush goroutine took and has not
|
// writing is how many updates the flush goroutine took and has not
|
||||||
// finished writing.
|
// finished writing.
|
||||||
writing int
|
writing int
|
||||||
// writingTableChanges is how many changes the flush goroutine took from
|
// deleting is how many peers from peersDown the flush goroutine took and
|
||||||
// the routing table and has not finished writing.
|
// has not yet deleted the routes of.
|
||||||
writingTableChanges int
|
deleting int
|
||||||
// batchTaken is signalled when the flush goroutine takes batch; its L is
|
// batchTaken is signalled when the flush goroutine takes batch; its L is
|
||||||
// &mu.
|
// &mu.
|
||||||
batchTaken sync.Cond
|
batchTaken sync.Cond
|
||||||
batchFullCh chan struct{} // wakes the flush goroutine when batch is full
|
batchFullCh chan struct{} // wakes the flush goroutine when batch is full
|
||||||
|
lastFlush time.Time // used only by the flush goroutine
|
||||||
// stopCh hands Stop's context to the flush goroutine. It holds one, so
|
// stopCh hands Stop's context to the flush goroutine. It holds one, so
|
||||||
// Stop does not wait for a write in progress to hand it over.
|
// Stop does not wait for a write in progress to hand it over.
|
||||||
stopCh chan context.Context
|
stopCh chan context.Context
|
||||||
done chan struct{} // closed when the flush goroutine returns
|
done chan struct{} // closed when the flush goroutine returns
|
||||||
}
|
}
|
||||||
|
|
||||||
// seenPrefix is what the batch keeps of an update: the prefix it names and its
|
|
||||||
// time. A flush needs only these to store each new prefix's first-seen time.
|
|
||||||
type seenPrefix struct {
|
|
||||||
prefix string
|
|
||||||
timestamp time.Time
|
|
||||||
}
|
|
||||||
|
|
||||||
// prefixUpdate is one prefix of an update message, as applyToTable applies it
|
|
||||||
// to the routing table.
|
|
||||||
type prefixUpdate struct {
|
type prefixUpdate struct {
|
||||||
prefix string
|
prefix string
|
||||||
originASN int
|
originASN int
|
||||||
@@ -103,45 +97,21 @@ type prefixUpdate struct {
|
|||||||
endsInSet bool
|
endsInSet bool
|
||||||
}
|
}
|
||||||
|
|
||||||
// announcesRoute says whether u gives the peer a route to the prefix: an
|
// NewPrefixHandler creates a new batched prefix tracking handler
|
||||||
// announcement whose path gives an origin, or ends in an AS set of several
|
func NewPrefixHandler(db database.Store, logger *logger.Logger) *PrefixHandler {
|
||||||
// members, which gives origin 0 (none). Any other update removes the peer's
|
|
||||||
// route for the prefix: a withdrawal, or an announcement whose path is empty
|
|
||||||
// or holds AS 0 anywhere (the decoder empties such a path). Such an
|
|
||||||
// announcement replaces the peer's route, as any announcement does, with none
|
|
||||||
// (RFC 7607 treats AS 0 in a path as a withdrawal).
|
|
||||||
func (u prefixUpdate) announcesRoute() bool {
|
|
||||||
return u.messageType == "announcement" && (u.originASN > 0 || u.endsInSet)
|
|
||||||
}
|
|
||||||
|
|
||||||
// NewPrefixHandler creates a new batched prefix tracking handler, which applies
|
|
||||||
// the updates to table, the routing table read from db at startup.
|
|
||||||
func NewPrefixHandler(
|
|
||||||
db database.Store, logger *logger.Logger, table *routetable.Table,
|
|
||||||
) *PrefixHandler {
|
|
||||||
return newPrefixHandler(db, logger, table, prefixBatchTimeout)
|
|
||||||
}
|
|
||||||
|
|
||||||
// newPrefixHandler is NewPrefixHandler with the flush timer's interval,
|
|
||||||
// batchTimeout, in place of prefixBatchTimeout. A test that the timer must not
|
|
||||||
// disturb gives an interval longer than the test runs.
|
|
||||||
func newPrefixHandler(
|
|
||||||
db database.Store, logger *logger.Logger, table *routetable.Table,
|
|
||||||
batchTimeout time.Duration,
|
|
||||||
) *PrefixHandler {
|
|
||||||
h := &PrefixHandler{
|
h := &PrefixHandler{
|
||||||
db: db,
|
db: db,
|
||||||
logger: logger,
|
logger: logger,
|
||||||
table: table,
|
batch: make([]prefixUpdate, 0, prefixBatchSize),
|
||||||
batch: make([]seenPrefix, 0, prefixBatchSize),
|
|
||||||
batchFullCh: make(chan struct{}, 1),
|
batchFullCh: make(chan struct{}, 1),
|
||||||
|
lastFlush: time.Now(),
|
||||||
stopCh: make(chan context.Context, 1),
|
stopCh: make(chan context.Context, 1),
|
||||||
done: make(chan struct{}),
|
done: make(chan struct{}),
|
||||||
}
|
}
|
||||||
h.batchTaken.L = &h.mu
|
h.batchTaken.L = &h.mu
|
||||||
|
|
||||||
// Start the flush timer goroutine
|
// Start the flush timer goroutine
|
||||||
go flushLoop(batchTimeout, h.batchFullCh, h.stopCh, h.done, h.flushBatch)
|
go h.flushLoop()
|
||||||
|
|
||||||
return h
|
return h
|
||||||
}
|
}
|
||||||
@@ -170,7 +140,7 @@ func (h *PrefixHandler) QueueCapacity() int {
|
|||||||
func (h *PrefixHandler) HandleMessage(msg *ristypes.RISMessage) {
|
func (h *PrefixHandler) HandleMessage(msg *ristypes.RISMessage) {
|
||||||
if msg.Type == ristypes.MessageTypeState {
|
if msg.Type == ristypes.MessageTypeState {
|
||||||
if msg.State == ristypes.StateDown {
|
if msg.State == ristypes.StateDown {
|
||||||
h.table.PeerDown(msg.Peer)
|
h.peerSessionDown(msg.Peer)
|
||||||
}
|
}
|
||||||
|
|
||||||
return
|
return
|
||||||
@@ -195,21 +165,18 @@ func (h *PrefixHandler) HandleMessage(msg *ristypes.RISMessage) {
|
|||||||
h.batchTaken.Wait()
|
h.batchTaken.Wait()
|
||||||
}
|
}
|
||||||
|
|
||||||
// Withdrawals are applied to the routing table before announcements, so
|
// Withdrawals are queued before announcements, because the batch keeps
|
||||||
// a message that lists a prefix as both withdrawn and announced keeps the
|
// each peer's last update for a prefix: a message that lists a prefix as
|
||||||
// route (RFC 4271 section 4.3). A withdrawal names only the prefix and
|
// both withdrawn and announced keeps the route (RFC 4271 section 4.3).
|
||||||
// peer: the message's AS path, and so its origin AS, belongs to the
|
// A withdrawal names only the prefix and peer: the message's AS path, and
|
||||||
// announced prefixes.
|
// so its origin AS, belongs to the announced prefixes.
|
||||||
for _, prefix := range msg.Withdrawals {
|
for _, prefix := range msg.Withdrawals {
|
||||||
update := prefixUpdate{
|
h.batch = append(h.batch, prefixUpdate{
|
||||||
prefix: canonicalPrefix(prefix),
|
prefix: canonicalPrefix(prefix),
|
||||||
peer: msg.Peer,
|
peer: msg.Peer,
|
||||||
messageType: "withdrawal",
|
messageType: "withdrawal",
|
||||||
timestamp: timestamp,
|
timestamp: timestamp,
|
||||||
}
|
})
|
||||||
h.batch = append(h.batch, seenPrefix{update.prefix, update.timestamp})
|
|
||||||
h.applyToTable(update)
|
|
||||||
|
|
||||||
// Record withdrawal in metrics
|
// Record withdrawal in metrics
|
||||||
if h.metrics != nil {
|
if h.metrics != nil {
|
||||||
h.metrics.RecordWithdrawal()
|
h.metrics.RecordWithdrawal()
|
||||||
@@ -218,7 +185,7 @@ func (h *PrefixHandler) HandleMessage(msg *ristypes.RISMessage) {
|
|||||||
|
|
||||||
for _, announcement := range msg.Announcements {
|
for _, announcement := range msg.Announcements {
|
||||||
for _, prefix := range announcement.Prefixes {
|
for _, prefix := range announcement.Prefixes {
|
||||||
update := prefixUpdate{
|
h.batch = append(h.batch, prefixUpdate{
|
||||||
prefix: canonicalPrefix(prefix),
|
prefix: canonicalPrefix(prefix),
|
||||||
originASN: originASN,
|
originASN: originASN,
|
||||||
peer: msg.Peer,
|
peer: msg.Peer,
|
||||||
@@ -226,10 +193,7 @@ func (h *PrefixHandler) HandleMessage(msg *ristypes.RISMessage) {
|
|||||||
timestamp: timestamp,
|
timestamp: timestamp,
|
||||||
path: path,
|
path: path,
|
||||||
endsInSet: len(msg.Path.Set) > 0,
|
endsInSet: len(msg.Path.Set) > 0,
|
||||||
}
|
})
|
||||||
h.batch = append(h.batch, seenPrefix{update.prefix, update.timestamp})
|
|
||||||
h.applyToTable(update)
|
|
||||||
|
|
||||||
// Record announcement in metrics
|
// Record announcement in metrics
|
||||||
if h.metrics != nil {
|
if h.metrics != nil {
|
||||||
h.metrics.RecordAnnouncement()
|
h.metrics.RecordAnnouncement()
|
||||||
@@ -248,13 +212,12 @@ func (h *PrefixHandler) HandleMessage(msg *ristypes.RISMessage) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// Stop stops the flush goroutine, which makes one last flush before it returns,
|
// Stop stops the flush goroutine, which writes the updates and peer downs the
|
||||||
// and waits for it until ctx ends. That flush writes at most
|
// handler holds before it returns, and waits for it until ctx ends. Once ctx
|
||||||
// maxTableChangesPerWrite of the routing table's changes, and the prefixes of
|
// has ended the flush goroutine skips that last write, and a write still
|
||||||
// the updates the handler holds. Once ctx has ended the flush goroutine skips
|
// running is lost when the process exits. A peer whose route delete fails in
|
||||||
// that last flush, and a write still running is lost when the process exits. A
|
// the last write is not retried either. Stop logs how many updates are not
|
||||||
// failed write of the table's changes is not retried either. Stop logs how many
|
// written and how many peer downs have not had their routes deleted.
|
||||||
// updates and how many of the table's changes are not written.
|
|
||||||
func (h *PrefixHandler) Stop(ctx context.Context) {
|
func (h *PrefixHandler) Stop(ctx context.Context) {
|
||||||
h.stopCh <- ctx
|
h.stopCh <- ctx
|
||||||
|
|
||||||
@@ -264,156 +227,341 @@ func (h *PrefixHandler) Stop(ctx context.Context) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
h.mu.Lock()
|
h.mu.Lock()
|
||||||
unwritten := h.writing + len(h.batch)
|
unwritten := h.writing + len(h.batch) + len(h.held)
|
||||||
tableChanges := h.writingTableChanges + h.table.ChangeCount()
|
peersDown := h.deleting + len(h.peersDown)
|
||||||
h.mu.Unlock()
|
h.mu.Unlock()
|
||||||
|
|
||||||
if unwritten == 0 && tableChanges == 0 {
|
if unwritten == 0 && peersDown == 0 {
|
||||||
return
|
return
|
||||||
}
|
}
|
||||||
|
|
||||||
if ctx.Err() != nil {
|
if ctx.Err() != nil {
|
||||||
h.logger.Error("Stop deadline passed before the prefix handler wrote "+
|
h.logger.Error("Stop deadline passed before the prefix handler wrote "+
|
||||||
"its updates", "updates", unwritten, "table_changes", tableChanges)
|
"its updates", "updates", unwritten, "peers_down", peersDown)
|
||||||
|
|
||||||
return
|
return
|
||||||
}
|
}
|
||||||
|
|
||||||
h.logger.Error("The prefix handler stopped before it wrote everything",
|
h.logger.Error("The prefix handler stopped without deleting the routes "+
|
||||||
"updates", unwritten, "table_changes", tableChanges)
|
"of peers whose session went down", "updates", unwritten,
|
||||||
|
"peers_down", peersDown)
|
||||||
}
|
}
|
||||||
|
|
||||||
// takeBatch returns the pending updates and leaves an empty batch in their
|
// peerSessionDown notes that the peer's BGP session went down, so that the next
|
||||||
// place for HandleMessage to fill. The updates count in writing until
|
// flush deletes all of the peer's live routes before it writes the batch. The
|
||||||
// flushBatch has written them.
|
// peer's updates still in the batch or held came before the down, which
|
||||||
func (h *PrefixHandler) takeBatch() []seenPrefix {
|
// replaces them, so they are dropped; updates the peer sends later stay in the
|
||||||
|
// batch and are written after the delete. A peer already in peersDown is not
|
||||||
|
// added again: one delete removes all of its routes.
|
||||||
|
func (h *PrefixHandler) peerSessionDown(peer string) {
|
||||||
h.mu.Lock()
|
h.mu.Lock()
|
||||||
defer h.mu.Unlock()
|
defer h.mu.Unlock()
|
||||||
|
|
||||||
if len(h.batch) == 0 {
|
isPeers := func(update prefixUpdate) bool { return update.peer == peer }
|
||||||
return nil
|
h.batch = slices.DeleteFunc(h.batch, isPeers)
|
||||||
|
h.held = slices.DeleteFunc(h.held, isPeers)
|
||||||
|
|
||||||
|
if !slices.Contains(h.peersDown, peer) {
|
||||||
|
h.peersDown = append(h.peersDown, peer)
|
||||||
}
|
}
|
||||||
|
|
||||||
batch := h.batch
|
|
||||||
h.writing = len(batch)
|
|
||||||
h.batch = make([]seenPrefix, 0, prefixBatchSize)
|
|
||||||
h.batchTaken.Broadcast()
|
|
||||||
|
|
||||||
return batch
|
|
||||||
}
|
}
|
||||||
|
|
||||||
// flushBatch writes the routing table's changes, then takes the pending batch
|
// flushLoop runs in a goroutine, the only one that writes the batch: it
|
||||||
// and stores the prefixes it names, without holding the mutex during the
|
// flushes when the batch is full, periodically, and once more on Stop.
|
||||||
// writes.
|
func (h *PrefixHandler) flushLoop() {
|
||||||
func (h *PrefixHandler) flushBatch() {
|
defer close(h.done)
|
||||||
h.writeTableChanges()
|
|
||||||
|
|
||||||
batch := h.takeBatch()
|
ticker := time.NewTicker(prefixBatchTimeout)
|
||||||
if len(batch) == 0 {
|
defer ticker.Stop()
|
||||||
|
|
||||||
|
for {
|
||||||
|
select {
|
||||||
|
case <-ticker.C:
|
||||||
|
if time.Since(h.lastFlush) >= prefixBatchTimeout {
|
||||||
|
h.flushBatch()
|
||||||
|
}
|
||||||
|
case <-h.batchFullCh:
|
||||||
|
h.flushBatch()
|
||||||
|
case ctx := <-h.stopCh:
|
||||||
|
// Final flush, unless the stop deadline has passed
|
||||||
|
if ctx.Err() == nil {
|
||||||
|
h.flushBatch()
|
||||||
|
}
|
||||||
|
|
||||||
|
return
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// takeBatch returns the pending updates, the held ones first, and the peers in
|
||||||
|
// peersDown, and leaves an empty batch, no held updates and no peers in their
|
||||||
|
// place for HandleMessage to fill. The updates count in writing until
|
||||||
|
// flushBatch has written them, and the peers in deleting until their routes
|
||||||
|
// are deleted.
|
||||||
|
func (h *PrefixHandler) takeBatch() ([]prefixUpdate, []string) {
|
||||||
|
h.mu.Lock()
|
||||||
|
defer h.mu.Unlock()
|
||||||
|
|
||||||
|
if len(h.held) == 0 && len(h.batch) == 0 && len(h.peersDown) == 0 {
|
||||||
|
return nil, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
batch, peersDown := h.batch, h.peersDown
|
||||||
|
if len(h.held) > 0 {
|
||||||
|
batch = slices.Concat(h.held, h.batch)
|
||||||
|
}
|
||||||
|
|
||||||
|
h.writing = len(batch)
|
||||||
|
h.deleting = len(peersDown)
|
||||||
|
h.held = nil
|
||||||
|
h.batch = make([]prefixUpdate, 0, prefixBatchSize)
|
||||||
|
h.peersDown = nil
|
||||||
|
h.batchTaken.Broadcast()
|
||||||
|
|
||||||
|
return batch, peersDown
|
||||||
|
}
|
||||||
|
|
||||||
|
// flushBatch takes the pending batch and writes it to the database, without
|
||||||
|
// holding the mutex during the write.
|
||||||
|
func (h *PrefixHandler) flushBatch() {
|
||||||
|
batch, peersDown := h.takeBatch()
|
||||||
|
if len(batch) == 0 && len(peersDown) == 0 {
|
||||||
return
|
return
|
||||||
}
|
}
|
||||||
|
|
||||||
startTime := time.Now()
|
startTime := time.Now()
|
||||||
|
batchSize := len(batch)
|
||||||
|
|
||||||
|
// The routes of a peer whose session went down are deleted first: the
|
||||||
|
// batch holds only the updates the peer sent after the down. A peer whose
|
||||||
|
// delete failed is kept, with those updates, for the next flush to retry.
|
||||||
|
failedPeers := h.deleteLiveRoutesOfPeers(peersDown)
|
||||||
|
if len(failedPeers) > 0 {
|
||||||
|
batch = h.keepForNextFlush(batch, failedPeers)
|
||||||
|
}
|
||||||
|
|
||||||
|
// For each prefix and peer, keep only the last update the feed delivered.
|
||||||
|
// Updates from different peers for the same prefix are all kept. Arrival
|
||||||
|
// order decides, not the timestamp: timestamps are whole seconds, so
|
||||||
|
// updates within one second carry the same one.
|
||||||
|
type prefixAndPeer struct{ prefix, peer string }
|
||||||
|
|
||||||
|
lastUpdates := make(map[prefixAndPeer]prefixUpdate)
|
||||||
|
|
||||||
// Each prefix the batch names, announced or withdrawn, with the earliest
|
// Each prefix the batch names, announced or withdrawn, with the earliest
|
||||||
// timestamp among its updates: the prefix tables store only first_seen,
|
// timestamp among all of its updates, not only each peer's last one: the
|
||||||
// written when the prefix is new and never changed after.
|
// prefix tables store only first_seen, written when the prefix is new and
|
||||||
|
// never changed after.
|
||||||
prefixesToUpdate := make(map[string]time.Time)
|
prefixesToUpdate := make(map[string]time.Time)
|
||||||
|
|
||||||
for _, update := range batch {
|
for _, update := range batch {
|
||||||
|
lastUpdates[prefixAndPeer{update.prefix, update.peer}] = update
|
||||||
|
|
||||||
seen, exists := prefixesToUpdate[update.prefix]
|
seen, exists := prefixesToUpdate[update.prefix]
|
||||||
if !exists || update.timestamp.Before(seen) {
|
if !exists || update.timestamp.Before(seen) {
|
||||||
prefixesToUpdate[update.prefix] = update.timestamp
|
prefixesToUpdate[update.prefix] = update.timestamp
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
err := h.db.UpdatePrefixesBatch(prefixesToUpdate)
|
// Collect routes to upsert and delete
|
||||||
if err != nil {
|
var (
|
||||||
h.logger.Error("Failed to update prefix batch",
|
routesToUpsert []*database.LiveRoute
|
||||||
"error", err, "count", len(prefixesToUpdate))
|
routesToDelete []database.LiveRouteDeletion
|
||||||
|
)
|
||||||
|
|
||||||
|
for _, update := range lastUpdates {
|
||||||
|
// An announcement is stored when its path gives an origin, or when it
|
||||||
|
// ends in an AS set of several members: that route is stored with
|
||||||
|
// origin 0 (none). Any other update deletes the peer's route for the
|
||||||
|
// prefix: a withdrawal, or an announcement whose path is empty or
|
||||||
|
// holds AS 0 anywhere (the decoder empties such a path). Such an
|
||||||
|
// announcement replaces the peer's route, as any announcement does,
|
||||||
|
// with none (RFC 7607 treats AS 0 in a path as a withdrawal).
|
||||||
|
if update.messageType == "announcement" &&
|
||||||
|
(update.originASN > 0 || update.endsInSet) {
|
||||||
|
// Create live route for batch upsert
|
||||||
|
route := h.createLiveRoute(update)
|
||||||
|
if route != nil {
|
||||||
|
routesToUpsert = append(routesToUpsert, route)
|
||||||
|
}
|
||||||
|
} else {
|
||||||
|
// Parse CIDR to get IP version
|
||||||
|
_, ipVersion, err := parseCIDR(update.prefix)
|
||||||
|
if err != nil {
|
||||||
|
h.logger.Error("Failed to parse CIDR for route deletion",
|
||||||
|
"prefix", update.prefix, "error", err)
|
||||||
|
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
|
||||||
|
// Create deletion record for batch delete
|
||||||
|
routesToDelete = append(routesToDelete, database.LiveRouteDeletion{
|
||||||
|
Prefix: update.prefix,
|
||||||
|
PeerIP: update.peer,
|
||||||
|
IPVersion: ipVersion,
|
||||||
|
})
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
successCount := h.writeBatch(routesToUpsert, routesToDelete, prefixesToUpdate)
|
||||||
|
|
||||||
h.mu.Lock()
|
h.mu.Lock()
|
||||||
h.writing = 0
|
h.writing = 0
|
||||||
h.mu.Unlock()
|
h.mu.Unlock()
|
||||||
|
|
||||||
|
elapsed := time.Since(startTime)
|
||||||
h.logger.Debug("Flushed prefix batch",
|
h.logger.Debug("Flushed prefix batch",
|
||||||
"batch_size", len(batch),
|
"batch_size", batchSize,
|
||||||
|
"peers_down", len(peersDown),
|
||||||
"unique_prefixes", len(prefixesToUpdate),
|
"unique_prefixes", len(prefixesToUpdate),
|
||||||
"duration_ms", time.Since(startTime).Milliseconds(),
|
"success", successCount,
|
||||||
|
"duration_ms", elapsed.Milliseconds(),
|
||||||
)
|
)
|
||||||
|
|
||||||
|
h.lastFlush = time.Now()
|
||||||
}
|
}
|
||||||
|
|
||||||
// applyToTable applies update to the routing table in memory: as an
|
// deleteLiveRoutesOfPeers deletes all live routes of each of peers, the peers
|
||||||
// announcement when it gives the peer a route, otherwise as a withdrawal. A
|
// whose BGP session went down, and returns those whose delete failed. Most
|
||||||
// prefix that does not parse is logged and left out. Each announcement counts
|
// downs come from peers that have no routes, so only a delete that removed
|
||||||
// in the IPv4 or IPv6 route update rate.
|
// routes is logged. A peer whose delete succeeded no longer counts in deleting;
|
||||||
func (h *PrefixHandler) applyToTable(update prefixUpdate) {
|
// a failed one counts until keepForNextFlush puts it back in peersDown.
|
||||||
prefix, err := netip.ParsePrefix(update.prefix)
|
func (h *PrefixHandler) deleteLiveRoutesOfPeers(peers []string) []string {
|
||||||
if err != nil {
|
var failed []string
|
||||||
h.logger.Error("Failed to parse prefix",
|
|
||||||
"prefix", update.prefix, "error", err)
|
|
||||||
|
|
||||||
return
|
for _, peer := range peers {
|
||||||
}
|
deleted, err := h.db.DeleteLiveRoutesOfPeer(peer)
|
||||||
|
if err != nil {
|
||||||
|
h.logger.Error(
|
||||||
|
"Failed to delete the live routes of a peer whose session "+
|
||||||
|
"went down; the next flush retries",
|
||||||
|
"peer", peer, "routes_deleted", deleted, "error", err)
|
||||||
|
|
||||||
if !update.announcesRoute() {
|
failed = append(failed, peer)
|
||||||
h.table.Withdraw(prefix, update.peer)
|
|
||||||
|
|
||||||
return
|
continue
|
||||||
}
|
}
|
||||||
|
|
||||||
if h.metrics != nil {
|
h.mu.Lock()
|
||||||
if prefix.Addr().Is4() {
|
h.deleting--
|
||||||
h.metrics.RecordIPv4Update()
|
h.mu.Unlock()
|
||||||
} else {
|
|
||||||
h.metrics.RecordIPv6Update()
|
if deleted > 0 {
|
||||||
|
h.logger.Info(
|
||||||
|
"Deleted the live routes of a peer whose session went down",
|
||||||
|
"peer", peer, "routes", deleted)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
h.table.Announce(prefix, update.originASN, routetable.Route{
|
return failed
|
||||||
Peer: update.peer,
|
|
||||||
ASPath: update.path,
|
|
||||||
Time: update.timestamp,
|
|
||||||
})
|
|
||||||
}
|
}
|
||||||
|
|
||||||
// writeTableChanges writes at most maxTableChangesPerWrite changes of the
|
// keepForNextFlush puts the peers whose route delete failed back in peersDown,
|
||||||
// routing table in memory, and every peer the table numbered since the last
|
// and their updates from batch in held, so that the next flush retries each
|
||||||
// write.
|
// delete before it writes the peer's updates; written now, they would be
|
||||||
// When the write fails, they go back to the table, so that a later write writes
|
// deleted by the retry. The updates of a peer whose session went down again
|
||||||
// them again, each pair with its content then.
|
// since the batch was taken came before that down, so they are dropped, as
|
||||||
func (h *PrefixHandler) writeTableChanges() {
|
// peerSessionDown drops them, and the peer is already in peersDown. It returns
|
||||||
h.mu.Lock()
|
// the updates of the other peers, for this flush to write.
|
||||||
changes, newPeers := h.table.TakeChanges(maxTableChangesPerWrite)
|
//
|
||||||
h.writingTableChanges = len(changes)
|
// While a delete keeps failing, the held updates would grow with every flush,
|
||||||
h.mu.Unlock()
|
// so only the last prefixBatchSize of them are kept and the older ones are
|
||||||
|
// dropped. This never leaves in place a route the peer withdrew: a prefix with
|
||||||
|
// a kept update keeps its last one, and the retry deletes all of the peer's
|
||||||
|
// routes, so a prefix whose updates were all dropped has no route from the peer
|
||||||
|
// until the peer announces it again.
|
||||||
|
func (h *PrefixHandler) keepForNextFlush(
|
||||||
|
batch []prefixUpdate, failedPeers []string,
|
||||||
|
) []prefixUpdate {
|
||||||
|
var kept, toWrite []prefixUpdate
|
||||||
|
|
||||||
if len(changes) == 0 && len(newPeers) == 0 {
|
for _, update := range batch {
|
||||||
return
|
if slices.Contains(failedPeers, update.peer) {
|
||||||
|
kept = append(kept, update)
|
||||||
|
} else {
|
||||||
|
toWrite = append(toWrite, update)
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
err := h.db.WriteRouteChanges(changes, newPeers)
|
|
||||||
|
|
||||||
h.mu.Lock()
|
h.mu.Lock()
|
||||||
defer h.mu.Unlock()
|
defer h.mu.Unlock()
|
||||||
|
|
||||||
h.writingTableChanges = 0
|
kept = slices.DeleteFunc(kept, func(update prefixUpdate) bool {
|
||||||
|
return slices.Contains(h.peersDown, update.peer)
|
||||||
|
})
|
||||||
|
|
||||||
if err != nil {
|
if len(kept) > prefixBatchSize {
|
||||||
h.logger.Error("Failed to write the routing table's changes; the next "+
|
h.logger.Error(
|
||||||
"flush retries", "changes", len(changes), "new_peers", len(newPeers),
|
"Dropped the oldest updates of peers whose route delete failed",
|
||||||
"error", err)
|
"peers", failedPeers, "dropped", len(kept)-prefixBatchSize)
|
||||||
|
|
||||||
h.table.PutBackChanges(changes, newPeers)
|
// A copy, so that the dropped updates are freed.
|
||||||
|
kept = slices.Clone(kept[len(kept)-prefixBatchSize:])
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// takeBatch left held empty, and only this flush fills it. The kept
|
||||||
|
// updates now count in held, not in writing, and the failed peers in
|
||||||
|
// peersDown, not in deleting.
|
||||||
|
h.held = kept
|
||||||
|
h.writing = len(toWrite)
|
||||||
|
h.deleting = 0
|
||||||
|
|
||||||
|
for _, peer := range failedPeers {
|
||||||
|
if !slices.Contains(h.peersDown, peer) {
|
||||||
|
h.peersDown = append(h.peersDown, peer)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
return toWrite
|
||||||
|
}
|
||||||
|
|
||||||
|
// writeBatch writes the collected route upserts, route deletions and prefix
|
||||||
|
// updates to the database and returns how many routes it wrote.
|
||||||
|
func (h *PrefixHandler) writeBatch(
|
||||||
|
routesToUpsert []*database.LiveRoute,
|
||||||
|
routesToDelete []database.LiveRouteDeletion,
|
||||||
|
prefixesToUpdate map[string]time.Time,
|
||||||
|
) int {
|
||||||
|
successCount := 0
|
||||||
|
|
||||||
|
if len(routesToUpsert) > 0 {
|
||||||
|
err := h.db.UpsertLiveRouteBatch(routesToUpsert)
|
||||||
|
if err != nil {
|
||||||
|
h.logger.Error("Failed to upsert route batch",
|
||||||
|
"error", err, "count", len(routesToUpsert))
|
||||||
|
} else {
|
||||||
|
successCount += len(routesToUpsert)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
if len(routesToDelete) > 0 {
|
||||||
|
err := h.db.DeleteLiveRouteBatch(routesToDelete)
|
||||||
|
if err != nil {
|
||||||
|
h.logger.Error("Failed to delete route batch",
|
||||||
|
"error", err, "count", len(routesToDelete))
|
||||||
|
} else {
|
||||||
|
successCount += len(routesToDelete)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Update prefix tables
|
||||||
|
if len(prefixesToUpdate) > 0 {
|
||||||
|
err := h.db.UpdatePrefixesBatch(prefixesToUpdate)
|
||||||
|
if err != nil {
|
||||||
|
h.logger.Error("Failed to update prefix batch",
|
||||||
|
"error", err, "count", len(prefixesToUpdate))
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
return successCount
|
||||||
}
|
}
|
||||||
|
|
||||||
// canonicalPrefix returns prefix in the text form net/netip prints, the form
|
// canonicalPrefix returns prefix in the text form net/netip prints, the form
|
||||||
// the prefix tables store and the IP lookup reads them in. The feed sends IPv6
|
// the IP lookup builds when it looks a prefix up. The feed sends IPv6
|
||||||
// announcements compressed ("2001:db8::/32") but IPv6 withdrawals uncompressed
|
// announcements compressed ("2001:db8::/32") but IPv6 withdrawals uncompressed
|
||||||
// ("2001:db8:0:0:0:0:0:0/32"). A prefix that does not parse is returned
|
// ("2001:db8:0:0:0:0:0:0/32"); stored as received, a withdrawal would not match
|
||||||
// unchanged, and applyToTable logs it.
|
// the route its announcement stored. A prefix that does not parse is returned
|
||||||
|
// unchanged, and the batch flush reports it.
|
||||||
func canonicalPrefix(prefix string) string {
|
func canonicalPrefix(prefix string) string {
|
||||||
p, err := netip.ParsePrefix(prefix)
|
p, err := netip.ParsePrefix(prefix)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
@@ -422,3 +570,55 @@ func canonicalPrefix(prefix string) string {
|
|||||||
|
|
||||||
return p.Masked().String()
|
return p.Masked().String()
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// parseCIDR returns the mask length and the IP version of a prefix string
|
||||||
|
func parseCIDR(prefix string) (int, int, error) {
|
||||||
|
_, ipNet, err := net.ParseCIDR(prefix)
|
||||||
|
if err != nil {
|
||||||
|
return 0, 0, err
|
||||||
|
}
|
||||||
|
|
||||||
|
ones, _ := ipNet.Mask.Size()
|
||||||
|
if strings.Contains(prefix, ":") {
|
||||||
|
return ones, ipv6Version, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
return ones, ipv4Version, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// createLiveRoute creates a LiveRoute from a prefix update
|
||||||
|
func (h *PrefixHandler) createLiveRoute(update prefixUpdate) *database.LiveRoute {
|
||||||
|
// Parse CIDR to get mask length
|
||||||
|
maskLength, ipVersion, err := parseCIDR(update.prefix)
|
||||||
|
if err != nil {
|
||||||
|
h.logger.Error("Failed to parse CIDR",
|
||||||
|
"prefix", update.prefix,
|
||||||
|
"error", err,
|
||||||
|
)
|
||||||
|
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// Track route update metrics
|
||||||
|
if h.metrics != nil {
|
||||||
|
if ipVersion == ipv4Version {
|
||||||
|
h.metrics.RecordIPv4Update()
|
||||||
|
} else {
|
||||||
|
h.metrics.RecordIPv6Update()
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Create live route record
|
||||||
|
liveRoute := &database.LiveRoute{
|
||||||
|
Prefix: update.prefix,
|
||||||
|
MaskLength: maskLength,
|
||||||
|
IPVersion: ipVersion,
|
||||||
|
OriginASN: update.originASN,
|
||||||
|
PeerIP: update.peer,
|
||||||
|
ASPath: update.path,
|
||||||
|
NextHop: update.peer, // Using peer as next hop
|
||||||
|
LastUpdated: update.timestamp,
|
||||||
|
}
|
||||||
|
|
||||||
|
return liveRoute
|
||||||
|
}
|
||||||
|
|||||||
File diff suppressed because it is too large
Load Diff
@@ -1,607 +0,0 @@
|
|||||||
package routewatch
|
|
||||||
|
|
||||||
import (
|
|
||||||
"bytes"
|
|
||||||
"encoding/json"
|
|
||||||
"errors"
|
|
||||||
"log/slog"
|
|
||||||
"net/netip"
|
|
||||||
"reflect"
|
|
||||||
"slices"
|
|
||||||
"strings"
|
|
||||||
"testing"
|
|
||||||
"time"
|
|
||||||
|
|
||||||
"git.eeqj.de/sneak/routewatch/internal/database"
|
|
||||||
"git.eeqj.de/sneak/routewatch/internal/logger"
|
|
||||||
"git.eeqj.de/sneak/routewatch/internal/ristypes"
|
|
||||||
"git.eeqj.de/sneak/routewatch/internal/routetable"
|
|
||||||
)
|
|
||||||
|
|
||||||
// More peers and prefixes for the tests of the routing table in memory.
|
|
||||||
const (
|
|
||||||
secondPeerIP = "2001:db8:ffff::2"
|
|
||||||
thirdPeerIP = "2001:db8:ffff::3"
|
|
||||||
fourthPeerIP = "192.0.2.4"
|
|
||||||
fifthPeerIP = "192.0.2.5"
|
|
||||||
|
|
||||||
otherV4Prefix = "198.51.100.0/24"
|
|
||||||
setV4Prefix = "203.0.113.0/24"
|
|
||||||
|
|
||||||
// firstManyPrefix is the first prefix announceManyPrefixes announces.
|
|
||||||
firstManyPrefix = "10.0.0.0/32"
|
|
||||||
)
|
|
||||||
|
|
||||||
// tableWriteTimeout is how long a test waits for the flush timer to write the
|
|
||||||
// routing table's changes: a few ticks of it.
|
|
||||||
const tableWriteTimeout = 5 * prefixBatchTimeout
|
|
||||||
|
|
||||||
// tableCase is one test of the routing table in memory: the messages handed to
|
|
||||||
// the prefix handler, and for each prefix the peers that carry it with each
|
|
||||||
// origin AS after them.
|
|
||||||
type tableCase struct {
|
|
||||||
name string
|
|
||||||
messages []*ristypes.RISMessage
|
|
||||||
want map[string]map[int][]string
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestPrefixHandlerAppliesUpdatesToTheRoutingTable hands the prefix handler
|
|
||||||
// announcements, withdrawals and a peer's session going down.
|
|
||||||
func TestPrefixHandlerAppliesUpdatesToTheRoutingTable(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
announceAndWithdraw := peerAnnouncement(testPeerIP, testV4Prefix)
|
|
||||||
announceAndWithdraw.Withdrawals = []string{testV4Prefix}
|
|
||||||
|
|
||||||
checkTableCases(t, []tableCase{
|
|
||||||
{
|
|
||||||
name: "an announcement",
|
|
||||||
messages: []*ristypes.RISMessage{peerAnnouncement(testPeerIP, testV4Prefix)},
|
|
||||||
want: map[string]map[int][]string{testV4Prefix: {testASNB: {testPeerIP}}},
|
|
||||||
},
|
|
||||||
{
|
|
||||||
name: "a withdrawal",
|
|
||||||
messages: []*ristypes.RISMessage{
|
|
||||||
peerAnnouncement(testPeerIP, testV4Prefix),
|
|
||||||
peerWithdrawal(testPeerIP, testV4Prefix),
|
|
||||||
},
|
|
||||||
want: map[string]map[int][]string{testV4Prefix: {}},
|
|
||||||
},
|
|
||||||
{
|
|
||||||
// RFC 4271 section 4.3: the announcement counts, not the withdrawal.
|
|
||||||
name: "a message that announces and withdraws the prefix",
|
|
||||||
messages: []*ristypes.RISMessage{announceAndWithdraw},
|
|
||||||
want: map[string]map[int][]string{testV4Prefix: {testASNB: {testPeerIP}}},
|
|
||||||
},
|
|
||||||
{
|
|
||||||
name: "a peer's session going down",
|
|
||||||
messages: []*ristypes.RISMessage{
|
|
||||||
peerAnnouncement(testPeerIP, testV4Prefix, testV6Prefix),
|
|
||||||
peerAnnouncement(secondPeerIP, testV4Prefix),
|
|
||||||
sessionDown(),
|
|
||||||
},
|
|
||||||
want: map[string]map[int][]string{
|
|
||||||
testV4Prefix: {testASNB: {secondPeerIP}},
|
|
||||||
testV6Prefix: {},
|
|
||||||
},
|
|
||||||
},
|
|
||||||
{
|
|
||||||
name: "an announcement after the peer's session went down",
|
|
||||||
messages: []*ristypes.RISMessage{
|
|
||||||
peerAnnouncement(testPeerIP, testV4Prefix),
|
|
||||||
sessionDown(),
|
|
||||||
peerAnnouncement(testPeerIP, otherV4Prefix),
|
|
||||||
},
|
|
||||||
want: map[string]map[int][]string{
|
|
||||||
testV4Prefix: {},
|
|
||||||
otherV4Prefix: {testASNB: {testPeerIP}},
|
|
||||||
},
|
|
||||||
},
|
|
||||||
})
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestPrefixHandlerAppliesAPathWithoutAnOriginToTheRoutingTable hands the
|
|
||||||
// prefix handler announcements whose AS path gives no origin. One whose path
|
|
||||||
// is empty or holds AS 0 removes the peer's route, as a withdrawal does; one
|
|
||||||
// whose path ends in an AS set of several members is kept with origin 0.
|
|
||||||
func TestPrefixHandlerAppliesAPathWithoutAnOriginToTheRoutingTable(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
announceWithPath := func(path ristypes.ASPath) *ristypes.RISMessage {
|
|
||||||
msg := peerAnnouncement(testPeerIP, testV4Prefix)
|
|
||||||
msg.Path = path
|
|
||||||
|
|
||||||
return msg
|
|
||||||
}
|
|
||||||
|
|
||||||
checkTableCases(t, []tableCase{
|
|
||||||
{
|
|
||||||
name: "a path whose last AS is 0",
|
|
||||||
messages: []*ristypes.RISMessage{
|
|
||||||
peerAnnouncement(testPeerIP, testV4Prefix),
|
|
||||||
announceWithPath(ristypes.ASPath{Sequence: []int{testASNA, 0}}),
|
|
||||||
},
|
|
||||||
want: map[string]map[int][]string{testV4Prefix: {}},
|
|
||||||
},
|
|
||||||
{
|
|
||||||
name: "an empty path",
|
|
||||||
messages: []*ristypes.RISMessage{
|
|
||||||
peerAnnouncement(testPeerIP, testV4Prefix),
|
|
||||||
announceWithPath(ristypes.ASPath{}),
|
|
||||||
},
|
|
||||||
want: map[string]map[int][]string{testV4Prefix: {}},
|
|
||||||
},
|
|
||||||
{
|
|
||||||
name: "a path with AS 0 before its end",
|
|
||||||
messages: []*ristypes.RISMessage{
|
|
||||||
peerAnnouncement(testPeerIP, testV4Prefix),
|
|
||||||
announceWithPath(decodePath(t, testASNA, 0, testASNB)),
|
|
||||||
},
|
|
||||||
want: map[string]map[int][]string{testV4Prefix: {}},
|
|
||||||
},
|
|
||||||
{
|
|
||||||
name: "a path that ends in an AS set",
|
|
||||||
messages: []*ristypes.RISMessage{
|
|
||||||
announceWithPath(decodePath(t, testASNA, []int{testASNC, testASND})),
|
|
||||||
},
|
|
||||||
want: map[string]map[int][]string{testV4Prefix: {0: {testPeerIP}}},
|
|
||||||
},
|
|
||||||
})
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestPrefixHandlerKeepsThePathOfAPathEndingInASet announces a prefix with AS
|
|
||||||
// paths that end in an AS set. Any member of a set of several may be the
|
|
||||||
// origin, so the routing table keeps the route with origin 0 (none); a set of
|
|
||||||
// one member counts as that AS. The kept AS path lists the set's members after
|
|
||||||
// the rest of the path.
|
|
||||||
func TestPrefixHandlerKeepsThePathOfAPathEndingInASet(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
tests := []struct {
|
|
||||||
name string
|
|
||||||
path []any
|
|
||||||
wantOrigin int
|
|
||||||
wantPath []int
|
|
||||||
}{
|
|
||||||
{
|
|
||||||
name: "a set of several members",
|
|
||||||
path: []any{testASNA, testASNB, []int{testASNC, testASND}},
|
|
||||||
wantPath: []int{testASNA, testASNB, testASNC, testASND},
|
|
||||||
},
|
|
||||||
{
|
|
||||||
name: "a set of one member",
|
|
||||||
path: []any{testASNA, testASNB, []int{testASNC}},
|
|
||||||
wantOrigin: testASNC,
|
|
||||||
wantPath: []int{testASNA, testASNB, testASNC},
|
|
||||||
},
|
|
||||||
}
|
|
||||||
|
|
||||||
for _, tt := range tests {
|
|
||||||
t.Run(tt.name, func(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
msg := peerAnnouncement(testPeerIP, testV4Prefix)
|
|
||||||
msg.Path = decodePath(t, tt.path...)
|
|
||||||
|
|
||||||
h := prefixHandlerWithoutTimer(newMockStore())
|
|
||||||
h.HandleMessage(msg)
|
|
||||||
|
|
||||||
origins := h.table.Origins(netip.MustParsePrefix(testV4Prefix))
|
|
||||||
if len(origins) != 1 || origins[0].ASN != tt.wantOrigin ||
|
|
||||||
len(origins[0].Routes) != 1 ||
|
|
||||||
!slices.Equal(origins[0].Routes[0].ASPath, tt.wantPath) {
|
|
||||||
t.Errorf("Origins(%s) = %+v, want one route with origin AS%d "+
|
|
||||||
"and AS path %v", testV4Prefix, origins, tt.wantOrigin,
|
|
||||||
tt.wantPath)
|
|
||||||
}
|
|
||||||
})
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// checkTableCases hands the messages of each case to a new prefix handler and
|
|
||||||
// checks its routing table right after, with no flush.
|
|
||||||
func checkTableCases(t *testing.T, tests []tableCase) {
|
|
||||||
t.Helper()
|
|
||||||
|
|
||||||
for _, tt := range tests {
|
|
||||||
t.Run(tt.name, func(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
h := prefixHandlerWithoutTimer(newMockStore())
|
|
||||||
for _, msg := range tt.messages {
|
|
||||||
h.HandleMessage(msg)
|
|
||||||
}
|
|
||||||
|
|
||||||
for prefix, want := range tt.want {
|
|
||||||
got := tableOrigins(h.table, prefix)
|
|
||||||
if !reflect.DeepEqual(got, want) {
|
|
||||||
t.Errorf("peers of %s by origin AS = %v, want %v",
|
|
||||||
prefix, got, want)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
})
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// tableWritesStore is a real database that hands each write of the routing
|
|
||||||
// table's changes to written once it is made.
|
|
||||||
type tableWritesStore struct {
|
|
||||||
*database.Database
|
|
||||||
|
|
||||||
written chan []routetable.Change
|
|
||||||
}
|
|
||||||
|
|
||||||
func (s tableWritesStore) WriteRouteChanges(
|
|
||||||
changes []routetable.Change, newPeers map[string]uint,
|
|
||||||
) error {
|
|
||||||
err := s.Database.WriteRouteChanges(changes, newPeers)
|
|
||||||
s.written <- changes
|
|
||||||
|
|
||||||
return err
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestPrefixHandlerWritesTheTableChangesEachSecond hands a running prefix
|
|
||||||
// handler an announcement and then its withdrawal, each far short of a full
|
|
||||||
// batch. The flush timer must write the change of each to the database.
|
|
||||||
func TestPrefixHandlerWritesTheTableChangesEachSecond(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
// The two writes the test waits for, and room for one more.
|
|
||||||
const writesBuffered = 3
|
|
||||||
|
|
||||||
store := tableWritesStore{
|
|
||||||
Database: newTestDatabase(t),
|
|
||||||
written: make(chan []routetable.Change, writesBuffered),
|
|
||||||
}
|
|
||||||
h := NewPrefixHandler(store, logger.New(), routetable.New(routetable.Params{}))
|
|
||||||
|
|
||||||
defer h.Stop(t.Context())
|
|
||||||
|
|
||||||
prefix := netip.MustParsePrefix(testV4Prefix)
|
|
||||||
|
|
||||||
for _, step := range []struct {
|
|
||||||
msg *ristypes.RISMessage
|
|
||||||
wantRemoved bool
|
|
||||||
}{
|
|
||||||
{msg: peerAnnouncement(testPeerIP, testV4Prefix)},
|
|
||||||
{msg: peerWithdrawal(testPeerIP, testV4Prefix), wantRemoved: true},
|
|
||||||
} {
|
|
||||||
h.HandleMessage(step.msg)
|
|
||||||
|
|
||||||
select {
|
|
||||||
case changes := <-store.written:
|
|
||||||
if len(changes) != 1 || changes[0].Prefix != prefix ||
|
|
||||||
changes[0].OriginASN != testASNB ||
|
|
||||||
changes[0].Removed != step.wantRemoved {
|
|
||||||
t.Fatalf("wrote %+v, want the pair of %s and AS%d with "+
|
|
||||||
"removed %t", changes, prefix, testASNB, step.wantRemoved)
|
|
||||||
}
|
|
||||||
case <-time.After(tableWriteTimeout):
|
|
||||||
t.Fatalf("the routing table's change was not written within %s",
|
|
||||||
tableWriteTimeout)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// errTestTableWrite is the error a test store's failing write of the routing
|
|
||||||
// table's changes returns.
|
|
||||||
var errTestTableWrite = errors.New("test: the write of the table's changes failed")
|
|
||||||
|
|
||||||
// firstTableWriteFailsStore is a real database whose first write of the
|
|
||||||
// routing table's changes fails, writing nothing. writes counts the writes
|
|
||||||
// asked for.
|
|
||||||
type firstTableWriteFailsStore struct {
|
|
||||||
*database.Database
|
|
||||||
|
|
||||||
writes int
|
|
||||||
}
|
|
||||||
|
|
||||||
func (s *firstTableWriteFailsStore) WriteRouteChanges(
|
|
||||||
changes []routetable.Change, newPeers map[string]uint,
|
|
||||||
) error {
|
|
||||||
s.writes++
|
|
||||||
if s.writes == 1 {
|
|
||||||
return errTestTableWrite
|
|
||||||
}
|
|
||||||
|
|
||||||
return s.Database.WriteRouteChanges(changes, newPeers)
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestPrefixHandlerWritesAFailedTableWriteAgain runs two flushes into a
|
|
||||||
// database whose first write of the routing table's changes fails. Before the
|
|
||||||
// first, two new peers announce a prefix each; before the second, one of them
|
|
||||||
// announces its prefix again with another path to the same origin AS. The
|
|
||||||
// second write must hold the changes and new peers of the first, with the later
|
|
||||||
// path, so the database holds the table that is in memory.
|
|
||||||
func TestPrefixHandlerWritesAFailedTableWriteAgain(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
db := newTestDatabase(t)
|
|
||||||
store := &firstTableWriteFailsStore{Database: db}
|
|
||||||
h := prefixHandlerWithoutTimer(store)
|
|
||||||
|
|
||||||
first := peerAnnouncement(testPeerIP, testV4Prefix)
|
|
||||||
h.HandleMessage(first)
|
|
||||||
h.HandleMessage(peerAnnouncement(secondPeerIP, otherV4Prefix))
|
|
||||||
h.flushBatch()
|
|
||||||
|
|
||||||
laterPath := []int{testASNC, testASNB}
|
|
||||||
later := peerAnnouncement(testPeerIP, testV4Prefix)
|
|
||||||
later.Path = ristypes.ASPath{Sequence: laterPath}
|
|
||||||
later.ParsedTimestamp = first.ParsedTimestamp.Add(secondUpdateAfter)
|
|
||||||
h.HandleMessage(later)
|
|
||||||
h.flushBatch()
|
|
||||||
|
|
||||||
if store.writes != 2 {
|
|
||||||
t.Fatalf("the store was asked for %d writes of the table's changes, "+
|
|
||||||
"want 2", store.writes)
|
|
||||||
}
|
|
||||||
|
|
||||||
read, err := db.ReadRouteTable(t.Context())
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("ReadRouteTable: %v", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
checkSameTable(t, read, h.table, testV4Prefix, otherV4Prefix)
|
|
||||||
|
|
||||||
origins := read.Origins(netip.MustParsePrefix(testV4Prefix))
|
|
||||||
if len(origins) != 1 || len(origins[0].Routes) != 1 ||
|
|
||||||
!slices.Equal(origins[0].Routes[0].ASPath, laterPath) {
|
|
||||||
t.Errorf("Origins(%s) = %+v, want one route with the path %v",
|
|
||||||
testV4Prefix, origins, laterPath)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// tableWriteSizesStore is a real database that records how many changes and
|
|
||||||
// how many new peers each write of the routing table's changes holds.
|
|
||||||
type tableWriteSizesStore struct {
|
|
||||||
*database.Database
|
|
||||||
|
|
||||||
changes []int
|
|
||||||
newPeers []int
|
|
||||||
}
|
|
||||||
|
|
||||||
func (s *tableWriteSizesStore) WriteRouteChanges(
|
|
||||||
changes []routetable.Change, newPeers map[string]uint,
|
|
||||||
) error {
|
|
||||||
s.changes = append(s.changes, len(changes))
|
|
||||||
s.newPeers = append(s.newPeers, len(newPeers))
|
|
||||||
|
|
||||||
return s.Database.WriteRouteChanges(changes, newPeers)
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestPrefixHandlerWritesManyTableChangesOverSeveralFlushes has one new peer
|
|
||||||
// change one and a half times as many prefix and origin AS pairs as one write
|
|
||||||
// takes, then flushes three times. The first write must hold as many pairs as
|
|
||||||
// one write takes, and the peer. The second must hold the other pairs, and the
|
|
||||||
// third flush must write nothing. The database must then hold the table in
|
|
||||||
// memory.
|
|
||||||
func TestPrefixHandlerWritesManyTableChangesOverSeveralFlushes(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
const (
|
|
||||||
pairs = maxTableChangesPerWrite * 3 / 2
|
|
||||||
flushes = 3
|
|
||||||
)
|
|
||||||
|
|
||||||
db := newTestDatabase(t)
|
|
||||||
store := &tableWriteSizesStore{Database: db}
|
|
||||||
h := prefixHandlerWithoutTimer(store)
|
|
||||||
announceManyPrefixes(h.table, pairs)
|
|
||||||
|
|
||||||
for range flushes {
|
|
||||||
h.flushBatch()
|
|
||||||
}
|
|
||||||
|
|
||||||
wantChanges := []int{maxTableChangesPerWrite, pairs - maxTableChangesPerWrite}
|
|
||||||
wantNewPeers := []int{1, 0}
|
|
||||||
|
|
||||||
if !slices.Equal(store.changes, wantChanges) ||
|
|
||||||
!slices.Equal(store.newPeers, wantNewPeers) {
|
|
||||||
t.Errorf("the writes held %v pairs and %v new peers, want %v and %v",
|
|
||||||
store.changes, store.newPeers, wantChanges, wantNewPeers)
|
|
||||||
}
|
|
||||||
|
|
||||||
read, err := db.ReadRouteTable(t.Context())
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("ReadRouteTable: %v", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
checkSameTable(t, read, h.table, firstManyPrefix)
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestPrefixHandlerStopLogsTheTableChangesLeftUnwritten stops a prefix handler,
|
|
||||||
// with no deadline, whose routing table holds more changed pairs than one write
|
|
||||||
// takes. Its last flush writes one write's worth, so Stop must log the pairs
|
|
||||||
// that no write took, and must not say that the stop deadline passed. The
|
|
||||||
// flush timer must not fire, or its writes could leave no pairs for Stop to
|
|
||||||
// log.
|
|
||||||
func TestPrefixHandlerStopLogsTheTableChangesLeftUnwritten(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
const pairs = maxTableChangesPerWrite + 1
|
|
||||||
|
|
||||||
var logs bytes.Buffer
|
|
||||||
|
|
||||||
store := &tableWriteSizesStore{Database: newTestDatabase(t)}
|
|
||||||
table := routetable.New(routetable.Params{})
|
|
||||||
announceManyPrefixes(table, pairs)
|
|
||||||
|
|
||||||
h := newPrefixHandler(store, newErrorLogger(&logs), table, longerThanAnyTest)
|
|
||||||
h.Stop(t.Context())
|
|
||||||
|
|
||||||
written := 0
|
|
||||||
for _, n := range store.changes {
|
|
||||||
written += n
|
|
||||||
}
|
|
||||||
|
|
||||||
logged := lastLogged(t, &logs)
|
|
||||||
if strings.Contains(logged.Msg, "deadline") {
|
|
||||||
t.Errorf("Stop logged %q although its deadline had not passed",
|
|
||||||
logged.Msg)
|
|
||||||
}
|
|
||||||
|
|
||||||
if logged.TableChanges != pairs-written {
|
|
||||||
t.Errorf("Stop logged %d table changes not written, want %d",
|
|
||||||
logged.TableChanges, pairs-written)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// readLog holds what readRouteTable logs.
|
|
||||||
//
|
|
||||||
//nolint:tagliatelle // field names are the keys readRouteTable logs
|
|
||||||
type readLog struct {
|
|
||||||
Msg string `json:"msg"`
|
|
||||||
Pairs int `json:"pairs"`
|
|
||||||
Peers int `json:"peers"`
|
|
||||||
DurationMs *int64 `json:"duration_ms"`
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestARestartReadsTheRoutingTableBack runs announcements, a withdrawal and a
|
|
||||||
// peer's session going down through a prefix handler into a database, then
|
|
||||||
// reads the table back as startup does. The table read must answer as the one
|
|
||||||
// in memory, and the log must give its pairs and peers. A new handler on the
|
|
||||||
// table read then takes later updates and writes them as it stops: the
|
|
||||||
// database must then hold what the first table holds after the same updates.
|
|
||||||
func TestARestartReadsTheRoutingTableBack(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
prefixes := []string{testV4Prefix, testV6Prefix, otherV4Prefix, setV4Prefix}
|
|
||||||
|
|
||||||
db := newTestDatabase(t)
|
|
||||||
h := prefixHandlerWithoutTimer(db)
|
|
||||||
|
|
||||||
viaOtherOrigin := peerAnnouncement(thirdPeerIP, testV4Prefix)
|
|
||||||
viaOtherOrigin.Path = ristypes.ASPath{Sequence: []int{testASNA, testASNC}}
|
|
||||||
endsInSet := peerAnnouncement(secondPeerIP, setV4Prefix)
|
|
||||||
endsInSet.Path = decodePath(t, testASNA, []int{testASNC, testASND})
|
|
||||||
|
|
||||||
for _, msg := range []*ristypes.RISMessage{
|
|
||||||
peerAnnouncement(testPeerIP, testV4Prefix, testV6Prefix),
|
|
||||||
peerAnnouncement(secondPeerIP, testV4Prefix),
|
|
||||||
viaOtherOrigin,
|
|
||||||
endsInSet,
|
|
||||||
peerAnnouncement(thirdPeerIP, otherV4Prefix),
|
|
||||||
peerWithdrawal(thirdPeerIP, otherV4Prefix),
|
|
||||||
peerAnnouncement(fourthPeerIP, otherV4Prefix),
|
|
||||||
sessionDownOf(fourthPeerIP),
|
|
||||||
} {
|
|
||||||
h.HandleMessage(msg)
|
|
||||||
}
|
|
||||||
|
|
||||||
h.flushBatch()
|
|
||||||
|
|
||||||
var logs bytes.Buffer
|
|
||||||
|
|
||||||
table, err := readRouteTable(db,
|
|
||||||
&logger.Logger{Logger: slog.New(slog.NewJSONHandler(&logs, nil))})
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("readRouteTable: %v", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
checkSameTable(t, table, h.table, prefixes...)
|
|
||||||
|
|
||||||
// testV4Prefix with two origin ASes, testV6Prefix and setV4Prefix; four
|
|
||||||
// peers, the last of which carries nothing.
|
|
||||||
const wantPairs, wantPeers = 4, 4
|
|
||||||
|
|
||||||
var logged readLog
|
|
||||||
|
|
||||||
err = json.Unmarshal(bytes.TrimSpace(logs.Bytes()), &logged)
|
|
||||||
if err != nil || logged.Pairs != wantPairs || logged.Peers != wantPeers ||
|
|
||||||
logged.DurationMs == nil {
|
|
||||||
t.Errorf("readRouteTable logged %q, want %d pairs, %d peers and the "+
|
|
||||||
"duration", logs.String(), wantPairs, wantPeers)
|
|
||||||
}
|
|
||||||
|
|
||||||
restarted := NewPrefixHandler(db, logger.New(), table)
|
|
||||||
|
|
||||||
for _, msg := range []*ristypes.RISMessage{
|
|
||||||
peerWithdrawal(testPeerIP, testV6Prefix),
|
|
||||||
sessionDownOf(secondPeerIP),
|
|
||||||
peerAnnouncement(fourthPeerIP, otherV4Prefix),
|
|
||||||
peerAnnouncement(fifthPeerIP, setV4Prefix),
|
|
||||||
} {
|
|
||||||
// The first handler only applies the updates to its table.
|
|
||||||
h.HandleMessage(msg)
|
|
||||||
restarted.HandleMessage(msg)
|
|
||||||
}
|
|
||||||
|
|
||||||
restarted.Stop(t.Context())
|
|
||||||
|
|
||||||
read, err := db.ReadRouteTable(t.Context())
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("ReadRouteTable after the restart: %v", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
checkSameTable(t, read, h.table, prefixes...)
|
|
||||||
}
|
|
||||||
|
|
||||||
// peerWithdrawal returns an UPDATE message in which peer withdraws prefixes.
|
|
||||||
func peerWithdrawal(peer string, prefixes ...string) *ristypes.RISMessage {
|
|
||||||
return &ristypes.RISMessage{
|
|
||||||
Type: ristypes.MessageTypeUpdate,
|
|
||||||
Peer: peer,
|
|
||||||
ParsedTimestamp: time.Date(2026, 1, 2, 3, 4, 5, 0, time.UTC),
|
|
||||||
Withdrawals: prefixes,
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// announceManyPrefixes has testPeerIP announce n prefixes in table, with the
|
|
||||||
// path of peerAnnouncement: the /32 of each address from firstManyPrefix on.
|
|
||||||
func announceManyPrefixes(table *routetable.Table, n int) {
|
|
||||||
addr := netip.MustParsePrefix(firstManyPrefix).Addr()
|
|
||||||
route := routetable.Route{
|
|
||||||
Peer: testPeerIP,
|
|
||||||
ASPath: []int{testASNA, testASNB},
|
|
||||||
Time: time.Date(2026, 1, 2, 3, 4, 5, 0, time.UTC),
|
|
||||||
}
|
|
||||||
|
|
||||||
for range n {
|
|
||||||
table.Announce(netip.PrefixFrom(addr, addr.BitLen()), testASNB, route)
|
|
||||||
addr = addr.Next()
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// sessionDownOf returns the STATE message that says the BGP session of peer
|
|
||||||
// went down.
|
|
||||||
func sessionDownOf(peer string) *ristypes.RISMessage {
|
|
||||||
return &ristypes.RISMessage{
|
|
||||||
Type: ristypes.MessageTypeState,
|
|
||||||
Peer: peer,
|
|
||||||
State: ristypes.StateDown,
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// tableOrigins returns, for each origin AS of prefix in table, the peers that
|
|
||||||
// carry the prefix with it.
|
|
||||||
func tableOrigins(table *routetable.Table, prefix string) map[int][]string {
|
|
||||||
origins := make(map[int][]string)
|
|
||||||
for _, o := range table.Origins(netip.MustParsePrefix(prefix)) {
|
|
||||||
origins[o.ASN] = o.Peers
|
|
||||||
}
|
|
||||||
|
|
||||||
return origins
|
|
||||||
}
|
|
||||||
|
|
||||||
// checkSameTable checks that got answers as want does for each of prefixes, and
|
|
||||||
// that the two tables have the same counts.
|
|
||||||
func checkSameTable(
|
|
||||||
t *testing.T, got, want *routetable.Table, prefixes ...string,
|
|
||||||
) {
|
|
||||||
t.Helper()
|
|
||||||
|
|
||||||
for _, text := range prefixes {
|
|
||||||
prefix := netip.MustParsePrefix(text)
|
|
||||||
|
|
||||||
gotOrigins, wantOrigins := got.Origins(prefix), want.Origins(prefix)
|
|
||||||
if !reflect.DeepEqual(gotOrigins, wantOrigins) {
|
|
||||||
t.Errorf("Origins(%s) = %+v, want %+v", text, gotOrigins, wantOrigins)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
gotCounts, wantCounts := got.Counts(), want.Counts()
|
|
||||||
if !reflect.DeepEqual(gotCounts, wantCounts) {
|
|
||||||
t.Errorf("Counts() = %+v, want %+v", gotCounts, wantCounts)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
+292
-503
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
@@ -1,397 +0,0 @@
|
|||||||
package server
|
|
||||||
|
|
||||||
import (
|
|
||||||
"encoding/json"
|
|
||||||
"net/http"
|
|
||||||
"net/http/httptest"
|
|
||||||
"slices"
|
|
||||||
"strconv"
|
|
||||||
"strings"
|
|
||||||
"testing"
|
|
||||||
"time"
|
|
||||||
|
|
||||||
"git.eeqj.de/sneak/routewatch/internal/database"
|
|
||||||
"git.eeqj.de/sneak/routewatch/internal/ristypes"
|
|
||||||
"git.eeqj.de/sneak/routewatch/internal/routetable"
|
|
||||||
)
|
|
||||||
|
|
||||||
// The origin ASes in the routing table that newTwoOriginTable fills.
|
|
||||||
const (
|
|
||||||
mostPeersASN = 64500
|
|
||||||
fewerPeersASN = 64501
|
|
||||||
v6OriginASN = 64502
|
|
||||||
)
|
|
||||||
|
|
||||||
// noKeptRoutePrefix is the prefix of newTwoOriginTable whose origin keeps no
|
|
||||||
// route.
|
|
||||||
const noKeptRoutePrefix = "198.51.100.0/24"
|
|
||||||
|
|
||||||
// routeTimeFormat is how the pages write a route's time.
|
|
||||||
const routeTimeFormat = "2006-01-02 15:04:05"
|
|
||||||
|
|
||||||
// newTwoOriginTable returns a routing table that holds:
|
|
||||||
// - testPrefixV4 with two origin ASes. Peers 1, 2 and 3 carry it with
|
|
||||||
// mostPeersASN, which keeps the routes of peers 3 and 2. Peer 4 carries
|
|
||||||
// it with fewerPeersASN, in the prefix's newest route.
|
|
||||||
// - noKeptRoutePrefix with mostPeersASN. Only peer 7 carries it, and its
|
|
||||||
// route is not kept: peers 5 and 6, whose routes were, withdrew it.
|
|
||||||
// - testPrefixV6 with v6OriginASN.
|
|
||||||
func newTwoOriginTable() *routetable.Table {
|
|
||||||
table := emptyTable()
|
|
||||||
|
|
||||||
for n := 1; n <= 3; n++ {
|
|
||||||
announce(table, testPrefixV4, mostPeersASN, testPeer(n), testRouteTime(n))
|
|
||||||
}
|
|
||||||
|
|
||||||
announce(table, testPrefixV4, fewerPeersASN, testPeer(4), testRouteTime(5))
|
|
||||||
|
|
||||||
announce(table, noKeptRoutePrefix, mostPeersASN, testPeer(5), testRouteTime(4))
|
|
||||||
announce(table, noKeptRoutePrefix, mostPeersASN, testPeer(6), testRouteTime(6))
|
|
||||||
announce(table, noKeptRoutePrefix, mostPeersASN, testPeer(7), testRouteTime(0))
|
|
||||||
withdraw(table, noKeptRoutePrefix, testPeer(5))
|
|
||||||
withdraw(table, noKeptRoutePrefix, testPeer(6))
|
|
||||||
|
|
||||||
announce(table, testPrefixV6, v6OriginASN, testPeerV6, testRouteTime(0))
|
|
||||||
|
|
||||||
return table
|
|
||||||
}
|
|
||||||
|
|
||||||
// get sends a GET for path through s's router and fails the test unless it
|
|
||||||
// answers 200.
|
|
||||||
func get(t *testing.T, s *Server, path string) *httptest.ResponseRecorder {
|
|
||||||
t.Helper()
|
|
||||||
|
|
||||||
rec := httptest.NewRecorder()
|
|
||||||
s.router.ServeHTTP(rec, httptest.NewRequestWithContext(
|
|
||||||
t.Context(), http.MethodGet, path, nil))
|
|
||||||
|
|
||||||
if rec.Code != http.StatusOK {
|
|
||||||
t.Fatalf("%s: status %d, want %d; body %s",
|
|
||||||
path, rec.Code, http.StatusOK, rec.Body)
|
|
||||||
}
|
|
||||||
|
|
||||||
return rec
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestIPLookupGivesTheOriginMostPeersCarry looks up an address in a prefix
|
|
||||||
// with two origin ASes, through each IP lookup route. The answer names the
|
|
||||||
// origin AS that more peers carry, counts every peer, and gives the time of
|
|
||||||
// the prefix's newest kept route. The prefix's first-seen time and the AS's
|
|
||||||
// WHOIS details come from the database.
|
|
||||||
func TestIPLookupGivesTheOriginMostPeersCarry(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
s := newServerWithTable(t, newTwoOriginTable())
|
|
||||||
s.resolver = &fakeResolver{}
|
|
||||||
|
|
||||||
firstSeen := testRouteTime(-1)
|
|
||||||
|
|
||||||
err := s.db.UpdatePrefixesBatch(map[string]time.Time{testPrefixV4: firstSeen})
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("UpdatePrefixesBatch: %v", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
_, err = s.db.GetOrCreateASN(mostPeersASN, firstSeen)
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("GetOrCreateASN: %v", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
whois := &database.ASNWHOISUpdate{
|
|
||||||
ASN: mostPeersASN, ASName: "TEST-AS", CountryCode: "DE",
|
|
||||||
}
|
|
||||||
|
|
||||||
err = s.db.UpdateASNWHOIS(t.Context(), whois)
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("UpdateASNWHOIS: %v", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
// ownAddressPath looks up the request's own address, which httptest
|
|
||||||
// gives as addrInTestPrefixV4.
|
|
||||||
for _, path := range []string{
|
|
||||||
"/api/v1/ip/" + addrInTestPrefixV4, "/ip/" + addrInTestPrefixV4,
|
|
||||||
ownAddressPath,
|
|
||||||
} {
|
|
||||||
var body struct {
|
|
||||||
Data IPLookupResponse `json:"data"`
|
|
||||||
}
|
|
||||||
|
|
||||||
err := json.Unmarshal(get(t, s, path).Body.Bytes(), &body)
|
|
||||||
if err != nil || len(body.Data.Results) != 1 ||
|
|
||||||
body.Data.Results[0].IPInfo == nil {
|
|
||||||
t.Fatalf("%s: answer %+v (%v), want one result with a route",
|
|
||||||
path, body.Data, err)
|
|
||||||
}
|
|
||||||
|
|
||||||
got := body.Data.Results[0].IPInfo
|
|
||||||
if got.Netblock != testPrefixV4 || got.ASN != mostPeersASN ||
|
|
||||||
got.NumPeers != 4 || !got.LastSeen.Equal(testRouteTime(5)) ||
|
|
||||||
!got.FirstSeen.Equal(firstSeen) || got.ASName != whois.ASName ||
|
|
||||||
got.CountryCode != whois.CountryCode {
|
|
||||||
t.Errorf("%s: route %+v, want %s from AS%d, 4 peers, last seen %v, "+
|
|
||||||
"first seen %v, %s in %s", path, *got, testPrefixV4,
|
|
||||||
mostPeersASN, testRouteTime(5), firstSeen, whois.ASName,
|
|
||||||
whois.CountryCode)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestIPLookupAsksForNoWHOISLookupOfOriginZero checks that the IP lookup asks
|
|
||||||
// for no WHOIS lookup of AS 0, which means no known origin. It asks for one of
|
|
||||||
// an AS whose WHOIS details were never fetched or are stale, and not of one
|
|
||||||
// whose details are fresh.
|
|
||||||
func TestIPLookupAsksForNoWHOISLookupOfOriginZero(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
fresh := time.Now().Add(-time.Hour)
|
|
||||||
stale := time.Now().Add(-2 * whoisStaleThreshold)
|
|
||||||
|
|
||||||
tests := []struct {
|
|
||||||
name string
|
|
||||||
asInfo database.ASN
|
|
||||||
want bool
|
|
||||||
}{
|
|
||||||
{"no known origin", database.ASN{ASN: ristypes.NoOrigin}, false},
|
|
||||||
{"details never fetched", database.ASN{ASN: mostPeersASN}, true},
|
|
||||||
{"stale details", database.ASN{ASN: mostPeersASN, WHOISUpdatedAt: &stale}, true},
|
|
||||||
{"fresh details", database.ASN{ASN: mostPeersASN, WHOISUpdatedAt: &fresh}, false},
|
|
||||||
}
|
|
||||||
|
|
||||||
for _, tt := range tests {
|
|
||||||
if got := needsWHOISRefresh(&tt.asInfo); got != tt.want {
|
|
||||||
t.Errorf("%s: needsWHOISRefresh = %t, want %t", tt.name, got, tt.want)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// prefixRoute is one route of the prefix API's answer. A field the answer
|
|
||||||
// leaves out stays nil.
|
|
||||||
//
|
|
||||||
//nolint:tagliatelle // field names are the existing JSON API
|
|
||||||
type prefixRoute struct {
|
|
||||||
OriginASN int `json:"origin_asn"`
|
|
||||||
PeerIP string `json:"peer_ip"`
|
|
||||||
ASPath []int `json:"as_path"`
|
|
||||||
NextHop *string `json:"next_hop"`
|
|
||||||
LastUpdated *time.Time `json:"last_updated"`
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestPrefixAnswersListEveryPeer asks the prefix API and page for a prefix
|
|
||||||
// with two origin ASes, and for one whose origin keeps no route. Each answer
|
|
||||||
// lists every peer that carries the prefix, by origin AS. Only the kept routes
|
|
||||||
// have an AS path, a next hop and a time, newest first. The other peers follow
|
|
||||||
// without them.
|
|
||||||
func TestPrefixAnswersListEveryPeer(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
s := newServerWithTable(t, newTwoOriginTable())
|
|
||||||
|
|
||||||
// keptAt[peer] is the time of the peer's kept route; a peer not in it is
|
|
||||||
// listed without a route.
|
|
||||||
keptAt := map[string]time.Time{
|
|
||||||
testPeer(3): testRouteTime(3),
|
|
||||||
testPeer(2): testRouteTime(2),
|
|
||||||
testPeer(4): testRouteTime(5),
|
|
||||||
}
|
|
||||||
|
|
||||||
tests := []struct {
|
|
||||||
prefix string
|
|
||||||
wantPeers []string
|
|
||||||
wantOrigins []int
|
|
||||||
}{
|
|
||||||
{
|
|
||||||
testPrefixV4,
|
|
||||||
[]string{testPeer(3), testPeer(2), testPeer(1), testPeer(4)},
|
|
||||||
[]int{mostPeersASN, mostPeersASN, mostPeersASN, fewerPeersASN},
|
|
||||||
},
|
|
||||||
{noKeptRoutePrefix, []string{testPeer(7)}, []int{mostPeersASN}},
|
|
||||||
}
|
|
||||||
|
|
||||||
for _, tt := range tests {
|
|
||||||
//nolint:tagliatelle // field names are the existing JSON API
|
|
||||||
var body struct {
|
|
||||||
Data struct {
|
|
||||||
Routes []prefixRoute `json:"routes"`
|
|
||||||
PeerCount int `json:"peer_count"`
|
|
||||||
OriginCount int `json:"origin_count"`
|
|
||||||
} `json:"data"`
|
|
||||||
}
|
|
||||||
|
|
||||||
path := "/api/v1/prefix/" + tt.prefix
|
|
||||||
|
|
||||||
err := json.Unmarshal(get(t, s, path).Body.Bytes(), &body)
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("%s: decoding the answer: %v", path, err)
|
|
||||||
}
|
|
||||||
|
|
||||||
wantOriginCount := len(slices.Compact(slices.Clone(tt.wantOrigins)))
|
|
||||||
if body.Data.PeerCount != len(tt.wantPeers) ||
|
|
||||||
body.Data.OriginCount != wantOriginCount {
|
|
||||||
t.Errorf("%s: %d peers and %d origins, want %d and %d", path,
|
|
||||||
body.Data.PeerCount, body.Data.OriginCount, len(tt.wantPeers),
|
|
||||||
wantOriginCount)
|
|
||||||
}
|
|
||||||
|
|
||||||
checkPrefixRoutes(t, path, body.Data.Routes, tt.wantPeers, tt.wantOrigins,
|
|
||||||
keptAt)
|
|
||||||
checkPrefixPage(t, s, tt.prefix, tt.wantPeers, keptAt)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// checkPrefixRoutes checks that routes come from wantPeers, in that order,
|
|
||||||
// with wantOrigins as their origin ASes. The route of a peer in keptAt must
|
|
||||||
// have an AS path, the peer as next hop, and the time keptAt gives. Any other
|
|
||||||
// route must have none of the three.
|
|
||||||
func checkPrefixRoutes(
|
|
||||||
t *testing.T, path string, routes []prefixRoute, wantPeers []string,
|
|
||||||
wantOrigins []int, keptAt map[string]time.Time,
|
|
||||||
) {
|
|
||||||
t.Helper()
|
|
||||||
|
|
||||||
if len(routes) != len(wantPeers) {
|
|
||||||
t.Fatalf("%s: routes %+v, want one each from %v", path, routes, wantPeers)
|
|
||||||
}
|
|
||||||
|
|
||||||
for i, route := range routes {
|
|
||||||
if route.PeerIP != wantPeers[i] || route.OriginASN != wantOrigins[i] {
|
|
||||||
t.Errorf("%s: route %d from %s with AS%d, want from %s with AS%d",
|
|
||||||
path, i, route.PeerIP, route.OriginASN, wantPeers[i], wantOrigins[i])
|
|
||||||
}
|
|
||||||
|
|
||||||
at, kept := keptAt[route.PeerIP]
|
|
||||||
if !kept {
|
|
||||||
if route.ASPath != nil || route.NextHop != nil || route.LastUpdated != nil {
|
|
||||||
t.Errorf("%s: route from %s is not kept but has %+v",
|
|
||||||
path, route.PeerIP, route)
|
|
||||||
}
|
|
||||||
|
|
||||||
continue
|
|
||||||
}
|
|
||||||
|
|
||||||
if route.ASPath == nil || route.NextHop == nil ||
|
|
||||||
*route.NextHop != route.PeerIP || route.LastUpdated == nil ||
|
|
||||||
!route.LastUpdated.Equal(at) {
|
|
||||||
t.Errorf("%s: kept route from %s is %+v, want its AS path, the peer "+
|
|
||||||
"as next hop and the time %v", path, route.PeerIP, route, at)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// checkPrefixPage checks that the page of prefix shows each of wantPeers, the
|
|
||||||
// time of each kept route among them, and no time that was never set.
|
|
||||||
func checkPrefixPage(
|
|
||||||
t *testing.T, s *Server, prefix string, wantPeers []string,
|
|
||||||
keptAt map[string]time.Time,
|
|
||||||
) {
|
|
||||||
t.Helper()
|
|
||||||
|
|
||||||
path := "/prefix/" + prefix
|
|
||||||
page := get(t, s, path).Body.String()
|
|
||||||
|
|
||||||
for _, peer := range wantPeers {
|
|
||||||
want := `<td class="peer-ip">` + peer + `</td>`
|
|
||||||
if !strings.Contains(page, want) {
|
|
||||||
t.Errorf("%s does not list the peer %s", path, peer)
|
|
||||||
}
|
|
||||||
|
|
||||||
at, kept := keptAt[peer]
|
|
||||||
if kept && !strings.Contains(page, at.Format(routeTimeFormat)) {
|
|
||||||
t.Errorf("%s does not show the time %v of the route from %s",
|
|
||||||
path, at, peer)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
if strings.Contains(page, zeroDate) {
|
|
||||||
t.Errorf("%s shows the date %s", path, zeroDate)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestASAnswersGiveEachPrefixItsNewestKeptRoute asks the AS API and page for
|
|
||||||
// mostPeersASN. Its prefixes are testPrefixV4, whose newest kept route with
|
|
||||||
// that origin AS is peer 3's, and noKeptRoutePrefix, which keeps no route. The
|
|
||||||
// first has that route's time. The second is listed without a time.
|
|
||||||
func TestASAnswersGiveEachPrefixItsNewestKeptRoute(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
s := newServerWithTable(t, newTwoOriginTable())
|
|
||||||
path := "/api/v1/as/" + strconv.Itoa(mostPeersASN)
|
|
||||||
|
|
||||||
//nolint:tagliatelle // field names are the existing JSON API
|
|
||||||
var body struct {
|
|
||||||
Data struct {
|
|
||||||
IPv4Prefixes []struct {
|
|
||||||
Prefix string `json:"prefix"`
|
|
||||||
LastUpdated *time.Time `json:"last_updated"`
|
|
||||||
} `json:"ipv4_prefixes"`
|
|
||||||
} `json:"data"`
|
|
||||||
}
|
|
||||||
|
|
||||||
err := json.Unmarshal(get(t, s, path).Body.Bytes(), &body)
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("%s: decoding the answer: %v", path, err)
|
|
||||||
}
|
|
||||||
|
|
||||||
prefixes := body.Data.IPv4Prefixes
|
|
||||||
if len(prefixes) != 2 || prefixes[0].Prefix != testPrefixV4 ||
|
|
||||||
prefixes[0].LastUpdated == nil ||
|
|
||||||
!prefixes[0].LastUpdated.Equal(testRouteTime(3)) ||
|
|
||||||
prefixes[1].Prefix != noKeptRoutePrefix || prefixes[1].LastUpdated != nil {
|
|
||||||
t.Errorf("%s: prefixes %+v, want %s updated %v and %s without a time",
|
|
||||||
path, prefixes, testPrefixV4, testRouteTime(3), noKeptRoutePrefix)
|
|
||||||
}
|
|
||||||
|
|
||||||
path = "/as/" + strconv.Itoa(mostPeersASN)
|
|
||||||
page := get(t, s, path).Body.String()
|
|
||||||
|
|
||||||
for _, want := range []string{
|
|
||||||
testPrefixV4, noKeptRoutePrefix, testRouteTime(3).Format(routeTimeFormat),
|
|
||||||
} {
|
|
||||||
if !strings.Contains(page, want) {
|
|
||||||
t.Errorf("%s does not show %s", path, want)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
if strings.Contains(page, zeroDate) {
|
|
||||||
t.Errorf("%s shows the date %s", path, zeroDate)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestPrefixLengthPagesShowTheOriginMostPeersCarry asks for the IPv4 /24
|
|
||||||
// page and the IPv6 /32 page. The first lists both /24 prefixes with
|
|
||||||
// mostPeersASN, the origin AS more peers carry testPrefixV4 with, and gives
|
|
||||||
// noKeptRoutePrefix no age. The second lists testPrefixV6.
|
|
||||||
func TestPrefixLengthPagesShowTheOriginMostPeersCarry(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
s := newServerWithTable(t, newTwoOriginTable())
|
|
||||||
|
|
||||||
page := get(t, s, prefixLengthPagePath).Body.String()
|
|
||||||
|
|
||||||
for _, want := range []string{
|
|
||||||
">" + testPrefixV4 + "<", ">" + noKeptRoutePrefix + "<",
|
|
||||||
`href="/as/` + strconv.Itoa(mostPeersASN) + `"`,
|
|
||||||
} {
|
|
||||||
if !strings.Contains(page, want) {
|
|
||||||
t.Errorf("%s does not show %s", prefixLengthPagePath, want)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
if strings.Contains(page, `href="/as/`+strconv.Itoa(fewerPeersASN)+`"`) {
|
|
||||||
t.Errorf("%s links to AS%d, which fewer peers carry",
|
|
||||||
prefixLengthPagePath, fewerPeersASN)
|
|
||||||
}
|
|
||||||
|
|
||||||
const noAge = `<td class="age"></td>`
|
|
||||||
if strings.Count(page, noAge) != 1 {
|
|
||||||
t.Errorf("%s gives %d prefixes no age, want 1 (%s)",
|
|
||||||
prefixLengthPagePath, strings.Count(page, noAge), noKeptRoutePrefix)
|
|
||||||
}
|
|
||||||
|
|
||||||
const v6PagePath = "/prefixlength6/32"
|
|
||||||
|
|
||||||
page = get(t, s, v6PagePath).Body.String()
|
|
||||||
if !strings.Contains(page, ">"+testPrefixV6+"<") ||
|
|
||||||
!strings.Contains(page, `href="/as/`+strconv.Itoa(v6OriginASN)+`"`) {
|
|
||||||
t.Errorf("%s does not show %s with AS%d", v6PagePath, testPrefixV6,
|
|
||||||
v6OriginASN)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
@@ -11,12 +11,11 @@ import (
|
|||||||
)
|
)
|
||||||
|
|
||||||
// The addresses the forwarding tests use: the client a proxy names in
|
// The addresses the forwarding tests use: the client a proxy names in
|
||||||
// X-Forwarded-For, the range of trusted proxies, a peer inside it, and a peer
|
// X-Forwarded-For, a peer inside the trusted range of
|
||||||
// outside it, which lies in testPrefixV4 so that an IP lookup of it finds a
|
// TestRealIPMiddlewareTrustsOnlyConfiguredProxies, and a peer outside it,
|
||||||
// route.
|
// which lies in testPrefixV4 so that an IP lookup of it finds a route.
|
||||||
const (
|
const (
|
||||||
forwardedClient = "203.0.113.9"
|
forwardedClient = "203.0.113.9"
|
||||||
trustedProxyRange = "10.0.0.0/8"
|
|
||||||
trustedPeer = "10.0.0.2:1234"
|
trustedPeer = "10.0.0.2:1234"
|
||||||
untrustedPeerAddr = "192.0.2.1"
|
untrustedPeerAddr = "192.0.2.1"
|
||||||
untrustedPeer = untrustedPeerAddr + ":1234"
|
untrustedPeer = untrustedPeerAddr + ":1234"
|
||||||
@@ -190,7 +189,7 @@ func TestForwardedForFromUntrustedPeerIsIgnored(t *testing.T) {
|
|||||||
func TestRealIPMiddlewareTrustsOnlyConfiguredProxies(t *testing.T) {
|
func TestRealIPMiddlewareTrustsOnlyConfiguredProxies(t *testing.T) {
|
||||||
t.Parallel()
|
t.Parallel()
|
||||||
|
|
||||||
trusted := []netip.Prefix{netip.MustParsePrefix(trustedProxyRange)}
|
trusted := []netip.Prefix{netip.MustParsePrefix("10.0.0.0/8")}
|
||||||
|
|
||||||
tests := []struct {
|
tests := []struct {
|
||||||
name string
|
name string
|
||||||
|
|||||||
@@ -1,67 +0,0 @@
|
|||||||
package server
|
|
||||||
|
|
||||||
import (
|
|
||||||
"errors"
|
|
||||||
"time"
|
|
||||||
)
|
|
||||||
|
|
||||||
var (
|
|
||||||
// ErrInvalidIP says that an IP address is malformed. The IP lookup returns
|
|
||||||
// it.
|
|
||||||
ErrInvalidIP = errors.New("invalid IP address")
|
|
||||||
// ErrNoRoute says that the routing table has no route for an address, a
|
|
||||||
// prefix or an AS. The lookups return it.
|
|
||||||
ErrNoRoute = errors.New("no route found")
|
|
||||||
)
|
|
||||||
|
|
||||||
// LiveRoute represents a route in the live routing table. JSON leaves out an
|
|
||||||
// empty AS path, next hop or time: the routing table in memory keeps them only
|
|
||||||
// for the two newest routes of each prefix and origin AS.
|
|
||||||
//
|
|
||||||
//nolint:tagliatelle // field names are the existing JSON API
|
|
||||||
type LiveRoute struct {
|
|
||||||
Prefix string `json:"prefix"`
|
|
||||||
MaskLength int `json:"mask_length"`
|
|
||||||
IPVersion int `json:"ip_version"`
|
|
||||||
OriginASN int `json:"origin_asn"`
|
|
||||||
PeerIP string `json:"peer_ip"`
|
|
||||||
ASPath []int `json:"as_path,omitempty"`
|
|
||||||
NextHop string `json:"next_hop,omitempty"`
|
|
||||||
LastUpdated time.Time `json:"last_updated,omitzero"`
|
|
||||||
}
|
|
||||||
|
|
||||||
// PrefixDistribution represents the distribution of prefixes by mask length
|
|
||||||
//
|
|
||||||
//nolint:tagliatelle // field names are the existing JSON API
|
|
||||||
type PrefixDistribution struct {
|
|
||||||
MaskLength int `json:"mask_length"`
|
|
||||||
Count int `json:"count"`
|
|
||||||
}
|
|
||||||
|
|
||||||
// IPInfo represents comprehensive IP information for the /ip endpoint
|
|
||||||
//
|
|
||||||
//nolint:tagliatelle // field names are the existing JSON API
|
|
||||||
type IPInfo struct {
|
|
||||||
IP string `json:"ip"`
|
|
||||||
Netblock string `json:"netblock"`
|
|
||||||
MaskLength int `json:"mask_length"`
|
|
||||||
IPVersion int `json:"ip_version"`
|
|
||||||
NumPeers int `json:"num_peers"`
|
|
||||||
// AS information
|
|
||||||
ASN int `json:"asn"`
|
|
||||||
ASName string `json:"as_name,omitempty"`
|
|
||||||
Handle string `json:"handle,omitempty"`
|
|
||||||
Description string `json:"description,omitempty"`
|
|
||||||
OrgName string `json:"org_name,omitempty"`
|
|
||||||
OrgID string `json:"org_id,omitempty"`
|
|
||||||
Address string `json:"address,omitempty"`
|
|
||||||
CountryCode string `json:"country_code,omitempty"`
|
|
||||||
AbuseEmail string `json:"abuse_email,omitempty"`
|
|
||||||
RIR string `json:"rir,omitempty"`
|
|
||||||
// Timestamps. LastSeen is zero, and left out of JSON, when the prefix
|
|
||||||
// keeps no route.
|
|
||||||
FirstSeen time.Time `json:"first_seen"`
|
|
||||||
LastSeen time.Time `json:"last_seen,omitzero"`
|
|
||||||
// Indicates if WHOIS data needs refresh (not serialized)
|
|
||||||
NeedsWHOISRefresh bool `json:"-"`
|
|
||||||
}
|
|
||||||
@@ -3,10 +3,6 @@
|
|||||||
// - the AS, prefix and prefix length pages, and the JSON answers behind them
|
// - the AS, prefix and prefix length pages, and the JSON answers behind them
|
||||||
// - the IP lookups
|
// - the IP lookups
|
||||||
// - the AS name search
|
// - the AS name search
|
||||||
//
|
|
||||||
// Every route it shows comes from the routing table in memory. AS names,
|
|
||||||
// WHOIS details and the times a prefix and an AS were first seen come from
|
|
||||||
// the database.
|
|
||||||
package server
|
package server
|
||||||
|
|
||||||
import (
|
import (
|
||||||
@@ -19,11 +15,9 @@ import (
|
|||||||
"strconv"
|
"strconv"
|
||||||
"time"
|
"time"
|
||||||
|
|
||||||
"git.eeqj.de/sneak/routewatch/internal/blocklist"
|
|
||||||
"git.eeqj.de/sneak/routewatch/internal/config"
|
"git.eeqj.de/sneak/routewatch/internal/config"
|
||||||
"git.eeqj.de/sneak/routewatch/internal/database"
|
"git.eeqj.de/sneak/routewatch/internal/database"
|
||||||
"git.eeqj.de/sneak/routewatch/internal/logger"
|
"git.eeqj.de/sneak/routewatch/internal/logger"
|
||||||
"git.eeqj.de/sneak/routewatch/internal/routetable"
|
|
||||||
"git.eeqj.de/sneak/routewatch/internal/streamer"
|
"git.eeqj.de/sneak/routewatch/internal/streamer"
|
||||||
"github.com/go-chi/chi/v5"
|
"github.com/go-chi/chi/v5"
|
||||||
)
|
)
|
||||||
@@ -42,13 +36,6 @@ type ASNFetcher interface {
|
|||||||
GetStats() ASNFetcherStats
|
GetStats() ASNFetcherStats
|
||||||
}
|
}
|
||||||
|
|
||||||
// Blocklists tells which public blocklists list an address, from copies it
|
|
||||||
// keeps in memory, as *blocklist.Blocklists does.
|
|
||||||
type Blocklists interface {
|
|
||||||
Lookup(addr netip.Addr) []blocklist.Listing
|
|
||||||
ShortestInterval() time.Duration
|
|
||||||
}
|
|
||||||
|
|
||||||
// dnsResolver looks up the addresses of host names and the names of addresses
|
// dnsResolver looks up the addresses of host names and the names of addresses
|
||||||
// in DNS, as *net.Resolver does.
|
// in DNS, as *net.Resolver does.
|
||||||
type dnsResolver interface {
|
type dnsResolver interface {
|
||||||
@@ -60,13 +47,11 @@ type dnsResolver interface {
|
|||||||
type Server struct {
|
type Server struct {
|
||||||
router *chi.Mux
|
router *chi.Mux
|
||||||
db database.Store
|
db database.Store
|
||||||
table *routetable.Table
|
|
||||||
streamer *streamer.Streamer
|
streamer *streamer.Streamer
|
||||||
logger *logger.Logger
|
logger *logger.Logger
|
||||||
port int
|
port int
|
||||||
srv *http.Server
|
srv *http.Server
|
||||||
asnFetcher ASNFetcher
|
asnFetcher ASNFetcher
|
||||||
blocklists Blocklists
|
|
||||||
|
|
||||||
// trustedProxies are the reverse proxies whose X-Forwarded-For header
|
// trustedProxies are the reverse proxies whose X-Forwarded-For header
|
||||||
// RealIPMiddleware believes
|
// RealIPMiddleware believes
|
||||||
@@ -77,18 +62,15 @@ type Server struct {
|
|||||||
resolver dnsResolver
|
resolver dnsResolver
|
||||||
}
|
}
|
||||||
|
|
||||||
// New creates a new HTTP server that reads routes from table, the routing
|
// New creates a new HTTP server
|
||||||
// table that the prefix handler keeps current.
|
|
||||||
func New(
|
func New(
|
||||||
db database.Store,
|
db database.Store,
|
||||||
table *routetable.Table,
|
|
||||||
streamer *streamer.Streamer,
|
streamer *streamer.Streamer,
|
||||||
logger *logger.Logger,
|
logger *logger.Logger,
|
||||||
cfg *config.Config,
|
cfg *config.Config,
|
||||||
) *Server {
|
) *Server {
|
||||||
s := &Server{
|
s := &Server{
|
||||||
db: db,
|
db: db,
|
||||||
table: table,
|
|
||||||
streamer: streamer,
|
streamer: streamer,
|
||||||
logger: logger,
|
logger: logger,
|
||||||
port: cfg.Port,
|
port: cfg.Port,
|
||||||
@@ -166,9 +148,3 @@ func (s *Server) Stop(ctx context.Context) error {
|
|||||||
func (s *Server) SetASNFetcher(fetcher ASNFetcher) {
|
func (s *Server) SetASNFetcher(fetcher ASNFetcher) {
|
||||||
s.asnFetcher = fetcher
|
s.asnFetcher = fetcher
|
||||||
}
|
}
|
||||||
|
|
||||||
// SetBlocklists sets the blocklists the IP lookups report. Call it before
|
|
||||||
// Start: the handlers read the blocklists without a lock.
|
|
||||||
func (s *Server) SetBlocklists(blocklists Blocklists) {
|
|
||||||
s.blocklists = blocklists
|
|
||||||
}
|
|
||||||
|
|||||||
@@ -34,7 +34,7 @@ func TestStartFailsWhenPortIsHeld(t *testing.T) {
|
|||||||
|
|
||||||
port := addr.Port
|
port := addr.Port
|
||||||
log := logger.New()
|
log := logger.New()
|
||||||
s := server.New(nil, nil, streamer.New(log, metrics.New()), log,
|
s := server.New(nil, streamer.New(log, metrics.New()), log,
|
||||||
&config.Config{Port: port})
|
&config.Config{Port: port})
|
||||||
|
|
||||||
defer func() { _ = s.Stop(context.Background()) }()
|
defer func() { _ = s.Stop(context.Background()) }()
|
||||||
|
|||||||
@@ -126,26 +126,6 @@
|
|||||||
margin: 0;
|
margin: 0;
|
||||||
color: #2c3e50;
|
color: #2c3e50;
|
||||||
}
|
}
|
||||||
/* A click on a prefix list's heading folds or unfolds the list. Not
|
|
||||||
every browser shows its own marker on a flex element, so the page
|
|
||||||
hides it and draws a triangle before the heading. The triangle
|
|
||||||
points down while the list is open. */
|
|
||||||
summary.prefix-header {
|
|
||||||
cursor: pointer;
|
|
||||||
list-style: none;
|
|
||||||
}
|
|
||||||
summary.prefix-header::-webkit-details-marker {
|
|
||||||
display: none;
|
|
||||||
}
|
|
||||||
summary.prefix-header h2::before {
|
|
||||||
content: "\25B8";
|
|
||||||
display: inline-block;
|
|
||||||
margin-right: 10px;
|
|
||||||
transition: transform 0.2s;
|
|
||||||
}
|
|
||||||
details[open] > summary.prefix-header h2::before {
|
|
||||||
transform: rotate(90deg);
|
|
||||||
}
|
|
||||||
.prefix-count {
|
.prefix-count {
|
||||||
background: #e74c3c;
|
background: #e74c3c;
|
||||||
color: white;
|
color: white;
|
||||||
@@ -262,11 +242,11 @@
|
|||||||
</div>
|
</div>
|
||||||
|
|
||||||
{{if .IPv4Prefixes}}
|
{{if .IPv4Prefixes}}
|
||||||
<details class="prefix-section" open>
|
<div class="prefix-section">
|
||||||
<summary class="prefix-header">
|
<div class="prefix-header">
|
||||||
<h2>IPv4 Prefixes</h2>
|
<h2>IPv4 Prefixes</h2>
|
||||||
<span class="prefix-count">{{.IPv4Count}}</span>
|
<span class="prefix-count">{{.IPv4Count}}</span>
|
||||||
</summary>
|
</div>
|
||||||
<table class="prefix-table">
|
<table class="prefix-table">
|
||||||
<thead>
|
<thead>
|
||||||
<tr>
|
<tr>
|
||||||
@@ -281,21 +261,21 @@
|
|||||||
<tr>
|
<tr>
|
||||||
<td><a href="{{.Prefix | prefixURL}}" class="prefix-link">{{.Prefix}}</a></td>
|
<td><a href="{{.Prefix | prefixURL}}" class="prefix-link">{{.Prefix}}</a></td>
|
||||||
<td>/{{.MaskLength}}</td>
|
<td>/{{.MaskLength}}</td>
|
||||||
<td>{{if not .LastUpdated.IsZero}}{{.LastUpdated.Format "2006-01-02 15:04:05"}}{{end}}</td>
|
<td>{{.LastUpdated.Format "2006-01-02 15:04:05"}}</td>
|
||||||
<td class="age">{{if not .LastUpdated.IsZero}}{{.LastUpdated | timeSince}}{{end}}</td>
|
<td class="age">{{.LastUpdated | timeSince}}</td>
|
||||||
</tr>
|
</tr>
|
||||||
{{end}}
|
{{end}}
|
||||||
</tbody>
|
</tbody>
|
||||||
</table>
|
</table>
|
||||||
</details>
|
</div>
|
||||||
{{end}}
|
{{end}}
|
||||||
|
|
||||||
{{if .IPv6Prefixes}}
|
{{if .IPv6Prefixes}}
|
||||||
<details class="prefix-section" open>
|
<div class="prefix-section">
|
||||||
<summary class="prefix-header">
|
<div class="prefix-header">
|
||||||
<h2>IPv6 Prefixes</h2>
|
<h2>IPv6 Prefixes</h2>
|
||||||
<span class="prefix-count">{{.IPv6Count}}</span>
|
<span class="prefix-count">{{.IPv6Count}}</span>
|
||||||
</summary>
|
</div>
|
||||||
<table class="prefix-table">
|
<table class="prefix-table">
|
||||||
<thead>
|
<thead>
|
||||||
<tr>
|
<tr>
|
||||||
@@ -310,13 +290,13 @@
|
|||||||
<tr>
|
<tr>
|
||||||
<td><a href="{{.Prefix | prefixURL}}" class="prefix-link">{{.Prefix}}</a></td>
|
<td><a href="{{.Prefix | prefixURL}}" class="prefix-link">{{.Prefix}}</a></td>
|
||||||
<td>/{{.MaskLength}}</td>
|
<td>/{{.MaskLength}}</td>
|
||||||
<td>{{if not .LastUpdated.IsZero}}{{.LastUpdated.Format "2006-01-02 15:04:05"}}{{end}}</td>
|
<td>{{.LastUpdated.Format "2006-01-02 15:04:05"}}</td>
|
||||||
<td class="age">{{if not .LastUpdated.IsZero}}{{.LastUpdated | timeSince}}{{end}}</td>
|
<td class="age">{{.LastUpdated | timeSince}}</td>
|
||||||
</tr>
|
</tr>
|
||||||
{{end}}
|
{{end}}
|
||||||
</tbody>
|
</tbody>
|
||||||
</table>
|
</table>
|
||||||
</details>
|
</div>
|
||||||
{{end}}
|
{{end}}
|
||||||
|
|
||||||
{{if eq .TotalCount 0}}
|
{{if eq .TotalCount 0}}
|
||||||
|
|||||||
@@ -107,15 +107,9 @@
|
|||||||
.stat-card.connected .stat-value {
|
.stat-card.connected .stat-value {
|
||||||
color: #27ae60;
|
color: #27ae60;
|
||||||
}
|
}
|
||||||
.stat-card.disconnected .stat-value,
|
.stat-card.disconnected .stat-value {
|
||||||
.stale {
|
|
||||||
color: #e74c3c;
|
color: #e74c3c;
|
||||||
}
|
}
|
||||||
/* The card has no room beside "Connected", so the spinner and "stale" go under it */
|
|
||||||
#freshness {
|
|
||||||
display: block;
|
|
||||||
font-size: 20px;
|
|
||||||
}
|
|
||||||
|
|
||||||
/* Search section */
|
/* Search section */
|
||||||
.search-section {
|
.search-section {
|
||||||
@@ -239,8 +233,7 @@
|
|||||||
cursor: pointer;
|
cursor: pointer;
|
||||||
font-size: 12px;
|
font-size: 12px;
|
||||||
}
|
}
|
||||||
.ip-result pre,
|
.ip-result pre {
|
||||||
#ip-api pre {
|
|
||||||
background: #2c3e50;
|
background: #2c3e50;
|
||||||
color: #ecf0f1;
|
color: #ecf0f1;
|
||||||
padding: 16px;
|
padding: 16px;
|
||||||
@@ -252,30 +245,18 @@
|
|||||||
max-height: 400px;
|
max-height: 400px;
|
||||||
overflow-y: auto;
|
overflow-y: auto;
|
||||||
}
|
}
|
||||||
.ip-result .error,
|
.ip-result .error {
|
||||||
#ip-api .error {
|
|
||||||
background: #fee;
|
background: #fee;
|
||||||
color: #c00;
|
color: #c00;
|
||||||
padding: 12px;
|
padding: 12px;
|
||||||
border-radius: 6px;
|
border-radius: 6px;
|
||||||
font-size: 14px;
|
font-size: 14px;
|
||||||
}
|
}
|
||||||
.ip-result .loading,
|
.ip-result .loading {
|
||||||
#ip-api .loading {
|
|
||||||
color: #7f8c8d;
|
color: #7f8c8d;
|
||||||
font-style: italic;
|
font-style: italic;
|
||||||
}
|
}
|
||||||
|
|
||||||
/* IP information API card: the command, then its answer */
|
|
||||||
#ip-api p {
|
|
||||||
margin: 0;
|
|
||||||
color: #7f8c8d;
|
|
||||||
font-size: 14px;
|
|
||||||
}
|
|
||||||
#ip-api pre {
|
|
||||||
margin-top: 12px;
|
|
||||||
}
|
|
||||||
|
|
||||||
/* Footer */
|
/* Footer */
|
||||||
.footer {
|
.footer {
|
||||||
margin-top: 40px;
|
margin-top: 40px;
|
||||||
@@ -316,10 +297,7 @@
|
|||||||
<main class="main-content">
|
<main class="main-content">
|
||||||
<div class="stats-grid">
|
<div class="stats-grid">
|
||||||
<div class="stat-card" id="status-card">
|
<div class="stat-card" id="status-card">
|
||||||
<div class="stat-value">
|
<div class="stat-value" id="stat-status">-</div>
|
||||||
<span id="stat-status">-</span>
|
|
||||||
<span id="freshness"></span>
|
|
||||||
</div>
|
|
||||||
<div class="stat-label">Status</div>
|
<div class="stat-label">Status</div>
|
||||||
</div>
|
</div>
|
||||||
<div class="stat-card">
|
<div class="stat-card">
|
||||||
@@ -380,13 +358,6 @@
|
|||||||
</div>
|
</div>
|
||||||
</div>
|
</div>
|
||||||
</div>
|
</div>
|
||||||
|
|
||||||
<div class="search-card" id="ip-api">
|
|
||||||
<h2>IP Information API</h2>
|
|
||||||
<p>The API answers what RouteWatch knows about an IP address, such as its AS number and country. <code>/ip/</code> with no address answers for the caller's own address.</p>
|
|
||||||
<pre id="ip-api-command" data-client-ip="{{.ClientIP}}"></pre>
|
|
||||||
<pre id="ip-api-answer" class="loading">Loading...</pre>
|
|
||||||
</div>
|
|
||||||
</main>
|
</main>
|
||||||
|
|
||||||
<footer class="footer">
|
<footer class="footer">
|
||||||
@@ -398,44 +369,6 @@
|
|||||||
</footer>
|
</footer>
|
||||||
|
|
||||||
<script nonce="{{.CSPNonce}}">
|
<script nonce="{{.CSPNonce}}">
|
||||||
// The stats are fetched every REFRESH_PERIOD_MS. Next to the status, a
|
|
||||||
// spinner turns one frame each time an answer counts more messages
|
|
||||||
// than the one before. After STALE_AFTER_REFRESHES refreshes without
|
|
||||||
// such an answer, the spinner gives way to "stale" and the time since.
|
|
||||||
const REFRESH_PERIOD_MS = 5000;
|
|
||||||
const STALE_AFTER_REFRESHES = 3;
|
|
||||||
const MS_PER_SECOND = 1000;
|
|
||||||
const SPINNER_FRAMES = ['⠋', '⠙', '⠹', '⠸', '⠼', '⠴', '⠦', '⠧', '⠇', '⠏'];
|
|
||||||
const SPINNER_COLOURS = ['#e74c3c', '#e67e22', '#f1c40f', '#27ae60', '#3498db', '#9b59b6'];
|
|
||||||
|
|
||||||
let spinnerFrame = 0;
|
|
||||||
let lastMessageCount = null;
|
|
||||||
let lastFreshAt = Date.now();
|
|
||||||
|
|
||||||
// Turn the spinner when the message count has risen since the last answer.
|
|
||||||
function noteMessageCount(count) {
|
|
||||||
if (lastMessageCount !== null && count > lastMessageCount) {
|
|
||||||
spinnerFrame = (spinnerFrame + 1) % SPINNER_FRAMES.length;
|
|
||||||
lastFreshAt = Date.now();
|
|
||||||
}
|
|
||||||
lastMessageCount = count;
|
|
||||||
showFreshness();
|
|
||||||
}
|
|
||||||
|
|
||||||
function showFreshness() {
|
|
||||||
const freshness = document.getElementById('freshness');
|
|
||||||
const sinceFreshMs = Date.now() - lastFreshAt;
|
|
||||||
const stale = sinceFreshMs >= STALE_AFTER_REFRESHES * REFRESH_PERIOD_MS;
|
|
||||||
freshness.classList.toggle('stale', stale);
|
|
||||||
if (stale) {
|
|
||||||
freshness.textContent = 'stale for ' + Math.floor(sinceFreshMs / MS_PER_SECOND) + 's';
|
|
||||||
freshness.style.color = '';
|
|
||||||
return;
|
|
||||||
}
|
|
||||||
freshness.textContent = SPINNER_FRAMES[spinnerFrame];
|
|
||||||
freshness.style.color = SPINNER_COLOURS[spinnerFrame % SPINNER_COLOURS.length];
|
|
||||||
}
|
|
||||||
|
|
||||||
function formatNumber(num) {
|
function formatNumber(num) {
|
||||||
if (num >= 1000000) {
|
if (num >= 1000000) {
|
||||||
return (num / 1000000).toFixed(1) + 'M';
|
return (num / 1000000).toFixed(1) + 'M';
|
||||||
@@ -457,7 +390,6 @@
|
|||||||
const statusEl = document.getElementById('stat-status');
|
const statusEl = document.getElementById('stat-status');
|
||||||
statusEl.textContent = data.connected ? 'Connected' : 'Disconnected';
|
statusEl.textContent = data.connected ? 'Connected' : 'Disconnected';
|
||||||
statusCard.className = 'stat-card ' + (data.connected ? 'connected' : 'disconnected');
|
statusCard.className = 'stat-card ' + (data.connected ? 'connected' : 'disconnected');
|
||||||
noteMessageCount(data.total_messages);
|
|
||||||
|
|
||||||
document.getElementById('stat-routes').textContent = formatNumber(data.live_routes);
|
document.getElementById('stat-routes').textContent = formatNumber(data.live_routes);
|
||||||
document.getElementById('stat-asns').textContent = formatNumber(data.asns);
|
document.getElementById('stat-asns').textContent = formatNumber(data.asns);
|
||||||
@@ -604,33 +536,9 @@
|
|||||||
document.getElementById('ip-input').value = '';
|
document.getElementById('ip-input').value = '';
|
||||||
});
|
});
|
||||||
|
|
||||||
// IP information API card: show the curl command that asks /ip/ about
|
// Initial load and refresh stats every 5 seconds
|
||||||
// the visitor's own address, then make that same request and show the
|
|
||||||
// JSON it answers, an error answer included. The answer holds outside
|
|
||||||
// data, so it is set with textContent, never innerHTML.
|
|
||||||
const ipAPICommand = document.getElementById('ip-api-command');
|
|
||||||
const ipAPIAnswer = document.getElementById('ip-api-answer');
|
|
||||||
const ipAPIURL = window.location.origin + '/ip/' + ipAPICommand.dataset.clientIp;
|
|
||||||
|
|
||||||
ipAPICommand.textContent = 'curl ' + ipAPIURL;
|
|
||||||
|
|
||||||
fetch(ipAPIURL)
|
|
||||||
.then(response => response.json())
|
|
||||||
.then(answer => {
|
|
||||||
ipAPIAnswer.className = '';
|
|
||||||
ipAPIAnswer.textContent = JSON.stringify(answer, null, 2);
|
|
||||||
})
|
|
||||||
.catch(error => {
|
|
||||||
ipAPIAnswer.className = 'error';
|
|
||||||
ipAPIAnswer.textContent = 'Error: ' + error.message;
|
|
||||||
});
|
|
||||||
|
|
||||||
// Initial load and refresh stats every REFRESH_PERIOD_MS. The freshness
|
|
||||||
// check runs every second on its own, so a request that never returns
|
|
||||||
// still turns the spinner into "stale".
|
|
||||||
updateStats();
|
updateStats();
|
||||||
setInterval(updateStats, REFRESH_PERIOD_MS);
|
setInterval(updateStats, 5000);
|
||||||
setInterval(showFreshness, MS_PER_SECOND);
|
|
||||||
</script>
|
</script>
|
||||||
</body>
|
</body>
|
||||||
</html>
|
</html>
|
||||||
|
|||||||
@@ -329,8 +329,8 @@
|
|||||||
<td class="peer-ip">{{.PeerIP}}</td>
|
<td class="peer-ip">{{.PeerIP}}</td>
|
||||||
<td class="as-path">{{range $i, $as := .ASPathWithHandle}}{{if $i}} → {{end}}<a href="/as/{{$as.ASN}}" class="as-link">{{if $as.Handle}}{{$as.Handle}}{{else}}AS{{$as.ASN}}{{end}}</a>{{end}}</td>
|
<td class="as-path">{{range $i, $as := .ASPathWithHandle}}{{if $i}} → {{end}}<a href="/as/{{$as.ASN}}" class="as-link">{{if $as.Handle}}{{$as.Handle}}{{else}}AS{{$as.ASN}}{{end}}</a>{{end}}</td>
|
||||||
<td class="peer-ip">{{.NextHop}}</td>
|
<td class="peer-ip">{{.NextHop}}</td>
|
||||||
<td>{{if not .LastUpdated.IsZero}}{{.LastUpdated.Format "2006-01-02 15:04:05"}}{{end}}</td>
|
<td>{{.LastUpdated.Format "2006-01-02 15:04:05"}}</td>
|
||||||
<td class="age">{{if not .LastUpdated.IsZero}}{{.LastUpdated | timeSince}}{{end}}</td>
|
<td class="age">{{.LastUpdated | timeSince}}</td>
|
||||||
</tr>
|
</tr>
|
||||||
{{end}}
|
{{end}}
|
||||||
</tbody>
|
</tbody>
|
||||||
|
|||||||
@@ -126,8 +126,7 @@
|
|||||||
.connected {
|
.connected {
|
||||||
color: #22c55e;
|
color: #22c55e;
|
||||||
}
|
}
|
||||||
.disconnected,
|
.disconnected {
|
||||||
.stale {
|
|
||||||
color: #ef4444;
|
color: #ef4444;
|
||||||
}
|
}
|
||||||
.error {
|
.error {
|
||||||
@@ -181,10 +180,7 @@
|
|||||||
<h2>RouteWatch</h2>
|
<h2>RouteWatch</h2>
|
||||||
<div class="metric">
|
<div class="metric">
|
||||||
<span class="metric-label">Status</span>
|
<span class="metric-label">Status</span>
|
||||||
<span>
|
<span class="metric-value" id="connected">-</span>
|
||||||
<span class="metric-value" id="connected">-</span>
|
|
||||||
<span class="metric-value" id="freshness"></span>
|
|
||||||
</span>
|
|
||||||
</div>
|
</div>
|
||||||
<div class="metric">
|
<div class="metric">
|
||||||
<span class="metric-label">Uptime</span>
|
<span class="metric-label">Uptime</span>
|
||||||
@@ -318,10 +314,6 @@
|
|||||||
<span class="metric-label">IPv6 Routes</span>
|
<span class="metric-label">IPv6 Routes</span>
|
||||||
<span class="metric-value" id="ipv6_routes">-</span>
|
<span class="metric-value" id="ipv6_routes">-</span>
|
||||||
</div>
|
</div>
|
||||||
<div class="metric">
|
|
||||||
<span class="metric-label">Origins Without a Kept Route</span>
|
|
||||||
<span class="metric-value" id="origins_without_kept_route">-</span>
|
|
||||||
</div>
|
|
||||||
<div class="metric">
|
<div class="metric">
|
||||||
<span class="metric-label">IPv4 Updates/sec</span>
|
<span class="metric-label">IPv4 Updates/sec</span>
|
||||||
<span class="metric-value" id="ipv4_updates_per_sec">-</span>
|
<span class="metric-value" id="ipv4_updates_per_sec">-</span>
|
||||||
@@ -394,44 +386,6 @@
|
|||||||
</div>
|
</div>
|
||||||
|
|
||||||
<script nonce="{{.CSPNonce}}">
|
<script nonce="{{.CSPNonce}}">
|
||||||
// The stats are fetched every REFRESH_PERIOD_MS. Next to the status, a
|
|
||||||
// spinner turns one frame each time an answer counts more messages
|
|
||||||
// than the one before. After STALE_AFTER_REFRESHES refreshes without
|
|
||||||
// such an answer, the spinner gives way to "stale" and the time since.
|
|
||||||
const REFRESH_PERIOD_MS = 2000;
|
|
||||||
const STALE_AFTER_REFRESHES = 3;
|
|
||||||
const MS_PER_SECOND = 1000;
|
|
||||||
const SPINNER_FRAMES = ['⠋', '⠙', '⠹', '⠸', '⠼', '⠴', '⠦', '⠧', '⠇', '⠏'];
|
|
||||||
const SPINNER_COLOURS = ['#e74c3c', '#e67e22', '#f1c40f', '#27ae60', '#3498db', '#9b59b6'];
|
|
||||||
|
|
||||||
let spinnerFrame = 0;
|
|
||||||
let lastMessageCount = null;
|
|
||||||
let lastFreshAt = Date.now();
|
|
||||||
|
|
||||||
// Turn the spinner when the message count has risen since the last answer.
|
|
||||||
function noteMessageCount(count) {
|
|
||||||
if (lastMessageCount !== null && count > lastMessageCount) {
|
|
||||||
spinnerFrame = (spinnerFrame + 1) % SPINNER_FRAMES.length;
|
|
||||||
lastFreshAt = Date.now();
|
|
||||||
}
|
|
||||||
lastMessageCount = count;
|
|
||||||
showFreshness();
|
|
||||||
}
|
|
||||||
|
|
||||||
function showFreshness() {
|
|
||||||
const freshness = document.getElementById('freshness');
|
|
||||||
const sinceFreshMs = Date.now() - lastFreshAt;
|
|
||||||
const stale = sinceFreshMs >= STALE_AFTER_REFRESHES * REFRESH_PERIOD_MS;
|
|
||||||
freshness.classList.toggle('stale', stale);
|
|
||||||
if (stale) {
|
|
||||||
freshness.textContent = 'stale for ' + Math.floor(sinceFreshMs / MS_PER_SECOND) + 's';
|
|
||||||
freshness.style.color = '';
|
|
||||||
return;
|
|
||||||
}
|
|
||||||
freshness.textContent = SPINNER_FRAMES[spinnerFrame];
|
|
||||||
freshness.style.color = SPINNER_COLOURS[spinnerFrame % SPINNER_COLOURS.length];
|
|
||||||
}
|
|
||||||
|
|
||||||
function formatBytes(bytes) {
|
function formatBytes(bytes) {
|
||||||
if (bytes === 0) return '0 B';
|
if (bytes === 0) return '0 B';
|
||||||
const k = 1024;
|
const k = 1024;
|
||||||
@@ -575,7 +529,6 @@
|
|||||||
document.getElementById('live_routes').textContent = '-';
|
document.getElementById('live_routes').textContent = '-';
|
||||||
document.getElementById('ipv4_routes').textContent = '-';
|
document.getElementById('ipv4_routes').textContent = '-';
|
||||||
document.getElementById('ipv6_routes').textContent = '-';
|
document.getElementById('ipv6_routes').textContent = '-';
|
||||||
document.getElementById('origins_without_kept_route').textContent = '-';
|
|
||||||
document.getElementById('ipv4_updates_per_sec').textContent = '-';
|
document.getElementById('ipv4_updates_per_sec').textContent = '-';
|
||||||
document.getElementById('ipv6_updates_per_sec').textContent = '-';
|
document.getElementById('ipv6_updates_per_sec').textContent = '-';
|
||||||
document.getElementById('oldest_route').textContent = '-';
|
document.getElementById('oldest_route').textContent = '-';
|
||||||
@@ -616,7 +569,6 @@
|
|||||||
const connectedEl = document.getElementById('connected');
|
const connectedEl = document.getElementById('connected');
|
||||||
connectedEl.textContent = data.connected ? 'Connected' : 'Disconnected';
|
connectedEl.textContent = data.connected ? 'Connected' : 'Disconnected';
|
||||||
connectedEl.className = 'metric-value ' + (data.connected ? 'connected' : 'disconnected');
|
connectedEl.className = 'metric-value ' + (data.connected ? 'connected' : 'disconnected');
|
||||||
noteMessageCount(data.total_messages);
|
|
||||||
|
|
||||||
// Update all metrics
|
// Update all metrics
|
||||||
document.getElementById('uptime').textContent = data.uptime;
|
document.getElementById('uptime').textContent = data.uptime;
|
||||||
@@ -639,7 +591,6 @@
|
|||||||
document.getElementById('live_routes').textContent = formatNumber(data.live_routes);
|
document.getElementById('live_routes').textContent = formatNumber(data.live_routes);
|
||||||
document.getElementById('ipv4_routes').textContent = formatNumber(data.ipv4_routes);
|
document.getElementById('ipv4_routes').textContent = formatNumber(data.ipv4_routes);
|
||||||
document.getElementById('ipv6_routes').textContent = formatNumber(data.ipv6_routes);
|
document.getElementById('ipv6_routes').textContent = formatNumber(data.ipv6_routes);
|
||||||
document.getElementById('origins_without_kept_route').textContent = formatNumber(data.origins_without_kept_route);
|
|
||||||
document.getElementById('ipv4_updates_per_sec').textContent = formatNumber(Math.round(data.ipv4_updates_per_sec));
|
document.getElementById('ipv4_updates_per_sec').textContent = formatNumber(Math.round(data.ipv4_updates_per_sec));
|
||||||
document.getElementById('ipv6_updates_per_sec').textContent = formatNumber(Math.round(data.ipv6_updates_per_sec));
|
document.getElementById('ipv6_updates_per_sec').textContent = formatNumber(Math.round(data.ipv6_updates_per_sec));
|
||||||
document.getElementById('oldest_route').textContent = formatRelativeTime(data.oldest_route);
|
document.getElementById('oldest_route').textContent = formatRelativeTime(data.oldest_route);
|
||||||
@@ -692,12 +643,9 @@
|
|||||||
});
|
});
|
||||||
}
|
}
|
||||||
|
|
||||||
// Update immediately and then every REFRESH_PERIOD_MS. The freshness
|
// Update immediately and then every 2 seconds
|
||||||
// check runs every second on its own, so a request that never returns
|
|
||||||
// still turns the spinner into "stale".
|
|
||||||
updateStatus();
|
updateStatus();
|
||||||
setInterval(updateStatus, REFRESH_PERIOD_MS);
|
setInterval(updateStatus, 2000);
|
||||||
setInterval(showFreshness, MS_PER_SECOND);
|
|
||||||
</script>
|
</script>
|
||||||
</main>
|
</main>
|
||||||
|
|
||||||
|
|||||||
@@ -7,9 +7,9 @@ import (
|
|||||||
"net/url"
|
"net/url"
|
||||||
"strings"
|
"strings"
|
||||||
"sync"
|
"sync"
|
||||||
|
"time"
|
||||||
|
|
||||||
"git.eeqj.de/sneak/routewatch/internal/version"
|
"git.eeqj.de/sneak/routewatch/internal/version"
|
||||||
"github.com/dustin/go-humanize"
|
|
||||||
)
|
)
|
||||||
|
|
||||||
//go:embed index.html
|
//go:embed index.html
|
||||||
@@ -48,7 +48,48 @@ var (
|
|||||||
once sync.Once
|
once sync.Once
|
||||||
)
|
)
|
||||||
|
|
||||||
const cidrPartCount = 2 // A CIDR has two parts: prefix and length
|
const (
|
||||||
|
hoursPerDay = 24
|
||||||
|
daysPerMonth = 30
|
||||||
|
cidrPartCount = 2 // A CIDR has two parts: prefix and length
|
||||||
|
)
|
||||||
|
|
||||||
|
// timeSince returns a human-readable duration since the given time
|
||||||
|
func timeSince(t time.Time) string {
|
||||||
|
duration := time.Since(t)
|
||||||
|
if duration < time.Minute {
|
||||||
|
return "just now"
|
||||||
|
}
|
||||||
|
|
||||||
|
if duration < time.Hour {
|
||||||
|
minutes := int(duration.Minutes())
|
||||||
|
if minutes == 1 {
|
||||||
|
return "1 minute ago"
|
||||||
|
}
|
||||||
|
|
||||||
|
return duration.Truncate(time.Minute).String() + " ago"
|
||||||
|
}
|
||||||
|
|
||||||
|
if duration < hoursPerDay*time.Hour {
|
||||||
|
hours := int(duration.Hours())
|
||||||
|
if hours == 1 {
|
||||||
|
return "1 hour ago"
|
||||||
|
}
|
||||||
|
|
||||||
|
return duration.Truncate(time.Hour).String() + " ago"
|
||||||
|
}
|
||||||
|
|
||||||
|
days := int(duration.Hours() / hoursPerDay)
|
||||||
|
if days == 1 {
|
||||||
|
return "1 day ago"
|
||||||
|
}
|
||||||
|
|
||||||
|
if days < daysPerMonth {
|
||||||
|
return duration.Truncate(hoursPerDay*time.Hour).String() + " ago"
|
||||||
|
}
|
||||||
|
|
||||||
|
return t.Format("2006-01-02")
|
||||||
|
}
|
||||||
|
|
||||||
// prefixURL generates a URL path for a prefix in CIDR notation.
|
// prefixURL generates a URL path for a prefix in CIDR notation.
|
||||||
// Takes a prefix like "192.168.1.0/24" and returns "/prefix/192.168.1.0/24"
|
// Takes a prefix like "192.168.1.0/24" and returns "/prefix/192.168.1.0/24"
|
||||||
@@ -72,7 +113,7 @@ func initTemplates() {
|
|||||||
|
|
||||||
// Create common template functions
|
// Create common template functions
|
||||||
funcs := template.FuncMap{
|
funcs := template.FuncMap{
|
||||||
"timeSince": humanize.Time,
|
"timeSince": timeSince,
|
||||||
"urlEncode": url.QueryEscape,
|
"urlEncode": url.QueryEscape,
|
||||||
"prefixURL": prefixURL,
|
"prefixURL": prefixURL,
|
||||||
"appName": func() string { return version.Name },
|
"appName": func() string { return version.Name },
|
||||||
|
|||||||
@@ -6,26 +6,9 @@ import (
|
|||||||
"strings"
|
"strings"
|
||||||
"testing"
|
"testing"
|
||||||
|
|
||||||
"git.eeqj.de/sneak/routewatch/internal/database"
|
|
||||||
"git.eeqj.de/sneak/routewatch/internal/server"
|
|
||||||
"git.eeqj.de/sneak/routewatch/internal/templates"
|
"git.eeqj.de/sneak/routewatch/internal/templates"
|
||||||
)
|
)
|
||||||
|
|
||||||
// renderPage renders a page with the data every page needs, and the visitor's
|
|
||||||
// address, which the home page also needs.
|
|
||||||
func renderPage(t *testing.T, tmpl *template.Template) string {
|
|
||||||
t.Helper()
|
|
||||||
|
|
||||||
var page bytes.Buffer
|
|
||||||
|
|
||||||
err := tmpl.Execute(&page, struct{ CSPNonce, ClientIP string }{})
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("rendering the page: %v", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
return page.String()
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestFooterLicense renders the home and status pages and expects each footer
|
// TestFooterLicense renders the home and status pages and expects each footer
|
||||||
// to name the licence in LICENSE, MIT, and not WTFPL.
|
// to name the licence in LICENSE, MIT, and not WTFPL.
|
||||||
func TestFooterLicense(t *testing.T) {
|
func TestFooterLicense(t *testing.T) {
|
||||||
@@ -45,7 +28,14 @@ func TestFooterLicense(t *testing.T) {
|
|||||||
t.Run(name, func(t *testing.T) {
|
t.Run(name, func(t *testing.T) {
|
||||||
t.Parallel()
|
t.Parallel()
|
||||||
|
|
||||||
_, footer, found := strings.Cut(renderPage(t, tmpl), "<footer")
|
var page bytes.Buffer
|
||||||
|
|
||||||
|
err := tmpl.Execute(&page, struct{ CSPNonce string }{})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("rendering the page: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
_, footer, found := strings.Cut(page.String(), "<footer")
|
||||||
if !found {
|
if !found {
|
||||||
t.Fatal("the page has no footer")
|
t.Fatal("the page has no footer")
|
||||||
}
|
}
|
||||||
@@ -62,121 +52,3 @@ func TestFooterLicense(t *testing.T) {
|
|||||||
})
|
})
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// TestFreshnessIndicator renders the home and status pages and expects each to
|
|
||||||
// have, right after the status element, the element where the page's script
|
|
||||||
// shows its spinner or "stale".
|
|
||||||
func TestFreshnessIndicator(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
pages := map[string]struct {
|
|
||||||
tmpl *template.Template
|
|
||||||
status string
|
|
||||||
indicator string
|
|
||||||
}{
|
|
||||||
"index": {
|
|
||||||
tmpl: templates.IndexTemplate(),
|
|
||||||
status: `<span id="stat-status">-</span>`,
|
|
||||||
indicator: `<span id="freshness"></span>`,
|
|
||||||
},
|
|
||||||
"status": {
|
|
||||||
tmpl: templates.StatusTemplate(),
|
|
||||||
status: `<span class="metric-value" id="connected">-</span>`,
|
|
||||||
indicator: `<span class="metric-value" id="freshness"></span>`,
|
|
||||||
},
|
|
||||||
}
|
|
||||||
|
|
||||||
for name, page := range pages {
|
|
||||||
t.Run(name, func(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
_, afterStatus, found := strings.Cut(renderPage(t, page.tmpl), page.status)
|
|
||||||
if !found {
|
|
||||||
t.Fatalf("the page has no status element %s", page.status)
|
|
||||||
}
|
|
||||||
|
|
||||||
if !strings.HasPrefix(strings.TrimSpace(afterStatus), page.indicator) {
|
|
||||||
t.Errorf("the status element is not followed by %s", page.indicator)
|
|
||||||
}
|
|
||||||
})
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestASDetailPrefixListsFold renders the AS page with one IPv4 and one IPv6
|
|
||||||
// prefix. It expects each prefix list inside its own <details> element that
|
|
||||||
// starts open and whose <summary> holds the list's heading, so that a click on
|
|
||||||
// the heading folds the list.
|
|
||||||
func TestASDetailPrefixListsFold(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
const (
|
|
||||||
asn = 64500
|
|
||||||
prefixV4 = "192.0.2.0/24"
|
|
||||||
prefixV6 = "2001:db8::/32"
|
|
||||||
)
|
|
||||||
|
|
||||||
ipv4 := []server.LiveRoute{{Prefix: prefixV4}}
|
|
||||||
ipv6 := []server.LiveRoute{{Prefix: prefixV6}}
|
|
||||||
|
|
||||||
// The fields the AS page handler gives the template.
|
|
||||||
data := struct {
|
|
||||||
ASN *database.ASN
|
|
||||||
IPv4Prefixes []server.LiveRoute
|
|
||||||
IPv6Prefixes []server.LiveRoute
|
|
||||||
TotalCount int
|
|
||||||
IPv4Count int
|
|
||||||
IPv6Count int
|
|
||||||
Peers []database.ASPeer
|
|
||||||
PeerCount int
|
|
||||||
CSPNonce string
|
|
||||||
}{
|
|
||||||
ASN: &database.ASN{ASN: asn},
|
|
||||||
IPv4Prefixes: ipv4,
|
|
||||||
IPv6Prefixes: ipv6,
|
|
||||||
TotalCount: len(ipv4) + len(ipv6),
|
|
||||||
IPv4Count: len(ipv4),
|
|
||||||
IPv6Count: len(ipv6),
|
|
||||||
}
|
|
||||||
|
|
||||||
var rendered bytes.Buffer
|
|
||||||
|
|
||||||
err := templates.ASDetailTemplate().Execute(&rendered, data)
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("rendering the page: %v", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
page := rendered.String()
|
|
||||||
|
|
||||||
lists := []struct{ heading, prefix string }{
|
|
||||||
{"IPv4 Prefixes", prefixV4},
|
|
||||||
{"IPv6 Prefixes", prefixV6},
|
|
||||||
}
|
|
||||||
|
|
||||||
for _, list := range lists {
|
|
||||||
before, _, found := strings.Cut(page, list.prefix)
|
|
||||||
if !found {
|
|
||||||
t.Fatalf("the page does not list %s", list.prefix)
|
|
||||||
}
|
|
||||||
|
|
||||||
// The list's section starts at the last <details before its prefix.
|
|
||||||
start := strings.LastIndex(before, "<details")
|
|
||||||
if start < 0 || strings.Contains(before[start:], "</details>") {
|
|
||||||
t.Errorf("%s: the list is not inside a <details> element", list.heading)
|
|
||||||
|
|
||||||
continue
|
|
||||||
}
|
|
||||||
|
|
||||||
tag, section, _ := strings.Cut(before[start:], ">")
|
|
||||||
if !strings.Contains(tag, " open") {
|
|
||||||
t.Errorf("%s: the <details> element does not start open: %s>",
|
|
||||||
list.heading, tag)
|
|
||||||
}
|
|
||||||
|
|
||||||
_, summary, _ := strings.Cut(section, "<summary")
|
|
||||||
summary, _, _ = strings.Cut(summary, "</summary>")
|
|
||||||
|
|
||||||
if !strings.Contains(summary, list.heading) {
|
|
||||||
t.Errorf("%s: the <summary> does not hold the heading", list.heading)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|||||||
+32
-224
@@ -7,15 +7,10 @@ import (
|
|||||||
"errors"
|
"errors"
|
||||||
"fmt"
|
"fmt"
|
||||||
"io"
|
"io"
|
||||||
"log/slog"
|
|
||||||
"net"
|
"net"
|
||||||
"regexp"
|
"regexp"
|
||||||
"strings"
|
"strings"
|
||||||
"sync"
|
|
||||||
"time"
|
"time"
|
||||||
"unicode/utf8"
|
|
||||||
|
|
||||||
"golang.org/x/text/encoding/charmap"
|
|
||||||
)
|
)
|
||||||
|
|
||||||
// Timeout constants for WHOIS queries.
|
// Timeout constants for WHOIS queries.
|
||||||
@@ -31,18 +26,10 @@ const maxAnswerSize = 1 << 20
|
|||||||
|
|
||||||
// Parsing constants.
|
// Parsing constants.
|
||||||
const (
|
const (
|
||||||
|
keyValueParts = 2 // Expected parts when splitting "key: value"
|
||||||
lacnicDateFormatLen = 8 // Length of YYYYMMDD date format
|
lacnicDateFormatLen = 8 // Length of YYYYMMDD date format
|
||||||
)
|
)
|
||||||
|
|
||||||
// The classes of the two objects that describe an AS in an answer from
|
|
||||||
// RIPE, APNIC or AFRINIC, and the key of the aut-num object's line that
|
|
||||||
// gives the ID of the AS's organisation object.
|
|
||||||
const (
|
|
||||||
autNumClass = "aut-num"
|
|
||||||
organisationClass = "organisation"
|
|
||||||
orgKey = "org"
|
|
||||||
)
|
|
||||||
|
|
||||||
// WHOIS server addresses.
|
// WHOIS server addresses.
|
||||||
const (
|
const (
|
||||||
whoisServerIANA = "whois.iana.org:43"
|
whoisServerIANA = "whois.iana.org:43"
|
||||||
@@ -51,32 +38,8 @@ const (
|
|||||||
whoisServerAPNIC = "whois.apnic.net:43"
|
whoisServerAPNIC = "whois.apnic.net:43"
|
||||||
whoisServerLACNIC = "whois.lacnic.net:43"
|
whoisServerLACNIC = "whois.lacnic.net:43"
|
||||||
whoisServerAFRINIC = "whois.afrinic.net:43"
|
whoisServerAFRINIC = "whois.afrinic.net:43"
|
||||||
|
|
||||||
// NIC.br registers Brazil's ASes under LACNIC.
|
|
||||||
whoisServerNICbr = "whois.registro.br:43"
|
|
||||||
)
|
)
|
||||||
|
|
||||||
// nicbrHeader starts the copyright notice at the top of NIC.br's WHOIS
|
|
||||||
// answers. LACNIC's answer for an AS that NIC.br registers is NIC.br's, and
|
|
||||||
// carries the same notice:
|
|
||||||
//
|
|
||||||
// % Copyright (c) Nic.br - Use of this data is governed by the Use and
|
|
||||||
// % Privacy Policy at https://registro.br/upp . Distribution,
|
|
||||||
const nicbrHeader = "% Copyright (c) Nic.br"
|
|
||||||
|
|
||||||
// nicbrLimitReached is in the line that marks NIC.br's reduced answer to a
|
|
||||||
// client over its query limit. NIC.br publishes no limit. Its Use and
|
|
||||||
// Privacy Policy, https://registro.br/upp, says only that it restricts
|
|
||||||
// queries to stop the data being taken in bulk.
|
|
||||||
const nicbrLimitReached = "Query rate limit exceeded"
|
|
||||||
|
|
||||||
// NICbrSkipTime is how long the client stops asking NIC.br after NIC.br
|
|
||||||
// answers that the client is over its query limit. NIC.br publishes
|
|
||||||
// neither the limit nor how long it lasts. An hour outlasts a limit
|
|
||||||
// counted per minute or per hour. The ASes looked up meanwhile get
|
|
||||||
// NIC.br's fields when they are next refreshed.
|
|
||||||
const NICbrSkipTime = time.Hour
|
|
||||||
|
|
||||||
// RIR identifiers.
|
// RIR identifiers.
|
||||||
const (
|
const (
|
||||||
RIRARIN = "ARIN"
|
RIRARIN = "ARIN"
|
||||||
@@ -149,42 +112,21 @@ type Client struct {
|
|||||||
// host:port. NewClient sets a net.Dialer's; tests replace it to reach
|
// host:port. NewClient sets a net.Dialer's; tests replace it to reach
|
||||||
// local servers instead.
|
// local servers instead.
|
||||||
DialContext func(ctx context.Context, network, address string) (net.Conn, error)
|
DialContext func(ctx context.Context, network, address string) (net.Conn, error)
|
||||||
|
|
||||||
// Now returns the current time. NewClient sets time.Now; tests replace
|
|
||||||
// it to move the clock.
|
|
||||||
Now func() time.Time
|
|
||||||
|
|
||||||
logger *slog.Logger
|
|
||||||
|
|
||||||
// mu guards nicbrSkipUntil.
|
|
||||||
mu sync.Mutex
|
|
||||||
|
|
||||||
// nicbrSkipUntil is when the client next asks NIC.br, after NIC.br
|
|
||||||
// answered that the client is over its query limit.
|
|
||||||
nicbrSkipUntil time.Time
|
|
||||||
}
|
}
|
||||||
|
|
||||||
// NewClient creates a new WHOIS client that logs to logger.
|
// NewClient creates a new WHOIS client.
|
||||||
func NewClient(logger *slog.Logger) *Client {
|
func NewClient() *Client {
|
||||||
dialer := &net.Dialer{Timeout: dialTimeout}
|
dialer := &net.Dialer{Timeout: dialTimeout}
|
||||||
|
|
||||||
return &Client{
|
return &Client{DialContext: dialer.DialContext}
|
||||||
DialContext: dialer.DialContext,
|
|
||||||
Now: time.Now,
|
|
||||||
logger: logger,
|
|
||||||
}
|
|
||||||
}
|
}
|
||||||
|
|
||||||
// LookupASN asks IANA which RIR holds the given ASN, asks that RIR, and
|
// LookupASN asks IANA which RIR holds the given ASN, asks that RIR, and
|
||||||
// returns what the RIR's answer says. When LACNIC's answer is NIC.br's, it
|
// returns what the RIR's answer says. It returns ErrNoReferral when IANA
|
||||||
// returns what NIC.br's own server says instead, unless that query fails or
|
// refers the AS to no RIR, and another error when a query fails, when the
|
||||||
// is skipped, as askNICbr says.
|
// RIR refuses it, when IANA's answer has neither a referral nor the AS's
|
||||||
//
|
// block, when IANA refers the AS to a server none of the five RIRs run, or
|
||||||
// It returns ErrNoReferral when IANA refers the AS to no RIR, and another
|
// when the RIR's answer carries none of ASNInfo's fields.
|
||||||
// error when the query to IANA or to the RIR fails, when the RIR refuses
|
|
||||||
// it, when IANA's answer has neither a referral nor the AS's block, when
|
|
||||||
// IANA refers the AS to a server none of the five RIRs run, or when the
|
|
||||||
// answer carries none of ASNInfo's fields.
|
|
||||||
func (c *Client) LookupASN(ctx context.Context, asn int) (*ASNInfo, error) {
|
func (c *Client) LookupASN(ctx context.Context, asn int) (*ASNInfo, error) {
|
||||||
// Query IANA first to find the authoritative RIR
|
// Query IANA first to find the authoritative RIR
|
||||||
query := fmt.Sprintf("AS%d", asn)
|
query := fmt.Sprintf("AS%d", asn)
|
||||||
@@ -216,77 +158,18 @@ func (c *Client) LookupASN(ctx context.Context, asn int) (*ASNInfo, error) {
|
|||||||
return nil, fmt.Errorf("RIR query failed: %w", err)
|
return nil, fmt.Errorf("RIR query failed: %w", err)
|
||||||
}
|
}
|
||||||
|
|
||||||
// The fields never come from IANA's answer, which describes the block
|
// The fields come from the RIR's answer alone: IANA's describes the
|
||||||
// the AS is in.
|
// block the AS is in.
|
||||||
info := c.parseResponse(asn, rir, rirResp)
|
info := c.parseResponse(asn, rir, rirResp)
|
||||||
|
|
||||||
if rir == RIRLACNIC && strings.Contains(rirResp, nicbrHeader) {
|
|
||||||
info = c.askNICbr(ctx, query, info)
|
|
||||||
}
|
|
||||||
|
|
||||||
if info.isEmpty() {
|
if info.isEmpty() {
|
||||||
return nil, fmt.Errorf("%s from %s: %w", query, whoisServer, errNoFields)
|
return nil, fmt.Errorf("%s from %s: %w", query, whoisServer, errNoFields)
|
||||||
}
|
}
|
||||||
|
|
||||||
info.RawResponse = ianaResp + "\n" + info.RawResponse
|
info.RawResponse = ianaResp + "\n" + rirResp
|
||||||
|
|
||||||
return info, nil
|
return info, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
// askNICbr asks NIC.br's server about an AS whose answer from LACNIC is
|
|
||||||
// NIC.br's, and returns what NIC.br's answer says. LACNIC replaces each
|
|
||||||
// accented letter in such an answer with U+FFFD. NIC.br's own server keeps
|
|
||||||
// them, in Latin-1.
|
|
||||||
//
|
|
||||||
// askNICbr returns lacnicInfo, what LACNIC's answer says, and logs a
|
|
||||||
// warning when the query fails, when NIC.br sends the reduced answer for a
|
|
||||||
// client over its limit, or when NIC.br's answer carries none of ASNInfo's
|
|
||||||
// fields. After the reduced answer, it returns lacnicInfo without asking
|
|
||||||
// NIC.br until NICbrSkipTime has passed.
|
|
||||||
func (c *Client) askNICbr(
|
|
||||||
ctx context.Context, asQuery string, lacnicInfo *ASNInfo,
|
|
||||||
) *ASNInfo {
|
|
||||||
asn := lacnicInfo.ASN
|
|
||||||
|
|
||||||
c.mu.Lock()
|
|
||||||
skipping := c.Now().Before(c.nicbrSkipUntil)
|
|
||||||
c.mu.Unlock()
|
|
||||||
|
|
||||||
if skipping {
|
|
||||||
return lacnicInfo
|
|
||||||
}
|
|
||||||
|
|
||||||
answer, err := c.query(ctx, whoisServerNICbr, asQuery)
|
|
||||||
if err != nil {
|
|
||||||
c.logger.Warn("NIC.br query failed, keeping LACNIC's fields",
|
|
||||||
"asn", asn, "error", err)
|
|
||||||
|
|
||||||
return lacnicInfo
|
|
||||||
}
|
|
||||||
|
|
||||||
if strings.Contains(answer, nicbrLimitReached) {
|
|
||||||
c.mu.Lock()
|
|
||||||
c.nicbrSkipUntil = c.Now().Add(NICbrSkipTime)
|
|
||||||
c.mu.Unlock()
|
|
||||||
|
|
||||||
c.logger.Warn("NIC.br answered that the client is over its query "+
|
|
||||||
"limit, keeping LACNIC's fields and skipping NIC.br",
|
|
||||||
"asn", asn, "skip_for", NICbrSkipTime)
|
|
||||||
|
|
||||||
return lacnicInfo
|
|
||||||
}
|
|
||||||
|
|
||||||
info := c.parseResponse(asn, RIRLACNIC, answer)
|
|
||||||
if info.isEmpty() {
|
|
||||||
c.logger.Warn("NIC.br answered with no WHOIS fields, "+
|
|
||||||
"keeping LACNIC's fields", "asn", asn)
|
|
||||||
|
|
||||||
return lacnicInfo
|
|
||||||
}
|
|
||||||
|
|
||||||
return info
|
|
||||||
}
|
|
||||||
|
|
||||||
// isEmpty reports whether info holds none of the fields a WHOIS answer
|
// isEmpty reports whether info holds none of the fields a WHOIS answer
|
||||||
// fills, as when the answer is only comment and error lines.
|
// fills, as when the answer is only comment and error lines.
|
||||||
func (info *ASNInfo) isEmpty() bool {
|
func (info *ASNInfo) isEmpty() bool {
|
||||||
@@ -330,8 +213,8 @@ func (c *Client) queryRIR(
|
|||||||
return answer, nil
|
return answer, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
// query performs a raw WHOIS query to the specified server and returns the
|
// query performs a raw WHOIS query to the specified server. An answer that
|
||||||
// answer in UTF-8. An answer that reaches maxAnswerSize is an error.
|
// reaches maxAnswerSize is an error.
|
||||||
func (c *Client) query(ctx context.Context, server, query string) (string, error) {
|
func (c *Client) query(ctx context.Context, server, query string) (string, error) {
|
||||||
conn, err := c.DialContext(ctx, "tcp", server)
|
conn, err := c.DialContext(ctx, "tcp", server)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
@@ -378,25 +261,7 @@ func (c *Client) query(ctx context.Context, server, query string) (string, error
|
|||||||
return "", fmt.Errorf("read response: %w", errAnswerTooLarge)
|
return "", fmt.Errorf("read response: %w", errAnswerTooLarge)
|
||||||
}
|
}
|
||||||
|
|
||||||
return toUTF8(sb.String())
|
return sb.String(), nil
|
||||||
}
|
|
||||||
|
|
||||||
// toUTF8 returns a WHOIS answer in UTF-8. An answer that is valid UTF-8 is
|
|
||||||
// returned as it is. Any other answer is decoded as Latin-1, which RIPE,
|
|
||||||
// APNIC and NIC.br send. LACNIC, ARIN and AFRINIC send UTF-8. RIPE's
|
|
||||||
// database working group lists the five RIRs here:
|
|
||||||
// https://www.ripe.net/ripe/mail/archives/db-wg/2024-May/008075.html
|
|
||||||
func toUTF8(answer string) (string, error) {
|
|
||||||
if utf8.ValidString(answer) {
|
|
||||||
return answer, nil
|
|
||||||
}
|
|
||||||
|
|
||||||
decoded, err := charmap.ISO8859_1.NewDecoder().String(answer)
|
|
||||||
if err != nil {
|
|
||||||
return "", fmt.Errorf("decode Latin-1 answer: %w", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
return decoded, nil
|
|
||||||
}
|
}
|
||||||
|
|
||||||
// parseIANAReferral extracts the RIR and WHOIS server from an IANA response.
|
// parseIANAReferral extracts the RIR and WHOIS server from an IANA response.
|
||||||
@@ -432,9 +297,7 @@ func (c *Client) parseIANAReferral(response string) (string, string, error) {
|
|||||||
return "", "", nil
|
return "", "", nil
|
||||||
}
|
}
|
||||||
|
|
||||||
// parseResponse extracts ASN information from a WHOIS response. Of an
|
// parseResponse extracts ASN information from a WHOIS response.
|
||||||
// answer from RIPE, APNIC or AFRINIC it reads only the lines asObjectLines
|
|
||||||
// returns. Of an answer from ARIN or LACNIC it reads every line.
|
|
||||||
func (c *Client) parseResponse(asn int, rir, response string) *ASNInfo {
|
func (c *Client) parseResponse(asn int, rir, response string) *ASNInfo {
|
||||||
info := &ASNInfo{
|
info := &ASNInfo{
|
||||||
ASN: asn,
|
ASN: asn,
|
||||||
@@ -443,15 +306,26 @@ func (c *Client) parseResponse(asn int, rir, response string) *ASNInfo {
|
|||||||
}
|
}
|
||||||
|
|
||||||
lines := strings.Split(response, "\n")
|
lines := strings.Split(response, "\n")
|
||||||
if rir == RIRRIPE || rir == RIRAPNIC || rir == RIRAFRNIC {
|
|
||||||
lines = asObjectLines(lines)
|
|
||||||
}
|
|
||||||
|
|
||||||
var addressLines []string
|
var addressLines []string
|
||||||
|
|
||||||
for _, line := range lines {
|
for _, line := range lines {
|
||||||
key, value, ok := splitField(line)
|
line = strings.TrimSpace(line)
|
||||||
if !ok || value == "" {
|
if line == "" || strings.HasPrefix(line, "%") ||
|
||||||
|
strings.HasPrefix(line, "#") {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
|
||||||
|
// Split on first colon
|
||||||
|
parts := strings.SplitN(line, ":", keyValueParts)
|
||||||
|
if len(parts) != keyValueParts {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
|
||||||
|
key := strings.TrimSpace(strings.ToLower(parts[0]))
|
||||||
|
value := strings.TrimSpace(parts[1])
|
||||||
|
|
||||||
|
if value == "" {
|
||||||
continue
|
continue
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -478,72 +352,6 @@ func (c *Client) parseResponse(asn int, rir, response string) *ASNInfo {
|
|||||||
return info
|
return info
|
||||||
}
|
}
|
||||||
|
|
||||||
// splitField splits a "key: value" line of a WHOIS answer into its key, in
|
|
||||||
// lower case, and its value. The third result is false for a line that
|
|
||||||
// holds no field: a blank line, a comment line or a line without a colon.
|
|
||||||
func splitField(line string) (string, string, bool) {
|
|
||||||
line = strings.TrimSpace(line)
|
|
||||||
if strings.HasPrefix(line, "%") || strings.HasPrefix(line, "#") {
|
|
||||||
return "", "", false
|
|
||||||
}
|
|
||||||
|
|
||||||
key, value, ok := strings.Cut(line, ":")
|
|
||||||
|
|
||||||
return strings.ToLower(strings.TrimSpace(key)), strings.TrimSpace(value), ok
|
|
||||||
}
|
|
||||||
|
|
||||||
// asObjectLines returns the lines of the aut-num object in an answer from
|
|
||||||
// RIPE, APNIC or AFRINIC, followed by the lines of the organisation object
|
|
||||||
// whose ID the aut-num object gives on its "org:" line. The answer's other
|
|
||||||
// objects are left out. RIPE and APNIC start their answer with the
|
|
||||||
// as-block object of the block the AS is in, whose dates are not the AS's.
|
|
||||||
//
|
|
||||||
// These answers are in RPSL. A blank line ends an object. An object's first
|
|
||||||
// line has the object's class as its key and the object's ID as its value,
|
|
||||||
// as in "organisation: ORG-APNI1-AP".
|
|
||||||
func asObjectLines(lines []string) []string {
|
|
||||||
var autNum []string
|
|
||||||
|
|
||||||
// The lines of each organisation object, by its ID.
|
|
||||||
organisations := make(map[string][]string)
|
|
||||||
|
|
||||||
// The class and ID of the object the line is in, "" between objects.
|
|
||||||
var class, id string
|
|
||||||
|
|
||||||
// The ID of the organisation object the aut-num object refers to.
|
|
||||||
var orgID string
|
|
||||||
|
|
||||||
for _, line := range lines {
|
|
||||||
if strings.TrimSpace(line) == "" {
|
|
||||||
class = ""
|
|
||||||
|
|
||||||
continue
|
|
||||||
}
|
|
||||||
|
|
||||||
key, value, ok := splitField(line)
|
|
||||||
if !ok {
|
|
||||||
continue
|
|
||||||
}
|
|
||||||
|
|
||||||
if class == "" {
|
|
||||||
class, id = key, value
|
|
||||||
}
|
|
||||||
|
|
||||||
switch class {
|
|
||||||
case autNumClass:
|
|
||||||
autNum = append(autNum, line)
|
|
||||||
|
|
||||||
if key == orgKey {
|
|
||||||
orgID = value
|
|
||||||
}
|
|
||||||
case organisationClass:
|
|
||||||
organisations[id] = append(organisations[id], line)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
return append(autNum, organisations[orgID]...)
|
|
||||||
}
|
|
||||||
|
|
||||||
// setResponseField stores the value of one "key: value" line of a WHOIS
|
// setResponseField stores the value of one "key: value" line of a WHOIS
|
||||||
// response in info, keeping the first value seen for each field.
|
// response in info, keeping the first value seen for each field.
|
||||||
func (c *Client) setResponseField(info *ASNInfo, key, value string) {
|
func (c *Client) setResponseField(info *ASNInfo, key, value string) {
|
||||||
|
|||||||
@@ -2,13 +2,9 @@ package whois_test
|
|||||||
|
|
||||||
import (
|
import (
|
||||||
"bufio"
|
"bufio"
|
||||||
"bytes"
|
|
||||||
"context"
|
"context"
|
||||||
"encoding/json"
|
|
||||||
"errors"
|
"errors"
|
||||||
"io"
|
"io"
|
||||||
"log/slog"
|
|
||||||
"maps"
|
|
||||||
"net"
|
"net"
|
||||||
"slices"
|
"slices"
|
||||||
"strings"
|
"strings"
|
||||||
@@ -16,7 +12,6 @@ import (
|
|||||||
"syscall"
|
"syscall"
|
||||||
"testing"
|
"testing"
|
||||||
"time"
|
"time"
|
||||||
"unicode/utf8"
|
|
||||||
|
|
||||||
"git.eeqj.de/sneak/routewatch/internal/whois"
|
"git.eeqj.de/sneak/routewatch/internal/whois"
|
||||||
)
|
)
|
||||||
@@ -54,30 +49,14 @@ changed: 2002-09-01
|
|||||||
source: IANA
|
source: IANA
|
||||||
`
|
`
|
||||||
|
|
||||||
// lacnicReferral is IANA's answer for an AS in a block it gave to LACNIC,
|
|
||||||
// shortened to the referral.
|
|
||||||
const lacnicReferral = "refer: whois.lacnic.net\n"
|
|
||||||
|
|
||||||
// ripeRecord is RIPE's answer for the AS when asked to leave out person and
|
// ripeRecord is RIPE's answer for the AS when asked to leave out person and
|
||||||
// role objects. It starts with the as-block object of the block the AS is
|
// role objects: the abuse address as a comment, the AS's object and the
|
||||||
// in. Then come the abuse address as a comment, the AS's object and the
|
|
||||||
// organisation object it refers to.
|
// organisation object it refers to.
|
||||||
const ripeRecord = `% This is the RIPE Database query service.
|
const ripeRecord = `% This is the RIPE Database query service.
|
||||||
% The objects are in RPSL format.
|
% The objects are in RPSL format.
|
||||||
|
|
||||||
% Note: --no-personal means ALL personal data has been filtered from this output.
|
% Note: --no-personal means ALL personal data has been filtered from this output.
|
||||||
|
|
||||||
% Information related to 'AS3209 - AS3353'
|
|
||||||
|
|
||||||
as-block: AS3209 - AS3353
|
|
||||||
descr: RIPE NCC ASN block
|
|
||||||
remarks: These AS Numbers are assigned to network operators
|
|
||||||
remarks: in the RIPE NCC service region.
|
|
||||||
mnt-by: RIPE-NCC-HM-MNT
|
|
||||||
created: 2018-11-22T15:27:19Z
|
|
||||||
last-modified: 2018-11-22T15:27:19Z
|
|
||||||
source: RIPE
|
|
||||||
|
|
||||||
% Information related to 'AS3333'
|
% Information related to 'AS3333'
|
||||||
|
|
||||||
% Abuse contact for 'AS3333' is 'abuse@ripe.net'
|
% Abuse contact for 'AS3333' is 'abuse@ripe.net'
|
||||||
@@ -139,78 +118,6 @@ country: ZA
|
|||||||
address: 1 Example Road
|
address: 1 Example Road
|
||||||
`
|
`
|
||||||
|
|
||||||
// apnicServer is APNIC's WHOIS server, and apnicASN an AS that APNIC holds.
|
|
||||||
const (
|
|
||||||
apnicServer = "whois.apnic.net:43"
|
|
||||||
apnicASN = 4608
|
|
||||||
)
|
|
||||||
|
|
||||||
// The keys of the queries LookupASN sends for apnicASN: IANA's, then
|
|
||||||
// APNIC's, which asks APNIC to leave out person and role objects.
|
|
||||||
const (
|
|
||||||
apnicIANAQuery = ianaServer + " AS4608"
|
|
||||||
apnicQuery = apnicServer + " --no-personal AS4608"
|
|
||||||
)
|
|
||||||
|
|
||||||
// apnicRecord is APNIC's answer for apnicASN when asked to leave out person
|
|
||||||
// and role objects, shortened. It starts with the as-block object of the
|
|
||||||
// block the AS is in. Then come the abuse address as a comment, the AS's
|
|
||||||
// object, and the irt and organisation objects it refers to.
|
|
||||||
const apnicRecord = `% [whois.apnic.net]
|
|
||||||
% Whois data copyright terms http://www.apnic.net/db/dbcopyright.html
|
|
||||||
|
|
||||||
% Note: --no-personal means ALL personal data has been filtered from this output.
|
|
||||||
|
|
||||||
% Information related to 'AS4608 - AS4865'
|
|
||||||
|
|
||||||
as-block: AS4608 - AS4865
|
|
||||||
descr: APNIC ASN block
|
|
||||||
remarks: These AS numbers are further assigned by APNIC
|
|
||||||
remarks: to APNIC members and end-users in the APNIC region
|
|
||||||
mnt-by: APNIC-HM
|
|
||||||
mnt-lower: APNIC-HM
|
|
||||||
last-modified: 2019-05-20T03:52:26Z
|
|
||||||
source: APNIC
|
|
||||||
|
|
||||||
% Information related to 'AS4608'
|
|
||||||
|
|
||||||
% Abuse contact for 'AS4608' is 'abuse@apnic.net'
|
|
||||||
|
|
||||||
aut-num: AS4608
|
|
||||||
as-name: APNIC-SERVICES
|
|
||||||
descr: Asia Pacific Network Information Centre
|
|
||||||
country: AU
|
|
||||||
org: ORG-APNI1-AP
|
|
||||||
admin-c: AIC1-AP
|
|
||||||
abuse-c: AA1589-AP
|
|
||||||
mnt-by: APNIC-HM
|
|
||||||
mnt-irt: IRT-APNIC-IS-AP
|
|
||||||
last-modified: 2026-08-26T00:40:00Z
|
|
||||||
source: APNIC
|
|
||||||
|
|
||||||
irt: IRT-APNIC-IS-AP
|
|
||||||
address: South Brisbane, Australia
|
|
||||||
e-mail: helpdesk@apnic.net
|
|
||||||
abuse-mailbox: abuse@apnic.net
|
|
||||||
mnt-by: MAINT-APNIC-IS-AP
|
|
||||||
last-modified: 2025-11-18T00:27:08Z
|
|
||||||
source: APNIC
|
|
||||||
|
|
||||||
organisation: ORG-APNI1-AP
|
|
||||||
org-name: Asia Pacific Network Information Centre
|
|
||||||
org-type: LIR
|
|
||||||
country: AU
|
|
||||||
address: PO Box 3646
|
|
||||||
address: South Brisbane, QLD 4101,
|
|
||||||
address: Australia
|
|
||||||
e-mail: noc@apnic.net
|
|
||||||
mnt-by: APNIC-HM
|
|
||||||
last-modified: 2026-08-24T12:58:43Z
|
|
||||||
source: APNIC
|
|
||||||
|
|
||||||
% This query was served by the APNIC Whois Service version 1.88.48 (WHOIS-UK2)
|
|
||||||
`
|
|
||||||
|
|
||||||
// ianaPrivateUse is IANA's answer for a private-use AS: its block and no
|
// ianaPrivateUse is IANA's answer for a private-use AS: its block and no
|
||||||
// referral.
|
// referral.
|
||||||
const ianaPrivateUse = `% IANA WHOIS server
|
const ianaPrivateUse = `% IANA WHOIS server
|
||||||
@@ -280,7 +187,7 @@ func lookUp(
|
|||||||
) (*whois.ASNInfo, error) {
|
) (*whois.ASNInfo, error) {
|
||||||
t.Helper()
|
t.Helper()
|
||||||
|
|
||||||
client := whois.NewClient(slog.New(slog.DiscardHandler))
|
client := whois.NewClient()
|
||||||
client.DialContext = fakeServers(t, answers)
|
client.DialContext = fakeServers(t, answers)
|
||||||
|
|
||||||
return client.LookupASN(t.Context(), testASN)
|
return client.LookupASN(t.Context(), testASN)
|
||||||
@@ -290,8 +197,7 @@ func lookUp(
|
|||||||
// answers only a query that leaves out person and role objects, and its
|
// answers only a query that leaves out person and role objects, and its
|
||||||
// answer to that still fills the AS's fields: the abuse address from the
|
// answer to that still fills the AS's fields: the abuse address from the
|
||||||
// comment line, and the organisation's name and country from its object.
|
// comment line, and the organisation's name and country from its object.
|
||||||
// The dates are the AS's. IANA's "changed:" line dates the AS's block, and
|
// The dates are the AS's: IANA's "changed:" line dates its block.
|
||||||
// so do the dates of the as-block object at the start of RIPE's answer.
|
|
||||||
func TestLookupASNReadsTheRIRAnswer(t *testing.T) {
|
func TestLookupASNReadsTheRIRAnswer(t *testing.T) {
|
||||||
t.Parallel()
|
t.Parallel()
|
||||||
|
|
||||||
@@ -331,46 +237,6 @@ func TestLookupASNReadsTheRIRAnswer(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// TestLookupASNReadsOnlyTheASAndItsOrganisation looks up an AS at APNIC,
|
|
||||||
// whose answer starts with the as-block object of the block the AS is in.
|
|
||||||
// The last-modified date is the aut-num object's, not the block's. The
|
|
||||||
// address is the organisation object's alone, without the irt object's.
|
|
||||||
func TestLookupASNReadsOnlyTheASAndItsOrganisation(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
client := whois.NewClient(slog.New(slog.DiscardHandler))
|
|
||||||
client.DialContext = fakeServers(t, map[string]string{
|
|
||||||
apnicIANAQuery: "refer: whois.apnic.net\n",
|
|
||||||
apnicQuery: apnicRecord,
|
|
||||||
})
|
|
||||||
|
|
||||||
info, err := client.LookupASN(t.Context(), apnicASN)
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("LookupASN: %v", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
wantLastMod := time.Date(2026, 8, 26, 0, 40, 0, 0, time.UTC)
|
|
||||||
if info.LastMod == nil || !info.LastMod.Equal(wantLastMod) {
|
|
||||||
t.Errorf("LookupASN last modified = %v, want %v",
|
|
||||||
info.LastMod, wantLastMod)
|
|
||||||
}
|
|
||||||
|
|
||||||
got := []string{
|
|
||||||
info.ASName, info.OrgID, info.OrgName, info.CountryCode,
|
|
||||||
info.Address, info.AbuseEmail,
|
|
||||||
}
|
|
||||||
want := []string{
|
|
||||||
"APNIC-SERVICES", "ORG-APNI1-AP",
|
|
||||||
"Asia Pacific Network Information Centre", "AU",
|
|
||||||
"PO Box 3646\nSouth Brisbane, QLD 4101,\nAustralia", "abuse@apnic.net",
|
|
||||||
}
|
|
||||||
|
|
||||||
if !slices.Equal(got, want) {
|
|
||||||
t.Errorf("LookupASN = AS name, org ID, org name, country, address, "+
|
|
||||||
"abuse address %q; want %q", got, want)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestLookupASNAsksAPNICAndAFRINICToLeaveOutPersonalData looks up an AS at
|
// TestLookupASNAsksAPNICAndAFRINICToLeaveOutPersonalData looks up an AS at
|
||||||
// APNIC and at AFRINIC. Each answers only a query that leaves out person
|
// APNIC and at AFRINIC. Each answers only a query that leaves out person
|
||||||
// and role objects, as RIPE does. That one answer still holds the
|
// and role objects, as RIPE does. That one answer still holds the
|
||||||
@@ -407,361 +273,6 @@ func TestLookupASNAsksAPNICAndAFRINICToLeaveOutPersonalData(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// The name of an organisation with accented letters, in Latin-1, one byte for
|
|
||||||
// each letter, and in UTF-8.
|
|
||||||
const (
|
|
||||||
latin1OrgName = "Fundaci\xf3n Ra\xedces de Espa\xf1a"
|
|
||||||
utf8OrgName = "Fundación Raíces de España"
|
|
||||||
)
|
|
||||||
|
|
||||||
// TestLookupASNStoresAnswersInUTF8 looks up an AS at RIPE, which answers in
|
|
||||||
// Latin-1, and at LACNIC, which answers in UTF-8. Both answers give the
|
|
||||||
// organisation's name with its accented letters in UTF-8, in the parsed field
|
|
||||||
// and in the stored answer. Before the fix the Latin-1 answer was kept as
|
|
||||||
// received, which is not valid UTF-8.
|
|
||||||
func TestLookupASNStoresAnswersInUTF8(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
for name, answers := range map[string]map[string]string{
|
|
||||||
"RIPE in Latin-1": {
|
|
||||||
ianaQuery: ianaReferral,
|
|
||||||
ripeQuery: "aut-num: AS3333\n" +
|
|
||||||
"org-name: " + latin1OrgName + "\n",
|
|
||||||
},
|
|
||||||
"LACNIC in UTF-8": {
|
|
||||||
ianaQuery: lacnicReferral,
|
|
||||||
"whois.lacnic.net:43 " + asQuery: "aut-num: AS3333\n" +
|
|
||||||
"owner: " + utf8OrgName + "\n",
|
|
||||||
},
|
|
||||||
} {
|
|
||||||
t.Run(name, func(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
info, err := lookUp(t, answers)
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("LookupASN: %v", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
if info.OrgName != utf8OrgName {
|
|
||||||
t.Errorf("LookupASN org name = %q, want %q",
|
|
||||||
info.OrgName, utf8OrgName)
|
|
||||||
}
|
|
||||||
|
|
||||||
if !utf8.ValidString(info.RawResponse) ||
|
|
||||||
!strings.Contains(info.RawResponse, utf8OrgName) {
|
|
||||||
t.Errorf("LookupASN answer = %q, want UTF-8 holding %q",
|
|
||||||
info.RawResponse, utf8OrgName)
|
|
||||||
}
|
|
||||||
})
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// lacnicServer is LACNIC's WHOIS server and nicbrServer NIC.br's, which
|
|
||||||
// registers Brazil's ASes under LACNIC. brazilASN is one of those ASes.
|
|
||||||
const (
|
|
||||||
lacnicServer = "whois.lacnic.net:43"
|
|
||||||
nicbrServer = "whois.registro.br:43"
|
|
||||||
brazilASN = 52751
|
|
||||||
)
|
|
||||||
|
|
||||||
// The keys of the queries LookupASN sends for brazilASN: IANA's, LACNIC's,
|
|
||||||
// then NIC.br's.
|
|
||||||
const (
|
|
||||||
brazilIANAQuery = ianaServer + " AS52751"
|
|
||||||
brazilLACNICQuery = lacnicServer + " AS52751"
|
|
||||||
brazilNICbrQuery = nicbrServer + " AS52751"
|
|
||||||
)
|
|
||||||
|
|
||||||
// The name of brazilASN's organisation in three forms: as LACNIC sends it,
|
|
||||||
// with each accented letter replaced by U+FFFD; as NIC.br sends it, in
|
|
||||||
// Latin-1; and in UTF-8.
|
|
||||||
const (
|
|
||||||
lacnicBrazilOrgName = "Brasrede Telecomunica\U0000FFFD\U0000FFFDes LTDA"
|
|
||||||
nicbrBrazilOrgName = "Brasrede Telecomunica\xe7\xf5es LTDA"
|
|
||||||
brazilOrgName = "Brasrede Telecomunicações LTDA"
|
|
||||||
)
|
|
||||||
|
|
||||||
// nicbrCopyright is the copyright notice at the top of NIC.br's answers.
|
|
||||||
const nicbrCopyright = "% Copyright (c) Nic.br - " +
|
|
||||||
"Use of this data is governed by the Use and\n" +
|
|
||||||
"% Privacy Policy at https://registro.br/upp . Distribution,\n" +
|
|
||||||
"% commercialization, reproduction, and use for advertising or similar\n" +
|
|
||||||
"% purposes are expressly prohibited.\n"
|
|
||||||
|
|
||||||
// lacnicBrazilRecord is LACNIC's answer for brazilASN, recorded on
|
|
||||||
// 2026-10-26. It is NIC.br's answer, which LACNIC passes on after a line
|
|
||||||
// of its own and a space. It is shortened. The client's address is a
|
|
||||||
// documentation address. The contact's name, the contact's object and the
|
|
||||||
// footer are left out.
|
|
||||||
const lacnicBrazilRecord = "% IP Client: 192.0.2.1\n" +
|
|
||||||
" " + nicbrCopyright + `% 2026-10-26T23:12:22-03:00 - 192.0.2.1
|
|
||||||
|
|
||||||
aut-num: AS52751
|
|
||||||
owner: ` + lacnicBrazilOrgName + `
|
|
||||||
owner-c: AVL217
|
|
||||||
routing-c: AVL217
|
|
||||||
abuse-c: AVL217
|
|
||||||
created: 20120829
|
|
||||||
changed: 20130306
|
|
||||||
inetnum: 177.38.152.0/21
|
|
||||||
inetnum: 2804:b34::/32
|
|
||||||
inetnum: 191.243.4.0/22
|
|
||||||
inetnum: 131.72.248.0/22
|
|
||||||
inetnum: 138.94.228.0/22
|
|
||||||
inetnum: 170.233.8.0/22
|
|
||||||
inetnum: 170.246.4.0/22
|
|
||||||
`
|
|
||||||
|
|
||||||
// nicbrRecord is NIC.br's answer for brazilASN, in Latin-1, recorded on
|
|
||||||
// 2026-10-26 and shortened as lacnicBrazilRecord is.
|
|
||||||
const nicbrRecord = nicbrCopyright + `% 2026-10-26T23:14:41-03:00 - 192.0.2.1
|
|
||||||
|
|
||||||
aut-num: AS52751
|
|
||||||
owner: ` + nicbrBrazilOrgName + `
|
|
||||||
owner-c: AVL217
|
|
||||||
routing-c: AVL217
|
|
||||||
abuse-c: AVL217
|
|
||||||
created: 20120829
|
|
||||||
changed: 20130306
|
|
||||||
inetnum: 177.38.152.0/21
|
|
||||||
inetnum: 2804:b34::/32
|
|
||||||
inetnum: 191.243.4.0/22
|
|
||||||
inetnum: 131.72.248.0/22
|
|
||||||
inetnum: 138.94.228.0/22
|
|
||||||
inetnum: 170.233.8.0/22
|
|
||||||
inetnum: 170.246.4.0/22
|
|
||||||
`
|
|
||||||
|
|
||||||
// nicbrOverLimit stands for NIC.br's answer for brazilASN to a client over
|
|
||||||
// its query limit, and is not recorded. Such an answer leaves information
|
|
||||||
// out, here the owner, and says so in its first line.
|
|
||||||
const nicbrOverLimit = `% Query rate limit exceeded. Reduced information.
|
|
||||||
|
|
||||||
aut-num: AS52751
|
|
||||||
created: 20120829
|
|
||||||
changed: 20130306
|
|
||||||
`
|
|
||||||
|
|
||||||
// newNICbrClient returns a client whose fake servers send answers and
|
|
||||||
// whose logger writes JSON lines to logs. It adds one to *nicbrAsks each
|
|
||||||
// time it connects to NIC.br's server.
|
|
||||||
func newNICbrClient(
|
|
||||||
t *testing.T, answers map[string]string, logs io.Writer, nicbrAsks *int,
|
|
||||||
) *whois.Client {
|
|
||||||
t.Helper()
|
|
||||||
|
|
||||||
dial := fakeServers(t, answers)
|
|
||||||
|
|
||||||
client := whois.NewClient(slog.New(slog.NewJSONHandler(logs, nil)))
|
|
||||||
client.DialContext = func(
|
|
||||||
ctx context.Context, network, address string,
|
|
||||||
) (net.Conn, error) {
|
|
||||||
if address == nicbrServer {
|
|
||||||
*nicbrAsks++
|
|
||||||
}
|
|
||||||
|
|
||||||
return dial(ctx, network, address)
|
|
||||||
}
|
|
||||||
|
|
||||||
return client
|
|
||||||
}
|
|
||||||
|
|
||||||
// lookUpBrazilianAS looks brazilASN up with fake servers sending answers,
|
|
||||||
// and reports whether the client connected to NIC.br's server.
|
|
||||||
func lookUpBrazilianAS(
|
|
||||||
t *testing.T, answers map[string]string,
|
|
||||||
) (*whois.ASNInfo, bool, error) {
|
|
||||||
t.Helper()
|
|
||||||
|
|
||||||
nicbrAsks := 0
|
|
||||||
client := newNICbrClient(t, answers, io.Discard, &nicbrAsks)
|
|
||||||
|
|
||||||
info, err := client.LookupASN(t.Context(), brazilASN)
|
|
||||||
|
|
||||||
return info, nicbrAsks > 0, err
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestLookupASNTakesABrazilianASFromNICbr looks up an AS that NIC.br
|
|
||||||
// registers. LACNIC's answer is NIC.br's with each accented letter replaced
|
|
||||||
// by U+FFFD, so the client also asks NIC.br's server, which answers in
|
|
||||||
// Latin-1. The organisation's name has its accented letters, in the parsed
|
|
||||||
// field and in the stored answer.
|
|
||||||
func TestLookupASNTakesABrazilianASFromNICbr(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
info, _, err := lookUpBrazilianAS(t, map[string]string{
|
|
||||||
brazilIANAQuery: lacnicReferral,
|
|
||||||
brazilLACNICQuery: lacnicBrazilRecord,
|
|
||||||
brazilNICbrQuery: nicbrRecord,
|
|
||||||
})
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("LookupASN: %v", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
if info.OrgName != brazilOrgName || info.RIR != whois.RIRLACNIC {
|
|
||||||
t.Errorf("LookupASN org name, RIR = %q, %q; want %q, %q",
|
|
||||||
info.OrgName, info.RIR, brazilOrgName, whois.RIRLACNIC)
|
|
||||||
}
|
|
||||||
|
|
||||||
if !strings.Contains(info.RawResponse, brazilOrgName) {
|
|
||||||
t.Errorf("LookupASN answer = %q, want one holding %q",
|
|
||||||
info.RawResponse, brazilOrgName)
|
|
||||||
}
|
|
||||||
|
|
||||||
wantRegDate := time.Date(2012, 8, 29, 0, 0, 0, 0, time.UTC)
|
|
||||||
if info.RegDate == nil || !info.RegDate.Equal(wantRegDate) {
|
|
||||||
t.Errorf("LookupASN registered = %v, want %v",
|
|
||||||
info.RegDate, wantRegDate)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestLookupASNAsksNICbrOnlyForItsASes looks up an AS at LACNIC whose
|
|
||||||
// answer does not carry NIC.br's copyright notice, as for an AS that LACNIC
|
|
||||||
// registers itself. The client does not connect to NIC.br's server, which
|
|
||||||
// would answer with a record, and keeps LACNIC's fields.
|
|
||||||
func TestLookupASNAsksNICbrOnlyForItsASes(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
info, askedNICbr, err := lookUpBrazilianAS(t, map[string]string{
|
|
||||||
brazilIANAQuery: lacnicReferral,
|
|
||||||
brazilLACNICQuery: "aut-num: AS52751\n" +
|
|
||||||
"owner: " + utf8OrgName + "\n",
|
|
||||||
brazilNICbrQuery: nicbrRecord,
|
|
||||||
})
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("LookupASN: %v", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
if askedNICbr {
|
|
||||||
t.Error("LookupASN asked NIC.br's server about an AS whose " +
|
|
||||||
"answer from LACNIC is not NIC.br's")
|
|
||||||
}
|
|
||||||
|
|
||||||
if info.OrgName != utf8OrgName {
|
|
||||||
t.Errorf("LookupASN org name = %q, want LACNIC's %q",
|
|
||||||
info.OrgName, utf8OrgName)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestLookupASNKeepsLACNICsAnswerWhenNICbrFails looks up an AS that NIC.br
|
|
||||||
// registers when NIC.br's server refuses the connection, sends nothing, or
|
|
||||||
// sends the reduced answer for a client over its query limit. The client
|
|
||||||
// asks NIC.br, and the lookup still succeeds with LACNIC's fields. The
|
|
||||||
// client logs one warning, which names the AS and the reason.
|
|
||||||
func TestLookupASNKeepsLACNICsAnswerWhenNICbrFails(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
for name, test := range map[string]struct {
|
|
||||||
nicbrAnswer map[string]string
|
|
||||||
reason string
|
|
||||||
}{
|
|
||||||
"connection refused": {
|
|
||||||
nicbrAnswer: map[string]string{},
|
|
||||||
reason: syscall.ECONNREFUSED.Error(),
|
|
||||||
},
|
|
||||||
"no answer": {
|
|
||||||
nicbrAnswer: map[string]string{brazilNICbrQuery: ""},
|
|
||||||
reason: "no WHOIS fields",
|
|
||||||
},
|
|
||||||
"over the query limit": {
|
|
||||||
nicbrAnswer: map[string]string{brazilNICbrQuery: nicbrOverLimit},
|
|
||||||
reason: "over its query limit",
|
|
||||||
},
|
|
||||||
} {
|
|
||||||
t.Run(name, func(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
answers := map[string]string{
|
|
||||||
brazilIANAQuery: lacnicReferral,
|
|
||||||
brazilLACNICQuery: lacnicBrazilRecord,
|
|
||||||
}
|
|
||||||
maps.Copy(answers, test.nicbrAnswer)
|
|
||||||
|
|
||||||
var logs bytes.Buffer
|
|
||||||
|
|
||||||
nicbrAsks := 0
|
|
||||||
client := newNICbrClient(t, answers, &logs, &nicbrAsks)
|
|
||||||
|
|
||||||
info, err := client.LookupASN(t.Context(), brazilASN)
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("LookupASN: %v", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
if nicbrAsks == 0 {
|
|
||||||
t.Error("LookupASN did not ask NIC.br's server")
|
|
||||||
}
|
|
||||||
|
|
||||||
if info.OrgName != lacnicBrazilOrgName {
|
|
||||||
t.Errorf("LookupASN org name = %q, want LACNIC's %q",
|
|
||||||
info.OrgName, lacnicBrazilOrgName)
|
|
||||||
}
|
|
||||||
|
|
||||||
var logged struct {
|
|
||||||
Level string `json:"level"`
|
|
||||||
ASN int `json:"asn"`
|
|
||||||
}
|
|
||||||
|
|
||||||
err = json.Unmarshal(logs.Bytes(), &logged)
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("want one log line, got %q: %v", logs.String(), err)
|
|
||||||
}
|
|
||||||
|
|
||||||
if logged.Level != slog.LevelWarn.String() ||
|
|
||||||
logged.ASN != brazilASN ||
|
|
||||||
!strings.Contains(logs.String(), test.reason) {
|
|
||||||
t.Errorf("logged %q, want a warning for AS%d holding %q",
|
|
||||||
logs.String(), brazilASN, test.reason)
|
|
||||||
}
|
|
||||||
})
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestLookupASNSkipsNICbrAfterItsLimit looks up an AS that NIC.br
|
|
||||||
// registers three times with one client, while NIC.br answers that the
|
|
||||||
// client is over its query limit. The first lookup asks NIC.br. The
|
|
||||||
// second, a second before NICbrSkipTime has passed, does not. The third,
|
|
||||||
// once NICbrSkipTime has passed, asks again. Each keeps LACNIC's fields.
|
|
||||||
func TestLookupASNSkipsNICbrAfterItsLimit(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
nicbrAsks := 0
|
|
||||||
client := newNICbrClient(t, map[string]string{
|
|
||||||
brazilIANAQuery: lacnicReferral,
|
|
||||||
brazilLACNICQuery: lacnicBrazilRecord,
|
|
||||||
brazilNICbrQuery: nicbrOverLimit,
|
|
||||||
}, io.Discard, &nicbrAsks)
|
|
||||||
|
|
||||||
start := time.Date(2026, 10, 27, 0, 0, 0, 0, time.UTC)
|
|
||||||
now := start
|
|
||||||
client.Now = func() time.Time { return now }
|
|
||||||
|
|
||||||
for _, step := range []struct {
|
|
||||||
after time.Duration
|
|
||||||
wantAsks int
|
|
||||||
}{
|
|
||||||
{after: 0, wantAsks: 1},
|
|
||||||
{after: whois.NICbrSkipTime - time.Second, wantAsks: 1},
|
|
||||||
{after: whois.NICbrSkipTime, wantAsks: 2},
|
|
||||||
} {
|
|
||||||
now = start.Add(step.after)
|
|
||||||
|
|
||||||
info, err := client.LookupASN(t.Context(), brazilASN)
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("LookupASN after %v: %v", step.after, err)
|
|
||||||
}
|
|
||||||
|
|
||||||
if nicbrAsks != step.wantAsks {
|
|
||||||
t.Errorf("after %v NIC.br's server was asked %d times, want %d",
|
|
||||||
step.after, nicbrAsks, step.wantAsks)
|
|
||||||
}
|
|
||||||
|
|
||||||
if info.OrgName != lacnicBrazilOrgName {
|
|
||||||
t.Errorf("LookupASN after %v org name = %q, want LACNIC's %q",
|
|
||||||
step.after, info.OrgName, lacnicBrazilOrgName)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestLookupASNReportsARefusal looks up an AS IANA refers to RIPE when RIPE
|
// TestLookupASNReportsARefusal looks up an AS IANA refers to RIPE when RIPE
|
||||||
// refuses the query, with each of the error lines RIPE refuses with. The
|
// refuses the query, with each of the error lines RIPE refuses with. The
|
||||||
// error holds RIPE's line, so the log says why the lookup failed.
|
// error holds RIPE's line, so the log says why the lookup failed.
|
||||||
|
|||||||
Binary file not shown.
@@ -22,7 +22,7 @@ func TestGet(t *testing.T) {
|
|||||||
name: "AS1 Level 3",
|
name: "AS1 Level 3",
|
||||||
asn: 1,
|
asn: 1,
|
||||||
wantOK: true,
|
wantOK: true,
|
||||||
wantDesc: "Level 3 Parent LLC",
|
wantDesc: "Level 3 Parent, LLC",
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
name: "AS15169 Google",
|
name: "AS15169 Google",
|
||||||
@@ -36,10 +36,10 @@ func TestGet(t *testing.T) {
|
|||||||
wantOK: false,
|
wantOK: false,
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
name: "AS1251 with accented letters",
|
name: "AS0 IANA Reserved",
|
||||||
asn: 1251,
|
asn: 0,
|
||||||
wantOK: true,
|
wantOK: true,
|
||||||
wantDesc: "FUNDAÇÃO DE AMPARO À PESQUISA DO ESTADO SÃO PAULO",
|
wantDesc: "Internet Assigned Numbers Authority",
|
||||||
},
|
},
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -69,7 +69,7 @@ func TestGetDescription(t *testing.T) {
|
|||||||
asn int
|
asn int
|
||||||
want string
|
want string
|
||||||
}{
|
}{
|
||||||
{1, "Level 3 Parent LLC"},
|
{1, "Level 3 Parent, LLC"},
|
||||||
{999999999, ""},
|
{999999999, ""},
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -92,7 +92,8 @@ func TestGetHandle(t *testing.T) {
|
|||||||
asn int
|
asn int
|
||||||
want string
|
want string
|
||||||
}{
|
}{
|
||||||
{1, "LVLT-1"},
|
{1, "LVLT"},
|
||||||
|
{0, "IANA-RSVD"},
|
||||||
{999999999, ""},
|
{999999999, ""},
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -141,11 +142,6 @@ func TestSearch(t *testing.T) {
|
|||||||
query: "University",
|
query: "University",
|
||||||
wantMin: 10,
|
wantMin: 10,
|
||||||
},
|
},
|
||||||
{
|
|
||||||
name: "Search for São Paulo",
|
|
||||||
query: "São Paulo",
|
|
||||||
wantMin: 1,
|
|
||||||
},
|
|
||||||
{
|
{
|
||||||
name: "Non-existent org",
|
name: "Non-existent org",
|
||||||
query: "XYZABC123456",
|
query: "XYZABC123456",
|
||||||
@@ -193,13 +189,13 @@ func TestSearchEmptyQueryMatchesNothing(t *testing.T) {
|
|||||||
func TestSearchIgnoresCase(t *testing.T) {
|
func TestSearchIgnoresCase(t *testing.T) {
|
||||||
t.Parallel()
|
t.Parallel()
|
||||||
|
|
||||||
const cloudflareASN = 13335 // handle CLOUDFLARENET, "Cloudflare Inc."
|
const cloudflareASN = 13335 // handle CLOUDFLARENET, "Cloudflare, Inc."
|
||||||
|
|
||||||
isCloudflare := func(info asinfo.Info) bool {
|
isCloudflare := func(info asinfo.Info) bool {
|
||||||
return info.ASN == cloudflareASN
|
return info.ASN == cloudflareASN
|
||||||
}
|
}
|
||||||
|
|
||||||
for _, query := range []string{"cloudflarenet", "CLOUDFLARE INC."} {
|
for _, query := range []string{"cloudflarenet", "CLOUDFLARE, INC."} {
|
||||||
results, _ := asinfo.Search(query, asinfo.Total())
|
results, _ := asinfo.Search(query, asinfo.Total())
|
||||||
if !slices.ContainsFunc(results, isCloudflare) {
|
if !slices.ContainsFunc(results, isCloudflare) {
|
||||||
t.Errorf("Search(%q) did not find AS%d", query, cloudflareASN)
|
t.Errorf("Search(%q) did not find AS%d", query, cloudflareASN)
|
||||||
|
|||||||
+1
-1
@@ -2,7 +2,7 @@
|
|||||||
Package asinfo provides information about Autonomous Systems (AS).
|
Package asinfo provides information about Autonomous Systems (AS).
|
||||||
|
|
||||||
The package embeds a comprehensive database of AS numbers, handles, and descriptions
|
The package embeds a comprehensive database of AS numbers, handles, and descriptions
|
||||||
sourced from https://github.com/ipverse/as-metadata. The data is embedded at compile
|
sourced from https://github.com/ipverse/asn-info. The data is embedded at compile
|
||||||
time and provides fast, offline lookups.
|
time and provides fast, offline lookups.
|
||||||
|
|
||||||
Basic usage:
|
Basic usage:
|
||||||
|
|||||||
@@ -17,7 +17,7 @@ func ExampleGet() {
|
|||||||
func ExampleGetDescription() {
|
func ExampleGetDescription() {
|
||||||
desc := asinfo.GetDescription(13335)
|
desc := asinfo.GetDescription(13335)
|
||||||
fmt.Println(desc)
|
fmt.Println(desc)
|
||||||
// Output: Cloudflare Inc.
|
// Output: Cloudflare, Inc.
|
||||||
}
|
}
|
||||||
|
|
||||||
func ExampleSearch() {
|
func ExampleSearch() {
|
||||||
|
|||||||
@@ -1,34 +0,0 @@
|
|||||||
#!/bin/sh
|
|
||||||
# script/asupdate: rebuild the AS list built into RouteWatch from
|
|
||||||
# as.csv with cmd/asinfo-gen, which fails when it finds no AS. The new
|
|
||||||
# list replaces the old one only when the generator succeeds. Otherwise
|
|
||||||
# the old list stays as it was and the script exits non-zero.
|
|
||||||
set -eu
|
|
||||||
|
|
||||||
ROOT="$(cd "$(dirname "$0")/.." && pwd -P)"
|
|
||||||
|
|
||||||
AS_DATA="pkg/asinfo/asdata.json.gz"
|
|
||||||
# Next to AS_DATA, so the final mv is a rename on one file system.
|
|
||||||
AS_DATA_TMP="$AS_DATA.tmp"
|
|
||||||
|
|
||||||
main() {
|
|
||||||
cd "$ROOT"
|
|
||||||
|
|
||||||
json="$(mktemp)"
|
|
||||||
trap 'rm -f "$json" "$AS_DATA_TMP"' EXIT
|
|
||||||
# A signal exits through the EXIT trap too.
|
|
||||||
trap 'exit 1' HUP INT TERM
|
|
||||||
|
|
||||||
echo "Updating AS info data..."
|
|
||||||
if ! go run ./cmd/asinfo-gen >"$json"; then
|
|
||||||
echo "asupdate: the generator failed; $AS_DATA is unchanged" >&2
|
|
||||||
exit 1
|
|
||||||
fi
|
|
||||||
|
|
||||||
# -n leaves the time out of the file, so an unchanged list gives the
|
|
||||||
# same bytes.
|
|
||||||
gzip -n -c <"$json" >"$AS_DATA_TMP"
|
|
||||||
mv "$AS_DATA_TMP" "$AS_DATA"
|
|
||||||
}
|
|
||||||
|
|
||||||
main "$@"
|
|
||||||
Reference in New Issue
Block a user