sneak/routewatch
1
0
Fork 0
Code Issues 4 Pull Requests 2 Actions 6 Packages Projects Releases Wiki Activity

Compare commits

base: sneak/routewatch:555f01920d1fe3d4fdf019f4cf41eb25518bd207
Branches Tags
sneak/routewatch:main sneak/routewatch:next sneak/routewatch:issue-157-readme sneak/routewatch:prod sneak/routewatch:issue-27-cached-stats
...
compare: sneak/routewatch:next
Branches Tags
sneak/routewatch:next sneak/routewatch:issue-157-readme sneak/routewatch:prod sneak/routewatch:main sneak/routewatch:issue-27-cached-stats
11 Commits
555f01920d ... next
Author SHA1 Message Date
clawbot 38b43549b9 Make the routewatch tests pass on a loaded host (closes #213)
check / check (push) Waiting to run
Details 
Five prefix handler tests gave HandleMessage two seconds to apply a full
batch of 25,000 prefixes, which under the race detector on a loaded host
takes longer. A test now waits up to stepTimeout, 15 seconds, for each step
it expects.

The slowest tests filled production limits, such as the peering handler's
500,000 paths. The peering handler now takes its path limit, and the prefix
handler its limit on one write of the table's changes, as they take their
timer interval. Those tests give small limits, and the tests of the maximum
batch size lower it to one batch. This also fixes
#215.

The test helpers bound every wait, Stop included, so a test that hangs in
one fails with its own message.

Model: opus-5-5
 2026-10-28 01:44:47 +01:00
clawbot 5ea1bb17fe Keep working out the home page cards after startup (closes #211)
check / check (push) Canceled after 0s
Details 
Server.Start ran the loop that works out the home page's cards on the
context it was given. fx ends that context once start-up is done, so the
cards were worked out once and never again. The loop now runs on a
context that only Stop ends, as the other background parts do. Stop
waits for the loop to return, until its own context ends, before it
stops the HTTP server.

A new test starts a server in a synctest bubble and ends its context at
once. On the bubble's fake clock, the cards must be worked out again a
minute later, and not after Stop.

Model: opus-5-5
 2026-10-27 22:55:17 +01:00
clawbot 6d4f3e3419 Say in README why RouteWatch exists (closes #204)
check / check (push) Canceled after 0s
Details 
The Rationale section now gives the owner's reason for the program: it
is an internal API that our other services ask about IP addresses. It
sees which AS routes each address and which announced prefix holds it
from its own routing table. WHOIS details of each AS, kept in its
database, add to that.

What RIS Live does and does not give stays, shortened. Planned geojs
lookups are left out, because the README describes only what is built.

Model: opus-5-5
 2026-10-27 21:34:08 +01:00
clawbot 47aed8761e Give every JSON answer @meta, errors included (closes #201)
check / check (push) Canceled after 0s
Details 
A panic in a JSON route's handler answered 500 with an empty body,
because the router's recoverer sits outside the middleware that adds
`@meta`. `JSONValidationMiddleware` now wraps the handler in the same
recoverer, and answers the panic with the JSON error body.

`/status.json` and the health check answered chi's empty 405 to any
method they do not take. They now answer with the same JSON 405 as the
routes under `/api/v1` and `/ip`.

The home page's IP lookup card showed only the answer's `data`. It now
shows the whole answer, `@meta` included.

A table test checks `@meta` on every JSON route, in its successes and
its errors.

Model: opus-5-5
 2026-10-27 21:20:07 +01:00
clawbot e8d04f1304 Give every API value with a unit as a number (closes #199)
check / check (push) Canceled after 0s
Details 
A value with a unit is now a number, and its field name ends in the
unit, as the vendored API_POLICIES.md requires. @meta.execution_time
becomes execution_time_ms. In the stats answers, uptime,
connection_duration, memory_usage and whois_stats.current_interval
become uptime_seconds, connection_duration_seconds, memory_usage_bytes
and whois_stats.current_interval_seconds. connection_duration_seconds
is null while the feed is not connected. The status page formats the
numbers itself. REPO_POLICIES.md is re-vendored from the same
sneak/prompts commit.

Model: opus-5-5
 2026-10-27 20:19:01 +01:00
clawbot a671b8a0b0 Give every IP lookup result the same fields (closes #200)
check / check (push) Canceled after 0s
Details 
Every result of /ip/, /ip/{addr} and /api/v1/ip/{ip} now has every
field. A field without a value is null instead of left out, so an
address with no live route has all its route and AS fields set to null.
ptr is an empty list when there are no PTR names.

Two new fields, num_origins and origin_asns, give how many origin ASes
peers carry the netblock with, and which. The routing table's prefix
summary now lists them. asn still names the origin that most peers
carry.

first_seen is null, no longer the zero time, when the database has not
stored the netblock yet.

Model: opus-5-5
 2026-10-27 19:54:02 +01:00
clawbot fb5e98feab Trust reverse proxies at private addresses by default (closes #198)
check / check (push) Canceled after 0s
Details 
With TRUSTED_PROXIES unset, the server now trusts reverse proxies in the
private ranges of RFC 1918. A proxy such as traefik on a Docker network is
then believed. /ip/, the home page's curl card and the request log give the
client's address from X-Forwarded-For instead of the proxy's address.

A list in TRUSTED_PROXIES replaces the default. TRUSTED_PROXIES set but
empty trusts no proxy, so the config package reads it with os.LookupEnv.
A direct client at a public address still cannot choose its address with
a forged header.

Model: opus-5-5
 2026-10-27 19:18:34 +01:00
clawbot 281d162e27 Show figures of the routing table on the home page (closes #154)
check / check (push) Canceled after 0s
Details 
The home page gets cards for the newest and oldest kept route, the newest
and oldest updated AS, the AS with the most prefixes, the ASes with a
single prefix and fifteen well-known networks. Two more cards count the
bogon announcements and the prefixes with several origin ASes. They link
to the new pages /bogons and /multiple-origins, which list up to 1,000
entries each.

The new package internal/bogon tells reserved prefixes and AS numbers,
from the IANA special-purpose and address space registries. Table.Overview
walks the table and lets the read lock go every 10,000 prefixes. The
server works the cards out once a minute and serves that copy. A new index
on asns(last_seen) finds the newest and oldest updated AS.

Model: opus-5-5
 2026-10-27 18:27:01 +01:00
sneak e68be8254a Merge branch 'main' into next
check / check (push) Canceled after 0s
Details
2026-10-27 18:14:58 +01:00
clawbot 1c65656b58 Remove the per-peer route tables (closes #122)
check / check (push) Canceled after 0s
Details 
The routing table in memory now holds every route, so the two per-peer route
tables go, with every write, delete, count and read that served them. The
prefix handler applies each update to the table, writes the table's changes
every second, and stores each new prefix with its first-seen time. The peer
delete, its retry on the next flush and the updates held for it go too. The
types and errors of the server's answers move to internal/server.

The image's memory settings fit a full table of about 1 GB: GOMEMLIMIT of
2.5 GiB, a 512 MiB page cache for the writing connection, and SQLite heap
limits of 768 MiB and 1 GiB. The README describes the new table, its
database copy and the memory budget.

Model: opus-5-5
 2026-10-27 17:22:32 +01:00
sneak 20a29e389d Merge pull request 'next into main' (#87) from next into main
check / check (push) Canceled after 0s
Details 
Reviewed-on: #87
 2026-10-22 18:25:14 +02:00
54 changed files with 4488 additions and 5655 deletions
Expand all files Collapse all files
API_POLICIES.md
+20
View File
@@ -0,0 +1,20 @@
---
title: API Policies
last_modified: 2026-10-27
---
A value in an API response is never a string with a label or unit in it. A value
of a native type is sent as that type, and its unit goes in the field name, for
example `_ms`.
Wrong:
```json
{ "execution_time": "0 ms" }
```
Right:
```json
{ "execution_time_ms": 0 }
```
Dockerfile
+5 -4
View File
@@ -121,10 +121,11 @@ RUN chown -R routewatch:routewatch /app
ENV XDG_DATA_HOME=/var/lib ENV XDG_DATA_HOME=/var/lib
# Cap the Go heap at 1.5 GiB so the runtime collects harder before the # Cap the Go heap at 2.5 GiB so the runtime collects harder before the
# container's memory limit is reached. setpriv in the entrypoint preserves this # container's memory limit is reached. A full routing table takes about 1 GB
# the way it does XDG_DATA_HOME above. # of it (README "Memory"). setpriv in the entrypoint preserves this the way it
ENV GOMEMLIMIT=1536MiB # does XDG_DATA_HOME above.
ENV GOMEMLIMIT=2560MiB
# Cap glibc's malloc arenas. The SQLite C library allocates and frees millions # Cap glibc's malloc arenas. The SQLite C library allocates and frees millions
# of small page-cache chunks from many threads; glibc otherwise creates up to # of small page-cache chunks from many threads; glibc otherwise creates up to
README.md
+294 -149
View File
@@ -4,24 +4,43 @@ RouteWatch is an MIT-licensed Go daemon by @sneak that monitors the BGP routing
table in real time. It works like this: table in real time. It works like this:
- It streams BGP UPDATE messages from the RIPE RIS Live service. - It streams BGP UPDATE messages from the RIPE RIS Live service.
- It keeps a live routing table in SQLite. - It keeps the whole routing table in memory, with a copy in SQLite.
- It serves web pages and a JSON API for looking routes up. - It serves web pages and a JSON API for looking routes up.
## Rationale ## Rationale
RIPE RIS Live streams the BGP updates that RIPE's route collectors receive, as RouteWatch exists to serve our other services as an internal API for information
they happen, from the moment a client connects. It reports changes, not the about IP addresses.
routing table they add up to, so it cannot say which AS an IP address is routed
to now or which routes a prefix has. RouteWatch applies the feed's updates to a
routing table that it keeps in memory, with a copy in its own SQLite database.
It answers IP-to-AS lookups and prefix pages from that table.
The feed sends no table when a client connects. So the routing table holds only RouteWatch follows BGP routing table updates from RIPE RIS Live. RIS Live
the routes announced since RouteWatch first started. An IP-to-AS lookup finds streams the updates that RIPE's route collectors receive, as they happen, from
nothing for an address whose routes have not been announced since then. The the moment a client connects. It reports changes, not the routing table they add
routing table also misses updates RouteWatch drops when it falls behind, and up to. RouteWatch applies those changes to a routing table that it keeps in
updates sent while it is stopped. So a lookup or a prefix page can show a route memory, with a copy in its own SQLite database. From that table, RouteWatch sees
the feed has withdrawn. directly:
- which AS each IP address is routed to
- which announced prefix holds each IP address, and how big that prefix is
RouteWatch adds details of each AS from WHOIS, the public directory of the
internet registries. The details include the organisation that holds the AS.
RouteWatch keeps them in its database and looks them up again once they are 30
days old. An IP lookup answers from those stored details and never waits for
WHOIS.
So our own services can ask RouteWatch about an IP address, instead of each
working it out for itself.
RIS Live sends no routing table when a client connects. So the routing table
holds only the routes announced since RouteWatch first started. An IP lookup
finds no route for an address until a prefix that holds it is announced. The
routing table also misses these updates:
- those sent while RouteWatch is stopped
- those that RouteWatch drops when it falls behind
A lookup or a prefix page can therefore show a route that has since been
withdrawn.
## Features ## Features
@@ -70,12 +89,56 @@ one of its peers sends an update.
- `GET /` - Home page (HTML): headline figures, lookups by AS number and by IP - `GET /` - Home page (HTML): headline figures, lookups by AS number and by IP
address, and a search of AS names. A card shows the `curl` command that looks address, and a search of AS names. A card shows the `curl` command that looks
up the visitor's own address at `/ip/{addr}`, and that command's live answer. up the visitor's own address at `/ip/{addr}`, and that command's live answer.
Other cards show figures of the routing table, with links to the AS and prefix
pages:
- the newest and the oldest kept route, with its prefix, origin AS and time
- the AS last seen in an AS path most recently, and the one last seen
longest ago
- the AS that originates the most prefixes
- how many ASes originate a single prefix, and the most recently updated of
them
- the number of bogon announcements, linked to `/bogons`
- the number of prefixes with more than one origin AS, linked to
`/multiple-origins`
- well-known networks, such as Apple, Google, Cogent and MIT, each with its
number of prefixes and the time of its newest kept route, newest first
RouteWatch works these figures out once a minute, so they can be a minute
old.
- `GET /bogons` - The bogon announcements (HTML): the routes whose prefix or
origin AS is reserved. A prefix is reserved when it lies inside one of these:
- a block that the IANA special-purpose address registries mark as not
globally reachable
- the multicast space, `224.0.0.0/4` and `ff00::/8`, which the IANA IPv4 and
IPv6 address space registries list as Multicast
A reserved AS number is one of these:
- an AS number for private use or for documentation
- `AS23456`
- the last AS number of the 16-bit or the 32-bit range
Address space and AS numbers that the IANA has not handed out yet are not
counted. Each row shows the prefix, the origin AS, the number of peers and
why it is a bogon. The page shows at most 1,000 rows, in prefix order, and
says how many there are in all.
- `GET /multiple-origins` - The prefixes announced by more than one origin AS
(HTML). Each row shows the origin ASes and how many peers carry the prefix
with each. Like `/bogons`, it shows at most 1,000 rows, in prefix order, and
says how many there are in all.
- `GET /status` - HTML status dashboard, which reads `/api/v1/stats` - `GET /status` - HTML status dashboard, which reads `/api/v1/stats`
- `GET /status.json` - JSON statistics: feed, memory, database and route - `GET /status.json` - JSON statistics: feed, memory, database and route
figures. The route figures come from the routing table. `oldest_route` and figures. The route figures come from the routing table. `oldest_route` and
`newest_route` are the times of the oldest and the newest kept routes. `newest_route` are the times of the oldest and the newest kept routes.
`origins_without_kept_route` counts, across all prefixes, the origin ASes that `origins_without_kept_route` counts, across all prefixes, the origin ASes that
have no kept route. have no kept route. Other figures include these:
- `uptime_seconds`: the number of seconds since the RIS Live feed last
connected, or 0 when it has never connected
- `memory_usage_bytes`: the size of the Go heap in use, in bytes
- `whois_stats.current_interval_seconds`: the WHOIS fetcher's wait between
two fetches, in seconds, which can include a fraction of a second
- `GET /.well-known/healthcheck.json` - Health check (JSON): 200 when the - `GET /.well-known/healthcheck.json` - Health check (JSON): 200 when the
database answers and the RIS Live feed is connected, otherwise 503; database answers and the RIS Live feed is connected, otherwise 503;
`data.status` is `ok` or `error` and `data.checks` says which check failed `data.status` is `ok` or `error` and `data.checks` says which check failed
@@ -98,20 +161,45 @@ one of its peers sends an update.
### API v1 ### API v1
- `GET /api/v1/stats` - The `/status.json` figures plus connection, garbage - `GET /api/v1/stats` - The `/status.json` figures plus connection, garbage
collector, stream and handler queue statistics. `stream.route_churn_per_min` collector, stream and handler queue statistics. `connection_duration_seconds`
is the number of prefix announcements and withdrawals, from all peers, in the is the number of seconds the RIS Live feed has been connected, or null while
last full clock minute. it is not connected. `stream.route_churn_per_min` is the number of prefix
announcements and withdrawals, from all peers, in the last full clock minute.
- `GET /api/v1/ip/{ip}` - Looks up each address. `{ip}` is an IP address, a host - `GET /api/v1/ip/{ip}` - Looks up each address. `{ip}` is an IP address, a host
name, or a comma-separated list of at most 10 of these. `results` has one name, or a comma-separated list of at most 10 of these. `results` has one
entry for each address, with: entry for each address. Every entry has the same fields:
- `ip` and `ptr`: the address and its PTR names - `ip`: the address
- `netblock` and the fields beside it: the most specific prefix that holds - `ptr`: the address's PTR names, or an empty list when it has none
the address, and its origin AS with that AS's WHOIS details. For a prefix - `blocklists`: the public blocklists the address is on (see Blocklists), or
with several origin ASes, `asn` is the one that most peers carry. an empty list when it is on none
`last_seen` is the time of the prefix's newest kept route.
- `blocklists`: the public blocklists the address is on (see Blocklists) Every other field is about the address's live route. Each is null when the
address has no live route. The fields about the route:
- `netblock`: the most specific prefix that holds the address
- `mask_length`: the netblock's mask length
- `ip_version`: the netblock's IP version, 4 or 6
- `num_peers`: the number of peers that carry the netblock
- `asn`: the origin AS that the most peers carry the netblock with
- `num_origins`: the number of origin ASes that peers carry the netblock
with
- `origin_asns`: those origin ASes, in AS number order
- `first_seen`: when RouteWatch first saw the netblock, or null when the
database has not stored the netblock yet
- `last_seen`: the time of the netblock's newest kept route, or null when
the netblock keeps no route
The fields about the AS in `asn` come from the database. Each is also null
when the database does not have it:
- `handle`: the AS's handle
- `description`: the AS's description
- `as_name`: the AS's name in its WHOIS details
- `org_name`: the name of the organisation that holds the AS
- `org_id`: the organisation's ID at its registry
- `address`: the organisation's postal address
- `country_code`: the organisation's country code
- `abuse_email`: the address that takes abuse reports for the AS
- `rir`: the regional internet registry of the AS
An address with no live route has a result without the route and AS fields.
Of the addresses a host name resolves to, only public ones are looked up. A Of the addresses a host name resolves to, only public ones are looked up. A
name with none, such as one that resolves to a private address, is reported name with none, such as one that resolves to a private address, is reported
as `host name has no public address`, without its addresses. `errors` has as `host name has no public address`, without its addresses. `errors` has
@@ -145,10 +233,16 @@ one of its peers sends an update.
missing, empty or only spaces answers 400. missing, empty or only spaces answers 400.
A JSON answer is `{"status": "ok", "data": ...}`, or for an error A JSON answer is `{"status": "ok", "data": ...}`, or for an error
`{"status": "error", "error": {"msg": ..., "code": ...}}`, and carries an `{"status": "error", "error": {"msg": ..., "code": ...}}`. The health check is
`@meta` object with the time zone, the API version and the time the request the exception: its 503 answer is also `{"status": "ok", "data": ...}`, with
took. The health check is the exception: its 503 answer is also `error` in `data.status`. Every JSON answer also carries an `@meta` object:
`{"status": "ok", "data": ...}`, with `error` in `data.status`.
- `time_zone`: always `UTC`
- `api_version`: `1`
- `execution_time_ms`: the number of milliseconds the request took
Every value with a unit is a number whose field name ends in the unit, such as
`_ms`, `_seconds` or `_bytes`.
A request that runs past its time limit answers 504: the limit is 30 seconds, A request that runs past its time limit answers 504: the limit is 30 seconds,
and 4 seconds for the database reads behind `/status.json` and `/api/v1/stats`. and 4 seconds for the database reads behind `/status.json` and `/api/v1/stats`.
@@ -158,11 +252,17 @@ originates no prefix and has no record in the database or in the AS list built
into RouteWatch. AS 0 always answers 404, because it means no known origin. The into RouteWatch. AS 0 always answers 404, because it means no known origin. The
AS name search never lists it. A prefix with no live route answers 404 too. The AS name search never lists it. A prefix with no live route answers 404 too. The
HTML pages give these errors as plain text. Under `/api/v1` and `/ip/`, a path HTML pages give these errors as plain text. Under `/api/v1` and `/ip/`, a path
no route matches answers 404 with the message `Not found`, and `POST`, `PUT`, that matches no route answers 404 with the message `Not found`. A request with
`PATCH`, `DELETE`, `OPTIONS`, `TRACE` or `CONNECT` answers 405 with `POST`, `PUT`, `PATCH`, `DELETE`, `OPTIONS`, `TRACE` or `CONNECT` answers 405
`Method not allowed`. A 500 answer says only `Internal server error`, and a host with `Method not allowed` on these routes:
name that does not resolve only `could not resolve host name`; the detail goes
to the log. - the routes under `/api/v1` and `/ip/`
- `/status.json`
- `/.well-known/healthcheck.json`
A 500 answer says only `Internal server error`. The error for a host name that
does not resolve says only `could not resolve host name`. The detail goes to the
log.
Every answer carries `Strict-Transport-Security`, `Content-Security-Policy`, Every answer carries `Strict-Transport-Security`, `Content-Security-Policy`,
`X-Frame-Options`, `X-Content-Type-Options`, `Referrer-Policy` and `X-Frame-Options`, `X-Content-Type-Options`, `Referrer-Policy` and
@@ -173,10 +273,15 @@ so each answer's policy names a random nonce made for that answer, and the
browser applies or runs only the blocks that carry it. The pages therefore use browser applies or runs only the blocks that carry it. The pages therefore use
no `style` attributes, which a nonce cannot cover. no `style` attributes, which a nonce cannot cover.
An origin AS of 0 (`origin_asn` in a route, `asn` in an IP lookup, the key `0` An origin AS of 0 means the route has no known origin. Its AS path ends in an AS
in a prefix's `origins`) means the route has no known origin: its AS path ends set of several members, any of which may have originated it. The 0 appears in
in an AS set of several members, any of which may have originated it. The pages these fields:
show such a route as having no known origin.
- `origin_asn` in a route
- `asn` and `origin_asns` in an IP lookup
- the key `0` in a prefix's `origins`
The pages show such a route as having no known origin.
## Blocklists ## Blocklists
@@ -248,6 +353,7 @@ routewatch/
│ └── streamdumper/ # Debug utility for raw stream output │ └── streamdumper/ # Debug utility for raw stream output
├── internal/ ├── internal/
│ ├── blocklist/ # Public IP blocklists: downloads and lookups │ ├── blocklist/ # Public IP blocklists: downloads and lookups
│ ├── bogon/ # Reserved prefixes and AS numbers
│ ├── routewatch/ # Core application logic │ ├── routewatch/ # Core application logic
│ ├── server/ # HTTP server and handlers │ ├── server/ # HTTP server and handlers
│ ├── database/ # SQLite storage layer │ ├── database/ # SQLite storage layer
@@ -268,31 +374,36 @@ routewatch/
### Component Relationships ### Component Relationships
`internal/routewatch/app.go` builds and starts these parts:
``` ```
┌─────────────────────────────────────────────────────────────────┐ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐
│ RouteWatch │ │ Streamer │───▶│ Handlers │───▶│ Database │
│ (internal/routewatch/app.go - main orchestrator) │ │ │ │ │ │ │
├─────────────────────────────────────────────────────────────────┤ │ RIS Live │ │ - ASHandler │ │ SQLite with │
│ │ │ HTTP stream │ │ - PeerHandler│ │ WAL mode │
│ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ │ │ client │ │ - PrefixHdlr │ │ │
│ │ Streamer │───▶│ Handlers │───▶│ Database │ │ │ │ │ - PeeringHdlr│ │ Tables: │
│ │ │ │ │ │ │ │ └──────────────┘ └──────┬───────┘ │ - asns │
│ │ RIS Live │ │ - ASHandler │ │ SQLite with │ │ │ │ - prefixes │
│ │ HTTP stream │ │ - PeerHandler│ │ WAL mode │ │ ▼ │ - routes │
│ │ client │ │ - PrefixHdlr │ │ │ │ ┌──────────────┐ │ - route_peers│
│ │ │ │ - PeeringHdlr│ │ Tables: │ │ │ Routing table│───▶│ - peerings │
│ └──────────────┘ └──────────────┘ │ - asns │ │ │ in memory │ │ - bgp_peers │
│ │ - prefixes │ │ └──────────────┘ └──────────────┘
│ ┌──────────────┐ ┌──────────────┐ │ - live_routes│ │ ▲ ▲
│ │ Server │───▶│ Handlers │───▶│ - peerings │ │ ┌──────────────┐ ┌──────┴───────┐ │
│ │ │ │ │ │ - bgp_peers │ │ │ Server │───▶│ Handlers │───────────┘
│ │ Chi router │ │ Status, API │ └──────────────┘ │ │ │ │ │
│ │ port 8080 │ │ AS, Prefix │ │ │ Chi router │ │ Status, API │
│ └──────────────┘ └──────────────┘ │ │ port 8080 │ │ AS, Prefix │
│ │ └──────────────┘ └──────────────┘
└─────────────────────────────────────────────────────────────────┘
``` ```
The prefix handler keeps the routing table in memory current. It writes the
table's changes to the database every second. The web handlers read routes from
the routing table, and everything else from the database.
### Execution Flow ### Execution Flow
1. **Startup** (`cmd/routewatch/main.go` → `internal/routewatch/cli.go`) 1. **Startup** (`cmd/routewatch/main.go` → `internal/routewatch/cli.go`)
@@ -301,15 +412,17 @@ routewatch/
2. **Initialization** (`internal/routewatch/app.go`) 2. **Initialization** (`internal/routewatch/app.go`)
- Database created with SQLite schema (WAL mode, page caches as in "Memory") - Database created with SQLite schema (WAL mode, page caches as in "Memory")
- The routing table read from the database into memory, before the streamer - The routing table read from the `routes` table into memory, before the
connects streamer connects. A full table takes about 25 seconds to read.
- Message handlers registered with the streamer - Message handlers registered with the streamer
- HTTP server started on configured port - HTTP server started on configured port
3. **Message Processing Pipeline** 3. **Message Processing Pipeline**
``` ```
RIS Live Stream → JSON Parser → Message Dispatcher → Handler Queues → Batch Writers → SQLite RIS Live Stream → JSON Parser → Message Dispatcher → Handler Queues
├→ Batch Writers → SQLite
└→ PrefixHandler → Routing Table in Memory → SQLite, every second
``` ```
- Streamer connects to `ris-live.ripe.net` via HTTP - Streamer connects to `ris-live.ripe.net` via HTTP
@@ -324,24 +437,41 @@ routewatch/
4. **Handler Details** 4. **Handler Details**
- **ASHandler**: Tracks all ASNs seen in AS paths - **ASHandler**: Tracks all ASNs seen in AS paths
- **PeerHandler**: Records BGP peer information - **PeerHandler**: Records BGP peer information
- **PrefixHandler**: Maintains live routing table (upserts on announcement, - **PrefixHandler**: Keeps the routing table in memory current. It applies
deletes on withdrawal or on an announcement whose AS path is empty or each update to the table as it handles it:
holds AS 0 anywhere, deletes all of a peer's routes when its BGP session - An announcement gives the peer a route to the prefix.
goes down). Routes are kept per peer IP, so when one peer IP has sessions - A withdrawal removes the peer's route. So does an announcement whose
on two collectors, a session going down on one also removes the routes the AS path is empty or holds AS 0 anywhere.
other still carries, until they are announced again. It also applies each - A peer whose BGP session goes down leaves every prefix.
update to the routing table in memory, and writes the table's changes to
the database every second. A write takes at most 20,000 changed prefix and Every second it writes the changed prefix and origin AS pairs to the
origin AS pairs, and the rest follow in the next writes. `routes` table. A write takes at most 20,000 pairs, and the rest follow
in the next writes. It also stores each new prefix with the time it was
first seen. Peers are known by their IP address. A peer IP can have
sessions on two collectors. A session going down on one then also
removes the routes the other still carries, until they are announced
again.
- **PeeringHandler**: Extracts AS peering relationships from AS paths - **PeeringHandler**: Extracts AS peering relationships from AS paths
5. **HTTP Request Flow** 5. **HTTP Request Flow**
``` ```
Request → Chi Router → Middleware (timeout, logging) → Handler → Database Query → Response Request → Chi Router → Middleware (timeout, logging) → Handler → Routing Table or Database → Response
``` ```
### Key Design Patterns ### Key Design Patterns
- **Routing Table in Memory**: `internal/routetable` holds the whole routing
table. For each prefix and origin AS it keeps:
- every peer whose current route to the prefix has that origin AS, one bit
per peer;
- the newest two of those routes in full, with AS path and time.
Of the other routes, only the peer is kept. The prefix handler writes the
changed pairs to the `routes` table every second, one row per prefix and
origin AS. Startup reads that table back into memory before the streamer
connects. A crash loses only the changes not yet written.
- **Batched Writes**: The handlers write the feed's updates to the database in - **Batched Writes**: The handlers write the feed's updates to the database in
batches for performance batches for performance
- **Backpressure**: Probabilistic message dropping when queues exceed 50% - **Backpressure**: Probabilistic message dropping when queues exceed 50%
@@ -373,13 +503,14 @@ asns(asn PRIMARY KEY, handle, description,
prefixes_v4(prefix PRIMARY KEY, first_seen) WITHOUT ROWID prefixes_v4(prefix PRIMARY KEY, first_seen) WITHOUT ROWID
prefixes_v6(prefix PRIMARY KEY, first_seen) WITHOUT ROWID prefixes_v6(prefix PRIMARY KEY, first_seen) WITHOUT ROWID
-- Live routing tables (one per IP version), one route per peer and prefix -- The routing table, one row per prefix and origin AS that some peer
live_routes_v4(id INTEGER PRIMARY KEY, prefix, mask_length, origin_asn, -- carries: the peer set, and the newest two routes in full
peer_ip, as_path, next_hop, last_updated, routes(prefix, origin_asn, peers,
UNIQUE(peer_ip, prefix)) newest_peer, newest_as_path, newest_time,
live_routes_v6(id INTEGER PRIMARY KEY, prefix, mask_length, origin_asn, older_peer, older_as_path, older_time,
peer_ip, as_path, next_hop, last_updated, PRIMARY KEY (prefix, origin_asn)) WITHOUT ROWID
UNIQUE(peer_ip, prefix)) -- The peers of the routes table, each with its number in the peer sets
route_peers(number PRIMARY KEY, address UNIQUE)
-- Relationship tracking -- Relationship tracking
peerings(id PRIMARY KEY, as_a, as_b, first_seen, last_seen, peerings(id PRIMARY KEY, as_a, as_b, first_seen, last_seen,
@@ -403,21 +534,31 @@ Configuration is handled via environment variables and OS-specific paths:
| `PORT` | `8080` | HTTP server port, a whole number from 1 to 65535 | | `PORT` | `8080` | HTTP server port, a whole number from 1 to 65535 |
| `DEBUG` | (empty) | Set to `routewatch` for debug logging | | `DEBUG` | (empty) | Set to `routewatch` for debug logging |
| `XDG_DATA_HOME` | `/var/lib` (in the Docker image) | Base of the state directory; must be an absolute path | | `XDG_DATA_HOME` | `/var/lib` (in the Docker image) | Base of the state directory; must be an absolute path |
| `GOMEMLIMIT` | `1536MiB` (in the Docker image) | Go soft memory limit; see Memory | | `GOMEMLIMIT` | `2560MiB` (in the Docker image) | Go soft memory limit; see Memory |
| `MALLOC_ARENA_MAX` | `2` (in the Docker image) | glibc malloc arena cap, a positive whole number; see Memory | | `MALLOC_ARENA_MAX` | `2` (in the Docker image) | glibc malloc arena cap, a positive whole number; see Memory |
| `TRUSTED_PROXIES` | (empty) | Addresses and CIDR ranges of trusted proxies; see below | | `TRUSTED_PROXIES` | the RFC 1918 ranges | Addresses and CIDR ranges of trusted proxies; see below |
A variable that is set to an invalid value stops the start with an error and a A variable that is set to an invalid value stops the start with an error and a
non-zero exit. An empty variable counts as unset. non-zero exit. An empty variable counts as unset. `TRUSTED_PROXIES` is the one
exception: set but empty, it makes the server trust no proxy.
`TRUSTED_PROXIES` lists the reverse proxies whose `X-Forwarded-For` the server `TRUSTED_PROXIES` names the reverse proxies whose `X-Forwarded-For` the server
believes, comma-separated, such as `10.0.0.0/8,192.0.2.1`. For a request from believes:
one of them, the request log and `/ip/` use the last address in
`X-Forwarded-For` that is not a trusted proxy: each proxy adds the address it - When it is not set, the server trusts the private ranges of RFC 1918:
received the request from to the end, so the entries before it are whatever the - `10.0.0.0/8`
client sent. For a request from any other address they use the address the - `172.16.0.0/12`
request came from, whatever forwarding headers it carries. `X-Real-IP` and - `192.168.0.0/16`
`True-Client-IP` are not read. - When it lists addresses and CIDR ranges, comma-separated, such as
`10.0.0.0/8,192.0.2.1`, the server trusts those instead.
- When it is set but empty (`TRUSTED_PROXIES=`), the server trusts no proxy.
For a request from a trusted proxy, the request log and `/ip/` use the last
address in `X-Forwarded-For` that is not a trusted proxy. Each proxy adds the
address it received the request from to the end, so the entries before it are
whatever the client sent. For a request from any other address they use the
address the request came from, whatever forwarding headers it carries.
`X-Real-IP` and `True-Client-IP` are not read.
State directory (the database and the blocklist copies): State directory (the database and the blocklist copies):
@@ -429,28 +570,37 @@ State directory (the database and the blocklist copies):
## Memory ## Memory
The daemon holds a live routing table, so its memory grows with the size of the The daemon keeps the whole routing table in memory. The image sets ceilings that
data it tracks. The image sets ceilings that keep it inside a 5 GiB container. keep it inside a 5 GiB container.
Budget: Budget:
- Go heap: a 1.5 GiB soft limit (`GOMEMLIMIT=1536MiB`, set in the image). - Go heap: a 2.5 GiB soft limit (`GOMEMLIMIT=2560MiB`, set in the image).
- SQLite: 1 GiB of page cache for the one connection every write uses, which - The routing table takes about 700 bytes per prefix and origin AS. A full
keeps the pages the route writes touch from one batch to the next, and 16 MiB table has about 1.4 million of these pairs, so it takes about 1 GB.
for each of the 9 reading connections, whose cache SQLite empties whenever - The rest of the heap takes up to about 0.5 GiB. Most of it is the
another connection has written: about 1.15 GiB in all. A 1.5 GiB soft heap handlers' batches and queues, described below.
limit sits above the caches, so it does not shrink them, and a 2 GiB hard heap - The remaining 1 GiB is room for the garbage collector. With much less, it
limit caps the C library. would run almost all the time.
- glibc allocator: the SQLite C library runs on glibc `malloc`, which frees - SQLite: 656 MiB of page cache.
page-cache chunks back to per-arena free lists rather than to the kernel, so - The connection that does every write has 512 MiB. It keeps the pages the
process RSS tracks the high-water mark of those arenas, not SQLite's live writes touch from one write to the next. With a full routing table, the
heap. glibc creates up to eight arenas per core, so on a many-core host the `routes` table takes about 450 MB, so its pages fit.
retained memory — and thus RSS — grows with the core count. The image sets - Each of the 9 reading connections has 16 MiB. SQLite empties a reading
`MALLOC_ARENA_MAX=2` to bound it; the two-arena cap costs nothing here because connection's cache whenever another connection has written.
database writes are already serialized. - A 768 MiB soft heap limit sits above the caches, so it does not shrink
them. A 1 GiB hard heap limit caps the C library.
- glibc allocator: the SQLite C library allocates memory with glibc `malloc`.
- glibc keeps freed page-cache chunks in per-arena free lists instead of
returning them to the kernel. So the process's RSS follows the high-water
mark of those arenas, not SQLite's live heap.
- glibc creates up to eight arenas per core, so on a many-core host RSS
grows with the core count.
- The image sets `MALLOC_ARENA_MAX=2` to bound it. Two arenas cost nothing
here, because database writes already run one at a time.
- About 0.2 GiB for everything else in the runtime. - About 0.2 GiB for everything else in the runtime.
The ceilings add up to 3.7 GiB: the Go heap's 1.5 GiB, SQLite's 2 GiB and the The ceilings add up to 3.7 GiB: the Go heap's 2.5 GiB, SQLite's 1 GiB and the
0.2 GiB for the rest. 0.2 GiB for the rest.
Run the container with a memory limit of 5 GiB and swap disabled: Run the container with a memory limit of 5 GiB and swap disabled:
@@ -462,38 +612,36 @@ docker run --memory=5g --memory-swap=5g ...
or the equivalent in your deployment tool. This leaves headroom above the or the equivalent in your deployment tool. This leaves headroom above the
ceilings for spikes and the kernel page cache. ceilings for spikes and the kernel page cache.
Override the Go soft limit by setting `GOMEMLIMIT` in the environment (for To replace the image's Go soft limit, set `GOMEMLIMIT` in the environment, for
example `-e GOMEMLIMIT=1GiB`); this replaces the image default. example `-e GOMEMLIMIT=3GiB`. A limit close to the routing table's size makes
`MALLOC_ARENA_MAX` can be overridden the same way, but raising it lets RSS climb the garbage collector run almost all the time. `MALLOC_ARENA_MAX` can be
again on a many-core host. overridden the same way, but raising it lets RSS climb again on a many-core
host.
What happens at each limit: What happens at each limit:
- Go soft limit: as the heap approaches `GOMEMLIMIT`, the runtime runs garbage - Go soft limit: as the heap approaches `GOMEMLIMIT`, the runtime runs garbage
collection more aggressively rather than growing further. collection more aggressively rather than growing further.
- SQLite: at the 1.5 GiB soft heap limit it recycles its page cache instead of - SQLite: at the 768 MiB soft heap limit it recycles its page cache instead of
allocating more; at the 2 GiB hard heap limit a statement fails with an allocating more. At the 1 GiB hard heap limit a statement fails with an
out-of-memory error, and the handler logs it and drops that batch. The process out-of-memory error, and the handler logs it. The prefix handler writes the
keeps running. routing table's changes again on a later write. The other writes drop their
batch. The process keeps running.
- Handler queues: each of the four handler queues holds at most 20,000 messages. - Handler queues: each of the four handler queues holds at most 20,000 messages.
When a queue fills, the streamer drops messages instead of blocking, except a When a queue fills, the streamer drops messages instead of blocking, except a
`STATE` message, which it holds, in feed order, until there is room: a few `STATE` message, which it holds, in feed order, until there is room: a few
hundred bytes each, about ten a second. hundred bytes each, about ten a second.
- Handler batches: while the AS, peer or prefix handler writes a batch to the - Handler batches: while the AS, peer or prefix handler writes a batch to the
database, the updates that arrive collect in its next batch, which stops at database, the updates that arrive collect in its next batch. That batch stops
ten times the batch size: 300,000 updates for the AS handler at ten times the batch size:
(`asnBatchMaxSize`, about 10 MiB), 100,000 for the peer handler - 300,000 updates for the AS handler (`asnBatchMaxSize`, about 10 MiB);
(`peerBatchMaxSize`, about 10 MiB) and 250,000 for the prefix handler - 100,000 for the peer handler (`peerBatchMaxSize`, about 10 MiB);
(`prefixBatchMaxSize`, about 60 MiB). At that size the handler takes nothing - 250,000 for the prefix handler (`prefixBatchMaxSize`, about 15 MiB).
more off its queue until the write ends, so the queue fills and the streamer
drops messages as above. When the delete of a peer's live routes after its At that size the handler takes nothing more off its queue until the write
session went down fails, the prefix handler keeps the updates the peer sent ends. The queue then fills, and the streamer drops messages as above. With
after the down for the next write, which retries the delete first; on top of the batches being written, the three handlers' batches take at most about
its 250,000 it keeps at most the last 25,000 of them (`prefixBatchSize`, about 0.1 GiB of the Go heap.
6 MiB) and drops and logs the older ones, so while the delete keeps failing
the other peers' updates still come off the queue. With the batches being
written, the three handlers' batches take at most about 0.25 GiB of the Go
heap's 1.5 GiB.
With `DEBUG=routewatch` the daemon logs a `System stats` line every 60 seconds With `DEBUG=routewatch` the daemon logs a `System stats` line every 60 seconds
with the goroutine count and Go memory figures. with the goroutine count and Go memory figures.
@@ -504,12 +652,11 @@ What the [upaas](https://git.eeqj.de/sneak/upaas) app needs:
- Container port: `8080`. - Container port: `8080`.
- Volume: one, at container path `/var/lib/berlin.sneak.app.routewatch`. - Volume: one, at container path `/var/lib/berlin.sneak.app.routewatch`.
- Environment: `TRUSTED_PROXIES`, set to the address the reverse proxy in front - Environment: nothing. The server trusts a reverse proxy at an RFC 1918 address
of the app connects from, as the container sees it; without it the request log without `TRUSTED_PROXIES` (see Configuration). Set it only when the proxy
and `/ip/` give the proxy's address for every request. Nothing else is needed: connects from any other address. Leave `XDG_DATA_HOME`, `GOMEMLIMIT` and
leave `XDG_DATA_HOME`, `GOMEMLIMIT` and `MALLOC_ARENA_MAX` at the image's `MALLOC_ARENA_MAX` at the image's values. `DEBUG=routewatch` is optional and
values. `DEBUG=routewatch` is optional and adds the `System stats` memory line adds the `System stats` memory line to the log.
to the log.
- Memory Limit: `5g`, the 5 GiB limit from Memory above. upaas sets no swap - Memory Limit: `5g`, the 5 GiB limit from Memory above. upaas sets no swap
limit, so on a host with swap Docker allows the same amount of swap again. limit, so on a host with swap Docker allows the same amount of swap again.
- Health check: the image's `HEALTHCHECK` requests - Health check: the image's `HEALTHCHECK` requests
@@ -521,21 +668,19 @@ What the [upaas](https://git.eeqj.de/sneak/upaas) app needs:
- Stop grace: the container needs 9 seconds between SIGTERM and SIGKILL. upaas - Stop grace: the container needs 9 seconds between SIGTERM and SIGKILL. upaas
gives 10 (`stopTimeoutSeconds` in its `internal/docker/client.go`), as gives 10 (`stopTimeoutSeconds` in its `internal/docker/client.go`), as
`docker stop` does by default. The daemon allows 8 seconds for the handlers to `docker stop` does by default. The daemon allows 8 seconds for the handlers to
write what they hold and for the requests in progress to finish, then stops write what they hold and for the requests in progress to finish. Then it stops
the HTTP server, closes the database and exits with status 0. A longer grace the HTTP server, closes the database and exits with status 0. A longer grace
changes nothing. What is not written after 8 seconds is lost, and the streamer changes nothing. What is not written after 8 seconds is lost, and the streamer
and each handler log how much. When the prefix handler has fallen behind the and each handler log how much.
feed, a stop loses far more: its batch is full at 250,000 updates and the - The prefix handler's last write takes at most 20,000 of the routing
streamer's worker for it waits for room, so the streamer's stop takes the table's changed pairs. A full-table peer's session going down changes
whole 8 seconds and every handler then skips its last write. Lost then are the about a million pairs, which take about a minute to write. A stop in that
messages left on the four handlers' queues, what the AS, peer and peering minute leaves the rest unwritten.
handlers hold, the prefix handler's batch, and the write it has in progress, - A lost withdrawal or peer down leaves its routes in the `routes` table,
up to another 250,000 updates, which take about 20 seconds and are cut off by and the next start reads them back. They stay until the peer sends another
the exit; withdrawals and peer downs are among them. A route withdrawn in them update for the prefix, or its session goes down.
stays in the live route table until its peer announces or withdraws the prefix - When the 8 seconds run out the database is not closed. When the database
again or its session goes down. When the 8 seconds run out the database is not next opens, SQLite rolls back any write the exit cut off.
closed, and SQLite rolls back a write the exit cut off when the database next
opens.
## Development ## Development
REPO_POLICIES.md
+92 -22
View File
@@ -1,6 +1,6 @@
--- ---
title: Repository Policies title: Repository Policies
last_modified: 2026-10-18 last_modified: 2026-10-27
--- ---
This document covers repository structure, tooling, and workflow standards. Code This document covers repository structure, tooling, and workflow standards. Code
@@ -12,6 +12,7 @@ style conventions are in separate documents:
- [JavaScript](https://git.eeqj.de/sneak/prompts/raw/branch/main/prompts/CODE_STYLEGUIDE_JS.md) - [JavaScript](https://git.eeqj.de/sneak/prompts/raw/branch/main/prompts/CODE_STYLEGUIDE_JS.md)
- [Python](https://git.eeqj.de/sneak/prompts/raw/branch/main/prompts/CODE_STYLEGUIDE_PYTHON.md) - [Python](https://git.eeqj.de/sneak/prompts/raw/branch/main/prompts/CODE_STYLEGUIDE_PYTHON.md)
- [Go HTTP Server Conventions](https://git.eeqj.de/sneak/prompts/raw/branch/main/prompts/GO_HTTP_SERVER_CONVENTIONS.md) - [Go HTTP Server Conventions](https://git.eeqj.de/sneak/prompts/raw/branch/main/prompts/GO_HTTP_SERVER_CONVENTIONS.md)
- [API Policies](https://git.eeqj.de/sneak/prompts/raw/branch/main/prompts/API_POLICIES.md)
--- ---
@@ -174,27 +175,32 @@ style conventions are in separate documents:
`script/cibuild`. `script/lint`, `script/test`, `script/cibuild` and `script/cibuild`. `script/lint`, `script/test`, `script/cibuild` and
`script/docker` are the four, and there is no fifth — `script/check` runs the `script/docker` are the four, and there is no fifth — `script/check` runs the
two gate phases and `script/fmt-check`, and builds no image of its own. In two gate phases and `script/fmt-check`, and builds no image of its own. In
every stage of the `Dockerfile`, `ARG CACHEBUST` comes right after the steps every stage of the `Dockerfile`, every step that lints, tests or compiles
that install system packages and download Go's modules, or right after `FROM` comes after `ARG CACHEBUST`. `ARG CACHEBUST` comes right after these steps:
in a stage with neither, and every step that lints, tests or compiles comes - the steps that install system packages
after it. A step that installs anything else as well, such as a - the steps that download Go's modules
`script/bootstrap` that also installs node packages, comes after it too. A new - the step that runs `script/bootstrap` to install a Python `test` phase's
value reruns every `RUN` step after that line; `COPY` steps are compared by packages from `Pipfile.lock`
content and need nothing. Docker keeps the layers above the line until the
base image, their own lines or the files they copy change. A stage without the A stage with none of these steps declares `ARG CACHEBUST` right after
`ARG` line is served from cache on an unchanged tree, so a repository `FROM`. A step that installs anything else as well, such as a
re-vendors the scripts and adds the `ARG` lines in one commit. The scripts do `script/bootstrap` that also installs node packages, comes after it too. A
not pass `--no-cache`: it reruns every step too, but it also builds the layers new value reruns every `RUN` step after that line; `COPY` steps are compared
above the line again and empties every cache mount, so Go's build cache and by content and need nothing. Docker keeps the layers above the line until
golangci-lint's, which the canonical Go `Dockerfile` below mounts so that they the base image, their own lines or the files they copy change. A stage
survive between check builds, would start empty every time. A bare without the `ARG` line is served from cache on an unchanged tree, so a
`docker build .` is not evidence that anything ran: a sub-second build repository re-vendors the scripts and adds the `ARG` lines in one commit.
reporting success is a cache hit, not a result. Never invalidate by pruning — The scripts do not pass `--no-cache`: it reruns every step too, but it also
`docker builder prune` and friends destroy a build cache shared with every builds the layers above the line again and empties every cache mount, so
other build on the host. When a check is added or changed, prove it works by Go's build cache and golangci-lint's, which the canonical Go `Dockerfile`
planting a defect it must catch and watching the run fail on it, then revert below mounts so that they survive between check builds, would start empty
the defect. A green run alone shows neither that the check ran nor that it every time. A bare `docker build .` is not evidence that anything ran: a
covers what it should. sub-second build reporting success is a cache hit, not a result. Never
invalidate by pruning — `docker builder prune` and friends destroy a build
cache shared with every other build on the host. When a check is added or
changed, prove it works by planting a defect it must catch and watching the
run fail on it, then revert the defect. A green run alone shows neither that
the check ran nor that it covers what it should.
- **The gate phases are separate stages, and the build stage depends on both.** - **The gate phases are separate stages, and the build stage depends on both.**
The lint phase is based on the `golangci/golangci-lint` image (pinned by The lint phase is based on the `golangci/golangci-lint` image (pinned by
@@ -309,6 +315,9 @@ style conventions are in separate documents:
a package's findings only when the package's files and those of every a package's findings only when the package's files and those of every
package it imports, `go.mod`, the linters and their settings in package it imports, `go.mod`, the linters and their settings in
`.golangci.yml`, and the golangci-lint version are all unchanged. `.golangci.yml`, and the golangci-lint version are all unchanged.
- In a Python `test` phase, the layer of the step that runs
`script/bootstrap` to install the packages from `Pipfile.lock`, as
described for that phase below.
- If the project uses `//go:embed` directives that reference build artifacts - If the project uses `//go:embed` directives that reference build artifacts
(e.g. a web frontend compiled in a separate stage), the lint phase must (e.g. a web frontend compiled in a separate stage), the lint phase must
create placeholder files so the embed directives resolve. Example: create placeholder files so the embed directives resolve. Example:
@@ -363,6 +372,67 @@ style conventions are in separate documents:
`--build-arg VERSION=...`, as `script/docker` and `script/cibuild` already `--build-arg VERSION=...`, as `script/docker` and `script/cibuild` already
do. do.
- **A Python `test` phase that installs its packages from `Pipfile.lock` does it
above `ARG CACHEBUST`, so check builds keep them.** The steps are
`COPY script/ script/`, `COPY Pipfile Pipfile.lock ./` and
`RUN script/bootstrap`, with `ARG CACHEBUST` right after them. Docker keeps
what that `RUN` installed until the base image, `script/`, `Pipfile`,
`Pipfile.lock` or a line up to and including the `RUN` changes. The tests run
after `ARG CACHEBUST`, on every check build. In every other stage, a step that
installs Python packages comes after `ARG CACHEBUST`.
- When `script/bootstrap` runs in that step, it installs nothing beyond
system packages, pipenv and its dependencies, and the packages from
`Pipfile.lock`. It installs no Python there: the tests use the base
image's Python, which the image's hash pins. If it installs anything else
there, the step goes after `ARG CACHEBUST`.
- Every external reference must be pinned by hash, so `script/bootstrap`
installs pipenv and its dependencies by hash. It does so with
`pip install --require-hashes -r` and a requirements file in `script/`
that lists each of them with `==` and its `--hash`. pip refuses to install
a dependency the file does not list.
- It then runs `pipenv install --deploy`, with `--dev` when the tests need
the development packages. `--deploy` installs only what `Pipfile.lock`
names, checks every package against the lockfile's hashes, and fails when
the lockfile is out of date with `Pipfile`.
- A kept layer holds nothing a new install would refuse: pip checked pipenv
and its dependencies, and pipenv checked the lockfile's packages, against
their hashes. pipenv creates its virtual environment with the `virtualenv`
package. The packages that `virtualenv` adds come from wheels inside that
package, so the hash pip checked for it covers them. Two things in the
layer are not fixed by a hash. A package with no ready-built wheel for the
image is built from source. pip downloads the build tools without a hash
check, so a kept layer's build of such a package may differ from a new
one. And system packages keep the versions the package manager gave when
the layer was built; a new install may get newer ones.
- As with Go's modules, the packages are kept in a layer, not a cache mount.
A cache mount is shared by every build on the host, and nothing checks its
files again.
In this example, `script/bootstrap` finds `make` and `git` already
installed, installs pipenv and its dependencies by hash, and runs
`pipenv install --dev --deploy`:
```dockerfile
# Test phase. The 90-second timeout covers both pytest runs.
# python:3.x-slim, YYYY-MM-DD
FROM python@sha256:... AS test
# make and git are base tools that script/bootstrap installs when they
# are missing. Installed here, they stay in their own layer when
# script/ changes.
RUN apt-get update \
&& apt-get install -y --no-install-recommends make git \
&& rm -rf /var/lib/apt/lists/*
WORKDIR /app
COPY script/ script/
COPY Pipfile Pipfile.lock ./
RUN script/bootstrap
ARG CACHEBUST
COPY . .
RUN timeout 90 sh -c 'pipenv run pytest || { \
echo "--- Rerunning with -v for details ---"; \
pipenv run pytest -v; exit 1; }'
```
- Every repo should have a Gitea Actions workflow (`.gitea/workflows/`) that - Every repo should have a Gitea Actions workflow (`.gitea/workflows/`) that
runs `script/cibuild` on push, and checks out the repo as its only other step, runs `script/cibuild` on push, and checks out the repo as its only other step,
with `persist-credentials: false`: `script/cibuild` needs no token, and with `persist-credentials: false`: `script/cibuild` needs no token, and
TODO.md
+60 -5
View File
@@ -25,14 +25,70 @@ are measured by a benchmark, not yet on the live feed.
# Next Step # Next Step
Run `next` on the live feed, as README "Running under upaas" describes, for the Run `next` on the live feed, as README "Running under upaas" describes, until
final check https://git.eeqj.de/sneak/routewatch/issues/96 names: whether the the routing table stops growing. Post its size and memory figures on
prefix handler still drops updates after the first hour. If it does, a new issue https://git.eeqj.de/sneak/routewatch/issues/122. The same run makes the final
check https://git.eeqj.de/sneak/routewatch/issues/96 names: whether the prefix
handler still drops updates after the first hour. If it does, a new issue
follows with the numbers. Then an audit of the code at the commit that run follows with the numbers. Then an audit of the code at the commit that run
tested. tested.
# Completed Steps # Completed Steps
- 2026-10-27: the tests of `internal/routewatch` no longer fail on a loaded
host. Five of them gave `HandleMessage` two seconds to apply a full prefix
batch, which under the race detector can take longer. A test now waits up to
15 seconds for each step it expects. The tests of the handlers' limits give
the handlers small ones, so the package no longer runs past its 90-second
limit (closes #213, #215)
- 2026-10-27: the server works out the home page's cards every minute until it
stops. It used to work them out only once, at startup: the end of the context
that `Start` is given stopped the refreshes (closes #211)
- 2026-10-27: README "Rationale" says why RouteWatch exists: to answer our other
services' questions about IP addresses, from its own routing table and the
WHOIS details it keeps. It still says what RIS Live does not give (closes
#204)
- 2026-10-27: every JSON answer carries `@meta`. A panic on a JSON route answers
500 with the JSON error body. `/status.json` and the health check answer 405
with the JSON error body to any method they do not take. The 500 and the 405
used to have an empty body. The home page's IP lookup shows the whole JSON
answer, `@meta` included. It used to show only the answer's `data`, or only
the message of an error answer. A test checks `@meta` on every JSON route, in
its successes and its errors (closes #201)
- 2026-10-27: every value in a JSON answer that has a unit is a number, with the
unit at the end of its field name. `@meta.execution_time` is now
`execution_time_ms`. The stats' `uptime`, `connection_duration` and
`memory_usage` are now `uptime_seconds`, `connection_duration_seconds` and
`memory_usage_bytes`. `whois_stats.current_interval` is now
`current_interval_seconds`. They used to be strings such as "12 ms",
"1h18m34s" or "1.1 GB". `connection_duration_seconds` is null while the feed
is not connected. `API_POLICIES.md` is vendored from `sneak/prompts`, and
`REPO_POLICIES.md` re-vendored (closes #199)
- 2026-10-27: every result of the IP lookup has the same fields. A field without
a value is null. It used to be left out. An address with no live route has
every route and AS field set to null, and `ptr` is an empty list when there
are no PTR names. New fields give the number of origin ASes of the netblock,
in `num_origins`, and which they are, in `origin_asns` (closes #200)
- 2026-10-27: with `TRUSTED_PROXIES` unset, the server trusts reverse proxies at
the private addresses of RFC 1918. `/ip/`, the home page's `curl` card and the
request log then give the client's address from `X-Forwarded-For`. They used
to give the proxy's address. A list in `TRUSTED_PROXIES` replaces the default,
and `TRUSTED_PROXIES` set but empty trusts no proxy (closes #198)
- 2026-10-27: the home page has cards with figures of the routing table: the
newest and oldest kept route, the newest and oldest updated AS, the AS with
the most prefixes, the ASes with a single prefix, and fifteen well-known
networks. Two cards count the bogon announcements and the prefixes with more
than one origin AS, and link to the new pages `/bogons` and
`/multiple-origins` that list them. A new package, `internal/bogon`, tells
which prefixes and AS numbers are reserved. The server works the figures out
once a minute (closes #154)
- 2026-10-27: the per-peer route tables are gone, with the code that wrote,
counted and read them. The prefix handler applies each update to the routing
table in memory and writes the table's changes every second. It also stores
each new prefix with its first-seen time. The image's memory limits are set
for a full routing table of about 1 GB: `GOMEMLIMIT` is 2.5 GiB, the writing
connection's page cache 512 MiB, and SQLite's heap limits 768 MiB and 1 GiB
(closes #122)
- 2026-10-27: the pages, the API and the stats read every route from the routing - 2026-10-27: the pages, the API and the stats read every route from the routing
table in memory. For a prefix with several origin ASes, the IP lookup gives table in memory. For a prefix with several origin ASes, the IP lookup gives
the one most peers carry. Its `last_seen` is the newest kept route's time. The the one most peers carry. Its `last_seen` is the newest kept route's time. The
@@ -277,5 +333,4 @@ tested.
# Future Steps # Future Steps
- the rest of https://git.eeqj.de/sneak/routewatch/issues/122: removing the None.
per-peer route tables
internal/bogon/bogon.go
+202
View File
@@ -0,0 +1,202 @@
// Package bogon tells whether a prefix or an AS number is reserved: set aside
// for a special purpose, so that no network should announce it on the
// internet. A route is a bogon announcement when its prefix is reserved or its
// origin AS is.
//
// Most reserved prefixes come from the IANA special-purpose address registries
// for IPv4 and IPv6. The two multicast blocks, 224.0.0.0/4 and ff00::/8, come
// from the IANA IPv4 and IPv6 address space registries. The reserved AS
// numbers come from the IANA special-purpose AS numbers registry:
// - the AS numbers for private use
// - the AS numbers for documentation
// - AS 23456, which stands in for a 32-bit AS number
// - the last AS number of the 16-bit and of the 32-bit range
//
// Address space and AS numbers that the IANA has not handed out yet are not
// counted.
package bogon
import (
"net/netip"
"strconv"
)
// Prefix returns why prefix is reserved, and true, when it lies inside a
// reserved block. The most specific registry block that holds prefix decides,
// because some blocks that may be announced lie inside reserved ones. It
// returns "" and false when prefix is not reserved.
func Prefix(prefix netip.Prefix) (string, bool) {
var holder *block
for i, b := range registryBlocks {
inside := b.prefix.Bits() <= prefix.Bits() && b.prefix.Contains(prefix.Addr())
if inside && (holder == nil || b.prefix.Bits() > holder.prefix.Bits()) {
holder = &registryBlocks[i]
}
}
if holder == nil || !holder.reserved {
return "", false
}
return holder.name + " (" + holder.prefix.String() + ")", true
}
// ASN returns why asn is reserved, and true, when it is. It returns "" and
// false when asn is not reserved.
func ASN(asn int) (string, bool) {
for _, r := range reservedASRanges {
if asn < r.first || asn > r.last {
continue
}
if r.first == r.last {
return r.name + " (AS" + strconv.Itoa(asn) + ")", true
}
return r.name + " (AS" + strconv.Itoa(r.first) + "-AS" +
strconv.Itoa(r.last) + ")", true
}
return "", false
}
// The names that the registries give to more than one row.
const (
privateUse = "Private-Use"
documentation = "Documentation"
benchmarking = "Benchmarking"
ietfProtocolAssignments = "IETF Protocol Assignments"
portControlAnycast = "Port Control Protocol Anycast"
natTraversalAnycast = "Traversal Using Relays around NAT Anycast"
nat64Discovery = "NAT64/DNS64 Discovery"
amt = "AMT"
as112DirectDelegation = "Direct Delegation AS112 Service"
ipv4IPv6Translation = "IPv4-IPv6 Translat."
multicast = "Multicast"
)
// block is one row of an IANA address registry.
type block struct {
prefix netip.Prefix
name string
// reserved says that no network should announce the row's addresses on
// the internet.
reserved bool
}
// reservedBlock returns a reserved row: a multicast block, or a special-purpose
// row whose Globally Reachable column is False.
func reservedBlock(prefix, name string) block {
return block{prefix: netip.MustParsePrefix(prefix), name: name, reserved: true}
}
// reachableBlock returns a special-purpose row whose Globally Reachable column
// is True or N/A. Such a row matters only inside a reserved block, whose
// addresses it lets be announced.
func reachableBlock(prefix, name string) block {
return block{prefix: netip.MustParsePrefix(prefix), name: name}
}
// registryBlocks are the rows of the IANA special-purpose address registries,
// without the deprecated ones:
// - https://www.iana.org/assignments/iana-ipv4-special-registry/
// - https://www.iana.org/assignments/iana-ipv6-special-registry/
//
// They also hold the multicast blocks, which the IANA address space registries
// list as Multicast:
// - https://www.iana.org/assignments/ipv4-address-space/
// - https://www.iana.org/assignments/ipv6-address-space/
//
//nolint:gochecknoglobals // a fixed list, parsed once
var registryBlocks = []block{
reservedBlock("0.0.0.0/8", "This network"),
reservedBlock("0.0.0.0/32", "This host on this network"),
reservedBlock("10.0.0.0/8", privateUse),
reservedBlock("100.64.0.0/10", "Shared Address Space"),
reservedBlock("127.0.0.0/8", "Loopback"),
reservedBlock("169.254.0.0/16", "Link Local"),
reservedBlock("172.16.0.0/12", privateUse),
reservedBlock("192.0.0.0/24", ietfProtocolAssignments),
reservedBlock("192.0.0.0/29", "IPv4 Service Continuity Prefix"),
reservedBlock("192.0.0.8/32", "IPv4 dummy address"),
reachableBlock("192.0.0.9/32", portControlAnycast),
reachableBlock("192.0.0.10/32", natTraversalAnycast),
reservedBlock("192.0.0.170/32", nat64Discovery),
reservedBlock("192.0.0.171/32", nat64Discovery),
reservedBlock("192.0.2.0/24", "Documentation (TEST-NET-1)"),
reachableBlock("192.31.196.0/24", "AS112-v4"),
reachableBlock("192.52.193.0/24", amt),
reservedBlock("192.88.99.2/32", "6a44-relay anycast address"),
reservedBlock("192.168.0.0/16", privateUse),
reachableBlock("192.175.48.0/24", as112DirectDelegation),
reservedBlock("198.18.0.0/15", benchmarking),
reservedBlock("198.51.100.0/24", "Documentation (TEST-NET-2)"),
reservedBlock("203.0.113.0/24", "Documentation (TEST-NET-3)"),
reservedBlock("224.0.0.0/4", multicast),
reservedBlock("240.0.0.0/4", "Reserved"),
reservedBlock("255.255.255.255/32", "Limited Broadcast"),
reservedBlock("::1/128", "Loopback Address"),
reservedBlock("::/128", "Unspecified Address"),
reservedBlock("::ffff:0:0/96", "IPv4-mapped Address"),
reachableBlock("64:ff9b::/96", ipv4IPv6Translation),
reservedBlock("64:ff9b:1::/48", ipv4IPv6Translation),
reservedBlock("100::/64", "Discard-Only Address Block"),
reservedBlock("100:0:0:1::/64", "Dummy IPv6 Prefix"),
reservedBlock("2001::/23", ietfProtocolAssignments),
reachableBlock("2001::/32", "TEREDO"),
reachableBlock("2001:1::1/128", portControlAnycast),
reachableBlock("2001:1::2/128", natTraversalAnycast),
reachableBlock("2001:1::3/128", "DNS-SD Service Registration Protocol Anycast"),
reservedBlock("2001:2::/48", benchmarking),
reachableBlock("2001:3::/32", amt),
reachableBlock("2001:4:112::/48", "AS112-v6"),
reachableBlock("2001:20::/28", "ORCHIDv2"),
reachableBlock("2001:30::/28", "Drone Remote ID Protocol Entity Tags (DETs) Prefix"),
reservedBlock("2001:db8::/32", documentation),
reachableBlock("2002::/16", "6to4"),
reachableBlock("2620:4f:8000::/48", as112DirectDelegation),
reservedBlock("3fff::/20", documentation),
reservedBlock("5f00::/16", "Segment Routing (SRv6) SIDs"),
reservedBlock("fc00::/7", "Unique-Local"),
reservedBlock("fe80::/10", "Link-Local Unicast"),
reservedBlock("ff00::/8", multicast),
}
// asRange is a range of reserved AS numbers, first to last.
type asRange struct {
first, last int
name string
}
// The reserved AS numbers of the IANA special-purpose AS numbers registry.
const (
asTrans = 23456
firstDocumentationAS16 = 64496
lastDocumentationAS16 = 64511
firstPrivateUseAS16 = 64512
lastPrivateUseAS16 = 65534
lastAS16 = 65535
firstDocumentationAS32 = 65536
lastDocumentationAS32 = 65551
firstPrivateUseAS32 = 4200000000
lastPrivateUseAS32 = 4294967294
lastAS32 = 4294967295
)
// reservedASRanges are the reserved rows of the IANA special-purpose AS numbers
// registry, https://www.iana.org/assignments/iana-as-numbers-special-registry/.
// AS 0 is left out: RouteWatch uses it for a route with no known origin. AS 112
// is left out because the AS112 project announces routes with it.
//
//nolint:gochecknoglobals // a fixed list
var reservedASRanges = []asRange{
{first: asTrans, last: asTrans, name: "AS_TRANS"},
{first: firstDocumentationAS16, last: lastDocumentationAS16, name: documentation},
{first: firstPrivateUseAS16, last: lastPrivateUseAS16, name: privateUse},
{first: lastAS16, last: lastAS16, name: "Last 16-bit AS number"},
{first: firstDocumentationAS32, last: lastDocumentationAS32, name: documentation},
{first: firstPrivateUseAS32, last: lastPrivateUseAS32, name: privateUse},
{first: lastAS32, last: lastAS32, name: "Last 32-bit AS number"},
}
internal/bogon/bogon_test.go
+95
View File
@@ -0,0 +1,95 @@
package bogon_test
import (
"net/netip"
"strconv"
"testing"
"git.eeqj.de/sneak/routewatch/internal/bogon"
)
func TestPrefix(t *testing.T) {
t.Parallel()
tests := []struct {
prefix string
// want is why the prefix is reserved, or "" when it is not.
want string
}{
{"10.1.2.0/24", "Private-Use (10.0.0.0/8)"},
{"10.0.0.0/8", "Private-Use (10.0.0.0/8)"},
{"192.0.2.0/24", "Documentation (TEST-NET-1) (192.0.2.0/24)"},
{"2001:db8:1::/48", "Documentation (2001:db8::/32)"},
{"fd00::/8", "Unique-Local (fc00::/7)"},
// Multicast space, from the IANA address space registries.
{"224.0.0.0/4", "Multicast (224.0.0.0/4)"},
{"239.255.0.0/16", "Multicast (224.0.0.0/4)"},
{"ff00::/8", "Multicast (ff00::/8)"},
{"ff02::/16", "Multicast (ff00::/8)"},
// The most specific block names the reason.
{"0.0.0.0/32", "This host on this network (0.0.0.0/32)"},
{"2001:2::/48", "Benchmarking (2001:2::/48)"},
// Blocks that may be announced, inside reserved blocks.
{"192.0.0.9/32", ""},
{"2001:4:112::/48", ""},
// Blocks that may be announced, inside no reserved block.
{"192.175.48.0/24", ""},
{"2002::/16", ""},
// A prefix that holds a reserved block is not inside it.
{"192.0.0.0/16", ""},
{"0.0.0.0/0", ""},
// Prefixes anyone may announce.
{"8.8.8.0/24", ""},
{"2001:4860::/32", ""},
}
for _, tc := range tests {
t.Run(tc.prefix, func(t *testing.T) {
t.Parallel()
reason, reserved := bogon.Prefix(netip.MustParsePrefix(tc.prefix))
if reason != tc.want || reserved != (tc.want != "") {
t.Errorf("Prefix(%s) = %q, %v; want %q", tc.prefix, reason,
reserved, tc.want)
}
})
}
}
func TestASN(t *testing.T) {
t.Parallel()
tests := []struct {
asn int
// want is why the AS number is reserved, or "" when it is not.
want string
}{
{23456, "AS_TRANS (AS23456)"},
{64496, "Documentation (AS64496-AS64511)"},
{64512, "Private-Use (AS64512-AS65534)"},
{65534, "Private-Use (AS64512-AS65534)"},
{65535, "Last 16-bit AS number (AS65535)"},
{65551, "Documentation (AS65536-AS65551)"},
{4200000000, "Private-Use (AS4200000000-AS4294967294)"},
{4294967295, "Last 32-bit AS number (AS4294967295)"},
// AS 0 means no known origin, and the AS112 project uses AS 112.
{0, ""},
{112, ""},
// Not handed out yet, which is not counted.
{65552, ""},
{15169, ""},
{64495, ""},
}
for _, tc := range tests {
t.Run(strconv.Itoa(tc.asn), func(t *testing.T) {
t.Parallel()
reason, reserved := bogon.ASN(tc.asn)
if reason != tc.want || reserved != (tc.want != "") {
t.Errorf("ASN(%d) = %q, %v; want %q", tc.asn, reason, reserved,
tc.want)
}
})
}
}
internal/config/config.go
+22 -4
View File
@@ -30,6 +30,14 @@ const (
maxPort = 65535 maxPort = 65535
) )
// The private address ranges of RFC 1918. The HTTP server trusts a reverse
// proxy in any of them when TRUSTED_PROXIES is not set.
const (
privateRange10 = "10.0.0.0/8"
privateRange172 = "172.16.0.0/12"
privateRange192 = "192.168.0.0/16"
)
var ( var (
errRelativeXDGDataHome = errors.New("XDG_DATA_HOME must be an absolute path") errRelativeXDGDataHome = errors.New("XDG_DATA_HOME must be an absolute path")
errUnsupportedOS = errors.New("unsupported operating system") errUnsupportedOS = errors.New("unsupported operating system")
@@ -159,11 +167,21 @@ func getPort() (int, error) {
return int(port), nil return int(port), nil
} }
// getTrustedProxies returns the addresses and CIDR ranges listed in // getTrustedProxies returns the reverse proxies the HTTP server trusts. When
// TRUSTED_PROXIES, such as "10.0.0.0/8,192.0.2.1", or none when it is not set. // TRUSTED_PROXIES is not set, they are the private ranges of RFC 1918. When it
// An address is returned as a range holding only that address. // is set, they are the addresses and CIDR ranges it lists, such as
// "10.0.0.0/8,192.0.2.1", and none when it is empty. An address is returned as
// a range holding only that address.
func getTrustedProxies() ([]netip.Prefix, error) { func getTrustedProxies() ([]netip.Prefix, error) {
value := os.Getenv("TRUSTED_PROXIES") value, set := os.LookupEnv("TRUSTED_PROXIES")
if !set {
return []netip.Prefix{
netip.MustParsePrefix(privateRange10),
netip.MustParsePrefix(privateRange172),
netip.MustParsePrefix(privateRange192),
}, nil
}
if value == "" { if value == "" {
return nil, nil return nil, nil
} }
internal/config/config_internal_test.go
+40 -5
View File
@@ -2,6 +2,7 @@ package config
import ( import (
"net/netip" "net/netip"
"os"
"runtime" "runtime"
"slices" "slices"
"testing" "testing"
@@ -46,25 +47,47 @@ func TestNewRefusesInvalidPort(t *testing.T) {
} }
} }
func TestNewReadsTrustedProxies(t *testing.T) { // TestNewTrustsPrivateRangesByDefault checks that with TRUSTED_PROXIES unset
// the trusted proxies are the private ranges of RFC 1918. Before the fix no
// proxy was trusted, so a reverse proxy at a private address was taken for the
// client (https://git.eeqj.de/sneak/routewatch/issues/198).
func TestNewTrustsPrivateRangesByDefault(t *testing.T) {
t.Setenv("PORT", "") t.Setenv("PORT", "")
t.Setenv("XDG_DATA_HOME", "") t.Setenv("XDG_DATA_HOME", "")
// t.Setenv puts the variable back when the test ends.
t.Setenv("TRUSTED_PROXIES", "") t.Setenv("TRUSTED_PROXIES", "")
err := os.Unsetenv("TRUSTED_PROXIES")
if err != nil {
t.Fatal(err)
}
cfg, err := New() cfg, err := New()
if err != nil { if err != nil {
t.Fatalf("New() without TRUSTED_PROXIES: %v", err) t.Fatalf("New() without TRUSTED_PROXIES: %v", err)
} }
if len(cfg.TrustedProxies) != 0 { want := []netip.Prefix{
t.Errorf("New() without TRUSTED_PROXIES: TrustedProxies = %v, want none", netip.MustParsePrefix("10.0.0.0/8"),
cfg.TrustedProxies) netip.MustParsePrefix("172.16.0.0/12"),
netip.MustParsePrefix("192.168.0.0/16"),
} }
if !slices.Equal(cfg.TrustedProxies, want) {
t.Errorf("New() without TRUSTED_PROXIES: TrustedProxies = %v, want %v",
cfg.TrustedProxies, want)
}
}
// TestNewReadsTrustedProxies checks that a list in TRUSTED_PROXIES replaces
// the default, and that TRUSTED_PROXIES set but empty trusts no proxy.
func TestNewReadsTrustedProxies(t *testing.T) {
t.Setenv("PORT", "")
t.Setenv("XDG_DATA_HOME", "")
t.Setenv("TRUSTED_PROXIES", "10.0.0.0/8, 192.0.2.1,2001:db8::/32") t.Setenv("TRUSTED_PROXIES", "10.0.0.0/8, 192.0.2.1,2001:db8::/32")
cfg, err = New() cfg, err := New()
if err != nil { if err != nil {
t.Fatalf("New() with TRUSTED_PROXIES: %v", err) t.Fatalf("New() with TRUSTED_PROXIES: %v", err)
} }
@@ -77,6 +100,18 @@ func TestNewReadsTrustedProxies(t *testing.T) {
if !slices.Equal(cfg.TrustedProxies, want) { if !slices.Equal(cfg.TrustedProxies, want) {
t.Errorf("TrustedProxies = %v, want %v", cfg.TrustedProxies, want) t.Errorf("TrustedProxies = %v, want %v", cfg.TrustedProxies, want)
} }
t.Setenv("TRUSTED_PROXIES", "")
cfg, err = New()
if err != nil {
t.Fatalf("New() with TRUSTED_PROXIES set but empty: %v", err)
}
if len(cfg.TrustedProxies) != 0 {
t.Errorf("New() with TRUSTED_PROXIES set but empty: TrustedProxies = %v, "+
"want none", cfg.TrustedProxies)
}
} }
func TestNewRefusesInvalidTrustedProxies(t *testing.T) { func TestNewRefusesInvalidTrustedProxies(t *testing.T) {
internal/database/counts.go
+9 -100
View File
@@ -6,11 +6,10 @@ import (
"sync" "sync"
) )
// liveCounts holds the running row counts and the prefix distribution that the // liveCounts holds the running row counts that the stats endpoints report.
// stats endpoints report. They are seeded once at startup from the tables and // They are seeded once at startup from the tables and then adjusted on every
// then adjusted on every write, so a stats read serves them from memory instead // write, so a stats read serves them from memory instead of running a query
// of running a query over the tables. The COUNT(*) scans (issue 27) and then // over the tables. The COUNT(*) scans (issue 27) grew with the database until
// the prefix distribution query (issue 30) each grew with the database until
// they took the whole request timeout and made /api/v1/stats return 500. // they took the whole request timeout and made /api/v1/stats return 500.
// //
// A single mutex guards all fields so the stats reader takes a consistent // A single mutex guards all fields so the stats reader takes a consistent
@@ -23,20 +22,11 @@ type liveCounts struct {
prefixesV6 int prefixesV6 int
peerings int peerings int
peers int peers int
routesV4 int
routesV6 int
// The prefix distribution: for each mask length, the number of distinct
// prefixes that have at least one live route.
distributionV4 [ipv4Bits + 1]int
distributionV6 [ipv6Bits + 1]int
} }
// seed sets every count to the value read from the tables at startup. It runs // seed sets every count to the value read from the tables at startup. It runs
// before any writer, so it needs no coordination with the adjust methods. // before any writer, so it needs no coordination with the adjust methods.
func (c *liveCounts) seed( func (c *liveCounts) seed(asns, prefixesV4, prefixesV6, peerings, peers int) {
asns, prefixesV4, prefixesV6, peerings, peers, routesV4, routesV6 int,
distributionV4, distributionV6 []PrefixDistribution,
) {
c.mu.Lock() c.mu.Lock()
defer c.mu.Unlock() defer c.mu.Unlock()
@@ -45,16 +35,6 @@ func (c *liveCounts) seed(
c.prefixesV6 = prefixesV6 c.prefixesV6 = prefixesV6
c.peerings = peerings c.peerings = peerings
c.peers = peers c.peers = peers
c.routesV4 = routesV4
c.routesV6 = routesV6
for _, entry := range distributionV4 {
addAtMaskLength(c.distributionV4[:], entry.MaskLength, entry.Count)
}
for _, entry := range distributionV6 {
addAtMaskLength(c.distributionV6[:], entry.MaskLength, entry.Count)
}
} }
// addASNs adds n to the ASN count. // addASNs adds n to the ASN count.
@@ -86,41 +66,7 @@ func (c *liveCounts) addPeers(n int) {
c.mu.Unlock() c.mu.Unlock()
} }
// addRoutes adds to the IPv4 and IPv6 live-route counts. Deletions pass // fill copies the counts into a Stats, including the derived total, under a
// negative values.
func (c *liveCounts) addRoutes(v4, v6 int) {
c.mu.Lock()
c.routesV4 += v4
c.routesV6 += v6
c.mu.Unlock()
}
// addToDistribution adds n to the IPv4 and IPv6 prefix distributions once for
// each listed mask length. A write lists the mask lengths of the prefixes it
// gave their first live route with n = 1, and of the prefixes it left with no
// live route with n = -1.
func (c *liveCounts) addToDistribution(maskLengthsV4, maskLengthsV6 []int, n int) {
c.mu.Lock()
defer c.mu.Unlock()
for _, maskLength := range maskLengthsV4 {
addAtMaskLength(c.distributionV4[:], maskLength, n)
}
for _, maskLength := range maskLengthsV6 {
addAtMaskLength(c.distributionV6[:], maskLength, n)
}
}
// addAtMaskLength adds n to counts[maskLength]. A mask length the array has no
// entry for is ignored, so a malformed route cannot crash the daemon.
func addAtMaskLength(counts []int, maskLength, n int) {
if maskLength >= 0 && maskLength < len(counts) {
counts[maskLength] += n
}
}
// fill copies the counts into a Stats, including the derived totals, under a
// single read lock so the reader sees one consistent snapshot. // single read lock so the reader sees one consistent snapshot.
func (c *liveCounts) fill(s *Stats) { func (c *liveCounts) fill(s *Stats) {
c.mu.RLock() c.mu.RLock()
@@ -132,26 +78,6 @@ func (c *liveCounts) fill(s *Stats) {
s.Prefixes = c.prefixesV4 + c.prefixesV6 s.Prefixes = c.prefixesV4 + c.prefixesV6
s.Peerings = c.peerings s.Peerings = c.peerings
s.Peers = c.peers s.Peers = c.peers
s.IPv4Routes = c.routesV4
s.IPv6Routes = c.routesV6
s.LiveRoutes = c.routesV4 + c.routesV6
s.IPv4PrefixDistribution = distributionList(c.distributionV4[:])
s.IPv6PrefixDistribution = distributionList(c.distributionV6[:])
}
// distributionList lists the mask lengths that have at least one prefix, in
// ascending order, the way the distribution query returns them.
func distributionList(counts []int) []PrefixDistribution {
var list []PrefixDistribution
for maskLength, count := range counts {
if count > 0 {
list = append(list,
PrefixDistribution{MaskLength: maskLength, Count: count})
}
}
return list
} }
// countRows returns the number of rows in the named table. It is used only at // countRows returns the number of rows in the named table. It is used only at
@@ -167,9 +93,8 @@ func (d *Database) countRows(ctx context.Context, table string) (int, error) {
return n, nil return n, nil
} }
// seedCounts reads the current row counts and prefix distribution from the // seedCounts reads the current row counts from the tables into the in-memory
// tables into the in-memory counters. It runs once at startup, before the // counters. It runs once at startup, before the streamer begins writing.
// streamer begins writing.
func (d *Database) seedCounts(ctx context.Context) error { func (d *Database) seedCounts(ctx context.Context) error {
asns, err := d.countRows(ctx, "asns") asns, err := d.countRows(ctx, "asns")
if err != nil { if err != nil {
@@ -196,23 +121,7 @@ func (d *Database) seedCounts(ctx context.Context) error {
return err return err
} }
routesV4, err := d.countRows(ctx, tableLiveRoutesV4) d.counts.seed(asns, prefixesV4, prefixesV6, peerings, peers)
if err != nil {
return err
}
routesV6, err := d.countRows(ctx, tableLiveRoutesV6)
if err != nil {
return err
}
distributionV4, distributionV6, err := d.GetPrefixDistributionContext(ctx)
if err != nil {
return err
}
d.counts.seed(asns, prefixesV4, prefixesV6, peerings, peers, routesV4, routesV6,
distributionV4, distributionV6)
return nil return nil
} }
internal/database/counts_internal_test.go
+25 -577
View File
@@ -3,62 +3,23 @@ package database
import ( import (
"context" "context"
"fmt" "fmt"
"slices"
"sync" "sync"
"testing" "testing"
"time" "time"
"git.eeqj.de/sneak/routewatch/internal/config" "git.eeqj.de/sneak/routewatch/internal/config"
"git.eeqj.de/sneak/routewatch/internal/logger" "git.eeqj.de/sneak/routewatch/internal/logger"
"github.com/google/uuid"
) )
// testV4PeerIP is the peer the IPv4 routes in these tests are learned from. // testV4PeerIP is the peer the IPv4 rows in these tests are learned from.
const testV4PeerIP = "192.0.2.1" const testV4PeerIP = "192.0.2.1"
// testV6Prefix is the prefix of the IPv6 routes in these tests. // testV6Prefix is the IPv6 prefix of these tests.
const testV6Prefix = "2001:db8::/32" const testV6Prefix = "2001:db8::/32"
// mkV4Route builds an IPv4 live route with its mask length taken from the
// prefix.
func mkV4Route(t *testing.T, prefix string, asn int, ts time.Time) *LiveRoute {
t.Helper()
maskLength, err := prefixMaskLength(prefix)
if err != nil {
t.Fatalf("prefixMaskLength(%s): %v", prefix, err)
}
return &LiveRoute{
Prefix: prefix,
MaskLength: maskLength,
IPVersion: ipVersionV4,
OriginASN: asn,
PeerIP: testV4PeerIP,
ASPath: []int{asn},
NextHop: "192.0.2.254",
LastUpdated: ts,
}
}
// mkV6Route builds an IPv6 live route.
func mkV6Route(prefix string, asn int, ts time.Time) *LiveRoute {
return &LiveRoute{
Prefix: prefix,
MaskLength: 32,
IPVersion: ipVersionV6,
OriginASN: asn,
PeerIP: "2001:db8::1",
ASPath: []int{asn},
NextHop: "2001:db8::ffff",
LastUpdated: ts,
}
}
// TestLiveCountsTrackWritesInRealtime checks that the stats counts start at // TestLiveCountsTrackWritesInRealtime checks that the stats counts start at
// zero, reflect each write the moment it commits (no recompute, no timer), do // zero and reflect each write the moment it commits (no recompute, no timer).
// not move when a route is merely re-announced, and drop when a route is // These counts are what /api/v1/stats reports; before this change the
// deleted. These counts are what /api/v1/stats reports; before this change the
// endpoint recomputed them with a COUNT(*) over each table on every request. // endpoint recomputed them with a COUNT(*) over each table on every request.
func TestLiveCountsTrackWritesInRealtime(t *testing.T) { func TestLiveCountsTrackWritesInRealtime(t *testing.T) {
t.Parallel() t.Parallel()
@@ -79,7 +40,7 @@ func TestLiveCountsTrackWritesInRealtime(t *testing.T) {
} }
if empty.ASNs != 0 || empty.Prefixes != 0 || empty.Peerings != 0 || if empty.ASNs != 0 || empty.Prefixes != 0 || empty.Peerings != 0 ||
empty.Peers != 0 || empty.LiveRoutes != 0 { empty.Peers != 0 {
t.Fatalf("empty database counts nonzero: %+v", empty) t.Fatalf("empty database counts nonzero: %+v", empty)
} }
@@ -87,17 +48,6 @@ func TestLiveCountsTrackWritesInRealtime(t *testing.T) {
writeASNsPrefixesAndPeers(t, db, ts) writeASNsPrefixesAndPeers(t, db, ts)
routes := []*LiveRoute{
mkV4Route(t, "198.51.100.0/24", 64500, ts),
mkV4Route(t, "203.0.113.0/24", 64501, ts.Add(time.Minute)),
mkV6Route(testV6Prefix, 64502, ts.Add(2*time.Minute)),
}
err = db.UpsertLiveRouteBatch(routes)
if err != nil {
t.Fatalf("UpsertLiveRouteBatch: %v", err)
}
stats, err := db.GetStatsContext(ctx) stats, err := db.GetStatsContext(ctx)
if err != nil { if err != nil {
t.Fatalf("GetStatsContext: %v", err) t.Fatalf("GetStatsContext: %v", err)
@@ -105,46 +55,11 @@ func TestLiveCountsTrackWritesInRealtime(t *testing.T) {
assertCounts(t, "after inserts", stats, wantCounts{ assertCounts(t, "after inserts", stats, wantCounts{
asns: 2, prefixes: 2, peerings: 1, peers: 1, asns: 2, prefixes: 2, peerings: 1, peers: 1,
ipv4Routes: 2, ipv6Routes: 1, liveRoutes: 3,
})
// Re-announcing the same routes is an update, not an insert: counts hold.
err = db.UpsertLiveRouteBatch(routes)
if err != nil {
t.Fatalf("UpsertLiveRouteBatch (re-announce): %v", err)
}
stats, err = db.GetStatsContext(ctx)
if err != nil {
t.Fatalf("GetStatsContext: %v", err)
}
assertCounts(t, "after re-announce", stats, wantCounts{
asns: 2, prefixes: 2, peerings: 1, peers: 1,
ipv4Routes: 2, ipv6Routes: 1, liveRoutes: 3,
})
// A withdrawal removes one route.
err = db.DeleteLiveRouteBatch([]LiveRouteDeletion{
{Prefix: "203.0.113.0/24", PeerIP: testV4PeerIP, IPVersion: ipVersionV4},
})
if err != nil {
t.Fatalf("DeleteLiveRouteBatch: %v", err)
}
stats, err = db.GetStatsContext(ctx)
if err != nil {
t.Fatalf("GetStatsContext: %v", err)
}
assertCounts(t, "after delete", stats, wantCounts{
asns: 2, prefixes: 2, peerings: 1, peers: 1,
ipv4Routes: 1, ipv6Routes: 1, liveRoutes: 2,
}) })
} }
// writeASNsPrefixesAndPeers writes two ASNs, two prefixes, one peer and one // writeASNsPrefixesAndPeers writes two ASNs, two prefixes, one peer and one
// peering: the rows besides live routes that the stats count. // peering: the rows the stats count.
func writeASNsPrefixesAndPeers(t *testing.T, db *Database, ts time.Time) { func writeASNsPrefixesAndPeers(t *testing.T, db *Database, ts time.Time) {
t.Helper() t.Helper()
@@ -180,8 +95,8 @@ func writeASNsPrefixesAndPeers(t *testing.T, db *Database, ts time.Time) {
} }
// TestLiveCountsSeededFromDatabaseAtStartup writes rows, reopens the same // TestLiveCountsSeededFromDatabaseAtStartup writes rows, reopens the same
// database file, and checks the counts and the prefix distribution come back // database file, and checks the counts come back from the seed scan rather
// from the seed scan rather than starting at zero. // than starting at zero.
func TestLiveCountsSeededFromDatabaseAtStartup(t *testing.T) { func TestLiveCountsSeededFromDatabaseAtStartup(t *testing.T) {
t.Parallel() t.Parallel()
@@ -192,23 +107,7 @@ func TestLiveCountsSeededFromDatabaseAtStartup(t *testing.T) {
t.Fatalf("failed to create database: %v", err) t.Fatalf("failed to create database: %v", err)
} }
ts := time.Date(2026, 1, 2, 3, 4, 5, 0, time.UTC) writeASNsPrefixesAndPeers(t, db, time.Date(2026, 1, 2, 3, 4, 5, 0, time.UTC))
seen := SeenTimes{First: ts, Last: ts}
err = db.GetOrCreateASNBatch(
map[int]SeenTimes{64500: seen, 64501: seen, 64502: seen})
if err != nil {
t.Fatalf("GetOrCreateASNBatch: %v", err)
}
err = db.UpsertLiveRouteBatch([]*LiveRoute{
mkV4Route(t, "198.51.100.0/24", 64500, ts),
mkV6Route(testV6Prefix, 64502, ts),
})
if err != nil {
t.Fatalf("UpsertLiveRouteBatch: %v", err)
}
err = db.Close() err = db.Close()
if err != nil { if err != nil {
@@ -226,435 +125,9 @@ func TestLiveCountsSeededFromDatabaseAtStartup(t *testing.T) {
t.Fatalf("GetStatsContext after reopen: %v", err) t.Fatalf("GetStatsContext after reopen: %v", err)
} }
if stats.ASNs != 3 { assertCounts(t, "after reopen", stats, wantCounts{
t.Errorf("seeded ASNs = %d, want 3", stats.ASNs) asns: 2, prefixes: 2, peerings: 1, peers: 1,
}
if stats.IPv4Routes != 1 || stats.IPv6Routes != 1 || stats.LiveRoutes != 2 {
t.Errorf("seeded routes = (v4 %d, v6 %d, total %d), want (1, 1, 2)",
stats.IPv4Routes, stats.IPv6Routes, stats.LiveRoutes)
}
assertDistribution(t, "seeded IPv4 distribution", stats.IPv4PrefixDistribution,
[]PrefixDistribution{{MaskLength: 24, Count: 1}})
assertDistribution(t, "seeded IPv6 distribution", stats.IPv6PrefixDistribution,
[]PrefixDistribution{{MaskLength: 32, Count: 1}})
}
// TestPrefixDistributionTracksWrites checks that the prefix distribution the
// stats read reports stays exact across each kind of live-route write, and that
// after every step it equals what the distribution query reads from the route
// tables. The steps run once through the batch methods the prefix handler uses
// and once through the single-route methods.
func TestPrefixDistributionTracksWrites(t *testing.T) {
t.Parallel()
steps := prefixDistributionSteps(t)
for _, batch := range []bool{true, false} {
name := "single-route writes"
if batch {
name = "batch writes"
}
t.Run(name, func(t *testing.T) {
t.Parallel()
db, err := New(&config.Config{StateDir: t.TempDir()}, logger.New())
if err != nil {
t.Fatalf("failed to create database: %v", err)
}
defer func() { _ = db.Close() }()
ctx := context.Background()
for _, step := range steps {
err := announceRoutes(db, batch, step.announce)
if err != nil {
t.Fatalf("%s: announce: %v", step.name, err)
}
err = withdrawRoutes(db, batch, step.withdraw)
if err != nil {
t.Fatalf("%s: withdraw: %v", step.name, err)
}
stats, err := db.GetStatsContext(ctx)
if err != nil {
t.Fatalf("%s: GetStatsContext: %v", step.name, err)
}
queryV4, queryV6, err := db.GetPrefixDistributionContext(ctx)
if err != nil {
t.Fatalf("%s: GetPrefixDistributionContext: %v", step.name, err)
}
assertDistribution(t, step.name+": IPv4 distribution",
stats.IPv4PrefixDistribution, step.wantV4)
assertDistribution(t, step.name+": IPv6 distribution",
stats.IPv6PrefixDistribution, step.wantV6)
assertDistribution(t, step.name+": IPv4 distribution query",
queryV4, step.wantV4)
assertDistribution(t, step.name+": IPv6 distribution query",
queryV6, step.wantV6)
}
})
}
}
// prefixDistributionStep is one step of TestPrefixDistributionTracksWrites:
// the routes it announces, then the routes it withdraws, then the
// distributions both the stats and the distribution query must report.
type prefixDistributionStep struct {
name string
announce []*LiveRoute
withdraw []*LiveRoute
wantV4 []PrefixDistribution
wantV6 []PrefixDistribution
}
// prefixDistributionSteps returns the steps of
// TestPrefixDistributionTracksWrites, in the order they run.
func prefixDistributionSteps(t *testing.T) []prefixDistributionStep {
t.Helper()
ts := time.Date(2026, 1, 2, 3, 4, 5, 0, time.UTC)
shared := mkV4Route(t, "198.51.100.0/24", 64500, ts)
sharedSecondPeer := mkV4Route(t, "198.51.100.0/24", 64500, ts)
sharedSecondPeer.PeerIP = "192.0.2.2"
other := mkV4Route(t, "203.0.113.0/24", 64501, ts)
wide := mkV4Route(t, "172.16.0.0/16", 64502, ts)
v6 := mkV6Route(testV6Prefix, 64503, ts)
v6SecondPeer := mkV6Route(testV6Prefix, 64503, ts)
v6SecondPeer.PeerIP = "2001:db8::2"
all := []PrefixDistribution{{MaskLength: 16, Count: 1}, {MaskLength: 24, Count: 2}}
v6Only := []PrefixDistribution{{MaskLength: 32, Count: 1}}
return []prefixDistributionStep{
{
name: "new routes",
announce: []*LiveRoute{shared, other, wide, v6},
wantV4: all, wantV6: v6Only,
},
{
name: "re-announcement",
announce: []*LiveRoute{shared, other, wide, v6},
wantV4: all, wantV6: v6Only,
},
{
name: "second peer announces a prefix that has a live route",
announce: []*LiveRoute{sharedSecondPeer, v6SecondPeer},
wantV4: all, wantV6: v6Only,
},
{
name: "withdrawal of a route that is not the last for its prefix",
withdraw: []*LiveRoute{shared, v6},
wantV4: all, wantV6: v6Only,
},
{
name: "withdrawal of the last route for a prefix",
withdraw: []*LiveRoute{sharedSecondPeer, v6SecondPeer},
wantV4: []PrefixDistribution{
{MaskLength: 16, Count: 1}, {MaskLength: 24, Count: 1},
},
},
{
name: "withdrawal of every remaining route",
withdraw: []*LiveRoute{other, wide},
},
{
name: "two peers announce a new prefix together",
announce: []*LiveRoute{shared, sharedSecondPeer, v6, v6SecondPeer},
wantV4: []PrefixDistribution{{MaskLength: 24, Count: 1}},
wantV6: v6Only,
},
{
name: "both routes for a prefix withdrawn together",
withdraw: []*LiveRoute{shared, sharedSecondPeer, v6, v6SecondPeer},
},
// The feed often withdraws a route that is not live. That must not take
// the prefix out of the distribution. A count wrongly taken below zero is
// left out of the answer, so the next step shows it: its announcement
// at the same mask length would then not be counted.
{
name: "withdrawal of routes that are not live",
withdraw: []*LiveRoute{other, v6},
},
{
name: "announcement after a withdrawal of routes that are not live",
announce: []*LiveRoute{other, v6},
wantV4: []PrefixDistribution{{MaskLength: 24, Count: 1}},
wantV6: v6Only,
},
}
}
// announceRoutes writes routes in one UpsertLiveRouteBatch, or with one
// UpsertLiveRoute each.
func announceRoutes(db *Database, batch bool, routes []*LiveRoute) error {
if batch {
return db.UpsertLiveRouteBatch(routes)
}
for _, route := range routes {
err := db.UpsertLiveRoute(route)
if err != nil {
return err
}
}
return nil
}
// withdrawRoutes removes routes in one DeleteLiveRouteBatch, or with one
// DeleteLiveRoute each.
func withdrawRoutes(db *Database, batch bool, routes []*LiveRoute) error {
if !batch {
for _, route := range routes {
err := db.DeleteLiveRoute(route.Prefix, route.PeerIP)
if err != nil {
return err
}
}
return nil
}
deletions := make([]LiveRouteDeletion, 0, len(routes))
for _, route := range routes {
deletions = append(deletions, LiveRouteDeletion{
Prefix: route.Prefix, PeerIP: route.PeerIP, IPVersion: route.IPVersion,
})
}
return db.DeleteLiveRouteBatch(deletions)
}
// TestDeleteLiveRoutesOfPeer gives two peers IPv4 and IPv6 live routes, some
// for the same prefixes, and deletes all of one peer's routes. The other peer's
// routes stay, and the route counts and prefix distribution held in memory
// equal what is read from the route tables.
func TestDeleteLiveRoutesOfPeer(t *testing.T) {
t.Parallel()
const (
peerIP = "192.0.2.1"
otherPeerIP = "192.0.2.2"
sharedV4 = "198.51.100.0/24"
sharedV6 = "2001:db8::/32"
)
db, err := New(&config.Config{StateDir: t.TempDir()}, logger.New())
if err != nil {
t.Fatalf("failed to create database: %v", err)
}
defer func() { _ = db.Close() }()
ts := time.Date(2026, 1, 2, 3, 4, 5, 0, time.UTC)
ofPeer := func(peer string, route *LiveRoute) *LiveRoute {
route.PeerIP = peer
return route
}
onlyV6 := mkV6Route("2001:db8:1::/48", 64503, ts)
onlyV6.MaskLength = 48
err = db.UpsertLiveRouteBatch([]*LiveRoute{
ofPeer(peerIP, mkV4Route(t, sharedV4, 64500, ts)),
ofPeer(otherPeerIP, mkV4Route(t, sharedV4, 64500, ts)),
ofPeer(peerIP, mkV4Route(t, "203.0.113.0/24", 64501, ts)),
ofPeer(peerIP, mkV6Route(sharedV6, 64502, ts)),
ofPeer(otherPeerIP, mkV6Route(sharedV6, 64502, ts)),
ofPeer(peerIP, onlyV6),
}) })
if err != nil {
t.Fatalf("UpsertLiveRouteBatch: %v", err)
}
// peerIP has two IPv4 and two IPv6 routes; a second delete finds none.
for _, want := range []int{4, 0} {
deleted, err := db.DeleteLiveRoutesOfPeer(peerIP)
if err != nil {
t.Fatalf("DeleteLiveRoutesOfPeer: %v", err)
}
if deleted != want {
t.Errorf("DeleteLiveRoutesOfPeer deleted %d routes, want %d",
deleted, want)
}
}
for _, prefix := range []string{sharedV4, sharedV6} {
routes, _, err := db.GetPrefixDetailsContext(context.Background(), prefix)
if err != nil {
t.Fatalf("GetPrefixDetailsContext(%s): %v", prefix, err)
}
if len(routes) != 1 || routes[0].PeerIP != otherPeerIP {
t.Errorf("live routes for %s = %+v, want only the one from %s",
prefix, routes, otherPeerIP)
}
}
checkOneLiveRoutePerIPVersion(t, db)
}
// TestDeleteLiveRoutesOfPeerDeletesInChunks gives a peer one IPv4 route more
// than a chunk holds, two of them for prefixes another peer also has. One chunk
// deletes peerRouteChunkSize of the peer's routes and commits, leaving the last
// one. The routes are written again, and DeleteLiveRoutesOfPeer deletes all of
// them, a chunk and then the last one. After each step the route count and
// prefix distribution held in memory and those read from the route table are
// the ones wanted.
func TestDeleteLiveRoutesOfPeerDeletesInChunks(t *testing.T) {
t.Parallel()
const (
peerIP = "192.0.2.1"
otherPeerIP = "192.0.2.2"
// How many of the peer's prefixes the other peer also has, the first
// ones in prefix order.
sharedPrefixes = 2
)
db, err := New(&config.Config{StateDir: t.TempDir()}, logger.New())
if err != nil {
t.Fatalf("failed to create database: %v", err)
}
defer func() { _ = db.Close() }()
ts := time.Date(2026, 1, 2, 3, 4, 5, 0, time.UTC)
peerRoutes := make([]*LiveRoute, 0, peerRouteChunkSize+1)
otherPeerRoutes := make([]*LiveRoute, 0, sharedPrefixes)
for i := range peerRouteChunkSize + 1 {
prefix := fmt.Sprintf("10.%d.%d.0/24", i/256, i%256)
route := mkV4Route(t, prefix, 64500, ts)
route.PeerIP = peerIP
peerRoutes = append(peerRoutes, route)
if i < sharedPrefixes {
route := mkV4Route(t, prefix, 64500, ts)
route.PeerIP = otherPeerIP
otherPeerRoutes = append(otherPeerRoutes, route)
}
}
err = db.UpsertLiveRouteBatch(slices.Concat(peerRoutes, otherPeerRoutes))
if err != nil {
t.Fatalf("UpsertLiveRouteBatch: %v", err)
}
deletedV4, deletedV6, err := db.deletePeerRouteChunk(peerIP)
if err != nil {
t.Fatalf("deletePeerRouteChunk: %v", err)
}
if deletedV4 != peerRouteChunkSize || deletedV6 != 0 {
t.Errorf("deletePeerRouteChunk deleted (%d, %d) routes, want (%d, 0)",
deletedV4, deletedV6, peerRouteChunkSize)
}
// The shared prefixes went with the chunk, so each route left has a
// prefix of its own.
checkV4RoutesOfOwnPrefixes(t, db, "after one chunk", 1+sharedPrefixes)
err = db.UpsertLiveRouteBatch(peerRoutes)
if err != nil {
t.Fatalf("UpsertLiveRouteBatch: %v", err)
}
deleted, err := db.DeleteLiveRoutesOfPeer(peerIP)
if err != nil {
t.Fatalf("DeleteLiveRoutesOfPeer: %v", err)
}
if deleted != len(peerRoutes) {
t.Errorf("DeleteLiveRoutesOfPeer deleted %d routes, want %d",
deleted, len(peerRoutes))
}
checkV4RoutesOfOwnPrefixes(t, db, "after DeleteLiveRoutesOfPeer",
sharedPrefixes)
}
// checkV4RoutesOfOwnPrefixes checks that the database has wantRoutes IPv4 live
// routes, each for a /24 no other live route has, both as counted in memory and
// as read from the route table. when says which step of the test checks.
func checkV4RoutesOfOwnPrefixes(
t *testing.T, db *Database, when string, wantRoutes int,
) {
t.Helper()
ctx := context.Background()
stats, err := db.GetStatsContext(ctx)
if err != nil {
t.Fatalf("GetStatsContext: %v", err)
}
tableRoutes, _, err := db.GetLiveRouteCountsContext(ctx)
if err != nil {
t.Fatalf("GetLiveRouteCountsContext: %v", err)
}
if stats.IPv4Routes != wantRoutes || tableRoutes != wantRoutes {
t.Errorf("%s: IPv4 live routes: %d in memory, %d in the table, want %d",
when, stats.IPv4Routes, tableRoutes, wantRoutes)
}
queryDistribution, _, err := db.GetPrefixDistributionContext(ctx)
if err != nil {
t.Fatalf("GetPrefixDistributionContext: %v", err)
}
want := []PrefixDistribution{{MaskLength: 24, Count: wantRoutes}}
assertDistribution(t, when+": IPv4 distribution",
stats.IPv4PrefixDistribution, want)
assertDistribution(t, when+": IPv4 distribution query",
queryDistribution, want)
}
// checkOneLiveRoutePerIPVersion checks that the database has one IPv4 live
// route, for a /24, and one IPv6 live route, for a /32, both as counted in
// memory and as read from the route tables.
func checkOneLiveRoutePerIPVersion(t *testing.T, db *Database) {
t.Helper()
ctx := context.Background()
stats, err := db.GetStatsContext(ctx)
if err != nil {
t.Fatalf("GetStatsContext: %v", err)
}
tableV4, tableV6, err := db.GetLiveRouteCountsContext(ctx)
if err != nil {
t.Fatalf("GetLiveRouteCountsContext: %v", err)
}
if stats.IPv4Routes != 1 || tableV4 != 1 || stats.IPv6Routes != 1 ||
tableV6 != 1 {
t.Errorf("live routes (IPv4, IPv6): (%d, %d) in memory, (%d, %d) in "+
"the tables, want (1, 1)",
stats.IPv4Routes, stats.IPv6Routes, tableV4, tableV6)
}
queryV4, queryV6, err := db.GetPrefixDistributionContext(ctx)
if err != nil {
t.Fatalf("GetPrefixDistributionContext: %v", err)
}
wantV4 := []PrefixDistribution{{MaskLength: 24, Count: 1}}
wantV6 := []PrefixDistribution{{MaskLength: 32, Count: 1}}
assertDistribution(t, "IPv4 distribution", stats.IPv4PrefixDistribution, wantV4)
assertDistribution(t, "IPv6 distribution", stats.IPv6PrefixDistribution, wantV6)
assertDistribution(t, "IPv4 distribution query", queryV4, wantV4)
assertDistribution(t, "IPv6 distribution query", queryV6, wantV6)
} }
// TestLiveCountsConcurrentReadWrite runs writers and stats readers at once so // TestLiveCountsConcurrentReadWrite runs writers and stats readers at once so
@@ -672,7 +145,10 @@ func TestLiveCountsConcurrentReadWrite(t *testing.T) {
ts := time.Date(2026, 1, 2, 3, 4, 5, 0, time.UTC) ts := time.Date(2026, 1, 2, 3, 4, 5, 0, time.UTC)
const writers = 4 const (
writers = 4
prefixesPerWrite = 25
)
var wg sync.WaitGroup var wg sync.WaitGroup
wg.Add(writers) wg.Add(writers)
@@ -681,14 +157,12 @@ func TestLiveCountsConcurrentReadWrite(t *testing.T) {
go func(base int) { go func(base int) {
defer wg.Done() defer wg.Done()
for i := range 25 { for i := range prefixesPerWrite {
asn := 65000 + base*100 + i prefix := fmt.Sprintf("10.%d.%d.0/24", base, i)
route := mkV6Route(testV6Prefix, asn, ts)
route.PeerIP = "2001:db8::" + uuid.NewString()
err := db.UpsertLiveRoute(route) err := db.UpdatePrefixesBatch(map[string]time.Time{prefix: ts})
if err != nil { if err != nil {
t.Errorf("UpsertLiveRoute: %v", err) t.Errorf("UpdatePrefixesBatch: %v", err)
return return
} }
@@ -728,22 +202,16 @@ func TestLiveCountsConcurrentReadWrite(t *testing.T) {
t.Fatalf("GetStatsContext: %v", err) t.Fatalf("GetStatsContext: %v", err)
} }
if want := writers * 25; stats.IPv6Routes != want { if want := writers * prefixesPerWrite; stats.IPv4Prefixes != want {
t.Errorf("IPv6Routes = %d, want %d", stats.IPv6Routes, want) t.Errorf("IPv4Prefixes = %d, want %d", stats.IPv4Prefixes, want)
} }
// Every writer announced the same prefix, so it counts once.
assertDistribution(t, "IPv6 distribution", stats.IPv6PrefixDistribution,
[]PrefixDistribution{{MaskLength: 32, Count: 1}})
} }
type wantCounts struct { type wantCounts struct {
asns int asns int
prefixes int prefixes int
peerings int peerings int
peers int peers int
ipv4Routes int
ipv6Routes int
liveRoutes int
} }
func assertCounts(t *testing.T, when string, got Stats, want wantCounts) { func assertCounts(t *testing.T, when string, got Stats, want wantCounts) {
@@ -764,24 +232,4 @@ func assertCounts(t *testing.T, when string, got Stats, want wantCounts) {
if got.Peers != want.peers { if got.Peers != want.peers {
t.Errorf("%s: Peers = %d, want %d", when, got.Peers, want.peers) t.Errorf("%s: Peers = %d, want %d", when, got.Peers, want.peers)
} }
if got.IPv4Routes != want.ipv4Routes {
t.Errorf("%s: IPv4Routes = %d, want %d", when, got.IPv4Routes, want.ipv4Routes)
}
if got.IPv6Routes != want.ipv6Routes {
t.Errorf("%s: IPv6Routes = %d, want %d", when, got.IPv6Routes, want.ipv6Routes)
}
if got.LiveRoutes != want.liveRoutes {
t.Errorf("%s: LiveRoutes = %d, want %d", when, got.LiveRoutes, want.liveRoutes)
}
}
func assertDistribution(t *testing.T, what string, got, want []PrefixDistribution) {
t.Helper()
if !slices.Equal(got, want) {
t.Errorf("%s = %v, want %v", what, got, want)
}
} }
internal/database/database.go
+66 -1454
View File
File diff suppressed because it is too large Load Diff
internal/database/database_internal_test.go
+111 -1270
View File
File diff suppressed because it is too large Load Diff
internal/database/interface.go
+15 -47
View File
@@ -1,7 +1,6 @@
// Package database provides SQLite storage for BGP routing data including ASNs, // Package database provides SQLite storage for BGP routing data: ASNs,
// prefixes, announcements, peerings, and live route tables. Its routes table // prefixes, announcements, peerings and BGP peers. Its routes table holds a
// holds a copy of the whole routing table that internal/routetable keeps in // copy of the whole routing table that internal/routetable keeps in memory.
// memory.
package database package database
import ( import (
@@ -12,26 +11,21 @@ import (
"git.eeqj.de/sneak/routewatch/internal/routetable" "git.eeqj.de/sneak/routewatch/internal/routetable"
) )
// Stats contains database statistics including counts for ASNs, prefixes, // Stats contains database statistics: counts of ASNs, prefixes, peerings and
// peerings, peers, and live routes, as well as file size and prefix distribution data. // peers, and the file size.
type Stats struct { type Stats struct {
ASNs int ASNs int
Prefixes int Prefixes int
IPv4Prefixes int IPv4Prefixes int
IPv6Prefixes int IPv6Prefixes int
Peerings int Peerings int
Peers int Peers int
FileSizeBytes int64 FileSizeBytes int64
LiveRoutes int
IPv4Routes int
IPv6Routes int
IPv4PrefixDistribution []PrefixDistribution
IPv6PrefixDistribution []PrefixDistribution
} }
// Store defines the interface for database operations. It provides methods for // Store defines the interface for database operations. It provides methods for
// managing ASNs, prefixes, announcements, peerings, BGP peers, and live routes. // managing ASNs, prefixes, announcements, peerings, BGP peers, and the copy of
// Implementations must be safe for concurrent use. // the routing table. Implementations must be safe for concurrent use.
// //
//nolint:interfacebloat // the one seam handlers and tests depend on //nolint:interfacebloat // the one seam handlers and tests depend on
type Store interface { type Store interface {
@@ -57,27 +51,10 @@ type Store interface {
UpdatePeer(peerIP string, peerASN int, messageType string, timestamp time.Time) error UpdatePeer(peerIP string, peerASN int, messageType string, timestamp time.Time) error
UpdatePeerBatch(peers map[string]PeerUpdate) error UpdatePeerBatch(peers map[string]PeerUpdate) error
// Live route operations
UpsertLiveRoute(route *LiveRoute) error
UpsertLiveRouteBatch(routes []*LiveRoute) error
DeleteLiveRoute(prefix, peerIP string) error
DeleteLiveRouteBatch(deletions []LiveRouteDeletion) error
DeleteLiveRoutesOfPeer(peerIP string) (deleted int, err error)
GetPrefixDistribution() (
ipv4 []PrefixDistribution, ipv6 []PrefixDistribution, err error)
GetPrefixDistributionContext(ctx context.Context) (
ipv4 []PrefixDistribution, ipv6 []PrefixDistribution, err error)
GetLiveRouteCounts() (ipv4Count, ipv6Count int, err error)
GetLiveRouteCountsContext(ctx context.Context) (ipv4Count, ipv6Count int, err error)
// Routing table operations: the database's copy of the table in memory // Routing table operations: the database's copy of the table in memory
WriteRouteChanges(changes []routetable.Change, newPeers map[string]uint) error WriteRouteChanges(changes []routetable.Change, newPeers map[string]uint) error
ReadRouteTable(ctx context.Context) (*routetable.Table, error) ReadRouteTable(ctx context.Context) (*routetable.Table, error)
// IP lookup operations
GetIPInfo(ip string) (*IPInfo, error)
GetIPInfoContext(ctx context.Context, ip string) (*IPInfo, error)
// ASN WHOIS operations // ASN WHOIS operations
GetNextStaleASN(ctx context.Context, staleThreshold time.Duration) (int, error) GetNextStaleASN(ctx context.Context, staleThreshold time.Duration) (int, error)
UpdateASNWHOIS(ctx context.Context, update *ASNWHOISUpdate) error UpdateASNWHOIS(ctx context.Context, update *ASNWHOISUpdate) error
@@ -85,20 +62,11 @@ type Store interface {
// AS and prefix detail operations // AS and prefix detail operations
GetAS(ctx context.Context, asn int) (*ASN, bool, error) GetAS(ctx context.Context, asn int) (*ASN, bool, error)
GetLastSeenASes(ctx context.Context) (*ASN, *ASN, error)
GetASWHOIS(ctx context.Context, asn int) (*ASN, error) GetASWHOIS(ctx context.Context, asn int) (*ASN, error)
GetPrefixFirstSeen(ctx context.Context, prefix netip.Prefix) (time.Time, error) GetPrefixFirstSeen(ctx context.Context, prefix netip.Prefix) (time.Time, error)
GetASDetails(asn int) (*ASN, []LiveRoute, error)
GetASDetailsContext(ctx context.Context, asn int) (*ASN, []LiveRoute, error)
GetASPeers(asn int) ([]ASPeer, error) GetASPeers(asn int) ([]ASPeer, error)
GetASPeersContext(ctx context.Context, asn int) ([]ASPeer, error) GetASPeersContext(ctx context.Context, asn int) ([]ASPeer, error)
GetPrefixDetails(prefix string) ([]LiveRoute, map[int]ASN, error)
GetPrefixDetailsContext(
ctx context.Context, prefix string,
) ([]LiveRoute, map[int]ASN, error)
GetRandomPrefixesByLength(maskLength, ipVersion, limit int) ([]LiveRoute, error)
GetRandomPrefixesByLengthContext(
ctx context.Context, maskLength, ipVersion, limit int,
) ([]LiveRoute, error)
// Lifecycle // Lifecycle
Close() error Close() error
internal/database/models.go
-59
View File
@@ -66,65 +66,6 @@ type ASNPeering struct {
LastSeen time.Time LastSeen time.Time
} }
// LiveRoute represents a route in the live routing table. JSON leaves out an
// empty AS path, next hop or time: the routing table in memory keeps them only
// for the two newest routes of each prefix and origin AS.
//
//nolint:tagliatelle // field names are the existing JSON API
type LiveRoute struct {
Prefix string `json:"prefix"`
MaskLength int `json:"mask_length"`
IPVersion int `json:"ip_version"`
OriginASN int `json:"origin_asn"`
PeerIP string `json:"peer_ip"`
ASPath []int `json:"as_path,omitempty"`
NextHop string `json:"next_hop,omitempty"`
LastUpdated time.Time `json:"last_updated,omitzero"`
}
// PrefixDistribution represents the distribution of prefixes by mask length
//
//nolint:tagliatelle // field names are the existing JSON API
type PrefixDistribution struct {
MaskLength int `json:"mask_length"`
Count int `json:"count"`
}
// IPInfo represents comprehensive IP information for the /ip endpoint
//
//nolint:tagliatelle // field names are the existing JSON API
type IPInfo struct {
IP string `json:"ip"`
Netblock string `json:"netblock"`
MaskLength int `json:"mask_length"`
IPVersion int `json:"ip_version"`
NumPeers int `json:"num_peers"`
// AS information
ASN int `json:"asn"`
ASName string `json:"as_name,omitempty"`
Handle string `json:"handle,omitempty"`
Description string `json:"description,omitempty"`
OrgName string `json:"org_name,omitempty"`
OrgID string `json:"org_id,omitempty"`
Address string `json:"address,omitempty"`
CountryCode string `json:"country_code,omitempty"`
AbuseEmail string `json:"abuse_email,omitempty"`
RIR string `json:"rir,omitempty"`
// Timestamps. LastSeen is zero, and left out of JSON, when the prefix
// keeps no route.
FirstSeen time.Time `json:"first_seen"`
LastSeen time.Time `json:"last_seen,omitzero"`
// Indicates if WHOIS data needs refresh (not serialized)
NeedsWHOISRefresh bool `json:"-"`
}
// LiveRouteDeletion represents parameters for deleting a live route
type LiveRouteDeletion struct {
Prefix string
PeerIP string
IPVersion int
}
// SeenTimes is the earliest and the latest time one batch of updates saw an AS, // SeenTimes is the earliest and the latest time one batch of updates saw an AS,
// a peer or a peering. A row the batch adds is stored first seen at First and // a peer or a peering. A row the batch adds is stored first seen at First and
// last seen at Last. A row already stored takes First as its first_seen only // last seen at Last. A row already stored takes First as its first_seen only
internal/database/schema.sql
+3 -59
View File
@@ -83,71 +83,15 @@ CREATE INDEX IF NOT EXISTS idx_peerings_lookup ON peerings(as_a, as_b);
-- Indexes for asns table -- Indexes for asns table
CREATE INDEX IF NOT EXISTS idx_asns_asn ON asns(asn); CREATE INDEX IF NOT EXISTS idx_asns_asn ON asns(asn);
CREATE INDEX IF NOT EXISTS idx_asns_whois_updated_at ON asns(whois_updated_at); CREATE INDEX IF NOT EXISTS idx_asns_whois_updated_at ON asns(whois_updated_at);
-- Finds the AS last seen most recently and the one last seen longest ago, for
-- the home page.
CREATE INDEX IF NOT EXISTS idx_asns_last_seen ON asns(last_seen);
-- Indexes for bgp_peers table -- Indexes for bgp_peers table
CREATE INDEX IF NOT EXISTS idx_bgp_peers_asn ON bgp_peers(peer_asn); CREATE INDEX IF NOT EXISTS idx_bgp_peers_asn ON bgp_peers(peer_asn);
CREATE INDEX IF NOT EXISTS idx_bgp_peers_last_seen ON bgp_peers(last_seen); CREATE INDEX IF NOT EXISTS idx_bgp_peers_last_seen ON bgp_peers(last_seen);
CREATE INDEX IF NOT EXISTS idx_bgp_peers_ip ON bgp_peers(peer_ip); CREATE INDEX IF NOT EXISTS idx_bgp_peers_ip ON bgp_peers(peer_ip);
-- IPv4 routing table maintained by PrefixHandler. A route write changes the
-- table and its entry in every index on a column the write sets, so the table
-- has only the indexes that the queries named with them use.
CREATE TABLE IF NOT EXISTS live_routes_v4 (
-- The row id, which the prefix-length pages' sample draws through the
-- mask_length index. Declared as a column, it stays the same for as long
-- as the route is live, also through a VACUUM.
id INTEGER PRIMARY KEY,
prefix TEXT NOT NULL,
mask_length INTEGER NOT NULL, -- CIDR mask length (0-32)
origin_asn INTEGER NOT NULL,
peer_ip TEXT NOT NULL,
as_path TEXT NOT NULL, -- JSON array
next_hop TEXT NOT NULL,
last_updated DATETIME NOT NULL,
-- A peer has at most one route for a prefix. This key finds the route a
-- write updates or deletes and, peer_ip coming first, all of a peer's
-- routes when its BGP session goes down.
UNIQUE(peer_ip, prefix)
);
-- IPv6 routing table maintained by PrefixHandler, as live_routes_v4.
CREATE TABLE IF NOT EXISTS live_routes_v6 (
id INTEGER PRIMARY KEY,
prefix TEXT NOT NULL,
mask_length INTEGER NOT NULL, -- CIDR mask length (0-128)
origin_asn INTEGER NOT NULL,
peer_ip TEXT NOT NULL,
as_path TEXT NOT NULL, -- JSON array
next_hop TEXT NOT NULL,
last_updated DATETIME NOT NULL,
UNIQUE(peer_ip, prefix)
);
-- Indexes for both live route tables, and the queries that use each.
--
-- prefix, last_updated: a prefix's routes, for the prefix page and its API
-- answer, the IP lookup and its peer count, and the check whether a prefix
-- still has a live route; and a prefix's newest route for the prefix-length
-- pages and the AS page, in one search whatever the number of peers.
CREATE INDEX IF NOT EXISTS idx_live_routes_v4_prefix_last_updated ON live_routes_v4(prefix, last_updated);
CREATE INDEX IF NOT EXISTS idx_live_routes_v6_prefix_last_updated ON live_routes_v6(prefix, last_updated);
-- mask_length: the prefix-length pages' random sample, the row id range of a
-- mask length and the first route of it at or after a row id.
CREATE INDEX IF NOT EXISTS idx_live_routes_v4_mask_length ON live_routes_v4(mask_length);
CREATE INDEX IF NOT EXISTS idx_live_routes_v6_mask_length ON live_routes_v6(mask_length);
-- mask_length, prefix: the prefix-length pages' list of a mask length's
-- prefixes in text order, and the prefix distribution read at startup.
CREATE INDEX IF NOT EXISTS idx_live_routes_v4_mask_prefix ON live_routes_v4(mask_length, prefix);
CREATE INDEX IF NOT EXISTS idx_live_routes_v6_mask_prefix ON live_routes_v6(mask_length, prefix);
-- origin_asn, prefix: the prefixes an AS originates, each in one search
-- whatever the number of peers, for the AS page and its API answer.
CREATE INDEX IF NOT EXISTS idx_live_routes_v4_origin_prefix ON live_routes_v4(origin_asn, prefix);
CREATE INDEX IF NOT EXISTS idx_live_routes_v6_origin_prefix ON live_routes_v6(origin_asn, prefix);
-- last_updated: the oldest and newest route times in /api/v1/stats and
-- /status.json.
CREATE INDEX IF NOT EXISTS idx_live_routes_v4_last_updated ON live_routes_v4(last_updated);
CREATE INDEX IF NOT EXISTS idx_live_routes_v6_last_updated ON live_routes_v6(last_updated);
-- The whole routing table, as internal/routetable keeps it in memory: one row -- The whole routing table, as internal/routetable keeps it in memory: one row
-- for each prefix and origin AS that some peer carries. Startup reads every -- for each prefix and origin AS that some peer carries. Startup reads every
-- row, and the rows of the pairs that change are written back. -- row, and the rows of the pairs that change are written back.
internal/database/utils.go
-14
View File
@@ -1,8 +1,6 @@
package database package database
import ( import (
"fmt"
"net"
"strings" "strings"
"github.com/google/uuid" "github.com/google/uuid"
@@ -20,15 +18,3 @@ func detectIPVersion(prefix string) int {
return ipVersionV4 return ipVersionV4
} }
// prefixMaskLength returns the mask length of a prefix such as 192.0.2.0/24.
func prefixMaskLength(prefix string) (int, error) {
_, network, err := net.ParseCIDR(prefix)
if err != nil {
return 0, fmt.Errorf("invalid prefix %s: %w", prefix, err)
}
maskLength, _ := network.Mask.Size()
return maskLength, nil
}
internal/routetable/overview.go
+259
View File
@@ -0,0 +1,259 @@
package routetable
import (
"cmp"
"net/netip"
"slices"
"time"
"git.eeqj.de/sneak/routewatch/internal/bogon"
"git.eeqj.de/sneak/routewatch/internal/ristypes"
)
// walkBatch is how many prefixes the walk over the whole table reads in one
// hold of the read lock. The updates waiting for the lock get it in between.
const walkBatch = 10000
// Overview holds figures about the whole table. Working them out takes a walk
// over every prefix, so a caller asks for them now and then and keeps them.
type Overview struct {
// NewestRoute and OldestRoute are the newest and the oldest kept routes,
// or nil when the table keeps none. Of several with the same time, it is
// any one of them.
NewestRoute *PrefixRoute
OldestRoute *PrefixRoute
// MostPrefixesASN is the AS that originates the most prefixes, and
// MostPrefixes is how many it originates. Of several with as many, it is
// the lowest AS number. AS 0, no known origin, counts neither here nor
// in the single-prefix figures.
MostPrefixesASN int
MostPrefixes int
// SinglePrefixASes counts the ASes that originate one prefix.
SinglePrefixASes int
// NewestSinglePrefix is the newest kept route of those ASes, or nil when
// they keep none.
NewestSinglePrefix *PrefixRoute
// Bogons are the bogon announcements, in prefix order: at most the
// number Overview is asked for. BogonCount counts them all.
Bogons []Bogon
BogonCount int
// MultipleOrigins are the prefixes that peers carry with more than one
// origin AS, in prefix order: at most the number Overview is asked for.
// MultipleOriginCount counts them all.
MultipleOrigins []MultipleOriginPrefix
MultipleOriginCount int
}
// PrefixRoute is the time of a kept route, with its prefix and origin AS.
type PrefixRoute struct {
Prefix netip.Prefix
OriginASN int
Time time.Time
}
// Bogon is a bogon announcement: a prefix and origin AS pair whose prefix or
// origin AS is reserved, as package bogon tells.
type Bogon struct {
Prefix netip.Prefix
OriginASN int
// Peers is how many peers carry the prefix with this origin AS.
Peers int
// Reasons say why the prefix, the origin AS or both are reserved.
Reasons []string
}
// MultipleOriginPrefix is a prefix that peers carry with more than one origin
// AS.
type MultipleOriginPrefix struct {
Prefix netip.Prefix
// Origins are its origin ASes, in AS number order.
Origins []OriginPeers
}
// OriginPeers is one origin AS of a prefix, and how many peers carry the prefix
// with it.
type OriginPeers struct {
ASN int
Peers int
}
// Overview works out the table's overview. It lists at most maxListed bogon
// announcements and at most maxListed prefixes with more than one origin AS:
// the first in prefix order. It walks over every prefix, but holds the read
// lock for walkBatch prefixes at a time.
func (t *Table) Overview(maxListed int) Overview {
var overview Overview
t.walkPrefixes(func(prefix netip.Prefix, origins []*origin) {
overview.noteKeptRoutes(prefix, origins)
overview.noteBogons(prefix, origins)
overview.noteMultipleOrigins(prefix, origins)
})
t.countPrefixesPerOrigin(&overview)
slices.SortFunc(overview.Bogons, func(a, b Bogon) int {
return cmp.Or(a.Prefix.Compare(b.Prefix),
cmp.Compare(a.OriginASN, b.OriginASN))
})
overview.BogonCount = len(overview.Bogons)
overview.Bogons = overview.Bogons[:min(maxListed, overview.BogonCount)]
slices.SortFunc(overview.MultipleOrigins, func(a, b MultipleOriginPrefix) int {
return a.Prefix.Compare(b.Prefix)
})
overview.MultipleOriginCount = len(overview.MultipleOrigins)
overview.MultipleOrigins =
overview.MultipleOrigins[:min(maxListed, overview.MultipleOriginCount)]
return overview
}
// walkPrefixes calls visit with every prefix of the table and its origins. It
// holds the read lock, but lets it go after every walkBatch prefixes, so the
// updates waiting for it get in. A prefix that is added or removed in such a
// gap may or may not be visited, as Go allows for a map that changes during a
// range over it. visit must not keep origins once it returns.
func (t *Table) walkPrefixes(visit func(prefix netip.Prefix, origins []*origin)) {
t.mu.RLock()
defer t.mu.RUnlock()
visited := 0
for _, f := range []*family{&t.ipv4, &t.ipv6} {
for _, prefixes := range f.byLength {
for prefix, origins := range prefixes {
visit(prefix, origins)
visited++
if visited%walkBatch == 0 {
t.mu.RUnlock()
t.mu.RLock()
}
}
}
}
}
// countPrefixesPerOrigin works out the overview's figures of how many prefixes
// each AS originates. It holds the read lock once, for one look at each origin
// AS, of which there are far fewer than prefixes.
func (t *Table) countPrefixesPerOrigin(overview *Overview) {
t.mu.RLock()
defer t.mu.RUnlock()
for asn, prefixes := range t.byOrigin {
if asn == ristypes.NoOrigin {
continue
}
count := len(prefixes)
if count > overview.MostPrefixes ||
(count == overview.MostPrefixes && asn < overview.MostPrefixesASN) {
overview.MostPrefixesASN = asn
overview.MostPrefixes = count
}
if count == 1 {
overview.SinglePrefixASes++
overview.noteSinglePrefix(asn, prefixes)
}
}
}
// noteSinglePrefix makes the newest kept route of asn, whose one prefix is in
// prefixes, the overview's newest single-prefix route when it is newer.
func (overview *Overview) noteSinglePrefix(
asn int, prefixes map[netip.Prefix]*origin,
) {
for prefix, o := range prefixes {
newest := o.newestTime()
if newest == nil {
continue
}
if overview.NewestSinglePrefix == nil ||
newest.After(overview.NewestSinglePrefix.Time) {
overview.NewestSinglePrefix = &PrefixRoute{
Prefix: prefix, OriginASN: asn, Time: *newest,
}
}
}
}
// noteKeptRoutes makes the newest and the oldest kept routes of prefix's
// origins the overview's newest and oldest routes when they are newer or older.
func (overview *Overview) noteKeptRoutes(prefix netip.Prefix, origins []*origin) {
for _, o := range origins {
if len(o.routes) == 0 {
continue
}
newest := o.routes[0].time
if overview.NewestRoute == nil || newest.After(overview.NewestRoute.Time) {
overview.NewestRoute = &PrefixRoute{
Prefix: prefix, OriginASN: o.asn, Time: newest,
}
}
oldest := o.routes[len(o.routes)-1].time
if overview.OldestRoute == nil || oldest.Before(overview.OldestRoute.Time) {
overview.OldestRoute = &PrefixRoute{
Prefix: prefix, OriginASN: o.asn, Time: oldest,
}
}
}
}
// noteBogons adds each origin of prefix to the overview's bogon announcements
// when the prefix or the origin AS is reserved.
func (overview *Overview) noteBogons(prefix netip.Prefix, origins []*origin) {
prefixReason, prefixReserved := bogon.Prefix(prefix)
for _, o := range origins {
asnReason, asnReserved := bogon.ASN(o.asn)
if !prefixReserved && !asnReserved {
continue
}
announcement := Bogon{
Prefix: prefix, OriginASN: o.asn, Peers: int(o.peers.Count()),
}
if prefixReserved {
announcement.Reasons = append(announcement.Reasons, prefixReason)
}
if asnReserved {
announcement.Reasons = append(announcement.Reasons, asnReason)
}
overview.Bogons = append(overview.Bogons, announcement)
}
}
// noteMultipleOrigins adds prefix to the overview's prefixes with more than one
// origin AS when it has more than one.
func (overview *Overview) noteMultipleOrigins(
prefix netip.Prefix, origins []*origin,
) {
if len(origins) <= 1 {
return
}
entry := MultipleOriginPrefix{
Prefix: prefix,
Origins: make([]OriginPeers, 0, len(origins)),
}
for _, o := range origins {
entry.Origins = append(entry.Origins,
OriginPeers{ASN: o.asn, Peers: int(o.peers.Count())})
}
overview.MultipleOrigins = append(overview.MultipleOrigins, entry)
}
internal/routetable/overview_internal_test.go
+133
View File
@@ -0,0 +1,133 @@
package routetable
import (
"net/netip"
"runtime"
"testing"
"time"
)
// TestWalkPrefixesVisitsEveryPrefix walks a table with more prefixes than two
// of the walk's batches, while another goroutine keeps adding and removing
// other prefixes of the same mask length. The walk lets the lock go between
// batches, so those updates get in while it runs. Every prefix that was in
// the table all along must be visited once.
func TestWalkPrefixesVisitsEveryPrefix(t *testing.T) {
t.Parallel()
const (
asn = 1000
peer = "192.0.2.1"
// prefixCount is more prefixes than two of the walk's batches.
prefixCount = 2*walkBatch + 1
)
routeTime := time.Date(2026, 1, 2, 3, 4, 5, 0, time.UTC)
table := New(Params{})
prefixes := make([]netip.Prefix, 0, prefixCount)
addr := netip.MustParseAddr("1.0.0.0")
for range prefixCount {
prefix := netip.PrefixFrom(addr, addr.BitLen())
table.Announce(prefix, asn, Route{Peer: peer, Time: routeTime})
prefixes = append(prefixes, prefix)
addr = addr.Next()
}
stop := make(chan struct{})
updated := make(chan struct{})
go func() {
defer close(updated)
other := netip.MustParseAddr("2.0.0.0")
for {
select {
case <-stop:
return
default:
}
prefix := netip.PrefixFrom(other, other.BitLen())
table.Announce(prefix, asn, Route{Peer: peer, Time: routeTime})
table.Withdraw(prefix, peer)
other = other.Next()
}
}()
visits := make(map[netip.Prefix]int)
table.walkPrefixes(func(prefix netip.Prefix, _ []*origin) {
visits[prefix]++
})
close(stop)
<-updated
for _, prefix := range prefixes {
if visits[prefix] != 1 {
t.Fatalf("the walk visited %s %d times, want once", prefix,
visits[prefix])
}
}
}
// TestWalkPrefixesLetsUpdatesIn starts an update during the walk's first
// batch. The update must have been made by the time the walk visits the first
// prefix of its second batch, because the walk lets the lock go in between.
func TestWalkPrefixesLetsUpdatesIn(t *testing.T) {
t.Parallel()
const (
asn = 1000
peer = "192.0.2.1"
)
routeTime := time.Date(2026, 1, 2, 3, 4, 5, 0, time.UTC)
table := New(Params{})
addr := netip.MustParseAddr("1.0.0.0")
for range walkBatch + 1 {
prefix := netip.PrefixFrom(addr, addr.BitLen())
table.Announce(prefix, asn, Route{Peer: peer, Time: routeTime})
addr = addr.Next()
}
updatePrefix := netip.MustParsePrefix("2.0.0.0/8")
updateDone := make(chan struct{})
visited := 0
madeBeforeSecondBatch := false
table.walkPrefixes(func(netip.Prefix, []*origin) {
visited++
switch visited {
case 1:
go func() {
defer close(updateDone)
table.Announce(updatePrefix, asn, Route{Peer: peer, Time: routeTime})
}()
// Wait until the update waits for the lock. TryRLock fails
// once a writer waits.
for table.mu.TryRLock() {
table.mu.RUnlock()
runtime.Gosched()
}
case walkBatch + 1:
// The walk holds the read lock, so the table can be read.
madeBeforeSecondBatch = table.origins(updatePrefix) != nil
}
})
<-updateDone
if !madeBeforeSecondBatch {
t.Fatal("an update started during the walk's first batch was not made " +
"before its second batch")
}
}
internal/routetable/overview_test.go
+231
View File
@@ -0,0 +1,231 @@
package routetable_test
import (
"net/netip"
"reflect"
"testing"
"git.eeqj.de/sneak/routewatch/internal/routetable"
)
// Origin ASes that are not reserved, unlike asA, asB and asC, which are
// documentation AS numbers and so make bogon announcements.
const (
asOne = 1000
asTwo = 2000
asThree = 3000
asFour = 4000
asFive = 5000
// noKnownOrigin is the origin AS of a route whose path ends in an AS
// set.
noKnownOrigin = 0
// privateUseAS and asTrans are reserved AS numbers.
privateUseAS = 64512
asTrans = 23456
// maxListed is how many bogon announcements and prefixes with more than
// one origin AS the tests ask Overview for.
maxListed = 10
)
// prefixRoute returns the PrefixRoute of prefix with origin AS asn, second
// seconds after the start.
func prefixRoute(prefix string, asn, second int) *routetable.PrefixRoute {
return &routetable.PrefixRoute{
Prefix: netip.MustParsePrefix(prefix), OriginASN: asn, Time: at(second),
}
}
func TestOverviewOfAnEmptyTable(t *testing.T) {
t.Parallel()
got := routetable.New(routetable.Params{}).Overview(maxListed)
if !reflect.DeepEqual(got, routetable.Overview{}) {
t.Errorf("Overview() = %+v, want no figures", got)
}
}
// TestOverviewNewestAndOldestRoutes checks that the newest and the oldest kept
// routes are found among every origin's kept routes, and that withdrawn routes
// no longer count.
func TestOverviewNewestAndOldestRoutes(t *testing.T) {
t.Parallel()
table := routetable.New(routetable.Params{})
announceAll(table, "1.0.0.0/24", asOne, 20, peerA, peerB)
// The oldest is the older of this origin's two kept routes.
announceAll(table, "2.0.0.0/24", asTwo, 10, peerA, peerB)
announceAll(table, "2001:4860::/32", asThree, 40, peerC)
// Withdrawn routes, newer and older than every other.
announceAll(table, "3.0.0.0/24", asOne, 50, peerA)
announceAll(table, "4.0.0.0/24", asOne, 5, peerB)
table.Withdraw(netip.MustParsePrefix("3.0.0.0/24"), peerA)
table.Withdraw(netip.MustParsePrefix("4.0.0.0/24"), peerB)
got := table.Overview(maxListed)
want := prefixRoute("2001:4860::/32", asThree, 40)
if !reflect.DeepEqual(got.NewestRoute, want) {
t.Errorf("NewestRoute = %+v, want %+v", got.NewestRoute, want)
}
want = prefixRoute("2.0.0.0/24", asTwo, 10)
if !reflect.DeepEqual(got.OldestRoute, want) {
t.Errorf("OldestRoute = %+v, want %+v", got.OldestRoute, want)
}
}
// TestOverviewPrefixesPerOrigin checks the AS that originates the most
// prefixes and the ASes that originate one.
func TestOverviewPrefixesPerOrigin(t *testing.T) {
t.Parallel()
table := routetable.New(routetable.Params{})
// asOne and asTwo originate two prefixes each; the lower AS number counts.
announceAll(table, "1.0.0.0/24", asTwo, 10, peerA)
announceAll(table, "1.0.1.0/24", asTwo, 10, peerA)
announceAll(table, "2.0.0.0/24", asOne, 10, peerA)
announceAll(table, "2.0.1.0/24", asOne, 10, peerB)
// Three ASes originate one prefix. asThree's route is the newest, and
// asFive's origin keeps no route: peers B and C withdrew theirs.
announceAll(table, "3.0.0.0/24", asThree, 30, peerA)
announceAll(table, "4.0.0.0/24", asFour, 20, peerA)
announceAll(table, "5.0.0.0/24", asFive, 40, peerA, peerB, peerC)
table.Withdraw(netip.MustParsePrefix("5.0.0.0/24"), peerB)
table.Withdraw(netip.MustParsePrefix("5.0.0.0/24"), peerC)
got := table.Overview(maxListed)
if got.MostPrefixesASN != asOne || got.MostPrefixes != 2 {
t.Errorf("most prefixes: AS%d with %d, want AS%d with 2",
got.MostPrefixesASN, got.MostPrefixes, asOne)
}
if got.SinglePrefixASes != 3 {
t.Errorf("SinglePrefixASes = %d, want 3", got.SinglePrefixASes)
}
want := prefixRoute("3.0.0.0/24", asThree, 30)
if !reflect.DeepEqual(got.NewestSinglePrefix, want) {
t.Errorf("NewestSinglePrefix = %+v, want %+v", got.NewestSinglePrefix, want)
}
}
// TestOverviewLeavesOutNoKnownOrigin checks that AS 0, which stands for no
// known origin, counts in neither the most nor the single-prefix figures. It
// originates one prefix, with a route newer than asOne's, and asOne would lose
// the tie for the most prefixes to it.
func TestOverviewLeavesOutNoKnownOrigin(t *testing.T) {
t.Parallel()
table := routetable.New(routetable.Params{})
announceAll(table, "1.0.0.0/24", asOne, 10, peerA)
announceAll(table, "2.0.0.0/24", noKnownOrigin, 20, peerA)
got := table.Overview(maxListed)
if got.MostPrefixesASN != asOne || got.SinglePrefixASes != 1 {
t.Errorf("most prefixes AS%d, %d single-prefix ASes; want AS%d, 1",
got.MostPrefixesASN, got.SinglePrefixASes, asOne)
}
want := prefixRoute("1.0.0.0/24", asOne, 10)
if !reflect.DeepEqual(got.NewestSinglePrefix, want) {
t.Errorf("NewestSinglePrefix = %+v, want %+v", got.NewestSinglePrefix, want)
}
}
// TestOverviewBogons checks which routes are bogon announcements and why, in
// prefix order, and that a smaller maxListed lists fewer but counts them all.
func TestOverviewBogons(t *testing.T) {
t.Parallel()
table := routetable.New(routetable.Params{})
// A reserved prefix, carried by two peers.
announceAll(table, "192.168.0.0/16", asOne, 10, peerA, peerB)
// A reserved origin AS.
announceAll(table, "1.0.0.0/24", privateUseAS, 10, peerA)
// Both.
announceAll(table, "10.0.0.0/8", asTrans, 10, peerA)
// Neither: a normal route, and a block that may be announced although it
// lies inside a reserved one.
announceAll(table, "1.0.1.0/24", asOne, 10, peerA)
announceAll(table, "2001:4:112::/48", asTwo, 10, peerA)
want := []routetable.Bogon{
{
Prefix: netip.MustParsePrefix("1.0.0.0/24"), OriginASN: privateUseAS,
Peers: 1, Reasons: []string{"Private-Use (AS64512-AS65534)"},
},
{
Prefix: netip.MustParsePrefix("10.0.0.0/8"), OriginASN: asTrans,
Peers: 1,
Reasons: []string{
"Private-Use (10.0.0.0/8)", "AS_TRANS (AS23456)",
},
},
{
Prefix: netip.MustParsePrefix("192.168.0.0/16"), OriginASN: asOne,
Peers: 2, Reasons: []string{"Private-Use (192.168.0.0/16)"},
},
}
got := table.Overview(maxListed)
if !reflect.DeepEqual(got.Bogons, want) || got.BogonCount != len(want) {
t.Errorf("Overview(%d): %d bogons %+v, want %d %+v", maxListed,
got.BogonCount, got.Bogons, len(want), want)
}
const fewer = 2
got = table.Overview(fewer)
if !reflect.DeepEqual(got.Bogons, want[:fewer]) || got.BogonCount != len(want) {
t.Errorf("Overview(%d): %d bogons %+v, want %d %+v", fewer,
got.BogonCount, got.Bogons, len(want), want[:fewer])
}
}
// TestOverviewMultipleOrigins checks the prefixes with more than one origin
// AS, in prefix order, and that a smaller maxListed lists fewer but counts them
// all.
func TestOverviewMultipleOrigins(t *testing.T) {
t.Parallel()
table := routetable.New(routetable.Params{})
announceAll(table, "2001:4860::/32", asThree, 10, peerA)
announceAll(table, "2001:4860::/32", asTwo, 10, peerB)
announceAll(table, "1.0.0.0/24", asOne, 10, peerA, peerB)
announceAll(table, "1.0.0.0/24", asTwo, 30, peerC)
// One origin only.
announceAll(table, "2.0.0.0/24", asOne, 10, peerA, peerB)
want := []routetable.MultipleOriginPrefix{
{
Prefix: netip.MustParsePrefix("1.0.0.0/24"),
Origins: []routetable.OriginPeers{
{ASN: asOne, Peers: 2}, {ASN: asTwo, Peers: 1},
},
},
{
Prefix: netip.MustParsePrefix("2001:4860::/32"),
Origins: []routetable.OriginPeers{
{ASN: asTwo, Peers: 1}, {ASN: asThree, Peers: 1},
},
},
}
got := table.Overview(maxListed)
if !reflect.DeepEqual(got.MultipleOrigins, want) ||
got.MultipleOriginCount != len(want) {
t.Errorf("Overview(%d): %d prefixes %+v, want %d %+v", maxListed,
got.MultipleOriginCount, got.MultipleOrigins, len(want), want)
}
got = table.Overview(1)
if !reflect.DeepEqual(got.MultipleOrigins, want[:1]) ||
got.MultipleOriginCount != len(want) {
t.Errorf("Overview(1): %d prefixes %+v, want %d %+v",
got.MultipleOriginCount, got.MultipleOrigins, len(want), want[:1])
}
}
internal/routetable/reads.go
+6 -1
View File
@@ -14,6 +14,9 @@ type PrefixSummary struct {
// OriginASN is the origin AS that the most peers carry the prefix with. // OriginASN is the origin AS that the most peers carry the prefix with.
// When several have as many peers, it is the lowest AS number. // When several have as many peers, it is the lowest AS number.
OriginASN int OriginASN int
// OriginASNs are all the origin ASes that peers carry the prefix with, in
// AS number order.
OriginASNs []int
// Peers is how many peers carry the prefix. // Peers is how many peers carry the prefix.
Peers int Peers int
// Newest is the time of the newest kept route of the prefix, or nil when // Newest is the time of the newest kept route of the prefix, or nil when
@@ -211,10 +214,12 @@ func (t *Table) addresses(peers *bitset.BitSet) []string {
// summarize returns the summary of prefix, whose origins are in AS number // summarize returns the summary of prefix, whose origins are in AS number
// order. // order.
func summarize(prefix netip.Prefix, origins []*origin) PrefixSummary { func summarize(prefix netip.Prefix, origins []*origin) PrefixSummary {
summary := PrefixSummary{Prefix: prefix} summary := PrefixSummary{Prefix: prefix, OriginASNs: make([]int, 0, len(origins))}
mostPeers := 0 mostPeers := 0
for _, o := range origins { for _, o := range origins {
summary.OriginASNs = append(summary.OriginASNs, o.asn)
peers := int(o.peers.Count()) peers := int(o.peers.Count())
summary.Peers += peers summary.Peers += peers
internal/routetable/reads_test.go
+21 -18
View File
@@ -46,21 +46,22 @@ func TestLookup(t *testing.T) {
tests := []struct { tests := []struct {
addr string addr string
// want is the summary's prefix, or empty when none is found. // want is the summary's prefix, or empty when none is found.
want string want string
originASN int originASN int
peers int originASNs []int
newest int // in seconds after the start; 0 for none peers int
newest int // in seconds after the start; 0 for none
}{ }{
{"10.1.2.3", prefixV4, asA, 2, 20}, {"10.1.2.3", prefixV4, asA, []int{asA, asB}, 2, 20},
{"::ffff:10.1.2.3", prefixV4, asA, 2, 20}, {"::ffff:10.1.2.3", prefixV4, asA, []int{asA, asB}, 2, 20},
{"10.1.3.4", "10.1.0.0/16", asB, 3, 30}, {"10.1.3.4", "10.1.0.0/16", asB, []int{asA, asB}, 3, 30},
{"10.2.0.1", "10.0.0.0/8", asA, 1, 10}, {"10.2.0.1", "10.0.0.0/8", asA, []int{asA}, 1, 10},
{"10.9.0.1", "10.9.0.0/16", asA, 1, 0}, {"10.9.0.1", "10.9.0.0/16", asA, []int{asA}, 1, 0},
{"192.0.2.1", "", 0, 0, 0}, {"192.0.2.1", "", 0, nil, 0, 0},
{"2001:db8:1::5", prefixV6, asB, 1, 20}, {"2001:db8:1::5", prefixV6, asB, []int{asB}, 1, 20},
{"2001:db8:2::1", "2001:db8::/32", asA, 1, 10}, {"2001:db8:2::1", "2001:db8::/32", asA, []int{asA}, 1, 10},
{"3000::1", "3000::/16", asA, 1, 10}, {"3000::1", "3000::/16", asA, []int{asA}, 1, 10},
{"2001:db9::1", "", 0, 0, 0}, {"2001:db9::1", "", 0, nil, 0, 0},
} }
for _, tc := range tests { for _, tc := range tests {
@@ -77,9 +78,10 @@ func TestLookup(t *testing.T) {
} }
want := routetable.PrefixSummary{ want := routetable.PrefixSummary{
Prefix: netip.MustParsePrefix(tc.want), Prefix: netip.MustParsePrefix(tc.want),
OriginASN: tc.originASN, OriginASN: tc.originASN,
Peers: tc.peers, OriginASNs: tc.originASNs,
Peers: tc.peers,
} }
if tc.newest != 0 { if tc.newest != 0 {
want.Newest = timeAt(tc.newest) want.Newest = timeAt(tc.newest)
@@ -166,7 +168,8 @@ func TestSamplePrefixes(t *testing.T) {
for _, got := range sample { for _, got := range sample {
want := routetable.PrefixSummary{ want := routetable.PrefixSummary{
Prefix: got.Prefix, OriginASN: asA, Peers: 1, Newest: timeAt(10), Prefix: got.Prefix, OriginASN: asA, OriginASNs: []int{asA},
Peers: 1, Newest: timeAt(10),
} }
if seen[got.Prefix] || got.Prefix.Bits() != tc.maskLength || if seen[got.Prefix] || got.Prefix.Bits() != tc.maskLength ||
!reflect.DeepEqual(got, want) { !reflect.DeepEqual(got, want) {
internal/routetable/routetable_test.go
+1
View File
@@ -625,6 +625,7 @@ func TestConcurrentUse(t *testing.T) {
func(int) { table.OriginatedPrefixes(asA) }, func(int) { table.OriginatedPrefixes(asA) },
func(int) { table.SamplePrefixes(ipVersion, prefix.Bits(), sampleSize) }, func(int) { table.SamplePrefixes(ipVersion, prefix.Bits(), sampleSize) },
func(int) { table.Counts() }, func(int) { table.Counts() },
func(int) { table.Overview(sampleSize) },
} }
var wg sync.WaitGroup var wg sync.WaitGroup
internal/routewatch/app.go
+3 -4
View File
@@ -109,8 +109,9 @@ func (rw *RouteWatch) Run(ctx context.Context) error {
rw.peerHandler = NewPeerHandler(rw.db, rw.logger) rw.peerHandler = NewPeerHandler(rw.db, rw.logger)
rw.streamer.RegisterHandler(rw.peerHandler) rw.streamer.RegisterHandler(rw.peerHandler)
// PrefixHandler maintains the prefixes and live_routes tables, and // PrefixHandler keeps the routing table in memory current, and writes
// the routing table in memory and its copy in the routes table // its changes to the routes table and new prefixes to the prefix
// tables
rw.prefixHandler = NewPrefixHandler(rw.db, rw.logger, rw.table) rw.prefixHandler = NewPrefixHandler(rw.db, rw.logger, rw.table)
rw.prefixHandler.SetMetricsTracker(rw.streamer.GetMetricsTracker()) rw.prefixHandler.SetMetricsTracker(rw.streamer.GetMetricsTracker())
rw.streamer.RegisterHandler(rw.prefixHandler) rw.streamer.RegisterHandler(rw.prefixHandler)
@@ -125,8 +126,6 @@ func (rw *RouteWatch) Run(ctx context.Context) error {
return errNonBatchedNotImplemented return errNonBatchedNotImplemented
} }
// No longer need routing table handler - PrefixHandler maintains live_routes table
// Start streaming // Start streaming
err := rw.streamer.Start(ctx) err := rw.streamer.Start(ctx)
if err != nil { if err != nil {
internal/routewatch/app_integration_internal_test.go
+56 -183
View File
@@ -33,14 +33,12 @@ type mockStore struct {
ASNCount int ASNCount int
PrefixCount int PrefixCount int
PeeringCount int PeeringCount int
RouteCount int
WithdrawalCount int WithdrawalCount int
// Track unique items // Track unique items
ASNs map[int]*database.ASN ASNs map[int]*database.ASN
Prefixes map[string]*database.Prefix Prefixes map[string]*database.Prefix
Peerings map[string]bool // key is "from_to" Peerings map[string]bool // key is "from_to"
Routes map[string]bool // key is "prefix_origin_peer"
// TablePrefixes holds the prefixes that the written changes of the // TablePrefixes holds the prefixes that the written changes of the
// routing table leave some peer carrying. // routing table leave some peer carrying.
TablePrefixes map[string]bool TablePrefixes map[string]bool
@@ -56,7 +54,6 @@ func newMockStore() *mockStore {
ASNs: make(map[int]*database.ASN), ASNs: make(map[int]*database.ASN),
Prefixes: make(map[string]*database.Prefix), Prefixes: make(map[string]*database.Prefix),
Peerings: make(map[string]bool), Peerings: make(map[string]bool),
Routes: make(map[string]bool),
TablePrefixes: make(map[string]bool), TablePrefixes: make(map[string]bool),
} }
} }
@@ -107,6 +104,13 @@ func (m *mockStore) UpdatePrefixesBatch(prefixes map[string]time.Time) error {
IPVersion: ipVersion, IPVersion: ipVersion,
FirstSeen: timestamp, FirstSeen: timestamp,
} }
m.PrefixCount++
if ipVersion == ipVersionV4 {
m.IPv4Prefixes++
} else {
m.IPv6Prefixes++
}
} }
} }
@@ -207,65 +211,6 @@ func (m *mockStore) GetStatsContext(_ context.Context) (database.Stats, error) {
return m.GetStats() return m.GetStats()
} }
// UpsertLiveRoute mock implementation
func (m *mockStore) UpsertLiveRoute(_ *database.LiveRoute) error {
// Simple mock - just return nil
return nil
}
// DeleteLiveRoute mock implementation
func (m *mockStore) DeleteLiveRoute(_, _ string) error {
// Simple mock - just return nil
return nil
}
// GetPrefixDistribution mock implementation
func (m *mockStore) GetPrefixDistribution() (
[]database.PrefixDistribution, []database.PrefixDistribution, error,
) {
// Return empty distributions for now
return nil, nil, nil
}
// GetPrefixDistributionContext mock implementation with context support
func (m *mockStore) GetPrefixDistributionContext(_ context.Context) (
[]database.PrefixDistribution, []database.PrefixDistribution, error,
) {
return m.GetPrefixDistribution()
}
// GetLiveRouteCounts mock implementation
func (m *mockStore) GetLiveRouteCounts() (int, int, error) {
// Return mock counts
return m.RouteCount / 2, m.RouteCount / 2, nil
}
// GetLiveRouteCountsContext mock implementation with context support
func (m *mockStore) GetLiveRouteCountsContext(_ context.Context) (int, int, error) {
return m.GetLiveRouteCounts()
}
// GetASDetails mock implementation
func (m *mockStore) GetASDetails(asn int) (*database.ASN, []database.LiveRoute, error) {
m.mu.Lock()
defer m.mu.Unlock()
// Check if ASN exists
if asnInfo, exists := m.ASNs[asn]; exists {
// Return empty prefixes for now
return asnInfo, []database.LiveRoute{}, nil
}
return nil, nil, database.ErrNoRoute
}
// GetASDetailsContext mock implementation with context support
func (m *mockStore) GetASDetailsContext(
_ context.Context, asn int,
) (*database.ASN, []database.LiveRoute, error) {
return m.GetASDetails(asn)
}
// GetAS returns the stored AS, or only its number when none is stored. // GetAS returns the stored AS, or only its number when none is stored.
func (m *mockStore) GetAS(_ context.Context, asn int) (*database.ASN, bool, error) { func (m *mockStore) GetAS(_ context.Context, asn int) (*database.ASN, bool, error) {
m.mu.Lock() m.mu.Lock()
@@ -278,6 +223,35 @@ func (m *mockStore) GetAS(_ context.Context, asn int) (*database.ASN, bool, erro
return &database.ASN{ASN: asn}, false, nil return &database.ASN{ASN: asn}, false, nil
} }
// GetLastSeenASes returns copies of the stored ASes last seen most recently
// and longest ago, or neither when none is stored.
func (m *mockStore) GetLastSeenASes(
_ context.Context,
) (*database.ASN, *database.ASN, error) {
m.mu.Lock()
defer m.mu.Unlock()
var newest, oldest *database.ASN
for _, asn := range m.ASNs {
if newest == nil || asn.LastSeen.After(newest.LastSeen) {
newest = asn
}
if oldest == nil || asn.LastSeen.Before(oldest.LastSeen) {
oldest = asn
}
}
if newest == nil {
return nil, nil, nil
}
newestCopy, oldestCopy := *newest, *oldest
return &newestCopy, &oldestCopy, nil
}
// GetASWHOIS returns the stored AS, or only its number when none is stored. // GetASWHOIS returns the stored AS, or only its number when none is stored.
func (m *mockStore) GetASWHOIS(ctx context.Context, asn int) (*database.ASN, error) { func (m *mockStore) GetASWHOIS(ctx context.Context, asn int) (*database.ASN, error) {
asnInfo, _, err := m.GetAS(ctx, asn) asnInfo, _, err := m.GetAS(ctx, asn)
@@ -292,35 +266,6 @@ func (m *mockStore) GetPrefixFirstSeen(
return time.Time{}, nil return time.Time{}, nil
} }
// GetPrefixDetails mock implementation
func (m *mockStore) GetPrefixDetails(
_ string,
) ([]database.LiveRoute, map[int]database.ASN, error) {
// Return empty routes for now
return []database.LiveRoute{}, map[int]database.ASN{}, nil
}
// GetPrefixDetailsContext mock implementation with context support
func (m *mockStore) GetPrefixDetailsContext(
_ context.Context, prefix string,
) ([]database.LiveRoute, map[int]database.ASN, error) {
return m.GetPrefixDetails(prefix)
}
func (m *mockStore) GetRandomPrefixesByLength(
_, _, _ int,
) ([]database.LiveRoute, error) {
// Return empty routes for now
return []database.LiveRoute{}, nil
}
// GetRandomPrefixesByLengthContext mock implementation with context support
func (m *mockStore) GetRandomPrefixesByLengthContext(
_ context.Context, maskLength, ipVersion, limit int,
) ([]database.LiveRoute, error) {
return m.GetRandomPrefixesByLength(maskLength, ipVersion, limit)
}
// GetASPeers mock implementation // GetASPeers mock implementation
func (m *mockStore) GetASPeers(_ int) ([]database.ASPeer, error) { func (m *mockStore) GetASPeers(_ int) ([]database.ASPeer, error) {
// Return empty peers for now // Return empty peers for now
@@ -334,32 +279,6 @@ func (m *mockStore) GetASPeersContext(
return m.GetASPeers(asn) return m.GetASPeers(asn)
} }
// GetIPInfo mock implementation
func (m *mockStore) GetIPInfo(ip string) (*database.IPInfo, error) {
return m.GetIPInfoContext(context.Background(), ip)
}
// GetIPInfoContext mock implementation with context support
func (m *mockStore) GetIPInfoContext(
_ context.Context, ip string,
) (*database.IPInfo, error) {
now := time.Now()
return &database.IPInfo{
IP: ip,
Netblock: "8.8.8.0/24",
MaskLength: 24,
IPVersion: 4,
NumPeers: 3,
ASN: 15169,
Handle: "GOOGLE",
Description: "Google LLC",
CountryCode: "US",
FirstSeen: now.Add(-24 * time.Hour),
LastSeen: now,
}, nil
}
// GetNextStaleASN mock implementation // GetNextStaleASN mock implementation
func (m *mockStore) GetNextStaleASN(_ context.Context, _ time.Duration) (int, error) { func (m *mockStore) GetNextStaleASN(_ context.Context, _ time.Duration) (int, error) {
return 0, database.ErrNoStaleASN return 0, database.ErrNoStaleASN
@@ -387,46 +306,6 @@ func (m *mockStore) GetWHOISStats(
}, nil }, nil
} }
// UpsertLiveRouteBatch mock implementation
func (m *mockStore) UpsertLiveRouteBatch(routes []*database.LiveRoute) error {
m.mu.Lock()
defer m.mu.Unlock()
for _, route := range routes {
// Track prefix
if _, exists := m.Prefixes[route.Prefix]; !exists {
m.Prefixes[route.Prefix] = &database.Prefix{
Prefix: route.Prefix,
IPVersion: route.IPVersion,
FirstSeen: route.LastUpdated,
}
m.PrefixCount++
if route.IPVersion == 4 {
m.IPv4Prefixes++
} else {
m.IPv6Prefixes++
}
}
m.RouteCount++
}
return nil
}
// DeleteLiveRouteBatch mock implementation
func (m *mockStore) DeleteLiveRouteBatch(_ []database.LiveRouteDeletion) error {
// Simple mock - just return nil
return nil
}
// DeleteLiveRoutesOfPeer mock implementation
func (m *mockStore) DeleteLiveRoutesOfPeer(_ string) (int, error) {
// Simple mock - deletes nothing
return 0, nil
}
// WriteRouteChanges records the prefixes the changes leave some peer carrying, // WriteRouteChanges records the prefixes the changes leave some peer carrying,
// whatever their origin AS. // whatever their origin AS.
func (m *mockStore) WriteRouteChanges( func (m *mockStore) WriteRouteChanges(
@@ -630,8 +509,6 @@ func announcementFeed(prefixes []string) string {
func TestShutdownWritesTheUpdatesQueuedInTheStreamer(t *testing.T) { func TestShutdownWritesTheUpdatesQueuedInTheStreamer(t *testing.T) {
t.Parallel() t.Parallel()
const maxWait = 10 * time.Second
prefixes := []string{"192.0.2.0/24", "198.51.100.0/24", "203.0.113.0/24"} prefixes := []string{"192.0.2.0/24", "198.51.100.0/24", "203.0.113.0/24"}
feed := announcementFeed(prefixes) feed := announcementFeed(prefixes)
@@ -677,9 +554,9 @@ func TestShutdownWritesTheUpdatesQueuedInTheStreamer(t *testing.T) {
for range cap(linesSeen) { for range cap(linesSeen) {
select { select {
case <-linesSeen: case <-linesSeen:
case <-time.After(maxWait): case <-time.After(stepTimeout):
t.Fatalf("the streamer did not read %d lines within %s", t.Fatalf("the streamer did not read %d lines within %s",
cap(linesSeen), maxWait) cap(linesSeen), stepTimeout)
} }
} }
@@ -692,8 +569,8 @@ func TestShutdownWritesTheUpdatesQueuedInTheStreamer(t *testing.T) {
select { select {
case <-shutDown: case <-shutDown:
case <-time.After(maxWait): case <-time.After(stepTimeout):
t.Fatalf("Shutdown did not return within %s", maxWait) t.Fatalf("Shutdown did not return within %s", stepTimeout)
} }
store.mu.Lock() store.mu.Lock()
@@ -730,21 +607,16 @@ func freePort(t *testing.T) int {
return addr.Port return addr.Port
} }
// TestShutdownReturnsByItsDeadlineWhileAWriteIsBlocked starts the HTTP server // TestShutdownDoesNotWaitForABlockedWrite starts the HTTP server and has the
// and has the AS handler start a write that the store holds until the test // AS handler start a write that the store holds until the test ends, past
// ends, past Shutdown's deadline. Shutdown must return within its deadline // Shutdown's deadline. Shutdown must return without waiting for the write, with
// plus a small margin, with the HTTP server stopped, and must neither // the HTTP server stopped, and must neither checkpoint nor close the database:
// checkpoint nor close the database: the real one would wait for the write. // the real one would wait for the write.
// Before the fix Shutdown waited for the write. // Before the fix Shutdown waited for the write.
func TestShutdownReturnsByItsDeadlineWhileAWriteIsBlocked(t *testing.T) { func TestShutdownDoesNotWaitForABlockedWrite(t *testing.T) {
t.Parallel() t.Parallel()
const ( const deadline = 100 * time.Millisecond
deadline = 100 * time.Millisecond
// margin is how long Shutdown may take after its deadline, to stop
// the HTTP server.
margin = 2 * time.Second
)
blocking := newBlockingStore() blocking := newBlockingStore()
defer blocking.releaseWrites() defer blocking.releaseWrites()
@@ -755,12 +627,15 @@ func TestShutdownReturnsByItsDeadlineWhileAWriteIsBlocked(t *testing.T) {
cfg := &config.Config{Port: port} cfg := &config.Config{Port: port}
s := streamer.New(log, metrics.New()) s := streamer.New(log, metrics.New())
// A started server reads its routing table in the background, so it
// needs one.
rw := New(Dependencies{ rw := New(Dependencies{
DB: store, DB: store,
Streamer: s, Streamer: s,
Server: server.New(store, nil, s, log, cfg), Server: server.New(store, routetable.New(routetable.Params{}), s, log,
Logger: log, cfg),
Config: cfg, Logger: log,
Config: cfg,
}) })
err := rw.server.Start(t.Context()) err := rw.server.Start(t.Context())
@@ -785,9 +660,9 @@ func TestShutdownReturnsByItsDeadlineWhileAWriteIsBlocked(t *testing.T) {
select { select {
case <-shutDown: case <-shutDown:
case <-time.After(deadline + margin): case <-time.After(deadline + stepTimeout):
t.Fatalf("Shutdown did not return within %s of its deadline while "+ t.Fatalf("Shutdown did not return within %s of its deadline while "+
"a write was blocked", margin) "a write was blocked", stepTimeout)
} }
var dialer net.Dialer var dialer net.Dialer
@@ -862,8 +737,6 @@ func (s *closeOrderStore) record(call string) {
func TestShutdownCheckpointsAndClosesTheDatabaseLast(t *testing.T) { func TestShutdownCheckpointsAndClosesTheDatabaseLast(t *testing.T) {
t.Parallel() t.Parallel()
const deadline = 10 * time.Second
store := &closeOrderStore{Store: newMockStore()} store := &closeOrderStore{Store: newMockStore()}
log := logger.New() log := logger.New()
cfg := &config.Config{} cfg := &config.Config{}
@@ -881,7 +754,7 @@ func TestShutdownCheckpointsAndClosesTheDatabaseLast(t *testing.T) {
Path: ristypes.ASPath{Sequence: []int{testASNA}}, Path: ristypes.ASPath{Sequence: []int{testASNA}},
}) })
ctx, cancel := context.WithTimeout(t.Context(), deadline) ctx, cancel := context.WithTimeout(t.Context(), stepTimeout)
defer cancel() defer cancel()
rw.Shutdown(ctx) rw.Shutdown(ctx)
internal/routewatch/ashandler.go
+14 -9
View File
@@ -40,9 +40,13 @@ type ASHandler struct {
db database.Store db database.Store
logger *logger.Logger logger *logger.Logger
// batchMaxSize is asnBatchMaxSize. Only HandleMessage reads it, so a test
// may lower it before it hands the handler a message.
batchMaxSize int
// Batching. mu guards batch and writing only: the flush goroutine takes // Batching. mu guards batch and writing only: the flush goroutine takes
// the batch under mu and writes it after releasing mu, so HandleMessage // the batch under mu and writes it after releasing mu, so HandleMessage
// waits for a database write only while batch is at asnBatchMaxSize. // waits for a database write only while batch is at batchMaxSize.
mu sync.Mutex mu sync.Mutex
batch []asnOp batch []asnOp
// writing is how many operations the flush goroutine took from batch // writing is how many operations the flush goroutine took from batch
@@ -79,12 +83,13 @@ func newASHandler(
db database.Store, logger *logger.Logger, batchTimeout time.Duration, db database.Store, logger *logger.Logger, batchTimeout time.Duration,
) *ASHandler { ) *ASHandler {
h := &ASHandler{ h := &ASHandler{
db: db, db: db,
logger: logger, logger: logger,
batch: make([]asnOp, 0, asnBatchSize), batchMaxSize: asnBatchMaxSize,
batchFullCh: make(chan struct{}, 1), batch: make([]asnOp, 0, asnBatchSize),
stopCh: make(chan context.Context, 1), batchFullCh: make(chan struct{}, 1),
done: make(chan struct{}), stopCh: make(chan context.Context, 1),
done: make(chan struct{}),
} }
h.batchTaken.L = &h.mu h.batchTaken.L = &h.mu
@@ -127,7 +132,7 @@ func (h *ASHandler) HandleMessage(msg *ristypes.RISMessage) {
// While the batch is at its maximum size, wait until the flush goroutine // While the batch is at its maximum size, wait until the flush goroutine
// takes it. Meanwhile the handler takes nothing off its queue, which // takes it. Meanwhile the handler takes nothing off its queue, which
// fills, and the streamer drops messages for it. // fills, and the streamer drops messages for it.
for len(h.batch) >= asnBatchMaxSize { for len(h.batch) >= h.batchMaxSize {
h.batchTaken.Wait() h.batchTaken.Wait()
} }
@@ -151,7 +156,7 @@ func (h *ASHandler) HandleMessage(msg *ristypes.RISMessage) {
// A full batch wakes the flush goroutine. If it is writing, the wake-up // A full batch wakes the flush goroutine. If it is writing, the wake-up
// waits in batchFullCh and the batch keeps growing until the write ends // waits in batchFullCh and the batch keeps growing until the write ends
// or the batch reaches asnBatchMaxSize. // or the batch reaches batchMaxSize.
if len(h.batch) >= asnBatchSize { if len(h.batch) >= asnBatchSize {
select { select {
case h.batchFullCh <- struct{}{}: case h.batchFullCh <- struct{}{}:
internal/routewatch/ashandler_internal_test.go
+4 -4
View File
@@ -19,7 +19,7 @@ func TestASHandlerStoresTheEarliestAndLatestTimesOfANewAS(t *testing.T) {
db := newTestDatabase(t) db := newTestDatabase(t)
// Built without NewASHandler's flush goroutine, so the test flushes itself. // Built without NewASHandler's flush goroutine, so the test flushes itself.
h := &ASHandler{db: db, logger: logger.New()} h := &ASHandler{db: db, logger: logger.New(), batchMaxSize: asnBatchMaxSize}
first := peerAnnouncement(testPeerIP, testV4Prefix) first := peerAnnouncement(testPeerIP, testV4Prefix)
second := peerAnnouncement(testPeerIP, testV4Prefix) second := peerAnnouncement(testPeerIP, testV4Prefix)
@@ -29,9 +29,9 @@ func TestASHandlerStoresTheEarliestAndLatestTimesOfANewAS(t *testing.T) {
h.HandleMessage(second) h.HandleMessage(second)
h.flushBatch() h.flushBatch()
asn, _, err := db.GetASDetails(testASNA) asn, _, err := db.GetAS(t.Context(), testASNA)
if err != nil { if err != nil {
t.Fatalf("GetASDetails(%d): %v", testASNA, err) t.Fatalf("GetAS(%d): %v", testASNA, err)
} }
if !asn.FirstSeen.Equal(first.ParsedTimestamp) { if !asn.FirstSeen.Equal(first.ParsedTimestamp) {
@@ -54,7 +54,7 @@ func TestASHandlerRecordsNothingFromAPathWithASZero(t *testing.T) {
store := newMockStore() store := newMockStore()
// Built without NewASHandler's flush goroutine, so the test flushes itself. // Built without NewASHandler's flush goroutine, so the test flushes itself.
h := &ASHandler{db: store, logger: logger.New()} h := &ASHandler{db: store, logger: logger.New(), batchMaxSize: asnBatchMaxSize}
msg := peerAnnouncement(testPeerIP, testV4Prefix) msg := peerAnnouncement(testPeerIP, testV4Prefix)
msg.Path = decodePath(t, testASNA, 0, testASNB) msg.Path = decodePath(t, testASNA, 0, testASNB)
internal/routewatch/asnfetcher_internal_test.go
+4 -4
View File
@@ -224,7 +224,7 @@ func TestASNFetcherStopReturnsByItsDeadline(t *testing.T) {
select { select {
case <-lookupStarted: case <-lookupStarted:
case <-time.After(handleMessageTimeout): case <-time.After(stepTimeout):
t.Fatal("the fetcher did not start the lookup") t.Fatal("the fetcher did not start the lookup")
} }
@@ -240,22 +240,22 @@ func TestASNFetcherStopReturnsByItsDeadline(t *testing.T) {
select { select {
case <-f.done: case <-f.done:
case <-time.After(handleMessageTimeout): case <-time.After(stepTimeout):
t.Fatal("the fetcher goroutine did not return after the lookup ended") t.Fatal("the fetcher goroutine did not return after the lookup ended")
} }
} }
// TestRecordKeepsOnlyTheStatsWindow records a success and a failure every // TestRecordKeepsOnlyTheStatsWindow records a success and a failure every
// second, by a clock the test sets, far longer than statsWindow, and never // second, by a clock the test sets, for three times statsWindow, and never
// asks for the stats. The fetcher must keep only the times within the last // asks for the stats. The fetcher must keep only the times within the last
// statsWindow. // statsWindow.
func TestRecordKeepsOnlyTheStatsWindow(t *testing.T) { func TestRecordKeepsOnlyTheStatsWindow(t *testing.T) {
t.Parallel() t.Parallel()
const ( const (
records = 100_000
step = time.Second step = time.Second
want = int(statsWindow / step) want = int(statsWindow / step)
records = 3 * want
) )
clock := time.Now() clock := time.Now()
internal/routewatch/batchwrite_internal_test.go
+82 -52
View File
@@ -19,9 +19,15 @@ import (
) )
const ( const (
// handleMessageTimeout is how long a test waits for HandleMessage to // stepTimeout is how long a test waits for a step it expects, such as
// return, or for a write to start, before it fails. // HandleMessage returning, a write starting or Shutdown returning, before
handleMessageTimeout = 2 * time.Second // it fails. Only a step that never comes should reach it. On a loaded
// host, under the race detector, HandleMessage takes seconds to apply a
// full prefix batch. The helpers below bound every wait with it, Stop
// included. A test that hangs in one of them waits out at most two of
// these bounds, its step's and Stop's, and then fails with its own
// message. That leaves time before go test's 90-second timeout.
stepTimeout = 15 * time.Second
// stillWaitingAfter is how long a test watches a HandleMessage that must // stillWaitingAfter is how long a test watches a HandleMessage that must
// wait. One that does not wait returns well within it. // wait. One that does not wait returns well within it.
@@ -81,9 +87,9 @@ func TestFlushLoopWritesOnEveryTick(t *testing.T) {
} }
// blockingStore is a mockStore whose batch writes for the AS, peer and prefix // blockingStore is a mockStore whose batch writes for the AS, peer and prefix
// handlers, delete of a peer's routes and peering write wait until the test // handlers and peering write wait until the test calls releaseWrites.
// calls releaseWrites. writeStarted is closed when the first of them starts. It // writeStarted is closed when the first of them starts. It records what each
// records what each batch write was given. // batch write was given.
type blockingStore struct { type blockingStore struct {
*mockStore *mockStore
@@ -109,13 +115,13 @@ func newBlockingStore() *blockingStore {
} }
} }
func (s *blockingStore) UpsertLiveRouteBatch(routes []*database.LiveRoute) error { func (s *blockingStore) UpdatePrefixesBatch(prefixes map[string]time.Time) error {
s.waitForRelease() s.waitForRelease()
s.mu.Lock() s.mu.Lock()
defer s.mu.Unlock() defer s.mu.Unlock()
for _, route := range routes { for prefix := range prefixes {
s.prefixes[route.Prefix] = true s.prefixes[prefix] = true
} }
return nil return nil
@@ -145,12 +151,6 @@ func (s *blockingStore) UpdatePeerBatch(peers map[string]database.PeerUpdate) er
return nil return nil
} }
func (s *blockingStore) DeleteLiveRoutesOfPeer(peerIP string) (int, error) {
s.waitForRelease()
return s.mockStore.DeleteLiveRoutesOfPeer(peerIP)
}
func (s *blockingStore) RecordPeering( func (s *blockingStore) RecordPeering(
asA, asB int, seen database.SeenTimes, asA, asB int, seen database.SeenTimes,
) error { ) error {
@@ -176,7 +176,6 @@ func (s *blockingStore) releaseWrites() {
type stopLog struct { type stopLog struct {
Msg string `json:"msg"` Msg string `json:"msg"`
Updates int `json:"updates"` Updates int `json:"updates"`
PeersDown int `json:"peers_down"`
TableChanges int `json:"table_changes"` TableChanges int `json:"table_changes"`
Paths int `json:"paths"` Paths int `json:"paths"`
} }
@@ -205,7 +204,7 @@ func lastLogged(t *testing.T, logs *bytes.Buffer) stopLog {
return logged return logged
} }
// returnsInTime reports whether f returns within handleMessageTimeout. // returnsInTime reports whether f returns within stepTimeout.
func returnsInTime(f func()) bool { func returnsInTime(f func()) bool {
done := make(chan struct{}) done := make(chan struct{})
@@ -217,11 +216,22 @@ func returnsInTime(f func()) bool {
select { select {
case <-done: case <-done:
return true return true
case <-time.After(handleMessageTimeout): case <-time.After(stepTimeout):
return false return false
} }
} }
// stopInTime stops h, with no deadline, and fails the test if Stop does not
// return within stepTimeout. It only marks the test failed, so it can be
// deferred.
func stopInTime(t *testing.T, h interface{ Stop(ctx context.Context) }) {
t.Helper()
if !returnsInTime(func() { h.Stop(t.Context()) }) {
t.Errorf("Stop did not return within %s", stepTimeout)
}
}
// startBlockedWrite hands h the messages in fill and waits until h starts to // startBlockedWrite hands h the messages in fill and waits until h starts to
// write them to store. The prefix, AS and peer handlers start when fill fills // write them to store. The prefix, AS and peer handlers start when fill fills
// their batch, and the peering handler on its timer. store holds the write // their batch, and the peering handler on its timer. store holds the write
@@ -247,7 +257,7 @@ func startBlockedWrite(
select { select {
case <-store.writeStarted: case <-store.writeStarted:
case <-time.After(handleMessageTimeout): case <-time.After(stepTimeout):
t.Fatal("the handler did not start to write the messages") t.Fatal("the handler did not start to write the messages")
} }
} }
@@ -268,7 +278,7 @@ func handleWhileAWriteIsBlocked(
) { ) {
t.Helper() t.Helper()
defer h.Stop(t.Context()) defer stopInTime(t, h)
defer store.releaseWrites() defer store.releaseWrites()
startBlockedWrite(t, store, h, fill) startBlockedWrite(t, store, h, fill)
@@ -282,7 +292,9 @@ func handleWhileAWriteIsBlocked(
// while store holds the write, hands h the messages in toMax, which bring its // while store holds the write, hands h the messages in toMax, which bring its
// next batch to its maximum size, and the message extra. HandleMessage for // next batch to its maximum size, and the message extra. HandleMessage for
// extra must wait until the test lets the write finish, and return after. Then // extra must wait until the test lets the write finish, and return after. Then
// it stops h, which writes what it holds. // it stops h, which writes what it holds. The tests lower h's maximum batch
// size to one batch, so toMax is one batch: the real maximum of ten batches is
// slow to reach under the race detector.
func handleAtMaxBatchSize( func handleAtMaxBatchSize(
t *testing.T, t *testing.T,
store *blockingStore, store *blockingStore,
@@ -295,13 +307,18 @@ func handleAtMaxBatchSize(
) { ) {
t.Helper() t.Helper()
defer h.Stop(t.Context()) defer stopInTime(t, h)
defer store.releaseWrites() defer store.releaseWrites()
startBlockedWrite(t, store, h, fill) startBlockedWrite(t, store, h, fill)
for _, msg := range toMax { if !returnsInTime(func() {
h.HandleMessage(msg) for _, msg := range toMax {
h.HandleMessage(msg)
}
}) {
t.Fatal("HandleMessage did not return while the batch grew to its " +
"maximum size")
} }
returned := make(chan struct{}) returned := make(chan struct{})
@@ -322,7 +339,7 @@ func handleAtMaxBatchSize(
select { select {
case <-returned: case <-returned:
case <-time.After(handleMessageTimeout): case <-time.After(stepTimeout):
t.Fatal("HandleMessage did not return after the database write finished") t.Fatal("HandleMessage did not return after the database write finished")
} }
} }
@@ -352,8 +369,12 @@ func stopWhileAWriteIsBlocked(
startBlockedWrite(t, store, h, fill) startBlockedWrite(t, store, h, fill)
for _, msg := range extra { if !returnsInTime(func() {
h.HandleMessage(msg) for _, msg := range extra {
h.HandleMessage(msg)
}
}) {
t.Fatal("HandleMessage did not return while a database write was blocked")
} }
ctx, cancel := context.WithTimeout(t.Context(), stopDeadline) ctx, cancel := context.WithTimeout(t.Context(), stopDeadline)
@@ -368,16 +389,16 @@ func stopWhileAWriteIsBlocked(
select { select {
case <-done: case <-done:
case <-time.After(handleMessageTimeout): case <-time.After(stepTimeout):
t.Fatal("the flush goroutine did not return after the write finished") t.Fatal("the flush goroutine did not return after the write finished")
} }
} }
// TestPrefixHandlerSkipsItsLastWriteAfterTheStopDeadline also checks the // TestPrefixHandlerSkipsItsLastWriteAfterTheStopDeadline also checks the
// routing table's changes: the flush whose delete is held wrote those of the // routing table's changes: the flush whose prefix write is held wrote those of
// fill, so Stop must log the one of the extra prefix as not written. The flush // the fill, so Stop must log the one of the extra prefix as not written. The
// timer must not fire. A flush it started before the fill would be the one // flush timer must not fire. A flush it started before the fill would be the
// held, and it would not have written the fill's change. // one held, and it would not have written the fill's change.
func TestPrefixHandlerSkipsItsLastWriteAfterTheStopDeadline(t *testing.T) { func TestPrefixHandlerSkipsItsLastWriteAfterTheStopDeadline(t *testing.T) {
t.Parallel() t.Parallel()
@@ -387,12 +408,11 @@ func TestPrefixHandlerSkipsItsLastWriteAfterTheStopDeadline(t *testing.T) {
store := newBlockingStore() store := newBlockingStore()
h := newPrefixHandler(store, newErrorLogger(&logs), h := newPrefixHandler(store, newErrorLogger(&logs),
routetable.New(routetable.Params{}), longerThanAnyTest) routetable.New(routetable.Params{}), longerThanAnyTest,
maxTableChangesPerWrite)
fill := slices.Repeat([]string{testV4Prefix}, prefixBatchSize) fill := slices.Repeat([]string{testV4Prefix}, prefixBatchSize)
// The peer's session goes down first, so the write held is the delete of
// its routes, and Stop must log that peer down too.
stopWhileAWriteIsBlocked(t, store, h, h.done, stopWhileAWriteIsBlocked(t, store, h, h.done,
[]*ristypes.RISMessage{sessionDown(), prefixAnnouncement(fill)}, []*ristypes.RISMessage{prefixAnnouncement(fill)},
prefixAnnouncement([]string{extraPrefix})) prefixAnnouncement([]string{extraPrefix}))
store.mu.Lock() store.mu.Lock()
@@ -403,11 +423,10 @@ func TestPrefixHandlerSkipsItsLastWriteAfterTheStopDeadline(t *testing.T) {
} }
logged := lastLogged(t, &logs) logged := lastLogged(t, &logs)
if logged.Updates != prefixBatchSize+1 || logged.PeersDown != 1 || if logged.Updates != prefixBatchSize+1 || logged.TableChanges != 1 {
logged.TableChanges != 1 { t.Errorf("Stop logged %d updates and %d table changes not written, "+
t.Errorf("Stop logged %d updates, %d peer downs and %d table changes "+ "want %d and 1", logged.Updates, logged.TableChanges,
"not written, want %d, 1 and 1", logged.Updates, logged.PeersDown, prefixBatchSize+1)
logged.TableChanges, prefixBatchSize+1)
} }
} }
@@ -607,7 +626,7 @@ func TestPeeringHandlerStopCountsTheWriteInProgress(t *testing.T) {
select { select {
case <-h.done: case <-h.done:
case <-time.After(handleMessageTimeout): case <-time.After(stepTimeout):
t.Fatal("the processing goroutine did not return after the write " + t.Fatal("the processing goroutine did not return after the write " +
"finished") "finished")
} }
@@ -628,7 +647,7 @@ func TestPeeringHandlerStartsNoWriteAfterTheStopDeadline(t *testing.T) {
const processInterval = 10 * time.Millisecond const processInterval = 10 * time.Millisecond
store := newBlockingStore() store := newBlockingStore()
h := newPeeringHandler(store, logger.New(), processInterval) h := newPeeringHandler(store, logger.New(), processInterval, maxTrackedPaths)
stopWhileAWriteIsBlocked(t, store, h, h.done, stopWhileAWriteIsBlocked(t, store, h, h.done,
[]*ristypes.RISMessage{ []*ristypes.RISMessage{
{Path: ristypes.ASPath{Sequence: []int{testASNA, testASNB}}}, {Path: ristypes.ASPath{Sequence: []int{testASNA, testASNB}}},
@@ -724,9 +743,11 @@ func TestPrefixHandlerWaitsAtTheMaximumBatchSize(t *testing.T) {
const extraPrefix = "198.51.100.0/24" const extraPrefix = "198.51.100.0/24"
store := newBlockingStore() store := newBlockingStore()
h := NewPrefixHandler(store, logger.New(), routetable.New(routetable.Params{})) h := newPrefixHandler(store, logger.New(), routetable.New(routetable.Params{}),
longerThanAnyTest, maxTableChangesPerWrite)
h.batchMaxSize = prefixBatchSize
fill := slices.Repeat([]string{testV4Prefix}, prefixBatchSize) fill := slices.Repeat([]string{testV4Prefix}, prefixBatchSize)
toMax := slices.Repeat([]string{testV4Prefix}, prefixBatchMaxSize) toMax := slices.Repeat([]string{testV4Prefix}, prefixBatchSize)
handleAtMaxBatchSize(t, store, h, handleAtMaxBatchSize(t, store, h,
[]*ristypes.RISMessage{prefixAnnouncement(fill)}, []*ristypes.RISMessage{prefixAnnouncement(fill)},
[]*ristypes.RISMessage{prefixAnnouncement(toMax)}, []*ristypes.RISMessage{prefixAnnouncement(toMax)},
@@ -746,8 +767,9 @@ func TestASHandlerWaitsAtTheMaximumBatchSize(t *testing.T) {
store := newBlockingStore() store := newBlockingStore()
h := newASHandler(store, logger.New(), longerThanAnyTest) h := newASHandler(store, logger.New(), longerThanAnyTest)
h.batchMaxSize = asnBatchSize
fill := ristypes.ASPath{Sequence: slices.Repeat([]int{testASNA}, asnBatchSize)} fill := ristypes.ASPath{Sequence: slices.Repeat([]int{testASNA}, asnBatchSize)}
toMax := ristypes.ASPath{Sequence: slices.Repeat([]int{testASNA}, asnBatchMaxSize)} toMax := ristypes.ASPath{Sequence: slices.Repeat([]int{testASNA}, asnBatchSize)}
handleAtMaxBatchSize(t, store, h, handleAtMaxBatchSize(t, store, h,
[]*ristypes.RISMessage{{Path: fill}}, []*ristypes.RISMessage{{Path: fill}},
[]*ristypes.RISMessage{{Path: toMax}}, []*ristypes.RISMessage{{Path: toMax}},
@@ -765,7 +787,7 @@ func TestASHandlerWaitsAtTheMaximumBatchSize(t *testing.T) {
// TestPeerHandlerWaitsAtTheMaximumBatchSize must not have its flush timer fire. // TestPeerHandlerWaitsAtTheMaximumBatchSize must not have its flush timer fire.
// A timer flush during the fill would take only part of it. The batch would // A timer flush during the fill would take only part of it. The batch would
// then reach its maximum size before the test's last message, and the test // then reach its maximum size before the test's last message, and the test
// would hang. // would fail.
func TestPeerHandlerWaitsAtTheMaximumBatchSize(t *testing.T) { func TestPeerHandlerWaitsAtTheMaximumBatchSize(t *testing.T) {
t.Parallel() t.Parallel()
@@ -773,10 +795,11 @@ func TestPeerHandlerWaitsAtTheMaximumBatchSize(t *testing.T) {
store := newBlockingStore() store := newBlockingStore()
h := newPeerHandler(store, logger.New(), longerThanAnyTest) h := newPeerHandler(store, logger.New(), longerThanAnyTest)
h.batchMaxSize = peerBatchSize
message := []*ristypes.RISMessage{{Peer: testPeerIP}} message := []*ristypes.RISMessage{{Peer: testPeerIP}}
handleAtMaxBatchSize(t, store, h, handleAtMaxBatchSize(t, store, h,
slices.Repeat(message, peerBatchSize), slices.Repeat(message, peerBatchSize),
slices.Repeat(message, peerBatchMaxSize), slices.Repeat(message, peerBatchSize),
&ristypes.RISMessage{Peer: otherPeerIP}) &ristypes.RISMessage{Peer: otherPeerIP})
store.mu.Lock() store.mu.Lock()
@@ -806,8 +829,9 @@ func TestHandlersWakeTheFlushGoroutineWhenTheBatchIsFull(t *testing.T) {
batchSize: prefixBatchSize, batchSize: prefixBatchSize,
handle: func(n int) chan struct{} { handle: func(n int) chan struct{} {
h := &PrefixHandler{ h := &PrefixHandler{
batchFullCh: make(chan struct{}, 1), batchMaxSize: prefixBatchMaxSize,
table: routetable.New(routetable.Params{}), batchFullCh: make(chan struct{}, 1),
table: routetable.New(routetable.Params{}),
} }
h.HandleMessage(prefixAnnouncement( h.HandleMessage(prefixAnnouncement(
slices.Repeat([]string{testV4Prefix}, n))) slices.Repeat([]string{testV4Prefix}, n)))
@@ -819,7 +843,10 @@ func TestHandlersWakeTheFlushGoroutineWhenTheBatchIsFull(t *testing.T) {
name: "AS", name: "AS",
batchSize: asnBatchSize, batchSize: asnBatchSize,
handle: func(n int) chan struct{} { handle: func(n int) chan struct{} {
h := &ASHandler{batchFullCh: make(chan struct{}, 1)} h := &ASHandler{
batchMaxSize: asnBatchMaxSize,
batchFullCh: make(chan struct{}, 1),
}
// The origin AS and each AS in the path are one update each. // The origin AS and each AS in the path are one update each.
h.HandleMessage(&ristypes.RISMessage{ h.HandleMessage(&ristypes.RISMessage{
Path: ristypes.ASPath{Sequence: slices.Repeat([]int{testASNA}, n-1)}, Path: ristypes.ASPath{Sequence: slices.Repeat([]int{testASNA}, n-1)},
@@ -832,7 +859,10 @@ func TestHandlersWakeTheFlushGoroutineWhenTheBatchIsFull(t *testing.T) {
name: "peer", name: "peer",
batchSize: peerBatchSize, batchSize: peerBatchSize,
handle: func(n int) chan struct{} { handle: func(n int) chan struct{} {
h := &PeerHandler{batchFullCh: make(chan struct{}, 1)} h := &PeerHandler{
batchMaxSize: peerBatchMaxSize,
batchFullCh: make(chan struct{}, 1),
}
for range n { for range n {
h.HandleMessage(&ristypes.RISMessage{Peer: testPeerIP}) h.HandleMessage(&ristypes.RISMessage{Peer: testPeerIP})
} }
internal/routewatch/dbmaintainer_internal_test.go
+2 -2
View File
@@ -97,7 +97,7 @@ func TestDBMaintainerStopReturnsByItsDeadline(t *testing.T) {
select { select {
case <-store.started: case <-store.started:
case <-time.After(handleMessageTimeout): case <-time.After(stepTimeout):
t.Fatal("the maintainer did not start the checkpoint") t.Fatal("the maintainer did not start the checkpoint")
} }
@@ -113,7 +113,7 @@ func TestDBMaintainerStopReturnsByItsDeadline(t *testing.T) {
select { select {
case <-m.done: case <-m.done:
case <-time.After(handleMessageTimeout): case <-time.After(stepTimeout):
t.Fatal("the maintenance goroutine did not return after the " + t.Fatal("the maintenance goroutine did not return after the " +
"checkpoint ended") "checkpoint ended")
} }
internal/routewatch/peerhandler.go
+14 -9
View File
@@ -43,10 +43,14 @@ type PeerHandler struct {
db database.Store db database.Store
logger *logger.Logger logger *logger.Logger
// batchMaxSize is peerBatchMaxSize. Only HandleMessage reads it, so a test
// may lower it before it hands the handler a message.
batchMaxSize int
// Batching. mu guards peerBatch and writing only: the flush goroutine // Batching. mu guards peerBatch and writing only: the flush goroutine
// takes the batch under mu and writes it after releasing mu, so // takes the batch under mu and writes it after releasing mu, so
// HandleMessage waits for a database write only while peerBatch is at // HandleMessage waits for a database write only while peerBatch is at
// peerBatchMaxSize. // batchMaxSize.
mu sync.Mutex mu sync.Mutex
peerBatch []peerUpdate peerBatch []peerUpdate
// writing is how many updates the flush goroutine took from peerBatch // writing is how many updates the flush goroutine took from peerBatch
@@ -84,12 +88,13 @@ func newPeerHandler(
db database.Store, logger *logger.Logger, batchTimeout time.Duration, db database.Store, logger *logger.Logger, batchTimeout time.Duration,
) *PeerHandler { ) *PeerHandler {
h := &PeerHandler{ h := &PeerHandler{
db: db, db: db,
logger: logger, logger: logger,
peerBatch: make([]peerUpdate, 0, peerBatchSize), batchMaxSize: peerBatchMaxSize,
batchFullCh: make(chan struct{}, 1), peerBatch: make([]peerUpdate, 0, peerBatchSize),
stopCh: make(chan context.Context, 1), batchFullCh: make(chan struct{}, 1),
done: make(chan struct{}), stopCh: make(chan context.Context, 1),
done: make(chan struct{}),
} }
h.batchTaken.L = &h.mu h.batchTaken.L = &h.mu
@@ -135,7 +140,7 @@ func (h *PeerHandler) HandleMessage(msg *ristypes.RISMessage) {
// While the batch is at its maximum size, wait until the flush goroutine // While the batch is at its maximum size, wait until the flush goroutine
// takes it. Meanwhile the handler takes nothing off its queue, which // takes it. Meanwhile the handler takes nothing off its queue, which
// fills, and the streamer drops messages for it. // fills, and the streamer drops messages for it.
for len(h.peerBatch) >= peerBatchMaxSize { for len(h.peerBatch) >= h.batchMaxSize {
h.batchTaken.Wait() h.batchTaken.Wait()
} }
@@ -149,7 +154,7 @@ func (h *PeerHandler) HandleMessage(msg *ristypes.RISMessage) {
// A full batch wakes the flush goroutine. If it is writing, the wake-up // A full batch wakes the flush goroutine. If it is writing, the wake-up
// waits in batchFullCh and the batch keeps growing until the write ends // waits in batchFullCh and the batch keeps growing until the write ends
// or the batch reaches peerBatchMaxSize. // or the batch reaches batchMaxSize.
if len(h.peerBatch) >= peerBatchSize { if len(h.peerBatch) >= peerBatchSize {
select { select {
case h.batchFullCh <- struct{}{}: case h.batchFullCh <- struct{}{}:
internal/routewatch/peerhandler_internal_test.go
+1 -1
View File
@@ -35,7 +35,7 @@ func TestPeerHandlerWritesTheEarliestAndLatestTimesOfANewPeer(t *testing.T) {
// Built without NewPeerHandler's flush goroutine, so the test flushes // Built without NewPeerHandler's flush goroutine, so the test flushes
// itself. // itself.
h := &PeerHandler{db: store, logger: logger.New()} h := &PeerHandler{db: store, logger: logger.New(), batchMaxSize: peerBatchMaxSize}
first := peerAnnouncement(testPeerIP, testV4Prefix) first := peerAnnouncement(testPeerIP, testV4Prefix)
second := peerAnnouncement(testPeerIP, testV4Prefix) second := peerAnnouncement(testPeerIP, testV4Prefix)
internal/routewatch/peeringhandler.go
+16 -9
View File
@@ -42,6 +42,10 @@ type PeeringHandler struct {
db database.Store db database.Store
logger *logger.Logger logger *logger.Logger
// maxPaths is the most paths asPaths holds: maxTrackedPaths, except in
// tests that give a smaller one.
maxPaths int
// In-memory AS path tracking: each path, JSON-encoded, with the earliest // In-memory AS path tracking: each path, JSON-encoded, with the earliest
// and the latest time an update carried it. // and the latest time an update carried it.
mu sync.Mutex mu sync.Mutex
@@ -62,21 +66,24 @@ type PeeringHandler struct {
// periodically processes accumulated AS paths into peering records. The // periodically processes accumulated AS paths into peering records. The
// handler begins processing immediately upon creation. // handler begins processing immediately upon creation.
func NewPeeringHandler(db database.Store, logger *logger.Logger) *PeeringHandler { func NewPeeringHandler(db database.Store, logger *logger.Logger) *PeeringHandler {
return newPeeringHandler(db, logger, peeringProcessInterval) return newPeeringHandler(db, logger, peeringProcessInterval, maxTrackedPaths)
} }
// newPeeringHandler is NewPeeringHandler with the processing timer's interval, // newPeeringHandler is NewPeeringHandler with the processing timer's interval,
// processInterval, in place of peeringProcessInterval. A test gives a short // processInterval, in place of peeringProcessInterval, and the most paths it
// one to have the timer start a write. // holds, maxPaths, in place of maxTrackedPaths. A test gives a short interval to
// have the timer start a write, and a small maxPaths to fill the map quickly.
func newPeeringHandler( func newPeeringHandler(
db database.Store, logger *logger.Logger, processInterval time.Duration, db database.Store, logger *logger.Logger, processInterval time.Duration,
maxPaths int,
) *PeeringHandler { ) *PeeringHandler {
h := &PeeringHandler{ h := &PeeringHandler{
db: db, db: db,
logger: logger, logger: logger,
asPaths: make(map[string]database.SeenTimes), asPaths: make(map[string]database.SeenTimes),
stopCh: make(chan context.Context, 1), maxPaths: maxPaths,
done: make(chan struct{}), stopCh: make(chan context.Context, 1),
done: make(chan struct{}),
} }
// Start the periodic processing goroutine // Start the periodic processing goroutine
@@ -131,7 +138,7 @@ func (h *PeeringHandler) HandleMessage(msg *ristypes.RISMessage) {
defer h.mu.Unlock() defer h.mu.Unlock()
seen, exists := h.asPaths[key] seen, exists := h.asPaths[key]
if !exists && len(h.asPaths) >= maxTrackedPaths { if !exists && len(h.asPaths) >= h.maxPaths {
// Map is full; drop this new path and count it. // Map is full; drop this new path and count it.
h.droppedPaths++ h.droppedPaths++
internal/routewatch/peeringhandler_internal_test.go
+31 -18
View File
@@ -5,7 +5,6 @@ import (
"fmt" "fmt"
"maps" "maps"
"slices" "slices"
"strconv"
"sync" "sync"
"testing" "testing"
"time" "time"
@@ -20,6 +19,10 @@ const (
testASNB = 64501 testASNB = 64501
testASNC = 64502 testASNC = 64502
testASND = 64503 testASND = 64503
// testMaxTrackedPaths is a limit of the path map that a test fills
// quickly, in place of maxTrackedPaths.
testMaxTrackedPaths = 10
) )
// recordingStore wraps mockStore to count every RecordPeering call, so a // recordingStore wraps mockStore to count every RecordPeering call, so a
@@ -52,9 +55,10 @@ func (r *recordingStore) callCount() int {
// processing goroutine, so tests drive processing explicitly. // processing goroutine, so tests drive processing explicitly.
func newTestHandler(db database.Store) *PeeringHandler { func newTestHandler(db database.Store) *PeeringHandler {
return &PeeringHandler{ return &PeeringHandler{
db: db, db: db,
logger: logger.New(), logger: logger.New(),
asPaths: make(map[string]database.SeenTimes), asPaths: make(map[string]database.SeenTimes),
maxPaths: maxTrackedPaths,
} }
} }
@@ -244,25 +248,28 @@ func TestPeeringHandlerStopWritesTheRemainingPaths(t *testing.T) {
// TestPeeringHandlerCapDropsAndCounts verifies that a full map drops new paths // TestPeeringHandlerCapDropsAndCounts verifies that a full map drops new paths
// and counts them, while a path already tracked is refreshed rather than // and counts them, while a path already tracked is refreshed rather than
// dropped. // dropped. The handler's limit is small, and its timer does not fire.
func TestPeeringHandlerCapDropsAndCounts(t *testing.T) { func TestPeeringHandlerCapDropsAndCounts(t *testing.T) {
t.Parallel() t.Parallel()
store := &recordingStore{mockStore: newMockStore()} store := &recordingStore{mockStore: newMockStore()}
h := newTestHandler(store) h := newPeeringHandler(store, logger.New(), longerThanAnyTest,
testMaxTrackedPaths)
defer stopInTime(t, h)
now := time.Now().UTC() now := time.Now().UTC()
// Fill the map to exactly maxTrackedPaths, including one real path key so // Fill the map to exactly its limit, including one real path key so the
// the "already tracked" branch can be exercised. The filler keys are never // "already tracked" branch can be exercised. The other paths are of one
// processed in this test, so their contents do not matter. // AS, so they carry no peering.
existing := pathKey(t, testASNA, testASNB) existing := pathKey(t, testASNA, testASNB)
h.mu.Lock() h.mu.Lock()
h.asPaths[existing] = database.SeenTimes{First: now, Last: now} h.asPaths[existing] = database.SeenTimes{First: now, Last: now}
for i := 0; len(h.asPaths) < maxTrackedPaths; i++ { for asn := 1; len(h.asPaths) < testMaxTrackedPaths; asn++ {
h.asPaths[strconv.Itoa(i)] = database.SeenTimes{First: now, Last: now} h.asPaths[pathKey(t, asn)] = database.SeenTimes{First: now, Last: now}
} }
h.mu.Unlock() h.mu.Unlock()
@@ -270,8 +277,9 @@ func TestPeeringHandlerCapDropsAndCounts(t *testing.T) {
handle(h, now.Add(time.Second), testASNA, testASNC) handle(h, now.Add(time.Second), testASNA, testASNC)
tracked, dropped := snapshot(h) tracked, dropped := snapshot(h)
if tracked != maxTrackedPaths { if tracked != testMaxTrackedPaths {
t.Fatalf("want map size %d after drop, got %d", maxTrackedPaths, tracked) t.Fatalf("want map size %d after drop, got %d", testMaxTrackedPaths,
tracked)
} }
if dropped != 1 { if dropped != 1 {
@@ -283,8 +291,9 @@ func TestPeeringHandlerCapDropsAndCounts(t *testing.T) {
handle(h, refreshed, testASNA, testASNB) handle(h, refreshed, testASNA, testASNB)
tracked, dropped = snapshot(h) tracked, dropped = snapshot(h)
if tracked != maxTrackedPaths { if tracked != testMaxTrackedPaths {
t.Fatalf("want map size %d after refresh, got %d", maxTrackedPaths, tracked) t.Fatalf("want map size %d after refresh, got %d", testMaxTrackedPaths,
tracked)
} }
if dropped != 1 { if dropped != 1 {
@@ -303,17 +312,21 @@ func TestPeeringHandlerCapDropsAndCounts(t *testing.T) {
// TestPeeringHandlerRunResetsTheDroppedCount fills the map, has the handler // TestPeeringHandlerRunResetsTheDroppedCount fills the map, has the handler
// drop one path, and runs the processing. The count of dropped paths is the // drop one path, and runs the processing. The count of dropped paths is the
// run's own, so it is 0 again after the run. // run's own, so it is 0 again after the run. The handler's limit is small, and
// its timer does not fire.
func TestPeeringHandlerRunResetsTheDroppedCount(t *testing.T) { func TestPeeringHandlerRunResetsTheDroppedCount(t *testing.T) {
t.Parallel() t.Parallel()
h := newTestHandler(newMockStore()) h := newPeeringHandler(newMockStore(), logger.New(), longerThanAnyTest,
testMaxTrackedPaths)
defer stopInTime(t, h)
now := time.Now().UTC() now := time.Now().UTC()
// Paths of one AS carry no peering, so the run records nothing. // Paths of one AS carry no peering, so the run records nothing.
h.mu.Lock() h.mu.Lock()
for asn := 1; len(h.asPaths) < maxTrackedPaths; asn++ { for asn := 1; len(h.asPaths) < testMaxTrackedPaths; asn++ {
h.asPaths[pathKey(t, asn)] = database.SeenTimes{First: now, Last: now} h.asPaths[pathKey(t, asn)] = database.SeenTimes{First: now, Last: now}
} }
h.mu.Unlock() h.mu.Unlock()
internal/routewatch/prefixhandler.go
+106 -368
View File
@@ -2,10 +2,8 @@ package routewatch
import ( import (
"context" "context"
"net"
"net/netip" "net/netip"
"slices" "slices"
"strings"
"sync" "sync"
"time" "time"
@@ -28,11 +26,9 @@ const (
prefixBatchSize = 25000 prefixBatchSize = 25000
// prefixBatchMaxSize is the most prefix updates the batch holds while the // prefixBatchMaxSize is the most prefix updates the batch holds while the
// flush goroutine writes the previous one: ten batches, about 60 MiB // flush goroutine writes the previous one: ten batches, about 15 MiB
// (README "Memory"). At this size HandleMessage waits until the flush // (README "Memory"). At this size HandleMessage waits until the flush
// goroutine takes the batch. On top of these, the handler holds at most // goroutine takes the batch.
// prefixBatchSize updates, about 6 MiB, that a flush kept for the peers
// whose route delete failed (held).
prefixBatchMaxSize = 250000 prefixBatchMaxSize = 250000
// prefixBatchTimeout is the maximum time to wait before flushing a batch // prefixBatchTimeout is the maximum time to wait before flushing a batch
@@ -48,47 +44,38 @@ const (
// most one pair. So the million pairs that a full-table peer's session // most one pair. So the million pairs that a full-table peer's session
// going down changes are written in about a minute. // going down changes are written in about a minute.
maxTableChangesPerWrite = 20000 maxTableChangesPerWrite = 20000
// IP version constants
ipv4Version = 4
ipv6Version = 6
) )
// PrefixHandler tracks BGP prefixes and maintains a live routing table in the // PrefixHandler keeps the routing table in memory current from the feed. It
// database. Routes are added on announcement and deleted on withdrawal or on an // applies each update to the table as it handles it. An announcement gives
// announcement whose AS path is empty or holds AS 0 anywhere, and all routes of // the peer a route to the prefix. A withdrawal removes the peer's route, as
// a peer are deleted when its BGP session goes down. // does an announcement whose AS path is empty or holds AS 0 anywhere. A peer
// whose BGP session goes down leaves every prefix.
// //
// It applies the same updates to the routing table in memory as it handles // Each flush writes the table's changes to the database. It also stores each
// them, and writes the table's changes to the database with each flush. // prefix the updates name with the time it was first seen.
type PrefixHandler struct { type PrefixHandler struct {
db database.Store db database.Store
logger *logger.Logger logger *logger.Logger
metrics *metrics.Tracker metrics *metrics.Tracker
table *routetable.Table table *routetable.Table
// Batching. mu guards batch, held, peersDown, writing, deleting and // batchMaxSize is prefixBatchMaxSize. Only HandleMessage reads it, so a
// writingTableChanges only: the flush goroutine takes them under mu and // test may lower it before it hands the handler a message.
// writes them after releasing mu, so HandleMessage waits for a database batchMaxSize int
// write only while batch is at prefixBatchMaxSize. // tableChangesPerWrite is maxTableChangesPerWrite, except in tests that
// give a smaller one.
tableChangesPerWrite int
// Batching. mu guards batch, writing and writingTableChanges only: the
// flush goroutine takes them under mu and writes them after releasing
// mu, so HandleMessage waits for a database write only while batch is at
// batchMaxSize.
mu sync.Mutex mu sync.Mutex
batch []prefixUpdate batch []seenPrefix
// held holds the updates that the peers whose route delete failed in the
// last flush sent after their session went down, for the next flush to
// write after it retries the delete. They are not in batch, so they do
// not count toward a full batch: a delete that keeps failing is retried
// at the pace of the flush timer.
held []prefixUpdate
// peersDown holds, each once, the peers whose BGP session went down
// since the batch was taken, and those whose route delete failed in the
// last flush.
peersDown []string
// writing is how many updates the flush goroutine took and has not // writing is how many updates the flush goroutine took and has not
// finished writing. // finished writing.
writing int writing int
// deleting is how many peers from peersDown the flush goroutine took and
// has not yet deleted the routes of.
deleting int
// writingTableChanges is how many changes the flush goroutine took from // writingTableChanges is how many changes the flush goroutine took from
// the routing table and has not finished writing. // the routing table and has not finished writing.
writingTableChanges int writingTableChanges int
@@ -102,6 +89,15 @@ type PrefixHandler struct {
done chan struct{} // closed when the flush goroutine returns done chan struct{} // closed when the flush goroutine returns
} }
// seenPrefix is what the batch keeps of an update: the prefix it names and its
// time. A flush needs only these to store each new prefix's first-seen time.
type seenPrefix struct {
prefix string
timestamp time.Time
}
// prefixUpdate is one prefix of an update message, as applyToTable applies it
// to the routing table.
type prefixUpdate struct { type prefixUpdate struct {
prefix string prefix string
originASN int originASN int
@@ -130,24 +126,30 @@ func (u prefixUpdate) announcesRoute() bool {
func NewPrefixHandler( func NewPrefixHandler(
db database.Store, logger *logger.Logger, table *routetable.Table, db database.Store, logger *logger.Logger, table *routetable.Table,
) *PrefixHandler { ) *PrefixHandler {
return newPrefixHandler(db, logger, table, prefixBatchTimeout) return newPrefixHandler(db, logger, table, prefixBatchTimeout,
maxTableChangesPerWrite)
} }
// newPrefixHandler is NewPrefixHandler with the flush timer's interval, // newPrefixHandler is NewPrefixHandler with the flush timer's interval,
// batchTimeout, in place of prefixBatchTimeout. A test that the timer must not // batchTimeout, in place of prefixBatchTimeout, and tableChangesPerWrite in
// disturb gives an interval longer than the test runs. // place of maxTableChangesPerWrite. A test that the timer must not disturb
// gives an interval longer than the test runs. A test of writes split over
// several flushes gives a small tableChangesPerWrite, so that it exceeds it
// with few changes.
func newPrefixHandler( func newPrefixHandler(
db database.Store, logger *logger.Logger, table *routetable.Table, db database.Store, logger *logger.Logger, table *routetable.Table,
batchTimeout time.Duration, batchTimeout time.Duration, tableChangesPerWrite int,
) *PrefixHandler { ) *PrefixHandler {
h := &PrefixHandler{ h := &PrefixHandler{
db: db, db: db,
logger: logger, logger: logger,
table: table, table: table,
batch: make([]prefixUpdate, 0, prefixBatchSize), batch: make([]seenPrefix, 0, prefixBatchSize),
batchFullCh: make(chan struct{}, 1), batchMaxSize: prefixBatchMaxSize,
stopCh: make(chan context.Context, 1), tableChangesPerWrite: tableChangesPerWrite,
done: make(chan struct{}), batchFullCh: make(chan struct{}, 1),
stopCh: make(chan context.Context, 1),
done: make(chan struct{}),
} }
h.batchTaken.L = &h.mu h.batchTaken.L = &h.mu
@@ -181,7 +183,7 @@ func (h *PrefixHandler) QueueCapacity() int {
func (h *PrefixHandler) HandleMessage(msg *ristypes.RISMessage) { func (h *PrefixHandler) HandleMessage(msg *ristypes.RISMessage) {
if msg.Type == ristypes.MessageTypeState { if msg.Type == ristypes.MessageTypeState {
if msg.State == ristypes.StateDown { if msg.State == ristypes.StateDown {
h.peerSessionDown(msg.Peer) h.table.PeerDown(msg.Peer)
} }
return return
@@ -202,16 +204,15 @@ func (h *PrefixHandler) HandleMessage(msg *ristypes.RISMessage) {
// While the batch is at its maximum size, wait until the flush goroutine // While the batch is at its maximum size, wait until the flush goroutine
// takes it. Meanwhile the handler takes nothing off its queue, which // takes it. Meanwhile the handler takes nothing off its queue, which
// fills, and the streamer drops messages for it. // fills, and the streamer drops messages for it.
for len(h.batch) >= prefixBatchMaxSize { for len(h.batch) >= h.batchMaxSize {
h.batchTaken.Wait() h.batchTaken.Wait()
} }
// Withdrawals are queued, and applied to the routing table, before // Withdrawals are applied to the routing table before announcements, so
// announcements, because the batch keeps each peer's last update for a // a message that lists a prefix as both withdrawn and announced keeps the
// prefix: a message that lists a prefix as both withdrawn and announced // route (RFC 4271 section 4.3). A withdrawal names only the prefix and
// keeps the route (RFC 4271 section 4.3). A withdrawal names only the // peer: the message's AS path, and so its origin AS, belongs to the
// prefix and peer: the message's AS path, and so its origin AS, belongs // announced prefixes.
// to the announced prefixes.
for _, prefix := range msg.Withdrawals { for _, prefix := range msg.Withdrawals {
update := prefixUpdate{ update := prefixUpdate{
prefix: canonicalPrefix(prefix), prefix: canonicalPrefix(prefix),
@@ -219,7 +220,7 @@ func (h *PrefixHandler) HandleMessage(msg *ristypes.RISMessage) {
messageType: "withdrawal", messageType: "withdrawal",
timestamp: timestamp, timestamp: timestamp,
} }
h.batch = append(h.batch, update) h.batch = append(h.batch, seenPrefix{update.prefix, update.timestamp})
h.applyToTable(update) h.applyToTable(update)
// Record withdrawal in metrics // Record withdrawal in metrics
@@ -239,7 +240,7 @@ func (h *PrefixHandler) HandleMessage(msg *ristypes.RISMessage) {
path: path, path: path,
endsInSet: len(msg.Path.Set) > 0, endsInSet: len(msg.Path.Set) > 0,
} }
h.batch = append(h.batch, update) h.batch = append(h.batch, seenPrefix{update.prefix, update.timestamp})
h.applyToTable(update) h.applyToTable(update)
// Record announcement in metrics // Record announcement in metrics
@@ -251,7 +252,7 @@ func (h *PrefixHandler) HandleMessage(msg *ristypes.RISMessage) {
// A full batch wakes the flush goroutine. If it is writing, the wake-up // A full batch wakes the flush goroutine. If it is writing, the wake-up
// waits in batchFullCh and the batch keeps growing until the write ends // waits in batchFullCh and the batch keeps growing until the write ends
// or the batch reaches prefixBatchMaxSize. // or the batch reaches batchMaxSize.
if len(h.batch) >= prefixBatchSize { if len(h.batch) >= prefixBatchSize {
select { select {
case h.batchFullCh <- struct{}{}: case h.batchFullCh <- struct{}{}:
@@ -261,14 +262,12 @@ func (h *PrefixHandler) HandleMessage(msg *ristypes.RISMessage) {
} }
// Stop stops the flush goroutine, which makes one last flush before it returns, // Stop stops the flush goroutine, which makes one last flush before it returns,
// and waits for it until ctx ends. That flush writes the updates and peer downs // and waits for it until ctx ends. That flush writes at most
// the handler holds, and at most maxTableChangesPerWrite of the routing table's // tableChangesPerWrite of the routing table's changes, and the prefixes of
// changes. Once ctx has ended the flush goroutine skips that last flush, and a // the updates the handler holds. Once ctx has ended the flush goroutine skips
// write still running is lost when the process exits. A peer whose route delete // that last flush, and a write still running is lost when the process exits. A
// fails in the last flush is not retried either, nor is a failed write of the // failed write of the table's changes is not retried either. Stop logs how many
// table's changes. Stop logs how many updates are not written, how many peer // updates and how many of the table's changes are not written.
// downs have not had their routes deleted, and how many of the table's changes
// are not written.
func (h *PrefixHandler) Stop(ctx context.Context) { func (h *PrefixHandler) Stop(ctx context.Context) {
h.stopCh <- ctx h.stopCh <- ctx
@@ -278,176 +277,96 @@ func (h *PrefixHandler) Stop(ctx context.Context) {
} }
h.mu.Lock() h.mu.Lock()
unwritten := h.writing + len(h.batch) + len(h.held) unwritten := h.writing + len(h.batch)
peersDown := h.deleting + len(h.peersDown)
tableChanges := h.writingTableChanges + h.table.ChangeCount() tableChanges := h.writingTableChanges + h.table.ChangeCount()
h.mu.Unlock() h.mu.Unlock()
if unwritten == 0 && peersDown == 0 && tableChanges == 0 { if unwritten == 0 && tableChanges == 0 {
return return
} }
if ctx.Err() != nil { if ctx.Err() != nil {
h.logger.Error("Stop deadline passed before the prefix handler wrote "+ h.logger.Error("Stop deadline passed before the prefix handler wrote "+
"its updates", "updates", unwritten, "peers_down", peersDown, "its updates", "updates", unwritten, "table_changes", tableChanges)
"table_changes", tableChanges)
return return
} }
h.logger.Error("The prefix handler stopped before it wrote everything", h.logger.Error("The prefix handler stopped before it wrote everything",
"updates", unwritten, "peers_down", peersDown, "updates", unwritten, "table_changes", tableChanges)
"table_changes", tableChanges)
} }
// peerSessionDown notes that the peer's BGP session went down, so that the next // takeBatch returns the pending updates and leaves an empty batch in their
// flush deletes all of the peer's live routes before it writes the batch. The
// peer's updates still in the batch or held came before the down, which
// replaces them, so they are dropped; updates the peer sends later stay in the
// batch and are written after the delete. A peer already in peersDown is not
// added again: one delete removes all of its routes. In the routing table in
// memory the down takes effect at once.
func (h *PrefixHandler) peerSessionDown(peer string) {
h.mu.Lock()
defer h.mu.Unlock()
h.table.PeerDown(peer)
isPeers := func(update prefixUpdate) bool { return update.peer == peer }
h.batch = slices.DeleteFunc(h.batch, isPeers)
h.held = slices.DeleteFunc(h.held, isPeers)
if !slices.Contains(h.peersDown, peer) {
h.peersDown = append(h.peersDown, peer)
}
}
// takeBatch returns the pending updates, the held ones first, and the peers in
// peersDown, and leaves an empty batch, no held updates and no peers in their
// place for HandleMessage to fill. The updates count in writing until // place for HandleMessage to fill. The updates count in writing until
// flushBatch has written them, and the peers in deleting until their routes // flushBatch has written them.
// are deleted. func (h *PrefixHandler) takeBatch() []seenPrefix {
func (h *PrefixHandler) takeBatch() ([]prefixUpdate, []string) {
h.mu.Lock() h.mu.Lock()
defer h.mu.Unlock() defer h.mu.Unlock()
if len(h.held) == 0 && len(h.batch) == 0 && len(h.peersDown) == 0 { if len(h.batch) == 0 {
return nil, nil return nil
}
batch, peersDown := h.batch, h.peersDown
if len(h.held) > 0 {
batch = slices.Concat(h.held, h.batch)
} }
batch := h.batch
h.writing = len(batch) h.writing = len(batch)
h.deleting = len(peersDown) h.batch = make([]seenPrefix, 0, prefixBatchSize)
h.held = nil
h.batch = make([]prefixUpdate, 0, prefixBatchSize)
h.peersDown = nil
h.batchTaken.Broadcast() h.batchTaken.Broadcast()
return batch, peersDown return batch
} }
// flushBatch writes the routing table's changes, then takes the pending batch // flushBatch writes the routing table's changes, then takes the pending batch
// and writes it to the database, without holding the mutex during the writes. // and stores the prefixes it names, without holding the mutex during the
// writes.
func (h *PrefixHandler) flushBatch() { func (h *PrefixHandler) flushBatch() {
h.writeTableChanges() h.writeTableChanges()
batch, peersDown := h.takeBatch() batch := h.takeBatch()
if len(batch) == 0 && len(peersDown) == 0 { if len(batch) == 0 {
return return
} }
startTime := time.Now() startTime := time.Now()
batchSize := len(batch)
// The routes of a peer whose session went down are deleted first: the
// batch holds only the updates the peer sent after the down. A peer whose
// delete failed is kept, with those updates, for the next flush to retry.
failedPeers := h.deleteLiveRoutesOfPeers(peersDown)
if len(failedPeers) > 0 {
batch = h.keepForNextFlush(batch, failedPeers)
}
// For each prefix and peer, keep only the last update the feed delivered.
// Updates from different peers for the same prefix are all kept. Arrival
// order decides, not the timestamp: timestamps are whole seconds, so
// updates within one second carry the same one.
type prefixAndPeer struct{ prefix, peer string }
lastUpdates := make(map[prefixAndPeer]prefixUpdate)
// Each prefix the batch names, announced or withdrawn, with the earliest // Each prefix the batch names, announced or withdrawn, with the earliest
// timestamp among all of its updates, not only each peer's last one: the // timestamp among its updates: the prefix tables store only first_seen,
// prefix tables store only first_seen, written when the prefix is new and // written when the prefix is new and never changed after.
// never changed after.
prefixesToUpdate := make(map[string]time.Time) prefixesToUpdate := make(map[string]time.Time)
for _, update := range batch { for _, update := range batch {
lastUpdates[prefixAndPeer{update.prefix, update.peer}] = update
seen, exists := prefixesToUpdate[update.prefix] seen, exists := prefixesToUpdate[update.prefix]
if !exists || update.timestamp.Before(seen) { if !exists || update.timestamp.Before(seen) {
prefixesToUpdate[update.prefix] = update.timestamp prefixesToUpdate[update.prefix] = update.timestamp
} }
} }
// Collect routes to upsert and delete err := h.db.UpdatePrefixesBatch(prefixesToUpdate)
var ( if err != nil {
routesToUpsert []*database.LiveRoute h.logger.Error("Failed to update prefix batch",
routesToDelete []database.LiveRouteDeletion "error", err, "count", len(prefixesToUpdate))
)
for _, update := range lastUpdates {
if update.announcesRoute() {
// Create live route for batch upsert
route := h.createLiveRoute(update)
if route != nil {
routesToUpsert = append(routesToUpsert, route)
}
} else {
// Parse CIDR to get IP version
_, ipVersion, err := parseCIDR(update.prefix)
if err != nil {
h.logger.Error("Failed to parse CIDR for route deletion",
"prefix", update.prefix, "error", err)
continue
}
// Create deletion record for batch delete
routesToDelete = append(routesToDelete, database.LiveRouteDeletion{
Prefix: update.prefix,
PeerIP: update.peer,
IPVersion: ipVersion,
})
}
} }
successCount := h.writeBatch(routesToUpsert, routesToDelete, prefixesToUpdate)
h.mu.Lock() h.mu.Lock()
h.writing = 0 h.writing = 0
h.mu.Unlock() h.mu.Unlock()
elapsed := time.Since(startTime)
h.logger.Debug("Flushed prefix batch", h.logger.Debug("Flushed prefix batch",
"batch_size", batchSize, "batch_size", len(batch),
"peers_down", len(peersDown),
"unique_prefixes", len(prefixesToUpdate), "unique_prefixes", len(prefixesToUpdate),
"success", successCount, "duration_ms", time.Since(startTime).Milliseconds(),
"duration_ms", elapsed.Milliseconds(),
) )
} }
// applyToTable applies update to the routing table in memory: as an // applyToTable applies update to the routing table in memory: as an
// announcement when it gives the peer a route, otherwise as a withdrawal. A // announcement when it gives the peer a route, otherwise as a withdrawal. A
// prefix that does not parse is left out; flushBatch logs it. // prefix that does not parse is logged and left out. Each announcement counts
// in the IPv4 or IPv6 route update rate.
func (h *PrefixHandler) applyToTable(update prefixUpdate) { func (h *PrefixHandler) applyToTable(update prefixUpdate) {
prefix, err := netip.ParsePrefix(update.prefix) prefix, err := netip.ParsePrefix(update.prefix)
if err != nil { if err != nil {
h.logger.Error("Failed to parse prefix",
"prefix", update.prefix, "error", err)
return return
} }
@@ -457,6 +376,14 @@ func (h *PrefixHandler) applyToTable(update prefixUpdate) {
return return
} }
if h.metrics != nil {
if prefix.Addr().Is4() {
h.metrics.RecordIPv4Update()
} else {
h.metrics.RecordIPv6Update()
}
}
h.table.Announce(prefix, update.originASN, routetable.Route{ h.table.Announce(prefix, update.originASN, routetable.Route{
Peer: update.peer, Peer: update.peer,
ASPath: update.path, ASPath: update.path,
@@ -464,14 +391,13 @@ func (h *PrefixHandler) applyToTable(update prefixUpdate) {
}) })
} }
// writeTableChanges writes at most maxTableChangesPerWrite changes of the // writeTableChanges writes at most tableChangesPerWrite changes of the routing
// routing table in memory, and every peer the table numbered since the last // table in memory, and every peer the table numbered since the last write.
// write.
// When the write fails, they go back to the table, so that a later write writes // When the write fails, they go back to the table, so that a later write writes
// them again, each pair with its content then. // them again, each pair with its content then.
func (h *PrefixHandler) writeTableChanges() { func (h *PrefixHandler) writeTableChanges() {
h.mu.Lock() h.mu.Lock()
changes, newPeers := h.table.TakeChanges(maxTableChangesPerWrite) changes, newPeers := h.table.TakeChanges(h.tableChangesPerWrite)
h.writingTableChanges = len(changes) h.writingTableChanges = len(changes)
h.mu.Unlock() h.mu.Unlock()
@@ -495,147 +421,11 @@ func (h *PrefixHandler) writeTableChanges() {
} }
} }
// deleteLiveRoutesOfPeers deletes all live routes of each of peers, the peers
// whose BGP session went down, and returns those whose delete failed. Most
// downs come from peers that have no routes, so only a delete that removed
// routes is logged. A peer whose delete succeeded no longer counts in deleting;
// a failed one counts until keepForNextFlush puts it back in peersDown.
func (h *PrefixHandler) deleteLiveRoutesOfPeers(peers []string) []string {
var failed []string
for _, peer := range peers {
deleted, err := h.db.DeleteLiveRoutesOfPeer(peer)
if err != nil {
h.logger.Error(
"Failed to delete the live routes of a peer whose session "+
"went down; the next flush retries",
"peer", peer, "routes_deleted", deleted, "error", err)
failed = append(failed, peer)
continue
}
h.mu.Lock()
h.deleting--
h.mu.Unlock()
if deleted > 0 {
h.logger.Info(
"Deleted the live routes of a peer whose session went down",
"peer", peer, "routes", deleted)
}
}
return failed
}
// keepForNextFlush puts the peers whose route delete failed back in peersDown,
// and their updates from batch in held, so that the next flush retries each
// delete before it writes the peer's updates; written now, they would be
// deleted by the retry. The updates of a peer whose session went down again
// since the batch was taken came before that down, so they are dropped, as
// peerSessionDown drops them, and the peer is already in peersDown. It returns
// the updates of the other peers, for this flush to write.
//
// While a delete keeps failing, the held updates would grow with every flush,
// so only the last prefixBatchSize of them are kept and the older ones are
// dropped. This never leaves in place a route the peer withdrew: a prefix with
// a kept update keeps its last one, and the retry deletes all of the peer's
// routes, so a prefix whose updates were all dropped has no route from the peer
// until the peer announces it again.
func (h *PrefixHandler) keepForNextFlush(
batch []prefixUpdate, failedPeers []string,
) []prefixUpdate {
var kept, toWrite []prefixUpdate
for _, update := range batch {
if slices.Contains(failedPeers, update.peer) {
kept = append(kept, update)
} else {
toWrite = append(toWrite, update)
}
}
h.mu.Lock()
defer h.mu.Unlock()
kept = slices.DeleteFunc(kept, func(update prefixUpdate) bool {
return slices.Contains(h.peersDown, update.peer)
})
if len(kept) > prefixBatchSize {
h.logger.Error(
"Dropped the oldest updates of peers whose route delete failed",
"peers", failedPeers, "dropped", len(kept)-prefixBatchSize)
// A copy, so that the dropped updates are freed.
kept = slices.Clone(kept[len(kept)-prefixBatchSize:])
}
// takeBatch left held empty, and only this flush fills it. The kept
// updates now count in held, not in writing, and the failed peers in
// peersDown, not in deleting.
h.held = kept
h.writing = len(toWrite)
h.deleting = 0
for _, peer := range failedPeers {
if !slices.Contains(h.peersDown, peer) {
h.peersDown = append(h.peersDown, peer)
}
}
return toWrite
}
// writeBatch writes the collected route upserts, route deletions and prefix
// updates to the database and returns how many routes it wrote.
func (h *PrefixHandler) writeBatch(
routesToUpsert []*database.LiveRoute,
routesToDelete []database.LiveRouteDeletion,
prefixesToUpdate map[string]time.Time,
) int {
successCount := 0
if len(routesToUpsert) > 0 {
err := h.db.UpsertLiveRouteBatch(routesToUpsert)
if err != nil {
h.logger.Error("Failed to upsert route batch",
"error", err, "count", len(routesToUpsert))
} else {
successCount += len(routesToUpsert)
}
}
if len(routesToDelete) > 0 {
err := h.db.DeleteLiveRouteBatch(routesToDelete)
if err != nil {
h.logger.Error("Failed to delete route batch",
"error", err, "count", len(routesToDelete))
} else {
successCount += len(routesToDelete)
}
}
// Update prefix tables
if len(prefixesToUpdate) > 0 {
err := h.db.UpdatePrefixesBatch(prefixesToUpdate)
if err != nil {
h.logger.Error("Failed to update prefix batch",
"error", err, "count", len(prefixesToUpdate))
}
}
return successCount
}
// canonicalPrefix returns prefix in the text form net/netip prints, the form // canonicalPrefix returns prefix in the text form net/netip prints, the form
// the IP lookup builds when it looks a prefix up. The feed sends IPv6 // the prefix tables store and the IP lookup reads them in. The feed sends IPv6
// announcements compressed ("2001:db8::/32") but IPv6 withdrawals uncompressed // announcements compressed ("2001:db8::/32") but IPv6 withdrawals uncompressed
// ("2001:db8:0:0:0:0:0:0/32"); stored as received, a withdrawal would not match // ("2001:db8:0:0:0:0:0:0/32"). A prefix that does not parse is returned
// the route its announcement stored. A prefix that does not parse is returned // unchanged, and applyToTable logs it.
// unchanged, and the batch flush reports it.
func canonicalPrefix(prefix string) string { func canonicalPrefix(prefix string) string {
p, err := netip.ParsePrefix(prefix) p, err := netip.ParsePrefix(prefix)
if err != nil { if err != nil {
@@ -644,55 +434,3 @@ func canonicalPrefix(prefix string) string {
return p.Masked().String() return p.Masked().String()
} }
// parseCIDR returns the mask length and the IP version of a prefix string
func parseCIDR(prefix string) (int, int, error) {
_, ipNet, err := net.ParseCIDR(prefix)
if err != nil {
return 0, 0, err
}
ones, _ := ipNet.Mask.Size()
if strings.Contains(prefix, ":") {
return ones, ipv6Version, nil
}
return ones, ipv4Version, nil
}
// createLiveRoute creates a LiveRoute from a prefix update
func (h *PrefixHandler) createLiveRoute(update prefixUpdate) *database.LiveRoute {
// Parse CIDR to get mask length
maskLength, ipVersion, err := parseCIDR(update.prefix)
if err != nil {
h.logger.Error("Failed to parse CIDR",
"prefix", update.prefix,
"error", err,
)
return nil
}
// Track route update metrics
if h.metrics != nil {
if ipVersion == ipv4Version {
h.metrics.RecordIPv4Update()
} else {
h.metrics.RecordIPv6Update()
}
}
// Create live route record
liveRoute := &database.LiveRoute{
Prefix: update.prefix,
MaskLength: maskLength,
IPVersion: ipVersion,
OriginASN: update.originASN,
PeerIP: update.peer,
ASPath: update.path,
NextHop: update.peer, // Using peer as next hop
LastUpdated: update.timestamp,
}
return liveRoute
}
internal/routewatch/prefixhandler_internal_test.go
+54 -921
View File
File diff suppressed because it is too large Load Diff
internal/routewatch/prefixhandler_table_internal_test.go
+104 -18
View File
@@ -8,6 +8,7 @@ import (
"net/netip" "net/netip"
"reflect" "reflect"
"slices" "slices"
"strings"
"testing" "testing"
"time" "time"
@@ -29,11 +30,12 @@ const (
// firstManyPrefix is the first prefix announceManyPrefixes announces. // firstManyPrefix is the first prefix announceManyPrefixes announces.
firstManyPrefix = "10.0.0.0/32" firstManyPrefix = "10.0.0.0/32"
)
// tableWriteTimeout is how long a test waits for the flush timer to write the // testMaxTableChangesPerWrite is a limit of one write of the routing
// routing table's changes: a few ticks of it. // table's changes that a test exceeds quickly, in place of
const tableWriteTimeout = 5 * prefixBatchTimeout // maxTableChangesPerWrite.
testMaxTableChangesPerWrite = 10
)
// tableCase is one test of the routing table in memory: the messages handed to // tableCase is one test of the routing table in memory: the messages handed to
// the prefix handler, and for each prefix the peers that carry it with each // the prefix handler, and for each prefix the peers that carry it with each
@@ -84,13 +86,25 @@ func TestPrefixHandlerAppliesUpdatesToTheRoutingTable(t *testing.T) {
testV6Prefix: {}, testV6Prefix: {},
}, },
}, },
{
name: "an announcement after the peer's session went down",
messages: []*ristypes.RISMessage{
peerAnnouncement(testPeerIP, testV4Prefix),
sessionDown(),
peerAnnouncement(testPeerIP, otherV4Prefix),
},
want: map[string]map[int][]string{
testV4Prefix: {},
otherV4Prefix: {testASNB: {testPeerIP}},
},
},
}) })
} }
// TestPrefixHandlerAppliesAPathWithoutAnOriginToTheRoutingTable hands the // TestPrefixHandlerAppliesAPathWithoutAnOriginToTheRoutingTable hands the
// prefix handler announcements whose AS path gives no origin. One whose path // prefix handler announcements whose AS path gives no origin. One whose path
// holds AS 0 removes the peer's route, as a withdrawal does; one whose path ends // is empty or holds AS 0 removes the peer's route, as a withdrawal does; one
// in an AS set of several members is kept with origin 0. // whose path ends in an AS set of several members is kept with origin 0.
func TestPrefixHandlerAppliesAPathWithoutAnOriginToTheRoutingTable(t *testing.T) { func TestPrefixHandlerAppliesAPathWithoutAnOriginToTheRoutingTable(t *testing.T) {
t.Parallel() t.Parallel()
@@ -110,6 +124,14 @@ func TestPrefixHandlerAppliesAPathWithoutAnOriginToTheRoutingTable(t *testing.T)
}, },
want: map[string]map[int][]string{testV4Prefix: {}}, want: map[string]map[int][]string{testV4Prefix: {}},
}, },
{
name: "an empty path",
messages: []*ristypes.RISMessage{
peerAnnouncement(testPeerIP, testV4Prefix),
announceWithPath(ristypes.ASPath{}),
},
want: map[string]map[int][]string{testV4Prefix: {}},
},
{ {
name: "a path with AS 0 before its end", name: "a path with AS 0 before its end",
messages: []*ristypes.RISMessage{ messages: []*ristypes.RISMessage{
@@ -128,6 +150,55 @@ func TestPrefixHandlerAppliesAPathWithoutAnOriginToTheRoutingTable(t *testing.T)
}) })
} }
// TestPrefixHandlerKeepsThePathOfAPathEndingInASet announces a prefix with AS
// paths that end in an AS set. Any member of a set of several may be the
// origin, so the routing table keeps the route with origin 0 (none); a set of
// one member counts as that AS. The kept AS path lists the set's members after
// the rest of the path.
func TestPrefixHandlerKeepsThePathOfAPathEndingInASet(t *testing.T) {
t.Parallel()
tests := []struct {
name string
path []any
wantOrigin int
wantPath []int
}{
{
name: "a set of several members",
path: []any{testASNA, testASNB, []int{testASNC, testASND}},
wantPath: []int{testASNA, testASNB, testASNC, testASND},
},
{
name: "a set of one member",
path: []any{testASNA, testASNB, []int{testASNC}},
wantOrigin: testASNC,
wantPath: []int{testASNA, testASNB, testASNC},
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
t.Parallel()
msg := peerAnnouncement(testPeerIP, testV4Prefix)
msg.Path = decodePath(t, tt.path...)
h := prefixHandlerWithoutTimer(newMockStore())
h.HandleMessage(msg)
origins := h.table.Origins(netip.MustParsePrefix(testV4Prefix))
if len(origins) != 1 || origins[0].ASN != tt.wantOrigin ||
len(origins[0].Routes) != 1 ||
!slices.Equal(origins[0].Routes[0].ASPath, tt.wantPath) {
t.Errorf("Origins(%s) = %+v, want one route with origin AS%d "+
"and AS path %v", testV4Prefix, origins, tt.wantOrigin,
tt.wantPath)
}
})
}
}
// checkTableCases hands the messages of each case to a new prefix handler and // checkTableCases hands the messages of each case to a new prefix handler and
// checks its routing table right after, with no flush. // checks its routing table right after, with no flush.
func checkTableCases(t *testing.T, tests []tableCase) { func checkTableCases(t *testing.T, tests []tableCase) {
@@ -206,9 +277,9 @@ func TestPrefixHandlerWritesTheTableChangesEachSecond(t *testing.T) {
t.Fatalf("wrote %+v, want the pair of %s and AS%d with "+ t.Fatalf("wrote %+v, want the pair of %s and AS%d with "+
"removed %t", changes, prefix, testASNB, step.wantRemoved) "removed %t", changes, prefix, testASNB, step.wantRemoved)
} }
case <-time.After(tableWriteTimeout): case <-time.After(stepTimeout):
t.Fatalf("the routing table's change was not written within %s", t.Fatalf("the routing table's change was not written within %s",
tableWriteTimeout) stepTimeout)
} }
} }
} }
@@ -305,25 +376,32 @@ func (s *tableWriteSizesStore) WriteRouteChanges(
// takes, then flushes three times. The first write must hold as many pairs as // takes, then flushes three times. The first write must hold as many pairs as
// one write takes, and the peer. The second must hold the other pairs, and the // one write takes, and the peer. The second must hold the other pairs, and the
// third flush must write nothing. The database must then hold the table in // third flush must write nothing. The database must then hold the table in
// memory. // memory. The handler's limit is small, and its timer does not fire.
func TestPrefixHandlerWritesManyTableChangesOverSeveralFlushes(t *testing.T) { func TestPrefixHandlerWritesManyTableChangesOverSeveralFlushes(t *testing.T) {
t.Parallel() t.Parallel()
const ( const (
pairs = maxTableChangesPerWrite * 3 / 2 pairs = testMaxTableChangesPerWrite * 3 / 2
flushes = 3 flushes = 3
) )
db := newTestDatabase(t) db := newTestDatabase(t)
store := &tableWriteSizesStore{Database: db} store := &tableWriteSizesStore{Database: db}
h := prefixHandlerWithoutTimer(store) table := routetable.New(routetable.Params{})
announceManyPrefixes(h.table, pairs) announceManyPrefixes(table, pairs)
h := newPrefixHandler(store, logger.New(), table, longerThanAnyTest,
testMaxTableChangesPerWrite)
defer stopInTime(t, h)
for range flushes { for range flushes {
h.flushBatch() h.flushBatch()
} }
wantChanges := []int{maxTableChangesPerWrite, pairs - maxTableChangesPerWrite} wantChanges := []int{
testMaxTableChangesPerWrite, pairs - testMaxTableChangesPerWrite,
}
wantNewPeers := []int{1, 0} wantNewPeers := []int{1, 0}
if !slices.Equal(store.changes, wantChanges) || if !slices.Equal(store.changes, wantChanges) ||
@@ -343,12 +421,13 @@ func TestPrefixHandlerWritesManyTableChangesOverSeveralFlushes(t *testing.T) {
// TestPrefixHandlerStopLogsTheTableChangesLeftUnwritten stops a prefix handler, // TestPrefixHandlerStopLogsTheTableChangesLeftUnwritten stops a prefix handler,
// with no deadline, whose routing table holds more changed pairs than one write // with no deadline, whose routing table holds more changed pairs than one write
// takes. Its last flush writes one write's worth, so Stop must log the pairs // takes. Its last flush writes one write's worth, so Stop must log the pairs
// that no write took. The flush timer must not fire, or its writes could leave // that no write took, and must not say that the stop deadline passed. The
// no pairs for Stop to log. // flush timer must not fire, or its writes could leave no pairs for Stop to
// log. The handler's limit of one write is small.
func TestPrefixHandlerStopLogsTheTableChangesLeftUnwritten(t *testing.T) { func TestPrefixHandlerStopLogsTheTableChangesLeftUnwritten(t *testing.T) {
t.Parallel() t.Parallel()
const pairs = maxTableChangesPerWrite + 1 const pairs = testMaxTableChangesPerWrite + 1
var logs bytes.Buffer var logs bytes.Buffer
@@ -356,7 +435,8 @@ func TestPrefixHandlerStopLogsTheTableChangesLeftUnwritten(t *testing.T) {
table := routetable.New(routetable.Params{}) table := routetable.New(routetable.Params{})
announceManyPrefixes(table, pairs) announceManyPrefixes(table, pairs)
h := newPrefixHandler(store, newErrorLogger(&logs), table, longerThanAnyTest) h := newPrefixHandler(store, newErrorLogger(&logs), table, longerThanAnyTest,
testMaxTableChangesPerWrite)
h.Stop(t.Context()) h.Stop(t.Context())
written := 0 written := 0
@@ -364,7 +444,13 @@ func TestPrefixHandlerStopLogsTheTableChangesLeftUnwritten(t *testing.T) {
written += n written += n
} }
if logged := lastLogged(t, &logs); logged.TableChanges != pairs-written { logged := lastLogged(t, &logs)
if strings.Contains(logged.Msg, "deadline") {
t.Errorf("Stop logged %q although its deadline had not passed",
logged.Msg)
}
if logged.TableChanges != pairs-written {
t.Errorf("Stop logged %d table changes not written, want %d", t.Errorf("Stop logged %d table changes not written, want %d",
logged.TableChanges, pairs-written) logged.TableChanges, pairs-written)
} }
internal/server/handlers.go
+187 -162
View File
@@ -23,7 +23,6 @@ import (
"git.eeqj.de/sneak/routewatch/internal/streamer" "git.eeqj.de/sneak/routewatch/internal/streamer"
"git.eeqj.de/sneak/routewatch/internal/templates" "git.eeqj.de/sneak/routewatch/internal/templates"
asinfo "git.eeqj.de/sneak/routewatch/pkg/asinfo" asinfo "git.eeqj.de/sneak/routewatch/pkg/asinfo"
"github.com/dustin/go-humanize"
"github.com/go-chi/chi/v5" "github.com/go-chi/chi/v5"
) )
@@ -83,7 +82,7 @@ const (
maxASSearchResults = 50 maxASSearchResults = 50
) )
// IP versions as database.LiveRoute.IPVersion records them. // IP versions as LiveRoute.IPVersion records them.
const ( const (
ipVersionV4 = 4 ipVersionV4 = 4
ipVersionV6 = 6 ipVersionV6 = 6
@@ -160,6 +159,7 @@ func (s *Server) handleIndex() http.HandlerFunc {
err := tmpl.Execute(w, indexPageData{ err := tmpl.Execute(w, indexPageData{
CSPNonce: cspNonce(r.Context()), CSPNonce: cspNonce(r.Context()),
ClientIP: extractClientIP(r), ClientIP: extractClientIP(r),
Cards: s.cards.Load(),
}) })
if err != nil { if err != nil {
s.logger.Error("Failed to render index template", "error", err) s.logger.Error("Failed to render index template", "error", err)
@@ -181,6 +181,9 @@ type indexPageData struct {
// ClientIP is the visitor's address, which the IP information API card // ClientIP is the visitor's address, which the IP information API card
// looks up: the address /ip/ would look up for the same request. // looks up: the address /ip/ would look up for the same request.
ClientIP string ClientIP string
// Cards is the copy of the cards that show figures of the routing table,
// or nil before the server has first worked them out.
Cards *homeCards
} }
// writeJSONError writes a standardized JSON error response with the given // writeJSONError writes a standardized JSON error response with the given
@@ -222,45 +225,50 @@ type WHOISStatsInfo struct {
NeverFetched int `json:"never_fetched"` NeverFetched int `json:"never_fetched"`
SuccessesLastHour int `json:"successes_last_hour"` SuccessesLastHour int `json:"successes_last_hour"`
ErrorsLastHour int `json:"errors_last_hour"` ErrorsLastHour int `json:"errors_last_hour"`
CurrentInterval string `json:"current_interval"`
ConsecutiveFails int `json:"consecutive_fails"` ConsecutiveFails int `json:"consecutive_fails"`
FreshPercent float64 `json:"fresh_percent"` FreshPercent float64 `json:"fresh_percent"`
// CurrentIntervalSeconds is the wait between two fetches. It can have a
// fractional part, such as 22.5.
CurrentIntervalSeconds float64 `json:"current_interval_seconds"`
} }
// Stats is the /status.json answer: uptime, message counts, database stats, // Stats is the /status.json answer: uptime, message counts, database stats,
// and the routing table's figures. OldestRoute and NewestRoute are the times // and the routing table's figures. UptimeSeconds is the time since the stream
// of the oldest and the newest kept routes. // last connected, or 0 if it has never connected. MemoryUsageBytes is the Go
// heap in use. OldestRoute and NewestRoute are the times of the oldest and the
// newest kept routes.
// //
//nolint:tagliatelle // field names are the existing JSON API //nolint:tagliatelle // field names are the existing JSON API
type Stats struct { type Stats struct {
Uptime string `json:"uptime"` UptimeSeconds int64 `json:"uptime_seconds"`
TotalMessages uint64 `json:"total_messages"` TotalMessages uint64 `json:"total_messages"`
TotalBytes uint64 `json:"total_bytes"` TotalBytes uint64 `json:"total_bytes"`
TotalWireBytes uint64 `json:"total_wire_bytes"` TotalWireBytes uint64 `json:"total_wire_bytes"`
MessagesPerSec float64 `json:"messages_per_sec"` MessagesPerSec float64 `json:"messages_per_sec"`
MbitsPerSec float64 `json:"mbits_per_sec"` MbitsPerSec float64 `json:"mbits_per_sec"`
WireMbitsPerSec float64 `json:"wire_mbits_per_sec"` WireMbitsPerSec float64 `json:"wire_mbits_per_sec"`
Connected bool `json:"connected"` Connected bool `json:"connected"`
GoVersion string `json:"go_version"` GoVersion string `json:"go_version"`
Goroutines int `json:"goroutines"` Goroutines int `json:"goroutines"`
MemoryUsage string `json:"memory_usage"` MemoryUsageBytes uint64 `json:"memory_usage_bytes"`
ASNs int `json:"asns"` ASNs int `json:"asns"`
Prefixes int `json:"prefixes"` Prefixes int `json:"prefixes"`
IPv4Prefixes int `json:"ipv4_prefixes"` IPv4Prefixes int `json:"ipv4_prefixes"`
IPv6Prefixes int `json:"ipv6_prefixes"` IPv6Prefixes int `json:"ipv6_prefixes"`
Peerings int `json:"peerings"` Peerings int `json:"peerings"`
Peers int `json:"peers"` Peers int `json:"peers"`
DatabaseSizeBytes int64 `json:"database_size_bytes"` DatabaseSizeBytes int64 `json:"database_size_bytes"`
LiveRoutes int `json:"live_routes"` LiveRoutes int `json:"live_routes"`
IPv4Routes int `json:"ipv4_routes"` IPv4Routes int `json:"ipv4_routes"`
IPv6Routes int `json:"ipv6_routes"` IPv6Routes int `json:"ipv6_routes"`
OldestRoute *time.Time `json:"oldest_route,omitempty"` OldestRoute *time.Time `json:"oldest_route,omitempty"`
NewestRoute *time.Time `json:"newest_route,omitempty"` NewestRoute *time.Time `json:"newest_route,omitempty"`
IPv4UpdatesPerSec float64 `json:"ipv4_updates_per_sec"` IPv4UpdatesPerSec float64 `json:"ipv4_updates_per_sec"`
IPv6UpdatesPerSec float64 `json:"ipv6_updates_per_sec"` IPv6UpdatesPerSec float64 `json:"ipv6_updates_per_sec"`
IPv4PrefixDistribution []database.PrefixDistribution `json:"ipv4_prefix_distribution"` IPv4PrefixDistribution []PrefixDistribution `json:"ipv4_prefix_distribution"`
IPv6PrefixDistribution []database.PrefixDistribution `json:"ipv6_prefix_distribution"` IPv6PrefixDistribution []PrefixDistribution `json:"ipv6_prefix_distribution"`
WHOISStats *WHOISStatsInfo `json:"whois_stats,omitempty"` WHOISStats *WHOISStatsInfo `json:"whois_stats,omitempty"`
// OriginsWithoutKeptRoute counts the prefix and origin AS pairs that peers // OriginsWithoutKeptRoute counts the prefix and origin AS pairs that peers
// carry but whose kept routes were all withdrawn. // carry but whose kept routes were all withdrawn.
@@ -358,7 +366,7 @@ func (s *Server) statusStats(
runtime.ReadMemStats(&memStats) runtime.ReadMemStats(&memStats)
return Stats{ return Stats{
Uptime: streamUptime(streamMetrics.ConnectedSince), UptimeSeconds: uptimeSeconds(streamMetrics.ConnectedSince),
TotalMessages: streamMetrics.TotalMessages, TotalMessages: streamMetrics.TotalMessages,
TotalBytes: streamMetrics.TotalBytes, TotalBytes: streamMetrics.TotalBytes,
TotalWireBytes: streamMetrics.TotalWireBytes, TotalWireBytes: streamMetrics.TotalWireBytes,
@@ -368,7 +376,7 @@ func (s *Server) statusStats(
Connected: streamMetrics.Connected, Connected: streamMetrics.Connected,
GoVersion: runtime.Version(), GoVersion: runtime.Version(),
Goroutines: runtime.NumGoroutine(), Goroutines: runtime.NumGoroutine(),
MemoryUsage: humanize.Bytes(memStats.Alloc), MemoryUsageBytes: memStats.Alloc,
ASNs: dbStats.ASNs, ASNs: dbStats.ASNs,
Prefixes: counts.IPv4Prefixes + counts.IPv6Prefixes, Prefixes: counts.IPv4Prefixes + counts.IPv6Prefixes,
IPv4Prefixes: counts.IPv4Prefixes, IPv4Prefixes: counts.IPv4Prefixes,
@@ -394,12 +402,12 @@ func (s *Server) statusStats(
// prefixDistribution lists the mask lengths that have at least one prefix, in // prefixDistribution lists the mask lengths that have at least one prefix, in
// ascending order, with their prefix counts. countsByLength[n] is the number // ascending order, with their prefix counts. countsByLength[n] is the number
// of prefixes of mask length n. // of prefixes of mask length n.
func prefixDistribution(countsByLength []int) []database.PrefixDistribution { func prefixDistribution(countsByLength []int) []PrefixDistribution {
var distribution []database.PrefixDistribution var distribution []PrefixDistribution
for maskLength, count := range countsByLength { for maskLength, count := range countsByLength {
if count > 0 { if count > 0 {
distribution = append(distribution, database.PrefixDistribution{ distribution = append(distribution, PrefixDistribution{
MaskLength: maskLength, Count: count, MaskLength: maskLength, Count: count,
}) })
} }
@@ -435,30 +443,31 @@ func (s *Server) getWHOISStats(ctx context.Context) *WHOISStatsInfo {
NeverFetched: dbStats.NeverFetched, NeverFetched: dbStats.NeverFetched,
SuccessesLastHour: fetcherStats.SuccessesLastHour, SuccessesLastHour: fetcherStats.SuccessesLastHour,
ErrorsLastHour: fetcherStats.ErrorsLastHour, ErrorsLastHour: fetcherStats.ErrorsLastHour,
CurrentInterval: fetcherStats.CurrentInterval.String(),
ConsecutiveFails: fetcherStats.ConsecutiveFails, ConsecutiveFails: fetcherStats.ConsecutiveFails,
FreshPercent: freshPercent, FreshPercent: freshPercent,
CurrentIntervalSeconds: fetcherStats.CurrentInterval.Seconds(),
} }
} }
// streamUptime is how long the stream has been connected, to the second, or // uptimeSeconds is the number of whole seconds since the stream last
// "0s" if it has never connected. // connected, or 0 if it has never connected.
func streamUptime(connectedSince time.Time) string { func uptimeSeconds(connectedSince time.Time) int64 {
if connectedSince.IsZero() { if connectedSince.IsZero() {
return "0s" return 0
} }
return time.Since(connectedSince).Truncate(time.Second).String() return int64(time.Since(connectedSince) / time.Second)
} }
// connectionDuration is how long the stream has been connected, to the // connectionDurationSeconds is the number of whole seconds the stream has
// second, or "disconnected". // been connected, or nil while it is not connected.
func connectionDuration(connected bool, connectedSince time.Time) string { func connectionDurationSeconds(connected bool, connectedSince time.Time) *int64 {
if !connected || connectedSince.IsZero() { if !connected || connectedSince.IsZero() {
return "disconnected" return nil
} }
return time.Since(connectedSince).Truncate(time.Second).String() return new(uptimeSeconds(connectedSince))
} }
// whoisStaleThreshold matches the fetcher's threshold for consistency. // whoisStaleThreshold matches the fetcher's threshold for consistency.
@@ -508,39 +517,41 @@ type StreamStats struct {
// //
//nolint:tagliatelle // field names are the existing JSON API //nolint:tagliatelle // field names are the existing JSON API
type StatsResponse struct { type StatsResponse struct {
Uptime string `json:"uptime"` UptimeSeconds int64 `json:"uptime_seconds"`
TotalMessages uint64 `json:"total_messages"` TotalMessages uint64 `json:"total_messages"`
TotalBytes uint64 `json:"total_bytes"` TotalBytes uint64 `json:"total_bytes"`
TotalWireBytes uint64 `json:"total_wire_bytes"` TotalWireBytes uint64 `json:"total_wire_bytes"`
MessagesPerSec float64 `json:"messages_per_sec"` MessagesPerSec float64 `json:"messages_per_sec"`
MbitsPerSec float64 `json:"mbits_per_sec"` MbitsPerSec float64 `json:"mbits_per_sec"`
WireMbitsPerSec float64 `json:"wire_mbits_per_sec"` WireMbitsPerSec float64 `json:"wire_mbits_per_sec"`
Connected bool `json:"connected"` Connected bool `json:"connected"`
ConnectionDuration string `json:"connection_duration"` ReconnectCount uint64 `json:"reconnect_count"`
ReconnectCount uint64 `json:"reconnect_count"` GoVersion string `json:"go_version"`
GoVersion string `json:"go_version"` Goroutines int `json:"goroutines"`
Goroutines int `json:"goroutines"` MemoryUsageBytes uint64 `json:"memory_usage_bytes"`
MemoryUsage string `json:"memory_usage"` GC GCStats `json:"gc"`
GC GCStats `json:"gc"` Stream StreamStats `json:"stream"`
Stream StreamStats `json:"stream"` ASNs int `json:"asns"`
ASNs int `json:"asns"` Prefixes int `json:"prefixes"`
Prefixes int `json:"prefixes"` IPv4Prefixes int `json:"ipv4_prefixes"`
IPv4Prefixes int `json:"ipv4_prefixes"` IPv6Prefixes int `json:"ipv6_prefixes"`
IPv6Prefixes int `json:"ipv6_prefixes"` Peerings int `json:"peerings"`
Peerings int `json:"peerings"` Peers int `json:"peers"`
Peers int `json:"peers"` DatabaseSizeBytes int64 `json:"database_size_bytes"`
DatabaseSizeBytes int64 `json:"database_size_bytes"` LiveRoutes int `json:"live_routes"`
LiveRoutes int `json:"live_routes"` IPv4Routes int `json:"ipv4_routes"`
IPv4Routes int `json:"ipv4_routes"` IPv6Routes int `json:"ipv6_routes"`
IPv6Routes int `json:"ipv6_routes"` OldestRoute *time.Time `json:"oldest_route,omitempty"`
OldestRoute *time.Time `json:"oldest_route,omitempty"` NewestRoute *time.Time `json:"newest_route,omitempty"`
NewestRoute *time.Time `json:"newest_route,omitempty"` IPv4UpdatesPerSec float64 `json:"ipv4_updates_per_sec"`
IPv4UpdatesPerSec float64 `json:"ipv4_updates_per_sec"` IPv6UpdatesPerSec float64 `json:"ipv6_updates_per_sec"`
IPv6UpdatesPerSec float64 `json:"ipv6_updates_per_sec"` HandlerStats []HandlerStatsInfo `json:"handler_stats"`
HandlerStats []HandlerStatsInfo `json:"handler_stats"` IPv4PrefixDistribution []PrefixDistribution `json:"ipv4_prefix_distribution"`
IPv4PrefixDistribution []database.PrefixDistribution `json:"ipv4_prefix_distribution"` IPv6PrefixDistribution []PrefixDistribution `json:"ipv6_prefix_distribution"`
IPv6PrefixDistribution []database.PrefixDistribution `json:"ipv6_prefix_distribution"` WHOISStats *WHOISStatsInfo `json:"whois_stats,omitempty"`
WHOISStats *WHOISStatsInfo `json:"whois_stats,omitempty"`
// ConnectionDurationSeconds is null while the stream is not connected.
ConnectionDurationSeconds *int64 `json:"connection_duration_seconds"`
OriginsWithoutKeptRoute int `json:"origins_without_kept_route"` OriginsWithoutKeptRoute int `json:"origins_without_kept_route"`
} }
@@ -680,20 +691,18 @@ func (s *Server) statsResponse(
runtime.ReadMemStats(&memStats) runtime.ReadMemStats(&memStats)
return StatsResponse{ return StatsResponse{
Uptime: streamUptime(streamMetrics.ConnectedSince), UptimeSeconds: uptimeSeconds(streamMetrics.ConnectedSince),
TotalMessages: streamMetrics.TotalMessages, TotalMessages: streamMetrics.TotalMessages,
TotalBytes: streamMetrics.TotalBytes, TotalBytes: streamMetrics.TotalBytes,
TotalWireBytes: streamMetrics.TotalWireBytes, TotalWireBytes: streamMetrics.TotalWireBytes,
MessagesPerSec: streamMetrics.MessagesPerSec, MessagesPerSec: streamMetrics.MessagesPerSec,
MbitsPerSec: streamMetrics.BitsPerSec / bitsPerMegabit, MbitsPerSec: streamMetrics.BitsPerSec / bitsPerMegabit,
WireMbitsPerSec: streamMetrics.WireBitsPerSec / bitsPerMegabit, WireMbitsPerSec: streamMetrics.WireBitsPerSec / bitsPerMegabit,
Connected: streamMetrics.Connected, Connected: streamMetrics.Connected,
ConnectionDuration: connectionDuration(
streamMetrics.Connected, streamMetrics.ConnectedSince),
ReconnectCount: streamMetrics.ReconnectCount, ReconnectCount: streamMetrics.ReconnectCount,
GoVersion: runtime.Version(), GoVersion: runtime.Version(),
Goroutines: runtime.NumGoroutine(), Goroutines: runtime.NumGoroutine(),
MemoryUsage: humanize.Bytes(memStats.Alloc), MemoryUsageBytes: memStats.Alloc,
GC: gcStats(&memStats), GC: gcStats(&memStats),
Stream: s.streamStats(), Stream: s.streamStats(),
ASNs: dbStats.ASNs, ASNs: dbStats.ASNs,
@@ -715,6 +724,9 @@ func (s *Server) statsResponse(
IPv6PrefixDistribution: prefixDistribution(counts.IPv6PrefixesByLength), IPv6PrefixDistribution: prefixDistribution(counts.IPv6PrefixesByLength),
WHOISStats: whoisStats, WHOISStats: whoisStats,
ConnectionDurationSeconds: connectionDurationSeconds(
streamMetrics.Connected, streamMetrics.ConnectedSince),
OriginsWithoutKeptRoute: counts.OriginsWithoutKeptRoute, OriginsWithoutKeptRoute: counts.OriginsWithoutKeptRoute,
} }
} }
@@ -748,15 +760,14 @@ type IPLookupResponse struct {
} }
// IPLookupResult is the answer for one address: its PTR names, its live route // IPLookupResult is the answer for one address: its PTR names, its live route
// and that route's AS, and the blocklists that list it. IPInfo is nil when // and that route's AS, and the blocklists that list it. Every result has every
// the address has no live route, and the answer then has no route or AS // field. The fields of IPInfo are null when the address has no live route.
// fields. IP hides the IPInfo field of the same name, so that an address with
// no route has it too.
type IPLookupResult struct { type IPLookupResult struct {
*database.IPInfo IPInfo
IP string `json:"ip"` IP string `json:"ip"`
PTR []string `json:"ptr,omitempty"` // PTR is empty, never nil, when the address has no PTR names.
PTR []string `json:"ptr"`
// Blocklists has one entry for each blocklist the address is on. // Blocklists has one entry for each blocklist the address is on.
Blocklists []blocklist.Listing `json:"blocklists"` Blocklists []blocklist.Listing `json:"blocklists"`
@@ -813,10 +824,10 @@ func (s *Server) handleIPInfo() http.HandlerFunc {
} }
switch { switch {
case errors.Is(err, database.ErrNoRoute): case errors.Is(err, ErrNoRoute):
// The address still gets a result, without route fields. // The address still gets a result, with every field of
ipInfo = nil // ipInfo null.
case errors.Is(err, database.ErrInvalidIP): case errors.Is(err, ErrInvalidIP):
response.Errors = append(response.Errors, ip+": "+err.Error()) response.Errors = append(response.Errors, ip+": "+err.Error())
continue continue
@@ -842,8 +853,8 @@ func (s *Server) handleIPInfo() http.HandlerFunc {
Blocklists: s.blocklistListings(ip), Blocklists: s.blocklistListings(ip),
}) })
if ipInfo != nil && ipInfo.NeedsWHOISRefresh { if ipInfo.NeedsWHOISRefresh {
refreshASNs[ipInfo.ASN] = true refreshASNs[*ipInfo.ASN] = true
} }
} }
@@ -854,60 +865,72 @@ func (s *Server) handleIPInfo() http.HandlerFunc {
} }
} }
// ipInfo returns the IP lookup's answer for ip. The routing table gives the // ipInfo returns what the IP lookup knows about the route of ip. The routing
// most specific prefix that holds the address, the number of peers that carry // table gives the most specific prefix that holds the address, the number of
// it, the origin AS that most of them carry, and the time of its newest kept // peers that carry it, its origin ASes and the one that most peers carry, and
// route. The database gives the prefix's first-seen time and the AS's WHOIS // the time of its newest kept route. The database gives the prefix's
// details. It returns ErrNoRoute when no prefix holds the address, and // first-seen time and the AS's WHOIS details. It returns ErrNoRoute when no
// ErrInvalidIP when ip does not parse. // prefix holds the address, and ErrInvalidIP when ip does not parse. With an
func (s *Server) ipInfo(ctx context.Context, ip string) (*database.IPInfo, error) { // error, every field of the IPInfo it returns is nil.
func (s *Server) ipInfo(ctx context.Context, ip string) (IPInfo, error) {
addr, err := netip.ParseAddr(ip) addr, err := netip.ParseAddr(ip)
if err != nil { if err != nil {
return nil, fmt.Errorf("%w: %s", database.ErrInvalidIP, ip) return IPInfo{}, fmt.Errorf("%w: %s", ErrInvalidIP, ip)
} }
summary, found := s.table.Lookup(addr) summary, found := s.table.Lookup(addr)
if !found { if !found {
return nil, fmt.Errorf("%w for IP %s", database.ErrNoRoute, ip) return IPInfo{}, fmt.Errorf("%w for IP %s", ErrNoRoute, ip)
} }
firstSeen, err := s.db.GetPrefixFirstSeen(ctx, summary.Prefix) firstSeen, err := s.db.GetPrefixFirstSeen(ctx, summary.Prefix)
if err != nil { if err != nil {
return nil, err return IPInfo{}, err
} }
asInfo, err := s.db.GetASWHOIS(ctx, summary.OriginASN) asInfo, err := s.db.GetASWHOIS(ctx, summary.OriginASN)
if err != nil { if err != nil {
return nil, err return IPInfo{}, err
} }
info := &database.IPInfo{ info := IPInfo{
IP: ip, Netblock: new(summary.Prefix.String()),
Netblock: summary.Prefix.String(), MaskLength: new(summary.Prefix.Bits()),
MaskLength: summary.Prefix.Bits(), IPVersion: new(ipVersionOf(summary.Prefix)),
IPVersion: ipVersionOf(summary.Prefix), NumPeers: new(summary.Peers),
NumPeers: summary.Peers, ASN: new(summary.OriginASN),
ASN: summary.OriginASN, NumOrigins: new(len(summary.OriginASNs)),
ASName: asInfo.ASName, OriginASNs: summary.OriginASNs,
Handle: asInfo.Handle, ASName: nilIfEmpty(asInfo.ASName),
Description: asInfo.Description, Handle: nilIfEmpty(asInfo.Handle),
OrgName: asInfo.OrgName, Description: nilIfEmpty(asInfo.Description),
OrgID: asInfo.OrgID, OrgName: nilIfEmpty(asInfo.OrgName),
Address: asInfo.Address, OrgID: nilIfEmpty(asInfo.OrgID),
CountryCode: asInfo.CountryCode, Address: nilIfEmpty(asInfo.Address),
AbuseEmail: asInfo.AbuseEmail, CountryCode: nilIfEmpty(asInfo.CountryCode),
RIR: asInfo.RIR, AbuseEmail: nilIfEmpty(asInfo.AbuseEmail),
FirstSeen: firstSeen, RIR: nilIfEmpty(asInfo.RIR),
LastSeen: summary.Newest,
NeedsWHOISRefresh: needsWHOISRefresh(asInfo), NeedsWHOISRefresh: needsWHOISRefresh(asInfo),
} }
if summary.Newest != nil { if !firstSeen.IsZero() {
info.LastSeen = *summary.Newest info.FirstSeen = &firstSeen
} }
return info, nil return info, nil
} }
// nilIfEmpty returns a pointer to s, or nil when s is empty, so that JSON
// gives a detail the database does not hold as null.
func nilIfEmpty(s string) *string {
if s == "" {
return nil
}
return &s
}
// needsWHOISRefresh says whether the WHOIS details of asInfo were never // needsWHOISRefresh says whether the WHOIS details of asInfo were never
// fetched or are older than whoisStaleThreshold. AS 0, no known origin, has // fetched or are older than whoisStaleThreshold. AS 0, no known origin, has
// none to fetch. // none to fetch.
@@ -1113,15 +1136,17 @@ func publicAddresses(addrs []string) []string {
return public return public
} }
// lookupPTRNames returns the PTR records of ip without their trailing dots, or // lookupPTRNames returns the PTR records of ip without their trailing dots. It
// nil when the lookup fails or runs past ptrLookupTimeout. // returns an empty list, never nil, when there are none, when the lookup fails
// and when it runs past ptrLookupTimeout, so that an answer's ptr is [] rather
// than null.
func (s *Server) lookupPTRNames(ctx context.Context, ip string) []string { func (s *Server) lookupPTRNames(ctx context.Context, ip string) []string {
ctx, cancel := context.WithTimeout(ctx, ptrLookupTimeout) ctx, cancel := context.WithTimeout(ctx, ptrLookupTimeout)
defer cancel() defer cancel()
ptrs, err := s.resolver.LookupAddr(ctx, ip) ptrs, err := s.resolver.LookupAddr(ctx, ip)
if err != nil { if err != nil || ptrs == nil {
return nil return []string{}
} }
// Remove trailing dots from PTR records // Remove trailing dots from PTR records
@@ -1165,7 +1190,7 @@ func (s *Server) handleASDetailJSON() http.HandlerFunc {
} }
if err != nil { if err != nil {
if errors.Is(err, database.ErrNoRoute) { if errors.Is(err, ErrNoRoute) {
s.writeJSONError(w, http.StatusNotFound, err.Error()) s.writeJSONError(w, http.StatusNotFound, err.Error())
} else { } else {
s.logger.Error("Failed to get AS details", "asn", asn, "error", err) s.logger.Error("Failed to get AS details", "asn", asn, "error", err)
@@ -1201,9 +1226,9 @@ func (s *Server) handleASDetailJSON() http.HandlerFunc {
// routing table all lack. // routing table all lack.
func (s *Server) asDetails( func (s *Server) asDetails(
ctx context.Context, asn int, ctx context.Context, asn int,
) (*database.ASN, []database.LiveRoute, error) { ) (*database.ASN, []LiveRoute, error) {
if asn == ristypes.NoOrigin { if asn == ristypes.NoOrigin {
return nil, nil, fmt.Errorf("%w: AS%d", database.ErrNoRoute, asn) return nil, nil, fmt.Errorf("%w: AS%d", ErrNoRoute, asn)
} }
asInfo, known, err := s.db.GetAS(ctx, asn) asInfo, known, err := s.db.GetAS(ctx, asn)
@@ -1213,13 +1238,13 @@ func (s *Server) asDetails(
originated := s.table.OriginatedPrefixes(asn) originated := s.table.OriginatedPrefixes(asn)
if !known && len(originated) == 0 { if !known && len(originated) == 0 {
return nil, nil, fmt.Errorf("%w: AS%d", database.ErrNoRoute, asn) return nil, nil, fmt.Errorf("%w: AS%d", ErrNoRoute, asn)
} }
prefixes := make([]database.LiveRoute, 0, len(originated)) prefixes := make([]LiveRoute, 0, len(originated))
for _, p := range originated { for _, p := range originated {
route := database.LiveRoute{ route := LiveRoute{
Prefix: p.Prefix.String(), Prefix: p.Prefix.String(),
MaskLength: p.Prefix.Bits(), MaskLength: p.Prefix.Bits(),
IPVersion: ipVersionOf(p.Prefix), IPVersion: ipVersionOf(p.Prefix),
@@ -1239,9 +1264,9 @@ func (s *Server) asDetails(
// splitRoutesByIPVersion groups routes into the IPv4 and the IPv6 ones. Neither // splitRoutesByIPVersion groups routes into the IPv4 and the IPv6 ones. Neither
// group is nil, so that a JSON answer holds [] for an empty one and not null. // group is nil, so that a JSON answer holds [] for an empty one and not null.
func splitRoutesByIPVersion( func splitRoutesByIPVersion(
routes []database.LiveRoute, routes []LiveRoute,
) ([]database.LiveRoute, []database.LiveRoute) { ) ([]LiveRoute, []LiveRoute) {
ipv4Routes, ipv6Routes := []database.LiveRoute{}, []database.LiveRoute{} ipv4Routes, ipv6Routes := []LiveRoute{}, []LiveRoute{}
for _, route := range routes { for _, route := range routes {
if route.IPVersion == ipVersionV4 { if route.IPVersion == ipVersionV4 {
@@ -1256,8 +1281,8 @@ func splitRoutesByIPVersion(
// sortRoutesByMaskLength sorts routes by mask length, shortest first, and // sortRoutesByMaskLength sorts routes by mask length, shortest first, and
// routes with the same mask length by network address. // routes with the same mask length by network address.
func sortRoutesByMaskLength(routes []database.LiveRoute) { func sortRoutesByMaskLength(routes []LiveRoute) {
slices.SortFunc(routes, func(a, b database.LiveRoute) int { slices.SortFunc(routes, func(a, b LiveRoute) int {
prefixA, _ := netip.ParsePrefix(a.Prefix) prefixA, _ := netip.ParsePrefix(a.Prefix)
prefixB, _ := netip.ParsePrefix(b.Prefix) prefixB, _ := netip.ParsePrefix(b.Prefix)
@@ -1294,7 +1319,7 @@ func (s *Server) handlePrefixDetailJSON() http.HandlerFunc {
origins := s.table.Origins(parsed) origins := s.table.Origins(parsed)
if len(origins) == 0 { if len(origins) == 0 {
s.writeJSONError(w, http.StatusNotFound, s.writeJSONError(w, http.StatusNotFound,
database.ErrNoRoute.Error()+": "+prefix) ErrNoRoute.Error()+": "+prefix)
return return
} }
@@ -1302,7 +1327,7 @@ func (s *Server) handlePrefixDetailJSON() http.HandlerFunc {
routes := prefixRoutes(parsed, origins) routes := prefixRoutes(parsed, origins)
// Group by origin AS // Group by origin AS
originMap := make(map[int][]database.LiveRoute) originMap := make(map[int][]LiveRoute)
for _, route := range routes { for _, route := range routes {
originMap[route.OriginASN] = append(originMap[route.OriginASN], route) originMap[route.OriginASN] = append(originMap[route.OriginASN], route)
} }
@@ -1392,7 +1417,7 @@ func (s *Server) handleASDetail() http.HandlerFunc {
} }
if err != nil { if err != nil {
if errors.Is(err, database.ErrNoRoute) { if errors.Is(err, ErrNoRoute) {
http.Error(w, "AS not found", http.StatusNotFound) http.Error(w, "AS not found", http.StatusNotFound)
} else { } else {
s.logger.Error("Failed to get AS details", "error", err) s.logger.Error("Failed to get AS details", "error", err)
@@ -1425,8 +1450,8 @@ func (s *Server) handleASDetail() http.HandlerFunc {
// Prepare template data // Prepare template data
data := struct { data := struct {
ASN *database.ASN ASN *database.ASN
IPv4Prefixes []database.LiveRoute IPv4Prefixes []LiveRoute
IPv6Prefixes []database.LiveRoute IPv6Prefixes []LiveRoute
TotalCount int TotalCount int
IPv4Count int IPv4Count int
IPv6Count int IPv6Count int
@@ -1585,11 +1610,11 @@ func (s *Server) originASNs(
// them, since the routing table keeps only their addresses. // them, since the routing table keeps only their addresses.
func prefixRoutes( func prefixRoutes(
prefix netip.Prefix, origins []routetable.Origin, prefix netip.Prefix, origins []routetable.Origin,
) []database.LiveRoute { ) []LiveRoute {
var routes []database.LiveRoute var routes []LiveRoute
for _, origin := range origins { for _, origin := range origins {
base := database.LiveRoute{ base := LiveRoute{
Prefix: prefix.String(), Prefix: prefix.String(),
MaskLength: prefix.Bits(), MaskLength: prefix.Bits(),
IPVersion: ipVersionOf(prefix), IPVersion: ipVersionOf(prefix),
@@ -1630,14 +1655,14 @@ type ASPathEntry struct {
// EnhancedRoute is a live route with the handle of each AS in its path. // EnhancedRoute is a live route with the handle of each AS in its path.
type EnhancedRoute struct { type EnhancedRoute struct {
database.LiveRoute LiveRoute
ASPathWithHandle []ASPathEntry ASPathWithHandle []ASPathEntry
} }
// routesWithASPathHandles looks up the handle of each AS in the path of each // routesWithASPathHandles looks up the handle of each AS in the path of each
// route. // route.
func routesWithASPathHandles(routes []database.LiveRoute) []EnhancedRoute { func routesWithASPathHandles(routes []LiveRoute) []EnhancedRoute {
enhancedRoutes := make([]EnhancedRoute, len(routes)) enhancedRoutes := make([]EnhancedRoute, len(routes))
for i, route := range routes { for i, route := range routes {
@@ -1736,7 +1761,7 @@ func (s *Server) prefixLengthHandler(ipVersion, maxMaskLength int) http.HandlerF
// EnhancedPrefix is a prefix on the prefix length page, with the origin AS // EnhancedPrefix is a prefix on the prefix length page, with the origin AS
// that most peers carry it with, that AS's description, and its age. // that most peers carry it with, that AS's description, and its age.
type EnhancedPrefix struct { type EnhancedPrefix struct {
database.LiveRoute LiveRoute
OriginASDescription string OriginASDescription string
Age string Age string
internal/server/handlers_internal_test.go
+303 -43
View File
@@ -5,6 +5,7 @@ import (
"encoding/json" "encoding/json"
"errors" "errors"
"html" "html"
"maps"
"net" "net"
"net/http" "net/http"
"net/http/httptest" "net/http/httptest"
@@ -72,8 +73,8 @@ func emptyTable() *routetable.Table {
// Paths several tests request. // Paths several tests request.
const ( const (
asPagePath = "/as/64500" asPagePath = "/as/64500"
asJSONPath = "/api/v1/as/64500"
prefixLengthPagePath = "/prefixlength/24" prefixLengthPagePath = "/prefixlength/24"
statusJSONPath = "/status.json"
statsPath = "/api/v1/stats" statsPath = "/api/v1/stats"
) )
@@ -158,8 +159,8 @@ type tableStats struct {
OldestRoute time.Time `json:"oldest_route"` OldestRoute time.Time `json:"oldest_route"`
NewestRoute time.Time `json:"newest_route"` NewestRoute time.Time `json:"newest_route"`
IPv4Distribution []database.PrefixDistribution `json:"ipv4_prefix_distribution"` IPv4Distribution []PrefixDistribution `json:"ipv4_prefix_distribution"`
IPv6Distribution []database.PrefixDistribution `json:"ipv6_prefix_distribution"` IPv6Distribution []PrefixDistribution `json:"ipv6_prefix_distribution"`
} }
// TestStatsHandlersAnswerFromMemory checks that both stats handlers answer 200 // TestStatsHandlersAnswerFromMemory checks that both stats handlers answer 200
@@ -211,8 +212,8 @@ func TestStatsHandlersAnswerFromMemory(t *testing.T) {
OriginsWithoutKeptRoute: 1, OriginsWithoutKeptRoute: 1,
OldestRoute: testRouteTime(0), OldestRoute: testRouteTime(0),
NewestRoute: testRouteTime(4), NewestRoute: testRouteTime(4),
IPv4Distribution: []database.PrefixDistribution{{MaskLength: 24, Count: 2}}, IPv4Distribution: []PrefixDistribution{{MaskLength: 24, Count: 2}},
IPv6Distribution: []database.PrefixDistribution{{MaskLength: 32, Count: 1}}, IPv6Distribution: []PrefixDistribution{{MaskLength: 32, Count: 1}},
} }
for name, handler := range handlers { for name, handler := range handlers {
@@ -294,6 +295,167 @@ func TestStatsAnswerCarriesChurnPerMinute(t *testing.T) {
} }
} }
// fixedIntervalASNFetcher is a WHOIS fetcher that does nothing and reports
// interval as its wait between fetches.
type fixedIntervalASNFetcher struct {
idleASNFetcher
interval time.Duration
}
func (f fixedIntervalASNFetcher) GetStats() ASNFetcherStats {
return ASNFetcherStats{CurrentInterval: f.interval}
}
// unitStats holds the figures of a stats answer that carry a unit. Each is a
// pointer, so that a figure missing from the answer is nil.
//
//nolint:tagliatelle // field names are the existing JSON API
type unitStats struct {
UptimeSeconds *int64 `json:"uptime_seconds"`
MemoryUsageBytes *uint64 `json:"memory_usage_bytes"`
WHOISStats struct {
CurrentIntervalSeconds *float64 `json:"current_interval_seconds"`
} `json:"whois_stats"`
GC struct {
HeapAllocBytes uint64 `json:"heap_alloc_bytes"`
} `json:"gc"`
}
// TestStatsAnswersGiveUnitsInFieldNames checks that the figures of both stats
// answers that carry a unit are numbers, with the unit at the end of their
// names. They used to be strings with the unit in them, such as "1h18m34s" or
// "1.1 GB" (https://git.eeqj.de/sneak/routewatch/issues/199). The stream has
// never connected, so the uptime is 0. The memory use in /api/v1/stats is the
// same number of bytes as its gc.heap_alloc_bytes.
func TestStatsAnswersGiveUnitsInFieldNames(t *testing.T) {
t.Parallel()
const fetchInterval = 18200 * time.Millisecond
s := newServerWithTestRoutes(t)
s.SetASNFetcher(fixedIntervalASNFetcher{interval: fetchInterval})
for _, path := range []string{statusJSONPath, statsPath} {
answer := get(t, s, path).Body.Bytes()
var body struct {
Data unitStats `json:"data"`
}
err := json.Unmarshal(answer, &body)
if err != nil {
t.Errorf("%s: figures are not numbers: %v; answer %s", path, err, answer)
continue
}
got := body.Data
if got.UptimeSeconds == nil || *got.UptimeSeconds != 0 {
t.Errorf("%s: uptime_seconds is not 0: %s", path, answer)
}
if got.MemoryUsageBytes == nil || *got.MemoryUsageBytes == 0 {
t.Errorf("%s: memory_usage_bytes is not a number of bytes: %s",
path, answer)
}
interval := got.WHOISStats.CurrentIntervalSeconds
if interval == nil || *interval != fetchInterval.Seconds() {
t.Errorf("%s: whois_stats.current_interval_seconds is not %v: %s",
path, fetchInterval.Seconds(), answer)
}
if path == statsPath && got.MemoryUsageBytes != nil &&
*got.MemoryUsageBytes != got.GC.HeapAllocBytes {
t.Errorf("%s: memory_usage_bytes %d, want gc.heap_alloc_bytes %d",
path, *got.MemoryUsageBytes, got.GC.HeapAllocBytes)
}
}
}
// TestStatsAnswerGivesConnectionDurationInSeconds checks that the
// connection_duration_seconds of /api/v1/stats is null while the stream is not
// connected, and a whole number of seconds once it is. It used to be the
// string "disconnected", or a Go duration such as "1h18m34s"
// (https://git.eeqj.de/sneak/routewatch/issues/199).
func TestStatsAnswerGivesConnectionDurationInSeconds(t *testing.T) {
t.Parallel()
s := newServerWithTestRoutes(t)
connectionDuration := func() json.RawMessage {
answer := get(t, s, statsPath).Body.Bytes()
var body struct {
Data map[string]json.RawMessage `json:"data"`
}
err := json.Unmarshal(answer, &body)
if err != nil {
t.Fatalf("decoding the answer: %v; answer %s", err, answer)
}
duration, ok := body.Data["connection_duration_seconds"]
if !ok {
t.Fatalf("no connection_duration_seconds: %s", answer)
}
return duration
}
disconnected := connectionDuration()
if string(disconnected) != "null" {
t.Errorf("disconnected: connection_duration_seconds = %s, want null",
disconnected)
}
s.streamer.GetMetricsTracker().SetConnected(true)
var seconds int64
connected := connectionDuration()
err := json.Unmarshal(connected, &seconds)
if err != nil || seconds != 0 {
t.Errorf("just connected: connection_duration_seconds = %s, want 0",
connected)
}
}
// TestStreamDurationsCountSeconds checks that the uptime and the connection
// duration of a stream that connected 90 seconds ago are both 90. A stream
// that has never connected has an uptime of 0, and one that is not connected
// has no connection duration.
func TestStreamDurationsCountSeconds(t *testing.T) {
t.Parallel()
const connectedFor = 90 * time.Second
connectedSince := time.Now().Add(-connectedFor)
want := int64(connectedFor / time.Second)
if got := uptimeSeconds(connectedSince); got != want {
t.Errorf("uptimeSeconds = %d, want %d", got, want)
}
if got := uptimeSeconds(time.Time{}); got != 0 {
t.Errorf("uptimeSeconds before any connection = %d, want 0", got)
}
switch got := connectionDurationSeconds(true, connectedSince); {
case got == nil:
t.Errorf("connectionDurationSeconds = nil, want %d", want)
case *got != want:
t.Errorf("connectionDurationSeconds = %d, want %d", *got, want)
}
if got := connectionDurationSeconds(false, connectedSince); got != nil {
t.Errorf("connectionDurationSeconds while not connected = %d, want nil",
*got)
}
}
// newTestTable returns a routing table that holds a route for testPrefixV4, // newTestTable returns a routing table that holds a route for testPrefixV4,
// with origin AS 64500, and one for testPrefixV6, with origin AS 64501. // with origin AS 64500, and one for testPrefixV6, with origin AS 64501.
func newTestTable() *routetable.Table { func newTestTable() *routetable.Table {
@@ -422,27 +584,9 @@ func TestIPLookupsReadPercentEncodedAddresses(t *testing.T) {
} }
} }
// errASDetailsLookupFailed is what failingASDetailsDB answers every AS details
// lookup with.
var errASDetailsLookupFailed = errors.New("AS details lookup failed")
// failingASDetailsDB wraps a real store and fails every AS details lookup.
type failingASDetailsDB struct {
database.Store
}
func (failingASDetailsDB) GetASDetailsContext(
_ context.Context, _ int,
) (*database.ASN, []database.LiveRoute, error) {
return nil, nil, errASDetailsLookupFailed
}
// TestPrefixDetailShowsOriginNames checks that the prefix page shows each // TestPrefixDetailShowsOriginNames checks that the prefix page shows each
// origin AS's handle and description without looking up the AS's details, // origin AS's handle and description. The AS path column shows handles too, so
// which also read every route the AS originates. Before the fix the page did // the test looks for the origin list's "(handle)" form.
// that lookup once per origin, so with it failing the names were missing
// (https://git.eeqj.de/sneak/routewatch/issues/67). The AS path column shows
// handles too, so the test looks for the origin list's "(handle)" form.
func TestPrefixDetailShowsOriginNames(t *testing.T) { func TestPrefixDetailShowsOriginNames(t *testing.T) {
t.Parallel() t.Parallel()
@@ -465,7 +609,7 @@ func TestPrefixDetailShowsOriginNames(t *testing.T) {
announce(table, testPrefixV4, asn, testPeer(i+1), testRouteTime(0)) announce(table, testPrefixV4, asn, testPeer(i+1), testRouteTime(0))
} }
s := New(failingASDetailsDB{db}, table, s := New(db, table,
streamer.New(logger.New(), metrics.New()), logger.New(), &config.Config{}) streamer.New(logger.New(), metrics.New()), logger.New(), &config.Config{})
rec := httptest.NewRecorder() rec := httptest.NewRecorder()
s.router.ServeHTTP(rec, httptest.NewRequestWithContext( s.router.ServeHTTP(rec, httptest.NewRequestWithContext(
@@ -783,7 +927,7 @@ func TestRoutesAnswer504WhenDatabaseReadIsCutOff(t *testing.T) {
{path: "/ip/" + addrInTestPrefixV4, wantJSONBody: true}, {path: "/ip/" + addrInTestPrefixV4, wantJSONBody: true},
{path: statsPath, wantJSONBody: true}, {path: statsPath, wantJSONBody: true},
{path: "/api/v1/ip/" + addrInTestPrefixV4, wantJSONBody: true}, {path: "/api/v1/ip/" + addrInTestPrefixV4, wantJSONBody: true},
{path: "/api/v1/as/64500", wantJSONBody: true}, {path: asJSONPath, wantJSONBody: true},
} }
for _, tt := range tests { for _, tt := range tests {
@@ -964,10 +1108,10 @@ func TestJSONRoutesDoNotSendErrorDetails(t *testing.T) {
path string path string
wantCode int wantCode int
}{ }{
{"/.well-known/healthcheck.json", http.StatusServiceUnavailable}, {healthCheckPath, http.StatusServiceUnavailable},
{statusJSONPath, http.StatusInternalServerError}, {statusJSONPath, http.StatusInternalServerError},
{statsPath, http.StatusInternalServerError}, {statsPath, http.StatusInternalServerError},
{"/api/v1/as/64500", http.StatusInternalServerError}, {asJSONPath, http.StatusInternalServerError},
{"/api/v1/ip/" + addrInTestPrefixV4, http.StatusInternalServerError}, {"/api/v1/ip/" + addrInTestPrefixV4, http.StatusInternalServerError},
{"/api/v1/prefix/foo/24", http.StatusBadRequest}, {"/api/v1/prefix/foo/24", http.StatusBadRequest},
} }
@@ -1237,7 +1381,8 @@ func TestIPLookupKeepsRouteWhenPTRLookupDoesNotAnswer(t *testing.T) {
t.Fatalf("decoding the answer %q: %v", rec.Body, err) t.Fatalf("decoding the answer %q: %v", rec.Body, err)
} }
if len(body.Data.Results) != 1 || body.Data.Results[0].Netblock != testPrefixV4 { if len(body.Data.Results) != 1 || body.Data.Results[0].Netblock == nil ||
*body.Data.Results[0].Netblock != testPrefixV4 {
t.Errorf("answer %s, want one result in %s", rec.Body, testPrefixV4) t.Errorf("answer %s, want one result in %s", rec.Body, testPrefixV4)
} }
} }
@@ -1329,12 +1474,39 @@ func TestIPLookupReportsBlocklists(t *testing.T) {
} }
} }
// jsonNull is how JSON writes a field without a value.
const jsonNull = "null"
// jsonEmptyList is how JSON writes an empty list.
const jsonEmptyList = "[]"
// resultFields decodes an IP lookup answer and returns each of its results as
// its JSON fields.
func resultFields(t *testing.T, answer []byte) []map[string]json.RawMessage {
t.Helper()
var body struct {
Data struct {
Results []map[string]json.RawMessage `json:"results"`
} `json:"data"`
}
err := json.Unmarshal(answer, &body)
if err != nil {
t.Fatalf("decoding the answer %s: %v", answer, err)
}
return body.Data.Results
}
// TestIPLookupReportsBlocklistsOfAnAddressWithNoRoute looks up a listed // TestIPLookupReportsBlocklistsOfAnAddressWithNoRoute looks up a listed
// address that has no live route, alone and beside a routed address. Each // address that has no live route, alone and beside a routed address. Each
// answer is 200 and has a result for the address, with its blocklists and no // answer is 200 and has a result for the address, with its blocklists and
// route fields. Before the fix the lookup of that address alone answered // every route field set to null. Before the fix the lookup of that address
// 404, and beside a routed address it was only named in errors, so its // alone answered 404, and beside a routed address it was only named in
// blocklists were never reported. // errors, so its blocklists were never reported. After that fix the result
// still left out the route fields
// (https://git.eeqj.de/sneak/routewatch/issues/200).
func TestIPLookupReportsBlocklistsOfAnAddressWithNoRoute(t *testing.T) { func TestIPLookupReportsBlocklistsOfAnAddressWithNoRoute(t *testing.T) {
t.Parallel() t.Parallel()
@@ -1342,6 +1514,15 @@ func TestIPLookupReportsBlocklistsOfAnAddressWithNoRoute(t *testing.T) {
s.resolver = &fakeResolver{} s.resolver = &fakeResolver{}
s.SetBlocklists(fakeBlocklists{}) s.SetBlocklists(fakeBlocklists{})
// routeFields are the fields of a result that come from the address's
// route and that route's origin AS.
routeFields := []string{
"netblock", "mask_length", "ip_version", "num_peers", "asn",
"num_origins", "origin_asns", "as_name", "handle", "description",
"org_name", "org_id", "address", "country_code", "abuse_email", "rir",
"first_seen", "last_seen",
}
for _, target := range []string{ for _, target := range []string{
listedUnroutedAddr, listedUnroutedAddr,
addrInTestPrefixV4 + "," + listedUnroutedAddr, addrInTestPrefixV4 + "," + listedUnroutedAddr,
@@ -1372,12 +1553,21 @@ func TestIPLookupReportsBlocklistsOfAnAddressWithNoRoute(t *testing.T) {
continue continue
} }
result := body.Data.Results[i] fields := resultFields(t, rec.Body.Bytes())[i]
if result.IPInfo != nil { for _, name := range routeFields {
t.Errorf("%s: result for %s has route fields %+v", value, present := fields[name]
target, listedUnroutedAddr, result.IPInfo)
switch {
case !present:
t.Errorf("%s: the result for %s has no %s",
target, listedUnroutedAddr, name)
case string(value) != jsonNull:
t.Errorf("%s: %s of %s is %s, want %s",
target, name, listedUnroutedAddr, value, jsonNull)
}
} }
result := body.Data.Results[i]
if len(result.Blocklists) != 1 || if len(result.Blocklists) != 1 ||
result.Blocklists[0].Name != testListing().Name { result.Blocklists[0].Name != testListing().Name {
t.Errorf("%s: blocklists %+v, want [%+v]", t.Errorf("%s: blocklists %+v, want [%+v]",
@@ -1386,6 +1576,76 @@ func TestIPLookupReportsBlocklistsOfAnAddressWithNoRoute(t *testing.T) {
} }
} }
// twoAddressHostName is a host name that resolves to addrInTestPrefixV4 and
// listedUnroutedAddr.
const twoAddressHostName = "two-addresses.test"
// TestIPLookupResultsHaveTheSameFields looks up a host name, addresses and
// the caller's own address, through each IP lookup route. Some of the
// addresses have a route and some do not, and none has PTR names. Every
// result has the same fields. Before the fix a result left out every field
// without a value (https://git.eeqj.de/sneak/routewatch/issues/200).
func TestIPLookupResultsHaveTheSameFields(t *testing.T) {
t.Parallel()
s := newServerWithTestRoutes(t)
s.resolver = &fakeResolver{hosts: map[string][]string{
twoAddressHostName: {addrInTestPrefixV4, listedUnroutedAddr},
}}
s.SetBlocklists(fakeBlocklists{})
// want is the fields of the first result.
var want []string
for _, path := range []string{
"/api/v1/ip/" + twoAddressHostName,
"/ip/" + addrInTestPrefixV6 + "," + listedUnroutedAddr,
ownAddressPath,
} {
for _, result := range resultFields(t, get(t, s, path).Body.Bytes()) {
got := slices.Sorted(maps.Keys(result))
if want == nil {
want = got
}
if !slices.Equal(got, want) {
t.Errorf("%s: a result has the fields %v, want %v", path, got, want)
}
}
}
}
// TestIPLookupOfAnAddressWithoutPTRNamesOrStoredNetblock looks up a routed
// address that has no PTR names, and whose netblock the database has not
// stored. Its ptr is an empty list and its first_seen is null. Before the fix
// ptr was left out, and first_seen was 0001-01-01T00:00:00Z
// (https://git.eeqj.de/sneak/routewatch/issues/200).
func TestIPLookupOfAnAddressWithoutPTRNamesOrStoredNetblock(t *testing.T) {
t.Parallel()
s := newServerWithTestRoutes(t)
s.resolver = &fakeResolver{}
path := "/api/v1/ip/" + addrInTestPrefixV4
results := resultFields(t, get(t, s, path).Body.Bytes())
if len(results) != 1 {
t.Fatalf("%s: %d results, want 1", path, len(results))
}
want := map[string]string{
"ptr": jsonEmptyList,
"first_seen": jsonNull,
}
for name, value := range want {
got := string(results[0][name])
if got != value {
t.Errorf("%s: %s is %s, want %s", path, name, got, value)
}
}
}
// TestIPLookupAnswersMayBeCached checks the Cache-Control header of the IP // TestIPLookupAnswersMayBeCached checks the Cache-Control header of the IP
// lookup answers. Caches may keep them for the blocklists' shortest interval, // lookup answers. Caches may keep them for the blocklists' shortest interval,
// and only the caller's own browser may keep the answer about the caller's // and only the caller's own browser may keep the answer about the caller's
@@ -1716,9 +1976,9 @@ func TestASDetailJSONAnswersBuiltInAS(t *testing.T) {
//nolint:tagliatelle // field names are the existing JSON API //nolint:tagliatelle // field names are the existing JSON API
var body struct { var body struct {
Data struct { Data struct {
ASN database.ASN `json:"asn"` ASN database.ASN `json:"asn"`
IPv4Prefixes []database.LiveRoute `json:"ipv4_prefixes"` IPv4Prefixes []LiveRoute `json:"ipv4_prefixes"`
IPv6Prefixes []database.LiveRoute `json:"ipv6_prefixes"` IPv6Prefixes []LiveRoute `json:"ipv6_prefixes"`
} `json:"data"` } `json:"data"`
} }
@@ -1863,9 +2123,9 @@ func TestSortRoutesByMaskLength(t *testing.T) {
} }
for _, tt := range tests { for _, tt := range tests {
routes := make([]database.LiveRoute, 0, len(tt.prefixes)) routes := make([]LiveRoute, 0, len(tt.prefixes))
for _, prefix := range tt.prefixes { for _, prefix := range tt.prefixes {
routes = append(routes, database.LiveRoute{Prefix: prefix}) routes = append(routes, LiveRoute{Prefix: prefix})
} }
sortRoutesByMaskLength(routes) sortRoutesByMaskLength(routes)
internal/server/handlers_table_internal_test.go
+62 -17
View File
@@ -4,6 +4,8 @@ import (
"encoding/json" "encoding/json"
"net/http" "net/http"
"net/http/httptest" "net/http/httptest"
"net/netip"
"reflect"
"slices" "slices"
"strconv" "strconv"
"strings" "strings"
@@ -11,6 +13,7 @@ import (
"time" "time"
"git.eeqj.de/sneak/routewatch/internal/database" "git.eeqj.de/sneak/routewatch/internal/database"
"git.eeqj.de/sneak/routewatch/internal/ristypes"
"git.eeqj.de/sneak/routewatch/internal/routetable" "git.eeqj.de/sneak/routewatch/internal/routetable"
) )
@@ -74,9 +77,10 @@ func get(t *testing.T, s *Server, path string) *httptest.ResponseRecorder {
// TestIPLookupGivesTheOriginMostPeersCarry looks up an address in a prefix // TestIPLookupGivesTheOriginMostPeersCarry looks up an address in a prefix
// with two origin ASes, through each IP lookup route. The answer names the // with two origin ASes, through each IP lookup route. The answer names the
// origin AS that more peers carry, counts every peer, and gives the time of // origin AS that more peers carry, counts every peer, lists both origin ASes,
// the prefix's newest kept route. The prefix's first-seen time and the AS's // and gives the time of the prefix's newest kept route. The prefix's
// WHOIS details come from the database. // first-seen time and the AS's WHOIS details come from the database. A WHOIS
// detail the database does not hold is null.
func TestIPLookupGivesTheOriginMostPeersCarry(t *testing.T) { func TestIPLookupGivesTheOriginMostPeersCarry(t *testing.T) {
t.Parallel() t.Parallel()
@@ -104,6 +108,26 @@ func TestIPLookupGivesTheOriginMostPeersCarry(t *testing.T) {
t.Fatalf("UpdateASNWHOIS: %v", err) t.Fatalf("UpdateASNWHOIS: %v", err)
} }
// The details the database does not hold, such as org_name, are null.
want := IPInfo{
Netblock: new(testPrefixV4),
MaskLength: new(netip.MustParsePrefix(testPrefixV4).Bits()),
IPVersion: new(ipVersionV4),
NumPeers: new(4),
ASN: new(mostPeersASN),
NumOrigins: new(2),
OriginASNs: []int{mostPeersASN, fewerPeersASN},
ASName: &whois.ASName,
CountryCode: &whois.CountryCode,
FirstSeen: &firstSeen,
LastSeen: new(testRouteTime(5)),
}
wantJSON, err := json.Marshal(want)
if err != nil {
t.Fatalf("encoding %+v: %v", want, err)
}
// ownAddressPath looks up the request's own address, which httptest // ownAddressPath looks up the request's own address, which httptest
// gives as addrInTestPrefixV4. // gives as addrInTestPrefixV4.
for _, path := range []string{ for _, path := range []string{
@@ -114,22 +138,43 @@ func TestIPLookupGivesTheOriginMostPeersCarry(t *testing.T) {
Data IPLookupResponse `json:"data"` Data IPLookupResponse `json:"data"`
} }
err := json.Unmarshal(get(t, s, path).Body.Bytes(), &body) answer := get(t, s, path).Body.Bytes()
if err != nil || len(body.Data.Results) != 1 ||
body.Data.Results[0].IPInfo == nil { err := json.Unmarshal(answer, &body)
t.Fatalf("%s: answer %+v (%v), want one result with a route", if err != nil || len(body.Data.Results) != 1 {
path, body.Data, err) t.Fatalf("%s: answer %s (%v), want one result", path, answer, err)
} }
got := body.Data.Results[0].IPInfo if !reflect.DeepEqual(body.Data.Results[0].IPInfo, want) {
if got.Netblock != testPrefixV4 || got.ASN != mostPeersASN || t.Errorf("%s: answer %s, want the fields %s", path, answer, wantJSON)
got.NumPeers != 4 || !got.LastSeen.Equal(testRouteTime(5)) || }
!got.FirstSeen.Equal(firstSeen) || got.ASName != whois.ASName || }
got.CountryCode != whois.CountryCode { }
t.Errorf("%s: route %+v, want %s from AS%d, 4 peers, last seen %v, "+
"first seen %v, %s in %s", path, *got, testPrefixV4, // TestIPLookupAsksForNoWHOISLookupOfOriginZero checks that the IP lookup asks
mostPeersASN, testRouteTime(5), firstSeen, whois.ASName, // for no WHOIS lookup of AS 0, which means no known origin. It asks for one of
whois.CountryCode) // an AS whose WHOIS details were never fetched or are stale, and not of one
// whose details are fresh.
func TestIPLookupAsksForNoWHOISLookupOfOriginZero(t *testing.T) {
t.Parallel()
fresh := time.Now().Add(-time.Hour)
stale := time.Now().Add(-2 * whoisStaleThreshold)
tests := []struct {
name string
asInfo database.ASN
want bool
}{
{"no known origin", database.ASN{ASN: ristypes.NoOrigin}, false},
{"details never fetched", database.ASN{ASN: mostPeersASN}, true},
{"stale details", database.ASN{ASN: mostPeersASN, WHOISUpdatedAt: &stale}, true},
{"fresh details", database.ASN{ASN: mostPeersASN, WHOISUpdatedAt: &fresh}, false},
}
for _, tt := range tests {
if got := needsWHOISRefresh(&tt.asInfo); got != tt.want {
t.Errorf("%s: needsWHOISRefresh = %t, want %t", tt.name, got, tt.want)
} }
} }
} }
internal/server/home.go
+206
View File
@@ -0,0 +1,206 @@
package server
import (
"context"
"html/template"
"net/http"
"slices"
"time"
"git.eeqj.de/sneak/routewatch/internal/database"
"git.eeqj.de/sneak/routewatch/internal/routetable"
"git.eeqj.de/sneak/routewatch/internal/templates"
asinfo "git.eeqj.de/sneak/routewatch/pkg/asinfo"
)
const (
// homeCardsRefreshInterval is how often the server works out the home
// page's cards again. Some of their figures take a walk over the whole
// routing table, so no request works them out.
homeCardsRefreshInterval = time.Minute
// homeCardsDatabaseTimeout is how long the database read behind the
// cards of the newest and the oldest AS may take.
homeCardsDatabaseTimeout = 4 * time.Second
// maxListedEntries is how many entries the bogon and multiple-origin
// pages list at most.
maxListedEntries = 1000
)
// homeCards is what the home page's cards show. The bogon and multiple-origin
// pages list the entries their cards count. The server works it out every
// homeCardsRefreshInterval and serves this copy in between.
type homeCards struct {
routetable.Overview
// NewestAS and OldestAS are the ASes the asns table last saw most
// recently and longest ago. They are nil when the database did not
// answer or holds no AS.
NewestAS *database.ASN
OldestAS *database.ASN
// WellKnownNetworks are the networks of wellKnownASNs, newest update
// first.
WellKnownNetworks []wellKnownNetwork
}
// wellKnownNetwork is one network on the home page's card of well-known
// networks.
type wellKnownNetwork struct {
ASN int
// Name is its description in the AS list built into RouteWatch.
Name string
// Prefixes is how many prefixes it originates.
Prefixes int
// Newest is the time of its newest kept route, or nil when it keeps none.
Newest *time.Time
}
// listPageData is what the bogon and multiple-origin pages are rendered with.
type listPageData struct {
// CSPNonce is the nonce the page's style block carries.
CSPNonce string
// Cards is the copy of the home page's cards, or nil before the server
// has first worked them out.
Cards *homeCards
}
// The AS numbers of the networks the home page lists as well known.
const (
appleASN = 714
googleASN = 15169
amazonASN = 16509
microsoftASN = 8075
metaASN = 32934
cloudflareASN = 13335
akamaiASN = 20940
netflixASN = 2906
dodNetworkInfoCtrASN = 721
cogentASN = 174
hurricaneElectricASN = 6939
level3ASN = 3356
arelionASN = 1299
nttASN = 2914
mitASN = 3
)
// wellKnownASNs returns the AS numbers of the networks the home page lists as
// well known: large companies, large transit networks, the US Department of
// Defense's Network Information Center and a university.
func wellKnownASNs() []int {
return []int{
appleASN, googleASN, amazonASN, microsoftASN, metaASN, cloudflareASN,
akamaiASN, netflixASN, dodNetworkInfoCtrASN, cogentASN,
hurricaneElectricASN, level3ASN, arelionASN, nttASN, mitASN,
}
}
// keepHomeCardsFresh works out the home page's cards now, and again every
// interval until ctx ends.
func (s *Server) keepHomeCardsFresh(ctx context.Context, interval time.Duration) {
ticker := time.NewTicker(interval)
defer ticker.Stop()
for {
s.refreshHomeCards(ctx)
select {
case <-ctx.Done():
return
case <-ticker.C:
}
}
}
// refreshHomeCards works out the home page's cards and keeps them for the
// pages to serve. When the database does not answer, the failure is logged
// and the cards of the newest and the oldest AS stay empty.
func (s *Server) refreshHomeCards(ctx context.Context) {
cards := &homeCards{
Overview: s.table.Overview(maxListedEntries),
WellKnownNetworks: s.wellKnownNetworks(),
}
ctx, cancel := context.WithTimeout(ctx, homeCardsDatabaseTimeout)
defer cancel()
newest, oldest, err := s.db.GetLastSeenASes(ctx)
if err != nil {
s.logger.Warn("Failed to read the newest and the oldest AS", "error", err)
} else {
cards.NewestAS, cards.OldestAS = newest, oldest
}
s.cards.Store(cards)
}
// wellKnownNetworks returns the networks of wellKnownASNs, each with how many
// prefixes it originates and the time of its newest kept route. They come
// newest first, and those that keep no route last.
func (s *Server) wellKnownNetworks() []wellKnownNetwork {
asns := wellKnownASNs()
networks := make([]wellKnownNetwork, 0, len(asns))
for _, asn := range asns {
originated := s.table.OriginatedPrefixes(asn)
network := wellKnownNetwork{ASN: asn, Prefixes: len(originated)}
if info, ok := asinfo.Get(asn); ok {
network.Name = info.Description
}
for _, p := range originated {
if p.Newest != nil &&
(network.Newest == nil || p.Newest.After(*network.Newest)) {
network.Newest = p.Newest
}
}
networks = append(networks, network)
}
slices.SortStableFunc(networks, func(a, b wellKnownNetwork) int {
switch {
case a.Newest == nil && b.Newest == nil:
return 0
case a.Newest == nil:
return 1
case b.Newest == nil:
return -1
}
return b.Newest.Compare(*a.Newest)
})
return networks
}
// handleBogons returns a handler that serves the page listing the bogon
// announcements.
func (s *Server) handleBogons() http.HandlerFunc {
return s.listPageHandler(templates.BogonsTemplate())
}
// handleMultipleOrigins returns a handler that serves the page listing the
// prefixes announced by more than one origin AS.
func (s *Server) handleMultipleOrigins() http.HandlerFunc {
return s.listPageHandler(templates.MultipleOriginsTemplate())
}
// listPageHandler returns a handler that renders tmpl, a page that lists
// entries the home page's cards count, from the copy of the cards.
func (s *Server) listPageHandler(tmpl *template.Template) http.HandlerFunc {
return func(w http.ResponseWriter, r *http.Request) {
w.Header().Set("Content-Type", "text/html; charset=utf-8")
err := tmpl.Execute(w, listPageData{
CSPNonce: cspNonce(r.Context()),
Cards: s.cards.Load(),
})
if err != nil {
s.logger.Error("Failed to render a list page", "error", err)
http.Error(w, "Internal Server Error", http.StatusInternalServerError)
}
}
}
internal/server/home_internal_test.go
+419
View File
@@ -0,0 +1,419 @@
package server
import (
"context"
"net/netip"
"strconv"
"strings"
"testing"
"testing/synctest"
"time"
"git.eeqj.de/sneak/routewatch/internal/database"
"git.eeqj.de/sneak/routewatch/internal/routetable"
)
// The ASes of the routing table that newCardsTable fills. Each card test
// finds one of them on its card.
const (
twoPrefixASN = 13335 // originates two prefixes, the most
newestRouteASN = 15169 // the newest route
oldestRouteASN = 3 // the oldest route
secondOrigin = 16509 // a second origin AS of twoPrefixASN's prefix
privateUseASN = 64512 // reserved, so its route is a bogon announcement
)
// Paths of the pages that list what two of the home page's cards count.
const (
bogonsPath = "/bogons"
multipleOriginsPath = "/multiple-origins"
)
// cardsNotReadyText is what the pages say before the cards are first worked
// out.
const cardsNotReadyText = "being worked out"
// newCardsTable returns a routing table in which:
// - twoPrefixASN originates 1.0.0.0/24 and 1.0.4.0/22;
// - secondOrigin originates 1.0.0.0/24 too, which so has two origin ASes;
// - newestRouteASN's 8.8.8.0/24 has the newest route;
// - oldestRouteASN's 18.0.0.0/8 has the oldest;
// - privateUseASN originates 10.0.0.0/8, carried by two peers: a bogon
// announcement for both its prefix and its AS.
//
// The last four ASes originate one prefix each.
func newCardsTable() *routetable.Table {
table := emptyTable()
announce(table, "1.0.0.0/24", twoPrefixASN, testPeer(1), testRouteTime(10))
announce(table, "1.0.4.0/22", twoPrefixASN, testPeer(1), testRouteTime(5))
announce(table, "1.0.0.0/24", secondOrigin, testPeer(3), testRouteTime(12))
announce(table, "8.8.8.0/24", newestRouteASN, testPeer(2), testRouteTime(20))
announce(table, "18.0.0.0/8", oldestRouteASN, testPeer(2), testRouteTime(1))
announce(table, "10.0.0.0/8", privateUseASN, testPeer(1), testRouteTime(15))
announce(table, "10.0.0.0/8", privateUseASN, testPeer(2), testRouteTime(15))
return table
}
// homeCard returns the HTML of the home page's card with heading, up to the
// end of the card.
func homeCard(t *testing.T, page, heading string) string {
t.Helper()
_, card, found := strings.Cut(page, "<h2>"+heading+"</h2>")
if !found {
t.Fatalf("the home page has no card %q", heading)
}
card, _, _ = strings.Cut(card, "</div>")
return card
}
// asLink returns the HTML attribute of a link to the AS page of asn.
func asLink(asn int) string {
return `href="/as/` + strconv.Itoa(asn) + `"`
}
// TestHomePageCardsShowTheTableFigures fills a routing table and the asns
// table, works out the cards, and checks that each card shows its figure.
func TestHomePageCardsShowTheTableFigures(t *testing.T) {
t.Parallel()
const (
newestSeenASN = 6939
oldestSeenASN = 174
)
s := newServerWithTable(t, newCardsTable())
err := s.db.GetOrCreateASNBatch(map[int]database.SeenTimes{
newestSeenASN: {First: testRouteTime(0), Last: testRouteTime(30)},
oldestSeenASN: {First: testRouteTime(-30), Last: testRouteTime(-30)},
})
if err != nil {
t.Fatalf("GetOrCreateASNBatch: %v", err)
}
s.refreshHomeCards(t.Context())
page := get(t, s, "/").Body.String()
tests := []struct {
heading string
want []string
}{
{"Newest Updated Route", []string{
"8.8.8.0/24", asLink(newestRouteASN),
testRouteTime(20).Format(routeTimeFormat),
}},
{"Oldest Updated Route", []string{
"18.0.0.0/8", asLink(oldestRouteASN),
testRouteTime(1).Format(routeTimeFormat),
}},
{"Newest Updated AS", []string{
asLink(newestSeenASN), testRouteTime(30).Format(routeTimeFormat),
}},
{"Oldest Updated AS", []string{
asLink(oldestSeenASN), testRouteTime(-30).Format(routeTimeFormat),
}},
{"AS with the Most Prefixes", []string{
asLink(twoPrefixASN), "Prefixes it originates: 2",
}},
{"ASes with the Fewest Prefixes", []string{
"ASes that originate a single prefix: 4", asLink(newestRouteASN),
"8.8.8.0/24",
}},
{"Bogon Announcements", []string{`href="/bogons">1<`}},
{"Prefixes with Several Origins", []string{`href="/multiple-origins">1<`}},
}
for _, tc := range tests {
card := homeCard(t, page, tc.heading)
for _, want := range tc.want {
if !strings.Contains(card, want) {
t.Errorf("card %q does not hold %s: %s", tc.heading, want, card)
}
}
}
// The well-known networks come newest update first, and those without
// a route after them.
networks := homeCard(t, page, "Well-Known Networks")
order := []int{newestRouteASN, secondOrigin, twoPrefixASN, oldestRouteASN, 714}
last := -1
for _, asn := range order {
i := strings.Index(networks, asLink(asn))
if i <= last {
t.Fatalf("well-known networks: AS%d is not after the ASes before "+
"it in %v: %s", asn, order, networks)
}
last = i
}
if !strings.Contains(networks, "<td>2</td>") {
t.Errorf("well-known networks: no network with 2 prefixes: %s", networks)
}
}
// TestListPagesListTheirEntries checks the rows of the bogon and multiple-origin
// pages for the table of newCardsTable.
func TestListPagesListTheirEntries(t *testing.T) {
t.Parallel()
s := newServerWithTable(t, newCardsTable())
s.refreshHomeCards(t.Context())
pages := map[string][]string{
bogonsPath: {
"Bogon announcements: 1.", "10.0.0.0/8", asLink(privateUseASN),
"<td>2</td>",
"Private-Use (10.0.0.0/8); Private-Use (AS64512-AS65534)",
},
multipleOriginsPath: {
"Prefixes announced by more than one origin AS: 1.", "1.0.0.0/24",
asLink(twoPrefixASN), asLink(secondOrigin), ">1 peer<",
},
}
for path, wants := range pages {
page := get(t, s, path).Body.String()
for _, want := range wants {
if !strings.Contains(page, want) {
t.Errorf("%s does not hold %s", path, want)
}
}
}
}
// TestListPagesListAtMostMaxListedEntries fills a table with more prefixes than
// the list pages list, each announced by two origin ASes in reserved address
// space. Each page lists maxListedEntries of them and gives how many there are
// in all.
func TestListPagesListAtMostMaxListedEntries(t *testing.T) {
t.Parallel()
const prefixCount = maxListedEntries + 1
table := emptyTable()
addr := netip.MustParseAddr("10.0.0.0")
for range prefixCount {
prefix := netip.PrefixFrom(addr, addr.BitLen()).String()
announce(table, prefix, twoPrefixASN, testPeer(1), testRouteTime(0))
announce(table, prefix, secondOrigin, testPeer(2), testRouteTime(0))
addr = addr.Next()
}
s := newServerWithTable(t, table)
s.refreshHomeCards(t.Context())
listed := "The first " + strconv.Itoa(maxListedEntries) + " are listed"
tests := []struct {
path string
total string
}{
{bogonsPath, "Bogon announcements: " + strconv.Itoa(2*prefixCount) + "."},
{multipleOriginsPath, "Prefixes announced by more than one origin AS: " +
strconv.Itoa(prefixCount) + "."},
}
for _, tc := range tests {
page := get(t, s, tc.path).Body.String()
rows := strings.Count(page, `class="prefix-link"`)
if rows != maxListedEntries {
t.Errorf("%s lists %d entries, want %d", tc.path, rows,
maxListedEntries)
}
if !strings.Contains(page, tc.total) || !strings.Contains(page, listed) {
t.Errorf("%s does not say %q and %q", tc.path, tc.total, listed)
}
}
}
// TestPagesServeTheCopyOfTheCards checks that the pages show the cards as they
// were last worked out, not as the routing table is now, and that they say so
// before the cards are first worked out.
func TestPagesServeTheCopyOfTheCards(t *testing.T) {
t.Parallel()
table := emptyTable()
announce(table, "1.0.0.0/24", twoPrefixASN, testPeer(1), testRouteTime(0))
s := newServerWithTable(t, table)
for _, path := range []string{"/", bogonsPath, multipleOriginsPath} {
if !strings.Contains(get(t, s, path).Body.String(), cardsNotReadyText) {
t.Errorf("%s does not say the cards are %s", path, cardsNotReadyText)
}
}
s.refreshHomeCards(t.Context())
announce(table, "8.8.8.0/24", newestRouteASN, testPeer(2), testRouteTime(1))
card := homeCard(t, get(t, s, "/").Body.String(), "Newest Updated Route")
if !strings.Contains(card, "1.0.0.0/24") {
t.Errorf("before the cards are worked out again, the newest route "+
"is not the one they were worked out with: %s", card)
}
s.refreshHomeCards(t.Context())
card = homeCard(t, get(t, s, "/").Body.String(), "Newest Updated Route")
if !strings.Contains(card, "8.8.8.0/24") {
t.Errorf("after the cards are worked out again, the newest route is "+
"not the table's: %s", card)
}
}
// TestKeepHomeCardsFreshWorksThemOutEveryInterval runs keepHomeCardsFresh with
// a short interval. It works the cards out at once, again after the table
// changes, and stops when its context ends.
func TestKeepHomeCardsFreshWorksThemOutEveryInterval(t *testing.T) {
t.Parallel()
const (
interval = 10 * time.Millisecond
maxWait = 5 * time.Second
)
table := emptyTable()
announce(table, "1.0.0.0/24", twoPrefixASN, testPeer(1), testRouteTime(0))
s := newServerWithTable(t, table)
ctx, cancel := context.WithCancel(t.Context())
stopped := make(chan struct{})
go func() {
s.keepHomeCardsFresh(ctx, interval)
close(stopped)
}()
newestRouteASNIs := func(asn int) bool {
cards := s.cards.Load()
return cards != nil && cards.NewestRoute != nil &&
cards.NewestRoute.OriginASN == asn
}
waitUntil(t, maxWait, func() bool { return newestRouteASNIs(twoPrefixASN) })
announce(table, "8.8.8.0/24", newestRouteASN, testPeer(2), testRouteTime(1))
waitUntil(t, maxWait, func() bool { return newestRouteASNIs(newestRouteASN) })
cancel()
select {
case <-stopped:
case <-time.After(maxWait):
t.Fatal("keepHomeCardsFresh did not stop when its context ended")
}
}
// TestStartedServerWorksOutTheCards starts a server and waits until its home
// page shows the cards.
func TestStartedServerWorksOutTheCards(t *testing.T) {
t.Parallel()
const maxWait = 5 * time.Second
s := newServerWithTable(t, newCardsTable())
err := s.Start(t.Context())
if err != nil {
t.Fatalf("Start: %v", err)
}
defer func() { _ = s.Stop(context.Background()) }()
waitUntil(t, maxWait, func() bool {
return !strings.Contains(get(t, s, "/").Body.String(), cardsNotReadyText)
})
card := homeCard(t, get(t, s, "/").Body.String(), "Newest Updated Route")
if !strings.Contains(card, "8.8.8.0/24") {
t.Errorf("the newest route is not the table's: %s", card)
}
}
// TestStartedServerWorksOutTheCardsUntilStop starts a server with a context
// that ends as soon as Start returns. It runs in a synctest bubble, whose clock
// is fake: time.Sleep there moves the clock on and returns at once. The server
// must work the cards out again a minute later, and no more after Stop. Each
// time it works them out, it keeps a new copy. Before the fix the end of
// Start's context stopped the goroutine that works them out.
func TestStartedServerWorksOutTheCardsUntilStop(t *testing.T) {
t.Parallel()
// The server is made outside the bubble. The streamer's metrics start a
// goroutine that never returns, and a bubble waits for each of its
// goroutines to return.
s := newServerWithTable(t, newCardsTable())
synctest.Test(t, func(t *testing.T) {
ctx, cancel := context.WithCancel(t.Context())
err := s.Start(ctx)
if err != nil {
t.Fatalf("Start: %v", err)
}
cancel()
// The bubble's clock moves only while each of its goroutines waits
// on the bubble, and the goroutine that serves HTTP waits on the
// network. The test sends no request, so it closes the HTTP server.
_ = s.srv.Close()
synctest.Wait()
started := s.cards.Load()
if started == nil {
t.Fatal("the server did not work out the cards when it started")
}
time.Sleep(homeCardsRefreshInterval)
synctest.Wait()
if s.cards.Load() == started {
t.Error("the server did not work out the cards again a minute " +
"after its start context ended")
}
err = s.Stop(t.Context())
if err != nil {
t.Fatalf("Stop: %v", err)
}
stopped := s.cards.Load()
time.Sleep(homeCardsRefreshInterval)
synctest.Wait()
if s.cards.Load() != stopped {
t.Error("the server worked out the cards again after Stop")
}
})
}
// waitUntil fails the test unless done reports true within maxWait.
func waitUntil(t *testing.T, maxWait time.Duration, done func() bool) {
t.Helper()
const pollInterval = 5 * time.Millisecond
deadline := time.Now().Add(maxWait)
for !done() {
if time.Now().After(deadline) {
t.Fatalf("the cards did not change within %s", maxWait)
}
time.Sleep(pollInterval)
}
}
internal/server/middleware.go
+19 -7
View File
@@ -16,6 +16,7 @@ import (
"time" "time"
"github.com/go-chi/chi/v5" "github.com/go-chi/chi/v5"
"github.com/go-chi/chi/v5/middleware"
) )
// responseWriter wraps http.ResponseWriter to capture the response // responseWriter wraps http.ResponseWriter to capture the response
@@ -53,6 +54,15 @@ func (rw *responseWriter) Header() http.Header {
return rw.ResponseWriter.Header() return rw.ResponseWriter.Header()
} }
// The @meta object JSONResponseMiddleware adds to a JSON answer: its key, and
// the keys of its fields.
const (
jsonKeyMeta = "@meta"
metaKeyTimeZone = "time_zone"
metaKeyAPIVersion = "api_version"
metaKeyExecutionTimeMs = "execution_time_ms"
)
// JSONResponseMiddleware is an HTTP middleware that wraps all JSON responses // JSONResponseMiddleware is an HTTP middleware that wraps all JSON responses
// with a @meta field containing execution metadata. The metadata includes the // with a @meta field containing execution metadata. The metadata includes the
// time zone (always UTC), API version, and request execution time in milliseconds. // time zone (always UTC), API version, and request execution time in milliseconds.
@@ -106,10 +116,10 @@ func JSONResponseMiddleware(next http.Handler) http.Handler {
} }
// Add @meta field // Add @meta field
originalResponse["@meta"] = map[string]any{ originalResponse[jsonKeyMeta] = map[string]any{
"time_zone": "UTC", metaKeyTimeZone: "UTC",
"api_version": 1, metaKeyAPIVersion: 1,
"execution_time": fmt.Sprintf("%d ms", execTime.Milliseconds()), metaKeyExecutionTimeMs: execTime.Milliseconds(),
} }
// Write the enhanced response // Write the enhanced response
@@ -145,7 +155,8 @@ func deadlinePassed(ctx context.Context) bool {
// has passed, it answers 504 with the error message "Request timeout". A 4xx // has passed, it answers 504 with the error message "Request timeout". A 4xx
// keeps its status, with the status's standard text as the message. For other // keeps its status, with the status's standard text as the message. For other
// errors, it returns "Internal server error", with status 500 if the original // errors, it returns "Internal server error", with status 500 if the original
// status was 200. // status was 200. A panic in the handler answers 500 with "Internal server
// error" too.
// //
// Non-JSON responses are passed through unchanged. // Non-JSON responses are passed through unchanged.
func JSONValidationMiddleware(next http.Handler) http.Handler { func JSONValidationMiddleware(next http.Handler) http.Handler {
@@ -157,8 +168,9 @@ func JSONValidationMiddleware(next http.Handler) http.Handler {
statusCode: http.StatusOK, statusCode: http.StatusOK,
} }
// Serve the request // Serve the request. Recoverer logs a panic in the handler and turns it
next.ServeHTTP(rw, r) // into a 500 without a body, which gets the JSON error body below.
middleware.Recoverer(next).ServeHTTP(rw, r)
// Check if it's meant to be a JSON response // Check if it's meant to be a JSON response
contentType := rw.Header().Get("Content-Type") contentType := rw.Header().Get("Content-Type")
internal/server/middleware_internal_test.go
+364 -1
View File
@@ -1,25 +1,40 @@
package server package server
import ( import (
"bytes"
"context"
"encoding/json" "encoding/json"
"io"
"log/slog"
"net"
"net/http" "net/http"
"net/http/httptest" "net/http/httptest"
"net/netip" "net/netip"
"os"
"regexp"
"strings" "strings"
"testing" "testing"
"time" "time"
"git.eeqj.de/sneak/routewatch/internal/config"
"git.eeqj.de/sneak/routewatch/internal/database"
"git.eeqj.de/sneak/routewatch/internal/logger"
"git.eeqj.de/sneak/routewatch/internal/metrics"
"git.eeqj.de/sneak/routewatch/internal/streamer"
) )
// The addresses the forwarding tests use: the client a proxy names in // The addresses the forwarding tests use: the client a proxy names in
// X-Forwarded-For, the range of trusted proxies, a peer inside it, and a peer // X-Forwarded-For, the range of trusted proxies, a peer inside it, and a peer
// outside it, which lies in testPrefixV4 so that an IP lookup of it finds a // outside it, which lies in testPrefixV4 so that an IP lookup of it finds a
// route. // route. reverseProxyPeer is a reverse proxy at a private address, as Docker
// gives one.
const ( const (
forwardedClient = "203.0.113.9" forwardedClient = "203.0.113.9"
trustedProxyRange = "10.0.0.0/8" trustedProxyRange = "10.0.0.0/8"
trustedPeer = "10.0.0.2:1234" trustedPeer = "10.0.0.2:1234"
untrustedPeerAddr = "192.0.2.1" untrustedPeerAddr = "192.0.2.1"
untrustedPeer = untrustedPeerAddr + ":1234" untrustedPeer = untrustedPeerAddr + ":1234"
reverseProxyPeer = "172.19.0.14:1234"
) )
// TestResponsesCarrySecurityHeaders checks that a page, a JSON route and a // TestResponsesCarrySecurityHeaders checks that a page, a JSON route and a
@@ -87,6 +102,219 @@ func TestJSONValidationKeeps4xxWithoutBody(t *testing.T) {
checkJSONErrorMessage(t, name, rec, http.StatusText(http.StatusBadRequest)) checkJSONErrorMessage(t, name, rec, http.StatusText(http.StatusBadRequest))
} }
// TestJSONAnswersPutNoUnitInAString sends a request to each JSON route and
// checks each answer. Its @meta.execution_time_ms is a number, and none of its
// strings is a number with a unit, such as "12 ms", "1.1 GB" or "1h18m34s"
// (https://git.eeqj.de/sneak/routewatch/issues/199). The health check answers
// 503, since the stream is not connected, and the last path matches no route.
func TestJSONAnswersPutNoUnitInAString(t *testing.T) {
t.Parallel()
numberWithUnit := regexp.MustCompile(
`^([0-9]+(\.[0-9]+)? ?(ns|µs|ms|s|m|h|B|kB|MB|GB|TB))+$`)
s := newServerWithTestRoutes(t)
s.resolver = &fakeResolver{}
s.SetASNFetcher(fixedIntervalASNFetcher{interval: time.Second})
paths := []string{
statusJSONPath,
statsPath,
healthCheckPath,
"/api/v1/ip/" + untrustedPeerAddr,
"/ip/" + untrustedPeerAddr,
asJSONPath,
"/api/v1/prefix/" + testPrefixV4,
"/api/v1/search/as?q=google",
"/api/v1/no-such-route",
}
for _, path := range paths {
rec := httptest.NewRecorder()
s.router.ServeHTTP(rec, httptest.NewRequestWithContext(
t.Context(), http.MethodGet, path, nil))
var answer map[string]any
err := json.Unmarshal(rec.Body.Bytes(), &answer)
if err != nil {
t.Errorf("%s: decoding the answer: %v; answer %s", path, err, rec.Body)
continue
}
meta, _ := answer["@meta"].(map[string]any)
if _, ok := meta["execution_time_ms"].(float64); !ok {
t.Errorf("%s: @meta.execution_time_ms is not a number: %s",
path, rec.Body)
}
for _, value := range stringsIn(answer) {
if numberWithUnit.MatchString(value) {
t.Errorf("%s: %q is a number with a unit in a string", path, value)
}
}
}
}
// TestJSONAnswersCarryMeta sends requests to every JSON route through the
// server's router, and checks that each answer carries @meta with its three
// fields. The answers include successes, and errors that the handlers, the
// router and the middleware write. Before the fix a panic in a handler, and a
// method /status.json or the health check does not take, answered with an
// empty body (https://git.eeqj.de/sneak/routewatch/issues/201).
func TestJSONAnswersCarryMeta(t *testing.T) {
t.Parallel()
// JSON routes whose handlers write nothing or panic, so that the JSON
// check on the route writes the error answer itself.
const (
writesNothingPath = "/writes-nothing.json"
panicsPath = "/panics.json"
)
s := newServerWithTestRoutes(t)
s.resolver = &fakeResolver{}
s.streamer.GetMetricsTracker().SetConnected(true)
s.router.Get(writesNothingPath, JSONValidationMiddleware(http.HandlerFunc(
func(http.ResponseWriter, *http.Request) {})).ServeHTTP)
s.router.Get(panicsPath, JSONValidationMiddleware(http.HandlerFunc(
func(http.ResponseWriter, *http.Request) { panic(errDiskIO) })).ServeHTTP)
tests := []struct {
method string
path string
wantCode int
}{
// Answers that succeed
{http.MethodGet, statusJSONPath, http.StatusOK},
{http.MethodGet, healthCheckPath, http.StatusOK},
{http.MethodGet, "/ip", http.StatusOK},
{http.MethodGet, ownAddressPath, http.StatusOK},
{http.MethodGet, "/ip/" + addrInTestPrefixV4, http.StatusOK},
{http.MethodGet, statsPath, http.StatusOK},
{http.MethodGet, "/api/v1/ip/" + addrInTestPrefixV4, http.StatusOK},
{http.MethodGet, asJSONPath, http.StatusOK},
{http.MethodGet, "/api/v1/prefix/" + testPrefixV4, http.StatusOK},
{http.MethodGet, "/api/v1/search/as?q=google", http.StatusOK},
// Errors the handlers write
{http.MethodGet, "/ip/no-such-host.example", http.StatusBadRequest},
{http.MethodGet, "/api/v1/ip/no-such-host.example", http.StatusBadRequest},
{http.MethodGet, "/api/v1/as/foo", http.StatusBadRequest},
{http.MethodGet, "/api/v1/as/0", http.StatusNotFound},
{http.MethodGet, "/api/v1/prefix/192.0.2.0/33", http.StatusBadRequest},
{http.MethodGet, "/api/v1/prefix/198.51.100.0/24", http.StatusNotFound},
{http.MethodGet, "/api/v1/search/as", http.StatusBadRequest},
// Errors the router and the middleware write
{http.MethodGet, "/api/v1/no-such-route", http.StatusNotFound},
{http.MethodGet, "/ip/192.0.2.1/nothing", http.StatusNotFound},
{http.MethodPost, statsPath, http.StatusMethodNotAllowed},
{http.MethodPost, ownAddressPath, http.StatusMethodNotAllowed},
{http.MethodPost, statusJSONPath, http.StatusMethodNotAllowed},
{http.MethodPut, healthCheckPath, http.StatusMethodNotAllowed},
{http.MethodGet, writesNothingPath, http.StatusInternalServerError},
{http.MethodGet, panicsPath, http.StatusInternalServerError},
}
for _, tt := range tests {
checkAnswerCarriesMeta(t, s, httptest.NewRequestWithContext(
t.Context(), tt.method, tt.path, nil), tt.wantCode)
}
// A request whose time limit has already passed answers 504. The handlers
// write the answers of the first five routes, and the JSON check on the
// route writes the last.
pastTimeLimit, cancel := context.WithDeadline(t.Context(), time.Now())
defer cancel()
for _, path := range []string{
statusJSONPath, statsPath, "/ip/" + addrInTestPrefixV4,
"/api/v1/ip/" + addrInTestPrefixV4, asJSONPath, writesNothingPath,
} {
checkAnswerCarriesMeta(t, s, httptest.NewRequestWithContext(
pastTimeLimit, http.MethodGet, path, nil), http.StatusGatewayTimeout)
}
// With a store whose reads fail, the health check answers 503, and the
// handlers of the other routes 500.
failing := New(failingDB{}, newTestTable(),
streamer.New(logger.New(), metrics.New()), logger.New(), &config.Config{})
failing.resolver = &fakeResolver{}
checkAnswerCarriesMeta(t, failing, httptest.NewRequestWithContext(
t.Context(), http.MethodGet, healthCheckPath, nil),
http.StatusServiceUnavailable)
for _, path := range []string{
statusJSONPath, statsPath, asJSONPath,
"/api/v1/ip/" + addrInTestPrefixV4, "/ip/" + addrInTestPrefixV4,
} {
checkAnswerCarriesMeta(t, failing, httptest.NewRequestWithContext(
t.Context(), http.MethodGet, path, nil), http.StatusInternalServerError)
}
}
// checkAnswerCarriesMeta serves req through the server's router, and checks
// that it answers wantCode with @meta and its three fields.
func checkAnswerCarriesMeta(
t *testing.T, s *Server, req *http.Request, wantCode int,
) {
t.Helper()
// The values README "API v1" gives the two fields that never change.
const (
wantTimeZone = "UTC"
wantAPIVersion = 1
)
name := req.Method + " " + req.URL.String()
rec := httptest.NewRecorder()
s.router.ServeHTTP(rec, req)
if rec.Code != wantCode {
t.Errorf("%s: status %d, want %d; body %s", name, rec.Code, wantCode, rec.Body)
}
var answer map[string]any
err := json.Unmarshal(rec.Body.Bytes(), &answer)
if err != nil {
t.Errorf("%s: decoding the answer %q: %v", name, rec.Body, err)
return
}
meta, _ := answer[jsonKeyMeta].(map[string]any)
_, timeIsNumber := meta[metaKeyExecutionTimeMs].(float64)
if meta[metaKeyTimeZone] != wantTimeZone ||
meta[metaKeyAPIVersion] != float64(wantAPIVersion) || !timeIsNumber {
t.Errorf("%s: @meta %v, want time_zone %q, api_version %d and a "+
"number in execution_time_ms", name, meta, wantTimeZone, wantAPIVersion)
}
}
// stringsIn returns every string in a decoded JSON value, at any depth.
func stringsIn(value any) []string {
var found []string
switch v := value.(type) {
case string:
found = append(found, v)
case []any:
for _, item := range v {
found = append(found, stringsIn(item)...)
}
case map[string]any:
for _, item := range v {
found = append(found, stringsIn(item)...)
}
}
return found
}
// TestPagesCarryTheCSPNonce checks that every script and style block of each // TestPagesCarryTheCSPNonce checks that every script and style block of each
// page carries the nonce its Content-Security-Policy header names, so that the // page carries the nonce its Content-Security-Policy header names, so that the
// browser runs and applies them, that each answer has a nonce of its own, and // browser runs and applies them, that each answer has a nonce of its own, and
@@ -183,6 +411,141 @@ func TestForwardedForFromUntrustedPeerIsIgnored(t *testing.T) {
} }
} }
// TestDefaultTrustedProxiesGiveTheForwardedClient builds the server from the
// configuration with TRUSTED_PROXIES unset. A request from a reverse proxy at
// a private address must be looked up by /ip/, shown on the home page's card
// and logged as the client its X-Forwarded-For names. A direct request from a
// public address with a forged X-Forwarded-For keeps its own address. Before
// the fix no proxy was trusted unless TRUSTED_PROXIES named it, so /ip/ gave
// the proxy's address (https://git.eeqj.de/sneak/routewatch/issues/198).
func TestDefaultTrustedProxiesGiveTheForwardedClient(t *testing.T) {
t.Setenv("PORT", "")
t.Setenv("XDG_DATA_HOME", "")
// t.Setenv puts the variable back when the test ends.
t.Setenv("TRUSTED_PROXIES", "")
err := os.Unsetenv("TRUSTED_PROXIES")
if err != nil {
t.Fatal(err)
}
cfg, err := config.New()
if err != nil {
t.Fatalf("config.New: %v", err)
}
var requestLog bytes.Buffer
s := newServerWithConfig(t, cfg, &requestLog)
tests := []struct {
name string
peer string
want string
}{
{"through a reverse proxy", reverseProxyPeer, forwardedClient},
{"direct from a public address", untrustedPeer, untrustedPeerAddr},
}
for _, tt := range tests {
requestLog.Reset()
get := func(path string) *httptest.ResponseRecorder {
req := httptest.NewRequestWithContext(t.Context(), http.MethodGet, path, nil)
req.RemoteAddr = tt.peer
req.Header.Set("X-Forwarded-For", forwardedClient)
rec := httptest.NewRecorder()
s.router.ServeHTTP(rec, req)
return rec
}
var answer struct {
Data IPLookupResponse `json:"data"`
}
rec := get("/ip/")
err := json.Unmarshal(rec.Body.Bytes(), &answer)
if err != nil {
t.Fatalf("%s: decoding the answer %q: %v", tt.name, rec.Body, err)
}
if answer.Data.Query != tt.want {
t.Errorf("%s: /ip/ looked up %q, want %q", tt.name, answer.Data.Query, tt.want)
}
page := get("/").Body.String()
if !strings.Contains(page, `data-client-ip="`+tt.want+`"`) {
t.Errorf("%s: the home page's card does not hold %s", tt.name, tt.want)
}
logged := loggedAddresses(requestLog.String())
if len(logged) == 0 {
t.Errorf("%s: the request log names no address", tt.name)
}
for _, addr := range logged {
if addr != tt.want {
t.Errorf("%s: the request log gives %s, want %s", tt.name, addr, tt.want)
}
}
}
}
// newServerWithConfig returns a server built with cfg, on a new database, that
// reads the routes of newTestTable and logs its requests as JSON to
// requestLog.
func newServerWithConfig(
t *testing.T, cfg *config.Config, requestLog io.Writer,
) *Server {
t.Helper()
cfg.StateDir = t.TempDir()
db, err := database.New(cfg, logger.New())
if err != nil {
t.Fatalf("database.New: %v", err)
}
t.Cleanup(func() { _ = db.Close() })
s := New(db, newTestTable(), streamer.New(logger.New(), metrics.New()),
&logger.Logger{Logger: slog.New(slog.NewJSONHandler(requestLog, nil))}, cfg)
s.resolver = &fakeResolver{}
return s
}
// loggedAddresses returns the address each line of the JSON request log
// names, without the port a direct request is logged with.
func loggedAddresses(requestLog string) []string {
var addrs []string
for line := range strings.Lines(requestLog) {
//nolint:tagliatelle // the field name is the key the request log uses
var entry struct {
RemoteAddr string `json:"remote_addr"`
}
err := json.Unmarshal([]byte(line), &entry)
if err != nil || entry.RemoteAddr == "" {
continue
}
addr, _, err := net.SplitHostPort(entry.RemoteAddr)
if err != nil {
addr = entry.RemoteAddr
}
addrs = append(addrs, addr)
}
return addrs
}
// TestRealIPMiddlewareTrustsOnlyConfiguredProxies checks which address // TestRealIPMiddlewareTrustsOnlyConfiguredProxies checks which address
// RealIPMiddleware gives a request: the TCP peer, unless the peer is a trusted // RealIPMiddleware gives a request: the TCP peer, unless the peer is a trusted
// proxy; then the last X-Forwarded-For entry that is not a trusted proxy, // proxy; then the last X-Forwarded-For entry that is not a trusted proxy,
internal/server/models.go
+74
View File
@@ -0,0 +1,74 @@
package server
import (
"errors"
"time"
)
var (
// ErrInvalidIP says that an IP address is malformed. The IP lookup returns
// it.
ErrInvalidIP = errors.New("invalid IP address")
// ErrNoRoute says that the routing table has no route for an address, a
// prefix or an AS. The lookups return it.
ErrNoRoute = errors.New("no route found")
)
// LiveRoute represents a route in the live routing table. JSON leaves out an
// empty AS path, next hop or time: the routing table in memory keeps them only
// for the two newest routes of each prefix and origin AS.
//
//nolint:tagliatelle // field names are the existing JSON API
type LiveRoute struct {
Prefix string `json:"prefix"`
MaskLength int `json:"mask_length"`
IPVersion int `json:"ip_version"`
OriginASN int `json:"origin_asn"`
PeerIP string `json:"peer_ip"`
ASPath []int `json:"as_path,omitempty"`
NextHop string `json:"next_hop,omitempty"`
LastUpdated time.Time `json:"last_updated,omitzero"`
}
// PrefixDistribution represents the distribution of prefixes by mask length
//
//nolint:tagliatelle // field names are the existing JSON API
type PrefixDistribution struct {
MaskLength int `json:"mask_length"`
Count int `json:"count"`
}
// IPInfo is what the IP lookup knows about the route of an address and that
// route's origin AS. A field is nil, null in JSON, when its value is not known.
// Every field is nil for an address with no route.
//
//nolint:tagliatelle // field names are the existing JSON API
type IPInfo struct {
// Netblock is the most specific prefix that holds the address.
Netblock *string `json:"netblock"`
MaskLength *int `json:"mask_length"`
IPVersion *int `json:"ip_version"`
NumPeers *int `json:"num_peers"`
// ASN is the origin AS that the most peers carry the netblock with.
ASN *int `json:"asn"`
// NumOrigins and OriginASNs are how many origin ASes peers carry the
// netblock with, and which, in AS number order.
NumOrigins *int `json:"num_origins"`
OriginASNs []int `json:"origin_asns"`
// The details the database holds about the AS in ASN.
ASName *string `json:"as_name"`
Handle *string `json:"handle"`
Description *string `json:"description"`
OrgName *string `json:"org_name"`
OrgID *string `json:"org_id"`
Address *string `json:"address"`
CountryCode *string `json:"country_code"`
AbuseEmail *string `json:"abuse_email"`
RIR *string `json:"rir"`
// FirstSeen is when RouteWatch first saw the netblock. LastSeen is the
// time of the netblock's newest kept route.
FirstSeen *time.Time `json:"first_seen"`
LastSeen *time.Time `json:"last_seen"`
// Indicates if WHOIS data needs refresh (not serialized)
NeedsWHOISRefresh bool `json:"-"`
}
internal/server/routes.go
+30 -7
View File
@@ -8,8 +8,8 @@ import (
"github.com/go-chi/chi/v5/middleware" "github.com/go-chi/chi/v5/middleware"
) )
// The error messages the JSON routes under /api/v1 and /ip answer with when no // The error messages the JSON routes answer with when no route matches the
// route matches the path, and when none takes the method. // path, and when the route does not take the method.
const ( const (
notFoundMessage = "Not found" notFoundMessage = "Not found"
methodNotAllowedMessage = "Method not allowed" methodNotAllowedMessage = "Method not allowed"
@@ -19,6 +19,13 @@ const (
jsonRoutesAllow = "GET, HEAD" jsonRoutesAllow = "GET, HEAD"
) )
// The paths of the status figures and the health check, the two JSON routes
// outside /api/v1 and /ip.
const (
statusJSONPath = "/status.json"
healthCheckPath = "/.well-known/healthcheck.json"
)
// setupRoutes configures the HTTP routes // setupRoutes configures the HTTP routes
func (s *Server) setupRoutes() { func (s *Server) setupRoutes() {
r := chi.NewRouter() r := chi.NewRouter()
@@ -32,6 +39,8 @@ func (s *Server) setupRoutes() {
r.Use(middleware.RequestID) r.Use(middleware.RequestID)
r.Use(RealIPMiddleware(s.trustedProxies)) r.Use(RealIPMiddleware(s.trustedProxies))
r.Use(RequestLoggerMiddleware(s.logger.Logger)) // Structured request logging r.Use(RequestLoggerMiddleware(s.logger.Logger)) // Structured request logging
// Recoverer answers a panic on a page. JSONValidationMiddleware recovers a
// panic on a JSON route itself, so that the answer is JSON with @meta.
r.Use(middleware.Recoverer) r.Use(middleware.Recoverer)
r.Use(JSONResponseMiddleware) r.Use(JSONResponseMiddleware)
// Timeout must come after JSONResponseMiddleware, which sends its answer // Timeout must come after JSONResponseMiddleware, which sends its answer
@@ -43,11 +52,6 @@ func (s *Server) setupRoutes() {
// Routes // Routes
r.Get("/", s.handleIndex()) r.Get("/", s.handleIndex())
r.Get("/status", s.handleStatusHTML()) r.Get("/status", s.handleStatusHTML())
r.Get("/status.json", JSONValidationMiddleware(s.handleStatusJSON()).ServeHTTP)
r.Get(
"/.well-known/healthcheck.json",
JSONValidationMiddleware(s.handleHealthCheck()).ServeHTTP,
)
// AS and prefix detail pages // AS and prefix detail pages
r.Get("/as/{asn}", s.handleASDetail()) r.Get("/as/{asn}", s.handleASDetail())
@@ -55,6 +59,10 @@ func (s *Server) setupRoutes() {
r.Get("/prefixlength/{length}", s.handlePrefixLength()) r.Get("/prefixlength/{length}", s.handlePrefixLength())
r.Get("/prefixlength6/{length}", s.handlePrefixLength6()) r.Get("/prefixlength6/{length}", s.handlePrefixLength6())
// The pages that list what two of the home page's cards count
r.Get("/bogons", s.handleBogons())
r.Get("/multiple-origins", s.handleMultipleOrigins())
// The JSON routes answer a path or method they do not take with the JSON // The JSON routes answer a path or method they do not take with the JSON
// error body, as they answer everything else. // error body, as they answer everything else.
jsonNotFound := func(w http.ResponseWriter, _ *http.Request) { jsonNotFound := func(w http.ResponseWriter, _ *http.Request) {
@@ -66,6 +74,21 @@ func (s *Server) setupRoutes() {
s.writeJSONError(w, http.StatusMethodNotAllowed, methodNotAllowedMessage) s.writeJSONError(w, http.StatusMethodNotAllowed, methodNotAllowedMessage)
} }
// The status figures and the health check. chi sets one 405 answer for a
// whole router. These two routes are on the pages' router, whose 405 stays
// chi's own. So each method they do not take is routed to the JSON 405
// here.
r.Get(statusJSONPath, JSONValidationMiddleware(s.handleStatusJSON()).ServeHTTP)
r.Get(healthCheckPath, JSONValidationMiddleware(s.handleHealthCheck()).ServeHTTP)
for _, method := range []string{
http.MethodPost, http.MethodPut, http.MethodPatch, http.MethodDelete,
http.MethodOptions, http.MethodTrace, http.MethodConnect,
} {
r.MethodFunc(method, statusJSONPath, jsonMethodNotAllowed)
r.MethodFunc(method, healthCheckPath, jsonMethodNotAllowed)
}
// IP info JSON endpoints (replaces old /ip redirect) // IP info JSON endpoints (replaces old /ip redirect)
r.Route("/ip", func(r chi.Router) { r.Route("/ip", func(r chi.Router) {
r.Use(JSONValidationMiddleware) r.Use(JSONValidationMiddleware)
internal/server/routes_internal_test.go
+13 -2
View File
@@ -17,7 +17,7 @@ func TestRoutesAnswerHEAD(t *testing.T) {
s := newServerWithTestRoutes(t) s := newServerWithTestRoutes(t)
for _, path := range []string{"/", statsPath, "/ip", "/ip/"} { for _, path := range []string{"/", statusJSONPath, statsPath, "/ip", "/ip/"} {
rec := httptest.NewRecorder() rec := httptest.NewRecorder()
s.router.ServeHTTP(rec, httptest.NewRequestWithContext( s.router.ServeHTTP(rec, httptest.NewRequestWithContext(
t.Context(), http.MethodHead, path, nil)) t.Context(), http.MethodHead, path, nil))
@@ -31,10 +31,13 @@ func TestRoutesAnswerHEAD(t *testing.T) {
// TestJSONRoutesAnswerUnmatchedRequestsWithJSONErrors sends a method the JSON // TestJSONRoutesAnswerUnmatchedRequestsWithJSONErrors sends a method the JSON
// routes do not take, and a path none of them matches, to each group of JSON // routes do not take, and a path none of them matches, to each group of JSON
// routes through the server's router. Each must answer with the JSON error // routes through the server's router. It also sends a method they do not take
// to /status.json and the health check. Each must answer with the JSON error
// body, and a 405 must name the methods the routes take in its Allow header. // body, and a 405 must name the methods the routes take in its Allow header.
// Before the fix a 405 said "Internal server error" and a 404 was chi's plain // Before the fix a 405 said "Internal server error" and a 404 was chi's plain
// text answer (https://git.eeqj.de/sneak/routewatch/issues/111). // text answer (https://git.eeqj.de/sneak/routewatch/issues/111).
// /status.json and the health check gave chi's empty 405
// (https://git.eeqj.de/sneak/routewatch/issues/201).
func TestJSONRoutesAnswerUnmatchedRequestsWithJSONErrors(t *testing.T) { func TestJSONRoutesAnswerUnmatchedRequestsWithJSONErrors(t *testing.T) {
t.Parallel() t.Parallel()
@@ -48,6 +51,14 @@ func TestJSONRoutesAnswerUnmatchedRequestsWithJSONErrors(t *testing.T) {
}{ }{
{http.MethodPost, statsPath, http.StatusMethodNotAllowed, methodNotAllowedMessage}, {http.MethodPost, statsPath, http.StatusMethodNotAllowed, methodNotAllowedMessage},
{http.MethodPost, "/ip/", http.StatusMethodNotAllowed, methodNotAllowedMessage}, {http.MethodPost, "/ip/", http.StatusMethodNotAllowed, methodNotAllowedMessage},
{
http.MethodDelete, statusJSONPath,
http.StatusMethodNotAllowed, methodNotAllowedMessage,
},
{
http.MethodOptions, healthCheckPath,
http.StatusMethodNotAllowed, methodNotAllowedMessage,
},
{http.MethodGet, "/api/v1/nothing", http.StatusNotFound, notFoundMessage}, {http.MethodGet, "/api/v1/nothing", http.StatusNotFound, notFoundMessage},
{http.MethodGet, "/ip/192.0.2.1/nothing", http.StatusNotFound, notFoundMessage}, {http.MethodGet, "/ip/192.0.2.1/nothing", http.StatusNotFound, notFoundMessage},
} }
internal/server/server.go
+39 -4
View File
@@ -1,5 +1,7 @@
// Package server serves RouteWatch's web pages and JSON API: // Package server serves RouteWatch's web pages and JSON API:
// - the home page, the status figures and the health check // - the home page, the status figures and the health check
// - the pages that list the bogon announcements and the prefixes announced
// by more than one origin AS
// - the AS, prefix and prefix length pages, and the JSON answers behind them // - the AS, prefix and prefix length pages, and the JSON answers behind them
// - the IP lookups // - the IP lookups
// - the AS name search // - the AS name search
@@ -17,6 +19,7 @@ import (
"net/http" "net/http"
"net/netip" "net/netip"
"strconv" "strconv"
"sync/atomic"
"time" "time"
"git.eeqj.de/sneak/routewatch/internal/blocklist" "git.eeqj.de/sneak/routewatch/internal/blocklist"
@@ -75,6 +78,16 @@ type Server struct {
// resolver makes the IP lookup's DNS lookups: net.DefaultResolver, or a // resolver makes the IP lookup's DNS lookups: net.DefaultResolver, or a
// fake in tests // fake in tests
resolver dnsResolver resolver dnsResolver
// cards is the copy of the home page's cards that the pages serve, nil
// until the server first works them out
cards atomic.Pointer[homeCards]
// stopCards ends the goroutine that works out the cards, and
// cardsStopped is closed once it has returned. Both are nil until Start
// starts the goroutine.
stopCards context.CancelFunc
cardsStopped chan struct{}
} }
// New creates a new HTTP server that reads routes from table, the routing // New creates a new HTTP server that reads routes from table, the routing
@@ -103,7 +116,9 @@ func New(
// Start listens on the configured port and serves HTTP in the background. // Start listens on the configured port and serves HTTP in the background.
// It returns an error if it cannot listen, for example because the port is // It returns an error if it cannot listen, for example because the port is
// already in use. ctx bounds only opening the port, not serving. // already in use. ctx bounds opening the port, not serving. Once listening,
// the server works out the home page's cards every minute until Stop: the end
// of ctx does not stop it.
func (s *Server) Start(ctx context.Context) error { func (s *Server) Start(ctx context.Context) error {
const ( const (
network = "tcp" network = "tcp"
@@ -140,12 +155,22 @@ func (s *Server) Start(ctx context.Context) error {
} }
}() }()
ctx, s.stopCards = context.WithCancel(context.WithoutCancel(ctx))
s.cardsStopped = make(chan struct{})
go func() {
defer close(s.cardsStopped)
s.keepHomeCardsFresh(ctx, homeCardsRefreshInterval)
}()
return nil return nil
} }
// Stop stops the HTTP server. It waits, until ctx ends, for the requests in // Stop stops the goroutine that works out the home page's cards, and waits
// progress to finish, and then closes the connections of those still running // until it has returned or ctx ends. It then stops the HTTP server. It waits,
// and returns ctx's error. // until ctx ends, for the requests in progress to finish, and then closes the
// connections of those still running and returns ctx's error.
func (s *Server) Stop(ctx context.Context) error { func (s *Server) Stop(ctx context.Context) error {
if s.srv == nil { if s.srv == nil {
return nil return nil
@@ -153,6 +178,16 @@ func (s *Server) Stop(ctx context.Context) error {
s.logger.Info("Stopping HTTP server") s.logger.Info("Stopping HTTP server")
// Start returns before it starts the goroutine when it cannot listen.
if s.stopCards != nil {
s.stopCards()
select {
case <-s.cardsStopped:
case <-ctx.Done():
}
}
err := s.srv.Shutdown(ctx) err := s.srv.Shutdown(ctx)
if err != nil { if err != nil {
return errors.Join(err, s.srv.Close()) return errors.Join(err, s.srv.Close())
internal/templates/bogons.html
+170
View File
@@ -0,0 +1,170 @@
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<title>Bogon Announcements - RouteWatch</title>
<style nonce="{{.CSPNonce}}">
* {
box-sizing: border-box;
}
body {
font-family: -apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, sans-serif;
margin: 0;
padding: 0;
background: #f5f5f5;
}
/* Navbar styles */
.navbar {
background: #2c3e50;
padding: 0 20px;
display: flex;
align-items: center;
justify-content: space-between;
height: 56px;
box-shadow: 0 2px 4px rgba(0,0,0,0.1);
}
.navbar-brand {
display: flex;
align-items: center;
gap: 8px;
font-size: 18px;
font-weight: 600;
}
.navbar-brand a {
color: white;
text-decoration: none;
}
.navbar-brand a:hover {
color: #ecf0f1;
}
.navbar-brand .by {
font-weight: normal;
color: #95a5a6;
font-size: 14px;
}
.navbar-brand .author {
color: #3498db;
font-weight: normal;
}
.navbar-brand .author:hover {
text-decoration: underline;
}
.navbar-links {
display: flex;
gap: 20px;
}
.navbar-links a {
color: #ecf0f1;
text-decoration: none;
font-size: 14px;
padding: 8px 12px;
border-radius: 4px;
transition: background-color 0.2s;
}
.navbar-links a:hover {
background: rgba(255,255,255,0.1);
}
/* Main content */
.main-content {
max-width: 1200px;
margin: 0 auto;
padding: 20px;
}
h1 {
color: #333;
margin-bottom: 10px;
}
.subtitle {
color: #666;
margin-bottom: 30px;
}
table {
width: 100%;
border-collapse: collapse;
background: white;
box-shadow: 0 2px 4px rgba(0,0,0,0.1);
border-radius: 8px;
overflow: hidden;
}
th {
background: #f8f9fa;
padding: 12px;
text-align: left;
font-weight: 600;
color: #333;
border-bottom: 2px solid #dee2e6;
}
td {
padding: 12px;
border-bottom: 1px solid #eee;
}
tr:last-child td {
border-bottom: none;
}
tr:hover {
background: #f8f9fa;
}
a {
color: #0066cc;
text-decoration: none;
}
a:hover {
text-decoration: underline;
}
.prefix-link {
font-family: 'SF Mono', Monaco, 'Cascadia Mono', 'Roboto Mono', Consolas, 'Courier New', monospace;
}
</style>
</head>
<body>
<nav class="navbar">
<div class="navbar-brand">
<a href="/">routewatch</a>
<span class="by">by</span>
<a href="https://sneak.berlin" class="author">@sneak</a>
</div>
<div class="navbar-links">
<a href="/">Home</a>
<a href="/status">Status</a>
</div>
</nav>
<main class="main-content">
<h1>Bogon Announcements</h1>
{{ with .Cards }}
<p class="subtitle">
Bogon announcements: {{ .BogonCount }}.
A bogon announcement is a route that announces a reserved prefix, or that comes from a reserved AS number.
{{ if lt (len .Bogons) .BogonCount }}The first {{ len .Bogons }} are listed, in prefix order.{{ end }}
The list is worked out once a minute.
</p>
<table>
<thead>
<tr>
<th>Prefix</th>
<th>Origin AS</th>
<th>Peers</th>
<th>Why it is a bogon</th>
</tr>
</thead>
<tbody>
{{ range .Bogons }}
<tr>
<td><a href="{{ prefixURL .Prefix.String }}" class="prefix-link">{{ .Prefix }}</a></td>
<td>{{ if .OriginASN }}<a href="/as/{{ .OriginASN }}">AS{{ .OriginASN }}</a>{{ else }}No known origin{{ end }}</td>
<td>{{ .Peers }}</td>
<td>{{ range $i, $reason := .Reasons }}{{ if $i }}; {{ end }}{{ $reason }}{{ end }}</td>
</tr>
{{ end }}
</tbody>
</table>
{{ else }}
<p class="subtitle">The list is being worked out. Reload the page in a minute.</p>
{{ end }}
</main>
</body>
</html>
internal/templates/index.html
+154 -11
View File
@@ -276,6 +276,52 @@
margin-top: 12px; margin-top: 12px;
} }
/* Cards with figures of the routing table, worked out once a minute */
#table-cards p,
#well-known p {
margin: 0 0 8px 0;
font-size: 14px;
}
#table-cards a,
#well-known a {
color: #3498db;
text-decoration: none;
}
#table-cards a:hover,
#well-known a:hover {
text-decoration: underline;
}
#table-cards .card-figure {
font-size: 32px;
font-weight: bold;
}
#table-cards .card-detail,
#table-cards .card-time {
color: #7f8c8d;
font-size: 12px;
}
#table-cards .prefix {
font-family: 'SF Mono', Monaco, 'Cascadia Mono', 'Roboto Mono', Consolas, 'Courier New', monospace;
}
#well-known {
margin-bottom: 40px;
}
#well-known table {
width: 100%;
border-collapse: collapse;
font-size: 14px;
}
#well-known th,
#well-known td {
text-align: left;
padding: 8px 4px;
border-bottom: 1px solid #eee;
}
#well-known th {
color: #7f8c8d;
font-weight: 600;
}
/* Footer */ /* Footer */
.footer { .footer {
margin-top: 40px; margin-top: 40px;
@@ -381,6 +427,109 @@
</div> </div>
</div> </div>
{{ with .Cards }}
<div class="search-section" id="table-cards">
<div class="search-card">
<h2>Newest Updated Route</h2>
{{ with .NewestRoute }}
<p><a href="{{ prefixURL .Prefix.String }}" class="prefix">{{ .Prefix }}</a> from {{ if .OriginASN }}<a href="/as/{{ .OriginASN }}">AS{{ .OriginASN }}</a>{{ else }}no known origin{{ end }}</p>
<p class="card-time">{{ .Time.UTC.Format "2006-01-02 15:04:05" }} UTC, {{ timeSince .Time }}</p>
{{ else }}
<p>The routing table keeps no route yet.</p>
{{ end }}
</div>
<div class="search-card">
<h2>Oldest Updated Route</h2>
{{ with .OldestRoute }}
<p><a href="{{ prefixURL .Prefix.String }}" class="prefix">{{ .Prefix }}</a> from {{ if .OriginASN }}<a href="/as/{{ .OriginASN }}">AS{{ .OriginASN }}</a>{{ else }}no known origin{{ end }}</p>
<p class="card-time">{{ .Time.UTC.Format "2006-01-02 15:04:05" }} UTC, {{ timeSince .Time }}</p>
{{ else }}
<p>The routing table keeps no route yet.</p>
{{ end }}
</div>
<div class="search-card">
<h2>Newest Updated AS</h2>
{{ with .NewestAS }}
<p><a href="/as/{{ .ASN }}">AS{{ .ASN }}</a> {{ .Handle }}</p>
<p class="card-detail">{{ .Description }}</p>
<p class="card-time">Last seen {{ .LastSeen.UTC.Format "2006-01-02 15:04:05" }} UTC, {{ timeSince .LastSeen }}</p>
{{ else }}
<p>No AS has been seen yet.</p>
{{ end }}
</div>
<div class="search-card">
<h2>Oldest Updated AS</h2>
{{ with .OldestAS }}
<p><a href="/as/{{ .ASN }}">AS{{ .ASN }}</a> {{ .Handle }}</p>
<p class="card-detail">{{ .Description }}</p>
<p class="card-time">Last seen {{ .LastSeen.UTC.Format "2006-01-02 15:04:05" }} UTC, {{ timeSince .LastSeen }}</p>
{{ else }}
<p>No AS has been seen yet.</p>
{{ end }}
</div>
<div class="search-card">
<h2>AS with the Most Prefixes</h2>
{{ if .MostPrefixes }}
<p><a href="/as/{{ .MostPrefixesASN }}">AS{{ .MostPrefixesASN }}</a></p>
<p>Prefixes it originates: {{ .MostPrefixes }}</p>
{{ else }}
<p>No AS originates a prefix yet.</p>
{{ end }}
</div>
<div class="search-card">
<h2>ASes with the Fewest Prefixes</h2>
<p>ASes that originate a single prefix: {{ .SinglePrefixASes }}</p>
{{ with .NewestSinglePrefix }}
<p>The most recently updated of them: <a href="/as/{{ .OriginASN }}">AS{{ .OriginASN }}</a>, with <a href="{{ prefixURL .Prefix.String }}" class="prefix">{{ .Prefix }}</a></p>
<p class="card-time">{{ .Time.UTC.Format "2006-01-02 15:04:05" }} UTC, {{ timeSince .Time }}</p>
{{ end }}
</div>
<div class="search-card">
<h2>Bogon Announcements</h2>
<p class="card-figure"><a href="/bogons">{{ .BogonCount }}</a></p>
<p>Routes that announce a reserved prefix, or that come from a reserved AS number</p>
</div>
<div class="search-card">
<h2>Prefixes with Several Origins</h2>
<p class="card-figure"><a href="/multiple-origins">{{ .MultipleOriginCount }}</a></p>
<p>Prefixes announced by more than one origin AS</p>
</div>
</div>
<div class="search-card" id="well-known">
<h2>Well-Known Networks</h2>
<table>
<thead>
<tr>
<th>Network</th>
<th>Prefixes</th>
<th>Last Update</th>
</tr>
</thead>
<tbody>
{{ range .WellKnownNetworks }}
<tr>
<td><a href="/as/{{ .ASN }}">AS{{ .ASN }}</a> {{ .Name }}</td>
<td>{{ .Prefixes }}</td>
<td>{{ with .Newest }}{{ timeSince . }}{{ else }}no route{{ end }}</td>
</tr>
{{ end }}
</tbody>
</table>
</div>
{{ else }}
<div class="search-card" id="well-known">
<p>The figures of the routing table are being worked out. Reload the page in a minute.</p>
</div>
{{ end }}
<div class="search-card" id="ip-api"> <div class="search-card" id="ip-api">
<h2>IP Information API</h2> <h2>IP Information API</h2>
<p>The API answers what RouteWatch knows about an IP address, such as its AS number and country. <code>/ip/</code> with no address answers for the caller's own address.</p> <p>The API answers what RouteWatch knows about an IP address, such as its AS number and country. <code>/ip/</code> with no address answers for the caller's own address.</p>
@@ -565,7 +714,8 @@
}); });
}); });
// IP lookup // IP lookup: show the whole JSON answer, its @meta and an error answer
// included, as the IP information API card does.
document.getElementById('ip-form').addEventListener('submit', function(e) { document.getElementById('ip-form').addEventListener('submit', function(e) {
e.preventDefault(); e.preventDefault();
const ip = document.getElementById('ip-input').value.trim(); const ip = document.getElementById('ip-input').value.trim();
@@ -581,18 +731,11 @@
fetch('/ip/' + encodeURIComponent(ip)) fetch('/ip/' + encodeURIComponent(ip))
.then(response => response.json()) .then(response => response.json())
.then(response => { .then(answer => {
contentEl.classList.remove('loading'); contentEl.className = '';
if (response.status === 'error') { contentEl.textContent = JSON.stringify(answer, null, 2);
contentEl.className = 'error';
contentEl.textContent = 'Error: ' + response.error.msg;
} else {
contentEl.className = '';
contentEl.textContent = JSON.stringify(response.data, null, 2);
}
}) })
.catch(error => { .catch(error => {
contentEl.classList.remove('loading');
contentEl.className = 'error'; contentEl.className = 'error';
contentEl.textContent = 'Error: ' + error.message; contentEl.textContent = 'Error: ' + error.message;
}); });
internal/templates/multiple_origins.html
+182
View File
@@ -0,0 +1,182 @@
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<title>Prefixes with Several Origins - RouteWatch</title>
<style nonce="{{.CSPNonce}}">
* {
box-sizing: border-box;
}
body {
font-family: -apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, sans-serif;
margin: 0;
padding: 0;
background: #f5f5f5;
}
/* Navbar styles */
.navbar {
background: #2c3e50;
padding: 0 20px;
display: flex;
align-items: center;
justify-content: space-between;
height: 56px;
box-shadow: 0 2px 4px rgba(0,0,0,0.1);
}
.navbar-brand {
display: flex;
align-items: center;
gap: 8px;
font-size: 18px;
font-weight: 600;
}
.navbar-brand a {
color: white;
text-decoration: none;
}
.navbar-brand a:hover {
color: #ecf0f1;
}
.navbar-brand .by {
font-weight: normal;
color: #95a5a6;
font-size: 14px;
}
.navbar-brand .author {
color: #3498db;
font-weight: normal;
}
.navbar-brand .author:hover {
text-decoration: underline;
}
.navbar-links {
display: flex;
gap: 20px;
}
.navbar-links a {
color: #ecf0f1;
text-decoration: none;
font-size: 14px;
padding: 8px 12px;
border-radius: 4px;
transition: background-color 0.2s;
}
.navbar-links a:hover {
background: rgba(255,255,255,0.1);
}
/* Main content */
.main-content {
max-width: 1200px;
margin: 0 auto;
padding: 20px;
}
h1 {
color: #333;
margin-bottom: 10px;
}
.subtitle {
color: #666;
margin-bottom: 30px;
}
table {
width: 100%;
border-collapse: collapse;
background: white;
box-shadow: 0 2px 4px rgba(0,0,0,0.1);
border-radius: 8px;
overflow: hidden;
}
th {
background: #f8f9fa;
padding: 12px;
text-align: left;
font-weight: 600;
color: #333;
border-bottom: 2px solid #dee2e6;
}
td {
padding: 12px;
border-bottom: 1px solid #eee;
vertical-align: top;
}
tr:last-child td {
border-bottom: none;
}
tr:hover {
background: #f8f9fa;
}
a {
color: #0066cc;
text-decoration: none;
}
a:hover {
text-decoration: underline;
}
.prefix-link {
font-family: 'SF Mono', Monaco, 'Cascadia Mono', 'Roboto Mono', Consolas, 'Courier New', monospace;
}
.origins {
list-style: none;
margin: 0;
padding: 0;
}
.peers {
color: #666;
font-size: 0.9em;
}
</style>
</head>
<body>
<nav class="navbar">
<div class="navbar-brand">
<a href="/">routewatch</a>
<span class="by">by</span>
<a href="https://sneak.berlin" class="author">@sneak</a>
</div>
<div class="navbar-links">
<a href="/">Home</a>
<a href="/status">Status</a>
</div>
</nav>
<main class="main-content">
<h1>Prefixes with Several Origin ASes</h1>
{{ with .Cards }}
<p class="subtitle">
Prefixes announced by more than one origin AS: {{ .MultipleOriginCount }}.
Such a prefix can be a hijack.
{{ if lt (len .MultipleOrigins) .MultipleOriginCount }}The first {{ len .MultipleOrigins }} are listed, in prefix order.{{ end }}
The list is worked out once a minute.
</p>
<table>
<thead>
<tr>
<th>Prefix</th>
<th>Origin ASes</th>
</tr>
</thead>
<tbody>
{{ range .MultipleOrigins }}
<tr>
<td><a href="{{ prefixURL .Prefix.String }}" class="prefix-link">{{ .Prefix }}</a></td>
<td>
<ul class="origins">
{{ range .Origins }}
<li>{{ if .ASN }}<a href="/as/{{ .ASN }}">AS{{ .ASN }}</a>{{ else }}No known origin{{ end }} <span class="peers">{{ .Peers }} {{ if eq .Peers 1 }}peer{{ else }}peers{{ end }}</span></li>
{{ end }}
</ul>
</td>
</tr>
{{ end }}
</tbody>
</table>
{{ else }}
<p class="subtitle">The list is being worked out. Reload the page in a minute.</p>
{{ end }}
</main>
</body>
</html>
internal/templates/status.html
+20 -5
View File
@@ -401,6 +401,8 @@
const REFRESH_PERIOD_MS = 2000; const REFRESH_PERIOD_MS = 2000;
const STALE_AFTER_REFRESHES = 3; const STALE_AFTER_REFRESHES = 3;
const MS_PER_SECOND = 1000; const MS_PER_SECOND = 1000;
const SECONDS_PER_MINUTE = 60;
const SECONDS_PER_HOUR = 3600;
const SPINNER_FRAMES = ['⠋', '⠙', '⠹', '⠸', '⠼', '⠴', '⠦', '⠧', '⠇', '⠏']; const SPINNER_FRAMES = ['⠋', '⠙', '⠹', '⠸', '⠼', '⠴', '⠦', '⠧', '⠇', '⠏'];
const SPINNER_COLOURS = ['#e74c3c', '#e67e22', '#f1c40f', '#27ae60', '#3498db', '#9b59b6']; const SPINNER_COLOURS = ['#e74c3c', '#e67e22', '#f1c40f', '#27ae60', '#3498db', '#9b59b6'];
@@ -443,7 +445,18 @@
function formatNumber(num) { function formatNumber(num) {
return num.toLocaleString(); return num.toLocaleString();
} }
// formatDuration writes a number of seconds in hours, minutes and
// seconds, such as "1h 18m 34s" or "18.2s", with at most one decimal.
function formatDuration(totalSeconds) {
const hours = Math.floor(totalSeconds / SECONDS_PER_HOUR);
const minutes = Math.floor(totalSeconds % SECONDS_PER_HOUR / SECONDS_PER_MINUTE);
const seconds = parseFloat((totalSeconds % SECONDS_PER_MINUTE).toFixed(1));
if (hours > 0) return hours + 'h ' + minutes + 'm ' + seconds + 's';
if (minutes > 0) return minutes + 'm ' + seconds + 's';
return seconds + 's';
}
function formatProcessingTime(ms) { function formatProcessingTime(ms) {
if (ms < 0.001) { if (ms < 0.001) {
return (ms * 1000).toFixed(0) + ' µs'; return (ms * 1000).toFixed(0) + ' µs';
@@ -619,11 +632,13 @@
noteMessageCount(data.total_messages); noteMessageCount(data.total_messages);
// Update all metrics // Update all metrics
document.getElementById('uptime').textContent = data.uptime; document.getElementById('uptime').textContent = formatDuration(data.uptime_seconds);
document.getElementById('go_version').textContent = data.go_version; document.getElementById('go_version').textContent = data.go_version;
document.getElementById('goroutines').textContent = formatNumber(data.goroutines); document.getElementById('goroutines').textContent = formatNumber(data.goroutines);
document.getElementById('memory_usage').textContent = data.memory_usage; document.getElementById('memory_usage').textContent = formatBytes(data.memory_usage_bytes);
document.getElementById('connection_duration').textContent = data.connection_duration; // connection_duration_seconds is null while the stream is not connected.
document.getElementById('connection_duration').textContent =
data.connection_duration_seconds === null ? 'disconnected' : formatDuration(data.connection_duration_seconds);
document.getElementById('reconnect_count').textContent = formatNumber(data.reconnect_count); document.getElementById('reconnect_count').textContent = formatNumber(data.reconnect_count);
document.getElementById('total_messages').textContent = formatNumber(data.total_messages); document.getElementById('total_messages').textContent = formatNumber(data.total_messages);
document.getElementById('messages_per_sec').textContent = formatNumber(Math.round(data.messages_per_sec)); document.getElementById('messages_per_sec').textContent = formatNumber(Math.round(data.messages_per_sec));
@@ -672,7 +687,7 @@
const errorsEl = document.getElementById('whois_errors'); const errorsEl = document.getElementById('whois_errors');
errorsEl.textContent = formatNumber(data.whois_stats.errors_last_hour); errorsEl.textContent = formatNumber(data.whois_stats.errors_last_hour);
errorsEl.className = 'metric-value' + (data.whois_stats.errors_last_hour > 0 ? ' disconnected' : ''); errorsEl.className = 'metric-value' + (data.whois_stats.errors_last_hour > 0 ? ' disconnected' : '');
document.getElementById('whois_interval').textContent = data.whois_stats.current_interval; document.getElementById('whois_interval').textContent = formatDuration(data.whois_stats.current_interval_seconds);
} }
// Update handler stats // Update handler stats
internal/templates/templates.go
+34
View File
@@ -27,6 +27,12 @@ var prefixDetailHTML string
//go:embed prefix_length.html //go:embed prefix_length.html
var prefixLengthHTML string var prefixLengthHTML string
//go:embed bogons.html
var bogonsHTML string
//go:embed multiple_origins.html
var multipleOriginsHTML string
// Templates contains all parsed templates // Templates contains all parsed templates
type Templates struct { type Templates struct {
// Index is the template for the home page // Index is the template for the home page
@@ -39,6 +45,11 @@ type Templates struct {
PrefixDetail *template.Template PrefixDetail *template.Template
// PrefixLength is the template for displaying prefixes by length // PrefixLength is the template for displaying prefixes by length
PrefixLength *template.Template PrefixLength *template.Template
// Bogons is the template of the page that lists the bogon announcements
Bogons *template.Template
// MultipleOrigins is the template of the page that lists the prefixes
// announced by more than one origin AS
MultipleOrigins *template.Template
} }
var ( var (
@@ -116,6 +127,18 @@ func initTemplates() {
if err != nil { if err != nil {
panic("failed to parse prefix length template: " + err.Error()) panic("failed to parse prefix length template: " + err.Error())
} }
defaultTemplates.Bogons, err = template.New("bogons").
Funcs(funcs).Parse(bogonsHTML)
if err != nil {
panic("failed to parse bogons template: " + err.Error())
}
defaultTemplates.MultipleOrigins, err = template.New("multipleOrigins").
Funcs(funcs).Parse(multipleOriginsHTML)
if err != nil {
panic("failed to parse multiple origins template: " + err.Error())
}
} }
// Get returns the singleton Templates instance // Get returns the singleton Templates instance
@@ -149,3 +172,14 @@ func PrefixDetailTemplate() *template.Template {
func PrefixLengthTemplate() *template.Template { func PrefixLengthTemplate() *template.Template {
return Get().PrefixLength return Get().PrefixLength
} }
// BogonsTemplate returns the parsed template of the bogon announcements page
func BogonsTemplate() *template.Template {
return Get().Bogons
}
// MultipleOriginsTemplate returns the parsed template of the page of prefixes
// announced by more than one origin AS
func MultipleOriginsTemplate() *template.Template {
return Get().MultipleOrigins
}
internal/templates/templates_test.go
+11 -6
View File
@@ -7,17 +7,22 @@ import (
"testing" "testing"
"git.eeqj.de/sneak/routewatch/internal/database" "git.eeqj.de/sneak/routewatch/internal/database"
"git.eeqj.de/sneak/routewatch/internal/server"
"git.eeqj.de/sneak/routewatch/internal/templates" "git.eeqj.de/sneak/routewatch/internal/templates"
) )
// renderPage renders a page with the data every page needs, and the visitor's // renderPage renders a page with the data every page needs, and the visitor's
// address, which the home page also needs. // address and the routing table's cards, which the home page also needs. The
// cards are left out, as before the server first works them out.
func renderPage(t *testing.T, tmpl *template.Template) string { func renderPage(t *testing.T, tmpl *template.Template) string {
t.Helper() t.Helper()
var page bytes.Buffer var page bytes.Buffer
err := tmpl.Execute(&page, struct{ CSPNonce, ClientIP string }{}) err := tmpl.Execute(&page, struct {
CSPNonce, ClientIP string
Cards any
}{})
if err != nil { if err != nil {
t.Fatalf("rendering the page: %v", err) t.Fatalf("rendering the page: %v", err)
} }
@@ -114,14 +119,14 @@ func TestASDetailPrefixListsFold(t *testing.T) {
prefixV6 = "2001:db8::/32" prefixV6 = "2001:db8::/32"
) )
ipv4 := []database.LiveRoute{{Prefix: prefixV4}} ipv4 := []server.LiveRoute{{Prefix: prefixV4}}
ipv6 := []database.LiveRoute{{Prefix: prefixV6}} ipv6 := []server.LiveRoute{{Prefix: prefixV6}}
// The fields the AS page handler gives the template. // The fields the AS page handler gives the template.
data := struct { data := struct {
ASN *database.ASN ASN *database.ASN
IPv4Prefixes []database.LiveRoute IPv4Prefixes []server.LiveRoute
IPv6Prefixes []database.LiveRoute IPv6Prefixes []server.LiveRoute
TotalCount int TotalCount int
IPv4Count int IPv4Count int
IPv6Count int IPv6Count int