From ebb7bf4d7151287bc009228965c660a7df8bee9a Mon Sep 17 00:00:00 2001 From: sneak Date: Tue, 27 Oct 2026 21:03:44 +0000 Subject: [PATCH] Rewrite the README sentence by sentence (closes #157) The old text failed the issue's prose rules: a 60-word sentence on the content security policy, a markdown table of settings, comma-chained lists of batch sizes and error codes, undefined jargon and an edge case in the opening. The new text leads with what RouteWatch is, why it exists and how to run it, then the pages, the API and the blocklists, then the settings, memory and upaas, and last the design and the development workflow. Every statement was checked against the code on next. The PR body lists where the old text disagreed with the code. Model: fable-5-1 --- README.md | 1180 +++++++++++++++++++++++++++-------------------------- 1 file changed, 602 insertions(+), 578 deletions(-) diff --git a/README.md b/README.md index 5f003b7..ff57aec 100644 --- a/README.md +++ b/README.md @@ -1,225 +1,191 @@ # RouteWatch -RouteWatch is an MIT-licensed Go daemon by @sneak that monitors the BGP routing -table in real time. It works like this: +RouteWatch is an MIT-licensed Go daemon by @sneak that keeps a live copy of the +internet's BGP routing table and answers questions about IP addresses over HTTP. -- It streams BGP UPDATE messages from the RIPE RIS Live service. -- It keeps the whole routing table in memory, with a copy in SQLite. -- It serves web pages and a JSON API for looking routes up. +BGP is the protocol by which the networks of the internet tell each other which +addresses they can reach. Each network is an autonomous system, an AS, with a +number such as `AS15169`. A prefix is a block of addresses, such as +`192.0.2.0/24`. A route says which AS announces a prefix and through which ASes +the announcement came. The AS that announces the prefix is its origin AS. The +list of ASes is the AS path. + +RouteWatch does three things: + +- It streams BGP updates from the RIPE RIS Live service. +- It applies them to a routing table that it keeps in memory, with a copy in a + SQLite database. +- It serves web pages and a JSON API that answer from that table. ## Rationale RouteWatch exists to serve our other services as an internal API for information -about IP addresses. +about IP addresses. A service asks RouteWatch instead of working the answer out +for itself. -RouteWatch follows BGP routing table updates from RIPE RIS Live. RIS Live -streams the updates that RIPE's route collectors receive, as they happen, from -the moment a client connects. It reports changes, not the routing table they add -up to. RouteWatch applies those changes to a routing table that it keeps in -memory, with a copy in its own SQLite database. From that table, RouteWatch sees -directly: +RIPE runs route collectors around the world. Many routers, the RIS peers, send +the collectors their routes. RIS Live streams every change the collectors +receive, as it happens. RouteWatch applies those changes to its own routing +table. From that table it sees directly: - which AS each IP address is routed to -- which announced prefix holds each IP address, and how big that prefix is +- which announced prefix holds the address, and how big that prefix is +- how many peers carry the prefix, and with which origin ASes RouteWatch adds details of each AS from WHOIS, the public directory of the -internet registries. The details include the organisation that holds the AS. -RouteWatch keeps them in its database and looks them up again once they are 30 -days old. An IP lookup answers from those stored details and never waits for -WHOIS. +internet registries. The details include the organisation that holds the AS and +its abuse contact. RouteWatch keeps them in its database and fetches them again +once they are 30 days old. An IP lookup answers from the stored details and +never waits for WHOIS. -So our own services can ask RouteWatch about an IP address, instead of each -working it out for itself. +An IP lookup also says which public blocklists list the address (see +Blocklists). -RIS Live sends no routing table when a client connects. So the routing table -holds only the routes announced since RouteWatch first started. An IP lookup +## Getting started + +RouteWatch is built to run in Docker. Build the image, which is tagged +`routewatch`: + +```bash +script/docker +``` + +The build runs the tests and the linter first, and fails when either fails. Run +the image: + +```bash +docker run --memory=5g --memory-swap=5g -p 8080:8080 \ + -v routewatch:/var/lib/berlin.sneak.app.routewatch routewatch +``` + +The memory limit is explained in Memory. The volume holds the database and the +copies of the blocklists. + +To run the binary on the host instead, build it with `make build`. The build +needs Go and a C compiler, because the SQLite library is C: + +```bash +make build +./bin/routewatch +``` + +`make run` builds the binary and runs it with debug logging, with a copy of the +log in `log.txt`. + +Then open `http://localhost:8080/` in a browser, or ask about an address: + +```bash +curl http://localhost:8080/ip/192.0.2.1 +``` + +The routing table starts empty. RIS Live sends no routing table when a client +connects, only the changes from then on. So the table holds the routes announced +since RouteWatch first started, and it fills as the feed runs. An IP lookup finds no route for an address until a prefix that holds it is announced. The -routing table also misses these updates: +database keeps the table across restarts. -- those sent while RouteWatch is stopped -- those that RouteWatch drops when it falls behind +## HTTP -A lookup or a prefix page can therefore show a route that has since been -withdrawn. +Every route answers `HEAD` as it answers `GET`, without the body. The server +listens on `PORT`, `8080` by default. -## Features +The pages and the API read routes from the routing table in memory. For each +prefix and origin AS, the table keeps the two newest routes in full, with their +AS paths and times. This document calls them the kept routes. The table keeps +only the address of every other peer that carries the prefix with that origin +AS. -- Real-time streaming of BGP updates from RIPE RIS Live -- Maintains live IPv4 and IPv6 routing tables -- Tracks AS peering relationships -- HTTP API for IP-to-AS lookups, prefix details, and AS information -- IP lookups report the public blocklists an address is on, from copies kept in - memory and on disk -- Automatic reconnection with exponential backoff -- Batched database writes for high performance -- Backpressure handling to prevent memory exhaustion +### Web pages -## Installation - -```bash -go build -o routewatch ./cmd/routewatch -``` - -## Usage - -```bash -# Run the daemon (listens on port 8080 by default) -./routewatch - -# Set custom port -PORT=3000 ./routewatch - -# Enable debug logging -DEBUG=routewatch ./routewatch -``` - -## HTTP Endpoints - -Every route answers `HEAD` as it answers `GET`, without the body. - -The pages and the API read routes from the routing table that RouteWatch keeps -in memory. For each prefix and origin AS, the table keeps the two newest routes -in full, with their AS paths and times. These are the kept routes. The table -keeps only the address of every other peer that carries the prefix. When both -kept routes of an origin AS are withdrawn, that origin has no kept route until -one of its peers sends an update. - -### Web Interface - -- `GET /` - Home page (HTML): headline figures, lookups by AS number and by IP - address, and a search of AS names. A card shows the `curl` command that looks - up the visitor's own address at `/ip/{addr}`, and that command's live answer. - Other cards show figures of the routing table, with links to the AS and prefix - pages: - - the newest and the oldest kept route, with its prefix, origin AS and time - - the AS last seen in an AS path most recently, and the one last seen - longest ago +- `/`: the home page. It has a lookup by AS number, a search of AS names and a + lookup by IP address. A card shows the `curl` command that looks up the + visitor's own address, and that command's live answer. Other cards show + figures of the routing table, with links to the AS and prefix pages: + - the newest and the oldest kept route, each with its prefix, origin AS and + time + - the AS most recently seen in an AS path, and the AS seen longest ago - the AS that originates the most prefixes - - how many ASes originate a single prefix, and the most recently updated of - them + - how many ASes originate a single prefix, and the newest kept route of + those ASes - the number of bogon announcements, linked to `/bogons` - the number of prefixes with more than one origin AS, linked to `/multiple-origins` - - well-known networks, such as Apple, Google, Cogent and MIT, each with its - number of prefixes and the time of its newest kept route, newest first + - fifteen well-known networks, such as Apple, Google, Cogent and MIT, each + with its number of prefixes and the time of its newest kept route, newest + first RouteWatch works these figures out once a minute, so they can be a minute old. -- `GET /bogons` - The bogon announcements (HTML): the routes whose prefix or - origin AS is reserved. A prefix is reserved when it lies inside one of these: - - a block that the IANA special-purpose address registries mark as not - globally reachable - - the multicast space, `224.0.0.0/4` and `ff00::/8`, which the IANA IPv4 and - IPv6 address space registries list as Multicast +- `/as/{asn}`: an AS, the prefixes it originates and the ASes it peers with. + When the database has no record of the AS, its handle and description come + from the AS list built into RouteWatch. The page then shows no first-seen + date. +- `/prefix/{prefix}/{length}`: a prefix, such as `/prefix/192.0.2.0/24`, with + its origin ASes and every peer that carries it. Only the kept routes show an + AS path, a next hop and a time. +- `/prefixlength/{length}`: up to 500 IPv4 prefixes with that mask length, + picked at random. Each shows the origin AS that the most peers carry it with, + and the age of its newest kept route. +- `/prefixlength6/{length}`: the same for IPv6. +- `/bogons`: the bogon announcements. A bogon announcement is a route whose + prefix or origin AS is reserved. Each row shows the prefix, the origin AS, the + number of peers and the reason. The page lists at most 1,000 rows, in prefix + order, and says how many there are in all. +- `/multiple-origins`: the prefixes that more than one origin AS announces. Each + row shows the origin ASes and how many peers carry the prefix with each. The + page lists at most 1,000 rows, in prefix order, and says how many there are in + all. +- `/status`: a dashboard of the figures of `/api/v1/stats`, refreshed every 2 + seconds. - A reserved AS number is one of these: - - an AS number for private use or for documentation - - `AS23456` - - the last AS number of the 16-bit or the 32-bit range +A prefix is reserved when it lies inside one of these: - Address space and AS numbers that the IANA has not handed out yet are not - counted. Each row shows the prefix, the origin AS, the number of peers and - why it is a bogon. The page shows at most 1,000 rows, in prefix order, and - says how many there are in all. +- a block that the IANA special-purpose address registries mark as not globally + reachable +- the multicast space, `224.0.0.0/4` and `ff00::/8` -- `GET /multiple-origins` - The prefixes announced by more than one origin AS - (HTML). Each row shows the origin ASes and how many peers carry the prefix - with each. Like `/bogons`, it shows at most 1,000 rows, in prefix order, and - says how many there are in all. -- `GET /status` - HTML status dashboard, which reads `/api/v1/stats` -- `GET /status.json` - JSON statistics: feed, memory, database and route - figures. The route figures come from the routing table. `oldest_route` and - `newest_route` are the times of the oldest and the newest kept routes. - `origins_without_kept_route` counts, across all prefixes, the origin ASes that - have no kept route. Other figures include these: - - `uptime_seconds`: the number of seconds since the RIS Live feed last - connected, or 0 when it has never connected - - `memory_usage_bytes`: the size of the Go heap in use, in bytes - - `whois_stats.current_interval_seconds`: the WHOIS fetcher's wait between - two fetches, in seconds, which can include a fraction of a second +An AS number is reserved when it is one of these: -- `GET /.well-known/healthcheck.json` - Health check (JSON): 200 when the - database answers and the RIS Live feed is connected, otherwise 503; - `data.status` is `ok` or `error` and `data.checks` says which check failed -- `GET /as/{asn}` - AS detail page (HTML): the prefixes the AS originates and - its peers. When the database has no record of the AS, its handle and - description come from the AS list built into RouteWatch. The page then shows - no first-seen date. -- `GET /prefix/{prefix}/{length}` - Prefix detail page (HTML), such as - `/prefix/192.0.2.0/24`: its origin ASes and every peer that carries it. Only - the kept routes show an AS path, a next hop and a time. -- `GET /prefixlength/{length}` - Up to 500 IPv4 prefixes with that mask length, - picked at random (HTML). Each shows the origin AS that most peers carry it - with, and the age of its newest kept route. -- `GET /prefixlength6/{length}` - The same for IPv6 -- `GET /ip/{addr}` - The same JSON answer as `/api/v1/ip/{ip}` -- `GET /ip/` - The same, for the caller's own address: the address the request - came from or, when that is a trusted proxy (`TRUSTED_PROXIES`, see - Configuration), the client address in its `X-Forwarded-For` +- an AS number for private use or for documentation +- `AS23456`, which stands in for a 32-bit AS number +- the last AS number of the 16-bit range or of the 32-bit range -### API v1 +Address space and AS numbers that the IANA has not handed out yet are not +counted. -- `GET /api/v1/stats` - The `/status.json` figures plus connection, garbage - collector, stream and handler queue statistics. `connection_duration_seconds` - is the number of seconds the RIS Live feed has been connected, or null while - it is not connected. `stream.route_churn_per_min` is the number of prefix - announcements and withdrawals, from all peers, in the last full clock minute. -- `GET /api/v1/ip/{ip}` - Looks up each address. `{ip}` is an IP address, a host - name, or a comma-separated list of at most 10 of these. `results` has one - entry for each address. Every entry has the same fields: - - `ip`: the address - - `ptr`: the address's PTR names, or an empty list when it has none - - `blocklists`: the public blocklists the address is on (see Blocklists), or - an empty list when it is on none +### JSON API - Every other field is about the address's live route. Each is null when the - address has no live route. The fields about the route: - - `netblock`: the most specific prefix that holds the address - - `mask_length`: the netblock's mask length - - `ip_version`: the netblock's IP version, 4 or 6 - - `num_peers`: the number of peers that carry the netblock - - `asn`: the origin AS that the most peers carry the netblock with - - `num_origins`: the number of origin ASes that peers carry the netblock - with - - `origin_asns`: those origin ASes, in AS number order - - `first_seen`: when RouteWatch first saw the netblock, or null when the - database has not stored the netblock yet - - `last_seen`: the time of the netblock's newest kept route, or null when - the netblock keeps no route +A JSON answer is `{"status": "ok", "data": ...}`. An error answer is +`{"status": "error", "error": {"msg": ..., "code": ...}}`. Every JSON answer +also carries an `@meta` object: - The fields about the AS in `asn` come from the database. Each is also null - when the database does not have it: - - `handle`: the AS's handle - - `description`: the AS's description - - `as_name`: the AS's name in its WHOIS details - - `org_name`: the name of the organisation that holds the AS - - `org_id`: the organisation's ID at its registry - - `address`: the organisation's postal address - - `country_code`: the organisation's country code - - `abuse_email`: the address that takes abuse reports for the AS - - `rir`: the regional internet registry of the AS +- `time_zone`: always `UTC` +- `api_version`: `1` +- `execution_time_ms`: the number of milliseconds the request took - Of the addresses a host name resolves to, only public ones are looked up. A - name with none, such as one that resolves to a private address, is reported - as `host name has no public address`, without its addresses. `errors` has - one entry for each host name that failed. The answer is 400 when: - - `{ip}` lists more than 10 - - `{ip}` comes to more than 10 addresses once its names are resolved - - nothing in `{ip}` is an IP address or a host name with a public address +Every value with a unit is a number. The unit ends the field name: `_ms`, +`_seconds` or `_bytes`. Every time is in UTC. - A PTR lookup gets 2 seconds. An address whose lookup takes longer is - answered without PTR names. +The routes: -- `GET /api/v1/as/{asn}` - The AS and the prefixes it originates, IPv4 and IPv6 - apart, each with the time of its newest kept route. A prefix that keeps no - route with the AS has no `last_updated`. When the database has no record of - the AS, its handle and description come from the AS list built into +- `GET /api/v1/ip/{ip}`: what RouteWatch knows about each address in `{ip}`. See + IP lookup below. +- `GET /ip/{addr}`: the same answer as `/api/v1/ip/{ip}`. +- `GET /ip/`: the same answer, for the caller's own address. That is the address + the request came from. For a request from a trusted proxy, it is the client + address in `X-Forwarded-For` (see `TRUSTED_PROXIES` in Configuration). +- `GET /api/v1/as/{asn}`: the AS in `asn`, and the prefixes it originates in + `ipv4_prefixes` and `ipv6_prefixes`, counted in `total_count`. Each prefix has + `last_updated`, the time of its newest kept route with the AS. A prefix that + keeps no route with the AS has no `last_updated`. When the database has no + record of the AS, its handle and description come from the AS list built into RouteWatch. The answer then has no `first_seen` or `last_seen`. -- `GET /api/v1/prefix/{prefix}/{length}` - One route for each peer that carries - the prefix, and the same routes grouped by origin AS in `origins`. Only the - kept routes have `as_path`, `next_hop` and `last_updated`. -- `GET /api/v1/search/as?q={text}` - Finds the ASes whose handle or description +- `GET /api/v1/prefix/{prefix}/{length}`: the prefix, one route for each peer + that carries it in `routes`, and the same routes grouped by origin AS in + `origins`. `peer_count` and `origin_count` count them. Only the kept routes + have `as_path`, `next_hop` and `last_updated`. +- `GET /api/v1/search/as?q={text}`: the ASes whose handle or description contains `{text}`, ignoring case. `data` holds: - `query`: `{text}` without its leading and trailing spaces - `total`: the number of ASes that match @@ -228,74 +194,144 @@ one of its peers sends an update. The search reads the AS list built into RouteWatch, not the database. The list comes from - [ipverse/as-metadata](https://github.com/ipverse/as-metadata). A match can - therefore be an AS that the database has no record of. A `q` that is - missing, empty or only spaces answers 400. + [ipverse/as-metadata](https://github.com/ipverse/as-metadata). So a match + can be an AS that the database has no record of. -A JSON answer is `{"status": "ok", "data": ...}`, or for an error -`{"status": "error", "error": {"msg": ..., "code": ...}}`. The health check is -the exception: its 503 answer is also `{"status": "ok", "data": ...}`, with -`error` in `data.status`. Every JSON answer also carries an `@meta` object: +- `GET /status.json`: figures of the feed, the memory, the database and the + routing table. The figures whose names do not say what they mean: + - `uptime_seconds`: the seconds since the feed last connected, or 0 when it + has never connected + - `memory_usage_bytes`: the size of the Go heap in use + - `live_routes`, `ipv4_routes` and `ipv6_routes`: the number of peer and + prefix pairs, where each peer that carries a prefix counts once for it + - `oldest_route` and `newest_route`: the times of the oldest and the newest + kept route + - `origins_without_kept_route`: the number of prefix and origin AS pairs + that peers carry but that keep no route + - `whois_stats.current_interval_seconds`: the WHOIS fetcher's wait between + two fetches, which can include a fraction of a second +- `GET /api/v1/stats`: the `/status.json` figures plus the reconnection count, + and figures of the garbage collector, the stream and the handler queues. + `connection_duration_seconds` is the seconds the feed has been connected, or + null while it is not connected. `stream.route_churn_per_min` is the number of + announcements and withdrawals, from all peers, in the last full clock minute. +- `GET /.well-known/healthcheck.json`: the health check. It answers 200 when the + database answers and the feed is connected, otherwise 503. `data.status` is + `ok` or `error`, and `data.checks` says which check failed. The 503 answer + keeps the `{"status": "ok", "data": ...}` form. -- `time_zone`: always `UTC` -- `api_version`: `1` -- `execution_time_ms`: the number of milliseconds the request took +### IP lookup -Every value with a unit is a number whose field name ends in the unit, such as -`_ms`, `_seconds` or `_bytes`. +`{ip}` is an IP address, a host name, or a comma-separated list of at most 10 of +these. A host name is resolved, and only its public addresses are looked up. The +answer holds `query`, the text looked up, and `results`, with one entry for each +address. Every entry has the same fields: -A request that runs past its time limit answers 504: the limit is 30 seconds, -and 4 seconds for the database reads behind `/status.json` and `/api/v1/stats`. -An AS number, prefix or mask length that does not parse answers 400, as does a -mask length out of range for its IP version. An AS answers 404 when it -originates no prefix and has no record in the database or in the AS list built -into RouteWatch. AS 0 always answers 404, because it means no known origin. The -AS name search never lists it. A prefix with no live route answers 404 too. The -HTML pages give these errors as plain text. Under `/api/v1` and `/ip/`, a path -that matches no route answers 404 with the message `Not found`. A request with -`POST`, `PUT`, `PATCH`, `DELETE`, `OPTIONS`, `TRACE` or `CONNECT` answers 405 -with `Method not allowed` on these routes: +- `ip`: the address +- `ptr`: the names DNS gives for the address, its PTR names, or an empty list + when it has none +- `blocklists`: the public blocklists the address is on (see Blocklists), or an + empty list when it is on none -- the routes under `/api/v1` and `/ip/` -- `/status.json` -- `/.well-known/healthcheck.json` +Every other field is about the address's live route. Each is null when the +address has no live route: -A 500 answer says only `Internal server error`. The error for a host name that -does not resolve says only `could not resolve host name`. The detail goes to the -log. +- `netblock`: the most specific prefix that holds the address +- `mask_length`: the netblock's mask length +- `ip_version`: the netblock's IP version, 4 or 6 +- `num_peers`: the number of peers that carry the netblock +- `asn`: the origin AS that the most peers carry the netblock with +- `num_origins`: the number of origin ASes that peers carry the netblock with +- `origin_asns`: those origin ASes, in AS number order +- `first_seen`: when RouteWatch first saw the netblock, or null when the + database has not stored the netblock yet +- `last_seen`: the time of the netblock's newest kept route, or null when the + netblock keeps no route -Every answer carries `Strict-Transport-Security`, `Content-Security-Policy`, -`X-Frame-Options`, `X-Content-Type-Options`, `Referrer-Policy` and -`Permissions-Policy`. The Content-Security-Policy lets a page load content only -from the server, and does not allow inline content with `'unsafe-inline'`: the -pages keep their CSS and JavaScript in inline `