routewatch/README.md

RouteWatch

RouteWatch is an MIT-licensed Go daemon by @sneak that follows the internet's BGP routing table as it changes and answers questions about IP addresses over HTTP.

BGP is the protocol by which the networks of the internet tell each other which addresses they can reach. Each network is an autonomous system, an AS, with a number such as AS15169. A prefix is a block of addresses, such as 192.0.2.0/24. The number after the slash is the mask length. A shorter mask length means a bigger block. A route says which AS announces a prefix and through which ASes the announcement came. The AS that announces the prefix is its origin AS. The ASes the announcement came through, in order, are the AS path.

RouteWatch does three things:

• It streams BGP updates from the RIPE RIS Live service. This document calls that stream the feed.
• It applies them to a routing table that it keeps in memory, with a copy in a SQLite database.
• It serves web pages and a JSON API that answer from that table.

Rationale

RouteWatch exists to serve our other services as an internal API for information about IP addresses. A service asks RouteWatch instead of working the answer out for itself.

RIPE runs route collectors around the world. Many routers, the RIS peers, send the collectors their routes. The connection between a peer and a collector is a BGP session. A peer carries a prefix when the peer has a route to that prefix. RIS Live streams every change the collectors receive, as it happens. RouteWatch applies those changes to its own routing table. From that table it sees directly:

• which AS each IP address is routed to
• which announced prefix holds the address, and how big that prefix is
• how many peers carry the prefix, and with which origin ASes

RouteWatch adds details of each AS from WHOIS, the public directory of the internet registries. The details include the organisation that holds the AS and its abuse contact. RouteWatch keeps them in its database and fetches them again once they are 30 days old. An IP lookup answers from the stored details and never waits for WHOIS.

An IP lookup also says which public IP lists hold the address (see IP lists).

Getting started

The usual way to run RouteWatch is its Docker image. Build the image, which is tagged routewatch:

make docker

The build runs the tests and the linter first, and fails when either fails. Run the image:

docker run --memory=5g --memory-swap=5g -p 8080:8080 \
    -v routewatch:/var/lib/berlin.sneak.app.routewatch routewatch

The memory limit is explained in Memory. The volume holds the database, the copies of the IP lists and the geolocation files.

To run the binary on the host instead, build it with make build. The build needs Go and a C compiler, because the SQLite library is written in C:

make build
./bin/routewatch

make run builds the binary and runs it with debug logging. It also writes the log to log.txt.

Then open http://localhost:8080/ in a browser, or ask about an address:

curl http://localhost:8080/ip/8.8.8.8

The routing table starts empty. RIS Live sends no routing table when a client connects, only the changes from then on. So the table holds the routes announced since RouteWatch first started, and it fills as the feed runs. An IP lookup finds no route for an address until a prefix that holds it is announced. The database keeps the table across restarts.

HTTP

The server listens on PORT, 8080 by default. Every page and API path answers HEAD as it answers GET, without the body.

The pages and the API read routes from the routing table in memory. For each prefix and origin AS, the table keeps the two newest routes in full, with their AS paths and times. This document calls them the kept routes. The table keeps only the address of every other peer that carries the prefix with that origin AS. When a kept route is withdrawn, no older route takes its place. So a prefix and origin AS can have two kept routes, one or none.

Web pages

• /: the home page. Its top row shows these figures, refreshed every 5 seconds:

  • whether the feed is connected
  • how many routes, ASes, prefixes and peers there are
  • the route updates per second

  Below the figures are a lookup by AS number, a search of AS names and a lookup by IP address. A card shows the curl command that looks up the visitor's own address, and that command's live answer. Other cards show figures of the routing table, with links to the AS and prefix pages:

  • the newest and the oldest kept route, each with its prefix, origin AS and time
  • the AS most recently seen in an AS path, and the AS seen longest ago
  • the AS that originates the most prefixes
  • how many ASes originate a single prefix, and the newest kept route of those ASes
  • the number of bogon announcements, linked to /bogons. A bogon announcement is a route whose prefix or origin AS is reserved, as the rules below say.
  • the number of prefixes with more than one origin AS, linked to /multiple-origins
  • fifteen well-known networks, such as Apple, Google, Cogent and MIT, each with its number of prefixes and the age of its newest kept route, newest first

  RouteWatch works out the figures on these cards once a minute, so they can be a minute old. Until the first figures are ready, the page says that they are being worked out.

• /as/{asn}: an AS, the prefixes it originates and the ASes next to it in AS paths. The prefixes come in two lists, IPv4 and IPv6, shortest mask length first. An AS has a handle, a short name such as GOOGLE, and a description, such as Google LLC. When the database has no record of the AS, both come from the AS list built into RouteWatch. The page then shows no first-seen date.

• /prefix/{prefix}/{length}: a prefix, such as /prefix/192.0.2.0/24, with its origin ASes and every peer that carries it. Only the kept routes show an AS path, a next hop and a time. The next hop is the router the peer sends the prefix's traffic to. RouteWatch records the peer itself as the next hop. The page also lists the prefix's origin changes, newest first. An origin change is recorded when the first peer starts to carry the prefix with an origin AS, or when the last peer stops. The page shows at most the newest 5,000. Each row shows:

  • the time
  • announced or withdrawn
  • the origin AS
  • the peer count: how many peers carry the prefix just after the change, with any origin AS

• /prefixlength/{length}: up to 500 IPv4 prefixes with that mask length, picked at random. Each shows its origin AS and the age of its newest kept route. When peers give a prefix different origin ASes, the page shows the one that most peers give.

• /prefixlength6/{length}: the same for IPv6.

• /bogons: the bogon announcements. Each row shows the prefix, the origin AS, the number of peers and the reason. The page lists at most 1,000 rows, in prefix order, and says how many there are in all. The list is worked out once a minute, with the home page's figures.

• /multiple-origins: the prefixes that more than one origin AS announces. Each row shows the origin ASes and how many peers carry the prefix with each. Like /bogons, the page lists at most 1,000 rows, in prefix order, and says how many there are in all.

• /status: a dashboard of the figures of /api/v1/stats, refreshed every 2 seconds.

A prefix is reserved when it lies inside one of these:

• a block that the IANA special-purpose address registries mark as not globally reachable. The IANA is the body that hands out the internet's address space and AS numbers.
• the multicast blocks, 224.0.0.0/4 and ff00::/8. A multicast address stands for a group of receivers, not for one host.

A few blocks that may be announced lie inside reserved blocks. Of the blocks that hold the prefix, the one with the longest mask length decides.

An AS number is reserved when it is one of these:

• an AS number for private use or for documentation
• AS23456, which stands in for a 32-bit AS number
• the last AS number of the 16-bit range or of the 32-bit range

Address space and AS numbers that the IANA has not handed out yet are not counted.

JSON API

A JSON answer is {"status": "ok", "data": ...}. An error answer is {"status": "error", "error": {"msg": ..., "code": ...}}. Every JSON answer also carries an @meta object:

• time_zone: always UTC
• api_version: 1
• execution_time_ms: the number of milliseconds the request took

Every value with a unit is a number. The unit ends the field name, as in _ms, _seconds, _bytes, _per_sec and _per_min. latitude and longitude are in degrees, and their names do not say so.

The paths:

• GET /api/v1/ip/{ip}: what RouteWatch knows about each address in {ip}. See IP lookup below.

• GET /ip/{addr}: the same answer as /api/v1/ip/{ip}.

• GET /ip/: the same answer, for the caller's own address. That is the address the request came from. For a request from a trusted proxy, it is the client address in X-Forwarded-For (see TRUSTED_PROXIES in Configuration).

• GET /api/v1/as/{asn}: the AS and the prefixes it originates. data holds:

  • asn: the AS, with its handle, description, first_seen and last_seen
  • ipv4_prefixes and ipv6_prefixes: the prefixes it originates, each with last_updated, the time of its newest kept route with the AS
  • total_count: how many prefixes the two lists hold together

  last_updated is left out when the kept routes of the prefix with the AS were all withdrawn. When the database has no record of the AS, its handle and description come from the AS list built into RouteWatch. The answer then has no first_seen or last_seen.

• GET /api/v1/prefix/{prefix}/{length}: the prefix and its routes. data holds:

  • prefix: the prefix
  • routes: one route for each peer that carries the prefix
  • origins: the same routes, grouped by origin AS
  • peer_count and origin_count: how many routes and how many origin ASes there are

  A route has prefix, mask_length, ip_version, origin_asn and peer_ip. Only the kept routes also have as_path, next_hop and last_updated.

• GET /api/v1/search/as?q={text}: the ASes whose handle or description contains {text}, ignoring case. data holds:

  • query: {text} without its leading and trailing spaces
  • total: the number of ASes that match
  • results: the first 50 of them in AS number order, each with asn, handle and description

  The search reads the AS list built into RouteWatch, not the database. The list comes from ipverse/as-metadata. So a match can be an AS that the database has no record of.

• GET /status.json: the daemon's figures. The figures whose names do not say what they mean:

  • uptime_seconds: the seconds since the feed last connected, or 0 when it has never connected
  • memory_usage_bytes: the size of the Go heap in use
  • live_routes, ipv4_routes and ipv6_routes: the number of peer and prefix pairs, where each peer that carries a prefix counts once for it
  • oldest_route and newest_route: the times of the oldest and the newest kept route
  • origins_without_kept_route: the number of prefix and origin AS pairs that peers carry but whose kept routes were all withdrawn
  • whois_stats.current_interval_seconds: the WHOIS fetcher's wait between two fetches, which can include a fraction of a second

• GET /api/v1/stats: the /status.json figures and five more:

  • reconnect_count: how many times the feed has reconnected since the start
  • connection_duration_seconds: the seconds the feed has been connected, or null while it is not connected
  • gc: figures of the garbage collector, the part of the Go runtime that frees memory no longer in use
  • stream: figures of the feed. stream.route_churn_per_min is the number of prefix announcements and withdrawals, from all peers, in the last full clock minute.
  • handler_stats: figures of the handlers' queues. The handlers are the four parts that take the feed's messages (see Design).

• GET /.well-known/healthcheck.json: the health check. It answers 200 when the database answers and the feed is connected, otherwise 503. data.status is ok or error, and data.checks says which check failed. The 503 answer keeps the {"status": "ok", "data": ...} form.

IP lookup

{ip} is an IP address, a host name, or a comma-separated list of at most 10 of these. A host name is resolved, and only its public addresses are looked up. The answer holds query, the text looked up, and results, with one entry for each address. Every entry has the same fields:

• ip: the address
• ptr: the address's PTR names, the host names that DNS gives for it (see PTR lookups), or an empty list when it has none
• flags: yes-or-no facts about the address:
  • is_tor: true when the address is on the tor_exits list
  • is_vpn: true when the address is on any list of VPN servers (see IP lists)
  • is_relay: true when the address is on the icloud_private_relay list
  • is_bogon: true when the address is reserved, so that no network should announce it on the internet
• vpn_providers: the names of the lists of VPN servers the address is on, or an empty list when it is on none
• ip_lists: the public IP lists the address is on (see IP lists), or an empty list when it is on none
• geo: where the address is (see Geolocation), or null when no source answers

Every other field is about the prefix in the routing table that holds the address. Each is null when no prefix in the routing table holds the address:

• netblock: the most specific prefix that holds the address, the one with the longest mask length
• mask_length: the netblock's mask length
• ip_version: the netblock's IP version, 4 or 6
• num_peers: the number of peers that carry the netblock
• asn: the netblock's origin AS. When peers give different origin ASes, it is the one that most peers give.
• num_origins: how many different origin ASes the netblock has
• origin_asns: those origin ASes, in AS number order
• first_seen: when RouteWatch first saw the netblock, or null when the database has not stored the netblock yet
• last_seen: the time of the netblock's newest kept route, or null when its kept routes were all withdrawn

The fields about the AS in asn come from the database. Each is null when the database does not have it:

• handle: the AS's handle
• description: the AS's description
• as_name: the AS's name in its WHOIS details
• org_name: the name of the organisation that holds the AS
• org_id: the organisation's ID at its registry
• address: the organisation's postal address
• country_code: the organisation's country code
• abuse_email: the address that takes abuse reports for the AS
• rir: the regional internet registry that holds the AS, one of ARIN, RIPE, APNIC, LACNIC and AFRINIC

When a host name fails, errors lists it with the reason. A name that does not resolve is reported as could not resolve host name. A name with no public address, such as one that resolves to a private address, is reported as host name has no public address. The answer is 400 when:

• {ip} lists more than 10 addresses and host names
• {ip} comes to more than 10 addresses once its names are resolved
• nothing in {ip} is an IP address or a host name with a public address

A successful answer carries Cache-Control with a max-age of 15 minutes, the shortest IP list interval. The answer for the caller's own address (/ip/) is private, and the others are public.

Errors

• 400, for:
  • an AS number, prefix or mask length that does not parse
  • a mask length out of range for its IP version
  • an AS name search without q, or with only spaces in it
  • an IP lookup with too many addresses, or none (see IP lookup)
• 404, for:
  • AS 0
  • an AS that originates no prefix and has no record in the database or in the AS list built into RouteWatch
  • a prefix that no peer carries
  • under /api/v1 and /ip/, a path that matches nothing, with the message Not found
  • any other path that matches nothing, with the text 404 page not found
• 405, with the message Method not allowed, for POST, PUT, PATCH, DELETE, OPTIONS, TRACE or CONNECT on these paths:
  • the paths under /api/v1 and /ip/
  • /status.json
  • /.well-known/healthcheck.json
• 500: an internal error. The JSON paths say only Internal server error. The detail goes to the log.
• 504: Request timeout. A request may take 30 seconds. The database reads behind /status.json and /api/v1/stats may take 4 seconds.

The pages give these errors as plain text.

AS 0

An origin AS of 0 means the route has no known origin. Its AS path ends in an AS set: a group of several ASes, any of which may have originated the route. The 0 appears in these fields:

• origin_asn in a route
• asn and origin_asns in an IP lookup
• the key 0 in a prefix's origins

The pages show such a route as having no known origin. /as/0 answers 404, and the AS name search never lists AS 0.

Headers

Every answer carries these headers:

• Strict-Transport-Security
• Content-Security-Policy
• X-Frame-Options
• X-Content-Type-Options
• Referrer-Policy
• Permissions-Policy

The content security policy lets a page load content only from the server, and lets no other site frame the pages. The pages keep their CSS and JavaScript in inline <style> and <script> blocks. Each answer's policy names a random value, a nonce, made for that answer. The browser applies or runs only the blocks that carry it.

IP lists

An IP lookup reports which public IP lists hold each address. Each entry in ip_lists has:

• name: the list's name, from the lists below
• description: what being on the list means
• updated: when RouteWatch last downloaded the list, or last heard from its publisher that its copy was current

The lists:

• dshield: the 20 networks that DShield saw scan the most targets in the last three days. From DShield, downloaded every hour.
• feodo_tracker: botnet command-and-control servers. From abuse.ch Feodo Tracker, downloaded every 15 minutes.
• firehol_abusers_30d: addresses reported for comment, form or forum spam in the last 30 days. From FireHOL, downloaded every 5 hours.
• firehol_level3: addresses reported for attacks, spyware or viruses in the last 30 days. From FireHOL, downloaded every 3 hours.
• icloud_private_relay: the networks where iCloud Private Relay traffic leaves. From Apple, downloaded once a day.
• ivpn: VPN servers run by IVPN. From IVPN, downloaded once a day.
• mullvad: VPN servers run by Mullvad. From Mullvad, downloaded once a day.
• nordvpn: VPN servers run by NordVPN. From NordVPN, downloaded once a day.
• pia: VPN servers run by Private Internet Access. From Private Internet Access, downloaded once a day.
• protonvpn: the /24 networks that hold ProtonVPN servers. The whole /24 is listed because the ProtonVPN-IPs README says traffic leaves from unpublished addresses in it. From the public ProtonVPN-IPs mirror of Proton's server list, downloaded once a day.
• protonvpn_entry: the IPv4 and IPv6 addresses of ProtonVPN servers. From the same ProtonVPN-IPs mirror, downloaded once a day.
• spamhaus_drop and spamhaus_dropv6: IPv4 and IPv6 networks hijacked or leased by spammers or cyber criminals, according to Spamhaus. From Spamhaus DROP, downloaded once a day.
• tor_exits: the IPv4 and IPv6 addresses of Tor exit nodes, where traffic from the Tor network leaves. From the Tor Project's Onionoo service, downloaded every hour.
• x4bnet and x4bnet_v6: IPv4 and IPv6 networks of VPN services, according to a community list that names no service. From X4BNet, downloaded once a day.

No list is downloaded more often than its publisher updates it or allows. Each can be downloaded whole without an account or a key.

How RouteWatch keeps the lists:

• It holds each list in memory, so a lookup makes no outside request.
• It keeps a copy of each list in the iplists directory of the state directory (see Configuration). Each copy is a file named after its list.
• Beside each copy, a file ending in .validators keeps the Last-Modified and ETag headers the publisher sent with that copy.
• At startup it loads each list from its copy. It downloads a list at once when the copy is older than the list's interval, or when there is no copy.
• A download sends the copy's Last-Modified as If-Modified-Since and its ETag as If-None-Match. The publisher then sends the list only when it has changed.
• A download fails when it takes over a minute or its file is over 200 MiB.
• When a download fails, or its file does not parse, RouteWatch keeps the previous copy and logs the failure. It tries again one interval later.

PTR lookups

An IP lookup finds each address's PTR names over DNS-over-HTTPS. It asks these public resolvers in order, until one answers:

• Google: https://dns.google/dns-query
• Cloudflare: https://cloudflare-dns.com/dns-query
• AdGuard: https://unfiltered.adguard-dns.com/dns-query
• Quad9: https://dns.quad9.net/dns-query

One address's lookup gets 2 seconds across all four. Each resolver gets an equal share of the time left, so one that does not answer leaves time for the next. When none answers in time, the address's ptr is an empty list.

A reserved address, one whose is_bogon flag is true, is not sent to the resolvers. Its ptr is an empty list.

What goes outside: each resolver asked receives the address being looked up. For /ip/ and the home page, that address is the visitor's own. A lookup answered from the cache sends nothing.

How RouteWatch keeps the answers:

• It holds answers in memory. An answer that has not expired is served from there.
• It writes new answers every 5 seconds to cache.db in the state directory (see Configuration), and loads them from there at startup.
• Each DNS record carries a TTL, the time for which it may be kept. An answer expires after the shortest TTL of its records. RouteWatch keeps it for at least 60 seconds and at most 24 hours.
• When an address has no PTR names, the answer's SOA record says how long that answer may be kept. The answer expires after the SOA record's minimum field, or the record's own TTL when that is shorter. RouteWatch keeps it for at most 1 hour.
• An answer with no PTR names and no SOA record is not kept. Neither is one whose SOA record has a TTL or minimum field of 0.
• A failed lookup is not kept, so the next lookup of the address asks again.
• Every 10 minutes it deletes expired answers from memory and from cache.db.
• It keeps answers for at most 250,000 addresses, and keeps no answer for a new address while it holds that many.

Geolocation

An IP lookup finds where each address is, on a best-effort basis. It uses two sources:

• geojs is asked first. One request carries every address of the lookup that the cache does not hold. geojs has 1 second to answer.
• DB-IP City Lite answers when geojs has no answer for the address. It is a database file that RouteWatch reads itself.

geojs has no answer when its request fails or runs out of time. An answer with no country, city or coordinates also counts as no answer, and RouteWatch does not cache it.

The geo object of a result has these fields:

• country_code: the two-letter country code
• country: the country's name in English
• continent_code: the two-letter continent code
• region: the region or state
• city: the city
• latitude and longitude: the location in degrees
• accuracy_radius_km: how far from that location the address may be, in km
• time_zone: the time zone's name, such as Europe/Berlin
• source: the source of the answer, geojs or dbip
• updated: when the answer was fetched from geojs, or when the database copy was downloaded

A field the source does not know is null. DB-IP has no accuracy_radius_km and no time_zone, so both are null in its answers. geo itself is null when neither source answers. A reserved address, one whose is_bogon flag is true, goes to neither source, so its geo is null.

What goes outside: geojs receives each address that the cache does not hold. For /ip/ and the home page, that address is the visitor's own. A lookup answered from the cache sends nothing. DB-IP receives only the monthly request for its file.

How RouteWatch keeps the answers of geojs:

• It holds at most 250,000 addresses in memory, and drops the least recently used first. That takes about 100 MB of the Go heap.
• It writes new answers every 5 seconds to cache.db in the state directory.
• It loads the newest 250,000 answers from cache.db at startup.
• An address not in memory is read from cache.db before geojs is asked.
• An answer expires 7 days after its fetch. Every 10 minutes RouteWatch deletes expired answers from memory and from cache.db.

Answers from the database are not cached, because reading the file is local.

How RouteWatch keeps the database:

• It keeps one copy in the state directory, named dbip-city-lite.mmdb.
• It downloads the current month's file at the first start, and then once a month. Until the first download finishes, only geojs answers.
• A download fails when it takes over 10 minutes or unpacks to over 200 MiB.
• A new file replaces the old copy only once it opens. After a failed download RouteWatch keeps the old copy and tries again an hour later.
• The file is about 130 MB. RouteWatch maps it into memory, outside the Go heap. The kernel loads the parts that are read, and can drop them again when memory is short.

The home page's footer says "IP Geolocation by DB-IP" with a link to db-ip.com.

Configuration

RouteWatch reads its settings from environment variables. It takes no command-line flags.

• PORT: the port the HTTP server listens on. A whole number from 1 to 65535. Default 8080.
• DEBUG: set it to routewatch for debug logging. The daemon then also logs a System stats line every 60 seconds, with the Go memory figures and the number of goroutines, Go's lightweight threads. Default empty.
• TRUSTED_PROXIES: the reverse proxies whose X-Forwarded-For header the server believes. A reverse proxy is a server in front of RouteWatch that passes requests on to it. See below. Default: the private ranges of RFC 1918.
• XDG_DATA_HOME: the base of the state directory (see below) on Linux and the BSDs, when RouteWatch does not run as root. It must be an absolute path. The Docker image sets /var/lib.
• GOMEMLIMIT: the soft limit of the Go heap. The Docker image sets 2560MiB. See Memory.
• MALLOC_ARENA_MAX: how many memory pools, called arenas, the C library may create. The Docker image sets 2. See Memory.

The start fails, with an error and a non-zero exit, when PORT, XDG_DATA_HOME or TRUSTED_PROXIES has an invalid value. In the Docker image, entrypoint.sh refuses a MALLOC_ARENA_MAX that is not a positive whole number. An empty variable counts as unset. TRUSTED_PROXIES is the one exception: set but empty, it makes the server trust no proxy.

TRUSTED_PROXIES works like this:

• When it is not set, the server trusts 10.0.0.0/8, 172.16.0.0/12 and 192.168.0.0/16.
• When it lists addresses and prefixes, comma-separated, such as 10.0.0.0/8,192.0.2.1, the server trusts those instead.
• When it is set but empty (TRUSTED_PROXIES=), the server trusts no proxy.

Three parts of RouteWatch read X-Forwarded-For:

• the request log
• /ip/
• the home page's curl card

For a request from a trusted proxy, they use the last address in the header that is not a trusted proxy. Each proxy adds, at the end of the header, the address that sent it the request. The entries left of the client's address were written by the client. For a request from any other address, they use the address the request came from, whatever headers it carries. X-Real-IP and True-Client-IP are not read.

The state directory holds:

• db.sqlite: the database
• cache.db: the answers of PTR lookups and of geojs
• dbip-city-lite.mmdb: the copy of DB-IP's City Lite database (see Geolocation)
• iplists: the IP list copies

It is at:

• macOS: ~/Library/Application Support/berlin.sneak.app.routewatch/
• Linux and the BSDs, as root: /var/lib/berlin.sneak.app.routewatch/
• Linux and the BSDs, as another user: $XDG_DATA_HOME/berlin.sneak.app.routewatch/, or ~/.local/share/berlin.sneak.app.routewatch/ when XDG_DATA_HOME is not set
• the Docker image: /var/lib/berlin.sneak.app.routewatch/

RouteWatch logs to standard output. It writes text when standard output is a terminal, and JSON lines otherwise.

Memory

The daemon keeps the whole routing table in memory. RouteWatch and its Docker image set ceilings that keep it inside a 5 GiB container. Run the container with a memory limit of 5 GiB and swap disabled:

docker run --memory=5g --memory-swap=5g ...

The 5 GiB leaves room above the ceilings for spikes, for the kernel's cache of the database file, and for the mapped copy of DB-IP's database (see Geolocation).

The ceilings add up to 3.7 GiB:

• The Go heap, the memory the Go program allocates: a 2.5 GiB soft limit, GOMEMLIMIT=2560MiB, set by the image. The routing table takes about 700 bytes for each prefix and origin AS pair. A full table has about 1.4 million pairs, so it takes about 1 GB. The rest of the heap takes up to about 0.5 GiB, most of it the handlers' batches and queues (see below). The remaining 1 GiB is room for the garbage collector, which frees memory the program no longer uses. With much less, it would run almost all the time. The routing table also keeps at most 100,000 origin changes waiting to be written, about 8 MB.
• SQLite: 1 GiB, set in the code. The connection that does every write has a 512 MiB page cache: the database pages it keeps in memory from one write to the next. The routes table of a full routing table takes about 450 MB, so its pages fit. Each of the 9 reading connections has 16 MiB. A 768 MiB soft heap limit sits above the caches, so it does not shrink them. A 1 GiB hard heap limit caps the SQLite library.
• About 0.2 GiB for everything else in the runtime.

The icloud_private_relay list has about 285,000 networks. Neighbouring networks merge, which leaves about 6,900 address ranges. Loaded, the list adds about 0.3 MiB to the Go heap and under 6 MiB to the process's resident memory. Parsing the file allocates about 94 MiB in about 0.2 seconds. The next garbage collection frees all of it but the 0.3 MiB above. Parsing happens at startup and whenever a download brings a new version.

The SQLite library gets its memory from glibc, the C library. glibc keeps freed memory in pools called arenas, and hands it back to the kernel slowly. It creates up to eight arenas per processor core. So on a host with many cores the process holds far more memory than SQLite uses. The image sets MALLOC_ARENA_MAX=2, which caps the arenas at two. Database writes run one at a time, so the cap does not slow them.

What happens at each limit:

• The Go soft limit: as the heap nears GOMEMLIMIT, the runtime collects garbage more often rather than growing further.

• The SQLite soft heap limit: SQLite recycles its page cache instead of allocating more.

• The SQLite hard heap limit: a database statement fails with an out-of-memory error, which the handler logs. The prefix handler writes the routing table's changes and its origin changes again in a later write. Everything else in a handler's failed write is dropped. The PTR cache and the geolocation part write their answers to cache.db again in a later write. The process keeps running.

• The handler queues: the handlers are the four parts that take the feed's messages (see Design). The prefix handler's queue holds at most 100,000 messages, about 100 MiB. Each of the other three holds at most 20,000. When a queue fills, the streamer drops messages instead of waiting. The streamer is the part that reads the feed. The exception is a STATE message, which it holds, in feed order, until there is room. A STATE message says that a peer's BGP session changed state, for example went down. These are a few hundred bytes each, about ten a second.

• The handler batches: while the AS, peer or prefix handler writes a batch to the database, the updates that arrive collect in its next batch.

  • The AS handler's batch holds one entry per AS. About 80,000 ASes appear in the feed, so it takes at most about 10 MiB.
  • The peer handler's batch stops at 100,000 updates, about 10 MiB.
  • The prefix handler's batch stops at 250,000 prefix updates, about 15 MiB. Each prefix an update announces or withdraws counts once.

  At that size the peer or prefix handler takes nothing more off its queue until the write ends. The queue fills, and the streamer drops messages as above.

• Held-back messages: the prefix handler holds back a peer's messages after the peer's session goes down. It applies them after the peer has left every prefix and after the held-back messages of every peer whose session went down before it. It holds at most 150,000 such messages, about 150 MiB. At that number it takes nothing more off its queue until it has applied one of them.

• Waiting origin changes: the routing table keeps at most 100,000 origin changes waiting to be written. Past that, it drops the oldest, and the prefix handler logs how many. The prefix page never shows a dropped origin change.

To replace the image's Go soft limit, set GOMEMLIMIT in the environment, for example -e GOMEMLIMIT=3GiB. A limit close to the routing table's size makes the garbage collector run almost all the time. MALLOC_ARENA_MAX can be replaced the same way, but raising it lets the process grow again on a host with many cores.

Running under upaas

What upaas, the deployment tool, needs:

• Container port: 8080.
• Volume: one, at the container path /var/lib/berlin.sneak.app.routewatch.
• Environment: nothing. When TRUSTED_PROXIES is not set, the server trusts a reverse proxy at an RFC 1918 address. Set it only when the proxy connects from another address. Leave XDG_DATA_HOME, GOMEMLIMIT and MALLOC_ARENA_MAX at the image's values. DEBUG=routewatch is optional.
• Memory limit: 5g. upaas sets no swap limit, so on a host with swap Docker allows the same amount of swap again.
• Health check: the image's HEALTHCHECK requests /.well-known/healthcheck.json on the container port. upaas reads the container's health 60 seconds after a deploy and fails the deploy unless it is healthy. In those 60 seconds, Docker 25 and later runs the check every 5 seconds, and the first pass makes the container healthy. Startup first reads the routing table from the database, which takes about 25 seconds when the table is full. The check also needs the feed connected, so a container that cannot reach RIS Live never becomes healthy.
• Stop grace: upaas gives the container 10 seconds between SIGTERM and SIGKILL, as docker stop does by default. The daemon gives its parts 8 seconds to stop and then exits with status 0. A stop that is still running after 9 seconds exits with status 1 instead. See Shutdown under Design for what a stop writes and what it can lose.

Design

Parts

The parts are built in two steps:

• The fx library, which wires a Go program's parts together, builds these:
  • the database
  • the routing table in memory
  • the streamer
  • the HTTP server
• RouteWatch.Run in internal/routewatch/app.go builds the other parts and starts them all.

The parts:

• The streamer connects to RIS Live, reads the JSON stream and parses the BGP messages. It gives each message to the handlers that want it, through one queue per handler.
• The prefix handler keeps the routing table in memory current. Every second it writes the table's changes to the routes table and the table's origin changes to the origin_changes table of the database. It also stores each new prefix with the time it was first seen.
• The AS handler records every AS seen in an AS path in the asns table, with the times it was first and last seen.
• The peer handler records every peer in the bgp_peers table.
• The peering handler takes the pairs of neighbouring ASes out of the AS paths and writes them to the peerings table every 30 seconds.
• The WHOIS fetcher fetches the details of one AS at a time: those never fetched, and those fetched more than 30 days ago. It asks IANA which registry holds the AS, then asks that registry. Between these fetches it waits. The wait starts at 15 seconds. It halves after a success, down to 1 second, and grows by half after a failure, up to a minute. An IP lookup that finds an AS with old or missing details asks the fetcher for it at once, without the wait.
• The IP lists part downloads the IP lists and answers which lists hold an address.
• The PTR cache looks up the PTR names of addresses and keeps the answers (see PTR lookups).
• The geolocation part finds where addresses are, keeps the answers of geojs and downloads DB-IP's database (see Geolocation).
• The database maintainer checkpoints the database every 5 seconds: it folds the write-ahead log, where SQLite first writes changes, into the database file. It reclaims free space every 10 minutes. Every hour it refreshes the statistics SQLite uses to plan its queries, and prunes the origin_changes table. A prune keeps each prefix's rows from the last 72 hours, or its newest 500, whichever keeps more.
• The HTTP server serves the pages and the API. Every route it shows comes from the routing table in memory. WHOIS details, first-seen times and origin changes come from the database. AS names come from the database or from the AS list built into RouteWatch. The AS name search reads the AS list built into RouteWatch.

The routing table

internal/routetable holds the whole routing table. For each prefix and origin AS it keeps:

• every peer whose current route to the prefix has that origin AS, as one bit per peer
• the two newest of those routes in full, with their AS paths and times

When a kept route is withdrawn, no older route takes its place: the table does not have the older routes in full any more.

The prefix handler applies the updates this way:

• An announcement gives the peer a route to the prefix. A peer carries a prefix with one origin AS at most.
• A withdrawal removes the peer's route. So does an announcement whose AS path is empty or holds AS 0 anywhere.
• A STATE message that says the peer's BGP session went down removes every route of the peer.
• The peer's later updates wait while its routes are removed. Then they are applied in order, after the held-back updates of every peer whose session went down before it.
• Within one message, the withdrawals are applied before the announcements.

The address lookup finds the most specific prefix that holds the address. It looks the address up at each mask length, longest first: at most 25 lookups for IPv4 and 113 for IPv6. Prefixes shorter than /8 for IPv4 or /16 for IPv6 are not used. Such a prefix says only where a peer sends traffic that matches no better route; it does not say who holds the address. One of them is the default route, 0.0.0.0/0 or ::/0, which matches every address.

The database's routes table has one row for each prefix and origin AS. Every second the prefix handler writes the pairs that changed, at most 20,000 in one write. The rest follow in the next writes. Startup reads the whole table back into memory before the streamer connects. A full table takes about 25 seconds to read. If the daemon crashes, it loses only the changes not yet written. The database runs with synchronous=OFF, so a power loss or an operating system crash can lose more.

What the table can miss

RIS Live sends no routing table when a client connects, so the table holds only the routes announced since RouteWatch first started. It also misses:

• the updates sent while RouteWatch is stopped
• the updates RouteWatch drops when it falls behind (see Feed handling)
• the updates lost at a stop (see Shutdown)

A lookup or a prefix page can therefore show a route that has since been withdrawn.

Peers are known by their IP address, and a peer can have sessions on two collectors. When its session on one goes down, the routes the other still carries are removed too, until they are announced again.

Feed handling

The streamer reads the stream line by line. Each line is one message as JSON. It passes an UPDATE or STATE message to each handler that wants it, through that handler's queue. An UPDATE message carries announcements and withdrawals. A STATE message says that the BGP session between a collector and a peer changed state, for example went down.

The prefix handler's queue holds 100,000 messages. Each of the other handlers' queues holds 20,000. A message that finds its handler's queue full is dropped for that handler. A STATE message is never dropped. One that finds the queue full is held until there is room, and the handler's other messages are dropped until the held ones are in.

The AS, peer and prefix handlers write to the database in batches. A batch is written when it is full, or when its time is up:

• the AS handler: 30,000 ASes, or after 2 seconds
• the peer handler: 10,000 updates, or after 2 seconds
• the prefix handler: 25,000 prefix updates, or after 1 second

The peering handler writes every 30 seconds.

Reconnection

Any end of the stream other than a shutdown marks the feed disconnected. That includes RIS Live ending the response normally, and a stream that sends no line for 60 seconds. The streamer then reconnects after a wait. The wait starts at 5 seconds and doubles each time, up to 320 seconds. A stream that lasted more than 30 seconds resets the wait to 5 seconds.

Shutdown

On SIGTERM or SIGINT the daemon stops its parts in this order:

1. the streamer
2. the handlers, each of which writes what it holds
3. the WHOIS fetcher
4. the database maintainer
5. the IP list downloads
6. the HTTP server
7. the geolocation part, which writes the geojs answers it holds
8. the PTR cache, which writes what it holds

Last, the daemon checkpoints and closes the database. It gives all this 8 seconds and then exits with status 0. A stop that is still running after 9 seconds exits with status 1 instead. When RouteWatch stops on an error, it exits with status 1.

What a stop can lose:

• The prefix handler's last write takes at most 20,000 of the routing table's changed pairs. A full-table peer is a peer that carries the whole routing table. Its session going down changes about a million pairs, which take about a minute to write. A stop in that minute leaves the rest unwritten.
• The prefix handler's last write of origin changes takes at most 20,000. The rest are lost.
• A withdrawal, or a session going down, that is lost this way leaves its routes in the routes table. The next start reads them back. They stay until the peer sends another update for the prefix, or its session goes down again.
• When the 8 seconds run out, the streamer and the handlers log what they could not hand over or write. The database is then not closed. When it next opens, SQLite rolls back any write the exit cut off.
• When the 8 seconds run out, the PTR answers not yet in cache.db are lost. RouteWatch asks the resolvers for them again the next time they are needed.
• When the 8 seconds run out, the geojs answers not yet in cache.db are lost. RouteWatch asks geojs for them again the next time they are needed.

Code layout

routewatch/
├── cmd/
│   ├── routewatch/         # The daemon's entry point
│   ├── asinfo-gen/         # Builds the AS list built into RouteWatch
│   └── streamdumper/       # Prints the raw RIS Live stream, for debugging
├── docs/
│   └── message-examples.json  # Examples of RIS Live messages
├── internal/
│   ├── bogon/              # Reserved prefixes and AS numbers
│   ├── cachedb/            # cache.db, the caches of outside answers
│   ├── config/             # Settings from the environment
│   ├── database/           # SQLite storage
│   ├── geo/                # Geolocation: geojs, DB-IP City Lite, the cache
│   ├── iplist/             # Public IP lists: downloads and lookups
│   ├── logger/             # Structured logging
│   ├── metrics/            # Feed and route update figures
│   ├── ptr/                # PTR lookups over DNS-over-HTTPS, and their cache
│   ├── ristypes/           # The RIS Live message types
│   ├── routetable/         # The whole routing table in memory
│   ├── routewatch/         # The daemon: startup, handlers, WHOIS fetcher
│   ├── server/             # HTTP server, pages and API
│   ├── streamer/           # RIS Live client
│   ├── templates/          # HTML templates
│   ├── version/            # Build version information
│   └── whois/              # WHOIS lookups of AS details
└── pkg/
    └── asinfo/             # The AS list built into RouteWatch

Database schema

asns(asn PRIMARY KEY, handle, description,
     as_name, org_name, org_id, address, country_code, abuse_email,
     abuse_phone, tech_email, tech_phone, rir, rir_registration_date,
     rir_last_modified, whois_raw,
     first_seen, last_seen, whois_updated_at)
prefixes_v4(prefix PRIMARY KEY, first_seen) WITHOUT ROWID
prefixes_v6(prefix PRIMARY KEY, first_seen) WITHOUT ROWID

-- The routing table, one row per prefix and origin AS that some peer
-- carries: the peer set, and the newest two routes in full
routes(prefix, origin_asn, peers,
       newest_peer, newest_as_path, newest_time,
       older_peer, older_as_path, older_time,
       PRIMARY KEY (prefix, origin_asn)) WITHOUT ROWID
-- The peers of the routes table, each with its number in the peer sets
route_peers(number PRIMARY KEY, address UNIQUE)

-- The origin changes of the routing table: each time a prefix and origin AS
-- gained its first peer or lost its last
origin_changes(id PRIMARY KEY, prefix, origin_asn, announced, peer_count, time)

peerings(id PRIMARY KEY, as_a, as_b, first_seen, last_seen,
         UNIQUE(as_a, as_b))
bgp_peers(id PRIMARY KEY, peer_ip UNIQUE, peer_asn, first_seen, last_seen,
          last_message_type)

-- Nothing in the daemon writes this table
announcements(id PRIMARY KEY, prefix_id, peer_asn, origin_asn, path,
              next_hop, timestamp, is_withdrawal)

The indexes are in internal/database/schema.sql.

cache.db is a separate file, with two tables:

-- The answers of PTR lookups. names is empty for an address with no PTR names.
hostnames(address PRIMARY KEY, names, expires_at)

-- The answers of geojs. A part geojs does not know is NULL.
geolocations(address PRIMARY KEY, country_code, country, continent_code,
             region, city, latitude, longitude, accuracy_radius_km, time_zone,
             fetched_at)

Development

The make targets:

• make bootstrap: install the development dependencies (see script/bootstrap below)
• make setup: make a fresh clone ready for development
• make test: run the tests
• make lint: run the linter
• make fmt: format the Go code and the markdown
• make fmt-check: check the formatting without writing
• make check: run the tests, the linter and the format check
• make hooks: install the git hook that runs script/precommit before each commit
• make build: build bin/routewatch
• make run: build the binary and run it with debug logging
• make clean: delete bin/
• make docker: build the Docker image
• make asupdate: rebuild the AS list built into RouteWatch

make with no target runs the tests.

The tests and the linter run only in Docker, as the test and lint phases of the Dockerfile. The image cannot be built unless both pass. Every docker build in script/ passes a new value of the CACHEBUST build argument, so each run executes every check. Go's build cache and the linter's cache are kept between builds.

The test suite runs with -short, which skips TestRouteWatchLiveFeed. That test streams the live feed for a few seconds. No make target runs it.

Entrypoints

This repository adheres to the Scripts to Rule Them All standard: the scripts in script/ are the entrypoints of the development workflow. Every make target except make build, make run and make clean calls one of them. The scripts:

• script/bootstrap: install the development dependencies:
  • git, make and Go, when they are missing
  • the pinned versions of node and yarn
  • prettier, the markdown formatter, from yarn.lock
  • the Go modules
• script/setup: make a fresh clone ready for development. It runs script/bootstrap, then script/install-precommit.
• script/projectname: print the project name, routewatch, which script/docker uses as the image tag.
• script/test: build the test phase of the Dockerfile alone. It runs go test -short -count=1 -timeout 90s -race -cover ./.... On failure it runs the tests again with -v and without -cover.
• script/lint: build the lint phase of the Dockerfile alone. It runs golangci-lint v2.14.0 with .golangci.yml.
• script/fmt: format the Go code with go fmt and the markdown with prettier. It writes.
• script/fmt-check: check the formatting without writing.
• script/check: run script/test, script/lint and script/fmt-check.
• script/docker: build the Docker image, tagged routewatch.
• script/cibuild: the CI entrypoint. It runs script/bootstrap and script/check, then builds the image.
• script/precommit: the pre-commit gate. It runs go mod tidy and go fmt, then script/check. go mod tidy must leave go.mod and go.sum unchanged; go fmt writes.
• script/install-precommit: install the git hook that runs script/precommit before each commit.
• script/asupdate: rebuild the AS list built into RouteWatch, pkg/asinfo/asdata.json.gz, from the as.csv file of ipverse/as-metadata. It needs the network. When the generator finds no AS, the old list stays and the script exits non-zero.

TODO

TODO.md holds the working list: the next step and the steps planned.

License

MIT. See LICENSE.

Author

@sneak