RouteWatch

RouteWatch is an MIT-licensed Go daemon by @sneak that monitors the BGP routing table in real time. It works like this:

• It streams BGP UPDATE messages from the RIPE RIS Live service.
• It keeps the whole routing table in memory, with a copy in SQLite.
• It serves web pages and a JSON API for looking routes up.

Rationale

RIPE RIS Live streams the BGP updates that RIPE's route collectors receive, as they happen, from the moment a client connects. It reports changes, not the routing table they add up to, so it cannot say which AS an IP address is routed to now or which routes a prefix has. RouteWatch applies the feed's updates to a routing table that it keeps in memory, with a copy in its own SQLite database. It answers IP-to-AS lookups and prefix pages from that table.

The feed sends no table when a client connects. So the routing table holds only the routes announced since RouteWatch first started. An IP-to-AS lookup finds nothing for an address whose routes have not been announced since then. The routing table also misses updates RouteWatch drops when it falls behind, and updates sent while it is stopped. So a lookup or a prefix page can show a route the feed has withdrawn.

Features

• Real-time streaming of BGP updates from RIPE RIS Live
• Maintains live IPv4 and IPv6 routing tables
• Tracks AS peering relationships
• HTTP API for IP-to-AS lookups, prefix details, and AS information
• IP lookups report the public blocklists an address is on, from copies kept in memory and on disk
• Automatic reconnection with exponential backoff
• Batched database writes for high performance
• Backpressure handling to prevent memory exhaustion

Installation

go build -o routewatch ./cmd/routewatch

Usage

# Run the daemon (listens on port 8080 by default)
./routewatch

# Set custom port
PORT=3000 ./routewatch

# Enable debug logging
DEBUG=routewatch ./routewatch

HTTP Endpoints

Every route answers HEAD as it answers GET, without the body.

The pages and the API read routes from the routing table that RouteWatch keeps in memory. For each prefix and origin AS, the table keeps the two newest routes in full, with their AS paths and times. These are the kept routes. The table keeps only the address of every other peer that carries the prefix. When both kept routes of an origin AS are withdrawn, that origin has no kept route until one of its peers sends an update.

Web Interface

• GET / - Home page (HTML): headline figures, lookups by AS number and by IP address, and a search of AS names. A card shows the curl command that looks up the visitor's own address at /ip/{addr}, and that command's live answer. Other cards show figures of the routing table, with links to the AS and prefix pages:

  • the newest and the oldest kept route, with its prefix, origin AS and time
  • the AS last seen in an AS path most recently, and the one last seen longest ago
  • the AS that originates the most prefixes
  • how many ASes originate a single prefix, and the most recently updated of them
  • the number of bogon announcements, linked to /bogons
  • the number of prefixes with more than one origin AS, linked to /multiple-origins
  • well-known networks, such as Apple, Google, Cogent and MIT, each with its number of prefixes and the time of its newest kept route, newest first

  RouteWatch works these figures out once a minute, so they can be a minute old.

• GET /bogons - The bogon announcements (HTML): the routes whose prefix or origin AS is reserved. A prefix is reserved when it lies inside one of these:

  • a block that the IANA special-purpose address registries mark as not globally reachable
  • the multicast space, 224.0.0.0/4 and ff00::/8, which the IANA IPv4 and IPv6 address space registries list as Multicast

  A reserved AS number is one of these:

  • an AS number for private use or for documentation
  • AS23456
  • the last AS number of the 16-bit or the 32-bit range

  Address space and AS numbers that the IANA has not handed out yet are not counted. Each row shows the prefix, the origin AS, the number of peers and why it is a bogon. The page shows at most 1,000 rows, in prefix order, and says how many there are in all.

• GET /multiple-origins - The prefixes announced by more than one origin AS (HTML). Each row shows the origin ASes and how many peers carry the prefix with each. Like /bogons, it shows at most 1,000 rows, in prefix order, and says how many there are in all.

• GET /status - HTML status dashboard, which reads /api/v1/stats

• GET /status.json - JSON statistics: feed, memory, database and route figures. The route figures come from the routing table. oldest_route and newest_route are the times of the oldest and the newest kept routes. origins_without_kept_route counts, across all prefixes, the origin ASes that have no kept route. Other figures include these:

  • uptime_seconds: the number of seconds since the RIS Live feed last connected, or 0 when it has never connected
  • memory_usage_bytes: the size of the Go heap in use, in bytes
  • whois_stats.current_interval_seconds: the WHOIS fetcher's wait between two fetches, in seconds, which can include a fraction of a second

• GET /.well-known/healthcheck.json - Health check (JSON): 200 when the database answers and the RIS Live feed is connected, otherwise 503; data.status is ok or error and data.checks says which check failed

• GET /as/{asn} - AS detail page (HTML): the prefixes the AS originates and its peers. When the database has no record of the AS, its handle and description come from the AS list built into RouteWatch. The page then shows no first-seen date.

• GET /prefix/{prefix}/{length} - Prefix detail page (HTML), such as /prefix/192.0.2.0/24: its origin ASes and every peer that carries it. Only the kept routes show an AS path, a next hop and a time.

• GET /prefixlength/{length} - Up to 500 IPv4 prefixes with that mask length, picked at random (HTML). Each shows the origin AS that most peers carry it with, and the age of its newest kept route.

• GET /prefixlength6/{length} - The same for IPv6

• GET /ip/{addr} - The same JSON answer as /api/v1/ip/{ip}

• GET /ip/ - The same, for the caller's own address: the address the request came from or, when that is a trusted proxy (TRUSTED_PROXIES, see Configuration), the client address in its X-Forwarded-For

API v1

• GET /api/v1/stats - The /status.json figures plus connection, garbage collector, stream and handler queue statistics. connection_duration_seconds is the number of seconds the RIS Live feed has been connected, or null while it is not connected. stream.route_churn_per_min is the number of prefix announcements and withdrawals, from all peers, in the last full clock minute.

• GET /api/v1/ip/{ip} - Looks up each address. {ip} is an IP address, a host name, or a comma-separated list of at most 10 of these. results has one entry for each address. Every entry has the same fields:

  • ip: the address
  • ptr: the address's PTR names, or an empty list when it has none
  • blocklists: the public blocklists the address is on (see Blocklists), or an empty list when it is on none

  Every other field is about the address's live route. Each is null when the address has no live route. The fields about the route:

  • netblock: the most specific prefix that holds the address
  • mask_length: the netblock's mask length
  • ip_version: the netblock's IP version, 4 or 6
  • num_peers: the number of peers that carry the netblock
  • asn: the origin AS that the most peers carry the netblock with
  • num_origins: the number of origin ASes that peers carry the netblock with
  • origin_asns: those origin ASes, in AS number order
  • first_seen: when RouteWatch first saw the netblock, or null when the database has not stored the netblock yet
  • last_seen: the time of the netblock's newest kept route, or null when the netblock keeps no route

  The fields about the AS in asn come from the database. Each is also null when the database does not have it:

  • handle: the AS's handle
  • description: the AS's description
  • as_name: the AS's name in its WHOIS details
  • org_name: the name of the organisation that holds the AS
  • org_id: the organisation's ID at its registry
  • address: the organisation's postal address
  • country_code: the organisation's country code
  • abuse_email: the address that takes abuse reports for the AS
  • rir: the regional internet registry of the AS

  Of the addresses a host name resolves to, only public ones are looked up. A name with none, such as one that resolves to a private address, is reported as host name has no public address, without its addresses. errors has one entry for each host name that failed. The answer is 400 when:

  • {ip} lists more than 10
  • {ip} comes to more than 10 addresses once its names are resolved
  • nothing in {ip} is an IP address or a host name with a public address

  A PTR lookup gets 2 seconds. An address whose lookup takes longer is answered without PTR names.

• GET /api/v1/as/{asn} - The AS and the prefixes it originates, IPv4 and IPv6 apart, each with the time of its newest kept route. A prefix that keeps no route with the AS has no last_updated. When the database has no record of the AS, its handle and description come from the AS list built into RouteWatch. The answer then has no first_seen or last_seen.

• GET /api/v1/prefix/{prefix}/{length} - One route for each peer that carries the prefix, and the same routes grouped by origin AS in origins. Only the kept routes have as_path, next_hop and last_updated.

• GET /api/v1/search/as?q={text} - Finds the ASes whose handle or description contains {text}, ignoring case. data holds:

  • query: {text} without its leading and trailing spaces
  • total: the number of ASes that match
  • results: the first 50 of them in AS number order, each with asn, handle and description

  The search reads the AS list built into RouteWatch, not the database. The list comes from ipverse/as-metadata. A match can therefore be an AS that the database has no record of. A q that is missing, empty or only spaces answers 400.

A JSON answer is {"status": "ok", "data": ...}, or for an error {"status": "error", "error": {"msg": ..., "code": ...}}. The health check is the exception: its 503 answer is also {"status": "ok", "data": ...}, with error in data.status. Every JSON answer also carries an @meta object:

• time_zone: always UTC
• api_version: 1
• execution_time_ms: the number of milliseconds the request took

Every value with a unit is a number whose field name ends in the unit, such as _ms, _seconds or _bytes.

A request that runs past its time limit answers 504: the limit is 30 seconds, and 4 seconds for the database reads behind /status.json and /api/v1/stats. An AS number, prefix or mask length that does not parse answers 400, as does a mask length out of range for its IP version. An AS answers 404 when it originates no prefix and has no record in the database or in the AS list built into RouteWatch. AS 0 always answers 404, because it means no known origin. The AS name search never lists it. A prefix with no live route answers 404 too. The HTML pages give these errors as plain text. Under /api/v1 and /ip/, a path that matches no route answers 404 with the message Not found. A request with POST, PUT, PATCH, DELETE, OPTIONS, TRACE or CONNECT answers 405 with Method not allowed on these routes:

• the routes under /api/v1 and /ip/
• /status.json
• /.well-known/healthcheck.json

A 500 answer says only Internal server error. The error for a host name that does not resolve says only could not resolve host name. The detail goes to the log.

Every answer carries Strict-Transport-Security, Content-Security-Policy, X-Frame-Options, X-Content-Type-Options, Referrer-Policy and Permissions-Policy. The Content-Security-Policy lets a page load content only from the server, and does not allow inline content with 'unsafe-inline': the pages keep their CSS and JavaScript in inline <style> and <script> blocks, so each answer's policy names a random nonce made for that answer, and the browser applies or runs only the blocks that carry it. The pages therefore use no style attributes, which a nonce cannot cover.

An origin AS of 0 means the route has no known origin. Its AS path ends in an AS set of several members, any of which may have originated it. The 0 appears in these fields:

• origin_asn in a route
• asn and origin_asns in an IP lookup
• the key 0 in a prefix's origins

The pages show such a route as having no known origin.

Blocklists

The IP lookup reports which public blocklists list each address. Each result has a blocklists list with one entry for each list the address is on:

• name: the list's name, from the lists below
• description: what being on the list means
• updated: when RouteWatch last downloaded the list, or last heard from its publisher that its copy was current

An address on no list has an empty blocklists. An address with no live route still has a result with its blocklists.

The lists:

• dshield: the 20 networks that DShield saw scan the most targets in the last three days. From DShield, downloaded every hour.
• feodo_tracker: botnet command-and-control servers. From abuse.ch Feodo Tracker, downloaded every 15 minutes.
• firehol_abusers_30d: addresses reported for comment, form or forum spam in the last 30 days. From FireHOL, downloaded every 5 hours.
• firehol_level3: addresses reported for attacks, spyware or viruses in the last 30 days. From FireHOL, downloaded every 3 hours.
• spamhaus_drop and spamhaus_dropv6: IPv4 and IPv6 networks that Spamhaus says spammers or cyber criminals have hijacked or lease. From Spamhaus DROP, downloaded once a day. Spamhaus merged its EDROP list into DROP.
• tor_exits: Tor exit nodes, where traffic from the Tor network leaves. From the Tor Project, downloaded every hour.

No list is downloaded more often than its publisher updates it or allows. Each can be downloaded whole without an account or a key.

How RouteWatch keeps the lists:

• It holds each list in memory, so a lookup makes no outside request.
• It keeps a copy of each list in the blocklists directory of the state directory (see Configuration). Each copy is a file named after its list.
• Beside each copy, a file ending in .validators keeps the Last-Modified and ETag the publisher sent with that copy.
• At startup it loads each list from its copy. It downloads a list at once when the copy is older than the list's interval, or when there is no copy.
• A download sends the copy's Last-Modified as If-Modified-Since and its ETag as If-None-Match. The publisher then sends the list only when it has changed. Without either, the download asks for the whole list.
• A download fails when it takes over a minute or its file is over 16 MiB.
• When a download fails, or its file does not parse, RouteWatch keeps the previous copy and logs the failure. It tries again one interval later.

A successful IP lookup answer carries Cache-Control with a max-age of 15 minutes, the shortest list interval. The answer for the caller's own address (/ip/) is private, and the others are public.

Code Structure

routewatch/
├── cmd/
│   ├── routewatch/         # Main daemon entry point
│   ├── asinfo-gen/         # Utility to generate AS info data
│   └── streamdumper/       # Debug utility for raw stream output
├── internal/
│   ├── blocklist/          # Public IP blocklists: downloads and lookups
│   ├── bogon/              # Reserved prefixes and AS numbers
│   ├── routewatch/         # Core application logic
│   ├── server/             # HTTP server and handlers
│   ├── database/           # SQLite storage layer
│   ├── routetable/         # The whole routing table in memory
│   ├── streamer/           # RIPE RIS Live client
│   ├── ristypes/           # BGP message data structures
│   ├── logger/             # Structured logging wrapper
│   ├── metrics/            # Performance metrics tracking
│   ├── config/             # Configuration management
│   ├── templates/          # HTML templates
│   ├── version/            # Build version information
│   └── whois/              # WHOIS lookups of AS details
└── pkg/
    └── asinfo/             # AS information lookup (public API)

Architecture Overview

Component Relationships

internal/routewatch/app.go builds and starts these parts:

┌──────────────┐    ┌──────────────┐    ┌──────────────┐
│   Streamer   │───▶│   Handlers   │───▶│   Database   │
│              │    │              │    │              │
│ RIS Live     │    │ - ASHandler  │    │ SQLite with  │
│ HTTP stream  │    │ - PeerHandler│    │ WAL mode     │
│ client       │    │ - PrefixHdlr │    │              │
│              │    │ - PeeringHdlr│    │ Tables:      │
└──────────────┘    └──────┬───────┘    │ - asns       │
                           │            │ - prefixes   │
                           ▼            │ - routes     │
                    ┌──────────────┐    │ - route_peers│
                    │ Routing table│───▶│ - peerings   │
                    │ in memory    │    │ - bgp_peers  │
                    └──────────────┘    └──────────────┘
                           ▲                   ▲
┌──────────────┐    ┌──────┴───────┐           │
│    Server    │───▶│   Handlers   │───────────┘
│              │    │              │
│ Chi router   │    │ Status, API  │
│ port 8080    │    │ AS, Prefix   │
└──────────────┘    └──────────────┘

The prefix handler keeps the routing table in memory current. It writes the table's changes to the database every second. The web handlers read routes from the routing table, and everything else from the database.

Execution Flow

1. Startup (cmd/routewatch/main.go → internal/routewatch/cli.go)

   • Uber fx dependency injection initializes all components
   • Signal handlers registered for graceful shutdown

2. Initialization (internal/routewatch/app.go)

   • Database created with SQLite schema (WAL mode, page caches as in "Memory")
   • The routing table read from the routes table into memory, before the streamer connects. A full table takes about 25 seconds to read.
   • Message handlers registered with the streamer
   • HTTP server started on configured port

3. Message Processing Pipeline

   RIS Live Stream → JSON Parser → Message Dispatcher → Handler Queues
       ├→ Batch Writers → SQLite
       └→ PrefixHandler → Routing Table in Memory → SQLite, every second
   

   • Streamer connects to ris-live.ripe.net via HTTP
   • Parses BGP UPDATE messages from JSON stream
   • Dispatches to registered handlers based on message type
   • Each handler has its own queue with backpressure handling
   • Handlers batch writes for efficiency: the AS, peer and prefix handlers write 30,000, 10,000 and 25,000 updates at a time, or after 2, 2 and 1 seconds; the peering handler writes the peerings it has collected every 30 seconds

4. Handler Details

   • ASHandler: Tracks all ASNs seen in AS paths

   • PeerHandler: Records BGP peer information

   • PrefixHandler: Keeps the routing table in memory current. It applies each update to the table as it handles it:

     • An announcement gives the peer a route to the prefix.
     • A withdrawal removes the peer's route. So does an announcement whose AS path is empty or holds AS 0 anywhere.
     • A peer whose BGP session goes down leaves every prefix.

     Every second it writes the changed prefix and origin AS pairs to the routes table. A write takes at most 20,000 pairs, and the rest follow in the next writes. It also stores each new prefix with the time it was first seen. Peers are known by their IP address. A peer IP can have sessions on two collectors. A session going down on one then also removes the routes the other still carries, until they are announced again.

   • PeeringHandler: Extracts AS peering relationships from AS paths

5. HTTP Request Flow

   Request → Chi Router → Middleware (timeout, logging) → Handler → Routing Table or Database → Response
   

Key Design Patterns

• Routing Table in Memory: internal/routetable holds the whole routing table. For each prefix and origin AS it keeps:

  • every peer whose current route to the prefix has that origin AS, one bit per peer;
  • the newest two of those routes in full, with AS path and time.

  Of the other routes, only the peer is kept. The prefix handler writes the changed pairs to the routes table every second, one row per prefix and origin AS. Startup reads that table back into memory before the streamer connects. A crash loses only the changes not yet written.

• Batched Writes: The handlers write the feed's updates to the database in batches for performance

• Backpressure: Probabilistic message dropping when queues exceed 50% capacity

• Graceful Shutdown: on SIGTERM or SIGINT the streamer stops, the handlers write the batches they hold, the HTTP server stops and the database is checkpointed and closed. The daemon gives this 8 seconds and exits with status 0; "Running under upaas" says what a stop loses, logs and leaves undone when the 8 seconds run out.

• Reconnection: Any end of the stream other than shutdown, including RIS Live ending the response normally, marks the feed disconnected and reconnects with exponential backoff (5s-320s). The backoff resets after a stream that lasted more than 30s. A stream with no line for 60s is dropped and reconnected.

• IP Lookup: the routing table in memory finds the most specific prefix that holds the address. It looks up the address's prefix at each mask length, longest first: at most 25 lookups for IPv4 and 113 for IPv6. Prefixes shorter than /8 for IPv4 or /16 for IPv6, such as a default route, are not used.

Database Schema

-- Core tables
asns(asn PRIMARY KEY, handle, description,
     as_name, org_name, org_id, address, country_code, abuse_email,
     abuse_phone, tech_email, tech_phone, rir, rir_registration_date,
     rir_last_modified, whois_raw,
     first_seen, last_seen, whois_updated_at)
prefixes_v4(prefix PRIMARY KEY, first_seen) WITHOUT ROWID
prefixes_v6(prefix PRIMARY KEY, first_seen) WITHOUT ROWID

-- The routing table, one row per prefix and origin AS that some peer
-- carries: the peer set, and the newest two routes in full
routes(prefix, origin_asn, peers,
       newest_peer, newest_as_path, newest_time,
       older_peer, older_as_path, older_time,
       PRIMARY KEY (prefix, origin_asn)) WITHOUT ROWID
-- The peers of the routes table, each with its number in the peer sets
route_peers(number PRIMARY KEY, address UNIQUE)

-- Relationship tracking
peerings(id PRIMARY KEY, as_a, as_b, first_seen, last_seen,
         UNIQUE(as_a, as_b))
bgp_peers(id PRIMARY KEY, peer_ip UNIQUE, peer_asn, first_seen, last_seen,
          last_message_type)

-- Nothing in the daemon writes this table
announcements(id PRIMARY KEY, prefix_id, peer_asn, origin_asn, path,
              next_hop, timestamp, is_withdrawal)

The indexes are in internal/database/schema.sql.

Configuration

Configuration is handled via environment variables and OS-specific paths:

Variable	Default	Description
PORT	8080	HTTP server port, a whole number from 1 to 65535
DEBUG	(empty)	Set to routewatch for debug logging
XDG_DATA_HOME	/var/lib (in the Docker image)	Base of the state directory; must be an absolute path
GOMEMLIMIT	2560MiB (in the Docker image)	Go soft memory limit; see Memory
MALLOC_ARENA_MAX	2 (in the Docker image)	glibc malloc arena cap, a positive whole number; see Memory
TRUSTED_PROXIES	the RFC 1918 ranges	Addresses and CIDR ranges of trusted proxies; see below

A variable that is set to an invalid value stops the start with an error and a non-zero exit. An empty variable counts as unset. TRUSTED_PROXIES is the one exception: set but empty, it makes the server trust no proxy.

TRUSTED_PROXIES names the reverse proxies whose X-Forwarded-For the server believes:

• When it is not set, the server trusts the private ranges of RFC 1918:
  • 10.0.0.0/8
  • 172.16.0.0/12
  • 192.168.0.0/16
• When it lists addresses and CIDR ranges, comma-separated, such as 10.0.0.0/8,192.0.2.1, the server trusts those instead.
• When it is set but empty (TRUSTED_PROXIES=), the server trusts no proxy.

For a request from a trusted proxy, the request log and /ip/ use the last address in X-Forwarded-For that is not a trusted proxy. Each proxy adds the address it received the request from to the end, so the entries before it are whatever the client sent. For a request from any other address they use the address the request came from, whatever forwarding headers it carries. X-Real-IP and True-Client-IP are not read.

State directory (the database and the blocklist copies):

• macOS: ~/Library/Application Support/berlin.sneak.app.routewatch/
• Linux: /var/lib/berlin.sneak.app.routewatch/ when running as root, otherwise $XDG_DATA_HOME/berlin.sneak.app.routewatch/ (with XDG_DATA_HOME unset, ~/.local/share/berlin.sneak.app.routewatch/). In the Docker image this is /var/lib/berlin.sneak.app.routewatch/.

Memory

The daemon keeps the whole routing table in memory. The image sets ceilings that keep it inside a 5 GiB container.

Budget:

• Go heap: a 2.5 GiB soft limit (GOMEMLIMIT=2560MiB, set in the image).
  • The routing table takes about 700 bytes per prefix and origin AS. A full table has about 1.4 million of these pairs, so it takes about 1 GB.
  • The rest of the heap takes up to about 0.5 GiB. Most of it is the handlers' batches and queues, described below.
  • The remaining 1 GiB is room for the garbage collector. With much less, it would run almost all the time.
• SQLite: 656 MiB of page cache.
  • The connection that does every write has 512 MiB. It keeps the pages the writes touch from one write to the next. With a full routing table, the routes table takes about 450 MB, so its pages fit.
  • Each of the 9 reading connections has 16 MiB. SQLite empties a reading connection's cache whenever another connection has written.
  • A 768 MiB soft heap limit sits above the caches, so it does not shrink them. A 1 GiB hard heap limit caps the C library.
• glibc allocator: the SQLite C library allocates memory with glibc malloc.
  • glibc keeps freed page-cache chunks in per-arena free lists instead of returning them to the kernel. So the process's RSS follows the high-water mark of those arenas, not SQLite's live heap.
  • glibc creates up to eight arenas per core, so on a many-core host RSS grows with the core count.
  • The image sets MALLOC_ARENA_MAX=2 to bound it. Two arenas cost nothing here, because database writes already run one at a time.
• About 0.2 GiB for everything else in the runtime.

The ceilings add up to 3.7 GiB: the Go heap's 2.5 GiB, SQLite's 1 GiB and the 0.2 GiB for the rest.

Run the container with a memory limit of 5 GiB and swap disabled:

docker run --memory=5g --memory-swap=5g ...

or the equivalent in your deployment tool. This leaves headroom above the ceilings for spikes and the kernel page cache.

To replace the image's Go soft limit, set GOMEMLIMIT in the environment, for example -e GOMEMLIMIT=3GiB. A limit close to the routing table's size makes the garbage collector run almost all the time. MALLOC_ARENA_MAX can be overridden the same way, but raising it lets RSS climb again on a many-core host.

What happens at each limit:

• Go soft limit: as the heap approaches GOMEMLIMIT, the runtime runs garbage collection more aggressively rather than growing further.

• SQLite: at the 768 MiB soft heap limit it recycles its page cache instead of allocating more. At the 1 GiB hard heap limit a statement fails with an out-of-memory error, and the handler logs it. The prefix handler writes the routing table's changes again on a later write. The other writes drop their batch. The process keeps running.

• Handler queues: each of the four handler queues holds at most 20,000 messages. When a queue fills, the streamer drops messages instead of blocking, except a STATE message, which it holds, in feed order, until there is room: a few hundred bytes each, about ten a second.

• Handler batches: while the AS, peer or prefix handler writes a batch to the database, the updates that arrive collect in its next batch. That batch stops at ten times the batch size:

  • 300,000 updates for the AS handler (asnBatchMaxSize, about 10 MiB);
  • 100,000 for the peer handler (peerBatchMaxSize, about 10 MiB);
  • 250,000 for the prefix handler (prefixBatchMaxSize, about 15 MiB).

  At that size the handler takes nothing more off its queue until the write ends. The queue then fills, and the streamer drops messages as above. With the batches being written, the three handlers' batches take at most about 0.1 GiB of the Go heap.

With DEBUG=routewatch the daemon logs a System stats line every 60 seconds with the goroutine count and Go memory figures.

Running under upaas

What the upaas app needs:

• Container port: 8080.
• Volume: one, at container path /var/lib/berlin.sneak.app.routewatch.
• Environment: nothing. The server trusts a reverse proxy at an RFC 1918 address without TRUSTED_PROXIES (see Configuration). Set it only when the proxy connects from any other address. Leave XDG_DATA_HOME, GOMEMLIMIT and MALLOC_ARENA_MAX at the image's values. DEBUG=routewatch is optional and adds the System stats memory line to the log.
• Memory Limit: 5g, the 5 GiB limit from Memory above. upaas sets no swap limit, so on a host with swap Docker allows the same amount of swap again.
• Health check: the image's HEALTHCHECK requests /.well-known/healthcheck.json on the container port. upaas reads the container's health 60 seconds after a deploy and fails the deploy unless it is healthy. For the first 60 seconds, Docker 25 and later runs the check every 5 seconds, and the first pass makes the container healthy. Startup first reads the routing table, which takes about 25 seconds when the table is full.
• Stop grace: the container needs 9 seconds between SIGTERM and SIGKILL. upaas gives 10 (stopTimeoutSeconds in its internal/docker/client.go), as docker stop does by default. The daemon allows 8 seconds for the handlers to write what they hold and for the requests in progress to finish. Then it stops the HTTP server, closes the database and exits with status 0. A longer grace changes nothing. What is not written after 8 seconds is lost, and the streamer and each handler log how much.
  • The prefix handler's last write takes at most 20,000 of the routing table's changed pairs. A full-table peer's session going down changes about a million pairs, which take about a minute to write. A stop in that minute leaves the rest unwritten.
  • A lost withdrawal or peer down leaves its routes in the routes table, and the next start reads them back. They stay until the peer sends another update for the prefix, or its session goes down.
  • When the 8 seconds run out the database is not closed. When the database next opens, SQLite rolls back any write the exit cut off.

Development

# Run tests
make test

# Format code
make fmt

# Run linter
make lint

# Build
make build

Entrypoints

This repository adheres to the Scripts to Rule Them All standard: normalized scripts in script/ are the entrypoints for the development workflow, and the Makefile targets are thin shims that call them. We provide:

• script/bootstrap — install all development dependencies (git, make, go, the pinned node and yarn, prettier from yarn.lock, Go module download)
• script/setup — make a fresh clone ready for development: runs script/bootstrap, then script/install-precommit
• script/projectname — print the project name (used for the Docker image tag)
• script/test — build the test phase of the Dockerfile alone, which runs go test -short -count=1 -timeout 90s -race -cover ./... (verbose rerun on failure). The -short flag skips the live-network integration test so the run is deterministic and offline.
• script/lint — build the lint phase of the Dockerfile alone, which runs golangci-lint v2.14.0 with .golangci.yml
• script/fmt — format the Go code with go fmt and the markdown with prettier (writes)
• script/fmt-check — check formatting (read-only)
• script/check — run script/test, script/lint, and script/fmt-check
• script/docker — build the Docker image tagged via script/projectname
• script/cibuild — CI entrypoint: runs script/bootstrap and script/check, then builds the image
• script/precommit — pre-commit gate: go mod tidy (must not change go.mod or go.sum) and go fmt (writes), then script/check
• script/install-precommit — install the git pre-commit hook that runs script/precommit
• script/asupdate — rebuild the AS list built into RouteWatch, pkg/asinfo/asdata.json.gz, from the as.csv file of ipverse/as-metadata. It needs the network. It runs cmd/asinfo-gen, which fails when it finds no AS. When cmd/asinfo-gen fails, the old list stays and the script exits non-zero.

Tests and the linter run only in Docker, as the test and lint phases of the Dockerfile; the image cannot be built unless both pass. Every docker build in script/ passes a new CACHEBUST value, so each run executes every check, while Go's build cache and golangci-lint's cache are kept between builds.

The live-network integration test TestRouteWatchLiveFeed streams the RIPE RIS feed for a few seconds and is skipped in short mode. To run it on demand, invoke go test directly without -short: go test -run TestRouteWatchLiveFeed ./internal/routewatch/.

License

MIT. See LICENSE.

Author

@sneak