sneak/routewatch
1
0
Fork 0
Code Issues 4 Pull Requests 2 Actions 19 Packages Projects Releases Wiki Activity

Remove the per-peer route tables (closes #122)
check / check (push) Waiting to run
Details

Browse Source 
The routing table in memory now holds every route, so the two per-peer route
tables go, with every write, delete, count and read that served them. The
prefix handler applies each update to the table, writes the table's changes
every second, and stores each new prefix with its first-seen time. The peer
delete, its retry on the next flush and the updates held for it go too. The
types and errors of the server's answers move to internal/server.

The image's memory settings fit a full table of about 1 GB: GOMEMLIMIT of
2.5 GiB, a 512 MiB page cache for the writing connection, and SQLite heap
limits of 768 MiB and 1 GiB. The README describes the new table, its
database copy and the memory budget.

Model: opus-5-5
This commit is contained in:
sneak
2026-10-27 16:10:27 +00:00
parent 506e289a63
commit 697f1ffd3e
23 changed files with 701 additions and 5273 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+126 -96
View File
@@ -4,7 +4,7 @@ RouteWatch is an MIT-licensed Go daemon by @sneak that monitors the BGP routing
table in real time. It works like this:
- It streams BGP UPDATE messages from the RIPE RIS Live service.
- It keeps a live routing table in SQLite.
- It keeps the whole routing table in memory, with a copy in SQLite.
- It serves web pages and a JSON API for looking routes up.
## Rationale
@@ -268,31 +268,36 @@ routewatch/
### Component Relationships
`internal/routewatch/app.go` builds and starts these parts:
```
┌─────────────────────────────────────────────────────────────────┐
│ RouteWatch │
│ (internal/routewatch/app.go - main orchestrator) │
├─────────────────────────────────────────────────────────────────┤
│ │
│ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ │
│ │ Streamer │───▶│ Handlers │───▶│ Database │ │
│ │ │ │ │ │ │ │
│ │ RIS Live │ │ - ASHandler │ │ SQLite with │ │
│ │ HTTP stream │ │ - PeerHandler│ │ WAL mode │ │
│ │ client │ │ - PrefixHdlr │ │ │ │
│ │ │ │ - PeeringHdlr│ │ Tables: │ │
│ └──────────────┘ └──────────────┘ │ - asns │ │
│ │ - prefixes │ │
│ ┌──────────────┐ ┌──────────────┐ │ - live_routes│ │
│ │ Server │───▶│ Handlers │───▶│ - peerings │ │
│ │ │ │ │ │ - bgp_peers │ │
│ │ Chi router │ │ Status, API │ └──────────────┘ │
│ │ port 8080 │ │ AS, Prefix │ │
│ └──────────────┘ └──────────────┘ │
│ │
└─────────────────────────────────────────────────────────────────┘
┌──────────────┐ ┌──────────────┐ ┌──────────────┐
│ Streamer │───▶│ Handlers │───▶│ Database │
│ │ │ │ │ │
│ RIS Live │ │ - ASHandler │ │ SQLite with │
│ HTTP stream │ │ - PeerHandler│ │ WAL mode │
│ client │ │ - PrefixHdlr │ │ │
│ │ │ - PeeringHdlr│ │ Tables: │
└──────────────┘ └──────┬───────┘ │ - asns │
│ │ - prefixes │
▼ │ - routes │
┌──────────────┐ │ - route_peers│
│ Routing table│───▶│ - peerings │
│ in memory │ │ - bgp_peers │
└──────────────┘ └──────────────┘
▲ ▲
┌──────────────┐ ┌──────┴───────┐ │
│ Server │───▶│ Handlers │───────────┘
│ │ │ │
│ Chi router │ │ Status, API │
│ port 8080 │ │ AS, Prefix │
└──────────────┘ └──────────────┘
```
The prefix handler keeps the routing table in memory current. It writes the
table's changes to the database every second. The web handlers read routes from
the routing table, and everything else from the database.
### Execution Flow
1. **Startup** (`cmd/routewatch/main.go` → `internal/routewatch/cli.go`)
@@ -301,15 +306,17 @@ routewatch/
2. **Initialization** (`internal/routewatch/app.go`)
- Database created with SQLite schema (WAL mode, page caches as in "Memory")
- The routing table read from the database into memory, before the streamer
connects
- The routing table read from the `routes` table into memory, before the
streamer connects. A full table takes about 25 seconds to read.
- Message handlers registered with the streamer
- HTTP server started on configured port
3. **Message Processing Pipeline**
```
RIS Live Stream → JSON Parser → Message Dispatcher → Handler Queues → Batch Writers → SQLite
RIS Live Stream → JSON Parser → Message Dispatcher → Handler Queues
├→ Batch Writers → SQLite
└→ PrefixHandler → Routing Table in Memory → SQLite, every second
```
- Streamer connects to `ris-live.ripe.net` via HTTP
@@ -324,24 +331,41 @@ routewatch/
4. **Handler Details**
- **ASHandler**: Tracks all ASNs seen in AS paths
- **PeerHandler**: Records BGP peer information
- **PrefixHandler**: Maintains live routing table (upserts on announcement,
deletes on withdrawal or on an announcement whose AS path is empty or
holds AS 0 anywhere, deletes all of a peer's routes when its BGP session
goes down). Routes are kept per peer IP, so when one peer IP has sessions
on two collectors, a session going down on one also removes the routes the
other still carries, until they are announced again. It also applies each
update to the routing table in memory, and writes the table's changes to
the database every second. A write takes at most 20,000 changed prefix and
origin AS pairs, and the rest follow in the next writes.
- **PrefixHandler**: Keeps the routing table in memory current. It applies
each update to the table as it handles it:
- An announcement gives the peer a route to the prefix.
- A withdrawal removes the peer's route. So does an announcement whose
AS path is empty or holds AS 0 anywhere.
- A peer whose BGP session goes down leaves every prefix.
Every second it writes the changed prefix and origin AS pairs to the
`routes` table. A write takes at most 20,000 pairs, and the rest follow
in the next writes. It also stores each new prefix with the time it was
first seen. Peers are known by their IP address. A peer IP can have
sessions on two collectors. A session going down on one then also
removes the routes the other still carries, until they are announced
again.
- **PeeringHandler**: Extracts AS peering relationships from AS paths
5. **HTTP Request Flow**
```
Request → Chi Router → Middleware (timeout, logging) → Handler → Database Query → Response
Request → Chi Router → Middleware (timeout, logging) → Handler → Routing Table or Database → Response
```
### Key Design Patterns
- **Routing Table in Memory**: `internal/routetable` holds the whole routing
table. For each prefix and origin AS it keeps:
- every peer whose current route to the prefix has that origin AS, one bit
per peer;
- the newest two of those routes in full, with AS path and time.
Of the other routes, only the peer is kept. The prefix handler writes the
changed pairs to the `routes` table every second, one row per prefix and
origin AS. Startup reads that table back into memory before the streamer
connects. A crash loses only the changes not yet written.
- **Batched Writes**: The handlers write the feed's updates to the database in
batches for performance
- **Backpressure**: Probabilistic message dropping when queues exceed 50%
@@ -373,13 +397,14 @@ asns(asn PRIMARY KEY, handle, description,
prefixes_v4(prefix PRIMARY KEY, first_seen) WITHOUT ROWID
prefixes_v6(prefix PRIMARY KEY, first_seen) WITHOUT ROWID
-- Live routing tables (one per IP version), one route per peer and prefix
live_routes_v4(id INTEGER PRIMARY KEY, prefix, mask_length, origin_asn,
peer_ip, as_path, next_hop, last_updated,
UNIQUE(peer_ip, prefix))
live_routes_v6(id INTEGER PRIMARY KEY, prefix, mask_length, origin_asn,
peer_ip, as_path, next_hop, last_updated,
UNIQUE(peer_ip, prefix))
-- The routing table, one row per prefix and origin AS that some peer
-- carries: the peer set, and the newest two routes in full
routes(prefix, origin_asn, peers,
newest_peer, newest_as_path, newest_time,
older_peer, older_as_path, older_time,
PRIMARY KEY (prefix, origin_asn)) WITHOUT ROWID
-- The peers of the routes table, each with its number in the peer sets
route_peers(number PRIMARY KEY, address UNIQUE)
-- Relationship tracking
peerings(id PRIMARY KEY, as_a, as_b, first_seen, last_seen,
@@ -403,7 +428,7 @@ Configuration is handled via environment variables and OS-specific paths:
| `PORT` | `8080` | HTTP server port, a whole number from 1 to 65535 |
| `DEBUG` | (empty) | Set to `routewatch` for debug logging |
| `XDG_DATA_HOME` | `/var/lib` (in the Docker image) | Base of the state directory; must be an absolute path |
| `GOMEMLIMIT` | `1536MiB` (in the Docker image) | Go soft memory limit; see Memory |
| `GOMEMLIMIT` | `2560MiB` (in the Docker image) | Go soft memory limit; see Memory |
| `MALLOC_ARENA_MAX` | `2` (in the Docker image) | glibc malloc arena cap, a positive whole number; see Memory |
| `TRUSTED_PROXIES` | (empty) | Addresses and CIDR ranges of trusted proxies; see below |
@@ -429,28 +454,37 @@ State directory (the database and the blocklist copies):
## Memory
The daemon holds a live routing table, so its memory grows with the size of the
data it tracks. The image sets ceilings that keep it inside a 5 GiB container.
The daemon keeps the whole routing table in memory. The image sets ceilings that
keep it inside a 5 GiB container.
Budget:
- Go heap: a 1.5 GiB soft limit (`GOMEMLIMIT=1536MiB`, set in the image).
- SQLite: 1 GiB of page cache for the one connection every write uses, which
keeps the pages the route writes touch from one batch to the next, and 16 MiB
for each of the 9 reading connections, whose cache SQLite empties whenever
another connection has written: about 1.15 GiB in all. A 1.5 GiB soft heap
limit sits above the caches, so it does not shrink them, and a 2 GiB hard heap
limit caps the C library.
- glibc allocator: the SQLite C library runs on glibc `malloc`, which frees
page-cache chunks back to per-arena free lists rather than to the kernel, so
process RSS tracks the high-water mark of those arenas, not SQLite's live
heap. glibc creates up to eight arenas per core, so on a many-core host the
retained memory — and thus RSS — grows with the core count. The image sets
`MALLOC_ARENA_MAX=2` to bound it; the two-arena cap costs nothing here because
database writes are already serialized.
- Go heap: a 2.5 GiB soft limit (`GOMEMLIMIT=2560MiB`, set in the image).
- The routing table takes about 700 bytes per prefix and origin AS. A full
table has about 1.4 million of these pairs, so it takes about 1 GB.
- The rest of the heap takes up to about 0.5 GiB. Most of it is the
handlers' batches and queues, described below.
- The remaining 1 GiB is room for the garbage collector. With much less, it
would run almost all the time.
- SQLite: 656 MiB of page cache.
- The connection that does every write has 512 MiB. It keeps the pages the
writes touch from one write to the next. With a full routing table, the
`routes` table takes about 450 MB, so its pages fit.
- Each of the 9 reading connections has 16 MiB. SQLite empties a reading
connection's cache whenever another connection has written.
- A 768 MiB soft heap limit sits above the caches, so it does not shrink
them. A 1 GiB hard heap limit caps the C library.
- glibc allocator: the SQLite C library allocates memory with glibc `malloc`.
- glibc keeps freed page-cache chunks in per-arena free lists instead of
returning them to the kernel. So the process's RSS follows the high-water
mark of those arenas, not SQLite's live heap.
- glibc creates up to eight arenas per core, so on a many-core host RSS
grows with the core count.
- The image sets `MALLOC_ARENA_MAX=2` to bound it. Two arenas cost nothing
here, because database writes already run one at a time.
- About 0.2 GiB for everything else in the runtime.
The ceilings add up to 3.7 GiB: the Go heap's 1.5 GiB, SQLite's 2 GiB and the
The ceilings add up to 3.7 GiB: the Go heap's 2.5 GiB, SQLite's 1 GiB and the
0.2 GiB for the rest.
Run the container with a memory limit of 5 GiB and swap disabled:
@@ -462,38 +496,36 @@ docker run --memory=5g --memory-swap=5g ...
or the equivalent in your deployment tool. This leaves headroom above the
ceilings for spikes and the kernel page cache.
Override the Go soft limit by setting `GOMEMLIMIT` in the environment (for
example `-e GOMEMLIMIT=1GiB`); this replaces the image default.
`MALLOC_ARENA_MAX` can be overridden the same way, but raising it lets RSS climb
again on a many-core host.
To replace the image's Go soft limit, set `GOMEMLIMIT` in the environment, for
example `-e GOMEMLIMIT=3GiB`. A limit close to the routing table's size makes
the garbage collector run almost all the time. `MALLOC_ARENA_MAX` can be
overridden the same way, but raising it lets RSS climb again on a many-core
host.
What happens at each limit:
- Go soft limit: as the heap approaches `GOMEMLIMIT`, the runtime runs garbage
collection more aggressively rather than growing further.
- SQLite: at the 1.5 GiB soft heap limit it recycles its page cache instead of
allocating more; at the 2 GiB hard heap limit a statement fails with an
out-of-memory error, and the handler logs it and drops that batch. The process
keeps running.
- SQLite: at the 768 MiB soft heap limit it recycles its page cache instead of
allocating more. At the 1 GiB hard heap limit a statement fails with an
out-of-memory error, and the handler logs it. The prefix handler writes the
routing table's changes again on a later write. The other writes drop their
batch. The process keeps running.
- Handler queues: each of the four handler queues holds at most 20,000 messages.
When a queue fills, the streamer drops messages instead of blocking, except a
`STATE` message, which it holds, in feed order, until there is room: a few
hundred bytes each, about ten a second.
- Handler batches: while the AS, peer or prefix handler writes a batch to the
database, the updates that arrive collect in its next batch, which stops at
ten times the batch size: 300,000 updates for the AS handler
(`asnBatchMaxSize`, about 10 MiB), 100,000 for the peer handler
(`peerBatchMaxSize`, about 10 MiB) and 250,000 for the prefix handler
(`prefixBatchMaxSize`, about 60 MiB). At that size the handler takes nothing
more off its queue until the write ends, so the queue fills and the streamer
drops messages as above. When the delete of a peer's live routes after its
session went down fails, the prefix handler keeps the updates the peer sent
after the down for the next write, which retries the delete first; on top of
its 250,000 it keeps at most the last 25,000 of them (`prefixBatchSize`, about
6 MiB) and drops and logs the older ones, so while the delete keeps failing
the other peers' updates still come off the queue. With the batches being
written, the three handlers' batches take at most about 0.25 GiB of the Go
heap's 1.5 GiB.
database, the updates that arrive collect in its next batch. That batch stops
at ten times the batch size:
- 300,000 updates for the AS handler (`asnBatchMaxSize`, about 10 MiB);
- 100,000 for the peer handler (`peerBatchMaxSize`, about 10 MiB);
- 250,000 for the prefix handler (`prefixBatchMaxSize`, about 15 MiB).
At that size the handler takes nothing more off its queue until the write
ends. The queue then fills, and the streamer drops messages as above. With
the batches being written, the three handlers' batches take at most about
0.1 GiB of the Go heap.
With `DEBUG=routewatch` the daemon logs a `System stats` line every 60 seconds
with the goroutine count and Go memory figures.
@@ -521,21 +553,19 @@ What the [upaas](https://git.eeqj.de/sneak/upaas) app needs:
- Stop grace: the container needs 9 seconds between SIGTERM and SIGKILL. upaas
gives 10 (`stopTimeoutSeconds` in its `internal/docker/client.go`), as
`docker stop` does by default. The daemon allows 8 seconds for the handlers to
write what they hold and for the requests in progress to finish, then stops
write what they hold and for the requests in progress to finish. Then it stops
the HTTP server, closes the database and exits with status 0. A longer grace
changes nothing. What is not written after 8 seconds is lost, and the streamer
and each handler log how much. When the prefix handler has fallen behind the
feed, a stop loses far more: its batch is full at 250,000 updates and the
streamer's worker for it waits for room, so the streamer's stop takes the
whole 8 seconds and every handler then skips its last write. Lost then are the
messages left on the four handlers' queues, what the AS, peer and peering
handlers hold, the prefix handler's batch, and the write it has in progress,
up to another 250,000 updates, which take about 20 seconds and are cut off by
the exit; withdrawals and peer downs are among them. A route withdrawn in them
stays in the live route table until its peer announces or withdraws the prefix
again or its session goes down. When the 8 seconds run out the database is not
closed, and SQLite rolls back a write the exit cut off when the database next
opens.
and each handler log how much.
- The prefix handler's last write takes at most 20,000 of the routing
table's changed pairs. A full-table peer's session going down changes
about a million pairs, which take about a minute to write. A stop in that
minute leaves the rest unwritten.
- A lost withdrawal or peer down leaves its routes in the `routes` table,
and the next start reads them back. They stay until the peer sends another
update for the prefix, or its session goes down.
- When the 8 seconds run out the database is not closed. When the database
next opens, SQLite rolls back any write the exit cut off.
## Development