clawbot
08f060045b
check / check (push) Successful in 3m44s
After the earlier memory work, SQLite's live heap is bounded but process RSS still climbed about 70 MiB/min, nearly all anonymous and outside the Go runtime and outside SQLite's own accounting, without any SQLITE_NOMEM. The cause is glibc: the SQLite C library allocates and frees millions of small page-cache chunks from many threads, and glibc keeps each arena's freed chunks resident. With arenas uncapped it creates up to eight per core, so on a large host RSS grows with the core count. Set MALLOC_ARENA_MAX=2 in the image to bound the retained memory. Writes are already serialized, so the two-arena cap costs no throughput. Update the README Memory section. Model: opus-4-8
278 lines
12 KiB
Markdown
278 lines
12 KiB
Markdown
# RouteWatch
|
|
|
|
RouteWatch is a real-time BGP routing table monitor that streams BGP UPDATE messages from the RIPE RIS Live service, maintains a live routing table in SQLite, and provides HTTP APIs for querying routing information.
|
|
|
|
|
|
## Features
|
|
|
|
- Real-time streaming of BGP updates from RIPE RIS Live
|
|
- Maintains live IPv4 and IPv6 routing tables
|
|
- Tracks AS peering relationships
|
|
- HTTP API for IP-to-AS lookups, prefix details, and AS information
|
|
- Automatic reconnection with exponential backoff
|
|
- Batched database writes for high performance
|
|
- Backpressure handling to prevent memory exhaustion
|
|
|
|
## Installation
|
|
|
|
```bash
|
|
go build -o routewatch ./cmd/routewatch
|
|
```
|
|
|
|
## Usage
|
|
|
|
```bash
|
|
# Run the daemon (listens on port 8080 by default)
|
|
./routewatch
|
|
|
|
# Set custom port
|
|
PORT=3000 ./routewatch
|
|
|
|
# Enable debug logging
|
|
DEBUG=routewatch ./routewatch
|
|
```
|
|
|
|
## HTTP Endpoints
|
|
|
|
### Web Interface
|
|
- `GET /` - Redirects to /status
|
|
- `GET /status` - HTML status dashboard
|
|
- `GET /status.json` - JSON statistics
|
|
- `GET /as/{asn}` - AS detail page (HTML)
|
|
- `GET /prefix/{prefix}` - Prefix detail page (HTML)
|
|
- `GET /prefixlength/{length}` - IPv4 prefixes by mask length
|
|
- `GET /prefixlength6/{length}` - IPv6 prefixes by mask length
|
|
- `GET /ip/{ip}` - Redirects to prefix containing the IP
|
|
|
|
### API v1
|
|
- `GET /api/v1/stats` - Detailed statistics with handler metrics
|
|
- `GET /api/v1/ip/{ip}` - Look up AS information for an IP address
|
|
- `GET /api/v1/as/{asn}` - Get prefixes announced by an AS
|
|
- `GET /api/v1/prefix/{prefix}` - Get routes for a specific prefix
|
|
|
|
## Code Structure
|
|
|
|
```
|
|
routewatch/
|
|
├── cmd/
|
|
│ ├── routewatch/ # Main daemon entry point
|
|
│ ├── asinfo-gen/ # Utility to generate AS info data
|
|
│ └── streamdumper/ # Debug utility for raw stream output
|
|
├── internal/
|
|
│ ├── routewatch/ # Core application logic
|
|
│ ├── server/ # HTTP server and handlers
|
|
│ ├── database/ # SQLite storage layer
|
|
│ ├── streamer/ # RIPE RIS Live client
|
|
│ ├── ristypes/ # BGP message data structures
|
|
│ ├── logger/ # Structured logging wrapper
|
|
│ ├── metrics/ # Performance metrics tracking
|
|
│ ├── config/ # Configuration management
|
|
│ └── templates/ # HTML templates
|
|
└── pkg/
|
|
└── asinfo/ # AS information lookup (public API)
|
|
```
|
|
|
|
## Architecture Overview
|
|
|
|
### Component Relationships
|
|
|
|
```
|
|
┌─────────────────────────────────────────────────────────────────┐
|
|
│ RouteWatch │
|
|
│ (internal/routewatch/app.go - main orchestrator) │
|
|
├─────────────────────────────────────────────────────────────────┤
|
|
│ │
|
|
│ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ │
|
|
│ │ Streamer │───▶│ Handlers │───▶│ Database │ │
|
|
│ │ │ │ │ │ │ │
|
|
│ │ RIS Live │ │ - ASHandler │ │ SQLite with │ │
|
|
│ │ WebSocket │ │ - PeerHandler│ │ WAL mode │ │
|
|
│ │ client │ │ - PrefixHdlr │ │ │ │
|
|
│ │ │ │ - PeeringHdlr│ │ Tables: │ │
|
|
│ └──────────────┘ └──────────────┘ │ - asns │ │
|
|
│ │ - prefixes │ │
|
|
│ ┌──────────────┐ ┌──────────────┐ │ - live_routes│ │
|
|
│ │ Server │───▶│ Handlers │───▶│ - peerings │ │
|
|
│ │ │ │ │ │ - bgp_peers │ │
|
|
│ │ Chi router │ │ Status, API │ └──────────────┘ │
|
|
│ │ port 8080 │ │ AS, Prefix │ │
|
|
│ └──────────────┘ └──────────────┘ │
|
|
│ │
|
|
└─────────────────────────────────────────────────────────────────┘
|
|
```
|
|
|
|
### Execution Flow
|
|
|
|
1. **Startup** (`cmd/routewatch/main.go` → `internal/routewatch/cli.go`)
|
|
- Uber fx dependency injection initializes all components
|
|
- Signal handlers registered for graceful shutdown
|
|
|
|
2. **Initialization** (`internal/routewatch/app.go`)
|
|
- Database created with SQLite schema (WAL mode, 3GB cache)
|
|
- Message handlers registered with the streamer
|
|
- HTTP server started on configured port
|
|
|
|
3. **Message Processing Pipeline**
|
|
```
|
|
RIS Live Stream → JSON Parser → Message Dispatcher → Handler Queues → Batch Writers → SQLite
|
|
```
|
|
- Streamer connects to `ris-live.ripe.net` via HTTP
|
|
- Parses BGP UPDATE messages from JSON stream
|
|
- Dispatches to registered handlers based on message type
|
|
- Each handler has its own queue with backpressure handling
|
|
- Handlers batch writes for efficiency (25K-30K ops, 1-2s timeout)
|
|
|
|
4. **Handler Details**
|
|
- **ASHandler**: Tracks all ASNs seen in AS paths
|
|
- **PeerHandler**: Records BGP peer information
|
|
- **PrefixHandler**: Maintains live routing table (upserts on announcement, deletes on withdrawal)
|
|
- **PeeringHandler**: Extracts AS peering relationships from AS paths
|
|
|
|
5. **HTTP Request Flow**
|
|
```
|
|
Request → Chi Router → Middleware (timeout, logging) → Handler → Database Query → Response
|
|
```
|
|
|
|
### Key Design Patterns
|
|
|
|
- **Batched Writes**: All database operations are batched for performance
|
|
- **Backpressure**: Probabilistic message dropping when queues exceed 50% capacity
|
|
- **Graceful Shutdown**: 60-second timeout, flushes all pending batches
|
|
- **Reconnection**: Exponential backoff (5s-320s) with reset after 30s of stable connection
|
|
- **IPv4 Optimization**: IP ranges stored as uint32 for O(1) lookups
|
|
|
|
### Database Schema
|
|
|
|
```sql
|
|
-- Core tables
|
|
asns(id, number, handle, description, first_seen, last_seen)
|
|
prefixes_v4(id, prefix, mask_length, first_seen, last_seen)
|
|
prefixes_v6(id, prefix, mask_length, first_seen, last_seen)
|
|
|
|
-- Live routing tables (one per IP version)
|
|
live_routes_v4(id, prefix, mask_length, origin_asn, peer_ip, as_path,
|
|
next_hop, last_updated, v4_ip_start, v4_ip_end)
|
|
live_routes_v6(id, prefix, mask_length, origin_asn, peer_ip, as_path,
|
|
next_hop, last_updated)
|
|
|
|
-- Relationship tracking
|
|
peerings(id, as_a, as_b, first_seen, last_seen)
|
|
bgp_peers(id, peer_ip, peer_asn, last_message_type, last_seen)
|
|
```
|
|
|
|
## Configuration
|
|
|
|
Configuration is handled via environment variables and OS-specific paths:
|
|
|
|
| Variable | Default | Description |
|
|
|----------|----------|-------------|
|
|
| `PORT` | `8080` | HTTP server port |
|
|
| `DEBUG` | (empty) | Set to `routewatch` for debug logging |
|
|
| `GOMEMLIMIT` | `1536MiB` (in the Docker image) | Go soft memory limit; see Memory |
|
|
| `MALLOC_ARENA_MAX` | `2` (in the Docker image) | glibc malloc arena cap; see Memory |
|
|
|
|
State directory (database location):
|
|
- macOS: `~/Library/Application Support/routewatch/`
|
|
- Linux: `/var/lib/routewatch/` or `~/.local/share/routewatch/`
|
|
|
|
## Memory
|
|
|
|
The daemon holds a live routing table, so its memory grows with the size of the
|
|
data it tracks. The image sets ceilings that keep it inside a 5 GiB container.
|
|
|
|
Budget:
|
|
- Go heap: a 1.5 GiB soft limit (`GOMEMLIMIT=1536MiB`, set in the image).
|
|
- SQLite: at most 640 MiB of page cache across the connection pool (64 MiB per
|
|
connection, 10 connections) and a 1.5 GiB hard heap limit for the C library.
|
|
- glibc allocator: the SQLite C library runs on glibc `malloc`, which frees
|
|
page-cache chunks back to per-arena free lists rather than to the kernel, so
|
|
process RSS tracks the high-water mark of those arenas, not SQLite's live
|
|
heap. glibc creates up to eight arenas per core, so on a many-core host the
|
|
retained memory — and thus RSS — grows with the core count. The image sets
|
|
`MALLOC_ARENA_MAX=2` to bound it; the two-arena cap costs nothing here because
|
|
database writes are already serialized.
|
|
- About 0.2 GiB for everything else in the runtime.
|
|
|
|
Run the container with a memory limit of 5 GiB and swap disabled:
|
|
|
|
```bash
|
|
docker run --memory=5g --memory-swap=5g ...
|
|
```
|
|
|
|
or the equivalent in your deployment tool. This leaves headroom above the
|
|
ceilings for spikes and the kernel page cache.
|
|
|
|
Override the Go soft limit by setting `GOMEMLIMIT` in the environment (for
|
|
example `-e GOMEMLIMIT=1GiB`); this replaces the image default. `MALLOC_ARENA_MAX`
|
|
can be overridden the same way, but raising it lets RSS climb again on a
|
|
many-core host.
|
|
|
|
What happens at each limit:
|
|
- Go soft limit: as the heap approaches `GOMEMLIMIT`, the runtime runs garbage
|
|
collection more aggressively rather than growing further.
|
|
- SQLite: at a 1 GiB soft heap limit it recycles its page cache instead of
|
|
allocating more; at the 1.5 GiB hard heap limit a statement fails with an
|
|
out-of-memory error, and the handler logs it and drops that batch. The process
|
|
keeps running.
|
|
- Handler queues: each of the four handler queues holds at most 20,000 messages.
|
|
When a queue fills, the streamer drops messages instead of blocking.
|
|
|
|
With `DEBUG=routewatch` the daemon logs a `System stats` line every 60 seconds
|
|
with the goroutine count and Go memory figures.
|
|
|
|
## Development
|
|
|
|
```bash
|
|
# Run tests
|
|
make test
|
|
|
|
# Format code
|
|
make fmt
|
|
|
|
# Run linter
|
|
make lint
|
|
|
|
# Build
|
|
make
|
|
```
|
|
|
|
## Entrypoints
|
|
|
|
This repository adheres to the
|
|
[Scripts to Rule Them All](https://github.com/github/scripts-to-rule-them-all)
|
|
standard: normalized scripts in `script/` are the entrypoints for the
|
|
development workflow, and the Makefile targets are thin shims that call
|
|
them. We provide:
|
|
|
|
- `script/bootstrap` — install all development dependencies (go,
|
|
golangci-lint, Go module download)
|
|
- `script/setup` — make a fresh clone ready for development: runs
|
|
`script/bootstrap`, then `script/install-precommit`
|
|
- `script/projectname` — print the project name (used for the Docker
|
|
image tag)
|
|
- `script/test` — run the test suite
|
|
(`go test -short -timeout 30s -race -cover ./...`, verbose rerun on
|
|
failure). The `-short` flag skips the live-network integration test so the
|
|
default run is deterministic and offline.
|
|
- `script/lint` — run `go vet ./...` and `golangci-lint run`
|
|
- `script/fmt` — format all code (writes)
|
|
- `script/fmt-check` — check formatting (read-only)
|
|
- `script/check` — run `script/test`, `script/lint`, and
|
|
`script/fmt-check`
|
|
- `script/docker` — build the Docker image tagged via
|
|
`script/projectname`
|
|
- `script/cibuild` — CI entrypoint: `docker build .`
|
|
- `script/precommit` — pre-commit gate: `go mod tidy` + `go fmt` (must
|
|
not change files), then `script/check`
|
|
- `script/install-precommit` — install the git pre-commit hook that
|
|
runs `script/precommit`
|
|
|
|
The live-network integration test `TestRouteWatchLiveFeed` streams the RIPE
|
|
RIS feed for a few seconds and is skipped in short mode. To run it on demand,
|
|
invoke `go test` directly without `-short`:
|
|
`go test -run TestRouteWatchLiveFeed ./internal/routewatch/`.
|
|
|
|
## License
|
|
|
|
See LICENSE file.
|