clawbot
b0cd884019
TestRouteWatchLiveFeed streams the live RIPE RIS feed for a few seconds, so the default test run depended on the network and tripped the race detector. It now skips under testing.Short(), and script/test passes -short in both go test lines, so make check runs offline and gives the same result every time. The README says how to run the live test on demand. No non-test code changed. The race itself is not fixed: the test reads the peeringHandler field of RouteWatch while Run is still setting it. Nothing outside the test reads that field concurrently. Model: opus-4-8 (implementation, review); fable-5-1 (summary)
231 lines
9.4 KiB
Markdown
231 lines
9.4 KiB
Markdown
# RouteWatch
|
|
|
|
RouteWatch is a real-time BGP routing table monitor that streams BGP UPDATE messages from the RIPE RIS Live service, maintains a live routing table in SQLite, and provides HTTP APIs for querying routing information.
|
|
|
|
|
|
## Features
|
|
|
|
- Real-time streaming of BGP updates from RIPE RIS Live
|
|
- Maintains live IPv4 and IPv6 routing tables
|
|
- Tracks AS peering relationships
|
|
- HTTP API for IP-to-AS lookups, prefix details, and AS information
|
|
- Automatic reconnection with exponential backoff
|
|
- Batched database writes for high performance
|
|
- Backpressure handling to prevent memory exhaustion
|
|
|
|
## Installation
|
|
|
|
```bash
|
|
go build -o routewatch ./cmd/routewatch
|
|
```
|
|
|
|
## Usage
|
|
|
|
```bash
|
|
# Run the daemon (listens on port 8080 by default)
|
|
./routewatch
|
|
|
|
# Set custom port
|
|
PORT=3000 ./routewatch
|
|
|
|
# Enable debug logging
|
|
DEBUG=routewatch ./routewatch
|
|
```
|
|
|
|
## HTTP Endpoints
|
|
|
|
### Web Interface
|
|
- `GET /` - Redirects to /status
|
|
- `GET /status` - HTML status dashboard
|
|
- `GET /status.json` - JSON statistics
|
|
- `GET /as/{asn}` - AS detail page (HTML)
|
|
- `GET /prefix/{prefix}` - Prefix detail page (HTML)
|
|
- `GET /prefixlength/{length}` - IPv4 prefixes by mask length
|
|
- `GET /prefixlength6/{length}` - IPv6 prefixes by mask length
|
|
- `GET /ip/{ip}` - Redirects to prefix containing the IP
|
|
|
|
### API v1
|
|
- `GET /api/v1/stats` - Detailed statistics with handler metrics
|
|
- `GET /api/v1/ip/{ip}` - Look up AS information for an IP address
|
|
- `GET /api/v1/as/{asn}` - Get prefixes announced by an AS
|
|
- `GET /api/v1/prefix/{prefix}` - Get routes for a specific prefix
|
|
|
|
## Code Structure
|
|
|
|
```
|
|
routewatch/
|
|
├── cmd/
|
|
│ ├── routewatch/ # Main daemon entry point
|
|
│ ├── asinfo-gen/ # Utility to generate AS info data
|
|
│ └── streamdumper/ # Debug utility for raw stream output
|
|
├── internal/
|
|
│ ├── routewatch/ # Core application logic
|
|
│ ├── server/ # HTTP server and handlers
|
|
│ ├── database/ # SQLite storage layer
|
|
│ ├── streamer/ # RIPE RIS Live client
|
|
│ ├── ristypes/ # BGP message data structures
|
|
│ ├── logger/ # Structured logging wrapper
|
|
│ ├── metrics/ # Performance metrics tracking
|
|
│ ├── config/ # Configuration management
|
|
│ └── templates/ # HTML templates
|
|
└── pkg/
|
|
└── asinfo/ # AS information lookup (public API)
|
|
```
|
|
|
|
## Architecture Overview
|
|
|
|
### Component Relationships
|
|
|
|
```
|
|
┌─────────────────────────────────────────────────────────────────┐
|
|
│ RouteWatch │
|
|
│ (internal/routewatch/app.go - main orchestrator) │
|
|
├─────────────────────────────────────────────────────────────────┤
|
|
│ │
|
|
│ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ │
|
|
│ │ Streamer │───▶│ Handlers │───▶│ Database │ │
|
|
│ │ │ │ │ │ │ │
|
|
│ │ RIS Live │ │ - ASHandler │ │ SQLite with │ │
|
|
│ │ WebSocket │ │ - PeerHandler│ │ WAL mode │ │
|
|
│ │ client │ │ - PrefixHdlr │ │ │ │
|
|
│ │ │ │ - PeeringHdlr│ │ Tables: │ │
|
|
│ └──────────────┘ └──────────────┘ │ - asns │ │
|
|
│ │ - prefixes │ │
|
|
│ ┌──────────────┐ ┌──────────────┐ │ - live_routes│ │
|
|
│ │ Server │───▶│ Handlers │───▶│ - peerings │ │
|
|
│ │ │ │ │ │ - bgp_peers │ │
|
|
│ │ Chi router │ │ Status, API │ └──────────────┘ │
|
|
│ │ port 8080 │ │ AS, Prefix │ │
|
|
│ └──────────────┘ └──────────────┘ │
|
|
│ │
|
|
└─────────────────────────────────────────────────────────────────┘
|
|
```
|
|
|
|
### Execution Flow
|
|
|
|
1. **Startup** (`cmd/routewatch/main.go` → `internal/routewatch/cli.go`)
|
|
- Uber fx dependency injection initializes all components
|
|
- Signal handlers registered for graceful shutdown
|
|
|
|
2. **Initialization** (`internal/routewatch/app.go`)
|
|
- Database created with SQLite schema (WAL mode, 3GB cache)
|
|
- Message handlers registered with the streamer
|
|
- HTTP server started on configured port
|
|
|
|
3. **Message Processing Pipeline**
|
|
```
|
|
RIS Live Stream → JSON Parser → Message Dispatcher → Handler Queues → Batch Writers → SQLite
|
|
```
|
|
- Streamer connects to `ris-live.ripe.net` via HTTP
|
|
- Parses BGP UPDATE messages from JSON stream
|
|
- Dispatches to registered handlers based on message type
|
|
- Each handler has its own queue with backpressure handling
|
|
- Handlers batch writes for efficiency (25K-30K ops, 1-2s timeout)
|
|
|
|
4. **Handler Details**
|
|
- **ASHandler**: Tracks all ASNs seen in AS paths
|
|
- **PeerHandler**: Records BGP peer information
|
|
- **PrefixHandler**: Maintains live routing table (upserts on announcement, deletes on withdrawal)
|
|
- **PeeringHandler**: Extracts AS peering relationships from AS paths
|
|
|
|
5. **HTTP Request Flow**
|
|
```
|
|
Request → Chi Router → Middleware (timeout, logging) → Handler → Database Query → Response
|
|
```
|
|
|
|
### Key Design Patterns
|
|
|
|
- **Batched Writes**: All database operations are batched for performance
|
|
- **Backpressure**: Probabilistic message dropping when queues exceed 50% capacity
|
|
- **Graceful Shutdown**: 60-second timeout, flushes all pending batches
|
|
- **Reconnection**: Exponential backoff (5s-320s) with reset after 30s of stable connection
|
|
- **IPv4 Optimization**: IP ranges stored as uint32 for O(1) lookups
|
|
|
|
### Database Schema
|
|
|
|
```sql
|
|
-- Core tables
|
|
asns(id, number, handle, description, first_seen, last_seen)
|
|
prefixes_v4(id, prefix, mask_length, first_seen, last_seen)
|
|
prefixes_v6(id, prefix, mask_length, first_seen, last_seen)
|
|
|
|
-- Live routing tables (one per IP version)
|
|
live_routes_v4(id, prefix, mask_length, origin_asn, peer_ip, as_path,
|
|
next_hop, last_updated, v4_ip_start, v4_ip_end)
|
|
live_routes_v6(id, prefix, mask_length, origin_asn, peer_ip, as_path,
|
|
next_hop, last_updated)
|
|
|
|
-- Relationship tracking
|
|
peerings(id, as_a, as_b, first_seen, last_seen)
|
|
bgp_peers(id, peer_ip, peer_asn, last_message_type, last_seen)
|
|
```
|
|
|
|
## Configuration
|
|
|
|
Configuration is handled via environment variables and OS-specific paths:
|
|
|
|
| Variable | Default | Description |
|
|
|----------|---------|-------------|
|
|
| `PORT` | `8080` | HTTP server port |
|
|
| `DEBUG` | (empty) | Set to `routewatch` for debug logging |
|
|
|
|
State directory (database location):
|
|
- macOS: `~/Library/Application Support/routewatch/`
|
|
- Linux: `/var/lib/routewatch/` or `~/.local/share/routewatch/`
|
|
|
|
## Development
|
|
|
|
```bash
|
|
# Run tests
|
|
make test
|
|
|
|
# Format code
|
|
make fmt
|
|
|
|
# Run linter
|
|
make lint
|
|
|
|
# Build
|
|
make
|
|
```
|
|
|
|
## Entrypoints
|
|
|
|
This repository adheres to the
|
|
[Scripts to Rule Them All](https://github.com/github/scripts-to-rule-them-all)
|
|
standard: normalized scripts in `script/` are the entrypoints for the
|
|
development workflow, and the Makefile targets are thin shims that call
|
|
them. We provide:
|
|
|
|
- `script/bootstrap` — install all development dependencies (go,
|
|
golangci-lint, Go module download)
|
|
- `script/setup` — make a fresh clone ready for development: runs
|
|
`script/bootstrap`, then `script/install-precommit`
|
|
- `script/projectname` — print the project name (used for the Docker
|
|
image tag)
|
|
- `script/test` — run the test suite
|
|
(`go test -short -timeout 30s -race -cover ./...`, verbose rerun on
|
|
failure). The `-short` flag skips the live-network integration test so the
|
|
default run is deterministic and offline.
|
|
- `script/lint` — run `go vet ./...` and `golangci-lint run`
|
|
- `script/fmt` — format all code (writes)
|
|
- `script/fmt-check` — check formatting (read-only)
|
|
- `script/check` — run `script/test`, `script/lint`, and
|
|
`script/fmt-check`
|
|
- `script/docker` — build the Docker image tagged via
|
|
`script/projectname`
|
|
- `script/cibuild` — CI entrypoint: `docker build .`
|
|
- `script/precommit` — pre-commit gate: `go mod tidy` + `go fmt` (must
|
|
not change files), then `script/check`
|
|
- `script/install-precommit` — install the git pre-commit hook that
|
|
runs `script/precommit`
|
|
|
|
The live-network integration test `TestRouteWatchLiveFeed` streams the RIPE
|
|
RIS feed for a few seconds and is skipped in short mode. To run it on demand,
|
|
invoke `go test` directly without `-short`:
|
|
`go test -run TestRouteWatchLiveFeed ./internal/routewatch/`.
|
|
|
|
## License
|
|
|
|
See LICENSE file.
|