Re-vendor the shared files from sneak/prompts at 0653579 (closes #52)
check / check (push) Canceled after 0s
check / check (push) Canceled after 0s
Takes .golangci.yml, .dockerignore, .gitignore, .editorconfig, the workflow and REPO_POLICIES.md from sneak/prompts 0653579, with this repository's own entries after the shared content. Lint (golangci-lint v2.14.0) and test are Dockerfile phases that script/lint and script/test build alone with a new CACHEBUST value, keeping Go's and golangci-lint's caches; the image build depends on both. make fmt and make fmt-check also format the markdown with prettier, pinned in yarn.lock. The agent guidance moves to AGENTS.md. The Go code is fixed for what the new configuration finds, and the AS and prefix detail reads now return the rows' error after their loops. Model: opus-5-5
This commit was merged in pull request #85.
This commit is contained in:
@@ -5,7 +5,6 @@ table in real time: it streams BGP UPDATE messages from the RIPE RIS Live
|
||||
service, maintains a live routing table in SQLite, and provides HTTP APIs for
|
||||
querying routing information.
|
||||
|
||||
|
||||
## Features
|
||||
|
||||
- Real-time streaming of BGP updates from RIPE RIS Live
|
||||
@@ -38,6 +37,7 @@ DEBUG=routewatch ./routewatch
|
||||
## HTTP Endpoints
|
||||
|
||||
### Web Interface
|
||||
|
||||
- `GET /` - Redirects to /status
|
||||
- `GET /status` - HTML status dashboard
|
||||
- `GET /status.json` - JSON statistics
|
||||
@@ -48,6 +48,7 @@ DEBUG=routewatch ./routewatch
|
||||
- `GET /ip/{ip}` - Redirects to prefix containing the IP
|
||||
|
||||
### API v1
|
||||
|
||||
- `GET /api/v1/stats` - Detailed statistics with handler metrics
|
||||
- `GET /api/v1/ip/{ip}` - Look up AS information for an IP address
|
||||
- `GET /api/v1/as/{asn}` - Get prefixes announced by an AS
|
||||
@@ -107,52 +108,57 @@ routewatch/
|
||||
### Execution Flow
|
||||
|
||||
1. **Startup** (`cmd/routewatch/main.go` → `internal/routewatch/cli.go`)
|
||||
- Uber fx dependency injection initializes all components
|
||||
- Signal handlers registered for graceful shutdown
|
||||
- Uber fx dependency injection initializes all components
|
||||
- Signal handlers registered for graceful shutdown
|
||||
|
||||
2. **Initialization** (`internal/routewatch/app.go`)
|
||||
- Database created with SQLite schema (WAL mode, 3GB cache)
|
||||
- Message handlers registered with the streamer
|
||||
- HTTP server started on configured port
|
||||
- Database created with SQLite schema (WAL mode, 3GB cache)
|
||||
- Message handlers registered with the streamer
|
||||
- HTTP server started on configured port
|
||||
|
||||
3. **Message Processing Pipeline**
|
||||
```
|
||||
RIS Live Stream → JSON Parser → Message Dispatcher → Handler Queues → Batch Writers → SQLite
|
||||
```
|
||||
- Streamer connects to `ris-live.ripe.net` via HTTP
|
||||
- Parses BGP UPDATE messages from JSON stream
|
||||
- Dispatches to registered handlers based on message type
|
||||
- Each handler has its own queue with backpressure handling
|
||||
- Handlers batch writes for efficiency (25K-30K ops, 1-2s timeout)
|
||||
|
||||
```
|
||||
RIS Live Stream → JSON Parser → Message Dispatcher → Handler Queues → Batch Writers → SQLite
|
||||
```
|
||||
|
||||
- Streamer connects to `ris-live.ripe.net` via HTTP
|
||||
- Parses BGP UPDATE messages from JSON stream
|
||||
- Dispatches to registered handlers based on message type
|
||||
- Each handler has its own queue with backpressure handling
|
||||
- Handlers batch writes for efficiency (25K-30K ops, 1-2s timeout)
|
||||
|
||||
4. **Handler Details**
|
||||
- **ASHandler**: Tracks all ASNs seen in AS paths
|
||||
- **PeerHandler**: Records BGP peer information
|
||||
- **PrefixHandler**: Maintains live routing table (upserts on announcement, deletes on withdrawal,
|
||||
deletes all of a peer's routes when its BGP session goes down). Routes are kept per peer IP, so
|
||||
when one peer IP has sessions on two collectors, a session going down on one also removes the
|
||||
routes the other still carries, until they are announced again.
|
||||
- **PeeringHandler**: Extracts AS peering relationships from AS paths
|
||||
- **ASHandler**: Tracks all ASNs seen in AS paths
|
||||
- **PeerHandler**: Records BGP peer information
|
||||
- **PrefixHandler**: Maintains live routing table (upserts on announcement,
|
||||
deletes on withdrawal, deletes all of a peer's routes when its BGP session
|
||||
goes down). Routes are kept per peer IP, so when one peer IP has sessions
|
||||
on two collectors, a session going down on one also removes the routes the
|
||||
other still carries, until they are announced again.
|
||||
- **PeeringHandler**: Extracts AS peering relationships from AS paths
|
||||
|
||||
5. **HTTP Request Flow**
|
||||
```
|
||||
Request → Chi Router → Middleware (timeout, logging) → Handler → Database Query → Response
|
||||
```
|
||||
```
|
||||
Request → Chi Router → Middleware (timeout, logging) → Handler → Database Query → Response
|
||||
```
|
||||
|
||||
### Key Design Patterns
|
||||
|
||||
- **Batched Writes**: All database operations are batched for performance
|
||||
- **Backpressure**: Probabilistic message dropping when queues exceed 50% capacity
|
||||
- **Backpressure**: Probabilistic message dropping when queues exceed 50%
|
||||
capacity
|
||||
- **Graceful Shutdown**: 60-second timeout, flushes all pending batches
|
||||
- **Reconnection**: Any end of the stream other than shutdown, including RIS Live
|
||||
ending the response normally, marks the feed disconnected and reconnects with
|
||||
exponential backoff (5s-320s). The backoff resets after a stream that lasted
|
||||
more than 30s. A stream with no line for 60s is dropped and reconnected.
|
||||
- **Reconnection**: Any end of the stream other than shutdown, including RIS
|
||||
Live ending the response normally, marks the feed disconnected and reconnects
|
||||
with exponential backoff (5s-320s). The backoff resets after a stream that
|
||||
lasted more than 30s. A stream with no line for 60s is dropped and
|
||||
reconnected.
|
||||
- **IP Lookup**: the most specific live route is found by looking up the
|
||||
address's prefix at each mask length, longest first, in the prefix index
|
||||
(at most 25 lookups for IPv4, 113 for IPv6); prefixes are stored in the
|
||||
text form Go's `net/netip` prints. Routes shorter than /8 for IPv4 or /16
|
||||
for IPv6, such as a default route, are not used.
|
||||
address's prefix at each mask length, longest first, in the prefix index (at
|
||||
most 25 lookups for IPv4, 113 for IPv6); prefixes are stored in the text form
|
||||
Go's `net/netip` prints. Routes shorter than /8 for IPv4 or /16 for IPv6, such
|
||||
as a default route, are not used.
|
||||
|
||||
### Database Schema
|
||||
|
||||
@@ -177,23 +183,24 @@ bgp_peers(id, peer_ip, peer_asn, last_message_type, last_seen)
|
||||
|
||||
Configuration is handled via environment variables and OS-specific paths:
|
||||
|
||||
| Variable | Default | Description |
|
||||
|----------|----------|-------------|
|
||||
| `PORT` | `8080` | HTTP server port, a whole number from 1 to 65535 |
|
||||
| `DEBUG` | (empty) | Set to `routewatch` for debug logging |
|
||||
| `XDG_DATA_HOME` | `/var/lib` (in the Docker image) | Base of the state directory; must be an absolute path |
|
||||
| `GOMEMLIMIT` | `1536MiB` (in the Docker image) | Go soft memory limit; see Memory |
|
||||
| `MALLOC_ARENA_MAX` | `2` (in the Docker image) | glibc malloc arena cap, a positive whole number; see Memory |
|
||||
| Variable | Default | Description |
|
||||
| ------------------ | -------------------------------- | ----------------------------------------------------------- |
|
||||
| `PORT` | `8080` | HTTP server port, a whole number from 1 to 65535 |
|
||||
| `DEBUG` | (empty) | Set to `routewatch` for debug logging |
|
||||
| `XDG_DATA_HOME` | `/var/lib` (in the Docker image) | Base of the state directory; must be an absolute path |
|
||||
| `GOMEMLIMIT` | `1536MiB` (in the Docker image) | Go soft memory limit; see Memory |
|
||||
| `MALLOC_ARENA_MAX` | `2` (in the Docker image) | glibc malloc arena cap, a positive whole number; see Memory |
|
||||
|
||||
A variable that is set to an invalid value stops the start with an error and a
|
||||
non-zero exit. An empty variable counts as unset.
|
||||
|
||||
State directory (database location):
|
||||
|
||||
- macOS: `~/Library/Application Support/routewatch/`
|
||||
- Linux: `/var/lib/berlin.sneak.app.routewatch/` when running as root,
|
||||
otherwise `$XDG_DATA_HOME/berlin.sneak.app.routewatch/` (with `XDG_DATA_HOME`
|
||||
unset, `~/.local/share/berlin.sneak.app.routewatch/`). In the Docker image
|
||||
this is `/var/lib/berlin.sneak.app.routewatch/`.
|
||||
- Linux: `/var/lib/berlin.sneak.app.routewatch/` when running as root, otherwise
|
||||
`$XDG_DATA_HOME/berlin.sneak.app.routewatch/` (with `XDG_DATA_HOME` unset,
|
||||
`~/.local/share/berlin.sneak.app.routewatch/`). In the Docker image this is
|
||||
`/var/lib/berlin.sneak.app.routewatch/`.
|
||||
|
||||
## Memory
|
||||
|
||||
@@ -201,6 +208,7 @@ The daemon holds a live routing table, so its memory grows with the size of the
|
||||
data it tracks. The image sets ceilings that keep it inside a 5 GiB container.
|
||||
|
||||
Budget:
|
||||
|
||||
- Go heap: a 1.5 GiB soft limit (`GOMEMLIMIT=1536MiB`, set in the image).
|
||||
- SQLite: at most 640 MiB of page cache across the connection pool (64 MiB per
|
||||
connection, 10 connections) and a 1.5 GiB hard heap limit for the C library.
|
||||
@@ -223,11 +231,12 @@ or the equivalent in your deployment tool. This leaves headroom above the
|
||||
ceilings for spikes and the kernel page cache.
|
||||
|
||||
Override the Go soft limit by setting `GOMEMLIMIT` in the environment (for
|
||||
example `-e GOMEMLIMIT=1GiB`); this replaces the image default. `MALLOC_ARENA_MAX`
|
||||
can be overridden the same way, but raising it lets RSS climb again on a
|
||||
many-core host.
|
||||
example `-e GOMEMLIMIT=1GiB`); this replaces the image default.
|
||||
`MALLOC_ARENA_MAX` can be overridden the same way, but raising it lets RSS climb
|
||||
again on a many-core host.
|
||||
|
||||
What happens at each limit:
|
||||
|
||||
- Go soft limit: as the heap approaches `GOMEMLIMIT`, the runtime runs garbage
|
||||
collection more aggressively rather than growing further.
|
||||
- SQLite: at a 1 GiB soft heap limit it recycles its page cache instead of
|
||||
@@ -264,8 +273,8 @@ What the [upaas](https://git.eeqj.de/sneak/upaas) app needs:
|
||||
limit, so on a host with swap Docker allows the same amount of swap again.
|
||||
- Health check: the image's `HEALTHCHECK` requests
|
||||
`/.well-known/healthcheck.json` on the container port. upaas reads the
|
||||
container's health 60 seconds after a deploy and fails the deploy unless it
|
||||
is `healthy`.
|
||||
container's health 60 seconds after a deploy and fails the deploy unless it is
|
||||
`healthy`.
|
||||
|
||||
## Development
|
||||
|
||||
@@ -288,35 +297,40 @@ make
|
||||
This repository adheres to the
|
||||
[Scripts to Rule Them All](https://github.com/github/scripts-to-rule-them-all)
|
||||
standard: normalized scripts in `script/` are the entrypoints for the
|
||||
development workflow, and the Makefile targets are thin shims that call
|
||||
them. We provide:
|
||||
development workflow, and the Makefile targets are thin shims that call them. We
|
||||
provide:
|
||||
|
||||
- `script/bootstrap` — install all development dependencies (go,
|
||||
golangci-lint, Go module download)
|
||||
- `script/bootstrap` — install all development dependencies (git, make, go, the
|
||||
pinned node and yarn, prettier from `yarn.lock`, Go module download)
|
||||
- `script/setup` — make a fresh clone ready for development: runs
|
||||
`script/bootstrap`, then `script/install-precommit`
|
||||
- `script/projectname` — print the project name (used for the Docker
|
||||
image tag)
|
||||
- `script/test` — run the test suite
|
||||
(`go test -short -timeout 30s -race -cover ./...`, verbose rerun on
|
||||
failure). The `-short` flag skips the live-network integration test so the
|
||||
default run is deterministic and offline.
|
||||
- `script/lint` — run `go vet ./...` and `golangci-lint run`
|
||||
- `script/fmt` — format all code (writes)
|
||||
- `script/projectname` — print the project name (used for the Docker image tag)
|
||||
- `script/test` — build the `test` phase of the `Dockerfile` alone, which runs
|
||||
`go test -short -count=1 -timeout 90s -race -cover ./...` (verbose rerun on
|
||||
failure). The `-short` flag skips the live-network integration test so the run
|
||||
is deterministic and offline.
|
||||
- `script/lint` — build the `lint` phase of the `Dockerfile` alone, which runs
|
||||
golangci-lint v2.14.0 with `.golangci.yml`
|
||||
- `script/fmt` — format the Go code with `go fmt` and the markdown with prettier
|
||||
(writes)
|
||||
- `script/fmt-check` — check formatting (read-only)
|
||||
- `script/check` — run `script/test`, `script/lint`, and
|
||||
`script/fmt-check`
|
||||
- `script/docker` — build the Docker image tagged via
|
||||
`script/projectname`
|
||||
- `script/cibuild` — CI entrypoint: `docker build .`
|
||||
- `script/precommit` — pre-commit gate: `go mod tidy` + `go fmt` (must
|
||||
not change files), then `script/check`
|
||||
- `script/install-precommit` — install the git pre-commit hook that
|
||||
runs `script/precommit`
|
||||
- `script/check` — run `script/test`, `script/lint`, and `script/fmt-check`
|
||||
- `script/docker` — build the Docker image tagged via `script/projectname`
|
||||
- `script/cibuild` — CI entrypoint: runs `script/bootstrap` and `script/check`,
|
||||
then builds the image
|
||||
- `script/precommit` — pre-commit gate: `go mod tidy` + `go fmt` (must not
|
||||
change files), then `script/check`
|
||||
- `script/install-precommit` — install the git pre-commit hook that runs
|
||||
`script/precommit`
|
||||
|
||||
The live-network integration test `TestRouteWatchLiveFeed` streams the RIPE
|
||||
RIS feed for a few seconds and is skipped in short mode. To run it on demand,
|
||||
invoke `go test` directly without `-short`:
|
||||
Tests and the linter run only in Docker, as the `test` and `lint` phases of the
|
||||
`Dockerfile`; the image cannot be built unless both pass. Every `docker build`
|
||||
in `script/` passes a new `CACHEBUST` value, so each run executes every check,
|
||||
while Go's build cache and golangci-lint's cache are kept between builds.
|
||||
|
||||
The live-network integration test `TestRouteWatchLiveFeed` streams the RIPE RIS
|
||||
feed for a few seconds and is skipped in short mode. To run it on demand, invoke
|
||||
`go test` directly without `-short`:
|
||||
`go test -run TestRouteWatchLiveFeed ./internal/routewatch/`.
|
||||
|
||||
## License
|
||||
|
||||
Reference in New Issue
Block a user