check / check (push) Waiting to run
A new package holds every prefix some peer carries. For each prefix and origin AS it keeps the set of peers that carry it, one bit per peer, and the two newest of their routes in full. It applies announcements, withdrawals and peers' sessions going down by the rule confirmed on the issue, and records which pairs changed for the database writes to come. It answers the IP lookup, an AS's prefixes, a prefix's origins, a sample of one mask length, and the counts the stats show. Nothing uses it yet. The peer sets use github.com/bits-and-blooms/bitset. The kept routes' times are held in order in github.com/google/btree, so the oldest and newest need no scan. Model: opus-5-5
587 lines
30 KiB
Markdown
587 lines
30 KiB
Markdown
# RouteWatch
|
|
|
|
RouteWatch is an MIT-licensed Go daemon by @sneak that monitors the BGP routing
|
|
table in real time. It works like this:
|
|
|
|
- It streams BGP UPDATE messages from the RIPE RIS Live service.
|
|
- It keeps a live routing table in SQLite.
|
|
- It serves web pages and a JSON API for looking routes up.
|
|
|
|
## Rationale
|
|
|
|
RIPE RIS Live streams the BGP updates that RIPE's route collectors receive, as
|
|
they happen, from the moment a client connects. It reports changes, not the
|
|
routing table they add up to, so it cannot say which AS an IP address is routed
|
|
to now or which routes a prefix has. RouteWatch applies the feed's updates to
|
|
live routing tables in its own SQLite database and answers IP-to-AS lookups and
|
|
prefix pages from them.
|
|
|
|
The feed sends no table when a client connects, so these tables hold only the
|
|
routes the feed has announced since RouteWatch first started: an IP-to-AS lookup
|
|
finds nothing for an address whose routes have not been announced since then.
|
|
The tables also miss the updates RouteWatch drops when it falls behind under
|
|
load and those sent while it is stopped, so a lookup or a prefix page can show a
|
|
route the feed has withdrawn.
|
|
|
|
## Features
|
|
|
|
- Real-time streaming of BGP updates from RIPE RIS Live
|
|
- Maintains live IPv4 and IPv6 routing tables
|
|
- Tracks AS peering relationships
|
|
- HTTP API for IP-to-AS lookups, prefix details, and AS information
|
|
- IP lookups report the public blocklists an address is on, from copies kept in
|
|
memory and on disk
|
|
- Automatic reconnection with exponential backoff
|
|
- Batched database writes for high performance
|
|
- Backpressure handling to prevent memory exhaustion
|
|
|
|
## Installation
|
|
|
|
```bash
|
|
go build -o routewatch ./cmd/routewatch
|
|
```
|
|
|
|
## Usage
|
|
|
|
```bash
|
|
# Run the daemon (listens on port 8080 by default)
|
|
./routewatch
|
|
|
|
# Set custom port
|
|
PORT=3000 ./routewatch
|
|
|
|
# Enable debug logging
|
|
DEBUG=routewatch ./routewatch
|
|
```
|
|
|
|
## HTTP Endpoints
|
|
|
|
Every route answers `HEAD` as it answers `GET`, without the body.
|
|
|
|
### Web Interface
|
|
|
|
- `GET /` - Home page (HTML): headline figures, lookups by AS number and by IP
|
|
address, and a search of AS names. A card shows the `curl` command that looks
|
|
up the visitor's own address at `/ip/{addr}`, and that command's live answer.
|
|
- `GET /status` - HTML status dashboard, which reads `/api/v1/stats`
|
|
- `GET /status.json` - JSON statistics: feed, memory, database and route figures
|
|
- `GET /.well-known/healthcheck.json` - Health check (JSON): 200 when the
|
|
database answers and the RIS Live feed is connected, otherwise 503;
|
|
`data.status` is `ok` or `error` and `data.checks` says which check failed
|
|
- `GET /as/{asn}` - AS detail page (HTML): the prefixes the AS originates and
|
|
its peers. When the database has no record of the AS, its handle and
|
|
description come from the AS list built into RouteWatch. The page then shows
|
|
no first-seen date.
|
|
- `GET /prefix/{prefix}/{length}` - Prefix detail page (HTML), such as
|
|
`/prefix/192.0.2.0/24`: every live route for the prefix, one per peer, and its
|
|
origin ASes
|
|
- `GET /prefixlength/{length}` - Up to 500 IPv4 prefixes with that mask length,
|
|
picked at random, each with its newest route (HTML)
|
|
- `GET /prefixlength6/{length}` - The same for IPv6
|
|
- `GET /ip/{addr}` - The same JSON answer as `/api/v1/ip/{ip}`
|
|
- `GET /ip/` - The same, for the caller's own address: the address the request
|
|
came from or, when that is a trusted proxy (`TRUSTED_PROXIES`, see
|
|
Configuration), the client address in its `X-Forwarded-For`
|
|
|
|
### API v1
|
|
|
|
- `GET /api/v1/stats` - The `/status.json` figures plus connection, garbage
|
|
collector, stream and handler queue statistics. `stream.route_churn_per_min`
|
|
is the number of prefix announcements and withdrawals, from all peers, in the
|
|
last full clock minute.
|
|
- `GET /api/v1/ip/{ip}` - Looks up each address. `{ip}` is an IP address, a host
|
|
name, or a comma-separated list of at most 10 of these. `results` has one
|
|
entry for each address, with:
|
|
- `ip` and `ptr`: the address and its PTR names
|
|
- `netblock` and the fields beside it: the address's most specific live
|
|
route, and that route's origin AS with its WHOIS details
|
|
- `blocklists`: the public blocklists the address is on (see Blocklists)
|
|
|
|
An address with no live route has a result without the route and AS fields.
|
|
Of the addresses a host name resolves to, only public ones are looked up. A
|
|
name with none, such as one that resolves to a private address, is reported
|
|
as `host name has no public address`, without its addresses. `errors` has
|
|
one entry for each host name that failed. The answer is 400 when:
|
|
- `{ip}` lists more than 10
|
|
- `{ip}` comes to more than 10 addresses once its names are resolved
|
|
- nothing in `{ip}` is an IP address or a host name with a public address
|
|
|
|
A PTR lookup gets 2 seconds. An address whose lookup takes longer is
|
|
answered without PTR names.
|
|
|
|
- `GET /api/v1/as/{asn}` - The AS and the prefixes it originates, IPv4 and IPv6
|
|
apart, each with when its newest route was updated. When the database has no
|
|
record of the AS, its handle and description come from the AS list built into
|
|
RouteWatch. The answer then has no `first_seen` or `last_seen`.
|
|
- `GET /api/v1/prefix/{prefix}/{length}` - Every live route for the prefix, one
|
|
per peer, and the same routes grouped by origin AS in `origins`
|
|
- `GET /api/v1/search/as?q={text}` - Finds the ASes whose handle or description
|
|
contains `{text}`, ignoring case. `data` holds:
|
|
- `query`: `{text}` without its leading and trailing spaces
|
|
- `total`: the number of ASes that match
|
|
- `results`: the first 50 of them in AS number order, each with `asn`,
|
|
`handle` and `description`
|
|
|
|
The search reads the AS list built into RouteWatch, not the database. The
|
|
list comes from
|
|
[ipverse/as-metadata](https://github.com/ipverse/as-metadata). A match can
|
|
therefore be an AS that the database has no record of. A `q` that is
|
|
missing, empty or only spaces answers 400.
|
|
|
|
A JSON answer is `{"status": "ok", "data": ...}`, or for an error
|
|
`{"status": "error", "error": {"msg": ..., "code": ...}}`, and carries an
|
|
`@meta` object with the time zone, the API version and the time the request
|
|
took. The health check is the exception: its 503 answer is also
|
|
`{"status": "ok", "data": ...}`, with `error` in `data.status`.
|
|
|
|
A request that runs past its time limit answers 504: the limit is 30 seconds,
|
|
and 4 seconds for the database reads behind `/status.json` and `/api/v1/stats`.
|
|
An AS number, prefix or mask length that does not parse answers 400, as does a
|
|
mask length out of range for its IP version. An AS answers 404 when it is not in
|
|
the AS list built into RouteWatch and the database has neither a record of it
|
|
nor a live route it originates. AS 0 always answers 404, because it means no
|
|
known origin. The AS name search never lists it. A prefix with no live route
|
|
answers 404 too. The HTML pages give these errors as plain text. Under `/api/v1`
|
|
and `/ip/`, a path no route matches answers 404 with the message `Not found`,
|
|
and `POST`, `PUT`, `PATCH`, `DELETE`, `OPTIONS`, `TRACE` or `CONNECT` answers
|
|
405 with `Method not allowed`. A 500 answer says only `Internal server error`,
|
|
and a host name that does not resolve only `could not resolve host name`; the
|
|
detail goes to the log.
|
|
|
|
Every answer carries `Strict-Transport-Security`, `Content-Security-Policy`,
|
|
`X-Frame-Options`, `X-Content-Type-Options`, `Referrer-Policy` and
|
|
`Permissions-Policy`. The Content-Security-Policy lets a page load content only
|
|
from the server, and does not allow inline content with `'unsafe-inline'`: the
|
|
pages keep their CSS and JavaScript in inline `<style>` and `<script>` blocks,
|
|
so each answer's policy names a random nonce made for that answer, and the
|
|
browser applies or runs only the blocks that carry it. The pages therefore use
|
|
no `style` attributes, which a nonce cannot cover.
|
|
|
|
An origin AS of 0 (`origin_asn` in a route, `asn` in an IP lookup, the key `0`
|
|
in a prefix's `origins`) means the route has no known origin: its AS path ends
|
|
in an AS set of several members, any of which may have originated it. The pages
|
|
show such a route as having no known origin.
|
|
|
|
## Blocklists
|
|
|
|
The IP lookup reports which public blocklists list each address. Each result has
|
|
a `blocklists` list with one entry for each list the address is on:
|
|
|
|
- `name`: the list's name, from the lists below
|
|
- `description`: what being on the list means
|
|
- `updated`: when RouteWatch last downloaded the list, or last heard from its
|
|
publisher that its copy was current
|
|
|
|
An address on no list has an empty `blocklists`. An address with no live route
|
|
still has a result with its `blocklists`.
|
|
|
|
The lists:
|
|
|
|
- `dshield`: the 20 networks that DShield saw scan the most targets in the last
|
|
three days. From [DShield](https://isc.sans.edu/feeds_doc.html), downloaded
|
|
every hour.
|
|
- `feodo_tracker`: botnet command-and-control servers. From abuse.ch
|
|
[Feodo Tracker](https://feodotracker.abuse.ch/blocklist/), downloaded every 15
|
|
minutes.
|
|
- `firehol_abusers_30d`: addresses reported for comment, form or forum spam in
|
|
the last 30 days. From
|
|
[FireHOL](https://iplists.firehol.org/?ipset=firehol_abusers_30d), downloaded
|
|
every 5 hours.
|
|
- `firehol_level3`: addresses reported for attacks, spyware or viruses in the
|
|
last 30 days. From
|
|
[FireHOL](https://iplists.firehol.org/?ipset=firehol_level3), downloaded every
|
|
3 hours.
|
|
- `spamhaus_drop` and `spamhaus_dropv6`: IPv4 and IPv6 networks that Spamhaus
|
|
says spammers or cyber criminals have hijacked or lease. From
|
|
[Spamhaus DROP](https://www.spamhaus.org/blocklists/do-not-route-or-peer/),
|
|
downloaded once a day. Spamhaus merged its EDROP list into DROP.
|
|
- `tor_exits`: Tor exit nodes, where traffic from the Tor network leaves. From
|
|
the [Tor Project](https://check.torproject.org/torbulkexitlist), downloaded
|
|
every hour.
|
|
|
|
No list is downloaded more often than its publisher updates it or allows. Each
|
|
can be downloaded whole without an account or a key.
|
|
|
|
How RouteWatch keeps the lists:
|
|
|
|
- It holds each list in memory, so a lookup makes no outside request.
|
|
- It keeps a copy of each list in the `blocklists` directory of the state
|
|
directory (see Configuration). Each copy is a file named after its list.
|
|
- Beside each copy, a file ending in `.validators` keeps the `Last-Modified` and
|
|
`ETag` the publisher sent with that copy.
|
|
- At startup it loads each list from its copy. It downloads a list at once when
|
|
the copy is older than the list's interval, or when there is no copy.
|
|
- A download sends the copy's `Last-Modified` as `If-Modified-Since` and its
|
|
`ETag` as `If-None-Match`. The publisher then sends the list only when it has
|
|
changed. Without either, the download asks for the whole list.
|
|
- A download fails when it takes over a minute or its file is over 16 MiB.
|
|
- When a download fails, or its file does not parse, RouteWatch keeps the
|
|
previous copy and logs the failure. It tries again one interval later.
|
|
|
|
A successful IP lookup answer carries `Cache-Control` with a `max-age` of 15
|
|
minutes, the shortest list interval. The answer for the caller's own address
|
|
(`/ip/`) is `private`, and the others are `public`.
|
|
|
|
## Code Structure
|
|
|
|
```
|
|
routewatch/
|
|
├── cmd/
|
|
│ ├── routewatch/ # Main daemon entry point
|
|
│ ├── asinfo-gen/ # Utility to generate AS info data
|
|
│ └── streamdumper/ # Debug utility for raw stream output
|
|
├── internal/
|
|
│ ├── blocklist/ # Public IP blocklists: downloads and lookups
|
|
│ ├── routewatch/ # Core application logic
|
|
│ ├── server/ # HTTP server and handlers
|
|
│ ├── database/ # SQLite storage layer
|
|
│ ├── routetable/ # The whole routing table in memory
|
|
│ ├── streamer/ # RIPE RIS Live client
|
|
│ ├── ristypes/ # BGP message data structures
|
|
│ ├── logger/ # Structured logging wrapper
|
|
│ ├── metrics/ # Performance metrics tracking
|
|
│ ├── config/ # Configuration management
|
|
│ ├── templates/ # HTML templates
|
|
│ ├── version/ # Build version information
|
|
│ └── whois/ # WHOIS lookups of AS details
|
|
└── pkg/
|
|
└── asinfo/ # AS information lookup (public API)
|
|
```
|
|
|
|
## Architecture Overview
|
|
|
|
### Component Relationships
|
|
|
|
```
|
|
┌─────────────────────────────────────────────────────────────────┐
|
|
│ RouteWatch │
|
|
│ (internal/routewatch/app.go - main orchestrator) │
|
|
├─────────────────────────────────────────────────────────────────┤
|
|
│ │
|
|
│ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ │
|
|
│ │ Streamer │───▶│ Handlers │───▶│ Database │ │
|
|
│ │ │ │ │ │ │ │
|
|
│ │ RIS Live │ │ - ASHandler │ │ SQLite with │ │
|
|
│ │ HTTP stream │ │ - PeerHandler│ │ WAL mode │ │
|
|
│ │ client │ │ - PrefixHdlr │ │ │ │
|
|
│ │ │ │ - PeeringHdlr│ │ Tables: │ │
|
|
│ └──────────────┘ └──────────────┘ │ - asns │ │
|
|
│ │ - prefixes │ │
|
|
│ ┌──────────────┐ ┌──────────────┐ │ - live_routes│ │
|
|
│ │ Server │───▶│ Handlers │───▶│ - peerings │ │
|
|
│ │ │ │ │ │ - bgp_peers │ │
|
|
│ │ Chi router │ │ Status, API │ └──────────────┘ │
|
|
│ │ port 8080 │ │ AS, Prefix │ │
|
|
│ └──────────────┘ └──────────────┘ │
|
|
│ │
|
|
└─────────────────────────────────────────────────────────────────┘
|
|
```
|
|
|
|
### Execution Flow
|
|
|
|
1. **Startup** (`cmd/routewatch/main.go` → `internal/routewatch/cli.go`)
|
|
- Uber fx dependency injection initializes all components
|
|
- Signal handlers registered for graceful shutdown
|
|
|
|
2. **Initialization** (`internal/routewatch/app.go`)
|
|
- Database created with SQLite schema (WAL mode, page caches as in "Memory")
|
|
- Message handlers registered with the streamer
|
|
- HTTP server started on configured port
|
|
|
|
3. **Message Processing Pipeline**
|
|
|
|
```
|
|
RIS Live Stream → JSON Parser → Message Dispatcher → Handler Queues → Batch Writers → SQLite
|
|
```
|
|
|
|
- Streamer connects to `ris-live.ripe.net` via HTTP
|
|
- Parses BGP UPDATE messages from JSON stream
|
|
- Dispatches to registered handlers based on message type
|
|
- Each handler has its own queue with backpressure handling
|
|
- Handlers batch writes for efficiency: the AS, peer and prefix handlers
|
|
write 30,000, 10,000 and 25,000 updates at a time, or after 2, 2 and 1
|
|
seconds; the peering handler writes the peerings it has collected every 30
|
|
seconds
|
|
|
|
4. **Handler Details**
|
|
- **ASHandler**: Tracks all ASNs seen in AS paths
|
|
- **PeerHandler**: Records BGP peer information
|
|
- **PrefixHandler**: Maintains live routing table (upserts on announcement,
|
|
deletes on withdrawal or on an announcement whose AS path is empty or
|
|
holds AS 0 anywhere, deletes all of a peer's routes when its BGP session
|
|
goes down). Routes are kept per peer IP, so when one peer IP has sessions
|
|
on two collectors, a session going down on one also removes the routes the
|
|
other still carries, until they are announced again.
|
|
- **PeeringHandler**: Extracts AS peering relationships from AS paths
|
|
|
|
5. **HTTP Request Flow**
|
|
```
|
|
Request → Chi Router → Middleware (timeout, logging) → Handler → Database Query → Response
|
|
```
|
|
|
|
### Key Design Patterns
|
|
|
|
- **Batched Writes**: The handlers write the feed's updates to the database in
|
|
batches for performance
|
|
- **Backpressure**: Probabilistic message dropping when queues exceed 50%
|
|
capacity
|
|
- **Graceful Shutdown**: on SIGTERM or SIGINT the streamer stops, the handlers
|
|
write the batches they hold, the HTTP server stops and the database is
|
|
checkpointed and closed. The daemon gives this 8 seconds and exits with status
|
|
0; "Running under upaas" says what a stop loses, logs and leaves undone when
|
|
the 8 seconds run out.
|
|
- **Reconnection**: Any end of the stream other than shutdown, including RIS
|
|
Live ending the response normally, marks the feed disconnected and reconnects
|
|
with exponential backoff (5s-320s). The backoff resets after a stream that
|
|
lasted more than 30s. A stream with no line for 60s is dropped and
|
|
reconnected.
|
|
- **IP Lookup**: the most specific live route is found by looking up the
|
|
address's prefix at each mask length, longest first, in the prefix index (at
|
|
most 25 lookups for IPv4, 113 for IPv6); prefixes are stored in the text form
|
|
Go's `net/netip` prints. Routes shorter than /8 for IPv4 or /16 for IPv6, such
|
|
as a default route, are not used.
|
|
|
|
### Database Schema
|
|
|
|
```sql
|
|
-- Core tables
|
|
asns(asn PRIMARY KEY, handle, description,
|
|
as_name, org_name, org_id, address, country_code, abuse_email,
|
|
abuse_phone, tech_email, tech_phone, rir, rir_registration_date,
|
|
rir_last_modified, whois_raw,
|
|
first_seen, last_seen, whois_updated_at)
|
|
prefixes_v4(prefix PRIMARY KEY, first_seen) WITHOUT ROWID
|
|
prefixes_v6(prefix PRIMARY KEY, first_seen) WITHOUT ROWID
|
|
|
|
-- Live routing tables (one per IP version), one route per peer and prefix
|
|
live_routes_v4(id INTEGER PRIMARY KEY, prefix, mask_length, origin_asn,
|
|
peer_ip, as_path, next_hop, last_updated,
|
|
UNIQUE(peer_ip, prefix))
|
|
live_routes_v6(id INTEGER PRIMARY KEY, prefix, mask_length, origin_asn,
|
|
peer_ip, as_path, next_hop, last_updated,
|
|
UNIQUE(peer_ip, prefix))
|
|
|
|
-- Relationship tracking
|
|
peerings(id PRIMARY KEY, as_a, as_b, first_seen, last_seen,
|
|
UNIQUE(as_a, as_b))
|
|
bgp_peers(id PRIMARY KEY, peer_ip UNIQUE, peer_asn, first_seen, last_seen,
|
|
last_message_type)
|
|
|
|
-- Nothing in the daemon writes this table
|
|
announcements(id PRIMARY KEY, prefix_id, peer_asn, origin_asn, path,
|
|
next_hop, timestamp, is_withdrawal)
|
|
```
|
|
|
|
The indexes are in `internal/database/schema.sql`.
|
|
|
|
## Configuration
|
|
|
|
Configuration is handled via environment variables and OS-specific paths:
|
|
|
|
| Variable | Default | Description |
|
|
| ------------------ | -------------------------------- | ----------------------------------------------------------- |
|
|
| `PORT` | `8080` | HTTP server port, a whole number from 1 to 65535 |
|
|
| `DEBUG` | (empty) | Set to `routewatch` for debug logging |
|
|
| `XDG_DATA_HOME` | `/var/lib` (in the Docker image) | Base of the state directory; must be an absolute path |
|
|
| `GOMEMLIMIT` | `1536MiB` (in the Docker image) | Go soft memory limit; see Memory |
|
|
| `MALLOC_ARENA_MAX` | `2` (in the Docker image) | glibc malloc arena cap, a positive whole number; see Memory |
|
|
| `TRUSTED_PROXIES` | (empty) | Addresses and CIDR ranges of trusted proxies; see below |
|
|
|
|
A variable that is set to an invalid value stops the start with an error and a
|
|
non-zero exit. An empty variable counts as unset.
|
|
|
|
`TRUSTED_PROXIES` lists the reverse proxies whose `X-Forwarded-For` the server
|
|
believes, comma-separated, such as `10.0.0.0/8,192.0.2.1`. For a request from
|
|
one of them, the request log and `/ip/` use the last address in
|
|
`X-Forwarded-For` that is not a trusted proxy: each proxy adds the address it
|
|
received the request from to the end, so the entries before it are whatever the
|
|
client sent. For a request from any other address they use the address the
|
|
request came from, whatever forwarding headers it carries. `X-Real-IP` and
|
|
`True-Client-IP` are not read.
|
|
|
|
State directory (the database and the blocklist copies):
|
|
|
|
- macOS: `~/Library/Application Support/berlin.sneak.app.routewatch/`
|
|
- Linux: `/var/lib/berlin.sneak.app.routewatch/` when running as root, otherwise
|
|
`$XDG_DATA_HOME/berlin.sneak.app.routewatch/` (with `XDG_DATA_HOME` unset,
|
|
`~/.local/share/berlin.sneak.app.routewatch/`). In the Docker image this is
|
|
`/var/lib/berlin.sneak.app.routewatch/`.
|
|
|
|
## Memory
|
|
|
|
The daemon holds a live routing table, so its memory grows with the size of the
|
|
data it tracks. The image sets ceilings that keep it inside a 5 GiB container.
|
|
|
|
Budget:
|
|
|
|
- Go heap: a 1.5 GiB soft limit (`GOMEMLIMIT=1536MiB`, set in the image).
|
|
- SQLite: 1 GiB of page cache for the one connection every write uses, which
|
|
keeps the pages the route writes touch from one batch to the next, and 16 MiB
|
|
for each of the 9 reading connections, whose cache SQLite empties whenever
|
|
another connection has written: about 1.15 GiB in all. A 1.5 GiB soft heap
|
|
limit sits above the caches, so it does not shrink them, and a 2 GiB hard heap
|
|
limit caps the C library.
|
|
- glibc allocator: the SQLite C library runs on glibc `malloc`, which frees
|
|
page-cache chunks back to per-arena free lists rather than to the kernel, so
|
|
process RSS tracks the high-water mark of those arenas, not SQLite's live
|
|
heap. glibc creates up to eight arenas per core, so on a many-core host the
|
|
retained memory — and thus RSS — grows with the core count. The image sets
|
|
`MALLOC_ARENA_MAX=2` to bound it; the two-arena cap costs nothing here because
|
|
database writes are already serialized.
|
|
- About 0.2 GiB for everything else in the runtime.
|
|
|
|
The ceilings add up to 3.7 GiB: the Go heap's 1.5 GiB, SQLite's 2 GiB and the
|
|
0.2 GiB for the rest.
|
|
|
|
Run the container with a memory limit of 5 GiB and swap disabled:
|
|
|
|
```bash
|
|
docker run --memory=5g --memory-swap=5g ...
|
|
```
|
|
|
|
or the equivalent in your deployment tool. This leaves headroom above the
|
|
ceilings for spikes and the kernel page cache.
|
|
|
|
Override the Go soft limit by setting `GOMEMLIMIT` in the environment (for
|
|
example `-e GOMEMLIMIT=1GiB`); this replaces the image default.
|
|
`MALLOC_ARENA_MAX` can be overridden the same way, but raising it lets RSS climb
|
|
again on a many-core host.
|
|
|
|
What happens at each limit:
|
|
|
|
- Go soft limit: as the heap approaches `GOMEMLIMIT`, the runtime runs garbage
|
|
collection more aggressively rather than growing further.
|
|
- SQLite: at the 1.5 GiB soft heap limit it recycles its page cache instead of
|
|
allocating more; at the 2 GiB hard heap limit a statement fails with an
|
|
out-of-memory error, and the handler logs it and drops that batch. The process
|
|
keeps running.
|
|
- Handler queues: each of the four handler queues holds at most 20,000 messages.
|
|
When a queue fills, the streamer drops messages instead of blocking, except a
|
|
`STATE` message, which it holds, in feed order, until there is room: a few
|
|
hundred bytes each, about ten a second.
|
|
- Handler batches: while the AS, peer or prefix handler writes a batch to the
|
|
database, the updates that arrive collect in its next batch, which stops at
|
|
ten times the batch size: 300,000 updates for the AS handler
|
|
(`asnBatchMaxSize`, about 10 MiB), 100,000 for the peer handler
|
|
(`peerBatchMaxSize`, about 10 MiB) and 250,000 for the prefix handler
|
|
(`prefixBatchMaxSize`, about 60 MiB). At that size the handler takes nothing
|
|
more off its queue until the write ends, so the queue fills and the streamer
|
|
drops messages as above. When the delete of a peer's live routes after its
|
|
session went down fails, the prefix handler keeps the updates the peer sent
|
|
after the down for the next write, which retries the delete first; on top of
|
|
its 250,000 it keeps at most the last 25,000 of them (`prefixBatchSize`, about
|
|
6 MiB) and drops and logs the older ones, so while the delete keeps failing
|
|
the other peers' updates still come off the queue. With the batches being
|
|
written, the three handlers' batches take at most about 0.25 GiB of the Go
|
|
heap's 1.5 GiB.
|
|
|
|
With `DEBUG=routewatch` the daemon logs a `System stats` line every 60 seconds
|
|
with the goroutine count and Go memory figures.
|
|
|
|
## Running under upaas
|
|
|
|
What the [upaas](https://git.eeqj.de/sneak/upaas) app needs:
|
|
|
|
- Container port: `8080`.
|
|
- Volume: one, at container path `/var/lib/berlin.sneak.app.routewatch`.
|
|
- Environment: `TRUSTED_PROXIES`, set to the address the reverse proxy in front
|
|
of the app connects from, as the container sees it; without it the request log
|
|
and `/ip/` give the proxy's address for every request. Nothing else is needed:
|
|
leave `XDG_DATA_HOME`, `GOMEMLIMIT` and `MALLOC_ARENA_MAX` at the image's
|
|
values. `DEBUG=routewatch` is optional and adds the `System stats` memory line
|
|
to the log.
|
|
- Memory Limit: `5g`, the 5 GiB limit from Memory above. upaas sets no swap
|
|
limit, so on a host with swap Docker allows the same amount of swap again.
|
|
- Health check: the image's `HEALTHCHECK` requests
|
|
`/.well-known/healthcheck.json` on the container port. upaas reads the
|
|
container's health 60 seconds after a deploy and fails the deploy unless it is
|
|
`healthy`.
|
|
- Stop grace: the container needs 9 seconds between SIGTERM and SIGKILL. upaas
|
|
gives 10 (`stopTimeoutSeconds` in its `internal/docker/client.go`), as
|
|
`docker stop` does by default. The daemon allows 8 seconds for the handlers to
|
|
write what they hold and for the requests in progress to finish, then stops
|
|
the HTTP server, closes the database and exits with status 0. A longer grace
|
|
changes nothing. What is not written after 8 seconds is lost, and the streamer
|
|
and each handler log how much. When the prefix handler has fallen behind the
|
|
feed, a stop loses far more: its batch is full at 250,000 updates and the
|
|
streamer's worker for it waits for room, so the streamer's stop takes the
|
|
whole 8 seconds and every handler then skips its last write. Lost then are the
|
|
messages left on the four handlers' queues, what the AS, peer and peering
|
|
handlers hold, the prefix handler's batch, and the write it has in progress,
|
|
up to another 250,000 updates, which take about 20 seconds and are cut off by
|
|
the exit; withdrawals and peer downs are among them. A route withdrawn in them
|
|
stays in the live route table until its peer announces or withdraws the prefix
|
|
again or its session goes down. When the 8 seconds run out the database is not
|
|
closed, and SQLite rolls back a write the exit cut off when the database next
|
|
opens.
|
|
|
|
## Development
|
|
|
|
```bash
|
|
# Run tests
|
|
make test
|
|
|
|
# Format code
|
|
make fmt
|
|
|
|
# Run linter
|
|
make lint
|
|
|
|
# Build
|
|
make build
|
|
```
|
|
|
|
## Entrypoints
|
|
|
|
This repository adheres to the
|
|
[Scripts to Rule Them All](https://github.com/github/scripts-to-rule-them-all)
|
|
standard: normalized scripts in `script/` are the entrypoints for the
|
|
development workflow, and the Makefile targets are thin shims that call them. We
|
|
provide:
|
|
|
|
- `script/bootstrap` — install all development dependencies (git, make, go, the
|
|
pinned node and yarn, prettier from `yarn.lock`, Go module download)
|
|
- `script/setup` — make a fresh clone ready for development: runs
|
|
`script/bootstrap`, then `script/install-precommit`
|
|
- `script/projectname` — print the project name (used for the Docker image tag)
|
|
- `script/test` — build the `test` phase of the `Dockerfile` alone, which runs
|
|
`go test -short -count=1 -timeout 90s -race -cover ./...` (verbose rerun on
|
|
failure). The `-short` flag skips the live-network integration test so the run
|
|
is deterministic and offline.
|
|
- `script/lint` — build the `lint` phase of the `Dockerfile` alone, which runs
|
|
golangci-lint v2.14.0 with `.golangci.yml`
|
|
- `script/fmt` — format the Go code with `go fmt` and the markdown with prettier
|
|
(writes)
|
|
- `script/fmt-check` — check formatting (read-only)
|
|
- `script/check` — run `script/test`, `script/lint`, and `script/fmt-check`
|
|
- `script/docker` — build the Docker image tagged via `script/projectname`
|
|
- `script/cibuild` — CI entrypoint: runs `script/bootstrap` and `script/check`,
|
|
then builds the image
|
|
- `script/precommit` — pre-commit gate: `go mod tidy` (must not change `go.mod`
|
|
or `go.sum`) and `go fmt` (writes), then `script/check`
|
|
- `script/install-precommit` — install the git pre-commit hook that runs
|
|
`script/precommit`
|
|
- `script/asupdate` — rebuild the AS list built into RouteWatch,
|
|
`pkg/asinfo/asdata.json.gz`, from the `as.csv` file of
|
|
[ipverse/as-metadata](https://github.com/ipverse/as-metadata). It needs the
|
|
network. It runs `cmd/asinfo-gen`, which fails when it finds no AS. When
|
|
`cmd/asinfo-gen` fails, the old list stays and the script exits non-zero.
|
|
|
|
Tests and the linter run only in Docker, as the `test` and `lint` phases of the
|
|
`Dockerfile`; the image cannot be built unless both pass. Every `docker build`
|
|
in `script/` passes a new `CACHEBUST` value, so each run executes every check,
|
|
while Go's build cache and golangci-lint's cache are kept between builds.
|
|
|
|
The live-network integration test `TestRouteWatchLiveFeed` streams the RIPE RIS
|
|
feed for a few seconds and is skipped in short mode. To run it on demand, invoke
|
|
`go test` directly without `-short`:
|
|
`go test -run TestRouteWatchLiveFeed ./internal/routewatch/`.
|
|
|
|
## License
|
|
|
|
MIT. See [`LICENSE`](LICENSE).
|
|
|
|
## Author
|
|
|
|
[@sneak](https://sneak.berlin)
|