sneak/routewatch
1
0
Fork 0
Code Issues 3 Pull Requests 1 Actions 4 Packages Projects Releases Wiki Activity
Files
6c0bfc1690810240f509314cd5bb5f87593ceb55
routewatch/README.md
T
sneak 6c0bfc1690
check / check (push) Canceled after 0s
Details
Rewrite the README sentence by sentence 
The README is rewritten from scratch against the prose rules of
#157: every sentence under
about 30 words, lists as bullets, every technical term defined where it
first appears, and every statement checked against the code on next. It
leads with what RouteWatch is and how to run it, then the pages, the API
and the blocklists, then the settings, memory and upaas, and last the
design and the development workflow. The dated entry goes into TODO.md.

Model: fable-5-1
2026-11-02 00:03:50 +00:00

853 lines
39 KiB
Markdown
Raw Blame History

# RouteWatch
RouteWatch is an MIT-licensed Go daemon by @sneak that follows the internet's
BGP routing table as it changes and answers questions about IP addresses over
HTTP.
BGP is the protocol by which the networks of the internet tell each other which
addresses they can reach. Each network is an autonomous system, an AS, with a
number such as `AS15169`. A prefix is a block of addresses, such as
`192.0.2.0/24`. The number after the slash is the mask length. A shorter mask
length means a bigger block. A route says which AS announces a prefix and
through which ASes the announcement came. The AS that announces the prefix is
its origin AS. The ASes the announcement came through, in order, are the AS
path.
RouteWatch does three things:
- It streams BGP updates from the RIPE RIS Live service.
- It applies them to a routing table that it keeps in memory, with a copy in a
SQLite database.
- It serves web pages and a JSON API that answer from that table.
## Rationale
RouteWatch exists to serve our other services as an internal API for information
about IP addresses. A service asks RouteWatch instead of working the answer out
for itself.
RIPE runs route collectors around the world. Many routers, the RIS peers, send
the collectors their routes over BGP sessions. RIS Live streams every change the
collectors receive, as it happens. RouteWatch applies those changes to its own
routing table. From that table it sees directly:
- which AS each IP address is routed to
- which announced prefix holds the address, and how big that prefix is
- how many peers carry the prefix, and with which origin ASes
RouteWatch adds details of each AS from WHOIS, the public directory of the
internet registries. The details include the organisation that holds the AS and
its abuse contact. RouteWatch keeps them in its database and fetches them again
once they are 30 days old. An IP lookup answers from the stored details and
never waits for WHOIS.
An IP lookup also says which public blocklists list the address (see
Blocklists).
## Getting started
The usual way to run RouteWatch is its Docker image. Build the image, which is
tagged `routewatch`:
```bash
make docker
```
The build runs the tests and the linter first, and fails when either fails. Run
the image:
```bash
docker run --memory=5g --memory-swap=5g -p 8080:8080 \
-v routewatch:/var/lib/berlin.sneak.app.routewatch routewatch
```
The memory limit is explained in Memory. The volume holds the database and the
copies of the blocklists.
To run the binary on the host instead, build it with `make build`. The build
needs Go and a C compiler, because the SQLite library is written in C:
```bash
make build
./bin/routewatch
```
`make run` builds the binary and runs it with debug logging. It also writes the
log to `log.txt`.
Then open `http://localhost:8080/` in a browser, or ask about an address:
```bash
curl http://localhost:8080/ip/8.8.8.8
```
The routing table starts empty. RIS Live sends no routing table when a client
connects, only the changes from then on. So the table holds the routes announced
since RouteWatch first started, and it fills as the feed runs. An IP lookup
finds no route for an address until a prefix that holds it is announced. The
database keeps the table across restarts.
## HTTP
The server listens on `PORT`, `8080` by default. Every route answers `HEAD` as
it answers `GET`, without the body.
The pages and the API read routes from the routing table in memory. For each
prefix and origin AS, the table keeps the two newest routes in full, with their
AS paths and times. This document calls them the kept routes. The table keeps
only the address of every other peer that carries the prefix with that origin
AS. When a kept route is withdrawn, no older route takes its place. So a prefix
and origin AS can have two kept routes, one or none.
### Web pages
- `/`: the home page. Its top row shows these figures, refreshed every 5
seconds:
- whether the feed is connected
- the number of live routes, ASes, prefixes and BGP peers
- the route updates per second
Below the figures are a lookup by AS number, a search of AS names and a
lookup by IP address. A card shows the `curl` command that looks up the
visitor's own address, and that command's live answer. Other cards show
figures of the routing table, with links to the AS and prefix pages:
- the newest and the oldest kept route, each with its prefix, origin AS and
time
- the AS most recently seen in an AS path, and the AS seen longest ago
- the AS that originates the most prefixes
- how many ASes originate a single prefix, and the newest kept route of
those ASes
- the number of bogon announcements, linked to `/bogons`. A bogon
announcement is a route whose prefix or origin AS is reserved.
- the number of prefixes with more than one origin AS, linked to
`/multiple-origins`
- fifteen well-known networks, such as Apple, Google, Cogent and MIT, each
with its number of prefixes and the age of its newest kept route, newest
first
RouteWatch works these figures out once a minute, so they can be a minute
old. Until the first figures are ready, the page says that they are being
worked out.
- `/as/{asn}`: an AS, the prefixes it originates and the ASes it peers with. The
prefixes come in two lists, IPv4 and IPv6, shortest mask length first. An AS
has a handle, a short name such as `GOOGLE`, and a description, such as
`Google LLC`. When the database has no record of the AS, both come from the AS
list built into RouteWatch. The page then shows no first-seen date.
- `/prefix/{prefix}/{length}`: a prefix, such as `/prefix/192.0.2.0/24`, with
its origin ASes and every peer that carries it. Only the kept routes show an
AS path, a next hop and a time. The next hop is the router the peer sends the
prefix's traffic to. RouteWatch records the peer itself as the next hop.
- `/prefixlength/{length}`: up to 500 IPv4 prefixes with that mask length,
picked at random. Each shows its origin AS and the age of its newest kept
route. When peers give a prefix different origin ASes, the page shows the one
that most peers give.
- `/prefixlength6/{length}`: the same for IPv6.
- `/bogons`: the bogon announcements. Each row shows the prefix, the origin AS,
the number of peers and the reason. The page lists at most 1,000 rows, in
prefix order, and says how many there are in all. The list is worked out once
a minute, with the home page's figures.
- `/multiple-origins`: the prefixes that more than one origin AS announces. Each
row shows the origin ASes and how many peers carry the prefix with each. Like
`/bogons`, the page lists at most 1,000 rows, in prefix order, and says how
many there are in all.
- `/status`: a dashboard of the figures of `/api/v1/stats`, refreshed every 2
seconds.
A prefix is reserved when it lies inside one of these:
- a block that the IANA special-purpose address registries mark as not globally
reachable
- the multicast space, `224.0.0.0/4` and `ff00::/8`
A few blocks that may be announced lie inside reserved blocks. The most specific
block that holds the prefix decides.
An AS number is reserved when it is one of these:
- an AS number for private use or for documentation
- `AS23456`, which stands in for a 32-bit AS number
- the last AS number of the 16-bit range or of the 32-bit range
Address space and AS numbers that the IANA has not handed out yet are not
counted.
### JSON API
A JSON answer is `{"status": "ok", "data": ...}`. An error answer is
`{"status": "error", "error": {"msg": ..., "code": ...}}`. Every JSON answer
also carries an `@meta` object:
- `time_zone`: always `UTC`
- `api_version`: `1`
- `execution_time_ms`: the number of milliseconds the request took
Every value with a unit is a number, and the unit ends the field name. Examples
are `_ms`, `_seconds`, `_bytes`, `_per_sec` and `_per_min`.
The routes:
- `GET /api/v1/ip/{ip}`: what RouteWatch knows about each address in `{ip}`. See
IP lookup below.
- `GET /ip/{addr}`: the same answer as `/api/v1/ip/{ip}`.
- `GET /ip/`: the same answer, for the caller's own address. That is the address
the request came from. For a request from a trusted proxy, it is the client
address in `X-Forwarded-For` (see `TRUSTED_PROXIES` in Configuration).
- `GET /api/v1/as/{asn}`: the AS and the prefixes it originates. `data` holds:
- `asn`: the AS, with its handle, description, `first_seen` and `last_seen`
- `ipv4_prefixes` and `ipv6_prefixes`: the prefixes it originates, each with
`last_updated`, the time of its newest kept route with the AS
- `total_count`: how many prefixes the two lists hold together
`last_updated` is left out when the kept routes of the prefix with the AS
were all withdrawn. When the database has no record of the AS, its handle
and description come from the AS list built into RouteWatch. The answer then
has no `first_seen` or `last_seen`.
- `GET /api/v1/prefix/{prefix}/{length}`: the prefix and its routes. `data`
holds:
- `prefix`: the prefix
- `routes`: one route for each peer that carries the prefix
- `origins`: the same routes, grouped by origin AS
- `peer_count` and `origin_count`: how many routes and how many origin ASes
there are
Only the kept routes have `as_path`, `next_hop` and `last_updated`.
- `GET /api/v1/search/as?q={text}`: the ASes whose handle or description
contains `{text}`, ignoring case. `data` holds:
- `query`: `{text}` without its leading and trailing spaces
- `total`: the number of ASes that match
- `results`: the first 50 of them in AS number order, each with `asn`,
`handle` and `description`
The search reads the AS list built into RouteWatch, not the database. The
list comes from
[ipverse/as-metadata](https://github.com/ipverse/as-metadata). So a match
can be an AS that the database has no record of.
- `GET /status.json`: figures of the feed, the memory, the database and the
routing table. The figures whose names do not say what they mean:
- `uptime_seconds`: the seconds since the feed last connected, or 0 when it
has never connected
- `memory_usage_bytes`: the size of the Go heap in use
- `live_routes`, `ipv4_routes` and `ipv6_routes`: the number of peer and
prefix pairs, where each peer that carries a prefix counts once for it
- `oldest_route` and `newest_route`: the times of the oldest and the newest
kept route
- `origins_without_kept_route`: the number of prefix and origin AS pairs
that peers carry but whose kept routes were all withdrawn
- `whois_stats.current_interval_seconds`: the WHOIS fetcher's wait between
two fetches, which can include a fraction of a second
- `GET /api/v1/stats`: the `/status.json` figures plus `reconnect_count`, and
figures of the garbage collector in `gc`, of the stream in `stream` and of the
handler queues in `handler_stats`. `connection_duration_seconds` is the
seconds the feed has been connected, or null while it is not connected.
`stream.route_churn_per_min` is the number of prefix announcements and
withdrawals, from all peers, in the last full clock minute.
- `GET /.well-known/healthcheck.json`: the health check. It answers 200 when the
database answers and the feed is connected, otherwise 503. `data.status` is
`ok` or `error`, and `data.checks` says which check failed. The 503 answer
keeps the `{"status": "ok", "data": ...}` form.
### IP lookup
`{ip}` is an IP address, a host name, or a comma-separated list of at most 10 of
these. A host name is resolved, and only its public addresses are looked up. The
answer holds `query`, the text looked up, and `results`, with one entry for each
address. Every entry has the same fields:
- `ip`: the address
- `ptr`: the address's PTR names, the host names that DNS gives for it, or an
empty list when it has none
- `blocklists`: the public blocklists the address is on (see Blocklists), or an
empty list when it is on none
Every other field is about the address's live route. Each is null when the
address has no live route:
- `netblock`: the most specific prefix that holds the address
- `mask_length`: the netblock's mask length
- `ip_version`: the netblock's IP version, 4 or 6
- `num_peers`: the number of peers that carry the netblock
- `asn`: the netblock's origin AS. When peers give different origin ASes, it is
the one that most peers give.
- `num_origins`: how many different origin ASes the netblock has
- `origin_asns`: those origin ASes, in AS number order
- `first_seen`: when RouteWatch first saw the netblock, or null when the
database has not stored the netblock yet
- `last_seen`: the time of the netblock's newest kept route, or null when its
kept routes were all withdrawn
The fields about the AS in `asn` come from the database. Each is null when the
database does not have it:
- `handle`: the AS's handle
- `description`: the AS's description
- `as_name`: the AS's name in its WHOIS details
- `org_name`: the name of the organisation that holds the AS
- `org_id`: the organisation's ID at its registry
- `address`: the organisation's postal address
- `country_code`: the organisation's country code
- `abuse_email`: the address that takes abuse reports for the AS
- `rir`: the regional internet registry of the AS
When a host name fails, `errors` lists it with the reason. A name that does not
resolve is reported as `could not resolve host name`. A name with no public
address, such as one that resolves to a private address, is reported as
`host name has no public address`. The answer is 400 when:
- `{ip}` lists more than 10 addresses and host names
- `{ip}` comes to more than 10 addresses once its names are resolved
- nothing in `{ip}` is an IP address or a host name with a public address
Each PTR lookup gets 2 seconds. An address whose lookup takes longer is answered
without PTR names.
A successful answer carries `Cache-Control` with a `max-age` of 15 minutes, the
shortest blocklist interval. The answer for the caller's own address (`/ip/`) is
`private`, and the others are `public`.
### Errors
- 400, for:
- an AS number, prefix or mask length that does not parse
- a mask length out of range for its IP version
- an AS name search without `q`, or with only spaces in it
- an IP lookup with too many addresses, or none (see IP lookup)
- 404, for:
- AS 0
- an AS that originates no prefix and has no record in the database or in
the AS list built into RouteWatch
- a prefix that no peer carries
- under `/api/v1` and `/ip/`, a path that matches no route, with the message
`Not found`
- 405, with the message `Method not allowed`, for `POST`, `PUT`, `PATCH`,
`DELETE`, `OPTIONS`, `TRACE` or `CONNECT` on these routes:
- the routes under `/api/v1` and `/ip/`
- `/status.json`
- `/.well-known/healthcheck.json`
- 500: an internal error. The JSON routes say only `Internal server error`. The
detail goes to the log.
- 504: `Request timeout`. A request may take 30 seconds. The database reads
behind `/status.json` and `/api/v1/stats` may take 4 seconds.
The pages give these errors as plain text.
### AS 0
An origin AS of 0 means the route has no known origin. Its AS path ends in an AS
set: a group of several ASes, any of which may have originated the route. The 0
appears in these fields:
- `origin_asn` in a route
- `asn` and `origin_asns` in an IP lookup
- the key `0` in a prefix's `origins`
The pages show such a route as having no known origin. `/as/0` answers 404, and
the AS name search never lists AS 0.
### Headers
Every answer carries these headers:
- `Strict-Transport-Security`
- `Content-Security-Policy`
- `X-Frame-Options`
- `X-Content-Type-Options`
- `Referrer-Policy`
- `Permissions-Policy`
The content security policy lets a page load content only from the server, and
lets no other site frame the pages. The pages keep their CSS and JavaScript in
inline `<style>` and `<script>` blocks. Each answer's policy names a random
value, a nonce, made for that answer. The browser applies or runs only the
blocks that carry it.
## Blocklists
An IP lookup reports which public blocklists list each address. Each entry in
`blocklists` has:
- `name`: the list's name, from the lists below
- `description`: what being on the list means
- `updated`: when RouteWatch last downloaded the list, or last heard from its
publisher that its copy was current
The lists:
- `dshield`: the 20 networks that DShield saw scan the most targets in the last
three days. From [DShield](https://isc.sans.edu/feeds_doc.html), downloaded
every hour.
- `feodo_tracker`: botnet command-and-control servers. From abuse.ch
[Feodo Tracker](https://feodotracker.abuse.ch/blocklist/), downloaded every 15
minutes.
- `firehol_abusers_30d`: addresses reported for comment, form or forum spam in
the last 30 days. From
[FireHOL](https://iplists.firehol.org/?ipset=firehol_abusers_30d), downloaded
every 5 hours.
- `firehol_level3`: addresses reported for attacks, spyware or viruses in the
last 30 days. From
[FireHOL](https://iplists.firehol.org/?ipset=firehol_level3), downloaded every
3 hours.
- `spamhaus_drop` and `spamhaus_dropv6`: IPv4 and IPv6 networks hijacked or
leased by spammers or cyber criminals, according to Spamhaus. From
[Spamhaus DROP](https://www.spamhaus.org/blocklists/do-not-route-or-peer/),
downloaded once a day.
- `tor_exits`: Tor exit nodes, where traffic from the Tor network leaves. From
the [Tor Project](https://check.torproject.org/torbulkexitlist), downloaded
every hour.
No list is downloaded more often than its publisher updates it or allows. Each
can be downloaded whole without an account or a key.
How RouteWatch keeps the lists:
- It holds each list in memory, so a lookup makes no outside request.
- It keeps a copy of each list in the `blocklists` directory of the state
directory (see Configuration). Each copy is a file named after its list.
- Beside each copy, a file ending in `.validators` keeps the `Last-Modified` and
`ETag` headers the publisher sent with that copy.
- At startup it loads each list from its copy. It downloads a list at once when
the copy is older than the list's interval, or when there is no copy.
- A download sends the copy's `Last-Modified` as `If-Modified-Since` and its
`ETag` as `If-None-Match`. The publisher then sends the list only when it has
changed.
- A download fails when it takes over a minute or its file is over 16 MiB.
- When a download fails, or its file does not parse, RouteWatch keeps the
previous copy and logs the failure. It tries again one interval later.
## Configuration
RouteWatch reads its settings from environment variables. It takes no
command-line flags.
- `PORT`: the port the HTTP server listens on. A whole number from 1 to 65535.
Default `8080`.
- `DEBUG`: set it to `routewatch` for debug logging. The daemon then also logs a
`System stats` line every 60 seconds, with the goroutine count and the Go
memory figures. Default empty.
- `TRUSTED_PROXIES`: the reverse proxies whose `X-Forwarded-For` header the
server believes. See below. Default: the private ranges of RFC 1918.
- `XDG_DATA_HOME`: the base of the state directory on Linux and the BSDs, when
RouteWatch does not run as root. It must be an absolute path. The Docker image
sets `/var/lib`.
- `GOMEMLIMIT`: the Go runtime's soft memory limit. The Docker image sets
`2560MiB`. See Memory.
- `MALLOC_ARENA_MAX`: how many memory pools, called arenas, the C library may
create. The Docker image sets `2`. See Memory.
`PORT`, `XDG_DATA_HOME` or `TRUSTED_PROXIES` set to an invalid value stops the
start with an error and a non-zero exit. The Docker image's entrypoint refuses a
`MALLOC_ARENA_MAX` that is not a positive whole number. An empty variable counts
as unset. `TRUSTED_PROXIES` is the one exception: set but empty, it makes the
server trust no proxy.
`TRUSTED_PROXIES` works like this:
- When it is not set, the server trusts `10.0.0.0/8`, `172.16.0.0/12` and
`192.168.0.0/16`.
- When it lists addresses and CIDR ranges, comma-separated, such as
`10.0.0.0/8,192.0.2.1`, the server trusts those instead.
- When it is set but empty (`TRUSTED_PROXIES=`), the server trusts no proxy.
For a request from a trusted proxy, the request log, `/ip/` and the home page's
`curl` card use the last address in `X-Forwarded-For` that is not a trusted
proxy. Each proxy adds the address it received the request from to the end of
the header. The entries before that are whatever the client sent. For a request
from any other address, they use the address the request came from, whatever
headers it carries. `X-Real-IP` and `True-Client-IP` are not read.
The state directory holds the database, `db.sqlite`, and the blocklist copies:
- macOS: `~/Library/Application Support/berlin.sneak.app.routewatch/`
- Linux and the BSDs, as root: `/var/lib/berlin.sneak.app.routewatch/`
- Linux and the BSDs, as another user:
`$XDG_DATA_HOME/berlin.sneak.app.routewatch/`, or
`~/.local/share/berlin.sneak.app.routewatch/` when `XDG_DATA_HOME` is not set
- the Docker image: `/var/lib/berlin.sneak.app.routewatch/`
RouteWatch logs to standard output. It writes text when standard output is a
terminal, and JSON lines otherwise.
## Memory
The daemon keeps the whole routing table in memory. RouteWatch and its Docker
image set ceilings that keep it inside a 5 GiB container. Run the container with
a memory limit of 5 GiB and swap disabled:
```bash
docker run --memory=5g --memory-swap=5g ...
```
That leaves room above the ceilings for spikes and for the kernel's page cache.
The ceilings add up to 3.7 GiB:
- The Go heap: a 2.5 GiB soft limit, `GOMEMLIMIT=2560MiB`, set by the image. The
routing table takes about 700 bytes for each prefix and origin AS pair. A full
table has about 1.4 million pairs, so it takes about 1 GB. The handlers'
batches and queues take up to about 0.5 GiB. The remaining 1 GiB is room for
the garbage collector. With much less, it would run almost all the time.
- SQLite: 1 GiB, set in the code. The connection that does every write has a 512
MiB page cache, the database pages it keeps in memory. It keeps the pages the
writes touch from one write to the next. The `routes` table of a full routing
table takes about 450 MB, so its pages fit. Each of the 9 reading connections
has 16 MiB. A 768 MiB soft heap limit sits above the caches, so it does not
shrink them. A 1 GiB hard heap limit caps the SQLite library.
- About 0.2 GiB for everything else in the runtime.
The SQLite library gets its memory from the C library, glibc. glibc keeps freed
memory in pools called arenas, and hands it back to the kernel slowly. It
creates up to eight arenas per processor core. So on a host with many cores the
process holds far more memory than SQLite uses. The image sets
`MALLOC_ARENA_MAX=2`, which caps the arenas at two. Database writes run one at a
time, so the cap does not slow them.
What happens at each limit:
- The Go soft limit: as the heap nears `GOMEMLIMIT`, the runtime collects
garbage more often rather than growing further.
- The SQLite soft heap limit: SQLite recycles its page cache instead of
allocating more.
- The SQLite hard heap limit: a statement fails with an out-of-memory error,
which the handler logs. The prefix handler writes the routing table's changes
again in a later write. Everything else in the failed write is dropped. The
process keeps running.
- The handler queues: each of the four handlers has a queue of at most 20,000
messages. When a queue fills, the streamer drops messages instead of waiting.
The exception is a `STATE` message, which it holds, in feed order, until there
is room. A `STATE` message says that a peer's BGP session changed state, for
example went down. These are a few hundred bytes each, about ten a second.
- The handler batches: while the AS, peer or prefix handler writes a batch to
the database, the updates that arrive collect in its next batch. That batch
stops at ten times the batch size:
- 300,000 AS numbers from the AS paths for the AS handler, about 10 MiB
- 100,000 updates for the peer handler, about 10 MiB
- 250,000 prefix updates for the prefix handler, about 15 MiB. Each prefix
an update announces or withdraws counts once.
The handler then takes nothing more off its queue until the write ends. The
queue fills, and the streamer drops messages as above.
To replace the image's Go soft limit, set `GOMEMLIMIT` in the environment, for
example `-e GOMEMLIMIT=3GiB`. A limit close to the routing table's size makes
the garbage collector run almost all the time. `MALLOC_ARENA_MAX` can be
replaced the same way, but raising it lets the process grow again on a host with
many cores.
## Running under upaas
What the [upaas](https://git.eeqj.de/sneak/upaas) app needs:
- Container port: `8080`.
- Volume: one, at the container path `/var/lib/berlin.sneak.app.routewatch`.
- Environment: nothing. When `TRUSTED_PROXIES` is not set, the server trusts a
reverse proxy at an RFC 1918 address. Set it only when the proxy connects from
another address. Leave `XDG_DATA_HOME`, `GOMEMLIMIT` and `MALLOC_ARENA_MAX` at
the image's values. `DEBUG=routewatch` is optional.
- Memory limit: `5g`. upaas sets no swap limit, so on a host with swap Docker
allows the same amount of swap again.
- Health check: the image's `HEALTHCHECK` requests
`/.well-known/healthcheck.json` on the container port. upaas reads the
container's health 60 seconds after a deploy and fails the deploy unless it is
`healthy`. In those 60 seconds, Docker 25 and later runs the check every 5
seconds, and the first pass makes the container healthy. Startup first reads
the routing table from the database, which takes about 25 seconds when the
table is full. The check also needs the feed connected, so a container that
cannot reach RIS Live never becomes healthy.
- Stop grace: upaas gives the container 10 seconds between `SIGTERM` and
`SIGKILL`, as `docker stop` does by default. The daemon gives its parts 8
seconds to stop and then exits. A stop still not done after 9 seconds ends
with status 1. See Shutdown under Design for what a stop writes and what it
can lose.
## Design
### Parts
The parts are built in two steps. The `fx` library, which wires a Go program's
parts together, builds the database, the routing table, the streamer and the
HTTP server. `RouteWatch.Run` in `internal/routewatch/app.go` builds the
handlers, the WHOIS fetcher, the blocklists and the database maintainer, and
starts every part. The parts:
- The streamer connects to RIS Live, reads the JSON stream and parses the BGP
messages. It gives each message to the handlers that want it, through one
queue per handler.
- The prefix handler keeps the routing table in memory current. Every second it
writes the table's changes to the `routes` table of the database. It also
stores each new prefix with the time it was first seen.
- The AS handler records every AS seen in an AS path in the `asns` table, with
the times it was first and last seen.
- The peer handler records every peer in the `bgp_peers` table.
- The peering handler takes the pairs of neighbouring ASes out of the AS paths
and writes them to the `peerings` table every 30 seconds.
- The WHOIS fetcher fetches the details of one AS at a time: those never
fetched, and those fetched more than 30 days ago. It asks IANA which registry
holds the AS, then asks that registry. Between these fetches it waits. The
wait starts at 15 seconds. It halves after a success, down to 1 second, and
grows by half after a failure, up to a minute. An IP lookup that finds an AS
with old or missing details asks the fetcher for it at once, without the wait.
- The blocklists part downloads the blocklists and answers which lists hold an
address.
- The database maintainer checkpoints the database every 5 seconds: it folds the
write-ahead log, where SQLite first writes changes, into the database file. It
reclaims free space every 10 minutes and refreshes the query statistics every
hour.
- The HTTP server serves the pages and the API. Every route it shows comes from
the routing table in memory. AS names, WHOIS details and first-seen times come
from the database. The AS name search reads the AS list built into RouteWatch.
### The routing table
`internal/routetable` holds the whole routing table. For each prefix and origin
AS it keeps:
- every peer whose current route to the prefix has that origin AS, as one bit
per peer
- the two newest of those routes in full, with their AS paths and times
When a kept route is withdrawn, no older route takes its place: the table does
not have the older routes in full any more.
The prefix handler applies each update as it handles it:
- An announcement gives the peer a route to the prefix. A peer carries a prefix
with one origin AS at most.
- A withdrawal removes the peer's route. So does an announcement whose AS path
is empty or holds AS 0 anywhere.
- A `STATE` message that says the peer's BGP session went down removes every
route of the peer.
- Within one message, the withdrawals are applied before the announcements.
The address lookup finds the most specific prefix that holds the address. It
looks the address up at each mask length, longest first: at most 25 lookups for
IPv4 and 113 for IPv6. Prefixes shorter than `/8` for IPv4 or `/16` for IPv6,
such as a default route, are not used.
The database's `routes` table has one row for each prefix and origin AS. Every
second the prefix handler writes the pairs that changed, at most 20,000 in one
write. The rest follow in the next writes. Startup reads the whole table back
into memory before the streamer connects. A full table takes about 25 seconds to
read. If the daemon crashes, it loses only the changes not yet written. The
database runs with `synchronous=OFF`, so a power loss or an operating system
crash can lose more.
### What the table can miss
RIS Live sends no routing table when a client connects, so the table holds only
the routes announced since RouteWatch first started. It also misses:
- the updates sent while RouteWatch is stopped
- the updates RouteWatch drops when it falls behind (see Feed handling)
- the updates lost at a stop (see Shutdown)
A lookup or a prefix page can therefore show a route that has since been
withdrawn.
Peers are known by their IP address, and a peer can have sessions on two
collectors. When its session on one goes down, the routes the other still
carries are removed too, until they are announced again.
### Feed handling
The streamer reads the stream line by line. Each line is one message as JSON. It
passes an `UPDATE` or `STATE` message to each handler that wants it, through
that handler's queue of 20,000 messages. An `UPDATE` message carries
announcements and withdrawals. A `STATE` message says that the BGP session
between a collector and a peer changed state, for example went down.
When a queue is more than half full, the streamer drops a share of the messages
for that handler. The share rises from none at half full to all at full. A
`STATE` message is never dropped. One that finds the queue full is held until
there is room, and the handler's other messages are dropped until the held ones
are in.
The AS, peer and prefix handlers write to the database in batches. A batch is
written when it is full, or when its time is up:
- the AS handler: 30,000 AS numbers from the AS paths, or after 2 seconds
- the peer handler: 10,000 updates, or after 2 seconds
- the prefix handler: 25,000 prefix updates, or after 1 second
The peering handler writes every 30 seconds.
### Reconnection
Any end of the stream other than a shutdown marks the feed disconnected. That
includes RIS Live ending the response normally, and a stream that sends no line
for 60 seconds. The streamer then reconnects after a wait. The wait starts at 5
seconds and doubles each time, up to 320 seconds. A stream that lasted more than
30 seconds resets the wait to 5 seconds.
### Shutdown
On `SIGTERM` or `SIGINT` the daemon stops its parts in this order:
1. the streamer
2. the handlers, each of which writes what it holds
3. the WHOIS fetcher
4. the database maintainer
5. the blocklist downloads
6. the HTTP server
Last, the daemon checkpoints and closes the database. It gives all this 8
seconds and then exits with status 0. A stop still not done after 9 seconds ends
with status 1. When RouteWatch stops on an error, it exits with status 1.
What a stop can lose:
- The prefix handler's last write takes at most 20,000 of the routing table's
changed pairs. A full-table peer is a peer that carries the whole routing
table. Its session going down changes about a million pairs, which take about
a minute to write. A stop in that minute leaves the rest unwritten.
- A withdrawal, or a session going down, that is lost this way leaves its routes
in the `routes` table. The next start reads them back. They stay until the
peer sends another update for the prefix, or its session goes down again.
- When the 8 seconds run out, the streamer and the handlers log what they could
not hand over or write. The database is then not closed. When it next opens,
SQLite rolls back any write the exit cut off.
### Code layout
```
routewatch/
├── cmd/
│ ├── routewatch/ # The daemon's entry point
│ ├── asinfo-gen/ # Builds the AS list built into RouteWatch
│ └── streamdumper/ # Prints the raw RIS Live stream, for debugging
├── docs/
│ └── message-examples.json # Examples of RIS Live messages
├── internal/
│ ├── blocklist/ # Public IP blocklists: downloads and lookups
│ ├── bogon/ # Reserved prefixes and AS numbers
│ ├── config/ # Settings from the environment
│ ├── database/ # SQLite storage
│ ├── logger/ # Structured logging
│ ├── metrics/ # Feed and route update figures
│ ├── ristypes/ # The RIS Live message types
│ ├── routetable/ # The whole routing table in memory
│ ├── routewatch/ # The daemon: startup, handlers, WHOIS fetcher
│ ├── server/ # HTTP server, pages and API
│ ├── streamer/ # RIS Live client
│ ├── templates/ # HTML templates
│ ├── version/ # Build version information
│ └── whois/ # WHOIS lookups of AS details
└── pkg/
└── asinfo/ # The AS list built into RouteWatch
```
### Database schema
```sql
asns(asn PRIMARY KEY, handle, description,
as_name, org_name, org_id, address, country_code, abuse_email,
abuse_phone, tech_email, tech_phone, rir, rir_registration_date,
rir_last_modified, whois_raw,
first_seen, last_seen, whois_updated_at)
prefixes_v4(prefix PRIMARY KEY, first_seen) WITHOUT ROWID
prefixes_v6(prefix PRIMARY KEY, first_seen) WITHOUT ROWID
-- The routing table, one row per prefix and origin AS that some peer
-- carries: the peer set, and the newest two routes in full
routes(prefix, origin_asn, peers,
newest_peer, newest_as_path, newest_time,
older_peer, older_as_path, older_time,
PRIMARY KEY (prefix, origin_asn)) WITHOUT ROWID
-- The peers of the routes table, each with its number in the peer sets
route_peers(number PRIMARY KEY, address UNIQUE)
peerings(id PRIMARY KEY, as_a, as_b, first_seen, last_seen,
UNIQUE(as_a, as_b))
bgp_peers(id PRIMARY KEY, peer_ip UNIQUE, peer_asn, first_seen, last_seen,
last_message_type)
-- Nothing in the daemon writes this table
announcements(id PRIMARY KEY, prefix_id, peer_asn, origin_asn, path,
next_hop, timestamp, is_withdrawal)
```
The indexes are in `internal/database/schema.sql`.
## Development
The `make` targets:
- `make bootstrap`: install the development dependencies (see `script/bootstrap`
below)
- `make setup`: make a fresh clone ready for development
- `make test`: run the tests
- `make lint`: run the linter
- `make fmt`: format the Go code and the markdown
- `make fmt-check`: check the formatting without writing
- `make check`: run the tests, the linter and the format check
- `make hooks`: install the git pre-commit hook
- `make build`: build `bin/routewatch`
- `make run`: build the binary and run it with debug logging
- `make clean`: delete `bin/`
- `make docker`: build the Docker image
- `make asupdate`: rebuild the AS list built into RouteWatch
`make` with no target runs the tests.
The tests and the linter run only in Docker, as the `test` and `lint` phases of
the `Dockerfile`. The image cannot be built unless both pass. Every
`docker build` in `script/` passes a new `CACHEBUST` value, so each run executes
every check. Go's build cache and the linter's cache are kept between builds.
The test suite runs with `-short`, which skips `TestRouteWatchLiveFeed`. That
test streams the live feed for a few seconds. No `make` target runs it.
### Entrypoints
This repository adheres to the
[Scripts to Rule Them All](https://github.com/github/scripts-to-rule-them-all)
standard: the scripts in `script/` are the entrypoints of the development
workflow, and the `Makefile` targets call them. The scripts:
- `script/bootstrap`: install the development dependencies:
- `git`, `make` and Go, when they are missing
- the pinned node and yarn
- prettier, from `yarn.lock`
- the Go modules
- `script/setup`: make a fresh clone ready for development. It runs
`script/bootstrap`, then `script/install-precommit`.
- `script/projectname`: print the project name, `routewatch`, which
`script/docker` uses as the image tag.
- `script/test`: build the `test` phase of the `Dockerfile` alone. It runs
`go test -short -count=1 -timeout 90s -race -cover ./...`, and reruns it with
`-v` on failure.
- `script/lint`: build the `lint` phase of the `Dockerfile` alone. It runs
golangci-lint v2.14.0 with `.golangci.yml`.
- `script/fmt`: format the Go code with `go fmt` and the markdown with prettier.
It writes.
- `script/fmt-check`: check the formatting without writing.
- `script/check`: run `script/test`, `script/lint` and `script/fmt-check`.
- `script/docker`: build the Docker image, tagged `routewatch`.
- `script/cibuild`: the CI entrypoint. It runs `script/bootstrap` and
`script/check`, then builds the image.
- `script/precommit`: the pre-commit gate. It runs `go mod tidy` and `go fmt`,
then `script/check`. `go mod tidy` must leave `go.mod` and `go.sum` unchanged;
`go fmt` writes.
- `script/install-precommit`: install the git pre-commit hook that runs
`script/precommit`.
- `script/asupdate`: rebuild the AS list built into RouteWatch,
`pkg/asinfo/asdata.json.gz`, from the `as.csv` file of
[ipverse/as-metadata](https://github.com/ipverse/as-metadata). It needs the
network. When the generator finds no AS, the old list stays and the script
exits non-zero.
## TODO
`TODO.md` holds the working list: the next step, the steps done and the steps
planned.
## License
MIT. See [`LICENSE`](LICENSE).
## Author
[@sneak](https://sneak.berlin)
Reference in New Issue View Git Blame Copy Permalink