# RouteWatch

RouteWatch is an MIT-licensed Go daemon by @sneak that monitors the BGP routing
table in real time. It works like this:

- It streams BGP UPDATE messages from the RIPE RIS Live service.
- It keeps the whole routing table in memory, with a copy in SQLite.
- It serves web pages and a JSON API for looking routes up.

## Rationale

RIPE RIS Live streams the BGP updates that RIPE's route collectors receive, as
they happen, from the moment a client connects. It reports changes, not the
routing table they add up to, so it cannot say which AS an IP address is routed
to now or which routes a prefix has. RouteWatch applies the feed's updates to a
routing table that it keeps in memory, with a copy in its own SQLite database.
It answers IP-to-AS lookups and prefix pages from that table.

The feed sends no table when a client connects. So the routing table holds only
the routes announced since RouteWatch first started. An IP-to-AS lookup finds
nothing for an address whose routes have not been announced since then. The
routing table also misses updates RouteWatch drops when it falls behind, and
updates sent while it is stopped. So a lookup or a prefix page can show a route
the feed has withdrawn.

## Features

- Real-time streaming of BGP updates from RIPE RIS Live
- Maintains live IPv4 and IPv6 routing tables
- Tracks AS peering relationships
- HTTP API for IP-to-AS lookups, prefix details, and AS information
- IP lookups report the public blocklists an address is on, from copies kept in
  memory and on disk
- Automatic reconnection with exponential backoff
- Batched database writes for high performance
- Backpressure handling to prevent memory exhaustion

## Installation

```bash
go build -o routewatch ./cmd/routewatch
```

## Usage

```bash
# Run the daemon (listens on port 8080 by default)
./routewatch

# Set custom port
PORT=3000 ./routewatch

# Enable debug logging
DEBUG=routewatch ./routewatch
```

## HTTP Endpoints

Every route answers `HEAD` as it answers `GET`, without the body.

The pages and the API read routes from the routing table that RouteWatch keeps
in memory. For each prefix and origin AS, the table keeps the two newest routes
in full, with their AS paths and times. These are the kept routes. The table
keeps only the address of every other peer that carries the prefix. When both
kept routes of an origin AS are withdrawn, that origin has no kept route until
one of its peers sends an update.

### Web Interface

- `GET /` - Home page (HTML): headline figures, lookups by AS number and by IP
  address, and a search of AS names. A card shows the `curl` command that looks
  up the visitor's own address at `/ip/{addr}`, and that command's live answer.
  Other cards show figures of the routing table, with links to the AS and prefix
  pages:
    - the newest and the oldest kept route, with its prefix, origin AS and time
    - the AS last seen in an AS path most recently, and the one last seen
      longest ago
    - the AS that originates the most prefixes
    - how many ASes originate a single prefix, and the most recently updated of
      them
    - the number of bogon announcements, linked to `/bogons`
    - the number of prefixes with more than one origin AS, linked to
      `/multiple-origins`
    - well-known networks, such as Apple, Google, Cogent and MIT, each with its
      number of prefixes and the time of its newest kept route, newest first

    RouteWatch works these figures out once a minute, so they can be a minute
    old.

- `GET /bogons` - The bogon announcements (HTML): the routes whose prefix or
  origin AS is reserved. A prefix is reserved when it lies inside one of these:
    - a block that the IANA special-purpose address registries mark as not
      globally reachable
    - the multicast space, `224.0.0.0/4` and `ff00::/8`, which the IANA IPv4 and
      IPv6 address space registries list as Multicast

    A reserved AS number is one of these:
    - an AS number for private use or for documentation
    - `AS23456`
    - the last AS number of the 16-bit or the 32-bit range

    Address space and AS numbers that the IANA has not handed out yet are not
    counted. Each row shows the prefix, the origin AS, the number of peers and
    why it is a bogon. The page shows at most 1,000 rows, in prefix order, and
    says how many there are in all.

- `GET /multiple-origins` - The prefixes announced by more than one origin AS
  (HTML). Each row shows the origin ASes and how many peers carry the prefix
  with each. Like `/bogons`, it shows at most 1,000 rows, in prefix order, and
  says how many there are in all.
- `GET /status` - HTML status dashboard, which reads `/api/v1/stats`
- `GET /status.json` - JSON statistics: feed, memory, database and route
  figures. The route figures come from the routing table. `oldest_route` and
  `newest_route` are the times of the oldest and the newest kept routes.
  `origins_without_kept_route` counts, across all prefixes, the origin ASes that
  have no kept route.
- `GET /.well-known/healthcheck.json` - Health check (JSON): 200 when the
  database answers and the RIS Live feed is connected, otherwise 503;
  `data.status` is `ok` or `error` and `data.checks` says which check failed
- `GET /as/{asn}` - AS detail page (HTML): the prefixes the AS originates and
  its peers. When the database has no record of the AS, its handle and
  description come from the AS list built into RouteWatch. The page then shows
  no first-seen date.
- `GET /prefix/{prefix}/{length}` - Prefix detail page (HTML), such as
  `/prefix/192.0.2.0/24`: its origin ASes and every peer that carries it. Only
  the kept routes show an AS path, a next hop and a time.
- `GET /prefixlength/{length}` - Up to 500 IPv4 prefixes with that mask length,
  picked at random (HTML). Each shows the origin AS that most peers carry it
  with, and the age of its newest kept route.
- `GET /prefixlength6/{length}` - The same for IPv6
- `GET /ip/{addr}` - The same JSON answer as `/api/v1/ip/{ip}`
- `GET /ip/` - The same, for the caller's own address: the address the request
  came from or, when that is a trusted proxy (`TRUSTED_PROXIES`, see
  Configuration), the client address in its `X-Forwarded-For`

### API v1

- `GET /api/v1/stats` - The `/status.json` figures plus connection, garbage
  collector, stream and handler queue statistics. `stream.route_churn_per_min`
  is the number of prefix announcements and withdrawals, from all peers, in the
  last full clock minute.
- `GET /api/v1/ip/{ip}` - Looks up each address. `{ip}` is an IP address, a host
  name, or a comma-separated list of at most 10 of these. `results` has one
  entry for each address. Every entry has the same fields:
    - `ip`: the address
    - `ptr`: the address's PTR names, or an empty list when it has none
    - `blocklists`: the public blocklists the address is on (see Blocklists), or
      an empty list when it is on none

    Every other field is about the address's live route. Each is null when the
    address has no live route. The fields about the route:
    - `netblock`: the most specific prefix that holds the address
    - `mask_length`: the netblock's mask length
    - `ip_version`: the netblock's IP version, 4 or 6
    - `num_peers`: the number of peers that carry the netblock
    - `asn`: the origin AS that the most peers carry the netblock with
    - `num_origins`: the number of origin ASes that peers carry the netblock
      with
    - `origin_asns`: those origin ASes, in AS number order
    - `first_seen`: when RouteWatch first saw the netblock, or null when the
      database has not stored the netblock yet
    - `last_seen`: the time of the netblock's newest kept route, or null when
      the netblock keeps no route

    The fields about the AS in `asn` come from the database. Each is also null
    when the database does not have it:
    - `handle`: the AS's handle
    - `description`: the AS's description
    - `as_name`: the AS's name in its WHOIS details
    - `org_name`: the name of the organisation that holds the AS
    - `org_id`: the organisation's ID at its registry
    - `address`: the organisation's postal address
    - `country_code`: the organisation's country code
    - `abuse_email`: the address that takes abuse reports for the AS
    - `rir`: the regional internet registry of the AS

    Of the addresses a host name resolves to, only public ones are looked up. A
    name with none, such as one that resolves to a private address, is reported
    as `host name has no public address`, without its addresses. `errors` has
    one entry for each host name that failed. The answer is 400 when:
    - `{ip}` lists more than 10
    - `{ip}` comes to more than 10 addresses once its names are resolved
    - nothing in `{ip}` is an IP address or a host name with a public address

    A PTR lookup gets 2 seconds. An address whose lookup takes longer is
    answered without PTR names.

- `GET /api/v1/as/{asn}` - The AS and the prefixes it originates, IPv4 and IPv6
  apart, each with the time of its newest kept route. A prefix that keeps no
  route with the AS has no `last_updated`. When the database has no record of
  the AS, its handle and description come from the AS list built into
  RouteWatch. The answer then has no `first_seen` or `last_seen`.
- `GET /api/v1/prefix/{prefix}/{length}` - One route for each peer that carries
  the prefix, and the same routes grouped by origin AS in `origins`. Only the
  kept routes have `as_path`, `next_hop` and `last_updated`.
- `GET /api/v1/search/as?q={text}` - Finds the ASes whose handle or description
  contains `{text}`, ignoring case. `data` holds:
    - `query`: `{text}` without its leading and trailing spaces
    - `total`: the number of ASes that match
    - `results`: the first 50 of them in AS number order, each with `asn`,
      `handle` and `description`

    The search reads the AS list built into RouteWatch, not the database. The
    list comes from
    [ipverse/as-metadata](https://github.com/ipverse/as-metadata). A match can
    therefore be an AS that the database has no record of. A `q` that is
    missing, empty or only spaces answers 400.

A JSON answer is `{"status": "ok", "data": ...}`, or for an error
`{"status": "error", "error": {"msg": ..., "code": ...}}`, and carries an
`@meta` object with the time zone, the API version and the time the request
took. The health check is the exception: its 503 answer is also
`{"status": "ok", "data": ...}`, with `error` in `data.status`.

A request that runs past its time limit answers 504: the limit is 30 seconds,
and 4 seconds for the database reads behind `/status.json` and `/api/v1/stats`.
An AS number, prefix or mask length that does not parse answers 400, as does a
mask length out of range for its IP version. An AS answers 404 when it
originates no prefix and has no record in the database or in the AS list built
into RouteWatch. AS 0 always answers 404, because it means no known origin. The
AS name search never lists it. A prefix with no live route answers 404 too. The
HTML pages give these errors as plain text. Under `/api/v1` and `/ip/`, a path
no route matches answers 404 with the message `Not found`, and `POST`, `PUT`,
`PATCH`, `DELETE`, `OPTIONS`, `TRACE` or `CONNECT` answers 405 with
`Method not allowed`. A 500 answer says only `Internal server error`, and a host
name that does not resolve only `could not resolve host name`; the detail goes
to the log.

Every answer carries `Strict-Transport-Security`, `Content-Security-Policy`,
`X-Frame-Options`, `X-Content-Type-Options`, `Referrer-Policy` and
`Permissions-Policy`. The Content-Security-Policy lets a page load content only
from the server, and does not allow inline content with `'unsafe-inline'`: the
pages keep their CSS and JavaScript in inline `<style>` and `<script>` blocks,
so each answer's policy names a random nonce made for that answer, and the
browser applies or runs only the blocks that carry it. The pages therefore use
no `style` attributes, which a nonce cannot cover.

An origin AS of 0 means the route has no known origin. Its AS path ends in an AS
set of several members, any of which may have originated it. The 0 appears in
these fields:

- `origin_asn` in a route
- `asn` and `origin_asns` in an IP lookup
- the key `0` in a prefix's `origins`

The pages show such a route as having no known origin.

## Blocklists

The IP lookup reports which public blocklists list each address. Each result has
a `blocklists` list with one entry for each list the address is on:

- `name`: the list's name, from the lists below
- `description`: what being on the list means
- `updated`: when RouteWatch last downloaded the list, or last heard from its
  publisher that its copy was current

An address on no list has an empty `blocklists`. An address with no live route
still has a result with its `blocklists`.

The lists:

- `dshield`: the 20 networks that DShield saw scan the most targets in the last
  three days. From [DShield](https://isc.sans.edu/feeds_doc.html), downloaded
  every hour.
- `feodo_tracker`: botnet command-and-control servers. From abuse.ch
  [Feodo Tracker](https://feodotracker.abuse.ch/blocklist/), downloaded every 15
  minutes.
- `firehol_abusers_30d`: addresses reported for comment, form or forum spam in
  the last 30 days. From
  [FireHOL](https://iplists.firehol.org/?ipset=firehol_abusers_30d), downloaded
  every 5 hours.
- `firehol_level3`: addresses reported for attacks, spyware or viruses in the
  last 30 days. From
  [FireHOL](https://iplists.firehol.org/?ipset=firehol_level3), downloaded every
  3 hours.
- `spamhaus_drop` and `spamhaus_dropv6`: IPv4 and IPv6 networks that Spamhaus
  says spammers or cyber criminals have hijacked or lease. From
  [Spamhaus DROP](https://www.spamhaus.org/blocklists/do-not-route-or-peer/),
  downloaded once a day. Spamhaus merged its EDROP list into DROP.
- `tor_exits`: Tor exit nodes, where traffic from the Tor network leaves. From
  the [Tor Project](https://check.torproject.org/torbulkexitlist), downloaded
  every hour.

No list is downloaded more often than its publisher updates it or allows. Each
can be downloaded whole without an account or a key.

How RouteWatch keeps the lists:

- It holds each list in memory, so a lookup makes no outside request.
- It keeps a copy of each list in the `blocklists` directory of the state
  directory (see Configuration). Each copy is a file named after its list.
- Beside each copy, a file ending in `.validators` keeps the `Last-Modified` and
  `ETag` the publisher sent with that copy.
- At startup it loads each list from its copy. It downloads a list at once when
  the copy is older than the list's interval, or when there is no copy.
- A download sends the copy's `Last-Modified` as `If-Modified-Since` and its
  `ETag` as `If-None-Match`. The publisher then sends the list only when it has
  changed. Without either, the download asks for the whole list.
- A download fails when it takes over a minute or its file is over 16 MiB.
- When a download fails, or its file does not parse, RouteWatch keeps the
  previous copy and logs the failure. It tries again one interval later.

A successful IP lookup answer carries `Cache-Control` with a `max-age` of 15
minutes, the shortest list interval. The answer for the caller's own address
(`/ip/`) is `private`, and the others are `public`.

## Code Structure

```
routewatch/
├── cmd/
│   ├── routewatch/         # Main daemon entry point
│   ├── asinfo-gen/         # Utility to generate AS info data
│   └── streamdumper/       # Debug utility for raw stream output
├── internal/
│   ├── blocklist/          # Public IP blocklists: downloads and lookups
│   ├── bogon/              # Reserved prefixes and AS numbers
│   ├── routewatch/         # Core application logic
│   ├── server/             # HTTP server and handlers
│   ├── database/           # SQLite storage layer
│   ├── routetable/         # The whole routing table in memory
│   ├── streamer/           # RIPE RIS Live client
│   ├── ristypes/           # BGP message data structures
│   ├── logger/             # Structured logging wrapper
│   ├── metrics/            # Performance metrics tracking
│   ├── config/             # Configuration management
│   ├── templates/          # HTML templates
│   ├── version/            # Build version information
│   └── whois/              # WHOIS lookups of AS details
└── pkg/
    └── asinfo/             # AS information lookup (public API)
```

## Architecture Overview

### Component Relationships

`internal/routewatch/app.go` builds and starts these parts:

```
┌──────────────┐    ┌──────────────┐    ┌──────────────┐
│   Streamer   │───▶│   Handlers   │───▶│   Database   │
│              │    │              │    │              │
│ RIS Live     │    │ - ASHandler  │    │ SQLite with  │
│ HTTP stream  │    │ - PeerHandler│    │ WAL mode     │
│ client       │    │ - PrefixHdlr │    │              │
│              │    │ - PeeringHdlr│    │ Tables:      │
└──────────────┘    └──────┬───────┘    │ - asns       │
                           │            │ - prefixes   │
                           ▼            │ - routes     │
                    ┌──────────────┐    │ - route_peers│
                    │ Routing table│───▶│ - peerings   │
                    │ in memory    │    │ - bgp_peers  │
                    └──────────────┘    └──────────────┘
                           ▲                   ▲
┌──────────────┐    ┌──────┴───────┐           │
│    Server    │───▶│   Handlers   │───────────┘
│              │    │              │
│ Chi router   │    │ Status, API  │
│ port 8080    │    │ AS, Prefix   │
└──────────────┘    └──────────────┘
```

The prefix handler keeps the routing table in memory current. It writes the
table's changes to the database every second. The web handlers read routes from
the routing table, and everything else from the database.

### Execution Flow

1. **Startup** (`cmd/routewatch/main.go` → `internal/routewatch/cli.go`)
    - Uber fx dependency injection initializes all components
    - Signal handlers registered for graceful shutdown

2. **Initialization** (`internal/routewatch/app.go`)
    - Database created with SQLite schema (WAL mode, page caches as in "Memory")
    - The routing table read from the `routes` table into memory, before the
      streamer connects. A full table takes about 25 seconds to read.
    - Message handlers registered with the streamer
    - HTTP server started on configured port

3. **Message Processing Pipeline**

    ```
    RIS Live Stream → JSON Parser → Message Dispatcher → Handler Queues
        ├→ Batch Writers → SQLite
        └→ PrefixHandler → Routing Table in Memory → SQLite, every second
    ```

    - Streamer connects to `ris-live.ripe.net` via HTTP
    - Parses BGP UPDATE messages from JSON stream
    - Dispatches to registered handlers based on message type
    - Each handler has its own queue with backpressure handling
    - Handlers batch writes for efficiency: the AS, peer and prefix handlers
      write 30,000, 10,000 and 25,000 updates at a time, or after 2, 2 and 1
      seconds; the peering handler writes the peerings it has collected every 30
      seconds

4. **Handler Details**
    - **ASHandler**: Tracks all ASNs seen in AS paths
    - **PeerHandler**: Records BGP peer information
    - **PrefixHandler**: Keeps the routing table in memory current. It applies
      each update to the table as it handles it:
        - An announcement gives the peer a route to the prefix.
        - A withdrawal removes the peer's route. So does an announcement whose
          AS path is empty or holds AS 0 anywhere.
        - A peer whose BGP session goes down leaves every prefix.

        Every second it writes the changed prefix and origin AS pairs to the
        `routes` table. A write takes at most 20,000 pairs, and the rest follow
        in the next writes. It also stores each new prefix with the time it was
        first seen. Peers are known by their IP address. A peer IP can have
        sessions on two collectors. A session going down on one then also
        removes the routes the other still carries, until they are announced
        again.

    - **PeeringHandler**: Extracts AS peering relationships from AS paths

5. **HTTP Request Flow**
    ```
    Request → Chi Router → Middleware (timeout, logging) → Handler → Routing Table or Database → Response
    ```

### Key Design Patterns

- **Routing Table in Memory**: `internal/routetable` holds the whole routing
  table. For each prefix and origin AS it keeps:
    - every peer whose current route to the prefix has that origin AS, one bit
      per peer;
    - the newest two of those routes in full, with AS path and time.

    Of the other routes, only the peer is kept. The prefix handler writes the
    changed pairs to the `routes` table every second, one row per prefix and
    origin AS. Startup reads that table back into memory before the streamer
    connects. A crash loses only the changes not yet written.

- **Batched Writes**: The handlers write the feed's updates to the database in
  batches for performance
- **Backpressure**: Probabilistic message dropping when queues exceed 50%
  capacity
- **Graceful Shutdown**: on SIGTERM or SIGINT the streamer stops, the handlers
  write the batches they hold, the HTTP server stops and the database is
  checkpointed and closed. The daemon gives this 8 seconds and exits with status
  0; "Running under upaas" says what a stop loses, logs and leaves undone when
  the 8 seconds run out.
- **Reconnection**: Any end of the stream other than shutdown, including RIS
  Live ending the response normally, marks the feed disconnected and reconnects
  with exponential backoff (5s-320s). The backoff resets after a stream that
  lasted more than 30s. A stream with no line for 60s is dropped and
  reconnected.
- **IP Lookup**: the routing table in memory finds the most specific prefix that
  holds the address. It looks up the address's prefix at each mask length,
  longest first: at most 25 lookups for IPv4 and 113 for IPv6. Prefixes shorter
  than /8 for IPv4 or /16 for IPv6, such as a default route, are not used.

### Database Schema

```sql
-- Core tables
asns(asn PRIMARY KEY, handle, description,
     as_name, org_name, org_id, address, country_code, abuse_email,
     abuse_phone, tech_email, tech_phone, rir, rir_registration_date,
     rir_last_modified, whois_raw,
     first_seen, last_seen, whois_updated_at)
prefixes_v4(prefix PRIMARY KEY, first_seen) WITHOUT ROWID
prefixes_v6(prefix PRIMARY KEY, first_seen) WITHOUT ROWID

-- The routing table, one row per prefix and origin AS that some peer
-- carries: the peer set, and the newest two routes in full
routes(prefix, origin_asn, peers,
       newest_peer, newest_as_path, newest_time,
       older_peer, older_as_path, older_time,
       PRIMARY KEY (prefix, origin_asn)) WITHOUT ROWID
-- The peers of the routes table, each with its number in the peer sets
route_peers(number PRIMARY KEY, address UNIQUE)

-- Relationship tracking
peerings(id PRIMARY KEY, as_a, as_b, first_seen, last_seen,
         UNIQUE(as_a, as_b))
bgp_peers(id PRIMARY KEY, peer_ip UNIQUE, peer_asn, first_seen, last_seen,
          last_message_type)

-- Nothing in the daemon writes this table
announcements(id PRIMARY KEY, prefix_id, peer_asn, origin_asn, path,
              next_hop, timestamp, is_withdrawal)
```

The indexes are in `internal/database/schema.sql`.

## Configuration

Configuration is handled via environment variables and OS-specific paths:

| Variable           | Default                          | Description                                                 |
| ------------------ | -------------------------------- | ----------------------------------------------------------- |
| `PORT`             | `8080`                           | HTTP server port, a whole number from 1 to 65535            |
| `DEBUG`            | (empty)                          | Set to `routewatch` for debug logging                       |
| `XDG_DATA_HOME`    | `/var/lib` (in the Docker image) | Base of the state directory; must be an absolute path       |
| `GOMEMLIMIT`       | `2560MiB` (in the Docker image)  | Go soft memory limit; see Memory                            |
| `MALLOC_ARENA_MAX` | `2` (in the Docker image)        | glibc malloc arena cap, a positive whole number; see Memory |
| `TRUSTED_PROXIES`  | the RFC 1918 ranges              | Addresses and CIDR ranges of trusted proxies; see below     |

A variable that is set to an invalid value stops the start with an error and a
non-zero exit. An empty variable counts as unset. `TRUSTED_PROXIES` is the one
exception: set but empty, it makes the server trust no proxy.

`TRUSTED_PROXIES` names the reverse proxies whose `X-Forwarded-For` the server
believes:

- When it is not set, the server trusts the private ranges of RFC 1918:
    - `10.0.0.0/8`
    - `172.16.0.0/12`
    - `192.168.0.0/16`
- When it lists addresses and CIDR ranges, comma-separated, such as
  `10.0.0.0/8,192.0.2.1`, the server trusts those instead.
- When it is set but empty (`TRUSTED_PROXIES=`), the server trusts no proxy.

For a request from a trusted proxy, the request log and `/ip/` use the last
address in `X-Forwarded-For` that is not a trusted proxy. Each proxy adds the
address it received the request from to the end, so the entries before it are
whatever the client sent. For a request from any other address they use the
address the request came from, whatever forwarding headers it carries.
`X-Real-IP` and `True-Client-IP` are not read.

State directory (the database and the blocklist copies):

- macOS: `~/Library/Application Support/berlin.sneak.app.routewatch/`
- Linux: `/var/lib/berlin.sneak.app.routewatch/` when running as root, otherwise
  `$XDG_DATA_HOME/berlin.sneak.app.routewatch/` (with `XDG_DATA_HOME` unset,
  `~/.local/share/berlin.sneak.app.routewatch/`). In the Docker image this is
  `/var/lib/berlin.sneak.app.routewatch/`.

## Memory

The daemon keeps the whole routing table in memory. The image sets ceilings that
keep it inside a 5 GiB container.

Budget:

- Go heap: a 2.5 GiB soft limit (`GOMEMLIMIT=2560MiB`, set in the image).
    - The routing table takes about 700 bytes per prefix and origin AS. A full
      table has about 1.4 million of these pairs, so it takes about 1 GB.
    - The rest of the heap takes up to about 0.5 GiB. Most of it is the
      handlers' batches and queues, described below.
    - The remaining 1 GiB is room for the garbage collector. With much less, it
      would run almost all the time.
- SQLite: 656 MiB of page cache.
    - The connection that does every write has 512 MiB. It keeps the pages the
      writes touch from one write to the next. With a full routing table, the
      `routes` table takes about 450 MB, so its pages fit.
    - Each of the 9 reading connections has 16 MiB. SQLite empties a reading
      connection's cache whenever another connection has written.
    - A 768 MiB soft heap limit sits above the caches, so it does not shrink
      them. A 1 GiB hard heap limit caps the C library.
- glibc allocator: the SQLite C library allocates memory with glibc `malloc`.
    - glibc keeps freed page-cache chunks in per-arena free lists instead of
      returning them to the kernel. So the process's RSS follows the high-water
      mark of those arenas, not SQLite's live heap.
    - glibc creates up to eight arenas per core, so on a many-core host RSS
      grows with the core count.
    - The image sets `MALLOC_ARENA_MAX=2` to bound it. Two arenas cost nothing
      here, because database writes already run one at a time.
- About 0.2 GiB for everything else in the runtime.

The ceilings add up to 3.7 GiB: the Go heap's 2.5 GiB, SQLite's 1 GiB and the
0.2 GiB for the rest.

Run the container with a memory limit of 5 GiB and swap disabled:

```bash
docker run --memory=5g --memory-swap=5g ...
```

or the equivalent in your deployment tool. This leaves headroom above the
ceilings for spikes and the kernel page cache.

To replace the image's Go soft limit, set `GOMEMLIMIT` in the environment, for
example `-e GOMEMLIMIT=3GiB`. A limit close to the routing table's size makes
the garbage collector run almost all the time. `MALLOC_ARENA_MAX` can be
overridden the same way, but raising it lets RSS climb again on a many-core
host.

What happens at each limit:

- Go soft limit: as the heap approaches `GOMEMLIMIT`, the runtime runs garbage
  collection more aggressively rather than growing further.
- SQLite: at the 768 MiB soft heap limit it recycles its page cache instead of
  allocating more. At the 1 GiB hard heap limit a statement fails with an
  out-of-memory error, and the handler logs it. The prefix handler writes the
  routing table's changes again on a later write. The other writes drop their
  batch. The process keeps running.
- Handler queues: each of the four handler queues holds at most 20,000 messages.
  When a queue fills, the streamer drops messages instead of blocking, except a
  `STATE` message, which it holds, in feed order, until there is room: a few
  hundred bytes each, about ten a second.
- Handler batches: while the AS, peer or prefix handler writes a batch to the
  database, the updates that arrive collect in its next batch. That batch stops
  at ten times the batch size:
    - 300,000 updates for the AS handler (`asnBatchMaxSize`, about 10 MiB);
    - 100,000 for the peer handler (`peerBatchMaxSize`, about 10 MiB);
    - 250,000 for the prefix handler (`prefixBatchMaxSize`, about 15 MiB).

    At that size the handler takes nothing more off its queue until the write
    ends. The queue then fills, and the streamer drops messages as above. With
    the batches being written, the three handlers' batches take at most about
    0.1 GiB of the Go heap.

With `DEBUG=routewatch` the daemon logs a `System stats` line every 60 seconds
with the goroutine count and Go memory figures.

## Running under upaas

What the [upaas](https://git.eeqj.de/sneak/upaas) app needs:

- Container port: `8080`.
- Volume: one, at container path `/var/lib/berlin.sneak.app.routewatch`.
- Environment: nothing. The server trusts a reverse proxy at an RFC 1918 address
  without `TRUSTED_PROXIES` (see Configuration). Set it only when the proxy
  connects from any other address. Leave `XDG_DATA_HOME`, `GOMEMLIMIT` and
  `MALLOC_ARENA_MAX` at the image's values. `DEBUG=routewatch` is optional and
  adds the `System stats` memory line to the log.
- Memory Limit: `5g`, the 5 GiB limit from Memory above. upaas sets no swap
  limit, so on a host with swap Docker allows the same amount of swap again.
- Health check: the image's `HEALTHCHECK` requests
  `/.well-known/healthcheck.json` on the container port. upaas reads the
  container's health 60 seconds after a deploy and fails the deploy unless it is
  `healthy`. For the first 60 seconds, Docker 25 and later runs the check every
  5 seconds, and the first pass makes the container healthy. Startup first reads
  the routing table, which takes about 25 seconds when the table is full.
- Stop grace: the container needs 9 seconds between SIGTERM and SIGKILL. upaas
  gives 10 (`stopTimeoutSeconds` in its `internal/docker/client.go`), as
  `docker stop` does by default. The daemon allows 8 seconds for the handlers to
  write what they hold and for the requests in progress to finish. Then it stops
  the HTTP server, closes the database and exits with status 0. A longer grace
  changes nothing. What is not written after 8 seconds is lost, and the streamer
  and each handler log how much.
    - The prefix handler's last write takes at most 20,000 of the routing
      table's changed pairs. A full-table peer's session going down changes
      about a million pairs, which take about a minute to write. A stop in that
      minute leaves the rest unwritten.
    - A lost withdrawal or peer down leaves its routes in the `routes` table,
      and the next start reads them back. They stay until the peer sends another
      update for the prefix, or its session goes down.
    - When the 8 seconds run out the database is not closed. When the database
      next opens, SQLite rolls back any write the exit cut off.

## Development

```bash
# Run tests
make test

# Format code
make fmt

# Run linter
make lint

# Build
make build
```

## Entrypoints

This repository adheres to the
[Scripts to Rule Them All](https://github.com/github/scripts-to-rule-them-all)
standard: normalized scripts in `script/` are the entrypoints for the
development workflow, and the Makefile targets are thin shims that call them. We
provide:

- `script/bootstrap` — install all development dependencies (git, make, go, the
  pinned node and yarn, prettier from `yarn.lock`, Go module download)
- `script/setup` — make a fresh clone ready for development: runs
  `script/bootstrap`, then `script/install-precommit`
- `script/projectname` — print the project name (used for the Docker image tag)
- `script/test` — build the `test` phase of the `Dockerfile` alone, which runs
  `go test -short -count=1 -timeout 90s -race -cover ./...` (verbose rerun on
  failure). The `-short` flag skips the live-network integration test so the run
  is deterministic and offline.
- `script/lint` — build the `lint` phase of the `Dockerfile` alone, which runs
  golangci-lint v2.14.0 with `.golangci.yml`
- `script/fmt` — format the Go code with `go fmt` and the markdown with prettier
  (writes)
- `script/fmt-check` — check formatting (read-only)
- `script/check` — run `script/test`, `script/lint`, and `script/fmt-check`
- `script/docker` — build the Docker image tagged via `script/projectname`
- `script/cibuild` — CI entrypoint: runs `script/bootstrap` and `script/check`,
  then builds the image
- `script/precommit` — pre-commit gate: `go mod tidy` (must not change `go.mod`
  or `go.sum`) and `go fmt` (writes), then `script/check`
- `script/install-precommit` — install the git pre-commit hook that runs
  `script/precommit`
- `script/asupdate` — rebuild the AS list built into RouteWatch,
  `pkg/asinfo/asdata.json.gz`, from the `as.csv` file of
  [ipverse/as-metadata](https://github.com/ipverse/as-metadata). It needs the
  network. It runs `cmd/asinfo-gen`, which fails when it finds no AS. When
  `cmd/asinfo-gen` fails, the old list stays and the script exits non-zero.

Tests and the linter run only in Docker, as the `test` and `lint` phases of the
`Dockerfile`; the image cannot be built unless both pass. Every `docker build`
in `script/` passes a new `CACHEBUST` value, so each run executes every check,
while Go's build cache and golangci-lint's cache are kept between builds.

The live-network integration test `TestRouteWatchLiveFeed` streams the RIPE RIS
feed for a few seconds and is skipped in short mode. To run it on demand, invoke
`go test` directly without `-short`:
`go test -run TestRouteWatchLiveFeed ./internal/routewatch/`.

## License

MIT. See [`LICENSE`](LICENSE).

## Author

[@sneak](https://sneak.berlin)