From 5c202beffb6396f5064fd1c24195ee59b98d6a0c Mon Sep 17 00:00:00 2001 From: sneak Date: Mon, 19 Oct 2026 12:24:10 +0000 Subject: [PATCH] Bring README and TODO.md up to date with the code (closes #72) README "HTTP Endpoints" lists every route setupRoutes registers with what it answers, including the error answers, and "Database Schema" matches schema.sql. Other sentences the code contradicts are corrected: the handlers' batch sizes, the stream client, the macOS state directory, the build command and what script/precommit checks; "Code Structure" gains the version and whois packages. TODO.md gets one line for each issue landed on next since 2026-10-03, a current Status and Future Steps, and a Next Step naming the run on the live feed and the audit after it. Model: opus-5-5 --- README.md | 112 +++++++++++++++++++++++++++++++++++++++--------------- TODO.md | 89 ++++++++++++++++++++++++++++++++++++------- 2 files changed, 157 insertions(+), 44 deletions(-) diff --git a/README.md b/README.md index 3526890..3e5a2e7 100644 --- a/README.md +++ b/README.md @@ -38,21 +38,53 @@ DEBUG=routewatch ./routewatch ### Web Interface -- `GET /` - Redirects to /status -- `GET /status` - HTML status dashboard -- `GET /status.json` - JSON statistics -- `GET /as/{asn}` - AS detail page (HTML) -- `GET /prefix/{prefix}` - Prefix detail page (HTML) -- `GET /prefixlength/{length}` - IPv4 prefixes by mask length -- `GET /prefixlength6/{length}` - IPv6 prefixes by mask length -- `GET /ip/{ip}` - Redirects to prefix containing the IP +- `GET /` - Home page (HTML): headline figures, and lookups by AS number and by + IP address +- `GET /status` - HTML status dashboard, which reads `/api/v1/stats` +- `GET /status.json` - JSON statistics: feed, memory, database and route figures +- `GET /.well-known/healthcheck.json` - Health check (JSON): 200 when the + database answers and the RIS Live feed is connected, otherwise 503; + `data.status` is `ok` or `error` and `data.checks` says which check failed +- `GET /as/{asn}` - AS detail page (HTML): the prefixes the AS originates and + its peers +- `GET /prefix/{prefix}/{length}` - Prefix detail page (HTML), such as + `/prefix/192.0.2.0/24`: every live route for the prefix, one per peer, and its + origin ASes +- `GET /prefixlength/{length}` - Up to 500 IPv4 prefixes with that mask length, + picked at random, each with its newest route (HTML) +- `GET /prefixlength6/{length}` - The same for IPv6 +- `GET /ip/{addr}` - The same JSON answer as `/api/v1/ip/{ip}` +- `GET /ip/` - The same, for the caller's own address, or the one a proxy names + in `True-Client-IP`, `X-Real-IP` or `X-Forwarded-For` ### API v1 -- `GET /api/v1/stats` - Detailed statistics with handler metrics -- `GET /api/v1/ip/{ip}` - Look up AS information for an IP address -- `GET /api/v1/as/{asn}` - Get prefixes announced by an AS -- `GET /api/v1/prefix/{prefix}` - Get routes for a specific prefix +- `GET /api/v1/stats` - The `/status.json` figures plus connection, garbage + collector, stream and handler queue statistics +- `GET /api/v1/ip/{ip}` - Looks up each address: its most specific live route + (`netblock`), that route's origin AS with its WHOIS details, and the address's + PTR names. `{ip}` is an IP address, a host name (each address it resolves to + is looked up), or a comma-separated list of these. `results` has one entry per + address with a live route and `errors` one per address or name that failed. + The answer is 400 when nothing in `{ip}` is an IP address or a host name that + resolves, and 404 when none of the addresses has a live route. +- `GET /api/v1/as/{asn}` - The AS and the prefixes it originates, IPv4 and IPv6 + apart, each with when its newest route was updated +- `GET /api/v1/prefix/{prefix}/{length}` - Every live route for the prefix, one + per peer, and the same routes grouped by origin AS in `origins` + +A JSON answer is `{"status": "ok", "data": ...}`, or for an error +`{"status": "error", "error": {"msg": ..., "code": ...}}`, and carries an +`@meta` object with the time zone, the API version and the time the request +took. The health check is the exception: its 503 answer is also +`{"status": "ok", "data": ...}`, with `error` in `data.status`. + +A request that runs past its time limit answers 504: the limit is 30 seconds, +and 4 seconds for the database reads behind `/status.json` and `/api/v1/stats`. +An AS number, prefix or mask length that does not parse answers 400, as does a +mask length out of range for its IP version. An AS the database does not hold, +or a prefix with no live route, answers 404. The HTML pages give these errors as +plain text. An origin AS of 0 (`origin_asn` in a route, `asn` in an IP lookup, the key `0` in a prefix's `origins`) means the route has no known origin: its AS path ends @@ -76,7 +108,9 @@ routewatch/ │ ├── logger/ # Structured logging wrapper │ ├── metrics/ # Performance metrics tracking │ ├── config/ # Configuration management -│ └── templates/ # HTML templates +│ ├── templates/ # HTML templates +│ ├── version/ # Build version information +│ └── whois/ # WHOIS lookups of AS details └── pkg/ └── asinfo/ # AS information lookup (public API) ``` @@ -95,7 +129,7 @@ routewatch/ │ │ Streamer │───▶│ Handlers │───▶│ Database │ │ │ │ │ │ │ │ │ │ │ │ RIS Live │ │ - ASHandler │ │ SQLite with │ │ -│ │ WebSocket │ │ - PeerHandler│ │ WAL mode │ │ +│ │ HTTP stream │ │ - PeerHandler│ │ WAL mode │ │ │ │ client │ │ - PrefixHdlr │ │ │ │ │ │ │ │ - PeeringHdlr│ │ Tables: │ │ │ └──────────────┘ └──────────────┘ │ - asns │ │ @@ -131,7 +165,10 @@ routewatch/ - Parses BGP UPDATE messages from JSON stream - Dispatches to registered handlers based on message type - Each handler has its own queue with backpressure handling - - Handlers batch writes for efficiency (25K-30K ops, 1-2s timeout) + - Handlers batch writes for efficiency: the AS, peer and prefix handlers + write 30,000, 10,000 and 25,000 updates at a time, or after 2, 2 and 1 + seconds; the peering handler writes the peerings it has collected every 30 + seconds 4. **Handler Details** - **ASHandler**: Tracks all ASNs seen in AS paths @@ -150,7 +187,8 @@ routewatch/ ### Key Design Patterns -- **Batched Writes**: All database operations are batched for performance +- **Batched Writes**: The handlers write the feed's updates to the database in + batches for performance - **Backpressure**: Probabilistic message dropping when queues exceed 50% capacity - **Graceful Shutdown**: 60-second timeout, flushes all pending batches @@ -169,21 +207,35 @@ routewatch/ ```sql -- Core tables -asns(id, number, handle, description, first_seen, last_seen) -prefixes_v4(prefix, first_seen) -prefixes_v6(prefix, first_seen) +asns(asn PRIMARY KEY, handle, description, + as_name, org_name, org_id, address, country_code, abuse_email, + abuse_phone, tech_email, tech_phone, rir, rir_registration_date, + rir_last_modified, whois_raw, + first_seen, last_seen, whois_updated_at) +prefixes_v4(prefix PRIMARY KEY, first_seen) WITHOUT ROWID +prefixes_v6(prefix PRIMARY KEY, first_seen) WITHOUT ROWID --- Live routing tables (one per IP version) -live_routes_v4(id, prefix, mask_length, origin_asn, peer_ip, as_path, - next_hop, last_updated) -live_routes_v6(id, prefix, mask_length, origin_asn, peer_ip, as_path, - next_hop, last_updated) +-- Live routing tables (one per IP version), one route per peer and prefix +live_routes_v4(id INTEGER PRIMARY KEY, prefix, mask_length, origin_asn, + peer_ip, as_path, next_hop, last_updated, + UNIQUE(peer_ip, prefix)) +live_routes_v6(id INTEGER PRIMARY KEY, prefix, mask_length, origin_asn, + peer_ip, as_path, next_hop, last_updated, + UNIQUE(peer_ip, prefix)) -- Relationship tracking -peerings(id, as_a, as_b, first_seen, last_seen) -bgp_peers(id, peer_ip, peer_asn, last_message_type, last_seen) +peerings(id PRIMARY KEY, as_a, as_b, first_seen, last_seen, + UNIQUE(as_a, as_b)) +bgp_peers(id PRIMARY KEY, peer_ip UNIQUE, peer_asn, first_seen, last_seen, + last_message_type) + +-- Nothing in the daemon writes this table +announcements(id PRIMARY KEY, prefix_id, peer_asn, origin_asn, path, + next_hop, timestamp, is_withdrawal) ``` +The indexes are in `internal/database/schema.sql`. + ## Configuration Configuration is handled via environment variables and OS-specific paths: @@ -201,7 +253,7 @@ non-zero exit. An empty variable counts as unset. State directory (database location): -- macOS: `~/Library/Application Support/routewatch/` +- macOS: `~/Library/Application Support/berlin.sneak.app.routewatch/` - Linux: `/var/lib/berlin.sneak.app.routewatch/` when running as root, otherwise `$XDG_DATA_HOME/berlin.sneak.app.routewatch/` (with `XDG_DATA_HOME` unset, `~/.local/share/berlin.sneak.app.routewatch/`). In the Docker image this is @@ -301,7 +353,7 @@ make fmt make lint # Build -make +make build ``` ## Entrypoints @@ -330,8 +382,8 @@ provide: - `script/docker` — build the Docker image tagged via `script/projectname` - `script/cibuild` — CI entrypoint: runs `script/bootstrap` and `script/check`, then builds the image -- `script/precommit` — pre-commit gate: `go mod tidy` + `go fmt` (must not - change files), then `script/check` +- `script/precommit` — pre-commit gate: `go mod tidy` (must not change `go.mod` + or `go.sum`) and `go fmt` (writes), then `script/check` - `script/install-precommit` — install the git pre-commit hook that runs `script/precommit` diff --git a/TODO.md b/TODO.md index ade4f7b..a4f5e9a 100644 --- a/TODO.md +++ b/TODO.md @@ -10,28 +10,91 @@ # Status -pre-1.0. No git tags. The linter and the tests are phases of the Docker build, -and the Gitea workflow runs `script/cibuild` (the format check of the Go code -and the markdown, both phases, then the image) on every push. The image sets -memory ceilings for a 5 GiB container (README "Memory") and the README says how -to run it under upaas (README "Running under upaas"). A 35-hour run of `3898daa` -on the live feed peaked at about 1 GiB, without a container memory limit. +pre-1.0. No git tags. `main` last took `next` on 2026-10-06, through +https://git.eeqj.de/sneak/routewatch/pulls/6. The linter and the tests are +phases of the Docker build, and the Gitea workflow runs `script/cibuild` (the +format check of the Go code and the markdown, both phases, then the image) on +every push. The image sets memory ceilings for a 5 GiB container (README +"Memory") and the README says how to run it under upaas (README "Running under +upaas"). A 25-hour run of `a9b6860` on the live feed peaked at about 0.6 GiB +(https://git.eeqj.de/sneak/routewatch/issues/3). A run of `825afa2` on the live +feed found the prefix handler dropping about a quarter of its updates because +its route writes were slower than the feed +(https://git.eeqj.de/sneak/routewatch/issues/96); the cheaper writes since then +are measured by a benchmark, not yet on the live feed. # Next Step -`next` waits for sneak to merge it to `main` through -https://git.eeqj.de/sneak/routewatch/pulls/6. After that, setting routewatch up -under upaas on fsn1app1 and deploying it are his -(https://git.eeqj.de/sneak/routewatch/issues/31), and so is the run under a real -5 GiB limit (https://git.eeqj.de/sneak/routewatch/issues/3). +Run `next` on the live feed, as README "Running under upaas" describes, for the +final check https://git.eeqj.de/sneak/routewatch/issues/96 names: whether the +prefix handler still drops updates after the first hour. If it does, a new issue +follows with the numbers. Then an audit of the code at the commit that run +tested. # Completed Steps +- 2026-10-19: README "HTTP Endpoints" lists every route with what it answers, + "Database Schema" matches `schema.sql`, and the rest of the README matches the + code; this file brought up to date (closes #72) +- 2026-10-19: route writes are cheaper: the live route and prefix tables lose + their random UUID keys and the prefix tables their duplicate index, a route + update sets only the columns that change, and every write goes through one + connection with a page cache of 1 GiB (README "Memory"). A benchmark writes + the prefix handler's largest batch faster than the feed sends it (closes #96) +- 2026-10-19: the WAL checkpoint, the incremental vacuum and ANALYZE take the + database lock every batch write holds, so a vacuum no longer fails with + "database is locked" and an ANALYZE no longer makes a batch write lose its + updates; ANALYZE runs as `PRAGMA optimize` with an analysis limit (closes #94) +- 2026-10-19: shutdown stops the streamer first, and each handler writes + everything left on its queue before it stops, so the updates waiting there are + no longer lost (closes #69) +- 2026-10-19: a path that ends in an AS set of several members has origin AS 0, + no known origin, instead of the set's last member, and the set adds no + peerings; the pages show such a route as having no known origin (closes #75) +- 2026-10-19: `bgp_peer_count` drops to 0 when the last peer goes down (closes + #92) +- 2026-10-19: `bgp_peer_count` counts every peer the feed has heard from, not + only those whose session opened since startup, and drops a peer whose session + goes down (closes #74) - 2026-10-19: the shared files are re-vendored from `sneak/prompts` at `0653579`: golangci-lint v2.14.0 and its configuration, lint and test as `Dockerfile` phases that keep Go's and golangci-lint's caches, markdown formatted with prettier, agent guidance in `AGENTS.md`; the code is fixed for what the new linter configuration finds (closes #52) +- 2026-10-19: the streamer reads feed lines up to 1 MiB, so a line over 64 KiB + no longer ends the connection (closes #76) +- 2026-10-19: a peer's live routes are removed when its BGP session goes down; + `STATE` messages are never dropped for backpressure, and one that finds its + queue full waits for room ahead of the handler's other messages (closes #56) +- 2026-10-19: the prefix-length pages pick their sample with a bounded number of + index lookups instead of reading every route of that length, and list each + prefix once (closes #66) +- 2026-10-19: a request past its time limit answers 504 instead of 408, and + `/api/v1/stats` answers 504 instead of 500 when its read runs out of time + (closes #70) +- 2026-10-19: the AS, peer and prefix handlers write each batch outside the lock + their message handling needs, so messages keep coming off their queues during + a write; meanwhile the next batch grows to at most ten batch sizes (closes + #65) +- 2026-10-19: a live route is identified by prefix and peer, so an announcement + with a new origin AS replaces the peer's route and a withdrawal removes it + whatever its origin AS (closes #55) +- 2026-10-14: the streamer reconnects when RIS Live ends the stream or sends + nothing for 60 seconds, and the health check reports the feed disconnected + meanwhile (closes #53) +- 2026-10-14: the prefix page takes the names of the origin ASes from the prefix + query instead of reading every route of each origin AS (closes #67) +- 2026-10-14: the daemon exits with status 1 when its web server cannot listen + on its port, instead of running on without it (closes #73) +- 2026-10-14: the prefix pages parse the prefix before looking it up: one that + does not parse answers 400, and any valid way of writing a prefix finds its + routes (closes #71) +- 2026-10-14: the IP lookup ignores routes shorter than /8 for IPv4 and /16 for + IPv6, so a default route is no longer named as an address's owner (closes #57) +- 2026-10-14: a prefix batch keeps each peer's last update for a prefix, instead + of one update per prefix across all peers (closes #54) +- 2026-10-14: `reconnect_count` in `/api/v1/stats` counts every reconnect + (closes #68) - 2026-10-03: `/api/v1/stats` serves the prefix distribution from memory, seeded at startup and adjusted on every live-route write, so a request no longer reads every live route (closes #30) @@ -116,6 +179,4 @@ under upaas on fsn1app1 and deploying it are his # Future Steps -- Production memory under 5 GiB: whether to test under a real 5 GiB container - limit on fsn1app1 is open for sneak - (https://git.eeqj.de/sneak/routewatch/issues/3) +None open.