Bring README and TODO.md up to date with the code (closes #72)

README "HTTP Endpoints" now lists every route the server registers with
what it answers, errors included: / is a home page and /ip/{addr} answers
JSON, neither redirects; /ip/ looks up the caller; the IP endpoints take
host names and comma-separated lists; 400, 404 and 504 are described, and
the health check's envelope. "Database Schema" matches schema.sql, and
other sentences the code contradicted are corrected. TODO.md gets a dated
line for each issue landed since 2026-10-03, with Status and Next Step
current.

Model: opus-5-5

README.md

@@ -38,21 +38,53 @@ DEBUG=routewatch ./routewatch
 
 ### Web Interface
 
-- `GET /` - Redirects to /status
+- `GET /` - Home page (HTML): headline figures, and lookups by AS number and by
-- `GET /status` - HTML status dashboard
+  IP address
-- `GET /status.json` - JSON statistics
+- `GET /status` - HTML status dashboard, which reads `/api/v1/stats`
-- `GET /as/{asn}` - AS detail page (HTML)
+- `GET /status.json` - JSON statistics: feed, memory, database and route figures
-- `GET /prefix/{prefix}` - Prefix detail page (HTML)
+- `GET /.well-known/healthcheck.json` - Health check (JSON): 200 when the
-- `GET /prefixlength/{length}` - IPv4 prefixes by mask length
+  database answers and the RIS Live feed is connected, otherwise 503;
-- `GET /prefixlength6/{length}` - IPv6 prefixes by mask length
+  `data.status` is `ok` or `error` and `data.checks` says which check failed
-- `GET /ip/{ip}` - Redirects to prefix containing the IP
+- `GET /as/{asn}` - AS detail page (HTML): the prefixes the AS originates and
+  its peers
+- `GET /prefix/{prefix}/{length}` - Prefix detail page (HTML), such as
+  `/prefix/192.0.2.0/24`: every live route for the prefix, one per peer, and its
+  origin ASes
+- `GET /prefixlength/{length}` - Up to 500 IPv4 prefixes with that mask length,
+  picked at random, each with its newest route (HTML)
+- `GET /prefixlength6/{length}` - The same for IPv6
+- `GET /ip/{addr}` - The same JSON answer as `/api/v1/ip/{ip}`
+- `GET /ip/` - The same, for the caller's own address, or the one a proxy names
+  in `True-Client-IP`, `X-Real-IP` or `X-Forwarded-For`
 
 ### API v1
 
-- `GET /api/v1/stats` - Detailed statistics with handler metrics
+- `GET /api/v1/stats` - The `/status.json` figures plus connection, garbage
-- `GET /api/v1/ip/{ip}` - Look up AS information for an IP address
+  collector, stream and handler queue statistics
-- `GET /api/v1/as/{asn}` - Get prefixes announced by an AS
+- `GET /api/v1/ip/{ip}` - Looks up each address: its most specific live route
-- `GET /api/v1/prefix/{prefix}` - Get routes for a specific prefix
+  (`netblock`), that route's origin AS with its WHOIS details, and the address's
+  PTR names. `{ip}` is an IP address, a host name (each address it resolves to
+  is looked up), or a comma-separated list of these. `results` has one entry per
+  address with a live route and `errors` one per address or name that failed.
+  The answer is 400 when nothing in `{ip}` is an IP address or a host name that
+  resolves, and 404 when none of the addresses has a live route.
+- `GET /api/v1/as/{asn}` - The AS and the prefixes it originates, IPv4 and IPv6
+  apart, each with when its newest route was updated
+- `GET /api/v1/prefix/{prefix}/{length}` - Every live route for the prefix, one
+  per peer, and the same routes grouped by origin AS in `origins`
+
+A JSON answer is `{"status": "ok", "data": ...}`, or for an error
+`{"status": "error", "error": {"msg": ..., "code": ...}}`, and carries an
+`@meta` object with the time zone, the API version and the time the request
+took. The health check is the exception: its 503 answer is also
+`{"status": "ok", "data": ...}`, with `error` in `data.status`.
+
+A request that runs past its time limit answers 504: the limit is 30 seconds,
+and 4 seconds for the database reads behind `/status.json` and `/api/v1/stats`.
+An AS number, prefix or mask length that does not parse answers 400, as does a
+mask length out of range for its IP version. An AS the database does not hold,
+or a prefix with no live route, answers 404. The HTML pages give these errors as
+plain text.
 
 An origin AS of 0 (`origin_asn` in a route, `asn` in an IP lookup, the key `0`
 in a prefix's `origins`) means the route has no known origin: its AS path ends
@@ -76,7 +108,9 @@ routewatch/
 │   ├── logger/             # Structured logging wrapper
 │   ├── metrics/            # Performance metrics tracking
 │   ├── config/             # Configuration management
-│   └── templates/          # HTML templates
+│   ├── templates/          # HTML templates
+│   ├── version/            # Build version information
+│   └── whois/              # WHOIS lookups of AS details
 └── pkg/
     └── asinfo/             # AS information lookup (public API)
 ```
@@ -95,7 +129,7 @@ routewatch/
 │  │   Streamer   │───▶│   Handlers   │───▶│   Database   │       │
 │  │              │    │              │    │              │       │
 │  │ RIS Live     │    │ - ASHandler  │    │ SQLite with  │       │
-│  │ WebSocket    │    │ - PeerHandler│    │ WAL mode     │       │
+│  │ HTTP stream  │    │ - PeerHandler│    │ WAL mode     │       │
 │  │ client       │    │ - PrefixHdlr │    │              │       │
 │  │              │    │ - PeeringHdlr│    │ Tables:      │       │
 │  └──────────────┘    └──────────────┘    │ - asns       │       │
@@ -131,7 +165,10 @@ routewatch/
     - Parses BGP UPDATE messages from JSON stream
     - Dispatches to registered handlers based on message type
     - Each handler has its own queue with backpressure handling
-    - Handlers batch writes for efficiency (25K-30K ops, 1-2s timeout)
+    - Handlers batch writes for efficiency: the AS, peer and prefix handlers
+      write 30,000, 10,000 and 25,000 updates at a time, or after 2, 2 and 1
+      seconds; the peering handler writes the peerings it has collected every 30
+      seconds
 
 4. **Handler Details**
     - **ASHandler**: Tracks all ASNs seen in AS paths
@@ -150,7 +187,8 @@ routewatch/
 
 ### Key Design Patterns
 
-- **Batched Writes**: All database operations are batched for performance
+- **Batched Writes**: The handlers write the feed's updates to the database in
+  batches for performance
 - **Backpressure**: Probabilistic message dropping when queues exceed 50%
   capacity
 - **Graceful Shutdown**: 60-second timeout, flushes all pending batches
@@ -169,21 +207,35 @@ routewatch/
 
 ```sql
 -- Core tables
-asns(id, number, handle, description, first_seen, last_seen)
+asns(asn PRIMARY KEY, handle, description,
-prefixes_v4(prefix, first_seen)
+     as_name, org_name, org_id, address, country_code, abuse_email,
-prefixes_v6(prefix, first_seen)
+     abuse_phone, tech_email, tech_phone, rir, rir_registration_date,
+     rir_last_modified, whois_raw,
+     first_seen, last_seen, whois_updated_at)
+prefixes_v4(prefix PRIMARY KEY, first_seen) WITHOUT ROWID
+prefixes_v6(prefix PRIMARY KEY, first_seen) WITHOUT ROWID
 
--- Live routing tables (one per IP version)
+-- Live routing tables (one per IP version), one route per peer and prefix
-live_routes_v4(id, prefix, mask_length, origin_asn, peer_ip, as_path,
+live_routes_v4(id INTEGER PRIMARY KEY, prefix, mask_length, origin_asn,
-               next_hop, last_updated)
+               peer_ip, as_path, next_hop, last_updated,
-live_routes_v6(id, prefix, mask_length, origin_asn, peer_ip, as_path,
+               UNIQUE(peer_ip, prefix))
-               next_hop, last_updated)
+live_routes_v6(id INTEGER PRIMARY KEY, prefix, mask_length, origin_asn,
+               peer_ip, as_path, next_hop, last_updated,
+               UNIQUE(peer_ip, prefix))
 
 -- Relationship tracking
-peerings(id, as_a, as_b, first_seen, last_seen)
+peerings(id PRIMARY KEY, as_a, as_b, first_seen, last_seen,
-bgp_peers(id, peer_ip, peer_asn, last_message_type, last_seen)
+         UNIQUE(as_a, as_b))
+bgp_peers(id PRIMARY KEY, peer_ip UNIQUE, peer_asn, first_seen, last_seen,
+          last_message_type)
+
+-- Nothing in the daemon writes this table
+announcements(id PRIMARY KEY, prefix_id, peer_asn, origin_asn, path,
+              next_hop, timestamp, is_withdrawal)
 ```
 
+The indexes are in `internal/database/schema.sql`.
+
 ## Configuration
 
 Configuration is handled via environment variables and OS-specific paths:
@@ -201,7 +253,7 @@ non-zero exit. An empty variable counts as unset.
 
 State directory (database location):
 
-- macOS: `~/Library/Application Support/routewatch/`
+- macOS: `~/Library/Application Support/berlin.sneak.app.routewatch/`
 - Linux: `/var/lib/berlin.sneak.app.routewatch/` when running as root, otherwise
   `$XDG_DATA_HOME/berlin.sneak.app.routewatch/` (with `XDG_DATA_HOME` unset,
   `~/.local/share/berlin.sneak.app.routewatch/`). In the Docker image this is
@@ -301,7 +353,7 @@ make fmt
 make lint
 
 # Build
-make
+make build
 ```
 
 ## Entrypoints
@@ -330,8 +382,8 @@ provide:
 - `script/docker` — build the Docker image tagged via `script/projectname`
 - `script/cibuild` — CI entrypoint: runs `script/bootstrap` and `script/check`,
   then builds the image
-- `script/precommit` — pre-commit gate: `go mod tidy` + `go fmt` (must not
+- `script/precommit` — pre-commit gate: `go mod tidy` (must not change `go.mod`
-  change files), then `script/check`
+  or `go.sum`) and `go fmt` (writes), then `script/check`
 - `script/install-precommit` — install the git pre-commit hook that runs
   `script/precommit`
 

TODO.md

@@ -10,28 +10,91 @@
 
 # Status
 
-pre-1.0. No git tags. The linter and the tests are phases of the Docker build,
+pre-1.0. No git tags. `main` last took `next` on 2026-10-06, through
-and the Gitea workflow runs `script/cibuild` (the format check of the Go code
+https://git.eeqj.de/sneak/routewatch/pulls/6. The linter and the tests are
-and the markdown, both phases, then the image) on every push. The image sets
+phases of the Docker build, and the Gitea workflow runs `script/cibuild` (the
-memory ceilings for a 5 GiB container (README "Memory") and the README says how
+format check of the Go code and the markdown, both phases, then the image) on
-to run it under upaas (README "Running under upaas"). A 35-hour run of `3898daa`
+every push. The image sets memory ceilings for a 5 GiB container (README
-on the live feed peaked at about 1 GiB, without a container memory limit.
+"Memory") and the README says how to run it under upaas (README "Running under
+upaas"). A 25-hour run of `a9b6860` on the live feed peaked at about 0.6 GiB
+(https://git.eeqj.de/sneak/routewatch/issues/3). A run of `825afa2` on the live
+feed found the prefix handler dropping about a quarter of its updates because
+its route writes were slower than the feed
+(https://git.eeqj.de/sneak/routewatch/issues/96); the cheaper writes since then
+are measured by a benchmark, not yet on the live feed.
 
 # Next Step
 
-`next` waits for sneak to merge it to `main` through
+Run `next` on the live feed, as README "Running under upaas" describes, for the
-https://git.eeqj.de/sneak/routewatch/pulls/6. After that, setting routewatch up
+final check https://git.eeqj.de/sneak/routewatch/issues/96 names: whether the
-under upaas on fsn1app1 and deploying it are his
+prefix handler still drops updates after the first hour. If it does, a new issue
-(https://git.eeqj.de/sneak/routewatch/issues/31), and so is the run under a real
+follows with the numbers. Then an audit of the code at the commit that run
-5 GiB limit (https://git.eeqj.de/sneak/routewatch/issues/3).
+tested.
 
 # Completed Steps
 
+- 2026-10-19: README "HTTP Endpoints" lists every route with what it answers,
+  "Database Schema" matches `schema.sql`, and the rest of the README matches the
+  code; this file brought up to date (closes #72)
+- 2026-10-19: route writes are cheaper: the live route and prefix tables lose
+  their random UUID keys and the prefix tables their duplicate index, a route
+  update sets only the columns that change, and every write goes through one
+  connection with a page cache of 1 GiB (README "Memory"). A benchmark writes
+  the prefix handler's largest batch faster than the feed sends it (closes #96)
+- 2026-10-19: the WAL checkpoint, the incremental vacuum and ANALYZE take the
+  database lock every batch write holds, so a vacuum no longer fails with
+  "database is locked" and an ANALYZE no longer makes a batch write lose its
+  updates; ANALYZE runs as `PRAGMA optimize` with an analysis limit (closes #94)
+- 2026-10-19: shutdown stops the streamer first, and each handler writes
+  everything left on its queue before it stops, so the updates waiting there are
+  no longer lost (closes #69)
+- 2026-10-19: a path that ends in an AS set of several members has origin AS 0,
+  no known origin, instead of the set's last member, and the set adds no
+  peerings; the pages show such a route as having no known origin (closes #75)
+- 2026-10-19: `bgp_peer_count` drops to 0 when the last peer goes down (closes
+  #92)
+- 2026-10-19: `bgp_peer_count` counts every peer the feed has heard from, not
+  only those whose session opened since startup, and drops a peer whose session
+  goes down (closes #74)
 - 2026-10-19: the shared files are re-vendored from `sneak/prompts` at
   `0653579`: golangci-lint v2.14.0 and its configuration, lint and test as
   `Dockerfile` phases that keep Go's and golangci-lint's caches, markdown
   formatted with prettier, agent guidance in `AGENTS.md`; the code is fixed for
   what the new linter configuration finds (closes #52)
+- 2026-10-19: the streamer reads feed lines up to 1 MiB, so a line over 64 KiB
+  no longer ends the connection (closes #76)
+- 2026-10-19: a peer's live routes are removed when its BGP session goes down;
+  `STATE` messages are never dropped for backpressure, and one that finds its
+  queue full waits for room ahead of the handler's other messages (closes #56)
+- 2026-10-19: the prefix-length pages pick their sample with a bounded number of
+  index lookups instead of reading every route of that length, and list each
+  prefix once (closes #66)
+- 2026-10-19: a request past its time limit answers 504 instead of 408, and
+  `/api/v1/stats` answers 504 instead of 500 when its read runs out of time
+  (closes #70)
+- 2026-10-19: the AS, peer and prefix handlers write each batch outside the lock
+  their message handling needs, so messages keep coming off their queues during
+  a write; meanwhile the next batch grows to at most ten batch sizes (closes
+  #65)
+- 2026-10-19: a live route is identified by prefix and peer, so an announcement
+  with a new origin AS replaces the peer's route and a withdrawal removes it
+  whatever its origin AS (closes #55)
+- 2026-10-14: the streamer reconnects when RIS Live ends the stream or sends
+  nothing for 60 seconds, and the health check reports the feed disconnected
+  meanwhile (closes #53)
+- 2026-10-14: the prefix page takes the names of the origin ASes from the prefix
+  query instead of reading every route of each origin AS (closes #67)
+- 2026-10-14: the daemon exits with status 1 when its web server cannot listen
+  on its port, instead of running on without it (closes #73)
+- 2026-10-14: the prefix pages parse the prefix before looking it up: one that
+  does not parse answers 400, and any valid way of writing a prefix finds its
+  routes (closes #71)
+- 2026-10-14: the IP lookup ignores routes shorter than /8 for IPv4 and /16 for
+  IPv6, so a default route is no longer named as an address's owner (closes #57)
+- 2026-10-14: a prefix batch keeps each peer's last update for a prefix, instead
+  of one update per prefix across all peers (closes #54)
+- 2026-10-14: `reconnect_count` in `/api/v1/stats` counts every reconnect
+  (closes #68)
 - 2026-10-03: `/api/v1/stats` serves the prefix distribution from memory, seeded
   at startup and adjusted on every live-route write, so a request no longer
   reads every live route (closes #30)
@@ -116,6 +179,4 @@ under upaas on fsn1app1 and deploying it are his
 
 # Future Steps
 
-- Production memory under 5 GiB: whether to test under a real 5 GiB container
+None open.
-  limit on fsn1app1 is open for sneak
-  (https://git.eeqj.de/sneak/routewatch/issues/3)