Question for sneak: which BGP peers should routewatch keep routes from?
routewatch keeps one route per prefix for every peer of every RIS collector, so its route tables never stop growing on the live feed. Measured on next (d84dd56) running as README "Running under upaas" describes:
After 6.5 hours: 30.1 million routes in a 10.8 GB database, still growing 4 to 5 million an hour; memory steady at 1.3 to 1.6 GB.
582 IPv4 peers have routes so far, only 4 of them a full table: routes are learned as the feed updates them, so each peer's table fills over days.
RIPEstat lists 1,427 RIS peers, 330 with full IPv4 tables; together they carry about 460 million routes, roughly 165 GB at today's size per route.
Writes keep up below about 15 million routes (no updates dropped in the first 3 hours); past 20 million the prefix handler drops again, 6 to 7 percent of its updates in the last half hour, and lost withdrawals leave routes in the table.
Options:
Chosen peers: a setting lists the peer addresses whose routes are kept, by default about ten full-table peers spread over several collectors. The tables stop at roughly 1.2 million routes per peer (about 12 million, under 5 GB). The AS and peering pages keep using every peer. IP lookups and prefix pages show those peers' view.
Chosen collectors: a setting lists RIS collectors instead. rrc00 alone has 109 peers carrying about 56 million routes (about 20 GB), beyond what writes keep up with today.
All peers: as now. Needs about 165 GB of disk and a different way of storing routes before writes keep up; a lot of work.
Recommendation: chosen peers. Until you answer, the other fixes continue.
Model: opus-5-5
**Question for sneak:** which BGP peers should routewatch keep routes from?
routewatch keeps one route per prefix for every peer of every RIS collector, so its route tables never stop growing on the live feed. Measured on `next` (`d84dd56`) running as README "Running under upaas" describes:
- After 6.5 hours: 30.1 million routes in a 10.8 GB database, still growing 4 to 5 million an hour; memory steady at 1.3 to 1.6 GB.
- 582 IPv4 peers have routes so far, only 4 of them a full table: routes are learned as the feed updates them, so each peer's table fills over days.
- RIPEstat lists 1,427 RIS peers, 330 with full IPv4 tables; together they carry about 460 million routes, roughly 165 GB at today's size per route.
- Writes keep up below about 15 million routes (no updates dropped in the first 3 hours); past 20 million the prefix handler drops again, 6 to 7 percent of its updates in the last half hour, and lost withdrawals leave routes in the table.
Options:
- **Chosen peers**: a setting lists the peer addresses whose routes are kept, by default about ten full-table peers spread over several collectors. The tables stop at roughly 1.2 million routes per peer (about 12 million, under 5 GB). The AS and peering pages keep using every peer. IP lookups and prefix pages show those peers' view.
- **Chosen collectors**: a setting lists RIS collectors instead. `rrc00` alone has 109 peers carrying about 56 million routes (about 20 GB), beyond what writes keep up with today.
- **All peers**: as now. Needs about 165 GB of disk and a different way of storing routes before writes keep up; a lot of work.
Recommendation: chosen peers. Until you answer, the other fixes continue.
Model: opus-5-5
sneak
was assigned by clawbot2026-10-19 22:40:42 +02:00
The same run after 12 hours: 50.4 million routes in 18.0 GB, still growing about 2 million an hour overnight (4 to 5 million an hour in the evening). The prefix handler lost 43 million of its 298 million updates, almost all of them after the tables passed 20 million routes; memory stayed between 1.3 and 1.9 GB.
Model: opus-5-5
The same run after 12 hours: 50.4 million routes in 18.0 GB, still growing about 2 million an hour overnight (4 to 5 million an hour in the evening). The prefix handler lost 43 million of its 298 million updates, almost all of them after the tables passed 20 million routes; memory stayed between 1.3 and 1.9 GB.
Model: opus-5-5
One more fact for the decision: the feed sends no starting table and routewatch loads none, so the tables hold only routes announced since it first started. An address whose route has not changed since then gets 404 from the IP lookup, whichever option you pick; with chosen peers, a full table for each would take days to arrive this way. Loading those peers' tables from RIS's published table dumps at start would fill them at once, but that is a new feature and not part of any option above.
Model: opus-5-5
One more fact for the decision: the feed sends no starting table and routewatch loads none, so the tables hold only routes announced since it first started. An address whose route has not changed since then gets 404 from the IP lookup, whichever option you pick; with chosen peers, a full table for each would take days to arrive this way. Loading those peers' tables from RIS's published table dumps at start would fill them at once, but that is a new feature and not part of any option above.
Model: opus-5-5
It needs to keep a full routing table in memory at all times, so there needs to be a new table in the db that keeps the live whole-internet routing table up to date at all times and is read on startup.
Is there a way we can prune the oldest routes, but only redundant ones (networks we have multiple newer routes for) to keep the table size manageable and not unbounded? Can you conceive a pruning algorithm that doesn’t lose fidelity for the info APIs?
It needs to keep a full routing table in memory at all times, so there needs to be a new table in the db that keeps the live whole-internet routing table up to date at all times and is read on startup.
Is there a way we can prune the oldest routes, but only redundant ones (networks we have multiple newer routes for) to keep the table size manageable and not unbounded? Can you conceive a pruning algorithm that doesn’t lose fidelity for the info APIs?
Recorded: routewatch keeps the whole routing table at all times, in a new database table kept current from the live feed and read at startup. The answer to your question, a pruning rule that drops only redundant old routes without losing anything the info APIs answer, comes here as a plain proposal for you to confirm before anything is built. routewatch gets its manager back for it as soon as an account has room; both worker accounts are at their week limit now (sneak/project-management#37).
Model: opus-5-5
Recorded: routewatch keeps the whole routing table at all times, in a new database table kept current from the live feed and read at startup. The answer to your question, a pruning rule that drops only redundant old routes without losing anything the info APIs answer, comes here as a plain proposal for you to confirm before anything is built. routewatch gets its manager back for it as soon as an account has room; both worker accounts are at their week limit now (https://git.eeqj.de/sneak/project-management/issues/37).
Model: opus-5-5
sneak
was unassigned by clawbot2026-10-21 06:09:10 +02:00
clawbot
self-assigned this 2026-10-21 06:09:10 +02:00
Proposal for you to confirm: pruning the routing table
Yes. routewatch can drop old routes and still answer the same: which prefix an address is in, which AS originates it, how many peers carry it, and which prefixes an AS originates. The one thing it cannot keep is every peer's own AS path; the prefix page loses that for older routes. Nothing is built until you confirm.
Which routes are redundant. A route is one peer's route to one prefix. A route is redundant when its prefix has at least two newer routes with the same origin AS. Counting only routes with the same origin means a prefix announced by two ASes (a hijack, or a network announced from two places) keeps both origins' routes, however old one side's routes are. The rule is applied on every update, so there is no separate clean-up run.
What is kept, for each prefix and origin AS:
the two newest routes in full: peer, AS path, next hop, time;
the list of every peer whose current route it is, one bit per peer (about 180 bytes for a prefix all 1,427 RIS peers carry).
A pruned route loses its AS path, next hop and time; its peer stays on the list until it withdraws the prefix, announces it with another origin, or its session goes down.
Why the answers stay the same. The peer list keeps them exact. Without it, when the two peers whose routes were kept withdraw a prefix, the prefix would vanish while hundreds of other peers still carry it, and IP lookups would fall back to a less specific prefix or answer 404. With it:
IP lookup: the most specific prefix any peer carries, its peer count and, for a prefix with one origin, its origin AS are exact.
AS page and API: the prefixes the AS originates, each with its newest route's time, are exact: each origin's newest route is always kept.
Prefix page and API: every peer that carries the prefix is listed with its origin, and the origins are exact.
Prefix-length pages, the prefix counts per mask length, and the route count on the status page are exact.
What changes:
The prefix page shows AS path, next hop and time only for each origin's two newest routes; the other peers are listed without them.
For a prefix with more than one origin, the IP lookup gives the origin most peers carry; today it gives the origin of whichever route is oldest. Its last_seen becomes the newest route's time instead of the oldest's.
When both kept routes of an origin are withdrawn while other peers still carry it, that origin shows its peers but no path or time until one of them sends an update. The live run will count how often this happens.
The status page's oldest-route time becomes the oldest kept route's.
Size on the live feed. A full table is about 1.1 million IPv4 and 0.26 million IPv6 prefixes (RIPEstat, today). At most two routes and one peer list per prefix and origin comes to about 1.4 million rows and at most 2.8 million routes. Today's tables held 50.4 million routes in 18.0 GB after 12 hours, on the way to 460 million (about 165 GB) once every peer's table arrives. Estimates once the table is full: about 0.5 GB on disk and 0.7 GB of memory for the table. With routes read from memory, SQLite's 1 GiB page cache can shrink, so the process should stay around 2 GB (1.3 to 1.9 GB measured so far), inside the 5 GiB container, and stop growing; the image's memory limits are reset to fit. Writes stay at about 1.4 million rows, far below the 20 million where the prefix handler started losing updates.
At startup and while running. The whole table lives in memory, and every route lookup reads it there. The database keeps a copy in one new table, one row per prefix and origin, which replaces today's per-peer route tables. At startup routewatch reads that table into memory before it connects to the feed; 1.4 million rows should take seconds, to be measured against upaas's 60-second health check. Each update from the feed changes the memory copy at once, and the rows it changed are written to the database every second, as the handlers batch writes now. A stop writes the last batch; a crash loses only the changes not yet written.
Withdrawals. A withdrawal removes the peer from its prefix's list and drops its route if it was one of the two kept; an announcement whose AS path is empty or holds AS 0 counts as one, as now. A peer's session going down removes that peer from every list. A prefix leaves the table only when no peer carries it any more. Withdrawals sent while routewatch is stopped are missed, as today: those routes stay until their peer sends another update for the prefix or its session goes down.
Not included: on a first start with an empty database the table still fills only from the feed's updates, which takes days. Filling it at first start from RIS's published table dumps would be a separate feature; say if you want it.
Confirm, or say what to change, and I build it in small reviewed PRs on next, then run it on the live feed until the table stops growing and post the figures here.
Model: opus-5-5
**Proposal for you to confirm: pruning the routing table**
Yes. routewatch can drop old routes and still answer the same: which prefix an address is in, which AS originates it, how many peers carry it, and which prefixes an AS originates. The one thing it cannot keep is every peer's own AS path; the prefix page loses that for older routes. Nothing is built until you confirm.
**Which routes are redundant.** A route is one peer's route to one prefix. A route is redundant when its prefix has at least two newer routes with the same origin AS. Counting only routes with the same origin means a prefix announced by two ASes (a hijack, or a network announced from two places) keeps both origins' routes, however old one side's routes are. The rule is applied on every update, so there is no separate clean-up run.
**What is kept**, for each prefix and origin AS:
- the two newest routes in full: peer, AS path, next hop, time;
- the list of every peer whose current route it is, one bit per peer (about 180 bytes for a prefix all 1,427 RIS peers carry).
A pruned route loses its AS path, next hop and time; its peer stays on the list until it withdraws the prefix, announces it with another origin, or its session goes down.
**Why the answers stay the same.** The peer list keeps them exact. Without it, when the two peers whose routes were kept withdraw a prefix, the prefix would vanish while hundreds of other peers still carry it, and IP lookups would fall back to a less specific prefix or answer 404. With it:
- IP lookup: the most specific prefix any peer carries, its peer count and, for a prefix with one origin, its origin AS are exact.
- AS page and API: the prefixes the AS originates, each with its newest route's time, are exact: each origin's newest route is always kept.
- Prefix page and API: every peer that carries the prefix is listed with its origin, and the origins are exact.
- Prefix-length pages, the prefix counts per mask length, and the route count on the status page are exact.
What changes:
- The prefix page shows AS path, next hop and time only for each origin's two newest routes; the other peers are listed without them.
- For a prefix with more than one origin, the IP lookup gives the origin most peers carry; today it gives the origin of whichever route is oldest. Its `last_seen` becomes the newest route's time instead of the oldest's.
- When both kept routes of an origin are withdrawn while other peers still carry it, that origin shows its peers but no path or time until one of them sends an update. The live run will count how often this happens.
- The status page's oldest-route time becomes the oldest kept route's.
**Size on the live feed.** A full table is about 1.1 million IPv4 and 0.26 million IPv6 prefixes (RIPEstat, today). At most two routes and one peer list per prefix and origin comes to about 1.4 million rows and at most 2.8 million routes. Today's tables held 50.4 million routes in 18.0 GB after 12 hours, on the way to 460 million (about 165 GB) once every peer's table arrives. Estimates once the table is full: about 0.5 GB on disk and 0.7 GB of memory for the table. With routes read from memory, SQLite's 1 GiB page cache can shrink, so the process should stay around 2 GB (1.3 to 1.9 GB measured so far), inside the 5 GiB container, and stop growing; the image's memory limits are reset to fit. Writes stay at about 1.4 million rows, far below the 20 million where the prefix handler started losing updates.
**At startup and while running.** The whole table lives in memory, and every route lookup reads it there. The database keeps a copy in one new table, one row per prefix and origin, which replaces today's per-peer route tables. At startup routewatch reads that table into memory before it connects to the feed; 1.4 million rows should take seconds, to be measured against upaas's 60-second health check. Each update from the feed changes the memory copy at once, and the rows it changed are written to the database every second, as the handlers batch writes now. A stop writes the last batch; a crash loses only the changes not yet written.
**Withdrawals.** A withdrawal removes the peer from its prefix's list and drops its route if it was one of the two kept; an announcement whose AS path is empty or holds AS 0 counts as one, as now. A peer's session going down removes that peer from every list. A prefix leaves the table only when no peer carries it any more. Withdrawals sent while routewatch is stopped are missed, as today: those routes stay until their peer sends another update for the prefix or its session goes down.
**Not included:** on a first start with an empty database the table still fills only from the feed's updates, which takes days. Filling it at first start from RIS's published table dumps would be a separate feature; say if you want it.
Confirm, or say what to change, and I build it in small reviewed PRs on `next`, then run it on the live feed until the table stops growing and post the figures here.
Model: opus-5-5
clawbot
removed their assignment 2026-10-21 12:14:58 +02:00
sneak
was assigned by clawbot2026-10-21 12:14:58 +02:00
clawbot
changed title from Which peers should routewatch keep routes from? Keeping every RIS peer, the route tables never stop growing to Keep the whole routing table in memory and in one database table, pruning only redundant routes so it stays bounded2026-10-21 12:15:12 +02:00
Recorded: routewatch builds the pruning rule exactly as proposed above, in small reviewed changes on next, then a run against the live feed to see its steady state. Its manager starts on it when an account has room: all fleet work is paused at your 60% guard on the primary, and the first worker account resets at 23:00 UTC on 25 October.
Model: opus-5-5
Recorded: routewatch builds the pruning rule exactly as proposed above, in small reviewed changes on `next`, then a run against the live feed to see its steady state. Its manager starts on it when an account has room: all fleet work is paused at your 60% guard on the primary, and the first worker account resets at 23:00 UTC on 25 October.
Model: opus-5-5
sneak
was unassigned by clawbot2026-10-21 23:22:28 +02:00
clawbot
self-assigned this 2026-10-21 23:22:28 +02:00
Implementation plan for the rule confirmed above, in five PRs into next, each reviewed before it merges. next keeps working after each one.
The table in memory. A new package, internal/routetable, holds the whole routing table. For each prefix and origin AS it keeps the set of peers that carry it (one bit per peer) and its two newest routes (peer, AS path, time). The next hop is the peer, as today, so it is not stored. It applies an announcement, a withdrawal and a peer's session going down exactly as the proposal says, and records which prefix and origin pairs changed. It answers the reads the pages need: the most specific prefix for an address, the prefixes of an AS, one prefix's origins and peers, a sample of prefixes of one mask length, and the counts the stats show. Nothing uses it yet.
The table in the database. In internal/database, a new routes table with one row per prefix and origin AS: the peer set, and the two newest routes. Peers get a small fixed number, kept in a table of their own, so the peer sets stay valid across restarts. Two functions: one reads every row at startup, one writes the changed rows in one transaction. Nothing uses them yet.
Writes. The prefix handler applies each update to the table in memory at once, and every second writes the rows that changed. At startup routewatch reads the table before it connects to the feed, and logs how many rows it read and how long that took. The old per-peer route tables are still written in this step, so the pages keep working.
Reads. The IP lookup, the AS page and API, the prefix page and API, the prefix-length pages and the stats read the table in memory, with the changed answers the proposal lists. The stats also report how many origins have peers but no kept route, which the live run counts.
Clean-up. The per-peer route tables and the code that writes and counts them are removed. The memory settings in the image and README "Memory" are set for the new table, and the README describes the new answers and the routes table.
Then the run on the live feed until the table stops growing, with the figures posted here.
Model: opus-5-5
**Implementation plan** for the rule confirmed above, in five PRs into `next`, each reviewed before it merges. `next` keeps working after each one.
1. **The table in memory.** A new package, `internal/routetable`, holds the whole routing table. For each prefix and origin AS it keeps the set of peers that carry it (one bit per peer) and its two newest routes (peer, AS path, time). The next hop is the peer, as today, so it is not stored. It applies an announcement, a withdrawal and a peer's session going down exactly as the proposal says, and records which prefix and origin pairs changed. It answers the reads the pages need: the most specific prefix for an address, the prefixes of an AS, one prefix's origins and peers, a sample of prefixes of one mask length, and the counts the stats show. Nothing uses it yet.
2. **The table in the database.** In `internal/database`, a new `routes` table with one row per prefix and origin AS: the peer set, and the two newest routes. Peers get a small fixed number, kept in a table of their own, so the peer sets stay valid across restarts. Two functions: one reads every row at startup, one writes the changed rows in one transaction. Nothing uses them yet.
3. **Writes.** The prefix handler applies each update to the table in memory at once, and every second writes the rows that changed. At startup routewatch reads the table before it connects to the feed, and logs how many rows it read and how long that took. The old per-peer route tables are still written in this step, so the pages keep working.
4. **Reads.** The IP lookup, the AS page and API, the prefix page and API, the prefix-length pages and the stats read the table in memory, with the changed answers the proposal lists. The stats also report how many origins have peers but no kept route, which the live run counts.
5. **Clean-up.** The per-peer route tables and the code that writes and counts them are removed. The memory settings in the image and README "Memory" are set for the new table, and the README describes the new answers and the `routes` table.
Then the run on the live feed until the table stops growing, with the figures posted here.
Model: opus-5-5
Step 1, the routing table in memory, is up for review in #185. Nothing uses it yet.
One figure for the later steps: the table takes about 700 bytes per prefix and origin pair, so a full table needs about 1 GB of memory, not the 0.7 GB the proposal estimated.
Model: opus-5-5
Step 1, the routing table in memory, is up for review in https://git.eeqj.de/sneak/routewatch/pulls/185. Nothing uses it yet.
One figure for the later steps: the table takes about 700 bytes per prefix and origin pair, so a full table needs about 1 GB of memory, not the 0.7 GB the proposal estimated.
Model: opus-5-5
Step 2, the routing table in the database, is up for review in #186. Nothing uses it yet.
Two figures for the later steps: reading a full table at startup should take about 25 seconds, inside the 60-second health check, and the database file needs about 450 MB for it.
Model: opus-5-5
Step 2, the routing table in the database, is up for review in https://git.eeqj.de/sneak/routewatch/pulls/186. Nothing uses it yet.
Two figures for the later steps: reading a full table at startup should take about 25 seconds, inside the 60-second health check, and the database file needs about 450 MB for it.
Model: opus-5-5
Step 3, the routing table in memory kept current from the feed and written every second, is up for review in #187. The pages still read the per-peer route tables.
One finding for the deploys: with the image's old 10-second health check start period, Docker's first check after it came 40 seconds after the start, not 60. The start period is now 60 seconds, so a start that reads a full table, about 20 to 25 seconds, is healthy before upaas looks.
Model: opus-5-5
Step 3, the routing table in memory kept current from the feed and written every second, is up for review in https://git.eeqj.de/sneak/routewatch/pulls/187. The pages still read the per-peer route tables.
One finding for the deploys: with the image's old 10-second health check start period, Docker's first check after it came 40 seconds after the start, not 60. The start period is now 60 seconds, so a start that reads a full table, about 20 to 25 seconds, is healthy before upaas looks.
Model: opus-5-5
Step 4, the pages, the API and the stats reading every route from the routing table in memory, is up for review in #194. The per-peer route tables are still written; step 5 removes them.
Model: opus-5-5
Step 4, the pages, the API and the stats reading every route from the routing table in memory, is up for review in https://git.eeqj.de/sneak/routewatch/pulls/194. The per-peer route tables are still written; step 5 removes them.
Model: opus-5-5
Step 5, the last, is up for review in #195. The per-peer route tables and the code that wrote, counted and read them are gone. The image's memory limits are set for a full routing table of about 1 GB, inside the 5 GiB container. After it merges, the run on the live feed posts its figures here.
Model: opus-5-5
Step 5, the last, is up for review in https://git.eeqj.de/sneak/routewatch/pulls/195. The per-peer route tables and the code that wrote, counted and read them are gone. The image's memory limits are set for a full routing table of about 1 GB, inside the 5 GiB container. After it merges, the run on the live feed posts its figures here.
Model: opus-5-5
The run on the live feed has started on 1c65656, the merge that closed this issue, with an empty database, as README "Running under upaas" describes. It samples memory, the table's size, the database size and every handler's dropped updates every 10 minutes. The figures go here once the table stops growing.
Model: opus-5-5
The run on the live feed has started on `1c65656`, the merge that closed this issue, with an empty database, as README "Running under upaas" describes. It samples memory, the table's size, the database size and every handler's dropped updates every 10 minutes. The figures go here once the table stops growing.
Model: opus-5-5
Figures from the run on the live feed (1c65656, empty database at start, about 4.5 hours so far):
The table reached its full size in about 45 minutes: 1.12 million IPv4 and 0.28 million IPv6 prefixes, 1.41 million prefix and origin pairs. It has stopped growing.
Memory held at 2.55 to 2.66 GB for the last 3.5 hours, inside the 5 GiB container; the Go heap moved between 0.9 and 1.7 GB.
The database holds 564 MB.
No handler dropped an update in the whole run.
Peer and prefix pairs keep arriving as more peers send updates (23.5 million now), without growing memory.
Origins carried by peers but with no kept route: about 2,000 most of the time. One burst of withdrawals raised it to 24,000 (1.7 percent of the table), back to 2,300 within an hour.
A restart read the whole table in 12.4 seconds, and the container was healthy 17 seconds after the start; a stop took under a second and lost nothing.
The run continues overnight to confirm memory stays flat as the peer and prefix pairs keep growing.
Model: opus-5-5
Figures from the run on the live feed (`1c65656`, empty database at start, about 4.5 hours so far):
- The table reached its full size in about 45 minutes: 1.12 million IPv4 and 0.28 million IPv6 prefixes, 1.41 million prefix and origin pairs. It has stopped growing.
- Memory held at 2.55 to 2.66 GB for the last 3.5 hours, inside the 5 GiB container; the Go heap moved between 0.9 and 1.7 GB.
- The database holds 564 MB.
- No handler dropped an update in the whole run.
- Peer and prefix pairs keep arriving as more peers send updates (23.5 million now), without growing memory.
- Origins carried by peers but with no kept route: about 2,000 most of the time. One burst of withdrawals raised it to 24,000 (1.7 percent of the table), back to 2,300 within an hour.
- A restart read the whole table in 12.4 seconds, and the container was healthy 17 seconds after the start; a stop took under a second and lost nothing.
The run continues overnight to confirm memory stays flat as the peer and prefix pairs keep growing.
Model: opus-5-5
The run of 1c65656 on the live feed ended after five days (27 October to 1 November).
Memory reached 3.1 GB in the first 15 hours and stayed there for four days, inside the 5 GiB container. The database reached 860 MB.
Prefix and origin pairs whose kept routes were all withdrawn rose from about 2,000 to 19,000, 1.3 percent of the table.
It found three defects: #216 and #217 (the prefix and AS handlers drop updates in bursts) and #218 (the WHOIS fetcher keeps asking registries that refuse it).
Model: opus-5-5
The run of `1c65656` on the live feed ended after five days (27 October to 1 November).
- Memory reached 3.1 GB in the first 15 hours and stayed there for four days, inside the 5 GiB container. The database reached 860 MB.
- Prefix and origin pairs whose kept routes were all withdrawn rose from about 2,000 to 19,000, 1.3 percent of the table.
- It found three defects: https://git.eeqj.de/sneak/routewatch/issues/216 and https://git.eeqj.de/sneak/routewatch/issues/217 (the prefix and AS handlers drop updates in bursts) and https://git.eeqj.de/sneak/routewatch/issues/218 (the WHOIS fetcher keeps asking registries that refuse it).
Model: opus-5-5
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Question for sneak: which BGP peers should routewatch keep routes from?
routewatch keeps one route per prefix for every peer of every RIS collector, so its route tables never stop growing on the live feed. Measured on
next(d84dd56) running as README "Running under upaas" describes:Options:
rrc00alone has 109 peers carrying about 56 million routes (about 20 GB), beyond what writes keep up with today.Recommendation: chosen peers. Until you answer, the other fixes continue.
Model: opus-5-5
The same run after 12 hours: 50.4 million routes in 18.0 GB, still growing about 2 million an hour overnight (4 to 5 million an hour in the evening). The prefix handler lost 43 million of its 298 million updates, almost all of them after the tables passed 20 million routes; memory stayed between 1.3 and 1.9 GB.
Model: opus-5-5
One more fact for the decision: the feed sends no starting table and routewatch loads none, so the tables hold only routes announced since it first started. An address whose route has not changed since then gets 404 from the IP lookup, whichever option you pick; with chosen peers, a full table for each would take days to arrive this way. Loading those peers' tables from RIS's published table dumps at start would fill them at once, but that is a new feature and not part of any option above.
Model: opus-5-5
It needs to keep a full routing table in memory at all times, so there needs to be a new table in the db that keeps the live whole-internet routing table up to date at all times and is read on startup.
Is there a way we can prune the oldest routes, but only redundant ones (networks we have multiple newer routes for) to keep the table size manageable and not unbounded? Can you conceive a pruning algorithm that doesn’t lose fidelity for the info APIs?
Recorded: routewatch keeps the whole routing table at all times, in a new database table kept current from the live feed and read at startup. The answer to your question, a pruning rule that drops only redundant old routes without losing anything the info APIs answer, comes here as a plain proposal for you to confirm before anything is built. routewatch gets its manager back for it as soon as an account has room; both worker accounts are at their week limit now (sneak/project-management#37).
Model: opus-5-5
Proposal for you to confirm: pruning the routing table
Yes. routewatch can drop old routes and still answer the same: which prefix an address is in, which AS originates it, how many peers carry it, and which prefixes an AS originates. The one thing it cannot keep is every peer's own AS path; the prefix page loses that for older routes. Nothing is built until you confirm.
Which routes are redundant. A route is one peer's route to one prefix. A route is redundant when its prefix has at least two newer routes with the same origin AS. Counting only routes with the same origin means a prefix announced by two ASes (a hijack, or a network announced from two places) keeps both origins' routes, however old one side's routes are. The rule is applied on every update, so there is no separate clean-up run.
What is kept, for each prefix and origin AS:
A pruned route loses its AS path, next hop and time; its peer stays on the list until it withdraws the prefix, announces it with another origin, or its session goes down.
Why the answers stay the same. The peer list keeps them exact. Without it, when the two peers whose routes were kept withdraw a prefix, the prefix would vanish while hundreds of other peers still carry it, and IP lookups would fall back to a less specific prefix or answer 404. With it:
What changes:
last_seenbecomes the newest route's time instead of the oldest's.Size on the live feed. A full table is about 1.1 million IPv4 and 0.26 million IPv6 prefixes (RIPEstat, today). At most two routes and one peer list per prefix and origin comes to about 1.4 million rows and at most 2.8 million routes. Today's tables held 50.4 million routes in 18.0 GB after 12 hours, on the way to 460 million (about 165 GB) once every peer's table arrives. Estimates once the table is full: about 0.5 GB on disk and 0.7 GB of memory for the table. With routes read from memory, SQLite's 1 GiB page cache can shrink, so the process should stay around 2 GB (1.3 to 1.9 GB measured so far), inside the 5 GiB container, and stop growing; the image's memory limits are reset to fit. Writes stay at about 1.4 million rows, far below the 20 million where the prefix handler started losing updates.
At startup and while running. The whole table lives in memory, and every route lookup reads it there. The database keeps a copy in one new table, one row per prefix and origin, which replaces today's per-peer route tables. At startup routewatch reads that table into memory before it connects to the feed; 1.4 million rows should take seconds, to be measured against upaas's 60-second health check. Each update from the feed changes the memory copy at once, and the rows it changed are written to the database every second, as the handlers batch writes now. A stop writes the last batch; a crash loses only the changes not yet written.
Withdrawals. A withdrawal removes the peer from its prefix's list and drops its route if it was one of the two kept; an announcement whose AS path is empty or holds AS 0 counts as one, as now. A peer's session going down removes that peer from every list. A prefix leaves the table only when no peer carries it any more. Withdrawals sent while routewatch is stopped are missed, as today: those routes stay until their peer sends another update for the prefix or its session goes down.
Not included: on a first start with an empty database the table still fills only from the feed's updates, which takes days. Filling it at first start from RIS's published table dumps would be a separate feature; say if you want it.
Confirm, or say what to change, and I build it in small reviewed PRs on
next, then run it on the live feed until the table stops growing and post the figures here.Model: opus-5-5
Which peers should routewatch keep routes from? Keeping every RIS peer, the route tables never stop growingto Keep the whole routing table in memory and in one database table, pruning only redundant routes so it stays boundedproceed with the plan.
Recorded: routewatch builds the pruning rule exactly as proposed above, in small reviewed changes on
next, then a run against the live feed to see its steady state. Its manager starts on it when an account has room: all fleet work is paused at your 60% guard on the primary, and the first worker account resets at 23:00 UTC on 25 October.Model: opus-5-5
Implementation plan for the rule confirmed above, in five PRs into
next, each reviewed before it merges.nextkeeps working after each one.internal/routetable, holds the whole routing table. For each prefix and origin AS it keeps the set of peers that carry it (one bit per peer) and its two newest routes (peer, AS path, time). The next hop is the peer, as today, so it is not stored. It applies an announcement, a withdrawal and a peer's session going down exactly as the proposal says, and records which prefix and origin pairs changed. It answers the reads the pages need: the most specific prefix for an address, the prefixes of an AS, one prefix's origins and peers, a sample of prefixes of one mask length, and the counts the stats show. Nothing uses it yet.internal/database, a newroutestable with one row per prefix and origin AS: the peer set, and the two newest routes. Peers get a small fixed number, kept in a table of their own, so the peer sets stay valid across restarts. Two functions: one reads every row at startup, one writes the changed rows in one transaction. Nothing uses them yet.routestable.Then the run on the live feed until the table stops growing, with the figures posted here.
Model: opus-5-5
Step 1, the routing table in memory, is up for review in #185. Nothing uses it yet.
One figure for the later steps: the table takes about 700 bytes per prefix and origin pair, so a full table needs about 1 GB of memory, not the 0.7 GB the proposal estimated.
Model: opus-5-5
Step 2, the routing table in the database, is up for review in #186. Nothing uses it yet.
Two figures for the later steps: reading a full table at startup should take about 25 seconds, inside the 60-second health check, and the database file needs about 450 MB for it.
Model: opus-5-5
Step 3, the routing table in memory kept current from the feed and written every second, is up for review in #187. The pages still read the per-peer route tables.
One finding for the deploys: with the image's old 10-second health check start period, Docker's first check after it came 40 seconds after the start, not 60. The start period is now 60 seconds, so a start that reads a full table, about 20 to 25 seconds, is healthy before upaas looks.
Model: opus-5-5
Step 4, the pages, the API and the stats reading every route from the routing table in memory, is up for review in #194. The per-peer route tables are still written; step 5 removes them.
Model: opus-5-5
Step 5, the last, is up for review in #195. The per-peer route tables and the code that wrote, counted and read them are gone. The image's memory limits are set for a full routing table of about 1 GB, inside the 5 GiB container. After it merges, the run on the live feed posts its figures here.
Model: opus-5-5
The run on the live feed has started on
1c65656, the merge that closed this issue, with an empty database, as README "Running under upaas" describes. It samples memory, the table's size, the database size and every handler's dropped updates every 10 minutes. The figures go here once the table stops growing.Model: opus-5-5
Figures from the run on the live feed (
1c65656, empty database at start, about 4.5 hours so far):The run continues overnight to confirm memory stays flat as the peer and prefix pairs keep growing.
Model: opus-5-5
The run of
1c65656on the live feed ended after five days (27 October to 1 November).Model: opus-5-5