On a five-day run of 1c65656 on the live feed (27 October to 1 November), the prefix handler dropped 549,667 of 2.56 billion messages, in bursts. A dropped withdrawal leaves a route in the table that no peer carries.
Cause.Table.PeerDown in internal/routetable/routetable.go visits all 1.55 million prefixes while it holds the table's write lock. The prefix handler calls it in internal/routewatch/prefixhandler.go for every session that goes down, and takes no message off its queue meanwhile.
The feed reported 91,578 sessions going down in the five days, often several within seconds.
The prefix handler's longest single message took 1.47 seconds. Its queue holds 20,000 messages, about 2.5 seconds of feed.
At 06:56 UTC on 28 October, five sessions went down within 12 seconds and the handler dropped 11,093 updates. At 15:39 UTC, four went down within 10 seconds and it dropped 14,838.
While the lock is held, IP lookups and the prefix pages wait too.
Recommended fix.
PeerDown releases the lock every 10,000 prefixes and takes it again. Go allows a map to change between the steps of a range over it.
The prefix handler runs PeerDown on a goroutine of its own, one peer at a time, so it keeps taking messages meanwhile.
The handler holds back the down peer's later messages until its PeerDown returns, then applies them in order. A route that peer announces after its session went down must survive.
Definition of done.
A session going down no longer stops the prefix handler, and no reader waits on the table's lock for the whole visit.
A route announced by the same peer after its session went down is in the table once PeerDown ends.
Tests that fail on the current code show both.
The code stays in packages routetable (its doc: it applies peers' sessions going down) and routewatch (its doc: it holds the handlers). No new package.
README and comments that describe a session going down match the code.
Model: opus-5-5
On a five-day run of `1c65656` on the live feed (27 October to 1 November), the prefix handler dropped 549,667 of 2.56 billion messages, in bursts. A dropped withdrawal leaves a route in the table that no peer carries.
**Cause.** `Table.PeerDown` in `internal/routetable/routetable.go` visits all 1.55 million prefixes while it holds the table's write lock. The prefix handler calls it in `internal/routewatch/prefixhandler.go` for every session that goes down, and takes no message off its queue meanwhile.
- The feed reported 91,578 sessions going down in the five days, often several within seconds.
- The prefix handler's longest single message took 1.47 seconds. Its queue holds 20,000 messages, about 2.5 seconds of feed.
- At 06:56 UTC on 28 October, five sessions went down within 12 seconds and the handler dropped 11,093 updates. At 15:39 UTC, four went down within 10 seconds and it dropped 14,838.
- While the lock is held, IP lookups and the prefix pages wait too.
**Recommended fix.**
1. `PeerDown` releases the lock every 10,000 prefixes and takes it again. Go allows a map to change between the steps of a range over it.
2. The prefix handler runs `PeerDown` on a goroutine of its own, one peer at a time, so it keeps taking messages meanwhile.
3. The handler holds back the down peer's later messages until its `PeerDown` returns, then applies them in order. A route that peer announces after its session went down must survive.
**Definition of done.**
- A session going down no longer stops the prefix handler, and no reader waits on the table's lock for the whole visit.
- A route announced by the same peer after its session went down is in the table once `PeerDown` ends.
- Tests that fail on the current code show both.
- The code stays in packages `routetable` (its doc: it applies peers' sessions going down) and `routewatch` (its doc: it holds the handlers). No new package.
- README and comments that describe a session going down match the code.
Model: opus-5-5
clawbot
self-assigned this 2026-11-02 00:16:14 +01:00
Between batches, PeerDown lets the lock go for as long as it held it for the batch, then takes it again. The handler and the readers then have the table at least half the time during a walk, which is far more than the feed needs.
The doc comments and the TODO.md entry say this in one plain sentence each.
A test runs a real PeerDown over many batches while the handler is handed another peer's announcements, and finds them applied before the walk ends. It fails on the PR's current code.
Model: opus-5-5
Direction for the second rework of https://git.eeqj.de/sneak/routewatch/pulls/220, after https://git.eeqj.de/sneak/routewatch/pulls/220#issuecomment-160559:
- Between batches, `PeerDown` lets the lock go for as long as it held it for the batch, then takes it again. The handler and the readers then have the table at least half the time during a walk, which is far more than the feed needs.
- The doc comments and the `TODO.md` entry say this in one plain sentence each.
- A test runs a real `PeerDown` over many batches while the handler is handed another peer's announcements, and finds them applied before the walk ends. It fails on the PR's current code.
Model: opus-5-5
A 24-hour run of 26d939a on the live feed, with an empty database (this host cannot cap a container's memory, so memory is read from the process and compared with 5 GiB), checks this fix and #217's: whether any handler still drops updates. What it finds is filed; the outcome goes here.
Model: opus-5-5
A 24-hour run of `26d939a` on the live feed, with an empty database (this host cannot cap a container's memory, so memory is read from the process and compared with 5 GiB), checks this fix and https://git.eeqj.de/sneak/routewatch/issues/217's: whether any handler still drops updates. What it finds is filed; the outcome goes here.
Model: opus-5-5
Outcome of the run of 26d939a, stopped after 5 hours because it had found its cause:
The prefix handler dropped 17,983 of 86 million messages. Every burst fell during the home page's once-a-minute walk of the table, #226, now fixed on next. None came from a session going down.
The AS handler dropped nothing. The feed did not reconnect in these 5 hours, so the burst after a reconnect that made it drop before did not happen.
Memory held at 2.64 GB.
A 24-hour run of 9bd0e91 follows; its outcome goes on #226.
Model: opus-5-5
Outcome of the run of `26d939a`, stopped after 5 hours because it had found its cause:
- The prefix handler dropped 17,983 of 86 million messages. Every burst fell during the home page's once-a-minute walk of the table, https://git.eeqj.de/sneak/routewatch/issues/226, now fixed on `next`. None came from a session going down.
- The AS handler dropped nothing. The feed did not reconnect in these 5 hours, so the burst after a reconnect that made it drop before did not happen.
- Memory held at 2.64 GB.
A 24-hour run of `9bd0e91` follows; its outcome goes on https://git.eeqj.de/sneak/routewatch/issues/226.
Model: opus-5-5
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
On a five-day run of
1c65656on the live feed (27 October to 1 November), the prefix handler dropped 549,667 of 2.56 billion messages, in bursts. A dropped withdrawal leaves a route in the table that no peer carries.Cause.
Table.PeerDownininternal/routetable/routetable.govisits all 1.55 million prefixes while it holds the table's write lock. The prefix handler calls it ininternal/routewatch/prefixhandler.gofor every session that goes down, and takes no message off its queue meanwhile.Recommended fix.
PeerDownreleases the lock every 10,000 prefixes and takes it again. Go allows a map to change between the steps of a range over it.PeerDownon a goroutine of its own, one peer at a time, so it keeps taking messages meanwhile.PeerDownreturns, then applies them in order. A route that peer announces after its session went down must survive.Definition of done.
PeerDownends.routetable(its doc: it applies peers' sessions going down) androutewatch(its doc: it holds the handlers). No new package.Model: opus-5-5
Direction for the second rework of #220, after #220 (comment):
PeerDownlets the lock go for as long as it held it for the batch, then takes it again. The handler and the readers then have the table at least half the time during a walk, which is far more than the feed needs.TODO.mdentry say this in one plain sentence each.PeerDownover many batches while the handler is handed another peer's announcements, and finds them applied before the walk ends. It fails on the PR's current code.Model: opus-5-5
clawbot referenced this issue2026-11-02 03:00:31 +01:00
A 24-hour run of
26d939aon the live feed, with an empty database (this host cannot cap a container's memory, so memory is read from the process and compared with 5 GiB), checks this fix and #217's: whether any handler still drops updates. What it finds is filed; the outcome goes here.Model: opus-5-5
Outcome of the run of
26d939a, stopped after 5 hours because it had found its cause:next. None came from a session going down.A 24-hour run of
9bd0e91follows; its outcome goes on #226.Model: opus-5-5