Drop an update only when its handler's queue is full (closes #229)
check / check (push) Waiting to run

The streamer dropped a random share of a handler's updates once its
queue was half full. When the feed paused and then sent its backlog
while the routing table's Overview walked every prefix, the prefix
handler lost about 10,000 updates with its queue never full.

The streamer now drops an update only when the queue is full. The
prefix handler's queue holds 100,000 messages, about 100 MiB, so it
can hold such a backlog. The README's Memory and Feed handling
sections give the new sizes and the rule.

Model: opus-5-5
This commit was merged in pull request #230.
This commit is contained in:
2026-11-03 18:27:09 +01:00
parent 742e75914c
commit 15bf20a894
5 changed files with 100 additions and 60 deletions
+10 -9
View File
@@ -543,7 +543,8 @@ What happens at each limit:
changes again in a later write. Everything else in the failed write is
dropped. The process keeps running.
- The handler queues: the handlers are the four parts that take the feed's
messages (see Design). Each has a queue of at most 20,000 messages. When a
messages (see Design). The prefix handler's queue holds at most 100,000
messages, about 100 MiB. Each of the other three holds at most 20,000. When a
queue fills, the streamer drops messages instead of waiting. The streamer is
the part that reads the feed. The exception is a `STATE` message, which it
holds, in feed order, until there is room. A `STATE` message says that a
@@ -703,15 +704,15 @@ carries are removed too, until they are announced again.
The streamer reads the stream line by line. Each line is one message as JSON. It
passes an `UPDATE` or `STATE` message to each handler that wants it, through
that handler's queue of 20,000 messages. An `UPDATE` message carries
announcements and withdrawals. A `STATE` message says that the BGP session
between a collector and a peer changed state, for example went down.
that handler's queue. An `UPDATE` message carries announcements and withdrawals.
A `STATE` message says that the BGP session between a collector and a peer
changed state, for example went down.
When a queue is more than half full, the streamer drops a share of the messages
for that handler. The share rises from none at half full to all at full. A
`STATE` message is never dropped. One that finds the queue full is held until
there is room, and the handler's other messages are dropped until the held ones
are in.
The prefix handler's queue holds 100,000 messages. Each of the other handlers'
queues holds 20,000. A message that finds its handler's queue full is dropped
for that handler. A `STATE` message is never dropped. One that finds the queue
full is held until there is room, and the handler's other messages are dropped
until the held ones are in.
The AS, peer and prefix handlers write to the database in batches. A batch is
written when it is full, or when its time is up: