Drop an update only when its handler's queue is full (closes #229)
check / check (push) Waiting to run
check / check (push) Waiting to run
The streamer dropped a random share of a handler's updates once its queue was half full. When the feed paused and then sent its backlog while the routing table's Overview walked every prefix, the prefix handler lost about 10,000 updates with its queue never full. The streamer now drops an update only when the queue is full. The prefix handler's queue holds 100,000 messages, about 100 MiB, so it can hold such a backlog. The README's Memory and Feed handling sections give the new sizes and the rule. Model: opus-5-5
This commit was merged in pull request #230.
This commit is contained in:
@@ -543,7 +543,8 @@ What happens at each limit:
|
||||
changes again in a later write. Everything else in the failed write is
|
||||
dropped. The process keeps running.
|
||||
- The handler queues: the handlers are the four parts that take the feed's
|
||||
messages (see Design). Each has a queue of at most 20,000 messages. When a
|
||||
messages (see Design). The prefix handler's queue holds at most 100,000
|
||||
messages, about 100 MiB. Each of the other three holds at most 20,000. When a
|
||||
queue fills, the streamer drops messages instead of waiting. The streamer is
|
||||
the part that reads the feed. The exception is a `STATE` message, which it
|
||||
holds, in feed order, until there is room. A `STATE` message says that a
|
||||
@@ -703,15 +704,15 @@ carries are removed too, until they are announced again.
|
||||
|
||||
The streamer reads the stream line by line. Each line is one message as JSON. It
|
||||
passes an `UPDATE` or `STATE` message to each handler that wants it, through
|
||||
that handler's queue of 20,000 messages. An `UPDATE` message carries
|
||||
announcements and withdrawals. A `STATE` message says that the BGP session
|
||||
between a collector and a peer changed state, for example went down.
|
||||
that handler's queue. An `UPDATE` message carries announcements and withdrawals.
|
||||
A `STATE` message says that the BGP session between a collector and a peer
|
||||
changed state, for example went down.
|
||||
|
||||
When a queue is more than half full, the streamer drops a share of the messages
|
||||
for that handler. The share rises from none at half full to all at full. A
|
||||
`STATE` message is never dropped. One that finds the queue full is held until
|
||||
there is room, and the handler's other messages are dropped until the held ones
|
||||
are in.
|
||||
The prefix handler's queue holds 100,000 messages. Each of the other handlers'
|
||||
queues holds 20,000. A message that finds its handler's queue full is dropped
|
||||
for that handler. A `STATE` message is never dropped. One that finds the queue
|
||||
full is held until there is room, and the handler's other messages are dropped
|
||||
until the held ones are in.
|
||||
|
||||
The AS, peer and prefix handlers write to the database in batches. A batch is
|
||||
written when it is full, or when its time is up:
|
||||
|
||||
Reference in New Issue
Block a user