Drop an update only when its handler's queue is full (closes #229)
check / check (push) Canceled after 0s
check / check (push) Canceled after 0s
The streamer dropped a random share of a handler's updates once its queue was half full. When the feed paused and then sent its backlog while the routing table's Overview walked every prefix, the prefix handler lost about 10,000 updates with its queue never full. The streamer now drops an update only when the queue is full. The prefix handler's queue holds 100,000 messages, about 100 MiB, so it can hold such a backlog. The README's Memory and Feed handling sections give the new sizes and the rule. Model: opus-5-5
This commit was merged in pull request #230.
This commit is contained in:
1 parent
742e75914c
commit
15bf20a894
5 files changed
+100
-60
No files matched your search
@@ -15,12 +15,15 @@ import (
|
||||
)
|
||||
|
||||
const (
|
||||
// prefixHandlerQueueSize is the queue capacity for prefix tracking
|
||||
// operations, about 4 seconds of feed at peak. The streamer drops updates
|
||||
// rather than blocks when a queue is full (a STATE message is held until
|
||||
// there is room), so this bounds memory. Batches still flush on a timer
|
||||
// (prefixBatchTimeout).
|
||||
prefixHandlerQueueSize = 20000
|
||||
// prefixHandlerQueueSize is the most messages the handler's queue holds.
|
||||
// While the routing table's Overview or PeersDown walks every prefix, it
|
||||
// holds the table's lock about half of the time. The handler then applies
|
||||
// updates more slowly. When the feed pauses for some seconds and then
|
||||
// catches up, its backlog can arrive faster than that, and the queue
|
||||
// holds the backlog. The streamer drops an update that finds the queue
|
||||
// full. A message takes about 1 KiB, so a full queue takes about 100 MiB
|
||||
// (README "Memory").
|
||||
prefixHandlerQueueSize = 100000
|
||||
|
||||
// prefixBatchSize is the number of prefix updates to batch together
|
||||
prefixBatchSize = 25000
|
||||
|
||||
Reference in new issue
Block a user