Drop an update only when its handler's queue is full (closes #229)
check / check (push) Canceled after 0s

The streamer dropped a random share of a handler's updates once its
queue was half full. When the feed paused and then sent its backlog
while the routing table's Overview walked every prefix, the prefix
handler lost about 10,000 updates with its queue never full.

The streamer now drops an update only when the queue is full. The
prefix handler's queue holds 100,000 messages, about 100 MiB, so it
can hold such a backlog. The README's Memory and Feed handling
sections give the new sizes and the rule.

Model: opus-5-5
This commit was merged in pull request #230.
This commit is contained in:
clawbot committed 2026-11-03 18:27:09 +01:00
1 parent 742e75914c
commit 15bf20a894
5 files changed
+100 -60

No files matched your search

+9 -6
View File
@@ -15,12 +15,15 @@ import (
)
const (
// prefixHandlerQueueSize is the queue capacity for prefix tracking
// operations, about 4 seconds of feed at peak. The streamer drops updates
// rather than blocks when a queue is full (a STATE message is held until
// there is room), so this bounds memory. Batches still flush on a timer
// (prefixBatchTimeout).
prefixHandlerQueueSize = 20000
// prefixHandlerQueueSize is the most messages the handler's queue holds.
// While the routing table's Overview or PeersDown walks every prefix, it
// holds the table's lock about half of the time. The handler then applies
// updates more slowly. When the feed pauses for some seconds and then
// catches up, its backlog can arrive faster than that, and the queue
// holds the backlog. The streamer drops an update that finds the queue
// full. A message takes about 1 KiB, so a full queue takes about 100 MiB
// (README "Memory").
prefixHandlerQueueSize = 100000
// prefixBatchSize is the number of prefix updates to batch together
prefixBatchSize = 25000