Write each handler batch from its flush goroutine, outside the mutex (closes #65)
check / check (push) Canceled after 0s
check / check (push) Canceled after 0s
The AS, peer and prefix handlers wrote their batch to the database while holding the mutex HandleMessage needs, so while a write ran nothing was taken off their queues and the streamer dropped updates. Each handler now writes only from its flush goroutine: it takes the batch under the mutex, leaves an empty one in its place, and writes after releasing it. A full batch wakes the flush goroutine. While a write runs the next batch may grow to ten batch sizes; at that size HandleMessage waits, so the queue fills and the streamer drops as before. README "Memory" gives the numbers and their cost. Model: opus-5-5
This commit was merged in pull request #83.
This commit is contained in:
@@ -233,6 +233,15 @@ What happens at each limit:
|
||||
keeps running.
|
||||
- Handler queues: each of the four handler queues holds at most 20,000 messages.
|
||||
When a queue fills, the streamer drops messages instead of blocking.
|
||||
- Handler batches: while the AS, peer or prefix handler writes a batch to the
|
||||
database, the updates that arrive collect in its next batch, which stops at
|
||||
ten times the batch size: 300,000 updates for the AS handler
|
||||
(`asnBatchMaxSize`, about 10 MiB), 100,000 for the peer handler
|
||||
(`peerBatchMaxSize`, about 10 MiB) and 250,000 for the prefix handler
|
||||
(`prefixBatchMaxSize`, about 60 MiB). At that size the handler takes nothing
|
||||
more off its queue until the write ends, so the queue fills and the streamer
|
||||
drops messages as above. With the batches being written, the three handlers'
|
||||
batches take at most about 0.25 GiB of the Go heap's 1.5 GiB.
|
||||
|
||||
With `DEBUG=routewatch` the daemon logs a `System stats` line every 60 seconds
|
||||
with the goroutine count and Go memory figures.
|
||||
|
||||
Reference in New Issue
Block a user