Write each handler batch from its flush goroutine, outside the mutex (closes #65)
check / check (push) Canceled after 0s
check / check (push) Canceled after 0s
The AS, peer and prefix handlers wrote their batch to the database while holding the mutex HandleMessage needs, and HandleMessage wrote a full batch itself. For as long as a write took, the handler took nothing off its queue, and the streamer dropped updates once the queue was half full. Now only the flush goroutine writes: it takes the batch under the mutex, releases it, then writes. A full batch wakes the flush goroutine instead. While a write runs, the next batch grows to at most ten batch sizes; at that size HandleMessage waits until the flush goroutine takes it (README "Memory"). Stop still writes what the handler holds. Model: opus-5-5
This commit is contained in:
@@ -233,6 +233,15 @@ What happens at each limit:
|
||||
keeps running.
|
||||
- Handler queues: each of the four handler queues holds at most 20,000 messages.
|
||||
When a queue fills, the streamer drops messages instead of blocking.
|
||||
- Handler batches: while the AS, peer or prefix handler writes a batch to the
|
||||
database, the updates that arrive collect in its next batch, which stops at
|
||||
ten times the batch size: 300,000 updates for the AS handler
|
||||
(`asnBatchMaxSize`, about 10 MiB), 100,000 for the peer handler
|
||||
(`peerBatchMaxSize`, about 10 MiB) and 250,000 for the prefix handler
|
||||
(`prefixBatchMaxSize`, about 60 MiB). At that size the handler takes nothing
|
||||
more off its queue until the write ends, so the queue fills and the streamer
|
||||
drops messages as above. With the batches being written, the three handlers'
|
||||
batches take at most about 0.25 GiB of the Go heap's 1.5 GiB.
|
||||
|
||||
With `DEBUG=routewatch` the daemon logs a `System stats` line every 60 seconds
|
||||
with the goroutine count and Go memory figures.
|
||||
|
||||
Reference in New Issue
Block a user