Write each handler batch from its flush goroutine, outside the mutex (closes #65)
check / check (push) Canceled after 0s

The AS, peer and prefix handlers wrote their batch to the database while
holding the mutex HandleMessage needs, so while a write ran nothing was
taken off their queues and the streamer dropped updates.

Each handler now writes only from its flush goroutine: it takes the batch
under the mutex, leaves an empty one in its place, and writes after
releasing it. A full batch wakes the flush goroutine. While a write runs the
next batch may grow to ten batch sizes; at that size HandleMessage waits, so
the queue fills and the streamer drops as before. README "Memory" gives the
numbers and their cost.

Model: opus-5-5
This commit was merged in pull request #83.
This commit is contained in:
2026-10-19 09:45:24 +02:00
parent f7c88c2496
commit 43c61ee0d0
6 changed files with 584 additions and 93 deletions
+9
View File
@@ -233,6 +233,15 @@ What happens at each limit:
keeps running.
- Handler queues: each of the four handler queues holds at most 20,000 messages.
When a queue fills, the streamer drops messages instead of blocking.
- Handler batches: while the AS, peer or prefix handler writes a batch to the
database, the updates that arrive collect in its next batch, which stops at
ten times the batch size: 300,000 updates for the AS handler
(`asnBatchMaxSize`, about 10 MiB), 100,000 for the peer handler
(`peerBatchMaxSize`, about 10 MiB) and 250,000 for the prefix handler
(`prefixBatchMaxSize`, about 60 MiB). At that size the handler takes nothing
more off its queue until the write ends, so the queue fills and the streamer
drops messages as above. With the batches being written, the three handlers'
batches take at most about 0.25 GiB of the Go heap's 1.5 GiB.
With `DEBUG=routewatch` the daemon logs a `System stats` line every 60 seconds
with the goroutine count and Go memory figures.