Write each handler batch from its flush goroutine, outside the mutex (closes #65)
check / check (push) Canceled after 0s

The AS, peer and prefix handlers wrote their batch to the database while
holding the mutex HandleMessage needs, and HandleMessage wrote a full batch
itself. For as long as a write took, the handler took nothing off its queue,
and the streamer dropped updates once the queue was half full.

Now only the flush goroutine writes: it takes the batch under the mutex,
releases it, then writes. A full batch wakes the flush goroutine instead.
While a write runs, the next batch grows to at most ten batch sizes; at that
size HandleMessage waits until the flush goroutine takes it (README
"Memory"). Stop still writes what the handler holds.

Model: opus-5-5
This commit is contained in:
2026-10-19 07:29:40 +00:00
parent f7c88c2496
commit 276a55ab60
6 changed files with 584 additions and 93 deletions
+9
View File
@@ -233,6 +233,15 @@ What happens at each limit:
keeps running.
- Handler queues: each of the four handler queues holds at most 20,000 messages.
When a queue fills, the streamer drops messages instead of blocking.
- Handler batches: while the AS, peer or prefix handler writes a batch to the
database, the updates that arrive collect in its next batch, which stops at
ten times the batch size: 300,000 updates for the AS handler
(`asnBatchMaxSize`, about 10 MiB), 100,000 for the peer handler
(`peerBatchMaxSize`, about 10 MiB) and 250,000 for the prefix handler
(`prefixBatchMaxSize`, about 60 MiB). At that size the handler takes nothing
more off its queue until the write ends, so the queue fills and the streamer
drops messages as above. With the batches being written, the three handlers'
batches take at most about 0.25 GiB of the Go heap's 1.5 GiB.
With `DEBUG=routewatch` the daemon logs a `System stats` line every 60 seconds
with the goroutine count and Go memory figures.