Keep one entry per AS in the AS handler's batch (closes #217)
check / check (push) Canceled after 0s
check / check (push) Canceled after 0s
HandleMessage added one entry to the batch for every AS of every path, and waited for the write in progress once the batch held 300,000. In a burst of the feed its queue then filled, and the streamer dropped messages. The batch is now a map from AS number to the earliest and the latest time of its updates. It holds one entry per AS, so it no longer grows with the feed's rate. HandleMessage never waits for a write. asnBatchSize now counts ASes. Stop logs the entries of the batch being written and of the next batch under "batch_entries". An AS in both counts twice. Model: opus-5-5
This commit was merged in pull request #219.
This commit is contained in:
@@ -429,10 +429,12 @@ the routing table, and everything else from the database.
|
||||
- Parses BGP UPDATE messages from JSON stream
|
||||
- Dispatches to registered handlers based on message type
|
||||
- Each handler has its own queue with backpressure handling
|
||||
- Handlers batch writes for efficiency: the AS, peer and prefix handlers
|
||||
write 30,000, 10,000 and 25,000 updates at a time, or after 2, 2 and 1
|
||||
seconds; the peering handler writes the peerings it has collected every 30
|
||||
seconds
|
||||
- Handlers batch writes for efficiency. The AS, peer and prefix handlers
|
||||
write a batch when it reaches a size, or after a time:
|
||||
- the AS handler at 30,000 ASes, or after 2 seconds;
|
||||
- the peer handler at 10,000 updates, or after 2 seconds;
|
||||
- the prefix handler at 25,000 updates, or after 1 second.
|
||||
- The peering handler writes the peerings it has collected every 30 seconds
|
||||
|
||||
4. **Handler Details**
|
||||
- **ASHandler**: Tracks all ASNs seen in AS paths
|
||||
@@ -632,16 +634,19 @@ What happens at each limit:
|
||||
`STATE` message, which it holds, in feed order, until there is room: a few
|
||||
hundred bytes each, about ten a second.
|
||||
- Handler batches: while the AS, peer or prefix handler writes a batch to the
|
||||
database, the updates that arrive collect in its next batch. That batch stops
|
||||
at ten times the batch size:
|
||||
- 300,000 updates for the AS handler (`asnBatchMaxSize`, about 10 MiB);
|
||||
- 100,000 for the peer handler (`peerBatchMaxSize`, about 10 MiB);
|
||||
- 250,000 for the prefix handler (`prefixBatchMaxSize`, about 15 MiB).
|
||||
database, the updates that arrive collect in its next batch.
|
||||
- The AS handler's batch holds one entry per AS, with the earliest and the
|
||||
latest time of its updates. About 80,000 ASes appear in the feed, so the
|
||||
batch takes at most about 10 MiB, however fast the feed runs.
|
||||
- The peer handler's batch stops at 100,000 updates (`peerBatchMaxSize`),
|
||||
about 10 MiB.
|
||||
- The prefix handler's batch stops at 250,000 updates
|
||||
(`prefixBatchMaxSize`), about 15 MiB.
|
||||
|
||||
At that size the handler takes nothing more off its queue until the write
|
||||
ends. The queue then fills, and the streamer drops messages as above. With
|
||||
the batches being written, the three handlers' batches take at most about
|
||||
0.1 GiB of the Go heap.
|
||||
At that maximum the peer or prefix handler takes nothing more off its queue
|
||||
until the write ends. The queue then fills, and the streamer drops messages
|
||||
as above. With the batches being written, the three handlers' batches take
|
||||
at most about 0.1 GiB of the Go heap.
|
||||
|
||||
With `DEBUG=routewatch` the daemon logs a `System stats` line every 60 seconds
|
||||
with the goroutine count and Go memory figures.
|
||||
|
||||
Reference in New Issue
Block a user