On a five-day run of 1c65656 on the live feed (27 October to 1 November), the AS handler dropped 37,526 messages in about ten bursts. Each burst came while sessions were coming back up and the feed ran at up to 24,600 messages a second, three times its usual rate.
Cause.HandleMessage in internal/routewatch/ashandler.go appends one item to the batch for every AS in every path, about five for each update. At 24,600 updates a second, that is about 120,000 items a second. While the flush goroutine writes one batch, the next batch grows. At 300,000 items (asnBatchMaxSize), about 2.5 seconds of such a burst, HandleMessage waits for the write to end. The queue of 20,000 then fills in under a second and the streamer drops messages. For example, at 14:39 UTC on 28 October one message took 380 milliseconds and the handler dropped 9,777 in ten seconds.
flushBatch reduces the batch to one first-seen and one last-seen time per AS anyway, and about 80,000 ASes appear in the feed.
Recommended fix. Keep the batch as a map from AS number to its seen times, adding each AS's time as the update arrives. The batch then holds at most one entry per AS, never grows with the feed rate, and HandleMessage never waits for a write. asnBatchMaxSize and the wait go away.
Definition of done.
HandleMessage never waits for a database write.
The batch holds one entry per AS, whatever the number of updates.
First seen and last seen stay the earliest and the latest time the batch saw for each AS.
A test that fails on the current code shows HandleMessage returns while a write is blocked.
The code stays in package routewatch (its doc: it holds the handlers that write the feed's updates in batches). No new package.
README text and comments about the AS handler's batch and memory match the code.
Model: opus-5-5
On a five-day run of `1c65656` on the live feed (27 October to 1 November), the AS handler dropped 37,526 messages in about ten bursts. Each burst came while sessions were coming back up and the feed ran at up to 24,600 messages a second, three times its usual rate.
**Cause.** `HandleMessage` in `internal/routewatch/ashandler.go` appends one item to the batch for every AS in every path, about five for each update. At 24,600 updates a second, that is about 120,000 items a second. While the flush goroutine writes one batch, the next batch grows. At 300,000 items (`asnBatchMaxSize`), about 2.5 seconds of such a burst, `HandleMessage` waits for the write to end. The queue of 20,000 then fills in under a second and the streamer drops messages. For example, at 14:39 UTC on 28 October one message took 380 milliseconds and the handler dropped 9,777 in ten seconds.
`flushBatch` reduces the batch to one first-seen and one last-seen time per AS anyway, and about 80,000 ASes appear in the feed.
**Recommended fix.** Keep the batch as a map from AS number to its seen times, adding each AS's time as the update arrives. The batch then holds at most one entry per AS, never grows with the feed rate, and `HandleMessage` never waits for a write. `asnBatchMaxSize` and the wait go away.
**Definition of done.**
- `HandleMessage` never waits for a database write.
- The batch holds one entry per AS, whatever the number of updates.
- First seen and last seen stay the earliest and the latest time the batch saw for each AS.
- A test that fails on the current code shows `HandleMessage` returns while a write is blocked.
- The code stays in package `routewatch` (its doc: it holds the handlers that write the feed's updates in batches). No new package.
- README text and comments about the AS handler's batch and memory match the code.
Model: opus-5-5
clawbot
self-assigned this 2026-11-02 00:16:15 +01:00
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
On a five-day run of
1c65656on the live feed (27 October to 1 November), the AS handler dropped 37,526 messages in about ten bursts. Each burst came while sessions were coming back up and the feed ran at up to 24,600 messages a second, three times its usual rate.Cause.
HandleMessageininternal/routewatch/ashandler.goappends one item to the batch for every AS in every path, about five for each update. At 24,600 updates a second, that is about 120,000 items a second. While the flush goroutine writes one batch, the next batch grows. At 300,000 items (asnBatchMaxSize), about 2.5 seconds of such a burst,HandleMessagewaits for the write to end. The queue of 20,000 then fills in under a second and the streamer drops messages. For example, at 14:39 UTC on 28 October one message took 380 milliseconds and the handler dropped 9,777 in ten seconds.flushBatchreduces the batch to one first-seen and one last-seen time per AS anyway, and about 80,000 ASes appear in the feed.Recommended fix. Keep the batch as a map from AS number to its seen times, adding each AS's time as the update arrives. The batch then holds at most one entry per AS, never grows with the feed rate, and
HandleMessagenever waits for a write.asnBatchMaxSizeand the wait go away.Definition of done.
HandleMessagenever waits for a database write.HandleMessagereturns while a write is blocked.routewatch(its doc: it holds the handlers that write the feed's updates in batches). No new package.Model: opus-5-5