The fixes for #216, #225 and #226 are on next. In the 24-hour run of 742e759 on the live feed, the prefix handler still dropped 10,096 updates in one burst, between 15:37:58 and 15:38:08 UTC on 2026-11-03, 16 hours in. The other handlers dropped none.
What the run shows around the burst.
At 15:37:59 the sessions of two peers went down: 2400:8800:f000:2::156 (AS134823) and 193.239.118.249 (AS41255). The first came back up 4 milliseconds later.
At 15:38:01 the peering handler logged a database write that took 3.2 seconds.
The feed ran at about 4,400 messages a second, less than at many other moments of the run without a drop.
The prefix handler's longest single message in the run took 0.23 seconds. Its queue's high-water mark is 15,231 of 20,000.
The held-back count, polled every 10 seconds, read 0 before and after the burst.
At 04:03 the same run handled ten sessions going down and nineteen opening within ten seconds without a drop.
Not known: what stopped the handler taking messages for the seconds the queue needed to fill. The cause is not yet worked out; a plan with figures from the code and the logs follows as a comment before any code.
Definition of done.
The cause is found and fixed, so a burst like this one no longer makes the prefix handler drop updates.
A test that fails on the current code shows it.
README text and comments about the handler match the code.
Model: opus-5-5
The fixes for https://git.eeqj.de/sneak/routewatch/issues/216, https://git.eeqj.de/sneak/routewatch/issues/225 and https://git.eeqj.de/sneak/routewatch/issues/226 are on `next`. In the 24-hour run of `742e759` on the live feed, the prefix handler still dropped 10,096 updates in one burst, between 15:37:58 and 15:38:08 UTC on 2026-11-03, 16 hours in. The other handlers dropped none.
**What the run shows around the burst.**
- At 15:37:59 the sessions of two peers went down: 2400:8800:f000:2::156 (AS134823) and 193.239.118.249 (AS41255). The first came back up 4 milliseconds later.
- At 15:38:01 the peering handler logged a database write that took 3.2 seconds.
- The feed ran at about 4,400 messages a second, less than at many other moments of the run without a drop.
- The prefix handler's longest single message in the run took 0.23 seconds. Its queue's high-water mark is 15,231 of 20,000.
- The held-back count, polled every 10 seconds, read 0 before and after the burst.
At 04:03 the same run handled ten sessions going down and nineteen opening within ten seconds without a drop.
**Not known:** what stopped the handler taking messages for the seconds the queue needed to fill. The cause is not yet worked out; a plan with figures from the code and the logs follows as a comment before any code.
**Definition of done.**
- The cause is found and fixed, so a burst like this one no longer makes the prefix handler drop updates.
- A test that fails on the current code shows it.
- README text and comments about the handler match the code.
Model: opus-5-5
clawbot
self-assigned this 2026-11-03 17:00:28 +01:00
Plan, from the run of 742e759 (container log, statistics every 10 s) and next.
Cause. The stream stalled, and its backlog then arrived faster than the prefix handler takes messages while the home page's refresh walks the table. Measured:
The stream sent 12,813 messages in the 10 s to 15:37:48 and 606 in the 10 s to 15:37:58, against about 40,000 around; then 136,597 in the 10 s to 15:38:08. Four peers send a NOTIFICATION once a minute all run (957 each). Their delayed ones show the backlog's pace: the messages due about 15:37:45 and 15:37:55 were read at 15:37:58.99 and 15:37:59.52, about 80,000 a second.
keepHomeCardsFresh (internal/server/home.go) started with the server at 23:27:58.5 and ticks once a minute, so Overview walked the 1.42 million prefixes from 15:37:58.5, holding the lock about half the time.
The queue never filled: queueMessage drops at random from 10,000 queued (half full, shouldDropForBackpressure), and the high-water mark is 15,231. The handler never stopped: its longest message stayed 0.23 s, the held-back count 0.
Of the run's 47 stalls, the one at 00:39:58 met the same walk, but its backlog came in over 5 s: no drop, queue under 3,400.
Inferred: during a walk the handler takes about half of its pace alone (8 µs a message), about 50,000 a second.
Not involved: the two peers' sessions, which go down every minute and carry no route, so PeersDown returns at once; the peering handler's 3.2 s write, since HandleMessage touches no database and the flush goroutine waits for the database mutex only between single peerings. No earlier burst: total_dropped went from 0 to 10,096 in one step.
Change.
internal/streamer/streamer.go: queueMessage drops an update only when the queue is full. shouldDropForBackpressure, backpressureThreshold and backpressureSlope go; the docs of MessageHandler.QueueCapacity and queueMessage say so. Tests that need drops fill the queue.
internal/routewatch/prefixhandler.go: prefixHandlerQueueSize becomes 100,000. Its doc says the queue holds a backlog arriving faster than the handler takes it while a walk runs. About 1 KiB a message (README "Memory"), 100 MiB at most. Whether it is enough is measured by the next run's high-water mark, as for the hold-back limit (#225 (comment)).
README "Memory" (handler queues) and "Feed handling" (the half-full paragraph); TODO.md entry.
Test, failing today: in internal/streamer, a handler whose HandleMessage waits on a channel is sent 20,000 updates. None is dropped before the queue is full, the next one is, and once released the handler gets all 20,000 in order. Today thousands are dropped past 10,000.
Rejected. Longer walk pauses: a figure from an inferred pace, and a faster backlog beats it. One walk at a time: no second walk ran. Larger queue alone: half of it goes to random drops. Smaller walk batches: the handler's share stays half.
Model: fable-5-1
Plan, from the run of `742e759` (container log, statistics every 10 s) and `next`.
**Cause.** The stream stalled, and its backlog then arrived faster than the prefix handler takes messages while the home page's refresh walks the table. Measured:
- The stream sent 12,813 messages in the 10 s to 15:37:48 and 606 in the 10 s to 15:37:58, against about 40,000 around; then 136,597 in the 10 s to 15:38:08. Four peers send a NOTIFICATION once a minute all run (957 each). Their delayed ones show the backlog's pace: the messages due about 15:37:45 and 15:37:55 were read at 15:37:58.99 and 15:37:59.52, about 80,000 a second.
- `keepHomeCardsFresh` (`internal/server/home.go`) started with the server at 23:27:58.5 and ticks once a minute, so `Overview` walked the 1.42 million prefixes from 15:37:58.5, holding the lock about half the time.
- The queue never filled: `queueMessage` drops at random from 10,000 queued (half full, `shouldDropForBackpressure`), and the high-water mark is 15,231. The handler never stopped: its longest message stayed 0.23 s, the held-back count 0.
- Of the run's 47 stalls, the one at 00:39:58 met the same walk, but its backlog came in over 5 s: no drop, queue under 3,400.
Inferred: during a walk the handler takes about half of its pace alone (8 µs a message), about 50,000 a second.
Not involved: the two peers' sessions, which go down every minute and carry no route, so `PeersDown` returns at once; the peering handler's 3.2 s write, since `HandleMessage` touches no database and the flush goroutine waits for the database mutex only between single peerings. No earlier burst: `total_dropped` went from 0 to 10,096 in one step.
**Change.**
1. `internal/streamer/streamer.go`: `queueMessage` drops an update only when the queue is full. `shouldDropForBackpressure`, `backpressureThreshold` and `backpressureSlope` go; the docs of `MessageHandler.QueueCapacity` and `queueMessage` say so. Tests that need drops fill the queue.
2. `internal/routewatch/prefixhandler.go`: `prefixHandlerQueueSize` becomes 100,000. Its doc says the queue holds a backlog arriving faster than the handler takes it while a walk runs. About 1 KiB a message (README "Memory"), 100 MiB at most. Whether it is enough is measured by the next run's high-water mark, as for the hold-back limit (https://git.eeqj.de/sneak/routewatch/issues/225#issuecomment-161180).
3. README "Memory" (handler queues) and "Feed handling" (the half-full paragraph); `TODO.md` entry.
**Test**, failing today: in `internal/streamer`, a handler whose `HandleMessage` waits on a channel is sent 20,000 updates. None is dropped before the queue is full, the next one is, and once released the handler gets all 20,000 in order. Today thousands are dropped past 10,000.
**Rejected.** Longer walk pauses: a figure from an inferred pace, and a faster backlog beats it. One walk at a time: no second walk ran. Larger queue alone: half of it goes to random drops. Smaller walk batches: the handler's share stays half.
Model: fable-5-1
A 24-hour run of 15bf20a on the live feed, with an empty database, checks this fix: whether any handler still drops updates, and the prefix queue's high-water mark against its 100,000. Its outcome goes here.
Model: opus-5-5
A 24-hour run of `15bf20a` on the live feed, with an empty database, checks this fix: whether any handler still drops updates, and the prefix queue's high-water mark against its 100,000. Its outcome goes here.
Model: opus-5-5
The run of 15bf20a was stopped after 3 hours, on 2026-11-03 at 20:18 UTC, when work on this repo was paused, so it is not the 24-hour check planned. What it measured: no handler dropped an update; the prefix queue peaked at 5,611 of 100,000; memory peaked at 2.39 GB; the feed did not stall or reconnect, so the case this issue fixed did not come up. A run of at least a day is still needed to confirm the fix.
Model: opus-5-5
The run of `15bf20a` was stopped after 3 hours, on 2026-11-03 at 20:18 UTC, when work on this repo was paused, so it is not the 24-hour check planned. What it measured: no handler dropped an update; the prefix queue peaked at 5,611 of 100,000; memory peaked at 2.39 GB; the feed did not stall or reconnect, so the case this issue fixed did not come up. A run of at least a day is still needed to confirm the fix.
Model: opus-5-5
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
The fixes for #216, #225 and #226 are on
next. In the 24-hour run of742e759on the live feed, the prefix handler still dropped 10,096 updates in one burst, between 15:37:58 and 15:38:08 UTC on 2026-11-03, 16 hours in. The other handlers dropped none.What the run shows around the burst.
At 04:03 the same run handled ten sessions going down and nineteen opening within ten seconds without a drop.
Not known: what stopped the handler taking messages for the seconds the queue needed to fill. The cause is not yet worked out; a plan with figures from the code and the logs follows as a comment before any code.
Definition of done.
Model: opus-5-5
Plan, from the run of
742e759(container log, statistics every 10 s) andnext.Cause. The stream stalled, and its backlog then arrived faster than the prefix handler takes messages while the home page's refresh walks the table. Measured:
keepHomeCardsFresh(internal/server/home.go) started with the server at 23:27:58.5 and ticks once a minute, soOverviewwalked the 1.42 million prefixes from 15:37:58.5, holding the lock about half the time.queueMessagedrops at random from 10,000 queued (half full,shouldDropForBackpressure), and the high-water mark is 15,231. The handler never stopped: its longest message stayed 0.23 s, the held-back count 0.Inferred: during a walk the handler takes about half of its pace alone (8 µs a message), about 50,000 a second.
Not involved: the two peers' sessions, which go down every minute and carry no route, so
PeersDownreturns at once; the peering handler's 3.2 s write, sinceHandleMessagetouches no database and the flush goroutine waits for the database mutex only between single peerings. No earlier burst:total_droppedwent from 0 to 10,096 in one step.Change.
internal/streamer/streamer.go:queueMessagedrops an update only when the queue is full.shouldDropForBackpressure,backpressureThresholdandbackpressureSlopego; the docs ofMessageHandler.QueueCapacityandqueueMessagesay so. Tests that need drops fill the queue.internal/routewatch/prefixhandler.go:prefixHandlerQueueSizebecomes 100,000. Its doc says the queue holds a backlog arriving faster than the handler takes it while a walk runs. About 1 KiB a message (README "Memory"), 100 MiB at most. Whether it is enough is measured by the next run's high-water mark, as for the hold-back limit (#225 (comment)).TODO.mdentry.Test, failing today: in
internal/streamer, a handler whoseHandleMessagewaits on a channel is sent 20,000 updates. None is dropped before the queue is full, the next one is, and once released the handler gets all 20,000 in order. Today thousands are dropped past 10,000.Rejected. Longer walk pauses: a figure from an inferred pace, and a faster backlog beats it. One walk at a time: no second walk ran. Larger queue alone: half of it goes to random drops. Smaller walk batches: the handler's share stays half.
Model: fable-5-1
A 24-hour run of
15bf20aon the live feed, with an empty database, checks this fix: whether any handler still drops updates, and the prefix queue's high-water mark against its 100,000. Its outcome goes here.Model: opus-5-5
The run of
15bf20awas stopped after 3 hours, on 2026-11-03 at 20:18 UTC, when work on this repo was paused, so it is not the 24-hour check planned. What it measured: no handler dropped an update; the prefix queue peaked at 5,611 of 100,000; memory peaked at 2.39 GB; the feed did not stall or reconnect, so the case this issue fixed did not come up. A run of at least a day is still needed to confirm the fix.Model: opus-5-5