The AS, peer and prefix handlers wrote their batch to the database while holding the mutex that HandleMessage needs, and HandleMessage wrote a full batch itself. While a write ran, the handler took nothing off its queue, and the streamer dropped updates once the queue was half full.
Now each handler writes only from its flush goroutine: it takes the batch under the mutex, releases the mutex, then writes. A full batch wakes the flush goroutine through a one-slot channel, so a wake-up sent during a write is picked up when that write ends. Stop still writes what the handler holds.
While a write runs, the next batch grows to at most ten batch sizes (asnBatchMaxSize, peerBatchMaxSize, prefixBatchMaxSize; numbers and memory cost in README "Memory"). At that size HandleMessage waits until the flush goroutine takes the batch, so the queue fills and the streamer drops as on next.
The tests hold a store's batch writes until released: HandleMessage returns during a held write, waits once the next batch is at its maximum size, and returns after the release; Stop writes what was handled meanwhile. A handler without its flush goroutine shows that a full batch leaves a wake-up and one update short leaves none.
Deviation: an update reaching a handler after its Stop is not written, and a maximum-size batch then leaves that handler's worker waiting until exit; #69 changes the shutdown order so none arrive.
Unverified: the README memory figures are estimated from the size of each batch entry, not measured.
Model: opus-5-5
Fixes https://git.eeqj.de/sneak/routewatch/issues/65.
The AS, peer and prefix handlers wrote their batch to the database while holding the mutex that `HandleMessage` needs, and `HandleMessage` wrote a full batch itself. While a write ran, the handler took nothing off its queue, and the streamer dropped updates once the queue was half full.
Now each handler writes only from its flush goroutine: it takes the batch under the mutex, releases the mutex, then writes. A full batch wakes the flush goroutine through a one-slot channel, so a wake-up sent during a write is picked up when that write ends. `Stop` still writes what the handler holds.
While a write runs, the next batch grows to at most ten batch sizes (`asnBatchMaxSize`, `peerBatchMaxSize`, `prefixBatchMaxSize`; numbers and memory cost in README "Memory"). At that size `HandleMessage` waits until the flush goroutine takes the batch, so the queue fills and the streamer drops as on `next`.
The tests hold a store's batch writes until released: `HandleMessage` returns during a held write, waits once the next batch is at its maximum size, and returns after the release; `Stop` writes what was handled meanwhile. A handler without its flush goroutine shows that a full batch leaves a wake-up and one update short leaves none.
- Deviation: an update reaching a handler after its `Stop` is not written, and a maximum-size batch then leaves that handler's worker waiting until exit; https://git.eeqj.de/sneak/routewatch/issues/69 changes the shutdown order so none arrive.
- Unverified: the README memory figures are estimated from the size of each batch entry, not measured.
Model: opus-5-5
Breaks next once rebased.TestPrefixHandlerKeepsOneRoutePerPrefixAndPeer in internal/routewatch/prefixhandler_test.go landed on next with #55 and still calls h.flushBatchLocked() under h.mu. This PR removes flushBatchLocked, so on current next the routewatch package no longer compiles. Git shows no conflict, so a squash merge would leave next red. Acceptable: rebase onto current next and make that test call h.flushBatch(), as the other tests in the file now do.
The pending batch has no ceiling. While a write runs, HandleMessage in ashandler.go, peerhandler.go and prefixhandler.go adds to the next batch with no limit. All database writes share one lock, so one slow write makes all three batches grow. Each bigger batch then goes in as one bigger transaction, which holds the lock longer. A database that stays slower than the feed pushes the Go heap past its 1.5 GiB soft limit until the 5 GiB container is killed, and every pending update is lost, withdrawals included. That is worse than the drops this PR removes, and it is the failure #3 fixed. The README Memory section now says the image's ceilings keep the daemon inside 5 GiB, and also that this batch has no limit; the new bullet sits under "What happens at each limit" but names none. Acceptable: a fixed ceiling on each handler's pending batch, sized from the memory budget, with its number and cost in the README Memory section. At the ceiling the batch stops growing: either HandleMessage waits until the flush goroutine takes the batch, so the queue fills and the streamer drops as on next, or the handler drops updates and logs how many. Add a test that holds a write and shows the batch stops at the ceiling.
No test covers the full-batch wake-up. The new tests would still pass without the send on batchFullCh in HandleMessage, because each handler's batch timer starts the write within their two-second wait. That send is part of the definition of done. Acceptable: a test that does not depend on timing. For example, build a handler without its flush goroutine and fill it to exactly the batch size: a wake-up must then be waiting in batchFullCh, and one update short there must be none.
Model: opus-5-5
1. **Breaks `next` once rebased.** `TestPrefixHandlerKeepsOneRoutePerPrefixAndPeer` in `internal/routewatch/prefixhandler_test.go` landed on `next` with https://git.eeqj.de/sneak/routewatch/issues/55 and still calls `h.flushBatchLocked()` under `h.mu`. This PR removes `flushBatchLocked`, so on current `next` the routewatch package no longer compiles. Git shows no conflict, so a squash merge would leave `next` red. Acceptable: rebase onto current `next` and make that test call `h.flushBatch()`, as the other tests in the file now do.
2. **The pending batch has no ceiling.** While a write runs, `HandleMessage` in `ashandler.go`, `peerhandler.go` and `prefixhandler.go` adds to the next batch with no limit. All database writes share one lock, so one slow write makes all three batches grow. Each bigger batch then goes in as one bigger transaction, which holds the lock longer. A database that stays slower than the feed pushes the Go heap past its 1.5 GiB soft limit until the 5 GiB container is killed, and every pending update is lost, withdrawals included. That is worse than the drops this PR removes, and it is the failure https://git.eeqj.de/sneak/routewatch/issues/3 fixed. The README Memory section now says the image's ceilings keep the daemon inside 5 GiB, and also that this batch has no limit; the new bullet sits under "What happens at each limit" but names none. Acceptable: a fixed ceiling on each handler's pending batch, sized from the memory budget, with its number and cost in the README Memory section. At the ceiling the batch stops growing: either `HandleMessage` waits until the flush goroutine takes the batch, so the queue fills and the streamer drops as on `next`, or the handler drops updates and logs how many. Add a test that holds a write and shows the batch stops at the ceiling.
3. **No test covers the full-batch wake-up.** The new tests would still pass without the send on `batchFullCh` in `HandleMessage`, because each handler's batch timer starts the write within their two-second wait. That send is part of the definition of done. Acceptable: a test that does not depend on timing. For example, build a handler without its flush goroutine and fill it to exactly the batch size: a wake-up must then be waiting in `batchFullCh`, and one update short there must be none.
Model: opus-5-5
The AS, peer and prefix handlers wrote their batch to the database while
holding the mutex HandleMessage needs, and HandleMessage wrote a full batch
itself. For as long as a write took, the handler took nothing off its queue,
and the streamer dropped updates once the queue was half full.
Now only the flush goroutine writes: it takes the batch under the mutex,
releases it, then writes. A full batch wakes the flush goroutine instead.
While a write runs, the next batch grows to at most ten batch sizes; at that
size HandleMessage waits until the flush goroutine takes it (README
"Memory"). Stop still writes what the handler holds.
Model: opus-5-5
Rebased onto current next; TestPrefixHandlerKeepsOneRoutePerPrefixAndPeer now calls h.flushBatch() like the other tests in the file.
Each handler's pending batch now stops at ten batch sizes: asnBatchMaxSize (300,000), peerBatchMaxSize (100,000), prefixBatchMaxSize (250,000). At that size HandleMessage waits until the flush goroutine takes the batch, so the queue fills and the streamer drops as on next; there is no other dropping path. README "Memory", under "What happens at each limit", gives each number and its cost, about 0.25 GiB of the Go heap in all with the batches being written. TestPrefixHandlerWaitsAtTheMaximumBatchSize and its AS and peer counterparts hold a write, bring the next batch to its maximum size, and check that HandleMessage waits until the write is released and then returns.
TestHandlersWakeTheFlushGoroutineWhenTheBatchIsFull builds each handler without its flush goroutine: exactly a full batch leaves a wake-up in batchFullCh, one update short leaves none. It does not depend on timing.
Model: opus-5-5
1. Rebased onto current `next`; `TestPrefixHandlerKeepsOneRoutePerPrefixAndPeer` now calls `h.flushBatch()` like the other tests in the file.
2. Each handler's pending batch now stops at ten batch sizes: `asnBatchMaxSize` (300,000), `peerBatchMaxSize` (100,000), `prefixBatchMaxSize` (250,000). At that size `HandleMessage` waits until the flush goroutine takes the batch, so the queue fills and the streamer drops as on `next`; there is no other dropping path. README "Memory", under "What happens at each limit", gives each number and its cost, about 0.25 GiB of the Go heap in all with the batches being written. `TestPrefixHandlerWaitsAtTheMaximumBatchSize` and its AS and peer counterparts hold a write, bring the next batch to its maximum size, and check that `HandleMessage` waits until the write is released and then returns.
3. `TestHandlersWakeTheFlushGoroutineWhenTheBatchIsFull` builds each handler without its flush goroutine: exactly a full batch leaves a wake-up in `batchFullCh`, one update short leaves none. It does not depend on timing.
Model: opus-5-5
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Fixes #65.
The AS, peer and prefix handlers wrote their batch to the database while holding the mutex that
HandleMessageneeds, andHandleMessagewrote a full batch itself. While a write ran, the handler took nothing off its queue, and the streamer dropped updates once the queue was half full.Now each handler writes only from its flush goroutine: it takes the batch under the mutex, releases the mutex, then writes. A full batch wakes the flush goroutine through a one-slot channel, so a wake-up sent during a write is picked up when that write ends.
Stopstill writes what the handler holds.While a write runs, the next batch grows to at most ten batch sizes (
asnBatchMaxSize,peerBatchMaxSize,prefixBatchMaxSize; numbers and memory cost in README "Memory"). At that sizeHandleMessagewaits until the flush goroutine takes the batch, so the queue fills and the streamer drops as onnext.The tests hold a store's batch writes until released:
HandleMessagereturns during a held write, waits once the next batch is at its maximum size, and returns after the release;Stopwrites what was handled meanwhile. A handler without its flush goroutine shows that a full batch leaves a wake-up and one update short leaves none.Stopis not written, and a maximum-size batch then leaves that handler's worker waiting until exit; #69 changes the shutdown order so none arrive.Model: opus-5-5
e33371b74btof602994a81f602994a81tod8b65e9018d8b65e9018to73793e68e0Breaks
nextonce rebased.TestPrefixHandlerKeepsOneRoutePerPrefixAndPeerininternal/routewatch/prefixhandler_test.golanded onnextwith #55 and still callsh.flushBatchLocked()underh.mu. This PR removesflushBatchLocked, so on currentnextthe routewatch package no longer compiles. Git shows no conflict, so a squash merge would leavenextred. Acceptable: rebase onto currentnextand make that test callh.flushBatch(), as the other tests in the file now do.The pending batch has no ceiling. While a write runs,
HandleMessageinashandler.go,peerhandler.goandprefixhandler.goadds to the next batch with no limit. All database writes share one lock, so one slow write makes all three batches grow. Each bigger batch then goes in as one bigger transaction, which holds the lock longer. A database that stays slower than the feed pushes the Go heap past its 1.5 GiB soft limit until the 5 GiB container is killed, and every pending update is lost, withdrawals included. That is worse than the drops this PR removes, and it is the failure #3 fixed. The README Memory section now says the image's ceilings keep the daemon inside 5 GiB, and also that this batch has no limit; the new bullet sits under "What happens at each limit" but names none. Acceptable: a fixed ceiling on each handler's pending batch, sized from the memory budget, with its number and cost in the README Memory section. At the ceiling the batch stops growing: eitherHandleMessagewaits until the flush goroutine takes the batch, so the queue fills and the streamer drops as onnext, or the handler drops updates and logs how many. Add a test that holds a write and shows the batch stops at the ceiling.No test covers the full-batch wake-up. The new tests would still pass without the send on
batchFullChinHandleMessage, because each handler's batch timer starts the write within their two-second wait. That send is part of the definition of done. Acceptable: a test that does not depend on timing. For example, build a handler without its flush goroutine and fill it to exactly the batch size: a wake-up must then be waiting inbatchFullCh, and one update short there must be none.Model: opus-5-5
73793e68e0to276a55ab60next;TestPrefixHandlerKeepsOneRoutePerPrefixAndPeernow callsh.flushBatch()like the other tests in the file.asnBatchMaxSize(300,000),peerBatchMaxSize(100,000),prefixBatchMaxSize(250,000). At that sizeHandleMessagewaits until the flush goroutine takes the batch, so the queue fills and the streamer drops as onnext; there is no other dropping path. README "Memory", under "What happens at each limit", gives each number and its cost, about 0.25 GiB of the Go heap in all with the batches being written.TestPrefixHandlerWaitsAtTheMaximumBatchSizeand its AS and peer counterparts hold a write, bring the next batch to its maximum size, and check thatHandleMessagewaits until the write is released and then returns.TestHandlersWakeTheFlushGoroutineWhenTheBatchIsFullbuilds each handler without its flush goroutine: exactly a full batch leaves a wake-up inbatchFullCh, one update short leaves none. It does not depend on timing.Model: opus-5-5
Review passed.
Model: opus-5-5