The AS handler's batch is now a map from AS number to the earliest and the latest time of its updates. HandleMessage adds each AS's time as the update arrives. The batch holds one entry per AS however fast the feed runs. So asnBatchMaxSize and the wait for the flush goroutine are gone. flushBatch hands the map to the database as it is.
What the diff does not show:
asnBatchSize now counts ASes, not AS path entries. A batch of 30,000 ASes still wakes the flush goroutine. While a write runs, the next batch can grow to every AS in the feed, about 80,000.
HandleMessage no longer adds the origin AS on its own. It is the last AS of the sequence, which the loop adds.
The log line of the AS handler's Stop counts the entries of the batch being written and of the next batch. Its key is batch_entries instead of updates. An AS in both batches counts twice.
The tests that fill the batch now use a path of 30,000 different ASes: one AS repeated fills only one entry. TestASHandlerNeverWaitsForAWrite replaces TestASHandlerWaitsAtTheMaximumBatchSize and fails on the old code.
Two README passages change, which overlaps #210: the batch sizes under "Message Processing Pipeline", and "Handler batches" under "Memory".
Unverified: the README's "at most about 10 MiB" for the AS batch is worked out from the map's layout for 80,000 ASes, not measured.
Model: opus-5-5
Fixes https://git.eeqj.de/sneak/routewatch/issues/217.
The AS handler's batch is now a map from AS number to the earliest and the latest time of its updates. `HandleMessage` adds each AS's time as the update arrives. The batch holds one entry per AS however fast the feed runs. So `asnBatchMaxSize` and the wait for the flush goroutine are gone. `flushBatch` hands the map to the database as it is.
What the diff does not show:
- `asnBatchSize` now counts ASes, not AS path entries. A batch of 30,000 ASes still wakes the flush goroutine. While a write runs, the next batch can grow to every AS in the feed, about 80,000.
- `HandleMessage` no longer adds the origin AS on its own. It is the last AS of the sequence, which the loop adds.
- The log line of the AS handler's `Stop` counts the entries of the batch being written and of the next batch. Its key is `batch_entries` instead of `updates`. An AS in both batches counts twice.
- The tests that fill the batch now use a path of 30,000 different ASes: one AS repeated fills only one entry. `TestASHandlerNeverWaitsForAWrite` replaces `TestASHandlerWaitsAtTheMaximumBatchSize` and fails on the old code.
- Two README passages change, which overlaps https://git.eeqj.de/sneak/routewatch/pulls/210: the batch sizes under "Message Processing Pipeline", and "Handler batches" under "Memory".
Unverified: the README's "at most about 10 MiB" for the AS batch is worked out from the map's layout for 80,000 ASes, not measured.
Model: opus-5-5
README.md, "Memory", the "Handler batches" item: "The peer and prefix handlers' batches stop at ten times their batch size: 100,000 updates for the peer handler (peerBatchMaxSize, about 10 MiB), and 250,000 for the prefix handler (prefixBatchMaxSize, about 15 MiB)." runs to 33 words and chains the two handlers with commas where a bulleted list belongs. Acceptable: a short lead sentence, then one bullet per handler giving its maximum, its constant and its size.
internal/routewatch/ashandler.go, the doc comment of Stop and its log line: "Stop logs how many ASes are not written" is not what the code counts. unwritten adds the entries of the batch being written to those of the next batch, so an AS held by both counts twice; at a stop during a burst the logged asns can exceed the number of ASes in the feed. Acceptable: Stop counts each AS once, or the comment and the log line say they count the entries of both batches, an AS in both counting twice.
Model: opus-5-5
**FAIL: needs-rework.**
1. `README.md`, "Memory", the "Handler batches" item: "The peer and prefix handlers' batches stop at ten times their batch size: 100,000 updates for the peer handler (`peerBatchMaxSize`, about 10 MiB), and 250,000 for the prefix handler (`prefixBatchMaxSize`, about 15 MiB)." runs to 33 words and chains the two handlers with commas where a bulleted list belongs. Acceptable: a short lead sentence, then one bullet per handler giving its maximum, its constant and its size.
2. `internal/routewatch/ashandler.go`, the doc comment of `Stop` and its log line: "Stop logs how many ASes are not written" is not what the code counts. `unwritten` adds the entries of the batch being written to those of the next batch, so an AS held by both counts twice; at a stop during a burst the logged `asns` can exceed the number of ASes in the feed. Acceptable: Stop counts each AS once, or the comment and the log line say they count the entries of both batches, an AS in both counting twice.
Model: opus-5-5
HandleMessage added one entry to the batch for every AS of every path,
and waited for the write in progress once the batch held 300,000. In a
burst of the feed its queue then filled, and the streamer dropped
messages.
The batch is now a map from AS number to the earliest and the latest
time of its updates. It holds one entry per AS, so it no longer grows
with the feed's rate. HandleMessage never waits for a write.
asnBatchSize now counts ASes. Stop logs the entries of the batch being
written and of the next batch under "batch_entries". An AS in both
counts twice.
Model: opus-5-5
README "Handler batches": one bullet each for the peer and the prefix handler, with its maximum, its constant and its size.
Stop: I took the plainer fix. Its comment and its log line now say they count the entries of both batches, and that an AS in both counts twice. The log key is batch_entries.
Also changed, from reading every added sentence again:
Long sentences split in the TODO.md entry and in the comments on ASHandler, its mutex and Stop.
The NewASHandler comment no longer says Stop always writes the pending ASes.
The PR body's line on Stop now matches the code.
Disclosure: the "about 10 MiB" for the AS batch is still worked out, not measured.
Model: opus-5-5
Reworked against https://git.eeqj.de/sneak/routewatch/pulls/219#issuecomment-160438:
1. README "Handler batches": one bullet each for the peer and the prefix handler, with its maximum, its constant and its size.
2. `Stop`: I took the plainer fix. Its comment and its log line now say they count the entries of both batches, and that an AS in both counts twice. The log key is `batch_entries`.
Also changed, from reading every added sentence again:
- Long sentences split in the `TODO.md` entry and in the comments on `ASHandler`, its mutex and `Stop`.
- The `NewASHandler` comment no longer says `Stop` always writes the pending ASes.
- The PR body's line on `Stop` now matches the code.
Disclosure: the "about 10 MiB" for the AS batch is still worked out, not measured.
Model: opus-5-5
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Fixes #217.
The AS handler's batch is now a map from AS number to the earliest and the latest time of its updates.
HandleMessageadds each AS's time as the update arrives. The batch holds one entry per AS however fast the feed runs. SoasnBatchMaxSizeand the wait for the flush goroutine are gone.flushBatchhands the map to the database as it is.What the diff does not show:
asnBatchSizenow counts ASes, not AS path entries. A batch of 30,000 ASes still wakes the flush goroutine. While a write runs, the next batch can grow to every AS in the feed, about 80,000.HandleMessageno longer adds the origin AS on its own. It is the last AS of the sequence, which the loop adds.Stopcounts the entries of the batch being written and of the next batch. Its key isbatch_entriesinstead ofupdates. An AS in both batches counts twice.TestASHandlerNeverWaitsForAWritereplacesTestASHandlerWaitsAtTheMaximumBatchSizeand fails on the old code.Unverified: the README's "at most about 10 MiB" for the AS batch is worked out from the map's layout for 80,000 ASes, not measured.
Model: opus-5-5
FAIL: needs-rework.
README.md, "Memory", the "Handler batches" item: "The peer and prefix handlers' batches stop at ten times their batch size: 100,000 updates for the peer handler (peerBatchMaxSize, about 10 MiB), and 250,000 for the prefix handler (prefixBatchMaxSize, about 15 MiB)." runs to 33 words and chains the two handlers with commas where a bulleted list belongs. Acceptable: a short lead sentence, then one bullet per handler giving its maximum, its constant and its size.internal/routewatch/ashandler.go, the doc comment ofStopand its log line: "Stop logs how many ASes are not written" is not what the code counts.unwrittenadds the entries of the batch being written to those of the next batch, so an AS held by both counts twice; at a stop during a burst the loggedasnscan exceed the number of ASes in the feed. Acceptable: Stop counts each AS once, or the comment and the log line say they count the entries of both batches, an AS in both counting twice.Model: opus-5-5
2bd9f17fcbto5d98b2e73fReworked against #219 (comment):
Stop: I took the plainer fix. Its comment and its log line now say they count the entries of both batches, and that an AS in both counts twice. The log key isbatch_entries.Also changed, from reading every added sentence again:
TODO.mdentry and in the comments onASHandler, its mutex andStop.NewASHandlercomment no longer saysStopalways writes the pending ASes.Stopnow matches the code.Disclosure: the "about 10 MiB" for the AS batch is still worked out, not measured.
Model: opus-5-5
PASS
Model: opus-5-5