internal/streamer: queueMessage drops an update only when the handler's queue is full. The random drop from half full is gone: shouldDropForBackpressure, its two constants and the random number generator it used.
internal/routewatch: the prefix handler's queue holds 100,000 messages instead of 20,000. Its doc says why. While Overview or PeersDown walks the table, the handler applies updates more slowly. A backlog after a pause in the feed can arrive faster than that.
Why
In the run described in #229, the prefix handler's queue never filled, yet about 10,000 updates were dropped. The plan in #229 (comment) traces this to the random drop while Overview walked the table.
README sentences changed
"Memory", the bullet on the handler queues: the size of each queue.
"Feed handling": the queue size in the first paragraph, and the half-full paragraph, which now gives the full-queue rule.
Worth knowing
TestPrefixHandlerTakesMessagesWhileAReturningPeerSendsItsTable sizes its burst from this queue, so it now sends 100,001 messages.
Deviation: the new test, TestUpdatesDroppedOnlyWhenTheQueueIsFull, runs no worker and reads the queue itself, as the file's other queue tests do. The plan had a handler that waits on a channel. Its queue holds 1,000, not 20,000. Both keep the test exact and quick.
Unverified: whether 100,000 is enough. The next run's high-water mark shows it.
Left as is: README "Memory" puts the rest of the heap at about 0.5 GiB. The larger queue adds about 80 MiB to it.
Model: opus-5-5
**What changed**
- `internal/streamer`: `queueMessage` drops an update only when the handler's queue is full. The random drop from half full is gone: `shouldDropForBackpressure`, its two constants and the random number generator it used.
- `internal/routewatch`: the prefix handler's queue holds 100,000 messages instead of 20,000. Its doc says why. While `Overview` or `PeersDown` walks the table, the handler applies updates more slowly. A backlog after a pause in the feed can arrive faster than that.
**Why**
In the run described in https://git.eeqj.de/sneak/routewatch/issues/229, the prefix handler's queue never filled, yet about 10,000 updates were dropped. The plan in https://git.eeqj.de/sneak/routewatch/issues/229#issuecomment-164902 traces this to the random drop while `Overview` walked the table.
**README sentences changed**
- "Memory", the bullet on the handler queues: the size of each queue.
- "Feed handling": the queue size in the first paragraph, and the half-full paragraph, which now gives the full-queue rule.
**Worth knowing**
- `TestPrefixHandlerTakesMessagesWhileAReturningPeerSendsItsTable` sizes its burst from this queue, so it now sends 100,001 messages.
- Deviation: the new test, `TestUpdatesDroppedOnlyWhenTheQueueIsFull`, runs no worker and reads the queue itself, as the file's other queue tests do. The plan had a handler that waits on a channel. Its queue holds 1,000, not 20,000. Both keep the test exact and quick.
- Unverified: whether 100,000 is enough. The next run's high-water mark shows it.
- Left as is: README "Memory" puts the rest of the heap at about 0.5 GiB. The larger queue adds about 80 MiB to it.
Model: opus-5-5
internal/streamer/streamer.go line 124, the doc of Streamer: "It never waits for room on a queue (see queueMessage)." is false. Stop waits for room on each queue to hand over the held STATE messages, until its context ends (line 272). Acceptable: a sentence that is true of the read loop only, for example "Its read loop never waits for room on a queue (see queueMessage)."
internal/streamer/streamer.go lines 887-891, the doc of queueMessage: the change re-wraps "Held messages go into the queue, oldest first, before anything else for the handler, as soon as there is room, or when Stop is called; while any is still held, the handler's other messages are dropped, so none of them overtakes a held down." It runs to 44 words, joins two ideas with a semicolon, and "a held down" is a coined short form. That breaks the sentence rules of #157. Acceptable: short sentences, one idea each. One says when held messages go into the queue. One says the handler's other messages are dropped while any is held, so none overtakes a held STATE message.
Model: opus-5-5
**FAIL: needs-rework.**
1. `internal/streamer/streamer.go` line 124, the doc of `Streamer`: "It never waits for room on a queue (see queueMessage)." is false. `Stop` waits for room on each queue to hand over the held `STATE` messages, until its context ends (line 272). Acceptable: a sentence that is true of the read loop only, for example "Its read loop never waits for room on a queue (see queueMessage)."
2. `internal/streamer/streamer.go` lines 887-891, the doc of `queueMessage`: the change re-wraps "Held messages go into the queue, oldest first, before anything else for the handler, as soon as there is room, or when Stop is called; while any is still held, the handler's other messages are dropped, so none of them overtakes a held down." It runs to 44 words, joins two ideas with a semicolon, and "a held down" is a coined short form. That breaks the sentence rules of https://git.eeqj.de/sneak/routewatch/issues/157. Acceptable: short sentences, one idea each. One says when held messages go into the queue. One says the handler's other messages are dropped while any is held, so none overtakes a held `STATE` message.
Model: opus-5-5
The streamer dropped a random share of a handler's updates once its
queue was half full. When the feed paused and then sent its backlog
while the routing table's Overview walked every prefix, the prefix
handler lost about 10,000 updates with its queue never full.
The streamer now drops an update only when the queue is full. The
prefix handler's queue holds 100,000 messages, about 100 MiB, so it
can hold such a backlog. The README's Memory and Feed handling
sections give the new sizes and the rule.
Model: opus-5-5
Rework for #230 (comment), both in internal/streamer/streamer.go, comments only:
Took the review's form: the doc of Streamer now says "Its read loop never waits for room on a queue."
Took the review's form: the doc of queueMessage is now short sentences. One says when held messages go into the queue. One says the handler's other messages are dropped while any message is held, so none of them overtakes a held STATE message.
Disclosures:
Judgement call: held messages go in "when a later message for the handler finds room", not "as soon as there is room". Nothing moves them until the handler's next message arrives.
Also changed, in the same two comments: the sentence that joined the drop rule and the reason for the STATE exception with a colon is now two sentences. The comma before "and" in the doc of Streamer is gone.
Model: opus-5-5
Rework for https://git.eeqj.de/sneak/routewatch/pulls/230#issuecomment-164947, both in `internal/streamer/streamer.go`, comments only:
1. Took the review's form: the doc of `Streamer` now says "Its read loop never waits for room on a queue."
2. Took the review's form: the doc of `queueMessage` is now short sentences. One says when held messages go into the queue. One says the handler's other messages are dropped while any message is held, so none of them overtakes a held `STATE` message.
Disclosures:
- Judgement call: held messages go in "when a later message for the handler finds room", not "as soon as there is room". Nothing moves them until the handler's next message arrives.
- Also changed, in the same two comments: the sentence that joined the drop rule and the reason for the `STATE` exception with a colon is now two sentences. The comma before "and" in the doc of `Streamer` is gone.
Model: opus-5-5
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
What changed
internal/streamer:queueMessagedrops an update only when the handler's queue is full. The random drop from half full is gone:shouldDropForBackpressure, its two constants and the random number generator it used.internal/routewatch: the prefix handler's queue holds 100,000 messages instead of 20,000. Its doc says why. WhileOvervieworPeersDownwalks the table, the handler applies updates more slowly. A backlog after a pause in the feed can arrive faster than that.Why
In the run described in #229, the prefix handler's queue never filled, yet about 10,000 updates were dropped. The plan in #229 (comment) traces this to the random drop while
Overviewwalked the table.README sentences changed
Worth knowing
TestPrefixHandlerTakesMessagesWhileAReturningPeerSendsItsTablesizes its burst from this queue, so it now sends 100,001 messages.TestUpdatesDroppedOnlyWhenTheQueueIsFull, runs no worker and reads the queue itself, as the file's other queue tests do. The plan had a handler that waits on a channel. Its queue holds 1,000, not 20,000. Both keep the test exact and quick.Model: opus-5-5
FAIL: needs-rework.
internal/streamer/streamer.goline 124, the doc ofStreamer: "It never waits for room on a queue (see queueMessage)." is false.Stopwaits for room on each queue to hand over the heldSTATEmessages, until its context ends (line 272). Acceptable: a sentence that is true of the read loop only, for example "Its read loop never waits for room on a queue (see queueMessage)."internal/streamer/streamer.golines 887-891, the doc ofqueueMessage: the change re-wraps "Held messages go into the queue, oldest first, before anything else for the handler, as soon as there is room, or when Stop is called; while any is still held, the handler's other messages are dropped, so none of them overtakes a held down." It runs to 44 words, joins two ideas with a semicolon, and "a held down" is a coined short form. That breaks the sentence rules of #157. Acceptable: short sentences, one idea each. One says when held messages go into the queue. One says the handler's other messages are dropped while any is held, so none overtakes a heldSTATEmessage.Model: opus-5-5
3d44848af2to49d2f4ede8Rework for #230 (comment), both in
internal/streamer/streamer.go, comments only:Streamernow says "Its read loop never waits for room on a queue."queueMessageis now short sentences. One says when held messages go into the queue. One says the handler's other messages are dropped while any message is held, so none of them overtakes a heldSTATEmessage.Disclosures:
STATEexception with a colon is now two sentences. The comma before "and" in the doc ofStreameris gone.Model: opus-5-5
PASS
Model: opus-5-5