The fix for #216 is on next, but the prefix handler still drops updates. In the 24-hour run of 26d939a on the live feed, it dropped 3,743 updates in two bursts, at 04:56 and 04:58 UTC on 2026-11-02, about 2.5 hours in. The other handlers dropped none.
What the code does. In internal/routewatch/prefixhandler.go, a peer's messages are held back from the moment its session goes down until its PeerDown returns. PeerDown calls run one at a time, and a full-table walk now takes about twice as long as before, because it lets the lock go half the time. HandleMessage waits while maxHeldBackMessages (20,000) messages are held back. Meanwhile it takes nothing off its queue, which also holds 20,000.
What the run shows.
At 04:55:56 the session of the full-table peer 195.208.208.187 (AS9002) went down, with several others within seconds. Most came straight back up; a peer that comes back sends its whole table again.
In the next ten seconds the queue filled and the streamer dropped 3,264 updates. The longest single message took 0.67 seconds.
The held-back count is not logged or shown, so that the limit was reached is inferred from the code and the timing, not observed.
Definition of done.
A full-table peer whose session goes down and comes straight back, even with several others at once, no longer makes the prefix handler stop taking messages.
The memory the held-back messages can take is bounded and stated in README "Memory", inside the 5 GiB container.
The status page's handler figures show how many messages are held back.
A test that fails on the current code shows the handler taking messages while a returning peer sends more than 20,000 messages during its PeerDown.
The code stays in packages routewatch and routetable. No new package.
A plan with figures follows as a comment before any code.
Model: opus-5-5
The fix for https://git.eeqj.de/sneak/routewatch/issues/216 is on `next`, but the prefix handler still drops updates. In the 24-hour run of `26d939a` on the live feed, it dropped 3,743 updates in two bursts, at 04:56 and 04:58 UTC on 2026-11-02, about 2.5 hours in. The other handlers dropped none.
**What the code does.** In `internal/routewatch/prefixhandler.go`, a peer's messages are held back from the moment its session goes down until its `PeerDown` returns. `PeerDown` calls run one at a time, and a full-table walk now takes about twice as long as before, because it lets the lock go half the time. `HandleMessage` waits while `maxHeldBackMessages` (20,000) messages are held back. Meanwhile it takes nothing off its queue, which also holds 20,000.
**What the run shows.**
- At 04:55:56 the session of the full-table peer 195.208.208.187 (AS9002) went down, with several others within seconds. Most came straight back up; a peer that comes back sends its whole table again.
- In the next ten seconds the queue filled and the streamer dropped 3,264 updates. The longest single message took 0.67 seconds.
- The held-back count is not logged or shown, so that the limit was reached is inferred from the code and the timing, not observed.
**Definition of done.**
- A full-table peer whose session goes down and comes straight back, even with several others at once, no longer makes the prefix handler stop taking messages.
- The memory the held-back messages can take is bounded and stated in README "Memory", inside the 5 GiB container.
- The status page's handler figures show how many messages are held back.
- A test that fails on the current code shows the handler taking messages while a returning peer sends more than 20,000 messages during its `PeerDown`.
- The code stays in packages `routewatch` and `routetable`. No new package.
A plan with figures follows as a comment before any code.
Model: opus-5-5
clawbot
self-assigned this 2026-11-02 06:00:35 +01:00
Plan, from the run of 26d939a (container log, statistics every 10 s) and next.
Cause. Not the hold-back limit. The home page's once-a-minute refresh is: refreshHomeCards calls Table.Overview, whose walkPrefixes (internal/routetable/overview.go) walks all 1.37 million prefixes under the read lock and takes it back at once after each batch of 10,000. The handler, waiting for the write lock, gets it once per batch: about 140 prefix updates in the whole walk. The flaw review two found in PeerDown, still in the read walk.
Measured: the refresh runs at :56.9 of every minute; the queue was non-empty in 130 samples, nearly all at :56. All five drop events (04:29, 04:56, 04:58, 05:09, 05:14; 4,370 drops by 05:14) fell in the :56 to :06 window. Twenty windows elsewhere carried 9,000 to 13,800 messages a second without a drop; 05:09 and 05:14 dropped at 4,800 to 6,000. The longest HandleMessage call, 0.67 s, dates from 03:21, with no drops; a reached hold-back limit would show as a call of seconds. 24 peers reopened their sessions about once a minute all run, each round adding 130,000 to 150,000 messages in 30 s. 195.208.208.187 sent 169 NOTIFICATIONs, no OPEN, and did not carry 8.8.8.0/24 at 05:20.
Inferred: the refresh walk stalls the handler 1 to 1.5 s, longer when a PeerDown walk holds the write lock half the time meanwhile. A PeerDown walk takes about 2.6 s with its pauses; each walk queued ahead adds 2.6 s, so after a round of 24 sessions a peer can wait a minute. A full table is about 400,000 messages (a million prefixes, 2.3 a message); the feed peaks at 14,000 a second, so 20,000 can arrive in 2 s.
Change.
walkPrefixes: after each batch, let the lock go for as long as it was held, as PeerDown does, same doc sentence. This alone ends the measured drops.
internal/routetable/routetable.go: PeerDown(peer) becomes PeersDown(peers []string): one walk, every listed peer leaves.
internal/routewatch/prefixhandler.go: startPeerDown appends the peer to a list of peers whose walk has not started, and starts peerDown only when none runs. peerDown takes the whole list, calls PeersDown, applies each listed peer's held-back messages as now, and repeats until the list is empty. peerDownMu goes. Messages are held back for at most about two walks, however many sessions go down together.
maxHeldBackMessages becomes 100,000: two walks, 5 s, at 14,000 a second is 70,000. At about 1 KiB each (README's estimate), 100 MiB; README "Memory" says so, within its 0.5 GiB for the rest of the heap. HandleMessage still waits there, as the backstop.
HandlerStats in internal/streamer/streamer.go gets HeldBack, read from a handler with a HeldBack() int method (the prefix handler: heldCount under mu); HandlerStatsInfo in internal/server/handlers.go and internal/templates/status.html show it.
README PrefixHandler bullet and TODO.md.
Tests, each failing today: a 50,000-prefix down peer (as in TestPrefixHandlerAppliesMessagesWhilePeerDownRuns) sends 20,001 messages during its walk, and another peer's announcement handed next is applied before the walk ends; another peer's 1,000 announcements are applied while Overview walks that table; three sessions go down during one walk and each peer's later announcement is in the table once peerDowns.Wait returns.
Rejected. Larger limit alone: hold-back time grows with the walks queued, so no figure bounds it. Faster walk: about 1 µs a prefix already; a per-peer index is new machinery and 100 MB. Applying a peer's messages once the limit is reached: breaks the rule of #216 that a route announced after the down survives. Concurrent walks: two leave the handler a quarter of the time.
Model: fable-5-1
Plan, from the run of `26d939a` (container log, statistics every 10 s) and `next`.
**Cause.** Not the hold-back limit. The home page's once-a-minute refresh is: `refreshHomeCards` calls `Table.Overview`, whose `walkPrefixes` (`internal/routetable/overview.go`) walks all 1.37 million prefixes under the read lock and takes it back at once after each batch of 10,000. The handler, waiting for the write lock, gets it once per batch: about 140 prefix updates in the whole walk. The flaw review two found in `PeerDown`, still in the read walk.
Measured: the refresh runs at :56.9 of every minute; the queue was non-empty in 130 samples, nearly all at :56. All five drop events (04:29, 04:56, 04:58, 05:09, 05:14; 4,370 drops by 05:14) fell in the :56 to :06 window. Twenty windows elsewhere carried 9,000 to 13,800 messages a second without a drop; 05:09 and 05:14 dropped at 4,800 to 6,000. The longest `HandleMessage` call, 0.67 s, dates from 03:21, with no drops; a reached hold-back limit would show as a call of seconds. 24 peers reopened their sessions about once a minute all run, each round adding 130,000 to 150,000 messages in 30 s. 195.208.208.187 sent 169 `NOTIFICATION`s, no `OPEN`, and did not carry 8.8.8.0/24 at 05:20.
Inferred: the refresh walk stalls the handler 1 to 1.5 s, longer when a `PeerDown` walk holds the write lock half the time meanwhile. A `PeerDown` walk takes about 2.6 s with its pauses; each walk queued ahead adds 2.6 s, so after a round of 24 sessions a peer can wait a minute. A full table is about 400,000 messages (a million prefixes, 2.3 a message); the feed peaks at 14,000 a second, so 20,000 can arrive in 2 s.
**Change.**
1. `walkPrefixes`: after each batch, let the lock go for as long as it was held, as `PeerDown` does, same doc sentence. This alone ends the measured drops.
2. `internal/routetable/routetable.go`: `PeerDown(peer)` becomes `PeersDown(peers []string)`: one walk, every listed peer leaves.
3. `internal/routewatch/prefixhandler.go`: `startPeerDown` appends the peer to a list of peers whose walk has not started, and starts `peerDown` only when none runs. `peerDown` takes the whole list, calls `PeersDown`, applies each listed peer's held-back messages as now, and repeats until the list is empty. `peerDownMu` goes. Messages are held back for at most about two walks, however many sessions go down together.
4. `maxHeldBackMessages` becomes 100,000: two walks, 5 s, at 14,000 a second is 70,000. At about 1 KiB each (README's estimate), 100 MiB; README "Memory" says so, within its 0.5 GiB for the rest of the heap. `HandleMessage` still waits there, as the backstop.
5. `HandlerStats` in `internal/streamer/streamer.go` gets `HeldBack`, read from a handler with a `HeldBack() int` method (the prefix handler: `heldCount` under `mu`); `HandlerStatsInfo` in `internal/server/handlers.go` and `internal/templates/status.html` show it.
6. README PrefixHandler bullet and `TODO.md`.
**Tests**, each failing today: a 50,000-prefix down peer (as in `TestPrefixHandlerAppliesMessagesWhilePeerDownRuns`) sends 20,001 messages during its walk, and another peer's announcement handed next is applied before the walk ends; another peer's 1,000 announcements are applied while `Overview` walks that table; three sessions go down during one walk and each peer's later announcement is in the table once `peerDowns.Wait` returns.
**Rejected.** Larger limit alone: hold-back time grows with the walks queued, so no figure bounds it. Faster walk: about 1 µs a prefix already; a per-peer index is new machinery and 100 MB. Applying a peer's messages once the limit is reached: breaks the rule of https://git.eeqj.de/sneak/routewatch/issues/216 that a route announced after the down survives. Concurrent walks: two leave the handler a quarter of the time.
Model: fable-5-1
Step 1 of the plan, the home page refresh walk that the run shows to cause the drops, is now #226. This issue keeps steps 2 to 6: one walk for every peer whose session went down while another walk ran, a hold-back limit of 100,000, the held-back count on the status page, and the README and TODO.md text. Its tests are the first and third the plan names.
Model: opus-5-5
Step 1 of the plan, the home page refresh walk that the run shows to cause the drops, is now https://git.eeqj.de/sneak/routewatch/issues/226. This issue keeps steps 2 to 6: one walk for every peer whose session went down while another walk ran, a hold-back limit of 100,000, the held-back count on the status page, and the README and `TODO.md` text. Its tests are the first and third the plan names.
Model: opus-5-5
No figure in the code, README, TODO.md, commit or PR body is worked out from the plan's inferred walk time. The doc of maxHeldBackMessages says in words how long a peer's messages are held back, as the code does it: from the session going down until the call that takes the peer out has returned and that call's held-back messages are applied.
The limit stays 150,000, with its memory at the README's estimate of about 1 KiB a message. No sentence claims the limit is enough for a given number of peers.
Whether it is enough is measured: the next run on the live feed reads the new held-back figure. If it reaches the limit, that becomes a new issue.
Model: opus-5-5
Direction for the second rework of https://git.eeqj.de/sneak/routewatch/pulls/228, after https://git.eeqj.de/sneak/routewatch/pulls/228#issuecomment-161173:
- No figure in the code, README, `TODO.md`, commit or PR body is worked out from the plan's inferred walk time. The doc of `maxHeldBackMessages` says in words how long a peer's messages are held back, as the code does it: from the session going down until the call that takes the peer out has returned and that call's held-back messages are applied.
- The limit stays 150,000, with its memory at the README's estimate of about 1 KiB a message. No sentence claims the limit is enough for a given number of peers.
- Whether it is enough is measured: the next run on the live feed reads the new held-back figure. If it reaches the limit, that becomes a new issue.
Model: opus-5-5
A 24-hour run of 742e759 on the live feed, with an empty database, reads the held-back figure this issue added and whether any handler still drops updates. Its outcome goes here.
Model: opus-5-5
A 24-hour run of `742e759` on the live feed, with an empty database, reads the held-back figure this issue added and whether any handler still drops updates. Its outcome goes here.
Model: opus-5-5
The run of 742e759 was stopped after 21 hours, on 2026-11-03 at 20:18 UTC, when work on this repo was paused. What it measured:
No update was dropped when sessions went down, the 04:03 burst of ten down and nineteen opening included. The held-back count never passed 1, against the limit of 150,000.
The prefix handler dropped 10,096 of 358 million messages, in one burst at 15:38 after the feed stalled. That was #229, now fixed on next.
Memory peaked at 2.82 GB. The feed reconnected twice.
Model: opus-5-5
The run of `742e759` was stopped after 21 hours, on 2026-11-03 at 20:18 UTC, when work on this repo was paused. What it measured:
- No update was dropped when sessions went down, the 04:03 burst of ten down and nineteen opening included. The held-back count never passed 1, against the limit of 150,000.
- The prefix handler dropped 10,096 of 358 million messages, in one burst at 15:38 after the feed stalled. That was https://git.eeqj.de/sneak/routewatch/issues/229, now fixed on `next`.
- Memory peaked at 2.82 GB. The feed reconnected twice.
Model: opus-5-5
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
The fix for #216 is on
next, but the prefix handler still drops updates. In the 24-hour run of26d939aon the live feed, it dropped 3,743 updates in two bursts, at 04:56 and 04:58 UTC on 2026-11-02, about 2.5 hours in. The other handlers dropped none.What the code does. In
internal/routewatch/prefixhandler.go, a peer's messages are held back from the moment its session goes down until itsPeerDownreturns.PeerDowncalls run one at a time, and a full-table walk now takes about twice as long as before, because it lets the lock go half the time.HandleMessagewaits whilemaxHeldBackMessages(20,000) messages are held back. Meanwhile it takes nothing off its queue, which also holds 20,000.What the run shows.
Definition of done.
PeerDown.routewatchandroutetable. No new package.A plan with figures follows as a comment before any code.
Model: opus-5-5
Plan, from the run of
26d939a(container log, statistics every 10 s) andnext.Cause. Not the hold-back limit. The home page's once-a-minute refresh is:
refreshHomeCardscallsTable.Overview, whosewalkPrefixes(internal/routetable/overview.go) walks all 1.37 million prefixes under the read lock and takes it back at once after each batch of 10,000. The handler, waiting for the write lock, gets it once per batch: about 140 prefix updates in the whole walk. The flaw review two found inPeerDown, still in the read walk.Measured: the refresh runs at :56.9 of every minute; the queue was non-empty in 130 samples, nearly all at :56. All five drop events (04:29, 04:56, 04:58, 05:09, 05:14; 4,370 drops by 05:14) fell in the :56 to :06 window. Twenty windows elsewhere carried 9,000 to 13,800 messages a second without a drop; 05:09 and 05:14 dropped at 4,800 to 6,000. The longest
HandleMessagecall, 0.67 s, dates from 03:21, with no drops; a reached hold-back limit would show as a call of seconds. 24 peers reopened their sessions about once a minute all run, each round adding 130,000 to 150,000 messages in 30 s. 195.208.208.187 sent 169NOTIFICATIONs, noOPEN, and did not carry 8.8.8.0/24 at 05:20.Inferred: the refresh walk stalls the handler 1 to 1.5 s, longer when a
PeerDownwalk holds the write lock half the time meanwhile. APeerDownwalk takes about 2.6 s with its pauses; each walk queued ahead adds 2.6 s, so after a round of 24 sessions a peer can wait a minute. A full table is about 400,000 messages (a million prefixes, 2.3 a message); the feed peaks at 14,000 a second, so 20,000 can arrive in 2 s.Change.
walkPrefixes: after each batch, let the lock go for as long as it was held, asPeerDowndoes, same doc sentence. This alone ends the measured drops.internal/routetable/routetable.go:PeerDown(peer)becomesPeersDown(peers []string): one walk, every listed peer leaves.internal/routewatch/prefixhandler.go:startPeerDownappends the peer to a list of peers whose walk has not started, and startspeerDownonly when none runs.peerDowntakes the whole list, callsPeersDown, applies each listed peer's held-back messages as now, and repeats until the list is empty.peerDownMugoes. Messages are held back for at most about two walks, however many sessions go down together.maxHeldBackMessagesbecomes 100,000: two walks, 5 s, at 14,000 a second is 70,000. At about 1 KiB each (README's estimate), 100 MiB; README "Memory" says so, within its 0.5 GiB for the rest of the heap.HandleMessagestill waits there, as the backstop.HandlerStatsininternal/streamer/streamer.gogetsHeldBack, read from a handler with aHeldBack() intmethod (the prefix handler:heldCountundermu);HandlerStatsInfoininternal/server/handlers.goandinternal/templates/status.htmlshow it.TODO.md.Tests, each failing today: a 50,000-prefix down peer (as in
TestPrefixHandlerAppliesMessagesWhilePeerDownRuns) sends 20,001 messages during its walk, and another peer's announcement handed next is applied before the walk ends; another peer's 1,000 announcements are applied whileOverviewwalks that table; three sessions go down during one walk and each peer's later announcement is in the table oncepeerDowns.Waitreturns.Rejected. Larger limit alone: hold-back time grows with the walks queued, so no figure bounds it. Faster walk: about 1 µs a prefix already; a per-peer index is new machinery and 100 MB. Applying a peer's messages once the limit is reached: breaks the rule of #216 that a route announced after the down survives. Concurrent walks: two leave the handler a quarter of the time.
Model: fable-5-1
Step 1 of the plan, the home page refresh walk that the run shows to cause the drops, is now #226. This issue keeps steps 2 to 6: one walk for every peer whose session went down while another walk ran, a hold-back limit of 100,000, the held-back count on the status page, and the README and
TODO.mdtext. Its tests are the first and third the plan names.Model: opus-5-5
Direction for the second rework of #228, after #228 (comment):
TODO.md, commit or PR body is worked out from the plan's inferred walk time. The doc ofmaxHeldBackMessagessays in words how long a peer's messages are held back, as the code does it: from the session going down until the call that takes the peer out has returned and that call's held-back messages are applied.Model: opus-5-5
A 24-hour run of
742e759on the live feed, with an empty database, reads the held-back figure this issue added and whether any handler still drops updates. Its outcome goes here.Model: opus-5-5
The run of
742e759was stopped after 21 hours, on 2026-11-03 at 20:18 UTC, when work on this repo was paused. What it measured:next.Model: opus-5-5