Take every down peer out of the table in one walk (closes #225)

The prefix handler ran one PeerDown per peer, one after another, and
stopped taking messages at 20,000 held-back messages. Sessions going
down together queued up their walks, so a returning full-table peer
could reach that limit.

The table's PeersDown now takes several peers out of every prefix in
one walk. One walk runs at a time. The next one takes every peer whose
session went down meanwhile, so a message is held back for two walks at
most. Each walk takes about 2.6 seconds for each full-table peer it
takes out. The limit is now 150,000 messages, about 150 MiB: enough
when four full-table peers go down together. The status page shows how
many messages each handler holds back.

Model: opus-5-5

README.md

@@ -652,9 +652,10 @@ What happens at each limit:
 
 - Held-back messages: the prefix handler holds back a peer's messages after the
   peer's session goes down. It applies them once the peer has left every prefix.
-  It holds at most 100,000 such messages (`maxHeldBackMessages`), about 100 MiB.
-  At that number it takes nothing more off its queue until it has applied one of
-  them.
+  It holds at most 150,000 such messages (`maxHeldBackMessages`), about 150 MiB.
+  The limit is enough when the sessions of four full-table peers go down
+  together. At the limit it takes nothing more off its queue until it has
+  applied one of them.
 
 With `DEBUG=routewatch` the daemon logs a `System stats` line every 60 seconds
 with the goroutine count and Go memory figures.

TODO.md

@@ -39,10 +39,12 @@ tested.
   goes on taking messages. The table's `PeersDown` takes several peers out of
   every prefix in one walk over the table. One walk runs at a time. The next one
   takes every peer whose session went down meanwhile. So a peer's messages are
-  held back for about two walks at most. The handler holds back at most 100,000
-  messages, about 100 MiB. The status page shows how many it holds back. Each
-  peer used to have a walk of its own, and the handler stopped taking messages
-  at 20,000 held back (closes #225)
+  held back for two walks at most. Each walk takes about 2.6 seconds for each
+  full-table peer it takes out. The handler holds back at most 150,000 messages,
+  about 150 MiB. The limit is enough when the sessions of four full-table peers
+  go down together. The status page shows how many messages the handler holds
+  back. Each peer used to have a walk of its own, and the handler stopped taking
+  messages at 20,000 held back (closes #225)
 - 2026-11-02: like a failed WHOIS lookup, a lookup skipped because its RIR is
   paused makes the fetcher wait longer before the next lookup. The longest wait
   is one minute. The status page does not count a skip as a failure. A skip used

internal/routetable/routetable.go

@@ -242,7 +242,8 @@ func (t *Table) PeersDown(peers []string) {
 					if o != nil {
 						t.leave(prefix, o, number)
 
-						// leave may have removed an origin of the prefix.
+						// leave may have removed an origin of the
+						// prefix, or the prefix itself.
 						origins = prefixes[prefix]
 					}
 				}

internal/routetable/routetable_test.go

@@ -284,7 +284,6 @@ func TestWithdrawalsAndPeerDowns(t *testing.T) {
 				Routes: []routetable.Route{routeOf(peerA, asA, 10)}}},
 		},
 		{
-			// peerA's leaving removes the origin asA, before peerB leaves.
 			name: "peers going down together each leave the prefix",
 			steps: []step{announce(peerA, asA, 10), announce(peerB, asB, 20),
 				announce(peerC, asB, 30), peersDown(peerA, peerB)},
@@ -341,6 +340,32 @@ func TestPeersDownLeavesEveryPrefix(t *testing.T) {
 	}
 }
 
+// TestPeersDownTakesOutPeersOfDifferentPrefixes takes peerA and peerB out in
+// one PeersDown. peerA carries only prefixV4, and peerB only prefixV6. So
+// peerA's leaving removes prefixV4 before PeersDown looks for peerB there. Both
+// prefixes must then be gone.
+func TestPeersDownTakesOutPeersOfDifferentPrefixes(t *testing.T) {
+	t.Parallel()
+
+	table := routetable.New(routetable.Params{})
+	prefixes := []netip.Prefix{
+		netip.MustParsePrefix(prefixV4),
+		netip.MustParsePrefix(prefixV6),
+	}
+
+	table.Announce(prefixes[0], asA, routeOf(peerA, asA, 10))
+	table.Announce(prefixes[1], asB, routeOf(peerB, asB, 20))
+	table.PeersDown([]string{peerA, peerB})
+
+	for _, prefix := range prefixes {
+		got := table.Origins(prefix)
+		if got != nil {
+			t.Errorf("Origins(%s) = %+v after its only peer went down, want nil",
+				prefix, got)
+		}
+	}
+}
+
 func TestCountsOriginsWithoutKeptRoute(t *testing.T) {
 	t.Parallel()
 

internal/routewatch/prefixhandler.go

@@ -46,13 +46,19 @@ const (
 	maxTableChangesPerWrite = 20000
 
 	// maxHeldBackMessages is the most messages the handler holds back from
-	// peers whose session went down. A message is held back for about two
-	// full-table PeersDown calls at most, about 5 seconds. At its peak of
-	// about 14,000 messages a second, the feed sends about 70,000 in that
-	// time. A message takes about 1 KiB, so 100,000 take about 100 MiB
-	// (README "Memory"). At this number HandleMessage waits until one of them
-	// is applied.
-	maxHeldBackMessages = 100000
+	// peers whose session went down. A peer's messages are held back while
+	// up to two PeersDown calls run: the one running when its session goes
+	// down, and the next, which takes the peer out. A call takes about 2.6
+	// seconds for each full-table peer it takes out. Take a full-table peer
+	// whose session goes down together with those of three other full-table
+	// peers. The two calls then take about 10.4 seconds, 2.6 for each of the
+	// four. The feed peaks at about 14,000 messages a second, so the four
+	// peers send at most about 146,000 messages in that time. Once the second
+	// call has returned, the handler applies the held-back messages faster
+	// than the feed sends more. A message takes about 1 KiB, so 150,000 take
+	// about 150 MiB (README "Memory"). At this number HandleMessage waits
+	// until one of them is applied.
+	maxHeldBackMessages = 150000
 )
 
 // PrefixHandler keeps the routing table in memory current from the feed. It
@@ -445,7 +451,8 @@ func (h *PrefixHandler) peerDown() {
 // mu for one message at a time, so HandleMessage takes new messages in
 // between. The peer's new messages join the end of the held-back ones. A
 // held-back message that says the peer's session went down again adds the
-// peer to downPeers. The rest stay held back until the next PeersDown.
+// peer to downPeers. The rest stay held back until the next PeersDown has
+// returned.
 func (h *PrefixHandler) applyHeldBack(peer string) {
 	for {
 		h.mu.Lock()