Take every down peer out of the table in one walk (closes #225)

The prefix handler ran one PeerDown per peer, one after another. When
the sessions of several peers went down together, their walks queued up.
A returning full-table peer's messages stayed held back while the walks
ahead of its own ran, and could reach the limit of 20,000.

The table's PeersDown now takes several peers out of every prefix in one
walk. One walk runs at a time. The next one takes every peer whose
session went down since the previous one began. The doc of
maxHeldBackMessages says how long a peer's messages are held back. The
limit is now 150,000 messages, about 150 MiB. The status page shows how
many messages each handler holds back.

Model: opus-5-5

README.md

@@ -653,9 +653,8 @@ What happens at each limit:
 - Held-back messages: the prefix handler holds back a peer's messages after the
   peer's session goes down. It applies them once the peer has left every prefix.
   It holds at most 150,000 such messages (`maxHeldBackMessages`), about 150 MiB.
-  The limit is enough when the sessions of four full-table peers go down
+  At that number it takes nothing more off its queue until it has applied one of
-  together. At the limit it takes nothing more off its queue until it has
+  them.
-  applied one of them.
 
 With `DEBUG=routewatch` the daemon logs a `System stats` line every 60 seconds
 with the goroutine count and Go memory figures.

TODO.md

@@ -35,16 +35,20 @@ tested.
 
 # Completed Steps
 
-- 2026-11-02: when several peers' sessions go down at once, the prefix handler
+- 2026-11-02: when the sessions of several peers go down, the prefix handler
-  goes on taking messages. The table's `PeersDown` takes several peers out of
+  takes the peers out of the routing table together. The table's `PeersDown`
-  every prefix in one walk over the table. One walk runs at a time. The next one
+  takes several peers out of every prefix in one walk. One walk runs at a time.
-  takes every peer whose session went down meanwhile. So a peer's messages are
+  The next one takes every peer whose session went down since the previous one
-  held back for two walks at most. Each walk takes about 2.6 seconds for each
+  began. Each peer used to have a walk of its own. A peer's messages stayed held
-  full-table peer it takes out. The handler holds back at most 150,000 messages,
+  back while the walks queued ahead of its own ran. The handler holds back at
-  about 150 MiB. The limit is enough when the sessions of four full-table peers
+  most 150,000 messages, about 150 MiB. It used to hold back at most 20,000. The
-  go down together. The status page shows how many messages the handler holds
+  status page shows how many messages the handler holds back (closes #225)
-  back. Each peer used to have a walk of its own, and the handler stopped taking
+- 2026-11-02: working out the home page's cards no longer stops the prefix
-  messages at 20,000 held back (closes #225)
+  handler. The walk over the routing table holds the table's lock for 10,000
+  prefixes at a time, then lets the lock go for as long again, as `PeerDown`
+  does. It used to take the lock back at once. The prefix handler then applied
+  about one update per 10,000 prefixes walked. During a walk, its queue could
+  fill, and the streamer then dropped updates for it (closes #226)
 - 2026-11-02: like a failed WHOIS lookup, a lookup skipped because its RIR is
   paused makes the fetcher wait longer before the next lookup. The longest wait
   is one minute. The status page does not count a skip as a failure. A skip used

internal/routetable/overview.go

@@ -79,8 +79,9 @@ type OriginPeers struct {
 
 // Overview works out the table's overview. It lists at most maxListed bogon
 // announcements and at most maxListed prefixes with more than one origin AS:
-// the first in prefix order. It walks over every prefix, but holds the read
+// the first in prefix order. It visits every prefix in the table, in batches
-// lock for walkBatch prefixes at a time.
+// of walkBatch. After each batch, it lets the table's lock go for as long as
+// it held the lock for the batch.
 func (t *Table) Overview(maxListed int) Overview {
 	var overview Overview
 
@@ -110,15 +111,18 @@ func (t *Table) Overview(maxListed int) Overview {
 }
 
 // walkPrefixes calls visit with every prefix of the table and its origins. It
-// holds the read lock, but lets it go after every walkBatch prefixes, so the
+// holds the read lock for walkBatch prefixes at a time. After each batch, it
-// updates waiting for it get in. A prefix that is added or removed in such a
+// lets the table's lock go for as long as it held the lock for the batch.
-// gap may or may not be visited, as Go allows for a map that changes during a
+// During a walk over many batches, updates can then take the lock about half
-// range over it. visit must not keep origins once it returns.
+// of the time. The walk may or may not visit a prefix that is added or removed
+// while the lock is let go. Go allows this in a range over a map that changes.
+// visit must not keep origins once it returns.
 func (t *Table) walkPrefixes(visit func(prefix netip.Prefix, origins []*origin)) {
 	t.mu.RLock()
 	defer t.mu.RUnlock()
 
 	visited := 0
+	batchStart := time.Now()
 
 	for _, f := range []*family{&t.ipv4, &t.ipv6} {
 		for _, prefixes := range f.byLength {
@@ -127,8 +131,13 @@ func (t *Table) walkPrefixes(visit func(prefix netip.Prefix, origins []*origin))
 
 				visited++
 				if visited%walkBatch == 0 {
+					held := time.Since(batchStart)
+
 					t.mu.RUnlock()
+					time.Sleep(held)
 					t.mu.RLock()
+
+					batchStart = time.Now()
 				}
 			}
 		}

internal/routetable/overview_internal_test.go

@@ -131,3 +131,61 @@ func TestWalkPrefixesLetsUpdatesIn(t *testing.T) {
 			"before its second batch")
 	}
 }
+
+// TestOverviewLetsUpdatesIn applies another peer's 1,000 announcements while
+// Overview walks a table with more prefixes than five of the walk's batches.
+// After each batch, the walk lets the lock go for as long as it held the lock.
+// So all 1,000 announcements must be applied before Overview returns. A walk
+// that takes the lock back at once lets in about one update per batch.
+func TestOverviewLetsUpdatesIn(t *testing.T) {
+	t.Parallel()
+
+	const (
+		asn       = 1000
+		peer      = "192.0.2.1"
+		otherPeer = "192.0.2.2"
+		// prefixCount is more prefixes than five of the walk's batches.
+		prefixCount   = 5*walkBatch + 1
+		announcements = 1000
+	)
+
+	routeTime := time.Date(2026, 1, 2, 3, 4, 5, 0, time.UTC)
+	table := New(Params{})
+	prefixes := make([]netip.Prefix, 0, prefixCount)
+	addr := netip.MustParseAddr("1.0.0.0")
+
+	for range prefixCount {
+		prefix := netip.PrefixFrom(addr, addr.BitLen())
+		table.Announce(prefix, asn, Route{Peer: peer, Time: routeTime})
+		prefixes = append(prefixes, prefix)
+		addr = addr.Next()
+	}
+
+	overviewDone := make(chan struct{})
+
+	go func() {
+		defer close(overviewDone)
+
+		table.Overview(0)
+	}()
+
+	// Wait until Overview holds the read lock. TryLock fails while a reader
+	// holds it.
+	for table.mu.TryLock() {
+		table.mu.Unlock()
+		runtime.Gosched()
+	}
+
+	for _, prefix := range prefixes[:announcements] {
+		table.Announce(prefix, asn, Route{Peer: otherPeer, Time: routeTime})
+	}
+
+	select {
+	case <-overviewDone:
+		t.Fatal("Overview returned before another peer's 1,000 announcements " +
+			"were applied")
+	default:
+	}
+
+	<-overviewDone
+}

internal/routewatch/prefixhandler.go

@@ -46,18 +46,18 @@ const (
 	maxTableChangesPerWrite = 20000
 
 	// maxHeldBackMessages is the most messages the handler holds back from
-	// peers whose session went down. A peer's messages are held back while
+	// peers whose session went down. A peer's messages are held back from
-	// up to two PeersDown calls run: the one running when its session goes
+	// the moment its session goes down until peerDown has applied them. If
-	// down, and the next, which takes the peer out. A call takes about 2.6
+	// peerDown is running at that moment, the peer waits for the rest of
-	// seconds for each full-table peer it takes out. Take a full-table peer
+	// peerDown's current PeersDown call. The peer also waits while peerDown
-	// whose session goes down together with those of three other full-table
+	// applies the messages held back from that call's peers, those that
-	// peers. The two calls then take about 10.4 seconds, 2.6 for each of the
+	// arrive meanwhile included. The next PeersDown call takes the peer out.
-	// four. The feed peaks at about 14,000 messages a second, so the four
+	// Once that call returns, peerDown applies the messages held back from
-	// peers send at most about 146,000 messages in that time. Once the second
+	// that call's peers, one peer after another, in the order the call lists
-	// call has returned, the handler applies the held-back messages faster
+	// them. A held-back message that says the peer's session went down again
-	// than the feed sends more. A message takes about 1 KiB, so 150,000 take
+	// makes the peer's later messages wait for the next call. A message takes
-	// about 150 MiB (README "Memory"). At this number HandleMessage waits
+	// about 1 KiB, so 150,000 take about 150 MiB (README "Memory"). At this
-	// until one of them is applied.
+	// number HandleMessage waits until one of them is applied.
 	maxHeldBackMessages = 150000
 )
 
@@ -70,12 +70,12 @@ const (
 // The table's PeersDown applies sessions going down. It visits every prefix,
 // which takes seconds on a full table. So the handler runs PeersDown on a
 // separate goroutine and goes on taking messages meanwhile. One PeersDown runs
-// at a time. The next one takes every peer whose session went down meanwhile.
+// at a time. The next one takes every peer whose session went down since the
-// PeersDown lets the table's lock go about half of the time, so the handler
+// previous one began. PeersDown lets the table's lock go about half of the
-// applies other peers' messages while PeersDown runs. The handler holds back a
+// time, so the handler applies other peers' messages while PeersDown runs. The
-// down peer's later messages until the peer has left every prefix. Then the
+// handler holds back a down peer's later messages until the peer has left every
-// handler applies them in order, one at a time, and takes new messages in
+// prefix. Then the handler applies them in order, one at a time, and takes new
-// between. It holds back at most maxHeldBackMessages messages.
+// messages in between. It holds back at most maxHeldBackMessages messages.
 //
 // Each flush writes the table's changes to the database. It also stores each
 // prefix the updates name with the time it was first seen.

internal/routewatch/prefixhandler_table_internal_test.go

@@ -164,11 +164,12 @@ func TestPrefixHandlerTakesMessagesWhileAPeerGoesDown(t *testing.T) {
 	})
 }
 
-// TestPrefixHandlerTakesPeersDownTogether holds up the PeersDown of a peer
+// TestPrefixHandlerTakesPeersDownTogether has four peers announce testV4Prefix.
-// whose session went down. Meanwhile the sessions of three more peers go down,
+// It holds up the PeersDown of the first peer, whose session goes down.
-// and each of them announces a prefix. Once the first PeersDown is let go, the
+// Meanwhile the sessions of the three others go down, and each of them
-// next one must take the three peers together. Each one's announcement must
+// announces otherV4Prefix. Once the first PeersDown is let go, the next one
-// then be in the table.
+// must take the three others together. No peer may then carry testV4Prefix,
+// and the three others must carry otherV4Prefix.
 func TestPrefixHandlerTakesPeersDownTogether(t *testing.T) {
 	t.Parallel()