Take every down peer out of the table in one walk (closes #225)

The prefix handler ran one PeerDown per peer, one after another. When
the sessions of several peers went down together, their walks queued up.
A returning full-table peer's messages stayed held back while the walks
ahead of its own ran, and could reach the limit of 20,000.

The table's PeersDown now takes several peers out of every prefix in one
walk. One walk runs at a time. The next one takes every peer whose
session went down since the previous one began. The doc of
maxHeldBackMessages says how long a peer's messages are held back. The
limit is now 150,000 messages, about 150 MiB. The status page shows how
many messages each handler holds back.

Model: opus-5-5

README.md

@@ -653,9 +653,8 @@ What happens at each limit:
 - Held-back messages: the prefix handler holds back a peer's messages after the
   peer's session goes down. It applies them once the peer has left every prefix.
   It holds at most 150,000 such messages (`maxHeldBackMessages`), about 150 MiB.
-  The limit is enough when the sessions of four full-table peers go down
-  together. At the limit it takes nothing more off its queue until it has
-  applied one of them.
+  At that number it takes nothing more off its queue until it has applied one of
+  them.
 
 With `DEBUG=routewatch` the daemon logs a `System stats` line every 60 seconds
 with the goroutine count and Go memory figures.

TODO.md

@@ -35,16 +35,20 @@ tested.
 
 # Completed Steps
 
-- 2026-11-02: when several peers' sessions go down at once, the prefix handler
-  goes on taking messages. The table's `PeersDown` takes several peers out of
-  every prefix in one walk over the table. One walk runs at a time. The next one
-  takes every peer whose session went down meanwhile. So a peer's messages are
-  held back for two walks at most. Each walk takes about 2.6 seconds for each
-  full-table peer it takes out. The handler holds back at most 150,000 messages,
-  about 150 MiB. The limit is enough when the sessions of four full-table peers
-  go down together. The status page shows how many messages the handler holds
-  back. Each peer used to have a walk of its own, and the handler stopped taking
-  messages at 20,000 held back (closes #225)
+- 2026-11-02: when the sessions of several peers go down, the prefix handler
+  takes the peers out of the routing table together. The table's `PeersDown`
+  takes several peers out of every prefix in one walk. One walk runs at a time.
+  The next one takes every peer whose session went down since the previous one
+  began. Each peer used to have a walk of its own. A peer's messages stayed held
+  back while the walks queued ahead of its own ran. The handler holds back at
+  most 150,000 messages, about 150 MiB. It used to hold back at most 20,000. The
+  status page shows how many messages the handler holds back (closes #225)
+- 2026-11-02: working out the home page's cards no longer stops the prefix
+  handler. The walk over the routing table holds the table's lock for 10,000
+  prefixes at a time, then lets the lock go for as long again, as `PeerDown`
+  does. It used to take the lock back at once. The prefix handler then applied
+  about one update per 10,000 prefixes walked. During a walk, its queue could
+  fill, and the streamer then dropped updates for it (closes #226)
 - 2026-11-02: like a failed WHOIS lookup, a lookup skipped because its RIR is
   paused makes the fetcher wait longer before the next lookup. The longest wait
   is one minute. The status page does not count a skip as a failure. A skip used

internal/routetable/overview.go

@@ -79,8 +79,9 @@ type OriginPeers struct {
 
 // Overview works out the table's overview. It lists at most maxListed bogon
 // announcements and at most maxListed prefixes with more than one origin AS:
-// the first in prefix order. It walks over every prefix, but holds the read
-// lock for walkBatch prefixes at a time.
+// the first in prefix order. It visits every prefix in the table, in batches
+// of walkBatch. After each batch, it lets the table's lock go for as long as
+// it held the lock for the batch.
 func (t *Table) Overview(maxListed int) Overview {
 	var overview Overview
 
@@ -110,15 +111,18 @@ func (t *Table) Overview(maxListed int) Overview {
 }
 
 // walkPrefixes calls visit with every prefix of the table and its origins. It
-// holds the read lock, but lets it go after every walkBatch prefixes, so the
-// updates waiting for it get in. A prefix that is added or removed in such a
-// gap may or may not be visited, as Go allows for a map that changes during a
-// range over it. visit must not keep origins once it returns.
+// holds the read lock for walkBatch prefixes at a time. After each batch, it
+// lets the table's lock go for as long as it held the lock for the batch.
+// During a walk over many batches, updates can then take the lock about half
+// of the time. The walk may or may not visit a prefix that is added or removed
+// while the lock is let go. Go allows this in a range over a map that changes.
+// visit must not keep origins once it returns.
 func (t *Table) walkPrefixes(visit func(prefix netip.Prefix, origins []*origin)) {
 	t.mu.RLock()
 	defer t.mu.RUnlock()
 
 	visited := 0
+	batchStart := time.Now()
 
 	for _, f := range []*family{&t.ipv4, &t.ipv6} {
 		for _, prefixes := range f.byLength {
@@ -127,8 +131,13 @@ func (t *Table) walkPrefixes(visit func(prefix netip.Prefix, origins []*origin))
 
 				visited++
 				if visited%walkBatch == 0 {
+					held := time.Since(batchStart)
+
 					t.mu.RUnlock()
+					time.Sleep(held)
 					t.mu.RLock()
+
+					batchStart = time.Now()
 				}
 			}
 		}

internal/routetable/overview_internal_test.go

@@ -131,3 +131,61 @@ func TestWalkPrefixesLetsUpdatesIn(t *testing.T) {
 			"before its second batch")
 	}
 }
+
+// TestOverviewLetsUpdatesIn applies another peer's 1,000 announcements while
+// Overview walks a table with more prefixes than five of the walk's batches.
+// After each batch, the walk lets the lock go for as long as it held the lock.
+// So all 1,000 announcements must be applied before Overview returns. A walk
+// that takes the lock back at once lets in about one update per batch.
+func TestOverviewLetsUpdatesIn(t *testing.T) {
+	t.Parallel()
+
+	const (
+		asn       = 1000
+		peer      = "192.0.2.1"
+		otherPeer = "192.0.2.2"
+		// prefixCount is more prefixes than five of the walk's batches.
+		prefixCount   = 5*walkBatch + 1
+		announcements = 1000
+	)
+
+	routeTime := time.Date(2026, 1, 2, 3, 4, 5, 0, time.UTC)
+	table := New(Params{})
+	prefixes := make([]netip.Prefix, 0, prefixCount)
+	addr := netip.MustParseAddr("1.0.0.0")
+
+	for range prefixCount {
+		prefix := netip.PrefixFrom(addr, addr.BitLen())
+		table.Announce(prefix, asn, Route{Peer: peer, Time: routeTime})
+		prefixes = append(prefixes, prefix)
+		addr = addr.Next()
+	}
+
+	overviewDone := make(chan struct{})
+
+	go func() {
+		defer close(overviewDone)
+
+		table.Overview(0)
+	}()
+
+	// Wait until Overview holds the read lock. TryLock fails while a reader
+	// holds it.
+	for table.mu.TryLock() {
+		table.mu.Unlock()
+		runtime.Gosched()
+	}
+
+	for _, prefix := range prefixes[:announcements] {
+		table.Announce(prefix, asn, Route{Peer: otherPeer, Time: routeTime})
+	}
+
+	select {
+	case <-overviewDone:
+		t.Fatal("Overview returned before another peer's 1,000 announcements " +
+			"were applied")
+	default:
+	}
+
+	<-overviewDone
+}

internal/routewatch/prefixhandler.go

@@ -46,18 +46,18 @@ const (
 	maxTableChangesPerWrite = 20000
 
 	// maxHeldBackMessages is the most messages the handler holds back from
-	// peers whose session went down. A peer's messages are held back while
-	// up to two PeersDown calls run: the one running when its session goes
-	// down, and the next, which takes the peer out. A call takes about 2.6
-	// seconds for each full-table peer it takes out. Take a full-table peer
-	// whose session goes down together with those of three other full-table
-	// peers. The two calls then take about 10.4 seconds, 2.6 for each of the
-	// four. The feed peaks at about 14,000 messages a second, so the four
-	// peers send at most about 146,000 messages in that time. Once the second
-	// call has returned, the handler applies the held-back messages faster
-	// than the feed sends more. A message takes about 1 KiB, so 150,000 take
-	// about 150 MiB (README "Memory"). At this number HandleMessage waits
-	// until one of them is applied.
+	// peers whose session went down. A peer's messages are held back from
+	// the moment its session goes down until peerDown has applied them. If
+	// peerDown is running at that moment, the peer waits for the rest of
+	// peerDown's current PeersDown call. The peer also waits while peerDown
+	// applies the messages held back from that call's peers, those that
+	// arrive meanwhile included. The next PeersDown call takes the peer out.
+	// Once that call returns, peerDown applies the messages held back from
+	// that call's peers, one peer after another, in the order the call lists
+	// them. A held-back message that says the peer's session went down again
+	// makes the peer's later messages wait for the next call. A message takes
+	// about 1 KiB, so 150,000 take about 150 MiB (README "Memory"). At this
+	// number HandleMessage waits until one of them is applied.
 	maxHeldBackMessages = 150000
 )
 
@@ -70,12 +70,12 @@ const (
 // The table's PeersDown applies sessions going down. It visits every prefix,
 // which takes seconds on a full table. So the handler runs PeersDown on a
 // separate goroutine and goes on taking messages meanwhile. One PeersDown runs
-// at a time. The next one takes every peer whose session went down meanwhile.
-// PeersDown lets the table's lock go about half of the time, so the handler
-// applies other peers' messages while PeersDown runs. The handler holds back a
-// down peer's later messages until the peer has left every prefix. Then the
-// handler applies them in order, one at a time, and takes new messages in
-// between. It holds back at most maxHeldBackMessages messages.
+// at a time. The next one takes every peer whose session went down since the
+// previous one began. PeersDown lets the table's lock go about half of the
+// time, so the handler applies other peers' messages while PeersDown runs. The
+// handler holds back a down peer's later messages until the peer has left every
+// prefix. Then the handler applies them in order, one at a time, and takes new
+// messages in between. It holds back at most maxHeldBackMessages messages.
 //
 // Each flush writes the table's changes to the database. It also stores each
 // prefix the updates name with the time it was first seen.

internal/routewatch/prefixhandler_table_internal_test.go

@@ -164,11 +164,12 @@ func TestPrefixHandlerTakesMessagesWhileAPeerGoesDown(t *testing.T) {
 	})
 }
 
-// TestPrefixHandlerTakesPeersDownTogether holds up the PeersDown of a peer
-// whose session went down. Meanwhile the sessions of three more peers go down,
-// and each of them announces a prefix. Once the first PeersDown is let go, the
-// next one must take the three peers together. Each one's announcement must
-// then be in the table.
+// TestPrefixHandlerTakesPeersDownTogether has four peers announce testV4Prefix.
+// It holds up the PeersDown of the first peer, whose session goes down.
+// Meanwhile the sessions of the three others go down, and each of them
+// announces otherV4Prefix. Once the first PeersDown is let go, the next one
+// must take the three others together. No peer may then carry testV4Prefix,
+// and the three others must carry otherV4Prefix.
 func TestPrefixHandlerTakesPeersDownTogether(t *testing.T) {
 	t.Parallel()