Keep one entry per AS in the AS handler's batch (closes #217)

HandleMessage added one entry to the batch for every AS of every path,
and waited for the write in progress once the batch held 300,000. In a
burst of the feed its queue then filled, and the streamer dropped
messages.

The batch is now a map from AS number to the earliest and the latest
time of its updates. It holds one entry per AS, so it no longer grows
with the feed's rate, and HandleMessage never waits for a write.
asnBatchSize now counts ASes. Stop logs the ASes it did not write under
"asns".

Model: opus-5-5

README.md

@@ -638,15 +638,14 @@ What happens at each limit:
     - The AS handler's batch holds one entry per AS, with the earliest and the
       latest time of its updates. About 80,000 ASes appear in the feed, so the
       batch takes at most about 10 MiB, however fast the feed runs.
-    - The peer handler's batch stops at 100,000 updates (`peerBatchMaxSize`),
+    - The peer and prefix handlers' batches stop at ten times their batch size:
-      about 10 MiB.
+      100,000 updates for the peer handler (`peerBatchMaxSize`, about 10 MiB),
-    - The prefix handler's batch stops at 250,000 updates
+      and 250,000 for the prefix handler (`prefixBatchMaxSize`, about 15 MiB).
-      (`prefixBatchMaxSize`), about 15 MiB.
+      At that size the handler takes nothing more off its queue until the write
+      ends. The queue then fills, and the streamer drops messages as above.
 
-    At that maximum the peer or prefix handler takes nothing more off its queue
+    With the batches being written, the three handlers' batches take at most
-    until the write ends. The queue then fills, and the streamer drops messages
+    about 0.1 GiB of the Go heap.
-    as above. With the batches being written, the three handlers' batches take
-    at most about 0.1 GiB of the Go heap.
 
 With `DEBUG=routewatch` the daemon logs a `System stats` line every 60 seconds
 with the goroutine count and Go memory figures.

TODO.md

@@ -36,10 +36,10 @@ tested.
 # Completed Steps
 
 - 2026-11-02: the AS handler never waits for a database write. Its batch holds
-  one entry per AS, with the earliest and the latest time of its updates. So the
+  one entry per AS, with the earliest and the latest time of its updates, so it
-  batch no longer grows with the feed's rate. It used to hold one entry for each
+  no longer grows with the feed's rate. It used to hold one entry for each AS of
-  AS of each path and to wait once it held 300,000. In a burst of the feed its
+  each path and to wait once it held 300,000. In a burst of the feed its queue
-  queue then filled, and the streamer dropped messages for it (closes #217)
+  then filled, and the streamer dropped messages for it (closes #217)
 - 2026-10-27: the tests of `internal/routewatch` no longer fail on a loaded
   host. Five of them gave `HandleMessage` two seconds to apply a full prefix
   batch, which under the race detector can take longer. A test now waits up to

internal/routewatch/ashandler.go

@@ -27,16 +27,16 @@ const (
 	asnBatchTimeout = 2 * time.Second
 )
 
-// ASHandler records every AS in the AS paths of BGP UPDATE messages. Its batch
+// ASHandler records every AS in the AS paths of BGP UPDATE messages. It
-// holds the earliest and the latest time of each AS's updates. The flush
+// collects, for each AS, the earliest and the latest time of its updates in a
-// goroutine writes the batch to the database when it holds asnBatchSize ASes,
+// batch. The flush goroutine writes the batch to the database when it holds
-// and every asnBatchTimeout.
+// asnBatchSize ASes, and every asnBatchTimeout.
 type ASHandler struct {
 	db     database.Store
 	logger *logger.Logger
 
-	// Batching. mu guards batch and writing only. The flush goroutine takes
+	// Batching. mu guards batch and writing only: the flush goroutine takes
-	// the batch under mu and writes it after releasing mu. So HandleMessage
+	// the batch under mu and writes it after releasing mu, so HandleMessage
 	// never waits for a database write.
 	mu sync.Mutex
 	// batch holds the earliest and the latest time of the updates for each AS
@@ -56,7 +56,8 @@ type ASHandler struct {
 // NewASHandler creates and returns a new ASHandler instance. It initializes
 // the batching system and starts a background goroutine that periodically
 // writes the collected ASes to the database. The caller must call Stop when
-// finished, to write the pending ASes and end the background goroutine.
+// finished to ensure all pending ASes are written and the background goroutine
+// is terminated.
 func NewASHandler(db database.Store, logger *logger.Logger) *ASHandler {
 	return newASHandler(db, logger, asnBatchTimeout)
 }
@@ -100,7 +101,7 @@ func (h *ASHandler) QueueCapacity() int {
 
 // HandleMessage adds the time of msg to the batch entry of each AS in its AS
 // path, the origin AS and the members of an AS set included. When the batch
-// holds asnBatchSize ASes, HandleMessage wakes the flush goroutine to write it.
+// holds asnBatchSize ASes, it wakes the flush goroutine to write it.
 func (h *ASHandler) HandleMessage(msg *ristypes.RISMessage) {
 	h.mu.Lock()
 	defer h.mu.Unlock()
@@ -125,11 +126,10 @@ func (h *ASHandler) HandleMessage(msg *ristypes.RISMessage) {
 	}
 }
 
-// Stop stops the flush goroutine and waits for it until ctx ends. The flush
+// Stop stops the flush goroutine, which writes the ASes the handler holds before
-// goroutine writes the batch before it returns, unless ctx has ended. A write
+// it returns, and waits for it until ctx ends. Once ctx has ended the flush
-// still running when the process exits is lost. Stop logs how many entries are
+// goroutine skips that last write, and a write still running is lost when the
-// not written, in the batch being written and in the next batch. An AS in both
+// process exits, so Stop logs how many ASes are not written.
-// batches counts twice.
 func (h *ASHandler) Stop(ctx context.Context) {
 	h.stopCh <- ctx
 
@@ -144,7 +144,7 @@ func (h *ASHandler) Stop(ctx context.Context) {
 
 	if unwritten > 0 {
 		h.logger.Error("Stop deadline passed before the AS handler wrote "+
-			"its batches", "batch_entries", unwritten)
+			"its ASes", "asns", unwritten)
 	}
 }
 

internal/routewatch/batchwrite_internal_test.go

@@ -178,7 +178,7 @@ type stopLog struct {
 	Updates      int    `json:"updates"`
 	TableChanges int    `json:"table_changes"`
 	Paths        int    `json:"paths"`
-	BatchEntries int    `json:"batch_entries"`
+	ASNs         int    `json:"asns"`
 }
 
 // newErrorLogger returns a logger that writes only errors, as JSON lines, to
@@ -497,12 +497,12 @@ func TestPrefixHandlerStopCountsTheTableChangesOfAWriteInProgress(t *testing.T)
 }
 
 // TestASHandlerSkipsItsLastWriteAfterTheStopDeadline also checks what Stop
-// logs: the asnBatchSize entries of the held write and the entry for testASNB,
+// logs: the asnBatchSize ASes of the held write and testASNB, handled since,
-// handled since, are not written.
+// are not written.
 func TestASHandlerSkipsItsLastWriteAfterTheStopDeadline(t *testing.T) {
 	t.Parallel()
 
-	const wantEntries = asnBatchSize + 1
+	const wantASNs = asnBatchSize + 1
 
 	var logs bytes.Buffer
 
@@ -519,9 +519,9 @@ func TestASHandlerSkipsItsLastWriteAfterTheStopDeadline(t *testing.T) {
 		t.Errorf("AS%d was written after the stop deadline", testASNB)
 	}
 
-	if logged := lastLogged(t, &logs); logged.BatchEntries != wantEntries {
+	if logged := lastLogged(t, &logs); logged.ASNs != wantASNs {
-		t.Errorf("Stop logged %d batch entries not written, want %d",
+		t.Errorf("Stop logged %d ASes not written, want %d",
-			logged.BatchEntries, wantEntries)
+			logged.ASNs, wantASNs)
 	}
 }