The prefix handler ran one PeerDown per peer, one after another. When
the sessions of several peers went down together, their walks queued up.
A returning full-table peer's messages stayed held back while the walks
ahead of its own ran, and could reach the limit of 20,000.
The table's PeersDown now takes several peers out of every prefix in one
walk. One walk runs at a time. The next one takes every peer whose
session went down since the previous one began. The doc of
maxHeldBackMessages says how long a peer's messages are held back. The
limit is now 150,000 messages, about 150 MiB. The status page shows how
many messages each handler holds back.
Model: opus-5-5
The table's PeerDown visits every prefix under the write lock, which
takes over a second on a full table. The prefix handler called it
inline, so its queue filled and the streamer dropped updates.
PeerDown now holds the lock for 10,000 prefixes at a time, then lets
the lock go for as long again. The handler runs PeerDown on a separate
goroutine, one peer at a time. The handler holds back the down peer's
later messages until that PeerDown returns. Then the handler applies
them in order, one at a time, and takes new messages in between. It
holds back at most 20,000 messages. Stop waits for all of this before
its last flush.
Model: opus-5-5
Five prefix handler tests gave HandleMessage two seconds to apply a full
batch of 25,000 prefixes, which under the race detector on a loaded host
takes longer. A test now waits up to stepTimeout, 15 seconds, for each step
it expects.
The slowest tests filled production limits, such as the peering handler's
500,000 paths. The peering handler now takes its path limit, and the prefix
handler its limit on one write of the table's changes, as they take their
timer interval. Those tests give small limits, and the tests of the maximum
batch size lower it to one batch. This also fixes
#215.
The test helpers bound every wait, Stop included, so a test that hangs in
one fails with its own message.
Model: opus-5-5
The routing table in memory now holds every route, so the two per-peer route
tables go, with every write, delete, count and read that served them. The
prefix handler applies each update to the table, writes the table's changes
every second, and stores each new prefix with its first-seen time. The peer
delete, its retry on the next flush and the updates held for it go too. The
types and errors of the server's answers move to internal/server.
The image's memory settings fit a full table of about 1 GB: GOMEMLIMIT of
2.5 GiB, a 512 MiB page cache for the writing connection, and SQLite heap
limits of 768 MiB and 1 GiB. The README describes the new table, its
database copy and the memory budget.
Model: opus-5-5
The IP lookups, the AS and prefix pages and their API answers, the prefix-length pages and the stats read routes only from the routing table the prefix handler keeps. fx reads the table at startup and gives it to both. AS names, WHOIS details and first-seen times still come from the database.
The IP lookup names the origin AS most peers carry, with the time of the newest kept route. The prefix answers list peers whose routes are not kept without a path or time. The stats gain origins_without_kept_route. They no longer read route times from the per-peer tables, so the two helpers that did are gone.
Model: opus-5-5
When a write ran past the stop deadline, a full batch's wake-up or a tick
could wait beside Stop's context. The flush goroutine's select picked one of
them at random, so about half the time it wrote that batch after the
deadline. That was the intermittent failure of
TestPrefixHandlerSkipsItsLastWriteAfterTheStopDeadline. The flush goroutine
now takes a waiting stop first. A new test hands over a full batch while a
write is held; it fails about half its runs on the old code.
TestPrefixHandlerSkipsItsLastWriteAfterTheStopDeadline and
TestPrefixHandlerStopLogsTheTableChangesLeftUnwritten also failed when the
flush timer fired between two of their steps. Both now give the handler a
timer interval longer than the test.
Model: opus-5-5
The prefix handler applies each update to the routing table in memory as
it handles it, sorted into announcements and withdrawals as for the live
route tables. Each flush first writes at most 20,000 of the table's
changed pairs, with every new peer; the rest follow in the next flushes.
A failed write puts its changes and new peers back in the table. Stop
makes one last flush, and logs the changed pairs left.
Startup reads the table before the streamer connects, and logs its pairs,
peers and time. The per-peer counts are a slice, so the read takes about
a third less time. The health check's start period is 60 seconds, so a
full table's read fits before upaas looks.
Model: opus-5-5
clawbot