Every IP lookup result gains `geo`. One request to geojs carries all uncached addresses of a lookup, within 1 second. On a timeout, an error or an answer with no location, the lookup reads DB-IP City Lite, downloaded monthly into the state directory. Geojs answers stay in memory, at most 250,000, and in `cache.db` for 7 days. The home page footer credits DB-IP.
Judgement call: a reserved address goes to neither source, and its `geo` is null.
Judgement call: `updated` for a DB-IP answer is when the copy was downloaded.
Deviation: the cache is SQLite, by sneak's words, not the JSON file the standing rule asks for.
Model: opus-5-5
`ptr` keeps its field and shape. It now comes from Google, Cloudflare, AdGuard and Quad9 over RFC 8484, tried in turn within 2 seconds, in place of the host's resolver with no cache. Answers stay in memory and in a new `cache.db` for the record's TTL, at most 250,000 addresses. A reserved address is not sent to any resolver and has an empty `ptr`.
Judgement call: each resolver gets an equal share of the time left, so a hung one does not use up the limit.
Judgement call: the negative TTL is the smaller of the SOA minimum and the SOA's own TTL.
Deviation: the cache is SQLite, by sneak's words, not the JSON file the standing rule asks for.
Model: opus-5-5
The lists are not only used for blocking, so the package, the `blocklists` field of the IP lookup answer and the state directory are now IP lists (`internal/iplist`, `ip_lists`, `iplists`). Every result gains a `flags` object with `is_tor` and `is_bogon`. The largest list file accepted is 200 MiB.
Judgement call: the dated 2026-10-27 `TODO.md` entry keeps the old word.
Rule suppressed: `tagliatelle` on the two result structs, for snake_case keys.
Model: opus-5-5
The routing table records a row each time a prefix and origin AS gets its first peer (announced) or loses its last (withdrawn), with the time and how many peers carry the prefix just after. The prefix handler writes the rows to a new origin_changes table, apart from routes. Every hour the database maintainer keeps each prefix's rows from the last 72 hours or its newest 500, whichever keeps more. The prefix page lists them newest first, at most 5,000.
At most 100,000 rows wait in memory to be written; past that the oldest are dropped and logged. A stop loses the rows past its last write.
Model: deepseek-v4-pro (implementation); opus-5-5 (commit message)
The streamer dropped a random share of a handler's updates once its
queue was half full. When the feed paused and then sent its backlog
while the routing table's Overview walked every prefix, the prefix
handler lost about 10,000 updates with its queue never full.
The streamer now drops an update only when the queue is full. The
prefix handler's queue holds 100,000 messages, about 100 MiB, so it
can hold such a backlog. The README's Memory and Feed handling
sections give the new sizes and the rule.
Model: opus-5-5
The prefix handler ran one PeerDown per peer, one after another. When
the sessions of several peers went down together, their walks queued up.
A returning full-table peer's messages stayed held back while the walks
ahead of its own ran, and could reach the limit of 20,000.
The table's PeersDown now takes several peers out of every prefix in one
walk. One walk runs at a time. The next one takes every peer whose
session went down since the previous one began. The doc of
maxHeldBackMessages says how long a peer's messages are held back. The
limit is now 150,000 messages, about 150 MiB. The status page shows how
many messages each handler holds back.
Model: opus-5-5
A lookup whose RIR is paused is skipped. The skip left the fetcher's wait
as it was. Each lookup asks IANA before it finds its RIR paused. So while
most stale ASes belonged to a paused RIR, the fetcher could ask IANA every
second.
Like a failure, a skipped lookup now makes the wait 1.5 times as long, up
to one minute. The skip is still not counted as a failure, so the status
page's failure figures leave it out.
Model: opus-5-5
The table's PeerDown visits every prefix under the write lock, which
takes over a second on a full table. The prefix handler called it
inline, so its queue filled and the streamer dropped updates.
PeerDown now holds the lock for 10,000 prefixes at a time, then lets
the lock go for as long again. The handler runs PeerDown on a separate
goroutine, one peer at a time. The handler holds back the down peer's
later messages until that PeerDown returns. Then the handler applies
them in order, one at a time, and takes new messages in between. It
holds back at most 20,000 messages. Stop waits for all of this before
its last flush.
Model: opus-5-5
The fetcher kept one interval for every registry, so a registry that
refused a query was asked again within seconds. RIPE blocked the test
host's address that way.
The WHOIS client now pauses an RIR's or NIC.br's server for an hour
after the server refuses a query. A refused or reset connection is a
refusal, and so is an %ERROR:201: or %ERROR:202: line. Lookups at the
other servers go on. A lookup whose RIR is paused leaves the AS stale,
and counts as neither a success nor a failure. IANA is never paused,
because a pause of IANA would stop every lookup. NIC.br's hour-long
skip after its over-limit answer is now the same pause.
Model: opus-5-5
HandleMessage added one entry to the batch for every AS of every path,
and waited for the write in progress once the batch held 300,000. In a
burst of the feed its queue then filled, and the streamer dropped
messages.
The batch is now a map from AS number to the earliest and the latest
time of its updates. It holds one entry per AS, so it no longer grows
with the feed's rate. HandleMessage never waits for a write.
asnBatchSize now counts ASes. Stop logs the entries of the batch being
written and of the next batch under "batch_entries". An AS in both
counts twice.
Model: opus-5-5
Five prefix handler tests gave HandleMessage two seconds to apply a full
batch of 25,000 prefixes, which under the race detector on a loaded host
takes longer. A test now waits up to stepTimeout, 15 seconds, for each step
it expects.
The slowest tests filled production limits, such as the peering handler's
500,000 paths. The peering handler now takes its path limit, and the prefix
handler its limit on one write of the table's changes, as they take their
timer interval. Those tests give small limits, and the tests of the maximum
batch size lower it to one batch. This also fixes
#215.
The test helpers bound every wait, Stop included, so a test that hangs in
one fails with its own message.
Model: opus-5-5
The home page gets cards for the newest and oldest kept route, the newest
and oldest updated AS, the AS with the most prefixes, the ASes with a
single prefix and fifteen well-known networks. Two more cards count the
bogon announcements and the prefixes with several origin ASes. They link
to the new pages /bogons and /multiple-origins, which list up to 1,000
entries each.
The new package internal/bogon tells reserved prefixes and AS numbers,
from the IANA special-purpose and address space registries. Table.Overview
walks the table and lets the read lock go every 10,000 prefixes. The
server works the cards out once a minute and serves that copy. A new index
on asns(last_seen) finds the newest and oldest updated AS.
Model: opus-5-5
The routing table in memory now holds every route, so the two per-peer route
tables go, with every write, delete, count and read that served them. The
prefix handler applies each update to the table, writes the table's changes
every second, and stores each new prefix with its first-seen time. The peer
delete, its retry on the next flush and the updates held for it go too. The
types and errors of the server's answers move to internal/server.
The image's memory settings fit a full table of about 1 GB: GOMEMLIMIT of
2.5 GiB, a 512 MiB page cache for the writing connection, and SQLite heap
limits of 768 MiB and 1 GiB. The README describes the new table, its
database copy and the memory budget.
Model: opus-5-5
The IP lookups, the AS and prefix pages and their API answers, the prefix-length pages and the stats read routes only from the routing table the prefix handler keeps. fx reads the table at startup and gives it to both. AS names, WHOIS details and first-seen times still come from the database.
The IP lookup names the origin AS most peers carry, with the time of the newest kept route. The prefix answers list peers whose routes are not kept without a path or time. The stats gain origins_without_kept_route. They no longer read route times from the per-peer tables, so the two helpers that did are gone.
Model: opus-5-5
The prefix, AS and peer handlers skipped a timed write unless one
interval had passed since the last write ended. The next tick came a
little less than one interval after that, so every other tick was
skipped. Each tick now writes. A full batch's write restarts the timer,
so the next timed write comes one interval after that write began.
Comparing clock readings would not do: two ticks one interval apart can
read a hair under it. flushBatch no longer reports whether it wrote a
batch, since nothing reads that. A new test runs the loop on a fake
clock and fails on the old code.
Model: opus-5-5
A write can run past the stop deadline while a full batch's wake-up or a
tick waits beside Stop's context. The select picked one at random, so about
half the time the AS, peer or peering handler started another write after
the deadline. Each loop now takes a waiting stop first, as the prefix
handler's does. The prefix, AS and peer handlers' flush loops were
near-copies, so they are now one function, flushLoop. A new test for each
handler fails about half its runs on the old code.
The AS and peer handler tests give the flush timer an interval longer than
the test. A timer flush during the fill can no longer hang
TestPeerHandlerWaitsAtTheMaximumBatchSize.
Model: opus-5-5
When a write ran past the stop deadline, a full batch's wake-up or a tick
could wait beside Stop's context. The flush goroutine's select picked one of
them at random, so about half the time it wrote that batch after the
deadline. That was the intermittent failure of
TestPrefixHandlerSkipsItsLastWriteAfterTheStopDeadline. The flush goroutine
now takes a waiting stop first. A new test hands over a full batch while a
write is held; it fails about half its runs on the old code.
TestPrefixHandlerSkipsItsLastWriteAfterTheStopDeadline and
TestPrefixHandlerStopLogsTheTableChangesLeftUnwritten also failed when the
flush timer fired between two of their steps. Both now give the handler a
timer interval longer than the test.
Model: opus-5-5
The prefix handler applies each update to the routing table in memory as
it handles it, sorted into announcements and withdrawals as for the live
route tables. Each flush first writes at most 20,000 of the table's
changed pairs, with every new peer; the rest follow in the next flushes.
A failed write puts its changes and new peers back in the table. Stop
makes one last flush, and logs the changed pairs left.
Startup reads the table before the streamer connects, and logs its pairs,
peers and time. The per-peer counts are a slice, so the read takes about
a third less time. The health check's start period is 60 seconds, so a
full table's read fits before upaas looks.
Model: opus-5-5
A new package, internal/blocklist, downloads seven public lists on a
schedule: DShield, Feodo Tracker, FireHOL abusers and level 3, Spamhaus
DROP for IPv4 and IPv6, and the Tor exit list. It keeps the last good
copy of each in memory and on disk. A download sends back the
Last-Modified and ETag the publisher gave for the copy, also kept on
disk.
Every IP lookup result carries blocklists. An address with no live route
now gets a result too, without route fields, so the lookup no longer
answers 404. A successful answer carries Cache-Control, private for the
caller's own address. The address sets come from go4.org/netipx.
Model: opus-5-5
LACNIC's answer for an AS that NIC.br registers is NIC.br's answer with
each accented letter replaced by U+FFFD, so a name such as AS52751's
cannot be recovered from it. When LACNIC's answer carries NIC.br's
copyright notice, the client now also asks whois.registro.br for the AS
and takes the fields from that answer, which is Latin-1 and decoded as
before. When that query fails, when the answer has no fields, or when
NIC.br says the client is over its query limit, LACNIC's fields are kept
and a warning is logged. After an over-limit answer the client skips
NIC.br for an hour. The tests use recorded answers for AS52751.
Model: opus-5-5
A failed lookup now makes the fetcher's wait 1.5 times as long, up to
one minute instead of five. A success still halves it.
RIPE counts the person and role objects an answer holds against a daily
limit per IP address, and refuses an address for good that passes it
too often. RIPE, APNIC and AFRINIC are now asked with --no-personal,
which leaves those objects out but keeps the organisation object.
An answer with RIPE's access-denied or limit-reached error line is now
reported as a refusal holding that line, not as an answer with no
fields.
Model: opus-5-5
An AS path that holds AS 0 anywhere, a member of an AS set included, now
decodes as an empty path. RFC 7607 makes such a path malformed and its
announcements withdrawals. An empty path already gets that handling:
the prefix handler deletes the peer's route for the prefix, and the AS
and peering handlers record nothing from it. Before, a path such as
[64496, 0, 64497] stored the route with origin AS64497, recorded both
ASes, and recorded the peerings between the path's other ASes.
Model: opus-5-5
UpdatePrefixesBatch now skips a prefix that does not parse, so it never
becomes a row in prefixes_v4 or prefixes_v6 and never counts in the prefix
total in the stats. Before, the prefix handler stored no route for such a
prefix and logged it once per prefix and peer, but still passed it to
UpdatePrefixesBatch, which stored it.
Model: opus-5-5
recordSuccess and recordFailure appended a time on every fetch, and only
GetStats, run when a stats endpoint is requested, dropped the times older
than statsWindow, so a daemon nobody polled kept every one. Each now drops
those times before it appends. The drop takes them off the front of the
list instead of copying the rest: the times are in the order they were
recorded, and it now runs on every fetch.
The fetcher reads the time through a now field so a test can set the
clock. The test records 100,000 successes and failures a second apart
without asking for the stats and finds one hour's worth of each kept.
Model: opus-5-5
processPeerings replaced the path map every run but kept the count of
paths dropped while the map was full, so every later "Processed AS
peerings" line reported all drops since start next to that run's own
figures. The count is now set to 0 where the map is replaced, under the
same lock. A test fills the map, has one path dropped, runs the
processing and finds the count back at 0.
Model: opus-5-5
An announcement whose AS path is empty or ends in AS 0 now deletes the
peer's route for the prefix, as a withdrawal does. Before, it was neither
stored nor applied: the peer's old route stayed live, and within one
batch it cancelled the peer's earlier withdrawal of the prefix.
Model: opus-5-5
The read loop added to a handler's drop count atomically while the
statistics endpoint and the metrics log read it as a plain field, a data
race whenever messages were dropped. The count is now an atomic.Uint64,
read with Load. A test reads the handler statistics while the read loop
drops messages for a handler that never finishes one.
Run created the WHOIS fetcher and gave it to the HTTP server after the
server had started, while request handlers could read it. The fetcher is
now set before the server starts.
Model: opus-5-5
Each handler took the latest time per key in a batch and inserted it as
a new row's first_seen, so a key updated twice in one batch, seconds
apart, was stored first seen at the later time, and the IP lookup could
answer with a prefix first_seen after the route's last_seen.
The prefix handler now keeps the earliest time of all of the batch's
updates for each prefix. The AS, peer and peering handlers keep the
earliest and the latest time per key in database.SeenTimes, and
GetOrCreateASNBatch, UpdatePeerBatch and RecordPeering store a new row
first seen at the earliest and last seen at the latest. A stored row
still has only its last_seen set, to the latest.
Model: opus-5-5
The start hook registered its own handler for SIGINT and SIGTERM and
asked fx to shut down from it, although fx already stops the daemon on
both signals. A stop signal was handled twice, and the handler stayed
registered after the stop. fx now handles both signals alone, as
GO_HTTP_SERVER_CONVENTIONS.md "Signal Handling and Graceful Shutdown"
requires.
Model: opus-5-5
Shutdown passes its context to every stage: the streamer, the four
handlers, the WHOIS fetcher, the database maintainer and the HTTP server
each stop waiting when it ends. A handler then skips its last write and
logs how many updates it did not write, and the HTTP server closes the
connections still open. The database is checkpointed and closed last,
and left open once the deadline has passed, since a write may still
hold it. The stop hook gives Shutdown 8 seconds and fx 9, inside the 10
seconds upaas and docker stop allow, so a stop signal exits 0.
Model: opus-5-5
The 5-second checkpoint ran in TRUNCATE mode, which waits up to the
5-second busy timeout for every open read while holding the database
lock every batch write needs. It now runs in PASSIVE mode, which never
waits. A WAL file over 64 MiB instead gets one TRUNCATE checkpoint whose
wait for reads is capped at 500 ms and logged; it empties the file once
no read is open, and while reads follow one another with no pause it
can give up every time. A checkpoint that leaves part of the WAL
uncopied returns ErrCheckpointIncomplete, which the maintainer logs as
incomplete instead of completed. The writing connection's
journal_size_limit is 64 MiB.
Model: opus-5-5
A lookup whose RIR query failed, or whose RIR answer carried none of the
WHOIS fields (a refusal of service, for one), returned IANA's answer as
a success. The fetcher stored it as complete with blank fields, did not
look the AS up again for 30 days, and shortened its interval. Such a
lookup is now an error: the fetcher backs off and leaves the row as it
was. The fields are read from the RIR's answer alone. An AS IANA refers
to no RIR is stored with no WHOIS data, so it is not looked up again
until it is stale, counting as neither a success nor a failure. Tests
reach fake WHOIS servers through the client's DialContext.
Model: opus-5-5
When the delete of a peer's live routes after its session went down
failed, the down was dropped and the routes stayed. The prefix handler
now keeps such a peer, once, with the updates it sent after the down,
for the next flush, which retries the delete before it writes them; each
failure is logged. At most the last 25,000 of those updates are kept,
outside the batch, so they do not start a flush. The delete runs in
chunks of at most 10,000 routes per table, taken in prefix order
through the peer's key, each chunk in its own transaction, so the
database lock is released between chunks and a failure undoes at most
one chunk.
Model: opus-5-5
The live route tables lose their random UUID key; the row id the
prefix-length sample draws is declared as the integer key. The prefix
tables are keyed by prefix and keep only first_seen; a batch inserts only
prefixes not stored yet. A route update sets only the columns that
change. Each kept index names the queries using it. Every write goes
through one connection kept for writing, with a 1 GiB page cache; reading
connections get 16 MiB, heap limits rise to 1.5 and 2 GiB (README
"Memory"). Batch writes go in key order. A benchmark run by hand writes
250,000 updates on 5 million routes faster than the feed's 10,000 a
second; a live run follows.
Model: opus-5-5
Shutdown stopped the handlers before the streamer, so updates still in the
handler queues, up to 20,000 per queue, were not written, and a lost
withdrawal left its route in the table after a restart.
Shutdown now stops the streamer first. Streamer.Stop queues the STATE
messages it holds for a full queue, closes the queues and returns once
every worker has handed its handler everything on its queue; then each
handler writes what it holds before its Stop returns. PeeringHandler.Stop
now waits for its loop's final write instead of running a second one
beside it.
Model: opus-5-5
A path ending in an AS set, such as [64496, 64497, [64498, 64499]], was
flattened into the sequence, so the route was stored as originated by the
last set member and the peering handler recorded peerings the path does not
show. ASPath now keeps the sequence and a trailing set of several members
apart; Origin() gives the last AS of the sequence, or 0 (no known origin)
for such a set. The handlers use it; peerings come from the sequence only.
The pages show origin 0 as no known origin, and the IP lookup asks no WHOIS
lookup for it. Empty paths are unchanged.
Model: opus-5-5
Takes .golangci.yml, .dockerignore, .gitignore, .editorconfig, the
workflow and REPO_POLICIES.md from sneak/prompts 0653579, with this
repository's own entries after the shared content. Lint (golangci-lint
v2.14.0) and test are Dockerfile phases that script/lint and script/test
build alone with a new CACHEBUST value, keeping Go's and golangci-lint's
caches; the image build depends on both. make fmt and make fmt-check also
format the markdown with prettier, pinned in yarn.lock. The agent
guidance moves to AGENTS.md. The Go code is fixed for what the new
configuration finds, and the AS and prefix detail reads now return the
rows' error after their loops.
Model: opus-5-5
RIS sends no withdrawals for the routes of a peer whose BGP session goes
down, so routewatch kept them.
The streamer now hands STATE messages to the prefix handler. They are never
dropped for backpressure: one that finds the queue full is held and queued
first once there is room, and the handler's other messages are dropped
meanwhile, so none overtakes it. Nothing waits. On a down, the prefix
handler drops the peer's pending updates and its next flush deletes all the
peer's live routes before the batch's other writes, keeping the in-memory
counts exact. The live route tables are unique on peer and prefix, which
also finds a peer's routes.
Model: opus-5-5
The AS, peer and prefix handlers wrote their batch to the database while
holding the mutex HandleMessage needs, so while a write ran nothing was
taken off their queues and the streamer dropped updates.
Each handler now writes only from its flush goroutine: it takes the batch
under the mutex, leaves an empty one in its place, and writes after
releasing it. A full batch wakes the flush goroutine. While a write runs the
next batch may grow to ten batch sizes; at that size HandleMessage waits, so
the queue fills and the streamer drops as before. README "Memory" gives the
numbers and their cost.
Model: opus-5-5
A peer has at most one route per prefix, but both live route tables were
unique on prefix, origin AS and peer. A peer that changed a prefix's origin
AS left its old route behind, and a withdrawal sent in the same message as
announcements was given their origin AS and missed the withdrawn route.
Both tables are now unique on prefix and peer. An announcement replaces the
peer's row for the prefix, origin AS included; a withdrawal deletes it
without looking at an origin AS. The route counts and prefix distribution
kept in memory stay exact, for IPv4 and IPv6.
Model: opus-5-5
The prefix page looked up each origin AS's details only to show its handle and description, and that lookup reads every live route the AS originates. `GetPrefixDetailsContext` already selected the handle and description and dropped them; it now returns them keyed by AS number, and the page uses them. The origin list now also shows the description after the handle, which it never rendered. The JSON prefix answer is unchanged. The test serves the page from a store whose AS lookup always fails.
Model: opus-5-5
`Server.Start` opened its port inside a goroutine and returned nil at once, so a port already in use was only logged and the daemon kept running with no web server. It now opens the listener before returning and returns that error; serving runs in the background on the open listener. `CLIEntry` turns an error from `Run` into an fx shutdown with exit status 1, after the normal stop sequence, so a container restart policy can act. A serving error after a successful listen is still only logged.
Model: opus-5-5
The prefix handler kept one update per prefix in each batch across all peers, chosen by a timestamp cut to whole seconds, so most peers' announcements and withdrawals were lost and the first update in a second won. A batch now keeps the last update per prefix and peer in feed order. A message's withdrawals are queued before its announcements, so a prefix both withdrawn and announced in one message keeps its route (RFC 4271 section 4.3). Tests run each case through the handler into a real database and check the in-memory counts.
Model: opus-5-5
Looking up an address on /ip/ and /api/v1/ip/ read every IPv6 live route and about half of the IPv4 range index, so on a day-sized database it passed the 30-second request timeout. The lookup is now one function for both families: from the longest mask length down, it looks up the address's network prefix on the existing prefix index, and the first live route wins.
The feed sends IPv6 withdrawals uncompressed, so they never matched the stored compressed prefix and never removed a route. Prefixes are now stored in the text form net/netip prints, so IPv6 withdrawals take effect.
ip_start, ip_end and GetASInfoForIP are removed; nothing read them. A database from before this change must be deleted.
Model: opus-5-5
A set but invalid PORT (anything but plain digits from 1 to 65535) or a relative XDG_DATA_HOME now stops the start before the database opens; before, a bad PORT left the daemon running without HTTP. entrypoint.sh refuses a MALLOC_ARENA_MAX that is not a positive whole number, since glibc ignores a bad one silently. The HEALTHCHECK probes the port PORT names, 8080 when unset. PORT is now read in internal/config, so server.New takes the config.
The README gives the real Linux state directory, lists XDG_DATA_HOME, and adds "Running under upaas": port, volume, environment, the 5g memory limit and the health check.
Unverified: the 5g memory limit could not be exercised on the build host.
Model: opus-5-5
Each handler queue held 100,000 message pointers; a message is retained
until the slowest handler drains it, so all four full was a derived worst
case near 800 MiB. Twenty thousand is about four seconds of feed at peak
and caps that at roughly 160 MiB. The streamer already drops rather than
blocks on a full queue, so the smaller bound is safe.
Batch sizes are unchanged. The largest, asnBatchSize, is 30,000 and now
exceeds its queue, but each queued message contributes every ASN in its
path, and every handler also flushes on its own timer regardless of fill,
so batches still flush and no size change is warranted.
Model: opus-4-8
processPeerings now takes the accumulated AS-path map under the lock and
replaces it with a fresh empty one, so each path is processed once and the
map never grows past a single 30-second interval's traffic. HandleMessage
stops adding new paths once maxTrackedPaths (500000) is reached and counts
the drops, which are logged with each run. The 30-minute prune and its
ticker are removed as dead code under the swap, and the map mutex drops
from RWMutex to Mutex since the read path is gone. RecordPeering already
upserts last_seen, so stored peerings are unchanged.
Model: opus-4-8
TestRouteWatchLiveFeed streams the live RIPE RIS feed for a few seconds, so
the default test run depended on the network and tripped the race detector.
It now skips under testing.Short(), and script/test passes -short in both
go test lines, so make check runs offline and gives the same result every
time. The README says how to run the live test on demand. No non-test code
changed.
The race itself is not fixed: the test reads the peeringHandler field of
RouteWatch while Run is still setting it. Nothing outside the test reads
that field concurrently.
Model: opus-4-8 (implementation, review); fable-5-1 (summary)
Under heavy write load, 30 seconds is too long between checkpoints,
causing the WAL to grow and slow down read queries. More aggressive
checkpointing keeps the WAL small and maintains read performance.
clawbot
sneak