The prefix, AS and peer handlers skipped a timed write unless one
interval had passed since the last write ended. The next tick came a
little less than one interval after that, so every other tick was
skipped. Each tick now writes. A full batch's write restarts the timer,
so the next timed write comes one interval after that write began.
Comparing clock readings would not do: two ticks one interval apart can
read a hair under it. flushBatch no longer reports whether it wrote a
batch, since nothing reads that. A new test runs the loop on a fake
clock and fails on the old code.
Model: opus-5-5
A write can run past the stop deadline while a full batch's wake-up or a
tick waits beside Stop's context. The select picked one at random, so about
half the time the AS, peer or peering handler started another write after
the deadline. Each loop now takes a waiting stop first, as the prefix
handler's does. The prefix, AS and peer handlers' flush loops were
near-copies, so they are now one function, flushLoop. A new test for each
handler fails about half its runs on the old code.
The AS and peer handler tests give the flush timer an interval longer than
the test. A timer flush during the fill can no longer hang
TestPeerHandlerWaitsAtTheMaximumBatchSize.
Model: opus-5-5
When a write ran past the stop deadline, a full batch's wake-up or a tick
could wait beside Stop's context. The flush goroutine's select picked one of
them at random, so about half the time it wrote that batch after the
deadline. That was the intermittent failure of
TestPrefixHandlerSkipsItsLastWriteAfterTheStopDeadline. The flush goroutine
now takes a waiting stop first. A new test hands over a full batch while a
write is held; it fails about half its runs on the old code.
TestPrefixHandlerSkipsItsLastWriteAfterTheStopDeadline and
TestPrefixHandlerStopLogsTheTableChangesLeftUnwritten also failed when the
flush timer fired between two of their steps. Both now give the handler a
timer interval longer than the test.
Model: opus-5-5
The prefix handler applies each update to the routing table in memory as
it handles it, sorted into announcements and withdrawals as for the live
route tables. Each flush first writes at most 20,000 of the table's
changed pairs, with every new peer; the rest follow in the next flushes.
A failed write puts its changes and new peers back in the table. Stop
makes one last flush, and logs the changed pairs left.
Startup reads the table before the streamer connects, and logs its pairs,
peers and time. The per-peer counts are a slice, so the read takes about
a third less time. The health check's start period is 60 seconds, so a
full table's read fits before upaas looks.
Model: opus-5-5
A new routes table holds one row per prefix and origin AS: the peer set
as the bitset's bytes, and the two kept routes, each with its peer's
number, AS path and time in seconds. A new route_peers table holds each
peer's number and address.
WriteRouteChanges writes one TakeChanges result in a transaction.
ReadRouteTable reads both tables into a new routing table, through the
new Table.Load, which records no change. It fails on a row that names a
peer number missing from route_peers. Nothing uses them yet.
Model: opus-5-5
A new package holds every prefix some peer carries. For each prefix and
origin AS it keeps the set of peers that carry it, one bit per peer,
and the two newest of their routes in full. It applies announcements,
withdrawals and peers' sessions going down by the rule confirmed on the
issue, and records which pairs changed for the database writes to come.
It answers the IP lookup, an AS's prefixes, a prefix's origins, a sample
of one mask length, and the counts the stats show. Nothing uses it yet.
The peer sets use github.com/bits-and-blooms/bitset. The kept routes'
times are held in order in github.com/google/btree, so the oldest and
newest need no scan.
Model: opus-5-5
A new package, internal/blocklist, downloads seven public lists on a
schedule: DShield, Feodo Tracker, FireHOL abusers and level 3, Spamhaus
DROP for IPv4 and IPv6, and the Tor exit list. It keeps the last good
copy of each in memory and on disk. A download sends back the
Last-Modified and ETag the publisher gave for the copy, also kept on
disk.
Every IP lookup result carries blocklists. An address with no live route
now gets a result too, without route fields, so the lookup no longer
answers 404. A successful answer carries Cache-Control, private for the
caller's own address. The address sets come from go4.org/netipx.
Model: opus-5-5
The home page gets an IP information API card. It shows the curl
command that looks up the visitor's own address at /ip/{addr}. The page
then makes that same request and shows the JSON answer, pretty-printed.
An error answer, such as a 404 for an address with no route, is shown
the same way. A request that fails shows an error line.
The server gives the page the visitor's address, found as /ip/ finds
it, so a trusted proxy's X-Forwarded-For is honoured. The command takes
its origin from the browser, so it names the host the visitor used.
Model: opus-5-5
The IPv4 and IPv6 prefix lists on the AS page are each a <details>
element. Its <summary> holds the list's heading and count, so a click on
the heading folds or unfolds the list without a script. Both lists start
open. The peer list is unchanged.
The summary keeps the look of the other section headings. It hides the
browser's own marker, which not every browser shows on a flex element,
and draws a triangle before the heading that points down while the list
is open.
Model: opus-5-5
The AS page sorted its IPv4 and IPv6 prefix lists by network address,
then by mask length. It now sorts them by mask length, shortest first,
and by network address within one mask length, so a /16 comes before a
/24. The sort function is renamed from sortRoutesByNetwork to
sortRoutesByMaskLength to say so.
The handler of /api/v1/as/{asn} does not sort its lists, so it is
unchanged.
Model: opus-5-5
Each page's script turns a one-character braille spinner next to the
status whenever a stats answer counts more messages than the one
before. Its colour cycles as rtnetmon's does. A separate check runs
every second. Once no such answer has come for three refresh periods,
15 seconds on / and 6 on /status, it shows "stale" and the seconds
since, in the page's disconnected colour. A failed or hung request
counts as no fresh answer. On / the card has no room beside
"Connected", so the indicator sits on the line under it. A test renders
both pages and checks the indicator follows the status element.
Model: opus-5-5
LACNIC's answer for an AS that NIC.br registers is NIC.br's answer with
each accented letter replaced by U+FFFD, so a name such as AS52751's
cannot be recovered from it. When LACNIC's answer carries NIC.br's
copyright notice, the client now also asks whois.registro.br for the AS
and takes the fields from that answer, which is Latin-1 and decoded as
before. When that query fails, when the answer has no fields, or when
NIC.br says the client is over its query limit, LACNIC's fields are kept
and a warning is logged. After an over-limit answer the client skips
NIC.br for an hour. The tests use recorded answers for AS52751.
Model: opus-5-5
parseCSV skipped a record on any read error and asked for the next one.
When the download of as.csv broke off, the response body returned the
same error on every read, so the generator logged it forever and make
asupdate never finished.
parseCSV now skips only a record that does not parse, a csv.ParseError.
Any other read error ends it with that error, so the generator exits
non-zero and script/asupdate keeps the old list.
Model: opus-5-5
make asupdate piped the generator into gzip, so the target saw gzip's
exit status. A failed download replaced pkg/asinfo/asdata.json.gz with
an empty file and the target still exited 0.
The target now runs the new script/asupdate. It writes the output of
cmd/asinfo-gen to a temporary file. Only when the generator succeeds
does it compress the list and move it into place. Otherwise it leaves
the file alone and exits non-zero. The generator now fails when as.csv
holds no AS.
The generator reads as.csv from ipverse/as-metadata, the source's new
name. The list it builds there is unchanged, so no new list is
committed.
Model: opus-5-5
APNIC and RIPE start their answer for an AS with the as-block object of
the block the AS is in. The parser kept the first value of each field
across the whole answer, so it stored the block's dates as the AS's.
The parser now reads an answer from RIPE, APNIC or AFRINIC object by
object. It takes the fields from the aut-num object and the organisation
object it names, and leaves out every other object. The irt object's
address therefore no longer joins the organisation's. ARIN and LACNIC
answers are parsed as before.
Model: opus-5-5
The built-in AS list had no letter outside ASCII, and make asupdate
rejected today's as.csv, which added a country code column.
The generator now checks only the first three header columns and reads
past the rest; the parsing moved into parseCSV. The list is regenerated.
The new source spells some names differently and no longer lists AS 0
or the documentation AS numbers, so the tests that relied on those
entries changed.
A WHOIS answer that is not valid UTF-8 is now decoded as Latin-1, which
RIPE and APNIC send, before it is parsed and stored. LACNIC answers in
UTF-8.
Model: opus-5-5
The Age column printed Go's duration format for every plural age, such
as "5m0s ago" or "360h0m0s ago".
The templates now register humanize.Time from go-humanize, which the
server already uses, as the timeSince function, and the hand-written
function is gone. Ages read "5 minutes ago", "3 hours ago" or "2 weeks
ago". An age under a minute reads "now" or in seconds instead of "just
now". An age over a month reads in months or years instead of a date.
Model: opus-5-5
Some RIS collectors send updates days old while they catch up on a
backlog. Every update to a stored AS, peering or peer set last_seen to
the update's time, even an older one, and never lowered first_seen. An
AS page could then list a peer first seen after it was last seen.
The update statements for asns, peerings and bgp_peers now keep the
earlier of the stored and new first_seen and the later of the stored
and new last_seen, with MIN and MAX in the SQL. The batch and the
single-row writes share each statement. GetOrCreateASN returns the
times the row now holds. Route times are unchanged.
Model: opus-5-5
The status page showed route churn per second as a one-minute moving
average, so a quiet feed gave a fraction such as 0.1. The average was
correct.
The metrics tracker now also counts prefix announcements and
withdrawals, from all peers, in each clock minute. /api/v1/stats
reports the count for the last full minute as route_churn_per_min in
stream, in place of route_churn_per_sec. The status page shows it as
"Route Churn/min".
The status and home pages round messages and updates per second to
whole numbers.
Model: opus-5-5
A failed lookup now makes the fetcher's wait 1.5 times as long, up to
one minute instead of five. A success still halves it.
RIPE counts the person and role objects an answer holds against a daily
limit per IP address, and refuses an address for good that passes it
too often. RIPE, APNIC and AFRINIC are now asked with --no-personal,
which leaves those objects out but keeps the organisation object.
An answer with RIPE's access-denied or limit-reached error line is now
reported as a refusal holding that line, not as an answer with no
fields.
Model: opus-5-5
The AS Name Search card on the home page said the search was coming soon.
It now asks the new GET /api/v1/search/as?q=<text> and lists the matches
as links to their AS pages. asinfo.Search ignores case, returns the first
matches up to a limit in AS number order, and counts all matches. An
empty q answers 400.
When the database has no record of an AS, its page and /api/v1/as/{asn}
take its handle and description from asinfo and still list its prefixes.
They answer 404 only for an AS with no record, no asinfo entry and no
prefix. AS 0 means no known origin: it always answers 404, and the search
never lists it.
Model: opus-5-5
Initialize scanned the row of PRAGMA wal_checkpoint(TRUNCATE), which is
the busy flag, the frames in the WAL and the frames copied, into
wal_pages, checkpointed and moved; and a completed TRUNCATE reads 0 for
both counts. The line now logs busy, wal_frames and checkpointed_frames,
plus wal_bytes, the WAL file's size, which New reads before it opens the
database because the open writes a frame to the WAL. The line is a
warning when busy is 1. Tests open a database a second time while the
first stays open, with and without a read open on the first, and check
the line's level and figures.
Model: opus-5-5
REPO_POLICIES.md requires a Rationale section saying why the program
exists. RIS Live sends only the changes to the routing table, so it
cannot answer which AS an address is routed to now; routewatch applies
the changes to its own tables and answers from them. The section says
those tables hold only the routes announced since routewatch first
started, and miss the updates it drops under load or while it is
stopped.
Model: opus-5-5
An AS path that holds AS 0 anywhere, a member of an AS set included, now
decodes as an empty path. RFC 7607 makes such a path malformed and its
announcements withdrawals. An empty path already gets that handling:
the prefix handler deletes the peer's route for the prefix, and the AS
and peering handlers record nothing from it. Before, a path such as
[64496, 0, 64497] stored the route with origin AS64497, recorded both
ASes, and recorded the peerings between the path's other ASes.
Model: opus-5-5
The daemon's database is db.sqlite with its -wal and -shm files, which
the .db patterns in .dockerignore and .gitignore did not match, so a
database left in the clone shipped in the image's source archive.
.dockerignore now also excludes the logs and the .exe, .dll, .so and
.dylib files .gitignore already ignores. A test opens a database and
checks that .dockerignore has an any-depth pattern for each of its
files.
Model: opus-5-5
UpdatePrefixesBatch now skips a prefix that does not parse, so it never
becomes a row in prefixes_v4 or prefixes_v6 and never counts in the prefix
total in the stats. Before, the prefix handler stored no route for such a
prefix and logged it once per prefix and peer, but still passed it to
UpdatePrefixesBatch, which stored it.
Model: opus-5-5
version.License, which the home and status page footers show, still
said WTFPL after LICENSE and the README moved to MIT. It now says MIT.
A test renders both pages and expects each footer to name MIT and not
WTFPL.
Model: opus-5-5
recordSuccess and recordFailure appended a time on every fetch, and only
GetStats, run when a stats endpoint is requested, dropped the times older
than statsWindow, so a daemon nobody polled kept every one. Each now drops
those times before it appends. The drop takes them off the front of the
list instead of copying the rest: the times are in the order they were
recorded, and it now runs on every fetch.
The fetcher reads the time through a now field so a test can set the
clock. The test records 100,000 successes and failures a second apart
without asking for the stats and finds one hour's worth of each kept.
Model: opus-5-5
processPeerings replaced the path map every run but kept the count of
paths dropped while the map was full, so every later "Processed AS
peerings" line reported all drops since start next to that run's own
figures. The count is now set to 0 where the map is replaced, under the
same lock. A test fills the map, has one path dropped, runs the
processing and finds the count back at 0.
Model: opus-5-5
An announcement whose AS path is empty or ends in AS 0 now deletes the
peer's route for the prefix, as a withdrawal does. Before, it was neither
stored nor applied: the peer's old route stayed live, and within one
batch it cancelled the peer's earlier withdrawal of the prefix.
Model: opus-5-5
The read loop added to a handler's drop count atomically while the
statistics endpoint and the metrics log read it as a plain field, a data
race whenever messages were dropped. The count is now an atomic.Uint64,
read with Load. A test reads the handler statistics while the read loop
drops messages for a handler that never finishes one.
Run created the WHOIS fetcher and gave it to the HTTP server after the
server had started, while request handlers could read it. The fetcher is
now set before the server starts.
Model: opus-5-5
Every route was registered for GET only, so HEAD answered 405, and on
the JSON routes that 405 said "Internal server error". chi's GetHead
now serves HEAD with the GET handler; /ip and /ip/ register HEAD
themselves, as GetHead does not reach them. The /api/v1 and /ip route
groups answer a path they lack with a JSON 404 "Not found", and the
other methods the router knows (POST, PUT, PATCH, DELETE, OPTIONS,
TRACE, CONNECT) with a JSON 405 "Method not allowed" whose Allow header
names GET and HEAD. JSONValidationMiddleware gives a 4xx sent without a
JSON body the status's standard text instead of "Internal server error".
Model: opus-5-5
Each handler took the latest time per key in a batch and inserted it as
a new row's first_seen, so a key updated twice in one batch, seconds
apart, was stored first seen at the later time, and the IP lookup could
answer with a prefix first_seen after the route's last_seen.
The prefix handler now keeps the earliest time of all of the batch's
updates for each prefix. The AS, peer and peering handlers keep the
earliest and the latest time per key in database.SeenTimes, and
GetOrCreateASNBatch, UpdatePeerBatch and RecordPeering store a new row
first seen at the earliest and last seen at the latest. A stored row
still has only its last_seen set, to the latest.
Model: opus-5-5
The start hook registered its own handler for SIGINT and SIGTERM and
asked fx to shut down from it, although fx already stops the daemon on
both signals. A stop signal was handled twice, and the handler stayed
registered after the stop. fx now handles both signals alone, as
GO_HTTP_SERVER_CONVENTIONS.md "Signal Handling and Graceful Shutdown"
requires.
Model: opus-5-5
An IP lookup took any number of addresses and host names, gave each PTR
lookup until the request's 30-second limit, and looked up and named
every address a host name resolved to. It now answers 400 for more than
10 entries, or more than 10 addresses once the names are resolved,
before reading the database. Each PTR lookup gets 2 seconds; one that
runs out leaves the address's result without PTR names. Of a host
name's addresses only public ones are kept: global unicast and not
private. A name with none is reported as having no public address. The
tests give the server a fake resolver.
Model: opus-5-5
Shutdown passes its context to every stage: the streamer, the four
handlers, the WHOIS fetcher, the database maintainer and the HTTP server
each stop waiting when it ends. A handler then skips its last write and
logs how many updates it did not write, and the HTTP server closes the
connections still open. The database is checkpointed and closed last,
and left open once the deadline has passed, since a write may still
hold it. The stop hook gives Shutdown 8 seconds and fx 9, inside the 10
seconds upaas and docker stop allow, so a stop signal exits 0.
Model: opus-5-5
A failed database read for an address was listed in the answer's errors
as if the address had no route, so the lookup answered 404, or 200 next
to a routed address. Only a missing route or an invalid address is
listed now; any other read error is logged and answers 500 with the JSON
error body, as the AS and prefix JSON routes do.
Model: opus-5-5
Every answer carries the six security headers REPO_POLICIES.md lists.
The Content-Security-Policy allows inline content only with a nonce made
for that answer; the pages put it on their style and script blocks and
drop their four style attributes.
The request log and /ip/ take a forwarded address only from a proxy
listed in TRUSTED_PROXIES, which stops the start when set but invalid:
the last X-Forwarded-For entry that is not a trusted proxy. X-Real-IP
and True-Client-IP are no longer read.
The JSON routes and the health check send fixed messages in place of
database, parser and resolver errors, and log the detail.
Model: opus-5-5
The 5-second checkpoint ran in TRUNCATE mode, which waits up to the
5-second busy timeout for every open read while holding the database
lock every batch write needs. It now runs in PASSIVE mode, which never
waits. A WAL file over 64 MiB instead gets one TRUNCATE checkpoint whose
wait for reads is capped at 500 ms and logged; it empties the file once
no read is open, and while reads follow one another with no pause it
can give up every time. A checkpoint that leaves part of the WAL
uncopied returns ErrCheckpointIncomplete, which the maintainer logs as
incomplete instead of completed. The writing connection's
journal_size_limit is 64 MiB.
Model: opus-5-5
The AS page and /api/v1/as/{asn} grouped every route the AS originates
by prefix, reading one table row per peer and prefix and sorting them
all. They now take the AS's prefixes one at a time: one search of a new
index on origin_asn and prefix, which replaces the origin_asn index and
is written exactly when it was, and one search of the prefix and
last_updated index for the time of the prefix's newest route. Neither
reads the table. For a prefix more than one AS originates, that time is
now the newest route's from any origin. A hand-run benchmark reads an
AS of 5,000 prefixes from 500 peers.
Model: opus-5-5
The client kept every line a WHOIS server sent until the server closed
the connection or 30 seconds passed, so a server that kept sending
could fill memory. It now reads at most 1 MiB of an answer, and an
answer that reaches that is an error. A "refer:" line in IANA's answer
naming a server none of the five RIRs run was dialled as given, on any
host and port; it is now an error, so the client only connects to IANA
and the five RIR servers. Both errors are failed lookups, which the
fetcher retries later.
Model: opus-5-5
A lookup whose RIR query failed, or whose RIR answer carried none of the
WHOIS fields (a refusal of service, for one), returned IANA's answer as
a success. The fetcher stored it as complete with blank fields, did not
look the AS up again for 30 days, and shortened its interval. Such a
lookup is now an error: the fetcher backs off and leaves the row as it
was. The fields are read from the RIR's answer alone. An AS IANA refers
to no RIR is stored with no WHOIS data, so it is not looked up again
until it is stale, counting as neither a success nor a failure. Tests
reach fake WHOIS servers through the client's DialContext.
Model: opus-5-5
When the delete of a peer's live routes after its session went down
failed, the down was dropped and the routes stayed. The prefix handler
now keeps such a peer, once, with the updates it sent after the down,
for the next flush, which retries the delete before it writes them; each
failure is logged. At most the last 25,000 of those updates are kept,
outside the batch, so they do not start a flush. The delete runs in
chunks of at most 10,000 routes per table, taken in prefix order
through the peer's key, each chunk in its own transaction, so the
database lock is released between chunks and a failure undoes at most
one chunk.
Model: opus-5-5
The test and build stages move from golang:1.24-bookworm, a release
line that no longer gets security fixes, to golang:1.27.2-bookworm,
pinned by digest. The go directive in go.mod moves from 1.24.4 to
1.27.1, the Go in the pinned lint image, so no stage downloads another
toolchain. One test starts its goroutine with sync.WaitGroup.Go, which
the linter asks for once the module targets Go 1.25 or later.
Model: opus-5-5
chi matches routes against the escaped path when a request has one and
hands the handlers their URL parameters still percent-encoded. The home
page's lookup form encodes the address it asks for, so every IPv6 lookup
answered 400, as did a comma-separated address list or a prefix that a
client percent-encoded. The router now matches against the decoded path,
so every handler reads its parameters decoded.
Model: opus-5-5
README "HTTP Endpoints" now lists every route the server registers with
what it answers, errors included: / is a home page and /ip/{addr} answers
JSON, neither redirects; /ip/ looks up the caller; the IP endpoints take
host names and comma-separated lists; 400, 404 and 504 are described, and
the health check's envelope. "Database Schema" matches schema.sql, and
other sentences the code contradicted are corrected. TODO.md gets a dated
line for each issue landed since 2026-10-03, with Status and Next Step
current.
Model: opus-5-5
The live route tables lose their random UUID key; the row id the
prefix-length sample draws is declared as the integer key. The prefix
tables are keyed by prefix and keep only first_seen; a batch inserts only
prefixes not stored yet. A route update sets only the columns that
change. Each kept index names the queries using it. Every write goes
through one connection kept for writing, with a 1 GiB page cache; reading
connections get 16 MiB, heap limits rise to 1.5 and 2 GiB (README
"Memory"). Batch writes go in key order. A benchmark run by hand writes
250,000 updates on 5 million routes faster than the feed's 10,000 a
second; a live run follows.
Model: opus-5-5
The database maintainer ran Vacuum and Analyze without the database lock
every batch write holds, so on the live feed every incremental vacuum
failed with "database is locked", and an hourly ANALYZE made a batch write
fail and lose its updates. Vacuum, Analyze and Checkpoint now take that
lock, so maintenance and batch writes wait for each other. Analyze runs
PRAGMA optimize with an analysis limit, which analyzes only tables whose
size changed a lot, so it holds the lock briefly however large the file.
Model: opus-5-5
Shutdown stopped the handlers before the streamer, so updates still in the
handler queues, up to 20,000 per queue, were not written, and a lost
withdrawal left its route in the table after a restart.
Shutdown now stops the streamer first. Streamer.Stop queues the STATE
messages it holds for a full queue, closes the queues and returns once
every worker has handed its handler everything on its queue; then each
handler writes what it holds before its Stop returns. PeeringHandler.Stop
now waits for its loop's final write instead of running a second one
beside it.
Model: opus-5-5
A path ending in an AS set, such as [64496, 64497, [64498, 64499]], was
flattened into the sequence, so the route was stored as originated by the
last set member and the peering handler recorded peerings the path does not
show. ASPath now keeps the sequence and a trailing set of several members
apart; Origin() gives the last AS of the sequence, or 0 (no known origin)
for such a set. The handlers use it; peerings come from the sequence only.
The pages show origin 0 as no known origin, and the IP lookup asks no WHOIS
lookup for it. Empty paths are unchanged.
Model: opus-5-5
clawbot