On next since the last merge to main: the routing table of #122, your 22 October reports and UI requests, and the defects found on the way.
Routing table: kept in memory with only redundant routes pruned, written to the database every second, read at startup; every page and API answer reads it. The per-peer route tables are gone. A full table is about 1 GB of memory and 450 MB on disk; the memory limits are set for it.
Pages: cards on / with the newest and oldest routes and ASes, the largest AS, bogon and multiple-origin counts with their own list pages, and well-known networks; AS name search; a freshness spinner by the connection status; ages in words; the AS page's prefixes sorted by mask length and folding; a live curl example of the IP API on /.
IP API: each address lists the public blocklists that name it, and an address with no route answers with its tags.
WHOIS: slower backoff, at most a minute; no personal-data queries to RIPE, APNIC and AFRINIC; names keep their accents.
Stats: stream.route_churn_per_min, a whole count, replaces route_churn_per_sec.
Shutdown: no write starts after the stop deadline; timed writes happen on every tick.
For a deploy:
Start on a fresh database: an existing file keeps the old per-peer tables, tens of GB on your instance, which nothing reads.
routewatch downloads the blocklists itself into a blocklists directory in its state directory.
The health check start period is now 60 seconds, so the table's read fits before upaas checks.
TRUSTED_PROXIES must name the reverse proxy, as before.
Model: opus-5-5
On `next` since the last merge to `main`: the routing table of https://git.eeqj.de/sneak/routewatch/issues/122, your 22 October reports and UI requests, and the defects found on the way.
- Routing table: kept in memory with only redundant routes pruned, written to the database every second, read at startup; every page and API answer reads it. The per-peer route tables are gone. A full table is about 1 GB of memory and 450 MB on disk; the memory limits are set for it.
- Pages: cards on `/` with the newest and oldest routes and ASes, the largest AS, bogon and multiple-origin counts with their own list pages, and well-known networks; AS name search; a freshness spinner by the connection status; ages in words; the AS page's prefixes sorted by mask length and folding; a live `curl` example of the IP API on `/`.
- IP API: each address lists the public blocklists that name it, and an address with no route answers with its tags.
- WHOIS: slower backoff, at most a minute; no personal-data queries to RIPE, APNIC and AFRINIC; names keep their accents.
- Stats: `stream.route_churn_per_min`, a whole count, replaces `route_churn_per_sec`.
- Shutdown: no write starts after the stop deadline; timed writes happen on every tick.
For a deploy:
- Start on a fresh database: an existing file keeps the old per-peer tables, tens of GB on your instance, which nothing reads.
- routewatch downloads the blocklists itself into a `blocklists` directory in its state directory.
- The health check start period is now 60 seconds, so the table's read fits before upaas checks.
- `TRUSTED_PROXIES` must name the reverse proxy, as before.
Model: opus-5-5
The AS Name Search card on the home page said the search was coming soon.
It now asks the new GET /api/v1/search/as?q=<text> and lists the matches
as links to their AS pages. asinfo.Search ignores case, returns the first
matches up to a limit in AS number order, and counts all matches. An
empty q answers 400.
When the database has no record of an AS, its page and /api/v1/as/{asn}
take its handle and description from asinfo and still list its prefixes.
They answer 404 only for an AS with no record, no asinfo entry and no
prefix. AS 0 means no known origin: it always answers 404, and the search
never lists it.
Model: opus-5-5
A failed lookup now makes the fetcher's wait 1.5 times as long, up to
one minute instead of five. A success still halves it.
RIPE counts the person and role objects an answer holds against a daily
limit per IP address, and refuses an address for good that passes it
too often. RIPE, APNIC and AFRINIC are now asked with --no-personal,
which leaves those objects out but keeps the organisation object.
An answer with RIPE's access-denied or limit-reached error line is now
reported as a refusal holding that line, not as an answer with no
fields.
Model: opus-5-5
The status page showed route churn per second as a one-minute moving
average, so a quiet feed gave a fraction such as 0.1. The average was
correct.
The metrics tracker now also counts prefix announcements and
withdrawals, from all peers, in each clock minute. /api/v1/stats
reports the count for the last full minute as route_churn_per_min in
stream, in place of route_churn_per_sec. The status page shows it as
"Route Churn/min".
The status and home pages round messages and updates per second to
whole numbers.
Model: opus-5-5
Some RIS collectors send updates days old while they catch up on a
backlog. Every update to a stored AS, peering or peer set last_seen to
the update's time, even an older one, and never lowered first_seen. An
AS page could then list a peer first seen after it was last seen.
The update statements for asns, peerings and bgp_peers now keep the
earlier of the stored and new first_seen and the later of the stored
and new last_seen, with MIN and MAX in the SQL. The batch and the
single-row writes share each statement. GetOrCreateASN returns the
times the row now holds. Route times are unchanged.
Model: opus-5-5
The Age column printed Go's duration format for every plural age, such
as "5m0s ago" or "360h0m0s ago".
The templates now register humanize.Time from go-humanize, which the
server already uses, as the timeSince function, and the hand-written
function is gone. Ages read "5 minutes ago", "3 hours ago" or "2 weeks
ago". An age under a minute reads "now" or in seconds instead of "just
now". An age over a month reads in months or years instead of a date.
Model: opus-5-5
The built-in AS list had no letter outside ASCII, and make asupdate
rejected today's as.csv, which added a country code column.
The generator now checks only the first three header columns and reads
past the rest; the parsing moved into parseCSV. The list is regenerated.
The new source spells some names differently and no longer lists AS 0
or the documentation AS numbers, so the tests that relied on those
entries changed.
A WHOIS answer that is not valid UTF-8 is now decoded as Latin-1, which
RIPE and APNIC send, before it is parsed and stored. LACNIC answers in
UTF-8.
Model: opus-5-5
APNIC and RIPE start their answer for an AS with the as-block object of
the block the AS is in. The parser kept the first value of each field
across the whole answer, so it stored the block's dates as the AS's.
The parser now reads an answer from RIPE, APNIC or AFRINIC object by
object. It takes the fields from the aut-num object and the organisation
object it names, and leaves out every other object. The irt object's
address therefore no longer joins the organisation's. ARIN and LACNIC
answers are parsed as before.
Model: opus-5-5
make asupdate piped the generator into gzip, so the target saw gzip's
exit status. A failed download replaced pkg/asinfo/asdata.json.gz with
an empty file and the target still exited 0.
The target now runs the new script/asupdate. It writes the output of
cmd/asinfo-gen to a temporary file. Only when the generator succeeds
does it compress the list and move it into place. Otherwise it leaves
the file alone and exits non-zero. The generator now fails when as.csv
holds no AS.
The generator reads as.csv from ipverse/as-metadata, the source's new
name. The list it builds there is unchanged, so no new list is
committed.
Model: opus-5-5
parseCSV skipped a record on any read error and asked for the next one.
When the download of as.csv broke off, the response body returned the
same error on every read, so the generator logged it forever and make
asupdate never finished.
parseCSV now skips only a record that does not parse, a csv.ParseError.
Any other read error ends it with that error, so the generator exits
non-zero and script/asupdate keeps the old list.
Model: opus-5-5
LACNIC's answer for an AS that NIC.br registers is NIC.br's answer with
each accented letter replaced by U+FFFD, so a name such as AS52751's
cannot be recovered from it. When LACNIC's answer carries NIC.br's
copyright notice, the client now also asks whois.registro.br for the AS
and takes the fields from that answer, which is Latin-1 and decoded as
before. When that query fails, when the answer has no fields, or when
NIC.br says the client is over its query limit, LACNIC's fields are kept
and a warning is logged. After an over-limit answer the client skips
NIC.br for an hour. The tests use recorded answers for AS52751.
Model: opus-5-5
Each page's script turns a one-character braille spinner next to the
status whenever a stats answer counts more messages than the one
before. Its colour cycles as rtnetmon's does. A separate check runs
every second. Once no such answer has come for three refresh periods,
15 seconds on / and 6 on /status, it shows "stale" and the seconds
since, in the page's disconnected colour. A failed or hung request
counts as no fresh answer. On / the card has no room beside
"Connected", so the indicator sits on the line under it. A test renders
both pages and checks the indicator follows the status element.
Model: opus-5-5
The AS page sorted its IPv4 and IPv6 prefix lists by network address,
then by mask length. It now sorts them by mask length, shortest first,
and by network address within one mask length, so a /16 comes before a
/24. The sort function is renamed from sortRoutesByNetwork to
sortRoutesByMaskLength to say so.
The handler of /api/v1/as/{asn} does not sort its lists, so it is
unchanged.
Model: opus-5-5
The IPv4 and IPv6 prefix lists on the AS page are each a <details>
element. Its <summary> holds the list's heading and count, so a click on
the heading folds or unfolds the list without a script. Both lists start
open. The peer list is unchanged.
The summary keeps the look of the other section headings. It hides the
browser's own marker, which not every browser shows on a flex element,
and draws a triangle before the heading that points down while the list
is open.
Model: opus-5-5
The home page gets an IP information API card. It shows the curl
command that looks up the visitor's own address at /ip/{addr}. The page
then makes that same request and shows the JSON answer, pretty-printed.
An error answer, such as a 404 for an address with no route, is shown
the same way. A request that fails shows an error line.
The server gives the page the visitor's address, found as /ip/ finds
it, so a trusted proxy's X-Forwarded-For is honoured. The command takes
its origin from the browser, so it names the host the visitor used.
Model: opus-5-5
A new package, internal/blocklist, downloads seven public lists on a
schedule: DShield, Feodo Tracker, FireHOL abusers and level 3, Spamhaus
DROP for IPv4 and IPv6, and the Tor exit list. It keeps the last good
copy of each in memory and on disk. A download sends back the
Last-Modified and ETag the publisher gave for the copy, also kept on
disk.
Every IP lookup result carries blocklists. An address with no live route
now gets a result too, without route fields, so the lookup no longer
answers 404. A successful answer carries Cache-Control, private for the
caller's own address. The address sets come from go4.org/netipx.
Model: opus-5-5
A new package holds every prefix some peer carries. For each prefix and
origin AS it keeps the set of peers that carry it, one bit per peer,
and the two newest of their routes in full. It applies announcements,
withdrawals and peers' sessions going down by the rule confirmed on the
issue, and records which pairs changed for the database writes to come.
It answers the IP lookup, an AS's prefixes, a prefix's origins, a sample
of one mask length, and the counts the stats show. Nothing uses it yet.
The peer sets use github.com/bits-and-blooms/bitset. The kept routes'
times are held in order in github.com/google/btree, so the oldest and
newest need no scan.
Model: opus-5-5
A new routes table holds one row per prefix and origin AS: the peer set
as the bitset's bytes, and the two kept routes, each with its peer's
number, AS path and time in seconds. A new route_peers table holds each
peer's number and address.
WriteRouteChanges writes one TakeChanges result in a transaction.
ReadRouteTable reads both tables into a new routing table, through the
new Table.Load, which records no change. It fails on a row that names a
peer number missing from route_peers. Nothing uses them yet.
Model: opus-5-5
The prefix handler applies each update to the routing table in memory as
it handles it, sorted into announcements and withdrawals as for the live
route tables. Each flush first writes at most 20,000 of the table's
changed pairs, with every new peer; the rest follow in the next flushes.
A failed write puts its changes and new peers back in the table. Stop
makes one last flush, and logs the changed pairs left.
Startup reads the table before the streamer connects, and logs its pairs,
peers and time. The per-peer counts are a slice, so the read takes about
a third less time. The health check's start period is 60 seconds, so a
full table's read fits before upaas looks.
Model: opus-5-5
When a write ran past the stop deadline, a full batch's wake-up or a tick
could wait beside Stop's context. The flush goroutine's select picked one of
them at random, so about half the time it wrote that batch after the
deadline. That was the intermittent failure of
TestPrefixHandlerSkipsItsLastWriteAfterTheStopDeadline. The flush goroutine
now takes a waiting stop first. A new test hands over a full batch while a
write is held; it fails about half its runs on the old code.
TestPrefixHandlerSkipsItsLastWriteAfterTheStopDeadline and
TestPrefixHandlerStopLogsTheTableChangesLeftUnwritten also failed when the
flush timer fired between two of their steps. Both now give the handler a
timer interval longer than the test.
Model: opus-5-5
clawbot
removed their assignment 2026-10-27 13:13:56 +01:00
sneak
was assigned by clawbot2026-10-27 13:13:56 +01:00
A write can run past the stop deadline while a full batch's wake-up or a
tick waits beside Stop's context. The select picked one at random, so about
half the time the AS, peer or peering handler started another write after
the deadline. Each loop now takes a waiting stop first, as the prefix
handler's does. The prefix, AS and peer handlers' flush loops were
near-copies, so they are now one function, flushLoop. A new test for each
handler fails about half its runs on the old code.
The AS and peer handler tests give the flush timer an interval longer than
the test. A timer flush during the fill can no longer hang
TestPeerHandlerWaitsAtTheMaximumBatchSize.
Model: opus-5-5
The prefix, AS and peer handlers skipped a timed write unless one
interval had passed since the last write ended. The next tick came a
little less than one interval after that, so every other tick was
skipped. Each tick now writes. A full batch's write restarts the timer,
so the next timed write comes one interval after that write began.
Comparing clock readings would not do: two ticks one interval apart can
read a hair under it. flushBatch no longer reports whether it wrote a
batch, since nothing reads that. A new test runs the loop on a fake
clock and fails on the old code.
Model: opus-5-5
The IP lookups, the AS and prefix pages and their API answers, the prefix-length pages and the stats read routes only from the routing table the prefix handler keeps. fx reads the table at startup and gives it to both. AS names, WHOIS details and first-seen times still come from the database.
The IP lookup names the origin AS most peers carry, with the time of the newest kept route. The prefix answers list peers whose routes are not kept without a path or time. The stats gain origins_without_kept_route. They no longer read route times from the per-peer tables, so the two helpers that did are gone.
Model: opus-5-5
The routing table in memory now holds every route, so the two per-peer route
tables go, with every write, delete, count and read that served them. The
prefix handler applies each update to the table, writes the table's changes
every second, and stores each new prefix with its first-seen time. The peer
delete, its retry on the next flush and the updates held for it go too. The
types and errors of the server's answers move to internal/server.
The image's memory settings fit a full table of about 1 GB: GOMEMLIMIT of
2.5 GiB, a 512 MiB page cache for the writing connection, and SQLite heap
limits of 768 MiB and 1 GiB. The README describes the new table, its
database copy and the memory budget.
Model: opus-5-5
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
On
nextsince the last merge tomain: the routing table of #122, your 22 October reports and UI requests, and the defects found on the way./with the newest and oldest routes and ASes, the largest AS, bogon and multiple-origin counts with their own list pages, and well-known networks; AS name search; a freshness spinner by the connection status; ages in words; the AS page's prefixes sorted by mask length and folding; a livecurlexample of the IP API on/.stream.route_churn_per_min, a whole count, replacesroute_churn_per_sec.For a deploy:
blocklistsdirectory in its state directory.TRUSTED_PROXIESmust name the reverse proxy, as before.Model: opus-5-5
The AS Name Search card on the home page said the search was coming soon. It now asks the new GET /api/v1/search/as?q=<text> and lists the matches as links to their AS pages. asinfo.Search ignores case, returns the first matches up to a limit in AS number order, and counts all matches. An empty q answers 400. When the database has no record of an AS, its page and /api/v1/as/{asn} take its handle and description from asinfo and still list its prefixes. They answer 404 only for an AS with no record, no asinfo entry and no prefix. AS 0 means no known origin: it always answers 404, and the search never lists it. Model: opus-5-5The AS page sorted its IPv4 and IPv6 prefix lists by network address, then by mask length. It now sorts them by mask length, shortest first, and by network address within one mask length, so a /16 comes before a /24. The sort function is renamed from sortRoutesByNetwork to sortRoutesByMaskLength to say so. The handler of /api/v1/as/{asn} does not sort its lists, so it is unchanged. Model: opus-5-5The home page gets an IP information API card. It shows the curl command that looks up the visitor's own address at /ip/{addr}. The page then makes that same request and shows the JSON answer, pretty-printed. An error answer, such as a 404 for an address with no route, is shown the same way. A request that fails shows an error line. The server gives the page the visitor's address, found as /ip/ finds it, so a trusted proxy's X-Forwarded-For is honoured. The command takes its origin from the browser, so it names the host the visitor used. Model: opus-5-5