REPO_POLICIES.md requires a Rationale section saying why the program
exists. RIS Live sends only the changes to the routing table, so it
cannot answer which AS an address is routed to now; routewatch applies
the changes to its own tables and answers from them. The section says
those tables hold only the routes announced since routewatch first
started, and miss the updates it drops under load or while it is
stopped.
Model: opus-5-5
An AS path that holds AS 0 anywhere, a member of an AS set included, now
decodes as an empty path. RFC 7607 makes such a path malformed and its
announcements withdrawals. An empty path already gets that handling:
the prefix handler deletes the peer's route for the prefix, and the AS
and peering handlers record nothing from it. Before, a path such as
[64496, 0, 64497] stored the route with origin AS64497, recorded both
ASes, and recorded the peerings between the path's other ASes.
Model: opus-5-5
The daemon's database is db.sqlite with its -wal and -shm files, which
the .db patterns in .dockerignore and .gitignore did not match, so a
database left in the clone shipped in the image's source archive.
.dockerignore now also excludes the logs and the .exe, .dll, .so and
.dylib files .gitignore already ignores. A test opens a database and
checks that .dockerignore has an any-depth pattern for each of its
files.
Model: opus-5-5
UpdatePrefixesBatch now skips a prefix that does not parse, so it never
becomes a row in prefixes_v4 or prefixes_v6 and never counts in the prefix
total in the stats. Before, the prefix handler stored no route for such a
prefix and logged it once per prefix and peer, but still passed it to
UpdatePrefixesBatch, which stored it.
Model: opus-5-5
version.License, which the home and status page footers show, still
said WTFPL after LICENSE and the README moved to MIT. It now says MIT.
A test renders both pages and expects each footer to name MIT and not
WTFPL.
Model: opus-5-5
recordSuccess and recordFailure appended a time on every fetch, and only
GetStats, run when a stats endpoint is requested, dropped the times older
than statsWindow, so a daemon nobody polled kept every one. Each now drops
those times before it appends. The drop takes them off the front of the
list instead of copying the rest: the times are in the order they were
recorded, and it now runs on every fetch.
The fetcher reads the time through a now field so a test can set the
clock. The test records 100,000 successes and failures a second apart
without asking for the stats and finds one hour's worth of each kept.
Model: opus-5-5
processPeerings replaced the path map every run but kept the count of
paths dropped while the map was full, so every later "Processed AS
peerings" line reported all drops since start next to that run's own
figures. The count is now set to 0 where the map is replaced, under the
same lock. A test fills the map, has one path dropped, runs the
processing and finds the count back at 0.
Model: opus-5-5
An announcement whose AS path is empty or ends in AS 0 now deletes the
peer's route for the prefix, as a withdrawal does. Before, it was neither
stored nor applied: the peer's old route stayed live, and within one
batch it cancelled the peer's earlier withdrawal of the prefix.
Model: opus-5-5
The read loop added to a handler's drop count atomically while the
statistics endpoint and the metrics log read it as a plain field, a data
race whenever messages were dropped. The count is now an atomic.Uint64,
read with Load. A test reads the handler statistics while the read loop
drops messages for a handler that never finishes one.
Run created the WHOIS fetcher and gave it to the HTTP server after the
server had started, while request handlers could read it. The fetcher is
now set before the server starts.
Model: opus-5-5
Every route was registered for GET only, so HEAD answered 405, and on
the JSON routes that 405 said "Internal server error". chi's GetHead
now serves HEAD with the GET handler; /ip and /ip/ register HEAD
themselves, as GetHead does not reach them. The /api/v1 and /ip route
groups answer a path they lack with a JSON 404 "Not found", and the
other methods the router knows (POST, PUT, PATCH, DELETE, OPTIONS,
TRACE, CONNECT) with a JSON 405 "Method not allowed" whose Allow header
names GET and HEAD. JSONValidationMiddleware gives a 4xx sent without a
JSON body the status's standard text instead of "Internal server error".
Model: opus-5-5
Each handler took the latest time per key in a batch and inserted it as
a new row's first_seen, so a key updated twice in one batch, seconds
apart, was stored first seen at the later time, and the IP lookup could
answer with a prefix first_seen after the route's last_seen.
The prefix handler now keeps the earliest time of all of the batch's
updates for each prefix. The AS, peer and peering handlers keep the
earliest and the latest time per key in database.SeenTimes, and
GetOrCreateASNBatch, UpdatePeerBatch and RecordPeering store a new row
first seen at the earliest and last seen at the latest. A stored row
still has only its last_seen set, to the latest.
Model: opus-5-5
The start hook registered its own handler for SIGINT and SIGTERM and
asked fx to shut down from it, although fx already stops the daemon on
both signals. A stop signal was handled twice, and the handler stayed
registered after the stop. fx now handles both signals alone, as
GO_HTTP_SERVER_CONVENTIONS.md "Signal Handling and Graceful Shutdown"
requires.
Model: opus-5-5
An IP lookup took any number of addresses and host names, gave each PTR
lookup until the request's 30-second limit, and looked up and named
every address a host name resolved to. It now answers 400 for more than
10 entries, or more than 10 addresses once the names are resolved,
before reading the database. Each PTR lookup gets 2 seconds; one that
runs out leaves the address's result without PTR names. Of a host
name's addresses only public ones are kept: global unicast and not
private. A name with none is reported as having no public address. The
tests give the server a fake resolver.
Model: opus-5-5
Shutdown passes its context to every stage: the streamer, the four
handlers, the WHOIS fetcher, the database maintainer and the HTTP server
each stop waiting when it ends. A handler then skips its last write and
logs how many updates it did not write, and the HTTP server closes the
connections still open. The database is checkpointed and closed last,
and left open once the deadline has passed, since a write may still
hold it. The stop hook gives Shutdown 8 seconds and fx 9, inside the 10
seconds upaas and docker stop allow, so a stop signal exits 0.
Model: opus-5-5
A failed database read for an address was listed in the answer's errors
as if the address had no route, so the lookup answered 404, or 200 next
to a routed address. Only a missing route or an invalid address is
listed now; any other read error is logged and answers 500 with the JSON
error body, as the AS and prefix JSON routes do.
Model: opus-5-5
Every answer carries the six security headers REPO_POLICIES.md lists.
The Content-Security-Policy allows inline content only with a nonce made
for that answer; the pages put it on their style and script blocks and
drop their four style attributes.
The request log and /ip/ take a forwarded address only from a proxy
listed in TRUSTED_PROXIES, which stops the start when set but invalid:
the last X-Forwarded-For entry that is not a trusted proxy. X-Real-IP
and True-Client-IP are no longer read.
The JSON routes and the health check send fixed messages in place of
database, parser and resolver errors, and log the detail.
Model: opus-5-5
The 5-second checkpoint ran in TRUNCATE mode, which waits up to the
5-second busy timeout for every open read while holding the database
lock every batch write needs. It now runs in PASSIVE mode, which never
waits. A WAL file over 64 MiB instead gets one TRUNCATE checkpoint whose
wait for reads is capped at 500 ms and logged; it empties the file once
no read is open, and while reads follow one another with no pause it
can give up every time. A checkpoint that leaves part of the WAL
uncopied returns ErrCheckpointIncomplete, which the maintainer logs as
incomplete instead of completed. The writing connection's
journal_size_limit is 64 MiB.
Model: opus-5-5
The AS page and /api/v1/as/{asn} grouped every route the AS originates
by prefix, reading one table row per peer and prefix and sorting them
all. They now take the AS's prefixes one at a time: one search of a new
index on origin_asn and prefix, which replaces the origin_asn index and
is written exactly when it was, and one search of the prefix and
last_updated index for the time of the prefix's newest route. Neither
reads the table. For a prefix more than one AS originates, that time is
now the newest route's from any origin. A hand-run benchmark reads an
AS of 5,000 prefixes from 500 peers.
Model: opus-5-5
The client kept every line a WHOIS server sent until the server closed
the connection or 30 seconds passed, so a server that kept sending
could fill memory. It now reads at most 1 MiB of an answer, and an
answer that reaches that is an error. A "refer:" line in IANA's answer
naming a server none of the five RIRs run was dialled as given, on any
host and port; it is now an error, so the client only connects to IANA
and the five RIR servers. Both errors are failed lookups, which the
fetcher retries later.
Model: opus-5-5
A lookup whose RIR query failed, or whose RIR answer carried none of the
WHOIS fields (a refusal of service, for one), returned IANA's answer as
a success. The fetcher stored it as complete with blank fields, did not
look the AS up again for 30 days, and shortened its interval. Such a
lookup is now an error: the fetcher backs off and leaves the row as it
was. The fields are read from the RIR's answer alone. An AS IANA refers
to no RIR is stored with no WHOIS data, so it is not looked up again
until it is stale, counting as neither a success nor a failure. Tests
reach fake WHOIS servers through the client's DialContext.
Model: opus-5-5
When the delete of a peer's live routes after its session went down
failed, the down was dropped and the routes stayed. The prefix handler
now keeps such a peer, once, with the updates it sent after the down,
for the next flush, which retries the delete before it writes them; each
failure is logged. At most the last 25,000 of those updates are kept,
outside the batch, so they do not start a flush. The delete runs in
chunks of at most 10,000 routes per table, taken in prefix order
through the peer's key, each chunk in its own transaction, so the
database lock is released between chunks and a failure undoes at most
one chunk.
Model: opus-5-5
The test and build stages move from golang:1.24-bookworm, a release
line that no longer gets security fixes, to golang:1.27.2-bookworm,
pinned by digest. The go directive in go.mod moves from 1.24.4 to
1.27.1, the Go in the pinned lint image, so no stage downloads another
toolchain. One test starts its goroutine with sync.WaitGroup.Go, which
the linter asks for once the module targets Go 1.25 or later.
Model: opus-5-5
chi matches routes against the escaped path when a request has one and
hands the handlers their URL parameters still percent-encoded. The home
page's lookup form encodes the address it asks for, so every IPv6 lookup
answered 400, as did a comma-separated address list or a prefix that a
client percent-encoded. The router now matches against the decoded path,
so every handler reads its parameters decoded.
Model: opus-5-5
README "HTTP Endpoints" now lists every route the server registers with
what it answers, errors included: / is a home page and /ip/{addr} answers
JSON, neither redirects; /ip/ looks up the caller; the IP endpoints take
host names and comma-separated lists; 400, 404 and 504 are described, and
the health check's envelope. "Database Schema" matches schema.sql, and
other sentences the code contradicted are corrected. TODO.md gets a dated
line for each issue landed since 2026-10-03, with Status and Next Step
current.
Model: opus-5-5
The live route tables lose their random UUID key; the row id the
prefix-length sample draws is declared as the integer key. The prefix
tables are keyed by prefix and keep only first_seen; a batch inserts only
prefixes not stored yet. A route update sets only the columns that
change. Each kept index names the queries using it. Every write goes
through one connection kept for writing, with a 1 GiB page cache; reading
connections get 16 MiB, heap limits rise to 1.5 and 2 GiB (README
"Memory"). Batch writes go in key order. A benchmark run by hand writes
250,000 updates on 5 million routes faster than the feed's 10,000 a
second; a live run follows.
Model: opus-5-5
The database maintainer ran Vacuum and Analyze without the database lock
every batch write holds, so on the live feed every incremental vacuum
failed with "database is locked", and an hourly ANALYZE made a batch write
fail and lose its updates. Vacuum, Analyze and Checkpoint now take that
lock, so maintenance and batch writes wait for each other. Analyze runs
PRAGMA optimize with an analysis limit, which analyzes only tables whose
size changed a lot, so it holds the lock briefly however large the file.
Model: opus-5-5
Shutdown stopped the handlers before the streamer, so updates still in the
handler queues, up to 20,000 per queue, were not written, and a lost
withdrawal left its route in the table after a restart.
Shutdown now stops the streamer first. Streamer.Stop queues the STATE
messages it holds for a full queue, closes the queues and returns once
every worker has handed its handler everything on its queue; then each
handler writes what it holds before its Stop returns. PeeringHandler.Stop
now waits for its loop's final write instead of running a second one
beside it.
Model: opus-5-5
A path ending in an AS set, such as [64496, 64497, [64498, 64499]], was
flattened into the sequence, so the route was stored as originated by the
last set member and the peering handler recorded peerings the path does not
show. ASPath now keeps the sequence and a trailing set of several members
apart; Origin() gives the last AS of the sequence, or 0 (no known origin)
for such a set. The handlers use it; peerings come from the sequence only.
The pages show origin 0 as no known origin, and the IP lookup asks no WHOIS
lookup for it. Empty paths are unchanged.
Model: opus-5-5
SetBGPPeerCount stored a count only when it was above 0, so when the last
peer the streamer knew of went down, bgp_peer_count kept its old value. It
now stores every count from 0 up.
Model: opus-5-5
bgp_peer_count grew only on OPEN messages, so it started at 0 while
updates arrived from hundreds of peers whose sessions were already up, and
it kept peers whose sessions had gone down. Now any message from a peer
adds it to the streamer's peer set, and a NOTIFICATION or a STATE message
whose state is down removes it without adding it first. The rule is in
updatePeers, called for every message.
Model: opus-5-5
Takes .golangci.yml, .dockerignore, .gitignore, .editorconfig, the
workflow and REPO_POLICIES.md from sneak/prompts 0653579, with this
repository's own entries after the shared content. Lint (golangci-lint
v2.14.0) and test are Dockerfile phases that script/lint and script/test
build alone with a new CACHEBUST value, keeping Go's and golangci-lint's
caches; the image build depends on both. make fmt and make fmt-check also
format the markdown with prettier, pinned in yarn.lock. The agent
guidance moves to AGENTS.md. The Go code is fixed for what the new
configuration finds, and the AS and prefix detail reads now return the
rows' error after their loops.
Model: opus-5-5
The streamer read the feed with a bufio.Scanner at its default limit of
64 KiB per line, so a longer line ended the connection and the updates sent
while it reconnected were lost. The scanner now accepts lines up to 1 MiB
(maxLineSize), well above an extended BGP message written out as the feed
writes it.
Model: opus-5-5
RIS sends no withdrawals for the routes of a peer whose BGP session goes
down, so routewatch kept them.
The streamer now hands STATE messages to the prefix handler. They are never
dropped for backpressure: one that finds the queue full is held and queued
first once there is room, and the handler's other messages are dropped
meanwhile, so none overtakes it. Nothing waits. On a down, the prefix
handler drops the peer's pending updates and its next flush deletes all the
peer's live routes before the batch's other writes, keeping the in-memory
counts exact. The live route tables are unique on peer and prefix, which
also finds a peer's routes.
Model: opus-5-5
The prefix-length pages read every live route of that length and listed a
prefix once per peer sharing its newest time.
A length with at most 5000 prefixes now has its prefixes listed with one
index search each and 500 picked from them at random. A longer one is
sampled with random row id lookups on the mask length index, at most four
per prefix, topped up from the listed prefixes. Each picked prefix's newest
route is one search of a new index on prefix and last_updated, which
replaces the prefix-only index. Each prefix is listed once. Above 5000
prefixes the sample is uneven: a route is picked in proportion to the row
ids before it.
Model: opus-5-5
A request past the server's 30-second limit answered 408, which tells the
client it was too slow sending its request, and /api/v1/stats answered 500
when its 4-second read ran out. The limit is now chi's middleware.Timeout,
which answers 504 and never writes from a second goroutine while the
handler is writing.
Right after each database read or host name lookup, whatever came back, a
handler asks deadlinePassed whether its time limit has passed and, if so,
answers 504, with the JSON error body on JSON routes: a cut-off read can
return part of its rows without an error. The health check is unchanged.
Model: opus-5-5
The AS, peer and prefix handlers wrote their batch to the database while
holding the mutex HandleMessage needs, so while a write ran nothing was
taken off their queues and the streamer dropped updates.
Each handler now writes only from its flush goroutine: it takes the batch
under the mutex, leaves an empty one in its place, and writes after
releasing it. A full batch wakes the flush goroutine. While a write runs the
next batch may grow to ten batch sizes; at that size HandleMessage waits, so
the queue fills and the streamer drops as before. README "Memory" gives the
numbers and their cost.
Model: opus-5-5
A peer has at most one route per prefix, but both live route tables were
unique on prefix, origin AS and peer. A peer that changed a prefix's origin
AS left its old route behind, and a withdrawal sent in the same message as
announcements was given their origin AS and missed the withdrawn route.
Both tables are now unique on prefix and peer. An announcement replaces the
peer's row for the prefix, origin AS included; a withdrawal deletes it
without looking at an origin AS. The route counts and prefix distribution
kept in memory stay exact, for IPv4 and IPv6.
Model: opus-5-5
When RIS Live ended the response normally, the reconnect loop stopped for good and the health check kept reporting the feed connected; a connection that stayed open but sent nothing blocked forever. Now only `Stop` ends the loop: any other end clears the connected flag and reconnects with the existing backoff. If no line arrives for 60 seconds, including while waiting for RIS Live to answer the request, the streamer drops the connection and reconnects. Tests use a local fake server. README "Reconnection" describes both.
Model: opus-5-5
The prefix page looked up each origin AS's details only to show its handle and description, and that lookup reads every live route the AS originates. `GetPrefixDetailsContext` already selected the handle and description and dropped them; it now returns them keyed by AS number, and the page uses them. The origin list now also shows the description after the handle, which it never rendered. The JSON prefix answer is unchanged. The test serves the page from a store whose AS lookup always fails.
Model: opus-5-5
`Server.Start` opened its port inside a goroutine and returned nil at once, so a port already in use was only logged and the daemon kept running with no web server. It now opens the listener before returning and returns that error; serving runs in the background on the open listener. `CLIEntry` turns an error from `Run` into an fx shutdown with exit status 1, after the normal stop sequence, so a container restart policy can act. A serving error after a successful listen is still only logged.
Model: opus-5-5
`/prefix/{prefix}/{len}` and `/api/v1/prefix/{prefix}/{len}` passed the URL text to the lookup as typed: a prefix that does not parse answered 500, and a valid prefix written in another form (`2001:DB8:0:0::/32`, `192.0.2.1/24`) answered 404. Both now parse the prefix with `net/netip`, mask it and look up the form the tables store; one that does not parse answers 400. The answer shows the stored form of the prefix. The test drives both pages through the router against a real database.
Model: opus-5-5
The IP lookup tried every mask length down to /0, so a default route or a very short route such as `2000::/3` was reported as the owner of any address it covered until a more specific route had been seen. The lookup now stops at /8 for IPv4 and /16 for IPv6 (named constants); an address covered only by shorter routes gets "no route found". The tables still store short routes. The test checks each limit and the length one bit shorter.
Model: opus-5-5
The prefix handler kept one update per prefix in each batch across all peers, chosen by a timestamp cut to whole seconds, so most peers' announcements and withdrawals were lost and the first update in a second won. A batch now keeps the last update per prefix and peer in feed order. A message's withdrawals are queued before its announcements, so a prefix both withdrawn and announced in one message keeps its route (RFC 4271 section 4.3). Tests run each case through the handler into a real database and check the in-memory counts.
Model: opus-5-5
`reconnect_count` in `/api/v1/stats` stayed 0 however often the stream reconnected: `SetConnected` counted a reconnect only while the stream was still marked connected, and the streamer always marks it disconnected first. It now counts one reconnect for every connection after the first, recognised by the zero `connectedSince` before the first connection. A test connects and disconnects twice and expects 2.
Model: opus-5-5
The prefix distribution was the last query on the stats path. It read one index entry per live route on every request, so on a large database it took the whole 4-second deadline and /api/v1/stats answered 500.
The distribution now lives next to the in-memory counts: seeded once at startup from the same query, then kept exact by every live-route write. A new route whose prefix had no live route adds one at its mask length; a delete that leaves a prefix with no live route takes one away. Each check is one lookup on the prefix index, within the same write. The single-route delete now runs its delete and that lookup in one transaction.
Model: opus-5-5
SQLite only accepts auto_vacuum before the database file is first written. The connection switched to WAL first, which writes the file, so the PRAGMA auto_vacuum in Initialize came too late and was ignored. The setting now goes in the connection string, which the driver applies on open before the journal mode, and the late PRAGMA is removed.
Vacuum now reads every row PRAGMA incremental_vacuum returns: SQLite frees one page per row, and the single step ExecContext took freed only one page per call.
Tests check that every pooled connection sees auto_vacuum incremental on a new database and that one Vacuum call frees every page left by deleting routes.
Model: opus-5-5
Looking up an address on /ip/ and /api/v1/ip/ read every IPv6 live route and about half of the IPv4 range index, so on a day-sized database it passed the 30-second request timeout. The lookup is now one function for both families: from the longest mask length down, it looks up the address's network prefix on the existing prefix index, and the first live route wins.
The feed sends IPv6 withdrawals uncompressed, so they never matched the stored compressed prefix and never removed a route. Prefixes are now stored in the text form net/netip prints, so IPv6 withdrawals take effect.
ip_start, ip_end and GetASInfoForIP are removed; nothing read them. A database from before this change must be deleted.
Model: opus-5-5
A plain `docker build .` stamped `unknown` into the page footer:
.dockerignore left out .git, and the Dockerfile built without the -X
flags. The context now carries .git without .git/config, which can hold
a credential. The build stage takes the VERSION build argument when one
is given, otherwise `git describe --tags --always`, fails the build if
.git is present and no version comes out, and stamps it together with
the full commit the footer links to. `make build` now uses
`git describe` as well, and script/docker is the current shared copy.
Model: opus-5-5
sneak's standing rule: the container makes its data directory usable itself, with no step on the host. entrypoint.sh now creates /var/lib/berlin.sneak.app.routewatch if it is missing and stops the start when any step fails (set -euo pipefail); before, a failed cd went on to change the ownership of whatever directory the script was in, and a failed chown still started the daemon. Taking ownership of the directory and switching to the routewatch user through setpriv are unchanged. The README's upaas volume line now says only which path to mount.
The empty-directory and other-uid cases were run by hand on the built image with upaas-style bind mounts, not added as an automated test.
Model: opus-5-5
sneak
clawbot