IP lookup answer: every result has the same fields. New: num_origins, origin_asns, flags (is_tor, is_vpn, is_relay, is_bogon), vpn_providers and geo. blocklists is now ip_lists. ptr comes from public DNS-over-HTTPS resolvers, and geo from geojs with DB-IP's City Lite database as the fallback; both are cached in cache.db.
Other API changes: no value carries its unit inside a string, and every JSON answer has @meta.
Pages: new home page cards and list pages. The prefix page lists each time an AS started or stopped announcing the prefix.
Client address: taken from X-Forwarded-For when the direct peer is a trusted proxy.
Live feed: a handler drops an update only when its queue is full, and a burst of sessions going down no longer stops the prefix handler. The WHOIS fetcher pauses a registry that refuses it.
Tests: time-dependent tests allow for a stalled build machine.
README.md and TODO.md are rewritten.
For a deploy:
Nothing to set. With TRUSTED_PROXIES unset, every RFC 1918 address is trusted.
A service that reads blocklists must read ip_lists, and one that reads the old unit-in-a-string fields must read their replacements.
The lists download again into iplists in the state directory; the old blocklists directory can be deleted.
Looked-up addresses now go to Google, Cloudflare, AdGuard or Quad9 for PTR names and to geojs for locations.
The first start downloads the 121 MB City Lite database. cache.db and the origin_changes table are created at startup.
The ProtonVPN list comes from one person's public mirror.
Merged into next:
Make two tests independent of host load (closes#256): #258
Drop the work log from TODO.md; keep only open work: #257
Model: opus-5-5
On `next` since the last merge to `main`:
- **IP lookup answer:** every result has the same fields. New: `num_origins`, `origin_asns`, `flags` (`is_tor`, `is_vpn`, `is_relay`, `is_bogon`), `vpn_providers` and `geo`. `blocklists` is now `ip_lists`. `ptr` comes from public DNS-over-HTTPS resolvers, and `geo` from geojs with DB-IP's City Lite database as the fallback; both are cached in `cache.db`.
- **Other API changes:** no value carries its unit inside a string, and every JSON answer has `@meta`.
- **Pages:** new home page cards and list pages. The prefix page lists each time an AS started or stopped announcing the prefix.
- **Client address:** taken from `X-Forwarded-For` when the direct peer is a trusted proxy.
- **Live feed:** a handler drops an update only when its queue is full, and a burst of sessions going down no longer stops the prefix handler. The WHOIS fetcher pauses a registry that refuses it.
- **Tests:** time-dependent tests allow for a stalled build machine.
- `README.md` and `TODO.md` are rewritten.
For a deploy:
- Nothing to set. With `TRUSTED_PROXIES` unset, every RFC 1918 address is trusted.
- A service that reads `blocklists` must read `ip_lists`, and one that reads the old unit-in-a-string fields must read their replacements.
- The lists download again into `iplists` in the state directory; the old `blocklists` directory can be deleted.
- Looked-up addresses now go to Google, Cloudflare, AdGuard or Quad9 for PTR names and to geojs for locations.
- The first start downloads the 121 MB City Lite database. `cache.db` and the `origin_changes` table are created at startup.
- The ProtonVPN list comes from one person's public mirror.
Merged into `next`:
- Make two tests independent of host load (closes #256): https://git.eeqj.de/sneak/routewatch/pulls/258
- Drop the work log from TODO.md; keep only open work: https://git.eeqj.de/sneak/routewatch/pulls/257
Model: opus-5-5
The home page gets cards for the newest and oldest kept route, the newest
and oldest updated AS, the AS with the most prefixes, the ASes with a
single prefix and fifteen well-known networks. Two more cards count the
bogon announcements and the prefixes with several origin ASes. They link
to the new pages /bogons and /multiple-origins, which list up to 1,000
entries each.
The new package internal/bogon tells reserved prefixes and AS numbers,
from the IANA special-purpose and address space registries. Table.Overview
walks the table and lets the read lock go every 10,000 prefixes. The
server works the cards out once a minute and serves that copy. A new index
on asns(last_seen) finds the newest and oldest updated AS.
Model: opus-5-5
sneak
was assigned by clawbot2026-10-27 18:39:24 +01:00
With TRUSTED_PROXIES unset, the server now trusts reverse proxies in the
private ranges of RFC 1918. A proxy such as traefik on a Docker network is
then believed. /ip/, the home page's curl card and the request log give the
client's address from X-Forwarded-For instead of the proxy's address.
A list in TRUSTED_PROXIES replaces the default. TRUSTED_PROXIES set but
empty trusts no proxy, so the config package reads it with os.LookupEnv.
A direct client at a public address still cannot choose its address with
a forged header.
Model: opus-5-5
Every result of /ip/, /ip/{addr} and /api/v1/ip/{ip} now has every
field. A field without a value is null instead of left out, so an
address with no live route has all its route and AS fields set to null.
ptr is an empty list when there are no PTR names.
Two new fields, num_origins and origin_asns, give how many origin ASes
peers carry the netblock with, and which. The routing table's prefix
summary now lists them. asn still names the origin that most peers
carry.
first_seen is null, no longer the zero time, when the database has not
stored the netblock yet.
Model: opus-5-5
A value with a unit is now a number, and its field name ends in the
unit, as the vendored API_POLICIES.md requires. @meta.execution_time
becomes execution_time_ms. In the stats answers, uptime,
connection_duration, memory_usage and whois_stats.current_interval
become uptime_seconds, connection_duration_seconds, memory_usage_bytes
and whois_stats.current_interval_seconds. connection_duration_seconds
is null while the feed is not connected. The status page formats the
numbers itself. REPO_POLICIES.md is re-vendored from the same
sneak/prompts commit.
Model: opus-5-5
A panic in a JSON route's handler answered 500 with an empty body,
because the router's recoverer sits outside the middleware that adds
`@meta`. `JSONValidationMiddleware` now wraps the handler in the same
recoverer, and answers the panic with the JSON error body.
`/status.json` and the health check answered chi's empty 405 to any
method they do not take. They now answer with the same JSON 405 as the
routes under `/api/v1` and `/ip`.
The home page's IP lookup card showed only the answer's `data`. It now
shows the whole answer, `@meta` included.
A table test checks `@meta` on every JSON route, in its successes and
its errors.
Model: opus-5-5
The Rationale section now gives the owner's reason for the program: it
is an internal API that our other services ask about IP addresses. It
sees which AS routes each address and which announced prefix holds it
from its own routing table. WHOIS details of each AS, kept in its
database, add to that.
What RIS Live does and does not give stays, shortened. Planned geojs
lookups are left out, because the README describes only what is built.
Model: opus-5-5
Server.Start ran the loop that works out the home page's cards on the
context it was given. fx ends that context once start-up is done, so the
cards were worked out once and never again. The loop now runs on a
context that only Stop ends, as the other background parts do. Stop
waits for the loop to return, until its own context ends, before it
stops the HTTP server.
A new test starts a server in a synctest bubble and ends its context at
once. On the bubble's fake clock, the cards must be worked out again a
minute later, and not after Stop.
Model: opus-5-5
Five prefix handler tests gave HandleMessage two seconds to apply a full
batch of 25,000 prefixes, which under the race detector on a loaded host
takes longer. A test now waits up to stepTimeout, 15 seconds, for each step
it expects.
The slowest tests filled production limits, such as the peering handler's
500,000 paths. The peering handler now takes its path limit, and the prefix
handler its limit on one write of the table's changes, as they take their
timer interval. Those tests give small limits, and the tests of the maximum
batch size lower it to one batch. This also fixes
#215.
The test helpers bound every wait, Stop included, so a test that hangs in
one fails with its own message.
Model: opus-5-5
HandleMessage added one entry to the batch for every AS of every path,
and waited for the write in progress once the batch held 300,000. In a
burst of the feed its queue then filled, and the streamer dropped
messages.
The batch is now a map from AS number to the earliest and the latest
time of its updates. It holds one entry per AS, so it no longer grows
with the feed's rate. HandleMessage never waits for a write.
asnBatchSize now counts ASes. Stop logs the entries of the batch being
written and of the next batch under "batch_entries". An AS in both
counts twice.
Model: opus-5-5
The fetcher kept one interval for every registry, so a registry that
refused a query was asked again within seconds. RIPE blocked the test
host's address that way.
The WHOIS client now pauses an RIR's or NIC.br's server for an hour
after the server refuses a query. A refused or reset connection is a
refusal, and so is an %ERROR:201: or %ERROR:202: line. Lookups at the
other servers go on. A lookup whose RIR is paused leaves the AS stale,
and counts as neither a success nor a failure. IANA is never paused,
because a pause of IANA would stop every lookup. NIC.br's hour-long
skip after its over-limit answer is now the same pause.
Model: opus-5-5
The table's PeerDown visits every prefix under the write lock, which
takes over a second on a full table. The prefix handler called it
inline, so its queue filled and the streamer dropped updates.
PeerDown now holds the lock for 10,000 prefixes at a time, then lets
the lock go for as long again. The handler runs PeerDown on a separate
goroutine, one peer at a time. The handler holds back the down peer's
later messages until that PeerDown returns. Then the handler applies
them in order, one at a time, and takes new messages in between. It
holds back at most 20,000 messages. Stop waits for all of this before
its last flush.
Model: opus-5-5
A lookup whose RIR is paused is skipped. The skip left the fetcher's wait
as it was. Each lookup asks IANA before it finds its RIR paused. So while
most stale ASes belonged to a paused RIR, the fetcher could ask IANA every
second.
Like a failure, a skipped lookup now makes the wait 1.5 times as long, up
to one minute. The skip is still not counted as a failure, so the status
page's failure figures leave it out.
Model: opus-5-5
The home page's cards are worked out once a minute by Overview, which
walks every prefix of the routing table under the table's read lock.
After each batch of 10,000 prefixes, the walk let the lock go and took
it back at once. The prefix handler, waiting for the write lock, then got
in only once per batch. During a walk, its queue could fill, and the
streamer then dropped updates for it.
The walk now lets the lock go for as long as it held the lock for the
batch, as PeerDown does. A test applies another peer's 1,000
announcements while Overview walks a table with more prefixes than five
of the walk's batches.
Model: opus-5-5
The prefix handler ran one PeerDown per peer, one after another. When
the sessions of several peers went down together, their walks queued up.
A returning full-table peer's messages stayed held back while the walks
ahead of its own ran, and could reach the limit of 20,000.
The table's PeersDown now takes several peers out of every prefix in one
walk. One walk runs at a time. The next one takes every peer whose
session went down since the previous one began. The doc of
maxHeldBackMessages says how long a peer's messages are held back. The
limit is now 150,000 messages, about 150 MiB. The status page shows how
many messages each handler holds back.
Model: opus-5-5
Every sentence of TODO.md is rewritten from scratch against the rules
of #157: about 30 words at
most, one idea each, subject and verb early, identifiers in backticks.
Status and Next Step describe the repo as the tracker shows it now.
Every Completed Steps entry keeps its date, its facts and its issue
reference, checked against the commit it names. The Workflow section
keeps its meaning. The entry for this rewrite is added.
Model: fable-5-1
The README is rewritten from scratch against the prose rules of
#157: every sentence under
about 30 words, lists as bullets, every technical term defined where it
first appears, and every statement checked against the code on next. It
leads with what RouteWatch is and how to run it, then the pages, the API
and the blocklists, then the settings, memory and upaas, and last the
design and the development workflow. The dated entry goes into TODO.md.
Model: fable-5-1
The streamer dropped a random share of a handler's updates once its
queue was half full. When the feed paused and then sent its backlog
while the routing table's Overview walked every prefix, the prefix
handler lost about 10,000 updates with its queue never full.
The streamer now drops an update only when the queue is full. The
prefix handler's queue holds 100,000 messages, about 100 MiB, so it
can hold such a backlog. The README's Memory and Feed handling
sections give the new sizes and the rule.
Model: opus-5-5
The routing table records a row each time a prefix and origin AS gets its first peer (announced) or loses its last (withdrawn), with the time and how many peers carry the prefix just after. The prefix handler writes the rows to a new origin_changes table, apart from routes. Every hour the database maintainer keeps each prefix's rows from the last 72 hours or its newest 500, whichever keeps more. The prefix page lists them newest first, at most 5,000.
At most 100,000 rows wait in memory to be written; past that the oldest are dropped and logged. A stop loses the rows past its last write.
Model: deepseek-v4-pro (implementation); opus-5-5 (commit message)
The lists are not only used for blocking, so the package, the `blocklists` field of the IP lookup answer and the state directory are now IP lists (`internal/iplist`, `ip_lists`, `iplists`). Every result gains a `flags` object with `is_tor` and `is_bogon`. The largest list file accepted is 200 MiB.
Judgement call: the dated 2026-10-27 `TODO.md` entry keeps the old word.
Rule suppressed: `tagliatelle` on the two result structs, for snake_case keys.
Model: opus-5-5
`check.torproject.org` lists IPv4 exits only, so `is_tor` was false for an IPv6 exit. `tor_exits` now comes from Onionoo through a new format that reads `relays[].exit_addresses` and `relays[].or_addresses`. IPv6 addresses come only from `or_addresses`, the address a relay accepts Tor connections on, so an exit leaving from another IPv6 address still has `is_tor` false.
Judgement call: a document with no addresses fails the parse.
Rule suppressed: `tagliatelle` on the struct holding Onionoo's field names.
Model: opus-5-5
The IP lookup now checks the public VPN server lists of Mullvad, NordVPN, Private Internet Access and IVPN, and X4BNet's community list, each downloaded once a day. `List` gains `VPN`. Every result gains `flags.is_vpn` and `vpn_providers`, which is empty when no list holds the address.
Judgement call: X4BNet's IPv4 and IPv6 files are two lists, `x4bnet` and `x4bnet_v6`, as Spamhaus's are.
Judgement call: Private Internet Access's signature is neither read nor checked.
Judgement call: a document with no addresses fails the parse.
Rule suppressed: `tagliatelle` on the structs holding the publishers' field names.
Model: opus-5-5
Tests that wait on the real clock failed when the host stalled for a few seconds, although the code was right. Waits for something that should happen are now mostly 15 seconds. Tests whose result depends on a short idle timeout or a production request limit run under `synctest.Test`, on a fake clock, with a fake transport where a real connection would stop that clock. Test code only.
Judgement call: the 5-second waits in three `internal/streamer` tests also became 15 seconds.
Model: opus-5-5
`ptr` keeps its field and shape. It now comes from Google, Cloudflare, AdGuard and Quad9 over RFC 8484, tried in turn within 2 seconds, in place of the host's resolver with no cache. Answers stay in memory and in a new `cache.db` for the record's TTL, at most 250,000 addresses. A reserved address is not sent to any resolver and has an empty `ptr`.
Judgement call: each resolver gets an equal share of the time left, so a hung one does not use up the limit.
Judgement call: the negative TTL is the smaller of the SOA minimum and the SOA's own TTL.
Deviation: the cache is SQLite, by sneak's words, not the JSON file the standing rule asks for.
Model: opus-5-5
Proton's own server list answers only a logged-in session, so the lists come from the public mirror https://github.com/tn3w/ProtonVPN-IPs, downloaded once a day: `protonvpn`, its /24 networks, and `protonvpn_entry`, its servers' IPv4 and IPv6 addresses. Both feed `flags.is_vpn` and `vpn_providers`.
Judgement call: the /24 networks are listed as well as the exact addresses, because coverage matters more than attribution, so neighbouring addresses in those networks count too.
Risk: the mirror is one person's repository; if it stops updating, downloads keep succeeding with old data.
Model: opus-5-5
Adds Apple's iCloud Private Relay egress list as `icloud_private_relay`, downloaded once a day, and `flags.is_relay`, true when an address is on it. Loaded, the list adds about 0.3 MiB to the Go heap; parsing it allocates about 94 MiB, freed by the next garbage collection.
Judgement call: a line with no comma fails the parse, so a plain network list served at Apple's URL cannot replace a good copy.
Model: opus-5-5
Every IP lookup result gains `geo`. One request to geojs carries all uncached addresses of a lookup, within 1 second. On a timeout, an error or an answer with no location, the lookup reads DB-IP City Lite, downloaded monthly into the state directory. Geojs answers stay in memory, at most 250,000, and in `cache.db` for 7 days. The home page footer credits DB-IP.
Judgement call: a reserved address goes to neither source, and its `geo` is null.
Judgement call: `updated` for a DB-IP answer is when the copy was downloaded.
Deviation: the cache is SQLite, by sneak's words, not the JSON file the standing rule asks for.
Model: opus-5-5
`useradd -r` asks for a system user and UID 1000 is above the system range, so the image build warned. Dropping `-r` ends the warning; the user keeps UID 1000, so the owner of the data directory does not change.
Model: opus-5-5
A fresh read of `README.md` and `TODO.md` against the prose rules, with every statement except measured figures and what outside services do checked against the code. Four README sentences failed the prose rules and five disagreed with the code; each is rewritten. The Status and Next Step of `TODO.md` now describe the run of `e58c663`.
Model: fable-5-1 (rewrite); opus-5-5 (commit message)
`go-sqlite3` v1.14.29 printed two `-Wdiscarded-qualifiers` warnings from its copy of SQLite when compiled with a recent `gcc`. v1.14.53 prints neither.
Model: opus-5-5
The home-card refresh test runs keepHomeCardsFresh in a synctest bubble,
so its waits are exact on a fake clock instead of 15-second polls.
The geo fallback subtest in which geojs never answers used a 50 ms time
limit and counted requests at the fake server, which a slow host could
miss. It now uses a transport that counts each request as it is sent and
holds it until the time limit ends it.
Model: opus-5-5
TODO.md held a dated log of finished steps, merge history and accounts
of earlier live-feed runs. It now holds the Status line, the open Next
Step and an empty Future Steps. The Workflow section deletes the
finished Next Step instead of moving it to Completed Steps. README
"TODO" no longer says the file lists the steps done.
Model: opus-5-5
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
On
nextsince the last merge tomain:num_origins,origin_asns,flags(is_tor,is_vpn,is_relay,is_bogon),vpn_providersandgeo.blocklistsis nowip_lists.ptrcomes from public DNS-over-HTTPS resolvers, andgeofrom geojs with DB-IP's City Lite database as the fallback; both are cached incache.db.@meta.X-Forwarded-Forwhen the direct peer is a trusted proxy.README.mdandTODO.mdare rewritten.For a deploy:
TRUSTED_PROXIESunset, every RFC 1918 address is trusted.blocklistsmust readip_lists, and one that reads the old unit-in-a-string fields must read their replacements.iplistsin the state directory; the oldblocklistsdirectory can be deleted.cache.dband theorigin_changestable are created at startup.Merged into
next:Model: opus-5-5
Every result of /ip/, /ip/{addr} and /api/v1/ip/{ip} now has every field. A field without a value is null instead of left out, so an address with no live route has all its route and AS fields set to null. ptr is an empty list when there are no PTR names. Two new fields, num_origins and origin_asns, give how many origin ASes peers carry the netblock with, and which. The routing table's prefix summary now lists them. asn still names the origin that most peers carry. first_seen is null, no longer the zero time, when the database has not stored the netblock yet. Model: opus-5-5View command line instructions
Checkout
From your project repository, check out a new branch and test the changes.