next into main #202

Open
clawbot wants to merge 31 commits from next into main
pull from: next
31 Commits
Author SHA1 Message Date
clawbot 5111697885 Drop the work log from TODO.md; keep only open work (closes #255)
check / check (push) Successful in 1m12s
TODO.md held a dated log of finished steps, merge history and accounts
of earlier live-feed runs. It now holds the Status line, the open Next
Step and an empty Future Steps. The Workflow section deletes the
finished Next Step instead of moving it to Completed Steps. README
"TODO" no longer says the file lists the steps done.

Model: opus-5-5
2026-11-11 20:25:40 +01:00
clawbot caed69249c Make two tests independent of host load (closes #256)
check / check (push) Canceled after 0s
The home-card refresh test runs keepHomeCardsFresh in a synctest bubble,
so its waits are exact on a fake clock instead of 15-second polls.

The geo fallback subtest in which geojs never answers used a 50 ms time
limit and counted requests at the fake server, which a slow host could
miss. It now uses a transport that counts each request as it is sent and
holds it until the time limit ends it.

Model: opus-5-5
2026-11-11 19:40:39 +01:00
clawbot 760ba4746b Update go-sqlite3 to v1.14.53 to drop two C compiler warnings (closes #245)
check / check (push) Successful in 1m2s
`go-sqlite3` v1.14.29 printed two `-Wdiscarded-qualifiers` warnings from its copy of SQLite when compiled with a recent `gcc`. v1.14.53 prints neither.

Model: opus-5-5
2026-11-10 18:13:50 +01:00
clawbot dcb3935643 Rewrite the README and TODO.md sentences that fail the prose rules (closes #251)
check / check (push) Successful in 1m7s
A fresh read of `README.md` and `TODO.md` against the prose rules, with every statement except measured figures and what outside services do checked against the code. Four README sentences failed the prose rules and five disagreed with the code; each is rewritten. The Status and Next Step of `TODO.md` now describe the run of `e58c663`.

Model: fable-5-1 (rewrite); opus-5-5 (commit message)
2026-11-10 18:01:51 +01:00
clawbot 6de44ac054 Create the image's app user without the system-user flag (closes #248)
check / check (push) Successful in 1m8s
`useradd -r` asks for a system user and UID 1000 is above the system range, so the image build warned. Dropping `-r` ends the warning; the user keeps UID 1000, so the owner of the data directory does not change.

Model: opus-5-5
2026-11-10 16:23:20 +01:00
clawbot cca402c73d Add best-effort geolocation to the IP lookup from geojs and DB-IP City Lite (closes #239)
check / check (push) Successful in 1m41s
Every IP lookup result gains `geo`. One request to geojs carries all uncached addresses of a lookup, within 1 second. On a timeout, an error or an answer with no location, the lookup reads DB-IP City Lite, downloaded monthly into the state directory. Geojs answers stay in memory, at most 250,000, and in `cache.db` for 7 days. The home page footer credits DB-IP.

Judgement call: a reserved address goes to neither source, and its `geo` is null.
Judgement call: `updated` for a DB-IP answer is when the copy was downloaded.
Deviation: the cache is SQLite, by sneak's words, not the JSON file the standing rule asks for.

Model: opus-5-5
2026-11-10 15:48:12 +01:00
clawbot 2c3f6f71dc Add the iCloud Private Relay list to the IP lookup, with is_relay (closes #237)
check / check (push) Failing after 17m2s
Adds Apple's iCloud Private Relay egress list as `icloud_private_relay`, downloaded once a day, and `flags.is_relay`, true when an address is on it. Loaded, the list adds about 0.3 MiB to the Go heap; parsing it allocates about 94 MiB, freed by the next garbage collection.

Judgement call: a line with no comma fails the parse, so a plain network list served at Apple's URL cannot replace a good copy.

Model: opus-5-5
2026-11-10 15:13:55 +01:00
clawbot b2553f044d Add the ProtonVPN server networks and addresses to the IP lookup (closes #236)
check / check (push) Successful in 1m27s
Proton's own server list answers only a logged-in session, so the lists come from the public mirror https://github.com/tn3w/ProtonVPN-IPs, downloaded once a day: `protonvpn`, its /24 networks, and `protonvpn_entry`, its servers' IPv4 and IPv6 addresses. Both feed `flags.is_vpn` and `vpn_providers`.

Judgement call: the /24 networks are listed as well as the exact addresses, because coverage matters more than attribution, so neighbouring addresses in those networks count too.
Risk: the mirror is one person's repository; if it stops updating, downloads keep succeeding with old data.

Model: opus-5-5
2026-11-10 14:14:06 +01:00
clawbot 0cbd9cd943 Resolve PTR names over DNS-over-HTTPS and cache them for the record TTL (closes #238)
check / check (push) Successful in 4m39s
`ptr` keeps its field and shape. It now comes from Google, Cloudflare, AdGuard and Quad9 over RFC 8484, tried in turn within 2 seconds, in place of the host's resolver with no cache. Answers stay in memory and in a new `cache.db` for the record's TTL, at most 250,000 addresses. A reserved address is not sent to any resolver and has an empty `ptr`.

Judgement call: each resolver gets an equal share of the time left, so a hung one does not use up the limit.
Judgement call: the negative TTL is the smaller of the SOA minimum and the SOA's own TTL.
Deviation: the cache is SQLite, by sneak's words, not the JSON file the standing rule asks for.

Model: opus-5-5
2026-11-10 13:48:06 +01:00
clawbot 641798bb22 Make the tests pass on a busy host (closes #240)
check / check (push) Successful in 1m51s
Tests that wait on the real clock failed when the host stalled for a few seconds, although the code was right. Waits for something that should happen are now mostly 15 seconds. Tests whose result depends on a short idle timeout or a production request limit run under `synctest.Test`, on a fake clock, with a fake transport where a real connection would stop that clock. Test code only.

Judgement call: the 5-second waits in three `internal/streamer` tests also became 15 seconds.

Model: opus-5-5
2026-11-10 12:48:07 +01:00
clawbot 28ef0275ba Add the public VPN server lists, with is_vpn and vpn_providers (closes #235)
check / check (push) Failing after 4m35s
The IP lookup now checks the public VPN server lists of Mullvad, NordVPN, Private Internet Access and IVPN, and X4BNet's community list, each downloaded once a day. `List` gains `VPN`. Every result gains `flags.is_vpn` and `vpn_providers`, which is empty when no list holds the address.

Judgement call: X4BNet's IPv4 and IPv6 files are two lists, `x4bnet` and `x4bnet_v6`, as Spamhaus's are.
Judgement call: Private Internet Access's signature is neither read nor checked.
Judgement call: a document with no addresses fails the parse.
Rule suppressed: `tagliatelle` on the structs holding the publishers' field names.

Model: opus-5-5
2026-11-10 12:15:32 +01:00
clawbot 3204b7ee4b Take the Tor exit list from Onionoo, which has the IPv6 exits (closes #234)
check / check (push) Failing after 1m50s
`check.torproject.org` lists IPv4 exits only, so `is_tor` was false for an IPv6 exit. `tor_exits` now comes from Onionoo through a new format that reads `relays[].exit_addresses` and `relays[].or_addresses`. IPv6 addresses come only from `or_addresses`, the address a relay accepts Tor connections on, so an exit leaving from another IPv6 address still has `is_tor` false.

Judgement call: a document with no addresses fails the parse.
Rule suppressed: `tagliatelle` on the struct holding Onionoo's field names.

Model: opus-5-5
2026-11-10 11:47:56 +01:00
clawbot cf50d195ad Rename blocklists to IP lists, add IP lookup flags, raise the download limit to 200 MiB (closes #233)
check / check (push) Failing after 10m45s
The lists are not only used for blocking, so the package, the `blocklists` field of the IP lookup answer and the state directory are now IP lists (`internal/iplist`, `ip_lists`, `iplists`). Every result gains a `flags` object with `is_tor` and `is_bogon`. The largest list file accepted is 200 MiB.

Judgement call: the dated 2026-10-27 `TODO.md` entry keeps the old word.
Rule suppressed: `tagliatelle` on the two result structs, for snake_case keys.

Model: opus-5-5
2026-11-10 10:48:23 +01:00
clawbot e58c6635e7 Record origin changes and show them on the prefix page (closes #161)
check / check (push) Successful in 1m3s
The routing table records a row each time a prefix and origin AS gets its first peer (announced) or loses its last (withdrawn), with the time and how many peers carry the prefix just after. The prefix handler writes the rows to a new origin_changes table, apart from routes. Every hour the database maintainer keeps each prefix's rows from the last 72 hours or its newest 500, whichever keeps more. The prefix page lists them newest first, at most 5,000.

At most 100,000 rows wait in memory to be written; past that the oldest are dropped and logged. A stop loses the rows past its last write.

Model: deepseek-v4-pro (implementation); opus-5-5 (commit message)
2026-11-05 06:57:41 +01:00
clawbot 15bf20a894 Drop an update only when its handler's queue is full (closes #229)
check / check (push) Canceled after 0s
The streamer dropped a random share of a handler's updates once its
queue was half full. When the feed paused and then sent its backlog
while the routing table's Overview walked every prefix, the prefix
handler lost about 10,000 updates with its queue never full.

The streamer now drops an update only when the queue is full. The
prefix handler's queue holds 100,000 messages, about 100 MiB, so it
can hold such a backlog. The README's Memory and Feed handling
sections give the new sizes and the rule.

Model: opus-5-5
2026-11-03 18:27:09 +01:00
clawbot 742e75914c Rewrite the README sentence by sentence (closes #157)
check / check (push) Canceled after 0s
The README is rewritten from scratch against the prose rules of
#157: every sentence under
about 30 words, lists as bullets, every technical term defined where it
first appears, and every statement checked against the code on next. It
leads with what RouteWatch is and how to run it, then the pages, the API
and the blocklists, then the settings, memory and upaas, and last the
design and the development workflow. The dated entry goes into TODO.md.

Model: fable-5-1
2026-11-03 00:10:08 +01:00
clawbot 37cecedc5b Rewrite TODO.md sentence by sentence (part of #157)
check / check (push) Canceled after 0s
Every sentence of TODO.md is rewritten from scratch against the rules
of #157: about 30 words at
most, one idea each, subject and verb early, identifiers in backticks.
Status and Next Step describe the repo as the tracker shows it now.
Every Completed Steps entry keeps its date, its facts and its issue
reference, checked against the commit it names. The Workflow section
keeps its meaning. The entry for this rewrite is added.

Model: fable-5-1
2026-11-02 23:44:09 +01:00
clawbot e5afd94a39 Take every down peer out of the table in one walk (closes #225)
check / check (push) Canceled after 0s
The prefix handler ran one PeerDown per peer, one after another. When
the sessions of several peers went down together, their walks queued up.
A returning full-table peer's messages stayed held back while the walks
ahead of its own ran, and could reach the limit of 20,000.

The table's PeersDown now takes several peers out of every prefix in one
walk. One walk runs at a time. The next one takes every peer whose
session went down since the previous one began. The doc of
maxHeldBackMessages says how long a peer's messages are held back. The
limit is now 150,000 messages, about 150 MiB. The status page shows how
many messages each handler holds back.

Model: opus-5-5
2026-11-02 23:10:09 +01:00
clawbot 9bd0e91a3b Let the lock go for as long as it was held during Overview's walk (closes #226)
check / check (push) Canceled after 0s
The home page's cards are worked out once a minute by Overview, which
walks every prefix of the routing table under the table's read lock.
After each batch of 10,000 prefixes, the walk let the lock go and took
it back at once. The prefix handler, waiting for the write lock, then got
in only once per batch. During a walk, its queue could fill, and the
streamer then dropped updates for it.

The walk now lets the lock go for as long as it held the lock for the
batch, as PeerDown does. A test applies another peer's 1,000
announcements while Overview walks a table with more prefixes than five
of the walk's batches.

Model: opus-5-5
2026-11-02 08:00:00 +01:00
clawbot d66c59ef07 Lengthen the WHOIS fetcher's wait after a skipped lookup (closes #222)
check / check (push) Canceled after 0s
A lookup whose RIR is paused is skipped. The skip left the fetcher's wait
as it was. Each lookup asks IANA before it finds its RIR paused. So while
most stale ASes belonged to a paused RIR, the fetcher could ask IANA every
second.

Like a failure, a skipped lookup now makes the wait 1.5 times as long, up
to one minute. The skip is still not counted as a failure, so the status
page's failure figures leave it out.

Model: opus-5-5
2026-11-02 03:59:47 +01:00
clawbot 26d939ad72 Keep taking messages while a peer's session goes down (closes #216)
check / check (push) Canceled after 0s
The table's PeerDown visits every prefix under the write lock, which
takes over a second on a full table. The prefix handler called it
inline, so its queue filled and the streamer dropped updates.

PeerDown now holds the lock for 10,000 prefixes at a time, then lets
the lock go for as long again. The handler runs PeerDown on a separate
goroutine, one peer at a time. The handler holds back the down peer's
later messages until that PeerDown returns. Then the handler applies
them in order, one at a time, and takes new messages in between. It
holds back at most 20,000 messages. Stop waits for all of this before
its last flush.

Model: opus-5-5
2026-11-02 03:27:10 +01:00
clawbot df15e2b2cd Pause a registry's WHOIS server after it refuses a query (closes #218)
check / check (push) Canceled after 0s
The fetcher kept one interval for every registry, so a registry that
refused a query was asked again within seconds. RIPE blocked the test
host's address that way.

The WHOIS client now pauses an RIR's or NIC.br's server for an hour
after the server refuses a query. A refused or reset connection is a
refusal, and so is an %ERROR:201: or %ERROR:202: line. Lookups at the
other servers go on. A lookup whose RIR is paused leaves the AS stale,
and counts as neither a success nor a failure. IANA is never paused,
because a pause of IANA would stop every lookup. NIC.br's hour-long
skip after its over-limit answer is now the same pause.

Model: opus-5-5
2026-11-02 02:59:57 +01:00
clawbot 6b004dfbc3 Keep one entry per AS in the AS handler's batch (closes #217)
check / check (push) Canceled after 0s
HandleMessage added one entry to the batch for every AS of every path,
and waited for the write in progress once the batch held 300,000. In a
burst of the feed its queue then filled, and the streamer dropped
messages.

The batch is now a map from AS number to the earliest and the latest
time of its updates. It holds one entry per AS, so it no longer grows
with the feed's rate. HandleMessage never waits for a write.
asnBatchSize now counts ASes. Stop logs the entries of the batch being
written and of the next batch under "batch_entries". An AS in both
counts twice.

Model: opus-5-5
2026-11-02 02:10:17 +01:00
clawbot 38b43549b9 Make the routewatch tests pass on a loaded host (closes #213)
check / check (push) Canceled after 0s
Five prefix handler tests gave HandleMessage two seconds to apply a full
batch of 25,000 prefixes, which under the race detector on a loaded host
takes longer. A test now waits up to stepTimeout, 15 seconds, for each step
it expects.

The slowest tests filled production limits, such as the peering handler's
500,000 paths. The peering handler now takes its path limit, and the prefix
handler its limit on one write of the table's changes, as they take their
timer interval. Those tests give small limits, and the tests of the maximum
batch size lower it to one batch. This also fixes
#215.

The test helpers bound every wait, Stop included, so a test that hangs in
one fails with its own message.

Model: opus-5-5
2026-10-28 01:44:47 +01:00
clawbot 5ea1bb17fe Keep working out the home page cards after startup (closes #211)
check / check (push) Canceled after 0s
Server.Start ran the loop that works out the home page's cards on the
context it was given. fx ends that context once start-up is done, so the
cards were worked out once and never again. The loop now runs on a
context that only Stop ends, as the other background parts do. Stop
waits for the loop to return, until its own context ends, before it
stops the HTTP server.

A new test starts a server in a synctest bubble and ends its context at
once. On the bubble's fake clock, the cards must be worked out again a
minute later, and not after Stop.

Model: opus-5-5
2026-10-27 22:55:17 +01:00
clawbot 6d4f3e3419 Say in README why RouteWatch exists (closes #204)
check / check (push) Canceled after 0s
The Rationale section now gives the owner's reason for the program: it
is an internal API that our other services ask about IP addresses. It
sees which AS routes each address and which announced prefix holds it
from its own routing table. WHOIS details of each AS, kept in its
database, add to that.

What RIS Live does and does not give stays, shortened. Planned geojs
lookups are left out, because the README describes only what is built.

Model: opus-5-5
2026-10-27 21:34:08 +01:00
clawbot 47aed8761e Give every JSON answer @meta, errors included (closes #201)
check / check (push) Canceled after 0s
A panic in a JSON route's handler answered 500 with an empty body,
because the router's recoverer sits outside the middleware that adds
`@meta`. `JSONValidationMiddleware` now wraps the handler in the same
recoverer, and answers the panic with the JSON error body.

`/status.json` and the health check answered chi's empty 405 to any
method they do not take. They now answer with the same JSON 405 as the
routes under `/api/v1` and `/ip`.

The home page's IP lookup card showed only the answer's `data`. It now
shows the whole answer, `@meta` included.

A table test checks `@meta` on every JSON route, in its successes and
its errors.

Model: opus-5-5
2026-10-27 21:20:07 +01:00
clawbot e8d04f1304 Give every API value with a unit as a number (closes #199)
check / check (push) Canceled after 0s
A value with a unit is now a number, and its field name ends in the
unit, as the vendored API_POLICIES.md requires. @meta.execution_time
becomes execution_time_ms. In the stats answers, uptime,
connection_duration, memory_usage and whois_stats.current_interval
become uptime_seconds, connection_duration_seconds, memory_usage_bytes
and whois_stats.current_interval_seconds. connection_duration_seconds
is null while the feed is not connected. The status page formats the
numbers itself. REPO_POLICIES.md is re-vendored from the same
sneak/prompts commit.

Model: opus-5-5
2026-10-27 20:19:01 +01:00
clawbot a671b8a0b0 Give every IP lookup result the same fields (closes #200)
check / check (push) Canceled after 0s
Every result of /ip/, /ip/{addr} and /api/v1/ip/{ip} now has every
field. A field without a value is null instead of left out, so an
address with no live route has all its route and AS fields set to null.
ptr is an empty list when there are no PTR names.

Two new fields, num_origins and origin_asns, give how many origin ASes
peers carry the netblock with, and which. The routing table's prefix
summary now lists them. asn still names the origin that most peers
carry.

first_seen is null, no longer the zero time, when the database has not
stored the netblock yet.

Model: opus-5-5
2026-10-27 19:54:02 +01:00
clawbot fb5e98feab Trust reverse proxies at private addresses by default (closes #198)
check / check (push) Canceled after 0s
With TRUSTED_PROXIES unset, the server now trusts reverse proxies in the
private ranges of RFC 1918. A proxy such as traefik on a Docker network is
then believed. /ip/, the home page's curl card and the request log give the
client's address from X-Forwarded-For instead of the proxy's address.

A list in TRUSTED_PROXIES replaces the default. TRUSTED_PROXIES set but
empty trusts no proxy, so the config package reads it with os.LookupEnv.
A direct client at a public address still cannot choose its address with
a forged header.

Model: opus-5-5
2026-10-27 19:18:34 +01:00
clawbot 281d162e27 Show figures of the routing table on the home page (closes #154)
check / check (push) Canceled after 0s
The home page gets cards for the newest and oldest kept route, the newest
and oldest updated AS, the AS with the most prefixes, the ASes with a
single prefix and fifteen well-known networks. Two more cards count the
bogon announcements and the prefixes with several origin ASes. They link
to the new pages /bogons and /multiple-origins, which list up to 1,000
entries each.

The new package internal/bogon tells reserved prefixes and AS numbers,
from the IANA special-purpose and address space registries. Table.Overview
walks the table and lets the read lock go every 10,000 prefixes. The
server works the cards out once a minute and serves that copy. A new index
on asns(last_seen) finds the newest and oldest updated AS.

Model: opus-5-5
2026-10-27 18:27:01 +01:00