2 Commits
Author SHA1 Message Date
sneak 3bf20f4125 Bring TODO.md Status and Next Step to the run in progress (closes #251)
check / check (push) Successful in 1m14s
Status and Next Step described the 24-hour run of 9bd0e91, whose outcome
was posted on issue 226 on 3 November. They now describe the run of
e58c663 on issue 232: at least 24 hours, then on until the feed stalls or
reconnects, up to 72 hours, and what its outcome reports. Two Completed
Steps entries of 10 November are rewritten: the ProtonVPN entry ran past
30 words, and the busy-host test entry referred to waits it had not named.
The PTR entry defines TTL. An entry for this read is added.

Model: fable-5-1
2026-11-10 16:07:33 +00:00
sneak ed7a565e91 Rewrite the README sentences that fail the prose rules or disagree with the code
A fresh read of README.md whole against the rules of issue 157, with every
statement except measured figures checked against the code on next. Four
sentences fail the prose rules, all in sections added since 742e759: the
origin change definition, the first uses of TTL and SOA record, and the
geojs request. Five disagree with the code: the geojs request carries only
the addresses the cache does not hold; after a write fails on SQLite's hard
heap limit, the prefix handler also writes its origin changes again, and
the PTR and geolocation caches write their answers again; the test rerun on
failure drops -cover; and latitude and longitude are in degrees, which
their names do not say.

Model: fable-5-1
2026-11-10 16:07:33 +00:00
2 changed files with 63 additions and 43 deletions

No files matched your search

+21 -15
View File
@@ -143,9 +143,9 @@ and origin AS can have two kept routes, one or none.
AS path, a next hop and a time. The next hop is the router the peer sends the
prefix's traffic to. RouteWatch records the peer itself as the next hop. The
page also lists the prefix's origin changes, newest first. An origin change is
an AS starting to announce the prefix, when its first peer carries it, or
stopping, when its last peer no longer does. The page shows at most the newest
5,000. Each row shows:
recorded when the first peer starts to carry the prefix with an origin AS, or
when the last peer stops. The page shows at most the newest 5,000. Each row
shows:
- the time
- announced or withdrawn
- the origin AS
@@ -197,8 +197,9 @@ also carries an `@meta` object:
- `api_version`: `1`
- `execution_time_ms`: the number of milliseconds the request took
Every value with a unit is a number, and the unit ends the field name. Examples
are `_ms`, `_seconds`, `_bytes`, `_per_sec` and `_per_min`.
Every value with a unit is a number. The unit ends the field name, as in `_ms`,
`_seconds`, `_bytes`, `_per_sec` and `_per_min`. `latitude` and `longitude` are
in degrees, and their names do not say so.
The paths:
@@ -502,11 +503,13 @@ How RouteWatch keeps the answers:
there.
- It writes new answers every 5 seconds to `cache.db` in the state directory
(see Configuration), and loads them from there at startup.
- An answer expires after the TTL of its records, the shortest one when there
are several. RouteWatch keeps it for at least 60 seconds and at most 24 hours.
- When an address has no PTR names, that answer expires after the SOA record's
minimum field, or the record's own TTL when that is shorter. RouteWatch keeps
it for at most 1 hour.
- Each DNS record carries a TTL, the time for which it may be kept. An answer
expires after the shortest TTL of its records. RouteWatch keeps it for at
least 60 seconds and at most 24 hours.
- When an address has no PTR names, the answer's SOA record says how long that
answer may be kept. The answer expires after the SOA record's minimum field,
or the record's own TTL when that is shorter. RouteWatch keeps it for at most
1 hour.
- An answer with no PTR names and no SOA record is not kept. Neither is one
whose SOA record has a TTL or minimum field of 0.
- A failed lookup is not kept, so the next lookup of the address asks again.
@@ -520,7 +523,8 @@ An IP lookup finds where each address is, on a best-effort basis. It uses two
sources:
- [geojs](https://www.geojs.io/) is asked first. One request carries every
address of the lookup, and it has 1 second to answer.
address of the lookup that the cache does not hold. geojs has 1 second to
answer.
- [DB-IP City Lite](https://db-ip.com/db/download/ip-to-city-lite) answers when
geojs has no answer for the address. It is a database file that RouteWatch
reads itself.
@@ -702,8 +706,10 @@ What happens at each limit:
allocating more.
- The SQLite hard heap limit: a database statement fails with an out-of-memory
error, which the handler logs. The prefix handler writes the routing table's
changes again in a later write. Everything else in the failed write is
dropped. The process keeps running.
changes and its origin changes again in a later write. Everything else in a
handler's failed write is dropped. The PTR cache and the geolocation part
write their answers to `cache.db` again in a later write. The process keeps
running.
- The handler queues: the handlers are the four parts that take the feed's
messages (see Design). The prefix handler's queue holds at most 100,000
messages, about 100 MiB. Each of the other three holds at most 20,000. When a
@@ -1070,8 +1076,8 @@ calls one of them. The scripts:
- `script/projectname`: print the project name, `routewatch`, which
`script/docker` uses as the image tag.
- `script/test`: build the `test` phase of the `Dockerfile` alone. It runs
`go test -short -count=1 -timeout 90s -race -cover ./...`, and reruns it with
`-v` on failure.
`go test -short -count=1 -timeout 90s -race -cover ./...`. On failure it runs
the tests again with `-v` and without `-cover`.
- `script/lint`: build the `lint` phase of the `Dockerfile` alone. It runs
`golangci-lint` v2.14.0 with `.golangci.yml`.
- `script/fmt`: format the Go code with `go fmt` and the markdown with
+42 -28
View File
@@ -22,37 +22,49 @@ formatting of the Go code and the markdown. Then it builds the image.
The image sets memory ceilings for a 5 GiB container (README "Memory"). README
"Running under upaas" says how to run it.
A five-day run of `1c65656` on the live feed, from 2026-10-27 to 2026-11-01,
found three problems:
Five runs of `next` on the live feed have ended since 2026-10-27, from three
hours to five days long. The runs found the handlers dropping updates for five
causes, and the WHOIS fetcher asking a registry that had refused it. Each is
fixed on `next`, and its Completed Steps entry below names its issue. A 21-hour
run of `742e759`, which has every fix but one, dropped no update except after
the feed stalled. That cause, https://git.eeqj.de/sneak/routewatch/issues/229,
is fixed on `next`.
- The prefix handler dropped updates while a peer's session went down
(https://git.eeqj.de/sneak/routewatch/issues/216).
- The AS handler dropped updates when the feed burst
(https://git.eeqj.de/sneak/routewatch/issues/217).
- The WHOIS fetcher kept asking a registry that had refused it
(https://git.eeqj.de/sneak/routewatch/issues/218).
All three are fixed on `next`. So is
https://git.eeqj.de/sneak/routewatch/issues/222, which the third fix brought up.
A run of `26d939a` on the live feed showed the prefix handler still dropping
updates while the home page's cards were worked out.
https://git.eeqj.de/sneak/routewatch/issues/226 fixed that on `next`.
https://git.eeqj.de/sneak/routewatch/issues/225, also on `next`, takes the peers
whose sessions go down together out of the table in one walk. A 24-hour run of
`9bd0e91` on the live feed is checking whether any handler still drops updates.
A run of `e58c663` on the live feed began at 09:19 UTC on 2026-11-10
(https://git.eeqj.de/sneak/routewatch/issues/232). It is to confirm that fix.
Next Step lists what its outcome reports. The seven IP lookup additions of
https://git.eeqj.de/sneak/routewatch/issues/203 landed after it began, so it
does not exercise them.
The order of the project's remaining work is kept on
https://git.eeqj.de/sneak/routewatch/issues/86.
# Next Step
Post the outcome of the 24-hour run of `9bd0e91` on
https://git.eeqj.de/sneak/routewatch/issues/226: whether any handler still drops
updates, and the process's peak memory against 5 GiB. File an issue for each
problem the run finds.
Post the outcome of the run of `e58c663` on
https://git.eeqj.de/sneak/routewatch/issues/232. The run lasts at least 24
hours. If the feed has not stalled or reconnected by then, the run goes on until
it does, up to 72 hours in all. The outcome says:
- whether the feed stalled or reconnected
- whether any handler dropped an update, and the most messages the prefix
handler's queue held at once
- the most messages the prefix handler held back at once
- the size of the `origin_changes` table on disk
- the process's peak memory against 5 GiB
File an issue for each problem the run finds.
# Completed Steps
- 2026-11-10: `README.md` and this file are read whole against the rules of
https://git.eeqj.de/sneak/routewatch/issues/157, and every statement except
measured figures is checked against the code on `next`. Four README sentences
failed the prose rules, on origin changes, PTR lookups and geolocation, and
five disagreed with the code, on units, geolocation, a failed database write
and `script/test`. Each is rewritten. Status and Next Step describe the
live-feed run in progress (closes
https://git.eeqj.de/sneak/routewatch/issues/251).
- 2026-11-10: the image creates the `routewatch` user as a regular user with UID
1000, so the build no longer warns about the system UID range (closes
https://git.eeqj.de/sneak/routewatch/issues/248).
@@ -63,16 +75,18 @@ problem the run finds.
- 2026-11-10: the IP lookup checks the `icloud_private_relay` list of Apple's
iCloud Private Relay networks, downloaded once a day. Each result has
`flags.is_relay` (closes https://git.eeqj.de/sneak/routewatch/issues/237).
- 2026-11-10: the IP lookup checks the `protonvpn` list of ProtonVPN's /24
networks and the `protonvpn_entry` list of its servers' IPv4 and IPv6
addresses, from a public mirror of Proton's server list (closes
https://git.eeqj.de/sneak/routewatch/issues/236).
- 2026-11-10: the IP lookup checks two ProtonVPN lists from a public mirror of
Proton's server list. `protonvpn` holds the /24 networks of ProtonVPN's
servers, and `protonvpn_entry` holds the servers' IPv4 and IPv6 addresses
(closes https://git.eeqj.de/sneak/routewatch/issues/236).
- 2026-11-10: the IP lookup finds PTR names over DNS-over-HTTPS and caches them
for the record's TTL, in memory and in `cache.db` (closes
in memory and in `cache.db` for the record's TTL. A TTL is the time for which
a DNS record may be kept (closes
https://git.eeqj.de/sneak/routewatch/issues/238).
- 2026-11-10: the tests no longer fail when a busy host stalls for several
seconds. Most such waits are now 15 seconds. The test of the stream's idle
timeout runs on a fake clock (closes #240)
seconds. Most waits for something that should happen are now 15 seconds. Tests
that depend on a short idle timeout or a request limit run on a fake clock
(closes https://git.eeqj.de/sneak/routewatch/issues/240)
- 2026-11-10: the IP lookup checks the public VPN server lists of Mullvad,
NordVPN, Private Internet Access and IVPN, and X4BNet's community list. Each
result has `flags.is_vpn` and `vpn_providers` (closes