Compare commits
3
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
136cf0b7a4 | ||
|
|
6de44ac054 | ||
|
|
cca402c73d |
No files matched your search
+1
-1
@@ -105,7 +105,7 @@ RUN apt-get update && apt-get install -y --no-install-recommends \
|
|||||||
ARG CACHEBUST
|
ARG CACHEBUST
|
||||||
|
|
||||||
# Create non-root user
|
# Create non-root user
|
||||||
RUN useradd -r -u 1000 -m routewatch
|
RUN useradd -u 1000 -m routewatch
|
||||||
|
|
||||||
RUN mkdir -p /var/lib/berlin.sneak.app.routewatch && chown routewatch:routewatch /var/lib/berlin.sneak.app.routewatch
|
RUN mkdir -p /var/lib/berlin.sneak.app.routewatch && chown routewatch:routewatch /var/lib/berlin.sneak.app.routewatch
|
||||||
|
|
||||||
|
|||||||
@@ -143,9 +143,9 @@ and origin AS can have two kept routes, one or none.
|
|||||||
AS path, a next hop and a time. The next hop is the router the peer sends the
|
AS path, a next hop and a time. The next hop is the router the peer sends the
|
||||||
prefix's traffic to. RouteWatch records the peer itself as the next hop. The
|
prefix's traffic to. RouteWatch records the peer itself as the next hop. The
|
||||||
page also lists the prefix's origin changes, newest first. An origin change is
|
page also lists the prefix's origin changes, newest first. An origin change is
|
||||||
an AS starting to announce the prefix, when its first peer carries it, or
|
recorded when the first peer starts to carry the prefix with an origin AS, or
|
||||||
stopping, when its last peer no longer does. The page shows at most the newest
|
when the last peer stops. The page shows at most the newest 5,000. Each row
|
||||||
5,000. Each row shows:
|
shows:
|
||||||
- the time
|
- the time
|
||||||
- announced or withdrawn
|
- announced or withdrawn
|
||||||
- the origin AS
|
- the origin AS
|
||||||
@@ -197,8 +197,9 @@ also carries an `@meta` object:
|
|||||||
- `api_version`: `1`
|
- `api_version`: `1`
|
||||||
- `execution_time_ms`: the number of milliseconds the request took
|
- `execution_time_ms`: the number of milliseconds the request took
|
||||||
|
|
||||||
Every value with a unit is a number, and the unit ends the field name. Examples
|
Every value with a unit is a number. The unit ends the field name, as in `_ms`,
|
||||||
are `_ms`, `_seconds`, `_bytes`, `_per_sec` and `_per_min`.
|
`_seconds`, `_bytes`, `_per_sec` and `_per_min`. `latitude` and `longitude` are
|
||||||
|
in degrees, and their names do not say so.
|
||||||
|
|
||||||
The paths:
|
The paths:
|
||||||
|
|
||||||
@@ -502,11 +503,13 @@ How RouteWatch keeps the answers:
|
|||||||
there.
|
there.
|
||||||
- It writes new answers every 5 seconds to `cache.db` in the state directory
|
- It writes new answers every 5 seconds to `cache.db` in the state directory
|
||||||
(see Configuration), and loads them from there at startup.
|
(see Configuration), and loads them from there at startup.
|
||||||
- An answer expires after the TTL of its records, the shortest one when there
|
- Each DNS record carries a TTL, the time for which it may be kept. An answer
|
||||||
are several. RouteWatch keeps it for at least 60 seconds and at most 24 hours.
|
expires after the shortest TTL of its records. RouteWatch keeps it for at
|
||||||
- When an address has no PTR names, that answer expires after the SOA record's
|
least 60 seconds and at most 24 hours.
|
||||||
minimum field, or the record's own TTL when that is shorter. RouteWatch keeps
|
- When an address has no PTR names, the answer's SOA record says how long that
|
||||||
it for at most 1 hour.
|
answer may be kept. The answer expires after the SOA record's minimum field,
|
||||||
|
or the record's own TTL when that is shorter. RouteWatch keeps it for at most
|
||||||
|
1 hour.
|
||||||
- An answer with no PTR names and no SOA record is not kept. Neither is one
|
- An answer with no PTR names and no SOA record is not kept. Neither is one
|
||||||
whose SOA record has a TTL or minimum field of 0.
|
whose SOA record has a TTL or minimum field of 0.
|
||||||
- A failed lookup is not kept, so the next lookup of the address asks again.
|
- A failed lookup is not kept, so the next lookup of the address asks again.
|
||||||
@@ -520,7 +523,8 @@ An IP lookup finds where each address is, on a best-effort basis. It uses two
|
|||||||
sources:
|
sources:
|
||||||
|
|
||||||
- [geojs](https://www.geojs.io/) is asked first. One request carries every
|
- [geojs](https://www.geojs.io/) is asked first. One request carries every
|
||||||
address of the lookup, and it has 1 second to answer.
|
address of the lookup that the cache does not hold. geojs has 1 second to
|
||||||
|
answer.
|
||||||
- [DB-IP City Lite](https://db-ip.com/db/download/ip-to-city-lite) answers when
|
- [DB-IP City Lite](https://db-ip.com/db/download/ip-to-city-lite) answers when
|
||||||
geojs has no answer for the address. It is a database file that RouteWatch
|
geojs has no answer for the address. It is a database file that RouteWatch
|
||||||
reads itself.
|
reads itself.
|
||||||
@@ -702,8 +706,10 @@ What happens at each limit:
|
|||||||
allocating more.
|
allocating more.
|
||||||
- The SQLite hard heap limit: a database statement fails with an out-of-memory
|
- The SQLite hard heap limit: a database statement fails with an out-of-memory
|
||||||
error, which the handler logs. The prefix handler writes the routing table's
|
error, which the handler logs. The prefix handler writes the routing table's
|
||||||
changes again in a later write. Everything else in the failed write is
|
changes and its origin changes again in a later write. Everything else in a
|
||||||
dropped. The process keeps running.
|
handler's failed write is dropped. The PTR cache and the geolocation part
|
||||||
|
write their answers to `cache.db` again in a later write. The process keeps
|
||||||
|
running.
|
||||||
- The handler queues: the handlers are the four parts that take the feed's
|
- The handler queues: the handlers are the four parts that take the feed's
|
||||||
messages (see Design). The prefix handler's queue holds at most 100,000
|
messages (see Design). The prefix handler's queue holds at most 100,000
|
||||||
messages, about 100 MiB. Each of the other three holds at most 20,000. When a
|
messages, about 100 MiB. Each of the other three holds at most 20,000. When a
|
||||||
@@ -1070,8 +1076,8 @@ calls one of them. The scripts:
|
|||||||
- `script/projectname`: print the project name, `routewatch`, which
|
- `script/projectname`: print the project name, `routewatch`, which
|
||||||
`script/docker` uses as the image tag.
|
`script/docker` uses as the image tag.
|
||||||
- `script/test`: build the `test` phase of the `Dockerfile` alone. It runs
|
- `script/test`: build the `test` phase of the `Dockerfile` alone. It runs
|
||||||
`go test -short -count=1 -timeout 90s -race -cover ./...`, and reruns it with
|
`go test -short -count=1 -timeout 90s -race -cover ./...`. On failure it runs
|
||||||
`-v` on failure.
|
the tests again with `-v` and without `-cover`.
|
||||||
- `script/lint`: build the `lint` phase of the `Dockerfile` alone. It runs
|
- `script/lint`: build the `lint` phase of the `Dockerfile` alone. It runs
|
||||||
`golangci-lint` v2.14.0 with `.golangci.yml`.
|
`golangci-lint` v2.14.0 with `.golangci.yml`.
|
||||||
- `script/fmt`: format the Go code with `go fmt` and the markdown with
|
- `script/fmt`: format the Go code with `go fmt` and the markdown with
|
||||||
|
|||||||
@@ -22,37 +22,52 @@ formatting of the Go code and the markdown. Then it builds the image.
|
|||||||
The image sets memory ceilings for a 5 GiB container (README "Memory"). README
|
The image sets memory ceilings for a 5 GiB container (README "Memory"). README
|
||||||
"Running under upaas" says how to run it.
|
"Running under upaas" says how to run it.
|
||||||
|
|
||||||
A five-day run of `1c65656` on the live feed, from 2026-10-27 to 2026-11-01,
|
Five runs of `next` on the live feed have ended since 2026-10-27, from three
|
||||||
found three problems:
|
hours to five days long. The runs found the handlers dropping updates for five
|
||||||
|
causes, and the WHOIS fetcher asking a registry that had refused it. Each is
|
||||||
|
fixed on `next`, and its Completed Steps entry below names its issue. A 21-hour
|
||||||
|
run of `742e759`, which has every fix but one, dropped no update except after
|
||||||
|
the feed stalled. That cause, https://git.eeqj.de/sneak/routewatch/issues/229,
|
||||||
|
is fixed on `next`.
|
||||||
|
|
||||||
- The prefix handler dropped updates while a peer's session went down
|
A run of `e58c663` on the live feed began at 09:19 UTC on 2026-11-10
|
||||||
(https://git.eeqj.de/sneak/routewatch/issues/216).
|
(https://git.eeqj.de/sneak/routewatch/issues/232). It is to confirm that fix.
|
||||||
- The AS handler dropped updates when the feed burst
|
Next Step lists what its outcome reports. The seven IP lookup additions of
|
||||||
(https://git.eeqj.de/sneak/routewatch/issues/217).
|
https://git.eeqj.de/sneak/routewatch/issues/203 landed after it began, so it
|
||||||
- The WHOIS fetcher kept asking a registry that had refused it
|
does not exercise them.
|
||||||
(https://git.eeqj.de/sneak/routewatch/issues/218).
|
|
||||||
|
|
||||||
All three are fixed on `next`. So is
|
|
||||||
https://git.eeqj.de/sneak/routewatch/issues/222, which the third fix brought up.
|
|
||||||
A run of `26d939a` on the live feed showed the prefix handler still dropping
|
|
||||||
updates while the home page's cards were worked out.
|
|
||||||
https://git.eeqj.de/sneak/routewatch/issues/226 fixed that on `next`.
|
|
||||||
https://git.eeqj.de/sneak/routewatch/issues/225, also on `next`, takes the peers
|
|
||||||
whose sessions go down together out of the table in one walk. A 24-hour run of
|
|
||||||
`9bd0e91` on the live feed is checking whether any handler still drops updates.
|
|
||||||
|
|
||||||
The order of the project's remaining work is kept on
|
The order of the project's remaining work is kept on
|
||||||
https://git.eeqj.de/sneak/routewatch/issues/86.
|
https://git.eeqj.de/sneak/routewatch/issues/86.
|
||||||
|
|
||||||
# Next Step
|
# Next Step
|
||||||
|
|
||||||
Post the outcome of the 24-hour run of `9bd0e91` on
|
Post the outcome of the run of `e58c663` on
|
||||||
https://git.eeqj.de/sneak/routewatch/issues/226: whether any handler still drops
|
https://git.eeqj.de/sneak/routewatch/issues/232. The run lasts at least 24
|
||||||
updates, and the process's peak memory against 5 GiB. File an issue for each
|
hours. If the feed has not stalled or reconnected by then, the run goes on until
|
||||||
problem the run finds.
|
it does, up to 72 hours in all. The outcome says:
|
||||||
|
|
||||||
|
- whether the feed stalled or reconnected
|
||||||
|
- whether any handler dropped an update, and the most messages the prefix
|
||||||
|
handler's queue held at once
|
||||||
|
- the most messages the prefix handler held back at once
|
||||||
|
- the size of the `origin_changes` table on disk
|
||||||
|
- the process's peak memory against 5 GiB
|
||||||
|
|
||||||
|
File an issue for each problem the run finds.
|
||||||
|
|
||||||
# Completed Steps
|
# Completed Steps
|
||||||
|
|
||||||
|
- 2026-11-10: `README.md` and this file are read whole against the rules of
|
||||||
|
https://git.eeqj.de/sneak/routewatch/issues/157, and every statement except
|
||||||
|
measured figures is checked against the code on `next`. Four README sentences
|
||||||
|
failed the prose rules, on origin changes, PTR lookups and geolocation, and
|
||||||
|
five disagreed with the code, on units, geolocation, a failed database write
|
||||||
|
and `script/test`. Each is rewritten. Status and Next Step describe the
|
||||||
|
live-feed run in progress (closes
|
||||||
|
https://git.eeqj.de/sneak/routewatch/issues/251).
|
||||||
|
- 2026-11-10: the image creates the `routewatch` user as a regular user with UID
|
||||||
|
1000, so the build no longer warns about the system UID range (closes
|
||||||
|
https://git.eeqj.de/sneak/routewatch/issues/248).
|
||||||
- 2026-11-10: each result of the IP lookup has `geo`, where the address is. It
|
- 2026-11-10: each result of the IP lookup has `geo`, where the address is. It
|
||||||
comes from geojs, with DB-IP's City Lite database as the fallback. The geojs
|
comes from geojs, with DB-IP's City Lite database as the fallback. The geojs
|
||||||
answers stay in memory and in `cache.db` for 7 days (closes
|
answers stay in memory and in `cache.db` for 7 days (closes
|
||||||
@@ -60,16 +75,18 @@ problem the run finds.
|
|||||||
- 2026-11-10: the IP lookup checks the `icloud_private_relay` list of Apple's
|
- 2026-11-10: the IP lookup checks the `icloud_private_relay` list of Apple's
|
||||||
iCloud Private Relay networks, downloaded once a day. Each result has
|
iCloud Private Relay networks, downloaded once a day. Each result has
|
||||||
`flags.is_relay` (closes https://git.eeqj.de/sneak/routewatch/issues/237).
|
`flags.is_relay` (closes https://git.eeqj.de/sneak/routewatch/issues/237).
|
||||||
- 2026-11-10: the IP lookup checks the `protonvpn` list of ProtonVPN's /24
|
- 2026-11-10: the IP lookup checks two ProtonVPN lists from a public mirror of
|
||||||
networks and the `protonvpn_entry` list of its servers' IPv4 and IPv6
|
Proton's server list. `protonvpn` holds the /24 networks of ProtonVPN's
|
||||||
addresses, from a public mirror of Proton's server list (closes
|
servers, and `protonvpn_entry` holds the servers' IPv4 and IPv6 addresses
|
||||||
https://git.eeqj.de/sneak/routewatch/issues/236).
|
(closes https://git.eeqj.de/sneak/routewatch/issues/236).
|
||||||
- 2026-11-10: the IP lookup finds PTR names over DNS-over-HTTPS and caches them
|
- 2026-11-10: the IP lookup finds PTR names over DNS-over-HTTPS and caches them
|
||||||
for the record's TTL, in memory and in `cache.db` (closes
|
in memory and in `cache.db` for the record's TTL. A TTL is the time for which
|
||||||
|
a DNS record may be kept (closes
|
||||||
https://git.eeqj.de/sneak/routewatch/issues/238).
|
https://git.eeqj.de/sneak/routewatch/issues/238).
|
||||||
- 2026-11-10: the tests no longer fail when a busy host stalls for several
|
- 2026-11-10: the tests no longer fail when a busy host stalls for several
|
||||||
seconds. Most such waits are now 15 seconds. The test of the stream's idle
|
seconds. Most waits for something that should happen are now 15 seconds. Tests
|
||||||
timeout runs on a fake clock (closes #240)
|
that depend on a short idle timeout or a request limit run on a fake clock
|
||||||
|
(closes https://git.eeqj.de/sneak/routewatch/issues/240)
|
||||||
- 2026-11-10: the IP lookup checks the public VPN server lists of Mullvad,
|
- 2026-11-10: the IP lookup checks the public VPN server lists of Mullvad,
|
||||||
NordVPN, Private Internet Access and IVPN, and X4BNet's community list. Each
|
NordVPN, Private Internet Access and IVPN, and X4BNet's community list. Each
|
||||||
result has `flags.is_vpn` and `vpn_providers` (closes
|
result has `flags.is_vpn` and `vpn_providers` (closes
|
||||||
|
|||||||
Reference in new issue
Block a user