Author SHA1 Message Date
sneak d9497319c3 Bring TODO.md Status and Next Step to the run in progress (closes #251)
check / check (push) Successful in 6m29s
Status and Next Step described the 24-hour run of 9bd0e91, whose outcome
was posted on issue 226 on 3 November. They now describe the run of
e58c663 on issue 232: at least 24 hours, then on until the feed stalls or
reconnects, up to 72 hours, and what its outcome reports. Two Completed
Steps entries of 10 November are rewritten: the ProtonVPN entry ran past
30 words, and the busy-host test entry referred to waits it had not named.
The PTR entry defines TTL. An entry for this read is added.

Model: fable-5-1
2026-11-10 16:32:28 +00:00
sneak 7e8aae1875 Rewrite the README sentences that fail the prose rules or disagree with the code
A fresh read of README.md whole against the rules of issue 157, with every
statement except measured figures and what outside services do checked
against the code on next. Four sentences fail the prose rules, all in
sections added since 742e759: the origin change definition, the first uses
of TTL and SOA record, and the geojs request. Five disagree with the code:
the geojs request carries only the addresses the cache does not hold; after
a write fails on SQLite's hard heap limit, the prefix handler also writes
its origin changes again, and the PTR and geolocation caches write their
answers again; the test rerun on failure drops -cover; and latitude and
longitude are in degrees, which their names do not say.

Model: fable-5-1
2026-11-10 16:32:04 +00:00
clawbot 6de44ac054 Create the image's app user without the system-user flag (closes #248)
check / check (push) Successful in 1m8s
`useradd -r` asks for a system user and UID 1000 is above the system range, so the image build warned. Dropping `-r` ends the warning; the user keeps UID 1000, so the owner of the data directory does not change.

Model: opus-5-5
2026-11-10 16:23:20 +01:00
clawbot cca402c73d Add best-effort geolocation to the IP lookup from geojs and DB-IP City Lite (closes #239)
check / check (push) Successful in 1m41s
Every IP lookup result gains `geo`. One request to geojs carries all uncached addresses of a lookup, within 1 second. On a timeout, an error or an answer with no location, the lookup reads DB-IP City Lite, downloaded monthly into the state directory. Geojs answers stay in memory, at most 250,000, and in `cache.db` for 7 days. The home page footer credits DB-IP.

Judgement call: a reserved address goes to neither source, and its `geo` is null.
Judgement call: `updated` for a DB-IP answer is when the copy was downloaded.
Deviation: the cache is SQLite, by sneak's words, not the JSON file the standing rule asks for.

Model: opus-5-5
2026-11-10 15:48:12 +01:00
clawbot 2c3f6f71dc Add the iCloud Private Relay list to the IP lookup, with is_relay (closes #237)
check / check (push) Failing after 17m2s
Adds Apple's iCloud Private Relay egress list as `icloud_private_relay`, downloaded once a day, and `flags.is_relay`, true when an address is on it. Loaded, the list adds about 0.3 MiB to the Go heap; parsing it allocates about 94 MiB, freed by the next garbage collection.

Judgement call: a line with no comma fails the parse, so a plain network list served at Apple's URL cannot replace a good copy.

Model: opus-5-5
2026-11-10 15:13:55 +01:00
clawbot b2553f044d Add the ProtonVPN server networks and addresses to the IP lookup (closes #236)
check / check (push) Successful in 1m27s
Proton's own server list answers only a logged-in session, so the lists come from the public mirror https://github.com/tn3w/ProtonVPN-IPs, downloaded once a day: `protonvpn`, its /24 networks, and `protonvpn_entry`, its servers' IPv4 and IPv6 addresses. Both feed `flags.is_vpn` and `vpn_providers`.

Judgement call: the /24 networks are listed as well as the exact addresses, because coverage matters more than attribution, so neighbouring addresses in those networks count too.
Risk: the mirror is one person's repository; if it stops updating, downloads keep succeeding with old data.

Model: opus-5-5
2026-11-10 14:14:06 +01:00
10 changed files with 350 additions and 71 deletions

No files matched your search

+1 -1
View File
@@ -105,7 +105,7 @@ RUN apt-get update && apt-get install -y --no-install-recommends \
ARG CACHEBUST ARG CACHEBUST
# Create non-root user # Create non-root user
RUN useradd -r -u 1000 -m routewatch RUN useradd -u 1000 -m routewatch
RUN mkdir -p /var/lib/berlin.sneak.app.routewatch && chown routewatch:routewatch /var/lib/berlin.sneak.app.routewatch RUN mkdir -p /var/lib/berlin.sneak.app.routewatch && chown routewatch:routewatch /var/lib/berlin.sneak.app.routewatch
+41 -16
View File
@@ -143,9 +143,9 @@ and origin AS can have two kept routes, one or none.
AS path, a next hop and a time. The next hop is the router the peer sends the AS path, a next hop and a time. The next hop is the router the peer sends the
prefix's traffic to. RouteWatch records the peer itself as the next hop. The prefix's traffic to. RouteWatch records the peer itself as the next hop. The
page also lists the prefix's origin changes, newest first. An origin change is page also lists the prefix's origin changes, newest first. An origin change is
an AS starting to announce the prefix, when its first peer carries it, or recorded when the first peer starts to carry the prefix with an origin AS, or
stopping, when its last peer no longer does. The page shows at most the newest when the last peer stops. The page shows at most the newest 5,000. Each row
5,000. Each row shows: shows:
- the time - the time
- announced or withdrawn - announced or withdrawn
- the origin AS - the origin AS
@@ -197,8 +197,9 @@ also carries an `@meta` object:
- `api_version`: `1` - `api_version`: `1`
- `execution_time_ms`: the number of milliseconds the request took - `execution_time_ms`: the number of milliseconds the request took
Every value with a unit is a number, and the unit ends the field name. Examples Every value with a unit is a number. The unit ends the field name, as in `_ms`,
are `_ms`, `_seconds`, `_bytes`, `_per_sec` and `_per_min`. `_seconds`, `_bytes`, `_per_sec` and `_per_min`. `latitude` and `longitude` are
in degrees, and their names do not say so.
The paths: The paths:
@@ -287,6 +288,7 @@ address. Every entry has the same fields:
- `is_tor`: true when the address is on the `tor_exits` list - `is_tor`: true when the address is on the `tor_exits` list
- `is_vpn`: true when the address is on any list of VPN servers (see IP - `is_vpn`: true when the address is on any list of VPN servers (see IP
lists) lists)
- `is_relay`: true when the address is on the `icloud_private_relay` list
- `is_bogon`: true when the address is reserved, so that no network should - `is_bogon`: true when the address is reserved, so that no network should
announce it on the internet announce it on the internet
- `vpn_providers`: the names of the lists of VPN servers the address is on, or - `vpn_providers`: the names of the lists of VPN servers the address is on, or
@@ -422,6 +424,9 @@ The lists:
last 30 days. From last 30 days. From
[FireHOL](https://iplists.firehol.org/?ipset=firehol_level3), downloaded every [FireHOL](https://iplists.firehol.org/?ipset=firehol_level3), downloaded every
3 hours. 3 hours.
- `icloud_private_relay`: the networks where iCloud Private Relay traffic
leaves. From [Apple](https://mask-api.icloud.com/egress-ip-ranges.csv),
downloaded once a day.
- `ivpn`: VPN servers run by IVPN. From - `ivpn`: VPN servers run by IVPN. From
[IVPN](https://api.ivpn.net/v5/servers.json), downloaded once a day. [IVPN](https://api.ivpn.net/v5/servers.json), downloaded once a day.
- `mullvad`: VPN servers run by Mullvad. From - `mullvad`: VPN servers run by Mullvad. From
@@ -432,6 +437,14 @@ The lists:
- `pia`: VPN servers run by Private Internet Access. From - `pia`: VPN servers run by Private Internet Access. From
[Private Internet Access](https://serverlist.piaservers.net/vpninfo/servers/v6), [Private Internet Access](https://serverlist.piaservers.net/vpninfo/servers/v6),
downloaded once a day. downloaded once a day.
- `protonvpn`: the /24 networks that hold ProtonVPN servers. The whole /24 is
listed because the ProtonVPN-IPs README says traffic leaves from unpublished
addresses in it. From the public
[ProtonVPN-IPs](https://github.com/tn3w/ProtonVPN-IPs) mirror of Proton's
server list, downloaded once a day.
- `protonvpn_entry`: the IPv4 and IPv6 addresses of ProtonVPN servers. From the
same [ProtonVPN-IPs](https://github.com/tn3w/ProtonVPN-IPs) mirror, downloaded
once a day.
- `spamhaus_drop` and `spamhaus_dropv6`: IPv4 and IPv6 networks hijacked or - `spamhaus_drop` and `spamhaus_dropv6`: IPv4 and IPv6 networks hijacked or
leased by spammers or cyber criminals, according to Spamhaus. From leased by spammers or cyber criminals, according to Spamhaus. From
[Spamhaus DROP](https://www.spamhaus.org/blocklists/do-not-route-or-peer/), [Spamhaus DROP](https://www.spamhaus.org/blocklists/do-not-route-or-peer/),
@@ -490,11 +503,13 @@ How RouteWatch keeps the answers:
there. there.
- It writes new answers every 5 seconds to `cache.db` in the state directory - It writes new answers every 5 seconds to `cache.db` in the state directory
(see Configuration), and loads them from there at startup. (see Configuration), and loads them from there at startup.
- An answer expires after the TTL of its records, the shortest one when there - Each DNS record carries a TTL, the time for which it may be kept. An answer
are several. RouteWatch keeps it for at least 60 seconds and at most 24 hours. expires after the shortest TTL of its records. RouteWatch keeps it for at
- When an address has no PTR names, that answer expires after the SOA record's least 60 seconds and at most 24 hours.
minimum field, or the record's own TTL when that is shorter. RouteWatch keeps - When an address has no PTR names, the answer's SOA record says how long that
it for at most 1 hour. answer may be kept. The answer expires after the SOA record's minimum field,
or the record's own TTL when that is shorter. RouteWatch keeps it for at most
1 hour.
- An answer with no PTR names and no SOA record is not kept. Neither is one - An answer with no PTR names and no SOA record is not kept. Neither is one
whose SOA record has a TTL or minimum field of 0. whose SOA record has a TTL or minimum field of 0.
- A failed lookup is not kept, so the next lookup of the address asks again. - A failed lookup is not kept, so the next lookup of the address asks again.
@@ -508,7 +523,8 @@ An IP lookup finds where each address is, on a best-effort basis. It uses two
sources: sources:
- [geojs](https://www.geojs.io/) is asked first. One request carries every - [geojs](https://www.geojs.io/) is asked first. One request carries every
address of the lookup, and it has 1 second to answer. address of the lookup that the cache does not hold. geojs has 1 second to
answer.
- [DB-IP City Lite](https://db-ip.com/db/download/ip-to-city-lite) answers when - [DB-IP City Lite](https://db-ip.com/db/download/ip-to-city-lite) answers when
geojs has no answer for the address. It is a database file that RouteWatch geojs has no answer for the address. It is a database file that RouteWatch
reads itself. reads itself.
@@ -544,7 +560,7 @@ file.
How RouteWatch keeps the answers of geojs: How RouteWatch keeps the answers of geojs:
- It holds at most 250,000 addresses in memory, and drops the least recently - It holds at most 250,000 addresses in memory, and drops the least recently
used first. That takes about 75 MB of the Go heap. used first. That takes about 100 MB of the Go heap.
- It writes new answers every 5 seconds to `cache.db` in the state directory. - It writes new answers every 5 seconds to `cache.db` in the state directory.
- It loads the newest 250,000 answers from `cache.db` at startup. - It loads the newest 250,000 answers from `cache.db` at startup.
- An address not in memory is read from `cache.db` before geojs is asked. - An address not in memory is read from `cache.db` before geojs is asked.
@@ -668,6 +684,13 @@ The ceilings add up to 3.7 GiB:
limit caps the SQLite library. limit caps the SQLite library.
- About 0.2 GiB for everything else in the runtime. - About 0.2 GiB for everything else in the runtime.
The `icloud_private_relay` list has about 285,000 networks. Neighbouring
networks merge, which leaves about 6,900 address ranges. Loaded, the list adds
about 0.3 MiB to the Go heap and under 6 MiB to the process's resident memory.
Parsing the file allocates about 94 MiB in about 0.2 seconds. The next garbage
collection frees all of it but the 0.3 MiB above. Parsing happens at startup and
whenever a download brings a new version.
The SQLite library gets its memory from glibc, the C library. glibc keeps freed The SQLite library gets its memory from glibc, the C library. glibc keeps freed
memory in pools called arenas, and hands it back to the kernel slowly. It memory in pools called arenas, and hands it back to the kernel slowly. It
creates up to eight arenas per processor core. So on a host with many cores the creates up to eight arenas per processor core. So on a host with many cores the
@@ -683,8 +706,10 @@ What happens at each limit:
allocating more. allocating more.
- The SQLite hard heap limit: a database statement fails with an out-of-memory - The SQLite hard heap limit: a database statement fails with an out-of-memory
error, which the handler logs. The prefix handler writes the routing table's error, which the handler logs. The prefix handler writes the routing table's
changes again in a later write. Everything else in the failed write is changes and its origin changes again in a later write. Everything else in a
dropped. The process keeps running. handler's failed write is dropped. The PTR cache and the geolocation part
write their answers to `cache.db` again in a later write. The process keeps
running.
- The handler queues: the handlers are the four parts that take the feed's - The handler queues: the handlers are the four parts that take the feed's
messages (see Design). The prefix handler's queue holds at most 100,000 messages (see Design). The prefix handler's queue holds at most 100,000
messages, about 100 MiB. Each of the other three holds at most 20,000. When a messages, about 100 MiB. Each of the other three holds at most 20,000. When a
@@ -1051,8 +1076,8 @@ calls one of them. The scripts:
- `script/projectname`: print the project name, `routewatch`, which - `script/projectname`: print the project name, `routewatch`, which
`script/docker` uses as the image tag. `script/docker` uses as the image tag.
- `script/test`: build the `test` phase of the `Dockerfile` alone. It runs - `script/test`: build the `test` phase of the `Dockerfile` alone. It runs
`go test -short -count=1 -timeout 90s -race -cover ./...`, and reruns it with `go test -short -count=1 -timeout 90s -race -cover ./...`. On failure it runs
`-v` on failure. the tests again with `-v` and without `-cover`.
- `script/lint`: build the `lint` phase of the `Dockerfile` alone. It runs - `script/lint`: build the `lint` phase of the `Dockerfile` alone. It runs
`golangci-lint` v2.14.0 with `.golangci.yml`. `golangci-lint` v2.14.0 with `.golangci.yml`.
- `script/fmt`: format the Go code with `go fmt` and the markdown with - `script/fmt`: format the Go code with `go fmt` and the markdown with
+46 -24
View File
@@ -22,47 +22,69 @@ formatting of the Go code and the markdown. Then it builds the image.
The image sets memory ceilings for a 5 GiB container (README "Memory"). README The image sets memory ceilings for a 5 GiB container (README "Memory"). README
"Running under upaas" says how to run it. "Running under upaas" says how to run it.
A five-day run of `1c65656` on the live feed, from 2026-10-27 to 2026-11-01, Five runs of `next` on the live feed have ended since 2026-10-27, from three
found three problems: hours to five days long. The runs found the handlers dropping updates for five
causes, and the WHOIS fetcher asking a registry that had refused it. Each is
fixed on `next`, and its Completed Steps entry below names its issue. A 21-hour
run of `742e759`, which has every fix but one, dropped no update except after
the feed stalled. That cause, https://git.eeqj.de/sneak/routewatch/issues/229,
is fixed on `next`.
- The prefix handler dropped updates while a peer's session went down A run of `e58c663` on the live feed began at 09:19 UTC on 2026-11-10
(https://git.eeqj.de/sneak/routewatch/issues/216). (https://git.eeqj.de/sneak/routewatch/issues/232). It is to confirm that fix.
- The AS handler dropped updates when the feed burst Next Step lists what its outcome reports. The seven IP lookup additions of
(https://git.eeqj.de/sneak/routewatch/issues/217). https://git.eeqj.de/sneak/routewatch/issues/203 landed after it began, so it
- The WHOIS fetcher kept asking a registry that had refused it does not exercise them.
(https://git.eeqj.de/sneak/routewatch/issues/218).
All three are fixed on `next`. So is
https://git.eeqj.de/sneak/routewatch/issues/222, which the third fix brought up.
A run of `26d939a` on the live feed showed the prefix handler still dropping
updates while the home page's cards were worked out.
https://git.eeqj.de/sneak/routewatch/issues/226 fixed that on `next`.
https://git.eeqj.de/sneak/routewatch/issues/225, also on `next`, takes the peers
whose sessions go down together out of the table in one walk. A 24-hour run of
`9bd0e91` on the live feed is checking whether any handler still drops updates.
The order of the project's remaining work is kept on The order of the project's remaining work is kept on
https://git.eeqj.de/sneak/routewatch/issues/86. https://git.eeqj.de/sneak/routewatch/issues/86.
# Next Step # Next Step
Post the outcome of the 24-hour run of `9bd0e91` on Post the outcome of the run of `e58c663` on
https://git.eeqj.de/sneak/routewatch/issues/226: whether any handler still drops https://git.eeqj.de/sneak/routewatch/issues/232. The run lasts at least 24
updates, and the process's peak memory against 5 GiB. File an issue for each hours. If the feed has not stalled or reconnected by then, the run goes on until
problem the run finds. it does, up to 72 hours in all. The outcome says:
- whether the feed stalled or reconnected
- whether any handler dropped an update, and the most messages the prefix
handler's queue held at once
- the most messages the prefix handler held back at once
- the size of the `origin_changes` table on disk
- the process's peak memory against 5 GiB
File an issue for each problem the run finds.
# Completed Steps # Completed Steps
- 2026-11-10: `README.md` and this file are read whole against the rules of
https://git.eeqj.de/sneak/routewatch/issues/157, and every statement except
measured figures and what outside services do is checked against the code on
`next`. Four README sentences failed the prose rules. Five disagreed with the
code. Each is rewritten. Status and Next Step describe the live-feed run in
progress (closes https://git.eeqj.de/sneak/routewatch/issues/251).
- 2026-11-10: the image creates the `routewatch` user as a regular user with UID
1000, so the build no longer warns about the system UID range (closes
https://git.eeqj.de/sneak/routewatch/issues/248).
- 2026-11-10: each result of the IP lookup has `geo`, where the address is. It - 2026-11-10: each result of the IP lookup has `geo`, where the address is. It
comes from geojs, with DB-IP's City Lite database as the fallback. The geojs comes from geojs, with DB-IP's City Lite database as the fallback. The geojs
answers stay in memory and in `cache.db` for 7 days (closes answers stay in memory and in `cache.db` for 7 days (closes
https://git.eeqj.de/sneak/routewatch/issues/239). https://git.eeqj.de/sneak/routewatch/issues/239).
- 2026-11-10: the IP lookup checks the `icloud_private_relay` list of Apple's
iCloud Private Relay networks, downloaded once a day. Each result has
`flags.is_relay` (closes https://git.eeqj.de/sneak/routewatch/issues/237).
- 2026-11-10: the IP lookup checks two ProtonVPN lists from a public mirror of
Proton's server list. `protonvpn` holds the /24 networks of ProtonVPN's
servers, and `protonvpn_entry` holds the servers' IPv4 and IPv6 addresses
(closes https://git.eeqj.de/sneak/routewatch/issues/236).
- 2026-11-10: the IP lookup finds PTR names over DNS-over-HTTPS and caches them - 2026-11-10: the IP lookup finds PTR names over DNS-over-HTTPS and caches them
for the record's TTL, in memory and in `cache.db` (closes in memory and in `cache.db` for the record's TTL. A TTL is the time for which
a DNS record may be kept (closes
https://git.eeqj.de/sneak/routewatch/issues/238). https://git.eeqj.de/sneak/routewatch/issues/238).
- 2026-11-10: the tests no longer fail when a busy host stalls for several - 2026-11-10: the tests no longer fail when a busy host stalls for several
seconds. Most such waits are now 15 seconds. The test of the stream's idle seconds. Most waits for something that should happen are now 15 seconds. Tests
timeout runs on a fake clock (closes #240) that depend on a short idle timeout or a request limit run on a fake clock
(closes https://git.eeqj.de/sneak/routewatch/issues/240)
- 2026-11-10: the IP lookup checks the public VPN server lists of Mullvad, - 2026-11-10: the IP lookup checks the public VPN server lists of Mullvad,
NordVPN, Private Internet Access and IVPN, and X4BNet's community list. Each NordVPN, Private Internet Access and IVPN, and X4BNet's community list. Each
result has `flags.is_vpn` and `vpn_providers` (closes result has `flags.is_vpn` and `vpn_providers` (closes
+1 -1
View File
@@ -21,7 +21,7 @@ const (
downloadTimeout = time.Minute downloadTimeout = time.Minute
// maxDownloadSize is the size of the largest list file accepted, in // maxDownloadSize is the size of the largest list file accepted, in
// bytes. The largest list, FireHOL abusers, is about 2.5 MiB. // bytes. The largest list, iCloud Private Relay, is about 12 MiB.
maxDownloadSize = 200 << 20 maxDownloadSize = 200 << 20
) )
+10
View File
@@ -51,6 +51,16 @@ const (
// FormatIVPNJSON is IVPN's JSON server list. The hosts of its wireguard // FormatIVPNJSON is IVPN's JSON server list. The hosts of its wireguard
// and openvpn gateways have their addresses in host and v2ray. // and openvpn gateways have their addresses in host and v2ray.
FormatIVPNJSON FormatIVPNJSON
// FormatProtonVPN has one address or network per line, with comment lines
// that start with #. Unlike FormatLines, a file with no addresses or
// networks fails.
FormatProtonVPN
// FormatICloudPrivateRelayCSV is Apple's CSV file of iCloud Private Relay
// networks. Each line starts with a network, followed by its country,
// region and city.
FormatICloudPrivateRelayCSV
) )
// List is one public IP list. // List is one public IP list.
+135 -13
View File
@@ -70,6 +70,31 @@ const (
"config":{"api":{"ips":["192.0.2.73"]}}}` "config":{"api":{"ips":["192.0.2.73"]}}}`
) )
// protonVPNBody is a small file in the format of the ProtonVPN-IPs list.
const protonVPNBody = "#\n# protonvpn_ip_ranges.txt\n" +
"# https://github.com/tn3w/ProtonVPN-IPs\n#\n" +
"# The /24 block of every advertised IPv4.\n#\n" +
"192.0.2.0/24\n198.51.100.0/24\n"
// protonVPNEntryBody is a small file in the format of the ProtonVPN-IPs list
// of server addresses.
const protonVPNEntryBody = "#\n# protonvpn_entry_ips.txt\n" +
"# https://github.com/tn3w/ProtonVPN-IPs\n#\n" +
"# Entry IPs clients connect to, resolved from ProtonVPN hostnames.\n#\n" +
"192.0.2.80\n2001:db8::80\n"
// iCloudPrivateRelayAddr is an address on iCloudPrivateRelayBody.
const iCloudPrivateRelayAddr = "2001:db8:1::5"
// iCloudPrivateRelayBody is a small file in the format of Apple's iCloud
// Private Relay file.
const iCloudPrivateRelayBody = "192.0.2.0/31,GB,GB-EN,London,\n" +
"198.51.100.32/27,US,US-CA,Los Angeles,\n" +
"2001:db8:1::/64,US,US-AK,Anchorage,\n"
// emptyFileCase names the case of a wrong document that is an empty file.
const emptyFileCase = "an empty file"
// errorPage is a file that is no list, as a publisher's error page is. // errorPage is a file that is no list, as a publisher's error page is.
const errorPage = "<!DOCTYPE html>\n<html></html>\n" const errorPage = "<!DOCTYPE html>\n<html></html>\n"
@@ -230,6 +255,15 @@ func TestLookupFindsListedAddresses(t *testing.T) {
}, },
unlisted: []string{"192.0.2.12", "2001:db8::2"}, unlisted: []string{"192.0.2.12", "2001:db8::2"},
}, },
{
name: "iCloud Private Relay networks",
format: FormatICloudPrivateRelayCSV,
body: iCloudPrivateRelayBody,
listed: []string{
"192.0.2.1", "198.51.100.63", iCloudPrivateRelayAddr,
},
unlisted: []string{"192.0.2.2", "198.51.100.64", "2001:db8:2::1"},
},
}) })
} }
@@ -267,6 +301,20 @@ func TestLookupFindsVPNServers(t *testing.T) {
listed: []string{"192.0.2.70", "192.0.2.71", "192.0.2.72"}, listed: []string{"192.0.2.70", "192.0.2.71", "192.0.2.72"},
unlisted: []string{"192.0.2.73", "2001:db8::70"}, unlisted: []string{"192.0.2.73", "2001:db8::70"},
}, },
{
name: "ProtonVPN networks",
format: FormatProtonVPN,
body: protonVPNBody,
listed: []string{"192.0.2.0", "192.0.2.255", "198.51.100.7"},
unlisted: []string{"192.0.3.1", "2001:db8::90"},
},
{
name: "ProtonVPN server addresses",
format: FormatProtonVPN,
body: protonVPNEntryBody,
listed: []string{"192.0.2.80", "2001:db8::80"},
unlisted: []string{"192.0.2.81", "2001:db8::81"},
},
}) })
} }
@@ -458,12 +506,12 @@ func TestWrongVPNDocumentFailsToParse(t *testing.T) {
for _, f := range formats { for _, f := range formats {
bodies := map[string]string{ bodies := map[string]string{
"not JSON": errorPage, "not JSON": errorPage,
"other JSON": `{"status":"ok","data":{"count":1}}`, "other JSON": `{"status":"ok","data":{"count":1}}`,
"no addresses": f.empty, "no addresses": f.empty,
"bad address": f.badAddress, "bad address": f.badAddress,
"cut off": f.good[:len(f.good)/2], "cut off": f.good[:len(f.good)/2],
"an empty file": "", emptyFileCase: "",
} }
for name, body := range bodies { for name, body := range bodies {
@@ -479,6 +527,59 @@ func TestWrongVPNDocumentFailsToParse(t *testing.T) {
} }
} }
// TestWrongProtonVPNDocumentFailsToParse parses documents that are not a
// ProtonVPN-IPs list. Each fails, so none can replace a good copy.
func TestWrongProtonVPNDocumentFailsToParse(t *testing.T) {
t.Parallel()
bodies := map[string]string{
"error page": errorPage,
"JSON": `{"status":"ok"}`,
"comments only": "#\n# protonvpn_ip_ranges.txt\n#\n",
"bad network": "192.0.2.0/24\nexample\n",
emptyFileCase: "",
}
for name, body := range bodies {
t.Run(name, func(t *testing.T) {
t.Parallel()
addresses, err := parse(strings.NewReader(body), FormatProtonVPN)
if err == nil {
t.Errorf("parse returned %v and no error", addresses.Ranges())
}
})
}
}
// TestWrongICloudPrivateRelayDocumentFailsToParse parses documents that are
// not Apple's iCloud Private Relay file. Each fails, so none can replace a
// good copy.
func TestWrongICloudPrivateRelayDocumentFailsToParse(t *testing.T) {
t.Parallel()
bodies := map[string]string{
"error page": errorPage,
"JSON": `{"status":"ok","count":1}`,
"no other field": "192.0.2.0/24\n",
"bad network": "192.0.2.0/24,GB,GB-EN,London,\nexample,GB,GB-EN,London,\n",
"an address": "192.0.2.1,GB,GB-EN,London,\n",
emptyFileCase: "",
}
for name, body := range bodies {
t.Run(name, func(t *testing.T) {
t.Parallel()
addresses, err := parse(strings.NewReader(body),
FormatICloudPrivateRelayCSV)
if err == nil {
t.Errorf("parse returned %v and no error", addresses.Ranges())
}
})
}
}
// TestLookupMarksVPNLists looks up an address on a VPN list and on another // TestLookupMarksVPNLists looks up an address on a VPN list and on another
// list. Only the VPN list's listing says it is one. // list. Only the VPN list's listing says it is one.
func TestLookupMarksVPNLists(t *testing.T) { func TestLookupMarksVPNLists(t *testing.T) {
@@ -523,7 +624,10 @@ func TestPublicListsMarkTheVPNLists(t *testing.T) {
} }
} }
want := []string{"ivpn", "mullvad", "nordvpn", "pia", "x4bnet", "x4bnet_v6"} want := []string{
"ivpn", "mullvad", "nordvpn", "pia", "protonvpn", "protonvpn_entry",
"x4bnet", "x4bnet_v6",
}
if !slices.Equal(got, want) { if !slices.Equal(got, want) {
t.Errorf("VPN lists %v, want %v", got, want) t.Errorf("VPN lists %v, want %v", got, want)
} }
@@ -830,15 +934,33 @@ func TestStopCancelsADownloadInProgress(t *testing.T) {
func TestTorExitsListHoldsIPv6Exits(t *testing.T) { func TestTorExitsListHoldsIPv6Exits(t *testing.T) {
t.Parallel() t.Parallel()
checkPublicListHolds(t, TorExitsList, onionooBody, onionooIPv6ExitAddr)
}
// TestICloudPrivateRelayListHoldsItsNetworks downloads a file in Apple's
// format as the public iCloud Private Relay list, which is what an address's
// is_relay answer comes from.
func TestICloudPrivateRelayListHoldsItsNetworks(t *testing.T) {
t.Parallel()
checkPublicListHolds(t, ICloudPrivateRelayList, iCloudPrivateRelayBody,
iCloudPrivateRelayAddr)
}
// checkPublicListHolds downloads body as the public list called name and
// checks that addr has one listing, which names that list.
func checkPublicListHolds(t *testing.T, name, body, addr string) {
t.Helper()
index := slices.IndexFunc(Lists(), func(list List) bool { index := slices.IndexFunc(Lists(), func(list List) bool {
return list.Name == TorExitsList return list.Name == name
}) })
if index < 0 { if index < 0 {
t.Fatalf("no list named %s", TorExitsList) t.Fatalf("no list named %s", name)
} }
list := Lists()[index] list := Lists()[index]
_, list.URL = newPublisher(t, onionooBody) _, list.URL = newPublisher(t, body)
b := newTestIPLists(t.TempDir(), list) b := newTestIPLists(t.TempDir(), list)
err := b.download(t.Context(), list) err := b.download(t.Context(), list)
@@ -846,9 +968,9 @@ func TestTorExitsListHoldsIPv6Exits(t *testing.T) {
t.Fatalf("download: %v", err) t.Fatalf("download: %v", err)
} }
listings := lookup(t, b, onionooIPv6ExitAddr) listings := lookup(t, b, addr)
if len(listings) != 1 || listings[0].Name != TorExitsList { if len(listings) != 1 || listings[0].Name != name {
t.Errorf("listings %+v, want one for %s", listings, TorExitsList) t.Errorf("listings %+v, want one for %s", listings, name)
} }
} }
+39
View File
@@ -8,6 +8,13 @@ import (
// TorExitsList is the name of the list of Tor exit nodes. // TorExitsList is the name of the list of Tor exit nodes.
const TorExitsList = "tor_exits" const TorExitsList = "tor_exits"
// ICloudPrivateRelayList is the name of the list of iCloud Private Relay's
// networks.
const ICloudPrivateRelayList = "icloud_private_relay"
// protonVPNIPsURL is where the ProtonVPN-IPs mirror publishes its files.
const protonVPNIPsURL = "https://raw.githubusercontent.com/tn3w/ProtonVPN-IPs/master/"
// How often each list is downloaded. No list is downloaded more often than // How often each list is downloaded. No list is downloaded more often than
// its publisher updates it or allows. // its publisher updates it or allows.
const ( const (
@@ -27,6 +34,9 @@ const (
fireHOLAbusersInterval = 5 * time.Hour fireHOLAbusersInterval = 5 * time.Hour
fireHOLLevel3Interval = 3 * time.Hour fireHOLLevel3Interval = 3 * time.Hour
// Apple's iCloud Private Relay file is 12 MB, so once a day.
iCloudPrivateRelayInterval = 24 * time.Hour
// Spamhaus asks that DROP be downloaded at most once a day: // Spamhaus asks that DROP be downloaded at most once a day:
// https://www.spamhaus.org/blocklists/do-not-route-or-peer/ // https://www.spamhaus.org/blocklists/do-not-route-or-peer/
spamhausDROPInterval = 24 * time.Hour spamhausDROPInterval = 24 * time.Hour
@@ -81,6 +91,13 @@ func nonVPNLists() []List {
Interval: fireHOLLevel3Interval, Interval: fireHOLLevel3Interval,
Format: FormatLines, Format: FormatLines,
}, },
{
Name: ICloudPrivateRelayList,
Description: "In a network where iCloud Private Relay traffic leaves",
URL: "https://mask-api.icloud.com/egress-ip-ranges.csv",
Interval: iCloudPrivateRelayInterval,
Format: FormatICloudPrivateRelayCSV,
},
{ {
Name: "spamhaus_drop", Name: "spamhaus_drop",
Description: "In an IPv4 network that Spamhaus says spammers " + Description: "In an IPv4 network that Spamhaus says spammers " +
@@ -146,6 +163,28 @@ func vpnLists() []List {
Format: FormatPIAJSON, Format: FormatPIAJSON,
VPN: true, VPN: true,
}, },
{
// Proton's own server list needs a logged-in session, so this is
// the public ProtonVPN-IPs mirror of it. Proton publishes only
// the addresses clients connect to. The mirror's README says
// traffic leaves from others in the same /24, so this file of
// /24 networks is the one it says to use for detecting exits.
Name: "protonvpn",
Description: "In a /24 network that holds a server of the " +
"ProtonVPN service",
URL: protonVPNIPsURL + "protonvpn_ip_ranges.txt",
Interval: vpnListInterval,
Format: FormatProtonVPN,
VPN: true,
},
{
Name: "protonvpn_entry",
Description: "A server of the ProtonVPN service",
URL: protonVPNIPsURL + "protonvpn_entry_ips.txt",
Interval: vpnListInterval,
Format: FormatProtonVPN,
VPN: true,
},
{ {
Name: "x4bnet", Name: "x4bnet",
Description: "In an IPv4 network of a VPN service, according to " + Description: "In an IPv4 network of a VPN service, according to " +
+43
View File
@@ -25,6 +25,10 @@ const (
// dshieldFieldSeparator separates the fields of a DShield line. // dshieldFieldSeparator separates the fields of a DShield line.
dshieldFieldSeparator = "\t" dshieldFieldSeparator = "\t"
// iCloudPrivateRelayFieldSeparator separates the fields of a line of
// Apple's iCloud Private Relay file.
iCloudPrivateRelayFieldSeparator = ","
// spamhausMetadataType is the type of the line that ends a Spamhaus // spamhausMetadataType is the type of the line that ends a Spamhaus
// file. That line holds the file's metadata and no network. // file. That line holds the file's metadata and no network.
spamhausMetadataType = "metadata" spamhausMetadataType = "metadata"
@@ -58,6 +62,10 @@ func parse(r io.Reader, format Format) (*netipx.IPSet, error) {
return parsePIA(r) return parsePIA(r)
case FormatIVPNJSON: case FormatIVPNJSON:
return parseIVPN(r) return parseIVPN(r)
case FormatProtonVPN:
return parseLinesNotEmpty(r, addAddressOrNetwork)
case FormatICloudPrivateRelayCSV:
return parseLinesNotEmpty(r, addICloudPrivateRelayNetwork)
default: default:
return nil, errUnknownFormat return nil, errUnknownFormat
} }
@@ -329,6 +337,41 @@ func parseIVPN(r io.Reader) (*netipx.IPSet, error) {
return addressSet(texts) return addressSet(texts)
} }
// parseLinesNotEmpty is parseLines for a list that always holds addresses, so
// a file with none fails.
func parseLinesNotEmpty(
r io.Reader, addLine func(*netipx.IPSetBuilder, string) error,
) (*netipx.IPSet, error) {
addresses, err := parseLines(r, addLine)
if err != nil {
return nil, err
}
if len(addresses.Ranges()) == 0 {
return nil, errNoAddresses
}
return addresses, nil
}
// addICloudPrivateRelayNetwork adds the network at the start of a line of
// Apple's iCloud Private Relay file to builder.
func addICloudPrivateRelayNetwork(builder *netipx.IPSetBuilder, line string) error {
field, _, found := strings.Cut(line, iCloudPrivateRelayFieldSeparator)
if !found {
return errNoNetwork
}
network, err := netip.ParsePrefix(field)
if err != nil {
return err
}
builder.AddPrefix(network)
return nil
}
// readJSON reads r to its end and decodes it, as one JSON document, into v. // readJSON reads r to its end and decodes it, as one JSON document, into v.
func readJSON(r io.Reader, v any) error { func readJSON(r io.Reader, v any) error {
data, err := io.ReadAll(r) data, err := io.ReadAll(r)
+6
View File
@@ -801,6 +801,9 @@ type IPLookupFlags struct {
// IsVPN says that the address is on a list of VPN servers. // IsVPN says that the address is on a list of VPN servers.
IsVPN bool `json:"is_vpn"` IsVPN bool `json:"is_vpn"`
// IsRelay says that the address is on the iCloud Private Relay list.
IsRelay bool `json:"is_relay"`
// IsBogon says that the address is reserved, as package bogon tells. // IsBogon says that the address is reserved, as package bogon tells.
IsBogon bool `json:"is_bogon"` IsBogon bool `json:"is_bogon"`
} }
@@ -1033,6 +1036,9 @@ func ipLookupFlags(ip string, listings []iplist.Listing) IPLookupFlags {
IsVPN: slices.ContainsFunc(listings, func(l iplist.Listing) bool { IsVPN: slices.ContainsFunc(listings, func(l iplist.Listing) bool {
return l.VPN return l.VPN
}), }),
IsRelay: slices.ContainsFunc(listings, func(l iplist.Listing) bool {
return l.Name == iplist.ICloudPrivateRelayList
}),
IsBogon: isBogon(ip), IsBogon: isBogon(ip),
} }
} }
+28 -16
View File
@@ -1506,6 +1506,10 @@ const listedUnroutedAddr = "203.0.113.1"
// torExitAddr is the address that fakeIPLists lists on the Tor exit list. // torExitAddr is the address that fakeIPLists lists on the Tor exit list.
const torExitAddr = "203.0.113.2" const torExitAddr = "203.0.113.2"
// relayAddr is the address that fakeIPLists lists on the iCloud Private Relay
// list.
const relayAddr = "203.0.113.5"
// The addresses that fakeIPLists lists on one and on two VPN lists, and the // The addresses that fakeIPLists lists on one and on two VPN lists, and the
// names of those lists. // names of those lists.
const ( const (
@@ -1515,10 +1519,11 @@ const (
otherTestVPN = "other_test_vpn" otherTestVPN = "other_test_vpn"
) )
// fakeIPLists has four lists. The test list holds addrInTestPrefixV4, // fakeIPLists has five lists. The test list holds addrInTestPrefixV4,
// listedUnroutedAddr and twoVPNListsAddr. The Tor exit list holds torExitAddr. // listedUnroutedAddr and twoVPNListsAddr. The Tor exit list holds torExitAddr.
// The test VPN list holds oneVPNListAddr and twoVPNListsAddr, and the other // The iCloud Private Relay list holds relayAddr. The test VPN list holds
// test VPN list holds twoVPNListsAddr. // oneVPNListAddr and twoVPNListsAddr, and the other test VPN list holds
// twoVPNListsAddr.
type fakeIPLists struct{} type fakeIPLists struct{}
func testListing() iplist.Listing { func testListing() iplist.Listing {
@@ -1539,6 +1544,10 @@ func (fakeIPLists) Lookup(addr netip.Addr) []iplist.Listing {
return []iplist.Listing{{Name: iplist.TorExitsList}} return []iplist.Listing{{Name: iplist.TorExitsList}}
} }
if addr == netip.MustParseAddr(relayAddr) {
return []iplist.Listing{{Name: iplist.ICloudPrivateRelayList}}
}
if addr == netip.MustParseAddr(oneVPNListAddr) { if addr == netip.MustParseAddr(oneVPNListAddr) {
return []iplist.Listing{{Name: testVPNList, VPN: true}} return []iplist.Listing{{Name: testVPNList, VPN: true}}
} }
@@ -1571,20 +1580,22 @@ func TestIPLookupFlags(t *testing.T) {
name string name string
addr string addr string
tor bool tor bool
relay bool
bogon bool bogon bool
}{ }{
{"on the Tor exit list", torExitAddr, true, true}, {"on the Tor exit list", torExitAddr, true, false, true},
{"on another list", listedUnroutedAddr, false, true}, {"on the iCloud Private Relay list", relayAddr, false, true, true},
{"routed IPv4", addrInTestPrefixV4, false, true}, {"on another list", listedUnroutedAddr, false, false, true},
{"routed IPv6", addrInTestPrefixV6, false, true}, {"routed IPv4", addrInTestPrefixV4, false, false, true},
{"private IPv4", "10.0.0.1", false, true}, {"routed IPv6", addrInTestPrefixV6, false, false, true},
{"loopback IPv4", "127.0.0.1", false, true}, {"private IPv4", "10.0.0.1", false, false, true},
{"public IPv4", "1.1.1.1", false, false}, {"loopback IPv4", "127.0.0.1", false, false, true},
{"private IPv6", "fd00::1", false, true}, {"public IPv4", "1.1.1.1", false, false, false},
{"loopback IPv6", "::1", false, true}, {"private IPv6", "fd00::1", false, false, true},
{"public IPv6", "2606:4700:4700::1111", false, false}, {"loopback IPv6", "::1", false, false, true},
{"public IPv4 in IPv6 form", "::ffff:1.1.1.1", false, false}, {"public IPv6", "2606:4700:4700::1111", false, false, false},
{"private IPv4 in IPv6 form", "::ffff:10.0.0.1", false, true}, {"public IPv4 in IPv6 form", "::ffff:1.1.1.1", false, false, false},
{"private IPv4 in IPv6 form", "::ffff:10.0.0.1", false, false, true},
} }
for _, tt := range tests { for _, tt := range tests {
@@ -1606,7 +1617,8 @@ func TestIPLookupFlags(t *testing.T) {
} }
want := map[string]bool{ want := map[string]bool{
"is_tor": tt.tor, "is_vpn": false, "is_bogon": tt.bogon, "is_tor": tt.tor, "is_vpn": false, "is_relay": tt.relay,
"is_bogon": tt.bogon,
} }
if !maps.Equal(got, want) { if !maps.Equal(got, want) {
t.Errorf("%s: flags %v, want %v", path, got, want) t.Errorf("%s: flags %v, want %v", path, got, want)