Author SHA1 Message Date
clawbot 01b18d00ff Create the image's app user without the system-user flag (closes #248)
check / check (push) Successful in 4m49s
`useradd -r` with UID 1000 made the image build warn, since 1000 is above the system range. The user keeps UID 1000, so the data directory's owner does not change.

Model: opus-5-5
2026-11-10 15:04:21 +00:00
clawbot cca402c73d Add best-effort geolocation to the IP lookup from geojs and DB-IP City Lite (closes #239)
check / check (push) Successful in 1m41s
Every IP lookup result gains `geo`. One request to geojs carries all uncached addresses of a lookup, within 1 second. On a timeout, an error or an answer with no location, the lookup reads DB-IP City Lite, downloaded monthly into the state directory. Geojs answers stay in memory, at most 250,000, and in `cache.db` for 7 days. The home page footer credits DB-IP.

Judgement call: a reserved address goes to neither source, and its `geo` is null.
Judgement call: `updated` for a DB-IP answer is when the copy was downloaded.
Deviation: the cache is SQLite, by sneak's words, not the JSON file the standing rule asks for.

Model: opus-5-5
2026-11-10 15:48:12 +01:00
clawbot 2c3f6f71dc Add the iCloud Private Relay list to the IP lookup, with is_relay (closes #237)
check / check (push) Failing after 17m2s
Adds Apple's iCloud Private Relay egress list as `icloud_private_relay`, downloaded once a day, and `flags.is_relay`, true when an address is on it. Loaded, the list adds about 0.3 MiB to the Go heap; parsing it allocates about 94 MiB, freed by the next garbage collection.

Judgement call: a line with no comma fails the parse, so a plain network list served at Apple's URL cannot replace a good copy.

Model: opus-5-5
2026-11-10 15:13:55 +01:00
clawbot b2553f044d Add the ProtonVPN server networks and addresses to the IP lookup (closes #236)
check / check (push) Successful in 1m27s
Proton's own server list answers only a logged-in session, so the lists come from the public mirror https://github.com/tn3w/ProtonVPN-IPs, downloaded once a day: `protonvpn`, its /24 networks, and `protonvpn_entry`, its servers' IPv4 and IPv6 addresses. Both feed `flags.is_vpn` and `vpn_providers`.

Judgement call: the /24 networks are listed as well as the exact addresses, because coverage matters more than attribution, so neighbouring addresses in those networks count too.
Risk: the mirror is one person's repository; if it stops updating, downloads keep succeeding with old data.

Model: opus-5-5
2026-11-10 14:14:06 +01:00
15 changed files with 440 additions and 51 deletions

No files matched your search

+1 -1
View File
@@ -105,7 +105,7 @@ RUN apt-get update && apt-get install -y --no-install-recommends \
ARG CACHEBUST
# Create non-root user
RUN useradd -r -u 1000 -m routewatch
RUN useradd -u 1000 -m routewatch
RUN mkdir -p /var/lib/berlin.sneak.app.routewatch && chown routewatch:routewatch /var/lib/berlin.sneak.app.routewatch
+29 -6
View File
@@ -287,6 +287,7 @@ address. Every entry has the same fields:
- `is_tor`: true when the address is on the `tor_exits` list
- `is_vpn`: true when the address is on any list of VPN servers (see IP
lists)
- `is_relay`: true when the address is on the `icloud_private_relay` list
- `is_bogon`: true when the address is reserved, so that no network should
announce it on the internet
- `vpn_providers`: the names of the lists of VPN servers the address is on, or
@@ -422,6 +423,9 @@ The lists:
last 30 days. From
[FireHOL](https://iplists.firehol.org/?ipset=firehol_level3), downloaded every
3 hours.
- `icloud_private_relay`: the networks where iCloud Private Relay traffic
leaves. From [Apple](https://mask-api.icloud.com/egress-ip-ranges.csv),
downloaded once a day.
- `ivpn`: VPN servers run by IVPN. From
[IVPN](https://api.ivpn.net/v5/servers.json), downloaded once a day.
- `mullvad`: VPN servers run by Mullvad. From
@@ -432,6 +436,14 @@ The lists:
- `pia`: VPN servers run by Private Internet Access. From
[Private Internet Access](https://serverlist.piaservers.net/vpninfo/servers/v6),
downloaded once a day.
- `protonvpn`: the /24 networks that hold ProtonVPN servers. The whole /24 is
listed because the ProtonVPN-IPs README says traffic leaves from unpublished
addresses in it. From the public
[ProtonVPN-IPs](https://github.com/tn3w/ProtonVPN-IPs) mirror of Proton's
server list, downloaded once a day.
- `protonvpn_entry`: the IPv4 and IPv6 addresses of ProtonVPN servers. From the
same [ProtonVPN-IPs](https://github.com/tn3w/ProtonVPN-IPs) mirror, downloaded
once a day.
- `spamhaus_drop` and `spamhaus_dropv6`: IPv4 and IPv6 networks hijacked or
leased by spammers or cyber criminals, according to Spamhaus. From
[Spamhaus DROP](https://www.spamhaus.org/blocklists/do-not-route-or-peer/),
@@ -510,9 +522,13 @@ sources:
- [geojs](https://www.geojs.io/) is asked first. One request carries every
address of the lookup, and it has 1 second to answer.
- [DB-IP City Lite](https://db-ip.com/db/download/ip-to-city-lite) answers when
geojs fails or does not answer in time. It is a database file that RouteWatch
geojs has no answer for the address. It is a database file that RouteWatch
reads itself.
geojs has no answer when its request fails or runs out of time. An answer with
no country, city or coordinates also counts as no answer, and RouteWatch does
not cache it.
The `geo` object of a result has these fields:
- `country_code`: the two-letter country code
@@ -534,13 +550,13 @@ goes to neither source, so its `geo` is null.
What goes outside: geojs receives each address that the cache does not hold. For
`/ip/` and the home page, that address is the visitor's own. A lookup answered
from the cache or from the database sends nothing. DB-IP receives only the
monthly request for its file.
from the cache sends nothing. DB-IP receives only the monthly request for its
file.
How RouteWatch keeps the answers of geojs:
- It holds at most 250,000 addresses in memory, and drops the least recently
used first. That takes about 75 MB of the Go heap.
used first. That takes about 100 MB of the Go heap.
- It writes new answers every 5 seconds to `cache.db` in the state directory.
- It loads the newest 250,000 answers from `cache.db` at startup.
- An address not in memory is read from `cache.db` before geojs is asked.
@@ -561,8 +577,8 @@ How RouteWatch keeps the database:
The kernel loads the parts that are read, and can drop them again when memory
is short.
DB-IP's licence requires an attribution, so the home page's footer says "IP
Geolocation by DB-IP" with a link to [db-ip.com](https://db-ip.com).
The home page's footer says "IP Geolocation by DB-IP" with a link to
[db-ip.com](https://db-ip.com).
## Configuration
@@ -664,6 +680,13 @@ The ceilings add up to 3.7 GiB:
limit caps the SQLite library.
- About 0.2 GiB for everything else in the runtime.
The `icloud_private_relay` list has about 285,000 networks. Neighbouring
networks merge, which leaves about 6,900 address ranges. Loaded, the list adds
about 0.3 MiB to the Go heap and under 6 MiB to the process's resident memory.
Parsing the file allocates about 94 MiB in about 0.2 seconds. The next garbage
collection frees all of it but the 0.3 MiB above. Parsing happens at startup and
whenever a download brings a new version.
The SQLite library gets its memory from glibc, the C library. glibc keeps freed
memory in pools called arenas, and hands it back to the kernel slowly. It
creates up to eight arenas per processor core. So on a host with many cores the
+10
View File
@@ -53,10 +53,20 @@ problem the run finds.
# Completed Steps
- 2026-11-10: the image creates the `routewatch` user as a regular user with UID
1000, so the build no longer warns about the system UID range (closes
https://git.eeqj.de/sneak/routewatch/issues/248).
- 2026-11-10: each result of the IP lookup has `geo`, where the address is. It
comes from geojs, with DB-IP's City Lite database as the fallback. The geojs
answers stay in memory and in `cache.db` for 7 days (closes
https://git.eeqj.de/sneak/routewatch/issues/239).
- 2026-11-10: the IP lookup checks the `icloud_private_relay` list of Apple's
iCloud Private Relay networks, downloaded once a day. Each result has
`flags.is_relay` (closes https://git.eeqj.de/sneak/routewatch/issues/237).
- 2026-11-10: the IP lookup checks the `protonvpn` list of ProtonVPN's /24
networks and the `protonvpn_entry` list of its servers' IPv4 and IPv6
addresses, from a public mirror of Proton's server list (closes
https://git.eeqj.de/sneak/routewatch/issues/236).
- 2026-11-10: the IP lookup finds PTR names over DNS-over-HTTPS and caches them
for the record's TTL, in memory and in `cache.db` (closes
https://git.eeqj.de/sneak/routewatch/issues/238).
+12 -2
View File
@@ -107,7 +107,8 @@ func nilIfEmpty(text string) *string {
}
// fromDatabase returns where the copy of the database places addr. It returns
// nil when there is no copy yet or the copy has no record for addr.
// nil when there is no copy yet or the copy has no record for addr. It also
// returns nil, and logs the error, when reading the copy fails.
func (l *Locator) fromDatabase(addr netip.Addr) *Location {
l.databaseMu.RLock()
defer l.databaseMu.RUnlock()
@@ -117,13 +118,22 @@ func (l *Locator) fromDatabase(addr netip.Addr) *Location {
}
result := l.database.Lookup(addr)
err := result.Err()
if err != nil {
l.logger.Error("Failed to look up an address in the database",
"address", addr, "error", err)
return nil
}
if !result.Found() {
return nil
}
var record dbipRecord
err := result.Decode(&record)
err = result.Decode(&record)
if err != nil {
l.logger.Error("Failed to read a database record",
"address", addr, "error", err)
+12 -2
View File
@@ -51,7 +51,8 @@ type geojsAnswer struct {
}
// askGeoJS asks geojs about addresses in one request, and returns its answers
// by address. An address geojs left out has no entry.
// by address. An address geojs left out, or gave no country, city or
// coordinates for, has no entry.
func (l *Locator) askGeoJS(
ctx context.Context, addresses []string,
) (map[string]*Location, error) {
@@ -97,12 +98,21 @@ func (l *Locator) askGeoJS(
continue
}
answers[addr.Unmap().String()] = answer.location(fetched)
location := answer.location(fetched)
if hasLocation(location) {
answers[plainAddress(addr)] = location
}
}
return answers, nil
}
// hasLocation says whether found has a country, a city or a coordinate.
func hasLocation(found *Location) bool {
return found.CountryCode != nil || found.Country != nil ||
found.City != nil || found.Latitude != nil || found.Longitude != nil
}
// location returns the answer as a Location fetched at the time given.
func (a geojsAnswer) location(fetched time.Time) *Location {
return &Location{
+12 -4
View File
@@ -1,6 +1,7 @@
// Package geo finds where IP addresses are. It asks geojs.io, and keeps those
// answers in memory and in cache.db. When geojs.io does not answer in time, it
// reads a copy of DB-IP's City Lite database, which it downloads once a month.
// answers in memory and in cache.db. When geojs.io gives no location for an
// address, it reads a copy of DB-IP's City Lite database, which it downloads
// once a month.
package geo
import (
@@ -141,7 +142,7 @@ func (l *Locator) Lookup(ctx context.Context, addrs []netip.Addr) []*Location {
var unanswered []string
for i, addr := range addrs {
address := addr.Unmap().String()
address := plainAddress(addr)
locations[i] = l.cached(ctx, address)
if locations[i] == nil {
@@ -163,7 +164,7 @@ func (l *Locator) Lookup(ctx context.Context, addrs []netip.Addr) []*Location {
continue
}
address := addr.Unmap().String()
address := plainAddress(addr)
answer, ok := answers[address]
if ok {
@@ -179,6 +180,13 @@ func (l *Locator) Lookup(ctx context.Context, addrs []netip.Addr) []*Location {
return locations
}
// plainAddress returns addr as the requests to geojs and the cache name it: an
// IPv4 address mapped into IPv6 as plain IPv4, and with no zone. A client
// chooses the zone, so it must not reach the request.
func plainAddress(addr netip.Addr) string {
return addr.Unmap().WithZone("").String()
}
// Start loads the cached answers that have not expired and opens the copy of
// the database, if there is one. It then starts the goroutines that write new
// answers, delete expired ones and download the database. They run until Stop:
+100 -4
View File
@@ -53,9 +53,18 @@ const (
"country": "Germany", "country_code": "DE", "ip": "` + testAddress + `",
"latitude": "52.6171", "longitude": "13.1207",
"region": "Brandenburg", "timezone": "Europe/Berlin"},
{"ip": "` + otherAddress + `", "latitude": "nil", "longitude": "nil"}
{"country": "Germany", "ip": "` + otherAddress + `",
"latitude": "nil", "longitude": "nil"}
]`
geojsOneAddress = `[{"ip": "` + testAddress + `", "city": "` + testCity + `"}]`
// geojsNoLocation has no country, city or coordinates, as geojs sends for
// some addresses. Its other parts do not count as a location.
geojsNoLocation = `[{"accuracy": 1000, "area_code": "0", "asn": 64496,
"continent_code": "EU", "ip": "` + testAddress + `",
"latitude": "nil", "longitude": "nil",
"organization": "AS64496 Example", "organization_name": "Example",
"region": "Brandenburg", "timezone": "Europe/Berlin"}]`
)
// fixedTime returns the time a test's clock starts at.
@@ -315,9 +324,8 @@ func TestLookupAsksGeoJSOnceForEveryAddress(t *testing.T) {
}
}
// TestLookupGivesUnknownPartsAsNil checks an answer whose coordinates are the
// string "nil" and which has no other part: the parts are nil, and the answer
// is still an answer.
// TestLookupGivesUnknownPartsAsNil checks an answer with only a country name,
// whose coordinates are the string "nil": the other parts are nil.
func TestLookupGivesUnknownPartsAsNil(t *testing.T) {
t.Parallel()
@@ -368,6 +376,7 @@ func TestLookupFallsBackToTheDatabase(t *testing.T) {
{name: "error status", status: http.StatusBadGateway},
{name: "unreadable answer", body: "not JSON"},
{name: "address left out", body: "[]"},
{name: "answer with no location", body: geojsNoLocation},
}
for _, tt := range tests {
@@ -411,6 +420,93 @@ func TestLookupFallsBackToTheDatabase(t *testing.T) {
}
}
// TestAnyCountryCityOrCoordinateIsAnAnswer checks that each of those parts,
// alone, makes an answer from geojs that is returned and kept.
func TestAnyCountryCityOrCoordinateIsAnAnswer(t *testing.T) {
t.Parallel()
parts := []string{
`"country": "Germany"`,
`"country_code": "DE"`,
`"city": "` + testCity + `"`,
`"latitude": "52.5"`,
`"longitude": "13.5"`,
}
for _, part := range parts {
t.Run(part, func(t *testing.T) {
t.Parallel()
fake := newFakeGeoJS(t, `[{"ip": "`+testAddress+`", `+part+`}]`)
dir := t.TempDir()
locator := newTestLocator(t, fake, openCacheDB(t, dir), dir)
got := lookupOne(t, locator, testAddress)
if got == nil || got.Source != sourceGeoJS {
t.Fatalf("location %+v, want the answer from geojs", got)
}
if !locator.entries.Contains(testAddress) {
t.Error("the answer is not in memory")
}
})
}
}
// TestLookupSendsNoZoneToGeoJS looks up an address with a zone, which is text
// a client chooses. geojs receives only the plain address, and the answer is
// kept under it.
func TestLookupSendsNoZoneToGeoJS(t *testing.T) {
t.Parallel()
fake := newFakeGeoJS(t, geojsTwoAddresses)
dir := t.TempDir()
locator := newTestLocator(t, fake, openCacheDB(t, dir), dir)
got := lookupOne(t, locator, otherAddress+"%x&callback=evil")
requests := fake.requestsSeen()
if len(requests) != 1 || requests[0] != otherAddress {
t.Errorf("requests %v, want one naming only %s", requests, otherAddress)
}
if got == nil || got.Source != sourceGeoJS {
t.Errorf("location %+v, want the answer from geojs", got)
}
if !locator.entries.Contains(otherAddress) {
t.Error("the answer is not kept under the plain address")
}
}
// TestFailedDatabaseLookupIsLogged closes the copy under the locator, so that
// the lookup in it fails. The location is nil and the failure is logged.
func TestFailedDatabaseLookupIsLogged(t *testing.T) {
t.Parallel()
fake := newFakeGeoJS(t, "[]")
dir := t.TempDir()
locator := newTestLocator(t, fake, openCacheDB(t, dir), dir)
giveDatabase(t, locator)
err := locator.database.Close()
if err != nil {
t.Fatalf("closing the copy: %v", err)
}
var logged bytes.Buffer
locator.logger = slog.New(slog.NewTextHandler(&logged, nil))
if got := locator.fromDatabase(parseAddresses(t, testAddress)[0]); got != nil {
t.Errorf("location %+v from a failed lookup, want nil", got)
}
if !strings.Contains(logged.String(), "address="+testAddress) {
t.Errorf("the failure was not logged: %q", logged.String())
}
}
// TestLookupIsNilWhenNeitherSourceAnswers makes geojs fail with no database
// copy, and then with a copy that does not cover the address.
func TestLookupIsNilWhenNeitherSourceAnswers(t *testing.T) {
+1 -1
View File
@@ -21,7 +21,7 @@ const (
downloadTimeout = time.Minute
// maxDownloadSize is the size of the largest list file accepted, in
// bytes. The largest list, FireHOL abusers, is about 2.5 MiB.
// bytes. The largest list, iCloud Private Relay, is about 12 MiB.
maxDownloadSize = 200 << 20
)
+10
View File
@@ -51,6 +51,16 @@ const (
// FormatIVPNJSON is IVPN's JSON server list. The hosts of its wireguard
// and openvpn gateways have their addresses in host and v2ray.
FormatIVPNJSON
// FormatProtonVPN has one address or network per line, with comment lines
// that start with #. Unlike FormatLines, a file with no addresses or
// networks fails.
FormatProtonVPN
// FormatICloudPrivateRelayCSV is Apple's CSV file of iCloud Private Relay
// networks. Each line starts with a network, followed by its country,
// region and city.
FormatICloudPrivateRelayCSV
)
// List is one public IP list.
+135 -13
View File
@@ -70,6 +70,31 @@ const (
"config":{"api":{"ips":["192.0.2.73"]}}}`
)
// protonVPNBody is a small file in the format of the ProtonVPN-IPs list.
const protonVPNBody = "#\n# protonvpn_ip_ranges.txt\n" +
"# https://github.com/tn3w/ProtonVPN-IPs\n#\n" +
"# The /24 block of every advertised IPv4.\n#\n" +
"192.0.2.0/24\n198.51.100.0/24\n"
// protonVPNEntryBody is a small file in the format of the ProtonVPN-IPs list
// of server addresses.
const protonVPNEntryBody = "#\n# protonvpn_entry_ips.txt\n" +
"# https://github.com/tn3w/ProtonVPN-IPs\n#\n" +
"# Entry IPs clients connect to, resolved from ProtonVPN hostnames.\n#\n" +
"192.0.2.80\n2001:db8::80\n"
// iCloudPrivateRelayAddr is an address on iCloudPrivateRelayBody.
const iCloudPrivateRelayAddr = "2001:db8:1::5"
// iCloudPrivateRelayBody is a small file in the format of Apple's iCloud
// Private Relay file.
const iCloudPrivateRelayBody = "192.0.2.0/31,GB,GB-EN,London,\n" +
"198.51.100.32/27,US,US-CA,Los Angeles,\n" +
"2001:db8:1::/64,US,US-AK,Anchorage,\n"
// emptyFileCase names the case of a wrong document that is an empty file.
const emptyFileCase = "an empty file"
// errorPage is a file that is no list, as a publisher's error page is.
const errorPage = "<!DOCTYPE html>\n<html></html>\n"
@@ -230,6 +255,15 @@ func TestLookupFindsListedAddresses(t *testing.T) {
},
unlisted: []string{"192.0.2.12", "2001:db8::2"},
},
{
name: "iCloud Private Relay networks",
format: FormatICloudPrivateRelayCSV,
body: iCloudPrivateRelayBody,
listed: []string{
"192.0.2.1", "198.51.100.63", iCloudPrivateRelayAddr,
},
unlisted: []string{"192.0.2.2", "198.51.100.64", "2001:db8:2::1"},
},
})
}
@@ -267,6 +301,20 @@ func TestLookupFindsVPNServers(t *testing.T) {
listed: []string{"192.0.2.70", "192.0.2.71", "192.0.2.72"},
unlisted: []string{"192.0.2.73", "2001:db8::70"},
},
{
name: "ProtonVPN networks",
format: FormatProtonVPN,
body: protonVPNBody,
listed: []string{"192.0.2.0", "192.0.2.255", "198.51.100.7"},
unlisted: []string{"192.0.3.1", "2001:db8::90"},
},
{
name: "ProtonVPN server addresses",
format: FormatProtonVPN,
body: protonVPNEntryBody,
listed: []string{"192.0.2.80", "2001:db8::80"},
unlisted: []string{"192.0.2.81", "2001:db8::81"},
},
})
}
@@ -458,12 +506,12 @@ func TestWrongVPNDocumentFailsToParse(t *testing.T) {
for _, f := range formats {
bodies := map[string]string{
"not JSON": errorPage,
"other JSON": `{"status":"ok","data":{"count":1}}`,
"no addresses": f.empty,
"bad address": f.badAddress,
"cut off": f.good[:len(f.good)/2],
"an empty file": "",
"not JSON": errorPage,
"other JSON": `{"status":"ok","data":{"count":1}}`,
"no addresses": f.empty,
"bad address": f.badAddress,
"cut off": f.good[:len(f.good)/2],
emptyFileCase: "",
}
for name, body := range bodies {
@@ -479,6 +527,59 @@ func TestWrongVPNDocumentFailsToParse(t *testing.T) {
}
}
// TestWrongProtonVPNDocumentFailsToParse parses documents that are not a
// ProtonVPN-IPs list. Each fails, so none can replace a good copy.
func TestWrongProtonVPNDocumentFailsToParse(t *testing.T) {
t.Parallel()
bodies := map[string]string{
"error page": errorPage,
"JSON": `{"status":"ok"}`,
"comments only": "#\n# protonvpn_ip_ranges.txt\n#\n",
"bad network": "192.0.2.0/24\nexample\n",
emptyFileCase: "",
}
for name, body := range bodies {
t.Run(name, func(t *testing.T) {
t.Parallel()
addresses, err := parse(strings.NewReader(body), FormatProtonVPN)
if err == nil {
t.Errorf("parse returned %v and no error", addresses.Ranges())
}
})
}
}
// TestWrongICloudPrivateRelayDocumentFailsToParse parses documents that are
// not Apple's iCloud Private Relay file. Each fails, so none can replace a
// good copy.
func TestWrongICloudPrivateRelayDocumentFailsToParse(t *testing.T) {
t.Parallel()
bodies := map[string]string{
"error page": errorPage,
"JSON": `{"status":"ok","count":1}`,
"no other field": "192.0.2.0/24\n",
"bad network": "192.0.2.0/24,GB,GB-EN,London,\nexample,GB,GB-EN,London,\n",
"an address": "192.0.2.1,GB,GB-EN,London,\n",
emptyFileCase: "",
}
for name, body := range bodies {
t.Run(name, func(t *testing.T) {
t.Parallel()
addresses, err := parse(strings.NewReader(body),
FormatICloudPrivateRelayCSV)
if err == nil {
t.Errorf("parse returned %v and no error", addresses.Ranges())
}
})
}
}
// TestLookupMarksVPNLists looks up an address on a VPN list and on another
// list. Only the VPN list's listing says it is one.
func TestLookupMarksVPNLists(t *testing.T) {
@@ -523,7 +624,10 @@ func TestPublicListsMarkTheVPNLists(t *testing.T) {
}
}
want := []string{"ivpn", "mullvad", "nordvpn", "pia", "x4bnet", "x4bnet_v6"}
want := []string{
"ivpn", "mullvad", "nordvpn", "pia", "protonvpn", "protonvpn_entry",
"x4bnet", "x4bnet_v6",
}
if !slices.Equal(got, want) {
t.Errorf("VPN lists %v, want %v", got, want)
}
@@ -830,15 +934,33 @@ func TestStopCancelsADownloadInProgress(t *testing.T) {
func TestTorExitsListHoldsIPv6Exits(t *testing.T) {
t.Parallel()
checkPublicListHolds(t, TorExitsList, onionooBody, onionooIPv6ExitAddr)
}
// TestICloudPrivateRelayListHoldsItsNetworks downloads a file in Apple's
// format as the public iCloud Private Relay list, which is what an address's
// is_relay answer comes from.
func TestICloudPrivateRelayListHoldsItsNetworks(t *testing.T) {
t.Parallel()
checkPublicListHolds(t, ICloudPrivateRelayList, iCloudPrivateRelayBody,
iCloudPrivateRelayAddr)
}
// checkPublicListHolds downloads body as the public list called name and
// checks that addr has one listing, which names that list.
func checkPublicListHolds(t *testing.T, name, body, addr string) {
t.Helper()
index := slices.IndexFunc(Lists(), func(list List) bool {
return list.Name == TorExitsList
return list.Name == name
})
if index < 0 {
t.Fatalf("no list named %s", TorExitsList)
t.Fatalf("no list named %s", name)
}
list := Lists()[index]
_, list.URL = newPublisher(t, onionooBody)
_, list.URL = newPublisher(t, body)
b := newTestIPLists(t.TempDir(), list)
err := b.download(t.Context(), list)
@@ -846,9 +968,9 @@ func TestTorExitsListHoldsIPv6Exits(t *testing.T) {
t.Fatalf("download: %v", err)
}
listings := lookup(t, b, onionooIPv6ExitAddr)
if len(listings) != 1 || listings[0].Name != TorExitsList {
t.Errorf("listings %+v, want one for %s", listings, TorExitsList)
listings := lookup(t, b, addr)
if len(listings) != 1 || listings[0].Name != name {
t.Errorf("listings %+v, want one for %s", listings, name)
}
}
+39
View File
@@ -8,6 +8,13 @@ import (
// TorExitsList is the name of the list of Tor exit nodes.
const TorExitsList = "tor_exits"
// ICloudPrivateRelayList is the name of the list of iCloud Private Relay's
// networks.
const ICloudPrivateRelayList = "icloud_private_relay"
// protonVPNIPsURL is where the ProtonVPN-IPs mirror publishes its files.
const protonVPNIPsURL = "https://raw.githubusercontent.com/tn3w/ProtonVPN-IPs/master/"
// How often each list is downloaded. No list is downloaded more often than
// its publisher updates it or allows.
const (
@@ -27,6 +34,9 @@ const (
fireHOLAbusersInterval = 5 * time.Hour
fireHOLLevel3Interval = 3 * time.Hour
// Apple's iCloud Private Relay file is 12 MB, so once a day.
iCloudPrivateRelayInterval = 24 * time.Hour
// Spamhaus asks that DROP be downloaded at most once a day:
// https://www.spamhaus.org/blocklists/do-not-route-or-peer/
spamhausDROPInterval = 24 * time.Hour
@@ -81,6 +91,13 @@ func nonVPNLists() []List {
Interval: fireHOLLevel3Interval,
Format: FormatLines,
},
{
Name: ICloudPrivateRelayList,
Description: "In a network where iCloud Private Relay traffic leaves",
URL: "https://mask-api.icloud.com/egress-ip-ranges.csv",
Interval: iCloudPrivateRelayInterval,
Format: FormatICloudPrivateRelayCSV,
},
{
Name: "spamhaus_drop",
Description: "In an IPv4 network that Spamhaus says spammers " +
@@ -146,6 +163,28 @@ func vpnLists() []List {
Format: FormatPIAJSON,
VPN: true,
},
{
// Proton's own server list needs a logged-in session, so this is
// the public ProtonVPN-IPs mirror of it. Proton publishes only
// the addresses clients connect to. The mirror's README says
// traffic leaves from others in the same /24, so this file of
// /24 networks is the one it says to use for detecting exits.
Name: "protonvpn",
Description: "In a /24 network that holds a server of the " +
"ProtonVPN service",
URL: protonVPNIPsURL + "protonvpn_ip_ranges.txt",
Interval: vpnListInterval,
Format: FormatProtonVPN,
VPN: true,
},
{
Name: "protonvpn_entry",
Description: "A server of the ProtonVPN service",
URL: protonVPNIPsURL + "protonvpn_entry_ips.txt",
Interval: vpnListInterval,
Format: FormatProtonVPN,
VPN: true,
},
{
Name: "x4bnet",
Description: "In an IPv4 network of a VPN service, according to " +
+43
View File
@@ -25,6 +25,10 @@ const (
// dshieldFieldSeparator separates the fields of a DShield line.
dshieldFieldSeparator = "\t"
// iCloudPrivateRelayFieldSeparator separates the fields of a line of
// Apple's iCloud Private Relay file.
iCloudPrivateRelayFieldSeparator = ","
// spamhausMetadataType is the type of the line that ends a Spamhaus
// file. That line holds the file's metadata and no network.
spamhausMetadataType = "metadata"
@@ -58,6 +62,10 @@ func parse(r io.Reader, format Format) (*netipx.IPSet, error) {
return parsePIA(r)
case FormatIVPNJSON:
return parseIVPN(r)
case FormatProtonVPN:
return parseLinesNotEmpty(r, addAddressOrNetwork)
case FormatICloudPrivateRelayCSV:
return parseLinesNotEmpty(r, addICloudPrivateRelayNetwork)
default:
return nil, errUnknownFormat
}
@@ -329,6 +337,41 @@ func parseIVPN(r io.Reader) (*netipx.IPSet, error) {
return addressSet(texts)
}
// parseLinesNotEmpty is parseLines for a list that always holds addresses, so
// a file with none fails.
func parseLinesNotEmpty(
r io.Reader, addLine func(*netipx.IPSetBuilder, string) error,
) (*netipx.IPSet, error) {
addresses, err := parseLines(r, addLine)
if err != nil {
return nil, err
}
if len(addresses.Ranges()) == 0 {
return nil, errNoAddresses
}
return addresses, nil
}
// addICloudPrivateRelayNetwork adds the network at the start of a line of
// Apple's iCloud Private Relay file to builder.
func addICloudPrivateRelayNetwork(builder *netipx.IPSetBuilder, line string) error {
field, _, found := strings.Cut(line, iCloudPrivateRelayFieldSeparator)
if !found {
return errNoNetwork
}
network, err := netip.ParsePrefix(field)
if err != nil {
return err
}
builder.AddPrefix(network)
return nil
}
// readJSON reads r to its end and decodes it, as one JSON document, into v.
func readJSON(r io.Reader, v any) error {
data, err := io.ReadAll(r)
+6
View File
@@ -801,6 +801,9 @@ type IPLookupFlags struct {
// IsVPN says that the address is on a list of VPN servers.
IsVPN bool `json:"is_vpn"`
// IsRelay says that the address is on the iCloud Private Relay list.
IsRelay bool `json:"is_relay"`
// IsBogon says that the address is reserved, as package bogon tells.
IsBogon bool `json:"is_bogon"`
}
@@ -1033,6 +1036,9 @@ func ipLookupFlags(ip string, listings []iplist.Listing) IPLookupFlags {
IsVPN: slices.ContainsFunc(listings, func(l iplist.Listing) bool {
return l.VPN
}),
IsRelay: slices.ContainsFunc(listings, func(l iplist.Listing) bool {
return l.Name == iplist.ICloudPrivateRelayList
}),
IsBogon: isBogon(ip),
}
}
+28 -16
View File
@@ -1506,6 +1506,10 @@ const listedUnroutedAddr = "203.0.113.1"
// torExitAddr is the address that fakeIPLists lists on the Tor exit list.
const torExitAddr = "203.0.113.2"
// relayAddr is the address that fakeIPLists lists on the iCloud Private Relay
// list.
const relayAddr = "203.0.113.5"
// The addresses that fakeIPLists lists on one and on two VPN lists, and the
// names of those lists.
const (
@@ -1515,10 +1519,11 @@ const (
otherTestVPN = "other_test_vpn"
)
// fakeIPLists has four lists. The test list holds addrInTestPrefixV4,
// fakeIPLists has five lists. The test list holds addrInTestPrefixV4,
// listedUnroutedAddr and twoVPNListsAddr. The Tor exit list holds torExitAddr.
// The test VPN list holds oneVPNListAddr and twoVPNListsAddr, and the other
// test VPN list holds twoVPNListsAddr.
// The iCloud Private Relay list holds relayAddr. The test VPN list holds
// oneVPNListAddr and twoVPNListsAddr, and the other test VPN list holds
// twoVPNListsAddr.
type fakeIPLists struct{}
func testListing() iplist.Listing {
@@ -1539,6 +1544,10 @@ func (fakeIPLists) Lookup(addr netip.Addr) []iplist.Listing {
return []iplist.Listing{{Name: iplist.TorExitsList}}
}
if addr == netip.MustParseAddr(relayAddr) {
return []iplist.Listing{{Name: iplist.ICloudPrivateRelayList}}
}
if addr == netip.MustParseAddr(oneVPNListAddr) {
return []iplist.Listing{{Name: testVPNList, VPN: true}}
}
@@ -1571,20 +1580,22 @@ func TestIPLookupFlags(t *testing.T) {
name string
addr string
tor bool
relay bool
bogon bool
}{
{"on the Tor exit list", torExitAddr, true, true},
{"on another list", listedUnroutedAddr, false, true},
{"routed IPv4", addrInTestPrefixV4, false, true},
{"routed IPv6", addrInTestPrefixV6, false, true},
{"private IPv4", "10.0.0.1", false, true},
{"loopback IPv4", "127.0.0.1", false, true},
{"public IPv4", "1.1.1.1", false, false},
{"private IPv6", "fd00::1", false, true},
{"loopback IPv6", "::1", false, true},
{"public IPv6", "2606:4700:4700::1111", false, false},
{"public IPv4 in IPv6 form", "::ffff:1.1.1.1", false, false},
{"private IPv4 in IPv6 form", "::ffff:10.0.0.1", false, true},
{"on the Tor exit list", torExitAddr, true, false, true},
{"on the iCloud Private Relay list", relayAddr, false, true, true},
{"on another list", listedUnroutedAddr, false, false, true},
{"routed IPv4", addrInTestPrefixV4, false, false, true},
{"routed IPv6", addrInTestPrefixV6, false, false, true},
{"private IPv4", "10.0.0.1", false, false, true},
{"loopback IPv4", "127.0.0.1", false, false, true},
{"public IPv4", "1.1.1.1", false, false, false},
{"private IPv6", "fd00::1", false, false, true},
{"loopback IPv6", "::1", false, false, true},
{"public IPv6", "2606:4700:4700::1111", false, false, false},
{"public IPv4 in IPv6 form", "::ffff:1.1.1.1", false, false, false},
{"private IPv4 in IPv6 form", "::ffff:10.0.0.1", false, false, true},
}
for _, tt := range tests {
@@ -1606,7 +1617,8 @@ func TestIPLookupFlags(t *testing.T) {
}
want := map[string]bool{
"is_tor": tt.tor, "is_vpn": false, "is_bogon": tt.bogon,
"is_tor": tt.tor, "is_vpn": false, "is_relay": tt.relay,
"is_bogon": tt.bogon,
}
if !maps.Equal(got, want) {
t.Errorf("%s: flags %v, want %v", path, got, want)
+2 -2
View File
@@ -67,8 +67,8 @@ func TestFooterLicense(t *testing.T) {
}
}
// TestHomePageCreditsDBIP expects the home page's footer to carry the
// attribution the DB-IP licence requires.
// TestHomePageCreditsDBIP expects the home page's footer to carry the credit
// to DB-IP.
func TestHomePageCreditsDBIP(t *testing.T) {
t.Parallel()