Author SHA1 Message Date
sneak 6e903d511f Add best-effort geolocation to the IP lookup from geojs and DB-IP City Lite (closes #239)
check / check (push) Successful in 1m50s
Each result gains `geo`. One request to geojs carries every address, 1 second in all. On a timeout or error the lookup reads DB-IP City Lite, downloaded monthly into the state directory. geojs answers stay in memory (250,000, least recently used dropped) and in a new `cache.db` table for 7 days.

Judgement call: a reserved address goes to neither source, and its `geo` is null.
Judgement call: an answer from geojs with every part unknown is still an answer.
Judgement call: `updated` for DB-IP is when the copy was downloaded.
Deviation: the cache is SQLite, by sneak's words.

Model: opus-5-5
2026-11-10 13:03:58 +00:00
14 changed files with 50 additions and 436 deletions

No files matched your search

+6 -29
View File
@@ -287,7 +287,6 @@ address. Every entry has the same fields:
- `is_tor`: true when the address is on the `tor_exits` list - `is_tor`: true when the address is on the `tor_exits` list
- `is_vpn`: true when the address is on any list of VPN servers (see IP - `is_vpn`: true when the address is on any list of VPN servers (see IP
lists) lists)
- `is_relay`: true when the address is on the `icloud_private_relay` list
- `is_bogon`: true when the address is reserved, so that no network should - `is_bogon`: true when the address is reserved, so that no network should
announce it on the internet announce it on the internet
- `vpn_providers`: the names of the lists of VPN servers the address is on, or - `vpn_providers`: the names of the lists of VPN servers the address is on, or
@@ -423,9 +422,6 @@ The lists:
last 30 days. From last 30 days. From
[FireHOL](https://iplists.firehol.org/?ipset=firehol_level3), downloaded every [FireHOL](https://iplists.firehol.org/?ipset=firehol_level3), downloaded every
3 hours. 3 hours.
- `icloud_private_relay`: the networks where iCloud Private Relay traffic
leaves. From [Apple](https://mask-api.icloud.com/egress-ip-ranges.csv),
downloaded once a day.
- `ivpn`: VPN servers run by IVPN. From - `ivpn`: VPN servers run by IVPN. From
[IVPN](https://api.ivpn.net/v5/servers.json), downloaded once a day. [IVPN](https://api.ivpn.net/v5/servers.json), downloaded once a day.
- `mullvad`: VPN servers run by Mullvad. From - `mullvad`: VPN servers run by Mullvad. From
@@ -436,14 +432,6 @@ The lists:
- `pia`: VPN servers run by Private Internet Access. From - `pia`: VPN servers run by Private Internet Access. From
[Private Internet Access](https://serverlist.piaservers.net/vpninfo/servers/v6), [Private Internet Access](https://serverlist.piaservers.net/vpninfo/servers/v6),
downloaded once a day. downloaded once a day.
- `protonvpn`: the /24 networks that hold ProtonVPN servers. The whole /24 is
listed because the ProtonVPN-IPs README says traffic leaves from unpublished
addresses in it. From the public
[ProtonVPN-IPs](https://github.com/tn3w/ProtonVPN-IPs) mirror of Proton's
server list, downloaded once a day.
- `protonvpn_entry`: the IPv4 and IPv6 addresses of ProtonVPN servers. From the
same [ProtonVPN-IPs](https://github.com/tn3w/ProtonVPN-IPs) mirror, downloaded
once a day.
- `spamhaus_drop` and `spamhaus_dropv6`: IPv4 and IPv6 networks hijacked or - `spamhaus_drop` and `spamhaus_dropv6`: IPv4 and IPv6 networks hijacked or
leased by spammers or cyber criminals, according to Spamhaus. From leased by spammers or cyber criminals, according to Spamhaus. From
[Spamhaus DROP](https://www.spamhaus.org/blocklists/do-not-route-or-peer/), [Spamhaus DROP](https://www.spamhaus.org/blocklists/do-not-route-or-peer/),
@@ -522,13 +510,9 @@ sources:
- [geojs](https://www.geojs.io/) is asked first. One request carries every - [geojs](https://www.geojs.io/) is asked first. One request carries every
address of the lookup, and it has 1 second to answer. address of the lookup, and it has 1 second to answer.
- [DB-IP City Lite](https://db-ip.com/db/download/ip-to-city-lite) answers when - [DB-IP City Lite](https://db-ip.com/db/download/ip-to-city-lite) answers when
geojs has no answer for the address. It is a database file that RouteWatch geojs fails or does not answer in time. It is a database file that RouteWatch
reads itself. reads itself.
geojs has no answer when its request fails or runs out of time. An answer with
no country, city or coordinates also counts as no answer, and RouteWatch does
not cache it.
The `geo` object of a result has these fields: The `geo` object of a result has these fields:
- `country_code`: the two-letter country code - `country_code`: the two-letter country code
@@ -550,13 +534,13 @@ goes to neither source, so its `geo` is null.
What goes outside: geojs receives each address that the cache does not hold. For What goes outside: geojs receives each address that the cache does not hold. For
`/ip/` and the home page, that address is the visitor's own. A lookup answered `/ip/` and the home page, that address is the visitor's own. A lookup answered
from the cache sends nothing. DB-IP receives only the monthly request for its from the cache or from the database sends nothing. DB-IP receives only the
file. monthly request for its file.
How RouteWatch keeps the answers of geojs: How RouteWatch keeps the answers of geojs:
- It holds at most 250,000 addresses in memory, and drops the least recently - It holds at most 250,000 addresses in memory, and drops the least recently
used first. That takes about 100 MB of the Go heap. used first. That takes about 75 MB of the Go heap.
- It writes new answers every 5 seconds to `cache.db` in the state directory. - It writes new answers every 5 seconds to `cache.db` in the state directory.
- It loads the newest 250,000 answers from `cache.db` at startup. - It loads the newest 250,000 answers from `cache.db` at startup.
- An address not in memory is read from `cache.db` before geojs is asked. - An address not in memory is read from `cache.db` before geojs is asked.
@@ -577,8 +561,8 @@ How RouteWatch keeps the database:
The kernel loads the parts that are read, and can drop them again when memory The kernel loads the parts that are read, and can drop them again when memory
is short. is short.
The home page's footer says "IP Geolocation by DB-IP" with a link to DB-IP's licence requires an attribution, so the home page's footer says "IP
[db-ip.com](https://db-ip.com). Geolocation by DB-IP" with a link to [db-ip.com](https://db-ip.com).
## Configuration ## Configuration
@@ -680,13 +664,6 @@ The ceilings add up to 3.7 GiB:
limit caps the SQLite library. limit caps the SQLite library.
- About 0.2 GiB for everything else in the runtime. - About 0.2 GiB for everything else in the runtime.
The `icloud_private_relay` list has about 285,000 networks. Neighbouring
networks merge, which leaves about 6,900 address ranges. Loaded, the list adds
about 0.3 MiB to the Go heap and under 6 MiB to the process's resident memory.
Parsing the file allocates about 94 MiB in about 0.2 seconds. The next garbage
collection frees all of it but the 0.3 MiB above. Parsing happens at startup and
whenever a download brings a new version.
The SQLite library gets its memory from glibc, the C library. glibc keeps freed The SQLite library gets its memory from glibc, the C library. glibc keeps freed
memory in pools called arenas, and hands it back to the kernel slowly. It memory in pools called arenas, and hands it back to the kernel slowly. It
creates up to eight arenas per processor core. So on a host with many cores the creates up to eight arenas per processor core. So on a host with many cores the
-7
View File
@@ -57,13 +57,6 @@ problem the run finds.
comes from geojs, with DB-IP's City Lite database as the fallback. The geojs comes from geojs, with DB-IP's City Lite database as the fallback. The geojs
answers stay in memory and in `cache.db` for 7 days (closes answers stay in memory and in `cache.db` for 7 days (closes
https://git.eeqj.de/sneak/routewatch/issues/239). https://git.eeqj.de/sneak/routewatch/issues/239).
- 2026-11-10: the IP lookup checks the `icloud_private_relay` list of Apple's
iCloud Private Relay networks, downloaded once a day. Each result has
`flags.is_relay` (closes https://git.eeqj.de/sneak/routewatch/issues/237).
- 2026-11-10: the IP lookup checks the `protonvpn` list of ProtonVPN's /24
networks and the `protonvpn_entry` list of its servers' IPv4 and IPv6
addresses, from a public mirror of Proton's server list (closes
https://git.eeqj.de/sneak/routewatch/issues/236).
- 2026-11-10: the IP lookup finds PTR names over DNS-over-HTTPS and caches them - 2026-11-10: the IP lookup finds PTR names over DNS-over-HTTPS and caches them
for the record's TTL, in memory and in `cache.db` (closes for the record's TTL, in memory and in `cache.db` (closes
https://git.eeqj.de/sneak/routewatch/issues/238). https://git.eeqj.de/sneak/routewatch/issues/238).
+2 -12
View File
@@ -107,8 +107,7 @@ func nilIfEmpty(text string) *string {
} }
// fromDatabase returns where the copy of the database places addr. It returns // fromDatabase returns where the copy of the database places addr. It returns
// nil when there is no copy yet or the copy has no record for addr. It also // nil when there is no copy yet or the copy has no record for addr.
// returns nil, and logs the error, when reading the copy fails.
func (l *Locator) fromDatabase(addr netip.Addr) *Location { func (l *Locator) fromDatabase(addr netip.Addr) *Location {
l.databaseMu.RLock() l.databaseMu.RLock()
defer l.databaseMu.RUnlock() defer l.databaseMu.RUnlock()
@@ -118,22 +117,13 @@ func (l *Locator) fromDatabase(addr netip.Addr) *Location {
} }
result := l.database.Lookup(addr) result := l.database.Lookup(addr)
err := result.Err()
if err != nil {
l.logger.Error("Failed to look up an address in the database",
"address", addr, "error", err)
return nil
}
if !result.Found() { if !result.Found() {
return nil return nil
} }
var record dbipRecord var record dbipRecord
err = result.Decode(&record) err := result.Decode(&record)
if err != nil { if err != nil {
l.logger.Error("Failed to read a database record", l.logger.Error("Failed to read a database record",
"address", addr, "error", err) "address", addr, "error", err)
+2 -12
View File
@@ -51,8 +51,7 @@ type geojsAnswer struct {
} }
// askGeoJS asks geojs about addresses in one request, and returns its answers // askGeoJS asks geojs about addresses in one request, and returns its answers
// by address. An address geojs left out, or gave no country, city or // by address. An address geojs left out has no entry.
// coordinates for, has no entry.
func (l *Locator) askGeoJS( func (l *Locator) askGeoJS(
ctx context.Context, addresses []string, ctx context.Context, addresses []string,
) (map[string]*Location, error) { ) (map[string]*Location, error) {
@@ -98,21 +97,12 @@ func (l *Locator) askGeoJS(
continue continue
} }
location := answer.location(fetched) answers[addr.Unmap().String()] = answer.location(fetched)
if hasLocation(location) {
answers[plainAddress(addr)] = location
}
} }
return answers, nil return answers, nil
} }
// hasLocation says whether found has a country, a city or a coordinate.
func hasLocation(found *Location) bool {
return found.CountryCode != nil || found.Country != nil ||
found.City != nil || found.Latitude != nil || found.Longitude != nil
}
// location returns the answer as a Location fetched at the time given. // location returns the answer as a Location fetched at the time given.
func (a geojsAnswer) location(fetched time.Time) *Location { func (a geojsAnswer) location(fetched time.Time) *Location {
return &Location{ return &Location{
+4 -12
View File
@@ -1,7 +1,6 @@
// Package geo finds where IP addresses are. It asks geojs.io, and keeps those // Package geo finds where IP addresses are. It asks geojs.io, and keeps those
// answers in memory and in cache.db. When geojs.io gives no location for an // answers in memory and in cache.db. When geojs.io does not answer in time, it
// address, it reads a copy of DB-IP's City Lite database, which it downloads // reads a copy of DB-IP's City Lite database, which it downloads once a month.
// once a month.
package geo package geo
import ( import (
@@ -142,7 +141,7 @@ func (l *Locator) Lookup(ctx context.Context, addrs []netip.Addr) []*Location {
var unanswered []string var unanswered []string
for i, addr := range addrs { for i, addr := range addrs {
address := plainAddress(addr) address := addr.Unmap().String()
locations[i] = l.cached(ctx, address) locations[i] = l.cached(ctx, address)
if locations[i] == nil { if locations[i] == nil {
@@ -164,7 +163,7 @@ func (l *Locator) Lookup(ctx context.Context, addrs []netip.Addr) []*Location {
continue continue
} }
address := plainAddress(addr) address := addr.Unmap().String()
answer, ok := answers[address] answer, ok := answers[address]
if ok { if ok {
@@ -180,13 +179,6 @@ func (l *Locator) Lookup(ctx context.Context, addrs []netip.Addr) []*Location {
return locations return locations
} }
// plainAddress returns addr as the requests to geojs and the cache name it: an
// IPv4 address mapped into IPv6 as plain IPv4, and with no zone. A client
// chooses the zone, so it must not reach the request.
func plainAddress(addr netip.Addr) string {
return addr.Unmap().WithZone("").String()
}
// Start loads the cached answers that have not expired and opens the copy of // Start loads the cached answers that have not expired and opens the copy of
// the database, if there is one. It then starts the goroutines that write new // the database, if there is one. It then starts the goroutines that write new
// answers, delete expired ones and download the database. They run until Stop: // answers, delete expired ones and download the database. They run until Stop:
+4 -100
View File
@@ -53,18 +53,9 @@ const (
"country": "Germany", "country_code": "DE", "ip": "` + testAddress + `", "country": "Germany", "country_code": "DE", "ip": "` + testAddress + `",
"latitude": "52.6171", "longitude": "13.1207", "latitude": "52.6171", "longitude": "13.1207",
"region": "Brandenburg", "timezone": "Europe/Berlin"}, "region": "Brandenburg", "timezone": "Europe/Berlin"},
{"country": "Germany", "ip": "` + otherAddress + `", {"ip": "` + otherAddress + `", "latitude": "nil", "longitude": "nil"}
"latitude": "nil", "longitude": "nil"}
]` ]`
geojsOneAddress = `[{"ip": "` + testAddress + `", "city": "` + testCity + `"}]` geojsOneAddress = `[{"ip": "` + testAddress + `", "city": "` + testCity + `"}]`
// geojsNoLocation has no country, city or coordinates, as geojs sends for
// some addresses. Its other parts do not count as a location.
geojsNoLocation = `[{"accuracy": 1000, "area_code": "0", "asn": 64496,
"continent_code": "EU", "ip": "` + testAddress + `",
"latitude": "nil", "longitude": "nil",
"organization": "AS64496 Example", "organization_name": "Example",
"region": "Brandenburg", "timezone": "Europe/Berlin"}]`
) )
// fixedTime returns the time a test's clock starts at. // fixedTime returns the time a test's clock starts at.
@@ -324,8 +315,9 @@ func TestLookupAsksGeoJSOnceForEveryAddress(t *testing.T) {
} }
} }
// TestLookupGivesUnknownPartsAsNil checks an answer with only a country name, // TestLookupGivesUnknownPartsAsNil checks an answer whose coordinates are the
// whose coordinates are the string "nil": the other parts are nil. // string "nil" and which has no other part: the parts are nil, and the answer
// is still an answer.
func TestLookupGivesUnknownPartsAsNil(t *testing.T) { func TestLookupGivesUnknownPartsAsNil(t *testing.T) {
t.Parallel() t.Parallel()
@@ -376,7 +368,6 @@ func TestLookupFallsBackToTheDatabase(t *testing.T) {
{name: "error status", status: http.StatusBadGateway}, {name: "error status", status: http.StatusBadGateway},
{name: "unreadable answer", body: "not JSON"}, {name: "unreadable answer", body: "not JSON"},
{name: "address left out", body: "[]"}, {name: "address left out", body: "[]"},
{name: "answer with no location", body: geojsNoLocation},
} }
for _, tt := range tests { for _, tt := range tests {
@@ -420,93 +411,6 @@ func TestLookupFallsBackToTheDatabase(t *testing.T) {
} }
} }
// TestAnyCountryCityOrCoordinateIsAnAnswer checks that each of those parts,
// alone, makes an answer from geojs that is returned and kept.
func TestAnyCountryCityOrCoordinateIsAnAnswer(t *testing.T) {
t.Parallel()
parts := []string{
`"country": "Germany"`,
`"country_code": "DE"`,
`"city": "` + testCity + `"`,
`"latitude": "52.5"`,
`"longitude": "13.5"`,
}
for _, part := range parts {
t.Run(part, func(t *testing.T) {
t.Parallel()
fake := newFakeGeoJS(t, `[{"ip": "`+testAddress+`", `+part+`}]`)
dir := t.TempDir()
locator := newTestLocator(t, fake, openCacheDB(t, dir), dir)
got := lookupOne(t, locator, testAddress)
if got == nil || got.Source != sourceGeoJS {
t.Fatalf("location %+v, want the answer from geojs", got)
}
if !locator.entries.Contains(testAddress) {
t.Error("the answer is not in memory")
}
})
}
}
// TestLookupSendsNoZoneToGeoJS looks up an address with a zone, which is text
// a client chooses. geojs receives only the plain address, and the answer is
// kept under it.
func TestLookupSendsNoZoneToGeoJS(t *testing.T) {
t.Parallel()
fake := newFakeGeoJS(t, geojsTwoAddresses)
dir := t.TempDir()
locator := newTestLocator(t, fake, openCacheDB(t, dir), dir)
got := lookupOne(t, locator, otherAddress+"%x&callback=evil")
requests := fake.requestsSeen()
if len(requests) != 1 || requests[0] != otherAddress {
t.Errorf("requests %v, want one naming only %s", requests, otherAddress)
}
if got == nil || got.Source != sourceGeoJS {
t.Errorf("location %+v, want the answer from geojs", got)
}
if !locator.entries.Contains(otherAddress) {
t.Error("the answer is not kept under the plain address")
}
}
// TestFailedDatabaseLookupIsLogged closes the copy under the locator, so that
// the lookup in it fails. The location is nil and the failure is logged.
func TestFailedDatabaseLookupIsLogged(t *testing.T) {
t.Parallel()
fake := newFakeGeoJS(t, "[]")
dir := t.TempDir()
locator := newTestLocator(t, fake, openCacheDB(t, dir), dir)
giveDatabase(t, locator)
err := locator.database.Close()
if err != nil {
t.Fatalf("closing the copy: %v", err)
}
var logged bytes.Buffer
locator.logger = slog.New(slog.NewTextHandler(&logged, nil))
if got := locator.fromDatabase(parseAddresses(t, testAddress)[0]); got != nil {
t.Errorf("location %+v from a failed lookup, want nil", got)
}
if !strings.Contains(logged.String(), "address="+testAddress) {
t.Errorf("the failure was not logged: %q", logged.String())
}
}
// TestLookupIsNilWhenNeitherSourceAnswers makes geojs fail with no database // TestLookupIsNilWhenNeitherSourceAnswers makes geojs fail with no database
// copy, and then with a copy that does not cover the address. // copy, and then with a copy that does not cover the address.
func TestLookupIsNilWhenNeitherSourceAnswers(t *testing.T) { func TestLookupIsNilWhenNeitherSourceAnswers(t *testing.T) {
+1 -1
View File
@@ -21,7 +21,7 @@ const (
downloadTimeout = time.Minute downloadTimeout = time.Minute
// maxDownloadSize is the size of the largest list file accepted, in // maxDownloadSize is the size of the largest list file accepted, in
// bytes. The largest list, iCloud Private Relay, is about 12 MiB. // bytes. The largest list, FireHOL abusers, is about 2.5 MiB.
maxDownloadSize = 200 << 20 maxDownloadSize = 200 << 20
) )
-10
View File
@@ -51,16 +51,6 @@ const (
// FormatIVPNJSON is IVPN's JSON server list. The hosts of its wireguard // FormatIVPNJSON is IVPN's JSON server list. The hosts of its wireguard
// and openvpn gateways have their addresses in host and v2ray. // and openvpn gateways have their addresses in host and v2ray.
FormatIVPNJSON FormatIVPNJSON
// FormatProtonVPN has one address or network per line, with comment lines
// that start with #. Unlike FormatLines, a file with no addresses or
// networks fails.
FormatProtonVPN
// FormatICloudPrivateRelayCSV is Apple's CSV file of iCloud Private Relay
// networks. Each line starts with a network, followed by its country,
// region and city.
FormatICloudPrivateRelayCSV
) )
// List is one public IP list. // List is one public IP list.
+13 -135
View File
@@ -70,31 +70,6 @@ const (
"config":{"api":{"ips":["192.0.2.73"]}}}` "config":{"api":{"ips":["192.0.2.73"]}}}`
) )
// protonVPNBody is a small file in the format of the ProtonVPN-IPs list.
const protonVPNBody = "#\n# protonvpn_ip_ranges.txt\n" +
"# https://github.com/tn3w/ProtonVPN-IPs\n#\n" +
"# The /24 block of every advertised IPv4.\n#\n" +
"192.0.2.0/24\n198.51.100.0/24\n"
// protonVPNEntryBody is a small file in the format of the ProtonVPN-IPs list
// of server addresses.
const protonVPNEntryBody = "#\n# protonvpn_entry_ips.txt\n" +
"# https://github.com/tn3w/ProtonVPN-IPs\n#\n" +
"# Entry IPs clients connect to, resolved from ProtonVPN hostnames.\n#\n" +
"192.0.2.80\n2001:db8::80\n"
// iCloudPrivateRelayAddr is an address on iCloudPrivateRelayBody.
const iCloudPrivateRelayAddr = "2001:db8:1::5"
// iCloudPrivateRelayBody is a small file in the format of Apple's iCloud
// Private Relay file.
const iCloudPrivateRelayBody = "192.0.2.0/31,GB,GB-EN,London,\n" +
"198.51.100.32/27,US,US-CA,Los Angeles,\n" +
"2001:db8:1::/64,US,US-AK,Anchorage,\n"
// emptyFileCase names the case of a wrong document that is an empty file.
const emptyFileCase = "an empty file"
// errorPage is a file that is no list, as a publisher's error page is. // errorPage is a file that is no list, as a publisher's error page is.
const errorPage = "<!DOCTYPE html>\n<html></html>\n" const errorPage = "<!DOCTYPE html>\n<html></html>\n"
@@ -255,15 +230,6 @@ func TestLookupFindsListedAddresses(t *testing.T) {
}, },
unlisted: []string{"192.0.2.12", "2001:db8::2"}, unlisted: []string{"192.0.2.12", "2001:db8::2"},
}, },
{
name: "iCloud Private Relay networks",
format: FormatICloudPrivateRelayCSV,
body: iCloudPrivateRelayBody,
listed: []string{
"192.0.2.1", "198.51.100.63", iCloudPrivateRelayAddr,
},
unlisted: []string{"192.0.2.2", "198.51.100.64", "2001:db8:2::1"},
},
}) })
} }
@@ -301,20 +267,6 @@ func TestLookupFindsVPNServers(t *testing.T) {
listed: []string{"192.0.2.70", "192.0.2.71", "192.0.2.72"}, listed: []string{"192.0.2.70", "192.0.2.71", "192.0.2.72"},
unlisted: []string{"192.0.2.73", "2001:db8::70"}, unlisted: []string{"192.0.2.73", "2001:db8::70"},
}, },
{
name: "ProtonVPN networks",
format: FormatProtonVPN,
body: protonVPNBody,
listed: []string{"192.0.2.0", "192.0.2.255", "198.51.100.7"},
unlisted: []string{"192.0.3.1", "2001:db8::90"},
},
{
name: "ProtonVPN server addresses",
format: FormatProtonVPN,
body: protonVPNEntryBody,
listed: []string{"192.0.2.80", "2001:db8::80"},
unlisted: []string{"192.0.2.81", "2001:db8::81"},
},
}) })
} }
@@ -506,12 +458,12 @@ func TestWrongVPNDocumentFailsToParse(t *testing.T) {
for _, f := range formats { for _, f := range formats {
bodies := map[string]string{ bodies := map[string]string{
"not JSON": errorPage, "not JSON": errorPage,
"other JSON": `{"status":"ok","data":{"count":1}}`, "other JSON": `{"status":"ok","data":{"count":1}}`,
"no addresses": f.empty, "no addresses": f.empty,
"bad address": f.badAddress, "bad address": f.badAddress,
"cut off": f.good[:len(f.good)/2], "cut off": f.good[:len(f.good)/2],
emptyFileCase: "", "an empty file": "",
} }
for name, body := range bodies { for name, body := range bodies {
@@ -527,59 +479,6 @@ func TestWrongVPNDocumentFailsToParse(t *testing.T) {
} }
} }
// TestWrongProtonVPNDocumentFailsToParse parses documents that are not a
// ProtonVPN-IPs list. Each fails, so none can replace a good copy.
func TestWrongProtonVPNDocumentFailsToParse(t *testing.T) {
t.Parallel()
bodies := map[string]string{
"error page": errorPage,
"JSON": `{"status":"ok"}`,
"comments only": "#\n# protonvpn_ip_ranges.txt\n#\n",
"bad network": "192.0.2.0/24\nexample\n",
emptyFileCase: "",
}
for name, body := range bodies {
t.Run(name, func(t *testing.T) {
t.Parallel()
addresses, err := parse(strings.NewReader(body), FormatProtonVPN)
if err == nil {
t.Errorf("parse returned %v and no error", addresses.Ranges())
}
})
}
}
// TestWrongICloudPrivateRelayDocumentFailsToParse parses documents that are
// not Apple's iCloud Private Relay file. Each fails, so none can replace a
// good copy.
func TestWrongICloudPrivateRelayDocumentFailsToParse(t *testing.T) {
t.Parallel()
bodies := map[string]string{
"error page": errorPage,
"JSON": `{"status":"ok","count":1}`,
"no other field": "192.0.2.0/24\n",
"bad network": "192.0.2.0/24,GB,GB-EN,London,\nexample,GB,GB-EN,London,\n",
"an address": "192.0.2.1,GB,GB-EN,London,\n",
emptyFileCase: "",
}
for name, body := range bodies {
t.Run(name, func(t *testing.T) {
t.Parallel()
addresses, err := parse(strings.NewReader(body),
FormatICloudPrivateRelayCSV)
if err == nil {
t.Errorf("parse returned %v and no error", addresses.Ranges())
}
})
}
}
// TestLookupMarksVPNLists looks up an address on a VPN list and on another // TestLookupMarksVPNLists looks up an address on a VPN list and on another
// list. Only the VPN list's listing says it is one. // list. Only the VPN list's listing says it is one.
func TestLookupMarksVPNLists(t *testing.T) { func TestLookupMarksVPNLists(t *testing.T) {
@@ -624,10 +523,7 @@ func TestPublicListsMarkTheVPNLists(t *testing.T) {
} }
} }
want := []string{ want := []string{"ivpn", "mullvad", "nordvpn", "pia", "x4bnet", "x4bnet_v6"}
"ivpn", "mullvad", "nordvpn", "pia", "protonvpn", "protonvpn_entry",
"x4bnet", "x4bnet_v6",
}
if !slices.Equal(got, want) { if !slices.Equal(got, want) {
t.Errorf("VPN lists %v, want %v", got, want) t.Errorf("VPN lists %v, want %v", got, want)
} }
@@ -934,33 +830,15 @@ func TestStopCancelsADownloadInProgress(t *testing.T) {
func TestTorExitsListHoldsIPv6Exits(t *testing.T) { func TestTorExitsListHoldsIPv6Exits(t *testing.T) {
t.Parallel() t.Parallel()
checkPublicListHolds(t, TorExitsList, onionooBody, onionooIPv6ExitAddr)
}
// TestICloudPrivateRelayListHoldsItsNetworks downloads a file in Apple's
// format as the public iCloud Private Relay list, which is what an address's
// is_relay answer comes from.
func TestICloudPrivateRelayListHoldsItsNetworks(t *testing.T) {
t.Parallel()
checkPublicListHolds(t, ICloudPrivateRelayList, iCloudPrivateRelayBody,
iCloudPrivateRelayAddr)
}
// checkPublicListHolds downloads body as the public list called name and
// checks that addr has one listing, which names that list.
func checkPublicListHolds(t *testing.T, name, body, addr string) {
t.Helper()
index := slices.IndexFunc(Lists(), func(list List) bool { index := slices.IndexFunc(Lists(), func(list List) bool {
return list.Name == name return list.Name == TorExitsList
}) })
if index < 0 { if index < 0 {
t.Fatalf("no list named %s", name) t.Fatalf("no list named %s", TorExitsList)
} }
list := Lists()[index] list := Lists()[index]
_, list.URL = newPublisher(t, body) _, list.URL = newPublisher(t, onionooBody)
b := newTestIPLists(t.TempDir(), list) b := newTestIPLists(t.TempDir(), list)
err := b.download(t.Context(), list) err := b.download(t.Context(), list)
@@ -968,9 +846,9 @@ func checkPublicListHolds(t *testing.T, name, body, addr string) {
t.Fatalf("download: %v", err) t.Fatalf("download: %v", err)
} }
listings := lookup(t, b, addr) listings := lookup(t, b, onionooIPv6ExitAddr)
if len(listings) != 1 || listings[0].Name != name { if len(listings) != 1 || listings[0].Name != TorExitsList {
t.Errorf("listings %+v, want one for %s", listings, name) t.Errorf("listings %+v, want one for %s", listings, TorExitsList)
} }
} }
-39
View File
@@ -8,13 +8,6 @@ import (
// TorExitsList is the name of the list of Tor exit nodes. // TorExitsList is the name of the list of Tor exit nodes.
const TorExitsList = "tor_exits" const TorExitsList = "tor_exits"
// ICloudPrivateRelayList is the name of the list of iCloud Private Relay's
// networks.
const ICloudPrivateRelayList = "icloud_private_relay"
// protonVPNIPsURL is where the ProtonVPN-IPs mirror publishes its files.
const protonVPNIPsURL = "https://raw.githubusercontent.com/tn3w/ProtonVPN-IPs/master/"
// How often each list is downloaded. No list is downloaded more often than // How often each list is downloaded. No list is downloaded more often than
// its publisher updates it or allows. // its publisher updates it or allows.
const ( const (
@@ -34,9 +27,6 @@ const (
fireHOLAbusersInterval = 5 * time.Hour fireHOLAbusersInterval = 5 * time.Hour
fireHOLLevel3Interval = 3 * time.Hour fireHOLLevel3Interval = 3 * time.Hour
// Apple's iCloud Private Relay file is 12 MB, so once a day.
iCloudPrivateRelayInterval = 24 * time.Hour
// Spamhaus asks that DROP be downloaded at most once a day: // Spamhaus asks that DROP be downloaded at most once a day:
// https://www.spamhaus.org/blocklists/do-not-route-or-peer/ // https://www.spamhaus.org/blocklists/do-not-route-or-peer/
spamhausDROPInterval = 24 * time.Hour spamhausDROPInterval = 24 * time.Hour
@@ -91,13 +81,6 @@ func nonVPNLists() []List {
Interval: fireHOLLevel3Interval, Interval: fireHOLLevel3Interval,
Format: FormatLines, Format: FormatLines,
}, },
{
Name: ICloudPrivateRelayList,
Description: "In a network where iCloud Private Relay traffic leaves",
URL: "https://mask-api.icloud.com/egress-ip-ranges.csv",
Interval: iCloudPrivateRelayInterval,
Format: FormatICloudPrivateRelayCSV,
},
{ {
Name: "spamhaus_drop", Name: "spamhaus_drop",
Description: "In an IPv4 network that Spamhaus says spammers " + Description: "In an IPv4 network that Spamhaus says spammers " +
@@ -163,28 +146,6 @@ func vpnLists() []List {
Format: FormatPIAJSON, Format: FormatPIAJSON,
VPN: true, VPN: true,
}, },
{
// Proton's own server list needs a logged-in session, so this is
// the public ProtonVPN-IPs mirror of it. Proton publishes only
// the addresses clients connect to. The mirror's README says
// traffic leaves from others in the same /24, so this file of
// /24 networks is the one it says to use for detecting exits.
Name: "protonvpn",
Description: "In a /24 network that holds a server of the " +
"ProtonVPN service",
URL: protonVPNIPsURL + "protonvpn_ip_ranges.txt",
Interval: vpnListInterval,
Format: FormatProtonVPN,
VPN: true,
},
{
Name: "protonvpn_entry",
Description: "A server of the ProtonVPN service",
URL: protonVPNIPsURL + "protonvpn_entry_ips.txt",
Interval: vpnListInterval,
Format: FormatProtonVPN,
VPN: true,
},
{ {
Name: "x4bnet", Name: "x4bnet",
Description: "In an IPv4 network of a VPN service, according to " + Description: "In an IPv4 network of a VPN service, according to " +
-43
View File
@@ -25,10 +25,6 @@ const (
// dshieldFieldSeparator separates the fields of a DShield line. // dshieldFieldSeparator separates the fields of a DShield line.
dshieldFieldSeparator = "\t" dshieldFieldSeparator = "\t"
// iCloudPrivateRelayFieldSeparator separates the fields of a line of
// Apple's iCloud Private Relay file.
iCloudPrivateRelayFieldSeparator = ","
// spamhausMetadataType is the type of the line that ends a Spamhaus // spamhausMetadataType is the type of the line that ends a Spamhaus
// file. That line holds the file's metadata and no network. // file. That line holds the file's metadata and no network.
spamhausMetadataType = "metadata" spamhausMetadataType = "metadata"
@@ -62,10 +58,6 @@ func parse(r io.Reader, format Format) (*netipx.IPSet, error) {
return parsePIA(r) return parsePIA(r)
case FormatIVPNJSON: case FormatIVPNJSON:
return parseIVPN(r) return parseIVPN(r)
case FormatProtonVPN:
return parseLinesNotEmpty(r, addAddressOrNetwork)
case FormatICloudPrivateRelayCSV:
return parseLinesNotEmpty(r, addICloudPrivateRelayNetwork)
default: default:
return nil, errUnknownFormat return nil, errUnknownFormat
} }
@@ -337,41 +329,6 @@ func parseIVPN(r io.Reader) (*netipx.IPSet, error) {
return addressSet(texts) return addressSet(texts)
} }
// parseLinesNotEmpty is parseLines for a list that always holds addresses, so
// a file with none fails.
func parseLinesNotEmpty(
r io.Reader, addLine func(*netipx.IPSetBuilder, string) error,
) (*netipx.IPSet, error) {
addresses, err := parseLines(r, addLine)
if err != nil {
return nil, err
}
if len(addresses.Ranges()) == 0 {
return nil, errNoAddresses
}
return addresses, nil
}
// addICloudPrivateRelayNetwork adds the network at the start of a line of
// Apple's iCloud Private Relay file to builder.
func addICloudPrivateRelayNetwork(builder *netipx.IPSetBuilder, line string) error {
field, _, found := strings.Cut(line, iCloudPrivateRelayFieldSeparator)
if !found {
return errNoNetwork
}
network, err := netip.ParsePrefix(field)
if err != nil {
return err
}
builder.AddPrefix(network)
return nil
}
// readJSON reads r to its end and decodes it, as one JSON document, into v. // readJSON reads r to its end and decodes it, as one JSON document, into v.
func readJSON(r io.Reader, v any) error { func readJSON(r io.Reader, v any) error {
data, err := io.ReadAll(r) data, err := io.ReadAll(r)
-6
View File
@@ -801,9 +801,6 @@ type IPLookupFlags struct {
// IsVPN says that the address is on a list of VPN servers. // IsVPN says that the address is on a list of VPN servers.
IsVPN bool `json:"is_vpn"` IsVPN bool `json:"is_vpn"`
// IsRelay says that the address is on the iCloud Private Relay list.
IsRelay bool `json:"is_relay"`
// IsBogon says that the address is reserved, as package bogon tells. // IsBogon says that the address is reserved, as package bogon tells.
IsBogon bool `json:"is_bogon"` IsBogon bool `json:"is_bogon"`
} }
@@ -1036,9 +1033,6 @@ func ipLookupFlags(ip string, listings []iplist.Listing) IPLookupFlags {
IsVPN: slices.ContainsFunc(listings, func(l iplist.Listing) bool { IsVPN: slices.ContainsFunc(listings, func(l iplist.Listing) bool {
return l.VPN return l.VPN
}), }),
IsRelay: slices.ContainsFunc(listings, func(l iplist.Listing) bool {
return l.Name == iplist.ICloudPrivateRelayList
}),
IsBogon: isBogon(ip), IsBogon: isBogon(ip),
} }
} }
+16 -28
View File
@@ -1506,10 +1506,6 @@ const listedUnroutedAddr = "203.0.113.1"
// torExitAddr is the address that fakeIPLists lists on the Tor exit list. // torExitAddr is the address that fakeIPLists lists on the Tor exit list.
const torExitAddr = "203.0.113.2" const torExitAddr = "203.0.113.2"
// relayAddr is the address that fakeIPLists lists on the iCloud Private Relay
// list.
const relayAddr = "203.0.113.5"
// The addresses that fakeIPLists lists on one and on two VPN lists, and the // The addresses that fakeIPLists lists on one and on two VPN lists, and the
// names of those lists. // names of those lists.
const ( const (
@@ -1519,11 +1515,10 @@ const (
otherTestVPN = "other_test_vpn" otherTestVPN = "other_test_vpn"
) )
// fakeIPLists has five lists. The test list holds addrInTestPrefixV4, // fakeIPLists has four lists. The test list holds addrInTestPrefixV4,
// listedUnroutedAddr and twoVPNListsAddr. The Tor exit list holds torExitAddr. // listedUnroutedAddr and twoVPNListsAddr. The Tor exit list holds torExitAddr.
// The iCloud Private Relay list holds relayAddr. The test VPN list holds // The test VPN list holds oneVPNListAddr and twoVPNListsAddr, and the other
// oneVPNListAddr and twoVPNListsAddr, and the other test VPN list holds // test VPN list holds twoVPNListsAddr.
// twoVPNListsAddr.
type fakeIPLists struct{} type fakeIPLists struct{}
func testListing() iplist.Listing { func testListing() iplist.Listing {
@@ -1544,10 +1539,6 @@ func (fakeIPLists) Lookup(addr netip.Addr) []iplist.Listing {
return []iplist.Listing{{Name: iplist.TorExitsList}} return []iplist.Listing{{Name: iplist.TorExitsList}}
} }
if addr == netip.MustParseAddr(relayAddr) {
return []iplist.Listing{{Name: iplist.ICloudPrivateRelayList}}
}
if addr == netip.MustParseAddr(oneVPNListAddr) { if addr == netip.MustParseAddr(oneVPNListAddr) {
return []iplist.Listing{{Name: testVPNList, VPN: true}} return []iplist.Listing{{Name: testVPNList, VPN: true}}
} }
@@ -1580,22 +1571,20 @@ func TestIPLookupFlags(t *testing.T) {
name string name string
addr string addr string
tor bool tor bool
relay bool
bogon bool bogon bool
}{ }{
{"on the Tor exit list", torExitAddr, true, false, true}, {"on the Tor exit list", torExitAddr, true, true},
{"on the iCloud Private Relay list", relayAddr, false, true, true}, {"on another list", listedUnroutedAddr, false, true},
{"on another list", listedUnroutedAddr, false, false, true}, {"routed IPv4", addrInTestPrefixV4, false, true},
{"routed IPv4", addrInTestPrefixV4, false, false, true}, {"routed IPv6", addrInTestPrefixV6, false, true},
{"routed IPv6", addrInTestPrefixV6, false, false, true}, {"private IPv4", "10.0.0.1", false, true},
{"private IPv4", "10.0.0.1", false, false, true}, {"loopback IPv4", "127.0.0.1", false, true},
{"loopback IPv4", "127.0.0.1", false, false, true}, {"public IPv4", "1.1.1.1", false, false},
{"public IPv4", "1.1.1.1", false, false, false}, {"private IPv6", "fd00::1", false, true},
{"private IPv6", "fd00::1", false, false, true}, {"loopback IPv6", "::1", false, true},
{"loopback IPv6", "::1", false, false, true}, {"public IPv6", "2606:4700:4700::1111", false, false},
{"public IPv6", "2606:4700:4700::1111", false, false, false}, {"public IPv4 in IPv6 form", "::ffff:1.1.1.1", false, false},
{"public IPv4 in IPv6 form", "::ffff:1.1.1.1", false, false, false}, {"private IPv4 in IPv6 form", "::ffff:10.0.0.1", false, true},
{"private IPv4 in IPv6 form", "::ffff:10.0.0.1", false, false, true},
} }
for _, tt := range tests { for _, tt := range tests {
@@ -1617,8 +1606,7 @@ func TestIPLookupFlags(t *testing.T) {
} }
want := map[string]bool{ want := map[string]bool{
"is_tor": tt.tor, "is_vpn": false, "is_relay": tt.relay, "is_tor": tt.tor, "is_vpn": false, "is_bogon": tt.bogon,
"is_bogon": tt.bogon,
} }
if !maps.Equal(got, want) { if !maps.Equal(got, want) {
t.Errorf("%s: flags %v, want %v", path, got, want) t.Errorf("%s: flags %v, want %v", path, got, want)
+2 -2
View File
@@ -67,8 +67,8 @@ func TestFooterLicense(t *testing.T) {
} }
} }
// TestHomePageCreditsDBIP expects the home page's footer to carry the credit // TestHomePageCreditsDBIP expects the home page's footer to carry the
// to DB-IP. // attribution the DB-IP licence requires.
func TestHomePageCreditsDBIP(t *testing.T) { func TestHomePageCreditsDBIP(t *testing.T) {
t.Parallel() t.Parallel()