Author SHA1 Message Date
sneak 9aaf877e2f Add the iCloud Private Relay list to the IP lookup, with is_relay (closes #237)
check / check (push) Successful in 1m11s
`icloud_private_relay` is Apple's list of Private Relay networks, downloaded once a day. Every result gains `flags.is_relay`.

`parseProtonVPN` becomes `parseLinesNotEmpty`, shared with the new parser, which reads only each line's first column.

Judgement call: a line with no comma fails the parse, so a plain list of networks at Apple's URL cannot replace a good copy.

Model: opus-5-5
2026-11-10 13:52:01 +00:00
clawbot b2553f044d Add the ProtonVPN server networks and addresses to the IP lookup (closes #236)
check / check (push) Successful in 1m27s
Proton's own server list answers only a logged-in session, so the lists come from the public mirror https://github.com/tn3w/ProtonVPN-IPs, downloaded once a day: `protonvpn`, its /24 networks, and `protonvpn_entry`, its servers' IPv4 and IPv6 addresses. Both feed `flags.is_vpn` and `vpn_providers`.

Judgement call: the /24 networks are listed as well as the exact addresses, because coverage matters more than attribution, so neighbouring addresses in those networks count too.
Risk: the mirror is one person's repository; if it stops updating, downloads keep succeeding with old data.

Model: opus-5-5
2026-11-10 14:14:06 +01:00
clawbot 0cbd9cd943 Resolve PTR names over DNS-over-HTTPS and cache them for the record TTL (closes #238)
check / check (push) Successful in 4m39s
`ptr` keeps its field and shape. It now comes from Google, Cloudflare, AdGuard and Quad9 over RFC 8484, tried in turn within 2 seconds, in place of the host's resolver with no cache. Answers stay in memory and in a new `cache.db` for the record's TTL, at most 250,000 addresses. A reserved address is not sent to any resolver and has an empty `ptr`.

Judgement call: each resolver gets an equal share of the time left, so a hung one does not use up the limit.
Judgement call: the negative TTL is the smaller of the SOA minimum and the SOA's own TTL.
Deviation: the cache is SQLite, by sneak's words, not the JSON file the standing rule asks for.

Model: opus-5-5
2026-11-10 13:48:06 +01:00
clawbot 641798bb22 Make the tests pass on a busy host (closes #240)
check / check (push) Successful in 1m51s
Tests that wait on the real clock failed when the host stalled for a few seconds, although the code was right. Waits for something that should happen are now mostly 15 seconds. Tests whose result depends on a short idle timeout or a production request limit run under `synctest.Test`, on a fake clock, with a fake transport where a real connection would stop that clock. Test code only.

Judgement call: the 5-second waits in three `internal/streamer` tests also became 15 seconds.

Model: opus-5-5
2026-11-10 12:48:07 +01:00
19 changed files with 802 additions and 237 deletions

No files matched your search

+21
View File
@@ -287,6 +287,7 @@ address. Every entry has the same fields:
- `is_tor`: true when the address is on the `tor_exits` list - `is_tor`: true when the address is on the `tor_exits` list
- `is_vpn`: true when the address is on any list of VPN servers (see IP - `is_vpn`: true when the address is on any list of VPN servers (see IP
lists) lists)
- `is_relay`: true when the address is on the `icloud_private_relay` list
- `is_bogon`: true when the address is reserved, so that no network should - `is_bogon`: true when the address is reserved, so that no network should
announce it on the internet announce it on the internet
- `vpn_providers`: the names of the lists of VPN servers the address is on, or - `vpn_providers`: the names of the lists of VPN servers the address is on, or
@@ -421,6 +422,9 @@ The lists:
last 30 days. From last 30 days. From
[FireHOL](https://iplists.firehol.org/?ipset=firehol_level3), downloaded every [FireHOL](https://iplists.firehol.org/?ipset=firehol_level3), downloaded every
3 hours. 3 hours.
- `icloud_private_relay`: the networks where iCloud Private Relay traffic
leaves. From [Apple](https://mask-api.icloud.com/egress-ip-ranges.csv),
downloaded once a day.
- `ivpn`: VPN servers run by IVPN. From - `ivpn`: VPN servers run by IVPN. From
[IVPN](https://api.ivpn.net/v5/servers.json), downloaded once a day. [IVPN](https://api.ivpn.net/v5/servers.json), downloaded once a day.
- `mullvad`: VPN servers run by Mullvad. From - `mullvad`: VPN servers run by Mullvad. From
@@ -431,6 +435,14 @@ The lists:
- `pia`: VPN servers run by Private Internet Access. From - `pia`: VPN servers run by Private Internet Access. From
[Private Internet Access](https://serverlist.piaservers.net/vpninfo/servers/v6), [Private Internet Access](https://serverlist.piaservers.net/vpninfo/servers/v6),
downloaded once a day. downloaded once a day.
- `protonvpn`: the /24 networks that hold ProtonVPN servers. The whole /24 is
listed because the ProtonVPN-IPs README says traffic leaves from unpublished
addresses in it. From the public
[ProtonVPN-IPs](https://github.com/tn3w/ProtonVPN-IPs) mirror of Proton's
server list, downloaded once a day.
- `protonvpn_entry`: the IPv4 and IPv6 addresses of ProtonVPN servers. From the
same [ProtonVPN-IPs](https://github.com/tn3w/ProtonVPN-IPs) mirror, downloaded
once a day.
- `spamhaus_drop` and `spamhaus_dropv6`: IPv4 and IPv6 networks hijacked or - `spamhaus_drop` and `spamhaus_dropv6`: IPv4 and IPv6 networks hijacked or
leased by spammers or cyber criminals, according to Spamhaus. From leased by spammers or cyber criminals, according to Spamhaus. From
[Spamhaus DROP](https://www.spamhaus.org/blocklists/do-not-route-or-peer/), [Spamhaus DROP](https://www.spamhaus.org/blocklists/do-not-route-or-peer/),
@@ -498,6 +510,8 @@ How RouteWatch keeps the answers:
whose SOA record has a TTL or minimum field of 0. whose SOA record has a TTL or minimum field of 0.
- A failed lookup is not kept, so the next lookup of the address asks again. - A failed lookup is not kept, so the next lookup of the address asks again.
- Every 10 minutes it deletes expired answers from memory and from `cache.db`. - Every 10 minutes it deletes expired answers from memory and from `cache.db`.
- It keeps answers for at most 250,000 addresses, and keeps no answer for a new
address while it holds that many.
## Configuration ## Configuration
@@ -596,6 +610,13 @@ The ceilings add up to 3.7 GiB:
limit caps the SQLite library. limit caps the SQLite library.
- About 0.2 GiB for everything else in the runtime. - About 0.2 GiB for everything else in the runtime.
The `icloud_private_relay` list has about 285,000 networks. Neighbouring
networks merge, which leaves about 6,900 address ranges. Loaded, the list adds
about 0.3 MiB to the Go heap and under 6 MiB to the process's resident memory.
Parsing the file allocates about 94 MiB in about 0.2 seconds. The next garbage
collection frees all of it but the 0.3 MiB above. Parsing happens at startup and
whenever a download brings a new version.
The SQLite library gets its memory from glibc, the C library. glibc keeps freed The SQLite library gets its memory from glibc, the C library. glibc keeps freed
memory in pools called arenas, and hands it back to the kernel slowly. It memory in pools called arenas, and hands it back to the kernel slowly. It
creates up to eight arenas per processor core. So on a host with many cores the creates up to eight arenas per processor core. So on a host with many cores the
+10
View File
@@ -53,9 +53,19 @@ problem the run finds.
# Completed Steps # Completed Steps
- 2026-11-10: the IP lookup checks the `icloud_private_relay` list of Apple's
iCloud Private Relay networks, downloaded once a day. Each result has
`flags.is_relay` (closes https://git.eeqj.de/sneak/routewatch/issues/237).
- 2026-11-10: the IP lookup checks the `protonvpn` list of ProtonVPN's /24
networks and the `protonvpn_entry` list of its servers' IPv4 and IPv6
addresses, from a public mirror of Proton's server list (closes
https://git.eeqj.de/sneak/routewatch/issues/236).
- 2026-11-10: the IP lookup finds PTR names over DNS-over-HTTPS and caches them - 2026-11-10: the IP lookup finds PTR names over DNS-over-HTTPS and caches them
for the record's TTL, in memory and in `cache.db` (closes for the record's TTL, in memory and in `cache.db` (closes
https://git.eeqj.de/sneak/routewatch/issues/238). https://git.eeqj.de/sneak/routewatch/issues/238).
- 2026-11-10: the tests no longer fail when a busy host stalls for several
seconds. Most such waits are now 15 seconds. The test of the stream's idle
timeout runs on a fake clock (closes #240)
- 2026-11-10: the IP lookup checks the public VPN server lists of Mullvad, - 2026-11-10: the IP lookup checks the public VPN server lists of Mullvad,
NordVPN, Private Internet Access and IVPN, and X4BNet's community list. Each NordVPN, Private Internet Access and IVPN, and X4BNet's community list. Each
result has `flags.is_vpn` and `vpn_providers` (closes result has `flags.is_vpn` and `vpn_providers` (closes
+1 -1
View File
@@ -12,7 +12,7 @@ import (
// parseTimeLimit is how long a test waits for parseCSV to return before it // parseTimeLimit is how long a test waits for parseCSV to return before it
// fails. // fails.
const parseTimeLimit = 2 * time.Second const parseTimeLimit = 15 * time.Second
// TestParseCSVReadsTodaysLayout parses an as.csv with the country code column // TestParseCSVReadsTodaysLayout parses an as.csv with the country code column
// that the source added after the description. It expects each AS with its // that the source added after the description. It expects each AS with its
+25 -5
View File
@@ -72,17 +72,34 @@ const (
// lockTestPrefix is the prefix the batch writes in the checkpoint tests // lockTestPrefix is the prefix the batch writes in the checkpoint tests
// store. // store.
lockTestPrefix = "192.0.2.0/24" lockTestPrefix = "192.0.2.0/24"
// longBusyTimeoutMs is the busy timeout setLongBusyTimeout gives the
// writing connection. A checkpoint that waits for an open read takes this
// long.
longBusyTimeoutMs = 30000
// readerCheckpointLimit is how long a checkpoint, and a batch write // readerCheckpointLimit is how long a checkpoint, and a batch write
// waiting for it, may take while a reader is open: well under the busy // waiting for it, may take while a reader is open: well under
// timeout, which a checkpoint that waits for the reader takes. // longBusyTimeoutMs, with room for a stall on a busy host.
readerCheckpointLimit = time.Second readerCheckpointLimit = 15 * time.Second
// truncateCheckpointLimit is how long a checkpoint of a WAL file over // truncateCheckpointLimit is how long a checkpoint of a WAL file over
// sqliteWALSizeLimitBytes may take to give up while a reader is open: its // sqliteWALSizeLimitBytes may take to give up while a reader is open: its
// wait for the reader plus a second, well under the busy timeout. // wait for the reader plus readerCheckpointLimit.
truncateCheckpointLimit = truncateBusyTimeoutMs*time.Millisecond + truncateCheckpointLimit = truncateBusyTimeoutMs*time.Millisecond +
time.Second readerCheckpointLimit
) )
// setLongBusyTimeout gives the writing connection of db longBusyTimeoutMs, so
// that a checkpoint that waits for an open read takes far longer than a stall
// on a busy host.
func setLongBusyTimeout(ctx context.Context, t *testing.T, db *Database) {
t.Helper()
_, err := db.writer.ExecContext(ctx,
fmt.Sprintf("PRAGMA busy_timeout=%d", longBusyTimeoutMs))
if err != nil {
t.Fatalf("failed to set the busy timeout: %v", err)
}
}
// TestConnectionPoolPragmas holds several pooled connections open at once and // TestConnectionPoolPragmas holds several pooled connections open at once and
// checks each one carries the per-connection settings from the DSN, plus the // checks each one carries the per-connection settings from the DSN, plus the
// process-wide heap limits, and sees the new file with auto_vacuum // process-wide heap limits, and sees the new file with auto_vacuum
@@ -579,6 +596,8 @@ func TestCheckpointDoesNotWaitForReader(t *testing.T) {
ctx := context.Background() ctx := context.Background()
ts := time.Now().UTC() ts := time.Now().UTC()
setLongBusyTimeout(ctx, t, db)
err = db.UpdatePrefixesBatch(map[string]time.Time{"198.51.100.0/24": ts}) err = db.UpdatePrefixesBatch(map[string]time.Time{"198.51.100.0/24": ts})
if err != nil { if err != nil {
t.Fatalf("failed to store the prefix the reader reads: %v", err) t.Fatalf("failed to store the prefix the reader reads: %v", err)
@@ -690,6 +709,7 @@ func TestCheckpointEmptiesWALOnceNoReadIsOpen(t *testing.T) {
} }
fillWALPastLimit(ctx, t, db) fillWALPastLimit(ctx, t, db)
setLongBusyTimeout(ctx, t, db)
started := time.Now() started := time.Now()
+1 -1
View File
@@ -21,7 +21,7 @@ const (
downloadTimeout = time.Minute downloadTimeout = time.Minute
// maxDownloadSize is the size of the largest list file accepted, in // maxDownloadSize is the size of the largest list file accepted, in
// bytes. The largest list, FireHOL abusers, is about 2.5 MiB. // bytes. The largest list, iCloud Private Relay, is about 12 MiB.
maxDownloadSize = 200 << 20 maxDownloadSize = 200 << 20
) )
+10
View File
@@ -51,6 +51,16 @@ const (
// FormatIVPNJSON is IVPN's JSON server list. The hosts of its wireguard // FormatIVPNJSON is IVPN's JSON server list. The hosts of its wireguard
// and openvpn gateways have their addresses in host and v2ray. // and openvpn gateways have their addresses in host and v2ray.
FormatIVPNJSON FormatIVPNJSON
// FormatProtonVPN has one address or network per line, with comment lines
// that start with #. Unlike FormatLines, a file with no addresses or
// networks fails.
FormatProtonVPN
// FormatICloudPrivateRelayCSV is Apple's CSV file of iCloud Private Relay
// networks. Each line starts with a network, followed by its country,
// region and city.
FormatICloudPrivateRelayCSV
) )
// List is one public IP list. // List is one public IP list.
+140 -14
View File
@@ -70,6 +70,31 @@ const (
"config":{"api":{"ips":["192.0.2.73"]}}}` "config":{"api":{"ips":["192.0.2.73"]}}}`
) )
// protonVPNBody is a small file in the format of the ProtonVPN-IPs list.
const protonVPNBody = "#\n# protonvpn_ip_ranges.txt\n" +
"# https://github.com/tn3w/ProtonVPN-IPs\n#\n" +
"# The /24 block of every advertised IPv4.\n#\n" +
"192.0.2.0/24\n198.51.100.0/24\n"
// protonVPNEntryBody is a small file in the format of the ProtonVPN-IPs list
// of server addresses.
const protonVPNEntryBody = "#\n# protonvpn_entry_ips.txt\n" +
"# https://github.com/tn3w/ProtonVPN-IPs\n#\n" +
"# Entry IPs clients connect to, resolved from ProtonVPN hostnames.\n#\n" +
"192.0.2.80\n2001:db8::80\n"
// iCloudPrivateRelayAddr is an address on iCloudPrivateRelayBody.
const iCloudPrivateRelayAddr = "2001:db8:1::5"
// iCloudPrivateRelayBody is a small file in the format of Apple's iCloud
// Private Relay file.
const iCloudPrivateRelayBody = "192.0.2.0/31,GB,GB-EN,London,\n" +
"198.51.100.32/27,US,US-CA,Los Angeles,\n" +
"2001:db8:1::/64,US,US-AK,Anchorage,\n"
// emptyFileCase names the case of a wrong document that is an empty file.
const emptyFileCase = "an empty file"
// errorPage is a file that is no list, as a publisher's error page is. // errorPage is a file that is no list, as a publisher's error page is.
const errorPage = "<!DOCTYPE html>\n<html></html>\n" const errorPage = "<!DOCTYPE html>\n<html></html>\n"
@@ -230,6 +255,15 @@ func TestLookupFindsListedAddresses(t *testing.T) {
}, },
unlisted: []string{"192.0.2.12", "2001:db8::2"}, unlisted: []string{"192.0.2.12", "2001:db8::2"},
}, },
{
name: "iCloud Private Relay networks",
format: FormatICloudPrivateRelayCSV,
body: iCloudPrivateRelayBody,
listed: []string{
"192.0.2.1", "198.51.100.63", iCloudPrivateRelayAddr,
},
unlisted: []string{"192.0.2.2", "198.51.100.64", "2001:db8:2::1"},
},
}) })
} }
@@ -267,6 +301,20 @@ func TestLookupFindsVPNServers(t *testing.T) {
listed: []string{"192.0.2.70", "192.0.2.71", "192.0.2.72"}, listed: []string{"192.0.2.70", "192.0.2.71", "192.0.2.72"},
unlisted: []string{"192.0.2.73", "2001:db8::70"}, unlisted: []string{"192.0.2.73", "2001:db8::70"},
}, },
{
name: "ProtonVPN networks",
format: FormatProtonVPN,
body: protonVPNBody,
listed: []string{"192.0.2.0", "192.0.2.255", "198.51.100.7"},
unlisted: []string{"192.0.3.1", "2001:db8::90"},
},
{
name: "ProtonVPN server addresses",
format: FormatProtonVPN,
body: protonVPNEntryBody,
listed: []string{"192.0.2.80", "2001:db8::80"},
unlisted: []string{"192.0.2.81", "2001:db8::81"},
},
}) })
} }
@@ -458,12 +506,12 @@ func TestWrongVPNDocumentFailsToParse(t *testing.T) {
for _, f := range formats { for _, f := range formats {
bodies := map[string]string{ bodies := map[string]string{
"not JSON": errorPage, "not JSON": errorPage,
"other JSON": `{"status":"ok","data":{"count":1}}`, "other JSON": `{"status":"ok","data":{"count":1}}`,
"no addresses": f.empty, "no addresses": f.empty,
"bad address": f.badAddress, "bad address": f.badAddress,
"cut off": f.good[:len(f.good)/2], "cut off": f.good[:len(f.good)/2],
"an empty file": "", emptyFileCase: "",
} }
for name, body := range bodies { for name, body := range bodies {
@@ -479,6 +527,59 @@ func TestWrongVPNDocumentFailsToParse(t *testing.T) {
} }
} }
// TestWrongProtonVPNDocumentFailsToParse parses documents that are not a
// ProtonVPN-IPs list. Each fails, so none can replace a good copy.
func TestWrongProtonVPNDocumentFailsToParse(t *testing.T) {
t.Parallel()
bodies := map[string]string{
"error page": errorPage,
"JSON": `{"status":"ok"}`,
"comments only": "#\n# protonvpn_ip_ranges.txt\n#\n",
"bad network": "192.0.2.0/24\nexample\n",
emptyFileCase: "",
}
for name, body := range bodies {
t.Run(name, func(t *testing.T) {
t.Parallel()
addresses, err := parse(strings.NewReader(body), FormatProtonVPN)
if err == nil {
t.Errorf("parse returned %v and no error", addresses.Ranges())
}
})
}
}
// TestWrongICloudPrivateRelayDocumentFailsToParse parses documents that are
// not Apple's iCloud Private Relay file. Each fails, so none can replace a
// good copy.
func TestWrongICloudPrivateRelayDocumentFailsToParse(t *testing.T) {
t.Parallel()
bodies := map[string]string{
"error page": errorPage,
"JSON": `{"status":"ok","count":1}`,
"no other field": "192.0.2.0/24\n",
"bad network": "192.0.2.0/24,GB,GB-EN,London,\nexample,GB,GB-EN,London,\n",
"an address": "192.0.2.1,GB,GB-EN,London,\n",
emptyFileCase: "",
}
for name, body := range bodies {
t.Run(name, func(t *testing.T) {
t.Parallel()
addresses, err := parse(strings.NewReader(body),
FormatICloudPrivateRelayCSV)
if err == nil {
t.Errorf("parse returned %v and no error", addresses.Ranges())
}
})
}
}
// TestLookupMarksVPNLists looks up an address on a VPN list and on another // TestLookupMarksVPNLists looks up an address on a VPN list and on another
// list. Only the VPN list's listing says it is one. // list. Only the VPN list's listing says it is one.
func TestLookupMarksVPNLists(t *testing.T) { func TestLookupMarksVPNLists(t *testing.T) {
@@ -523,7 +624,10 @@ func TestPublicListsMarkTheVPNLists(t *testing.T) {
} }
} }
want := []string{"ivpn", "mullvad", "nordvpn", "pia", "x4bnet", "x4bnet_v6"} want := []string{
"ivpn", "mullvad", "nordvpn", "pia", "protonvpn", "protonvpn_entry",
"x4bnet", "x4bnet_v6",
}
if !slices.Equal(got, want) { if !slices.Equal(got, want) {
t.Errorf("VPN lists %v, want %v", got, want) t.Errorf("VPN lists %v, want %v", got, want)
} }
@@ -646,9 +750,13 @@ func TestRestartLoadsCopiesFromDisk(t *testing.T) {
second.Start(t.Context()) second.Start(t.Context())
second.Stop(t.Context()) second.Stop(t.Context())
// The download takes its time after writing the file, and the restart
// reads the file's time. A busy host can put seconds between the two.
const maxUpdatedDifference = 15 * time.Second
listings := lookup(t, second, listedAddr) listings := lookup(t, second, listedAddr)
if len(listings) != 1 || if len(listings) != 1 ||
listings[0].Updated.Sub(downloaded).Abs() > time.Second { listings[0].Updated.Sub(downloaded).Abs() > maxUpdatedDifference {
t.Errorf("after the restart, listings %+v, want one updated at %s", t.Errorf("after the restart, listings %+v, want one updated at %s",
listings, downloaded) listings, downloaded)
} }
@@ -826,15 +934,33 @@ func TestStopCancelsADownloadInProgress(t *testing.T) {
func TestTorExitsListHoldsIPv6Exits(t *testing.T) { func TestTorExitsListHoldsIPv6Exits(t *testing.T) {
t.Parallel() t.Parallel()
checkPublicListHolds(t, TorExitsList, onionooBody, onionooIPv6ExitAddr)
}
// TestICloudPrivateRelayListHoldsItsNetworks downloads a file in Apple's
// format as the public iCloud Private Relay list, which is what an address's
// is_relay answer comes from.
func TestICloudPrivateRelayListHoldsItsNetworks(t *testing.T) {
t.Parallel()
checkPublicListHolds(t, ICloudPrivateRelayList, iCloudPrivateRelayBody,
iCloudPrivateRelayAddr)
}
// checkPublicListHolds downloads body as the public list called name and
// checks that addr has one listing, which names that list.
func checkPublicListHolds(t *testing.T, name, body, addr string) {
t.Helper()
index := slices.IndexFunc(Lists(), func(list List) bool { index := slices.IndexFunc(Lists(), func(list List) bool {
return list.Name == TorExitsList return list.Name == name
}) })
if index < 0 { if index < 0 {
t.Fatalf("no list named %s", TorExitsList) t.Fatalf("no list named %s", name)
} }
list := Lists()[index] list := Lists()[index]
_, list.URL = newPublisher(t, onionooBody) _, list.URL = newPublisher(t, body)
b := newTestIPLists(t.TempDir(), list) b := newTestIPLists(t.TempDir(), list)
err := b.download(t.Context(), list) err := b.download(t.Context(), list)
@@ -842,9 +968,9 @@ func TestTorExitsListHoldsIPv6Exits(t *testing.T) {
t.Fatalf("download: %v", err) t.Fatalf("download: %v", err)
} }
listings := lookup(t, b, onionooIPv6ExitAddr) listings := lookup(t, b, addr)
if len(listings) != 1 || listings[0].Name != TorExitsList { if len(listings) != 1 || listings[0].Name != name {
t.Errorf("listings %+v, want one for %s", listings, TorExitsList) t.Errorf("listings %+v, want one for %s", listings, name)
} }
} }
+39
View File
@@ -8,6 +8,13 @@ import (
// TorExitsList is the name of the list of Tor exit nodes. // TorExitsList is the name of the list of Tor exit nodes.
const TorExitsList = "tor_exits" const TorExitsList = "tor_exits"
// ICloudPrivateRelayList is the name of the list of iCloud Private Relay's
// networks.
const ICloudPrivateRelayList = "icloud_private_relay"
// protonVPNIPsURL is where the ProtonVPN-IPs mirror publishes its files.
const protonVPNIPsURL = "https://raw.githubusercontent.com/tn3w/ProtonVPN-IPs/master/"
// How often each list is downloaded. No list is downloaded more often than // How often each list is downloaded. No list is downloaded more often than
// its publisher updates it or allows. // its publisher updates it or allows.
const ( const (
@@ -27,6 +34,9 @@ const (
fireHOLAbusersInterval = 5 * time.Hour fireHOLAbusersInterval = 5 * time.Hour
fireHOLLevel3Interval = 3 * time.Hour fireHOLLevel3Interval = 3 * time.Hour
// Apple's iCloud Private Relay file is 12 MB, so once a day.
iCloudPrivateRelayInterval = 24 * time.Hour
// Spamhaus asks that DROP be downloaded at most once a day: // Spamhaus asks that DROP be downloaded at most once a day:
// https://www.spamhaus.org/blocklists/do-not-route-or-peer/ // https://www.spamhaus.org/blocklists/do-not-route-or-peer/
spamhausDROPInterval = 24 * time.Hour spamhausDROPInterval = 24 * time.Hour
@@ -81,6 +91,13 @@ func nonVPNLists() []List {
Interval: fireHOLLevel3Interval, Interval: fireHOLLevel3Interval,
Format: FormatLines, Format: FormatLines,
}, },
{
Name: ICloudPrivateRelayList,
Description: "In a network where iCloud Private Relay traffic leaves",
URL: "https://mask-api.icloud.com/egress-ip-ranges.csv",
Interval: iCloudPrivateRelayInterval,
Format: FormatICloudPrivateRelayCSV,
},
{ {
Name: "spamhaus_drop", Name: "spamhaus_drop",
Description: "In an IPv4 network that Spamhaus says spammers " + Description: "In an IPv4 network that Spamhaus says spammers " +
@@ -146,6 +163,28 @@ func vpnLists() []List {
Format: FormatPIAJSON, Format: FormatPIAJSON,
VPN: true, VPN: true,
}, },
{
// Proton's own server list needs a logged-in session, so this is
// the public ProtonVPN-IPs mirror of it. Proton publishes only
// the addresses clients connect to. The mirror's README says
// traffic leaves from others in the same /24, so this file of
// /24 networks is the one it says to use for detecting exits.
Name: "protonvpn",
Description: "In a /24 network that holds a server of the " +
"ProtonVPN service",
URL: protonVPNIPsURL + "protonvpn_ip_ranges.txt",
Interval: vpnListInterval,
Format: FormatProtonVPN,
VPN: true,
},
{
Name: "protonvpn_entry",
Description: "A server of the ProtonVPN service",
URL: protonVPNIPsURL + "protonvpn_entry_ips.txt",
Interval: vpnListInterval,
Format: FormatProtonVPN,
VPN: true,
},
{ {
Name: "x4bnet", Name: "x4bnet",
Description: "In an IPv4 network of a VPN service, according to " + Description: "In an IPv4 network of a VPN service, according to " +
+43
View File
@@ -25,6 +25,10 @@ const (
// dshieldFieldSeparator separates the fields of a DShield line. // dshieldFieldSeparator separates the fields of a DShield line.
dshieldFieldSeparator = "\t" dshieldFieldSeparator = "\t"
// iCloudPrivateRelayFieldSeparator separates the fields of a line of
// Apple's iCloud Private Relay file.
iCloudPrivateRelayFieldSeparator = ","
// spamhausMetadataType is the type of the line that ends a Spamhaus // spamhausMetadataType is the type of the line that ends a Spamhaus
// file. That line holds the file's metadata and no network. // file. That line holds the file's metadata and no network.
spamhausMetadataType = "metadata" spamhausMetadataType = "metadata"
@@ -58,6 +62,10 @@ func parse(r io.Reader, format Format) (*netipx.IPSet, error) {
return parsePIA(r) return parsePIA(r)
case FormatIVPNJSON: case FormatIVPNJSON:
return parseIVPN(r) return parseIVPN(r)
case FormatProtonVPN:
return parseLinesNotEmpty(r, addAddressOrNetwork)
case FormatICloudPrivateRelayCSV:
return parseLinesNotEmpty(r, addICloudPrivateRelayNetwork)
default: default:
return nil, errUnknownFormat return nil, errUnknownFormat
} }
@@ -329,6 +337,41 @@ func parseIVPN(r io.Reader) (*netipx.IPSet, error) {
return addressSet(texts) return addressSet(texts)
} }
// parseLinesNotEmpty is parseLines for a list that always holds addresses, so
// a file with none fails.
func parseLinesNotEmpty(
r io.Reader, addLine func(*netipx.IPSetBuilder, string) error,
) (*netipx.IPSet, error) {
addresses, err := parseLines(r, addLine)
if err != nil {
return nil, err
}
if len(addresses.Ranges()) == 0 {
return nil, errNoAddresses
}
return addresses, nil
}
// addICloudPrivateRelayNetwork adds the network at the start of a line of
// Apple's iCloud Private Relay file to builder.
func addICloudPrivateRelayNetwork(builder *netipx.IPSetBuilder, line string) error {
field, _, found := strings.Cut(line, iCloudPrivateRelayFieldSeparator)
if !found {
return errNoNetwork
}
network, err := netip.ParsePrefix(field)
if err != nil {
return err
}
builder.AddPrefix(network)
return nil
}
// readJSON reads r to its end and decodes it, as one JSON document, into v. // readJSON reads r to its end and decodes it, as one JSON document, into v.
func readJSON(r io.Reader, v any) error { func readJSON(r io.Reader, v any) error {
data, err := io.ReadAll(r) data, err := io.ReadAll(r)
+33 -20
View File
@@ -1,6 +1,6 @@
// Package ptr looks up the PTR names of addresses over DNS-over-HTTPS. It // Package ptr looks up the PTR names of addresses over DNS-over-HTTPS. It
// keeps each answer in memory for as long as DNS allows, and in cache.db so // keeps answers in memory for as long as DNS allows, and in cache.db so that
// that the answers survive a restart. // the answers survive a restart.
package ptr package ptr
import ( import (
@@ -29,6 +29,10 @@ const (
// nameSeparator separates the names in a row of the hostnames table. // nameSeparator separates the names in a row of the hostnames table.
nameSeparator = " " nameSeparator = " "
// maxAddresses limits how many addresses the cache holds, so that a public
// lookup cannot grow memory and cache.db without bound.
maxAddresses = 250_000
) )
// Params is what New needs. // Params is what New needs.
@@ -52,15 +56,16 @@ type entry struct {
// Cache answers PTR lookups from memory, and asks the resolvers for an address // Cache answers PTR lookups from memory, and asks the resolvers for an address
// it holds no current answer for. Start loads it from cache.db. // it holds no current answer for. Start loads it from cache.db.
type Cache struct { type Cache struct {
db *sql.DB db *sql.DB
resolvers []string resolvers []string
timeLimit time.Duration timeLimit time.Duration
logger *slog.Logger maxEntries int
client *http.Client logger *slog.Logger
now func() time.Time client *http.Client
now func() time.Time
// entries holds every answer by address. pending holds those that // entries holds the kept answers by address, at most maxEntries of them.
// cache.db does not have yet. // pending holds those that cache.db does not have yet.
mu sync.RWMutex mu sync.RWMutex
entries map[string]entry entries map[string]entry
pending map[string]entry pending map[string]entry
@@ -73,15 +78,16 @@ type Cache struct {
// New returns an empty Cache. Start loads it. // New returns an empty Cache. Start loads it.
func New(params Params) *Cache { func New(params Params) *Cache {
return &Cache{ return &Cache{
db: params.DB, db: params.DB,
resolvers: params.Resolvers, resolvers: params.Resolvers,
timeLimit: LookupTimeout, timeLimit: LookupTimeout,
logger: params.Logger.With("component", "ptr"), maxEntries: maxAddresses,
client: &http.Client{}, logger: params.Logger.With("component", "ptr"),
now: time.Now, client: &http.Client{},
entries: make(map[string]entry), now: time.Now,
pending: make(map[string]entry), entries: make(map[string]entry),
done: make(chan struct{}), pending: make(map[string]entry),
done: make(chan struct{}),
} }
} }
@@ -168,11 +174,18 @@ func (c *Cache) current(address string) (entry, bool) {
return found, true return found, true
} }
// store keeps an answer in memory and queues it for cache.db. // store keeps an answer in memory and queues it for cache.db. Once entries
// holds maxEntries addresses it keeps no new address. It still replaces the
// answer of an address already held.
func (c *Cache) store(address string, answer entry) { func (c *Cache) store(address string, answer entry) {
c.mu.Lock() c.mu.Lock()
defer c.mu.Unlock() defer c.mu.Unlock()
_, held := c.entries[address]
if !held && len(c.entries) >= c.maxEntries {
return
}
c.entries[address] = answer c.entries[address] = answer
c.pending[address] = answer c.pending[address] = answer
} }
+69
View File
@@ -880,3 +880,72 @@ func TestFlushKeepsAnswersItCouldNotWrite(t *testing.T) {
t.Errorf("%d rows after the second write, want 1", got) t.Errorf("%d rows after the second write, want 1", got)
} }
} }
// TestNewAddressIsNotKeptAtTheLimit fills a cache that holds one address. A
// lookup of a second address still returns its answer, and the cache keeps
// nothing for it until the sweep has deleted the first address's expired
// answer.
func TestNewAddressIsNotKeptAtTheLimit(t *testing.T) {
t.Parallel()
db := openTestDB(t, t.TempDir())
resolver := newFakeResolver(t, ptrReply(300, testName))
cache, clock := newTestCache(t, db, resolver)
cache.maxEntries = 1
lookupOK(t, cache, testAddress)
names := lookupOK(t, cache, otherAddress)
if !slices.Equal(names, []string{testName}) {
t.Errorf("names %v at the limit, want %s", names, testName)
}
_, inEntries := cache.entries[otherAddress]
_, inPending := cache.pending[otherAddress]
if inEntries || inPending {
t.Errorf("a new address was kept at the limit: in entries %t, in pending %t",
inEntries, inPending)
}
cache.flush(t.Context())
if got := countRows(t, db); got != 1 {
t.Errorf("%d rows at the limit, want 1", got)
}
clock.advance(300 * time.Second)
cache.sweep(t.Context())
lookupOK(t, cache, otherAddress)
_, inEntries = cache.entries[otherAddress]
if !inEntries {
t.Error("a new address was not kept after the sweep made room")
}
}
// TestHeldAddressIsReplacedAtTheLimit lets the answer of the only address a
// cache holds expire. The new answer replaces it in memory and is queued for
// cache.db.
func TestHeldAddressIsReplacedAtTheLimit(t *testing.T) {
t.Parallel()
resolver := newFakeResolver(t, ptrReply(300, testName))
cache, clock := newTestCache(t, openTestDB(t, t.TempDir()), resolver)
cache.maxEntries = 1
lookupOK(t, cache, testAddress)
clock.advance(300 * time.Second)
resolver.setReply(ptrReply(300, otherName))
lookupOK(t, cache, testAddress)
want := []string{otherName}
if got := cache.entries[testAddress].names; !slices.Equal(got, want) {
t.Errorf("entries holds %v, want %v", got, want)
}
if got := cache.pending[testAddress].names; !slices.Equal(got, want) {
t.Errorf("pending holds %v, want %v", got, want)
}
}
@@ -8,6 +8,8 @@ import (
"net/http" "net/http"
"net/http/httptest" "net/http/httptest"
"net/netip" "net/netip"
"os"
"path/filepath"
"slices" "slices"
"strconv" "strconv"
"strings" "strings"
@@ -783,3 +785,74 @@ func TestShutdownCheckpointsAndClosesTheDatabaseLast(t *testing.T) {
t.Errorf("Shutdown made the calls %q, want %q", store.calls, want) t.Errorf("Shutdown made the calls %q, want %q", store.calls, want)
} }
} }
// TestPTRCacheAnswersTheIPLookup starts the PTR cache and the HTTP server as
// Run does, and looks up a reserved address, which asks no resolver. Without
// the PTR cache the server has no PTR resolver, and every IP lookup answers
// 500.
func TestPTRCacheAnswersTheIPLookup(t *testing.T) {
t.Parallel()
store := newMockStore()
log := logger.New()
port := freePort(t)
cfg := &config.Config{Port: port, StateDir: t.TempDir()}
s := streamer.New(log, metrics.New())
// A started server reads its routing table in the background, so it
// needs one.
rw := New(Dependencies{
DB: store,
Streamer: s,
Server: server.New(store, routetable.New(routetable.Params{}), s, log,
cfg),
Logger: log,
Config: cfg,
})
err := rw.startPTRCache(t.Context())
if err != nil {
t.Fatalf("failed to start the PTR cache: %v", err)
}
defer rw.stopPTRCache(t.Context())
err = rw.server.Start(t.Context())
if err != nil {
t.Fatalf("failed to start the HTTP server: %v", err)
}
defer func() { _ = rw.server.Stop(t.Context()) }()
req, err := http.NewRequestWithContext(t.Context(), http.MethodGet,
"http://"+net.JoinHostPort("127.0.0.1", strconv.Itoa(port))+
"/api/v1/ip/10.0.0.1", nil)
if err != nil {
t.Fatalf("failed to build the request: %v", err)
}
resp, err := http.DefaultClient.Do(req)
if err != nil {
t.Fatalf("the IP lookup failed: %v", err)
}
defer func() { _ = resp.Body.Close() }()
body, err := io.ReadAll(resp.Body)
if err != nil {
t.Fatalf("failed to read the answer: %v", err)
}
if resp.StatusCode != http.StatusOK {
t.Fatalf("status %d, want %d; body %s", resp.StatusCode, http.StatusOK, body)
}
if !strings.Contains(string(body), `"ptr":[]`) {
t.Errorf("answer %s, want an empty ptr", body)
}
_, err = os.Stat(filepath.Join(cfg.StateDir, "cache.db"))
if err != nil {
t.Errorf("cache.db is not in the state directory: %v", err)
}
}
+6
View File
@@ -797,6 +797,9 @@ type IPLookupFlags struct {
// IsVPN says that the address is on a list of VPN servers. // IsVPN says that the address is on a list of VPN servers.
IsVPN bool `json:"is_vpn"` IsVPN bool `json:"is_vpn"`
// IsRelay says that the address is on the iCloud Private Relay list.
IsRelay bool `json:"is_relay"`
// IsBogon says that the address is reserved, as package bogon tells. // IsBogon says that the address is reserved, as package bogon tells.
IsBogon bool `json:"is_bogon"` IsBogon bool `json:"is_bogon"`
} }
@@ -1025,6 +1028,9 @@ func ipLookupFlags(ip string, listings []iplist.Listing) IPLookupFlags {
IsVPN: slices.ContainsFunc(listings, func(l iplist.Listing) bool { IsVPN: slices.ContainsFunc(listings, func(l iplist.Listing) bool {
return l.VPN return l.VPN
}), }),
IsRelay: slices.ContainsFunc(listings, func(l iplist.Listing) bool {
return l.Name == iplist.ICloudPrivateRelayList
}),
IsBogon: isBogon(ip), IsBogon: isBogon(ip),
} }
} }
+141 -83
View File
@@ -17,6 +17,7 @@ import (
"strings" "strings"
"sync/atomic" "sync/atomic"
"testing" "testing"
"testing/synctest"
"time" "time"
"git.eeqj.de/sneak/routewatch/internal/config" "git.eeqj.de/sneak/routewatch/internal/config"
@@ -174,6 +175,9 @@ type tableStats struct {
// carried 203.0.113.0/24, the two with kept routes withdrew it, so its origin // carried 203.0.113.0/24, the two with kept routes withdrew it, so its origin
// keeps no route. The oldest and newest routes are those of the other two // keeps no route. The oldest and newest routes are those of the other two
// prefixes. // prefixes.
//
// The requests run in a synctest bubble, whose clock is fake, so a stall on
// the host does not use up the handlers' 4-second limit.
func TestStatsHandlersAnswerFromMemory(t *testing.T) { func TestStatsHandlersAnswerFromMemory(t *testing.T) {
t.Parallel() t.Parallel()
@@ -219,7 +223,11 @@ func TestStatsHandlersAnswerFromMemory(t *testing.T) {
for name, handler := range handlers { for name, handler := range handlers {
rec := httptest.NewRecorder() rec := httptest.NewRecorder()
handler(rec, httptest.NewRequestWithContext(t.Context(), http.MethodGet, "/", nil))
synctest.Test(t, func(t *testing.T) {
handler(rec,
httptest.NewRequestWithContext(t.Context(), http.MethodGet, "/", nil))
})
if rec.Code != http.StatusOK { if rec.Code != http.StatusOK {
t.Errorf("%s: status %d, want %d; body %s", t.Errorf("%s: status %d, want %d; body %s",
@@ -254,15 +262,20 @@ func TestStatsHandlersAnswerFromMemory(t *testing.T) {
// TestStatsAnswerCarriesChurnPerMinute checks that the stream object of the // TestStatsAnswerCarriesChurnPerMinute checks that the stream object of the
// /api/v1/stats answer holds route_churn_per_min as a whole number, and no // /api/v1/stats answer holds route_churn_per_min as a whole number, and no
// longer route_churn_per_sec, the moving average it replaces // longer route_churn_per_sec, the moving average it replaces
// (https://git.eeqj.de/sneak/routewatch/issues/160). // (https://git.eeqj.de/sneak/routewatch/issues/160). The request runs in a
// synctest bubble, whose clock is fake, so a stall on the host does not use up
// the handler's 4-second limit.
func TestStatsAnswerCarriesChurnPerMinute(t *testing.T) { func TestStatsAnswerCarriesChurnPerMinute(t *testing.T) {
t.Parallel() t.Parallel()
s := newServerWithTestRoutes(t) s := newServerWithTestRoutes(t)
rec := httptest.NewRecorder() rec := httptest.NewRecorder()
s.handleStats()(rec,
httptest.NewRequestWithContext(t.Context(), http.MethodGet, statsPath, nil)) synctest.Test(t, func(t *testing.T) {
s.handleStats()(rec,
httptest.NewRequestWithContext(t.Context(), http.MethodGet, statsPath, nil))
})
if rec.Code != http.StatusOK { if rec.Code != http.StatusOK {
t.Fatalf("status %d, want %d; body %s", rec.Code, http.StatusOK, rec.Body) t.Fatalf("status %d, want %d; body %s", rec.Code, http.StatusOK, rec.Body)
@@ -310,18 +323,26 @@ func (heldBackHandler) HeldBackCount() int { return heldBackMess
// TestStatsAnswerGivesHowManyMessagesAHandlerHoldsBack registers a handler // TestStatsAnswerGivesHowManyMessagesAHandlerHoldsBack registers a handler
// that holds back messages. Its figures in /api/v1/stats, which the status // that holds back messages. Its figures in /api/v1/stats, which the status
// page shows, must give how many. // page shows, must give how many. The request runs in a synctest bubble, whose
// clock is fake, so a stall on the host does not use up the handler's 4-second
// limit.
func TestStatsAnswerGivesHowManyMessagesAHandlerHoldsBack(t *testing.T) { func TestStatsAnswerGivesHowManyMessagesAHandlerHoldsBack(t *testing.T) {
t.Parallel() t.Parallel()
s := newServerWithTestRoutes(t) s := newServerWithTestRoutes(t)
s.streamer.RegisterHandler(heldBackHandler{}) s.streamer.RegisterHandler(heldBackHandler{})
var answer []byte
synctest.Test(t, func(t *testing.T) {
answer = get(t, s, statsPath).Body.Bytes()
})
var body struct { var body struct {
Data StatsResponse `json:"data"` Data StatsResponse `json:"data"`
} }
err := json.Unmarshal(get(t, s, statsPath).Body.Bytes(), &body) err := json.Unmarshal(answer, &body)
if err != nil { if err != nil {
t.Fatalf("decoding the answer: %v", err) t.Fatalf("decoding the answer: %v", err)
} }
@@ -365,7 +386,9 @@ type unitStats struct {
// names. They used to be strings with the unit in them, such as "1h18m34s" or // names. They used to be strings with the unit in them, such as "1h18m34s" or
// "1.1 GB" (https://git.eeqj.de/sneak/routewatch/issues/199). The stream has // "1.1 GB" (https://git.eeqj.de/sneak/routewatch/issues/199). The stream has
// never connected, so the uptime is 0. The memory use in /api/v1/stats is the // never connected, so the uptime is 0. The memory use in /api/v1/stats is the
// same number of bytes as its gc.heap_alloc_bytes. // same number of bytes as its gc.heap_alloc_bytes. The requests run in a
// synctest bubble, whose clock is fake, so a stall on the host does not use up
// the handlers' 4-second limit.
func TestStatsAnswersGiveUnitsInFieldNames(t *testing.T) { func TestStatsAnswersGiveUnitsInFieldNames(t *testing.T) {
t.Parallel() t.Parallel()
@@ -375,7 +398,11 @@ func TestStatsAnswersGiveUnitsInFieldNames(t *testing.T) {
s.SetASNFetcher(fixedIntervalASNFetcher{interval: fetchInterval}) s.SetASNFetcher(fixedIntervalASNFetcher{interval: fetchInterval})
for _, path := range []string{statusJSONPath, statsPath} { for _, path := range []string{statusJSONPath, statsPath} {
answer := get(t, s, path).Body.Bytes() var answer []byte
synctest.Test(t, func(t *testing.T) {
answer = get(t, s, path).Body.Bytes()
})
var body struct { var body struct {
Data unitStats `json:"data"` Data unitStats `json:"data"`
@@ -416,82 +443,89 @@ func TestStatsAnswersGiveUnitsInFieldNames(t *testing.T) {
// connection_duration_seconds of /api/v1/stats is null while the stream is not // connection_duration_seconds of /api/v1/stats is null while the stream is not
// connected, and a whole number of seconds once it is. It used to be the // connected, and a whole number of seconds once it is. It used to be the
// string "disconnected", or a Go duration such as "1h18m34s" // string "disconnected", or a Go duration such as "1h18m34s"
// (https://git.eeqj.de/sneak/routewatch/issues/199). // (https://git.eeqj.de/sneak/routewatch/issues/199). The test runs in a
// synctest bubble, whose clock is fake, so a stall on the host does not use up
// the handler's 4-second limit.
func TestStatsAnswerGivesConnectionDurationInSeconds(t *testing.T) { func TestStatsAnswerGivesConnectionDurationInSeconds(t *testing.T) {
t.Parallel() t.Parallel()
s := newServerWithTestRoutes(t) s := newServerWithTestRoutes(t)
connectionDuration := func() json.RawMessage { synctest.Test(t, func(t *testing.T) {
answer := get(t, s, statsPath).Body.Bytes() connectionDuration := func() json.RawMessage {
answer := get(t, s, statsPath).Body.Bytes()
var body struct { var body struct {
Data map[string]json.RawMessage `json:"data"` Data map[string]json.RawMessage `json:"data"`
}
err := json.Unmarshal(answer, &body)
if err != nil {
t.Fatalf("decoding the answer: %v; answer %s", err, answer)
}
duration, ok := body.Data["connection_duration_seconds"]
if !ok {
t.Fatalf("no connection_duration_seconds: %s", answer)
}
return duration
} }
err := json.Unmarshal(answer, &body) disconnected := connectionDuration()
if err != nil { if string(disconnected) != "null" {
t.Fatalf("decoding the answer: %v; answer %s", err, answer) t.Errorf("disconnected: connection_duration_seconds = %s, want null",
disconnected)
} }
duration, ok := body.Data["connection_duration_seconds"] s.streamer.GetMetricsTracker().SetConnected(true)
if !ok {
t.Fatalf("no connection_duration_seconds: %s", answer) var seconds int64
connected := connectionDuration()
err := json.Unmarshal(connected, &seconds)
if err != nil || seconds != 0 {
t.Errorf("just connected: connection_duration_seconds = %s, want 0",
connected)
} }
})
return duration
}
disconnected := connectionDuration()
if string(disconnected) != "null" {
t.Errorf("disconnected: connection_duration_seconds = %s, want null",
disconnected)
}
s.streamer.GetMetricsTracker().SetConnected(true)
var seconds int64
connected := connectionDuration()
err := json.Unmarshal(connected, &seconds)
if err != nil || seconds != 0 {
t.Errorf("just connected: connection_duration_seconds = %s, want 0",
connected)
}
} }
// TestStreamDurationsCountSeconds checks that the uptime and the connection // TestStreamDurationsCountSeconds checks that the uptime and the connection
// duration of a stream that connected 90 seconds ago are both 90. A stream // duration of a stream that connected 90 seconds ago are both 90. A stream
// that has never connected has an uptime of 0, and one that is not connected // that has never connected has an uptime of 0, and one that is not connected
// has no connection duration. // has no connection duration. The test runs in a synctest bubble, whose clock
// is fake, so a stall on the host does not add to the 90 seconds.
func TestStreamDurationsCountSeconds(t *testing.T) { func TestStreamDurationsCountSeconds(t *testing.T) {
t.Parallel() t.Parallel()
const connectedFor = 90 * time.Second synctest.Test(t, func(t *testing.T) {
const connectedFor = 90 * time.Second
connectedSince := time.Now().Add(-connectedFor) connectedSince := time.Now().Add(-connectedFor)
want := int64(connectedFor / time.Second) want := int64(connectedFor / time.Second)
if got := uptimeSeconds(connectedSince); got != want { if got := uptimeSeconds(connectedSince); got != want {
t.Errorf("uptimeSeconds = %d, want %d", got, want) t.Errorf("uptimeSeconds = %d, want %d", got, want)
} }
if got := uptimeSeconds(time.Time{}); got != 0 { if got := uptimeSeconds(time.Time{}); got != 0 {
t.Errorf("uptimeSeconds before any connection = %d, want 0", got) t.Errorf("uptimeSeconds before any connection = %d, want 0", got)
} }
switch got := connectionDurationSeconds(true, connectedSince); { switch got := connectionDurationSeconds(true, connectedSince); {
case got == nil: case got == nil:
t.Errorf("connectionDurationSeconds = nil, want %d", want) t.Errorf("connectionDurationSeconds = nil, want %d", want)
case *got != want: case *got != want:
t.Errorf("connectionDurationSeconds = %d, want %d", *got, want) t.Errorf("connectionDurationSeconds = %d, want %d", *got, want)
} }
if got := connectionDurationSeconds(false, connectedSince); got != nil { if got := connectionDurationSeconds(false, connectedSince); got != nil {
t.Errorf("connectionDurationSeconds while not connected = %d, want nil", t.Errorf("connectionDurationSeconds while not connected = %d, want nil",
*got) *got)
} }
})
} }
// newTestTable returns a routing table that holds a route for testPrefixV4, // newTestTable returns a routing table that holds a route for testPrefixV4,
@@ -1138,7 +1172,9 @@ func (failingDB) GetPrefixFirstSeen(
// with the driver's error text. It also sends a prefix that does not parse. // with the driver's error text. It also sends a prefix that does not parse.
// No answer may carry the driver's or the prefix parser's text. Before the fix // No answer may carry the driver's or the prefix parser's text. Before the fix
// the answers sent it to the client // the answers sent it to the client
// (https://git.eeqj.de/sneak/routewatch/issues/108). // (https://git.eeqj.de/sneak/routewatch/issues/108). The requests run in a
// synctest bubble, whose clock is fake, so a stall on the host does not use up
// the stats handlers' 4-second limit.
func TestJSONRoutesDoNotSendErrorDetails(t *testing.T) { func TestJSONRoutesDoNotSendErrorDetails(t *testing.T) {
t.Parallel() t.Parallel()
@@ -1159,8 +1195,11 @@ func TestJSONRoutesDoNotSendErrorDetails(t *testing.T) {
for _, tt := range tests { for _, tt := range tests {
rec := httptest.NewRecorder() rec := httptest.NewRecorder()
s.router.ServeHTTP(rec, httptest.NewRequestWithContext(
t.Context(), http.MethodGet, tt.path, nil)) synctest.Test(t, func(t *testing.T) {
s.router.ServeHTTP(rec, httptest.NewRequestWithContext(
t.Context(), http.MethodGet, tt.path, nil))
})
if rec.Code != tt.wantCode { if rec.Code != tt.wantCode {
t.Errorf("%s: status %d, want %d; body %s", t.Errorf("%s: status %d, want %d; body %s",
@@ -1414,8 +1453,11 @@ func TestIPLookupKeepsRouteWhenPTRLookupDoesNotAnswer(t *testing.T) {
s := newServerWithTestRoutes(t) s := newServerWithTestRoutes(t)
useFakeResolver(s, &fakeResolver{ptrNeverAnswers: true}) useFakeResolver(s, &fakeResolver{ptrNeverAnswers: true})
// The request's time limit passes after the PTR lookup's. // The request's time limit passes well after the PTR lookup's, so that a
ctx, cancel := context.WithTimeout(t.Context(), 2*ptr.LookupTimeout) // stall on a busy host does not use up the difference.
const requestTimeLimit = ptr.LookupTimeout + 15*time.Second
ctx, cancel := context.WithTimeout(t.Context(), requestTimeLimit)
defer cancel() defer cancel()
rec := httptest.NewRecorder() rec := httptest.NewRecorder()
@@ -1462,6 +1504,10 @@ const listedUnroutedAddr = "203.0.113.1"
// torExitAddr is the address that fakeIPLists lists on the Tor exit list. // torExitAddr is the address that fakeIPLists lists on the Tor exit list.
const torExitAddr = "203.0.113.2" const torExitAddr = "203.0.113.2"
// relayAddr is the address that fakeIPLists lists on the iCloud Private Relay
// list.
const relayAddr = "203.0.113.5"
// The addresses that fakeIPLists lists on one and on two VPN lists, and the // The addresses that fakeIPLists lists on one and on two VPN lists, and the
// names of those lists. // names of those lists.
const ( const (
@@ -1471,10 +1517,11 @@ const (
otherTestVPN = "other_test_vpn" otherTestVPN = "other_test_vpn"
) )
// fakeIPLists has four lists. The test list holds addrInTestPrefixV4, // fakeIPLists has five lists. The test list holds addrInTestPrefixV4,
// listedUnroutedAddr and twoVPNListsAddr. The Tor exit list holds torExitAddr. // listedUnroutedAddr and twoVPNListsAddr. The Tor exit list holds torExitAddr.
// The test VPN list holds oneVPNListAddr and twoVPNListsAddr, and the other // The iCloud Private Relay list holds relayAddr. The test VPN list holds
// test VPN list holds twoVPNListsAddr. // oneVPNListAddr and twoVPNListsAddr, and the other test VPN list holds
// twoVPNListsAddr.
type fakeIPLists struct{} type fakeIPLists struct{}
func testListing() iplist.Listing { func testListing() iplist.Listing {
@@ -1495,6 +1542,10 @@ func (fakeIPLists) Lookup(addr netip.Addr) []iplist.Listing {
return []iplist.Listing{{Name: iplist.TorExitsList}} return []iplist.Listing{{Name: iplist.TorExitsList}}
} }
if addr == netip.MustParseAddr(relayAddr) {
return []iplist.Listing{{Name: iplist.ICloudPrivateRelayList}}
}
if addr == netip.MustParseAddr(oneVPNListAddr) { if addr == netip.MustParseAddr(oneVPNListAddr) {
return []iplist.Listing{{Name: testVPNList, VPN: true}} return []iplist.Listing{{Name: testVPNList, VPN: true}}
} }
@@ -1527,20 +1578,22 @@ func TestIPLookupFlags(t *testing.T) {
name string name string
addr string addr string
tor bool tor bool
relay bool
bogon bool bogon bool
}{ }{
{"on the Tor exit list", torExitAddr, true, true}, {"on the Tor exit list", torExitAddr, true, false, true},
{"on another list", listedUnroutedAddr, false, true}, {"on the iCloud Private Relay list", relayAddr, false, true, true},
{"routed IPv4", addrInTestPrefixV4, false, true}, {"on another list", listedUnroutedAddr, false, false, true},
{"routed IPv6", addrInTestPrefixV6, false, true}, {"routed IPv4", addrInTestPrefixV4, false, false, true},
{"private IPv4", "10.0.0.1", false, true}, {"routed IPv6", addrInTestPrefixV6, false, false, true},
{"loopback IPv4", "127.0.0.1", false, true}, {"private IPv4", "10.0.0.1", false, false, true},
{"public IPv4", "1.1.1.1", false, false}, {"loopback IPv4", "127.0.0.1", false, false, true},
{"private IPv6", "fd00::1", false, true}, {"public IPv4", "1.1.1.1", false, false, false},
{"loopback IPv6", "::1", false, true}, {"private IPv6", "fd00::1", false, false, true},
{"public IPv6", "2606:4700:4700::1111", false, false}, {"loopback IPv6", "::1", false, false, true},
{"public IPv4 in IPv6 form", "::ffff:1.1.1.1", false, false}, {"public IPv6", "2606:4700:4700::1111", false, false, false},
{"private IPv4 in IPv6 form", "::ffff:10.0.0.1", false, true}, {"public IPv4 in IPv6 form", "::ffff:1.1.1.1", false, false, false},
{"private IPv4 in IPv6 form", "::ffff:10.0.0.1", false, false, true},
} }
for _, tt := range tests { for _, tt := range tests {
@@ -1562,7 +1615,8 @@ func TestIPLookupFlags(t *testing.T) {
} }
want := map[string]bool{ want := map[string]bool{
"is_tor": tt.tor, "is_vpn": false, "is_bogon": tt.bogon, "is_tor": tt.tor, "is_vpn": false, "is_relay": tt.relay,
"is_bogon": tt.bogon,
} }
if !maps.Equal(got, want) { if !maps.Equal(got, want) {
t.Errorf("%s: flags %v, want %v", path, got, want) t.Errorf("%s: flags %v, want %v", path, got, want)
@@ -2414,9 +2468,13 @@ func settledGoroutineCount() int {
return prev return prev
} }
// waitForGoroutines waits until the goroutine count drops to target or below. // waitForGoroutines waits up to 15 seconds for the goroutine count to drop to
// target or below.
func waitForGoroutines(target int) bool { func waitForGoroutines(target int) bool {
for range 100 { const maxWait = 15 * time.Second
deadline := time.Now().Add(maxWait)
for time.Now().Before(deadline) {
if runtime.NumGoroutine() <= target { if runtime.NumGoroutine() <= target {
return true return true
} }
+8 -4
View File
@@ -76,7 +76,9 @@ func asLink(asn int) string {
} }
// TestHomePageCardsShowTheTableFigures fills a routing table and the asns // TestHomePageCardsShowTheTableFigures fills a routing table and the asns
// table, works out the cards, and checks that each card shows its figure. // table, works out the cards, and checks that each card shows its figure. The
// cards are worked out in a synctest bubble, whose clock is fake, so a stall on
// the host does not use up the 4-second limit on their database read.
func TestHomePageCardsShowTheTableFigures(t *testing.T) { func TestHomePageCardsShowTheTableFigures(t *testing.T) {
t.Parallel() t.Parallel()
@@ -95,7 +97,9 @@ func TestHomePageCardsShowTheTableFigures(t *testing.T) {
t.Fatalf("GetOrCreateASNBatch: %v", err) t.Fatalf("GetOrCreateASNBatch: %v", err)
} }
s.refreshHomeCards(t.Context()) synctest.Test(t, func(t *testing.T) {
s.refreshHomeCards(t.Context())
})
page := get(t, s, "/").Body.String() page := get(t, s, "/").Body.String()
@@ -279,7 +283,7 @@ func TestKeepHomeCardsFreshWorksThemOutEveryInterval(t *testing.T) {
const ( const (
interval = 10 * time.Millisecond interval = 10 * time.Millisecond
maxWait = 5 * time.Second maxWait = 15 * time.Second
) )
table := emptyTable() table := emptyTable()
@@ -321,7 +325,7 @@ func TestKeepHomeCardsFreshWorksThemOutEveryInterval(t *testing.T) {
func TestStartedServerWorksOutTheCards(t *testing.T) { func TestStartedServerWorksOutTheCards(t *testing.T) {
t.Parallel() t.Parallel()
const maxWait = 5 * time.Second const maxWait = 15 * time.Second
s := newServerWithTable(t, newCardsTable()) s := newServerWithTable(t, newCardsTable())
+39 -32
View File
@@ -14,6 +14,7 @@ import (
"regexp" "regexp"
"strings" "strings"
"testing" "testing"
"testing/synctest"
"time" "time"
"git.eeqj.de/sneak/routewatch/internal/config" "git.eeqj.de/sneak/routewatch/internal/config"
@@ -163,6 +164,10 @@ func TestJSONAnswersPutNoUnitInAString(t *testing.T) {
// router and the middleware write. Before the fix a panic in a handler, and a // router and the middleware write. Before the fix a panic in a handler, and a
// method /status.json or the health check does not take, answered with an // method /status.json or the health check does not take, answered with an
// empty body (https://git.eeqj.de/sneak/routewatch/issues/201). // empty body (https://git.eeqj.de/sneak/routewatch/issues/201).
//
// The requests run in a synctest bubble, whose clock is fake, so a stall on
// the host does not use up the 2-second limit of the health check or the
// 4-second limit of the stats handlers.
func TestJSONAnswersCarryMeta(t *testing.T) { func TestJSONAnswersCarryMeta(t *testing.T) {
t.Parallel() t.Parallel()
@@ -181,6 +186,10 @@ func TestJSONAnswersCarryMeta(t *testing.T) {
s.router.Get(panicsPath, JSONValidationMiddleware(http.HandlerFunc( s.router.Get(panicsPath, JSONValidationMiddleware(http.HandlerFunc(
func(http.ResponseWriter, *http.Request) { panic(errDiskIO) })).ServeHTTP) func(http.ResponseWriter, *http.Request) { panic(errDiskIO) })).ServeHTTP)
failing := New(failingDB{}, newTestTable(),
streamer.New(logger.New(), metrics.New()), logger.New(), &config.Config{})
useFakeResolver(failing, &fakeResolver{})
tests := []struct { tests := []struct {
method string method string
path string path string
@@ -216,42 +225,40 @@ func TestJSONAnswersCarryMeta(t *testing.T) {
{http.MethodGet, panicsPath, http.StatusInternalServerError}, {http.MethodGet, panicsPath, http.StatusInternalServerError},
} }
for _, tt := range tests { synctest.Test(t, func(t *testing.T) {
checkAnswerCarriesMeta(t, s, httptest.NewRequestWithContext( for _, tt := range tests {
t.Context(), tt.method, tt.path, nil), tt.wantCode) checkAnswerCarriesMeta(t, s, httptest.NewRequestWithContext(
} t.Context(), tt.method, tt.path, nil), tt.wantCode)
}
// A request whose time limit has already passed answers 504. The handlers // A request whose time limit has already passed answers 504. The
// write the answers of the first five routes, and the JSON check on the // handlers write the answers of the first five routes, and the JSON
// route writes the last. // check on the route writes the last.
pastTimeLimit, cancel := context.WithDeadline(t.Context(), time.Now()) pastTimeLimit, cancel := context.WithDeadline(t.Context(), time.Now())
defer cancel() defer cancel()
for _, path := range []string{ for _, path := range []string{
statusJSONPath, statsPath, "/ip/" + addrInTestPrefixV4, statusJSONPath, statsPath, "/ip/" + addrInTestPrefixV4,
"/api/v1/ip/" + addrInTestPrefixV4, asJSONPath, writesNothingPath, "/api/v1/ip/" + addrInTestPrefixV4, asJSONPath, writesNothingPath,
} { } {
checkAnswerCarriesMeta(t, s, httptest.NewRequestWithContext( checkAnswerCarriesMeta(t, s, httptest.NewRequestWithContext(
pastTimeLimit, http.MethodGet, path, nil), http.StatusGatewayTimeout) pastTimeLimit, http.MethodGet, path, nil), http.StatusGatewayTimeout)
} }
// With a store whose reads fail, the health check answers 503, and the // With a store whose reads fail, the health check answers 503, and
// handlers of the other routes 500. // the handlers of the other routes 500.
failing := New(failingDB{}, newTestTable(),
streamer.New(logger.New(), metrics.New()), logger.New(), &config.Config{})
useFakeResolver(failing, &fakeResolver{})
checkAnswerCarriesMeta(t, failing, httptest.NewRequestWithContext(
t.Context(), http.MethodGet, healthCheckPath, nil),
http.StatusServiceUnavailable)
for _, path := range []string{
statusJSONPath, statsPath, asJSONPath,
"/api/v1/ip/" + addrInTestPrefixV4, "/ip/" + addrInTestPrefixV4,
} {
checkAnswerCarriesMeta(t, failing, httptest.NewRequestWithContext( checkAnswerCarriesMeta(t, failing, httptest.NewRequestWithContext(
t.Context(), http.MethodGet, path, nil), http.StatusInternalServerError) t.Context(), http.MethodGet, healthCheckPath, nil),
} http.StatusServiceUnavailable)
for _, path := range []string{
statusJSONPath, statsPath, asJSONPath,
"/api/v1/ip/" + addrInTestPrefixV4, "/ip/" + addrInTestPrefixV4,
} {
checkAnswerCarriesMeta(t, failing, httptest.NewRequestWithContext(
t.Context(), http.MethodGet, path, nil), http.StatusInternalServerError)
}
})
} }
// checkAnswerCarriesMeta serves req through the server's router, and checks // checkAnswerCarriesMeta serves req through the server's router, and checks
+11 -2
View File
@@ -5,6 +5,7 @@ import (
"net/http" "net/http"
"net/http/httptest" "net/http/httptest"
"testing" "testing"
"testing/synctest"
) )
// TestRoutesAnswerHEAD sends HEAD for a page and JSON routes through the // TestRoutesAnswerHEAD sends HEAD for a page and JSON routes through the
@@ -12,15 +13,23 @@ import (
// lookup of the caller's own address, are in the list because chi's GetHead // lookup of the caller's own address, are in the list because chi's GetHead
// does not reach them. Before the fix every route answered HEAD with 405 // does not reach them. Before the fix every route answered HEAD with 405
// (https://git.eeqj.de/sneak/routewatch/issues/111). // (https://git.eeqj.de/sneak/routewatch/issues/111).
//
// The requests run in a synctest bubble, whose clock is fake, so a stall on
// the host does not use up the stats handlers' 4-second limit. The resolver
// is fake because no time limit would cut off a real DNS lookup in a bubble.
func TestRoutesAnswerHEAD(t *testing.T) { func TestRoutesAnswerHEAD(t *testing.T) {
t.Parallel() t.Parallel()
s := newServerWithTestRoutes(t) s := newServerWithTestRoutes(t)
s.resolver = &fakeResolver{}
for _, path := range []string{"/", statusJSONPath, statsPath, "/ip", "/ip/"} { for _, path := range []string{"/", statusJSONPath, statsPath, "/ip", "/ip/"} {
rec := httptest.NewRecorder() rec := httptest.NewRecorder()
s.router.ServeHTTP(rec, httptest.NewRequestWithContext(
t.Context(), http.MethodHead, path, nil)) synctest.Test(t, func(t *testing.T) {
s.router.ServeHTTP(rec, httptest.NewRequestWithContext(
t.Context(), http.MethodHead, path, nil))
})
if rec.Code != http.StatusOK { if rec.Code != http.StatusOK {
t.Errorf("HEAD %s: status %d, want %d; body %s", t.Errorf("HEAD %s: status %d, want %d; body %s",
+2 -2
View File
@@ -22,10 +22,10 @@ func TestStopClosesTheConnectionOfARequestStillRunning(t *testing.T) {
stopDeadline = 100 * time.Millisecond stopDeadline = 100 * time.Millisecond
// waitTimeout is how long the test waits for the request to reach // waitTimeout is how long the test waits for the request to reach
// the handler, and then for the client to see its connection closed. // the handler, and then for the client to see its connection closed.
waitTimeout = 2 * time.Second waitTimeout = 15 * time.Second
// readHeaderTimeout bounds the test server's read of the request // readHeaderTimeout bounds the test server's read of the request
// header, as every http.Server must. // header, as every http.Server must.
readHeaderTimeout = time.Second readHeaderTimeout = waitTimeout
) )
release := make(chan struct{}) release := make(chan struct{})
+130 -73
View File
@@ -14,6 +14,7 @@ import (
"strings" "strings"
"sync/atomic" "sync/atomic"
"testing" "testing"
"testing/synctest"
"time" "time"
"git.eeqj.de/sneak/routewatch/internal/logger" "git.eeqj.de/sneak/routewatch/internal/logger"
@@ -398,7 +399,7 @@ func TestStopQueuesTheHeldStateMessages(t *testing.T) {
const ( const (
held = 2 held = 2
lines = held + 1 lines = held + 1
maxWait = 5 * time.Second maxWait = 15 * time.Second
) )
srv := httptest.NewServer(http.HandlerFunc( srv := httptest.NewServer(http.HandlerFunc(
@@ -478,15 +479,15 @@ func TestStopQueuesTheHeldStateMessages(t *testing.T) {
// TestStopReturnsByItsDeadline stops a streamer whose handler has a full queue // TestStopReturnsByItsDeadline stops a streamer whose handler has a full queue
// and a STATE message held for it, and a worker that stays busy until the test // and a STATE message held for it, and a worker that stays busy until the test
// ends, as one does whose handler waits for a database write. Stop must return // ends, as one does whose handler waits for a database write. Stop must return
// within its deadline plus a small margin instead of waiting for room on the // within its deadline plus a margin instead of waiting for room on the queue
// queue and for the worker, and log as not handed over the held message and // and for the worker, and log as not handed over the held message and every
// every message on the queue. // message on the queue.
func TestStopReturnsByItsDeadline(t *testing.T) { func TestStopReturnsByItsDeadline(t *testing.T) {
t.Parallel() t.Parallel()
const ( const (
deadline = 100 * time.Millisecond deadline = 100 * time.Millisecond
margin = 2 * time.Second margin = 15 * time.Second
) )
var logs bytes.Buffer var logs bytes.Buffer
@@ -577,7 +578,7 @@ func TestStopBeforeMessageReachesQueues(t *testing.T) {
func TestReconnectWhenStreamEnds(t *testing.T) { func TestReconnectWhenStreamEnds(t *testing.T) {
t.Parallel() t.Parallel()
endResponse := func(http.ResponseWriter, *http.Request) {} endResponse := pipeTransport{write: func(context.Context, io.Writer) {}}
checkReconnect(t, endResponse, streamIdleTimeout) checkReconnect(t, endResponse, streamIdleTimeout)
} }
@@ -589,9 +590,8 @@ func TestReconnectWhenStreamGoesIdle(t *testing.T) {
const idleTimeout = 50 * time.Millisecond const idleTimeout = 50 * time.Millisecond
holdResponse := func(w http.ResponseWriter, r *http.Request) { holdResponse := pipeTransport{
flush(w) write: func(ctx context.Context, _ io.Writer) { <-ctx.Done() },
<-r.Context().Done()
} }
checkReconnect(t, holdResponse, idleTimeout) checkReconnect(t, holdResponse, idleTimeout)
} }
@@ -604,16 +604,51 @@ func TestReconnectWhenServerNeverAnswers(t *testing.T) {
const idleTimeout = 50 * time.Millisecond const idleTimeout = 50 * time.Millisecond
neverAnswer := func(_ http.ResponseWriter, r *http.Request) { neverAnswer := roundTripFunc(
<-r.Context().Done() func(req *http.Request) (*http.Response, error) {
} <-req.Context().Done()
return nil, req.Context().Err()
},
)
checkReconnect(t, neverAnswer, idleTimeout) checkReconnect(t, neverAnswer, idleTimeout)
} }
// TestStreamNotDroppedWhileLinesArrive has the server send lines well within // pipeTransport answers every request with a response whose body write fills.
// the idle timeout for several times the timeout, then end the response. Each // The body fails when the request's context ends, as a connection's does.
// line must start the idle timeout over, so the stream lasts until the server // write gets that context, so that it can end when the request does.
// ends it instead of being dropped as idle. type pipeTransport struct {
write func(context.Context, io.Writer)
}
func (p pipeTransport) RoundTrip(req *http.Request) (*http.Response, error) {
body, w := io.Pipe()
context.AfterFunc(req.Context(), func() {
_ = w.CloseWithError(req.Context().Err())
})
go func(ctx context.Context) {
p.write(ctx, w)
_ = w.Close()
}(req.Context())
return &http.Response{StatusCode: http.StatusOK, Body: body}, nil
}
// roundTripFunc makes a function an http.RoundTripper.
type roundTripFunc func(*http.Request) (*http.Response, error)
func (f roundTripFunc) RoundTrip(req *http.Request) (*http.Response, error) {
return f(req)
}
// TestStreamNotDroppedWhileLinesArrive has a response send lines well within
// the idle timeout for several times the timeout, then end. Each line must
// start the idle timeout over, so the stream lasts until the response ends
// instead of being dropped as idle. It runs in a synctest bubble, whose clock
// is fake, so a stall on the host does not use up the timeout.
func TestStreamNotDroppedWhileLinesArrive(t *testing.T) { func TestStreamNotDroppedWhileLinesArrive(t *testing.T) {
t.Parallel() t.Parallel()
@@ -623,26 +658,35 @@ func TestStreamNotDroppedWhileLinesArrive(t *testing.T) {
sendFor = 5 * idleTimeout sendFor = 5 * idleTimeout
) )
srv := httptest.NewServer(http.HandlerFunc( // The streamer is made outside the bubble. Its metrics start a goroutine
func(w http.ResponseWriter, _ *http.Request) { // that never returns, and a bubble waits for each of its goroutines to
// return.
s := New(logger.New(), metrics.New())
s.idleTimeout = idleTimeout
s.client = &http.Client{Transport: pipeTransport{
write: func(ctx context.Context, w io.Writer) {
deadline := time.Now().Add(sendFor) deadline := time.Now().Add(sendFor)
for time.Now().Before(deadline) { for time.Now().Before(deadline) {
_, _ = io.WriteString(w, updateLine) _, _ = io.WriteString(w, updateLine)
flush(w) // The wait ends with the stream. A bubble's clock stops when
time.Sleep(lineInterval) // the test function returns, and a goroutine then still
// asleep panics the test binary.
select {
case <-ctx.Done():
return
case <-time.After(lineInterval):
}
} }
})) },
defer srv.Close() }}
s := New(logger.New(), metrics.New()) synctest.Test(t, func(t *testing.T) {
s.url = srv.URL err := s.stream(context.Background())
s.idleTimeout = idleTimeout if !errors.Is(err, errStreamEnded) {
t.Fatalf("stream returned %v, want %v", err, errStreamEnded)
err := s.stream(context.Background()) }
if !errors.Is(err, errStreamEnded) { })
t.Fatalf("stream returned %v, want %v", err, errStreamEnded)
}
} }
// TestStreamReadsLongLine has the server send one UPDATE line of about // TestStreamReadsLongLine has the server send one UPDATE line of about
@@ -706,7 +750,7 @@ func TestHandlerStatsReadWhileMessagesAreDropped(t *testing.T) {
const ( const (
lines = 5 lines = 5
wantDropped = lines - 2 wantDropped = lines - 2
maxWait = 5 * time.Second maxWait = 15 * time.Second
) )
srv := httptest.NewServer(http.HandlerFunc( srv := httptest.NewServer(http.HandlerFunc(
@@ -820,61 +864,70 @@ func streamLines(t *testing.T, s *Streamer, lines string) {
// line and then stays open. The streamer must come back for a second // line and then stays open. The streamer must come back for a second
// response, must not be marked connected when it does, and must handle the // response, must not be marked connected when it does, and must handle the
// UPDATE in it. // UPDATE in it.
//
// It runs in a synctest bubble, whose clock is fake, so a stall on the host
// does not use up the idle timeout. The fake server gets the requests in the
// order the streamer makes them, because no real connection carries them.
func checkReconnect( func checkReconnect(
t *testing.T, firstResponse http.HandlerFunc, idleTimeout time.Duration, t *testing.T, firstResponse http.RoundTripper, idleTimeout time.Duration,
) { ) {
t.Helper() t.Helper()
const maxWait = 5 * time.Second const maxWait = 15 * time.Second
// The tracker is made outside the bubble. It starts a goroutine that never
// returns, and a bubble waits for each of its goroutines to return.
tracker := metrics.New() tracker := metrics.New()
connectedAtReconnect := make(chan bool, 1)
var requests atomic.Int32 synctest.Test(t, func(t *testing.T) {
connectedAtReconnect := make(chan bool, 1)
laterResponse := pipeTransport{
write: func(ctx context.Context, w io.Writer) {
_, _ = io.WriteString(w, updateLine)
srv := httptest.NewServer(http.HandlerFunc( <-ctx.Done()
func(w http.ResponseWriter, r *http.Request) { },
if requests.Add(1) == 1 { }
firstResponse(w, r)
return var requests atomic.Int32
}
select { s := New(logger.New(), tracker)
case connectedAtReconnect <- tracker.IsConnected(): s.client = &http.Client{Transport: roundTripFunc(
default: func(req *http.Request) (*http.Response, error) {
} if requests.Add(1) == 1 {
return firstResponse.RoundTrip(req)
}
_, _ = io.WriteString(w, updateLine) select {
case connectedAtReconnect <- tracker.IsConnected():
default:
}
flush(w) return laterResponse.RoundTrip(req)
<-r.Context().Done() },
})) )}
defer srv.Close() s.idleTimeout = idleTimeout
s.minBackoff = time.Millisecond
handler := updateHandler{handled: make(chan struct{}, 1)}
s.RegisterHandler(handler)
s := New(logger.New(), tracker) err := s.Start(t.Context())
s.url = srv.URL if err != nil {
s.idleTimeout = idleTimeout t.Fatalf("Start returned error: %v", err)
s.minBackoff = time.Millisecond }
handler := updateHandler{handled: make(chan struct{}, 1)} defer s.Stop(t.Context())
s.RegisterHandler(handler)
err := s.Start(t.Context()) select {
if err != nil { case <-handler.handled:
t.Fatalf("Start returned error: %v", err) case <-time.After(maxWait):
} t.Fatalf("no message from a second response handled within %s", maxWait)
defer s.Stop(t.Context()) }
select { if <-connectedAtReconnect {
case <-handler.handled: t.Error("streamer was still marked connected when it came back " +
case <-time.After(maxWait): "for a second response")
t.Fatalf("no message from a second response handled within %s", maxWait) }
} })
if <-connectedAtReconnect {
t.Error("streamer was still marked connected when it came back " +
"for a second response")
}
} }
// settledGoroutineCount lets transient goroutines finish, then reports the // settledGoroutineCount lets transient goroutines finish, then reports the
@@ -896,9 +949,13 @@ func settledGoroutineCount() int {
return prev return prev
} }
// waitForGoroutines waits until the goroutine count drops to target or below. // waitForGoroutines waits up to 15 seconds for the goroutine count to drop to
// target or below.
func waitForGoroutines(target int) bool { func waitForGoroutines(target int) bool {
for range 100 { const maxWait = 15 * time.Second
deadline := time.Now().Add(maxWait)
for time.Now().Before(deadline) {
if runtime.NumGoroutine() <= target { if runtime.NumGoroutine() <= target {
return true return true
} }