Compare commits
1
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
5051aff4dd |
No files matched your search
@@ -287,7 +287,6 @@ address. Every entry has the same fields:
|
|||||||
- `is_tor`: true when the address is on the `tor_exits` list
|
- `is_tor`: true when the address is on the `tor_exits` list
|
||||||
- `is_vpn`: true when the address is on any list of VPN servers (see IP
|
- `is_vpn`: true when the address is on any list of VPN servers (see IP
|
||||||
lists)
|
lists)
|
||||||
- `is_relay`: true when the address is on the `icloud_private_relay` list
|
|
||||||
- `is_bogon`: true when the address is reserved, so that no network should
|
- `is_bogon`: true when the address is reserved, so that no network should
|
||||||
announce it on the internet
|
announce it on the internet
|
||||||
- `vpn_providers`: the names of the lists of VPN servers the address is on, or
|
- `vpn_providers`: the names of the lists of VPN servers the address is on, or
|
||||||
@@ -422,9 +421,6 @@ The lists:
|
|||||||
last 30 days. From
|
last 30 days. From
|
||||||
[FireHOL](https://iplists.firehol.org/?ipset=firehol_level3), downloaded every
|
[FireHOL](https://iplists.firehol.org/?ipset=firehol_level3), downloaded every
|
||||||
3 hours.
|
3 hours.
|
||||||
- `icloud_private_relay`: the networks where iCloud Private Relay traffic
|
|
||||||
leaves. From [Apple](https://mask-api.icloud.com/egress-ip-ranges.csv),
|
|
||||||
downloaded once a day.
|
|
||||||
- `ivpn`: VPN servers run by IVPN. From
|
- `ivpn`: VPN servers run by IVPN. From
|
||||||
[IVPN](https://api.ivpn.net/v5/servers.json), downloaded once a day.
|
[IVPN](https://api.ivpn.net/v5/servers.json), downloaded once a day.
|
||||||
- `mullvad`: VPN servers run by Mullvad. From
|
- `mullvad`: VPN servers run by Mullvad. From
|
||||||
@@ -435,14 +431,6 @@ The lists:
|
|||||||
- `pia`: VPN servers run by Private Internet Access. From
|
- `pia`: VPN servers run by Private Internet Access. From
|
||||||
[Private Internet Access](https://serverlist.piaservers.net/vpninfo/servers/v6),
|
[Private Internet Access](https://serverlist.piaservers.net/vpninfo/servers/v6),
|
||||||
downloaded once a day.
|
downloaded once a day.
|
||||||
- `protonvpn`: the /24 networks that hold ProtonVPN servers. The whole /24 is
|
|
||||||
listed because the ProtonVPN-IPs README says traffic leaves from unpublished
|
|
||||||
addresses in it. From the public
|
|
||||||
[ProtonVPN-IPs](https://github.com/tn3w/ProtonVPN-IPs) mirror of Proton's
|
|
||||||
server list, downloaded once a day.
|
|
||||||
- `protonvpn_entry`: the IPv4 and IPv6 addresses of ProtonVPN servers. From the
|
|
||||||
same [ProtonVPN-IPs](https://github.com/tn3w/ProtonVPN-IPs) mirror, downloaded
|
|
||||||
once a day.
|
|
||||||
- `spamhaus_drop` and `spamhaus_dropv6`: IPv4 and IPv6 networks hijacked or
|
- `spamhaus_drop` and `spamhaus_dropv6`: IPv4 and IPv6 networks hijacked or
|
||||||
leased by spammers or cyber criminals, according to Spamhaus. From
|
leased by spammers or cyber criminals, according to Spamhaus. From
|
||||||
[Spamhaus DROP](https://www.spamhaus.org/blocklists/do-not-route-or-peer/),
|
[Spamhaus DROP](https://www.spamhaus.org/blocklists/do-not-route-or-peer/),
|
||||||
@@ -510,8 +498,6 @@ How RouteWatch keeps the answers:
|
|||||||
whose SOA record has a TTL or minimum field of 0.
|
whose SOA record has a TTL or minimum field of 0.
|
||||||
- A failed lookup is not kept, so the next lookup of the address asks again.
|
- A failed lookup is not kept, so the next lookup of the address asks again.
|
||||||
- Every 10 minutes it deletes expired answers from memory and from `cache.db`.
|
- Every 10 minutes it deletes expired answers from memory and from `cache.db`.
|
||||||
- It keeps answers for at most 250,000 addresses, and keeps no answer for a new
|
|
||||||
address while it holds that many.
|
|
||||||
|
|
||||||
## Configuration
|
## Configuration
|
||||||
|
|
||||||
@@ -610,13 +596,6 @@ The ceilings add up to 3.7 GiB:
|
|||||||
limit caps the SQLite library.
|
limit caps the SQLite library.
|
||||||
- About 0.2 GiB for everything else in the runtime.
|
- About 0.2 GiB for everything else in the runtime.
|
||||||
|
|
||||||
The `icloud_private_relay` list has about 285,000 networks. Neighbouring
|
|
||||||
networks merge, which leaves about 6,900 address ranges. Loaded, the list adds
|
|
||||||
about 0.3 MiB to the Go heap and under 6 MiB to the process's resident memory.
|
|
||||||
Parsing the file allocates about 94 MiB in about 0.2 seconds. The next garbage
|
|
||||||
collection frees all of it but the 0.3 MiB above. Parsing happens at startup and
|
|
||||||
whenever a download brings a new version.
|
|
||||||
|
|
||||||
The SQLite library gets its memory from glibc, the C library. glibc keeps freed
|
The SQLite library gets its memory from glibc, the C library. glibc keeps freed
|
||||||
memory in pools called arenas, and hands it back to the kernel slowly. It
|
memory in pools called arenas, and hands it back to the kernel slowly. It
|
||||||
creates up to eight arenas per processor core. So on a host with many cores the
|
creates up to eight arenas per processor core. So on a host with many cores the
|
||||||
|
|||||||
@@ -53,19 +53,9 @@ problem the run finds.
|
|||||||
|
|
||||||
# Completed Steps
|
# Completed Steps
|
||||||
|
|
||||||
- 2026-11-10: the IP lookup checks the `icloud_private_relay` list of Apple's
|
|
||||||
iCloud Private Relay networks, downloaded once a day. Each result has
|
|
||||||
`flags.is_relay` (closes https://git.eeqj.de/sneak/routewatch/issues/237).
|
|
||||||
- 2026-11-10: the IP lookup checks the `protonvpn` list of ProtonVPN's /24
|
|
||||||
networks and the `protonvpn_entry` list of its servers' IPv4 and IPv6
|
|
||||||
addresses, from a public mirror of Proton's server list (closes
|
|
||||||
https://git.eeqj.de/sneak/routewatch/issues/236).
|
|
||||||
- 2026-11-10: the IP lookup finds PTR names over DNS-over-HTTPS and caches them
|
- 2026-11-10: the IP lookup finds PTR names over DNS-over-HTTPS and caches them
|
||||||
for the record's TTL, in memory and in `cache.db` (closes
|
for the record's TTL, in memory and in `cache.db` (closes
|
||||||
https://git.eeqj.de/sneak/routewatch/issues/238).
|
https://git.eeqj.de/sneak/routewatch/issues/238).
|
||||||
- 2026-11-10: the tests no longer fail when a busy host stalls for several
|
|
||||||
seconds. Most such waits are now 15 seconds. The test of the stream's idle
|
|
||||||
timeout runs on a fake clock (closes #240)
|
|
||||||
- 2026-11-10: the IP lookup checks the public VPN server lists of Mullvad,
|
- 2026-11-10: the IP lookup checks the public VPN server lists of Mullvad,
|
||||||
NordVPN, Private Internet Access and IVPN, and X4BNet's community list. Each
|
NordVPN, Private Internet Access and IVPN, and X4BNet's community list. Each
|
||||||
result has `flags.is_vpn` and `vpn_providers` (closes
|
result has `flags.is_vpn` and `vpn_providers` (closes
|
||||||
|
|||||||
@@ -12,7 +12,7 @@ import (
|
|||||||
|
|
||||||
// parseTimeLimit is how long a test waits for parseCSV to return before it
|
// parseTimeLimit is how long a test waits for parseCSV to return before it
|
||||||
// fails.
|
// fails.
|
||||||
const parseTimeLimit = 15 * time.Second
|
const parseTimeLimit = 2 * time.Second
|
||||||
|
|
||||||
// TestParseCSVReadsTodaysLayout parses an as.csv with the country code column
|
// TestParseCSVReadsTodaysLayout parses an as.csv with the country code column
|
||||||
// that the source added after the description. It expects each AS with its
|
// that the source added after the description. It expects each AS with its
|
||||||
|
|||||||
@@ -72,34 +72,17 @@ const (
|
|||||||
// lockTestPrefix is the prefix the batch writes in the checkpoint tests
|
// lockTestPrefix is the prefix the batch writes in the checkpoint tests
|
||||||
// store.
|
// store.
|
||||||
lockTestPrefix = "192.0.2.0/24"
|
lockTestPrefix = "192.0.2.0/24"
|
||||||
// longBusyTimeoutMs is the busy timeout setLongBusyTimeout gives the
|
|
||||||
// writing connection. A checkpoint that waits for an open read takes this
|
|
||||||
// long.
|
|
||||||
longBusyTimeoutMs = 30000
|
|
||||||
// readerCheckpointLimit is how long a checkpoint, and a batch write
|
// readerCheckpointLimit is how long a checkpoint, and a batch write
|
||||||
// waiting for it, may take while a reader is open: well under
|
// waiting for it, may take while a reader is open: well under the busy
|
||||||
// longBusyTimeoutMs, with room for a stall on a busy host.
|
// timeout, which a checkpoint that waits for the reader takes.
|
||||||
readerCheckpointLimit = 15 * time.Second
|
readerCheckpointLimit = time.Second
|
||||||
// truncateCheckpointLimit is how long a checkpoint of a WAL file over
|
// truncateCheckpointLimit is how long a checkpoint of a WAL file over
|
||||||
// sqliteWALSizeLimitBytes may take to give up while a reader is open: its
|
// sqliteWALSizeLimitBytes may take to give up while a reader is open: its
|
||||||
// wait for the reader plus readerCheckpointLimit.
|
// wait for the reader plus a second, well under the busy timeout.
|
||||||
truncateCheckpointLimit = truncateBusyTimeoutMs*time.Millisecond +
|
truncateCheckpointLimit = truncateBusyTimeoutMs*time.Millisecond +
|
||||||
readerCheckpointLimit
|
time.Second
|
||||||
)
|
)
|
||||||
|
|
||||||
// setLongBusyTimeout gives the writing connection of db longBusyTimeoutMs, so
|
|
||||||
// that a checkpoint that waits for an open read takes far longer than a stall
|
|
||||||
// on a busy host.
|
|
||||||
func setLongBusyTimeout(ctx context.Context, t *testing.T, db *Database) {
|
|
||||||
t.Helper()
|
|
||||||
|
|
||||||
_, err := db.writer.ExecContext(ctx,
|
|
||||||
fmt.Sprintf("PRAGMA busy_timeout=%d", longBusyTimeoutMs))
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("failed to set the busy timeout: %v", err)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestConnectionPoolPragmas holds several pooled connections open at once and
|
// TestConnectionPoolPragmas holds several pooled connections open at once and
|
||||||
// checks each one carries the per-connection settings from the DSN, plus the
|
// checks each one carries the per-connection settings from the DSN, plus the
|
||||||
// process-wide heap limits, and sees the new file with auto_vacuum
|
// process-wide heap limits, and sees the new file with auto_vacuum
|
||||||
@@ -596,8 +579,6 @@ func TestCheckpointDoesNotWaitForReader(t *testing.T) {
|
|||||||
ctx := context.Background()
|
ctx := context.Background()
|
||||||
ts := time.Now().UTC()
|
ts := time.Now().UTC()
|
||||||
|
|
||||||
setLongBusyTimeout(ctx, t, db)
|
|
||||||
|
|
||||||
err = db.UpdatePrefixesBatch(map[string]time.Time{"198.51.100.0/24": ts})
|
err = db.UpdatePrefixesBatch(map[string]time.Time{"198.51.100.0/24": ts})
|
||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatalf("failed to store the prefix the reader reads: %v", err)
|
t.Fatalf("failed to store the prefix the reader reads: %v", err)
|
||||||
@@ -709,7 +690,6 @@ func TestCheckpointEmptiesWALOnceNoReadIsOpen(t *testing.T) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
fillWALPastLimit(ctx, t, db)
|
fillWALPastLimit(ctx, t, db)
|
||||||
setLongBusyTimeout(ctx, t, db)
|
|
||||||
|
|
||||||
started := time.Now()
|
started := time.Now()
|
||||||
|
|
||||||
|
|||||||
@@ -21,7 +21,7 @@ const (
|
|||||||
downloadTimeout = time.Minute
|
downloadTimeout = time.Minute
|
||||||
|
|
||||||
// maxDownloadSize is the size of the largest list file accepted, in
|
// maxDownloadSize is the size of the largest list file accepted, in
|
||||||
// bytes. The largest list, iCloud Private Relay, is about 12 MiB.
|
// bytes. The largest list, FireHOL abusers, is about 2.5 MiB.
|
||||||
maxDownloadSize = 200 << 20
|
maxDownloadSize = 200 << 20
|
||||||
)
|
)
|
||||||
|
|
||||||
|
|||||||
@@ -51,16 +51,6 @@ const (
|
|||||||
// FormatIVPNJSON is IVPN's JSON server list. The hosts of its wireguard
|
// FormatIVPNJSON is IVPN's JSON server list. The hosts of its wireguard
|
||||||
// and openvpn gateways have their addresses in host and v2ray.
|
// and openvpn gateways have their addresses in host and v2ray.
|
||||||
FormatIVPNJSON
|
FormatIVPNJSON
|
||||||
|
|
||||||
// FormatProtonVPN has one address or network per line, with comment lines
|
|
||||||
// that start with #. Unlike FormatLines, a file with no addresses or
|
|
||||||
// networks fails.
|
|
||||||
FormatProtonVPN
|
|
||||||
|
|
||||||
// FormatICloudPrivateRelayCSV is Apple's CSV file of iCloud Private Relay
|
|
||||||
// networks. Each line starts with a network, followed by its country,
|
|
||||||
// region and city.
|
|
||||||
FormatICloudPrivateRelayCSV
|
|
||||||
)
|
)
|
||||||
|
|
||||||
// List is one public IP list.
|
// List is one public IP list.
|
||||||
|
|||||||
@@ -70,31 +70,6 @@ const (
|
|||||||
"config":{"api":{"ips":["192.0.2.73"]}}}`
|
"config":{"api":{"ips":["192.0.2.73"]}}}`
|
||||||
)
|
)
|
||||||
|
|
||||||
// protonVPNBody is a small file in the format of the ProtonVPN-IPs list.
|
|
||||||
const protonVPNBody = "#\n# protonvpn_ip_ranges.txt\n" +
|
|
||||||
"# https://github.com/tn3w/ProtonVPN-IPs\n#\n" +
|
|
||||||
"# The /24 block of every advertised IPv4.\n#\n" +
|
|
||||||
"192.0.2.0/24\n198.51.100.0/24\n"
|
|
||||||
|
|
||||||
// protonVPNEntryBody is a small file in the format of the ProtonVPN-IPs list
|
|
||||||
// of server addresses.
|
|
||||||
const protonVPNEntryBody = "#\n# protonvpn_entry_ips.txt\n" +
|
|
||||||
"# https://github.com/tn3w/ProtonVPN-IPs\n#\n" +
|
|
||||||
"# Entry IPs clients connect to, resolved from ProtonVPN hostnames.\n#\n" +
|
|
||||||
"192.0.2.80\n2001:db8::80\n"
|
|
||||||
|
|
||||||
// iCloudPrivateRelayAddr is an address on iCloudPrivateRelayBody.
|
|
||||||
const iCloudPrivateRelayAddr = "2001:db8:1::5"
|
|
||||||
|
|
||||||
// iCloudPrivateRelayBody is a small file in the format of Apple's iCloud
|
|
||||||
// Private Relay file.
|
|
||||||
const iCloudPrivateRelayBody = "192.0.2.0/31,GB,GB-EN,London,\n" +
|
|
||||||
"198.51.100.32/27,US,US-CA,Los Angeles,\n" +
|
|
||||||
"2001:db8:1::/64,US,US-AK,Anchorage,\n"
|
|
||||||
|
|
||||||
// emptyFileCase names the case of a wrong document that is an empty file.
|
|
||||||
const emptyFileCase = "an empty file"
|
|
||||||
|
|
||||||
// errorPage is a file that is no list, as a publisher's error page is.
|
// errorPage is a file that is no list, as a publisher's error page is.
|
||||||
const errorPage = "<!DOCTYPE html>\n<html></html>\n"
|
const errorPage = "<!DOCTYPE html>\n<html></html>\n"
|
||||||
|
|
||||||
@@ -255,15 +230,6 @@ func TestLookupFindsListedAddresses(t *testing.T) {
|
|||||||
},
|
},
|
||||||
unlisted: []string{"192.0.2.12", "2001:db8::2"},
|
unlisted: []string{"192.0.2.12", "2001:db8::2"},
|
||||||
},
|
},
|
||||||
{
|
|
||||||
name: "iCloud Private Relay networks",
|
|
||||||
format: FormatICloudPrivateRelayCSV,
|
|
||||||
body: iCloudPrivateRelayBody,
|
|
||||||
listed: []string{
|
|
||||||
"192.0.2.1", "198.51.100.63", iCloudPrivateRelayAddr,
|
|
||||||
},
|
|
||||||
unlisted: []string{"192.0.2.2", "198.51.100.64", "2001:db8:2::1"},
|
|
||||||
},
|
|
||||||
})
|
})
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -301,20 +267,6 @@ func TestLookupFindsVPNServers(t *testing.T) {
|
|||||||
listed: []string{"192.0.2.70", "192.0.2.71", "192.0.2.72"},
|
listed: []string{"192.0.2.70", "192.0.2.71", "192.0.2.72"},
|
||||||
unlisted: []string{"192.0.2.73", "2001:db8::70"},
|
unlisted: []string{"192.0.2.73", "2001:db8::70"},
|
||||||
},
|
},
|
||||||
{
|
|
||||||
name: "ProtonVPN networks",
|
|
||||||
format: FormatProtonVPN,
|
|
||||||
body: protonVPNBody,
|
|
||||||
listed: []string{"192.0.2.0", "192.0.2.255", "198.51.100.7"},
|
|
||||||
unlisted: []string{"192.0.3.1", "2001:db8::90"},
|
|
||||||
},
|
|
||||||
{
|
|
||||||
name: "ProtonVPN server addresses",
|
|
||||||
format: FormatProtonVPN,
|
|
||||||
body: protonVPNEntryBody,
|
|
||||||
listed: []string{"192.0.2.80", "2001:db8::80"},
|
|
||||||
unlisted: []string{"192.0.2.81", "2001:db8::81"},
|
|
||||||
},
|
|
||||||
})
|
})
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -506,12 +458,12 @@ func TestWrongVPNDocumentFailsToParse(t *testing.T) {
|
|||||||
|
|
||||||
for _, f := range formats {
|
for _, f := range formats {
|
||||||
bodies := map[string]string{
|
bodies := map[string]string{
|
||||||
"not JSON": errorPage,
|
"not JSON": errorPage,
|
||||||
"other JSON": `{"status":"ok","data":{"count":1}}`,
|
"other JSON": `{"status":"ok","data":{"count":1}}`,
|
||||||
"no addresses": f.empty,
|
"no addresses": f.empty,
|
||||||
"bad address": f.badAddress,
|
"bad address": f.badAddress,
|
||||||
"cut off": f.good[:len(f.good)/2],
|
"cut off": f.good[:len(f.good)/2],
|
||||||
emptyFileCase: "",
|
"an empty file": "",
|
||||||
}
|
}
|
||||||
|
|
||||||
for name, body := range bodies {
|
for name, body := range bodies {
|
||||||
@@ -527,59 +479,6 @@ func TestWrongVPNDocumentFailsToParse(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// TestWrongProtonVPNDocumentFailsToParse parses documents that are not a
|
|
||||||
// ProtonVPN-IPs list. Each fails, so none can replace a good copy.
|
|
||||||
func TestWrongProtonVPNDocumentFailsToParse(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
bodies := map[string]string{
|
|
||||||
"error page": errorPage,
|
|
||||||
"JSON": `{"status":"ok"}`,
|
|
||||||
"comments only": "#\n# protonvpn_ip_ranges.txt\n#\n",
|
|
||||||
"bad network": "192.0.2.0/24\nexample\n",
|
|
||||||
emptyFileCase: "",
|
|
||||||
}
|
|
||||||
|
|
||||||
for name, body := range bodies {
|
|
||||||
t.Run(name, func(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
addresses, err := parse(strings.NewReader(body), FormatProtonVPN)
|
|
||||||
if err == nil {
|
|
||||||
t.Errorf("parse returned %v and no error", addresses.Ranges())
|
|
||||||
}
|
|
||||||
})
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestWrongICloudPrivateRelayDocumentFailsToParse parses documents that are
|
|
||||||
// not Apple's iCloud Private Relay file. Each fails, so none can replace a
|
|
||||||
// good copy.
|
|
||||||
func TestWrongICloudPrivateRelayDocumentFailsToParse(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
bodies := map[string]string{
|
|
||||||
"error page": errorPage,
|
|
||||||
"JSON": `{"status":"ok","count":1}`,
|
|
||||||
"no other field": "192.0.2.0/24\n",
|
|
||||||
"bad network": "192.0.2.0/24,GB,GB-EN,London,\nexample,GB,GB-EN,London,\n",
|
|
||||||
"an address": "192.0.2.1,GB,GB-EN,London,\n",
|
|
||||||
emptyFileCase: "",
|
|
||||||
}
|
|
||||||
|
|
||||||
for name, body := range bodies {
|
|
||||||
t.Run(name, func(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
addresses, err := parse(strings.NewReader(body),
|
|
||||||
FormatICloudPrivateRelayCSV)
|
|
||||||
if err == nil {
|
|
||||||
t.Errorf("parse returned %v and no error", addresses.Ranges())
|
|
||||||
}
|
|
||||||
})
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestLookupMarksVPNLists looks up an address on a VPN list and on another
|
// TestLookupMarksVPNLists looks up an address on a VPN list and on another
|
||||||
// list. Only the VPN list's listing says it is one.
|
// list. Only the VPN list's listing says it is one.
|
||||||
func TestLookupMarksVPNLists(t *testing.T) {
|
func TestLookupMarksVPNLists(t *testing.T) {
|
||||||
@@ -624,10 +523,7 @@ func TestPublicListsMarkTheVPNLists(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
want := []string{
|
want := []string{"ivpn", "mullvad", "nordvpn", "pia", "x4bnet", "x4bnet_v6"}
|
||||||
"ivpn", "mullvad", "nordvpn", "pia", "protonvpn", "protonvpn_entry",
|
|
||||||
"x4bnet", "x4bnet_v6",
|
|
||||||
}
|
|
||||||
if !slices.Equal(got, want) {
|
if !slices.Equal(got, want) {
|
||||||
t.Errorf("VPN lists %v, want %v", got, want)
|
t.Errorf("VPN lists %v, want %v", got, want)
|
||||||
}
|
}
|
||||||
@@ -750,13 +646,9 @@ func TestRestartLoadsCopiesFromDisk(t *testing.T) {
|
|||||||
second.Start(t.Context())
|
second.Start(t.Context())
|
||||||
second.Stop(t.Context())
|
second.Stop(t.Context())
|
||||||
|
|
||||||
// The download takes its time after writing the file, and the restart
|
|
||||||
// reads the file's time. A busy host can put seconds between the two.
|
|
||||||
const maxUpdatedDifference = 15 * time.Second
|
|
||||||
|
|
||||||
listings := lookup(t, second, listedAddr)
|
listings := lookup(t, second, listedAddr)
|
||||||
if len(listings) != 1 ||
|
if len(listings) != 1 ||
|
||||||
listings[0].Updated.Sub(downloaded).Abs() > maxUpdatedDifference {
|
listings[0].Updated.Sub(downloaded).Abs() > time.Second {
|
||||||
t.Errorf("after the restart, listings %+v, want one updated at %s",
|
t.Errorf("after the restart, listings %+v, want one updated at %s",
|
||||||
listings, downloaded)
|
listings, downloaded)
|
||||||
}
|
}
|
||||||
@@ -934,33 +826,15 @@ func TestStopCancelsADownloadInProgress(t *testing.T) {
|
|||||||
func TestTorExitsListHoldsIPv6Exits(t *testing.T) {
|
func TestTorExitsListHoldsIPv6Exits(t *testing.T) {
|
||||||
t.Parallel()
|
t.Parallel()
|
||||||
|
|
||||||
checkPublicListHolds(t, TorExitsList, onionooBody, onionooIPv6ExitAddr)
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestICloudPrivateRelayListHoldsItsNetworks downloads a file in Apple's
|
|
||||||
// format as the public iCloud Private Relay list, which is what an address's
|
|
||||||
// is_relay answer comes from.
|
|
||||||
func TestICloudPrivateRelayListHoldsItsNetworks(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
checkPublicListHolds(t, ICloudPrivateRelayList, iCloudPrivateRelayBody,
|
|
||||||
iCloudPrivateRelayAddr)
|
|
||||||
}
|
|
||||||
|
|
||||||
// checkPublicListHolds downloads body as the public list called name and
|
|
||||||
// checks that addr has one listing, which names that list.
|
|
||||||
func checkPublicListHolds(t *testing.T, name, body, addr string) {
|
|
||||||
t.Helper()
|
|
||||||
|
|
||||||
index := slices.IndexFunc(Lists(), func(list List) bool {
|
index := slices.IndexFunc(Lists(), func(list List) bool {
|
||||||
return list.Name == name
|
return list.Name == TorExitsList
|
||||||
})
|
})
|
||||||
if index < 0 {
|
if index < 0 {
|
||||||
t.Fatalf("no list named %s", name)
|
t.Fatalf("no list named %s", TorExitsList)
|
||||||
}
|
}
|
||||||
|
|
||||||
list := Lists()[index]
|
list := Lists()[index]
|
||||||
_, list.URL = newPublisher(t, body)
|
_, list.URL = newPublisher(t, onionooBody)
|
||||||
b := newTestIPLists(t.TempDir(), list)
|
b := newTestIPLists(t.TempDir(), list)
|
||||||
|
|
||||||
err := b.download(t.Context(), list)
|
err := b.download(t.Context(), list)
|
||||||
@@ -968,9 +842,9 @@ func checkPublicListHolds(t *testing.T, name, body, addr string) {
|
|||||||
t.Fatalf("download: %v", err)
|
t.Fatalf("download: %v", err)
|
||||||
}
|
}
|
||||||
|
|
||||||
listings := lookup(t, b, addr)
|
listings := lookup(t, b, onionooIPv6ExitAddr)
|
||||||
if len(listings) != 1 || listings[0].Name != name {
|
if len(listings) != 1 || listings[0].Name != TorExitsList {
|
||||||
t.Errorf("listings %+v, want one for %s", listings, name)
|
t.Errorf("listings %+v, want one for %s", listings, TorExitsList)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
@@ -8,13 +8,6 @@ import (
|
|||||||
// TorExitsList is the name of the list of Tor exit nodes.
|
// TorExitsList is the name of the list of Tor exit nodes.
|
||||||
const TorExitsList = "tor_exits"
|
const TorExitsList = "tor_exits"
|
||||||
|
|
||||||
// ICloudPrivateRelayList is the name of the list of iCloud Private Relay's
|
|
||||||
// networks.
|
|
||||||
const ICloudPrivateRelayList = "icloud_private_relay"
|
|
||||||
|
|
||||||
// protonVPNIPsURL is where the ProtonVPN-IPs mirror publishes its files.
|
|
||||||
const protonVPNIPsURL = "https://raw.githubusercontent.com/tn3w/ProtonVPN-IPs/master/"
|
|
||||||
|
|
||||||
// How often each list is downloaded. No list is downloaded more often than
|
// How often each list is downloaded. No list is downloaded more often than
|
||||||
// its publisher updates it or allows.
|
// its publisher updates it or allows.
|
||||||
const (
|
const (
|
||||||
@@ -34,9 +27,6 @@ const (
|
|||||||
fireHOLAbusersInterval = 5 * time.Hour
|
fireHOLAbusersInterval = 5 * time.Hour
|
||||||
fireHOLLevel3Interval = 3 * time.Hour
|
fireHOLLevel3Interval = 3 * time.Hour
|
||||||
|
|
||||||
// Apple's iCloud Private Relay file is 12 MB, so once a day.
|
|
||||||
iCloudPrivateRelayInterval = 24 * time.Hour
|
|
||||||
|
|
||||||
// Spamhaus asks that DROP be downloaded at most once a day:
|
// Spamhaus asks that DROP be downloaded at most once a day:
|
||||||
// https://www.spamhaus.org/blocklists/do-not-route-or-peer/
|
// https://www.spamhaus.org/blocklists/do-not-route-or-peer/
|
||||||
spamhausDROPInterval = 24 * time.Hour
|
spamhausDROPInterval = 24 * time.Hour
|
||||||
@@ -91,13 +81,6 @@ func nonVPNLists() []List {
|
|||||||
Interval: fireHOLLevel3Interval,
|
Interval: fireHOLLevel3Interval,
|
||||||
Format: FormatLines,
|
Format: FormatLines,
|
||||||
},
|
},
|
||||||
{
|
|
||||||
Name: ICloudPrivateRelayList,
|
|
||||||
Description: "In a network where iCloud Private Relay traffic leaves",
|
|
||||||
URL: "https://mask-api.icloud.com/egress-ip-ranges.csv",
|
|
||||||
Interval: iCloudPrivateRelayInterval,
|
|
||||||
Format: FormatICloudPrivateRelayCSV,
|
|
||||||
},
|
|
||||||
{
|
{
|
||||||
Name: "spamhaus_drop",
|
Name: "spamhaus_drop",
|
||||||
Description: "In an IPv4 network that Spamhaus says spammers " +
|
Description: "In an IPv4 network that Spamhaus says spammers " +
|
||||||
@@ -163,28 +146,6 @@ func vpnLists() []List {
|
|||||||
Format: FormatPIAJSON,
|
Format: FormatPIAJSON,
|
||||||
VPN: true,
|
VPN: true,
|
||||||
},
|
},
|
||||||
{
|
|
||||||
// Proton's own server list needs a logged-in session, so this is
|
|
||||||
// the public ProtonVPN-IPs mirror of it. Proton publishes only
|
|
||||||
// the addresses clients connect to. The mirror's README says
|
|
||||||
// traffic leaves from others in the same /24, so this file of
|
|
||||||
// /24 networks is the one it says to use for detecting exits.
|
|
||||||
Name: "protonvpn",
|
|
||||||
Description: "In a /24 network that holds a server of the " +
|
|
||||||
"ProtonVPN service",
|
|
||||||
URL: protonVPNIPsURL + "protonvpn_ip_ranges.txt",
|
|
||||||
Interval: vpnListInterval,
|
|
||||||
Format: FormatProtonVPN,
|
|
||||||
VPN: true,
|
|
||||||
},
|
|
||||||
{
|
|
||||||
Name: "protonvpn_entry",
|
|
||||||
Description: "A server of the ProtonVPN service",
|
|
||||||
URL: protonVPNIPsURL + "protonvpn_entry_ips.txt",
|
|
||||||
Interval: vpnListInterval,
|
|
||||||
Format: FormatProtonVPN,
|
|
||||||
VPN: true,
|
|
||||||
},
|
|
||||||
{
|
{
|
||||||
Name: "x4bnet",
|
Name: "x4bnet",
|
||||||
Description: "In an IPv4 network of a VPN service, according to " +
|
Description: "In an IPv4 network of a VPN service, according to " +
|
||||||
|
|||||||
@@ -25,10 +25,6 @@ const (
|
|||||||
// dshieldFieldSeparator separates the fields of a DShield line.
|
// dshieldFieldSeparator separates the fields of a DShield line.
|
||||||
dshieldFieldSeparator = "\t"
|
dshieldFieldSeparator = "\t"
|
||||||
|
|
||||||
// iCloudPrivateRelayFieldSeparator separates the fields of a line of
|
|
||||||
// Apple's iCloud Private Relay file.
|
|
||||||
iCloudPrivateRelayFieldSeparator = ","
|
|
||||||
|
|
||||||
// spamhausMetadataType is the type of the line that ends a Spamhaus
|
// spamhausMetadataType is the type of the line that ends a Spamhaus
|
||||||
// file. That line holds the file's metadata and no network.
|
// file. That line holds the file's metadata and no network.
|
||||||
spamhausMetadataType = "metadata"
|
spamhausMetadataType = "metadata"
|
||||||
@@ -62,10 +58,6 @@ func parse(r io.Reader, format Format) (*netipx.IPSet, error) {
|
|||||||
return parsePIA(r)
|
return parsePIA(r)
|
||||||
case FormatIVPNJSON:
|
case FormatIVPNJSON:
|
||||||
return parseIVPN(r)
|
return parseIVPN(r)
|
||||||
case FormatProtonVPN:
|
|
||||||
return parseLinesNotEmpty(r, addAddressOrNetwork)
|
|
||||||
case FormatICloudPrivateRelayCSV:
|
|
||||||
return parseLinesNotEmpty(r, addICloudPrivateRelayNetwork)
|
|
||||||
default:
|
default:
|
||||||
return nil, errUnknownFormat
|
return nil, errUnknownFormat
|
||||||
}
|
}
|
||||||
@@ -337,41 +329,6 @@ func parseIVPN(r io.Reader) (*netipx.IPSet, error) {
|
|||||||
return addressSet(texts)
|
return addressSet(texts)
|
||||||
}
|
}
|
||||||
|
|
||||||
// parseLinesNotEmpty is parseLines for a list that always holds addresses, so
|
|
||||||
// a file with none fails.
|
|
||||||
func parseLinesNotEmpty(
|
|
||||||
r io.Reader, addLine func(*netipx.IPSetBuilder, string) error,
|
|
||||||
) (*netipx.IPSet, error) {
|
|
||||||
addresses, err := parseLines(r, addLine)
|
|
||||||
if err != nil {
|
|
||||||
return nil, err
|
|
||||||
}
|
|
||||||
|
|
||||||
if len(addresses.Ranges()) == 0 {
|
|
||||||
return nil, errNoAddresses
|
|
||||||
}
|
|
||||||
|
|
||||||
return addresses, nil
|
|
||||||
}
|
|
||||||
|
|
||||||
// addICloudPrivateRelayNetwork adds the network at the start of a line of
|
|
||||||
// Apple's iCloud Private Relay file to builder.
|
|
||||||
func addICloudPrivateRelayNetwork(builder *netipx.IPSetBuilder, line string) error {
|
|
||||||
field, _, found := strings.Cut(line, iCloudPrivateRelayFieldSeparator)
|
|
||||||
if !found {
|
|
||||||
return errNoNetwork
|
|
||||||
}
|
|
||||||
|
|
||||||
network, err := netip.ParsePrefix(field)
|
|
||||||
if err != nil {
|
|
||||||
return err
|
|
||||||
}
|
|
||||||
|
|
||||||
builder.AddPrefix(network)
|
|
||||||
|
|
||||||
return nil
|
|
||||||
}
|
|
||||||
|
|
||||||
// readJSON reads r to its end and decodes it, as one JSON document, into v.
|
// readJSON reads r to its end and decodes it, as one JSON document, into v.
|
||||||
func readJSON(r io.Reader, v any) error {
|
func readJSON(r io.Reader, v any) error {
|
||||||
data, err := io.ReadAll(r)
|
data, err := io.ReadAll(r)
|
||||||
|
|||||||
+20
-33
@@ -1,6 +1,6 @@
|
|||||||
// Package ptr looks up the PTR names of addresses over DNS-over-HTTPS. It
|
// Package ptr looks up the PTR names of addresses over DNS-over-HTTPS. It
|
||||||
// keeps answers in memory for as long as DNS allows, and in cache.db so that
|
// keeps each answer in memory for as long as DNS allows, and in cache.db so
|
||||||
// the answers survive a restart.
|
// that the answers survive a restart.
|
||||||
package ptr
|
package ptr
|
||||||
|
|
||||||
import (
|
import (
|
||||||
@@ -29,10 +29,6 @@ const (
|
|||||||
|
|
||||||
// nameSeparator separates the names in a row of the hostnames table.
|
// nameSeparator separates the names in a row of the hostnames table.
|
||||||
nameSeparator = " "
|
nameSeparator = " "
|
||||||
|
|
||||||
// maxAddresses limits how many addresses the cache holds, so that a public
|
|
||||||
// lookup cannot grow memory and cache.db without bound.
|
|
||||||
maxAddresses = 250_000
|
|
||||||
)
|
)
|
||||||
|
|
||||||
// Params is what New needs.
|
// Params is what New needs.
|
||||||
@@ -56,16 +52,15 @@ type entry struct {
|
|||||||
// Cache answers PTR lookups from memory, and asks the resolvers for an address
|
// Cache answers PTR lookups from memory, and asks the resolvers for an address
|
||||||
// it holds no current answer for. Start loads it from cache.db.
|
// it holds no current answer for. Start loads it from cache.db.
|
||||||
type Cache struct {
|
type Cache struct {
|
||||||
db *sql.DB
|
db *sql.DB
|
||||||
resolvers []string
|
resolvers []string
|
||||||
timeLimit time.Duration
|
timeLimit time.Duration
|
||||||
maxEntries int
|
logger *slog.Logger
|
||||||
logger *slog.Logger
|
client *http.Client
|
||||||
client *http.Client
|
now func() time.Time
|
||||||
now func() time.Time
|
|
||||||
|
|
||||||
// entries holds the kept answers by address, at most maxEntries of them.
|
// entries holds every answer by address. pending holds those that
|
||||||
// pending holds those that cache.db does not have yet.
|
// cache.db does not have yet.
|
||||||
mu sync.RWMutex
|
mu sync.RWMutex
|
||||||
entries map[string]entry
|
entries map[string]entry
|
||||||
pending map[string]entry
|
pending map[string]entry
|
||||||
@@ -78,16 +73,15 @@ type Cache struct {
|
|||||||
// New returns an empty Cache. Start loads it.
|
// New returns an empty Cache. Start loads it.
|
||||||
func New(params Params) *Cache {
|
func New(params Params) *Cache {
|
||||||
return &Cache{
|
return &Cache{
|
||||||
db: params.DB,
|
db: params.DB,
|
||||||
resolvers: params.Resolvers,
|
resolvers: params.Resolvers,
|
||||||
timeLimit: LookupTimeout,
|
timeLimit: LookupTimeout,
|
||||||
maxEntries: maxAddresses,
|
logger: params.Logger.With("component", "ptr"),
|
||||||
logger: params.Logger.With("component", "ptr"),
|
client: &http.Client{},
|
||||||
client: &http.Client{},
|
now: time.Now,
|
||||||
now: time.Now,
|
entries: make(map[string]entry),
|
||||||
entries: make(map[string]entry),
|
pending: make(map[string]entry),
|
||||||
pending: make(map[string]entry),
|
done: make(chan struct{}),
|
||||||
done: make(chan struct{}),
|
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -174,18 +168,11 @@ func (c *Cache) current(address string) (entry, bool) {
|
|||||||
return found, true
|
return found, true
|
||||||
}
|
}
|
||||||
|
|
||||||
// store keeps an answer in memory and queues it for cache.db. Once entries
|
// store keeps an answer in memory and queues it for cache.db.
|
||||||
// holds maxEntries addresses it keeps no new address. It still replaces the
|
|
||||||
// answer of an address already held.
|
|
||||||
func (c *Cache) store(address string, answer entry) {
|
func (c *Cache) store(address string, answer entry) {
|
||||||
c.mu.Lock()
|
c.mu.Lock()
|
||||||
defer c.mu.Unlock()
|
defer c.mu.Unlock()
|
||||||
|
|
||||||
_, held := c.entries[address]
|
|
||||||
if !held && len(c.entries) >= c.maxEntries {
|
|
||||||
return
|
|
||||||
}
|
|
||||||
|
|
||||||
c.entries[address] = answer
|
c.entries[address] = answer
|
||||||
c.pending[address] = answer
|
c.pending[address] = answer
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -880,72 +880,3 @@ func TestFlushKeepsAnswersItCouldNotWrite(t *testing.T) {
|
|||||||
t.Errorf("%d rows after the second write, want 1", got)
|
t.Errorf("%d rows after the second write, want 1", got)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// TestNewAddressIsNotKeptAtTheLimit fills a cache that holds one address. A
|
|
||||||
// lookup of a second address still returns its answer, and the cache keeps
|
|
||||||
// nothing for it until the sweep has deleted the first address's expired
|
|
||||||
// answer.
|
|
||||||
func TestNewAddressIsNotKeptAtTheLimit(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
db := openTestDB(t, t.TempDir())
|
|
||||||
resolver := newFakeResolver(t, ptrReply(300, testName))
|
|
||||||
cache, clock := newTestCache(t, db, resolver)
|
|
||||||
cache.maxEntries = 1
|
|
||||||
|
|
||||||
lookupOK(t, cache, testAddress)
|
|
||||||
|
|
||||||
names := lookupOK(t, cache, otherAddress)
|
|
||||||
if !slices.Equal(names, []string{testName}) {
|
|
||||||
t.Errorf("names %v at the limit, want %s", names, testName)
|
|
||||||
}
|
|
||||||
|
|
||||||
_, inEntries := cache.entries[otherAddress]
|
|
||||||
_, inPending := cache.pending[otherAddress]
|
|
||||||
|
|
||||||
if inEntries || inPending {
|
|
||||||
t.Errorf("a new address was kept at the limit: in entries %t, in pending %t",
|
|
||||||
inEntries, inPending)
|
|
||||||
}
|
|
||||||
|
|
||||||
cache.flush(t.Context())
|
|
||||||
|
|
||||||
if got := countRows(t, db); got != 1 {
|
|
||||||
t.Errorf("%d rows at the limit, want 1", got)
|
|
||||||
}
|
|
||||||
|
|
||||||
clock.advance(300 * time.Second)
|
|
||||||
cache.sweep(t.Context())
|
|
||||||
lookupOK(t, cache, otherAddress)
|
|
||||||
|
|
||||||
_, inEntries = cache.entries[otherAddress]
|
|
||||||
if !inEntries {
|
|
||||||
t.Error("a new address was not kept after the sweep made room")
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestHeldAddressIsReplacedAtTheLimit lets the answer of the only address a
|
|
||||||
// cache holds expire. The new answer replaces it in memory and is queued for
|
|
||||||
// cache.db.
|
|
||||||
func TestHeldAddressIsReplacedAtTheLimit(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
resolver := newFakeResolver(t, ptrReply(300, testName))
|
|
||||||
cache, clock := newTestCache(t, openTestDB(t, t.TempDir()), resolver)
|
|
||||||
cache.maxEntries = 1
|
|
||||||
|
|
||||||
lookupOK(t, cache, testAddress)
|
|
||||||
clock.advance(300 * time.Second)
|
|
||||||
resolver.setReply(ptrReply(300, otherName))
|
|
||||||
lookupOK(t, cache, testAddress)
|
|
||||||
|
|
||||||
want := []string{otherName}
|
|
||||||
|
|
||||||
if got := cache.entries[testAddress].names; !slices.Equal(got, want) {
|
|
||||||
t.Errorf("entries holds %v, want %v", got, want)
|
|
||||||
}
|
|
||||||
|
|
||||||
if got := cache.pending[testAddress].names; !slices.Equal(got, want) {
|
|
||||||
t.Errorf("pending holds %v, want %v", got, want)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
@@ -8,8 +8,6 @@ import (
|
|||||||
"net/http"
|
"net/http"
|
||||||
"net/http/httptest"
|
"net/http/httptest"
|
||||||
"net/netip"
|
"net/netip"
|
||||||
"os"
|
|
||||||
"path/filepath"
|
|
||||||
"slices"
|
"slices"
|
||||||
"strconv"
|
"strconv"
|
||||||
"strings"
|
"strings"
|
||||||
@@ -785,74 +783,3 @@ func TestShutdownCheckpointsAndClosesTheDatabaseLast(t *testing.T) {
|
|||||||
t.Errorf("Shutdown made the calls %q, want %q", store.calls, want)
|
t.Errorf("Shutdown made the calls %q, want %q", store.calls, want)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// TestPTRCacheAnswersTheIPLookup starts the PTR cache and the HTTP server as
|
|
||||||
// Run does, and looks up a reserved address, which asks no resolver. Without
|
|
||||||
// the PTR cache the server has no PTR resolver, and every IP lookup answers
|
|
||||||
// 500.
|
|
||||||
func TestPTRCacheAnswersTheIPLookup(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
store := newMockStore()
|
|
||||||
log := logger.New()
|
|
||||||
port := freePort(t)
|
|
||||||
cfg := &config.Config{Port: port, StateDir: t.TempDir()}
|
|
||||||
s := streamer.New(log, metrics.New())
|
|
||||||
|
|
||||||
// A started server reads its routing table in the background, so it
|
|
||||||
// needs one.
|
|
||||||
rw := New(Dependencies{
|
|
||||||
DB: store,
|
|
||||||
Streamer: s,
|
|
||||||
Server: server.New(store, routetable.New(routetable.Params{}), s, log,
|
|
||||||
cfg),
|
|
||||||
Logger: log,
|
|
||||||
Config: cfg,
|
|
||||||
})
|
|
||||||
|
|
||||||
err := rw.startPTRCache(t.Context())
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("failed to start the PTR cache: %v", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
defer rw.stopPTRCache(t.Context())
|
|
||||||
|
|
||||||
err = rw.server.Start(t.Context())
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("failed to start the HTTP server: %v", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
defer func() { _ = rw.server.Stop(t.Context()) }()
|
|
||||||
|
|
||||||
req, err := http.NewRequestWithContext(t.Context(), http.MethodGet,
|
|
||||||
"http://"+net.JoinHostPort("127.0.0.1", strconv.Itoa(port))+
|
|
||||||
"/api/v1/ip/10.0.0.1", nil)
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("failed to build the request: %v", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
resp, err := http.DefaultClient.Do(req)
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("the IP lookup failed: %v", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
defer func() { _ = resp.Body.Close() }()
|
|
||||||
|
|
||||||
body, err := io.ReadAll(resp.Body)
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("failed to read the answer: %v", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
if resp.StatusCode != http.StatusOK {
|
|
||||||
t.Fatalf("status %d, want %d; body %s", resp.StatusCode, http.StatusOK, body)
|
|
||||||
}
|
|
||||||
|
|
||||||
if !strings.Contains(string(body), `"ptr":[]`) {
|
|
||||||
t.Errorf("answer %s, want an empty ptr", body)
|
|
||||||
}
|
|
||||||
|
|
||||||
_, err = os.Stat(filepath.Join(cfg.StateDir, "cache.db"))
|
|
||||||
if err != nil {
|
|
||||||
t.Errorf("cache.db is not in the state directory: %v", err)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
@@ -797,9 +797,6 @@ type IPLookupFlags struct {
|
|||||||
// IsVPN says that the address is on a list of VPN servers.
|
// IsVPN says that the address is on a list of VPN servers.
|
||||||
IsVPN bool `json:"is_vpn"`
|
IsVPN bool `json:"is_vpn"`
|
||||||
|
|
||||||
// IsRelay says that the address is on the iCloud Private Relay list.
|
|
||||||
IsRelay bool `json:"is_relay"`
|
|
||||||
|
|
||||||
// IsBogon says that the address is reserved, as package bogon tells.
|
// IsBogon says that the address is reserved, as package bogon tells.
|
||||||
IsBogon bool `json:"is_bogon"`
|
IsBogon bool `json:"is_bogon"`
|
||||||
}
|
}
|
||||||
@@ -1028,9 +1025,6 @@ func ipLookupFlags(ip string, listings []iplist.Listing) IPLookupFlags {
|
|||||||
IsVPN: slices.ContainsFunc(listings, func(l iplist.Listing) bool {
|
IsVPN: slices.ContainsFunc(listings, func(l iplist.Listing) bool {
|
||||||
return l.VPN
|
return l.VPN
|
||||||
}),
|
}),
|
||||||
IsRelay: slices.ContainsFunc(listings, func(l iplist.Listing) bool {
|
|
||||||
return l.Name == iplist.ICloudPrivateRelayList
|
|
||||||
}),
|
|
||||||
IsBogon: isBogon(ip),
|
IsBogon: isBogon(ip),
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -17,7 +17,6 @@ import (
|
|||||||
"strings"
|
"strings"
|
||||||
"sync/atomic"
|
"sync/atomic"
|
||||||
"testing"
|
"testing"
|
||||||
"testing/synctest"
|
|
||||||
"time"
|
"time"
|
||||||
|
|
||||||
"git.eeqj.de/sneak/routewatch/internal/config"
|
"git.eeqj.de/sneak/routewatch/internal/config"
|
||||||
@@ -175,9 +174,6 @@ type tableStats struct {
|
|||||||
// carried 203.0.113.0/24, the two with kept routes withdrew it, so its origin
|
// carried 203.0.113.0/24, the two with kept routes withdrew it, so its origin
|
||||||
// keeps no route. The oldest and newest routes are those of the other two
|
// keeps no route. The oldest and newest routes are those of the other two
|
||||||
// prefixes.
|
// prefixes.
|
||||||
//
|
|
||||||
// The requests run in a synctest bubble, whose clock is fake, so a stall on
|
|
||||||
// the host does not use up the handlers' 4-second limit.
|
|
||||||
func TestStatsHandlersAnswerFromMemory(t *testing.T) {
|
func TestStatsHandlersAnswerFromMemory(t *testing.T) {
|
||||||
t.Parallel()
|
t.Parallel()
|
||||||
|
|
||||||
@@ -223,11 +219,7 @@ func TestStatsHandlersAnswerFromMemory(t *testing.T) {
|
|||||||
|
|
||||||
for name, handler := range handlers {
|
for name, handler := range handlers {
|
||||||
rec := httptest.NewRecorder()
|
rec := httptest.NewRecorder()
|
||||||
|
handler(rec, httptest.NewRequestWithContext(t.Context(), http.MethodGet, "/", nil))
|
||||||
synctest.Test(t, func(t *testing.T) {
|
|
||||||
handler(rec,
|
|
||||||
httptest.NewRequestWithContext(t.Context(), http.MethodGet, "/", nil))
|
|
||||||
})
|
|
||||||
|
|
||||||
if rec.Code != http.StatusOK {
|
if rec.Code != http.StatusOK {
|
||||||
t.Errorf("%s: status %d, want %d; body %s",
|
t.Errorf("%s: status %d, want %d; body %s",
|
||||||
@@ -262,20 +254,15 @@ func TestStatsHandlersAnswerFromMemory(t *testing.T) {
|
|||||||
// TestStatsAnswerCarriesChurnPerMinute checks that the stream object of the
|
// TestStatsAnswerCarriesChurnPerMinute checks that the stream object of the
|
||||||
// /api/v1/stats answer holds route_churn_per_min as a whole number, and no
|
// /api/v1/stats answer holds route_churn_per_min as a whole number, and no
|
||||||
// longer route_churn_per_sec, the moving average it replaces
|
// longer route_churn_per_sec, the moving average it replaces
|
||||||
// (https://git.eeqj.de/sneak/routewatch/issues/160). The request runs in a
|
// (https://git.eeqj.de/sneak/routewatch/issues/160).
|
||||||
// synctest bubble, whose clock is fake, so a stall on the host does not use up
|
|
||||||
// the handler's 4-second limit.
|
|
||||||
func TestStatsAnswerCarriesChurnPerMinute(t *testing.T) {
|
func TestStatsAnswerCarriesChurnPerMinute(t *testing.T) {
|
||||||
t.Parallel()
|
t.Parallel()
|
||||||
|
|
||||||
s := newServerWithTestRoutes(t)
|
s := newServerWithTestRoutes(t)
|
||||||
|
|
||||||
rec := httptest.NewRecorder()
|
rec := httptest.NewRecorder()
|
||||||
|
s.handleStats()(rec,
|
||||||
synctest.Test(t, func(t *testing.T) {
|
httptest.NewRequestWithContext(t.Context(), http.MethodGet, statsPath, nil))
|
||||||
s.handleStats()(rec,
|
|
||||||
httptest.NewRequestWithContext(t.Context(), http.MethodGet, statsPath, nil))
|
|
||||||
})
|
|
||||||
|
|
||||||
if rec.Code != http.StatusOK {
|
if rec.Code != http.StatusOK {
|
||||||
t.Fatalf("status %d, want %d; body %s", rec.Code, http.StatusOK, rec.Body)
|
t.Fatalf("status %d, want %d; body %s", rec.Code, http.StatusOK, rec.Body)
|
||||||
@@ -323,26 +310,18 @@ func (heldBackHandler) HeldBackCount() int { return heldBackMess
|
|||||||
|
|
||||||
// TestStatsAnswerGivesHowManyMessagesAHandlerHoldsBack registers a handler
|
// TestStatsAnswerGivesHowManyMessagesAHandlerHoldsBack registers a handler
|
||||||
// that holds back messages. Its figures in /api/v1/stats, which the status
|
// that holds back messages. Its figures in /api/v1/stats, which the status
|
||||||
// page shows, must give how many. The request runs in a synctest bubble, whose
|
// page shows, must give how many.
|
||||||
// clock is fake, so a stall on the host does not use up the handler's 4-second
|
|
||||||
// limit.
|
|
||||||
func TestStatsAnswerGivesHowManyMessagesAHandlerHoldsBack(t *testing.T) {
|
func TestStatsAnswerGivesHowManyMessagesAHandlerHoldsBack(t *testing.T) {
|
||||||
t.Parallel()
|
t.Parallel()
|
||||||
|
|
||||||
s := newServerWithTestRoutes(t)
|
s := newServerWithTestRoutes(t)
|
||||||
s.streamer.RegisterHandler(heldBackHandler{})
|
s.streamer.RegisterHandler(heldBackHandler{})
|
||||||
|
|
||||||
var answer []byte
|
|
||||||
|
|
||||||
synctest.Test(t, func(t *testing.T) {
|
|
||||||
answer = get(t, s, statsPath).Body.Bytes()
|
|
||||||
})
|
|
||||||
|
|
||||||
var body struct {
|
var body struct {
|
||||||
Data StatsResponse `json:"data"`
|
Data StatsResponse `json:"data"`
|
||||||
}
|
}
|
||||||
|
|
||||||
err := json.Unmarshal(answer, &body)
|
err := json.Unmarshal(get(t, s, statsPath).Body.Bytes(), &body)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatalf("decoding the answer: %v", err)
|
t.Fatalf("decoding the answer: %v", err)
|
||||||
}
|
}
|
||||||
@@ -386,9 +365,7 @@ type unitStats struct {
|
|||||||
// names. They used to be strings with the unit in them, such as "1h18m34s" or
|
// names. They used to be strings with the unit in them, such as "1h18m34s" or
|
||||||
// "1.1 GB" (https://git.eeqj.de/sneak/routewatch/issues/199). The stream has
|
// "1.1 GB" (https://git.eeqj.de/sneak/routewatch/issues/199). The stream has
|
||||||
// never connected, so the uptime is 0. The memory use in /api/v1/stats is the
|
// never connected, so the uptime is 0. The memory use in /api/v1/stats is the
|
||||||
// same number of bytes as its gc.heap_alloc_bytes. The requests run in a
|
// same number of bytes as its gc.heap_alloc_bytes.
|
||||||
// synctest bubble, whose clock is fake, so a stall on the host does not use up
|
|
||||||
// the handlers' 4-second limit.
|
|
||||||
func TestStatsAnswersGiveUnitsInFieldNames(t *testing.T) {
|
func TestStatsAnswersGiveUnitsInFieldNames(t *testing.T) {
|
||||||
t.Parallel()
|
t.Parallel()
|
||||||
|
|
||||||
@@ -398,11 +375,7 @@ func TestStatsAnswersGiveUnitsInFieldNames(t *testing.T) {
|
|||||||
s.SetASNFetcher(fixedIntervalASNFetcher{interval: fetchInterval})
|
s.SetASNFetcher(fixedIntervalASNFetcher{interval: fetchInterval})
|
||||||
|
|
||||||
for _, path := range []string{statusJSONPath, statsPath} {
|
for _, path := range []string{statusJSONPath, statsPath} {
|
||||||
var answer []byte
|
answer := get(t, s, path).Body.Bytes()
|
||||||
|
|
||||||
synctest.Test(t, func(t *testing.T) {
|
|
||||||
answer = get(t, s, path).Body.Bytes()
|
|
||||||
})
|
|
||||||
|
|
||||||
var body struct {
|
var body struct {
|
||||||
Data unitStats `json:"data"`
|
Data unitStats `json:"data"`
|
||||||
@@ -443,89 +416,82 @@ func TestStatsAnswersGiveUnitsInFieldNames(t *testing.T) {
|
|||||||
// connection_duration_seconds of /api/v1/stats is null while the stream is not
|
// connection_duration_seconds of /api/v1/stats is null while the stream is not
|
||||||
// connected, and a whole number of seconds once it is. It used to be the
|
// connected, and a whole number of seconds once it is. It used to be the
|
||||||
// string "disconnected", or a Go duration such as "1h18m34s"
|
// string "disconnected", or a Go duration such as "1h18m34s"
|
||||||
// (https://git.eeqj.de/sneak/routewatch/issues/199). The test runs in a
|
// (https://git.eeqj.de/sneak/routewatch/issues/199).
|
||||||
// synctest bubble, whose clock is fake, so a stall on the host does not use up
|
|
||||||
// the handler's 4-second limit.
|
|
||||||
func TestStatsAnswerGivesConnectionDurationInSeconds(t *testing.T) {
|
func TestStatsAnswerGivesConnectionDurationInSeconds(t *testing.T) {
|
||||||
t.Parallel()
|
t.Parallel()
|
||||||
|
|
||||||
s := newServerWithTestRoutes(t)
|
s := newServerWithTestRoutes(t)
|
||||||
|
|
||||||
synctest.Test(t, func(t *testing.T) {
|
connectionDuration := func() json.RawMessage {
|
||||||
connectionDuration := func() json.RawMessage {
|
answer := get(t, s, statsPath).Body.Bytes()
|
||||||
answer := get(t, s, statsPath).Body.Bytes()
|
|
||||||
|
|
||||||
var body struct {
|
var body struct {
|
||||||
Data map[string]json.RawMessage `json:"data"`
|
Data map[string]json.RawMessage `json:"data"`
|
||||||
}
|
|
||||||
|
|
||||||
err := json.Unmarshal(answer, &body)
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("decoding the answer: %v; answer %s", err, answer)
|
|
||||||
}
|
|
||||||
|
|
||||||
duration, ok := body.Data["connection_duration_seconds"]
|
|
||||||
if !ok {
|
|
||||||
t.Fatalf("no connection_duration_seconds: %s", answer)
|
|
||||||
}
|
|
||||||
|
|
||||||
return duration
|
|
||||||
}
|
}
|
||||||
|
|
||||||
disconnected := connectionDuration()
|
err := json.Unmarshal(answer, &body)
|
||||||
if string(disconnected) != "null" {
|
if err != nil {
|
||||||
t.Errorf("disconnected: connection_duration_seconds = %s, want null",
|
t.Fatalf("decoding the answer: %v; answer %s", err, answer)
|
||||||
disconnected)
|
|
||||||
}
|
}
|
||||||
|
|
||||||
s.streamer.GetMetricsTracker().SetConnected(true)
|
duration, ok := body.Data["connection_duration_seconds"]
|
||||||
|
if !ok {
|
||||||
var seconds int64
|
t.Fatalf("no connection_duration_seconds: %s", answer)
|
||||||
|
|
||||||
connected := connectionDuration()
|
|
||||||
|
|
||||||
err := json.Unmarshal(connected, &seconds)
|
|
||||||
if err != nil || seconds != 0 {
|
|
||||||
t.Errorf("just connected: connection_duration_seconds = %s, want 0",
|
|
||||||
connected)
|
|
||||||
}
|
}
|
||||||
})
|
|
||||||
|
return duration
|
||||||
|
}
|
||||||
|
|
||||||
|
disconnected := connectionDuration()
|
||||||
|
if string(disconnected) != "null" {
|
||||||
|
t.Errorf("disconnected: connection_duration_seconds = %s, want null",
|
||||||
|
disconnected)
|
||||||
|
}
|
||||||
|
|
||||||
|
s.streamer.GetMetricsTracker().SetConnected(true)
|
||||||
|
|
||||||
|
var seconds int64
|
||||||
|
|
||||||
|
connected := connectionDuration()
|
||||||
|
|
||||||
|
err := json.Unmarshal(connected, &seconds)
|
||||||
|
if err != nil || seconds != 0 {
|
||||||
|
t.Errorf("just connected: connection_duration_seconds = %s, want 0",
|
||||||
|
connected)
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// TestStreamDurationsCountSeconds checks that the uptime and the connection
|
// TestStreamDurationsCountSeconds checks that the uptime and the connection
|
||||||
// duration of a stream that connected 90 seconds ago are both 90. A stream
|
// duration of a stream that connected 90 seconds ago are both 90. A stream
|
||||||
// that has never connected has an uptime of 0, and one that is not connected
|
// that has never connected has an uptime of 0, and one that is not connected
|
||||||
// has no connection duration. The test runs in a synctest bubble, whose clock
|
// has no connection duration.
|
||||||
// is fake, so a stall on the host does not add to the 90 seconds.
|
|
||||||
func TestStreamDurationsCountSeconds(t *testing.T) {
|
func TestStreamDurationsCountSeconds(t *testing.T) {
|
||||||
t.Parallel()
|
t.Parallel()
|
||||||
|
|
||||||
synctest.Test(t, func(t *testing.T) {
|
const connectedFor = 90 * time.Second
|
||||||
const connectedFor = 90 * time.Second
|
|
||||||
|
|
||||||
connectedSince := time.Now().Add(-connectedFor)
|
connectedSince := time.Now().Add(-connectedFor)
|
||||||
want := int64(connectedFor / time.Second)
|
want := int64(connectedFor / time.Second)
|
||||||
|
|
||||||
if got := uptimeSeconds(connectedSince); got != want {
|
if got := uptimeSeconds(connectedSince); got != want {
|
||||||
t.Errorf("uptimeSeconds = %d, want %d", got, want)
|
t.Errorf("uptimeSeconds = %d, want %d", got, want)
|
||||||
}
|
}
|
||||||
|
|
||||||
if got := uptimeSeconds(time.Time{}); got != 0 {
|
if got := uptimeSeconds(time.Time{}); got != 0 {
|
||||||
t.Errorf("uptimeSeconds before any connection = %d, want 0", got)
|
t.Errorf("uptimeSeconds before any connection = %d, want 0", got)
|
||||||
}
|
}
|
||||||
|
|
||||||
switch got := connectionDurationSeconds(true, connectedSince); {
|
switch got := connectionDurationSeconds(true, connectedSince); {
|
||||||
case got == nil:
|
case got == nil:
|
||||||
t.Errorf("connectionDurationSeconds = nil, want %d", want)
|
t.Errorf("connectionDurationSeconds = nil, want %d", want)
|
||||||
case *got != want:
|
case *got != want:
|
||||||
t.Errorf("connectionDurationSeconds = %d, want %d", *got, want)
|
t.Errorf("connectionDurationSeconds = %d, want %d", *got, want)
|
||||||
}
|
}
|
||||||
|
|
||||||
if got := connectionDurationSeconds(false, connectedSince); got != nil {
|
if got := connectionDurationSeconds(false, connectedSince); got != nil {
|
||||||
t.Errorf("connectionDurationSeconds while not connected = %d, want nil",
|
t.Errorf("connectionDurationSeconds while not connected = %d, want nil",
|
||||||
*got)
|
*got)
|
||||||
}
|
}
|
||||||
})
|
|
||||||
}
|
}
|
||||||
|
|
||||||
// newTestTable returns a routing table that holds a route for testPrefixV4,
|
// newTestTable returns a routing table that holds a route for testPrefixV4,
|
||||||
@@ -1172,9 +1138,7 @@ func (failingDB) GetPrefixFirstSeen(
|
|||||||
// with the driver's error text. It also sends a prefix that does not parse.
|
// with the driver's error text. It also sends a prefix that does not parse.
|
||||||
// No answer may carry the driver's or the prefix parser's text. Before the fix
|
// No answer may carry the driver's or the prefix parser's text. Before the fix
|
||||||
// the answers sent it to the client
|
// the answers sent it to the client
|
||||||
// (https://git.eeqj.de/sneak/routewatch/issues/108). The requests run in a
|
// (https://git.eeqj.de/sneak/routewatch/issues/108).
|
||||||
// synctest bubble, whose clock is fake, so a stall on the host does not use up
|
|
||||||
// the stats handlers' 4-second limit.
|
|
||||||
func TestJSONRoutesDoNotSendErrorDetails(t *testing.T) {
|
func TestJSONRoutesDoNotSendErrorDetails(t *testing.T) {
|
||||||
t.Parallel()
|
t.Parallel()
|
||||||
|
|
||||||
@@ -1195,11 +1159,8 @@ func TestJSONRoutesDoNotSendErrorDetails(t *testing.T) {
|
|||||||
|
|
||||||
for _, tt := range tests {
|
for _, tt := range tests {
|
||||||
rec := httptest.NewRecorder()
|
rec := httptest.NewRecorder()
|
||||||
|
s.router.ServeHTTP(rec, httptest.NewRequestWithContext(
|
||||||
synctest.Test(t, func(t *testing.T) {
|
t.Context(), http.MethodGet, tt.path, nil))
|
||||||
s.router.ServeHTTP(rec, httptest.NewRequestWithContext(
|
|
||||||
t.Context(), http.MethodGet, tt.path, nil))
|
|
||||||
})
|
|
||||||
|
|
||||||
if rec.Code != tt.wantCode {
|
if rec.Code != tt.wantCode {
|
||||||
t.Errorf("%s: status %d, want %d; body %s",
|
t.Errorf("%s: status %d, want %d; body %s",
|
||||||
@@ -1453,11 +1414,8 @@ func TestIPLookupKeepsRouteWhenPTRLookupDoesNotAnswer(t *testing.T) {
|
|||||||
s := newServerWithTestRoutes(t)
|
s := newServerWithTestRoutes(t)
|
||||||
useFakeResolver(s, &fakeResolver{ptrNeverAnswers: true})
|
useFakeResolver(s, &fakeResolver{ptrNeverAnswers: true})
|
||||||
|
|
||||||
// The request's time limit passes well after the PTR lookup's, so that a
|
// The request's time limit passes after the PTR lookup's.
|
||||||
// stall on a busy host does not use up the difference.
|
ctx, cancel := context.WithTimeout(t.Context(), 2*ptr.LookupTimeout)
|
||||||
const requestTimeLimit = ptr.LookupTimeout + 15*time.Second
|
|
||||||
|
|
||||||
ctx, cancel := context.WithTimeout(t.Context(), requestTimeLimit)
|
|
||||||
defer cancel()
|
defer cancel()
|
||||||
|
|
||||||
rec := httptest.NewRecorder()
|
rec := httptest.NewRecorder()
|
||||||
@@ -1504,10 +1462,6 @@ const listedUnroutedAddr = "203.0.113.1"
|
|||||||
// torExitAddr is the address that fakeIPLists lists on the Tor exit list.
|
// torExitAddr is the address that fakeIPLists lists on the Tor exit list.
|
||||||
const torExitAddr = "203.0.113.2"
|
const torExitAddr = "203.0.113.2"
|
||||||
|
|
||||||
// relayAddr is the address that fakeIPLists lists on the iCloud Private Relay
|
|
||||||
// list.
|
|
||||||
const relayAddr = "203.0.113.5"
|
|
||||||
|
|
||||||
// The addresses that fakeIPLists lists on one and on two VPN lists, and the
|
// The addresses that fakeIPLists lists on one and on two VPN lists, and the
|
||||||
// names of those lists.
|
// names of those lists.
|
||||||
const (
|
const (
|
||||||
@@ -1517,11 +1471,10 @@ const (
|
|||||||
otherTestVPN = "other_test_vpn"
|
otherTestVPN = "other_test_vpn"
|
||||||
)
|
)
|
||||||
|
|
||||||
// fakeIPLists has five lists. The test list holds addrInTestPrefixV4,
|
// fakeIPLists has four lists. The test list holds addrInTestPrefixV4,
|
||||||
// listedUnroutedAddr and twoVPNListsAddr. The Tor exit list holds torExitAddr.
|
// listedUnroutedAddr and twoVPNListsAddr. The Tor exit list holds torExitAddr.
|
||||||
// The iCloud Private Relay list holds relayAddr. The test VPN list holds
|
// The test VPN list holds oneVPNListAddr and twoVPNListsAddr, and the other
|
||||||
// oneVPNListAddr and twoVPNListsAddr, and the other test VPN list holds
|
// test VPN list holds twoVPNListsAddr.
|
||||||
// twoVPNListsAddr.
|
|
||||||
type fakeIPLists struct{}
|
type fakeIPLists struct{}
|
||||||
|
|
||||||
func testListing() iplist.Listing {
|
func testListing() iplist.Listing {
|
||||||
@@ -1542,10 +1495,6 @@ func (fakeIPLists) Lookup(addr netip.Addr) []iplist.Listing {
|
|||||||
return []iplist.Listing{{Name: iplist.TorExitsList}}
|
return []iplist.Listing{{Name: iplist.TorExitsList}}
|
||||||
}
|
}
|
||||||
|
|
||||||
if addr == netip.MustParseAddr(relayAddr) {
|
|
||||||
return []iplist.Listing{{Name: iplist.ICloudPrivateRelayList}}
|
|
||||||
}
|
|
||||||
|
|
||||||
if addr == netip.MustParseAddr(oneVPNListAddr) {
|
if addr == netip.MustParseAddr(oneVPNListAddr) {
|
||||||
return []iplist.Listing{{Name: testVPNList, VPN: true}}
|
return []iplist.Listing{{Name: testVPNList, VPN: true}}
|
||||||
}
|
}
|
||||||
@@ -1578,22 +1527,20 @@ func TestIPLookupFlags(t *testing.T) {
|
|||||||
name string
|
name string
|
||||||
addr string
|
addr string
|
||||||
tor bool
|
tor bool
|
||||||
relay bool
|
|
||||||
bogon bool
|
bogon bool
|
||||||
}{
|
}{
|
||||||
{"on the Tor exit list", torExitAddr, true, false, true},
|
{"on the Tor exit list", torExitAddr, true, true},
|
||||||
{"on the iCloud Private Relay list", relayAddr, false, true, true},
|
{"on another list", listedUnroutedAddr, false, true},
|
||||||
{"on another list", listedUnroutedAddr, false, false, true},
|
{"routed IPv4", addrInTestPrefixV4, false, true},
|
||||||
{"routed IPv4", addrInTestPrefixV4, false, false, true},
|
{"routed IPv6", addrInTestPrefixV6, false, true},
|
||||||
{"routed IPv6", addrInTestPrefixV6, false, false, true},
|
{"private IPv4", "10.0.0.1", false, true},
|
||||||
{"private IPv4", "10.0.0.1", false, false, true},
|
{"loopback IPv4", "127.0.0.1", false, true},
|
||||||
{"loopback IPv4", "127.0.0.1", false, false, true},
|
{"public IPv4", "1.1.1.1", false, false},
|
||||||
{"public IPv4", "1.1.1.1", false, false, false},
|
{"private IPv6", "fd00::1", false, true},
|
||||||
{"private IPv6", "fd00::1", false, false, true},
|
{"loopback IPv6", "::1", false, true},
|
||||||
{"loopback IPv6", "::1", false, false, true},
|
{"public IPv6", "2606:4700:4700::1111", false, false},
|
||||||
{"public IPv6", "2606:4700:4700::1111", false, false, false},
|
{"public IPv4 in IPv6 form", "::ffff:1.1.1.1", false, false},
|
||||||
{"public IPv4 in IPv6 form", "::ffff:1.1.1.1", false, false, false},
|
{"private IPv4 in IPv6 form", "::ffff:10.0.0.1", false, true},
|
||||||
{"private IPv4 in IPv6 form", "::ffff:10.0.0.1", false, false, true},
|
|
||||||
}
|
}
|
||||||
|
|
||||||
for _, tt := range tests {
|
for _, tt := range tests {
|
||||||
@@ -1615,8 +1562,7 @@ func TestIPLookupFlags(t *testing.T) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
want := map[string]bool{
|
want := map[string]bool{
|
||||||
"is_tor": tt.tor, "is_vpn": false, "is_relay": tt.relay,
|
"is_tor": tt.tor, "is_vpn": false, "is_bogon": tt.bogon,
|
||||||
"is_bogon": tt.bogon,
|
|
||||||
}
|
}
|
||||||
if !maps.Equal(got, want) {
|
if !maps.Equal(got, want) {
|
||||||
t.Errorf("%s: flags %v, want %v", path, got, want)
|
t.Errorf("%s: flags %v, want %v", path, got, want)
|
||||||
@@ -2468,13 +2414,9 @@ func settledGoroutineCount() int {
|
|||||||
return prev
|
return prev
|
||||||
}
|
}
|
||||||
|
|
||||||
// waitForGoroutines waits up to 15 seconds for the goroutine count to drop to
|
// waitForGoroutines waits until the goroutine count drops to target or below.
|
||||||
// target or below.
|
|
||||||
func waitForGoroutines(target int) bool {
|
func waitForGoroutines(target int) bool {
|
||||||
const maxWait = 15 * time.Second
|
for range 100 {
|
||||||
|
|
||||||
deadline := time.Now().Add(maxWait)
|
|
||||||
for time.Now().Before(deadline) {
|
|
||||||
if runtime.NumGoroutine() <= target {
|
if runtime.NumGoroutine() <= target {
|
||||||
return true
|
return true
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -76,9 +76,7 @@ func asLink(asn int) string {
|
|||||||
}
|
}
|
||||||
|
|
||||||
// TestHomePageCardsShowTheTableFigures fills a routing table and the asns
|
// TestHomePageCardsShowTheTableFigures fills a routing table and the asns
|
||||||
// table, works out the cards, and checks that each card shows its figure. The
|
// table, works out the cards, and checks that each card shows its figure.
|
||||||
// cards are worked out in a synctest bubble, whose clock is fake, so a stall on
|
|
||||||
// the host does not use up the 4-second limit on their database read.
|
|
||||||
func TestHomePageCardsShowTheTableFigures(t *testing.T) {
|
func TestHomePageCardsShowTheTableFigures(t *testing.T) {
|
||||||
t.Parallel()
|
t.Parallel()
|
||||||
|
|
||||||
@@ -97,9 +95,7 @@ func TestHomePageCardsShowTheTableFigures(t *testing.T) {
|
|||||||
t.Fatalf("GetOrCreateASNBatch: %v", err)
|
t.Fatalf("GetOrCreateASNBatch: %v", err)
|
||||||
}
|
}
|
||||||
|
|
||||||
synctest.Test(t, func(t *testing.T) {
|
s.refreshHomeCards(t.Context())
|
||||||
s.refreshHomeCards(t.Context())
|
|
||||||
})
|
|
||||||
|
|
||||||
page := get(t, s, "/").Body.String()
|
page := get(t, s, "/").Body.String()
|
||||||
|
|
||||||
@@ -283,7 +279,7 @@ func TestKeepHomeCardsFreshWorksThemOutEveryInterval(t *testing.T) {
|
|||||||
|
|
||||||
const (
|
const (
|
||||||
interval = 10 * time.Millisecond
|
interval = 10 * time.Millisecond
|
||||||
maxWait = 15 * time.Second
|
maxWait = 5 * time.Second
|
||||||
)
|
)
|
||||||
|
|
||||||
table := emptyTable()
|
table := emptyTable()
|
||||||
@@ -325,7 +321,7 @@ func TestKeepHomeCardsFreshWorksThemOutEveryInterval(t *testing.T) {
|
|||||||
func TestStartedServerWorksOutTheCards(t *testing.T) {
|
func TestStartedServerWorksOutTheCards(t *testing.T) {
|
||||||
t.Parallel()
|
t.Parallel()
|
||||||
|
|
||||||
const maxWait = 15 * time.Second
|
const maxWait = 5 * time.Second
|
||||||
|
|
||||||
s := newServerWithTable(t, newCardsTable())
|
s := newServerWithTable(t, newCardsTable())
|
||||||
|
|
||||||
|
|||||||
@@ -14,7 +14,6 @@ import (
|
|||||||
"regexp"
|
"regexp"
|
||||||
"strings"
|
"strings"
|
||||||
"testing"
|
"testing"
|
||||||
"testing/synctest"
|
|
||||||
"time"
|
"time"
|
||||||
|
|
||||||
"git.eeqj.de/sneak/routewatch/internal/config"
|
"git.eeqj.de/sneak/routewatch/internal/config"
|
||||||
@@ -164,10 +163,6 @@ func TestJSONAnswersPutNoUnitInAString(t *testing.T) {
|
|||||||
// router and the middleware write. Before the fix a panic in a handler, and a
|
// router and the middleware write. Before the fix a panic in a handler, and a
|
||||||
// method /status.json or the health check does not take, answered with an
|
// method /status.json or the health check does not take, answered with an
|
||||||
// empty body (https://git.eeqj.de/sneak/routewatch/issues/201).
|
// empty body (https://git.eeqj.de/sneak/routewatch/issues/201).
|
||||||
//
|
|
||||||
// The requests run in a synctest bubble, whose clock is fake, so a stall on
|
|
||||||
// the host does not use up the 2-second limit of the health check or the
|
|
||||||
// 4-second limit of the stats handlers.
|
|
||||||
func TestJSONAnswersCarryMeta(t *testing.T) {
|
func TestJSONAnswersCarryMeta(t *testing.T) {
|
||||||
t.Parallel()
|
t.Parallel()
|
||||||
|
|
||||||
@@ -186,10 +181,6 @@ func TestJSONAnswersCarryMeta(t *testing.T) {
|
|||||||
s.router.Get(panicsPath, JSONValidationMiddleware(http.HandlerFunc(
|
s.router.Get(panicsPath, JSONValidationMiddleware(http.HandlerFunc(
|
||||||
func(http.ResponseWriter, *http.Request) { panic(errDiskIO) })).ServeHTTP)
|
func(http.ResponseWriter, *http.Request) { panic(errDiskIO) })).ServeHTTP)
|
||||||
|
|
||||||
failing := New(failingDB{}, newTestTable(),
|
|
||||||
streamer.New(logger.New(), metrics.New()), logger.New(), &config.Config{})
|
|
||||||
useFakeResolver(failing, &fakeResolver{})
|
|
||||||
|
|
||||||
tests := []struct {
|
tests := []struct {
|
||||||
method string
|
method string
|
||||||
path string
|
path string
|
||||||
@@ -225,40 +216,42 @@ func TestJSONAnswersCarryMeta(t *testing.T) {
|
|||||||
{http.MethodGet, panicsPath, http.StatusInternalServerError},
|
{http.MethodGet, panicsPath, http.StatusInternalServerError},
|
||||||
}
|
}
|
||||||
|
|
||||||
synctest.Test(t, func(t *testing.T) {
|
for _, tt := range tests {
|
||||||
for _, tt := range tests {
|
checkAnswerCarriesMeta(t, s, httptest.NewRequestWithContext(
|
||||||
checkAnswerCarriesMeta(t, s, httptest.NewRequestWithContext(
|
t.Context(), tt.method, tt.path, nil), tt.wantCode)
|
||||||
t.Context(), tt.method, tt.path, nil), tt.wantCode)
|
}
|
||||||
}
|
|
||||||
|
|
||||||
// A request whose time limit has already passed answers 504. The
|
// A request whose time limit has already passed answers 504. The handlers
|
||||||
// handlers write the answers of the first five routes, and the JSON
|
// write the answers of the first five routes, and the JSON check on the
|
||||||
// check on the route writes the last.
|
// route writes the last.
|
||||||
pastTimeLimit, cancel := context.WithDeadline(t.Context(), time.Now())
|
pastTimeLimit, cancel := context.WithDeadline(t.Context(), time.Now())
|
||||||
defer cancel()
|
defer cancel()
|
||||||
|
|
||||||
for _, path := range []string{
|
for _, path := range []string{
|
||||||
statusJSONPath, statsPath, "/ip/" + addrInTestPrefixV4,
|
statusJSONPath, statsPath, "/ip/" + addrInTestPrefixV4,
|
||||||
"/api/v1/ip/" + addrInTestPrefixV4, asJSONPath, writesNothingPath,
|
"/api/v1/ip/" + addrInTestPrefixV4, asJSONPath, writesNothingPath,
|
||||||
} {
|
} {
|
||||||
checkAnswerCarriesMeta(t, s, httptest.NewRequestWithContext(
|
checkAnswerCarriesMeta(t, s, httptest.NewRequestWithContext(
|
||||||
pastTimeLimit, http.MethodGet, path, nil), http.StatusGatewayTimeout)
|
pastTimeLimit, http.MethodGet, path, nil), http.StatusGatewayTimeout)
|
||||||
}
|
}
|
||||||
|
|
||||||
// With a store whose reads fail, the health check answers 503, and
|
// With a store whose reads fail, the health check answers 503, and the
|
||||||
// the handlers of the other routes 500.
|
// handlers of the other routes 500.
|
||||||
|
failing := New(failingDB{}, newTestTable(),
|
||||||
|
streamer.New(logger.New(), metrics.New()), logger.New(), &config.Config{})
|
||||||
|
useFakeResolver(failing, &fakeResolver{})
|
||||||
|
|
||||||
|
checkAnswerCarriesMeta(t, failing, httptest.NewRequestWithContext(
|
||||||
|
t.Context(), http.MethodGet, healthCheckPath, nil),
|
||||||
|
http.StatusServiceUnavailable)
|
||||||
|
|
||||||
|
for _, path := range []string{
|
||||||
|
statusJSONPath, statsPath, asJSONPath,
|
||||||
|
"/api/v1/ip/" + addrInTestPrefixV4, "/ip/" + addrInTestPrefixV4,
|
||||||
|
} {
|
||||||
checkAnswerCarriesMeta(t, failing, httptest.NewRequestWithContext(
|
checkAnswerCarriesMeta(t, failing, httptest.NewRequestWithContext(
|
||||||
t.Context(), http.MethodGet, healthCheckPath, nil),
|
t.Context(), http.MethodGet, path, nil), http.StatusInternalServerError)
|
||||||
http.StatusServiceUnavailable)
|
}
|
||||||
|
|
||||||
for _, path := range []string{
|
|
||||||
statusJSONPath, statsPath, asJSONPath,
|
|
||||||
"/api/v1/ip/" + addrInTestPrefixV4, "/ip/" + addrInTestPrefixV4,
|
|
||||||
} {
|
|
||||||
checkAnswerCarriesMeta(t, failing, httptest.NewRequestWithContext(
|
|
||||||
t.Context(), http.MethodGet, path, nil), http.StatusInternalServerError)
|
|
||||||
}
|
|
||||||
})
|
|
||||||
}
|
}
|
||||||
|
|
||||||
// checkAnswerCarriesMeta serves req through the server's router, and checks
|
// checkAnswerCarriesMeta serves req through the server's router, and checks
|
||||||
|
|||||||
@@ -5,7 +5,6 @@ import (
|
|||||||
"net/http"
|
"net/http"
|
||||||
"net/http/httptest"
|
"net/http/httptest"
|
||||||
"testing"
|
"testing"
|
||||||
"testing/synctest"
|
|
||||||
)
|
)
|
||||||
|
|
||||||
// TestRoutesAnswerHEAD sends HEAD for a page and JSON routes through the
|
// TestRoutesAnswerHEAD sends HEAD for a page and JSON routes through the
|
||||||
@@ -13,23 +12,15 @@ import (
|
|||||||
// lookup of the caller's own address, are in the list because chi's GetHead
|
// lookup of the caller's own address, are in the list because chi's GetHead
|
||||||
// does not reach them. Before the fix every route answered HEAD with 405
|
// does not reach them. Before the fix every route answered HEAD with 405
|
||||||
// (https://git.eeqj.de/sneak/routewatch/issues/111).
|
// (https://git.eeqj.de/sneak/routewatch/issues/111).
|
||||||
//
|
|
||||||
// The requests run in a synctest bubble, whose clock is fake, so a stall on
|
|
||||||
// the host does not use up the stats handlers' 4-second limit. The resolver
|
|
||||||
// is fake because no time limit would cut off a real DNS lookup in a bubble.
|
|
||||||
func TestRoutesAnswerHEAD(t *testing.T) {
|
func TestRoutesAnswerHEAD(t *testing.T) {
|
||||||
t.Parallel()
|
t.Parallel()
|
||||||
|
|
||||||
s := newServerWithTestRoutes(t)
|
s := newServerWithTestRoutes(t)
|
||||||
s.resolver = &fakeResolver{}
|
|
||||||
|
|
||||||
for _, path := range []string{"/", statusJSONPath, statsPath, "/ip", "/ip/"} {
|
for _, path := range []string{"/", statusJSONPath, statsPath, "/ip", "/ip/"} {
|
||||||
rec := httptest.NewRecorder()
|
rec := httptest.NewRecorder()
|
||||||
|
s.router.ServeHTTP(rec, httptest.NewRequestWithContext(
|
||||||
synctest.Test(t, func(t *testing.T) {
|
t.Context(), http.MethodHead, path, nil))
|
||||||
s.router.ServeHTTP(rec, httptest.NewRequestWithContext(
|
|
||||||
t.Context(), http.MethodHead, path, nil))
|
|
||||||
})
|
|
||||||
|
|
||||||
if rec.Code != http.StatusOK {
|
if rec.Code != http.StatusOK {
|
||||||
t.Errorf("HEAD %s: status %d, want %d; body %s",
|
t.Errorf("HEAD %s: status %d, want %d; body %s",
|
||||||
|
|||||||
@@ -22,10 +22,10 @@ func TestStopClosesTheConnectionOfARequestStillRunning(t *testing.T) {
|
|||||||
stopDeadline = 100 * time.Millisecond
|
stopDeadline = 100 * time.Millisecond
|
||||||
// waitTimeout is how long the test waits for the request to reach
|
// waitTimeout is how long the test waits for the request to reach
|
||||||
// the handler, and then for the client to see its connection closed.
|
// the handler, and then for the client to see its connection closed.
|
||||||
waitTimeout = 15 * time.Second
|
waitTimeout = 2 * time.Second
|
||||||
// readHeaderTimeout bounds the test server's read of the request
|
// readHeaderTimeout bounds the test server's read of the request
|
||||||
// header, as every http.Server must.
|
// header, as every http.Server must.
|
||||||
readHeaderTimeout = waitTimeout
|
readHeaderTimeout = time.Second
|
||||||
)
|
)
|
||||||
|
|
||||||
release := make(chan struct{})
|
release := make(chan struct{})
|
||||||
|
|||||||
@@ -14,7 +14,6 @@ import (
|
|||||||
"strings"
|
"strings"
|
||||||
"sync/atomic"
|
"sync/atomic"
|
||||||
"testing"
|
"testing"
|
||||||
"testing/synctest"
|
|
||||||
"time"
|
"time"
|
||||||
|
|
||||||
"git.eeqj.de/sneak/routewatch/internal/logger"
|
"git.eeqj.de/sneak/routewatch/internal/logger"
|
||||||
@@ -399,7 +398,7 @@ func TestStopQueuesTheHeldStateMessages(t *testing.T) {
|
|||||||
const (
|
const (
|
||||||
held = 2
|
held = 2
|
||||||
lines = held + 1
|
lines = held + 1
|
||||||
maxWait = 15 * time.Second
|
maxWait = 5 * time.Second
|
||||||
)
|
)
|
||||||
|
|
||||||
srv := httptest.NewServer(http.HandlerFunc(
|
srv := httptest.NewServer(http.HandlerFunc(
|
||||||
@@ -479,15 +478,15 @@ func TestStopQueuesTheHeldStateMessages(t *testing.T) {
|
|||||||
// TestStopReturnsByItsDeadline stops a streamer whose handler has a full queue
|
// TestStopReturnsByItsDeadline stops a streamer whose handler has a full queue
|
||||||
// and a STATE message held for it, and a worker that stays busy until the test
|
// and a STATE message held for it, and a worker that stays busy until the test
|
||||||
// ends, as one does whose handler waits for a database write. Stop must return
|
// ends, as one does whose handler waits for a database write. Stop must return
|
||||||
// within its deadline plus a margin instead of waiting for room on the queue
|
// within its deadline plus a small margin instead of waiting for room on the
|
||||||
// and for the worker, and log as not handed over the held message and every
|
// queue and for the worker, and log as not handed over the held message and
|
||||||
// message on the queue.
|
// every message on the queue.
|
||||||
func TestStopReturnsByItsDeadline(t *testing.T) {
|
func TestStopReturnsByItsDeadline(t *testing.T) {
|
||||||
t.Parallel()
|
t.Parallel()
|
||||||
|
|
||||||
const (
|
const (
|
||||||
deadline = 100 * time.Millisecond
|
deadline = 100 * time.Millisecond
|
||||||
margin = 15 * time.Second
|
margin = 2 * time.Second
|
||||||
)
|
)
|
||||||
|
|
||||||
var logs bytes.Buffer
|
var logs bytes.Buffer
|
||||||
@@ -578,7 +577,7 @@ func TestStopBeforeMessageReachesQueues(t *testing.T) {
|
|||||||
func TestReconnectWhenStreamEnds(t *testing.T) {
|
func TestReconnectWhenStreamEnds(t *testing.T) {
|
||||||
t.Parallel()
|
t.Parallel()
|
||||||
|
|
||||||
endResponse := pipeTransport{write: func(context.Context, io.Writer) {}}
|
endResponse := func(http.ResponseWriter, *http.Request) {}
|
||||||
checkReconnect(t, endResponse, streamIdleTimeout)
|
checkReconnect(t, endResponse, streamIdleTimeout)
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -590,8 +589,9 @@ func TestReconnectWhenStreamGoesIdle(t *testing.T) {
|
|||||||
|
|
||||||
const idleTimeout = 50 * time.Millisecond
|
const idleTimeout = 50 * time.Millisecond
|
||||||
|
|
||||||
holdResponse := pipeTransport{
|
holdResponse := func(w http.ResponseWriter, r *http.Request) {
|
||||||
write: func(ctx context.Context, _ io.Writer) { <-ctx.Done() },
|
flush(w)
|
||||||
|
<-r.Context().Done()
|
||||||
}
|
}
|
||||||
checkReconnect(t, holdResponse, idleTimeout)
|
checkReconnect(t, holdResponse, idleTimeout)
|
||||||
}
|
}
|
||||||
@@ -604,51 +604,16 @@ func TestReconnectWhenServerNeverAnswers(t *testing.T) {
|
|||||||
|
|
||||||
const idleTimeout = 50 * time.Millisecond
|
const idleTimeout = 50 * time.Millisecond
|
||||||
|
|
||||||
neverAnswer := roundTripFunc(
|
neverAnswer := func(_ http.ResponseWriter, r *http.Request) {
|
||||||
func(req *http.Request) (*http.Response, error) {
|
<-r.Context().Done()
|
||||||
<-req.Context().Done()
|
}
|
||||||
|
|
||||||
return nil, req.Context().Err()
|
|
||||||
},
|
|
||||||
)
|
|
||||||
checkReconnect(t, neverAnswer, idleTimeout)
|
checkReconnect(t, neverAnswer, idleTimeout)
|
||||||
}
|
}
|
||||||
|
|
||||||
// pipeTransport answers every request with a response whose body write fills.
|
// TestStreamNotDroppedWhileLinesArrive has the server send lines well within
|
||||||
// The body fails when the request's context ends, as a connection's does.
|
// the idle timeout for several times the timeout, then end the response. Each
|
||||||
// write gets that context, so that it can end when the request does.
|
// line must start the idle timeout over, so the stream lasts until the server
|
||||||
type pipeTransport struct {
|
// ends it instead of being dropped as idle.
|
||||||
write func(context.Context, io.Writer)
|
|
||||||
}
|
|
||||||
|
|
||||||
func (p pipeTransport) RoundTrip(req *http.Request) (*http.Response, error) {
|
|
||||||
body, w := io.Pipe()
|
|
||||||
|
|
||||||
context.AfterFunc(req.Context(), func() {
|
|
||||||
_ = w.CloseWithError(req.Context().Err())
|
|
||||||
})
|
|
||||||
|
|
||||||
go func(ctx context.Context) {
|
|
||||||
p.write(ctx, w)
|
|
||||||
|
|
||||||
_ = w.Close()
|
|
||||||
}(req.Context())
|
|
||||||
|
|
||||||
return &http.Response{StatusCode: http.StatusOK, Body: body}, nil
|
|
||||||
}
|
|
||||||
|
|
||||||
// roundTripFunc makes a function an http.RoundTripper.
|
|
||||||
type roundTripFunc func(*http.Request) (*http.Response, error)
|
|
||||||
|
|
||||||
func (f roundTripFunc) RoundTrip(req *http.Request) (*http.Response, error) {
|
|
||||||
return f(req)
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestStreamNotDroppedWhileLinesArrive has a response send lines well within
|
|
||||||
// the idle timeout for several times the timeout, then end. Each line must
|
|
||||||
// start the idle timeout over, so the stream lasts until the response ends
|
|
||||||
// instead of being dropped as idle. It runs in a synctest bubble, whose clock
|
|
||||||
// is fake, so a stall on the host does not use up the timeout.
|
|
||||||
func TestStreamNotDroppedWhileLinesArrive(t *testing.T) {
|
func TestStreamNotDroppedWhileLinesArrive(t *testing.T) {
|
||||||
t.Parallel()
|
t.Parallel()
|
||||||
|
|
||||||
@@ -658,35 +623,26 @@ func TestStreamNotDroppedWhileLinesArrive(t *testing.T) {
|
|||||||
sendFor = 5 * idleTimeout
|
sendFor = 5 * idleTimeout
|
||||||
)
|
)
|
||||||
|
|
||||||
// The streamer is made outside the bubble. Its metrics start a goroutine
|
srv := httptest.NewServer(http.HandlerFunc(
|
||||||
// that never returns, and a bubble waits for each of its goroutines to
|
func(w http.ResponseWriter, _ *http.Request) {
|
||||||
// return.
|
|
||||||
s := New(logger.New(), metrics.New())
|
|
||||||
s.idleTimeout = idleTimeout
|
|
||||||
s.client = &http.Client{Transport: pipeTransport{
|
|
||||||
write: func(ctx context.Context, w io.Writer) {
|
|
||||||
deadline := time.Now().Add(sendFor)
|
deadline := time.Now().Add(sendFor)
|
||||||
for time.Now().Before(deadline) {
|
for time.Now().Before(deadline) {
|
||||||
_, _ = io.WriteString(w, updateLine)
|
_, _ = io.WriteString(w, updateLine)
|
||||||
|
|
||||||
// The wait ends with the stream. A bubble's clock stops when
|
flush(w)
|
||||||
// the test function returns, and a goroutine then still
|
time.Sleep(lineInterval)
|
||||||
// asleep panics the test binary.
|
|
||||||
select {
|
|
||||||
case <-ctx.Done():
|
|
||||||
return
|
|
||||||
case <-time.After(lineInterval):
|
|
||||||
}
|
|
||||||
}
|
}
|
||||||
},
|
}))
|
||||||
}}
|
defer srv.Close()
|
||||||
|
|
||||||
synctest.Test(t, func(t *testing.T) {
|
s := New(logger.New(), metrics.New())
|
||||||
err := s.stream(context.Background())
|
s.url = srv.URL
|
||||||
if !errors.Is(err, errStreamEnded) {
|
s.idleTimeout = idleTimeout
|
||||||
t.Fatalf("stream returned %v, want %v", err, errStreamEnded)
|
|
||||||
}
|
err := s.stream(context.Background())
|
||||||
})
|
if !errors.Is(err, errStreamEnded) {
|
||||||
|
t.Fatalf("stream returned %v, want %v", err, errStreamEnded)
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// TestStreamReadsLongLine has the server send one UPDATE line of about
|
// TestStreamReadsLongLine has the server send one UPDATE line of about
|
||||||
@@ -750,7 +706,7 @@ func TestHandlerStatsReadWhileMessagesAreDropped(t *testing.T) {
|
|||||||
const (
|
const (
|
||||||
lines = 5
|
lines = 5
|
||||||
wantDropped = lines - 2
|
wantDropped = lines - 2
|
||||||
maxWait = 15 * time.Second
|
maxWait = 5 * time.Second
|
||||||
)
|
)
|
||||||
|
|
||||||
srv := httptest.NewServer(http.HandlerFunc(
|
srv := httptest.NewServer(http.HandlerFunc(
|
||||||
@@ -864,70 +820,61 @@ func streamLines(t *testing.T, s *Streamer, lines string) {
|
|||||||
// line and then stays open. The streamer must come back for a second
|
// line and then stays open. The streamer must come back for a second
|
||||||
// response, must not be marked connected when it does, and must handle the
|
// response, must not be marked connected when it does, and must handle the
|
||||||
// UPDATE in it.
|
// UPDATE in it.
|
||||||
//
|
|
||||||
// It runs in a synctest bubble, whose clock is fake, so a stall on the host
|
|
||||||
// does not use up the idle timeout. The fake server gets the requests in the
|
|
||||||
// order the streamer makes them, because no real connection carries them.
|
|
||||||
func checkReconnect(
|
func checkReconnect(
|
||||||
t *testing.T, firstResponse http.RoundTripper, idleTimeout time.Duration,
|
t *testing.T, firstResponse http.HandlerFunc, idleTimeout time.Duration,
|
||||||
) {
|
) {
|
||||||
t.Helper()
|
t.Helper()
|
||||||
|
|
||||||
const maxWait = 15 * time.Second
|
const maxWait = 5 * time.Second
|
||||||
|
|
||||||
// The tracker is made outside the bubble. It starts a goroutine that never
|
|
||||||
// returns, and a bubble waits for each of its goroutines to return.
|
|
||||||
tracker := metrics.New()
|
tracker := metrics.New()
|
||||||
|
connectedAtReconnect := make(chan bool, 1)
|
||||||
|
|
||||||
synctest.Test(t, func(t *testing.T) {
|
var requests atomic.Int32
|
||||||
connectedAtReconnect := make(chan bool, 1)
|
|
||||||
laterResponse := pipeTransport{
|
|
||||||
write: func(ctx context.Context, w io.Writer) {
|
|
||||||
_, _ = io.WriteString(w, updateLine)
|
|
||||||
|
|
||||||
<-ctx.Done()
|
srv := httptest.NewServer(http.HandlerFunc(
|
||||||
},
|
func(w http.ResponseWriter, r *http.Request) {
|
||||||
}
|
if requests.Add(1) == 1 {
|
||||||
|
firstResponse(w, r)
|
||||||
|
|
||||||
var requests atomic.Int32
|
return
|
||||||
|
}
|
||||||
|
|
||||||
s := New(logger.New(), tracker)
|
select {
|
||||||
s.client = &http.Client{Transport: roundTripFunc(
|
case connectedAtReconnect <- tracker.IsConnected():
|
||||||
func(req *http.Request) (*http.Response, error) {
|
default:
|
||||||
if requests.Add(1) == 1 {
|
}
|
||||||
return firstResponse.RoundTrip(req)
|
|
||||||
}
|
|
||||||
|
|
||||||
select {
|
_, _ = io.WriteString(w, updateLine)
|
||||||
case connectedAtReconnect <- tracker.IsConnected():
|
|
||||||
default:
|
|
||||||
}
|
|
||||||
|
|
||||||
return laterResponse.RoundTrip(req)
|
flush(w)
|
||||||
},
|
<-r.Context().Done()
|
||||||
)}
|
}))
|
||||||
s.idleTimeout = idleTimeout
|
defer srv.Close()
|
||||||
s.minBackoff = time.Millisecond
|
|
||||||
handler := updateHandler{handled: make(chan struct{}, 1)}
|
|
||||||
s.RegisterHandler(handler)
|
|
||||||
|
|
||||||
err := s.Start(t.Context())
|
s := New(logger.New(), tracker)
|
||||||
if err != nil {
|
s.url = srv.URL
|
||||||
t.Fatalf("Start returned error: %v", err)
|
s.idleTimeout = idleTimeout
|
||||||
}
|
s.minBackoff = time.Millisecond
|
||||||
defer s.Stop(t.Context())
|
handler := updateHandler{handled: make(chan struct{}, 1)}
|
||||||
|
s.RegisterHandler(handler)
|
||||||
|
|
||||||
select {
|
err := s.Start(t.Context())
|
||||||
case <-handler.handled:
|
if err != nil {
|
||||||
case <-time.After(maxWait):
|
t.Fatalf("Start returned error: %v", err)
|
||||||
t.Fatalf("no message from a second response handled within %s", maxWait)
|
}
|
||||||
}
|
defer s.Stop(t.Context())
|
||||||
|
|
||||||
if <-connectedAtReconnect {
|
select {
|
||||||
t.Error("streamer was still marked connected when it came back " +
|
case <-handler.handled:
|
||||||
"for a second response")
|
case <-time.After(maxWait):
|
||||||
}
|
t.Fatalf("no message from a second response handled within %s", maxWait)
|
||||||
})
|
}
|
||||||
|
|
||||||
|
if <-connectedAtReconnect {
|
||||||
|
t.Error("streamer was still marked connected when it came back " +
|
||||||
|
"for a second response")
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// settledGoroutineCount lets transient goroutines finish, then reports the
|
// settledGoroutineCount lets transient goroutines finish, then reports the
|
||||||
@@ -949,13 +896,9 @@ func settledGoroutineCount() int {
|
|||||||
return prev
|
return prev
|
||||||
}
|
}
|
||||||
|
|
||||||
// waitForGoroutines waits up to 15 seconds for the goroutine count to drop to
|
// waitForGoroutines waits until the goroutine count drops to target or below.
|
||||||
// target or below.
|
|
||||||
func waitForGoroutines(target int) bool {
|
func waitForGoroutines(target int) bool {
|
||||||
const maxWait = 15 * time.Second
|
for range 100 {
|
||||||
|
|
||||||
deadline := time.Now().Add(maxWait)
|
|
||||||
for time.Now().Before(deadline) {
|
|
||||||
if runtime.NumGoroutine() <= target {
|
if runtime.NumGoroutine() <= target {
|
||||||
return true
|
return true
|
||||||
}
|
}
|
||||||
|
|||||||
Reference in new issue
Block a user