Author SHA1 Message Date
sneak a749d7abe6 Resolve PTR names over DNS-over-HTTPS and cache them for the record TTL (closes #238)
check / check (push) Canceled after 0s
`internal/ptr` asks Google, Cloudflare, AdGuard and Quad9 in turn, 2 seconds in all per address. It keeps answers in memory and writes them through to `cache.db`, opened by the new `internal/cachedb`. A reserved address is not sent to the resolvers.

Judgement calls:
- Each resolver gets an equal share of the time left, so a hung resolver does not use up the 2 seconds.
- The negative TTL is the smaller of the SOA minimum and the SOA's own TTL (RFC 2308).
- Names that are not host names are dropped, as `net.Resolver.LookupAddr` does.

Model: opus-5-5
2026-11-10 11:35:16 +00:00
19 changed files with 239 additions and 1105 deletions

No files matched your search

-19
View File
@@ -285,12 +285,8 @@ address. Every entry has the same fields:
lookups), or an empty list when it has none lookups), or an empty list when it has none
- `flags`: yes-or-no facts about the address: - `flags`: yes-or-no facts about the address:
- `is_tor`: true when the address is on the `tor_exits` list - `is_tor`: true when the address is on the `tor_exits` list
- `is_vpn`: true when the address is on any list of VPN servers (see IP
lists)
- `is_bogon`: true when the address is reserved, so that no network should - `is_bogon`: true when the address is reserved, so that no network should
announce it on the internet announce it on the internet
- `vpn_providers`: the names of the lists of VPN servers the address is on, or
an empty list when it is on none
- `ip_lists`: the public IP lists the address is on (see IP lists), or an empty - `ip_lists`: the public IP lists the address is on (see IP lists), or an empty
list when it is on none list when it is on none
@@ -421,16 +417,6 @@ The lists:
last 30 days. From last 30 days. From
[FireHOL](https://iplists.firehol.org/?ipset=firehol_level3), downloaded every [FireHOL](https://iplists.firehol.org/?ipset=firehol_level3), downloaded every
3 hours. 3 hours.
- `ivpn`: VPN servers run by IVPN. From
[IVPN](https://api.ivpn.net/v5/servers.json), downloaded once a day.
- `mullvad`: VPN servers run by Mullvad. From
[Mullvad](https://api.mullvad.net/www/relays/all/), downloaded once a day.
- `nordvpn`: VPN servers run by NordVPN. From
[NordVPN](https://api.nordvpn.com/v1/servers?limit=0&fields[servers.station]&fields[servers.ipv6_station]&fields[servers.hostname]),
downloaded once a day.
- `pia`: VPN servers run by Private Internet Access. From
[Private Internet Access](https://serverlist.piaservers.net/vpninfo/servers/v6),
downloaded once a day.
- `spamhaus_drop` and `spamhaus_dropv6`: IPv4 and IPv6 networks hijacked or - `spamhaus_drop` and `spamhaus_dropv6`: IPv4 and IPv6 networks hijacked or
leased by spammers or cyber criminals, according to Spamhaus. From leased by spammers or cyber criminals, according to Spamhaus. From
[Spamhaus DROP](https://www.spamhaus.org/blocklists/do-not-route-or-peer/), [Spamhaus DROP](https://www.spamhaus.org/blocklists/do-not-route-or-peer/),
@@ -439,9 +425,6 @@ The lists:
the Tor network leaves. From the Tor Project's the Tor network leaves. From the Tor Project's
[Onionoo](https://metrics.torproject.org/onionoo.html) service, downloaded [Onionoo](https://metrics.torproject.org/onionoo.html) service, downloaded
every hour. every hour.
- `x4bnet` and `x4bnet_v6`: IPv4 and IPv6 networks of VPN services, according to
a community list that names no service. From
[X4BNet](https://github.com/X4BNet/lists_vpn), downloaded once a day.
No list is downloaded more often than its publisher updates it or allows. Each No list is downloaded more often than its publisher updates it or allows. Each
can be downloaded whole without an account or a key. can be downloaded whole without an account or a key.
@@ -498,8 +481,6 @@ How RouteWatch keeps the answers:
whose SOA record has a TTL or minimum field of 0. whose SOA record has a TTL or minimum field of 0.
- A failed lookup is not kept, so the next lookup of the address asks again. - A failed lookup is not kept, so the next lookup of the address asks again.
- Every 10 minutes it deletes expired answers from memory and from `cache.db`. - Every 10 minutes it deletes expired answers from memory and from `cache.db`.
- It keeps answers for at most 250,000 addresses, and keeps no answer for a new
address while it holds that many.
## Configuration ## Configuration
-7
View File
@@ -56,13 +56,6 @@ problem the run finds.
- 2026-11-10: the IP lookup finds PTR names over DNS-over-HTTPS and caches them - 2026-11-10: the IP lookup finds PTR names over DNS-over-HTTPS and caches them
for the record's TTL, in memory and in `cache.db` (closes for the record's TTL, in memory and in `cache.db` (closes
https://git.eeqj.de/sneak/routewatch/issues/238). https://git.eeqj.de/sneak/routewatch/issues/238).
- 2026-11-10: the tests no longer fail when a busy host stalls for several
seconds. Most such waits are now 15 seconds. The test of the stream's idle
timeout runs on a fake clock (closes #240)
- 2026-11-10: the IP lookup checks the public VPN server lists of Mullvad,
NordVPN, Private Internet Access and IVPN, and X4BNet's community list. Each
result has `flags.is_vpn` and `vpn_providers` (closes
https://git.eeqj.de/sneak/routewatch/issues/235).
- 2026-11-10: `tor_exits` comes from Onionoo, so it holds IPv6 exits and - 2026-11-10: `tor_exits` comes from Onionoo, so it holds IPv6 exits and
`is_tor` is true for them (closes `is_tor` is true for them (closes
https://git.eeqj.de/sneak/routewatch/issues/234). https://git.eeqj.de/sneak/routewatch/issues/234).
+1 -1
View File
@@ -12,7 +12,7 @@ import (
// parseTimeLimit is how long a test waits for parseCSV to return before it // parseTimeLimit is how long a test waits for parseCSV to return before it
// fails. // fails.
const parseTimeLimit = 15 * time.Second const parseTimeLimit = 2 * time.Second
// TestParseCSVReadsTodaysLayout parses an as.csv with the country code column // TestParseCSVReadsTodaysLayout parses an as.csv with the country code column
// that the source added after the description. It expects each AS with its // that the source added after the description. It expects each AS with its
+4 -7
View File
@@ -8,9 +8,6 @@ import (
"git.eeqj.de/sneak/routewatch/internal/bogon" "git.eeqj.de/sneak/routewatch/internal/bogon"
) )
// privateUseTen is why a prefix or an address inside 10.0.0.0/8 is reserved.
const privateUseTen = "Private-Use (10.0.0.0/8)"
func TestPrefix(t *testing.T) { func TestPrefix(t *testing.T) {
t.Parallel() t.Parallel()
@@ -19,8 +16,8 @@ func TestPrefix(t *testing.T) {
// want is why the prefix is reserved, or "" when it is not. // want is why the prefix is reserved, or "" when it is not.
want string want string
}{ }{
{"10.1.2.0/24", privateUseTen}, {"10.1.2.0/24", "Private-Use (10.0.0.0/8)"},
{"10.0.0.0/8", privateUseTen}, {"10.0.0.0/8", "Private-Use (10.0.0.0/8)"},
{"192.0.2.0/24", "Documentation (TEST-NET-1) (192.0.2.0/24)"}, {"192.0.2.0/24", "Documentation (TEST-NET-1) (192.0.2.0/24)"},
{"2001:db8:1::/48", "Documentation (2001:db8::/32)"}, {"2001:db8:1::/48", "Documentation (2001:db8::/32)"},
{"fd00::/8", "Unique-Local (fc00::/7)"}, {"fd00::/8", "Unique-Local (fc00::/7)"},
@@ -67,10 +64,10 @@ func TestAddr(t *testing.T) {
// want is why the address is reserved, or "" when it is not. // want is why the address is reserved, or "" when it is not.
want string want string
}{ }{
{"10.0.0.1", privateUseTen}, {"10.0.0.1", "Private-Use (10.0.0.0/8)"},
{"fd00::1", "Unique-Local (fc00::/7)"}, {"fd00::1", "Unique-Local (fc00::/7)"},
// An IPv4 address written in IPv6 form counts as the IPv4 address. // An IPv4 address written in IPv6 form counts as the IPv4 address.
{"::ffff:10.0.0.1", privateUseTen}, {"::ffff:10.0.0.1", "Private-Use (10.0.0.0/8)"},
{"::ffff:8.8.8.8", ""}, {"::ffff:8.8.8.8", ""},
// An address that may be announced, inside a reserved block. // An address that may be announced, inside a reserved block.
{"192.0.0.9", ""}, {"192.0.0.9", ""},
+5 -25
View File
@@ -72,34 +72,17 @@ const (
// lockTestPrefix is the prefix the batch writes in the checkpoint tests // lockTestPrefix is the prefix the batch writes in the checkpoint tests
// store. // store.
lockTestPrefix = "192.0.2.0/24" lockTestPrefix = "192.0.2.0/24"
// longBusyTimeoutMs is the busy timeout setLongBusyTimeout gives the
// writing connection. A checkpoint that waits for an open read takes this
// long.
longBusyTimeoutMs = 30000
// readerCheckpointLimit is how long a checkpoint, and a batch write // readerCheckpointLimit is how long a checkpoint, and a batch write
// waiting for it, may take while a reader is open: well under // waiting for it, may take while a reader is open: well under the busy
// longBusyTimeoutMs, with room for a stall on a busy host. // timeout, which a checkpoint that waits for the reader takes.
readerCheckpointLimit = 15 * time.Second readerCheckpointLimit = time.Second
// truncateCheckpointLimit is how long a checkpoint of a WAL file over // truncateCheckpointLimit is how long a checkpoint of a WAL file over
// sqliteWALSizeLimitBytes may take to give up while a reader is open: its // sqliteWALSizeLimitBytes may take to give up while a reader is open: its
// wait for the reader plus readerCheckpointLimit. // wait for the reader plus a second, well under the busy timeout.
truncateCheckpointLimit = truncateBusyTimeoutMs*time.Millisecond + truncateCheckpointLimit = truncateBusyTimeoutMs*time.Millisecond +
readerCheckpointLimit time.Second
) )
// setLongBusyTimeout gives the writing connection of db longBusyTimeoutMs, so
// that a checkpoint that waits for an open read takes far longer than a stall
// on a busy host.
func setLongBusyTimeout(ctx context.Context, t *testing.T, db *Database) {
t.Helper()
_, err := db.writer.ExecContext(ctx,
fmt.Sprintf("PRAGMA busy_timeout=%d", longBusyTimeoutMs))
if err != nil {
t.Fatalf("failed to set the busy timeout: %v", err)
}
}
// TestConnectionPoolPragmas holds several pooled connections open at once and // TestConnectionPoolPragmas holds several pooled connections open at once and
// checks each one carries the per-connection settings from the DSN, plus the // checks each one carries the per-connection settings from the DSN, plus the
// process-wide heap limits, and sees the new file with auto_vacuum // process-wide heap limits, and sees the new file with auto_vacuum
@@ -596,8 +579,6 @@ func TestCheckpointDoesNotWaitForReader(t *testing.T) {
ctx := context.Background() ctx := context.Background()
ts := time.Now().UTC() ts := time.Now().UTC()
setLongBusyTimeout(ctx, t, db)
err = db.UpdatePrefixesBatch(map[string]time.Time{"198.51.100.0/24": ts}) err = db.UpdatePrefixesBatch(map[string]time.Time{"198.51.100.0/24": ts})
if err != nil { if err != nil {
t.Fatalf("failed to store the prefix the reader reads: %v", err) t.Fatalf("failed to store the prefix the reader reads: %v", err)
@@ -709,7 +690,6 @@ func TestCheckpointEmptiesWALOnceNoReadIsOpen(t *testing.T) {
} }
fillWALPastLimit(ctx, t, db) fillWALPastLimit(ctx, t, db)
setLongBusyTimeout(ctx, t, db)
started := time.Now() started := time.Now()
-25
View File
@@ -34,23 +34,6 @@ const (
// addresses in two fields: exit_addresses, which has IPv4 only, and // addresses in two fields: exit_addresses, which has IPv4 only, and
// or_addresses, which has IPv4 and IPv6, each with a port. // or_addresses, which has IPv4 and IPv6, each with a port.
FormatOnionooJSON FormatOnionooJSON
// FormatMullvadJSON is Mullvad's JSON list of relays. A relay's addresses
// are in ipv4_addr_in, ipv6_addr_in and ipv4_v2ray.
FormatMullvadJSON
// FormatNordVPNJSON is NordVPN's JSON list of servers. A server's
// addresses are in station and ipv6_station.
FormatNordVPNJSON
// FormatPIAJSON is Private Internet Access's JSON server list, then a
// blank line and a signature. Each region's servers has lists of servers
// by protocol, and each server has its address in ip.
FormatPIAJSON
// FormatIVPNJSON is IVPN's JSON server list. The hosts of its wireguard
// and openvpn gateways have their addresses in host and v2ray.
FormatIVPNJSON
) )
// List is one public IP list. // List is one public IP list.
@@ -70,9 +53,6 @@ type List struct {
// Format is how the list's file is written. // Format is how the list's file is written.
Format Format Format Format
// VPN says that the list holds servers of VPN services.
VPN bool
} }
// Listing says that a list contains an address. // Listing says that a list contains an address.
@@ -83,10 +63,6 @@ type Listing struct {
// Updated is when routewatch last downloaded the list, or last heard from // Updated is when routewatch last downloaded the list, or last heard from
// its publisher that its copy was still current. // its publisher that its copy was still current.
Updated time.Time `json:"updated"` Updated time.Time `json:"updated"`
// VPN says that the list holds servers of VPN services. A lookup answer
// reports it through is_vpn and vpn_providers.
VPN bool `json:"-"`
} }
// listCopy is a list's last good copy in memory. // listCopy is a list's last good copy in memory.
@@ -153,7 +129,6 @@ func (b *IPLists) Lookup(addr netip.Addr) []Listing {
Name: list.Name, Name: list.Name,
Description: list.Description, Description: list.Description,
Updated: c.updated, Updated: c.updated,
VPN: list.VPN,
}) })
} }
} }
+12 -197
View File
@@ -43,36 +43,6 @@ const (
],"bridges":[]}` ],"bridges":[]}`
) )
// Small documents in the VPN services' formats. PIA's and IVPN's have no
// public IPv6 addresses, and IVPN's holds addresses that are not its servers,
// which must not be read.
const (
mullvadBody = `[
{"hostname":"a-wg-001","ipv4_addr_in":"192.0.2.40","ipv6_addr_in":"2001:db8::40"},
{"hostname":"a-br-001","ipv4_addr_in":"192.0.2.41","ipv6_addr_in":null,
"ipv4_v2ray":"192.0.2.42"}
]`
nordVPNBody = `[
{"hostname":"a1.example.com","station":"192.0.2.50","ipv6_station":""},
{"hostname":"a2.example.com","station":"192.0.2.51","ipv6_station":"2001:db8::51"}
]`
piaBody = `{"groups":{"wg":[{"name":"wireguard","ports":[1337]}]},"regions":[
{"id":"a","servers":{
"wg":[{"ip":"192.0.2.60","cn":"a1"}],
"ikev2":[{"ip":"192.0.2.61","cn":"a1"}]}},
{"id":"b","servers":{"meta":[{"ip":"192.0.2.62","cn":"b1","van":true}]}}
]}` + "\n\nc2lnbmF0dXJl\nbGluZSB0d28=\n"
ivpnBody = `{"wireguard":[{"gateway":"a.wg.example.com","hosts":[
{"hostname":"a1","host":"192.0.2.70","v2ray":"192.0.2.71",
"ipv6":{"local_ip":"2001:db8::70/96"}}]}],
"openvpn":[{"gateway":"a.gw.example.com","hosts":[
{"hostname":"a1","host":"192.0.2.72"}]}],
"config":{"api":{"ips":["192.0.2.73"]}}}`
)
// errorPage is a file that is no list, as a publisher's error page is.
const errorPage = "<!DOCTYPE html>\n<html></html>\n"
// testETag is the ETag a publisher sends with a version of the test list. // testETag is the ETag a publisher sends with a version of the test list.
const testETag = `"v1"` const testETag = `"v1"`
@@ -188,7 +158,13 @@ func lookup(t *testing.T, b *IPLists, addr string) []Listing {
func TestLookupFindsListedAddresses(t *testing.T) { func TestLookupFindsListedAddresses(t *testing.T) {
t.Parallel() t.Parallel()
runLookupCases(t, []lookupCase{ tests := []struct {
name string
format Format
body string
listed []string
unlisted []string
}{
{ {
name: "single addresses", name: "single addresses",
format: FormatLines, format: FormatLines,
@@ -230,60 +206,9 @@ func TestLookupFindsListedAddresses(t *testing.T) {
}, },
unlisted: []string{"192.0.2.12", "2001:db8::2"}, unlisted: []string{"192.0.2.12", "2001:db8::2"},
}, },
}) }
}
// TestLookupFindsVPNServers downloads a list in each VPN service's format and for _, tt := range tests {
// looks up addresses on it and off it.
func TestLookupFindsVPNServers(t *testing.T) {
t.Parallel()
runLookupCases(t, []lookupCase{
{
name: "Mullvad relays",
format: FormatMullvadJSON,
body: mullvadBody,
listed: []string{"192.0.2.40", "192.0.2.41", "192.0.2.42", "2001:db8::40"},
unlisted: []string{"192.0.2.43", "2001:db8::41"},
},
{
name: "NordVPN servers",
format: FormatNordVPNJSON,
body: nordVPNBody,
listed: []string{"192.0.2.50", "192.0.2.51", "2001:db8::51"},
unlisted: []string{"192.0.2.52", "2001:db8::50"},
},
{
name: "PIA servers, with the signature after the JSON",
format: FormatPIAJSON,
body: piaBody,
listed: []string{"192.0.2.60", "192.0.2.61", "192.0.2.62"},
unlisted: []string{"192.0.2.63", "2001:db8::60"},
},
{
name: "IVPN hosts",
format: FormatIVPNJSON,
body: ivpnBody,
listed: []string{"192.0.2.70", "192.0.2.71", "192.0.2.72"},
unlisted: []string{"192.0.2.73", "2001:db8::70"},
},
})
}
// lookupCase is a list file, the addresses it holds and some it does not.
type lookupCase struct {
name string
format Format
body string
listed []string
unlisted []string
}
// runLookupCases downloads each case's list file and checks its addresses.
func runLookupCases(t *testing.T, cases []lookupCase) {
t.Helper()
for _, tt := range cases {
t.Run(tt.name, func(t *testing.T) { t.Run(tt.name, func(t *testing.T) {
t.Parallel() t.Parallel()
@@ -335,7 +260,7 @@ func TestFailedDownloadKeepsThePreviousCopy(t *testing.T) {
body string body string
}{ }{
{"server error", http.StatusInternalServerError, "unavailable"}, {"server error", http.StatusInternalServerError, "unavailable"},
{"not a list", http.StatusOK, errorPage}, {"not a list", http.StatusOK, "<!DOCTYPE html>\n<html></html>\n"},
} }
for _, tt := range tests { for _, tt := range tests {
@@ -381,7 +306,7 @@ func TestFileThatIsNotAnOnionooDocumentKeepsThePreviousCopy(t *testing.T) {
name string name string
body string body string
}{ }{
{"not JSON", errorPage}, {"not JSON", "<!DOCTYPE html>\n<html></html>\n"},
{"other JSON", `{"status":"ok","data":{"relays_count":1}}`}, {"other JSON", `{"status":"ok","data":{"relays_count":1}}`},
{"JSON array", `[{"or_addresses":["192.0.2.1:443"]}]`}, {"JSON array", `[{"or_addresses":["192.0.2.1:443"]}]`},
{"no relays", `{"relays":[],"bridges":[]}`}, {"no relays", `{"relays":[],"bridges":[]}`},
@@ -423,112 +348,6 @@ func TestFileThatIsNotAnOnionooDocumentKeepsThePreviousCopy(t *testing.T) {
} }
} }
// TestWrongVPNDocumentFailsToParse parses documents that are not a VPN
// service's list. Each fails, so none can replace a good copy.
func TestWrongVPNDocumentFailsToParse(t *testing.T) {
t.Parallel()
formats := []struct {
name string
format Format
good string
empty string
badAddress string
}{
{
"Mullvad", FormatMullvadJSON, mullvadBody,
`[{"hostname":"a"}]`, `[{"ipv4_addr_in":"example"}]`,
},
{
"NordVPN", FormatNordVPNJSON, nordVPNBody,
`[{"hostname":"a","station":"","ipv6_station":""}]`,
`[{"station":"example"}]`,
},
{
"PIA", FormatPIAJSON, piaBody,
`{"regions":[{"id":"a","servers":{}}]}`,
`{"regions":[{"servers":{"wg":[{"ip":"example"}]}}]}`,
},
{
"IVPN", FormatIVPNJSON, ivpnBody,
`{"wireguard":[{"hosts":[]}],"openvpn":[]}`,
`{"wireguard":[{"hosts":[{"host":"example"}]}]}`,
},
}
for _, f := range formats {
bodies := map[string]string{
"not JSON": errorPage,
"other JSON": `{"status":"ok","data":{"count":1}}`,
"no addresses": f.empty,
"bad address": f.badAddress,
"cut off": f.good[:len(f.good)/2],
"an empty file": "",
}
for name, body := range bodies {
t.Run(f.name+" "+name, func(t *testing.T) {
t.Parallel()
addresses, err := parse(strings.NewReader(body), f.format)
if err == nil {
t.Errorf("parse returned %v and no error", addresses.Ranges())
}
})
}
}
}
// TestLookupMarksVPNLists looks up an address on a VPN list and on another
// list. Only the VPN list's listing says it is one.
func TestLookupMarksVPNLists(t *testing.T) {
t.Parallel()
_, url := newPublisher(t, goodListBody)
vpnList := testList(url, FormatLines)
vpnList.Name = "test_vpn_list"
vpnList.VPN = true
otherList := testList(url, FormatLines)
b := New(Params{
Dir: t.TempDir(),
Lists: []List{vpnList, otherList},
Logger: slog.New(slog.DiscardHandler),
})
for _, list := range b.lists {
err := b.download(t.Context(), list)
if err != nil {
t.Fatalf("download of %s: %v", list.Name, err)
}
}
listings := lookup(t, b, listedAddr)
if len(listings) != 2 || !listings[0].VPN || listings[1].VPN {
t.Errorf("listings %+v, want the first of two marked as a VPN list",
listings)
}
}
// TestPublicListsMarkTheVPNLists checks which public lists are VPN lists,
// since their names are what vpn_providers reports.
func TestPublicListsMarkTheVPNLists(t *testing.T) {
t.Parallel()
var got []string
for _, list := range Lists() {
if list.VPN {
got = append(got, list.Name)
}
}
want := []string{"ivpn", "mullvad", "nordvpn", "pia", "x4bnet", "x4bnet_v6"}
if !slices.Equal(got, want) {
t.Errorf("VPN lists %v, want %v", got, want)
}
}
// checkGoodCopyOnDisk checks that dir holds the good version of the test list // checkGoodCopyOnDisk checks that dir holds the good version of the test list
// and no part of a failed download. // and no part of a failed download.
func checkGoodCopyOnDisk(t *testing.T, dir string) { func checkGoodCopyOnDisk(t *testing.T, dir string) {
@@ -646,13 +465,9 @@ func TestRestartLoadsCopiesFromDisk(t *testing.T) {
second.Start(t.Context()) second.Start(t.Context())
second.Stop(t.Context()) second.Stop(t.Context())
// The download takes its time after writing the file, and the restart
// reads the file's time. A busy host can put seconds between the two.
const maxUpdatedDifference = 15 * time.Second
listings := lookup(t, second, listedAddr) listings := lookup(t, second, listedAddr)
if len(listings) != 1 || if len(listings) != 1 ||
listings[0].Updated.Sub(downloaded).Abs() > maxUpdatedDifference { listings[0].Updated.Sub(downloaded).Abs() > time.Second {
t.Errorf("after the restart, listings %+v, want one updated at %s", t.Errorf("after the restart, listings %+v, want one updated at %s",
listings, downloaded) listings, downloaded)
} }
+1 -73
View File
@@ -1,9 +1,6 @@
package iplist package iplist
import ( import "time"
"slices"
"time"
)
// TorExitsList is the name of the list of Tor exit nodes. // TorExitsList is the name of the list of Tor exit nodes.
const TorExitsList = "tor_exits" const TorExitsList = "tor_exits"
@@ -35,20 +32,12 @@ const (
// Onionoo sets no download limit: // Onionoo sets no download limit:
// https://metrics.torproject.org/onionoo.html // https://metrics.torproject.org/onionoo.html
torExitsInterval = time.Hour torExitsInterval = time.Hour
// A VPN service's servers change slowly, so once a day is enough.
vpnListInterval = 24 * time.Hour
) )
// Lists returns the public IP lists routewatch downloads. Each can be // Lists returns the public IP lists routewatch downloads. Each can be
// downloaded whole without an account or a key. Spamhaus merged its EDROP // downloaded whole without an account or a key. Spamhaus merged its EDROP
// list into DROP, so DROP covers both. // list into DROP, so DROP covers both.
func Lists() []List { func Lists() []List {
return slices.Concat(nonVPNLists(), vpnLists())
}
// nonVPNLists returns the public lists that are not lists of VPN servers.
func nonVPNLists() []List {
return []List{ return []List{
{ {
Name: "dshield", Name: "dshield",
@@ -107,64 +96,3 @@ func nonVPNLists() []List {
}, },
} }
} }
// vpnLists returns the public lists of VPN servers. Their names are what
// vpn_providers reports.
func vpnLists() []List {
return []List{
{
Name: "ivpn",
Description: "A server of the IVPN service",
URL: "https://api.ivpn.net/v5/servers.json",
Interval: vpnListInterval,
Format: FormatIVPNJSON,
VPN: true,
},
{
Name: "mullvad",
Description: "A server of the Mullvad VPN service",
URL: "https://api.mullvad.net/www/relays/all/",
Interval: vpnListInterval,
Format: FormatMullvadJSON,
VPN: true,
},
{
Name: "nordvpn",
Description: "A server of the NordVPN service",
URL: "https://api.nordvpn.com/v1/servers?limit=0" +
"&fields[servers.station]&fields[servers.ipv6_station]" +
"&fields[servers.hostname]",
Interval: vpnListInterval,
Format: FormatNordVPNJSON,
VPN: true,
},
{
Name: "pia",
Description: "A server of the Private Internet Access VPN service",
URL: "https://serverlist.piaservers.net/vpninfo/servers/v6",
Interval: vpnListInterval,
Format: FormatPIAJSON,
VPN: true,
},
{
Name: "x4bnet",
Description: "In an IPv4 network of a VPN service, according to " +
"X4BNet's community list",
URL: "https://raw.githubusercontent.com/X4BNet/lists_vpn/main/" +
"output/vpn/ipv4.txt",
Interval: vpnListInterval,
Format: FormatLines,
VPN: true,
},
{
Name: "x4bnet_v6",
Description: "In an IPv6 network of a VPN service, according to " +
"X4BNet's community list",
URL: "https://raw.githubusercontent.com/X4BNet/lists_vpn/main/" +
"output/vpn/ipv6.txt",
Interval: vpnListInterval,
Format: FormatLines,
VPN: true,
},
}
}
+7 -158
View File
@@ -8,7 +8,6 @@ import (
"io" "io"
"net" "net"
"net/netip" "net/netip"
"slices"
"strings" "strings"
"go4.org/netipx" "go4.org/netipx"
@@ -34,7 +33,7 @@ var (
errUnknownFormat = errors.New("unknown list format") errUnknownFormat = errors.New("unknown list format")
errNoNetwork = errors.New("no network on the line") errNoNetwork = errors.New("no network on the line")
errBadRange = errors.New("first and last address make no range") errBadRange = errors.New("first and last address make no range")
errNoAddresses = errors.New("no addresses in the document") errNoAddresses = errors.New("no relay addresses in the document")
) )
// parse reads a list file written in format and returns the addresses it // parse reads a list file written in format and returns the addresses it
@@ -50,14 +49,6 @@ func parse(r io.Reader, format Format) (*netipx.IPSet, error) {
return parseLines(r, addDShieldNetwork) return parseLines(r, addDShieldNetwork)
case FormatOnionooJSON: case FormatOnionooJSON:
return parseOnionoo(r) return parseOnionoo(r)
case FormatMullvadJSON:
return parseMullvad(r)
case FormatNordVPNJSON:
return parseNordVPN(r)
case FormatPIAJSON:
return parsePIA(r)
case FormatIVPNJSON:
return parseIVPN(r)
default: default:
return nil, errUnknownFormat return nil, errUnknownFormat
} }
@@ -177,6 +168,11 @@ func addDShieldNetwork(builder *netipx.IPSetBuilder, line string) error {
// fields of the relays in an Onionoo details document. A document with no // fields of the relays in an Onionoo details document. A document with no
// addresses fails, because the Tor network always has relays. // addresses fails, because the Tor network always has relays.
func parseOnionoo(r io.Reader) (*netipx.IPSet, error) { func parseOnionoo(r io.Reader) (*netipx.IPSet, error) {
data, err := io.ReadAll(r)
if err != nil {
return nil, err
}
//nolint:tagliatelle // Onionoo's own field names //nolint:tagliatelle // Onionoo's own field names
var details struct { var details struct {
Relays []struct { Relays []struct {
@@ -185,7 +181,7 @@ func parseOnionoo(r io.Reader) (*netipx.IPSet, error) {
} `json:"relays"` } `json:"relays"`
} }
err := readJSON(r, &details) err = json.Unmarshal(data, &details)
if err != nil { if err != nil {
return nil, err return nil, err
} }
@@ -220,153 +216,6 @@ func parseOnionoo(r io.Reader) (*netipx.IPSet, error) {
return addresses, nil return addresses, nil
} }
// parseMullvad returns the addresses of the relays in a Mullvad document. A
// relay with no IPv6 address has null for it, and only bridges have
// ipv4_v2ray.
func parseMullvad(r io.Reader) (*netipx.IPSet, error) {
//nolint:tagliatelle // Mullvad's own field names
var relays []struct {
IPv4 string `json:"ipv4_addr_in"`
IPv6 string `json:"ipv6_addr_in"`
IPv4V2Ray string `json:"ipv4_v2ray"`
}
err := readJSON(r, &relays)
if err != nil {
return nil, err
}
var texts []string
for _, relay := range relays {
texts = append(texts, relay.IPv4, relay.IPv6, relay.IPv4V2Ray)
}
return addressSet(texts)
}
// parseNordVPN returns the addresses of the servers in a NordVPN document. A
// server with no IPv6 address has an empty ipv6_station.
func parseNordVPN(r io.Reader) (*netipx.IPSet, error) {
//nolint:tagliatelle // NordVPN's own field names
var servers []struct {
Station string `json:"station"`
IPv6Station string `json:"ipv6_station"`
}
err := readJSON(r, &servers)
if err != nil {
return nil, err
}
var texts []string
for _, server := range servers {
texts = append(texts, server.Station, server.IPv6Station)
}
return addressSet(texts)
}
// parsePIA returns the addresses of the servers in a Private Internet Access
// document. json.Decoder stops at the end of the JSON, so the signature after
// it is not read.
func parsePIA(r io.Reader) (*netipx.IPSet, error) {
var list struct {
Regions []struct {
Servers map[string][]struct {
IP string `json:"ip"`
} `json:"servers"`
} `json:"regions"`
}
err := json.NewDecoder(r).Decode(&list)
if err != nil {
return nil, err
}
var texts []string
for _, region := range list.Regions {
for _, servers := range region.Servers {
for _, server := range servers {
texts = append(texts, server.IP)
}
}
}
return addressSet(texts)
}
// parseIVPN returns the addresses of the hosts in an IVPN document. A host's
// ipv6 object holds only an address inside the VPN, so it is not read.
func parseIVPN(r io.Reader) (*netipx.IPSet, error) {
type gateway struct {
Hosts []struct {
Host string `json:"host"`
V2Ray string `json:"v2ray"`
} `json:"hosts"`
}
var list struct {
WireGuard []gateway `json:"wireguard"`
OpenVPN []gateway `json:"openvpn"`
}
err := readJSON(r, &list)
if err != nil {
return nil, err
}
var texts []string
for _, gw := range slices.Concat(list.WireGuard, list.OpenVPN) {
for _, host := range gw.Hosts {
texts = append(texts, host.Host, host.V2Ray)
}
}
return addressSet(texts)
}
// readJSON reads r to its end and decodes it, as one JSON document, into v.
func readJSON(r io.Reader, v any) error {
data, err := io.ReadAll(r)
if err != nil {
return err
}
return json.Unmarshal(data, v)
}
// addressSet returns the set of the addresses in texts, skipping empty ones.
// A list with no addresses fails, because every VPN service has servers.
func addressSet(texts []string) (*netipx.IPSet, error) {
var builder netipx.IPSetBuilder
for _, text := range texts {
if text == "" {
continue
}
err := addAddress(&builder, text)
if err != nil {
return nil, err
}
}
addresses, err := builder.IPSet()
if err != nil {
return nil, err
}
if len(addresses.Ranges()) == 0 {
return nil, errNoAddresses
}
return addresses, nil
}
// addAddress adds an address, such as 192.0.2.1, to builder. // addAddress adds an address, such as 192.0.2.1, to builder.
func addAddress(builder *netipx.IPSetBuilder, text string) error { func addAddress(builder *netipx.IPSetBuilder, text string) error {
addr, err := netip.ParseAddr(text) addr, err := netip.ParseAddr(text)
+20 -33
View File
@@ -1,6 +1,6 @@
// Package ptr looks up the PTR names of addresses over DNS-over-HTTPS. It // Package ptr looks up the PTR names of addresses over DNS-over-HTTPS. It
// keeps answers in memory for as long as DNS allows, and in cache.db so that // keeps each answer in memory for as long as DNS allows, and in cache.db so
// the answers survive a restart. // that the answers survive a restart.
package ptr package ptr
import ( import (
@@ -29,10 +29,6 @@ const (
// nameSeparator separates the names in a row of the hostnames table. // nameSeparator separates the names in a row of the hostnames table.
nameSeparator = " " nameSeparator = " "
// maxAddresses limits how many addresses the cache holds, so that a public
// lookup cannot grow memory and cache.db without bound.
maxAddresses = 250_000
) )
// Params is what New needs. // Params is what New needs.
@@ -56,16 +52,15 @@ type entry struct {
// Cache answers PTR lookups from memory, and asks the resolvers for an address // Cache answers PTR lookups from memory, and asks the resolvers for an address
// it holds no current answer for. Start loads it from cache.db. // it holds no current answer for. Start loads it from cache.db.
type Cache struct { type Cache struct {
db *sql.DB db *sql.DB
resolvers []string resolvers []string
timeLimit time.Duration timeLimit time.Duration
maxEntries int logger *slog.Logger
logger *slog.Logger client *http.Client
client *http.Client now func() time.Time
now func() time.Time
// entries holds the kept answers by address, at most maxEntries of them. // entries holds every answer by address. pending holds those that
// pending holds those that cache.db does not have yet. // cache.db does not have yet.
mu sync.RWMutex mu sync.RWMutex
entries map[string]entry entries map[string]entry
pending map[string]entry pending map[string]entry
@@ -78,16 +73,15 @@ type Cache struct {
// New returns an empty Cache. Start loads it. // New returns an empty Cache. Start loads it.
func New(params Params) *Cache { func New(params Params) *Cache {
return &Cache{ return &Cache{
db: params.DB, db: params.DB,
resolvers: params.Resolvers, resolvers: params.Resolvers,
timeLimit: LookupTimeout, timeLimit: LookupTimeout,
maxEntries: maxAddresses, logger: params.Logger.With("component", "ptr"),
logger: params.Logger.With("component", "ptr"), client: &http.Client{},
client: &http.Client{}, now: time.Now,
now: time.Now, entries: make(map[string]entry),
entries: make(map[string]entry), pending: make(map[string]entry),
pending: make(map[string]entry), done: make(chan struct{}),
done: make(chan struct{}),
} }
} }
@@ -174,18 +168,11 @@ func (c *Cache) current(address string) (entry, bool) {
return found, true return found, true
} }
// store keeps an answer in memory and queues it for cache.db. Once entries // store keeps an answer in memory and queues it for cache.db.
// holds maxEntries addresses it keeps no new address. It still replaces the
// answer of an address already held.
func (c *Cache) store(address string, answer entry) { func (c *Cache) store(address string, answer entry) {
c.mu.Lock() c.mu.Lock()
defer c.mu.Unlock() defer c.mu.Unlock()
_, held := c.entries[address]
if !held && len(c.entries) >= c.maxEntries {
return
}
c.entries[address] = answer c.entries[address] = answer
c.pending[address] = answer c.pending[address] = answer
} }
-69
View File
@@ -880,72 +880,3 @@ func TestFlushKeepsAnswersItCouldNotWrite(t *testing.T) {
t.Errorf("%d rows after the second write, want 1", got) t.Errorf("%d rows after the second write, want 1", got)
} }
} }
// TestNewAddressIsNotKeptAtTheLimit fills a cache that holds one address. A
// lookup of a second address still returns its answer, and the cache keeps
// nothing for it until the sweep has deleted the first address's expired
// answer.
func TestNewAddressIsNotKeptAtTheLimit(t *testing.T) {
t.Parallel()
db := openTestDB(t, t.TempDir())
resolver := newFakeResolver(t, ptrReply(300, testName))
cache, clock := newTestCache(t, db, resolver)
cache.maxEntries = 1
lookupOK(t, cache, testAddress)
names := lookupOK(t, cache, otherAddress)
if !slices.Equal(names, []string{testName}) {
t.Errorf("names %v at the limit, want %s", names, testName)
}
_, inEntries := cache.entries[otherAddress]
_, inPending := cache.pending[otherAddress]
if inEntries || inPending {
t.Errorf("a new address was kept at the limit: in entries %t, in pending %t",
inEntries, inPending)
}
cache.flush(t.Context())
if got := countRows(t, db); got != 1 {
t.Errorf("%d rows at the limit, want 1", got)
}
clock.advance(300 * time.Second)
cache.sweep(t.Context())
lookupOK(t, cache, otherAddress)
_, inEntries = cache.entries[otherAddress]
if !inEntries {
t.Error("a new address was not kept after the sweep made room")
}
}
// TestHeldAddressIsReplacedAtTheLimit lets the answer of the only address a
// cache holds expire. The new answer replaces it in memory and is queued for
// cache.db.
func TestHeldAddressIsReplacedAtTheLimit(t *testing.T) {
t.Parallel()
resolver := newFakeResolver(t, ptrReply(300, testName))
cache, clock := newTestCache(t, openTestDB(t, t.TempDir()), resolver)
cache.maxEntries = 1
lookupOK(t, cache, testAddress)
clock.advance(300 * time.Second)
resolver.setReply(ptrReply(300, otherName))
lookupOK(t, cache, testAddress)
want := []string{otherName}
if got := cache.entries[testAddress].names; !slices.Equal(got, want) {
t.Errorf("entries holds %v, want %v", got, want)
}
if got := cache.pending[testAddress].names; !slices.Equal(got, want) {
t.Errorf("pending holds %v, want %v", got, want)
}
}
@@ -8,8 +8,6 @@ import (
"net/http" "net/http"
"net/http/httptest" "net/http/httptest"
"net/netip" "net/netip"
"os"
"path/filepath"
"slices" "slices"
"strconv" "strconv"
"strings" "strings"
@@ -785,74 +783,3 @@ func TestShutdownCheckpointsAndClosesTheDatabaseLast(t *testing.T) {
t.Errorf("Shutdown made the calls %q, want %q", store.calls, want) t.Errorf("Shutdown made the calls %q, want %q", store.calls, want)
} }
} }
// TestPTRCacheAnswersTheIPLookup starts the PTR cache and the HTTP server as
// Run does, and looks up a reserved address, which asks no resolver. Without
// the PTR cache the server has no PTR resolver, and every IP lookup answers
// 500.
func TestPTRCacheAnswersTheIPLookup(t *testing.T) {
t.Parallel()
store := newMockStore()
log := logger.New()
port := freePort(t)
cfg := &config.Config{Port: port, StateDir: t.TempDir()}
s := streamer.New(log, metrics.New())
// A started server reads its routing table in the background, so it
// needs one.
rw := New(Dependencies{
DB: store,
Streamer: s,
Server: server.New(store, routetable.New(routetable.Params{}), s, log,
cfg),
Logger: log,
Config: cfg,
})
err := rw.startPTRCache(t.Context())
if err != nil {
t.Fatalf("failed to start the PTR cache: %v", err)
}
defer rw.stopPTRCache(t.Context())
err = rw.server.Start(t.Context())
if err != nil {
t.Fatalf("failed to start the HTTP server: %v", err)
}
defer func() { _ = rw.server.Stop(t.Context()) }()
req, err := http.NewRequestWithContext(t.Context(), http.MethodGet,
"http://"+net.JoinHostPort("127.0.0.1", strconv.Itoa(port))+
"/api/v1/ip/10.0.0.1", nil)
if err != nil {
t.Fatalf("failed to build the request: %v", err)
}
resp, err := http.DefaultClient.Do(req)
if err != nil {
t.Fatalf("the IP lookup failed: %v", err)
}
defer func() { _ = resp.Body.Close() }()
body, err := io.ReadAll(resp.Body)
if err != nil {
t.Fatalf("failed to read the answer: %v", err)
}
if resp.StatusCode != http.StatusOK {
t.Fatalf("status %d, want %d; body %s", resp.StatusCode, http.StatusOK, body)
}
if !strings.Contains(string(body), `"ptr":[]`) {
t.Errorf("answer %s, want an empty ptr", body)
}
_, err = os.Stat(filepath.Join(cfg.StateDir, "cache.db"))
if err != nil {
t.Errorf("cache.db is not in the state directory: %v", err)
}
}
+5 -31
View File
@@ -778,10 +778,6 @@ type IPLookupResult struct {
Flags IPLookupFlags `json:"flags"` Flags IPLookupFlags `json:"flags"`
// VPNProviders names the VPN lists the address is on. It is empty, never
// nil, when there are none.
VPNProviders []string `json:"vpn_providers"`
// IPLists has one entry for each IP list the address is on. // IPLists has one entry for each IP list the address is on.
IPLists []iplist.Listing `json:"ip_lists"` IPLists []iplist.Listing `json:"ip_lists"`
} }
@@ -794,9 +790,6 @@ type IPLookupFlags struct {
// IsTor says that the address is on the Tor exit list. // IsTor says that the address is on the Tor exit list.
IsTor bool `json:"is_tor"` IsTor bool `json:"is_tor"`
// IsVPN says that the address is on a list of VPN servers.
IsVPN bool `json:"is_vpn"`
// IsBogon says that the address is reserved, as package bogon tells. // IsBogon says that the address is reserved, as package bogon tells.
IsBogon bool `json:"is_bogon"` IsBogon bool `json:"is_bogon"`
} }
@@ -981,30 +974,14 @@ func (s *Server) ipLookupResult(ip string, ptrs []string, info IPInfo) IPLookupR
listings := s.ipListListings(ip) listings := s.ipListListings(ip)
return IPLookupResult{ return IPLookupResult{
IP: ip, IP: ip,
PTR: ptrs, PTR: ptrs,
IPInfo: info, IPInfo: info,
Flags: ipLookupFlags(ip, listings), Flags: ipLookupFlags(ip, listings),
VPNProviders: vpnProviders(listings), IPLists: listings,
IPLists: listings,
} }
} }
// vpnProviders returns the names of the VPN lists among listings. It returns
// an empty list, never nil, so that an answer's vpn_providers is [] rather
// than null.
func vpnProviders(listings []iplist.Listing) []string {
names := []string{}
for _, listing := range listings {
if listing.VPN {
names = append(names, listing.Name)
}
}
return names
}
// ipListListings returns the IP lists that contain ip. It returns an empty // ipListListings returns the IP lists that contain ip. It returns an empty
// list, never nil, so that an answer's ip_lists is [] rather than null. // list, never nil, so that an answer's ip_lists is [] rather than null.
func (s *Server) ipListListings(ip string) []iplist.Listing { func (s *Server) ipListListings(ip string) []iplist.Listing {
@@ -1022,9 +999,6 @@ func ipLookupFlags(ip string, listings []iplist.Listing) IPLookupFlags {
IsTor: slices.ContainsFunc(listings, func(l iplist.Listing) bool { IsTor: slices.ContainsFunc(listings, func(l iplist.Listing) bool {
return l.Name == iplist.TorExitsList return l.Name == iplist.TorExitsList
}), }),
IsVPN: slices.ContainsFunc(listings, func(l iplist.Listing) bool {
return l.VPN
}),
IsBogon: isBogon(ip), IsBogon: isBogon(ip),
} }
} }
+71 -197
View File
@@ -17,7 +17,6 @@ import (
"strings" "strings"
"sync/atomic" "sync/atomic"
"testing" "testing"
"testing/synctest"
"time" "time"
"git.eeqj.de/sneak/routewatch/internal/config" "git.eeqj.de/sneak/routewatch/internal/config"
@@ -175,9 +174,6 @@ type tableStats struct {
// carried 203.0.113.0/24, the two with kept routes withdrew it, so its origin // carried 203.0.113.0/24, the two with kept routes withdrew it, so its origin
// keeps no route. The oldest and newest routes are those of the other two // keeps no route. The oldest and newest routes are those of the other two
// prefixes. // prefixes.
//
// The requests run in a synctest bubble, whose clock is fake, so a stall on
// the host does not use up the handlers' 4-second limit.
func TestStatsHandlersAnswerFromMemory(t *testing.T) { func TestStatsHandlersAnswerFromMemory(t *testing.T) {
t.Parallel() t.Parallel()
@@ -223,11 +219,7 @@ func TestStatsHandlersAnswerFromMemory(t *testing.T) {
for name, handler := range handlers { for name, handler := range handlers {
rec := httptest.NewRecorder() rec := httptest.NewRecorder()
handler(rec, httptest.NewRequestWithContext(t.Context(), http.MethodGet, "/", nil))
synctest.Test(t, func(t *testing.T) {
handler(rec,
httptest.NewRequestWithContext(t.Context(), http.MethodGet, "/", nil))
})
if rec.Code != http.StatusOK { if rec.Code != http.StatusOK {
t.Errorf("%s: status %d, want %d; body %s", t.Errorf("%s: status %d, want %d; body %s",
@@ -262,20 +254,15 @@ func TestStatsHandlersAnswerFromMemory(t *testing.T) {
// TestStatsAnswerCarriesChurnPerMinute checks that the stream object of the // TestStatsAnswerCarriesChurnPerMinute checks that the stream object of the
// /api/v1/stats answer holds route_churn_per_min as a whole number, and no // /api/v1/stats answer holds route_churn_per_min as a whole number, and no
// longer route_churn_per_sec, the moving average it replaces // longer route_churn_per_sec, the moving average it replaces
// (https://git.eeqj.de/sneak/routewatch/issues/160). The request runs in a // (https://git.eeqj.de/sneak/routewatch/issues/160).
// synctest bubble, whose clock is fake, so a stall on the host does not use up
// the handler's 4-second limit.
func TestStatsAnswerCarriesChurnPerMinute(t *testing.T) { func TestStatsAnswerCarriesChurnPerMinute(t *testing.T) {
t.Parallel() t.Parallel()
s := newServerWithTestRoutes(t) s := newServerWithTestRoutes(t)
rec := httptest.NewRecorder() rec := httptest.NewRecorder()
s.handleStats()(rec,
synctest.Test(t, func(t *testing.T) { httptest.NewRequestWithContext(t.Context(), http.MethodGet, statsPath, nil))
s.handleStats()(rec,
httptest.NewRequestWithContext(t.Context(), http.MethodGet, statsPath, nil))
})
if rec.Code != http.StatusOK { if rec.Code != http.StatusOK {
t.Fatalf("status %d, want %d; body %s", rec.Code, http.StatusOK, rec.Body) t.Fatalf("status %d, want %d; body %s", rec.Code, http.StatusOK, rec.Body)
@@ -323,26 +310,18 @@ func (heldBackHandler) HeldBackCount() int { return heldBackMess
// TestStatsAnswerGivesHowManyMessagesAHandlerHoldsBack registers a handler // TestStatsAnswerGivesHowManyMessagesAHandlerHoldsBack registers a handler
// that holds back messages. Its figures in /api/v1/stats, which the status // that holds back messages. Its figures in /api/v1/stats, which the status
// page shows, must give how many. The request runs in a synctest bubble, whose // page shows, must give how many.
// clock is fake, so a stall on the host does not use up the handler's 4-second
// limit.
func TestStatsAnswerGivesHowManyMessagesAHandlerHoldsBack(t *testing.T) { func TestStatsAnswerGivesHowManyMessagesAHandlerHoldsBack(t *testing.T) {
t.Parallel() t.Parallel()
s := newServerWithTestRoutes(t) s := newServerWithTestRoutes(t)
s.streamer.RegisterHandler(heldBackHandler{}) s.streamer.RegisterHandler(heldBackHandler{})
var answer []byte
synctest.Test(t, func(t *testing.T) {
answer = get(t, s, statsPath).Body.Bytes()
})
var body struct { var body struct {
Data StatsResponse `json:"data"` Data StatsResponse `json:"data"`
} }
err := json.Unmarshal(answer, &body) err := json.Unmarshal(get(t, s, statsPath).Body.Bytes(), &body)
if err != nil { if err != nil {
t.Fatalf("decoding the answer: %v", err) t.Fatalf("decoding the answer: %v", err)
} }
@@ -386,9 +365,7 @@ type unitStats struct {
// names. They used to be strings with the unit in them, such as "1h18m34s" or // names. They used to be strings with the unit in them, such as "1h18m34s" or
// "1.1 GB" (https://git.eeqj.de/sneak/routewatch/issues/199). The stream has // "1.1 GB" (https://git.eeqj.de/sneak/routewatch/issues/199). The stream has
// never connected, so the uptime is 0. The memory use in /api/v1/stats is the // never connected, so the uptime is 0. The memory use in /api/v1/stats is the
// same number of bytes as its gc.heap_alloc_bytes. The requests run in a // same number of bytes as its gc.heap_alloc_bytes.
// synctest bubble, whose clock is fake, so a stall on the host does not use up
// the handlers' 4-second limit.
func TestStatsAnswersGiveUnitsInFieldNames(t *testing.T) { func TestStatsAnswersGiveUnitsInFieldNames(t *testing.T) {
t.Parallel() t.Parallel()
@@ -398,11 +375,7 @@ func TestStatsAnswersGiveUnitsInFieldNames(t *testing.T) {
s.SetASNFetcher(fixedIntervalASNFetcher{interval: fetchInterval}) s.SetASNFetcher(fixedIntervalASNFetcher{interval: fetchInterval})
for _, path := range []string{statusJSONPath, statsPath} { for _, path := range []string{statusJSONPath, statsPath} {
var answer []byte answer := get(t, s, path).Body.Bytes()
synctest.Test(t, func(t *testing.T) {
answer = get(t, s, path).Body.Bytes()
})
var body struct { var body struct {
Data unitStats `json:"data"` Data unitStats `json:"data"`
@@ -443,89 +416,82 @@ func TestStatsAnswersGiveUnitsInFieldNames(t *testing.T) {
// connection_duration_seconds of /api/v1/stats is null while the stream is not // connection_duration_seconds of /api/v1/stats is null while the stream is not
// connected, and a whole number of seconds once it is. It used to be the // connected, and a whole number of seconds once it is. It used to be the
// string "disconnected", or a Go duration such as "1h18m34s" // string "disconnected", or a Go duration such as "1h18m34s"
// (https://git.eeqj.de/sneak/routewatch/issues/199). The test runs in a // (https://git.eeqj.de/sneak/routewatch/issues/199).
// synctest bubble, whose clock is fake, so a stall on the host does not use up
// the handler's 4-second limit.
func TestStatsAnswerGivesConnectionDurationInSeconds(t *testing.T) { func TestStatsAnswerGivesConnectionDurationInSeconds(t *testing.T) {
t.Parallel() t.Parallel()
s := newServerWithTestRoutes(t) s := newServerWithTestRoutes(t)
synctest.Test(t, func(t *testing.T) { connectionDuration := func() json.RawMessage {
connectionDuration := func() json.RawMessage { answer := get(t, s, statsPath).Body.Bytes()
answer := get(t, s, statsPath).Body.Bytes()
var body struct { var body struct {
Data map[string]json.RawMessage `json:"data"` Data map[string]json.RawMessage `json:"data"`
}
err := json.Unmarshal(answer, &body)
if err != nil {
t.Fatalf("decoding the answer: %v; answer %s", err, answer)
}
duration, ok := body.Data["connection_duration_seconds"]
if !ok {
t.Fatalf("no connection_duration_seconds: %s", answer)
}
return duration
} }
disconnected := connectionDuration() err := json.Unmarshal(answer, &body)
if string(disconnected) != "null" { if err != nil {
t.Errorf("disconnected: connection_duration_seconds = %s, want null", t.Fatalf("decoding the answer: %v; answer %s", err, answer)
disconnected)
} }
s.streamer.GetMetricsTracker().SetConnected(true) duration, ok := body.Data["connection_duration_seconds"]
if !ok {
var seconds int64 t.Fatalf("no connection_duration_seconds: %s", answer)
connected := connectionDuration()
err := json.Unmarshal(connected, &seconds)
if err != nil || seconds != 0 {
t.Errorf("just connected: connection_duration_seconds = %s, want 0",
connected)
} }
})
return duration
}
disconnected := connectionDuration()
if string(disconnected) != "null" {
t.Errorf("disconnected: connection_duration_seconds = %s, want null",
disconnected)
}
s.streamer.GetMetricsTracker().SetConnected(true)
var seconds int64
connected := connectionDuration()
err := json.Unmarshal(connected, &seconds)
if err != nil || seconds != 0 {
t.Errorf("just connected: connection_duration_seconds = %s, want 0",
connected)
}
} }
// TestStreamDurationsCountSeconds checks that the uptime and the connection // TestStreamDurationsCountSeconds checks that the uptime and the connection
// duration of a stream that connected 90 seconds ago are both 90. A stream // duration of a stream that connected 90 seconds ago are both 90. A stream
// that has never connected has an uptime of 0, and one that is not connected // that has never connected has an uptime of 0, and one that is not connected
// has no connection duration. The test runs in a synctest bubble, whose clock // has no connection duration.
// is fake, so a stall on the host does not add to the 90 seconds.
func TestStreamDurationsCountSeconds(t *testing.T) { func TestStreamDurationsCountSeconds(t *testing.T) {
t.Parallel() t.Parallel()
synctest.Test(t, func(t *testing.T) { const connectedFor = 90 * time.Second
const connectedFor = 90 * time.Second
connectedSince := time.Now().Add(-connectedFor) connectedSince := time.Now().Add(-connectedFor)
want := int64(connectedFor / time.Second) want := int64(connectedFor / time.Second)
if got := uptimeSeconds(connectedSince); got != want { if got := uptimeSeconds(connectedSince); got != want {
t.Errorf("uptimeSeconds = %d, want %d", got, want) t.Errorf("uptimeSeconds = %d, want %d", got, want)
} }
if got := uptimeSeconds(time.Time{}); got != 0 { if got := uptimeSeconds(time.Time{}); got != 0 {
t.Errorf("uptimeSeconds before any connection = %d, want 0", got) t.Errorf("uptimeSeconds before any connection = %d, want 0", got)
} }
switch got := connectionDurationSeconds(true, connectedSince); { switch got := connectionDurationSeconds(true, connectedSince); {
case got == nil: case got == nil:
t.Errorf("connectionDurationSeconds = nil, want %d", want) t.Errorf("connectionDurationSeconds = nil, want %d", want)
case *got != want: case *got != want:
t.Errorf("connectionDurationSeconds = %d, want %d", *got, want) t.Errorf("connectionDurationSeconds = %d, want %d", *got, want)
} }
if got := connectionDurationSeconds(false, connectedSince); got != nil { if got := connectionDurationSeconds(false, connectedSince); got != nil {
t.Errorf("connectionDurationSeconds while not connected = %d, want nil", t.Errorf("connectionDurationSeconds while not connected = %d, want nil",
*got) *got)
} }
})
} }
// newTestTable returns a routing table that holds a route for testPrefixV4, // newTestTable returns a routing table that holds a route for testPrefixV4,
@@ -1172,9 +1138,7 @@ func (failingDB) GetPrefixFirstSeen(
// with the driver's error text. It also sends a prefix that does not parse. // with the driver's error text. It also sends a prefix that does not parse.
// No answer may carry the driver's or the prefix parser's text. Before the fix // No answer may carry the driver's or the prefix parser's text. Before the fix
// the answers sent it to the client // the answers sent it to the client
// (https://git.eeqj.de/sneak/routewatch/issues/108). The requests run in a // (https://git.eeqj.de/sneak/routewatch/issues/108).
// synctest bubble, whose clock is fake, so a stall on the host does not use up
// the stats handlers' 4-second limit.
func TestJSONRoutesDoNotSendErrorDetails(t *testing.T) { func TestJSONRoutesDoNotSendErrorDetails(t *testing.T) {
t.Parallel() t.Parallel()
@@ -1195,11 +1159,8 @@ func TestJSONRoutesDoNotSendErrorDetails(t *testing.T) {
for _, tt := range tests { for _, tt := range tests {
rec := httptest.NewRecorder() rec := httptest.NewRecorder()
s.router.ServeHTTP(rec, httptest.NewRequestWithContext(
synctest.Test(t, func(t *testing.T) { t.Context(), http.MethodGet, tt.path, nil))
s.router.ServeHTTP(rec, httptest.NewRequestWithContext(
t.Context(), http.MethodGet, tt.path, nil))
})
if rec.Code != tt.wantCode { if rec.Code != tt.wantCode {
t.Errorf("%s: status %d, want %d; body %s", t.Errorf("%s: status %d, want %d; body %s",
@@ -1453,11 +1414,8 @@ func TestIPLookupKeepsRouteWhenPTRLookupDoesNotAnswer(t *testing.T) {
s := newServerWithTestRoutes(t) s := newServerWithTestRoutes(t)
useFakeResolver(s, &fakeResolver{ptrNeverAnswers: true}) useFakeResolver(s, &fakeResolver{ptrNeverAnswers: true})
// The request's time limit passes well after the PTR lookup's, so that a // The request's time limit passes after the PTR lookup's.
// stall on a busy host does not use up the difference. ctx, cancel := context.WithTimeout(t.Context(), 2*ptr.LookupTimeout)
const requestTimeLimit = ptr.LookupTimeout + 15*time.Second
ctx, cancel := context.WithTimeout(t.Context(), requestTimeLimit)
defer cancel() defer cancel()
rec := httptest.NewRecorder() rec := httptest.NewRecorder()
@@ -1504,19 +1462,8 @@ const listedUnroutedAddr = "203.0.113.1"
// torExitAddr is the address that fakeIPLists lists on the Tor exit list. // torExitAddr is the address that fakeIPLists lists on the Tor exit list.
const torExitAddr = "203.0.113.2" const torExitAddr = "203.0.113.2"
// The addresses that fakeIPLists lists on one and on two VPN lists, and the // fakeIPLists has two lists. The test list holds addrInTestPrefixV4 and
// names of those lists. // listedUnroutedAddr, and the Tor exit list holds torExitAddr.
const (
oneVPNListAddr = "203.0.113.3"
twoVPNListsAddr = "203.0.113.4"
testVPNList = "test_vpn"
otherTestVPN = "other_test_vpn"
)
// fakeIPLists has four lists. The test list holds addrInTestPrefixV4,
// listedUnroutedAddr and twoVPNListsAddr. The Tor exit list holds torExitAddr.
// The test VPN list holds oneVPNListAddr and twoVPNListsAddr, and the other
// test VPN list holds twoVPNListsAddr.
type fakeIPLists struct{} type fakeIPLists struct{}
func testListing() iplist.Listing { func testListing() iplist.Listing {
@@ -1537,18 +1484,6 @@ func (fakeIPLists) Lookup(addr netip.Addr) []iplist.Listing {
return []iplist.Listing{{Name: iplist.TorExitsList}} return []iplist.Listing{{Name: iplist.TorExitsList}}
} }
if addr == netip.MustParseAddr(oneVPNListAddr) {
return []iplist.Listing{{Name: testVPNList, VPN: true}}
}
if addr == netip.MustParseAddr(twoVPNListsAddr) {
return []iplist.Listing{
{Name: otherTestVPN, VPN: true},
testListing(),
{Name: testVPNList, VPN: true},
}
}
return []iplist.Listing{} return []iplist.Listing{}
} }
@@ -1557,7 +1492,7 @@ func (fakeIPLists) ShortestInterval() time.Duration {
} }
// TestIPLookupFlags looks up addresses with and without a route. Every result // TestIPLookupFlags looks up addresses with and without a route. Every result
// has every flag, under the JSON keys the API documents. // has both flags, under the JSON keys the API documents.
func TestIPLookupFlags(t *testing.T) { func TestIPLookupFlags(t *testing.T) {
t.Parallel() t.Parallel()
@@ -1603,9 +1538,7 @@ func TestIPLookupFlags(t *testing.T) {
t.Fatalf("%s: decoding flags %s: %v", path, results[0]["flags"], err) t.Fatalf("%s: decoding flags %s: %v", path, results[0]["flags"], err)
} }
want := map[string]bool{ want := map[string]bool{"is_tor": tt.tor, "is_bogon": tt.bogon}
"is_tor": tt.tor, "is_vpn": false, "is_bogon": tt.bogon,
}
if !maps.Equal(got, want) { if !maps.Equal(got, want) {
t.Errorf("%s: flags %v, want %v", path, got, want) t.Errorf("%s: flags %v, want %v", path, got, want)
} }
@@ -1613,61 +1546,6 @@ func TestIPLookupFlags(t *testing.T) {
} }
} }
// TestIPLookupReportsVPNProviders looks up addresses on no list, on a list
// that is not a VPN list, on one VPN list and on two. is_vpn says whether any
// VPN list holds the address, and vpn_providers names only those lists. An
// address on none has [], not null.
func TestIPLookupReportsVPNProviders(t *testing.T) {
t.Parallel()
s := newServerWithTestRoutes(t)
s.resolver = &fakeResolver{}
s.SetIPLists(fakeIPLists{})
tests := []struct {
name string
addr string
isVPN string
providers string
}{
{"on no list", "1.1.1.1", "false", `[]`},
{"on a list that is not a VPN list", listedUnroutedAddr, "false", `[]`},
{"on one VPN list", oneVPNListAddr, "true", `["` + testVPNList + `"]`},
{
"on two VPN lists", twoVPNListsAddr, "true",
`["` + otherTestVPN + `","` + testVPNList + `"]`,
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
t.Parallel()
path := "/api/v1/ip/" + tt.addr
results := resultFields(t, get(t, s, path).Body.Bytes())
if len(results) != 1 {
t.Fatalf("%s: %d results, want 1", path, len(results))
}
var flags map[string]json.RawMessage
err := json.Unmarshal(results[0]["flags"], &flags)
if err != nil {
t.Fatalf("%s: decoding flags %s: %v", path, results[0]["flags"], err)
}
if got := string(flags["is_vpn"]); got != tt.isVPN {
t.Errorf("%s: is_vpn %s, want %s", path, got, tt.isVPN)
}
if got := string(results[0]["vpn_providers"]); got != tt.providers {
t.Errorf("%s: vpn_providers %s, want %s", path, got, tt.providers)
}
})
}
}
// TestIPLookupReportsIPLists looks up two routed addresses, one of them on an // TestIPLookupReportsIPLists looks up two routed addresses, one of them on an
// IP list. The listed address's result names the list, says what it means and // IP list. The listed address's result names the list, says what it means and
// when it was updated. The other's result has an empty list. // when it was updated. The other's result has an empty list.
@@ -2456,13 +2334,9 @@ func settledGoroutineCount() int {
return prev return prev
} }
// waitForGoroutines waits up to 15 seconds for the goroutine count to drop to // waitForGoroutines waits until the goroutine count drops to target or below.
// target or below.
func waitForGoroutines(target int) bool { func waitForGoroutines(target int) bool {
const maxWait = 15 * time.Second for range 100 {
deadline := time.Now().Add(maxWait)
for time.Now().Before(deadline) {
if runtime.NumGoroutine() <= target { if runtime.NumGoroutine() <= target {
return true return true
} }
+4 -8
View File
@@ -76,9 +76,7 @@ func asLink(asn int) string {
} }
// TestHomePageCardsShowTheTableFigures fills a routing table and the asns // TestHomePageCardsShowTheTableFigures fills a routing table and the asns
// table, works out the cards, and checks that each card shows its figure. The // table, works out the cards, and checks that each card shows its figure.
// cards are worked out in a synctest bubble, whose clock is fake, so a stall on
// the host does not use up the 4-second limit on their database read.
func TestHomePageCardsShowTheTableFigures(t *testing.T) { func TestHomePageCardsShowTheTableFigures(t *testing.T) {
t.Parallel() t.Parallel()
@@ -97,9 +95,7 @@ func TestHomePageCardsShowTheTableFigures(t *testing.T) {
t.Fatalf("GetOrCreateASNBatch: %v", err) t.Fatalf("GetOrCreateASNBatch: %v", err)
} }
synctest.Test(t, func(t *testing.T) { s.refreshHomeCards(t.Context())
s.refreshHomeCards(t.Context())
})
page := get(t, s, "/").Body.String() page := get(t, s, "/").Body.String()
@@ -283,7 +279,7 @@ func TestKeepHomeCardsFreshWorksThemOutEveryInterval(t *testing.T) {
const ( const (
interval = 10 * time.Millisecond interval = 10 * time.Millisecond
maxWait = 15 * time.Second maxWait = 5 * time.Second
) )
table := emptyTable() table := emptyTable()
@@ -325,7 +321,7 @@ func TestKeepHomeCardsFreshWorksThemOutEveryInterval(t *testing.T) {
func TestStartedServerWorksOutTheCards(t *testing.T) { func TestStartedServerWorksOutTheCards(t *testing.T) {
t.Parallel() t.Parallel()
const maxWait = 15 * time.Second const maxWait = 5 * time.Second
s := newServerWithTable(t, newCardsTable()) s := newServerWithTable(t, newCardsTable())
+32 -39
View File
@@ -14,7 +14,6 @@ import (
"regexp" "regexp"
"strings" "strings"
"testing" "testing"
"testing/synctest"
"time" "time"
"git.eeqj.de/sneak/routewatch/internal/config" "git.eeqj.de/sneak/routewatch/internal/config"
@@ -164,10 +163,6 @@ func TestJSONAnswersPutNoUnitInAString(t *testing.T) {
// router and the middleware write. Before the fix a panic in a handler, and a // router and the middleware write. Before the fix a panic in a handler, and a
// method /status.json or the health check does not take, answered with an // method /status.json or the health check does not take, answered with an
// empty body (https://git.eeqj.de/sneak/routewatch/issues/201). // empty body (https://git.eeqj.de/sneak/routewatch/issues/201).
//
// The requests run in a synctest bubble, whose clock is fake, so a stall on
// the host does not use up the 2-second limit of the health check or the
// 4-second limit of the stats handlers.
func TestJSONAnswersCarryMeta(t *testing.T) { func TestJSONAnswersCarryMeta(t *testing.T) {
t.Parallel() t.Parallel()
@@ -186,10 +181,6 @@ func TestJSONAnswersCarryMeta(t *testing.T) {
s.router.Get(panicsPath, JSONValidationMiddleware(http.HandlerFunc( s.router.Get(panicsPath, JSONValidationMiddleware(http.HandlerFunc(
func(http.ResponseWriter, *http.Request) { panic(errDiskIO) })).ServeHTTP) func(http.ResponseWriter, *http.Request) { panic(errDiskIO) })).ServeHTTP)
failing := New(failingDB{}, newTestTable(),
streamer.New(logger.New(), metrics.New()), logger.New(), &config.Config{})
useFakeResolver(failing, &fakeResolver{})
tests := []struct { tests := []struct {
method string method string
path string path string
@@ -225,40 +216,42 @@ func TestJSONAnswersCarryMeta(t *testing.T) {
{http.MethodGet, panicsPath, http.StatusInternalServerError}, {http.MethodGet, panicsPath, http.StatusInternalServerError},
} }
synctest.Test(t, func(t *testing.T) { for _, tt := range tests {
for _, tt := range tests { checkAnswerCarriesMeta(t, s, httptest.NewRequestWithContext(
checkAnswerCarriesMeta(t, s, httptest.NewRequestWithContext( t.Context(), tt.method, tt.path, nil), tt.wantCode)
t.Context(), tt.method, tt.path, nil), tt.wantCode) }
}
// A request whose time limit has already passed answers 504. The // A request whose time limit has already passed answers 504. The handlers
// handlers write the answers of the first five routes, and the JSON // write the answers of the first five routes, and the JSON check on the
// check on the route writes the last. // route writes the last.
pastTimeLimit, cancel := context.WithDeadline(t.Context(), time.Now()) pastTimeLimit, cancel := context.WithDeadline(t.Context(), time.Now())
defer cancel() defer cancel()
for _, path := range []string{ for _, path := range []string{
statusJSONPath, statsPath, "/ip/" + addrInTestPrefixV4, statusJSONPath, statsPath, "/ip/" + addrInTestPrefixV4,
"/api/v1/ip/" + addrInTestPrefixV4, asJSONPath, writesNothingPath, "/api/v1/ip/" + addrInTestPrefixV4, asJSONPath, writesNothingPath,
} { } {
checkAnswerCarriesMeta(t, s, httptest.NewRequestWithContext( checkAnswerCarriesMeta(t, s, httptest.NewRequestWithContext(
pastTimeLimit, http.MethodGet, path, nil), http.StatusGatewayTimeout) pastTimeLimit, http.MethodGet, path, nil), http.StatusGatewayTimeout)
} }
// With a store whose reads fail, the health check answers 503, and // With a store whose reads fail, the health check answers 503, and the
// the handlers of the other routes 500. // handlers of the other routes 500.
failing := New(failingDB{}, newTestTable(),
streamer.New(logger.New(), metrics.New()), logger.New(), &config.Config{})
useFakeResolver(failing, &fakeResolver{})
checkAnswerCarriesMeta(t, failing, httptest.NewRequestWithContext(
t.Context(), http.MethodGet, healthCheckPath, nil),
http.StatusServiceUnavailable)
for _, path := range []string{
statusJSONPath, statsPath, asJSONPath,
"/api/v1/ip/" + addrInTestPrefixV4, "/ip/" + addrInTestPrefixV4,
} {
checkAnswerCarriesMeta(t, failing, httptest.NewRequestWithContext( checkAnswerCarriesMeta(t, failing, httptest.NewRequestWithContext(
t.Context(), http.MethodGet, healthCheckPath, nil), t.Context(), http.MethodGet, path, nil), http.StatusInternalServerError)
http.StatusServiceUnavailable) }
for _, path := range []string{
statusJSONPath, statsPath, asJSONPath,
"/api/v1/ip/" + addrInTestPrefixV4, "/ip/" + addrInTestPrefixV4,
} {
checkAnswerCarriesMeta(t, failing, httptest.NewRequestWithContext(
t.Context(), http.MethodGet, path, nil), http.StatusInternalServerError)
}
})
} }
// checkAnswerCarriesMeta serves req through the server's router, and checks // checkAnswerCarriesMeta serves req through the server's router, and checks
+2 -11
View File
@@ -5,7 +5,6 @@ import (
"net/http" "net/http"
"net/http/httptest" "net/http/httptest"
"testing" "testing"
"testing/synctest"
) )
// TestRoutesAnswerHEAD sends HEAD for a page and JSON routes through the // TestRoutesAnswerHEAD sends HEAD for a page and JSON routes through the
@@ -13,23 +12,15 @@ import (
// lookup of the caller's own address, are in the list because chi's GetHead // lookup of the caller's own address, are in the list because chi's GetHead
// does not reach them. Before the fix every route answered HEAD with 405 // does not reach them. Before the fix every route answered HEAD with 405
// (https://git.eeqj.de/sneak/routewatch/issues/111). // (https://git.eeqj.de/sneak/routewatch/issues/111).
//
// The requests run in a synctest bubble, whose clock is fake, so a stall on
// the host does not use up the stats handlers' 4-second limit. The resolver
// is fake because no time limit would cut off a real DNS lookup in a bubble.
func TestRoutesAnswerHEAD(t *testing.T) { func TestRoutesAnswerHEAD(t *testing.T) {
t.Parallel() t.Parallel()
s := newServerWithTestRoutes(t) s := newServerWithTestRoutes(t)
s.resolver = &fakeResolver{}
for _, path := range []string{"/", statusJSONPath, statsPath, "/ip", "/ip/"} { for _, path := range []string{"/", statusJSONPath, statsPath, "/ip", "/ip/"} {
rec := httptest.NewRecorder() rec := httptest.NewRecorder()
s.router.ServeHTTP(rec, httptest.NewRequestWithContext(
synctest.Test(t, func(t *testing.T) { t.Context(), http.MethodHead, path, nil))
s.router.ServeHTTP(rec, httptest.NewRequestWithContext(
t.Context(), http.MethodHead, path, nil))
})
if rec.Code != http.StatusOK { if rec.Code != http.StatusOK {
t.Errorf("HEAD %s: status %d, want %d; body %s", t.Errorf("HEAD %s: status %d, want %d; body %s",
+2 -2
View File
@@ -22,10 +22,10 @@ func TestStopClosesTheConnectionOfARequestStillRunning(t *testing.T) {
stopDeadline = 100 * time.Millisecond stopDeadline = 100 * time.Millisecond
// waitTimeout is how long the test waits for the request to reach // waitTimeout is how long the test waits for the request to reach
// the handler, and then for the client to see its connection closed. // the handler, and then for the client to see its connection closed.
waitTimeout = 15 * time.Second waitTimeout = 2 * time.Second
// readHeaderTimeout bounds the test server's read of the request // readHeaderTimeout bounds the test server's read of the request
// header, as every http.Server must. // header, as every http.Server must.
readHeaderTimeout = waitTimeout readHeaderTimeout = time.Second
) )
release := make(chan struct{}) release := make(chan struct{})
+73 -130
View File
@@ -14,7 +14,6 @@ import (
"strings" "strings"
"sync/atomic" "sync/atomic"
"testing" "testing"
"testing/synctest"
"time" "time"
"git.eeqj.de/sneak/routewatch/internal/logger" "git.eeqj.de/sneak/routewatch/internal/logger"
@@ -399,7 +398,7 @@ func TestStopQueuesTheHeldStateMessages(t *testing.T) {
const ( const (
held = 2 held = 2
lines = held + 1 lines = held + 1
maxWait = 15 * time.Second maxWait = 5 * time.Second
) )
srv := httptest.NewServer(http.HandlerFunc( srv := httptest.NewServer(http.HandlerFunc(
@@ -479,15 +478,15 @@ func TestStopQueuesTheHeldStateMessages(t *testing.T) {
// TestStopReturnsByItsDeadline stops a streamer whose handler has a full queue // TestStopReturnsByItsDeadline stops a streamer whose handler has a full queue
// and a STATE message held for it, and a worker that stays busy until the test // and a STATE message held for it, and a worker that stays busy until the test
// ends, as one does whose handler waits for a database write. Stop must return // ends, as one does whose handler waits for a database write. Stop must return
// within its deadline plus a margin instead of waiting for room on the queue // within its deadline plus a small margin instead of waiting for room on the
// and for the worker, and log as not handed over the held message and every // queue and for the worker, and log as not handed over the held message and
// message on the queue. // every message on the queue.
func TestStopReturnsByItsDeadline(t *testing.T) { func TestStopReturnsByItsDeadline(t *testing.T) {
t.Parallel() t.Parallel()
const ( const (
deadline = 100 * time.Millisecond deadline = 100 * time.Millisecond
margin = 15 * time.Second margin = 2 * time.Second
) )
var logs bytes.Buffer var logs bytes.Buffer
@@ -578,7 +577,7 @@ func TestStopBeforeMessageReachesQueues(t *testing.T) {
func TestReconnectWhenStreamEnds(t *testing.T) { func TestReconnectWhenStreamEnds(t *testing.T) {
t.Parallel() t.Parallel()
endResponse := pipeTransport{write: func(context.Context, io.Writer) {}} endResponse := func(http.ResponseWriter, *http.Request) {}
checkReconnect(t, endResponse, streamIdleTimeout) checkReconnect(t, endResponse, streamIdleTimeout)
} }
@@ -590,8 +589,9 @@ func TestReconnectWhenStreamGoesIdle(t *testing.T) {
const idleTimeout = 50 * time.Millisecond const idleTimeout = 50 * time.Millisecond
holdResponse := pipeTransport{ holdResponse := func(w http.ResponseWriter, r *http.Request) {
write: func(ctx context.Context, _ io.Writer) { <-ctx.Done() }, flush(w)
<-r.Context().Done()
} }
checkReconnect(t, holdResponse, idleTimeout) checkReconnect(t, holdResponse, idleTimeout)
} }
@@ -604,51 +604,16 @@ func TestReconnectWhenServerNeverAnswers(t *testing.T) {
const idleTimeout = 50 * time.Millisecond const idleTimeout = 50 * time.Millisecond
neverAnswer := roundTripFunc( neverAnswer := func(_ http.ResponseWriter, r *http.Request) {
func(req *http.Request) (*http.Response, error) { <-r.Context().Done()
<-req.Context().Done() }
return nil, req.Context().Err()
},
)
checkReconnect(t, neverAnswer, idleTimeout) checkReconnect(t, neverAnswer, idleTimeout)
} }
// pipeTransport answers every request with a response whose body write fills. // TestStreamNotDroppedWhileLinesArrive has the server send lines well within
// The body fails when the request's context ends, as a connection's does. // the idle timeout for several times the timeout, then end the response. Each
// write gets that context, so that it can end when the request does. // line must start the idle timeout over, so the stream lasts until the server
type pipeTransport struct { // ends it instead of being dropped as idle.
write func(context.Context, io.Writer)
}
func (p pipeTransport) RoundTrip(req *http.Request) (*http.Response, error) {
body, w := io.Pipe()
context.AfterFunc(req.Context(), func() {
_ = w.CloseWithError(req.Context().Err())
})
go func(ctx context.Context) {
p.write(ctx, w)
_ = w.Close()
}(req.Context())
return &http.Response{StatusCode: http.StatusOK, Body: body}, nil
}
// roundTripFunc makes a function an http.RoundTripper.
type roundTripFunc func(*http.Request) (*http.Response, error)
func (f roundTripFunc) RoundTrip(req *http.Request) (*http.Response, error) {
return f(req)
}
// TestStreamNotDroppedWhileLinesArrive has a response send lines well within
// the idle timeout for several times the timeout, then end. Each line must
// start the idle timeout over, so the stream lasts until the response ends
// instead of being dropped as idle. It runs in a synctest bubble, whose clock
// is fake, so a stall on the host does not use up the timeout.
func TestStreamNotDroppedWhileLinesArrive(t *testing.T) { func TestStreamNotDroppedWhileLinesArrive(t *testing.T) {
t.Parallel() t.Parallel()
@@ -658,35 +623,26 @@ func TestStreamNotDroppedWhileLinesArrive(t *testing.T) {
sendFor = 5 * idleTimeout sendFor = 5 * idleTimeout
) )
// The streamer is made outside the bubble. Its metrics start a goroutine srv := httptest.NewServer(http.HandlerFunc(
// that never returns, and a bubble waits for each of its goroutines to func(w http.ResponseWriter, _ *http.Request) {
// return.
s := New(logger.New(), metrics.New())
s.idleTimeout = idleTimeout
s.client = &http.Client{Transport: pipeTransport{
write: func(ctx context.Context, w io.Writer) {
deadline := time.Now().Add(sendFor) deadline := time.Now().Add(sendFor)
for time.Now().Before(deadline) { for time.Now().Before(deadline) {
_, _ = io.WriteString(w, updateLine) _, _ = io.WriteString(w, updateLine)
// The wait ends with the stream. A bubble's clock stops when flush(w)
// the test function returns, and a goroutine then still time.Sleep(lineInterval)
// asleep panics the test binary.
select {
case <-ctx.Done():
return
case <-time.After(lineInterval):
}
} }
}, }))
}} defer srv.Close()
synctest.Test(t, func(t *testing.T) { s := New(logger.New(), metrics.New())
err := s.stream(context.Background()) s.url = srv.URL
if !errors.Is(err, errStreamEnded) { s.idleTimeout = idleTimeout
t.Fatalf("stream returned %v, want %v", err, errStreamEnded)
} err := s.stream(context.Background())
}) if !errors.Is(err, errStreamEnded) {
t.Fatalf("stream returned %v, want %v", err, errStreamEnded)
}
} }
// TestStreamReadsLongLine has the server send one UPDATE line of about // TestStreamReadsLongLine has the server send one UPDATE line of about
@@ -750,7 +706,7 @@ func TestHandlerStatsReadWhileMessagesAreDropped(t *testing.T) {
const ( const (
lines = 5 lines = 5
wantDropped = lines - 2 wantDropped = lines - 2
maxWait = 15 * time.Second maxWait = 5 * time.Second
) )
srv := httptest.NewServer(http.HandlerFunc( srv := httptest.NewServer(http.HandlerFunc(
@@ -864,70 +820,61 @@ func streamLines(t *testing.T, s *Streamer, lines string) {
// line and then stays open. The streamer must come back for a second // line and then stays open. The streamer must come back for a second
// response, must not be marked connected when it does, and must handle the // response, must not be marked connected when it does, and must handle the
// UPDATE in it. // UPDATE in it.
//
// It runs in a synctest bubble, whose clock is fake, so a stall on the host
// does not use up the idle timeout. The fake server gets the requests in the
// order the streamer makes them, because no real connection carries them.
func checkReconnect( func checkReconnect(
t *testing.T, firstResponse http.RoundTripper, idleTimeout time.Duration, t *testing.T, firstResponse http.HandlerFunc, idleTimeout time.Duration,
) { ) {
t.Helper() t.Helper()
const maxWait = 15 * time.Second const maxWait = 5 * time.Second
// The tracker is made outside the bubble. It starts a goroutine that never
// returns, and a bubble waits for each of its goroutines to return.
tracker := metrics.New() tracker := metrics.New()
connectedAtReconnect := make(chan bool, 1)
synctest.Test(t, func(t *testing.T) { var requests atomic.Int32
connectedAtReconnect := make(chan bool, 1)
laterResponse := pipeTransport{
write: func(ctx context.Context, w io.Writer) {
_, _ = io.WriteString(w, updateLine)
<-ctx.Done() srv := httptest.NewServer(http.HandlerFunc(
}, func(w http.ResponseWriter, r *http.Request) {
} if requests.Add(1) == 1 {
firstResponse(w, r)
var requests atomic.Int32 return
}
s := New(logger.New(), tracker) select {
s.client = &http.Client{Transport: roundTripFunc( case connectedAtReconnect <- tracker.IsConnected():
func(req *http.Request) (*http.Response, error) { default:
if requests.Add(1) == 1 { }
return firstResponse.RoundTrip(req)
}
select { _, _ = io.WriteString(w, updateLine)
case connectedAtReconnect <- tracker.IsConnected():
default:
}
return laterResponse.RoundTrip(req) flush(w)
}, <-r.Context().Done()
)} }))
s.idleTimeout = idleTimeout defer srv.Close()
s.minBackoff = time.Millisecond
handler := updateHandler{handled: make(chan struct{}, 1)}
s.RegisterHandler(handler)
err := s.Start(t.Context()) s := New(logger.New(), tracker)
if err != nil { s.url = srv.URL
t.Fatalf("Start returned error: %v", err) s.idleTimeout = idleTimeout
} s.minBackoff = time.Millisecond
defer s.Stop(t.Context()) handler := updateHandler{handled: make(chan struct{}, 1)}
s.RegisterHandler(handler)
select { err := s.Start(t.Context())
case <-handler.handled: if err != nil {
case <-time.After(maxWait): t.Fatalf("Start returned error: %v", err)
t.Fatalf("no message from a second response handled within %s", maxWait) }
} defer s.Stop(t.Context())
if <-connectedAtReconnect { select {
t.Error("streamer was still marked connected when it came back " + case <-handler.handled:
"for a second response") case <-time.After(maxWait):
} t.Fatalf("no message from a second response handled within %s", maxWait)
}) }
if <-connectedAtReconnect {
t.Error("streamer was still marked connected when it came back " +
"for a second response")
}
} }
// settledGoroutineCount lets transient goroutines finish, then reports the // settledGoroutineCount lets transient goroutines finish, then reports the
@@ -949,13 +896,9 @@ func settledGoroutineCount() int {
return prev return prev
} }
// waitForGoroutines waits up to 15 seconds for the goroutine count to drop to // waitForGoroutines waits until the goroutine count drops to target or below.
// target or below.
func waitForGoroutines(target int) bool { func waitForGoroutines(target int) bool {
const maxWait = 15 * time.Second for range 100 {
deadline := time.Now().Add(maxWait)
for time.Now().Before(deadline) {
if runtime.NumGoroutine() <= target { if runtime.NumGoroutine() <= target {
return true return true
} }