Give every IP lookup result the same fields (closes #200) #206
@@ -141,15 +141,39 @@ one of its peers sends an update.
|
|||||||
last full clock minute.
|
last full clock minute.
|
||||||
- `GET /api/v1/ip/{ip}` - Looks up each address. `{ip}` is an IP address, a host
|
- `GET /api/v1/ip/{ip}` - Looks up each address. `{ip}` is an IP address, a host
|
||||||
name, or a comma-separated list of at most 10 of these. `results` has one
|
name, or a comma-separated list of at most 10 of these. `results` has one
|
||||||
entry for each address, with:
|
entry for each address. Every entry has the same fields:
|
||||||
- `ip` and `ptr`: the address and its PTR names
|
- `ip`: the address
|
||||||
- `netblock` and the fields beside it: the most specific prefix that holds
|
- `ptr`: the address's PTR names, or an empty list when it has none
|
||||||
the address, and its origin AS with that AS's WHOIS details. For a prefix
|
- `blocklists`: the public blocklists the address is on (see Blocklists), or
|
||||||
with several origin ASes, `asn` is the one that most peers carry.
|
an empty list when it is on none
|
||||||
`last_seen` is the time of the prefix's newest kept route.
|
|
||||||
- `blocklists`: the public blocklists the address is on (see Blocklists)
|
Every other field is about the address's live route. Each is null when the
|
||||||
|
address has no live route. The fields about the route:
|
||||||
|
- `netblock`: the most specific prefix that holds the address
|
||||||
|
- `mask_length`: the netblock's mask length
|
||||||
|
- `ip_version`: the netblock's IP version, 4 or 6
|
||||||
|
- `num_peers`: the number of peers that carry the netblock
|
||||||
|
- `asn`: the origin AS that the most peers carry the netblock with
|
||||||
|
- `num_origins`: the number of origin ASes that peers carry the netblock
|
||||||
|
with
|
||||||
|
- `origin_asns`: those origin ASes, in AS number order
|
||||||
|
- `first_seen`: when RouteWatch first saw the netblock, or null when the
|
||||||
|
database has not stored the netblock yet
|
||||||
|
- `last_seen`: the time of the netblock's newest kept route, or null when
|
||||||
|
the netblock keeps no route
|
||||||
|
|
||||||
|
The fields about the AS in `asn` come from the database. Each is also null
|
||||||
|
when the database does not have it:
|
||||||
|
- `handle`: the AS's handle
|
||||||
|
- `description`: the AS's description
|
||||||
|
- `as_name`: the AS's name in its WHOIS details
|
||||||
|
- `org_name`: the name of the organisation that holds the AS
|
||||||
|
- `org_id`: the organisation's ID at its registry
|
||||||
|
- `address`: the organisation's postal address
|
||||||
|
- `country_code`: the organisation's country code
|
||||||
|
- `abuse_email`: the address that takes abuse reports for the AS
|
||||||
|
- `rir`: the regional internet registry of the AS
|
||||||
|
|
||||||
An address with no live route has a result without the route and AS fields.
|
|
||||||
Of the addresses a host name resolves to, only public ones are looked up. A
|
Of the addresses a host name resolves to, only public ones are looked up. A
|
||||||
name with none, such as one that resolves to a private address, is reported
|
name with none, such as one that resolves to a private address, is reported
|
||||||
as `host name has no public address`, without its addresses. `errors` has
|
as `host name has no public address`, without its addresses. `errors` has
|
||||||
@@ -211,10 +235,15 @@ so each answer's policy names a random nonce made for that answer, and the
|
|||||||
browser applies or runs only the blocks that carry it. The pages therefore use
|
browser applies or runs only the blocks that carry it. The pages therefore use
|
||||||
no `style` attributes, which a nonce cannot cover.
|
no `style` attributes, which a nonce cannot cover.
|
||||||
|
|
||||||
An origin AS of 0 (`origin_asn` in a route, `asn` in an IP lookup, the key `0`
|
An origin AS of 0 means the route has no known origin. Its AS path ends in an AS
|
||||||
in a prefix's `origins`) means the route has no known origin: its AS path ends
|
set of several members, any of which may have originated it. The 0 appears in
|
||||||
in an AS set of several members, any of which may have originated it. The pages
|
these fields:
|
||||||
show such a route as having no known origin.
|
|
||||||
|
- `origin_asn` in a route
|
||||||
|
- `asn` and `origin_asns` in an IP lookup
|
||||||
|
- the key `0` in a prefix's `origins`
|
||||||
|
|
||||||
|
The pages show such a route as having no known origin.
|
||||||
|
|
||||||
## Blocklists
|
## Blocklists
|
||||||
|
|
||||||
|
|||||||
@@ -35,6 +35,11 @@ tested.
|
|||||||
|
|
||||||
# Completed Steps
|
# Completed Steps
|
||||||
|
|
||||||
|
- 2026-10-27: every result of the IP lookup has the same fields. A field without
|
||||||
|
a value is null. It used to be left out. An address with no live route has
|
||||||
|
every route and AS field set to null, and `ptr` is an empty list when there
|
||||||
|
are no PTR names. New fields give the number of origin ASes of the netblock,
|
||||||
|
in `num_origins`, and which they are, in `origin_asns` (closes #200)
|
||||||
- 2026-10-27: with `TRUSTED_PROXIES` unset, the server trusts reverse proxies at
|
- 2026-10-27: with `TRUSTED_PROXIES` unset, the server trusts reverse proxies at
|
||||||
the private addresses of RFC 1918. `/ip/`, the home page's `curl` card and the
|
the private addresses of RFC 1918. `/ip/`, the home page's `curl` card and the
|
||||||
request log then give the client's address from `X-Forwarded-For`. They used
|
request log then give the client's address from `X-Forwarded-For`. They used
|
||||||
|
|||||||
@@ -14,6 +14,9 @@ type PrefixSummary struct {
|
|||||||
// OriginASN is the origin AS that the most peers carry the prefix with.
|
// OriginASN is the origin AS that the most peers carry the prefix with.
|
||||||
// When several have as many peers, it is the lowest AS number.
|
// When several have as many peers, it is the lowest AS number.
|
||||||
OriginASN int
|
OriginASN int
|
||||||
|
// OriginASNs are all the origin ASes that peers carry the prefix with, in
|
||||||
|
// AS number order.
|
||||||
|
OriginASNs []int
|
||||||
// Peers is how many peers carry the prefix.
|
// Peers is how many peers carry the prefix.
|
||||||
Peers int
|
Peers int
|
||||||
// Newest is the time of the newest kept route of the prefix, or nil when
|
// Newest is the time of the newest kept route of the prefix, or nil when
|
||||||
@@ -211,10 +214,12 @@ func (t *Table) addresses(peers *bitset.BitSet) []string {
|
|||||||
// summarize returns the summary of prefix, whose origins are in AS number
|
// summarize returns the summary of prefix, whose origins are in AS number
|
||||||
// order.
|
// order.
|
||||||
func summarize(prefix netip.Prefix, origins []*origin) PrefixSummary {
|
func summarize(prefix netip.Prefix, origins []*origin) PrefixSummary {
|
||||||
summary := PrefixSummary{Prefix: prefix}
|
summary := PrefixSummary{Prefix: prefix, OriginASNs: make([]int, 0, len(origins))}
|
||||||
mostPeers := 0
|
mostPeers := 0
|
||||||
|
|
||||||
for _, o := range origins {
|
for _, o := range origins {
|
||||||
|
summary.OriginASNs = append(summary.OriginASNs, o.asn)
|
||||||
|
|
||||||
peers := int(o.peers.Count())
|
peers := int(o.peers.Count())
|
||||||
summary.Peers += peers
|
summary.Peers += peers
|
||||||
|
|
||||||
|
|||||||
@@ -46,21 +46,22 @@ func TestLookup(t *testing.T) {
|
|||||||
tests := []struct {
|
tests := []struct {
|
||||||
addr string
|
addr string
|
||||||
// want is the summary's prefix, or empty when none is found.
|
// want is the summary's prefix, or empty when none is found.
|
||||||
want string
|
want string
|
||||||
originASN int
|
originASN int
|
||||||
peers int
|
originASNs []int
|
||||||
newest int // in seconds after the start; 0 for none
|
peers int
|
||||||
|
newest int // in seconds after the start; 0 for none
|
||||||
}{
|
}{
|
||||||
{"10.1.2.3", prefixV4, asA, 2, 20},
|
{"10.1.2.3", prefixV4, asA, []int{asA, asB}, 2, 20},
|
||||||
{"::ffff:10.1.2.3", prefixV4, asA, 2, 20},
|
{"::ffff:10.1.2.3", prefixV4, asA, []int{asA, asB}, 2, 20},
|
||||||
{"10.1.3.4", "10.1.0.0/16", asB, 3, 30},
|
{"10.1.3.4", "10.1.0.0/16", asB, []int{asA, asB}, 3, 30},
|
||||||
{"10.2.0.1", "10.0.0.0/8", asA, 1, 10},
|
{"10.2.0.1", "10.0.0.0/8", asA, []int{asA}, 1, 10},
|
||||||
{"10.9.0.1", "10.9.0.0/16", asA, 1, 0},
|
{"10.9.0.1", "10.9.0.0/16", asA, []int{asA}, 1, 0},
|
||||||
{"192.0.2.1", "", 0, 0, 0},
|
{"192.0.2.1", "", 0, nil, 0, 0},
|
||||||
{"2001:db8:1::5", prefixV6, asB, 1, 20},
|
{"2001:db8:1::5", prefixV6, asB, []int{asB}, 1, 20},
|
||||||
{"2001:db8:2::1", "2001:db8::/32", asA, 1, 10},
|
{"2001:db8:2::1", "2001:db8::/32", asA, []int{asA}, 1, 10},
|
||||||
{"3000::1", "3000::/16", asA, 1, 10},
|
{"3000::1", "3000::/16", asA, []int{asA}, 1, 10},
|
||||||
{"2001:db9::1", "", 0, 0, 0},
|
{"2001:db9::1", "", 0, nil, 0, 0},
|
||||||
}
|
}
|
||||||
|
|
||||||
for _, tc := range tests {
|
for _, tc := range tests {
|
||||||
@@ -77,9 +78,10 @@ func TestLookup(t *testing.T) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
want := routetable.PrefixSummary{
|
want := routetable.PrefixSummary{
|
||||||
Prefix: netip.MustParsePrefix(tc.want),
|
Prefix: netip.MustParsePrefix(tc.want),
|
||||||
OriginASN: tc.originASN,
|
OriginASN: tc.originASN,
|
||||||
Peers: tc.peers,
|
OriginASNs: tc.originASNs,
|
||||||
|
Peers: tc.peers,
|
||||||
}
|
}
|
||||||
if tc.newest != 0 {
|
if tc.newest != 0 {
|
||||||
want.Newest = timeAt(tc.newest)
|
want.Newest = timeAt(tc.newest)
|
||||||
@@ -166,7 +168,8 @@ func TestSamplePrefixes(t *testing.T) {
|
|||||||
|
|
||||||
for _, got := range sample {
|
for _, got := range sample {
|
||||||
want := routetable.PrefixSummary{
|
want := routetable.PrefixSummary{
|
||||||
Prefix: got.Prefix, OriginASN: asA, Peers: 1, Newest: timeAt(10),
|
Prefix: got.Prefix, OriginASN: asA, OriginASNs: []int{asA},
|
||||||
|
Peers: 1, Newest: timeAt(10),
|
||||||
}
|
}
|
||||||
if seen[got.Prefix] || got.Prefix.Bits() != tc.maskLength ||
|
if seen[got.Prefix] || got.Prefix.Bits() != tc.maskLength ||
|
||||||
!reflect.DeepEqual(got, want) {
|
!reflect.DeepEqual(got, want) {
|
||||||
|
|||||||
+58
-45
@@ -752,15 +752,14 @@ type IPLookupResponse struct {
|
|||||||
}
|
}
|
||||||
|
|
||||||
// IPLookupResult is the answer for one address: its PTR names, its live route
|
// IPLookupResult is the answer for one address: its PTR names, its live route
|
||||||
// and that route's AS, and the blocklists that list it. IPInfo is nil when
|
// and that route's AS, and the blocklists that list it. Every result has every
|
||||||
// the address has no live route, and the answer then has no route or AS
|
// field. The fields of IPInfo are null when the address has no live route.
|
||||||
// fields. IP hides the IPInfo field of the same name, so that an address with
|
|
||||||
// no route has it too.
|
|
||||||
type IPLookupResult struct {
|
type IPLookupResult struct {
|
||||||
*IPInfo
|
IPInfo
|
||||||
|
|
||||||
IP string `json:"ip"`
|
IP string `json:"ip"`
|
||||||
PTR []string `json:"ptr,omitempty"`
|
// PTR is empty, never nil, when the address has no PTR names.
|
||||||
|
PTR []string `json:"ptr"`
|
||||||
|
|
||||||
// Blocklists has one entry for each blocklist the address is on.
|
// Blocklists has one entry for each blocklist the address is on.
|
||||||
Blocklists []blocklist.Listing `json:"blocklists"`
|
Blocklists []blocklist.Listing `json:"blocklists"`
|
||||||
@@ -818,8 +817,8 @@ func (s *Server) handleIPInfo() http.HandlerFunc {
|
|||||||
|
|
||||||
switch {
|
switch {
|
||||||
case errors.Is(err, ErrNoRoute):
|
case errors.Is(err, ErrNoRoute):
|
||||||
// The address still gets a result, without route fields.
|
// The address still gets a result, with every field of
|
||||||
ipInfo = nil
|
// ipInfo null.
|
||||||
case errors.Is(err, ErrInvalidIP):
|
case errors.Is(err, ErrInvalidIP):
|
||||||
response.Errors = append(response.Errors, ip+": "+err.Error())
|
response.Errors = append(response.Errors, ip+": "+err.Error())
|
||||||
|
|
||||||
@@ -846,8 +845,8 @@ func (s *Server) handleIPInfo() http.HandlerFunc {
|
|||||||
Blocklists: s.blocklistListings(ip),
|
Blocklists: s.blocklistListings(ip),
|
||||||
})
|
})
|
||||||
|
|
||||||
if ipInfo != nil && ipInfo.NeedsWHOISRefresh {
|
if ipInfo.NeedsWHOISRefresh {
|
||||||
refreshASNs[ipInfo.ASN] = true
|
refreshASNs[*ipInfo.ASN] = true
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -858,60 +857,72 @@ func (s *Server) handleIPInfo() http.HandlerFunc {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// ipInfo returns the IP lookup's answer for ip. The routing table gives the
|
// ipInfo returns what the IP lookup knows about the route of ip. The routing
|
||||||
// most specific prefix that holds the address, the number of peers that carry
|
// table gives the most specific prefix that holds the address, the number of
|
||||||
// it, the origin AS that most of them carry, and the time of its newest kept
|
// peers that carry it, its origin ASes and the one that most peers carry, and
|
||||||
// route. The database gives the prefix's first-seen time and the AS's WHOIS
|
// the time of its newest kept route. The database gives the prefix's
|
||||||
// details. It returns ErrNoRoute when no prefix holds the address, and
|
// first-seen time and the AS's WHOIS details. It returns ErrNoRoute when no
|
||||||
// ErrInvalidIP when ip does not parse.
|
// prefix holds the address, and ErrInvalidIP when ip does not parse. With an
|
||||||
func (s *Server) ipInfo(ctx context.Context, ip string) (*IPInfo, error) {
|
// error, every field of the IPInfo it returns is nil.
|
||||||
|
func (s *Server) ipInfo(ctx context.Context, ip string) (IPInfo, error) {
|
||||||
addr, err := netip.ParseAddr(ip)
|
addr, err := netip.ParseAddr(ip)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return nil, fmt.Errorf("%w: %s", ErrInvalidIP, ip)
|
return IPInfo{}, fmt.Errorf("%w: %s", ErrInvalidIP, ip)
|
||||||
}
|
}
|
||||||
|
|
||||||
summary, found := s.table.Lookup(addr)
|
summary, found := s.table.Lookup(addr)
|
||||||
if !found {
|
if !found {
|
||||||
return nil, fmt.Errorf("%w for IP %s", ErrNoRoute, ip)
|
return IPInfo{}, fmt.Errorf("%w for IP %s", ErrNoRoute, ip)
|
||||||
}
|
}
|
||||||
|
|
||||||
firstSeen, err := s.db.GetPrefixFirstSeen(ctx, summary.Prefix)
|
firstSeen, err := s.db.GetPrefixFirstSeen(ctx, summary.Prefix)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return nil, err
|
return IPInfo{}, err
|
||||||
}
|
}
|
||||||
|
|
||||||
asInfo, err := s.db.GetASWHOIS(ctx, summary.OriginASN)
|
asInfo, err := s.db.GetASWHOIS(ctx, summary.OriginASN)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return nil, err
|
return IPInfo{}, err
|
||||||
}
|
}
|
||||||
|
|
||||||
info := &IPInfo{
|
info := IPInfo{
|
||||||
IP: ip,
|
Netblock: new(summary.Prefix.String()),
|
||||||
Netblock: summary.Prefix.String(),
|
MaskLength: new(summary.Prefix.Bits()),
|
||||||
MaskLength: summary.Prefix.Bits(),
|
IPVersion: new(ipVersionOf(summary.Prefix)),
|
||||||
IPVersion: ipVersionOf(summary.Prefix),
|
NumPeers: new(summary.Peers),
|
||||||
NumPeers: summary.Peers,
|
ASN: new(summary.OriginASN),
|
||||||
ASN: summary.OriginASN,
|
NumOrigins: new(len(summary.OriginASNs)),
|
||||||
ASName: asInfo.ASName,
|
OriginASNs: summary.OriginASNs,
|
||||||
Handle: asInfo.Handle,
|
ASName: nilIfEmpty(asInfo.ASName),
|
||||||
Description: asInfo.Description,
|
Handle: nilIfEmpty(asInfo.Handle),
|
||||||
OrgName: asInfo.OrgName,
|
Description: nilIfEmpty(asInfo.Description),
|
||||||
OrgID: asInfo.OrgID,
|
OrgName: nilIfEmpty(asInfo.OrgName),
|
||||||
Address: asInfo.Address,
|
OrgID: nilIfEmpty(asInfo.OrgID),
|
||||||
CountryCode: asInfo.CountryCode,
|
Address: nilIfEmpty(asInfo.Address),
|
||||||
AbuseEmail: asInfo.AbuseEmail,
|
CountryCode: nilIfEmpty(asInfo.CountryCode),
|
||||||
RIR: asInfo.RIR,
|
AbuseEmail: nilIfEmpty(asInfo.AbuseEmail),
|
||||||
FirstSeen: firstSeen,
|
RIR: nilIfEmpty(asInfo.RIR),
|
||||||
|
LastSeen: summary.Newest,
|
||||||
NeedsWHOISRefresh: needsWHOISRefresh(asInfo),
|
NeedsWHOISRefresh: needsWHOISRefresh(asInfo),
|
||||||
}
|
}
|
||||||
|
|
||||||
if summary.Newest != nil {
|
if !firstSeen.IsZero() {
|
||||||
info.LastSeen = *summary.Newest
|
info.FirstSeen = &firstSeen
|
||||||
}
|
}
|
||||||
|
|
||||||
return info, nil
|
return info, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// nilIfEmpty returns a pointer to s, or nil when s is empty, so that JSON
|
||||||
|
// gives a detail the database does not hold as null.
|
||||||
|
func nilIfEmpty(s string) *string {
|
||||||
|
if s == "" {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
return &s
|
||||||
|
}
|
||||||
|
|
||||||
// needsWHOISRefresh says whether the WHOIS details of asInfo were never
|
// needsWHOISRefresh says whether the WHOIS details of asInfo were never
|
||||||
// fetched or are older than whoisStaleThreshold. AS 0, no known origin, has
|
// fetched or are older than whoisStaleThreshold. AS 0, no known origin, has
|
||||||
// none to fetch.
|
// none to fetch.
|
||||||
@@ -1117,15 +1128,17 @@ func publicAddresses(addrs []string) []string {
|
|||||||
return public
|
return public
|
||||||
}
|
}
|
||||||
|
|
||||||
// lookupPTRNames returns the PTR records of ip without their trailing dots, or
|
// lookupPTRNames returns the PTR records of ip without their trailing dots. It
|
||||||
// nil when the lookup fails or runs past ptrLookupTimeout.
|
// returns an empty list, never nil, when there are none, when the lookup fails
|
||||||
|
// and when it runs past ptrLookupTimeout, so that an answer's ptr is [] rather
|
||||||
|
// than null.
|
||||||
func (s *Server) lookupPTRNames(ctx context.Context, ip string) []string {
|
func (s *Server) lookupPTRNames(ctx context.Context, ip string) []string {
|
||||||
ctx, cancel := context.WithTimeout(ctx, ptrLookupTimeout)
|
ctx, cancel := context.WithTimeout(ctx, ptrLookupTimeout)
|
||||||
defer cancel()
|
defer cancel()
|
||||||
|
|
||||||
ptrs, err := s.resolver.LookupAddr(ctx, ip)
|
ptrs, err := s.resolver.LookupAddr(ctx, ip)
|
||||||
if err != nil {
|
if err != nil || ptrs == nil {
|
||||||
return nil
|
return []string{}
|
||||||
}
|
}
|
||||||
|
|
||||||
// Remove trailing dots from PTR records
|
// Remove trailing dots from PTR records
|
||||||
|
|||||||
@@ -5,6 +5,7 @@ import (
|
|||||||
"encoding/json"
|
"encoding/json"
|
||||||
"errors"
|
"errors"
|
||||||
"html"
|
"html"
|
||||||
|
"maps"
|
||||||
"net"
|
"net"
|
||||||
"net/http"
|
"net/http"
|
||||||
"net/http/httptest"
|
"net/http/httptest"
|
||||||
@@ -1219,7 +1220,8 @@ func TestIPLookupKeepsRouteWhenPTRLookupDoesNotAnswer(t *testing.T) {
|
|||||||
t.Fatalf("decoding the answer %q: %v", rec.Body, err)
|
t.Fatalf("decoding the answer %q: %v", rec.Body, err)
|
||||||
}
|
}
|
||||||
|
|
||||||
if len(body.Data.Results) != 1 || body.Data.Results[0].Netblock != testPrefixV4 {
|
if len(body.Data.Results) != 1 || body.Data.Results[0].Netblock == nil ||
|
||||||
|
*body.Data.Results[0].Netblock != testPrefixV4 {
|
||||||
t.Errorf("answer %s, want one result in %s", rec.Body, testPrefixV4)
|
t.Errorf("answer %s, want one result in %s", rec.Body, testPrefixV4)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -1311,12 +1313,39 @@ func TestIPLookupReportsBlocklists(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// jsonNull is how JSON writes a field without a value.
|
||||||
|
const jsonNull = "null"
|
||||||
|
|
||||||
|
// jsonEmptyList is how JSON writes an empty list.
|
||||||
|
const jsonEmptyList = "[]"
|
||||||
|
|
||||||
|
// resultFields decodes an IP lookup answer and returns each of its results as
|
||||||
|
// its JSON fields.
|
||||||
|
func resultFields(t *testing.T, answer []byte) []map[string]json.RawMessage {
|
||||||
|
t.Helper()
|
||||||
|
|
||||||
|
var body struct {
|
||||||
|
Data struct {
|
||||||
|
Results []map[string]json.RawMessage `json:"results"`
|
||||||
|
} `json:"data"`
|
||||||
|
}
|
||||||
|
|
||||||
|
err := json.Unmarshal(answer, &body)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("decoding the answer %s: %v", answer, err)
|
||||||
|
}
|
||||||
|
|
||||||
|
return body.Data.Results
|
||||||
|
}
|
||||||
|
|
||||||
// TestIPLookupReportsBlocklistsOfAnAddressWithNoRoute looks up a listed
|
// TestIPLookupReportsBlocklistsOfAnAddressWithNoRoute looks up a listed
|
||||||
// address that has no live route, alone and beside a routed address. Each
|
// address that has no live route, alone and beside a routed address. Each
|
||||||
// answer is 200 and has a result for the address, with its blocklists and no
|
// answer is 200 and has a result for the address, with its blocklists and
|
||||||
// route fields. Before the fix the lookup of that address alone answered
|
// every route field set to null. Before the fix the lookup of that address
|
||||||
// 404, and beside a routed address it was only named in errors, so its
|
// alone answered 404, and beside a routed address it was only named in
|
||||||
// blocklists were never reported.
|
// errors, so its blocklists were never reported. After that fix the result
|
||||||
|
// still left out the route fields
|
||||||
|
// (https://git.eeqj.de/sneak/routewatch/issues/200).
|
||||||
func TestIPLookupReportsBlocklistsOfAnAddressWithNoRoute(t *testing.T) {
|
func TestIPLookupReportsBlocklistsOfAnAddressWithNoRoute(t *testing.T) {
|
||||||
t.Parallel()
|
t.Parallel()
|
||||||
|
|
||||||
@@ -1324,6 +1353,15 @@ func TestIPLookupReportsBlocklistsOfAnAddressWithNoRoute(t *testing.T) {
|
|||||||
s.resolver = &fakeResolver{}
|
s.resolver = &fakeResolver{}
|
||||||
s.SetBlocklists(fakeBlocklists{})
|
s.SetBlocklists(fakeBlocklists{})
|
||||||
|
|
||||||
|
// routeFields are the fields of a result that come from the address's
|
||||||
|
// route and that route's origin AS.
|
||||||
|
routeFields := []string{
|
||||||
|
"netblock", "mask_length", "ip_version", "num_peers", "asn",
|
||||||
|
"num_origins", "origin_asns", "as_name", "handle", "description",
|
||||||
|
"org_name", "org_id", "address", "country_code", "abuse_email", "rir",
|
||||||
|
"first_seen", "last_seen",
|
||||||
|
}
|
||||||
|
|
||||||
for _, target := range []string{
|
for _, target := range []string{
|
||||||
listedUnroutedAddr,
|
listedUnroutedAddr,
|
||||||
addrInTestPrefixV4 + "," + listedUnroutedAddr,
|
addrInTestPrefixV4 + "," + listedUnroutedAddr,
|
||||||
@@ -1354,12 +1392,21 @@ func TestIPLookupReportsBlocklistsOfAnAddressWithNoRoute(t *testing.T) {
|
|||||||
continue
|
continue
|
||||||
}
|
}
|
||||||
|
|
||||||
result := body.Data.Results[i]
|
fields := resultFields(t, rec.Body.Bytes())[i]
|
||||||
if result.IPInfo != nil {
|
for _, name := range routeFields {
|
||||||
t.Errorf("%s: result for %s has route fields %+v",
|
value, present := fields[name]
|
||||||
target, listedUnroutedAddr, result.IPInfo)
|
|
||||||
|
switch {
|
||||||
|
case !present:
|
||||||
|
t.Errorf("%s: the result for %s has no %s",
|
||||||
|
target, listedUnroutedAddr, name)
|
||||||
|
case string(value) != jsonNull:
|
||||||
|
t.Errorf("%s: %s of %s is %s, want %s",
|
||||||
|
target, name, listedUnroutedAddr, value, jsonNull)
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
result := body.Data.Results[i]
|
||||||
if len(result.Blocklists) != 1 ||
|
if len(result.Blocklists) != 1 ||
|
||||||
result.Blocklists[0].Name != testListing().Name {
|
result.Blocklists[0].Name != testListing().Name {
|
||||||
t.Errorf("%s: blocklists %+v, want [%+v]",
|
t.Errorf("%s: blocklists %+v, want [%+v]",
|
||||||
@@ -1368,6 +1415,76 @@ func TestIPLookupReportsBlocklistsOfAnAddressWithNoRoute(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// twoAddressHostName is a host name that resolves to addrInTestPrefixV4 and
|
||||||
|
// listedUnroutedAddr.
|
||||||
|
const twoAddressHostName = "two-addresses.test"
|
||||||
|
|
||||||
|
// TestIPLookupResultsHaveTheSameFields looks up a host name, addresses and
|
||||||
|
// the caller's own address, through each IP lookup route. Some of the
|
||||||
|
// addresses have a route and some do not, and none has PTR names. Every
|
||||||
|
// result has the same fields. Before the fix a result left out every field
|
||||||
|
// without a value (https://git.eeqj.de/sneak/routewatch/issues/200).
|
||||||
|
func TestIPLookupResultsHaveTheSameFields(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
s := newServerWithTestRoutes(t)
|
||||||
|
s.resolver = &fakeResolver{hosts: map[string][]string{
|
||||||
|
twoAddressHostName: {addrInTestPrefixV4, listedUnroutedAddr},
|
||||||
|
}}
|
||||||
|
s.SetBlocklists(fakeBlocklists{})
|
||||||
|
|
||||||
|
// want is the fields of the first result.
|
||||||
|
var want []string
|
||||||
|
|
||||||
|
for _, path := range []string{
|
||||||
|
"/api/v1/ip/" + twoAddressHostName,
|
||||||
|
"/ip/" + addrInTestPrefixV6 + "," + listedUnroutedAddr,
|
||||||
|
ownAddressPath,
|
||||||
|
} {
|
||||||
|
for _, result := range resultFields(t, get(t, s, path).Body.Bytes()) {
|
||||||
|
got := slices.Sorted(maps.Keys(result))
|
||||||
|
if want == nil {
|
||||||
|
want = got
|
||||||
|
}
|
||||||
|
|
||||||
|
if !slices.Equal(got, want) {
|
||||||
|
t.Errorf("%s: a result has the fields %v, want %v", path, got, want)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestIPLookupOfAnAddressWithoutPTRNamesOrStoredNetblock looks up a routed
|
||||||
|
// address that has no PTR names, and whose netblock the database has not
|
||||||
|
// stored. Its ptr is an empty list and its first_seen is null. Before the fix
|
||||||
|
// ptr was left out, and first_seen was 0001-01-01T00:00:00Z
|
||||||
|
// (https://git.eeqj.de/sneak/routewatch/issues/200).
|
||||||
|
func TestIPLookupOfAnAddressWithoutPTRNamesOrStoredNetblock(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
s := newServerWithTestRoutes(t)
|
||||||
|
s.resolver = &fakeResolver{}
|
||||||
|
|
||||||
|
path := "/api/v1/ip/" + addrInTestPrefixV4
|
||||||
|
|
||||||
|
results := resultFields(t, get(t, s, path).Body.Bytes())
|
||||||
|
if len(results) != 1 {
|
||||||
|
t.Fatalf("%s: %d results, want 1", path, len(results))
|
||||||
|
}
|
||||||
|
|
||||||
|
want := map[string]string{
|
||||||
|
"ptr": jsonEmptyList,
|
||||||
|
"first_seen": jsonNull,
|
||||||
|
}
|
||||||
|
|
||||||
|
for name, value := range want {
|
||||||
|
got := string(results[0][name])
|
||||||
|
if got != value {
|
||||||
|
t.Errorf("%s: %s is %s, want %s", path, name, got, value)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
// TestIPLookupAnswersMayBeCached checks the Cache-Control header of the IP
|
// TestIPLookupAnswersMayBeCached checks the Cache-Control header of the IP
|
||||||
// lookup answers. Caches may keep them for the blocklists' shortest interval,
|
// lookup answers. Caches may keep them for the blocklists' shortest interval,
|
||||||
// and only the caller's own browser may keep the answer about the caller's
|
// and only the caller's own browser may keep the answer about the caller's
|
||||||
|
|||||||
@@ -4,6 +4,8 @@ import (
|
|||||||
"encoding/json"
|
"encoding/json"
|
||||||
"net/http"
|
"net/http"
|
||||||
"net/http/httptest"
|
"net/http/httptest"
|
||||||
|
"net/netip"
|
||||||
|
"reflect"
|
||||||
"slices"
|
"slices"
|
||||||
"strconv"
|
"strconv"
|
||||||
"strings"
|
"strings"
|
||||||
@@ -75,9 +77,10 @@ func get(t *testing.T, s *Server, path string) *httptest.ResponseRecorder {
|
|||||||
|
|
||||||
// TestIPLookupGivesTheOriginMostPeersCarry looks up an address in a prefix
|
// TestIPLookupGivesTheOriginMostPeersCarry looks up an address in a prefix
|
||||||
// with two origin ASes, through each IP lookup route. The answer names the
|
// with two origin ASes, through each IP lookup route. The answer names the
|
||||||
// origin AS that more peers carry, counts every peer, and gives the time of
|
// origin AS that more peers carry, counts every peer, lists both origin ASes,
|
||||||
// the prefix's newest kept route. The prefix's first-seen time and the AS's
|
// and gives the time of the prefix's newest kept route. The prefix's
|
||||||
// WHOIS details come from the database.
|
// first-seen time and the AS's WHOIS details come from the database. A WHOIS
|
||||||
|
// detail the database does not hold is null.
|
||||||
func TestIPLookupGivesTheOriginMostPeersCarry(t *testing.T) {
|
func TestIPLookupGivesTheOriginMostPeersCarry(t *testing.T) {
|
||||||
t.Parallel()
|
t.Parallel()
|
||||||
|
|
||||||
@@ -105,6 +108,26 @@ func TestIPLookupGivesTheOriginMostPeersCarry(t *testing.T) {
|
|||||||
t.Fatalf("UpdateASNWHOIS: %v", err)
|
t.Fatalf("UpdateASNWHOIS: %v", err)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// The details the database does not hold, such as org_name, are null.
|
||||||
|
want := IPInfo{
|
||||||
|
Netblock: new(testPrefixV4),
|
||||||
|
MaskLength: new(netip.MustParsePrefix(testPrefixV4).Bits()),
|
||||||
|
IPVersion: new(ipVersionV4),
|
||||||
|
NumPeers: new(4),
|
||||||
|
ASN: new(mostPeersASN),
|
||||||
|
NumOrigins: new(2),
|
||||||
|
OriginASNs: []int{mostPeersASN, fewerPeersASN},
|
||||||
|
ASName: &whois.ASName,
|
||||||
|
CountryCode: &whois.CountryCode,
|
||||||
|
FirstSeen: &firstSeen,
|
||||||
|
LastSeen: new(testRouteTime(5)),
|
||||||
|
}
|
||||||
|
|
||||||
|
wantJSON, err := json.Marshal(want)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("encoding %+v: %v", want, err)
|
||||||
|
}
|
||||||
|
|
||||||
// ownAddressPath looks up the request's own address, which httptest
|
// ownAddressPath looks up the request's own address, which httptest
|
||||||
// gives as addrInTestPrefixV4.
|
// gives as addrInTestPrefixV4.
|
||||||
for _, path := range []string{
|
for _, path := range []string{
|
||||||
@@ -115,22 +138,15 @@ func TestIPLookupGivesTheOriginMostPeersCarry(t *testing.T) {
|
|||||||
Data IPLookupResponse `json:"data"`
|
Data IPLookupResponse `json:"data"`
|
||||||
}
|
}
|
||||||
|
|
||||||
err := json.Unmarshal(get(t, s, path).Body.Bytes(), &body)
|
answer := get(t, s, path).Body.Bytes()
|
||||||
if err != nil || len(body.Data.Results) != 1 ||
|
|
||||||
body.Data.Results[0].IPInfo == nil {
|
err := json.Unmarshal(answer, &body)
|
||||||
t.Fatalf("%s: answer %+v (%v), want one result with a route",
|
if err != nil || len(body.Data.Results) != 1 {
|
||||||
path, body.Data, err)
|
t.Fatalf("%s: answer %s (%v), want one result", path, answer, err)
|
||||||
}
|
}
|
||||||
|
|
||||||
got := body.Data.Results[0].IPInfo
|
if !reflect.DeepEqual(body.Data.Results[0].IPInfo, want) {
|
||||||
if got.Netblock != testPrefixV4 || got.ASN != mostPeersASN ||
|
t.Errorf("%s: answer %s, want the fields %s", path, answer, wantJSON)
|
||||||
got.NumPeers != 4 || !got.LastSeen.Equal(testRouteTime(5)) ||
|
|
||||||
!got.FirstSeen.Equal(firstSeen) || got.ASName != whois.ASName ||
|
|
||||||
got.CountryCode != whois.CountryCode {
|
|
||||||
t.Errorf("%s: route %+v, want %s from AS%d, 4 peers, last seen %v, "+
|
|
||||||
"first seen %v, %s in %s", path, *got, testPrefixV4,
|
|
||||||
mostPeersASN, testRouteTime(5), firstSeen, whois.ASName,
|
|
||||||
whois.CountryCode)
|
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
+28
-21
@@ -38,30 +38,37 @@ type PrefixDistribution struct {
|
|||||||
Count int `json:"count"`
|
Count int `json:"count"`
|
||||||
}
|
}
|
||||||
|
|
||||||
// IPInfo represents comprehensive IP information for the /ip endpoint
|
// IPInfo is what the IP lookup knows about the route of an address and that
|
||||||
|
// route's origin AS. A field is nil, null in JSON, when its value is not known.
|
||||||
|
// Every field is nil for an address with no route.
|
||||||
//
|
//
|
||||||
//nolint:tagliatelle // field names are the existing JSON API
|
//nolint:tagliatelle // field names are the existing JSON API
|
||||||
type IPInfo struct {
|
type IPInfo struct {
|
||||||
IP string `json:"ip"`
|
// Netblock is the most specific prefix that holds the address.
|
||||||
Netblock string `json:"netblock"`
|
Netblock *string `json:"netblock"`
|
||||||
MaskLength int `json:"mask_length"`
|
MaskLength *int `json:"mask_length"`
|
||||||
IPVersion int `json:"ip_version"`
|
IPVersion *int `json:"ip_version"`
|
||||||
NumPeers int `json:"num_peers"`
|
NumPeers *int `json:"num_peers"`
|
||||||
// AS information
|
// ASN is the origin AS that the most peers carry the netblock with.
|
||||||
ASN int `json:"asn"`
|
ASN *int `json:"asn"`
|
||||||
ASName string `json:"as_name,omitempty"`
|
// NumOrigins and OriginASNs are how many origin ASes peers carry the
|
||||||
Handle string `json:"handle,omitempty"`
|
// netblock with, and which, in AS number order.
|
||||||
Description string `json:"description,omitempty"`
|
NumOrigins *int `json:"num_origins"`
|
||||||
OrgName string `json:"org_name,omitempty"`
|
OriginASNs []int `json:"origin_asns"`
|
||||||
OrgID string `json:"org_id,omitempty"`
|
// The details the database holds about the AS in ASN.
|
||||||
Address string `json:"address,omitempty"`
|
ASName *string `json:"as_name"`
|
||||||
CountryCode string `json:"country_code,omitempty"`
|
Handle *string `json:"handle"`
|
||||||
AbuseEmail string `json:"abuse_email,omitempty"`
|
Description *string `json:"description"`
|
||||||
RIR string `json:"rir,omitempty"`
|
OrgName *string `json:"org_name"`
|
||||||
// Timestamps. LastSeen is zero, and left out of JSON, when the prefix
|
OrgID *string `json:"org_id"`
|
||||||
// keeps no route.
|
Address *string `json:"address"`
|
||||||
FirstSeen time.Time `json:"first_seen"`
|
CountryCode *string `json:"country_code"`
|
||||||
LastSeen time.Time `json:"last_seen,omitzero"`
|
AbuseEmail *string `json:"abuse_email"`
|
||||||
|
RIR *string `json:"rir"`
|
||||||
|
// FirstSeen is when RouteWatch first saw the netblock. LastSeen is the
|
||||||
|
// time of the netblock's newest kept route.
|
||||||
|
FirstSeen *time.Time `json:"first_seen"`
|
||||||
|
LastSeen *time.Time `json:"last_seen"`
|
||||||
// Indicates if WHOIS data needs refresh (not serialized)
|
// Indicates if WHOIS data needs refresh (not serialized)
|
||||||
NeedsWHOISRefresh bool `json:"-"`
|
NeedsWHOISRefresh bool `json:"-"`
|
||||||
}
|
}
|
||||||
|
|||||||
Reference in New Issue
Block a user