diff --git a/README.md b/README.md index c907cfa..3eb652b 100644 --- a/README.md +++ b/README.md @@ -12,16 +12,16 @@ table in real time. It works like this: RIPE RIS Live streams the BGP updates that RIPE's route collectors receive, as they happen, from the moment a client connects. It reports changes, not the routing table they add up to, so it cannot say which AS an IP address is routed -to now or which routes a prefix has. RouteWatch applies the feed's updates to -live routing tables in its own SQLite database and answers IP-to-AS lookups and -prefix pages from them. +to now or which routes a prefix has. RouteWatch applies the feed's updates to a +routing table that it keeps in memory, with a copy in its own SQLite database. +It answers IP-to-AS lookups and prefix pages from that table. -The feed sends no table when a client connects, so these tables hold only the -routes the feed has announced since RouteWatch first started: an IP-to-AS lookup -finds nothing for an address whose routes have not been announced since then. -The tables also miss the updates RouteWatch drops when it falls behind under -load and those sent while it is stopped, so a lookup or a prefix page can show a -route the feed has withdrawn. +The feed sends no table when a client connects. So the routing table holds only +the routes announced since RouteWatch first started. An IP-to-AS lookup finds +nothing for an address whose routes have not been announced since then. The +routing table also misses updates RouteWatch drops when it falls behind, and +updates sent while it is stopped. So a lookup or a prefix page can show a route +the feed has withdrawn. ## Features @@ -58,13 +58,24 @@ DEBUG=routewatch ./routewatch Every route answers `HEAD` as it answers `GET`, without the body. +The pages and the API read routes from the routing table that RouteWatch keeps +in memory. For each prefix and origin AS, the table keeps the two newest routes +in full, with their AS paths and times. These are the kept routes. The table +keeps only the address of every other peer that carries the prefix. When both +kept routes of an origin AS are withdrawn, that origin has no kept route until +one of its peers sends an update. + ### Web Interface - `GET /` - Home page (HTML): headline figures, lookups by AS number and by IP address, and a search of AS names. A card shows the `curl` command that looks up the visitor's own address at `/ip/{addr}`, and that command's live answer. - `GET /status` - HTML status dashboard, which reads `/api/v1/stats` -- `GET /status.json` - JSON statistics: feed, memory, database and route figures +- `GET /status.json` - JSON statistics: feed, memory, database and route + figures. The route figures come from the routing table. `oldest_route` and + `newest_route` are the times of the oldest and the newest kept routes. + `origins_without_kept_route` counts, across all prefixes, the origin ASes that + have no kept route. - `GET /.well-known/healthcheck.json` - Health check (JSON): 200 when the database answers and the RIS Live feed is connected, otherwise 503; `data.status` is `ok` or `error` and `data.checks` says which check failed @@ -73,10 +84,11 @@ Every route answers `HEAD` as it answers `GET`, without the body. description come from the AS list built into RouteWatch. The page then shows no first-seen date. - `GET /prefix/{prefix}/{length}` - Prefix detail page (HTML), such as - `/prefix/192.0.2.0/24`: every live route for the prefix, one per peer, and its - origin ASes + `/prefix/192.0.2.0/24`: its origin ASes and every peer that carries it. Only + the kept routes show an AS path, a next hop and a time. - `GET /prefixlength/{length}` - Up to 500 IPv4 prefixes with that mask length, - picked at random, each with its newest route (HTML) + picked at random (HTML). Each shows the origin AS that most peers carry it + with, and the age of its newest kept route. - `GET /prefixlength6/{length}` - The same for IPv6 - `GET /ip/{addr}` - The same JSON answer as `/api/v1/ip/{ip}` - `GET /ip/` - The same, for the caller's own address: the address the request @@ -93,8 +105,10 @@ Every route answers `HEAD` as it answers `GET`, without the body. name, or a comma-separated list of at most 10 of these. `results` has one entry for each address, with: - `ip` and `ptr`: the address and its PTR names - - `netblock` and the fields beside it: the address's most specific live - route, and that route's origin AS with its WHOIS details + - `netblock` and the fields beside it: the most specific prefix that holds + the address, and its origin AS with that AS's WHOIS details. For a prefix + with several origin ASes, `asn` is the one that most peers carry. + `last_seen` is the time of the prefix's newest kept route. - `blocklists`: the public blocklists the address is on (see Blocklists) An address with no live route has a result without the route and AS fields. @@ -110,11 +124,13 @@ Every route answers `HEAD` as it answers `GET`, without the body. answered without PTR names. - `GET /api/v1/as/{asn}` - The AS and the prefixes it originates, IPv4 and IPv6 - apart, each with when its newest route was updated. When the database has no - record of the AS, its handle and description come from the AS list built into + apart, each with the time of its newest kept route. A prefix that keeps no + route with the AS has no `last_updated`. When the database has no record of + the AS, its handle and description come from the AS list built into RouteWatch. The answer then has no `first_seen` or `last_seen`. -- `GET /api/v1/prefix/{prefix}/{length}` - Every live route for the prefix, one - per peer, and the same routes grouped by origin AS in `origins` +- `GET /api/v1/prefix/{prefix}/{length}` - One route for each peer that carries + the prefix, and the same routes grouped by origin AS in `origins`. Only the + kept routes have `as_path`, `next_hop` and `last_updated`. - `GET /api/v1/search/as?q={text}` - Finds the ASes whose handle or description contains `{text}`, ignoring case. `data` holds: - `query`: `{text}` without its leading and trailing spaces @@ -137,16 +153,16 @@ took. The health check is the exception: its 503 answer is also A request that runs past its time limit answers 504: the limit is 30 seconds, and 4 seconds for the database reads behind `/status.json` and `/api/v1/stats`. An AS number, prefix or mask length that does not parse answers 400, as does a -mask length out of range for its IP version. An AS answers 404 when it is not in -the AS list built into RouteWatch and the database has neither a record of it -nor a live route it originates. AS 0 always answers 404, because it means no -known origin. The AS name search never lists it. A prefix with no live route -answers 404 too. The HTML pages give these errors as plain text. Under `/api/v1` -and `/ip/`, a path no route matches answers 404 with the message `Not found`, -and `POST`, `PUT`, `PATCH`, `DELETE`, `OPTIONS`, `TRACE` or `CONNECT` answers -405 with `Method not allowed`. A 500 answer says only `Internal server error`, -and a host name that does not resolve only `could not resolve host name`; the -detail goes to the log. +mask length out of range for its IP version. An AS answers 404 when it +originates no prefix and has no record in the database or in the AS list built +into RouteWatch. AS 0 always answers 404, because it means no known origin. The +AS name search never lists it. A prefix with no live route answers 404 too. The +HTML pages give these errors as plain text. Under `/api/v1` and `/ip/`, a path +no route matches answers 404 with the message `Not found`, and `POST`, `PUT`, +`PATCH`, `DELETE`, `OPTIONS`, `TRACE` or `CONNECT` answers 405 with +`Method not allowed`. A 500 answer says only `Internal server error`, and a host +name that does not resolve only `could not resolve host name`; the detail goes +to the log. Every answer carries `Strict-Transport-Security`, `Content-Security-Policy`, `X-Frame-Options`, `X-Content-Type-Options`, `Referrer-Policy` and @@ -340,11 +356,10 @@ routewatch/ with exponential backoff (5s-320s). The backoff resets after a stream that lasted more than 30s. A stream with no line for 60s is dropped and reconnected. -- **IP Lookup**: the most specific live route is found by looking up the - address's prefix at each mask length, longest first, in the prefix index (at - most 25 lookups for IPv4, 113 for IPv6); prefixes are stored in the text form - Go's `net/netip` prints. Routes shorter than /8 for IPv4 or /16 for IPv6, such - as a default route, are not used. +- **IP Lookup**: the routing table in memory finds the most specific prefix that + holds the address. It looks up the address's prefix at each mask length, + longest first: at most 25 lookups for IPv4 and 113 for IPv6. Prefixes shorter + than /8 for IPv4 or /16 for IPv6, such as a default route, are not used. ### Database Schema diff --git a/TODO.md b/TODO.md index f0538d9..2a3e50a 100644 --- a/TODO.md +++ b/TODO.md @@ -33,6 +33,12 @@ tested. # Completed Steps +- 2026-10-27: the pages, the API and the stats read every route from the routing + table in memory. For a prefix with several origin ASes, the IP lookup gives + the one most peers carry. Its `last_seen` is the newest kept route's time. The + prefix page and API list every peer that carries the prefix, with an AS path, + next hop and time only for the kept routes. The stats count the origins whose + kept routes were all withdrawn, in `origins_without_kept_route` (part of #122) - 2026-10-27: the prefix, AS and peer handlers make a timed write on every tick of their timers, every 1, 2 and 2 seconds. They used to skip every other tick. A full batch's write restarts the timer, so the next timed write comes one @@ -271,6 +277,5 @@ tested. # Future Steps -- the rest of https://git.eeqj.de/sneak/routewatch/issues/122, one PR each: the - pages and the API reading from the routing table, then removing the per-peer - route tables +- the rest of https://git.eeqj.de/sneak/routewatch/issues/122: removing the + per-peer route tables diff --git a/internal/database/counts_internal_test.go b/internal/database/counts_internal_test.go index 3fcd7b5..805830d 100644 --- a/internal/database/counts_internal_test.go +++ b/internal/database/counts_internal_test.go @@ -657,60 +657,6 @@ func checkOneLiveRoutePerIPVersion(t *testing.T, db *Database) { assertDistribution(t, "IPv6 distribution query", queryV6, wantV6) } -// TestStatsRouteTimestamps checks the oldest/newest route timestamps are read -// from the right rows across both tables and parse into time.Time. The old -// MIN/MAX union query read its result into *time.Time, which the driver could -// not parse, so it logged a warning every call and left both timestamps nil. -func TestStatsRouteTimestamps(t *testing.T) { - t.Parallel() - - cfg := &config.Config{StateDir: t.TempDir()} - - db, err := New(cfg, logger.New()) - if err != nil { - t.Fatalf("failed to create database: %v", err) - } - defer func() { _ = db.Close() }() - - ctx := context.Background() - - empty, err := db.GetStatsContext(ctx) - if err != nil { - t.Fatalf("GetStatsContext on empty database: %v", err) - } - - if empty.OldestRoute != nil || empty.NewestRoute != nil { - t.Fatalf("empty database timestamps = (%v, %v), want (nil, nil)", - empty.OldestRoute, empty.NewestRoute) - } - - base := time.Date(2026, 1, 2, 3, 4, 5, 0, time.UTC) - oldest := base - newest := base.Add(2 * time.Minute) - - err = db.UpsertLiveRouteBatch([]*LiveRoute{ - mkV4Route(t, "198.51.100.0/24", 64500, base.Add(time.Minute)), - mkV4Route(t, "203.0.113.0/24", 64501, oldest), - mkV6Route(testV6Prefix, 64502, newest), - }) - if err != nil { - t.Fatalf("UpsertLiveRouteBatch: %v", err) - } - - stats, err := db.GetStatsContext(ctx) - if err != nil { - t.Fatalf("GetStatsContext: %v", err) - } - - if stats.OldestRoute == nil || !stats.OldestRoute.Equal(oldest) { - t.Errorf("OldestRoute = %v, want %v", stats.OldestRoute, oldest) - } - - if stats.NewestRoute == nil || !stats.NewestRoute.Equal(newest) { - t.Errorf("NewestRoute = %v, want %v", stats.NewestRoute, newest) - } -} - // TestLiveCountsConcurrentReadWrite runs writers and stats readers at once so // the race detector proves the counters are safe under concurrent use. func TestLiveCountsConcurrentReadWrite(t *testing.T) { diff --git a/internal/database/database.go b/internal/database/database.go index 8a68944..5bb7a41 100644 --- a/internal/database/database.go +++ b/internal/database/database.go @@ -1452,10 +1452,9 @@ func (d *Database) GetStats() (Stats, error) { // // The row counts (ASNs, prefixes, peerings, peers, live routes) and the prefix // distribution come from the in-memory counters, seeded at startup and kept -// current on every write. The oldest/newest route timestamps are read from the -// ends of the last_updated index, and the file size from a stat(). No part of -// the read scans a table or a whole index. -func (d *Database) GetStatsContext(ctx context.Context) (Stats, error) { +// current on every write, and the file size from a stat(). The read touches +// no table. +func (d *Database) GetStatsContext(_ context.Context) (Stats, error) { var stats Stats // Row counts and prefix distribution from memory, as a single consistent @@ -1470,20 +1469,6 @@ func (d *Database) GetStatsContext(ctx context.Context) (Stats, error) { stats.FileSizeBytes = fileInfo.Size() } - // Oldest and newest route timestamps read one row from each end of the - // last_updated index (a log-time lookup, not a scan). Selecting the column - // directly lets the driver parse the DATETIME into time.Time; the old - // MIN/MAX union scan returned an untyped string that failed to scan and - // logged a warning on every call. - oldest, newest, err := d.routeTimestampRange(ctx) - if err != nil { - // Display-only fields; log but keep the rest of the stats. - d.logger.Warn("Failed to get route timestamps", "error", err) - } else { - stats.OldestRoute = oldest - stats.NewestRoute = newest - } - return stats, nil } @@ -1778,7 +1763,35 @@ func (d *Database) GetASDetailsContext( return nil, nil, fmt.Errorf("%w: AS%d", ErrNoRoute, asn) } - // Get AS information + asnInfo, known, err := d.GetAS(ctx, asn) + if err != nil { + return nil, nil, err + } + + allPrefixes, err := d.originPrefixes(ctx, asn, ipVersionV4) + if err != nil { + return asnInfo, nil, err + } + + prefixesV6, err := d.originPrefixes(ctx, asn, ipVersionV6) + if err != nil { + return asnInfo, allPrefixes, err + } + + allPrefixes = append(allPrefixes, prefixesV6...) + + if !known && len(allPrefixes) == 0 { + return nil, nil, fmt.Errorf("%w: AS%d", ErrNoRoute, asn) + } + + return asnInfo, allPrefixes, nil +} + +// GetAS returns the AS's handle, description and first and last seen times +// from the asns table. When the table has no record of the AS, its handle and +// description come from the AS info lookup, and it has no seen times. The bool +// says whether the asns table or the AS info lookup holds the AS. +func (d *Database) GetAS(ctx context.Context, asn int) (*ASN, bool, error) { var ( asnInfo ASN handle, description sql.NullString @@ -1791,9 +1804,6 @@ func (d *Database) GetASDetailsContext( ).Scan(&asnInfo.ASN, &handle, &description, &asnInfo.FirstSeen, &asnInfo.LastSeen) - // known is whether the asns table or the AS info lookup holds the AS. - known := true - switch { case errors.Is(err, sql.ErrNoRows): asnInfo.ASN = asn @@ -1804,31 +1814,96 @@ func (d *Database) GetASDetailsContext( asnInfo.Description = info.Description } - known = ok + return &asnInfo, ok, nil case err != nil: - return nil, nil, fmt.Errorf("failed to query AS: %w", err) - default: - asnInfo.Handle = handle.String - asnInfo.Description = description.String + return nil, false, fmt.Errorf("failed to query AS: %w", err) + } + + asnInfo.Handle = handle.String + asnInfo.Description = description.String + + return &asnInfo, true, nil +} + +// asWHOISQuery reads the handle, description and WHOIS details of one AS, and +// when the WHOIS details were fetched. +const asWHOISQuery = ` + SELECT handle, description, as_name, org_name, org_id, address, + country_code, abuse_email, rir, whois_updated_at + FROM asns WHERE asn = ?` + +// GetASWHOIS returns the handle, description and WHOIS details that the asns +// table holds for the AS, and when the WHOIS details were fetched. When the +// table has no record of the AS, it returns an ASN with only its number. +func (d *Database) GetASWHOIS(ctx context.Context, asn int) (*ASN, error) { + var ( + handle, description, asName, orgName, orgID sql.NullString + address, countryCode, abuseEmail, rir sql.NullString + whoisUpdatedAt sql.NullTime + ) + + err := d.db.QueryRowContext(ctx, asWHOISQuery, asn).Scan( + &handle, &description, &asName, &orgName, &orgID, &address, + &countryCode, &abuseEmail, &rir, &whoisUpdatedAt, + ) + if errors.Is(err, sql.ErrNoRows) { + return &ASN{ASN: asn}, nil } - allPrefixes, err := d.originPrefixes(ctx, asn, ipVersionV4) if err != nil { - return &asnInfo, nil, err + return nil, fmt.Errorf("failed to query AS: %w", err) + } + + info := &ASN{ + ASN: asn, + Handle: handle.String, + Description: description.String, + ASName: asName.String, + OrgName: orgName.String, + OrgID: orgID.String, + Address: address.String, + CountryCode: countryCode.String, + AbuseEmail: abuseEmail.String, + RIR: rir.String, + } + + if whoisUpdatedAt.Valid { + info.WHOISUpdatedAt = &whoisUpdatedAt.Time + } + + return info, nil +} + +// The queries GetPrefixFirstSeen reads an IPv4 and an IPv6 prefix's first-seen +// time with. +const ( + prefixFirstSeenV4SQL = `SELECT first_seen FROM prefixes_v4 WHERE prefix = ?` + prefixFirstSeenV6SQL = `SELECT first_seen FROM prefixes_v6 WHERE prefix = ?` +) + +// GetPrefixFirstSeen returns when the prefix tables first saw prefix, or the +// zero time when they have no record of it. +func (d *Database) GetPrefixFirstSeen( + ctx context.Context, prefix netip.Prefix, +) (time.Time, error) { + query := prefixFirstSeenV6SQL + if prefix.Addr().Is4() { + query = prefixFirstSeenV4SQL + } + + var firstSeen time.Time + + err := d.db.QueryRowContext(ctx, query, prefix.Masked().String()). + Scan(&firstSeen) + if errors.Is(err, sql.ErrNoRows) { + return time.Time{}, nil } - prefixesV6, err := d.originPrefixes(ctx, asn, ipVersionV6) if err != nil { - return &asnInfo, allPrefixes, err + return time.Time{}, fmt.Errorf("failed to query prefix: %w", err) } - allPrefixes = append(allPrefixes, prefixesV6...) - - if !known && len(allPrefixes) == 0 { - return nil, nil, fmt.Errorf("%w: AS%d", ErrNoRoute, asn) - } - - return &asnInfo, allPrefixes, nil + return firstSeen, nil } // ASPeer represents a peering relationship with another AS including handle, @@ -2564,76 +2639,6 @@ func (d *Database) deletePeerRouteChunk(peerIP string) (int, int, error) { return int(deletedV4), int(deletedV6), nil } -// routeTimestampRange returns the earliest and latest last_updated across both -// live route tables, or nil values when both tables are empty. Each query reads -// one row from an end of the last_updated index rather than scanning the tables. -func (d *Database) routeTimestampRange( - ctx context.Context, -) (*time.Time, *time.Time, error) { - var oldest, newest *time.Time - - oldestV4, ok, err := d.scanRouteTimestamp(ctx, "SELECT last_updated "+ - "FROM live_routes_v4 ORDER BY last_updated ASC LIMIT 1") - if err != nil { - return nil, nil, err - } - - if ok { - oldest = &oldestV4 - } - - oldestV6, ok, err := d.scanRouteTimestamp(ctx, "SELECT last_updated "+ - "FROM live_routes_v6 ORDER BY last_updated ASC LIMIT 1") - if err != nil { - return nil, nil, err - } - - if ok && (oldest == nil || oldestV6.Before(*oldest)) { - oldest = &oldestV6 - } - - newestV4, ok, err := d.scanRouteTimestamp(ctx, "SELECT last_updated "+ - "FROM live_routes_v4 ORDER BY last_updated DESC LIMIT 1") - if err != nil { - return nil, nil, err - } - - if ok { - newest = &newestV4 - } - - newestV6, ok, err := d.scanRouteTimestamp(ctx, "SELECT last_updated "+ - "FROM live_routes_v6 ORDER BY last_updated DESC LIMIT 1") - if err != nil { - return nil, nil, err - } - - if ok && (newest == nil || newestV6.After(*newest)) { - newest = &newestV6 - } - - return oldest, newest, nil -} - -// scanRouteTimestamp runs a single-row timestamp query. The bool is false when -// the table is empty. The query selects the last_updated column directly so the -// driver parses the DATETIME value into a time.Time. -func (d *Database) scanRouteTimestamp( - ctx context.Context, query string, -) (time.Time, bool, error) { - var ts time.Time - - err := d.db.QueryRowContext(ctx, query).Scan(&ts) - switch { - case errors.Is(err, sql.ErrNoRows): - return time.Time{}, false, nil - case err != nil: - return time.Time{}, false, err - default: - return ts, true, nil - } -} - // The AS page lists the prefixes an AS originates one at a time, each in two // index searches that read no table row: one of the origin_asn and prefix // index for the AS's first prefix after the previous one, and one of the diff --git a/internal/database/database_internal_test.go b/internal/database/database_internal_test.go index f5182d0..6891927 100644 --- a/internal/database/database_internal_test.go +++ b/internal/database/database_internal_test.go @@ -1780,6 +1780,165 @@ func TestGetASDetailsOfASNotInDatabase(t *testing.T) { } } +// TestGetASReadsTheRecordOrTheBuiltInList asks GetAS for an AS the asns table +// holds, one only the AS info lookup holds, and one neither holds. The first +// gets its record's seen times, the second its handle and description from the +// lookup, and only the third is not known. +func TestGetASReadsTheRecordOrTheBuiltInList(t *testing.T) { + t.Parallel() + + db, err := New(&config.Config{StateDir: t.TempDir()}, logger.New()) + if err != nil { + t.Fatalf("failed to create database: %v", err) + } + defer func() { _ = db.Close() }() + + const ( + storedASN = 64500 // a documentation AS, not in the lookup + listedASN = 15169 // in the lookup, not stored + unknownASN = 4200000001 // in neither + ) + + ts := time.Date(2026, 1, 2, 3, 4, 5, 0, time.UTC) + + _, err = db.GetOrCreateASN(storedASN, ts) + if err != nil { + t.Fatalf("GetOrCreateASN: %v", err) + } + + listed, ok := asinfo.Get(listedASN) + if !ok { + t.Fatalf("AS%d is not in the AS info data", listedASN) + } + + tests := []struct { + asn int + want ASN + wantKnown bool + }{ + {storedASN, ASN{ASN: storedASN, FirstSeen: ts, LastSeen: ts}, true}, + {listedASN, ASN{ + ASN: listedASN, Handle: listed.Handle, Description: listed.Description, + }, true}, + {unknownASN, ASN{ASN: unknownASN}, false}, + } + + for _, tt := range tests { + got, known, err := db.GetAS(t.Context(), tt.asn) + if err != nil { + t.Errorf("GetAS(%d): %v", tt.asn, err) + + continue + } + + if !got.FirstSeen.Equal(tt.want.FirstSeen) || + !got.LastSeen.Equal(tt.want.LastSeen) { + t.Errorf("GetAS(%d) seen %v to %v, want %v to %v", tt.asn, + got.FirstSeen, got.LastSeen, tt.want.FirstSeen, tt.want.LastSeen) + } + + got.FirstSeen, got.LastSeen = tt.want.FirstSeen, tt.want.LastSeen + if *got != tt.want || known != tt.wantKnown { + t.Errorf("GetAS(%d) = %+v, known %t; want %+v, known %t", + tt.asn, *got, known, tt.want, tt.wantKnown) + } + } +} + +// TestGetASWHOISReadsTheStoredDetails stores an AS with WHOIS details and asks +// GetASWHOIS for it and for an AS the asns table does not hold. The first gets +// its stored handle, description and details, and the time they were fetched. +// The second gets only its number, although the AS info lookup holds it. +func TestGetASWHOISReadsTheStoredDetails(t *testing.T) { + t.Parallel() + + db, err := New(&config.Config{StateDir: t.TempDir()}, logger.New()) + if err != nil { + t.Fatalf("failed to create database: %v", err) + } + defer func() { _ = db.Close() }() + + const storedASN, unstoredASN = 15169, 13335 + + _, err = db.GetOrCreateASN(storedASN, time.Date(2026, 1, 2, 3, 4, 5, 0, time.UTC)) + if err != nil { + t.Fatalf("GetOrCreateASN: %v", err) + } + + update := &ASNWHOISUpdate{ + ASN: storedASN, ASName: "TEST-AS", OrgName: "Test Org", OrgID: "ORG-1", + Address: "1 Test Street", CountryCode: "DE", AbuseEmail: "abuse@test", + RIR: "RIPE", + } + + err = db.UpdateASNWHOIS(t.Context(), update) + if err != nil { + t.Fatalf("UpdateASNWHOIS: %v", err) + } + + got, err := db.GetASWHOIS(t.Context(), storedASN) + if err != nil { + t.Fatalf("GetASWHOIS(%d): %v", storedASN, err) + } + + want := ASN{ + ASN: storedASN, Handle: asinfo.GetHandle(storedASN), + Description: asinfo.GetDescription(storedASN), ASName: update.ASName, + OrgName: update.OrgName, OrgID: update.OrgID, Address: update.Address, + CountryCode: update.CountryCode, AbuseEmail: update.AbuseEmail, + RIR: update.RIR, + } + + if got.WHOISUpdatedAt == nil { + t.Errorf("GetASWHOIS(%d) has no time the details were fetched", storedASN) + } + + got.WHOISUpdatedAt = nil + if *got != want { + t.Errorf("GetASWHOIS(%d) = %+v, want %+v", storedASN, *got, want) + } + + got, err = db.GetASWHOIS(t.Context(), unstoredASN) + if err != nil || *got != (ASN{ASN: unstoredASN}) { + t.Errorf("GetASWHOIS(%d) = %+v, %v; want only the number", + unstoredASN, got, err) + } +} + +// TestGetPrefixFirstSeenReadsThePrefixTables stores an IPv4 and an IPv6 +// prefix and expects GetPrefixFirstSeen to give each its first-seen time, and +// the zero time for a prefix that is not stored. +func TestGetPrefixFirstSeenReadsThePrefixTables(t *testing.T) { + t.Parallel() + + db, err := New(&config.Config{StateDir: t.TempDir()}, logger.New()) + if err != nil { + t.Fatalf("failed to create database: %v", err) + } + defer func() { _ = db.Close() }() + + ts := time.Date(2026, 1, 2, 3, 4, 5, 0, time.UTC) + stored := map[string]time.Time{ + "192.0.2.0/24": ts, + testV6Prefix: ts.Add(time.Hour), + } + + err = db.UpdatePrefixesBatch(stored) + if err != nil { + t.Fatalf("UpdatePrefixesBatch: %v", err) + } + + stored["198.51.100.0/24"] = time.Time{} + + for prefix, want := range stored { + got, err := db.GetPrefixFirstSeen(t.Context(), netip.MustParsePrefix(prefix)) + if err != nil || !got.Equal(want) { + t.Errorf("GetPrefixFirstSeen(%s) = %v, %v; want %v", + prefix, got, err, want) + } + } +} + // TestASPrefixQueriesUseCoveringIndexes checks the plan of each query that // lists the prefixes an AS originates: a search of a covering index and no // temporary B-tree, so the list reads one index entry per prefix, not one diff --git a/internal/database/interface.go b/internal/database/interface.go index 1e8bf5f..195c267 100644 --- a/internal/database/interface.go +++ b/internal/database/interface.go @@ -6,6 +6,7 @@ package database import ( "context" + "net/netip" "time" "git.eeqj.de/sneak/routewatch/internal/routetable" @@ -24,8 +25,6 @@ type Stats struct { LiveRoutes int IPv4Routes int IPv6Routes int - OldestRoute *time.Time - NewestRoute *time.Time IPv4PrefixDistribution []PrefixDistribution IPv6PrefixDistribution []PrefixDistribution } @@ -85,6 +84,9 @@ type Store interface { GetWHOISStats(ctx context.Context, staleThreshold time.Duration) (*WHOISStats, error) // AS and prefix detail operations + GetAS(ctx context.Context, asn int) (*ASN, bool, error) + GetASWHOIS(ctx context.Context, asn int) (*ASN, error) + GetPrefixFirstSeen(ctx context.Context, prefix netip.Prefix) (time.Time, error) GetASDetails(asn int) (*ASN, []LiveRoute, error) GetASDetailsContext(ctx context.Context, asn int) (*ASN, []LiveRoute, error) GetASPeers(asn int) ([]ASPeer, error) diff --git a/internal/database/models.go b/internal/database/models.go index 8bdc4d6..36cc035 100644 --- a/internal/database/models.go +++ b/internal/database/models.go @@ -66,7 +66,9 @@ type ASNPeering struct { LastSeen time.Time } -// LiveRoute represents a route in the live routing table +// LiveRoute represents a route in the live routing table. JSON leaves out an +// empty AS path, next hop or time: the routing table in memory keeps them only +// for the two newest routes of each prefix and origin AS. // //nolint:tagliatelle // field names are the existing JSON API type LiveRoute struct { @@ -75,9 +77,9 @@ type LiveRoute struct { IPVersion int `json:"ip_version"` OriginASN int `json:"origin_asn"` PeerIP string `json:"peer_ip"` - ASPath []int `json:"as_path"` - NextHop string `json:"next_hop"` - LastUpdated time.Time `json:"last_updated"` + ASPath []int `json:"as_path,omitempty"` + NextHop string `json:"next_hop,omitempty"` + LastUpdated time.Time `json:"last_updated,omitzero"` } // PrefixDistribution represents the distribution of prefixes by mask length @@ -108,9 +110,10 @@ type IPInfo struct { CountryCode string `json:"country_code,omitempty"` AbuseEmail string `json:"abuse_email,omitempty"` RIR string `json:"rir,omitempty"` - // Timestamps + // Timestamps. LastSeen is zero, and left out of JSON, when the prefix + // keeps no route. FirstSeen time.Time `json:"first_seen"` - LastSeen time.Time `json:"last_seen"` + LastSeen time.Time `json:"last_seen,omitzero"` // Indicates if WHOIS data needs refresh (not serialized) NeedsWHOISRefresh bool `json:"-"` } diff --git a/internal/routewatch/app.go b/internal/routewatch/app.go index f0eafef..7832669 100644 --- a/internal/routewatch/app.go +++ b/internal/routewatch/app.go @@ -31,11 +31,13 @@ import ( // are disabled, since only the batched handlers exist. var errNonBatchedNotImplemented = errors.New("non-batched handlers not implemented") -// Dependencies contains all dependencies for RouteWatch +// Dependencies contains all dependencies for RouteWatch. Table is the routing +// table read from the database at startup, which the server reads too. type Dependencies struct { fx.In DB database.Store + Table *routetable.Table Streamer *streamer.Streamer Server *server.Server Logger *logger.Logger @@ -45,6 +47,7 @@ type Dependencies struct { // RouteWatch represents the main application instance type RouteWatch struct { db database.Store + table *routetable.Table streamer *streamer.Streamer server *server.Server logger *logger.Logger @@ -69,6 +72,7 @@ const blocklistsDirName = "blocklists" func New(deps Dependencies) *RouteWatch { rw := &RouteWatch{ db: deps.DB, + table: deps.Table, streamer: deps.Streamer, server: deps.Server, logger: deps.Logger, @@ -97,13 +101,6 @@ func (rw *RouteWatch) Run(ctx context.Context) error { if rw.config.EnableBatchedDatabaseWrites { rw.logger.Info("Using batched database handlers for improved performance") - // The routing table is read before the streamer connects, so that - // the prefix handler applies the feed's updates to all of it. - table, err := rw.readRouteTable(ctx) - if err != nil { - return err - } - // ASHandler maintains the asns table rw.dbHandler = NewASHandler(rw.db, rw.logger) rw.streamer.RegisterHandler(rw.dbHandler) @@ -114,7 +111,7 @@ func (rw *RouteWatch) Run(ctx context.Context) error { // PrefixHandler maintains the prefixes and live_routes tables, and // the routing table in memory and its copy in the routes table - rw.prefixHandler = NewPrefixHandler(rw.db, rw.logger, table) + rw.prefixHandler = NewPrefixHandler(rw.db, rw.logger, rw.table) rw.prefixHandler.SetMetricsTracker(rw.streamer.GetMetricsTracker()) rw.streamer.RegisterHandler(rw.prefixHandler) @@ -269,20 +266,21 @@ func (rw *RouteWatch) closeDatabase(ctx context.Context) { // readRouteTable reads the routing table from the database and logs how many // prefix and origin AS pairs and peers it holds and how long the read took. -// ctx is fx's start context, which ends 15 seconds after the start. Reading a -// full table can take longer, so the read does not stop with ctx. -func (rw *RouteWatch) readRouteTable( - ctx context.Context, +// fx calls it once, to build the server and RouteWatch, before the streamer +// connects, so the prefix handler applies the feed's updates to all of it. No +// start timeout limits the read: fx builds them before it runs a start hook. +func readRouteTable( + db database.Store, logger *logger.Logger, ) (*routetable.Table, error) { start := time.Now() - table, err := rw.db.ReadRouteTable(context.WithoutCancel(ctx)) + table, err := db.ReadRouteTable(context.Background()) if err != nil { return nil, fmt.Errorf("failed to read the routing table: %w", err) } counts := table.Counts() - rw.logger.Info("Read the routing table", "pairs", counts.Pairs, + logger.Info("Read the routing table", "pairs", counts.Pairs, "peers", counts.Peers, "duration_ms", time.Since(start).Milliseconds()) return table, nil @@ -299,6 +297,7 @@ func getModule() []fx.Option { database.New, fx.As(new(database.Store)), ), + readRouteTable, streamer.New, server.New, New, diff --git a/internal/routewatch/app_integration_internal_test.go b/internal/routewatch/app_integration_internal_test.go index 86c78bb..f6743e9 100644 --- a/internal/routewatch/app_integration_internal_test.go +++ b/internal/routewatch/app_integration_internal_test.go @@ -7,6 +7,7 @@ import ( "net" "net/http" "net/http/httptest" + "net/netip" "slices" "strconv" "strings" @@ -265,6 +266,32 @@ func (m *mockStore) GetASDetailsContext( return m.GetASDetails(asn) } +// GetAS returns the stored AS, or only its number when none is stored. +func (m *mockStore) GetAS(_ context.Context, asn int) (*database.ASN, bool, error) { + m.mu.Lock() + defer m.mu.Unlock() + + if asnInfo, exists := m.ASNs[asn]; exists { + return asnInfo, true, nil + } + + return &database.ASN{ASN: asn}, false, nil +} + +// GetASWHOIS returns the stored AS, or only its number when none is stored. +func (m *mockStore) GetASWHOIS(ctx context.Context, asn int) (*database.ASN, error) { + asnInfo, _, err := m.GetAS(ctx, asn) + + return asnInfo, err +} + +// GetPrefixFirstSeen returns the zero time: the mock keeps no prefix times. +func (m *mockStore) GetPrefixFirstSeen( + _ context.Context, _ netip.Prefix, +) (time.Time, error) { + return time.Time{}, nil +} + // GetPrefixDetails mock implementation func (m *mockStore) GetPrefixDetails( _ string, @@ -502,12 +529,15 @@ func TestRouteWatchLiveFeed(t *testing.T) { EnableBatchedDatabaseWrites: true, } + table := routetable.New(routetable.Params{}) + // Create server - srv := server.New(mockDB, s, logger, cfg) + srv := server.New(mockDB, table, s, logger, cfg) // Create RouteWatch with 5 second limit deps := Dependencies{ DB: mockDB, + Table: table, Streamer: s, Server: srv, Logger: logger, @@ -626,7 +656,7 @@ func TestShutdownWritesTheUpdatesQueuedInTheStreamer(t *testing.T) { rw := New(Dependencies{ DB: store, Streamer: s, - Server: server.New(store, s, log, cfg), + Server: server.New(store, nil, s, log, cfg), Logger: log, Config: cfg, }) @@ -728,7 +758,7 @@ func TestShutdownReturnsByItsDeadlineWhileAWriteIsBlocked(t *testing.T) { rw := New(Dependencies{ DB: store, Streamer: s, - Server: server.New(store, s, log, cfg), + Server: server.New(store, nil, s, log, cfg), Logger: log, Config: cfg, }) @@ -842,7 +872,7 @@ func TestShutdownCheckpointsAndClosesTheDatabaseLast(t *testing.T) { rw := New(Dependencies{ DB: store, Streamer: s, - Server: server.New(store, s, log, cfg), + Server: server.New(store, nil, s, log, cfg), Logger: log, Config: cfg, }) diff --git a/internal/routewatch/prefixhandler_table_internal_test.go b/internal/routewatch/prefixhandler_table_internal_test.go index dc8a7cd..97969e2 100644 --- a/internal/routewatch/prefixhandler_table_internal_test.go +++ b/internal/routewatch/prefixhandler_table_internal_test.go @@ -416,12 +416,8 @@ func TestARestartReadsTheRoutingTableBack(t *testing.T) { var logs bytes.Buffer - rw := &RouteWatch{ - db: db, - logger: &logger.Logger{Logger: slog.New(slog.NewJSONHandler(&logs, nil))}, - } - - table, err := rw.readRouteTable(t.Context()) + table, err := readRouteTable(db, + &logger.Logger{Logger: slog.New(slog.NewJSONHandler(&logs, nil))}) if err != nil { t.Fatalf("readRouteTable: %v", err) } diff --git a/internal/server/handlers.go b/internal/server/handlers.go index de829ce..1f5fa29 100644 --- a/internal/server/handlers.go +++ b/internal/server/handlers.go @@ -5,12 +5,12 @@ import ( "context" "encoding/json" "errors" + "fmt" "net" "net/http" "net/netip" "runtime" "slices" - "sort" "strconv" "strings" "time" @@ -19,6 +19,7 @@ import ( "git.eeqj.de/sneak/routewatch/internal/database" "git.eeqj.de/sneak/routewatch/internal/metrics" "git.eeqj.de/sneak/routewatch/internal/ristypes" + "git.eeqj.de/sneak/routewatch/internal/routetable" "git.eeqj.de/sneak/routewatch/internal/streamer" "git.eeqj.de/sneak/routewatch/internal/templates" asinfo "git.eeqj.de/sneak/routewatch/pkg/asinfo" @@ -38,7 +39,7 @@ const ( const ( // maxLookupTargets is the most addresses and host names an IP lookup may // list, and the most addresses it may have once the names are resolved: - // each address costs up to 113 database reads and a PTR lookup. + // each address costs two database reads and a PTR lookup. maxLookupTargets = 10 // ptrLookupTimeout is each PTR lookup's own time limit, so that ten @@ -227,7 +228,8 @@ type WHOISStatsInfo struct { } // Stats is the /status.json answer: uptime, message counts, database stats, -// and route information. +// and the routing table's figures. OldestRoute and NewestRoute are the times +// of the oldest and the newest kept routes. // //nolint:tagliatelle // field names are the existing JSON API type Stats struct { @@ -259,6 +261,10 @@ type Stats struct { IPv4PrefixDistribution []database.PrefixDistribution `json:"ipv4_prefix_distribution"` IPv6PrefixDistribution []database.PrefixDistribution `json:"ipv6_prefix_distribution"` WHOISStats *WHOISStatsInfo `json:"whois_stats,omitempty"` + + // OriginsWithoutKeptRoute counts the prefix and origin AS pairs that peers + // carry but whose kept routes were all withdrawn. + OriginsWithoutKeptRoute int `json:"origins_without_kept_route"` } // databaseStats gets the database stats, giving up when ctx is done; it then @@ -317,15 +323,6 @@ func (s *Server) handleStatusJSON() http.HandlerFunc { return } - const bitsPerMegabit = 1000000.0 - - // Get route update metrics - routeMetrics := s.streamer.GetMetricsTracker().GetRouteMetrics() - - // Get memory stats - var memStats runtime.MemStats - runtime.ReadMemStats(&memStats) - // Get WHOIS stats if fetcher is available var whoisStats *WHOISStatsInfo if s.asnFetcher != nil { @@ -337,44 +334,80 @@ func (s *Server) handleStatusJSON() http.HandlerFunc { } } - stats := Stats{ - Uptime: streamUptime(metrics.ConnectedSince), - TotalMessages: metrics.TotalMessages, - TotalBytes: metrics.TotalBytes, - TotalWireBytes: metrics.TotalWireBytes, - MessagesPerSec: metrics.MessagesPerSec, - MbitsPerSec: metrics.BitsPerSec / bitsPerMegabit, - WireMbitsPerSec: metrics.WireBitsPerSec / bitsPerMegabit, - Connected: metrics.Connected, - GoVersion: runtime.Version(), - Goroutines: runtime.NumGoroutine(), - MemoryUsage: humanize.Bytes(memStats.Alloc), - ASNs: dbStats.ASNs, - Prefixes: dbStats.Prefixes, - IPv4Prefixes: dbStats.IPv4Prefixes, - IPv6Prefixes: dbStats.IPv6Prefixes, - Peerings: dbStats.Peerings, - Peers: dbStats.Peers, - DatabaseSizeBytes: dbStats.FileSizeBytes, - LiveRoutes: dbStats.LiveRoutes, - IPv4Routes: dbStats.IPv4Routes, - IPv6Routes: dbStats.IPv6Routes, - OldestRoute: dbStats.OldestRoute, - NewestRoute: dbStats.NewestRoute, - IPv4UpdatesPerSec: routeMetrics.IPv4UpdatesPerSec, - IPv6UpdatesPerSec: routeMetrics.IPv6UpdatesPerSec, - IPv4PrefixDistribution: dbStats.IPv4PrefixDistribution, - IPv6PrefixDistribution: dbStats.IPv6PrefixDistribution, - WHOISStats: whoisStats, - } - - err = writeJSONSuccess(w, stats) + err = writeJSONSuccess(w, s.statusStats(metrics, dbStats, whoisStats)) if err != nil { s.logger.Error("Failed to encode stats", "error", err) } } } +// statusStats builds the /status.json answer from the stream metrics, the +// database stats, the routing table's figures and the WHOIS stats, with the +// route update rates and memory use as they are now. +func (s *Server) statusStats( + streamMetrics metrics.StreamMetrics, + dbStats database.Stats, + whoisStats *WHOISStatsInfo, +) Stats { + const bitsPerMegabit = 1000000.0 + + routeMetrics := s.streamer.GetMetricsTracker().GetRouteMetrics() + counts := s.table.Counts() + + var memStats runtime.MemStats + runtime.ReadMemStats(&memStats) + + return Stats{ + Uptime: streamUptime(streamMetrics.ConnectedSince), + TotalMessages: streamMetrics.TotalMessages, + TotalBytes: streamMetrics.TotalBytes, + TotalWireBytes: streamMetrics.TotalWireBytes, + MessagesPerSec: streamMetrics.MessagesPerSec, + MbitsPerSec: streamMetrics.BitsPerSec / bitsPerMegabit, + WireMbitsPerSec: streamMetrics.WireBitsPerSec / bitsPerMegabit, + Connected: streamMetrics.Connected, + GoVersion: runtime.Version(), + Goroutines: runtime.NumGoroutine(), + MemoryUsage: humanize.Bytes(memStats.Alloc), + ASNs: dbStats.ASNs, + Prefixes: counts.IPv4Prefixes + counts.IPv6Prefixes, + IPv4Prefixes: counts.IPv4Prefixes, + IPv6Prefixes: counts.IPv6Prefixes, + Peerings: dbStats.Peerings, + Peers: dbStats.Peers, + DatabaseSizeBytes: dbStats.FileSizeBytes, + LiveRoutes: counts.IPv4Routes + counts.IPv6Routes, + IPv4Routes: counts.IPv4Routes, + IPv6Routes: counts.IPv6Routes, + OldestRoute: counts.OldestKeptRoute, + NewestRoute: counts.NewestKeptRoute, + IPv4UpdatesPerSec: routeMetrics.IPv4UpdatesPerSec, + IPv6UpdatesPerSec: routeMetrics.IPv6UpdatesPerSec, + IPv4PrefixDistribution: prefixDistribution(counts.IPv4PrefixesByLength), + IPv6PrefixDistribution: prefixDistribution(counts.IPv6PrefixesByLength), + WHOISStats: whoisStats, + + OriginsWithoutKeptRoute: counts.OriginsWithoutKeptRoute, + } +} + +// prefixDistribution lists the mask lengths that have at least one prefix, in +// ascending order, with their prefix counts. countsByLength[n] is the number +// of prefixes of mask length n. +func prefixDistribution(countsByLength []int) []database.PrefixDistribution { + var distribution []database.PrefixDistribution + + for maskLength, count := range countsByLength { + if count > 0 { + distribution = append(distribution, database.PrefixDistribution{ + MaskLength: maskLength, Count: count, + }) + } + } + + return distribution +} + // getWHOISStats builds WHOIS statistics from database and fetcher. func (s *Server) getWHOISStats(ctx context.Context) *WHOISStatsInfo { // Get database WHOIS stats @@ -470,7 +503,8 @@ type StreamStats struct { BGPPeerCount int `json:"bgp_peer_count"` } -// StatsResponse represents the API statistics response +// StatsResponse represents the API statistics response. The fields it shares +// with Stats mean what they mean there. // //nolint:tagliatelle // field names are the existing JSON API type StatsResponse struct { @@ -507,6 +541,8 @@ type StatsResponse struct { IPv4PrefixDistribution []database.PrefixDistribution `json:"ipv4_prefix_distribution"` IPv6PrefixDistribution []database.PrefixDistribution `json:"ipv6_prefix_distribution"` WHOISStats *WHOISStatsInfo `json:"whois_stats,omitempty"` + + OriginsWithoutKeptRoute int `json:"origins_without_kept_route"` } // handlerStatsInfo converts the streamer's handler statistics for the API @@ -626,8 +662,8 @@ func (s *Server) handleStats() http.HandlerFunc { } // statsResponse builds the /api/v1/stats answer from the stream metrics, the -// database stats and the WHOIS stats, with the route update rates, memory use -// and handler queues as they are now. +// database stats, the routing table's figures and the WHOIS stats, with the +// route update rates, memory use and handler queues as they are now. func (s *Server) statsResponse( streamMetrics metrics.StreamMetrics, dbStats database.Stats, @@ -637,6 +673,7 @@ func (s *Server) statsResponse( // Get route update metrics routeMetrics := s.streamer.GetMetricsTracker().GetRouteMetrics() + counts := s.table.Counts() // Get memory stats var memStats runtime.MemStats @@ -660,23 +697,25 @@ func (s *Server) statsResponse( GC: gcStats(&memStats), Stream: s.streamStats(), ASNs: dbStats.ASNs, - Prefixes: dbStats.Prefixes, - IPv4Prefixes: dbStats.IPv4Prefixes, - IPv6Prefixes: dbStats.IPv6Prefixes, + Prefixes: counts.IPv4Prefixes + counts.IPv6Prefixes, + IPv4Prefixes: counts.IPv4Prefixes, + IPv6Prefixes: counts.IPv6Prefixes, Peerings: dbStats.Peerings, Peers: dbStats.Peers, DatabaseSizeBytes: dbStats.FileSizeBytes, - LiveRoutes: dbStats.LiveRoutes, - IPv4Routes: dbStats.IPv4Routes, - IPv6Routes: dbStats.IPv6Routes, - OldestRoute: dbStats.OldestRoute, - NewestRoute: dbStats.NewestRoute, + LiveRoutes: counts.IPv4Routes + counts.IPv6Routes, + IPv4Routes: counts.IPv4Routes, + IPv6Routes: counts.IPv6Routes, + OldestRoute: counts.OldestKeptRoute, + NewestRoute: counts.NewestKeptRoute, IPv4UpdatesPerSec: routeMetrics.IPv4UpdatesPerSec, IPv6UpdatesPerSec: routeMetrics.IPv6UpdatesPerSec, HandlerStats: handlerStatsInfo(s.streamer.GetHandlerStats()), - IPv4PrefixDistribution: dbStats.IPv4PrefixDistribution, - IPv6PrefixDistribution: dbStats.IPv6PrefixDistribution, + IPv4PrefixDistribution: prefixDistribution(counts.IPv4PrefixesByLength), + IPv6PrefixDistribution: prefixDistribution(counts.IPv6PrefixesByLength), WHOISStats: whoisStats, + + OriginsWithoutKeptRoute: counts.OriginsWithoutKeptRoute, } } @@ -766,7 +805,7 @@ func (s *Server) handleIPInfo() http.HandlerFunc { // Look up each IP for _, ip := range ipsToLookup { - ipInfo, err := s.db.GetIPInfoContext(ctx, ip) + ipInfo, err := s.ipInfo(ctx, ip) if deadlinePassed(ctx) { s.writeJSONError(w, http.StatusGatewayTimeout, requestTimeoutMessage) @@ -815,6 +854,81 @@ func (s *Server) handleIPInfo() http.HandlerFunc { } } +// ipInfo returns the IP lookup's answer for ip. The routing table gives the +// most specific prefix that holds the address, the number of peers that carry +// it, the origin AS that most of them carry, and the time of its newest kept +// route. The database gives the prefix's first-seen time and the AS's WHOIS +// details. It returns ErrNoRoute when no prefix holds the address, and +// ErrInvalidIP when ip does not parse. +func (s *Server) ipInfo(ctx context.Context, ip string) (*database.IPInfo, error) { + addr, err := netip.ParseAddr(ip) + if err != nil { + return nil, fmt.Errorf("%w: %s", database.ErrInvalidIP, ip) + } + + summary, found := s.table.Lookup(addr) + if !found { + return nil, fmt.Errorf("%w for IP %s", database.ErrNoRoute, ip) + } + + firstSeen, err := s.db.GetPrefixFirstSeen(ctx, summary.Prefix) + if err != nil { + return nil, err + } + + asInfo, err := s.db.GetASWHOIS(ctx, summary.OriginASN) + if err != nil { + return nil, err + } + + info := &database.IPInfo{ + IP: ip, + Netblock: summary.Prefix.String(), + MaskLength: summary.Prefix.Bits(), + IPVersion: ipVersionOf(summary.Prefix), + NumPeers: summary.Peers, + ASN: summary.OriginASN, + ASName: asInfo.ASName, + Handle: asInfo.Handle, + Description: asInfo.Description, + OrgName: asInfo.OrgName, + OrgID: asInfo.OrgID, + Address: asInfo.Address, + CountryCode: asInfo.CountryCode, + AbuseEmail: asInfo.AbuseEmail, + RIR: asInfo.RIR, + FirstSeen: firstSeen, + NeedsWHOISRefresh: needsWHOISRefresh(asInfo), + } + + if summary.Newest != nil { + info.LastSeen = *summary.Newest + } + + return info, nil +} + +// needsWHOISRefresh says whether the WHOIS details of asInfo were never +// fetched or are older than whoisStaleThreshold. AS 0, no known origin, has +// none to fetch. +func needsWHOISRefresh(asInfo *database.ASN) bool { + if asInfo.ASN == ristypes.NoOrigin { + return false + } + + return asInfo.WHOISUpdatedAt == nil || + time.Since(*asInfo.WHOISUpdatedAt) > whoisStaleThreshold +} + +// ipVersionOf returns 4 or 6, the IP version of prefix. +func ipVersionOf(prefix netip.Prefix) int { + if prefix.Addr().Is4() { + return ipVersionV4 + } + + return ipVersionV6 +} + // blocklistListings returns the blocklists that list ip. It returns an empty // list, never nil, so that an answer's blocklists is [] rather than null. func (s *Server) blocklistListings(ip string) []blocklist.Listing { @@ -1043,7 +1157,7 @@ func (s *Server) handleASDetailJSON() http.HandlerFunc { return } - asInfo, prefixes, err := s.db.GetASDetailsContext(r.Context(), asn) + asInfo, prefixes, err := s.asDetails(r.Context(), asn) if deadlinePassed(r.Context()) { s.writeJSONError(w, http.StatusGatewayTimeout, requestTimeoutMessage) @@ -1079,6 +1193,49 @@ func (s *Server) handleASDetailJSON() http.HandlerFunc { } } +// asDetails returns the AS's handle, description and seen times, from the +// database, and the prefixes it originates, from the routing table. Each +// prefix has the time of its newest kept route, if it keeps one. It returns +// ErrNoRoute for AS 0, which means no known origin. It also returns ErrNoRoute +// for an AS that the database, the AS list built into RouteWatch and the +// routing table all lack. +func (s *Server) asDetails( + ctx context.Context, asn int, +) (*database.ASN, []database.LiveRoute, error) { + if asn == ristypes.NoOrigin { + return nil, nil, fmt.Errorf("%w: AS%d", database.ErrNoRoute, asn) + } + + asInfo, known, err := s.db.GetAS(ctx, asn) + if err != nil { + return nil, nil, err + } + + originated := s.table.OriginatedPrefixes(asn) + if !known && len(originated) == 0 { + return nil, nil, fmt.Errorf("%w: AS%d", database.ErrNoRoute, asn) + } + + prefixes := make([]database.LiveRoute, 0, len(originated)) + + for _, p := range originated { + route := database.LiveRoute{ + Prefix: p.Prefix.String(), + MaskLength: p.Prefix.Bits(), + IPVersion: ipVersionOf(p.Prefix), + OriginASN: asn, + } + + if p.Newest != nil { + route.LastUpdated = *p.Newest + } + + prefixes = append(prefixes, route) + } + + return asInfo, prefixes, nil +} + // splitRoutesByIPVersion groups routes into the IPv4 and the IPv6 ones. Neither // group is nil, so that a JSON answer holds [] for an empty one and not null. func splitRoutesByIPVersion( @@ -1123,8 +1280,7 @@ func (s *Server) handlePrefixDetailJSON() http.HandlerFunc { return } - // Look the prefix up in the text form net/netip prints, the form the - // tables store, so that any valid way of writing it finds the route. + // Masking the prefix lets any valid way of writing it find its routes. parsed, err := netip.ParsePrefix(prefixParam + "/" + lenParam) if err != nil { s.writeJSONError(w, http.StatusBadRequest, invalidPrefixMessage) @@ -1132,27 +1288,18 @@ func (s *Server) handlePrefixDetailJSON() http.HandlerFunc { return } - prefix := parsed.Masked().String() + parsed = parsed.Masked() + prefix := parsed.String() - routes, _, err := s.db.GetPrefixDetailsContext(r.Context(), prefix) - if deadlinePassed(r.Context()) { - s.writeJSONError(w, http.StatusGatewayTimeout, requestTimeoutMessage) + origins := s.table.Origins(parsed) + if len(origins) == 0 { + s.writeJSONError(w, http.StatusNotFound, + database.ErrNoRoute.Error()+": "+prefix) return } - if err != nil { - if errors.Is(err, database.ErrNoRoute) { - s.writeJSONError(w, http.StatusNotFound, err.Error()) - } else { - s.logger.Error("Failed to get prefix details", - "prefix", prefix, "error", err) - s.writeJSONError(w, http.StatusInternalServerError, - internalErrorMessage) - } - - return - } + routes := prefixRoutes(parsed, origins) // Group by origin AS originMap := make(map[int][]database.LiveRoute) @@ -1237,7 +1384,7 @@ func (s *Server) handleASDetail() http.HandlerFunc { return } - asInfo, prefixes, err := s.db.GetASDetailsContext(r.Context(), asn) + asInfo, prefixes, err := s.asDetails(r.Context(), asn) if deadlinePassed(r.Context()) { http.Error(w, requestTimeoutMessage, http.StatusGatewayTimeout) @@ -1327,8 +1474,7 @@ func (s *Server) handlePrefixDetail() http.HandlerFunc { return } - // Look the prefix up in the text form net/netip prints, the form the - // tables store, so that any valid way of writing it finds the route. + // Masking the prefix lets any valid way of writing it find its routes. parsed, err := netip.ParsePrefix(prefixParam + "/" + lenParam) if err != nil { http.Error(w, invalidPrefixMessage, http.StatusBadRequest) @@ -1336,9 +1482,16 @@ func (s *Server) handlePrefixDetail() http.HandlerFunc { return } - prefix := parsed.Masked().String() + parsed = parsed.Masked() - routes, originNames, err := s.db.GetPrefixDetailsContext(r.Context(), prefix) + origins := s.table.Origins(parsed) + if len(origins) == 0 { + http.Error(w, "Prefix not found", http.StatusNotFound) + + return + } + + originInfos, err := s.originASNs(r.Context(), origins) if deadlinePassed(r.Context()) { http.Error(w, requestTimeoutMessage, http.StatusGatewayTimeout) @@ -1346,27 +1499,13 @@ func (s *Server) handlePrefixDetail() http.HandlerFunc { } if err != nil { - if errors.Is(err, database.ErrNoRoute) { - http.Error(w, "Prefix not found", http.StatusNotFound) - } else { - s.logger.Error("Failed to get prefix details", "error", err) - http.Error(w, internalErrorMessage, http.StatusInternalServerError) - } + s.logger.Error("Failed to get the origin ASes' names", "error", err) + http.Error(w, internalErrorMessage, http.StatusInternalServerError) return } - // Group by origin AS and collect unique AS info - origins := originASNs(routes, originNames) - - // Get the first route to extract some common info - var maskLength, ipVersion int - if len(routes) > 0 { - maskLength, ipVersion = prefixMaskLengthAndIPVersion(prefix) - } - - // Create enhanced routes with AS path handles - enhancedRoutes := routesWithASPathHandles(routes) + routes := prefixRoutes(parsed, origins) // Prepare template data data := struct { @@ -1379,13 +1518,13 @@ func (s *Server) handlePrefixDetail() http.HandlerFunc { OriginCount int CSPNonce string }{ - Prefix: prefix, - MaskLength: maskLength, - IPVersion: ipVersion, - Routes: enhancedRoutes, - Origins: origins, + Prefix: parsed.String(), + MaskLength: parsed.Bits(), + IPVersion: ipVersionOf(parsed), + Routes: routesWithASPathHandles(routes), + Origins: originInfos, PeerCount: len(routes), - OriginCount: len(origins), + OriginCount: len(originInfos), CSPNonce: cspNonce(r.Context()), } @@ -1415,34 +1554,72 @@ type ASNInfo struct { PeerCount int } -// originASNs groups routes by origin AS, with each AS's handle and description -// from names, as the prefix query returned them, and the number of routes it -// originates. The origin ASes come in no particular order. -func originASNs( - routes []database.LiveRoute, - names map[int]database.ASN, -) []*ASNInfo { - originMap := make(map[int]*ASNInfo) +// originASNs returns each of origins, a prefix's origin ASes, with its handle +// and description from the database and the number of peers that carry the +// prefix with it. +func (s *Server) originASNs( + ctx context.Context, origins []routetable.Origin, +) ([]*ASNInfo, error) { + infos := make([]*ASNInfo, 0, len(origins)) - for _, route := range routes { - if _, exists := originMap[route.OriginASN]; !exists { - originMap[route.OriginASN] = &ASNInfo{ - ASN: route.OriginASN, - Handle: names[route.OriginASN].Handle, - Description: names[route.OriginASN].Description, - PeerCount: 0, - } + for _, origin := range origins { + asInfo, _, err := s.db.GetAS(ctx, origin.ASN) + if err != nil { + return nil, err } - originMap[route.OriginASN].PeerCount++ + infos = append(infos, &ASNInfo{ + ASN: origin.ASN, + Handle: asInfo.Handle, + Description: asInfo.Description, + PeerCount: len(origin.Peers), + }) } - origins := make([]*ASNInfo, 0, len(originMap)) - for _, origin := range originMap { - origins = append(origins, origin) + return infos, nil +} + +// prefixRoutes returns one route for each peer that carries prefix, by origin +// AS in the order of origins. Each origin's kept routes come first, newest +// first, with their AS path, next hop and time. Its other peers follow without +// them, since the routing table keeps only their addresses. +func prefixRoutes( + prefix netip.Prefix, origins []routetable.Origin, +) []database.LiveRoute { + var routes []database.LiveRoute + + for _, origin := range origins { + base := database.LiveRoute{ + Prefix: prefix.String(), + MaskLength: prefix.Bits(), + IPVersion: ipVersionOf(prefix), + OriginASN: origin.ASN, + } + + kept := make(map[string]bool, len(origin.Routes)) + + for _, r := range origin.Routes { + kept[r.Peer] = true + + route := base + route.PeerIP = r.Peer + route.ASPath = r.ASPath + // The next hop is the peer, as the prefix handler records it. + route.NextHop = r.Peer + route.LastUpdated = r.Time + routes = append(routes, route) + } + + for _, peer := range origin.Peers { + if !kept[peer] { + route := base + route.PeerIP = peer + routes = append(routes, route) + } + } } - return origins + return routes } // ASPathEntry is one AS of a route's AS path, with its handle. @@ -1484,23 +1661,6 @@ func routesWithASPathHandles(routes []database.LiveRoute) []EnhancedRoute { return enhancedRoutes } -// prefixMaskLengthAndIPVersion returns the mask length and IP version of a -// prefix in CIDR notation, or zeros if it does not parse. -func prefixMaskLengthAndIPVersion(prefix string) (int, int) { - _, ipNet, err := net.ParseCIDR(prefix) - if err != nil { - return 0, 0 - } - - maskLength, _ := ipNet.Mask.Size() - - if ipNet.IP.To4() != nil { - return maskLength, ipVersionV4 - } - - return maskLength, ipVersionV6 -} - // handlePrefixLength shows a random sample of IPv4 prefixes with the // specified mask length func (s *Server) handlePrefixLength() http.HandlerFunc { @@ -1539,29 +1699,11 @@ func (s *Server) prefixLengthHandler(ipVersion, maxMaskLength int) http.HandlerF // Get random sample of prefixes const maxPrefixes = 500 - prefixes, err := s.db.GetRandomPrefixesByLengthContext( - r.Context(), maskLength, ipVersion, maxPrefixes) - if deadlinePassed(r.Context()) { - http.Error(w, requestTimeoutMessage, http.StatusGatewayTimeout) - - return - } - - if err != nil { - s.logger.Error("Failed to get prefixes by length", "error", err) - http.Error(w, internalErrorMessage, http.StatusInternalServerError) - - return - } + prefixes := s.table.SamplePrefixes(ipVersion, maskLength, maxPrefixes) // Sort prefixes for display - sort.Slice(prefixes, func(i, j int) bool { - // First compare by IP version - if prefixes[i].IPVersion != prefixes[j].IPVersion { - return prefixes[i].IPVersion < prefixes[j].IPVersion - } - // Then by prefix - return prefixes[i].Prefix < prefixes[j].Prefix + slices.SortFunc(prefixes, func(a, b routetable.PrefixSummary) int { + return a.Prefix.Compare(b.Prefix) }) // Render template @@ -1591,8 +1733,8 @@ func (s *Server) prefixLengthHandler(ipVersion, maxMaskLength int) http.HandlerF } } -// EnhancedPrefix is a live route on the prefix length page, with the -// description of its origin AS and its age. +// EnhancedPrefix is a prefix on the prefix length page, with the origin AS +// that most peers carry it with, that AS's description, and its age. type EnhancedPrefix struct { database.LiveRoute @@ -1601,13 +1743,23 @@ type EnhancedPrefix struct { } // prefixesWithASDescriptions looks up the description of the origin AS of -// each prefix and gives each one's age. -func prefixesWithASDescriptions(prefixes []database.LiveRoute) []EnhancedPrefix { +// each prefix. It gives each prefix the age of its newest kept route, or no +// age when it keeps none. +func prefixesWithASDescriptions( + prefixes []routetable.PrefixSummary, +) []EnhancedPrefix { enhancedPrefixes := make([]EnhancedPrefix, len(prefixes)) for i, prefix := range prefixes { enhancedPrefixes[i] = EnhancedPrefix{ - LiveRoute: prefix, - Age: formatAge(prefix.LastUpdated), + Prefix: prefix.Prefix.String(), + MaskLength: prefix.Prefix.Bits(), + IPVersion: ipVersionOf(prefix.Prefix), + OriginASN: prefix.OriginASN, + } + + if prefix.Newest != nil { + enhancedPrefixes[i].LastUpdated = *prefix.Newest + enhancedPrefixes[i].Age = formatAge(*prefix.Newest) } // Get AS description diff --git a/internal/server/handlers_internal_test.go b/internal/server/handlers_internal_test.go index cc7872e..d2b0152 100644 --- a/internal/server/handlers_internal_test.go +++ b/internal/server/handlers_internal_test.go @@ -9,6 +9,7 @@ import ( "net/http" "net/http/httptest" "net/netip" + "reflect" "runtime" "slices" "strconv" @@ -23,6 +24,7 @@ import ( "git.eeqj.de/sneak/routewatch/internal/logger" "git.eeqj.de/sneak/routewatch/internal/metrics" "git.eeqj.de/sneak/routewatch/internal/ristypes" + "git.eeqj.de/sneak/routewatch/internal/routetable" "git.eeqj.de/sneak/routewatch/internal/streamer" "git.eeqj.de/sneak/routewatch/pkg/asinfo" ) @@ -35,6 +37,38 @@ const ( testPeerV6 = "2001:db8::1" ) +// testPeer returns the address of the nth of the IPv4 peers that tests +// announce routes from: 198.51.100.n. +func testPeer(n int) string { + return "198.51.100." + strconv.Itoa(n) +} + +// testRouteTime is the time the tests' routes are announced at, or a number of +// minutes after it. +func testRouteTime(minutes int) time.Time { + return time.Date(2026, 1, 2, 3, 4, 5, 0, time.UTC). + Add(time.Duration(minutes) * time.Minute) +} + +// announce has peer announce prefix in table at time at, with an AS path +// that holds only the origin AS asn. +func announce( + table *routetable.Table, prefix string, asn int, peer string, at time.Time, +) { + table.Announce(netip.MustParsePrefix(prefix), asn, + routetable.Route{Peer: peer, ASPath: []int{asn}, Time: at}) +} + +// withdraw has peer withdraw prefix in table. +func withdraw(table *routetable.Table, prefix, peer string) { + table.Withdraw(netip.MustParsePrefix(prefix), peer) +} + +// emptyTable returns a routing table that holds no route. +func emptyTable() *routetable.Table { + return routetable.New(routetable.Params{}) +} + // Paths several tests request. const ( asPagePath = "/as/64500" @@ -76,7 +110,8 @@ func (d blockingStatsDB) GetStatsContext(_ context.Context) (database.Stats, err func TestStatsHandlersDoNotLeakOnTimeout(t *testing.T) { release := make(chan struct{}) db := blockingStatsDB{release: release} - s := New(db, streamer.New(logger.New(), metrics.New()), logger.New(), &config.Config{}) + s := New(db, emptyTable(), streamer.New(logger.New(), metrics.New()), + logger.New(), &config.Config{}) handlers := map[string]http.HandlerFunc{ statusJSONHandlerName: s.handleStatusJSON(), @@ -108,51 +143,78 @@ func TestStatsHandlersDoNotLeakOnTimeout(t *testing.T) { } } +// tableStats holds the figures of a stats answer that come from the routing +// table. +// +//nolint:tagliatelle // field names are the existing JSON API +type tableStats struct { + Prefixes int `json:"prefixes"` + IPv4Prefixes int `json:"ipv4_prefixes"` + IPv6Prefixes int `json:"ipv6_prefixes"` + LiveRoutes int `json:"live_routes"` + IPv4Routes int `json:"ipv4_routes"` + IPv6Routes int `json:"ipv6_routes"` + OriginsWithoutKeptRoute int `json:"origins_without_kept_route"` + OldestRoute time.Time `json:"oldest_route"` + NewestRoute time.Time `json:"newest_route"` + + IPv4Distribution []database.PrefixDistribution `json:"ipv4_prefix_distribution"` + IPv6Distribution []database.PrefixDistribution `json:"ipv6_prefix_distribution"` +} + // TestStatsHandlersAnswerFromMemory checks that both stats handlers answer 200 -// with the live route counts and the prefix distribution while the database is -// closed, so that any query would fail: the request path reads them from -// memory. The prefix distribution query it used to run read every live route -// and, on a large database, took the whole 4-second deadline, so +// with the routing table's figures while the database is closed, so that any +// query would fail. The prefix distribution query they used to run read every +// live route and, on a large database, took the whole 4-second deadline, so // /api/v1/stats answered 500 (https://git.eeqj.de/sneak/routewatch/issues/30). -// The oldest and newest route times still come from one-row lookups at the ends -// of an index; with the database closed they are left out of the answer. +// +// The table holds two IPv4 prefixes and an IPv6 one. Of the three peers that +// carried 203.0.113.0/24, the two with kept routes withdrew it, so its origin +// keeps no route. The oldest and newest routes are those of the other two +// prefixes. func TestStatsHandlersAnswerFromMemory(t *testing.T) { t.Parallel() + const withdrawnPrefix = "203.0.113.0/24" + + table := emptyTable() + announce(table, testPrefixV4, 64502, testPeer(1), testRouteTime(0)) + + for n := 1; n <= 3; n++ { + announce(table, withdrawnPrefix, 64500, testPeer(n), testRouteTime(n)) + } + + withdraw(table, withdrawnPrefix, testPeer(2)) + withdraw(table, withdrawnPrefix, testPeer(3)) + announce(table, testPrefixV6, 64501, testPeerV6, testRouteTime(4)) + db, err := database.New(&config.Config{StateDir: t.TempDir()}, logger.New()) if err != nil { t.Fatalf("database.New: %v", err) } - ts := time.Date(2026, 1, 2, 3, 4, 5, 0, time.UTC) - - err = db.UpsertLiveRouteBatch([]*database.LiveRoute{ - { - Prefix: "198.51.100.0/24", MaskLength: 24, IPVersion: 4, - OriginASN: 64500, PeerIP: "192.0.2.1", ASPath: []int{64500}, - NextHop: "192.0.2.1", LastUpdated: ts, - }, - { - Prefix: "2001:db8::/32", MaskLength: 32, IPVersion: 6, - OriginASN: 64501, PeerIP: testPeerV6, ASPath: []int{64501}, - NextHop: testPeerV6, LastUpdated: ts, - }, - }) - if err != nil { - t.Fatalf("UpsertLiveRouteBatch: %v", err) - } - err = db.Close() if err != nil { t.Fatalf("Close: %v", err) } - s := New(db, streamer.New(logger.New(), metrics.New()), logger.New(), &config.Config{}) + s := New(db, table, streamer.New(logger.New(), metrics.New()), logger.New(), + &config.Config{}) handlers := map[string]http.HandlerFunc{ statusJSONHandlerName: s.handleStatusJSON(), statsHandlerName: s.handleStats(), } + want := tableStats{ + Prefixes: 3, IPv4Prefixes: 2, IPv6Prefixes: 1, + LiveRoutes: 3, IPv4Routes: 2, IPv6Routes: 1, + OriginsWithoutKeptRoute: 1, + OldestRoute: testRouteTime(0), + NewestRoute: testRouteTime(4), + IPv4Distribution: []database.PrefixDistribution{{MaskLength: 24, Count: 2}}, + IPv6Distribution: []database.PrefixDistribution{{MaskLength: 32, Count: 1}}, + } + for name, handler := range handlers { rec := httptest.NewRecorder() handler(rec, httptest.NewRequestWithContext(t.Context(), http.MethodGet, "/", nil)) @@ -164,14 +226,8 @@ func TestStatsHandlersAnswerFromMemory(t *testing.T) { continue } - //nolint:tagliatelle // field names are the existing JSON API var body struct { - Data struct { - IPv4Routes int `json:"ipv4_routes"` - IPv6Routes int `json:"ipv6_routes"` - IPv4Distribution []database.PrefixDistribution `json:"ipv4_prefix_distribution"` - IPv6Distribution []database.PrefixDistribution `json:"ipv6_prefix_distribution"` - } `json:"data"` + Data tableStats `json:"data"` } err := json.Unmarshal(rec.Body.Bytes(), &body) @@ -179,21 +235,16 @@ func TestStatsHandlersAnswerFromMemory(t *testing.T) { t.Fatalf("%s: decoding the answer: %v", name, err) } - if body.Data.IPv4Routes != 1 || body.Data.IPv6Routes != 1 { - t.Errorf("%s: routes = (v4 %d, v6 %d), want (1, 1)", - name, body.Data.IPv4Routes, body.Data.IPv6Routes) + got := body.Data + if !got.OldestRoute.Equal(want.OldestRoute) || + !got.NewestRoute.Equal(want.NewestRoute) { + t.Errorf("%s: routes from %v to %v, want from %v to %v", name, + got.OldestRoute, got.NewestRoute, want.OldestRoute, want.NewestRoute) } - wantV4 := []database.PrefixDistribution{{MaskLength: 24, Count: 1}} - if !slices.Equal(body.Data.IPv4Distribution, wantV4) { - t.Errorf("%s: IPv4 distribution = %v, want %v", - name, body.Data.IPv4Distribution, wantV4) - } - - wantV6 := []database.PrefixDistribution{{MaskLength: 32, Count: 1}} - if !slices.Equal(body.Data.IPv6Distribution, wantV6) { - t.Errorf("%s: IPv6 distribution = %v, want %v", - name, body.Data.IPv6Distribution, wantV6) + got.OldestRoute, got.NewestRoute = want.OldestRoute, want.NewestRoute + if !reflect.DeepEqual(got, want) { + t.Errorf("%s: figures %+v, want %+v", name, got, want) } } } @@ -243,11 +294,29 @@ func TestStatsAnswerCarriesChurnPerMinute(t *testing.T) { } } -// newServerWithTestRoutes returns a server on a new database that holds a -// route for testPrefixV4 and one for testPrefixV6. +// newTestTable returns a routing table that holds a route for testPrefixV4, +// with origin AS 64500, and one for testPrefixV6, with origin AS 64501. +func newTestTable() *routetable.Table { + table := emptyTable() + announce(table, testPrefixV4, 64500, testPeerV4, testRouteTime(0)) + announce(table, testPrefixV6, 64501, testPeerV6, testRouteTime(0)) + + return table +} + +// newServerWithTestRoutes returns a server on a new database that reads the +// routes of newTestTable. func newServerWithTestRoutes(t *testing.T) *Server { t.Helper() + return newServerWithTable(t, newTestTable()) +} + +// newServerWithTable returns a server on a new database that reads routes from +// table. +func newServerWithTable(t *testing.T, table *routetable.Table) *Server { + t.Helper() + db, err := database.New(&config.Config{StateDir: t.TempDir()}, logger.New()) if err != nil { t.Fatalf("database.New: %v", err) @@ -255,25 +324,7 @@ func newServerWithTestRoutes(t *testing.T) *Server { t.Cleanup(func() { _ = db.Close() }) - ts := time.Date(2026, 1, 2, 3, 4, 5, 0, time.UTC) - - err = db.UpsertLiveRouteBatch([]*database.LiveRoute{ - { - Prefix: testPrefixV4, MaskLength: 24, IPVersion: 4, - OriginASN: 64500, PeerIP: testPeerV4, ASPath: []int{64500}, - NextHop: testPeerV4, LastUpdated: ts, - }, - { - Prefix: testPrefixV6, MaskLength: 32, IPVersion: 6, - OriginASN: 64501, PeerIP: testPeerV6, ASPath: []int{64501}, - NextHop: testPeerV6, LastUpdated: ts, - }, - }) - if err != nil { - t.Fatalf("UpsertLiveRouteBatch: %v", err) - } - - return New(db, streamer.New(logger.New(), metrics.New()), logger.New(), + return New(db, table, streamer.New(logger.New(), metrics.New()), logger.New(), &config.Config{}) } @@ -402,34 +453,20 @@ func TestPrefixDetailShowsOriginNames(t *testing.T) { t.Cleanup(func() { _ = db.Close() }) - ts := time.Date(2026, 1, 2, 3, 4, 5, 0, time.UTC) origins := []int{15169, 13335} + table := emptyTable() - for _, asn := range origins { - _, err := db.GetOrCreateASN(asn, ts) + for i, asn := range origins { + _, err := db.GetOrCreateASN(asn, testRouteTime(0)) if err != nil { t.Fatalf("GetOrCreateASN(%d): %v", asn, err) } + + announce(table, testPrefixV4, asn, testPeer(i+1), testRouteTime(0)) } - err = db.UpsertLiveRouteBatch([]*database.LiveRoute{ - { - Prefix: testPrefixV4, MaskLength: 24, IPVersion: 4, - OriginASN: origins[0], PeerIP: testPeerV4, - ASPath: []int{origins[0]}, NextHop: testPeerV4, LastUpdated: ts, - }, - { - Prefix: testPrefixV4, MaskLength: 24, IPVersion: 4, - OriginASN: origins[1], PeerIP: "198.51.100.2", - ASPath: []int{origins[1]}, NextHop: "198.51.100.2", LastUpdated: ts, - }, - }) - if err != nil { - t.Fatalf("UpsertLiveRouteBatch: %v", err) - } - - s := New(failingASDetailsDB{db}, streamer.New(logger.New(), metrics.New()), - logger.New(), &config.Config{}) + s := New(failingASDetailsDB{db}, table, + streamer.New(logger.New(), metrics.New()), logger.New(), &config.Config{}) rec := httptest.NewRecorder() s.router.ServeHTTP(rec, httptest.NewRequestWithContext( t.Context(), http.MethodGet, "/prefix/"+testPrefixV4, nil)) @@ -466,23 +503,14 @@ func TestPrefixDetailShowsOriginNames(t *testing.T) { func TestPagesSayOriginZeroIsNoKnownOrigin(t *testing.T) { t.Parallel() - db, err := database.New(&config.Config{StateDir: t.TempDir()}, logger.New()) - if err != nil { - t.Fatalf("database.New: %v", err) - } + table := emptyTable() + table.Announce(netip.MustParsePrefix(testPrefixV4), ristypes.NoOrigin, + routetable.Route{ + Peer: testPeerV4, ASPath: []int{64500, 64501, 64502}, + Time: testRouteTime(0), + }) - t.Cleanup(func() { _ = db.Close() }) - - err = db.UpsertLiveRouteBatch([]*database.LiveRoute{{ - Prefix: testPrefixV4, MaskLength: 24, IPVersion: 4, - OriginASN: 0, PeerIP: testPeerV4, ASPath: []int{64500, 64501, 64502}, - NextHop: testPeerV4, LastUpdated: time.Date(2026, 1, 2, 3, 4, 5, 0, time.UTC), - }}) - if err != nil { - t.Fatalf("UpsertLiveRouteBatch: %v", err) - } - - s := New(db, streamer.New(logger.New(), metrics.New()), logger.New(), &config.Config{}) + s := newServerWithTable(t, table) for _, path := range []string{"/prefix/" + testPrefixV4, prefixLengthPagePath} { rec := httptest.NewRecorder() @@ -533,8 +561,9 @@ func TestIndexShowsTheClientAddress(t *testing.T) { } for _, tt := range tests { - // The home page reads nothing from the database, so the server has none. - s := New(nil, streamer.New(logger.New(), metrics.New()), logger.New(), + // The home page reads no route and nothing from the database, so the + // server has neither a routing table nor a database. + s := New(nil, nil, streamer.New(logger.New(), metrics.New()), logger.New(), &config.Config{TrustedProxies: tt.trustedProxies}) req := httptest.NewRequestWithContext(t.Context(), http.MethodGet, "/", nil) @@ -571,13 +600,6 @@ func TestIndexShowsTheClientAddress(t *testing.T) { func TestPagesShowAgesInWords(t *testing.T) { t.Parallel() - db, err := database.New(&config.Config{StateDir: t.TempDir()}, logger.New()) - if err != nil { - t.Fatalf("database.New: %v", err) - } - - t.Cleanup(func() { _ = db.Close() }) - // Each age is well past a whole number of its unit, so the time the test // takes to run does not change the words. ages := []struct { @@ -591,22 +613,13 @@ func TestPagesShowAgesInWords(t *testing.T) { } now := time.Now().UTC() - routes := make([]*database.LiveRoute, 0, len(ages)) + table := emptyTable() for _, a := range ages { - routes = append(routes, &database.LiveRoute{ - Prefix: a.prefix, MaskLength: 24, IPVersion: 4, - OriginASN: 64500, PeerIP: testPeerV4, ASPath: []int{64500}, - NextHop: testPeerV4, LastUpdated: now.Add(-a.age), - }) + announce(table, a.prefix, 64500, testPeerV4, now.Add(-a.age)) } - err = db.UpsertLiveRouteBatch(routes) - if err != nil { - t.Fatalf("UpsertLiveRouteBatch: %v", err) - } - - s := New(db, streamer.New(logger.New(), metrics.New()), logger.New(), &config.Config{}) + s := newServerWithTable(t, table) for _, a := range ages { cell := `` + a.want + `` @@ -643,7 +656,7 @@ const shortRequestDeadline = 50 * time.Millisecond func TestRequestPastTimeLimitAnswers504(t *testing.T) { t.Parallel() - s := New(nil, streamer.New(logger.New(), metrics.New()), logger.New(), + s := New(nil, nil, streamer.New(logger.New(), metrics.New()), logger.New(), &config.Config{}) waitForTimeLimit := func(_ http.ResponseWriter, r *http.Request) { @@ -693,8 +706,8 @@ func TestStatsHandlersAnswer504WhenDatabaseReadRunsOut(t *testing.T) { defer close(release) db := blockingStatsDB{release: release} - s := New(db, streamer.New(logger.New(), metrics.New()), logger.New(), - &config.Config{}) + s := New(db, emptyTable(), streamer.New(logger.New(), metrics.New()), + logger.New(), &config.Config{}) handlers := map[string]http.HandlerFunc{ statusJSONHandlerName: s.handleStatusJSON(), @@ -720,10 +733,9 @@ func TestStatsHandlersAnswer504WhenDatabaseReadRunsOut(t *testing.T) { // slowDB embeds database.Store (left nil) and overrides the reads the routes // in TestRoutesAnswer504WhenDatabaseReadIsCutOff make. Each read waits until // its context ends and then answers as the real store does when the time -// limit cuts it off: the IP read with the context's error, the prefix read -// that there is no route, and the others with what they had read, here -// nothing, and no error. Every other Store method is unused and would panic -// if called. +// limit cuts it off: the stats read with what it had read, here nothing, and +// no error, and the others with the context's error. Every other Store method +// is unused and would panic if called. type slowDB struct { database.Store } @@ -734,66 +746,44 @@ func (slowDB) GetStatsContext(ctx context.Context) (database.Stats, error) { return database.Stats{}, nil } -func (slowDB) GetIPInfoContext( - ctx context.Context, _ string, -) (*database.IPInfo, error) { +func (slowDB) GetAS(ctx context.Context, _ int) (*database.ASN, bool, error) { <-ctx.Done() - return nil, ctx.Err() + return nil, false, ctx.Err() } -func (slowDB) GetASDetailsContext( - ctx context.Context, asn int, -) (*database.ASN, []database.LiveRoute, error) { +func (slowDB) GetPrefixFirstSeen( + ctx context.Context, _ netip.Prefix, +) (time.Time, error) { <-ctx.Done() - return &database.ASN{ASN: asn}, nil, nil -} - -func (slowDB) GetPrefixDetailsContext( - ctx context.Context, _ string, -) ([]database.LiveRoute, map[int]database.ASN, error) { - <-ctx.Done() - - return nil, nil, database.ErrNoRoute -} - -func (slowDB) GetRandomPrefixesByLengthContext( - ctx context.Context, _, _, _ int, -) ([]database.LiveRoute, error) { - <-ctx.Done() - - return nil, nil + return time.Time{}, ctx.Err() } // TestRoutesAnswer504WhenDatabaseReadIsCutOff sends a request through the // server's router for each route that reads the database, with a store whose // reads are cut off when the request's time limit passes. Each route answers // 504 with the time limit message. Without its check after the read, a route -// answers with what the cut-off read returned: 500, 404, 200, or, on a page -// that stops before rendering once the request has ended, 504 with an empty -// body. The health check is left out: it answers 503 when its own 2-second -// ping runs out. +// answers with what the cut-off read returned: 500 or 200. The health check is +// left out: it answers 503 when its own 2-second ping runs out. The prefix +// length pages and the prefix API read only the routing table. func TestRoutesAnswer504WhenDatabaseReadIsCutOff(t *testing.T) { t.Parallel() - s := New(slowDB{}, streamer.New(logger.New(), metrics.New()), logger.New(), - &config.Config{}) + s := New(slowDB{}, newTestTable(), streamer.New(logger.New(), metrics.New()), + logger.New(), &config.Config{}) tests := []struct { path string wantJSONBody bool }{ {path: asPagePath}, - {path: "/prefix/192.0.2.0/24"}, - {path: prefixLengthPagePath}, - {path: "/prefixlength6/48"}, + {path: "/prefix/" + testPrefixV4}, {path: statusJSONPath, wantJSONBody: true}, - {path: "/ip/192.0.2.1", wantJSONBody: true}, + {path: "/ip/" + addrInTestPrefixV4, wantJSONBody: true}, {path: statsPath, wantJSONBody: true}, - {path: "/api/v1/ip/192.0.2.1", wantJSONBody: true}, + {path: "/api/v1/ip/" + addrInTestPrefixV4, wantJSONBody: true}, {path: "/api/v1/as/64500", wantJSONBody: true}, - {path: "/api/v1/prefix/192.0.2.0/24", wantJSONBody: true}, } for _, tt := range tests { @@ -810,10 +800,8 @@ type slowSecondReadDB struct { database.Store } -func (slowSecondReadDB) GetASDetailsContext( - _ context.Context, asn int, -) (*database.ASN, []database.LiveRoute, error) { - return &database.ASN{ASN: asn}, nil, nil +func (slowSecondReadDB) GetAS(_ context.Context, asn int) (*database.ASN, bool, error) { + return &database.ASN{ASN: asn}, true, nil } func (slowSecondReadDB) GetASPeersContext( @@ -855,8 +843,8 @@ func (idleASNFetcher) GetStats() ASNFetcherStats { func TestRoutesAnswer504WhenSecondDatabaseReadIsCutOff(t *testing.T) { t.Parallel() - s := New(slowSecondReadDB{}, streamer.New(logger.New(), metrics.New()), - logger.New(), &config.Config{}) + s := New(slowSecondReadDB{}, emptyTable(), + streamer.New(logger.New(), metrics.New()), logger.New(), &config.Config{}) s.SetASNFetcher(idleASNFetcher{}) tests := []struct { @@ -950,34 +938,27 @@ func (failingDB) GetStatsContext(_ context.Context) (database.Stats, error) { return database.Stats{}, errDiskIO } -func (failingDB) GetIPInfoContext( - _ context.Context, _ string, -) (*database.IPInfo, error) { - return nil, errDiskIO +func (failingDB) GetAS(_ context.Context, _ int) (*database.ASN, bool, error) { + return nil, false, errDiskIO } -func (failingDB) GetASDetailsContext( - _ context.Context, _ int, -) (*database.ASN, []database.LiveRoute, error) { - return nil, nil, errDiskIO +func (failingDB) GetPrefixFirstSeen( + _ context.Context, _ netip.Prefix, +) (time.Time, error) { + return time.Time{}, errDiskIO } -func (failingDB) GetPrefixDetailsContext( - _ context.Context, _ string, -) ([]database.LiveRoute, map[int]database.ASN, error) { - return nil, nil, errDiskIO -} - -// TestJSONRoutesDoNotSendErrorDetails sends a request to each JSON route and -// the health check with a store whose reads fail with the driver's error text, -// and a prefix that does not parse, and checks that no answer carries the -// driver's or the prefix parser's text. Before the fix the answers sent it to -// the client (https://git.eeqj.de/sneak/routewatch/issues/108). +// TestJSONRoutesDoNotSendErrorDetails sends a request to each JSON route that +// reads the database, and to the health check, with a store whose reads fail +// with the driver's error text. It also sends a prefix that does not parse. +// No answer may carry the driver's or the prefix parser's text. Before the fix +// the answers sent it to the client +// (https://git.eeqj.de/sneak/routewatch/issues/108). func TestJSONRoutesDoNotSendErrorDetails(t *testing.T) { t.Parallel() - s := New(failingDB{}, streamer.New(logger.New(), metrics.New()), logger.New(), - &config.Config{}) + s := New(failingDB{}, newTestTable(), streamer.New(logger.New(), metrics.New()), + logger.New(), &config.Config{}) tests := []struct { path string @@ -987,8 +968,7 @@ func TestJSONRoutesDoNotSendErrorDetails(t *testing.T) { {statusJSONPath, http.StatusInternalServerError}, {statsPath, http.StatusInternalServerError}, {"/api/v1/as/64500", http.StatusInternalServerError}, - {"/api/v1/prefix/192.0.2.0/24", http.StatusInternalServerError}, - {"/api/v1/ip/192.0.2.1", http.StatusInternalServerError}, + {"/api/v1/ip/" + addrInTestPrefixV4, http.StatusInternalServerError}, {"/api/v1/prefix/foo/24", http.StatusBadRequest}, } @@ -1011,45 +991,47 @@ func TestJSONRoutesDoNotSendErrorDetails(t *testing.T) { } } -// The addresses failingIPLookupDB fails the read of and finds a route for. +// The address whose read failingIPLookupDB fails, in testPrefixV4, and one it +// reads, in testPrefixV6. const ( - failingLookupIP = "192.0.2.1" - routedLookupIP = "192.0.2.2" + failingLookupIP = addrInTestPrefixV4 + routedLookupIP = addrInTestPrefixV6 ) -// failingIPLookupDB embeds database.Store (left nil) and answers the IP lookup -// with errDiskIO for failingLookupIP, a route for routedLookupIP, and no route -// for any other address. Every other Store method is unused and would panic if -// called. +// failingIPLookupDB embeds database.Store (left nil). It fails the read of +// testPrefixV4's first-seen time with errDiskIO, and answers the IP lookup's +// other reads with nothing stored. Every other Store method is unused and +// would panic if called. type failingIPLookupDB struct { database.Store } -func (failingIPLookupDB) GetIPInfoContext( - _ context.Context, ip string, -) (*database.IPInfo, error) { - switch ip { - case failingLookupIP: - return nil, errDiskIO - case routedLookupIP: - return &database.IPInfo{IP: ip}, nil - default: - return nil, database.ErrNoRoute +func (failingIPLookupDB) GetPrefixFirstSeen( + _ context.Context, prefix netip.Prefix, +) (time.Time, error) { + if prefix == netip.MustParsePrefix(testPrefixV4) { + return time.Time{}, errDiskIO } + + return time.Time{}, nil +} + +func (failingIPLookupDB) GetASWHOIS(_ context.Context, asn int) (*database.ASN, error) { + return &database.ASN{ASN: asn}, nil } // TestIPLookupAnswers500WhenDatabaseReadFails sends IP lookups through the -// server's router with a store whose read of one address fails with the +// server's router with a store whose read for one address fails with the // driver's error. A lookup that asks for that address answers 500 with the -// JSON error body, as the AS and prefix JSON routes do; an address with no -// route answers 200. Before the fix the failed read was reported as if the -// address had no route: 404 for that address alone, and 200 beside a routed -// address (https://git.eeqj.de/sneak/routewatch/issues/109). +// JSON error body, as the AS JSON route does; an address with no route answers +// 200. Before the fix the failed read was reported as if the address had no +// route: 404 for that address alone, and 200 beside a routed address +// (https://git.eeqj.de/sneak/routewatch/issues/109). func TestIPLookupAnswers500WhenDatabaseReadFails(t *testing.T) { t.Parallel() - s := New(failingIPLookupDB{}, streamer.New(logger.New(), metrics.New()), - logger.New(), &config.Config{}) + s := New(failingIPLookupDB{}, newTestTable(), + streamer.New(logger.New(), metrics.New()), logger.New(), &config.Config{}) s.resolver = &fakeResolver{} tests := []struct { @@ -1122,30 +1104,37 @@ func (r *fakeResolver) LookupAddr(ctx context.Context, _ string) ([]string, erro return nil, errNoSuchHost } -// countingIPLookupDB embeds database.Store (left nil) and counts its IP -// lookups, answering each that the address has no route. Every other Store -// method is unused and would panic if called. +// countingIPLookupDB embeds database.Store (left nil) and answers the IP +// lookup's reads with nothing stored. It counts the lookups, as the reads of a +// prefix's first-seen time. Every other Store method is unused and would panic +// if called. type countingIPLookupDB struct { database.Store lookups atomic.Int64 } -func (d *countingIPLookupDB) GetIPInfoContext( - _ context.Context, _ string, -) (*database.IPInfo, error) { +func (d *countingIPLookupDB) GetPrefixFirstSeen( + _ context.Context, _ netip.Prefix, +) (time.Time, error) { d.lookups.Add(1) - return nil, database.ErrNoRoute + return time.Time{}, nil +} + +func (d *countingIPLookupDB) GetASWHOIS( + _ context.Context, asn int, +) (*database.ASN, error) { + return &database.ASN{ASN: asn}, nil } // TestIPLookupRefusesTooManyAddresses sends IP lookups of maxLookupTargets -// addresses and of one more, listed or as a host name that resolves to them, -// and a lookup listing maxLookupTargets+1 host names. The longer lookups answer -// 400 without reading the database, and the list of host names without -// resolving any of them; the others read the database once per address. Before -// the fix every address was read and every name resolved, however many there -// were (https://git.eeqj.de/sneak/routewatch/issues/110). +// addresses in testPrefixV4 and of one more, listed or as a host name that +// resolves to them, and a lookup listing maxLookupTargets+1 host names. The +// longer lookups answer 400 without reading the database, and the list of host +// names without resolving any of them; the others read the database for each +// address. Before the fix every address was read and every name resolved, +// however many there were (https://git.eeqj.de/sneak/routewatch/issues/110). func TestIPLookupRefusesTooManyAddresses(t *testing.T) { t.Parallel() @@ -1180,8 +1169,8 @@ func TestIPLookupRefusesTooManyAddresses(t *testing.T) { for _, tt := range tests { db := &countingIPLookupDB{} - s := New(db, streamer.New(logger.New(), metrics.New()), logger.New(), - &config.Config{}) + s := New(db, newTestTable(), streamer.New(logger.New(), metrics.New()), + logger.New(), &config.Config{}) resolver := &fakeResolver{hosts: hosts} s.resolver = resolver @@ -1901,16 +1890,11 @@ func TestNoOriginHasNoASAnswers(t *testing.T) { const noOriginPrefix = "203.0.113.0/24" s := newServerWithTestRoutes(t) - - err := s.db.UpsertLiveRouteBatch([]*database.LiveRoute{{ - Prefix: noOriginPrefix, MaskLength: 24, IPVersion: 4, - OriginASN: ristypes.NoOrigin, PeerIP: testPeerV4, - ASPath: []int{64500, 64501, 64502}, NextHop: testPeerV4, - LastUpdated: time.Date(2026, 1, 2, 3, 4, 5, 0, time.UTC), - }}) - if err != nil { - t.Fatalf("UpsertLiveRouteBatch: %v", err) - } + s.table.Announce(netip.MustParsePrefix(noOriginPrefix), ristypes.NoOrigin, + routetable.Route{ + Peer: testPeerV4, ASPath: []int{64500, 64501, 64502}, + Time: testRouteTime(0), + }) asn := strconv.Itoa(ristypes.NoOrigin) diff --git a/internal/server/handlers_table_internal_test.go b/internal/server/handlers_table_internal_test.go new file mode 100644 index 0000000..515a163 --- /dev/null +++ b/internal/server/handlers_table_internal_test.go @@ -0,0 +1,368 @@ +package server + +import ( + "encoding/json" + "net/http" + "net/http/httptest" + "slices" + "strconv" + "strings" + "testing" + "time" + + "git.eeqj.de/sneak/routewatch/internal/database" + "git.eeqj.de/sneak/routewatch/internal/routetable" +) + +// The origin ASes in the routing table that newTwoOriginTable fills. +const ( + mostPeersASN = 64500 + fewerPeersASN = 64501 + v6OriginASN = 64502 +) + +// noKeptRoutePrefix is the prefix of newTwoOriginTable whose origin keeps no +// route. +const noKeptRoutePrefix = "198.51.100.0/24" + +// routeTimeFormat is how the pages write a route's time. +const routeTimeFormat = "2006-01-02 15:04:05" + +// newTwoOriginTable returns a routing table that holds: +// - testPrefixV4 with two origin ASes. Peers 1, 2 and 3 carry it with +// mostPeersASN, which keeps the routes of peers 3 and 2. Peer 4 carries +// it with fewerPeersASN, in the prefix's newest route. +// - noKeptRoutePrefix with mostPeersASN. Only peer 7 carries it, and its +// route is not kept: peers 5 and 6, whose routes were, withdrew it. +// - testPrefixV6 with v6OriginASN. +func newTwoOriginTable() *routetable.Table { + table := emptyTable() + + for n := 1; n <= 3; n++ { + announce(table, testPrefixV4, mostPeersASN, testPeer(n), testRouteTime(n)) + } + + announce(table, testPrefixV4, fewerPeersASN, testPeer(4), testRouteTime(5)) + + announce(table, noKeptRoutePrefix, mostPeersASN, testPeer(5), testRouteTime(4)) + announce(table, noKeptRoutePrefix, mostPeersASN, testPeer(6), testRouteTime(6)) + announce(table, noKeptRoutePrefix, mostPeersASN, testPeer(7), testRouteTime(0)) + withdraw(table, noKeptRoutePrefix, testPeer(5)) + withdraw(table, noKeptRoutePrefix, testPeer(6)) + + announce(table, testPrefixV6, v6OriginASN, testPeerV6, testRouteTime(0)) + + return table +} + +// get sends a GET for path through s's router and fails the test unless it +// answers 200. +func get(t *testing.T, s *Server, path string) *httptest.ResponseRecorder { + t.Helper() + + rec := httptest.NewRecorder() + s.router.ServeHTTP(rec, httptest.NewRequestWithContext( + t.Context(), http.MethodGet, path, nil)) + + if rec.Code != http.StatusOK { + t.Fatalf("%s: status %d, want %d; body %s", + path, rec.Code, http.StatusOK, rec.Body) + } + + return rec +} + +// TestIPLookupGivesTheOriginMostPeersCarry looks up an address in a prefix +// with two origin ASes, through each IP lookup route. The answer names the +// origin AS that more peers carry, counts every peer, and gives the time of +// the prefix's newest kept route. The prefix's first-seen time and the AS's +// WHOIS details come from the database. +func TestIPLookupGivesTheOriginMostPeersCarry(t *testing.T) { + t.Parallel() + + s := newServerWithTable(t, newTwoOriginTable()) + s.resolver = &fakeResolver{} + + firstSeen := testRouteTime(-1) + + err := s.db.UpdatePrefixesBatch(map[string]time.Time{testPrefixV4: firstSeen}) + if err != nil { + t.Fatalf("UpdatePrefixesBatch: %v", err) + } + + _, err = s.db.GetOrCreateASN(mostPeersASN, firstSeen) + if err != nil { + t.Fatalf("GetOrCreateASN: %v", err) + } + + whois := &database.ASNWHOISUpdate{ + ASN: mostPeersASN, ASName: "TEST-AS", CountryCode: "DE", + } + + err = s.db.UpdateASNWHOIS(t.Context(), whois) + if err != nil { + t.Fatalf("UpdateASNWHOIS: %v", err) + } + + // ownAddressPath looks up the request's own address, which httptest + // gives as addrInTestPrefixV4. + for _, path := range []string{ + "/api/v1/ip/" + addrInTestPrefixV4, "/ip/" + addrInTestPrefixV4, + ownAddressPath, + } { + var body struct { + Data IPLookupResponse `json:"data"` + } + + err := json.Unmarshal(get(t, s, path).Body.Bytes(), &body) + if err != nil || len(body.Data.Results) != 1 || + body.Data.Results[0].IPInfo == nil { + t.Fatalf("%s: answer %+v (%v), want one result with a route", + path, body.Data, err) + } + + got := body.Data.Results[0].IPInfo + if got.Netblock != testPrefixV4 || got.ASN != mostPeersASN || + got.NumPeers != 4 || !got.LastSeen.Equal(testRouteTime(5)) || + !got.FirstSeen.Equal(firstSeen) || got.ASName != whois.ASName || + got.CountryCode != whois.CountryCode { + t.Errorf("%s: route %+v, want %s from AS%d, 4 peers, last seen %v, "+ + "first seen %v, %s in %s", path, *got, testPrefixV4, + mostPeersASN, testRouteTime(5), firstSeen, whois.ASName, + whois.CountryCode) + } + } +} + +// prefixRoute is one route of the prefix API's answer. A field the answer +// leaves out stays nil. +// +//nolint:tagliatelle // field names are the existing JSON API +type prefixRoute struct { + OriginASN int `json:"origin_asn"` + PeerIP string `json:"peer_ip"` + ASPath []int `json:"as_path"` + NextHop *string `json:"next_hop"` + LastUpdated *time.Time `json:"last_updated"` +} + +// TestPrefixAnswersListEveryPeer asks the prefix API and page for a prefix +// with two origin ASes, and for one whose origin keeps no route. Each answer +// lists every peer that carries the prefix, by origin AS. Only the kept routes +// have an AS path, a next hop and a time, newest first. The other peers follow +// without them. +func TestPrefixAnswersListEveryPeer(t *testing.T) { + t.Parallel() + + s := newServerWithTable(t, newTwoOriginTable()) + + // keptAt[peer] is the time of the peer's kept route; a peer not in it is + // listed without a route. + keptAt := map[string]time.Time{ + testPeer(3): testRouteTime(3), + testPeer(2): testRouteTime(2), + testPeer(4): testRouteTime(5), + } + + tests := []struct { + prefix string + wantPeers []string + wantOrigins []int + }{ + { + testPrefixV4, + []string{testPeer(3), testPeer(2), testPeer(1), testPeer(4)}, + []int{mostPeersASN, mostPeersASN, mostPeersASN, fewerPeersASN}, + }, + {noKeptRoutePrefix, []string{testPeer(7)}, []int{mostPeersASN}}, + } + + for _, tt := range tests { + //nolint:tagliatelle // field names are the existing JSON API + var body struct { + Data struct { + Routes []prefixRoute `json:"routes"` + PeerCount int `json:"peer_count"` + OriginCount int `json:"origin_count"` + } `json:"data"` + } + + path := "/api/v1/prefix/" + tt.prefix + + err := json.Unmarshal(get(t, s, path).Body.Bytes(), &body) + if err != nil { + t.Fatalf("%s: decoding the answer: %v", path, err) + } + + wantOriginCount := len(slices.Compact(slices.Clone(tt.wantOrigins))) + if body.Data.PeerCount != len(tt.wantPeers) || + body.Data.OriginCount != wantOriginCount { + t.Errorf("%s: %d peers and %d origins, want %d and %d", path, + body.Data.PeerCount, body.Data.OriginCount, len(tt.wantPeers), + wantOriginCount) + } + + checkPrefixRoutes(t, path, body.Data.Routes, tt.wantPeers, tt.wantOrigins, + keptAt) + checkPrefixPage(t, s, tt.prefix, tt.wantPeers, keptAt) + } +} + +// checkPrefixRoutes checks that routes come from wantPeers, in that order, +// with wantOrigins as their origin ASes. The route of a peer in keptAt must +// have an AS path, the peer as next hop, and the time keptAt gives. Any other +// route must have none of the three. +func checkPrefixRoutes( + t *testing.T, path string, routes []prefixRoute, wantPeers []string, + wantOrigins []int, keptAt map[string]time.Time, +) { + t.Helper() + + if len(routes) != len(wantPeers) { + t.Fatalf("%s: routes %+v, want one each from %v", path, routes, wantPeers) + } + + for i, route := range routes { + if route.PeerIP != wantPeers[i] || route.OriginASN != wantOrigins[i] { + t.Errorf("%s: route %d from %s with AS%d, want from %s with AS%d", + path, i, route.PeerIP, route.OriginASN, wantPeers[i], wantOrigins[i]) + } + + at, kept := keptAt[route.PeerIP] + if !kept { + if route.ASPath != nil || route.NextHop != nil || route.LastUpdated != nil { + t.Errorf("%s: route from %s is not kept but has %+v", + path, route.PeerIP, route) + } + + continue + } + + if route.ASPath == nil || route.NextHop == nil || + *route.NextHop != route.PeerIP || route.LastUpdated == nil || + !route.LastUpdated.Equal(at) { + t.Errorf("%s: kept route from %s is %+v, want its AS path, the peer "+ + "as next hop and the time %v", path, route.PeerIP, route, at) + } + } +} + +// checkPrefixPage checks that the page of prefix shows each of wantPeers, the +// time of each kept route among them, and no time that was never set. +func checkPrefixPage( + t *testing.T, s *Server, prefix string, wantPeers []string, + keptAt map[string]time.Time, +) { + t.Helper() + + path := "/prefix/" + prefix + page := get(t, s, path).Body.String() + + for _, peer := range wantPeers { + want := `` + peer + `` + if !strings.Contains(page, want) { + t.Errorf("%s does not list the peer %s", path, peer) + } + + at, kept := keptAt[peer] + if kept && !strings.Contains(page, at.Format(routeTimeFormat)) { + t.Errorf("%s does not show the time %v of the route from %s", + path, at, peer) + } + } + + if strings.Contains(page, zeroDate) { + t.Errorf("%s shows the date %s", path, zeroDate) + } +} + +// TestASAnswersGiveEachPrefixItsNewestKeptRoute asks the AS API and page for +// mostPeersASN. Its prefixes are testPrefixV4, whose newest kept route with +// that origin AS is peer 3's, and noKeptRoutePrefix, which keeps no route. The +// first has that route's time. The second is listed without a time. +func TestASAnswersGiveEachPrefixItsNewestKeptRoute(t *testing.T) { + t.Parallel() + + s := newServerWithTable(t, newTwoOriginTable()) + path := "/api/v1/as/" + strconv.Itoa(mostPeersASN) + + //nolint:tagliatelle // field names are the existing JSON API + var body struct { + Data struct { + IPv4Prefixes []struct { + Prefix string `json:"prefix"` + LastUpdated *time.Time `json:"last_updated"` + } `json:"ipv4_prefixes"` + } `json:"data"` + } + + err := json.Unmarshal(get(t, s, path).Body.Bytes(), &body) + if err != nil { + t.Fatalf("%s: decoding the answer: %v", path, err) + } + + prefixes := body.Data.IPv4Prefixes + if len(prefixes) != 2 || prefixes[0].Prefix != testPrefixV4 || + prefixes[0].LastUpdated == nil || + !prefixes[0].LastUpdated.Equal(testRouteTime(3)) || + prefixes[1].Prefix != noKeptRoutePrefix || prefixes[1].LastUpdated != nil { + t.Errorf("%s: prefixes %+v, want %s updated %v and %s without a time", + path, prefixes, testPrefixV4, testRouteTime(3), noKeptRoutePrefix) + } + + path = "/as/" + strconv.Itoa(mostPeersASN) + page := get(t, s, path).Body.String() + + for _, want := range []string{ + testPrefixV4, noKeptRoutePrefix, testRouteTime(3).Format(routeTimeFormat), + } { + if !strings.Contains(page, want) { + t.Errorf("%s does not show %s", path, want) + } + } + + if strings.Contains(page, zeroDate) { + t.Errorf("%s shows the date %s", path, zeroDate) + } +} + +// TestPrefixLengthPagesShowTheOriginMostPeersCarry asks for the IPv4 /24 +// page and the IPv6 /32 page. The first lists both /24 prefixes with +// mostPeersASN, the origin AS more peers carry testPrefixV4 with, and gives +// noKeptRoutePrefix no age. The second lists testPrefixV6. +func TestPrefixLengthPagesShowTheOriginMostPeersCarry(t *testing.T) { + t.Parallel() + + s := newServerWithTable(t, newTwoOriginTable()) + + page := get(t, s, prefixLengthPagePath).Body.String() + + for _, want := range []string{ + ">" + testPrefixV4 + "<", ">" + noKeptRoutePrefix + "<", + `href="/as/` + strconv.Itoa(mostPeersASN) + `"`, + } { + if !strings.Contains(page, want) { + t.Errorf("%s does not show %s", prefixLengthPagePath, want) + } + } + + if strings.Contains(page, `href="/as/`+strconv.Itoa(fewerPeersASN)+`"`) { + t.Errorf("%s links to AS%d, which fewer peers carry", + prefixLengthPagePath, fewerPeersASN) + } + + const noAge = `` + if strings.Count(page, noAge) != 1 { + t.Errorf("%s gives %d prefixes no age, want 1 (%s)", + prefixLengthPagePath, strings.Count(page, noAge), noKeptRoutePrefix) + } + + const v6PagePath = "/prefixlength6/32" + + page = get(t, s, v6PagePath).Body.String() + if !strings.Contains(page, ">"+testPrefixV6+"<") || + !strings.Contains(page, `href="/as/`+strconv.Itoa(v6OriginASN)+`"`) { + t.Errorf("%s does not show %s with AS%d", v6PagePath, testPrefixV6, + v6OriginASN) + } +} diff --git a/internal/server/server.go b/internal/server/server.go index b0cb889..6d4aadc 100644 --- a/internal/server/server.go +++ b/internal/server/server.go @@ -3,6 +3,10 @@ // - the AS, prefix and prefix length pages, and the JSON answers behind them // - the IP lookups // - the AS name search +// +// Every route it shows comes from the routing table in memory. AS names, +// WHOIS details and the times a prefix and an AS were first seen come from +// the database. package server import ( @@ -19,6 +23,7 @@ import ( "git.eeqj.de/sneak/routewatch/internal/config" "git.eeqj.de/sneak/routewatch/internal/database" "git.eeqj.de/sneak/routewatch/internal/logger" + "git.eeqj.de/sneak/routewatch/internal/routetable" "git.eeqj.de/sneak/routewatch/internal/streamer" "github.com/go-chi/chi/v5" ) @@ -55,6 +60,7 @@ type dnsResolver interface { type Server struct { router *chi.Mux db database.Store + table *routetable.Table streamer *streamer.Streamer logger *logger.Logger port int @@ -71,15 +77,18 @@ type Server struct { resolver dnsResolver } -// New creates a new HTTP server +// New creates a new HTTP server that reads routes from table, the routing +// table that the prefix handler keeps current. func New( db database.Store, + table *routetable.Table, streamer *streamer.Streamer, logger *logger.Logger, cfg *config.Config, ) *Server { s := &Server{ db: db, + table: table, streamer: streamer, logger: logger, port: cfg.Port, diff --git a/internal/server/server_test.go b/internal/server/server_test.go index 1f6964b..5bfec36 100644 --- a/internal/server/server_test.go +++ b/internal/server/server_test.go @@ -34,7 +34,7 @@ func TestStartFailsWhenPortIsHeld(t *testing.T) { port := addr.Port log := logger.New() - s := server.New(nil, streamer.New(log, metrics.New()), log, + s := server.New(nil, nil, streamer.New(log, metrics.New()), log, &config.Config{Port: port}) defer func() { _ = s.Stop(context.Background()) }() diff --git a/internal/templates/as_detail.html b/internal/templates/as_detail.html index 9fa6951..984dcc6 100644 --- a/internal/templates/as_detail.html +++ b/internal/templates/as_detail.html @@ -281,8 +281,8 @@ {{.Prefix}} /{{.MaskLength}} - {{.LastUpdated.Format "2006-01-02 15:04:05"}} - {{.LastUpdated | timeSince}} + {{if not .LastUpdated.IsZero}}{{.LastUpdated.Format "2006-01-02 15:04:05"}}{{end}} + {{if not .LastUpdated.IsZero}}{{.LastUpdated | timeSince}}{{end}} {{end}} @@ -310,8 +310,8 @@ {{.Prefix}} /{{.MaskLength}} - {{.LastUpdated.Format "2006-01-02 15:04:05"}} - {{.LastUpdated | timeSince}} + {{if not .LastUpdated.IsZero}}{{.LastUpdated.Format "2006-01-02 15:04:05"}}{{end}} + {{if not .LastUpdated.IsZero}}{{.LastUpdated | timeSince}}{{end}} {{end}} diff --git a/internal/templates/prefix_detail.html b/internal/templates/prefix_detail.html index 18fe813..de44b3f 100644 --- a/internal/templates/prefix_detail.html +++ b/internal/templates/prefix_detail.html @@ -329,8 +329,8 @@ {{.PeerIP}} {{range $i, $as := .ASPathWithHandle}}{{if $i}} → {{end}}{{if $as.Handle}}{{$as.Handle}}{{else}}AS{{$as.ASN}}{{end}}{{end}} {{.NextHop}} - {{.LastUpdated.Format "2006-01-02 15:04:05"}} - {{.LastUpdated | timeSince}} + {{if not .LastUpdated.IsZero}}{{.LastUpdated.Format "2006-01-02 15:04:05"}}{{end}} + {{if not .LastUpdated.IsZero}}{{.LastUpdated | timeSince}}{{end}} {{end}} diff --git a/internal/templates/status.html b/internal/templates/status.html index 4061f7f..b277a22 100644 --- a/internal/templates/status.html +++ b/internal/templates/status.html @@ -318,6 +318,10 @@ IPv6 Routes - +
+ Origins Without a Kept Route + - +
IPv4 Updates/sec - @@ -571,6 +575,7 @@ document.getElementById('live_routes').textContent = '-'; document.getElementById('ipv4_routes').textContent = '-'; document.getElementById('ipv6_routes').textContent = '-'; + document.getElementById('origins_without_kept_route').textContent = '-'; document.getElementById('ipv4_updates_per_sec').textContent = '-'; document.getElementById('ipv6_updates_per_sec').textContent = '-'; document.getElementById('oldest_route').textContent = '-'; @@ -634,6 +639,7 @@ document.getElementById('live_routes').textContent = formatNumber(data.live_routes); document.getElementById('ipv4_routes').textContent = formatNumber(data.ipv4_routes); document.getElementById('ipv6_routes').textContent = formatNumber(data.ipv6_routes); + document.getElementById('origins_without_kept_route').textContent = formatNumber(data.origins_without_kept_route); document.getElementById('ipv4_updates_per_sec').textContent = formatNumber(Math.round(data.ipv4_updates_per_sec)); document.getElementById('ipv6_updates_per_sec').textContent = formatNumber(Math.round(data.ipv6_updates_per_sec)); document.getElementById('oldest_route').textContent = formatRelativeTime(data.oldest_route);