From 506e289a63e724b077112fb5766915075517db18 Mon Sep 17 00:00:00 2001 From: clawbot <35+clawbot@noreply.example.org> Date: Tue, 27 Oct 2026 16:21:59 +0100 Subject: [PATCH] Read every route from the routing table in memory (part of #122) The IP lookups, the AS and prefix pages and their API answers, the prefix-length pages and the stats read routes only from the routing table the prefix handler keeps. fx reads the table at startup and gives it to both. AS names, WHOIS details and first-seen times still come from the database. The IP lookup names the origin AS most peers carry, with the time of the newest kept route. The prefix answers list peers whose routes are not kept without a path or time. The stats gain origins_without_kept_route. They no longer read route times from the per-peer tables, so the two helpers that did are gone. Model: opus-5-5 --- README.md | 83 +-- TODO.md | 11 +- internal/database/counts_internal_test.go | 54 -- internal/database/database.go | 221 ++++---- internal/database/database_internal_test.go | 159 ++++++ internal/database/interface.go | 6 +- internal/database/models.go | 15 +- internal/routewatch/app.go | 29 +- .../app_integration_internal_test.go | 38 +- .../prefixhandler_table_internal_test.go | 8 +- internal/server/handlers.go | 492 ++++++++++++------ internal/server/handlers_internal_test.go | 476 ++++++++--------- .../server/handlers_table_internal_test.go | 368 +++++++++++++ internal/server/server.go | 11 +- internal/server/server_test.go | 2 +- internal/templates/as_detail.html | 8 +- internal/templates/prefix_detail.html | 4 +- internal/templates/status.html | 6 + 18 files changed, 1335 insertions(+), 656 deletions(-) create mode 100644 internal/server/handlers_table_internal_test.go diff --git a/README.md b/README.md index c907cfa..3eb652b 100644 --- a/README.md +++ b/README.md @@ -12,16 +12,16 @@ table in real time. It works like this: RIPE RIS Live streams the BGP updates that RIPE's route collectors receive, as they happen, from the moment a client connects. It reports changes, not the routing table they add up to, so it cannot say which AS an IP address is routed -to now or which routes a prefix has. RouteWatch applies the feed's updates to -live routing tables in its own SQLite database and answers IP-to-AS lookups and -prefix pages from them. +to now or which routes a prefix has. RouteWatch applies the feed's updates to a +routing table that it keeps in memory, with a copy in its own SQLite database. +It answers IP-to-AS lookups and prefix pages from that table. -The feed sends no table when a client connects, so these tables hold only the -routes the feed has announced since RouteWatch first started: an IP-to-AS lookup -finds nothing for an address whose routes have not been announced since then. -The tables also miss the updates RouteWatch drops when it falls behind under -load and those sent while it is stopped, so a lookup or a prefix page can show a -route the feed has withdrawn. +The feed sends no table when a client connects. So the routing table holds only +the routes announced since RouteWatch first started. An IP-to-AS lookup finds +nothing for an address whose routes have not been announced since then. The +routing table also misses updates RouteWatch drops when it falls behind, and +updates sent while it is stopped. So a lookup or a prefix page can show a route +the feed has withdrawn. ## Features @@ -58,13 +58,24 @@ DEBUG=routewatch ./routewatch Every route answers `HEAD` as it answers `GET`, without the body. +The pages and the API read routes from the routing table that RouteWatch keeps +in memory. For each prefix and origin AS, the table keeps the two newest routes +in full, with their AS paths and times. These are the kept routes. The table +keeps only the address of every other peer that carries the prefix. When both +kept routes of an origin AS are withdrawn, that origin has no kept route until +one of its peers sends an update. + ### Web Interface - `GET /` - Home page (HTML): headline figures, lookups by AS number and by IP address, and a search of AS names. A card shows the `curl` command that looks up the visitor's own address at `/ip/{addr}`, and that command's live answer. - `GET /status` - HTML status dashboard, which reads `/api/v1/stats` -- `GET /status.json` - JSON statistics: feed, memory, database and route figures +- `GET /status.json` - JSON statistics: feed, memory, database and route + figures. The route figures come from the routing table. `oldest_route` and + `newest_route` are the times of the oldest and the newest kept routes. + `origins_without_kept_route` counts, across all prefixes, the origin ASes that + have no kept route. - `GET /.well-known/healthcheck.json` - Health check (JSON): 200 when the database answers and the RIS Live feed is connected, otherwise 503; `data.status` is `ok` or `error` and `data.checks` says which check failed @@ -73,10 +84,11 @@ Every route answers `HEAD` as it answers `GET`, without the body. description come from the AS list built into RouteWatch. The page then shows no first-seen date. - `GET /prefix/{prefix}/{length}` - Prefix detail page (HTML), such as - `/prefix/192.0.2.0/24`: every live route for the prefix, one per peer, and its - origin ASes + `/prefix/192.0.2.0/24`: its origin ASes and every peer that carries it. Only + the kept routes show an AS path, a next hop and a time. - `GET /prefixlength/{length}` - Up to 500 IPv4 prefixes with that mask length, - picked at random, each with its newest route (HTML) + picked at random (HTML). Each shows the origin AS that most peers carry it + with, and the age of its newest kept route. - `GET /prefixlength6/{length}` - The same for IPv6 - `GET /ip/{addr}` - The same JSON answer as `/api/v1/ip/{ip}` - `GET /ip/` - The same, for the caller's own address: the address the request @@ -93,8 +105,10 @@ Every route answers `HEAD` as it answers `GET`, without the body. name, or a comma-separated list of at most 10 of these. `results` has one entry for each address, with: - `ip` and `ptr`: the address and its PTR names - - `netblock` and the fields beside it: the address's most specific live - route, and that route's origin AS with its WHOIS details + - `netblock` and the fields beside it: the most specific prefix that holds + the address, and its origin AS with that AS's WHOIS details. For a prefix + with several origin ASes, `asn` is the one that most peers carry. + `last_seen` is the time of the prefix's newest kept route. - `blocklists`: the public blocklists the address is on (see Blocklists) An address with no live route has a result without the route and AS fields. @@ -110,11 +124,13 @@ Every route answers `HEAD` as it answers `GET`, without the body. answered without PTR names. - `GET /api/v1/as/{asn}` - The AS and the prefixes it originates, IPv4 and IPv6 - apart, each with when its newest route was updated. When the database has no - record of the AS, its handle and description come from the AS list built into + apart, each with the time of its newest kept route. A prefix that keeps no + route with the AS has no `last_updated`. When the database has no record of + the AS, its handle and description come from the AS list built into RouteWatch. The answer then has no `first_seen` or `last_seen`. -- `GET /api/v1/prefix/{prefix}/{length}` - Every live route for the prefix, one - per peer, and the same routes grouped by origin AS in `origins` +- `GET /api/v1/prefix/{prefix}/{length}` - One route for each peer that carries + the prefix, and the same routes grouped by origin AS in `origins`. Only the + kept routes have `as_path`, `next_hop` and `last_updated`. - `GET /api/v1/search/as?q={text}` - Finds the ASes whose handle or description contains `{text}`, ignoring case. `data` holds: - `query`: `{text}` without its leading and trailing spaces @@ -137,16 +153,16 @@ took. The health check is the exception: its 503 answer is also A request that runs past its time limit answers 504: the limit is 30 seconds, and 4 seconds for the database reads behind `/status.json` and `/api/v1/stats`. An AS number, prefix or mask length that does not parse answers 400, as does a -mask length out of range for its IP version. An AS answers 404 when it is not in -the AS list built into RouteWatch and the database has neither a record of it -nor a live route it originates. AS 0 always answers 404, because it means no -known origin. The AS name search never lists it. A prefix with no live route -answers 404 too. The HTML pages give these errors as plain text. Under `/api/v1` -and `/ip/`, a path no route matches answers 404 with the message `Not found`, -and `POST`, `PUT`, `PATCH`, `DELETE`, `OPTIONS`, `TRACE` or `CONNECT` answers -405 with `Method not allowed`. A 500 answer says only `Internal server error`, -and a host name that does not resolve only `could not resolve host name`; the -detail goes to the log. +mask length out of range for its IP version. An AS answers 404 when it +originates no prefix and has no record in the database or in the AS list built +into RouteWatch. AS 0 always answers 404, because it means no known origin. The +AS name search never lists it. A prefix with no live route answers 404 too. The +HTML pages give these errors as plain text. Under `/api/v1` and `/ip/`, a path +no route matches answers 404 with the message `Not found`, and `POST`, `PUT`, +`PATCH`, `DELETE`, `OPTIONS`, `TRACE` or `CONNECT` answers 405 with +`Method not allowed`. A 500 answer says only `Internal server error`, and a host +name that does not resolve only `could not resolve host name`; the detail goes +to the log. Every answer carries `Strict-Transport-Security`, `Content-Security-Policy`, `X-Frame-Options`, `X-Content-Type-Options`, `Referrer-Policy` and @@ -340,11 +356,10 @@ routewatch/ with exponential backoff (5s-320s). The backoff resets after a stream that lasted more than 30s. A stream with no line for 60s is dropped and reconnected. -- **IP Lookup**: the most specific live route is found by looking up the - address's prefix at each mask length, longest first, in the prefix index (at - most 25 lookups for IPv4, 113 for IPv6); prefixes are stored in the text form - Go's `net/netip` prints. Routes shorter than /8 for IPv4 or /16 for IPv6, such - as a default route, are not used. +- **IP Lookup**: the routing table in memory finds the most specific prefix that + holds the address. It looks up the address's prefix at each mask length, + longest first: at most 25 lookups for IPv4 and 113 for IPv6. Prefixes shorter + than /8 for IPv4 or /16 for IPv6, such as a default route, are not used. ### Database Schema diff --git a/TODO.md b/TODO.md index f0538d9..2a3e50a 100644 --- a/TODO.md +++ b/TODO.md @@ -33,6 +33,12 @@ tested. # Completed Steps +- 2026-10-27: the pages, the API and the stats read every route from the routing + table in memory. For a prefix with several origin ASes, the IP lookup gives + the one most peers carry. Its `last_seen` is the newest kept route's time. The + prefix page and API list every peer that carries the prefix, with an AS path, + next hop and time only for the kept routes. The stats count the origins whose + kept routes were all withdrawn, in `origins_without_kept_route` (part of #122) - 2026-10-27: the prefix, AS and peer handlers make a timed write on every tick of their timers, every 1, 2 and 2 seconds. They used to skip every other tick. A full batch's write restarts the timer, so the next timed write comes one @@ -271,6 +277,5 @@ tested. # Future Steps -- the rest of https://git.eeqj.de/sneak/routewatch/issues/122, one PR each: the - pages and the API reading from the routing table, then removing the per-peer - route tables +- the rest of https://git.eeqj.de/sneak/routewatch/issues/122: removing the + per-peer route tables diff --git a/internal/database/counts_internal_test.go b/internal/database/counts_internal_test.go index 3fcd7b5..805830d 100644 --- a/internal/database/counts_internal_test.go +++ b/internal/database/counts_internal_test.go @@ -657,60 +657,6 @@ func checkOneLiveRoutePerIPVersion(t *testing.T, db *Database) { assertDistribution(t, "IPv6 distribution query", queryV6, wantV6) } -// TestStatsRouteTimestamps checks the oldest/newest route timestamps are read -// from the right rows across both tables and parse into time.Time. The old -// MIN/MAX union query read its result into *time.Time, which the driver could -// not parse, so it logged a warning every call and left both timestamps nil. -func TestStatsRouteTimestamps(t *testing.T) { - t.Parallel() - - cfg := &config.Config{StateDir: t.TempDir()} - - db, err := New(cfg, logger.New()) - if err != nil { - t.Fatalf("failed to create database: %v", err) - } - defer func() { _ = db.Close() }() - - ctx := context.Background() - - empty, err := db.GetStatsContext(ctx) - if err != nil { - t.Fatalf("GetStatsContext on empty database: %v", err) - } - - if empty.OldestRoute != nil || empty.NewestRoute != nil { - t.Fatalf("empty database timestamps = (%v, %v), want (nil, nil)", - empty.OldestRoute, empty.NewestRoute) - } - - base := time.Date(2026, 1, 2, 3, 4, 5, 0, time.UTC) - oldest := base - newest := base.Add(2 * time.Minute) - - err = db.UpsertLiveRouteBatch([]*LiveRoute{ - mkV4Route(t, "198.51.100.0/24", 64500, base.Add(time.Minute)), - mkV4Route(t, "203.0.113.0/24", 64501, oldest), - mkV6Route(testV6Prefix, 64502, newest), - }) - if err != nil { - t.Fatalf("UpsertLiveRouteBatch: %v", err) - } - - stats, err := db.GetStatsContext(ctx) - if err != nil { - t.Fatalf("GetStatsContext: %v", err) - } - - if stats.OldestRoute == nil || !stats.OldestRoute.Equal(oldest) { - t.Errorf("OldestRoute = %v, want %v", stats.OldestRoute, oldest) - } - - if stats.NewestRoute == nil || !stats.NewestRoute.Equal(newest) { - t.Errorf("NewestRoute = %v, want %v", stats.NewestRoute, newest) - } -} - // TestLiveCountsConcurrentReadWrite runs writers and stats readers at once so // the race detector proves the counters are safe under concurrent use. func TestLiveCountsConcurrentReadWrite(t *testing.T) { diff --git a/internal/database/database.go b/internal/database/database.go index 8a68944..5bb7a41 100644 --- a/internal/database/database.go +++ b/internal/database/database.go @@ -1452,10 +1452,9 @@ func (d *Database) GetStats() (Stats, error) { // // The row counts (ASNs, prefixes, peerings, peers, live routes) and the prefix // distribution come from the in-memory counters, seeded at startup and kept -// current on every write. The oldest/newest route timestamps are read from the -// ends of the last_updated index, and the file size from a stat(). No part of -// the read scans a table or a whole index. -func (d *Database) GetStatsContext(ctx context.Context) (Stats, error) { +// current on every write, and the file size from a stat(). The read touches +// no table. +func (d *Database) GetStatsContext(_ context.Context) (Stats, error) { var stats Stats // Row counts and prefix distribution from memory, as a single consistent @@ -1470,20 +1469,6 @@ func (d *Database) GetStatsContext(ctx context.Context) (Stats, error) { stats.FileSizeBytes = fileInfo.Size() } - // Oldest and newest route timestamps read one row from each end of the - // last_updated index (a log-time lookup, not a scan). Selecting the column - // directly lets the driver parse the DATETIME into time.Time; the old - // MIN/MAX union scan returned an untyped string that failed to scan and - // logged a warning on every call. - oldest, newest, err := d.routeTimestampRange(ctx) - if err != nil { - // Display-only fields; log but keep the rest of the stats. - d.logger.Warn("Failed to get route timestamps", "error", err) - } else { - stats.OldestRoute = oldest - stats.NewestRoute = newest - } - return stats, nil } @@ -1778,7 +1763,35 @@ func (d *Database) GetASDetailsContext( return nil, nil, fmt.Errorf("%w: AS%d", ErrNoRoute, asn) } - // Get AS information + asnInfo, known, err := d.GetAS(ctx, asn) + if err != nil { + return nil, nil, err + } + + allPrefixes, err := d.originPrefixes(ctx, asn, ipVersionV4) + if err != nil { + return asnInfo, nil, err + } + + prefixesV6, err := d.originPrefixes(ctx, asn, ipVersionV6) + if err != nil { + return asnInfo, allPrefixes, err + } + + allPrefixes = append(allPrefixes, prefixesV6...) + + if !known && len(allPrefixes) == 0 { + return nil, nil, fmt.Errorf("%w: AS%d", ErrNoRoute, asn) + } + + return asnInfo, allPrefixes, nil +} + +// GetAS returns the AS's handle, description and first and last seen times +// from the asns table. When the table has no record of the AS, its handle and +// description come from the AS info lookup, and it has no seen times. The bool +// says whether the asns table or the AS info lookup holds the AS. +func (d *Database) GetAS(ctx context.Context, asn int) (*ASN, bool, error) { var ( asnInfo ASN handle, description sql.NullString @@ -1791,9 +1804,6 @@ func (d *Database) GetASDetailsContext( ).Scan(&asnInfo.ASN, &handle, &description, &asnInfo.FirstSeen, &asnInfo.LastSeen) - // known is whether the asns table or the AS info lookup holds the AS. - known := true - switch { case errors.Is(err, sql.ErrNoRows): asnInfo.ASN = asn @@ -1804,31 +1814,96 @@ func (d *Database) GetASDetailsContext( asnInfo.Description = info.Description } - known = ok + return &asnInfo, ok, nil case err != nil: - return nil, nil, fmt.Errorf("failed to query AS: %w", err) - default: - asnInfo.Handle = handle.String - asnInfo.Description = description.String + return nil, false, fmt.Errorf("failed to query AS: %w", err) + } + + asnInfo.Handle = handle.String + asnInfo.Description = description.String + + return &asnInfo, true, nil +} + +// asWHOISQuery reads the handle, description and WHOIS details of one AS, and +// when the WHOIS details were fetched. +const asWHOISQuery = ` + SELECT handle, description, as_name, org_name, org_id, address, + country_code, abuse_email, rir, whois_updated_at + FROM asns WHERE asn = ?` + +// GetASWHOIS returns the handle, description and WHOIS details that the asns +// table holds for the AS, and when the WHOIS details were fetched. When the +// table has no record of the AS, it returns an ASN with only its number. +func (d *Database) GetASWHOIS(ctx context.Context, asn int) (*ASN, error) { + var ( + handle, description, asName, orgName, orgID sql.NullString + address, countryCode, abuseEmail, rir sql.NullString + whoisUpdatedAt sql.NullTime + ) + + err := d.db.QueryRowContext(ctx, asWHOISQuery, asn).Scan( + &handle, &description, &asName, &orgName, &orgID, &address, + &countryCode, &abuseEmail, &rir, &whoisUpdatedAt, + ) + if errors.Is(err, sql.ErrNoRows) { + return &ASN{ASN: asn}, nil } - allPrefixes, err := d.originPrefixes(ctx, asn, ipVersionV4) if err != nil { - return &asnInfo, nil, err + return nil, fmt.Errorf("failed to query AS: %w", err) + } + + info := &ASN{ + ASN: asn, + Handle: handle.String, + Description: description.String, + ASName: asName.String, + OrgName: orgName.String, + OrgID: orgID.String, + Address: address.String, + CountryCode: countryCode.String, + AbuseEmail: abuseEmail.String, + RIR: rir.String, + } + + if whoisUpdatedAt.Valid { + info.WHOISUpdatedAt = &whoisUpdatedAt.Time + } + + return info, nil +} + +// The queries GetPrefixFirstSeen reads an IPv4 and an IPv6 prefix's first-seen +// time with. +const ( + prefixFirstSeenV4SQL = `SELECT first_seen FROM prefixes_v4 WHERE prefix = ?` + prefixFirstSeenV6SQL = `SELECT first_seen FROM prefixes_v6 WHERE prefix = ?` +) + +// GetPrefixFirstSeen returns when the prefix tables first saw prefix, or the +// zero time when they have no record of it. +func (d *Database) GetPrefixFirstSeen( + ctx context.Context, prefix netip.Prefix, +) (time.Time, error) { + query := prefixFirstSeenV6SQL + if prefix.Addr().Is4() { + query = prefixFirstSeenV4SQL + } + + var firstSeen time.Time + + err := d.db.QueryRowContext(ctx, query, prefix.Masked().String()). + Scan(&firstSeen) + if errors.Is(err, sql.ErrNoRows) { + return time.Time{}, nil } - prefixesV6, err := d.originPrefixes(ctx, asn, ipVersionV6) if err != nil { - return &asnInfo, allPrefixes, err + return time.Time{}, fmt.Errorf("failed to query prefix: %w", err) } - allPrefixes = append(allPrefixes, prefixesV6...) - - if !known && len(allPrefixes) == 0 { - return nil, nil, fmt.Errorf("%w: AS%d", ErrNoRoute, asn) - } - - return &asnInfo, allPrefixes, nil + return firstSeen, nil } // ASPeer represents a peering relationship with another AS including handle, @@ -2564,76 +2639,6 @@ func (d *Database) deletePeerRouteChunk(peerIP string) (int, int, error) { return int(deletedV4), int(deletedV6), nil } -// routeTimestampRange returns the earliest and latest last_updated across both -// live route tables, or nil values when both tables are empty. Each query reads -// one row from an end of the last_updated index rather than scanning the tables. -func (d *Database) routeTimestampRange( - ctx context.Context, -) (*time.Time, *time.Time, error) { - var oldest, newest *time.Time - - oldestV4, ok, err := d.scanRouteTimestamp(ctx, "SELECT last_updated "+ - "FROM live_routes_v4 ORDER BY last_updated ASC LIMIT 1") - if err != nil { - return nil, nil, err - } - - if ok { - oldest = &oldestV4 - } - - oldestV6, ok, err := d.scanRouteTimestamp(ctx, "SELECT last_updated "+ - "FROM live_routes_v6 ORDER BY last_updated ASC LIMIT 1") - if err != nil { - return nil, nil, err - } - - if ok && (oldest == nil || oldestV6.Before(*oldest)) { - oldest = &oldestV6 - } - - newestV4, ok, err := d.scanRouteTimestamp(ctx, "SELECT last_updated "+ - "FROM live_routes_v4 ORDER BY last_updated DESC LIMIT 1") - if err != nil { - return nil, nil, err - } - - if ok { - newest = &newestV4 - } - - newestV6, ok, err := d.scanRouteTimestamp(ctx, "SELECT last_updated "+ - "FROM live_routes_v6 ORDER BY last_updated DESC LIMIT 1") - if err != nil { - return nil, nil, err - } - - if ok && (newest == nil || newestV6.After(*newest)) { - newest = &newestV6 - } - - return oldest, newest, nil -} - -// scanRouteTimestamp runs a single-row timestamp query. The bool is false when -// the table is empty. The query selects the last_updated column directly so the -// driver parses the DATETIME value into a time.Time. -func (d *Database) scanRouteTimestamp( - ctx context.Context, query string, -) (time.Time, bool, error) { - var ts time.Time - - err := d.db.QueryRowContext(ctx, query).Scan(&ts) - switch { - case errors.Is(err, sql.ErrNoRows): - return time.Time{}, false, nil - case err != nil: - return time.Time{}, false, err - default: - return ts, true, nil - } -} - // The AS page lists the prefixes an AS originates one at a time, each in two // index searches that read no table row: one of the origin_asn and prefix // index for the AS's first prefix after the previous one, and one of the diff --git a/internal/database/database_internal_test.go b/internal/database/database_internal_test.go index f5182d0..6891927 100644 --- a/internal/database/database_internal_test.go +++ b/internal/database/database_internal_test.go @@ -1780,6 +1780,165 @@ func TestGetASDetailsOfASNotInDatabase(t *testing.T) { } } +// TestGetASReadsTheRecordOrTheBuiltInList asks GetAS for an AS the asns table +// holds, one only the AS info lookup holds, and one neither holds. The first +// gets its record's seen times, the second its handle and description from the +// lookup, and only the third is not known. +func TestGetASReadsTheRecordOrTheBuiltInList(t *testing.T) { + t.Parallel() + + db, err := New(&config.Config{StateDir: t.TempDir()}, logger.New()) + if err != nil { + t.Fatalf("failed to create database: %v", err) + } + defer func() { _ = db.Close() }() + + const ( + storedASN = 64500 // a documentation AS, not in the lookup + listedASN = 15169 // in the lookup, not stored + unknownASN = 4200000001 // in neither + ) + + ts := time.Date(2026, 1, 2, 3, 4, 5, 0, time.UTC) + + _, err = db.GetOrCreateASN(storedASN, ts) + if err != nil { + t.Fatalf("GetOrCreateASN: %v", err) + } + + listed, ok := asinfo.Get(listedASN) + if !ok { + t.Fatalf("AS%d is not in the AS info data", listedASN) + } + + tests := []struct { + asn int + want ASN + wantKnown bool + }{ + {storedASN, ASN{ASN: storedASN, FirstSeen: ts, LastSeen: ts}, true}, + {listedASN, ASN{ + ASN: listedASN, Handle: listed.Handle, Description: listed.Description, + }, true}, + {unknownASN, ASN{ASN: unknownASN}, false}, + } + + for _, tt := range tests { + got, known, err := db.GetAS(t.Context(), tt.asn) + if err != nil { + t.Errorf("GetAS(%d): %v", tt.asn, err) + + continue + } + + if !got.FirstSeen.Equal(tt.want.FirstSeen) || + !got.LastSeen.Equal(tt.want.LastSeen) { + t.Errorf("GetAS(%d) seen %v to %v, want %v to %v", tt.asn, + got.FirstSeen, got.LastSeen, tt.want.FirstSeen, tt.want.LastSeen) + } + + got.FirstSeen, got.LastSeen = tt.want.FirstSeen, tt.want.LastSeen + if *got != tt.want || known != tt.wantKnown { + t.Errorf("GetAS(%d) = %+v, known %t; want %+v, known %t", + tt.asn, *got, known, tt.want, tt.wantKnown) + } + } +} + +// TestGetASWHOISReadsTheStoredDetails stores an AS with WHOIS details and asks +// GetASWHOIS for it and for an AS the asns table does not hold. The first gets +// its stored handle, description and details, and the time they were fetched. +// The second gets only its number, although the AS info lookup holds it. +func TestGetASWHOISReadsTheStoredDetails(t *testing.T) { + t.Parallel() + + db, err := New(&config.Config{StateDir: t.TempDir()}, logger.New()) + if err != nil { + t.Fatalf("failed to create database: %v", err) + } + defer func() { _ = db.Close() }() + + const storedASN, unstoredASN = 15169, 13335 + + _, err = db.GetOrCreateASN(storedASN, time.Date(2026, 1, 2, 3, 4, 5, 0, time.UTC)) + if err != nil { + t.Fatalf("GetOrCreateASN: %v", err) + } + + update := &ASNWHOISUpdate{ + ASN: storedASN, ASName: "TEST-AS", OrgName: "Test Org", OrgID: "ORG-1", + Address: "1 Test Street", CountryCode: "DE", AbuseEmail: "abuse@test", + RIR: "RIPE", + } + + err = db.UpdateASNWHOIS(t.Context(), update) + if err != nil { + t.Fatalf("UpdateASNWHOIS: %v", err) + } + + got, err := db.GetASWHOIS(t.Context(), storedASN) + if err != nil { + t.Fatalf("GetASWHOIS(%d): %v", storedASN, err) + } + + want := ASN{ + ASN: storedASN, Handle: asinfo.GetHandle(storedASN), + Description: asinfo.GetDescription(storedASN), ASName: update.ASName, + OrgName: update.OrgName, OrgID: update.OrgID, Address: update.Address, + CountryCode: update.CountryCode, AbuseEmail: update.AbuseEmail, + RIR: update.RIR, + } + + if got.WHOISUpdatedAt == nil { + t.Errorf("GetASWHOIS(%d) has no time the details were fetched", storedASN) + } + + got.WHOISUpdatedAt = nil + if *got != want { + t.Errorf("GetASWHOIS(%d) = %+v, want %+v", storedASN, *got, want) + } + + got, err = db.GetASWHOIS(t.Context(), unstoredASN) + if err != nil || *got != (ASN{ASN: unstoredASN}) { + t.Errorf("GetASWHOIS(%d) = %+v, %v; want only the number", + unstoredASN, got, err) + } +} + +// TestGetPrefixFirstSeenReadsThePrefixTables stores an IPv4 and an IPv6 +// prefix and expects GetPrefixFirstSeen to give each its first-seen time, and +// the zero time for a prefix that is not stored. +func TestGetPrefixFirstSeenReadsThePrefixTables(t *testing.T) { + t.Parallel() + + db, err := New(&config.Config{StateDir: t.TempDir()}, logger.New()) + if err != nil { + t.Fatalf("failed to create database: %v", err) + } + defer func() { _ = db.Close() }() + + ts := time.Date(2026, 1, 2, 3, 4, 5, 0, time.UTC) + stored := map[string]time.Time{ + "192.0.2.0/24": ts, + testV6Prefix: ts.Add(time.Hour), + } + + err = db.UpdatePrefixesBatch(stored) + if err != nil { + t.Fatalf("UpdatePrefixesBatch: %v", err) + } + + stored["198.51.100.0/24"] = time.Time{} + + for prefix, want := range stored { + got, err := db.GetPrefixFirstSeen(t.Context(), netip.MustParsePrefix(prefix)) + if err != nil || !got.Equal(want) { + t.Errorf("GetPrefixFirstSeen(%s) = %v, %v; want %v", + prefix, got, err, want) + } + } +} + // TestASPrefixQueriesUseCoveringIndexes checks the plan of each query that // lists the prefixes an AS originates: a search of a covering index and no // temporary B-tree, so the list reads one index entry per prefix, not one diff --git a/internal/database/interface.go b/internal/database/interface.go index 1e8bf5f..195c267 100644 --- a/internal/database/interface.go +++ b/internal/database/interface.go @@ -6,6 +6,7 @@ package database import ( "context" + "net/netip" "time" "git.eeqj.de/sneak/routewatch/internal/routetable" @@ -24,8 +25,6 @@ type Stats struct { LiveRoutes int IPv4Routes int IPv6Routes int - OldestRoute *time.Time - NewestRoute *time.Time IPv4PrefixDistribution []PrefixDistribution IPv6PrefixDistribution []PrefixDistribution } @@ -85,6 +84,9 @@ type Store interface { GetWHOISStats(ctx context.Context, staleThreshold time.Duration) (*WHOISStats, error) // AS and prefix detail operations + GetAS(ctx context.Context, asn int) (*ASN, bool, error) + GetASWHOIS(ctx context.Context, asn int) (*ASN, error) + GetPrefixFirstSeen(ctx context.Context, prefix netip.Prefix) (time.Time, error) GetASDetails(asn int) (*ASN, []LiveRoute, error) GetASDetailsContext(ctx context.Context, asn int) (*ASN, []LiveRoute, error) GetASPeers(asn int) ([]ASPeer, error) diff --git a/internal/database/models.go b/internal/database/models.go index 8bdc4d6..36cc035 100644 --- a/internal/database/models.go +++ b/internal/database/models.go @@ -66,7 +66,9 @@ type ASNPeering struct { LastSeen time.Time } -// LiveRoute represents a route in the live routing table +// LiveRoute represents a route in the live routing table. JSON leaves out an +// empty AS path, next hop or time: the routing table in memory keeps them only +// for the two newest routes of each prefix and origin AS. // //nolint:tagliatelle // field names are the existing JSON API type LiveRoute struct { @@ -75,9 +77,9 @@ type LiveRoute struct { IPVersion int `json:"ip_version"` OriginASN int `json:"origin_asn"` PeerIP string `json:"peer_ip"` - ASPath []int `json:"as_path"` - NextHop string `json:"next_hop"` - LastUpdated time.Time `json:"last_updated"` + ASPath []int `json:"as_path,omitempty"` + NextHop string `json:"next_hop,omitempty"` + LastUpdated time.Time `json:"last_updated,omitzero"` } // PrefixDistribution represents the distribution of prefixes by mask length @@ -108,9 +110,10 @@ type IPInfo struct { CountryCode string `json:"country_code,omitempty"` AbuseEmail string `json:"abuse_email,omitempty"` RIR string `json:"rir,omitempty"` - // Timestamps + // Timestamps. LastSeen is zero, and left out of JSON, when the prefix + // keeps no route. FirstSeen time.Time `json:"first_seen"` - LastSeen time.Time `json:"last_seen"` + LastSeen time.Time `json:"last_seen,omitzero"` // Indicates if WHOIS data needs refresh (not serialized) NeedsWHOISRefresh bool `json:"-"` } diff --git a/internal/routewatch/app.go b/internal/routewatch/app.go index f0eafef..7832669 100644 --- a/internal/routewatch/app.go +++ b/internal/routewatch/app.go @@ -31,11 +31,13 @@ import ( // are disabled, since only the batched handlers exist. var errNonBatchedNotImplemented = errors.New("non-batched handlers not implemented") -// Dependencies contains all dependencies for RouteWatch +// Dependencies contains all dependencies for RouteWatch. Table is the routing +// table read from the database at startup, which the server reads too. type Dependencies struct { fx.In DB database.Store + Table *routetable.Table Streamer *streamer.Streamer Server *server.Server Logger *logger.Logger @@ -45,6 +47,7 @@ type Dependencies struct { // RouteWatch represents the main application instance type RouteWatch struct { db database.Store + table *routetable.Table streamer *streamer.Streamer server *server.Server logger *logger.Logger @@ -69,6 +72,7 @@ const blocklistsDirName = "blocklists" func New(deps Dependencies) *RouteWatch { rw := &RouteWatch{ db: deps.DB, + table: deps.Table, streamer: deps.Streamer, server: deps.Server, logger: deps.Logger, @@ -97,13 +101,6 @@ func (rw *RouteWatch) Run(ctx context.Context) error { if rw.config.EnableBatchedDatabaseWrites { rw.logger.Info("Using batched database handlers for improved performance") - // The routing table is read before the streamer connects, so that - // the prefix handler applies the feed's updates to all of it. - table, err := rw.readRouteTable(ctx) - if err != nil { - return err - } - // ASHandler maintains the asns table rw.dbHandler = NewASHandler(rw.db, rw.logger) rw.streamer.RegisterHandler(rw.dbHandler) @@ -114,7 +111,7 @@ func (rw *RouteWatch) Run(ctx context.Context) error { // PrefixHandler maintains the prefixes and live_routes tables, and // the routing table in memory and its copy in the routes table - rw.prefixHandler = NewPrefixHandler(rw.db, rw.logger, table) + rw.prefixHandler = NewPrefixHandler(rw.db, rw.logger, rw.table) rw.prefixHandler.SetMetricsTracker(rw.streamer.GetMetricsTracker()) rw.streamer.RegisterHandler(rw.prefixHandler) @@ -269,20 +266,21 @@ func (rw *RouteWatch) closeDatabase(ctx context.Context) { // readRouteTable reads the routing table from the database and logs how many // prefix and origin AS pairs and peers it holds and how long the read took. -// ctx is fx's start context, which ends 15 seconds after the start. Reading a -// full table can take longer, so the read does not stop with ctx. -func (rw *RouteWatch) readRouteTable( - ctx context.Context, +// fx calls it once, to build the server and RouteWatch, before the streamer +// connects, so the prefix handler applies the feed's updates to all of it. No +// start timeout limits the read: fx builds them before it runs a start hook. +func readRouteTable( + db database.Store, logger *logger.Logger, ) (*routetable.Table, error) { start := time.Now() - table, err := rw.db.ReadRouteTable(context.WithoutCancel(ctx)) + table, err := db.ReadRouteTable(context.Background()) if err != nil { return nil, fmt.Errorf("failed to read the routing table: %w", err) } counts := table.Counts() - rw.logger.Info("Read the routing table", "pairs", counts.Pairs, + logger.Info("Read the routing table", "pairs", counts.Pairs, "peers", counts.Peers, "duration_ms", time.Since(start).Milliseconds()) return table, nil @@ -299,6 +297,7 @@ func getModule() []fx.Option { database.New, fx.As(new(database.Store)), ), + readRouteTable, streamer.New, server.New, New, diff --git a/internal/routewatch/app_integration_internal_test.go b/internal/routewatch/app_integration_internal_test.go index 86c78bb..f6743e9 100644 --- a/internal/routewatch/app_integration_internal_test.go +++ b/internal/routewatch/app_integration_internal_test.go @@ -7,6 +7,7 @@ import ( "net" "net/http" "net/http/httptest" + "net/netip" "slices" "strconv" "strings" @@ -265,6 +266,32 @@ func (m *mockStore) GetASDetailsContext( return m.GetASDetails(asn) } +// GetAS returns the stored AS, or only its number when none is stored. +func (m *mockStore) GetAS(_ context.Context, asn int) (*database.ASN, bool, error) { + m.mu.Lock() + defer m.mu.Unlock() + + if asnInfo, exists := m.ASNs[asn]; exists { + return asnInfo, true, nil + } + + return &database.ASN{ASN: asn}, false, nil +} + +// GetASWHOIS returns the stored AS, or only its number when none is stored. +func (m *mockStore) GetASWHOIS(ctx context.Context, asn int) (*database.ASN, error) { + asnInfo, _, err := m.GetAS(ctx, asn) + + return asnInfo, err +} + +// GetPrefixFirstSeen returns the zero time: the mock keeps no prefix times. +func (m *mockStore) GetPrefixFirstSeen( + _ context.Context, _ netip.Prefix, +) (time.Time, error) { + return time.Time{}, nil +} + // GetPrefixDetails mock implementation func (m *mockStore) GetPrefixDetails( _ string, @@ -502,12 +529,15 @@ func TestRouteWatchLiveFeed(t *testing.T) { EnableBatchedDatabaseWrites: true, } + table := routetable.New(routetable.Params{}) + // Create server - srv := server.New(mockDB, s, logger, cfg) + srv := server.New(mockDB, table, s, logger, cfg) // Create RouteWatch with 5 second limit deps := Dependencies{ DB: mockDB, + Table: table, Streamer: s, Server: srv, Logger: logger, @@ -626,7 +656,7 @@ func TestShutdownWritesTheUpdatesQueuedInTheStreamer(t *testing.T) { rw := New(Dependencies{ DB: store, Streamer: s, - Server: server.New(store, s, log, cfg), + Server: server.New(store, nil, s, log, cfg), Logger: log, Config: cfg, }) @@ -728,7 +758,7 @@ func TestShutdownReturnsByItsDeadlineWhileAWriteIsBlocked(t *testing.T) { rw := New(Dependencies{ DB: store, Streamer: s, - Server: server.New(store, s, log, cfg), + Server: server.New(store, nil, s, log, cfg), Logger: log, Config: cfg, }) @@ -842,7 +872,7 @@ func TestShutdownCheckpointsAndClosesTheDatabaseLast(t *testing.T) { rw := New(Dependencies{ DB: store, Streamer: s, - Server: server.New(store, s, log, cfg), + Server: server.New(store, nil, s, log, cfg), Logger: log, Config: cfg, }) diff --git a/internal/routewatch/prefixhandler_table_internal_test.go b/internal/routewatch/prefixhandler_table_internal_test.go index dc8a7cd..97969e2 100644 --- a/internal/routewatch/prefixhandler_table_internal_test.go +++ b/internal/routewatch/prefixhandler_table_internal_test.go @@ -416,12 +416,8 @@ func TestARestartReadsTheRoutingTableBack(t *testing.T) { var logs bytes.Buffer - rw := &RouteWatch{ - db: db, - logger: &logger.Logger{Logger: slog.New(slog.NewJSONHandler(&logs, nil))}, - } - - table, err := rw.readRouteTable(t.Context()) + table, err := readRouteTable(db, + &logger.Logger{Logger: slog.New(slog.NewJSONHandler(&logs, nil))}) if err != nil { t.Fatalf("readRouteTable: %v", err) } diff --git a/internal/server/handlers.go b/internal/server/handlers.go index de829ce..1f5fa29 100644 --- a/internal/server/handlers.go +++ b/internal/server/handlers.go @@ -5,12 +5,12 @@ import ( "context" "encoding/json" "errors" + "fmt" "net" "net/http" "net/netip" "runtime" "slices" - "sort" "strconv" "strings" "time" @@ -19,6 +19,7 @@ import ( "git.eeqj.de/sneak/routewatch/internal/database" "git.eeqj.de/sneak/routewatch/internal/metrics" "git.eeqj.de/sneak/routewatch/internal/ristypes" + "git.eeqj.de/sneak/routewatch/internal/routetable" "git.eeqj.de/sneak/routewatch/internal/streamer" "git.eeqj.de/sneak/routewatch/internal/templates" asinfo "git.eeqj.de/sneak/routewatch/pkg/asinfo" @@ -38,7 +39,7 @@ const ( const ( // maxLookupTargets is the most addresses and host names an IP lookup may // list, and the most addresses it may have once the names are resolved: - // each address costs up to 113 database reads and a PTR lookup. + // each address costs two database reads and a PTR lookup. maxLookupTargets = 10 // ptrLookupTimeout is each PTR lookup's own time limit, so that ten @@ -227,7 +228,8 @@ type WHOISStatsInfo struct { } // Stats is the /status.json answer: uptime, message counts, database stats, -// and route information. +// and the routing table's figures. OldestRoute and NewestRoute are the times +// of the oldest and the newest kept routes. // //nolint:tagliatelle // field names are the existing JSON API type Stats struct { @@ -259,6 +261,10 @@ type Stats struct { IPv4PrefixDistribution []database.PrefixDistribution `json:"ipv4_prefix_distribution"` IPv6PrefixDistribution []database.PrefixDistribution `json:"ipv6_prefix_distribution"` WHOISStats *WHOISStatsInfo `json:"whois_stats,omitempty"` + + // OriginsWithoutKeptRoute counts the prefix and origin AS pairs that peers + // carry but whose kept routes were all withdrawn. + OriginsWithoutKeptRoute int `json:"origins_without_kept_route"` } // databaseStats gets the database stats, giving up when ctx is done; it then @@ -317,15 +323,6 @@ func (s *Server) handleStatusJSON() http.HandlerFunc { return } - const bitsPerMegabit = 1000000.0 - - // Get route update metrics - routeMetrics := s.streamer.GetMetricsTracker().GetRouteMetrics() - - // Get memory stats - var memStats runtime.MemStats - runtime.ReadMemStats(&memStats) - // Get WHOIS stats if fetcher is available var whoisStats *WHOISStatsInfo if s.asnFetcher != nil { @@ -337,44 +334,80 @@ func (s *Server) handleStatusJSON() http.HandlerFunc { } } - stats := Stats{ - Uptime: streamUptime(metrics.ConnectedSince), - TotalMessages: metrics.TotalMessages, - TotalBytes: metrics.TotalBytes, - TotalWireBytes: metrics.TotalWireBytes, - MessagesPerSec: metrics.MessagesPerSec, - MbitsPerSec: metrics.BitsPerSec / bitsPerMegabit, - WireMbitsPerSec: metrics.WireBitsPerSec / bitsPerMegabit, - Connected: metrics.Connected, - GoVersion: runtime.Version(), - Goroutines: runtime.NumGoroutine(), - MemoryUsage: humanize.Bytes(memStats.Alloc), - ASNs: dbStats.ASNs, - Prefixes: dbStats.Prefixes, - IPv4Prefixes: dbStats.IPv4Prefixes, - IPv6Prefixes: dbStats.IPv6Prefixes, - Peerings: dbStats.Peerings, - Peers: dbStats.Peers, - DatabaseSizeBytes: dbStats.FileSizeBytes, - LiveRoutes: dbStats.LiveRoutes, - IPv4Routes: dbStats.IPv4Routes, - IPv6Routes: dbStats.IPv6Routes, - OldestRoute: dbStats.OldestRoute, - NewestRoute: dbStats.NewestRoute, - IPv4UpdatesPerSec: routeMetrics.IPv4UpdatesPerSec, - IPv6UpdatesPerSec: routeMetrics.IPv6UpdatesPerSec, - IPv4PrefixDistribution: dbStats.IPv4PrefixDistribution, - IPv6PrefixDistribution: dbStats.IPv6PrefixDistribution, - WHOISStats: whoisStats, - } - - err = writeJSONSuccess(w, stats) + err = writeJSONSuccess(w, s.statusStats(metrics, dbStats, whoisStats)) if err != nil { s.logger.Error("Failed to encode stats", "error", err) } } } +// statusStats builds the /status.json answer from the stream metrics, the +// database stats, the routing table's figures and the WHOIS stats, with the +// route update rates and memory use as they are now. +func (s *Server) statusStats( + streamMetrics metrics.StreamMetrics, + dbStats database.Stats, + whoisStats *WHOISStatsInfo, +) Stats { + const bitsPerMegabit = 1000000.0 + + routeMetrics := s.streamer.GetMetricsTracker().GetRouteMetrics() + counts := s.table.Counts() + + var memStats runtime.MemStats + runtime.ReadMemStats(&memStats) + + return Stats{ + Uptime: streamUptime(streamMetrics.ConnectedSince), + TotalMessages: streamMetrics.TotalMessages, + TotalBytes: streamMetrics.TotalBytes, + TotalWireBytes: streamMetrics.TotalWireBytes, + MessagesPerSec: streamMetrics.MessagesPerSec, + MbitsPerSec: streamMetrics.BitsPerSec / bitsPerMegabit, + WireMbitsPerSec: streamMetrics.WireBitsPerSec / bitsPerMegabit, + Connected: streamMetrics.Connected, + GoVersion: runtime.Version(), + Goroutines: runtime.NumGoroutine(), + MemoryUsage: humanize.Bytes(memStats.Alloc), + ASNs: dbStats.ASNs, + Prefixes: counts.IPv4Prefixes + counts.IPv6Prefixes, + IPv4Prefixes: counts.IPv4Prefixes, + IPv6Prefixes: counts.IPv6Prefixes, + Peerings: dbStats.Peerings, + Peers: dbStats.Peers, + DatabaseSizeBytes: dbStats.FileSizeBytes, + LiveRoutes: counts.IPv4Routes + counts.IPv6Routes, + IPv4Routes: counts.IPv4Routes, + IPv6Routes: counts.IPv6Routes, + OldestRoute: counts.OldestKeptRoute, + NewestRoute: counts.NewestKeptRoute, + IPv4UpdatesPerSec: routeMetrics.IPv4UpdatesPerSec, + IPv6UpdatesPerSec: routeMetrics.IPv6UpdatesPerSec, + IPv4PrefixDistribution: prefixDistribution(counts.IPv4PrefixesByLength), + IPv6PrefixDistribution: prefixDistribution(counts.IPv6PrefixesByLength), + WHOISStats: whoisStats, + + OriginsWithoutKeptRoute: counts.OriginsWithoutKeptRoute, + } +} + +// prefixDistribution lists the mask lengths that have at least one prefix, in +// ascending order, with their prefix counts. countsByLength[n] is the number +// of prefixes of mask length n. +func prefixDistribution(countsByLength []int) []database.PrefixDistribution { + var distribution []database.PrefixDistribution + + for maskLength, count := range countsByLength { + if count > 0 { + distribution = append(distribution, database.PrefixDistribution{ + MaskLength: maskLength, Count: count, + }) + } + } + + return distribution +} + // getWHOISStats builds WHOIS statistics from database and fetcher. func (s *Server) getWHOISStats(ctx context.Context) *WHOISStatsInfo { // Get database WHOIS stats @@ -470,7 +503,8 @@ type StreamStats struct { BGPPeerCount int `json:"bgp_peer_count"` } -// StatsResponse represents the API statistics response +// StatsResponse represents the API statistics response. The fields it shares +// with Stats mean what they mean there. // //nolint:tagliatelle // field names are the existing JSON API type StatsResponse struct { @@ -507,6 +541,8 @@ type StatsResponse struct { IPv4PrefixDistribution []database.PrefixDistribution `json:"ipv4_prefix_distribution"` IPv6PrefixDistribution []database.PrefixDistribution `json:"ipv6_prefix_distribution"` WHOISStats *WHOISStatsInfo `json:"whois_stats,omitempty"` + + OriginsWithoutKeptRoute int `json:"origins_without_kept_route"` } // handlerStatsInfo converts the streamer's handler statistics for the API @@ -626,8 +662,8 @@ func (s *Server) handleStats() http.HandlerFunc { } // statsResponse builds the /api/v1/stats answer from the stream metrics, the -// database stats and the WHOIS stats, with the route update rates, memory use -// and handler queues as they are now. +// database stats, the routing table's figures and the WHOIS stats, with the +// route update rates, memory use and handler queues as they are now. func (s *Server) statsResponse( streamMetrics metrics.StreamMetrics, dbStats database.Stats, @@ -637,6 +673,7 @@ func (s *Server) statsResponse( // Get route update metrics routeMetrics := s.streamer.GetMetricsTracker().GetRouteMetrics() + counts := s.table.Counts() // Get memory stats var memStats runtime.MemStats @@ -660,23 +697,25 @@ func (s *Server) statsResponse( GC: gcStats(&memStats), Stream: s.streamStats(), ASNs: dbStats.ASNs, - Prefixes: dbStats.Prefixes, - IPv4Prefixes: dbStats.IPv4Prefixes, - IPv6Prefixes: dbStats.IPv6Prefixes, + Prefixes: counts.IPv4Prefixes + counts.IPv6Prefixes, + IPv4Prefixes: counts.IPv4Prefixes, + IPv6Prefixes: counts.IPv6Prefixes, Peerings: dbStats.Peerings, Peers: dbStats.Peers, DatabaseSizeBytes: dbStats.FileSizeBytes, - LiveRoutes: dbStats.LiveRoutes, - IPv4Routes: dbStats.IPv4Routes, - IPv6Routes: dbStats.IPv6Routes, - OldestRoute: dbStats.OldestRoute, - NewestRoute: dbStats.NewestRoute, + LiveRoutes: counts.IPv4Routes + counts.IPv6Routes, + IPv4Routes: counts.IPv4Routes, + IPv6Routes: counts.IPv6Routes, + OldestRoute: counts.OldestKeptRoute, + NewestRoute: counts.NewestKeptRoute, IPv4UpdatesPerSec: routeMetrics.IPv4UpdatesPerSec, IPv6UpdatesPerSec: routeMetrics.IPv6UpdatesPerSec, HandlerStats: handlerStatsInfo(s.streamer.GetHandlerStats()), - IPv4PrefixDistribution: dbStats.IPv4PrefixDistribution, - IPv6PrefixDistribution: dbStats.IPv6PrefixDistribution, + IPv4PrefixDistribution: prefixDistribution(counts.IPv4PrefixesByLength), + IPv6PrefixDistribution: prefixDistribution(counts.IPv6PrefixesByLength), WHOISStats: whoisStats, + + OriginsWithoutKeptRoute: counts.OriginsWithoutKeptRoute, } } @@ -766,7 +805,7 @@ func (s *Server) handleIPInfo() http.HandlerFunc { // Look up each IP for _, ip := range ipsToLookup { - ipInfo, err := s.db.GetIPInfoContext(ctx, ip) + ipInfo, err := s.ipInfo(ctx, ip) if deadlinePassed(ctx) { s.writeJSONError(w, http.StatusGatewayTimeout, requestTimeoutMessage) @@ -815,6 +854,81 @@ func (s *Server) handleIPInfo() http.HandlerFunc { } } +// ipInfo returns the IP lookup's answer for ip. The routing table gives the +// most specific prefix that holds the address, the number of peers that carry +// it, the origin AS that most of them carry, and the time of its newest kept +// route. The database gives the prefix's first-seen time and the AS's WHOIS +// details. It returns ErrNoRoute when no prefix holds the address, and +// ErrInvalidIP when ip does not parse. +func (s *Server) ipInfo(ctx context.Context, ip string) (*database.IPInfo, error) { + addr, err := netip.ParseAddr(ip) + if err != nil { + return nil, fmt.Errorf("%w: %s", database.ErrInvalidIP, ip) + } + + summary, found := s.table.Lookup(addr) + if !found { + return nil, fmt.Errorf("%w for IP %s", database.ErrNoRoute, ip) + } + + firstSeen, err := s.db.GetPrefixFirstSeen(ctx, summary.Prefix) + if err != nil { + return nil, err + } + + asInfo, err := s.db.GetASWHOIS(ctx, summary.OriginASN) + if err != nil { + return nil, err + } + + info := &database.IPInfo{ + IP: ip, + Netblock: summary.Prefix.String(), + MaskLength: summary.Prefix.Bits(), + IPVersion: ipVersionOf(summary.Prefix), + NumPeers: summary.Peers, + ASN: summary.OriginASN, + ASName: asInfo.ASName, + Handle: asInfo.Handle, + Description: asInfo.Description, + OrgName: asInfo.OrgName, + OrgID: asInfo.OrgID, + Address: asInfo.Address, + CountryCode: asInfo.CountryCode, + AbuseEmail: asInfo.AbuseEmail, + RIR: asInfo.RIR, + FirstSeen: firstSeen, + NeedsWHOISRefresh: needsWHOISRefresh(asInfo), + } + + if summary.Newest != nil { + info.LastSeen = *summary.Newest + } + + return info, nil +} + +// needsWHOISRefresh says whether the WHOIS details of asInfo were never +// fetched or are older than whoisStaleThreshold. AS 0, no known origin, has +// none to fetch. +func needsWHOISRefresh(asInfo *database.ASN) bool { + if asInfo.ASN == ristypes.NoOrigin { + return false + } + + return asInfo.WHOISUpdatedAt == nil || + time.Since(*asInfo.WHOISUpdatedAt) > whoisStaleThreshold +} + +// ipVersionOf returns 4 or 6, the IP version of prefix. +func ipVersionOf(prefix netip.Prefix) int { + if prefix.Addr().Is4() { + return ipVersionV4 + } + + return ipVersionV6 +} + // blocklistListings returns the blocklists that list ip. It returns an empty // list, never nil, so that an answer's blocklists is [] rather than null. func (s *Server) blocklistListings(ip string) []blocklist.Listing { @@ -1043,7 +1157,7 @@ func (s *Server) handleASDetailJSON() http.HandlerFunc { return } - asInfo, prefixes, err := s.db.GetASDetailsContext(r.Context(), asn) + asInfo, prefixes, err := s.asDetails(r.Context(), asn) if deadlinePassed(r.Context()) { s.writeJSONError(w, http.StatusGatewayTimeout, requestTimeoutMessage) @@ -1079,6 +1193,49 @@ func (s *Server) handleASDetailJSON() http.HandlerFunc { } } +// asDetails returns the AS's handle, description and seen times, from the +// database, and the prefixes it originates, from the routing table. Each +// prefix has the time of its newest kept route, if it keeps one. It returns +// ErrNoRoute for AS 0, which means no known origin. It also returns ErrNoRoute +// for an AS that the database, the AS list built into RouteWatch and the +// routing table all lack. +func (s *Server) asDetails( + ctx context.Context, asn int, +) (*database.ASN, []database.LiveRoute, error) { + if asn == ristypes.NoOrigin { + return nil, nil, fmt.Errorf("%w: AS%d", database.ErrNoRoute, asn) + } + + asInfo, known, err := s.db.GetAS(ctx, asn) + if err != nil { + return nil, nil, err + } + + originated := s.table.OriginatedPrefixes(asn) + if !known && len(originated) == 0 { + return nil, nil, fmt.Errorf("%w: AS%d", database.ErrNoRoute, asn) + } + + prefixes := make([]database.LiveRoute, 0, len(originated)) + + for _, p := range originated { + route := database.LiveRoute{ + Prefix: p.Prefix.String(), + MaskLength: p.Prefix.Bits(), + IPVersion: ipVersionOf(p.Prefix), + OriginASN: asn, + } + + if p.Newest != nil { + route.LastUpdated = *p.Newest + } + + prefixes = append(prefixes, route) + } + + return asInfo, prefixes, nil +} + // splitRoutesByIPVersion groups routes into the IPv4 and the IPv6 ones. Neither // group is nil, so that a JSON answer holds [] for an empty one and not null. func splitRoutesByIPVersion( @@ -1123,8 +1280,7 @@ func (s *Server) handlePrefixDetailJSON() http.HandlerFunc { return } - // Look the prefix up in the text form net/netip prints, the form the - // tables store, so that any valid way of writing it finds the route. + // Masking the prefix lets any valid way of writing it find its routes. parsed, err := netip.ParsePrefix(prefixParam + "/" + lenParam) if err != nil { s.writeJSONError(w, http.StatusBadRequest, invalidPrefixMessage) @@ -1132,27 +1288,18 @@ func (s *Server) handlePrefixDetailJSON() http.HandlerFunc { return } - prefix := parsed.Masked().String() + parsed = parsed.Masked() + prefix := parsed.String() - routes, _, err := s.db.GetPrefixDetailsContext(r.Context(), prefix) - if deadlinePassed(r.Context()) { - s.writeJSONError(w, http.StatusGatewayTimeout, requestTimeoutMessage) + origins := s.table.Origins(parsed) + if len(origins) == 0 { + s.writeJSONError(w, http.StatusNotFound, + database.ErrNoRoute.Error()+": "+prefix) return } - if err != nil { - if errors.Is(err, database.ErrNoRoute) { - s.writeJSONError(w, http.StatusNotFound, err.Error()) - } else { - s.logger.Error("Failed to get prefix details", - "prefix", prefix, "error", err) - s.writeJSONError(w, http.StatusInternalServerError, - internalErrorMessage) - } - - return - } + routes := prefixRoutes(parsed, origins) // Group by origin AS originMap := make(map[int][]database.LiveRoute) @@ -1237,7 +1384,7 @@ func (s *Server) handleASDetail() http.HandlerFunc { return } - asInfo, prefixes, err := s.db.GetASDetailsContext(r.Context(), asn) + asInfo, prefixes, err := s.asDetails(r.Context(), asn) if deadlinePassed(r.Context()) { http.Error(w, requestTimeoutMessage, http.StatusGatewayTimeout) @@ -1327,8 +1474,7 @@ func (s *Server) handlePrefixDetail() http.HandlerFunc { return } - // Look the prefix up in the text form net/netip prints, the form the - // tables store, so that any valid way of writing it finds the route. + // Masking the prefix lets any valid way of writing it find its routes. parsed, err := netip.ParsePrefix(prefixParam + "/" + lenParam) if err != nil { http.Error(w, invalidPrefixMessage, http.StatusBadRequest) @@ -1336,9 +1482,16 @@ func (s *Server) handlePrefixDetail() http.HandlerFunc { return } - prefix := parsed.Masked().String() + parsed = parsed.Masked() - routes, originNames, err := s.db.GetPrefixDetailsContext(r.Context(), prefix) + origins := s.table.Origins(parsed) + if len(origins) == 0 { + http.Error(w, "Prefix not found", http.StatusNotFound) + + return + } + + originInfos, err := s.originASNs(r.Context(), origins) if deadlinePassed(r.Context()) { http.Error(w, requestTimeoutMessage, http.StatusGatewayTimeout) @@ -1346,27 +1499,13 @@ func (s *Server) handlePrefixDetail() http.HandlerFunc { } if err != nil { - if errors.Is(err, database.ErrNoRoute) { - http.Error(w, "Prefix not found", http.StatusNotFound) - } else { - s.logger.Error("Failed to get prefix details", "error", err) - http.Error(w, internalErrorMessage, http.StatusInternalServerError) - } + s.logger.Error("Failed to get the origin ASes' names", "error", err) + http.Error(w, internalErrorMessage, http.StatusInternalServerError) return } - // Group by origin AS and collect unique AS info - origins := originASNs(routes, originNames) - - // Get the first route to extract some common info - var maskLength, ipVersion int - if len(routes) > 0 { - maskLength, ipVersion = prefixMaskLengthAndIPVersion(prefix) - } - - // Create enhanced routes with AS path handles - enhancedRoutes := routesWithASPathHandles(routes) + routes := prefixRoutes(parsed, origins) // Prepare template data data := struct { @@ -1379,13 +1518,13 @@ func (s *Server) handlePrefixDetail() http.HandlerFunc { OriginCount int CSPNonce string }{ - Prefix: prefix, - MaskLength: maskLength, - IPVersion: ipVersion, - Routes: enhancedRoutes, - Origins: origins, + Prefix: parsed.String(), + MaskLength: parsed.Bits(), + IPVersion: ipVersionOf(parsed), + Routes: routesWithASPathHandles(routes), + Origins: originInfos, PeerCount: len(routes), - OriginCount: len(origins), + OriginCount: len(originInfos), CSPNonce: cspNonce(r.Context()), } @@ -1415,34 +1554,72 @@ type ASNInfo struct { PeerCount int } -// originASNs groups routes by origin AS, with each AS's handle and description -// from names, as the prefix query returned them, and the number of routes it -// originates. The origin ASes come in no particular order. -func originASNs( - routes []database.LiveRoute, - names map[int]database.ASN, -) []*ASNInfo { - originMap := make(map[int]*ASNInfo) +// originASNs returns each of origins, a prefix's origin ASes, with its handle +// and description from the database and the number of peers that carry the +// prefix with it. +func (s *Server) originASNs( + ctx context.Context, origins []routetable.Origin, +) ([]*ASNInfo, error) { + infos := make([]*ASNInfo, 0, len(origins)) - for _, route := range routes { - if _, exists := originMap[route.OriginASN]; !exists { - originMap[route.OriginASN] = &ASNInfo{ - ASN: route.OriginASN, - Handle: names[route.OriginASN].Handle, - Description: names[route.OriginASN].Description, - PeerCount: 0, - } + for _, origin := range origins { + asInfo, _, err := s.db.GetAS(ctx, origin.ASN) + if err != nil { + return nil, err } - originMap[route.OriginASN].PeerCount++ + infos = append(infos, &ASNInfo{ + ASN: origin.ASN, + Handle: asInfo.Handle, + Description: asInfo.Description, + PeerCount: len(origin.Peers), + }) } - origins := make([]*ASNInfo, 0, len(originMap)) - for _, origin := range originMap { - origins = append(origins, origin) + return infos, nil +} + +// prefixRoutes returns one route for each peer that carries prefix, by origin +// AS in the order of origins. Each origin's kept routes come first, newest +// first, with their AS path, next hop and time. Its other peers follow without +// them, since the routing table keeps only their addresses. +func prefixRoutes( + prefix netip.Prefix, origins []routetable.Origin, +) []database.LiveRoute { + var routes []database.LiveRoute + + for _, origin := range origins { + base := database.LiveRoute{ + Prefix: prefix.String(), + MaskLength: prefix.Bits(), + IPVersion: ipVersionOf(prefix), + OriginASN: origin.ASN, + } + + kept := make(map[string]bool, len(origin.Routes)) + + for _, r := range origin.Routes { + kept[r.Peer] = true + + route := base + route.PeerIP = r.Peer + route.ASPath = r.ASPath + // The next hop is the peer, as the prefix handler records it. + route.NextHop = r.Peer + route.LastUpdated = r.Time + routes = append(routes, route) + } + + for _, peer := range origin.Peers { + if !kept[peer] { + route := base + route.PeerIP = peer + routes = append(routes, route) + } + } } - return origins + return routes } // ASPathEntry is one AS of a route's AS path, with its handle. @@ -1484,23 +1661,6 @@ func routesWithASPathHandles(routes []database.LiveRoute) []EnhancedRoute { return enhancedRoutes } -// prefixMaskLengthAndIPVersion returns the mask length and IP version of a -// prefix in CIDR notation, or zeros if it does not parse. -func prefixMaskLengthAndIPVersion(prefix string) (int, int) { - _, ipNet, err := net.ParseCIDR(prefix) - if err != nil { - return 0, 0 - } - - maskLength, _ := ipNet.Mask.Size() - - if ipNet.IP.To4() != nil { - return maskLength, ipVersionV4 - } - - return maskLength, ipVersionV6 -} - // handlePrefixLength shows a random sample of IPv4 prefixes with the // specified mask length func (s *Server) handlePrefixLength() http.HandlerFunc { @@ -1539,29 +1699,11 @@ func (s *Server) prefixLengthHandler(ipVersion, maxMaskLength int) http.HandlerF // Get random sample of prefixes const maxPrefixes = 500 - prefixes, err := s.db.GetRandomPrefixesByLengthContext( - r.Context(), maskLength, ipVersion, maxPrefixes) - if deadlinePassed(r.Context()) { - http.Error(w, requestTimeoutMessage, http.StatusGatewayTimeout) - - return - } - - if err != nil { - s.logger.Error("Failed to get prefixes by length", "error", err) - http.Error(w, internalErrorMessage, http.StatusInternalServerError) - - return - } + prefixes := s.table.SamplePrefixes(ipVersion, maskLength, maxPrefixes) // Sort prefixes for display - sort.Slice(prefixes, func(i, j int) bool { - // First compare by IP version - if prefixes[i].IPVersion != prefixes[j].IPVersion { - return prefixes[i].IPVersion < prefixes[j].IPVersion - } - // Then by prefix - return prefixes[i].Prefix < prefixes[j].Prefix + slices.SortFunc(prefixes, func(a, b routetable.PrefixSummary) int { + return a.Prefix.Compare(b.Prefix) }) // Render template @@ -1591,8 +1733,8 @@ func (s *Server) prefixLengthHandler(ipVersion, maxMaskLength int) http.HandlerF } } -// EnhancedPrefix is a live route on the prefix length page, with the -// description of its origin AS and its age. +// EnhancedPrefix is a prefix on the prefix length page, with the origin AS +// that most peers carry it with, that AS's description, and its age. type EnhancedPrefix struct { database.LiveRoute @@ -1601,13 +1743,23 @@ type EnhancedPrefix struct { } // prefixesWithASDescriptions looks up the description of the origin AS of -// each prefix and gives each one's age. -func prefixesWithASDescriptions(prefixes []database.LiveRoute) []EnhancedPrefix { +// each prefix. It gives each prefix the age of its newest kept route, or no +// age when it keeps none. +func prefixesWithASDescriptions( + prefixes []routetable.PrefixSummary, +) []EnhancedPrefix { enhancedPrefixes := make([]EnhancedPrefix, len(prefixes)) for i, prefix := range prefixes { enhancedPrefixes[i] = EnhancedPrefix{ - LiveRoute: prefix, - Age: formatAge(prefix.LastUpdated), + Prefix: prefix.Prefix.String(), + MaskLength: prefix.Prefix.Bits(), + IPVersion: ipVersionOf(prefix.Prefix), + OriginASN: prefix.OriginASN, + } + + if prefix.Newest != nil { + enhancedPrefixes[i].LastUpdated = *prefix.Newest + enhancedPrefixes[i].Age = formatAge(*prefix.Newest) } // Get AS description diff --git a/internal/server/handlers_internal_test.go b/internal/server/handlers_internal_test.go index cc7872e..d2b0152 100644 --- a/internal/server/handlers_internal_test.go +++ b/internal/server/handlers_internal_test.go @@ -9,6 +9,7 @@ import ( "net/http" "net/http/httptest" "net/netip" + "reflect" "runtime" "slices" "strconv" @@ -23,6 +24,7 @@ import ( "git.eeqj.de/sneak/routewatch/internal/logger" "git.eeqj.de/sneak/routewatch/internal/metrics" "git.eeqj.de/sneak/routewatch/internal/ristypes" + "git.eeqj.de/sneak/routewatch/internal/routetable" "git.eeqj.de/sneak/routewatch/internal/streamer" "git.eeqj.de/sneak/routewatch/pkg/asinfo" ) @@ -35,6 +37,38 @@ const ( testPeerV6 = "2001:db8::1" ) +// testPeer returns the address of the nth of the IPv4 peers that tests +// announce routes from: 198.51.100.n. +func testPeer(n int) string { + return "198.51.100." + strconv.Itoa(n) +} + +// testRouteTime is the time the tests' routes are announced at, or a number of +// minutes after it. +func testRouteTime(minutes int) time.Time { + return time.Date(2026, 1, 2, 3, 4, 5, 0, time.UTC). + Add(time.Duration(minutes) * time.Minute) +} + +// announce has peer announce prefix in table at time at, with an AS path +// that holds only the origin AS asn. +func announce( + table *routetable.Table, prefix string, asn int, peer string, at time.Time, +) { + table.Announce(netip.MustParsePrefix(prefix), asn, + routetable.Route{Peer: peer, ASPath: []int{asn}, Time: at}) +} + +// withdraw has peer withdraw prefix in table. +func withdraw(table *routetable.Table, prefix, peer string) { + table.Withdraw(netip.MustParsePrefix(prefix), peer) +} + +// emptyTable returns a routing table that holds no route. +func emptyTable() *routetable.Table { + return routetable.New(routetable.Params{}) +} + // Paths several tests request. const ( asPagePath = "/as/64500" @@ -76,7 +110,8 @@ func (d blockingStatsDB) GetStatsContext(_ context.Context) (database.Stats, err func TestStatsHandlersDoNotLeakOnTimeout(t *testing.T) { release := make(chan struct{}) db := blockingStatsDB{release: release} - s := New(db, streamer.New(logger.New(), metrics.New()), logger.New(), &config.Config{}) + s := New(db, emptyTable(), streamer.New(logger.New(), metrics.New()), + logger.New(), &config.Config{}) handlers := map[string]http.HandlerFunc{ statusJSONHandlerName: s.handleStatusJSON(), @@ -108,51 +143,78 @@ func TestStatsHandlersDoNotLeakOnTimeout(t *testing.T) { } } +// tableStats holds the figures of a stats answer that come from the routing +// table. +// +//nolint:tagliatelle // field names are the existing JSON API +type tableStats struct { + Prefixes int `json:"prefixes"` + IPv4Prefixes int `json:"ipv4_prefixes"` + IPv6Prefixes int `json:"ipv6_prefixes"` + LiveRoutes int `json:"live_routes"` + IPv4Routes int `json:"ipv4_routes"` + IPv6Routes int `json:"ipv6_routes"` + OriginsWithoutKeptRoute int `json:"origins_without_kept_route"` + OldestRoute time.Time `json:"oldest_route"` + NewestRoute time.Time `json:"newest_route"` + + IPv4Distribution []database.PrefixDistribution `json:"ipv4_prefix_distribution"` + IPv6Distribution []database.PrefixDistribution `json:"ipv6_prefix_distribution"` +} + // TestStatsHandlersAnswerFromMemory checks that both stats handlers answer 200 -// with the live route counts and the prefix distribution while the database is -// closed, so that any query would fail: the request path reads them from -// memory. The prefix distribution query it used to run read every live route -// and, on a large database, took the whole 4-second deadline, so +// with the routing table's figures while the database is closed, so that any +// query would fail. The prefix distribution query they used to run read every +// live route and, on a large database, took the whole 4-second deadline, so // /api/v1/stats answered 500 (https://git.eeqj.de/sneak/routewatch/issues/30). -// The oldest and newest route times still come from one-row lookups at the ends -// of an index; with the database closed they are left out of the answer. +// +// The table holds two IPv4 prefixes and an IPv6 one. Of the three peers that +// carried 203.0.113.0/24, the two with kept routes withdrew it, so its origin +// keeps no route. The oldest and newest routes are those of the other two +// prefixes. func TestStatsHandlersAnswerFromMemory(t *testing.T) { t.Parallel() + const withdrawnPrefix = "203.0.113.0/24" + + table := emptyTable() + announce(table, testPrefixV4, 64502, testPeer(1), testRouteTime(0)) + + for n := 1; n <= 3; n++ { + announce(table, withdrawnPrefix, 64500, testPeer(n), testRouteTime(n)) + } + + withdraw(table, withdrawnPrefix, testPeer(2)) + withdraw(table, withdrawnPrefix, testPeer(3)) + announce(table, testPrefixV6, 64501, testPeerV6, testRouteTime(4)) + db, err := database.New(&config.Config{StateDir: t.TempDir()}, logger.New()) if err != nil { t.Fatalf("database.New: %v", err) } - ts := time.Date(2026, 1, 2, 3, 4, 5, 0, time.UTC) - - err = db.UpsertLiveRouteBatch([]*database.LiveRoute{ - { - Prefix: "198.51.100.0/24", MaskLength: 24, IPVersion: 4, - OriginASN: 64500, PeerIP: "192.0.2.1", ASPath: []int{64500}, - NextHop: "192.0.2.1", LastUpdated: ts, - }, - { - Prefix: "2001:db8::/32", MaskLength: 32, IPVersion: 6, - OriginASN: 64501, PeerIP: testPeerV6, ASPath: []int{64501}, - NextHop: testPeerV6, LastUpdated: ts, - }, - }) - if err != nil { - t.Fatalf("UpsertLiveRouteBatch: %v", err) - } - err = db.Close() if err != nil { t.Fatalf("Close: %v", err) } - s := New(db, streamer.New(logger.New(), metrics.New()), logger.New(), &config.Config{}) + s := New(db, table, streamer.New(logger.New(), metrics.New()), logger.New(), + &config.Config{}) handlers := map[string]http.HandlerFunc{ statusJSONHandlerName: s.handleStatusJSON(), statsHandlerName: s.handleStats(), } + want := tableStats{ + Prefixes: 3, IPv4Prefixes: 2, IPv6Prefixes: 1, + LiveRoutes: 3, IPv4Routes: 2, IPv6Routes: 1, + OriginsWithoutKeptRoute: 1, + OldestRoute: testRouteTime(0), + NewestRoute: testRouteTime(4), + IPv4Distribution: []database.PrefixDistribution{{MaskLength: 24, Count: 2}}, + IPv6Distribution: []database.PrefixDistribution{{MaskLength: 32, Count: 1}}, + } + for name, handler := range handlers { rec := httptest.NewRecorder() handler(rec, httptest.NewRequestWithContext(t.Context(), http.MethodGet, "/", nil)) @@ -164,14 +226,8 @@ func TestStatsHandlersAnswerFromMemory(t *testing.T) { continue } - //nolint:tagliatelle // field names are the existing JSON API var body struct { - Data struct { - IPv4Routes int `json:"ipv4_routes"` - IPv6Routes int `json:"ipv6_routes"` - IPv4Distribution []database.PrefixDistribution `json:"ipv4_prefix_distribution"` - IPv6Distribution []database.PrefixDistribution `json:"ipv6_prefix_distribution"` - } `json:"data"` + Data tableStats `json:"data"` } err := json.Unmarshal(rec.Body.Bytes(), &body) @@ -179,21 +235,16 @@ func TestStatsHandlersAnswerFromMemory(t *testing.T) { t.Fatalf("%s: decoding the answer: %v", name, err) } - if body.Data.IPv4Routes != 1 || body.Data.IPv6Routes != 1 { - t.Errorf("%s: routes = (v4 %d, v6 %d), want (1, 1)", - name, body.Data.IPv4Routes, body.Data.IPv6Routes) + got := body.Data + if !got.OldestRoute.Equal(want.OldestRoute) || + !got.NewestRoute.Equal(want.NewestRoute) { + t.Errorf("%s: routes from %v to %v, want from %v to %v", name, + got.OldestRoute, got.NewestRoute, want.OldestRoute, want.NewestRoute) } - wantV4 := []database.PrefixDistribution{{MaskLength: 24, Count: 1}} - if !slices.Equal(body.Data.IPv4Distribution, wantV4) { - t.Errorf("%s: IPv4 distribution = %v, want %v", - name, body.Data.IPv4Distribution, wantV4) - } - - wantV6 := []database.PrefixDistribution{{MaskLength: 32, Count: 1}} - if !slices.Equal(body.Data.IPv6Distribution, wantV6) { - t.Errorf("%s: IPv6 distribution = %v, want %v", - name, body.Data.IPv6Distribution, wantV6) + got.OldestRoute, got.NewestRoute = want.OldestRoute, want.NewestRoute + if !reflect.DeepEqual(got, want) { + t.Errorf("%s: figures %+v, want %+v", name, got, want) } } } @@ -243,11 +294,29 @@ func TestStatsAnswerCarriesChurnPerMinute(t *testing.T) { } } -// newServerWithTestRoutes returns a server on a new database that holds a -// route for testPrefixV4 and one for testPrefixV6. +// newTestTable returns a routing table that holds a route for testPrefixV4, +// with origin AS 64500, and one for testPrefixV6, with origin AS 64501. +func newTestTable() *routetable.Table { + table := emptyTable() + announce(table, testPrefixV4, 64500, testPeerV4, testRouteTime(0)) + announce(table, testPrefixV6, 64501, testPeerV6, testRouteTime(0)) + + return table +} + +// newServerWithTestRoutes returns a server on a new database that reads the +// routes of newTestTable. func newServerWithTestRoutes(t *testing.T) *Server { t.Helper() + return newServerWithTable(t, newTestTable()) +} + +// newServerWithTable returns a server on a new database that reads routes from +// table. +func newServerWithTable(t *testing.T, table *routetable.Table) *Server { + t.Helper() + db, err := database.New(&config.Config{StateDir: t.TempDir()}, logger.New()) if err != nil { t.Fatalf("database.New: %v", err) @@ -255,25 +324,7 @@ func newServerWithTestRoutes(t *testing.T) *Server { t.Cleanup(func() { _ = db.Close() }) - ts := time.Date(2026, 1, 2, 3, 4, 5, 0, time.UTC) - - err = db.UpsertLiveRouteBatch([]*database.LiveRoute{ - { - Prefix: testPrefixV4, MaskLength: 24, IPVersion: 4, - OriginASN: 64500, PeerIP: testPeerV4, ASPath: []int{64500}, - NextHop: testPeerV4, LastUpdated: ts, - }, - { - Prefix: testPrefixV6, MaskLength: 32, IPVersion: 6, - OriginASN: 64501, PeerIP: testPeerV6, ASPath: []int{64501}, - NextHop: testPeerV6, LastUpdated: ts, - }, - }) - if err != nil { - t.Fatalf("UpsertLiveRouteBatch: %v", err) - } - - return New(db, streamer.New(logger.New(), metrics.New()), logger.New(), + return New(db, table, streamer.New(logger.New(), metrics.New()), logger.New(), &config.Config{}) } @@ -402,34 +453,20 @@ func TestPrefixDetailShowsOriginNames(t *testing.T) { t.Cleanup(func() { _ = db.Close() }) - ts := time.Date(2026, 1, 2, 3, 4, 5, 0, time.UTC) origins := []int{15169, 13335} + table := emptyTable() - for _, asn := range origins { - _, err := db.GetOrCreateASN(asn, ts) + for i, asn := range origins { + _, err := db.GetOrCreateASN(asn, testRouteTime(0)) if err != nil { t.Fatalf("GetOrCreateASN(%d): %v", asn, err) } + + announce(table, testPrefixV4, asn, testPeer(i+1), testRouteTime(0)) } - err = db.UpsertLiveRouteBatch([]*database.LiveRoute{ - { - Prefix: testPrefixV4, MaskLength: 24, IPVersion: 4, - OriginASN: origins[0], PeerIP: testPeerV4, - ASPath: []int{origins[0]}, NextHop: testPeerV4, LastUpdated: ts, - }, - { - Prefix: testPrefixV4, MaskLength: 24, IPVersion: 4, - OriginASN: origins[1], PeerIP: "198.51.100.2", - ASPath: []int{origins[1]}, NextHop: "198.51.100.2", LastUpdated: ts, - }, - }) - if err != nil { - t.Fatalf("UpsertLiveRouteBatch: %v", err) - } - - s := New(failingASDetailsDB{db}, streamer.New(logger.New(), metrics.New()), - logger.New(), &config.Config{}) + s := New(failingASDetailsDB{db}, table, + streamer.New(logger.New(), metrics.New()), logger.New(), &config.Config{}) rec := httptest.NewRecorder() s.router.ServeHTTP(rec, httptest.NewRequestWithContext( t.Context(), http.MethodGet, "/prefix/"+testPrefixV4, nil)) @@ -466,23 +503,14 @@ func TestPrefixDetailShowsOriginNames(t *testing.T) { func TestPagesSayOriginZeroIsNoKnownOrigin(t *testing.T) { t.Parallel() - db, err := database.New(&config.Config{StateDir: t.TempDir()}, logger.New()) - if err != nil { - t.Fatalf("database.New: %v", err) - } + table := emptyTable() + table.Announce(netip.MustParsePrefix(testPrefixV4), ristypes.NoOrigin, + routetable.Route{ + Peer: testPeerV4, ASPath: []int{64500, 64501, 64502}, + Time: testRouteTime(0), + }) - t.Cleanup(func() { _ = db.Close() }) - - err = db.UpsertLiveRouteBatch([]*database.LiveRoute{{ - Prefix: testPrefixV4, MaskLength: 24, IPVersion: 4, - OriginASN: 0, PeerIP: testPeerV4, ASPath: []int{64500, 64501, 64502}, - NextHop: testPeerV4, LastUpdated: time.Date(2026, 1, 2, 3, 4, 5, 0, time.UTC), - }}) - if err != nil { - t.Fatalf("UpsertLiveRouteBatch: %v", err) - } - - s := New(db, streamer.New(logger.New(), metrics.New()), logger.New(), &config.Config{}) + s := newServerWithTable(t, table) for _, path := range []string{"/prefix/" + testPrefixV4, prefixLengthPagePath} { rec := httptest.NewRecorder() @@ -533,8 +561,9 @@ func TestIndexShowsTheClientAddress(t *testing.T) { } for _, tt := range tests { - // The home page reads nothing from the database, so the server has none. - s := New(nil, streamer.New(logger.New(), metrics.New()), logger.New(), + // The home page reads no route and nothing from the database, so the + // server has neither a routing table nor a database. + s := New(nil, nil, streamer.New(logger.New(), metrics.New()), logger.New(), &config.Config{TrustedProxies: tt.trustedProxies}) req := httptest.NewRequestWithContext(t.Context(), http.MethodGet, "/", nil) @@ -571,13 +600,6 @@ func TestIndexShowsTheClientAddress(t *testing.T) { func TestPagesShowAgesInWords(t *testing.T) { t.Parallel() - db, err := database.New(&config.Config{StateDir: t.TempDir()}, logger.New()) - if err != nil { - t.Fatalf("database.New: %v", err) - } - - t.Cleanup(func() { _ = db.Close() }) - // Each age is well past a whole number of its unit, so the time the test // takes to run does not change the words. ages := []struct { @@ -591,22 +613,13 @@ func TestPagesShowAgesInWords(t *testing.T) { } now := time.Now().UTC() - routes := make([]*database.LiveRoute, 0, len(ages)) + table := emptyTable() for _, a := range ages { - routes = append(routes, &database.LiveRoute{ - Prefix: a.prefix, MaskLength: 24, IPVersion: 4, - OriginASN: 64500, PeerIP: testPeerV4, ASPath: []int{64500}, - NextHop: testPeerV4, LastUpdated: now.Add(-a.age), - }) + announce(table, a.prefix, 64500, testPeerV4, now.Add(-a.age)) } - err = db.UpsertLiveRouteBatch(routes) - if err != nil { - t.Fatalf("UpsertLiveRouteBatch: %v", err) - } - - s := New(db, streamer.New(logger.New(), metrics.New()), logger.New(), &config.Config{}) + s := newServerWithTable(t, table) for _, a := range ages { cell := `` + a.want + `` @@ -643,7 +656,7 @@ const shortRequestDeadline = 50 * time.Millisecond func TestRequestPastTimeLimitAnswers504(t *testing.T) { t.Parallel() - s := New(nil, streamer.New(logger.New(), metrics.New()), logger.New(), + s := New(nil, nil, streamer.New(logger.New(), metrics.New()), logger.New(), &config.Config{}) waitForTimeLimit := func(_ http.ResponseWriter, r *http.Request) { @@ -693,8 +706,8 @@ func TestStatsHandlersAnswer504WhenDatabaseReadRunsOut(t *testing.T) { defer close(release) db := blockingStatsDB{release: release} - s := New(db, streamer.New(logger.New(), metrics.New()), logger.New(), - &config.Config{}) + s := New(db, emptyTable(), streamer.New(logger.New(), metrics.New()), + logger.New(), &config.Config{}) handlers := map[string]http.HandlerFunc{ statusJSONHandlerName: s.handleStatusJSON(), @@ -720,10 +733,9 @@ func TestStatsHandlersAnswer504WhenDatabaseReadRunsOut(t *testing.T) { // slowDB embeds database.Store (left nil) and overrides the reads the routes // in TestRoutesAnswer504WhenDatabaseReadIsCutOff make. Each read waits until // its context ends and then answers as the real store does when the time -// limit cuts it off: the IP read with the context's error, the prefix read -// that there is no route, and the others with what they had read, here -// nothing, and no error. Every other Store method is unused and would panic -// if called. +// limit cuts it off: the stats read with what it had read, here nothing, and +// no error, and the others with the context's error. Every other Store method +// is unused and would panic if called. type slowDB struct { database.Store } @@ -734,66 +746,44 @@ func (slowDB) GetStatsContext(ctx context.Context) (database.Stats, error) { return database.Stats{}, nil } -func (slowDB) GetIPInfoContext( - ctx context.Context, _ string, -) (*database.IPInfo, error) { +func (slowDB) GetAS(ctx context.Context, _ int) (*database.ASN, bool, error) { <-ctx.Done() - return nil, ctx.Err() + return nil, false, ctx.Err() } -func (slowDB) GetASDetailsContext( - ctx context.Context, asn int, -) (*database.ASN, []database.LiveRoute, error) { +func (slowDB) GetPrefixFirstSeen( + ctx context.Context, _ netip.Prefix, +) (time.Time, error) { <-ctx.Done() - return &database.ASN{ASN: asn}, nil, nil -} - -func (slowDB) GetPrefixDetailsContext( - ctx context.Context, _ string, -) ([]database.LiveRoute, map[int]database.ASN, error) { - <-ctx.Done() - - return nil, nil, database.ErrNoRoute -} - -func (slowDB) GetRandomPrefixesByLengthContext( - ctx context.Context, _, _, _ int, -) ([]database.LiveRoute, error) { - <-ctx.Done() - - return nil, nil + return time.Time{}, ctx.Err() } // TestRoutesAnswer504WhenDatabaseReadIsCutOff sends a request through the // server's router for each route that reads the database, with a store whose // reads are cut off when the request's time limit passes. Each route answers // 504 with the time limit message. Without its check after the read, a route -// answers with what the cut-off read returned: 500, 404, 200, or, on a page -// that stops before rendering once the request has ended, 504 with an empty -// body. The health check is left out: it answers 503 when its own 2-second -// ping runs out. +// answers with what the cut-off read returned: 500 or 200. The health check is +// left out: it answers 503 when its own 2-second ping runs out. The prefix +// length pages and the prefix API read only the routing table. func TestRoutesAnswer504WhenDatabaseReadIsCutOff(t *testing.T) { t.Parallel() - s := New(slowDB{}, streamer.New(logger.New(), metrics.New()), logger.New(), - &config.Config{}) + s := New(slowDB{}, newTestTable(), streamer.New(logger.New(), metrics.New()), + logger.New(), &config.Config{}) tests := []struct { path string wantJSONBody bool }{ {path: asPagePath}, - {path: "/prefix/192.0.2.0/24"}, - {path: prefixLengthPagePath}, - {path: "/prefixlength6/48"}, + {path: "/prefix/" + testPrefixV4}, {path: statusJSONPath, wantJSONBody: true}, - {path: "/ip/192.0.2.1", wantJSONBody: true}, + {path: "/ip/" + addrInTestPrefixV4, wantJSONBody: true}, {path: statsPath, wantJSONBody: true}, - {path: "/api/v1/ip/192.0.2.1", wantJSONBody: true}, + {path: "/api/v1/ip/" + addrInTestPrefixV4, wantJSONBody: true}, {path: "/api/v1/as/64500", wantJSONBody: true}, - {path: "/api/v1/prefix/192.0.2.0/24", wantJSONBody: true}, } for _, tt := range tests { @@ -810,10 +800,8 @@ type slowSecondReadDB struct { database.Store } -func (slowSecondReadDB) GetASDetailsContext( - _ context.Context, asn int, -) (*database.ASN, []database.LiveRoute, error) { - return &database.ASN{ASN: asn}, nil, nil +func (slowSecondReadDB) GetAS(_ context.Context, asn int) (*database.ASN, bool, error) { + return &database.ASN{ASN: asn}, true, nil } func (slowSecondReadDB) GetASPeersContext( @@ -855,8 +843,8 @@ func (idleASNFetcher) GetStats() ASNFetcherStats { func TestRoutesAnswer504WhenSecondDatabaseReadIsCutOff(t *testing.T) { t.Parallel() - s := New(slowSecondReadDB{}, streamer.New(logger.New(), metrics.New()), - logger.New(), &config.Config{}) + s := New(slowSecondReadDB{}, emptyTable(), + streamer.New(logger.New(), metrics.New()), logger.New(), &config.Config{}) s.SetASNFetcher(idleASNFetcher{}) tests := []struct { @@ -950,34 +938,27 @@ func (failingDB) GetStatsContext(_ context.Context) (database.Stats, error) { return database.Stats{}, errDiskIO } -func (failingDB) GetIPInfoContext( - _ context.Context, _ string, -) (*database.IPInfo, error) { - return nil, errDiskIO +func (failingDB) GetAS(_ context.Context, _ int) (*database.ASN, bool, error) { + return nil, false, errDiskIO } -func (failingDB) GetASDetailsContext( - _ context.Context, _ int, -) (*database.ASN, []database.LiveRoute, error) { - return nil, nil, errDiskIO +func (failingDB) GetPrefixFirstSeen( + _ context.Context, _ netip.Prefix, +) (time.Time, error) { + return time.Time{}, errDiskIO } -func (failingDB) GetPrefixDetailsContext( - _ context.Context, _ string, -) ([]database.LiveRoute, map[int]database.ASN, error) { - return nil, nil, errDiskIO -} - -// TestJSONRoutesDoNotSendErrorDetails sends a request to each JSON route and -// the health check with a store whose reads fail with the driver's error text, -// and a prefix that does not parse, and checks that no answer carries the -// driver's or the prefix parser's text. Before the fix the answers sent it to -// the client (https://git.eeqj.de/sneak/routewatch/issues/108). +// TestJSONRoutesDoNotSendErrorDetails sends a request to each JSON route that +// reads the database, and to the health check, with a store whose reads fail +// with the driver's error text. It also sends a prefix that does not parse. +// No answer may carry the driver's or the prefix parser's text. Before the fix +// the answers sent it to the client +// (https://git.eeqj.de/sneak/routewatch/issues/108). func TestJSONRoutesDoNotSendErrorDetails(t *testing.T) { t.Parallel() - s := New(failingDB{}, streamer.New(logger.New(), metrics.New()), logger.New(), - &config.Config{}) + s := New(failingDB{}, newTestTable(), streamer.New(logger.New(), metrics.New()), + logger.New(), &config.Config{}) tests := []struct { path string @@ -987,8 +968,7 @@ func TestJSONRoutesDoNotSendErrorDetails(t *testing.T) { {statusJSONPath, http.StatusInternalServerError}, {statsPath, http.StatusInternalServerError}, {"/api/v1/as/64500", http.StatusInternalServerError}, - {"/api/v1/prefix/192.0.2.0/24", http.StatusInternalServerError}, - {"/api/v1/ip/192.0.2.1", http.StatusInternalServerError}, + {"/api/v1/ip/" + addrInTestPrefixV4, http.StatusInternalServerError}, {"/api/v1/prefix/foo/24", http.StatusBadRequest}, } @@ -1011,45 +991,47 @@ func TestJSONRoutesDoNotSendErrorDetails(t *testing.T) { } } -// The addresses failingIPLookupDB fails the read of and finds a route for. +// The address whose read failingIPLookupDB fails, in testPrefixV4, and one it +// reads, in testPrefixV6. const ( - failingLookupIP = "192.0.2.1" - routedLookupIP = "192.0.2.2" + failingLookupIP = addrInTestPrefixV4 + routedLookupIP = addrInTestPrefixV6 ) -// failingIPLookupDB embeds database.Store (left nil) and answers the IP lookup -// with errDiskIO for failingLookupIP, a route for routedLookupIP, and no route -// for any other address. Every other Store method is unused and would panic if -// called. +// failingIPLookupDB embeds database.Store (left nil). It fails the read of +// testPrefixV4's first-seen time with errDiskIO, and answers the IP lookup's +// other reads with nothing stored. Every other Store method is unused and +// would panic if called. type failingIPLookupDB struct { database.Store } -func (failingIPLookupDB) GetIPInfoContext( - _ context.Context, ip string, -) (*database.IPInfo, error) { - switch ip { - case failingLookupIP: - return nil, errDiskIO - case routedLookupIP: - return &database.IPInfo{IP: ip}, nil - default: - return nil, database.ErrNoRoute +func (failingIPLookupDB) GetPrefixFirstSeen( + _ context.Context, prefix netip.Prefix, +) (time.Time, error) { + if prefix == netip.MustParsePrefix(testPrefixV4) { + return time.Time{}, errDiskIO } + + return time.Time{}, nil +} + +func (failingIPLookupDB) GetASWHOIS(_ context.Context, asn int) (*database.ASN, error) { + return &database.ASN{ASN: asn}, nil } // TestIPLookupAnswers500WhenDatabaseReadFails sends IP lookups through the -// server's router with a store whose read of one address fails with the +// server's router with a store whose read for one address fails with the // driver's error. A lookup that asks for that address answers 500 with the -// JSON error body, as the AS and prefix JSON routes do; an address with no -// route answers 200. Before the fix the failed read was reported as if the -// address had no route: 404 for that address alone, and 200 beside a routed -// address (https://git.eeqj.de/sneak/routewatch/issues/109). +// JSON error body, as the AS JSON route does; an address with no route answers +// 200. Before the fix the failed read was reported as if the address had no +// route: 404 for that address alone, and 200 beside a routed address +// (https://git.eeqj.de/sneak/routewatch/issues/109). func TestIPLookupAnswers500WhenDatabaseReadFails(t *testing.T) { t.Parallel() - s := New(failingIPLookupDB{}, streamer.New(logger.New(), metrics.New()), - logger.New(), &config.Config{}) + s := New(failingIPLookupDB{}, newTestTable(), + streamer.New(logger.New(), metrics.New()), logger.New(), &config.Config{}) s.resolver = &fakeResolver{} tests := []struct { @@ -1122,30 +1104,37 @@ func (r *fakeResolver) LookupAddr(ctx context.Context, _ string) ([]string, erro return nil, errNoSuchHost } -// countingIPLookupDB embeds database.Store (left nil) and counts its IP -// lookups, answering each that the address has no route. Every other Store -// method is unused and would panic if called. +// countingIPLookupDB embeds database.Store (left nil) and answers the IP +// lookup's reads with nothing stored. It counts the lookups, as the reads of a +// prefix's first-seen time. Every other Store method is unused and would panic +// if called. type countingIPLookupDB struct { database.Store lookups atomic.Int64 } -func (d *countingIPLookupDB) GetIPInfoContext( - _ context.Context, _ string, -) (*database.IPInfo, error) { +func (d *countingIPLookupDB) GetPrefixFirstSeen( + _ context.Context, _ netip.Prefix, +) (time.Time, error) { d.lookups.Add(1) - return nil, database.ErrNoRoute + return time.Time{}, nil +} + +func (d *countingIPLookupDB) GetASWHOIS( + _ context.Context, asn int, +) (*database.ASN, error) { + return &database.ASN{ASN: asn}, nil } // TestIPLookupRefusesTooManyAddresses sends IP lookups of maxLookupTargets -// addresses and of one more, listed or as a host name that resolves to them, -// and a lookup listing maxLookupTargets+1 host names. The longer lookups answer -// 400 without reading the database, and the list of host names without -// resolving any of them; the others read the database once per address. Before -// the fix every address was read and every name resolved, however many there -// were (https://git.eeqj.de/sneak/routewatch/issues/110). +// addresses in testPrefixV4 and of one more, listed or as a host name that +// resolves to them, and a lookup listing maxLookupTargets+1 host names. The +// longer lookups answer 400 without reading the database, and the list of host +// names without resolving any of them; the others read the database for each +// address. Before the fix every address was read and every name resolved, +// however many there were (https://git.eeqj.de/sneak/routewatch/issues/110). func TestIPLookupRefusesTooManyAddresses(t *testing.T) { t.Parallel() @@ -1180,8 +1169,8 @@ func TestIPLookupRefusesTooManyAddresses(t *testing.T) { for _, tt := range tests { db := &countingIPLookupDB{} - s := New(db, streamer.New(logger.New(), metrics.New()), logger.New(), - &config.Config{}) + s := New(db, newTestTable(), streamer.New(logger.New(), metrics.New()), + logger.New(), &config.Config{}) resolver := &fakeResolver{hosts: hosts} s.resolver = resolver @@ -1901,16 +1890,11 @@ func TestNoOriginHasNoASAnswers(t *testing.T) { const noOriginPrefix = "203.0.113.0/24" s := newServerWithTestRoutes(t) - - err := s.db.UpsertLiveRouteBatch([]*database.LiveRoute{{ - Prefix: noOriginPrefix, MaskLength: 24, IPVersion: 4, - OriginASN: ristypes.NoOrigin, PeerIP: testPeerV4, - ASPath: []int{64500, 64501, 64502}, NextHop: testPeerV4, - LastUpdated: time.Date(2026, 1, 2, 3, 4, 5, 0, time.UTC), - }}) - if err != nil { - t.Fatalf("UpsertLiveRouteBatch: %v", err) - } + s.table.Announce(netip.MustParsePrefix(noOriginPrefix), ristypes.NoOrigin, + routetable.Route{ + Peer: testPeerV4, ASPath: []int{64500, 64501, 64502}, + Time: testRouteTime(0), + }) asn := strconv.Itoa(ristypes.NoOrigin) diff --git a/internal/server/handlers_table_internal_test.go b/internal/server/handlers_table_internal_test.go new file mode 100644 index 0000000..515a163 --- /dev/null +++ b/internal/server/handlers_table_internal_test.go @@ -0,0 +1,368 @@ +package server + +import ( + "encoding/json" + "net/http" + "net/http/httptest" + "slices" + "strconv" + "strings" + "testing" + "time" + + "git.eeqj.de/sneak/routewatch/internal/database" + "git.eeqj.de/sneak/routewatch/internal/routetable" +) + +// The origin ASes in the routing table that newTwoOriginTable fills. +const ( + mostPeersASN = 64500 + fewerPeersASN = 64501 + v6OriginASN = 64502 +) + +// noKeptRoutePrefix is the prefix of newTwoOriginTable whose origin keeps no +// route. +const noKeptRoutePrefix = "198.51.100.0/24" + +// routeTimeFormat is how the pages write a route's time. +const routeTimeFormat = "2006-01-02 15:04:05" + +// newTwoOriginTable returns a routing table that holds: +// - testPrefixV4 with two origin ASes. Peers 1, 2 and 3 carry it with +// mostPeersASN, which keeps the routes of peers 3 and 2. Peer 4 carries +// it with fewerPeersASN, in the prefix's newest route. +// - noKeptRoutePrefix with mostPeersASN. Only peer 7 carries it, and its +// route is not kept: peers 5 and 6, whose routes were, withdrew it. +// - testPrefixV6 with v6OriginASN. +func newTwoOriginTable() *routetable.Table { + table := emptyTable() + + for n := 1; n <= 3; n++ { + announce(table, testPrefixV4, mostPeersASN, testPeer(n), testRouteTime(n)) + } + + announce(table, testPrefixV4, fewerPeersASN, testPeer(4), testRouteTime(5)) + + announce(table, noKeptRoutePrefix, mostPeersASN, testPeer(5), testRouteTime(4)) + announce(table, noKeptRoutePrefix, mostPeersASN, testPeer(6), testRouteTime(6)) + announce(table, noKeptRoutePrefix, mostPeersASN, testPeer(7), testRouteTime(0)) + withdraw(table, noKeptRoutePrefix, testPeer(5)) + withdraw(table, noKeptRoutePrefix, testPeer(6)) + + announce(table, testPrefixV6, v6OriginASN, testPeerV6, testRouteTime(0)) + + return table +} + +// get sends a GET for path through s's router and fails the test unless it +// answers 200. +func get(t *testing.T, s *Server, path string) *httptest.ResponseRecorder { + t.Helper() + + rec := httptest.NewRecorder() + s.router.ServeHTTP(rec, httptest.NewRequestWithContext( + t.Context(), http.MethodGet, path, nil)) + + if rec.Code != http.StatusOK { + t.Fatalf("%s: status %d, want %d; body %s", + path, rec.Code, http.StatusOK, rec.Body) + } + + return rec +} + +// TestIPLookupGivesTheOriginMostPeersCarry looks up an address in a prefix +// with two origin ASes, through each IP lookup route. The answer names the +// origin AS that more peers carry, counts every peer, and gives the time of +// the prefix's newest kept route. The prefix's first-seen time and the AS's +// WHOIS details come from the database. +func TestIPLookupGivesTheOriginMostPeersCarry(t *testing.T) { + t.Parallel() + + s := newServerWithTable(t, newTwoOriginTable()) + s.resolver = &fakeResolver{} + + firstSeen := testRouteTime(-1) + + err := s.db.UpdatePrefixesBatch(map[string]time.Time{testPrefixV4: firstSeen}) + if err != nil { + t.Fatalf("UpdatePrefixesBatch: %v", err) + } + + _, err = s.db.GetOrCreateASN(mostPeersASN, firstSeen) + if err != nil { + t.Fatalf("GetOrCreateASN: %v", err) + } + + whois := &database.ASNWHOISUpdate{ + ASN: mostPeersASN, ASName: "TEST-AS", CountryCode: "DE", + } + + err = s.db.UpdateASNWHOIS(t.Context(), whois) + if err != nil { + t.Fatalf("UpdateASNWHOIS: %v", err) + } + + // ownAddressPath looks up the request's own address, which httptest + // gives as addrInTestPrefixV4. + for _, path := range []string{ + "/api/v1/ip/" + addrInTestPrefixV4, "/ip/" + addrInTestPrefixV4, + ownAddressPath, + } { + var body struct { + Data IPLookupResponse `json:"data"` + } + + err := json.Unmarshal(get(t, s, path).Body.Bytes(), &body) + if err != nil || len(body.Data.Results) != 1 || + body.Data.Results[0].IPInfo == nil { + t.Fatalf("%s: answer %+v (%v), want one result with a route", + path, body.Data, err) + } + + got := body.Data.Results[0].IPInfo + if got.Netblock != testPrefixV4 || got.ASN != mostPeersASN || + got.NumPeers != 4 || !got.LastSeen.Equal(testRouteTime(5)) || + !got.FirstSeen.Equal(firstSeen) || got.ASName != whois.ASName || + got.CountryCode != whois.CountryCode { + t.Errorf("%s: route %+v, want %s from AS%d, 4 peers, last seen %v, "+ + "first seen %v, %s in %s", path, *got, testPrefixV4, + mostPeersASN, testRouteTime(5), firstSeen, whois.ASName, + whois.CountryCode) + } + } +} + +// prefixRoute is one route of the prefix API's answer. A field the answer +// leaves out stays nil. +// +//nolint:tagliatelle // field names are the existing JSON API +type prefixRoute struct { + OriginASN int `json:"origin_asn"` + PeerIP string `json:"peer_ip"` + ASPath []int `json:"as_path"` + NextHop *string `json:"next_hop"` + LastUpdated *time.Time `json:"last_updated"` +} + +// TestPrefixAnswersListEveryPeer asks the prefix API and page for a prefix +// with two origin ASes, and for one whose origin keeps no route. Each answer +// lists every peer that carries the prefix, by origin AS. Only the kept routes +// have an AS path, a next hop and a time, newest first. The other peers follow +// without them. +func TestPrefixAnswersListEveryPeer(t *testing.T) { + t.Parallel() + + s := newServerWithTable(t, newTwoOriginTable()) + + // keptAt[peer] is the time of the peer's kept route; a peer not in it is + // listed without a route. + keptAt := map[string]time.Time{ + testPeer(3): testRouteTime(3), + testPeer(2): testRouteTime(2), + testPeer(4): testRouteTime(5), + } + + tests := []struct { + prefix string + wantPeers []string + wantOrigins []int + }{ + { + testPrefixV4, + []string{testPeer(3), testPeer(2), testPeer(1), testPeer(4)}, + []int{mostPeersASN, mostPeersASN, mostPeersASN, fewerPeersASN}, + }, + {noKeptRoutePrefix, []string{testPeer(7)}, []int{mostPeersASN}}, + } + + for _, tt := range tests { + //nolint:tagliatelle // field names are the existing JSON API + var body struct { + Data struct { + Routes []prefixRoute `json:"routes"` + PeerCount int `json:"peer_count"` + OriginCount int `json:"origin_count"` + } `json:"data"` + } + + path := "/api/v1/prefix/" + tt.prefix + + err := json.Unmarshal(get(t, s, path).Body.Bytes(), &body) + if err != nil { + t.Fatalf("%s: decoding the answer: %v", path, err) + } + + wantOriginCount := len(slices.Compact(slices.Clone(tt.wantOrigins))) + if body.Data.PeerCount != len(tt.wantPeers) || + body.Data.OriginCount != wantOriginCount { + t.Errorf("%s: %d peers and %d origins, want %d and %d", path, + body.Data.PeerCount, body.Data.OriginCount, len(tt.wantPeers), + wantOriginCount) + } + + checkPrefixRoutes(t, path, body.Data.Routes, tt.wantPeers, tt.wantOrigins, + keptAt) + checkPrefixPage(t, s, tt.prefix, tt.wantPeers, keptAt) + } +} + +// checkPrefixRoutes checks that routes come from wantPeers, in that order, +// with wantOrigins as their origin ASes. The route of a peer in keptAt must +// have an AS path, the peer as next hop, and the time keptAt gives. Any other +// route must have none of the three. +func checkPrefixRoutes( + t *testing.T, path string, routes []prefixRoute, wantPeers []string, + wantOrigins []int, keptAt map[string]time.Time, +) { + t.Helper() + + if len(routes) != len(wantPeers) { + t.Fatalf("%s: routes %+v, want one each from %v", path, routes, wantPeers) + } + + for i, route := range routes { + if route.PeerIP != wantPeers[i] || route.OriginASN != wantOrigins[i] { + t.Errorf("%s: route %d from %s with AS%d, want from %s with AS%d", + path, i, route.PeerIP, route.OriginASN, wantPeers[i], wantOrigins[i]) + } + + at, kept := keptAt[route.PeerIP] + if !kept { + if route.ASPath != nil || route.NextHop != nil || route.LastUpdated != nil { + t.Errorf("%s: route from %s is not kept but has %+v", + path, route.PeerIP, route) + } + + continue + } + + if route.ASPath == nil || route.NextHop == nil || + *route.NextHop != route.PeerIP || route.LastUpdated == nil || + !route.LastUpdated.Equal(at) { + t.Errorf("%s: kept route from %s is %+v, want its AS path, the peer "+ + "as next hop and the time %v", path, route.PeerIP, route, at) + } + } +} + +// checkPrefixPage checks that the page of prefix shows each of wantPeers, the +// time of each kept route among them, and no time that was never set. +func checkPrefixPage( + t *testing.T, s *Server, prefix string, wantPeers []string, + keptAt map[string]time.Time, +) { + t.Helper() + + path := "/prefix/" + prefix + page := get(t, s, path).Body.String() + + for _, peer := range wantPeers { + want := `` + peer + `` + if !strings.Contains(page, want) { + t.Errorf("%s does not list the peer %s", path, peer) + } + + at, kept := keptAt[peer] + if kept && !strings.Contains(page, at.Format(routeTimeFormat)) { + t.Errorf("%s does not show the time %v of the route from %s", + path, at, peer) + } + } + + if strings.Contains(page, zeroDate) { + t.Errorf("%s shows the date %s", path, zeroDate) + } +} + +// TestASAnswersGiveEachPrefixItsNewestKeptRoute asks the AS API and page for +// mostPeersASN. Its prefixes are testPrefixV4, whose newest kept route with +// that origin AS is peer 3's, and noKeptRoutePrefix, which keeps no route. The +// first has that route's time. The second is listed without a time. +func TestASAnswersGiveEachPrefixItsNewestKeptRoute(t *testing.T) { + t.Parallel() + + s := newServerWithTable(t, newTwoOriginTable()) + path := "/api/v1/as/" + strconv.Itoa(mostPeersASN) + + //nolint:tagliatelle // field names are the existing JSON API + var body struct { + Data struct { + IPv4Prefixes []struct { + Prefix string `json:"prefix"` + LastUpdated *time.Time `json:"last_updated"` + } `json:"ipv4_prefixes"` + } `json:"data"` + } + + err := json.Unmarshal(get(t, s, path).Body.Bytes(), &body) + if err != nil { + t.Fatalf("%s: decoding the answer: %v", path, err) + } + + prefixes := body.Data.IPv4Prefixes + if len(prefixes) != 2 || prefixes[0].Prefix != testPrefixV4 || + prefixes[0].LastUpdated == nil || + !prefixes[0].LastUpdated.Equal(testRouteTime(3)) || + prefixes[1].Prefix != noKeptRoutePrefix || prefixes[1].LastUpdated != nil { + t.Errorf("%s: prefixes %+v, want %s updated %v and %s without a time", + path, prefixes, testPrefixV4, testRouteTime(3), noKeptRoutePrefix) + } + + path = "/as/" + strconv.Itoa(mostPeersASN) + page := get(t, s, path).Body.String() + + for _, want := range []string{ + testPrefixV4, noKeptRoutePrefix, testRouteTime(3).Format(routeTimeFormat), + } { + if !strings.Contains(page, want) { + t.Errorf("%s does not show %s", path, want) + } + } + + if strings.Contains(page, zeroDate) { + t.Errorf("%s shows the date %s", path, zeroDate) + } +} + +// TestPrefixLengthPagesShowTheOriginMostPeersCarry asks for the IPv4 /24 +// page and the IPv6 /32 page. The first lists both /24 prefixes with +// mostPeersASN, the origin AS more peers carry testPrefixV4 with, and gives +// noKeptRoutePrefix no age. The second lists testPrefixV6. +func TestPrefixLengthPagesShowTheOriginMostPeersCarry(t *testing.T) { + t.Parallel() + + s := newServerWithTable(t, newTwoOriginTable()) + + page := get(t, s, prefixLengthPagePath).Body.String() + + for _, want := range []string{ + ">" + testPrefixV4 + "<", ">" + noKeptRoutePrefix + "<", + `href="/as/` + strconv.Itoa(mostPeersASN) + `"`, + } { + if !strings.Contains(page, want) { + t.Errorf("%s does not show %s", prefixLengthPagePath, want) + } + } + + if strings.Contains(page, `href="/as/`+strconv.Itoa(fewerPeersASN)+`"`) { + t.Errorf("%s links to AS%d, which fewer peers carry", + prefixLengthPagePath, fewerPeersASN) + } + + const noAge = `` + if strings.Count(page, noAge) != 1 { + t.Errorf("%s gives %d prefixes no age, want 1 (%s)", + prefixLengthPagePath, strings.Count(page, noAge), noKeptRoutePrefix) + } + + const v6PagePath = "/prefixlength6/32" + + page = get(t, s, v6PagePath).Body.String() + if !strings.Contains(page, ">"+testPrefixV6+"<") || + !strings.Contains(page, `href="/as/`+strconv.Itoa(v6OriginASN)+`"`) { + t.Errorf("%s does not show %s with AS%d", v6PagePath, testPrefixV6, + v6OriginASN) + } +} diff --git a/internal/server/server.go b/internal/server/server.go index b0cb889..6d4aadc 100644 --- a/internal/server/server.go +++ b/internal/server/server.go @@ -3,6 +3,10 @@ // - the AS, prefix and prefix length pages, and the JSON answers behind them // - the IP lookups // - the AS name search +// +// Every route it shows comes from the routing table in memory. AS names, +// WHOIS details and the times a prefix and an AS were first seen come from +// the database. package server import ( @@ -19,6 +23,7 @@ import ( "git.eeqj.de/sneak/routewatch/internal/config" "git.eeqj.de/sneak/routewatch/internal/database" "git.eeqj.de/sneak/routewatch/internal/logger" + "git.eeqj.de/sneak/routewatch/internal/routetable" "git.eeqj.de/sneak/routewatch/internal/streamer" "github.com/go-chi/chi/v5" ) @@ -55,6 +60,7 @@ type dnsResolver interface { type Server struct { router *chi.Mux db database.Store + table *routetable.Table streamer *streamer.Streamer logger *logger.Logger port int @@ -71,15 +77,18 @@ type Server struct { resolver dnsResolver } -// New creates a new HTTP server +// New creates a new HTTP server that reads routes from table, the routing +// table that the prefix handler keeps current. func New( db database.Store, + table *routetable.Table, streamer *streamer.Streamer, logger *logger.Logger, cfg *config.Config, ) *Server { s := &Server{ db: db, + table: table, streamer: streamer, logger: logger, port: cfg.Port, diff --git a/internal/server/server_test.go b/internal/server/server_test.go index 1f6964b..5bfec36 100644 --- a/internal/server/server_test.go +++ b/internal/server/server_test.go @@ -34,7 +34,7 @@ func TestStartFailsWhenPortIsHeld(t *testing.T) { port := addr.Port log := logger.New() - s := server.New(nil, streamer.New(log, metrics.New()), log, + s := server.New(nil, nil, streamer.New(log, metrics.New()), log, &config.Config{Port: port}) defer func() { _ = s.Stop(context.Background()) }() diff --git a/internal/templates/as_detail.html b/internal/templates/as_detail.html index 9fa6951..984dcc6 100644 --- a/internal/templates/as_detail.html +++ b/internal/templates/as_detail.html @@ -281,8 +281,8 @@ {{.Prefix}} /{{.MaskLength}} - {{.LastUpdated.Format "2006-01-02 15:04:05"}} - {{.LastUpdated | timeSince}} + {{if not .LastUpdated.IsZero}}{{.LastUpdated.Format "2006-01-02 15:04:05"}}{{end}} + {{if not .LastUpdated.IsZero}}{{.LastUpdated | timeSince}}{{end}} {{end}} @@ -310,8 +310,8 @@ {{.Prefix}} /{{.MaskLength}} - {{.LastUpdated.Format "2006-01-02 15:04:05"}} - {{.LastUpdated | timeSince}} + {{if not .LastUpdated.IsZero}}{{.LastUpdated.Format "2006-01-02 15:04:05"}}{{end}} + {{if not .LastUpdated.IsZero}}{{.LastUpdated | timeSince}}{{end}} {{end}} diff --git a/internal/templates/prefix_detail.html b/internal/templates/prefix_detail.html index 18fe813..de44b3f 100644 --- a/internal/templates/prefix_detail.html +++ b/internal/templates/prefix_detail.html @@ -329,8 +329,8 @@ {{.PeerIP}} {{range $i, $as := .ASPathWithHandle}}{{if $i}} → {{end}}{{if $as.Handle}}{{$as.Handle}}{{else}}AS{{$as.ASN}}{{end}}{{end}} {{.NextHop}} - {{.LastUpdated.Format "2006-01-02 15:04:05"}} - {{.LastUpdated | timeSince}} + {{if not .LastUpdated.IsZero}}{{.LastUpdated.Format "2006-01-02 15:04:05"}}{{end}} + {{if not .LastUpdated.IsZero}}{{.LastUpdated | timeSince}}{{end}} {{end}} diff --git a/internal/templates/status.html b/internal/templates/status.html index 4061f7f..b277a22 100644 --- a/internal/templates/status.html +++ b/internal/templates/status.html @@ -318,6 +318,10 @@ IPv6 Routes - +
+ Origins Without a Kept Route + - +
IPv4 Updates/sec - @@ -571,6 +575,7 @@ document.getElementById('live_routes').textContent = '-'; document.getElementById('ipv4_routes').textContent = '-'; document.getElementById('ipv6_routes').textContent = '-'; + document.getElementById('origins_without_kept_route').textContent = '-'; document.getElementById('ipv4_updates_per_sec').textContent = '-'; document.getElementById('ipv6_updates_per_sec').textContent = '-'; document.getElementById('oldest_route').textContent = '-'; @@ -634,6 +639,7 @@ document.getElementById('live_routes').textContent = formatNumber(data.live_routes); document.getElementById('ipv4_routes').textContent = formatNumber(data.ipv4_routes); document.getElementById('ipv6_routes').textContent = formatNumber(data.ipv6_routes); + document.getElementById('origins_without_kept_route').textContent = formatNumber(data.origins_without_kept_route); document.getElementById('ipv4_updates_per_sec').textContent = formatNumber(Math.round(data.ipv4_updates_per_sec)); document.getElementById('ipv6_updates_per_sec').textContent = formatNumber(Math.round(data.ipv6_updates_per_sec)); document.getElementById('oldest_route').textContent = formatRelativeTime(data.oldest_route);