diff --git a/README.md b/README.md index 7c3629f..7b5d2ec 100644 --- a/README.md +++ b/README.md @@ -56,15 +56,17 @@ Every route answers `HEAD` as it answers `GET`, without the body. ### Web Interface -- `GET /` - Home page (HTML): headline figures, and lookups by AS number and by - IP address +- `GET /` - Home page (HTML): headline figures, lookups by AS number and by IP + address, and a search of AS names - `GET /status` - HTML status dashboard, which reads `/api/v1/stats` - `GET /status.json` - JSON statistics: feed, memory, database and route figures - `GET /.well-known/healthcheck.json` - Health check (JSON): 200 when the database answers and the RIS Live feed is connected, otherwise 503; `data.status` is `ok` or `error` and `data.checks` says which check failed - `GET /as/{asn}` - AS detail page (HTML): the prefixes the AS originates and - its peers + its peers. When the database has no record of the AS, its handle and + description come from the AS list built into RouteWatch. The page then shows + no first-seen date. - `GET /prefix/{prefix}/{length}` - Prefix detail page (HTML), such as `/prefix/192.0.2.0/24`: every live route for the prefix, one per peer, and its origin ASes @@ -93,9 +95,22 @@ Every route answers `HEAD` as it answers `GET`, without the body. and 404 when none of the addresses has a live route. A PTR lookup gets 2 seconds; an address whose lookup takes longer is answered without PTR names. - `GET /api/v1/as/{asn}` - The AS and the prefixes it originates, IPv4 and IPv6 - apart, each with when its newest route was updated + apart, each with when its newest route was updated. When the database has no + record of the AS, its handle and description come from the AS list built into + RouteWatch. The answer then has no `first_seen` or `last_seen`. - `GET /api/v1/prefix/{prefix}/{length}` - Every live route for the prefix, one per peer, and the same routes grouped by origin AS in `origins` +- `GET /api/v1/search/as?q={text}` - Finds the ASes whose handle or description + contains `{text}`, ignoring case. `data` holds: + - `query`: `{text}` without its leading and trailing spaces + - `total`: the number of ASes that match + - `results`: the first 50 of them in AS number order, each with `asn`, + `handle` and `description` + + The search reads the AS list built into RouteWatch, not the database. The + list comes from [ipverse/asn-info](https://github.com/ipverse/asn-info). A + match can therefore be an AS that the database has no record of. A `q` that + is missing, empty or only spaces answers 400. A JSON answer is `{"status": "ok", "data": ...}`, or for an error `{"status": "error", "error": {"msg": ..., "code": ...}}`, and carries an @@ -106,13 +121,16 @@ took. The health check is the exception: its 503 answer is also A request that runs past its time limit answers 504: the limit is 30 seconds, and 4 seconds for the database reads behind `/status.json` and `/api/v1/stats`. An AS number, prefix or mask length that does not parse answers 400, as does a -mask length out of range for its IP version. An AS the database does not hold, -or a prefix with no live route, answers 404. The HTML pages give these errors as -plain text. Under `/api/v1` and `/ip/`, a path no route matches answers 404 with -the message `Not found`, and `POST`, `PUT`, `PATCH`, `DELETE`, `OPTIONS`, -`TRACE` or `CONNECT` answers 405 with `Method not allowed`. A 500 answer says -only `Internal server error`, and a host name that does not resolve only -`could not resolve host name`; the detail goes to the log. +mask length out of range for its IP version. An AS answers 404 when it is not in +the AS list built into RouteWatch and the database has neither a record of it +nor a live route it originates. AS 0 always answers 404, because it means no +known origin. The AS name search never lists it. A prefix with no live route +answers 404 too. The HTML pages give these errors as plain text. Under `/api/v1` +and `/ip/`, a path no route matches answers 404 with the message `Not found`, +and `POST`, `PUT`, `PATCH`, `DELETE`, `OPTIONS`, `TRACE` or `CONNECT` answers +405 with `Method not allowed`. A 500 answer says only `Internal server error`, +and a host name that does not resolve only `could not resolve host name`; the +detail goes to the log. Every answer carries `Strict-Transport-Security`, `Content-Security-Policy`, `X-Frame-Options`, `X-Content-Type-Options`, `Referrer-Policy` and diff --git a/TODO.md b/TODO.md index a4f5e9a..6c89695 100644 --- a/TODO.md +++ b/TODO.md @@ -33,6 +33,11 @@ tested. # Completed Steps +- 2026-10-26: the home page's AS name search lists the ASes whose handle or + description holds the text, as links to their pages, from the new + `/api/v1/search/as`; the search ignores case. When the database has no record + of an AS, its page takes the handle and description from the AS list built + into RouteWatch and still lists the AS's prefixes (closes #151) - 2026-10-19: README "HTTP Endpoints" lists every route with what it answers, "Database Schema" matches `schema.sql`, and the rest of the README matches the code; this file brought up to date (closes #72) diff --git a/internal/database/database.go b/internal/database/database.go index caa7f69..b62d2a3 100644 --- a/internal/database/database.go +++ b/internal/database/database.go @@ -21,6 +21,7 @@ import ( "git.eeqj.de/sneak/routewatch/internal/config" "git.eeqj.de/sneak/routewatch/internal/logger" + "git.eeqj.de/sneak/routewatch/internal/ristypes" "git.eeqj.de/sneak/routewatch/pkg/asinfo" _ "github.com/mattn/go-sqlite3" // CGO SQLite driver ) @@ -1748,9 +1749,21 @@ func (d *Database) GetASDetails(asn int) (*ASN, []LiveRoute, error) { // GetASDetailsContext returns the AS and the prefixes it originates, the IPv4 // ones first, each once with the time of its newest route. It makes two index // searches per prefix, however many peers carry the prefix. +// +// The routes of an AS can be stored before its record in the asns table. When +// the AS has no record, its handle and description come from the AS info +// lookup, and it has no seen times. An AS with no record, no lookup entry and +// no prefix is ErrNoRoute. +// +// ristypes.NoOrigin means no known origin, not an AS, so it is ErrNoRoute +// whatever is stored. func (d *Database) GetASDetailsContext( ctx context.Context, asn int, ) (*ASN, []LiveRoute, error) { + if asn == ristypes.NoOrigin { + return nil, nil, fmt.Errorf("%w: AS%d", ErrNoRoute, asn) + } + // Get AS information var ( asnInfo ASN @@ -1763,17 +1776,28 @@ func (d *Database) GetASDetailsContext( asn, ).Scan(&asnInfo.ASN, &handle, &description, &asnInfo.FirstSeen, &asnInfo.LastSeen) - if err != nil { - if errors.Is(err, sql.ErrNoRows) { - return nil, nil, fmt.Errorf("%w: AS%d", ErrNoRoute, asn) + + // known is whether the asns table or the AS info lookup holds the AS. + known := true + + switch { + case errors.Is(err, sql.ErrNoRows): + asnInfo.ASN = asn + + info, ok := asinfo.Get(asn) + if ok { + asnInfo.Handle = info.Handle + asnInfo.Description = info.Description } + known = ok + case err != nil: return nil, nil, fmt.Errorf("failed to query AS: %w", err) + default: + asnInfo.Handle = handle.String + asnInfo.Description = description.String } - asnInfo.Handle = handle.String - asnInfo.Description = description.String - allPrefixes, err := d.originPrefixes(ctx, asn, ipVersionV4) if err != nil { return &asnInfo, nil, err @@ -1784,7 +1808,13 @@ func (d *Database) GetASDetailsContext( return &asnInfo, allPrefixes, err } - return &asnInfo, append(allPrefixes, prefixesV6...), nil + allPrefixes = append(allPrefixes, prefixesV6...) + + if !known && len(allPrefixes) == 0 { + return nil, nil, fmt.Errorf("%w: AS%d", ErrNoRoute, asn) + } + + return &asnInfo, allPrefixes, nil } // ASPeer represents a peering relationship with another AS including handle, diff --git a/internal/database/database_internal_test.go b/internal/database/database_internal_test.go index 2f8f242..e08e82d 100644 --- a/internal/database/database_internal_test.go +++ b/internal/database/database_internal_test.go @@ -20,6 +20,7 @@ import ( "git.eeqj.de/sneak/routewatch/internal/config" "git.eeqj.de/sneak/routewatch/internal/logger" + "git.eeqj.de/sneak/routewatch/pkg/asinfo" "github.com/mattn/go-sqlite3" ) @@ -1699,6 +1700,85 @@ func TestGetASDetailsListsEachPrefixOnce(t *testing.T) { } } +// TestGetASDetailsOfASNotInDatabase asks for ASes that the database has no +// record of, some with stored routes. Each gets the prefixes of its routes and +// no seen times. One that the AS info lookup knows gets its handle and +// description from it. An AS with no record, no lookup entry and no route is +// ErrNoRoute. Before the fix every AS with no record was ErrNoRoute. +func TestGetASDetailsOfASNotInDatabase(t *testing.T) { + t.Parallel() + + db, err := New(&config.Config{StateDir: t.TempDir()}, logger.New()) + if err != nil { + t.Fatalf("failed to create database: %v", err) + } + defer func() { _ = db.Close() }() + + ts := time.Date(2026, 1, 2, 3, 4, 5, 0, time.UTC) + + tests := []struct { + asn int + listed bool // in the AS info lookup + prefixes []string // of the routes stored for the AS + }{ + {15169, true, []string{"192.0.2.0/24"}}, + {13335, true, nil}, + {4200000000, false, []string{"198.51.100.0/24"}}, + } + + var routes []*LiveRoute + + for _, tt := range tests { + if _, ok := asinfo.Get(tt.asn); ok != tt.listed { + t.Fatalf("AS%d in the AS info data: %t, want %t", tt.asn, ok, tt.listed) + } + + for _, prefix := range tt.prefixes { + routes = append(routes, mkV4Route(t, prefix, tt.asn, ts)) + } + } + + err = db.UpsertLiveRouteBatch(routes) + if err != nil { + t.Fatalf("UpsertLiveRouteBatch: %v", err) + } + + for _, tt := range tests { + got, prefixes, err := db.GetASDetails(tt.asn) + if err != nil { + t.Errorf("GetASDetails(%d): %v", tt.asn, err) + + continue + } + + want := ASN{ASN: tt.asn} + if info, ok := asinfo.Get(tt.asn); ok { + want.Handle, want.Description = info.Handle, info.Description + } + + var gotPrefixes []string + for _, route := range prefixes { + gotPrefixes = append(gotPrefixes, route.Prefix) + } + + if *got != want || !slices.Equal(gotPrefixes, tt.prefixes) { + t.Errorf("GetASDetails(%d) = %+v with prefixes %v, want %+v with %v", + tt.asn, *got, gotPrefixes, want, tt.prefixes) + } + } + + const absentASN = 4200000001 // no record, no lookup entry and no route + + if _, ok := asinfo.Get(absentASN); ok { + t.Fatalf("AS%d is in the AS info data", absentASN) + } + + _, _, err = db.GetASDetails(absentASN) + if !errors.Is(err, ErrNoRoute) { + t.Errorf("GetASDetails(%d): error %v, want %v", absentASN, err, ErrNoRoute) + } +} + // TestASPrefixQueriesUseCoveringIndexes checks the plan of each query that // lists the prefixes an AS originates: a search of a covering index and no // temporary B-tree, so the list reads one index entry per prefix, not one diff --git a/internal/database/models.go b/internal/database/models.go index 8797ae7..9b94d8e 100644 --- a/internal/database/models.go +++ b/internal/database/models.go @@ -28,9 +28,10 @@ type ASN struct { RIRRegDate *time.Time `json:"rir_registration_date,omitempty"` RIRLastMod *time.Time `json:"rir_last_modified,omitempty"` WHOISRaw string `json:"whois_raw,omitempty"` - // Timestamps - FirstSeen time.Time `json:"first_seen"` - LastSeen time.Time `json:"last_seen"` + // Timestamps. They are zero, and left out of JSON, for an AS that has no + // record in the asns table. + FirstSeen time.Time `json:"first_seen,omitzero"` + LastSeen time.Time `json:"last_seen,omitzero"` WHOISUpdatedAt *time.Time `json:"whois_updated_at,omitempty"` } diff --git a/internal/ristypes/ris.go b/internal/ristypes/ris.go index a33f06a..9c40881 100644 --- a/internal/ristypes/ris.go +++ b/internal/ristypes/ris.go @@ -39,12 +39,16 @@ type ASPath struct { Set []int } +// NoOrigin is the origin AS of a route whose origin is not known. AS 0 is +// reserved, so no AS has this number. +const NoOrigin = 0 + // Origin returns the AS that originated the route: the last AS of the -// sequence, or 0 (none) when the path ends in a set of several members or is +// sequence, or NoOrigin when the path ends in a set of several members or is // empty. func (p *ASPath) Origin() int { if len(p.Set) > 0 || len(p.Sequence) == 0 { - return 0 + return NoOrigin } return p.Sequence[len(p.Sequence)-1] diff --git a/internal/server/handlers.go b/internal/server/handlers.go index 2132c0c..c33963d 100644 --- a/internal/server/handlers.go +++ b/internal/server/handlers.go @@ -16,6 +16,7 @@ import ( "git.eeqj.de/sneak/routewatch/internal/database" "git.eeqj.de/sneak/routewatch/internal/metrics" + "git.eeqj.de/sneak/routewatch/internal/ristypes" "git.eeqj.de/sneak/routewatch/internal/streamer" "git.eeqj.de/sneak/routewatch/internal/templates" asinfo "git.eeqj.de/sneak/routewatch/pkg/asinfo" @@ -56,10 +57,18 @@ const ( // Error messages an answer sends, in place of the error behind it where there // is one. const ( - invalidPrefixMessage = "Invalid prefix" - hostNotResolvedMessage = "could not resolve host name" - hostNotPublicMessage = "host name has no public address" - tooManyTargetsMessage = "Too many addresses or host names" + invalidPrefixMessage = "Invalid prefix" + hostNotResolvedMessage = "could not resolve host name" + hostNotPublicMessage = "host name has no public address" + tooManyTargetsMessage = "Too many addresses or host names" + searchQueryMissingMessage = "Missing search query" +) + +// The AS name search reads the text it looks for from the URL query parameter +// asSearchQueryParam, and answers with at most maxASSearchResults matches. +const ( + asSearchQueryParam = "q" + maxASSearchResults = 50 ) // IP versions as database.LiveRoute.IPVersion records them. @@ -1001,11 +1010,12 @@ func (s *Server) handleASDetailJSON() http.HandlerFunc { } } -// splitRoutesByIPVersion groups routes into the IPv4 and the IPv6 ones. +// splitRoutesByIPVersion groups routes into the IPv4 and the IPv6 ones. Neither +// group is nil, so that a JSON answer holds [] for an empty one and not null. func splitRoutesByIPVersion( routes []database.LiveRoute, ) ([]database.LiveRoute, []database.LiveRoute) { - var ipv4Routes, ipv6Routes []database.LiveRoute + ipv4Routes, ipv6Routes := []database.LiveRoute{}, []database.LiveRoute{} for _, route := range routes { if route.IPVersion == ipVersionV4 { @@ -1105,6 +1115,56 @@ func (s *Server) handlePrefixDetailJSON() http.HandlerFunc { } } +// ASSearchResponse is the answer to an AS name search: the text searched for, +// the number of ASes that match it, and the first maxASSearchResults of them in +// AS number order. +type ASSearchResponse struct { + Query string `json:"query"` + Total int `json:"total"` + Results []asinfo.Info `json:"results"` +} + +// handleASSearch returns a handler that searches the AS list built into +// asinfo for the text in the q parameter, without its leading and trailing +// spaces. It answers 400 when no text is left. +func (s *Server) handleASSearch() http.HandlerFunc { + return func(w http.ResponseWriter, r *http.Request) { + query := strings.TrimSpace(r.URL.Query().Get(asSearchQueryParam)) + if query == "" { + s.writeJSONError(w, http.StatusBadRequest, searchQueryMissingMessage) + + return + } + + // AS 0 means no known origin and has no AS page, so the answer leaves + // it out. The matches come in AS number order, so AS 0 can only be the + // first. The search asks for one match more than the answer holds, to + // take its place. + results, total := asinfo.Search(query, maxASSearchResults+1) + if len(results) > 0 && results[0].ASN == ristypes.NoOrigin { + results = results[1:] + total-- + } + + results = results[:min(len(results), maxASSearchResults)] + + // results is an empty list rather than nil when nothing matches, so + // that the answer holds [] and not null. + if results == nil { + results = []asinfo.Info{} + } + + err := writeJSONSuccess(w, ASSearchResponse{ + Query: query, + Total: total, + Results: results, + }) + if err != nil { + s.logger.Error("Failed to encode AS search results", "error", err) + } + } +} + // handleASDetail returns a handler that serves the AS detail HTML page func (s *Server) handleASDetail() http.HandlerFunc { return func(w http.ResponseWriter, r *http.Request) { diff --git a/internal/server/handlers_internal_test.go b/internal/server/handlers_internal_test.go index a734953..9736468 100644 --- a/internal/server/handlers_internal_test.go +++ b/internal/server/handlers_internal_test.go @@ -19,6 +19,7 @@ import ( "git.eeqj.de/sneak/routewatch/internal/database" "git.eeqj.de/sneak/routewatch/internal/logger" "git.eeqj.de/sneak/routewatch/internal/metrics" + "git.eeqj.de/sneak/routewatch/internal/ristypes" "git.eeqj.de/sneak/routewatch/internal/streamer" "git.eeqj.de/sneak/routewatch/pkg/asinfo" ) @@ -26,6 +27,7 @@ import ( // The prefix and peer addresses the prefix page tests store routes for. const ( testPrefixV4 = "192.0.2.0/24" + testPrefixV6 = "2001:db8::/32" testPeerV4 = "198.51.100.1" testPeerV6 = "2001:db8::1" ) @@ -194,7 +196,7 @@ func TestStatsHandlersAnswerFromMemory(t *testing.T) { } // newServerWithTestRoutes returns a server on a new database that holds a -// route for testPrefixV4 and one for 2001:db8::/32. +// route for testPrefixV4 and one for testPrefixV6. func newServerWithTestRoutes(t *testing.T) *Server { t.Helper() @@ -214,7 +216,7 @@ func newServerWithTestRoutes(t *testing.T) *Server { NextHop: testPeerV4, LastUpdated: ts, }, { - Prefix: "2001:db8::/32", MaskLength: 32, IPVersion: 6, + Prefix: testPrefixV6, MaskLength: 32, IPVersion: 6, OriginASN: 64501, PeerIP: testPeerV6, ASPath: []int{64501}, NextHop: testPeerV6, LastUpdated: ts, }, @@ -1166,6 +1168,389 @@ func TestIPLookupHidesPrivateAddresses(t *testing.T) { } } +// asSearchPath is the path of the AS name search. +const asSearchPath = "/api/v1/search/as" + +// searchAS sends an AS name search with the URL query rawQuery through s's +// router, expects a 200 answer, and returns its data. +func searchAS(t *testing.T, s *Server, rawQuery string) ASSearchResponse { + t.Helper() + + rec := httptest.NewRecorder() + s.router.ServeHTTP(rec, httptest.NewRequestWithContext( + t.Context(), http.MethodGet, asSearchPath+"?"+rawQuery, nil)) + + if rec.Code != http.StatusOK { + t.Fatalf("%s: status %d, want %d; body %s", + rawQuery, rec.Code, http.StatusOK, rec.Body) + } + + var body struct { + Data ASSearchResponse `json:"data"` + } + + err := json.Unmarshal(rec.Body.Bytes(), &body) + if err != nil { + t.Fatalf("%s: decoding the answer %q: %v", rawQuery, rec.Body, err) + } + + return body.Data +} + +// TestASSearchAnswersMatches searches for Cloudflare's handle in lower case, +// with spaces around it, and expects AS13335 among the matches. Before the fix +// there was no AS name search, and the home page's search card said it was +// coming soon (https://git.eeqj.de/sneak/routewatch/issues/151). +func TestASSearchAnswersMatches(t *testing.T) { + t.Parallel() + + const query = "cloudflarenet" + + data := searchAS(t, newServerWithTestRoutes(t), "q=%20"+query+"%20") + + if data.Query != query { + t.Errorf("query %q, want %q", data.Query, query) + } + + if data.Total != len(data.Results) { + t.Errorf("total %d, but %d results", data.Total, len(data.Results)) + } + + cloudflare := asinfo.Info{ + ASN: 13335, Handle: "CLOUDFLARENET", Description: "Cloudflare, Inc.", + } + if !slices.Contains(data.Results, cloudflare) { + t.Errorf("results %v do not hold %v", data.Results, cloudflare) + } +} + +// TestASSearchAnswersEmptyListWhenNothingMatches expects a search that matches +// nothing to answer 200 with a total of 0 and an empty list, not null. +func TestASSearchAnswersEmptyListWhenNothingMatches(t *testing.T) { + t.Parallel() + + data := searchAS(t, newServerWithTestRoutes(t), "q=XYZABC123456") + + if data.Total != 0 || data.Results == nil || len(data.Results) != 0 { + t.Errorf("total %d and results %#v, want 0 and an empty list", + data.Total, data.Results) + } +} + +// TestASSearchAnswers400ForEmptyQuery expects a search with no q, an empty q +// or a q of only spaces to answer 400 with the JSON error body. +func TestASSearchAnswers400ForEmptyQuery(t *testing.T) { + t.Parallel() + + s := newServerWithTestRoutes(t) + + for _, path := range []string{ + asSearchPath, asSearchPath + "?q=", asSearchPath + "?q=%20%20", + } { + rec := httptest.NewRecorder() + s.router.ServeHTTP(rec, httptest.NewRequestWithContext( + t.Context(), http.MethodGet, path, nil)) + + if rec.Code != http.StatusBadRequest { + t.Errorf("%s: status %d, want %d; body %s", + path, rec.Code, http.StatusBadRequest, rec.Body) + } + + checkJSONErrorCode(t, path, rec, http.StatusBadRequest) + checkJSONErrorMessage(t, path, rec, searchQueryMissingMessage) + } +} + +// TestASSearchShowsFirstMatchesAndTotal searches for a word with more matches +// than an answer holds, and expects the first maxASSearchResults of them, in +// AS number order, and the number of all of them. +func TestASSearchShowsFirstMatchesAndTotal(t *testing.T) { + t.Parallel() + + const query = "university" + + all, _ := asinfo.Search(query, asinfo.Total()) + if len(all) <= maxASSearchResults { + t.Fatalf("Search(%q) has %d matches; the test needs more than %d", + query, len(all), maxASSearchResults) + } + + data := searchAS(t, newServerWithTestRoutes(t), "q="+query) + + if data.Total != len(all) { + t.Errorf("total %d, want %d", data.Total, len(all)) + } + + if !slices.Equal(data.Results, all[:maxASSearchResults]) { + t.Errorf("results %v, want the first %d matches %v", + data.Results, maxASSearchResults, all[:maxASSearchResults]) + } +} + +// The AS numbers the AS detail tests ask for. storedASN originates the route +// for testPrefixV4 and is in the database. routeOnlyASN originates the route +// for testPrefixV6, but the database has no record of the AS itself, as when +// a route is stored before its AS. builtInOnlyASN is in the AS list built into +// asinfo but not in the database. unknownASN is in neither. +const ( + storedASN = 64500 + routeOnlyASN = 64501 + builtInOnlyASN = 15169 + unknownASN = 4200000000 +) + +// zeroDate is how a page writes a time that was never set. +const zeroDate = "0001-01-01" + +// firstSeenCard is the label of the AS page's first-seen date. +const firstSeenCard = `
Search for autonomous systems by organization name
+Search for autonomous systems by organization name or handle
@@ -387,24 +419,82 @@ } }); - // AS name search + // AS name search. The handles and descriptions come from outside data, + // so the results are built with textContent, never innerHTML. + const asNameResults = document.getElementById('asname-results'); + + // Replace the search results with one line of text. + function showASNameMessage(text, className) { + const line = document.createElement('p'); + line.className = className; + line.textContent = text; + asNameResults.replaceChildren(line); + } + + // List the matches of a search answer, each as a link to its AS page, + // and say how many more matched than the answer holds. + function showASNameMatches(data) { + if (data.total === 0) { + showASNameMessage('No AS matches "' + data.query + '".', 'search-message'); + return; + } + + const list = document.createElement('ul'); + list.className = 'asname-list'; + for (const match of data.results) { + const asn = document.createElement('span'); + asn.className = 'asn'; + asn.textContent = 'AS' + match.asn; + + const description = document.createElement('span'); + description.className = 'description'; + description.textContent = match.description; + + const link = document.createElement('a'); + link.href = '/as/' + match.asn; + link.append(asn, ' ' + match.handle, description); + + const item = document.createElement('li'); + item.append(link); + list.append(item); + } + asNameResults.replaceChildren(list); + + const more = data.total - data.results.length; + if (more > 0) { + const line = document.createElement('p'); + line.className = 'search-message'; + line.textContent = more.toLocaleString() + ' more matched. Narrow the search to see them.'; + asNameResults.append(line); + } + } + document.getElementById('asname-form').addEventListener('submit', function(e) { e.preventDefault(); const query = document.getElementById('asname-input').value.trim(); if (!query) return; - const resultsDiv = document.getElementById('asname-results'); - resultsDiv.innerHTML = 'Searching...
'; - - // Use a simple client-side search against the asinfo data - // For now, redirect to AS page if it looks like an ASN - if (/^\d+$/.test(query)) { - window.location.href = '/as/' + query; + // A number, or AS and a number, opens that AS's page + const asn = query.replace(/^AS/i, ''); + if (/^\d+$/.test(asn)) { + window.location.href = '/as/' + asn; return; } - // Show a message that server-side search is coming - resultsDiv.innerHTML = ''; + showASNameMessage('Searching...', 'loading'); + + fetch('/api/v1/search/as?q=' + encodeURIComponent(query)) + .then(response => response.json()) + .then(response => { + if (response.status === 'error') { + showASNameMessage('Error: ' + response.error.msg, 'error'); + } else { + showASNameMatches(response.data); + } + }) + .catch(error => { + showASNameMessage('Error: ' + error.message, 'error'); + }); }); // IP lookup diff --git a/pkg/asinfo/asinfo.go b/pkg/asinfo/asinfo.go index 372f2e1..a617edd 100644 --- a/pkg/asinfo/asinfo.go +++ b/pkg/asinfo/asinfo.go @@ -2,10 +2,13 @@ package asinfo import ( "bytes" + "cmp" "compress/gzip" _ "embed" "encoding/json" "io" + "slices" + "strings" "sync" ) @@ -23,6 +26,17 @@ type Info struct { type Registry struct { mu sync.RWMutex byASN map[int]*Info + // searchEntries holds every entry in AS number order, for Search. + searchEntries []searchEntry +} + +// searchEntry is an entry with its handle and description in lower case. They +// are put in lower case once, when the data loads, so that Search only +// compares. +type searchEntry struct { + info *Info + lowerHandle string + lowerDescription string } var ( @@ -59,9 +73,21 @@ func initRegistry() { panic("failed to unmarshal embedded AS data: " + err.Error()) } + slices.SortFunc(asInfos, func(a, b Info) int { + return cmp.Compare(a.ASN, b.ASN) + }) + + defaultRegistry.searchEntries = make([]searchEntry, 0, len(asInfos)) + for i := range asInfos { info := &asInfos[i] defaultRegistry.byASN[info.ASN] = info + defaultRegistry.searchEntries = append(defaultRegistry.searchEntries, + searchEntry{ + info: info, + lowerHandle: strings.ToLower(info.Handle), + lowerDescription: strings.ToLower(info.Description), + }) } } @@ -129,36 +155,36 @@ func All() []Info { return result } -// Search searches for AS entries where the handle or description contains the query -// This is a simple case-sensitive substring search -func Search(query string) []Info { +// Search finds the AS entries whose handle or description contains query, +// ignoring case. It returns the first limit of them in AS number order, and the +// number of entries that match. An empty query matches nothing. +func Search(query string, limit int) ([]Info, int) { once.Do(initRegistry) + if query == "" { + return nil, 0 + } + + query = strings.ToLower(query) + defaultRegistry.mu.RLock() defer defaultRegistry.mu.RUnlock() - var results []Info + var ( + results []Info + total int + ) - for _, info := range defaultRegistry.byASN { - if contains(info.Handle, query) || contains(info.Description, query) { - results = append(results, *info) + for _, entry := range defaultRegistry.searchEntries { + if strings.Contains(entry.lowerHandle, query) || + strings.Contains(entry.lowerDescription, query) { + if total < limit { + results = append(results, *entry.info) + } + + total++ } } - return results -} - -// contains is a simple substring search -func contains(s, substr string) bool { - return len(substr) > 0 && len(s) >= len(substr) && containsImpl(s, substr) -} - -func containsImpl(s, substr string) bool { - for i := 0; i <= len(s)-len(substr); i++ { - if s[i:i+len(substr)] == substr { - return true - } - } - - return false + return results, total } diff --git a/pkg/asinfo/asinfo_test.go b/pkg/asinfo/asinfo_test.go index 3dabfa6..f87d697 100644 --- a/pkg/asinfo/asinfo_test.go +++ b/pkg/asinfo/asinfo_test.go @@ -1,6 +1,8 @@ package asinfo_test import ( + "cmp" + "slices" "strings" "testing" @@ -140,11 +142,6 @@ func TestSearch(t *testing.T) { query: "University", wantMin: 10, }, - { - name: "Empty search", - query: "", - wantMin: 0, - }, { name: "Non-existent org", query: "XYZABC123456", @@ -156,26 +153,115 @@ func TestSearch(t *testing.T) { t.Run(tt.name, func(t *testing.T) { t.Parallel() - results := asinfo.Search(tt.query) + results, _ := asinfo.Search(tt.query, asinfo.Total()) if len(results) < tt.wantMin { t.Errorf("Search(%q) returned %d results, want at least %d", tt.query, len(results), tt.wantMin) } - // Verify all results contain the query - if tt.query != "" { - for _, r := range results { - if !strings.Contains(r.Handle, tt.query) && - !strings.Contains(r.Description, tt.query) { - t.Errorf("Search result ASN %d doesn't contain query %q", - r.ASN, tt.query) - } + // Verify all results contain the query, ignoring case + query := strings.ToLower(tt.query) + for _, r := range results { + if !strings.Contains(strings.ToLower(r.Handle), query) && + !strings.Contains(strings.ToLower(r.Description), query) { + t.Errorf("Search result ASN %d doesn't contain query %q", + r.ASN, tt.query) } } }) } } +// TestSearchEmptyQueryMatchesNothing expects an empty query to match no AS, +// although every handle contains the empty string. +func TestSearchEmptyQueryMatchesNothing(t *testing.T) { + t.Parallel() + + results, total := asinfo.Search("", asinfo.Total()) + if len(results) != 0 || total != 0 { + t.Errorf(`Search("") returned %d results and a total of %d, want none`, + len(results), total) + } +} + +// TestSearchIgnoresCase searches for Cloudflare's handle in lower case and its +// description in upper case, and expects both searches to find AS13335. +func TestSearchIgnoresCase(t *testing.T) { + t.Parallel() + + const cloudflareASN = 13335 // handle CLOUDFLARENET, "Cloudflare, Inc." + + isCloudflare := func(info asinfo.Info) bool { + return info.ASN == cloudflareASN + } + + for _, query := range []string{"cloudflarenet", "CLOUDFLARE, INC."} { + results, _ := asinfo.Search(query, asinfo.Total()) + if !slices.ContainsFunc(results, isCloudflare) { + t.Errorf("Search(%q) did not find AS%d", query, cloudflareASN) + } + } +} + +// TestSearchOrdersByASN expects the matches of a search with many of them to +// come in AS number order. +func TestSearchOrdersByASN(t *testing.T) { + t.Parallel() + + const query = "University" + + results, _ := asinfo.Search(query, asinfo.Total()) + + for i := 1; i < len(results); i++ { + if results[i-1].ASN >= results[i].ASN { + t.Fatalf("Search(%q): AS%d comes after AS%d", + query, results[i].ASN, results[i-1].ASN) + } + } +} + +// TestSearchStopsAtLimit searches for a word that matches more entries than +// the limit. It expects exactly limit entries, the first matches in AS number +// order, and the number of all matches, counted here from All. +func TestSearchStopsAtLimit(t *testing.T) { + t.Parallel() + + const ( + query = "university" + limit = 10 + ) + + var matches []asinfo.Info + + for _, info := range asinfo.All() { + if strings.Contains(strings.ToLower(info.Handle), query) || + strings.Contains(strings.ToLower(info.Description), query) { + matches = append(matches, info) + } + } + + if len(matches) <= limit { + t.Fatalf("%q matches %d entries; the test needs more than %d", + query, len(matches), limit) + } + + slices.SortFunc(matches, func(a, b asinfo.Info) int { + return cmp.Compare(a.ASN, b.ASN) + }) + + results, total := asinfo.Search(query, limit) + + if total != len(matches) { + t.Errorf("Search(%q, %d): total %d, want %d", + query, limit, total, len(matches)) + } + + if !slices.Equal(results, matches[:limit]) { + t.Errorf("Search(%q, %d) = %v, want %v", + query, limit, results, matches[:limit]) + } +} + func TestDataIntegrity(t *testing.T) { t.Parallel() @@ -215,11 +301,13 @@ func BenchmarkGet(b *testing.B) { } func BenchmarkSearch(b *testing.B) { + const limit = 50 + queries := []string{"Google", "Amazon", "Microsoft", "University"} b.ResetTimer() for i := range b.N { - asinfo.Search(queries[i%len(queries)]) + asinfo.Search(queries[i%len(queries)], limit) } } diff --git a/pkg/asinfo/doc.go b/pkg/asinfo/doc.go index c2b4074..c26823c 100644 --- a/pkg/asinfo/doc.go +++ b/pkg/asinfo/doc.go @@ -16,8 +16,10 @@ Basic usage: // Get just the description desc := asinfo.GetDescription(15169) // "Google LLC" - // Search for AS entries - results := asinfo.Search("University") + // Search the handles and descriptions, ignoring case, for the first 10 + // matches in AS number order and the number of all matches + results, total := asinfo.Search("university", 10) + fmt.Printf("%d matches\n", total) for _, as := range results { fmt.Printf("AS%d: %s\n", as.ASN, as.Description) } diff --git a/pkg/asinfo/example_test.go b/pkg/asinfo/example_test.go index 9b6f831..9612e80 100644 --- a/pkg/asinfo/example_test.go +++ b/pkg/asinfo/example_test.go @@ -21,9 +21,16 @@ func ExampleGetDescription() { } func ExampleSearch() { - results := asinfo.Search("MIT-GATEWAY") + const limit = 10 + + results, total := asinfo.Search("mit-gateway", limit) + + fmt.Printf("%d of %d matches\n", len(results), total) + for _, info := range results { fmt.Printf("AS%d: %s - %s\n", info.ASN, info.Handle, info.Description) } - // Output: AS3: MIT-GATEWAYS - Massachusetts Institute of Technology + // Output: + // 1 of 1 matches + // AS3: MIT-GATEWAYS - Massachusetts Institute of Technology }