4 Commits
Author SHA1 Message Date
clawbot 087b8de9ae Back off WHOIS lookups more slowly and stop asking for personal data (closes #159)
check / check (push) Waiting to run
A failed lookup now makes the fetcher's wait 1.5 times as long, up to
one minute instead of five. A success still halves it.

RIPE counts the person and role objects an answer holds against a daily
limit per IP address, and refuses an address for good that passes it
too often. RIPE, APNIC and AFRINIC are now asked with --no-personal,
which leaves those objects out but keeps the organisation object.

An answer with RIPE's access-denied or limit-reached error line is now
reported as a refusal holding that line, not as an answer with no
fields.

Model: opus-5-5
2026-10-26 23:40:13 +01:00
clawbot 99c7efc156 Make the home page's AS name search work (closes #151)
check / check (push) Canceled after 0s
The AS Name Search card on the home page said the search was coming soon.
It now asks the new GET /api/v1/search/as?q=<text> and lists the matches
as links to their AS pages. asinfo.Search ignores case, returns the first
matches up to a limit in AS number order, and counts all matches. An
empty q answers 400.

When the database has no record of an AS, its page and /api/v1/as/{asn}
take its handle and description from asinfo and still list its prefixes.
They answer 404 only for an AS with no record, no asinfo entry and no
prefix. AS 0 means no known origin: it always answers 404, and the search
never lists it.

Model: opus-5-5
2026-10-26 22:51:12 +01:00
clawbot 71ee2aeda7 Log the start-up checkpoint's figures under their own names (closes #117)
check / check (push) Canceled after 0s
Initialize scanned the row of PRAGMA wal_checkpoint(TRUNCATE), which is
the busy flag, the frames in the WAL and the frames copied, into
wal_pages, checkpointed and moved; and a completed TRUNCATE reads 0 for
both counts. The line now logs busy, wal_frames and checkpointed_frames,
plus wal_bytes, the WAL file's size, which New reads before it opens the
database because the open writes a frame to the WAL. The line is a
warning when busy is 1. Tests open a database a second time while the
first stays open, with and without a read open on the first, and check
the line's level and figures.

Model: opus-5-5
2026-10-20 04:32:08 +02:00
clawbot 8d16bf9ea6 Add the Rationale section to README (closes #121)
check / check (push) Canceled after 0s
REPO_POLICIES.md requires a Rationale section saying why the program
exists. RIS Live sends only the changes to the routing table, so it
cannot answer which AS an address is routed to now; routewatch applies
the changes to its own tables and answers from them. The section says
those tables hold only the routes announced since routewatch first
started, and miss the updates it drops under load or while it is
stopped.

Model: opus-5-5
2026-10-20 04:24:41 +02:00
20 changed files with 1424 additions and 139 deletions
+45 -11
View File
@@ -5,6 +5,22 @@ table in real time: it streams BGP UPDATE messages from the RIPE RIS Live
service, maintains a live routing table in SQLite, and provides HTTP APIs for service, maintains a live routing table in SQLite, and provides HTTP APIs for
querying routing information. querying routing information.
## Rationale
RIPE RIS Live streams the BGP updates that RIPE's route collectors receive, as
they happen, from the moment a client connects. It reports changes, not the
routing table they add up to, so it cannot say which AS an IP address is routed
to now or which routes a prefix has. RouteWatch applies the feed's updates to
live routing tables in its own SQLite database and answers IP-to-AS lookups and
prefix pages from them.
The feed sends no table when a client connects, so these tables hold only the
routes the feed has announced since RouteWatch first started: an IP-to-AS lookup
finds nothing for an address whose routes have not been announced since then.
The tables also miss the updates RouteWatch drops when it falls behind under
load and those sent while it is stopped, so a lookup or a prefix page can show a
route the feed has withdrawn.
## Features ## Features
- Real-time streaming of BGP updates from RIPE RIS Live - Real-time streaming of BGP updates from RIPE RIS Live
@@ -40,15 +56,17 @@ Every route answers `HEAD` as it answers `GET`, without the body.
### Web Interface ### Web Interface
- `GET /` - Home page (HTML): headline figures, and lookups by AS number and by - `GET /` - Home page (HTML): headline figures, lookups by AS number and by IP
IP address address, and a search of AS names
- `GET /status` - HTML status dashboard, which reads `/api/v1/stats` - `GET /status` - HTML status dashboard, which reads `/api/v1/stats`
- `GET /status.json` - JSON statistics: feed, memory, database and route figures - `GET /status.json` - JSON statistics: feed, memory, database and route figures
- `GET /.well-known/healthcheck.json` - Health check (JSON): 200 when the - `GET /.well-known/healthcheck.json` - Health check (JSON): 200 when the
database answers and the RIS Live feed is connected, otherwise 503; database answers and the RIS Live feed is connected, otherwise 503;
`data.status` is `ok` or `error` and `data.checks` says which check failed `data.status` is `ok` or `error` and `data.checks` says which check failed
- `GET /as/{asn}` - AS detail page (HTML): the prefixes the AS originates and - `GET /as/{asn}` - AS detail page (HTML): the prefixes the AS originates and
its peers its peers. When the database has no record of the AS, its handle and
description come from the AS list built into RouteWatch. The page then shows
no first-seen date.
- `GET /prefix/{prefix}/{length}` - Prefix detail page (HTML), such as - `GET /prefix/{prefix}/{length}` - Prefix detail page (HTML), such as
`/prefix/192.0.2.0/24`: every live route for the prefix, one per peer, and its `/prefix/192.0.2.0/24`: every live route for the prefix, one per peer, and its
origin ASes origin ASes
@@ -77,9 +95,22 @@ Every route answers `HEAD` as it answers `GET`, without the body.
and 404 when none of the addresses has a live route. A PTR lookup gets 2 and 404 when none of the addresses has a live route. A PTR lookup gets 2
seconds; an address whose lookup takes longer is answered without PTR names. seconds; an address whose lookup takes longer is answered without PTR names.
- `GET /api/v1/as/{asn}` - The AS and the prefixes it originates, IPv4 and IPv6 - `GET /api/v1/as/{asn}` - The AS and the prefixes it originates, IPv4 and IPv6
apart, each with when its newest route was updated apart, each with when its newest route was updated. When the database has no
record of the AS, its handle and description come from the AS list built into
RouteWatch. The answer then has no `first_seen` or `last_seen`.
- `GET /api/v1/prefix/{prefix}/{length}` - Every live route for the prefix, one - `GET /api/v1/prefix/{prefix}/{length}` - Every live route for the prefix, one
per peer, and the same routes grouped by origin AS in `origins` per peer, and the same routes grouped by origin AS in `origins`
- `GET /api/v1/search/as?q={text}` - Finds the ASes whose handle or description
contains `{text}`, ignoring case. `data` holds:
- `query`: `{text}` without its leading and trailing spaces
- `total`: the number of ASes that match
- `results`: the first 50 of them in AS number order, each with `asn`,
`handle` and `description`
The search reads the AS list built into RouteWatch, not the database. The
list comes from [ipverse/asn-info](https://github.com/ipverse/asn-info). A
match can therefore be an AS that the database has no record of. A `q` that
is missing, empty or only spaces answers 400.
A JSON answer is `{"status": "ok", "data": ...}`, or for an error A JSON answer is `{"status": "ok", "data": ...}`, or for an error
`{"status": "error", "error": {"msg": ..., "code": ...}}`, and carries an `{"status": "error", "error": {"msg": ..., "code": ...}}`, and carries an
@@ -90,13 +121,16 @@ took. The health check is the exception: its 503 answer is also
A request that runs past its time limit answers 504: the limit is 30 seconds, A request that runs past its time limit answers 504: the limit is 30 seconds,
and 4 seconds for the database reads behind `/status.json` and `/api/v1/stats`. and 4 seconds for the database reads behind `/status.json` and `/api/v1/stats`.
An AS number, prefix or mask length that does not parse answers 400, as does a An AS number, prefix or mask length that does not parse answers 400, as does a
mask length out of range for its IP version. An AS the database does not hold, mask length out of range for its IP version. An AS answers 404 when it is not in
or a prefix with no live route, answers 404. The HTML pages give these errors as the AS list built into RouteWatch and the database has neither a record of it
plain text. Under `/api/v1` and `/ip/`, a path no route matches answers 404 with nor a live route it originates. AS 0 always answers 404, because it means no
the message `Not found`, and `POST`, `PUT`, `PATCH`, `DELETE`, `OPTIONS`, known origin. The AS name search never lists it. A prefix with no live route
`TRACE` or `CONNECT` answers 405 with `Method not allowed`. A 500 answer says answers 404 too. The HTML pages give these errors as plain text. Under `/api/v1`
only `Internal server error`, and a host name that does not resolve only and `/ip/`, a path no route matches answers 404 with the message `Not found`,
`could not resolve host name`; the detail goes to the log. and `POST`, `PUT`, `PATCH`, `DELETE`, `OPTIONS`, `TRACE` or `CONNECT` answers
405 with `Method not allowed`. A 500 answer says only `Internal server error`,
and a host name that does not resolve only `could not resolve host name`; the
detail goes to the log.
Every answer carries `Strict-Transport-Security`, `Content-Security-Policy`, Every answer carries `Strict-Transport-Security`, `Content-Security-Policy`,
`X-Frame-Options`, `X-Content-Type-Options`, `Referrer-Policy` and `X-Frame-Options`, `X-Content-Type-Options`, `Referrer-Policy` and
+10
View File
@@ -33,6 +33,16 @@ tested.
# Completed Steps # Completed Steps
- 2026-10-26: the WHOIS fetcher backs off more slowly. A failure makes its wait
1.5 times as long, up to one minute instead of five. Its queries to RIPE,
APNIC and AFRINIC leave out person and role objects, which RIPE counts against
a daily limit per address. A refused query is logged with the RIR's error line
(closes #159)
- 2026-10-26: the home page's AS name search lists the ASes whose handle or
description holds the text, as links to their pages, from the new
`/api/v1/search/as`; the search ignores case. When the database has no record
of an AS, its page takes the handle and description from the AS list built
into RouteWatch and still lists the AS's prefixes (closes #151)
- 2026-10-19: README "HTTP Endpoints" lists every route with what it answers, - 2026-10-19: README "HTTP Endpoints" lists every route with what it answers,
"Database Schema" matches `schema.sql`, and the rest of the README matches the "Database Schema" matches `schema.sql`, and the rest of the README matches the
code; this file brought up to date (closes #72) code; this file brought up to date (closes #72)
+85 -19
View File
@@ -21,6 +21,7 @@ import (
"git.eeqj.de/sneak/routewatch/internal/config" "git.eeqj.de/sneak/routewatch/internal/config"
"git.eeqj.de/sneak/routewatch/internal/logger" "git.eeqj.de/sneak/routewatch/internal/logger"
"git.eeqj.de/sneak/routewatch/internal/ristypes"
"git.eeqj.de/sneak/routewatch/pkg/asinfo" "git.eeqj.de/sneak/routewatch/pkg/asinfo"
_ "github.com/mattn/go-sqlite3" // CGO SQLite driver _ "github.com/mattn/go-sqlite3" // CGO SQLite driver
) )
@@ -169,6 +170,20 @@ func New(cfg *config.Config, logger *logger.Logger) (*Database, error) {
return nil, fmt.Errorf("failed to create database directory: %w", err) return nil, fmt.Errorf("failed to create database directory: %w", err)
} }
// The size of the WAL file the previous run left, which Initialize logs
// with the start-up checkpoint. It is read before the database is opened,
// because opening it writes a frame to the WAL. There is no WAL file when
// the database is new or the last run closed the database, which deletes
// it.
var walBytes int64
wal, err := os.Stat(dbPath + walFileSuffix)
if err == nil {
walBytes = wal.Size()
} else if !errors.Is(err, os.ErrNotExist) {
logger.Warn("Failed to read the WAL file's size on startup", "error", err)
}
// Per-connection SQLite settings go in the DSN so every pooled connection // Per-connection SQLite settings go in the DSN so every pooled connection
// gets them, not just the one that runs the Initialize pragmas. _txlock= // gets them, not just the one that runs the Initialize pragmas. _txlock=
// immediate makes every transaction take the write lock at BEGIN. Without it // immediate makes every transaction take the write lock at BEGIN. Without it
@@ -229,7 +244,7 @@ func New(cfg *config.Config, logger *logger.Logger) (*Database, error) {
counts: &liveCounts{}, counts: &liveCounts{},
} }
err = database.Initialize() err = database.Initialize(walBytes)
if err != nil { if err != nil {
return nil, fmt.Errorf("failed to initialize database: %w", err) return nil, fmt.Errorf("failed to initialize database: %w", err)
} }
@@ -245,8 +260,16 @@ func New(cfg *config.Config, logger *logger.Logger) (*Database, error) {
return database, nil return database, nil
} }
// Initialize creates the database schema if it doesn't exist. // The messages Initialize logs the start-up WAL checkpoint with.
func (d *Database) Initialize() error { const (
startupCheckpointMessage = "WAL checkpoint on startup"
startupCheckpointIncompleteMessage = "WAL checkpoint on startup incomplete"
)
// Initialize creates the database schema if it doesn't exist. walBytes is the
// size of the WAL file the previous run left, which New reads before it opens
// the database.
func (d *Database) Initialize(walBytes int64) error {
// Set SQLite pragmas for performance. Per-connection settings (cache_size, // Set SQLite pragmas for performance. Per-connection settings (cache_size,
// synchronous, busy_timeout, journal_mode) live in the DSN, and New sets the // synchronous, busy_timeout, journal_mode) live in the DSN, and New sets the
// writing connection's cache_size; temp_store is left at its default so // writing connection's cache_size; temp_store is left at its default so
@@ -266,19 +289,33 @@ func (d *Database) Initialize() error {
} }
} }
// Run WAL checkpoint on startup to consolidate any existing WAL // The start-up checkpoint copies the WAL a previous run left into the
var walPages, checkpointedPages, movedPages int // database file and empties the WAL file; its frame counts then read 0,
// so walBytes is logged with them.
var busy, walFrames, checkpointedFrames int
err := d.writer.QueryRowContext(context.Background(), err := d.writer.QueryRowContext(context.Background(),
"PRAGMA wal_checkpoint(TRUNCATE)", "PRAGMA wal_checkpoint(TRUNCATE)",
).Scan(&walPages, &checkpointedPages, &movedPages) ).Scan(&busy, &walFrames, &checkpointedFrames)
if err != nil {
switch {
case err != nil:
d.logger.Warn("Failed to checkpoint WAL on startup", "error", err) d.logger.Warn("Failed to checkpoint WAL on startup", "error", err)
} else { case busy != 0:
d.logger.Info("WAL checkpoint on startup", // Another process using the database kept the checkpoint from
"wal_pages", walPages, // copying or emptying all of the WAL.
"checkpointed", checkpointedPages, d.logger.Warn(startupCheckpointIncompleteMessage,
"moved", movedPages, "wal_bytes", walBytes,
"busy", busy,
"wal_frames", walFrames,
"checkpointed_frames", checkpointedFrames,
)
default:
d.logger.Info(startupCheckpointMessage,
"wal_bytes", walBytes,
"busy", busy,
"wal_frames", walFrames,
"checkpointed_frames", checkpointedFrames,
) )
} }
@@ -1712,9 +1749,21 @@ func (d *Database) GetASDetails(asn int) (*ASN, []LiveRoute, error) {
// GetASDetailsContext returns the AS and the prefixes it originates, the IPv4 // GetASDetailsContext returns the AS and the prefixes it originates, the IPv4
// ones first, each once with the time of its newest route. It makes two index // ones first, each once with the time of its newest route. It makes two index
// searches per prefix, however many peers carry the prefix. // searches per prefix, however many peers carry the prefix.
//
// The routes of an AS can be stored before its record in the asns table. When
// the AS has no record, its handle and description come from the AS info
// lookup, and it has no seen times. An AS with no record, no lookup entry and
// no prefix is ErrNoRoute.
//
// ristypes.NoOrigin means no known origin, not an AS, so it is ErrNoRoute
// whatever is stored.
func (d *Database) GetASDetailsContext( func (d *Database) GetASDetailsContext(
ctx context.Context, asn int, ctx context.Context, asn int,
) (*ASN, []LiveRoute, error) { ) (*ASN, []LiveRoute, error) {
if asn == ristypes.NoOrigin {
return nil, nil, fmt.Errorf("%w: AS%d", ErrNoRoute, asn)
}
// Get AS information // Get AS information
var ( var (
asnInfo ASN asnInfo ASN
@@ -1727,17 +1776,28 @@ func (d *Database) GetASDetailsContext(
asn, asn,
).Scan(&asnInfo.ASN, &handle, &description, &asnInfo.FirstSeen, ).Scan(&asnInfo.ASN, &handle, &description, &asnInfo.FirstSeen,
&asnInfo.LastSeen) &asnInfo.LastSeen)
if err != nil {
if errors.Is(err, sql.ErrNoRows) { // known is whether the asns table or the AS info lookup holds the AS.
return nil, nil, fmt.Errorf("%w: AS%d", ErrNoRoute, asn) known := true
switch {
case errors.Is(err, sql.ErrNoRows):
asnInfo.ASN = asn
info, ok := asinfo.Get(asn)
if ok {
asnInfo.Handle = info.Handle
asnInfo.Description = info.Description
} }
known = ok
case err != nil:
return nil, nil, fmt.Errorf("failed to query AS: %w", err) return nil, nil, fmt.Errorf("failed to query AS: %w", err)
default:
asnInfo.Handle = handle.String
asnInfo.Description = description.String
} }
asnInfo.Handle = handle.String
asnInfo.Description = description.String
allPrefixes, err := d.originPrefixes(ctx, asn, ipVersionV4) allPrefixes, err := d.originPrefixes(ctx, asn, ipVersionV4)
if err != nil { if err != nil {
return &asnInfo, nil, err return &asnInfo, nil, err
@@ -1748,7 +1808,13 @@ func (d *Database) GetASDetailsContext(
return &asnInfo, allPrefixes, err return &asnInfo, allPrefixes, err
} }
return &asnInfo, append(allPrefixes, prefixesV6...), nil allPrefixes = append(allPrefixes, prefixesV6...)
if !known && len(allPrefixes) == 0 {
return nil, nil, fmt.Errorf("%w: AS%d", ErrNoRoute, asn)
}
return &asnInfo, allPrefixes, nil
} }
// ASPeer represents a peering relationship with another AS including handle, // ASPeer represents a peering relationship with another AS including handle,
+252
View File
@@ -1,8 +1,10 @@
package database package database
import ( import (
"bytes"
"context" "context"
"database/sql" "database/sql"
"encoding/json"
"errors" "errors"
"fmt" "fmt"
"log/slog" "log/slog"
@@ -18,6 +20,7 @@ import (
"git.eeqj.de/sneak/routewatch/internal/config" "git.eeqj.de/sneak/routewatch/internal/config"
"git.eeqj.de/sneak/routewatch/internal/logger" "git.eeqj.de/sneak/routewatch/internal/logger"
"git.eeqj.de/sneak/routewatch/pkg/asinfo"
"github.com/mattn/go-sqlite3" "github.com/mattn/go-sqlite3"
) )
@@ -1052,6 +1055,176 @@ func walFileSize(t *testing.T, db *Database) int64 {
return wal.Size() return wal.Size()
} }
// TestStartupCheckpointLogsItsFigures opens a new database, writes a route,
// and opens the same file again while the first stays open, so that its WAL
// file stays. The first open finds no WAL file: it logs a size of 0 and no
// warning. The second logs the size of the WAL file the first left, and the
// checkpoint's busy flag and frame counts, each under its own name; the
// counts read 0 once the checkpoint has emptied the file.
// Initialize logged the busy flag as wal_pages and the counts under each
// other's names (https://git.eeqj.de/sneak/routewatch/issues/117).
func TestStartupCheckpointLogsItsFigures(t *testing.T) {
t.Parallel()
cfg := &config.Config{StateDir: t.TempDir()}
db, lines := openLogged(t, cfg)
for _, line := range lines {
if line[slog.LevelKey] == slog.LevelWarn.String() {
t.Errorf("opening a new database logged a warning: %v", line)
}
}
line := findLine(t, lines, startupCheckpointMessage)
if line["wal_bytes"] != 0.0 {
t.Errorf("new database: start-up checkpoint logged wal_bytes = %v, "+
"want 0", line["wal_bytes"])
}
err := db.UpsertLiveRouteBatch([]*LiveRoute{
mkV4Route(t, "192.0.2.0/24", lockTestASN, time.Now().UTC()),
})
if err != nil {
t.Fatalf("failed to store a route: %v", err)
}
walBytes := walFileSize(t, db)
if walBytes == 0 {
t.Fatal("WAL file empty after the route write, want frames in it")
}
_, lines = openLogged(t, cfg)
line = findLine(t, lines, startupCheckpointMessage)
for _, key := range []string{"busy", "wal_frames", "checkpointed_frames"} {
if line[key] != 0.0 {
t.Errorf("start-up checkpoint logged %s = %v, want 0", key, line[key])
}
}
if line["wal_bytes"] != float64(walBytes) {
t.Errorf("start-up checkpoint logged wal_bytes = %v, want the %d "+
"bytes the first database left", line["wal_bytes"], walBytes)
}
}
// TestStartupCheckpointWarnsWhenIncomplete keeps a read open on a database
// while a route is written after the read began, then opens the same file
// again. The second open's start-up checkpoint cannot copy the frames written
// after the read began, and gives up once the busy timeout has passed. It
// checks that the checkpoint is logged as a warning with busy 1 and fewer
// frames copied than the WAL holds.
func TestStartupCheckpointWarnsWhenIncomplete(t *testing.T) {
t.Parallel()
cfg := &config.Config{StateDir: t.TempDir()}
db, _ := openLogged(t, cfg)
ctx := context.Background()
reader, err := db.db.Conn(ctx)
if err != nil {
t.Fatalf("failed to open the reading connection: %v", err)
}
defer func() { _ = reader.Close() }()
// A plain BEGIN starts a transaction that only reads, unlike the BEGIN
// IMMEDIATE of _txlock. It keeps the snapshot its first read takes until
// it ends.
_, err = reader.ExecContext(ctx, "BEGIN")
if err != nil {
t.Fatalf("failed to begin the read transaction: %v", err)
}
defer func() { _, _ = reader.ExecContext(ctx, "ROLLBACK") }()
var routes int
err = reader.QueryRowContext(ctx,
"SELECT count(*) FROM live_routes_v4").Scan(&routes)
if err != nil {
t.Fatalf("failed to read the live routes: %v", err)
}
err = db.UpsertLiveRouteBatch([]*LiveRoute{
mkV4Route(t, "192.0.2.0/24", lockTestASN, time.Now().UTC()),
})
if err != nil {
t.Fatalf("failed to store a route after the read: %v", err)
}
_, lines := openLogged(t, cfg)
line := findLine(t, lines, startupCheckpointIncompleteMessage)
if line[slog.LevelKey] != slog.LevelWarn.String() {
t.Errorf("start-up checkpoint logged at level %v, want %s",
line[slog.LevelKey], slog.LevelWarn)
}
if line["busy"] != 1.0 {
t.Errorf("start-up checkpoint logged busy = %v, want 1", line["busy"])
}
walFrames, _ := line["wal_frames"].(float64)
checkpointedFrames, _ := line["checkpointed_frames"].(float64)
if checkpointedFrames >= walFrames {
t.Errorf("start-up checkpoint logged checkpointed_frames = %v and "+
"wal_frames = %v with a read open, want fewer copied than the WAL "+
"holds", line["checkpointed_frames"], line["wal_frames"])
}
}
// openLogged opens the database of cfg with a logger that writes JSON, and
// returns it with the lines it logged while it opened, decoded.
func openLogged(t *testing.T, cfg *config.Config) (*Database, []map[string]any) {
t.Helper()
var logged bytes.Buffer
db, err := New(cfg,
&logger.Logger{Logger: slog.New(slog.NewJSONHandler(&logged, nil))})
if err != nil {
t.Fatalf("failed to open the database: %v", err)
}
t.Cleanup(func() { _ = db.Close() })
var lines []map[string]any
decoder := json.NewDecoder(&logged)
for decoder.More() {
var line map[string]any
err := decoder.Decode(&line)
if err != nil {
t.Fatalf("failed to decode the log: %v", err)
}
lines = append(lines, line)
}
return db, lines
}
// findLine returns the first of lines logged with message.
func findLine(
t *testing.T,
lines []map[string]any,
message string,
) map[string]any {
t.Helper()
for _, line := range lines {
if line[slog.MessageKey] == message {
return line
}
}
t.Fatalf("no line %q in the log", message)
return nil
}
// TestGetRandomPrefixesByLengthListsPrefixOnce stores one prefix of each // TestGetRandomPrefixesByLengthListsPrefixOnce stores one prefix of each
// family learned from two peers in the same second and checks that the // family learned from two peers in the same second and checks that the
// prefix-length sample lists each of them once. // prefix-length sample lists each of them once.
@@ -1527,6 +1700,85 @@ func TestGetASDetailsListsEachPrefixOnce(t *testing.T) {
} }
} }
// TestGetASDetailsOfASNotInDatabase asks for ASes that the database has no
// record of, some with stored routes. Each gets the prefixes of its routes and
// no seen times. One that the AS info lookup knows gets its handle and
// description from it. An AS with no record, no lookup entry and no route is
// ErrNoRoute. Before the fix every AS with no record was ErrNoRoute.
func TestGetASDetailsOfASNotInDatabase(t *testing.T) {
t.Parallel()
db, err := New(&config.Config{StateDir: t.TempDir()}, logger.New())
if err != nil {
t.Fatalf("failed to create database: %v", err)
}
defer func() { _ = db.Close() }()
ts := time.Date(2026, 1, 2, 3, 4, 5, 0, time.UTC)
tests := []struct {
asn int
listed bool // in the AS info lookup
prefixes []string // of the routes stored for the AS
}{
{15169, true, []string{"192.0.2.0/24"}},
{13335, true, nil},
{4200000000, false, []string{"198.51.100.0/24"}},
}
var routes []*LiveRoute
for _, tt := range tests {
if _, ok := asinfo.Get(tt.asn); ok != tt.listed {
t.Fatalf("AS%d in the AS info data: %t, want %t", tt.asn, ok, tt.listed)
}
for _, prefix := range tt.prefixes {
routes = append(routes, mkV4Route(t, prefix, tt.asn, ts))
}
}
err = db.UpsertLiveRouteBatch(routes)
if err != nil {
t.Fatalf("UpsertLiveRouteBatch: %v", err)
}
for _, tt := range tests {
got, prefixes, err := db.GetASDetails(tt.asn)
if err != nil {
t.Errorf("GetASDetails(%d): %v", tt.asn, err)
continue
}
want := ASN{ASN: tt.asn}
if info, ok := asinfo.Get(tt.asn); ok {
want.Handle, want.Description = info.Handle, info.Description
}
var gotPrefixes []string
for _, route := range prefixes {
gotPrefixes = append(gotPrefixes, route.Prefix)
}
if *got != want || !slices.Equal(gotPrefixes, tt.prefixes) {
t.Errorf("GetASDetails(%d) = %+v with prefixes %v, want %+v with %v",
tt.asn, *got, gotPrefixes, want, tt.prefixes)
}
}
const absentASN = 4200000001 // no record, no lookup entry and no route
if _, ok := asinfo.Get(absentASN); ok {
t.Fatalf("AS%d is in the AS info data", absentASN)
}
_, _, err = db.GetASDetails(absentASN)
if !errors.Is(err, ErrNoRoute) {
t.Errorf("GetASDetails(%d): error %v, want %v", absentASN, err, ErrNoRoute)
}
}
// TestASPrefixQueriesUseCoveringIndexes checks the plan of each query that // TestASPrefixQueriesUseCoveringIndexes checks the plan of each query that
// lists the prefixes an AS originates: a search of a covering index and no // lists the prefixes an AS originates: a search of a covering index and no
// temporary B-tree, so the list reads one index entry per prefix, not one // temporary B-tree, so the list reads one index entry per prefix, not one
+4 -3
View File
@@ -28,9 +28,10 @@ type ASN struct {
RIRRegDate *time.Time `json:"rir_registration_date,omitempty"` RIRRegDate *time.Time `json:"rir_registration_date,omitempty"`
RIRLastMod *time.Time `json:"rir_last_modified,omitempty"` RIRLastMod *time.Time `json:"rir_last_modified,omitempty"`
WHOISRaw string `json:"whois_raw,omitempty"` WHOISRaw string `json:"whois_raw,omitempty"`
// Timestamps // Timestamps. They are zero, and left out of JSON, for an AS that has no
FirstSeen time.Time `json:"first_seen"` // record in the asns table.
LastSeen time.Time `json:"last_seen"` FirstSeen time.Time `json:"first_seen,omitzero"`
LastSeen time.Time `json:"last_seen,omitzero"`
WHOISUpdatedAt *time.Time `json:"whois_updated_at,omitempty"` WHOISUpdatedAt *time.Time `json:"whois_updated_at,omitempty"`
} }
+6 -2
View File
@@ -39,12 +39,16 @@ type ASPath struct {
Set []int Set []int
} }
// NoOrigin is the origin AS of a route whose origin is not known. AS 0 is
// reserved, so no AS has this number.
const NoOrigin = 0
// Origin returns the AS that originated the route: the last AS of the // Origin returns the AS that originated the route: the last AS of the
// sequence, or 0 (none) when the path ends in a set of several members or is // sequence, or NoOrigin when the path ends in a set of several members or is
// empty. // empty.
func (p *ASPath) Origin() int { func (p *ASPath) Origin() int {
if len(p.Set) > 0 || len(p.Sequence) == 0 { if len(p.Set) > 0 || len(p.Sequence) == 0 {
return 0 return NoOrigin
} }
return p.Sequence[len(p.Sequence)-1] return p.Sequence[len(p.Sequence)-1]
+13 -6
View File
@@ -21,10 +21,16 @@ const (
minInterval = 1 * time.Second minInterval = 1 * time.Second
// maxInterval is the maximum interval after failures (backoff cap). // maxInterval is the maximum interval after failures (backoff cap).
maxInterval = 5 * time.Minute maxInterval = time.Minute
// backoffMultiplier is how much to multiply interval on failure. // A failure multiplies the interval by backoffNumerator divided by
backoffMultiplier = 2 // backoffDenominator, 1.5, up to maxInterval.
backoffNumerator = 3
backoffDenominator = 2
// A success divides the interval by successDivisor, down to
// minInterval.
successDivisor = 2
// whoisStaleThreshold is how old WHOIS data can be before refresh. // whoisStaleThreshold is how old WHOIS data can be before refresh.
whoisStaleThreshold = 30 * 24 * time.Hour // 30 days whoisStaleThreshold = 30 * 24 * time.Hour // 30 days
@@ -160,7 +166,7 @@ func (f *ASNFetcher) recordSuccess() {
f.consecutiveFails = 0 f.consecutiveFails = 0
// Decrease interval by half, but not below minimum // Decrease interval by half, but not below minimum
newInterval := max(f.currentInterval/backoffMultiplier, minInterval) newInterval := max(f.currentInterval/successDivisor, minInterval)
if newInterval != f.currentInterval { if newInterval != f.currentInterval {
f.logger.Debug("Decreased fetch interval", f.logger.Debug("Decreased fetch interval",
@@ -184,8 +190,9 @@ func (f *ASNFetcher) recordFailure() {
f.intervalMu.Lock() f.intervalMu.Lock()
f.consecutiveFails++ f.consecutiveFails++
// Exponential backoff: multiply by 2, capped at max // Exponential backoff: multiply by 1.5, capped at max
newInterval := min(f.currentInterval*backoffMultiplier, maxInterval) newInterval := min(
f.currentInterval*backoffNumerator/backoffDenominator, maxInterval)
if newInterval != f.currentInterval { if newInterval != f.currentInterval {
f.logger.Debug("Increased fetch interval due to failure", f.logger.Debug("Increased fetch interval due to failure",
@@ -120,17 +120,58 @@ func TestFetchAndUpdateKeepsTheRowWhenTheLookupFails(t *testing.T) {
"was stored", stats.NeverFetched) "was stored", stats.NeverFetched)
} }
want := baseInterval * backoffNumerator / backoffDenominator
got := f.GetStats() got := f.GetStats()
if got.CurrentInterval != baseInterval*backoffMultiplier || if got.CurrentInterval != want || got.ConsecutiveFails != 1 {
got.ConsecutiveFails != 1 {
t.Errorf("interval %v after %d failures, want %v after 1", t.Errorf("interval %v after %d failures, want %v after 1",
got.CurrentInterval, got.ConsecutiveFails, got.CurrentInterval, got.ConsecutiveFails, want)
baseInterval*backoffMultiplier)
} }
}) })
} }
} }
// TestRecordStepsTheInterval has the fetcher fail from its base interval of
// 15 seconds, then succeed. Each failure makes the interval 1.5 times as
// long, up to one minute, and each success halves it, down to one second.
func TestRecordStepsTheInterval(t *testing.T) {
t.Parallel()
f := NewASNFetcher(newMockStore(), logger.New().Logger)
for i, want := range []time.Duration{
22500 * time.Millisecond,
33750 * time.Millisecond,
50625 * time.Millisecond,
time.Minute,
time.Minute,
} {
f.recordFailure()
if got := f.getInterval(); got != want {
t.Errorf("interval after %d failures = %v, want %v",
i+1, got, want)
}
}
for i, want := range []time.Duration{
30 * time.Second,
15 * time.Second,
7500 * time.Millisecond,
3750 * time.Millisecond,
1875 * time.Millisecond,
time.Second,
time.Second,
} {
f.recordSuccess()
if got := f.getInterval(); got != want {
t.Errorf("interval after %d successes = %v, want %v",
i+1, got, want)
}
}
}
// TestFetchAndUpdateMarksAnASWithoutReferral looks up a private-use AS, which // TestFetchAndUpdateMarksAnASWithoutReferral looks up a private-use AS, which
// IANA refers to no RIR. The AS must not be looked up again until it is // IANA refers to no RIR. The AS must not be looked up again until it is
// stale, and the lookup counts as neither a success nor a failure. // stale, and the lookup counts as neither a success nor a failure.
+66 -6
View File
@@ -16,6 +16,7 @@ import (
"git.eeqj.de/sneak/routewatch/internal/database" "git.eeqj.de/sneak/routewatch/internal/database"
"git.eeqj.de/sneak/routewatch/internal/metrics" "git.eeqj.de/sneak/routewatch/internal/metrics"
"git.eeqj.de/sneak/routewatch/internal/ristypes"
"git.eeqj.de/sneak/routewatch/internal/streamer" "git.eeqj.de/sneak/routewatch/internal/streamer"
"git.eeqj.de/sneak/routewatch/internal/templates" "git.eeqj.de/sneak/routewatch/internal/templates"
asinfo "git.eeqj.de/sneak/routewatch/pkg/asinfo" asinfo "git.eeqj.de/sneak/routewatch/pkg/asinfo"
@@ -56,10 +57,18 @@ const (
// Error messages an answer sends, in place of the error behind it where there // Error messages an answer sends, in place of the error behind it where there
// is one. // is one.
const ( const (
invalidPrefixMessage = "Invalid prefix" invalidPrefixMessage = "Invalid prefix"
hostNotResolvedMessage = "could not resolve host name" hostNotResolvedMessage = "could not resolve host name"
hostNotPublicMessage = "host name has no public address" hostNotPublicMessage = "host name has no public address"
tooManyTargetsMessage = "Too many addresses or host names" tooManyTargetsMessage = "Too many addresses or host names"
searchQueryMissingMessage = "Missing search query"
)
// The AS name search reads the text it looks for from the URL query parameter
// asSearchQueryParam, and answers with at most maxASSearchResults matches.
const (
asSearchQueryParam = "q"
maxASSearchResults = 50
) )
// IP versions as database.LiveRoute.IPVersion records them. // IP versions as database.LiveRoute.IPVersion records them.
@@ -1001,11 +1010,12 @@ func (s *Server) handleASDetailJSON() http.HandlerFunc {
} }
} }
// splitRoutesByIPVersion groups routes into the IPv4 and the IPv6 ones. // splitRoutesByIPVersion groups routes into the IPv4 and the IPv6 ones. Neither
// group is nil, so that a JSON answer holds [] for an empty one and not null.
func splitRoutesByIPVersion( func splitRoutesByIPVersion(
routes []database.LiveRoute, routes []database.LiveRoute,
) ([]database.LiveRoute, []database.LiveRoute) { ) ([]database.LiveRoute, []database.LiveRoute) {
var ipv4Routes, ipv6Routes []database.LiveRoute ipv4Routes, ipv6Routes := []database.LiveRoute{}, []database.LiveRoute{}
for _, route := range routes { for _, route := range routes {
if route.IPVersion == ipVersionV4 { if route.IPVersion == ipVersionV4 {
@@ -1105,6 +1115,56 @@ func (s *Server) handlePrefixDetailJSON() http.HandlerFunc {
} }
} }
// ASSearchResponse is the answer to an AS name search: the text searched for,
// the number of ASes that match it, and the first maxASSearchResults of them in
// AS number order.
type ASSearchResponse struct {
Query string `json:"query"`
Total int `json:"total"`
Results []asinfo.Info `json:"results"`
}
// handleASSearch returns a handler that searches the AS list built into
// asinfo for the text in the q parameter, without its leading and trailing
// spaces. It answers 400 when no text is left.
func (s *Server) handleASSearch() http.HandlerFunc {
return func(w http.ResponseWriter, r *http.Request) {
query := strings.TrimSpace(r.URL.Query().Get(asSearchQueryParam))
if query == "" {
s.writeJSONError(w, http.StatusBadRequest, searchQueryMissingMessage)
return
}
// AS 0 means no known origin and has no AS page, so the answer leaves
// it out. The matches come in AS number order, so AS 0 can only be the
// first. The search asks for one match more than the answer holds, to
// take its place.
results, total := asinfo.Search(query, maxASSearchResults+1)
if len(results) > 0 && results[0].ASN == ristypes.NoOrigin {
results = results[1:]
total--
}
results = results[:min(len(results), maxASSearchResults)]
// results is an empty list rather than nil when nothing matches, so
// that the answer holds [] and not null.
if results == nil {
results = []asinfo.Info{}
}
err := writeJSONSuccess(w, ASSearchResponse{
Query: query,
Total: total,
Results: results,
})
if err != nil {
s.logger.Error("Failed to encode AS search results", "error", err)
}
}
}
// handleASDetail returns a handler that serves the AS detail HTML page // handleASDetail returns a handler that serves the AS detail HTML page
func (s *Server) handleASDetail() http.HandlerFunc { func (s *Server) handleASDetail() http.HandlerFunc {
return func(w http.ResponseWriter, r *http.Request) { return func(w http.ResponseWriter, r *http.Request) {
+387 -2
View File
@@ -19,6 +19,7 @@ import (
"git.eeqj.de/sneak/routewatch/internal/database" "git.eeqj.de/sneak/routewatch/internal/database"
"git.eeqj.de/sneak/routewatch/internal/logger" "git.eeqj.de/sneak/routewatch/internal/logger"
"git.eeqj.de/sneak/routewatch/internal/metrics" "git.eeqj.de/sneak/routewatch/internal/metrics"
"git.eeqj.de/sneak/routewatch/internal/ristypes"
"git.eeqj.de/sneak/routewatch/internal/streamer" "git.eeqj.de/sneak/routewatch/internal/streamer"
"git.eeqj.de/sneak/routewatch/pkg/asinfo" "git.eeqj.de/sneak/routewatch/pkg/asinfo"
) )
@@ -26,6 +27,7 @@ import (
// The prefix and peer addresses the prefix page tests store routes for. // The prefix and peer addresses the prefix page tests store routes for.
const ( const (
testPrefixV4 = "192.0.2.0/24" testPrefixV4 = "192.0.2.0/24"
testPrefixV6 = "2001:db8::/32"
testPeerV4 = "198.51.100.1" testPeerV4 = "198.51.100.1"
testPeerV6 = "2001:db8::1" testPeerV6 = "2001:db8::1"
) )
@@ -194,7 +196,7 @@ func TestStatsHandlersAnswerFromMemory(t *testing.T) {
} }
// newServerWithTestRoutes returns a server on a new database that holds a // newServerWithTestRoutes returns a server on a new database that holds a
// route for testPrefixV4 and one for 2001:db8::/32. // route for testPrefixV4 and one for testPrefixV6.
func newServerWithTestRoutes(t *testing.T) *Server { func newServerWithTestRoutes(t *testing.T) *Server {
t.Helper() t.Helper()
@@ -214,7 +216,7 @@ func newServerWithTestRoutes(t *testing.T) *Server {
NextHop: testPeerV4, LastUpdated: ts, NextHop: testPeerV4, LastUpdated: ts,
}, },
{ {
Prefix: "2001:db8::/32", MaskLength: 32, IPVersion: 6, Prefix: testPrefixV6, MaskLength: 32, IPVersion: 6,
OriginASN: 64501, PeerIP: testPeerV6, ASPath: []int{64501}, OriginASN: 64501, PeerIP: testPeerV6, ASPath: []int{64501},
NextHop: testPeerV6, LastUpdated: ts, NextHop: testPeerV6, LastUpdated: ts,
}, },
@@ -1166,6 +1168,389 @@ func TestIPLookupHidesPrivateAddresses(t *testing.T) {
} }
} }
// asSearchPath is the path of the AS name search.
const asSearchPath = "/api/v1/search/as"
// searchAS sends an AS name search with the URL query rawQuery through s's
// router, expects a 200 answer, and returns its data.
func searchAS(t *testing.T, s *Server, rawQuery string) ASSearchResponse {
t.Helper()
rec := httptest.NewRecorder()
s.router.ServeHTTP(rec, httptest.NewRequestWithContext(
t.Context(), http.MethodGet, asSearchPath+"?"+rawQuery, nil))
if rec.Code != http.StatusOK {
t.Fatalf("%s: status %d, want %d; body %s",
rawQuery, rec.Code, http.StatusOK, rec.Body)
}
var body struct {
Data ASSearchResponse `json:"data"`
}
err := json.Unmarshal(rec.Body.Bytes(), &body)
if err != nil {
t.Fatalf("%s: decoding the answer %q: %v", rawQuery, rec.Body, err)
}
return body.Data
}
// TestASSearchAnswersMatches searches for Cloudflare's handle in lower case,
// with spaces around it, and expects AS13335 among the matches. Before the fix
// there was no AS name search, and the home page's search card said it was
// coming soon (https://git.eeqj.de/sneak/routewatch/issues/151).
func TestASSearchAnswersMatches(t *testing.T) {
t.Parallel()
const query = "cloudflarenet"
data := searchAS(t, newServerWithTestRoutes(t), "q=%20"+query+"%20")
if data.Query != query {
t.Errorf("query %q, want %q", data.Query, query)
}
if data.Total != len(data.Results) {
t.Errorf("total %d, but %d results", data.Total, len(data.Results))
}
cloudflare := asinfo.Info{
ASN: 13335, Handle: "CLOUDFLARENET", Description: "Cloudflare, Inc.",
}
if !slices.Contains(data.Results, cloudflare) {
t.Errorf("results %v do not hold %v", data.Results, cloudflare)
}
}
// TestASSearchAnswersEmptyListWhenNothingMatches expects a search that matches
// nothing to answer 200 with a total of 0 and an empty list, not null.
func TestASSearchAnswersEmptyListWhenNothingMatches(t *testing.T) {
t.Parallel()
data := searchAS(t, newServerWithTestRoutes(t), "q=XYZABC123456")
if data.Total != 0 || data.Results == nil || len(data.Results) != 0 {
t.Errorf("total %d and results %#v, want 0 and an empty list",
data.Total, data.Results)
}
}
// TestASSearchAnswers400ForEmptyQuery expects a search with no q, an empty q
// or a q of only spaces to answer 400 with the JSON error body.
func TestASSearchAnswers400ForEmptyQuery(t *testing.T) {
t.Parallel()
s := newServerWithTestRoutes(t)
for _, path := range []string{
asSearchPath, asSearchPath + "?q=", asSearchPath + "?q=%20%20",
} {
rec := httptest.NewRecorder()
s.router.ServeHTTP(rec, httptest.NewRequestWithContext(
t.Context(), http.MethodGet, path, nil))
if rec.Code != http.StatusBadRequest {
t.Errorf("%s: status %d, want %d; body %s",
path, rec.Code, http.StatusBadRequest, rec.Body)
}
checkJSONErrorCode(t, path, rec, http.StatusBadRequest)
checkJSONErrorMessage(t, path, rec, searchQueryMissingMessage)
}
}
// TestASSearchShowsFirstMatchesAndTotal searches for a word with more matches
// than an answer holds, and expects the first maxASSearchResults of them, in
// AS number order, and the number of all of them.
func TestASSearchShowsFirstMatchesAndTotal(t *testing.T) {
t.Parallel()
const query = "university"
all, _ := asinfo.Search(query, asinfo.Total())
if len(all) <= maxASSearchResults {
t.Fatalf("Search(%q) has %d matches; the test needs more than %d",
query, len(all), maxASSearchResults)
}
data := searchAS(t, newServerWithTestRoutes(t), "q="+query)
if data.Total != len(all) {
t.Errorf("total %d, want %d", data.Total, len(all))
}
if !slices.Equal(data.Results, all[:maxASSearchResults]) {
t.Errorf("results %v, want the first %d matches %v",
data.Results, maxASSearchResults, all[:maxASSearchResults])
}
}
// The AS numbers the AS detail tests ask for. storedASN originates the route
// for testPrefixV4 and is in the database. routeOnlyASN originates the route
// for testPrefixV6, but the database has no record of the AS itself, as when
// a route is stored before its AS. builtInOnlyASN is in the AS list built into
// asinfo but not in the database. unknownASN is in neither.
const (
storedASN = 64500
routeOnlyASN = 64501
builtInOnlyASN = 15169
unknownASN = 4200000000
)
// zeroDate is how a page writes a time that was never set.
const zeroDate = "0001-01-01"
// firstSeenCard is the label of the AS page's first-seen date.
const firstSeenCard = `<div class="info-label">First Seen</div>`
// newServerWithStoredAS returns newServerWithTestRoutes's server with storedASN
// added to the database. It fails the test when the AS list built into asinfo
// lacks routeOnlyASN or builtInOnlyASN, or holds unknownASN.
func newServerWithStoredAS(t *testing.T) *Server {
t.Helper()
for _, asn := range []int{routeOnlyASN, builtInOnlyASN} {
if _, ok := asinfo.Get(asn); !ok {
t.Fatalf("AS%d is not in the AS info data", asn)
}
}
if _, ok := asinfo.Get(unknownASN); ok {
t.Fatalf("AS%d is in the AS info data", unknownASN)
}
s := newServerWithTestRoutes(t)
_, err := s.db.GetOrCreateASN(storedASN, time.Date(2026, 1, 2, 3, 4, 5, 0, time.UTC))
if err != nil {
t.Fatalf("GetOrCreateASN: %v", err)
}
return s
}
// TestASDetailJSONAnswersBuiltInAS asks /api/v1/as/ for each of the ASes above.
// All but unknownASN answer 200 with the handle and description from the
// built-in AS list and their prefixes, as lists even when empty. Only storedASN
// has a first_seen time. unknownASN answers 404. Before the fix builtInOnlyASN
// and routeOnlyASN answered 404 too
// (https://git.eeqj.de/sneak/routewatch/issues/151).
func TestASDetailJSONAnswersBuiltInAS(t *testing.T) {
t.Parallel()
s := newServerWithStoredAS(t)
tests := []struct {
asn int
wantCode int
wantPrefixes []string
wantFirstSeen bool
}{
{storedASN, http.StatusOK, []string{testPrefixV4}, true},
{routeOnlyASN, http.StatusOK, []string{testPrefixV6}, false},
{builtInOnlyASN, http.StatusOK, nil, false},
{unknownASN, http.StatusNotFound, nil, false},
}
for _, tt := range tests {
path := "/api/v1/as/" + strconv.Itoa(tt.asn)
rec := httptest.NewRecorder()
s.router.ServeHTTP(rec, httptest.NewRequestWithContext(
t.Context(), http.MethodGet, path, nil))
if rec.Code != tt.wantCode {
t.Errorf("%s: status %d, want %d; body %s",
path, rec.Code, tt.wantCode, rec.Body)
continue
}
if tt.wantCode != http.StatusOK {
checkJSONErrorCode(t, path, rec, tt.wantCode)
continue
}
//nolint:tagliatelle // field names are the existing JSON API
var body struct {
Data struct {
ASN database.ASN `json:"asn"`
IPv4Prefixes []database.LiveRoute `json:"ipv4_prefixes"`
IPv6Prefixes []database.LiveRoute `json:"ipv6_prefixes"`
} `json:"data"`
}
err := json.Unmarshal(rec.Body.Bytes(), &body)
if err != nil {
t.Errorf("%s: decoding the answer %q: %v", path, rec.Body, err)
continue
}
info, _ := asinfo.Get(tt.asn)
got := body.Data.ASN
if got.ASN != tt.asn || got.Handle != info.Handle ||
got.Description != info.Description {
t.Errorf("%s: AS %d %q %q, want %d %q %q", path, got.ASN,
got.Handle, got.Description, tt.asn, info.Handle, info.Description)
}
if body.Data.IPv4Prefixes == nil || body.Data.IPv6Prefixes == nil {
t.Errorf("%s: a prefix list is null; body %s", path, rec.Body)
}
routes := slices.Concat(body.Data.IPv4Prefixes, body.Data.IPv6Prefixes)
gotPrefixes := make([]string, 0, len(routes))
for _, route := range routes {
gotPrefixes = append(gotPrefixes, route.Prefix)
}
if !slices.Equal(gotPrefixes, tt.wantPrefixes) {
t.Errorf("%s: prefixes %v, want %v", path, gotPrefixes, tt.wantPrefixes)
}
gotFirstSeen := strings.Contains(rec.Body.String(), `"first_seen"`)
if gotFirstSeen != tt.wantFirstSeen {
t.Errorf("%s: has first_seen %t, want %t; body %s",
path, gotFirstSeen, tt.wantFirstSeen, rec.Body)
}
}
}
// TestASDetailPageShowsBuiltInAS asks /as/ for each of the ASes above. All but
// unknownASN answer 200 with the handle and description from the built-in AS
// list. storedASN and routeOnlyASN list their prefixes, and builtInOnlyASN says
// the AS announces none. Only storedASN shows a first-seen date, and no page
// shows a date that was never set. unknownASN answers 404. Before the fix
// builtInOnlyASN and routeOnlyASN answered 404 too
// (https://git.eeqj.de/sneak/routewatch/issues/151).
func TestASDetailPageShowsBuiltInAS(t *testing.T) {
t.Parallel()
s := newServerWithStoredAS(t)
tests := []struct {
asn int
wantCode int
wantText string
wantFirstSeen bool
}{
{storedASN, http.StatusOK, testPrefixV4, true},
{routeOnlyASN, http.StatusOK, testPrefixV6, false},
{builtInOnlyASN, http.StatusOK, "No prefixes announced by this AS", false},
{unknownASN, http.StatusNotFound, "", false},
}
for _, tt := range tests {
path := "/as/" + strconv.Itoa(tt.asn)
rec := httptest.NewRecorder()
s.router.ServeHTTP(rec, httptest.NewRequestWithContext(
t.Context(), http.MethodGet, path, nil))
if rec.Code != tt.wantCode {
t.Errorf("%s: status %d, want %d; body %s",
path, rec.Code, tt.wantCode, rec.Body)
continue
}
if tt.wantCode != http.StatusOK {
continue
}
info, _ := asinfo.Get(tt.asn)
page := rec.Body.String()
for _, want := range []string{
html.EscapeString(info.Handle),
html.EscapeString(info.Description),
tt.wantText,
} {
if !strings.Contains(page, want) {
t.Errorf("%s: page does not show %q", path, want)
}
}
if strings.Contains(page, zeroDate) {
t.Errorf("%s: page shows the date %s", path, zeroDate)
}
gotFirstSeen := strings.Contains(page, firstSeenCard)
if gotFirstSeen != tt.wantFirstSeen {
t.Errorf("%s: shows a first-seen date %t, want %t",
path, gotFirstSeen, tt.wantFirstSeen)
}
}
}
// TestNoOriginHasNoASAnswers stores a route with no known origin, which has
// origin AS 0 (https://git.eeqj.de/sneak/routewatch/issues/75). It expects
// /as/0 and /api/v1/as/0 to answer 404. It expects a search for "iana" to leave
// AS 0 out of its results and its total, although the AS list built into
// asinfo holds AS 0. That search has more matches than an answer holds, so it
// still lists maxASSearchResults of them.
func TestNoOriginHasNoASAnswers(t *testing.T) {
t.Parallel()
const (
noOriginPrefix = "203.0.113.0/24"
query = "iana"
)
s := newServerWithTestRoutes(t)
err := s.db.UpsertLiveRouteBatch([]*database.LiveRoute{{
Prefix: noOriginPrefix, MaskLength: 24, IPVersion: 4,
OriginASN: ristypes.NoOrigin, PeerIP: testPeerV4,
ASPath: []int{64500, 64501, 64502}, NextHop: testPeerV4,
LastUpdated: time.Date(2026, 1, 2, 3, 4, 5, 0, time.UTC),
}})
if err != nil {
t.Fatalf("UpsertLiveRouteBatch: %v", err)
}
asn := strconv.Itoa(ristypes.NoOrigin)
for _, path := range []string{"/as/" + asn, "/api/v1/as/" + asn} {
rec := httptest.NewRecorder()
s.router.ServeHTTP(rec, httptest.NewRequestWithContext(
t.Context(), http.MethodGet, path, nil))
if rec.Code != http.StatusNotFound {
t.Errorf("%s: status %d, want %d; body %s",
path, rec.Code, http.StatusNotFound, rec.Body)
}
}
first, total := asinfo.Search(query, 1)
if len(first) != 1 || first[0].ASN != ristypes.NoOrigin {
t.Fatalf("AS%d is not the first match for %q in the AS info data",
ristypes.NoOrigin, query)
}
data := searchAS(t, s, "q="+query)
for _, info := range data.Results {
if info.ASN == ristypes.NoOrigin {
t.Errorf("the search for %q lists AS%d", query, info.ASN)
}
}
if data.Total != total-1 {
t.Errorf("the search for %q has a total of %d, want %d",
query, data.Total, total-1)
}
if len(data.Results) != maxASSearchResults {
t.Errorf("the search for %q lists %d matches, want %d",
query, len(data.Results), maxASSearchResults)
}
}
// settledGoroutineCount lets transient goroutines finish, then reports the // settledGoroutineCount lets transient goroutines finish, then reports the
// current count. // current count.
func settledGoroutineCount() int { func settledGoroutineCount() int {
+1
View File
@@ -87,6 +87,7 @@ func (s *Server) setupRoutes() {
r.Get("/ip/{ip}", s.handleIPLookup()) r.Get("/ip/{ip}", s.handleIPLookup())
r.Get("/as/{asn}", s.handleASDetailJSON()) r.Get("/as/{asn}", s.handleASDetailJSON())
r.Get("/prefix/{prefix}/{len}", s.handlePrefixDetailJSON()) r.Get("/prefix/{prefix}/{len}", s.handlePrefixDetailJSON())
r.Get("/search/as", s.handleASSearch())
}) })
s.router = r s.router = r
+5 -1
View File
@@ -1,4 +1,8 @@
// Package server provides HTTP endpoints for status monitoring and statistics // Package server serves RouteWatch's web pages and JSON API:
// - the home page, the status figures and the health check
// - the AS, prefix and prefix length pages, and the JSON answers behind them
// - the IP lookups
// - the AS name search
package server package server
import ( import (
+2
View File
@@ -233,10 +233,12 @@
<div class="info-label">Peer ASNs</div> <div class="info-label">Peer ASNs</div>
<div class="info-value">{{.PeerCount}}</div> <div class="info-value">{{.PeerCount}}</div>
</div> </div>
{{if not .ASN.FirstSeen.IsZero}}
<div class="info-card"> <div class="info-card">
<div class="info-label">First Seen</div> <div class="info-label">First Seen</div>
<div class="info-value">{{.ASN.FirstSeen.Format "2006-01-02"}}</div> <div class="info-value">{{.ASN.FirstSeen.Format "2006-01-02"}}</div>
</div> </div>
{{end}}
</div> </div>
{{if .IPv4Prefixes}} {{if .IPv4Prefixes}}
+101 -11
View File
@@ -174,6 +174,38 @@
#asname-results .search-message { #asname-results .search-message {
font-size: 13px; font-size: 13px;
} }
#asname-results .error {
color: #c00;
}
.asname-list {
list-style: none;
margin: 12px 0 0 0;
padding: 0;
max-height: 400px;
overflow-y: auto;
}
.asname-list li {
border-bottom: 1px solid #eee;
}
.asname-list a {
display: block;
padding: 8px 4px;
color: #2c3e50;
text-decoration: none;
font-size: 14px;
}
.asname-list a:hover {
background: #f5f5f5;
}
.asname-list .asn {
font-weight: 600;
color: #3498db;
}
.asname-list .description {
display: block;
color: #7f8c8d;
font-size: 12px;
}
/* IP Lookup result */ /* IP Lookup result */
.ip-result { .ip-result {
@@ -306,7 +338,7 @@
<input type="text" id="asname-input" placeholder="e.g., Google, Cloudflare" autocomplete="off"> <input type="text" id="asname-input" placeholder="e.g., Google, Cloudflare" autocomplete="off">
<button type="submit">Search</button> <button type="submit">Search</button>
</form> </form>
<p class="search-hint">Search for autonomous systems by organization name</p> <p class="search-hint">Search for autonomous systems by organization name or handle</p>
<div id="asname-results"></div> <div id="asname-results"></div>
</div> </div>
@@ -387,24 +419,82 @@
} }
}); });
// AS name search // AS name search. The handles and descriptions come from outside data,
// so the results are built with textContent, never innerHTML.
const asNameResults = document.getElementById('asname-results');
// Replace the search results with one line of text.
function showASNameMessage(text, className) {
const line = document.createElement('p');
line.className = className;
line.textContent = text;
asNameResults.replaceChildren(line);
}
// List the matches of a search answer, each as a link to its AS page,
// and say how many more matched than the answer holds.
function showASNameMatches(data) {
if (data.total === 0) {
showASNameMessage('No AS matches "' + data.query + '".', 'search-message');
return;
}
const list = document.createElement('ul');
list.className = 'asname-list';
for (const match of data.results) {
const asn = document.createElement('span');
asn.className = 'asn';
asn.textContent = 'AS' + match.asn;
const description = document.createElement('span');
description.className = 'description';
description.textContent = match.description;
const link = document.createElement('a');
link.href = '/as/' + match.asn;
link.append(asn, ' ' + match.handle, description);
const item = document.createElement('li');
item.append(link);
list.append(item);
}
asNameResults.replaceChildren(list);
const more = data.total - data.results.length;
if (more > 0) {
const line = document.createElement('p');
line.className = 'search-message';
line.textContent = more.toLocaleString() + ' more matched. Narrow the search to see them.';
asNameResults.append(line);
}
}
document.getElementById('asname-form').addEventListener('submit', function(e) { document.getElementById('asname-form').addEventListener('submit', function(e) {
e.preventDefault(); e.preventDefault();
const query = document.getElementById('asname-input').value.trim(); const query = document.getElementById('asname-input').value.trim();
if (!query) return; if (!query) return;
const resultsDiv = document.getElementById('asname-results'); // A number, or AS and a number, opens that AS's page
resultsDiv.innerHTML = '<p class="loading">Searching...</p>'; const asn = query.replace(/^AS/i, '');
if (/^\d+$/.test(asn)) {
// Use a simple client-side search against the asinfo data window.location.href = '/as/' + asn;
// For now, redirect to AS page if it looks like an ASN
if (/^\d+$/.test(query)) {
window.location.href = '/as/' + query;
return; return;
} }
// Show a message that server-side search is coming showASNameMessage('Searching...', 'loading');
resultsDiv.innerHTML = '<p class="search-message">AS name search coming soon. For now, try an AS number.</p>';
fetch('/api/v1/search/as?q=' + encodeURIComponent(query))
.then(response => response.json())
.then(response => {
if (response.status === 'error') {
showASNameMessage('Error: ' + response.error.msg, 'error');
} else {
showASNameMatches(response.data);
}
})
.catch(error => {
showASNameMessage('Error: ' + error.message, 'error');
});
}); });
// IP lookup // IP lookup
+64 -5
View File
@@ -49,6 +49,31 @@ const (
RIRAFRNIC = "AFRINIC" RIRAFRNIC = "AFRINIC"
) )
// noPersonalFlag keeps person and role objects out of an answer. The
// organisation object stays in it. RIPE counts the person and role objects
// it returns against a daily limit per IP address. Role objects for abuse
// contacts do not count. RIPE refuses an address for good that passes the
// limit too often:
// https://docs.db.ripe.net/Types-of-Queries/Access-Control-for-Queries/
//
// RIPE documents the flag here:
// https://docs.db.ripe.net/Types-of-Queries/Referenced-Objects-in-Query-Response/
// APNIC and AFRINIC list it in their answers to the query "help". AFRINIC
// sends readers to that answer for its full list of flags:
// https://afrinic.net/whois-getting-started
const noPersonalFlag = "--no-personal"
// The error lines with which RIPE's database software refuses a query:
// access denied to the client's IP address, and that address's limit
// reached. APNIC and AFRINIC run the same software. The lines are listed
// under "Access Errors" on this page, whose URL is split over two lines:
// https://docs.db.ripe.net/Appendices/
// Appendix-C--RIPE-Database-Query-Server-Response-Codes-and-Messages/
const (
accessDeniedError = "%ERROR:201:"
accessLimitError = "%ERROR:202:"
)
// Errors LookupASN returns when a WHOIS answer has nothing to store. // Errors LookupASN returns when a WHOIS answer has nothing to store.
var ( var (
// ErrNoReferral is returned when IANA refers the AS to no RIR, as it // ErrNoReferral is returned when IANA refers the AS to no RIR, as it
@@ -58,6 +83,7 @@ var (
errNoASBlock = errors.New("IANA answered with neither a referral " + errNoASBlock = errors.New("IANA answered with neither a referral " +
"nor an AS block") "nor an AS block")
errNoFields = errors.New("RIR answered with no WHOIS fields") errNoFields = errors.New("RIR answered with no WHOIS fields")
errRefused = errors.New("RIR refused the query")
errUnknownServer = errors.New("IANA refers the AS to a server no RIR runs") errUnknownServer = errors.New("IANA refers the AS to a server no RIR runs")
errAnswerTooLarge = errors.New("WHOIS answer reached the size cap") errAnswerTooLarge = errors.New("WHOIS answer reached the size cap")
) )
@@ -97,10 +123,10 @@ func NewClient() *Client {
// LookupASN asks IANA which RIR holds the given ASN, asks that RIR, and // LookupASN asks IANA which RIR holds the given ASN, asks that RIR, and
// returns what the RIR's answer says. It returns ErrNoReferral when IANA // returns what the RIR's answer says. It returns ErrNoReferral when IANA
// refers the AS to no RIR, and another error when a query fails, when IANA's // refers the AS to no RIR, and another error when a query fails, when the
// answer has neither a referral nor the AS's block, when IANA refers the AS // RIR refuses it, when IANA's answer has neither a referral nor the AS's
// to a server none of the five RIRs run, or when the RIR's answer carries // block, when IANA refers the AS to a server none of the five RIRs run, or
// none of ASNInfo's fields, as a refusal of service does. // when the RIR's answer carries none of ASNInfo's fields.
func (c *Client) LookupASN(ctx context.Context, asn int) (*ASNInfo, error) { func (c *Client) LookupASN(ctx context.Context, asn int) (*ASNInfo, error) {
// Query IANA first to find the authoritative RIR // Query IANA first to find the authoritative RIR
query := fmt.Sprintf("AS%d", asn) query := fmt.Sprintf("AS%d", asn)
@@ -127,7 +153,7 @@ func (c *Client) LookupASN(ctx context.Context, asn int) (*ASNInfo, error) {
} }
// Query the authoritative RIR // Query the authoritative RIR
rirResp, err := c.query(ctx, whoisServer, query) rirResp, err := c.queryRIR(ctx, whoisServer, rirQuery(rir, query))
if err != nil { if err != nil {
return nil, fmt.Errorf("RIR query failed: %w", err) return nil, fmt.Errorf("RIR query failed: %w", err)
} }
@@ -154,6 +180,39 @@ func (info *ASNInfo) isEmpty() bool {
info.RegDate == nil && info.LastMod == nil info.RegDate == nil && info.LastMod == nil
} }
// rirQuery returns the query sent to rir's server for asQuery, an AS
// number such as "AS3333". RIPE, APNIC and AFRINIC are asked to keep person
// and role objects out of the answer.
func rirQuery(rir, asQuery string) string {
switch rir {
case RIRRIPE, RIRAPNIC, RIRAFRNIC:
return noPersonalFlag + " " + asQuery
default:
return asQuery
}
}
// queryRIR sends query to an RIR's server and returns the answer. An answer
// holding an error line with which the server refuses the query is an
// error, and the error holds that line.
func (c *Client) queryRIR(
ctx context.Context, server, query string,
) (string, error) {
answer, err := c.query(ctx, server, query)
if err != nil {
return "", err
}
for line := range strings.SplitSeq(answer, "\n") {
if strings.HasPrefix(line, accessDeniedError) ||
strings.HasPrefix(line, accessLimitError) {
return "", fmt.Errorf("%w: %s", errRefused, line)
}
}
return answer, nil
}
// query performs a raw WHOIS query to the specified server. An answer that // query performs a raw WHOIS query to the specified server. An answer that
// reaches maxAnswerSize is an error. // reaches maxAnswerSize is an error.
func (c *Client) query(ctx context.Context, server, query string) (string, error) { func (c *Client) query(ctx context.Context, server, query string) (string, error) {
+173 -27
View File
@@ -6,6 +6,7 @@ import (
"errors" "errors"
"io" "io"
"net" "net"
"slices"
"strings" "strings"
"sync" "sync"
"syscall" "syscall"
@@ -21,8 +22,20 @@ const (
ripeServer = "whois.ripe.net:43" ripeServer = "whois.ripe.net:43"
) )
// testASN is the AS the tests look up. // testASN is the AS the tests look up, and asQuery the query for it.
const testASN = 3333 const (
testASN = 3333
asQuery = "AS3333"
)
// The fake servers' answers are keyed by a server's address, a space and a
// query. These are the keys of the queries LookupASN sends for testASN:
// IANA's, then RIPE's, which asks RIPE to leave out person and role
// objects.
const (
ianaQuery = ianaServer + " " + asQuery
ripeQuery = ripeServer + " --no-personal " + asQuery
)
// ianaReferral is IANA's answer for an AS in a block it gave to RIPE. // ianaReferral is IANA's answer for an AS in a block it gave to RIPE.
const ianaReferral = `% IANA WHOIS server const ianaReferral = `% IANA WHOIS server
@@ -36,27 +49,75 @@ changed: 2002-09-01
source: IANA source: IANA
` `
// ripeRecord is RIPE's answer for the AS. // ripeRecord is RIPE's answer for the AS when asked to leave out person and
// role objects: the abuse address as a comment, the AS's object and the
// organisation object it refers to.
const ripeRecord = `% This is the RIPE Database query service. const ripeRecord = `% This is the RIPE Database query service.
% The objects are in RPSL format.
% Note: --no-personal means ALL personal data has been filtered from this output.
% Information related to 'AS3333'
% Abuse contact for 'AS3333' is 'abuse@ripe.net'
aut-num: AS3333 aut-num: AS3333
as-name: RIPE-NCC-AS as-name: RIPE-NCC-AS
org: ORG-RIEN1-RIPE org: ORG-RIEN1-RIPE
admin-c: BRD-RIPE
tech-c: OPS4-RIPE
abuse-c: OPS4-RIPE
mnt-by: RIPE-NCC-MNT
created: 2002-09-17T13:08:32Z created: 2002-09-17T13:08:32Z
last-modified: 2024-01-09T08:55:41Z last-modified: 2024-01-09T08:55:41Z
source: RIPE source: RIPE
organisation: ORG-RIEN1-RIPE
org-name: Reseaux IP Europeens Network Coordination Centre (RIPE NCC)
country: NL
org-type: RIR
address: P.O. Box 10096
address: 1001EB Amsterdam
abuse-c: OPS4-RIPE
created: 2012-03-09T13:08:29Z
last-modified: 2024-03-26T10:55:38Z
source: RIPE
` `
// The error lines with which RIPE refuses a query: access denied to the
// host, and the host's limit on returned objects reached.
const (
ripeAccessDenied = "%ERROR:201: access denied for 192.0.2.1"
ripeLimitReached = "%ERROR:202: access control limit reached"
)
// ripeRefusal is RIPE's answer to a host it no longer serves. // ripeRefusal is RIPE's answer to a host it no longer serves.
const ripeRefusal = `% This is the RIPE Database query service. const ripeRefusal = `% This is the RIPE Database query service.
% The objects are in RPSL format. % The objects are in RPSL format.
% %
%ERROR:201: access denied for 192.0.2.1 ` + ripeAccessDenied + `
% %
% Sorry, access from your host has been permanently % Sorry, access from your host has been permanently
% denied because of a repeated excessive querying. % denied because of a repeated excessive querying.
` `
// apnicAfrinicRecord is the answer of APNIC or AFRINIC for the AS when asked
// to leave out person and role objects. Like RIPE's, it holds the abuse
// address as a comment, the AS's object and the organisation object.
const apnicAfrinicRecord = `% Abuse contact for 'AS3333' is 'abuse@example.net'
aut-num: AS3333
as-name: EXAMPLE-AS
org: ORG-EXAMPLE1
admin-c: EX1-TEST
last-modified: 2026-08-26T00:40:00Z
organisation: ORG-EXAMPLE1
org-name: Example Networks
country: ZA
address: 1 Example Road
`
// ianaPrivateUse is IANA's answer for a private-use AS: its block and no // ianaPrivateUse is IANA's answer for a private-use AS: its block and no
// referral. // referral.
const ianaPrivateUse = `% IANA WHOIS server const ianaPrivateUse = `% IANA WHOIS server
@@ -80,20 +141,29 @@ const fillerLine = "remarks: and more\n"
const endlessAnswerSize = 4 << 20 const endlessAnswerSize = 4 << 20
// fakeServers returns a DialContext for a whois.Client that connects to a // fakeServers returns a DialContext for a whois.Client that connects to a
// fake server for each address in answers, which reads the query and sends // fake server at each address answers names. Each key of answers is a
// answers[address]. Connecting to any other address is refused. // server's address, a space and a query, and its value is that server's
// answer to that query. A server reads one query and sends its answer, or
// nothing for a query it has no answer for. Connecting to any other address
// is refused.
func fakeServers( func fakeServers(
t *testing.T, answers map[string]string, t *testing.T, answers map[string]string,
) func(context.Context, string, string) (net.Conn, error) { ) func(context.Context, string, string) (net.Conn, error) {
t.Helper() t.Helper()
addresses := make(map[string]bool)
for key := range answers {
address, _, _ := strings.Cut(key, " ")
addresses[address] = true
}
var servers sync.WaitGroup var servers sync.WaitGroup
t.Cleanup(servers.Wait) t.Cleanup(servers.Wait)
return func(_ context.Context, _, address string) (net.Conn, error) { return func(_ context.Context, _, address string) (net.Conn, error) {
answer, ok := answers[address] if !addresses[address] {
if !ok {
return nil, syscall.ECONNREFUSED return nil, syscall.ECONNREFUSED
} }
@@ -102,7 +172,8 @@ func fakeServers(
servers.Go(func() { servers.Go(func() {
defer func() { _ = server.Close() }() defer func() { _ = server.Close() }()
_, _ = bufio.NewReader(server).ReadString('\n') query, _ := bufio.NewReader(server).ReadString('\n')
answer := answers[address+" "+strings.TrimSpace(query)]
_, _ = io.WriteString(server, answer) _, _ = io.WriteString(server, answer)
}) })
@@ -122,25 +193,41 @@ func lookUp(
return client.LookupASN(t.Context(), testASN) return client.LookupASN(t.Context(), testASN)
} }
// TestLookupASNReadsTheRIRAnswer looks up an AS IANA refers to RIPE. The // TestLookupASNReadsTheRIRAnswer looks up an AS IANA refers to RIPE. RIPE
// fields come from RIPE's answer, the last-modified date included: IANA's // answers only a query that leaves out person and role objects, and its
// "changed:" line dates its block, not the AS. // answer to that still fills the AS's fields: the abuse address from the
// comment line, and the organisation's name and country from its object.
// The dates are the AS's: IANA's "changed:" line dates its block.
func TestLookupASNReadsTheRIRAnswer(t *testing.T) { func TestLookupASNReadsTheRIRAnswer(t *testing.T) {
t.Parallel() t.Parallel()
info, err := lookUp(t, map[string]string{ info, err := lookUp(t, map[string]string{
ianaServer: ianaReferral, ianaQuery: ianaReferral,
ripeServer: ripeRecord, ripeQuery: ripeRecord,
}) })
if err != nil { if err != nil {
t.Fatalf("LookupASN: %v", err) t.Fatalf("LookupASN: %v", err)
} }
if info.ASName != "RIPE-NCC-AS" || info.OrgID != "ORG-RIEN1-RIPE" || got := []string{
info.RIR != whois.RIRRIPE { info.ASName, info.OrgID, info.OrgName, info.CountryCode,
t.Errorf("LookupASN = AS name %q, org %q, RIR %q; "+ info.AbuseEmail, info.RIR,
"want RIPE-NCC-AS, ORG-RIEN1-RIPE, RIPE", }
info.ASName, info.OrgID, info.RIR) want := []string{
"RIPE-NCC-AS", "ORG-RIEN1-RIPE",
"Reseaux IP Europeens Network Coordination Centre (RIPE NCC)", "NL",
"abuse@ripe.net", whois.RIRRIPE,
}
if !slices.Equal(got, want) {
t.Errorf("LookupASN = AS name, org ID, org name, country, abuse "+
"address, RIR %q; want %q", got, want)
}
wantRegDate := time.Date(2002, 9, 17, 13, 8, 32, 0, time.UTC)
if info.RegDate == nil || !info.RegDate.Equal(wantRegDate) {
t.Errorf("LookupASN registered = %v, want %v",
info.RegDate, wantRegDate)
} }
wantLastMod := time.Date(2024, 1, 9, 8, 55, 41, 0, time.UTC) wantLastMod := time.Date(2024, 1, 9, 8, 55, 41, 0, time.UTC)
@@ -150,6 +237,65 @@ func TestLookupASNReadsTheRIRAnswer(t *testing.T) {
} }
} }
// TestLookupASNAsksAPNICAndAFRINICToLeaveOutPersonalData looks up an AS at
// APNIC and at AFRINIC. Each answers only a query that leaves out person
// and role objects, as RIPE does. That one answer still holds the
// organisation object, so the organisation's name and country are filled.
func TestLookupASNAsksAPNICAndAFRINICToLeaveOutPersonalData(t *testing.T) {
t.Parallel()
for _, server := range []string{"whois.apnic.net", "whois.afrinic.net"} {
t.Run(server, func(t *testing.T) {
t.Parallel()
address := server + ":43"
info, err := lookUp(t, map[string]string{
ianaQuery: "refer: " + server + "\n",
address + " --no-personal " + asQuery: apnicAfrinicRecord,
})
if err != nil {
t.Fatalf("LookupASN: %v", err)
}
got := []string{
info.ASName, info.OrgName, info.CountryCode, info.AbuseEmail,
}
want := []string{
"EXAMPLE-AS", "Example Networks", "ZA", "abuse@example.net",
}
if !slices.Equal(got, want) {
t.Errorf("LookupASN = AS name, org name, country, abuse "+
"address %q; want %q", got, want)
}
})
}
}
// TestLookupASNReportsARefusal looks up an AS IANA refers to RIPE when RIPE
// refuses the query, with each of the error lines RIPE refuses with. The
// error holds RIPE's line, so the log says why the lookup failed.
func TestLookupASNReportsARefusal(t *testing.T) {
t.Parallel()
for _, line := range []string{ripeAccessDenied, ripeLimitReached} {
t.Run(line, func(t *testing.T) {
t.Parallel()
_, err := lookUp(t, map[string]string{
ianaQuery: ianaReferral,
ripeQuery: "% This is the RIPE Database query service.\n" +
line + "\n",
})
if err == nil || !strings.Contains(err.Error(), line) {
t.Errorf("LookupASN error = %v, want one holding %q",
err, line)
}
})
}
}
// TestLookupASNFailsWhenTheRIRGivesNoAnswer looks up an AS IANA refers to // TestLookupASNFailsWhenTheRIRGivesNoAnswer looks up an AS IANA refers to
// RIPE when RIPE's server refuses the connection, and when it answers only // RIPE when RIPE's server refuses the connection, and when it answers only
// with an error. Neither is a lookup that found the AS. // with an error. Neither is a lookup that found the AS.
@@ -157,8 +303,8 @@ func TestLookupASNFailsWhenTheRIRGivesNoAnswer(t *testing.T) {
t.Parallel() t.Parallel()
for name, answers := range map[string]map[string]string{ for name, answers := range map[string]map[string]string{
"connection refused": {ianaServer: ianaReferral}, "connection refused": {ianaQuery: ianaReferral},
"only an error": {ianaServer: ianaReferral, ripeServer: ripeRefusal}, "only an error": {ianaQuery: ianaReferral, ripeQuery: ripeRefusal},
} { } {
t.Run(name, func(t *testing.T) { t.Run(name, func(t *testing.T) {
t.Parallel() t.Parallel()
@@ -177,7 +323,7 @@ func TestLookupASNFailsWhenTheRIRGivesNoAnswer(t *testing.T) {
func TestLookupASNReportsNoReferral(t *testing.T) { func TestLookupASNReportsNoReferral(t *testing.T) {
t.Parallel() t.Parallel()
_, err := lookUp(t, map[string]string{ianaServer: ianaPrivateUse}) _, err := lookUp(t, map[string]string{ianaQuery: ianaPrivateUse})
if !errors.Is(err, whois.ErrNoReferral) { if !errors.Is(err, whois.ErrNoReferral) {
t.Errorf("LookupASN error = %v, want ErrNoReferral", err) t.Errorf("LookupASN error = %v, want ErrNoReferral", err)
} }
@@ -189,7 +335,7 @@ func TestLookupASNReportsNoReferral(t *testing.T) {
func TestLookupASNFailsWhenIANAAnswersNoBlock(t *testing.T) { func TestLookupASNFailsWhenIANAAnswersNoBlock(t *testing.T) {
t.Parallel() t.Parallel()
_, err := lookUp(t, map[string]string{ianaServer: ianaNoObject}) _, err := lookUp(t, map[string]string{ianaQuery: ianaNoObject})
if err == nil || errors.Is(err, whois.ErrNoReferral) { if err == nil || errors.Is(err, whois.ErrNoReferral) {
t.Errorf("LookupASN error = %v, want one that is not ErrNoReferral", t.Errorf("LookupASN error = %v, want one that is not ErrNoReferral",
err) err)
@@ -206,8 +352,8 @@ func TestLookupASNFailsWhenTheRIRAnswerDoesNotEnd(t *testing.T) {
strings.Repeat(fillerLine, endlessAnswerSize/len(fillerLine)) strings.Repeat(fillerLine, endlessAnswerSize/len(fillerLine))
_, err := lookUp(t, map[string]string{ _, err := lookUp(t, map[string]string{
ianaServer: ianaReferral, ianaQuery: ianaReferral,
ripeServer: endless, ripeQuery: endless,
}) })
if err == nil || errors.Is(err, whois.ErrNoReferral) { if err == nil || errors.Is(err, whois.ErrNoReferral) {
t.Errorf("LookupASN error = %v, want one that is not ErrNoReferral", t.Errorf("LookupASN error = %v, want one that is not ErrNoReferral",
@@ -233,9 +379,9 @@ func TestLookupASNFailsOnAReferralToAnotherServer(t *testing.T) {
t.Parallel() t.Parallel()
_, err := lookUp(t, map[string]string{ _, err := lookUp(t, map[string]string{
ianaServer: "refer: " + refer + "\n\n" + ianaQuery: "refer: " + refer + "\n\n" +
"as-block: 3154-3353\n", "as-block: 3154-3353\n",
address: ripeRecord, address + " " + asQuery: ripeRecord,
}) })
if err == nil || errors.Is(err, whois.ErrNoReferral) { if err == nil || errors.Is(err, whois.ErrNoReferral) {
t.Errorf("LookupASN error = %v, want one that is not "+ t.Errorf("LookupASN error = %v, want one that is not "+
+49 -23
View File
@@ -2,10 +2,13 @@ package asinfo
import ( import (
"bytes" "bytes"
"cmp"
"compress/gzip" "compress/gzip"
_ "embed" _ "embed"
"encoding/json" "encoding/json"
"io" "io"
"slices"
"strings"
"sync" "sync"
) )
@@ -23,6 +26,17 @@ type Info struct {
type Registry struct { type Registry struct {
mu sync.RWMutex mu sync.RWMutex
byASN map[int]*Info byASN map[int]*Info
// searchEntries holds every entry in AS number order, for Search.
searchEntries []searchEntry
}
// searchEntry is an entry with its handle and description in lower case. They
// are put in lower case once, when the data loads, so that Search only
// compares.
type searchEntry struct {
info *Info
lowerHandle string
lowerDescription string
} }
var ( var (
@@ -59,9 +73,21 @@ func initRegistry() {
panic("failed to unmarshal embedded AS data: " + err.Error()) panic("failed to unmarshal embedded AS data: " + err.Error())
} }
slices.SortFunc(asInfos, func(a, b Info) int {
return cmp.Compare(a.ASN, b.ASN)
})
defaultRegistry.searchEntries = make([]searchEntry, 0, len(asInfos))
for i := range asInfos { for i := range asInfos {
info := &asInfos[i] info := &asInfos[i]
defaultRegistry.byASN[info.ASN] = info defaultRegistry.byASN[info.ASN] = info
defaultRegistry.searchEntries = append(defaultRegistry.searchEntries,
searchEntry{
info: info,
lowerHandle: strings.ToLower(info.Handle),
lowerDescription: strings.ToLower(info.Description),
})
} }
} }
@@ -129,36 +155,36 @@ func All() []Info {
return result return result
} }
// Search searches for AS entries where the handle or description contains the query // Search finds the AS entries whose handle or description contains query,
// This is a simple case-sensitive substring search // ignoring case. It returns the first limit of them in AS number order, and the
func Search(query string) []Info { // number of entries that match. An empty query matches nothing.
func Search(query string, limit int) ([]Info, int) {
once.Do(initRegistry) once.Do(initRegistry)
if query == "" {
return nil, 0
}
query = strings.ToLower(query)
defaultRegistry.mu.RLock() defaultRegistry.mu.RLock()
defer defaultRegistry.mu.RUnlock() defer defaultRegistry.mu.RUnlock()
var results []Info var (
results []Info
total int
)
for _, info := range defaultRegistry.byASN { for _, entry := range defaultRegistry.searchEntries {
if contains(info.Handle, query) || contains(info.Description, query) { if strings.Contains(entry.lowerHandle, query) ||
results = append(results, *info) strings.Contains(entry.lowerDescription, query) {
if total < limit {
results = append(results, *entry.info)
}
total++
} }
} }
return results return results, total
}
// contains is a simple substring search
func contains(s, substr string) bool {
return len(substr) > 0 && len(s) >= len(substr) && containsImpl(s, substr)
}
func containsImpl(s, substr string) bool {
for i := 0; i <= len(s)-len(substr); i++ {
if s[i:i+len(substr)] == substr {
return true
}
}
return false
} }
+103 -15
View File
@@ -1,6 +1,8 @@
package asinfo_test package asinfo_test
import ( import (
"cmp"
"slices"
"strings" "strings"
"testing" "testing"
@@ -140,11 +142,6 @@ func TestSearch(t *testing.T) {
query: "University", query: "University",
wantMin: 10, wantMin: 10,
}, },
{
name: "Empty search",
query: "",
wantMin: 0,
},
{ {
name: "Non-existent org", name: "Non-existent org",
query: "XYZABC123456", query: "XYZABC123456",
@@ -156,26 +153,115 @@ func TestSearch(t *testing.T) {
t.Run(tt.name, func(t *testing.T) { t.Run(tt.name, func(t *testing.T) {
t.Parallel() t.Parallel()
results := asinfo.Search(tt.query) results, _ := asinfo.Search(tt.query, asinfo.Total())
if len(results) < tt.wantMin { if len(results) < tt.wantMin {
t.Errorf("Search(%q) returned %d results, want at least %d", t.Errorf("Search(%q) returned %d results, want at least %d",
tt.query, len(results), tt.wantMin) tt.query, len(results), tt.wantMin)
} }
// Verify all results contain the query // Verify all results contain the query, ignoring case
if tt.query != "" { query := strings.ToLower(tt.query)
for _, r := range results { for _, r := range results {
if !strings.Contains(r.Handle, tt.query) && if !strings.Contains(strings.ToLower(r.Handle), query) &&
!strings.Contains(r.Description, tt.query) { !strings.Contains(strings.ToLower(r.Description), query) {
t.Errorf("Search result ASN %d doesn't contain query %q", t.Errorf("Search result ASN %d doesn't contain query %q",
r.ASN, tt.query) r.ASN, tt.query)
}
} }
} }
}) })
} }
} }
// TestSearchEmptyQueryMatchesNothing expects an empty query to match no AS,
// although every handle contains the empty string.
func TestSearchEmptyQueryMatchesNothing(t *testing.T) {
t.Parallel()
results, total := asinfo.Search("", asinfo.Total())
if len(results) != 0 || total != 0 {
t.Errorf(`Search("") returned %d results and a total of %d, want none`,
len(results), total)
}
}
// TestSearchIgnoresCase searches for Cloudflare's handle in lower case and its
// description in upper case, and expects both searches to find AS13335.
func TestSearchIgnoresCase(t *testing.T) {
t.Parallel()
const cloudflareASN = 13335 // handle CLOUDFLARENET, "Cloudflare, Inc."
isCloudflare := func(info asinfo.Info) bool {
return info.ASN == cloudflareASN
}
for _, query := range []string{"cloudflarenet", "CLOUDFLARE, INC."} {
results, _ := asinfo.Search(query, asinfo.Total())
if !slices.ContainsFunc(results, isCloudflare) {
t.Errorf("Search(%q) did not find AS%d", query, cloudflareASN)
}
}
}
// TestSearchOrdersByASN expects the matches of a search with many of them to
// come in AS number order.
func TestSearchOrdersByASN(t *testing.T) {
t.Parallel()
const query = "University"
results, _ := asinfo.Search(query, asinfo.Total())
for i := 1; i < len(results); i++ {
if results[i-1].ASN >= results[i].ASN {
t.Fatalf("Search(%q): AS%d comes after AS%d",
query, results[i].ASN, results[i-1].ASN)
}
}
}
// TestSearchStopsAtLimit searches for a word that matches more entries than
// the limit. It expects exactly limit entries, the first matches in AS number
// order, and the number of all matches, counted here from All.
func TestSearchStopsAtLimit(t *testing.T) {
t.Parallel()
const (
query = "university"
limit = 10
)
var matches []asinfo.Info
for _, info := range asinfo.All() {
if strings.Contains(strings.ToLower(info.Handle), query) ||
strings.Contains(strings.ToLower(info.Description), query) {
matches = append(matches, info)
}
}
if len(matches) <= limit {
t.Fatalf("%q matches %d entries; the test needs more than %d",
query, len(matches), limit)
}
slices.SortFunc(matches, func(a, b asinfo.Info) int {
return cmp.Compare(a.ASN, b.ASN)
})
results, total := asinfo.Search(query, limit)
if total != len(matches) {
t.Errorf("Search(%q, %d): total %d, want %d",
query, limit, total, len(matches))
}
if !slices.Equal(results, matches[:limit]) {
t.Errorf("Search(%q, %d) = %v, want %v",
query, limit, results, matches[:limit])
}
}
func TestDataIntegrity(t *testing.T) { func TestDataIntegrity(t *testing.T) {
t.Parallel() t.Parallel()
@@ -215,11 +301,13 @@ func BenchmarkGet(b *testing.B) {
} }
func BenchmarkSearch(b *testing.B) { func BenchmarkSearch(b *testing.B) {
const limit = 50
queries := []string{"Google", "Amazon", "Microsoft", "University"} queries := []string{"Google", "Amazon", "Microsoft", "University"}
b.ResetTimer() b.ResetTimer()
for i := range b.N { for i := range b.N {
asinfo.Search(queries[i%len(queries)]) asinfo.Search(queries[i%len(queries)], limit)
} }
} }
+4 -2
View File
@@ -16,8 +16,10 @@ Basic usage:
// Get just the description // Get just the description
desc := asinfo.GetDescription(15169) // "Google LLC" desc := asinfo.GetDescription(15169) // "Google LLC"
// Search for AS entries // Search the handles and descriptions, ignoring case, for the first 10
results := asinfo.Search("University") // matches in AS number order and the number of all matches
results, total := asinfo.Search("university", 10)
fmt.Printf("%d matches\n", total)
for _, as := range results { for _, as := range results {
fmt.Printf("AS%d: %s\n", as.ASN, as.Description) fmt.Printf("AS%d: %s\n", as.ASN, as.Description)
} }
+9 -2
View File
@@ -21,9 +21,16 @@ func ExampleGetDescription() {
} }
func ExampleSearch() { func ExampleSearch() {
results := asinfo.Search("MIT-GATEWAY") const limit = 10
results, total := asinfo.Search("mit-gateway", limit)
fmt.Printf("%d of %d matches\n", len(results), total)
for _, info := range results { for _, info := range results {
fmt.Printf("AS%d: %s - %s\n", info.ASN, info.Handle, info.Description) fmt.Printf("AS%d: %s - %s\n", info.ASN, info.Handle, info.Description)
} }
// Output: AS3: MIT-GATEWAYS - Massachusetts Institute of Technology // Output:
// 1 of 1 matches
// AS3: MIT-GATEWAYS - Massachusetts Institute of Technology
} }