14 Commits
Author SHA1 Message Date
clawbot 99c7efc156 Make the home page's AS name search work (closes #151)
check / check (push) Waiting to run
The AS Name Search card on the home page said the search was coming soon.
It now asks the new GET /api/v1/search/as?q=<text> and lists the matches
as links to their AS pages. asinfo.Search ignores case, returns the first
matches up to a limit in AS number order, and counts all matches. An
empty q answers 400.

When the database has no record of an AS, its page and /api/v1/as/{asn}
take its handle and description from asinfo and still list its prefixes.
They answer 404 only for an AS with no record, no asinfo entry and no
prefix. AS 0 means no known origin: it always answers 404, and the search
never lists it.

Model: opus-5-5
2026-10-26 22:51:12 +01:00
clawbot 71ee2aeda7 Log the start-up checkpoint's figures under their own names (closes #117)
check / check (push) Canceled after 0s
Initialize scanned the row of PRAGMA wal_checkpoint(TRUNCATE), which is
the busy flag, the frames in the WAL and the frames copied, into
wal_pages, checkpointed and moved; and a completed TRUNCATE reads 0 for
both counts. The line now logs busy, wal_frames and checkpointed_frames,
plus wal_bytes, the WAL file's size, which New reads before it opens the
database because the open writes a frame to the WAL. The line is a
warning when busy is 1. Tests open a database a second time while the
first stays open, with and without a read open on the first, and check
the line's level and figures.

Model: opus-5-5
2026-10-20 04:32:08 +02:00
clawbot 8d16bf9ea6 Add the Rationale section to README (closes #121)
check / check (push) Canceled after 0s
REPO_POLICIES.md requires a Rationale section saying why the program
exists. RIS Live sends only the changes to the routing table, so it
cannot answer which AS an address is routed to now; routewatch applies
the changes to its own tables and answers from them. The section says
those tables hold only the routes announced since routewatch first
started, and miss the updates it drops under load or while it is
stopped.

Model: opus-5-5
2026-10-20 04:24:41 +02:00
clawbot 651a9e22f6 Treat a path with AS 0 anywhere as a withdrawal (closes #141)
check / check (push) Canceled after 0s
An AS path that holds AS 0 anywhere, a member of an AS set included, now
decodes as an empty path. RFC 7607 makes such a path malformed and its
announcements withdrawals. An empty path already gets that handling:
the prefix handler deletes the peer's route for the prefix, and the AS
and peering handlers record nothing from it. Before, a path such as
[64496, 0, 64497] stored the route with origin AS64497, recorded both
ASes, and recorded the peerings between the path's other ASes.

Model: opus-5-5
2026-10-20 04:07:41 +02:00
clawbot 9280e2eb2c Keep the database files, logs and binaries out of the build context (closes #120)
check / check (push) Canceled after 0s
The daemon's database is db.sqlite with its -wal and -shm files, which
the .db patterns in .dockerignore and .gitignore did not match, so a
database left in the clone shipped in the image's source archive.
.dockerignore now also excludes the logs and the .exe, .dll, .so and
.dylib files .gitignore already ignores. A test opens a database and
checks that .dockerignore has an any-depth pattern for each of its
files.

Model: opus-5-5
2026-10-20 03:58:56 +02:00
clawbot 100689a4fb Keep prefixes that do not parse out of the prefix tables (closes #114)
check / check (push) Canceled after 0s
UpdatePrefixesBatch now skips a prefix that does not parse, so it never
becomes a row in prefixes_v4 or prefixes_v6 and never counts in the prefix
total in the stats. Before, the prefix handler stored no route for such a
prefix and logged it once per prefix and peer, but still passed it to
UpdatePrefixesBatch, which stored it.

Model: opus-5-5
2026-10-20 03:51:10 +02:00
clawbot e13240cb04 Show the MIT licence in the page footers (closes #119)
check / check (push) Canceled after 0s
version.License, which the home and status page footers show, still
said WTFPL after LICENSE and the README moved to MIT. It now says MIT.
A test renders both pages and expects each footer to name MIT and not
WTFPL.

Model: opus-5-5
2026-10-20 03:46:44 +02:00
clawbot 55ecb22846 Bound the WHOIS fetcher's success and error time lists (closes #118)
check / check (push) Canceled after 0s
recordSuccess and recordFailure appended a time on every fetch, and only
GetStats, run when a stats endpoint is requested, dropped the times older
than statsWindow, so a daemon nobody polled kept every one. Each now drops
those times before it appends. The drop takes them off the front of the
list instead of copying the rest: the times are in the order they were
recorded, and it now runs on every fetch.

The fetcher reads the time through a now field so a test can set the
clock. The test records 100,000 successes and failures a second apart
without asking for the stats and finds one hour's worth of each kept.

Model: opus-5-5
2026-10-20 03:38:51 +02:00
clawbot bae5a14d76 Reset the peering handler's dropped_paths figure after each log line (closes #116)
check / check (push) Canceled after 0s
processPeerings replaced the path map every run but kept the count of
paths dropped while the map was full, so every later "Processed AS
peerings" line reported all drops since start next to that run's own
figures. The count is now set to 0 where the map is replaced, under the
same lock. A test fills the map, has one path dropped, runs the
processing and finds the count back at 0.

Model: opus-5-5
2026-10-20 03:28:06 +02:00
clawbot 467c295f94 Handle an announcement without an origin like a replacement (closes #113)
check / check (push) Canceled after 0s
An announcement whose AS path is empty or ends in AS 0 now deletes the
peer's route for the prefix, as a withdrawal does. Before, it was neither
stored nor applied: the peer's old route stayed live, and within one
batch it cancelled the peer's earlier withdrawal of the prefix.

Model: opus-5-5
2026-10-20 03:25:41 +02:00
clawbot 2f238236bd Read the drop counter atomically and set the WHOIS fetcher before serving (closes #115)
check / check (push) Canceled after 0s
The read loop added to a handler's drop count atomically while the
statistics endpoint and the metrics log read it as a plain field, a data
race whenever messages were dropped. The count is now an atomic.Uint64,
read with Load. A test reads the handler statistics while the read loop
drops messages for a handler that never finishes one.

Run created the WHOIS fetcher and gave it to the HTTP server after the
server had started, while request handlers could read it. The fetcher is
now set before the server starts.

Model: opus-5-5
2026-10-20 03:18:00 +02:00
clawbot f489321e3a Answer HEAD requests and give 405 a plain message (closes #111)
check / check (push) Canceled after 0s
Every route was registered for GET only, so HEAD answered 405, and on
the JSON routes that 405 said "Internal server error". chi's GetHead
now serves HEAD with the GET handler; /ip and /ip/ register HEAD
themselves, as GetHead does not reach them. The /api/v1 and /ip route
groups answer a path they lack with a JSON 404 "Not found", and the
other methods the router knows (POST, PUT, PATCH, DELETE, OPTIONS,
TRACE, CONNECT) with a JSON 405 "Method not allowed" whose Allow header
names GET and HEAD. JSONValidationMiddleware gives a 4xx sent without a
JSON body the status's standard text instead of "Internal server error".

Model: opus-5-5
2026-10-20 03:06:23 +02:00
clawbot 444457bcf8 Store a new row's earliest timestamp as first_seen (closes #112)
check / check (push) Canceled after 0s
Each handler took the latest time per key in a batch and inserted it as
a new row's first_seen, so a key updated twice in one batch, seconds
apart, was stored first seen at the later time, and the IP lookup could
answer with a prefix first_seen after the route's last_seen.

The prefix handler now keeps the earliest time of all of the batch's
updates for each prefix. The AS, peer and peering handlers keep the
earliest and the latest time per key in database.SeenTimes, and
GetOrCreateASNBatch, UpdatePeerBatch and RecordPeering store a new row
first seen at the earliest and last seen at the latest. A stored row
still has only its last_seen set, to the latest.

Model: opus-5-5
2026-10-20 02:59:17 +02:00
clawbot c4dadd1477 Leave SIGINT and SIGTERM to fx (closes #131)
check / check (push) Canceled after 0s
The start hook registered its own handler for SIGINT and SIGTERM and
asked fx to shut down from it, although fx already stops the daemon on
both signals. A stop signal was handled twice, and the handler stayed
registered after the stop. fx now handles both signals alone, as
GO_HTTP_SERVER_CONVENTIONS.md "Signal Handling and Graceful Shutdown"
requires.

Model: opus-5-5
2026-10-20 02:34:04 +02:00
43 changed files with 2240 additions and 267 deletions
+12 -2
View File
@@ -73,7 +73,8 @@
# This repository's host-built artifacts: `make build` writes bin/, `go # This repository's host-built artifacts: `make build` writes bin/, `go
# build` in the root the binaries, `make run` log.txt, `make asupdate` its # build` in the root the binaries, `make run` log.txt, `make asupdate` its
# .tmp file; test binaries, coverage profiles and source archives. # .tmp file; logs, test binaries, coverage profiles, source archives, and
# the binaries and libraries a build for another platform writes.
/bin /bin
/routewatch /routewatch
/asinfo-gen /asinfo-gen
@@ -82,15 +83,24 @@
/out /out
/pkg/asinfo/asdata.json /pkg/asinfo/asdata.json
**/*.tar.zst **/*.tar.zst
**/*.log
**/*.test **/*.test
**/*.out **/*.out
**/*.tmp **/*.tmp
**/*.exe
**/*.dll
**/*.so
**/*.dylib
# Local databases. The image carries a source archive of the whole build # Local databases. The image carries a source archive of the whole build
# context, so these would otherwise ship inside it. # context, so these would otherwise ship inside it. The daemon's database
# is db.sqlite, with its -wal and -shm files while it runs.
**/*.db **/*.db
**/*.db-journal **/*.db-journal
**/*.db-wal **/*.db-wal
**/*.sqlite
**/*.sqlite-wal
**/*.sqlite-shm
# A local Go workspace points at directories outside the build context. # A local Go workspace points at directories outside the build context.
/go.work /go.work
+5 -1
View File
@@ -67,10 +67,14 @@ node_modules/
go.work go.work
go.work.sum go.work.sum
# Local databases, temporary files and debug output. # Local databases, temporary files and debug output. The daemon's
# database is db.sqlite, with its -wal and -shm files while it runs.
*.db *.db
*.db-journal *.db-journal
*.db-wal *.db-wal
*.sqlite
*.sqlite-wal
*.sqlite-shm
*.tmp *.tmp
log.txt log.txt
out out
+49 -9
View File
@@ -5,6 +5,22 @@ table in real time: it streams BGP UPDATE messages from the RIPE RIS Live
service, maintains a live routing table in SQLite, and provides HTTP APIs for service, maintains a live routing table in SQLite, and provides HTTP APIs for
querying routing information. querying routing information.
## Rationale
RIPE RIS Live streams the BGP updates that RIPE's route collectors receive, as
they happen, from the moment a client connects. It reports changes, not the
routing table they add up to, so it cannot say which AS an IP address is routed
to now or which routes a prefix has. RouteWatch applies the feed's updates to
live routing tables in its own SQLite database and answers IP-to-AS lookups and
prefix pages from them.
The feed sends no table when a client connects, so these tables hold only the
routes the feed has announced since RouteWatch first started: an IP-to-AS lookup
finds nothing for an address whose routes have not been announced since then.
The tables also miss the updates RouteWatch drops when it falls behind under
load and those sent while it is stopped, so a lookup or a prefix page can show a
route the feed has withdrawn.
## Features ## Features
- Real-time streaming of BGP updates from RIPE RIS Live - Real-time streaming of BGP updates from RIPE RIS Live
@@ -36,17 +52,21 @@ DEBUG=routewatch ./routewatch
## HTTP Endpoints ## HTTP Endpoints
Every route answers `HEAD` as it answers `GET`, without the body.
### Web Interface ### Web Interface
- `GET /` - Home page (HTML): headline figures, and lookups by AS number and by - `GET /` - Home page (HTML): headline figures, lookups by AS number and by IP
IP address address, and a search of AS names
- `GET /status` - HTML status dashboard, which reads `/api/v1/stats` - `GET /status` - HTML status dashboard, which reads `/api/v1/stats`
- `GET /status.json` - JSON statistics: feed, memory, database and route figures - `GET /status.json` - JSON statistics: feed, memory, database and route figures
- `GET /.well-known/healthcheck.json` - Health check (JSON): 200 when the - `GET /.well-known/healthcheck.json` - Health check (JSON): 200 when the
database answers and the RIS Live feed is connected, otherwise 503; database answers and the RIS Live feed is connected, otherwise 503;
`data.status` is `ok` or `error` and `data.checks` says which check failed `data.status` is `ok` or `error` and `data.checks` says which check failed
- `GET /as/{asn}` - AS detail page (HTML): the prefixes the AS originates and - `GET /as/{asn}` - AS detail page (HTML): the prefixes the AS originates and
its peers its peers. When the database has no record of the AS, its handle and
description come from the AS list built into RouteWatch. The page then shows
no first-seen date.
- `GET /prefix/{prefix}/{length}` - Prefix detail page (HTML), such as - `GET /prefix/{prefix}/{length}` - Prefix detail page (HTML), such as
`/prefix/192.0.2.0/24`: every live route for the prefix, one per peer, and its `/prefix/192.0.2.0/24`: every live route for the prefix, one per peer, and its
origin ASes origin ASes
@@ -75,9 +95,22 @@ DEBUG=routewatch ./routewatch
and 404 when none of the addresses has a live route. A PTR lookup gets 2 and 404 when none of the addresses has a live route. A PTR lookup gets 2
seconds; an address whose lookup takes longer is answered without PTR names. seconds; an address whose lookup takes longer is answered without PTR names.
- `GET /api/v1/as/{asn}` - The AS and the prefixes it originates, IPv4 and IPv6 - `GET /api/v1/as/{asn}` - The AS and the prefixes it originates, IPv4 and IPv6
apart, each with when its newest route was updated apart, each with when its newest route was updated. When the database has no
record of the AS, its handle and description come from the AS list built into
RouteWatch. The answer then has no `first_seen` or `last_seen`.
- `GET /api/v1/prefix/{prefix}/{length}` - Every live route for the prefix, one - `GET /api/v1/prefix/{prefix}/{length}` - Every live route for the prefix, one
per peer, and the same routes grouped by origin AS in `origins` per peer, and the same routes grouped by origin AS in `origins`
- `GET /api/v1/search/as?q={text}` - Finds the ASes whose handle or description
contains `{text}`, ignoring case. `data` holds:
- `query`: `{text}` without its leading and trailing spaces
- `total`: the number of ASes that match
- `results`: the first 50 of them in AS number order, each with `asn`,
`handle` and `description`
The search reads the AS list built into RouteWatch, not the database. The
list comes from [ipverse/asn-info](https://github.com/ipverse/asn-info). A
match can therefore be an AS that the database has no record of. A `q` that
is missing, empty or only spaces answers 400.
A JSON answer is `{"status": "ok", "data": ...}`, or for an error A JSON answer is `{"status": "ok", "data": ...}`, or for an error
`{"status": "error", "error": {"msg": ..., "code": ...}}`, and carries an `{"status": "error", "error": {"msg": ..., "code": ...}}`, and carries an
@@ -88,10 +121,16 @@ took. The health check is the exception: its 503 answer is also
A request that runs past its time limit answers 504: the limit is 30 seconds, A request that runs past its time limit answers 504: the limit is 30 seconds,
and 4 seconds for the database reads behind `/status.json` and `/api/v1/stats`. and 4 seconds for the database reads behind `/status.json` and `/api/v1/stats`.
An AS number, prefix or mask length that does not parse answers 400, as does a An AS number, prefix or mask length that does not parse answers 400, as does a
mask length out of range for its IP version. An AS the database does not hold, mask length out of range for its IP version. An AS answers 404 when it is not in
or a prefix with no live route, answers 404. The HTML pages give these errors as the AS list built into RouteWatch and the database has neither a record of it
plain text. A 500 answer says only `Internal server error`, and a host name that nor a live route it originates. AS 0 always answers 404, because it means no
does not resolve only `could not resolve host name`; the detail goes to the log. known origin. The AS name search never lists it. A prefix with no live route
answers 404 too. The HTML pages give these errors as plain text. Under `/api/v1`
and `/ip/`, a path no route matches answers 404 with the message `Not found`,
and `POST`, `PUT`, `PATCH`, `DELETE`, `OPTIONS`, `TRACE` or `CONNECT` answers
405 with `Method not allowed`. A 500 answer says only `Internal server error`,
and a host name that does not resolve only `could not resolve host name`; the
detail goes to the log.
Every answer carries `Strict-Transport-Security`, `Content-Security-Policy`, Every answer carries `Strict-Transport-Security`, `Content-Security-Policy`,
`X-Frame-Options`, `X-Content-Type-Options`, `Referrer-Policy` and `X-Frame-Options`, `X-Content-Type-Options`, `Referrer-Policy` and
@@ -190,7 +229,8 @@ routewatch/
- **ASHandler**: Tracks all ASNs seen in AS paths - **ASHandler**: Tracks all ASNs seen in AS paths
- **PeerHandler**: Records BGP peer information - **PeerHandler**: Records BGP peer information
- **PrefixHandler**: Maintains live routing table (upserts on announcement, - **PrefixHandler**: Maintains live routing table (upserts on announcement,
deletes on withdrawal, deletes all of a peer's routes when its BGP session deletes on withdrawal or on an announcement whose AS path is empty or
holds AS 0 anywhere, deletes all of a peer's routes when its BGP session
goes down). Routes are kept per peer IP, so when one peer IP has sessions goes down). Routes are kept per peer IP, so when one peer IP has sessions
on two collectors, a session going down on one also removes the routes the on two collectors, a session going down on one also removes the routes the
other still carries, until they are announced again. other still carries, until they are announced again.
+5
View File
@@ -33,6 +33,11 @@ tested.
# Completed Steps # Completed Steps
- 2026-10-26: the home page's AS name search lists the ASes whose handle or
description holds the text, as links to their pages, from the new
`/api/v1/search/as`; the search ignores case. When the database has no record
of an AS, its page takes the handle and description from the AS list built
into RouteWatch and still lists the AS's prefixes (closes #151)
- 2026-10-19: README "HTTP Endpoints" lists every route with what it answers, - 2026-10-19: README "HTTP Endpoints" lists every route with what it answers,
"Database Schema" matches `schema.sql`, and the rest of the README matches the "Database Schema" matches `schema.sql`, and the rest of the README matches the
code; this file brought up to date (closes #72) code; this file brought up to date (closes #72)
+9 -4
View File
@@ -148,7 +148,9 @@ func TestLiveCountsTrackWritesInRealtime(t *testing.T) {
func writeASNsPrefixesAndPeers(t *testing.T, db *Database, ts time.Time) { func writeASNsPrefixesAndPeers(t *testing.T, db *Database, ts time.Time) {
t.Helper() t.Helper()
err := db.GetOrCreateASNBatch(map[int]time.Time{64500: ts, 64501: ts}) seen := SeenTimes{First: ts, Last: ts}
err := db.GetOrCreateASNBatch(map[int]SeenTimes{64500: seen, 64501: seen})
if err != nil { if err != nil {
t.Fatalf("GetOrCreateASNBatch: %v", err) t.Fatalf("GetOrCreateASNBatch: %v", err)
} }
@@ -164,14 +166,14 @@ func writeASNsPrefixesAndPeers(t *testing.T, db *Database, ts time.Time) {
err = db.UpdatePeerBatch(map[string]PeerUpdate{ err = db.UpdatePeerBatch(map[string]PeerUpdate{
testV4PeerIP: { testV4PeerIP: {
PeerIP: testV4PeerIP, PeerASN: 64500, PeerIP: testV4PeerIP, PeerASN: 64500,
MessageType: "UPDATE", Timestamp: ts, MessageType: "UPDATE", Seen: seen,
}, },
}) })
if err != nil { if err != nil {
t.Fatalf("UpdatePeerBatch: %v", err) t.Fatalf("UpdatePeerBatch: %v", err)
} }
err = db.RecordPeering(64500, 64501, ts) err = db.RecordPeering(64500, 64501, seen)
if err != nil { if err != nil {
t.Fatalf("RecordPeering: %v", err) t.Fatalf("RecordPeering: %v", err)
} }
@@ -192,7 +194,10 @@ func TestLiveCountsSeededFromDatabaseAtStartup(t *testing.T) {
ts := time.Date(2026, 1, 2, 3, 4, 5, 0, time.UTC) ts := time.Date(2026, 1, 2, 3, 4, 5, 0, time.UTC)
err = db.GetOrCreateASNBatch(map[int]time.Time{64500: ts, 64501: ts, 64502: ts}) seen := SeenTimes{First: ts, Last: ts}
err = db.GetOrCreateASNBatch(
map[int]SeenTimes{64500: seen, 64501: seen, 64502: seen})
if err != nil { if err != nil {
t.Fatalf("GetOrCreateASNBatch: %v", err) t.Fatalf("GetOrCreateASNBatch: %v", err)
} }
+109 -37
View File
@@ -21,6 +21,7 @@ import (
"git.eeqj.de/sneak/routewatch/internal/config" "git.eeqj.de/sneak/routewatch/internal/config"
"git.eeqj.de/sneak/routewatch/internal/logger" "git.eeqj.de/sneak/routewatch/internal/logger"
"git.eeqj.de/sneak/routewatch/internal/ristypes"
"git.eeqj.de/sneak/routewatch/pkg/asinfo" "git.eeqj.de/sneak/routewatch/pkg/asinfo"
_ "github.com/mattn/go-sqlite3" // CGO SQLite driver _ "github.com/mattn/go-sqlite3" // CGO SQLite driver
) )
@@ -169,6 +170,20 @@ func New(cfg *config.Config, logger *logger.Logger) (*Database, error) {
return nil, fmt.Errorf("failed to create database directory: %w", err) return nil, fmt.Errorf("failed to create database directory: %w", err)
} }
// The size of the WAL file the previous run left, which Initialize logs
// with the start-up checkpoint. It is read before the database is opened,
// because opening it writes a frame to the WAL. There is no WAL file when
// the database is new or the last run closed the database, which deletes
// it.
var walBytes int64
wal, err := os.Stat(dbPath + walFileSuffix)
if err == nil {
walBytes = wal.Size()
} else if !errors.Is(err, os.ErrNotExist) {
logger.Warn("Failed to read the WAL file's size on startup", "error", err)
}
// Per-connection SQLite settings go in the DSN so every pooled connection // Per-connection SQLite settings go in the DSN so every pooled connection
// gets them, not just the one that runs the Initialize pragmas. _txlock= // gets them, not just the one that runs the Initialize pragmas. _txlock=
// immediate makes every transaction take the write lock at BEGIN. Without it // immediate makes every transaction take the write lock at BEGIN. Without it
@@ -229,7 +244,7 @@ func New(cfg *config.Config, logger *logger.Logger) (*Database, error) {
counts: &liveCounts{}, counts: &liveCounts{},
} }
err = database.Initialize() err = database.Initialize(walBytes)
if err != nil { if err != nil {
return nil, fmt.Errorf("failed to initialize database: %w", err) return nil, fmt.Errorf("failed to initialize database: %w", err)
} }
@@ -245,8 +260,16 @@ func New(cfg *config.Config, logger *logger.Logger) (*Database, error) {
return database, nil return database, nil
} }
// Initialize creates the database schema if it doesn't exist. // The messages Initialize logs the start-up WAL checkpoint with.
func (d *Database) Initialize() error { const (
startupCheckpointMessage = "WAL checkpoint on startup"
startupCheckpointIncompleteMessage = "WAL checkpoint on startup incomplete"
)
// Initialize creates the database schema if it doesn't exist. walBytes is the
// size of the WAL file the previous run left, which New reads before it opens
// the database.
func (d *Database) Initialize(walBytes int64) error {
// Set SQLite pragmas for performance. Per-connection settings (cache_size, // Set SQLite pragmas for performance. Per-connection settings (cache_size,
// synchronous, busy_timeout, journal_mode) live in the DSN, and New sets the // synchronous, busy_timeout, journal_mode) live in the DSN, and New sets the
// writing connection's cache_size; temp_store is left at its default so // writing connection's cache_size; temp_store is left at its default so
@@ -266,19 +289,33 @@ func (d *Database) Initialize() error {
} }
} }
// Run WAL checkpoint on startup to consolidate any existing WAL // The start-up checkpoint copies the WAL a previous run left into the
var walPages, checkpointedPages, movedPages int // database file and empties the WAL file; its frame counts then read 0,
// so walBytes is logged with them.
var busy, walFrames, checkpointedFrames int
err := d.writer.QueryRowContext(context.Background(), err := d.writer.QueryRowContext(context.Background(),
"PRAGMA wal_checkpoint(TRUNCATE)", "PRAGMA wal_checkpoint(TRUNCATE)",
).Scan(&walPages, &checkpointedPages, &movedPages) ).Scan(&busy, &walFrames, &checkpointedFrames)
if err != nil {
switch {
case err != nil:
d.logger.Warn("Failed to checkpoint WAL on startup", "error", err) d.logger.Warn("Failed to checkpoint WAL on startup", "error", err)
} else { case busy != 0:
d.logger.Info("WAL checkpoint on startup", // Another process using the database kept the checkpoint from
"wal_pages", walPages, // copying or emptying all of the WAL.
"checkpointed", checkpointedPages, d.logger.Warn(startupCheckpointIncompleteMessage,
"moved", movedPages, "wal_bytes", walBytes,
"busy", busy,
"wal_frames", walFrames,
"checkpointed_frames", checkpointedFrames,
)
default:
d.logger.Info(startupCheckpointMessage,
"wal_bytes", walBytes,
"busy", busy,
"wal_frames", walFrames,
"checkpointed_frames", checkpointedFrames,
) )
} }
@@ -855,7 +892,7 @@ func insertPrefixesInTable(
// UpdatePrefixesBatch stores, in a single transaction, the prefixes that are // UpdatePrefixesBatch stores, in a single transaction, the prefixes that are
// not stored yet, each first seen at its time in prefixes: the IPv4 prefixes // not stored yet, each first seen at its time in prefixes: the IPv4 prefixes
// first and then the IPv6 prefixes. // first and then the IPv6 prefixes. A prefix that does not parse is not stored.
func (d *Database) UpdatePrefixesBatch(prefixes map[string]time.Time) error { func (d *Database) UpdatePrefixesBatch(prefixes map[string]time.Time) error {
if len(prefixes) == 0 { if len(prefixes) == 0 {
return nil return nil
@@ -874,6 +911,11 @@ func (d *Database) UpdatePrefixesBatch(prefixes map[string]time.Time) error {
prefixesV6 := make(map[string]time.Time) prefixesV6 := make(map[string]time.Time)
for prefix, timestamp := range prefixes { for prefix, timestamp := range prefixes {
_, err := netip.ParsePrefix(prefix)
if err != nil {
continue
}
if detectIPVersion(prefix) == ipVersionV4 { if detectIPVersion(prefix) == ipVersionV4 {
prefixesV4[prefix] = timestamp prefixesV4[prefix] = timestamp
} else { } else {
@@ -903,13 +945,14 @@ func (d *Database) UpdatePrefixesBatch(prefixes map[string]time.Time) error {
return nil return nil
} }
// newASN returns a new ASN record first and last seen at timestamp, with the // newASN returns a new ASN record first seen at seen.First and last seen at
// handle and description from the AS info lookup when it knows the number. // seen.Last, with the handle and description from the AS info lookup when it
func newASN(number int, timestamp time.Time) ASN { // knows the number.
func newASN(number int, seen SeenTimes) ASN {
asn := ASN{ asn := ASN{
ASN: number, ASN: number,
FirstSeen: timestamp, FirstSeen: seen.First,
LastSeen: timestamp, LastSeen: seen.Last,
} }
// Look up ASN info // Look up ASN info
@@ -921,12 +964,12 @@ func newASN(number int, timestamp time.Time) ASN {
return asn return asn
} }
// upsertASN updates the last_seen time of ASN number with the given // upsertASN sets the last_seen time of ASN number to seen.Last with the given
// statements, inserting the ASN when it is new. It reports whether it inserted // statements, inserting the ASN when it is new. It reports whether it inserted
// it. // it.
func upsertASN( func upsertASN(
ctx context.Context, selectStmt, updateStmt, insertStmt *sql.Stmt, ctx context.Context, selectStmt, updateStmt, insertStmt *sql.Stmt,
number int, timestamp time.Time, number int, seen SeenTimes,
) (bool, error) { ) (bool, error) {
var ( var (
asn ASN asn ASN
@@ -937,7 +980,7 @@ func upsertASN(
&description, &asn.FirstSeen, &asn.LastSeen) &description, &asn.FirstSeen, &asn.LastSeen)
if err == nil { if err == nil {
// ASN exists, update last_seen // ASN exists, update last_seen
_, err = updateStmt.ExecContext(ctx, timestamp, number) _, err = updateStmt.ExecContext(ctx, seen.Last, number)
if err != nil { if err != nil {
return false, fmt.Errorf("failed to update ASN %d: %w", number, err) return false, fmt.Errorf("failed to update ASN %d: %w", number, err)
} }
@@ -950,7 +993,7 @@ func upsertASN(
} }
// ASN doesn't exist, create it // ASN doesn't exist, create it
asn = newASN(number, timestamp) asn = newASN(number, seen)
_, err = insertStmt.ExecContext(ctx, asn.ASN, asn.Handle, _, err = insertStmt.ExecContext(ctx, asn.ASN, asn.Handle,
asn.Description, asn.FirstSeen, asn.LastSeen) asn.Description, asn.FirstSeen, asn.LastSeen)
@@ -962,7 +1005,7 @@ func upsertASN(
} }
// GetOrCreateASNBatch creates or updates multiple ASNs in a single transaction // GetOrCreateASNBatch creates or updates multiple ASNs in a single transaction
func (d *Database) GetOrCreateASNBatch(asns map[int]time.Time) error { func (d *Database) GetOrCreateASNBatch(asns map[int]SeenTimes) error {
if len(asns) == 0 { if len(asns) == 0 {
return nil return nil
} }
@@ -1004,9 +1047,9 @@ func (d *Database) GetOrCreateASNBatch(asns map[int]time.Time) error {
var newASNs int var newASNs int
for number, timestamp := range asns { for number, seen := range asns {
inserted, err := upsertASN(ctx, selectStmt, updateStmt, insertStmt, inserted, err := upsertASN(ctx, selectStmt, updateStmt, insertStmt,
number, timestamp) number, seen)
if err != nil { if err != nil {
return err return err
} }
@@ -1075,7 +1118,7 @@ func (d *Database) GetOrCreateASN(number int, timestamp time.Time) (*ASN, error)
} }
// ASN doesn't exist, create it with ASN info lookup // ASN doesn't exist, create it with ASN info lookup
asn = newASN(number, timestamp) asn = newASN(number, SeenTimes{First: timestamp, Last: timestamp})
_, err = tx.Exec("INSERT INTO asns "+ _, err = tx.Exec("INSERT INTO asns "+
"(asn, handle, description, first_seen, last_seen) "+ "(asn, handle, description, first_seen, last_seen) "+
@@ -1182,7 +1225,7 @@ func (d *Database) RecordAnnouncement(announcement *Announcement) error {
} }
// RecordPeering records a peering relationship between two ASNs. // RecordPeering records a peering relationship between two ASNs.
func (d *Database) RecordPeering(asA, asB int, timestamp time.Time) error { func (d *Database) RecordPeering(asA, asB int, seen SeenTimes) error {
// Validate ASNs // Validate ASNs
if asA <= 0 || asB <= 0 { if asA <= 0 || asB <= 0 {
return fmt.Errorf("%w: asA=%d, asB=%d", errInvalidASN, asA, asB) return fmt.Errorf("%w: asA=%d, asB=%d", errInvalidASN, asA, asB)
@@ -1218,12 +1261,12 @@ func (d *Database) RecordPeering(asA, asB int, timestamp time.Time) error {
if exists { if exists {
_, err = tx.Exec( _, err = tx.Exec(
"UPDATE peerings SET last_seen = ? WHERE as_a = ? AND as_b = ?", "UPDATE peerings SET last_seen = ? WHERE as_a = ? AND as_b = ?",
timestamp, asA, asB) seen.Last, asA, asB)
} else { } else {
_, err = tx.Exec(` _, err = tx.Exec(`
INSERT INTO peerings (id, as_a, as_b, first_seen, last_seen) INSERT INTO peerings (id, as_a, as_b, first_seen, last_seen)
VALUES (?, ?, ?, ?, ?)`, VALUES (?, ?, ?, ?, ?)`,
generateUUID().String(), asA, asB, timestamp, timestamp) generateUUID().String(), asA, asB, seen.First, seen.Last)
} }
if err != nil { if err != nil {
@@ -1301,11 +1344,11 @@ func (d *Database) UpdatePeerBatch(peers map[string]PeerUpdate) error {
if exists { if exists {
_, err = updateStmt.ExecContext(ctx, update.PeerASN, _, err = updateStmt.ExecContext(ctx, update.PeerASN,
update.Timestamp, update.MessageType, update.PeerIP) update.Seen.Last, update.MessageType, update.PeerIP)
} else { } else {
_, err = insertStmt.ExecContext(ctx, _, err = insertStmt.ExecContext(ctx,
generateUUID().String(), update.PeerIP, update.PeerASN, generateUUID().String(), update.PeerIP, update.PeerASN,
update.Timestamp, update.Timestamp, update.MessageType) update.Seen.First, update.Seen.Last, update.MessageType)
} }
if err != nil { if err != nil {
@@ -1706,9 +1749,21 @@ func (d *Database) GetASDetails(asn int) (*ASN, []LiveRoute, error) {
// GetASDetailsContext returns the AS and the prefixes it originates, the IPv4 // GetASDetailsContext returns the AS and the prefixes it originates, the IPv4
// ones first, each once with the time of its newest route. It makes two index // ones first, each once with the time of its newest route. It makes two index
// searches per prefix, however many peers carry the prefix. // searches per prefix, however many peers carry the prefix.
//
// The routes of an AS can be stored before its record in the asns table. When
// the AS has no record, its handle and description come from the AS info
// lookup, and it has no seen times. An AS with no record, no lookup entry and
// no prefix is ErrNoRoute.
//
// ristypes.NoOrigin means no known origin, not an AS, so it is ErrNoRoute
// whatever is stored.
func (d *Database) GetASDetailsContext( func (d *Database) GetASDetailsContext(
ctx context.Context, asn int, ctx context.Context, asn int,
) (*ASN, []LiveRoute, error) { ) (*ASN, []LiveRoute, error) {
if asn == ristypes.NoOrigin {
return nil, nil, fmt.Errorf("%w: AS%d", ErrNoRoute, asn)
}
// Get AS information // Get AS information
var ( var (
asnInfo ASN asnInfo ASN
@@ -1721,16 +1776,27 @@ func (d *Database) GetASDetailsContext(
asn, asn,
).Scan(&asnInfo.ASN, &handle, &description, &asnInfo.FirstSeen, ).Scan(&asnInfo.ASN, &handle, &description, &asnInfo.FirstSeen,
&asnInfo.LastSeen) &asnInfo.LastSeen)
if err != nil {
if errors.Is(err, sql.ErrNoRows) { // known is whether the asns table or the AS info lookup holds the AS.
return nil, nil, fmt.Errorf("%w: AS%d", ErrNoRoute, asn) known := true
switch {
case errors.Is(err, sql.ErrNoRows):
asnInfo.ASN = asn
info, ok := asinfo.Get(asn)
if ok {
asnInfo.Handle = info.Handle
asnInfo.Description = info.Description
} }
known = ok
case err != nil:
return nil, nil, fmt.Errorf("failed to query AS: %w", err) return nil, nil, fmt.Errorf("failed to query AS: %w", err)
} default:
asnInfo.Handle = handle.String asnInfo.Handle = handle.String
asnInfo.Description = description.String asnInfo.Description = description.String
}
allPrefixes, err := d.originPrefixes(ctx, asn, ipVersionV4) allPrefixes, err := d.originPrefixes(ctx, asn, ipVersionV4)
if err != nil { if err != nil {
@@ -1742,7 +1808,13 @@ func (d *Database) GetASDetailsContext(
return &asnInfo, allPrefixes, err return &asnInfo, allPrefixes, err
} }
return &asnInfo, append(allPrefixes, prefixesV6...), nil allPrefixes = append(allPrefixes, prefixesV6...)
if !known && len(allPrefixes) == 0 {
return nil, nil, fmt.Errorf("%w: AS%d", ErrNoRoute, asn)
}
return &asnInfo, allPrefixes, nil
} }
// ASPeer represents a peering relationship with another AS including handle, // ASPeer represents a peering relationship with another AS including handle,
+301 -4
View File
@@ -1,8 +1,10 @@
package database package database
import ( import (
"bytes"
"context" "context"
"database/sql" "database/sql"
"encoding/json"
"errors" "errors"
"fmt" "fmt"
"log/slog" "log/slog"
@@ -18,6 +20,7 @@ import (
"git.eeqj.de/sneak/routewatch/internal/config" "git.eeqj.de/sneak/routewatch/internal/config"
"git.eeqj.de/sneak/routewatch/internal/logger" "git.eeqj.de/sneak/routewatch/internal/logger"
"git.eeqj.de/sneak/routewatch/pkg/asinfo"
"github.com/mattn/go-sqlite3" "github.com/mattn/go-sqlite3"
) )
@@ -615,10 +618,12 @@ func TestBatchWriteDuringCheckpoint(t *testing.T) {
}) })
ts := time.Now().UTC() ts := time.Now().UTC()
seen := SeenTimes{First: ts, Last: ts}
for i := range contentionIterations { for i := range contentionIterations {
asns := map[int]time.Time{ asns := map[int]SeenTimes{
i % contendedASNCount: ts, i % contendedASNCount: seen,
(i % contendedASNCount) + asnSecondBand: ts, (i % contendedASNCount) + asnSecondBand: seen,
} }
err := db.GetOrCreateASNBatch(asns) err := db.GetOrCreateASNBatch(asns)
@@ -747,7 +752,8 @@ func TestBatchWriteWaitsForAnalyze(t *testing.T) {
db, messages := newDatabaseWithLockMessages(t) db, messages := newDatabaseWithLockMessages(t)
ctx := context.Background() ctx := context.Background()
asns := map[int]time.Time{lockTestASN: time.Now().UTC()} now := time.Now().UTC()
asns := map[int]SeenTimes{lockTestASN: {First: now, Last: now}}
// A stored AS gives ANALYZE a table to analyze and statistics to write. // A stored AS gives ANALYZE a table to analyze and statistics to write.
err := db.GetOrCreateASNBatch(asns) err := db.GetOrCreateASNBatch(asns)
@@ -1049,6 +1055,176 @@ func walFileSize(t *testing.T, db *Database) int64 {
return wal.Size() return wal.Size()
} }
// TestStartupCheckpointLogsItsFigures opens a new database, writes a route,
// and opens the same file again while the first stays open, so that its WAL
// file stays. The first open finds no WAL file: it logs a size of 0 and no
// warning. The second logs the size of the WAL file the first left, and the
// checkpoint's busy flag and frame counts, each under its own name; the
// counts read 0 once the checkpoint has emptied the file.
// Initialize logged the busy flag as wal_pages and the counts under each
// other's names (https://git.eeqj.de/sneak/routewatch/issues/117).
func TestStartupCheckpointLogsItsFigures(t *testing.T) {
t.Parallel()
cfg := &config.Config{StateDir: t.TempDir()}
db, lines := openLogged(t, cfg)
for _, line := range lines {
if line[slog.LevelKey] == slog.LevelWarn.String() {
t.Errorf("opening a new database logged a warning: %v", line)
}
}
line := findLine(t, lines, startupCheckpointMessage)
if line["wal_bytes"] != 0.0 {
t.Errorf("new database: start-up checkpoint logged wal_bytes = %v, "+
"want 0", line["wal_bytes"])
}
err := db.UpsertLiveRouteBatch([]*LiveRoute{
mkV4Route(t, "192.0.2.0/24", lockTestASN, time.Now().UTC()),
})
if err != nil {
t.Fatalf("failed to store a route: %v", err)
}
walBytes := walFileSize(t, db)
if walBytes == 0 {
t.Fatal("WAL file empty after the route write, want frames in it")
}
_, lines = openLogged(t, cfg)
line = findLine(t, lines, startupCheckpointMessage)
for _, key := range []string{"busy", "wal_frames", "checkpointed_frames"} {
if line[key] != 0.0 {
t.Errorf("start-up checkpoint logged %s = %v, want 0", key, line[key])
}
}
if line["wal_bytes"] != float64(walBytes) {
t.Errorf("start-up checkpoint logged wal_bytes = %v, want the %d "+
"bytes the first database left", line["wal_bytes"], walBytes)
}
}
// TestStartupCheckpointWarnsWhenIncomplete keeps a read open on a database
// while a route is written after the read began, then opens the same file
// again. The second open's start-up checkpoint cannot copy the frames written
// after the read began, and gives up once the busy timeout has passed. It
// checks that the checkpoint is logged as a warning with busy 1 and fewer
// frames copied than the WAL holds.
func TestStartupCheckpointWarnsWhenIncomplete(t *testing.T) {
t.Parallel()
cfg := &config.Config{StateDir: t.TempDir()}
db, _ := openLogged(t, cfg)
ctx := context.Background()
reader, err := db.db.Conn(ctx)
if err != nil {
t.Fatalf("failed to open the reading connection: %v", err)
}
defer func() { _ = reader.Close() }()
// A plain BEGIN starts a transaction that only reads, unlike the BEGIN
// IMMEDIATE of _txlock. It keeps the snapshot its first read takes until
// it ends.
_, err = reader.ExecContext(ctx, "BEGIN")
if err != nil {
t.Fatalf("failed to begin the read transaction: %v", err)
}
defer func() { _, _ = reader.ExecContext(ctx, "ROLLBACK") }()
var routes int
err = reader.QueryRowContext(ctx,
"SELECT count(*) FROM live_routes_v4").Scan(&routes)
if err != nil {
t.Fatalf("failed to read the live routes: %v", err)
}
err = db.UpsertLiveRouteBatch([]*LiveRoute{
mkV4Route(t, "192.0.2.0/24", lockTestASN, time.Now().UTC()),
})
if err != nil {
t.Fatalf("failed to store a route after the read: %v", err)
}
_, lines := openLogged(t, cfg)
line := findLine(t, lines, startupCheckpointIncompleteMessage)
if line[slog.LevelKey] != slog.LevelWarn.String() {
t.Errorf("start-up checkpoint logged at level %v, want %s",
line[slog.LevelKey], slog.LevelWarn)
}
if line["busy"] != 1.0 {
t.Errorf("start-up checkpoint logged busy = %v, want 1", line["busy"])
}
walFrames, _ := line["wal_frames"].(float64)
checkpointedFrames, _ := line["checkpointed_frames"].(float64)
if checkpointedFrames >= walFrames {
t.Errorf("start-up checkpoint logged checkpointed_frames = %v and "+
"wal_frames = %v with a read open, want fewer copied than the WAL "+
"holds", line["checkpointed_frames"], line["wal_frames"])
}
}
// openLogged opens the database of cfg with a logger that writes JSON, and
// returns it with the lines it logged while it opened, decoded.
func openLogged(t *testing.T, cfg *config.Config) (*Database, []map[string]any) {
t.Helper()
var logged bytes.Buffer
db, err := New(cfg,
&logger.Logger{Logger: slog.New(slog.NewJSONHandler(&logged, nil))})
if err != nil {
t.Fatalf("failed to open the database: %v", err)
}
t.Cleanup(func() { _ = db.Close() })
var lines []map[string]any
decoder := json.NewDecoder(&logged)
for decoder.More() {
var line map[string]any
err := decoder.Decode(&line)
if err != nil {
t.Fatalf("failed to decode the log: %v", err)
}
lines = append(lines, line)
}
return db, lines
}
// findLine returns the first of lines logged with message.
func findLine(
t *testing.T,
lines []map[string]any,
message string,
) map[string]any {
t.Helper()
for _, line := range lines {
if line[slog.MessageKey] == message {
return line
}
}
t.Fatalf("no line %q in the log", message)
return nil
}
// TestGetRandomPrefixesByLengthListsPrefixOnce stores one prefix of each // TestGetRandomPrefixesByLengthListsPrefixOnce stores one prefix of each
// family learned from two peers in the same second and checks that the // family learned from two peers in the same second and checks that the
// prefix-length sample lists each of them once. // prefix-length sample lists each of them once.
@@ -1524,6 +1700,85 @@ func TestGetASDetailsListsEachPrefixOnce(t *testing.T) {
} }
} }
// TestGetASDetailsOfASNotInDatabase asks for ASes that the database has no
// record of, some with stored routes. Each gets the prefixes of its routes and
// no seen times. One that the AS info lookup knows gets its handle and
// description from it. An AS with no record, no lookup entry and no route is
// ErrNoRoute. Before the fix every AS with no record was ErrNoRoute.
func TestGetASDetailsOfASNotInDatabase(t *testing.T) {
t.Parallel()
db, err := New(&config.Config{StateDir: t.TempDir()}, logger.New())
if err != nil {
t.Fatalf("failed to create database: %v", err)
}
defer func() { _ = db.Close() }()
ts := time.Date(2026, 1, 2, 3, 4, 5, 0, time.UTC)
tests := []struct {
asn int
listed bool // in the AS info lookup
prefixes []string // of the routes stored for the AS
}{
{15169, true, []string{"192.0.2.0/24"}},
{13335, true, nil},
{4200000000, false, []string{"198.51.100.0/24"}},
}
var routes []*LiveRoute
for _, tt := range tests {
if _, ok := asinfo.Get(tt.asn); ok != tt.listed {
t.Fatalf("AS%d in the AS info data: %t, want %t", tt.asn, ok, tt.listed)
}
for _, prefix := range tt.prefixes {
routes = append(routes, mkV4Route(t, prefix, tt.asn, ts))
}
}
err = db.UpsertLiveRouteBatch(routes)
if err != nil {
t.Fatalf("UpsertLiveRouteBatch: %v", err)
}
for _, tt := range tests {
got, prefixes, err := db.GetASDetails(tt.asn)
if err != nil {
t.Errorf("GetASDetails(%d): %v", tt.asn, err)
continue
}
want := ASN{ASN: tt.asn}
if info, ok := asinfo.Get(tt.asn); ok {
want.Handle, want.Description = info.Handle, info.Description
}
var gotPrefixes []string
for _, route := range prefixes {
gotPrefixes = append(gotPrefixes, route.Prefix)
}
if *got != want || !slices.Equal(gotPrefixes, tt.prefixes) {
t.Errorf("GetASDetails(%d) = %+v with prefixes %v, want %+v with %v",
tt.asn, *got, gotPrefixes, want, tt.prefixes)
}
}
const absentASN = 4200000001 // no record, no lookup entry and no route
if _, ok := asinfo.Get(absentASN); ok {
t.Fatalf("AS%d is in the AS info data", absentASN)
}
_, _, err = db.GetASDetails(absentASN)
if !errors.Is(err, ErrNoRoute) {
t.Errorf("GetASDetails(%d): error %v, want %v", absentASN, err, ErrNoRoute)
}
}
// TestASPrefixQueriesUseCoveringIndexes checks the plan of each query that // TestASPrefixQueriesUseCoveringIndexes checks the plan of each query that
// lists the prefixes an AS originates: a search of a covering index and no // lists the prefixes an AS originates: a search of a covering index and no
// temporary B-tree, so the list reads one index entry per prefix, not one // temporary B-tree, so the list reads one index entry per prefix, not one
@@ -1663,6 +1918,48 @@ func TestUpdatePrefixesBatchKeepsFirstSeen(t *testing.T) {
} }
} }
// TestUpdatePeerBatchStoresTheEarliestAndLatestTimesOfANewPeer writes a new
// peer whose updates span 5 seconds. Its row is first seen at the earliest time
// and last seen at the latest.
func TestUpdatePeerBatchStoresTheEarliestAndLatestTimesOfANewPeer(
t *testing.T,
) {
t.Parallel()
db, err := New(&config.Config{StateDir: t.TempDir()}, logger.New())
if err != nil {
t.Fatalf("failed to create database: %v", err)
}
defer func() { _ = db.Close() }()
first := time.Date(2026, 1, 2, 3, 4, 5, 0, time.UTC)
last := first.Add(5 * time.Second)
err = db.UpdatePeerBatch(map[string]PeerUpdate{
testPeerIP: {
PeerIP: testPeerIP, PeerASN: 64500, MessageType: "UPDATE",
Seen: SeenTimes{First: first, Last: last},
},
})
if err != nil {
t.Fatalf("UpdatePeerBatch: %v", err)
}
var firstSeen, lastSeen time.Time
err = db.db.QueryRowContext(t.Context(),
"SELECT first_seen, last_seen FROM bgp_peers WHERE peer_ip = ?",
testPeerIP).Scan(&firstSeen, &lastSeen)
if err != nil {
t.Fatalf("reading peer %s: %v", testPeerIP, err)
}
if !firstSeen.Equal(first) || !lastSeen.Equal(last) {
t.Errorf("peer %s: first seen %s, last seen %s; want %s, %s",
testPeerIP, firstSeen, lastSeen, first, last)
}
}
// TestUpsertLiveRouteBatchWritesInKeyOrder writes new routes of two peers out // TestUpsertLiveRouteBatchWritesInKeyOrder writes new routes of two peers out
// of key order, the last one a second announcement of the first, and checks // of key order, the last one a second announcement of the first, and checks
// that the rows were inserted in key order, peer then prefix (each new row's id // that the rows were inserted in key order, peer then prefix (each new row's id
@@ -0,0 +1,61 @@
package database
import (
"os"
"path/filepath"
"strings"
"testing"
"git.eeqj.de/sneak/routewatch/internal/config"
"git.eeqj.de/sneak/routewatch/internal/logger"
)
// TestDockerignoreExcludesDatabaseFiles checks that .dockerignore keeps the
// database file, and the WAL and shared-memory files SQLite writes next to it,
// out of the build context wherever they sit in the clone. The image ships the
// whole build context as a source archive.
func TestDockerignoreExcludesDatabaseFiles(t *testing.T) {
t.Parallel()
db, err := New(&config.Config{StateDir: t.TempDir()}, logger.New())
if err != nil {
t.Fatalf("failed to create database: %v", err)
}
defer func() { _ = db.Close() }()
data, err := os.ReadFile("../../.dockerignore")
if err != nil {
t.Fatalf("failed to read .dockerignore: %v", err)
}
// Docker reads `**/` as any number of directories and the rest of the
// pattern as filepath.Match does, so `**/` followed by a pattern that
// matches a file's name excludes that file at any depth.
var anyDepth []string
for line := range strings.Lines(string(data)) {
pattern, ok := strings.CutPrefix(strings.TrimSpace(line), "**/")
if ok {
anyDepth = append(anyDepth, pattern)
}
}
name := filepath.Base(db.path)
for _, file := range []string{name, name + "-wal", name + "-shm"} {
excluded := false
for _, pattern := range anyDepth {
matched, err := filepath.Match(pattern, file)
if err != nil {
t.Fatalf("bad .dockerignore pattern **/%s: %v", pattern, err)
}
excluded = excluded || matched
}
if !excluded {
t.Errorf(".dockerignore has no **/ pattern that excludes %s", file)
}
}
}
+2 -2
View File
@@ -34,7 +34,7 @@ type Stats struct {
type Store interface { type Store interface {
// ASN operations // ASN operations
GetOrCreateASN(number int, timestamp time.Time) (*ASN, error) GetOrCreateASN(number int, timestamp time.Time) (*ASN, error)
GetOrCreateASNBatch(asns map[int]time.Time) error GetOrCreateASNBatch(asns map[int]SeenTimes) error
// Prefix operations // Prefix operations
GetOrCreatePrefix(prefix string, timestamp time.Time) (*Prefix, error) GetOrCreatePrefix(prefix string, timestamp time.Time) (*Prefix, error)
@@ -44,7 +44,7 @@ type Store interface {
RecordAnnouncement(announcement *Announcement) error RecordAnnouncement(announcement *Announcement) error
// Peering operations // Peering operations
RecordPeering(asA, asB int, timestamp time.Time) error RecordPeering(asA, asB int, seen SeenTimes) error
// Statistics // Statistics
GetStats() (Stats, error) GetStats() (Stats, error)
+29 -5
View File
@@ -28,9 +28,10 @@ type ASN struct {
RIRRegDate *time.Time `json:"rir_registration_date,omitempty"` RIRRegDate *time.Time `json:"rir_registration_date,omitempty"`
RIRLastMod *time.Time `json:"rir_last_modified,omitempty"` RIRLastMod *time.Time `json:"rir_last_modified,omitempty"`
WHOISRaw string `json:"whois_raw,omitempty"` WHOISRaw string `json:"whois_raw,omitempty"`
// Timestamps // Timestamps. They are zero, and left out of JSON, for an AS that has no
FirstSeen time.Time `json:"first_seen"` // record in the asns table.
LastSeen time.Time `json:"last_seen"` FirstSeen time.Time `json:"first_seen,omitzero"`
LastSeen time.Time `json:"last_seen,omitzero"`
WHOISUpdatedAt *time.Time `json:"whois_updated_at,omitempty"` WHOISUpdatedAt *time.Time `json:"whois_updated_at,omitempty"`
} }
@@ -122,12 +123,35 @@ type LiveRouteDeletion struct {
IPVersion int IPVersion int
} }
// PeerUpdate represents parameters for updating a peer // SeenTimes is the earliest and the latest time one batch of updates saw an AS,
// a peer or a peering. A row the batch adds is stored first seen at First and
// last seen at Last; a row already stored keeps its first_seen and has its
// last_seen set to Last.
type SeenTimes struct {
First time.Time
Last time.Time
}
// Add makes t the earliest time of s when it is earlier than First, and the
// latest when it is later than Last. The zero SeenTimes holds no time yet, so
// the first Add sets both.
func (s *SeenTimes) Add(t time.Time) {
if s.First.IsZero() || t.Before(s.First) {
s.First = t
}
if s.Last.IsZero() || t.After(s.Last) {
s.Last = t
}
}
// PeerUpdate represents parameters for updating a peer. PeerASN and
// MessageType are those of the peer's latest update.
type PeerUpdate struct { type PeerUpdate struct {
PeerIP string PeerIP string
PeerASN int PeerASN int
MessageType string MessageType string
Timestamp time.Time Seen SeenTimes
} }
// ASNWHOISUpdate contains WHOIS data for updating an ASN record. // ASNWHOISUpdate contains WHOIS data for updating an ASN record.
+33 -11
View File
@@ -4,6 +4,7 @@ package ristypes
import ( import (
"encoding/json" "encoding/json"
"slices"
"time" "time"
) )
@@ -24,7 +25,8 @@ const (
) )
// ASPath is a BGP AS path as the feed writes it: an array of AS numbers in // ASPath is a BGP AS path as the feed writes it: an array of AS numbers in
// which an AS set is an array of its members, such as [1, 2, [3, 4]]. // which an AS set is an array of its members, such as [1, 2, [3, 4]]. A path
// that holds AS 0 decodes as an empty one (UnmarshalJSON).
type ASPath struct { type ASPath struct {
// Sequence is the path's AS numbers in order. A set of one member that // Sequence is the path's AS numbers in order. A set of one member that
// ends the path counts as that AS and is its last element. The members of // ends the path counts as that AS and is its last element. The members of
@@ -37,34 +39,56 @@ type ASPath struct {
Set []int Set []int
} }
// NoOrigin is the origin AS of a route whose origin is not known. AS 0 is
// reserved, so no AS has this number.
const NoOrigin = 0
// Origin returns the AS that originated the route: the last AS of the // Origin returns the AS that originated the route: the last AS of the
// sequence, or 0 (none) when the path ends in a set of several members or is // sequence, or NoOrigin when the path ends in a set of several members or is
// empty. // empty.
func (p *ASPath) Origin() int { func (p *ASPath) Origin() int {
if len(p.Set) > 0 || len(p.Sequence) == 0 { if len(p.Set) > 0 || len(p.Sequence) == 0 {
return 0 return NoOrigin
} }
return p.Sequence[len(p.Sequence)-1] return p.Sequence[len(p.Sequence)-1]
} }
// UnmarshalJSON implements the json.Unmarshaler interface for ASPath. // UnmarshalJSON implements the json.Unmarshaler interface for ASPath. A path
// that holds AS 0 anywhere, a member of an AS set included, decodes as an empty
// path: RFC 7607 makes such a path malformed and its announcements
// withdrawals. On an announcement with an empty path the prefix handler
// deletes the peer's route, and the AS and peering handlers record nothing.
func (p *ASPath) UnmarshalJSON(data []byte) error { func (p *ASPath) UnmarshalJSON(data []byte) error {
path, err := decodeASPath(data)
if err != nil {
return err
}
if slices.Contains(path.Sequence, 0) || slices.Contains(path.Set, 0) {
path = ASPath{}
}
*p = path
return nil
}
// decodeASPath decodes an AS path as the feed writes it.
func decodeASPath(data []byte) (ASPath, error) {
// Most paths have no AS set and decode as a plain array of integers. // Most paths have no AS set and decode as a plain array of integers.
var sequence []int var sequence []int
err := json.Unmarshal(data, &sequence) err := json.Unmarshal(data, &sequence)
if err == nil { if err == nil {
*p = ASPath{Sequence: sequence} return ASPath{Sequence: sequence}, nil
return nil
} }
var raw []any var raw []any
err = json.Unmarshal(data, &raw) err = json.Unmarshal(data, &raw)
if err != nil { if err != nil {
return err return ASPath{}, err
} }
path := ASPath{Sequence: make([]int, 0, len(raw))} path := ASPath{Sequence: make([]int, 0, len(raw))}
@@ -90,9 +114,7 @@ func (p *ASPath) UnmarshalJSON(data []byte) error {
} }
} }
*p = path return path, nil
return nil
} }
// RISLiveMessage represents the outer wrapper message from the RIPE RIS Live stream. // RISLiveMessage represents the outer wrapper message from the RIPE RIS Live stream.
+30
View File
@@ -145,3 +145,33 @@ func TestASPathKeepsATrailingSetApart(t *testing.T) {
}) })
} }
} }
// TestASPathWithASZeroDecodesAsAnEmptyPath decodes AS paths that hold AS 0
// before their end, at their end, or in an AS set. RFC 7607 makes such a path
// malformed, so each decodes as an empty path.
func TestASPathWithASZeroDecodesAsAnEmptyPath(t *testing.T) {
t.Parallel()
for _, path := range []string{
"[64496, 0, 64497]",
"[64496, 0]",
"[64496, [0, 64497], 64498]",
"[64496, [64497, 0]]",
} {
t.Run(path, func(t *testing.T) {
t.Parallel()
var got ristypes.ASPath
err := json.Unmarshal([]byte(path), &got)
if err != nil {
t.Fatalf("unmarshal %s: %v", path, err)
}
if len(got.Sequence) > 0 || len(got.Set) > 0 {
t.Errorf("%s decoded to sequence %v and set %v, want an empty path",
path, got.Sequence, got.Set)
}
})
}
}
+6 -5
View File
@@ -116,17 +116,18 @@ func (rw *RouteWatch) Run(ctx context.Context) error {
return err return err
} }
// Start ASN WHOIS fetcher for background updates. The HTTP handlers read
// the server's fetcher without a lock, so it is set before they run.
rw.asnFetcher = NewASNFetcher(rw.db, rw.logger.Logger)
rw.asnFetcher.Start(ctx)
rw.server.SetASNFetcher(rw.asnFetcher)
// Start HTTP server // Start HTTP server
err = rw.server.Start(ctx) err = rw.server.Start(ctx)
if err != nil { if err != nil {
return err return err
} }
// Start ASN WHOIS fetcher for background updates
rw.asnFetcher = NewASNFetcher(rw.db, rw.logger.Logger)
rw.asnFetcher.Start(ctx)
rw.server.SetASNFetcher(rw.asnFetcher)
// Start database maintenance goroutine // Start database maintenance goroutine
rw.dbMaintainer = NewDBMaintainer(rw.db, rw.logger.Logger) rw.dbMaintainer = NewDBMaintainer(rw.db, rw.logger.Logger)
rw.dbMaintainer.Start(ctx) rw.dbMaintainer.Start(ctx)
@@ -152,7 +152,7 @@ func (m *mockStore) RecordAnnouncement(_ *database.Announcement) error {
} }
// RecordPeering mock implementation // RecordPeering mock implementation
func (m *mockStore) RecordPeering(asA, asB int, _ time.Time) error { func (m *mockStore) RecordPeering(asA, asB int, _ database.SeenTimes) error {
m.mu.Lock() m.mu.Lock()
defer m.mu.Unlock() defer m.mu.Unlock()
@@ -396,16 +396,16 @@ func (m *mockStore) DeleteLiveRoutesOfPeer(_ string) (int, error) {
} }
// GetOrCreateASNBatch mock implementation // GetOrCreateASNBatch mock implementation
func (m *mockStore) GetOrCreateASNBatch(asns map[int]time.Time) error { func (m *mockStore) GetOrCreateASNBatch(asns map[int]database.SeenTimes) error {
m.mu.Lock() m.mu.Lock()
defer m.mu.Unlock() defer m.mu.Unlock()
for number, timestamp := range asns { for number, seen := range asns {
if _, exists := m.ASNs[number]; !exists { if _, exists := m.ASNs[number]; !exists {
m.ASNs[number] = &database.ASN{ m.ASNs[number] = &database.ASN{
ASN: number, ASN: number,
FirstSeen: timestamp, FirstSeen: seen.First,
LastSeen: timestamp, LastSeen: seen.Last,
} }
m.ASNCount++ m.ASNCount++
} }
@@ -766,7 +766,9 @@ const (
callClose = "close" callClose = "close"
) )
func (s *closeOrderStore) GetOrCreateASNBatch(asns map[int]time.Time) error { func (s *closeOrderStore) GetOrCreateASNBatch(
asns map[int]database.SeenTimes,
) error {
s.record(callWrite) s.record(callWrite)
return s.Store.GetOrCreateASNBatch(asns) return s.Store.GetOrCreateASNBatch(asns)
+9 -7
View File
@@ -229,19 +229,21 @@ func (h *ASHandler) flushBatch() {
return return
} }
// Process ASNs first (deduped) // The earliest and the latest time the batch saw each AS. The batch holds
asnMap := make(map[int]time.Time) // several updates for an AS, from many peers, so a new AS is stored first
// seen at the earliest of them and last seen at the latest.
asns := make(map[int]database.SeenTimes)
for _, op := range batch { for _, op := range batch {
if existing, ok := asnMap[op.number]; !ok || op.timestamp.After(existing) { seen := asns[op.number]
asnMap[op.number] = op.timestamp seen.Add(op.timestamp)
} asns[op.number] = seen
} }
// Process all ASNs in a single batch transaction // Process all ASNs in a single batch transaction
err := h.db.GetOrCreateASNBatch(asnMap) err := h.db.GetOrCreateASNBatch(asns)
if err != nil { if err != nil {
h.logger.Error("Failed to process ASN batch", h.logger.Error("Failed to process ASN batch",
"error", err, "count", len(asnMap)) "error", err, "count", len(asns))
} }
h.mu.Lock() h.mu.Lock()
@@ -0,0 +1,68 @@
package routewatch
import (
"maps"
"slices"
"testing"
"git.eeqj.de/sneak/routewatch/internal/logger"
)
// TestASHandlerStoresTheEarliestAndLatestTimesOfANewAS handles two updates
// whose path carries an AS the database does not hold yet, the second 5
// seconds after the first, in one batch into a real database. The AS is stored
// first seen at the time of the first update and last seen at the time of the
// second.
func TestASHandlerStoresTheEarliestAndLatestTimesOfANewAS(t *testing.T) {
t.Parallel()
db := newTestDatabase(t)
// Built without NewASHandler's flush goroutine, so the test flushes itself.
h := &ASHandler{db: db, logger: logger.New()}
first := peerAnnouncement(testPeerIP, testV4Prefix)
second := peerAnnouncement(testPeerIP, testV4Prefix)
second.ParsedTimestamp = first.ParsedTimestamp.Add(secondUpdateAfter)
h.HandleMessage(first)
h.HandleMessage(second)
h.flushBatch()
asn, _, err := db.GetASDetails(testASNA)
if err != nil {
t.Fatalf("GetASDetails(%d): %v", testASNA, err)
}
if !asn.FirstSeen.Equal(first.ParsedTimestamp) {
t.Errorf("first_seen of AS%d = %s, want %s", testASNA,
asn.FirstSeen, first.ParsedTimestamp)
}
if !asn.LastSeen.Equal(second.ParsedTimestamp) {
t.Errorf("last_seen of AS%d = %s, want %s", testASNA,
asn.LastSeen, second.ParsedTimestamp)
}
}
// TestASHandlerRecordsNothingFromAPathWithASZero handles an update whose AS
// path holds AS 0 between two other ASes. RFC 7607 makes such a path
// malformed, so neither AS is recorded.
func TestASHandlerRecordsNothingFromAPathWithASZero(t *testing.T) {
t.Parallel()
store := newMockStore()
// Built without NewASHandler's flush goroutine, so the test flushes itself.
h := &ASHandler{db: store, logger: logger.New()}
msg := peerAnnouncement(testPeerIP, testV4Prefix)
msg.Path = decodePath(t, testASNA, 0, testASNB)
h.HandleMessage(msg)
h.flushBatch()
if len(store.ASNs) > 0 {
t.Errorf("ASes recorded = %v, want none",
slices.Sorted(maps.Keys(store.ASNs)))
}
}
+18 -13
View File
@@ -53,7 +53,8 @@ type ASNFetcher struct {
currentInterval time.Duration currentInterval time.Duration
consecutiveFails int consecutiveFails int
// hourly stats tracking // hourly stats tracking; now is the clock, time.Now except in tests
now func() time.Time
statsMu sync.Mutex statsMu sync.Mutex
successTimes []time.Time successTimes []time.Time
errorTimes []time.Time errorTimes []time.Time
@@ -69,6 +70,7 @@ func NewASNFetcher(db database.Store, logger *slog.Logger) *ASNFetcher {
stopCh: make(chan struct{}), stopCh: make(chan struct{}),
done: make(chan struct{}), done: make(chan struct{}),
currentInterval: baseInterval, currentInterval: baseInterval,
now: time.Now,
successTimes: make([]time.Time, 0), successTimes: make([]time.Time, 0),
errorTimes: make([]time.Time, 0), errorTimes: make([]time.Time, 0),
} }
@@ -121,7 +123,7 @@ func (f *ASNFetcher) GetStats() server.ASNFetcherStats {
f.intervalMu.Unlock() f.intervalMu.Unlock()
// Prune old entries and count // Prune old entries and count
cutoff := time.Now().Add(-statsWindow) cutoff := f.now().Add(-statsWindow)
f.successTimes = pruneOldTimes(f.successTimes, cutoff) f.successTimes = pruneOldTimes(f.successTimes, cutoff)
f.errorTimes = pruneOldTimes(f.errorTimes, cutoff) f.errorTimes = pruneOldTimes(f.errorTimes, cutoff)
@@ -133,16 +135,15 @@ func (f *ASNFetcher) GetStats() server.ASNFetcherStats {
} }
} }
// pruneOldTimes removes times older than cutoff and returns the pruned slice. // pruneOldTimes drops the times at or before cutoff from the front of times,
// which holds times in the order they were recorded, and returns the rest. It
// copies nothing, so it is cheap enough to run on every append.
func pruneOldTimes(times []time.Time, cutoff time.Time) []time.Time { func pruneOldTimes(times []time.Time, cutoff time.Time) []time.Time {
result := make([]time.Time, 0, len(times)) for len(times) > 0 && !times[0].After(cutoff) {
for _, t := range times { times = times[1:]
if t.After(cutoff) {
result = append(result, t)
}
} }
return result return times
} }
// getInterval returns the current fetch interval. // getInterval returns the current fetch interval.
@@ -170,9 +171,11 @@ func (f *ASNFetcher) recordSuccess() {
} }
f.intervalMu.Unlock() f.intervalMu.Unlock()
// Record success time for stats // Record success time for stats, dropping those older than statsWindow
f.statsMu.Lock() f.statsMu.Lock()
f.successTimes = append(f.successTimes, time.Now()) now := f.now()
f.successTimes = pruneOldTimes(f.successTimes, now.Add(-statsWindow))
f.successTimes = append(f.successTimes, now)
f.statsMu.Unlock() f.statsMu.Unlock()
} }
@@ -194,9 +197,11 @@ func (f *ASNFetcher) recordFailure() {
} }
f.intervalMu.Unlock() f.intervalMu.Unlock()
// Record error time for stats // Record error time for stats, dropping those older than statsWindow
f.statsMu.Lock() f.statsMu.Lock()
f.errorTimes = append(f.errorTimes, time.Now()) now := f.now()
f.errorTimes = pruneOldTimes(f.errorTimes, now.Add(-statsWindow))
f.errorTimes = append(f.errorTimes, now)
f.statsMu.Unlock() f.statsMu.Unlock()
} }
@@ -203,3 +203,35 @@ func TestASNFetcherStopReturnsByItsDeadline(t *testing.T) {
t.Fatal("the fetcher goroutine did not return after the lookup ended") t.Fatal("the fetcher goroutine did not return after the lookup ended")
} }
} }
// TestRecordKeepsOnlyTheStatsWindow records a success and a failure every
// second, by a clock the test sets, far longer than statsWindow, and never
// asks for the stats. The fetcher must keep only the times within the last
// statsWindow.
func TestRecordKeepsOnlyTheStatsWindow(t *testing.T) {
t.Parallel()
const (
records = 100_000
step = time.Second
want = int(statsWindow / step)
)
clock := time.Now()
f := NewASNFetcher(newMockStore(), logger.New().Logger)
f.now = func() time.Time { return clock }
for range records {
clock = clock.Add(step)
f.recordSuccess()
f.recordFailure()
}
if len(f.successTimes) != want || len(f.errorTimes) != want {
t.Errorf("%d success and %d error times kept, want %d of each, "+
"those of the last %s", len(f.successTimes), len(f.errorTimes),
want, statsWindow)
}
}
@@ -66,7 +66,7 @@ func (s *blockingStore) UpsertLiveRouteBatch(routes []*database.LiveRoute) error
return nil return nil
} }
func (s *blockingStore) GetOrCreateASNBatch(asns map[int]time.Time) error { func (s *blockingStore) GetOrCreateASNBatch(asns map[int]database.SeenTimes) error {
s.waitForRelease() s.waitForRelease()
s.mu.Lock() s.mu.Lock()
defer s.mu.Unlock() defer s.mu.Unlock()
@@ -96,10 +96,12 @@ func (s *blockingStore) DeleteLiveRoutesOfPeer(peerIP string) (int, error) {
return s.mockStore.DeleteLiveRoutesOfPeer(peerIP) return s.mockStore.DeleteLiveRoutesOfPeer(peerIP)
} }
func (s *blockingStore) RecordPeering(asA, asB int, ts time.Time) error { func (s *blockingStore) RecordPeering(
asA, asB int, seen database.SeenTimes,
) error {
s.waitForRelease() s.waitForRelease()
return s.mockStore.RecordPeering(asA, asB, ts) return s.mockStore.RecordPeering(asA, asB, seen)
} }
func (s *blockingStore) waitForRelease() { func (s *blockingStore) waitForRelease() {
+4 -20
View File
@@ -3,10 +3,8 @@ package routewatch
import ( import (
"context" "context"
"os" "os"
"os/signal"
"runtime" "runtime"
"strings" "strings"
"syscall"
"time" "time"
"git.eeqj.de/sneak/routewatch/internal/logger" "git.eeqj.de/sneak/routewatch/internal/logger"
@@ -62,10 +60,10 @@ func logDebugStats(logger *logger.Logger) {
} }
// CLIEntry is the main entry point for the routewatch command-line interface. // CLIEntry is the main entry point for the routewatch command-line interface.
// It initializes the application using the fx dependency injection framework, // It initializes the application using the fx dependency injection framework
// sets up signal handling for graceful shutdown, and starts the RouteWatch // and starts the RouteWatch service. It blocks until fx receives SIGINT or
// service. This function blocks until the application receives a shutdown // SIGTERM or RouteWatch stops on an error; fx then runs the stop hook and sets
// signal or encounters a fatal error. // the exit code.
func CLIEntry() { func CLIEntry() {
app := fx.New( app := fx.New(
fx.Options(getModule()...), fx.Options(getModule()...),
@@ -81,20 +79,6 @@ func CLIEntry() {
// Start debug stats logging // Start debug stats logging
go logDebugStats(logger) go logDebugStats(logger)
// Handle shutdown signals
sigCh := make(chan os.Signal, 1)
signal.Notify(sigCh, os.Interrupt, syscall.SIGTERM)
go func() {
<-sigCh
logger.Info("Received shutdown signal")
err := shutdowner.Shutdown()
if err != nil {
logger.Error("Failed to shutdown gracefully", "error", err)
}
}()
go func() { go func() {
err := rw.Run(ctx) err := rw.Run(ctx)
if err != nil { if err != nil {
+16 -19
View File
@@ -37,8 +37,8 @@ const (
// PeerHandler tracks BGP peers from UPDATE messages using batched operations. // PeerHandler tracks BGP peers from UPDATE messages using batched operations.
// It maintains a queue of peer updates and periodically flushes them to the database // It maintains a queue of peer updates and periodically flushes them to the database
// in batches to improve performance. The handler deduplicates peer updates within // in batches to improve performance. The handler writes one row per peer IP
// each batch, keeping only the most recent update for each peer IP address. // address for each batch, with the earliest and the latest time of its updates.
type PeerHandler struct { type PeerHandler struct {
db database.Store db database.Store
logger *logger.Logger logger *logger.Logger
@@ -227,31 +227,28 @@ func (h *PeerHandler) flushBatch() {
return return
} }
// Deduplicate by peer IP, keeping the latest update for each peer // The earliest and the latest time the batch saw each peer, with the peer
peerMap := make(map[string]peerUpdate) // AS and message type of its latest update. The batch holds several
// updates for a peer, so a new peer is stored first seen at the earliest of
// them and last seen at the latest.
peers := make(map[string]database.PeerUpdate)
for _, update := range batch { for _, update := range batch {
existing, ok := peerMap[update.peerIP] peer, ok := peers[update.peerIP]
if !ok || update.timestamp.After(existing.timestamp) { if !ok || update.timestamp.After(peer.Seen.Last) {
peerMap[update.peerIP] = update peer.PeerIP = update.peerIP
} peer.PeerASN = update.peerASN
peer.MessageType = update.messageType
} }
// Convert to database format peer.Seen.Add(update.timestamp)
dbPeerMap := make(map[string]database.PeerUpdate) peers[update.peerIP] = peer
for peerIP, update := range peerMap {
dbPeerMap[peerIP] = database.PeerUpdate{
PeerIP: update.peerIP,
PeerASN: update.peerASN,
MessageType: update.messageType,
Timestamp: update.timestamp,
}
} }
// Process all peers in a single batch transaction // Process all peers in a single batch transaction
err := h.db.UpdatePeerBatch(dbPeerMap) err := h.db.UpdatePeerBatch(peers)
if err != nil { if err != nil {
h.logger.Error("Failed to process peer batch", h.logger.Error("Failed to process peer batch",
"error", err, "count", len(dbPeerMap)) "error", err, "count", len(peers))
} }
h.mu.Lock() h.mu.Lock()
@@ -0,0 +1,59 @@
package routewatch
import (
"testing"
"git.eeqj.de/sneak/routewatch/internal/database"
"git.eeqj.de/sneak/routewatch/internal/logger"
)
// peerBatchStore keeps the peers of the last UpdatePeerBatch call. The
// database has no read of the peer table, so the peer handler's test checks
// what the handler writes.
type peerBatchStore struct {
*mockStore
peers map[string]database.PeerUpdate
}
func (s *peerBatchStore) UpdatePeerBatch(
peers map[string]database.PeerUpdate,
) error {
s.peers = peers
return nil
}
// TestPeerHandlerWritesTheEarliestAndLatestTimesOfANewPeer handles two updates
// from one peer, the second 5 seconds after the first, in one batch. The peer
// handler writes the peer first seen at the time of the first update and last
// seen at the time of the second.
func TestPeerHandlerWritesTheEarliestAndLatestTimesOfANewPeer(t *testing.T) {
t.Parallel()
store := &peerBatchStore{mockStore: newMockStore()}
// Built without NewPeerHandler's flush goroutine, so the test flushes
// itself.
h := &PeerHandler{db: store, logger: logger.New()}
first := peerAnnouncement(testPeerIP, testV4Prefix)
second := peerAnnouncement(testPeerIP, testV4Prefix)
second.ParsedTimestamp = first.ParsedTimestamp.Add(secondUpdateAfter)
h.HandleMessage(first)
h.HandleMessage(second)
h.flushBatch()
seen := store.peers[testPeerIP].Seen
if !seen.First.Equal(first.ParsedTimestamp) {
t.Errorf("first_seen of peer %s = %s, want %s", testPeerIP,
seen.First, first.ParsedTimestamp)
}
if !seen.Last.Equal(second.ParsedTimestamp) {
t.Errorf("last_seen of peer %s = %s, want %s", testPeerIP,
seen.Last, second.ParsedTimestamp)
}
}
+31 -21
View File
@@ -42,10 +42,11 @@ type PeeringHandler struct {
db database.Store db database.Store
logger *logger.Logger logger *logger.Logger
// In-memory AS path tracking // In-memory AS path tracking: each path, JSON-encoded, with the earliest
// and the latest time an update carried it.
mu sync.Mutex mu sync.Mutex
asPaths map[string]time.Time // key is JSON-encoded AS path asPaths map[string]database.SeenTimes
droppedPaths int // paths dropped because the map was full droppedPaths int // paths dropped since the last run because the map was full
// writing is how many paths processPeerings took from asPaths and has // writing is how many paths processPeerings took from asPaths and has
// not finished writing the peerings of. // not finished writing the peerings of.
writing int writing int
@@ -64,7 +65,7 @@ func NewPeeringHandler(db database.Store, logger *logger.Logger) *PeeringHandler
h := &PeeringHandler{ h := &PeeringHandler{
db: db, db: db,
logger: logger, logger: logger,
asPaths: make(map[string]time.Time), asPaths: make(map[string]database.SeenTimes),
stopCh: make(chan context.Context, 1), stopCh: make(chan context.Context, 1),
done: make(chan struct{}), done: make(chan struct{}),
} }
@@ -118,16 +119,18 @@ func (h *PeeringHandler) HandleMessage(msg *ristypes.RISMessage) {
key := string(pathJSON) key := string(pathJSON)
h.mu.Lock() h.mu.Lock()
if _, exists := h.asPaths[key]; exists { defer h.mu.Unlock()
// Already tracked: refresh its timestamp.
h.asPaths[key] = timestamp seen, exists := h.asPaths[key]
} else if len(h.asPaths) >= maxTrackedPaths { if !exists && len(h.asPaths) >= maxTrackedPaths {
// Map is full; drop this new path and count it. // Map is full; drop this new path and count it.
h.droppedPaths++ h.droppedPaths++
} else {
h.asPaths[key] = timestamp return
} }
h.mu.Unlock()
seen.Add(timestamp)
h.asPaths[key] = seen
} }
// ProcessPeeringsNow triggers immediate processing of all accumulated AS // ProcessPeeringsNow triggers immediate processing of all accumulated AS
@@ -187,12 +190,14 @@ func (h *PeeringHandler) processLoop() {
func (h *PeeringHandler) processPeerings() { func (h *PeeringHandler) processPeerings() {
// Take the accumulated paths and replace the map with a fresh empty one // Take the accumulated paths and replace the map with a fresh empty one
// under the lock. Each path is processed exactly once and the memory is // under the lock. Each path is processed exactly once and the memory is
// released, so the map never grows past a single interval's traffic. // released, so the map never grows past a single interval's traffic. The
// count of dropped paths starts again with the new map.
h.mu.Lock() h.mu.Lock()
paths := h.asPaths paths := h.asPaths
h.asPaths = make(map[string]time.Time) h.asPaths = make(map[string]database.SeenTimes)
h.writing = len(paths) h.writing = len(paths)
dropped := h.droppedPaths dropped := h.droppedPaths
h.droppedPaths = 0
h.mu.Unlock() h.mu.Unlock()
if len(paths) == 0 { if len(paths) == 0 {
@@ -204,9 +209,13 @@ func (h *PeeringHandler) processPeerings() {
low, high int low, high int
} }
peerings := make(map[peeringKey]time.Time) // The earliest and the latest time an update carried each peering. Several
// paths, each carried at several times, can hold a peering, so a new
// peering is stored first seen at the earliest of them and last seen at the
// latest.
peerings := make(map[peeringKey]database.SeenTimes)
for pathJSON, timestamp := range paths { for pathJSON, pathSeen := range paths {
var path []int var path []int
err := json.Unmarshal([]byte(pathJSON), &path) err := json.Unmarshal([]byte(pathJSON), &path)
@@ -233,10 +242,11 @@ func (h *PeeringHandler) processPeerings() {
} }
key := peeringKey{low: low, high: high} key := peeringKey{low: low, high: high}
// Update timestamp if this is newer
if existing, ok := peerings[key]; !ok || timestamp.After(existing) { seen := peerings[key]
peerings[key] = timestamp seen.Add(pathSeen.First)
} seen.Add(pathSeen.Last)
peerings[key] = seen
} }
} }
@@ -244,8 +254,8 @@ func (h *PeeringHandler) processPeerings() {
start := time.Now() start := time.Now()
successCount := 0 successCount := 0
for key, ts := range peerings { for key, seen := range peerings {
err := h.db.RecordPeering(key.low, key.high, ts) err := h.db.RecordPeering(key.low, key.high, seen)
if err != nil { if err != nil {
h.logger.Error("Failed to record peering", h.logger.Error("Failed to record peering",
"as_a", key.low, "as_a", key.low,
@@ -31,12 +31,14 @@ type recordingStore struct {
calls int calls int
} }
func (r *recordingStore) RecordPeering(asA, asB int, ts time.Time) error { func (r *recordingStore) RecordPeering(
asA, asB int, seen database.SeenTimes,
) error {
r.mu.Lock() r.mu.Lock()
r.calls++ r.calls++
r.mu.Unlock() r.mu.Unlock()
return r.mockStore.RecordPeering(asA, asB, ts) return r.mockStore.RecordPeering(asA, asB, seen)
} }
func (r *recordingStore) callCount() int { func (r *recordingStore) callCount() int {
@@ -52,7 +54,7 @@ func newTestHandler(db database.Store) *PeeringHandler {
return &PeeringHandler{ return &PeeringHandler{
db: db, db: db,
logger: logger.New(), logger: logger.New(),
asPaths: make(map[string]time.Time), asPaths: make(map[string]database.SeenTimes),
} }
} }
@@ -151,6 +153,43 @@ func TestPeeringHandlerLeavesATrailingASSetOutOfThePeerings(t *testing.T) {
} }
} }
// TestPeeringHandlerRecordsNothingFromAPathWithASZero handles AS paths that
// hold AS 0. RFC 7607 makes such a path malformed, so no peering is recorded
// from it, not even one between two other ASes of the path.
func TestPeeringHandlerRecordsNothingFromAPathWithASZero(t *testing.T) {
t.Parallel()
tests := []struct {
name string
path []any
}{
{name: "AS 0 between two ASes", path: []any{testASNA, 0, testASNB}},
{
name: "AS 0 after two ASes",
path: []any{testASNA, testASNB, 0, testASNC},
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
t.Parallel()
store := newMockStore()
h := newTestHandler(store)
h.HandleMessage(&ristypes.RISMessage{
Path: decodePath(t, tt.path...),
ParsedTimestamp: time.Now().UTC(),
})
h.ProcessPeeringsNow()
if len(store.Peerings) > 0 {
t.Errorf("peerings recorded = %v, want none",
slices.Sorted(maps.Keys(store.Peerings)))
}
})
}
}
// TestPeeringHandlerProcessesEachRunAndEmpties verifies that a processing run // TestPeeringHandlerProcessesEachRunAndEmpties verifies that a processing run
// empties the path map (the swap) and that a path seen again after a run is // empties the path map (the swap) and that a path seen again after a run is
// recorded in the next run too. // recorded in the next run too.
@@ -220,10 +259,10 @@ func TestPeeringHandlerCapDropsAndCounts(t *testing.T) {
existing := pathKey(t, testASNA, testASNB) existing := pathKey(t, testASNA, testASNB)
h.mu.Lock() h.mu.Lock()
h.asPaths[existing] = now h.asPaths[existing] = database.SeenTimes{First: now, Last: now}
for i := 0; len(h.asPaths) < maxTrackedPaths; i++ { for i := 0; len(h.asPaths) < maxTrackedPaths; i++ {
h.asPaths[strconv.Itoa(i)] = now h.asPaths[strconv.Itoa(i)] = database.SeenTimes{First: now, Last: now}
} }
h.mu.Unlock() h.mu.Unlock()
@@ -253,11 +292,83 @@ func TestPeeringHandlerCapDropsAndCounts(t *testing.T) {
} }
h.mu.Lock() h.mu.Lock()
gotTS := h.asPaths[existing] gotSeen := h.asPaths[existing]
h.mu.Unlock() h.mu.Unlock()
if !gotTS.Equal(refreshed) { if !gotSeen.Last.Equal(refreshed) {
t.Fatalf("existing path timestamp not refreshed: want %v, got %v", t.Fatalf("existing path last seen not refreshed: want %v, got %v",
refreshed, gotTS) refreshed, gotSeen.Last)
}
}
// TestPeeringHandlerRunResetsTheDroppedCount fills the map, has the handler
// drop one path, and runs the processing. The count of dropped paths is the
// run's own, so it is 0 again after the run.
func TestPeeringHandlerRunResetsTheDroppedCount(t *testing.T) {
t.Parallel()
h := newTestHandler(newMockStore())
now := time.Now().UTC()
// Paths of one AS carry no peering, so the run records nothing.
h.mu.Lock()
for asn := 1; len(h.asPaths) < maxTrackedPaths; asn++ {
h.asPaths[pathKey(t, asn)] = database.SeenTimes{First: now, Last: now}
}
h.mu.Unlock()
handle(h, now, testASNA, testASNB)
if _, dropped := snapshot(h); dropped != 1 {
t.Fatalf("want dropped count 1 with the map full, got %d", dropped)
}
h.ProcessPeeringsNow()
if _, dropped := snapshot(h); dropped != 0 {
t.Errorf("want dropped count 0 after a run, got %d", dropped)
}
}
// TestPeeringHandlerStoresTheEarliestAndLatestTimesOfANewPeering handles two
// updates whose path carries a peering the database does not hold yet, the
// second 5 seconds after the first, in one run into a real database. The
// peering is stored first seen at the time of the first update and last seen
// at the time of the second.
func TestPeeringHandlerStoresTheEarliestAndLatestTimesOfANewPeering(
t *testing.T,
) {
t.Parallel()
db := newTestDatabase(t)
h := newTestHandler(db)
first := peerAnnouncement(testPeerIP, testV4Prefix)
second := peerAnnouncement(testPeerIP, testV4Prefix)
second.ParsedTimestamp = first.ParsedTimestamp.Add(secondUpdateAfter)
h.HandleMessage(first)
h.HandleMessage(second)
h.ProcessPeeringsNow()
peers, err := db.GetASPeers(testASNA)
if err != nil {
t.Fatalf("GetASPeers(%d): %v", testASNA, err)
}
if len(peers) != 1 || peers[0].ASN != testASNB {
t.Fatalf("peers of AS%d = %+v, want AS%d alone", testASNA, peers,
testASNB)
}
if !peers[0].FirstSeen.Equal(first.ParsedTimestamp) {
t.Errorf("first_seen of peering AS%d-AS%d = %s, want %s", testASNA,
testASNB, peers[0].FirstSeen, first.ParsedTimestamp)
}
if !peers[0].LastSeen.Equal(second.ParsedTimestamp) {
t.Errorf("last_seen of peering AS%d-AS%d = %s, want %s", testASNA,
testASNB, peers[0].LastSeen, second.ParsedTimestamp)
} }
} }
+21 -14
View File
@@ -44,8 +44,9 @@ const (
) )
// PrefixHandler tracks BGP prefixes and maintains a live routing table in the // PrefixHandler tracks BGP prefixes and maintains a live routing table in the
// database. Routes are added on announcement and deleted on withdrawal, and all // database. Routes are added on announcement and deleted on withdrawal or on an
// routes of a peer are deleted when its BGP session goes down. // announcement whose AS path is empty or holds AS 0 anywhere, and all routes of
// a peer are deleted when its BGP session goes down.
type PrefixHandler struct { type PrefixHandler struct {
db database.Store db database.Store
logger *logger.Logger logger *logger.Logger
@@ -347,8 +348,19 @@ func (h *PrefixHandler) flushBatch() {
lastUpdates := make(map[prefixAndPeer]prefixUpdate) lastUpdates := make(map[prefixAndPeer]prefixUpdate)
// Each prefix the batch names, announced or withdrawn, with the earliest
// timestamp among all of its updates, not only each peer's last one: the
// prefix tables store only first_seen, written when the prefix is new and
// never changed after.
prefixesToUpdate := make(map[string]time.Time)
for _, update := range batch { for _, update := range batch {
lastUpdates[prefixAndPeer{update.prefix, update.peer}] = update lastUpdates[prefixAndPeer{update.prefix, update.peer}] = update
seen, exists := prefixesToUpdate[update.prefix]
if !exists || update.timestamp.Before(seen) {
prefixesToUpdate[update.prefix] = update.timestamp
}
} }
// Collect routes to upsert and delete // Collect routes to upsert and delete
@@ -357,19 +369,14 @@ func (h *PrefixHandler) flushBatch() {
routesToDelete []database.LiveRouteDeletion routesToDelete []database.LiveRouteDeletion
) )
// Collect unique prefixes to update
prefixesToUpdate := make(map[string]time.Time)
for _, update := range lastUpdates { for _, update := range lastUpdates {
// Track prefix for both announcements and withdrawals
seen, exists := prefixesToUpdate[update.prefix]
if !exists || update.timestamp.After(seen) {
prefixesToUpdate[update.prefix] = update.timestamp
}
// An announcement is stored when its path gives an origin, or when it // An announcement is stored when its path gives an origin, or when it
// ends in an AS set of several members: that route is stored with // ends in an AS set of several members: that route is stored with
// origin 0 (none). An announcement with an empty path is not stored. // origin 0 (none). Any other update deletes the peer's route for the
// prefix: a withdrawal, or an announcement whose path is empty or
// holds AS 0 anywhere (the decoder empties such a path). Such an
// announcement replaces the peer's route, as any announcement does,
// with none (RFC 7607 treats AS 0 in a path as a withdrawal).
if update.messageType == "announcement" && if update.messageType == "announcement" &&
(update.originASN > 0 || update.endsInSet) { (update.originASN > 0 || update.endsInSet) {
// Create live route for batch upsert // Create live route for batch upsert
@@ -377,11 +384,11 @@ func (h *PrefixHandler) flushBatch() {
if route != nil { if route != nil {
routesToUpsert = append(routesToUpsert, route) routesToUpsert = append(routesToUpsert, route)
} }
} else if update.messageType == "withdrawal" { } else {
// Parse CIDR to get IP version // Parse CIDR to get IP version
_, ipVersion, err := parseCIDR(update.prefix) _, ipVersion, err := parseCIDR(update.prefix)
if err != nil { if err != nil {
h.logger.Error("Failed to parse CIDR for withdrawal", h.logger.Error("Failed to parse CIDR for route deletion",
"prefix", update.prefix, "error", err) "prefix", update.prefix, "error", err)
continue continue
@@ -26,6 +26,10 @@ const (
testV6Prefix = "2001:db8:1::/48" testV6Prefix = "2001:db8:1::/48"
) )
// secondUpdateAfter is how long after the first update the tests of first_seen
// and last_seen send the second, in the same batch.
const secondUpdateAfter = 5 * time.Second
// TestPrefixHandlerStoresPrefixesTheIPLookupFinds runs announcements and // TestPrefixHandlerStoresPrefixesTheIPLookupFinds runs announcements and
// withdrawals through the prefix handler into a real database and looks the // withdrawals through the prefix handler into a real database and looks the
// addresses up. The prefixes are written the way the feed sends them: IPv6 // addresses up. The prefixes are written the way the feed sends them: IPv6
@@ -83,6 +87,49 @@ func TestPrefixHandlerStoresPrefixesTheIPLookupFinds(t *testing.T) {
} }
} }
// TestPrefixHandlerStoresANewPrefixFirstSeenAtItsEarliestUpdate announces a
// prefix the database does not hold yet twice in one batch, the second time 5
// seconds after the first, through the prefix handler into a real database. The
// prefix is stored first seen at the time of the first announcement, whether
// the two come from two peers or from one, and the IP lookup answers with that
// time.
func TestPrefixHandlerStoresANewPrefixFirstSeenAtItsEarliestUpdate(
t *testing.T,
) {
t.Parallel()
const otherPeerIP = "2001:db8:ffff::2"
tests := []struct {
name string
firstPeer, secondPeer string
}{
{name: "two peers", firstPeer: testPeerIP, secondPeer: otherPeerIP},
{name: "one peer", firstPeer: testPeerIP, secondPeer: testPeerIP},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
t.Parallel()
first := peerAnnouncement(tt.firstPeer, testV4Prefix)
second := peerAnnouncement(tt.secondPeer, testV4Prefix)
second.ParsedTimestamp = first.ParsedTimestamp.Add(secondUpdateAfter)
db := handleBatches(t, [][]*ristypes.RISMessage{{first, second}})
info, err := db.GetIPInfo("192.0.2.1")
if err != nil {
t.Fatalf("GetIPInfo(192.0.2.1): %v", err)
}
if !info.FirstSeen.Equal(first.ParsedTimestamp) {
t.Errorf("first_seen of %s = %s, want %s", testV4Prefix,
info.FirstSeen, first.ParsedTimestamp)
}
})
}
}
// TestPrefixHandlerAppliesEachPeersLastUpdateInABatch runs updates for one // TestPrefixHandlerAppliesEachPeersLastUpdateInABatch runs updates for one
// prefix through the prefix handler into a real database. All updates carry the // prefix through the prefix handler into a real database. All updates carry the
// same timestamp, as updates within one second of the feed do. Within a batch, // same timestamp, as updates within one second of the feed do. Within a batch,
@@ -331,6 +378,133 @@ func TestPrefixHandlerDoesNotStoreAnAnnouncementWithAnEmptyPath(t *testing.T) {
assertPeers(t, db, map[string][]string{testV4Prefix: nil}) assertPeers(t, db, map[string][]string{testV4Prefix: nil})
} }
// TestPrefixHandlerRemovesTheRouteOnAnAnnouncementWithoutAnOrigin runs, through
// the prefix handler into a real database, a peer's announcement of a prefix
// with origin AS testASNB and then the peer's announcement of the prefix with
// an AS path that gives no origin: one whose last AS is 0, an empty one, or one
// that holds AS 0 before its end, which decodes as an empty path. As any later
// announcement from the peer does, the second replaces the peer's route, here
// with none, whether it comes in a later batch or follows a withdrawal in one
// batch. The in-memory route count and prefix distribution must match the
// route table.
func TestPrefixHandlerRemovesTheRouteOnAnAnnouncementWithoutAnOrigin(
t *testing.T,
) {
t.Parallel()
const prefixMaskLength = 24
noOrigin := func(path ristypes.ASPath) *ristypes.RISMessage {
msg := peerAnnouncement(testPeerIP, testV4Prefix)
msg.Path = path
return msg
}
endsInZero := ristypes.ASPath{Sequence: []int{testASNA, 0}}
withdrawal := &ristypes.RISMessage{
Type: ristypes.MessageTypeUpdate,
Peer: testPeerIP,
ParsedTimestamp: time.Date(2026, 1, 2, 3, 4, 5, 0, time.UTC),
Withdrawals: []string{testV4Prefix},
}
tests := []struct {
name string
// Handled and flushed as one batch after the announcement with origin
// AS testASNB.
batch []*ristypes.RISMessage
}{
{
name: "a path whose last AS is 0",
batch: []*ristypes.RISMessage{noOrigin(endsInZero)},
},
{
name: "an empty path",
batch: []*ristypes.RISMessage{noOrigin(ristypes.ASPath{})},
},
{
name: "a path with AS 0 before its end",
batch: []*ristypes.RISMessage{
noOrigin(decodePath(t, testASNA, 0, testASNB)),
},
},
{
name: "a withdrawal and then a path whose last AS is 0",
batch: []*ristypes.RISMessage{withdrawal, noOrigin(endsInZero)},
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
t.Parallel()
db := handleBatches(t, [][]*ristypes.RISMessage{
{peerAnnouncement(testPeerIP, testV4Prefix)},
tt.batch,
})
checkLivePeers(t, db, testV4Prefix, prefixMaskLength, nil)
})
}
}
// TestPrefixHandlerDoesNotStoreAPrefixThatDoesNotParse runs an announcement of
// "not-a-prefix" and a withdrawal of "also/bad" through the prefix handler into
// a real database. Neither is stored in the prefix tables: the prefix count in
// the stats stays 0, both as kept in memory and as counted from the tables when
// the database is opened again.
func TestPrefixHandlerDoesNotStoreAPrefixThatDoesNotParse(t *testing.T) {
t.Parallel()
cfg := &config.Config{StateDir: t.TempDir()}
db, err := database.New(cfg, logger.New())
if err != nil {
t.Fatalf("failed to create database: %v", err)
}
// Built without NewPrefixHandler's flush timer, so the test flushes itself.
h := &PrefixHandler{db: db, logger: logger.New()}
withdrawal := &ristypes.RISMessage{
Type: ristypes.MessageTypeUpdate,
Peer: testPeerIP,
ParsedTimestamp: time.Date(2026, 1, 2, 3, 4, 5, 0, time.UTC),
Withdrawals: []string{"also/bad"},
}
h.HandleMessage(peerAnnouncement(testPeerIP, "not-a-prefix"))
h.HandleMessage(withdrawal)
h.flushBatch()
checkNoPrefixes := func(when string) {
t.Helper()
stats, err := db.GetStatsContext(context.Background())
if err != nil {
t.Fatalf("GetStatsContext %s: %v", when, err)
}
if stats.Prefixes != 0 {
t.Errorf("prefix count %s = %d, want 0", when, stats.Prefixes)
}
}
checkNoPrefixes("after the flush")
err = db.Close()
if err != nil {
t.Fatalf("Close: %v", err)
}
db, err = database.New(cfg, logger.New())
if err != nil {
t.Fatalf("failed to open the database again: %v", err)
}
t.Cleanup(func() { _ = db.Close() })
checkNoPrefixes("after opening the database again")
}
// TestStateMessagesGoToThePrefixHandlerNotThePeerHandler checks that STATE // TestStateMessagesGoToThePrefixHandlerNotThePeerHandler checks that STATE
// messages go to the prefix handler, which acts on a down, and not to the peer // messages go to the prefix handler, which acts on a down, and not to the peer
// handler, which would record a peer whose session never comes up as active. // handler, which would record a peer whose session never comes up as active.
+62 -2
View File
@@ -16,6 +16,7 @@ import (
"git.eeqj.de/sneak/routewatch/internal/database" "git.eeqj.de/sneak/routewatch/internal/database"
"git.eeqj.de/sneak/routewatch/internal/metrics" "git.eeqj.de/sneak/routewatch/internal/metrics"
"git.eeqj.de/sneak/routewatch/internal/ristypes"
"git.eeqj.de/sneak/routewatch/internal/streamer" "git.eeqj.de/sneak/routewatch/internal/streamer"
"git.eeqj.de/sneak/routewatch/internal/templates" "git.eeqj.de/sneak/routewatch/internal/templates"
asinfo "git.eeqj.de/sneak/routewatch/pkg/asinfo" asinfo "git.eeqj.de/sneak/routewatch/pkg/asinfo"
@@ -60,6 +61,14 @@ const (
hostNotResolvedMessage = "could not resolve host name" hostNotResolvedMessage = "could not resolve host name"
hostNotPublicMessage = "host name has no public address" hostNotPublicMessage = "host name has no public address"
tooManyTargetsMessage = "Too many addresses or host names" tooManyTargetsMessage = "Too many addresses or host names"
searchQueryMissingMessage = "Missing search query"
)
// The AS name search reads the text it looks for from the URL query parameter
// asSearchQueryParam, and answers with at most maxASSearchResults matches.
const (
asSearchQueryParam = "q"
maxASSearchResults = 50
) )
// IP versions as database.LiveRoute.IPVersion records them. // IP versions as database.LiveRoute.IPVersion records them.
@@ -1001,11 +1010,12 @@ func (s *Server) handleASDetailJSON() http.HandlerFunc {
} }
} }
// splitRoutesByIPVersion groups routes into the IPv4 and the IPv6 ones. // splitRoutesByIPVersion groups routes into the IPv4 and the IPv6 ones. Neither
// group is nil, so that a JSON answer holds [] for an empty one and not null.
func splitRoutesByIPVersion( func splitRoutesByIPVersion(
routes []database.LiveRoute, routes []database.LiveRoute,
) ([]database.LiveRoute, []database.LiveRoute) { ) ([]database.LiveRoute, []database.LiveRoute) {
var ipv4Routes, ipv6Routes []database.LiveRoute ipv4Routes, ipv6Routes := []database.LiveRoute{}, []database.LiveRoute{}
for _, route := range routes { for _, route := range routes {
if route.IPVersion == ipVersionV4 { if route.IPVersion == ipVersionV4 {
@@ -1105,6 +1115,56 @@ func (s *Server) handlePrefixDetailJSON() http.HandlerFunc {
} }
} }
// ASSearchResponse is the answer to an AS name search: the text searched for,
// the number of ASes that match it, and the first maxASSearchResults of them in
// AS number order.
type ASSearchResponse struct {
Query string `json:"query"`
Total int `json:"total"`
Results []asinfo.Info `json:"results"`
}
// handleASSearch returns a handler that searches the AS list built into
// asinfo for the text in the q parameter, without its leading and trailing
// spaces. It answers 400 when no text is left.
func (s *Server) handleASSearch() http.HandlerFunc {
return func(w http.ResponseWriter, r *http.Request) {
query := strings.TrimSpace(r.URL.Query().Get(asSearchQueryParam))
if query == "" {
s.writeJSONError(w, http.StatusBadRequest, searchQueryMissingMessage)
return
}
// AS 0 means no known origin and has no AS page, so the answer leaves
// it out. The matches come in AS number order, so AS 0 can only be the
// first. The search asks for one match more than the answer holds, to
// take its place.
results, total := asinfo.Search(query, maxASSearchResults+1)
if len(results) > 0 && results[0].ASN == ristypes.NoOrigin {
results = results[1:]
total--
}
results = results[:min(len(results), maxASSearchResults)]
// results is an empty list rather than nil when nothing matches, so
// that the answer holds [] and not null.
if results == nil {
results = []asinfo.Info{}
}
err := writeJSONSuccess(w, ASSearchResponse{
Query: query,
Total: total,
Results: results,
})
if err != nil {
s.logger.Error("Failed to encode AS search results", "error", err)
}
}
}
// handleASDetail returns a handler that serves the AS detail HTML page // handleASDetail returns a handler that serves the AS detail HTML page
func (s *Server) handleASDetail() http.HandlerFunc { func (s *Server) handleASDetail() http.HandlerFunc {
return func(w http.ResponseWriter, r *http.Request) { return func(w http.ResponseWriter, r *http.Request) {
+387 -2
View File
@@ -19,6 +19,7 @@ import (
"git.eeqj.de/sneak/routewatch/internal/database" "git.eeqj.de/sneak/routewatch/internal/database"
"git.eeqj.de/sneak/routewatch/internal/logger" "git.eeqj.de/sneak/routewatch/internal/logger"
"git.eeqj.de/sneak/routewatch/internal/metrics" "git.eeqj.de/sneak/routewatch/internal/metrics"
"git.eeqj.de/sneak/routewatch/internal/ristypes"
"git.eeqj.de/sneak/routewatch/internal/streamer" "git.eeqj.de/sneak/routewatch/internal/streamer"
"git.eeqj.de/sneak/routewatch/pkg/asinfo" "git.eeqj.de/sneak/routewatch/pkg/asinfo"
) )
@@ -26,6 +27,7 @@ import (
// The prefix and peer addresses the prefix page tests store routes for. // The prefix and peer addresses the prefix page tests store routes for.
const ( const (
testPrefixV4 = "192.0.2.0/24" testPrefixV4 = "192.0.2.0/24"
testPrefixV6 = "2001:db8::/32"
testPeerV4 = "198.51.100.1" testPeerV4 = "198.51.100.1"
testPeerV6 = "2001:db8::1" testPeerV6 = "2001:db8::1"
) )
@@ -194,7 +196,7 @@ func TestStatsHandlersAnswerFromMemory(t *testing.T) {
} }
// newServerWithTestRoutes returns a server on a new database that holds a // newServerWithTestRoutes returns a server on a new database that holds a
// route for testPrefixV4 and one for 2001:db8::/32. // route for testPrefixV4 and one for testPrefixV6.
func newServerWithTestRoutes(t *testing.T) *Server { func newServerWithTestRoutes(t *testing.T) *Server {
t.Helper() t.Helper()
@@ -214,7 +216,7 @@ func newServerWithTestRoutes(t *testing.T) *Server {
NextHop: testPeerV4, LastUpdated: ts, NextHop: testPeerV4, LastUpdated: ts,
}, },
{ {
Prefix: "2001:db8::/32", MaskLength: 32, IPVersion: 6, Prefix: testPrefixV6, MaskLength: 32, IPVersion: 6,
OriginASN: 64501, PeerIP: testPeerV6, ASPath: []int{64501}, OriginASN: 64501, PeerIP: testPeerV6, ASPath: []int{64501},
NextHop: testPeerV6, LastUpdated: ts, NextHop: testPeerV6, LastUpdated: ts,
}, },
@@ -1166,6 +1168,389 @@ func TestIPLookupHidesPrivateAddresses(t *testing.T) {
} }
} }
// asSearchPath is the path of the AS name search.
const asSearchPath = "/api/v1/search/as"
// searchAS sends an AS name search with the URL query rawQuery through s's
// router, expects a 200 answer, and returns its data.
func searchAS(t *testing.T, s *Server, rawQuery string) ASSearchResponse {
t.Helper()
rec := httptest.NewRecorder()
s.router.ServeHTTP(rec, httptest.NewRequestWithContext(
t.Context(), http.MethodGet, asSearchPath+"?"+rawQuery, nil))
if rec.Code != http.StatusOK {
t.Fatalf("%s: status %d, want %d; body %s",
rawQuery, rec.Code, http.StatusOK, rec.Body)
}
var body struct {
Data ASSearchResponse `json:"data"`
}
err := json.Unmarshal(rec.Body.Bytes(), &body)
if err != nil {
t.Fatalf("%s: decoding the answer %q: %v", rawQuery, rec.Body, err)
}
return body.Data
}
// TestASSearchAnswersMatches searches for Cloudflare's handle in lower case,
// with spaces around it, and expects AS13335 among the matches. Before the fix
// there was no AS name search, and the home page's search card said it was
// coming soon (https://git.eeqj.de/sneak/routewatch/issues/151).
func TestASSearchAnswersMatches(t *testing.T) {
t.Parallel()
const query = "cloudflarenet"
data := searchAS(t, newServerWithTestRoutes(t), "q=%20"+query+"%20")
if data.Query != query {
t.Errorf("query %q, want %q", data.Query, query)
}
if data.Total != len(data.Results) {
t.Errorf("total %d, but %d results", data.Total, len(data.Results))
}
cloudflare := asinfo.Info{
ASN: 13335, Handle: "CLOUDFLARENET", Description: "Cloudflare, Inc.",
}
if !slices.Contains(data.Results, cloudflare) {
t.Errorf("results %v do not hold %v", data.Results, cloudflare)
}
}
// TestASSearchAnswersEmptyListWhenNothingMatches expects a search that matches
// nothing to answer 200 with a total of 0 and an empty list, not null.
func TestASSearchAnswersEmptyListWhenNothingMatches(t *testing.T) {
t.Parallel()
data := searchAS(t, newServerWithTestRoutes(t), "q=XYZABC123456")
if data.Total != 0 || data.Results == nil || len(data.Results) != 0 {
t.Errorf("total %d and results %#v, want 0 and an empty list",
data.Total, data.Results)
}
}
// TestASSearchAnswers400ForEmptyQuery expects a search with no q, an empty q
// or a q of only spaces to answer 400 with the JSON error body.
func TestASSearchAnswers400ForEmptyQuery(t *testing.T) {
t.Parallel()
s := newServerWithTestRoutes(t)
for _, path := range []string{
asSearchPath, asSearchPath + "?q=", asSearchPath + "?q=%20%20",
} {
rec := httptest.NewRecorder()
s.router.ServeHTTP(rec, httptest.NewRequestWithContext(
t.Context(), http.MethodGet, path, nil))
if rec.Code != http.StatusBadRequest {
t.Errorf("%s: status %d, want %d; body %s",
path, rec.Code, http.StatusBadRequest, rec.Body)
}
checkJSONErrorCode(t, path, rec, http.StatusBadRequest)
checkJSONErrorMessage(t, path, rec, searchQueryMissingMessage)
}
}
// TestASSearchShowsFirstMatchesAndTotal searches for a word with more matches
// than an answer holds, and expects the first maxASSearchResults of them, in
// AS number order, and the number of all of them.
func TestASSearchShowsFirstMatchesAndTotal(t *testing.T) {
t.Parallel()
const query = "university"
all, _ := asinfo.Search(query, asinfo.Total())
if len(all) <= maxASSearchResults {
t.Fatalf("Search(%q) has %d matches; the test needs more than %d",
query, len(all), maxASSearchResults)
}
data := searchAS(t, newServerWithTestRoutes(t), "q="+query)
if data.Total != len(all) {
t.Errorf("total %d, want %d", data.Total, len(all))
}
if !slices.Equal(data.Results, all[:maxASSearchResults]) {
t.Errorf("results %v, want the first %d matches %v",
data.Results, maxASSearchResults, all[:maxASSearchResults])
}
}
// The AS numbers the AS detail tests ask for. storedASN originates the route
// for testPrefixV4 and is in the database. routeOnlyASN originates the route
// for testPrefixV6, but the database has no record of the AS itself, as when
// a route is stored before its AS. builtInOnlyASN is in the AS list built into
// asinfo but not in the database. unknownASN is in neither.
const (
storedASN = 64500
routeOnlyASN = 64501
builtInOnlyASN = 15169
unknownASN = 4200000000
)
// zeroDate is how a page writes a time that was never set.
const zeroDate = "0001-01-01"
// firstSeenCard is the label of the AS page's first-seen date.
const firstSeenCard = `<div class="info-label">First Seen</div>`
// newServerWithStoredAS returns newServerWithTestRoutes's server with storedASN
// added to the database. It fails the test when the AS list built into asinfo
// lacks routeOnlyASN or builtInOnlyASN, or holds unknownASN.
func newServerWithStoredAS(t *testing.T) *Server {
t.Helper()
for _, asn := range []int{routeOnlyASN, builtInOnlyASN} {
if _, ok := asinfo.Get(asn); !ok {
t.Fatalf("AS%d is not in the AS info data", asn)
}
}
if _, ok := asinfo.Get(unknownASN); ok {
t.Fatalf("AS%d is in the AS info data", unknownASN)
}
s := newServerWithTestRoutes(t)
_, err := s.db.GetOrCreateASN(storedASN, time.Date(2026, 1, 2, 3, 4, 5, 0, time.UTC))
if err != nil {
t.Fatalf("GetOrCreateASN: %v", err)
}
return s
}
// TestASDetailJSONAnswersBuiltInAS asks /api/v1/as/ for each of the ASes above.
// All but unknownASN answer 200 with the handle and description from the
// built-in AS list and their prefixes, as lists even when empty. Only storedASN
// has a first_seen time. unknownASN answers 404. Before the fix builtInOnlyASN
// and routeOnlyASN answered 404 too
// (https://git.eeqj.de/sneak/routewatch/issues/151).
func TestASDetailJSONAnswersBuiltInAS(t *testing.T) {
t.Parallel()
s := newServerWithStoredAS(t)
tests := []struct {
asn int
wantCode int
wantPrefixes []string
wantFirstSeen bool
}{
{storedASN, http.StatusOK, []string{testPrefixV4}, true},
{routeOnlyASN, http.StatusOK, []string{testPrefixV6}, false},
{builtInOnlyASN, http.StatusOK, nil, false},
{unknownASN, http.StatusNotFound, nil, false},
}
for _, tt := range tests {
path := "/api/v1/as/" + strconv.Itoa(tt.asn)
rec := httptest.NewRecorder()
s.router.ServeHTTP(rec, httptest.NewRequestWithContext(
t.Context(), http.MethodGet, path, nil))
if rec.Code != tt.wantCode {
t.Errorf("%s: status %d, want %d; body %s",
path, rec.Code, tt.wantCode, rec.Body)
continue
}
if tt.wantCode != http.StatusOK {
checkJSONErrorCode(t, path, rec, tt.wantCode)
continue
}
//nolint:tagliatelle // field names are the existing JSON API
var body struct {
Data struct {
ASN database.ASN `json:"asn"`
IPv4Prefixes []database.LiveRoute `json:"ipv4_prefixes"`
IPv6Prefixes []database.LiveRoute `json:"ipv6_prefixes"`
} `json:"data"`
}
err := json.Unmarshal(rec.Body.Bytes(), &body)
if err != nil {
t.Errorf("%s: decoding the answer %q: %v", path, rec.Body, err)
continue
}
info, _ := asinfo.Get(tt.asn)
got := body.Data.ASN
if got.ASN != tt.asn || got.Handle != info.Handle ||
got.Description != info.Description {
t.Errorf("%s: AS %d %q %q, want %d %q %q", path, got.ASN,
got.Handle, got.Description, tt.asn, info.Handle, info.Description)
}
if body.Data.IPv4Prefixes == nil || body.Data.IPv6Prefixes == nil {
t.Errorf("%s: a prefix list is null; body %s", path, rec.Body)
}
routes := slices.Concat(body.Data.IPv4Prefixes, body.Data.IPv6Prefixes)
gotPrefixes := make([]string, 0, len(routes))
for _, route := range routes {
gotPrefixes = append(gotPrefixes, route.Prefix)
}
if !slices.Equal(gotPrefixes, tt.wantPrefixes) {
t.Errorf("%s: prefixes %v, want %v", path, gotPrefixes, tt.wantPrefixes)
}
gotFirstSeen := strings.Contains(rec.Body.String(), `"first_seen"`)
if gotFirstSeen != tt.wantFirstSeen {
t.Errorf("%s: has first_seen %t, want %t; body %s",
path, gotFirstSeen, tt.wantFirstSeen, rec.Body)
}
}
}
// TestASDetailPageShowsBuiltInAS asks /as/ for each of the ASes above. All but
// unknownASN answer 200 with the handle and description from the built-in AS
// list. storedASN and routeOnlyASN list their prefixes, and builtInOnlyASN says
// the AS announces none. Only storedASN shows a first-seen date, and no page
// shows a date that was never set. unknownASN answers 404. Before the fix
// builtInOnlyASN and routeOnlyASN answered 404 too
// (https://git.eeqj.de/sneak/routewatch/issues/151).
func TestASDetailPageShowsBuiltInAS(t *testing.T) {
t.Parallel()
s := newServerWithStoredAS(t)
tests := []struct {
asn int
wantCode int
wantText string
wantFirstSeen bool
}{
{storedASN, http.StatusOK, testPrefixV4, true},
{routeOnlyASN, http.StatusOK, testPrefixV6, false},
{builtInOnlyASN, http.StatusOK, "No prefixes announced by this AS", false},
{unknownASN, http.StatusNotFound, "", false},
}
for _, tt := range tests {
path := "/as/" + strconv.Itoa(tt.asn)
rec := httptest.NewRecorder()
s.router.ServeHTTP(rec, httptest.NewRequestWithContext(
t.Context(), http.MethodGet, path, nil))
if rec.Code != tt.wantCode {
t.Errorf("%s: status %d, want %d; body %s",
path, rec.Code, tt.wantCode, rec.Body)
continue
}
if tt.wantCode != http.StatusOK {
continue
}
info, _ := asinfo.Get(tt.asn)
page := rec.Body.String()
for _, want := range []string{
html.EscapeString(info.Handle),
html.EscapeString(info.Description),
tt.wantText,
} {
if !strings.Contains(page, want) {
t.Errorf("%s: page does not show %q", path, want)
}
}
if strings.Contains(page, zeroDate) {
t.Errorf("%s: page shows the date %s", path, zeroDate)
}
gotFirstSeen := strings.Contains(page, firstSeenCard)
if gotFirstSeen != tt.wantFirstSeen {
t.Errorf("%s: shows a first-seen date %t, want %t",
path, gotFirstSeen, tt.wantFirstSeen)
}
}
}
// TestNoOriginHasNoASAnswers stores a route with no known origin, which has
// origin AS 0 (https://git.eeqj.de/sneak/routewatch/issues/75). It expects
// /as/0 and /api/v1/as/0 to answer 404. It expects a search for "iana" to leave
// AS 0 out of its results and its total, although the AS list built into
// asinfo holds AS 0. That search has more matches than an answer holds, so it
// still lists maxASSearchResults of them.
func TestNoOriginHasNoASAnswers(t *testing.T) {
t.Parallel()
const (
noOriginPrefix = "203.0.113.0/24"
query = "iana"
)
s := newServerWithTestRoutes(t)
err := s.db.UpsertLiveRouteBatch([]*database.LiveRoute{{
Prefix: noOriginPrefix, MaskLength: 24, IPVersion: 4,
OriginASN: ristypes.NoOrigin, PeerIP: testPeerV4,
ASPath: []int{64500, 64501, 64502}, NextHop: testPeerV4,
LastUpdated: time.Date(2026, 1, 2, 3, 4, 5, 0, time.UTC),
}})
if err != nil {
t.Fatalf("UpsertLiveRouteBatch: %v", err)
}
asn := strconv.Itoa(ristypes.NoOrigin)
for _, path := range []string{"/as/" + asn, "/api/v1/as/" + asn} {
rec := httptest.NewRecorder()
s.router.ServeHTTP(rec, httptest.NewRequestWithContext(
t.Context(), http.MethodGet, path, nil))
if rec.Code != http.StatusNotFound {
t.Errorf("%s: status %d, want %d; body %s",
path, rec.Code, http.StatusNotFound, rec.Body)
}
}
first, total := asinfo.Search(query, 1)
if len(first) != 1 || first[0].ASN != ristypes.NoOrigin {
t.Fatalf("AS%d is not the first match for %q in the AS info data",
ristypes.NoOrigin, query)
}
data := searchAS(t, s, "q="+query)
for _, info := range data.Results {
if info.ASN == ristypes.NoOrigin {
t.Errorf("the search for %q lists AS%d", query, info.ASN)
}
}
if data.Total != total-1 {
t.Errorf("the search for %q has a total of %d, want %d",
query, data.Total, total-1)
}
if len(data.Results) != maxASSearchResults {
t.Errorf("the search for %q lists %d matches, want %d",
query, len(data.Results), maxASSearchResults)
}
}
// settledGoroutineCount lets transient goroutines finish, then reports the // settledGoroutineCount lets transient goroutines finish, then reports the
// current count. // current count.
func settledGoroutineCount() int { func settledGoroutineCount() int {
+9 -3
View File
@@ -142,8 +142,9 @@ func deadlinePassed(ctx context.Context) bool {
// //
// If a response is not valid JSON or is empty when JSON is expected, the middleware // If a response is not valid JSON or is empty when JSON is expected, the middleware
// returns a properly formatted JSON error response. If the request's time limit // returns a properly formatted JSON error response. If the request's time limit
// has passed, it answers 504 with the error message "Request timeout". For other // has passed, it answers 504 with the error message "Request timeout". A 4xx
// errors, it returns "Internal server error" with status 500 if the original // keeps its status, with the status's standard text as the message. For other
// errors, it returns "Internal server error", with status 500 if the original
// status was 200. // status was 200.
// //
// Non-JSON responses are passed through unchanged. // Non-JSON responses are passed through unchanged.
@@ -203,8 +204,13 @@ func JSONValidationMiddleware(next http.Handler) http.Handler {
w.WriteHeader(statusCode) w.WriteHeader(statusCode)
errorMsg := internalErrorMessage errorMsg := internalErrorMessage
if statusCode == http.StatusGatewayTimeout {
switch {
case statusCode == http.StatusGatewayTimeout:
errorMsg = requestTimeoutMessage errorMsg = requestTimeoutMessage
case statusCode >= http.StatusBadRequest &&
statusCode < http.StatusInternalServerError:
errorMsg = http.StatusText(statusCode)
} }
response := map[string]any{ response := map[string]any{
@@ -58,6 +58,34 @@ func TestResponsesCarrySecurityHeaders(t *testing.T) {
} }
} }
// TestJSONValidationKeeps4xxWithoutBody wraps a handler that answers 400
// without a body in JSONValidationMiddleware and checks that the JSON error
// body keeps the 400 and gives the status's own text. Before the fix it said
// "Internal server error"
// (https://git.eeqj.de/sneak/routewatch/issues/111).
func TestJSONValidationKeeps4xxWithoutBody(t *testing.T) {
t.Parallel()
handler := JSONValidationMiddleware(http.HandlerFunc(
func(w http.ResponseWriter, _ *http.Request) {
w.WriteHeader(http.StatusBadRequest)
}))
rec := httptest.NewRecorder()
handler.ServeHTTP(rec, httptest.NewRequestWithContext(
t.Context(), http.MethodGet, "/", nil))
const name = "400 without a body"
if rec.Code != http.StatusBadRequest {
t.Errorf("%s: status %d, want %d; body %s",
name, rec.Code, http.StatusBadRequest, rec.Body)
}
checkJSONErrorCode(t, name, rec, http.StatusBadRequest)
checkJSONErrorMessage(t, name, rec, http.StatusText(http.StatusBadRequest))
}
// TestPagesCarryTheCSPNonce checks that every script and style block of each // TestPagesCarryTheCSPNonce checks that every script and style block of each
// page carries the nonce its Content-Security-Policy header names, so that the // page carries the nonce its Content-Security-Policy header names, so that the
// browser runs and applies them, that each answer has a nonce of its own, and // browser runs and applies them, that each answer has a nonce of its own, and
+34
View File
@@ -1,18 +1,33 @@
package server package server
import ( import (
"net/http"
"time" "time"
"github.com/go-chi/chi/v5" "github.com/go-chi/chi/v5"
"github.com/go-chi/chi/v5/middleware" "github.com/go-chi/chi/v5/middleware"
) )
// The error messages the JSON routes under /api/v1 and /ip answer with when no
// route matches the path, and when none takes the method.
const (
notFoundMessage = "Not found"
methodNotAllowedMessage = "Method not allowed"
// jsonRoutesAllow is the Allow header of their 405 answer: every one of
// them takes GET and HEAD, and no other method.
jsonRoutesAllow = "GET, HEAD"
)
// setupRoutes configures the HTTP routes // setupRoutes configures the HTTP routes
func (s *Server) setupRoutes() { func (s *Server) setupRoutes() {
r := chi.NewRouter() r := chi.NewRouter()
// Middleware // Middleware
r.Use(DecodedPathMiddleware) r.Use(DecodedPathMiddleware)
// GetHead serves HEAD with the GET handler. It looks the route up by the
// path DecodedPathMiddleware sets, so it comes after it.
r.Use(middleware.GetHead)
r.Use(SecurityHeadersMiddleware) r.Use(SecurityHeadersMiddleware)
r.Use(middleware.RequestID) r.Use(middleware.RequestID)
r.Use(RealIPMiddleware(s.trustedProxies)) r.Use(RealIPMiddleware(s.trustedProxies))
@@ -40,20 +55,39 @@ func (s *Server) setupRoutes() {
r.Get("/prefixlength/{length}", s.handlePrefixLength()) r.Get("/prefixlength/{length}", s.handlePrefixLength())
r.Get("/prefixlength6/{length}", s.handlePrefixLength6()) r.Get("/prefixlength6/{length}", s.handlePrefixLength6())
// The JSON routes answer a path or method they do not take with the JSON
// error body, as they answer everything else.
jsonNotFound := func(w http.ResponseWriter, _ *http.Request) {
s.writeJSONError(w, http.StatusNotFound, notFoundMessage)
}
jsonMethodNotAllowed := func(w http.ResponseWriter, _ *http.Request) {
// chi's own 405 answer sets Allow; one set in its place must too.
w.Header().Set("Allow", jsonRoutesAllow)
s.writeJSONError(w, http.StatusMethodNotAllowed, methodNotAllowedMessage)
}
// IP info JSON endpoints (replaces old /ip redirect) // IP info JSON endpoints (replaces old /ip redirect)
r.Route("/ip", func(r chi.Router) { r.Route("/ip", func(r chi.Router) {
r.Use(JSONValidationMiddleware) r.Use(JSONValidationMiddleware)
r.NotFound(jsonNotFound)
r.MethodNotAllowed(jsonMethodNotAllowed)
r.Get("/", s.handleIPInfo()) // Client IP r.Get("/", s.handleIPInfo()) // Client IP
// GetHead does not reach this route: mounting the group registers
// /ip and /ip/ for every method, so HEAD arrives here as HEAD.
r.Head("/", s.handleIPInfo())
r.Get("/{addr}", s.handleIPInfo()) // Specified IP r.Get("/{addr}", s.handleIPInfo()) // Specified IP
}) })
// API routes // API routes
r.Route("/api/v1", func(r chi.Router) { r.Route("/api/v1", func(r chi.Router) {
r.Use(JSONValidationMiddleware) r.Use(JSONValidationMiddleware)
r.NotFound(jsonNotFound)
r.MethodNotAllowed(jsonMethodNotAllowed)
r.Get("/stats", s.handleStats()) r.Get("/stats", s.handleStats())
r.Get("/ip/{ip}", s.handleIPLookup()) r.Get("/ip/{ip}", s.handleIPLookup())
r.Get("/as/{asn}", s.handleASDetailJSON()) r.Get("/as/{asn}", s.handleASDetailJSON())
r.Get("/prefix/{prefix}/{len}", s.handlePrefixDetailJSON()) r.Get("/prefix/{prefix}/{len}", s.handlePrefixDetailJSON())
r.Get("/search/as", s.handleASSearch())
}) })
s.router = r s.router = r
+100
View File
@@ -0,0 +1,100 @@
package server
import (
"encoding/json"
"net/http"
"net/http/httptest"
"testing"
)
// TestRoutesAnswerHEAD sends HEAD for a page and JSON routes through the
// server's router and checks that each answers as GET does. /ip and /ip/, the
// lookup of the caller's own address, are in the list because chi's GetHead
// does not reach them. Before the fix every route answered HEAD with 405
// (https://git.eeqj.de/sneak/routewatch/issues/111).
func TestRoutesAnswerHEAD(t *testing.T) {
t.Parallel()
s := newServerWithTestRoutes(t)
for _, path := range []string{"/", statsPath, "/ip", "/ip/"} {
rec := httptest.NewRecorder()
s.router.ServeHTTP(rec, httptest.NewRequestWithContext(
t.Context(), http.MethodHead, path, nil))
if rec.Code != http.StatusOK {
t.Errorf("HEAD %s: status %d, want %d; body %s",
path, rec.Code, http.StatusOK, rec.Body)
}
}
}
// TestJSONRoutesAnswerUnmatchedRequestsWithJSONErrors sends a method the JSON
// routes do not take, and a path none of them matches, to each group of JSON
// routes through the server's router. Each must answer with the JSON error
// body, and a 405 must name the methods the routes take in its Allow header.
// Before the fix a 405 said "Internal server error" and a 404 was chi's plain
// text answer (https://git.eeqj.de/sneak/routewatch/issues/111).
func TestJSONRoutesAnswerUnmatchedRequestsWithJSONErrors(t *testing.T) {
t.Parallel()
s := newServerWithTestRoutes(t)
tests := []struct {
method string
path string
wantCode int
wantMsg string
}{
{http.MethodPost, statsPath, http.StatusMethodNotAllowed, methodNotAllowedMessage},
{http.MethodPost, "/ip/", http.StatusMethodNotAllowed, methodNotAllowedMessage},
{http.MethodGet, "/api/v1/nothing", http.StatusNotFound, notFoundMessage},
{http.MethodGet, "/ip/192.0.2.1/nothing", http.StatusNotFound, notFoundMessage},
}
for _, tt := range tests {
name := tt.method + " " + tt.path
rec := httptest.NewRecorder()
s.router.ServeHTTP(rec, httptest.NewRequestWithContext(
t.Context(), tt.method, tt.path, nil))
if rec.Code != tt.wantCode {
t.Errorf("%s: status %d, want %d; body %s",
name, rec.Code, tt.wantCode, rec.Body)
}
checkJSONErrorCode(t, name, rec, tt.wantCode)
checkJSONErrorMessage(t, name, rec, tt.wantMsg)
allow := rec.Header().Get("Allow")
if tt.wantCode == http.StatusMethodNotAllowed && allow != jsonRoutesAllow {
t.Errorf("%s: Allow %q, want %q", name, allow, jsonRoutesAllow)
}
}
}
// checkJSONErrorMessage checks that rec holds the JSON error body with the
// given message.
func checkJSONErrorMessage(
t *testing.T, name string, rec *httptest.ResponseRecorder, want string,
) {
t.Helper()
var body struct {
Error struct {
Msg string `json:"msg"`
} `json:"error"`
}
err := json.Unmarshal(rec.Body.Bytes(), &body)
if err != nil {
t.Errorf("%s: decoding the answer %q: %v", name, rec.Body, err)
return
}
if body.Error.Msg != want {
t.Errorf("%s: error message %q, want %q", name, body.Error.Msg, want)
}
}
+7 -2
View File
@@ -1,4 +1,8 @@
// Package server provides HTTP endpoints for status monitoring and statistics // Package server serves RouteWatch's web pages and JSON API:
// - the home page, the status figures and the health check
// - the AS, prefix and prefix length pages, and the JSON answers behind them
// - the IP lookups
// - the AS name search
package server package server
import ( import (
@@ -139,7 +143,8 @@ func (s *Server) Stop(ctx context.Context) error {
return nil return nil
} }
// SetASNFetcher sets the ASN WHOIS fetcher for on-demand lookups. // SetASNFetcher sets the ASN WHOIS fetcher for on-demand lookups. Call it
// before Start: the handlers read the fetcher without a lock.
func (s *Server) SetASNFetcher(fetcher ASNFetcher) { func (s *Server) SetASNFetcher(fetcher ASNFetcher) {
s.asnFetcher = fetcher s.asnFetcher = fetcher
} }
+5 -5
View File
@@ -106,12 +106,12 @@ type RawMessageHandler func(line string)
// handlerMetrics tracks performance metrics for a handler // handlerMetrics tracks performance metrics for a handler
type handlerMetrics struct { type handlerMetrics struct {
processedCount uint64 // Total messages processed processedCount uint64 // Total messages processed
droppedCount uint64 // Total messages dropped droppedCount atomic.Uint64 // Total messages dropped; not under mu
totalTime time.Duration // Total processing time (for average calculation) totalTime time.Duration // Total processing time (for average calculation)
minTime time.Duration // Minimum processing time minTime time.Duration // Minimum processing time
maxTime time.Duration // Maximum processing time maxTime time.Duration // Maximum processing time
queueHighWaterMark int // Maximum queue length seen queueHighWaterMark int // Maximum queue length seen
mu sync.Mutex // Protects the metrics mu sync.Mutex // Protects the other metrics
} }
// handlerInfo wraps a handler with its queue and metrics // handlerInfo wraps a handler with its queue and metrics
@@ -371,7 +371,7 @@ func (s *Streamer) GetHandlerStats() []HandlerStats {
QueueCapacity: cap(info.queue), QueueCapacity: cap(info.queue),
QueueHighWaterMark: info.metrics.queueHighWaterMark, QueueHighWaterMark: info.metrics.queueHighWaterMark,
ProcessedCount: info.metrics.processedCount, ProcessedCount: info.metrics.processedCount,
DroppedCount: info.metrics.droppedCount, DroppedCount: info.metrics.droppedCount.Load(),
MinProcessTime: info.metrics.minTime, MinProcessTime: info.metrics.minTime,
MaxProcessTime: info.metrics.maxTime, MaxProcessTime: info.metrics.maxTime,
} }
@@ -488,7 +488,7 @@ func (s *Streamer) logMetrics() {
"queue_len", len(info.queue), "queue_len", len(info.queue),
"queue_cap", cap(info.queue), "queue_cap", cap(info.queue),
"processed", info.metrics.processedCount, "processed", info.metrics.processedCount,
"dropped", info.metrics.droppedCount, "dropped", info.metrics.droppedCount.Load(),
"avg_time", avgTime, "avg_time", avgTime,
"min_time", info.metrics.minTime, "min_time", info.metrics.minTime,
"max_time", info.metrics.maxTime, "max_time", info.metrics.maxTime,
@@ -858,7 +858,7 @@ func (s *Streamer) dispatchMessage(
} }
if !s.queueMessage(info, msg) { if !s.queueMessage(info, msg) {
atomic.AddUint64(&info.metrics.droppedCount, 1) info.metrics.droppedCount.Add(1)
s.totalDropped.Add(1) s.totalDropped.Add(1)
continue continue
@@ -621,6 +621,68 @@ func TestStreamReadsLongLine(t *testing.T) {
} }
} }
// blockedHandler wants UPDATE messages, has room on its queue for one, and
// does not return from HandleMessage until release is closed.
type blockedHandler struct {
release chan struct{}
}
func (blockedHandler) WantsMessage(messageType string) bool {
return messageType == ristypes.MessageTypeUpdate
}
func (blockedHandler) QueueCapacity() int { return 1 }
func (h blockedHandler) HandleMessage(*ristypes.RISMessage) { <-h.release }
// TestHandlerStatsReadWhileMessagesAreDropped has the server send a few UPDATE
// lines to a blockedHandler, so the read loop drops all but the one the
// handler holds and the one on its queue, while the test reads the handler's
// statistics. The drop count must reach the number dropped, and the race
// detector must find the count read and written safely.
func TestHandlerStatsReadWhileMessagesAreDropped(t *testing.T) {
t.Parallel()
const (
lines = 5
wantDropped = lines - 2
maxWait = 5 * time.Second
)
srv := httptest.NewServer(http.HandlerFunc(
func(w http.ResponseWriter, r *http.Request) {
_, _ = io.WriteString(w, strings.Repeat(updateLine, lines))
flush(w)
<-r.Context().Done()
}))
defer srv.Close()
s := New(logger.New(), metrics.New())
s.url = srv.URL
handler := blockedHandler{release: make(chan struct{})}
s.RegisterHandler(handler)
err := s.Start(t.Context())
if err != nil {
t.Fatalf("Start returned error: %v", err)
}
defer s.Stop(t.Context())
defer close(handler.release)
// The test polls instead of waiting for a signal from the read loop: a
// signal would order the drops before the reads here, and the race
// detector would have nothing to report.
deadline := time.Now().Add(maxWait)
for s.GetHandlerStats()[0].DroppedCount < wantDropped {
if time.Now().After(deadline) {
t.Fatalf("fewer than %d messages counted as dropped within %s",
wantDropped, maxWait)
}
}
}
// TestPeerCountCountsEveryPeerHeardFrom has the server send an UPDATE from // TestPeerCountCountsEveryPeerHeardFrom has the server send an UPDATE from
// stateDownPeer and a KEEPALIVE from another peer, with no OPEN from either, // stateDownPeer and a KEEPALIVE from another peer, with no OPEN from either,
// and then a message that ends stateDownPeer's session: a NOTIFICATION, or a // and then a message that ends stateDownPeer's session: a NOTIFICATION, or a
+2
View File
@@ -233,10 +233,12 @@
<div class="info-label">Peer ASNs</div> <div class="info-label">Peer ASNs</div>
<div class="info-value">{{.PeerCount}}</div> <div class="info-value">{{.PeerCount}}</div>
</div> </div>
{{if not .ASN.FirstSeen.IsZero}}
<div class="info-card"> <div class="info-card">
<div class="info-label">First Seen</div> <div class="info-label">First Seen</div>
<div class="info-value">{{.ASN.FirstSeen.Format "2006-01-02"}}</div> <div class="info-value">{{.ASN.FirstSeen.Format "2006-01-02"}}</div>
</div> </div>
{{end}}
</div> </div>
{{if .IPv4Prefixes}} {{if .IPv4Prefixes}}
+101 -11
View File
@@ -174,6 +174,38 @@
#asname-results .search-message { #asname-results .search-message {
font-size: 13px; font-size: 13px;
} }
#asname-results .error {
color: #c00;
}
.asname-list {
list-style: none;
margin: 12px 0 0 0;
padding: 0;
max-height: 400px;
overflow-y: auto;
}
.asname-list li {
border-bottom: 1px solid #eee;
}
.asname-list a {
display: block;
padding: 8px 4px;
color: #2c3e50;
text-decoration: none;
font-size: 14px;
}
.asname-list a:hover {
background: #f5f5f5;
}
.asname-list .asn {
font-weight: 600;
color: #3498db;
}
.asname-list .description {
display: block;
color: #7f8c8d;
font-size: 12px;
}
/* IP Lookup result */ /* IP Lookup result */
.ip-result { .ip-result {
@@ -306,7 +338,7 @@
<input type="text" id="asname-input" placeholder="e.g., Google, Cloudflare" autocomplete="off"> <input type="text" id="asname-input" placeholder="e.g., Google, Cloudflare" autocomplete="off">
<button type="submit">Search</button> <button type="submit">Search</button>
</form> </form>
<p class="search-hint">Search for autonomous systems by organization name</p> <p class="search-hint">Search for autonomous systems by organization name or handle</p>
<div id="asname-results"></div> <div id="asname-results"></div>
</div> </div>
@@ -387,24 +419,82 @@
} }
}); });
// AS name search // AS name search. The handles and descriptions come from outside data,
// so the results are built with textContent, never innerHTML.
const asNameResults = document.getElementById('asname-results');
// Replace the search results with one line of text.
function showASNameMessage(text, className) {
const line = document.createElement('p');
line.className = className;
line.textContent = text;
asNameResults.replaceChildren(line);
}
// List the matches of a search answer, each as a link to its AS page,
// and say how many more matched than the answer holds.
function showASNameMatches(data) {
if (data.total === 0) {
showASNameMessage('No AS matches "' + data.query + '".', 'search-message');
return;
}
const list = document.createElement('ul');
list.className = 'asname-list';
for (const match of data.results) {
const asn = document.createElement('span');
asn.className = 'asn';
asn.textContent = 'AS' + match.asn;
const description = document.createElement('span');
description.className = 'description';
description.textContent = match.description;
const link = document.createElement('a');
link.href = '/as/' + match.asn;
link.append(asn, ' ' + match.handle, description);
const item = document.createElement('li');
item.append(link);
list.append(item);
}
asNameResults.replaceChildren(list);
const more = data.total - data.results.length;
if (more > 0) {
const line = document.createElement('p');
line.className = 'search-message';
line.textContent = more.toLocaleString() + ' more matched. Narrow the search to see them.';
asNameResults.append(line);
}
}
document.getElementById('asname-form').addEventListener('submit', function(e) { document.getElementById('asname-form').addEventListener('submit', function(e) {
e.preventDefault(); e.preventDefault();
const query = document.getElementById('asname-input').value.trim(); const query = document.getElementById('asname-input').value.trim();
if (!query) return; if (!query) return;
const resultsDiv = document.getElementById('asname-results'); // A number, or AS and a number, opens that AS's page
resultsDiv.innerHTML = '<p class="loading">Searching...</p>'; const asn = query.replace(/^AS/i, '');
if (/^\d+$/.test(asn)) {
// Use a simple client-side search against the asinfo data window.location.href = '/as/' + asn;
// For now, redirect to AS page if it looks like an ASN
if (/^\d+$/.test(query)) {
window.location.href = '/as/' + query;
return; return;
} }
// Show a message that server-side search is coming showASNameMessage('Searching...', 'loading');
resultsDiv.innerHTML = '<p class="search-message">AS name search coming soon. For now, try an AS number.</p>';
fetch('/api/v1/search/as?q=' + encodeURIComponent(query))
.then(response => response.json())
.then(response => {
if (response.status === 'error') {
showASNameMessage('Error: ' + response.error.msg, 'error');
} else {
showASNameMatches(response.data);
}
})
.catch(error => {
showASNameMessage('Error: ' + error.message, 'error');
});
}); });
// IP lookup // IP lookup
+54
View File
@@ -0,0 +1,54 @@
package templates_test
import (
"bytes"
"html/template"
"strings"
"testing"
"git.eeqj.de/sneak/routewatch/internal/templates"
)
// TestFooterLicense renders the home and status pages and expects each footer
// to name the licence in LICENSE, MIT, and not WTFPL.
func TestFooterLicense(t *testing.T) {
t.Parallel()
const (
license = "MIT"
wrongLicense = "WTFPL"
)
pages := map[string]*template.Template{
"index": templates.IndexTemplate(),
"status": templates.StatusTemplate(),
}
for name, tmpl := range pages {
t.Run(name, func(t *testing.T) {
t.Parallel()
var page bytes.Buffer
err := tmpl.Execute(&page, struct{ CSPNonce string }{})
if err != nil {
t.Fatalf("rendering the page: %v", err)
}
_, footer, found := strings.Cut(page.String(), "<footer")
if !found {
t.Fatal("the page has no footer")
}
footer, _, _ = strings.Cut(footer, "</footer>")
if !strings.Contains(footer, license) {
t.Errorf("footer does not name %s: %q", license, footer)
}
if strings.Contains(footer, wrongLicense) {
t.Errorf("footer names %s: %q", wrongLicense, footer)
}
})
}
}
+2 -2
View File
@@ -19,8 +19,8 @@ const (
Author = "@sneak" Author = "@sneak"
// AuthorURL is the author's website // AuthorURL is the author's website
AuthorURL = "https://sneak.berlin" AuthorURL = "https://sneak.berlin"
// License is the program license // License is the program license, as LICENSE grants it
License = "WTFPL" License = "MIT"
// RepoURL is the git repository URL // RepoURL is the git repository URL
RepoURL = "https://git.eeqj.de/sneak/routewatch" RepoURL = "https://git.eeqj.de/sneak/routewatch"
) )
+49 -23
View File
@@ -2,10 +2,13 @@ package asinfo
import ( import (
"bytes" "bytes"
"cmp"
"compress/gzip" "compress/gzip"
_ "embed" _ "embed"
"encoding/json" "encoding/json"
"io" "io"
"slices"
"strings"
"sync" "sync"
) )
@@ -23,6 +26,17 @@ type Info struct {
type Registry struct { type Registry struct {
mu sync.RWMutex mu sync.RWMutex
byASN map[int]*Info byASN map[int]*Info
// searchEntries holds every entry in AS number order, for Search.
searchEntries []searchEntry
}
// searchEntry is an entry with its handle and description in lower case. They
// are put in lower case once, when the data loads, so that Search only
// compares.
type searchEntry struct {
info *Info
lowerHandle string
lowerDescription string
} }
var ( var (
@@ -59,9 +73,21 @@ func initRegistry() {
panic("failed to unmarshal embedded AS data: " + err.Error()) panic("failed to unmarshal embedded AS data: " + err.Error())
} }
slices.SortFunc(asInfos, func(a, b Info) int {
return cmp.Compare(a.ASN, b.ASN)
})
defaultRegistry.searchEntries = make([]searchEntry, 0, len(asInfos))
for i := range asInfos { for i := range asInfos {
info := &asInfos[i] info := &asInfos[i]
defaultRegistry.byASN[info.ASN] = info defaultRegistry.byASN[info.ASN] = info
defaultRegistry.searchEntries = append(defaultRegistry.searchEntries,
searchEntry{
info: info,
lowerHandle: strings.ToLower(info.Handle),
lowerDescription: strings.ToLower(info.Description),
})
} }
} }
@@ -129,36 +155,36 @@ func All() []Info {
return result return result
} }
// Search searches for AS entries where the handle or description contains the query // Search finds the AS entries whose handle or description contains query,
// This is a simple case-sensitive substring search // ignoring case. It returns the first limit of them in AS number order, and the
func Search(query string) []Info { // number of entries that match. An empty query matches nothing.
func Search(query string, limit int) ([]Info, int) {
once.Do(initRegistry) once.Do(initRegistry)
if query == "" {
return nil, 0
}
query = strings.ToLower(query)
defaultRegistry.mu.RLock() defaultRegistry.mu.RLock()
defer defaultRegistry.mu.RUnlock() defer defaultRegistry.mu.RUnlock()
var results []Info var (
results []Info
total int
)
for _, info := range defaultRegistry.byASN { for _, entry := range defaultRegistry.searchEntries {
if contains(info.Handle, query) || contains(info.Description, query) { if strings.Contains(entry.lowerHandle, query) ||
results = append(results, *info) strings.Contains(entry.lowerDescription, query) {
if total < limit {
results = append(results, *entry.info)
}
total++
} }
} }
return results return results, total
}
// contains is a simple substring search
func contains(s, substr string) bool {
return len(substr) > 0 && len(s) >= len(substr) && containsImpl(s, substr)
}
func containsImpl(s, substr string) bool {
for i := 0; i <= len(s)-len(substr); i++ {
if s[i:i+len(substr)] == substr {
return true
}
}
return false
} }
+100 -12
View File
@@ -1,6 +1,8 @@
package asinfo_test package asinfo_test
import ( import (
"cmp"
"slices"
"strings" "strings"
"testing" "testing"
@@ -140,11 +142,6 @@ func TestSearch(t *testing.T) {
query: "University", query: "University",
wantMin: 10, wantMin: 10,
}, },
{
name: "Empty search",
query: "",
wantMin: 0,
},
{ {
name: "Non-existent org", name: "Non-existent org",
query: "XYZABC123456", query: "XYZABC123456",
@@ -156,26 +153,115 @@ func TestSearch(t *testing.T) {
t.Run(tt.name, func(t *testing.T) { t.Run(tt.name, func(t *testing.T) {
t.Parallel() t.Parallel()
results := asinfo.Search(tt.query) results, _ := asinfo.Search(tt.query, asinfo.Total())
if len(results) < tt.wantMin { if len(results) < tt.wantMin {
t.Errorf("Search(%q) returned %d results, want at least %d", t.Errorf("Search(%q) returned %d results, want at least %d",
tt.query, len(results), tt.wantMin) tt.query, len(results), tt.wantMin)
} }
// Verify all results contain the query // Verify all results contain the query, ignoring case
if tt.query != "" { query := strings.ToLower(tt.query)
for _, r := range results { for _, r := range results {
if !strings.Contains(r.Handle, tt.query) && if !strings.Contains(strings.ToLower(r.Handle), query) &&
!strings.Contains(r.Description, tt.query) { !strings.Contains(strings.ToLower(r.Description), query) {
t.Errorf("Search result ASN %d doesn't contain query %q", t.Errorf("Search result ASN %d doesn't contain query %q",
r.ASN, tt.query) r.ASN, tt.query)
} }
} }
}
}) })
} }
} }
// TestSearchEmptyQueryMatchesNothing expects an empty query to match no AS,
// although every handle contains the empty string.
func TestSearchEmptyQueryMatchesNothing(t *testing.T) {
t.Parallel()
results, total := asinfo.Search("", asinfo.Total())
if len(results) != 0 || total != 0 {
t.Errorf(`Search("") returned %d results and a total of %d, want none`,
len(results), total)
}
}
// TestSearchIgnoresCase searches for Cloudflare's handle in lower case and its
// description in upper case, and expects both searches to find AS13335.
func TestSearchIgnoresCase(t *testing.T) {
t.Parallel()
const cloudflareASN = 13335 // handle CLOUDFLARENET, "Cloudflare, Inc."
isCloudflare := func(info asinfo.Info) bool {
return info.ASN == cloudflareASN
}
for _, query := range []string{"cloudflarenet", "CLOUDFLARE, INC."} {
results, _ := asinfo.Search(query, asinfo.Total())
if !slices.ContainsFunc(results, isCloudflare) {
t.Errorf("Search(%q) did not find AS%d", query, cloudflareASN)
}
}
}
// TestSearchOrdersByASN expects the matches of a search with many of them to
// come in AS number order.
func TestSearchOrdersByASN(t *testing.T) {
t.Parallel()
const query = "University"
results, _ := asinfo.Search(query, asinfo.Total())
for i := 1; i < len(results); i++ {
if results[i-1].ASN >= results[i].ASN {
t.Fatalf("Search(%q): AS%d comes after AS%d",
query, results[i].ASN, results[i-1].ASN)
}
}
}
// TestSearchStopsAtLimit searches for a word that matches more entries than
// the limit. It expects exactly limit entries, the first matches in AS number
// order, and the number of all matches, counted here from All.
func TestSearchStopsAtLimit(t *testing.T) {
t.Parallel()
const (
query = "university"
limit = 10
)
var matches []asinfo.Info
for _, info := range asinfo.All() {
if strings.Contains(strings.ToLower(info.Handle), query) ||
strings.Contains(strings.ToLower(info.Description), query) {
matches = append(matches, info)
}
}
if len(matches) <= limit {
t.Fatalf("%q matches %d entries; the test needs more than %d",
query, len(matches), limit)
}
slices.SortFunc(matches, func(a, b asinfo.Info) int {
return cmp.Compare(a.ASN, b.ASN)
})
results, total := asinfo.Search(query, limit)
if total != len(matches) {
t.Errorf("Search(%q, %d): total %d, want %d",
query, limit, total, len(matches))
}
if !slices.Equal(results, matches[:limit]) {
t.Errorf("Search(%q, %d) = %v, want %v",
query, limit, results, matches[:limit])
}
}
func TestDataIntegrity(t *testing.T) { func TestDataIntegrity(t *testing.T) {
t.Parallel() t.Parallel()
@@ -215,11 +301,13 @@ func BenchmarkGet(b *testing.B) {
} }
func BenchmarkSearch(b *testing.B) { func BenchmarkSearch(b *testing.B) {
const limit = 50
queries := []string{"Google", "Amazon", "Microsoft", "University"} queries := []string{"Google", "Amazon", "Microsoft", "University"}
b.ResetTimer() b.ResetTimer()
for i := range b.N { for i := range b.N {
asinfo.Search(queries[i%len(queries)]) asinfo.Search(queries[i%len(queries)], limit)
} }
} }
+4 -2
View File
@@ -16,8 +16,10 @@ Basic usage:
// Get just the description // Get just the description
desc := asinfo.GetDescription(15169) // "Google LLC" desc := asinfo.GetDescription(15169) // "Google LLC"
// Search for AS entries // Search the handles and descriptions, ignoring case, for the first 10
results := asinfo.Search("University") // matches in AS number order and the number of all matches
results, total := asinfo.Search("university", 10)
fmt.Printf("%d matches\n", total)
for _, as := range results { for _, as := range results {
fmt.Printf("AS%d: %s\n", as.ASN, as.Description) fmt.Printf("AS%d: %s\n", as.ASN, as.Description)
} }
+9 -2
View File
@@ -21,9 +21,16 @@ func ExampleGetDescription() {
} }
func ExampleSearch() { func ExampleSearch() {
results := asinfo.Search("MIT-GATEWAY") const limit = 10
results, total := asinfo.Search("mit-gateway", limit)
fmt.Printf("%d of %d matches\n", len(results), total)
for _, info := range results { for _, info := range results {
fmt.Printf("AS%d: %s - %s\n", info.ASN, info.Handle, info.Description) fmt.Printf("AS%d: %s - %s\n", info.ASN, info.Handle, info.Description)
} }
// Output: AS3: MIT-GATEWAYS - Massachusetts Institute of Technology // Output:
// 1 of 1 matches
// AS3: MIT-GATEWAYS - Massachusetts Institute of Technology
} }