15 Commits
Author SHA1 Message Date
clawbot 203a4e13a5 Let the AS page's prefix lists fold (closes #155)
check / check (push) Waiting to run
The IPv4 and IPv6 prefix lists on the AS page are each a <details>
element. Its <summary> holds the list's heading and count, so a click on
the heading folds or unfolds the list without a script. Both lists start
open. The peer list is unchanged.

The summary keeps the look of the other section headings. It hides the
browser's own marker, which not every browser shows on a flex element,
and draws a triangle before the heading that points down while the list
is open.

Model: opus-5-5
2026-10-27 04:56:58 +01:00
clawbot 2375480e2e List an AS's prefixes by mask length, shortest first (closes #153)
check / check (push) Canceled after 0s
The AS page sorted its IPv4 and IPv6 prefix lists by network address,
then by mask length. It now sorts them by mask length, shortest first,
and by network address within one mask length, so a /16 comes before a
/24. The sort function is renamed from sortRoutesByNetwork to
sortRoutesByMaskLength to say so.

The handler of /api/v1/as/{asn} does not sort its lists, so it is
unchanged.

Model: opus-5-5
2026-10-27 04:37:17 +01:00
clawbot a5bff4e13c Show whether the status on / and /status is fresh (closes #152)
check / check (push) Canceled after 0s
Each page's script turns a one-character braille spinner next to the
status whenever a stats answer counts more messages than the one
before. Its colour cycles as rtnetmon's does. A separate check runs
every second. Once no such answer has come for three refresh periods,
15 seconds on / and 6 on /status, it shows "stale" and the seconds
since, in the page's disconnected colour. A failed or hung request
counts as no fresh answer. On / the card has no room beside
"Connected", so the indicator sits on the line under it. A test renders
both pages and checks the indicator follows the status element.

Model: opus-5-5
2026-10-27 04:24:53 +01:00
clawbot 73f53c8220 Take Brazilian AS details from NIC.br's WHOIS server (closes #174)
check / check (push) Canceled after 0s
LACNIC's answer for an AS that NIC.br registers is NIC.br's answer with
each accented letter replaced by U+FFFD, so a name such as AS52751's
cannot be recovered from it. When LACNIC's answer carries NIC.br's
copyright notice, the client now also asks whois.registro.br for the AS
and takes the fields from that answer, which is Latin-1 and decoded as
before. When that query fails, when the answer has no fields, or when
NIC.br says the client is over its query limit, LACNIC's fields are kept
and a warning is logged. After an over-limit answer the client skips
NIC.br for an hour. The tests use recorded answers for AS52751.

Model: opus-5-5
2026-10-27 03:59:11 +01:00
clawbot cd1b9b94a4 Stop asinfo-gen when the as.csv download breaks off (closes #177)
check / check (push) Canceled after 0s
parseCSV skipped a record on any read error and asked for the next one.
When the download of as.csv broke off, the response body returned the
same error on every read, so the generator logged it forever and make
asupdate never finished.

parseCSV now skips only a record that does not parse, a csv.ParseError.
Any other read error ends it with that error, so the generator exits
non-zero and script/asupdate keeps the old list.

Model: opus-5-5
2026-10-27 03:10:40 +01:00
clawbot 023c3f9b41 Keep the AS list when the asupdate generator fails (closes #173)
check / check (push) Canceled after 0s
make asupdate piped the generator into gzip, so the target saw gzip's
exit status. A failed download replaced pkg/asinfo/asdata.json.gz with
an empty file and the target still exited 0.

The target now runs the new script/asupdate. It writes the output of
cmd/asinfo-gen to a temporary file. Only when the generator succeeds
does it compress the list and move it into place. Otherwise it leaves
the file alone and exits non-zero. The generator now fails when as.csv
holds no AS.

The generator reads as.csv from ipverse/as-metadata, the source's new
name. The list it builds there is unchanged, so no new list is
committed.

Model: opus-5-5
2026-10-27 02:43:35 +01:00
clawbot 57469b6370 Read only the AS's objects in RIPE, APNIC and AFRINIC answers (closes #168)
check / check (push) Canceled after 0s
APNIC and RIPE start their answer for an AS with the as-block object of
the block the AS is in. The parser kept the first value of each field
across the whole answer, so it stored the block's dates as the AS's.

The parser now reads an answer from RIPE, APNIC or AFRINIC object by
object. It takes the fields from the aut-num object and the organisation
object it names, and leaves out every other object. The irt object's
address therefore no longer joins the organisation's. ARIN and LACNIC
answers are parsed as before.

Model: opus-5-5
2026-10-27 02:12:48 +01:00
clawbot 6d55d39737 Keep accented letters in AS names (closes #165)
check / check (push) Canceled after 0s
The built-in AS list had no letter outside ASCII, and make asupdate
rejected today's as.csv, which added a country code column.

The generator now checks only the first three header columns and reads
past the rest; the parsing moved into parseCSV. The list is regenerated.
The new source spells some names differently and no longer lists AS 0
or the documentation AS numbers, so the tests that relied on those
entries changed.

A WHOIS answer that is not valid UTF-8 is now decoded as Latin-1, which
RIPE and APNIC send, before it is parsed and stored. LACNIC answers in
UTF-8.

Model: opus-5-5
2026-10-27 01:50:34 +01:00
clawbot 86e1f20e11 Show ages on the AS and prefix pages in words (closes #163)
check / check (push) Canceled after 0s
The Age column printed Go's duration format for every plural age, such
as "5m0s ago" or "360h0m0s ago".

The templates now register humanize.Time from go-humanize, which the
server already uses, as the timeSince function, and the hand-written
function is gone. Ages read "5 minutes ago", "3 hours ago" or "2 weeks
ago". An age under a minute reads "now" or in seconds instead of "just
now". An age over a month reads in months or years instead of a date.

Model: opus-5-5
2026-10-27 01:23:50 +01:00
clawbot a1de67ebe1 Never move last seen back or first seen forward (closes #162)
check / check (push) Canceled after 0s
Some RIS collectors send updates days old while they catch up on a
backlog. Every update to a stored AS, peering or peer set last_seen to
the update's time, even an older one, and never lowered first_seen. An
AS page could then list a peer first seen after it was last seen.

The update statements for asns, peerings and bgp_peers now keep the
earlier of the stored and new first_seen and the later of the stored
and new last_seen, with MIN and MAX in the SQL. The batch and the
single-row writes share each statement. GetOrCreateASN returns the
times the row now holds. Route times are unchanged.

Model: opus-5-5
2026-10-27 01:10:01 +01:00
clawbot 355cd37496 Show route churn as a whole count per minute (closes #160)
check / check (push) Canceled after 0s
The status page showed route churn per second as a one-minute moving
average, so a quiet feed gave a fraction such as 0.1. The average was
correct.

The metrics tracker now also counts prefix announcements and
withdrawals, from all peers, in each clock minute. /api/v1/stats
reports the count for the last full minute as route_churn_per_min in
stream, in place of route_churn_per_sec. The status page shows it as
"Route Churn/min".

The status and home pages round messages and updates per second to
whole numbers.

Model: opus-5-5
2026-10-27 00:14:24 +01:00
clawbot 087b8de9ae Back off WHOIS lookups more slowly and stop asking for personal data (closes #159)
check / check (push) Canceled after 0s
A failed lookup now makes the fetcher's wait 1.5 times as long, up to
one minute instead of five. A success still halves it.

RIPE counts the person and role objects an answer holds against a daily
limit per IP address, and refuses an address for good that passes it
too often. RIPE, APNIC and AFRINIC are now asked with --no-personal,
which leaves those objects out but keeps the organisation object.

An answer with RIPE's access-denied or limit-reached error line is now
reported as a refusal holding that line, not as an answer with no
fields.

Model: opus-5-5
2026-10-26 23:40:13 +01:00
clawbot 99c7efc156 Make the home page's AS name search work (closes #151)
check / check (push) Canceled after 0s
The AS Name Search card on the home page said the search was coming soon.
It now asks the new GET /api/v1/search/as?q=<text> and lists the matches
as links to their AS pages. asinfo.Search ignores case, returns the first
matches up to a limit in AS number order, and counts all matches. An
empty q answers 400.

When the database has no record of an AS, its page and /api/v1/as/{asn}
take its handle and description from asinfo and still list its prefixes.
They answer 404 only for an AS with no record, no asinfo entry and no
prefix. AS 0 means no known origin: it always answers 404, and the search
never lists it.

Model: opus-5-5
2026-10-26 22:51:12 +01:00
clawbot 71ee2aeda7 Log the start-up checkpoint's figures under their own names (closes #117)
check / check (push) Canceled after 0s
Initialize scanned the row of PRAGMA wal_checkpoint(TRUNCATE), which is
the busy flag, the frames in the WAL and the frames copied, into
wal_pages, checkpointed and moved; and a completed TRUNCATE reads 0 for
both counts. The line now logs busy, wal_frames and checkpointed_frames,
plus wal_bytes, the WAL file's size, which New reads before it opens the
database because the open writes a frame to the WAL. The line is a
warning when busy is 1. Tests open a database a second time while the
first stays open, with and without a read open on the first, and check
the line's level and figures.

Model: opus-5-5
2026-10-20 04:32:08 +02:00
clawbot 8d16bf9ea6 Add the Rationale section to README (closes #121)
check / check (push) Canceled after 0s
REPO_POLICIES.md requires a Rationale section saying why the program
exists. RIS Live sends only the changes to the routing table, so it
cannot answer which AS an address is routed to now; routewatch applies
the changes to its own tables and answers from them. The section says
those tables hold only the routes announced since routewatch first
started, and miss the updates it drops under load or while it is
stopped.

Model: opus-5-5
2026-10-20 04:24:41 +02:00
33 changed files with 3218 additions and 353 deletions
+2 -3
View File
@@ -52,7 +52,6 @@ clean:
run: build run: build
DEBUG=routewatch ./bin/routewatch 2>&1 | tee log.txt DEBUG=routewatch ./bin/routewatch 2>&1 | tee log.txt
# Rebuild the AS list built into RouteWatch from as.csv (needs the network).
asupdate: asupdate:
@echo "Updating AS info data..." @script/asupdate
@go run cmd/asinfo-gen/main.go | gzip > pkg/asinfo/asdata.json.gz.tmp && \
mv pkg/asinfo/asdata.json.gz.tmp pkg/asinfo/asdata.json.gz
+54 -12
View File
@@ -5,6 +5,22 @@ table in real time: it streams BGP UPDATE messages from the RIPE RIS Live
service, maintains a live routing table in SQLite, and provides HTTP APIs for service, maintains a live routing table in SQLite, and provides HTTP APIs for
querying routing information. querying routing information.
## Rationale
RIPE RIS Live streams the BGP updates that RIPE's route collectors receive, as
they happen, from the moment a client connects. It reports changes, not the
routing table they add up to, so it cannot say which AS an IP address is routed
to now or which routes a prefix has. RouteWatch applies the feed's updates to
live routing tables in its own SQLite database and answers IP-to-AS lookups and
prefix pages from them.
The feed sends no table when a client connects, so these tables hold only the
routes the feed has announced since RouteWatch first started: an IP-to-AS lookup
finds nothing for an address whose routes have not been announced since then.
The tables also miss the updates RouteWatch drops when it falls behind under
load and those sent while it is stopped, so a lookup or a prefix page can show a
route the feed has withdrawn.
## Features ## Features
- Real-time streaming of BGP updates from RIPE RIS Live - Real-time streaming of BGP updates from RIPE RIS Live
@@ -40,15 +56,17 @@ Every route answers `HEAD` as it answers `GET`, without the body.
### Web Interface ### Web Interface
- `GET /` - Home page (HTML): headline figures, and lookups by AS number and by - `GET /` - Home page (HTML): headline figures, lookups by AS number and by IP
IP address address, and a search of AS names
- `GET /status` - HTML status dashboard, which reads `/api/v1/stats` - `GET /status` - HTML status dashboard, which reads `/api/v1/stats`
- `GET /status.json` - JSON statistics: feed, memory, database and route figures - `GET /status.json` - JSON statistics: feed, memory, database and route figures
- `GET /.well-known/healthcheck.json` - Health check (JSON): 200 when the - `GET /.well-known/healthcheck.json` - Health check (JSON): 200 when the
database answers and the RIS Live feed is connected, otherwise 503; database answers and the RIS Live feed is connected, otherwise 503;
`data.status` is `ok` or `error` and `data.checks` says which check failed `data.status` is `ok` or `error` and `data.checks` says which check failed
- `GET /as/{asn}` - AS detail page (HTML): the prefixes the AS originates and - `GET /as/{asn}` - AS detail page (HTML): the prefixes the AS originates and
its peers its peers. When the database has no record of the AS, its handle and
description come from the AS list built into RouteWatch. The page then shows
no first-seen date.
- `GET /prefix/{prefix}/{length}` - Prefix detail page (HTML), such as - `GET /prefix/{prefix}/{length}` - Prefix detail page (HTML), such as
`/prefix/192.0.2.0/24`: every live route for the prefix, one per peer, and its `/prefix/192.0.2.0/24`: every live route for the prefix, one per peer, and its
origin ASes origin ASes
@@ -63,7 +81,9 @@ Every route answers `HEAD` as it answers `GET`, without the body.
### API v1 ### API v1
- `GET /api/v1/stats` - The `/status.json` figures plus connection, garbage - `GET /api/v1/stats` - The `/status.json` figures plus connection, garbage
collector, stream and handler queue statistics collector, stream and handler queue statistics. `stream.route_churn_per_min`
is the number of prefix announcements and withdrawals, from all peers, in the
last full clock minute.
- `GET /api/v1/ip/{ip}` - Looks up each address: its most specific live route - `GET /api/v1/ip/{ip}` - Looks up each address: its most specific live route
(`netblock`), that route's origin AS with its WHOIS details, and the address's (`netblock`), that route's origin AS with its WHOIS details, and the address's
PTR names. `{ip}` is an IP address, a host name, or a comma-separated list of PTR names. `{ip}` is an IP address, a host name, or a comma-separated list of
@@ -77,9 +97,23 @@ Every route answers `HEAD` as it answers `GET`, without the body.
and 404 when none of the addresses has a live route. A PTR lookup gets 2 and 404 when none of the addresses has a live route. A PTR lookup gets 2
seconds; an address whose lookup takes longer is answered without PTR names. seconds; an address whose lookup takes longer is answered without PTR names.
- `GET /api/v1/as/{asn}` - The AS and the prefixes it originates, IPv4 and IPv6 - `GET /api/v1/as/{asn}` - The AS and the prefixes it originates, IPv4 and IPv6
apart, each with when its newest route was updated apart, each with when its newest route was updated. When the database has no
record of the AS, its handle and description come from the AS list built into
RouteWatch. The answer then has no `first_seen` or `last_seen`.
- `GET /api/v1/prefix/{prefix}/{length}` - Every live route for the prefix, one - `GET /api/v1/prefix/{prefix}/{length}` - Every live route for the prefix, one
per peer, and the same routes grouped by origin AS in `origins` per peer, and the same routes grouped by origin AS in `origins`
- `GET /api/v1/search/as?q={text}` - Finds the ASes whose handle or description
contains `{text}`, ignoring case. `data` holds:
- `query`: `{text}` without its leading and trailing spaces
- `total`: the number of ASes that match
- `results`: the first 50 of them in AS number order, each with `asn`,
`handle` and `description`
The search reads the AS list built into RouteWatch, not the database. The
list comes from
[ipverse/as-metadata](https://github.com/ipverse/as-metadata). A match can
therefore be an AS that the database has no record of. A `q` that is
missing, empty or only spaces answers 400.
A JSON answer is `{"status": "ok", "data": ...}`, or for an error A JSON answer is `{"status": "ok", "data": ...}`, or for an error
`{"status": "error", "error": {"msg": ..., "code": ...}}`, and carries an `{"status": "error", "error": {"msg": ..., "code": ...}}`, and carries an
@@ -90,13 +124,16 @@ took. The health check is the exception: its 503 answer is also
A request that runs past its time limit answers 504: the limit is 30 seconds, A request that runs past its time limit answers 504: the limit is 30 seconds,
and 4 seconds for the database reads behind `/status.json` and `/api/v1/stats`. and 4 seconds for the database reads behind `/status.json` and `/api/v1/stats`.
An AS number, prefix or mask length that does not parse answers 400, as does a An AS number, prefix or mask length that does not parse answers 400, as does a
mask length out of range for its IP version. An AS the database does not hold, mask length out of range for its IP version. An AS answers 404 when it is not in
or a prefix with no live route, answers 404. The HTML pages give these errors as the AS list built into RouteWatch and the database has neither a record of it
plain text. Under `/api/v1` and `/ip/`, a path no route matches answers 404 with nor a live route it originates. AS 0 always answers 404, because it means no
the message `Not found`, and `POST`, `PUT`, `PATCH`, `DELETE`, `OPTIONS`, known origin. The AS name search never lists it. A prefix with no live route
`TRACE` or `CONNECT` answers 405 with `Method not allowed`. A 500 answer says answers 404 too. The HTML pages give these errors as plain text. Under `/api/v1`
only `Internal server error`, and a host name that does not resolve only and `/ip/`, a path no route matches answers 404 with the message `Not found`,
`could not resolve host name`; the detail goes to the log. and `POST`, `PUT`, `PATCH`, `DELETE`, `OPTIONS`, `TRACE` or `CONNECT` answers
405 with `Method not allowed`. A 500 answer says only `Internal server error`,
and a host name that does not resolve only `could not resolve host name`; the
detail goes to the log.
Every answer carries `Strict-Transport-Security`, `Content-Security-Policy`, Every answer carries `Strict-Transport-Security`, `Content-Security-Policy`,
`X-Frame-Options`, `X-Content-Type-Options`, `Referrer-Policy` and `X-Frame-Options`, `X-Content-Type-Options`, `Referrer-Policy` and
@@ -449,6 +486,11 @@ provide:
or `go.sum`) and `go fmt` (writes), then `script/check` or `go.sum`) and `go fmt` (writes), then `script/check`
- `script/install-precommit` — install the git pre-commit hook that runs - `script/install-precommit` — install the git pre-commit hook that runs
`script/precommit` `script/precommit`
- `script/asupdate` — rebuild the AS list built into RouteWatch,
`pkg/asinfo/asdata.json.gz`, from the `as.csv` file of
[ipverse/as-metadata](https://github.com/ipverse/as-metadata). It needs the
network. It runs `cmd/asinfo-gen`, which fails when it finds no AS. When
`cmd/asinfo-gen` fails, the old list stays and the script exits non-zero.
Tests and the linter run only in Docker, as the `test` and `lint` phases of the Tests and the linter run only in Docker, as the `test` and `lint` phases of the
`Dockerfile`; the image cannot be built unless both pass. Every `docker build` `Dockerfile`; the image cannot be built unless both pass. Every `docker build`
+55
View File
@@ -33,6 +33,61 @@ tested.
# Completed Steps # Completed Steps
- 2026-10-27: on the AS page, a click on the heading of the IPv4 or the IPv6
prefix list folds or unfolds the list. Both lists start open (closes #155)
- 2026-10-27: the AS page lists its IPv4 and its IPv6 prefixes by mask length,
shortest first, and prefixes of one mask length by address. They used to be
listed by address (closes #153)
- 2026-10-27: on `/` and `/status`, a spinner beside the status turns one frame
each time the stats show more messages than the time before. A page that has
not seen more messages for three refreshes shows "stale" and the time since
then instead, in red (closes #152)
- 2026-10-27: the WHOIS details of a Brazilian AS keep their accented letters.
LACNIC's answer for such an AS comes from NIC.br, Brazil's registry, with each
accented letter replaced. The details now come from NIC.br's own server,
`whois.registro.br`. When that query fails, the details from LACNIC's answer
are kept and a warning is logged. After NIC.br answers that RouteWatch is over
its query limit, RouteWatch stops asking NIC.br for an hour (closes #174)
- 2026-10-27: `make asupdate` stops with an error when the download of `as.csv`
breaks off. The generator used to retry the failed read forever. A record that
does not parse is still logged and skipped (closes #177)
- 2026-10-27: `make asupdate` runs the new `script/asupdate`. It replaces the AS
list built into RouteWatch only when the generator succeeds and finds at least
one AS. A failed download used to replace the list with an empty file. The
generator reads `as.csv` from `ipverse/as-metadata`, the source's new name
(closes #173)
- 2026-10-27: the WHOIS details of an AS at RIPE, APNIC or AFRINIC come only
from its `aut-num` object and the organisation object it names. RIPE and APNIC
start their answer with the `as-block` object of the AS's block. Its dates
were stored as the AS's (closes #168)
- 2026-10-27: AS names keep their accented letters. The AS list built into
RouteWatch is rebuilt from today's `as.csv`, which added a country code
column. That file no longer lists AS 0 or the documentation AS numbers. A
WHOIS answer that is not valid UTF-8 is decoded as Latin-1, which RIPE and
APNIC send, before it is parsed and stored (closes #165)
- 2026-10-27: the Age column on the AS and prefix pages reads in words, such as
"5 minutes ago" or "2 weeks ago", instead of Go durations such as "360h0m0s
ago". An age under a minute reads in seconds, and one over a month in months
or years instead of a date (closes #163)
- 2026-10-26: a write to a stored AS, peering or peer keeps the earlier of the
two first-seen times and the later of the two last-seen times. An old update
from a collector catching up on a backlog no longer moves last seen back
(closes #162)
- 2026-10-26: the status page shows "Route Churn/min", the number of prefix
announcements and withdrawals, from all peers, in the last full clock minute.
`/api/v1/stats` reports it as `route_churn_per_min` in `stream`, which
replaces the moving average `route_churn_per_sec`. The messages and updates
per second on `/` and `/status` are whole numbers (closes #160)
- 2026-10-26: the WHOIS fetcher backs off more slowly. A failure makes its wait
1.5 times as long, up to one minute instead of five. Its queries to RIPE,
APNIC and AFRINIC leave out person and role objects, which RIPE counts against
a daily limit per address. A refused query is logged with the RIR's error line
(closes #159)
- 2026-10-26: the home page's AS name search lists the ASes whose handle or
description holds the text, as links to their pages, from the new
`/api/v1/search/as`; the search ignores case. When the database has no record
of an AS, its page takes the handle and description from the AS list built
into RouteWatch and still lists the AS's prefixes (closes #151)
- 2026-10-19: README "HTTP Endpoints" lists every route with what it answers, - 2026-10-19: README "HTTP Endpoints" lists every route with what it answers,
"Database Schema" matches `schema.sql`, and the rest of the README matches the "Database Schema" matches `schema.sql`, and the rest of the README matches the
code; this file brought up to date (closes #72) code; this file brought up to date (closes #72)
+60 -28
View File
@@ -6,6 +6,7 @@ import (
"encoding/csv" "encoding/csv"
"encoding/json" "encoding/json"
"errors" "errors"
"fmt"
"io" "io"
"log" "log"
"net/http" "net/http"
@@ -14,7 +15,21 @@ import (
"strings" "strings"
) )
const asCsvURL = "https://github.com/ipverse/asn-info/raw/refs/heads/master/as.csv" const asCsvURL = "https://github.com/ipverse/as-metadata/raw/refs/heads/master/as.csv"
// The first columns of as.csv, the ones the generator reads. It reads past
// the columns after them, such as the country code.
const (
asnColumn = "asn"
handleColumn = "handle"
descriptionColumn = "description"
columnsRead = 3
)
var (
errUnexpectedHeader = errors.New("unexpected CSV header")
errNoASes = errors.New("no AS in the CSV")
)
// ASInfo represents information about an Autonomous System // ASInfo represents information about an Autonomous System
type ASInfo struct { type ASInfo struct {
@@ -45,40 +60,66 @@ func main() {
log.Fatalf("Failed to fetch CSV: HTTP %d", resp.StatusCode) log.Fatalf("Failed to fetch CSV: HTTP %d", resp.StatusCode)
} }
// Parse CSV asInfos, err := parseCSV(resp.Body)
reader := csv.NewReader(resp.Body)
// Closed here rather than deferred: log.Fatalf exits without running
// deferred calls.
_ = resp.Body.Close()
if err != nil {
log.Fatalf("Failed to parse CSV: %v", err)
}
log.Printf("Parsed %d AS records", len(asInfos))
// Convert to JSON and output to stdout
encoder := json.NewEncoder(os.Stdout)
encoder.SetIndent("", "\t")
err = encoder.Encode(asInfos)
if err != nil {
log.Fatalf("Failed to encode JSON: %v", err)
}
}
// parseCSV reads as.csv from r and returns its ASes, with the handle and the
// description as the file spells them. A header whose first columns are not
// the AS number, the handle and the description is an error. A record that
// does not parse is logged and skipped. A read that fails, such as on a broken
// download, is an error. A file left with no AS is an error.
func parseCSV(r io.Reader) ([]ASInfo, error) {
reader := csv.NewReader(r)
// Read header
header, err := reader.Read() header, err := reader.Read()
if err != nil { if err != nil {
log.Fatalf("Failed to read CSV header: %v", err) return nil, fmt.Errorf("read CSV header: %w", err)
} }
if len(header) != 3 || header[0] != "asn" || header[1] != "handle" || if len(header) < columnsRead || header[0] != asnColumn ||
header[2] != "description" { header[1] != handleColumn || header[2] != descriptionColumn {
log.Fatalf("Unexpected CSV header: %v", header) return nil, fmt.Errorf("%w: %v", errUnexpectedHeader, header)
} }
// Read all records
var asInfos []ASInfo var asInfos []ASInfo
for { for {
// The reader returns an error for a record with a different number
// of fields than the header.
record, err := reader.Read() record, err := reader.Read()
if errors.Is(err, io.EOF) { if errors.Is(err, io.EOF) {
break break
} }
if err != nil { if _, ok := errors.AsType[*csv.ParseError](err); ok {
log.Printf("Error reading CSV record: %v", err) log.Printf("Error reading CSV record: %v", err)
continue continue
} }
const expectedFields = 3 // Any other error comes from the download, such as a broken
if len(record) != expectedFields { // connection. Every later read returns it again.
log.Printf("Skipping invalid record: %v", record) if err != nil {
return nil, fmt.Errorf("read CSV record: %w", err)
continue
} }
asn, err := strconv.Atoi(record[0]) asn, err := strconv.Atoi(record[0])
@@ -95,18 +136,9 @@ func main() {
}) })
} }
// Closed here rather than deferred: log.Fatalf exits without running if len(asInfos) == 0 {
// deferred calls. return nil, errNoASes
_ = resp.Body.Close()
log.Printf("Parsed %d AS records", len(asInfos))
// Convert to JSON and output to stdout
encoder := json.NewEncoder(os.Stdout)
encoder.SetIndent("", "\t")
err = encoder.Encode(asInfos)
if err != nil {
log.Fatalf("Failed to encode JSON: %v", err)
} }
return asInfos, nil
} }
+142
View File
@@ -0,0 +1,142 @@
package main
import (
"errors"
"io"
"slices"
"strings"
"testing"
"testing/iotest"
"time"
)
// parseTimeLimit is how long a test waits for parseCSV to return before it
// fails.
const parseTimeLimit = 2 * time.Second
// TestParseCSVReadsTodaysLayout parses an as.csv with the country code column
// that the source added after the description. It expects each AS with its
// handle and description spelled exactly as in the file, accents included.
func TestParseCSVReadsTodaysLayout(t *testing.T) {
t.Parallel()
const asCSV = `asn,handle,description,country-code
1,LVLT-1,Level 3 Parent LLC,US
1251,FUNDACAO-AMPARO-PESQUISA,FUNDAÇÃO DE AMPARO À PESQUISA DO ESTADO SÃO PAULO,BR
`
got, err := parseCSV(strings.NewReader(asCSV))
if err != nil {
t.Fatalf("parseCSV: %v", err)
}
want := []ASInfo{
{ASN: 1, Handle: "LVLT-1", Description: "Level 3 Parent LLC"},
{
ASN: 1251,
Handle: "FUNDACAO-AMPARO-PESQUISA",
Description: "FUNDAÇÃO DE AMPARO À PESQUISA DO ESTADO " +
"SÃO PAULO",
},
}
if !slices.Equal(got, want) {
t.Errorf("parseCSV = %+v, want %+v", got, want)
}
}
// TestParseCSVRejectsAnotherHeader parses files whose first columns are not
// the AS number, the handle and the description. It expects an error rather
// than a list read from the wrong columns.
func TestParseCSVRejectsAnotherHeader(t *testing.T) {
t.Parallel()
for _, asCSV := range []string{
"asn,handle\n1,LVLT-1\n",
"asn,description,handle\n1,Level 3 Parent LLC,LVLT-1\n",
} {
_, err := parseCSV(strings.NewReader(asCSV))
if err == nil {
t.Errorf("parseCSV(%q): no error", asCSV)
}
}
}
// TestParseCSVRejectsAFileWithNoASes parses files that hold no AS, one with
// only the header and one whose only record has no valid AS number. It
// expects errNoASes, so that make asupdate keeps the old list.
func TestParseCSVRejectsAFileWithNoASes(t *testing.T) {
t.Parallel()
for _, asCSV := range []string{
"asn,handle,description,country-code\n",
"asn,handle,description,country-code\nAS1,LVLT-1,Level 3,US\n",
} {
_, err := parseCSV(strings.NewReader(asCSV))
if !errors.Is(err, errNoASes) {
t.Errorf("parseCSV(%q) = %v, want %v", asCSV, err, errNoASes)
}
}
}
// TestParseCSVSkipsARecordThatDoesNotParse parses an as.csv with a record
// that has too few fields between two valid records. It expects both valid
// ASes and no error, so that one bad record does not stop make asupdate.
func TestParseCSVSkipsARecordThatDoesNotParse(t *testing.T) {
t.Parallel()
const asCSV = "asn,handle,description,country-code\n" +
"1,LVLT-1,Level 3 Parent LLC,US\n" +
"2,UDEL-DCN\n" +
"3,MIT-GATEWAYS,Massachusetts Institute of Technology,US\n"
got, err := parseCSV(strings.NewReader(asCSV))
if err != nil {
t.Fatalf("parseCSV: %v", err)
}
want := []ASInfo{
{ASN: 1, Handle: "LVLT-1", Description: "Level 3 Parent LLC"},
{
ASN: 3,
Handle: "MIT-GATEWAYS",
Description: "Massachusetts Institute of Technology",
},
}
if !slices.Equal(got, want) {
t.Errorf("parseCSV = %+v, want %+v", got, want)
}
}
// TestParseCSVStopsWhenTheDownloadBreaksOff parses an as.csv whose download
// breaks off after the first record. From then on every read fails, as on a
// broken connection. It expects parseCSV to return that error, so that make
// asupdate keeps the old list.
func TestParseCSVStopsWhenTheDownloadBreaksOff(t *testing.T) {
t.Parallel()
const asCSV = "asn,handle,description,country-code\n" +
"1,LVLT-1,Level 3 Parent LLC,US\n"
download := io.MultiReader(
strings.NewReader(asCSV), iotest.ErrReader(io.ErrUnexpectedEOF))
// parseCSV runs on its own goroutine, so that a parseCSV that never
// returns fails the test instead of hanging it.
parsed := make(chan error, 1)
go func() {
_, err := parseCSV(download)
parsed <- err
}()
select {
case err := <-parsed:
if !errors.Is(err, io.ErrUnexpectedEOF) {
t.Errorf("parseCSV = %v, want %v", err, io.ErrUnexpectedEOF)
}
case <-time.After(parseTimeLimit):
t.Fatalf("parseCSV did not return within %v", parseTimeLimit)
}
}
+1
View File
@@ -10,6 +10,7 @@ require (
github.com/rcrowley/go-metrics v0.0.0-20250401214520-65e299d6c5c9 github.com/rcrowley/go-metrics v0.0.0-20250401214520-65e299d6c5c9
go.uber.org/fx v1.24.0 go.uber.org/fx v1.24.0
golang.org/x/term v0.33.0 golang.org/x/term v0.33.0
golang.org/x/text v0.43.0
) )
require ( require (
+2
View File
@@ -28,5 +28,7 @@ golang.org/x/sys v0.34.0 h1:H5Y5sJ2L2JRdyv7ROF1he/lPdvFsd0mJHFw2ThKHxLA=
golang.org/x/sys v0.34.0/go.mod h1:BJP2sWEmIv4KK5OTEluFJCKSidICx8ciO85XgH3Ak8k= golang.org/x/sys v0.34.0/go.mod h1:BJP2sWEmIv4KK5OTEluFJCKSidICx8ciO85XgH3Ak8k=
golang.org/x/term v0.33.0 h1:NuFncQrRcaRvVmgRkvM3j/F00gWIAlcmlB8ACEKmGIg= golang.org/x/term v0.33.0 h1:NuFncQrRcaRvVmgRkvM3j/F00gWIAlcmlB8ACEKmGIg=
golang.org/x/term v0.33.0/go.mod h1:s18+ql9tYWp1IfpV9DmCtQDDSRBUjKaw9M1eAv5UeF0= golang.org/x/term v0.33.0/go.mod h1:s18+ql9tYWp1IfpV9DmCtQDDSRBUjKaw9M1eAv5UeF0=
golang.org/x/text v0.43.0 h1:1QivrlhwAsnMOcqgOdl9my7s9OyZl8O5/UGUTJVXoWc=
golang.org/x/text v0.43.0/go.mod h1:hbSIYA/amXcRXTFSZqh8tWHEoqRvvbKPJ2uz/KU+a+s=
gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA= gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA=
gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM= gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
+122 -42
View File
@@ -21,6 +21,7 @@ import (
"git.eeqj.de/sneak/routewatch/internal/config" "git.eeqj.de/sneak/routewatch/internal/config"
"git.eeqj.de/sneak/routewatch/internal/logger" "git.eeqj.de/sneak/routewatch/internal/logger"
"git.eeqj.de/sneak/routewatch/internal/ristypes"
"git.eeqj.de/sneak/routewatch/pkg/asinfo" "git.eeqj.de/sneak/routewatch/pkg/asinfo"
_ "github.com/mattn/go-sqlite3" // CGO SQLite driver _ "github.com/mattn/go-sqlite3" // CGO SQLite driver
) )
@@ -169,6 +170,20 @@ func New(cfg *config.Config, logger *logger.Logger) (*Database, error) {
return nil, fmt.Errorf("failed to create database directory: %w", err) return nil, fmt.Errorf("failed to create database directory: %w", err)
} }
// The size of the WAL file the previous run left, which Initialize logs
// with the start-up checkpoint. It is read before the database is opened,
// because opening it writes a frame to the WAL. There is no WAL file when
// the database is new or the last run closed the database, which deletes
// it.
var walBytes int64
wal, err := os.Stat(dbPath + walFileSuffix)
if err == nil {
walBytes = wal.Size()
} else if !errors.Is(err, os.ErrNotExist) {
logger.Warn("Failed to read the WAL file's size on startup", "error", err)
}
// Per-connection SQLite settings go in the DSN so every pooled connection // Per-connection SQLite settings go in the DSN so every pooled connection
// gets them, not just the one that runs the Initialize pragmas. _txlock= // gets them, not just the one that runs the Initialize pragmas. _txlock=
// immediate makes every transaction take the write lock at BEGIN. Without it // immediate makes every transaction take the write lock at BEGIN. Without it
@@ -229,7 +244,7 @@ func New(cfg *config.Config, logger *logger.Logger) (*Database, error) {
counts: &liveCounts{}, counts: &liveCounts{},
} }
err = database.Initialize() err = database.Initialize(walBytes)
if err != nil { if err != nil {
return nil, fmt.Errorf("failed to initialize database: %w", err) return nil, fmt.Errorf("failed to initialize database: %w", err)
} }
@@ -245,8 +260,16 @@ func New(cfg *config.Config, logger *logger.Logger) (*Database, error) {
return database, nil return database, nil
} }
// Initialize creates the database schema if it doesn't exist. // The messages Initialize logs the start-up WAL checkpoint with.
func (d *Database) Initialize() error { const (
startupCheckpointMessage = "WAL checkpoint on startup"
startupCheckpointIncompleteMessage = "WAL checkpoint on startup incomplete"
)
// Initialize creates the database schema if it doesn't exist. walBytes is the
// size of the WAL file the previous run left, which New reads before it opens
// the database.
func (d *Database) Initialize(walBytes int64) error {
// Set SQLite pragmas for performance. Per-connection settings (cache_size, // Set SQLite pragmas for performance. Per-connection settings (cache_size,
// synchronous, busy_timeout, journal_mode) live in the DSN, and New sets the // synchronous, busy_timeout, journal_mode) live in the DSN, and New sets the
// writing connection's cache_size; temp_store is left at its default so // writing connection's cache_size; temp_store is left at its default so
@@ -266,19 +289,33 @@ func (d *Database) Initialize() error {
} }
} }
// Run WAL checkpoint on startup to consolidate any existing WAL // The start-up checkpoint copies the WAL a previous run left into the
var walPages, checkpointedPages, movedPages int // database file and empties the WAL file; its frame counts then read 0,
// so walBytes is logged with them.
var busy, walFrames, checkpointedFrames int
err := d.writer.QueryRowContext(context.Background(), err := d.writer.QueryRowContext(context.Background(),
"PRAGMA wal_checkpoint(TRUNCATE)", "PRAGMA wal_checkpoint(TRUNCATE)",
).Scan(&walPages, &checkpointedPages, &movedPages) ).Scan(&busy, &walFrames, &checkpointedFrames)
if err != nil {
switch {
case err != nil:
d.logger.Warn("Failed to checkpoint WAL on startup", "error", err) d.logger.Warn("Failed to checkpoint WAL on startup", "error", err)
} else { case busy != 0:
d.logger.Info("WAL checkpoint on startup", // Another process using the database kept the checkpoint from
"wal_pages", walPages, // copying or emptying all of the WAL.
"checkpointed", checkpointedPages, d.logger.Warn(startupCheckpointIncompleteMessage,
"moved", movedPages, "wal_bytes", walBytes,
"busy", busy,
"wal_frames", walFrames,
"checkpointed_frames", checkpointedFrames,
)
default:
d.logger.Info(startupCheckpointMessage,
"wal_bytes", walBytes,
"busy", busy,
"wal_frames", walFrames,
"checkpointed_frames", checkpointedFrames,
) )
} }
@@ -908,6 +945,26 @@ func (d *Database) UpdatePrefixesBatch(prefixes map[string]time.Time) error {
return nil return nil
} }
// These statements update a stored AS, peering or peer. Each keeps the earlier
// of the stored and the new first_seen, and the later of the stored and the new
// last_seen. Some collectors send updates days old while they catch up on a
// backlog. MIN and MAX compare the times as text, which is time order here: the
// driver writes every time in one format, and every time is UTC.
const (
updateASNSQL = `UPDATE asns
SET first_seen = MIN(first_seen, ?), last_seen = MAX(last_seen, ?)
WHERE asn = ?`
updatePeeringSQL = `UPDATE peerings
SET first_seen = MIN(first_seen, ?), last_seen = MAX(last_seen, ?)
WHERE as_a = ? AND as_b = ?`
updatePeerSQL = `UPDATE bgp_peers
SET peer_asn = ?, last_message_type = ?,
first_seen = MIN(first_seen, ?), last_seen = MAX(last_seen, ?)
WHERE peer_ip = ?`
)
// newASN returns a new ASN record first seen at seen.First and last seen at // newASN returns a new ASN record first seen at seen.First and last seen at
// seen.Last, with the handle and description from the AS info lookup when it // seen.Last, with the handle and description from the AS info lookup when it
// knows the number. // knows the number.
@@ -927,9 +984,8 @@ func newASN(number int, seen SeenTimes) ASN {
return asn return asn
} }
// upsertASN sets the last_seen time of ASN number to seen.Last with the given // upsertASN stores the seen times of ASN number with the given statements,
// statements, inserting the ASN when it is new. It reports whether it inserted // inserting the ASN when it is new. It reports whether it inserted it.
// it.
func upsertASN( func upsertASN(
ctx context.Context, selectStmt, updateStmt, insertStmt *sql.Stmt, ctx context.Context, selectStmt, updateStmt, insertStmt *sql.Stmt,
number int, seen SeenTimes, number int, seen SeenTimes,
@@ -942,8 +998,8 @@ func upsertASN(
err := selectStmt.QueryRowContext(ctx, number).Scan(&asn.ASN, &handle, err := selectStmt.QueryRowContext(ctx, number).Scan(&asn.ASN, &handle,
&description, &asn.FirstSeen, &asn.LastSeen) &description, &asn.FirstSeen, &asn.LastSeen)
if err == nil { if err == nil {
// ASN exists, update last_seen // ASN exists, update its seen times
_, err = updateStmt.ExecContext(ctx, seen.Last, number) _, err = updateStmt.ExecContext(ctx, seen.First, seen.Last, number)
if err != nil { if err != nil {
return false, fmt.Errorf("failed to update ASN %d: %w", number, err) return false, fmt.Errorf("failed to update ASN %d: %w", number, err)
} }
@@ -993,8 +1049,7 @@ func (d *Database) GetOrCreateASNBatch(asns map[int]SeenTimes) error {
} }
defer func() { _ = selectStmt.Close() }() defer func() { _ = selectStmt.Close() }()
updateStmt, err := tx.PrepareContext(ctx, updateStmt, err := tx.PrepareContext(ctx, updateASNSQL)
"UPDATE asns SET last_seen = ? WHERE asn = ?")
if err != nil { if err != nil {
return fmt.Errorf("failed to prepare update statement: %w", err) return fmt.Errorf("failed to prepare update statement: %w", err)
} }
@@ -1053,17 +1108,19 @@ func (d *Database) GetOrCreateASN(number int, timestamp time.Time) (*ASN, error)
"FROM asns WHERE asn = ?", number). "FROM asns WHERE asn = ?", number).
Scan(&asn.ASN, &handle, &description, &asn.FirstSeen, &asn.LastSeen) Scan(&asn.ASN, &handle, &description, &asn.FirstSeen, &asn.LastSeen)
if err == nil { if err == nil {
// ASN exists, update last_seen // ASN exists, update its seen times
asn.Handle = handle.String asn.Handle = handle.String
asn.Description = description.String asn.Description = description.String
_, err = tx.Exec("UPDATE asns SET last_seen = ? WHERE asn = ?", _, err = tx.Exec(updateASNSQL, timestamp, timestamp, number)
timestamp, number)
if err != nil { if err != nil {
return nil, err return nil, err
} }
asn.LastSeen = timestamp // Return the seen times the update stored.
seen := SeenTimes{First: asn.FirstSeen, Last: asn.LastSeen}
seen.Add(timestamp)
asn.FirstSeen, asn.LastSeen = seen.First, seen.Last
err = tx.Commit() err = tx.Commit()
if err != nil { if err != nil {
@@ -1222,9 +1279,7 @@ func (d *Database) RecordPeering(asA, asB int, seen SeenTimes) error {
} }
if exists { if exists {
_, err = tx.Exec( _, err = tx.Exec(updatePeeringSQL, seen.First, seen.Last, asA, asB)
"UPDATE peerings SET last_seen = ? WHERE as_a = ? AND as_b = ?",
seen.Last, asA, asB)
} else { } else {
_, err = tx.Exec(` _, err = tx.Exec(`
INSERT INTO peerings (id, as_a, as_b, first_seen, last_seen) INSERT INTO peerings (id, as_a, as_b, first_seen, last_seen)
@@ -1279,9 +1334,7 @@ func (d *Database) UpdatePeerBatch(peers map[string]PeerUpdate) error {
} }
defer func() { _ = checkStmt.Close() }() defer func() { _ = checkStmt.Close() }()
updateStmt, err := tx.PrepareContext(ctx, updateStmt, err := tx.PrepareContext(ctx, updatePeerSQL)
"UPDATE bgp_peers SET peer_asn = ?, last_seen = ?, "+
"last_message_type = ? WHERE peer_ip = ?")
if err != nil { if err != nil {
return fmt.Errorf("failed to prepare update statement: %w", err) return fmt.Errorf("failed to prepare update statement: %w", err)
} }
@@ -1307,7 +1360,8 @@ func (d *Database) UpdatePeerBatch(peers map[string]PeerUpdate) error {
if exists { if exists {
_, err = updateStmt.ExecContext(ctx, update.PeerASN, _, err = updateStmt.ExecContext(ctx, update.PeerASN,
update.Seen.Last, update.MessageType, update.PeerIP) update.MessageType, update.Seen.First, update.Seen.Last,
update.PeerIP)
} else { } else {
_, err = insertStmt.ExecContext(ctx, _, err = insertStmt.ExecContext(ctx,
generateUUID().String(), update.PeerIP, update.PeerASN, generateUUID().String(), update.PeerIP, update.PeerASN,
@@ -1356,11 +1410,8 @@ func (d *Database) UpdatePeer(
} }
if exists { if exists {
_, err = tx.Exec( _, err = tx.Exec(updatePeerSQL,
"UPDATE bgp_peers SET peer_asn = ?, last_seen = ?, "+ peerASN, messageType, timestamp, timestamp, peerIP)
"last_message_type = ? WHERE peer_ip = ?",
peerASN, timestamp, messageType, peerIP,
)
} else { } else {
_, err = tx.Exec( _, err = tx.Exec(
"INSERT INTO bgp_peers (id, peer_ip, peer_asn, first_seen, "+ "INSERT INTO bgp_peers (id, peer_ip, peer_asn, first_seen, "+
@@ -1712,9 +1763,21 @@ func (d *Database) GetASDetails(asn int) (*ASN, []LiveRoute, error) {
// GetASDetailsContext returns the AS and the prefixes it originates, the IPv4 // GetASDetailsContext returns the AS and the prefixes it originates, the IPv4
// ones first, each once with the time of its newest route. It makes two index // ones first, each once with the time of its newest route. It makes two index
// searches per prefix, however many peers carry the prefix. // searches per prefix, however many peers carry the prefix.
//
// The routes of an AS can be stored before its record in the asns table. When
// the AS has no record, its handle and description come from the AS info
// lookup, and it has no seen times. An AS with no record, no lookup entry and
// no prefix is ErrNoRoute.
//
// ristypes.NoOrigin means no known origin, not an AS, so it is ErrNoRoute
// whatever is stored.
func (d *Database) GetASDetailsContext( func (d *Database) GetASDetailsContext(
ctx context.Context, asn int, ctx context.Context, asn int,
) (*ASN, []LiveRoute, error) { ) (*ASN, []LiveRoute, error) {
if asn == ristypes.NoOrigin {
return nil, nil, fmt.Errorf("%w: AS%d", ErrNoRoute, asn)
}
// Get AS information // Get AS information
var ( var (
asnInfo ASN asnInfo ASN
@@ -1727,17 +1790,28 @@ func (d *Database) GetASDetailsContext(
asn, asn,
).Scan(&asnInfo.ASN, &handle, &description, &asnInfo.FirstSeen, ).Scan(&asnInfo.ASN, &handle, &description, &asnInfo.FirstSeen,
&asnInfo.LastSeen) &asnInfo.LastSeen)
if err != nil {
if errors.Is(err, sql.ErrNoRows) { // known is whether the asns table or the AS info lookup holds the AS.
return nil, nil, fmt.Errorf("%w: AS%d", ErrNoRoute, asn) known := true
switch {
case errors.Is(err, sql.ErrNoRows):
asnInfo.ASN = asn
info, ok := asinfo.Get(asn)
if ok {
asnInfo.Handle = info.Handle
asnInfo.Description = info.Description
} }
known = ok
case err != nil:
return nil, nil, fmt.Errorf("failed to query AS: %w", err) return nil, nil, fmt.Errorf("failed to query AS: %w", err)
default:
asnInfo.Handle = handle.String
asnInfo.Description = description.String
} }
asnInfo.Handle = handle.String
asnInfo.Description = description.String
allPrefixes, err := d.originPrefixes(ctx, asn, ipVersionV4) allPrefixes, err := d.originPrefixes(ctx, asn, ipVersionV4)
if err != nil { if err != nil {
return &asnInfo, nil, err return &asnInfo, nil, err
@@ -1748,7 +1822,13 @@ func (d *Database) GetASDetailsContext(
return &asnInfo, allPrefixes, err return &asnInfo, allPrefixes, err
} }
return &asnInfo, append(allPrefixes, prefixesV6...), nil allPrefixes = append(allPrefixes, prefixesV6...)
if !known && len(allPrefixes) == 0 {
return nil, nil, fmt.Errorf("%w: AS%d", ErrNoRoute, asn)
}
return &asnInfo, allPrefixes, nil
} }
// ASPeer represents a peering relationship with another AS including handle, // ASPeer represents a peering relationship with another AS including handle,
+405
View File
@@ -1,8 +1,10 @@
package database package database
import ( import (
"bytes"
"context" "context"
"database/sql" "database/sql"
"encoding/json"
"errors" "errors"
"fmt" "fmt"
"log/slog" "log/slog"
@@ -18,6 +20,8 @@ import (
"git.eeqj.de/sneak/routewatch/internal/config" "git.eeqj.de/sneak/routewatch/internal/config"
"git.eeqj.de/sneak/routewatch/internal/logger" "git.eeqj.de/sneak/routewatch/internal/logger"
"git.eeqj.de/sneak/routewatch/internal/ristypes"
"git.eeqj.de/sneak/routewatch/pkg/asinfo"
"github.com/mattn/go-sqlite3" "github.com/mattn/go-sqlite3"
) )
@@ -1052,6 +1056,176 @@ func walFileSize(t *testing.T, db *Database) int64 {
return wal.Size() return wal.Size()
} }
// TestStartupCheckpointLogsItsFigures opens a new database, writes a route,
// and opens the same file again while the first stays open, so that its WAL
// file stays. The first open finds no WAL file: it logs a size of 0 and no
// warning. The second logs the size of the WAL file the first left, and the
// checkpoint's busy flag and frame counts, each under its own name; the
// counts read 0 once the checkpoint has emptied the file.
// Initialize logged the busy flag as wal_pages and the counts under each
// other's names (https://git.eeqj.de/sneak/routewatch/issues/117).
func TestStartupCheckpointLogsItsFigures(t *testing.T) {
t.Parallel()
cfg := &config.Config{StateDir: t.TempDir()}
db, lines := openLogged(t, cfg)
for _, line := range lines {
if line[slog.LevelKey] == slog.LevelWarn.String() {
t.Errorf("opening a new database logged a warning: %v", line)
}
}
line := findLine(t, lines, startupCheckpointMessage)
if line["wal_bytes"] != 0.0 {
t.Errorf("new database: start-up checkpoint logged wal_bytes = %v, "+
"want 0", line["wal_bytes"])
}
err := db.UpsertLiveRouteBatch([]*LiveRoute{
mkV4Route(t, "192.0.2.0/24", lockTestASN, time.Now().UTC()),
})
if err != nil {
t.Fatalf("failed to store a route: %v", err)
}
walBytes := walFileSize(t, db)
if walBytes == 0 {
t.Fatal("WAL file empty after the route write, want frames in it")
}
_, lines = openLogged(t, cfg)
line = findLine(t, lines, startupCheckpointMessage)
for _, key := range []string{"busy", "wal_frames", "checkpointed_frames"} {
if line[key] != 0.0 {
t.Errorf("start-up checkpoint logged %s = %v, want 0", key, line[key])
}
}
if line["wal_bytes"] != float64(walBytes) {
t.Errorf("start-up checkpoint logged wal_bytes = %v, want the %d "+
"bytes the first database left", line["wal_bytes"], walBytes)
}
}
// TestStartupCheckpointWarnsWhenIncomplete keeps a read open on a database
// while a route is written after the read began, then opens the same file
// again. The second open's start-up checkpoint cannot copy the frames written
// after the read began, and gives up once the busy timeout has passed. It
// checks that the checkpoint is logged as a warning with busy 1 and fewer
// frames copied than the WAL holds.
func TestStartupCheckpointWarnsWhenIncomplete(t *testing.T) {
t.Parallel()
cfg := &config.Config{StateDir: t.TempDir()}
db, _ := openLogged(t, cfg)
ctx := context.Background()
reader, err := db.db.Conn(ctx)
if err != nil {
t.Fatalf("failed to open the reading connection: %v", err)
}
defer func() { _ = reader.Close() }()
// A plain BEGIN starts a transaction that only reads, unlike the BEGIN
// IMMEDIATE of _txlock. It keeps the snapshot its first read takes until
// it ends.
_, err = reader.ExecContext(ctx, "BEGIN")
if err != nil {
t.Fatalf("failed to begin the read transaction: %v", err)
}
defer func() { _, _ = reader.ExecContext(ctx, "ROLLBACK") }()
var routes int
err = reader.QueryRowContext(ctx,
"SELECT count(*) FROM live_routes_v4").Scan(&routes)
if err != nil {
t.Fatalf("failed to read the live routes: %v", err)
}
err = db.UpsertLiveRouteBatch([]*LiveRoute{
mkV4Route(t, "192.0.2.0/24", lockTestASN, time.Now().UTC()),
})
if err != nil {
t.Fatalf("failed to store a route after the read: %v", err)
}
_, lines := openLogged(t, cfg)
line := findLine(t, lines, startupCheckpointIncompleteMessage)
if line[slog.LevelKey] != slog.LevelWarn.String() {
t.Errorf("start-up checkpoint logged at level %v, want %s",
line[slog.LevelKey], slog.LevelWarn)
}
if line["busy"] != 1.0 {
t.Errorf("start-up checkpoint logged busy = %v, want 1", line["busy"])
}
walFrames, _ := line["wal_frames"].(float64)
checkpointedFrames, _ := line["checkpointed_frames"].(float64)
if checkpointedFrames >= walFrames {
t.Errorf("start-up checkpoint logged checkpointed_frames = %v and "+
"wal_frames = %v with a read open, want fewer copied than the WAL "+
"holds", line["checkpointed_frames"], line["wal_frames"])
}
}
// openLogged opens the database of cfg with a logger that writes JSON, and
// returns it with the lines it logged while it opened, decoded.
func openLogged(t *testing.T, cfg *config.Config) (*Database, []map[string]any) {
t.Helper()
var logged bytes.Buffer
db, err := New(cfg,
&logger.Logger{Logger: slog.New(slog.NewJSONHandler(&logged, nil))})
if err != nil {
t.Fatalf("failed to open the database: %v", err)
}
t.Cleanup(func() { _ = db.Close() })
var lines []map[string]any
decoder := json.NewDecoder(&logged)
for decoder.More() {
var line map[string]any
err := decoder.Decode(&line)
if err != nil {
t.Fatalf("failed to decode the log: %v", err)
}
lines = append(lines, line)
}
return db, lines
}
// findLine returns the first of lines logged with message.
func findLine(
t *testing.T,
lines []map[string]any,
message string,
) map[string]any {
t.Helper()
for _, line := range lines {
if line[slog.MessageKey] == message {
return line
}
}
t.Fatalf("no line %q in the log", message)
return nil
}
// TestGetRandomPrefixesByLengthListsPrefixOnce stores one prefix of each // TestGetRandomPrefixesByLengthListsPrefixOnce stores one prefix of each
// family learned from two peers in the same second and checks that the // family learned from two peers in the same second and checks that the
// prefix-length sample lists each of them once. // prefix-length sample lists each of them once.
@@ -1527,6 +1701,85 @@ func TestGetASDetailsListsEachPrefixOnce(t *testing.T) {
} }
} }
// TestGetASDetailsOfASNotInDatabase asks for ASes that the database has no
// record of, some with stored routes. Each gets the prefixes of its routes and
// no seen times. One that the AS info lookup knows gets its handle and
// description from it. An AS with no record, no lookup entry and no route is
// ErrNoRoute. Before the fix every AS with no record was ErrNoRoute.
func TestGetASDetailsOfASNotInDatabase(t *testing.T) {
t.Parallel()
db, err := New(&config.Config{StateDir: t.TempDir()}, logger.New())
if err != nil {
t.Fatalf("failed to create database: %v", err)
}
defer func() { _ = db.Close() }()
ts := time.Date(2026, 1, 2, 3, 4, 5, 0, time.UTC)
tests := []struct {
asn int
listed bool // in the AS info lookup
prefixes []string // of the routes stored for the AS
}{
{15169, true, []string{"192.0.2.0/24"}},
{13335, true, nil},
{4200000000, false, []string{"198.51.100.0/24"}},
}
var routes []*LiveRoute
for _, tt := range tests {
if _, ok := asinfo.Get(tt.asn); ok != tt.listed {
t.Fatalf("AS%d in the AS info data: %t, want %t", tt.asn, ok, tt.listed)
}
for _, prefix := range tt.prefixes {
routes = append(routes, mkV4Route(t, prefix, tt.asn, ts))
}
}
err = db.UpsertLiveRouteBatch(routes)
if err != nil {
t.Fatalf("UpsertLiveRouteBatch: %v", err)
}
for _, tt := range tests {
got, prefixes, err := db.GetASDetails(tt.asn)
if err != nil {
t.Errorf("GetASDetails(%d): %v", tt.asn, err)
continue
}
want := ASN{ASN: tt.asn}
if info, ok := asinfo.Get(tt.asn); ok {
want.Handle, want.Description = info.Handle, info.Description
}
var gotPrefixes []string
for _, route := range prefixes {
gotPrefixes = append(gotPrefixes, route.Prefix)
}
if *got != want || !slices.Equal(gotPrefixes, tt.prefixes) {
t.Errorf("GetASDetails(%d) = %+v with prefixes %v, want %+v with %v",
tt.asn, *got, gotPrefixes, want, tt.prefixes)
}
}
const absentASN = 4200000001 // no record, no lookup entry and no route
if _, ok := asinfo.Get(absentASN); ok {
t.Fatalf("AS%d is in the AS info data", absentASN)
}
_, _, err = db.GetASDetails(absentASN)
if !errors.Is(err, ErrNoRoute) {
t.Errorf("GetASDetails(%d): error %v, want %v", absentASN, err, ErrNoRoute)
}
}
// TestASPrefixQueriesUseCoveringIndexes checks the plan of each query that // TestASPrefixQueriesUseCoveringIndexes checks the plan of each query that
// lists the prefixes an AS originates: a search of a covering index and no // lists the prefixes an AS originates: a search of a covering index and no
// temporary B-tree, so the list reads one index entry per prefix, not one // temporary B-tree, so the list reads one index entry per prefix, not one
@@ -1708,6 +1961,158 @@ func TestUpdatePeerBatchStoresTheEarliestAndLatestTimesOfANewPeer(
} }
} }
// backlogAge is how much older one write is than the other in the tests of
// first_seen and last_seen, as when a collector catches up on a backlog.
const backlogAge = 24 * time.Hour
// batchSpan is how much earlier a batch's First is than its Last in the tests
// of first_seen and last_seen.
const batchSpan = time.Hour
// TestWritesKeepEarliestFirstSeenAndLatestLastSeen writes a row of each of
// asns, peerings and bgp_peers through each function that writes one, at a
// newer time and at an older one. Each row takes the older write's First as
// first_seen and the newer write's Last as last_seen. The test writes in both
// orders, each in a database of its own. Newer then older is a collector
// catching up on a backlog; older then newer is a feed that arrives in order.
// Each function writes a row of its own. A batch write's First is batchSpan
// before its Last, so the test catches the two swapped. A single-row write
// takes one time.
func TestWritesKeepEarliestFirstSeenAndLatestLastSeen(t *testing.T) {
t.Parallel()
const (
batchASN, singleASN = 64500, 64501
peerASN = 64502
asnQuery = "SELECT first_seen, last_seen FROM asns WHERE asn = ?"
peeringQuery = "SELECT first_seen, last_seen FROM peerings " +
"WHERE as_a = ? AND as_b = ?"
peerQuery = "SELECT first_seen, last_seen FROM bgp_peers " +
"WHERE peer_ip = ?"
)
for name, olderFirst := range map[string]bool{
"newer then older": false,
"older then newer": true,
} {
t.Run(name, func(t *testing.T) {
t.Parallel()
db, err := New(&config.Config{StateDir: t.TempDir()}, logger.New())
if err != nil {
t.Fatalf("failed to create database: %v", err)
}
defer func() { _ = db.Close() }()
checkSeenTimes(t, db, olderFirst, func(seen SeenTimes) error {
return db.GetOrCreateASNBatch(map[int]SeenTimes{batchASN: seen})
}, batchSpan, asnQuery, batchASN)
checkSeenTimes(t, db, olderFirst, func(seen SeenTimes) error {
_, err := db.GetOrCreateASN(singleASN, seen.Last)
return err
}, 0, asnQuery, singleASN)
checkSeenTimes(t, db, olderFirst, func(seen SeenTimes) error {
return db.RecordPeering(batchASN, singleASN, seen)
}, batchSpan, peeringQuery, batchASN, singleASN)
checkSeenTimes(t, db, olderFirst, func(seen SeenTimes) error {
return db.UpdatePeerBatch(map[string]PeerUpdate{
testV4PeerIP: {
PeerIP: testV4PeerIP, PeerASN: peerASN,
MessageType: ristypes.MessageTypeUpdate,
Seen: seen,
},
})
}, batchSpan, peerQuery, testV4PeerIP)
checkSeenTimes(t, db, olderFirst, func(seen SeenTimes) error {
return db.UpdatePeer(testPeerIP, peerASN,
ristypes.MessageTypeUpdate, seen.Last)
}, 0, peerQuery, testPeerIP)
})
}
}
// checkSeenTimes calls write twice, each time with First span before Last: at
// newer times and at times backlogAge older. It writes the older times first
// when olderFirst is true. It checks that query, given args, reads the older
// write's First as first_seen and the newer write's Last as last_seen.
func checkSeenTimes(
t *testing.T, db *Database, olderFirst bool,
write func(seen SeenTimes) error,
span time.Duration, query string, args ...any,
) {
t.Helper()
newerLast := time.Date(2026, 1, 2, 3, 4, 5, 0, time.UTC)
newer := SeenTimes{First: newerLast.Add(-span), Last: newerLast}
older := SeenTimes{
First: newer.First.Add(-backlogAge),
Last: newer.Last.Add(-backlogAge),
}
writes := []SeenTimes{newer, older}
if olderFirst {
writes = []SeenTimes{older, newer}
}
for _, seen := range writes {
err := write(seen)
if err != nil {
t.Fatalf("writing first %s, last %s: %v", seen.First, seen.Last, err)
}
}
var firstSeen, lastSeen time.Time
err := db.db.QueryRowContext(t.Context(), query, args...).
Scan(&firstSeen, &lastSeen)
if err != nil {
t.Fatalf("reading the row: %v", err)
}
if !firstSeen.Equal(older.First) || !lastSeen.Equal(newer.Last) {
t.Errorf("first seen %s, last seen %s; want %s, %s",
firstSeen, lastSeen, older.First, newer.Last)
}
}
// TestGetOrCreateASNReturnsStoredSeenTimes stores an AS at a newer time, then
// at an older one. The second call returns the times the row now holds: the
// older first_seen and the newer last_seen.
func TestGetOrCreateASNReturnsStoredSeenTimes(t *testing.T) {
t.Parallel()
db, err := New(&config.Config{StateDir: t.TempDir()}, logger.New())
if err != nil {
t.Fatalf("failed to create database: %v", err)
}
defer func() { _ = db.Close() }()
const asn = 64500
newer := time.Date(2026, 1, 2, 3, 4, 5, 0, time.UTC)
older := newer.Add(-backlogAge)
_, err = db.GetOrCreateASN(asn, newer)
if err != nil {
t.Fatalf("GetOrCreateASN at %s: %v", newer, err)
}
got, err := db.GetOrCreateASN(asn, older)
if err != nil {
t.Fatalf("GetOrCreateASN at %s: %v", older, err)
}
if !got.FirstSeen.Equal(older) || !got.LastSeen.Equal(newer) {
t.Errorf("first seen %s, last seen %s; want %s, %s",
got.FirstSeen, got.LastSeen, older, newer)
}
}
// TestUpsertLiveRouteBatchWritesInKeyOrder writes new routes of two peers out // TestUpsertLiveRouteBatchWritesInKeyOrder writes new routes of two peers out
// of key order, the last one a second announcement of the first, and checks // of key order, the last one a second announcement of the first, and checks
// that the rows were inserted in key order, peer then prefix (each new row's id // that the rows were inserted in key order, peer then prefix (each new row's id
+6 -5
View File
@@ -28,9 +28,10 @@ type ASN struct {
RIRRegDate *time.Time `json:"rir_registration_date,omitempty"` RIRRegDate *time.Time `json:"rir_registration_date,omitempty"`
RIRLastMod *time.Time `json:"rir_last_modified,omitempty"` RIRLastMod *time.Time `json:"rir_last_modified,omitempty"`
WHOISRaw string `json:"whois_raw,omitempty"` WHOISRaw string `json:"whois_raw,omitempty"`
// Timestamps // Timestamps. They are zero, and left out of JSON, for an AS that has no
FirstSeen time.Time `json:"first_seen"` // record in the asns table.
LastSeen time.Time `json:"last_seen"` FirstSeen time.Time `json:"first_seen,omitzero"`
LastSeen time.Time `json:"last_seen,omitzero"`
WHOISUpdatedAt *time.Time `json:"whois_updated_at,omitempty"` WHOISUpdatedAt *time.Time `json:"whois_updated_at,omitempty"`
} }
@@ -124,8 +125,8 @@ type LiveRouteDeletion struct {
// SeenTimes is the earliest and the latest time one batch of updates saw an AS, // SeenTimes is the earliest and the latest time one batch of updates saw an AS,
// a peer or a peering. A row the batch adds is stored first seen at First and // a peer or a peering. A row the batch adds is stored first seen at First and
// last seen at Last; a row already stored keeps its first_seen and has its // last seen at Last. A row already stored takes First as its first_seen only
// last_seen set to Last. // when First is earlier, and Last as its last_seen only when Last is later.
type SeenTimes struct { type SeenTimes struct {
First time.Time First time.Time
Last time.Time Last time.Time
+8 -7
View File
@@ -34,7 +34,7 @@ type Tracker struct {
// Announcement/withdrawal metrics // Announcement/withdrawal metrics
announcementCounter metrics.Counter announcementCounter metrics.Counter
withdrawalCounter metrics.Counter withdrawalCounter metrics.Counter
churnRate metrics.Meter // combined announcements + withdrawals per second churnPerMinute minuteCounter // prefix announcements and withdrawals
// BGP peer tracking // BGP peer tracking
bgpPeerCount atomic.Int32 bgpPeerCount atomic.Int32
@@ -56,7 +56,7 @@ func New() *Tracker {
ipv6UpdateRate: metrics.NewMeter(), ipv6UpdateRate: metrics.NewMeter(),
announcementCounter: metrics.NewCounter(), announcementCounter: metrics.NewCounter(),
withdrawalCounter: metrics.NewCounter(), withdrawalCounter: metrics.NewCounter(),
churnRate: metrics.NewMeter(), churnPerMinute: minuteCounter{now: time.Now},
} }
} }
@@ -156,13 +156,13 @@ func (t *Tracker) RecordIPv6Update() {
// RecordAnnouncement records a route announcement // RecordAnnouncement records a route announcement
func (t *Tracker) RecordAnnouncement() { func (t *Tracker) RecordAnnouncement() {
t.announcementCounter.Inc(1) t.announcementCounter.Inc(1)
t.churnRate.Mark(1) t.churnPerMinute.add()
} }
// RecordWithdrawal records a route withdrawal // RecordWithdrawal records a route withdrawal
func (t *Tracker) RecordWithdrawal() { func (t *Tracker) RecordWithdrawal() {
t.withdrawalCounter.Inc(1) t.withdrawalCounter.Inc(1)
t.churnRate.Mark(1) t.churnPerMinute.add()
} }
// SetBGPPeerCount updates the current BGP peer count // SetBGPPeerCount updates the current BGP peer count
@@ -198,9 +198,10 @@ func (t *Tracker) GetWithdrawalCount() uint64 {
return uint64(count) return uint64(count)
} }
// GetChurnRate returns the route churn rate per second // GetChurnPerMinute returns the number of prefix announcements and
func (t *Tracker) GetChurnRate() float64 { // withdrawals, from all peers, in the last full clock minute.
return t.churnRate.Rate1() func (t *Tracker) GetChurnPerMinute() int {
return t.churnPerMinute.lastFullMinute()
} }
// GetRouteMetrics returns current route update metrics // GetRouteMetrics returns current route update metrics
+59
View File
@@ -0,0 +1,59 @@
package metrics
import (
"sync"
"time"
)
// minuteCounter counts events in each clock minute and reports the count of
// the last full minute. It is safe for concurrent use.
type minuteCounter struct {
mu sync.Mutex
now func() time.Time
// minute is the start of the minute that count belongs to.
minute time.Time
count int
// lastMinuteCount is the count of the minute just before minute.
lastMinuteCount int
}
// add counts one event in the current minute.
func (c *minuteCounter) add() {
c.mu.Lock()
defer c.mu.Unlock()
c.moveToCurrentMinute()
c.count++
}
// lastFullMinute returns the number of events in the minute that ended most
// recently.
func (c *minuteCounter) lastFullMinute() int {
c.mu.Lock()
defer c.mu.Unlock()
c.moveToCurrentMinute()
return c.lastMinuteCount
}
// moveToCurrentMinute starts a new count when the clock has moved into
// another minute. The count it ends becomes the last full minute's count when
// its minute is the one just before the current minute. Otherwise the last
// full minute had no events. The caller holds mu.
func (c *minuteCounter) moveToCurrentMinute() {
minute := c.now().Truncate(time.Minute)
if minute.Equal(c.minute) {
return
}
if minute.Sub(c.minute) == time.Minute {
c.lastMinuteCount = c.count
} else {
c.lastMinuteCount = 0
}
c.minute = minute
c.count = 0
}
@@ -0,0 +1,86 @@
package metrics
import (
"sync"
"testing"
"time"
)
// TestChurnPerMinuteCountsLastFullMinute announces and withdraws prefixes on
// an injected clock and expects each minute's whole count once that minute
// has ended, and 0 for a minute with none.
func TestChurnPerMinuteCountsLastFullMinute(t *testing.T) {
t.Parallel()
now := time.Date(2026, 10, 26, 12, 0, 30, 0, time.UTC)
tracker := New()
tracker.churnPerMinute.now = func() time.Time { return now }
expectChurn := func(want int, why string) {
t.Helper()
got := tracker.GetChurnPerMinute()
if got != want {
t.Fatalf("at %s churn = %d, want %d: %s",
now.Format(time.TimeOnly), got, want, why)
}
}
const firstMinuteChurn = 2
tracker.RecordAnnouncement()
tracker.RecordWithdrawal()
expectChurn(0, "12:00 has not ended")
now = time.Date(2026, 10, 26, 12, 1, 59, 0, time.UTC)
tracker.RecordAnnouncement()
expectChurn(firstMinuteChurn, "12:00 had one announcement and one withdrawal")
now = time.Date(2026, 10, 26, 12, 2, 0, 0, time.UTC)
expectChurn(1, "12:01 had one announcement")
now = time.Date(2026, 10, 26, 12, 4, 0, 0, time.UTC)
expectChurn(0, "12:03 had none")
tracker.RecordWithdrawal()
now = time.Date(2026, 10, 26, 12, 6, 0, 0, time.UTC)
expectChurn(0, "12:05 had none, though 12:04 had one withdrawal")
}
// TestChurnPerMinuteCountsConcurrentRecords announces prefixes from several
// goroutines at once and expects every one counted.
func TestChurnPerMinuteCountsConcurrentRecords(t *testing.T) {
t.Parallel()
const (
goroutines = 8
recordsPerGoroutine = 1000
)
now := time.Date(2026, 10, 26, 12, 0, 0, 0, time.UTC)
tracker := New()
tracker.churnPerMinute.now = func() time.Time { return now }
var wg sync.WaitGroup
for range goroutines {
wg.Go(func() {
for range recordsPerGoroutine {
tracker.RecordAnnouncement()
}
})
}
wg.Wait()
now = now.Add(time.Minute)
want := goroutines * recordsPerGoroutine
if got := tracker.GetChurnPerMinute(); got != want {
t.Fatalf("churn = %d, want %d", got, want)
}
}
+6 -2
View File
@@ -39,12 +39,16 @@ type ASPath struct {
Set []int Set []int
} }
// NoOrigin is the origin AS of a route whose origin is not known. AS 0 is
// reserved, so no AS has this number.
const NoOrigin = 0
// Origin returns the AS that originated the route: the last AS of the // Origin returns the AS that originated the route: the last AS of the
// sequence, or 0 (none) when the path ends in a set of several members or is // sequence, or NoOrigin when the path ends in a set of several members or is
// empty. // empty.
func (p *ASPath) Origin() int { func (p *ASPath) Origin() int {
if len(p.Set) > 0 || len(p.Sequence) == 0 { if len(p.Set) > 0 || len(p.Sequence) == 0 {
return 0 return NoOrigin
} }
return p.Sequence[len(p.Sequence)-1] return p.Sequence[len(p.Sequence)-1]
+17 -8
View File
@@ -21,10 +21,16 @@ const (
minInterval = 1 * time.Second minInterval = 1 * time.Second
// maxInterval is the maximum interval after failures (backoff cap). // maxInterval is the maximum interval after failures (backoff cap).
maxInterval = 5 * time.Minute maxInterval = time.Minute
// backoffMultiplier is how much to multiply interval on failure. // A failure multiplies the interval by backoffNumerator divided by
backoffMultiplier = 2 // backoffDenominator, 1.5, up to maxInterval.
backoffNumerator = 3
backoffDenominator = 2
// A success divides the interval by successDivisor, down to
// minInterval.
successDivisor = 2
// whoisStaleThreshold is how old WHOIS data can be before refresh. // whoisStaleThreshold is how old WHOIS data can be before refresh.
whoisStaleThreshold = 30 * 24 * time.Hour // 30 days whoisStaleThreshold = 30 * 24 * time.Hour // 30 days
@@ -62,10 +68,12 @@ type ASNFetcher struct {
// NewASNFetcher creates a new ASN fetcher. // NewASNFetcher creates a new ASN fetcher.
func NewASNFetcher(db database.Store, logger *slog.Logger) *ASNFetcher { func NewASNFetcher(db database.Store, logger *slog.Logger) *ASNFetcher {
logger = logger.With("component", "asn_fetcher")
return &ASNFetcher{ return &ASNFetcher{
db: db, db: db,
whoisClient: whois.NewClient(), whoisClient: whois.NewClient(logger),
logger: logger.With("component", "asn_fetcher"), logger: logger,
immediateQueue: make(chan int, immediateQueueSize), immediateQueue: make(chan int, immediateQueueSize),
stopCh: make(chan struct{}), stopCh: make(chan struct{}),
done: make(chan struct{}), done: make(chan struct{}),
@@ -160,7 +168,7 @@ func (f *ASNFetcher) recordSuccess() {
f.consecutiveFails = 0 f.consecutiveFails = 0
// Decrease interval by half, but not below minimum // Decrease interval by half, but not below minimum
newInterval := max(f.currentInterval/backoffMultiplier, minInterval) newInterval := max(f.currentInterval/successDivisor, minInterval)
if newInterval != f.currentInterval { if newInterval != f.currentInterval {
f.logger.Debug("Decreased fetch interval", f.logger.Debug("Decreased fetch interval",
@@ -184,8 +192,9 @@ func (f *ASNFetcher) recordFailure() {
f.intervalMu.Lock() f.intervalMu.Lock()
f.consecutiveFails++ f.consecutiveFails++
// Exponential backoff: multiply by 2, capped at max // Exponential backoff: multiply by 1.5, capped at max
newInterval := min(f.currentInterval*backoffMultiplier, maxInterval) newInterval := min(
f.currentInterval*backoffNumerator/backoffDenominator, maxInterval)
if newInterval != f.currentInterval { if newInterval != f.currentInterval {
f.logger.Debug("Increased fetch interval due to failure", f.logger.Debug("Increased fetch interval due to failure",
@@ -120,17 +120,58 @@ func TestFetchAndUpdateKeepsTheRowWhenTheLookupFails(t *testing.T) {
"was stored", stats.NeverFetched) "was stored", stats.NeverFetched)
} }
want := baseInterval * backoffNumerator / backoffDenominator
got := f.GetStats() got := f.GetStats()
if got.CurrentInterval != baseInterval*backoffMultiplier || if got.CurrentInterval != want || got.ConsecutiveFails != 1 {
got.ConsecutiveFails != 1 {
t.Errorf("interval %v after %d failures, want %v after 1", t.Errorf("interval %v after %d failures, want %v after 1",
got.CurrentInterval, got.ConsecutiveFails, got.CurrentInterval, got.ConsecutiveFails, want)
baseInterval*backoffMultiplier)
} }
}) })
} }
} }
// TestRecordStepsTheInterval has the fetcher fail from its base interval of
// 15 seconds, then succeed. Each failure makes the interval 1.5 times as
// long, up to one minute, and each success halves it, down to one second.
func TestRecordStepsTheInterval(t *testing.T) {
t.Parallel()
f := NewASNFetcher(newMockStore(), logger.New().Logger)
for i, want := range []time.Duration{
22500 * time.Millisecond,
33750 * time.Millisecond,
50625 * time.Millisecond,
time.Minute,
time.Minute,
} {
f.recordFailure()
if got := f.getInterval(); got != want {
t.Errorf("interval after %d failures = %v, want %v",
i+1, got, want)
}
}
for i, want := range []time.Duration{
30 * time.Second,
15 * time.Second,
7500 * time.Millisecond,
3750 * time.Millisecond,
1875 * time.Millisecond,
time.Second,
time.Second,
} {
f.recordSuccess()
if got := f.getInterval(); got != want {
t.Errorf("interval after %d successes = %v, want %v",
i+1, got, want)
}
}
}
// TestFetchAndUpdateMarksAnASWithoutReferral looks up a private-use AS, which // TestFetchAndUpdateMarksAnASWithoutReferral looks up a private-use AS, which
// IANA refers to no RIR. The AS must not be looked up again until it is // IANA refers to no RIR. The AS must not be looked up again until it is
// stale, and the lookup counts as neither a success nor a failure. // stale, and the lookup counts as neither a success nor a failure.
+83 -32
View File
@@ -1,7 +1,7 @@
package server package server
import ( import (
"bytes" "cmp"
"context" "context"
"encoding/json" "encoding/json"
"errors" "errors"
@@ -9,6 +9,7 @@ import (
"net/http" "net/http"
"net/netip" "net/netip"
"runtime" "runtime"
"slices"
"sort" "sort"
"strconv" "strconv"
"strings" "strings"
@@ -16,6 +17,7 @@ import (
"git.eeqj.de/sneak/routewatch/internal/database" "git.eeqj.de/sneak/routewatch/internal/database"
"git.eeqj.de/sneak/routewatch/internal/metrics" "git.eeqj.de/sneak/routewatch/internal/metrics"
"git.eeqj.de/sneak/routewatch/internal/ristypes"
"git.eeqj.de/sneak/routewatch/internal/streamer" "git.eeqj.de/sneak/routewatch/internal/streamer"
"git.eeqj.de/sneak/routewatch/internal/templates" "git.eeqj.de/sneak/routewatch/internal/templates"
asinfo "git.eeqj.de/sneak/routewatch/pkg/asinfo" asinfo "git.eeqj.de/sneak/routewatch/pkg/asinfo"
@@ -56,10 +58,18 @@ const (
// Error messages an answer sends, in place of the error behind it where there // Error messages an answer sends, in place of the error behind it where there
// is one. // is one.
const ( const (
invalidPrefixMessage = "Invalid prefix" invalidPrefixMessage = "Invalid prefix"
hostNotResolvedMessage = "could not resolve host name" hostNotResolvedMessage = "could not resolve host name"
hostNotPublicMessage = "host name has no public address" hostNotPublicMessage = "host name has no public address"
tooManyTargetsMessage = "Too many addresses or host names" tooManyTargetsMessage = "Too many addresses or host names"
searchQueryMissingMessage = "Missing search query"
)
// The AS name search reads the text it looks for from the URL query parameter
// asSearchQueryParam, and answers with at most maxASSearchResults matches.
const (
asSearchQueryParam = "q"
maxASSearchResults = 50
) )
// IP versions as database.LiveRoute.IPVersion records them. // IP versions as database.LiveRoute.IPVersion records them.
@@ -432,10 +442,10 @@ type GCStats struct {
// //
//nolint:tagliatelle // field names are the existing JSON API //nolint:tagliatelle // field names are the existing JSON API
type StreamStats struct { type StreamStats struct {
Announcements uint64 `json:"announcements"` Announcements uint64 `json:"announcements"`
Withdrawals uint64 `json:"withdrawals"` Withdrawals uint64 `json:"withdrawals"`
RouteChurnPerSec float64 `json:"route_churn_per_sec"` RouteChurnPerMin int `json:"route_churn_per_min"`
BGPPeerCount int `json:"bgp_peer_count"` BGPPeerCount int `json:"bgp_peer_count"`
} }
// StatsResponse represents the API statistics response // StatsResponse represents the API statistics response
@@ -536,7 +546,7 @@ func (s *Server) streamStats() StreamStats {
return StreamStats{ return StreamStats{
Announcements: metricsTracker.GetAnnouncementCount(), Announcements: metricsTracker.GetAnnouncementCount(),
Withdrawals: metricsTracker.GetWithdrawalCount(), Withdrawals: metricsTracker.GetWithdrawalCount(),
RouteChurnPerSec: metricsTracker.GetChurnRate(), RouteChurnPerMin: metricsTracker.GetChurnPerMinute(),
BGPPeerCount: metricsTracker.GetBGPPeerCount(), BGPPeerCount: metricsTracker.GetBGPPeerCount(),
} }
} }
@@ -1001,11 +1011,12 @@ func (s *Server) handleASDetailJSON() http.HandlerFunc {
} }
} }
// splitRoutesByIPVersion groups routes into the IPv4 and the IPv6 ones. // splitRoutesByIPVersion groups routes into the IPv4 and the IPv6 ones. Neither
// group is nil, so that a JSON answer holds [] for an empty one and not null.
func splitRoutesByIPVersion( func splitRoutesByIPVersion(
routes []database.LiveRoute, routes []database.LiveRoute,
) ([]database.LiveRoute, []database.LiveRoute) { ) ([]database.LiveRoute, []database.LiveRoute) {
var ipv4Routes, ipv6Routes []database.LiveRoute ipv4Routes, ipv6Routes := []database.LiveRoute{}, []database.LiveRoute{}
for _, route := range routes { for _, route := range routes {
if route.IPVersion == ipVersionV4 { if route.IPVersion == ipVersionV4 {
@@ -1018,24 +1029,15 @@ func splitRoutesByIPVersion(
return ipv4Routes, ipv6Routes return ipv4Routes, ipv6Routes
} }
// sortRoutesByNetwork sorts routes by network address, then by mask length. // sortRoutesByMaskLength sorts routes by mask length, shortest first, and
func sortRoutesByNetwork(routes []database.LiveRoute) { // routes with the same mask length by network address.
sort.Slice(routes, func(i, j int) bool { func sortRoutesByMaskLength(routes []database.LiveRoute) {
// Parse the prefixes to compare network addresses slices.SortFunc(routes, func(a, b database.LiveRoute) int {
ipI, netI, _ := net.ParseCIDR(routes[i].Prefix) prefixA, _ := netip.ParsePrefix(a.Prefix)
ipJ, netJ, _ := net.ParseCIDR(routes[j].Prefix) prefixB, _ := netip.ParsePrefix(b.Prefix)
// Compare by network address first return cmp.Or(cmp.Compare(prefixA.Bits(), prefixB.Bits()),
cmp := bytes.Compare(ipI.To16(), ipJ.To16()) prefixA.Addr().Compare(prefixB.Addr()))
if cmp != 0 {
return cmp < 0
}
// If network addresses are equal, compare by mask length
onesI, _ := netI.Mask.Size()
onesJ, _ := netJ.Mask.Size()
return onesI < onesJ
}) })
} }
@@ -1105,6 +1107,56 @@ func (s *Server) handlePrefixDetailJSON() http.HandlerFunc {
} }
} }
// ASSearchResponse is the answer to an AS name search: the text searched for,
// the number of ASes that match it, and the first maxASSearchResults of them in
// AS number order.
type ASSearchResponse struct {
Query string `json:"query"`
Total int `json:"total"`
Results []asinfo.Info `json:"results"`
}
// handleASSearch returns a handler that searches the AS list built into
// asinfo for the text in the q parameter, without its leading and trailing
// spaces. It answers 400 when no text is left.
func (s *Server) handleASSearch() http.HandlerFunc {
return func(w http.ResponseWriter, r *http.Request) {
query := strings.TrimSpace(r.URL.Query().Get(asSearchQueryParam))
if query == "" {
s.writeJSONError(w, http.StatusBadRequest, searchQueryMissingMessage)
return
}
// AS 0 means no known origin and has no AS page, so the answer leaves
// it out. The matches come in AS number order, so AS 0 can only be the
// first. The search asks for one match more than the answer holds, to
// take its place.
results, total := asinfo.Search(query, maxASSearchResults+1)
if len(results) > 0 && results[0].ASN == ristypes.NoOrigin {
results = results[1:]
total--
}
results = results[:min(len(results), maxASSearchResults)]
// results is an empty list rather than nil when nothing matches, so
// that the answer holds [] and not null.
if results == nil {
results = []asinfo.Info{}
}
err := writeJSONSuccess(w, ASSearchResponse{
Query: query,
Total: total,
Results: results,
})
if err != nil {
s.logger.Error("Failed to encode AS search results", "error", err)
}
}
}
// handleASDetail returns a handler that serves the AS detail HTML page // handleASDetail returns a handler that serves the AS detail HTML page
func (s *Server) handleASDetail() http.HandlerFunc { func (s *Server) handleASDetail() http.HandlerFunc {
return func(w http.ResponseWriter, r *http.Request) { return func(w http.ResponseWriter, r *http.Request) {
@@ -1152,9 +1204,8 @@ func (s *Server) handleASDetail() http.HandlerFunc {
// Group prefixes by IP version // Group prefixes by IP version
ipv4Prefixes, ipv6Prefixes := splitRoutesByIPVersion(prefixes) ipv4Prefixes, ipv6Prefixes := splitRoutesByIPVersion(prefixes)
// Sort prefixes by network address sortRoutesByMaskLength(ipv4Prefixes)
sortRoutesByNetwork(ipv4Prefixes) sortRoutesByMaskLength(ipv6Prefixes)
sortRoutesByNetwork(ipv6Prefixes)
// Prepare template data // Prepare template data
data := struct { data := struct {
+520 -11
View File
@@ -19,6 +19,7 @@ import (
"git.eeqj.de/sneak/routewatch/internal/database" "git.eeqj.de/sneak/routewatch/internal/database"
"git.eeqj.de/sneak/routewatch/internal/logger" "git.eeqj.de/sneak/routewatch/internal/logger"
"git.eeqj.de/sneak/routewatch/internal/metrics" "git.eeqj.de/sneak/routewatch/internal/metrics"
"git.eeqj.de/sneak/routewatch/internal/ristypes"
"git.eeqj.de/sneak/routewatch/internal/streamer" "git.eeqj.de/sneak/routewatch/internal/streamer"
"git.eeqj.de/sneak/routewatch/pkg/asinfo" "git.eeqj.de/sneak/routewatch/pkg/asinfo"
) )
@@ -26,6 +27,7 @@ import (
// The prefix and peer addresses the prefix page tests store routes for. // The prefix and peer addresses the prefix page tests store routes for.
const ( const (
testPrefixV4 = "192.0.2.0/24" testPrefixV4 = "192.0.2.0/24"
testPrefixV6 = "2001:db8::/32"
testPeerV4 = "198.51.100.1" testPeerV4 = "198.51.100.1"
testPeerV6 = "2001:db8::1" testPeerV6 = "2001:db8::1"
) )
@@ -193,8 +195,53 @@ func TestStatsHandlersAnswerFromMemory(t *testing.T) {
} }
} }
// TestStatsAnswerCarriesChurnPerMinute checks that the stream object of the
// /api/v1/stats answer holds route_churn_per_min as a whole number, and no
// longer route_churn_per_sec, the moving average it replaces
// (https://git.eeqj.de/sneak/routewatch/issues/160).
func TestStatsAnswerCarriesChurnPerMinute(t *testing.T) {
t.Parallel()
s := newServerWithTestRoutes(t)
rec := httptest.NewRecorder()
s.handleStats()(rec,
httptest.NewRequestWithContext(t.Context(), http.MethodGet, statsPath, nil))
if rec.Code != http.StatusOK {
t.Fatalf("status %d, want %d; body %s", rec.Code, http.StatusOK, rec.Body)
}
var body struct {
Data struct {
Stream map[string]json.RawMessage `json:"stream"`
} `json:"data"`
}
err := json.Unmarshal(rec.Body.Bytes(), &body)
if err != nil {
t.Fatalf("decoding the answer: %v", err)
}
churn, ok := body.Data.Stream["route_churn_per_min"]
if !ok {
t.Fatalf("stream has no route_churn_per_min: %s", rec.Body)
}
var count int
err = json.Unmarshal(churn, &count)
if err != nil {
t.Errorf("route_churn_per_min = %s, want a whole number: %v", churn, err)
}
if _, ok := body.Data.Stream["route_churn_per_sec"]; ok {
t.Errorf("stream still has route_churn_per_sec: %s", rec.Body)
}
}
// newServerWithTestRoutes returns a server on a new database that holds a // newServerWithTestRoutes returns a server on a new database that holds a
// route for testPrefixV4 and one for 2001:db8::/32. // route for testPrefixV4 and one for testPrefixV6.
func newServerWithTestRoutes(t *testing.T) *Server { func newServerWithTestRoutes(t *testing.T) *Server {
t.Helper() t.Helper()
@@ -214,7 +261,7 @@ func newServerWithTestRoutes(t *testing.T) *Server {
NextHop: testPeerV4, LastUpdated: ts, NextHop: testPeerV4, LastUpdated: ts,
}, },
{ {
Prefix: "2001:db8::/32", MaskLength: 32, IPVersion: 6, Prefix: testPrefixV6, MaskLength: 32, IPVersion: 6,
OriginASN: 64501, PeerIP: testPeerV6, ASPath: []int{64501}, OriginASN: 64501, PeerIP: testPeerV6, ASPath: []int{64501},
NextHop: testPeerV6, LastUpdated: ts, NextHop: testPeerV6, LastUpdated: ts,
}, },
@@ -411,8 +458,8 @@ func TestPrefixDetailShowsOriginNames(t *testing.T) {
// TestPagesSayOriginZeroIsNoKnownOrigin stores a route with origin 0, as the // TestPagesSayOriginZeroIsNoKnownOrigin stores a route with origin 0, as the
// prefix handler stores a route whose AS path ends in an AS set of several // prefix handler stores a route whose AS path ends in an AS set of several
// members, and checks that the prefix page and the prefix length page say it // members, and checks that the prefix page and the prefix length page say it
// has no known origin, without a link to AS0 or the description of the // has no known origin, without a link to AS0
// reserved AS 0 (https://git.eeqj.de/sneak/routewatch/issues/75). // (https://git.eeqj.de/sneak/routewatch/issues/75).
func TestPagesSayOriginZeroIsNoKnownOrigin(t *testing.T) { func TestPagesSayOriginZeroIsNoKnownOrigin(t *testing.T) {
t.Parallel() t.Parallel()
@@ -432,11 +479,6 @@ func TestPagesSayOriginZeroIsNoKnownOrigin(t *testing.T) {
t.Fatalf("UpsertLiveRouteBatch: %v", err) t.Fatalf("UpsertLiveRouteBatch: %v", err)
} }
reserved, ok := asinfo.Get(0)
if !ok {
t.Fatal("AS0 is not in the AS info data")
}
s := New(db, streamer.New(logger.New(), metrics.New()), logger.New(), &config.Config{}) s := New(db, streamer.New(logger.New(), metrics.New()), logger.New(), &config.Config{})
for _, path := range []string{"/prefix/" + testPrefixV4, prefixLengthPagePath} { for _, path := range []string{"/prefix/" + testPrefixV4, prefixLengthPagePath} {
@@ -458,9 +500,70 @@ func TestPagesSayOriginZeroIsNoKnownOrigin(t *testing.T) {
if strings.Contains(page, `href="/as/0"`) { if strings.Contains(page, `href="/as/0"`) {
t.Errorf("%s links to AS0", path) t.Errorf("%s links to AS0", path)
} }
}
}
if strings.Contains(page, html.EscapeString(reserved.Description)) { // TestPagesShowAgesInWords stores routes last updated about 5 minutes, 3 hours
t.Errorf("%s shows AS0's description %q", path, reserved.Description) // and 15 days ago, and checks that the AS page and the prefix pages show each
// age in words. Before the fix the ages read "5m0s ago", "3h0m0s ago" and
// "360h0m0s ago" (https://git.eeqj.de/sneak/routewatch/issues/163).
func TestPagesShowAgesInWords(t *testing.T) {
t.Parallel()
db, err := database.New(&config.Config{StateDir: t.TempDir()}, logger.New())
if err != nil {
t.Fatalf("database.New: %v", err)
}
t.Cleanup(func() { _ = db.Close() })
// Each age is well past a whole number of its unit, so the time the test
// takes to run does not change the words.
ages := []struct {
prefix string
age time.Duration
want string
}{
{testPrefixV4, 5*time.Minute + 30*time.Second, "5 minutes ago"},
{"198.51.100.0/24", 3*time.Hour + 30*time.Minute, "3 hours ago"},
{"203.0.113.0/24", 15 * 24 * time.Hour, "2 weeks ago"},
}
now := time.Now().UTC()
routes := make([]*database.LiveRoute, 0, len(ages))
for _, a := range ages {
routes = append(routes, &database.LiveRoute{
Prefix: a.prefix, MaskLength: 24, IPVersion: 4,
OriginASN: 64500, PeerIP: testPeerV4, ASPath: []int{64500},
NextHop: testPeerV4, LastUpdated: now.Add(-a.age),
})
}
err = db.UpsertLiveRouteBatch(routes)
if err != nil {
t.Fatalf("UpsertLiveRouteBatch: %v", err)
}
s := New(db, streamer.New(logger.New(), metrics.New()), logger.New(), &config.Config{})
for _, a := range ages {
cell := `<td class="age">` + a.want + `</td>`
for _, path := range []string{asPagePath, "/prefix/" + a.prefix} {
rec := httptest.NewRecorder()
s.router.ServeHTTP(rec, httptest.NewRequestWithContext(
t.Context(), http.MethodGet, path, nil))
if rec.Code != http.StatusOK {
t.Fatalf("%s: status %d, want %d; body %s",
path, rec.Code, http.StatusOK, rec.Body)
}
if !strings.Contains(rec.Body.String(), cell) {
t.Errorf("%s does not show the age of the route for %s as %q",
path, a.prefix, a.want)
}
} }
} }
} }
@@ -1166,6 +1269,412 @@ func TestIPLookupHidesPrivateAddresses(t *testing.T) {
} }
} }
// asSearchPath is the path of the AS name search.
const asSearchPath = "/api/v1/search/as"
// searchAS sends an AS name search with the URL query rawQuery through s's
// router, expects a 200 answer, and returns its data.
func searchAS(t *testing.T, s *Server, rawQuery string) ASSearchResponse {
t.Helper()
rec := httptest.NewRecorder()
s.router.ServeHTTP(rec, httptest.NewRequestWithContext(
t.Context(), http.MethodGet, asSearchPath+"?"+rawQuery, nil))
if rec.Code != http.StatusOK {
t.Fatalf("%s: status %d, want %d; body %s",
rawQuery, rec.Code, http.StatusOK, rec.Body)
}
var body struct {
Data ASSearchResponse `json:"data"`
}
err := json.Unmarshal(rec.Body.Bytes(), &body)
if err != nil {
t.Fatalf("%s: decoding the answer %q: %v", rawQuery, rec.Body, err)
}
return body.Data
}
// TestASSearchAnswersMatches searches for Cloudflare's handle in lower case,
// with spaces around it, and expects AS13335 among the matches. Before the fix
// there was no AS name search, and the home page's search card said it was
// coming soon (https://git.eeqj.de/sneak/routewatch/issues/151).
func TestASSearchAnswersMatches(t *testing.T) {
t.Parallel()
const query = "cloudflarenet"
data := searchAS(t, newServerWithTestRoutes(t), "q=%20"+query+"%20")
if data.Query != query {
t.Errorf("query %q, want %q", data.Query, query)
}
if data.Total != len(data.Results) {
t.Errorf("total %d, but %d results", data.Total, len(data.Results))
}
cloudflare := asinfo.Info{
ASN: 13335, Handle: "CLOUDFLARENET", Description: "Cloudflare Inc.",
}
if !slices.Contains(data.Results, cloudflare) {
t.Errorf("results %v do not hold %v", data.Results, cloudflare)
}
}
// TestASSearchAnswersEmptyListWhenNothingMatches expects a search that matches
// nothing to answer 200 with a total of 0 and an empty list, not null.
func TestASSearchAnswersEmptyListWhenNothingMatches(t *testing.T) {
t.Parallel()
data := searchAS(t, newServerWithTestRoutes(t), "q=XYZABC123456")
if data.Total != 0 || data.Results == nil || len(data.Results) != 0 {
t.Errorf("total %d and results %#v, want 0 and an empty list",
data.Total, data.Results)
}
}
// TestASSearchAnswers400ForEmptyQuery expects a search with no q, an empty q
// or a q of only spaces to answer 400 with the JSON error body.
func TestASSearchAnswers400ForEmptyQuery(t *testing.T) {
t.Parallel()
s := newServerWithTestRoutes(t)
for _, path := range []string{
asSearchPath, asSearchPath + "?q=", asSearchPath + "?q=%20%20",
} {
rec := httptest.NewRecorder()
s.router.ServeHTTP(rec, httptest.NewRequestWithContext(
t.Context(), http.MethodGet, path, nil))
if rec.Code != http.StatusBadRequest {
t.Errorf("%s: status %d, want %d; body %s",
path, rec.Code, http.StatusBadRequest, rec.Body)
}
checkJSONErrorCode(t, path, rec, http.StatusBadRequest)
checkJSONErrorMessage(t, path, rec, searchQueryMissingMessage)
}
}
// TestASSearchShowsFirstMatchesAndTotal searches for a word with more matches
// than an answer holds, and expects the first maxASSearchResults of them, in
// AS number order, and the number of all of them.
func TestASSearchShowsFirstMatchesAndTotal(t *testing.T) {
t.Parallel()
const query = "university"
all, _ := asinfo.Search(query, asinfo.Total())
if len(all) <= maxASSearchResults {
t.Fatalf("Search(%q) has %d matches; the test needs more than %d",
query, len(all), maxASSearchResults)
}
data := searchAS(t, newServerWithTestRoutes(t), "q="+query)
if data.Total != len(all) {
t.Errorf("total %d, want %d", data.Total, len(all))
}
if !slices.Equal(data.Results, all[:maxASSearchResults]) {
t.Errorf("results %v, want the first %d matches %v",
data.Results, maxASSearchResults, all[:maxASSearchResults])
}
}
// The AS numbers the AS detail tests ask for. storedASN originates the route
// for testPrefixV4 and is in the database. routeOnlyASN originates the route
// for testPrefixV6, but the database has no record of the AS itself, as when
// a route is stored before its AS. Both are documentation AS numbers, which
// the AS list built into asinfo does not hold. builtInOnlyASN is in that list
// but not in the database. unknownASN is in neither.
const (
storedASN = 64500
routeOnlyASN = 64501
builtInOnlyASN = 15169
unknownASN = 4200000000
)
// zeroDate is how a page writes a time that was never set.
const zeroDate = "0001-01-01"
// firstSeenCard is the label of the AS page's first-seen date.
const firstSeenCard = `<div class="info-label">First Seen</div>`
// newServerWithStoredAS returns newServerWithTestRoutes's server with storedASN
// added to the database. It fails the test when the AS list built into asinfo
// lacks builtInOnlyASN or holds unknownASN.
func newServerWithStoredAS(t *testing.T) *Server {
t.Helper()
if _, ok := asinfo.Get(builtInOnlyASN); !ok {
t.Fatalf("AS%d is not in the AS info data", builtInOnlyASN)
}
if _, ok := asinfo.Get(unknownASN); ok {
t.Fatalf("AS%d is in the AS info data", unknownASN)
}
s := newServerWithTestRoutes(t)
_, err := s.db.GetOrCreateASN(storedASN, time.Date(2026, 1, 2, 3, 4, 5, 0, time.UTC))
if err != nil {
t.Fatalf("GetOrCreateASN: %v", err)
}
return s
}
// TestASDetailJSONAnswersBuiltInAS asks /api/v1/as/ for each of the ASes above.
// All but unknownASN answer 200 with the handle and description the built-in
// AS list has for them, if any, and their prefixes, as lists even when empty.
// Only storedASN has a first_seen time. unknownASN answers 404. Before the fix
// builtInOnlyASN and routeOnlyASN answered 404 too
// (https://git.eeqj.de/sneak/routewatch/issues/151).
func TestASDetailJSONAnswersBuiltInAS(t *testing.T) {
t.Parallel()
s := newServerWithStoredAS(t)
tests := []struct {
asn int
wantCode int
wantPrefixes []string
wantFirstSeen bool
}{
{storedASN, http.StatusOK, []string{testPrefixV4}, true},
{routeOnlyASN, http.StatusOK, []string{testPrefixV6}, false},
{builtInOnlyASN, http.StatusOK, nil, false},
{unknownASN, http.StatusNotFound, nil, false},
}
for _, tt := range tests {
path := "/api/v1/as/" + strconv.Itoa(tt.asn)
rec := httptest.NewRecorder()
s.router.ServeHTTP(rec, httptest.NewRequestWithContext(
t.Context(), http.MethodGet, path, nil))
if rec.Code != tt.wantCode {
t.Errorf("%s: status %d, want %d; body %s",
path, rec.Code, tt.wantCode, rec.Body)
continue
}
if tt.wantCode != http.StatusOK {
checkJSONErrorCode(t, path, rec, tt.wantCode)
continue
}
//nolint:tagliatelle // field names are the existing JSON API
var body struct {
Data struct {
ASN database.ASN `json:"asn"`
IPv4Prefixes []database.LiveRoute `json:"ipv4_prefixes"`
IPv6Prefixes []database.LiveRoute `json:"ipv6_prefixes"`
} `json:"data"`
}
err := json.Unmarshal(rec.Body.Bytes(), &body)
if err != nil {
t.Errorf("%s: decoding the answer %q: %v", path, rec.Body, err)
continue
}
handle, description := asinfo.GetHandle(tt.asn), asinfo.GetDescription(tt.asn)
got := body.Data.ASN
if got.ASN != tt.asn || got.Handle != handle ||
got.Description != description {
t.Errorf("%s: AS %d %q %q, want %d %q %q", path, got.ASN,
got.Handle, got.Description, tt.asn, handle, description)
}
if body.Data.IPv4Prefixes == nil || body.Data.IPv6Prefixes == nil {
t.Errorf("%s: a prefix list is null; body %s", path, rec.Body)
}
routes := slices.Concat(body.Data.IPv4Prefixes, body.Data.IPv6Prefixes)
gotPrefixes := make([]string, 0, len(routes))
for _, route := range routes {
gotPrefixes = append(gotPrefixes, route.Prefix)
}
if !slices.Equal(gotPrefixes, tt.wantPrefixes) {
t.Errorf("%s: prefixes %v, want %v", path, gotPrefixes, tt.wantPrefixes)
}
gotFirstSeen := strings.Contains(rec.Body.String(), `"first_seen"`)
if gotFirstSeen != tt.wantFirstSeen {
t.Errorf("%s: has first_seen %t, want %t; body %s",
path, gotFirstSeen, tt.wantFirstSeen, rec.Body)
}
}
}
// TestASDetailPageShowsBuiltInAS asks /as/ for each of the ASes above. All but
// unknownASN answer 200 with the handle and description the built-in AS list
// has for them, if any. storedASN and routeOnlyASN list their prefixes, and
// builtInOnlyASN says the AS announces none. Only storedASN shows a first-seen
// date, and no page shows a date that was never set. unknownASN answers 404.
// Before the fix builtInOnlyASN and routeOnlyASN answered 404 too
// (https://git.eeqj.de/sneak/routewatch/issues/151).
func TestASDetailPageShowsBuiltInAS(t *testing.T) {
t.Parallel()
s := newServerWithStoredAS(t)
tests := []struct {
asn int
wantCode int
wantText string
wantFirstSeen bool
}{
{storedASN, http.StatusOK, testPrefixV4, true},
{routeOnlyASN, http.StatusOK, testPrefixV6, false},
{builtInOnlyASN, http.StatusOK, "No prefixes announced by this AS", false},
{unknownASN, http.StatusNotFound, "", false},
}
for _, tt := range tests {
path := "/as/" + strconv.Itoa(tt.asn)
rec := httptest.NewRecorder()
s.router.ServeHTTP(rec, httptest.NewRequestWithContext(
t.Context(), http.MethodGet, path, nil))
if rec.Code != tt.wantCode {
t.Errorf("%s: status %d, want %d; body %s",
path, rec.Code, tt.wantCode, rec.Body)
continue
}
if tt.wantCode != http.StatusOK {
continue
}
page := rec.Body.String()
for _, want := range []string{
html.EscapeString(asinfo.GetHandle(tt.asn)),
html.EscapeString(asinfo.GetDescription(tt.asn)),
tt.wantText,
} {
if !strings.Contains(page, want) {
t.Errorf("%s: page does not show %q", path, want)
}
}
if strings.Contains(page, zeroDate) {
t.Errorf("%s: page shows the date %s", path, zeroDate)
}
gotFirstSeen := strings.Contains(page, firstSeenCard)
if gotFirstSeen != tt.wantFirstSeen {
t.Errorf("%s: shows a first-seen date %t, want %t",
path, gotFirstSeen, tt.wantFirstSeen)
}
}
}
// TestSortRoutesByMaskLength expects the AS page's prefix lists in mask length
// order, shortest first, and in address order within one mask length. Before
// the fix they were in address order, so 10.1.0.0/16 came before 11.0.0.0/8
// (https://git.eeqj.de/sneak/routewatch/issues/153).
func TestSortRoutesByMaskLength(t *testing.T) {
t.Parallel()
tests := []struct {
name string
prefixes []string
want []string
}{
{
name: "IPv4",
prefixes: []string{
"100.64.1.0/24", "11.0.0.0/8", "10.1.0.0/16",
"20.0.1.0/24", "172.16.0.0/12", "9.0.0.0/8",
},
want: []string{
"9.0.0.0/8", "11.0.0.0/8", "172.16.0.0/12",
"10.1.0.0/16", "20.0.1.0/24", "100.64.1.0/24",
},
},
{
name: "IPv6",
prefixes: []string{
"2001:db8:1::/48", "2001:db8:8000::/33", "2001:db8:0:1::/64",
"2001:db8::/48", "2001:db8::/33",
},
want: []string{
"2001:db8::/33", "2001:db8:8000::/33", "2001:db8::/48",
"2001:db8:1::/48", "2001:db8:0:1::/64",
},
},
}
for _, tt := range tests {
routes := make([]database.LiveRoute, 0, len(tt.prefixes))
for _, prefix := range tt.prefixes {
routes = append(routes, database.LiveRoute{Prefix: prefix})
}
sortRoutesByMaskLength(routes)
got := make([]string, 0, len(routes))
for _, route := range routes {
got = append(got, route.Prefix)
}
if !slices.Equal(got, tt.want) {
t.Errorf("%s: order %v, want %v", tt.name, got, tt.want)
}
}
}
// TestNoOriginHasNoASAnswers stores a route with no known origin, which has
// origin AS 0 (https://git.eeqj.de/sneak/routewatch/issues/75). It expects
// /as/0 and /api/v1/as/0 to answer 404.
func TestNoOriginHasNoASAnswers(t *testing.T) {
t.Parallel()
const noOriginPrefix = "203.0.113.0/24"
s := newServerWithTestRoutes(t)
err := s.db.UpsertLiveRouteBatch([]*database.LiveRoute{{
Prefix: noOriginPrefix, MaskLength: 24, IPVersion: 4,
OriginASN: ristypes.NoOrigin, PeerIP: testPeerV4,
ASPath: []int{64500, 64501, 64502}, NextHop: testPeerV4,
LastUpdated: time.Date(2026, 1, 2, 3, 4, 5, 0, time.UTC),
}})
if err != nil {
t.Fatalf("UpsertLiveRouteBatch: %v", err)
}
asn := strconv.Itoa(ristypes.NoOrigin)
for _, path := range []string{"/as/" + asn, "/api/v1/as/" + asn} {
rec := httptest.NewRecorder()
s.router.ServeHTTP(rec, httptest.NewRequestWithContext(
t.Context(), http.MethodGet, path, nil))
if rec.Code != http.StatusNotFound {
t.Errorf("%s: status %d, want %d; body %s",
path, rec.Code, http.StatusNotFound, rec.Body)
}
}
}
// settledGoroutineCount lets transient goroutines finish, then reports the // settledGoroutineCount lets transient goroutines finish, then reports the
// current count. // current count.
func settledGoroutineCount() int { func settledGoroutineCount() int {
+1
View File
@@ -87,6 +87,7 @@ func (s *Server) setupRoutes() {
r.Get("/ip/{ip}", s.handleIPLookup()) r.Get("/ip/{ip}", s.handleIPLookup())
r.Get("/as/{asn}", s.handleASDetailJSON()) r.Get("/as/{asn}", s.handleASDetailJSON())
r.Get("/prefix/{prefix}/{len}", s.handlePrefixDetailJSON()) r.Get("/prefix/{prefix}/{len}", s.handlePrefixDetailJSON())
r.Get("/search/as", s.handleASSearch())
}) })
s.router = r s.router = r
+5 -1
View File
@@ -1,4 +1,8 @@
// Package server provides HTTP endpoints for status monitoring and statistics // Package server serves RouteWatch's web pages and JSON API:
// - the home page, the status figures and the health check
// - the AS, prefix and prefix length pages, and the JSON answers behind them
// - the IP lookups
// - the AS name search
package server package server
import ( import (
+30 -8
View File
@@ -126,6 +126,26 @@
margin: 0; margin: 0;
color: #2c3e50; color: #2c3e50;
} }
/* A click on a prefix list's heading folds or unfolds the list. Not
every browser shows its own marker on a flex element, so the page
hides it and draws a triangle before the heading. The triangle
points down while the list is open. */
summary.prefix-header {
cursor: pointer;
list-style: none;
}
summary.prefix-header::-webkit-details-marker {
display: none;
}
summary.prefix-header h2::before {
content: "\25B8";
display: inline-block;
margin-right: 10px;
transition: transform 0.2s;
}
details[open] > summary.prefix-header h2::before {
transform: rotate(90deg);
}
.prefix-count { .prefix-count {
background: #e74c3c; background: #e74c3c;
color: white; color: white;
@@ -233,18 +253,20 @@
<div class="info-label">Peer ASNs</div> <div class="info-label">Peer ASNs</div>
<div class="info-value">{{.PeerCount}}</div> <div class="info-value">{{.PeerCount}}</div>
</div> </div>
{{if not .ASN.FirstSeen.IsZero}}
<div class="info-card"> <div class="info-card">
<div class="info-label">First Seen</div> <div class="info-label">First Seen</div>
<div class="info-value">{{.ASN.FirstSeen.Format "2006-01-02"}}</div> <div class="info-value">{{.ASN.FirstSeen.Format "2006-01-02"}}</div>
</div> </div>
{{end}}
</div> </div>
{{if .IPv4Prefixes}} {{if .IPv4Prefixes}}
<div class="prefix-section"> <details class="prefix-section" open>
<div class="prefix-header"> <summary class="prefix-header">
<h2>IPv4 Prefixes</h2> <h2>IPv4 Prefixes</h2>
<span class="prefix-count">{{.IPv4Count}}</span> <span class="prefix-count">{{.IPv4Count}}</span>
</div> </summary>
<table class="prefix-table"> <table class="prefix-table">
<thead> <thead>
<tr> <tr>
@@ -265,15 +287,15 @@
{{end}} {{end}}
</tbody> </tbody>
</table> </table>
</div> </details>
{{end}} {{end}}
{{if .IPv6Prefixes}} {{if .IPv6Prefixes}}
<div class="prefix-section"> <details class="prefix-section" open>
<div class="prefix-header"> <summary class="prefix-header">
<h2>IPv6 Prefixes</h2> <h2>IPv6 Prefixes</h2>
<span class="prefix-count">{{.IPv6Count}}</span> <span class="prefix-count">{{.IPv6Count}}</span>
</div> </summary>
<table class="prefix-table"> <table class="prefix-table">
<thead> <thead>
<tr> <tr>
@@ -294,7 +316,7 @@
{{end}} {{end}}
</tbody> </tbody>
</table> </table>
</div> </details>
{{end}} {{end}}
{{if eq .TotalCount 0}} {{if eq .TotalCount 0}}
+157 -16
View File
@@ -107,9 +107,15 @@
.stat-card.connected .stat-value { .stat-card.connected .stat-value {
color: #27ae60; color: #27ae60;
} }
.stat-card.disconnected .stat-value { .stat-card.disconnected .stat-value,
.stale {
color: #e74c3c; color: #e74c3c;
} }
/* The card has no room beside "Connected", so the spinner and "stale" go under it */
#freshness {
display: block;
font-size: 20px;
}
/* Search section */ /* Search section */
.search-section { .search-section {
@@ -174,6 +180,38 @@
#asname-results .search-message { #asname-results .search-message {
font-size: 13px; font-size: 13px;
} }
#asname-results .error {
color: #c00;
}
.asname-list {
list-style: none;
margin: 12px 0 0 0;
padding: 0;
max-height: 400px;
overflow-y: auto;
}
.asname-list li {
border-bottom: 1px solid #eee;
}
.asname-list a {
display: block;
padding: 8px 4px;
color: #2c3e50;
text-decoration: none;
font-size: 14px;
}
.asname-list a:hover {
background: #f5f5f5;
}
.asname-list .asn {
font-weight: 600;
color: #3498db;
}
.asname-list .description {
display: block;
color: #7f8c8d;
font-size: 12px;
}
/* IP Lookup result */ /* IP Lookup result */
.ip-result { .ip-result {
@@ -265,7 +303,10 @@
<main class="main-content"> <main class="main-content">
<div class="stats-grid"> <div class="stats-grid">
<div class="stat-card" id="status-card"> <div class="stat-card" id="status-card">
<div class="stat-value" id="stat-status">-</div> <div class="stat-value">
<span id="stat-status">-</span>
<span id="freshness"></span>
</div>
<div class="stat-label">Status</div> <div class="stat-label">Status</div>
</div> </div>
<div class="stat-card"> <div class="stat-card">
@@ -306,7 +347,7 @@
<input type="text" id="asname-input" placeholder="e.g., Google, Cloudflare" autocomplete="off"> <input type="text" id="asname-input" placeholder="e.g., Google, Cloudflare" autocomplete="off">
<button type="submit">Search</button> <button type="submit">Search</button>
</form> </form>
<p class="search-hint">Search for autonomous systems by organization name</p> <p class="search-hint">Search for autonomous systems by organization name or handle</p>
<div id="asname-results"></div> <div id="asname-results"></div>
</div> </div>
@@ -337,6 +378,44 @@
</footer> </footer>
<script nonce="{{.CSPNonce}}"> <script nonce="{{.CSPNonce}}">
// The stats are fetched every REFRESH_PERIOD_MS. Next to the status, a
// spinner turns one frame each time an answer counts more messages
// than the one before. After STALE_AFTER_REFRESHES refreshes without
// such an answer, the spinner gives way to "stale" and the time since.
const REFRESH_PERIOD_MS = 5000;
const STALE_AFTER_REFRESHES = 3;
const MS_PER_SECOND = 1000;
const SPINNER_FRAMES = ['⠋', '⠙', '⠹', '⠸', '⠼', '⠴', '⠦', '⠧', '⠇', '⠏'];
const SPINNER_COLOURS = ['#e74c3c', '#e67e22', '#f1c40f', '#27ae60', '#3498db', '#9b59b6'];
let spinnerFrame = 0;
let lastMessageCount = null;
let lastFreshAt = Date.now();
// Turn the spinner when the message count has risen since the last answer.
function noteMessageCount(count) {
if (lastMessageCount !== null && count > lastMessageCount) {
spinnerFrame = (spinnerFrame + 1) % SPINNER_FRAMES.length;
lastFreshAt = Date.now();
}
lastMessageCount = count;
showFreshness();
}
function showFreshness() {
const freshness = document.getElementById('freshness');
const sinceFreshMs = Date.now() - lastFreshAt;
const stale = sinceFreshMs >= STALE_AFTER_REFRESHES * REFRESH_PERIOD_MS;
freshness.classList.toggle('stale', stale);
if (stale) {
freshness.textContent = 'stale for ' + Math.floor(sinceFreshMs / MS_PER_SECOND) + 's';
freshness.style.color = '';
return;
}
freshness.textContent = SPINNER_FRAMES[spinnerFrame];
freshness.style.color = SPINNER_COLOURS[spinnerFrame % SPINNER_COLOURS.length];
}
function formatNumber(num) { function formatNumber(num) {
if (num >= 1000000) { if (num >= 1000000) {
return (num / 1000000).toFixed(1) + 'M'; return (num / 1000000).toFixed(1) + 'M';
@@ -358,6 +437,7 @@
const statusEl = document.getElementById('stat-status'); const statusEl = document.getElementById('stat-status');
statusEl.textContent = data.connected ? 'Connected' : 'Disconnected'; statusEl.textContent = data.connected ? 'Connected' : 'Disconnected';
statusCard.className = 'stat-card ' + (data.connected ? 'connected' : 'disconnected'); statusCard.className = 'stat-card ' + (data.connected ? 'connected' : 'disconnected');
noteMessageCount(data.total_messages);
document.getElementById('stat-routes').textContent = formatNumber(data.live_routes); document.getElementById('stat-routes').textContent = formatNumber(data.live_routes);
document.getElementById('stat-asns').textContent = formatNumber(data.asns); document.getElementById('stat-asns').textContent = formatNumber(data.asns);
@@ -368,7 +448,7 @@
} }
const totalUpdates = data.ipv4_updates_per_sec + data.ipv6_updates_per_sec; const totalUpdates = data.ipv4_updates_per_sec + data.ipv6_updates_per_sec;
document.getElementById('stat-updates').textContent = totalUpdates.toFixed(1); document.getElementById('stat-updates').textContent = Math.round(totalUpdates).toLocaleString();
}) })
.catch(() => { .catch(() => {
document.getElementById('stat-status').textContent = 'Error'; document.getElementById('stat-status').textContent = 'Error';
@@ -387,24 +467,82 @@
} }
}); });
// AS name search // AS name search. The handles and descriptions come from outside data,
// so the results are built with textContent, never innerHTML.
const asNameResults = document.getElementById('asname-results');
// Replace the search results with one line of text.
function showASNameMessage(text, className) {
const line = document.createElement('p');
line.className = className;
line.textContent = text;
asNameResults.replaceChildren(line);
}
// List the matches of a search answer, each as a link to its AS page,
// and say how many more matched than the answer holds.
function showASNameMatches(data) {
if (data.total === 0) {
showASNameMessage('No AS matches "' + data.query + '".', 'search-message');
return;
}
const list = document.createElement('ul');
list.className = 'asname-list';
for (const match of data.results) {
const asn = document.createElement('span');
asn.className = 'asn';
asn.textContent = 'AS' + match.asn;
const description = document.createElement('span');
description.className = 'description';
description.textContent = match.description;
const link = document.createElement('a');
link.href = '/as/' + match.asn;
link.append(asn, ' ' + match.handle, description);
const item = document.createElement('li');
item.append(link);
list.append(item);
}
asNameResults.replaceChildren(list);
const more = data.total - data.results.length;
if (more > 0) {
const line = document.createElement('p');
line.className = 'search-message';
line.textContent = more.toLocaleString() + ' more matched. Narrow the search to see them.';
asNameResults.append(line);
}
}
document.getElementById('asname-form').addEventListener('submit', function(e) { document.getElementById('asname-form').addEventListener('submit', function(e) {
e.preventDefault(); e.preventDefault();
const query = document.getElementById('asname-input').value.trim(); const query = document.getElementById('asname-input').value.trim();
if (!query) return; if (!query) return;
const resultsDiv = document.getElementById('asname-results'); // A number, or AS and a number, opens that AS's page
resultsDiv.innerHTML = '<p class="loading">Searching...</p>'; const asn = query.replace(/^AS/i, '');
if (/^\d+$/.test(asn)) {
// Use a simple client-side search against the asinfo data window.location.href = '/as/' + asn;
// For now, redirect to AS page if it looks like an ASN
if (/^\d+$/.test(query)) {
window.location.href = '/as/' + query;
return; return;
} }
// Show a message that server-side search is coming showASNameMessage('Searching...', 'loading');
resultsDiv.innerHTML = '<p class="search-message">AS name search coming soon. For now, try an AS number.</p>';
fetch('/api/v1/search/as?q=' + encodeURIComponent(query))
.then(response => response.json())
.then(response => {
if (response.status === 'error') {
showASNameMessage('Error: ' + response.error.msg, 'error');
} else {
showASNameMatches(response.data);
}
})
.catch(error => {
showASNameMessage('Error: ' + error.message, 'error');
});
}); });
// IP lookup // IP lookup
@@ -446,9 +584,12 @@
document.getElementById('ip-input').value = ''; document.getElementById('ip-input').value = '';
}); });
// Initial load and refresh stats every 5 seconds // Initial load and refresh stats every REFRESH_PERIOD_MS. The freshness
// check runs every second on its own, so a request that never returns
// still turns the spinner into "stale".
updateStats(); updateStats();
setInterval(updateStats, 5000); setInterval(updateStats, REFRESH_PERIOD_MS);
setInterval(showFreshness, MS_PER_SECOND);
</script> </script>
</body> </body>
</html> </html>
+58 -12
View File
@@ -126,7 +126,8 @@
.connected { .connected {
color: #22c55e; color: #22c55e;
} }
.disconnected { .disconnected,
.stale {
color: #ef4444; color: #ef4444;
} }
.error { .error {
@@ -180,7 +181,10 @@
<h2>RouteWatch</h2> <h2>RouteWatch</h2>
<div class="metric"> <div class="metric">
<span class="metric-label">Status</span> <span class="metric-label">Status</span>
<span class="metric-value" id="connected">-</span> <span>
<span class="metric-value" id="connected">-</span>
<span class="metric-value" id="freshness"></span>
</span>
</div> </div>
<div class="metric"> <div class="metric">
<span class="metric-label">Uptime</span> <span class="metric-label">Uptime</span>
@@ -231,8 +235,8 @@
<span class="metric-value" id="withdrawals">-</span> <span class="metric-value" id="withdrawals">-</span>
</div> </div>
<div class="metric"> <div class="metric">
<span class="metric-label">Route Churn/sec</span> <span class="metric-label">Route Churn/min</span>
<span class="metric-value" id="route_churn_per_sec">-</span> <span class="metric-value" id="route_churn_per_min">-</span>
</div> </div>
<div class="metric"> <div class="metric">
<span class="metric-label">Total Data</span> <span class="metric-label">Total Data</span>
@@ -386,6 +390,44 @@
</div> </div>
<script nonce="{{.CSPNonce}}"> <script nonce="{{.CSPNonce}}">
// The stats are fetched every REFRESH_PERIOD_MS. Next to the status, a
// spinner turns one frame each time an answer counts more messages
// than the one before. After STALE_AFTER_REFRESHES refreshes without
// such an answer, the spinner gives way to "stale" and the time since.
const REFRESH_PERIOD_MS = 2000;
const STALE_AFTER_REFRESHES = 3;
const MS_PER_SECOND = 1000;
const SPINNER_FRAMES = ['⠋', '⠙', '⠹', '⠸', '⠼', '⠴', '⠦', '⠧', '⠇', '⠏'];
const SPINNER_COLOURS = ['#e74c3c', '#e67e22', '#f1c40f', '#27ae60', '#3498db', '#9b59b6'];
let spinnerFrame = 0;
let lastMessageCount = null;
let lastFreshAt = Date.now();
// Turn the spinner when the message count has risen since the last answer.
function noteMessageCount(count) {
if (lastMessageCount !== null && count > lastMessageCount) {
spinnerFrame = (spinnerFrame + 1) % SPINNER_FRAMES.length;
lastFreshAt = Date.now();
}
lastMessageCount = count;
showFreshness();
}
function showFreshness() {
const freshness = document.getElementById('freshness');
const sinceFreshMs = Date.now() - lastFreshAt;
const stale = sinceFreshMs >= STALE_AFTER_REFRESHES * REFRESH_PERIOD_MS;
freshness.classList.toggle('stale', stale);
if (stale) {
freshness.textContent = 'stale for ' + Math.floor(sinceFreshMs / MS_PER_SECOND) + 's';
freshness.style.color = '';
return;
}
freshness.textContent = SPINNER_FRAMES[spinnerFrame];
freshness.style.color = SPINNER_COLOURS[spinnerFrame % SPINNER_COLOURS.length];
}
function formatBytes(bytes) { function formatBytes(bytes) {
if (bytes === 0) return '0 B'; if (bytes === 0) return '0 B';
const k = 1024; const k = 1024;
@@ -511,7 +553,7 @@
document.getElementById('messages_per_sec').textContent = '-'; document.getElementById('messages_per_sec').textContent = '-';
document.getElementById('announcements').textContent = '-'; document.getElementById('announcements').textContent = '-';
document.getElementById('withdrawals').textContent = '-'; document.getElementById('withdrawals').textContent = '-';
document.getElementById('route_churn_per_sec').textContent = '-'; document.getElementById('route_churn_per_min').textContent = '-';
document.getElementById('total_wire_bytes').textContent = '-'; document.getElementById('total_wire_bytes').textContent = '-';
document.getElementById('wire_mbits_per_sec').textContent = '-'; document.getElementById('wire_mbits_per_sec').textContent = '-';
document.getElementById('gc_num').textContent = '-'; document.getElementById('gc_num').textContent = '-';
@@ -569,7 +611,8 @@
const connectedEl = document.getElementById('connected'); const connectedEl = document.getElementById('connected');
connectedEl.textContent = data.connected ? 'Connected' : 'Disconnected'; connectedEl.textContent = data.connected ? 'Connected' : 'Disconnected';
connectedEl.className = 'metric-value ' + (data.connected ? 'connected' : 'disconnected'); connectedEl.className = 'metric-value ' + (data.connected ? 'connected' : 'disconnected');
noteMessageCount(data.total_messages);
// Update all metrics // Update all metrics
document.getElementById('uptime').textContent = data.uptime; document.getElementById('uptime').textContent = data.uptime;
document.getElementById('go_version').textContent = data.go_version; document.getElementById('go_version').textContent = data.go_version;
@@ -578,7 +621,7 @@
document.getElementById('connection_duration').textContent = data.connection_duration; document.getElementById('connection_duration').textContent = data.connection_duration;
document.getElementById('reconnect_count').textContent = formatNumber(data.reconnect_count); document.getElementById('reconnect_count').textContent = formatNumber(data.reconnect_count);
document.getElementById('total_messages').textContent = formatNumber(data.total_messages); document.getElementById('total_messages').textContent = formatNumber(data.total_messages);
document.getElementById('messages_per_sec').textContent = data.messages_per_sec.toFixed(1); document.getElementById('messages_per_sec').textContent = formatNumber(Math.round(data.messages_per_sec));
document.getElementById('total_wire_bytes').textContent = formatBytes(data.total_wire_bytes); document.getElementById('total_wire_bytes').textContent = formatBytes(data.total_wire_bytes);
document.getElementById('wire_mbits_per_sec').textContent = data.wire_mbits_per_sec.toFixed(2) + ' Mbps'; document.getElementById('wire_mbits_per_sec').textContent = data.wire_mbits_per_sec.toFixed(2) + ' Mbps';
document.getElementById('asns').textContent = formatNumber(data.asns); document.getElementById('asns').textContent = formatNumber(data.asns);
@@ -591,8 +634,8 @@
document.getElementById('live_routes').textContent = formatNumber(data.live_routes); document.getElementById('live_routes').textContent = formatNumber(data.live_routes);
document.getElementById('ipv4_routes').textContent = formatNumber(data.ipv4_routes); document.getElementById('ipv4_routes').textContent = formatNumber(data.ipv4_routes);
document.getElementById('ipv6_routes').textContent = formatNumber(data.ipv6_routes); document.getElementById('ipv6_routes').textContent = formatNumber(data.ipv6_routes);
document.getElementById('ipv4_updates_per_sec').textContent = data.ipv4_updates_per_sec.toFixed(1); document.getElementById('ipv4_updates_per_sec').textContent = formatNumber(Math.round(data.ipv4_updates_per_sec));
document.getElementById('ipv6_updates_per_sec').textContent = data.ipv6_updates_per_sec.toFixed(1); document.getElementById('ipv6_updates_per_sec').textContent = formatNumber(Math.round(data.ipv6_updates_per_sec));
document.getElementById('oldest_route').textContent = formatRelativeTime(data.oldest_route); document.getElementById('oldest_route').textContent = formatRelativeTime(data.oldest_route);
document.getElementById('newest_route').textContent = formatRelativeTime(data.newest_route); document.getElementById('newest_route').textContent = formatRelativeTime(data.newest_route);
@@ -601,7 +644,7 @@
document.getElementById('bgp_peer_count').textContent = formatNumber(data.stream.bgp_peer_count); document.getElementById('bgp_peer_count').textContent = formatNumber(data.stream.bgp_peer_count);
document.getElementById('announcements').textContent = formatNumber(data.stream.announcements); document.getElementById('announcements').textContent = formatNumber(data.stream.announcements);
document.getElementById('withdrawals').textContent = formatNumber(data.stream.withdrawals); document.getElementById('withdrawals').textContent = formatNumber(data.stream.withdrawals);
document.getElementById('route_churn_per_sec').textContent = data.stream.route_churn_per_sec.toFixed(1); document.getElementById('route_churn_per_min').textContent = formatNumber(data.stream.route_churn_per_min);
} }
// Update GC stats // Update GC stats
@@ -643,9 +686,12 @@
}); });
} }
// Update immediately and then every 2 seconds // Update immediately and then every REFRESH_PERIOD_MS. The freshness
// check runs every second on its own, so a request that never returns
// still turns the spinner into "stale".
updateStatus(); updateStatus();
setInterval(updateStatus, 2000); setInterval(updateStatus, REFRESH_PERIOD_MS);
setInterval(showFreshness, MS_PER_SECOND);
</script> </script>
</main> </main>
+3 -44
View File
@@ -7,9 +7,9 @@ import (
"net/url" "net/url"
"strings" "strings"
"sync" "sync"
"time"
"git.eeqj.de/sneak/routewatch/internal/version" "git.eeqj.de/sneak/routewatch/internal/version"
"github.com/dustin/go-humanize"
) )
//go:embed index.html //go:embed index.html
@@ -48,48 +48,7 @@ var (
once sync.Once once sync.Once
) )
const ( const cidrPartCount = 2 // A CIDR has two parts: prefix and length
hoursPerDay = 24
daysPerMonth = 30
cidrPartCount = 2 // A CIDR has two parts: prefix and length
)
// timeSince returns a human-readable duration since the given time
func timeSince(t time.Time) string {
duration := time.Since(t)
if duration < time.Minute {
return "just now"
}
if duration < time.Hour {
minutes := int(duration.Minutes())
if minutes == 1 {
return "1 minute ago"
}
return duration.Truncate(time.Minute).String() + " ago"
}
if duration < hoursPerDay*time.Hour {
hours := int(duration.Hours())
if hours == 1 {
return "1 hour ago"
}
return duration.Truncate(time.Hour).String() + " ago"
}
days := int(duration.Hours() / hoursPerDay)
if days == 1 {
return "1 day ago"
}
if days < daysPerMonth {
return duration.Truncate(hoursPerDay*time.Hour).String() + " ago"
}
return t.Format("2006-01-02")
}
// prefixURL generates a URL path for a prefix in CIDR notation. // prefixURL generates a URL path for a prefix in CIDR notation.
// Takes a prefix like "192.168.1.0/24" and returns "/prefix/192.168.1.0/24" // Takes a prefix like "192.168.1.0/24" and returns "/prefix/192.168.1.0/24"
@@ -113,7 +72,7 @@ func initTemplates() {
// Create common template functions // Create common template functions
funcs := template.FuncMap{ funcs := template.FuncMap{
"timeSince": timeSince, "timeSince": humanize.Time,
"urlEncode": url.QueryEscape, "urlEncode": url.QueryEscape,
"prefixURL": prefixURL, "prefixURL": prefixURL,
"appName": func() string { return version.Name }, "appName": func() string { return version.Name },
+134 -8
View File
@@ -6,9 +6,24 @@ import (
"strings" "strings"
"testing" "testing"
"git.eeqj.de/sneak/routewatch/internal/database"
"git.eeqj.de/sneak/routewatch/internal/templates" "git.eeqj.de/sneak/routewatch/internal/templates"
) )
// renderPage renders a page with the data every page needs.
func renderPage(t *testing.T, tmpl *template.Template) string {
t.Helper()
var page bytes.Buffer
err := tmpl.Execute(&page, struct{ CSPNonce string }{})
if err != nil {
t.Fatalf("rendering the page: %v", err)
}
return page.String()
}
// TestFooterLicense renders the home and status pages and expects each footer // TestFooterLicense renders the home and status pages and expects each footer
// to name the licence in LICENSE, MIT, and not WTFPL. // to name the licence in LICENSE, MIT, and not WTFPL.
func TestFooterLicense(t *testing.T) { func TestFooterLicense(t *testing.T) {
@@ -28,14 +43,7 @@ func TestFooterLicense(t *testing.T) {
t.Run(name, func(t *testing.T) { t.Run(name, func(t *testing.T) {
t.Parallel() t.Parallel()
var page bytes.Buffer _, footer, found := strings.Cut(renderPage(t, tmpl), "<footer")
err := tmpl.Execute(&page, struct{ CSPNonce string }{})
if err != nil {
t.Fatalf("rendering the page: %v", err)
}
_, footer, found := strings.Cut(page.String(), "<footer")
if !found { if !found {
t.Fatal("the page has no footer") t.Fatal("the page has no footer")
} }
@@ -52,3 +60,121 @@ func TestFooterLicense(t *testing.T) {
}) })
} }
} }
// TestFreshnessIndicator renders the home and status pages and expects each to
// have, right after the status element, the element where the page's script
// shows its spinner or "stale".
func TestFreshnessIndicator(t *testing.T) {
t.Parallel()
pages := map[string]struct {
tmpl *template.Template
status string
indicator string
}{
"index": {
tmpl: templates.IndexTemplate(),
status: `<span id="stat-status">-</span>`,
indicator: `<span id="freshness"></span>`,
},
"status": {
tmpl: templates.StatusTemplate(),
status: `<span class="metric-value" id="connected">-</span>`,
indicator: `<span class="metric-value" id="freshness"></span>`,
},
}
for name, page := range pages {
t.Run(name, func(t *testing.T) {
t.Parallel()
_, afterStatus, found := strings.Cut(renderPage(t, page.tmpl), page.status)
if !found {
t.Fatalf("the page has no status element %s", page.status)
}
if !strings.HasPrefix(strings.TrimSpace(afterStatus), page.indicator) {
t.Errorf("the status element is not followed by %s", page.indicator)
}
})
}
}
// TestASDetailPrefixListsFold renders the AS page with one IPv4 and one IPv6
// prefix. It expects each prefix list inside its own <details> element that
// starts open and whose <summary> holds the list's heading, so that a click on
// the heading folds the list.
func TestASDetailPrefixListsFold(t *testing.T) {
t.Parallel()
const (
asn = 64500
prefixV4 = "192.0.2.0/24"
prefixV6 = "2001:db8::/32"
)
ipv4 := []database.LiveRoute{{Prefix: prefixV4}}
ipv6 := []database.LiveRoute{{Prefix: prefixV6}}
// The fields the AS page handler gives the template.
data := struct {
ASN *database.ASN
IPv4Prefixes []database.LiveRoute
IPv6Prefixes []database.LiveRoute
TotalCount int
IPv4Count int
IPv6Count int
Peers []database.ASPeer
PeerCount int
CSPNonce string
}{
ASN: &database.ASN{ASN: asn},
IPv4Prefixes: ipv4,
IPv6Prefixes: ipv6,
TotalCount: len(ipv4) + len(ipv6),
IPv4Count: len(ipv4),
IPv6Count: len(ipv6),
}
var rendered bytes.Buffer
err := templates.ASDetailTemplate().Execute(&rendered, data)
if err != nil {
t.Fatalf("rendering the page: %v", err)
}
page := rendered.String()
lists := []struct{ heading, prefix string }{
{"IPv4 Prefixes", prefixV4},
{"IPv6 Prefixes", prefixV6},
}
for _, list := range lists {
before, _, found := strings.Cut(page, list.prefix)
if !found {
t.Fatalf("the page does not list %s", list.prefix)
}
// The list's section starts at the last <details before its prefix.
start := strings.LastIndex(before, "<details")
if start < 0 || strings.Contains(before[start:], "</details>") {
t.Errorf("%s: the list is not inside a <details> element", list.heading)
continue
}
tag, section, _ := strings.Cut(before[start:], ">")
if !strings.Contains(tag, " open") {
t.Errorf("%s: the <details> element does not start open: %s>",
list.heading, tag)
}
_, summary, _ := strings.Cut(section, "<summary")
summary, _, _ = strings.Cut(summary, "</summary>")
if !strings.Contains(summary, list.heading) {
t.Errorf("%s: the <summary> does not hold the heading", list.heading)
}
}
}
+284 -33
View File
@@ -7,10 +7,15 @@ import (
"errors" "errors"
"fmt" "fmt"
"io" "io"
"log/slog"
"net" "net"
"regexp" "regexp"
"strings" "strings"
"sync"
"time" "time"
"unicode/utf8"
"golang.org/x/text/encoding/charmap"
) )
// Timeout constants for WHOIS queries. // Timeout constants for WHOIS queries.
@@ -26,10 +31,18 @@ const maxAnswerSize = 1 << 20
// Parsing constants. // Parsing constants.
const ( const (
keyValueParts = 2 // Expected parts when splitting "key: value"
lacnicDateFormatLen = 8 // Length of YYYYMMDD date format lacnicDateFormatLen = 8 // Length of YYYYMMDD date format
) )
// The classes of the two objects that describe an AS in an answer from
// RIPE, APNIC or AFRINIC, and the key of the aut-num object's line that
// gives the ID of the AS's organisation object.
const (
autNumClass = "aut-num"
organisationClass = "organisation"
orgKey = "org"
)
// WHOIS server addresses. // WHOIS server addresses.
const ( const (
whoisServerIANA = "whois.iana.org:43" whoisServerIANA = "whois.iana.org:43"
@@ -38,8 +51,32 @@ const (
whoisServerAPNIC = "whois.apnic.net:43" whoisServerAPNIC = "whois.apnic.net:43"
whoisServerLACNIC = "whois.lacnic.net:43" whoisServerLACNIC = "whois.lacnic.net:43"
whoisServerAFRINIC = "whois.afrinic.net:43" whoisServerAFRINIC = "whois.afrinic.net:43"
// NIC.br registers Brazil's ASes under LACNIC.
whoisServerNICbr = "whois.registro.br:43"
) )
// nicbrHeader starts the copyright notice at the top of NIC.br's WHOIS
// answers. LACNIC's answer for an AS that NIC.br registers is NIC.br's, and
// carries the same notice:
//
// % Copyright (c) Nic.br - Use of this data is governed by the Use and
// % Privacy Policy at https://registro.br/upp . Distribution,
const nicbrHeader = "% Copyright (c) Nic.br"
// nicbrLimitReached is in the line that marks NIC.br's reduced answer to a
// client over its query limit. NIC.br publishes no limit. Its Use and
// Privacy Policy, https://registro.br/upp, says only that it restricts
// queries to stop the data being taken in bulk.
const nicbrLimitReached = "Query rate limit exceeded"
// NICbrSkipTime is how long the client stops asking NIC.br after NIC.br
// answers that the client is over its query limit. NIC.br publishes
// neither the limit nor how long it lasts. An hour outlasts a limit
// counted per minute or per hour. The ASes looked up meanwhile get
// NIC.br's fields when they are next refreshed.
const NICbrSkipTime = time.Hour
// RIR identifiers. // RIR identifiers.
const ( const (
RIRARIN = "ARIN" RIRARIN = "ARIN"
@@ -49,6 +86,31 @@ const (
RIRAFRNIC = "AFRINIC" RIRAFRNIC = "AFRINIC"
) )
// noPersonalFlag keeps person and role objects out of an answer. The
// organisation object stays in it. RIPE counts the person and role objects
// it returns against a daily limit per IP address. Role objects for abuse
// contacts do not count. RIPE refuses an address for good that passes the
// limit too often:
// https://docs.db.ripe.net/Types-of-Queries/Access-Control-for-Queries/
//
// RIPE documents the flag here:
// https://docs.db.ripe.net/Types-of-Queries/Referenced-Objects-in-Query-Response/
// APNIC and AFRINIC list it in their answers to the query "help". AFRINIC
// sends readers to that answer for its full list of flags:
// https://afrinic.net/whois-getting-started
const noPersonalFlag = "--no-personal"
// The error lines with which RIPE's database software refuses a query:
// access denied to the client's IP address, and that address's limit
// reached. APNIC and AFRINIC run the same software. The lines are listed
// under "Access Errors" on this page, whose URL is split over two lines:
// https://docs.db.ripe.net/Appendices/
// Appendix-C--RIPE-Database-Query-Server-Response-Codes-and-Messages/
const (
accessDeniedError = "%ERROR:201:"
accessLimitError = "%ERROR:202:"
)
// Errors LookupASN returns when a WHOIS answer has nothing to store. // Errors LookupASN returns when a WHOIS answer has nothing to store.
var ( var (
// ErrNoReferral is returned when IANA refers the AS to no RIR, as it // ErrNoReferral is returned when IANA refers the AS to no RIR, as it
@@ -58,6 +120,7 @@ var (
errNoASBlock = errors.New("IANA answered with neither a referral " + errNoASBlock = errors.New("IANA answered with neither a referral " +
"nor an AS block") "nor an AS block")
errNoFields = errors.New("RIR answered with no WHOIS fields") errNoFields = errors.New("RIR answered with no WHOIS fields")
errRefused = errors.New("RIR refused the query")
errUnknownServer = errors.New("IANA refers the AS to a server no RIR runs") errUnknownServer = errors.New("IANA refers the AS to a server no RIR runs")
errAnswerTooLarge = errors.New("WHOIS answer reached the size cap") errAnswerTooLarge = errors.New("WHOIS answer reached the size cap")
) )
@@ -86,21 +149,42 @@ type Client struct {
// host:port. NewClient sets a net.Dialer's; tests replace it to reach // host:port. NewClient sets a net.Dialer's; tests replace it to reach
// local servers instead. // local servers instead.
DialContext func(ctx context.Context, network, address string) (net.Conn, error) DialContext func(ctx context.Context, network, address string) (net.Conn, error)
// Now returns the current time. NewClient sets time.Now; tests replace
// it to move the clock.
Now func() time.Time
logger *slog.Logger
// mu guards nicbrSkipUntil.
mu sync.Mutex
// nicbrSkipUntil is when the client next asks NIC.br, after NIC.br
// answered that the client is over its query limit.
nicbrSkipUntil time.Time
} }
// NewClient creates a new WHOIS client. // NewClient creates a new WHOIS client that logs to logger.
func NewClient() *Client { func NewClient(logger *slog.Logger) *Client {
dialer := &net.Dialer{Timeout: dialTimeout} dialer := &net.Dialer{Timeout: dialTimeout}
return &Client{DialContext: dialer.DialContext} return &Client{
DialContext: dialer.DialContext,
Now: time.Now,
logger: logger,
}
} }
// LookupASN asks IANA which RIR holds the given ASN, asks that RIR, and // LookupASN asks IANA which RIR holds the given ASN, asks that RIR, and
// returns what the RIR's answer says. It returns ErrNoReferral when IANA // returns what the RIR's answer says. When LACNIC's answer is NIC.br's, it
// refers the AS to no RIR, and another error when a query fails, when IANA's // returns what NIC.br's own server says instead, unless that query fails or
// answer has neither a referral nor the AS's block, when IANA refers the AS // is skipped, as askNICbr says.
// to a server none of the five RIRs run, or when the RIR's answer carries //
// none of ASNInfo's fields, as a refusal of service does. // It returns ErrNoReferral when IANA refers the AS to no RIR, and another
// error when the query to IANA or to the RIR fails, when the RIR refuses
// it, when IANA's answer has neither a referral nor the AS's block, when
// IANA refers the AS to a server none of the five RIRs run, or when the
// answer carries none of ASNInfo's fields.
func (c *Client) LookupASN(ctx context.Context, asn int) (*ASNInfo, error) { func (c *Client) LookupASN(ctx context.Context, asn int) (*ASNInfo, error) {
// Query IANA first to find the authoritative RIR // Query IANA first to find the authoritative RIR
query := fmt.Sprintf("AS%d", asn) query := fmt.Sprintf("AS%d", asn)
@@ -127,23 +211,82 @@ func (c *Client) LookupASN(ctx context.Context, asn int) (*ASNInfo, error) {
} }
// Query the authoritative RIR // Query the authoritative RIR
rirResp, err := c.query(ctx, whoisServer, query) rirResp, err := c.queryRIR(ctx, whoisServer, rirQuery(rir, query))
if err != nil { if err != nil {
return nil, fmt.Errorf("RIR query failed: %w", err) return nil, fmt.Errorf("RIR query failed: %w", err)
} }
// The fields come from the RIR's answer alone: IANA's describes the // The fields never come from IANA's answer, which describes the block
// block the AS is in. // the AS is in.
info := c.parseResponse(asn, rir, rirResp) info := c.parseResponse(asn, rir, rirResp)
if rir == RIRLACNIC && strings.Contains(rirResp, nicbrHeader) {
info = c.askNICbr(ctx, query, info)
}
if info.isEmpty() { if info.isEmpty() {
return nil, fmt.Errorf("%s from %s: %w", query, whoisServer, errNoFields) return nil, fmt.Errorf("%s from %s: %w", query, whoisServer, errNoFields)
} }
info.RawResponse = ianaResp + "\n" + rirResp info.RawResponse = ianaResp + "\n" + info.RawResponse
return info, nil return info, nil
} }
// askNICbr asks NIC.br's server about an AS whose answer from LACNIC is
// NIC.br's, and returns what NIC.br's answer says. LACNIC replaces each
// accented letter in such an answer with U+FFFD. NIC.br's own server keeps
// them, in Latin-1.
//
// askNICbr returns lacnicInfo, what LACNIC's answer says, and logs a
// warning when the query fails, when NIC.br sends the reduced answer for a
// client over its limit, or when NIC.br's answer carries none of ASNInfo's
// fields. After the reduced answer, it returns lacnicInfo without asking
// NIC.br until NICbrSkipTime has passed.
func (c *Client) askNICbr(
ctx context.Context, asQuery string, lacnicInfo *ASNInfo,
) *ASNInfo {
asn := lacnicInfo.ASN
c.mu.Lock()
skipping := c.Now().Before(c.nicbrSkipUntil)
c.mu.Unlock()
if skipping {
return lacnicInfo
}
answer, err := c.query(ctx, whoisServerNICbr, asQuery)
if err != nil {
c.logger.Warn("NIC.br query failed, keeping LACNIC's fields",
"asn", asn, "error", err)
return lacnicInfo
}
if strings.Contains(answer, nicbrLimitReached) {
c.mu.Lock()
c.nicbrSkipUntil = c.Now().Add(NICbrSkipTime)
c.mu.Unlock()
c.logger.Warn("NIC.br answered that the client is over its query "+
"limit, keeping LACNIC's fields and skipping NIC.br",
"asn", asn, "skip_for", NICbrSkipTime)
return lacnicInfo
}
info := c.parseResponse(asn, RIRLACNIC, answer)
if info.isEmpty() {
c.logger.Warn("NIC.br answered with no WHOIS fields, "+
"keeping LACNIC's fields", "asn", asn)
return lacnicInfo
}
return info
}
// isEmpty reports whether info holds none of the fields a WHOIS answer // isEmpty reports whether info holds none of the fields a WHOIS answer
// fills, as when the answer is only comment and error lines. // fills, as when the answer is only comment and error lines.
func (info *ASNInfo) isEmpty() bool { func (info *ASNInfo) isEmpty() bool {
@@ -154,8 +297,41 @@ func (info *ASNInfo) isEmpty() bool {
info.RegDate == nil && info.LastMod == nil info.RegDate == nil && info.LastMod == nil
} }
// query performs a raw WHOIS query to the specified server. An answer that // rirQuery returns the query sent to rir's server for asQuery, an AS
// reaches maxAnswerSize is an error. // number such as "AS3333". RIPE, APNIC and AFRINIC are asked to keep person
// and role objects out of the answer.
func rirQuery(rir, asQuery string) string {
switch rir {
case RIRRIPE, RIRAPNIC, RIRAFRNIC:
return noPersonalFlag + " " + asQuery
default:
return asQuery
}
}
// queryRIR sends query to an RIR's server and returns the answer. An answer
// holding an error line with which the server refuses the query is an
// error, and the error holds that line.
func (c *Client) queryRIR(
ctx context.Context, server, query string,
) (string, error) {
answer, err := c.query(ctx, server, query)
if err != nil {
return "", err
}
for line := range strings.SplitSeq(answer, "\n") {
if strings.HasPrefix(line, accessDeniedError) ||
strings.HasPrefix(line, accessLimitError) {
return "", fmt.Errorf("%w: %s", errRefused, line)
}
}
return answer, nil
}
// query performs a raw WHOIS query to the specified server and returns the
// answer in UTF-8. An answer that reaches maxAnswerSize is an error.
func (c *Client) query(ctx context.Context, server, query string) (string, error) { func (c *Client) query(ctx context.Context, server, query string) (string, error) {
conn, err := c.DialContext(ctx, "tcp", server) conn, err := c.DialContext(ctx, "tcp", server)
if err != nil { if err != nil {
@@ -202,7 +378,25 @@ func (c *Client) query(ctx context.Context, server, query string) (string, error
return "", fmt.Errorf("read response: %w", errAnswerTooLarge) return "", fmt.Errorf("read response: %w", errAnswerTooLarge)
} }
return sb.String(), nil return toUTF8(sb.String())
}
// toUTF8 returns a WHOIS answer in UTF-8. An answer that is valid UTF-8 is
// returned as it is. Any other answer is decoded as Latin-1, which RIPE,
// APNIC and NIC.br send. LACNIC, ARIN and AFRINIC send UTF-8. RIPE's
// database working group lists the five RIRs here:
// https://www.ripe.net/ripe/mail/archives/db-wg/2024-May/008075.html
func toUTF8(answer string) (string, error) {
if utf8.ValidString(answer) {
return answer, nil
}
decoded, err := charmap.ISO8859_1.NewDecoder().String(answer)
if err != nil {
return "", fmt.Errorf("decode Latin-1 answer: %w", err)
}
return decoded, nil
} }
// parseIANAReferral extracts the RIR and WHOIS server from an IANA response. // parseIANAReferral extracts the RIR and WHOIS server from an IANA response.
@@ -238,7 +432,9 @@ func (c *Client) parseIANAReferral(response string) (string, string, error) {
return "", "", nil return "", "", nil
} }
// parseResponse extracts ASN information from a WHOIS response. // parseResponse extracts ASN information from a WHOIS response. Of an
// answer from RIPE, APNIC or AFRINIC it reads only the lines asObjectLines
// returns. Of an answer from ARIN or LACNIC it reads every line.
func (c *Client) parseResponse(asn int, rir, response string) *ASNInfo { func (c *Client) parseResponse(asn int, rir, response string) *ASNInfo {
info := &ASNInfo{ info := &ASNInfo{
ASN: asn, ASN: asn,
@@ -247,26 +443,15 @@ func (c *Client) parseResponse(asn int, rir, response string) *ASNInfo {
} }
lines := strings.Split(response, "\n") lines := strings.Split(response, "\n")
if rir == RIRRIPE || rir == RIRAPNIC || rir == RIRAFRNIC {
lines = asObjectLines(lines)
}
var addressLines []string var addressLines []string
for _, line := range lines { for _, line := range lines {
line = strings.TrimSpace(line) key, value, ok := splitField(line)
if line == "" || strings.HasPrefix(line, "%") || if !ok || value == "" {
strings.HasPrefix(line, "#") {
continue
}
// Split on first colon
parts := strings.SplitN(line, ":", keyValueParts)
if len(parts) != keyValueParts {
continue
}
key := strings.TrimSpace(strings.ToLower(parts[0]))
value := strings.TrimSpace(parts[1])
if value == "" {
continue continue
} }
@@ -293,6 +478,72 @@ func (c *Client) parseResponse(asn int, rir, response string) *ASNInfo {
return info return info
} }
// splitField splits a "key: value" line of a WHOIS answer into its key, in
// lower case, and its value. The third result is false for a line that
// holds no field: a blank line, a comment line or a line without a colon.
func splitField(line string) (string, string, bool) {
line = strings.TrimSpace(line)
if strings.HasPrefix(line, "%") || strings.HasPrefix(line, "#") {
return "", "", false
}
key, value, ok := strings.Cut(line, ":")
return strings.ToLower(strings.TrimSpace(key)), strings.TrimSpace(value), ok
}
// asObjectLines returns the lines of the aut-num object in an answer from
// RIPE, APNIC or AFRINIC, followed by the lines of the organisation object
// whose ID the aut-num object gives on its "org:" line. The answer's other
// objects are left out. RIPE and APNIC start their answer with the
// as-block object of the block the AS is in, whose dates are not the AS's.
//
// These answers are in RPSL. A blank line ends an object. An object's first
// line has the object's class as its key and the object's ID as its value,
// as in "organisation: ORG-APNI1-AP".
func asObjectLines(lines []string) []string {
var autNum []string
// The lines of each organisation object, by its ID.
organisations := make(map[string][]string)
// The class and ID of the object the line is in, "" between objects.
var class, id string
// The ID of the organisation object the aut-num object refers to.
var orgID string
for _, line := range lines {
if strings.TrimSpace(line) == "" {
class = ""
continue
}
key, value, ok := splitField(line)
if !ok {
continue
}
if class == "" {
class, id = key, value
}
switch class {
case autNumClass:
autNum = append(autNum, line)
if key == orgKey {
orgID = value
}
case organisationClass:
organisations[id] = append(organisations[id], line)
}
}
return append(autNum, organisations[orgID]...)
}
// setResponseField stores the value of one "key: value" line of a WHOIS // setResponseField stores the value of one "key: value" line of a WHOIS
// response in info, keeping the first value seen for each field. // response in info, keeping the first value seen for each field.
func (c *Client) setResponseField(info *ASNInfo, key, value string) { func (c *Client) setResponseField(info *ASNInfo, key, value string) {
+663 -28
View File
@@ -2,15 +2,21 @@ package whois_test
import ( import (
"bufio" "bufio"
"bytes"
"context" "context"
"encoding/json"
"errors" "errors"
"io" "io"
"log/slog"
"maps"
"net" "net"
"slices"
"strings" "strings"
"sync" "sync"
"syscall" "syscall"
"testing" "testing"
"time" "time"
"unicode/utf8"
"git.eeqj.de/sneak/routewatch/internal/whois" "git.eeqj.de/sneak/routewatch/internal/whois"
) )
@@ -21,8 +27,20 @@ const (
ripeServer = "whois.ripe.net:43" ripeServer = "whois.ripe.net:43"
) )
// testASN is the AS the tests look up. // testASN is the AS the tests look up, and asQuery the query for it.
const testASN = 3333 const (
testASN = 3333
asQuery = "AS3333"
)
// The fake servers' answers are keyed by a server's address, a space and a
// query. These are the keys of the queries LookupASN sends for testASN:
// IANA's, then RIPE's, which asks RIPE to leave out person and role
// objects.
const (
ianaQuery = ianaServer + " " + asQuery
ripeQuery = ripeServer + " --no-personal " + asQuery
)
// ianaReferral is IANA's answer for an AS in a block it gave to RIPE. // ianaReferral is IANA's answer for an AS in a block it gave to RIPE.
const ianaReferral = `% IANA WHOIS server const ianaReferral = `% IANA WHOIS server
@@ -36,27 +54,163 @@ changed: 2002-09-01
source: IANA source: IANA
` `
// ripeRecord is RIPE's answer for the AS. // lacnicReferral is IANA's answer for an AS in a block it gave to LACNIC,
// shortened to the referral.
const lacnicReferral = "refer: whois.lacnic.net\n"
// ripeRecord is RIPE's answer for the AS when asked to leave out person and
// role objects. It starts with the as-block object of the block the AS is
// in. Then come the abuse address as a comment, the AS's object and the
// organisation object it refers to.
const ripeRecord = `% This is the RIPE Database query service. const ripeRecord = `% This is the RIPE Database query service.
% The objects are in RPSL format.
% Note: --no-personal means ALL personal data has been filtered from this output.
% Information related to 'AS3209 - AS3353'
as-block: AS3209 - AS3353
descr: RIPE NCC ASN block
remarks: These AS Numbers are assigned to network operators
remarks: in the RIPE NCC service region.
mnt-by: RIPE-NCC-HM-MNT
created: 2018-11-22T15:27:19Z
last-modified: 2018-11-22T15:27:19Z
source: RIPE
% Information related to 'AS3333'
% Abuse contact for 'AS3333' is 'abuse@ripe.net'
aut-num: AS3333 aut-num: AS3333
as-name: RIPE-NCC-AS as-name: RIPE-NCC-AS
org: ORG-RIEN1-RIPE org: ORG-RIEN1-RIPE
admin-c: BRD-RIPE
tech-c: OPS4-RIPE
abuse-c: OPS4-RIPE
mnt-by: RIPE-NCC-MNT
created: 2002-09-17T13:08:32Z created: 2002-09-17T13:08:32Z
last-modified: 2024-01-09T08:55:41Z last-modified: 2024-01-09T08:55:41Z
source: RIPE source: RIPE
organisation: ORG-RIEN1-RIPE
org-name: Reseaux IP Europeens Network Coordination Centre (RIPE NCC)
country: NL
org-type: RIR
address: P.O. Box 10096
address: 1001EB Amsterdam
abuse-c: OPS4-RIPE
created: 2012-03-09T13:08:29Z
last-modified: 2024-03-26T10:55:38Z
source: RIPE
` `
// The error lines with which RIPE refuses a query: access denied to the
// host, and the host's limit on returned objects reached.
const (
ripeAccessDenied = "%ERROR:201: access denied for 192.0.2.1"
ripeLimitReached = "%ERROR:202: access control limit reached"
)
// ripeRefusal is RIPE's answer to a host it no longer serves. // ripeRefusal is RIPE's answer to a host it no longer serves.
const ripeRefusal = `% This is the RIPE Database query service. const ripeRefusal = `% This is the RIPE Database query service.
% The objects are in RPSL format. % The objects are in RPSL format.
% %
%ERROR:201: access denied for 192.0.2.1 ` + ripeAccessDenied + `
% %
% Sorry, access from your host has been permanently % Sorry, access from your host has been permanently
% denied because of a repeated excessive querying. % denied because of a repeated excessive querying.
` `
// apnicAfrinicRecord is the answer of APNIC or AFRINIC for the AS when asked
// to leave out person and role objects. Like RIPE's, it holds the abuse
// address as a comment, the AS's object and the organisation object.
const apnicAfrinicRecord = `% Abuse contact for 'AS3333' is 'abuse@example.net'
aut-num: AS3333
as-name: EXAMPLE-AS
org: ORG-EXAMPLE1
admin-c: EX1-TEST
last-modified: 2026-08-26T00:40:00Z
organisation: ORG-EXAMPLE1
org-name: Example Networks
country: ZA
address: 1 Example Road
`
// apnicServer is APNIC's WHOIS server, and apnicASN an AS that APNIC holds.
const (
apnicServer = "whois.apnic.net:43"
apnicASN = 4608
)
// The keys of the queries LookupASN sends for apnicASN: IANA's, then
// APNIC's, which asks APNIC to leave out person and role objects.
const (
apnicIANAQuery = ianaServer + " AS4608"
apnicQuery = apnicServer + " --no-personal AS4608"
)
// apnicRecord is APNIC's answer for apnicASN when asked to leave out person
// and role objects, shortened. It starts with the as-block object of the
// block the AS is in. Then come the abuse address as a comment, the AS's
// object, and the irt and organisation objects it refers to.
const apnicRecord = `% [whois.apnic.net]
% Whois data copyright terms http://www.apnic.net/db/dbcopyright.html
% Note: --no-personal means ALL personal data has been filtered from this output.
% Information related to 'AS4608 - AS4865'
as-block: AS4608 - AS4865
descr: APNIC ASN block
remarks: These AS numbers are further assigned by APNIC
remarks: to APNIC members and end-users in the APNIC region
mnt-by: APNIC-HM
mnt-lower: APNIC-HM
last-modified: 2019-05-20T03:52:26Z
source: APNIC
% Information related to 'AS4608'
% Abuse contact for 'AS4608' is 'abuse@apnic.net'
aut-num: AS4608
as-name: APNIC-SERVICES
descr: Asia Pacific Network Information Centre
country: AU
org: ORG-APNI1-AP
admin-c: AIC1-AP
abuse-c: AA1589-AP
mnt-by: APNIC-HM
mnt-irt: IRT-APNIC-IS-AP
last-modified: 2026-08-26T00:40:00Z
source: APNIC
irt: IRT-APNIC-IS-AP
address: South Brisbane, Australia
e-mail: helpdesk@apnic.net
abuse-mailbox: abuse@apnic.net
mnt-by: MAINT-APNIC-IS-AP
last-modified: 2025-11-18T00:27:08Z
source: APNIC
organisation: ORG-APNI1-AP
org-name: Asia Pacific Network Information Centre
org-type: LIR
country: AU
address: PO Box 3646
address: South Brisbane, QLD 4101,
address: Australia
e-mail: noc@apnic.net
mnt-by: APNIC-HM
last-modified: 2026-08-24T12:58:43Z
source: APNIC
% This query was served by the APNIC Whois Service version 1.88.48 (WHOIS-UK2)
`
// ianaPrivateUse is IANA's answer for a private-use AS: its block and no // ianaPrivateUse is IANA's answer for a private-use AS: its block and no
// referral. // referral.
const ianaPrivateUse = `% IANA WHOIS server const ianaPrivateUse = `% IANA WHOIS server
@@ -80,20 +234,29 @@ const fillerLine = "remarks: and more\n"
const endlessAnswerSize = 4 << 20 const endlessAnswerSize = 4 << 20
// fakeServers returns a DialContext for a whois.Client that connects to a // fakeServers returns a DialContext for a whois.Client that connects to a
// fake server for each address in answers, which reads the query and sends // fake server at each address answers names. Each key of answers is a
// answers[address]. Connecting to any other address is refused. // server's address, a space and a query, and its value is that server's
// answer to that query. A server reads one query and sends its answer, or
// nothing for a query it has no answer for. Connecting to any other address
// is refused.
func fakeServers( func fakeServers(
t *testing.T, answers map[string]string, t *testing.T, answers map[string]string,
) func(context.Context, string, string) (net.Conn, error) { ) func(context.Context, string, string) (net.Conn, error) {
t.Helper() t.Helper()
addresses := make(map[string]bool)
for key := range answers {
address, _, _ := strings.Cut(key, " ")
addresses[address] = true
}
var servers sync.WaitGroup var servers sync.WaitGroup
t.Cleanup(servers.Wait) t.Cleanup(servers.Wait)
return func(_ context.Context, _, address string) (net.Conn, error) { return func(_ context.Context, _, address string) (net.Conn, error) {
answer, ok := answers[address] if !addresses[address] {
if !ok {
return nil, syscall.ECONNREFUSED return nil, syscall.ECONNREFUSED
} }
@@ -102,7 +265,8 @@ func fakeServers(
servers.Go(func() { servers.Go(func() {
defer func() { _ = server.Close() }() defer func() { _ = server.Close() }()
_, _ = bufio.NewReader(server).ReadString('\n') query, _ := bufio.NewReader(server).ReadString('\n')
answer := answers[address+" "+strings.TrimSpace(query)]
_, _ = io.WriteString(server, answer) _, _ = io.WriteString(server, answer)
}) })
@@ -116,31 +280,48 @@ func lookUp(
) (*whois.ASNInfo, error) { ) (*whois.ASNInfo, error) {
t.Helper() t.Helper()
client := whois.NewClient() client := whois.NewClient(slog.New(slog.DiscardHandler))
client.DialContext = fakeServers(t, answers) client.DialContext = fakeServers(t, answers)
return client.LookupASN(t.Context(), testASN) return client.LookupASN(t.Context(), testASN)
} }
// TestLookupASNReadsTheRIRAnswer looks up an AS IANA refers to RIPE. The // TestLookupASNReadsTheRIRAnswer looks up an AS IANA refers to RIPE. RIPE
// fields come from RIPE's answer, the last-modified date included: IANA's // answers only a query that leaves out person and role objects, and its
// "changed:" line dates its block, not the AS. // answer to that still fills the AS's fields: the abuse address from the
// comment line, and the organisation's name and country from its object.
// The dates are the AS's. IANA's "changed:" line dates the AS's block, and
// so do the dates of the as-block object at the start of RIPE's answer.
func TestLookupASNReadsTheRIRAnswer(t *testing.T) { func TestLookupASNReadsTheRIRAnswer(t *testing.T) {
t.Parallel() t.Parallel()
info, err := lookUp(t, map[string]string{ info, err := lookUp(t, map[string]string{
ianaServer: ianaReferral, ianaQuery: ianaReferral,
ripeServer: ripeRecord, ripeQuery: ripeRecord,
}) })
if err != nil { if err != nil {
t.Fatalf("LookupASN: %v", err) t.Fatalf("LookupASN: %v", err)
} }
if info.ASName != "RIPE-NCC-AS" || info.OrgID != "ORG-RIEN1-RIPE" || got := []string{
info.RIR != whois.RIRRIPE { info.ASName, info.OrgID, info.OrgName, info.CountryCode,
t.Errorf("LookupASN = AS name %q, org %q, RIR %q; "+ info.AbuseEmail, info.RIR,
"want RIPE-NCC-AS, ORG-RIEN1-RIPE, RIPE", }
info.ASName, info.OrgID, info.RIR) want := []string{
"RIPE-NCC-AS", "ORG-RIEN1-RIPE",
"Reseaux IP Europeens Network Coordination Centre (RIPE NCC)", "NL",
"abuse@ripe.net", whois.RIRRIPE,
}
if !slices.Equal(got, want) {
t.Errorf("LookupASN = AS name, org ID, org name, country, abuse "+
"address, RIR %q; want %q", got, want)
}
wantRegDate := time.Date(2002, 9, 17, 13, 8, 32, 0, time.UTC)
if info.RegDate == nil || !info.RegDate.Equal(wantRegDate) {
t.Errorf("LookupASN registered = %v, want %v",
info.RegDate, wantRegDate)
} }
wantLastMod := time.Date(2024, 1, 9, 8, 55, 41, 0, time.UTC) wantLastMod := time.Date(2024, 1, 9, 8, 55, 41, 0, time.UTC)
@@ -150,6 +331,460 @@ func TestLookupASNReadsTheRIRAnswer(t *testing.T) {
} }
} }
// TestLookupASNReadsOnlyTheASAndItsOrganisation looks up an AS at APNIC,
// whose answer starts with the as-block object of the block the AS is in.
// The last-modified date is the aut-num object's, not the block's. The
// address is the organisation object's alone, without the irt object's.
func TestLookupASNReadsOnlyTheASAndItsOrganisation(t *testing.T) {
t.Parallel()
client := whois.NewClient(slog.New(slog.DiscardHandler))
client.DialContext = fakeServers(t, map[string]string{
apnicIANAQuery: "refer: whois.apnic.net\n",
apnicQuery: apnicRecord,
})
info, err := client.LookupASN(t.Context(), apnicASN)
if err != nil {
t.Fatalf("LookupASN: %v", err)
}
wantLastMod := time.Date(2026, 8, 26, 0, 40, 0, 0, time.UTC)
if info.LastMod == nil || !info.LastMod.Equal(wantLastMod) {
t.Errorf("LookupASN last modified = %v, want %v",
info.LastMod, wantLastMod)
}
got := []string{
info.ASName, info.OrgID, info.OrgName, info.CountryCode,
info.Address, info.AbuseEmail,
}
want := []string{
"APNIC-SERVICES", "ORG-APNI1-AP",
"Asia Pacific Network Information Centre", "AU",
"PO Box 3646\nSouth Brisbane, QLD 4101,\nAustralia", "abuse@apnic.net",
}
if !slices.Equal(got, want) {
t.Errorf("LookupASN = AS name, org ID, org name, country, address, "+
"abuse address %q; want %q", got, want)
}
}
// TestLookupASNAsksAPNICAndAFRINICToLeaveOutPersonalData looks up an AS at
// APNIC and at AFRINIC. Each answers only a query that leaves out person
// and role objects, as RIPE does. That one answer still holds the
// organisation object, so the organisation's name and country are filled.
func TestLookupASNAsksAPNICAndAFRINICToLeaveOutPersonalData(t *testing.T) {
t.Parallel()
for _, server := range []string{"whois.apnic.net", "whois.afrinic.net"} {
t.Run(server, func(t *testing.T) {
t.Parallel()
address := server + ":43"
info, err := lookUp(t, map[string]string{
ianaQuery: "refer: " + server + "\n",
address + " --no-personal " + asQuery: apnicAfrinicRecord,
})
if err != nil {
t.Fatalf("LookupASN: %v", err)
}
got := []string{
info.ASName, info.OrgName, info.CountryCode, info.AbuseEmail,
}
want := []string{
"EXAMPLE-AS", "Example Networks", "ZA", "abuse@example.net",
}
if !slices.Equal(got, want) {
t.Errorf("LookupASN = AS name, org name, country, abuse "+
"address %q; want %q", got, want)
}
})
}
}
// The name of an organisation with accented letters, in Latin-1, one byte for
// each letter, and in UTF-8.
const (
latin1OrgName = "Fundaci\xf3n Ra\xedces de Espa\xf1a"
utf8OrgName = "Fundación Raíces de España"
)
// TestLookupASNStoresAnswersInUTF8 looks up an AS at RIPE, which answers in
// Latin-1, and at LACNIC, which answers in UTF-8. Both answers give the
// organisation's name with its accented letters in UTF-8, in the parsed field
// and in the stored answer. Before the fix the Latin-1 answer was kept as
// received, which is not valid UTF-8.
func TestLookupASNStoresAnswersInUTF8(t *testing.T) {
t.Parallel()
for name, answers := range map[string]map[string]string{
"RIPE in Latin-1": {
ianaQuery: ianaReferral,
ripeQuery: "aut-num: AS3333\n" +
"org-name: " + latin1OrgName + "\n",
},
"LACNIC in UTF-8": {
ianaQuery: lacnicReferral,
"whois.lacnic.net:43 " + asQuery: "aut-num: AS3333\n" +
"owner: " + utf8OrgName + "\n",
},
} {
t.Run(name, func(t *testing.T) {
t.Parallel()
info, err := lookUp(t, answers)
if err != nil {
t.Fatalf("LookupASN: %v", err)
}
if info.OrgName != utf8OrgName {
t.Errorf("LookupASN org name = %q, want %q",
info.OrgName, utf8OrgName)
}
if !utf8.ValidString(info.RawResponse) ||
!strings.Contains(info.RawResponse, utf8OrgName) {
t.Errorf("LookupASN answer = %q, want UTF-8 holding %q",
info.RawResponse, utf8OrgName)
}
})
}
}
// lacnicServer is LACNIC's WHOIS server and nicbrServer NIC.br's, which
// registers Brazil's ASes under LACNIC. brazilASN is one of those ASes.
const (
lacnicServer = "whois.lacnic.net:43"
nicbrServer = "whois.registro.br:43"
brazilASN = 52751
)
// The keys of the queries LookupASN sends for brazilASN: IANA's, LACNIC's,
// then NIC.br's.
const (
brazilIANAQuery = ianaServer + " AS52751"
brazilLACNICQuery = lacnicServer + " AS52751"
brazilNICbrQuery = nicbrServer + " AS52751"
)
// The name of brazilASN's organisation in three forms: as LACNIC sends it,
// with each accented letter replaced by U+FFFD; as NIC.br sends it, in
// Latin-1; and in UTF-8.
const (
lacnicBrazilOrgName = "Brasrede Telecomunica\U0000FFFD\U0000FFFDes LTDA"
nicbrBrazilOrgName = "Brasrede Telecomunica\xe7\xf5es LTDA"
brazilOrgName = "Brasrede Telecomunicações LTDA"
)
// nicbrCopyright is the copyright notice at the top of NIC.br's answers.
const nicbrCopyright = "% Copyright (c) Nic.br - " +
"Use of this data is governed by the Use and\n" +
"% Privacy Policy at https://registro.br/upp . Distribution,\n" +
"% commercialization, reproduction, and use for advertising or similar\n" +
"% purposes are expressly prohibited.\n"
// lacnicBrazilRecord is LACNIC's answer for brazilASN, recorded on
// 2026-10-26. It is NIC.br's answer, which LACNIC passes on after a line
// of its own and a space. It is shortened. The client's address is a
// documentation address. The contact's name, the contact's object and the
// footer are left out.
const lacnicBrazilRecord = "% IP Client: 192.0.2.1\n" +
" " + nicbrCopyright + `% 2026-10-26T23:12:22-03:00 - 192.0.2.1
aut-num: AS52751
owner: ` + lacnicBrazilOrgName + `
owner-c: AVL217
routing-c: AVL217
abuse-c: AVL217
created: 20120829
changed: 20130306
inetnum: 177.38.152.0/21
inetnum: 2804:b34::/32
inetnum: 191.243.4.0/22
inetnum: 131.72.248.0/22
inetnum: 138.94.228.0/22
inetnum: 170.233.8.0/22
inetnum: 170.246.4.0/22
`
// nicbrRecord is NIC.br's answer for brazilASN, in Latin-1, recorded on
// 2026-10-26 and shortened as lacnicBrazilRecord is.
const nicbrRecord = nicbrCopyright + `% 2026-10-26T23:14:41-03:00 - 192.0.2.1
aut-num: AS52751
owner: ` + nicbrBrazilOrgName + `
owner-c: AVL217
routing-c: AVL217
abuse-c: AVL217
created: 20120829
changed: 20130306
inetnum: 177.38.152.0/21
inetnum: 2804:b34::/32
inetnum: 191.243.4.0/22
inetnum: 131.72.248.0/22
inetnum: 138.94.228.0/22
inetnum: 170.233.8.0/22
inetnum: 170.246.4.0/22
`
// nicbrOverLimit stands for NIC.br's answer for brazilASN to a client over
// its query limit, and is not recorded. Such an answer leaves information
// out, here the owner, and says so in its first line.
const nicbrOverLimit = `% Query rate limit exceeded. Reduced information.
aut-num: AS52751
created: 20120829
changed: 20130306
`
// newNICbrClient returns a client whose fake servers send answers and
// whose logger writes JSON lines to logs. It adds one to *nicbrAsks each
// time it connects to NIC.br's server.
func newNICbrClient(
t *testing.T, answers map[string]string, logs io.Writer, nicbrAsks *int,
) *whois.Client {
t.Helper()
dial := fakeServers(t, answers)
client := whois.NewClient(slog.New(slog.NewJSONHandler(logs, nil)))
client.DialContext = func(
ctx context.Context, network, address string,
) (net.Conn, error) {
if address == nicbrServer {
*nicbrAsks++
}
return dial(ctx, network, address)
}
return client
}
// lookUpBrazilianAS looks brazilASN up with fake servers sending answers,
// and reports whether the client connected to NIC.br's server.
func lookUpBrazilianAS(
t *testing.T, answers map[string]string,
) (*whois.ASNInfo, bool, error) {
t.Helper()
nicbrAsks := 0
client := newNICbrClient(t, answers, io.Discard, &nicbrAsks)
info, err := client.LookupASN(t.Context(), brazilASN)
return info, nicbrAsks > 0, err
}
// TestLookupASNTakesABrazilianASFromNICbr looks up an AS that NIC.br
// registers. LACNIC's answer is NIC.br's with each accented letter replaced
// by U+FFFD, so the client also asks NIC.br's server, which answers in
// Latin-1. The organisation's name has its accented letters, in the parsed
// field and in the stored answer.
func TestLookupASNTakesABrazilianASFromNICbr(t *testing.T) {
t.Parallel()
info, _, err := lookUpBrazilianAS(t, map[string]string{
brazilIANAQuery: lacnicReferral,
brazilLACNICQuery: lacnicBrazilRecord,
brazilNICbrQuery: nicbrRecord,
})
if err != nil {
t.Fatalf("LookupASN: %v", err)
}
if info.OrgName != brazilOrgName || info.RIR != whois.RIRLACNIC {
t.Errorf("LookupASN org name, RIR = %q, %q; want %q, %q",
info.OrgName, info.RIR, brazilOrgName, whois.RIRLACNIC)
}
if !strings.Contains(info.RawResponse, brazilOrgName) {
t.Errorf("LookupASN answer = %q, want one holding %q",
info.RawResponse, brazilOrgName)
}
wantRegDate := time.Date(2012, 8, 29, 0, 0, 0, 0, time.UTC)
if info.RegDate == nil || !info.RegDate.Equal(wantRegDate) {
t.Errorf("LookupASN registered = %v, want %v",
info.RegDate, wantRegDate)
}
}
// TestLookupASNAsksNICbrOnlyForItsASes looks up an AS at LACNIC whose
// answer does not carry NIC.br's copyright notice, as for an AS that LACNIC
// registers itself. The client does not connect to NIC.br's server, which
// would answer with a record, and keeps LACNIC's fields.
func TestLookupASNAsksNICbrOnlyForItsASes(t *testing.T) {
t.Parallel()
info, askedNICbr, err := lookUpBrazilianAS(t, map[string]string{
brazilIANAQuery: lacnicReferral,
brazilLACNICQuery: "aut-num: AS52751\n" +
"owner: " + utf8OrgName + "\n",
brazilNICbrQuery: nicbrRecord,
})
if err != nil {
t.Fatalf("LookupASN: %v", err)
}
if askedNICbr {
t.Error("LookupASN asked NIC.br's server about an AS whose " +
"answer from LACNIC is not NIC.br's")
}
if info.OrgName != utf8OrgName {
t.Errorf("LookupASN org name = %q, want LACNIC's %q",
info.OrgName, utf8OrgName)
}
}
// TestLookupASNKeepsLACNICsAnswerWhenNICbrFails looks up an AS that NIC.br
// registers when NIC.br's server refuses the connection, sends nothing, or
// sends the reduced answer for a client over its query limit. The client
// asks NIC.br, and the lookup still succeeds with LACNIC's fields. The
// client logs one warning, which names the AS and the reason.
func TestLookupASNKeepsLACNICsAnswerWhenNICbrFails(t *testing.T) {
t.Parallel()
for name, test := range map[string]struct {
nicbrAnswer map[string]string
reason string
}{
"connection refused": {
nicbrAnswer: map[string]string{},
reason: syscall.ECONNREFUSED.Error(),
},
"no answer": {
nicbrAnswer: map[string]string{brazilNICbrQuery: ""},
reason: "no WHOIS fields",
},
"over the query limit": {
nicbrAnswer: map[string]string{brazilNICbrQuery: nicbrOverLimit},
reason: "over its query limit",
},
} {
t.Run(name, func(t *testing.T) {
t.Parallel()
answers := map[string]string{
brazilIANAQuery: lacnicReferral,
brazilLACNICQuery: lacnicBrazilRecord,
}
maps.Copy(answers, test.nicbrAnswer)
var logs bytes.Buffer
nicbrAsks := 0
client := newNICbrClient(t, answers, &logs, &nicbrAsks)
info, err := client.LookupASN(t.Context(), brazilASN)
if err != nil {
t.Fatalf("LookupASN: %v", err)
}
if nicbrAsks == 0 {
t.Error("LookupASN did not ask NIC.br's server")
}
if info.OrgName != lacnicBrazilOrgName {
t.Errorf("LookupASN org name = %q, want LACNIC's %q",
info.OrgName, lacnicBrazilOrgName)
}
var logged struct {
Level string `json:"level"`
ASN int `json:"asn"`
}
err = json.Unmarshal(logs.Bytes(), &logged)
if err != nil {
t.Fatalf("want one log line, got %q: %v", logs.String(), err)
}
if logged.Level != slog.LevelWarn.String() ||
logged.ASN != brazilASN ||
!strings.Contains(logs.String(), test.reason) {
t.Errorf("logged %q, want a warning for AS%d holding %q",
logs.String(), brazilASN, test.reason)
}
})
}
}
// TestLookupASNSkipsNICbrAfterItsLimit looks up an AS that NIC.br
// registers three times with one client, while NIC.br answers that the
// client is over its query limit. The first lookup asks NIC.br. The
// second, a second before NICbrSkipTime has passed, does not. The third,
// once NICbrSkipTime has passed, asks again. Each keeps LACNIC's fields.
func TestLookupASNSkipsNICbrAfterItsLimit(t *testing.T) {
t.Parallel()
nicbrAsks := 0
client := newNICbrClient(t, map[string]string{
brazilIANAQuery: lacnicReferral,
brazilLACNICQuery: lacnicBrazilRecord,
brazilNICbrQuery: nicbrOverLimit,
}, io.Discard, &nicbrAsks)
start := time.Date(2026, 10, 27, 0, 0, 0, 0, time.UTC)
now := start
client.Now = func() time.Time { return now }
for _, step := range []struct {
after time.Duration
wantAsks int
}{
{after: 0, wantAsks: 1},
{after: whois.NICbrSkipTime - time.Second, wantAsks: 1},
{after: whois.NICbrSkipTime, wantAsks: 2},
} {
now = start.Add(step.after)
info, err := client.LookupASN(t.Context(), brazilASN)
if err != nil {
t.Fatalf("LookupASN after %v: %v", step.after, err)
}
if nicbrAsks != step.wantAsks {
t.Errorf("after %v NIC.br's server was asked %d times, want %d",
step.after, nicbrAsks, step.wantAsks)
}
if info.OrgName != lacnicBrazilOrgName {
t.Errorf("LookupASN after %v org name = %q, want LACNIC's %q",
step.after, info.OrgName, lacnicBrazilOrgName)
}
}
}
// TestLookupASNReportsARefusal looks up an AS IANA refers to RIPE when RIPE
// refuses the query, with each of the error lines RIPE refuses with. The
// error holds RIPE's line, so the log says why the lookup failed.
func TestLookupASNReportsARefusal(t *testing.T) {
t.Parallel()
for _, line := range []string{ripeAccessDenied, ripeLimitReached} {
t.Run(line, func(t *testing.T) {
t.Parallel()
_, err := lookUp(t, map[string]string{
ianaQuery: ianaReferral,
ripeQuery: "% This is the RIPE Database query service.\n" +
line + "\n",
})
if err == nil || !strings.Contains(err.Error(), line) {
t.Errorf("LookupASN error = %v, want one holding %q",
err, line)
}
})
}
}
// TestLookupASNFailsWhenTheRIRGivesNoAnswer looks up an AS IANA refers to // TestLookupASNFailsWhenTheRIRGivesNoAnswer looks up an AS IANA refers to
// RIPE when RIPE's server refuses the connection, and when it answers only // RIPE when RIPE's server refuses the connection, and when it answers only
// with an error. Neither is a lookup that found the AS. // with an error. Neither is a lookup that found the AS.
@@ -157,8 +792,8 @@ func TestLookupASNFailsWhenTheRIRGivesNoAnswer(t *testing.T) {
t.Parallel() t.Parallel()
for name, answers := range map[string]map[string]string{ for name, answers := range map[string]map[string]string{
"connection refused": {ianaServer: ianaReferral}, "connection refused": {ianaQuery: ianaReferral},
"only an error": {ianaServer: ianaReferral, ripeServer: ripeRefusal}, "only an error": {ianaQuery: ianaReferral, ripeQuery: ripeRefusal},
} { } {
t.Run(name, func(t *testing.T) { t.Run(name, func(t *testing.T) {
t.Parallel() t.Parallel()
@@ -177,7 +812,7 @@ func TestLookupASNFailsWhenTheRIRGivesNoAnswer(t *testing.T) {
func TestLookupASNReportsNoReferral(t *testing.T) { func TestLookupASNReportsNoReferral(t *testing.T) {
t.Parallel() t.Parallel()
_, err := lookUp(t, map[string]string{ianaServer: ianaPrivateUse}) _, err := lookUp(t, map[string]string{ianaQuery: ianaPrivateUse})
if !errors.Is(err, whois.ErrNoReferral) { if !errors.Is(err, whois.ErrNoReferral) {
t.Errorf("LookupASN error = %v, want ErrNoReferral", err) t.Errorf("LookupASN error = %v, want ErrNoReferral", err)
} }
@@ -189,7 +824,7 @@ func TestLookupASNReportsNoReferral(t *testing.T) {
func TestLookupASNFailsWhenIANAAnswersNoBlock(t *testing.T) { func TestLookupASNFailsWhenIANAAnswersNoBlock(t *testing.T) {
t.Parallel() t.Parallel()
_, err := lookUp(t, map[string]string{ianaServer: ianaNoObject}) _, err := lookUp(t, map[string]string{ianaQuery: ianaNoObject})
if err == nil || errors.Is(err, whois.ErrNoReferral) { if err == nil || errors.Is(err, whois.ErrNoReferral) {
t.Errorf("LookupASN error = %v, want one that is not ErrNoReferral", t.Errorf("LookupASN error = %v, want one that is not ErrNoReferral",
err) err)
@@ -206,8 +841,8 @@ func TestLookupASNFailsWhenTheRIRAnswerDoesNotEnd(t *testing.T) {
strings.Repeat(fillerLine, endlessAnswerSize/len(fillerLine)) strings.Repeat(fillerLine, endlessAnswerSize/len(fillerLine))
_, err := lookUp(t, map[string]string{ _, err := lookUp(t, map[string]string{
ianaServer: ianaReferral, ianaQuery: ianaReferral,
ripeServer: endless, ripeQuery: endless,
}) })
if err == nil || errors.Is(err, whois.ErrNoReferral) { if err == nil || errors.Is(err, whois.ErrNoReferral) {
t.Errorf("LookupASN error = %v, want one that is not ErrNoReferral", t.Errorf("LookupASN error = %v, want one that is not ErrNoReferral",
@@ -233,9 +868,9 @@ func TestLookupASNFailsOnAReferralToAnotherServer(t *testing.T) {
t.Parallel() t.Parallel()
_, err := lookUp(t, map[string]string{ _, err := lookUp(t, map[string]string{
ianaServer: "refer: " + refer + "\n\n" + ianaQuery: "refer: " + refer + "\n\n" +
"as-block: 3154-3353\n", "as-block: 3154-3353\n",
address: ripeRecord, address + " " + asQuery: ripeRecord,
}) })
if err == nil || errors.Is(err, whois.ErrNoReferral) { if err == nil || errors.Is(err, whois.ErrNoReferral) {
t.Errorf("LookupASN error = %v, want one that is not "+ t.Errorf("LookupASN error = %v, want one that is not "+
Binary file not shown.
+49 -23
View File
@@ -2,10 +2,13 @@ package asinfo
import ( import (
"bytes" "bytes"
"cmp"
"compress/gzip" "compress/gzip"
_ "embed" _ "embed"
"encoding/json" "encoding/json"
"io" "io"
"slices"
"strings"
"sync" "sync"
) )
@@ -23,6 +26,17 @@ type Info struct {
type Registry struct { type Registry struct {
mu sync.RWMutex mu sync.RWMutex
byASN map[int]*Info byASN map[int]*Info
// searchEntries holds every entry in AS number order, for Search.
searchEntries []searchEntry
}
// searchEntry is an entry with its handle and description in lower case. They
// are put in lower case once, when the data loads, so that Search only
// compares.
type searchEntry struct {
info *Info
lowerHandle string
lowerDescription string
} }
var ( var (
@@ -59,9 +73,21 @@ func initRegistry() {
panic("failed to unmarshal embedded AS data: " + err.Error()) panic("failed to unmarshal embedded AS data: " + err.Error())
} }
slices.SortFunc(asInfos, func(a, b Info) int {
return cmp.Compare(a.ASN, b.ASN)
})
defaultRegistry.searchEntries = make([]searchEntry, 0, len(asInfos))
for i := range asInfos { for i := range asInfos {
info := &asInfos[i] info := &asInfos[i]
defaultRegistry.byASN[info.ASN] = info defaultRegistry.byASN[info.ASN] = info
defaultRegistry.searchEntries = append(defaultRegistry.searchEntries,
searchEntry{
info: info,
lowerHandle: strings.ToLower(info.Handle),
lowerDescription: strings.ToLower(info.Description),
})
} }
} }
@@ -129,36 +155,36 @@ func All() []Info {
return result return result
} }
// Search searches for AS entries where the handle or description contains the query // Search finds the AS entries whose handle or description contains query,
// This is a simple case-sensitive substring search // ignoring case. It returns the first limit of them in AS number order, and the
func Search(query string) []Info { // number of entries that match. An empty query matches nothing.
func Search(query string, limit int) ([]Info, int) {
once.Do(initRegistry) once.Do(initRegistry)
if query == "" {
return nil, 0
}
query = strings.ToLower(query)
defaultRegistry.mu.RLock() defaultRegistry.mu.RLock()
defer defaultRegistry.mu.RUnlock() defer defaultRegistry.mu.RUnlock()
var results []Info var (
results []Info
total int
)
for _, info := range defaultRegistry.byASN { for _, entry := range defaultRegistry.searchEntries {
if contains(info.Handle, query) || contains(info.Description, query) { if strings.Contains(entry.lowerHandle, query) ||
results = append(results, *info) strings.Contains(entry.lowerDescription, query) {
if total < limit {
results = append(results, *entry.info)
}
total++
} }
} }
return results return results, total
}
// contains is a simple substring search
func contains(s, substr string) bool {
return len(substr) > 0 && len(s) >= len(substr) && containsImpl(s, substr)
}
func containsImpl(s, substr string) bool {
for i := 0; i <= len(s)-len(substr); i++ {
if s[i:i+len(substr)] == substr {
return true
}
}
return false
} }
+112 -20
View File
@@ -1,6 +1,8 @@
package asinfo_test package asinfo_test
import ( import (
"cmp"
"slices"
"strings" "strings"
"testing" "testing"
@@ -20,7 +22,7 @@ func TestGet(t *testing.T) {
name: "AS1 Level 3", name: "AS1 Level 3",
asn: 1, asn: 1,
wantOK: true, wantOK: true,
wantDesc: "Level 3 Parent, LLC", wantDesc: "Level 3 Parent LLC",
}, },
{ {
name: "AS15169 Google", name: "AS15169 Google",
@@ -34,10 +36,10 @@ func TestGet(t *testing.T) {
wantOK: false, wantOK: false,
}, },
{ {
name: "AS0 IANA Reserved", name: "AS1251 with accented letters",
asn: 0, asn: 1251,
wantOK: true, wantOK: true,
wantDesc: "Internet Assigned Numbers Authority", wantDesc: "FUNDAÇÃO DE AMPARO À PESQUISA DO ESTADO SÃO PAULO",
}, },
} }
@@ -67,7 +69,7 @@ func TestGetDescription(t *testing.T) {
asn int asn int
want string want string
}{ }{
{1, "Level 3 Parent, LLC"}, {1, "Level 3 Parent LLC"},
{999999999, ""}, {999999999, ""},
} }
@@ -90,8 +92,7 @@ func TestGetHandle(t *testing.T) {
asn int asn int
want string want string
}{ }{
{1, "LVLT"}, {1, "LVLT-1"},
{0, "IANA-RSVD"},
{999999999, ""}, {999999999, ""},
} }
@@ -141,9 +142,9 @@ func TestSearch(t *testing.T) {
wantMin: 10, wantMin: 10,
}, },
{ {
name: "Empty search", name: "Search for São Paulo",
query: "", query: "São Paulo",
wantMin: 0, wantMin: 1,
}, },
{ {
name: "Non-existent org", name: "Non-existent org",
@@ -156,26 +157,115 @@ func TestSearch(t *testing.T) {
t.Run(tt.name, func(t *testing.T) { t.Run(tt.name, func(t *testing.T) {
t.Parallel() t.Parallel()
results := asinfo.Search(tt.query) results, _ := asinfo.Search(tt.query, asinfo.Total())
if len(results) < tt.wantMin { if len(results) < tt.wantMin {
t.Errorf("Search(%q) returned %d results, want at least %d", t.Errorf("Search(%q) returned %d results, want at least %d",
tt.query, len(results), tt.wantMin) tt.query, len(results), tt.wantMin)
} }
// Verify all results contain the query // Verify all results contain the query, ignoring case
if tt.query != "" { query := strings.ToLower(tt.query)
for _, r := range results { for _, r := range results {
if !strings.Contains(r.Handle, tt.query) && if !strings.Contains(strings.ToLower(r.Handle), query) &&
!strings.Contains(r.Description, tt.query) { !strings.Contains(strings.ToLower(r.Description), query) {
t.Errorf("Search result ASN %d doesn't contain query %q", t.Errorf("Search result ASN %d doesn't contain query %q",
r.ASN, tt.query) r.ASN, tt.query)
}
} }
} }
}) })
} }
} }
// TestSearchEmptyQueryMatchesNothing expects an empty query to match no AS,
// although every handle contains the empty string.
func TestSearchEmptyQueryMatchesNothing(t *testing.T) {
t.Parallel()
results, total := asinfo.Search("", asinfo.Total())
if len(results) != 0 || total != 0 {
t.Errorf(`Search("") returned %d results and a total of %d, want none`,
len(results), total)
}
}
// TestSearchIgnoresCase searches for Cloudflare's handle in lower case and its
// description in upper case, and expects both searches to find AS13335.
func TestSearchIgnoresCase(t *testing.T) {
t.Parallel()
const cloudflareASN = 13335 // handle CLOUDFLARENET, "Cloudflare Inc."
isCloudflare := func(info asinfo.Info) bool {
return info.ASN == cloudflareASN
}
for _, query := range []string{"cloudflarenet", "CLOUDFLARE INC."} {
results, _ := asinfo.Search(query, asinfo.Total())
if !slices.ContainsFunc(results, isCloudflare) {
t.Errorf("Search(%q) did not find AS%d", query, cloudflareASN)
}
}
}
// TestSearchOrdersByASN expects the matches of a search with many of them to
// come in AS number order.
func TestSearchOrdersByASN(t *testing.T) {
t.Parallel()
const query = "University"
results, _ := asinfo.Search(query, asinfo.Total())
for i := 1; i < len(results); i++ {
if results[i-1].ASN >= results[i].ASN {
t.Fatalf("Search(%q): AS%d comes after AS%d",
query, results[i].ASN, results[i-1].ASN)
}
}
}
// TestSearchStopsAtLimit searches for a word that matches more entries than
// the limit. It expects exactly limit entries, the first matches in AS number
// order, and the number of all matches, counted here from All.
func TestSearchStopsAtLimit(t *testing.T) {
t.Parallel()
const (
query = "university"
limit = 10
)
var matches []asinfo.Info
for _, info := range asinfo.All() {
if strings.Contains(strings.ToLower(info.Handle), query) ||
strings.Contains(strings.ToLower(info.Description), query) {
matches = append(matches, info)
}
}
if len(matches) <= limit {
t.Fatalf("%q matches %d entries; the test needs more than %d",
query, len(matches), limit)
}
slices.SortFunc(matches, func(a, b asinfo.Info) int {
return cmp.Compare(a.ASN, b.ASN)
})
results, total := asinfo.Search(query, limit)
if total != len(matches) {
t.Errorf("Search(%q, %d): total %d, want %d",
query, limit, total, len(matches))
}
if !slices.Equal(results, matches[:limit]) {
t.Errorf("Search(%q, %d) = %v, want %v",
query, limit, results, matches[:limit])
}
}
func TestDataIntegrity(t *testing.T) { func TestDataIntegrity(t *testing.T) {
t.Parallel() t.Parallel()
@@ -215,11 +305,13 @@ func BenchmarkGet(b *testing.B) {
} }
func BenchmarkSearch(b *testing.B) { func BenchmarkSearch(b *testing.B) {
const limit = 50
queries := []string{"Google", "Amazon", "Microsoft", "University"} queries := []string{"Google", "Amazon", "Microsoft", "University"}
b.ResetTimer() b.ResetTimer()
for i := range b.N { for i := range b.N {
asinfo.Search(queries[i%len(queries)]) asinfo.Search(queries[i%len(queries)], limit)
} }
} }
+5 -3
View File
@@ -2,7 +2,7 @@
Package asinfo provides information about Autonomous Systems (AS). Package asinfo provides information about Autonomous Systems (AS).
The package embeds a comprehensive database of AS numbers, handles, and descriptions The package embeds a comprehensive database of AS numbers, handles, and descriptions
sourced from https://github.com/ipverse/asn-info. The data is embedded at compile sourced from https://github.com/ipverse/as-metadata. The data is embedded at compile
time and provides fast, offline lookups. time and provides fast, offline lookups.
Basic usage: Basic usage:
@@ -16,8 +16,10 @@ Basic usage:
// Get just the description // Get just the description
desc := asinfo.GetDescription(15169) // "Google LLC" desc := asinfo.GetDescription(15169) // "Google LLC"
// Search for AS entries // Search the handles and descriptions, ignoring case, for the first 10
results := asinfo.Search("University") // matches in AS number order and the number of all matches
results, total := asinfo.Search("university", 10)
fmt.Printf("%d matches\n", total)
for _, as := range results { for _, as := range results {
fmt.Printf("AS%d: %s\n", as.ASN, as.Description) fmt.Printf("AS%d: %s\n", as.ASN, as.Description)
} }
+10 -3
View File
@@ -17,13 +17,20 @@ func ExampleGet() {
func ExampleGetDescription() { func ExampleGetDescription() {
desc := asinfo.GetDescription(13335) desc := asinfo.GetDescription(13335)
fmt.Println(desc) fmt.Println(desc)
// Output: Cloudflare, Inc. // Output: Cloudflare Inc.
} }
func ExampleSearch() { func ExampleSearch() {
results := asinfo.Search("MIT-GATEWAY") const limit = 10
results, total := asinfo.Search("mit-gateway", limit)
fmt.Printf("%d of %d matches\n", len(results), total)
for _, info := range results { for _, info := range results {
fmt.Printf("AS%d: %s - %s\n", info.ASN, info.Handle, info.Description) fmt.Printf("AS%d: %s - %s\n", info.ASN, info.Handle, info.Description)
} }
// Output: AS3: MIT-GATEWAYS - Massachusetts Institute of Technology // Output:
// 1 of 1 matches
// AS3: MIT-GATEWAYS - Massachusetts Institute of Technology
} }
+34
View File
@@ -0,0 +1,34 @@
#!/bin/sh
# script/asupdate: rebuild the AS list built into RouteWatch from
# as.csv with cmd/asinfo-gen, which fails when it finds no AS. The new
# list replaces the old one only when the generator succeeds. Otherwise
# the old list stays as it was and the script exits non-zero.
set -eu
ROOT="$(cd "$(dirname "$0")/.." && pwd -P)"
AS_DATA="pkg/asinfo/asdata.json.gz"
# Next to AS_DATA, so the final mv is a rename on one file system.
AS_DATA_TMP="$AS_DATA.tmp"
main() {
cd "$ROOT"
json="$(mktemp)"
trap 'rm -f "$json" "$AS_DATA_TMP"' EXIT
# A signal exits through the EXIT trap too.
trap 'exit 1' HUP INT TERM
echo "Updating AS info data..."
if ! go run ./cmd/asinfo-gen >"$json"; then
echo "asupdate: the generator failed; $AS_DATA is unchanged" >&2
exit 1
fi
# -n leaves the time out of the file, so an unchanged list gives the
# same bytes.
gzip -n -c <"$json" >"$AS_DATA_TMP"
mv "$AS_DATA_TMP" "$AS_DATA"
}
main "$@"