sneak/routewatch
1
0
Fork 0
Code Issues 5 Pull Requests 1 Actions 17 Packages Projects Releases Wiki Activity

Compare commits

base: sneak/routewatch:next
Branches Tags
sneak/routewatch:main sneak/routewatch:next sneak/routewatch:prod sneak/routewatch:issue-27-cached-stats
..
compare: sneak/routewatch:00ea3441f7747600384844b7cde912195fc199eb
Branches Tags
sneak/routewatch:next sneak/routewatch:prod sneak/routewatch:main sneak/routewatch:issue-27-cached-stats
1 Commits
next .. 00ea3441f7
Author SHA1 Message Date
sneak 00ea3441f7 Keep prefixes that do not parse out of the prefix tables (closes #114)
check / check (push) Canceled after 0s
Details 
The prefix handler now drops an announcement or withdrawal whose prefix
does not parse, and logs it as an error, so the prefix never reaches the
prefix tables or the prefix count in the stats. Before, no route was
stored for such a prefix, but it was stored as a permanent row in the
prefix tables and counted.

Model: opus-5-5
 2026-10-20 01:30:51 +00:00
55 changed files with 490 additions and 7189 deletions
Expand all files Collapse all files
.dockerignore
+2 -12
View File
@@ -73,8 +73,7 @@
# This repository's host-built artifacts: `make build` writes bin/, `go
# build` in the root the binaries, `make run` log.txt, `make asupdate` its
# .tmp file; logs, test binaries, coverage profiles, source archives, and
# the binaries and libraries a build for another platform writes.
# .tmp file; test binaries, coverage profiles and source archives.
/bin
/routewatch
/asinfo-gen
@@ -83,24 +82,15 @@
/out
/pkg/asinfo/asdata.json
**/*.tar.zst
**/*.log
**/*.test
**/*.out
**/*.tmp
**/*.exe
**/*.dll
**/*.so
**/*.dylib
# Local databases. The image carries a source archive of the whole build
# context, so these would otherwise ship inside it. The daemon's database
# is db.sqlite, with its -wal and -shm files while it runs.
# context, so these would otherwise ship inside it.
**/*.db
**/*.db-journal
**/*.db-wal
**/*.sqlite
**/*.sqlite-wal
**/*.sqlite-shm
# A local Go workspace points at directories outside the build context.
/go.work
.gitignore
+1 -5
View File
@@ -67,14 +67,10 @@ node_modules/
go.work
go.work.sum
# Local databases, temporary files and debug output. The daemon's
# database is db.sqlite, with its -wal and -shm files while it runs.
# Local databases, temporary files and debug output.
*.db
*.db-journal
*.db-wal
*.sqlite
*.sqlite-wal
*.sqlite-shm
*.tmp
log.txt
out
Makefile
+3 -2
View File
@@ -52,6 +52,7 @@ clean:
run: build
DEBUG=routewatch ./bin/routewatch 2>&1 | tee log.txt
# Rebuild the AS list built into RouteWatch from as.csv (needs the network).
asupdate:
@script/asupdate
@echo "Updating AS info data..."
@go run cmd/asinfo-gen/main.go | gzip > pkg/asinfo/asdata.json.gz.tmp && \
mv pkg/asinfo/asdata.json.gz.tmp pkg/asinfo/asdata.json.gz
README.md
+33 -150
View File
@@ -1,27 +1,9 @@
# RouteWatch
RouteWatch is an MIT-licensed Go daemon by @sneak that monitors the BGP routing
table in real time. It works like this:
- It streams BGP UPDATE messages from the RIPE RIS Live service.
- It keeps a live routing table in SQLite.
- It serves web pages and a JSON API for looking routes up.
## Rationale
RIPE RIS Live streams the BGP updates that RIPE's route collectors receive, as
they happen, from the moment a client connects. It reports changes, not the
routing table they add up to, so it cannot say which AS an IP address is routed
to now or which routes a prefix has. RouteWatch applies the feed's updates to
live routing tables in its own SQLite database and answers IP-to-AS lookups and
prefix pages from them.
The feed sends no table when a client connects, so these tables hold only the
routes the feed has announced since RouteWatch first started: an IP-to-AS lookup
finds nothing for an address whose routes have not been announced since then.
The tables also miss the updates RouteWatch drops when it falls behind under
load and those sent while it is stopped, so a lookup or a prefix page can show a
route the feed has withdrawn.
table in real time: it streams BGP UPDATE messages from the RIPE RIS Live
service, maintains a live routing table in SQLite, and provides HTTP APIs for
querying routing information.
## Features
@@ -29,8 +11,6 @@ route the feed has withdrawn.
- Maintains live IPv4 and IPv6 routing tables
- Tracks AS peering relationships
- HTTP API for IP-to-AS lookups, prefix details, and AS information
- IP lookups report the public blocklists an address is on, from copies kept in
memory and on disk
- Automatic reconnection with exponential backoff
- Batched database writes for high performance
- Backpressure handling to prevent memory exhaustion
@@ -60,18 +40,15 @@ Every route answers `HEAD` as it answers `GET`, without the body.
### Web Interface
- `GET /` - Home page (HTML): headline figures, lookups by AS number and by IP
address, and a search of AS names. A card shows the `curl` command that looks
up the visitor's own address at `/ip/{addr}`, and that command's live answer.
- `GET /` - Home page (HTML): headline figures, and lookups by AS number and by
IP address
- `GET /status` - HTML status dashboard, which reads `/api/v1/stats`
- `GET /status.json` - JSON statistics: feed, memory, database and route figures
- `GET /.well-known/healthcheck.json` - Health check (JSON): 200 when the
database answers and the RIS Live feed is connected, otherwise 503;
`data.status` is `ok` or `error` and `data.checks` says which check failed
- `GET /as/{asn}` - AS detail page (HTML): the prefixes the AS originates and
its peers. When the database has no record of the AS, its handle and
description come from the AS list built into RouteWatch. The page then shows
no first-seen date.
its peers
- `GET /prefix/{prefix}/{length}` - Prefix detail page (HTML), such as
`/prefix/192.0.2.0/24`: every live route for the prefix, one per peer, and its
origin ASes
@@ -86,47 +63,23 @@ Every route answers `HEAD` as it answers `GET`, without the body.
### API v1
- `GET /api/v1/stats` - The `/status.json` figures plus connection, garbage
collector, stream and handler queue statistics. `stream.route_churn_per_min`
is the number of prefix announcements and withdrawals, from all peers, in the
last full clock minute.
- `GET /api/v1/ip/{ip}` - Looks up each address. `{ip}` is an IP address, a host
name, or a comma-separated list of at most 10 of these. `results` has one
entry for each address, with:
- `ip` and `ptr`: the address and its PTR names
- `netblock` and the fields beside it: the address's most specific live
route, and that route's origin AS with its WHOIS details
- `blocklists`: the public blocklists the address is on (see Blocklists)
An address with no live route has a result without the route and AS fields.
Of the addresses a host name resolves to, only public ones are looked up. A
name with none, such as one that resolves to a private address, is reported
as `host name has no public address`, without its addresses. `errors` has
one entry for each host name that failed. The answer is 400 when:
- `{ip}` lists more than 10
- `{ip}` comes to more than 10 addresses once its names are resolved
- nothing in `{ip}` is an IP address or a host name with a public address
A PTR lookup gets 2 seconds. An address whose lookup takes longer is
answered without PTR names.
collector, stream and handler queue statistics
- `GET /api/v1/ip/{ip}` - Looks up each address: its most specific live route
(`netblock`), that route's origin AS with its WHOIS details, and the address's
PTR names. `{ip}` is an IP address, a host name, or a comma-separated list of
at most 10 of these. Of the addresses a host name resolves to, only public
ones are looked up: a name with none, such as one that resolves to a private
address, is reported as `host name has no public address`, without its
addresses. `results` has one entry per address with a live route and `errors`
one per address or name that failed. The answer is 400 when `{ip}` lists more
than 10, when it comes to more than 10 addresses once its names are resolved,
or when nothing in it is an IP address or a host name with a public address;
and 404 when none of the addresses has a live route. A PTR lookup gets 2
seconds; an address whose lookup takes longer is answered without PTR names.
- `GET /api/v1/as/{asn}` - The AS and the prefixes it originates, IPv4 and IPv6
apart, each with when its newest route was updated. When the database has no
record of the AS, its handle and description come from the AS list built into
RouteWatch. The answer then has no `first_seen` or `last_seen`.
apart, each with when its newest route was updated
- `GET /api/v1/prefix/{prefix}/{length}` - Every live route for the prefix, one
per peer, and the same routes grouped by origin AS in `origins`
- `GET /api/v1/search/as?q={text}` - Finds the ASes whose handle or description
contains `{text}`, ignoring case. `data` holds:
- `query`: `{text}` without its leading and trailing spaces
- `total`: the number of ASes that match
- `results`: the first 50 of them in AS number order, each with `asn`,
`handle` and `description`
The search reads the AS list built into RouteWatch, not the database. The
list comes from
[ipverse/as-metadata](https://github.com/ipverse/as-metadata). A match can
therefore be an AS that the database has no record of. A `q` that is
missing, empty or only spaces answers 400.
A JSON answer is `{"status": "ok", "data": ...}`, or for an error
`{"status": "error", "error": {"msg": ..., "code": ...}}`, and carries an
@@ -137,16 +90,13 @@ took. The health check is the exception: its 503 answer is also
A request that runs past its time limit answers 504: the limit is 30 seconds,
and 4 seconds for the database reads behind `/status.json` and `/api/v1/stats`.
An AS number, prefix or mask length that does not parse answers 400, as does a
mask length out of range for its IP version. An AS answers 404 when it is not in
the AS list built into RouteWatch and the database has neither a record of it
nor a live route it originates. AS 0 always answers 404, because it means no
known origin. The AS name search never lists it. A prefix with no live route
answers 404 too. The HTML pages give these errors as plain text. Under `/api/v1`
and `/ip/`, a path no route matches answers 404 with the message `Not found`,
and `POST`, `PUT`, `PATCH`, `DELETE`, `OPTIONS`, `TRACE` or `CONNECT` answers
405 with `Method not allowed`. A 500 answer says only `Internal server error`,
and a host name that does not resolve only `could not resolve host name`; the
detail goes to the log.
mask length out of range for its IP version. An AS the database does not hold,
or a prefix with no live route, answers 404. The HTML pages give these errors as
plain text. Under `/api/v1` and `/ip/`, a path no route matches answers 404 with
the message `Not found`, and `POST`, `PUT`, `PATCH`, `DELETE`, `OPTIONS`,
`TRACE` or `CONNECT` answers 405 with `Method not allowed`. A 500 answer says
only `Internal server error`, and a host name that does not resolve only
`could not resolve host name`; the detail goes to the log.
Every answer carries `Strict-Transport-Security`, `Content-Security-Policy`,
`X-Frame-Options`, `X-Content-Type-Options`, `Referrer-Policy` and
@@ -162,66 +112,6 @@ in a prefix's `origins`) means the route has no known origin: its AS path ends
in an AS set of several members, any of which may have originated it. The pages
show such a route as having no known origin.
## Blocklists
The IP lookup reports which public blocklists list each address. Each result has
a `blocklists` list with one entry for each list the address is on:
- `name`: the list's name, from the lists below
- `description`: what being on the list means
- `updated`: when RouteWatch last downloaded the list, or last heard from its
publisher that its copy was current
An address on no list has an empty `blocklists`. An address with no live route
still has a result with its `blocklists`.
The lists:
- `dshield`: the 20 networks that DShield saw scan the most targets in the last
three days. From [DShield](https://isc.sans.edu/feeds_doc.html), downloaded
every hour.
- `feodo_tracker`: botnet command-and-control servers. From abuse.ch
[Feodo Tracker](https://feodotracker.abuse.ch/blocklist/), downloaded every 15
minutes.
- `firehol_abusers_30d`: addresses reported for comment, form or forum spam in
the last 30 days. From
[FireHOL](https://iplists.firehol.org/?ipset=firehol_abusers_30d), downloaded
every 5 hours.
- `firehol_level3`: addresses reported for attacks, spyware or viruses in the
last 30 days. From
[FireHOL](https://iplists.firehol.org/?ipset=firehol_level3), downloaded every
3 hours.
- `spamhaus_drop` and `spamhaus_dropv6`: IPv4 and IPv6 networks that Spamhaus
says spammers or cyber criminals have hijacked or lease. From
[Spamhaus DROP](https://www.spamhaus.org/blocklists/do-not-route-or-peer/),
downloaded once a day. Spamhaus merged its EDROP list into DROP.
- `tor_exits`: Tor exit nodes, where traffic from the Tor network leaves. From
the [Tor Project](https://check.torproject.org/torbulkexitlist), downloaded
every hour.
No list is downloaded more often than its publisher updates it or allows. Each
can be downloaded whole without an account or a key.
How RouteWatch keeps the lists:
- It holds each list in memory, so a lookup makes no outside request.
- It keeps a copy of each list in the `blocklists` directory of the state
directory (see Configuration). Each copy is a file named after its list.
- Beside each copy, a file ending in `.validators` keeps the `Last-Modified` and
`ETag` the publisher sent with that copy.
- At startup it loads each list from its copy. It downloads a list at once when
the copy is older than the list's interval, or when there is no copy.
- A download sends the copy's `Last-Modified` as `If-Modified-Since` and its
`ETag` as `If-None-Match`. The publisher then sends the list only when it has
changed. Without either, the download asks for the whole list.
- A download fails when it takes over a minute or its file is over 16 MiB.
- When a download fails, or its file does not parse, RouteWatch keeps the
previous copy and logs the failure. It tries again one interval later.
A successful IP lookup answer carries `Cache-Control` with a `max-age` of 15
minutes, the shortest list interval. The answer for the caller's own address
(`/ip/`) is `private`, and the others are `public`.
## Code Structure
```
@@ -231,11 +121,9 @@ routewatch/
│ ├── asinfo-gen/ # Utility to generate AS info data
│ └── streamdumper/ # Debug utility for raw stream output
├── internal/
│ ├── blocklist/ # Public IP blocklists: downloads and lookups
│ ├── routewatch/ # Core application logic
│ ├── server/ # HTTP server and handlers
│ ├── database/ # SQLite storage layer
│ ├── routetable/ # The whole routing table in memory
│ ├── streamer/ # RIPE RIS Live client
│ ├── ristypes/ # BGP message data structures
│ ├── logger/ # Structured logging wrapper
@@ -307,11 +195,11 @@ routewatch/
- **ASHandler**: Tracks all ASNs seen in AS paths
- **PeerHandler**: Records BGP peer information
- **PrefixHandler**: Maintains live routing table (upserts on announcement,
deletes on withdrawal or on an announcement whose AS path is empty or
holds AS 0 anywhere, deletes all of a peer's routes when its BGP session
goes down). Routes are kept per peer IP, so when one peer IP has sessions
on two collectors, a session going down on one also removes the routes the
other still carries, until they are announced again.
deletes on withdrawal or on an announcement whose AS path is empty or ends
in AS 0, deletes all of a peer's routes when its BGP session goes down).
Routes are kept per peer IP, so when one peer IP has sessions on two
collectors, a session going down on one also removes the routes the other
still carries, until they are announced again.
- **PeeringHandler**: Extracts AS peering relationships from AS paths
5. **HTTP Request Flow**
@@ -399,7 +287,7 @@ client sent. For a request from any other address they use the address the
request came from, whatever forwarding headers it carries. `X-Real-IP` and
`True-Client-IP` are not read.
State directory (the database and the blocklist copies):
State directory (database location):
- macOS: `~/Library/Application Support/berlin.sneak.app.routewatch/`
- Linux: `/var/lib/berlin.sneak.app.routewatch/` when running as root, otherwise
@@ -561,11 +449,6 @@ provide:
or `go.sum`) and `go fmt` (writes), then `script/check`
- `script/install-precommit` — install the git pre-commit hook that runs
`script/precommit`
- `script/asupdate` — rebuild the AS list built into RouteWatch,
`pkg/asinfo/asdata.json.gz`, from the `as.csv` file of
[ipverse/as-metadata](https://github.com/ipverse/as-metadata). It needs the
network. It runs `cmd/asinfo-gen`, which fails when it finds no AS. When
`cmd/asinfo-gen` fails, the old list stays and the script exits non-zero.
Tests and the linter run only in Docker, as the `test` and `lint` phases of the
`Dockerfile`; the image cannot be built unless both pass. Every `docker build`
TODO.md
+1 -72
View File
@@ -33,75 +33,6 @@ tested.
# Completed Steps
- 2026-10-27: a new package, `internal/routetable`, holds the whole routing
table in memory. For each prefix and origin AS it keeps the peers that carry
it, one bit per peer, and the two newest of their routes. Nothing uses it yet
(part of #122)
- 2026-10-27: each result of the IP lookup lists the public blocklists the
address is on, in `blocklists`. The lists are DShield, Feodo Tracker, two
FireHOL lists, Spamhaus DROP and the Tor exit list. RouteWatch downloads each
list on its own schedule and answers from copies in memory. The copies are
also kept in the state directory's `blocklists` directory. IP lookup answers
carry `Cache-Control`. An address with no live route now has a result too,
without route fields, so the IP lookup no longer answers 404 (closes #158)
- 2026-10-27: the home page has an IP information API card. It shows the `curl`
command that looks up the visitor's own address at `/ip/{addr}`, and that
command's live answer (closes #156)
- 2026-10-27: on the AS page, a click on the heading of the IPv4 or the IPv6
prefix list folds or unfolds the list. Both lists start open (closes #155)
- 2026-10-27: the AS page lists its IPv4 and its IPv6 prefixes by mask length,
shortest first, and prefixes of one mask length by address. They used to be
listed by address (closes #153)
- 2026-10-27: on `/` and `/status`, a spinner beside the status turns one frame
each time the stats show more messages than the time before. A page that has
not seen more messages for three refreshes shows "stale" and the time since
then instead, in red (closes #152)
- 2026-10-27: the WHOIS details of a Brazilian AS keep their accented letters.
LACNIC's answer for such an AS comes from NIC.br, Brazil's registry, with each
accented letter replaced. The details now come from NIC.br's own server,
`whois.registro.br`. When that query fails, the details from LACNIC's answer
are kept and a warning is logged. After NIC.br answers that RouteWatch is over
its query limit, RouteWatch stops asking NIC.br for an hour (closes #174)
- 2026-10-27: `make asupdate` stops with an error when the download of `as.csv`
breaks off. The generator used to retry the failed read forever. A record that
does not parse is still logged and skipped (closes #177)
- 2026-10-27: `make asupdate` runs the new `script/asupdate`. It replaces the AS
list built into RouteWatch only when the generator succeeds and finds at least
one AS. A failed download used to replace the list with an empty file. The
generator reads `as.csv` from `ipverse/as-metadata`, the source's new name
(closes #173)
- 2026-10-27: the WHOIS details of an AS at RIPE, APNIC or AFRINIC come only
from its `aut-num` object and the organisation object it names. RIPE and APNIC
start their answer with the `as-block` object of the AS's block. Its dates
were stored as the AS's (closes #168)
- 2026-10-27: AS names keep their accented letters. The AS list built into
RouteWatch is rebuilt from today's `as.csv`, which added a country code
column. That file no longer lists AS 0 or the documentation AS numbers. A
WHOIS answer that is not valid UTF-8 is decoded as Latin-1, which RIPE and
APNIC send, before it is parsed and stored (closes #165)
- 2026-10-27: the Age column on the AS and prefix pages reads in words, such as
"5 minutes ago" or "2 weeks ago", instead of Go durations such as "360h0m0s
ago". An age under a minute reads in seconds, and one over a month in months
or years instead of a date (closes #163)
- 2026-10-26: a write to a stored AS, peering or peer keeps the earlier of the
two first-seen times and the later of the two last-seen times. An old update
from a collector catching up on a backlog no longer moves last seen back
(closes #162)
- 2026-10-26: the status page shows "Route Churn/min", the number of prefix
announcements and withdrawals, from all peers, in the last full clock minute.
`/api/v1/stats` reports it as `route_churn_per_min` in `stream`, which
replaces the moving average `route_churn_per_sec`. The messages and updates
per second on `/` and `/status` are whole numbers (closes #160)
- 2026-10-26: the WHOIS fetcher backs off more slowly. A failure makes its wait
1.5 times as long, up to one minute instead of five. Its queries to RIPE,
APNIC and AFRINIC leave out person and role objects, which RIPE counts against
a daily limit per address. A refused query is logged with the RIR's error line
(closes #159)
- 2026-10-26: the home page's AS name search lists the ASes whose handle or
description holds the text, as links to their pages, from the new
`/api/v1/search/as`; the search ignores case. When the database has no record
of an AS, its page takes the handle and description from the AS list built
into RouteWatch and still lists the AS's prefixes (closes #151)
- 2026-10-19: README "HTTP Endpoints" lists every route with what it answers,
"Database Schema" matches `schema.sql`, and the rest of the README matches the
code; this file brought up to date (closes #72)
@@ -248,6 +179,4 @@ tested.
# Future Steps
- the rest of https://git.eeqj.de/sneak/routewatch/issues/122, one PR each: the
routing table in the database, the prefix handler writing to it, the pages and
the API reading from it, then removing the per-peer route tables
None open.
cmd/asinfo-gen/main.go
+28 -60
View File
@@ -6,7 +6,6 @@ import (
"encoding/csv"
"encoding/json"
"errors"
"fmt"
"io"
"log"
"net/http"
@@ -15,21 +14,7 @@ import (
"strings"
)
const asCsvURL = "https://github.com/ipverse/as-metadata/raw/refs/heads/master/as.csv"
// The first columns of as.csv, the ones the generator reads. It reads past
// the columns after them, such as the country code.
const (
asnColumn = "asn"
handleColumn = "handle"
descriptionColumn = "description"
columnsRead = 3
)
var (
errUnexpectedHeader = errors.New("unexpected CSV header")
errNoASes = errors.New("no AS in the CSV")
)
const asCsvURL = "https://github.com/ipverse/asn-info/raw/refs/heads/master/as.csv"
// ASInfo represents information about an Autonomous System
type ASInfo struct {
@@ -60,66 +45,40 @@ func main() {
log.Fatalf("Failed to fetch CSV: HTTP %d", resp.StatusCode)
}
asInfos, err := parseCSV(resp.Body)
// Closed here rather than deferred: log.Fatalf exits without running
// deferred calls.
_ = resp.Body.Close()
if err != nil {
log.Fatalf("Failed to parse CSV: %v", err)
}
log.Printf("Parsed %d AS records", len(asInfos))
// Convert to JSON and output to stdout
encoder := json.NewEncoder(os.Stdout)
encoder.SetIndent("", "\t")
err = encoder.Encode(asInfos)
if err != nil {
log.Fatalf("Failed to encode JSON: %v", err)
}
}
// parseCSV reads as.csv from r and returns its ASes, with the handle and the
// description as the file spells them. A header whose first columns are not
// the AS number, the handle and the description is an error. A record that
// does not parse is logged and skipped. A read that fails, such as on a broken
// download, is an error. A file left with no AS is an error.
func parseCSV(r io.Reader) ([]ASInfo, error) {
reader := csv.NewReader(r)
// Parse CSV
reader := csv.NewReader(resp.Body)
// Read header
header, err := reader.Read()
if err != nil {
return nil, fmt.Errorf("read CSV header: %w", err)
log.Fatalf("Failed to read CSV header: %v", err)
}
if len(header) < columnsRead || header[0] != asnColumn ||
header[1] != handleColumn || header[2] != descriptionColumn {
return nil, fmt.Errorf("%w: %v", errUnexpectedHeader, header)
if len(header) != 3 || header[0] != "asn" || header[1] != "handle" ||
header[2] != "description" {
log.Fatalf("Unexpected CSV header: %v", header)
}
// Read all records
var asInfos []ASInfo
for {
// The reader returns an error for a record with a different number
// of fields than the header.
record, err := reader.Read()
if errors.Is(err, io.EOF) {
break
}
if _, ok := errors.AsType[*csv.ParseError](err); ok {
if err != nil {
log.Printf("Error reading CSV record: %v", err)
continue
}
// Any other error comes from the download, such as a broken
// connection. Every later read returns it again.
if err != nil {
return nil, fmt.Errorf("read CSV record: %w", err)
const expectedFields = 3
if len(record) != expectedFields {
log.Printf("Skipping invalid record: %v", record)
continue
}
asn, err := strconv.Atoi(record[0])
@@ -136,9 +95,18 @@ func parseCSV(r io.Reader) ([]ASInfo, error) {
})
}
if len(asInfos) == 0 {
return nil, errNoASes
}
// Closed here rather than deferred: log.Fatalf exits without running
// deferred calls.
_ = resp.Body.Close()
return asInfos, nil
log.Printf("Parsed %d AS records", len(asInfos))
// Convert to JSON and output to stdout
encoder := json.NewEncoder(os.Stdout)
encoder.SetIndent("", "\t")
err = encoder.Encode(asInfos)
if err != nil {
log.Fatalf("Failed to encode JSON: %v", err)
}
}
cmd/asinfo-gen/main_test.go
-142
View File
@@ -1,142 +0,0 @@
package main
import (
"errors"
"io"
"slices"
"strings"
"testing"
"testing/iotest"
"time"
)
// parseTimeLimit is how long a test waits for parseCSV to return before it
// fails.
const parseTimeLimit = 2 * time.Second
// TestParseCSVReadsTodaysLayout parses an as.csv with the country code column
// that the source added after the description. It expects each AS with its
// handle and description spelled exactly as in the file, accents included.
func TestParseCSVReadsTodaysLayout(t *testing.T) {
t.Parallel()
const asCSV = `asn,handle,description,country-code
1,LVLT-1,Level 3 Parent LLC,US
1251,FUNDACAO-AMPARO-PESQUISA,FUNDAÇÃO DE AMPARO À PESQUISA DO ESTADO SÃO PAULO,BR
`
got, err := parseCSV(strings.NewReader(asCSV))
if err != nil {
t.Fatalf("parseCSV: %v", err)
}
want := []ASInfo{
{ASN: 1, Handle: "LVLT-1", Description: "Level 3 Parent LLC"},
{
ASN: 1251,
Handle: "FUNDACAO-AMPARO-PESQUISA",
Description: "FUNDAÇÃO DE AMPARO À PESQUISA DO ESTADO " +
"SÃO PAULO",
},
}
if !slices.Equal(got, want) {
t.Errorf("parseCSV = %+v, want %+v", got, want)
}
}
// TestParseCSVRejectsAnotherHeader parses files whose first columns are not
// the AS number, the handle and the description. It expects an error rather
// than a list read from the wrong columns.
func TestParseCSVRejectsAnotherHeader(t *testing.T) {
t.Parallel()
for _, asCSV := range []string{
"asn,handle\n1,LVLT-1\n",
"asn,description,handle\n1,Level 3 Parent LLC,LVLT-1\n",
} {
_, err := parseCSV(strings.NewReader(asCSV))
if err == nil {
t.Errorf("parseCSV(%q): no error", asCSV)
}
}
}
// TestParseCSVRejectsAFileWithNoASes parses files that hold no AS, one with
// only the header and one whose only record has no valid AS number. It
// expects errNoASes, so that make asupdate keeps the old list.
func TestParseCSVRejectsAFileWithNoASes(t *testing.T) {
t.Parallel()
for _, asCSV := range []string{
"asn,handle,description,country-code\n",
"asn,handle,description,country-code\nAS1,LVLT-1,Level 3,US\n",
} {
_, err := parseCSV(strings.NewReader(asCSV))
if !errors.Is(err, errNoASes) {
t.Errorf("parseCSV(%q) = %v, want %v", asCSV, err, errNoASes)
}
}
}
// TestParseCSVSkipsARecordThatDoesNotParse parses an as.csv with a record
// that has too few fields between two valid records. It expects both valid
// ASes and no error, so that one bad record does not stop make asupdate.
func TestParseCSVSkipsARecordThatDoesNotParse(t *testing.T) {
t.Parallel()
const asCSV = "asn,handle,description,country-code\n" +
"1,LVLT-1,Level 3 Parent LLC,US\n" +
"2,UDEL-DCN\n" +
"3,MIT-GATEWAYS,Massachusetts Institute of Technology,US\n"
got, err := parseCSV(strings.NewReader(asCSV))
if err != nil {
t.Fatalf("parseCSV: %v", err)
}
want := []ASInfo{
{ASN: 1, Handle: "LVLT-1", Description: "Level 3 Parent LLC"},
{
ASN: 3,
Handle: "MIT-GATEWAYS",
Description: "Massachusetts Institute of Technology",
},
}
if !slices.Equal(got, want) {
t.Errorf("parseCSV = %+v, want %+v", got, want)
}
}
// TestParseCSVStopsWhenTheDownloadBreaksOff parses an as.csv whose download
// breaks off after the first record. From then on every read fails, as on a
// broken connection. It expects parseCSV to return that error, so that make
// asupdate keeps the old list.
func TestParseCSVStopsWhenTheDownloadBreaksOff(t *testing.T) {
t.Parallel()
const asCSV = "asn,handle,description,country-code\n" +
"1,LVLT-1,Level 3 Parent LLC,US\n"
download := io.MultiReader(
strings.NewReader(asCSV), iotest.ErrReader(io.ErrUnexpectedEOF))
// parseCSV runs on its own goroutine, so that a parseCSV that never
// returns fails the test instead of hanging it.
parsed := make(chan error, 1)
go func() {
_, err := parseCSV(download)
parsed <- err
}()
select {
case err := <-parsed:
if !errors.Is(err, io.ErrUnexpectedEOF) {
t.Errorf("parseCSV = %v, want %v", err, io.ErrUnexpectedEOF)
}
case <-time.After(parseTimeLimit):
t.Fatalf("parseCSV did not return within %v", parseTimeLimit)
}
}
go.mod
-4
View File
@@ -3,17 +3,13 @@ module git.eeqj.de/sneak/routewatch
go 1.27.1
require (
github.com/bits-and-blooms/bitset v1.26.0
github.com/dustin/go-humanize v1.0.1
github.com/go-chi/chi/v5 v5.2.2
github.com/google/btree v1.1.3
github.com/google/uuid v1.6.0
github.com/mattn/go-sqlite3 v1.14.29
github.com/rcrowley/go-metrics v0.0.0-20250401214520-65e299d6c5c9
go.uber.org/fx v1.24.0
go4.org/netipx v0.0.0-20260823151212-3075585bcbeb
golang.org/x/term v0.33.0
golang.org/x/text v0.43.0
)
require (
go.sum
-8
View File
@@ -1,13 +1,9 @@
github.com/bits-and-blooms/bitset v1.26.0 h1:WPvcPZbP1yuKofmjs2SYIFO1EjzIGhCP9ducElLsh5Q=
github.com/bits-and-blooms/bitset v1.26.0/go.mod h1:7hO7Gc7Pp1vODcmWvKMRA9BNmbv6a/7QIWpPxHddWR8=
github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c=
github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
github.com/dustin/go-humanize v1.0.1 h1:GzkhY7T5VNhEkwH0PVJgjz+fX1rhBrR7pRT3mDkpeCY=
github.com/dustin/go-humanize v1.0.1/go.mod h1:Mu1zIs6XwVuF/gI1OepvI0qD18qycQx+mFykh5fBlto=
github.com/go-chi/chi/v5 v5.2.2 h1:CMwsvRVTbXVytCk1Wd72Zy1LAsAh9GxMmSNWLHCG618=
github.com/go-chi/chi/v5 v5.2.2/go.mod h1:L2yAIGWB3H+phAw1NxKwWM+7eUH/lU8pOMm5hHcoops=
github.com/google/btree v1.1.3 h1:CVpQJjYgC4VbzxeGVHfvZrv1ctoYCAI8vbl07Fcxlyg=
github.com/google/btree v1.1.3/go.mod h1:qOPhT0dTNdNzV6Z/lhRX0YXUafgPLFUh+gZMl761Gm4=
github.com/google/uuid v1.6.0 h1:NIvaJDMOsjHA8n1jAhLSgzrAzy1Hgr+hNrb57e+94F0=
github.com/google/uuid v1.6.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo=
github.com/mattn/go-sqlite3 v1.14.29 h1:1O6nRLJKvsi1H2Sj0Hzdfojwt8GiGKm+LOfLaBFaouQ=
@@ -28,13 +24,9 @@ go.uber.org/multierr v1.10.0 h1:S0h4aNzvfcFsC3dRF1jLoaov7oRaKqRGC/pUEJ2yvPQ=
go.uber.org/multierr v1.10.0/go.mod h1:20+QtiLqy0Nd6FdQB9TLXag12DsQkrbs3htMFfDN80Y=
go.uber.org/zap v1.26.0 h1:sI7k6L95XOKS281NhVKOFCUNIvv9e0w4BF8N3u+tCRo=
go.uber.org/zap v1.26.0/go.mod h1:dtElttAiwGvoJ/vj4IwHBS/gXsEu/pZ50mUIRWuG0so=
go4.org/netipx v0.0.0-20260823151212-3075585bcbeb h1:XBM4hvfwGAttkkiTIFfeigdfcL1xIfdKXqFdgiHGtDs=
go4.org/netipx v0.0.0-20260823151212-3075585bcbeb/go.mod h1:PLyyIXexvUFg3Owu6p/WfdlivPbZJsZdgWZlrGope/Y=
golang.org/x/sys v0.34.0 h1:H5Y5sJ2L2JRdyv7ROF1he/lPdvFsd0mJHFw2ThKHxLA=
golang.org/x/sys v0.34.0/go.mod h1:BJP2sWEmIv4KK5OTEluFJCKSidICx8ciO85XgH3Ak8k=
golang.org/x/term v0.33.0 h1:NuFncQrRcaRvVmgRkvM3j/F00gWIAlcmlB8ACEKmGIg=
golang.org/x/term v0.33.0/go.mod h1:s18+ql9tYWp1IfpV9DmCtQDDSRBUjKaw9M1eAv5UeF0=
golang.org/x/text v0.43.0 h1:1QivrlhwAsnMOcqgOdl9my7s9OyZl8O5/UGUTJVXoWc=
golang.org/x/text v0.43.0/go.mod h1:hbSIYA/amXcRXTFSZqh8tWHEoqRvvbKPJ2uz/KU+a+s=
gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA=
gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
internal/blocklist/blocklist.go
-184
View File
@@ -1,184 +0,0 @@
// Package blocklist downloads public IP blocklists on a schedule, keeps the
// last good copy of each on disk and in memory, and answers which lists
// contain an address.
package blocklist
import (
"context"
"log/slog"
"net/netip"
"sync"
"time"
"go4.org/netipx"
)
// Format is how a list's file is written.
type Format int
const (
// FormatLines has one address or network per line, such as 192.0.2.1 or
// 192.0.2.0/24. Lines that start with # are comments.
FormatLines Format = iota
// FormatSpamhausJSON has one JSON object per line, with a network in its
// cidr field. The last line holds the file's metadata and no network.
FormatSpamhausJSON
// FormatDShield has one network per line, as tab-separated fields that
// start with the network's first and last address. Lines that start
// with # are comments.
FormatDShield
)
// List is one public blocklist.
type List struct {
// Name names the list in lookup answers, and names its copy on disk.
Name string
// Description says in one line what being on the list means.
Description string
// URL is where the publisher serves the whole list.
URL string
// Interval is how long a copy is used before the list is downloaded
// again.
Interval time.Duration
// Format is how the list's file is written.
Format Format
}
// Listing says that a list contains an address.
type Listing struct {
Name string `json:"name"`
Description string `json:"description"`
// Updated is when routewatch last downloaded the list, or last heard from
// its publisher that its copy was still current.
Updated time.Time `json:"updated"`
}
// listCopy is a list's last good copy in memory.
type listCopy struct {
addresses *netipx.IPSet
updated time.Time
// validators identify the version of the list this copy holds.
validators validators
}
// Params is what New needs.
type Params struct {
// Dir is the directory that holds the copies on disk.
Dir string
// Lists are the lists to download and look addresses up in.
Lists []List
Logger *slog.Logger
}
// Blocklists keeps a copy of each list and answers which lists contain an
// address. Start loads the copies from disk and starts downloading the lists.
type Blocklists struct {
dir string
lists []List
logger *slog.Logger
// copies holds each list's last good copy, by list name. A list with no
// good copy yet has none.
mu sync.RWMutex
copies map[string]listCopy
// stop ends the goroutine Start runs, and done is closed when it returns.
stop context.CancelFunc
done chan struct{}
}
// New returns a Blocklists with no copies. Start loads them.
func New(params Params) *Blocklists {
return &Blocklists{
dir: params.Dir,
lists: params.Lists,
logger: params.Logger.With("component", "blocklists"),
copies: make(map[string]listCopy),
done: make(chan struct{}),
}
}
// Lookup returns a Listing for each list whose copy contains addr, in the
// order of the lists given to New. It makes no outside request. When no list
// contains addr, it returns an empty list, not nil.
func (b *Blocklists) Lookup(addr netip.Addr) []Listing {
b.mu.RLock()
defer b.mu.RUnlock()
listings := []Listing{}
for _, list := range b.lists {
c, ok := b.copies[list.Name]
if ok && c.addresses.Contains(addr.Unmap()) {
listings = append(listings, Listing{
Name: list.Name,
Description: list.Description,
Updated: c.updated,
})
}
}
return listings
}
// ShortestInterval returns the shortest Interval of the lists. That is the
// soonest a copy can change, and so how long a lookup answer stays current.
func (b *Blocklists) ShortestInterval() time.Duration {
var shortest time.Duration
for i, list := range b.lists {
if i == 0 || list.Interval < shortest {
shortest = list.Interval
}
}
return shortest
}
// updated returns when the named list's copy was last downloaded or found
// current, or the zero time when the list has no copy.
func (b *Blocklists) updated(name string) time.Time {
b.mu.RLock()
defer b.mu.RUnlock()
return b.copies[name].updated
}
// copyValidators returns the validators of the named list's copy. They are
// empty when the list has no copy.
func (b *Blocklists) copyValidators(name string) validators {
b.mu.RLock()
defer b.mu.RUnlock()
return b.copies[name].validators
}
// setCopy makes c the named list's copy.
func (b *Blocklists) setCopy(name string, c listCopy) {
b.mu.Lock()
defer b.mu.Unlock()
b.copies[name] = c
}
// setUpdated records that the named list's copy was found current at t.
func (b *Blocklists) setUpdated(name string, t time.Time) {
b.mu.Lock()
defer b.mu.Unlock()
c, ok := b.copies[name]
if ok {
c.updated = t
b.copies[name] = c
}
}
internal/blocklist/blocklist_internal_test.go
-494
View File
@@ -1,494 +0,0 @@
package blocklist
import (
"context"
"errors"
"io"
"io/fs"
"log/slog"
"net/http"
"net/http/httptest"
"net/netip"
"os"
"path/filepath"
"strings"
"sync"
"testing"
"time"
)
// The test list's name and description, the address its good version lists,
// and the address a newer version lists instead.
const (
testListName = "test_list"
testListDescription = "On the test list"
listedAddr = "192.0.2.1"
goodListBody = "# A test list\n" + listedAddr + "\n"
newListedAddr = "192.0.2.2"
newListBody = "# A test list\n" + newListedAddr + "\n"
)
// testETag is the ETag a publisher sends with a version of the test list.
const testETag = `"v1"`
// maxWait is how long a test waits for the download goroutine.
const maxWait = 10 * time.Second
// publisher stands in for a list's publisher. It answers every request with
// its status and body, and keeps the headers of each request. A 200 answer
// carries lastModified and etag, when set, as Last-Modified and ETag. As
// from a real publisher, a request whose If-None-Match or If-Modified-Since
// matches them is answered "not modified".
type publisher struct {
mu sync.Mutex
status int
body string
lastModified time.Time
etag string
headers []http.Header
}
// newPublisher starts a publisher that answers 200 with body, and returns it
// and its URL.
func newPublisher(t *testing.T, body string) (*publisher, string) {
t.Helper()
p := &publisher{status: http.StatusOK, body: body}
srv := httptest.NewServer(p)
t.Cleanup(srv.Close)
return p, srv.URL
}
func (p *publisher) ServeHTTP(w http.ResponseWriter, r *http.Request) {
p.mu.Lock()
defer p.mu.Unlock()
p.headers = append(p.headers, r.Header.Clone())
if p.status != http.StatusOK {
w.WriteHeader(p.status)
_, _ = io.WriteString(w, p.body)
return
}
if p.etag != "" {
w.Header().Set(etagHeader, p.etag)
}
// ServeContent sends Last-Modified unless lastModified is zero, and
// answers "not modified" when the request's conditions match.
http.ServeContent(w, r, testListName, p.lastModified,
strings.NewReader(p.body))
}
// answer makes the publisher answer later requests with status and body.
func (p *publisher) answer(status int, body string) {
p.mu.Lock()
defer p.mu.Unlock()
p.status, p.body = status, body
}
// publish makes the publisher answer later requests with a version of the
// list: body, with lastModified and etag. A zero lastModified or an empty
// etag is not sent.
func (p *publisher) publish(body string, lastModified time.Time, etag string) {
p.mu.Lock()
defer p.mu.Unlock()
p.status, p.body = http.StatusOK, body
p.lastModified, p.etag = lastModified, etag
}
// requests returns the headers of the requests the publisher has had.
func (p *publisher) requests() []http.Header {
p.mu.Lock()
defer p.mu.Unlock()
return append([]http.Header(nil), p.headers...)
}
// testList returns a list served at url, written in format.
func testList(url string, format Format) List {
return List{
Name: testListName,
Description: testListDescription,
URL: url,
Interval: time.Hour,
Format: format,
}
}
func newTestBlocklists(dir string, list List) *Blocklists {
return New(Params{
Dir: dir,
Lists: []List{list},
Logger: slog.New(slog.DiscardHandler),
})
}
// lookup looks addr up in b.
func lookup(t *testing.T, b *Blocklists, addr string) []Listing {
t.Helper()
return b.Lookup(netip.MustParseAddr(addr))
}
// TestLookupFindsListedAddresses downloads a list in each format and looks
// up addresses on it, inside its networks, and off it, in IPv4 and IPv6.
// A listed address has one listing that names the list; any other has an
// empty list of listings.
func TestLookupFindsListedAddresses(t *testing.T) {
t.Parallel()
tests := []struct {
name string
format Format
body string
listed []string
unlisted []string
}{
{
name: "single addresses",
format: FormatLines,
body: "# comment\n192.0.2.1\n\n2001:db8::1\n",
listed: []string{"192.0.2.1", "::ffff:192.0.2.1", "2001:db8::1"},
unlisted: []string{"192.0.2.2", "2001:db8::2"},
},
{
name: "networks",
format: FormatLines,
body: "# comment\n198.51.100.0/24\n2001:db8:1::/48\n",
listed: []string{"198.51.100.0", "198.51.100.255", "2001:db8:1::5"},
unlisted: []string{"198.51.101.0", "2001:db8:2::1"},
},
{
name: "Spamhaus networks",
format: FormatSpamhausJSON,
body: `{"cidr":"203.0.113.0/25","sblid":"SBL1","rir":"arin"}` + "\n" +
`{"cidr":"2001:db8:3::/48","sblid":"SBL2","rir":"ripencc"}` + "\n" +
`{"type":"metadata","timestamp":1793074442,"records":2}` + "\n",
listed: []string{"203.0.113.1", "2001:db8:3::1"},
unlisted: []string{"203.0.113.200", "2001:db8:4::1"},
},
{
name: "DShield networks",
format: FormatDShield,
body: "# DShield.org Recommended Block List\n" +
"192.0.2.0\t192.0.2.255\t24\t327\tEXAMPLE\tUS\tNone\n",
listed: []string{"192.0.2.0", "192.0.2.77"},
unlisted: []string{"192.0.3.1", "2001:db8::1"},
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
t.Parallel()
_, url := newPublisher(t, tt.body)
b := newTestBlocklists(t.TempDir(), testList(url, tt.format))
err := b.download(t.Context(), b.lists[0])
if err != nil {
t.Fatalf("download: %v", err)
}
for _, addr := range tt.listed {
listings := lookup(t, b, addr)
if len(listings) != 1 || listings[0].Name != testListName ||
listings[0].Description != testListDescription ||
listings[0].Updated.IsZero() {
t.Errorf("%s: listings %+v, want one for %s", addr,
listings, testListName)
}
}
for _, addr := range tt.unlisted {
listings := lookup(t, b, addr)
if listings == nil || len(listings) != 0 {
t.Errorf("%s: listings %#v, want an empty list", addr, listings)
}
}
})
}
}
// TestFailedDownloadKeepsThePreviousCopy downloads a good version of a list,
// then one that fails. The good copy stays in memory and on disk, and no
// part of the failed download is left beside it.
func TestFailedDownloadKeepsThePreviousCopy(t *testing.T) {
t.Parallel()
tests := []struct {
name string
status int
body string
}{
{"server error", http.StatusInternalServerError, "unavailable"},
{"not a list", http.StatusOK, "<!DOCTYPE html>\n<html></html>\n"},
{
"over the size limit", http.StatusOK,
strings.Repeat(listedAddr+"\n", maxDownloadSize/len(listedAddr)),
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
t.Parallel()
dir := t.TempDir()
p, url := newPublisher(t, goodListBody)
b := newTestBlocklists(dir, testList(url, FormatLines))
err := b.download(t.Context(), b.lists[0])
if err != nil {
t.Fatalf("first download: %v", err)
}
before := lookup(t, b, listedAddr)
p.answer(tt.status, tt.body)
err = b.download(t.Context(), b.lists[0])
if err == nil {
t.Fatal("second download returned no error")
}
after := lookup(t, b, listedAddr)
if len(after) != 1 || len(before) != 1 ||
!after[0].Updated.Equal(before[0].Updated) {
t.Errorf("listings %+v after the failure, want %+v", after, before)
}
copies := os.DirFS(dir)
onDisk, err := fs.ReadFile(copies, testListName)
if err != nil || string(onDisk) != goodListBody {
t.Errorf("copy on disk %q (%v), want %q", onDisk, err, goodListBody)
}
_, err = fs.Stat(copies, testListName+downloadSuffix)
if !errors.Is(err, fs.ErrNotExist) {
t.Errorf("the failed download's file is still there (%v)", err)
}
})
}
}
// TestRestartLoadsCopiesFromDisk downloads a list, then starts a new
// Blocklists on the same directory, as a restart does. The new one answers
// from the copy on disk at once, and does not download the list again while
// the copy is younger than the list's interval.
func TestRestartLoadsCopiesFromDisk(t *testing.T) {
t.Parallel()
dir := t.TempDir()
p, url := newPublisher(t, goodListBody)
list := testList(url, FormatLines)
first := newTestBlocklists(dir, list)
err := first.download(t.Context(), list)
if err != nil {
t.Fatalf("download: %v", err)
}
downloaded := lookup(t, first, listedAddr)[0].Updated
second := newTestBlocklists(dir, list)
second.Start(t.Context())
second.Stop(t.Context())
listings := lookup(t, second, listedAddr)
if len(listings) != 1 ||
listings[0].Updated.Sub(downloaded).Abs() > time.Second {
t.Errorf("after the restart, listings %+v, want one updated at %s",
listings, downloaded)
}
if got := len(p.requests()); got != 1 {
t.Errorf("the publisher had %d requests, want 1", got)
}
}
// TestStartRefreshesAnOldCopy starts on a copy older than its list's
// interval. The list is downloaded at once, with routewatch's User-Agent and
// the Last-Modified and ETag the publisher sent with the copy, kept on disk.
// The publisher answers that the copy is still current, so the copy is kept
// and marked as updated now.
func TestStartRefreshesAnOldCopy(t *testing.T) {
t.Parallel()
dir := t.TempDir()
p, url := newPublisher(t, goodListBody)
list := testList(url, FormatLines)
lastModified := time.Now().Add(-3 * list.Interval).UTC().Truncate(time.Second)
p.publish(goodListBody, lastModified, testETag)
err := newTestBlocklists(dir, list).download(t.Context(), list)
if err != nil {
t.Fatalf("download: %v", err)
}
old := time.Now().Add(-2 * list.Interval).UTC().Truncate(time.Second)
err = os.Chtimes(filepath.Join(dir, testListName), old, old)
if err != nil {
t.Fatalf("Chtimes: %v", err)
}
b := newTestBlocklists(dir, list)
b.Start(t.Context())
deadline := time.Now().Add(maxWait)
for !lookup(t, b, listedAddr)[0].Updated.After(old) {
if time.Now().After(deadline) {
t.Fatalf("the copy was not marked updated within %s", maxWait)
}
time.Sleep(time.Millisecond)
}
b.Stop(t.Context())
requests := p.requests()
last := requests[len(requests)-1]
want := lastModified.Format(http.TimeFormat)
if got := last.Get(ifModifiedSinceHeader); got != want {
t.Errorf("If-Modified-Since %q, want %q", got, want)
}
if got := last.Get(ifNoneMatchHeader); got != testETag {
t.Errorf("If-None-Match %q, want %q", got, testETag)
}
if got := last.Get(userAgentHeader); got != userAgent {
t.Errorf("User-Agent %q, want %q", got, userAgent)
}
}
// TestNewVersionOlderThanTheLastDownloadReplacesTheCopy downloads a list.
// The publisher then puts out a new version whose Last-Modified is earlier
// than that download, as when a cache in front of the publisher passes the
// version on late. The next download sends the Last-Modified of the copy,
// not the time of the download, so it gets the new version.
func TestNewVersionOlderThanTheLastDownloadReplacesTheCopy(t *testing.T) {
t.Parallel()
p, url := newPublisher(t, goodListBody)
list := testList(url, FormatLines)
b := newTestBlocklists(t.TempDir(), list)
firstVersion := time.Now().Add(-2 * list.Interval).UTC().Truncate(time.Second)
p.publish(goodListBody, firstVersion, "")
err := b.download(t.Context(), list)
if err != nil {
t.Fatalf("first download: %v", err)
}
p.publish(newListBody, firstVersion.Add(list.Interval), "")
err = b.download(t.Context(), list)
if err != nil {
t.Fatalf("second download: %v", err)
}
if listings := lookup(t, b, newListedAddr); len(listings) != 1 {
t.Errorf("%s: listings %+v, want one from the new version",
newListedAddr, listings)
}
want := firstVersion.Format(http.TimeFormat)
if got := p.requests()[1].Get(ifModifiedSinceHeader); got != want {
t.Errorf("If-Modified-Since %q, want %q", got, want)
}
}
// TestDownloadWithoutLastModifiedOrETagSendsNoCondition downloads a list
// twice from a publisher that sends neither Last-Modified nor ETag. Neither
// request carries If-Modified-Since or If-None-Match, so the second download
// gets the whole list again.
func TestDownloadWithoutLastModifiedOrETagSendsNoCondition(t *testing.T) {
t.Parallel()
p, url := newPublisher(t, goodListBody)
list := testList(url, FormatLines)
b := newTestBlocklists(t.TempDir(), list)
err := b.download(t.Context(), list)
if err != nil {
t.Fatalf("first download: %v", err)
}
err = b.download(t.Context(), list)
if err != nil {
t.Fatalf("second download: %v", err)
}
for i, request := range p.requests() {
if request.Get(ifModifiedSinceHeader) != "" ||
request.Get(ifNoneMatchHeader) != "" {
t.Errorf("request %d has a condition: %v", i, request)
}
}
}
// TestStopCancelsADownloadInProgress stops while the publisher has not
// answered. Stop cancels the download and returns once the download
// goroutine has stopped.
func TestStopCancelsADownloadInProgress(t *testing.T) {
t.Parallel()
arrived := make(chan struct{})
var once sync.Once
srv := httptest.NewServer(http.HandlerFunc(
func(_ http.ResponseWriter, r *http.Request) {
once.Do(func() { close(arrived) })
<-r.Context().Done()
}))
t.Cleanup(srv.Close)
b := newTestBlocklists(t.TempDir(), testList(srv.URL, FormatLines))
b.Start(t.Context())
select {
case <-arrived:
case <-time.After(maxWait):
t.Fatalf("no download began within %s", maxWait)
}
ctx, cancel := context.WithTimeout(t.Context(), maxWait)
defer cancel()
b.Stop(ctx)
select {
case <-b.done:
default:
t.Error("Stop returned before the download goroutine stopped")
}
}
// TestPublicListsAreComplete checks that every public list has what a
// download and a lookup answer need, and a unique name.
func TestPublicListsAreComplete(t *testing.T) {
t.Parallel()
names := make(map[string]bool)
for _, list := range Lists() {
if list.Name == "" || list.Description == "" || list.URL == "" ||
list.Interval <= 0 || names[list.Name] {
t.Errorf("list %+v is incomplete or has a name used before", list)
}
names[list.Name] = true
}
}
internal/blocklist/download.go
-379
View File
@@ -1,379 +0,0 @@
package blocklist
import (
"context"
"encoding/json"
"errors"
"fmt"
"io"
"io/fs"
"net/http"
"os"
"slices"
"time"
"go4.org/netipx"
)
// Limits of one download.
const (
// downloadTimeout is how long one download may take.
downloadTimeout = time.Minute
// maxDownloadSize is the size of the largest list file accepted, in
// bytes. The largest list, FireHOL abusers, is about 2.5 MiB.
maxDownloadSize = 16 << 20
)
// The request headers a download sets.
const (
userAgentHeader = "User-Agent"
ifModifiedSinceHeader = "If-Modified-Since"
ifNoneMatchHeader = "If-None-Match"
// userAgent names routewatch to the publishers.
userAgent = "routewatch (+https://git.eeqj.de/sneak/routewatch)"
)
// The response headers that identify the version of a list a publisher sent.
const (
lastModifiedHeader = "Last-Modified"
etagHeader = "ETag"
)
const (
// downloadSuffix ends the name of the file a download is written to,
// beside the list's copy, until it has parsed.
downloadSuffix = ".download"
// validatorsSuffix ends the name of the file beside a list's copy that
// keeps the copy's validators.
validatorsSuffix = ".validators"
// dirPermissions are those of the directory that holds the copies.
dirPermissions = 0o750
// validatorsPermissions are those of the file that keeps a copy's
// validators.
validatorsPermissions = 0o600
)
// validators are the Last-Modified and ETag headers a publisher sent with a
// version of a list. A download sends them back, as If-Modified-Since and
// If-None-Match, so that the publisher sends the list only when it has
// changed. They are kept as the publisher wrote them.
//
//nolint:tagliatelle // snake_case keys, as the Go style guide asks
type validators struct {
LastModified string `json:"last_modified,omitempty"`
ETag string `json:"etag,omitempty"`
}
var (
errUnexpectedStatus = errors.New("unexpected HTTP status")
errTooLarge = errors.New("list is over the size limit")
)
// Start loads each list's copy from disk. It then starts the goroutine that
// downloads each list once its copy is older than its Interval, and a list
// with no copy at once. The goroutine runs until Stop: the end of ctx does
// not stop it.
func (b *Blocklists) Start(ctx context.Context) {
err := os.MkdirAll(b.dir, dirPermissions)
if err != nil {
b.logger.Error("Failed to create the blocklists directory",
"dir", b.dir, "error", err)
}
nextDownload := make([]time.Time, len(b.lists))
for i, list := range b.lists {
loadErr := b.loadCopy(list)
if loadErr != nil && !errors.Is(loadErr, fs.ErrNotExist) {
b.logger.Error("Failed to load a blocklist copy",
"list", list.Name, "error", loadErr)
}
// A list with no copy has the zero time, so it is due at once.
nextDownload[i] = b.updated(list.Name).Add(list.Interval)
}
ctx, b.stop = context.WithCancel(context.WithoutCancel(ctx))
go func() {
b.run(ctx, nextDownload)
}()
b.logger.Info("Blocklist downloads started", "dir", b.dir)
}
// Stop stops the downloads, cancelling one in progress, and waits until the
// goroutine has returned or ctx ends. The copies in memory stay, so Lookup
// still answers.
func (b *Blocklists) Stop(ctx context.Context) {
b.stop()
select {
case <-b.done:
b.logger.Info("Blocklist downloads stopped")
case <-ctx.Done():
b.logger.Warn("Stop deadline passed before the blocklist downloads " +
"stopped")
}
}
// run downloads each list when nextDownload says, until ctx ends. After each
// try, whether it worked or not, the list's next download is an Interval
// later.
func (b *Blocklists) run(ctx context.Context, nextDownload []time.Time) {
defer close(b.done)
if len(nextDownload) == 0 {
return
}
for {
soonest := slices.MinFunc(nextDownload, time.Time.Compare)
timer := time.NewTimer(time.Until(soonest))
select {
case <-ctx.Done():
timer.Stop()
return
case <-timer.C:
}
for i, list := range b.lists {
if time.Now().Before(nextDownload[i]) {
continue
}
err := b.download(ctx, list)
if ctx.Err() != nil {
return
}
if err != nil {
b.logger.Error("Blocklist download failed; keeping the "+
"previous copy", "list", list.Name, "error", err)
}
nextDownload[i] = time.Now().Add(list.Interval)
}
}
}
// download asks the publisher for list, sending the validators of the list's
// copy when it has them. A new version that parses becomes the list's copy,
// on disk and in memory. An answer that the copy is still current marks the
// copy as updated now. Anything else leaves the copy as it is and returns an
// error.
func (b *Blocklists) download(ctx context.Context, list List) error {
ctx, cancel := context.WithTimeout(ctx, downloadTimeout)
defer cancel()
req, err := http.NewRequestWithContext(ctx, http.MethodGet, list.URL, nil)
if err != nil {
return err
}
req.Header.Set(userAgentHeader, userAgent)
held := b.copyValidators(list.Name)
if held.LastModified != "" {
req.Header.Set(ifModifiedSinceHeader, held.LastModified)
}
if held.ETag != "" {
req.Header.Set(ifNoneMatchHeader, held.ETag)
}
resp, err := http.DefaultClient.Do(req)
if err != nil {
return err
}
defer func() { _ = resp.Body.Close() }()
switch resp.StatusCode {
case http.StatusOK:
return b.replaceCopy(list, resp.Body, validators{
LastModified: resp.Header.Get(lastModifiedHeader),
ETag: resp.Header.Get(etagHeader),
})
case http.StatusNotModified:
return b.markCurrent(list)
default:
return fmt.Errorf("%w: %s", errUnexpectedStatus, resp.Status)
}
}
// replaceCopy writes a new version of list, read from body, beside the list's
// copy. Only when it parses does it replace the copy, on disk and in memory.
// sent are the validators the publisher sent with the new version.
func (b *Blocklists) replaceCopy(list List, body io.Reader, sent validators) error {
root, err := os.OpenRoot(b.dir)
if err != nil {
return err
}
defer func() { _ = root.Close() }()
downloadName := list.Name + downloadSuffix
addresses, err := writeAndParse(root, downloadName, body, list.Format)
if err != nil {
_ = root.Remove(downloadName)
return err
}
// The old copy's validators are removed before the copy is replaced, so
// that they are never kept with the new copy.
err = root.Remove(list.Name + validatorsSuffix)
if err != nil && !errors.Is(err, fs.ErrNotExist) {
return err
}
err = root.Rename(downloadName, list.Name)
if err != nil {
return err
}
b.setCopy(list.Name, listCopy{
addresses: addresses, updated: time.Now().UTC(), validators: sent,
})
b.logger.Info("Downloaded a new version of a blocklist", "list", list.Name)
// Without this file, the first download after a restart asks for the
// whole list.
err = writeValidators(root, list.Name, sent)
if err != nil {
b.logger.Error("Failed to keep the validators of a blocklist copy",
"list", list.Name, "error", err)
}
return nil
}
// writeAndParse writes body to the file name in root, up to maxDownloadSize
// bytes, and parses the file as format.
func writeAndParse(
root *os.Root, name string, body io.Reader, format Format,
) (*netipx.IPSet, error) {
f, err := root.Create(name)
if err != nil {
return nil, err
}
defer func() { _ = f.Close() }()
written, err := io.Copy(f, io.LimitReader(body, maxDownloadSize+1))
if err != nil {
return nil, err
}
if written > maxDownloadSize {
return nil, errTooLarge
}
_, err = f.Seek(0, io.SeekStart)
if err != nil {
return nil, err
}
return parse(f, format)
}
// markCurrent records that the publisher says the list's copy is still
// current: the copy, on disk and in memory, is marked as updated now.
func (b *Blocklists) markCurrent(list List) error {
root, err := os.OpenRoot(b.dir)
if err != nil {
return err
}
defer func() { _ = root.Close() }()
now := time.Now().UTC()
err = root.Chtimes(list.Name, now, now)
if err != nil {
return err
}
b.setUpdated(list.Name, now)
b.logger.Debug("Blocklist copy is still current", "list", list.Name)
return nil
}
// loadCopy makes the list's copy on disk, with the validators kept beside it,
// its copy in memory. The file's modification time is when the copy was
// updated.
func (b *Blocklists) loadCopy(list List) error {
root, err := os.OpenRoot(b.dir)
if err != nil {
return err
}
defer func() { _ = root.Close() }()
f, err := root.Open(list.Name)
if err != nil {
return err
}
defer func() { _ = f.Close() }()
info, err := f.Stat()
if err != nil {
return err
}
addresses, err := parse(f, list.Format)
if err != nil {
return err
}
held, err := readValidators(root, list.Name)
if err != nil {
return err
}
b.setCopy(list.Name, listCopy{
addresses: addresses, updated: info.ModTime().UTC(), validators: held,
})
return nil
}
// writeValidators keeps held in the file beside the named list's copy.
func writeValidators(root *os.Root, name string, held validators) error {
data, err := json.Marshal(held)
if err != nil {
return err
}
return root.WriteFile(name+validatorsSuffix, data, validatorsPermissions)
}
// readValidators returns the validators kept beside the named list's copy.
// They are empty when there is no such file.
func readValidators(root *os.Root, name string) (validators, error) {
var held validators
data, err := root.ReadFile(name + validatorsSuffix)
if errors.Is(err, fs.ErrNotExist) {
return held, nil
}
if err != nil {
return held, err
}
err = json.Unmarshal(data, &held)
return held, err
}
internal/blocklist/lists.go
-93
View File
@@ -1,93 +0,0 @@
package blocklist
import "time"
// How often each list is downloaded. No list is downloaded more often than
// its publisher updates it or allows.
const (
// DShield asks that its feeds be downloaded at most once an hour:
// https://isc.sans.edu/feeds_doc.html
dshieldInterval = time.Hour
// abuse.ch makes the list every 5 minutes and recommends a download at
// least every 15: https://feodotracker.abuse.ch/blocklist/
feodoTrackerInterval = 15 * time.Minute
// FireHOL's page for each list gives how often the list changes on
// average: about every 5 hours for abusers and every 3 hours for level
// 3. FireHOL sets no download limit.
// https://iplists.firehol.org/?ipset=firehol_abusers_30d
// https://iplists.firehol.org/?ipset=firehol_level3
fireHOLAbusersInterval = 5 * time.Hour
fireHOLLevel3Interval = 3 * time.Hour
// Spamhaus asks that DROP be downloaded at most once a day:
// https://www.spamhaus.org/blocklists/do-not-route-or-peer/
spamhausDROPInterval = 24 * time.Hour
// Tor publishes a new exit list every hour and sets no download limit:
// https://collector.torproject.org/recent/exit-lists/
torExitsInterval = time.Hour
)
// Lists returns the public blocklists routewatch downloads. Each can be
// downloaded whole without an account or a key. Spamhaus merged its EDROP
// list into DROP, so DROP covers both.
func Lists() []List {
return []List{
{
Name: "dshield",
Description: "In one of the 20 networks that DShield saw scan " +
"the most targets in the last three days",
URL: "https://feeds.dshield.org/block.txt",
Interval: dshieldInterval,
Format: FormatDShield,
},
{
Name: "feodo_tracker",
Description: "A botnet command-and-control server",
URL: "https://feodotracker.abuse.ch/downloads/ipblocklist.txt",
Interval: feodoTrackerInterval,
Format: FormatLines,
},
{
Name: "firehol_abusers_30d",
Description: "Reported for comment, form or forum spam in the " +
"last 30 days",
URL: "https://iplists.firehol.org/files/firehol_abusers_30d.netset",
Interval: fireHOLAbusersInterval,
Format: FormatLines,
},
{
Name: "firehol_level3",
Description: "Reported for attacks, spyware or viruses in the " +
"last 30 days",
URL: "https://iplists.firehol.org/files/firehol_level3.netset",
Interval: fireHOLLevel3Interval,
Format: FormatLines,
},
{
Name: "spamhaus_drop",
Description: "In an IPv4 network that Spamhaus says spammers " +
"or cyber criminals have hijacked or lease",
URL: "https://www.spamhaus.org/drop/drop_v4.json",
Interval: spamhausDROPInterval,
Format: FormatSpamhausJSON,
},
{
Name: "spamhaus_dropv6",
Description: "In an IPv6 network that Spamhaus says spammers " +
"or cyber criminals have hijacked or lease",
URL: "https://www.spamhaus.org/drop/drop_v6.json",
Interval: spamhausDROPInterval,
Format: FormatSpamhausJSON,
},
{
Name: "tor_exits",
Description: "A Tor exit node, where traffic from the Tor network leaves",
URL: "https://check.torproject.org/torbulkexitlist",
Interval: torExitsInterval,
Format: FormatLines,
},
}
}
internal/blocklist/parse.go
-158
View File
@@ -1,158 +0,0 @@
package blocklist
import (
"bufio"
"encoding/json"
"errors"
"fmt"
"io"
"net/netip"
"strings"
"go4.org/netipx"
)
const (
// commentPrefix starts a comment line in a list file.
commentPrefix = "#"
// networkSeparator separates the address and the mask length of a
// network, as in 192.0.2.0/24.
networkSeparator = "/"
// dshieldFieldSeparator separates the fields of a DShield line.
dshieldFieldSeparator = "\t"
// spamhausMetadataType is the type of the line that ends a Spamhaus
// file. That line holds the file's metadata and no network.
spamhausMetadataType = "metadata"
)
var (
errUnknownFormat = errors.New("unknown list format")
errNoNetwork = errors.New("no network on the line")
errBadRange = errors.New("first and last address make no range")
)
// parse reads a list file written in format and returns the addresses it
// holds. A line it cannot read fails the whole file, so a file that is not
// the list, such as an error page, never replaces a good copy.
func parse(r io.Reader, format Format) (*netipx.IPSet, error) {
var builder netipx.IPSetBuilder
scanner := bufio.NewScanner(r)
for lineNumber := 1; scanner.Scan(); lineNumber++ {
line := strings.TrimSpace(scanner.Text())
if line == "" || strings.HasPrefix(line, commentPrefix) {
continue
}
err := addLine(&builder, line, format)
if err != nil {
return nil, fmt.Errorf("line %d: %w", lineNumber, err)
}
}
err := scanner.Err()
if err != nil {
return nil, err
}
return builder.IPSet()
}
// addLine adds the address or network on one line of a list file to builder.
func addLine(builder *netipx.IPSetBuilder, line string, format Format) error {
switch format {
case FormatLines:
return addAddressOrNetwork(builder, line)
case FormatSpamhausJSON:
return addSpamhausNetwork(builder, line)
case FormatDShield:
return addDShieldNetwork(builder, line)
default:
return errUnknownFormat
}
}
// addAddressOrNetwork adds an address, such as 192.0.2.1, or a network, such
// as 192.0.2.0/24, to builder.
func addAddressOrNetwork(builder *netipx.IPSetBuilder, text string) error {
if strings.Contains(text, networkSeparator) {
network, err := netip.ParsePrefix(text)
if err != nil {
return err
}
builder.AddPrefix(network)
return nil
}
addr, err := netip.ParseAddr(text)
if err != nil {
return err
}
builder.Add(addr)
return nil
}
// addSpamhausNetwork adds the network on a line of a Spamhaus file to
// builder. The metadata line adds nothing.
func addSpamhausNetwork(builder *netipx.IPSetBuilder, line string) error {
var entry struct {
CIDR string `json:"cidr"`
Type string `json:"type"`
}
err := json.Unmarshal([]byte(line), &entry)
if err != nil {
return err
}
if entry.Type == spamhausMetadataType {
return nil
}
network, err := netip.ParsePrefix(entry.CIDR)
if err != nil {
return err
}
builder.AddPrefix(network)
return nil
}
// addDShieldNetwork adds the network on a line of a DShield file to builder.
// The line's first two fields are the network's first and last address.
func addDShieldNetwork(builder *netipx.IPSetBuilder, line string) error {
firstField, rest, found := strings.Cut(line, dshieldFieldSeparator)
if !found {
return errNoNetwork
}
lastField, _, _ := strings.Cut(rest, dshieldFieldSeparator)
first, err := netip.ParseAddr(firstField)
if err != nil {
return err
}
last, err := netip.ParseAddr(lastField)
if err != nil {
return err
}
network := netipx.IPRangeFrom(first, last)
if !network.IsValid() {
return errBadRange
}
builder.AddRange(network)
return nil
}
internal/database/database.go
+43 -128
View File
@@ -21,7 +21,6 @@ import (
"git.eeqj.de/sneak/routewatch/internal/config"
"git.eeqj.de/sneak/routewatch/internal/logger"
"git.eeqj.de/sneak/routewatch/internal/ristypes"
"git.eeqj.de/sneak/routewatch/pkg/asinfo"
_ "github.com/mattn/go-sqlite3" // CGO SQLite driver
)
@@ -170,20 +169,6 @@ func New(cfg *config.Config, logger *logger.Logger) (*Database, error) {
return nil, fmt.Errorf("failed to create database directory: %w", err)
}
// The size of the WAL file the previous run left, which Initialize logs
// with the start-up checkpoint. It is read before the database is opened,
// because opening it writes a frame to the WAL. There is no WAL file when
// the database is new or the last run closed the database, which deletes
// it.
var walBytes int64
wal, err := os.Stat(dbPath + walFileSuffix)
if err == nil {
walBytes = wal.Size()
} else if !errors.Is(err, os.ErrNotExist) {
logger.Warn("Failed to read the WAL file's size on startup", "error", err)
}
// Per-connection SQLite settings go in the DSN so every pooled connection
// gets them, not just the one that runs the Initialize pragmas. _txlock=
// immediate makes every transaction take the write lock at BEGIN. Without it
@@ -244,7 +229,7 @@ func New(cfg *config.Config, logger *logger.Logger) (*Database, error) {
counts: &liveCounts{},
}
err = database.Initialize(walBytes)
err = database.Initialize()
if err != nil {
return nil, fmt.Errorf("failed to initialize database: %w", err)
}
@@ -260,16 +245,8 @@ func New(cfg *config.Config, logger *logger.Logger) (*Database, error) {
return database, nil
}
// The messages Initialize logs the start-up WAL checkpoint with.
const (
startupCheckpointMessage = "WAL checkpoint on startup"
startupCheckpointIncompleteMessage = "WAL checkpoint on startup incomplete"
)
// Initialize creates the database schema if it doesn't exist. walBytes is the
// size of the WAL file the previous run left, which New reads before it opens
// the database.
func (d *Database) Initialize(walBytes int64) error {
// Initialize creates the database schema if it doesn't exist.
func (d *Database) Initialize() error {
// Set SQLite pragmas for performance. Per-connection settings (cache_size,
// synchronous, busy_timeout, journal_mode) live in the DSN, and New sets the
// writing connection's cache_size; temp_store is left at its default so
@@ -289,33 +266,19 @@ func (d *Database) Initialize(walBytes int64) error {
}
}
// The start-up checkpoint copies the WAL a previous run left into the
// database file and empties the WAL file; its frame counts then read 0,
// so walBytes is logged with them.
var busy, walFrames, checkpointedFrames int
// Run WAL checkpoint on startup to consolidate any existing WAL
var walPages, checkpointedPages, movedPages int
err := d.writer.QueryRowContext(context.Background(),
"PRAGMA wal_checkpoint(TRUNCATE)",
).Scan(&busy, &walFrames, &checkpointedFrames)
switch {
case err != nil:
).Scan(&walPages, &checkpointedPages, &movedPages)
if err != nil {
d.logger.Warn("Failed to checkpoint WAL on startup", "error", err)
case busy != 0:
// Another process using the database kept the checkpoint from
// copying or emptying all of the WAL.
d.logger.Warn(startupCheckpointIncompleteMessage,
"wal_bytes", walBytes,
"busy", busy,
"wal_frames", walFrames,
"checkpointed_frames", checkpointedFrames,
)
default:
d.logger.Info(startupCheckpointMessage,
"wal_bytes", walBytes,
"busy", busy,
"wal_frames", walFrames,
"checkpointed_frames", checkpointedFrames,
} else {
d.logger.Info("WAL checkpoint on startup",
"wal_pages", walPages,
"checkpointed", checkpointedPages,
"moved", movedPages,
)
}
@@ -892,7 +855,7 @@ func insertPrefixesInTable(
// UpdatePrefixesBatch stores, in a single transaction, the prefixes that are
// not stored yet, each first seen at its time in prefixes: the IPv4 prefixes
// first and then the IPv6 prefixes. A prefix that does not parse is not stored.
// first and then the IPv6 prefixes.
func (d *Database) UpdatePrefixesBatch(prefixes map[string]time.Time) error {
if len(prefixes) == 0 {
return nil
@@ -911,11 +874,6 @@ func (d *Database) UpdatePrefixesBatch(prefixes map[string]time.Time) error {
prefixesV6 := make(map[string]time.Time)
for prefix, timestamp := range prefixes {
_, err := netip.ParsePrefix(prefix)
if err != nil {
continue
}
if detectIPVersion(prefix) == ipVersionV4 {
prefixesV4[prefix] = timestamp
} else {
@@ -945,26 +903,6 @@ func (d *Database) UpdatePrefixesBatch(prefixes map[string]time.Time) error {
return nil
}
// These statements update a stored AS, peering or peer. Each keeps the earlier
// of the stored and the new first_seen, and the later of the stored and the new
// last_seen. Some collectors send updates days old while they catch up on a
// backlog. MIN and MAX compare the times as text, which is time order here: the
// driver writes every time in one format, and every time is UTC.
const (
updateASNSQL = `UPDATE asns
SET first_seen = MIN(first_seen, ?), last_seen = MAX(last_seen, ?)
WHERE asn = ?`
updatePeeringSQL = `UPDATE peerings
SET first_seen = MIN(first_seen, ?), last_seen = MAX(last_seen, ?)
WHERE as_a = ? AND as_b = ?`
updatePeerSQL = `UPDATE bgp_peers
SET peer_asn = ?, last_message_type = ?,
first_seen = MIN(first_seen, ?), last_seen = MAX(last_seen, ?)
WHERE peer_ip = ?`
)
// newASN returns a new ASN record first seen at seen.First and last seen at
// seen.Last, with the handle and description from the AS info lookup when it
// knows the number.
@@ -984,8 +922,9 @@ func newASN(number int, seen SeenTimes) ASN {
return asn
}
// upsertASN stores the seen times of ASN number with the given statements,
// inserting the ASN when it is new. It reports whether it inserted it.
// upsertASN sets the last_seen time of ASN number to seen.Last with the given
// statements, inserting the ASN when it is new. It reports whether it inserted
// it.
func upsertASN(
ctx context.Context, selectStmt, updateStmt, insertStmt *sql.Stmt,
number int, seen SeenTimes,
@@ -998,8 +937,8 @@ func upsertASN(
err := selectStmt.QueryRowContext(ctx, number).Scan(&asn.ASN, &handle,
&description, &asn.FirstSeen, &asn.LastSeen)
if err == nil {
// ASN exists, update its seen times
_, err = updateStmt.ExecContext(ctx, seen.First, seen.Last, number)
// ASN exists, update last_seen
_, err = updateStmt.ExecContext(ctx, seen.Last, number)
if err != nil {
return false, fmt.Errorf("failed to update ASN %d: %w", number, err)
}
@@ -1049,7 +988,8 @@ func (d *Database) GetOrCreateASNBatch(asns map[int]SeenTimes) error {
}
defer func() { _ = selectStmt.Close() }()
updateStmt, err := tx.PrepareContext(ctx, updateASNSQL)
updateStmt, err := tx.PrepareContext(ctx,
"UPDATE asns SET last_seen = ? WHERE asn = ?")
if err != nil {
return fmt.Errorf("failed to prepare update statement: %w", err)
}
@@ -1108,19 +1048,17 @@ func (d *Database) GetOrCreateASN(number int, timestamp time.Time) (*ASN, error)
"FROM asns WHERE asn = ?", number).
Scan(&asn.ASN, &handle, &description, &asn.FirstSeen, &asn.LastSeen)
if err == nil {
// ASN exists, update its seen times
// ASN exists, update last_seen
asn.Handle = handle.String
asn.Description = description.String
_, err = tx.Exec(updateASNSQL, timestamp, timestamp, number)
_, err = tx.Exec("UPDATE asns SET last_seen = ? WHERE asn = ?",
timestamp, number)
if err != nil {
return nil, err
}
// Return the seen times the update stored.
seen := SeenTimes{First: asn.FirstSeen, Last: asn.LastSeen}
seen.Add(timestamp)
asn.FirstSeen, asn.LastSeen = seen.First, seen.Last
asn.LastSeen = timestamp
err = tx.Commit()
if err != nil {
@@ -1279,7 +1217,9 @@ func (d *Database) RecordPeering(asA, asB int, seen SeenTimes) error {
}
if exists {
_, err = tx.Exec(updatePeeringSQL, seen.First, seen.Last, asA, asB)
_, err = tx.Exec(
"UPDATE peerings SET last_seen = ? WHERE as_a = ? AND as_b = ?",
seen.Last, asA, asB)
} else {
_, err = tx.Exec(`
INSERT INTO peerings (id, as_a, as_b, first_seen, last_seen)
@@ -1334,7 +1274,9 @@ func (d *Database) UpdatePeerBatch(peers map[string]PeerUpdate) error {
}
defer func() { _ = checkStmt.Close() }()
updateStmt, err := tx.PrepareContext(ctx, updatePeerSQL)
updateStmt, err := tx.PrepareContext(ctx,
"UPDATE bgp_peers SET peer_asn = ?, last_seen = ?, "+
"last_message_type = ? WHERE peer_ip = ?")
if err != nil {
return fmt.Errorf("failed to prepare update statement: %w", err)
}
@@ -1360,8 +1302,7 @@ func (d *Database) UpdatePeerBatch(peers map[string]PeerUpdate) error {
if exists {
_, err = updateStmt.ExecContext(ctx, update.PeerASN,
update.MessageType, update.Seen.First, update.Seen.Last,
update.PeerIP)
update.Seen.Last, update.MessageType, update.PeerIP)
} else {
_, err = insertStmt.ExecContext(ctx,
generateUUID().String(), update.PeerIP, update.PeerASN,
@@ -1410,8 +1351,11 @@ func (d *Database) UpdatePeer(
}
if exists {
_, err = tx.Exec(updatePeerSQL,
peerASN, messageType, timestamp, timestamp, peerIP)
_, err = tx.Exec(
"UPDATE bgp_peers SET peer_asn = ?, last_seen = ?, "+
"last_message_type = ? WHERE peer_ip = ?",
peerASN, timestamp, messageType, peerIP,
)
} else {
_, err = tx.Exec(
"INSERT INTO bgp_peers (id, peer_ip, peer_asn, first_seen, "+
@@ -1763,21 +1707,9 @@ func (d *Database) GetASDetails(asn int) (*ASN, []LiveRoute, error) {
// GetASDetailsContext returns the AS and the prefixes it originates, the IPv4
// ones first, each once with the time of its newest route. It makes two index
// searches per prefix, however many peers carry the prefix.
//
// The routes of an AS can be stored before its record in the asns table. When
// the AS has no record, its handle and description come from the AS info
// lookup, and it has no seen times. An AS with no record, no lookup entry and
// no prefix is ErrNoRoute.
//
// ristypes.NoOrigin means no known origin, not an AS, so it is ErrNoRoute
// whatever is stored.
func (d *Database) GetASDetailsContext(
ctx context.Context, asn int,
) (*ASN, []LiveRoute, error) {
if asn == ristypes.NoOrigin {
return nil, nil, fmt.Errorf("%w: AS%d", ErrNoRoute, asn)
}
// Get AS information
var (
asnInfo ASN
@@ -1790,28 +1722,17 @@ func (d *Database) GetASDetailsContext(
asn,
).Scan(&asnInfo.ASN, &handle, &description, &asnInfo.FirstSeen,
&asnInfo.LastSeen)
// known is whether the asns table or the AS info lookup holds the AS.
known := true
switch {
case errors.Is(err, sql.ErrNoRows):
asnInfo.ASN = asn
info, ok := asinfo.Get(asn)
if ok {
asnInfo.Handle = info.Handle
asnInfo.Description = info.Description
if err != nil {
if errors.Is(err, sql.ErrNoRows) {
return nil, nil, fmt.Errorf("%w: AS%d", ErrNoRoute, asn)
}
known = ok
case err != nil:
return nil, nil, fmt.Errorf("failed to query AS: %w", err)
default:
asnInfo.Handle = handle.String
asnInfo.Description = description.String
}
asnInfo.Handle = handle.String
asnInfo.Description = description.String
allPrefixes, err := d.originPrefixes(ctx, asn, ipVersionV4)
if err != nil {
return &asnInfo, nil, err
@@ -1822,13 +1743,7 @@ func (d *Database) GetASDetailsContext(
return &asnInfo, allPrefixes, err
}
allPrefixes = append(allPrefixes, prefixesV6...)
if !known && len(allPrefixes) == 0 {
return nil, nil, fmt.Errorf("%w: AS%d", ErrNoRoute, asn)
}
return &asnInfo, allPrefixes, nil
return &asnInfo, append(allPrefixes, prefixesV6...), nil
}
// ASPeer represents a peering relationship with another AS including handle,
internal/database/database_internal_test.go
-405
View File
@@ -1,10 +1,8 @@
package database
import (
"bytes"
"context"
"database/sql"
"encoding/json"
"errors"
"fmt"
"log/slog"
@@ -20,8 +18,6 @@ import (
"git.eeqj.de/sneak/routewatch/internal/config"
"git.eeqj.de/sneak/routewatch/internal/logger"
"git.eeqj.de/sneak/routewatch/internal/ristypes"
"git.eeqj.de/sneak/routewatch/pkg/asinfo"
"github.com/mattn/go-sqlite3"
)
@@ -1056,176 +1052,6 @@ func walFileSize(t *testing.T, db *Database) int64 {
return wal.Size()
}
// TestStartupCheckpointLogsItsFigures opens a new database, writes a route,
// and opens the same file again while the first stays open, so that its WAL
// file stays. The first open finds no WAL file: it logs a size of 0 and no
// warning. The second logs the size of the WAL file the first left, and the
// checkpoint's busy flag and frame counts, each under its own name; the
// counts read 0 once the checkpoint has emptied the file.
// Initialize logged the busy flag as wal_pages and the counts under each
// other's names (https://git.eeqj.de/sneak/routewatch/issues/117).
func TestStartupCheckpointLogsItsFigures(t *testing.T) {
t.Parallel()
cfg := &config.Config{StateDir: t.TempDir()}
db, lines := openLogged(t, cfg)
for _, line := range lines {
if line[slog.LevelKey] == slog.LevelWarn.String() {
t.Errorf("opening a new database logged a warning: %v", line)
}
}
line := findLine(t, lines, startupCheckpointMessage)
if line["wal_bytes"] != 0.0 {
t.Errorf("new database: start-up checkpoint logged wal_bytes = %v, "+
"want 0", line["wal_bytes"])
}
err := db.UpsertLiveRouteBatch([]*LiveRoute{
mkV4Route(t, "192.0.2.0/24", lockTestASN, time.Now().UTC()),
})
if err != nil {
t.Fatalf("failed to store a route: %v", err)
}
walBytes := walFileSize(t, db)
if walBytes == 0 {
t.Fatal("WAL file empty after the route write, want frames in it")
}
_, lines = openLogged(t, cfg)
line = findLine(t, lines, startupCheckpointMessage)
for _, key := range []string{"busy", "wal_frames", "checkpointed_frames"} {
if line[key] != 0.0 {
t.Errorf("start-up checkpoint logged %s = %v, want 0", key, line[key])
}
}
if line["wal_bytes"] != float64(walBytes) {
t.Errorf("start-up checkpoint logged wal_bytes = %v, want the %d "+
"bytes the first database left", line["wal_bytes"], walBytes)
}
}
// TestStartupCheckpointWarnsWhenIncomplete keeps a read open on a database
// while a route is written after the read began, then opens the same file
// again. The second open's start-up checkpoint cannot copy the frames written
// after the read began, and gives up once the busy timeout has passed. It
// checks that the checkpoint is logged as a warning with busy 1 and fewer
// frames copied than the WAL holds.
func TestStartupCheckpointWarnsWhenIncomplete(t *testing.T) {
t.Parallel()
cfg := &config.Config{StateDir: t.TempDir()}
db, _ := openLogged(t, cfg)
ctx := context.Background()
reader, err := db.db.Conn(ctx)
if err != nil {
t.Fatalf("failed to open the reading connection: %v", err)
}
defer func() { _ = reader.Close() }()
// A plain BEGIN starts a transaction that only reads, unlike the BEGIN
// IMMEDIATE of _txlock. It keeps the snapshot its first read takes until
// it ends.
_, err = reader.ExecContext(ctx, "BEGIN")
if err != nil {
t.Fatalf("failed to begin the read transaction: %v", err)
}
defer func() { _, _ = reader.ExecContext(ctx, "ROLLBACK") }()
var routes int
err = reader.QueryRowContext(ctx,
"SELECT count(*) FROM live_routes_v4").Scan(&routes)
if err != nil {
t.Fatalf("failed to read the live routes: %v", err)
}
err = db.UpsertLiveRouteBatch([]*LiveRoute{
mkV4Route(t, "192.0.2.0/24", lockTestASN, time.Now().UTC()),
})
if err != nil {
t.Fatalf("failed to store a route after the read: %v", err)
}
_, lines := openLogged(t, cfg)
line := findLine(t, lines, startupCheckpointIncompleteMessage)
if line[slog.LevelKey] != slog.LevelWarn.String() {
t.Errorf("start-up checkpoint logged at level %v, want %s",
line[slog.LevelKey], slog.LevelWarn)
}
if line["busy"] != 1.0 {
t.Errorf("start-up checkpoint logged busy = %v, want 1", line["busy"])
}
walFrames, _ := line["wal_frames"].(float64)
checkpointedFrames, _ := line["checkpointed_frames"].(float64)
if checkpointedFrames >= walFrames {
t.Errorf("start-up checkpoint logged checkpointed_frames = %v and "+
"wal_frames = %v with a read open, want fewer copied than the WAL "+
"holds", line["checkpointed_frames"], line["wal_frames"])
}
}
// openLogged opens the database of cfg with a logger that writes JSON, and
// returns it with the lines it logged while it opened, decoded.
func openLogged(t *testing.T, cfg *config.Config) (*Database, []map[string]any) {
t.Helper()
var logged bytes.Buffer
db, err := New(cfg,
&logger.Logger{Logger: slog.New(slog.NewJSONHandler(&logged, nil))})
if err != nil {
t.Fatalf("failed to open the database: %v", err)
}
t.Cleanup(func() { _ = db.Close() })
var lines []map[string]any
decoder := json.NewDecoder(&logged)
for decoder.More() {
var line map[string]any
err := decoder.Decode(&line)
if err != nil {
t.Fatalf("failed to decode the log: %v", err)
}
lines = append(lines, line)
}
return db, lines
}
// findLine returns the first of lines logged with message.
func findLine(
t *testing.T,
lines []map[string]any,
message string,
) map[string]any {
t.Helper()
for _, line := range lines {
if line[slog.MessageKey] == message {
return line
}
}
t.Fatalf("no line %q in the log", message)
return nil
}
// TestGetRandomPrefixesByLengthListsPrefixOnce stores one prefix of each
// family learned from two peers in the same second and checks that the
// prefix-length sample lists each of them once.
@@ -1701,85 +1527,6 @@ func TestGetASDetailsListsEachPrefixOnce(t *testing.T) {
}
}
// TestGetASDetailsOfASNotInDatabase asks for ASes that the database has no
// record of, some with stored routes. Each gets the prefixes of its routes and
// no seen times. One that the AS info lookup knows gets its handle and
// description from it. An AS with no record, no lookup entry and no route is
// ErrNoRoute. Before the fix every AS with no record was ErrNoRoute.
func TestGetASDetailsOfASNotInDatabase(t *testing.T) {
t.Parallel()
db, err := New(&config.Config{StateDir: t.TempDir()}, logger.New())
if err != nil {
t.Fatalf("failed to create database: %v", err)
}
defer func() { _ = db.Close() }()
ts := time.Date(2026, 1, 2, 3, 4, 5, 0, time.UTC)
tests := []struct {
asn int
listed bool // in the AS info lookup
prefixes []string // of the routes stored for the AS
}{
{15169, true, []string{"192.0.2.0/24"}},
{13335, true, nil},
{4200000000, false, []string{"198.51.100.0/24"}},
}
var routes []*LiveRoute
for _, tt := range tests {
if _, ok := asinfo.Get(tt.asn); ok != tt.listed {
t.Fatalf("AS%d in the AS info data: %t, want %t", tt.asn, ok, tt.listed)
}
for _, prefix := range tt.prefixes {
routes = append(routes, mkV4Route(t, prefix, tt.asn, ts))
}
}
err = db.UpsertLiveRouteBatch(routes)
if err != nil {
t.Fatalf("UpsertLiveRouteBatch: %v", err)
}
for _, tt := range tests {
got, prefixes, err := db.GetASDetails(tt.asn)
if err != nil {
t.Errorf("GetASDetails(%d): %v", tt.asn, err)
continue
}
want := ASN{ASN: tt.asn}
if info, ok := asinfo.Get(tt.asn); ok {
want.Handle, want.Description = info.Handle, info.Description
}
var gotPrefixes []string
for _, route := range prefixes {
gotPrefixes = append(gotPrefixes, route.Prefix)
}
if *got != want || !slices.Equal(gotPrefixes, tt.prefixes) {
t.Errorf("GetASDetails(%d) = %+v with prefixes %v, want %+v with %v",
tt.asn, *got, gotPrefixes, want, tt.prefixes)
}
}
const absentASN = 4200000001 // no record, no lookup entry and no route
if _, ok := asinfo.Get(absentASN); ok {
t.Fatalf("AS%d is in the AS info data", absentASN)
}
_, _, err = db.GetASDetails(absentASN)
if !errors.Is(err, ErrNoRoute) {
t.Errorf("GetASDetails(%d): error %v, want %v", absentASN, err, ErrNoRoute)
}
}
// TestASPrefixQueriesUseCoveringIndexes checks the plan of each query that
// lists the prefixes an AS originates: a search of a covering index and no
// temporary B-tree, so the list reads one index entry per prefix, not one
@@ -1961,158 +1708,6 @@ func TestUpdatePeerBatchStoresTheEarliestAndLatestTimesOfANewPeer(
}
}
// backlogAge is how much older one write is than the other in the tests of
// first_seen and last_seen, as when a collector catches up on a backlog.
const backlogAge = 24 * time.Hour
// batchSpan is how much earlier a batch's First is than its Last in the tests
// of first_seen and last_seen.
const batchSpan = time.Hour
// TestWritesKeepEarliestFirstSeenAndLatestLastSeen writes a row of each of
// asns, peerings and bgp_peers through each function that writes one, at a
// newer time and at an older one. Each row takes the older write's First as
// first_seen and the newer write's Last as last_seen. The test writes in both
// orders, each in a database of its own. Newer then older is a collector
// catching up on a backlog; older then newer is a feed that arrives in order.
// Each function writes a row of its own. A batch write's First is batchSpan
// before its Last, so the test catches the two swapped. A single-row write
// takes one time.
func TestWritesKeepEarliestFirstSeenAndLatestLastSeen(t *testing.T) {
t.Parallel()
const (
batchASN, singleASN = 64500, 64501
peerASN = 64502
asnQuery = "SELECT first_seen, last_seen FROM asns WHERE asn = ?"
peeringQuery = "SELECT first_seen, last_seen FROM peerings " +
"WHERE as_a = ? AND as_b = ?"
peerQuery = "SELECT first_seen, last_seen FROM bgp_peers " +
"WHERE peer_ip = ?"
)
for name, olderFirst := range map[string]bool{
"newer then older": false,
"older then newer": true,
} {
t.Run(name, func(t *testing.T) {
t.Parallel()
db, err := New(&config.Config{StateDir: t.TempDir()}, logger.New())
if err != nil {
t.Fatalf("failed to create database: %v", err)
}
defer func() { _ = db.Close() }()
checkSeenTimes(t, db, olderFirst, func(seen SeenTimes) error {
return db.GetOrCreateASNBatch(map[int]SeenTimes{batchASN: seen})
}, batchSpan, asnQuery, batchASN)
checkSeenTimes(t, db, olderFirst, func(seen SeenTimes) error {
_, err := db.GetOrCreateASN(singleASN, seen.Last)
return err
}, 0, asnQuery, singleASN)
checkSeenTimes(t, db, olderFirst, func(seen SeenTimes) error {
return db.RecordPeering(batchASN, singleASN, seen)
}, batchSpan, peeringQuery, batchASN, singleASN)
checkSeenTimes(t, db, olderFirst, func(seen SeenTimes) error {
return db.UpdatePeerBatch(map[string]PeerUpdate{
testV4PeerIP: {
PeerIP: testV4PeerIP, PeerASN: peerASN,
MessageType: ristypes.MessageTypeUpdate,
Seen: seen,
},
})
}, batchSpan, peerQuery, testV4PeerIP)
checkSeenTimes(t, db, olderFirst, func(seen SeenTimes) error {
return db.UpdatePeer(testPeerIP, peerASN,
ristypes.MessageTypeUpdate, seen.Last)
}, 0, peerQuery, testPeerIP)
})
}
}
// checkSeenTimes calls write twice, each time with First span before Last: at
// newer times and at times backlogAge older. It writes the older times first
// when olderFirst is true. It checks that query, given args, reads the older
// write's First as first_seen and the newer write's Last as last_seen.
func checkSeenTimes(
t *testing.T, db *Database, olderFirst bool,
write func(seen SeenTimes) error,
span time.Duration, query string, args ...any,
) {
t.Helper()
newerLast := time.Date(2026, 1, 2, 3, 4, 5, 0, time.UTC)
newer := SeenTimes{First: newerLast.Add(-span), Last: newerLast}
older := SeenTimes{
First: newer.First.Add(-backlogAge),
Last: newer.Last.Add(-backlogAge),
}
writes := []SeenTimes{newer, older}
if olderFirst {
writes = []SeenTimes{older, newer}
}
for _, seen := range writes {
err := write(seen)
if err != nil {
t.Fatalf("writing first %s, last %s: %v", seen.First, seen.Last, err)
}
}
var firstSeen, lastSeen time.Time
err := db.db.QueryRowContext(t.Context(), query, args...).
Scan(&firstSeen, &lastSeen)
if err != nil {
t.Fatalf("reading the row: %v", err)
}
if !firstSeen.Equal(older.First) || !lastSeen.Equal(newer.Last) {
t.Errorf("first seen %s, last seen %s; want %s, %s",
firstSeen, lastSeen, older.First, newer.Last)
}
}
// TestGetOrCreateASNReturnsStoredSeenTimes stores an AS at a newer time, then
// at an older one. The second call returns the times the row now holds: the
// older first_seen and the newer last_seen.
func TestGetOrCreateASNReturnsStoredSeenTimes(t *testing.T) {
t.Parallel()
db, err := New(&config.Config{StateDir: t.TempDir()}, logger.New())
if err != nil {
t.Fatalf("failed to create database: %v", err)
}
defer func() { _ = db.Close() }()
const asn = 64500
newer := time.Date(2026, 1, 2, 3, 4, 5, 0, time.UTC)
older := newer.Add(-backlogAge)
_, err = db.GetOrCreateASN(asn, newer)
if err != nil {
t.Fatalf("GetOrCreateASN at %s: %v", newer, err)
}
got, err := db.GetOrCreateASN(asn, older)
if err != nil {
t.Fatalf("GetOrCreateASN at %s: %v", older, err)
}
if !got.FirstSeen.Equal(older) || !got.LastSeen.Equal(newer) {
t.Errorf("first seen %s, last seen %s; want %s, %s",
got.FirstSeen, got.LastSeen, older, newer)
}
}
// TestUpsertLiveRouteBatchWritesInKeyOrder writes new routes of two peers out
// of key order, the last one a second announcement of the first, and checks
// that the rows were inserted in key order, peer then prefix (each new row's id
internal/database/dockerignore_internal_test.go
-61
View File
@@ -1,61 +0,0 @@
package database
import (
"os"
"path/filepath"
"strings"
"testing"
"git.eeqj.de/sneak/routewatch/internal/config"
"git.eeqj.de/sneak/routewatch/internal/logger"
)
// TestDockerignoreExcludesDatabaseFiles checks that .dockerignore keeps the
// database file, and the WAL and shared-memory files SQLite writes next to it,
// out of the build context wherever they sit in the clone. The image ships the
// whole build context as a source archive.
func TestDockerignoreExcludesDatabaseFiles(t *testing.T) {
t.Parallel()
db, err := New(&config.Config{StateDir: t.TempDir()}, logger.New())
if err != nil {
t.Fatalf("failed to create database: %v", err)
}
defer func() { _ = db.Close() }()
data, err := os.ReadFile("../../.dockerignore")
if err != nil {
t.Fatalf("failed to read .dockerignore: %v", err)
}
// Docker reads `**/` as any number of directories and the rest of the
// pattern as filepath.Match does, so `**/` followed by a pattern that
// matches a file's name excludes that file at any depth.
var anyDepth []string
for line := range strings.Lines(string(data)) {
pattern, ok := strings.CutPrefix(strings.TrimSpace(line), "**/")
if ok {
anyDepth = append(anyDepth, pattern)
}
}
name := filepath.Base(db.path)
for _, file := range []string{name, name + "-wal", name + "-shm"} {
excluded := false
for _, pattern := range anyDepth {
matched, err := filepath.Match(pattern, file)
if err != nil {
t.Fatalf("bad .dockerignore pattern **/%s: %v", pattern, err)
}
excluded = excluded || matched
}
if !excluded {
t.Errorf(".dockerignore has no **/ pattern that excludes %s", file)
}
}
}
internal/database/models.go
+11 -11
View File
@@ -28,10 +28,9 @@ type ASN struct {
RIRRegDate *time.Time `json:"rir_registration_date,omitempty"`
RIRLastMod *time.Time `json:"rir_last_modified,omitempty"`
WHOISRaw string `json:"whois_raw,omitempty"`
// Timestamps. They are zero, and left out of JSON, for an AS that has no
// record in the asns table.
FirstSeen time.Time `json:"first_seen,omitzero"`
LastSeen time.Time `json:"last_seen,omitzero"`
// Timestamps
FirstSeen time.Time `json:"first_seen"`
LastSeen time.Time `json:"last_seen"`
WHOISUpdatedAt *time.Time `json:"whois_updated_at,omitempty"`
}
@@ -92,11 +91,12 @@ type PrefixDistribution struct {
//
//nolint:tagliatelle // field names are the existing JSON API
type IPInfo struct {
IP string `json:"ip"`
Netblock string `json:"netblock"`
MaskLength int `json:"mask_length"`
IPVersion int `json:"ip_version"`
NumPeers int `json:"num_peers"`
IP string `json:"ip"`
PTR []string `json:"ptr,omitempty"`
Netblock string `json:"netblock"`
MaskLength int `json:"mask_length"`
IPVersion int `json:"ip_version"`
NumPeers int `json:"num_peers"`
// AS information
ASN int `json:"asn"`
ASName string `json:"as_name,omitempty"`
@@ -124,8 +124,8 @@ type LiveRouteDeletion struct {
// SeenTimes is the earliest and the latest time one batch of updates saw an AS,
// a peer or a peering. A row the batch adds is stored first seen at First and
// last seen at Last. A row already stored takes First as its first_seen only
// when First is earlier, and Last as its last_seen only when Last is later.
// last seen at Last; a row already stored keeps its first_seen and has its
// last_seen set to Last.
type SeenTimes struct {
First time.Time
Last time.Time
internal/metrics/metrics.go
+7 -8
View File
@@ -34,7 +34,7 @@ type Tracker struct {
// Announcement/withdrawal metrics
announcementCounter metrics.Counter
withdrawalCounter metrics.Counter
churnPerMinute minuteCounter // prefix announcements and withdrawals
churnRate metrics.Meter // combined announcements + withdrawals per second
// BGP peer tracking
bgpPeerCount atomic.Int32
@@ -56,7 +56,7 @@ func New() *Tracker {
ipv6UpdateRate: metrics.NewMeter(),
announcementCounter: metrics.NewCounter(),
withdrawalCounter: metrics.NewCounter(),
churnPerMinute: minuteCounter{now: time.Now},
churnRate: metrics.NewMeter(),
}
}
@@ -156,13 +156,13 @@ func (t *Tracker) RecordIPv6Update() {
// RecordAnnouncement records a route announcement
func (t *Tracker) RecordAnnouncement() {
t.announcementCounter.Inc(1)
t.churnPerMinute.add()
t.churnRate.Mark(1)
}
// RecordWithdrawal records a route withdrawal
func (t *Tracker) RecordWithdrawal() {
t.withdrawalCounter.Inc(1)
t.churnPerMinute.add()
t.churnRate.Mark(1)
}
// SetBGPPeerCount updates the current BGP peer count
@@ -198,10 +198,9 @@ func (t *Tracker) GetWithdrawalCount() uint64 {
return uint64(count)
}
// GetChurnPerMinute returns the number of prefix announcements and
// withdrawals, from all peers, in the last full clock minute.
func (t *Tracker) GetChurnPerMinute() int {
return t.churnPerMinute.lastFullMinute()
// GetChurnRate returns the route churn rate per second
func (t *Tracker) GetChurnRate() float64 {
return t.churnRate.Rate1()
}
// GetRouteMetrics returns current route update metrics
internal/metrics/minutecounter.go
-59
View File
@@ -1,59 +0,0 @@
package metrics
import (
"sync"
"time"
)
// minuteCounter counts events in each clock minute and reports the count of
// the last full minute. It is safe for concurrent use.
type minuteCounter struct {
mu sync.Mutex
now func() time.Time
// minute is the start of the minute that count belongs to.
minute time.Time
count int
// lastMinuteCount is the count of the minute just before minute.
lastMinuteCount int
}
// add counts one event in the current minute.
func (c *minuteCounter) add() {
c.mu.Lock()
defer c.mu.Unlock()
c.moveToCurrentMinute()
c.count++
}
// lastFullMinute returns the number of events in the minute that ended most
// recently.
func (c *minuteCounter) lastFullMinute() int {
c.mu.Lock()
defer c.mu.Unlock()
c.moveToCurrentMinute()
return c.lastMinuteCount
}
// moveToCurrentMinute starts a new count when the clock has moved into
// another minute. The count it ends becomes the last full minute's count when
// its minute is the one just before the current minute. Otherwise the last
// full minute had no events. The caller holds mu.
func (c *minuteCounter) moveToCurrentMinute() {
minute := c.now().Truncate(time.Minute)
if minute.Equal(c.minute) {
return
}
if minute.Sub(c.minute) == time.Minute {
c.lastMinuteCount = c.count
} else {
c.lastMinuteCount = 0
}
c.minute = minute
c.count = 0
}
internal/metrics/minutecounter_internal_test.go
-86
View File
@@ -1,86 +0,0 @@
package metrics
import (
"sync"
"testing"
"time"
)
// TestChurnPerMinuteCountsLastFullMinute announces and withdraws prefixes on
// an injected clock and expects each minute's whole count once that minute
// has ended, and 0 for a minute with none.
func TestChurnPerMinuteCountsLastFullMinute(t *testing.T) {
t.Parallel()
now := time.Date(2026, 10, 26, 12, 0, 30, 0, time.UTC)
tracker := New()
tracker.churnPerMinute.now = func() time.Time { return now }
expectChurn := func(want int, why string) {
t.Helper()
got := tracker.GetChurnPerMinute()
if got != want {
t.Fatalf("at %s churn = %d, want %d: %s",
now.Format(time.TimeOnly), got, want, why)
}
}
const firstMinuteChurn = 2
tracker.RecordAnnouncement()
tracker.RecordWithdrawal()
expectChurn(0, "12:00 has not ended")
now = time.Date(2026, 10, 26, 12, 1, 59, 0, time.UTC)
tracker.RecordAnnouncement()
expectChurn(firstMinuteChurn, "12:00 had one announcement and one withdrawal")
now = time.Date(2026, 10, 26, 12, 2, 0, 0, time.UTC)
expectChurn(1, "12:01 had one announcement")
now = time.Date(2026, 10, 26, 12, 4, 0, 0, time.UTC)
expectChurn(0, "12:03 had none")
tracker.RecordWithdrawal()
now = time.Date(2026, 10, 26, 12, 6, 0, 0, time.UTC)
expectChurn(0, "12:05 had none, though 12:04 had one withdrawal")
}
// TestChurnPerMinuteCountsConcurrentRecords announces prefixes from several
// goroutines at once and expects every one counted.
func TestChurnPerMinuteCountsConcurrentRecords(t *testing.T) {
t.Parallel()
const (
goroutines = 8
recordsPerGoroutine = 1000
)
now := time.Date(2026, 10, 26, 12, 0, 0, 0, time.UTC)
tracker := New()
tracker.churnPerMinute.now = func() time.Time { return now }
var wg sync.WaitGroup
for range goroutines {
wg.Go(func() {
for range recordsPerGoroutine {
tracker.RecordAnnouncement()
}
})
}
wg.Wait()
now = now.Add(time.Minute)
want := goroutines * recordsPerGoroutine
if got := tracker.GetChurnPerMinute(); got != want {
t.Fatalf("churn = %d, want %d", got, want)
}
}
internal/ristypes/ris.go
+11 -33
View File
@@ -4,7 +4,6 @@ package ristypes
import (
"encoding/json"
"slices"
"time"
)
@@ -25,8 +24,7 @@ const (
)
// ASPath is a BGP AS path as the feed writes it: an array of AS numbers in
// which an AS set is an array of its members, such as [1, 2, [3, 4]]. A path
// that holds AS 0 decodes as an empty one (UnmarshalJSON).
// which an AS set is an array of its members, such as [1, 2, [3, 4]].
type ASPath struct {
// Sequence is the path's AS numbers in order. A set of one member that
// ends the path counts as that AS and is its last element. The members of
@@ -39,56 +37,34 @@ type ASPath struct {
Set []int
}
// NoOrigin is the origin AS of a route whose origin is not known. AS 0 is
// reserved, so no AS has this number.
const NoOrigin = 0
// Origin returns the AS that originated the route: the last AS of the
// sequence, or NoOrigin when the path ends in a set of several members or is
// sequence, or 0 (none) when the path ends in a set of several members or is
// empty.
func (p *ASPath) Origin() int {
if len(p.Set) > 0 || len(p.Sequence) == 0 {
return NoOrigin
return 0
}
return p.Sequence[len(p.Sequence)-1]
}
// UnmarshalJSON implements the json.Unmarshaler interface for ASPath. A path
// that holds AS 0 anywhere, a member of an AS set included, decodes as an empty
// path: RFC 7607 makes such a path malformed and its announcements
// withdrawals. On an announcement with an empty path the prefix handler
// deletes the peer's route, and the AS and peering handlers record nothing.
// UnmarshalJSON implements the json.Unmarshaler interface for ASPath.
func (p *ASPath) UnmarshalJSON(data []byte) error {
path, err := decodeASPath(data)
if err != nil {
return err
}
if slices.Contains(path.Sequence, 0) || slices.Contains(path.Set, 0) {
path = ASPath{}
}
*p = path
return nil
}
// decodeASPath decodes an AS path as the feed writes it.
func decodeASPath(data []byte) (ASPath, error) {
// Most paths have no AS set and decode as a plain array of integers.
var sequence []int
err := json.Unmarshal(data, &sequence)
if err == nil {
return ASPath{Sequence: sequence}, nil
*p = ASPath{Sequence: sequence}
return nil
}
var raw []any
err = json.Unmarshal(data, &raw)
if err != nil {
return ASPath{}, err
return err
}
path := ASPath{Sequence: make([]int, 0, len(raw))}
@@ -114,7 +90,9 @@ func decodeASPath(data []byte) (ASPath, error) {
}
}
return path, nil
*p = path
return nil
}
// RISLiveMessage represents the outer wrapper message from the RIPE RIS Live stream.
internal/ristypes/ris_test.go
-30
View File
@@ -145,33 +145,3 @@ func TestASPathKeepsATrailingSetApart(t *testing.T) {
})
}
}
// TestASPathWithASZeroDecodesAsAnEmptyPath decodes AS paths that hold AS 0
// before their end, at their end, or in an AS set. RFC 7607 makes such a path
// malformed, so each decodes as an empty path.
func TestASPathWithASZeroDecodesAsAnEmptyPath(t *testing.T) {
t.Parallel()
for _, path := range []string{
"[64496, 0, 64497]",
"[64496, 0]",
"[64496, [0, 64497], 64498]",
"[64496, [64497, 0]]",
} {
t.Run(path, func(t *testing.T) {
t.Parallel()
var got ristypes.ASPath
err := json.Unmarshal([]byte(path), &got)
if err != nil {
t.Fatalf("unmarshal %s: %v", path, err)
}
if len(got.Sequence) > 0 || len(got.Set) > 0 {
t.Errorf("%s decoded to sequence %v and set %v, want an empty path",
path, got.Sequence, got.Set)
}
})
}
}
internal/routetable/benchmark_test.go
-154
View File
@@ -1,154 +0,0 @@
package routetable_test
import (
"fmt"
"math/rand/v2"
"net/netip"
"runtime"
"testing"
"time"
"git.eeqj.de/sneak/routewatch/internal/routetable"
)
// Parameters for BenchmarkApplyMixedBatch.
const (
// benchPeers is how many peers there are, about as many as RIS has.
benchPeers = 1400
// benchPrefixes is how many IPv4 /24 prefixes the table holds, and
// benchPeersPerPrefix how many peers, picked at random, carry each.
benchPrefixes = 100000
benchPeersPerPrefix = 20
// benchBatchSize is how many updates the timed batch holds, as many as
// the prefix handler's largest batch (prefixBatchMaxSize in
// internal/routewatch).
benchBatchSize = 250000
// In the timed batch every benchWithdrawalEvery-th update is a
// withdrawal, and every benchNewOriginEvery-th announcement has an
// origin AS other than the prefix's.
benchWithdrawalEvery = 5
benchNewOriginEvery = 7
// benchOrigins is how many origin ASes the prefixes are spread over,
// from benchFirstASN on: 20 prefixes each, about as many as an origin AS
// has on average in the full table.
benchOrigins = 5000
benchFirstASN = 64500
// benchUpdatesPerSecond is how many updates share one time.
benchUpdatesPerSecond = 1000
benchMaskLength = 24
)
// benchUpdate is one update of the benchmark.
type benchUpdate struct {
prefix netip.Prefix
origin int
route routetable.Route
withdraw bool
}
// BenchmarkApplyMixedBatch times one batch of 250,000 updates of routes picked
// at random: announcements, withdrawals and announcements with a new origin AS.
// The table it applies them to holds 100,000 IPv4 prefixes from 5,000 origin
// ASes, each carried by 20 of 1,400 peers. It reports the heap bytes that table
// takes per prefix and origin AS pair. It is run by hand, not by make test, for
// example in the test image:
//
// go test -run '^$' -bench MixedBatch -benchtime 10x ./internal/routetable/
func BenchmarkApplyMixedBatch(b *testing.B) {
//nolint:gosec // The benchmark needs the same updates every run.
rng := rand.New(rand.NewPCG(1, 2))
peers := make([]string, benchPeers)
for i := range peers {
peers[i] = fmt.Sprintf("198.18.%d.%d", i/256, i%256)
}
before := heapBytes()
table := routetable.New(routetable.Params{})
updates := 0
for prefix := range benchPrefixes {
for range benchPeersPerPrefix {
peer := peers[rng.IntN(benchPeers)]
apply(table, benchAnnouncement(rng, prefix, peer, updates, false))
updates++
}
}
// The record of changes is handed over every second while routewatch
// runs, so it is not part of the table's size.
table.TakeChanges()
bytesPerPair := float64(heapBytes()-before) / benchPrefixes
batch := make([]benchUpdate, 0, benchBatchSize)
for i := range benchBatchSize {
update := benchAnnouncement(rng, rng.IntN(benchPrefixes),
peers[rng.IntN(benchPeers)], updates+i, i%benchNewOriginEvery == 0)
update.withdraw = i%benchWithdrawalEvery == 0
batch = append(batch, update)
}
for b.Loop() {
for _, update := range batch {
apply(table, update)
}
}
// After the loop: b.Loop drops the metrics reported before it starts.
b.ReportMetric(bytesPerPair, "bytes/pair")
}
// benchAnnouncement returns the announcement of the prefix numbered prefix, a
// /24 from 1.0.0.0/24 on, by peer. It is the updates-th update of the
// benchmark, and its AS path holds four ASes from the peer's AS to the origin
// AS. newOrigin gives it an origin AS other than the prefix's.
func benchAnnouncement(
rng *rand.Rand, prefix int, peer string, updates int, newOrigin bool,
) benchUpdate {
origin := benchFirstASN + prefix%benchOrigins
if newOrigin {
origin += benchOrigins
}
start := time.Date(2026, 1, 2, 3, 4, 5, 0, time.UTC)
return benchUpdate{
prefix: netip.MustParsePrefix(fmt.Sprintf("%d.%d.%d.0/%d",
1+prefix>>16, (prefix>>8)%256, prefix%256, benchMaskLength)),
origin: origin,
route: routetable.Route{
Peer: peer,
ASPath: []int{
benchFirstASN + rng.IntN(benchOrigins),
benchFirstASN + rng.IntN(benchOrigins),
benchFirstASN + rng.IntN(benchOrigins),
origin,
},
Time: start.Add(time.Duration(updates/benchUpdatesPerSecond) *
time.Second),
},
}
}
// apply applies update to table.
func apply(table *routetable.Table, update benchUpdate) {
if update.withdraw {
table.Withdraw(update.prefix, update.route.Peer)
return
}
table.Announce(update.prefix, update.origin, update.route)
}
// heapBytes returns the bytes of the live heap objects, after a collection.
func heapBytes() uint64 {
runtime.GC()
var stats runtime.MemStats
runtime.ReadMemStats(&stats)
return stats.HeapAlloc
}
internal/routetable/reads.go
-223
View File
@@ -1,223 +0,0 @@
package routetable
import (
"net/netip"
"slices"
"time"
"github.com/bits-and-blooms/bitset"
)
// PrefixSummary is what the table answers about one prefix.
type PrefixSummary struct {
Prefix netip.Prefix
// OriginASN is the origin AS that the most peers carry the prefix with.
// When several have as many peers, it is the lowest AS number.
OriginASN int
// Peers is how many peers carry the prefix.
Peers int
// Newest is the time of the newest kept route of the prefix, or nil when
// it keeps none.
Newest *time.Time
}
// OriginatedPrefix is one prefix an AS originates.
type OriginatedPrefix struct {
Prefix netip.Prefix
// Newest is the time of the newest kept route of the prefix with that
// origin AS, or nil when it keeps none.
Newest *time.Time
}
// Counts are the table's figures for the stats.
type Counts struct {
IPv4Prefixes int
IPv6Prefixes int
// IPv4PrefixesByLength[n] and IPv6PrefixesByLength[n] are how many
// prefixes of mask length n some peer carries.
IPv4PrefixesByLength []int
IPv6PrefixesByLength []int
// IPv4Routes and IPv6Routes count peer and prefix pairs: each peer that
// carries a prefix counts once for it.
IPv4Routes int
IPv6Routes int
// OriginsWithoutKeptRoute counts the prefix and origin AS pairs that peers
// carry but whose kept routes were all withdrawn.
OriginsWithoutKeptRoute int
// OldestKeptRoute and NewestKeptRoute are the times of the oldest and the
// newest kept routes, or nil when the table keeps none.
OldestKeptRoute *time.Time
NewestKeptRoute *time.Time
}
// Lookup returns the most specific prefix that holds addr and that some peer
// carries. It ignores prefixes shorter than /8 for IPv4 and /16 for IPv6. It
// makes one map lookup per mask length, and reports false when no prefix
// holds addr.
func (t *Table) Lookup(addr netip.Addr) (PrefixSummary, bool) {
t.mu.RLock()
defer t.mu.RUnlock()
// An IPv4 address written in IPv6 form (::ffff:192.0.2.1) is looked up as
// IPv4.
addr = addr.Unmap()
f := t.familyOf(addr)
for bits := addr.BitLen(); bits >= f.minLookupLength; bits-- {
prefix := netip.PrefixFrom(addr, bits).Masked()
origins, found := f.byLength[bits][prefix]
if found {
return summarize(prefix, origins), true
}
}
return PrefixSummary{}, false
}
// OriginatedPrefixes returns the prefixes AS asn originates, in prefix order.
func (t *Table) OriginatedPrefixes(asn int) []OriginatedPrefix {
t.mu.RLock()
defer t.mu.RUnlock()
prefixes := make([]OriginatedPrefix, 0, len(t.byOrigin[asn]))
for prefix, o := range t.byOrigin[asn] {
prefixes = append(prefixes,
OriginatedPrefix{Prefix: prefix, Newest: o.newestTime()})
}
slices.SortFunc(prefixes, func(a, b OriginatedPrefix) int {
return a.Prefix.Compare(b.Prefix)
})
return prefixes
}
// Origins returns each origin AS of prefix, in AS number order, with its peers
// and its kept routes. It returns nil when no peer carries the prefix.
func (t *Table) Origins(prefix netip.Prefix) []Origin {
t.mu.RLock()
defer t.mu.RUnlock()
if !prefix.IsValid() {
return nil
}
var origins []Origin
for _, o := range t.origins(prefix.Masked()) {
origins = append(origins, Origin{
ASN: o.asn,
Peers: t.addresses(&o.peers),
Routes: t.exportRoutes(o.routes),
})
}
return origins
}
// SamplePrefixes returns up to limit of the prefixes of IP version ipVersion
// (4 or 6) and mask length maskLength. It reads only the prefixes it returns.
// Go starts each walk over a map at a random place, so each call returns
// another sample.
func (t *Table) SamplePrefixes(ipVersion, maskLength, limit int) []PrefixSummary {
t.mu.RLock()
defer t.mu.RUnlock()
var f *family
switch ipVersion {
case ipVersion4:
f = &t.ipv4
case ipVersion6:
f = &t.ipv6
default:
return nil
}
if maskLength < 0 || maskLength >= len(f.byLength) || limit <= 0 {
return nil
}
prefixes := f.byLength[maskLength]
sample := make([]PrefixSummary, 0, min(limit, len(prefixes)))
for prefix, origins := range prefixes {
if len(sample) == limit {
break
}
sample = append(sample, summarize(prefix, origins))
}
return sample
}
// Counts returns the table's figures for the stats. The writes keep them up to
// date, so this reads no prefix.
func (t *Table) Counts() Counts {
t.mu.RLock()
defer t.mu.RUnlock()
counts := Counts{
IPv4PrefixesByLength: t.ipv4.prefixesByLength(),
IPv6PrefixesByLength: t.ipv6.prefixesByLength(),
IPv4Routes: t.ipv4.routes,
IPv6Routes: t.ipv6.routes,
OriginsWithoutKeptRoute: t.originsWithoutKeptRoute,
}
for _, n := range counts.IPv4PrefixesByLength {
counts.IPv4Prefixes += n
}
for _, n := range counts.IPv6PrefixesByLength {
counts.IPv6Prefixes += n
}
oldest, found := t.keptTimes.Min()
if found {
counts.OldestKeptRoute = &oldest.time
}
newest, found := t.keptTimes.Max()
if found {
counts.NewestKeptRoute = &newest.time
}
return counts
}
// addresses returns the addresses of the peers in peers, in number order.
func (t *Table) addresses(peers *bitset.BitSet) []string {
addresses := make([]string, 0, peers.Count())
for number := range peers.EachSet() {
addresses = append(addresses, t.peerAddresses[number])
}
return addresses
}
// summarize returns the summary of prefix, whose origins are in AS number
// order.
func summarize(prefix netip.Prefix, origins []*origin) PrefixSummary {
summary := PrefixSummary{Prefix: prefix}
mostPeers := 0
for _, o := range origins {
peers := int(o.peers.Count())
summary.Peers += peers
if peers > mostPeers {
mostPeers = peers
summary.OriginASN = o.asn
}
newest := o.newestTime()
if newest != nil && (summary.Newest == nil || newest.After(*summary.Newest)) {
summary.Newest = newest
}
}
return summary
}
internal/routetable/reads_test.go
-259
View File
@@ -1,259 +0,0 @@
package routetable_test
import (
"net/netip"
"reflect"
"testing"
"git.eeqj.de/sneak/routewatch/internal/routetable"
)
// announceAll has each listed peer announce prefix with origin AS asn, the
// peer at index i i*10 seconds after start.
func announceAll(
table *routetable.Table, prefix string, asn, start int, peers ...string,
) {
const secondsBetweenPeers = 10
for i, peer := range peers {
table.Announce(netip.MustParsePrefix(prefix), asn,
routeOf(peer, asn, start+i*secondsBetweenPeers))
}
}
func TestLookup(t *testing.T) {
t.Parallel()
table := routetable.New(routetable.Params{})
announceAll(table, "10.0.0.0/8", asA, 10, peerA)
// Three peers with two origins: asB has more of them.
announceAll(table, "10.1.0.0/16", asA, 10, peerA)
announceAll(table, "10.1.0.0/16", asB, 20, peerB, peerC)
// Two origins with one peer each: the lower AS number wins.
announceAll(table, prefixV4, asB, 20, peerB)
announceAll(table, prefixV4, asA, 10, peerA)
// An origin whose kept routes were withdrawn: no newest time.
announceAll(table, "10.9.0.0/16", asA, 10, peerA, peerB, peerC)
table.Withdraw(netip.MustParsePrefix("10.9.0.0/16"), peerB)
table.Withdraw(netip.MustParsePrefix("10.9.0.0/16"), peerC)
announceAll(table, "2001:db8::/32", asA, 10, peerA)
announceAll(table, prefixV6, asB, 20, peerB)
announceAll(table, "3000::/16", asA, 10, peerA)
// Shorter than the lookup uses.
announceAll(table, "192.0.0.0/7", asA, 10, peerA)
announceAll(table, "2000::/15", asA, 10, peerA)
tests := []struct {
addr string
// want is the summary's prefix, or empty when none is found.
want string
originASN int
peers int
newest int // in seconds after the start; 0 for none
}{
{"10.1.2.3", prefixV4, asA, 2, 20},
{"::ffff:10.1.2.3", prefixV4, asA, 2, 20},
{"10.1.3.4", "10.1.0.0/16", asB, 3, 30},
{"10.2.0.1", "10.0.0.0/8", asA, 1, 10},
{"10.9.0.1", "10.9.0.0/16", asA, 1, 0},
{"192.0.2.1", "", 0, 0, 0},
{"2001:db8:1::5", prefixV6, asB, 1, 20},
{"2001:db8:2::1", "2001:db8::/32", asA, 1, 10},
{"3000::1", "3000::/16", asA, 1, 10},
{"2001:db9::1", "", 0, 0, 0},
}
for _, tc := range tests {
t.Run(tc.addr, func(t *testing.T) {
t.Parallel()
got, found := table.Lookup(netip.MustParseAddr(tc.addr))
if tc.want == "" {
if found {
t.Errorf("Lookup(%s) = %+v, want none", tc.addr, got)
}
return
}
want := routetable.PrefixSummary{
Prefix: netip.MustParsePrefix(tc.want),
OriginASN: tc.originASN,
Peers: tc.peers,
}
if tc.newest != 0 {
want.Newest = timeAt(tc.newest)
}
if !found || !reflect.DeepEqual(got, want) {
t.Errorf("Lookup(%s) = %+v, %v; want %+v", tc.addr, got, found, want)
}
})
}
}
func TestOriginatedPrefixes(t *testing.T) {
t.Parallel()
table := routetable.New(routetable.Params{})
announceAll(table, "2001:db8::/32", asA, 30, peerA)
announceAll(table, prefixV4, asA, 10, peerA)
// asB's newer route for the same prefix does not count for asA.
announceAll(table, prefixV4, asB, 40, peerB)
announceAll(table, "10.1.0.0/16", asA, 10, peerA, peerB)
announceAll(table, "10.9.0.0/16", asA, 10, peerA, peerB, peerC)
table.Withdraw(netip.MustParsePrefix("10.9.0.0/16"), peerB)
table.Withdraw(netip.MustParsePrefix("10.9.0.0/16"), peerC)
want := []routetable.OriginatedPrefix{
{Prefix: netip.MustParsePrefix("10.1.0.0/16"), Newest: timeAt(20)},
{Prefix: netip.MustParsePrefix(prefixV4), Newest: timeAt(10)},
{Prefix: netip.MustParsePrefix("10.9.0.0/16")},
{Prefix: netip.MustParsePrefix("2001:db8::/32"), Newest: timeAt(30)},
}
got := table.OriginatedPrefixes(asA)
if !reflect.DeepEqual(got, want) {
t.Errorf("OriginatedPrefixes(%d) = %+v, want %+v", asA, got, want)
}
got = table.OriginatedPrefixes(asC)
if len(got) != 0 {
t.Errorf("OriginatedPrefixes(%d) = %+v, want none", asC, got)
}
}
func TestSamplePrefixes(t *testing.T) {
t.Parallel()
table := routetable.New(routetable.Params{})
announced := []string{
"10.0.0.0/24", "10.0.1.0/24", "10.0.2.0/24", "10.0.3.0/24",
"10.0.4.0/24", "10.0.0.0/16",
"2001:db8::/48", "2001:db8:1::/48", "2001:db8:2::/48",
}
for _, prefix := range announced {
announceAll(table, prefix, asA, 10, peerA)
}
tests := []struct {
name string
ipVersion, maskLength, limit int
want int
}{
{"fewer than there are", 4, 24, 3, 3},
{"more than there are", 4, 24, 10, 5},
{"IPv6", 6, 48, 10, 3},
{"a mask length with none", 4, 20, 10, 0},
{"no such IP version", 5, 24, 10, 0},
{"a mask length too long", 4, 33, 10, 0},
{"a negative mask length", 4, -1, 10, 0},
{"a limit of zero", 4, 24, 0, 0},
}
for _, tc := range tests {
t.Run(tc.name, func(t *testing.T) {
t.Parallel()
sample := table.SamplePrefixes(tc.ipVersion, tc.maskLength, tc.limit)
if len(sample) != tc.want {
t.Fatalf("SamplePrefixes(%d, %d, %d) returned %d prefixes, want %d",
tc.ipVersion, tc.maskLength, tc.limit, len(sample), tc.want)
}
seen := make(map[netip.Prefix]bool)
for _, got := range sample {
want := routetable.PrefixSummary{
Prefix: got.Prefix, OriginASN: asA, Peers: 1, Newest: timeAt(10),
}
if seen[got.Prefix] || got.Prefix.Bits() != tc.maskLength ||
!reflect.DeepEqual(got, want) {
t.Errorf("SamplePrefixes(%d, %d, %d) returned %+v",
tc.ipVersion, tc.maskLength, tc.limit, got)
}
seen[got.Prefix] = true
}
})
}
}
func TestCounts(t *testing.T) {
t.Parallel()
table := routetable.New(routetable.Params{})
checkCounts(t, table, routetable.Counts{
IPv4PrefixesByLength: byLength(ipv4Bits, nil),
IPv6PrefixesByLength: byLength(ipv6Bits, nil),
})
announceAll(table, prefixV4, asA, 10, peerA, peerB)
announceAll(table, "10.2.0.0/16", asA, 15, peerA)
announceAll(table, "2001:db8::/32", asA, 5, peerA)
announceAll(table, "2001:db8::/32", asA, 25, peerC)
checkCounts(t, table, routetable.Counts{
IPv4Prefixes: 2,
IPv6Prefixes: 1,
IPv4PrefixesByLength: byLength(ipv4Bits, map[int]int{16: 1, 24: 1}),
IPv6PrefixesByLength: byLength(ipv6Bits, map[int]int{32: 1}),
IPv4Routes: 3,
IPv6Routes: 2,
OldestKeptRoute: timeAt(5),
NewestKeptRoute: timeAt(25),
})
// peerC's route drops peerA's from the kept ones, and peerA leaves the
// IPv6 prefix.
announceAll(table, prefixV4, asA, 30, peerC)
table.Withdraw(netip.MustParsePrefix("2001:db8::/32"), peerA)
checkCounts(t, table, routetable.Counts{
IPv4Prefixes: 2,
IPv6Prefixes: 1,
IPv4PrefixesByLength: byLength(ipv4Bits, map[int]int{16: 1, 24: 1}),
IPv6PrefixesByLength: byLength(ipv6Bits, map[int]int{32: 1}),
IPv4Routes: 4,
IPv6Routes: 1,
OldestKeptRoute: timeAt(15),
NewestKeptRoute: timeAt(30),
})
// peerC was the last peer of the IPv6 prefix.
table.PeerDown(peerC)
checkCounts(t, table, routetable.Counts{
IPv4Prefixes: 2,
IPv4PrefixesByLength: byLength(ipv4Bits, map[int]int{16: 1, 24: 1}),
IPv6PrefixesByLength: byLength(ipv6Bits, nil),
IPv4Routes: 3,
OldestKeptRoute: timeAt(15),
NewestKeptRoute: timeAt(20),
})
}
// The address lengths of the two IP versions.
const (
ipv4Bits = 32
ipv6Bits = 128
)
// byLength returns the prefix counts by mask length, for addresses of bits
// bits, with counts[n] prefixes of mask length n and none of any other.
func byLength(bits int, counts map[int]int) []int {
all := make([]int, bits+1)
for n, count := range counts {
all[n] = count
}
return all
}
// checkCounts checks that the table's counts are want.
func checkCounts(t *testing.T, table *routetable.Table, want routetable.Counts) {
t.Helper()
got := table.Counts()
if !reflect.DeepEqual(got, want) {
t.Errorf("Counts() = %+v, want %+v", got, want)
}
}
internal/routetable/routetable.go
-578
View File
@@ -1,578 +0,0 @@
// Package routetable holds the whole routing table in memory. For each prefix
// and origin AS, it keeps the peers whose current route to the prefix has that
// origin AS. It also keeps the two newest of those routes in full. It applies
// announcements, withdrawals and peers' sessions going down. It answers the
// route reads that the pages and the API need.
package routetable
import (
"cmp"
"net/netip"
"slices"
"sync"
"time"
"github.com/bits-and-blooms/bitset"
"github.com/google/btree"
)
const (
// keptRoutesPerOrigin is how many routes of each prefix and origin AS are
// kept in full: the newest ones. Every other peer that carries the prefix
// with that origin AS keeps only its bit in the peer set.
keptRoutesPerOrigin = 2
// keptTimesDegree is the degree of the B-tree that holds the times of the
// kept routes in order: each of its nodes holds up to 63 times.
keptTimesDegree = 32
ipv4Bits = 32
ipv6Bits = 128
// The address lookup ignores prefixes shorter than these mask lengths. A
// default route or a route this short only says where a peer sends traffic
// it has no better route for; it does not say who holds the address.
minLookupMaskLengthV4 = 8
minLookupMaskLengthV6 = 16
// The IP versions SamplePrefixes takes.
ipVersion4 = 4
ipVersion6 = 6
)
// Route is one peer's route to one prefix. The next hop is the peer itself, so
// it is not stored.
type Route struct {
// Peer is the peer's address.
Peer string
ASPath []int
// Time is when the peer announced the route.
Time time.Time
}
// Origin is one origin AS of a prefix: the peers whose current route to the
// prefix has that origin AS, and the newest of those routes in full.
type Origin struct {
// ASN is the origin AS. 0 means no known origin: the path ends in an AS
// set of several members.
ASN int
// Peers are the peers' addresses, in the order of their numbers: the
// order they were first seen in.
Peers []string
// Routes are the kept routes, newest first: at most two, and none when
// both were withdrawn while other peers still carry the prefix.
Routes []Route
}
// Change is the current content of one prefix and origin AS pair that changed:
// what a database row for the pair holds.
type Change struct {
Prefix netip.Prefix
OriginASN int
// Removed says that no peer carries the prefix with this origin AS any
// more. Peers and Routes are then nil.
Removed bool
// Peers holds the number of each peer that carries the prefix with this
// origin AS.
Peers *bitset.BitSet
// Routes are the kept routes, newest first.
Routes []Route
}
// Params configures New.
type Params struct {
// PeerNumbers holds the number each peer was given before, by address. A
// peer not in it gets the next number above all of them the first time it
// announces a route.
PeerNumbers map[string]uint
}
// Table is the routing table. Its methods are safe for concurrent use.
type Table struct {
mu sync.RWMutex
ipv4 family
ipv6 family
// byOrigin holds, for each origin AS, the prefixes it originates.
byOrigin map[int]map[netip.Prefix]*origin
// peerNumbers and peerAddresses give each peer's number by its address
// and its address by its number. nextPeerNumber is the number the next
// new peer gets.
peerNumbers map[string]uint
peerAddresses map[uint]string
nextPeerNumber uint
// peerPrefixes counts the prefixes each peer carries, by its number.
peerPrefixes map[uint]int
// newPeers and changed are what TakeChanges hands over next: the peers
// numbered, and the prefix and origin AS pairs changed, since it last ran.
newPeers map[string]uint
changed map[pair]struct{}
// keptTimes counts the kept routes at each time, in time order, so its
// first and last items are the oldest and newest kept routes.
keptTimes *btree.BTreeG[keptTime]
// originsWithoutKeptRoute counts the origins that peers carry but that
// keep no route.
originsWithoutKeptRoute int
}
// New returns an empty table that numbers new peers on from
// params.PeerNumbers.
func New(params Params) *Table {
t := &Table{
ipv4: newFamily(ipv4Bits, minLookupMaskLengthV4),
ipv6: newFamily(ipv6Bits, minLookupMaskLengthV6),
byOrigin: make(map[int]map[netip.Prefix]*origin),
peerNumbers: make(map[string]uint, len(params.PeerNumbers)),
peerAddresses: make(map[uint]string, len(params.PeerNumbers)),
peerPrefixes: make(map[uint]int),
newPeers: make(map[string]uint),
changed: make(map[pair]struct{}),
keptTimes: btree.NewG(keptTimesDegree, func(a, b keptTime) bool {
return a.time.Before(b.time)
}),
}
for address, number := range params.PeerNumbers {
t.peerNumbers[address] = number
t.peerAddresses[number] = address
t.nextPeerNumber = max(t.nextPeerNumber, number+1)
}
return t
}
// Announce applies an announcement: the peer's current route to prefix is now
// r, with origin AS originASN. A peer that carried the prefix with another
// origin AS leaves that origin's peer set, and its route there is dropped. r
// is kept in full when it is one of the two newest routes of its prefix and
// origin AS; otherwise only the peer's bit is.
//
// An announcement whose path is empty or holds AS 0 is a withdrawal, and the
// caller passes it to Withdraw instead.
func (t *Table) Announce(prefix netip.Prefix, originASN int, r Route) {
t.mu.Lock()
defer t.mu.Unlock()
if !prefix.IsValid() {
return
}
prefix = prefix.Masked()
peer := t.numberPeer(r.Peer)
current := originWithPeer(t.origins(prefix), peer)
if current != nil && current.asn != originASN {
t.leave(prefix, current, peer)
}
t.join(prefix, t.findOrAddOrigin(prefix, originASN), route{
peer: peer,
path: slices.Clone(r.ASPath),
time: r.Time,
})
}
// Withdraw applies a withdrawal: the peer no longer carries prefix. It leaves
// the prefix's peer set, and its kept route, if any, is dropped.
func (t *Table) Withdraw(prefix netip.Prefix, peer string) {
t.mu.Lock()
defer t.mu.Unlock()
number, known := t.peerNumbers[peer]
if !known || !prefix.IsValid() {
return
}
prefix = prefix.Masked()
o := originWithPeer(t.origins(prefix), number)
if o != nil {
t.leave(prefix, o, number)
}
}
// PeerDown applies a peer's session going down: the peer leaves every peer
// set, and every route of it is dropped. When the peer carries any prefix, it
// visits every prefix in the table.
func (t *Table) PeerDown(peer string) {
t.mu.Lock()
defer t.mu.Unlock()
number, known := t.peerNumbers[peer]
if !known || t.peerPrefixes[number] == 0 {
return
}
for _, f := range []*family{&t.ipv4, &t.ipv6} {
for _, prefixes := range f.byLength {
for prefix, origins := range prefixes {
o := originWithPeer(origins, number)
if o != nil {
t.leave(prefix, o, number)
}
}
}
}
}
// TakeChanges returns the prefix and origin AS pairs that changed since it last
// ran, in prefix order, each with its current content. It also returns the
// peers numbered since then, by address. It then clears both records.
func (t *Table) TakeChanges() ([]Change, map[string]uint) {
t.mu.Lock()
defer t.mu.Unlock()
changes := make([]Change, 0, len(t.changed))
for key := range t.changed {
changes = append(changes, t.currentContent(key))
}
slices.SortFunc(changes, func(a, b Change) int {
return cmp.Or(a.Prefix.Compare(b.Prefix),
cmp.Compare(a.OriginASN, b.OriginASN))
})
newPeers := t.newPeers
t.newPeers = make(map[string]uint)
t.changed = make(map[pair]struct{})
return changes, newPeers
}
// numberPeer returns the peer's number, and gives it the next one the first
// time the table sees it.
func (t *Table) numberPeer(address string) uint {
number, known := t.peerNumbers[address]
if known {
return number
}
number = t.nextPeerNumber
t.nextPeerNumber++
t.peerNumbers[address] = number
t.peerAddresses[number] = address
t.newPeers[address] = number
return number
}
// familyOf returns the part of the table for addr's IP version.
func (t *Table) familyOf(addr netip.Addr) *family {
if addr.Is4() {
return &t.ipv4
}
return &t.ipv6
}
// prefixesOfLength returns the prefixes of the IP version and the mask length
// of prefix, a valid masked prefix.
func (t *Table) prefixesOfLength(prefix netip.Prefix) map[netip.Prefix][]*origin {
return t.familyOf(prefix.Addr()).byLength[prefix.Bits()]
}
// origins returns the origins of prefix, a valid masked prefix.
func (t *Table) origins(prefix netip.Prefix) []*origin {
return t.prefixesOfLength(prefix)[prefix]
}
// findOrAddOrigin returns the origin of prefix with AS asn, added with no
// peers when the prefix has none with that AS.
func (t *Table) findOrAddOrigin(prefix netip.Prefix, asn int) *origin {
prefixes := t.prefixesOfLength(prefix)
origins := prefixes[prefix]
i, found := slices.BinarySearchFunc(origins, asn,
func(o *origin, asn int) int { return cmp.Compare(o.asn, asn) })
if found {
return origins[i]
}
o := &origin{asn: asn}
prefixes[prefix] = slices.Insert(origins, i, o)
if t.byOrigin[asn] == nil {
t.byOrigin[asn] = make(map[netip.Prefix]*origin)
}
t.byOrigin[asn][prefix] = o
return o
}
// removeOrigin removes o, which no peer carries any more, from prefix, and the
// prefix from the table when it has no origin left.
func (t *Table) removeOrigin(prefix netip.Prefix, o *origin) {
prefixes := t.prefixesOfLength(prefix)
origins := slices.DeleteFunc(prefixes[prefix],
func(other *origin) bool { return other == o })
if len(origins) == 0 {
delete(prefixes, prefix)
} else {
prefixes[prefix] = origins
}
delete(t.byOrigin[o.asn], prefix)
if len(t.byOrigin[o.asn]) == 0 {
delete(t.byOrigin, o.asn)
}
}
// join puts the peer of r in the peer set of o, an origin of prefix, and keeps
// r in full when it is one of the two newest routes of o. The peer's earlier
// route, if o keeps one, is replaced: it is no longer the peer's current route.
// The pair is recorded as changed only when its peer set or kept routes change.
func (t *Table) join(prefix netip.Prefix, o *origin, r route) {
if slices.ContainsFunc(o.routes, r.equal) {
// The peer announced its kept route again.
return
}
wasWithoutKeptRoute := o.withoutKeptRoute()
joined := !o.peers.Test(r.peer)
if joined {
o.peers.Set(r.peer)
t.familyOf(prefix.Addr()).routes++
t.peerPrefixes[r.peer]++
}
dropped := t.dropKeptRoute(o, r.peer)
kept := t.keepIfNewest(o, r)
if joined || dropped || kept {
t.recordChange(prefix, o, wasWithoutKeptRoute)
}
}
// leave takes peer out of the peer set of o, an origin of prefix, and drops
// its kept route, if any.
func (t *Table) leave(prefix netip.Prefix, o *origin, peer uint) {
wasWithoutKeptRoute := o.withoutKeptRoute()
o.peers.Clear(peer)
t.dropKeptRoute(o, peer)
t.familyOf(prefix.Addr()).routes--
t.peerPrefixes[peer]--
t.recordChange(prefix, o, wasWithoutKeptRoute)
}
// keepIfNewest keeps r in o unless both kept routes of o are newer, and says
// whether it kept r. When o already keeps two routes, the older one is
// dropped, and its peer keeps only its bit.
func (t *Table) keepIfNewest(o *origin, r route) bool {
// The kept routes stay newest first: r goes before the first one that is
// not newer than it.
i := 0
for i < len(o.routes) && o.routes[i].time.After(r.time) {
i++
}
if i == keptRoutesPerOrigin {
return false
}
if len(o.routes) == keptRoutesPerOrigin {
t.dropKeptRoute(o, o.routes[keptRoutesPerOrigin-1].peer)
}
o.routes = slices.Insert(o.routes, i, r)
t.countKeptTime(r.time, 1)
return true
}
// dropKeptRoute drops the peer's route from the kept routes of o, if o keeps
// one, and says whether it did.
func (t *Table) dropKeptRoute(o *origin, peer uint) bool {
i := slices.IndexFunc(o.routes, func(r route) bool { return r.peer == peer })
if i < 0 {
return false
}
t.countKeptTime(o.routes[i].time, -1)
o.routes = slices.Delete(o.routes, i, i+1)
return true
}
// recordChange records that the pair of prefix and the AS of o changed, and
// brings the count of origins without a kept route up to date.
// wasWithoutKeptRoute says whether o was such an origin before the change. An
// origin that no peer carries any more is removed.
func (t *Table) recordChange(
prefix netip.Prefix, o *origin, wasWithoutKeptRoute bool,
) {
t.changed[pair{prefix: prefix, asn: o.asn}] = struct{}{}
if wasWithoutKeptRoute {
t.originsWithoutKeptRoute--
}
if o.withoutKeptRoute() {
t.originsWithoutKeptRoute++
}
if o.peers.None() {
t.removeOrigin(prefix, o)
}
}
// countKeptTime adds delta to the number of kept routes with time at.
func (t *Table) countKeptTime(at time.Time, delta int) {
count, _ := t.keptTimes.Get(keptTime{time: at})
count.time = at
count.routes += delta
if count.routes == 0 {
t.keptTimes.Delete(count)
return
}
t.keptTimes.ReplaceOrInsert(count)
}
// currentContent returns what the table holds for the pair now.
func (t *Table) currentContent(key pair) Change {
o := t.byOrigin[key.asn][key.prefix]
if o == nil {
return Change{Prefix: key.prefix, OriginASN: key.asn, Removed: true}
}
return Change{
Prefix: key.prefix,
OriginASN: key.asn,
Peers: o.peers.Clone(),
Routes: t.exportRoutes(o.routes),
}
}
// exportRoutes returns copies of routes that name each peer by its address, or
// nil when there are none.
func (t *Table) exportRoutes(routes []route) []Route {
if len(routes) == 0 {
return nil
}
exported := make([]Route, 0, len(routes))
for _, r := range routes {
exported = append(exported, Route{
Peer: t.peerAddresses[r.peer],
ASPath: slices.Clone(r.path),
Time: r.time,
})
}
return exported
}
// family is the part of the table for one IP version.
type family struct {
// byLength[n] holds the prefixes of mask length n that some peer carries,
// each with its origins in AS number order.
byLength []map[netip.Prefix][]*origin
// minLookupLength is the shortest mask length the address lookup uses.
minLookupLength int
// routes counts the peer and prefix pairs: each peer that carries a
// prefix counts once for it.
routes int
}
// newFamily returns an empty family for addresses of bits bits.
func newFamily(bits, minLookupLength int) family {
f := family{
byLength: make([]map[netip.Prefix][]*origin, bits+1),
minLookupLength: minLookupLength,
}
for n := range f.byLength {
f.byLength[n] = make(map[netip.Prefix][]*origin)
}
return f
}
// prefixesByLength returns how many prefixes of each mask length f holds.
func (f *family) prefixesByLength() []int {
counts := make([]int, len(f.byLength))
for n, prefixes := range f.byLength {
counts[n] = len(prefixes)
}
return counts
}
// origin is what the table holds for one prefix and origin AS.
type origin struct {
asn int
// peers holds the number of each peer whose current route to the prefix
// has this origin AS.
peers bitset.BitSet
// routes are the newest of those routes, newest first, at most
// keptRoutesPerOrigin.
routes []route
}
// withoutKeptRoute says whether peers carry o but o keeps no route.
func (o *origin) withoutKeptRoute() bool {
return o.peers.Any() && len(o.routes) == 0
}
// newestTime returns the time of the newest kept route of o, or nil when o
// keeps none.
func (o *origin) newestTime() *time.Time {
if len(o.routes) == 0 {
return nil
}
newest := o.routes[0].time
return &newest
}
// originWithPeer returns the origin whose peer set holds peer, or nil when
// none does. A peer carries a prefix with one origin AS at most.
func originWithPeer(origins []*origin, peer uint) *origin {
for _, o := range origins {
if o.peers.Test(peer) {
return o
}
}
return nil
}
// route is a kept route. peer is the peer's number.
type route struct {
peer uint
path []int
time time.Time
}
// equal says whether r and other are the same peer's route with the same path
// and time.
func (r route) equal(other route) bool {
return r.peer == other.peer && r.time.Equal(other.time) &&
slices.Equal(r.path, other.path)
}
// pair names one prefix and origin AS.
type pair struct {
prefix netip.Prefix
asn int
}
// keptTime counts the kept routes with one time.
type keptTime struct {
time time.Time
routes int
}
internal/routetable/routetable_test.go
-543
View File
@@ -1,543 +0,0 @@
package routetable_test
import (
"net/netip"
"reflect"
"slices"
"sync"
"testing"
"time"
"github.com/bits-and-blooms/bitset"
"git.eeqj.de/sneak/routewatch/internal/routetable"
)
const (
peerA = "192.0.2.1"
peerB = "192.0.2.2"
peerC = "2001:db8::3"
asA = 64500
asB = 64501
asC = 64502
// transitAS is the first AS of every test path, the peer's own.
transitAS = 64496
prefixV4 = "10.1.2.0/24"
prefixV6 = "2001:db8:1::/48"
)
// at returns the time second seconds after the start of every test.
func at(second int) time.Time {
return time.Date(2026, 1, 2, 3, 4, 5, 0, time.UTC).
Add(time.Duration(second) * time.Second)
}
// timeAt returns at(second) for a field that may be nil.
func timeAt(second int) *time.Time {
t := at(second)
return &t
}
// routeOf returns the route peer announces with origin AS asn second seconds
// after the start.
func routeOf(peer string, asn, second int) routetable.Route {
return routetable.Route{
Peer: peer, ASPath: []int{transitAS, asn}, Time: at(second),
}
}
// step is one update a test applies to a table for one prefix.
type step func(table *routetable.Table, prefix netip.Prefix)
// announce is the step of peer announcing prefix with origin AS asn second
// seconds after the start.
func announce(peer string, asn, second int) step {
return func(table *routetable.Table, prefix netip.Prefix) {
table.Announce(prefix, asn, routeOf(peer, asn, second))
}
}
// withdraw is the step of peer withdrawing prefix.
func withdraw(peer string) step {
return func(table *routetable.Table, prefix netip.Prefix) {
table.Withdraw(prefix, peer)
}
}
// peerDown is the step of peer's session going down.
func peerDown(peer string) step {
return func(table *routetable.Table, _ netip.Prefix) {
table.PeerDown(peer)
}
}
// rule is one case of the rule that decides what the table holds for a prefix:
// the steps applied to an empty table, and the prefix's origins after them.
// The peers are numbered in the order they first announce, which is the order
// Origin.Peers lists them in.
type rule struct {
name string
steps []step
want []routetable.Origin
}
// checkRules runs each case for an IPv4 and an IPv6 prefix.
func checkRules(t *testing.T, rules []rule) {
t.Helper()
for _, tc := range rules {
for _, text := range []string{prefixV4, prefixV6} {
t.Run(tc.name+" "+text, func(t *testing.T) {
t.Parallel()
prefix := netip.MustParsePrefix(text)
table := routetable.New(routetable.Params{})
for _, apply := range tc.steps {
apply(table, prefix)
}
got := table.Origins(prefix)
if !reflect.DeepEqual(got, tc.want) {
t.Errorf("Origins(%s) = %+v, want %+v", prefix, got, tc.want)
}
})
}
}
}
func TestAnnouncementsKeepTheTwoNewestRoutes(t *testing.T) {
t.Parallel()
checkRules(t, []rule{
{
name: "an announcement is kept",
steps: []step{announce(peerA, asA, 10)},
want: []routetable.Origin{{ASN: asA, Peers: []string{peerA},
Routes: []routetable.Route{routeOf(peerA, asA, 10)}}},
},
{
name: "a lagging peer's older route is not kept",
steps: []step{announce(peerA, asA, 20), announce(peerB, asA, 30),
announce(peerC, asA, 10)},
want: []routetable.Origin{{ASN: asA,
Peers: []string{peerA, peerB, peerC}, Routes: []routetable.Route{
routeOf(peerB, asA, 30), routeOf(peerA, asA, 20),
}}},
},
{
name: "a third newer route drops the older kept one, whose peer " +
"keeps its bit",
steps: []step{announce(peerA, asA, 10), announce(peerB, asA, 20),
announce(peerC, asA, 30)},
want: []routetable.Origin{{ASN: asA,
Peers: []string{peerA, peerB, peerC}, Routes: []routetable.Route{
routeOf(peerC, asA, 30), routeOf(peerB, asA, 20),
}}},
},
{
name: "a route as old as the older kept one replaces it",
steps: []step{announce(peerA, asA, 10), announce(peerB, asA, 20),
announce(peerC, asA, 10)},
want: []routetable.Origin{{ASN: asA,
Peers: []string{peerA, peerB, peerC}, Routes: []routetable.Route{
routeOf(peerB, asA, 20), routeOf(peerC, asA, 10),
}}},
},
{
name: "a peer's new route replaces its kept one",
steps: []step{announce(peerA, asA, 10), announce(peerB, asA, 20),
announce(peerA, asA, 30)},
want: []routetable.Origin{{ASN: asA, Peers: []string{peerA, peerB},
Routes: []routetable.Route{
routeOf(peerA, asA, 30), routeOf(peerB, asA, 20),
}}},
},
{
name: "a peer's new route replaces its kept one even when older",
steps: []step{announce(peerA, asA, 10), announce(peerB, asA, 20),
announce(peerB, asA, 5)},
want: []routetable.Origin{{ASN: asA, Peers: []string{peerA, peerB},
Routes: []routetable.Route{
routeOf(peerA, asA, 10), routeOf(peerB, asA, 5),
}}},
},
})
}
func TestAPeerCarriesAPrefixWithOneOrigin(t *testing.T) {
t.Parallel()
checkRules(t, []rule{
{
name: "a peer moving to a new origin leaves the old one",
steps: []step{announce(peerA, asA, 10), announce(peerB, asA, 20),
announce(peerA, asB, 30)},
want: []routetable.Origin{
{ASN: asA, Peers: []string{peerB},
Routes: []routetable.Route{routeOf(peerB, asA, 20)}},
{ASN: asB, Peers: []string{peerA},
Routes: []routetable.Route{routeOf(peerA, asB, 30)}},
},
},
{
name: "a peer whose route is not kept moves to a new origin",
steps: []step{announce(peerA, asA, 10), announce(peerB, asA, 20),
announce(peerC, asA, 30), announce(peerA, asB, 40)},
want: []routetable.Origin{
{ASN: asA, Peers: []string{peerB, peerC},
Routes: []routetable.Route{
routeOf(peerC, asA, 30), routeOf(peerB, asA, 20),
}},
{ASN: asB, Peers: []string{peerA},
Routes: []routetable.Route{routeOf(peerA, asB, 40)}},
},
},
{
name: "origins are listed in AS number order",
steps: []step{announce(peerA, asC, 10), announce(peerB, asA, 20),
announce(peerC, asB, 30)},
want: []routetable.Origin{
{ASN: asA, Peers: []string{peerB},
Routes: []routetable.Route{routeOf(peerB, asA, 20)}},
{ASN: asB, Peers: []string{peerC},
Routes: []routetable.Route{routeOf(peerC, asB, 30)}},
{ASN: asC, Peers: []string{peerA},
Routes: []routetable.Route{routeOf(peerA, asC, 10)}},
},
},
})
}
func TestWithdrawalsAndPeerDowns(t *testing.T) {
t.Parallel()
// threePeers is peerA, peerB and peerC announcing in turn; the routes of
// peerC and peerB are kept.
threePeers := []step{announce(peerA, asA, 10), announce(peerB, asA, 20),
announce(peerC, asA, 30)}
checkRules(t, []rule{
{
name: "a withdrawal drops the peer's kept route",
steps: append(slices.Clone(threePeers), withdraw(peerC)),
want: []routetable.Origin{{ASN: asA, Peers: []string{peerA, peerB},
Routes: []routetable.Route{routeOf(peerB, asA, 20)}}},
},
{
name: "a withdrawal of a route that is not kept leaves the kept ones",
steps: append(slices.Clone(threePeers), withdraw(peerA)),
want: []routetable.Origin{{ASN: asA, Peers: []string{peerB, peerC},
Routes: []routetable.Route{
routeOf(peerC, asA, 30), routeOf(peerB, asA, 20),
}}},
},
{
name: "an origin whose kept routes are withdrawn keeps its other " +
"peers",
steps: append(slices.Clone(threePeers), withdraw(peerB),
withdraw(peerC)),
want: []routetable.Origin{{ASN: asA, Peers: []string{peerA}}},
},
{
name: "an origin without kept routes keeps the next update",
steps: append(slices.Clone(threePeers), withdraw(peerB),
withdraw(peerC), announce(peerA, asA, 5)),
want: []routetable.Origin{{ASN: asA, Peers: []string{peerA},
Routes: []routetable.Route{routeOf(peerA, asA, 5)}}},
},
{
name: "an origin no peer carries is removed",
steps: []step{announce(peerA, asA, 10), announce(peerB, asB, 20),
withdraw(peerA)},
want: []routetable.Origin{{ASN: asB, Peers: []string{peerB},
Routes: []routetable.Route{routeOf(peerB, asB, 20)}}},
},
{
name: "a prefix no peer carries is removed",
steps: []step{announce(peerA, asA, 10), withdraw(peerA)},
want: nil,
},
{
name: "a withdrawal from a peer that does not carry it changes nothing",
steps: []step{announce(peerA, asA, 10), withdraw(peerB)},
want: []routetable.Origin{{ASN: asA, Peers: []string{peerA},
Routes: []routetable.Route{routeOf(peerA, asA, 10)}}},
},
{
name: "a peer going down leaves the prefix",
steps: []step{announce(peerA, asA, 10), announce(peerB, asA, 20),
peerDown(peerB)},
want: []routetable.Origin{{ASN: asA, Peers: []string{peerA},
Routes: []routetable.Route{routeOf(peerA, asA, 10)}}},
},
{
name: "a peer going down after it withdrew changes nothing",
steps: []step{announce(peerA, asA, 10), announce(peerB, asA, 20),
withdraw(peerB), peerDown(peerB)},
want: []routetable.Origin{{ASN: asA, Peers: []string{peerA},
Routes: []routetable.Route{routeOf(peerA, asA, 10)}}},
},
})
}
func TestPeerDownLeavesEveryPrefix(t *testing.T) {
t.Parallel()
table := routetable.New(routetable.Params{})
prefixes := []netip.Prefix{
netip.MustParsePrefix(prefixV4),
netip.MustParsePrefix("10.2.0.0/16"),
netip.MustParsePrefix(prefixV6),
}
// peerA carries every prefix with origin asA, peerB the first one with
// origin asB.
for _, prefix := range prefixes {
table.Announce(prefix, asA, routeOf(peerA, asA, 10))
}
table.Announce(prefixes[0], asB, routeOf(peerB, asB, 20))
table.PeerDown(peerA)
want := []routetable.Origin{{ASN: asB, Peers: []string{peerB},
Routes: []routetable.Route{routeOf(peerB, asB, 20)}}}
got := table.Origins(prefixes[0])
if !reflect.DeepEqual(got, want) {
t.Errorf("Origins(%s) = %+v, want %+v", prefixes[0], got, want)
}
for _, prefix := range prefixes[1:] {
got := table.Origins(prefix)
if got != nil {
t.Errorf("Origins(%s) = %+v after the only peer went down, want nil",
prefix, got)
}
}
counts := table.Counts()
if counts.IPv4Prefixes != 1 || counts.IPv6Prefixes != 0 ||
counts.IPv4Routes != 1 || counts.IPv6Routes != 0 {
t.Errorf("Counts() = %+v, want one IPv4 prefix and route", counts)
}
if len(table.OriginatedPrefixes(asA)) != 0 {
t.Errorf("OriginatedPrefixes(%d) = %+v, want none",
asA, table.OriginatedPrefixes(asA))
}
}
func TestCountsOriginsWithoutKeptRoute(t *testing.T) {
t.Parallel()
prefix := netip.MustParsePrefix(prefixV4)
table := routetable.New(routetable.Params{})
steps := []struct {
apply step
want int
}{
{announce(peerA, asA, 10), 0},
{announce(peerB, asA, 20), 0},
{announce(peerC, asA, 30), 0},
{withdraw(peerC), 0},
{withdraw(peerB), 1},
// An update from the last peer is kept.
{announce(peerA, asA, 40), 0},
{withdraw(peerA), 0},
// Once more without kept routes, then without peers: the origin
// is removed.
{announce(peerA, asA, 10), 0},
{announce(peerB, asA, 20), 0},
{announce(peerC, asA, 30), 0},
{withdraw(peerB), 0},
{withdraw(peerC), 1},
{peerDown(peerA), 0},
}
for i, s := range steps {
s.apply(table, prefix)
got := table.Counts().OriginsWithoutKeptRoute
if got != s.want {
t.Errorf("after step %d, OriginsWithoutKeptRoute = %d, want %d",
i, got, s.want)
}
}
}
// change is a routetable.Change with its peer set as a list of peer numbers.
type change struct {
Prefix string
OriginASN int
Removed bool
Peers []uint
Routes []routetable.Route
}
// peerNumbers returns the numbers in set, or nil for a nil set.
func peerNumbers(set *bitset.BitSet) []uint {
if set == nil {
return nil
}
return slices.Collect(set.EachSet())
}
func TestTakeChanges(t *testing.T) {
t.Parallel()
// peerA was numbered 5 before; the next new peer is 6.
table := routetable.New(routetable.Params{
PeerNumbers: map[string]uint{peerA: 5},
})
v4 := netip.MustParsePrefix(prefixV4)
v6 := netip.MustParsePrefix(prefixV6)
table.Announce(v6, asA, routeOf(peerA, asA, 10))
table.Announce(v4, asA, routeOf(peerA, asA, 10))
table.Announce(v4, asA, routeOf(peerB, asA, 20))
checkChanges(t, table, []change{
{Prefix: prefixV4, OriginASN: asA, Peers: []uint{5, 6},
Routes: []routetable.Route{
routeOf(peerB, asA, 20), routeOf(peerA, asA, 10),
}},
{Prefix: prefixV6, OriginASN: asA, Peers: []uint{5},
Routes: []routetable.Route{routeOf(peerA, asA, 10)}},
}, map[string]uint{peerB: 6})
// TakeChanges cleared the record.
checkChanges(t, table, []change{}, map[string]uint{})
// peerB moves to asB; peerA withdraws, which removes the pair of v4 and
// asA.
table.Announce(v4, asB, routeOf(peerB, asB, 30))
table.Withdraw(v4, peerA)
checkChanges(t, table, []change{
{Prefix: prefixV4, OriginASN: asA, Removed: true},
{Prefix: prefixV4, OriginASN: asB, Peers: []uint{6},
Routes: []routetable.Route{routeOf(peerB, asB, 30)}},
}, map[string]uint{})
}
func TestTakeChangesHoldsOnlyPairsThatChanged(t *testing.T) {
t.Parallel()
table := routetable.New(routetable.Params{})
v4 := netip.MustParsePrefix(prefixV4)
v6 := netip.MustParsePrefix(prefixV6)
// peerA, peerB and peerC are numbered 0, 1 and 2. The routes of peerC and
// peerB to v4 are kept; peerA keeps only its bit.
announceAll(table, prefixV4, asA, 10, peerA, peerB, peerC)
table.Announce(v6, asA, routeOf(peerB, asA, 10))
table.TakeChanges()
// peerA announces again, with a time older than both kept routes.
table.Announce(v4, asA, routeOf(peerA, asA, 15))
checkChanges(t, table, []change{}, map[string]uint{})
// peerC announces its kept route again.
table.Announce(v4, asA, routeOf(peerC, asA, 30))
checkChanges(t, table, []change{}, map[string]uint{})
// peerC announces the same path with a newer time, so its kept route
// changes.
table.Announce(v4, asA, routeOf(peerC, asA, 40))
checkChanges(t, table, []change{
{Prefix: prefixV4, OriginASN: asA, Peers: []uint{0, 1, 2},
Routes: []routetable.Route{
routeOf(peerC, asA, 40), routeOf(peerB, asA, 20),
}},
}, map[string]uint{})
// peerA withdraws v6, which it does not carry.
table.Withdraw(v6, peerA)
checkChanges(t, table, []change{}, map[string]uint{})
// peerA going down changes v4, the one prefix it carries.
table.PeerDown(peerA)
checkChanges(t, table, []change{
{Prefix: prefixV4, OriginASN: asA, Peers: []uint{1, 2},
Routes: []routetable.Route{
routeOf(peerC, asA, 40), routeOf(peerB, asA, 20),
}},
}, map[string]uint{})
// peerA, which carries nothing now, goes down again.
table.PeerDown(peerA)
checkChanges(t, table, []change{}, map[string]uint{})
}
// TestConcurrentUse has goroutines update one table while others read it. The
// race detector, which make test runs, fails it when a method uses the table
// without its lock.
func TestConcurrentUse(t *testing.T) {
t.Parallel()
const (
// rounds is how many times each goroutine calls its method.
rounds = 1000
ipVersion = 4
sampleSize = 10
)
table := routetable.New(routetable.Params{})
prefix := netip.MustParsePrefix(prefixV4)
uses := []func(round int){
func(round int) { table.Announce(prefix, asA, routeOf(peerA, asA, round)) },
func(round int) { table.Announce(prefix, asB, routeOf(peerB, asB, round)) },
func(int) { table.Withdraw(prefix, peerA) },
func(int) { table.PeerDown(peerB) },
func(int) { table.TakeChanges() },
func(int) { table.Lookup(prefix.Addr()) },
func(int) { table.Origins(prefix) },
func(int) { table.OriginatedPrefixes(asA) },
func(int) { table.SamplePrefixes(ipVersion, prefix.Bits(), sampleSize) },
func(int) { table.Counts() },
}
var wg sync.WaitGroup
for _, use := range uses {
wg.Go(func() {
for round := range rounds {
use(round)
}
})
}
wg.Wait()
}
// checkChanges checks that TakeChanges returns want and wantPeers.
func checkChanges(
t *testing.T, table *routetable.Table, want []change,
wantPeers map[string]uint,
) {
t.Helper()
changes, newPeers := table.TakeChanges()
got := make([]change, 0, len(changes))
for _, c := range changes {
got = append(got, change{
Prefix: c.Prefix.String(), OriginASN: c.OriginASN, Removed: c.Removed,
Peers: peerNumbers(c.Peers), Routes: c.Routes,
})
}
if !reflect.DeepEqual(got, want) {
t.Errorf("TakeChanges() changes = %+v, want %+v", got, want)
}
if !reflect.DeepEqual(newPeers, wantPeers) {
t.Errorf("TakeChanges() new peers = %v, want %v", newPeers, wantPeers)
}
}
internal/routewatch/app.go
+5 -29
View File
@@ -8,11 +8,9 @@ package routewatch
import (
"context"
"errors"
"path/filepath"
"sync"
"time"
"git.eeqj.de/sneak/routewatch/internal/blocklist"
"git.eeqj.de/sneak/routewatch/internal/config"
"git.eeqj.de/sneak/routewatch/internal/database"
"git.eeqj.de/sneak/routewatch/internal/logger"
@@ -54,13 +52,8 @@ type RouteWatch struct {
peeringHandler *PeeringHandler
asnFetcher *ASNFetcher
dbMaintainer *DBMaintainer
blocklists *blocklist.Blocklists
}
// blocklistsDirName is the directory in the state directory that holds the
// blocklist copies.
const blocklistsDirName = "blocklists"
// New creates a new RouteWatch instance
func New(deps Dependencies) *RouteWatch {
rw := &RouteWatch{
@@ -129,18 +122,6 @@ func (rw *RouteWatch) Run(ctx context.Context) error {
rw.asnFetcher.Start(ctx)
rw.server.SetASNFetcher(rw.asnFetcher)
// Load the blocklist copies from disk and start downloading the lists.
// The HTTP handlers read the server's blocklists without a lock, so they
// are set before the handlers run.
blocklists := blocklist.New(blocklist.Params{
Dir: filepath.Join(rw.config.StateDir, blocklistsDirName),
Lists: blocklist.Lists(),
Logger: rw.logger.Logger,
})
blocklists.Start(ctx)
rw.blocklists = blocklists
rw.server.SetBlocklists(blocklists)
// Start HTTP server
err = rw.server.Start(ctx)
if err != nil {
@@ -158,12 +139,11 @@ func (rw *RouteWatch) Run(ctx context.Context) error {
}
// Shutdown stops all services in order: the streamer, the handlers, which
// write what they hold, the WHOIS fetcher, the database maintainer, the
// blocklist downloads and the HTTP server, and last the database, which it
// checkpoints and closes. Each stage waits at most until ctx ends, so
// Shutdown returns soon after: a handler logs what it could not write by
// then, and the HTTP server closes the connections of requests still
// running.
// write what they hold, the WHOIS fetcher, the database maintainer and the
// HTTP server, and last the database, which it checkpoints and closes. Each
// stage waits at most until ctx ends, so Shutdown returns soon after: a
// handler logs what it could not write by then, and the HTTP server closes
// the connections of requests still running.
func (rw *RouteWatch) Shutdown(ctx context.Context) {
rw.mu.Lock()
if rw.shutdown {
@@ -210,10 +190,6 @@ func (rw *RouteWatch) Shutdown(ctx context.Context) {
rw.dbMaintainer.Stop(ctx)
}
if rw.blocklists != nil {
rw.blocklists.Stop(ctx)
}
err := rw.server.Stop(ctx)
if err != nil {
rw.logger.Error("Stopped the HTTP server before every request finished",
internal/routewatch/app_integration_internal_test.go
+2 -2
View File
@@ -466,9 +466,9 @@ func TestRouteWatchLiveFeed(t *testing.T) {
// Create streamer
s := streamer.New(logger, metricsTracker)
// The state directory holds the blocklist copies Run downloads.
// Create test config with empty state dir (no snapshot loading)
cfg := &config.Config{
StateDir: t.TempDir(),
StateDir: "",
MaxRuntime: 5 * time.Second,
EnableBatchedDatabaseWrites: true,
}
internal/routewatch/ashandler_internal_test.go
-24
View File
@@ -1,8 +1,6 @@
package routewatch
import (
"maps"
"slices"
"testing"
"git.eeqj.de/sneak/routewatch/internal/logger"
@@ -44,25 +42,3 @@ func TestASHandlerStoresTheEarliestAndLatestTimesOfANewAS(t *testing.T) {
asn.LastSeen, second.ParsedTimestamp)
}
}
// TestASHandlerRecordsNothingFromAPathWithASZero handles an update whose AS
// path holds AS 0 between two other ASes. RFC 7607 makes such a path
// malformed, so neither AS is recorded.
func TestASHandlerRecordsNothingFromAPathWithASZero(t *testing.T) {
t.Parallel()
store := newMockStore()
// Built without NewASHandler's flush goroutine, so the test flushes itself.
h := &ASHandler{db: store, logger: logger.New()}
msg := peerAnnouncement(testPeerIP, testV4Prefix)
msg.Path = decodePath(t, testASNA, 0, testASNB)
h.HandleMessage(msg)
h.flushBatch()
if len(store.ASNs) > 0 {
t.Errorf("ASes recorded = %v, want none",
slices.Sorted(maps.Keys(store.ASNs)))
}
}
internal/routewatch/asnfetcher.go
+21 -35
View File
@@ -21,16 +21,10 @@ const (
minInterval = 1 * time.Second
// maxInterval is the maximum interval after failures (backoff cap).
maxInterval = time.Minute
maxInterval = 5 * time.Minute
// A failure multiplies the interval by backoffNumerator divided by
// backoffDenominator, 1.5, up to maxInterval.
backoffNumerator = 3
backoffDenominator = 2
// A success divides the interval by successDivisor, down to
// minInterval.
successDivisor = 2
// backoffMultiplier is how much to multiply interval on failure.
backoffMultiplier = 2
// whoisStaleThreshold is how old WHOIS data can be before refresh.
whoisStaleThreshold = 30 * 24 * time.Hour // 30 days
@@ -59,8 +53,7 @@ type ASNFetcher struct {
currentInterval time.Duration
consecutiveFails int
// hourly stats tracking; now is the clock, time.Now except in tests
now func() time.Time
// hourly stats tracking
statsMu sync.Mutex
successTimes []time.Time
errorTimes []time.Time
@@ -68,17 +61,14 @@ type ASNFetcher struct {
// NewASNFetcher creates a new ASN fetcher.
func NewASNFetcher(db database.Store, logger *slog.Logger) *ASNFetcher {
logger = logger.With("component", "asn_fetcher")
return &ASNFetcher{
db: db,
whoisClient: whois.NewClient(logger),
logger: logger,
whoisClient: whois.NewClient(),
logger: logger.With("component", "asn_fetcher"),
immediateQueue: make(chan int, immediateQueueSize),
stopCh: make(chan struct{}),
done: make(chan struct{}),
currentInterval: baseInterval,
now: time.Now,
successTimes: make([]time.Time, 0),
errorTimes: make([]time.Time, 0),
}
@@ -131,7 +121,7 @@ func (f *ASNFetcher) GetStats() server.ASNFetcherStats {
f.intervalMu.Unlock()
// Prune old entries and count
cutoff := f.now().Add(-statsWindow)
cutoff := time.Now().Add(-statsWindow)
f.successTimes = pruneOldTimes(f.successTimes, cutoff)
f.errorTimes = pruneOldTimes(f.errorTimes, cutoff)
@@ -143,15 +133,16 @@ func (f *ASNFetcher) GetStats() server.ASNFetcherStats {
}
}
// pruneOldTimes drops the times at or before cutoff from the front of times,
// which holds times in the order they were recorded, and returns the rest. It
// copies nothing, so it is cheap enough to run on every append.
// pruneOldTimes removes times older than cutoff and returns the pruned slice.
func pruneOldTimes(times []time.Time, cutoff time.Time) []time.Time {
for len(times) > 0 && !times[0].After(cutoff) {
times = times[1:]
result := make([]time.Time, 0, len(times))
for _, t := range times {
if t.After(cutoff) {
result = append(result, t)
}
}
return times
return result
}
// getInterval returns the current fetch interval.
@@ -168,7 +159,7 @@ func (f *ASNFetcher) recordSuccess() {
f.consecutiveFails = 0
// Decrease interval by half, but not below minimum
newInterval := max(f.currentInterval/successDivisor, minInterval)
newInterval := max(f.currentInterval/backoffMultiplier, minInterval)
if newInterval != f.currentInterval {
f.logger.Debug("Decreased fetch interval",
@@ -179,11 +170,9 @@ func (f *ASNFetcher) recordSuccess() {
}
f.intervalMu.Unlock()
// Record success time for stats, dropping those older than statsWindow
// Record success time for stats
f.statsMu.Lock()
now := f.now()
f.successTimes = pruneOldTimes(f.successTimes, now.Add(-statsWindow))
f.successTimes = append(f.successTimes, now)
f.successTimes = append(f.successTimes, time.Now())
f.statsMu.Unlock()
}
@@ -192,9 +181,8 @@ func (f *ASNFetcher) recordFailure() {
f.intervalMu.Lock()
f.consecutiveFails++
// Exponential backoff: multiply by 1.5, capped at max
newInterval := min(
f.currentInterval*backoffNumerator/backoffDenominator, maxInterval)
// Exponential backoff: multiply by 2, capped at max
newInterval := min(f.currentInterval*backoffMultiplier, maxInterval)
if newInterval != f.currentInterval {
f.logger.Debug("Increased fetch interval due to failure",
@@ -206,11 +194,9 @@ func (f *ASNFetcher) recordFailure() {
}
f.intervalMu.Unlock()
// Record error time for stats, dropping those older than statsWindow
// Record error time for stats
f.statsMu.Lock()
now := f.now()
f.errorTimes = pruneOldTimes(f.errorTimes, now.Add(-statsWindow))
f.errorTimes = append(f.errorTimes, now)
f.errorTimes = append(f.errorTimes, time.Now())
f.statsMu.Unlock()
}
internal/routewatch/asnfetcher_internal_test.go
+4 -77
View File
@@ -120,58 +120,17 @@ func TestFetchAndUpdateKeepsTheRowWhenTheLookupFails(t *testing.T) {
"was stored", stats.NeverFetched)
}
want := baseInterval * backoffNumerator / backoffDenominator
got := f.GetStats()
if got.CurrentInterval != want || got.ConsecutiveFails != 1 {
if got.CurrentInterval != baseInterval*backoffMultiplier ||
got.ConsecutiveFails != 1 {
t.Errorf("interval %v after %d failures, want %v after 1",
got.CurrentInterval, got.ConsecutiveFails, want)
got.CurrentInterval, got.ConsecutiveFails,
baseInterval*backoffMultiplier)
}
})
}
}
// TestRecordStepsTheInterval has the fetcher fail from its base interval of
// 15 seconds, then succeed. Each failure makes the interval 1.5 times as
// long, up to one minute, and each success halves it, down to one second.
func TestRecordStepsTheInterval(t *testing.T) {
t.Parallel()
f := NewASNFetcher(newMockStore(), logger.New().Logger)
for i, want := range []time.Duration{
22500 * time.Millisecond,
33750 * time.Millisecond,
50625 * time.Millisecond,
time.Minute,
time.Minute,
} {
f.recordFailure()
if got := f.getInterval(); got != want {
t.Errorf("interval after %d failures = %v, want %v",
i+1, got, want)
}
}
for i, want := range []time.Duration{
30 * time.Second,
15 * time.Second,
7500 * time.Millisecond,
3750 * time.Millisecond,
1875 * time.Millisecond,
time.Second,
time.Second,
} {
f.recordSuccess()
if got := f.getInterval(); got != want {
t.Errorf("interval after %d successes = %v, want %v",
i+1, got, want)
}
}
}
// TestFetchAndUpdateMarksAnASWithoutReferral looks up a private-use AS, which
// IANA refers to no RIR. The AS must not be looked up again until it is
// stale, and the lookup counts as neither a success nor a failure.
@@ -244,35 +203,3 @@ func TestASNFetcherStopReturnsByItsDeadline(t *testing.T) {
t.Fatal("the fetcher goroutine did not return after the lookup ended")
}
}
// TestRecordKeepsOnlyTheStatsWindow records a success and a failure every
// second, by a clock the test sets, far longer than statsWindow, and never
// asks for the stats. The fetcher must keep only the times within the last
// statsWindow.
func TestRecordKeepsOnlyTheStatsWindow(t *testing.T) {
t.Parallel()
const (
records = 100_000
step = time.Second
want = int(statsWindow / step)
)
clock := time.Now()
f := NewASNFetcher(newMockStore(), logger.New().Logger)
f.now = func() time.Time { return clock }
for range records {
clock = clock.Add(step)
f.recordSuccess()
f.recordFailure()
}
if len(f.successTimes) != want || len(f.errorTimes) != want {
t.Errorf("%d success and %d error times kept, want %d of each, "+
"those of the last %s", len(f.successTimes), len(f.errorTimes),
want, statsWindow)
}
}
internal/routewatch/peeringhandler_internal_test.go
-37
View File
@@ -153,43 +153,6 @@ func TestPeeringHandlerLeavesATrailingASSetOutOfThePeerings(t *testing.T) {
}
}
// TestPeeringHandlerRecordsNothingFromAPathWithASZero handles AS paths that
// hold AS 0. RFC 7607 makes such a path malformed, so no peering is recorded
// from it, not even one between two other ASes of the path.
func TestPeeringHandlerRecordsNothingFromAPathWithASZero(t *testing.T) {
t.Parallel()
tests := []struct {
name string
path []any
}{
{name: "AS 0 between two ASes", path: []any{testASNA, 0, testASNB}},
{
name: "AS 0 after two ASes",
path: []any{testASNA, testASNB, 0, testASNC},
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
t.Parallel()
store := newMockStore()
h := newTestHandler(store)
h.HandleMessage(&ristypes.RISMessage{
Path: decodePath(t, tt.path...),
ParsedTimestamp: time.Now().UTC(),
})
h.ProcessPeeringsNow()
if len(store.Peerings) > 0 {
t.Errorf("peerings recorded = %v, want none",
slices.Sorted(maps.Keys(store.Peerings)))
}
})
}
}
// TestPeeringHandlerProcessesEachRunAndEmpties verifies that a processing run
// empties the path map (the swap) and that a path seen again after a run is
// recorded in the next run too.
internal/routewatch/prefixhandler.go
+34 -16
View File
@@ -45,8 +45,8 @@ const (
// PrefixHandler tracks BGP prefixes and maintains a live routing table in the
// database. Routes are added on announcement and deleted on withdrawal or on an
// announcement whose AS path is empty or holds AS 0 anywhere, and all routes of
// a peer are deleted when its BGP session goes down.
// announcement whose AS path is empty or ends in AS 0, and all routes of a peer
// are deleted when its BGP session goes down.
type PrefixHandler struct {
db database.Store
logger *logger.Logger
@@ -169,10 +169,19 @@ func (h *PrefixHandler) HandleMessage(msg *ristypes.RISMessage) {
// each peer's last update for a prefix: a message that lists a prefix as
// both withdrawn and announced keeps the route (RFC 4271 section 4.3).
// A withdrawal names only the prefix and peer: the message's AS path, and
// so its origin AS, belongs to the announced prefixes.
for _, prefix := range msg.Withdrawals {
// so its origin AS, belongs to the announced prefixes. An update whose
// prefix does not parse is logged and dropped.
for _, withdrawn := range msg.Withdrawals {
prefix, err := canonicalPrefix(withdrawn)
if err != nil {
h.logger.Error("Dropped a withdrawal whose prefix does not parse",
"prefix", withdrawn, "peer", msg.Peer, "error", err)
continue
}
h.batch = append(h.batch, prefixUpdate{
prefix: canonicalPrefix(prefix),
prefix: prefix,
peer: msg.Peer,
messageType: "withdrawal",
timestamp: timestamp,
@@ -184,9 +193,18 @@ func (h *PrefixHandler) HandleMessage(msg *ristypes.RISMessage) {
}
for _, announcement := range msg.Announcements {
for _, prefix := range announcement.Prefixes {
for _, announced := range announcement.Prefixes {
prefix, err := canonicalPrefix(announced)
if err != nil {
h.logger.Error(
"Dropped an announcement whose prefix does not parse",
"prefix", announced, "peer", msg.Peer, "error", err)
continue
}
h.batch = append(h.batch, prefixUpdate{
prefix: canonicalPrefix(prefix),
prefix: prefix,
originASN: originASN,
peer: msg.Peer,
messageType: "announcement",
@@ -373,10 +391,10 @@ func (h *PrefixHandler) flushBatch() {
// An announcement is stored when its path gives an origin, or when it
// ends in an AS set of several members: that route is stored with
// origin 0 (none). Any other update deletes the peer's route for the
// prefix: a withdrawal, or an announcement whose path is empty or
// holds AS 0 anywhere (the decoder empties such a path). Such an
// announcement replaces the peer's route, as any announcement does,
// with none (RFC 7607 treats AS 0 in a path as a withdrawal).
// prefix: a withdrawal, or an announcement whose path is empty or ends
// in AS 0. Such an announcement replaces the peer's route, as any
// announcement does, with none (RFC 7607 treats AS 0 in a path as a
// withdrawal).
if update.messageType == "announcement" &&
(update.originASN > 0 || update.endsInSet) {
// Create live route for batch upsert
@@ -560,15 +578,15 @@ func (h *PrefixHandler) writeBatch(
// the IP lookup builds when it looks a prefix up. The feed sends IPv6
// announcements compressed ("2001:db8::/32") but IPv6 withdrawals uncompressed
// ("2001:db8:0:0:0:0:0:0/32"); stored as received, a withdrawal would not match
// the route its announcement stored. A prefix that does not parse is returned
// unchanged, and the batch flush reports it.
func canonicalPrefix(prefix string) string {
// the route its announcement stored. It returns an error for a prefix that does
// not parse.
func canonicalPrefix(prefix string) (string, error) {
p, err := netip.ParsePrefix(prefix)
if err != nil {
return prefix
return "", err
}
return p.Masked().String()
return p.Masked().String(), nil
}
// parseCIDR returns the mask length and the IP version of a prefix string
internal/routewatch/prefixhandler_internal_test.go
+7 -14
View File
@@ -381,12 +381,11 @@ func TestPrefixHandlerDoesNotStoreAnAnnouncementWithAnEmptyPath(t *testing.T) {
// TestPrefixHandlerRemovesTheRouteOnAnAnnouncementWithoutAnOrigin runs, through
// the prefix handler into a real database, a peer's announcement of a prefix
// with origin AS testASNB and then the peer's announcement of the prefix with
// an AS path that gives no origin: one whose last AS is 0, an empty one, or one
// that holds AS 0 before its end, which decodes as an empty path. As any later
// announcement from the peer does, the second replaces the peer's route, here
// with none, whether it comes in a later batch or follows a withdrawal in one
// batch. The in-memory route count and prefix distribution must match the
// route table.
// an AS path that gives no origin: one whose last AS is 0, or an empty one. As
// any later announcement from the peer does, the second replaces the peer's
// route, here with none, whether it comes in a later batch or follows a
// withdrawal in one batch. The in-memory route count and prefix distribution
// must match the route table.
func TestPrefixHandlerRemovesTheRouteOnAnAnnouncementWithoutAnOrigin(
t *testing.T,
) {
@@ -422,12 +421,6 @@ func TestPrefixHandlerRemovesTheRouteOnAnAnnouncementWithoutAnOrigin(
name: "an empty path",
batch: []*ristypes.RISMessage{noOrigin(ristypes.ASPath{})},
},
{
name: "a path with AS 0 before its end",
batch: []*ristypes.RISMessage{
noOrigin(decodePath(t, testASNA, 0, testASNB)),
},
},
{
name: "a withdrawal and then a path whose last AS is 0",
batch: []*ristypes.RISMessage{withdrawal, noOrigin(endsInZero)},
@@ -447,12 +440,12 @@ func TestPrefixHandlerRemovesTheRouteOnAnAnnouncementWithoutAnOrigin(
}
}
// TestPrefixHandlerDoesNotStoreAPrefixThatDoesNotParse runs an announcement of
// TestPrefixHandlerDropsAPrefixThatDoesNotParse runs an announcement of
// "not-a-prefix" and a withdrawal of "also/bad" through the prefix handler into
// a real database. Neither is stored in the prefix tables: the prefix count in
// the stats stays 0, both as kept in memory and as counted from the tables when
// the database is opened again.
func TestPrefixHandlerDoesNotStoreAPrefixThatDoesNotParse(t *testing.T) {
func TestPrefixHandlerDropsAPrefixThatDoesNotParse(t *testing.T) {
t.Parallel()
cfg := &config.Config{StateDir: t.TempDir()}
internal/server/handlers.go
+58 -177
View File
@@ -1,7 +1,7 @@
package server
import (
"cmp"
"bytes"
"context"
"encoding/json"
"errors"
@@ -9,16 +9,13 @@ import (
"net/http"
"net/netip"
"runtime"
"slices"
"sort"
"strconv"
"strings"
"time"
"git.eeqj.de/sneak/routewatch/internal/blocklist"
"git.eeqj.de/sneak/routewatch/internal/database"
"git.eeqj.de/sneak/routewatch/internal/metrics"
"git.eeqj.de/sneak/routewatch/internal/ristypes"
"git.eeqj.de/sneak/routewatch/internal/streamer"
"git.eeqj.de/sneak/routewatch/internal/templates"
asinfo "git.eeqj.de/sneak/routewatch/pkg/asinfo"
@@ -46,15 +43,6 @@ const (
ptrLookupTimeout = 2 * time.Second
)
// The Cache-Control header of an IP lookup answer, such as
// "public, max-age=900".
const (
cacheControlHeader = "Cache-Control"
cacheControlPublic = "public"
cacheControlPrivate = "private"
cacheControlMaxAge = ", max-age="
)
// Keys of the JSON envelope the handlers and middleware write, and the status
// value of an error answer.
const (
@@ -68,18 +56,10 @@ const (
// Error messages an answer sends, in place of the error behind it where there
// is one.
const (
invalidPrefixMessage = "Invalid prefix"
hostNotResolvedMessage = "could not resolve host name"
hostNotPublicMessage = "host name has no public address"
tooManyTargetsMessage = "Too many addresses or host names"
searchQueryMissingMessage = "Missing search query"
)
// The AS name search reads the text it looks for from the URL query parameter
// asSearchQueryParam, and answers with at most maxASSearchResults matches.
const (
asSearchQueryParam = "q"
maxASSearchResults = 50
invalidPrefixMessage = "Invalid prefix"
hostNotResolvedMessage = "could not resolve host name"
hostNotPublicMessage = "host name has no public address"
tooManyTargetsMessage = "Too many addresses or host names"
)
// IP versions as database.LiveRoute.IPVersion records them.
@@ -156,10 +136,7 @@ func (s *Server) handleIndex() http.HandlerFunc {
tmpl := templates.IndexTemplate()
err := tmpl.Execute(w, indexPageData{
CSPNonce: cspNonce(r.Context()),
ClientIP: extractClientIP(r),
})
err := tmpl.Execute(w, pageData{CSPNonce: cspNonce(r.Context())})
if err != nil {
s.logger.Error("Failed to render index template", "error", err)
http.Error(w, "Internal Server Error", http.StatusInternalServerError)
@@ -173,15 +150,6 @@ type pageData struct {
CSPNonce string
}
// indexPageData is what the home page is rendered with.
type indexPageData struct {
// CSPNonce is the nonce the page's script and style blocks carry.
CSPNonce string
// ClientIP is the visitor's address, which the IP information API card
// looks up: the address /ip/ would look up for the same request.
ClientIP string
}
// writeJSONError writes a standardized JSON error response with the given
// status code and error message.
func (s *Server) writeJSONError(w http.ResponseWriter, statusCode int, message string) {
@@ -464,10 +432,10 @@ type GCStats struct {
//
//nolint:tagliatelle // field names are the existing JSON API
type StreamStats struct {
Announcements uint64 `json:"announcements"`
Withdrawals uint64 `json:"withdrawals"`
RouteChurnPerMin int `json:"route_churn_per_min"`
BGPPeerCount int `json:"bgp_peer_count"`
Announcements uint64 `json:"announcements"`
Withdrawals uint64 `json:"withdrawals"`
RouteChurnPerSec float64 `json:"route_churn_per_sec"`
BGPPeerCount int `json:"bgp_peer_count"`
}
// StatsResponse represents the API statistics response
@@ -568,7 +536,7 @@ func (s *Server) streamStats() StreamStats {
return StreamStats{
Announcements: metricsTracker.GetAnnouncementCount(),
Withdrawals: metricsTracker.GetWithdrawalCount(),
RouteChurnPerMin: metricsTracker.GetChurnPerMinute(),
RouteChurnPerSec: metricsTracker.GetChurnRate(),
BGPPeerCount: metricsTracker.GetBGPPeerCount(),
}
}
@@ -703,24 +671,9 @@ func (s *Server) handleIPLookup() http.HandlerFunc {
// IPLookupResponse is the standard response for IP/hostname lookups.
type IPLookupResponse struct {
Query string `json:"query"`
Results []IPLookupResult `json:"results"`
Errors []string `json:"errors,omitempty"`
}
// IPLookupResult is the answer for one address: its PTR names, its live route
// and that route's AS, and the blocklists that list it. IPInfo is nil when
// the address has no live route, and the answer then has no route or AS
// fields. IP hides the IPInfo field of the same name, so that an address with
// no route has it too.
type IPLookupResult struct {
*database.IPInfo
IP string `json:"ip"`
PTR []string `json:"ptr,omitempty"`
// Blocklists has one entry for each blocklist the address is on.
Blocklists []blocklist.Listing `json:"blocklists"`
Query string `json:"query"`
Results []*database.IPInfo `json:"results"`
Errors []string `json:"errors,omitempty"`
}
// handleIPInfo returns a handler that provides comprehensive IP information.
@@ -729,7 +682,7 @@ type IPLookupResult struct {
// Always returns the same response structure with PTR records for each IP.
func (s *Server) handleIPInfo() http.HandlerFunc {
return func(w http.ResponseWriter, r *http.Request) {
target, ownAddress := lookupTarget(r)
target := lookupTarget(r)
if target == "" {
s.writeJSONError(w, http.StatusBadRequest, "Could not determine IP address")
@@ -737,7 +690,7 @@ func (s *Server) handleIPInfo() http.HandlerFunc {
}
ctx := r.Context()
response := IPLookupResponse{Query: target, Results: make([]IPLookupResult, 0)}
response := IPLookupResponse{Query: target, Results: make([]*database.IPInfo, 0)}
// Collect all IPs to look up
ipsToLookup, resolveErrors, err := s.resolveLookupTargets(ctx, target)
@@ -774,10 +727,7 @@ func (s *Server) handleIPInfo() http.HandlerFunc {
}
switch {
case errors.Is(err, database.ErrNoRoute):
// The address still gets a result, without route fields.
ipInfo = nil
case errors.Is(err, database.ErrInvalidIP):
case errors.Is(err, database.ErrNoRoute), errors.Is(err, database.ErrInvalidIP):
response.Errors = append(response.Errors, ip+": "+err.Error())
continue
@@ -796,14 +746,13 @@ func (s *Server) handleIPInfo() http.HandlerFunc {
return
}
response.Results = append(response.Results, IPLookupResult{
IP: ip,
PTR: ptrs,
IPInfo: ipInfo,
Blocklists: s.blocklistListings(ip),
})
if len(ptrs) > 0 {
ipInfo.PTR = ptrs
}
if ipInfo != nil && ipInfo.NeedsWHOISRefresh {
response.Results = append(response.Results, ipInfo)
if ipInfo.NeedsWHOISRefresh {
refreshASNs[ipInfo.ASN] = true
}
}
@@ -811,40 +760,10 @@ func (s *Server) handleIPInfo() http.HandlerFunc {
// Queue WHOIS refresh for stale ASNs (non-blocking)
s.queueWHOISRefresh(refreshASNs)
s.writeIPLookupAnswer(w, response, ownAddress)
s.writeIPLookupAnswer(w, response)
}
}
// blocklistListings returns the blocklists that list ip. It returns an empty
// list, never nil, so that an answer's blocklists is [] rather than null.
func (s *Server) blocklistListings(ip string) []blocklist.Listing {
addr, err := netip.ParseAddr(ip)
if s.blocklists == nil || err != nil {
return []blocklist.Listing{}
}
return s.blocklists.Lookup(addr)
}
// setIPLookupCacheControl lets caches keep an IP lookup answer for the
// shortest blocklist interval, the soonest a blocklist copy can change. The
// answer about the caller's own address is private, so that a shared cache
// does not give it to another caller. Without blocklists it sets nothing.
func (s *Server) setIPLookupCacheControl(w http.ResponseWriter, ownAddress bool) {
if s.blocklists == nil {
return
}
visibility := cacheControlPublic
if ownAddress {
visibility = cacheControlPrivate
}
maxAgeSeconds := int(s.blocklists.ShortestInterval().Seconds())
w.Header().Set(cacheControlHeader,
visibility+cacheControlMaxAge+strconv.Itoa(maxAgeSeconds))
}
// noTargetsMessage returns the message of the 400 an IP lookup answers when
// no address is left to look up, followed by the first of resolveErrors, which
// says why, when there is one.
@@ -858,12 +777,16 @@ func noTargetsMessage(resolveErrors []string) string {
return message + ": " + resolveErrors[0]
}
// writeIPLookupAnswer sends response with its Cache-Control header.
// ownAddress says the request asked about the caller's own address.
// writeIPLookupAnswer sends response, or 404 with its first error when no
// address had a route.
func (s *Server) writeIPLookupAnswer(
w http.ResponseWriter, response IPLookupResponse, ownAddress bool,
w http.ResponseWriter, response IPLookupResponse,
) {
s.setIPLookupCacheControl(w, ownAddress)
if len(response.Results) == 0 && len(response.Errors) > 0 {
s.writeJSONError(w, http.StatusNotFound, "No routes found: "+response.Errors[0])
return
}
err := writeJSONSuccess(w, response)
if err != nil {
@@ -872,9 +795,8 @@ func (s *Server) writeIPLookupAnswer(
}
// lookupTarget returns what an IP lookup request asks for: the ip or addr URL
// parameter, or else the client's own address. The bool is true for the
// client's own address.
func lookupTarget(r *http.Request) (string, bool) {
// parameter, or else the client's own address.
func lookupTarget(r *http.Request) string {
target := chi.URLParam(r, "ip")
if target == "" {
target = chi.URLParam(r, "addr")
@@ -882,10 +804,10 @@ func lookupTarget(r *http.Request) (string, bool) {
if target == "" {
// Use client IP (RealIP middleware has already processed this)
return extractClientIP(r), true
target = extractClientIP(r)
}
return target, false
return target
}
// queueWHOISRefresh asks the WHOIS fetcher, when there is one, to refresh
@@ -1079,12 +1001,11 @@ func (s *Server) handleASDetailJSON() http.HandlerFunc {
}
}
// splitRoutesByIPVersion groups routes into the IPv4 and the IPv6 ones. Neither
// group is nil, so that a JSON answer holds [] for an empty one and not null.
// splitRoutesByIPVersion groups routes into the IPv4 and the IPv6 ones.
func splitRoutesByIPVersion(
routes []database.LiveRoute,
) ([]database.LiveRoute, []database.LiveRoute) {
ipv4Routes, ipv6Routes := []database.LiveRoute{}, []database.LiveRoute{}
var ipv4Routes, ipv6Routes []database.LiveRoute
for _, route := range routes {
if route.IPVersion == ipVersionV4 {
@@ -1097,15 +1018,24 @@ func splitRoutesByIPVersion(
return ipv4Routes, ipv6Routes
}
// sortRoutesByMaskLength sorts routes by mask length, shortest first, and
// routes with the same mask length by network address.
func sortRoutesByMaskLength(routes []database.LiveRoute) {
slices.SortFunc(routes, func(a, b database.LiveRoute) int {
prefixA, _ := netip.ParsePrefix(a.Prefix)
prefixB, _ := netip.ParsePrefix(b.Prefix)
// sortRoutesByNetwork sorts routes by network address, then by mask length.
func sortRoutesByNetwork(routes []database.LiveRoute) {
sort.Slice(routes, func(i, j int) bool {
// Parse the prefixes to compare network addresses
ipI, netI, _ := net.ParseCIDR(routes[i].Prefix)
ipJ, netJ, _ := net.ParseCIDR(routes[j].Prefix)
return cmp.Or(cmp.Compare(prefixA.Bits(), prefixB.Bits()),
prefixA.Addr().Compare(prefixB.Addr()))
// Compare by network address first
cmp := bytes.Compare(ipI.To16(), ipJ.To16())
if cmp != 0 {
return cmp < 0
}
// If network addresses are equal, compare by mask length
onesI, _ := netI.Mask.Size()
onesJ, _ := netJ.Mask.Size()
return onesI < onesJ
})
}
@@ -1175,56 +1105,6 @@ func (s *Server) handlePrefixDetailJSON() http.HandlerFunc {
}
}
// ASSearchResponse is the answer to an AS name search: the text searched for,
// the number of ASes that match it, and the first maxASSearchResults of them in
// AS number order.
type ASSearchResponse struct {
Query string `json:"query"`
Total int `json:"total"`
Results []asinfo.Info `json:"results"`
}
// handleASSearch returns a handler that searches the AS list built into
// asinfo for the text in the q parameter, without its leading and trailing
// spaces. It answers 400 when no text is left.
func (s *Server) handleASSearch() http.HandlerFunc {
return func(w http.ResponseWriter, r *http.Request) {
query := strings.TrimSpace(r.URL.Query().Get(asSearchQueryParam))
if query == "" {
s.writeJSONError(w, http.StatusBadRequest, searchQueryMissingMessage)
return
}
// AS 0 means no known origin and has no AS page, so the answer leaves
// it out. The matches come in AS number order, so AS 0 can only be the
// first. The search asks for one match more than the answer holds, to
// take its place.
results, total := asinfo.Search(query, maxASSearchResults+1)
if len(results) > 0 && results[0].ASN == ristypes.NoOrigin {
results = results[1:]
total--
}
results = results[:min(len(results), maxASSearchResults)]
// results is an empty list rather than nil when nothing matches, so
// that the answer holds [] and not null.
if results == nil {
results = []asinfo.Info{}
}
err := writeJSONSuccess(w, ASSearchResponse{
Query: query,
Total: total,
Results: results,
})
if err != nil {
s.logger.Error("Failed to encode AS search results", "error", err)
}
}
}
// handleASDetail returns a handler that serves the AS detail HTML page
func (s *Server) handleASDetail() http.HandlerFunc {
return func(w http.ResponseWriter, r *http.Request) {
@@ -1272,8 +1152,9 @@ func (s *Server) handleASDetail() http.HandlerFunc {
// Group prefixes by IP version
ipv4Prefixes, ipv6Prefixes := splitRoutesByIPVersion(prefixes)
sortRoutesByMaskLength(ipv4Prefixes)
sortRoutesByMaskLength(ipv6Prefixes)
// Sort prefixes by network address
sortRoutesByNetwork(ipv4Prefixes)
sortRoutesByNetwork(ipv6Prefixes)
// Prepare template data
data := struct {
internal/server/handlers_internal_test.go
+19 -779
View File
@@ -5,10 +5,8 @@ import (
"encoding/json"
"errors"
"html"
"net"
"net/http"
"net/http/httptest"
"net/netip"
"runtime"
"slices"
"strconv"
@@ -17,12 +15,10 @@ import (
"testing"
"time"
"git.eeqj.de/sneak/routewatch/internal/blocklist"
"git.eeqj.de/sneak/routewatch/internal/config"
"git.eeqj.de/sneak/routewatch/internal/database"
"git.eeqj.de/sneak/routewatch/internal/logger"
"git.eeqj.de/sneak/routewatch/internal/metrics"
"git.eeqj.de/sneak/routewatch/internal/ristypes"
"git.eeqj.de/sneak/routewatch/internal/streamer"
"git.eeqj.de/sneak/routewatch/pkg/asinfo"
)
@@ -30,7 +26,6 @@ import (
// The prefix and peer addresses the prefix page tests store routes for.
const (
testPrefixV4 = "192.0.2.0/24"
testPrefixV6 = "2001:db8::/32"
testPeerV4 = "198.51.100.1"
testPeerV6 = "2001:db8::1"
)
@@ -198,53 +193,8 @@ func TestStatsHandlersAnswerFromMemory(t *testing.T) {
}
}
// TestStatsAnswerCarriesChurnPerMinute checks that the stream object of the
// /api/v1/stats answer holds route_churn_per_min as a whole number, and no
// longer route_churn_per_sec, the moving average it replaces
// (https://git.eeqj.de/sneak/routewatch/issues/160).
func TestStatsAnswerCarriesChurnPerMinute(t *testing.T) {
t.Parallel()
s := newServerWithTestRoutes(t)
rec := httptest.NewRecorder()
s.handleStats()(rec,
httptest.NewRequestWithContext(t.Context(), http.MethodGet, statsPath, nil))
if rec.Code != http.StatusOK {
t.Fatalf("status %d, want %d; body %s", rec.Code, http.StatusOK, rec.Body)
}
var body struct {
Data struct {
Stream map[string]json.RawMessage `json:"stream"`
} `json:"data"`
}
err := json.Unmarshal(rec.Body.Bytes(), &body)
if err != nil {
t.Fatalf("decoding the answer: %v", err)
}
churn, ok := body.Data.Stream["route_churn_per_min"]
if !ok {
t.Fatalf("stream has no route_churn_per_min: %s", rec.Body)
}
var count int
err = json.Unmarshal(churn, &count)
if err != nil {
t.Errorf("route_churn_per_min = %s, want a whole number: %v", churn, err)
}
if _, ok := body.Data.Stream["route_churn_per_sec"]; ok {
t.Errorf("stream still has route_churn_per_sec: %s", rec.Body)
}
}
// newServerWithTestRoutes returns a server on a new database that holds a
// route for testPrefixV4 and one for testPrefixV6.
// route for testPrefixV4 and one for 2001:db8::/32.
func newServerWithTestRoutes(t *testing.T) *Server {
t.Helper()
@@ -264,7 +214,7 @@ func newServerWithTestRoutes(t *testing.T) *Server {
NextHop: testPeerV4, LastUpdated: ts,
},
{
Prefix: testPrefixV6, MaskLength: 32, IPVersion: 6,
Prefix: "2001:db8::/32", MaskLength: 32, IPVersion: 6,
OriginASN: 64501, PeerIP: testPeerV6, ASPath: []int{64501},
NextHop: testPeerV6, LastUpdated: ts,
},
@@ -461,8 +411,8 @@ func TestPrefixDetailShowsOriginNames(t *testing.T) {
// TestPagesSayOriginZeroIsNoKnownOrigin stores a route with origin 0, as the
// prefix handler stores a route whose AS path ends in an AS set of several
// members, and checks that the prefix page and the prefix length page say it
// has no known origin, without a link to AS0
// (https://git.eeqj.de/sneak/routewatch/issues/75).
// has no known origin, without a link to AS0 or the description of the
// reserved AS 0 (https://git.eeqj.de/sneak/routewatch/issues/75).
func TestPagesSayOriginZeroIsNoKnownOrigin(t *testing.T) {
t.Parallel()
@@ -482,6 +432,11 @@ func TestPagesSayOriginZeroIsNoKnownOrigin(t *testing.T) {
t.Fatalf("UpsertLiveRouteBatch: %v", err)
}
reserved, ok := asinfo.Get(0)
if !ok {
t.Fatal("AS0 is not in the AS info data")
}
s := New(db, streamer.New(logger.New(), metrics.New()), logger.New(), &config.Config{})
for _, path := range []string{"/prefix/" + testPrefixV4, prefixLengthPagePath} {
@@ -503,128 +458,9 @@ func TestPagesSayOriginZeroIsNoKnownOrigin(t *testing.T) {
if strings.Contains(page, `href="/as/0"`) {
t.Errorf("%s links to AS0", path)
}
}
}
// TestIndexShowsTheClientAddress requests the home page and checks that it has
// the IP information API card, holding the address /ip/ would look up for the
// same request: the TCP peer, or, when the peer is a trusted proxy, the client
// the proxy names in X-Forwarded-For
// (https://git.eeqj.de/sneak/routewatch/issues/156).
func TestIndexShowsTheClientAddress(t *testing.T) {
t.Parallel()
const cardHeading = "<h2>IP Information API</h2>"
// Both requests carry an X-Forwarded-For naming forwardedClient; only a
// trusted proxy's is believed.
tests := []struct {
name string
trustedProxies []netip.Prefix
peer string
want string
}{
{"direct request", nil, untrustedPeer, untrustedPeerAddr},
{
"request through a trusted proxy",
[]netip.Prefix{netip.MustParsePrefix(trustedProxyRange)},
trustedPeer, forwardedClient,
},
}
for _, tt := range tests {
// The home page reads nothing from the database, so the server has none.
s := New(nil, streamer.New(logger.New(), metrics.New()), logger.New(),
&config.Config{TrustedProxies: tt.trustedProxies})
req := httptest.NewRequestWithContext(t.Context(), http.MethodGet, "/", nil)
req.RemoteAddr = tt.peer
req.Header.Set("X-Forwarded-For", forwardedClient)
rec := httptest.NewRecorder()
s.router.ServeHTTP(rec, req)
if rec.Code != http.StatusOK {
t.Errorf("%s: status %d, want %d; body %s",
tt.name, rec.Code, http.StatusOK, rec.Body)
continue
}
page := rec.Body.String()
if !strings.Contains(page, cardHeading) {
t.Errorf("%s: the page has no IP information API card", tt.name)
}
want := `data-client-ip="` + tt.want + `"`
if !strings.Contains(page, want) {
t.Errorf("%s: the card does not hold %s", tt.name, want)
}
}
}
// TestPagesShowAgesInWords stores routes last updated about 5 minutes, 3 hours
// and 15 days ago, and checks that the AS page and the prefix pages show each
// age in words. Before the fix the ages read "5m0s ago", "3h0m0s ago" and
// "360h0m0s ago" (https://git.eeqj.de/sneak/routewatch/issues/163).
func TestPagesShowAgesInWords(t *testing.T) {
t.Parallel()
db, err := database.New(&config.Config{StateDir: t.TempDir()}, logger.New())
if err != nil {
t.Fatalf("database.New: %v", err)
}
t.Cleanup(func() { _ = db.Close() })
// Each age is well past a whole number of its unit, so the time the test
// takes to run does not change the words.
ages := []struct {
prefix string
age time.Duration
want string
}{
{testPrefixV4, 5*time.Minute + 30*time.Second, "5 minutes ago"},
{"198.51.100.0/24", 3*time.Hour + 30*time.Minute, "3 hours ago"},
{"203.0.113.0/24", 15 * 24 * time.Hour, "2 weeks ago"},
}
now := time.Now().UTC()
routes := make([]*database.LiveRoute, 0, len(ages))
for _, a := range ages {
routes = append(routes, &database.LiveRoute{
Prefix: a.prefix, MaskLength: 24, IPVersion: 4,
OriginASN: 64500, PeerIP: testPeerV4, ASPath: []int{64500},
NextHop: testPeerV4, LastUpdated: now.Add(-a.age),
})
}
err = db.UpsertLiveRouteBatch(routes)
if err != nil {
t.Fatalf("UpsertLiveRouteBatch: %v", err)
}
s := New(db, streamer.New(logger.New(), metrics.New()), logger.New(), &config.Config{})
for _, a := range ages {
cell := `<td class="age">` + a.want + `</td>`
for _, path := range []string{asPagePath, "/prefix/" + a.prefix} {
rec := httptest.NewRecorder()
s.router.ServeHTTP(rec, httptest.NewRequestWithContext(
t.Context(), http.MethodGet, path, nil))
if rec.Code != http.StatusOK {
t.Fatalf("%s: status %d, want %d; body %s",
path, rec.Code, http.StatusOK, rec.Body)
}
if !strings.Contains(rec.Body.String(), cell) {
t.Errorf("%s does not show the age of the route for %s as %q",
path, a.prefix, a.want)
}
if strings.Contains(page, html.EscapeString(reserved.Description)) {
t.Errorf("%s shows AS0's description %q", path, reserved.Description)
}
}
}
@@ -1042,15 +878,14 @@ func (failingIPLookupDB) GetIPInfoContext(
// server's router with a store whose read of one address fails with the
// driver's error. A lookup that asks for that address answers 500 with the
// JSON error body, as the AS and prefix JSON routes do; an address with no
// route answers 200. Before the fix the failed read was reported as if the
// address had no route: 404 for that address alone, and 200 beside a routed
// address (https://git.eeqj.de/sneak/routewatch/issues/109).
// route still answers 404. Before the fix the failed read was reported as if
// the address had no route: 404 for that address alone, and 200 beside a
// routed address (https://git.eeqj.de/sneak/routewatch/issues/109).
func TestIPLookupAnswers500WhenDatabaseReadFails(t *testing.T) {
t.Parallel()
s := New(failingIPLookupDB{}, streamer.New(logger.New(), metrics.New()),
logger.New(), &config.Config{})
s.resolver = &fakeResolver{}
tests := []struct {
path string
@@ -1066,7 +901,7 @@ func TestIPLookupAnswers500WhenDatabaseReadFails(t *testing.T) {
},
{
path: "/api/v1/ip/203.0.113.1",
wantCode: http.StatusOK,
wantCode: http.StatusNotFound,
},
}
@@ -1080,9 +915,7 @@ func TestIPLookupAnswers500WhenDatabaseReadFails(t *testing.T) {
tt.path, rec.Code, tt.wantCode, rec.Body)
}
if tt.wantCode != http.StatusOK {
checkJSONErrorCode(t, tt.path, rec, tt.wantCode)
}
checkJSONErrorCode(t, tt.path, rec, tt.wantCode)
}
}
@@ -1170,10 +1003,10 @@ func TestIPLookupRefusesTooManyAddresses(t *testing.T) {
}{
{
strings.Join(addrs[:maxLookupTargets], ","),
http.StatusOK, maxLookupTargets, 0,
http.StatusNotFound, maxLookupTargets, 0,
},
{strings.Join(addrs, ","), http.StatusBadRequest, 0, 0},
{"most.test", http.StatusOK, maxLookupTargets, 1},
{"most.test", http.StatusNotFound, maxLookupTargets, 1},
{"too-many.test", http.StatusBadRequest, 0, 1},
{strings.Join(names, ","), http.StatusBadRequest, 0, 0},
}
@@ -1194,9 +1027,7 @@ func TestIPLookupRefusesTooManyAddresses(t *testing.T) {
tt.target, rec.Code, tt.wantCode, rec.Body)
}
if tt.wantCode != http.StatusOK {
checkJSONErrorCode(t, tt.target, rec, tt.wantCode)
}
checkJSONErrorCode(t, tt.target, rec, tt.wantCode)
got := db.lookups.Load()
if got != tt.wantLookups {
@@ -1253,191 +1084,6 @@ func TestIPLookupKeepsRouteWhenPTRLookupDoesNotAnswer(t *testing.T) {
}
}
// addrInTestPrefixV6 is an address in testPrefixV6, which
// newServerWithTestRoutes stores a route for.
const addrInTestPrefixV6 = "2001:db8::1"
// testBlocklistInterval is fakeBlocklists' shortest interval.
const testBlocklistInterval = 15 * time.Minute
// ownAddressPath is the IP lookup of the caller's own address.
const ownAddressPath = "/ip/"
// listedUnroutedAddr is an address that fakeBlocklists lists, outside the
// routes newServerWithTestRoutes stores.
const listedUnroutedAddr = "203.0.113.1"
// fakeBlocklists has one list, which lists addrInTestPrefixV4 and
// listedUnroutedAddr and no other address.
type fakeBlocklists struct{}
func testListing() blocklist.Listing {
return blocklist.Listing{
Name: "test_list",
Description: "On the test list",
Updated: time.Date(2026, 10, 27, 4, 0, 0, 0, time.UTC),
}
}
func (fakeBlocklists) Lookup(addr netip.Addr) []blocklist.Listing {
if addr == netip.MustParseAddr(addrInTestPrefixV4) ||
addr == netip.MustParseAddr(listedUnroutedAddr) {
return []blocklist.Listing{testListing()}
}
return []blocklist.Listing{}
}
func (fakeBlocklists) ShortestInterval() time.Duration {
return testBlocklistInterval
}
// TestIPLookupReportsBlocklists looks up two routed addresses, one of them on
// a blocklist. The listed address's result names the list, says what it means
// and when it was updated. The other's result has an empty list.
func TestIPLookupReportsBlocklists(t *testing.T) {
t.Parallel()
s := newServerWithTestRoutes(t)
s.resolver = &fakeResolver{}
s.SetBlocklists(fakeBlocklists{})
rec := httptest.NewRecorder()
s.router.ServeHTTP(rec, httptest.NewRequestWithContext(t.Context(),
http.MethodGet,
"/api/v1/ip/"+addrInTestPrefixV4+","+addrInTestPrefixV6, nil))
if rec.Code != http.StatusOK {
t.Fatalf("status %d, want %d; body %s", rec.Code, http.StatusOK, rec.Body)
}
var body struct {
Data IPLookupResponse `json:"data"`
}
err := json.Unmarshal(rec.Body.Bytes(), &body)
if err != nil {
t.Fatalf("decoding the answer %q: %v", rec.Body, err)
}
results := body.Data.Results
if len(results) != 2 {
t.Fatalf("answer %s, want two results", rec.Body)
}
want := testListing()
listed := results[0].Blocklists
if len(listed) != 1 || listed[0].Name != want.Name ||
listed[0].Description != want.Description ||
!listed[0].Updated.Equal(want.Updated) {
t.Errorf("%s: blocklists %+v, want [%+v]", results[0].IP, listed, want)
}
if !strings.Contains(rec.Body.String(), `"blocklists":[]`) {
t.Errorf("answer %s, want an empty blocklists list for %s",
rec.Body, addrInTestPrefixV6)
}
}
// TestIPLookupReportsBlocklistsOfAnAddressWithNoRoute looks up a listed
// address that has no live route, alone and beside a routed address. Each
// answer is 200 and has a result for the address, with its blocklists and no
// route fields. Before the fix the lookup of that address alone answered
// 404, and beside a routed address it was only named in errors, so its
// blocklists were never reported.
func TestIPLookupReportsBlocklistsOfAnAddressWithNoRoute(t *testing.T) {
t.Parallel()
s := newServerWithTestRoutes(t)
s.resolver = &fakeResolver{}
s.SetBlocklists(fakeBlocklists{})
for _, target := range []string{
listedUnroutedAddr,
addrInTestPrefixV4 + "," + listedUnroutedAddr,
} {
rec := httptest.NewRecorder()
s.router.ServeHTTP(rec, httptest.NewRequestWithContext(t.Context(),
http.MethodGet, "/api/v1/ip/"+target, nil))
var body struct {
Data IPLookupResponse `json:"data"`
}
err := json.Unmarshal(rec.Body.Bytes(), &body)
if rec.Code != http.StatusOK || err != nil {
t.Errorf("%s: status %d (%v), want %d; body %s",
target, rec.Code, err, http.StatusOK, rec.Body)
continue
}
i := slices.IndexFunc(body.Data.Results, func(r IPLookupResult) bool {
return r.IP == listedUnroutedAddr
})
if i < 0 || len(body.Data.Errors) != 0 {
t.Errorf("%s: answer %s, want a result for %s and no errors",
target, rec.Body, listedUnroutedAddr)
continue
}
result := body.Data.Results[i]
if result.IPInfo != nil {
t.Errorf("%s: result for %s has route fields %+v",
target, listedUnroutedAddr, result.IPInfo)
}
if len(result.Blocklists) != 1 ||
result.Blocklists[0].Name != testListing().Name {
t.Errorf("%s: blocklists %+v, want [%+v]",
target, result.Blocklists, testListing())
}
}
}
// TestIPLookupAnswersMayBeCached checks the Cache-Control header of the IP
// lookup answers. Caches may keep them for the blocklists' shortest interval,
// and only the caller's own browser may keep the answer about the caller's
// own address.
func TestIPLookupAnswersMayBeCached(t *testing.T) {
t.Parallel()
s := newServerWithTestRoutes(t)
s.resolver = &fakeResolver{}
s.SetBlocklists(fakeBlocklists{})
tests := []struct {
path string
want string
}{
{"/api/v1/ip/" + addrInTestPrefixV4, "public, max-age=900"},
{ownAddressPath + addrInTestPrefixV4, "public, max-age=900"},
{ownAddressPath, "private, max-age=900"},
}
for _, tt := range tests {
req := httptest.NewRequestWithContext(t.Context(), http.MethodGet,
tt.path, nil)
req.RemoteAddr = net.JoinHostPort(addrInTestPrefixV4, "1234")
rec := httptest.NewRecorder()
s.router.ServeHTTP(rec, req)
if rec.Code != http.StatusOK {
t.Errorf("%s: status %d, want %d; body %s",
tt.path, rec.Code, http.StatusOK, rec.Body)
continue
}
if got := rec.Header().Get(cacheControlHeader); got != tt.want {
t.Errorf("%s: Cache-Control %q, want %q", tt.path, got, tt.want)
}
}
}
// The host name TestIPLookupHidesPrivateAddresses resolves to a private
// address, and that address.
const (
@@ -1520,412 +1166,6 @@ func TestIPLookupHidesPrivateAddresses(t *testing.T) {
}
}
// asSearchPath is the path of the AS name search.
const asSearchPath = "/api/v1/search/as"
// searchAS sends an AS name search with the URL query rawQuery through s's
// router, expects a 200 answer, and returns its data.
func searchAS(t *testing.T, s *Server, rawQuery string) ASSearchResponse {
t.Helper()
rec := httptest.NewRecorder()
s.router.ServeHTTP(rec, httptest.NewRequestWithContext(
t.Context(), http.MethodGet, asSearchPath+"?"+rawQuery, nil))
if rec.Code != http.StatusOK {
t.Fatalf("%s: status %d, want %d; body %s",
rawQuery, rec.Code, http.StatusOK, rec.Body)
}
var body struct {
Data ASSearchResponse `json:"data"`
}
err := json.Unmarshal(rec.Body.Bytes(), &body)
if err != nil {
t.Fatalf("%s: decoding the answer %q: %v", rawQuery, rec.Body, err)
}
return body.Data
}
// TestASSearchAnswersMatches searches for Cloudflare's handle in lower case,
// with spaces around it, and expects AS13335 among the matches. Before the fix
// there was no AS name search, and the home page's search card said it was
// coming soon (https://git.eeqj.de/sneak/routewatch/issues/151).
func TestASSearchAnswersMatches(t *testing.T) {
t.Parallel()
const query = "cloudflarenet"
data := searchAS(t, newServerWithTestRoutes(t), "q=%20"+query+"%20")
if data.Query != query {
t.Errorf("query %q, want %q", data.Query, query)
}
if data.Total != len(data.Results) {
t.Errorf("total %d, but %d results", data.Total, len(data.Results))
}
cloudflare := asinfo.Info{
ASN: 13335, Handle: "CLOUDFLARENET", Description: "Cloudflare Inc.",
}
if !slices.Contains(data.Results, cloudflare) {
t.Errorf("results %v do not hold %v", data.Results, cloudflare)
}
}
// TestASSearchAnswersEmptyListWhenNothingMatches expects a search that matches
// nothing to answer 200 with a total of 0 and an empty list, not null.
func TestASSearchAnswersEmptyListWhenNothingMatches(t *testing.T) {
t.Parallel()
data := searchAS(t, newServerWithTestRoutes(t), "q=XYZABC123456")
if data.Total != 0 || data.Results == nil || len(data.Results) != 0 {
t.Errorf("total %d and results %#v, want 0 and an empty list",
data.Total, data.Results)
}
}
// TestASSearchAnswers400ForEmptyQuery expects a search with no q, an empty q
// or a q of only spaces to answer 400 with the JSON error body.
func TestASSearchAnswers400ForEmptyQuery(t *testing.T) {
t.Parallel()
s := newServerWithTestRoutes(t)
for _, path := range []string{
asSearchPath, asSearchPath + "?q=", asSearchPath + "?q=%20%20",
} {
rec := httptest.NewRecorder()
s.router.ServeHTTP(rec, httptest.NewRequestWithContext(
t.Context(), http.MethodGet, path, nil))
if rec.Code != http.StatusBadRequest {
t.Errorf("%s: status %d, want %d; body %s",
path, rec.Code, http.StatusBadRequest, rec.Body)
}
checkJSONErrorCode(t, path, rec, http.StatusBadRequest)
checkJSONErrorMessage(t, path, rec, searchQueryMissingMessage)
}
}
// TestASSearchShowsFirstMatchesAndTotal searches for a word with more matches
// than an answer holds, and expects the first maxASSearchResults of them, in
// AS number order, and the number of all of them.
func TestASSearchShowsFirstMatchesAndTotal(t *testing.T) {
t.Parallel()
const query = "university"
all, _ := asinfo.Search(query, asinfo.Total())
if len(all) <= maxASSearchResults {
t.Fatalf("Search(%q) has %d matches; the test needs more than %d",
query, len(all), maxASSearchResults)
}
data := searchAS(t, newServerWithTestRoutes(t), "q="+query)
if data.Total != len(all) {
t.Errorf("total %d, want %d", data.Total, len(all))
}
if !slices.Equal(data.Results, all[:maxASSearchResults]) {
t.Errorf("results %v, want the first %d matches %v",
data.Results, maxASSearchResults, all[:maxASSearchResults])
}
}
// The AS numbers the AS detail tests ask for. storedASN originates the route
// for testPrefixV4 and is in the database. routeOnlyASN originates the route
// for testPrefixV6, but the database has no record of the AS itself, as when
// a route is stored before its AS. Both are documentation AS numbers, which
// the AS list built into asinfo does not hold. builtInOnlyASN is in that list
// but not in the database. unknownASN is in neither.
const (
storedASN = 64500
routeOnlyASN = 64501
builtInOnlyASN = 15169
unknownASN = 4200000000
)
// zeroDate is how a page writes a time that was never set.
const zeroDate = "0001-01-01"
// firstSeenCard is the label of the AS page's first-seen date.
const firstSeenCard = `<div class="info-label">First Seen</div>`
// newServerWithStoredAS returns newServerWithTestRoutes's server with storedASN
// added to the database. It fails the test when the AS list built into asinfo
// lacks builtInOnlyASN or holds unknownASN.
func newServerWithStoredAS(t *testing.T) *Server {
t.Helper()
if _, ok := asinfo.Get(builtInOnlyASN); !ok {
t.Fatalf("AS%d is not in the AS info data", builtInOnlyASN)
}
if _, ok := asinfo.Get(unknownASN); ok {
t.Fatalf("AS%d is in the AS info data", unknownASN)
}
s := newServerWithTestRoutes(t)
_, err := s.db.GetOrCreateASN(storedASN, time.Date(2026, 1, 2, 3, 4, 5, 0, time.UTC))
if err != nil {
t.Fatalf("GetOrCreateASN: %v", err)
}
return s
}
// TestASDetailJSONAnswersBuiltInAS asks /api/v1/as/ for each of the ASes above.
// All but unknownASN answer 200 with the handle and description the built-in
// AS list has for them, if any, and their prefixes, as lists even when empty.
// Only storedASN has a first_seen time. unknownASN answers 404. Before the fix
// builtInOnlyASN and routeOnlyASN answered 404 too
// (https://git.eeqj.de/sneak/routewatch/issues/151).
func TestASDetailJSONAnswersBuiltInAS(t *testing.T) {
t.Parallel()
s := newServerWithStoredAS(t)
tests := []struct {
asn int
wantCode int
wantPrefixes []string
wantFirstSeen bool
}{
{storedASN, http.StatusOK, []string{testPrefixV4}, true},
{routeOnlyASN, http.StatusOK, []string{testPrefixV6}, false},
{builtInOnlyASN, http.StatusOK, nil, false},
{unknownASN, http.StatusNotFound, nil, false},
}
for _, tt := range tests {
path := "/api/v1/as/" + strconv.Itoa(tt.asn)
rec := httptest.NewRecorder()
s.router.ServeHTTP(rec, httptest.NewRequestWithContext(
t.Context(), http.MethodGet, path, nil))
if rec.Code != tt.wantCode {
t.Errorf("%s: status %d, want %d; body %s",
path, rec.Code, tt.wantCode, rec.Body)
continue
}
if tt.wantCode != http.StatusOK {
checkJSONErrorCode(t, path, rec, tt.wantCode)
continue
}
//nolint:tagliatelle // field names are the existing JSON API
var body struct {
Data struct {
ASN database.ASN `json:"asn"`
IPv4Prefixes []database.LiveRoute `json:"ipv4_prefixes"`
IPv6Prefixes []database.LiveRoute `json:"ipv6_prefixes"`
} `json:"data"`
}
err := json.Unmarshal(rec.Body.Bytes(), &body)
if err != nil {
t.Errorf("%s: decoding the answer %q: %v", path, rec.Body, err)
continue
}
handle, description := asinfo.GetHandle(tt.asn), asinfo.GetDescription(tt.asn)
got := body.Data.ASN
if got.ASN != tt.asn || got.Handle != handle ||
got.Description != description {
t.Errorf("%s: AS %d %q %q, want %d %q %q", path, got.ASN,
got.Handle, got.Description, tt.asn, handle, description)
}
if body.Data.IPv4Prefixes == nil || body.Data.IPv6Prefixes == nil {
t.Errorf("%s: a prefix list is null; body %s", path, rec.Body)
}
routes := slices.Concat(body.Data.IPv4Prefixes, body.Data.IPv6Prefixes)
gotPrefixes := make([]string, 0, len(routes))
for _, route := range routes {
gotPrefixes = append(gotPrefixes, route.Prefix)
}
if !slices.Equal(gotPrefixes, tt.wantPrefixes) {
t.Errorf("%s: prefixes %v, want %v", path, gotPrefixes, tt.wantPrefixes)
}
gotFirstSeen := strings.Contains(rec.Body.String(), `"first_seen"`)
if gotFirstSeen != tt.wantFirstSeen {
t.Errorf("%s: has first_seen %t, want %t; body %s",
path, gotFirstSeen, tt.wantFirstSeen, rec.Body)
}
}
}
// TestASDetailPageShowsBuiltInAS asks /as/ for each of the ASes above. All but
// unknownASN answer 200 with the handle and description the built-in AS list
// has for them, if any. storedASN and routeOnlyASN list their prefixes, and
// builtInOnlyASN says the AS announces none. Only storedASN shows a first-seen
// date, and no page shows a date that was never set. unknownASN answers 404.
// Before the fix builtInOnlyASN and routeOnlyASN answered 404 too
// (https://git.eeqj.de/sneak/routewatch/issues/151).
func TestASDetailPageShowsBuiltInAS(t *testing.T) {
t.Parallel()
s := newServerWithStoredAS(t)
tests := []struct {
asn int
wantCode int
wantText string
wantFirstSeen bool
}{
{storedASN, http.StatusOK, testPrefixV4, true},
{routeOnlyASN, http.StatusOK, testPrefixV6, false},
{builtInOnlyASN, http.StatusOK, "No prefixes announced by this AS", false},
{unknownASN, http.StatusNotFound, "", false},
}
for _, tt := range tests {
path := "/as/" + strconv.Itoa(tt.asn)
rec := httptest.NewRecorder()
s.router.ServeHTTP(rec, httptest.NewRequestWithContext(
t.Context(), http.MethodGet, path, nil))
if rec.Code != tt.wantCode {
t.Errorf("%s: status %d, want %d; body %s",
path, rec.Code, tt.wantCode, rec.Body)
continue
}
if tt.wantCode != http.StatusOK {
continue
}
page := rec.Body.String()
for _, want := range []string{
html.EscapeString(asinfo.GetHandle(tt.asn)),
html.EscapeString(asinfo.GetDescription(tt.asn)),
tt.wantText,
} {
if !strings.Contains(page, want) {
t.Errorf("%s: page does not show %q", path, want)
}
}
if strings.Contains(page, zeroDate) {
t.Errorf("%s: page shows the date %s", path, zeroDate)
}
gotFirstSeen := strings.Contains(page, firstSeenCard)
if gotFirstSeen != tt.wantFirstSeen {
t.Errorf("%s: shows a first-seen date %t, want %t",
path, gotFirstSeen, tt.wantFirstSeen)
}
}
}
// TestSortRoutesByMaskLength expects the AS page's prefix lists in mask length
// order, shortest first, and in address order within one mask length. Before
// the fix they were in address order, so 10.1.0.0/16 came before 11.0.0.0/8
// (https://git.eeqj.de/sneak/routewatch/issues/153).
func TestSortRoutesByMaskLength(t *testing.T) {
t.Parallel()
tests := []struct {
name string
prefixes []string
want []string
}{
{
name: "IPv4",
prefixes: []string{
"100.64.1.0/24", "11.0.0.0/8", "10.1.0.0/16",
"20.0.1.0/24", "172.16.0.0/12", "9.0.0.0/8",
},
want: []string{
"9.0.0.0/8", "11.0.0.0/8", "172.16.0.0/12",
"10.1.0.0/16", "20.0.1.0/24", "100.64.1.0/24",
},
},
{
name: "IPv6",
prefixes: []string{
"2001:db8:1::/48", "2001:db8:8000::/33", "2001:db8:0:1::/64",
"2001:db8::/48", "2001:db8::/33",
},
want: []string{
"2001:db8::/33", "2001:db8:8000::/33", "2001:db8::/48",
"2001:db8:1::/48", "2001:db8:0:1::/64",
},
},
}
for _, tt := range tests {
routes := make([]database.LiveRoute, 0, len(tt.prefixes))
for _, prefix := range tt.prefixes {
routes = append(routes, database.LiveRoute{Prefix: prefix})
}
sortRoutesByMaskLength(routes)
got := make([]string, 0, len(routes))
for _, route := range routes {
got = append(got, route.Prefix)
}
if !slices.Equal(got, tt.want) {
t.Errorf("%s: order %v, want %v", tt.name, got, tt.want)
}
}
}
// TestNoOriginHasNoASAnswers stores a route with no known origin, which has
// origin AS 0 (https://git.eeqj.de/sneak/routewatch/issues/75). It expects
// /as/0 and /api/v1/as/0 to answer 404.
func TestNoOriginHasNoASAnswers(t *testing.T) {
t.Parallel()
const noOriginPrefix = "203.0.113.0/24"
s := newServerWithTestRoutes(t)
err := s.db.UpsertLiveRouteBatch([]*database.LiveRoute{{
Prefix: noOriginPrefix, MaskLength: 24, IPVersion: 4,
OriginASN: ristypes.NoOrigin, PeerIP: testPeerV4,
ASPath: []int{64500, 64501, 64502}, NextHop: testPeerV4,
LastUpdated: time.Date(2026, 1, 2, 3, 4, 5, 0, time.UTC),
}})
if err != nil {
t.Fatalf("UpsertLiveRouteBatch: %v", err)
}
asn := strconv.Itoa(ristypes.NoOrigin)
for _, path := range []string{"/as/" + asn, "/api/v1/as/" + asn} {
rec := httptest.NewRecorder()
s.router.ServeHTTP(rec, httptest.NewRequestWithContext(
t.Context(), http.MethodGet, path, nil))
if rec.Code != http.StatusNotFound {
t.Errorf("%s: status %d, want %d; body %s",
path, rec.Code, http.StatusNotFound, rec.Body)
}
}
}
// settledGoroutineCount lets transient goroutines finish, then reports the
// current count.
func settledGoroutineCount() int {
internal/server/middleware_internal_test.go
+4 -5
View File
@@ -11,12 +11,11 @@ import (
)
// The addresses the forwarding tests use: the client a proxy names in
// X-Forwarded-For, the range of trusted proxies, a peer inside it, and a peer
// outside it, which lies in testPrefixV4 so that an IP lookup of it finds a
// route.
// X-Forwarded-For, a peer inside the trusted range of
// TestRealIPMiddlewareTrustsOnlyConfiguredProxies, and a peer outside it,
// which lies in testPrefixV4 so that an IP lookup of it finds a route.
const (
forwardedClient = "203.0.113.9"
trustedProxyRange = "10.0.0.0/8"
trustedPeer = "10.0.0.2:1234"
untrustedPeerAddr = "192.0.2.1"
untrustedPeer = untrustedPeerAddr + ":1234"
@@ -190,7 +189,7 @@ func TestForwardedForFromUntrustedPeerIsIgnored(t *testing.T) {
func TestRealIPMiddlewareTrustsOnlyConfiguredProxies(t *testing.T) {
t.Parallel()
trusted := []netip.Prefix{netip.MustParsePrefix(trustedProxyRange)}
trusted := []netip.Prefix{netip.MustParsePrefix("10.0.0.0/8")}
tests := []struct {
name string
internal/server/routes.go
-1
View File
@@ -87,7 +87,6 @@ func (s *Server) setupRoutes() {
r.Get("/ip/{ip}", s.handleIPLookup())
r.Get("/as/{asn}", s.handleASDetailJSON())
r.Get("/prefix/{prefix}/{len}", s.handlePrefixDetailJSON())
r.Get("/search/as", s.handleASSearch())
})
s.router = r
internal/server/server.go
+1 -20
View File
@@ -1,8 +1,4 @@
// Package server serves RouteWatch's web pages and JSON API:
// - the home page, the status figures and the health check
// - the AS, prefix and prefix length pages, and the JSON answers behind them
// - the IP lookups
// - the AS name search
// Package server provides HTTP endpoints for status monitoring and statistics
package server
import (
@@ -15,7 +11,6 @@ import (
"strconv"
"time"
"git.eeqj.de/sneak/routewatch/internal/blocklist"
"git.eeqj.de/sneak/routewatch/internal/config"
"git.eeqj.de/sneak/routewatch/internal/database"
"git.eeqj.de/sneak/routewatch/internal/logger"
@@ -37,13 +32,6 @@ type ASNFetcher interface {
GetStats() ASNFetcherStats
}
// Blocklists tells which public blocklists list an address, from copies it
// keeps in memory, as *blocklist.Blocklists does.
type Blocklists interface {
Lookup(addr netip.Addr) []blocklist.Listing
ShortestInterval() time.Duration
}
// dnsResolver looks up the addresses of host names and the names of addresses
// in DNS, as *net.Resolver does.
type dnsResolver interface {
@@ -60,7 +48,6 @@ type Server struct {
port int
srv *http.Server
asnFetcher ASNFetcher
blocklists Blocklists
// trustedProxies are the reverse proxies whose X-Forwarded-For header
// RealIPMiddleware believes
@@ -157,9 +144,3 @@ func (s *Server) Stop(ctx context.Context) error {
func (s *Server) SetASNFetcher(fetcher ASNFetcher) {
s.asnFetcher = fetcher
}
// SetBlocklists sets the blocklists the IP lookups report. Call it before
// Start: the handlers read the blocklists without a lock.
func (s *Server) SetBlocklists(blocklists Blocklists) {
s.blocklists = blocklists
}
internal/templates/as_detail.html
+8 -30
View File
@@ -126,26 +126,6 @@
margin: 0;
color: #2c3e50;
}
/* A click on a prefix list's heading folds or unfolds the list. Not
every browser shows its own marker on a flex element, so the page
hides it and draws a triangle before the heading. The triangle
points down while the list is open. */
summary.prefix-header {
cursor: pointer;
list-style: none;
}
summary.prefix-header::-webkit-details-marker {
display: none;
}
summary.prefix-header h2::before {
content: "\25B8";
display: inline-block;
margin-right: 10px;
transition: transform 0.2s;
}
details[open] > summary.prefix-header h2::before {
transform: rotate(90deg);
}
.prefix-count {
background: #e74c3c;
color: white;
@@ -253,20 +233,18 @@
<div class="info-label">Peer ASNs</div>
<div class="info-value">{{.PeerCount}}</div>
</div>
{{if not .ASN.FirstSeen.IsZero}}
<div class="info-card">
<div class="info-label">First Seen</div>
<div class="info-value">{{.ASN.FirstSeen.Format "2006-01-02"}}</div>
</div>
{{end}}
</div>
{{if .IPv4Prefixes}}
<details class="prefix-section" open>
<summary class="prefix-header">
<div class="prefix-section">
<div class="prefix-header">
<h2>IPv4 Prefixes</h2>
<span class="prefix-count">{{.IPv4Count}}</span>
</summary>
</div>
<table class="prefix-table">
<thead>
<tr>
@@ -287,15 +265,15 @@
{{end}}
</tbody>
</table>
</details>
</div>
{{end}}
{{if .IPv6Prefixes}}
<details class="prefix-section" open>
<summary class="prefix-header">
<div class="prefix-section">
<div class="prefix-header">
<h2>IPv6 Prefixes</h2>
<span class="prefix-count">{{.IPv6Count}}</span>
</summary>
</div>
<table class="prefix-table">
<thead>
<tr>
@@ -316,7 +294,7 @@
{{end}}
</tbody>
</table>
</details>
</div>
{{end}}
{{if eq .TotalCount 0}}
internal/templates/index.html
+19 -201
View File
@@ -107,15 +107,9 @@
.stat-card.connected .stat-value {
color: #27ae60;
}
.stat-card.disconnected .stat-value,
.stale {
.stat-card.disconnected .stat-value {
color: #e74c3c;
}
/* The card has no room beside "Connected", so the spinner and "stale" go under it */
#freshness {
display: block;
font-size: 20px;
}
/* Search section */
.search-section {
@@ -180,38 +174,6 @@
#asname-results .search-message {
font-size: 13px;
}
#asname-results .error {
color: #c00;
}
.asname-list {
list-style: none;
margin: 12px 0 0 0;
padding: 0;
max-height: 400px;
overflow-y: auto;
}
.asname-list li {
border-bottom: 1px solid #eee;
}
.asname-list a {
display: block;
padding: 8px 4px;
color: #2c3e50;
text-decoration: none;
font-size: 14px;
}
.asname-list a:hover {
background: #f5f5f5;
}
.asname-list .asn {
font-weight: 600;
color: #3498db;
}
.asname-list .description {
display: block;
color: #7f8c8d;
font-size: 12px;
}
/* IP Lookup result */
.ip-result {
@@ -239,8 +201,7 @@
cursor: pointer;
font-size: 12px;
}
.ip-result pre,
#ip-api pre {
.ip-result pre {
background: #2c3e50;
color: #ecf0f1;
padding: 16px;
@@ -252,30 +213,18 @@
max-height: 400px;
overflow-y: auto;
}
.ip-result .error,
#ip-api .error {
.ip-result .error {
background: #fee;
color: #c00;
padding: 12px;
border-radius: 6px;
font-size: 14px;
}
.ip-result .loading,
#ip-api .loading {
.ip-result .loading {
color: #7f8c8d;
font-style: italic;
}
/* IP information API card: the command, then its answer */
#ip-api p {
margin: 0;
color: #7f8c8d;
font-size: 14px;
}
#ip-api pre {
margin-top: 12px;
}
/* Footer */
.footer {
margin-top: 40px;
@@ -316,10 +265,7 @@
<main class="main-content">
<div class="stats-grid">
<div class="stat-card" id="status-card">
<div class="stat-value">
<span id="stat-status">-</span>
<span id="freshness"></span>
</div>
<div class="stat-value" id="stat-status">-</div>
<div class="stat-label">Status</div>
</div>
<div class="stat-card">
@@ -360,7 +306,7 @@
<input type="text" id="asname-input" placeholder="e.g., Google, Cloudflare" autocomplete="off">
<button type="submit">Search</button>
</form>
<p class="search-hint">Search for autonomous systems by organization name or handle</p>
<p class="search-hint">Search for autonomous systems by organization name</p>
<div id="asname-results"></div>
</div>
@@ -380,13 +326,6 @@
</div>
</div>
</div>
<div class="search-card" id="ip-api">
<h2>IP Information API</h2>
<p>The API answers what RouteWatch knows about an IP address, such as its AS number and country. <code>/ip/</code> with no address answers for the caller's own address.</p>
<pre id="ip-api-command" data-client-ip="{{.ClientIP}}"></pre>
<pre id="ip-api-answer" class="loading">Loading...</pre>
</div>
</main>
<footer class="footer">
@@ -398,44 +337,6 @@
</footer>
<script nonce="{{.CSPNonce}}">
// The stats are fetched every REFRESH_PERIOD_MS. Next to the status, a
// spinner turns one frame each time an answer counts more messages
// than the one before. After STALE_AFTER_REFRESHES refreshes without
// such an answer, the spinner gives way to "stale" and the time since.
const REFRESH_PERIOD_MS = 5000;
const STALE_AFTER_REFRESHES = 3;
const MS_PER_SECOND = 1000;
const SPINNER_FRAMES = ['⠋', '⠙', '⠹', '⠸', '⠼', '⠴', '⠦', '⠧', '⠇', '⠏'];
const SPINNER_COLOURS = ['#e74c3c', '#e67e22', '#f1c40f', '#27ae60', '#3498db', '#9b59b6'];
let spinnerFrame = 0;
let lastMessageCount = null;
let lastFreshAt = Date.now();
// Turn the spinner when the message count has risen since the last answer.
function noteMessageCount(count) {
if (lastMessageCount !== null && count > lastMessageCount) {
spinnerFrame = (spinnerFrame + 1) % SPINNER_FRAMES.length;
lastFreshAt = Date.now();
}
lastMessageCount = count;
showFreshness();
}
function showFreshness() {
const freshness = document.getElementById('freshness');
const sinceFreshMs = Date.now() - lastFreshAt;
const stale = sinceFreshMs >= STALE_AFTER_REFRESHES * REFRESH_PERIOD_MS;
freshness.classList.toggle('stale', stale);
if (stale) {
freshness.textContent = 'stale for ' + Math.floor(sinceFreshMs / MS_PER_SECOND) + 's';
freshness.style.color = '';
return;
}
freshness.textContent = SPINNER_FRAMES[spinnerFrame];
freshness.style.color = SPINNER_COLOURS[spinnerFrame % SPINNER_COLOURS.length];
}
function formatNumber(num) {
if (num >= 1000000) {
return (num / 1000000).toFixed(1) + 'M';
@@ -457,7 +358,6 @@
const statusEl = document.getElementById('stat-status');
statusEl.textContent = data.connected ? 'Connected' : 'Disconnected';
statusCard.className = 'stat-card ' + (data.connected ? 'connected' : 'disconnected');
noteMessageCount(data.total_messages);
document.getElementById('stat-routes').textContent = formatNumber(data.live_routes);
document.getElementById('stat-asns').textContent = formatNumber(data.asns);
@@ -468,7 +368,7 @@
}
const totalUpdates = data.ipv4_updates_per_sec + data.ipv6_updates_per_sec;
document.getElementById('stat-updates').textContent = Math.round(totalUpdates).toLocaleString();
document.getElementById('stat-updates').textContent = totalUpdates.toFixed(1);
})
.catch(() => {
document.getElementById('stat-status').textContent = 'Error';
@@ -487,82 +387,24 @@
}
});
// AS name search. The handles and descriptions come from outside data,
// so the results are built with textContent, never innerHTML.
const asNameResults = document.getElementById('asname-results');
// Replace the search results with one line of text.
function showASNameMessage(text, className) {
const line = document.createElement('p');
line.className = className;
line.textContent = text;
asNameResults.replaceChildren(line);
}
// List the matches of a search answer, each as a link to its AS page,
// and say how many more matched than the answer holds.
function showASNameMatches(data) {
if (data.total === 0) {
showASNameMessage('No AS matches "' + data.query + '".', 'search-message');
return;
}
const list = document.createElement('ul');
list.className = 'asname-list';
for (const match of data.results) {
const asn = document.createElement('span');
asn.className = 'asn';
asn.textContent = 'AS' + match.asn;
const description = document.createElement('span');
description.className = 'description';
description.textContent = match.description;
const link = document.createElement('a');
link.href = '/as/' + match.asn;
link.append(asn, ' ' + match.handle, description);
const item = document.createElement('li');
item.append(link);
list.append(item);
}
asNameResults.replaceChildren(list);
const more = data.total - data.results.length;
if (more > 0) {
const line = document.createElement('p');
line.className = 'search-message';
line.textContent = more.toLocaleString() + ' more matched. Narrow the search to see them.';
asNameResults.append(line);
}
}
// AS name search
document.getElementById('asname-form').addEventListener('submit', function(e) {
e.preventDefault();
const query = document.getElementById('asname-input').value.trim();
if (!query) return;
// A number, or AS and a number, opens that AS's page
const asn = query.replace(/^AS/i, '');
if (/^\d+$/.test(asn)) {
window.location.href = '/as/' + asn;
const resultsDiv = document.getElementById('asname-results');
resultsDiv.innerHTML = '<p class="loading">Searching...</p>';
// Use a simple client-side search against the asinfo data
// For now, redirect to AS page if it looks like an ASN
if (/^\d+$/.test(query)) {
window.location.href = '/as/' + query;
return;
}
showASNameMessage('Searching...', 'loading');
fetch('/api/v1/search/as?q=' + encodeURIComponent(query))
.then(response => response.json())
.then(response => {
if (response.status === 'error') {
showASNameMessage('Error: ' + response.error.msg, 'error');
} else {
showASNameMatches(response.data);
}
})
.catch(error => {
showASNameMessage('Error: ' + error.message, 'error');
});
// Show a message that server-side search is coming
resultsDiv.innerHTML = '<p class="search-message">AS name search coming soon. For now, try an AS number.</p>';
});
// IP lookup
@@ -604,33 +446,9 @@
document.getElementById('ip-input').value = '';
});
// IP information API card: show the curl command that asks /ip/ about
// the visitor's own address, then make that same request and show the
// JSON it answers, an error answer included. The answer holds outside
// data, so it is set with textContent, never innerHTML.
const ipAPICommand = document.getElementById('ip-api-command');
const ipAPIAnswer = document.getElementById('ip-api-answer');
const ipAPIURL = window.location.origin + '/ip/' + ipAPICommand.dataset.clientIp;
ipAPICommand.textContent = 'curl ' + ipAPIURL;
fetch(ipAPIURL)
.then(response => response.json())
.then(answer => {
ipAPIAnswer.className = '';
ipAPIAnswer.textContent = JSON.stringify(answer, null, 2);
})
.catch(error => {
ipAPIAnswer.className = 'error';
ipAPIAnswer.textContent = 'Error: ' + error.message;
});
// Initial load and refresh stats every REFRESH_PERIOD_MS. The freshness
// check runs every second on its own, so a request that never returns
// still turns the spinner into "stale".
// Initial load and refresh stats every 5 seconds
updateStats();
setInterval(updateStats, REFRESH_PERIOD_MS);
setInterval(showFreshness, MS_PER_SECOND);
setInterval(updateStats, 5000);
</script>
</body>
</html>
internal/templates/status.html
+12 -58
View File
@@ -126,8 +126,7 @@
.connected {
color: #22c55e;
}
.disconnected,
.stale {
.disconnected {
color: #ef4444;
}
.error {
@@ -181,10 +180,7 @@
<h2>RouteWatch</h2>
<div class="metric">
<span class="metric-label">Status</span>
<span>
<span class="metric-value" id="connected">-</span>
<span class="metric-value" id="freshness"></span>
</span>
<span class="metric-value" id="connected">-</span>
</div>
<div class="metric">
<span class="metric-label">Uptime</span>
@@ -235,8 +231,8 @@
<span class="metric-value" id="withdrawals">-</span>
</div>
<div class="metric">
<span class="metric-label">Route Churn/min</span>
<span class="metric-value" id="route_churn_per_min">-</span>
<span class="metric-label">Route Churn/sec</span>
<span class="metric-value" id="route_churn_per_sec">-</span>
</div>
<div class="metric">
<span class="metric-label">Total Data</span>
@@ -390,44 +386,6 @@
</div>
<script nonce="{{.CSPNonce}}">
// The stats are fetched every REFRESH_PERIOD_MS. Next to the status, a
// spinner turns one frame each time an answer counts more messages
// than the one before. After STALE_AFTER_REFRESHES refreshes without
// such an answer, the spinner gives way to "stale" and the time since.
const REFRESH_PERIOD_MS = 2000;
const STALE_AFTER_REFRESHES = 3;
const MS_PER_SECOND = 1000;
const SPINNER_FRAMES = ['⠋', '⠙', '⠹', '⠸', '⠼', '⠴', '⠦', '⠧', '⠇', '⠏'];
const SPINNER_COLOURS = ['#e74c3c', '#e67e22', '#f1c40f', '#27ae60', '#3498db', '#9b59b6'];
let spinnerFrame = 0;
let lastMessageCount = null;
let lastFreshAt = Date.now();
// Turn the spinner when the message count has risen since the last answer.
function noteMessageCount(count) {
if (lastMessageCount !== null && count > lastMessageCount) {
spinnerFrame = (spinnerFrame + 1) % SPINNER_FRAMES.length;
lastFreshAt = Date.now();
}
lastMessageCount = count;
showFreshness();
}
function showFreshness() {
const freshness = document.getElementById('freshness');
const sinceFreshMs = Date.now() - lastFreshAt;
const stale = sinceFreshMs >= STALE_AFTER_REFRESHES * REFRESH_PERIOD_MS;
freshness.classList.toggle('stale', stale);
if (stale) {
freshness.textContent = 'stale for ' + Math.floor(sinceFreshMs / MS_PER_SECOND) + 's';
freshness.style.color = '';
return;
}
freshness.textContent = SPINNER_FRAMES[spinnerFrame];
freshness.style.color = SPINNER_COLOURS[spinnerFrame % SPINNER_COLOURS.length];
}
function formatBytes(bytes) {
if (bytes === 0) return '0 B';
const k = 1024;
@@ -553,7 +511,7 @@
document.getElementById('messages_per_sec').textContent = '-';
document.getElementById('announcements').textContent = '-';
document.getElementById('withdrawals').textContent = '-';
document.getElementById('route_churn_per_min').textContent = '-';
document.getElementById('route_churn_per_sec').textContent = '-';
document.getElementById('total_wire_bytes').textContent = '-';
document.getElementById('wire_mbits_per_sec').textContent = '-';
document.getElementById('gc_num').textContent = '-';
@@ -611,8 +569,7 @@
const connectedEl = document.getElementById('connected');
connectedEl.textContent = data.connected ? 'Connected' : 'Disconnected';
connectedEl.className = 'metric-value ' + (data.connected ? 'connected' : 'disconnected');
noteMessageCount(data.total_messages);
// Update all metrics
document.getElementById('uptime').textContent = data.uptime;
document.getElementById('go_version').textContent = data.go_version;
@@ -621,7 +578,7 @@
document.getElementById('connection_duration').textContent = data.connection_duration;
document.getElementById('reconnect_count').textContent = formatNumber(data.reconnect_count);
document.getElementById('total_messages').textContent = formatNumber(data.total_messages);
document.getElementById('messages_per_sec').textContent = formatNumber(Math.round(data.messages_per_sec));
document.getElementById('messages_per_sec').textContent = data.messages_per_sec.toFixed(1);
document.getElementById('total_wire_bytes').textContent = formatBytes(data.total_wire_bytes);
document.getElementById('wire_mbits_per_sec').textContent = data.wire_mbits_per_sec.toFixed(2) + ' Mbps';
document.getElementById('asns').textContent = formatNumber(data.asns);
@@ -634,8 +591,8 @@
document.getElementById('live_routes').textContent = formatNumber(data.live_routes);
document.getElementById('ipv4_routes').textContent = formatNumber(data.ipv4_routes);
document.getElementById('ipv6_routes').textContent = formatNumber(data.ipv6_routes);
document.getElementById('ipv4_updates_per_sec').textContent = formatNumber(Math.round(data.ipv4_updates_per_sec));
document.getElementById('ipv6_updates_per_sec').textContent = formatNumber(Math.round(data.ipv6_updates_per_sec));
document.getElementById('ipv4_updates_per_sec').textContent = data.ipv4_updates_per_sec.toFixed(1);
document.getElementById('ipv6_updates_per_sec').textContent = data.ipv6_updates_per_sec.toFixed(1);
document.getElementById('oldest_route').textContent = formatRelativeTime(data.oldest_route);
document.getElementById('newest_route').textContent = formatRelativeTime(data.newest_route);
@@ -644,7 +601,7 @@
document.getElementById('bgp_peer_count').textContent = formatNumber(data.stream.bgp_peer_count);
document.getElementById('announcements').textContent = formatNumber(data.stream.announcements);
document.getElementById('withdrawals').textContent = formatNumber(data.stream.withdrawals);
document.getElementById('route_churn_per_min').textContent = formatNumber(data.stream.route_churn_per_min);
document.getElementById('route_churn_per_sec').textContent = data.stream.route_churn_per_sec.toFixed(1);
}
// Update GC stats
@@ -686,12 +643,9 @@
});
}
// Update immediately and then every REFRESH_PERIOD_MS. The freshness
// check runs every second on its own, so a request that never returns
// still turns the spinner into "stale".
// Update immediately and then every 2 seconds
updateStatus();
setInterval(updateStatus, REFRESH_PERIOD_MS);
setInterval(showFreshness, MS_PER_SECOND);
setInterval(updateStatus, 2000);
</script>
</main>
internal/templates/templates.go
+44 -3
View File
@@ -7,9 +7,9 @@ import (
"net/url"
"strings"
"sync"
"time"
"git.eeqj.de/sneak/routewatch/internal/version"
"github.com/dustin/go-humanize"
)
//go:embed index.html
@@ -48,7 +48,48 @@ var (
once sync.Once
)
const cidrPartCount = 2 // A CIDR has two parts: prefix and length
const (
hoursPerDay = 24
daysPerMonth = 30
cidrPartCount = 2 // A CIDR has two parts: prefix and length
)
// timeSince returns a human-readable duration since the given time
func timeSince(t time.Time) string {
duration := time.Since(t)
if duration < time.Minute {
return "just now"
}
if duration < time.Hour {
minutes := int(duration.Minutes())
if minutes == 1 {
return "1 minute ago"
}
return duration.Truncate(time.Minute).String() + " ago"
}
if duration < hoursPerDay*time.Hour {
hours := int(duration.Hours())
if hours == 1 {
return "1 hour ago"
}
return duration.Truncate(time.Hour).String() + " ago"
}
days := int(duration.Hours() / hoursPerDay)
if days == 1 {
return "1 day ago"
}
if days < daysPerMonth {
return duration.Truncate(hoursPerDay*time.Hour).String() + " ago"
}
return t.Format("2006-01-02")
}
// prefixURL generates a URL path for a prefix in CIDR notation.
// Takes a prefix like "192.168.1.0/24" and returns "/prefix/192.168.1.0/24"
@@ -72,7 +113,7 @@ func initTemplates() {
// Create common template functions
funcs := template.FuncMap{
"timeSince": humanize.Time,
"timeSince": timeSince,
"urlEncode": url.QueryEscape,
"prefixURL": prefixURL,
"appName": func() string { return version.Name },
internal/templates/templates_test.go
-181
View File
@@ -1,181 +0,0 @@
package templates_test
import (
"bytes"
"html/template"
"strings"
"testing"
"git.eeqj.de/sneak/routewatch/internal/database"
"git.eeqj.de/sneak/routewatch/internal/templates"
)
// renderPage renders a page with the data every page needs, and the visitor's
// address, which the home page also needs.
func renderPage(t *testing.T, tmpl *template.Template) string {
t.Helper()
var page bytes.Buffer
err := tmpl.Execute(&page, struct{ CSPNonce, ClientIP string }{})
if err != nil {
t.Fatalf("rendering the page: %v", err)
}
return page.String()
}
// TestFooterLicense renders the home and status pages and expects each footer
// to name the licence in LICENSE, MIT, and not WTFPL.
func TestFooterLicense(t *testing.T) {
t.Parallel()
const (
license = "MIT"
wrongLicense = "WTFPL"
)
pages := map[string]*template.Template{
"index": templates.IndexTemplate(),
"status": templates.StatusTemplate(),
}
for name, tmpl := range pages {
t.Run(name, func(t *testing.T) {
t.Parallel()
_, footer, found := strings.Cut(renderPage(t, tmpl), "<footer")
if !found {
t.Fatal("the page has no footer")
}
footer, _, _ = strings.Cut(footer, "</footer>")
if !strings.Contains(footer, license) {
t.Errorf("footer does not name %s: %q", license, footer)
}
if strings.Contains(footer, wrongLicense) {
t.Errorf("footer names %s: %q", wrongLicense, footer)
}
})
}
}
// TestFreshnessIndicator renders the home and status pages and expects each to
// have, right after the status element, the element where the page's script
// shows its spinner or "stale".
func TestFreshnessIndicator(t *testing.T) {
t.Parallel()
pages := map[string]struct {
tmpl *template.Template
status string
indicator string
}{
"index": {
tmpl: templates.IndexTemplate(),
status: `<span id="stat-status">-</span>`,
indicator: `<span id="freshness"></span>`,
},
"status": {
tmpl: templates.StatusTemplate(),
status: `<span class="metric-value" id="connected">-</span>`,
indicator: `<span class="metric-value" id="freshness"></span>`,
},
}
for name, page := range pages {
t.Run(name, func(t *testing.T) {
t.Parallel()
_, afterStatus, found := strings.Cut(renderPage(t, page.tmpl), page.status)
if !found {
t.Fatalf("the page has no status element %s", page.status)
}
if !strings.HasPrefix(strings.TrimSpace(afterStatus), page.indicator) {
t.Errorf("the status element is not followed by %s", page.indicator)
}
})
}
}
// TestASDetailPrefixListsFold renders the AS page with one IPv4 and one IPv6
// prefix. It expects each prefix list inside its own <details> element that
// starts open and whose <summary> holds the list's heading, so that a click on
// the heading folds the list.
func TestASDetailPrefixListsFold(t *testing.T) {
t.Parallel()
const (
asn = 64500
prefixV4 = "192.0.2.0/24"
prefixV6 = "2001:db8::/32"
)
ipv4 := []database.LiveRoute{{Prefix: prefixV4}}
ipv6 := []database.LiveRoute{{Prefix: prefixV6}}
// The fields the AS page handler gives the template.
data := struct {
ASN *database.ASN
IPv4Prefixes []database.LiveRoute
IPv6Prefixes []database.LiveRoute
TotalCount int
IPv4Count int
IPv6Count int
Peers []database.ASPeer
PeerCount int
CSPNonce string
}{
ASN: &database.ASN{ASN: asn},
IPv4Prefixes: ipv4,
IPv6Prefixes: ipv6,
TotalCount: len(ipv4) + len(ipv6),
IPv4Count: len(ipv4),
IPv6Count: len(ipv6),
}
var rendered bytes.Buffer
err := templates.ASDetailTemplate().Execute(&rendered, data)
if err != nil {
t.Fatalf("rendering the page: %v", err)
}
page := rendered.String()
lists := []struct{ heading, prefix string }{
{"IPv4 Prefixes", prefixV4},
{"IPv6 Prefixes", prefixV6},
}
for _, list := range lists {
before, _, found := strings.Cut(page, list.prefix)
if !found {
t.Fatalf("the page does not list %s", list.prefix)
}
// The list's section starts at the last <details before its prefix.
start := strings.LastIndex(before, "<details")
if start < 0 || strings.Contains(before[start:], "</details>") {
t.Errorf("%s: the list is not inside a <details> element", list.heading)
continue
}
tag, section, _ := strings.Cut(before[start:], ">")
if !strings.Contains(tag, " open") {
t.Errorf("%s: the <details> element does not start open: %s>",
list.heading, tag)
}
_, summary, _ := strings.Cut(section, "<summary")
summary, _, _ = strings.Cut(summary, "</summary>")
if !strings.Contains(summary, list.heading) {
t.Errorf("%s: the <summary> does not hold the heading", list.heading)
}
}
}
internal/version/version.go
+2 -2
View File
@@ -19,8 +19,8 @@ const (
Author = "@sneak"
// AuthorURL is the author's website
AuthorURL = "https://sneak.berlin"
// License is the program license, as LICENSE grants it
License = "MIT"
// License is the program license
License = "WTFPL"
// RepoURL is the git repository URL
RepoURL = "https://git.eeqj.de/sneak/routewatch"
)
internal/whois/whois.go
+33 -284
View File
@@ -7,15 +7,10 @@ import (
"errors"
"fmt"
"io"
"log/slog"
"net"
"regexp"
"strings"
"sync"
"time"
"unicode/utf8"
"golang.org/x/text/encoding/charmap"
)
// Timeout constants for WHOIS queries.
@@ -31,18 +26,10 @@ const maxAnswerSize = 1 << 20
// Parsing constants.
const (
keyValueParts = 2 // Expected parts when splitting "key: value"
lacnicDateFormatLen = 8 // Length of YYYYMMDD date format
)
// The classes of the two objects that describe an AS in an answer from
// RIPE, APNIC or AFRINIC, and the key of the aut-num object's line that
// gives the ID of the AS's organisation object.
const (
autNumClass = "aut-num"
organisationClass = "organisation"
orgKey = "org"
)
// WHOIS server addresses.
const (
whoisServerIANA = "whois.iana.org:43"
@@ -51,32 +38,8 @@ const (
whoisServerAPNIC = "whois.apnic.net:43"
whoisServerLACNIC = "whois.lacnic.net:43"
whoisServerAFRINIC = "whois.afrinic.net:43"
// NIC.br registers Brazil's ASes under LACNIC.
whoisServerNICbr = "whois.registro.br:43"
)
// nicbrHeader starts the copyright notice at the top of NIC.br's WHOIS
// answers. LACNIC's answer for an AS that NIC.br registers is NIC.br's, and
// carries the same notice:
//
// % Copyright (c) Nic.br - Use of this data is governed by the Use and
// % Privacy Policy at https://registro.br/upp . Distribution,
const nicbrHeader = "% Copyright (c) Nic.br"
// nicbrLimitReached is in the line that marks NIC.br's reduced answer to a
// client over its query limit. NIC.br publishes no limit. Its Use and
// Privacy Policy, https://registro.br/upp, says only that it restricts
// queries to stop the data being taken in bulk.
const nicbrLimitReached = "Query rate limit exceeded"
// NICbrSkipTime is how long the client stops asking NIC.br after NIC.br
// answers that the client is over its query limit. NIC.br publishes
// neither the limit nor how long it lasts. An hour outlasts a limit
// counted per minute or per hour. The ASes looked up meanwhile get
// NIC.br's fields when they are next refreshed.
const NICbrSkipTime = time.Hour
// RIR identifiers.
const (
RIRARIN = "ARIN"
@@ -86,31 +49,6 @@ const (
RIRAFRNIC = "AFRINIC"
)
// noPersonalFlag keeps person and role objects out of an answer. The
// organisation object stays in it. RIPE counts the person and role objects
// it returns against a daily limit per IP address. Role objects for abuse
// contacts do not count. RIPE refuses an address for good that passes the
// limit too often:
// https://docs.db.ripe.net/Types-of-Queries/Access-Control-for-Queries/
//
// RIPE documents the flag here:
// https://docs.db.ripe.net/Types-of-Queries/Referenced-Objects-in-Query-Response/
// APNIC and AFRINIC list it in their answers to the query "help". AFRINIC
// sends readers to that answer for its full list of flags:
// https://afrinic.net/whois-getting-started
const noPersonalFlag = "--no-personal"
// The error lines with which RIPE's database software refuses a query:
// access denied to the client's IP address, and that address's limit
// reached. APNIC and AFRINIC run the same software. The lines are listed
// under "Access Errors" on this page, whose URL is split over two lines:
// https://docs.db.ripe.net/Appendices/
// Appendix-C--RIPE-Database-Query-Server-Response-Codes-and-Messages/
const (
accessDeniedError = "%ERROR:201:"
accessLimitError = "%ERROR:202:"
)
// Errors LookupASN returns when a WHOIS answer has nothing to store.
var (
// ErrNoReferral is returned when IANA refers the AS to no RIR, as it
@@ -120,7 +58,6 @@ var (
errNoASBlock = errors.New("IANA answered with neither a referral " +
"nor an AS block")
errNoFields = errors.New("RIR answered with no WHOIS fields")
errRefused = errors.New("RIR refused the query")
errUnknownServer = errors.New("IANA refers the AS to a server no RIR runs")
errAnswerTooLarge = errors.New("WHOIS answer reached the size cap")
)
@@ -149,42 +86,21 @@ type Client struct {
// host:port. NewClient sets a net.Dialer's; tests replace it to reach
// local servers instead.
DialContext func(ctx context.Context, network, address string) (net.Conn, error)
// Now returns the current time. NewClient sets time.Now; tests replace
// it to move the clock.
Now func() time.Time
logger *slog.Logger
// mu guards nicbrSkipUntil.
mu sync.Mutex
// nicbrSkipUntil is when the client next asks NIC.br, after NIC.br
// answered that the client is over its query limit.
nicbrSkipUntil time.Time
}
// NewClient creates a new WHOIS client that logs to logger.
func NewClient(logger *slog.Logger) *Client {
// NewClient creates a new WHOIS client.
func NewClient() *Client {
dialer := &net.Dialer{Timeout: dialTimeout}
return &Client{
DialContext: dialer.DialContext,
Now: time.Now,
logger: logger,
}
return &Client{DialContext: dialer.DialContext}
}
// LookupASN asks IANA which RIR holds the given ASN, asks that RIR, and
// returns what the RIR's answer says. When LACNIC's answer is NIC.br's, it
// returns what NIC.br's own server says instead, unless that query fails or
// is skipped, as askNICbr says.
//
// It returns ErrNoReferral when IANA refers the AS to no RIR, and another
// error when the query to IANA or to the RIR fails, when the RIR refuses
// it, when IANA's answer has neither a referral nor the AS's block, when
// IANA refers the AS to a server none of the five RIRs run, or when the
// answer carries none of ASNInfo's fields.
// returns what the RIR's answer says. It returns ErrNoReferral when IANA
// refers the AS to no RIR, and another error when a query fails, when IANA's
// answer has neither a referral nor the AS's block, when IANA refers the AS
// to a server none of the five RIRs run, or when the RIR's answer carries
// none of ASNInfo's fields, as a refusal of service does.
func (c *Client) LookupASN(ctx context.Context, asn int) (*ASNInfo, error) {
// Query IANA first to find the authoritative RIR
query := fmt.Sprintf("AS%d", asn)
@@ -211,82 +127,23 @@ func (c *Client) LookupASN(ctx context.Context, asn int) (*ASNInfo, error) {
}
// Query the authoritative RIR
rirResp, err := c.queryRIR(ctx, whoisServer, rirQuery(rir, query))
rirResp, err := c.query(ctx, whoisServer, query)
if err != nil {
return nil, fmt.Errorf("RIR query failed: %w", err)
}
// The fields never come from IANA's answer, which describes the block
// the AS is in.
// The fields come from the RIR's answer alone: IANA's describes the
// block the AS is in.
info := c.parseResponse(asn, rir, rirResp)
if rir == RIRLACNIC && strings.Contains(rirResp, nicbrHeader) {
info = c.askNICbr(ctx, query, info)
}
if info.isEmpty() {
return nil, fmt.Errorf("%s from %s: %w", query, whoisServer, errNoFields)
}
info.RawResponse = ianaResp + "\n" + info.RawResponse
info.RawResponse = ianaResp + "\n" + rirResp
return info, nil
}
// askNICbr asks NIC.br's server about an AS whose answer from LACNIC is
// NIC.br's, and returns what NIC.br's answer says. LACNIC replaces each
// accented letter in such an answer with U+FFFD. NIC.br's own server keeps
// them, in Latin-1.
//
// askNICbr returns lacnicInfo, what LACNIC's answer says, and logs a
// warning when the query fails, when NIC.br sends the reduced answer for a
// client over its limit, or when NIC.br's answer carries none of ASNInfo's
// fields. After the reduced answer, it returns lacnicInfo without asking
// NIC.br until NICbrSkipTime has passed.
func (c *Client) askNICbr(
ctx context.Context, asQuery string, lacnicInfo *ASNInfo,
) *ASNInfo {
asn := lacnicInfo.ASN
c.mu.Lock()
skipping := c.Now().Before(c.nicbrSkipUntil)
c.mu.Unlock()
if skipping {
return lacnicInfo
}
answer, err := c.query(ctx, whoisServerNICbr, asQuery)
if err != nil {
c.logger.Warn("NIC.br query failed, keeping LACNIC's fields",
"asn", asn, "error", err)
return lacnicInfo
}
if strings.Contains(answer, nicbrLimitReached) {
c.mu.Lock()
c.nicbrSkipUntil = c.Now().Add(NICbrSkipTime)
c.mu.Unlock()
c.logger.Warn("NIC.br answered that the client is over its query "+
"limit, keeping LACNIC's fields and skipping NIC.br",
"asn", asn, "skip_for", NICbrSkipTime)
return lacnicInfo
}
info := c.parseResponse(asn, RIRLACNIC, answer)
if info.isEmpty() {
c.logger.Warn("NIC.br answered with no WHOIS fields, "+
"keeping LACNIC's fields", "asn", asn)
return lacnicInfo
}
return info
}
// isEmpty reports whether info holds none of the fields a WHOIS answer
// fills, as when the answer is only comment and error lines.
func (info *ASNInfo) isEmpty() bool {
@@ -297,41 +154,8 @@ func (info *ASNInfo) isEmpty() bool {
info.RegDate == nil && info.LastMod == nil
}
// rirQuery returns the query sent to rir's server for asQuery, an AS
// number such as "AS3333". RIPE, APNIC and AFRINIC are asked to keep person
// and role objects out of the answer.
func rirQuery(rir, asQuery string) string {
switch rir {
case RIRRIPE, RIRAPNIC, RIRAFRNIC:
return noPersonalFlag + " " + asQuery
default:
return asQuery
}
}
// queryRIR sends query to an RIR's server and returns the answer. An answer
// holding an error line with which the server refuses the query is an
// error, and the error holds that line.
func (c *Client) queryRIR(
ctx context.Context, server, query string,
) (string, error) {
answer, err := c.query(ctx, server, query)
if err != nil {
return "", err
}
for line := range strings.SplitSeq(answer, "\n") {
if strings.HasPrefix(line, accessDeniedError) ||
strings.HasPrefix(line, accessLimitError) {
return "", fmt.Errorf("%w: %s", errRefused, line)
}
}
return answer, nil
}
// query performs a raw WHOIS query to the specified server and returns the
// answer in UTF-8. An answer that reaches maxAnswerSize is an error.
// query performs a raw WHOIS query to the specified server. An answer that
// reaches maxAnswerSize is an error.
func (c *Client) query(ctx context.Context, server, query string) (string, error) {
conn, err := c.DialContext(ctx, "tcp", server)
if err != nil {
@@ -378,25 +202,7 @@ func (c *Client) query(ctx context.Context, server, query string) (string, error
return "", fmt.Errorf("read response: %w", errAnswerTooLarge)
}
return toUTF8(sb.String())
}
// toUTF8 returns a WHOIS answer in UTF-8. An answer that is valid UTF-8 is
// returned as it is. Any other answer is decoded as Latin-1, which RIPE,
// APNIC and NIC.br send. LACNIC, ARIN and AFRINIC send UTF-8. RIPE's
// database working group lists the five RIRs here:
// https://www.ripe.net/ripe/mail/archives/db-wg/2024-May/008075.html
func toUTF8(answer string) (string, error) {
if utf8.ValidString(answer) {
return answer, nil
}
decoded, err := charmap.ISO8859_1.NewDecoder().String(answer)
if err != nil {
return "", fmt.Errorf("decode Latin-1 answer: %w", err)
}
return decoded, nil
return sb.String(), nil
}
// parseIANAReferral extracts the RIR and WHOIS server from an IANA response.
@@ -432,9 +238,7 @@ func (c *Client) parseIANAReferral(response string) (string, string, error) {
return "", "", nil
}
// parseResponse extracts ASN information from a WHOIS response. Of an
// answer from RIPE, APNIC or AFRINIC it reads only the lines asObjectLines
// returns. Of an answer from ARIN or LACNIC it reads every line.
// parseResponse extracts ASN information from a WHOIS response.
func (c *Client) parseResponse(asn int, rir, response string) *ASNInfo {
info := &ASNInfo{
ASN: asn,
@@ -443,15 +247,26 @@ func (c *Client) parseResponse(asn int, rir, response string) *ASNInfo {
}
lines := strings.Split(response, "\n")
if rir == RIRRIPE || rir == RIRAPNIC || rir == RIRAFRNIC {
lines = asObjectLines(lines)
}
var addressLines []string
for _, line := range lines {
key, value, ok := splitField(line)
if !ok || value == "" {
line = strings.TrimSpace(line)
if line == "" || strings.HasPrefix(line, "%") ||
strings.HasPrefix(line, "#") {
continue
}
// Split on first colon
parts := strings.SplitN(line, ":", keyValueParts)
if len(parts) != keyValueParts {
continue
}
key := strings.TrimSpace(strings.ToLower(parts[0]))
value := strings.TrimSpace(parts[1])
if value == "" {
continue
}
@@ -478,72 +293,6 @@ func (c *Client) parseResponse(asn int, rir, response string) *ASNInfo {
return info
}
// splitField splits a "key: value" line of a WHOIS answer into its key, in
// lower case, and its value. The third result is false for a line that
// holds no field: a blank line, a comment line or a line without a colon.
func splitField(line string) (string, string, bool) {
line = strings.TrimSpace(line)
if strings.HasPrefix(line, "%") || strings.HasPrefix(line, "#") {
return "", "", false
}
key, value, ok := strings.Cut(line, ":")
return strings.ToLower(strings.TrimSpace(key)), strings.TrimSpace(value), ok
}
// asObjectLines returns the lines of the aut-num object in an answer from
// RIPE, APNIC or AFRINIC, followed by the lines of the organisation object
// whose ID the aut-num object gives on its "org:" line. The answer's other
// objects are left out. RIPE and APNIC start their answer with the
// as-block object of the block the AS is in, whose dates are not the AS's.
//
// These answers are in RPSL. A blank line ends an object. An object's first
// line has the object's class as its key and the object's ID as its value,
// as in "organisation: ORG-APNI1-AP".
func asObjectLines(lines []string) []string {
var autNum []string
// The lines of each organisation object, by its ID.
organisations := make(map[string][]string)
// The class and ID of the object the line is in, "" between objects.
var class, id string
// The ID of the organisation object the aut-num object refers to.
var orgID string
for _, line := range lines {
if strings.TrimSpace(line) == "" {
class = ""
continue
}
key, value, ok := splitField(line)
if !ok {
continue
}
if class == "" {
class, id = key, value
}
switch class {
case autNumClass:
autNum = append(autNum, line)
if key == orgKey {
orgID = value
}
case organisationClass:
organisations[id] = append(organisations[id], line)
}
}
return append(autNum, organisations[orgID]...)
}
// setResponseField stores the value of one "key: value" line of a WHOIS
// response in info, keeping the first value seen for each field.
func (c *Client) setResponseField(info *ASNInfo, key, value string) {
internal/whois/whois_test.go
+28 -663
View File
@@ -2,21 +2,15 @@ package whois_test
import (
"bufio"
"bytes"
"context"
"encoding/json"
"errors"
"io"
"log/slog"
"maps"
"net"
"slices"
"strings"
"sync"
"syscall"
"testing"
"time"
"unicode/utf8"
"git.eeqj.de/sneak/routewatch/internal/whois"
)
@@ -27,20 +21,8 @@ const (
ripeServer = "whois.ripe.net:43"
)
// testASN is the AS the tests look up, and asQuery the query for it.
const (
testASN = 3333
asQuery = "AS3333"
)
// The fake servers' answers are keyed by a server's address, a space and a
// query. These are the keys of the queries LookupASN sends for testASN:
// IANA's, then RIPE's, which asks RIPE to leave out person and role
// objects.
const (
ianaQuery = ianaServer + " " + asQuery
ripeQuery = ripeServer + " --no-personal " + asQuery
)
// testASN is the AS the tests look up.
const testASN = 3333
// ianaReferral is IANA's answer for an AS in a block it gave to RIPE.
const ianaReferral = `% IANA WHOIS server
@@ -54,163 +36,27 @@ changed: 2002-09-01
source: IANA
`
// lacnicReferral is IANA's answer for an AS in a block it gave to LACNIC,
// shortened to the referral.
const lacnicReferral = "refer: whois.lacnic.net\n"
// ripeRecord is RIPE's answer for the AS when asked to leave out person and
// role objects. It starts with the as-block object of the block the AS is
// in. Then come the abuse address as a comment, the AS's object and the
// organisation object it refers to.
// ripeRecord is RIPE's answer for the AS.
const ripeRecord = `% This is the RIPE Database query service.
% The objects are in RPSL format.
% Note: --no-personal means ALL personal data has been filtered from this output.
% Information related to 'AS3209 - AS3353'
as-block: AS3209 - AS3353
descr: RIPE NCC ASN block
remarks: These AS Numbers are assigned to network operators
remarks: in the RIPE NCC service region.
mnt-by: RIPE-NCC-HM-MNT
created: 2018-11-22T15:27:19Z
last-modified: 2018-11-22T15:27:19Z
source: RIPE
% Information related to 'AS3333'
% Abuse contact for 'AS3333' is 'abuse@ripe.net'
aut-num: AS3333
as-name: RIPE-NCC-AS
org: ORG-RIEN1-RIPE
admin-c: BRD-RIPE
tech-c: OPS4-RIPE
abuse-c: OPS4-RIPE
mnt-by: RIPE-NCC-MNT
created: 2002-09-17T13:08:32Z
last-modified: 2024-01-09T08:55:41Z
source: RIPE
organisation: ORG-RIEN1-RIPE
org-name: Reseaux IP Europeens Network Coordination Centre (RIPE NCC)
country: NL
org-type: RIR
address: P.O. Box 10096
address: 1001EB Amsterdam
abuse-c: OPS4-RIPE
created: 2012-03-09T13:08:29Z
last-modified: 2024-03-26T10:55:38Z
source: RIPE
`
// The error lines with which RIPE refuses a query: access denied to the
// host, and the host's limit on returned objects reached.
const (
ripeAccessDenied = "%ERROR:201: access denied for 192.0.2.1"
ripeLimitReached = "%ERROR:202: access control limit reached"
)
// ripeRefusal is RIPE's answer to a host it no longer serves.
const ripeRefusal = `% This is the RIPE Database query service.
% The objects are in RPSL format.
%
` + ripeAccessDenied + `
%ERROR:201: access denied for 192.0.2.1
%
% Sorry, access from your host has been permanently
% denied because of a repeated excessive querying.
`
// apnicAfrinicRecord is the answer of APNIC or AFRINIC for the AS when asked
// to leave out person and role objects. Like RIPE's, it holds the abuse
// address as a comment, the AS's object and the organisation object.
const apnicAfrinicRecord = `% Abuse contact for 'AS3333' is 'abuse@example.net'
aut-num: AS3333
as-name: EXAMPLE-AS
org: ORG-EXAMPLE1
admin-c: EX1-TEST
last-modified: 2026-08-26T00:40:00Z
organisation: ORG-EXAMPLE1
org-name: Example Networks
country: ZA
address: 1 Example Road
`
// apnicServer is APNIC's WHOIS server, and apnicASN an AS that APNIC holds.
const (
apnicServer = "whois.apnic.net:43"
apnicASN = 4608
)
// The keys of the queries LookupASN sends for apnicASN: IANA's, then
// APNIC's, which asks APNIC to leave out person and role objects.
const (
apnicIANAQuery = ianaServer + " AS4608"
apnicQuery = apnicServer + " --no-personal AS4608"
)
// apnicRecord is APNIC's answer for apnicASN when asked to leave out person
// and role objects, shortened. It starts with the as-block object of the
// block the AS is in. Then come the abuse address as a comment, the AS's
// object, and the irt and organisation objects it refers to.
const apnicRecord = `% [whois.apnic.net]
% Whois data copyright terms http://www.apnic.net/db/dbcopyright.html
% Note: --no-personal means ALL personal data has been filtered from this output.
% Information related to 'AS4608 - AS4865'
as-block: AS4608 - AS4865
descr: APNIC ASN block
remarks: These AS numbers are further assigned by APNIC
remarks: to APNIC members and end-users in the APNIC region
mnt-by: APNIC-HM
mnt-lower: APNIC-HM
last-modified: 2019-05-20T03:52:26Z
source: APNIC
% Information related to 'AS4608'
% Abuse contact for 'AS4608' is 'abuse@apnic.net'
aut-num: AS4608
as-name: APNIC-SERVICES
descr: Asia Pacific Network Information Centre
country: AU
org: ORG-APNI1-AP
admin-c: AIC1-AP
abuse-c: AA1589-AP
mnt-by: APNIC-HM
mnt-irt: IRT-APNIC-IS-AP
last-modified: 2026-08-26T00:40:00Z
source: APNIC
irt: IRT-APNIC-IS-AP
address: South Brisbane, Australia
e-mail: helpdesk@apnic.net
abuse-mailbox: abuse@apnic.net
mnt-by: MAINT-APNIC-IS-AP
last-modified: 2025-11-18T00:27:08Z
source: APNIC
organisation: ORG-APNI1-AP
org-name: Asia Pacific Network Information Centre
org-type: LIR
country: AU
address: PO Box 3646
address: South Brisbane, QLD 4101,
address: Australia
e-mail: noc@apnic.net
mnt-by: APNIC-HM
last-modified: 2026-08-24T12:58:43Z
source: APNIC
% This query was served by the APNIC Whois Service version 1.88.48 (WHOIS-UK2)
`
// ianaPrivateUse is IANA's answer for a private-use AS: its block and no
// referral.
const ianaPrivateUse = `% IANA WHOIS server
@@ -234,29 +80,20 @@ const fillerLine = "remarks: and more\n"
const endlessAnswerSize = 4 << 20
// fakeServers returns a DialContext for a whois.Client that connects to a
// fake server at each address answers names. Each key of answers is a
// server's address, a space and a query, and its value is that server's
// answer to that query. A server reads one query and sends its answer, or
// nothing for a query it has no answer for. Connecting to any other address
// is refused.
// fake server for each address in answers, which reads the query and sends
// answers[address]. Connecting to any other address is refused.
func fakeServers(
t *testing.T, answers map[string]string,
) func(context.Context, string, string) (net.Conn, error) {
t.Helper()
addresses := make(map[string]bool)
for key := range answers {
address, _, _ := strings.Cut(key, " ")
addresses[address] = true
}
var servers sync.WaitGroup
t.Cleanup(servers.Wait)
return func(_ context.Context, _, address string) (net.Conn, error) {
if !addresses[address] {
answer, ok := answers[address]
if !ok {
return nil, syscall.ECONNREFUSED
}
@@ -265,8 +102,7 @@ func fakeServers(
servers.Go(func() {
defer func() { _ = server.Close() }()
query, _ := bufio.NewReader(server).ReadString('\n')
answer := answers[address+" "+strings.TrimSpace(query)]
_, _ = bufio.NewReader(server).ReadString('\n')
_, _ = io.WriteString(server, answer)
})
@@ -280,48 +116,31 @@ func lookUp(
) (*whois.ASNInfo, error) {
t.Helper()
client := whois.NewClient(slog.New(slog.DiscardHandler))
client := whois.NewClient()
client.DialContext = fakeServers(t, answers)
return client.LookupASN(t.Context(), testASN)
}
// TestLookupASNReadsTheRIRAnswer looks up an AS IANA refers to RIPE. RIPE
// answers only a query that leaves out person and role objects, and its
// answer to that still fills the AS's fields: the abuse address from the
// comment line, and the organisation's name and country from its object.
// The dates are the AS's. IANA's "changed:" line dates the AS's block, and
// so do the dates of the as-block object at the start of RIPE's answer.
// TestLookupASNReadsTheRIRAnswer looks up an AS IANA refers to RIPE. The
// fields come from RIPE's answer, the last-modified date included: IANA's
// "changed:" line dates its block, not the AS.
func TestLookupASNReadsTheRIRAnswer(t *testing.T) {
t.Parallel()
info, err := lookUp(t, map[string]string{
ianaQuery: ianaReferral,
ripeQuery: ripeRecord,
ianaServer: ianaReferral,
ripeServer: ripeRecord,
})
if err != nil {
t.Fatalf("LookupASN: %v", err)
}
got := []string{
info.ASName, info.OrgID, info.OrgName, info.CountryCode,
info.AbuseEmail, info.RIR,
}
want := []string{
"RIPE-NCC-AS", "ORG-RIEN1-RIPE",
"Reseaux IP Europeens Network Coordination Centre (RIPE NCC)", "NL",
"abuse@ripe.net", whois.RIRRIPE,
}
if !slices.Equal(got, want) {
t.Errorf("LookupASN = AS name, org ID, org name, country, abuse "+
"address, RIR %q; want %q", got, want)
}
wantRegDate := time.Date(2002, 9, 17, 13, 8, 32, 0, time.UTC)
if info.RegDate == nil || !info.RegDate.Equal(wantRegDate) {
t.Errorf("LookupASN registered = %v, want %v",
info.RegDate, wantRegDate)
if info.ASName != "RIPE-NCC-AS" || info.OrgID != "ORG-RIEN1-RIPE" ||
info.RIR != whois.RIRRIPE {
t.Errorf("LookupASN = AS name %q, org %q, RIR %q; "+
"want RIPE-NCC-AS, ORG-RIEN1-RIPE, RIPE",
info.ASName, info.OrgID, info.RIR)
}
wantLastMod := time.Date(2024, 1, 9, 8, 55, 41, 0, time.UTC)
@@ -331,460 +150,6 @@ func TestLookupASNReadsTheRIRAnswer(t *testing.T) {
}
}
// TestLookupASNReadsOnlyTheASAndItsOrganisation looks up an AS at APNIC,
// whose answer starts with the as-block object of the block the AS is in.
// The last-modified date is the aut-num object's, not the block's. The
// address is the organisation object's alone, without the irt object's.
func TestLookupASNReadsOnlyTheASAndItsOrganisation(t *testing.T) {
t.Parallel()
client := whois.NewClient(slog.New(slog.DiscardHandler))
client.DialContext = fakeServers(t, map[string]string{
apnicIANAQuery: "refer: whois.apnic.net\n",
apnicQuery: apnicRecord,
})
info, err := client.LookupASN(t.Context(), apnicASN)
if err != nil {
t.Fatalf("LookupASN: %v", err)
}
wantLastMod := time.Date(2026, 8, 26, 0, 40, 0, 0, time.UTC)
if info.LastMod == nil || !info.LastMod.Equal(wantLastMod) {
t.Errorf("LookupASN last modified = %v, want %v",
info.LastMod, wantLastMod)
}
got := []string{
info.ASName, info.OrgID, info.OrgName, info.CountryCode,
info.Address, info.AbuseEmail,
}
want := []string{
"APNIC-SERVICES", "ORG-APNI1-AP",
"Asia Pacific Network Information Centre", "AU",
"PO Box 3646\nSouth Brisbane, QLD 4101,\nAustralia", "abuse@apnic.net",
}
if !slices.Equal(got, want) {
t.Errorf("LookupASN = AS name, org ID, org name, country, address, "+
"abuse address %q; want %q", got, want)
}
}
// TestLookupASNAsksAPNICAndAFRINICToLeaveOutPersonalData looks up an AS at
// APNIC and at AFRINIC. Each answers only a query that leaves out person
// and role objects, as RIPE does. That one answer still holds the
// organisation object, so the organisation's name and country are filled.
func TestLookupASNAsksAPNICAndAFRINICToLeaveOutPersonalData(t *testing.T) {
t.Parallel()
for _, server := range []string{"whois.apnic.net", "whois.afrinic.net"} {
t.Run(server, func(t *testing.T) {
t.Parallel()
address := server + ":43"
info, err := lookUp(t, map[string]string{
ianaQuery: "refer: " + server + "\n",
address + " --no-personal " + asQuery: apnicAfrinicRecord,
})
if err != nil {
t.Fatalf("LookupASN: %v", err)
}
got := []string{
info.ASName, info.OrgName, info.CountryCode, info.AbuseEmail,
}
want := []string{
"EXAMPLE-AS", "Example Networks", "ZA", "abuse@example.net",
}
if !slices.Equal(got, want) {
t.Errorf("LookupASN = AS name, org name, country, abuse "+
"address %q; want %q", got, want)
}
})
}
}
// The name of an organisation with accented letters, in Latin-1, one byte for
// each letter, and in UTF-8.
const (
latin1OrgName = "Fundaci\xf3n Ra\xedces de Espa\xf1a"
utf8OrgName = "Fundación Raíces de España"
)
// TestLookupASNStoresAnswersInUTF8 looks up an AS at RIPE, which answers in
// Latin-1, and at LACNIC, which answers in UTF-8. Both answers give the
// organisation's name with its accented letters in UTF-8, in the parsed field
// and in the stored answer. Before the fix the Latin-1 answer was kept as
// received, which is not valid UTF-8.
func TestLookupASNStoresAnswersInUTF8(t *testing.T) {
t.Parallel()
for name, answers := range map[string]map[string]string{
"RIPE in Latin-1": {
ianaQuery: ianaReferral,
ripeQuery: "aut-num: AS3333\n" +
"org-name: " + latin1OrgName + "\n",
},
"LACNIC in UTF-8": {
ianaQuery: lacnicReferral,
"whois.lacnic.net:43 " + asQuery: "aut-num: AS3333\n" +
"owner: " + utf8OrgName + "\n",
},
} {
t.Run(name, func(t *testing.T) {
t.Parallel()
info, err := lookUp(t, answers)
if err != nil {
t.Fatalf("LookupASN: %v", err)
}
if info.OrgName != utf8OrgName {
t.Errorf("LookupASN org name = %q, want %q",
info.OrgName, utf8OrgName)
}
if !utf8.ValidString(info.RawResponse) ||
!strings.Contains(info.RawResponse, utf8OrgName) {
t.Errorf("LookupASN answer = %q, want UTF-8 holding %q",
info.RawResponse, utf8OrgName)
}
})
}
}
// lacnicServer is LACNIC's WHOIS server and nicbrServer NIC.br's, which
// registers Brazil's ASes under LACNIC. brazilASN is one of those ASes.
const (
lacnicServer = "whois.lacnic.net:43"
nicbrServer = "whois.registro.br:43"
brazilASN = 52751
)
// The keys of the queries LookupASN sends for brazilASN: IANA's, LACNIC's,
// then NIC.br's.
const (
brazilIANAQuery = ianaServer + " AS52751"
brazilLACNICQuery = lacnicServer + " AS52751"
brazilNICbrQuery = nicbrServer + " AS52751"
)
// The name of brazilASN's organisation in three forms: as LACNIC sends it,
// with each accented letter replaced by U+FFFD; as NIC.br sends it, in
// Latin-1; and in UTF-8.
const (
lacnicBrazilOrgName = "Brasrede Telecomunica\U0000FFFD\U0000FFFDes LTDA"
nicbrBrazilOrgName = "Brasrede Telecomunica\xe7\xf5es LTDA"
brazilOrgName = "Brasrede Telecomunicações LTDA"
)
// nicbrCopyright is the copyright notice at the top of NIC.br's answers.
const nicbrCopyright = "% Copyright (c) Nic.br - " +
"Use of this data is governed by the Use and\n" +
"% Privacy Policy at https://registro.br/upp . Distribution,\n" +
"% commercialization, reproduction, and use for advertising or similar\n" +
"% purposes are expressly prohibited.\n"
// lacnicBrazilRecord is LACNIC's answer for brazilASN, recorded on
// 2026-10-26. It is NIC.br's answer, which LACNIC passes on after a line
// of its own and a space. It is shortened. The client's address is a
// documentation address. The contact's name, the contact's object and the
// footer are left out.
const lacnicBrazilRecord = "% IP Client: 192.0.2.1\n" +
" " + nicbrCopyright + `% 2026-10-26T23:12:22-03:00 - 192.0.2.1
aut-num: AS52751
owner: ` + lacnicBrazilOrgName + `
owner-c: AVL217
routing-c: AVL217
abuse-c: AVL217
created: 20120829
changed: 20130306
inetnum: 177.38.152.0/21
inetnum: 2804:b34::/32
inetnum: 191.243.4.0/22
inetnum: 131.72.248.0/22
inetnum: 138.94.228.0/22
inetnum: 170.233.8.0/22
inetnum: 170.246.4.0/22
`
// nicbrRecord is NIC.br's answer for brazilASN, in Latin-1, recorded on
// 2026-10-26 and shortened as lacnicBrazilRecord is.
const nicbrRecord = nicbrCopyright + `% 2026-10-26T23:14:41-03:00 - 192.0.2.1
aut-num: AS52751
owner: ` + nicbrBrazilOrgName + `
owner-c: AVL217
routing-c: AVL217
abuse-c: AVL217
created: 20120829
changed: 20130306
inetnum: 177.38.152.0/21
inetnum: 2804:b34::/32
inetnum: 191.243.4.0/22
inetnum: 131.72.248.0/22
inetnum: 138.94.228.0/22
inetnum: 170.233.8.0/22
inetnum: 170.246.4.0/22
`
// nicbrOverLimit stands for NIC.br's answer for brazilASN to a client over
// its query limit, and is not recorded. Such an answer leaves information
// out, here the owner, and says so in its first line.
const nicbrOverLimit = `% Query rate limit exceeded. Reduced information.
aut-num: AS52751
created: 20120829
changed: 20130306
`
// newNICbrClient returns a client whose fake servers send answers and
// whose logger writes JSON lines to logs. It adds one to *nicbrAsks each
// time it connects to NIC.br's server.
func newNICbrClient(
t *testing.T, answers map[string]string, logs io.Writer, nicbrAsks *int,
) *whois.Client {
t.Helper()
dial := fakeServers(t, answers)
client := whois.NewClient(slog.New(slog.NewJSONHandler(logs, nil)))
client.DialContext = func(
ctx context.Context, network, address string,
) (net.Conn, error) {
if address == nicbrServer {
*nicbrAsks++
}
return dial(ctx, network, address)
}
return client
}
// lookUpBrazilianAS looks brazilASN up with fake servers sending answers,
// and reports whether the client connected to NIC.br's server.
func lookUpBrazilianAS(
t *testing.T, answers map[string]string,
) (*whois.ASNInfo, bool, error) {
t.Helper()
nicbrAsks := 0
client := newNICbrClient(t, answers, io.Discard, &nicbrAsks)
info, err := client.LookupASN(t.Context(), brazilASN)
return info, nicbrAsks > 0, err
}
// TestLookupASNTakesABrazilianASFromNICbr looks up an AS that NIC.br
// registers. LACNIC's answer is NIC.br's with each accented letter replaced
// by U+FFFD, so the client also asks NIC.br's server, which answers in
// Latin-1. The organisation's name has its accented letters, in the parsed
// field and in the stored answer.
func TestLookupASNTakesABrazilianASFromNICbr(t *testing.T) {
t.Parallel()
info, _, err := lookUpBrazilianAS(t, map[string]string{
brazilIANAQuery: lacnicReferral,
brazilLACNICQuery: lacnicBrazilRecord,
brazilNICbrQuery: nicbrRecord,
})
if err != nil {
t.Fatalf("LookupASN: %v", err)
}
if info.OrgName != brazilOrgName || info.RIR != whois.RIRLACNIC {
t.Errorf("LookupASN org name, RIR = %q, %q; want %q, %q",
info.OrgName, info.RIR, brazilOrgName, whois.RIRLACNIC)
}
if !strings.Contains(info.RawResponse, brazilOrgName) {
t.Errorf("LookupASN answer = %q, want one holding %q",
info.RawResponse, brazilOrgName)
}
wantRegDate := time.Date(2012, 8, 29, 0, 0, 0, 0, time.UTC)
if info.RegDate == nil || !info.RegDate.Equal(wantRegDate) {
t.Errorf("LookupASN registered = %v, want %v",
info.RegDate, wantRegDate)
}
}
// TestLookupASNAsksNICbrOnlyForItsASes looks up an AS at LACNIC whose
// answer does not carry NIC.br's copyright notice, as for an AS that LACNIC
// registers itself. The client does not connect to NIC.br's server, which
// would answer with a record, and keeps LACNIC's fields.
func TestLookupASNAsksNICbrOnlyForItsASes(t *testing.T) {
t.Parallel()
info, askedNICbr, err := lookUpBrazilianAS(t, map[string]string{
brazilIANAQuery: lacnicReferral,
brazilLACNICQuery: "aut-num: AS52751\n" +
"owner: " + utf8OrgName + "\n",
brazilNICbrQuery: nicbrRecord,
})
if err != nil {
t.Fatalf("LookupASN: %v", err)
}
if askedNICbr {
t.Error("LookupASN asked NIC.br's server about an AS whose " +
"answer from LACNIC is not NIC.br's")
}
if info.OrgName != utf8OrgName {
t.Errorf("LookupASN org name = %q, want LACNIC's %q",
info.OrgName, utf8OrgName)
}
}
// TestLookupASNKeepsLACNICsAnswerWhenNICbrFails looks up an AS that NIC.br
// registers when NIC.br's server refuses the connection, sends nothing, or
// sends the reduced answer for a client over its query limit. The client
// asks NIC.br, and the lookup still succeeds with LACNIC's fields. The
// client logs one warning, which names the AS and the reason.
func TestLookupASNKeepsLACNICsAnswerWhenNICbrFails(t *testing.T) {
t.Parallel()
for name, test := range map[string]struct {
nicbrAnswer map[string]string
reason string
}{
"connection refused": {
nicbrAnswer: map[string]string{},
reason: syscall.ECONNREFUSED.Error(),
},
"no answer": {
nicbrAnswer: map[string]string{brazilNICbrQuery: ""},
reason: "no WHOIS fields",
},
"over the query limit": {
nicbrAnswer: map[string]string{brazilNICbrQuery: nicbrOverLimit},
reason: "over its query limit",
},
} {
t.Run(name, func(t *testing.T) {
t.Parallel()
answers := map[string]string{
brazilIANAQuery: lacnicReferral,
brazilLACNICQuery: lacnicBrazilRecord,
}
maps.Copy(answers, test.nicbrAnswer)
var logs bytes.Buffer
nicbrAsks := 0
client := newNICbrClient(t, answers, &logs, &nicbrAsks)
info, err := client.LookupASN(t.Context(), brazilASN)
if err != nil {
t.Fatalf("LookupASN: %v", err)
}
if nicbrAsks == 0 {
t.Error("LookupASN did not ask NIC.br's server")
}
if info.OrgName != lacnicBrazilOrgName {
t.Errorf("LookupASN org name = %q, want LACNIC's %q",
info.OrgName, lacnicBrazilOrgName)
}
var logged struct {
Level string `json:"level"`
ASN int `json:"asn"`
}
err = json.Unmarshal(logs.Bytes(), &logged)
if err != nil {
t.Fatalf("want one log line, got %q: %v", logs.String(), err)
}
if logged.Level != slog.LevelWarn.String() ||
logged.ASN != brazilASN ||
!strings.Contains(logs.String(), test.reason) {
t.Errorf("logged %q, want a warning for AS%d holding %q",
logs.String(), brazilASN, test.reason)
}
})
}
}
// TestLookupASNSkipsNICbrAfterItsLimit looks up an AS that NIC.br
// registers three times with one client, while NIC.br answers that the
// client is over its query limit. The first lookup asks NIC.br. The
// second, a second before NICbrSkipTime has passed, does not. The third,
// once NICbrSkipTime has passed, asks again. Each keeps LACNIC's fields.
func TestLookupASNSkipsNICbrAfterItsLimit(t *testing.T) {
t.Parallel()
nicbrAsks := 0
client := newNICbrClient(t, map[string]string{
brazilIANAQuery: lacnicReferral,
brazilLACNICQuery: lacnicBrazilRecord,
brazilNICbrQuery: nicbrOverLimit,
}, io.Discard, &nicbrAsks)
start := time.Date(2026, 10, 27, 0, 0, 0, 0, time.UTC)
now := start
client.Now = func() time.Time { return now }
for _, step := range []struct {
after time.Duration
wantAsks int
}{
{after: 0, wantAsks: 1},
{after: whois.NICbrSkipTime - time.Second, wantAsks: 1},
{after: whois.NICbrSkipTime, wantAsks: 2},
} {
now = start.Add(step.after)
info, err := client.LookupASN(t.Context(), brazilASN)
if err != nil {
t.Fatalf("LookupASN after %v: %v", step.after, err)
}
if nicbrAsks != step.wantAsks {
t.Errorf("after %v NIC.br's server was asked %d times, want %d",
step.after, nicbrAsks, step.wantAsks)
}
if info.OrgName != lacnicBrazilOrgName {
t.Errorf("LookupASN after %v org name = %q, want LACNIC's %q",
step.after, info.OrgName, lacnicBrazilOrgName)
}
}
}
// TestLookupASNReportsARefusal looks up an AS IANA refers to RIPE when RIPE
// refuses the query, with each of the error lines RIPE refuses with. The
// error holds RIPE's line, so the log says why the lookup failed.
func TestLookupASNReportsARefusal(t *testing.T) {
t.Parallel()
for _, line := range []string{ripeAccessDenied, ripeLimitReached} {
t.Run(line, func(t *testing.T) {
t.Parallel()
_, err := lookUp(t, map[string]string{
ianaQuery: ianaReferral,
ripeQuery: "% This is the RIPE Database query service.\n" +
line + "\n",
})
if err == nil || !strings.Contains(err.Error(), line) {
t.Errorf("LookupASN error = %v, want one holding %q",
err, line)
}
})
}
}
// TestLookupASNFailsWhenTheRIRGivesNoAnswer looks up an AS IANA refers to
// RIPE when RIPE's server refuses the connection, and when it answers only
// with an error. Neither is a lookup that found the AS.
@@ -792,8 +157,8 @@ func TestLookupASNFailsWhenTheRIRGivesNoAnswer(t *testing.T) {
t.Parallel()
for name, answers := range map[string]map[string]string{
"connection refused": {ianaQuery: ianaReferral},
"only an error": {ianaQuery: ianaReferral, ripeQuery: ripeRefusal},
"connection refused": {ianaServer: ianaReferral},
"only an error": {ianaServer: ianaReferral, ripeServer: ripeRefusal},
} {
t.Run(name, func(t *testing.T) {
t.Parallel()
@@ -812,7 +177,7 @@ func TestLookupASNFailsWhenTheRIRGivesNoAnswer(t *testing.T) {
func TestLookupASNReportsNoReferral(t *testing.T) {
t.Parallel()
_, err := lookUp(t, map[string]string{ianaQuery: ianaPrivateUse})
_, err := lookUp(t, map[string]string{ianaServer: ianaPrivateUse})
if !errors.Is(err, whois.ErrNoReferral) {
t.Errorf("LookupASN error = %v, want ErrNoReferral", err)
}
@@ -824,7 +189,7 @@ func TestLookupASNReportsNoReferral(t *testing.T) {
func TestLookupASNFailsWhenIANAAnswersNoBlock(t *testing.T) {
t.Parallel()
_, err := lookUp(t, map[string]string{ianaQuery: ianaNoObject})
_, err := lookUp(t, map[string]string{ianaServer: ianaNoObject})
if err == nil || errors.Is(err, whois.ErrNoReferral) {
t.Errorf("LookupASN error = %v, want one that is not ErrNoReferral",
err)
@@ -841,8 +206,8 @@ func TestLookupASNFailsWhenTheRIRAnswerDoesNotEnd(t *testing.T) {
strings.Repeat(fillerLine, endlessAnswerSize/len(fillerLine))
_, err := lookUp(t, map[string]string{
ianaQuery: ianaReferral,
ripeQuery: endless,
ianaServer: ianaReferral,
ripeServer: endless,
})
if err == nil || errors.Is(err, whois.ErrNoReferral) {
t.Errorf("LookupASN error = %v, want one that is not ErrNoReferral",
@@ -868,9 +233,9 @@ func TestLookupASNFailsOnAReferralToAnotherServer(t *testing.T) {
t.Parallel()
_, err := lookUp(t, map[string]string{
ianaQuery: "refer: " + refer + "\n\n" +
ianaServer: "refer: " + refer + "\n\n" +
"as-block: 3154-3353\n",
address + " " + asQuery: ripeRecord,
address: ripeRecord,
})
if err == nil || errors.Is(err, whois.ErrNoReferral) {
t.Errorf("LookupASN error = %v, want one that is not "+
pkg/asinfo/asdata.json.gz
BIN
View File
Binary file not shown.
pkg/asinfo/asinfo.go
+23 -49
View File
@@ -2,13 +2,10 @@ package asinfo
import (
"bytes"
"cmp"
"compress/gzip"
_ "embed"
"encoding/json"
"io"
"slices"
"strings"
"sync"
)
@@ -26,17 +23,6 @@ type Info struct {
type Registry struct {
mu sync.RWMutex
byASN map[int]*Info
// searchEntries holds every entry in AS number order, for Search.
searchEntries []searchEntry
}
// searchEntry is an entry with its handle and description in lower case. They
// are put in lower case once, when the data loads, so that Search only
// compares.
type searchEntry struct {
info *Info
lowerHandle string
lowerDescription string
}
var (
@@ -73,21 +59,9 @@ func initRegistry() {
panic("failed to unmarshal embedded AS data: " + err.Error())
}
slices.SortFunc(asInfos, func(a, b Info) int {
return cmp.Compare(a.ASN, b.ASN)
})
defaultRegistry.searchEntries = make([]searchEntry, 0, len(asInfos))
for i := range asInfos {
info := &asInfos[i]
defaultRegistry.byASN[info.ASN] = info
defaultRegistry.searchEntries = append(defaultRegistry.searchEntries,
searchEntry{
info: info,
lowerHandle: strings.ToLower(info.Handle),
lowerDescription: strings.ToLower(info.Description),
})
}
}
@@ -155,36 +129,36 @@ func All() []Info {
return result
}
// Search finds the AS entries whose handle or description contains query,
// ignoring case. It returns the first limit of them in AS number order, and the
// number of entries that match. An empty query matches nothing.
func Search(query string, limit int) ([]Info, int) {
// Search searches for AS entries where the handle or description contains the query
// This is a simple case-sensitive substring search
func Search(query string) []Info {
once.Do(initRegistry)
if query == "" {
return nil, 0
}
query = strings.ToLower(query)
defaultRegistry.mu.RLock()
defer defaultRegistry.mu.RUnlock()
var (
results []Info
total int
)
var results []Info
for _, entry := range defaultRegistry.searchEntries {
if strings.Contains(entry.lowerHandle, query) ||
strings.Contains(entry.lowerDescription, query) {
if total < limit {
results = append(results, *entry.info)
}
total++
for _, info := range defaultRegistry.byASN {
if contains(info.Handle, query) || contains(info.Description, query) {
results = append(results, *info)
}
}
return results, total
return results
}
// contains is a simple substring search
func contains(s, substr string) bool {
return len(substr) > 0 && len(s) >= len(substr) && containsImpl(s, substr)
}
func containsImpl(s, substr string) bool {
for i := 0; i <= len(s)-len(substr); i++ {
if s[i:i+len(substr)] == substr {
return true
}
}
return false
}
pkg/asinfo/asinfo_test.go
+20 -112
View File
@@ -1,8 +1,6 @@
package asinfo_test
import (
"cmp"
"slices"
"strings"
"testing"
@@ -22,7 +20,7 @@ func TestGet(t *testing.T) {
name: "AS1 Level 3",
asn: 1,
wantOK: true,
wantDesc: "Level 3 Parent LLC",
wantDesc: "Level 3 Parent, LLC",
},
{
name: "AS15169 Google",
@@ -36,10 +34,10 @@ func TestGet(t *testing.T) {
wantOK: false,
},
{
name: "AS1251 with accented letters",
asn: 1251,
name: "AS0 IANA Reserved",
asn: 0,
wantOK: true,
wantDesc: "FUNDAÇÃO DE AMPARO À PESQUISA DO ESTADO SÃO PAULO",
wantDesc: "Internet Assigned Numbers Authority",
},
}
@@ -69,7 +67,7 @@ func TestGetDescription(t *testing.T) {
asn int
want string
}{
{1, "Level 3 Parent LLC"},
{1, "Level 3 Parent, LLC"},
{999999999, ""},
}
@@ -92,7 +90,8 @@ func TestGetHandle(t *testing.T) {
asn int
want string
}{
{1, "LVLT-1"},
{1, "LVLT"},
{0, "IANA-RSVD"},
{999999999, ""},
}
@@ -142,9 +141,9 @@ func TestSearch(t *testing.T) {
wantMin: 10,
},
{
name: "Search for São Paulo",
query: "São Paulo",
wantMin: 1,
name: "Empty search",
query: "",
wantMin: 0,
},
{
name: "Non-existent org",
@@ -157,115 +156,26 @@ func TestSearch(t *testing.T) {
t.Run(tt.name, func(t *testing.T) {
t.Parallel()
results, _ := asinfo.Search(tt.query, asinfo.Total())
results := asinfo.Search(tt.query)
if len(results) < tt.wantMin {
t.Errorf("Search(%q) returned %d results, want at least %d",
tt.query, len(results), tt.wantMin)
}
// Verify all results contain the query, ignoring case
query := strings.ToLower(tt.query)
for _, r := range results {
if !strings.Contains(strings.ToLower(r.Handle), query) &&
!strings.Contains(strings.ToLower(r.Description), query) {
t.Errorf("Search result ASN %d doesn't contain query %q",
r.ASN, tt.query)
// Verify all results contain the query
if tt.query != "" {
for _, r := range results {
if !strings.Contains(r.Handle, tt.query) &&
!strings.Contains(r.Description, tt.query) {
t.Errorf("Search result ASN %d doesn't contain query %q",
r.ASN, tt.query)
}
}
}
})
}
}
// TestSearchEmptyQueryMatchesNothing expects an empty query to match no AS,
// although every handle contains the empty string.
func TestSearchEmptyQueryMatchesNothing(t *testing.T) {
t.Parallel()
results, total := asinfo.Search("", asinfo.Total())
if len(results) != 0 || total != 0 {
t.Errorf(`Search("") returned %d results and a total of %d, want none`,
len(results), total)
}
}
// TestSearchIgnoresCase searches for Cloudflare's handle in lower case and its
// description in upper case, and expects both searches to find AS13335.
func TestSearchIgnoresCase(t *testing.T) {
t.Parallel()
const cloudflareASN = 13335 // handle CLOUDFLARENET, "Cloudflare Inc."
isCloudflare := func(info asinfo.Info) bool {
return info.ASN == cloudflareASN
}
for _, query := range []string{"cloudflarenet", "CLOUDFLARE INC."} {
results, _ := asinfo.Search(query, asinfo.Total())
if !slices.ContainsFunc(results, isCloudflare) {
t.Errorf("Search(%q) did not find AS%d", query, cloudflareASN)
}
}
}
// TestSearchOrdersByASN expects the matches of a search with many of them to
// come in AS number order.
func TestSearchOrdersByASN(t *testing.T) {
t.Parallel()
const query = "University"
results, _ := asinfo.Search(query, asinfo.Total())
for i := 1; i < len(results); i++ {
if results[i-1].ASN >= results[i].ASN {
t.Fatalf("Search(%q): AS%d comes after AS%d",
query, results[i].ASN, results[i-1].ASN)
}
}
}
// TestSearchStopsAtLimit searches for a word that matches more entries than
// the limit. It expects exactly limit entries, the first matches in AS number
// order, and the number of all matches, counted here from All.
func TestSearchStopsAtLimit(t *testing.T) {
t.Parallel()
const (
query = "university"
limit = 10
)
var matches []asinfo.Info
for _, info := range asinfo.All() {
if strings.Contains(strings.ToLower(info.Handle), query) ||
strings.Contains(strings.ToLower(info.Description), query) {
matches = append(matches, info)
}
}
if len(matches) <= limit {
t.Fatalf("%q matches %d entries; the test needs more than %d",
query, len(matches), limit)
}
slices.SortFunc(matches, func(a, b asinfo.Info) int {
return cmp.Compare(a.ASN, b.ASN)
})
results, total := asinfo.Search(query, limit)
if total != len(matches) {
t.Errorf("Search(%q, %d): total %d, want %d",
query, limit, total, len(matches))
}
if !slices.Equal(results, matches[:limit]) {
t.Errorf("Search(%q, %d) = %v, want %v",
query, limit, results, matches[:limit])
}
}
func TestDataIntegrity(t *testing.T) {
t.Parallel()
@@ -305,13 +215,11 @@ func BenchmarkGet(b *testing.B) {
}
func BenchmarkSearch(b *testing.B) {
const limit = 50
queries := []string{"Google", "Amazon", "Microsoft", "University"}
b.ResetTimer()
for i := range b.N {
asinfo.Search(queries[i%len(queries)], limit)
asinfo.Search(queries[i%len(queries)])
}
}
pkg/asinfo/doc.go
+3 -5
View File
@@ -2,7 +2,7 @@
Package asinfo provides information about Autonomous Systems (AS).
The package embeds a comprehensive database of AS numbers, handles, and descriptions
sourced from https://github.com/ipverse/as-metadata. The data is embedded at compile
sourced from https://github.com/ipverse/asn-info. The data is embedded at compile
time and provides fast, offline lookups.
Basic usage:
@@ -16,10 +16,8 @@ Basic usage:
// Get just the description
desc := asinfo.GetDescription(15169) // "Google LLC"
// Search the handles and descriptions, ignoring case, for the first 10
// matches in AS number order and the number of all matches
results, total := asinfo.Search("university", 10)
fmt.Printf("%d matches\n", total)
// Search for AS entries
results := asinfo.Search("University")
for _, as := range results {
fmt.Printf("AS%d: %s\n", as.ASN, as.Description)
}
pkg/asinfo/example_test.go
+3 -10
View File
@@ -17,20 +17,13 @@ func ExampleGet() {
func ExampleGetDescription() {
desc := asinfo.GetDescription(13335)
fmt.Println(desc)
// Output: Cloudflare Inc.
// Output: Cloudflare, Inc.
}
func ExampleSearch() {
const limit = 10
results, total := asinfo.Search("mit-gateway", limit)
fmt.Printf("%d of %d matches\n", len(results), total)
results := asinfo.Search("MIT-GATEWAY")
for _, info := range results {
fmt.Printf("AS%d: %s - %s\n", info.ASN, info.Handle, info.Description)
}
// Output:
// 1 of 1 matches
// AS3: MIT-GATEWAYS - Massachusetts Institute of Technology
// Output: AS3: MIT-GATEWAYS - Massachusetts Institute of Technology
}
script/asupdate
-34
View File
@@ -1,34 +0,0 @@
#!/bin/sh
# script/asupdate: rebuild the AS list built into RouteWatch from
# as.csv with cmd/asinfo-gen, which fails when it finds no AS. The new
# list replaces the old one only when the generator succeeds. Otherwise
# the old list stays as it was and the script exits non-zero.
set -eu
ROOT="$(cd "$(dirname "$0")/.." && pwd -P)"
AS_DATA="pkg/asinfo/asdata.json.gz"
# Next to AS_DATA, so the final mv is a rename on one file system.
AS_DATA_TMP="$AS_DATA.tmp"
main() {
cd "$ROOT"
json="$(mktemp)"
trap 'rm -f "$json" "$AS_DATA_TMP"' EXIT
# A signal exits through the EXIT trap too.
trap 'exit 1' HUP INT TERM
echo "Updating AS info data..."
if ! go run ./cmd/asinfo-gen >"$json"; then
echo "asupdate: the generator failed; $AS_DATA is unchanged" >&2
exit 1
fi
# -n leaves the time out of the file, so an unchanged list gives the
# same bytes.
gzip -n -c <"$json" >"$AS_DATA_TMP"
mv "$AS_DATA_TMP" "$AS_DATA"
}
main "$@"