sneak/routewatch
1
0
Fork 0
Code Issues 5 Pull Requests 2 Actions 17 Packages Projects Releases Wiki Activity

Compare commits

base: sneak/routewatch:next
Branches Tags
sneak/routewatch:main sneak/routewatch:issue-122-routetable sneak/routewatch:next sneak/routewatch:prod sneak/routewatch:issue-27-cached-stats
..
compare: sneak/routewatch:c360a82aa2390cfd0c5ebcad7659c2b2b5ab64fc
Branches Tags
sneak/routewatch:issue-122-routetable sneak/routewatch:next sneak/routewatch:prod sneak/routewatch:main sneak/routewatch:issue-27-cached-stats
1 Commits
next .. c360a82aa2
Author SHA1 Message Date
sneak c360a82aa2 Make the home page's AS name search work (closes #151)
check / check (push) Canceled after 0s
Details 
The AS Name Search card on the home page said the search was coming soon.
It now asks the new GET /api/v1/search/as?q=<text> and lists the matches
as links to their AS pages. asinfo.Search ignores case, returns the first
matches up to a limit in AS number order, and counts all matches. An
empty q answers 400.

When the database has no record of an AS, its page and /api/v1/as/{asn}
take its handle and description from asinfo and still list its prefixes.
They answer 404 only for an AS with no record, no asinfo entry and no
prefix. AS 0 means no known origin: it always answers 404, and the search
never lists it.

Model: opus-5-5
 2026-10-26 21:20:12 +00:00
38 changed files with 388 additions and 4021 deletions
Expand all files Collapse all files
Makefile
+3 -2
View File
@@ -52,6 +52,7 @@ clean:
run: build
DEBUG=routewatch ./bin/routewatch 2>&1 | tee log.txt
# Rebuild the AS list built into RouteWatch from as.csv (needs the network).
asupdate:
@script/asupdate
@echo "Updating AS info data..."
@go run cmd/asinfo-gen/main.go | gzip > pkg/asinfo/asdata.json.gz.tmp && \
mv pkg/asinfo/asdata.json.gz.tmp pkg/asinfo/asdata.json.gz
README.md
+18 -98
View File
@@ -27,8 +27,6 @@ route the feed has withdrawn.
- Maintains live IPv4 and IPv6 routing tables
- Tracks AS peering relationships
- HTTP API for IP-to-AS lookups, prefix details, and AS information
- IP lookups report the public blocklists an address is on, from copies kept in
memory and on disk
- Automatic reconnection with exponential backoff
- Batched database writes for high performance
- Backpressure handling to prevent memory exhaustion
@@ -59,8 +57,7 @@ Every route answers `HEAD` as it answers `GET`, without the body.
### Web Interface
- `GET /` - Home page (HTML): headline figures, lookups by AS number and by IP
address, and a search of AS names. A card shows the `curl` command that looks
up the visitor's own address at `/ip/{addr}`, and that command's live answer.
address, and a search of AS names
- `GET /status` - HTML status dashboard, which reads `/api/v1/stats`
- `GET /status.json` - JSON statistics: feed, memory, database and route figures
- `GET /.well-known/healthcheck.json` - Health check (JSON): 200 when the
@@ -84,29 +81,19 @@ Every route answers `HEAD` as it answers `GET`, without the body.
### API v1
- `GET /api/v1/stats` - The `/status.json` figures plus connection, garbage
collector, stream and handler queue statistics. `stream.route_churn_per_min`
is the number of prefix announcements and withdrawals, from all peers, in the
last full clock minute.
- `GET /api/v1/ip/{ip}` - Looks up each address. `{ip}` is an IP address, a host
name, or a comma-separated list of at most 10 of these. `results` has one
entry for each address, with:
- `ip` and `ptr`: the address and its PTR names
- `netblock` and the fields beside it: the address's most specific live
route, and that route's origin AS with its WHOIS details
- `blocklists`: the public blocklists the address is on (see Blocklists)
An address with no live route has a result without the route and AS fields.
Of the addresses a host name resolves to, only public ones are looked up. A
name with none, such as one that resolves to a private address, is reported
as `host name has no public address`, without its addresses. `errors` has
one entry for each host name that failed. The answer is 400 when:
- `{ip}` lists more than 10
- `{ip}` comes to more than 10 addresses once its names are resolved
- nothing in `{ip}` is an IP address or a host name with a public address
A PTR lookup gets 2 seconds. An address whose lookup takes longer is
answered without PTR names.
collector, stream and handler queue statistics
- `GET /api/v1/ip/{ip}` - Looks up each address: its most specific live route
(`netblock`), that route's origin AS with its WHOIS details, and the address's
PTR names. `{ip}` is an IP address, a host name, or a comma-separated list of
at most 10 of these. Of the addresses a host name resolves to, only public
ones are looked up: a name with none, such as one that resolves to a private
address, is reported as `host name has no public address`, without its
addresses. `results` has one entry per address with a live route and `errors`
one per address or name that failed. The answer is 400 when `{ip}` lists more
than 10, when it comes to more than 10 addresses once its names are resolved,
or when nothing in it is an IP address or a host name with a public address;
and 404 when none of the addresses has a live route. A PTR lookup gets 2
seconds; an address whose lookup takes longer is answered without PTR names.
- `GET /api/v1/as/{asn}` - The AS and the prefixes it originates, IPv4 and IPv6
apart, each with when its newest route was updated. When the database has no
record of the AS, its handle and description come from the AS list built into
@@ -121,10 +108,9 @@ Every route answers `HEAD` as it answers `GET`, without the body.
`handle` and `description`
The search reads the AS list built into RouteWatch, not the database. The
list comes from
[ipverse/as-metadata](https://github.com/ipverse/as-metadata). A match can
therefore be an AS that the database has no record of. A `q` that is
missing, empty or only spaces answers 400.
list comes from [ipverse/asn-info](https://github.com/ipverse/asn-info). A
match can therefore be an AS that the database has no record of. A `q` that
is missing, empty or only spaces answers 400.
A JSON answer is `{"status": "ok", "data": ...}`, or for an error
`{"status": "error", "error": {"msg": ..., "code": ...}}`, and carries an
@@ -160,66 +146,6 @@ in a prefix's `origins`) means the route has no known origin: its AS path ends
in an AS set of several members, any of which may have originated it. The pages
show such a route as having no known origin.
## Blocklists
The IP lookup reports which public blocklists list each address. Each result has
a `blocklists` list with one entry for each list the address is on:
- `name`: the list's name, from the lists below
- `description`: what being on the list means
- `updated`: when RouteWatch last downloaded the list, or last heard from its
publisher that its copy was current
An address on no list has an empty `blocklists`. An address with no live route
still has a result with its `blocklists`.
The lists:
- `dshield`: the 20 networks that DShield saw scan the most targets in the last
three days. From [DShield](https://isc.sans.edu/feeds_doc.html), downloaded
every hour.
- `feodo_tracker`: botnet command-and-control servers. From abuse.ch
[Feodo Tracker](https://feodotracker.abuse.ch/blocklist/), downloaded every 15
minutes.
- `firehol_abusers_30d`: addresses reported for comment, form or forum spam in
the last 30 days. From
[FireHOL](https://iplists.firehol.org/?ipset=firehol_abusers_30d), downloaded
every 5 hours.
- `firehol_level3`: addresses reported for attacks, spyware or viruses in the
last 30 days. From
[FireHOL](https://iplists.firehol.org/?ipset=firehol_level3), downloaded every
3 hours.
- `spamhaus_drop` and `spamhaus_dropv6`: IPv4 and IPv6 networks that Spamhaus
says spammers or cyber criminals have hijacked or lease. From
[Spamhaus DROP](https://www.spamhaus.org/blocklists/do-not-route-or-peer/),
downloaded once a day. Spamhaus merged its EDROP list into DROP.
- `tor_exits`: Tor exit nodes, where traffic from the Tor network leaves. From
the [Tor Project](https://check.torproject.org/torbulkexitlist), downloaded
every hour.
No list is downloaded more often than its publisher updates it or allows. Each
can be downloaded whole without an account or a key.
How RouteWatch keeps the lists:
- It holds each list in memory, so a lookup makes no outside request.
- It keeps a copy of each list in the `blocklists` directory of the state
directory (see Configuration). Each copy is a file named after its list.
- Beside each copy, a file ending in `.validators` keeps the `Last-Modified` and
`ETag` the publisher sent with that copy.
- At startup it loads each list from its copy. It downloads a list at once when
the copy is older than the list's interval, or when there is no copy.
- A download sends the copy's `Last-Modified` as `If-Modified-Since` and its
`ETag` as `If-None-Match`. The publisher then sends the list only when it has
changed. Without either, the download asks for the whole list.
- A download fails when it takes over a minute or its file is over 16 MiB.
- When a download fails, or its file does not parse, RouteWatch keeps the
previous copy and logs the failure. It tries again one interval later.
A successful IP lookup answer carries `Cache-Control` with a `max-age` of 15
minutes, the shortest list interval. The answer for the caller's own address
(`/ip/`) is `private`, and the others are `public`.
## Code Structure
```
@@ -229,7 +155,6 @@ routewatch/
│ ├── asinfo-gen/ # Utility to generate AS info data
│ └── streamdumper/ # Debug utility for raw stream output
├── internal/
│ ├── blocklist/ # Public IP blocklists: downloads and lookups
│ ├── routewatch/ # Core application logic
│ ├── server/ # HTTP server and handlers
│ ├── database/ # SQLite storage layer
@@ -396,7 +321,7 @@ client sent. For a request from any other address they use the address the
request came from, whatever forwarding headers it carries. `X-Real-IP` and
`True-Client-IP` are not read.
State directory (the database and the blocklist copies):
State directory (database location):
- macOS: `~/Library/Application Support/berlin.sneak.app.routewatch/`
- Linux: `/var/lib/berlin.sneak.app.routewatch/` when running as root, otherwise
@@ -558,11 +483,6 @@ provide:
or `go.sum`) and `go fmt` (writes), then `script/check`
- `script/install-precommit` — install the git pre-commit hook that runs
`script/precommit`
- `script/asupdate` — rebuild the AS list built into RouteWatch,
`pkg/asinfo/asdata.json.gz`, from the `as.csv` file of
[ipverse/as-metadata](https://github.com/ipverse/as-metadata). It needs the
network. It runs `cmd/asinfo-gen`, which fails when it finds no AS. When
`cmd/asinfo-gen` fails, the old list stays and the script exits non-zero.
Tests and the linter run only in Docker, as the `test` and `lint` phases of the
`Dockerfile`; the image cannot be built unless both pass. Every `docker build`
TODO.md
-60
View File
@@ -33,66 +33,6 @@ tested.
# Completed Steps
- 2026-10-27: each result of the IP lookup lists the public blocklists the
address is on, in `blocklists`. The lists are DShield, Feodo Tracker, two
FireHOL lists, Spamhaus DROP and the Tor exit list. RouteWatch downloads each
list on its own schedule and answers from copies in memory. The copies are
also kept in the state directory's `blocklists` directory. IP lookup answers
carry `Cache-Control`. An address with no live route now has a result too,
without route fields, so the IP lookup no longer answers 404 (closes #158)
- 2026-10-27: the home page has an IP information API card. It shows the `curl`
command that looks up the visitor's own address at `/ip/{addr}`, and that
command's live answer (closes #156)
- 2026-10-27: on the AS page, a click on the heading of the IPv4 or the IPv6
prefix list folds or unfolds the list. Both lists start open (closes #155)
- 2026-10-27: the AS page lists its IPv4 and its IPv6 prefixes by mask length,
shortest first, and prefixes of one mask length by address. They used to be
listed by address (closes #153)
- 2026-10-27: on `/` and `/status`, a spinner beside the status turns one frame
each time the stats show more messages than the time before. A page that has
not seen more messages for three refreshes shows "stale" and the time since
then instead, in red (closes #152)
- 2026-10-27: the WHOIS details of a Brazilian AS keep their accented letters.
LACNIC's answer for such an AS comes from NIC.br, Brazil's registry, with each
accented letter replaced. The details now come from NIC.br's own server,
`whois.registro.br`. When that query fails, the details from LACNIC's answer
are kept and a warning is logged. After NIC.br answers that RouteWatch is over
its query limit, RouteWatch stops asking NIC.br for an hour (closes #174)
- 2026-10-27: `make asupdate` stops with an error when the download of `as.csv`
breaks off. The generator used to retry the failed read forever. A record that
does not parse is still logged and skipped (closes #177)
- 2026-10-27: `make asupdate` runs the new `script/asupdate`. It replaces the AS
list built into RouteWatch only when the generator succeeds and finds at least
one AS. A failed download used to replace the list with an empty file. The
generator reads `as.csv` from `ipverse/as-metadata`, the source's new name
(closes #173)
- 2026-10-27: the WHOIS details of an AS at RIPE, APNIC or AFRINIC come only
from its `aut-num` object and the organisation object it names. RIPE and APNIC
start their answer with the `as-block` object of the AS's block. Its dates
were stored as the AS's (closes #168)
- 2026-10-27: AS names keep their accented letters. The AS list built into
RouteWatch is rebuilt from today's `as.csv`, which added a country code
column. That file no longer lists AS 0 or the documentation AS numbers. A
WHOIS answer that is not valid UTF-8 is decoded as Latin-1, which RIPE and
APNIC send, before it is parsed and stored (closes #165)
- 2026-10-27: the Age column on the AS and prefix pages reads in words, such as
"5 minutes ago" or "2 weeks ago", instead of Go durations such as "360h0m0s
ago". An age under a minute reads in seconds, and one over a month in months
or years instead of a date (closes #163)
- 2026-10-26: a write to a stored AS, peering or peer keeps the earlier of the
two first-seen times and the later of the two last-seen times. An old update
from a collector catching up on a backlog no longer moves last seen back
(closes #162)
- 2026-10-26: the status page shows "Route Churn/min", the number of prefix
announcements and withdrawals, from all peers, in the last full clock minute.
`/api/v1/stats` reports it as `route_churn_per_min` in `stream`, which
replaces the moving average `route_churn_per_sec`. The messages and updates
per second on `/` and `/status` are whole numbers (closes #160)
- 2026-10-26: the WHOIS fetcher backs off more slowly. A failure makes its wait
1.5 times as long, up to one minute instead of five. Its queries to RIPE,
APNIC and AFRINIC leave out person and role objects, which RIPE counts against
a daily limit per address. A refused query is logged with the RIR's error line
(closes #159)
- 2026-10-26: the home page's AS name search lists the ASes whose handle or
description holds the text, as links to their pages, from the new
`/api/v1/search/as`; the search ignores case. When the database has no record
cmd/asinfo-gen/main.go
+28 -60
View File
@@ -6,7 +6,6 @@ import (
"encoding/csv"
"encoding/json"
"errors"
"fmt"
"io"
"log"
"net/http"
@@ -15,21 +14,7 @@ import (
"strings"
)
const asCsvURL = "https://github.com/ipverse/as-metadata/raw/refs/heads/master/as.csv"
// The first columns of as.csv, the ones the generator reads. It reads past
// the columns after them, such as the country code.
const (
asnColumn = "asn"
handleColumn = "handle"
descriptionColumn = "description"
columnsRead = 3
)
var (
errUnexpectedHeader = errors.New("unexpected CSV header")
errNoASes = errors.New("no AS in the CSV")
)
const asCsvURL = "https://github.com/ipverse/asn-info/raw/refs/heads/master/as.csv"
// ASInfo represents information about an Autonomous System
type ASInfo struct {
@@ -60,66 +45,40 @@ func main() {
log.Fatalf("Failed to fetch CSV: HTTP %d", resp.StatusCode)
}
asInfos, err := parseCSV(resp.Body)
// Closed here rather than deferred: log.Fatalf exits without running
// deferred calls.
_ = resp.Body.Close()
if err != nil {
log.Fatalf("Failed to parse CSV: %v", err)
}
log.Printf("Parsed %d AS records", len(asInfos))
// Convert to JSON and output to stdout
encoder := json.NewEncoder(os.Stdout)
encoder.SetIndent("", "\t")
err = encoder.Encode(asInfos)
if err != nil {
log.Fatalf("Failed to encode JSON: %v", err)
}
}
// parseCSV reads as.csv from r and returns its ASes, with the handle and the
// description as the file spells them. A header whose first columns are not
// the AS number, the handle and the description is an error. A record that
// does not parse is logged and skipped. A read that fails, such as on a broken
// download, is an error. A file left with no AS is an error.
func parseCSV(r io.Reader) ([]ASInfo, error) {
reader := csv.NewReader(r)
// Parse CSV
reader := csv.NewReader(resp.Body)
// Read header
header, err := reader.Read()
if err != nil {
return nil, fmt.Errorf("read CSV header: %w", err)
log.Fatalf("Failed to read CSV header: %v", err)
}
if len(header) < columnsRead || header[0] != asnColumn ||
header[1] != handleColumn || header[2] != descriptionColumn {
return nil, fmt.Errorf("%w: %v", errUnexpectedHeader, header)
if len(header) != 3 || header[0] != "asn" || header[1] != "handle" ||
header[2] != "description" {
log.Fatalf("Unexpected CSV header: %v", header)
}
// Read all records
var asInfos []ASInfo
for {
// The reader returns an error for a record with a different number
// of fields than the header.
record, err := reader.Read()
if errors.Is(err, io.EOF) {
break
}
if _, ok := errors.AsType[*csv.ParseError](err); ok {
if err != nil {
log.Printf("Error reading CSV record: %v", err)
continue
}
// Any other error comes from the download, such as a broken
// connection. Every later read returns it again.
if err != nil {
return nil, fmt.Errorf("read CSV record: %w", err)
const expectedFields = 3
if len(record) != expectedFields {
log.Printf("Skipping invalid record: %v", record)
continue
}
asn, err := strconv.Atoi(record[0])
@@ -136,9 +95,18 @@ func parseCSV(r io.Reader) ([]ASInfo, error) {
})
}
if len(asInfos) == 0 {
return nil, errNoASes
}
// Closed here rather than deferred: log.Fatalf exits without running
// deferred calls.
_ = resp.Body.Close()
return asInfos, nil
log.Printf("Parsed %d AS records", len(asInfos))
// Convert to JSON and output to stdout
encoder := json.NewEncoder(os.Stdout)
encoder.SetIndent("", "\t")
err = encoder.Encode(asInfos)
if err != nil {
log.Fatalf("Failed to encode JSON: %v", err)
}
}
cmd/asinfo-gen/main_test.go
-142
View File
@@ -1,142 +0,0 @@
package main
import (
"errors"
"io"
"slices"
"strings"
"testing"
"testing/iotest"
"time"
)
// parseTimeLimit is how long a test waits for parseCSV to return before it
// fails.
const parseTimeLimit = 2 * time.Second
// TestParseCSVReadsTodaysLayout parses an as.csv with the country code column
// that the source added after the description. It expects each AS with its
// handle and description spelled exactly as in the file, accents included.
func TestParseCSVReadsTodaysLayout(t *testing.T) {
t.Parallel()
const asCSV = `asn,handle,description,country-code
1,LVLT-1,Level 3 Parent LLC,US
1251,FUNDACAO-AMPARO-PESQUISA,FUNDAÇÃO DE AMPARO À PESQUISA DO ESTADO SÃO PAULO,BR
`
got, err := parseCSV(strings.NewReader(asCSV))
if err != nil {
t.Fatalf("parseCSV: %v", err)
}
want := []ASInfo{
{ASN: 1, Handle: "LVLT-1", Description: "Level 3 Parent LLC"},
{
ASN: 1251,
Handle: "FUNDACAO-AMPARO-PESQUISA",
Description: "FUNDAÇÃO DE AMPARO À PESQUISA DO ESTADO " +
"SÃO PAULO",
},
}
if !slices.Equal(got, want) {
t.Errorf("parseCSV = %+v, want %+v", got, want)
}
}
// TestParseCSVRejectsAnotherHeader parses files whose first columns are not
// the AS number, the handle and the description. It expects an error rather
// than a list read from the wrong columns.
func TestParseCSVRejectsAnotherHeader(t *testing.T) {
t.Parallel()
for _, asCSV := range []string{
"asn,handle\n1,LVLT-1\n",
"asn,description,handle\n1,Level 3 Parent LLC,LVLT-1\n",
} {
_, err := parseCSV(strings.NewReader(asCSV))
if err == nil {
t.Errorf("parseCSV(%q): no error", asCSV)
}
}
}
// TestParseCSVRejectsAFileWithNoASes parses files that hold no AS, one with
// only the header and one whose only record has no valid AS number. It
// expects errNoASes, so that make asupdate keeps the old list.
func TestParseCSVRejectsAFileWithNoASes(t *testing.T) {
t.Parallel()
for _, asCSV := range []string{
"asn,handle,description,country-code\n",
"asn,handle,description,country-code\nAS1,LVLT-1,Level 3,US\n",
} {
_, err := parseCSV(strings.NewReader(asCSV))
if !errors.Is(err, errNoASes) {
t.Errorf("parseCSV(%q) = %v, want %v", asCSV, err, errNoASes)
}
}
}
// TestParseCSVSkipsARecordThatDoesNotParse parses an as.csv with a record
// that has too few fields between two valid records. It expects both valid
// ASes and no error, so that one bad record does not stop make asupdate.
func TestParseCSVSkipsARecordThatDoesNotParse(t *testing.T) {
t.Parallel()
const asCSV = "asn,handle,description,country-code\n" +
"1,LVLT-1,Level 3 Parent LLC,US\n" +
"2,UDEL-DCN\n" +
"3,MIT-GATEWAYS,Massachusetts Institute of Technology,US\n"
got, err := parseCSV(strings.NewReader(asCSV))
if err != nil {
t.Fatalf("parseCSV: %v", err)
}
want := []ASInfo{
{ASN: 1, Handle: "LVLT-1", Description: "Level 3 Parent LLC"},
{
ASN: 3,
Handle: "MIT-GATEWAYS",
Description: "Massachusetts Institute of Technology",
},
}
if !slices.Equal(got, want) {
t.Errorf("parseCSV = %+v, want %+v", got, want)
}
}
// TestParseCSVStopsWhenTheDownloadBreaksOff parses an as.csv whose download
// breaks off after the first record. From then on every read fails, as on a
// broken connection. It expects parseCSV to return that error, so that make
// asupdate keeps the old list.
func TestParseCSVStopsWhenTheDownloadBreaksOff(t *testing.T) {
t.Parallel()
const asCSV = "asn,handle,description,country-code\n" +
"1,LVLT-1,Level 3 Parent LLC,US\n"
download := io.MultiReader(
strings.NewReader(asCSV), iotest.ErrReader(io.ErrUnexpectedEOF))
// parseCSV runs on its own goroutine, so that a parseCSV that never
// returns fails the test instead of hanging it.
parsed := make(chan error, 1)
go func() {
_, err := parseCSV(download)
parsed <- err
}()
select {
case err := <-parsed:
if !errors.Is(err, io.ErrUnexpectedEOF) {
t.Errorf("parseCSV = %v, want %v", err, io.ErrUnexpectedEOF)
}
case <-time.After(parseTimeLimit):
t.Fatalf("parseCSV did not return within %v", parseTimeLimit)
}
}
go.mod
-2
View File
@@ -9,9 +9,7 @@ require (
github.com/mattn/go-sqlite3 v1.14.29
github.com/rcrowley/go-metrics v0.0.0-20250401214520-65e299d6c5c9
go.uber.org/fx v1.24.0
go4.org/netipx v0.0.0-20260823151212-3075585bcbeb
golang.org/x/term v0.33.0
golang.org/x/text v0.43.0
)
require (
go.sum
-4
View File
@@ -24,13 +24,9 @@ go.uber.org/multierr v1.10.0 h1:S0h4aNzvfcFsC3dRF1jLoaov7oRaKqRGC/pUEJ2yvPQ=
go.uber.org/multierr v1.10.0/go.mod h1:20+QtiLqy0Nd6FdQB9TLXag12DsQkrbs3htMFfDN80Y=
go.uber.org/zap v1.26.0 h1:sI7k6L95XOKS281NhVKOFCUNIvv9e0w4BF8N3u+tCRo=
go.uber.org/zap v1.26.0/go.mod h1:dtElttAiwGvoJ/vj4IwHBS/gXsEu/pZ50mUIRWuG0so=
go4.org/netipx v0.0.0-20260823151212-3075585bcbeb h1:XBM4hvfwGAttkkiTIFfeigdfcL1xIfdKXqFdgiHGtDs=
go4.org/netipx v0.0.0-20260823151212-3075585bcbeb/go.mod h1:PLyyIXexvUFg3Owu6p/WfdlivPbZJsZdgWZlrGope/Y=
golang.org/x/sys v0.34.0 h1:H5Y5sJ2L2JRdyv7ROF1he/lPdvFsd0mJHFw2ThKHxLA=
golang.org/x/sys v0.34.0/go.mod h1:BJP2sWEmIv4KK5OTEluFJCKSidICx8ciO85XgH3Ak8k=
golang.org/x/term v0.33.0 h1:NuFncQrRcaRvVmgRkvM3j/F00gWIAlcmlB8ACEKmGIg=
golang.org/x/term v0.33.0/go.mod h1:s18+ql9tYWp1IfpV9DmCtQDDSRBUjKaw9M1eAv5UeF0=
golang.org/x/text v0.43.0 h1:1QivrlhwAsnMOcqgOdl9my7s9OyZl8O5/UGUTJVXoWc=
golang.org/x/text v0.43.0/go.mod h1:hbSIYA/amXcRXTFSZqh8tWHEoqRvvbKPJ2uz/KU+a+s=
gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA=
gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
internal/blocklist/blocklist.go
-184
View File
@@ -1,184 +0,0 @@
// Package blocklist downloads public IP blocklists on a schedule, keeps the
// last good copy of each on disk and in memory, and answers which lists
// contain an address.
package blocklist
import (
"context"
"log/slog"
"net/netip"
"sync"
"time"
"go4.org/netipx"
)
// Format is how a list's file is written.
type Format int
const (
// FormatLines has one address or network per line, such as 192.0.2.1 or
// 192.0.2.0/24. Lines that start with # are comments.
FormatLines Format = iota
// FormatSpamhausJSON has one JSON object per line, with a network in its
// cidr field. The last line holds the file's metadata and no network.
FormatSpamhausJSON
// FormatDShield has one network per line, as tab-separated fields that
// start with the network's first and last address. Lines that start
// with # are comments.
FormatDShield
)
// List is one public blocklist.
type List struct {
// Name names the list in lookup answers, and names its copy on disk.
Name string
// Description says in one line what being on the list means.
Description string
// URL is where the publisher serves the whole list.
URL string
// Interval is how long a copy is used before the list is downloaded
// again.
Interval time.Duration
// Format is how the list's file is written.
Format Format
}
// Listing says that a list contains an address.
type Listing struct {
Name string `json:"name"`
Description string `json:"description"`
// Updated is when routewatch last downloaded the list, or last heard from
// its publisher that its copy was still current.
Updated time.Time `json:"updated"`
}
// listCopy is a list's last good copy in memory.
type listCopy struct {
addresses *netipx.IPSet
updated time.Time
// validators identify the version of the list this copy holds.
validators validators
}
// Params is what New needs.
type Params struct {
// Dir is the directory that holds the copies on disk.
Dir string
// Lists are the lists to download and look addresses up in.
Lists []List
Logger *slog.Logger
}
// Blocklists keeps a copy of each list and answers which lists contain an
// address. Start loads the copies from disk and starts downloading the lists.
type Blocklists struct {
dir string
lists []List
logger *slog.Logger
// copies holds each list's last good copy, by list name. A list with no
// good copy yet has none.
mu sync.RWMutex
copies map[string]listCopy
// stop ends the goroutine Start runs, and done is closed when it returns.
stop context.CancelFunc
done chan struct{}
}
// New returns a Blocklists with no copies. Start loads them.
func New(params Params) *Blocklists {
return &Blocklists{
dir: params.Dir,
lists: params.Lists,
logger: params.Logger.With("component", "blocklists"),
copies: make(map[string]listCopy),
done: make(chan struct{}),
}
}
// Lookup returns a Listing for each list whose copy contains addr, in the
// order of the lists given to New. It makes no outside request. When no list
// contains addr, it returns an empty list, not nil.
func (b *Blocklists) Lookup(addr netip.Addr) []Listing {
b.mu.RLock()
defer b.mu.RUnlock()
listings := []Listing{}
for _, list := range b.lists {
c, ok := b.copies[list.Name]
if ok && c.addresses.Contains(addr.Unmap()) {
listings = append(listings, Listing{
Name: list.Name,
Description: list.Description,
Updated: c.updated,
})
}
}
return listings
}
// ShortestInterval returns the shortest Interval of the lists. That is the
// soonest a copy can change, and so how long a lookup answer stays current.
func (b *Blocklists) ShortestInterval() time.Duration {
var shortest time.Duration
for i, list := range b.lists {
if i == 0 || list.Interval < shortest {
shortest = list.Interval
}
}
return shortest
}
// updated returns when the named list's copy was last downloaded or found
// current, or the zero time when the list has no copy.
func (b *Blocklists) updated(name string) time.Time {
b.mu.RLock()
defer b.mu.RUnlock()
return b.copies[name].updated
}
// copyValidators returns the validators of the named list's copy. They are
// empty when the list has no copy.
func (b *Blocklists) copyValidators(name string) validators {
b.mu.RLock()
defer b.mu.RUnlock()
return b.copies[name].validators
}
// setCopy makes c the named list's copy.
func (b *Blocklists) setCopy(name string, c listCopy) {
b.mu.Lock()
defer b.mu.Unlock()
b.copies[name] = c
}
// setUpdated records that the named list's copy was found current at t.
func (b *Blocklists) setUpdated(name string, t time.Time) {
b.mu.Lock()
defer b.mu.Unlock()
c, ok := b.copies[name]
if ok {
c.updated = t
b.copies[name] = c
}
}
internal/blocklist/blocklist_internal_test.go
-494
View File
@@ -1,494 +0,0 @@
package blocklist
import (
"context"
"errors"
"io"
"io/fs"
"log/slog"
"net/http"
"net/http/httptest"
"net/netip"
"os"
"path/filepath"
"strings"
"sync"
"testing"
"time"
)
// The test list's name and description, the address its good version lists,
// and the address a newer version lists instead.
const (
testListName = "test_list"
testListDescription = "On the test list"
listedAddr = "192.0.2.1"
goodListBody = "# A test list\n" + listedAddr + "\n"
newListedAddr = "192.0.2.2"
newListBody = "# A test list\n" + newListedAddr + "\n"
)
// testETag is the ETag a publisher sends with a version of the test list.
const testETag = `"v1"`
// maxWait is how long a test waits for the download goroutine.
const maxWait = 10 * time.Second
// publisher stands in for a list's publisher. It answers every request with
// its status and body, and keeps the headers of each request. A 200 answer
// carries lastModified and etag, when set, as Last-Modified and ETag. As
// from a real publisher, a request whose If-None-Match or If-Modified-Since
// matches them is answered "not modified".
type publisher struct {
mu sync.Mutex
status int
body string
lastModified time.Time
etag string
headers []http.Header
}
// newPublisher starts a publisher that answers 200 with body, and returns it
// and its URL.
func newPublisher(t *testing.T, body string) (*publisher, string) {
t.Helper()
p := &publisher{status: http.StatusOK, body: body}
srv := httptest.NewServer(p)
t.Cleanup(srv.Close)
return p, srv.URL
}
func (p *publisher) ServeHTTP(w http.ResponseWriter, r *http.Request) {
p.mu.Lock()
defer p.mu.Unlock()
p.headers = append(p.headers, r.Header.Clone())
if p.status != http.StatusOK {
w.WriteHeader(p.status)
_, _ = io.WriteString(w, p.body)
return
}
if p.etag != "" {
w.Header().Set(etagHeader, p.etag)
}
// ServeContent sends Last-Modified unless lastModified is zero, and
// answers "not modified" when the request's conditions match.
http.ServeContent(w, r, testListName, p.lastModified,
strings.NewReader(p.body))
}
// answer makes the publisher answer later requests with status and body.
func (p *publisher) answer(status int, body string) {
p.mu.Lock()
defer p.mu.Unlock()
p.status, p.body = status, body
}
// publish makes the publisher answer later requests with a version of the
// list: body, with lastModified and etag. A zero lastModified or an empty
// etag is not sent.
func (p *publisher) publish(body string, lastModified time.Time, etag string) {
p.mu.Lock()
defer p.mu.Unlock()
p.status, p.body = http.StatusOK, body
p.lastModified, p.etag = lastModified, etag
}
// requests returns the headers of the requests the publisher has had.
func (p *publisher) requests() []http.Header {
p.mu.Lock()
defer p.mu.Unlock()
return append([]http.Header(nil), p.headers...)
}
// testList returns a list served at url, written in format.
func testList(url string, format Format) List {
return List{
Name: testListName,
Description: testListDescription,
URL: url,
Interval: time.Hour,
Format: format,
}
}
func newTestBlocklists(dir string, list List) *Blocklists {
return New(Params{
Dir: dir,
Lists: []List{list},
Logger: slog.New(slog.DiscardHandler),
})
}
// lookup looks addr up in b.
func lookup(t *testing.T, b *Blocklists, addr string) []Listing {
t.Helper()
return b.Lookup(netip.MustParseAddr(addr))
}
// TestLookupFindsListedAddresses downloads a list in each format and looks
// up addresses on it, inside its networks, and off it, in IPv4 and IPv6.
// A listed address has one listing that names the list; any other has an
// empty list of listings.
func TestLookupFindsListedAddresses(t *testing.T) {
t.Parallel()
tests := []struct {
name string
format Format
body string
listed []string
unlisted []string
}{
{
name: "single addresses",
format: FormatLines,
body: "# comment\n192.0.2.1\n\n2001:db8::1\n",
listed: []string{"192.0.2.1", "::ffff:192.0.2.1", "2001:db8::1"},
unlisted: []string{"192.0.2.2", "2001:db8::2"},
},
{
name: "networks",
format: FormatLines,
body: "# comment\n198.51.100.0/24\n2001:db8:1::/48\n",
listed: []string{"198.51.100.0", "198.51.100.255", "2001:db8:1::5"},
unlisted: []string{"198.51.101.0", "2001:db8:2::1"},
},
{
name: "Spamhaus networks",
format: FormatSpamhausJSON,
body: `{"cidr":"203.0.113.0/25","sblid":"SBL1","rir":"arin"}` + "\n" +
`{"cidr":"2001:db8:3::/48","sblid":"SBL2","rir":"ripencc"}` + "\n" +
`{"type":"metadata","timestamp":1793074442,"records":2}` + "\n",
listed: []string{"203.0.113.1", "2001:db8:3::1"},
unlisted: []string{"203.0.113.200", "2001:db8:4::1"},
},
{
name: "DShield networks",
format: FormatDShield,
body: "# DShield.org Recommended Block List\n" +
"192.0.2.0\t192.0.2.255\t24\t327\tEXAMPLE\tUS\tNone\n",
listed: []string{"192.0.2.0", "192.0.2.77"},
unlisted: []string{"192.0.3.1", "2001:db8::1"},
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
t.Parallel()
_, url := newPublisher(t, tt.body)
b := newTestBlocklists(t.TempDir(), testList(url, tt.format))
err := b.download(t.Context(), b.lists[0])
if err != nil {
t.Fatalf("download: %v", err)
}
for _, addr := range tt.listed {
listings := lookup(t, b, addr)
if len(listings) != 1 || listings[0].Name != testListName ||
listings[0].Description != testListDescription ||
listings[0].Updated.IsZero() {
t.Errorf("%s: listings %+v, want one for %s", addr,
listings, testListName)
}
}
for _, addr := range tt.unlisted {
listings := lookup(t, b, addr)
if listings == nil || len(listings) != 0 {
t.Errorf("%s: listings %#v, want an empty list", addr, listings)
}
}
})
}
}
// TestFailedDownloadKeepsThePreviousCopy downloads a good version of a list,
// then one that fails. The good copy stays in memory and on disk, and no
// part of the failed download is left beside it.
func TestFailedDownloadKeepsThePreviousCopy(t *testing.T) {
t.Parallel()
tests := []struct {
name string
status int
body string
}{
{"server error", http.StatusInternalServerError, "unavailable"},
{"not a list", http.StatusOK, "<!DOCTYPE html>\n<html></html>\n"},
{
"over the size limit", http.StatusOK,
strings.Repeat(listedAddr+"\n", maxDownloadSize/len(listedAddr)),
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
t.Parallel()
dir := t.TempDir()
p, url := newPublisher(t, goodListBody)
b := newTestBlocklists(dir, testList(url, FormatLines))
err := b.download(t.Context(), b.lists[0])
if err != nil {
t.Fatalf("first download: %v", err)
}
before := lookup(t, b, listedAddr)
p.answer(tt.status, tt.body)
err = b.download(t.Context(), b.lists[0])
if err == nil {
t.Fatal("second download returned no error")
}
after := lookup(t, b, listedAddr)
if len(after) != 1 || len(before) != 1 ||
!after[0].Updated.Equal(before[0].Updated) {
t.Errorf("listings %+v after the failure, want %+v", after, before)
}
copies := os.DirFS(dir)
onDisk, err := fs.ReadFile(copies, testListName)
if err != nil || string(onDisk) != goodListBody {
t.Errorf("copy on disk %q (%v), want %q", onDisk, err, goodListBody)
}
_, err = fs.Stat(copies, testListName+downloadSuffix)
if !errors.Is(err, fs.ErrNotExist) {
t.Errorf("the failed download's file is still there (%v)", err)
}
})
}
}
// TestRestartLoadsCopiesFromDisk downloads a list, then starts a new
// Blocklists on the same directory, as a restart does. The new one answers
// from the copy on disk at once, and does not download the list again while
// the copy is younger than the list's interval.
func TestRestartLoadsCopiesFromDisk(t *testing.T) {
t.Parallel()
dir := t.TempDir()
p, url := newPublisher(t, goodListBody)
list := testList(url, FormatLines)
first := newTestBlocklists(dir, list)
err := first.download(t.Context(), list)
if err != nil {
t.Fatalf("download: %v", err)
}
downloaded := lookup(t, first, listedAddr)[0].Updated
second := newTestBlocklists(dir, list)
second.Start(t.Context())
second.Stop(t.Context())
listings := lookup(t, second, listedAddr)
if len(listings) != 1 ||
listings[0].Updated.Sub(downloaded).Abs() > time.Second {
t.Errorf("after the restart, listings %+v, want one updated at %s",
listings, downloaded)
}
if got := len(p.requests()); got != 1 {
t.Errorf("the publisher had %d requests, want 1", got)
}
}
// TestStartRefreshesAnOldCopy starts on a copy older than its list's
// interval. The list is downloaded at once, with routewatch's User-Agent and
// the Last-Modified and ETag the publisher sent with the copy, kept on disk.
// The publisher answers that the copy is still current, so the copy is kept
// and marked as updated now.
func TestStartRefreshesAnOldCopy(t *testing.T) {
t.Parallel()
dir := t.TempDir()
p, url := newPublisher(t, goodListBody)
list := testList(url, FormatLines)
lastModified := time.Now().Add(-3 * list.Interval).UTC().Truncate(time.Second)
p.publish(goodListBody, lastModified, testETag)
err := newTestBlocklists(dir, list).download(t.Context(), list)
if err != nil {
t.Fatalf("download: %v", err)
}
old := time.Now().Add(-2 * list.Interval).UTC().Truncate(time.Second)
err = os.Chtimes(filepath.Join(dir, testListName), old, old)
if err != nil {
t.Fatalf("Chtimes: %v", err)
}
b := newTestBlocklists(dir, list)
b.Start(t.Context())
deadline := time.Now().Add(maxWait)
for !lookup(t, b, listedAddr)[0].Updated.After(old) {
if time.Now().After(deadline) {
t.Fatalf("the copy was not marked updated within %s", maxWait)
}
time.Sleep(time.Millisecond)
}
b.Stop(t.Context())
requests := p.requests()
last := requests[len(requests)-1]
want := lastModified.Format(http.TimeFormat)
if got := last.Get(ifModifiedSinceHeader); got != want {
t.Errorf("If-Modified-Since %q, want %q", got, want)
}
if got := last.Get(ifNoneMatchHeader); got != testETag {
t.Errorf("If-None-Match %q, want %q", got, testETag)
}
if got := last.Get(userAgentHeader); got != userAgent {
t.Errorf("User-Agent %q, want %q", got, userAgent)
}
}
// TestNewVersionOlderThanTheLastDownloadReplacesTheCopy downloads a list.
// The publisher then puts out a new version whose Last-Modified is earlier
// than that download, as when a cache in front of the publisher passes the
// version on late. The next download sends the Last-Modified of the copy,
// not the time of the download, so it gets the new version.
func TestNewVersionOlderThanTheLastDownloadReplacesTheCopy(t *testing.T) {
t.Parallel()
p, url := newPublisher(t, goodListBody)
list := testList(url, FormatLines)
b := newTestBlocklists(t.TempDir(), list)
firstVersion := time.Now().Add(-2 * list.Interval).UTC().Truncate(time.Second)
p.publish(goodListBody, firstVersion, "")
err := b.download(t.Context(), list)
if err != nil {
t.Fatalf("first download: %v", err)
}
p.publish(newListBody, firstVersion.Add(list.Interval), "")
err = b.download(t.Context(), list)
if err != nil {
t.Fatalf("second download: %v", err)
}
if listings := lookup(t, b, newListedAddr); len(listings) != 1 {
t.Errorf("%s: listings %+v, want one from the new version",
newListedAddr, listings)
}
want := firstVersion.Format(http.TimeFormat)
if got := p.requests()[1].Get(ifModifiedSinceHeader); got != want {
t.Errorf("If-Modified-Since %q, want %q", got, want)
}
}
// TestDownloadWithoutLastModifiedOrETagSendsNoCondition downloads a list
// twice from a publisher that sends neither Last-Modified nor ETag. Neither
// request carries If-Modified-Since or If-None-Match, so the second download
// gets the whole list again.
func TestDownloadWithoutLastModifiedOrETagSendsNoCondition(t *testing.T) {
t.Parallel()
p, url := newPublisher(t, goodListBody)
list := testList(url, FormatLines)
b := newTestBlocklists(t.TempDir(), list)
err := b.download(t.Context(), list)
if err != nil {
t.Fatalf("first download: %v", err)
}
err = b.download(t.Context(), list)
if err != nil {
t.Fatalf("second download: %v", err)
}
for i, request := range p.requests() {
if request.Get(ifModifiedSinceHeader) != "" ||
request.Get(ifNoneMatchHeader) != "" {
t.Errorf("request %d has a condition: %v", i, request)
}
}
}
// TestStopCancelsADownloadInProgress stops while the publisher has not
// answered. Stop cancels the download and returns once the download
// goroutine has stopped.
func TestStopCancelsADownloadInProgress(t *testing.T) {
t.Parallel()
arrived := make(chan struct{})
var once sync.Once
srv := httptest.NewServer(http.HandlerFunc(
func(_ http.ResponseWriter, r *http.Request) {
once.Do(func() { close(arrived) })
<-r.Context().Done()
}))
t.Cleanup(srv.Close)
b := newTestBlocklists(t.TempDir(), testList(srv.URL, FormatLines))
b.Start(t.Context())
select {
case <-arrived:
case <-time.After(maxWait):
t.Fatalf("no download began within %s", maxWait)
}
ctx, cancel := context.WithTimeout(t.Context(), maxWait)
defer cancel()
b.Stop(ctx)
select {
case <-b.done:
default:
t.Error("Stop returned before the download goroutine stopped")
}
}
// TestPublicListsAreComplete checks that every public list has what a
// download and a lookup answer need, and a unique name.
func TestPublicListsAreComplete(t *testing.T) {
t.Parallel()
names := make(map[string]bool)
for _, list := range Lists() {
if list.Name == "" || list.Description == "" || list.URL == "" ||
list.Interval <= 0 || names[list.Name] {
t.Errorf("list %+v is incomplete or has a name used before", list)
}
names[list.Name] = true
}
}
internal/blocklist/download.go
-379
View File
@@ -1,379 +0,0 @@
package blocklist
import (
"context"
"encoding/json"
"errors"
"fmt"
"io"
"io/fs"
"net/http"
"os"
"slices"
"time"
"go4.org/netipx"
)
// Limits of one download.
const (
// downloadTimeout is how long one download may take.
downloadTimeout = time.Minute
// maxDownloadSize is the size of the largest list file accepted, in
// bytes. The largest list, FireHOL abusers, is about 2.5 MiB.
maxDownloadSize = 16 << 20
)
// The request headers a download sets.
const (
userAgentHeader = "User-Agent"
ifModifiedSinceHeader = "If-Modified-Since"
ifNoneMatchHeader = "If-None-Match"
// userAgent names routewatch to the publishers.
userAgent = "routewatch (+https://git.eeqj.de/sneak/routewatch)"
)
// The response headers that identify the version of a list a publisher sent.
const (
lastModifiedHeader = "Last-Modified"
etagHeader = "ETag"
)
const (
// downloadSuffix ends the name of the file a download is written to,
// beside the list's copy, until it has parsed.
downloadSuffix = ".download"
// validatorsSuffix ends the name of the file beside a list's copy that
// keeps the copy's validators.
validatorsSuffix = ".validators"
// dirPermissions are those of the directory that holds the copies.
dirPermissions = 0o750
// validatorsPermissions are those of the file that keeps a copy's
// validators.
validatorsPermissions = 0o600
)
// validators are the Last-Modified and ETag headers a publisher sent with a
// version of a list. A download sends them back, as If-Modified-Since and
// If-None-Match, so that the publisher sends the list only when it has
// changed. They are kept as the publisher wrote them.
//
//nolint:tagliatelle // snake_case keys, as the Go style guide asks
type validators struct {
LastModified string `json:"last_modified,omitempty"`
ETag string `json:"etag,omitempty"`
}
var (
errUnexpectedStatus = errors.New("unexpected HTTP status")
errTooLarge = errors.New("list is over the size limit")
)
// Start loads each list's copy from disk. It then starts the goroutine that
// downloads each list once its copy is older than its Interval, and a list
// with no copy at once. The goroutine runs until Stop: the end of ctx does
// not stop it.
func (b *Blocklists) Start(ctx context.Context) {
err := os.MkdirAll(b.dir, dirPermissions)
if err != nil {
b.logger.Error("Failed to create the blocklists directory",
"dir", b.dir, "error", err)
}
nextDownload := make([]time.Time, len(b.lists))
for i, list := range b.lists {
loadErr := b.loadCopy(list)
if loadErr != nil && !errors.Is(loadErr, fs.ErrNotExist) {
b.logger.Error("Failed to load a blocklist copy",
"list", list.Name, "error", loadErr)
}
// A list with no copy has the zero time, so it is due at once.
nextDownload[i] = b.updated(list.Name).Add(list.Interval)
}
ctx, b.stop = context.WithCancel(context.WithoutCancel(ctx))
go func() {
b.run(ctx, nextDownload)
}()
b.logger.Info("Blocklist downloads started", "dir", b.dir)
}
// Stop stops the downloads, cancelling one in progress, and waits until the
// goroutine has returned or ctx ends. The copies in memory stay, so Lookup
// still answers.
func (b *Blocklists) Stop(ctx context.Context) {
b.stop()
select {
case <-b.done:
b.logger.Info("Blocklist downloads stopped")
case <-ctx.Done():
b.logger.Warn("Stop deadline passed before the blocklist downloads " +
"stopped")
}
}
// run downloads each list when nextDownload says, until ctx ends. After each
// try, whether it worked or not, the list's next download is an Interval
// later.
func (b *Blocklists) run(ctx context.Context, nextDownload []time.Time) {
defer close(b.done)
if len(nextDownload) == 0 {
return
}
for {
soonest := slices.MinFunc(nextDownload, time.Time.Compare)
timer := time.NewTimer(time.Until(soonest))
select {
case <-ctx.Done():
timer.Stop()
return
case <-timer.C:
}
for i, list := range b.lists {
if time.Now().Before(nextDownload[i]) {
continue
}
err := b.download(ctx, list)
if ctx.Err() != nil {
return
}
if err != nil {
b.logger.Error("Blocklist download failed; keeping the "+
"previous copy", "list", list.Name, "error", err)
}
nextDownload[i] = time.Now().Add(list.Interval)
}
}
}
// download asks the publisher for list, sending the validators of the list's
// copy when it has them. A new version that parses becomes the list's copy,
// on disk and in memory. An answer that the copy is still current marks the
// copy as updated now. Anything else leaves the copy as it is and returns an
// error.
func (b *Blocklists) download(ctx context.Context, list List) error {
ctx, cancel := context.WithTimeout(ctx, downloadTimeout)
defer cancel()
req, err := http.NewRequestWithContext(ctx, http.MethodGet, list.URL, nil)
if err != nil {
return err
}
req.Header.Set(userAgentHeader, userAgent)
held := b.copyValidators(list.Name)
if held.LastModified != "" {
req.Header.Set(ifModifiedSinceHeader, held.LastModified)
}
if held.ETag != "" {
req.Header.Set(ifNoneMatchHeader, held.ETag)
}
resp, err := http.DefaultClient.Do(req)
if err != nil {
return err
}
defer func() { _ = resp.Body.Close() }()
switch resp.StatusCode {
case http.StatusOK:
return b.replaceCopy(list, resp.Body, validators{
LastModified: resp.Header.Get(lastModifiedHeader),
ETag: resp.Header.Get(etagHeader),
})
case http.StatusNotModified:
return b.markCurrent(list)
default:
return fmt.Errorf("%w: %s", errUnexpectedStatus, resp.Status)
}
}
// replaceCopy writes a new version of list, read from body, beside the list's
// copy. Only when it parses does it replace the copy, on disk and in memory.
// sent are the validators the publisher sent with the new version.
func (b *Blocklists) replaceCopy(list List, body io.Reader, sent validators) error {
root, err := os.OpenRoot(b.dir)
if err != nil {
return err
}
defer func() { _ = root.Close() }()
downloadName := list.Name + downloadSuffix
addresses, err := writeAndParse(root, downloadName, body, list.Format)
if err != nil {
_ = root.Remove(downloadName)
return err
}
// The old copy's validators are removed before the copy is replaced, so
// that they are never kept with the new copy.
err = root.Remove(list.Name + validatorsSuffix)
if err != nil && !errors.Is(err, fs.ErrNotExist) {
return err
}
err = root.Rename(downloadName, list.Name)
if err != nil {
return err
}
b.setCopy(list.Name, listCopy{
addresses: addresses, updated: time.Now().UTC(), validators: sent,
})
b.logger.Info("Downloaded a new version of a blocklist", "list", list.Name)
// Without this file, the first download after a restart asks for the
// whole list.
err = writeValidators(root, list.Name, sent)
if err != nil {
b.logger.Error("Failed to keep the validators of a blocklist copy",
"list", list.Name, "error", err)
}
return nil
}
// writeAndParse writes body to the file name in root, up to maxDownloadSize
// bytes, and parses the file as format.
func writeAndParse(
root *os.Root, name string, body io.Reader, format Format,
) (*netipx.IPSet, error) {
f, err := root.Create(name)
if err != nil {
return nil, err
}
defer func() { _ = f.Close() }()
written, err := io.Copy(f, io.LimitReader(body, maxDownloadSize+1))
if err != nil {
return nil, err
}
if written > maxDownloadSize {
return nil, errTooLarge
}
_, err = f.Seek(0, io.SeekStart)
if err != nil {
return nil, err
}
return parse(f, format)
}
// markCurrent records that the publisher says the list's copy is still
// current: the copy, on disk and in memory, is marked as updated now.
func (b *Blocklists) markCurrent(list List) error {
root, err := os.OpenRoot(b.dir)
if err != nil {
return err
}
defer func() { _ = root.Close() }()
now := time.Now().UTC()
err = root.Chtimes(list.Name, now, now)
if err != nil {
return err
}
b.setUpdated(list.Name, now)
b.logger.Debug("Blocklist copy is still current", "list", list.Name)
return nil
}
// loadCopy makes the list's copy on disk, with the validators kept beside it,
// its copy in memory. The file's modification time is when the copy was
// updated.
func (b *Blocklists) loadCopy(list List) error {
root, err := os.OpenRoot(b.dir)
if err != nil {
return err
}
defer func() { _ = root.Close() }()
f, err := root.Open(list.Name)
if err != nil {
return err
}
defer func() { _ = f.Close() }()
info, err := f.Stat()
if err != nil {
return err
}
addresses, err := parse(f, list.Format)
if err != nil {
return err
}
held, err := readValidators(root, list.Name)
if err != nil {
return err
}
b.setCopy(list.Name, listCopy{
addresses: addresses, updated: info.ModTime().UTC(), validators: held,
})
return nil
}
// writeValidators keeps held in the file beside the named list's copy.
func writeValidators(root *os.Root, name string, held validators) error {
data, err := json.Marshal(held)
if err != nil {
return err
}
return root.WriteFile(name+validatorsSuffix, data, validatorsPermissions)
}
// readValidators returns the validators kept beside the named list's copy.
// They are empty when there is no such file.
func readValidators(root *os.Root, name string) (validators, error) {
var held validators
data, err := root.ReadFile(name + validatorsSuffix)
if errors.Is(err, fs.ErrNotExist) {
return held, nil
}
if err != nil {
return held, err
}
err = json.Unmarshal(data, &held)
return held, err
}
internal/blocklist/lists.go
-93
View File
@@ -1,93 +0,0 @@
package blocklist
import "time"
// How often each list is downloaded. No list is downloaded more often than
// its publisher updates it or allows.
const (
// DShield asks that its feeds be downloaded at most once an hour:
// https://isc.sans.edu/feeds_doc.html
dshieldInterval = time.Hour
// abuse.ch makes the list every 5 minutes and recommends a download at
// least every 15: https://feodotracker.abuse.ch/blocklist/
feodoTrackerInterval = 15 * time.Minute
// FireHOL's page for each list gives how often the list changes on
// average: about every 5 hours for abusers and every 3 hours for level
// 3. FireHOL sets no download limit.
// https://iplists.firehol.org/?ipset=firehol_abusers_30d
// https://iplists.firehol.org/?ipset=firehol_level3
fireHOLAbusersInterval = 5 * time.Hour
fireHOLLevel3Interval = 3 * time.Hour
// Spamhaus asks that DROP be downloaded at most once a day:
// https://www.spamhaus.org/blocklists/do-not-route-or-peer/
spamhausDROPInterval = 24 * time.Hour
// Tor publishes a new exit list every hour and sets no download limit:
// https://collector.torproject.org/recent/exit-lists/
torExitsInterval = time.Hour
)
// Lists returns the public blocklists routewatch downloads. Each can be
// downloaded whole without an account or a key. Spamhaus merged its EDROP
// list into DROP, so DROP covers both.
func Lists() []List {
return []List{
{
Name: "dshield",
Description: "In one of the 20 networks that DShield saw scan " +
"the most targets in the last three days",
URL: "https://feeds.dshield.org/block.txt",
Interval: dshieldInterval,
Format: FormatDShield,
},
{
Name: "feodo_tracker",
Description: "A botnet command-and-control server",
URL: "https://feodotracker.abuse.ch/downloads/ipblocklist.txt",
Interval: feodoTrackerInterval,
Format: FormatLines,
},
{
Name: "firehol_abusers_30d",
Description: "Reported for comment, form or forum spam in the " +
"last 30 days",
URL: "https://iplists.firehol.org/files/firehol_abusers_30d.netset",
Interval: fireHOLAbusersInterval,
Format: FormatLines,
},
{
Name: "firehol_level3",
Description: "Reported for attacks, spyware or viruses in the " +
"last 30 days",
URL: "https://iplists.firehol.org/files/firehol_level3.netset",
Interval: fireHOLLevel3Interval,
Format: FormatLines,
},
{
Name: "spamhaus_drop",
Description: "In an IPv4 network that Spamhaus says spammers " +
"or cyber criminals have hijacked or lease",
URL: "https://www.spamhaus.org/drop/drop_v4.json",
Interval: spamhausDROPInterval,
Format: FormatSpamhausJSON,
},
{
Name: "spamhaus_dropv6",
Description: "In an IPv6 network that Spamhaus says spammers " +
"or cyber criminals have hijacked or lease",
URL: "https://www.spamhaus.org/drop/drop_v6.json",
Interval: spamhausDROPInterval,
Format: FormatSpamhausJSON,
},
{
Name: "tor_exits",
Description: "A Tor exit node, where traffic from the Tor network leaves",
URL: "https://check.torproject.org/torbulkexitlist",
Interval: torExitsInterval,
Format: FormatLines,
},
}
}
internal/blocklist/parse.go
-158
View File
@@ -1,158 +0,0 @@
package blocklist
import (
"bufio"
"encoding/json"
"errors"
"fmt"
"io"
"net/netip"
"strings"
"go4.org/netipx"
)
const (
// commentPrefix starts a comment line in a list file.
commentPrefix = "#"
// networkSeparator separates the address and the mask length of a
// network, as in 192.0.2.0/24.
networkSeparator = "/"
// dshieldFieldSeparator separates the fields of a DShield line.
dshieldFieldSeparator = "\t"
// spamhausMetadataType is the type of the line that ends a Spamhaus
// file. That line holds the file's metadata and no network.
spamhausMetadataType = "metadata"
)
var (
errUnknownFormat = errors.New("unknown list format")
errNoNetwork = errors.New("no network on the line")
errBadRange = errors.New("first and last address make no range")
)
// parse reads a list file written in format and returns the addresses it
// holds. A line it cannot read fails the whole file, so a file that is not
// the list, such as an error page, never replaces a good copy.
func parse(r io.Reader, format Format) (*netipx.IPSet, error) {
var builder netipx.IPSetBuilder
scanner := bufio.NewScanner(r)
for lineNumber := 1; scanner.Scan(); lineNumber++ {
line := strings.TrimSpace(scanner.Text())
if line == "" || strings.HasPrefix(line, commentPrefix) {
continue
}
err := addLine(&builder, line, format)
if err != nil {
return nil, fmt.Errorf("line %d: %w", lineNumber, err)
}
}
err := scanner.Err()
if err != nil {
return nil, err
}
return builder.IPSet()
}
// addLine adds the address or network on one line of a list file to builder.
func addLine(builder *netipx.IPSetBuilder, line string, format Format) error {
switch format {
case FormatLines:
return addAddressOrNetwork(builder, line)
case FormatSpamhausJSON:
return addSpamhausNetwork(builder, line)
case FormatDShield:
return addDShieldNetwork(builder, line)
default:
return errUnknownFormat
}
}
// addAddressOrNetwork adds an address, such as 192.0.2.1, or a network, such
// as 192.0.2.0/24, to builder.
func addAddressOrNetwork(builder *netipx.IPSetBuilder, text string) error {
if strings.Contains(text, networkSeparator) {
network, err := netip.ParsePrefix(text)
if err != nil {
return err
}
builder.AddPrefix(network)
return nil
}
addr, err := netip.ParseAddr(text)
if err != nil {
return err
}
builder.Add(addr)
return nil
}
// addSpamhausNetwork adds the network on a line of a Spamhaus file to
// builder. The metadata line adds nothing.
func addSpamhausNetwork(builder *netipx.IPSetBuilder, line string) error {
var entry struct {
CIDR string `json:"cidr"`
Type string `json:"type"`
}
err := json.Unmarshal([]byte(line), &entry)
if err != nil {
return err
}
if entry.Type == spamhausMetadataType {
return nil
}
network, err := netip.ParsePrefix(entry.CIDR)
if err != nil {
return err
}
builder.AddPrefix(network)
return nil
}
// addDShieldNetwork adds the network on a line of a DShield file to builder.
// The line's first two fields are the network's first and last address.
func addDShieldNetwork(builder *netipx.IPSetBuilder, line string) error {
firstField, rest, found := strings.Cut(line, dshieldFieldSeparator)
if !found {
return errNoNetwork
}
lastField, _, _ := strings.Cut(rest, dshieldFieldSeparator)
first, err := netip.ParseAddr(firstField)
if err != nil {
return err
}
last, err := netip.ParseAddr(lastField)
if err != nil {
return err
}
network := netipx.IPRangeFrom(first, last)
if !network.IsValid() {
return errBadRange
}
builder.AddRange(network)
return nil
}
internal/database/database.go
+23 -37
View File
@@ -945,26 +945,6 @@ func (d *Database) UpdatePrefixesBatch(prefixes map[string]time.Time) error {
return nil
}
// These statements update a stored AS, peering or peer. Each keeps the earlier
// of the stored and the new first_seen, and the later of the stored and the new
// last_seen. Some collectors send updates days old while they catch up on a
// backlog. MIN and MAX compare the times as text, which is time order here: the
// driver writes every time in one format, and every time is UTC.
const (
updateASNSQL = `UPDATE asns
SET first_seen = MIN(first_seen, ?), last_seen = MAX(last_seen, ?)
WHERE asn = ?`
updatePeeringSQL = `UPDATE peerings
SET first_seen = MIN(first_seen, ?), last_seen = MAX(last_seen, ?)
WHERE as_a = ? AND as_b = ?`
updatePeerSQL = `UPDATE bgp_peers
SET peer_asn = ?, last_message_type = ?,
first_seen = MIN(first_seen, ?), last_seen = MAX(last_seen, ?)
WHERE peer_ip = ?`
)
// newASN returns a new ASN record first seen at seen.First and last seen at
// seen.Last, with the handle and description from the AS info lookup when it
// knows the number.
@@ -984,8 +964,9 @@ func newASN(number int, seen SeenTimes) ASN {
return asn
}
// upsertASN stores the seen times of ASN number with the given statements,
// inserting the ASN when it is new. It reports whether it inserted it.
// upsertASN sets the last_seen time of ASN number to seen.Last with the given
// statements, inserting the ASN when it is new. It reports whether it inserted
// it.
func upsertASN(
ctx context.Context, selectStmt, updateStmt, insertStmt *sql.Stmt,
number int, seen SeenTimes,
@@ -998,8 +979,8 @@ func upsertASN(
err := selectStmt.QueryRowContext(ctx, number).Scan(&asn.ASN, &handle,
&description, &asn.FirstSeen, &asn.LastSeen)
if err == nil {
// ASN exists, update its seen times
_, err = updateStmt.ExecContext(ctx, seen.First, seen.Last, number)
// ASN exists, update last_seen
_, err = updateStmt.ExecContext(ctx, seen.Last, number)
if err != nil {
return false, fmt.Errorf("failed to update ASN %d: %w", number, err)
}
@@ -1049,7 +1030,8 @@ func (d *Database) GetOrCreateASNBatch(asns map[int]SeenTimes) error {
}
defer func() { _ = selectStmt.Close() }()
updateStmt, err := tx.PrepareContext(ctx, updateASNSQL)
updateStmt, err := tx.PrepareContext(ctx,
"UPDATE asns SET last_seen = ? WHERE asn = ?")
if err != nil {
return fmt.Errorf("failed to prepare update statement: %w", err)
}
@@ -1108,19 +1090,17 @@ func (d *Database) GetOrCreateASN(number int, timestamp time.Time) (*ASN, error)
"FROM asns WHERE asn = ?", number).
Scan(&asn.ASN, &handle, &description, &asn.FirstSeen, &asn.LastSeen)
if err == nil {
// ASN exists, update its seen times
// ASN exists, update last_seen
asn.Handle = handle.String
asn.Description = description.String
_, err = tx.Exec(updateASNSQL, timestamp, timestamp, number)
_, err = tx.Exec("UPDATE asns SET last_seen = ? WHERE asn = ?",
timestamp, number)
if err != nil {
return nil, err
}
// Return the seen times the update stored.
seen := SeenTimes{First: asn.FirstSeen, Last: asn.LastSeen}
seen.Add(timestamp)
asn.FirstSeen, asn.LastSeen = seen.First, seen.Last
asn.LastSeen = timestamp
err = tx.Commit()
if err != nil {
@@ -1279,7 +1259,9 @@ func (d *Database) RecordPeering(asA, asB int, seen SeenTimes) error {
}
if exists {
_, err = tx.Exec(updatePeeringSQL, seen.First, seen.Last, asA, asB)
_, err = tx.Exec(
"UPDATE peerings SET last_seen = ? WHERE as_a = ? AND as_b = ?",
seen.Last, asA, asB)
} else {
_, err = tx.Exec(`
INSERT INTO peerings (id, as_a, as_b, first_seen, last_seen)
@@ -1334,7 +1316,9 @@ func (d *Database) UpdatePeerBatch(peers map[string]PeerUpdate) error {
}
defer func() { _ = checkStmt.Close() }()
updateStmt, err := tx.PrepareContext(ctx, updatePeerSQL)
updateStmt, err := tx.PrepareContext(ctx,
"UPDATE bgp_peers SET peer_asn = ?, last_seen = ?, "+
"last_message_type = ? WHERE peer_ip = ?")
if err != nil {
return fmt.Errorf("failed to prepare update statement: %w", err)
}
@@ -1360,8 +1344,7 @@ func (d *Database) UpdatePeerBatch(peers map[string]PeerUpdate) error {
if exists {
_, err = updateStmt.ExecContext(ctx, update.PeerASN,
update.MessageType, update.Seen.First, update.Seen.Last,
update.PeerIP)
update.Seen.Last, update.MessageType, update.PeerIP)
} else {
_, err = insertStmt.ExecContext(ctx,
generateUUID().String(), update.PeerIP, update.PeerASN,
@@ -1410,8 +1393,11 @@ func (d *Database) UpdatePeer(
}
if exists {
_, err = tx.Exec(updatePeerSQL,
peerASN, messageType, timestamp, timestamp, peerIP)
_, err = tx.Exec(
"UPDATE bgp_peers SET peer_asn = ?, last_seen = ?, "+
"last_message_type = ? WHERE peer_ip = ?",
peerASN, timestamp, messageType, peerIP,
)
} else {
_, err = tx.Exec(
"INSERT INTO bgp_peers (id, peer_ip, peer_asn, first_seen, "+
internal/database/database_internal_test.go
+2 -154
View File
@@ -20,7 +20,6 @@ import (
"git.eeqj.de/sneak/routewatch/internal/config"
"git.eeqj.de/sneak/routewatch/internal/logger"
"git.eeqj.de/sneak/routewatch/internal/ristypes"
"git.eeqj.de/sneak/routewatch/pkg/asinfo"
"github.com/mattn/go-sqlite3"
)
@@ -1705,7 +1704,8 @@ func TestGetASDetailsListsEachPrefixOnce(t *testing.T) {
// record of, some with stored routes. Each gets the prefixes of its routes and
// no seen times. One that the AS info lookup knows gets its handle and
// description from it. An AS with no record, no lookup entry and no route is
// ErrNoRoute. Before the fix every AS with no record was ErrNoRoute.
// ErrNoRoute. Before the fix an AS with no record got no prefixes, although
// its routes were stored.
func TestGetASDetailsOfASNotInDatabase(t *testing.T) {
t.Parallel()
@@ -1961,158 +1961,6 @@ func TestUpdatePeerBatchStoresTheEarliestAndLatestTimesOfANewPeer(
}
}
// backlogAge is how much older one write is than the other in the tests of
// first_seen and last_seen, as when a collector catches up on a backlog.
const backlogAge = 24 * time.Hour
// batchSpan is how much earlier a batch's First is than its Last in the tests
// of first_seen and last_seen.
const batchSpan = time.Hour
// TestWritesKeepEarliestFirstSeenAndLatestLastSeen writes a row of each of
// asns, peerings and bgp_peers through each function that writes one, at a
// newer time and at an older one. Each row takes the older write's First as
// first_seen and the newer write's Last as last_seen. The test writes in both
// orders, each in a database of its own. Newer then older is a collector
// catching up on a backlog; older then newer is a feed that arrives in order.
// Each function writes a row of its own. A batch write's First is batchSpan
// before its Last, so the test catches the two swapped. A single-row write
// takes one time.
func TestWritesKeepEarliestFirstSeenAndLatestLastSeen(t *testing.T) {
t.Parallel()
const (
batchASN, singleASN = 64500, 64501
peerASN = 64502
asnQuery = "SELECT first_seen, last_seen FROM asns WHERE asn = ?"
peeringQuery = "SELECT first_seen, last_seen FROM peerings " +
"WHERE as_a = ? AND as_b = ?"
peerQuery = "SELECT first_seen, last_seen FROM bgp_peers " +
"WHERE peer_ip = ?"
)
for name, olderFirst := range map[string]bool{
"newer then older": false,
"older then newer": true,
} {
t.Run(name, func(t *testing.T) {
t.Parallel()
db, err := New(&config.Config{StateDir: t.TempDir()}, logger.New())
if err != nil {
t.Fatalf("failed to create database: %v", err)
}
defer func() { _ = db.Close() }()
checkSeenTimes(t, db, olderFirst, func(seen SeenTimes) error {
return db.GetOrCreateASNBatch(map[int]SeenTimes{batchASN: seen})
}, batchSpan, asnQuery, batchASN)
checkSeenTimes(t, db, olderFirst, func(seen SeenTimes) error {
_, err := db.GetOrCreateASN(singleASN, seen.Last)
return err
}, 0, asnQuery, singleASN)
checkSeenTimes(t, db, olderFirst, func(seen SeenTimes) error {
return db.RecordPeering(batchASN, singleASN, seen)
}, batchSpan, peeringQuery, batchASN, singleASN)
checkSeenTimes(t, db, olderFirst, func(seen SeenTimes) error {
return db.UpdatePeerBatch(map[string]PeerUpdate{
testV4PeerIP: {
PeerIP: testV4PeerIP, PeerASN: peerASN,
MessageType: ristypes.MessageTypeUpdate,
Seen: seen,
},
})
}, batchSpan, peerQuery, testV4PeerIP)
checkSeenTimes(t, db, olderFirst, func(seen SeenTimes) error {
return db.UpdatePeer(testPeerIP, peerASN,
ristypes.MessageTypeUpdate, seen.Last)
}, 0, peerQuery, testPeerIP)
})
}
}
// checkSeenTimes calls write twice, each time with First span before Last: at
// newer times and at times backlogAge older. It writes the older times first
// when olderFirst is true. It checks that query, given args, reads the older
// write's First as first_seen and the newer write's Last as last_seen.
func checkSeenTimes(
t *testing.T, db *Database, olderFirst bool,
write func(seen SeenTimes) error,
span time.Duration, query string, args ...any,
) {
t.Helper()
newerLast := time.Date(2026, 1, 2, 3, 4, 5, 0, time.UTC)
newer := SeenTimes{First: newerLast.Add(-span), Last: newerLast}
older := SeenTimes{
First: newer.First.Add(-backlogAge),
Last: newer.Last.Add(-backlogAge),
}
writes := []SeenTimes{newer, older}
if olderFirst {
writes = []SeenTimes{older, newer}
}
for _, seen := range writes {
err := write(seen)
if err != nil {
t.Fatalf("writing first %s, last %s: %v", seen.First, seen.Last, err)
}
}
var firstSeen, lastSeen time.Time
err := db.db.QueryRowContext(t.Context(), query, args...).
Scan(&firstSeen, &lastSeen)
if err != nil {
t.Fatalf("reading the row: %v", err)
}
if !firstSeen.Equal(older.First) || !lastSeen.Equal(newer.Last) {
t.Errorf("first seen %s, last seen %s; want %s, %s",
firstSeen, lastSeen, older.First, newer.Last)
}
}
// TestGetOrCreateASNReturnsStoredSeenTimes stores an AS at a newer time, then
// at an older one. The second call returns the times the row now holds: the
// older first_seen and the newer last_seen.
func TestGetOrCreateASNReturnsStoredSeenTimes(t *testing.T) {
t.Parallel()
db, err := New(&config.Config{StateDir: t.TempDir()}, logger.New())
if err != nil {
t.Fatalf("failed to create database: %v", err)
}
defer func() { _ = db.Close() }()
const asn = 64500
newer := time.Date(2026, 1, 2, 3, 4, 5, 0, time.UTC)
older := newer.Add(-backlogAge)
_, err = db.GetOrCreateASN(asn, newer)
if err != nil {
t.Fatalf("GetOrCreateASN at %s: %v", newer, err)
}
got, err := db.GetOrCreateASN(asn, older)
if err != nil {
t.Fatalf("GetOrCreateASN at %s: %v", older, err)
}
if !got.FirstSeen.Equal(older) || !got.LastSeen.Equal(newer) {
t.Errorf("first seen %s, last seen %s; want %s, %s",
got.FirstSeen, got.LastSeen, older, newer)
}
}
// TestUpsertLiveRouteBatchWritesInKeyOrder writes new routes of two peers out
// of key order, the last one a second announcement of the first, and checks
// that the rows were inserted in key order, peer then prefix (each new row's id
internal/database/models.go
+8 -7
View File
@@ -92,11 +92,12 @@ type PrefixDistribution struct {
//
//nolint:tagliatelle // field names are the existing JSON API
type IPInfo struct {
IP string `json:"ip"`
Netblock string `json:"netblock"`
MaskLength int `json:"mask_length"`
IPVersion int `json:"ip_version"`
NumPeers int `json:"num_peers"`
IP string `json:"ip"`
PTR []string `json:"ptr,omitempty"`
Netblock string `json:"netblock"`
MaskLength int `json:"mask_length"`
IPVersion int `json:"ip_version"`
NumPeers int `json:"num_peers"`
// AS information
ASN int `json:"asn"`
ASName string `json:"as_name,omitempty"`
@@ -124,8 +125,8 @@ type LiveRouteDeletion struct {
// SeenTimes is the earliest and the latest time one batch of updates saw an AS,
// a peer or a peering. A row the batch adds is stored first seen at First and
// last seen at Last. A row already stored takes First as its first_seen only
// when First is earlier, and Last as its last_seen only when Last is later.
// last seen at Last; a row already stored keeps its first_seen and has its
// last_seen set to Last.
type SeenTimes struct {
First time.Time
Last time.Time
internal/metrics/metrics.go
+7 -8
View File
@@ -34,7 +34,7 @@ type Tracker struct {
// Announcement/withdrawal metrics
announcementCounter metrics.Counter
withdrawalCounter metrics.Counter
churnPerMinute minuteCounter // prefix announcements and withdrawals
churnRate metrics.Meter // combined announcements + withdrawals per second
// BGP peer tracking
bgpPeerCount atomic.Int32
@@ -56,7 +56,7 @@ func New() *Tracker {
ipv6UpdateRate: metrics.NewMeter(),
announcementCounter: metrics.NewCounter(),
withdrawalCounter: metrics.NewCounter(),
churnPerMinute: minuteCounter{now: time.Now},
churnRate: metrics.NewMeter(),
}
}
@@ -156,13 +156,13 @@ func (t *Tracker) RecordIPv6Update() {
// RecordAnnouncement records a route announcement
func (t *Tracker) RecordAnnouncement() {
t.announcementCounter.Inc(1)
t.churnPerMinute.add()
t.churnRate.Mark(1)
}
// RecordWithdrawal records a route withdrawal
func (t *Tracker) RecordWithdrawal() {
t.withdrawalCounter.Inc(1)
t.churnPerMinute.add()
t.churnRate.Mark(1)
}
// SetBGPPeerCount updates the current BGP peer count
@@ -198,10 +198,9 @@ func (t *Tracker) GetWithdrawalCount() uint64 {
return uint64(count)
}
// GetChurnPerMinute returns the number of prefix announcements and
// withdrawals, from all peers, in the last full clock minute.
func (t *Tracker) GetChurnPerMinute() int {
return t.churnPerMinute.lastFullMinute()
// GetChurnRate returns the route churn rate per second
func (t *Tracker) GetChurnRate() float64 {
return t.churnRate.Rate1()
}
// GetRouteMetrics returns current route update metrics
internal/metrics/minutecounter.go
-59
View File
@@ -1,59 +0,0 @@
package metrics
import (
"sync"
"time"
)
// minuteCounter counts events in each clock minute and reports the count of
// the last full minute. It is safe for concurrent use.
type minuteCounter struct {
mu sync.Mutex
now func() time.Time
// minute is the start of the minute that count belongs to.
minute time.Time
count int
// lastMinuteCount is the count of the minute just before minute.
lastMinuteCount int
}
// add counts one event in the current minute.
func (c *minuteCounter) add() {
c.mu.Lock()
defer c.mu.Unlock()
c.moveToCurrentMinute()
c.count++
}
// lastFullMinute returns the number of events in the minute that ended most
// recently.
func (c *minuteCounter) lastFullMinute() int {
c.mu.Lock()
defer c.mu.Unlock()
c.moveToCurrentMinute()
return c.lastMinuteCount
}
// moveToCurrentMinute starts a new count when the clock has moved into
// another minute. The count it ends becomes the last full minute's count when
// its minute is the one just before the current minute. Otherwise the last
// full minute had no events. The caller holds mu.
func (c *minuteCounter) moveToCurrentMinute() {
minute := c.now().Truncate(time.Minute)
if minute.Equal(c.minute) {
return
}
if minute.Sub(c.minute) == time.Minute {
c.lastMinuteCount = c.count
} else {
c.lastMinuteCount = 0
}
c.minute = minute
c.count = 0
}
internal/metrics/minutecounter_internal_test.go
-86
View File
@@ -1,86 +0,0 @@
package metrics
import (
"sync"
"testing"
"time"
)
// TestChurnPerMinuteCountsLastFullMinute announces and withdraws prefixes on
// an injected clock and expects each minute's whole count once that minute
// has ended, and 0 for a minute with none.
func TestChurnPerMinuteCountsLastFullMinute(t *testing.T) {
t.Parallel()
now := time.Date(2026, 10, 26, 12, 0, 30, 0, time.UTC)
tracker := New()
tracker.churnPerMinute.now = func() time.Time { return now }
expectChurn := func(want int, why string) {
t.Helper()
got := tracker.GetChurnPerMinute()
if got != want {
t.Fatalf("at %s churn = %d, want %d: %s",
now.Format(time.TimeOnly), got, want, why)
}
}
const firstMinuteChurn = 2
tracker.RecordAnnouncement()
tracker.RecordWithdrawal()
expectChurn(0, "12:00 has not ended")
now = time.Date(2026, 10, 26, 12, 1, 59, 0, time.UTC)
tracker.RecordAnnouncement()
expectChurn(firstMinuteChurn, "12:00 had one announcement and one withdrawal")
now = time.Date(2026, 10, 26, 12, 2, 0, 0, time.UTC)
expectChurn(1, "12:01 had one announcement")
now = time.Date(2026, 10, 26, 12, 4, 0, 0, time.UTC)
expectChurn(0, "12:03 had none")
tracker.RecordWithdrawal()
now = time.Date(2026, 10, 26, 12, 6, 0, 0, time.UTC)
expectChurn(0, "12:05 had none, though 12:04 had one withdrawal")
}
// TestChurnPerMinuteCountsConcurrentRecords announces prefixes from several
// goroutines at once and expects every one counted.
func TestChurnPerMinuteCountsConcurrentRecords(t *testing.T) {
t.Parallel()
const (
goroutines = 8
recordsPerGoroutine = 1000
)
now := time.Date(2026, 10, 26, 12, 0, 0, 0, time.UTC)
tracker := New()
tracker.churnPerMinute.now = func() time.Time { return now }
var wg sync.WaitGroup
for range goroutines {
wg.Go(func() {
for range recordsPerGoroutine {
tracker.RecordAnnouncement()
}
})
}
wg.Wait()
now = now.Add(time.Minute)
want := goroutines * recordsPerGoroutine
if got := tracker.GetChurnPerMinute(); got != want {
t.Fatalf("churn = %d, want %d", got, want)
}
}
internal/routewatch/app.go
+5 -29
View File
@@ -8,11 +8,9 @@ package routewatch
import (
"context"
"errors"
"path/filepath"
"sync"
"time"
"git.eeqj.de/sneak/routewatch/internal/blocklist"
"git.eeqj.de/sneak/routewatch/internal/config"
"git.eeqj.de/sneak/routewatch/internal/database"
"git.eeqj.de/sneak/routewatch/internal/logger"
@@ -54,13 +52,8 @@ type RouteWatch struct {
peeringHandler *PeeringHandler
asnFetcher *ASNFetcher
dbMaintainer *DBMaintainer
blocklists *blocklist.Blocklists
}
// blocklistsDirName is the directory in the state directory that holds the
// blocklist copies.
const blocklistsDirName = "blocklists"
// New creates a new RouteWatch instance
func New(deps Dependencies) *RouteWatch {
rw := &RouteWatch{
@@ -129,18 +122,6 @@ func (rw *RouteWatch) Run(ctx context.Context) error {
rw.asnFetcher.Start(ctx)
rw.server.SetASNFetcher(rw.asnFetcher)
// Load the blocklist copies from disk and start downloading the lists.
// The HTTP handlers read the server's blocklists without a lock, so they
// are set before the handlers run.
blocklists := blocklist.New(blocklist.Params{
Dir: filepath.Join(rw.config.StateDir, blocklistsDirName),
Lists: blocklist.Lists(),
Logger: rw.logger.Logger,
})
blocklists.Start(ctx)
rw.blocklists = blocklists
rw.server.SetBlocklists(blocklists)
// Start HTTP server
err = rw.server.Start(ctx)
if err != nil {
@@ -158,12 +139,11 @@ func (rw *RouteWatch) Run(ctx context.Context) error {
}
// Shutdown stops all services in order: the streamer, the handlers, which
// write what they hold, the WHOIS fetcher, the database maintainer, the
// blocklist downloads and the HTTP server, and last the database, which it
// checkpoints and closes. Each stage waits at most until ctx ends, so
// Shutdown returns soon after: a handler logs what it could not write by
// then, and the HTTP server closes the connections of requests still
// running.
// write what they hold, the WHOIS fetcher, the database maintainer and the
// HTTP server, and last the database, which it checkpoints and closes. Each
// stage waits at most until ctx ends, so Shutdown returns soon after: a
// handler logs what it could not write by then, and the HTTP server closes
// the connections of requests still running.
func (rw *RouteWatch) Shutdown(ctx context.Context) {
rw.mu.Lock()
if rw.shutdown {
@@ -210,10 +190,6 @@ func (rw *RouteWatch) Shutdown(ctx context.Context) {
rw.dbMaintainer.Stop(ctx)
}
if rw.blocklists != nil {
rw.blocklists.Stop(ctx)
}
err := rw.server.Stop(ctx)
if err != nil {
rw.logger.Error("Stopped the HTTP server before every request finished",
internal/routewatch/app_integration_internal_test.go
+2 -2
View File
@@ -466,9 +466,9 @@ func TestRouteWatchLiveFeed(t *testing.T) {
// Create streamer
s := streamer.New(logger, metricsTracker)
// The state directory holds the blocklist copies Run downloads.
// Create test config with empty state dir (no snapshot loading)
cfg := &config.Config{
StateDir: t.TempDir(),
StateDir: "",
MaxRuntime: 5 * time.Second,
EnableBatchedDatabaseWrites: true,
}
internal/routewatch/asnfetcher.go
+8 -17
View File
@@ -21,16 +21,10 @@ const (
minInterval = 1 * time.Second
// maxInterval is the maximum interval after failures (backoff cap).
maxInterval = time.Minute
maxInterval = 5 * time.Minute
// A failure multiplies the interval by backoffNumerator divided by
// backoffDenominator, 1.5, up to maxInterval.
backoffNumerator = 3
backoffDenominator = 2
// A success divides the interval by successDivisor, down to
// minInterval.
successDivisor = 2
// backoffMultiplier is how much to multiply interval on failure.
backoffMultiplier = 2
// whoisStaleThreshold is how old WHOIS data can be before refresh.
whoisStaleThreshold = 30 * 24 * time.Hour // 30 days
@@ -68,12 +62,10 @@ type ASNFetcher struct {
// NewASNFetcher creates a new ASN fetcher.
func NewASNFetcher(db database.Store, logger *slog.Logger) *ASNFetcher {
logger = logger.With("component", "asn_fetcher")
return &ASNFetcher{
db: db,
whoisClient: whois.NewClient(logger),
logger: logger,
whoisClient: whois.NewClient(),
logger: logger.With("component", "asn_fetcher"),
immediateQueue: make(chan int, immediateQueueSize),
stopCh: make(chan struct{}),
done: make(chan struct{}),
@@ -168,7 +160,7 @@ func (f *ASNFetcher) recordSuccess() {
f.consecutiveFails = 0
// Decrease interval by half, but not below minimum
newInterval := max(f.currentInterval/successDivisor, minInterval)
newInterval := max(f.currentInterval/backoffMultiplier, minInterval)
if newInterval != f.currentInterval {
f.logger.Debug("Decreased fetch interval",
@@ -192,9 +184,8 @@ func (f *ASNFetcher) recordFailure() {
f.intervalMu.Lock()
f.consecutiveFails++
// Exponential backoff: multiply by 1.5, capped at max
newInterval := min(
f.currentInterval*backoffNumerator/backoffDenominator, maxInterval)
// Exponential backoff: multiply by 2, capped at max
newInterval := min(f.currentInterval*backoffMultiplier, maxInterval)
if newInterval != f.currentInterval {
f.logger.Debug("Increased fetch interval due to failure",
internal/routewatch/asnfetcher_internal_test.go
+4 -45
View File
@@ -120,58 +120,17 @@ func TestFetchAndUpdateKeepsTheRowWhenTheLookupFails(t *testing.T) {
"was stored", stats.NeverFetched)
}
want := baseInterval * backoffNumerator / backoffDenominator
got := f.GetStats()
if got.CurrentInterval != want || got.ConsecutiveFails != 1 {
if got.CurrentInterval != baseInterval*backoffMultiplier ||
got.ConsecutiveFails != 1 {
t.Errorf("interval %v after %d failures, want %v after 1",
got.CurrentInterval, got.ConsecutiveFails, want)
got.CurrentInterval, got.ConsecutiveFails,
baseInterval*backoffMultiplier)
}
})
}
}
// TestRecordStepsTheInterval has the fetcher fail from its base interval of
// 15 seconds, then succeed. Each failure makes the interval 1.5 times as
// long, up to one minute, and each success halves it, down to one second.
func TestRecordStepsTheInterval(t *testing.T) {
t.Parallel()
f := NewASNFetcher(newMockStore(), logger.New().Logger)
for i, want := range []time.Duration{
22500 * time.Millisecond,
33750 * time.Millisecond,
50625 * time.Millisecond,
time.Minute,
time.Minute,
} {
f.recordFailure()
if got := f.getInterval(); got != want {
t.Errorf("interval after %d failures = %v, want %v",
i+1, got, want)
}
}
for i, want := range []time.Duration{
30 * time.Second,
15 * time.Second,
7500 * time.Millisecond,
3750 * time.Millisecond,
1875 * time.Millisecond,
time.Second,
time.Second,
} {
f.recordSuccess()
if got := f.getInterval(); got != want {
t.Errorf("interval after %d successes = %v, want %v",
i+1, got, want)
}
}
}
// TestFetchAndUpdateMarksAnASWithoutReferral looks up a private-use AS, which
// IANA refers to no RIR. The AS must not be looked up again until it is
// stale, and the lookup counts as neither a success nor a failure.
internal/server/handlers.go
+52 -111
View File
@@ -1,7 +1,7 @@
package server
import (
"cmp"
"bytes"
"context"
"encoding/json"
"errors"
@@ -9,13 +9,11 @@ import (
"net/http"
"net/netip"
"runtime"
"slices"
"sort"
"strconv"
"strings"
"time"
"git.eeqj.de/sneak/routewatch/internal/blocklist"
"git.eeqj.de/sneak/routewatch/internal/database"
"git.eeqj.de/sneak/routewatch/internal/metrics"
"git.eeqj.de/sneak/routewatch/internal/ristypes"
@@ -46,15 +44,6 @@ const (
ptrLookupTimeout = 2 * time.Second
)
// The Cache-Control header of an IP lookup answer, such as
// "public, max-age=900".
const (
cacheControlHeader = "Cache-Control"
cacheControlPublic = "public"
cacheControlPrivate = "private"
cacheControlMaxAge = ", max-age="
)
// Keys of the JSON envelope the handlers and middleware write, and the status
// value of an error answer.
const (
@@ -156,10 +145,7 @@ func (s *Server) handleIndex() http.HandlerFunc {
tmpl := templates.IndexTemplate()
err := tmpl.Execute(w, indexPageData{
CSPNonce: cspNonce(r.Context()),
ClientIP: extractClientIP(r),
})
err := tmpl.Execute(w, pageData{CSPNonce: cspNonce(r.Context())})
if err != nil {
s.logger.Error("Failed to render index template", "error", err)
http.Error(w, "Internal Server Error", http.StatusInternalServerError)
@@ -173,15 +159,6 @@ type pageData struct {
CSPNonce string
}
// indexPageData is what the home page is rendered with.
type indexPageData struct {
// CSPNonce is the nonce the page's script and style blocks carry.
CSPNonce string
// ClientIP is the visitor's address, which the IP information API card
// looks up: the address /ip/ would look up for the same request.
ClientIP string
}
// writeJSONError writes a standardized JSON error response with the given
// status code and error message.
func (s *Server) writeJSONError(w http.ResponseWriter, statusCode int, message string) {
@@ -464,10 +441,10 @@ type GCStats struct {
//
//nolint:tagliatelle // field names are the existing JSON API
type StreamStats struct {
Announcements uint64 `json:"announcements"`
Withdrawals uint64 `json:"withdrawals"`
RouteChurnPerMin int `json:"route_churn_per_min"`
BGPPeerCount int `json:"bgp_peer_count"`
Announcements uint64 `json:"announcements"`
Withdrawals uint64 `json:"withdrawals"`
RouteChurnPerSec float64 `json:"route_churn_per_sec"`
BGPPeerCount int `json:"bgp_peer_count"`
}
// StatsResponse represents the API statistics response
@@ -568,7 +545,7 @@ func (s *Server) streamStats() StreamStats {
return StreamStats{
Announcements: metricsTracker.GetAnnouncementCount(),
Withdrawals: metricsTracker.GetWithdrawalCount(),
RouteChurnPerMin: metricsTracker.GetChurnPerMinute(),
RouteChurnPerSec: metricsTracker.GetChurnRate(),
BGPPeerCount: metricsTracker.GetBGPPeerCount(),
}
}
@@ -703,24 +680,9 @@ func (s *Server) handleIPLookup() http.HandlerFunc {
// IPLookupResponse is the standard response for IP/hostname lookups.
type IPLookupResponse struct {
Query string `json:"query"`
Results []IPLookupResult `json:"results"`
Errors []string `json:"errors,omitempty"`
}
// IPLookupResult is the answer for one address: its PTR names, its live route
// and that route's AS, and the blocklists that list it. IPInfo is nil when
// the address has no live route, and the answer then has no route or AS
// fields. IP hides the IPInfo field of the same name, so that an address with
// no route has it too.
type IPLookupResult struct {
*database.IPInfo
IP string `json:"ip"`
PTR []string `json:"ptr,omitempty"`
// Blocklists has one entry for each blocklist the address is on.
Blocklists []blocklist.Listing `json:"blocklists"`
Query string `json:"query"`
Results []*database.IPInfo `json:"results"`
Errors []string `json:"errors,omitempty"`
}
// handleIPInfo returns a handler that provides comprehensive IP information.
@@ -729,7 +691,7 @@ type IPLookupResult struct {
// Always returns the same response structure with PTR records for each IP.
func (s *Server) handleIPInfo() http.HandlerFunc {
return func(w http.ResponseWriter, r *http.Request) {
target, ownAddress := lookupTarget(r)
target := lookupTarget(r)
if target == "" {
s.writeJSONError(w, http.StatusBadRequest, "Could not determine IP address")
@@ -737,7 +699,7 @@ func (s *Server) handleIPInfo() http.HandlerFunc {
}
ctx := r.Context()
response := IPLookupResponse{Query: target, Results: make([]IPLookupResult, 0)}
response := IPLookupResponse{Query: target, Results: make([]*database.IPInfo, 0)}
// Collect all IPs to look up
ipsToLookup, resolveErrors, err := s.resolveLookupTargets(ctx, target)
@@ -774,10 +736,7 @@ func (s *Server) handleIPInfo() http.HandlerFunc {
}
switch {
case errors.Is(err, database.ErrNoRoute):
// The address still gets a result, without route fields.
ipInfo = nil
case errors.Is(err, database.ErrInvalidIP):
case errors.Is(err, database.ErrNoRoute), errors.Is(err, database.ErrInvalidIP):
response.Errors = append(response.Errors, ip+": "+err.Error())
continue
@@ -796,14 +755,13 @@ func (s *Server) handleIPInfo() http.HandlerFunc {
return
}
response.Results = append(response.Results, IPLookupResult{
IP: ip,
PTR: ptrs,
IPInfo: ipInfo,
Blocklists: s.blocklistListings(ip),
})
if len(ptrs) > 0 {
ipInfo.PTR = ptrs
}
if ipInfo != nil && ipInfo.NeedsWHOISRefresh {
response.Results = append(response.Results, ipInfo)
if ipInfo.NeedsWHOISRefresh {
refreshASNs[ipInfo.ASN] = true
}
}
@@ -811,40 +769,10 @@ func (s *Server) handleIPInfo() http.HandlerFunc {
// Queue WHOIS refresh for stale ASNs (non-blocking)
s.queueWHOISRefresh(refreshASNs)
s.writeIPLookupAnswer(w, response, ownAddress)
s.writeIPLookupAnswer(w, response)
}
}
// blocklistListings returns the blocklists that list ip. It returns an empty
// list, never nil, so that an answer's blocklists is [] rather than null.
func (s *Server) blocklistListings(ip string) []blocklist.Listing {
addr, err := netip.ParseAddr(ip)
if s.blocklists == nil || err != nil {
return []blocklist.Listing{}
}
return s.blocklists.Lookup(addr)
}
// setIPLookupCacheControl lets caches keep an IP lookup answer for the
// shortest blocklist interval, the soonest a blocklist copy can change. The
// answer about the caller's own address is private, so that a shared cache
// does not give it to another caller. Without blocklists it sets nothing.
func (s *Server) setIPLookupCacheControl(w http.ResponseWriter, ownAddress bool) {
if s.blocklists == nil {
return
}
visibility := cacheControlPublic
if ownAddress {
visibility = cacheControlPrivate
}
maxAgeSeconds := int(s.blocklists.ShortestInterval().Seconds())
w.Header().Set(cacheControlHeader,
visibility+cacheControlMaxAge+strconv.Itoa(maxAgeSeconds))
}
// noTargetsMessage returns the message of the 400 an IP lookup answers when
// no address is left to look up, followed by the first of resolveErrors, which
// says why, when there is one.
@@ -858,12 +786,16 @@ func noTargetsMessage(resolveErrors []string) string {
return message + ": " + resolveErrors[0]
}
// writeIPLookupAnswer sends response with its Cache-Control header.
// ownAddress says the request asked about the caller's own address.
// writeIPLookupAnswer sends response, or 404 with its first error when no
// address had a route.
func (s *Server) writeIPLookupAnswer(
w http.ResponseWriter, response IPLookupResponse, ownAddress bool,
w http.ResponseWriter, response IPLookupResponse,
) {
s.setIPLookupCacheControl(w, ownAddress)
if len(response.Results) == 0 && len(response.Errors) > 0 {
s.writeJSONError(w, http.StatusNotFound, "No routes found: "+response.Errors[0])
return
}
err := writeJSONSuccess(w, response)
if err != nil {
@@ -872,9 +804,8 @@ func (s *Server) writeIPLookupAnswer(
}
// lookupTarget returns what an IP lookup request asks for: the ip or addr URL
// parameter, or else the client's own address. The bool is true for the
// client's own address.
func lookupTarget(r *http.Request) (string, bool) {
// parameter, or else the client's own address.
func lookupTarget(r *http.Request) string {
target := chi.URLParam(r, "ip")
if target == "" {
target = chi.URLParam(r, "addr")
@@ -882,10 +813,10 @@ func lookupTarget(r *http.Request) (string, bool) {
if target == "" {
// Use client IP (RealIP middleware has already processed this)
return extractClientIP(r), true
target = extractClientIP(r)
}
return target, false
return target
}
// queueWHOISRefresh asks the WHOIS fetcher, when there is one, to refresh
@@ -1097,15 +1028,24 @@ func splitRoutesByIPVersion(
return ipv4Routes, ipv6Routes
}
// sortRoutesByMaskLength sorts routes by mask length, shortest first, and
// routes with the same mask length by network address.
func sortRoutesByMaskLength(routes []database.LiveRoute) {
slices.SortFunc(routes, func(a, b database.LiveRoute) int {
prefixA, _ := netip.ParsePrefix(a.Prefix)
prefixB, _ := netip.ParsePrefix(b.Prefix)
// sortRoutesByNetwork sorts routes by network address, then by mask length.
func sortRoutesByNetwork(routes []database.LiveRoute) {
sort.Slice(routes, func(i, j int) bool {
// Parse the prefixes to compare network addresses
ipI, netI, _ := net.ParseCIDR(routes[i].Prefix)
ipJ, netJ, _ := net.ParseCIDR(routes[j].Prefix)
return cmp.Or(cmp.Compare(prefixA.Bits(), prefixB.Bits()),
prefixA.Addr().Compare(prefixB.Addr()))
// Compare by network address first
cmp := bytes.Compare(ipI.To16(), ipJ.To16())
if cmp != 0 {
return cmp < 0
}
// If network addresses are equal, compare by mask length
onesI, _ := netI.Mask.Size()
onesJ, _ := netJ.Mask.Size()
return onesI < onesJ
})
}
@@ -1272,8 +1212,9 @@ func (s *Server) handleASDetail() http.HandlerFunc {
// Group prefixes by IP version
ipv4Prefixes, ipv6Prefixes := splitRoutesByIPVersion(prefixes)
sortRoutesByMaskLength(ipv4Prefixes)
sortRoutesByMaskLength(ipv6Prefixes)
// Sort prefixes by network address
sortRoutesByNetwork(ipv4Prefixes)
sortRoutesByNetwork(ipv6Prefixes)
// Prepare template data
data := struct {
internal/server/handlers_internal_test.go
+72 -450
View File
@@ -5,10 +5,8 @@ import (
"encoding/json"
"errors"
"html"
"net"
"net/http"
"net/http/httptest"
"net/netip"
"runtime"
"slices"
"strconv"
@@ -17,7 +15,6 @@ import (
"testing"
"time"
"git.eeqj.de/sneak/routewatch/internal/blocklist"
"git.eeqj.de/sneak/routewatch/internal/config"
"git.eeqj.de/sneak/routewatch/internal/database"
"git.eeqj.de/sneak/routewatch/internal/logger"
@@ -198,51 +195,6 @@ func TestStatsHandlersAnswerFromMemory(t *testing.T) {
}
}
// TestStatsAnswerCarriesChurnPerMinute checks that the stream object of the
// /api/v1/stats answer holds route_churn_per_min as a whole number, and no
// longer route_churn_per_sec, the moving average it replaces
// (https://git.eeqj.de/sneak/routewatch/issues/160).
func TestStatsAnswerCarriesChurnPerMinute(t *testing.T) {
t.Parallel()
s := newServerWithTestRoutes(t)
rec := httptest.NewRecorder()
s.handleStats()(rec,
httptest.NewRequestWithContext(t.Context(), http.MethodGet, statsPath, nil))
if rec.Code != http.StatusOK {
t.Fatalf("status %d, want %d; body %s", rec.Code, http.StatusOK, rec.Body)
}
var body struct {
Data struct {
Stream map[string]json.RawMessage `json:"stream"`
} `json:"data"`
}
err := json.Unmarshal(rec.Body.Bytes(), &body)
if err != nil {
t.Fatalf("decoding the answer: %v", err)
}
churn, ok := body.Data.Stream["route_churn_per_min"]
if !ok {
t.Fatalf("stream has no route_churn_per_min: %s", rec.Body)
}
var count int
err = json.Unmarshal(churn, &count)
if err != nil {
t.Errorf("route_churn_per_min = %s, want a whole number: %v", churn, err)
}
if _, ok := body.Data.Stream["route_churn_per_sec"]; ok {
t.Errorf("stream still has route_churn_per_sec: %s", rec.Body)
}
}
// newServerWithTestRoutes returns a server on a new database that holds a
// route for testPrefixV4 and one for testPrefixV6.
func newServerWithTestRoutes(t *testing.T) *Server {
@@ -461,8 +413,8 @@ func TestPrefixDetailShowsOriginNames(t *testing.T) {
// TestPagesSayOriginZeroIsNoKnownOrigin stores a route with origin 0, as the
// prefix handler stores a route whose AS path ends in an AS set of several
// members, and checks that the prefix page and the prefix length page say it
// has no known origin, without a link to AS0
// (https://git.eeqj.de/sneak/routewatch/issues/75).
// has no known origin, without a link to AS0 or the description of the
// reserved AS 0 (https://git.eeqj.de/sneak/routewatch/issues/75).
func TestPagesSayOriginZeroIsNoKnownOrigin(t *testing.T) {
t.Parallel()
@@ -482,6 +434,11 @@ func TestPagesSayOriginZeroIsNoKnownOrigin(t *testing.T) {
t.Fatalf("UpsertLiveRouteBatch: %v", err)
}
reserved, ok := asinfo.Get(0)
if !ok {
t.Fatal("AS0 is not in the AS info data")
}
s := New(db, streamer.New(logger.New(), metrics.New()), logger.New(), &config.Config{})
for _, path := range []string{"/prefix/" + testPrefixV4, prefixLengthPagePath} {
@@ -503,128 +460,9 @@ func TestPagesSayOriginZeroIsNoKnownOrigin(t *testing.T) {
if strings.Contains(page, `href="/as/0"`) {
t.Errorf("%s links to AS0", path)
}
}
}
// TestIndexShowsTheClientAddress requests the home page and checks that it has
// the IP information API card, holding the address /ip/ would look up for the
// same request: the TCP peer, or, when the peer is a trusted proxy, the client
// the proxy names in X-Forwarded-For
// (https://git.eeqj.de/sneak/routewatch/issues/156).
func TestIndexShowsTheClientAddress(t *testing.T) {
t.Parallel()
const cardHeading = "<h2>IP Information API</h2>"
// Both requests carry an X-Forwarded-For naming forwardedClient; only a
// trusted proxy's is believed.
tests := []struct {
name string
trustedProxies []netip.Prefix
peer string
want string
}{
{"direct request", nil, untrustedPeer, untrustedPeerAddr},
{
"request through a trusted proxy",
[]netip.Prefix{netip.MustParsePrefix(trustedProxyRange)},
trustedPeer, forwardedClient,
},
}
for _, tt := range tests {
// The home page reads nothing from the database, so the server has none.
s := New(nil, streamer.New(logger.New(), metrics.New()), logger.New(),
&config.Config{TrustedProxies: tt.trustedProxies})
req := httptest.NewRequestWithContext(t.Context(), http.MethodGet, "/", nil)
req.RemoteAddr = tt.peer
req.Header.Set("X-Forwarded-For", forwardedClient)
rec := httptest.NewRecorder()
s.router.ServeHTTP(rec, req)
if rec.Code != http.StatusOK {
t.Errorf("%s: status %d, want %d; body %s",
tt.name, rec.Code, http.StatusOK, rec.Body)
continue
}
page := rec.Body.String()
if !strings.Contains(page, cardHeading) {
t.Errorf("%s: the page has no IP information API card", tt.name)
}
want := `data-client-ip="` + tt.want + `"`
if !strings.Contains(page, want) {
t.Errorf("%s: the card does not hold %s", tt.name, want)
}
}
}
// TestPagesShowAgesInWords stores routes last updated about 5 minutes, 3 hours
// and 15 days ago, and checks that the AS page and the prefix pages show each
// age in words. Before the fix the ages read "5m0s ago", "3h0m0s ago" and
// "360h0m0s ago" (https://git.eeqj.de/sneak/routewatch/issues/163).
func TestPagesShowAgesInWords(t *testing.T) {
t.Parallel()
db, err := database.New(&config.Config{StateDir: t.TempDir()}, logger.New())
if err != nil {
t.Fatalf("database.New: %v", err)
}
t.Cleanup(func() { _ = db.Close() })
// Each age is well past a whole number of its unit, so the time the test
// takes to run does not change the words.
ages := []struct {
prefix string
age time.Duration
want string
}{
{testPrefixV4, 5*time.Minute + 30*time.Second, "5 minutes ago"},
{"198.51.100.0/24", 3*time.Hour + 30*time.Minute, "3 hours ago"},
{"203.0.113.0/24", 15 * 24 * time.Hour, "2 weeks ago"},
}
now := time.Now().UTC()
routes := make([]*database.LiveRoute, 0, len(ages))
for _, a := range ages {
routes = append(routes, &database.LiveRoute{
Prefix: a.prefix, MaskLength: 24, IPVersion: 4,
OriginASN: 64500, PeerIP: testPeerV4, ASPath: []int{64500},
NextHop: testPeerV4, LastUpdated: now.Add(-a.age),
})
}
err = db.UpsertLiveRouteBatch(routes)
if err != nil {
t.Fatalf("UpsertLiveRouteBatch: %v", err)
}
s := New(db, streamer.New(logger.New(), metrics.New()), logger.New(), &config.Config{})
for _, a := range ages {
cell := `<td class="age">` + a.want + `</td>`
for _, path := range []string{asPagePath, "/prefix/" + a.prefix} {
rec := httptest.NewRecorder()
s.router.ServeHTTP(rec, httptest.NewRequestWithContext(
t.Context(), http.MethodGet, path, nil))
if rec.Code != http.StatusOK {
t.Fatalf("%s: status %d, want %d; body %s",
path, rec.Code, http.StatusOK, rec.Body)
}
if !strings.Contains(rec.Body.String(), cell) {
t.Errorf("%s does not show the age of the route for %s as %q",
path, a.prefix, a.want)
}
if strings.Contains(page, html.EscapeString(reserved.Description)) {
t.Errorf("%s shows AS0's description %q", path, reserved.Description)
}
}
}
@@ -1042,15 +880,14 @@ func (failingIPLookupDB) GetIPInfoContext(
// server's router with a store whose read of one address fails with the
// driver's error. A lookup that asks for that address answers 500 with the
// JSON error body, as the AS and prefix JSON routes do; an address with no
// route answers 200. Before the fix the failed read was reported as if the
// address had no route: 404 for that address alone, and 200 beside a routed
// address (https://git.eeqj.de/sneak/routewatch/issues/109).
// route still answers 404. Before the fix the failed read was reported as if
// the address had no route: 404 for that address alone, and 200 beside a
// routed address (https://git.eeqj.de/sneak/routewatch/issues/109).
func TestIPLookupAnswers500WhenDatabaseReadFails(t *testing.T) {
t.Parallel()
s := New(failingIPLookupDB{}, streamer.New(logger.New(), metrics.New()),
logger.New(), &config.Config{})
s.resolver = &fakeResolver{}
tests := []struct {
path string
@@ -1066,7 +903,7 @@ func TestIPLookupAnswers500WhenDatabaseReadFails(t *testing.T) {
},
{
path: "/api/v1/ip/203.0.113.1",
wantCode: http.StatusOK,
wantCode: http.StatusNotFound,
},
}
@@ -1080,9 +917,7 @@ func TestIPLookupAnswers500WhenDatabaseReadFails(t *testing.T) {
tt.path, rec.Code, tt.wantCode, rec.Body)
}
if tt.wantCode != http.StatusOK {
checkJSONErrorCode(t, tt.path, rec, tt.wantCode)
}
checkJSONErrorCode(t, tt.path, rec, tt.wantCode)
}
}
@@ -1170,10 +1005,10 @@ func TestIPLookupRefusesTooManyAddresses(t *testing.T) {
}{
{
strings.Join(addrs[:maxLookupTargets], ","),
http.StatusOK, maxLookupTargets, 0,
http.StatusNotFound, maxLookupTargets, 0,
},
{strings.Join(addrs, ","), http.StatusBadRequest, 0, 0},
{"most.test", http.StatusOK, maxLookupTargets, 1},
{"most.test", http.StatusNotFound, maxLookupTargets, 1},
{"too-many.test", http.StatusBadRequest, 0, 1},
{strings.Join(names, ","), http.StatusBadRequest, 0, 0},
}
@@ -1194,9 +1029,7 @@ func TestIPLookupRefusesTooManyAddresses(t *testing.T) {
tt.target, rec.Code, tt.wantCode, rec.Body)
}
if tt.wantCode != http.StatusOK {
checkJSONErrorCode(t, tt.target, rec, tt.wantCode)
}
checkJSONErrorCode(t, tt.target, rec, tt.wantCode)
got := db.lookups.Load()
if got != tt.wantLookups {
@@ -1253,191 +1086,6 @@ func TestIPLookupKeepsRouteWhenPTRLookupDoesNotAnswer(t *testing.T) {
}
}
// addrInTestPrefixV6 is an address in testPrefixV6, which
// newServerWithTestRoutes stores a route for.
const addrInTestPrefixV6 = "2001:db8::1"
// testBlocklistInterval is fakeBlocklists' shortest interval.
const testBlocklistInterval = 15 * time.Minute
// ownAddressPath is the IP lookup of the caller's own address.
const ownAddressPath = "/ip/"
// listedUnroutedAddr is an address that fakeBlocklists lists, outside the
// routes newServerWithTestRoutes stores.
const listedUnroutedAddr = "203.0.113.1"
// fakeBlocklists has one list, which lists addrInTestPrefixV4 and
// listedUnroutedAddr and no other address.
type fakeBlocklists struct{}
func testListing() blocklist.Listing {
return blocklist.Listing{
Name: "test_list",
Description: "On the test list",
Updated: time.Date(2026, 10, 27, 4, 0, 0, 0, time.UTC),
}
}
func (fakeBlocklists) Lookup(addr netip.Addr) []blocklist.Listing {
if addr == netip.MustParseAddr(addrInTestPrefixV4) ||
addr == netip.MustParseAddr(listedUnroutedAddr) {
return []blocklist.Listing{testListing()}
}
return []blocklist.Listing{}
}
func (fakeBlocklists) ShortestInterval() time.Duration {
return testBlocklistInterval
}
// TestIPLookupReportsBlocklists looks up two routed addresses, one of them on
// a blocklist. The listed address's result names the list, says what it means
// and when it was updated. The other's result has an empty list.
func TestIPLookupReportsBlocklists(t *testing.T) {
t.Parallel()
s := newServerWithTestRoutes(t)
s.resolver = &fakeResolver{}
s.SetBlocklists(fakeBlocklists{})
rec := httptest.NewRecorder()
s.router.ServeHTTP(rec, httptest.NewRequestWithContext(t.Context(),
http.MethodGet,
"/api/v1/ip/"+addrInTestPrefixV4+","+addrInTestPrefixV6, nil))
if rec.Code != http.StatusOK {
t.Fatalf("status %d, want %d; body %s", rec.Code, http.StatusOK, rec.Body)
}
var body struct {
Data IPLookupResponse `json:"data"`
}
err := json.Unmarshal(rec.Body.Bytes(), &body)
if err != nil {
t.Fatalf("decoding the answer %q: %v", rec.Body, err)
}
results := body.Data.Results
if len(results) != 2 {
t.Fatalf("answer %s, want two results", rec.Body)
}
want := testListing()
listed := results[0].Blocklists
if len(listed) != 1 || listed[0].Name != want.Name ||
listed[0].Description != want.Description ||
!listed[0].Updated.Equal(want.Updated) {
t.Errorf("%s: blocklists %+v, want [%+v]", results[0].IP, listed, want)
}
if !strings.Contains(rec.Body.String(), `"blocklists":[]`) {
t.Errorf("answer %s, want an empty blocklists list for %s",
rec.Body, addrInTestPrefixV6)
}
}
// TestIPLookupReportsBlocklistsOfAnAddressWithNoRoute looks up a listed
// address that has no live route, alone and beside a routed address. Each
// answer is 200 and has a result for the address, with its blocklists and no
// route fields. Before the fix the lookup of that address alone answered
// 404, and beside a routed address it was only named in errors, so its
// blocklists were never reported.
func TestIPLookupReportsBlocklistsOfAnAddressWithNoRoute(t *testing.T) {
t.Parallel()
s := newServerWithTestRoutes(t)
s.resolver = &fakeResolver{}
s.SetBlocklists(fakeBlocklists{})
for _, target := range []string{
listedUnroutedAddr,
addrInTestPrefixV4 + "," + listedUnroutedAddr,
} {
rec := httptest.NewRecorder()
s.router.ServeHTTP(rec, httptest.NewRequestWithContext(t.Context(),
http.MethodGet, "/api/v1/ip/"+target, nil))
var body struct {
Data IPLookupResponse `json:"data"`
}
err := json.Unmarshal(rec.Body.Bytes(), &body)
if rec.Code != http.StatusOK || err != nil {
t.Errorf("%s: status %d (%v), want %d; body %s",
target, rec.Code, err, http.StatusOK, rec.Body)
continue
}
i := slices.IndexFunc(body.Data.Results, func(r IPLookupResult) bool {
return r.IP == listedUnroutedAddr
})
if i < 0 || len(body.Data.Errors) != 0 {
t.Errorf("%s: answer %s, want a result for %s and no errors",
target, rec.Body, listedUnroutedAddr)
continue
}
result := body.Data.Results[i]
if result.IPInfo != nil {
t.Errorf("%s: result for %s has route fields %+v",
target, listedUnroutedAddr, result.IPInfo)
}
if len(result.Blocklists) != 1 ||
result.Blocklists[0].Name != testListing().Name {
t.Errorf("%s: blocklists %+v, want [%+v]",
target, result.Blocklists, testListing())
}
}
}
// TestIPLookupAnswersMayBeCached checks the Cache-Control header of the IP
// lookup answers. Caches may keep them for the blocklists' shortest interval,
// and only the caller's own browser may keep the answer about the caller's
// own address.
func TestIPLookupAnswersMayBeCached(t *testing.T) {
t.Parallel()
s := newServerWithTestRoutes(t)
s.resolver = &fakeResolver{}
s.SetBlocklists(fakeBlocklists{})
tests := []struct {
path string
want string
}{
{"/api/v1/ip/" + addrInTestPrefixV4, "public, max-age=900"},
{ownAddressPath + addrInTestPrefixV4, "public, max-age=900"},
{ownAddressPath, "private, max-age=900"},
}
for _, tt := range tests {
req := httptest.NewRequestWithContext(t.Context(), http.MethodGet,
tt.path, nil)
req.RemoteAddr = net.JoinHostPort(addrInTestPrefixV4, "1234")
rec := httptest.NewRecorder()
s.router.ServeHTTP(rec, req)
if rec.Code != http.StatusOK {
t.Errorf("%s: status %d, want %d; body %s",
tt.path, rec.Code, http.StatusOK, rec.Body)
continue
}
if got := rec.Header().Get(cacheControlHeader); got != tt.want {
t.Errorf("%s: Cache-Control %q, want %q", tt.path, got, tt.want)
}
}
}
// The host name TestIPLookupHidesPrivateAddresses resolves to a private
// address, and that address.
const (
@@ -1569,7 +1217,7 @@ func TestASSearchAnswersMatches(t *testing.T) {
}
cloudflare := asinfo.Info{
ASN: 13335, Handle: "CLOUDFLARENET", Description: "Cloudflare Inc.",
ASN: 13335, Handle: "CLOUDFLARENET", Description: "Cloudflare, Inc.",
}
if !slices.Contains(data.Results, cloudflare) {
t.Errorf("results %v do not hold %v", data.Results, cloudflare)
@@ -1642,9 +1290,8 @@ func TestASSearchShowsFirstMatchesAndTotal(t *testing.T) {
// The AS numbers the AS detail tests ask for. storedASN originates the route
// for testPrefixV4 and is in the database. routeOnlyASN originates the route
// for testPrefixV6, but the database has no record of the AS itself, as when
// a route is stored before its AS. Both are documentation AS numbers, which
// the AS list built into asinfo does not hold. builtInOnlyASN is in that list
// but not in the database. unknownASN is in neither.
// a route is stored before its AS. builtInOnlyASN is in the AS list built into
// asinfo but not in the database. unknownASN is in neither.
const (
storedASN = 64500
routeOnlyASN = 64501
@@ -1660,12 +1307,14 @@ const firstSeenCard = `<div class="info-label">First Seen</div>`
// newServerWithStoredAS returns newServerWithTestRoutes's server with storedASN
// added to the database. It fails the test when the AS list built into asinfo
// lacks builtInOnlyASN or holds unknownASN.
// lacks routeOnlyASN or builtInOnlyASN, or holds unknownASN.
func newServerWithStoredAS(t *testing.T) *Server {
t.Helper()
if _, ok := asinfo.Get(builtInOnlyASN); !ok {
t.Fatalf("AS%d is not in the AS info data", builtInOnlyASN)
for _, asn := range []int{routeOnlyASN, builtInOnlyASN} {
if _, ok := asinfo.Get(asn); !ok {
t.Fatalf("AS%d is not in the AS info data", asn)
}
}
if _, ok := asinfo.Get(unknownASN); ok {
@@ -1683,11 +1332,12 @@ func newServerWithStoredAS(t *testing.T) *Server {
}
// TestASDetailJSONAnswersBuiltInAS asks /api/v1/as/ for each of the ASes above.
// All but unknownASN answer 200 with the handle and description the built-in
// AS list has for them, if any, and their prefixes, as lists even when empty.
// Only storedASN has a first_seen time. unknownASN answers 404. Before the fix
// builtInOnlyASN and routeOnlyASN answered 404 too
// (https://git.eeqj.de/sneak/routewatch/issues/151).
// All but unknownASN answer 200 with the handle and description from the
// built-in AS list and their prefixes, as lists even when empty. Only storedASN
// has a first_seen time. unknownASN answers 404. Before the fix builtInOnlyASN
// answered 404, so a result of the AS name search could lead nowhere
// (https://git.eeqj.de/sneak/routewatch/issues/151), and routeOnlyASN listed
// no prefixes.
func TestASDetailJSONAnswersBuiltInAS(t *testing.T) {
t.Parallel()
@@ -1740,13 +1390,13 @@ func TestASDetailJSONAnswersBuiltInAS(t *testing.T) {
continue
}
handle, description := asinfo.GetHandle(tt.asn), asinfo.GetDescription(tt.asn)
info, _ := asinfo.Get(tt.asn)
got := body.Data.ASN
if got.ASN != tt.asn || got.Handle != handle ||
got.Description != description {
if got.ASN != tt.asn || got.Handle != info.Handle ||
got.Description != info.Description {
t.Errorf("%s: AS %d %q %q, want %d %q %q", path, got.ASN,
got.Handle, got.Description, tt.asn, handle, description)
got.Handle, got.Description, tt.asn, info.Handle, info.Description)
}
if body.Data.IPv4Prefixes == nil || body.Data.IPv6Prefixes == nil {
@@ -1773,12 +1423,13 @@ func TestASDetailJSONAnswersBuiltInAS(t *testing.T) {
}
// TestASDetailPageShowsBuiltInAS asks /as/ for each of the ASes above. All but
// unknownASN answer 200 with the handle and description the built-in AS list
// has for them, if any. storedASN and routeOnlyASN list their prefixes, and
// builtInOnlyASN says the AS announces none. Only storedASN shows a first-seen
// date, and no page shows a date that was never set. unknownASN answers 404.
// Before the fix builtInOnlyASN and routeOnlyASN answered 404 too
// (https://git.eeqj.de/sneak/routewatch/issues/151).
// unknownASN answer 200 with the handle and description from the built-in AS
// list. storedASN and routeOnlyASN list their prefixes, and builtInOnlyASN says
// the AS announces none. Only storedASN shows a first-seen date, and no page
// shows a date that was never set. unknownASN answers 404. Before the fix
// builtInOnlyASN answered 404, so a result of the AS name search led nowhere
// (https://git.eeqj.de/sneak/routewatch/issues/151), and routeOnlyASN listed no
// prefixes.
func TestASDetailPageShowsBuiltInAS(t *testing.T) {
t.Parallel()
@@ -1813,11 +1464,12 @@ func TestASDetailPageShowsBuiltInAS(t *testing.T) {
continue
}
info, _ := asinfo.Get(tt.asn)
page := rec.Body.String()
for _, want := range []string{
html.EscapeString(asinfo.GetHandle(tt.asn)),
html.EscapeString(asinfo.GetDescription(tt.asn)),
html.EscapeString(info.Handle),
html.EscapeString(info.Description),
tt.wantText,
} {
if !strings.Contains(page, want) {
@@ -1837,68 +1489,19 @@ func TestASDetailPageShowsBuiltInAS(t *testing.T) {
}
}
// TestSortRoutesByMaskLength expects the AS page's prefix lists in mask length
// order, shortest first, and in address order within one mask length. Before
// the fix they were in address order, so 10.1.0.0/16 came before 11.0.0.0/8
// (https://git.eeqj.de/sneak/routewatch/issues/153).
func TestSortRoutesByMaskLength(t *testing.T) {
t.Parallel()
tests := []struct {
name string
prefixes []string
want []string
}{
{
name: "IPv4",
prefixes: []string{
"100.64.1.0/24", "11.0.0.0/8", "10.1.0.0/16",
"20.0.1.0/24", "172.16.0.0/12", "9.0.0.0/8",
},
want: []string{
"9.0.0.0/8", "11.0.0.0/8", "172.16.0.0/12",
"10.1.0.0/16", "20.0.1.0/24", "100.64.1.0/24",
},
},
{
name: "IPv6",
prefixes: []string{
"2001:db8:1::/48", "2001:db8:8000::/33", "2001:db8:0:1::/64",
"2001:db8::/48", "2001:db8::/33",
},
want: []string{
"2001:db8::/33", "2001:db8:8000::/33", "2001:db8::/48",
"2001:db8:1::/48", "2001:db8:0:1::/64",
},
},
}
for _, tt := range tests {
routes := make([]database.LiveRoute, 0, len(tt.prefixes))
for _, prefix := range tt.prefixes {
routes = append(routes, database.LiveRoute{Prefix: prefix})
}
sortRoutesByMaskLength(routes)
got := make([]string, 0, len(routes))
for _, route := range routes {
got = append(got, route.Prefix)
}
if !slices.Equal(got, tt.want) {
t.Errorf("%s: order %v, want %v", tt.name, got, tt.want)
}
}
}
// TestNoOriginHasNoASAnswers stores a route with no known origin, which has
// origin AS 0 (https://git.eeqj.de/sneak/routewatch/issues/75). It expects
// /as/0 and /api/v1/as/0 to answer 404.
// /as/0 and /api/v1/as/0 to answer 404. It expects a search for "iana" to leave
// AS 0 out of its results and its total, although the AS list built into
// asinfo holds AS 0. Before the fix both AS answers were 200 and listed the
// route's prefix, and the search listed AS 0 first.
func TestNoOriginHasNoASAnswers(t *testing.T) {
t.Parallel()
const noOriginPrefix = "203.0.113.0/24"
const (
noOriginPrefix = "203.0.113.0/24"
query = "iana"
)
s := newServerWithTestRoutes(t)
@@ -1924,6 +1527,25 @@ func TestNoOriginHasNoASAnswers(t *testing.T) {
path, rec.Code, http.StatusNotFound, rec.Body)
}
}
first, total := asinfo.Search(query, 1)
if len(first) != 1 || first[0].ASN != ristypes.NoOrigin {
t.Fatalf("AS%d is not the first match for %q in the AS info data",
ristypes.NoOrigin, query)
}
data := searchAS(t, s, "q="+query)
for _, info := range data.Results {
if info.ASN == ristypes.NoOrigin {
t.Errorf("the search for %q lists AS%d", query, info.ASN)
}
}
if data.Total != total-1 {
t.Errorf("the search for %q has a total of %d, want %d",
query, data.Total, total-1)
}
}
// settledGoroutineCount lets transient goroutines finish, then reports the
internal/server/middleware_internal_test.go
+4 -5
View File
@@ -11,12 +11,11 @@ import (
)
// The addresses the forwarding tests use: the client a proxy names in
// X-Forwarded-For, the range of trusted proxies, a peer inside it, and a peer
// outside it, which lies in testPrefixV4 so that an IP lookup of it finds a
// route.
// X-Forwarded-For, a peer inside the trusted range of
// TestRealIPMiddlewareTrustsOnlyConfiguredProxies, and a peer outside it,
// which lies in testPrefixV4 so that an IP lookup of it finds a route.
const (
forwardedClient = "203.0.113.9"
trustedProxyRange = "10.0.0.0/8"
trustedPeer = "10.0.0.2:1234"
untrustedPeerAddr = "192.0.2.1"
untrustedPeer = untrustedPeerAddr + ":1234"
@@ -190,7 +189,7 @@ func TestForwardedForFromUntrustedPeerIsIgnored(t *testing.T) {
func TestRealIPMiddlewareTrustsOnlyConfiguredProxies(t *testing.T) {
t.Parallel()
trusted := []netip.Prefix{netip.MustParsePrefix(trustedProxyRange)}
trusted := []netip.Prefix{netip.MustParsePrefix("10.0.0.0/8")}
tests := []struct {
name string
internal/server/server.go
-15
View File
@@ -15,7 +15,6 @@ import (
"strconv"
"time"
"git.eeqj.de/sneak/routewatch/internal/blocklist"
"git.eeqj.de/sneak/routewatch/internal/config"
"git.eeqj.de/sneak/routewatch/internal/database"
"git.eeqj.de/sneak/routewatch/internal/logger"
@@ -37,13 +36,6 @@ type ASNFetcher interface {
GetStats() ASNFetcherStats
}
// Blocklists tells which public blocklists list an address, from copies it
// keeps in memory, as *blocklist.Blocklists does.
type Blocklists interface {
Lookup(addr netip.Addr) []blocklist.Listing
ShortestInterval() time.Duration
}
// dnsResolver looks up the addresses of host names and the names of addresses
// in DNS, as *net.Resolver does.
type dnsResolver interface {
@@ -60,7 +52,6 @@ type Server struct {
port int
srv *http.Server
asnFetcher ASNFetcher
blocklists Blocklists
// trustedProxies are the reverse proxies whose X-Forwarded-For header
// RealIPMiddleware believes
@@ -157,9 +148,3 @@ func (s *Server) Stop(ctx context.Context) error {
func (s *Server) SetASNFetcher(fetcher ASNFetcher) {
s.asnFetcher = fetcher
}
// SetBlocklists sets the blocklists the IP lookups report. Call it before
// Start: the handlers read the blocklists without a lock.
func (s *Server) SetBlocklists(blocklists Blocklists) {
s.blocklists = blocklists
}
internal/templates/as_detail.html
+8 -28
View File
@@ -126,26 +126,6 @@
margin: 0;
color: #2c3e50;
}
/* A click on a prefix list's heading folds or unfolds the list. Not
every browser shows its own marker on a flex element, so the page
hides it and draws a triangle before the heading. The triangle
points down while the list is open. */
summary.prefix-header {
cursor: pointer;
list-style: none;
}
summary.prefix-header::-webkit-details-marker {
display: none;
}
summary.prefix-header h2::before {
content: "\25B8";
display: inline-block;
margin-right: 10px;
transition: transform 0.2s;
}
details[open] > summary.prefix-header h2::before {
transform: rotate(90deg);
}
.prefix-count {
background: #e74c3c;
color: white;
@@ -262,11 +242,11 @@
</div>
{{if .IPv4Prefixes}}
<details class="prefix-section" open>
<summary class="prefix-header">
<div class="prefix-section">
<div class="prefix-header">
<h2>IPv4 Prefixes</h2>
<span class="prefix-count">{{.IPv4Count}}</span>
</summary>
</div>
<table class="prefix-table">
<thead>
<tr>
@@ -287,15 +267,15 @@
{{end}}
</tbody>
</table>
</details>
</div>
{{end}}
{{if .IPv6Prefixes}}
<details class="prefix-section" open>
<summary class="prefix-header">
<div class="prefix-section">
<div class="prefix-header">
<h2>IPv6 Prefixes</h2>
<span class="prefix-count">{{.IPv6Count}}</span>
</summary>
</div>
<table class="prefix-table">
<thead>
<tr>
@@ -316,7 +296,7 @@
{{end}}
</tbody>
</table>
</details>
</div>
{{end}}
{{if eq .TotalCount 0}}
internal/templates/index.html
+8 -100
View File
@@ -107,15 +107,9 @@
.stat-card.connected .stat-value {
color: #27ae60;
}
.stat-card.disconnected .stat-value,
.stale {
.stat-card.disconnected .stat-value {
color: #e74c3c;
}
/* The card has no room beside "Connected", so the spinner and "stale" go under it */
#freshness {
display: block;
font-size: 20px;
}
/* Search section */
.search-section {
@@ -239,8 +233,7 @@
cursor: pointer;
font-size: 12px;
}
.ip-result pre,
#ip-api pre {
.ip-result pre {
background: #2c3e50;
color: #ecf0f1;
padding: 16px;
@@ -252,30 +245,18 @@
max-height: 400px;
overflow-y: auto;
}
.ip-result .error,
#ip-api .error {
.ip-result .error {
background: #fee;
color: #c00;
padding: 12px;
border-radius: 6px;
font-size: 14px;
}
.ip-result .loading,
#ip-api .loading {
.ip-result .loading {
color: #7f8c8d;
font-style: italic;
}
/* IP information API card: the command, then its answer */
#ip-api p {
margin: 0;
color: #7f8c8d;
font-size: 14px;
}
#ip-api pre {
margin-top: 12px;
}
/* Footer */
.footer {
margin-top: 40px;
@@ -316,10 +297,7 @@
<main class="main-content">
<div class="stats-grid">
<div class="stat-card" id="status-card">
<div class="stat-value">
<span id="stat-status">-</span>
<span id="freshness"></span>
</div>
<div class="stat-value" id="stat-status">-</div>
<div class="stat-label">Status</div>
</div>
<div class="stat-card">
@@ -380,13 +358,6 @@
</div>
</div>
</div>
<div class="search-card" id="ip-api">
<h2>IP Information API</h2>
<p>The API answers what RouteWatch knows about an IP address, such as its AS number and country. <code>/ip/</code> with no address answers for the caller's own address.</p>
<pre id="ip-api-command" data-client-ip="{{.ClientIP}}"></pre>
<pre id="ip-api-answer" class="loading">Loading...</pre>
</div>
</main>
<footer class="footer">
@@ -398,44 +369,6 @@
</footer>
<script nonce="{{.CSPNonce}}">
// The stats are fetched every REFRESH_PERIOD_MS. Next to the status, a
// spinner turns one frame each time an answer counts more messages
// than the one before. After STALE_AFTER_REFRESHES refreshes without
// such an answer, the spinner gives way to "stale" and the time since.
const REFRESH_PERIOD_MS = 5000;
const STALE_AFTER_REFRESHES = 3;
const MS_PER_SECOND = 1000;
const SPINNER_FRAMES = ['⠋', '⠙', '⠹', '⠸', '⠼', '⠴', '⠦', '⠧', '⠇', '⠏'];
const SPINNER_COLOURS = ['#e74c3c', '#e67e22', '#f1c40f', '#27ae60', '#3498db', '#9b59b6'];
let spinnerFrame = 0;
let lastMessageCount = null;
let lastFreshAt = Date.now();
// Turn the spinner when the message count has risen since the last answer.
function noteMessageCount(count) {
if (lastMessageCount !== null && count > lastMessageCount) {
spinnerFrame = (spinnerFrame + 1) % SPINNER_FRAMES.length;
lastFreshAt = Date.now();
}
lastMessageCount = count;
showFreshness();
}
function showFreshness() {
const freshness = document.getElementById('freshness');
const sinceFreshMs = Date.now() - lastFreshAt;
const stale = sinceFreshMs >= STALE_AFTER_REFRESHES * REFRESH_PERIOD_MS;
freshness.classList.toggle('stale', stale);
if (stale) {
freshness.textContent = 'stale for ' + Math.floor(sinceFreshMs / MS_PER_SECOND) + 's';
freshness.style.color = '';
return;
}
freshness.textContent = SPINNER_FRAMES[spinnerFrame];
freshness.style.color = SPINNER_COLOURS[spinnerFrame % SPINNER_COLOURS.length];
}
function formatNumber(num) {
if (num >= 1000000) {
return (num / 1000000).toFixed(1) + 'M';
@@ -457,7 +390,6 @@
const statusEl = document.getElementById('stat-status');
statusEl.textContent = data.connected ? 'Connected' : 'Disconnected';
statusCard.className = 'stat-card ' + (data.connected ? 'connected' : 'disconnected');
noteMessageCount(data.total_messages);
document.getElementById('stat-routes').textContent = formatNumber(data.live_routes);
document.getElementById('stat-asns').textContent = formatNumber(data.asns);
@@ -468,7 +400,7 @@
}
const totalUpdates = data.ipv4_updates_per_sec + data.ipv6_updates_per_sec;
document.getElementById('stat-updates').textContent = Math.round(totalUpdates).toLocaleString();
document.getElementById('stat-updates').textContent = totalUpdates.toFixed(1);
})
.catch(() => {
document.getElementById('stat-status').textContent = 'Error';
@@ -604,33 +536,9 @@
document.getElementById('ip-input').value = '';
});
// IP information API card: show the curl command that asks /ip/ about
// the visitor's own address, then make that same request and show the
// JSON it answers, an error answer included. The answer holds outside
// data, so it is set with textContent, never innerHTML.
const ipAPICommand = document.getElementById('ip-api-command');
const ipAPIAnswer = document.getElementById('ip-api-answer');
const ipAPIURL = window.location.origin + '/ip/' + ipAPICommand.dataset.clientIp;
ipAPICommand.textContent = 'curl ' + ipAPIURL;
fetch(ipAPIURL)
.then(response => response.json())
.then(answer => {
ipAPIAnswer.className = '';
ipAPIAnswer.textContent = JSON.stringify(answer, null, 2);
})
.catch(error => {
ipAPIAnswer.className = 'error';
ipAPIAnswer.textContent = 'Error: ' + error.message;
});
// Initial load and refresh stats every REFRESH_PERIOD_MS. The freshness
// check runs every second on its own, so a request that never returns
// still turns the spinner into "stale".
// Initial load and refresh stats every 5 seconds
updateStats();
setInterval(updateStats, REFRESH_PERIOD_MS);
setInterval(showFreshness, MS_PER_SECOND);
setInterval(updateStats, 5000);
</script>
</body>
</html>
internal/templates/status.html
+12 -58
View File
@@ -126,8 +126,7 @@
.connected {
color: #22c55e;
}
.disconnected,
.stale {
.disconnected {
color: #ef4444;
}
.error {
@@ -181,10 +180,7 @@
<h2>RouteWatch</h2>
<div class="metric">
<span class="metric-label">Status</span>
<span>
<span class="metric-value" id="connected">-</span>
<span class="metric-value" id="freshness"></span>
</span>
<span class="metric-value" id="connected">-</span>
</div>
<div class="metric">
<span class="metric-label">Uptime</span>
@@ -235,8 +231,8 @@
<span class="metric-value" id="withdrawals">-</span>
</div>
<div class="metric">
<span class="metric-label">Route Churn/min</span>
<span class="metric-value" id="route_churn_per_min">-</span>
<span class="metric-label">Route Churn/sec</span>
<span class="metric-value" id="route_churn_per_sec">-</span>
</div>
<div class="metric">
<span class="metric-label">Total Data</span>
@@ -390,44 +386,6 @@
</div>
<script nonce="{{.CSPNonce}}">
// The stats are fetched every REFRESH_PERIOD_MS. Next to the status, a
// spinner turns one frame each time an answer counts more messages
// than the one before. After STALE_AFTER_REFRESHES refreshes without
// such an answer, the spinner gives way to "stale" and the time since.
const REFRESH_PERIOD_MS = 2000;
const STALE_AFTER_REFRESHES = 3;
const MS_PER_SECOND = 1000;
const SPINNER_FRAMES = ['⠋', '⠙', '⠹', '⠸', '⠼', '⠴', '⠦', '⠧', '⠇', '⠏'];
const SPINNER_COLOURS = ['#e74c3c', '#e67e22', '#f1c40f', '#27ae60', '#3498db', '#9b59b6'];
let spinnerFrame = 0;
let lastMessageCount = null;
let lastFreshAt = Date.now();
// Turn the spinner when the message count has risen since the last answer.
function noteMessageCount(count) {
if (lastMessageCount !== null && count > lastMessageCount) {
spinnerFrame = (spinnerFrame + 1) % SPINNER_FRAMES.length;
lastFreshAt = Date.now();
}
lastMessageCount = count;
showFreshness();
}
function showFreshness() {
const freshness = document.getElementById('freshness');
const sinceFreshMs = Date.now() - lastFreshAt;
const stale = sinceFreshMs >= STALE_AFTER_REFRESHES * REFRESH_PERIOD_MS;
freshness.classList.toggle('stale', stale);
if (stale) {
freshness.textContent = 'stale for ' + Math.floor(sinceFreshMs / MS_PER_SECOND) + 's';
freshness.style.color = '';
return;
}
freshness.textContent = SPINNER_FRAMES[spinnerFrame];
freshness.style.color = SPINNER_COLOURS[spinnerFrame % SPINNER_COLOURS.length];
}
function formatBytes(bytes) {
if (bytes === 0) return '0 B';
const k = 1024;
@@ -553,7 +511,7 @@
document.getElementById('messages_per_sec').textContent = '-';
document.getElementById('announcements').textContent = '-';
document.getElementById('withdrawals').textContent = '-';
document.getElementById('route_churn_per_min').textContent = '-';
document.getElementById('route_churn_per_sec').textContent = '-';
document.getElementById('total_wire_bytes').textContent = '-';
document.getElementById('wire_mbits_per_sec').textContent = '-';
document.getElementById('gc_num').textContent = '-';
@@ -611,8 +569,7 @@
const connectedEl = document.getElementById('connected');
connectedEl.textContent = data.connected ? 'Connected' : 'Disconnected';
connectedEl.className = 'metric-value ' + (data.connected ? 'connected' : 'disconnected');
noteMessageCount(data.total_messages);
// Update all metrics
document.getElementById('uptime').textContent = data.uptime;
document.getElementById('go_version').textContent = data.go_version;
@@ -621,7 +578,7 @@
document.getElementById('connection_duration').textContent = data.connection_duration;
document.getElementById('reconnect_count').textContent = formatNumber(data.reconnect_count);
document.getElementById('total_messages').textContent = formatNumber(data.total_messages);
document.getElementById('messages_per_sec').textContent = formatNumber(Math.round(data.messages_per_sec));
document.getElementById('messages_per_sec').textContent = data.messages_per_sec.toFixed(1);
document.getElementById('total_wire_bytes').textContent = formatBytes(data.total_wire_bytes);
document.getElementById('wire_mbits_per_sec').textContent = data.wire_mbits_per_sec.toFixed(2) + ' Mbps';
document.getElementById('asns').textContent = formatNumber(data.asns);
@@ -634,8 +591,8 @@
document.getElementById('live_routes').textContent = formatNumber(data.live_routes);
document.getElementById('ipv4_routes').textContent = formatNumber(data.ipv4_routes);
document.getElementById('ipv6_routes').textContent = formatNumber(data.ipv6_routes);
document.getElementById('ipv4_updates_per_sec').textContent = formatNumber(Math.round(data.ipv4_updates_per_sec));
document.getElementById('ipv6_updates_per_sec').textContent = formatNumber(Math.round(data.ipv6_updates_per_sec));
document.getElementById('ipv4_updates_per_sec').textContent = data.ipv4_updates_per_sec.toFixed(1);
document.getElementById('ipv6_updates_per_sec').textContent = data.ipv6_updates_per_sec.toFixed(1);
document.getElementById('oldest_route').textContent = formatRelativeTime(data.oldest_route);
document.getElementById('newest_route').textContent = formatRelativeTime(data.newest_route);
@@ -644,7 +601,7 @@
document.getElementById('bgp_peer_count').textContent = formatNumber(data.stream.bgp_peer_count);
document.getElementById('announcements').textContent = formatNumber(data.stream.announcements);
document.getElementById('withdrawals').textContent = formatNumber(data.stream.withdrawals);
document.getElementById('route_churn_per_min').textContent = formatNumber(data.stream.route_churn_per_min);
document.getElementById('route_churn_per_sec').textContent = data.stream.route_churn_per_sec.toFixed(1);
}
// Update GC stats
@@ -686,12 +643,9 @@
});
}
// Update immediately and then every REFRESH_PERIOD_MS. The freshness
// check runs every second on its own, so a request that never returns
// still turns the spinner into "stale".
// Update immediately and then every 2 seconds
updateStatus();
setInterval(updateStatus, REFRESH_PERIOD_MS);
setInterval(showFreshness, MS_PER_SECOND);
setInterval(updateStatus, 2000);
</script>
</main>
internal/templates/templates.go
+44 -3
View File
@@ -7,9 +7,9 @@ import (
"net/url"
"strings"
"sync"
"time"
"git.eeqj.de/sneak/routewatch/internal/version"
"github.com/dustin/go-humanize"
)
//go:embed index.html
@@ -48,7 +48,48 @@ var (
once sync.Once
)
const cidrPartCount = 2 // A CIDR has two parts: prefix and length
const (
hoursPerDay = 24
daysPerMonth = 30
cidrPartCount = 2 // A CIDR has two parts: prefix and length
)
// timeSince returns a human-readable duration since the given time
func timeSince(t time.Time) string {
duration := time.Since(t)
if duration < time.Minute {
return "just now"
}
if duration < time.Hour {
minutes := int(duration.Minutes())
if minutes == 1 {
return "1 minute ago"
}
return duration.Truncate(time.Minute).String() + " ago"
}
if duration < hoursPerDay*time.Hour {
hours := int(duration.Hours())
if hours == 1 {
return "1 hour ago"
}
return duration.Truncate(time.Hour).String() + " ago"
}
days := int(duration.Hours() / hoursPerDay)
if days == 1 {
return "1 day ago"
}
if days < daysPerMonth {
return duration.Truncate(hoursPerDay*time.Hour).String() + " ago"
}
return t.Format("2006-01-02")
}
// prefixURL generates a URL path for a prefix in CIDR notation.
// Takes a prefix like "192.168.1.0/24" and returns "/prefix/192.168.1.0/24"
@@ -72,7 +113,7 @@ func initTemplates() {
// Create common template functions
funcs := template.FuncMap{
"timeSince": humanize.Time,
"timeSince": timeSince,
"urlEncode": url.QueryEscape,
"prefixURL": prefixURL,
"appName": func() string { return version.Name },
internal/templates/templates_test.go
+8 -135
View File
@@ -6,25 +6,9 @@ import (
"strings"
"testing"
"git.eeqj.de/sneak/routewatch/internal/database"
"git.eeqj.de/sneak/routewatch/internal/templates"
)
// renderPage renders a page with the data every page needs, and the visitor's
// address, which the home page also needs.
func renderPage(t *testing.T, tmpl *template.Template) string {
t.Helper()
var page bytes.Buffer
err := tmpl.Execute(&page, struct{ CSPNonce, ClientIP string }{})
if err != nil {
t.Fatalf("rendering the page: %v", err)
}
return page.String()
}
// TestFooterLicense renders the home and status pages and expects each footer
// to name the licence in LICENSE, MIT, and not WTFPL.
func TestFooterLicense(t *testing.T) {
@@ -44,7 +28,14 @@ func TestFooterLicense(t *testing.T) {
t.Run(name, func(t *testing.T) {
t.Parallel()
_, footer, found := strings.Cut(renderPage(t, tmpl), "<footer")
var page bytes.Buffer
err := tmpl.Execute(&page, struct{ CSPNonce string }{})
if err != nil {
t.Fatalf("rendering the page: %v", err)
}
_, footer, found := strings.Cut(page.String(), "<footer")
if !found {
t.Fatal("the page has no footer")
}
@@ -61,121 +52,3 @@ func TestFooterLicense(t *testing.T) {
})
}
}
// TestFreshnessIndicator renders the home and status pages and expects each to
// have, right after the status element, the element where the page's script
// shows its spinner or "stale".
func TestFreshnessIndicator(t *testing.T) {
t.Parallel()
pages := map[string]struct {
tmpl *template.Template
status string
indicator string
}{
"index": {
tmpl: templates.IndexTemplate(),
status: `<span id="stat-status">-</span>`,
indicator: `<span id="freshness"></span>`,
},
"status": {
tmpl: templates.StatusTemplate(),
status: `<span class="metric-value" id="connected">-</span>`,
indicator: `<span class="metric-value" id="freshness"></span>`,
},
}
for name, page := range pages {
t.Run(name, func(t *testing.T) {
t.Parallel()
_, afterStatus, found := strings.Cut(renderPage(t, page.tmpl), page.status)
if !found {
t.Fatalf("the page has no status element %s", page.status)
}
if !strings.HasPrefix(strings.TrimSpace(afterStatus), page.indicator) {
t.Errorf("the status element is not followed by %s", page.indicator)
}
})
}
}
// TestASDetailPrefixListsFold renders the AS page with one IPv4 and one IPv6
// prefix. It expects each prefix list inside its own <details> element that
// starts open and whose <summary> holds the list's heading, so that a click on
// the heading folds the list.
func TestASDetailPrefixListsFold(t *testing.T) {
t.Parallel()
const (
asn = 64500
prefixV4 = "192.0.2.0/24"
prefixV6 = "2001:db8::/32"
)
ipv4 := []database.LiveRoute{{Prefix: prefixV4}}
ipv6 := []database.LiveRoute{{Prefix: prefixV6}}
// The fields the AS page handler gives the template.
data := struct {
ASN *database.ASN
IPv4Prefixes []database.LiveRoute
IPv6Prefixes []database.LiveRoute
TotalCount int
IPv4Count int
IPv6Count int
Peers []database.ASPeer
PeerCount int
CSPNonce string
}{
ASN: &database.ASN{ASN: asn},
IPv4Prefixes: ipv4,
IPv6Prefixes: ipv6,
TotalCount: len(ipv4) + len(ipv6),
IPv4Count: len(ipv4),
IPv6Count: len(ipv6),
}
var rendered bytes.Buffer
err := templates.ASDetailTemplate().Execute(&rendered, data)
if err != nil {
t.Fatalf("rendering the page: %v", err)
}
page := rendered.String()
lists := []struct{ heading, prefix string }{
{"IPv4 Prefixes", prefixV4},
{"IPv6 Prefixes", prefixV6},
}
for _, list := range lists {
before, _, found := strings.Cut(page, list.prefix)
if !found {
t.Fatalf("the page does not list %s", list.prefix)
}
// The list's section starts at the last <details before its prefix.
start := strings.LastIndex(before, "<details")
if start < 0 || strings.Contains(before[start:], "</details>") {
t.Errorf("%s: the list is not inside a <details> element", list.heading)
continue
}
tag, section, _ := strings.Cut(before[start:], ">")
if !strings.Contains(tag, " open") {
t.Errorf("%s: the <details> element does not start open: %s>",
list.heading, tag)
}
_, summary, _ := strings.Cut(section, "<summary")
summary, _, _ = strings.Cut(summary, "</summary>")
if !strings.Contains(summary, list.heading) {
t.Errorf("%s: the <summary> does not hold the heading", list.heading)
}
}
}
internal/whois/whois.go
+33 -284
View File
@@ -7,15 +7,10 @@ import (
"errors"
"fmt"
"io"
"log/slog"
"net"
"regexp"
"strings"
"sync"
"time"
"unicode/utf8"
"golang.org/x/text/encoding/charmap"
)
// Timeout constants for WHOIS queries.
@@ -31,18 +26,10 @@ const maxAnswerSize = 1 << 20
// Parsing constants.
const (
keyValueParts = 2 // Expected parts when splitting "key: value"
lacnicDateFormatLen = 8 // Length of YYYYMMDD date format
)
// The classes of the two objects that describe an AS in an answer from
// RIPE, APNIC or AFRINIC, and the key of the aut-num object's line that
// gives the ID of the AS's organisation object.
const (
autNumClass = "aut-num"
organisationClass = "organisation"
orgKey = "org"
)
// WHOIS server addresses.
const (
whoisServerIANA = "whois.iana.org:43"
@@ -51,32 +38,8 @@ const (
whoisServerAPNIC = "whois.apnic.net:43"
whoisServerLACNIC = "whois.lacnic.net:43"
whoisServerAFRINIC = "whois.afrinic.net:43"
// NIC.br registers Brazil's ASes under LACNIC.
whoisServerNICbr = "whois.registro.br:43"
)
// nicbrHeader starts the copyright notice at the top of NIC.br's WHOIS
// answers. LACNIC's answer for an AS that NIC.br registers is NIC.br's, and
// carries the same notice:
//
// % Copyright (c) Nic.br - Use of this data is governed by the Use and
// % Privacy Policy at https://registro.br/upp . Distribution,
const nicbrHeader = "% Copyright (c) Nic.br"
// nicbrLimitReached is in the line that marks NIC.br's reduced answer to a
// client over its query limit. NIC.br publishes no limit. Its Use and
// Privacy Policy, https://registro.br/upp, says only that it restricts
// queries to stop the data being taken in bulk.
const nicbrLimitReached = "Query rate limit exceeded"
// NICbrSkipTime is how long the client stops asking NIC.br after NIC.br
// answers that the client is over its query limit. NIC.br publishes
// neither the limit nor how long it lasts. An hour outlasts a limit
// counted per minute or per hour. The ASes looked up meanwhile get
// NIC.br's fields when they are next refreshed.
const NICbrSkipTime = time.Hour
// RIR identifiers.
const (
RIRARIN = "ARIN"
@@ -86,31 +49,6 @@ const (
RIRAFRNIC = "AFRINIC"
)
// noPersonalFlag keeps person and role objects out of an answer. The
// organisation object stays in it. RIPE counts the person and role objects
// it returns against a daily limit per IP address. Role objects for abuse
// contacts do not count. RIPE refuses an address for good that passes the
// limit too often:
// https://docs.db.ripe.net/Types-of-Queries/Access-Control-for-Queries/
//
// RIPE documents the flag here:
// https://docs.db.ripe.net/Types-of-Queries/Referenced-Objects-in-Query-Response/
// APNIC and AFRINIC list it in their answers to the query "help". AFRINIC
// sends readers to that answer for its full list of flags:
// https://afrinic.net/whois-getting-started
const noPersonalFlag = "--no-personal"
// The error lines with which RIPE's database software refuses a query:
// access denied to the client's IP address, and that address's limit
// reached. APNIC and AFRINIC run the same software. The lines are listed
// under "Access Errors" on this page, whose URL is split over two lines:
// https://docs.db.ripe.net/Appendices/
// Appendix-C--RIPE-Database-Query-Server-Response-Codes-and-Messages/
const (
accessDeniedError = "%ERROR:201:"
accessLimitError = "%ERROR:202:"
)
// Errors LookupASN returns when a WHOIS answer has nothing to store.
var (
// ErrNoReferral is returned when IANA refers the AS to no RIR, as it
@@ -120,7 +58,6 @@ var (
errNoASBlock = errors.New("IANA answered with neither a referral " +
"nor an AS block")
errNoFields = errors.New("RIR answered with no WHOIS fields")
errRefused = errors.New("RIR refused the query")
errUnknownServer = errors.New("IANA refers the AS to a server no RIR runs")
errAnswerTooLarge = errors.New("WHOIS answer reached the size cap")
)
@@ -149,42 +86,21 @@ type Client struct {
// host:port. NewClient sets a net.Dialer's; tests replace it to reach
// local servers instead.
DialContext func(ctx context.Context, network, address string) (net.Conn, error)
// Now returns the current time. NewClient sets time.Now; tests replace
// it to move the clock.
Now func() time.Time
logger *slog.Logger
// mu guards nicbrSkipUntil.
mu sync.Mutex
// nicbrSkipUntil is when the client next asks NIC.br, after NIC.br
// answered that the client is over its query limit.
nicbrSkipUntil time.Time
}
// NewClient creates a new WHOIS client that logs to logger.
func NewClient(logger *slog.Logger) *Client {
// NewClient creates a new WHOIS client.
func NewClient() *Client {
dialer := &net.Dialer{Timeout: dialTimeout}
return &Client{
DialContext: dialer.DialContext,
Now: time.Now,
logger: logger,
}
return &Client{DialContext: dialer.DialContext}
}
// LookupASN asks IANA which RIR holds the given ASN, asks that RIR, and
// returns what the RIR's answer says. When LACNIC's answer is NIC.br's, it
// returns what NIC.br's own server says instead, unless that query fails or
// is skipped, as askNICbr says.
//
// It returns ErrNoReferral when IANA refers the AS to no RIR, and another
// error when the query to IANA or to the RIR fails, when the RIR refuses
// it, when IANA's answer has neither a referral nor the AS's block, when
// IANA refers the AS to a server none of the five RIRs run, or when the
// answer carries none of ASNInfo's fields.
// returns what the RIR's answer says. It returns ErrNoReferral when IANA
// refers the AS to no RIR, and another error when a query fails, when IANA's
// answer has neither a referral nor the AS's block, when IANA refers the AS
// to a server none of the five RIRs run, or when the RIR's answer carries
// none of ASNInfo's fields, as a refusal of service does.
func (c *Client) LookupASN(ctx context.Context, asn int) (*ASNInfo, error) {
// Query IANA first to find the authoritative RIR
query := fmt.Sprintf("AS%d", asn)
@@ -211,82 +127,23 @@ func (c *Client) LookupASN(ctx context.Context, asn int) (*ASNInfo, error) {
}
// Query the authoritative RIR
rirResp, err := c.queryRIR(ctx, whoisServer, rirQuery(rir, query))
rirResp, err := c.query(ctx, whoisServer, query)
if err != nil {
return nil, fmt.Errorf("RIR query failed: %w", err)
}
// The fields never come from IANA's answer, which describes the block
// the AS is in.
// The fields come from the RIR's answer alone: IANA's describes the
// block the AS is in.
info := c.parseResponse(asn, rir, rirResp)
if rir == RIRLACNIC && strings.Contains(rirResp, nicbrHeader) {
info = c.askNICbr(ctx, query, info)
}
if info.isEmpty() {
return nil, fmt.Errorf("%s from %s: %w", query, whoisServer, errNoFields)
}
info.RawResponse = ianaResp + "\n" + info.RawResponse
info.RawResponse = ianaResp + "\n" + rirResp
return info, nil
}
// askNICbr asks NIC.br's server about an AS whose answer from LACNIC is
// NIC.br's, and returns what NIC.br's answer says. LACNIC replaces each
// accented letter in such an answer with U+FFFD. NIC.br's own server keeps
// them, in Latin-1.
//
// askNICbr returns lacnicInfo, what LACNIC's answer says, and logs a
// warning when the query fails, when NIC.br sends the reduced answer for a
// client over its limit, or when NIC.br's answer carries none of ASNInfo's
// fields. After the reduced answer, it returns lacnicInfo without asking
// NIC.br until NICbrSkipTime has passed.
func (c *Client) askNICbr(
ctx context.Context, asQuery string, lacnicInfo *ASNInfo,
) *ASNInfo {
asn := lacnicInfo.ASN
c.mu.Lock()
skipping := c.Now().Before(c.nicbrSkipUntil)
c.mu.Unlock()
if skipping {
return lacnicInfo
}
answer, err := c.query(ctx, whoisServerNICbr, asQuery)
if err != nil {
c.logger.Warn("NIC.br query failed, keeping LACNIC's fields",
"asn", asn, "error", err)
return lacnicInfo
}
if strings.Contains(answer, nicbrLimitReached) {
c.mu.Lock()
c.nicbrSkipUntil = c.Now().Add(NICbrSkipTime)
c.mu.Unlock()
c.logger.Warn("NIC.br answered that the client is over its query "+
"limit, keeping LACNIC's fields and skipping NIC.br",
"asn", asn, "skip_for", NICbrSkipTime)
return lacnicInfo
}
info := c.parseResponse(asn, RIRLACNIC, answer)
if info.isEmpty() {
c.logger.Warn("NIC.br answered with no WHOIS fields, "+
"keeping LACNIC's fields", "asn", asn)
return lacnicInfo
}
return info
}
// isEmpty reports whether info holds none of the fields a WHOIS answer
// fills, as when the answer is only comment and error lines.
func (info *ASNInfo) isEmpty() bool {
@@ -297,41 +154,8 @@ func (info *ASNInfo) isEmpty() bool {
info.RegDate == nil && info.LastMod == nil
}
// rirQuery returns the query sent to rir's server for asQuery, an AS
// number such as "AS3333". RIPE, APNIC and AFRINIC are asked to keep person
// and role objects out of the answer.
func rirQuery(rir, asQuery string) string {
switch rir {
case RIRRIPE, RIRAPNIC, RIRAFRNIC:
return noPersonalFlag + " " + asQuery
default:
return asQuery
}
}
// queryRIR sends query to an RIR's server and returns the answer. An answer
// holding an error line with which the server refuses the query is an
// error, and the error holds that line.
func (c *Client) queryRIR(
ctx context.Context, server, query string,
) (string, error) {
answer, err := c.query(ctx, server, query)
if err != nil {
return "", err
}
for line := range strings.SplitSeq(answer, "\n") {
if strings.HasPrefix(line, accessDeniedError) ||
strings.HasPrefix(line, accessLimitError) {
return "", fmt.Errorf("%w: %s", errRefused, line)
}
}
return answer, nil
}
// query performs a raw WHOIS query to the specified server and returns the
// answer in UTF-8. An answer that reaches maxAnswerSize is an error.
// query performs a raw WHOIS query to the specified server. An answer that
// reaches maxAnswerSize is an error.
func (c *Client) query(ctx context.Context, server, query string) (string, error) {
conn, err := c.DialContext(ctx, "tcp", server)
if err != nil {
@@ -378,25 +202,7 @@ func (c *Client) query(ctx context.Context, server, query string) (string, error
return "", fmt.Errorf("read response: %w", errAnswerTooLarge)
}
return toUTF8(sb.String())
}
// toUTF8 returns a WHOIS answer in UTF-8. An answer that is valid UTF-8 is
// returned as it is. Any other answer is decoded as Latin-1, which RIPE,
// APNIC and NIC.br send. LACNIC, ARIN and AFRINIC send UTF-8. RIPE's
// database working group lists the five RIRs here:
// https://www.ripe.net/ripe/mail/archives/db-wg/2024-May/008075.html
func toUTF8(answer string) (string, error) {
if utf8.ValidString(answer) {
return answer, nil
}
decoded, err := charmap.ISO8859_1.NewDecoder().String(answer)
if err != nil {
return "", fmt.Errorf("decode Latin-1 answer: %w", err)
}
return decoded, nil
return sb.String(), nil
}
// parseIANAReferral extracts the RIR and WHOIS server from an IANA response.
@@ -432,9 +238,7 @@ func (c *Client) parseIANAReferral(response string) (string, string, error) {
return "", "", nil
}
// parseResponse extracts ASN information from a WHOIS response. Of an
// answer from RIPE, APNIC or AFRINIC it reads only the lines asObjectLines
// returns. Of an answer from ARIN or LACNIC it reads every line.
// parseResponse extracts ASN information from a WHOIS response.
func (c *Client) parseResponse(asn int, rir, response string) *ASNInfo {
info := &ASNInfo{
ASN: asn,
@@ -443,15 +247,26 @@ func (c *Client) parseResponse(asn int, rir, response string) *ASNInfo {
}
lines := strings.Split(response, "\n")
if rir == RIRRIPE || rir == RIRAPNIC || rir == RIRAFRNIC {
lines = asObjectLines(lines)
}
var addressLines []string
for _, line := range lines {
key, value, ok := splitField(line)
if !ok || value == "" {
line = strings.TrimSpace(line)
if line == "" || strings.HasPrefix(line, "%") ||
strings.HasPrefix(line, "#") {
continue
}
// Split on first colon
parts := strings.SplitN(line, ":", keyValueParts)
if len(parts) != keyValueParts {
continue
}
key := strings.TrimSpace(strings.ToLower(parts[0]))
value := strings.TrimSpace(parts[1])
if value == "" {
continue
}
@@ -478,72 +293,6 @@ func (c *Client) parseResponse(asn int, rir, response string) *ASNInfo {
return info
}
// splitField splits a "key: value" line of a WHOIS answer into its key, in
// lower case, and its value. The third result is false for a line that
// holds no field: a blank line, a comment line or a line without a colon.
func splitField(line string) (string, string, bool) {
line = strings.TrimSpace(line)
if strings.HasPrefix(line, "%") || strings.HasPrefix(line, "#") {
return "", "", false
}
key, value, ok := strings.Cut(line, ":")
return strings.ToLower(strings.TrimSpace(key)), strings.TrimSpace(value), ok
}
// asObjectLines returns the lines of the aut-num object in an answer from
// RIPE, APNIC or AFRINIC, followed by the lines of the organisation object
// whose ID the aut-num object gives on its "org:" line. The answer's other
// objects are left out. RIPE and APNIC start their answer with the
// as-block object of the block the AS is in, whose dates are not the AS's.
//
// These answers are in RPSL. A blank line ends an object. An object's first
// line has the object's class as its key and the object's ID as its value,
// as in "organisation: ORG-APNI1-AP".
func asObjectLines(lines []string) []string {
var autNum []string
// The lines of each organisation object, by its ID.
organisations := make(map[string][]string)
// The class and ID of the object the line is in, "" between objects.
var class, id string
// The ID of the organisation object the aut-num object refers to.
var orgID string
for _, line := range lines {
if strings.TrimSpace(line) == "" {
class = ""
continue
}
key, value, ok := splitField(line)
if !ok {
continue
}
if class == "" {
class, id = key, value
}
switch class {
case autNumClass:
autNum = append(autNum, line)
if key == orgKey {
orgID = value
}
case organisationClass:
organisations[id] = append(organisations[id], line)
}
}
return append(autNum, organisations[orgID]...)
}
// setResponseField stores the value of one "key: value" line of a WHOIS
// response in info, keeping the first value seen for each field.
func (c *Client) setResponseField(info *ASNInfo, key, value string) {
internal/whois/whois_test.go
+28 -663
View File
@@ -2,21 +2,15 @@ package whois_test
import (
"bufio"
"bytes"
"context"
"encoding/json"
"errors"
"io"
"log/slog"
"maps"
"net"
"slices"
"strings"
"sync"
"syscall"
"testing"
"time"
"unicode/utf8"
"git.eeqj.de/sneak/routewatch/internal/whois"
)
@@ -27,20 +21,8 @@ const (
ripeServer = "whois.ripe.net:43"
)
// testASN is the AS the tests look up, and asQuery the query for it.
const (
testASN = 3333
asQuery = "AS3333"
)
// The fake servers' answers are keyed by a server's address, a space and a
// query. These are the keys of the queries LookupASN sends for testASN:
// IANA's, then RIPE's, which asks RIPE to leave out person and role
// objects.
const (
ianaQuery = ianaServer + " " + asQuery
ripeQuery = ripeServer + " --no-personal " + asQuery
)
// testASN is the AS the tests look up.
const testASN = 3333
// ianaReferral is IANA's answer for an AS in a block it gave to RIPE.
const ianaReferral = `% IANA WHOIS server
@@ -54,163 +36,27 @@ changed: 2002-09-01
source: IANA
`
// lacnicReferral is IANA's answer for an AS in a block it gave to LACNIC,
// shortened to the referral.
const lacnicReferral = "refer: whois.lacnic.net\n"
// ripeRecord is RIPE's answer for the AS when asked to leave out person and
// role objects. It starts with the as-block object of the block the AS is
// in. Then come the abuse address as a comment, the AS's object and the
// organisation object it refers to.
// ripeRecord is RIPE's answer for the AS.
const ripeRecord = `% This is the RIPE Database query service.
% The objects are in RPSL format.
% Note: --no-personal means ALL personal data has been filtered from this output.
% Information related to 'AS3209 - AS3353'
as-block: AS3209 - AS3353
descr: RIPE NCC ASN block
remarks: These AS Numbers are assigned to network operators
remarks: in the RIPE NCC service region.
mnt-by: RIPE-NCC-HM-MNT
created: 2018-11-22T15:27:19Z
last-modified: 2018-11-22T15:27:19Z
source: RIPE
% Information related to 'AS3333'
% Abuse contact for 'AS3333' is 'abuse@ripe.net'
aut-num: AS3333
as-name: RIPE-NCC-AS
org: ORG-RIEN1-RIPE
admin-c: BRD-RIPE
tech-c: OPS4-RIPE
abuse-c: OPS4-RIPE
mnt-by: RIPE-NCC-MNT
created: 2002-09-17T13:08:32Z
last-modified: 2024-01-09T08:55:41Z
source: RIPE
organisation: ORG-RIEN1-RIPE
org-name: Reseaux IP Europeens Network Coordination Centre (RIPE NCC)
country: NL
org-type: RIR
address: P.O. Box 10096
address: 1001EB Amsterdam
abuse-c: OPS4-RIPE
created: 2012-03-09T13:08:29Z
last-modified: 2024-03-26T10:55:38Z
source: RIPE
`
// The error lines with which RIPE refuses a query: access denied to the
// host, and the host's limit on returned objects reached.
const (
ripeAccessDenied = "%ERROR:201: access denied for 192.0.2.1"
ripeLimitReached = "%ERROR:202: access control limit reached"
)
// ripeRefusal is RIPE's answer to a host it no longer serves.
const ripeRefusal = `% This is the RIPE Database query service.
% The objects are in RPSL format.
%
` + ripeAccessDenied + `
%ERROR:201: access denied for 192.0.2.1
%
% Sorry, access from your host has been permanently
% denied because of a repeated excessive querying.
`
// apnicAfrinicRecord is the answer of APNIC or AFRINIC for the AS when asked
// to leave out person and role objects. Like RIPE's, it holds the abuse
// address as a comment, the AS's object and the organisation object.
const apnicAfrinicRecord = `% Abuse contact for 'AS3333' is 'abuse@example.net'
aut-num: AS3333
as-name: EXAMPLE-AS
org: ORG-EXAMPLE1
admin-c: EX1-TEST
last-modified: 2026-08-26T00:40:00Z
organisation: ORG-EXAMPLE1
org-name: Example Networks
country: ZA
address: 1 Example Road
`
// apnicServer is APNIC's WHOIS server, and apnicASN an AS that APNIC holds.
const (
apnicServer = "whois.apnic.net:43"
apnicASN = 4608
)
// The keys of the queries LookupASN sends for apnicASN: IANA's, then
// APNIC's, which asks APNIC to leave out person and role objects.
const (
apnicIANAQuery = ianaServer + " AS4608"
apnicQuery = apnicServer + " --no-personal AS4608"
)
// apnicRecord is APNIC's answer for apnicASN when asked to leave out person
// and role objects, shortened. It starts with the as-block object of the
// block the AS is in. Then come the abuse address as a comment, the AS's
// object, and the irt and organisation objects it refers to.
const apnicRecord = `% [whois.apnic.net]
% Whois data copyright terms http://www.apnic.net/db/dbcopyright.html
% Note: --no-personal means ALL personal data has been filtered from this output.
% Information related to 'AS4608 - AS4865'
as-block: AS4608 - AS4865
descr: APNIC ASN block
remarks: These AS numbers are further assigned by APNIC
remarks: to APNIC members and end-users in the APNIC region
mnt-by: APNIC-HM
mnt-lower: APNIC-HM
last-modified: 2019-05-20T03:52:26Z
source: APNIC
% Information related to 'AS4608'
% Abuse contact for 'AS4608' is 'abuse@apnic.net'
aut-num: AS4608
as-name: APNIC-SERVICES
descr: Asia Pacific Network Information Centre
country: AU
org: ORG-APNI1-AP
admin-c: AIC1-AP
abuse-c: AA1589-AP
mnt-by: APNIC-HM
mnt-irt: IRT-APNIC-IS-AP
last-modified: 2026-08-26T00:40:00Z
source: APNIC
irt: IRT-APNIC-IS-AP
address: South Brisbane, Australia
e-mail: helpdesk@apnic.net
abuse-mailbox: abuse@apnic.net
mnt-by: MAINT-APNIC-IS-AP
last-modified: 2025-11-18T00:27:08Z
source: APNIC
organisation: ORG-APNI1-AP
org-name: Asia Pacific Network Information Centre
org-type: LIR
country: AU
address: PO Box 3646
address: South Brisbane, QLD 4101,
address: Australia
e-mail: noc@apnic.net
mnt-by: APNIC-HM
last-modified: 2026-08-24T12:58:43Z
source: APNIC
% This query was served by the APNIC Whois Service version 1.88.48 (WHOIS-UK2)
`
// ianaPrivateUse is IANA's answer for a private-use AS: its block and no
// referral.
const ianaPrivateUse = `% IANA WHOIS server
@@ -234,29 +80,20 @@ const fillerLine = "remarks: and more\n"
const endlessAnswerSize = 4 << 20
// fakeServers returns a DialContext for a whois.Client that connects to a
// fake server at each address answers names. Each key of answers is a
// server's address, a space and a query, and its value is that server's
// answer to that query. A server reads one query and sends its answer, or
// nothing for a query it has no answer for. Connecting to any other address
// is refused.
// fake server for each address in answers, which reads the query and sends
// answers[address]. Connecting to any other address is refused.
func fakeServers(
t *testing.T, answers map[string]string,
) func(context.Context, string, string) (net.Conn, error) {
t.Helper()
addresses := make(map[string]bool)
for key := range answers {
address, _, _ := strings.Cut(key, " ")
addresses[address] = true
}
var servers sync.WaitGroup
t.Cleanup(servers.Wait)
return func(_ context.Context, _, address string) (net.Conn, error) {
if !addresses[address] {
answer, ok := answers[address]
if !ok {
return nil, syscall.ECONNREFUSED
}
@@ -265,8 +102,7 @@ func fakeServers(
servers.Go(func() {
defer func() { _ = server.Close() }()
query, _ := bufio.NewReader(server).ReadString('\n')
answer := answers[address+" "+strings.TrimSpace(query)]
_, _ = bufio.NewReader(server).ReadString('\n')
_, _ = io.WriteString(server, answer)
})
@@ -280,48 +116,31 @@ func lookUp(
) (*whois.ASNInfo, error) {
t.Helper()
client := whois.NewClient(slog.New(slog.DiscardHandler))
client := whois.NewClient()
client.DialContext = fakeServers(t, answers)
return client.LookupASN(t.Context(), testASN)
}
// TestLookupASNReadsTheRIRAnswer looks up an AS IANA refers to RIPE. RIPE
// answers only a query that leaves out person and role objects, and its
// answer to that still fills the AS's fields: the abuse address from the
// comment line, and the organisation's name and country from its object.
// The dates are the AS's. IANA's "changed:" line dates the AS's block, and
// so do the dates of the as-block object at the start of RIPE's answer.
// TestLookupASNReadsTheRIRAnswer looks up an AS IANA refers to RIPE. The
// fields come from RIPE's answer, the last-modified date included: IANA's
// "changed:" line dates its block, not the AS.
func TestLookupASNReadsTheRIRAnswer(t *testing.T) {
t.Parallel()
info, err := lookUp(t, map[string]string{
ianaQuery: ianaReferral,
ripeQuery: ripeRecord,
ianaServer: ianaReferral,
ripeServer: ripeRecord,
})
if err != nil {
t.Fatalf("LookupASN: %v", err)
}
got := []string{
info.ASName, info.OrgID, info.OrgName, info.CountryCode,
info.AbuseEmail, info.RIR,
}
want := []string{
"RIPE-NCC-AS", "ORG-RIEN1-RIPE",
"Reseaux IP Europeens Network Coordination Centre (RIPE NCC)", "NL",
"abuse@ripe.net", whois.RIRRIPE,
}
if !slices.Equal(got, want) {
t.Errorf("LookupASN = AS name, org ID, org name, country, abuse "+
"address, RIR %q; want %q", got, want)
}
wantRegDate := time.Date(2002, 9, 17, 13, 8, 32, 0, time.UTC)
if info.RegDate == nil || !info.RegDate.Equal(wantRegDate) {
t.Errorf("LookupASN registered = %v, want %v",
info.RegDate, wantRegDate)
if info.ASName != "RIPE-NCC-AS" || info.OrgID != "ORG-RIEN1-RIPE" ||
info.RIR != whois.RIRRIPE {
t.Errorf("LookupASN = AS name %q, org %q, RIR %q; "+
"want RIPE-NCC-AS, ORG-RIEN1-RIPE, RIPE",
info.ASName, info.OrgID, info.RIR)
}
wantLastMod := time.Date(2024, 1, 9, 8, 55, 41, 0, time.UTC)
@@ -331,460 +150,6 @@ func TestLookupASNReadsTheRIRAnswer(t *testing.T) {
}
}
// TestLookupASNReadsOnlyTheASAndItsOrganisation looks up an AS at APNIC,
// whose answer starts with the as-block object of the block the AS is in.
// The last-modified date is the aut-num object's, not the block's. The
// address is the organisation object's alone, without the irt object's.
func TestLookupASNReadsOnlyTheASAndItsOrganisation(t *testing.T) {
t.Parallel()
client := whois.NewClient(slog.New(slog.DiscardHandler))
client.DialContext = fakeServers(t, map[string]string{
apnicIANAQuery: "refer: whois.apnic.net\n",
apnicQuery: apnicRecord,
})
info, err := client.LookupASN(t.Context(), apnicASN)
if err != nil {
t.Fatalf("LookupASN: %v", err)
}
wantLastMod := time.Date(2026, 8, 26, 0, 40, 0, 0, time.UTC)
if info.LastMod == nil || !info.LastMod.Equal(wantLastMod) {
t.Errorf("LookupASN last modified = %v, want %v",
info.LastMod, wantLastMod)
}
got := []string{
info.ASName, info.OrgID, info.OrgName, info.CountryCode,
info.Address, info.AbuseEmail,
}
want := []string{
"APNIC-SERVICES", "ORG-APNI1-AP",
"Asia Pacific Network Information Centre", "AU",
"PO Box 3646\nSouth Brisbane, QLD 4101,\nAustralia", "abuse@apnic.net",
}
if !slices.Equal(got, want) {
t.Errorf("LookupASN = AS name, org ID, org name, country, address, "+
"abuse address %q; want %q", got, want)
}
}
// TestLookupASNAsksAPNICAndAFRINICToLeaveOutPersonalData looks up an AS at
// APNIC and at AFRINIC. Each answers only a query that leaves out person
// and role objects, as RIPE does. That one answer still holds the
// organisation object, so the organisation's name and country are filled.
func TestLookupASNAsksAPNICAndAFRINICToLeaveOutPersonalData(t *testing.T) {
t.Parallel()
for _, server := range []string{"whois.apnic.net", "whois.afrinic.net"} {
t.Run(server, func(t *testing.T) {
t.Parallel()
address := server + ":43"
info, err := lookUp(t, map[string]string{
ianaQuery: "refer: " + server + "\n",
address + " --no-personal " + asQuery: apnicAfrinicRecord,
})
if err != nil {
t.Fatalf("LookupASN: %v", err)
}
got := []string{
info.ASName, info.OrgName, info.CountryCode, info.AbuseEmail,
}
want := []string{
"EXAMPLE-AS", "Example Networks", "ZA", "abuse@example.net",
}
if !slices.Equal(got, want) {
t.Errorf("LookupASN = AS name, org name, country, abuse "+
"address %q; want %q", got, want)
}
})
}
}
// The name of an organisation with accented letters, in Latin-1, one byte for
// each letter, and in UTF-8.
const (
latin1OrgName = "Fundaci\xf3n Ra\xedces de Espa\xf1a"
utf8OrgName = "Fundación Raíces de España"
)
// TestLookupASNStoresAnswersInUTF8 looks up an AS at RIPE, which answers in
// Latin-1, and at LACNIC, which answers in UTF-8. Both answers give the
// organisation's name with its accented letters in UTF-8, in the parsed field
// and in the stored answer. Before the fix the Latin-1 answer was kept as
// received, which is not valid UTF-8.
func TestLookupASNStoresAnswersInUTF8(t *testing.T) {
t.Parallel()
for name, answers := range map[string]map[string]string{
"RIPE in Latin-1": {
ianaQuery: ianaReferral,
ripeQuery: "aut-num: AS3333\n" +
"org-name: " + latin1OrgName + "\n",
},
"LACNIC in UTF-8": {
ianaQuery: lacnicReferral,
"whois.lacnic.net:43 " + asQuery: "aut-num: AS3333\n" +
"owner: " + utf8OrgName + "\n",
},
} {
t.Run(name, func(t *testing.T) {
t.Parallel()
info, err := lookUp(t, answers)
if err != nil {
t.Fatalf("LookupASN: %v", err)
}
if info.OrgName != utf8OrgName {
t.Errorf("LookupASN org name = %q, want %q",
info.OrgName, utf8OrgName)
}
if !utf8.ValidString(info.RawResponse) ||
!strings.Contains(info.RawResponse, utf8OrgName) {
t.Errorf("LookupASN answer = %q, want UTF-8 holding %q",
info.RawResponse, utf8OrgName)
}
})
}
}
// lacnicServer is LACNIC's WHOIS server and nicbrServer NIC.br's, which
// registers Brazil's ASes under LACNIC. brazilASN is one of those ASes.
const (
lacnicServer = "whois.lacnic.net:43"
nicbrServer = "whois.registro.br:43"
brazilASN = 52751
)
// The keys of the queries LookupASN sends for brazilASN: IANA's, LACNIC's,
// then NIC.br's.
const (
brazilIANAQuery = ianaServer + " AS52751"
brazilLACNICQuery = lacnicServer + " AS52751"
brazilNICbrQuery = nicbrServer + " AS52751"
)
// The name of brazilASN's organisation in three forms: as LACNIC sends it,
// with each accented letter replaced by U+FFFD; as NIC.br sends it, in
// Latin-1; and in UTF-8.
const (
lacnicBrazilOrgName = "Brasrede Telecomunica\U0000FFFD\U0000FFFDes LTDA"
nicbrBrazilOrgName = "Brasrede Telecomunica\xe7\xf5es LTDA"
brazilOrgName = "Brasrede Telecomunicações LTDA"
)
// nicbrCopyright is the copyright notice at the top of NIC.br's answers.
const nicbrCopyright = "% Copyright (c) Nic.br - " +
"Use of this data is governed by the Use and\n" +
"% Privacy Policy at https://registro.br/upp . Distribution,\n" +
"% commercialization, reproduction, and use for advertising or similar\n" +
"% purposes are expressly prohibited.\n"
// lacnicBrazilRecord is LACNIC's answer for brazilASN, recorded on
// 2026-10-26. It is NIC.br's answer, which LACNIC passes on after a line
// of its own and a space. It is shortened. The client's address is a
// documentation address. The contact's name, the contact's object and the
// footer are left out.
const lacnicBrazilRecord = "% IP Client: 192.0.2.1\n" +
" " + nicbrCopyright + `% 2026-10-26T23:12:22-03:00 - 192.0.2.1
aut-num: AS52751
owner: ` + lacnicBrazilOrgName + `
owner-c: AVL217
routing-c: AVL217
abuse-c: AVL217
created: 20120829
changed: 20130306
inetnum: 177.38.152.0/21
inetnum: 2804:b34::/32
inetnum: 191.243.4.0/22
inetnum: 131.72.248.0/22
inetnum: 138.94.228.0/22
inetnum: 170.233.8.0/22
inetnum: 170.246.4.0/22
`
// nicbrRecord is NIC.br's answer for brazilASN, in Latin-1, recorded on
// 2026-10-26 and shortened as lacnicBrazilRecord is.
const nicbrRecord = nicbrCopyright + `% 2026-10-26T23:14:41-03:00 - 192.0.2.1
aut-num: AS52751
owner: ` + nicbrBrazilOrgName + `
owner-c: AVL217
routing-c: AVL217
abuse-c: AVL217
created: 20120829
changed: 20130306
inetnum: 177.38.152.0/21
inetnum: 2804:b34::/32
inetnum: 191.243.4.0/22
inetnum: 131.72.248.0/22
inetnum: 138.94.228.0/22
inetnum: 170.233.8.0/22
inetnum: 170.246.4.0/22
`
// nicbrOverLimit stands for NIC.br's answer for brazilASN to a client over
// its query limit, and is not recorded. Such an answer leaves information
// out, here the owner, and says so in its first line.
const nicbrOverLimit = `% Query rate limit exceeded. Reduced information.
aut-num: AS52751
created: 20120829
changed: 20130306
`
// newNICbrClient returns a client whose fake servers send answers and
// whose logger writes JSON lines to logs. It adds one to *nicbrAsks each
// time it connects to NIC.br's server.
func newNICbrClient(
t *testing.T, answers map[string]string, logs io.Writer, nicbrAsks *int,
) *whois.Client {
t.Helper()
dial := fakeServers(t, answers)
client := whois.NewClient(slog.New(slog.NewJSONHandler(logs, nil)))
client.DialContext = func(
ctx context.Context, network, address string,
) (net.Conn, error) {
if address == nicbrServer {
*nicbrAsks++
}
return dial(ctx, network, address)
}
return client
}
// lookUpBrazilianAS looks brazilASN up with fake servers sending answers,
// and reports whether the client connected to NIC.br's server.
func lookUpBrazilianAS(
t *testing.T, answers map[string]string,
) (*whois.ASNInfo, bool, error) {
t.Helper()
nicbrAsks := 0
client := newNICbrClient(t, answers, io.Discard, &nicbrAsks)
info, err := client.LookupASN(t.Context(), brazilASN)
return info, nicbrAsks > 0, err
}
// TestLookupASNTakesABrazilianASFromNICbr looks up an AS that NIC.br
// registers. LACNIC's answer is NIC.br's with each accented letter replaced
// by U+FFFD, so the client also asks NIC.br's server, which answers in
// Latin-1. The organisation's name has its accented letters, in the parsed
// field and in the stored answer.
func TestLookupASNTakesABrazilianASFromNICbr(t *testing.T) {
t.Parallel()
info, _, err := lookUpBrazilianAS(t, map[string]string{
brazilIANAQuery: lacnicReferral,
brazilLACNICQuery: lacnicBrazilRecord,
brazilNICbrQuery: nicbrRecord,
})
if err != nil {
t.Fatalf("LookupASN: %v", err)
}
if info.OrgName != brazilOrgName || info.RIR != whois.RIRLACNIC {
t.Errorf("LookupASN org name, RIR = %q, %q; want %q, %q",
info.OrgName, info.RIR, brazilOrgName, whois.RIRLACNIC)
}
if !strings.Contains(info.RawResponse, brazilOrgName) {
t.Errorf("LookupASN answer = %q, want one holding %q",
info.RawResponse, brazilOrgName)
}
wantRegDate := time.Date(2012, 8, 29, 0, 0, 0, 0, time.UTC)
if info.RegDate == nil || !info.RegDate.Equal(wantRegDate) {
t.Errorf("LookupASN registered = %v, want %v",
info.RegDate, wantRegDate)
}
}
// TestLookupASNAsksNICbrOnlyForItsASes looks up an AS at LACNIC whose
// answer does not carry NIC.br's copyright notice, as for an AS that LACNIC
// registers itself. The client does not connect to NIC.br's server, which
// would answer with a record, and keeps LACNIC's fields.
func TestLookupASNAsksNICbrOnlyForItsASes(t *testing.T) {
t.Parallel()
info, askedNICbr, err := lookUpBrazilianAS(t, map[string]string{
brazilIANAQuery: lacnicReferral,
brazilLACNICQuery: "aut-num: AS52751\n" +
"owner: " + utf8OrgName + "\n",
brazilNICbrQuery: nicbrRecord,
})
if err != nil {
t.Fatalf("LookupASN: %v", err)
}
if askedNICbr {
t.Error("LookupASN asked NIC.br's server about an AS whose " +
"answer from LACNIC is not NIC.br's")
}
if info.OrgName != utf8OrgName {
t.Errorf("LookupASN org name = %q, want LACNIC's %q",
info.OrgName, utf8OrgName)
}
}
// TestLookupASNKeepsLACNICsAnswerWhenNICbrFails looks up an AS that NIC.br
// registers when NIC.br's server refuses the connection, sends nothing, or
// sends the reduced answer for a client over its query limit. The client
// asks NIC.br, and the lookup still succeeds with LACNIC's fields. The
// client logs one warning, which names the AS and the reason.
func TestLookupASNKeepsLACNICsAnswerWhenNICbrFails(t *testing.T) {
t.Parallel()
for name, test := range map[string]struct {
nicbrAnswer map[string]string
reason string
}{
"connection refused": {
nicbrAnswer: map[string]string{},
reason: syscall.ECONNREFUSED.Error(),
},
"no answer": {
nicbrAnswer: map[string]string{brazilNICbrQuery: ""},
reason: "no WHOIS fields",
},
"over the query limit": {
nicbrAnswer: map[string]string{brazilNICbrQuery: nicbrOverLimit},
reason: "over its query limit",
},
} {
t.Run(name, func(t *testing.T) {
t.Parallel()
answers := map[string]string{
brazilIANAQuery: lacnicReferral,
brazilLACNICQuery: lacnicBrazilRecord,
}
maps.Copy(answers, test.nicbrAnswer)
var logs bytes.Buffer
nicbrAsks := 0
client := newNICbrClient(t, answers, &logs, &nicbrAsks)
info, err := client.LookupASN(t.Context(), brazilASN)
if err != nil {
t.Fatalf("LookupASN: %v", err)
}
if nicbrAsks == 0 {
t.Error("LookupASN did not ask NIC.br's server")
}
if info.OrgName != lacnicBrazilOrgName {
t.Errorf("LookupASN org name = %q, want LACNIC's %q",
info.OrgName, lacnicBrazilOrgName)
}
var logged struct {
Level string `json:"level"`
ASN int `json:"asn"`
}
err = json.Unmarshal(logs.Bytes(), &logged)
if err != nil {
t.Fatalf("want one log line, got %q: %v", logs.String(), err)
}
if logged.Level != slog.LevelWarn.String() ||
logged.ASN != brazilASN ||
!strings.Contains(logs.String(), test.reason) {
t.Errorf("logged %q, want a warning for AS%d holding %q",
logs.String(), brazilASN, test.reason)
}
})
}
}
// TestLookupASNSkipsNICbrAfterItsLimit looks up an AS that NIC.br
// registers three times with one client, while NIC.br answers that the
// client is over its query limit. The first lookup asks NIC.br. The
// second, a second before NICbrSkipTime has passed, does not. The third,
// once NICbrSkipTime has passed, asks again. Each keeps LACNIC's fields.
func TestLookupASNSkipsNICbrAfterItsLimit(t *testing.T) {
t.Parallel()
nicbrAsks := 0
client := newNICbrClient(t, map[string]string{
brazilIANAQuery: lacnicReferral,
brazilLACNICQuery: lacnicBrazilRecord,
brazilNICbrQuery: nicbrOverLimit,
}, io.Discard, &nicbrAsks)
start := time.Date(2026, 10, 27, 0, 0, 0, 0, time.UTC)
now := start
client.Now = func() time.Time { return now }
for _, step := range []struct {
after time.Duration
wantAsks int
}{
{after: 0, wantAsks: 1},
{after: whois.NICbrSkipTime - time.Second, wantAsks: 1},
{after: whois.NICbrSkipTime, wantAsks: 2},
} {
now = start.Add(step.after)
info, err := client.LookupASN(t.Context(), brazilASN)
if err != nil {
t.Fatalf("LookupASN after %v: %v", step.after, err)
}
if nicbrAsks != step.wantAsks {
t.Errorf("after %v NIC.br's server was asked %d times, want %d",
step.after, nicbrAsks, step.wantAsks)
}
if info.OrgName != lacnicBrazilOrgName {
t.Errorf("LookupASN after %v org name = %q, want LACNIC's %q",
step.after, info.OrgName, lacnicBrazilOrgName)
}
}
}
// TestLookupASNReportsARefusal looks up an AS IANA refers to RIPE when RIPE
// refuses the query, with each of the error lines RIPE refuses with. The
// error holds RIPE's line, so the log says why the lookup failed.
func TestLookupASNReportsARefusal(t *testing.T) {
t.Parallel()
for _, line := range []string{ripeAccessDenied, ripeLimitReached} {
t.Run(line, func(t *testing.T) {
t.Parallel()
_, err := lookUp(t, map[string]string{
ianaQuery: ianaReferral,
ripeQuery: "% This is the RIPE Database query service.\n" +
line + "\n",
})
if err == nil || !strings.Contains(err.Error(), line) {
t.Errorf("LookupASN error = %v, want one holding %q",
err, line)
}
})
}
}
// TestLookupASNFailsWhenTheRIRGivesNoAnswer looks up an AS IANA refers to
// RIPE when RIPE's server refuses the connection, and when it answers only
// with an error. Neither is a lookup that found the AS.
@@ -792,8 +157,8 @@ func TestLookupASNFailsWhenTheRIRGivesNoAnswer(t *testing.T) {
t.Parallel()
for name, answers := range map[string]map[string]string{
"connection refused": {ianaQuery: ianaReferral},
"only an error": {ianaQuery: ianaReferral, ripeQuery: ripeRefusal},
"connection refused": {ianaServer: ianaReferral},
"only an error": {ianaServer: ianaReferral, ripeServer: ripeRefusal},
} {
t.Run(name, func(t *testing.T) {
t.Parallel()
@@ -812,7 +177,7 @@ func TestLookupASNFailsWhenTheRIRGivesNoAnswer(t *testing.T) {
func TestLookupASNReportsNoReferral(t *testing.T) {
t.Parallel()
_, err := lookUp(t, map[string]string{ianaQuery: ianaPrivateUse})
_, err := lookUp(t, map[string]string{ianaServer: ianaPrivateUse})
if !errors.Is(err, whois.ErrNoReferral) {
t.Errorf("LookupASN error = %v, want ErrNoReferral", err)
}
@@ -824,7 +189,7 @@ func TestLookupASNReportsNoReferral(t *testing.T) {
func TestLookupASNFailsWhenIANAAnswersNoBlock(t *testing.T) {
t.Parallel()
_, err := lookUp(t, map[string]string{ianaQuery: ianaNoObject})
_, err := lookUp(t, map[string]string{ianaServer: ianaNoObject})
if err == nil || errors.Is(err, whois.ErrNoReferral) {
t.Errorf("LookupASN error = %v, want one that is not ErrNoReferral",
err)
@@ -841,8 +206,8 @@ func TestLookupASNFailsWhenTheRIRAnswerDoesNotEnd(t *testing.T) {
strings.Repeat(fillerLine, endlessAnswerSize/len(fillerLine))
_, err := lookUp(t, map[string]string{
ianaQuery: ianaReferral,
ripeQuery: endless,
ianaServer: ianaReferral,
ripeServer: endless,
})
if err == nil || errors.Is(err, whois.ErrNoReferral) {
t.Errorf("LookupASN error = %v, want one that is not ErrNoReferral",
@@ -868,9 +233,9 @@ func TestLookupASNFailsOnAReferralToAnotherServer(t *testing.T) {
t.Parallel()
_, err := lookUp(t, map[string]string{
ianaQuery: "refer: " + refer + "\n\n" +
ianaServer: "refer: " + refer + "\n\n" +
"as-block: 3154-3353\n",
address + " " + asQuery: ripeRecord,
address: ripeRecord,
})
if err == nil || errors.Is(err, whois.ErrNoReferral) {
t.Errorf("LookupASN error = %v, want one that is not "+
pkg/asinfo/asdata.json.gz
BIN
View File
Binary file not shown.
pkg/asinfo/asinfo_test.go
+9 -13
View File
@@ -22,7 +22,7 @@ func TestGet(t *testing.T) {
name: "AS1 Level 3",
asn: 1,
wantOK: true,
wantDesc: "Level 3 Parent LLC",
wantDesc: "Level 3 Parent, LLC",
},
{
name: "AS15169 Google",
@@ -36,10 +36,10 @@ func TestGet(t *testing.T) {
wantOK: false,
},
{
name: "AS1251 with accented letters",
asn: 1251,
name: "AS0 IANA Reserved",
asn: 0,
wantOK: true,
wantDesc: "FUNDAÇÃO DE AMPARO À PESQUISA DO ESTADO SÃO PAULO",
wantDesc: "Internet Assigned Numbers Authority",
},
}
@@ -69,7 +69,7 @@ func TestGetDescription(t *testing.T) {
asn int
want string
}{
{1, "Level 3 Parent LLC"},
{1, "Level 3 Parent, LLC"},
{999999999, ""},
}
@@ -92,7 +92,8 @@ func TestGetHandle(t *testing.T) {
asn int
want string
}{
{1, "LVLT-1"},
{1, "LVLT"},
{0, "IANA-RSVD"},
{999999999, ""},
}
@@ -141,11 +142,6 @@ func TestSearch(t *testing.T) {
query: "University",
wantMin: 10,
},
{
name: "Search for São Paulo",
query: "São Paulo",
wantMin: 1,
},
{
name: "Non-existent org",
query: "XYZABC123456",
@@ -193,13 +189,13 @@ func TestSearchEmptyQueryMatchesNothing(t *testing.T) {
func TestSearchIgnoresCase(t *testing.T) {
t.Parallel()
const cloudflareASN = 13335 // handle CLOUDFLARENET, "Cloudflare Inc."
const cloudflareASN = 13335 // handle CLOUDFLARENET, "Cloudflare, Inc."
isCloudflare := func(info asinfo.Info) bool {
return info.ASN == cloudflareASN
}
for _, query := range []string{"cloudflarenet", "CLOUDFLARE INC."} {
for _, query := range []string{"cloudflarenet", "CLOUDFLARE, INC."} {
results, _ := asinfo.Search(query, asinfo.Total())
if !slices.ContainsFunc(results, isCloudflare) {
t.Errorf("Search(%q) did not find AS%d", query, cloudflareASN)
pkg/asinfo/doc.go
+1 -1
View File
@@ -2,7 +2,7 @@
Package asinfo provides information about Autonomous Systems (AS).
The package embeds a comprehensive database of AS numbers, handles, and descriptions
sourced from https://github.com/ipverse/as-metadata. The data is embedded at compile
sourced from https://github.com/ipverse/asn-info. The data is embedded at compile
time and provides fast, offline lookups.
Basic usage:
pkg/asinfo/example_test.go
+1 -1
View File
@@ -17,7 +17,7 @@ func ExampleGet() {
func ExampleGetDescription() {
desc := asinfo.GetDescription(13335)
fmt.Println(desc)
// Output: Cloudflare Inc.
// Output: Cloudflare, Inc.
}
func ExampleSearch() {
script/asupdate
-34
View File
@@ -1,34 +0,0 @@
#!/bin/sh
# script/asupdate: rebuild the AS list built into RouteWatch from
# as.csv with cmd/asinfo-gen, which fails when it finds no AS. The new
# list replaces the old one only when the generator succeeds. Otherwise
# the old list stays as it was and the script exits non-zero.
set -eu
ROOT="$(cd "$(dirname "$0")/.." && pwd -P)"
AS_DATA="pkg/asinfo/asdata.json.gz"
# Next to AS_DATA, so the final mv is a rename on one file system.
AS_DATA_TMP="$AS_DATA.tmp"
main() {
cd "$ROOT"
json="$(mktemp)"
trap 'rm -f "$json" "$AS_DATA_TMP"' EXIT
# A signal exits through the EXIT trap too.
trap 'exit 1' HUP INT TERM
echo "Updating AS info data..."
if ! go run ./cmd/asinfo-gen >"$json"; then
echo "asupdate: the generator failed; $AS_DATA is unchanged" >&2
exit 1
fi
# -n leaves the time out of the file, so an unchanged list gives the
# same bytes.
gzip -n -c <"$json" >"$AS_DATA_TMP"
mv "$AS_DATA_TMP" "$AS_DATA"
}
main "$@"