Compare commits
1
Commits
next
..
22a89685f9
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
22a89685f9 |
+2
-12
@@ -73,8 +73,7 @@
|
|||||||
|
|
||||||
# This repository's host-built artifacts: `make build` writes bin/, `go
|
# This repository's host-built artifacts: `make build` writes bin/, `go
|
||||||
# build` in the root the binaries, `make run` log.txt, `make asupdate` its
|
# build` in the root the binaries, `make run` log.txt, `make asupdate` its
|
||||||
# .tmp file; logs, test binaries, coverage profiles, source archives, and
|
# .tmp file; test binaries, coverage profiles and source archives.
|
||||||
# the binaries and libraries a build for another platform writes.
|
|
||||||
/bin
|
/bin
|
||||||
/routewatch
|
/routewatch
|
||||||
/asinfo-gen
|
/asinfo-gen
|
||||||
@@ -83,24 +82,15 @@
|
|||||||
/out
|
/out
|
||||||
/pkg/asinfo/asdata.json
|
/pkg/asinfo/asdata.json
|
||||||
**/*.tar.zst
|
**/*.tar.zst
|
||||||
**/*.log
|
|
||||||
**/*.test
|
**/*.test
|
||||||
**/*.out
|
**/*.out
|
||||||
**/*.tmp
|
**/*.tmp
|
||||||
**/*.exe
|
|
||||||
**/*.dll
|
|
||||||
**/*.so
|
|
||||||
**/*.dylib
|
|
||||||
|
|
||||||
# Local databases. The image carries a source archive of the whole build
|
# Local databases. The image carries a source archive of the whole build
|
||||||
# context, so these would otherwise ship inside it. The daemon's database
|
# context, so these would otherwise ship inside it.
|
||||||
# is db.sqlite, with its -wal and -shm files while it runs.
|
|
||||||
**/*.db
|
**/*.db
|
||||||
**/*.db-journal
|
**/*.db-journal
|
||||||
**/*.db-wal
|
**/*.db-wal
|
||||||
**/*.sqlite
|
|
||||||
**/*.sqlite-wal
|
|
||||||
**/*.sqlite-shm
|
|
||||||
|
|
||||||
# A local Go workspace points at directories outside the build context.
|
# A local Go workspace points at directories outside the build context.
|
||||||
/go.work
|
/go.work
|
||||||
|
|||||||
+1
-5
@@ -67,14 +67,10 @@ node_modules/
|
|||||||
go.work
|
go.work
|
||||||
go.work.sum
|
go.work.sum
|
||||||
|
|
||||||
# Local databases, temporary files and debug output. The daemon's
|
# Local databases, temporary files and debug output.
|
||||||
# database is db.sqlite, with its -wal and -shm files while it runs.
|
|
||||||
*.db
|
*.db
|
||||||
*.db-journal
|
*.db-journal
|
||||||
*.db-wal
|
*.db-wal
|
||||||
*.sqlite
|
|
||||||
*.sqlite-wal
|
|
||||||
*.sqlite-shm
|
|
||||||
*.tmp
|
*.tmp
|
||||||
log.txt
|
log.txt
|
||||||
out
|
out
|
||||||
|
|||||||
@@ -52,6 +52,7 @@ clean:
|
|||||||
run: build
|
run: build
|
||||||
DEBUG=routewatch ./bin/routewatch 2>&1 | tee log.txt
|
DEBUG=routewatch ./bin/routewatch 2>&1 | tee log.txt
|
||||||
|
|
||||||
# Rebuild the AS list built into RouteWatch from as.csv (needs the network).
|
|
||||||
asupdate:
|
asupdate:
|
||||||
@script/asupdate
|
@echo "Updating AS info data..."
|
||||||
|
@go run cmd/asinfo-gen/main.go | gzip > pkg/asinfo/asdata.json.gz.tmp && \
|
||||||
|
mv pkg/asinfo/asdata.json.gz.tmp pkg/asinfo/asdata.json.gz
|
||||||
|
|||||||
@@ -5,30 +5,12 @@ table in real time: it streams BGP UPDATE messages from the RIPE RIS Live
|
|||||||
service, maintains a live routing table in SQLite, and provides HTTP APIs for
|
service, maintains a live routing table in SQLite, and provides HTTP APIs for
|
||||||
querying routing information.
|
querying routing information.
|
||||||
|
|
||||||
## Rationale
|
|
||||||
|
|
||||||
RIPE RIS Live streams the BGP updates that RIPE's route collectors receive, as
|
|
||||||
they happen, from the moment a client connects. It reports changes, not the
|
|
||||||
routing table they add up to, so it cannot say which AS an IP address is routed
|
|
||||||
to now or which routes a prefix has. RouteWatch applies the feed's updates to
|
|
||||||
live routing tables in its own SQLite database and answers IP-to-AS lookups and
|
|
||||||
prefix pages from them.
|
|
||||||
|
|
||||||
The feed sends no table when a client connects, so these tables hold only the
|
|
||||||
routes the feed has announced since RouteWatch first started: an IP-to-AS lookup
|
|
||||||
finds nothing for an address whose routes have not been announced since then.
|
|
||||||
The tables also miss the updates RouteWatch drops when it falls behind under
|
|
||||||
load and those sent while it is stopped, so a lookup or a prefix page can show a
|
|
||||||
route the feed has withdrawn.
|
|
||||||
|
|
||||||
## Features
|
## Features
|
||||||
|
|
||||||
- Real-time streaming of BGP updates from RIPE RIS Live
|
- Real-time streaming of BGP updates from RIPE RIS Live
|
||||||
- Maintains live IPv4 and IPv6 routing tables
|
- Maintains live IPv4 and IPv6 routing tables
|
||||||
- Tracks AS peering relationships
|
- Tracks AS peering relationships
|
||||||
- HTTP API for IP-to-AS lookups, prefix details, and AS information
|
- HTTP API for IP-to-AS lookups, prefix details, and AS information
|
||||||
- IP lookups report the public blocklists an address is on, from copies kept in
|
|
||||||
memory and on disk
|
|
||||||
- Automatic reconnection with exponential backoff
|
- Automatic reconnection with exponential backoff
|
||||||
- Batched database writes for high performance
|
- Batched database writes for high performance
|
||||||
- Backpressure handling to prevent memory exhaustion
|
- Backpressure handling to prevent memory exhaustion
|
||||||
@@ -54,22 +36,17 @@ DEBUG=routewatch ./routewatch
|
|||||||
|
|
||||||
## HTTP Endpoints
|
## HTTP Endpoints
|
||||||
|
|
||||||
Every route answers `HEAD` as it answers `GET`, without the body.
|
|
||||||
|
|
||||||
### Web Interface
|
### Web Interface
|
||||||
|
|
||||||
- `GET /` - Home page (HTML): headline figures, lookups by AS number and by IP
|
- `GET /` - Home page (HTML): headline figures, and lookups by AS number and by
|
||||||
address, and a search of AS names. A card shows the `curl` command that looks
|
IP address
|
||||||
up the visitor's own address at `/ip/{addr}`, and that command's live answer.
|
|
||||||
- `GET /status` - HTML status dashboard, which reads `/api/v1/stats`
|
- `GET /status` - HTML status dashboard, which reads `/api/v1/stats`
|
||||||
- `GET /status.json` - JSON statistics: feed, memory, database and route figures
|
- `GET /status.json` - JSON statistics: feed, memory, database and route figures
|
||||||
- `GET /.well-known/healthcheck.json` - Health check (JSON): 200 when the
|
- `GET /.well-known/healthcheck.json` - Health check (JSON): 200 when the
|
||||||
database answers and the RIS Live feed is connected, otherwise 503;
|
database answers and the RIS Live feed is connected, otherwise 503;
|
||||||
`data.status` is `ok` or `error` and `data.checks` says which check failed
|
`data.status` is `ok` or `error` and `data.checks` says which check failed
|
||||||
- `GET /as/{asn}` - AS detail page (HTML): the prefixes the AS originates and
|
- `GET /as/{asn}` - AS detail page (HTML): the prefixes the AS originates and
|
||||||
its peers. When the database has no record of the AS, its handle and
|
its peers
|
||||||
description come from the AS list built into RouteWatch. The page then shows
|
|
||||||
no first-seen date.
|
|
||||||
- `GET /prefix/{prefix}/{length}` - Prefix detail page (HTML), such as
|
- `GET /prefix/{prefix}/{length}` - Prefix detail page (HTML), such as
|
||||||
`/prefix/192.0.2.0/24`: every live route for the prefix, one per peer, and its
|
`/prefix/192.0.2.0/24`: every live route for the prefix, one per peer, and its
|
||||||
origin ASes
|
origin ASes
|
||||||
@@ -84,47 +61,18 @@ Every route answers `HEAD` as it answers `GET`, without the body.
|
|||||||
### API v1
|
### API v1
|
||||||
|
|
||||||
- `GET /api/v1/stats` - The `/status.json` figures plus connection, garbage
|
- `GET /api/v1/stats` - The `/status.json` figures plus connection, garbage
|
||||||
collector, stream and handler queue statistics. `stream.route_churn_per_min`
|
collector, stream and handler queue statistics
|
||||||
is the number of prefix announcements and withdrawals, from all peers, in the
|
- `GET /api/v1/ip/{ip}` - Looks up each address: its most specific live route
|
||||||
last full clock minute.
|
(`netblock`), that route's origin AS with its WHOIS details, and the address's
|
||||||
- `GET /api/v1/ip/{ip}` - Looks up each address. `{ip}` is an IP address, a host
|
PTR names. `{ip}` is an IP address, a host name (each address it resolves to
|
||||||
name, or a comma-separated list of at most 10 of these. `results` has one
|
is looked up), or a comma-separated list of these. `results` has one entry per
|
||||||
entry for each address, with:
|
address with a live route and `errors` one per address or name that failed.
|
||||||
- `ip` and `ptr`: the address and its PTR names
|
The answer is 400 when nothing in `{ip}` is an IP address or a host name that
|
||||||
- `netblock` and the fields beside it: the address's most specific live
|
resolves, and 404 when none of the addresses has a live route.
|
||||||
route, and that route's origin AS with its WHOIS details
|
|
||||||
- `blocklists`: the public blocklists the address is on (see Blocklists)
|
|
||||||
|
|
||||||
An address with no live route has a result without the route and AS fields.
|
|
||||||
Of the addresses a host name resolves to, only public ones are looked up. A
|
|
||||||
name with none, such as one that resolves to a private address, is reported
|
|
||||||
as `host name has no public address`, without its addresses. `errors` has
|
|
||||||
one entry for each host name that failed. The answer is 400 when:
|
|
||||||
- `{ip}` lists more than 10
|
|
||||||
- `{ip}` comes to more than 10 addresses once its names are resolved
|
|
||||||
- nothing in `{ip}` is an IP address or a host name with a public address
|
|
||||||
|
|
||||||
A PTR lookup gets 2 seconds. An address whose lookup takes longer is
|
|
||||||
answered without PTR names.
|
|
||||||
|
|
||||||
- `GET /api/v1/as/{asn}` - The AS and the prefixes it originates, IPv4 and IPv6
|
- `GET /api/v1/as/{asn}` - The AS and the prefixes it originates, IPv4 and IPv6
|
||||||
apart, each with when its newest route was updated. When the database has no
|
apart, each with when its newest route was updated
|
||||||
record of the AS, its handle and description come from the AS list built into
|
|
||||||
RouteWatch. The answer then has no `first_seen` or `last_seen`.
|
|
||||||
- `GET /api/v1/prefix/{prefix}/{length}` - Every live route for the prefix, one
|
- `GET /api/v1/prefix/{prefix}/{length}` - Every live route for the prefix, one
|
||||||
per peer, and the same routes grouped by origin AS in `origins`
|
per peer, and the same routes grouped by origin AS in `origins`
|
||||||
- `GET /api/v1/search/as?q={text}` - Finds the ASes whose handle or description
|
|
||||||
contains `{text}`, ignoring case. `data` holds:
|
|
||||||
- `query`: `{text}` without its leading and trailing spaces
|
|
||||||
- `total`: the number of ASes that match
|
|
||||||
- `results`: the first 50 of them in AS number order, each with `asn`,
|
|
||||||
`handle` and `description`
|
|
||||||
|
|
||||||
The search reads the AS list built into RouteWatch, not the database. The
|
|
||||||
list comes from
|
|
||||||
[ipverse/as-metadata](https://github.com/ipverse/as-metadata). A match can
|
|
||||||
therefore be an AS that the database has no record of. A `q` that is
|
|
||||||
missing, empty or only spaces answers 400.
|
|
||||||
|
|
||||||
A JSON answer is `{"status": "ok", "data": ...}`, or for an error
|
A JSON answer is `{"status": "ok", "data": ...}`, or for an error
|
||||||
`{"status": "error", "error": {"msg": ..., "code": ...}}`, and carries an
|
`{"status": "error", "error": {"msg": ..., "code": ...}}`, and carries an
|
||||||
@@ -135,16 +83,10 @@ took. The health check is the exception: its 503 answer is also
|
|||||||
A request that runs past its time limit answers 504: the limit is 30 seconds,
|
A request that runs past its time limit answers 504: the limit is 30 seconds,
|
||||||
and 4 seconds for the database reads behind `/status.json` and `/api/v1/stats`.
|
and 4 seconds for the database reads behind `/status.json` and `/api/v1/stats`.
|
||||||
An AS number, prefix or mask length that does not parse answers 400, as does a
|
An AS number, prefix or mask length that does not parse answers 400, as does a
|
||||||
mask length out of range for its IP version. An AS answers 404 when it is not in
|
mask length out of range for its IP version. An AS the database does not hold,
|
||||||
the AS list built into RouteWatch and the database has neither a record of it
|
or a prefix with no live route, answers 404. The HTML pages give these errors as
|
||||||
nor a live route it originates. AS 0 always answers 404, because it means no
|
plain text. A 500 answer says only `Internal server error`, and a host name that
|
||||||
known origin. The AS name search never lists it. A prefix with no live route
|
does not resolve only `could not resolve host name`; the detail goes to the log.
|
||||||
answers 404 too. The HTML pages give these errors as plain text. Under `/api/v1`
|
|
||||||
and `/ip/`, a path no route matches answers 404 with the message `Not found`,
|
|
||||||
and `POST`, `PUT`, `PATCH`, `DELETE`, `OPTIONS`, `TRACE` or `CONNECT` answers
|
|
||||||
405 with `Method not allowed`. A 500 answer says only `Internal server error`,
|
|
||||||
and a host name that does not resolve only `could not resolve host name`; the
|
|
||||||
detail goes to the log.
|
|
||||||
|
|
||||||
Every answer carries `Strict-Transport-Security`, `Content-Security-Policy`,
|
Every answer carries `Strict-Transport-Security`, `Content-Security-Policy`,
|
||||||
`X-Frame-Options`, `X-Content-Type-Options`, `Referrer-Policy` and
|
`X-Frame-Options`, `X-Content-Type-Options`, `Referrer-Policy` and
|
||||||
@@ -160,66 +102,6 @@ in a prefix's `origins`) means the route has no known origin: its AS path ends
|
|||||||
in an AS set of several members, any of which may have originated it. The pages
|
in an AS set of several members, any of which may have originated it. The pages
|
||||||
show such a route as having no known origin.
|
show such a route as having no known origin.
|
||||||
|
|
||||||
## Blocklists
|
|
||||||
|
|
||||||
The IP lookup reports which public blocklists list each address. Each result has
|
|
||||||
a `blocklists` list with one entry for each list the address is on:
|
|
||||||
|
|
||||||
- `name`: the list's name, from the lists below
|
|
||||||
- `description`: what being on the list means
|
|
||||||
- `updated`: when RouteWatch last downloaded the list, or last heard from its
|
|
||||||
publisher that its copy was current
|
|
||||||
|
|
||||||
An address on no list has an empty `blocklists`. An address with no live route
|
|
||||||
still has a result with its `blocklists`.
|
|
||||||
|
|
||||||
The lists:
|
|
||||||
|
|
||||||
- `dshield`: the 20 networks that DShield saw scan the most targets in the last
|
|
||||||
three days. From [DShield](https://isc.sans.edu/feeds_doc.html), downloaded
|
|
||||||
every hour.
|
|
||||||
- `feodo_tracker`: botnet command-and-control servers. From abuse.ch
|
|
||||||
[Feodo Tracker](https://feodotracker.abuse.ch/blocklist/), downloaded every 15
|
|
||||||
minutes.
|
|
||||||
- `firehol_abusers_30d`: addresses reported for comment, form or forum spam in
|
|
||||||
the last 30 days. From
|
|
||||||
[FireHOL](https://iplists.firehol.org/?ipset=firehol_abusers_30d), downloaded
|
|
||||||
every 5 hours.
|
|
||||||
- `firehol_level3`: addresses reported for attacks, spyware or viruses in the
|
|
||||||
last 30 days. From
|
|
||||||
[FireHOL](https://iplists.firehol.org/?ipset=firehol_level3), downloaded every
|
|
||||||
3 hours.
|
|
||||||
- `spamhaus_drop` and `spamhaus_dropv6`: IPv4 and IPv6 networks that Spamhaus
|
|
||||||
says spammers or cyber criminals have hijacked or lease. From
|
|
||||||
[Spamhaus DROP](https://www.spamhaus.org/blocklists/do-not-route-or-peer/),
|
|
||||||
downloaded once a day. Spamhaus merged its EDROP list into DROP.
|
|
||||||
- `tor_exits`: Tor exit nodes, where traffic from the Tor network leaves. From
|
|
||||||
the [Tor Project](https://check.torproject.org/torbulkexitlist), downloaded
|
|
||||||
every hour.
|
|
||||||
|
|
||||||
No list is downloaded more often than its publisher updates it or allows. Each
|
|
||||||
can be downloaded whole without an account or a key.
|
|
||||||
|
|
||||||
How RouteWatch keeps the lists:
|
|
||||||
|
|
||||||
- It holds each list in memory, so a lookup makes no outside request.
|
|
||||||
- It keeps a copy of each list in the `blocklists` directory of the state
|
|
||||||
directory (see Configuration). Each copy is a file named after its list.
|
|
||||||
- Beside each copy, a file ending in `.validators` keeps the `Last-Modified` and
|
|
||||||
`ETag` the publisher sent with that copy.
|
|
||||||
- At startup it loads each list from its copy. It downloads a list at once when
|
|
||||||
the copy is older than the list's interval, or when there is no copy.
|
|
||||||
- A download sends the copy's `Last-Modified` as `If-Modified-Since` and its
|
|
||||||
`ETag` as `If-None-Match`. The publisher then sends the list only when it has
|
|
||||||
changed. Without either, the download asks for the whole list.
|
|
||||||
- A download fails when it takes over a minute or its file is over 16 MiB.
|
|
||||||
- When a download fails, or its file does not parse, RouteWatch keeps the
|
|
||||||
previous copy and logs the failure. It tries again one interval later.
|
|
||||||
|
|
||||||
A successful IP lookup answer carries `Cache-Control` with a `max-age` of 15
|
|
||||||
minutes, the shortest list interval. The answer for the caller's own address
|
|
||||||
(`/ip/`) is `private`, and the others are `public`.
|
|
||||||
|
|
||||||
## Code Structure
|
## Code Structure
|
||||||
|
|
||||||
```
|
```
|
||||||
@@ -229,7 +111,6 @@ routewatch/
|
|||||||
│ ├── asinfo-gen/ # Utility to generate AS info data
|
│ ├── asinfo-gen/ # Utility to generate AS info data
|
||||||
│ └── streamdumper/ # Debug utility for raw stream output
|
│ └── streamdumper/ # Debug utility for raw stream output
|
||||||
├── internal/
|
├── internal/
|
||||||
│ ├── blocklist/ # Public IP blocklists: downloads and lookups
|
|
||||||
│ ├── routewatch/ # Core application logic
|
│ ├── routewatch/ # Core application logic
|
||||||
│ ├── server/ # HTTP server and handlers
|
│ ├── server/ # HTTP server and handlers
|
||||||
│ ├── database/ # SQLite storage layer
|
│ ├── database/ # SQLite storage layer
|
||||||
@@ -304,8 +185,7 @@ routewatch/
|
|||||||
- **ASHandler**: Tracks all ASNs seen in AS paths
|
- **ASHandler**: Tracks all ASNs seen in AS paths
|
||||||
- **PeerHandler**: Records BGP peer information
|
- **PeerHandler**: Records BGP peer information
|
||||||
- **PrefixHandler**: Maintains live routing table (upserts on announcement,
|
- **PrefixHandler**: Maintains live routing table (upserts on announcement,
|
||||||
deletes on withdrawal or on an announcement whose AS path is empty or
|
deletes on withdrawal, deletes all of a peer's routes when its BGP session
|
||||||
holds AS 0 anywhere, deletes all of a peer's routes when its BGP session
|
|
||||||
goes down). Routes are kept per peer IP, so when one peer IP has sessions
|
goes down). Routes are kept per peer IP, so when one peer IP has sessions
|
||||||
on two collectors, a session going down on one also removes the routes the
|
on two collectors, a session going down on one also removes the routes the
|
||||||
other still carries, until they are announced again.
|
other still carries, until they are announced again.
|
||||||
@@ -396,7 +276,7 @@ client sent. For a request from any other address they use the address the
|
|||||||
request came from, whatever forwarding headers it carries. `X-Real-IP` and
|
request came from, whatever forwarding headers it carries. `X-Real-IP` and
|
||||||
`True-Client-IP` are not read.
|
`True-Client-IP` are not read.
|
||||||
|
|
||||||
State directory (the database and the blocklist copies):
|
State directory (database location):
|
||||||
|
|
||||||
- macOS: `~/Library/Application Support/berlin.sneak.app.routewatch/`
|
- macOS: `~/Library/Application Support/berlin.sneak.app.routewatch/`
|
||||||
- Linux: `/var/lib/berlin.sneak.app.routewatch/` when running as root, otherwise
|
- Linux: `/var/lib/berlin.sneak.app.routewatch/` when running as root, otherwise
|
||||||
@@ -558,11 +438,6 @@ provide:
|
|||||||
or `go.sum`) and `go fmt` (writes), then `script/check`
|
or `go.sum`) and `go fmt` (writes), then `script/check`
|
||||||
- `script/install-precommit` — install the git pre-commit hook that runs
|
- `script/install-precommit` — install the git pre-commit hook that runs
|
||||||
`script/precommit`
|
`script/precommit`
|
||||||
- `script/asupdate` — rebuild the AS list built into RouteWatch,
|
|
||||||
`pkg/asinfo/asdata.json.gz`, from the `as.csv` file of
|
|
||||||
[ipverse/as-metadata](https://github.com/ipverse/as-metadata). It needs the
|
|
||||||
network. It runs `cmd/asinfo-gen`, which fails when it finds no AS. When
|
|
||||||
`cmd/asinfo-gen` fails, the old list stays and the script exits non-zero.
|
|
||||||
|
|
||||||
Tests and the linter run only in Docker, as the `test` and `lint` phases of the
|
Tests and the linter run only in Docker, as the `test` and `lint` phases of the
|
||||||
`Dockerfile`; the image cannot be built unless both pass. Every `docker build`
|
`Dockerfile`; the image cannot be built unless both pass. Every `docker build`
|
||||||
|
|||||||
@@ -33,71 +33,6 @@ tested.
|
|||||||
|
|
||||||
# Completed Steps
|
# Completed Steps
|
||||||
|
|
||||||
- 2026-10-27: each result of the IP lookup lists the public blocklists the
|
|
||||||
address is on, in `blocklists`. The lists are DShield, Feodo Tracker, two
|
|
||||||
FireHOL lists, Spamhaus DROP and the Tor exit list. RouteWatch downloads each
|
|
||||||
list on its own schedule and answers from copies in memory. The copies are
|
|
||||||
also kept in the state directory's `blocklists` directory. IP lookup answers
|
|
||||||
carry `Cache-Control`. An address with no live route now has a result too,
|
|
||||||
without route fields, so the IP lookup no longer answers 404 (closes #158)
|
|
||||||
- 2026-10-27: the home page has an IP information API card. It shows the `curl`
|
|
||||||
command that looks up the visitor's own address at `/ip/{addr}`, and that
|
|
||||||
command's live answer (closes #156)
|
|
||||||
- 2026-10-27: on the AS page, a click on the heading of the IPv4 or the IPv6
|
|
||||||
prefix list folds or unfolds the list. Both lists start open (closes #155)
|
|
||||||
- 2026-10-27: the AS page lists its IPv4 and its IPv6 prefixes by mask length,
|
|
||||||
shortest first, and prefixes of one mask length by address. They used to be
|
|
||||||
listed by address (closes #153)
|
|
||||||
- 2026-10-27: on `/` and `/status`, a spinner beside the status turns one frame
|
|
||||||
each time the stats show more messages than the time before. A page that has
|
|
||||||
not seen more messages for three refreshes shows "stale" and the time since
|
|
||||||
then instead, in red (closes #152)
|
|
||||||
- 2026-10-27: the WHOIS details of a Brazilian AS keep their accented letters.
|
|
||||||
LACNIC's answer for such an AS comes from NIC.br, Brazil's registry, with each
|
|
||||||
accented letter replaced. The details now come from NIC.br's own server,
|
|
||||||
`whois.registro.br`. When that query fails, the details from LACNIC's answer
|
|
||||||
are kept and a warning is logged. After NIC.br answers that RouteWatch is over
|
|
||||||
its query limit, RouteWatch stops asking NIC.br for an hour (closes #174)
|
|
||||||
- 2026-10-27: `make asupdate` stops with an error when the download of `as.csv`
|
|
||||||
breaks off. The generator used to retry the failed read forever. A record that
|
|
||||||
does not parse is still logged and skipped (closes #177)
|
|
||||||
- 2026-10-27: `make asupdate` runs the new `script/asupdate`. It replaces the AS
|
|
||||||
list built into RouteWatch only when the generator succeeds and finds at least
|
|
||||||
one AS. A failed download used to replace the list with an empty file. The
|
|
||||||
generator reads `as.csv` from `ipverse/as-metadata`, the source's new name
|
|
||||||
(closes #173)
|
|
||||||
- 2026-10-27: the WHOIS details of an AS at RIPE, APNIC or AFRINIC come only
|
|
||||||
from its `aut-num` object and the organisation object it names. RIPE and APNIC
|
|
||||||
start their answer with the `as-block` object of the AS's block. Its dates
|
|
||||||
were stored as the AS's (closes #168)
|
|
||||||
- 2026-10-27: AS names keep their accented letters. The AS list built into
|
|
||||||
RouteWatch is rebuilt from today's `as.csv`, which added a country code
|
|
||||||
column. That file no longer lists AS 0 or the documentation AS numbers. A
|
|
||||||
WHOIS answer that is not valid UTF-8 is decoded as Latin-1, which RIPE and
|
|
||||||
APNIC send, before it is parsed and stored (closes #165)
|
|
||||||
- 2026-10-27: the Age column on the AS and prefix pages reads in words, such as
|
|
||||||
"5 minutes ago" or "2 weeks ago", instead of Go durations such as "360h0m0s
|
|
||||||
ago". An age under a minute reads in seconds, and one over a month in months
|
|
||||||
or years instead of a date (closes #163)
|
|
||||||
- 2026-10-26: a write to a stored AS, peering or peer keeps the earlier of the
|
|
||||||
two first-seen times and the later of the two last-seen times. An old update
|
|
||||||
from a collector catching up on a backlog no longer moves last seen back
|
|
||||||
(closes #162)
|
|
||||||
- 2026-10-26: the status page shows "Route Churn/min", the number of prefix
|
|
||||||
announcements and withdrawals, from all peers, in the last full clock minute.
|
|
||||||
`/api/v1/stats` reports it as `route_churn_per_min` in `stream`, which
|
|
||||||
replaces the moving average `route_churn_per_sec`. The messages and updates
|
|
||||||
per second on `/` and `/status` are whole numbers (closes #160)
|
|
||||||
- 2026-10-26: the WHOIS fetcher backs off more slowly. A failure makes its wait
|
|
||||||
1.5 times as long, up to one minute instead of five. Its queries to RIPE,
|
|
||||||
APNIC and AFRINIC leave out person and role objects, which RIPE counts against
|
|
||||||
a daily limit per address. A refused query is logged with the RIR's error line
|
|
||||||
(closes #159)
|
|
||||||
- 2026-10-26: the home page's AS name search lists the ASes whose handle or
|
|
||||||
description holds the text, as links to their pages, from the new
|
|
||||||
`/api/v1/search/as`; the search ignores case. When the database has no record
|
|
||||||
of an AS, its page takes the handle and description from the AS list built
|
|
||||||
into RouteWatch and still lists the AS's prefixes (closes #151)
|
|
||||||
- 2026-10-19: README "HTTP Endpoints" lists every route with what it answers,
|
- 2026-10-19: README "HTTP Endpoints" lists every route with what it answers,
|
||||||
"Database Schema" matches `schema.sql`, and the rest of the README matches the
|
"Database Schema" matches `schema.sql`, and the rest of the README matches the
|
||||||
code; this file brought up to date (closes #72)
|
code; this file brought up to date (closes #72)
|
||||||
|
|||||||
+28
-60
@@ -6,7 +6,6 @@ import (
|
|||||||
"encoding/csv"
|
"encoding/csv"
|
||||||
"encoding/json"
|
"encoding/json"
|
||||||
"errors"
|
"errors"
|
||||||
"fmt"
|
|
||||||
"io"
|
"io"
|
||||||
"log"
|
"log"
|
||||||
"net/http"
|
"net/http"
|
||||||
@@ -15,21 +14,7 @@ import (
|
|||||||
"strings"
|
"strings"
|
||||||
)
|
)
|
||||||
|
|
||||||
const asCsvURL = "https://github.com/ipverse/as-metadata/raw/refs/heads/master/as.csv"
|
const asCsvURL = "https://github.com/ipverse/asn-info/raw/refs/heads/master/as.csv"
|
||||||
|
|
||||||
// The first columns of as.csv, the ones the generator reads. It reads past
|
|
||||||
// the columns after them, such as the country code.
|
|
||||||
const (
|
|
||||||
asnColumn = "asn"
|
|
||||||
handleColumn = "handle"
|
|
||||||
descriptionColumn = "description"
|
|
||||||
columnsRead = 3
|
|
||||||
)
|
|
||||||
|
|
||||||
var (
|
|
||||||
errUnexpectedHeader = errors.New("unexpected CSV header")
|
|
||||||
errNoASes = errors.New("no AS in the CSV")
|
|
||||||
)
|
|
||||||
|
|
||||||
// ASInfo represents information about an Autonomous System
|
// ASInfo represents information about an Autonomous System
|
||||||
type ASInfo struct {
|
type ASInfo struct {
|
||||||
@@ -60,66 +45,40 @@ func main() {
|
|||||||
log.Fatalf("Failed to fetch CSV: HTTP %d", resp.StatusCode)
|
log.Fatalf("Failed to fetch CSV: HTTP %d", resp.StatusCode)
|
||||||
}
|
}
|
||||||
|
|
||||||
asInfos, err := parseCSV(resp.Body)
|
// Parse CSV
|
||||||
|
reader := csv.NewReader(resp.Body)
|
||||||
// Closed here rather than deferred: log.Fatalf exits without running
|
|
||||||
// deferred calls.
|
|
||||||
_ = resp.Body.Close()
|
|
||||||
|
|
||||||
if err != nil {
|
|
||||||
log.Fatalf("Failed to parse CSV: %v", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
log.Printf("Parsed %d AS records", len(asInfos))
|
|
||||||
|
|
||||||
// Convert to JSON and output to stdout
|
|
||||||
encoder := json.NewEncoder(os.Stdout)
|
|
||||||
encoder.SetIndent("", "\t")
|
|
||||||
|
|
||||||
err = encoder.Encode(asInfos)
|
|
||||||
if err != nil {
|
|
||||||
log.Fatalf("Failed to encode JSON: %v", err)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// parseCSV reads as.csv from r and returns its ASes, with the handle and the
|
|
||||||
// description as the file spells them. A header whose first columns are not
|
|
||||||
// the AS number, the handle and the description is an error. A record that
|
|
||||||
// does not parse is logged and skipped. A read that fails, such as on a broken
|
|
||||||
// download, is an error. A file left with no AS is an error.
|
|
||||||
func parseCSV(r io.Reader) ([]ASInfo, error) {
|
|
||||||
reader := csv.NewReader(r)
|
|
||||||
|
|
||||||
|
// Read header
|
||||||
header, err := reader.Read()
|
header, err := reader.Read()
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return nil, fmt.Errorf("read CSV header: %w", err)
|
log.Fatalf("Failed to read CSV header: %v", err)
|
||||||
}
|
}
|
||||||
|
|
||||||
if len(header) < columnsRead || header[0] != asnColumn ||
|
if len(header) != 3 || header[0] != "asn" || header[1] != "handle" ||
|
||||||
header[1] != handleColumn || header[2] != descriptionColumn {
|
header[2] != "description" {
|
||||||
return nil, fmt.Errorf("%w: %v", errUnexpectedHeader, header)
|
log.Fatalf("Unexpected CSV header: %v", header)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// Read all records
|
||||||
var asInfos []ASInfo
|
var asInfos []ASInfo
|
||||||
|
|
||||||
for {
|
for {
|
||||||
// The reader returns an error for a record with a different number
|
|
||||||
// of fields than the header.
|
|
||||||
record, err := reader.Read()
|
record, err := reader.Read()
|
||||||
if errors.Is(err, io.EOF) {
|
if errors.Is(err, io.EOF) {
|
||||||
break
|
break
|
||||||
}
|
}
|
||||||
|
|
||||||
if _, ok := errors.AsType[*csv.ParseError](err); ok {
|
if err != nil {
|
||||||
log.Printf("Error reading CSV record: %v", err)
|
log.Printf("Error reading CSV record: %v", err)
|
||||||
|
|
||||||
continue
|
continue
|
||||||
}
|
}
|
||||||
|
|
||||||
// Any other error comes from the download, such as a broken
|
const expectedFields = 3
|
||||||
// connection. Every later read returns it again.
|
if len(record) != expectedFields {
|
||||||
if err != nil {
|
log.Printf("Skipping invalid record: %v", record)
|
||||||
return nil, fmt.Errorf("read CSV record: %w", err)
|
|
||||||
|
continue
|
||||||
}
|
}
|
||||||
|
|
||||||
asn, err := strconv.Atoi(record[0])
|
asn, err := strconv.Atoi(record[0])
|
||||||
@@ -136,9 +95,18 @@ func parseCSV(r io.Reader) ([]ASInfo, error) {
|
|||||||
})
|
})
|
||||||
}
|
}
|
||||||
|
|
||||||
if len(asInfos) == 0 {
|
// Closed here rather than deferred: log.Fatalf exits without running
|
||||||
return nil, errNoASes
|
// deferred calls.
|
||||||
}
|
_ = resp.Body.Close()
|
||||||
|
|
||||||
return asInfos, nil
|
log.Printf("Parsed %d AS records", len(asInfos))
|
||||||
|
|
||||||
|
// Convert to JSON and output to stdout
|
||||||
|
encoder := json.NewEncoder(os.Stdout)
|
||||||
|
encoder.SetIndent("", "\t")
|
||||||
|
|
||||||
|
err = encoder.Encode(asInfos)
|
||||||
|
if err != nil {
|
||||||
|
log.Fatalf("Failed to encode JSON: %v", err)
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -1,142 +0,0 @@
|
|||||||
package main
|
|
||||||
|
|
||||||
import (
|
|
||||||
"errors"
|
|
||||||
"io"
|
|
||||||
"slices"
|
|
||||||
"strings"
|
|
||||||
"testing"
|
|
||||||
"testing/iotest"
|
|
||||||
"time"
|
|
||||||
)
|
|
||||||
|
|
||||||
// parseTimeLimit is how long a test waits for parseCSV to return before it
|
|
||||||
// fails.
|
|
||||||
const parseTimeLimit = 2 * time.Second
|
|
||||||
|
|
||||||
// TestParseCSVReadsTodaysLayout parses an as.csv with the country code column
|
|
||||||
// that the source added after the description. It expects each AS with its
|
|
||||||
// handle and description spelled exactly as in the file, accents included.
|
|
||||||
func TestParseCSVReadsTodaysLayout(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
const asCSV = `asn,handle,description,country-code
|
|
||||||
1,LVLT-1,Level 3 Parent LLC,US
|
|
||||||
1251,FUNDACAO-AMPARO-PESQUISA,FUNDAÇÃO DE AMPARO À PESQUISA DO ESTADO SÃO PAULO,BR
|
|
||||||
`
|
|
||||||
|
|
||||||
got, err := parseCSV(strings.NewReader(asCSV))
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("parseCSV: %v", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
want := []ASInfo{
|
|
||||||
{ASN: 1, Handle: "LVLT-1", Description: "Level 3 Parent LLC"},
|
|
||||||
{
|
|
||||||
ASN: 1251,
|
|
||||||
Handle: "FUNDACAO-AMPARO-PESQUISA",
|
|
||||||
Description: "FUNDAÇÃO DE AMPARO À PESQUISA DO ESTADO " +
|
|
||||||
"SÃO PAULO",
|
|
||||||
},
|
|
||||||
}
|
|
||||||
|
|
||||||
if !slices.Equal(got, want) {
|
|
||||||
t.Errorf("parseCSV = %+v, want %+v", got, want)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestParseCSVRejectsAnotherHeader parses files whose first columns are not
|
|
||||||
// the AS number, the handle and the description. It expects an error rather
|
|
||||||
// than a list read from the wrong columns.
|
|
||||||
func TestParseCSVRejectsAnotherHeader(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
for _, asCSV := range []string{
|
|
||||||
"asn,handle\n1,LVLT-1\n",
|
|
||||||
"asn,description,handle\n1,Level 3 Parent LLC,LVLT-1\n",
|
|
||||||
} {
|
|
||||||
_, err := parseCSV(strings.NewReader(asCSV))
|
|
||||||
if err == nil {
|
|
||||||
t.Errorf("parseCSV(%q): no error", asCSV)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestParseCSVRejectsAFileWithNoASes parses files that hold no AS, one with
|
|
||||||
// only the header and one whose only record has no valid AS number. It
|
|
||||||
// expects errNoASes, so that make asupdate keeps the old list.
|
|
||||||
func TestParseCSVRejectsAFileWithNoASes(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
for _, asCSV := range []string{
|
|
||||||
"asn,handle,description,country-code\n",
|
|
||||||
"asn,handle,description,country-code\nAS1,LVLT-1,Level 3,US\n",
|
|
||||||
} {
|
|
||||||
_, err := parseCSV(strings.NewReader(asCSV))
|
|
||||||
if !errors.Is(err, errNoASes) {
|
|
||||||
t.Errorf("parseCSV(%q) = %v, want %v", asCSV, err, errNoASes)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestParseCSVSkipsARecordThatDoesNotParse parses an as.csv with a record
|
|
||||||
// that has too few fields between two valid records. It expects both valid
|
|
||||||
// ASes and no error, so that one bad record does not stop make asupdate.
|
|
||||||
func TestParseCSVSkipsARecordThatDoesNotParse(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
const asCSV = "asn,handle,description,country-code\n" +
|
|
||||||
"1,LVLT-1,Level 3 Parent LLC,US\n" +
|
|
||||||
"2,UDEL-DCN\n" +
|
|
||||||
"3,MIT-GATEWAYS,Massachusetts Institute of Technology,US\n"
|
|
||||||
|
|
||||||
got, err := parseCSV(strings.NewReader(asCSV))
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("parseCSV: %v", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
want := []ASInfo{
|
|
||||||
{ASN: 1, Handle: "LVLT-1", Description: "Level 3 Parent LLC"},
|
|
||||||
{
|
|
||||||
ASN: 3,
|
|
||||||
Handle: "MIT-GATEWAYS",
|
|
||||||
Description: "Massachusetts Institute of Technology",
|
|
||||||
},
|
|
||||||
}
|
|
||||||
|
|
||||||
if !slices.Equal(got, want) {
|
|
||||||
t.Errorf("parseCSV = %+v, want %+v", got, want)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestParseCSVStopsWhenTheDownloadBreaksOff parses an as.csv whose download
|
|
||||||
// breaks off after the first record. From then on every read fails, as on a
|
|
||||||
// broken connection. It expects parseCSV to return that error, so that make
|
|
||||||
// asupdate keeps the old list.
|
|
||||||
func TestParseCSVStopsWhenTheDownloadBreaksOff(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
const asCSV = "asn,handle,description,country-code\n" +
|
|
||||||
"1,LVLT-1,Level 3 Parent LLC,US\n"
|
|
||||||
|
|
||||||
download := io.MultiReader(
|
|
||||||
strings.NewReader(asCSV), iotest.ErrReader(io.ErrUnexpectedEOF))
|
|
||||||
|
|
||||||
// parseCSV runs on its own goroutine, so that a parseCSV that never
|
|
||||||
// returns fails the test instead of hanging it.
|
|
||||||
parsed := make(chan error, 1)
|
|
||||||
|
|
||||||
go func() {
|
|
||||||
_, err := parseCSV(download)
|
|
||||||
parsed <- err
|
|
||||||
}()
|
|
||||||
|
|
||||||
select {
|
|
||||||
case err := <-parsed:
|
|
||||||
if !errors.Is(err, io.ErrUnexpectedEOF) {
|
|
||||||
t.Errorf("parseCSV = %v, want %v", err, io.ErrUnexpectedEOF)
|
|
||||||
}
|
|
||||||
case <-time.After(parseTimeLimit):
|
|
||||||
t.Fatalf("parseCSV did not return within %v", parseTimeLimit)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
@@ -9,9 +9,7 @@ require (
|
|||||||
github.com/mattn/go-sqlite3 v1.14.29
|
github.com/mattn/go-sqlite3 v1.14.29
|
||||||
github.com/rcrowley/go-metrics v0.0.0-20250401214520-65e299d6c5c9
|
github.com/rcrowley/go-metrics v0.0.0-20250401214520-65e299d6c5c9
|
||||||
go.uber.org/fx v1.24.0
|
go.uber.org/fx v1.24.0
|
||||||
go4.org/netipx v0.0.0-20260823151212-3075585bcbeb
|
|
||||||
golang.org/x/term v0.33.0
|
golang.org/x/term v0.33.0
|
||||||
golang.org/x/text v0.43.0
|
|
||||||
)
|
)
|
||||||
|
|
||||||
require (
|
require (
|
||||||
|
|||||||
@@ -24,13 +24,9 @@ go.uber.org/multierr v1.10.0 h1:S0h4aNzvfcFsC3dRF1jLoaov7oRaKqRGC/pUEJ2yvPQ=
|
|||||||
go.uber.org/multierr v1.10.0/go.mod h1:20+QtiLqy0Nd6FdQB9TLXag12DsQkrbs3htMFfDN80Y=
|
go.uber.org/multierr v1.10.0/go.mod h1:20+QtiLqy0Nd6FdQB9TLXag12DsQkrbs3htMFfDN80Y=
|
||||||
go.uber.org/zap v1.26.0 h1:sI7k6L95XOKS281NhVKOFCUNIvv9e0w4BF8N3u+tCRo=
|
go.uber.org/zap v1.26.0 h1:sI7k6L95XOKS281NhVKOFCUNIvv9e0w4BF8N3u+tCRo=
|
||||||
go.uber.org/zap v1.26.0/go.mod h1:dtElttAiwGvoJ/vj4IwHBS/gXsEu/pZ50mUIRWuG0so=
|
go.uber.org/zap v1.26.0/go.mod h1:dtElttAiwGvoJ/vj4IwHBS/gXsEu/pZ50mUIRWuG0so=
|
||||||
go4.org/netipx v0.0.0-20260823151212-3075585bcbeb h1:XBM4hvfwGAttkkiTIFfeigdfcL1xIfdKXqFdgiHGtDs=
|
|
||||||
go4.org/netipx v0.0.0-20260823151212-3075585bcbeb/go.mod h1:PLyyIXexvUFg3Owu6p/WfdlivPbZJsZdgWZlrGope/Y=
|
|
||||||
golang.org/x/sys v0.34.0 h1:H5Y5sJ2L2JRdyv7ROF1he/lPdvFsd0mJHFw2ThKHxLA=
|
golang.org/x/sys v0.34.0 h1:H5Y5sJ2L2JRdyv7ROF1he/lPdvFsd0mJHFw2ThKHxLA=
|
||||||
golang.org/x/sys v0.34.0/go.mod h1:BJP2sWEmIv4KK5OTEluFJCKSidICx8ciO85XgH3Ak8k=
|
golang.org/x/sys v0.34.0/go.mod h1:BJP2sWEmIv4KK5OTEluFJCKSidICx8ciO85XgH3Ak8k=
|
||||||
golang.org/x/term v0.33.0 h1:NuFncQrRcaRvVmgRkvM3j/F00gWIAlcmlB8ACEKmGIg=
|
golang.org/x/term v0.33.0 h1:NuFncQrRcaRvVmgRkvM3j/F00gWIAlcmlB8ACEKmGIg=
|
||||||
golang.org/x/term v0.33.0/go.mod h1:s18+ql9tYWp1IfpV9DmCtQDDSRBUjKaw9M1eAv5UeF0=
|
golang.org/x/term v0.33.0/go.mod h1:s18+ql9tYWp1IfpV9DmCtQDDSRBUjKaw9M1eAv5UeF0=
|
||||||
golang.org/x/text v0.43.0 h1:1QivrlhwAsnMOcqgOdl9my7s9OyZl8O5/UGUTJVXoWc=
|
|
||||||
golang.org/x/text v0.43.0/go.mod h1:hbSIYA/amXcRXTFSZqh8tWHEoqRvvbKPJ2uz/KU+a+s=
|
|
||||||
gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA=
|
gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA=
|
||||||
gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
|
gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
|
||||||
|
|||||||
@@ -1,184 +0,0 @@
|
|||||||
// Package blocklist downloads public IP blocklists on a schedule, keeps the
|
|
||||||
// last good copy of each on disk and in memory, and answers which lists
|
|
||||||
// contain an address.
|
|
||||||
package blocklist
|
|
||||||
|
|
||||||
import (
|
|
||||||
"context"
|
|
||||||
"log/slog"
|
|
||||||
"net/netip"
|
|
||||||
"sync"
|
|
||||||
"time"
|
|
||||||
|
|
||||||
"go4.org/netipx"
|
|
||||||
)
|
|
||||||
|
|
||||||
// Format is how a list's file is written.
|
|
||||||
type Format int
|
|
||||||
|
|
||||||
const (
|
|
||||||
// FormatLines has one address or network per line, such as 192.0.2.1 or
|
|
||||||
// 192.0.2.0/24. Lines that start with # are comments.
|
|
||||||
FormatLines Format = iota
|
|
||||||
|
|
||||||
// FormatSpamhausJSON has one JSON object per line, with a network in its
|
|
||||||
// cidr field. The last line holds the file's metadata and no network.
|
|
||||||
FormatSpamhausJSON
|
|
||||||
|
|
||||||
// FormatDShield has one network per line, as tab-separated fields that
|
|
||||||
// start with the network's first and last address. Lines that start
|
|
||||||
// with # are comments.
|
|
||||||
FormatDShield
|
|
||||||
)
|
|
||||||
|
|
||||||
// List is one public blocklist.
|
|
||||||
type List struct {
|
|
||||||
// Name names the list in lookup answers, and names its copy on disk.
|
|
||||||
Name string
|
|
||||||
|
|
||||||
// Description says in one line what being on the list means.
|
|
||||||
Description string
|
|
||||||
|
|
||||||
// URL is where the publisher serves the whole list.
|
|
||||||
URL string
|
|
||||||
|
|
||||||
// Interval is how long a copy is used before the list is downloaded
|
|
||||||
// again.
|
|
||||||
Interval time.Duration
|
|
||||||
|
|
||||||
// Format is how the list's file is written.
|
|
||||||
Format Format
|
|
||||||
}
|
|
||||||
|
|
||||||
// Listing says that a list contains an address.
|
|
||||||
type Listing struct {
|
|
||||||
Name string `json:"name"`
|
|
||||||
Description string `json:"description"`
|
|
||||||
|
|
||||||
// Updated is when routewatch last downloaded the list, or last heard from
|
|
||||||
// its publisher that its copy was still current.
|
|
||||||
Updated time.Time `json:"updated"`
|
|
||||||
}
|
|
||||||
|
|
||||||
// listCopy is a list's last good copy in memory.
|
|
||||||
type listCopy struct {
|
|
||||||
addresses *netipx.IPSet
|
|
||||||
updated time.Time
|
|
||||||
|
|
||||||
// validators identify the version of the list this copy holds.
|
|
||||||
validators validators
|
|
||||||
}
|
|
||||||
|
|
||||||
// Params is what New needs.
|
|
||||||
type Params struct {
|
|
||||||
// Dir is the directory that holds the copies on disk.
|
|
||||||
Dir string
|
|
||||||
|
|
||||||
// Lists are the lists to download and look addresses up in.
|
|
||||||
Lists []List
|
|
||||||
|
|
||||||
Logger *slog.Logger
|
|
||||||
}
|
|
||||||
|
|
||||||
// Blocklists keeps a copy of each list and answers which lists contain an
|
|
||||||
// address. Start loads the copies from disk and starts downloading the lists.
|
|
||||||
type Blocklists struct {
|
|
||||||
dir string
|
|
||||||
lists []List
|
|
||||||
logger *slog.Logger
|
|
||||||
|
|
||||||
// copies holds each list's last good copy, by list name. A list with no
|
|
||||||
// good copy yet has none.
|
|
||||||
mu sync.RWMutex
|
|
||||||
copies map[string]listCopy
|
|
||||||
|
|
||||||
// stop ends the goroutine Start runs, and done is closed when it returns.
|
|
||||||
stop context.CancelFunc
|
|
||||||
done chan struct{}
|
|
||||||
}
|
|
||||||
|
|
||||||
// New returns a Blocklists with no copies. Start loads them.
|
|
||||||
func New(params Params) *Blocklists {
|
|
||||||
return &Blocklists{
|
|
||||||
dir: params.Dir,
|
|
||||||
lists: params.Lists,
|
|
||||||
logger: params.Logger.With("component", "blocklists"),
|
|
||||||
copies: make(map[string]listCopy),
|
|
||||||
done: make(chan struct{}),
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// Lookup returns a Listing for each list whose copy contains addr, in the
|
|
||||||
// order of the lists given to New. It makes no outside request. When no list
|
|
||||||
// contains addr, it returns an empty list, not nil.
|
|
||||||
func (b *Blocklists) Lookup(addr netip.Addr) []Listing {
|
|
||||||
b.mu.RLock()
|
|
||||||
defer b.mu.RUnlock()
|
|
||||||
|
|
||||||
listings := []Listing{}
|
|
||||||
|
|
||||||
for _, list := range b.lists {
|
|
||||||
c, ok := b.copies[list.Name]
|
|
||||||
if ok && c.addresses.Contains(addr.Unmap()) {
|
|
||||||
listings = append(listings, Listing{
|
|
||||||
Name: list.Name,
|
|
||||||
Description: list.Description,
|
|
||||||
Updated: c.updated,
|
|
||||||
})
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
return listings
|
|
||||||
}
|
|
||||||
|
|
||||||
// ShortestInterval returns the shortest Interval of the lists. That is the
|
|
||||||
// soonest a copy can change, and so how long a lookup answer stays current.
|
|
||||||
func (b *Blocklists) ShortestInterval() time.Duration {
|
|
||||||
var shortest time.Duration
|
|
||||||
|
|
||||||
for i, list := range b.lists {
|
|
||||||
if i == 0 || list.Interval < shortest {
|
|
||||||
shortest = list.Interval
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
return shortest
|
|
||||||
}
|
|
||||||
|
|
||||||
// updated returns when the named list's copy was last downloaded or found
|
|
||||||
// current, or the zero time when the list has no copy.
|
|
||||||
func (b *Blocklists) updated(name string) time.Time {
|
|
||||||
b.mu.RLock()
|
|
||||||
defer b.mu.RUnlock()
|
|
||||||
|
|
||||||
return b.copies[name].updated
|
|
||||||
}
|
|
||||||
|
|
||||||
// copyValidators returns the validators of the named list's copy. They are
|
|
||||||
// empty when the list has no copy.
|
|
||||||
func (b *Blocklists) copyValidators(name string) validators {
|
|
||||||
b.mu.RLock()
|
|
||||||
defer b.mu.RUnlock()
|
|
||||||
|
|
||||||
return b.copies[name].validators
|
|
||||||
}
|
|
||||||
|
|
||||||
// setCopy makes c the named list's copy.
|
|
||||||
func (b *Blocklists) setCopy(name string, c listCopy) {
|
|
||||||
b.mu.Lock()
|
|
||||||
defer b.mu.Unlock()
|
|
||||||
|
|
||||||
b.copies[name] = c
|
|
||||||
}
|
|
||||||
|
|
||||||
// setUpdated records that the named list's copy was found current at t.
|
|
||||||
func (b *Blocklists) setUpdated(name string, t time.Time) {
|
|
||||||
b.mu.Lock()
|
|
||||||
defer b.mu.Unlock()
|
|
||||||
|
|
||||||
c, ok := b.copies[name]
|
|
||||||
if ok {
|
|
||||||
c.updated = t
|
|
||||||
b.copies[name] = c
|
|
||||||
}
|
|
||||||
}
|
|
||||||
@@ -1,494 +0,0 @@
|
|||||||
package blocklist
|
|
||||||
|
|
||||||
import (
|
|
||||||
"context"
|
|
||||||
"errors"
|
|
||||||
"io"
|
|
||||||
"io/fs"
|
|
||||||
"log/slog"
|
|
||||||
"net/http"
|
|
||||||
"net/http/httptest"
|
|
||||||
"net/netip"
|
|
||||||
"os"
|
|
||||||
"path/filepath"
|
|
||||||
"strings"
|
|
||||||
"sync"
|
|
||||||
"testing"
|
|
||||||
"time"
|
|
||||||
)
|
|
||||||
|
|
||||||
// The test list's name and description, the address its good version lists,
|
|
||||||
// and the address a newer version lists instead.
|
|
||||||
const (
|
|
||||||
testListName = "test_list"
|
|
||||||
testListDescription = "On the test list"
|
|
||||||
listedAddr = "192.0.2.1"
|
|
||||||
goodListBody = "# A test list\n" + listedAddr + "\n"
|
|
||||||
newListedAddr = "192.0.2.2"
|
|
||||||
newListBody = "# A test list\n" + newListedAddr + "\n"
|
|
||||||
)
|
|
||||||
|
|
||||||
// testETag is the ETag a publisher sends with a version of the test list.
|
|
||||||
const testETag = `"v1"`
|
|
||||||
|
|
||||||
// maxWait is how long a test waits for the download goroutine.
|
|
||||||
const maxWait = 10 * time.Second
|
|
||||||
|
|
||||||
// publisher stands in for a list's publisher. It answers every request with
|
|
||||||
// its status and body, and keeps the headers of each request. A 200 answer
|
|
||||||
// carries lastModified and etag, when set, as Last-Modified and ETag. As
|
|
||||||
// from a real publisher, a request whose If-None-Match or If-Modified-Since
|
|
||||||
// matches them is answered "not modified".
|
|
||||||
type publisher struct {
|
|
||||||
mu sync.Mutex
|
|
||||||
status int
|
|
||||||
body string
|
|
||||||
lastModified time.Time
|
|
||||||
etag string
|
|
||||||
headers []http.Header
|
|
||||||
}
|
|
||||||
|
|
||||||
// newPublisher starts a publisher that answers 200 with body, and returns it
|
|
||||||
// and its URL.
|
|
||||||
func newPublisher(t *testing.T, body string) (*publisher, string) {
|
|
||||||
t.Helper()
|
|
||||||
|
|
||||||
p := &publisher{status: http.StatusOK, body: body}
|
|
||||||
srv := httptest.NewServer(p)
|
|
||||||
t.Cleanup(srv.Close)
|
|
||||||
|
|
||||||
return p, srv.URL
|
|
||||||
}
|
|
||||||
|
|
||||||
func (p *publisher) ServeHTTP(w http.ResponseWriter, r *http.Request) {
|
|
||||||
p.mu.Lock()
|
|
||||||
defer p.mu.Unlock()
|
|
||||||
|
|
||||||
p.headers = append(p.headers, r.Header.Clone())
|
|
||||||
|
|
||||||
if p.status != http.StatusOK {
|
|
||||||
w.WriteHeader(p.status)
|
|
||||||
_, _ = io.WriteString(w, p.body)
|
|
||||||
|
|
||||||
return
|
|
||||||
}
|
|
||||||
|
|
||||||
if p.etag != "" {
|
|
||||||
w.Header().Set(etagHeader, p.etag)
|
|
||||||
}
|
|
||||||
|
|
||||||
// ServeContent sends Last-Modified unless lastModified is zero, and
|
|
||||||
// answers "not modified" when the request's conditions match.
|
|
||||||
http.ServeContent(w, r, testListName, p.lastModified,
|
|
||||||
strings.NewReader(p.body))
|
|
||||||
}
|
|
||||||
|
|
||||||
// answer makes the publisher answer later requests with status and body.
|
|
||||||
func (p *publisher) answer(status int, body string) {
|
|
||||||
p.mu.Lock()
|
|
||||||
defer p.mu.Unlock()
|
|
||||||
|
|
||||||
p.status, p.body = status, body
|
|
||||||
}
|
|
||||||
|
|
||||||
// publish makes the publisher answer later requests with a version of the
|
|
||||||
// list: body, with lastModified and etag. A zero lastModified or an empty
|
|
||||||
// etag is not sent.
|
|
||||||
func (p *publisher) publish(body string, lastModified time.Time, etag string) {
|
|
||||||
p.mu.Lock()
|
|
||||||
defer p.mu.Unlock()
|
|
||||||
|
|
||||||
p.status, p.body = http.StatusOK, body
|
|
||||||
p.lastModified, p.etag = lastModified, etag
|
|
||||||
}
|
|
||||||
|
|
||||||
// requests returns the headers of the requests the publisher has had.
|
|
||||||
func (p *publisher) requests() []http.Header {
|
|
||||||
p.mu.Lock()
|
|
||||||
defer p.mu.Unlock()
|
|
||||||
|
|
||||||
return append([]http.Header(nil), p.headers...)
|
|
||||||
}
|
|
||||||
|
|
||||||
// testList returns a list served at url, written in format.
|
|
||||||
func testList(url string, format Format) List {
|
|
||||||
return List{
|
|
||||||
Name: testListName,
|
|
||||||
Description: testListDescription,
|
|
||||||
URL: url,
|
|
||||||
Interval: time.Hour,
|
|
||||||
Format: format,
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
func newTestBlocklists(dir string, list List) *Blocklists {
|
|
||||||
return New(Params{
|
|
||||||
Dir: dir,
|
|
||||||
Lists: []List{list},
|
|
||||||
Logger: slog.New(slog.DiscardHandler),
|
|
||||||
})
|
|
||||||
}
|
|
||||||
|
|
||||||
// lookup looks addr up in b.
|
|
||||||
func lookup(t *testing.T, b *Blocklists, addr string) []Listing {
|
|
||||||
t.Helper()
|
|
||||||
|
|
||||||
return b.Lookup(netip.MustParseAddr(addr))
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestLookupFindsListedAddresses downloads a list in each format and looks
|
|
||||||
// up addresses on it, inside its networks, and off it, in IPv4 and IPv6.
|
|
||||||
// A listed address has one listing that names the list; any other has an
|
|
||||||
// empty list of listings.
|
|
||||||
func TestLookupFindsListedAddresses(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
tests := []struct {
|
|
||||||
name string
|
|
||||||
format Format
|
|
||||||
body string
|
|
||||||
listed []string
|
|
||||||
unlisted []string
|
|
||||||
}{
|
|
||||||
{
|
|
||||||
name: "single addresses",
|
|
||||||
format: FormatLines,
|
|
||||||
body: "# comment\n192.0.2.1\n\n2001:db8::1\n",
|
|
||||||
listed: []string{"192.0.2.1", "::ffff:192.0.2.1", "2001:db8::1"},
|
|
||||||
unlisted: []string{"192.0.2.2", "2001:db8::2"},
|
|
||||||
},
|
|
||||||
{
|
|
||||||
name: "networks",
|
|
||||||
format: FormatLines,
|
|
||||||
body: "# comment\n198.51.100.0/24\n2001:db8:1::/48\n",
|
|
||||||
listed: []string{"198.51.100.0", "198.51.100.255", "2001:db8:1::5"},
|
|
||||||
unlisted: []string{"198.51.101.0", "2001:db8:2::1"},
|
|
||||||
},
|
|
||||||
{
|
|
||||||
name: "Spamhaus networks",
|
|
||||||
format: FormatSpamhausJSON,
|
|
||||||
body: `{"cidr":"203.0.113.0/25","sblid":"SBL1","rir":"arin"}` + "\n" +
|
|
||||||
`{"cidr":"2001:db8:3::/48","sblid":"SBL2","rir":"ripencc"}` + "\n" +
|
|
||||||
`{"type":"metadata","timestamp":1793074442,"records":2}` + "\n",
|
|
||||||
listed: []string{"203.0.113.1", "2001:db8:3::1"},
|
|
||||||
unlisted: []string{"203.0.113.200", "2001:db8:4::1"},
|
|
||||||
},
|
|
||||||
{
|
|
||||||
name: "DShield networks",
|
|
||||||
format: FormatDShield,
|
|
||||||
body: "# DShield.org Recommended Block List\n" +
|
|
||||||
"192.0.2.0\t192.0.2.255\t24\t327\tEXAMPLE\tUS\tNone\n",
|
|
||||||
listed: []string{"192.0.2.0", "192.0.2.77"},
|
|
||||||
unlisted: []string{"192.0.3.1", "2001:db8::1"},
|
|
||||||
},
|
|
||||||
}
|
|
||||||
|
|
||||||
for _, tt := range tests {
|
|
||||||
t.Run(tt.name, func(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
_, url := newPublisher(t, tt.body)
|
|
||||||
b := newTestBlocklists(t.TempDir(), testList(url, tt.format))
|
|
||||||
|
|
||||||
err := b.download(t.Context(), b.lists[0])
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("download: %v", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
for _, addr := range tt.listed {
|
|
||||||
listings := lookup(t, b, addr)
|
|
||||||
if len(listings) != 1 || listings[0].Name != testListName ||
|
|
||||||
listings[0].Description != testListDescription ||
|
|
||||||
listings[0].Updated.IsZero() {
|
|
||||||
t.Errorf("%s: listings %+v, want one for %s", addr,
|
|
||||||
listings, testListName)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
for _, addr := range tt.unlisted {
|
|
||||||
listings := lookup(t, b, addr)
|
|
||||||
if listings == nil || len(listings) != 0 {
|
|
||||||
t.Errorf("%s: listings %#v, want an empty list", addr, listings)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
})
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestFailedDownloadKeepsThePreviousCopy downloads a good version of a list,
|
|
||||||
// then one that fails. The good copy stays in memory and on disk, and no
|
|
||||||
// part of the failed download is left beside it.
|
|
||||||
func TestFailedDownloadKeepsThePreviousCopy(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
tests := []struct {
|
|
||||||
name string
|
|
||||||
status int
|
|
||||||
body string
|
|
||||||
}{
|
|
||||||
{"server error", http.StatusInternalServerError, "unavailable"},
|
|
||||||
{"not a list", http.StatusOK, "<!DOCTYPE html>\n<html></html>\n"},
|
|
||||||
{
|
|
||||||
"over the size limit", http.StatusOK,
|
|
||||||
strings.Repeat(listedAddr+"\n", maxDownloadSize/len(listedAddr)),
|
|
||||||
},
|
|
||||||
}
|
|
||||||
|
|
||||||
for _, tt := range tests {
|
|
||||||
t.Run(tt.name, func(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
dir := t.TempDir()
|
|
||||||
p, url := newPublisher(t, goodListBody)
|
|
||||||
b := newTestBlocklists(dir, testList(url, FormatLines))
|
|
||||||
|
|
||||||
err := b.download(t.Context(), b.lists[0])
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("first download: %v", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
before := lookup(t, b, listedAddr)
|
|
||||||
|
|
||||||
p.answer(tt.status, tt.body)
|
|
||||||
|
|
||||||
err = b.download(t.Context(), b.lists[0])
|
|
||||||
if err == nil {
|
|
||||||
t.Fatal("second download returned no error")
|
|
||||||
}
|
|
||||||
|
|
||||||
after := lookup(t, b, listedAddr)
|
|
||||||
if len(after) != 1 || len(before) != 1 ||
|
|
||||||
!after[0].Updated.Equal(before[0].Updated) {
|
|
||||||
t.Errorf("listings %+v after the failure, want %+v", after, before)
|
|
||||||
}
|
|
||||||
|
|
||||||
copies := os.DirFS(dir)
|
|
||||||
|
|
||||||
onDisk, err := fs.ReadFile(copies, testListName)
|
|
||||||
if err != nil || string(onDisk) != goodListBody {
|
|
||||||
t.Errorf("copy on disk %q (%v), want %q", onDisk, err, goodListBody)
|
|
||||||
}
|
|
||||||
|
|
||||||
_, err = fs.Stat(copies, testListName+downloadSuffix)
|
|
||||||
if !errors.Is(err, fs.ErrNotExist) {
|
|
||||||
t.Errorf("the failed download's file is still there (%v)", err)
|
|
||||||
}
|
|
||||||
})
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestRestartLoadsCopiesFromDisk downloads a list, then starts a new
|
|
||||||
// Blocklists on the same directory, as a restart does. The new one answers
|
|
||||||
// from the copy on disk at once, and does not download the list again while
|
|
||||||
// the copy is younger than the list's interval.
|
|
||||||
func TestRestartLoadsCopiesFromDisk(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
dir := t.TempDir()
|
|
||||||
p, url := newPublisher(t, goodListBody)
|
|
||||||
list := testList(url, FormatLines)
|
|
||||||
|
|
||||||
first := newTestBlocklists(dir, list)
|
|
||||||
|
|
||||||
err := first.download(t.Context(), list)
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("download: %v", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
downloaded := lookup(t, first, listedAddr)[0].Updated
|
|
||||||
|
|
||||||
second := newTestBlocklists(dir, list)
|
|
||||||
second.Start(t.Context())
|
|
||||||
second.Stop(t.Context())
|
|
||||||
|
|
||||||
listings := lookup(t, second, listedAddr)
|
|
||||||
if len(listings) != 1 ||
|
|
||||||
listings[0].Updated.Sub(downloaded).Abs() > time.Second {
|
|
||||||
t.Errorf("after the restart, listings %+v, want one updated at %s",
|
|
||||||
listings, downloaded)
|
|
||||||
}
|
|
||||||
|
|
||||||
if got := len(p.requests()); got != 1 {
|
|
||||||
t.Errorf("the publisher had %d requests, want 1", got)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestStartRefreshesAnOldCopy starts on a copy older than its list's
|
|
||||||
// interval. The list is downloaded at once, with routewatch's User-Agent and
|
|
||||||
// the Last-Modified and ETag the publisher sent with the copy, kept on disk.
|
|
||||||
// The publisher answers that the copy is still current, so the copy is kept
|
|
||||||
// and marked as updated now.
|
|
||||||
func TestStartRefreshesAnOldCopy(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
dir := t.TempDir()
|
|
||||||
p, url := newPublisher(t, goodListBody)
|
|
||||||
list := testList(url, FormatLines)
|
|
||||||
|
|
||||||
lastModified := time.Now().Add(-3 * list.Interval).UTC().Truncate(time.Second)
|
|
||||||
p.publish(goodListBody, lastModified, testETag)
|
|
||||||
|
|
||||||
err := newTestBlocklists(dir, list).download(t.Context(), list)
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("download: %v", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
old := time.Now().Add(-2 * list.Interval).UTC().Truncate(time.Second)
|
|
||||||
|
|
||||||
err = os.Chtimes(filepath.Join(dir, testListName), old, old)
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("Chtimes: %v", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
b := newTestBlocklists(dir, list)
|
|
||||||
b.Start(t.Context())
|
|
||||||
|
|
||||||
deadline := time.Now().Add(maxWait)
|
|
||||||
for !lookup(t, b, listedAddr)[0].Updated.After(old) {
|
|
||||||
if time.Now().After(deadline) {
|
|
||||||
t.Fatalf("the copy was not marked updated within %s", maxWait)
|
|
||||||
}
|
|
||||||
|
|
||||||
time.Sleep(time.Millisecond)
|
|
||||||
}
|
|
||||||
|
|
||||||
b.Stop(t.Context())
|
|
||||||
|
|
||||||
requests := p.requests()
|
|
||||||
last := requests[len(requests)-1]
|
|
||||||
|
|
||||||
want := lastModified.Format(http.TimeFormat)
|
|
||||||
if got := last.Get(ifModifiedSinceHeader); got != want {
|
|
||||||
t.Errorf("If-Modified-Since %q, want %q", got, want)
|
|
||||||
}
|
|
||||||
|
|
||||||
if got := last.Get(ifNoneMatchHeader); got != testETag {
|
|
||||||
t.Errorf("If-None-Match %q, want %q", got, testETag)
|
|
||||||
}
|
|
||||||
|
|
||||||
if got := last.Get(userAgentHeader); got != userAgent {
|
|
||||||
t.Errorf("User-Agent %q, want %q", got, userAgent)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestNewVersionOlderThanTheLastDownloadReplacesTheCopy downloads a list.
|
|
||||||
// The publisher then puts out a new version whose Last-Modified is earlier
|
|
||||||
// than that download, as when a cache in front of the publisher passes the
|
|
||||||
// version on late. The next download sends the Last-Modified of the copy,
|
|
||||||
// not the time of the download, so it gets the new version.
|
|
||||||
func TestNewVersionOlderThanTheLastDownloadReplacesTheCopy(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
p, url := newPublisher(t, goodListBody)
|
|
||||||
list := testList(url, FormatLines)
|
|
||||||
b := newTestBlocklists(t.TempDir(), list)
|
|
||||||
|
|
||||||
firstVersion := time.Now().Add(-2 * list.Interval).UTC().Truncate(time.Second)
|
|
||||||
p.publish(goodListBody, firstVersion, "")
|
|
||||||
|
|
||||||
err := b.download(t.Context(), list)
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("first download: %v", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
p.publish(newListBody, firstVersion.Add(list.Interval), "")
|
|
||||||
|
|
||||||
err = b.download(t.Context(), list)
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("second download: %v", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
if listings := lookup(t, b, newListedAddr); len(listings) != 1 {
|
|
||||||
t.Errorf("%s: listings %+v, want one from the new version",
|
|
||||||
newListedAddr, listings)
|
|
||||||
}
|
|
||||||
|
|
||||||
want := firstVersion.Format(http.TimeFormat)
|
|
||||||
if got := p.requests()[1].Get(ifModifiedSinceHeader); got != want {
|
|
||||||
t.Errorf("If-Modified-Since %q, want %q", got, want)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestDownloadWithoutLastModifiedOrETagSendsNoCondition downloads a list
|
|
||||||
// twice from a publisher that sends neither Last-Modified nor ETag. Neither
|
|
||||||
// request carries If-Modified-Since or If-None-Match, so the second download
|
|
||||||
// gets the whole list again.
|
|
||||||
func TestDownloadWithoutLastModifiedOrETagSendsNoCondition(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
p, url := newPublisher(t, goodListBody)
|
|
||||||
list := testList(url, FormatLines)
|
|
||||||
b := newTestBlocklists(t.TempDir(), list)
|
|
||||||
|
|
||||||
err := b.download(t.Context(), list)
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("first download: %v", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
err = b.download(t.Context(), list)
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("second download: %v", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
for i, request := range p.requests() {
|
|
||||||
if request.Get(ifModifiedSinceHeader) != "" ||
|
|
||||||
request.Get(ifNoneMatchHeader) != "" {
|
|
||||||
t.Errorf("request %d has a condition: %v", i, request)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestStopCancelsADownloadInProgress stops while the publisher has not
|
|
||||||
// answered. Stop cancels the download and returns once the download
|
|
||||||
// goroutine has stopped.
|
|
||||||
func TestStopCancelsADownloadInProgress(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
arrived := make(chan struct{})
|
|
||||||
|
|
||||||
var once sync.Once
|
|
||||||
|
|
||||||
srv := httptest.NewServer(http.HandlerFunc(
|
|
||||||
func(_ http.ResponseWriter, r *http.Request) {
|
|
||||||
once.Do(func() { close(arrived) })
|
|
||||||
<-r.Context().Done()
|
|
||||||
}))
|
|
||||||
t.Cleanup(srv.Close)
|
|
||||||
|
|
||||||
b := newTestBlocklists(t.TempDir(), testList(srv.URL, FormatLines))
|
|
||||||
b.Start(t.Context())
|
|
||||||
|
|
||||||
select {
|
|
||||||
case <-arrived:
|
|
||||||
case <-time.After(maxWait):
|
|
||||||
t.Fatalf("no download began within %s", maxWait)
|
|
||||||
}
|
|
||||||
|
|
||||||
ctx, cancel := context.WithTimeout(t.Context(), maxWait)
|
|
||||||
defer cancel()
|
|
||||||
|
|
||||||
b.Stop(ctx)
|
|
||||||
|
|
||||||
select {
|
|
||||||
case <-b.done:
|
|
||||||
default:
|
|
||||||
t.Error("Stop returned before the download goroutine stopped")
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestPublicListsAreComplete checks that every public list has what a
|
|
||||||
// download and a lookup answer need, and a unique name.
|
|
||||||
func TestPublicListsAreComplete(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
names := make(map[string]bool)
|
|
||||||
|
|
||||||
for _, list := range Lists() {
|
|
||||||
if list.Name == "" || list.Description == "" || list.URL == "" ||
|
|
||||||
list.Interval <= 0 || names[list.Name] {
|
|
||||||
t.Errorf("list %+v is incomplete or has a name used before", list)
|
|
||||||
}
|
|
||||||
|
|
||||||
names[list.Name] = true
|
|
||||||
}
|
|
||||||
}
|
|
||||||
@@ -1,379 +0,0 @@
|
|||||||
package blocklist
|
|
||||||
|
|
||||||
import (
|
|
||||||
"context"
|
|
||||||
"encoding/json"
|
|
||||||
"errors"
|
|
||||||
"fmt"
|
|
||||||
"io"
|
|
||||||
"io/fs"
|
|
||||||
"net/http"
|
|
||||||
"os"
|
|
||||||
"slices"
|
|
||||||
"time"
|
|
||||||
|
|
||||||
"go4.org/netipx"
|
|
||||||
)
|
|
||||||
|
|
||||||
// Limits of one download.
|
|
||||||
const (
|
|
||||||
// downloadTimeout is how long one download may take.
|
|
||||||
downloadTimeout = time.Minute
|
|
||||||
|
|
||||||
// maxDownloadSize is the size of the largest list file accepted, in
|
|
||||||
// bytes. The largest list, FireHOL abusers, is about 2.5 MiB.
|
|
||||||
maxDownloadSize = 16 << 20
|
|
||||||
)
|
|
||||||
|
|
||||||
// The request headers a download sets.
|
|
||||||
const (
|
|
||||||
userAgentHeader = "User-Agent"
|
|
||||||
ifModifiedSinceHeader = "If-Modified-Since"
|
|
||||||
ifNoneMatchHeader = "If-None-Match"
|
|
||||||
|
|
||||||
// userAgent names routewatch to the publishers.
|
|
||||||
userAgent = "routewatch (+https://git.eeqj.de/sneak/routewatch)"
|
|
||||||
)
|
|
||||||
|
|
||||||
// The response headers that identify the version of a list a publisher sent.
|
|
||||||
const (
|
|
||||||
lastModifiedHeader = "Last-Modified"
|
|
||||||
etagHeader = "ETag"
|
|
||||||
)
|
|
||||||
|
|
||||||
const (
|
|
||||||
// downloadSuffix ends the name of the file a download is written to,
|
|
||||||
// beside the list's copy, until it has parsed.
|
|
||||||
downloadSuffix = ".download"
|
|
||||||
|
|
||||||
// validatorsSuffix ends the name of the file beside a list's copy that
|
|
||||||
// keeps the copy's validators.
|
|
||||||
validatorsSuffix = ".validators"
|
|
||||||
|
|
||||||
// dirPermissions are those of the directory that holds the copies.
|
|
||||||
dirPermissions = 0o750
|
|
||||||
|
|
||||||
// validatorsPermissions are those of the file that keeps a copy's
|
|
||||||
// validators.
|
|
||||||
validatorsPermissions = 0o600
|
|
||||||
)
|
|
||||||
|
|
||||||
// validators are the Last-Modified and ETag headers a publisher sent with a
|
|
||||||
// version of a list. A download sends them back, as If-Modified-Since and
|
|
||||||
// If-None-Match, so that the publisher sends the list only when it has
|
|
||||||
// changed. They are kept as the publisher wrote them.
|
|
||||||
//
|
|
||||||
//nolint:tagliatelle // snake_case keys, as the Go style guide asks
|
|
||||||
type validators struct {
|
|
||||||
LastModified string `json:"last_modified,omitempty"`
|
|
||||||
ETag string `json:"etag,omitempty"`
|
|
||||||
}
|
|
||||||
|
|
||||||
var (
|
|
||||||
errUnexpectedStatus = errors.New("unexpected HTTP status")
|
|
||||||
errTooLarge = errors.New("list is over the size limit")
|
|
||||||
)
|
|
||||||
|
|
||||||
// Start loads each list's copy from disk. It then starts the goroutine that
|
|
||||||
// downloads each list once its copy is older than its Interval, and a list
|
|
||||||
// with no copy at once. The goroutine runs until Stop: the end of ctx does
|
|
||||||
// not stop it.
|
|
||||||
func (b *Blocklists) Start(ctx context.Context) {
|
|
||||||
err := os.MkdirAll(b.dir, dirPermissions)
|
|
||||||
if err != nil {
|
|
||||||
b.logger.Error("Failed to create the blocklists directory",
|
|
||||||
"dir", b.dir, "error", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
nextDownload := make([]time.Time, len(b.lists))
|
|
||||||
|
|
||||||
for i, list := range b.lists {
|
|
||||||
loadErr := b.loadCopy(list)
|
|
||||||
if loadErr != nil && !errors.Is(loadErr, fs.ErrNotExist) {
|
|
||||||
b.logger.Error("Failed to load a blocklist copy",
|
|
||||||
"list", list.Name, "error", loadErr)
|
|
||||||
}
|
|
||||||
|
|
||||||
// A list with no copy has the zero time, so it is due at once.
|
|
||||||
nextDownload[i] = b.updated(list.Name).Add(list.Interval)
|
|
||||||
}
|
|
||||||
|
|
||||||
ctx, b.stop = context.WithCancel(context.WithoutCancel(ctx))
|
|
||||||
|
|
||||||
go func() {
|
|
||||||
b.run(ctx, nextDownload)
|
|
||||||
}()
|
|
||||||
|
|
||||||
b.logger.Info("Blocklist downloads started", "dir", b.dir)
|
|
||||||
}
|
|
||||||
|
|
||||||
// Stop stops the downloads, cancelling one in progress, and waits until the
|
|
||||||
// goroutine has returned or ctx ends. The copies in memory stay, so Lookup
|
|
||||||
// still answers.
|
|
||||||
func (b *Blocklists) Stop(ctx context.Context) {
|
|
||||||
b.stop()
|
|
||||||
|
|
||||||
select {
|
|
||||||
case <-b.done:
|
|
||||||
b.logger.Info("Blocklist downloads stopped")
|
|
||||||
case <-ctx.Done():
|
|
||||||
b.logger.Warn("Stop deadline passed before the blocklist downloads " +
|
|
||||||
"stopped")
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// run downloads each list when nextDownload says, until ctx ends. After each
|
|
||||||
// try, whether it worked or not, the list's next download is an Interval
|
|
||||||
// later.
|
|
||||||
func (b *Blocklists) run(ctx context.Context, nextDownload []time.Time) {
|
|
||||||
defer close(b.done)
|
|
||||||
|
|
||||||
if len(nextDownload) == 0 {
|
|
||||||
return
|
|
||||||
}
|
|
||||||
|
|
||||||
for {
|
|
||||||
soonest := slices.MinFunc(nextDownload, time.Time.Compare)
|
|
||||||
timer := time.NewTimer(time.Until(soonest))
|
|
||||||
|
|
||||||
select {
|
|
||||||
case <-ctx.Done():
|
|
||||||
timer.Stop()
|
|
||||||
|
|
||||||
return
|
|
||||||
case <-timer.C:
|
|
||||||
}
|
|
||||||
|
|
||||||
for i, list := range b.lists {
|
|
||||||
if time.Now().Before(nextDownload[i]) {
|
|
||||||
continue
|
|
||||||
}
|
|
||||||
|
|
||||||
err := b.download(ctx, list)
|
|
||||||
if ctx.Err() != nil {
|
|
||||||
return
|
|
||||||
}
|
|
||||||
|
|
||||||
if err != nil {
|
|
||||||
b.logger.Error("Blocklist download failed; keeping the "+
|
|
||||||
"previous copy", "list", list.Name, "error", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
nextDownload[i] = time.Now().Add(list.Interval)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// download asks the publisher for list, sending the validators of the list's
|
|
||||||
// copy when it has them. A new version that parses becomes the list's copy,
|
|
||||||
// on disk and in memory. An answer that the copy is still current marks the
|
|
||||||
// copy as updated now. Anything else leaves the copy as it is and returns an
|
|
||||||
// error.
|
|
||||||
func (b *Blocklists) download(ctx context.Context, list List) error {
|
|
||||||
ctx, cancel := context.WithTimeout(ctx, downloadTimeout)
|
|
||||||
defer cancel()
|
|
||||||
|
|
||||||
req, err := http.NewRequestWithContext(ctx, http.MethodGet, list.URL, nil)
|
|
||||||
if err != nil {
|
|
||||||
return err
|
|
||||||
}
|
|
||||||
|
|
||||||
req.Header.Set(userAgentHeader, userAgent)
|
|
||||||
|
|
||||||
held := b.copyValidators(list.Name)
|
|
||||||
if held.LastModified != "" {
|
|
||||||
req.Header.Set(ifModifiedSinceHeader, held.LastModified)
|
|
||||||
}
|
|
||||||
|
|
||||||
if held.ETag != "" {
|
|
||||||
req.Header.Set(ifNoneMatchHeader, held.ETag)
|
|
||||||
}
|
|
||||||
|
|
||||||
resp, err := http.DefaultClient.Do(req)
|
|
||||||
if err != nil {
|
|
||||||
return err
|
|
||||||
}
|
|
||||||
|
|
||||||
defer func() { _ = resp.Body.Close() }()
|
|
||||||
|
|
||||||
switch resp.StatusCode {
|
|
||||||
case http.StatusOK:
|
|
||||||
return b.replaceCopy(list, resp.Body, validators{
|
|
||||||
LastModified: resp.Header.Get(lastModifiedHeader),
|
|
||||||
ETag: resp.Header.Get(etagHeader),
|
|
||||||
})
|
|
||||||
case http.StatusNotModified:
|
|
||||||
return b.markCurrent(list)
|
|
||||||
default:
|
|
||||||
return fmt.Errorf("%w: %s", errUnexpectedStatus, resp.Status)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// replaceCopy writes a new version of list, read from body, beside the list's
|
|
||||||
// copy. Only when it parses does it replace the copy, on disk and in memory.
|
|
||||||
// sent are the validators the publisher sent with the new version.
|
|
||||||
func (b *Blocklists) replaceCopy(list List, body io.Reader, sent validators) error {
|
|
||||||
root, err := os.OpenRoot(b.dir)
|
|
||||||
if err != nil {
|
|
||||||
return err
|
|
||||||
}
|
|
||||||
|
|
||||||
defer func() { _ = root.Close() }()
|
|
||||||
|
|
||||||
downloadName := list.Name + downloadSuffix
|
|
||||||
|
|
||||||
addresses, err := writeAndParse(root, downloadName, body, list.Format)
|
|
||||||
if err != nil {
|
|
||||||
_ = root.Remove(downloadName)
|
|
||||||
|
|
||||||
return err
|
|
||||||
}
|
|
||||||
|
|
||||||
// The old copy's validators are removed before the copy is replaced, so
|
|
||||||
// that they are never kept with the new copy.
|
|
||||||
err = root.Remove(list.Name + validatorsSuffix)
|
|
||||||
if err != nil && !errors.Is(err, fs.ErrNotExist) {
|
|
||||||
return err
|
|
||||||
}
|
|
||||||
|
|
||||||
err = root.Rename(downloadName, list.Name)
|
|
||||||
if err != nil {
|
|
||||||
return err
|
|
||||||
}
|
|
||||||
|
|
||||||
b.setCopy(list.Name, listCopy{
|
|
||||||
addresses: addresses, updated: time.Now().UTC(), validators: sent,
|
|
||||||
})
|
|
||||||
b.logger.Info("Downloaded a new version of a blocklist", "list", list.Name)
|
|
||||||
|
|
||||||
// Without this file, the first download after a restart asks for the
|
|
||||||
// whole list.
|
|
||||||
err = writeValidators(root, list.Name, sent)
|
|
||||||
if err != nil {
|
|
||||||
b.logger.Error("Failed to keep the validators of a blocklist copy",
|
|
||||||
"list", list.Name, "error", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
return nil
|
|
||||||
}
|
|
||||||
|
|
||||||
// writeAndParse writes body to the file name in root, up to maxDownloadSize
|
|
||||||
// bytes, and parses the file as format.
|
|
||||||
func writeAndParse(
|
|
||||||
root *os.Root, name string, body io.Reader, format Format,
|
|
||||||
) (*netipx.IPSet, error) {
|
|
||||||
f, err := root.Create(name)
|
|
||||||
if err != nil {
|
|
||||||
return nil, err
|
|
||||||
}
|
|
||||||
|
|
||||||
defer func() { _ = f.Close() }()
|
|
||||||
|
|
||||||
written, err := io.Copy(f, io.LimitReader(body, maxDownloadSize+1))
|
|
||||||
if err != nil {
|
|
||||||
return nil, err
|
|
||||||
}
|
|
||||||
|
|
||||||
if written > maxDownloadSize {
|
|
||||||
return nil, errTooLarge
|
|
||||||
}
|
|
||||||
|
|
||||||
_, err = f.Seek(0, io.SeekStart)
|
|
||||||
if err != nil {
|
|
||||||
return nil, err
|
|
||||||
}
|
|
||||||
|
|
||||||
return parse(f, format)
|
|
||||||
}
|
|
||||||
|
|
||||||
// markCurrent records that the publisher says the list's copy is still
|
|
||||||
// current: the copy, on disk and in memory, is marked as updated now.
|
|
||||||
func (b *Blocklists) markCurrent(list List) error {
|
|
||||||
root, err := os.OpenRoot(b.dir)
|
|
||||||
if err != nil {
|
|
||||||
return err
|
|
||||||
}
|
|
||||||
|
|
||||||
defer func() { _ = root.Close() }()
|
|
||||||
|
|
||||||
now := time.Now().UTC()
|
|
||||||
|
|
||||||
err = root.Chtimes(list.Name, now, now)
|
|
||||||
if err != nil {
|
|
||||||
return err
|
|
||||||
}
|
|
||||||
|
|
||||||
b.setUpdated(list.Name, now)
|
|
||||||
b.logger.Debug("Blocklist copy is still current", "list", list.Name)
|
|
||||||
|
|
||||||
return nil
|
|
||||||
}
|
|
||||||
|
|
||||||
// loadCopy makes the list's copy on disk, with the validators kept beside it,
|
|
||||||
// its copy in memory. The file's modification time is when the copy was
|
|
||||||
// updated.
|
|
||||||
func (b *Blocklists) loadCopy(list List) error {
|
|
||||||
root, err := os.OpenRoot(b.dir)
|
|
||||||
if err != nil {
|
|
||||||
return err
|
|
||||||
}
|
|
||||||
|
|
||||||
defer func() { _ = root.Close() }()
|
|
||||||
|
|
||||||
f, err := root.Open(list.Name)
|
|
||||||
if err != nil {
|
|
||||||
return err
|
|
||||||
}
|
|
||||||
|
|
||||||
defer func() { _ = f.Close() }()
|
|
||||||
|
|
||||||
info, err := f.Stat()
|
|
||||||
if err != nil {
|
|
||||||
return err
|
|
||||||
}
|
|
||||||
|
|
||||||
addresses, err := parse(f, list.Format)
|
|
||||||
if err != nil {
|
|
||||||
return err
|
|
||||||
}
|
|
||||||
|
|
||||||
held, err := readValidators(root, list.Name)
|
|
||||||
if err != nil {
|
|
||||||
return err
|
|
||||||
}
|
|
||||||
|
|
||||||
b.setCopy(list.Name, listCopy{
|
|
||||||
addresses: addresses, updated: info.ModTime().UTC(), validators: held,
|
|
||||||
})
|
|
||||||
|
|
||||||
return nil
|
|
||||||
}
|
|
||||||
|
|
||||||
// writeValidators keeps held in the file beside the named list's copy.
|
|
||||||
func writeValidators(root *os.Root, name string, held validators) error {
|
|
||||||
data, err := json.Marshal(held)
|
|
||||||
if err != nil {
|
|
||||||
return err
|
|
||||||
}
|
|
||||||
|
|
||||||
return root.WriteFile(name+validatorsSuffix, data, validatorsPermissions)
|
|
||||||
}
|
|
||||||
|
|
||||||
// readValidators returns the validators kept beside the named list's copy.
|
|
||||||
// They are empty when there is no such file.
|
|
||||||
func readValidators(root *os.Root, name string) (validators, error) {
|
|
||||||
var held validators
|
|
||||||
|
|
||||||
data, err := root.ReadFile(name + validatorsSuffix)
|
|
||||||
if errors.Is(err, fs.ErrNotExist) {
|
|
||||||
return held, nil
|
|
||||||
}
|
|
||||||
|
|
||||||
if err != nil {
|
|
||||||
return held, err
|
|
||||||
}
|
|
||||||
|
|
||||||
err = json.Unmarshal(data, &held)
|
|
||||||
|
|
||||||
return held, err
|
|
||||||
}
|
|
||||||
@@ -1,93 +0,0 @@
|
|||||||
package blocklist
|
|
||||||
|
|
||||||
import "time"
|
|
||||||
|
|
||||||
// How often each list is downloaded. No list is downloaded more often than
|
|
||||||
// its publisher updates it or allows.
|
|
||||||
const (
|
|
||||||
// DShield asks that its feeds be downloaded at most once an hour:
|
|
||||||
// https://isc.sans.edu/feeds_doc.html
|
|
||||||
dshieldInterval = time.Hour
|
|
||||||
|
|
||||||
// abuse.ch makes the list every 5 minutes and recommends a download at
|
|
||||||
// least every 15: https://feodotracker.abuse.ch/blocklist/
|
|
||||||
feodoTrackerInterval = 15 * time.Minute
|
|
||||||
|
|
||||||
// FireHOL's page for each list gives how often the list changes on
|
|
||||||
// average: about every 5 hours for abusers and every 3 hours for level
|
|
||||||
// 3. FireHOL sets no download limit.
|
|
||||||
// https://iplists.firehol.org/?ipset=firehol_abusers_30d
|
|
||||||
// https://iplists.firehol.org/?ipset=firehol_level3
|
|
||||||
fireHOLAbusersInterval = 5 * time.Hour
|
|
||||||
fireHOLLevel3Interval = 3 * time.Hour
|
|
||||||
|
|
||||||
// Spamhaus asks that DROP be downloaded at most once a day:
|
|
||||||
// https://www.spamhaus.org/blocklists/do-not-route-or-peer/
|
|
||||||
spamhausDROPInterval = 24 * time.Hour
|
|
||||||
|
|
||||||
// Tor publishes a new exit list every hour and sets no download limit:
|
|
||||||
// https://collector.torproject.org/recent/exit-lists/
|
|
||||||
torExitsInterval = time.Hour
|
|
||||||
)
|
|
||||||
|
|
||||||
// Lists returns the public blocklists routewatch downloads. Each can be
|
|
||||||
// downloaded whole without an account or a key. Spamhaus merged its EDROP
|
|
||||||
// list into DROP, so DROP covers both.
|
|
||||||
func Lists() []List {
|
|
||||||
return []List{
|
|
||||||
{
|
|
||||||
Name: "dshield",
|
|
||||||
Description: "In one of the 20 networks that DShield saw scan " +
|
|
||||||
"the most targets in the last three days",
|
|
||||||
URL: "https://feeds.dshield.org/block.txt",
|
|
||||||
Interval: dshieldInterval,
|
|
||||||
Format: FormatDShield,
|
|
||||||
},
|
|
||||||
{
|
|
||||||
Name: "feodo_tracker",
|
|
||||||
Description: "A botnet command-and-control server",
|
|
||||||
URL: "https://feodotracker.abuse.ch/downloads/ipblocklist.txt",
|
|
||||||
Interval: feodoTrackerInterval,
|
|
||||||
Format: FormatLines,
|
|
||||||
},
|
|
||||||
{
|
|
||||||
Name: "firehol_abusers_30d",
|
|
||||||
Description: "Reported for comment, form or forum spam in the " +
|
|
||||||
"last 30 days",
|
|
||||||
URL: "https://iplists.firehol.org/files/firehol_abusers_30d.netset",
|
|
||||||
Interval: fireHOLAbusersInterval,
|
|
||||||
Format: FormatLines,
|
|
||||||
},
|
|
||||||
{
|
|
||||||
Name: "firehol_level3",
|
|
||||||
Description: "Reported for attacks, spyware or viruses in the " +
|
|
||||||
"last 30 days",
|
|
||||||
URL: "https://iplists.firehol.org/files/firehol_level3.netset",
|
|
||||||
Interval: fireHOLLevel3Interval,
|
|
||||||
Format: FormatLines,
|
|
||||||
},
|
|
||||||
{
|
|
||||||
Name: "spamhaus_drop",
|
|
||||||
Description: "In an IPv4 network that Spamhaus says spammers " +
|
|
||||||
"or cyber criminals have hijacked or lease",
|
|
||||||
URL: "https://www.spamhaus.org/drop/drop_v4.json",
|
|
||||||
Interval: spamhausDROPInterval,
|
|
||||||
Format: FormatSpamhausJSON,
|
|
||||||
},
|
|
||||||
{
|
|
||||||
Name: "spamhaus_dropv6",
|
|
||||||
Description: "In an IPv6 network that Spamhaus says spammers " +
|
|
||||||
"or cyber criminals have hijacked or lease",
|
|
||||||
URL: "https://www.spamhaus.org/drop/drop_v6.json",
|
|
||||||
Interval: spamhausDROPInterval,
|
|
||||||
Format: FormatSpamhausJSON,
|
|
||||||
},
|
|
||||||
{
|
|
||||||
Name: "tor_exits",
|
|
||||||
Description: "A Tor exit node, where traffic from the Tor network leaves",
|
|
||||||
URL: "https://check.torproject.org/torbulkexitlist",
|
|
||||||
Interval: torExitsInterval,
|
|
||||||
Format: FormatLines,
|
|
||||||
},
|
|
||||||
}
|
|
||||||
}
|
|
||||||
@@ -1,158 +0,0 @@
|
|||||||
package blocklist
|
|
||||||
|
|
||||||
import (
|
|
||||||
"bufio"
|
|
||||||
"encoding/json"
|
|
||||||
"errors"
|
|
||||||
"fmt"
|
|
||||||
"io"
|
|
||||||
"net/netip"
|
|
||||||
"strings"
|
|
||||||
|
|
||||||
"go4.org/netipx"
|
|
||||||
)
|
|
||||||
|
|
||||||
const (
|
|
||||||
// commentPrefix starts a comment line in a list file.
|
|
||||||
commentPrefix = "#"
|
|
||||||
|
|
||||||
// networkSeparator separates the address and the mask length of a
|
|
||||||
// network, as in 192.0.2.0/24.
|
|
||||||
networkSeparator = "/"
|
|
||||||
|
|
||||||
// dshieldFieldSeparator separates the fields of a DShield line.
|
|
||||||
dshieldFieldSeparator = "\t"
|
|
||||||
|
|
||||||
// spamhausMetadataType is the type of the line that ends a Spamhaus
|
|
||||||
// file. That line holds the file's metadata and no network.
|
|
||||||
spamhausMetadataType = "metadata"
|
|
||||||
)
|
|
||||||
|
|
||||||
var (
|
|
||||||
errUnknownFormat = errors.New("unknown list format")
|
|
||||||
errNoNetwork = errors.New("no network on the line")
|
|
||||||
errBadRange = errors.New("first and last address make no range")
|
|
||||||
)
|
|
||||||
|
|
||||||
// parse reads a list file written in format and returns the addresses it
|
|
||||||
// holds. A line it cannot read fails the whole file, so a file that is not
|
|
||||||
// the list, such as an error page, never replaces a good copy.
|
|
||||||
func parse(r io.Reader, format Format) (*netipx.IPSet, error) {
|
|
||||||
var builder netipx.IPSetBuilder
|
|
||||||
|
|
||||||
scanner := bufio.NewScanner(r)
|
|
||||||
|
|
||||||
for lineNumber := 1; scanner.Scan(); lineNumber++ {
|
|
||||||
line := strings.TrimSpace(scanner.Text())
|
|
||||||
if line == "" || strings.HasPrefix(line, commentPrefix) {
|
|
||||||
continue
|
|
||||||
}
|
|
||||||
|
|
||||||
err := addLine(&builder, line, format)
|
|
||||||
if err != nil {
|
|
||||||
return nil, fmt.Errorf("line %d: %w", lineNumber, err)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
err := scanner.Err()
|
|
||||||
if err != nil {
|
|
||||||
return nil, err
|
|
||||||
}
|
|
||||||
|
|
||||||
return builder.IPSet()
|
|
||||||
}
|
|
||||||
|
|
||||||
// addLine adds the address or network on one line of a list file to builder.
|
|
||||||
func addLine(builder *netipx.IPSetBuilder, line string, format Format) error {
|
|
||||||
switch format {
|
|
||||||
case FormatLines:
|
|
||||||
return addAddressOrNetwork(builder, line)
|
|
||||||
case FormatSpamhausJSON:
|
|
||||||
return addSpamhausNetwork(builder, line)
|
|
||||||
case FormatDShield:
|
|
||||||
return addDShieldNetwork(builder, line)
|
|
||||||
default:
|
|
||||||
return errUnknownFormat
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// addAddressOrNetwork adds an address, such as 192.0.2.1, or a network, such
|
|
||||||
// as 192.0.2.0/24, to builder.
|
|
||||||
func addAddressOrNetwork(builder *netipx.IPSetBuilder, text string) error {
|
|
||||||
if strings.Contains(text, networkSeparator) {
|
|
||||||
network, err := netip.ParsePrefix(text)
|
|
||||||
if err != nil {
|
|
||||||
return err
|
|
||||||
}
|
|
||||||
|
|
||||||
builder.AddPrefix(network)
|
|
||||||
|
|
||||||
return nil
|
|
||||||
}
|
|
||||||
|
|
||||||
addr, err := netip.ParseAddr(text)
|
|
||||||
if err != nil {
|
|
||||||
return err
|
|
||||||
}
|
|
||||||
|
|
||||||
builder.Add(addr)
|
|
||||||
|
|
||||||
return nil
|
|
||||||
}
|
|
||||||
|
|
||||||
// addSpamhausNetwork adds the network on a line of a Spamhaus file to
|
|
||||||
// builder. The metadata line adds nothing.
|
|
||||||
func addSpamhausNetwork(builder *netipx.IPSetBuilder, line string) error {
|
|
||||||
var entry struct {
|
|
||||||
CIDR string `json:"cidr"`
|
|
||||||
Type string `json:"type"`
|
|
||||||
}
|
|
||||||
|
|
||||||
err := json.Unmarshal([]byte(line), &entry)
|
|
||||||
if err != nil {
|
|
||||||
return err
|
|
||||||
}
|
|
||||||
|
|
||||||
if entry.Type == spamhausMetadataType {
|
|
||||||
return nil
|
|
||||||
}
|
|
||||||
|
|
||||||
network, err := netip.ParsePrefix(entry.CIDR)
|
|
||||||
if err != nil {
|
|
||||||
return err
|
|
||||||
}
|
|
||||||
|
|
||||||
builder.AddPrefix(network)
|
|
||||||
|
|
||||||
return nil
|
|
||||||
}
|
|
||||||
|
|
||||||
// addDShieldNetwork adds the network on a line of a DShield file to builder.
|
|
||||||
// The line's first two fields are the network's first and last address.
|
|
||||||
func addDShieldNetwork(builder *netipx.IPSetBuilder, line string) error {
|
|
||||||
firstField, rest, found := strings.Cut(line, dshieldFieldSeparator)
|
|
||||||
if !found {
|
|
||||||
return errNoNetwork
|
|
||||||
}
|
|
||||||
|
|
||||||
lastField, _, _ := strings.Cut(rest, dshieldFieldSeparator)
|
|
||||||
|
|
||||||
first, err := netip.ParseAddr(firstField)
|
|
||||||
if err != nil {
|
|
||||||
return err
|
|
||||||
}
|
|
||||||
|
|
||||||
last, err := netip.ParseAddr(lastField)
|
|
||||||
if err != nil {
|
|
||||||
return err
|
|
||||||
}
|
|
||||||
|
|
||||||
network := netipx.IPRangeFrom(first, last)
|
|
||||||
if !network.IsValid() {
|
|
||||||
return errBadRange
|
|
||||||
}
|
|
||||||
|
|
||||||
builder.AddRange(network)
|
|
||||||
|
|
||||||
return nil
|
|
||||||
}
|
|
||||||
@@ -148,9 +148,7 @@ func TestLiveCountsTrackWritesInRealtime(t *testing.T) {
|
|||||||
func writeASNsPrefixesAndPeers(t *testing.T, db *Database, ts time.Time) {
|
func writeASNsPrefixesAndPeers(t *testing.T, db *Database, ts time.Time) {
|
||||||
t.Helper()
|
t.Helper()
|
||||||
|
|
||||||
seen := SeenTimes{First: ts, Last: ts}
|
err := db.GetOrCreateASNBatch(map[int]time.Time{64500: ts, 64501: ts})
|
||||||
|
|
||||||
err := db.GetOrCreateASNBatch(map[int]SeenTimes{64500: seen, 64501: seen})
|
|
||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatalf("GetOrCreateASNBatch: %v", err)
|
t.Fatalf("GetOrCreateASNBatch: %v", err)
|
||||||
}
|
}
|
||||||
@@ -166,14 +164,14 @@ func writeASNsPrefixesAndPeers(t *testing.T, db *Database, ts time.Time) {
|
|||||||
err = db.UpdatePeerBatch(map[string]PeerUpdate{
|
err = db.UpdatePeerBatch(map[string]PeerUpdate{
|
||||||
testV4PeerIP: {
|
testV4PeerIP: {
|
||||||
PeerIP: testV4PeerIP, PeerASN: 64500,
|
PeerIP: testV4PeerIP, PeerASN: 64500,
|
||||||
MessageType: "UPDATE", Seen: seen,
|
MessageType: "UPDATE", Timestamp: ts,
|
||||||
},
|
},
|
||||||
})
|
})
|
||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatalf("UpdatePeerBatch: %v", err)
|
t.Fatalf("UpdatePeerBatch: %v", err)
|
||||||
}
|
}
|
||||||
|
|
||||||
err = db.RecordPeering(64500, 64501, seen)
|
err = db.RecordPeering(64500, 64501, ts)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatalf("RecordPeering: %v", err)
|
t.Fatalf("RecordPeering: %v", err)
|
||||||
}
|
}
|
||||||
@@ -194,10 +192,7 @@ func TestLiveCountsSeededFromDatabaseAtStartup(t *testing.T) {
|
|||||||
|
|
||||||
ts := time.Date(2026, 1, 2, 3, 4, 5, 0, time.UTC)
|
ts := time.Date(2026, 1, 2, 3, 4, 5, 0, time.UTC)
|
||||||
|
|
||||||
seen := SeenTimes{First: ts, Last: ts}
|
err = db.GetOrCreateASNBatch(map[int]time.Time{64500: ts, 64501: ts, 64502: ts})
|
||||||
|
|
||||||
err = db.GetOrCreateASNBatch(
|
|
||||||
map[int]SeenTimes{64500: seen, 64501: seen, 64502: seen})
|
|
||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatalf("GetOrCreateASNBatch: %v", err)
|
t.Fatalf("GetOrCreateASNBatch: %v", err)
|
||||||
}
|
}
|
||||||
|
|||||||
+57
-143
@@ -21,7 +21,6 @@ import (
|
|||||||
|
|
||||||
"git.eeqj.de/sneak/routewatch/internal/config"
|
"git.eeqj.de/sneak/routewatch/internal/config"
|
||||||
"git.eeqj.de/sneak/routewatch/internal/logger"
|
"git.eeqj.de/sneak/routewatch/internal/logger"
|
||||||
"git.eeqj.de/sneak/routewatch/internal/ristypes"
|
|
||||||
"git.eeqj.de/sneak/routewatch/pkg/asinfo"
|
"git.eeqj.de/sneak/routewatch/pkg/asinfo"
|
||||||
_ "github.com/mattn/go-sqlite3" // CGO SQLite driver
|
_ "github.com/mattn/go-sqlite3" // CGO SQLite driver
|
||||||
)
|
)
|
||||||
@@ -170,20 +169,6 @@ func New(cfg *config.Config, logger *logger.Logger) (*Database, error) {
|
|||||||
return nil, fmt.Errorf("failed to create database directory: %w", err)
|
return nil, fmt.Errorf("failed to create database directory: %w", err)
|
||||||
}
|
}
|
||||||
|
|
||||||
// The size of the WAL file the previous run left, which Initialize logs
|
|
||||||
// with the start-up checkpoint. It is read before the database is opened,
|
|
||||||
// because opening it writes a frame to the WAL. There is no WAL file when
|
|
||||||
// the database is new or the last run closed the database, which deletes
|
|
||||||
// it.
|
|
||||||
var walBytes int64
|
|
||||||
|
|
||||||
wal, err := os.Stat(dbPath + walFileSuffix)
|
|
||||||
if err == nil {
|
|
||||||
walBytes = wal.Size()
|
|
||||||
} else if !errors.Is(err, os.ErrNotExist) {
|
|
||||||
logger.Warn("Failed to read the WAL file's size on startup", "error", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
// Per-connection SQLite settings go in the DSN so every pooled connection
|
// Per-connection SQLite settings go in the DSN so every pooled connection
|
||||||
// gets them, not just the one that runs the Initialize pragmas. _txlock=
|
// gets them, not just the one that runs the Initialize pragmas. _txlock=
|
||||||
// immediate makes every transaction take the write lock at BEGIN. Without it
|
// immediate makes every transaction take the write lock at BEGIN. Without it
|
||||||
@@ -244,7 +229,7 @@ func New(cfg *config.Config, logger *logger.Logger) (*Database, error) {
|
|||||||
counts: &liveCounts{},
|
counts: &liveCounts{},
|
||||||
}
|
}
|
||||||
|
|
||||||
err = database.Initialize(walBytes)
|
err = database.Initialize()
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return nil, fmt.Errorf("failed to initialize database: %w", err)
|
return nil, fmt.Errorf("failed to initialize database: %w", err)
|
||||||
}
|
}
|
||||||
@@ -260,16 +245,8 @@ func New(cfg *config.Config, logger *logger.Logger) (*Database, error) {
|
|||||||
return database, nil
|
return database, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
// The messages Initialize logs the start-up WAL checkpoint with.
|
// Initialize creates the database schema if it doesn't exist.
|
||||||
const (
|
func (d *Database) Initialize() error {
|
||||||
startupCheckpointMessage = "WAL checkpoint on startup"
|
|
||||||
startupCheckpointIncompleteMessage = "WAL checkpoint on startup incomplete"
|
|
||||||
)
|
|
||||||
|
|
||||||
// Initialize creates the database schema if it doesn't exist. walBytes is the
|
|
||||||
// size of the WAL file the previous run left, which New reads before it opens
|
|
||||||
// the database.
|
|
||||||
func (d *Database) Initialize(walBytes int64) error {
|
|
||||||
// Set SQLite pragmas for performance. Per-connection settings (cache_size,
|
// Set SQLite pragmas for performance. Per-connection settings (cache_size,
|
||||||
// synchronous, busy_timeout, journal_mode) live in the DSN, and New sets the
|
// synchronous, busy_timeout, journal_mode) live in the DSN, and New sets the
|
||||||
// writing connection's cache_size; temp_store is left at its default so
|
// writing connection's cache_size; temp_store is left at its default so
|
||||||
@@ -289,33 +266,19 @@ func (d *Database) Initialize(walBytes int64) error {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// The start-up checkpoint copies the WAL a previous run left into the
|
// Run WAL checkpoint on startup to consolidate any existing WAL
|
||||||
// database file and empties the WAL file; its frame counts then read 0,
|
var walPages, checkpointedPages, movedPages int
|
||||||
// so walBytes is logged with them.
|
|
||||||
var busy, walFrames, checkpointedFrames int
|
|
||||||
|
|
||||||
err := d.writer.QueryRowContext(context.Background(),
|
err := d.writer.QueryRowContext(context.Background(),
|
||||||
"PRAGMA wal_checkpoint(TRUNCATE)",
|
"PRAGMA wal_checkpoint(TRUNCATE)",
|
||||||
).Scan(&busy, &walFrames, &checkpointedFrames)
|
).Scan(&walPages, &checkpointedPages, &movedPages)
|
||||||
|
if err != nil {
|
||||||
switch {
|
|
||||||
case err != nil:
|
|
||||||
d.logger.Warn("Failed to checkpoint WAL on startup", "error", err)
|
d.logger.Warn("Failed to checkpoint WAL on startup", "error", err)
|
||||||
case busy != 0:
|
} else {
|
||||||
// Another process using the database kept the checkpoint from
|
d.logger.Info("WAL checkpoint on startup",
|
||||||
// copying or emptying all of the WAL.
|
"wal_pages", walPages,
|
||||||
d.logger.Warn(startupCheckpointIncompleteMessage,
|
"checkpointed", checkpointedPages,
|
||||||
"wal_bytes", walBytes,
|
"moved", movedPages,
|
||||||
"busy", busy,
|
|
||||||
"wal_frames", walFrames,
|
|
||||||
"checkpointed_frames", checkpointedFrames,
|
|
||||||
)
|
|
||||||
default:
|
|
||||||
d.logger.Info(startupCheckpointMessage,
|
|
||||||
"wal_bytes", walBytes,
|
|
||||||
"busy", busy,
|
|
||||||
"wal_frames", walFrames,
|
|
||||||
"checkpointed_frames", checkpointedFrames,
|
|
||||||
)
|
)
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -892,7 +855,7 @@ func insertPrefixesInTable(
|
|||||||
|
|
||||||
// UpdatePrefixesBatch stores, in a single transaction, the prefixes that are
|
// UpdatePrefixesBatch stores, in a single transaction, the prefixes that are
|
||||||
// not stored yet, each first seen at its time in prefixes: the IPv4 prefixes
|
// not stored yet, each first seen at its time in prefixes: the IPv4 prefixes
|
||||||
// first and then the IPv6 prefixes. A prefix that does not parse is not stored.
|
// first and then the IPv6 prefixes.
|
||||||
func (d *Database) UpdatePrefixesBatch(prefixes map[string]time.Time) error {
|
func (d *Database) UpdatePrefixesBatch(prefixes map[string]time.Time) error {
|
||||||
if len(prefixes) == 0 {
|
if len(prefixes) == 0 {
|
||||||
return nil
|
return nil
|
||||||
@@ -911,11 +874,6 @@ func (d *Database) UpdatePrefixesBatch(prefixes map[string]time.Time) error {
|
|||||||
prefixesV6 := make(map[string]time.Time)
|
prefixesV6 := make(map[string]time.Time)
|
||||||
|
|
||||||
for prefix, timestamp := range prefixes {
|
for prefix, timestamp := range prefixes {
|
||||||
_, err := netip.ParsePrefix(prefix)
|
|
||||||
if err != nil {
|
|
||||||
continue
|
|
||||||
}
|
|
||||||
|
|
||||||
if detectIPVersion(prefix) == ipVersionV4 {
|
if detectIPVersion(prefix) == ipVersionV4 {
|
||||||
prefixesV4[prefix] = timestamp
|
prefixesV4[prefix] = timestamp
|
||||||
} else {
|
} else {
|
||||||
@@ -945,34 +903,13 @@ func (d *Database) UpdatePrefixesBatch(prefixes map[string]time.Time) error {
|
|||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
|
|
||||||
// These statements update a stored AS, peering or peer. Each keeps the earlier
|
// newASN returns a new ASN record first and last seen at timestamp, with the
|
||||||
// of the stored and the new first_seen, and the later of the stored and the new
|
// handle and description from the AS info lookup when it knows the number.
|
||||||
// last_seen. Some collectors send updates days old while they catch up on a
|
func newASN(number int, timestamp time.Time) ASN {
|
||||||
// backlog. MIN and MAX compare the times as text, which is time order here: the
|
|
||||||
// driver writes every time in one format, and every time is UTC.
|
|
||||||
const (
|
|
||||||
updateASNSQL = `UPDATE asns
|
|
||||||
SET first_seen = MIN(first_seen, ?), last_seen = MAX(last_seen, ?)
|
|
||||||
WHERE asn = ?`
|
|
||||||
|
|
||||||
updatePeeringSQL = `UPDATE peerings
|
|
||||||
SET first_seen = MIN(first_seen, ?), last_seen = MAX(last_seen, ?)
|
|
||||||
WHERE as_a = ? AND as_b = ?`
|
|
||||||
|
|
||||||
updatePeerSQL = `UPDATE bgp_peers
|
|
||||||
SET peer_asn = ?, last_message_type = ?,
|
|
||||||
first_seen = MIN(first_seen, ?), last_seen = MAX(last_seen, ?)
|
|
||||||
WHERE peer_ip = ?`
|
|
||||||
)
|
|
||||||
|
|
||||||
// newASN returns a new ASN record first seen at seen.First and last seen at
|
|
||||||
// seen.Last, with the handle and description from the AS info lookup when it
|
|
||||||
// knows the number.
|
|
||||||
func newASN(number int, seen SeenTimes) ASN {
|
|
||||||
asn := ASN{
|
asn := ASN{
|
||||||
ASN: number,
|
ASN: number,
|
||||||
FirstSeen: seen.First,
|
FirstSeen: timestamp,
|
||||||
LastSeen: seen.Last,
|
LastSeen: timestamp,
|
||||||
}
|
}
|
||||||
|
|
||||||
// Look up ASN info
|
// Look up ASN info
|
||||||
@@ -984,11 +921,12 @@ func newASN(number int, seen SeenTimes) ASN {
|
|||||||
return asn
|
return asn
|
||||||
}
|
}
|
||||||
|
|
||||||
// upsertASN stores the seen times of ASN number with the given statements,
|
// upsertASN updates the last_seen time of ASN number with the given
|
||||||
// inserting the ASN when it is new. It reports whether it inserted it.
|
// statements, inserting the ASN when it is new. It reports whether it inserted
|
||||||
|
// it.
|
||||||
func upsertASN(
|
func upsertASN(
|
||||||
ctx context.Context, selectStmt, updateStmt, insertStmt *sql.Stmt,
|
ctx context.Context, selectStmt, updateStmt, insertStmt *sql.Stmt,
|
||||||
number int, seen SeenTimes,
|
number int, timestamp time.Time,
|
||||||
) (bool, error) {
|
) (bool, error) {
|
||||||
var (
|
var (
|
||||||
asn ASN
|
asn ASN
|
||||||
@@ -998,8 +936,8 @@ func upsertASN(
|
|||||||
err := selectStmt.QueryRowContext(ctx, number).Scan(&asn.ASN, &handle,
|
err := selectStmt.QueryRowContext(ctx, number).Scan(&asn.ASN, &handle,
|
||||||
&description, &asn.FirstSeen, &asn.LastSeen)
|
&description, &asn.FirstSeen, &asn.LastSeen)
|
||||||
if err == nil {
|
if err == nil {
|
||||||
// ASN exists, update its seen times
|
// ASN exists, update last_seen
|
||||||
_, err = updateStmt.ExecContext(ctx, seen.First, seen.Last, number)
|
_, err = updateStmt.ExecContext(ctx, timestamp, number)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return false, fmt.Errorf("failed to update ASN %d: %w", number, err)
|
return false, fmt.Errorf("failed to update ASN %d: %w", number, err)
|
||||||
}
|
}
|
||||||
@@ -1012,7 +950,7 @@ func upsertASN(
|
|||||||
}
|
}
|
||||||
|
|
||||||
// ASN doesn't exist, create it
|
// ASN doesn't exist, create it
|
||||||
asn = newASN(number, seen)
|
asn = newASN(number, timestamp)
|
||||||
|
|
||||||
_, err = insertStmt.ExecContext(ctx, asn.ASN, asn.Handle,
|
_, err = insertStmt.ExecContext(ctx, asn.ASN, asn.Handle,
|
||||||
asn.Description, asn.FirstSeen, asn.LastSeen)
|
asn.Description, asn.FirstSeen, asn.LastSeen)
|
||||||
@@ -1024,7 +962,7 @@ func upsertASN(
|
|||||||
}
|
}
|
||||||
|
|
||||||
// GetOrCreateASNBatch creates or updates multiple ASNs in a single transaction
|
// GetOrCreateASNBatch creates or updates multiple ASNs in a single transaction
|
||||||
func (d *Database) GetOrCreateASNBatch(asns map[int]SeenTimes) error {
|
func (d *Database) GetOrCreateASNBatch(asns map[int]time.Time) error {
|
||||||
if len(asns) == 0 {
|
if len(asns) == 0 {
|
||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
@@ -1049,7 +987,8 @@ func (d *Database) GetOrCreateASNBatch(asns map[int]SeenTimes) error {
|
|||||||
}
|
}
|
||||||
defer func() { _ = selectStmt.Close() }()
|
defer func() { _ = selectStmt.Close() }()
|
||||||
|
|
||||||
updateStmt, err := tx.PrepareContext(ctx, updateASNSQL)
|
updateStmt, err := tx.PrepareContext(ctx,
|
||||||
|
"UPDATE asns SET last_seen = ? WHERE asn = ?")
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return fmt.Errorf("failed to prepare update statement: %w", err)
|
return fmt.Errorf("failed to prepare update statement: %w", err)
|
||||||
}
|
}
|
||||||
@@ -1065,9 +1004,9 @@ func (d *Database) GetOrCreateASNBatch(asns map[int]SeenTimes) error {
|
|||||||
|
|
||||||
var newASNs int
|
var newASNs int
|
||||||
|
|
||||||
for number, seen := range asns {
|
for number, timestamp := range asns {
|
||||||
inserted, err := upsertASN(ctx, selectStmt, updateStmt, insertStmt,
|
inserted, err := upsertASN(ctx, selectStmt, updateStmt, insertStmt,
|
||||||
number, seen)
|
number, timestamp)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return err
|
return err
|
||||||
}
|
}
|
||||||
@@ -1108,19 +1047,17 @@ func (d *Database) GetOrCreateASN(number int, timestamp time.Time) (*ASN, error)
|
|||||||
"FROM asns WHERE asn = ?", number).
|
"FROM asns WHERE asn = ?", number).
|
||||||
Scan(&asn.ASN, &handle, &description, &asn.FirstSeen, &asn.LastSeen)
|
Scan(&asn.ASN, &handle, &description, &asn.FirstSeen, &asn.LastSeen)
|
||||||
if err == nil {
|
if err == nil {
|
||||||
// ASN exists, update its seen times
|
// ASN exists, update last_seen
|
||||||
asn.Handle = handle.String
|
asn.Handle = handle.String
|
||||||
asn.Description = description.String
|
asn.Description = description.String
|
||||||
|
|
||||||
_, err = tx.Exec(updateASNSQL, timestamp, timestamp, number)
|
_, err = tx.Exec("UPDATE asns SET last_seen = ? WHERE asn = ?",
|
||||||
|
timestamp, number)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return nil, err
|
return nil, err
|
||||||
}
|
}
|
||||||
|
|
||||||
// Return the seen times the update stored.
|
asn.LastSeen = timestamp
|
||||||
seen := SeenTimes{First: asn.FirstSeen, Last: asn.LastSeen}
|
|
||||||
seen.Add(timestamp)
|
|
||||||
asn.FirstSeen, asn.LastSeen = seen.First, seen.Last
|
|
||||||
|
|
||||||
err = tx.Commit()
|
err = tx.Commit()
|
||||||
if err != nil {
|
if err != nil {
|
||||||
@@ -1138,7 +1075,7 @@ func (d *Database) GetOrCreateASN(number int, timestamp time.Time) (*ASN, error)
|
|||||||
}
|
}
|
||||||
|
|
||||||
// ASN doesn't exist, create it with ASN info lookup
|
// ASN doesn't exist, create it with ASN info lookup
|
||||||
asn = newASN(number, SeenTimes{First: timestamp, Last: timestamp})
|
asn = newASN(number, timestamp)
|
||||||
|
|
||||||
_, err = tx.Exec("INSERT INTO asns "+
|
_, err = tx.Exec("INSERT INTO asns "+
|
||||||
"(asn, handle, description, first_seen, last_seen) "+
|
"(asn, handle, description, first_seen, last_seen) "+
|
||||||
@@ -1245,7 +1182,7 @@ func (d *Database) RecordAnnouncement(announcement *Announcement) error {
|
|||||||
}
|
}
|
||||||
|
|
||||||
// RecordPeering records a peering relationship between two ASNs.
|
// RecordPeering records a peering relationship between two ASNs.
|
||||||
func (d *Database) RecordPeering(asA, asB int, seen SeenTimes) error {
|
func (d *Database) RecordPeering(asA, asB int, timestamp time.Time) error {
|
||||||
// Validate ASNs
|
// Validate ASNs
|
||||||
if asA <= 0 || asB <= 0 {
|
if asA <= 0 || asB <= 0 {
|
||||||
return fmt.Errorf("%w: asA=%d, asB=%d", errInvalidASN, asA, asB)
|
return fmt.Errorf("%w: asA=%d, asB=%d", errInvalidASN, asA, asB)
|
||||||
@@ -1279,12 +1216,14 @@ func (d *Database) RecordPeering(asA, asB int, seen SeenTimes) error {
|
|||||||
}
|
}
|
||||||
|
|
||||||
if exists {
|
if exists {
|
||||||
_, err = tx.Exec(updatePeeringSQL, seen.First, seen.Last, asA, asB)
|
_, err = tx.Exec(
|
||||||
|
"UPDATE peerings SET last_seen = ? WHERE as_a = ? AND as_b = ?",
|
||||||
|
timestamp, asA, asB)
|
||||||
} else {
|
} else {
|
||||||
_, err = tx.Exec(`
|
_, err = tx.Exec(`
|
||||||
INSERT INTO peerings (id, as_a, as_b, first_seen, last_seen)
|
INSERT INTO peerings (id, as_a, as_b, first_seen, last_seen)
|
||||||
VALUES (?, ?, ?, ?, ?)`,
|
VALUES (?, ?, ?, ?, ?)`,
|
||||||
generateUUID().String(), asA, asB, seen.First, seen.Last)
|
generateUUID().String(), asA, asB, timestamp, timestamp)
|
||||||
}
|
}
|
||||||
|
|
||||||
if err != nil {
|
if err != nil {
|
||||||
@@ -1334,7 +1273,9 @@ func (d *Database) UpdatePeerBatch(peers map[string]PeerUpdate) error {
|
|||||||
}
|
}
|
||||||
defer func() { _ = checkStmt.Close() }()
|
defer func() { _ = checkStmt.Close() }()
|
||||||
|
|
||||||
updateStmt, err := tx.PrepareContext(ctx, updatePeerSQL)
|
updateStmt, err := tx.PrepareContext(ctx,
|
||||||
|
"UPDATE bgp_peers SET peer_asn = ?, last_seen = ?, "+
|
||||||
|
"last_message_type = ? WHERE peer_ip = ?")
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return fmt.Errorf("failed to prepare update statement: %w", err)
|
return fmt.Errorf("failed to prepare update statement: %w", err)
|
||||||
}
|
}
|
||||||
@@ -1360,12 +1301,11 @@ func (d *Database) UpdatePeerBatch(peers map[string]PeerUpdate) error {
|
|||||||
|
|
||||||
if exists {
|
if exists {
|
||||||
_, err = updateStmt.ExecContext(ctx, update.PeerASN,
|
_, err = updateStmt.ExecContext(ctx, update.PeerASN,
|
||||||
update.MessageType, update.Seen.First, update.Seen.Last,
|
update.Timestamp, update.MessageType, update.PeerIP)
|
||||||
update.PeerIP)
|
|
||||||
} else {
|
} else {
|
||||||
_, err = insertStmt.ExecContext(ctx,
|
_, err = insertStmt.ExecContext(ctx,
|
||||||
generateUUID().String(), update.PeerIP, update.PeerASN,
|
generateUUID().String(), update.PeerIP, update.PeerASN,
|
||||||
update.Seen.First, update.Seen.Last, update.MessageType)
|
update.Timestamp, update.Timestamp, update.MessageType)
|
||||||
}
|
}
|
||||||
|
|
||||||
if err != nil {
|
if err != nil {
|
||||||
@@ -1410,8 +1350,11 @@ func (d *Database) UpdatePeer(
|
|||||||
}
|
}
|
||||||
|
|
||||||
if exists {
|
if exists {
|
||||||
_, err = tx.Exec(updatePeerSQL,
|
_, err = tx.Exec(
|
||||||
peerASN, messageType, timestamp, timestamp, peerIP)
|
"UPDATE bgp_peers SET peer_asn = ?, last_seen = ?, "+
|
||||||
|
"last_message_type = ? WHERE peer_ip = ?",
|
||||||
|
peerASN, timestamp, messageType, peerIP,
|
||||||
|
)
|
||||||
} else {
|
} else {
|
||||||
_, err = tx.Exec(
|
_, err = tx.Exec(
|
||||||
"INSERT INTO bgp_peers (id, peer_ip, peer_asn, first_seen, "+
|
"INSERT INTO bgp_peers (id, peer_ip, peer_asn, first_seen, "+
|
||||||
@@ -1763,21 +1706,9 @@ func (d *Database) GetASDetails(asn int) (*ASN, []LiveRoute, error) {
|
|||||||
// GetASDetailsContext returns the AS and the prefixes it originates, the IPv4
|
// GetASDetailsContext returns the AS and the prefixes it originates, the IPv4
|
||||||
// ones first, each once with the time of its newest route. It makes two index
|
// ones first, each once with the time of its newest route. It makes two index
|
||||||
// searches per prefix, however many peers carry the prefix.
|
// searches per prefix, however many peers carry the prefix.
|
||||||
//
|
|
||||||
// The routes of an AS can be stored before its record in the asns table. When
|
|
||||||
// the AS has no record, its handle and description come from the AS info
|
|
||||||
// lookup, and it has no seen times. An AS with no record, no lookup entry and
|
|
||||||
// no prefix is ErrNoRoute.
|
|
||||||
//
|
|
||||||
// ristypes.NoOrigin means no known origin, not an AS, so it is ErrNoRoute
|
|
||||||
// whatever is stored.
|
|
||||||
func (d *Database) GetASDetailsContext(
|
func (d *Database) GetASDetailsContext(
|
||||||
ctx context.Context, asn int,
|
ctx context.Context, asn int,
|
||||||
) (*ASN, []LiveRoute, error) {
|
) (*ASN, []LiveRoute, error) {
|
||||||
if asn == ristypes.NoOrigin {
|
|
||||||
return nil, nil, fmt.Errorf("%w: AS%d", ErrNoRoute, asn)
|
|
||||||
}
|
|
||||||
|
|
||||||
// Get AS information
|
// Get AS information
|
||||||
var (
|
var (
|
||||||
asnInfo ASN
|
asnInfo ASN
|
||||||
@@ -1790,28 +1721,17 @@ func (d *Database) GetASDetailsContext(
|
|||||||
asn,
|
asn,
|
||||||
).Scan(&asnInfo.ASN, &handle, &description, &asnInfo.FirstSeen,
|
).Scan(&asnInfo.ASN, &handle, &description, &asnInfo.FirstSeen,
|
||||||
&asnInfo.LastSeen)
|
&asnInfo.LastSeen)
|
||||||
|
if err != nil {
|
||||||
// known is whether the asns table or the AS info lookup holds the AS.
|
if errors.Is(err, sql.ErrNoRows) {
|
||||||
known := true
|
return nil, nil, fmt.Errorf("%w: AS%d", ErrNoRoute, asn)
|
||||||
|
|
||||||
switch {
|
|
||||||
case errors.Is(err, sql.ErrNoRows):
|
|
||||||
asnInfo.ASN = asn
|
|
||||||
|
|
||||||
info, ok := asinfo.Get(asn)
|
|
||||||
if ok {
|
|
||||||
asnInfo.Handle = info.Handle
|
|
||||||
asnInfo.Description = info.Description
|
|
||||||
}
|
}
|
||||||
|
|
||||||
known = ok
|
|
||||||
case err != nil:
|
|
||||||
return nil, nil, fmt.Errorf("failed to query AS: %w", err)
|
return nil, nil, fmt.Errorf("failed to query AS: %w", err)
|
||||||
default:
|
|
||||||
asnInfo.Handle = handle.String
|
|
||||||
asnInfo.Description = description.String
|
|
||||||
}
|
}
|
||||||
|
|
||||||
|
asnInfo.Handle = handle.String
|
||||||
|
asnInfo.Description = description.String
|
||||||
|
|
||||||
allPrefixes, err := d.originPrefixes(ctx, asn, ipVersionV4)
|
allPrefixes, err := d.originPrefixes(ctx, asn, ipVersionV4)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return &asnInfo, nil, err
|
return &asnInfo, nil, err
|
||||||
@@ -1822,13 +1742,7 @@ func (d *Database) GetASDetailsContext(
|
|||||||
return &asnInfo, allPrefixes, err
|
return &asnInfo, allPrefixes, err
|
||||||
}
|
}
|
||||||
|
|
||||||
allPrefixes = append(allPrefixes, prefixesV6...)
|
return &asnInfo, append(allPrefixes, prefixesV6...), nil
|
||||||
|
|
||||||
if !known && len(allPrefixes) == 0 {
|
|
||||||
return nil, nil, fmt.Errorf("%w: AS%d", ErrNoRoute, asn)
|
|
||||||
}
|
|
||||||
|
|
||||||
return &asnInfo, allPrefixes, nil
|
|
||||||
}
|
}
|
||||||
|
|
||||||
// ASPeer represents a peering relationship with another AS including handle,
|
// ASPeer represents a peering relationship with another AS including handle,
|
||||||
|
|||||||
@@ -1,10 +1,8 @@
|
|||||||
package database
|
package database
|
||||||
|
|
||||||
import (
|
import (
|
||||||
"bytes"
|
|
||||||
"context"
|
"context"
|
||||||
"database/sql"
|
"database/sql"
|
||||||
"encoding/json"
|
|
||||||
"errors"
|
"errors"
|
||||||
"fmt"
|
"fmt"
|
||||||
"log/slog"
|
"log/slog"
|
||||||
@@ -20,8 +18,6 @@ import (
|
|||||||
|
|
||||||
"git.eeqj.de/sneak/routewatch/internal/config"
|
"git.eeqj.de/sneak/routewatch/internal/config"
|
||||||
"git.eeqj.de/sneak/routewatch/internal/logger"
|
"git.eeqj.de/sneak/routewatch/internal/logger"
|
||||||
"git.eeqj.de/sneak/routewatch/internal/ristypes"
|
|
||||||
"git.eeqj.de/sneak/routewatch/pkg/asinfo"
|
|
||||||
"github.com/mattn/go-sqlite3"
|
"github.com/mattn/go-sqlite3"
|
||||||
)
|
)
|
||||||
|
|
||||||
@@ -619,12 +615,10 @@ func TestBatchWriteDuringCheckpoint(t *testing.T) {
|
|||||||
})
|
})
|
||||||
|
|
||||||
ts := time.Now().UTC()
|
ts := time.Now().UTC()
|
||||||
seen := SeenTimes{First: ts, Last: ts}
|
|
||||||
|
|
||||||
for i := range contentionIterations {
|
for i := range contentionIterations {
|
||||||
asns := map[int]SeenTimes{
|
asns := map[int]time.Time{
|
||||||
i % contendedASNCount: seen,
|
i % contendedASNCount: ts,
|
||||||
(i % contendedASNCount) + asnSecondBand: seen,
|
(i % contendedASNCount) + asnSecondBand: ts,
|
||||||
}
|
}
|
||||||
|
|
||||||
err := db.GetOrCreateASNBatch(asns)
|
err := db.GetOrCreateASNBatch(asns)
|
||||||
@@ -753,8 +747,7 @@ func TestBatchWriteWaitsForAnalyze(t *testing.T) {
|
|||||||
|
|
||||||
db, messages := newDatabaseWithLockMessages(t)
|
db, messages := newDatabaseWithLockMessages(t)
|
||||||
ctx := context.Background()
|
ctx := context.Background()
|
||||||
now := time.Now().UTC()
|
asns := map[int]time.Time{lockTestASN: time.Now().UTC()}
|
||||||
asns := map[int]SeenTimes{lockTestASN: {First: now, Last: now}}
|
|
||||||
|
|
||||||
// A stored AS gives ANALYZE a table to analyze and statistics to write.
|
// A stored AS gives ANALYZE a table to analyze and statistics to write.
|
||||||
err := db.GetOrCreateASNBatch(asns)
|
err := db.GetOrCreateASNBatch(asns)
|
||||||
@@ -1056,176 +1049,6 @@ func walFileSize(t *testing.T, db *Database) int64 {
|
|||||||
return wal.Size()
|
return wal.Size()
|
||||||
}
|
}
|
||||||
|
|
||||||
// TestStartupCheckpointLogsItsFigures opens a new database, writes a route,
|
|
||||||
// and opens the same file again while the first stays open, so that its WAL
|
|
||||||
// file stays. The first open finds no WAL file: it logs a size of 0 and no
|
|
||||||
// warning. The second logs the size of the WAL file the first left, and the
|
|
||||||
// checkpoint's busy flag and frame counts, each under its own name; the
|
|
||||||
// counts read 0 once the checkpoint has emptied the file.
|
|
||||||
// Initialize logged the busy flag as wal_pages and the counts under each
|
|
||||||
// other's names (https://git.eeqj.de/sneak/routewatch/issues/117).
|
|
||||||
func TestStartupCheckpointLogsItsFigures(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
cfg := &config.Config{StateDir: t.TempDir()}
|
|
||||||
|
|
||||||
db, lines := openLogged(t, cfg)
|
|
||||||
|
|
||||||
for _, line := range lines {
|
|
||||||
if line[slog.LevelKey] == slog.LevelWarn.String() {
|
|
||||||
t.Errorf("opening a new database logged a warning: %v", line)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
line := findLine(t, lines, startupCheckpointMessage)
|
|
||||||
if line["wal_bytes"] != 0.0 {
|
|
||||||
t.Errorf("new database: start-up checkpoint logged wal_bytes = %v, "+
|
|
||||||
"want 0", line["wal_bytes"])
|
|
||||||
}
|
|
||||||
|
|
||||||
err := db.UpsertLiveRouteBatch([]*LiveRoute{
|
|
||||||
mkV4Route(t, "192.0.2.0/24", lockTestASN, time.Now().UTC()),
|
|
||||||
})
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("failed to store a route: %v", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
walBytes := walFileSize(t, db)
|
|
||||||
if walBytes == 0 {
|
|
||||||
t.Fatal("WAL file empty after the route write, want frames in it")
|
|
||||||
}
|
|
||||||
|
|
||||||
_, lines = openLogged(t, cfg)
|
|
||||||
line = findLine(t, lines, startupCheckpointMessage)
|
|
||||||
|
|
||||||
for _, key := range []string{"busy", "wal_frames", "checkpointed_frames"} {
|
|
||||||
if line[key] != 0.0 {
|
|
||||||
t.Errorf("start-up checkpoint logged %s = %v, want 0", key, line[key])
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
if line["wal_bytes"] != float64(walBytes) {
|
|
||||||
t.Errorf("start-up checkpoint logged wal_bytes = %v, want the %d "+
|
|
||||||
"bytes the first database left", line["wal_bytes"], walBytes)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestStartupCheckpointWarnsWhenIncomplete keeps a read open on a database
|
|
||||||
// while a route is written after the read began, then opens the same file
|
|
||||||
// again. The second open's start-up checkpoint cannot copy the frames written
|
|
||||||
// after the read began, and gives up once the busy timeout has passed. It
|
|
||||||
// checks that the checkpoint is logged as a warning with busy 1 and fewer
|
|
||||||
// frames copied than the WAL holds.
|
|
||||||
func TestStartupCheckpointWarnsWhenIncomplete(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
cfg := &config.Config{StateDir: t.TempDir()}
|
|
||||||
db, _ := openLogged(t, cfg)
|
|
||||||
ctx := context.Background()
|
|
||||||
|
|
||||||
reader, err := db.db.Conn(ctx)
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("failed to open the reading connection: %v", err)
|
|
||||||
}
|
|
||||||
defer func() { _ = reader.Close() }()
|
|
||||||
|
|
||||||
// A plain BEGIN starts a transaction that only reads, unlike the BEGIN
|
|
||||||
// IMMEDIATE of _txlock. It keeps the snapshot its first read takes until
|
|
||||||
// it ends.
|
|
||||||
_, err = reader.ExecContext(ctx, "BEGIN")
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("failed to begin the read transaction: %v", err)
|
|
||||||
}
|
|
||||||
defer func() { _, _ = reader.ExecContext(ctx, "ROLLBACK") }()
|
|
||||||
|
|
||||||
var routes int
|
|
||||||
|
|
||||||
err = reader.QueryRowContext(ctx,
|
|
||||||
"SELECT count(*) FROM live_routes_v4").Scan(&routes)
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("failed to read the live routes: %v", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
err = db.UpsertLiveRouteBatch([]*LiveRoute{
|
|
||||||
mkV4Route(t, "192.0.2.0/24", lockTestASN, time.Now().UTC()),
|
|
||||||
})
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("failed to store a route after the read: %v", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
_, lines := openLogged(t, cfg)
|
|
||||||
line := findLine(t, lines, startupCheckpointIncompleteMessage)
|
|
||||||
|
|
||||||
if line[slog.LevelKey] != slog.LevelWarn.String() {
|
|
||||||
t.Errorf("start-up checkpoint logged at level %v, want %s",
|
|
||||||
line[slog.LevelKey], slog.LevelWarn)
|
|
||||||
}
|
|
||||||
|
|
||||||
if line["busy"] != 1.0 {
|
|
||||||
t.Errorf("start-up checkpoint logged busy = %v, want 1", line["busy"])
|
|
||||||
}
|
|
||||||
|
|
||||||
walFrames, _ := line["wal_frames"].(float64)
|
|
||||||
checkpointedFrames, _ := line["checkpointed_frames"].(float64)
|
|
||||||
|
|
||||||
if checkpointedFrames >= walFrames {
|
|
||||||
t.Errorf("start-up checkpoint logged checkpointed_frames = %v and "+
|
|
||||||
"wal_frames = %v with a read open, want fewer copied than the WAL "+
|
|
||||||
"holds", line["checkpointed_frames"], line["wal_frames"])
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// openLogged opens the database of cfg with a logger that writes JSON, and
|
|
||||||
// returns it with the lines it logged while it opened, decoded.
|
|
||||||
func openLogged(t *testing.T, cfg *config.Config) (*Database, []map[string]any) {
|
|
||||||
t.Helper()
|
|
||||||
|
|
||||||
var logged bytes.Buffer
|
|
||||||
|
|
||||||
db, err := New(cfg,
|
|
||||||
&logger.Logger{Logger: slog.New(slog.NewJSONHandler(&logged, nil))})
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("failed to open the database: %v", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
t.Cleanup(func() { _ = db.Close() })
|
|
||||||
|
|
||||||
var lines []map[string]any
|
|
||||||
|
|
||||||
decoder := json.NewDecoder(&logged)
|
|
||||||
for decoder.More() {
|
|
||||||
var line map[string]any
|
|
||||||
|
|
||||||
err := decoder.Decode(&line)
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("failed to decode the log: %v", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
lines = append(lines, line)
|
|
||||||
}
|
|
||||||
|
|
||||||
return db, lines
|
|
||||||
}
|
|
||||||
|
|
||||||
// findLine returns the first of lines logged with message.
|
|
||||||
func findLine(
|
|
||||||
t *testing.T,
|
|
||||||
lines []map[string]any,
|
|
||||||
message string,
|
|
||||||
) map[string]any {
|
|
||||||
t.Helper()
|
|
||||||
|
|
||||||
for _, line := range lines {
|
|
||||||
if line[slog.MessageKey] == message {
|
|
||||||
return line
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
t.Fatalf("no line %q in the log", message)
|
|
||||||
|
|
||||||
return nil
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestGetRandomPrefixesByLengthListsPrefixOnce stores one prefix of each
|
// TestGetRandomPrefixesByLengthListsPrefixOnce stores one prefix of each
|
||||||
// family learned from two peers in the same second and checks that the
|
// family learned from two peers in the same second and checks that the
|
||||||
// prefix-length sample lists each of them once.
|
// prefix-length sample lists each of them once.
|
||||||
@@ -1701,85 +1524,6 @@ func TestGetASDetailsListsEachPrefixOnce(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// TestGetASDetailsOfASNotInDatabase asks for ASes that the database has no
|
|
||||||
// record of, some with stored routes. Each gets the prefixes of its routes and
|
|
||||||
// no seen times. One that the AS info lookup knows gets its handle and
|
|
||||||
// description from it. An AS with no record, no lookup entry and no route is
|
|
||||||
// ErrNoRoute. Before the fix every AS with no record was ErrNoRoute.
|
|
||||||
func TestGetASDetailsOfASNotInDatabase(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
db, err := New(&config.Config{StateDir: t.TempDir()}, logger.New())
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("failed to create database: %v", err)
|
|
||||||
}
|
|
||||||
defer func() { _ = db.Close() }()
|
|
||||||
|
|
||||||
ts := time.Date(2026, 1, 2, 3, 4, 5, 0, time.UTC)
|
|
||||||
|
|
||||||
tests := []struct {
|
|
||||||
asn int
|
|
||||||
listed bool // in the AS info lookup
|
|
||||||
prefixes []string // of the routes stored for the AS
|
|
||||||
}{
|
|
||||||
{15169, true, []string{"192.0.2.0/24"}},
|
|
||||||
{13335, true, nil},
|
|
||||||
{4200000000, false, []string{"198.51.100.0/24"}},
|
|
||||||
}
|
|
||||||
|
|
||||||
var routes []*LiveRoute
|
|
||||||
|
|
||||||
for _, tt := range tests {
|
|
||||||
if _, ok := asinfo.Get(tt.asn); ok != tt.listed {
|
|
||||||
t.Fatalf("AS%d in the AS info data: %t, want %t", tt.asn, ok, tt.listed)
|
|
||||||
}
|
|
||||||
|
|
||||||
for _, prefix := range tt.prefixes {
|
|
||||||
routes = append(routes, mkV4Route(t, prefix, tt.asn, ts))
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
err = db.UpsertLiveRouteBatch(routes)
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("UpsertLiveRouteBatch: %v", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
for _, tt := range tests {
|
|
||||||
got, prefixes, err := db.GetASDetails(tt.asn)
|
|
||||||
if err != nil {
|
|
||||||
t.Errorf("GetASDetails(%d): %v", tt.asn, err)
|
|
||||||
|
|
||||||
continue
|
|
||||||
}
|
|
||||||
|
|
||||||
want := ASN{ASN: tt.asn}
|
|
||||||
if info, ok := asinfo.Get(tt.asn); ok {
|
|
||||||
want.Handle, want.Description = info.Handle, info.Description
|
|
||||||
}
|
|
||||||
|
|
||||||
var gotPrefixes []string
|
|
||||||
for _, route := range prefixes {
|
|
||||||
gotPrefixes = append(gotPrefixes, route.Prefix)
|
|
||||||
}
|
|
||||||
|
|
||||||
if *got != want || !slices.Equal(gotPrefixes, tt.prefixes) {
|
|
||||||
t.Errorf("GetASDetails(%d) = %+v with prefixes %v, want %+v with %v",
|
|
||||||
tt.asn, *got, gotPrefixes, want, tt.prefixes)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
const absentASN = 4200000001 // no record, no lookup entry and no route
|
|
||||||
|
|
||||||
if _, ok := asinfo.Get(absentASN); ok {
|
|
||||||
t.Fatalf("AS%d is in the AS info data", absentASN)
|
|
||||||
}
|
|
||||||
|
|
||||||
_, _, err = db.GetASDetails(absentASN)
|
|
||||||
if !errors.Is(err, ErrNoRoute) {
|
|
||||||
t.Errorf("GetASDetails(%d): error %v, want %v", absentASN, err, ErrNoRoute)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestASPrefixQueriesUseCoveringIndexes checks the plan of each query that
|
// TestASPrefixQueriesUseCoveringIndexes checks the plan of each query that
|
||||||
// lists the prefixes an AS originates: a search of a covering index and no
|
// lists the prefixes an AS originates: a search of a covering index and no
|
||||||
// temporary B-tree, so the list reads one index entry per prefix, not one
|
// temporary B-tree, so the list reads one index entry per prefix, not one
|
||||||
@@ -1919,200 +1663,6 @@ func TestUpdatePrefixesBatchKeepsFirstSeen(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// TestUpdatePeerBatchStoresTheEarliestAndLatestTimesOfANewPeer writes a new
|
|
||||||
// peer whose updates span 5 seconds. Its row is first seen at the earliest time
|
|
||||||
// and last seen at the latest.
|
|
||||||
func TestUpdatePeerBatchStoresTheEarliestAndLatestTimesOfANewPeer(
|
|
||||||
t *testing.T,
|
|
||||||
) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
db, err := New(&config.Config{StateDir: t.TempDir()}, logger.New())
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("failed to create database: %v", err)
|
|
||||||
}
|
|
||||||
defer func() { _ = db.Close() }()
|
|
||||||
|
|
||||||
first := time.Date(2026, 1, 2, 3, 4, 5, 0, time.UTC)
|
|
||||||
last := first.Add(5 * time.Second)
|
|
||||||
|
|
||||||
err = db.UpdatePeerBatch(map[string]PeerUpdate{
|
|
||||||
testPeerIP: {
|
|
||||||
PeerIP: testPeerIP, PeerASN: 64500, MessageType: "UPDATE",
|
|
||||||
Seen: SeenTimes{First: first, Last: last},
|
|
||||||
},
|
|
||||||
})
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("UpdatePeerBatch: %v", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
var firstSeen, lastSeen time.Time
|
|
||||||
|
|
||||||
err = db.db.QueryRowContext(t.Context(),
|
|
||||||
"SELECT first_seen, last_seen FROM bgp_peers WHERE peer_ip = ?",
|
|
||||||
testPeerIP).Scan(&firstSeen, &lastSeen)
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("reading peer %s: %v", testPeerIP, err)
|
|
||||||
}
|
|
||||||
|
|
||||||
if !firstSeen.Equal(first) || !lastSeen.Equal(last) {
|
|
||||||
t.Errorf("peer %s: first seen %s, last seen %s; want %s, %s",
|
|
||||||
testPeerIP, firstSeen, lastSeen, first, last)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// backlogAge is how much older one write is than the other in the tests of
|
|
||||||
// first_seen and last_seen, as when a collector catches up on a backlog.
|
|
||||||
const backlogAge = 24 * time.Hour
|
|
||||||
|
|
||||||
// batchSpan is how much earlier a batch's First is than its Last in the tests
|
|
||||||
// of first_seen and last_seen.
|
|
||||||
const batchSpan = time.Hour
|
|
||||||
|
|
||||||
// TestWritesKeepEarliestFirstSeenAndLatestLastSeen writes a row of each of
|
|
||||||
// asns, peerings and bgp_peers through each function that writes one, at a
|
|
||||||
// newer time and at an older one. Each row takes the older write's First as
|
|
||||||
// first_seen and the newer write's Last as last_seen. The test writes in both
|
|
||||||
// orders, each in a database of its own. Newer then older is a collector
|
|
||||||
// catching up on a backlog; older then newer is a feed that arrives in order.
|
|
||||||
// Each function writes a row of its own. A batch write's First is batchSpan
|
|
||||||
// before its Last, so the test catches the two swapped. A single-row write
|
|
||||||
// takes one time.
|
|
||||||
func TestWritesKeepEarliestFirstSeenAndLatestLastSeen(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
const (
|
|
||||||
batchASN, singleASN = 64500, 64501
|
|
||||||
peerASN = 64502
|
|
||||||
asnQuery = "SELECT first_seen, last_seen FROM asns WHERE asn = ?"
|
|
||||||
peeringQuery = "SELECT first_seen, last_seen FROM peerings " +
|
|
||||||
"WHERE as_a = ? AND as_b = ?"
|
|
||||||
peerQuery = "SELECT first_seen, last_seen FROM bgp_peers " +
|
|
||||||
"WHERE peer_ip = ?"
|
|
||||||
)
|
|
||||||
|
|
||||||
for name, olderFirst := range map[string]bool{
|
|
||||||
"newer then older": false,
|
|
||||||
"older then newer": true,
|
|
||||||
} {
|
|
||||||
t.Run(name, func(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
db, err := New(&config.Config{StateDir: t.TempDir()}, logger.New())
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("failed to create database: %v", err)
|
|
||||||
}
|
|
||||||
defer func() { _ = db.Close() }()
|
|
||||||
|
|
||||||
checkSeenTimes(t, db, olderFirst, func(seen SeenTimes) error {
|
|
||||||
return db.GetOrCreateASNBatch(map[int]SeenTimes{batchASN: seen})
|
|
||||||
}, batchSpan, asnQuery, batchASN)
|
|
||||||
|
|
||||||
checkSeenTimes(t, db, olderFirst, func(seen SeenTimes) error {
|
|
||||||
_, err := db.GetOrCreateASN(singleASN, seen.Last)
|
|
||||||
|
|
||||||
return err
|
|
||||||
}, 0, asnQuery, singleASN)
|
|
||||||
|
|
||||||
checkSeenTimes(t, db, olderFirst, func(seen SeenTimes) error {
|
|
||||||
return db.RecordPeering(batchASN, singleASN, seen)
|
|
||||||
}, batchSpan, peeringQuery, batchASN, singleASN)
|
|
||||||
|
|
||||||
checkSeenTimes(t, db, olderFirst, func(seen SeenTimes) error {
|
|
||||||
return db.UpdatePeerBatch(map[string]PeerUpdate{
|
|
||||||
testV4PeerIP: {
|
|
||||||
PeerIP: testV4PeerIP, PeerASN: peerASN,
|
|
||||||
MessageType: ristypes.MessageTypeUpdate,
|
|
||||||
Seen: seen,
|
|
||||||
},
|
|
||||||
})
|
|
||||||
}, batchSpan, peerQuery, testV4PeerIP)
|
|
||||||
|
|
||||||
checkSeenTimes(t, db, olderFirst, func(seen SeenTimes) error {
|
|
||||||
return db.UpdatePeer(testPeerIP, peerASN,
|
|
||||||
ristypes.MessageTypeUpdate, seen.Last)
|
|
||||||
}, 0, peerQuery, testPeerIP)
|
|
||||||
})
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// checkSeenTimes calls write twice, each time with First span before Last: at
|
|
||||||
// newer times and at times backlogAge older. It writes the older times first
|
|
||||||
// when olderFirst is true. It checks that query, given args, reads the older
|
|
||||||
// write's First as first_seen and the newer write's Last as last_seen.
|
|
||||||
func checkSeenTimes(
|
|
||||||
t *testing.T, db *Database, olderFirst bool,
|
|
||||||
write func(seen SeenTimes) error,
|
|
||||||
span time.Duration, query string, args ...any,
|
|
||||||
) {
|
|
||||||
t.Helper()
|
|
||||||
|
|
||||||
newerLast := time.Date(2026, 1, 2, 3, 4, 5, 0, time.UTC)
|
|
||||||
newer := SeenTimes{First: newerLast.Add(-span), Last: newerLast}
|
|
||||||
older := SeenTimes{
|
|
||||||
First: newer.First.Add(-backlogAge),
|
|
||||||
Last: newer.Last.Add(-backlogAge),
|
|
||||||
}
|
|
||||||
|
|
||||||
writes := []SeenTimes{newer, older}
|
|
||||||
if olderFirst {
|
|
||||||
writes = []SeenTimes{older, newer}
|
|
||||||
}
|
|
||||||
|
|
||||||
for _, seen := range writes {
|
|
||||||
err := write(seen)
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("writing first %s, last %s: %v", seen.First, seen.Last, err)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
var firstSeen, lastSeen time.Time
|
|
||||||
|
|
||||||
err := db.db.QueryRowContext(t.Context(), query, args...).
|
|
||||||
Scan(&firstSeen, &lastSeen)
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("reading the row: %v", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
if !firstSeen.Equal(older.First) || !lastSeen.Equal(newer.Last) {
|
|
||||||
t.Errorf("first seen %s, last seen %s; want %s, %s",
|
|
||||||
firstSeen, lastSeen, older.First, newer.Last)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestGetOrCreateASNReturnsStoredSeenTimes stores an AS at a newer time, then
|
|
||||||
// at an older one. The second call returns the times the row now holds: the
|
|
||||||
// older first_seen and the newer last_seen.
|
|
||||||
func TestGetOrCreateASNReturnsStoredSeenTimes(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
db, err := New(&config.Config{StateDir: t.TempDir()}, logger.New())
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("failed to create database: %v", err)
|
|
||||||
}
|
|
||||||
defer func() { _ = db.Close() }()
|
|
||||||
|
|
||||||
const asn = 64500
|
|
||||||
|
|
||||||
newer := time.Date(2026, 1, 2, 3, 4, 5, 0, time.UTC)
|
|
||||||
older := newer.Add(-backlogAge)
|
|
||||||
|
|
||||||
_, err = db.GetOrCreateASN(asn, newer)
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("GetOrCreateASN at %s: %v", newer, err)
|
|
||||||
}
|
|
||||||
|
|
||||||
got, err := db.GetOrCreateASN(asn, older)
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("GetOrCreateASN at %s: %v", older, err)
|
|
||||||
}
|
|
||||||
|
|
||||||
if !got.FirstSeen.Equal(older) || !got.LastSeen.Equal(newer) {
|
|
||||||
t.Errorf("first seen %s, last seen %s; want %s, %s",
|
|
||||||
got.FirstSeen, got.LastSeen, older, newer)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestUpsertLiveRouteBatchWritesInKeyOrder writes new routes of two peers out
|
// TestUpsertLiveRouteBatchWritesInKeyOrder writes new routes of two peers out
|
||||||
// of key order, the last one a second announcement of the first, and checks
|
// of key order, the last one a second announcement of the first, and checks
|
||||||
// that the rows were inserted in key order, peer then prefix (each new row's id
|
// that the rows were inserted in key order, peer then prefix (each new row's id
|
||||||
|
|||||||
@@ -1,61 +0,0 @@
|
|||||||
package database
|
|
||||||
|
|
||||||
import (
|
|
||||||
"os"
|
|
||||||
"path/filepath"
|
|
||||||
"strings"
|
|
||||||
"testing"
|
|
||||||
|
|
||||||
"git.eeqj.de/sneak/routewatch/internal/config"
|
|
||||||
"git.eeqj.de/sneak/routewatch/internal/logger"
|
|
||||||
)
|
|
||||||
|
|
||||||
// TestDockerignoreExcludesDatabaseFiles checks that .dockerignore keeps the
|
|
||||||
// database file, and the WAL and shared-memory files SQLite writes next to it,
|
|
||||||
// out of the build context wherever they sit in the clone. The image ships the
|
|
||||||
// whole build context as a source archive.
|
|
||||||
func TestDockerignoreExcludesDatabaseFiles(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
db, err := New(&config.Config{StateDir: t.TempDir()}, logger.New())
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("failed to create database: %v", err)
|
|
||||||
}
|
|
||||||
defer func() { _ = db.Close() }()
|
|
||||||
|
|
||||||
data, err := os.ReadFile("../../.dockerignore")
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("failed to read .dockerignore: %v", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
// Docker reads `**/` as any number of directories and the rest of the
|
|
||||||
// pattern as filepath.Match does, so `**/` followed by a pattern that
|
|
||||||
// matches a file's name excludes that file at any depth.
|
|
||||||
var anyDepth []string
|
|
||||||
|
|
||||||
for line := range strings.Lines(string(data)) {
|
|
||||||
pattern, ok := strings.CutPrefix(strings.TrimSpace(line), "**/")
|
|
||||||
if ok {
|
|
||||||
anyDepth = append(anyDepth, pattern)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
name := filepath.Base(db.path)
|
|
||||||
|
|
||||||
for _, file := range []string{name, name + "-wal", name + "-shm"} {
|
|
||||||
excluded := false
|
|
||||||
|
|
||||||
for _, pattern := range anyDepth {
|
|
||||||
matched, err := filepath.Match(pattern, file)
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("bad .dockerignore pattern **/%s: %v", pattern, err)
|
|
||||||
}
|
|
||||||
|
|
||||||
excluded = excluded || matched
|
|
||||||
}
|
|
||||||
|
|
||||||
if !excluded {
|
|
||||||
t.Errorf(".dockerignore has no **/ pattern that excludes %s", file)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
@@ -34,7 +34,7 @@ type Stats struct {
|
|||||||
type Store interface {
|
type Store interface {
|
||||||
// ASN operations
|
// ASN operations
|
||||||
GetOrCreateASN(number int, timestamp time.Time) (*ASN, error)
|
GetOrCreateASN(number int, timestamp time.Time) (*ASN, error)
|
||||||
GetOrCreateASNBatch(asns map[int]SeenTimes) error
|
GetOrCreateASNBatch(asns map[int]time.Time) error
|
||||||
|
|
||||||
// Prefix operations
|
// Prefix operations
|
||||||
GetOrCreatePrefix(prefix string, timestamp time.Time) (*Prefix, error)
|
GetOrCreatePrefix(prefix string, timestamp time.Time) (*Prefix, error)
|
||||||
@@ -44,7 +44,7 @@ type Store interface {
|
|||||||
RecordAnnouncement(announcement *Announcement) error
|
RecordAnnouncement(announcement *Announcement) error
|
||||||
|
|
||||||
// Peering operations
|
// Peering operations
|
||||||
RecordPeering(asA, asB int, seen SeenTimes) error
|
RecordPeering(asA, asB int, timestamp time.Time) error
|
||||||
|
|
||||||
// Statistics
|
// Statistics
|
||||||
GetStats() (Stats, error)
|
GetStats() (Stats, error)
|
||||||
|
|||||||
+11
-34
@@ -28,10 +28,9 @@ type ASN struct {
|
|||||||
RIRRegDate *time.Time `json:"rir_registration_date,omitempty"`
|
RIRRegDate *time.Time `json:"rir_registration_date,omitempty"`
|
||||||
RIRLastMod *time.Time `json:"rir_last_modified,omitempty"`
|
RIRLastMod *time.Time `json:"rir_last_modified,omitempty"`
|
||||||
WHOISRaw string `json:"whois_raw,omitempty"`
|
WHOISRaw string `json:"whois_raw,omitempty"`
|
||||||
// Timestamps. They are zero, and left out of JSON, for an AS that has no
|
// Timestamps
|
||||||
// record in the asns table.
|
FirstSeen time.Time `json:"first_seen"`
|
||||||
FirstSeen time.Time `json:"first_seen,omitzero"`
|
LastSeen time.Time `json:"last_seen"`
|
||||||
LastSeen time.Time `json:"last_seen,omitzero"`
|
|
||||||
WHOISUpdatedAt *time.Time `json:"whois_updated_at,omitempty"`
|
WHOISUpdatedAt *time.Time `json:"whois_updated_at,omitempty"`
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -92,11 +91,12 @@ type PrefixDistribution struct {
|
|||||||
//
|
//
|
||||||
//nolint:tagliatelle // field names are the existing JSON API
|
//nolint:tagliatelle // field names are the existing JSON API
|
||||||
type IPInfo struct {
|
type IPInfo struct {
|
||||||
IP string `json:"ip"`
|
IP string `json:"ip"`
|
||||||
Netblock string `json:"netblock"`
|
PTR []string `json:"ptr,omitempty"`
|
||||||
MaskLength int `json:"mask_length"`
|
Netblock string `json:"netblock"`
|
||||||
IPVersion int `json:"ip_version"`
|
MaskLength int `json:"mask_length"`
|
||||||
NumPeers int `json:"num_peers"`
|
IPVersion int `json:"ip_version"`
|
||||||
|
NumPeers int `json:"num_peers"`
|
||||||
// AS information
|
// AS information
|
||||||
ASN int `json:"asn"`
|
ASN int `json:"asn"`
|
||||||
ASName string `json:"as_name,omitempty"`
|
ASName string `json:"as_name,omitempty"`
|
||||||
@@ -122,35 +122,12 @@ type LiveRouteDeletion struct {
|
|||||||
IPVersion int
|
IPVersion int
|
||||||
}
|
}
|
||||||
|
|
||||||
// SeenTimes is the earliest and the latest time one batch of updates saw an AS,
|
// PeerUpdate represents parameters for updating a peer
|
||||||
// a peer or a peering. A row the batch adds is stored first seen at First and
|
|
||||||
// last seen at Last. A row already stored takes First as its first_seen only
|
|
||||||
// when First is earlier, and Last as its last_seen only when Last is later.
|
|
||||||
type SeenTimes struct {
|
|
||||||
First time.Time
|
|
||||||
Last time.Time
|
|
||||||
}
|
|
||||||
|
|
||||||
// Add makes t the earliest time of s when it is earlier than First, and the
|
|
||||||
// latest when it is later than Last. The zero SeenTimes holds no time yet, so
|
|
||||||
// the first Add sets both.
|
|
||||||
func (s *SeenTimes) Add(t time.Time) {
|
|
||||||
if s.First.IsZero() || t.Before(s.First) {
|
|
||||||
s.First = t
|
|
||||||
}
|
|
||||||
|
|
||||||
if s.Last.IsZero() || t.After(s.Last) {
|
|
||||||
s.Last = t
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// PeerUpdate represents parameters for updating a peer. PeerASN and
|
|
||||||
// MessageType are those of the peer's latest update.
|
|
||||||
type PeerUpdate struct {
|
type PeerUpdate struct {
|
||||||
PeerIP string
|
PeerIP string
|
||||||
PeerASN int
|
PeerASN int
|
||||||
MessageType string
|
MessageType string
|
||||||
Seen SeenTimes
|
Timestamp time.Time
|
||||||
}
|
}
|
||||||
|
|
||||||
// ASNWHOISUpdate contains WHOIS data for updating an ASN record.
|
// ASNWHOISUpdate contains WHOIS data for updating an ASN record.
|
||||||
|
|||||||
@@ -34,7 +34,7 @@ type Tracker struct {
|
|||||||
// Announcement/withdrawal metrics
|
// Announcement/withdrawal metrics
|
||||||
announcementCounter metrics.Counter
|
announcementCounter metrics.Counter
|
||||||
withdrawalCounter metrics.Counter
|
withdrawalCounter metrics.Counter
|
||||||
churnPerMinute minuteCounter // prefix announcements and withdrawals
|
churnRate metrics.Meter // combined announcements + withdrawals per second
|
||||||
|
|
||||||
// BGP peer tracking
|
// BGP peer tracking
|
||||||
bgpPeerCount atomic.Int32
|
bgpPeerCount atomic.Int32
|
||||||
@@ -56,7 +56,7 @@ func New() *Tracker {
|
|||||||
ipv6UpdateRate: metrics.NewMeter(),
|
ipv6UpdateRate: metrics.NewMeter(),
|
||||||
announcementCounter: metrics.NewCounter(),
|
announcementCounter: metrics.NewCounter(),
|
||||||
withdrawalCounter: metrics.NewCounter(),
|
withdrawalCounter: metrics.NewCounter(),
|
||||||
churnPerMinute: minuteCounter{now: time.Now},
|
churnRate: metrics.NewMeter(),
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -156,13 +156,13 @@ func (t *Tracker) RecordIPv6Update() {
|
|||||||
// RecordAnnouncement records a route announcement
|
// RecordAnnouncement records a route announcement
|
||||||
func (t *Tracker) RecordAnnouncement() {
|
func (t *Tracker) RecordAnnouncement() {
|
||||||
t.announcementCounter.Inc(1)
|
t.announcementCounter.Inc(1)
|
||||||
t.churnPerMinute.add()
|
t.churnRate.Mark(1)
|
||||||
}
|
}
|
||||||
|
|
||||||
// RecordWithdrawal records a route withdrawal
|
// RecordWithdrawal records a route withdrawal
|
||||||
func (t *Tracker) RecordWithdrawal() {
|
func (t *Tracker) RecordWithdrawal() {
|
||||||
t.withdrawalCounter.Inc(1)
|
t.withdrawalCounter.Inc(1)
|
||||||
t.churnPerMinute.add()
|
t.churnRate.Mark(1)
|
||||||
}
|
}
|
||||||
|
|
||||||
// SetBGPPeerCount updates the current BGP peer count
|
// SetBGPPeerCount updates the current BGP peer count
|
||||||
@@ -198,10 +198,9 @@ func (t *Tracker) GetWithdrawalCount() uint64 {
|
|||||||
return uint64(count)
|
return uint64(count)
|
||||||
}
|
}
|
||||||
|
|
||||||
// GetChurnPerMinute returns the number of prefix announcements and
|
// GetChurnRate returns the route churn rate per second
|
||||||
// withdrawals, from all peers, in the last full clock minute.
|
func (t *Tracker) GetChurnRate() float64 {
|
||||||
func (t *Tracker) GetChurnPerMinute() int {
|
return t.churnRate.Rate1()
|
||||||
return t.churnPerMinute.lastFullMinute()
|
|
||||||
}
|
}
|
||||||
|
|
||||||
// GetRouteMetrics returns current route update metrics
|
// GetRouteMetrics returns current route update metrics
|
||||||
|
|||||||
@@ -1,59 +0,0 @@
|
|||||||
package metrics
|
|
||||||
|
|
||||||
import (
|
|
||||||
"sync"
|
|
||||||
"time"
|
|
||||||
)
|
|
||||||
|
|
||||||
// minuteCounter counts events in each clock minute and reports the count of
|
|
||||||
// the last full minute. It is safe for concurrent use.
|
|
||||||
type minuteCounter struct {
|
|
||||||
mu sync.Mutex
|
|
||||||
now func() time.Time
|
|
||||||
|
|
||||||
// minute is the start of the minute that count belongs to.
|
|
||||||
minute time.Time
|
|
||||||
count int
|
|
||||||
// lastMinuteCount is the count of the minute just before minute.
|
|
||||||
lastMinuteCount int
|
|
||||||
}
|
|
||||||
|
|
||||||
// add counts one event in the current minute.
|
|
||||||
func (c *minuteCounter) add() {
|
|
||||||
c.mu.Lock()
|
|
||||||
defer c.mu.Unlock()
|
|
||||||
|
|
||||||
c.moveToCurrentMinute()
|
|
||||||
c.count++
|
|
||||||
}
|
|
||||||
|
|
||||||
// lastFullMinute returns the number of events in the minute that ended most
|
|
||||||
// recently.
|
|
||||||
func (c *minuteCounter) lastFullMinute() int {
|
|
||||||
c.mu.Lock()
|
|
||||||
defer c.mu.Unlock()
|
|
||||||
|
|
||||||
c.moveToCurrentMinute()
|
|
||||||
|
|
||||||
return c.lastMinuteCount
|
|
||||||
}
|
|
||||||
|
|
||||||
// moveToCurrentMinute starts a new count when the clock has moved into
|
|
||||||
// another minute. The count it ends becomes the last full minute's count when
|
|
||||||
// its minute is the one just before the current minute. Otherwise the last
|
|
||||||
// full minute had no events. The caller holds mu.
|
|
||||||
func (c *minuteCounter) moveToCurrentMinute() {
|
|
||||||
minute := c.now().Truncate(time.Minute)
|
|
||||||
if minute.Equal(c.minute) {
|
|
||||||
return
|
|
||||||
}
|
|
||||||
|
|
||||||
if minute.Sub(c.minute) == time.Minute {
|
|
||||||
c.lastMinuteCount = c.count
|
|
||||||
} else {
|
|
||||||
c.lastMinuteCount = 0
|
|
||||||
}
|
|
||||||
|
|
||||||
c.minute = minute
|
|
||||||
c.count = 0
|
|
||||||
}
|
|
||||||
@@ -1,86 +0,0 @@
|
|||||||
package metrics
|
|
||||||
|
|
||||||
import (
|
|
||||||
"sync"
|
|
||||||
"testing"
|
|
||||||
"time"
|
|
||||||
)
|
|
||||||
|
|
||||||
// TestChurnPerMinuteCountsLastFullMinute announces and withdraws prefixes on
|
|
||||||
// an injected clock and expects each minute's whole count once that minute
|
|
||||||
// has ended, and 0 for a minute with none.
|
|
||||||
func TestChurnPerMinuteCountsLastFullMinute(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
now := time.Date(2026, 10, 26, 12, 0, 30, 0, time.UTC)
|
|
||||||
tracker := New()
|
|
||||||
tracker.churnPerMinute.now = func() time.Time { return now }
|
|
||||||
|
|
||||||
expectChurn := func(want int, why string) {
|
|
||||||
t.Helper()
|
|
||||||
|
|
||||||
got := tracker.GetChurnPerMinute()
|
|
||||||
if got != want {
|
|
||||||
t.Fatalf("at %s churn = %d, want %d: %s",
|
|
||||||
now.Format(time.TimeOnly), got, want, why)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
const firstMinuteChurn = 2
|
|
||||||
|
|
||||||
tracker.RecordAnnouncement()
|
|
||||||
tracker.RecordWithdrawal()
|
|
||||||
expectChurn(0, "12:00 has not ended")
|
|
||||||
|
|
||||||
now = time.Date(2026, 10, 26, 12, 1, 59, 0, time.UTC)
|
|
||||||
|
|
||||||
tracker.RecordAnnouncement()
|
|
||||||
expectChurn(firstMinuteChurn, "12:00 had one announcement and one withdrawal")
|
|
||||||
|
|
||||||
now = time.Date(2026, 10, 26, 12, 2, 0, 0, time.UTC)
|
|
||||||
|
|
||||||
expectChurn(1, "12:01 had one announcement")
|
|
||||||
|
|
||||||
now = time.Date(2026, 10, 26, 12, 4, 0, 0, time.UTC)
|
|
||||||
|
|
||||||
expectChurn(0, "12:03 had none")
|
|
||||||
|
|
||||||
tracker.RecordWithdrawal()
|
|
||||||
|
|
||||||
now = time.Date(2026, 10, 26, 12, 6, 0, 0, time.UTC)
|
|
||||||
|
|
||||||
expectChurn(0, "12:05 had none, though 12:04 had one withdrawal")
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestChurnPerMinuteCountsConcurrentRecords announces prefixes from several
|
|
||||||
// goroutines at once and expects every one counted.
|
|
||||||
func TestChurnPerMinuteCountsConcurrentRecords(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
const (
|
|
||||||
goroutines = 8
|
|
||||||
recordsPerGoroutine = 1000
|
|
||||||
)
|
|
||||||
|
|
||||||
now := time.Date(2026, 10, 26, 12, 0, 0, 0, time.UTC)
|
|
||||||
tracker := New()
|
|
||||||
tracker.churnPerMinute.now = func() time.Time { return now }
|
|
||||||
|
|
||||||
var wg sync.WaitGroup
|
|
||||||
for range goroutines {
|
|
||||||
wg.Go(func() {
|
|
||||||
for range recordsPerGoroutine {
|
|
||||||
tracker.RecordAnnouncement()
|
|
||||||
}
|
|
||||||
})
|
|
||||||
}
|
|
||||||
|
|
||||||
wg.Wait()
|
|
||||||
|
|
||||||
now = now.Add(time.Minute)
|
|
||||||
|
|
||||||
want := goroutines * recordsPerGoroutine
|
|
||||||
if got := tracker.GetChurnPerMinute(); got != want {
|
|
||||||
t.Fatalf("churn = %d, want %d", got, want)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
+11
-33
@@ -4,7 +4,6 @@ package ristypes
|
|||||||
|
|
||||||
import (
|
import (
|
||||||
"encoding/json"
|
"encoding/json"
|
||||||
"slices"
|
|
||||||
"time"
|
"time"
|
||||||
)
|
)
|
||||||
|
|
||||||
@@ -25,8 +24,7 @@ const (
|
|||||||
)
|
)
|
||||||
|
|
||||||
// ASPath is a BGP AS path as the feed writes it: an array of AS numbers in
|
// ASPath is a BGP AS path as the feed writes it: an array of AS numbers in
|
||||||
// which an AS set is an array of its members, such as [1, 2, [3, 4]]. A path
|
// which an AS set is an array of its members, such as [1, 2, [3, 4]].
|
||||||
// that holds AS 0 decodes as an empty one (UnmarshalJSON).
|
|
||||||
type ASPath struct {
|
type ASPath struct {
|
||||||
// Sequence is the path's AS numbers in order. A set of one member that
|
// Sequence is the path's AS numbers in order. A set of one member that
|
||||||
// ends the path counts as that AS and is its last element. The members of
|
// ends the path counts as that AS and is its last element. The members of
|
||||||
@@ -39,56 +37,34 @@ type ASPath struct {
|
|||||||
Set []int
|
Set []int
|
||||||
}
|
}
|
||||||
|
|
||||||
// NoOrigin is the origin AS of a route whose origin is not known. AS 0 is
|
|
||||||
// reserved, so no AS has this number.
|
|
||||||
const NoOrigin = 0
|
|
||||||
|
|
||||||
// Origin returns the AS that originated the route: the last AS of the
|
// Origin returns the AS that originated the route: the last AS of the
|
||||||
// sequence, or NoOrigin when the path ends in a set of several members or is
|
// sequence, or 0 (none) when the path ends in a set of several members or is
|
||||||
// empty.
|
// empty.
|
||||||
func (p *ASPath) Origin() int {
|
func (p *ASPath) Origin() int {
|
||||||
if len(p.Set) > 0 || len(p.Sequence) == 0 {
|
if len(p.Set) > 0 || len(p.Sequence) == 0 {
|
||||||
return NoOrigin
|
return 0
|
||||||
}
|
}
|
||||||
|
|
||||||
return p.Sequence[len(p.Sequence)-1]
|
return p.Sequence[len(p.Sequence)-1]
|
||||||
}
|
}
|
||||||
|
|
||||||
// UnmarshalJSON implements the json.Unmarshaler interface for ASPath. A path
|
// UnmarshalJSON implements the json.Unmarshaler interface for ASPath.
|
||||||
// that holds AS 0 anywhere, a member of an AS set included, decodes as an empty
|
|
||||||
// path: RFC 7607 makes such a path malformed and its announcements
|
|
||||||
// withdrawals. On an announcement with an empty path the prefix handler
|
|
||||||
// deletes the peer's route, and the AS and peering handlers record nothing.
|
|
||||||
func (p *ASPath) UnmarshalJSON(data []byte) error {
|
func (p *ASPath) UnmarshalJSON(data []byte) error {
|
||||||
path, err := decodeASPath(data)
|
|
||||||
if err != nil {
|
|
||||||
return err
|
|
||||||
}
|
|
||||||
|
|
||||||
if slices.Contains(path.Sequence, 0) || slices.Contains(path.Set, 0) {
|
|
||||||
path = ASPath{}
|
|
||||||
}
|
|
||||||
|
|
||||||
*p = path
|
|
||||||
|
|
||||||
return nil
|
|
||||||
}
|
|
||||||
|
|
||||||
// decodeASPath decodes an AS path as the feed writes it.
|
|
||||||
func decodeASPath(data []byte) (ASPath, error) {
|
|
||||||
// Most paths have no AS set and decode as a plain array of integers.
|
// Most paths have no AS set and decode as a plain array of integers.
|
||||||
var sequence []int
|
var sequence []int
|
||||||
|
|
||||||
err := json.Unmarshal(data, &sequence)
|
err := json.Unmarshal(data, &sequence)
|
||||||
if err == nil {
|
if err == nil {
|
||||||
return ASPath{Sequence: sequence}, nil
|
*p = ASPath{Sequence: sequence}
|
||||||
|
|
||||||
|
return nil
|
||||||
}
|
}
|
||||||
|
|
||||||
var raw []any
|
var raw []any
|
||||||
|
|
||||||
err = json.Unmarshal(data, &raw)
|
err = json.Unmarshal(data, &raw)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return ASPath{}, err
|
return err
|
||||||
}
|
}
|
||||||
|
|
||||||
path := ASPath{Sequence: make([]int, 0, len(raw))}
|
path := ASPath{Sequence: make([]int, 0, len(raw))}
|
||||||
@@ -114,7 +90,9 @@ func decodeASPath(data []byte) (ASPath, error) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
return path, nil
|
*p = path
|
||||||
|
|
||||||
|
return nil
|
||||||
}
|
}
|
||||||
|
|
||||||
// RISLiveMessage represents the outer wrapper message from the RIPE RIS Live stream.
|
// RISLiveMessage represents the outer wrapper message from the RIPE RIS Live stream.
|
||||||
|
|||||||
@@ -145,33 +145,3 @@ func TestASPathKeepsATrailingSetApart(t *testing.T) {
|
|||||||
})
|
})
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// TestASPathWithASZeroDecodesAsAnEmptyPath decodes AS paths that hold AS 0
|
|
||||||
// before their end, at their end, or in an AS set. RFC 7607 makes such a path
|
|
||||||
// malformed, so each decodes as an empty path.
|
|
||||||
func TestASPathWithASZeroDecodesAsAnEmptyPath(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
for _, path := range []string{
|
|
||||||
"[64496, 0, 64497]",
|
|
||||||
"[64496, 0]",
|
|
||||||
"[64496, [0, 64497], 64498]",
|
|
||||||
"[64496, [64497, 0]]",
|
|
||||||
} {
|
|
||||||
t.Run(path, func(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
var got ristypes.ASPath
|
|
||||||
|
|
||||||
err := json.Unmarshal([]byte(path), &got)
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("unmarshal %s: %v", path, err)
|
|
||||||
}
|
|
||||||
|
|
||||||
if len(got.Sequence) > 0 || len(got.Set) > 0 {
|
|
||||||
t.Errorf("%s decoded to sequence %v and set %v, want an empty path",
|
|
||||||
path, got.Sequence, got.Set)
|
|
||||||
}
|
|
||||||
})
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|||||||
+10
-35
@@ -8,11 +8,9 @@ package routewatch
|
|||||||
import (
|
import (
|
||||||
"context"
|
"context"
|
||||||
"errors"
|
"errors"
|
||||||
"path/filepath"
|
|
||||||
"sync"
|
"sync"
|
||||||
"time"
|
"time"
|
||||||
|
|
||||||
"git.eeqj.de/sneak/routewatch/internal/blocklist"
|
|
||||||
"git.eeqj.de/sneak/routewatch/internal/config"
|
"git.eeqj.de/sneak/routewatch/internal/config"
|
||||||
"git.eeqj.de/sneak/routewatch/internal/database"
|
"git.eeqj.de/sneak/routewatch/internal/database"
|
||||||
"git.eeqj.de/sneak/routewatch/internal/logger"
|
"git.eeqj.de/sneak/routewatch/internal/logger"
|
||||||
@@ -54,13 +52,8 @@ type RouteWatch struct {
|
|||||||
peeringHandler *PeeringHandler
|
peeringHandler *PeeringHandler
|
||||||
asnFetcher *ASNFetcher
|
asnFetcher *ASNFetcher
|
||||||
dbMaintainer *DBMaintainer
|
dbMaintainer *DBMaintainer
|
||||||
blocklists *blocklist.Blocklists
|
|
||||||
}
|
}
|
||||||
|
|
||||||
// blocklistsDirName is the directory in the state directory that holds the
|
|
||||||
// blocklist copies.
|
|
||||||
const blocklistsDirName = "blocklists"
|
|
||||||
|
|
||||||
// New creates a new RouteWatch instance
|
// New creates a new RouteWatch instance
|
||||||
func New(deps Dependencies) *RouteWatch {
|
func New(deps Dependencies) *RouteWatch {
|
||||||
rw := &RouteWatch{
|
rw := &RouteWatch{
|
||||||
@@ -123,30 +116,17 @@ func (rw *RouteWatch) Run(ctx context.Context) error {
|
|||||||
return err
|
return err
|
||||||
}
|
}
|
||||||
|
|
||||||
// Start ASN WHOIS fetcher for background updates. The HTTP handlers read
|
|
||||||
// the server's fetcher without a lock, so it is set before they run.
|
|
||||||
rw.asnFetcher = NewASNFetcher(rw.db, rw.logger.Logger)
|
|
||||||
rw.asnFetcher.Start(ctx)
|
|
||||||
rw.server.SetASNFetcher(rw.asnFetcher)
|
|
||||||
|
|
||||||
// Load the blocklist copies from disk and start downloading the lists.
|
|
||||||
// The HTTP handlers read the server's blocklists without a lock, so they
|
|
||||||
// are set before the handlers run.
|
|
||||||
blocklists := blocklist.New(blocklist.Params{
|
|
||||||
Dir: filepath.Join(rw.config.StateDir, blocklistsDirName),
|
|
||||||
Lists: blocklist.Lists(),
|
|
||||||
Logger: rw.logger.Logger,
|
|
||||||
})
|
|
||||||
blocklists.Start(ctx)
|
|
||||||
rw.blocklists = blocklists
|
|
||||||
rw.server.SetBlocklists(blocklists)
|
|
||||||
|
|
||||||
// Start HTTP server
|
// Start HTTP server
|
||||||
err = rw.server.Start(ctx)
|
err = rw.server.Start(ctx)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return err
|
return err
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// Start ASN WHOIS fetcher for background updates
|
||||||
|
rw.asnFetcher = NewASNFetcher(rw.db, rw.logger.Logger)
|
||||||
|
rw.asnFetcher.Start(ctx)
|
||||||
|
rw.server.SetASNFetcher(rw.asnFetcher)
|
||||||
|
|
||||||
// Start database maintenance goroutine
|
// Start database maintenance goroutine
|
||||||
rw.dbMaintainer = NewDBMaintainer(rw.db, rw.logger.Logger)
|
rw.dbMaintainer = NewDBMaintainer(rw.db, rw.logger.Logger)
|
||||||
rw.dbMaintainer.Start(ctx)
|
rw.dbMaintainer.Start(ctx)
|
||||||
@@ -158,12 +138,11 @@ func (rw *RouteWatch) Run(ctx context.Context) error {
|
|||||||
}
|
}
|
||||||
|
|
||||||
// Shutdown stops all services in order: the streamer, the handlers, which
|
// Shutdown stops all services in order: the streamer, the handlers, which
|
||||||
// write what they hold, the WHOIS fetcher, the database maintainer, the
|
// write what they hold, the WHOIS fetcher, the database maintainer and the
|
||||||
// blocklist downloads and the HTTP server, and last the database, which it
|
// HTTP server, and last the database, which it checkpoints and closes. Each
|
||||||
// checkpoints and closes. Each stage waits at most until ctx ends, so
|
// stage waits at most until ctx ends, so Shutdown returns soon after: a
|
||||||
// Shutdown returns soon after: a handler logs what it could not write by
|
// handler logs what it could not write by then, and the HTTP server closes
|
||||||
// then, and the HTTP server closes the connections of requests still
|
// the connections of requests still running.
|
||||||
// running.
|
|
||||||
func (rw *RouteWatch) Shutdown(ctx context.Context) {
|
func (rw *RouteWatch) Shutdown(ctx context.Context) {
|
||||||
rw.mu.Lock()
|
rw.mu.Lock()
|
||||||
if rw.shutdown {
|
if rw.shutdown {
|
||||||
@@ -210,10 +189,6 @@ func (rw *RouteWatch) Shutdown(ctx context.Context) {
|
|||||||
rw.dbMaintainer.Stop(ctx)
|
rw.dbMaintainer.Stop(ctx)
|
||||||
}
|
}
|
||||||
|
|
||||||
if rw.blocklists != nil {
|
|
||||||
rw.blocklists.Stop(ctx)
|
|
||||||
}
|
|
||||||
|
|
||||||
err := rw.server.Stop(ctx)
|
err := rw.server.Stop(ctx)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
rw.logger.Error("Stopped the HTTP server before every request finished",
|
rw.logger.Error("Stopped the HTTP server before every request finished",
|
||||||
|
|||||||
@@ -152,7 +152,7 @@ func (m *mockStore) RecordAnnouncement(_ *database.Announcement) error {
|
|||||||
}
|
}
|
||||||
|
|
||||||
// RecordPeering mock implementation
|
// RecordPeering mock implementation
|
||||||
func (m *mockStore) RecordPeering(asA, asB int, _ database.SeenTimes) error {
|
func (m *mockStore) RecordPeering(asA, asB int, _ time.Time) error {
|
||||||
m.mu.Lock()
|
m.mu.Lock()
|
||||||
defer m.mu.Unlock()
|
defer m.mu.Unlock()
|
||||||
|
|
||||||
@@ -396,16 +396,16 @@ func (m *mockStore) DeleteLiveRoutesOfPeer(_ string) (int, error) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
// GetOrCreateASNBatch mock implementation
|
// GetOrCreateASNBatch mock implementation
|
||||||
func (m *mockStore) GetOrCreateASNBatch(asns map[int]database.SeenTimes) error {
|
func (m *mockStore) GetOrCreateASNBatch(asns map[int]time.Time) error {
|
||||||
m.mu.Lock()
|
m.mu.Lock()
|
||||||
defer m.mu.Unlock()
|
defer m.mu.Unlock()
|
||||||
|
|
||||||
for number, seen := range asns {
|
for number, timestamp := range asns {
|
||||||
if _, exists := m.ASNs[number]; !exists {
|
if _, exists := m.ASNs[number]; !exists {
|
||||||
m.ASNs[number] = &database.ASN{
|
m.ASNs[number] = &database.ASN{
|
||||||
ASN: number,
|
ASN: number,
|
||||||
FirstSeen: seen.First,
|
FirstSeen: timestamp,
|
||||||
LastSeen: seen.Last,
|
LastSeen: timestamp,
|
||||||
}
|
}
|
||||||
m.ASNCount++
|
m.ASNCount++
|
||||||
}
|
}
|
||||||
@@ -466,9 +466,9 @@ func TestRouteWatchLiveFeed(t *testing.T) {
|
|||||||
// Create streamer
|
// Create streamer
|
||||||
s := streamer.New(logger, metricsTracker)
|
s := streamer.New(logger, metricsTracker)
|
||||||
|
|
||||||
// The state directory holds the blocklist copies Run downloads.
|
// Create test config with empty state dir (no snapshot loading)
|
||||||
cfg := &config.Config{
|
cfg := &config.Config{
|
||||||
StateDir: t.TempDir(),
|
StateDir: "",
|
||||||
MaxRuntime: 5 * time.Second,
|
MaxRuntime: 5 * time.Second,
|
||||||
EnableBatchedDatabaseWrites: true,
|
EnableBatchedDatabaseWrites: true,
|
||||||
}
|
}
|
||||||
@@ -766,9 +766,7 @@ const (
|
|||||||
callClose = "close"
|
callClose = "close"
|
||||||
)
|
)
|
||||||
|
|
||||||
func (s *closeOrderStore) GetOrCreateASNBatch(
|
func (s *closeOrderStore) GetOrCreateASNBatch(asns map[int]time.Time) error {
|
||||||
asns map[int]database.SeenTimes,
|
|
||||||
) error {
|
|
||||||
s.record(callWrite)
|
s.record(callWrite)
|
||||||
|
|
||||||
return s.Store.GetOrCreateASNBatch(asns)
|
return s.Store.GetOrCreateASNBatch(asns)
|
||||||
|
|||||||
@@ -229,21 +229,19 @@ func (h *ASHandler) flushBatch() {
|
|||||||
return
|
return
|
||||||
}
|
}
|
||||||
|
|
||||||
// The earliest and the latest time the batch saw each AS. The batch holds
|
// Process ASNs first (deduped)
|
||||||
// several updates for an AS, from many peers, so a new AS is stored first
|
asnMap := make(map[int]time.Time)
|
||||||
// seen at the earliest of them and last seen at the latest.
|
|
||||||
asns := make(map[int]database.SeenTimes)
|
|
||||||
for _, op := range batch {
|
for _, op := range batch {
|
||||||
seen := asns[op.number]
|
if existing, ok := asnMap[op.number]; !ok || op.timestamp.After(existing) {
|
||||||
seen.Add(op.timestamp)
|
asnMap[op.number] = op.timestamp
|
||||||
asns[op.number] = seen
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// Process all ASNs in a single batch transaction
|
// Process all ASNs in a single batch transaction
|
||||||
err := h.db.GetOrCreateASNBatch(asns)
|
err := h.db.GetOrCreateASNBatch(asnMap)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
h.logger.Error("Failed to process ASN batch",
|
h.logger.Error("Failed to process ASN batch",
|
||||||
"error", err, "count", len(asns))
|
"error", err, "count", len(asnMap))
|
||||||
}
|
}
|
||||||
|
|
||||||
h.mu.Lock()
|
h.mu.Lock()
|
||||||
|
|||||||
@@ -1,68 +0,0 @@
|
|||||||
package routewatch
|
|
||||||
|
|
||||||
import (
|
|
||||||
"maps"
|
|
||||||
"slices"
|
|
||||||
"testing"
|
|
||||||
|
|
||||||
"git.eeqj.de/sneak/routewatch/internal/logger"
|
|
||||||
)
|
|
||||||
|
|
||||||
// TestASHandlerStoresTheEarliestAndLatestTimesOfANewAS handles two updates
|
|
||||||
// whose path carries an AS the database does not hold yet, the second 5
|
|
||||||
// seconds after the first, in one batch into a real database. The AS is stored
|
|
||||||
// first seen at the time of the first update and last seen at the time of the
|
|
||||||
// second.
|
|
||||||
func TestASHandlerStoresTheEarliestAndLatestTimesOfANewAS(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
db := newTestDatabase(t)
|
|
||||||
|
|
||||||
// Built without NewASHandler's flush goroutine, so the test flushes itself.
|
|
||||||
h := &ASHandler{db: db, logger: logger.New()}
|
|
||||||
|
|
||||||
first := peerAnnouncement(testPeerIP, testV4Prefix)
|
|
||||||
second := peerAnnouncement(testPeerIP, testV4Prefix)
|
|
||||||
second.ParsedTimestamp = first.ParsedTimestamp.Add(secondUpdateAfter)
|
|
||||||
|
|
||||||
h.HandleMessage(first)
|
|
||||||
h.HandleMessage(second)
|
|
||||||
h.flushBatch()
|
|
||||||
|
|
||||||
asn, _, err := db.GetASDetails(testASNA)
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("GetASDetails(%d): %v", testASNA, err)
|
|
||||||
}
|
|
||||||
|
|
||||||
if !asn.FirstSeen.Equal(first.ParsedTimestamp) {
|
|
||||||
t.Errorf("first_seen of AS%d = %s, want %s", testASNA,
|
|
||||||
asn.FirstSeen, first.ParsedTimestamp)
|
|
||||||
}
|
|
||||||
|
|
||||||
if !asn.LastSeen.Equal(second.ParsedTimestamp) {
|
|
||||||
t.Errorf("last_seen of AS%d = %s, want %s", testASNA,
|
|
||||||
asn.LastSeen, second.ParsedTimestamp)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestASHandlerRecordsNothingFromAPathWithASZero handles an update whose AS
|
|
||||||
// path holds AS 0 between two other ASes. RFC 7607 makes such a path
|
|
||||||
// malformed, so neither AS is recorded.
|
|
||||||
func TestASHandlerRecordsNothingFromAPathWithASZero(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
store := newMockStore()
|
|
||||||
|
|
||||||
// Built without NewASHandler's flush goroutine, so the test flushes itself.
|
|
||||||
h := &ASHandler{db: store, logger: logger.New()}
|
|
||||||
|
|
||||||
msg := peerAnnouncement(testPeerIP, testV4Prefix)
|
|
||||||
msg.Path = decodePath(t, testASNA, 0, testASNB)
|
|
||||||
h.HandleMessage(msg)
|
|
||||||
h.flushBatch()
|
|
||||||
|
|
||||||
if len(store.ASNs) > 0 {
|
|
||||||
t.Errorf("ASes recorded = %v, want none",
|
|
||||||
slices.Sorted(maps.Keys(store.ASNs)))
|
|
||||||
}
|
|
||||||
}
|
|
||||||
@@ -21,16 +21,10 @@ const (
|
|||||||
minInterval = 1 * time.Second
|
minInterval = 1 * time.Second
|
||||||
|
|
||||||
// maxInterval is the maximum interval after failures (backoff cap).
|
// maxInterval is the maximum interval after failures (backoff cap).
|
||||||
maxInterval = time.Minute
|
maxInterval = 5 * time.Minute
|
||||||
|
|
||||||
// A failure multiplies the interval by backoffNumerator divided by
|
// backoffMultiplier is how much to multiply interval on failure.
|
||||||
// backoffDenominator, 1.5, up to maxInterval.
|
backoffMultiplier = 2
|
||||||
backoffNumerator = 3
|
|
||||||
backoffDenominator = 2
|
|
||||||
|
|
||||||
// A success divides the interval by successDivisor, down to
|
|
||||||
// minInterval.
|
|
||||||
successDivisor = 2
|
|
||||||
|
|
||||||
// whoisStaleThreshold is how old WHOIS data can be before refresh.
|
// whoisStaleThreshold is how old WHOIS data can be before refresh.
|
||||||
whoisStaleThreshold = 30 * 24 * time.Hour // 30 days
|
whoisStaleThreshold = 30 * 24 * time.Hour // 30 days
|
||||||
@@ -59,8 +53,7 @@ type ASNFetcher struct {
|
|||||||
currentInterval time.Duration
|
currentInterval time.Duration
|
||||||
consecutiveFails int
|
consecutiveFails int
|
||||||
|
|
||||||
// hourly stats tracking; now is the clock, time.Now except in tests
|
// hourly stats tracking
|
||||||
now func() time.Time
|
|
||||||
statsMu sync.Mutex
|
statsMu sync.Mutex
|
||||||
successTimes []time.Time
|
successTimes []time.Time
|
||||||
errorTimes []time.Time
|
errorTimes []time.Time
|
||||||
@@ -68,17 +61,14 @@ type ASNFetcher struct {
|
|||||||
|
|
||||||
// NewASNFetcher creates a new ASN fetcher.
|
// NewASNFetcher creates a new ASN fetcher.
|
||||||
func NewASNFetcher(db database.Store, logger *slog.Logger) *ASNFetcher {
|
func NewASNFetcher(db database.Store, logger *slog.Logger) *ASNFetcher {
|
||||||
logger = logger.With("component", "asn_fetcher")
|
|
||||||
|
|
||||||
return &ASNFetcher{
|
return &ASNFetcher{
|
||||||
db: db,
|
db: db,
|
||||||
whoisClient: whois.NewClient(logger),
|
whoisClient: whois.NewClient(),
|
||||||
logger: logger,
|
logger: logger.With("component", "asn_fetcher"),
|
||||||
immediateQueue: make(chan int, immediateQueueSize),
|
immediateQueue: make(chan int, immediateQueueSize),
|
||||||
stopCh: make(chan struct{}),
|
stopCh: make(chan struct{}),
|
||||||
done: make(chan struct{}),
|
done: make(chan struct{}),
|
||||||
currentInterval: baseInterval,
|
currentInterval: baseInterval,
|
||||||
now: time.Now,
|
|
||||||
successTimes: make([]time.Time, 0),
|
successTimes: make([]time.Time, 0),
|
||||||
errorTimes: make([]time.Time, 0),
|
errorTimes: make([]time.Time, 0),
|
||||||
}
|
}
|
||||||
@@ -131,7 +121,7 @@ func (f *ASNFetcher) GetStats() server.ASNFetcherStats {
|
|||||||
f.intervalMu.Unlock()
|
f.intervalMu.Unlock()
|
||||||
|
|
||||||
// Prune old entries and count
|
// Prune old entries and count
|
||||||
cutoff := f.now().Add(-statsWindow)
|
cutoff := time.Now().Add(-statsWindow)
|
||||||
f.successTimes = pruneOldTimes(f.successTimes, cutoff)
|
f.successTimes = pruneOldTimes(f.successTimes, cutoff)
|
||||||
f.errorTimes = pruneOldTimes(f.errorTimes, cutoff)
|
f.errorTimes = pruneOldTimes(f.errorTimes, cutoff)
|
||||||
|
|
||||||
@@ -143,15 +133,16 @@ func (f *ASNFetcher) GetStats() server.ASNFetcherStats {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// pruneOldTimes drops the times at or before cutoff from the front of times,
|
// pruneOldTimes removes times older than cutoff and returns the pruned slice.
|
||||||
// which holds times in the order they were recorded, and returns the rest. It
|
|
||||||
// copies nothing, so it is cheap enough to run on every append.
|
|
||||||
func pruneOldTimes(times []time.Time, cutoff time.Time) []time.Time {
|
func pruneOldTimes(times []time.Time, cutoff time.Time) []time.Time {
|
||||||
for len(times) > 0 && !times[0].After(cutoff) {
|
result := make([]time.Time, 0, len(times))
|
||||||
times = times[1:]
|
for _, t := range times {
|
||||||
|
if t.After(cutoff) {
|
||||||
|
result = append(result, t)
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
return times
|
return result
|
||||||
}
|
}
|
||||||
|
|
||||||
// getInterval returns the current fetch interval.
|
// getInterval returns the current fetch interval.
|
||||||
@@ -168,7 +159,7 @@ func (f *ASNFetcher) recordSuccess() {
|
|||||||
f.consecutiveFails = 0
|
f.consecutiveFails = 0
|
||||||
|
|
||||||
// Decrease interval by half, but not below minimum
|
// Decrease interval by half, but not below minimum
|
||||||
newInterval := max(f.currentInterval/successDivisor, minInterval)
|
newInterval := max(f.currentInterval/backoffMultiplier, minInterval)
|
||||||
|
|
||||||
if newInterval != f.currentInterval {
|
if newInterval != f.currentInterval {
|
||||||
f.logger.Debug("Decreased fetch interval",
|
f.logger.Debug("Decreased fetch interval",
|
||||||
@@ -179,11 +170,9 @@ func (f *ASNFetcher) recordSuccess() {
|
|||||||
}
|
}
|
||||||
f.intervalMu.Unlock()
|
f.intervalMu.Unlock()
|
||||||
|
|
||||||
// Record success time for stats, dropping those older than statsWindow
|
// Record success time for stats
|
||||||
f.statsMu.Lock()
|
f.statsMu.Lock()
|
||||||
now := f.now()
|
f.successTimes = append(f.successTimes, time.Now())
|
||||||
f.successTimes = pruneOldTimes(f.successTimes, now.Add(-statsWindow))
|
|
||||||
f.successTimes = append(f.successTimes, now)
|
|
||||||
f.statsMu.Unlock()
|
f.statsMu.Unlock()
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -192,9 +181,8 @@ func (f *ASNFetcher) recordFailure() {
|
|||||||
f.intervalMu.Lock()
|
f.intervalMu.Lock()
|
||||||
f.consecutiveFails++
|
f.consecutiveFails++
|
||||||
|
|
||||||
// Exponential backoff: multiply by 1.5, capped at max
|
// Exponential backoff: multiply by 2, capped at max
|
||||||
newInterval := min(
|
newInterval := min(f.currentInterval*backoffMultiplier, maxInterval)
|
||||||
f.currentInterval*backoffNumerator/backoffDenominator, maxInterval)
|
|
||||||
|
|
||||||
if newInterval != f.currentInterval {
|
if newInterval != f.currentInterval {
|
||||||
f.logger.Debug("Increased fetch interval due to failure",
|
f.logger.Debug("Increased fetch interval due to failure",
|
||||||
@@ -206,11 +194,9 @@ func (f *ASNFetcher) recordFailure() {
|
|||||||
}
|
}
|
||||||
f.intervalMu.Unlock()
|
f.intervalMu.Unlock()
|
||||||
|
|
||||||
// Record error time for stats, dropping those older than statsWindow
|
// Record error time for stats
|
||||||
f.statsMu.Lock()
|
f.statsMu.Lock()
|
||||||
now := f.now()
|
f.errorTimes = append(f.errorTimes, time.Now())
|
||||||
f.errorTimes = pruneOldTimes(f.errorTimes, now.Add(-statsWindow))
|
|
||||||
f.errorTimes = append(f.errorTimes, now)
|
|
||||||
f.statsMu.Unlock()
|
f.statsMu.Unlock()
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
@@ -120,58 +120,17 @@ func TestFetchAndUpdateKeepsTheRowWhenTheLookupFails(t *testing.T) {
|
|||||||
"was stored", stats.NeverFetched)
|
"was stored", stats.NeverFetched)
|
||||||
}
|
}
|
||||||
|
|
||||||
want := baseInterval * backoffNumerator / backoffDenominator
|
|
||||||
|
|
||||||
got := f.GetStats()
|
got := f.GetStats()
|
||||||
if got.CurrentInterval != want || got.ConsecutiveFails != 1 {
|
if got.CurrentInterval != baseInterval*backoffMultiplier ||
|
||||||
|
got.ConsecutiveFails != 1 {
|
||||||
t.Errorf("interval %v after %d failures, want %v after 1",
|
t.Errorf("interval %v after %d failures, want %v after 1",
|
||||||
got.CurrentInterval, got.ConsecutiveFails, want)
|
got.CurrentInterval, got.ConsecutiveFails,
|
||||||
|
baseInterval*backoffMultiplier)
|
||||||
}
|
}
|
||||||
})
|
})
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// TestRecordStepsTheInterval has the fetcher fail from its base interval of
|
|
||||||
// 15 seconds, then succeed. Each failure makes the interval 1.5 times as
|
|
||||||
// long, up to one minute, and each success halves it, down to one second.
|
|
||||||
func TestRecordStepsTheInterval(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
f := NewASNFetcher(newMockStore(), logger.New().Logger)
|
|
||||||
|
|
||||||
for i, want := range []time.Duration{
|
|
||||||
22500 * time.Millisecond,
|
|
||||||
33750 * time.Millisecond,
|
|
||||||
50625 * time.Millisecond,
|
|
||||||
time.Minute,
|
|
||||||
time.Minute,
|
|
||||||
} {
|
|
||||||
f.recordFailure()
|
|
||||||
|
|
||||||
if got := f.getInterval(); got != want {
|
|
||||||
t.Errorf("interval after %d failures = %v, want %v",
|
|
||||||
i+1, got, want)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
for i, want := range []time.Duration{
|
|
||||||
30 * time.Second,
|
|
||||||
15 * time.Second,
|
|
||||||
7500 * time.Millisecond,
|
|
||||||
3750 * time.Millisecond,
|
|
||||||
1875 * time.Millisecond,
|
|
||||||
time.Second,
|
|
||||||
time.Second,
|
|
||||||
} {
|
|
||||||
f.recordSuccess()
|
|
||||||
|
|
||||||
if got := f.getInterval(); got != want {
|
|
||||||
t.Errorf("interval after %d successes = %v, want %v",
|
|
||||||
i+1, got, want)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestFetchAndUpdateMarksAnASWithoutReferral looks up a private-use AS, which
|
// TestFetchAndUpdateMarksAnASWithoutReferral looks up a private-use AS, which
|
||||||
// IANA refers to no RIR. The AS must not be looked up again until it is
|
// IANA refers to no RIR. The AS must not be looked up again until it is
|
||||||
// stale, and the lookup counts as neither a success nor a failure.
|
// stale, and the lookup counts as neither a success nor a failure.
|
||||||
@@ -244,35 +203,3 @@ func TestASNFetcherStopReturnsByItsDeadline(t *testing.T) {
|
|||||||
t.Fatal("the fetcher goroutine did not return after the lookup ended")
|
t.Fatal("the fetcher goroutine did not return after the lookup ended")
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// TestRecordKeepsOnlyTheStatsWindow records a success and a failure every
|
|
||||||
// second, by a clock the test sets, far longer than statsWindow, and never
|
|
||||||
// asks for the stats. The fetcher must keep only the times within the last
|
|
||||||
// statsWindow.
|
|
||||||
func TestRecordKeepsOnlyTheStatsWindow(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
const (
|
|
||||||
records = 100_000
|
|
||||||
step = time.Second
|
|
||||||
want = int(statsWindow / step)
|
|
||||||
)
|
|
||||||
|
|
||||||
clock := time.Now()
|
|
||||||
|
|
||||||
f := NewASNFetcher(newMockStore(), logger.New().Logger)
|
|
||||||
f.now = func() time.Time { return clock }
|
|
||||||
|
|
||||||
for range records {
|
|
||||||
clock = clock.Add(step)
|
|
||||||
|
|
||||||
f.recordSuccess()
|
|
||||||
f.recordFailure()
|
|
||||||
}
|
|
||||||
|
|
||||||
if len(f.successTimes) != want || len(f.errorTimes) != want {
|
|
||||||
t.Errorf("%d success and %d error times kept, want %d of each, "+
|
|
||||||
"those of the last %s", len(f.successTimes), len(f.errorTimes),
|
|
||||||
want, statsWindow)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|||||||
@@ -66,7 +66,7 @@ func (s *blockingStore) UpsertLiveRouteBatch(routes []*database.LiveRoute) error
|
|||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
|
|
||||||
func (s *blockingStore) GetOrCreateASNBatch(asns map[int]database.SeenTimes) error {
|
func (s *blockingStore) GetOrCreateASNBatch(asns map[int]time.Time) error {
|
||||||
s.waitForRelease()
|
s.waitForRelease()
|
||||||
s.mu.Lock()
|
s.mu.Lock()
|
||||||
defer s.mu.Unlock()
|
defer s.mu.Unlock()
|
||||||
@@ -96,12 +96,10 @@ func (s *blockingStore) DeleteLiveRoutesOfPeer(peerIP string) (int, error) {
|
|||||||
return s.mockStore.DeleteLiveRoutesOfPeer(peerIP)
|
return s.mockStore.DeleteLiveRoutesOfPeer(peerIP)
|
||||||
}
|
}
|
||||||
|
|
||||||
func (s *blockingStore) RecordPeering(
|
func (s *blockingStore) RecordPeering(asA, asB int, ts time.Time) error {
|
||||||
asA, asB int, seen database.SeenTimes,
|
|
||||||
) error {
|
|
||||||
s.waitForRelease()
|
s.waitForRelease()
|
||||||
|
|
||||||
return s.mockStore.RecordPeering(asA, asB, seen)
|
return s.mockStore.RecordPeering(asA, asB, ts)
|
||||||
}
|
}
|
||||||
|
|
||||||
func (s *blockingStore) waitForRelease() {
|
func (s *blockingStore) waitForRelease() {
|
||||||
|
|||||||
@@ -3,8 +3,10 @@ package routewatch
|
|||||||
import (
|
import (
|
||||||
"context"
|
"context"
|
||||||
"os"
|
"os"
|
||||||
|
"os/signal"
|
||||||
"runtime"
|
"runtime"
|
||||||
"strings"
|
"strings"
|
||||||
|
"syscall"
|
||||||
"time"
|
"time"
|
||||||
|
|
||||||
"git.eeqj.de/sneak/routewatch/internal/logger"
|
"git.eeqj.de/sneak/routewatch/internal/logger"
|
||||||
@@ -60,10 +62,10 @@ func logDebugStats(logger *logger.Logger) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
// CLIEntry is the main entry point for the routewatch command-line interface.
|
// CLIEntry is the main entry point for the routewatch command-line interface.
|
||||||
// It initializes the application using the fx dependency injection framework
|
// It initializes the application using the fx dependency injection framework,
|
||||||
// and starts the RouteWatch service. It blocks until fx receives SIGINT or
|
// sets up signal handling for graceful shutdown, and starts the RouteWatch
|
||||||
// SIGTERM or RouteWatch stops on an error; fx then runs the stop hook and sets
|
// service. This function blocks until the application receives a shutdown
|
||||||
// the exit code.
|
// signal or encounters a fatal error.
|
||||||
func CLIEntry() {
|
func CLIEntry() {
|
||||||
app := fx.New(
|
app := fx.New(
|
||||||
fx.Options(getModule()...),
|
fx.Options(getModule()...),
|
||||||
@@ -79,6 +81,20 @@ func CLIEntry() {
|
|||||||
// Start debug stats logging
|
// Start debug stats logging
|
||||||
go logDebugStats(logger)
|
go logDebugStats(logger)
|
||||||
|
|
||||||
|
// Handle shutdown signals
|
||||||
|
sigCh := make(chan os.Signal, 1)
|
||||||
|
signal.Notify(sigCh, os.Interrupt, syscall.SIGTERM)
|
||||||
|
|
||||||
|
go func() {
|
||||||
|
<-sigCh
|
||||||
|
logger.Info("Received shutdown signal")
|
||||||
|
|
||||||
|
err := shutdowner.Shutdown()
|
||||||
|
if err != nil {
|
||||||
|
logger.Error("Failed to shutdown gracefully", "error", err)
|
||||||
|
}
|
||||||
|
}()
|
||||||
|
|
||||||
go func() {
|
go func() {
|
||||||
err := rw.Run(ctx)
|
err := rw.Run(ctx)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
|
|||||||
@@ -37,8 +37,8 @@ const (
|
|||||||
|
|
||||||
// PeerHandler tracks BGP peers from UPDATE messages using batched operations.
|
// PeerHandler tracks BGP peers from UPDATE messages using batched operations.
|
||||||
// It maintains a queue of peer updates and periodically flushes them to the database
|
// It maintains a queue of peer updates and periodically flushes them to the database
|
||||||
// in batches to improve performance. The handler writes one row per peer IP
|
// in batches to improve performance. The handler deduplicates peer updates within
|
||||||
// address for each batch, with the earliest and the latest time of its updates.
|
// each batch, keeping only the most recent update for each peer IP address.
|
||||||
type PeerHandler struct {
|
type PeerHandler struct {
|
||||||
db database.Store
|
db database.Store
|
||||||
logger *logger.Logger
|
logger *logger.Logger
|
||||||
@@ -227,28 +227,31 @@ func (h *PeerHandler) flushBatch() {
|
|||||||
return
|
return
|
||||||
}
|
}
|
||||||
|
|
||||||
// The earliest and the latest time the batch saw each peer, with the peer
|
// Deduplicate by peer IP, keeping the latest update for each peer
|
||||||
// AS and message type of its latest update. The batch holds several
|
peerMap := make(map[string]peerUpdate)
|
||||||
// updates for a peer, so a new peer is stored first seen at the earliest of
|
|
||||||
// them and last seen at the latest.
|
|
||||||
peers := make(map[string]database.PeerUpdate)
|
|
||||||
for _, update := range batch {
|
for _, update := range batch {
|
||||||
peer, ok := peers[update.peerIP]
|
existing, ok := peerMap[update.peerIP]
|
||||||
if !ok || update.timestamp.After(peer.Seen.Last) {
|
if !ok || update.timestamp.After(existing.timestamp) {
|
||||||
peer.PeerIP = update.peerIP
|
peerMap[update.peerIP] = update
|
||||||
peer.PeerASN = update.peerASN
|
|
||||||
peer.MessageType = update.messageType
|
|
||||||
}
|
}
|
||||||
|
}
|
||||||
|
|
||||||
peer.Seen.Add(update.timestamp)
|
// Convert to database format
|
||||||
peers[update.peerIP] = peer
|
dbPeerMap := make(map[string]database.PeerUpdate)
|
||||||
|
for peerIP, update := range peerMap {
|
||||||
|
dbPeerMap[peerIP] = database.PeerUpdate{
|
||||||
|
PeerIP: update.peerIP,
|
||||||
|
PeerASN: update.peerASN,
|
||||||
|
MessageType: update.messageType,
|
||||||
|
Timestamp: update.timestamp,
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// Process all peers in a single batch transaction
|
// Process all peers in a single batch transaction
|
||||||
err := h.db.UpdatePeerBatch(peers)
|
err := h.db.UpdatePeerBatch(dbPeerMap)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
h.logger.Error("Failed to process peer batch",
|
h.logger.Error("Failed to process peer batch",
|
||||||
"error", err, "count", len(peers))
|
"error", err, "count", len(dbPeerMap))
|
||||||
}
|
}
|
||||||
|
|
||||||
h.mu.Lock()
|
h.mu.Lock()
|
||||||
|
|||||||
@@ -1,59 +0,0 @@
|
|||||||
package routewatch
|
|
||||||
|
|
||||||
import (
|
|
||||||
"testing"
|
|
||||||
|
|
||||||
"git.eeqj.de/sneak/routewatch/internal/database"
|
|
||||||
"git.eeqj.de/sneak/routewatch/internal/logger"
|
|
||||||
)
|
|
||||||
|
|
||||||
// peerBatchStore keeps the peers of the last UpdatePeerBatch call. The
|
|
||||||
// database has no read of the peer table, so the peer handler's test checks
|
|
||||||
// what the handler writes.
|
|
||||||
type peerBatchStore struct {
|
|
||||||
*mockStore
|
|
||||||
|
|
||||||
peers map[string]database.PeerUpdate
|
|
||||||
}
|
|
||||||
|
|
||||||
func (s *peerBatchStore) UpdatePeerBatch(
|
|
||||||
peers map[string]database.PeerUpdate,
|
|
||||||
) error {
|
|
||||||
s.peers = peers
|
|
||||||
|
|
||||||
return nil
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestPeerHandlerWritesTheEarliestAndLatestTimesOfANewPeer handles two updates
|
|
||||||
// from one peer, the second 5 seconds after the first, in one batch. The peer
|
|
||||||
// handler writes the peer first seen at the time of the first update and last
|
|
||||||
// seen at the time of the second.
|
|
||||||
func TestPeerHandlerWritesTheEarliestAndLatestTimesOfANewPeer(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
store := &peerBatchStore{mockStore: newMockStore()}
|
|
||||||
|
|
||||||
// Built without NewPeerHandler's flush goroutine, so the test flushes
|
|
||||||
// itself.
|
|
||||||
h := &PeerHandler{db: store, logger: logger.New()}
|
|
||||||
|
|
||||||
first := peerAnnouncement(testPeerIP, testV4Prefix)
|
|
||||||
second := peerAnnouncement(testPeerIP, testV4Prefix)
|
|
||||||
second.ParsedTimestamp = first.ParsedTimestamp.Add(secondUpdateAfter)
|
|
||||||
|
|
||||||
h.HandleMessage(first)
|
|
||||||
h.HandleMessage(second)
|
|
||||||
h.flushBatch()
|
|
||||||
|
|
||||||
seen := store.peers[testPeerIP].Seen
|
|
||||||
|
|
||||||
if !seen.First.Equal(first.ParsedTimestamp) {
|
|
||||||
t.Errorf("first_seen of peer %s = %s, want %s", testPeerIP,
|
|
||||||
seen.First, first.ParsedTimestamp)
|
|
||||||
}
|
|
||||||
|
|
||||||
if !seen.Last.Equal(second.ParsedTimestamp) {
|
|
||||||
t.Errorf("last_seen of peer %s = %s, want %s", testPeerIP,
|
|
||||||
seen.Last, second.ParsedTimestamp)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
@@ -42,11 +42,10 @@ type PeeringHandler struct {
|
|||||||
db database.Store
|
db database.Store
|
||||||
logger *logger.Logger
|
logger *logger.Logger
|
||||||
|
|
||||||
// In-memory AS path tracking: each path, JSON-encoded, with the earliest
|
// In-memory AS path tracking
|
||||||
// and the latest time an update carried it.
|
|
||||||
mu sync.Mutex
|
mu sync.Mutex
|
||||||
asPaths map[string]database.SeenTimes
|
asPaths map[string]time.Time // key is JSON-encoded AS path
|
||||||
droppedPaths int // paths dropped since the last run because the map was full
|
droppedPaths int // paths dropped because the map was full
|
||||||
// writing is how many paths processPeerings took from asPaths and has
|
// writing is how many paths processPeerings took from asPaths and has
|
||||||
// not finished writing the peerings of.
|
// not finished writing the peerings of.
|
||||||
writing int
|
writing int
|
||||||
@@ -65,7 +64,7 @@ func NewPeeringHandler(db database.Store, logger *logger.Logger) *PeeringHandler
|
|||||||
h := &PeeringHandler{
|
h := &PeeringHandler{
|
||||||
db: db,
|
db: db,
|
||||||
logger: logger,
|
logger: logger,
|
||||||
asPaths: make(map[string]database.SeenTimes),
|
asPaths: make(map[string]time.Time),
|
||||||
stopCh: make(chan context.Context, 1),
|
stopCh: make(chan context.Context, 1),
|
||||||
done: make(chan struct{}),
|
done: make(chan struct{}),
|
||||||
}
|
}
|
||||||
@@ -119,18 +118,16 @@ func (h *PeeringHandler) HandleMessage(msg *ristypes.RISMessage) {
|
|||||||
key := string(pathJSON)
|
key := string(pathJSON)
|
||||||
|
|
||||||
h.mu.Lock()
|
h.mu.Lock()
|
||||||
defer h.mu.Unlock()
|
if _, exists := h.asPaths[key]; exists {
|
||||||
|
// Already tracked: refresh its timestamp.
|
||||||
seen, exists := h.asPaths[key]
|
h.asPaths[key] = timestamp
|
||||||
if !exists && len(h.asPaths) >= maxTrackedPaths {
|
} else if len(h.asPaths) >= maxTrackedPaths {
|
||||||
// Map is full; drop this new path and count it.
|
// Map is full; drop this new path and count it.
|
||||||
h.droppedPaths++
|
h.droppedPaths++
|
||||||
|
} else {
|
||||||
return
|
h.asPaths[key] = timestamp
|
||||||
}
|
}
|
||||||
|
h.mu.Unlock()
|
||||||
seen.Add(timestamp)
|
|
||||||
h.asPaths[key] = seen
|
|
||||||
}
|
}
|
||||||
|
|
||||||
// ProcessPeeringsNow triggers immediate processing of all accumulated AS
|
// ProcessPeeringsNow triggers immediate processing of all accumulated AS
|
||||||
@@ -190,14 +187,12 @@ func (h *PeeringHandler) processLoop() {
|
|||||||
func (h *PeeringHandler) processPeerings() {
|
func (h *PeeringHandler) processPeerings() {
|
||||||
// Take the accumulated paths and replace the map with a fresh empty one
|
// Take the accumulated paths and replace the map with a fresh empty one
|
||||||
// under the lock. Each path is processed exactly once and the memory is
|
// under the lock. Each path is processed exactly once and the memory is
|
||||||
// released, so the map never grows past a single interval's traffic. The
|
// released, so the map never grows past a single interval's traffic.
|
||||||
// count of dropped paths starts again with the new map.
|
|
||||||
h.mu.Lock()
|
h.mu.Lock()
|
||||||
paths := h.asPaths
|
paths := h.asPaths
|
||||||
h.asPaths = make(map[string]database.SeenTimes)
|
h.asPaths = make(map[string]time.Time)
|
||||||
h.writing = len(paths)
|
h.writing = len(paths)
|
||||||
dropped := h.droppedPaths
|
dropped := h.droppedPaths
|
||||||
h.droppedPaths = 0
|
|
||||||
h.mu.Unlock()
|
h.mu.Unlock()
|
||||||
|
|
||||||
if len(paths) == 0 {
|
if len(paths) == 0 {
|
||||||
@@ -209,13 +204,9 @@ func (h *PeeringHandler) processPeerings() {
|
|||||||
low, high int
|
low, high int
|
||||||
}
|
}
|
||||||
|
|
||||||
// The earliest and the latest time an update carried each peering. Several
|
peerings := make(map[peeringKey]time.Time)
|
||||||
// paths, each carried at several times, can hold a peering, so a new
|
|
||||||
// peering is stored first seen at the earliest of them and last seen at the
|
|
||||||
// latest.
|
|
||||||
peerings := make(map[peeringKey]database.SeenTimes)
|
|
||||||
|
|
||||||
for pathJSON, pathSeen := range paths {
|
for pathJSON, timestamp := range paths {
|
||||||
var path []int
|
var path []int
|
||||||
|
|
||||||
err := json.Unmarshal([]byte(pathJSON), &path)
|
err := json.Unmarshal([]byte(pathJSON), &path)
|
||||||
@@ -242,11 +233,10 @@ func (h *PeeringHandler) processPeerings() {
|
|||||||
}
|
}
|
||||||
|
|
||||||
key := peeringKey{low: low, high: high}
|
key := peeringKey{low: low, high: high}
|
||||||
|
// Update timestamp if this is newer
|
||||||
seen := peerings[key]
|
if existing, ok := peerings[key]; !ok || timestamp.After(existing) {
|
||||||
seen.Add(pathSeen.First)
|
peerings[key] = timestamp
|
||||||
seen.Add(pathSeen.Last)
|
}
|
||||||
peerings[key] = seen
|
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -254,8 +244,8 @@ func (h *PeeringHandler) processPeerings() {
|
|||||||
start := time.Now()
|
start := time.Now()
|
||||||
successCount := 0
|
successCount := 0
|
||||||
|
|
||||||
for key, seen := range peerings {
|
for key, ts := range peerings {
|
||||||
err := h.db.RecordPeering(key.low, key.high, seen)
|
err := h.db.RecordPeering(key.low, key.high, ts)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
h.logger.Error("Failed to record peering",
|
h.logger.Error("Failed to record peering",
|
||||||
"as_a", key.low,
|
"as_a", key.low,
|
||||||
|
|||||||
@@ -31,14 +31,12 @@ type recordingStore struct {
|
|||||||
calls int
|
calls int
|
||||||
}
|
}
|
||||||
|
|
||||||
func (r *recordingStore) RecordPeering(
|
func (r *recordingStore) RecordPeering(asA, asB int, ts time.Time) error {
|
||||||
asA, asB int, seen database.SeenTimes,
|
|
||||||
) error {
|
|
||||||
r.mu.Lock()
|
r.mu.Lock()
|
||||||
r.calls++
|
r.calls++
|
||||||
r.mu.Unlock()
|
r.mu.Unlock()
|
||||||
|
|
||||||
return r.mockStore.RecordPeering(asA, asB, seen)
|
return r.mockStore.RecordPeering(asA, asB, ts)
|
||||||
}
|
}
|
||||||
|
|
||||||
func (r *recordingStore) callCount() int {
|
func (r *recordingStore) callCount() int {
|
||||||
@@ -54,7 +52,7 @@ func newTestHandler(db database.Store) *PeeringHandler {
|
|||||||
return &PeeringHandler{
|
return &PeeringHandler{
|
||||||
db: db,
|
db: db,
|
||||||
logger: logger.New(),
|
logger: logger.New(),
|
||||||
asPaths: make(map[string]database.SeenTimes),
|
asPaths: make(map[string]time.Time),
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -153,43 +151,6 @@ func TestPeeringHandlerLeavesATrailingASSetOutOfThePeerings(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// TestPeeringHandlerRecordsNothingFromAPathWithASZero handles AS paths that
|
|
||||||
// hold AS 0. RFC 7607 makes such a path malformed, so no peering is recorded
|
|
||||||
// from it, not even one between two other ASes of the path.
|
|
||||||
func TestPeeringHandlerRecordsNothingFromAPathWithASZero(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
tests := []struct {
|
|
||||||
name string
|
|
||||||
path []any
|
|
||||||
}{
|
|
||||||
{name: "AS 0 between two ASes", path: []any{testASNA, 0, testASNB}},
|
|
||||||
{
|
|
||||||
name: "AS 0 after two ASes",
|
|
||||||
path: []any{testASNA, testASNB, 0, testASNC},
|
|
||||||
},
|
|
||||||
}
|
|
||||||
|
|
||||||
for _, tt := range tests {
|
|
||||||
t.Run(tt.name, func(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
store := newMockStore()
|
|
||||||
h := newTestHandler(store)
|
|
||||||
h.HandleMessage(&ristypes.RISMessage{
|
|
||||||
Path: decodePath(t, tt.path...),
|
|
||||||
ParsedTimestamp: time.Now().UTC(),
|
|
||||||
})
|
|
||||||
h.ProcessPeeringsNow()
|
|
||||||
|
|
||||||
if len(store.Peerings) > 0 {
|
|
||||||
t.Errorf("peerings recorded = %v, want none",
|
|
||||||
slices.Sorted(maps.Keys(store.Peerings)))
|
|
||||||
}
|
|
||||||
})
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestPeeringHandlerProcessesEachRunAndEmpties verifies that a processing run
|
// TestPeeringHandlerProcessesEachRunAndEmpties verifies that a processing run
|
||||||
// empties the path map (the swap) and that a path seen again after a run is
|
// empties the path map (the swap) and that a path seen again after a run is
|
||||||
// recorded in the next run too.
|
// recorded in the next run too.
|
||||||
@@ -259,10 +220,10 @@ func TestPeeringHandlerCapDropsAndCounts(t *testing.T) {
|
|||||||
existing := pathKey(t, testASNA, testASNB)
|
existing := pathKey(t, testASNA, testASNB)
|
||||||
|
|
||||||
h.mu.Lock()
|
h.mu.Lock()
|
||||||
h.asPaths[existing] = database.SeenTimes{First: now, Last: now}
|
h.asPaths[existing] = now
|
||||||
|
|
||||||
for i := 0; len(h.asPaths) < maxTrackedPaths; i++ {
|
for i := 0; len(h.asPaths) < maxTrackedPaths; i++ {
|
||||||
h.asPaths[strconv.Itoa(i)] = database.SeenTimes{First: now, Last: now}
|
h.asPaths[strconv.Itoa(i)] = now
|
||||||
}
|
}
|
||||||
h.mu.Unlock()
|
h.mu.Unlock()
|
||||||
|
|
||||||
@@ -292,83 +253,11 @@ func TestPeeringHandlerCapDropsAndCounts(t *testing.T) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
h.mu.Lock()
|
h.mu.Lock()
|
||||||
gotSeen := h.asPaths[existing]
|
gotTS := h.asPaths[existing]
|
||||||
h.mu.Unlock()
|
h.mu.Unlock()
|
||||||
|
|
||||||
if !gotSeen.Last.Equal(refreshed) {
|
if !gotTS.Equal(refreshed) {
|
||||||
t.Fatalf("existing path last seen not refreshed: want %v, got %v",
|
t.Fatalf("existing path timestamp not refreshed: want %v, got %v",
|
||||||
refreshed, gotSeen.Last)
|
refreshed, gotTS)
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestPeeringHandlerRunResetsTheDroppedCount fills the map, has the handler
|
|
||||||
// drop one path, and runs the processing. The count of dropped paths is the
|
|
||||||
// run's own, so it is 0 again after the run.
|
|
||||||
func TestPeeringHandlerRunResetsTheDroppedCount(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
h := newTestHandler(newMockStore())
|
|
||||||
|
|
||||||
now := time.Now().UTC()
|
|
||||||
|
|
||||||
// Paths of one AS carry no peering, so the run records nothing.
|
|
||||||
h.mu.Lock()
|
|
||||||
for asn := 1; len(h.asPaths) < maxTrackedPaths; asn++ {
|
|
||||||
h.asPaths[pathKey(t, asn)] = database.SeenTimes{First: now, Last: now}
|
|
||||||
}
|
|
||||||
h.mu.Unlock()
|
|
||||||
|
|
||||||
handle(h, now, testASNA, testASNB)
|
|
||||||
|
|
||||||
if _, dropped := snapshot(h); dropped != 1 {
|
|
||||||
t.Fatalf("want dropped count 1 with the map full, got %d", dropped)
|
|
||||||
}
|
|
||||||
|
|
||||||
h.ProcessPeeringsNow()
|
|
||||||
|
|
||||||
if _, dropped := snapshot(h); dropped != 0 {
|
|
||||||
t.Errorf("want dropped count 0 after a run, got %d", dropped)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestPeeringHandlerStoresTheEarliestAndLatestTimesOfANewPeering handles two
|
|
||||||
// updates whose path carries a peering the database does not hold yet, the
|
|
||||||
// second 5 seconds after the first, in one run into a real database. The
|
|
||||||
// peering is stored first seen at the time of the first update and last seen
|
|
||||||
// at the time of the second.
|
|
||||||
func TestPeeringHandlerStoresTheEarliestAndLatestTimesOfANewPeering(
|
|
||||||
t *testing.T,
|
|
||||||
) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
db := newTestDatabase(t)
|
|
||||||
h := newTestHandler(db)
|
|
||||||
|
|
||||||
first := peerAnnouncement(testPeerIP, testV4Prefix)
|
|
||||||
second := peerAnnouncement(testPeerIP, testV4Prefix)
|
|
||||||
second.ParsedTimestamp = first.ParsedTimestamp.Add(secondUpdateAfter)
|
|
||||||
|
|
||||||
h.HandleMessage(first)
|
|
||||||
h.HandleMessage(second)
|
|
||||||
h.ProcessPeeringsNow()
|
|
||||||
|
|
||||||
peers, err := db.GetASPeers(testASNA)
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("GetASPeers(%d): %v", testASNA, err)
|
|
||||||
}
|
|
||||||
|
|
||||||
if len(peers) != 1 || peers[0].ASN != testASNB {
|
|
||||||
t.Fatalf("peers of AS%d = %+v, want AS%d alone", testASNA, peers,
|
|
||||||
testASNB)
|
|
||||||
}
|
|
||||||
|
|
||||||
if !peers[0].FirstSeen.Equal(first.ParsedTimestamp) {
|
|
||||||
t.Errorf("first_seen of peering AS%d-AS%d = %s, want %s", testASNA,
|
|
||||||
testASNB, peers[0].FirstSeen, first.ParsedTimestamp)
|
|
||||||
}
|
|
||||||
|
|
||||||
if !peers[0].LastSeen.Equal(second.ParsedTimestamp) {
|
|
||||||
t.Errorf("last_seen of peering AS%d-AS%d = %s, want %s", testASNA,
|
|
||||||
testASNB, peers[0].LastSeen, second.ParsedTimestamp)
|
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -44,9 +44,8 @@ const (
|
|||||||
)
|
)
|
||||||
|
|
||||||
// PrefixHandler tracks BGP prefixes and maintains a live routing table in the
|
// PrefixHandler tracks BGP prefixes and maintains a live routing table in the
|
||||||
// database. Routes are added on announcement and deleted on withdrawal or on an
|
// database. Routes are added on announcement and deleted on withdrawal, and all
|
||||||
// announcement whose AS path is empty or holds AS 0 anywhere, and all routes of
|
// routes of a peer are deleted when its BGP session goes down.
|
||||||
// a peer are deleted when its BGP session goes down.
|
|
||||||
type PrefixHandler struct {
|
type PrefixHandler struct {
|
||||||
db database.Store
|
db database.Store
|
||||||
logger *logger.Logger
|
logger *logger.Logger
|
||||||
@@ -348,19 +347,8 @@ func (h *PrefixHandler) flushBatch() {
|
|||||||
|
|
||||||
lastUpdates := make(map[prefixAndPeer]prefixUpdate)
|
lastUpdates := make(map[prefixAndPeer]prefixUpdate)
|
||||||
|
|
||||||
// Each prefix the batch names, announced or withdrawn, with the earliest
|
|
||||||
// timestamp among all of its updates, not only each peer's last one: the
|
|
||||||
// prefix tables store only first_seen, written when the prefix is new and
|
|
||||||
// never changed after.
|
|
||||||
prefixesToUpdate := make(map[string]time.Time)
|
|
||||||
|
|
||||||
for _, update := range batch {
|
for _, update := range batch {
|
||||||
lastUpdates[prefixAndPeer{update.prefix, update.peer}] = update
|
lastUpdates[prefixAndPeer{update.prefix, update.peer}] = update
|
||||||
|
|
||||||
seen, exists := prefixesToUpdate[update.prefix]
|
|
||||||
if !exists || update.timestamp.Before(seen) {
|
|
||||||
prefixesToUpdate[update.prefix] = update.timestamp
|
|
||||||
}
|
|
||||||
}
|
}
|
||||||
|
|
||||||
// Collect routes to upsert and delete
|
// Collect routes to upsert and delete
|
||||||
@@ -369,14 +357,19 @@ func (h *PrefixHandler) flushBatch() {
|
|||||||
routesToDelete []database.LiveRouteDeletion
|
routesToDelete []database.LiveRouteDeletion
|
||||||
)
|
)
|
||||||
|
|
||||||
|
// Collect unique prefixes to update
|
||||||
|
prefixesToUpdate := make(map[string]time.Time)
|
||||||
|
|
||||||
for _, update := range lastUpdates {
|
for _, update := range lastUpdates {
|
||||||
|
// Track prefix for both announcements and withdrawals
|
||||||
|
seen, exists := prefixesToUpdate[update.prefix]
|
||||||
|
if !exists || update.timestamp.After(seen) {
|
||||||
|
prefixesToUpdate[update.prefix] = update.timestamp
|
||||||
|
}
|
||||||
|
|
||||||
// An announcement is stored when its path gives an origin, or when it
|
// An announcement is stored when its path gives an origin, or when it
|
||||||
// ends in an AS set of several members: that route is stored with
|
// ends in an AS set of several members: that route is stored with
|
||||||
// origin 0 (none). Any other update deletes the peer's route for the
|
// origin 0 (none). An announcement with an empty path is not stored.
|
||||||
// prefix: a withdrawal, or an announcement whose path is empty or
|
|
||||||
// holds AS 0 anywhere (the decoder empties such a path). Such an
|
|
||||||
// announcement replaces the peer's route, as any announcement does,
|
|
||||||
// with none (RFC 7607 treats AS 0 in a path as a withdrawal).
|
|
||||||
if update.messageType == "announcement" &&
|
if update.messageType == "announcement" &&
|
||||||
(update.originASN > 0 || update.endsInSet) {
|
(update.originASN > 0 || update.endsInSet) {
|
||||||
// Create live route for batch upsert
|
// Create live route for batch upsert
|
||||||
@@ -384,11 +377,11 @@ func (h *PrefixHandler) flushBatch() {
|
|||||||
if route != nil {
|
if route != nil {
|
||||||
routesToUpsert = append(routesToUpsert, route)
|
routesToUpsert = append(routesToUpsert, route)
|
||||||
}
|
}
|
||||||
} else {
|
} else if update.messageType == "withdrawal" {
|
||||||
// Parse CIDR to get IP version
|
// Parse CIDR to get IP version
|
||||||
_, ipVersion, err := parseCIDR(update.prefix)
|
_, ipVersion, err := parseCIDR(update.prefix)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
h.logger.Error("Failed to parse CIDR for route deletion",
|
h.logger.Error("Failed to parse CIDR for withdrawal",
|
||||||
"prefix", update.prefix, "error", err)
|
"prefix", update.prefix, "error", err)
|
||||||
|
|
||||||
continue
|
continue
|
||||||
|
|||||||
@@ -26,10 +26,6 @@ const (
|
|||||||
testV6Prefix = "2001:db8:1::/48"
|
testV6Prefix = "2001:db8:1::/48"
|
||||||
)
|
)
|
||||||
|
|
||||||
// secondUpdateAfter is how long after the first update the tests of first_seen
|
|
||||||
// and last_seen send the second, in the same batch.
|
|
||||||
const secondUpdateAfter = 5 * time.Second
|
|
||||||
|
|
||||||
// TestPrefixHandlerStoresPrefixesTheIPLookupFinds runs announcements and
|
// TestPrefixHandlerStoresPrefixesTheIPLookupFinds runs announcements and
|
||||||
// withdrawals through the prefix handler into a real database and looks the
|
// withdrawals through the prefix handler into a real database and looks the
|
||||||
// addresses up. The prefixes are written the way the feed sends them: IPv6
|
// addresses up. The prefixes are written the way the feed sends them: IPv6
|
||||||
@@ -87,49 +83,6 @@ func TestPrefixHandlerStoresPrefixesTheIPLookupFinds(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// TestPrefixHandlerStoresANewPrefixFirstSeenAtItsEarliestUpdate announces a
|
|
||||||
// prefix the database does not hold yet twice in one batch, the second time 5
|
|
||||||
// seconds after the first, through the prefix handler into a real database. The
|
|
||||||
// prefix is stored first seen at the time of the first announcement, whether
|
|
||||||
// the two come from two peers or from one, and the IP lookup answers with that
|
|
||||||
// time.
|
|
||||||
func TestPrefixHandlerStoresANewPrefixFirstSeenAtItsEarliestUpdate(
|
|
||||||
t *testing.T,
|
|
||||||
) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
const otherPeerIP = "2001:db8:ffff::2"
|
|
||||||
|
|
||||||
tests := []struct {
|
|
||||||
name string
|
|
||||||
firstPeer, secondPeer string
|
|
||||||
}{
|
|
||||||
{name: "two peers", firstPeer: testPeerIP, secondPeer: otherPeerIP},
|
|
||||||
{name: "one peer", firstPeer: testPeerIP, secondPeer: testPeerIP},
|
|
||||||
}
|
|
||||||
|
|
||||||
for _, tt := range tests {
|
|
||||||
t.Run(tt.name, func(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
first := peerAnnouncement(tt.firstPeer, testV4Prefix)
|
|
||||||
second := peerAnnouncement(tt.secondPeer, testV4Prefix)
|
|
||||||
second.ParsedTimestamp = first.ParsedTimestamp.Add(secondUpdateAfter)
|
|
||||||
db := handleBatches(t, [][]*ristypes.RISMessage{{first, second}})
|
|
||||||
|
|
||||||
info, err := db.GetIPInfo("192.0.2.1")
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("GetIPInfo(192.0.2.1): %v", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
if !info.FirstSeen.Equal(first.ParsedTimestamp) {
|
|
||||||
t.Errorf("first_seen of %s = %s, want %s", testV4Prefix,
|
|
||||||
info.FirstSeen, first.ParsedTimestamp)
|
|
||||||
}
|
|
||||||
})
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestPrefixHandlerAppliesEachPeersLastUpdateInABatch runs updates for one
|
// TestPrefixHandlerAppliesEachPeersLastUpdateInABatch runs updates for one
|
||||||
// prefix through the prefix handler into a real database. All updates carry the
|
// prefix through the prefix handler into a real database. All updates carry the
|
||||||
// same timestamp, as updates within one second of the feed do. Within a batch,
|
// same timestamp, as updates within one second of the feed do. Within a batch,
|
||||||
@@ -378,133 +331,6 @@ func TestPrefixHandlerDoesNotStoreAnAnnouncementWithAnEmptyPath(t *testing.T) {
|
|||||||
assertPeers(t, db, map[string][]string{testV4Prefix: nil})
|
assertPeers(t, db, map[string][]string{testV4Prefix: nil})
|
||||||
}
|
}
|
||||||
|
|
||||||
// TestPrefixHandlerRemovesTheRouteOnAnAnnouncementWithoutAnOrigin runs, through
|
|
||||||
// the prefix handler into a real database, a peer's announcement of a prefix
|
|
||||||
// with origin AS testASNB and then the peer's announcement of the prefix with
|
|
||||||
// an AS path that gives no origin: one whose last AS is 0, an empty one, or one
|
|
||||||
// that holds AS 0 before its end, which decodes as an empty path. As any later
|
|
||||||
// announcement from the peer does, the second replaces the peer's route, here
|
|
||||||
// with none, whether it comes in a later batch or follows a withdrawal in one
|
|
||||||
// batch. The in-memory route count and prefix distribution must match the
|
|
||||||
// route table.
|
|
||||||
func TestPrefixHandlerRemovesTheRouteOnAnAnnouncementWithoutAnOrigin(
|
|
||||||
t *testing.T,
|
|
||||||
) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
const prefixMaskLength = 24
|
|
||||||
|
|
||||||
noOrigin := func(path ristypes.ASPath) *ristypes.RISMessage {
|
|
||||||
msg := peerAnnouncement(testPeerIP, testV4Prefix)
|
|
||||||
msg.Path = path
|
|
||||||
|
|
||||||
return msg
|
|
||||||
}
|
|
||||||
endsInZero := ristypes.ASPath{Sequence: []int{testASNA, 0}}
|
|
||||||
withdrawal := &ristypes.RISMessage{
|
|
||||||
Type: ristypes.MessageTypeUpdate,
|
|
||||||
Peer: testPeerIP,
|
|
||||||
ParsedTimestamp: time.Date(2026, 1, 2, 3, 4, 5, 0, time.UTC),
|
|
||||||
Withdrawals: []string{testV4Prefix},
|
|
||||||
}
|
|
||||||
|
|
||||||
tests := []struct {
|
|
||||||
name string
|
|
||||||
// Handled and flushed as one batch after the announcement with origin
|
|
||||||
// AS testASNB.
|
|
||||||
batch []*ristypes.RISMessage
|
|
||||||
}{
|
|
||||||
{
|
|
||||||
name: "a path whose last AS is 0",
|
|
||||||
batch: []*ristypes.RISMessage{noOrigin(endsInZero)},
|
|
||||||
},
|
|
||||||
{
|
|
||||||
name: "an empty path",
|
|
||||||
batch: []*ristypes.RISMessage{noOrigin(ristypes.ASPath{})},
|
|
||||||
},
|
|
||||||
{
|
|
||||||
name: "a path with AS 0 before its end",
|
|
||||||
batch: []*ristypes.RISMessage{
|
|
||||||
noOrigin(decodePath(t, testASNA, 0, testASNB)),
|
|
||||||
},
|
|
||||||
},
|
|
||||||
{
|
|
||||||
name: "a withdrawal and then a path whose last AS is 0",
|
|
||||||
batch: []*ristypes.RISMessage{withdrawal, noOrigin(endsInZero)},
|
|
||||||
},
|
|
||||||
}
|
|
||||||
|
|
||||||
for _, tt := range tests {
|
|
||||||
t.Run(tt.name, func(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
db := handleBatches(t, [][]*ristypes.RISMessage{
|
|
||||||
{peerAnnouncement(testPeerIP, testV4Prefix)},
|
|
||||||
tt.batch,
|
|
||||||
})
|
|
||||||
checkLivePeers(t, db, testV4Prefix, prefixMaskLength, nil)
|
|
||||||
})
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestPrefixHandlerDoesNotStoreAPrefixThatDoesNotParse runs an announcement of
|
|
||||||
// "not-a-prefix" and a withdrawal of "also/bad" through the prefix handler into
|
|
||||||
// a real database. Neither is stored in the prefix tables: the prefix count in
|
|
||||||
// the stats stays 0, both as kept in memory and as counted from the tables when
|
|
||||||
// the database is opened again.
|
|
||||||
func TestPrefixHandlerDoesNotStoreAPrefixThatDoesNotParse(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
cfg := &config.Config{StateDir: t.TempDir()}
|
|
||||||
|
|
||||||
db, err := database.New(cfg, logger.New())
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("failed to create database: %v", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
// Built without NewPrefixHandler's flush timer, so the test flushes itself.
|
|
||||||
h := &PrefixHandler{db: db, logger: logger.New()}
|
|
||||||
|
|
||||||
withdrawal := &ristypes.RISMessage{
|
|
||||||
Type: ristypes.MessageTypeUpdate,
|
|
||||||
Peer: testPeerIP,
|
|
||||||
ParsedTimestamp: time.Date(2026, 1, 2, 3, 4, 5, 0, time.UTC),
|
|
||||||
Withdrawals: []string{"also/bad"},
|
|
||||||
}
|
|
||||||
h.HandleMessage(peerAnnouncement(testPeerIP, "not-a-prefix"))
|
|
||||||
h.HandleMessage(withdrawal)
|
|
||||||
h.flushBatch()
|
|
||||||
|
|
||||||
checkNoPrefixes := func(when string) {
|
|
||||||
t.Helper()
|
|
||||||
|
|
||||||
stats, err := db.GetStatsContext(context.Background())
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("GetStatsContext %s: %v", when, err)
|
|
||||||
}
|
|
||||||
|
|
||||||
if stats.Prefixes != 0 {
|
|
||||||
t.Errorf("prefix count %s = %d, want 0", when, stats.Prefixes)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
checkNoPrefixes("after the flush")
|
|
||||||
|
|
||||||
err = db.Close()
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("Close: %v", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
db, err = database.New(cfg, logger.New())
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("failed to open the database again: %v", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
t.Cleanup(func() { _ = db.Close() })
|
|
||||||
|
|
||||||
checkNoPrefixes("after opening the database again")
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestStateMessagesGoToThePrefixHandlerNotThePeerHandler checks that STATE
|
// TestStateMessagesGoToThePrefixHandlerNotThePeerHandler checks that STATE
|
||||||
// messages go to the prefix handler, which acts on a down, and not to the peer
|
// messages go to the prefix handler, which acts on a down, and not to the peer
|
||||||
// handler, which would record a peer whose session never comes up as active.
|
// handler, which would record a peer whose session never comes up as active.
|
||||||
|
|||||||
+76
-296
@@ -1,7 +1,7 @@
|
|||||||
package server
|
package server
|
||||||
|
|
||||||
import (
|
import (
|
||||||
"cmp"
|
"bytes"
|
||||||
"context"
|
"context"
|
||||||
"encoding/json"
|
"encoding/json"
|
||||||
"errors"
|
"errors"
|
||||||
@@ -9,16 +9,13 @@ import (
|
|||||||
"net/http"
|
"net/http"
|
||||||
"net/netip"
|
"net/netip"
|
||||||
"runtime"
|
"runtime"
|
||||||
"slices"
|
|
||||||
"sort"
|
"sort"
|
||||||
"strconv"
|
"strconv"
|
||||||
"strings"
|
"strings"
|
||||||
"time"
|
"time"
|
||||||
|
|
||||||
"git.eeqj.de/sneak/routewatch/internal/blocklist"
|
|
||||||
"git.eeqj.de/sneak/routewatch/internal/database"
|
"git.eeqj.de/sneak/routewatch/internal/database"
|
||||||
"git.eeqj.de/sneak/routewatch/internal/metrics"
|
"git.eeqj.de/sneak/routewatch/internal/metrics"
|
||||||
"git.eeqj.de/sneak/routewatch/internal/ristypes"
|
|
||||||
"git.eeqj.de/sneak/routewatch/internal/streamer"
|
"git.eeqj.de/sneak/routewatch/internal/streamer"
|
||||||
"git.eeqj.de/sneak/routewatch/internal/templates"
|
"git.eeqj.de/sneak/routewatch/internal/templates"
|
||||||
asinfo "git.eeqj.de/sneak/routewatch/pkg/asinfo"
|
asinfo "git.eeqj.de/sneak/routewatch/pkg/asinfo"
|
||||||
@@ -34,27 +31,6 @@ const (
|
|||||||
healthCheckTimeout = 2 * time.Second
|
healthCheckTimeout = 2 * time.Second
|
||||||
)
|
)
|
||||||
|
|
||||||
// Limits of one IP lookup.
|
|
||||||
const (
|
|
||||||
// maxLookupTargets is the most addresses and host names an IP lookup may
|
|
||||||
// list, and the most addresses it may have once the names are resolved:
|
|
||||||
// each address costs up to 113 database reads and a PTR lookup.
|
|
||||||
maxLookupTargets = 10
|
|
||||||
|
|
||||||
// ptrLookupTimeout is each PTR lookup's own time limit, so that ten
|
|
||||||
// addresses whose PTR lookups never answer fit in the 30-second limit.
|
|
||||||
ptrLookupTimeout = 2 * time.Second
|
|
||||||
)
|
|
||||||
|
|
||||||
// The Cache-Control header of an IP lookup answer, such as
|
|
||||||
// "public, max-age=900".
|
|
||||||
const (
|
|
||||||
cacheControlHeader = "Cache-Control"
|
|
||||||
cacheControlPublic = "public"
|
|
||||||
cacheControlPrivate = "private"
|
|
||||||
cacheControlMaxAge = ", max-age="
|
|
||||||
)
|
|
||||||
|
|
||||||
// Keys of the JSON envelope the handlers and middleware write, and the status
|
// Keys of the JSON envelope the handlers and middleware write, and the status
|
||||||
// value of an error answer.
|
// value of an error answer.
|
||||||
const (
|
const (
|
||||||
@@ -65,21 +41,10 @@ const (
|
|||||||
jsonStatusError = "error"
|
jsonStatusError = "error"
|
||||||
)
|
)
|
||||||
|
|
||||||
// Error messages an answer sends, in place of the error behind it where there
|
// Error messages an answer sends in place of the error that caused it.
|
||||||
// is one.
|
|
||||||
const (
|
const (
|
||||||
invalidPrefixMessage = "Invalid prefix"
|
invalidPrefixMessage = "Invalid prefix"
|
||||||
hostNotResolvedMessage = "could not resolve host name"
|
hostNotResolvedMessage = "could not resolve host name"
|
||||||
hostNotPublicMessage = "host name has no public address"
|
|
||||||
tooManyTargetsMessage = "Too many addresses or host names"
|
|
||||||
searchQueryMissingMessage = "Missing search query"
|
|
||||||
)
|
|
||||||
|
|
||||||
// The AS name search reads the text it looks for from the URL query parameter
|
|
||||||
// asSearchQueryParam, and answers with at most maxASSearchResults matches.
|
|
||||||
const (
|
|
||||||
asSearchQueryParam = "q"
|
|
||||||
maxASSearchResults = 50
|
|
||||||
)
|
)
|
||||||
|
|
||||||
// IP versions as database.LiveRoute.IPVersion records them.
|
// IP versions as database.LiveRoute.IPVersion records them.
|
||||||
@@ -156,10 +121,7 @@ func (s *Server) handleIndex() http.HandlerFunc {
|
|||||||
|
|
||||||
tmpl := templates.IndexTemplate()
|
tmpl := templates.IndexTemplate()
|
||||||
|
|
||||||
err := tmpl.Execute(w, indexPageData{
|
err := tmpl.Execute(w, pageData{CSPNonce: cspNonce(r.Context())})
|
||||||
CSPNonce: cspNonce(r.Context()),
|
|
||||||
ClientIP: extractClientIP(r),
|
|
||||||
})
|
|
||||||
if err != nil {
|
if err != nil {
|
||||||
s.logger.Error("Failed to render index template", "error", err)
|
s.logger.Error("Failed to render index template", "error", err)
|
||||||
http.Error(w, "Internal Server Error", http.StatusInternalServerError)
|
http.Error(w, "Internal Server Error", http.StatusInternalServerError)
|
||||||
@@ -173,15 +135,6 @@ type pageData struct {
|
|||||||
CSPNonce string
|
CSPNonce string
|
||||||
}
|
}
|
||||||
|
|
||||||
// indexPageData is what the home page is rendered with.
|
|
||||||
type indexPageData struct {
|
|
||||||
// CSPNonce is the nonce the page's script and style blocks carry.
|
|
||||||
CSPNonce string
|
|
||||||
// ClientIP is the visitor's address, which the IP information API card
|
|
||||||
// looks up: the address /ip/ would look up for the same request.
|
|
||||||
ClientIP string
|
|
||||||
}
|
|
||||||
|
|
||||||
// writeJSONError writes a standardized JSON error response with the given
|
// writeJSONError writes a standardized JSON error response with the given
|
||||||
// status code and error message.
|
// status code and error message.
|
||||||
func (s *Server) writeJSONError(w http.ResponseWriter, statusCode int, message string) {
|
func (s *Server) writeJSONError(w http.ResponseWriter, statusCode int, message string) {
|
||||||
@@ -464,10 +417,10 @@ type GCStats struct {
|
|||||||
//
|
//
|
||||||
//nolint:tagliatelle // field names are the existing JSON API
|
//nolint:tagliatelle // field names are the existing JSON API
|
||||||
type StreamStats struct {
|
type StreamStats struct {
|
||||||
Announcements uint64 `json:"announcements"`
|
Announcements uint64 `json:"announcements"`
|
||||||
Withdrawals uint64 `json:"withdrawals"`
|
Withdrawals uint64 `json:"withdrawals"`
|
||||||
RouteChurnPerMin int `json:"route_churn_per_min"`
|
RouteChurnPerSec float64 `json:"route_churn_per_sec"`
|
||||||
BGPPeerCount int `json:"bgp_peer_count"`
|
BGPPeerCount int `json:"bgp_peer_count"`
|
||||||
}
|
}
|
||||||
|
|
||||||
// StatsResponse represents the API statistics response
|
// StatsResponse represents the API statistics response
|
||||||
@@ -568,7 +521,7 @@ func (s *Server) streamStats() StreamStats {
|
|||||||
return StreamStats{
|
return StreamStats{
|
||||||
Announcements: metricsTracker.GetAnnouncementCount(),
|
Announcements: metricsTracker.GetAnnouncementCount(),
|
||||||
Withdrawals: metricsTracker.GetWithdrawalCount(),
|
Withdrawals: metricsTracker.GetWithdrawalCount(),
|
||||||
RouteChurnPerMin: metricsTracker.GetChurnPerMinute(),
|
RouteChurnPerSec: metricsTracker.GetChurnRate(),
|
||||||
BGPPeerCount: metricsTracker.GetBGPPeerCount(),
|
BGPPeerCount: metricsTracker.GetBGPPeerCount(),
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -703,24 +656,9 @@ func (s *Server) handleIPLookup() http.HandlerFunc {
|
|||||||
|
|
||||||
// IPLookupResponse is the standard response for IP/hostname lookups.
|
// IPLookupResponse is the standard response for IP/hostname lookups.
|
||||||
type IPLookupResponse struct {
|
type IPLookupResponse struct {
|
||||||
Query string `json:"query"`
|
Query string `json:"query"`
|
||||||
Results []IPLookupResult `json:"results"`
|
Results []*database.IPInfo `json:"results"`
|
||||||
Errors []string `json:"errors,omitempty"`
|
Errors []string `json:"errors,omitempty"`
|
||||||
}
|
|
||||||
|
|
||||||
// IPLookupResult is the answer for one address: its PTR names, its live route
|
|
||||||
// and that route's AS, and the blocklists that list it. IPInfo is nil when
|
|
||||||
// the address has no live route, and the answer then has no route or AS
|
|
||||||
// fields. IP hides the IPInfo field of the same name, so that an address with
|
|
||||||
// no route has it too.
|
|
||||||
type IPLookupResult struct {
|
|
||||||
*database.IPInfo
|
|
||||||
|
|
||||||
IP string `json:"ip"`
|
|
||||||
PTR []string `json:"ptr,omitempty"`
|
|
||||||
|
|
||||||
// Blocklists has one entry for each blocklist the address is on.
|
|
||||||
Blocklists []blocklist.Listing `json:"blocklists"`
|
|
||||||
}
|
}
|
||||||
|
|
||||||
// handleIPInfo returns a handler that provides comprehensive IP information.
|
// handleIPInfo returns a handler that provides comprehensive IP information.
|
||||||
@@ -729,7 +667,7 @@ type IPLookupResult struct {
|
|||||||
// Always returns the same response structure with PTR records for each IP.
|
// Always returns the same response structure with PTR records for each IP.
|
||||||
func (s *Server) handleIPInfo() http.HandlerFunc {
|
func (s *Server) handleIPInfo() http.HandlerFunc {
|
||||||
return func(w http.ResponseWriter, r *http.Request) {
|
return func(w http.ResponseWriter, r *http.Request) {
|
||||||
target, ownAddress := lookupTarget(r)
|
target := lookupTarget(r)
|
||||||
if target == "" {
|
if target == "" {
|
||||||
s.writeJSONError(w, http.StatusBadRequest, "Could not determine IP address")
|
s.writeJSONError(w, http.StatusBadRequest, "Could not determine IP address")
|
||||||
|
|
||||||
@@ -737,26 +675,20 @@ func (s *Server) handleIPInfo() http.HandlerFunc {
|
|||||||
}
|
}
|
||||||
|
|
||||||
ctx := r.Context()
|
ctx := r.Context()
|
||||||
response := IPLookupResponse{Query: target, Results: make([]IPLookupResult, 0)}
|
response := IPLookupResponse{Query: target, Results: make([]*database.IPInfo, 0)}
|
||||||
|
|
||||||
// Collect all IPs to look up
|
// Collect all IPs to look up
|
||||||
ipsToLookup, resolveErrors, err := s.resolveLookupTargets(ctx, target)
|
ipsToLookup, resolveErrors := s.resolveLookupTargets(ctx, target)
|
||||||
if deadlinePassed(ctx) {
|
if deadlinePassed(ctx) {
|
||||||
s.writeJSONError(w, http.StatusGatewayTimeout, requestTimeoutMessage)
|
s.writeJSONError(w, http.StatusGatewayTimeout, requestTimeoutMessage)
|
||||||
|
|
||||||
return
|
return
|
||||||
}
|
}
|
||||||
|
|
||||||
if err != nil {
|
|
||||||
s.writeJSONError(w, http.StatusBadRequest, tooManyTargetsMessage)
|
|
||||||
|
|
||||||
return
|
|
||||||
}
|
|
||||||
|
|
||||||
response.Errors = append(response.Errors, resolveErrors...)
|
response.Errors = append(response.Errors, resolveErrors...)
|
||||||
|
|
||||||
if len(ipsToLookup) == 0 {
|
if len(ipsToLookup) == 0 {
|
||||||
s.writeJSONError(w, http.StatusBadRequest, noTargetsMessage(resolveErrors))
|
s.writeJSONError(w, http.StatusBadRequest, "No valid IPs or hostnames provided")
|
||||||
|
|
||||||
return
|
return
|
||||||
}
|
}
|
||||||
@@ -774,10 +706,7 @@ func (s *Server) handleIPInfo() http.HandlerFunc {
|
|||||||
}
|
}
|
||||||
|
|
||||||
switch {
|
switch {
|
||||||
case errors.Is(err, database.ErrNoRoute):
|
case errors.Is(err, database.ErrNoRoute), errors.Is(err, database.ErrInvalidIP):
|
||||||
// The address still gets a result, without route fields.
|
|
||||||
ipInfo = nil
|
|
||||||
case errors.Is(err, database.ErrInvalidIP):
|
|
||||||
response.Errors = append(response.Errors, ip+": "+err.Error())
|
response.Errors = append(response.Errors, ip+": "+err.Error())
|
||||||
|
|
||||||
continue
|
continue
|
||||||
@@ -789,21 +718,20 @@ func (s *Server) handleIPInfo() http.HandlerFunc {
|
|||||||
}
|
}
|
||||||
|
|
||||||
// Do PTR lookup for this IP
|
// Do PTR lookup for this IP
|
||||||
ptrs := s.lookupPTRNames(ctx, ip)
|
ptrs := lookupPTRNames(ctx, ip)
|
||||||
if deadlinePassed(ctx) {
|
if deadlinePassed(ctx) {
|
||||||
s.writeJSONError(w, http.StatusGatewayTimeout, requestTimeoutMessage)
|
s.writeJSONError(w, http.StatusGatewayTimeout, requestTimeoutMessage)
|
||||||
|
|
||||||
return
|
return
|
||||||
}
|
}
|
||||||
|
|
||||||
response.Results = append(response.Results, IPLookupResult{
|
if len(ptrs) > 0 {
|
||||||
IP: ip,
|
ipInfo.PTR = ptrs
|
||||||
PTR: ptrs,
|
}
|
||||||
IPInfo: ipInfo,
|
|
||||||
Blocklists: s.blocklistListings(ip),
|
|
||||||
})
|
|
||||||
|
|
||||||
if ipInfo != nil && ipInfo.NeedsWHOISRefresh {
|
response.Results = append(response.Results, ipInfo)
|
||||||
|
|
||||||
|
if ipInfo.NeedsWHOISRefresh {
|
||||||
refreshASNs[ipInfo.ASN] = true
|
refreshASNs[ipInfo.ASN] = true
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -811,70 +739,23 @@ func (s *Server) handleIPInfo() http.HandlerFunc {
|
|||||||
// Queue WHOIS refresh for stale ASNs (non-blocking)
|
// Queue WHOIS refresh for stale ASNs (non-blocking)
|
||||||
s.queueWHOISRefresh(refreshASNs)
|
s.queueWHOISRefresh(refreshASNs)
|
||||||
|
|
||||||
s.writeIPLookupAnswer(w, response, ownAddress)
|
// Return response (even if no results, include errors)
|
||||||
}
|
if len(response.Results) == 0 && len(response.Errors) > 0 {
|
||||||
}
|
s.writeJSONError(w, http.StatusNotFound, "No routes found: "+response.Errors[0])
|
||||||
|
|
||||||
// blocklistListings returns the blocklists that list ip. It returns an empty
|
return
|
||||||
// list, never nil, so that an answer's blocklists is [] rather than null.
|
}
|
||||||
func (s *Server) blocklistListings(ip string) []blocklist.Listing {
|
|
||||||
addr, err := netip.ParseAddr(ip)
|
|
||||||
if s.blocklists == nil || err != nil {
|
|
||||||
return []blocklist.Listing{}
|
|
||||||
}
|
|
||||||
|
|
||||||
return s.blocklists.Lookup(addr)
|
err := writeJSONSuccess(w, response)
|
||||||
}
|
if err != nil {
|
||||||
|
s.logger.Error("Failed to encode IP lookup response", "error", err)
|
||||||
// setIPLookupCacheControl lets caches keep an IP lookup answer for the
|
}
|
||||||
// shortest blocklist interval, the soonest a blocklist copy can change. The
|
|
||||||
// answer about the caller's own address is private, so that a shared cache
|
|
||||||
// does not give it to another caller. Without blocklists it sets nothing.
|
|
||||||
func (s *Server) setIPLookupCacheControl(w http.ResponseWriter, ownAddress bool) {
|
|
||||||
if s.blocklists == nil {
|
|
||||||
return
|
|
||||||
}
|
|
||||||
|
|
||||||
visibility := cacheControlPublic
|
|
||||||
if ownAddress {
|
|
||||||
visibility = cacheControlPrivate
|
|
||||||
}
|
|
||||||
|
|
||||||
maxAgeSeconds := int(s.blocklists.ShortestInterval().Seconds())
|
|
||||||
w.Header().Set(cacheControlHeader,
|
|
||||||
visibility+cacheControlMaxAge+strconv.Itoa(maxAgeSeconds))
|
|
||||||
}
|
|
||||||
|
|
||||||
// noTargetsMessage returns the message of the 400 an IP lookup answers when
|
|
||||||
// no address is left to look up, followed by the first of resolveErrors, which
|
|
||||||
// says why, when there is one.
|
|
||||||
func noTargetsMessage(resolveErrors []string) string {
|
|
||||||
const message = "No valid IPs or hostnames provided"
|
|
||||||
|
|
||||||
if len(resolveErrors) == 0 {
|
|
||||||
return message
|
|
||||||
}
|
|
||||||
|
|
||||||
return message + ": " + resolveErrors[0]
|
|
||||||
}
|
|
||||||
|
|
||||||
// writeIPLookupAnswer sends response with its Cache-Control header.
|
|
||||||
// ownAddress says the request asked about the caller's own address.
|
|
||||||
func (s *Server) writeIPLookupAnswer(
|
|
||||||
w http.ResponseWriter, response IPLookupResponse, ownAddress bool,
|
|
||||||
) {
|
|
||||||
s.setIPLookupCacheControl(w, ownAddress)
|
|
||||||
|
|
||||||
err := writeJSONSuccess(w, response)
|
|
||||||
if err != nil {
|
|
||||||
s.logger.Error("Failed to encode IP lookup response", "error", err)
|
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// lookupTarget returns what an IP lookup request asks for: the ip or addr URL
|
// lookupTarget returns what an IP lookup request asks for: the ip or addr URL
|
||||||
// parameter, or else the client's own address. The bool is true for the
|
// parameter, or else the client's own address.
|
||||||
// client's own address.
|
func lookupTarget(r *http.Request) string {
|
||||||
func lookupTarget(r *http.Request) (string, bool) {
|
|
||||||
target := chi.URLParam(r, "ip")
|
target := chi.URLParam(r, "ip")
|
||||||
if target == "" {
|
if target == "" {
|
||||||
target = chi.URLParam(r, "addr")
|
target = chi.URLParam(r, "addr")
|
||||||
@@ -882,10 +763,10 @@ func lookupTarget(r *http.Request) (string, bool) {
|
|||||||
|
|
||||||
if target == "" {
|
if target == "" {
|
||||||
// Use client IP (RealIP middleware has already processed this)
|
// Use client IP (RealIP middleware has already processed this)
|
||||||
return extractClientIP(r), true
|
target = extractClientIP(r)
|
||||||
}
|
}
|
||||||
|
|
||||||
return target, false
|
return target
|
||||||
}
|
}
|
||||||
|
|
||||||
// queueWHOISRefresh asks the WHOIS fetcher, when there is one, to refresh
|
// queueWHOISRefresh asks the WHOIS fetcher, when there is one, to refresh
|
||||||
@@ -900,16 +781,15 @@ func (s *Server) queueWHOISRefresh(asns map[int]bool) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// errTooManyTargets is what resolveLookupTargets returns for a list of more
|
// resolveLookupTargets splits a comma-separated list of IP addresses and
|
||||||
// than maxLookupTargets addresses and host names, or of names that resolve to
|
// hostnames and returns the IP addresses to look up, resolving each hostname.
|
||||||
// more than maxLookupTargets addresses.
|
// A hostname that does not resolve is reported in the returned errors instead,
|
||||||
var errTooManyTargets = errors.New("too many addresses or host names")
|
// with the resolver's own error only in the log, since it names the DNS server.
|
||||||
|
// It stops once ctx's deadline has passed; the caller then answers 504.
|
||||||
// splitLookupTargets returns the addresses and host names in a comma-separated
|
func (s *Server) resolveLookupTargets(
|
||||||
// list, leaving out blank entries, or errTooManyTargets when there are more
|
ctx context.Context, target string,
|
||||||
// than maxLookupTargets.
|
) ([]string, []string) {
|
||||||
func splitLookupTargets(target string) ([]string, error) {
|
var ipsToLookup, resolveErrors []string
|
||||||
var targets []string
|
|
||||||
|
|
||||||
for t := range strings.SplitSeq(target, ",") {
|
for t := range strings.SplitSeq(target, ",") {
|
||||||
t = strings.TrimSpace(t)
|
t = strings.TrimSpace(t)
|
||||||
@@ -917,36 +797,6 @@ func splitLookupTargets(target string) ([]string, error) {
|
|||||||
continue
|
continue
|
||||||
}
|
}
|
||||||
|
|
||||||
if len(targets) == maxLookupTargets {
|
|
||||||
return nil, errTooManyTargets
|
|
||||||
}
|
|
||||||
|
|
||||||
targets = append(targets, t)
|
|
||||||
}
|
|
||||||
|
|
||||||
return targets, nil
|
|
||||||
}
|
|
||||||
|
|
||||||
// resolveLookupTargets splits a comma-separated list of IP addresses and
|
|
||||||
// hostnames and returns the IP addresses to look up, resolving each hostname.
|
|
||||||
// A hostname that does not resolve is reported in the returned errors instead,
|
|
||||||
// with the resolver's own error only in the log, since it names the DNS server.
|
|
||||||
// Only the public addresses a hostname resolves to are kept, so the answer
|
|
||||||
// does not show the deployment's private addresses; a hostname with none is
|
|
||||||
// reported in the errors. More than maxLookupTargets entries, or addresses once
|
|
||||||
// resolved, return errTooManyTargets. It stops once ctx's deadline has passed;
|
|
||||||
// the caller then answers 504.
|
|
||||||
func (s *Server) resolveLookupTargets(
|
|
||||||
ctx context.Context, target string,
|
|
||||||
) ([]string, []string, error) {
|
|
||||||
targets, err := splitLookupTargets(target)
|
|
||||||
if err != nil {
|
|
||||||
return nil, nil, err
|
|
||||||
}
|
|
||||||
|
|
||||||
var ipsToLookup, resolveErrors []string
|
|
||||||
|
|
||||||
for _, t := range targets {
|
|
||||||
// Check if this target is an IP address
|
// Check if this target is an IP address
|
||||||
if net.ParseIP(t) != nil {
|
if net.ParseIP(t) != nil {
|
||||||
ipsToLookup = append(ipsToLookup, t)
|
ipsToLookup = append(ipsToLookup, t)
|
||||||
@@ -955,9 +805,9 @@ func (s *Server) resolveLookupTargets(
|
|||||||
}
|
}
|
||||||
|
|
||||||
// It's a hostname - resolve it
|
// It's a hostname - resolve it
|
||||||
resolved, err := s.resolver.LookupHost(ctx, t)
|
resolved, err := net.DefaultResolver.LookupHost(ctx, t)
|
||||||
if deadlinePassed(ctx) {
|
if deadlinePassed(ctx) {
|
||||||
return nil, nil, nil
|
return nil, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
if err != nil {
|
if err != nil {
|
||||||
@@ -967,45 +817,16 @@ func (s *Server) resolveLookupTargets(
|
|||||||
continue
|
continue
|
||||||
}
|
}
|
||||||
|
|
||||||
public := publicAddresses(resolved)
|
ipsToLookup = append(ipsToLookup, resolved...)
|
||||||
if len(public) == 0 {
|
|
||||||
resolveErrors = append(resolveErrors, t+": "+hostNotPublicMessage)
|
|
||||||
|
|
||||||
continue
|
|
||||||
}
|
|
||||||
|
|
||||||
ipsToLookup = append(ipsToLookup, public...)
|
|
||||||
}
|
}
|
||||||
|
|
||||||
if len(ipsToLookup) > maxLookupTargets {
|
return ipsToLookup, resolveErrors
|
||||||
return nil, nil, errTooManyTargets
|
|
||||||
}
|
|
||||||
|
|
||||||
return ipsToLookup, resolveErrors, nil
|
|
||||||
}
|
|
||||||
|
|
||||||
// publicAddresses returns the addresses in addrs that are global unicast and
|
|
||||||
// not private; Go counts the private ranges as global unicast.
|
|
||||||
func publicAddresses(addrs []string) []string {
|
|
||||||
var public []string
|
|
||||||
|
|
||||||
for _, a := range addrs {
|
|
||||||
ip, err := netip.ParseAddr(a)
|
|
||||||
if err == nil && ip.IsGlobalUnicast() && !ip.IsPrivate() {
|
|
||||||
public = append(public, a)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
return public
|
|
||||||
}
|
}
|
||||||
|
|
||||||
// lookupPTRNames returns the PTR records of ip without their trailing dots, or
|
// lookupPTRNames returns the PTR records of ip without their trailing dots, or
|
||||||
// nil when the lookup fails or runs past ptrLookupTimeout.
|
// nil when the lookup fails.
|
||||||
func (s *Server) lookupPTRNames(ctx context.Context, ip string) []string {
|
func lookupPTRNames(ctx context.Context, ip string) []string {
|
||||||
ctx, cancel := context.WithTimeout(ctx, ptrLookupTimeout)
|
ptrs, err := net.DefaultResolver.LookupAddr(ctx, ip)
|
||||||
defer cancel()
|
|
||||||
|
|
||||||
ptrs, err := s.resolver.LookupAddr(ctx, ip)
|
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
@@ -1079,12 +900,11 @@ func (s *Server) handleASDetailJSON() http.HandlerFunc {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// splitRoutesByIPVersion groups routes into the IPv4 and the IPv6 ones. Neither
|
// splitRoutesByIPVersion groups routes into the IPv4 and the IPv6 ones.
|
||||||
// group is nil, so that a JSON answer holds [] for an empty one and not null.
|
|
||||||
func splitRoutesByIPVersion(
|
func splitRoutesByIPVersion(
|
||||||
routes []database.LiveRoute,
|
routes []database.LiveRoute,
|
||||||
) ([]database.LiveRoute, []database.LiveRoute) {
|
) ([]database.LiveRoute, []database.LiveRoute) {
|
||||||
ipv4Routes, ipv6Routes := []database.LiveRoute{}, []database.LiveRoute{}
|
var ipv4Routes, ipv6Routes []database.LiveRoute
|
||||||
|
|
||||||
for _, route := range routes {
|
for _, route := range routes {
|
||||||
if route.IPVersion == ipVersionV4 {
|
if route.IPVersion == ipVersionV4 {
|
||||||
@@ -1097,15 +917,24 @@ func splitRoutesByIPVersion(
|
|||||||
return ipv4Routes, ipv6Routes
|
return ipv4Routes, ipv6Routes
|
||||||
}
|
}
|
||||||
|
|
||||||
// sortRoutesByMaskLength sorts routes by mask length, shortest first, and
|
// sortRoutesByNetwork sorts routes by network address, then by mask length.
|
||||||
// routes with the same mask length by network address.
|
func sortRoutesByNetwork(routes []database.LiveRoute) {
|
||||||
func sortRoutesByMaskLength(routes []database.LiveRoute) {
|
sort.Slice(routes, func(i, j int) bool {
|
||||||
slices.SortFunc(routes, func(a, b database.LiveRoute) int {
|
// Parse the prefixes to compare network addresses
|
||||||
prefixA, _ := netip.ParsePrefix(a.Prefix)
|
ipI, netI, _ := net.ParseCIDR(routes[i].Prefix)
|
||||||
prefixB, _ := netip.ParsePrefix(b.Prefix)
|
ipJ, netJ, _ := net.ParseCIDR(routes[j].Prefix)
|
||||||
|
|
||||||
return cmp.Or(cmp.Compare(prefixA.Bits(), prefixB.Bits()),
|
// Compare by network address first
|
||||||
prefixA.Addr().Compare(prefixB.Addr()))
|
cmp := bytes.Compare(ipI.To16(), ipJ.To16())
|
||||||
|
if cmp != 0 {
|
||||||
|
return cmp < 0
|
||||||
|
}
|
||||||
|
|
||||||
|
// If network addresses are equal, compare by mask length
|
||||||
|
onesI, _ := netI.Mask.Size()
|
||||||
|
onesJ, _ := netJ.Mask.Size()
|
||||||
|
|
||||||
|
return onesI < onesJ
|
||||||
})
|
})
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -1175,56 +1004,6 @@ func (s *Server) handlePrefixDetailJSON() http.HandlerFunc {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// ASSearchResponse is the answer to an AS name search: the text searched for,
|
|
||||||
// the number of ASes that match it, and the first maxASSearchResults of them in
|
|
||||||
// AS number order.
|
|
||||||
type ASSearchResponse struct {
|
|
||||||
Query string `json:"query"`
|
|
||||||
Total int `json:"total"`
|
|
||||||
Results []asinfo.Info `json:"results"`
|
|
||||||
}
|
|
||||||
|
|
||||||
// handleASSearch returns a handler that searches the AS list built into
|
|
||||||
// asinfo for the text in the q parameter, without its leading and trailing
|
|
||||||
// spaces. It answers 400 when no text is left.
|
|
||||||
func (s *Server) handleASSearch() http.HandlerFunc {
|
|
||||||
return func(w http.ResponseWriter, r *http.Request) {
|
|
||||||
query := strings.TrimSpace(r.URL.Query().Get(asSearchQueryParam))
|
|
||||||
if query == "" {
|
|
||||||
s.writeJSONError(w, http.StatusBadRequest, searchQueryMissingMessage)
|
|
||||||
|
|
||||||
return
|
|
||||||
}
|
|
||||||
|
|
||||||
// AS 0 means no known origin and has no AS page, so the answer leaves
|
|
||||||
// it out. The matches come in AS number order, so AS 0 can only be the
|
|
||||||
// first. The search asks for one match more than the answer holds, to
|
|
||||||
// take its place.
|
|
||||||
results, total := asinfo.Search(query, maxASSearchResults+1)
|
|
||||||
if len(results) > 0 && results[0].ASN == ristypes.NoOrigin {
|
|
||||||
results = results[1:]
|
|
||||||
total--
|
|
||||||
}
|
|
||||||
|
|
||||||
results = results[:min(len(results), maxASSearchResults)]
|
|
||||||
|
|
||||||
// results is an empty list rather than nil when nothing matches, so
|
|
||||||
// that the answer holds [] and not null.
|
|
||||||
if results == nil {
|
|
||||||
results = []asinfo.Info{}
|
|
||||||
}
|
|
||||||
|
|
||||||
err := writeJSONSuccess(w, ASSearchResponse{
|
|
||||||
Query: query,
|
|
||||||
Total: total,
|
|
||||||
Results: results,
|
|
||||||
})
|
|
||||||
if err != nil {
|
|
||||||
s.logger.Error("Failed to encode AS search results", "error", err)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// handleASDetail returns a handler that serves the AS detail HTML page
|
// handleASDetail returns a handler that serves the AS detail HTML page
|
||||||
func (s *Server) handleASDetail() http.HandlerFunc {
|
func (s *Server) handleASDetail() http.HandlerFunc {
|
||||||
return func(w http.ResponseWriter, r *http.Request) {
|
return func(w http.ResponseWriter, r *http.Request) {
|
||||||
@@ -1272,8 +1051,9 @@ func (s *Server) handleASDetail() http.HandlerFunc {
|
|||||||
// Group prefixes by IP version
|
// Group prefixes by IP version
|
||||||
ipv4Prefixes, ipv6Prefixes := splitRoutesByIPVersion(prefixes)
|
ipv4Prefixes, ipv6Prefixes := splitRoutesByIPVersion(prefixes)
|
||||||
|
|
||||||
sortRoutesByMaskLength(ipv4Prefixes)
|
// Sort prefixes by network address
|
||||||
sortRoutesByMaskLength(ipv6Prefixes)
|
sortRoutesByNetwork(ipv4Prefixes)
|
||||||
|
sortRoutesByNetwork(ipv6Prefixes)
|
||||||
|
|
||||||
// Prepare template data
|
// Prepare template data
|
||||||
data := struct {
|
data := struct {
|
||||||
|
|||||||
File diff suppressed because it is too large
Load Diff
@@ -142,9 +142,8 @@ func deadlinePassed(ctx context.Context) bool {
|
|||||||
//
|
//
|
||||||
// If a response is not valid JSON or is empty when JSON is expected, the middleware
|
// If a response is not valid JSON or is empty when JSON is expected, the middleware
|
||||||
// returns a properly formatted JSON error response. If the request's time limit
|
// returns a properly formatted JSON error response. If the request's time limit
|
||||||
// has passed, it answers 504 with the error message "Request timeout". A 4xx
|
// has passed, it answers 504 with the error message "Request timeout". For other
|
||||||
// keeps its status, with the status's standard text as the message. For other
|
// errors, it returns "Internal server error" with status 500 if the original
|
||||||
// errors, it returns "Internal server error", with status 500 if the original
|
|
||||||
// status was 200.
|
// status was 200.
|
||||||
//
|
//
|
||||||
// Non-JSON responses are passed through unchanged.
|
// Non-JSON responses are passed through unchanged.
|
||||||
@@ -204,13 +203,8 @@ func JSONValidationMiddleware(next http.Handler) http.Handler {
|
|||||||
w.WriteHeader(statusCode)
|
w.WriteHeader(statusCode)
|
||||||
|
|
||||||
errorMsg := internalErrorMessage
|
errorMsg := internalErrorMessage
|
||||||
|
if statusCode == http.StatusGatewayTimeout {
|
||||||
switch {
|
|
||||||
case statusCode == http.StatusGatewayTimeout:
|
|
||||||
errorMsg = requestTimeoutMessage
|
errorMsg = requestTimeoutMessage
|
||||||
case statusCode >= http.StatusBadRequest &&
|
|
||||||
statusCode < http.StatusInternalServerError:
|
|
||||||
errorMsg = http.StatusText(statusCode)
|
|
||||||
}
|
}
|
||||||
|
|
||||||
response := map[string]any{
|
response := map[string]any{
|
||||||
|
|||||||
@@ -11,12 +11,11 @@ import (
|
|||||||
)
|
)
|
||||||
|
|
||||||
// The addresses the forwarding tests use: the client a proxy names in
|
// The addresses the forwarding tests use: the client a proxy names in
|
||||||
// X-Forwarded-For, the range of trusted proxies, a peer inside it, and a peer
|
// X-Forwarded-For, a peer inside the trusted range of
|
||||||
// outside it, which lies in testPrefixV4 so that an IP lookup of it finds a
|
// TestRealIPMiddlewareTrustsOnlyConfiguredProxies, and a peer outside it,
|
||||||
// route.
|
// which lies in testPrefixV4 so that an IP lookup of it finds a route.
|
||||||
const (
|
const (
|
||||||
forwardedClient = "203.0.113.9"
|
forwardedClient = "203.0.113.9"
|
||||||
trustedProxyRange = "10.0.0.0/8"
|
|
||||||
trustedPeer = "10.0.0.2:1234"
|
trustedPeer = "10.0.0.2:1234"
|
||||||
untrustedPeerAddr = "192.0.2.1"
|
untrustedPeerAddr = "192.0.2.1"
|
||||||
untrustedPeer = untrustedPeerAddr + ":1234"
|
untrustedPeer = untrustedPeerAddr + ":1234"
|
||||||
@@ -59,34 +58,6 @@ func TestResponsesCarrySecurityHeaders(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// TestJSONValidationKeeps4xxWithoutBody wraps a handler that answers 400
|
|
||||||
// without a body in JSONValidationMiddleware and checks that the JSON error
|
|
||||||
// body keeps the 400 and gives the status's own text. Before the fix it said
|
|
||||||
// "Internal server error"
|
|
||||||
// (https://git.eeqj.de/sneak/routewatch/issues/111).
|
|
||||||
func TestJSONValidationKeeps4xxWithoutBody(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
handler := JSONValidationMiddleware(http.HandlerFunc(
|
|
||||||
func(w http.ResponseWriter, _ *http.Request) {
|
|
||||||
w.WriteHeader(http.StatusBadRequest)
|
|
||||||
}))
|
|
||||||
|
|
||||||
rec := httptest.NewRecorder()
|
|
||||||
handler.ServeHTTP(rec, httptest.NewRequestWithContext(
|
|
||||||
t.Context(), http.MethodGet, "/", nil))
|
|
||||||
|
|
||||||
const name = "400 without a body"
|
|
||||||
|
|
||||||
if rec.Code != http.StatusBadRequest {
|
|
||||||
t.Errorf("%s: status %d, want %d; body %s",
|
|
||||||
name, rec.Code, http.StatusBadRequest, rec.Body)
|
|
||||||
}
|
|
||||||
|
|
||||||
checkJSONErrorCode(t, name, rec, http.StatusBadRequest)
|
|
||||||
checkJSONErrorMessage(t, name, rec, http.StatusText(http.StatusBadRequest))
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestPagesCarryTheCSPNonce checks that every script and style block of each
|
// TestPagesCarryTheCSPNonce checks that every script and style block of each
|
||||||
// page carries the nonce its Content-Security-Policy header names, so that the
|
// page carries the nonce its Content-Security-Policy header names, so that the
|
||||||
// browser runs and applies them, that each answer has a nonce of its own, and
|
// browser runs and applies them, that each answer has a nonce of its own, and
|
||||||
@@ -190,7 +161,7 @@ func TestForwardedForFromUntrustedPeerIsIgnored(t *testing.T) {
|
|||||||
func TestRealIPMiddlewareTrustsOnlyConfiguredProxies(t *testing.T) {
|
func TestRealIPMiddlewareTrustsOnlyConfiguredProxies(t *testing.T) {
|
||||||
t.Parallel()
|
t.Parallel()
|
||||||
|
|
||||||
trusted := []netip.Prefix{netip.MustParsePrefix(trustedProxyRange)}
|
trusted := []netip.Prefix{netip.MustParsePrefix("10.0.0.0/8")}
|
||||||
|
|
||||||
tests := []struct {
|
tests := []struct {
|
||||||
name string
|
name string
|
||||||
|
|||||||
@@ -1,33 +1,18 @@
|
|||||||
package server
|
package server
|
||||||
|
|
||||||
import (
|
import (
|
||||||
"net/http"
|
|
||||||
"time"
|
"time"
|
||||||
|
|
||||||
"github.com/go-chi/chi/v5"
|
"github.com/go-chi/chi/v5"
|
||||||
"github.com/go-chi/chi/v5/middleware"
|
"github.com/go-chi/chi/v5/middleware"
|
||||||
)
|
)
|
||||||
|
|
||||||
// The error messages the JSON routes under /api/v1 and /ip answer with when no
|
|
||||||
// route matches the path, and when none takes the method.
|
|
||||||
const (
|
|
||||||
notFoundMessage = "Not found"
|
|
||||||
methodNotAllowedMessage = "Method not allowed"
|
|
||||||
|
|
||||||
// jsonRoutesAllow is the Allow header of their 405 answer: every one of
|
|
||||||
// them takes GET and HEAD, and no other method.
|
|
||||||
jsonRoutesAllow = "GET, HEAD"
|
|
||||||
)
|
|
||||||
|
|
||||||
// setupRoutes configures the HTTP routes
|
// setupRoutes configures the HTTP routes
|
||||||
func (s *Server) setupRoutes() {
|
func (s *Server) setupRoutes() {
|
||||||
r := chi.NewRouter()
|
r := chi.NewRouter()
|
||||||
|
|
||||||
// Middleware
|
// Middleware
|
||||||
r.Use(DecodedPathMiddleware)
|
r.Use(DecodedPathMiddleware)
|
||||||
// GetHead serves HEAD with the GET handler. It looks the route up by the
|
|
||||||
// path DecodedPathMiddleware sets, so it comes after it.
|
|
||||||
r.Use(middleware.GetHead)
|
|
||||||
r.Use(SecurityHeadersMiddleware)
|
r.Use(SecurityHeadersMiddleware)
|
||||||
r.Use(middleware.RequestID)
|
r.Use(middleware.RequestID)
|
||||||
r.Use(RealIPMiddleware(s.trustedProxies))
|
r.Use(RealIPMiddleware(s.trustedProxies))
|
||||||
@@ -55,39 +40,20 @@ func (s *Server) setupRoutes() {
|
|||||||
r.Get("/prefixlength/{length}", s.handlePrefixLength())
|
r.Get("/prefixlength/{length}", s.handlePrefixLength())
|
||||||
r.Get("/prefixlength6/{length}", s.handlePrefixLength6())
|
r.Get("/prefixlength6/{length}", s.handlePrefixLength6())
|
||||||
|
|
||||||
// The JSON routes answer a path or method they do not take with the JSON
|
|
||||||
// error body, as they answer everything else.
|
|
||||||
jsonNotFound := func(w http.ResponseWriter, _ *http.Request) {
|
|
||||||
s.writeJSONError(w, http.StatusNotFound, notFoundMessage)
|
|
||||||
}
|
|
||||||
jsonMethodNotAllowed := func(w http.ResponseWriter, _ *http.Request) {
|
|
||||||
// chi's own 405 answer sets Allow; one set in its place must too.
|
|
||||||
w.Header().Set("Allow", jsonRoutesAllow)
|
|
||||||
s.writeJSONError(w, http.StatusMethodNotAllowed, methodNotAllowedMessage)
|
|
||||||
}
|
|
||||||
|
|
||||||
// IP info JSON endpoints (replaces old /ip redirect)
|
// IP info JSON endpoints (replaces old /ip redirect)
|
||||||
r.Route("/ip", func(r chi.Router) {
|
r.Route("/ip", func(r chi.Router) {
|
||||||
r.Use(JSONValidationMiddleware)
|
r.Use(JSONValidationMiddleware)
|
||||||
r.NotFound(jsonNotFound)
|
r.Get("/", s.handleIPInfo()) // Client IP
|
||||||
r.MethodNotAllowed(jsonMethodNotAllowed)
|
|
||||||
r.Get("/", s.handleIPInfo()) // Client IP
|
|
||||||
// GetHead does not reach this route: mounting the group registers
|
|
||||||
// /ip and /ip/ for every method, so HEAD arrives here as HEAD.
|
|
||||||
r.Head("/", s.handleIPInfo())
|
|
||||||
r.Get("/{addr}", s.handleIPInfo()) // Specified IP
|
r.Get("/{addr}", s.handleIPInfo()) // Specified IP
|
||||||
})
|
})
|
||||||
|
|
||||||
// API routes
|
// API routes
|
||||||
r.Route("/api/v1", func(r chi.Router) {
|
r.Route("/api/v1", func(r chi.Router) {
|
||||||
r.Use(JSONValidationMiddleware)
|
r.Use(JSONValidationMiddleware)
|
||||||
r.NotFound(jsonNotFound)
|
|
||||||
r.MethodNotAllowed(jsonMethodNotAllowed)
|
|
||||||
r.Get("/stats", s.handleStats())
|
r.Get("/stats", s.handleStats())
|
||||||
r.Get("/ip/{ip}", s.handleIPLookup())
|
r.Get("/ip/{ip}", s.handleIPLookup())
|
||||||
r.Get("/as/{asn}", s.handleASDetailJSON())
|
r.Get("/as/{asn}", s.handleASDetailJSON())
|
||||||
r.Get("/prefix/{prefix}/{len}", s.handlePrefixDetailJSON())
|
r.Get("/prefix/{prefix}/{len}", s.handlePrefixDetailJSON())
|
||||||
r.Get("/search/as", s.handleASSearch())
|
|
||||||
})
|
})
|
||||||
|
|
||||||
s.router = r
|
s.router = r
|
||||||
|
|||||||
@@ -1,100 +0,0 @@
|
|||||||
package server
|
|
||||||
|
|
||||||
import (
|
|
||||||
"encoding/json"
|
|
||||||
"net/http"
|
|
||||||
"net/http/httptest"
|
|
||||||
"testing"
|
|
||||||
)
|
|
||||||
|
|
||||||
// TestRoutesAnswerHEAD sends HEAD for a page and JSON routes through the
|
|
||||||
// server's router and checks that each answers as GET does. /ip and /ip/, the
|
|
||||||
// lookup of the caller's own address, are in the list because chi's GetHead
|
|
||||||
// does not reach them. Before the fix every route answered HEAD with 405
|
|
||||||
// (https://git.eeqj.de/sneak/routewatch/issues/111).
|
|
||||||
func TestRoutesAnswerHEAD(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
s := newServerWithTestRoutes(t)
|
|
||||||
|
|
||||||
for _, path := range []string{"/", statsPath, "/ip", "/ip/"} {
|
|
||||||
rec := httptest.NewRecorder()
|
|
||||||
s.router.ServeHTTP(rec, httptest.NewRequestWithContext(
|
|
||||||
t.Context(), http.MethodHead, path, nil))
|
|
||||||
|
|
||||||
if rec.Code != http.StatusOK {
|
|
||||||
t.Errorf("HEAD %s: status %d, want %d; body %s",
|
|
||||||
path, rec.Code, http.StatusOK, rec.Body)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestJSONRoutesAnswerUnmatchedRequestsWithJSONErrors sends a method the JSON
|
|
||||||
// routes do not take, and a path none of them matches, to each group of JSON
|
|
||||||
// routes through the server's router. Each must answer with the JSON error
|
|
||||||
// body, and a 405 must name the methods the routes take in its Allow header.
|
|
||||||
// Before the fix a 405 said "Internal server error" and a 404 was chi's plain
|
|
||||||
// text answer (https://git.eeqj.de/sneak/routewatch/issues/111).
|
|
||||||
func TestJSONRoutesAnswerUnmatchedRequestsWithJSONErrors(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
s := newServerWithTestRoutes(t)
|
|
||||||
|
|
||||||
tests := []struct {
|
|
||||||
method string
|
|
||||||
path string
|
|
||||||
wantCode int
|
|
||||||
wantMsg string
|
|
||||||
}{
|
|
||||||
{http.MethodPost, statsPath, http.StatusMethodNotAllowed, methodNotAllowedMessage},
|
|
||||||
{http.MethodPost, "/ip/", http.StatusMethodNotAllowed, methodNotAllowedMessage},
|
|
||||||
{http.MethodGet, "/api/v1/nothing", http.StatusNotFound, notFoundMessage},
|
|
||||||
{http.MethodGet, "/ip/192.0.2.1/nothing", http.StatusNotFound, notFoundMessage},
|
|
||||||
}
|
|
||||||
|
|
||||||
for _, tt := range tests {
|
|
||||||
name := tt.method + " " + tt.path
|
|
||||||
|
|
||||||
rec := httptest.NewRecorder()
|
|
||||||
s.router.ServeHTTP(rec, httptest.NewRequestWithContext(
|
|
||||||
t.Context(), tt.method, tt.path, nil))
|
|
||||||
|
|
||||||
if rec.Code != tt.wantCode {
|
|
||||||
t.Errorf("%s: status %d, want %d; body %s",
|
|
||||||
name, rec.Code, tt.wantCode, rec.Body)
|
|
||||||
}
|
|
||||||
|
|
||||||
checkJSONErrorCode(t, name, rec, tt.wantCode)
|
|
||||||
checkJSONErrorMessage(t, name, rec, tt.wantMsg)
|
|
||||||
|
|
||||||
allow := rec.Header().Get("Allow")
|
|
||||||
if tt.wantCode == http.StatusMethodNotAllowed && allow != jsonRoutesAllow {
|
|
||||||
t.Errorf("%s: Allow %q, want %q", name, allow, jsonRoutesAllow)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// checkJSONErrorMessage checks that rec holds the JSON error body with the
|
|
||||||
// given message.
|
|
||||||
func checkJSONErrorMessage(
|
|
||||||
t *testing.T, name string, rec *httptest.ResponseRecorder, want string,
|
|
||||||
) {
|
|
||||||
t.Helper()
|
|
||||||
|
|
||||||
var body struct {
|
|
||||||
Error struct {
|
|
||||||
Msg string `json:"msg"`
|
|
||||||
} `json:"error"`
|
|
||||||
}
|
|
||||||
|
|
||||||
err := json.Unmarshal(rec.Body.Bytes(), &body)
|
|
||||||
if err != nil {
|
|
||||||
t.Errorf("%s: decoding the answer %q: %v", name, rec.Body, err)
|
|
||||||
|
|
||||||
return
|
|
||||||
}
|
|
||||||
|
|
||||||
if body.Error.Msg != want {
|
|
||||||
t.Errorf("%s: error message %q, want %q", name, body.Error.Msg, want)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
@@ -1,8 +1,4 @@
|
|||||||
// Package server serves RouteWatch's web pages and JSON API:
|
// Package server provides HTTP endpoints for status monitoring and statistics
|
||||||
// - the home page, the status figures and the health check
|
|
||||||
// - the AS, prefix and prefix length pages, and the JSON answers behind them
|
|
||||||
// - the IP lookups
|
|
||||||
// - the AS name search
|
|
||||||
package server
|
package server
|
||||||
|
|
||||||
import (
|
import (
|
||||||
@@ -15,7 +11,6 @@ import (
|
|||||||
"strconv"
|
"strconv"
|
||||||
"time"
|
"time"
|
||||||
|
|
||||||
"git.eeqj.de/sneak/routewatch/internal/blocklist"
|
|
||||||
"git.eeqj.de/sneak/routewatch/internal/config"
|
"git.eeqj.de/sneak/routewatch/internal/config"
|
||||||
"git.eeqj.de/sneak/routewatch/internal/database"
|
"git.eeqj.de/sneak/routewatch/internal/database"
|
||||||
"git.eeqj.de/sneak/routewatch/internal/logger"
|
"git.eeqj.de/sneak/routewatch/internal/logger"
|
||||||
@@ -37,20 +32,6 @@ type ASNFetcher interface {
|
|||||||
GetStats() ASNFetcherStats
|
GetStats() ASNFetcherStats
|
||||||
}
|
}
|
||||||
|
|
||||||
// Blocklists tells which public blocklists list an address, from copies it
|
|
||||||
// keeps in memory, as *blocklist.Blocklists does.
|
|
||||||
type Blocklists interface {
|
|
||||||
Lookup(addr netip.Addr) []blocklist.Listing
|
|
||||||
ShortestInterval() time.Duration
|
|
||||||
}
|
|
||||||
|
|
||||||
// dnsResolver looks up the addresses of host names and the names of addresses
|
|
||||||
// in DNS, as *net.Resolver does.
|
|
||||||
type dnsResolver interface {
|
|
||||||
LookupHost(ctx context.Context, host string) ([]string, error)
|
|
||||||
LookupAddr(ctx context.Context, addr string) ([]string, error)
|
|
||||||
}
|
|
||||||
|
|
||||||
// Server provides HTTP endpoints for status monitoring
|
// Server provides HTTP endpoints for status monitoring
|
||||||
type Server struct {
|
type Server struct {
|
||||||
router *chi.Mux
|
router *chi.Mux
|
||||||
@@ -60,15 +41,10 @@ type Server struct {
|
|||||||
port int
|
port int
|
||||||
srv *http.Server
|
srv *http.Server
|
||||||
asnFetcher ASNFetcher
|
asnFetcher ASNFetcher
|
||||||
blocklists Blocklists
|
|
||||||
|
|
||||||
// trustedProxies are the reverse proxies whose X-Forwarded-For header
|
// trustedProxies are the reverse proxies whose X-Forwarded-For header
|
||||||
// RealIPMiddleware believes
|
// RealIPMiddleware believes
|
||||||
trustedProxies []netip.Prefix
|
trustedProxies []netip.Prefix
|
||||||
|
|
||||||
// resolver makes the IP lookup's DNS lookups: net.DefaultResolver, or a
|
|
||||||
// fake in tests
|
|
||||||
resolver dnsResolver
|
|
||||||
}
|
}
|
||||||
|
|
||||||
// New creates a new HTTP server
|
// New creates a new HTTP server
|
||||||
@@ -84,7 +60,6 @@ func New(
|
|||||||
logger: logger,
|
logger: logger,
|
||||||
port: cfg.Port,
|
port: cfg.Port,
|
||||||
trustedProxies: cfg.TrustedProxies,
|
trustedProxies: cfg.TrustedProxies,
|
||||||
resolver: net.DefaultResolver,
|
|
||||||
}
|
}
|
||||||
|
|
||||||
s.setupRoutes()
|
s.setupRoutes()
|
||||||
@@ -152,14 +127,7 @@ func (s *Server) Stop(ctx context.Context) error {
|
|||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
|
|
||||||
// SetASNFetcher sets the ASN WHOIS fetcher for on-demand lookups. Call it
|
// SetASNFetcher sets the ASN WHOIS fetcher for on-demand lookups.
|
||||||
// before Start: the handlers read the fetcher without a lock.
|
|
||||||
func (s *Server) SetASNFetcher(fetcher ASNFetcher) {
|
func (s *Server) SetASNFetcher(fetcher ASNFetcher) {
|
||||||
s.asnFetcher = fetcher
|
s.asnFetcher = fetcher
|
||||||
}
|
}
|
||||||
|
|
||||||
// SetBlocklists sets the blocklists the IP lookups report. Call it before
|
|
||||||
// Start: the handlers read the blocklists without a lock.
|
|
||||||
func (s *Server) SetBlocklists(blocklists Blocklists) {
|
|
||||||
s.blocklists = blocklists
|
|
||||||
}
|
|
||||||
|
|||||||
@@ -106,12 +106,12 @@ type RawMessageHandler func(line string)
|
|||||||
// handlerMetrics tracks performance metrics for a handler
|
// handlerMetrics tracks performance metrics for a handler
|
||||||
type handlerMetrics struct {
|
type handlerMetrics struct {
|
||||||
processedCount uint64 // Total messages processed
|
processedCount uint64 // Total messages processed
|
||||||
droppedCount atomic.Uint64 // Total messages dropped; not under mu
|
droppedCount uint64 // Total messages dropped
|
||||||
totalTime time.Duration // Total processing time (for average calculation)
|
totalTime time.Duration // Total processing time (for average calculation)
|
||||||
minTime time.Duration // Minimum processing time
|
minTime time.Duration // Minimum processing time
|
||||||
maxTime time.Duration // Maximum processing time
|
maxTime time.Duration // Maximum processing time
|
||||||
queueHighWaterMark int // Maximum queue length seen
|
queueHighWaterMark int // Maximum queue length seen
|
||||||
mu sync.Mutex // Protects the other metrics
|
mu sync.Mutex // Protects the metrics
|
||||||
}
|
}
|
||||||
|
|
||||||
// handlerInfo wraps a handler with its queue and metrics
|
// handlerInfo wraps a handler with its queue and metrics
|
||||||
@@ -371,7 +371,7 @@ func (s *Streamer) GetHandlerStats() []HandlerStats {
|
|||||||
QueueCapacity: cap(info.queue),
|
QueueCapacity: cap(info.queue),
|
||||||
QueueHighWaterMark: info.metrics.queueHighWaterMark,
|
QueueHighWaterMark: info.metrics.queueHighWaterMark,
|
||||||
ProcessedCount: info.metrics.processedCount,
|
ProcessedCount: info.metrics.processedCount,
|
||||||
DroppedCount: info.metrics.droppedCount.Load(),
|
DroppedCount: info.metrics.droppedCount,
|
||||||
MinProcessTime: info.metrics.minTime,
|
MinProcessTime: info.metrics.minTime,
|
||||||
MaxProcessTime: info.metrics.maxTime,
|
MaxProcessTime: info.metrics.maxTime,
|
||||||
}
|
}
|
||||||
@@ -488,7 +488,7 @@ func (s *Streamer) logMetrics() {
|
|||||||
"queue_len", len(info.queue),
|
"queue_len", len(info.queue),
|
||||||
"queue_cap", cap(info.queue),
|
"queue_cap", cap(info.queue),
|
||||||
"processed", info.metrics.processedCount,
|
"processed", info.metrics.processedCount,
|
||||||
"dropped", info.metrics.droppedCount.Load(),
|
"dropped", info.metrics.droppedCount,
|
||||||
"avg_time", avgTime,
|
"avg_time", avgTime,
|
||||||
"min_time", info.metrics.minTime,
|
"min_time", info.metrics.minTime,
|
||||||
"max_time", info.metrics.maxTime,
|
"max_time", info.metrics.maxTime,
|
||||||
@@ -858,7 +858,7 @@ func (s *Streamer) dispatchMessage(
|
|||||||
}
|
}
|
||||||
|
|
||||||
if !s.queueMessage(info, msg) {
|
if !s.queueMessage(info, msg) {
|
||||||
info.metrics.droppedCount.Add(1)
|
atomic.AddUint64(&info.metrics.droppedCount, 1)
|
||||||
s.totalDropped.Add(1)
|
s.totalDropped.Add(1)
|
||||||
|
|
||||||
continue
|
continue
|
||||||
|
|||||||
@@ -621,68 +621,6 @@ func TestStreamReadsLongLine(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// blockedHandler wants UPDATE messages, has room on its queue for one, and
|
|
||||||
// does not return from HandleMessage until release is closed.
|
|
||||||
type blockedHandler struct {
|
|
||||||
release chan struct{}
|
|
||||||
}
|
|
||||||
|
|
||||||
func (blockedHandler) WantsMessage(messageType string) bool {
|
|
||||||
return messageType == ristypes.MessageTypeUpdate
|
|
||||||
}
|
|
||||||
|
|
||||||
func (blockedHandler) QueueCapacity() int { return 1 }
|
|
||||||
|
|
||||||
func (h blockedHandler) HandleMessage(*ristypes.RISMessage) { <-h.release }
|
|
||||||
|
|
||||||
// TestHandlerStatsReadWhileMessagesAreDropped has the server send a few UPDATE
|
|
||||||
// lines to a blockedHandler, so the read loop drops all but the one the
|
|
||||||
// handler holds and the one on its queue, while the test reads the handler's
|
|
||||||
// statistics. The drop count must reach the number dropped, and the race
|
|
||||||
// detector must find the count read and written safely.
|
|
||||||
func TestHandlerStatsReadWhileMessagesAreDropped(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
const (
|
|
||||||
lines = 5
|
|
||||||
wantDropped = lines - 2
|
|
||||||
maxWait = 5 * time.Second
|
|
||||||
)
|
|
||||||
|
|
||||||
srv := httptest.NewServer(http.HandlerFunc(
|
|
||||||
func(w http.ResponseWriter, r *http.Request) {
|
|
||||||
_, _ = io.WriteString(w, strings.Repeat(updateLine, lines))
|
|
||||||
|
|
||||||
flush(w)
|
|
||||||
<-r.Context().Done()
|
|
||||||
}))
|
|
||||||
defer srv.Close()
|
|
||||||
|
|
||||||
s := New(logger.New(), metrics.New())
|
|
||||||
s.url = srv.URL
|
|
||||||
handler := blockedHandler{release: make(chan struct{})}
|
|
||||||
s.RegisterHandler(handler)
|
|
||||||
|
|
||||||
err := s.Start(t.Context())
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("Start returned error: %v", err)
|
|
||||||
}
|
|
||||||
defer s.Stop(t.Context())
|
|
||||||
defer close(handler.release)
|
|
||||||
|
|
||||||
// The test polls instead of waiting for a signal from the read loop: a
|
|
||||||
// signal would order the drops before the reads here, and the race
|
|
||||||
// detector would have nothing to report.
|
|
||||||
deadline := time.Now().Add(maxWait)
|
|
||||||
|
|
||||||
for s.GetHandlerStats()[0].DroppedCount < wantDropped {
|
|
||||||
if time.Now().After(deadline) {
|
|
||||||
t.Fatalf("fewer than %d messages counted as dropped within %s",
|
|
||||||
wantDropped, maxWait)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestPeerCountCountsEveryPeerHeardFrom has the server send an UPDATE from
|
// TestPeerCountCountsEveryPeerHeardFrom has the server send an UPDATE from
|
||||||
// stateDownPeer and a KEEPALIVE from another peer, with no OPEN from either,
|
// stateDownPeer and a KEEPALIVE from another peer, with no OPEN from either,
|
||||||
// and then a message that ends stateDownPeer's session: a NOTIFICATION, or a
|
// and then a message that ends stateDownPeer's session: a NOTIFICATION, or a
|
||||||
|
|||||||
@@ -126,26 +126,6 @@
|
|||||||
margin: 0;
|
margin: 0;
|
||||||
color: #2c3e50;
|
color: #2c3e50;
|
||||||
}
|
}
|
||||||
/* A click on a prefix list's heading folds or unfolds the list. Not
|
|
||||||
every browser shows its own marker on a flex element, so the page
|
|
||||||
hides it and draws a triangle before the heading. The triangle
|
|
||||||
points down while the list is open. */
|
|
||||||
summary.prefix-header {
|
|
||||||
cursor: pointer;
|
|
||||||
list-style: none;
|
|
||||||
}
|
|
||||||
summary.prefix-header::-webkit-details-marker {
|
|
||||||
display: none;
|
|
||||||
}
|
|
||||||
summary.prefix-header h2::before {
|
|
||||||
content: "\25B8";
|
|
||||||
display: inline-block;
|
|
||||||
margin-right: 10px;
|
|
||||||
transition: transform 0.2s;
|
|
||||||
}
|
|
||||||
details[open] > summary.prefix-header h2::before {
|
|
||||||
transform: rotate(90deg);
|
|
||||||
}
|
|
||||||
.prefix-count {
|
.prefix-count {
|
||||||
background: #e74c3c;
|
background: #e74c3c;
|
||||||
color: white;
|
color: white;
|
||||||
@@ -253,20 +233,18 @@
|
|||||||
<div class="info-label">Peer ASNs</div>
|
<div class="info-label">Peer ASNs</div>
|
||||||
<div class="info-value">{{.PeerCount}}</div>
|
<div class="info-value">{{.PeerCount}}</div>
|
||||||
</div>
|
</div>
|
||||||
{{if not .ASN.FirstSeen.IsZero}}
|
|
||||||
<div class="info-card">
|
<div class="info-card">
|
||||||
<div class="info-label">First Seen</div>
|
<div class="info-label">First Seen</div>
|
||||||
<div class="info-value">{{.ASN.FirstSeen.Format "2006-01-02"}}</div>
|
<div class="info-value">{{.ASN.FirstSeen.Format "2006-01-02"}}</div>
|
||||||
</div>
|
</div>
|
||||||
{{end}}
|
|
||||||
</div>
|
</div>
|
||||||
|
|
||||||
{{if .IPv4Prefixes}}
|
{{if .IPv4Prefixes}}
|
||||||
<details class="prefix-section" open>
|
<div class="prefix-section">
|
||||||
<summary class="prefix-header">
|
<div class="prefix-header">
|
||||||
<h2>IPv4 Prefixes</h2>
|
<h2>IPv4 Prefixes</h2>
|
||||||
<span class="prefix-count">{{.IPv4Count}}</span>
|
<span class="prefix-count">{{.IPv4Count}}</span>
|
||||||
</summary>
|
</div>
|
||||||
<table class="prefix-table">
|
<table class="prefix-table">
|
||||||
<thead>
|
<thead>
|
||||||
<tr>
|
<tr>
|
||||||
@@ -287,15 +265,15 @@
|
|||||||
{{end}}
|
{{end}}
|
||||||
</tbody>
|
</tbody>
|
||||||
</table>
|
</table>
|
||||||
</details>
|
</div>
|
||||||
{{end}}
|
{{end}}
|
||||||
|
|
||||||
{{if .IPv6Prefixes}}
|
{{if .IPv6Prefixes}}
|
||||||
<details class="prefix-section" open>
|
<div class="prefix-section">
|
||||||
<summary class="prefix-header">
|
<div class="prefix-header">
|
||||||
<h2>IPv6 Prefixes</h2>
|
<h2>IPv6 Prefixes</h2>
|
||||||
<span class="prefix-count">{{.IPv6Count}}</span>
|
<span class="prefix-count">{{.IPv6Count}}</span>
|
||||||
</summary>
|
</div>
|
||||||
<table class="prefix-table">
|
<table class="prefix-table">
|
||||||
<thead>
|
<thead>
|
||||||
<tr>
|
<tr>
|
||||||
@@ -316,7 +294,7 @@
|
|||||||
{{end}}
|
{{end}}
|
||||||
</tbody>
|
</tbody>
|
||||||
</table>
|
</table>
|
||||||
</details>
|
</div>
|
||||||
{{end}}
|
{{end}}
|
||||||
|
|
||||||
{{if eq .TotalCount 0}}
|
{{if eq .TotalCount 0}}
|
||||||
|
|||||||
+19
-201
@@ -107,15 +107,9 @@
|
|||||||
.stat-card.connected .stat-value {
|
.stat-card.connected .stat-value {
|
||||||
color: #27ae60;
|
color: #27ae60;
|
||||||
}
|
}
|
||||||
.stat-card.disconnected .stat-value,
|
.stat-card.disconnected .stat-value {
|
||||||
.stale {
|
|
||||||
color: #e74c3c;
|
color: #e74c3c;
|
||||||
}
|
}
|
||||||
/* The card has no room beside "Connected", so the spinner and "stale" go under it */
|
|
||||||
#freshness {
|
|
||||||
display: block;
|
|
||||||
font-size: 20px;
|
|
||||||
}
|
|
||||||
|
|
||||||
/* Search section */
|
/* Search section */
|
||||||
.search-section {
|
.search-section {
|
||||||
@@ -180,38 +174,6 @@
|
|||||||
#asname-results .search-message {
|
#asname-results .search-message {
|
||||||
font-size: 13px;
|
font-size: 13px;
|
||||||
}
|
}
|
||||||
#asname-results .error {
|
|
||||||
color: #c00;
|
|
||||||
}
|
|
||||||
.asname-list {
|
|
||||||
list-style: none;
|
|
||||||
margin: 12px 0 0 0;
|
|
||||||
padding: 0;
|
|
||||||
max-height: 400px;
|
|
||||||
overflow-y: auto;
|
|
||||||
}
|
|
||||||
.asname-list li {
|
|
||||||
border-bottom: 1px solid #eee;
|
|
||||||
}
|
|
||||||
.asname-list a {
|
|
||||||
display: block;
|
|
||||||
padding: 8px 4px;
|
|
||||||
color: #2c3e50;
|
|
||||||
text-decoration: none;
|
|
||||||
font-size: 14px;
|
|
||||||
}
|
|
||||||
.asname-list a:hover {
|
|
||||||
background: #f5f5f5;
|
|
||||||
}
|
|
||||||
.asname-list .asn {
|
|
||||||
font-weight: 600;
|
|
||||||
color: #3498db;
|
|
||||||
}
|
|
||||||
.asname-list .description {
|
|
||||||
display: block;
|
|
||||||
color: #7f8c8d;
|
|
||||||
font-size: 12px;
|
|
||||||
}
|
|
||||||
|
|
||||||
/* IP Lookup result */
|
/* IP Lookup result */
|
||||||
.ip-result {
|
.ip-result {
|
||||||
@@ -239,8 +201,7 @@
|
|||||||
cursor: pointer;
|
cursor: pointer;
|
||||||
font-size: 12px;
|
font-size: 12px;
|
||||||
}
|
}
|
||||||
.ip-result pre,
|
.ip-result pre {
|
||||||
#ip-api pre {
|
|
||||||
background: #2c3e50;
|
background: #2c3e50;
|
||||||
color: #ecf0f1;
|
color: #ecf0f1;
|
||||||
padding: 16px;
|
padding: 16px;
|
||||||
@@ -252,30 +213,18 @@
|
|||||||
max-height: 400px;
|
max-height: 400px;
|
||||||
overflow-y: auto;
|
overflow-y: auto;
|
||||||
}
|
}
|
||||||
.ip-result .error,
|
.ip-result .error {
|
||||||
#ip-api .error {
|
|
||||||
background: #fee;
|
background: #fee;
|
||||||
color: #c00;
|
color: #c00;
|
||||||
padding: 12px;
|
padding: 12px;
|
||||||
border-radius: 6px;
|
border-radius: 6px;
|
||||||
font-size: 14px;
|
font-size: 14px;
|
||||||
}
|
}
|
||||||
.ip-result .loading,
|
.ip-result .loading {
|
||||||
#ip-api .loading {
|
|
||||||
color: #7f8c8d;
|
color: #7f8c8d;
|
||||||
font-style: italic;
|
font-style: italic;
|
||||||
}
|
}
|
||||||
|
|
||||||
/* IP information API card: the command, then its answer */
|
|
||||||
#ip-api p {
|
|
||||||
margin: 0;
|
|
||||||
color: #7f8c8d;
|
|
||||||
font-size: 14px;
|
|
||||||
}
|
|
||||||
#ip-api pre {
|
|
||||||
margin-top: 12px;
|
|
||||||
}
|
|
||||||
|
|
||||||
/* Footer */
|
/* Footer */
|
||||||
.footer {
|
.footer {
|
||||||
margin-top: 40px;
|
margin-top: 40px;
|
||||||
@@ -316,10 +265,7 @@
|
|||||||
<main class="main-content">
|
<main class="main-content">
|
||||||
<div class="stats-grid">
|
<div class="stats-grid">
|
||||||
<div class="stat-card" id="status-card">
|
<div class="stat-card" id="status-card">
|
||||||
<div class="stat-value">
|
<div class="stat-value" id="stat-status">-</div>
|
||||||
<span id="stat-status">-</span>
|
|
||||||
<span id="freshness"></span>
|
|
||||||
</div>
|
|
||||||
<div class="stat-label">Status</div>
|
<div class="stat-label">Status</div>
|
||||||
</div>
|
</div>
|
||||||
<div class="stat-card">
|
<div class="stat-card">
|
||||||
@@ -360,7 +306,7 @@
|
|||||||
<input type="text" id="asname-input" placeholder="e.g., Google, Cloudflare" autocomplete="off">
|
<input type="text" id="asname-input" placeholder="e.g., Google, Cloudflare" autocomplete="off">
|
||||||
<button type="submit">Search</button>
|
<button type="submit">Search</button>
|
||||||
</form>
|
</form>
|
||||||
<p class="search-hint">Search for autonomous systems by organization name or handle</p>
|
<p class="search-hint">Search for autonomous systems by organization name</p>
|
||||||
<div id="asname-results"></div>
|
<div id="asname-results"></div>
|
||||||
</div>
|
</div>
|
||||||
|
|
||||||
@@ -380,13 +326,6 @@
|
|||||||
</div>
|
</div>
|
||||||
</div>
|
</div>
|
||||||
</div>
|
</div>
|
||||||
|
|
||||||
<div class="search-card" id="ip-api">
|
|
||||||
<h2>IP Information API</h2>
|
|
||||||
<p>The API answers what RouteWatch knows about an IP address, such as its AS number and country. <code>/ip/</code> with no address answers for the caller's own address.</p>
|
|
||||||
<pre id="ip-api-command" data-client-ip="{{.ClientIP}}"></pre>
|
|
||||||
<pre id="ip-api-answer" class="loading">Loading...</pre>
|
|
||||||
</div>
|
|
||||||
</main>
|
</main>
|
||||||
|
|
||||||
<footer class="footer">
|
<footer class="footer">
|
||||||
@@ -398,44 +337,6 @@
|
|||||||
</footer>
|
</footer>
|
||||||
|
|
||||||
<script nonce="{{.CSPNonce}}">
|
<script nonce="{{.CSPNonce}}">
|
||||||
// The stats are fetched every REFRESH_PERIOD_MS. Next to the status, a
|
|
||||||
// spinner turns one frame each time an answer counts more messages
|
|
||||||
// than the one before. After STALE_AFTER_REFRESHES refreshes without
|
|
||||||
// such an answer, the spinner gives way to "stale" and the time since.
|
|
||||||
const REFRESH_PERIOD_MS = 5000;
|
|
||||||
const STALE_AFTER_REFRESHES = 3;
|
|
||||||
const MS_PER_SECOND = 1000;
|
|
||||||
const SPINNER_FRAMES = ['⠋', '⠙', '⠹', '⠸', '⠼', '⠴', '⠦', '⠧', '⠇', '⠏'];
|
|
||||||
const SPINNER_COLOURS = ['#e74c3c', '#e67e22', '#f1c40f', '#27ae60', '#3498db', '#9b59b6'];
|
|
||||||
|
|
||||||
let spinnerFrame = 0;
|
|
||||||
let lastMessageCount = null;
|
|
||||||
let lastFreshAt = Date.now();
|
|
||||||
|
|
||||||
// Turn the spinner when the message count has risen since the last answer.
|
|
||||||
function noteMessageCount(count) {
|
|
||||||
if (lastMessageCount !== null && count > lastMessageCount) {
|
|
||||||
spinnerFrame = (spinnerFrame + 1) % SPINNER_FRAMES.length;
|
|
||||||
lastFreshAt = Date.now();
|
|
||||||
}
|
|
||||||
lastMessageCount = count;
|
|
||||||
showFreshness();
|
|
||||||
}
|
|
||||||
|
|
||||||
function showFreshness() {
|
|
||||||
const freshness = document.getElementById('freshness');
|
|
||||||
const sinceFreshMs = Date.now() - lastFreshAt;
|
|
||||||
const stale = sinceFreshMs >= STALE_AFTER_REFRESHES * REFRESH_PERIOD_MS;
|
|
||||||
freshness.classList.toggle('stale', stale);
|
|
||||||
if (stale) {
|
|
||||||
freshness.textContent = 'stale for ' + Math.floor(sinceFreshMs / MS_PER_SECOND) + 's';
|
|
||||||
freshness.style.color = '';
|
|
||||||
return;
|
|
||||||
}
|
|
||||||
freshness.textContent = SPINNER_FRAMES[spinnerFrame];
|
|
||||||
freshness.style.color = SPINNER_COLOURS[spinnerFrame % SPINNER_COLOURS.length];
|
|
||||||
}
|
|
||||||
|
|
||||||
function formatNumber(num) {
|
function formatNumber(num) {
|
||||||
if (num >= 1000000) {
|
if (num >= 1000000) {
|
||||||
return (num / 1000000).toFixed(1) + 'M';
|
return (num / 1000000).toFixed(1) + 'M';
|
||||||
@@ -457,7 +358,6 @@
|
|||||||
const statusEl = document.getElementById('stat-status');
|
const statusEl = document.getElementById('stat-status');
|
||||||
statusEl.textContent = data.connected ? 'Connected' : 'Disconnected';
|
statusEl.textContent = data.connected ? 'Connected' : 'Disconnected';
|
||||||
statusCard.className = 'stat-card ' + (data.connected ? 'connected' : 'disconnected');
|
statusCard.className = 'stat-card ' + (data.connected ? 'connected' : 'disconnected');
|
||||||
noteMessageCount(data.total_messages);
|
|
||||||
|
|
||||||
document.getElementById('stat-routes').textContent = formatNumber(data.live_routes);
|
document.getElementById('stat-routes').textContent = formatNumber(data.live_routes);
|
||||||
document.getElementById('stat-asns').textContent = formatNumber(data.asns);
|
document.getElementById('stat-asns').textContent = formatNumber(data.asns);
|
||||||
@@ -468,7 +368,7 @@
|
|||||||
}
|
}
|
||||||
|
|
||||||
const totalUpdates = data.ipv4_updates_per_sec + data.ipv6_updates_per_sec;
|
const totalUpdates = data.ipv4_updates_per_sec + data.ipv6_updates_per_sec;
|
||||||
document.getElementById('stat-updates').textContent = Math.round(totalUpdates).toLocaleString();
|
document.getElementById('stat-updates').textContent = totalUpdates.toFixed(1);
|
||||||
})
|
})
|
||||||
.catch(() => {
|
.catch(() => {
|
||||||
document.getElementById('stat-status').textContent = 'Error';
|
document.getElementById('stat-status').textContent = 'Error';
|
||||||
@@ -487,82 +387,24 @@
|
|||||||
}
|
}
|
||||||
});
|
});
|
||||||
|
|
||||||
// AS name search. The handles and descriptions come from outside data,
|
// AS name search
|
||||||
// so the results are built with textContent, never innerHTML.
|
|
||||||
const asNameResults = document.getElementById('asname-results');
|
|
||||||
|
|
||||||
// Replace the search results with one line of text.
|
|
||||||
function showASNameMessage(text, className) {
|
|
||||||
const line = document.createElement('p');
|
|
||||||
line.className = className;
|
|
||||||
line.textContent = text;
|
|
||||||
asNameResults.replaceChildren(line);
|
|
||||||
}
|
|
||||||
|
|
||||||
// List the matches of a search answer, each as a link to its AS page,
|
|
||||||
// and say how many more matched than the answer holds.
|
|
||||||
function showASNameMatches(data) {
|
|
||||||
if (data.total === 0) {
|
|
||||||
showASNameMessage('No AS matches "' + data.query + '".', 'search-message');
|
|
||||||
return;
|
|
||||||
}
|
|
||||||
|
|
||||||
const list = document.createElement('ul');
|
|
||||||
list.className = 'asname-list';
|
|
||||||
for (const match of data.results) {
|
|
||||||
const asn = document.createElement('span');
|
|
||||||
asn.className = 'asn';
|
|
||||||
asn.textContent = 'AS' + match.asn;
|
|
||||||
|
|
||||||
const description = document.createElement('span');
|
|
||||||
description.className = 'description';
|
|
||||||
description.textContent = match.description;
|
|
||||||
|
|
||||||
const link = document.createElement('a');
|
|
||||||
link.href = '/as/' + match.asn;
|
|
||||||
link.append(asn, ' ' + match.handle, description);
|
|
||||||
|
|
||||||
const item = document.createElement('li');
|
|
||||||
item.append(link);
|
|
||||||
list.append(item);
|
|
||||||
}
|
|
||||||
asNameResults.replaceChildren(list);
|
|
||||||
|
|
||||||
const more = data.total - data.results.length;
|
|
||||||
if (more > 0) {
|
|
||||||
const line = document.createElement('p');
|
|
||||||
line.className = 'search-message';
|
|
||||||
line.textContent = more.toLocaleString() + ' more matched. Narrow the search to see them.';
|
|
||||||
asNameResults.append(line);
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
document.getElementById('asname-form').addEventListener('submit', function(e) {
|
document.getElementById('asname-form').addEventListener('submit', function(e) {
|
||||||
e.preventDefault();
|
e.preventDefault();
|
||||||
const query = document.getElementById('asname-input').value.trim();
|
const query = document.getElementById('asname-input').value.trim();
|
||||||
if (!query) return;
|
if (!query) return;
|
||||||
|
|
||||||
// A number, or AS and a number, opens that AS's page
|
const resultsDiv = document.getElementById('asname-results');
|
||||||
const asn = query.replace(/^AS/i, '');
|
resultsDiv.innerHTML = '<p class="loading">Searching...</p>';
|
||||||
if (/^\d+$/.test(asn)) {
|
|
||||||
window.location.href = '/as/' + asn;
|
// Use a simple client-side search against the asinfo data
|
||||||
|
// For now, redirect to AS page if it looks like an ASN
|
||||||
|
if (/^\d+$/.test(query)) {
|
||||||
|
window.location.href = '/as/' + query;
|
||||||
return;
|
return;
|
||||||
}
|
}
|
||||||
|
|
||||||
showASNameMessage('Searching...', 'loading');
|
// Show a message that server-side search is coming
|
||||||
|
resultsDiv.innerHTML = '<p class="search-message">AS name search coming soon. For now, try an AS number.</p>';
|
||||||
fetch('/api/v1/search/as?q=' + encodeURIComponent(query))
|
|
||||||
.then(response => response.json())
|
|
||||||
.then(response => {
|
|
||||||
if (response.status === 'error') {
|
|
||||||
showASNameMessage('Error: ' + response.error.msg, 'error');
|
|
||||||
} else {
|
|
||||||
showASNameMatches(response.data);
|
|
||||||
}
|
|
||||||
})
|
|
||||||
.catch(error => {
|
|
||||||
showASNameMessage('Error: ' + error.message, 'error');
|
|
||||||
});
|
|
||||||
});
|
});
|
||||||
|
|
||||||
// IP lookup
|
// IP lookup
|
||||||
@@ -604,33 +446,9 @@
|
|||||||
document.getElementById('ip-input').value = '';
|
document.getElementById('ip-input').value = '';
|
||||||
});
|
});
|
||||||
|
|
||||||
// IP information API card: show the curl command that asks /ip/ about
|
// Initial load and refresh stats every 5 seconds
|
||||||
// the visitor's own address, then make that same request and show the
|
|
||||||
// JSON it answers, an error answer included. The answer holds outside
|
|
||||||
// data, so it is set with textContent, never innerHTML.
|
|
||||||
const ipAPICommand = document.getElementById('ip-api-command');
|
|
||||||
const ipAPIAnswer = document.getElementById('ip-api-answer');
|
|
||||||
const ipAPIURL = window.location.origin + '/ip/' + ipAPICommand.dataset.clientIp;
|
|
||||||
|
|
||||||
ipAPICommand.textContent = 'curl ' + ipAPIURL;
|
|
||||||
|
|
||||||
fetch(ipAPIURL)
|
|
||||||
.then(response => response.json())
|
|
||||||
.then(answer => {
|
|
||||||
ipAPIAnswer.className = '';
|
|
||||||
ipAPIAnswer.textContent = JSON.stringify(answer, null, 2);
|
|
||||||
})
|
|
||||||
.catch(error => {
|
|
||||||
ipAPIAnswer.className = 'error';
|
|
||||||
ipAPIAnswer.textContent = 'Error: ' + error.message;
|
|
||||||
});
|
|
||||||
|
|
||||||
// Initial load and refresh stats every REFRESH_PERIOD_MS. The freshness
|
|
||||||
// check runs every second on its own, so a request that never returns
|
|
||||||
// still turns the spinner into "stale".
|
|
||||||
updateStats();
|
updateStats();
|
||||||
setInterval(updateStats, REFRESH_PERIOD_MS);
|
setInterval(updateStats, 5000);
|
||||||
setInterval(showFreshness, MS_PER_SECOND);
|
|
||||||
</script>
|
</script>
|
||||||
</body>
|
</body>
|
||||||
</html>
|
</html>
|
||||||
|
|||||||
@@ -126,8 +126,7 @@
|
|||||||
.connected {
|
.connected {
|
||||||
color: #22c55e;
|
color: #22c55e;
|
||||||
}
|
}
|
||||||
.disconnected,
|
.disconnected {
|
||||||
.stale {
|
|
||||||
color: #ef4444;
|
color: #ef4444;
|
||||||
}
|
}
|
||||||
.error {
|
.error {
|
||||||
@@ -181,10 +180,7 @@
|
|||||||
<h2>RouteWatch</h2>
|
<h2>RouteWatch</h2>
|
||||||
<div class="metric">
|
<div class="metric">
|
||||||
<span class="metric-label">Status</span>
|
<span class="metric-label">Status</span>
|
||||||
<span>
|
<span class="metric-value" id="connected">-</span>
|
||||||
<span class="metric-value" id="connected">-</span>
|
|
||||||
<span class="metric-value" id="freshness"></span>
|
|
||||||
</span>
|
|
||||||
</div>
|
</div>
|
||||||
<div class="metric">
|
<div class="metric">
|
||||||
<span class="metric-label">Uptime</span>
|
<span class="metric-label">Uptime</span>
|
||||||
@@ -235,8 +231,8 @@
|
|||||||
<span class="metric-value" id="withdrawals">-</span>
|
<span class="metric-value" id="withdrawals">-</span>
|
||||||
</div>
|
</div>
|
||||||
<div class="metric">
|
<div class="metric">
|
||||||
<span class="metric-label">Route Churn/min</span>
|
<span class="metric-label">Route Churn/sec</span>
|
||||||
<span class="metric-value" id="route_churn_per_min">-</span>
|
<span class="metric-value" id="route_churn_per_sec">-</span>
|
||||||
</div>
|
</div>
|
||||||
<div class="metric">
|
<div class="metric">
|
||||||
<span class="metric-label">Total Data</span>
|
<span class="metric-label">Total Data</span>
|
||||||
@@ -390,44 +386,6 @@
|
|||||||
</div>
|
</div>
|
||||||
|
|
||||||
<script nonce="{{.CSPNonce}}">
|
<script nonce="{{.CSPNonce}}">
|
||||||
// The stats are fetched every REFRESH_PERIOD_MS. Next to the status, a
|
|
||||||
// spinner turns one frame each time an answer counts more messages
|
|
||||||
// than the one before. After STALE_AFTER_REFRESHES refreshes without
|
|
||||||
// such an answer, the spinner gives way to "stale" and the time since.
|
|
||||||
const REFRESH_PERIOD_MS = 2000;
|
|
||||||
const STALE_AFTER_REFRESHES = 3;
|
|
||||||
const MS_PER_SECOND = 1000;
|
|
||||||
const SPINNER_FRAMES = ['⠋', '⠙', '⠹', '⠸', '⠼', '⠴', '⠦', '⠧', '⠇', '⠏'];
|
|
||||||
const SPINNER_COLOURS = ['#e74c3c', '#e67e22', '#f1c40f', '#27ae60', '#3498db', '#9b59b6'];
|
|
||||||
|
|
||||||
let spinnerFrame = 0;
|
|
||||||
let lastMessageCount = null;
|
|
||||||
let lastFreshAt = Date.now();
|
|
||||||
|
|
||||||
// Turn the spinner when the message count has risen since the last answer.
|
|
||||||
function noteMessageCount(count) {
|
|
||||||
if (lastMessageCount !== null && count > lastMessageCount) {
|
|
||||||
spinnerFrame = (spinnerFrame + 1) % SPINNER_FRAMES.length;
|
|
||||||
lastFreshAt = Date.now();
|
|
||||||
}
|
|
||||||
lastMessageCount = count;
|
|
||||||
showFreshness();
|
|
||||||
}
|
|
||||||
|
|
||||||
function showFreshness() {
|
|
||||||
const freshness = document.getElementById('freshness');
|
|
||||||
const sinceFreshMs = Date.now() - lastFreshAt;
|
|
||||||
const stale = sinceFreshMs >= STALE_AFTER_REFRESHES * REFRESH_PERIOD_MS;
|
|
||||||
freshness.classList.toggle('stale', stale);
|
|
||||||
if (stale) {
|
|
||||||
freshness.textContent = 'stale for ' + Math.floor(sinceFreshMs / MS_PER_SECOND) + 's';
|
|
||||||
freshness.style.color = '';
|
|
||||||
return;
|
|
||||||
}
|
|
||||||
freshness.textContent = SPINNER_FRAMES[spinnerFrame];
|
|
||||||
freshness.style.color = SPINNER_COLOURS[spinnerFrame % SPINNER_COLOURS.length];
|
|
||||||
}
|
|
||||||
|
|
||||||
function formatBytes(bytes) {
|
function formatBytes(bytes) {
|
||||||
if (bytes === 0) return '0 B';
|
if (bytes === 0) return '0 B';
|
||||||
const k = 1024;
|
const k = 1024;
|
||||||
@@ -553,7 +511,7 @@
|
|||||||
document.getElementById('messages_per_sec').textContent = '-';
|
document.getElementById('messages_per_sec').textContent = '-';
|
||||||
document.getElementById('announcements').textContent = '-';
|
document.getElementById('announcements').textContent = '-';
|
||||||
document.getElementById('withdrawals').textContent = '-';
|
document.getElementById('withdrawals').textContent = '-';
|
||||||
document.getElementById('route_churn_per_min').textContent = '-';
|
document.getElementById('route_churn_per_sec').textContent = '-';
|
||||||
document.getElementById('total_wire_bytes').textContent = '-';
|
document.getElementById('total_wire_bytes').textContent = '-';
|
||||||
document.getElementById('wire_mbits_per_sec').textContent = '-';
|
document.getElementById('wire_mbits_per_sec').textContent = '-';
|
||||||
document.getElementById('gc_num').textContent = '-';
|
document.getElementById('gc_num').textContent = '-';
|
||||||
@@ -611,7 +569,6 @@
|
|||||||
const connectedEl = document.getElementById('connected');
|
const connectedEl = document.getElementById('connected');
|
||||||
connectedEl.textContent = data.connected ? 'Connected' : 'Disconnected';
|
connectedEl.textContent = data.connected ? 'Connected' : 'Disconnected';
|
||||||
connectedEl.className = 'metric-value ' + (data.connected ? 'connected' : 'disconnected');
|
connectedEl.className = 'metric-value ' + (data.connected ? 'connected' : 'disconnected');
|
||||||
noteMessageCount(data.total_messages);
|
|
||||||
|
|
||||||
// Update all metrics
|
// Update all metrics
|
||||||
document.getElementById('uptime').textContent = data.uptime;
|
document.getElementById('uptime').textContent = data.uptime;
|
||||||
@@ -621,7 +578,7 @@
|
|||||||
document.getElementById('connection_duration').textContent = data.connection_duration;
|
document.getElementById('connection_duration').textContent = data.connection_duration;
|
||||||
document.getElementById('reconnect_count').textContent = formatNumber(data.reconnect_count);
|
document.getElementById('reconnect_count').textContent = formatNumber(data.reconnect_count);
|
||||||
document.getElementById('total_messages').textContent = formatNumber(data.total_messages);
|
document.getElementById('total_messages').textContent = formatNumber(data.total_messages);
|
||||||
document.getElementById('messages_per_sec').textContent = formatNumber(Math.round(data.messages_per_sec));
|
document.getElementById('messages_per_sec').textContent = data.messages_per_sec.toFixed(1);
|
||||||
document.getElementById('total_wire_bytes').textContent = formatBytes(data.total_wire_bytes);
|
document.getElementById('total_wire_bytes').textContent = formatBytes(data.total_wire_bytes);
|
||||||
document.getElementById('wire_mbits_per_sec').textContent = data.wire_mbits_per_sec.toFixed(2) + ' Mbps';
|
document.getElementById('wire_mbits_per_sec').textContent = data.wire_mbits_per_sec.toFixed(2) + ' Mbps';
|
||||||
document.getElementById('asns').textContent = formatNumber(data.asns);
|
document.getElementById('asns').textContent = formatNumber(data.asns);
|
||||||
@@ -634,8 +591,8 @@
|
|||||||
document.getElementById('live_routes').textContent = formatNumber(data.live_routes);
|
document.getElementById('live_routes').textContent = formatNumber(data.live_routes);
|
||||||
document.getElementById('ipv4_routes').textContent = formatNumber(data.ipv4_routes);
|
document.getElementById('ipv4_routes').textContent = formatNumber(data.ipv4_routes);
|
||||||
document.getElementById('ipv6_routes').textContent = formatNumber(data.ipv6_routes);
|
document.getElementById('ipv6_routes').textContent = formatNumber(data.ipv6_routes);
|
||||||
document.getElementById('ipv4_updates_per_sec').textContent = formatNumber(Math.round(data.ipv4_updates_per_sec));
|
document.getElementById('ipv4_updates_per_sec').textContent = data.ipv4_updates_per_sec.toFixed(1);
|
||||||
document.getElementById('ipv6_updates_per_sec').textContent = formatNumber(Math.round(data.ipv6_updates_per_sec));
|
document.getElementById('ipv6_updates_per_sec').textContent = data.ipv6_updates_per_sec.toFixed(1);
|
||||||
document.getElementById('oldest_route').textContent = formatRelativeTime(data.oldest_route);
|
document.getElementById('oldest_route').textContent = formatRelativeTime(data.oldest_route);
|
||||||
document.getElementById('newest_route').textContent = formatRelativeTime(data.newest_route);
|
document.getElementById('newest_route').textContent = formatRelativeTime(data.newest_route);
|
||||||
|
|
||||||
@@ -644,7 +601,7 @@
|
|||||||
document.getElementById('bgp_peer_count').textContent = formatNumber(data.stream.bgp_peer_count);
|
document.getElementById('bgp_peer_count').textContent = formatNumber(data.stream.bgp_peer_count);
|
||||||
document.getElementById('announcements').textContent = formatNumber(data.stream.announcements);
|
document.getElementById('announcements').textContent = formatNumber(data.stream.announcements);
|
||||||
document.getElementById('withdrawals').textContent = formatNumber(data.stream.withdrawals);
|
document.getElementById('withdrawals').textContent = formatNumber(data.stream.withdrawals);
|
||||||
document.getElementById('route_churn_per_min').textContent = formatNumber(data.stream.route_churn_per_min);
|
document.getElementById('route_churn_per_sec').textContent = data.stream.route_churn_per_sec.toFixed(1);
|
||||||
}
|
}
|
||||||
|
|
||||||
// Update GC stats
|
// Update GC stats
|
||||||
@@ -686,12 +643,9 @@
|
|||||||
});
|
});
|
||||||
}
|
}
|
||||||
|
|
||||||
// Update immediately and then every REFRESH_PERIOD_MS. The freshness
|
// Update immediately and then every 2 seconds
|
||||||
// check runs every second on its own, so a request that never returns
|
|
||||||
// still turns the spinner into "stale".
|
|
||||||
updateStatus();
|
updateStatus();
|
||||||
setInterval(updateStatus, REFRESH_PERIOD_MS);
|
setInterval(updateStatus, 2000);
|
||||||
setInterval(showFreshness, MS_PER_SECOND);
|
|
||||||
</script>
|
</script>
|
||||||
</main>
|
</main>
|
||||||
|
|
||||||
|
|||||||
@@ -7,9 +7,9 @@ import (
|
|||||||
"net/url"
|
"net/url"
|
||||||
"strings"
|
"strings"
|
||||||
"sync"
|
"sync"
|
||||||
|
"time"
|
||||||
|
|
||||||
"git.eeqj.de/sneak/routewatch/internal/version"
|
"git.eeqj.de/sneak/routewatch/internal/version"
|
||||||
"github.com/dustin/go-humanize"
|
|
||||||
)
|
)
|
||||||
|
|
||||||
//go:embed index.html
|
//go:embed index.html
|
||||||
@@ -48,7 +48,48 @@ var (
|
|||||||
once sync.Once
|
once sync.Once
|
||||||
)
|
)
|
||||||
|
|
||||||
const cidrPartCount = 2 // A CIDR has two parts: prefix and length
|
const (
|
||||||
|
hoursPerDay = 24
|
||||||
|
daysPerMonth = 30
|
||||||
|
cidrPartCount = 2 // A CIDR has two parts: prefix and length
|
||||||
|
)
|
||||||
|
|
||||||
|
// timeSince returns a human-readable duration since the given time
|
||||||
|
func timeSince(t time.Time) string {
|
||||||
|
duration := time.Since(t)
|
||||||
|
if duration < time.Minute {
|
||||||
|
return "just now"
|
||||||
|
}
|
||||||
|
|
||||||
|
if duration < time.Hour {
|
||||||
|
minutes := int(duration.Minutes())
|
||||||
|
if minutes == 1 {
|
||||||
|
return "1 minute ago"
|
||||||
|
}
|
||||||
|
|
||||||
|
return duration.Truncate(time.Minute).String() + " ago"
|
||||||
|
}
|
||||||
|
|
||||||
|
if duration < hoursPerDay*time.Hour {
|
||||||
|
hours := int(duration.Hours())
|
||||||
|
if hours == 1 {
|
||||||
|
return "1 hour ago"
|
||||||
|
}
|
||||||
|
|
||||||
|
return duration.Truncate(time.Hour).String() + " ago"
|
||||||
|
}
|
||||||
|
|
||||||
|
days := int(duration.Hours() / hoursPerDay)
|
||||||
|
if days == 1 {
|
||||||
|
return "1 day ago"
|
||||||
|
}
|
||||||
|
|
||||||
|
if days < daysPerMonth {
|
||||||
|
return duration.Truncate(hoursPerDay*time.Hour).String() + " ago"
|
||||||
|
}
|
||||||
|
|
||||||
|
return t.Format("2006-01-02")
|
||||||
|
}
|
||||||
|
|
||||||
// prefixURL generates a URL path for a prefix in CIDR notation.
|
// prefixURL generates a URL path for a prefix in CIDR notation.
|
||||||
// Takes a prefix like "192.168.1.0/24" and returns "/prefix/192.168.1.0/24"
|
// Takes a prefix like "192.168.1.0/24" and returns "/prefix/192.168.1.0/24"
|
||||||
@@ -72,7 +113,7 @@ func initTemplates() {
|
|||||||
|
|
||||||
// Create common template functions
|
// Create common template functions
|
||||||
funcs := template.FuncMap{
|
funcs := template.FuncMap{
|
||||||
"timeSince": humanize.Time,
|
"timeSince": timeSince,
|
||||||
"urlEncode": url.QueryEscape,
|
"urlEncode": url.QueryEscape,
|
||||||
"prefixURL": prefixURL,
|
"prefixURL": prefixURL,
|
||||||
"appName": func() string { return version.Name },
|
"appName": func() string { return version.Name },
|
||||||
|
|||||||
@@ -1,181 +0,0 @@
|
|||||||
package templates_test
|
|
||||||
|
|
||||||
import (
|
|
||||||
"bytes"
|
|
||||||
"html/template"
|
|
||||||
"strings"
|
|
||||||
"testing"
|
|
||||||
|
|
||||||
"git.eeqj.de/sneak/routewatch/internal/database"
|
|
||||||
"git.eeqj.de/sneak/routewatch/internal/templates"
|
|
||||||
)
|
|
||||||
|
|
||||||
// renderPage renders a page with the data every page needs, and the visitor's
|
|
||||||
// address, which the home page also needs.
|
|
||||||
func renderPage(t *testing.T, tmpl *template.Template) string {
|
|
||||||
t.Helper()
|
|
||||||
|
|
||||||
var page bytes.Buffer
|
|
||||||
|
|
||||||
err := tmpl.Execute(&page, struct{ CSPNonce, ClientIP string }{})
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("rendering the page: %v", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
return page.String()
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestFooterLicense renders the home and status pages and expects each footer
|
|
||||||
// to name the licence in LICENSE, MIT, and not WTFPL.
|
|
||||||
func TestFooterLicense(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
const (
|
|
||||||
license = "MIT"
|
|
||||||
wrongLicense = "WTFPL"
|
|
||||||
)
|
|
||||||
|
|
||||||
pages := map[string]*template.Template{
|
|
||||||
"index": templates.IndexTemplate(),
|
|
||||||
"status": templates.StatusTemplate(),
|
|
||||||
}
|
|
||||||
|
|
||||||
for name, tmpl := range pages {
|
|
||||||
t.Run(name, func(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
_, footer, found := strings.Cut(renderPage(t, tmpl), "<footer")
|
|
||||||
if !found {
|
|
||||||
t.Fatal("the page has no footer")
|
|
||||||
}
|
|
||||||
|
|
||||||
footer, _, _ = strings.Cut(footer, "</footer>")
|
|
||||||
|
|
||||||
if !strings.Contains(footer, license) {
|
|
||||||
t.Errorf("footer does not name %s: %q", license, footer)
|
|
||||||
}
|
|
||||||
|
|
||||||
if strings.Contains(footer, wrongLicense) {
|
|
||||||
t.Errorf("footer names %s: %q", wrongLicense, footer)
|
|
||||||
}
|
|
||||||
})
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestFreshnessIndicator renders the home and status pages and expects each to
|
|
||||||
// have, right after the status element, the element where the page's script
|
|
||||||
// shows its spinner or "stale".
|
|
||||||
func TestFreshnessIndicator(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
pages := map[string]struct {
|
|
||||||
tmpl *template.Template
|
|
||||||
status string
|
|
||||||
indicator string
|
|
||||||
}{
|
|
||||||
"index": {
|
|
||||||
tmpl: templates.IndexTemplate(),
|
|
||||||
status: `<span id="stat-status">-</span>`,
|
|
||||||
indicator: `<span id="freshness"></span>`,
|
|
||||||
},
|
|
||||||
"status": {
|
|
||||||
tmpl: templates.StatusTemplate(),
|
|
||||||
status: `<span class="metric-value" id="connected">-</span>`,
|
|
||||||
indicator: `<span class="metric-value" id="freshness"></span>`,
|
|
||||||
},
|
|
||||||
}
|
|
||||||
|
|
||||||
for name, page := range pages {
|
|
||||||
t.Run(name, func(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
_, afterStatus, found := strings.Cut(renderPage(t, page.tmpl), page.status)
|
|
||||||
if !found {
|
|
||||||
t.Fatalf("the page has no status element %s", page.status)
|
|
||||||
}
|
|
||||||
|
|
||||||
if !strings.HasPrefix(strings.TrimSpace(afterStatus), page.indicator) {
|
|
||||||
t.Errorf("the status element is not followed by %s", page.indicator)
|
|
||||||
}
|
|
||||||
})
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestASDetailPrefixListsFold renders the AS page with one IPv4 and one IPv6
|
|
||||||
// prefix. It expects each prefix list inside its own <details> element that
|
|
||||||
// starts open and whose <summary> holds the list's heading, so that a click on
|
|
||||||
// the heading folds the list.
|
|
||||||
func TestASDetailPrefixListsFold(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
const (
|
|
||||||
asn = 64500
|
|
||||||
prefixV4 = "192.0.2.0/24"
|
|
||||||
prefixV6 = "2001:db8::/32"
|
|
||||||
)
|
|
||||||
|
|
||||||
ipv4 := []database.LiveRoute{{Prefix: prefixV4}}
|
|
||||||
ipv6 := []database.LiveRoute{{Prefix: prefixV6}}
|
|
||||||
|
|
||||||
// The fields the AS page handler gives the template.
|
|
||||||
data := struct {
|
|
||||||
ASN *database.ASN
|
|
||||||
IPv4Prefixes []database.LiveRoute
|
|
||||||
IPv6Prefixes []database.LiveRoute
|
|
||||||
TotalCount int
|
|
||||||
IPv4Count int
|
|
||||||
IPv6Count int
|
|
||||||
Peers []database.ASPeer
|
|
||||||
PeerCount int
|
|
||||||
CSPNonce string
|
|
||||||
}{
|
|
||||||
ASN: &database.ASN{ASN: asn},
|
|
||||||
IPv4Prefixes: ipv4,
|
|
||||||
IPv6Prefixes: ipv6,
|
|
||||||
TotalCount: len(ipv4) + len(ipv6),
|
|
||||||
IPv4Count: len(ipv4),
|
|
||||||
IPv6Count: len(ipv6),
|
|
||||||
}
|
|
||||||
|
|
||||||
var rendered bytes.Buffer
|
|
||||||
|
|
||||||
err := templates.ASDetailTemplate().Execute(&rendered, data)
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("rendering the page: %v", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
page := rendered.String()
|
|
||||||
|
|
||||||
lists := []struct{ heading, prefix string }{
|
|
||||||
{"IPv4 Prefixes", prefixV4},
|
|
||||||
{"IPv6 Prefixes", prefixV6},
|
|
||||||
}
|
|
||||||
|
|
||||||
for _, list := range lists {
|
|
||||||
before, _, found := strings.Cut(page, list.prefix)
|
|
||||||
if !found {
|
|
||||||
t.Fatalf("the page does not list %s", list.prefix)
|
|
||||||
}
|
|
||||||
|
|
||||||
// The list's section starts at the last <details before its prefix.
|
|
||||||
start := strings.LastIndex(before, "<details")
|
|
||||||
if start < 0 || strings.Contains(before[start:], "</details>") {
|
|
||||||
t.Errorf("%s: the list is not inside a <details> element", list.heading)
|
|
||||||
|
|
||||||
continue
|
|
||||||
}
|
|
||||||
|
|
||||||
tag, section, _ := strings.Cut(before[start:], ">")
|
|
||||||
if !strings.Contains(tag, " open") {
|
|
||||||
t.Errorf("%s: the <details> element does not start open: %s>",
|
|
||||||
list.heading, tag)
|
|
||||||
}
|
|
||||||
|
|
||||||
_, summary, _ := strings.Cut(section, "<summary")
|
|
||||||
summary, _, _ = strings.Cut(summary, "</summary>")
|
|
||||||
|
|
||||||
if !strings.Contains(summary, list.heading) {
|
|
||||||
t.Errorf("%s: the <summary> does not hold the heading", list.heading)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
@@ -19,8 +19,8 @@ const (
|
|||||||
Author = "@sneak"
|
Author = "@sneak"
|
||||||
// AuthorURL is the author's website
|
// AuthorURL is the author's website
|
||||||
AuthorURL = "https://sneak.berlin"
|
AuthorURL = "https://sneak.berlin"
|
||||||
// License is the program license, as LICENSE grants it
|
// License is the program license
|
||||||
License = "MIT"
|
License = "WTFPL"
|
||||||
// RepoURL is the git repository URL
|
// RepoURL is the git repository URL
|
||||||
RepoURL = "https://git.eeqj.de/sneak/routewatch"
|
RepoURL = "https://git.eeqj.de/sneak/routewatch"
|
||||||
)
|
)
|
||||||
|
|||||||
+33
-284
@@ -7,15 +7,10 @@ import (
|
|||||||
"errors"
|
"errors"
|
||||||
"fmt"
|
"fmt"
|
||||||
"io"
|
"io"
|
||||||
"log/slog"
|
|
||||||
"net"
|
"net"
|
||||||
"regexp"
|
"regexp"
|
||||||
"strings"
|
"strings"
|
||||||
"sync"
|
|
||||||
"time"
|
"time"
|
||||||
"unicode/utf8"
|
|
||||||
|
|
||||||
"golang.org/x/text/encoding/charmap"
|
|
||||||
)
|
)
|
||||||
|
|
||||||
// Timeout constants for WHOIS queries.
|
// Timeout constants for WHOIS queries.
|
||||||
@@ -31,18 +26,10 @@ const maxAnswerSize = 1 << 20
|
|||||||
|
|
||||||
// Parsing constants.
|
// Parsing constants.
|
||||||
const (
|
const (
|
||||||
|
keyValueParts = 2 // Expected parts when splitting "key: value"
|
||||||
lacnicDateFormatLen = 8 // Length of YYYYMMDD date format
|
lacnicDateFormatLen = 8 // Length of YYYYMMDD date format
|
||||||
)
|
)
|
||||||
|
|
||||||
// The classes of the two objects that describe an AS in an answer from
|
|
||||||
// RIPE, APNIC or AFRINIC, and the key of the aut-num object's line that
|
|
||||||
// gives the ID of the AS's organisation object.
|
|
||||||
const (
|
|
||||||
autNumClass = "aut-num"
|
|
||||||
organisationClass = "organisation"
|
|
||||||
orgKey = "org"
|
|
||||||
)
|
|
||||||
|
|
||||||
// WHOIS server addresses.
|
// WHOIS server addresses.
|
||||||
const (
|
const (
|
||||||
whoisServerIANA = "whois.iana.org:43"
|
whoisServerIANA = "whois.iana.org:43"
|
||||||
@@ -51,32 +38,8 @@ const (
|
|||||||
whoisServerAPNIC = "whois.apnic.net:43"
|
whoisServerAPNIC = "whois.apnic.net:43"
|
||||||
whoisServerLACNIC = "whois.lacnic.net:43"
|
whoisServerLACNIC = "whois.lacnic.net:43"
|
||||||
whoisServerAFRINIC = "whois.afrinic.net:43"
|
whoisServerAFRINIC = "whois.afrinic.net:43"
|
||||||
|
|
||||||
// NIC.br registers Brazil's ASes under LACNIC.
|
|
||||||
whoisServerNICbr = "whois.registro.br:43"
|
|
||||||
)
|
)
|
||||||
|
|
||||||
// nicbrHeader starts the copyright notice at the top of NIC.br's WHOIS
|
|
||||||
// answers. LACNIC's answer for an AS that NIC.br registers is NIC.br's, and
|
|
||||||
// carries the same notice:
|
|
||||||
//
|
|
||||||
// % Copyright (c) Nic.br - Use of this data is governed by the Use and
|
|
||||||
// % Privacy Policy at https://registro.br/upp . Distribution,
|
|
||||||
const nicbrHeader = "% Copyright (c) Nic.br"
|
|
||||||
|
|
||||||
// nicbrLimitReached is in the line that marks NIC.br's reduced answer to a
|
|
||||||
// client over its query limit. NIC.br publishes no limit. Its Use and
|
|
||||||
// Privacy Policy, https://registro.br/upp, says only that it restricts
|
|
||||||
// queries to stop the data being taken in bulk.
|
|
||||||
const nicbrLimitReached = "Query rate limit exceeded"
|
|
||||||
|
|
||||||
// NICbrSkipTime is how long the client stops asking NIC.br after NIC.br
|
|
||||||
// answers that the client is over its query limit. NIC.br publishes
|
|
||||||
// neither the limit nor how long it lasts. An hour outlasts a limit
|
|
||||||
// counted per minute or per hour. The ASes looked up meanwhile get
|
|
||||||
// NIC.br's fields when they are next refreshed.
|
|
||||||
const NICbrSkipTime = time.Hour
|
|
||||||
|
|
||||||
// RIR identifiers.
|
// RIR identifiers.
|
||||||
const (
|
const (
|
||||||
RIRARIN = "ARIN"
|
RIRARIN = "ARIN"
|
||||||
@@ -86,31 +49,6 @@ const (
|
|||||||
RIRAFRNIC = "AFRINIC"
|
RIRAFRNIC = "AFRINIC"
|
||||||
)
|
)
|
||||||
|
|
||||||
// noPersonalFlag keeps person and role objects out of an answer. The
|
|
||||||
// organisation object stays in it. RIPE counts the person and role objects
|
|
||||||
// it returns against a daily limit per IP address. Role objects for abuse
|
|
||||||
// contacts do not count. RIPE refuses an address for good that passes the
|
|
||||||
// limit too often:
|
|
||||||
// https://docs.db.ripe.net/Types-of-Queries/Access-Control-for-Queries/
|
|
||||||
//
|
|
||||||
// RIPE documents the flag here:
|
|
||||||
// https://docs.db.ripe.net/Types-of-Queries/Referenced-Objects-in-Query-Response/
|
|
||||||
// APNIC and AFRINIC list it in their answers to the query "help". AFRINIC
|
|
||||||
// sends readers to that answer for its full list of flags:
|
|
||||||
// https://afrinic.net/whois-getting-started
|
|
||||||
const noPersonalFlag = "--no-personal"
|
|
||||||
|
|
||||||
// The error lines with which RIPE's database software refuses a query:
|
|
||||||
// access denied to the client's IP address, and that address's limit
|
|
||||||
// reached. APNIC and AFRINIC run the same software. The lines are listed
|
|
||||||
// under "Access Errors" on this page, whose URL is split over two lines:
|
|
||||||
// https://docs.db.ripe.net/Appendices/
|
|
||||||
// Appendix-C--RIPE-Database-Query-Server-Response-Codes-and-Messages/
|
|
||||||
const (
|
|
||||||
accessDeniedError = "%ERROR:201:"
|
|
||||||
accessLimitError = "%ERROR:202:"
|
|
||||||
)
|
|
||||||
|
|
||||||
// Errors LookupASN returns when a WHOIS answer has nothing to store.
|
// Errors LookupASN returns when a WHOIS answer has nothing to store.
|
||||||
var (
|
var (
|
||||||
// ErrNoReferral is returned when IANA refers the AS to no RIR, as it
|
// ErrNoReferral is returned when IANA refers the AS to no RIR, as it
|
||||||
@@ -120,7 +58,6 @@ var (
|
|||||||
errNoASBlock = errors.New("IANA answered with neither a referral " +
|
errNoASBlock = errors.New("IANA answered with neither a referral " +
|
||||||
"nor an AS block")
|
"nor an AS block")
|
||||||
errNoFields = errors.New("RIR answered with no WHOIS fields")
|
errNoFields = errors.New("RIR answered with no WHOIS fields")
|
||||||
errRefused = errors.New("RIR refused the query")
|
|
||||||
errUnknownServer = errors.New("IANA refers the AS to a server no RIR runs")
|
errUnknownServer = errors.New("IANA refers the AS to a server no RIR runs")
|
||||||
errAnswerTooLarge = errors.New("WHOIS answer reached the size cap")
|
errAnswerTooLarge = errors.New("WHOIS answer reached the size cap")
|
||||||
)
|
)
|
||||||
@@ -149,42 +86,21 @@ type Client struct {
|
|||||||
// host:port. NewClient sets a net.Dialer's; tests replace it to reach
|
// host:port. NewClient sets a net.Dialer's; tests replace it to reach
|
||||||
// local servers instead.
|
// local servers instead.
|
||||||
DialContext func(ctx context.Context, network, address string) (net.Conn, error)
|
DialContext func(ctx context.Context, network, address string) (net.Conn, error)
|
||||||
|
|
||||||
// Now returns the current time. NewClient sets time.Now; tests replace
|
|
||||||
// it to move the clock.
|
|
||||||
Now func() time.Time
|
|
||||||
|
|
||||||
logger *slog.Logger
|
|
||||||
|
|
||||||
// mu guards nicbrSkipUntil.
|
|
||||||
mu sync.Mutex
|
|
||||||
|
|
||||||
// nicbrSkipUntil is when the client next asks NIC.br, after NIC.br
|
|
||||||
// answered that the client is over its query limit.
|
|
||||||
nicbrSkipUntil time.Time
|
|
||||||
}
|
}
|
||||||
|
|
||||||
// NewClient creates a new WHOIS client that logs to logger.
|
// NewClient creates a new WHOIS client.
|
||||||
func NewClient(logger *slog.Logger) *Client {
|
func NewClient() *Client {
|
||||||
dialer := &net.Dialer{Timeout: dialTimeout}
|
dialer := &net.Dialer{Timeout: dialTimeout}
|
||||||
|
|
||||||
return &Client{
|
return &Client{DialContext: dialer.DialContext}
|
||||||
DialContext: dialer.DialContext,
|
|
||||||
Now: time.Now,
|
|
||||||
logger: logger,
|
|
||||||
}
|
|
||||||
}
|
}
|
||||||
|
|
||||||
// LookupASN asks IANA which RIR holds the given ASN, asks that RIR, and
|
// LookupASN asks IANA which RIR holds the given ASN, asks that RIR, and
|
||||||
// returns what the RIR's answer says. When LACNIC's answer is NIC.br's, it
|
// returns what the RIR's answer says. It returns ErrNoReferral when IANA
|
||||||
// returns what NIC.br's own server says instead, unless that query fails or
|
// refers the AS to no RIR, and another error when a query fails, when IANA's
|
||||||
// is skipped, as askNICbr says.
|
// answer has neither a referral nor the AS's block, when IANA refers the AS
|
||||||
//
|
// to a server none of the five RIRs run, or when the RIR's answer carries
|
||||||
// It returns ErrNoReferral when IANA refers the AS to no RIR, and another
|
// none of ASNInfo's fields, as a refusal of service does.
|
||||||
// error when the query to IANA or to the RIR fails, when the RIR refuses
|
|
||||||
// it, when IANA's answer has neither a referral nor the AS's block, when
|
|
||||||
// IANA refers the AS to a server none of the five RIRs run, or when the
|
|
||||||
// answer carries none of ASNInfo's fields.
|
|
||||||
func (c *Client) LookupASN(ctx context.Context, asn int) (*ASNInfo, error) {
|
func (c *Client) LookupASN(ctx context.Context, asn int) (*ASNInfo, error) {
|
||||||
// Query IANA first to find the authoritative RIR
|
// Query IANA first to find the authoritative RIR
|
||||||
query := fmt.Sprintf("AS%d", asn)
|
query := fmt.Sprintf("AS%d", asn)
|
||||||
@@ -211,82 +127,23 @@ func (c *Client) LookupASN(ctx context.Context, asn int) (*ASNInfo, error) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
// Query the authoritative RIR
|
// Query the authoritative RIR
|
||||||
rirResp, err := c.queryRIR(ctx, whoisServer, rirQuery(rir, query))
|
rirResp, err := c.query(ctx, whoisServer, query)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return nil, fmt.Errorf("RIR query failed: %w", err)
|
return nil, fmt.Errorf("RIR query failed: %w", err)
|
||||||
}
|
}
|
||||||
|
|
||||||
// The fields never come from IANA's answer, which describes the block
|
// The fields come from the RIR's answer alone: IANA's describes the
|
||||||
// the AS is in.
|
// block the AS is in.
|
||||||
info := c.parseResponse(asn, rir, rirResp)
|
info := c.parseResponse(asn, rir, rirResp)
|
||||||
|
|
||||||
if rir == RIRLACNIC && strings.Contains(rirResp, nicbrHeader) {
|
|
||||||
info = c.askNICbr(ctx, query, info)
|
|
||||||
}
|
|
||||||
|
|
||||||
if info.isEmpty() {
|
if info.isEmpty() {
|
||||||
return nil, fmt.Errorf("%s from %s: %w", query, whoisServer, errNoFields)
|
return nil, fmt.Errorf("%s from %s: %w", query, whoisServer, errNoFields)
|
||||||
}
|
}
|
||||||
|
|
||||||
info.RawResponse = ianaResp + "\n" + info.RawResponse
|
info.RawResponse = ianaResp + "\n" + rirResp
|
||||||
|
|
||||||
return info, nil
|
return info, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
// askNICbr asks NIC.br's server about an AS whose answer from LACNIC is
|
|
||||||
// NIC.br's, and returns what NIC.br's answer says. LACNIC replaces each
|
|
||||||
// accented letter in such an answer with U+FFFD. NIC.br's own server keeps
|
|
||||||
// them, in Latin-1.
|
|
||||||
//
|
|
||||||
// askNICbr returns lacnicInfo, what LACNIC's answer says, and logs a
|
|
||||||
// warning when the query fails, when NIC.br sends the reduced answer for a
|
|
||||||
// client over its limit, or when NIC.br's answer carries none of ASNInfo's
|
|
||||||
// fields. After the reduced answer, it returns lacnicInfo without asking
|
|
||||||
// NIC.br until NICbrSkipTime has passed.
|
|
||||||
func (c *Client) askNICbr(
|
|
||||||
ctx context.Context, asQuery string, lacnicInfo *ASNInfo,
|
|
||||||
) *ASNInfo {
|
|
||||||
asn := lacnicInfo.ASN
|
|
||||||
|
|
||||||
c.mu.Lock()
|
|
||||||
skipping := c.Now().Before(c.nicbrSkipUntil)
|
|
||||||
c.mu.Unlock()
|
|
||||||
|
|
||||||
if skipping {
|
|
||||||
return lacnicInfo
|
|
||||||
}
|
|
||||||
|
|
||||||
answer, err := c.query(ctx, whoisServerNICbr, asQuery)
|
|
||||||
if err != nil {
|
|
||||||
c.logger.Warn("NIC.br query failed, keeping LACNIC's fields",
|
|
||||||
"asn", asn, "error", err)
|
|
||||||
|
|
||||||
return lacnicInfo
|
|
||||||
}
|
|
||||||
|
|
||||||
if strings.Contains(answer, nicbrLimitReached) {
|
|
||||||
c.mu.Lock()
|
|
||||||
c.nicbrSkipUntil = c.Now().Add(NICbrSkipTime)
|
|
||||||
c.mu.Unlock()
|
|
||||||
|
|
||||||
c.logger.Warn("NIC.br answered that the client is over its query "+
|
|
||||||
"limit, keeping LACNIC's fields and skipping NIC.br",
|
|
||||||
"asn", asn, "skip_for", NICbrSkipTime)
|
|
||||||
|
|
||||||
return lacnicInfo
|
|
||||||
}
|
|
||||||
|
|
||||||
info := c.parseResponse(asn, RIRLACNIC, answer)
|
|
||||||
if info.isEmpty() {
|
|
||||||
c.logger.Warn("NIC.br answered with no WHOIS fields, "+
|
|
||||||
"keeping LACNIC's fields", "asn", asn)
|
|
||||||
|
|
||||||
return lacnicInfo
|
|
||||||
}
|
|
||||||
|
|
||||||
return info
|
|
||||||
}
|
|
||||||
|
|
||||||
// isEmpty reports whether info holds none of the fields a WHOIS answer
|
// isEmpty reports whether info holds none of the fields a WHOIS answer
|
||||||
// fills, as when the answer is only comment and error lines.
|
// fills, as when the answer is only comment and error lines.
|
||||||
func (info *ASNInfo) isEmpty() bool {
|
func (info *ASNInfo) isEmpty() bool {
|
||||||
@@ -297,41 +154,8 @@ func (info *ASNInfo) isEmpty() bool {
|
|||||||
info.RegDate == nil && info.LastMod == nil
|
info.RegDate == nil && info.LastMod == nil
|
||||||
}
|
}
|
||||||
|
|
||||||
// rirQuery returns the query sent to rir's server for asQuery, an AS
|
// query performs a raw WHOIS query to the specified server. An answer that
|
||||||
// number such as "AS3333". RIPE, APNIC and AFRINIC are asked to keep person
|
// reaches maxAnswerSize is an error.
|
||||||
// and role objects out of the answer.
|
|
||||||
func rirQuery(rir, asQuery string) string {
|
|
||||||
switch rir {
|
|
||||||
case RIRRIPE, RIRAPNIC, RIRAFRNIC:
|
|
||||||
return noPersonalFlag + " " + asQuery
|
|
||||||
default:
|
|
||||||
return asQuery
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// queryRIR sends query to an RIR's server and returns the answer. An answer
|
|
||||||
// holding an error line with which the server refuses the query is an
|
|
||||||
// error, and the error holds that line.
|
|
||||||
func (c *Client) queryRIR(
|
|
||||||
ctx context.Context, server, query string,
|
|
||||||
) (string, error) {
|
|
||||||
answer, err := c.query(ctx, server, query)
|
|
||||||
if err != nil {
|
|
||||||
return "", err
|
|
||||||
}
|
|
||||||
|
|
||||||
for line := range strings.SplitSeq(answer, "\n") {
|
|
||||||
if strings.HasPrefix(line, accessDeniedError) ||
|
|
||||||
strings.HasPrefix(line, accessLimitError) {
|
|
||||||
return "", fmt.Errorf("%w: %s", errRefused, line)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
return answer, nil
|
|
||||||
}
|
|
||||||
|
|
||||||
// query performs a raw WHOIS query to the specified server and returns the
|
|
||||||
// answer in UTF-8. An answer that reaches maxAnswerSize is an error.
|
|
||||||
func (c *Client) query(ctx context.Context, server, query string) (string, error) {
|
func (c *Client) query(ctx context.Context, server, query string) (string, error) {
|
||||||
conn, err := c.DialContext(ctx, "tcp", server)
|
conn, err := c.DialContext(ctx, "tcp", server)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
@@ -378,25 +202,7 @@ func (c *Client) query(ctx context.Context, server, query string) (string, error
|
|||||||
return "", fmt.Errorf("read response: %w", errAnswerTooLarge)
|
return "", fmt.Errorf("read response: %w", errAnswerTooLarge)
|
||||||
}
|
}
|
||||||
|
|
||||||
return toUTF8(sb.String())
|
return sb.String(), nil
|
||||||
}
|
|
||||||
|
|
||||||
// toUTF8 returns a WHOIS answer in UTF-8. An answer that is valid UTF-8 is
|
|
||||||
// returned as it is. Any other answer is decoded as Latin-1, which RIPE,
|
|
||||||
// APNIC and NIC.br send. LACNIC, ARIN and AFRINIC send UTF-8. RIPE's
|
|
||||||
// database working group lists the five RIRs here:
|
|
||||||
// https://www.ripe.net/ripe/mail/archives/db-wg/2024-May/008075.html
|
|
||||||
func toUTF8(answer string) (string, error) {
|
|
||||||
if utf8.ValidString(answer) {
|
|
||||||
return answer, nil
|
|
||||||
}
|
|
||||||
|
|
||||||
decoded, err := charmap.ISO8859_1.NewDecoder().String(answer)
|
|
||||||
if err != nil {
|
|
||||||
return "", fmt.Errorf("decode Latin-1 answer: %w", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
return decoded, nil
|
|
||||||
}
|
}
|
||||||
|
|
||||||
// parseIANAReferral extracts the RIR and WHOIS server from an IANA response.
|
// parseIANAReferral extracts the RIR and WHOIS server from an IANA response.
|
||||||
@@ -432,9 +238,7 @@ func (c *Client) parseIANAReferral(response string) (string, string, error) {
|
|||||||
return "", "", nil
|
return "", "", nil
|
||||||
}
|
}
|
||||||
|
|
||||||
// parseResponse extracts ASN information from a WHOIS response. Of an
|
// parseResponse extracts ASN information from a WHOIS response.
|
||||||
// answer from RIPE, APNIC or AFRINIC it reads only the lines asObjectLines
|
|
||||||
// returns. Of an answer from ARIN or LACNIC it reads every line.
|
|
||||||
func (c *Client) parseResponse(asn int, rir, response string) *ASNInfo {
|
func (c *Client) parseResponse(asn int, rir, response string) *ASNInfo {
|
||||||
info := &ASNInfo{
|
info := &ASNInfo{
|
||||||
ASN: asn,
|
ASN: asn,
|
||||||
@@ -443,15 +247,26 @@ func (c *Client) parseResponse(asn int, rir, response string) *ASNInfo {
|
|||||||
}
|
}
|
||||||
|
|
||||||
lines := strings.Split(response, "\n")
|
lines := strings.Split(response, "\n")
|
||||||
if rir == RIRRIPE || rir == RIRAPNIC || rir == RIRAFRNIC {
|
|
||||||
lines = asObjectLines(lines)
|
|
||||||
}
|
|
||||||
|
|
||||||
var addressLines []string
|
var addressLines []string
|
||||||
|
|
||||||
for _, line := range lines {
|
for _, line := range lines {
|
||||||
key, value, ok := splitField(line)
|
line = strings.TrimSpace(line)
|
||||||
if !ok || value == "" {
|
if line == "" || strings.HasPrefix(line, "%") ||
|
||||||
|
strings.HasPrefix(line, "#") {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
|
||||||
|
// Split on first colon
|
||||||
|
parts := strings.SplitN(line, ":", keyValueParts)
|
||||||
|
if len(parts) != keyValueParts {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
|
||||||
|
key := strings.TrimSpace(strings.ToLower(parts[0]))
|
||||||
|
value := strings.TrimSpace(parts[1])
|
||||||
|
|
||||||
|
if value == "" {
|
||||||
continue
|
continue
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -478,72 +293,6 @@ func (c *Client) parseResponse(asn int, rir, response string) *ASNInfo {
|
|||||||
return info
|
return info
|
||||||
}
|
}
|
||||||
|
|
||||||
// splitField splits a "key: value" line of a WHOIS answer into its key, in
|
|
||||||
// lower case, and its value. The third result is false for a line that
|
|
||||||
// holds no field: a blank line, a comment line or a line without a colon.
|
|
||||||
func splitField(line string) (string, string, bool) {
|
|
||||||
line = strings.TrimSpace(line)
|
|
||||||
if strings.HasPrefix(line, "%") || strings.HasPrefix(line, "#") {
|
|
||||||
return "", "", false
|
|
||||||
}
|
|
||||||
|
|
||||||
key, value, ok := strings.Cut(line, ":")
|
|
||||||
|
|
||||||
return strings.ToLower(strings.TrimSpace(key)), strings.TrimSpace(value), ok
|
|
||||||
}
|
|
||||||
|
|
||||||
// asObjectLines returns the lines of the aut-num object in an answer from
|
|
||||||
// RIPE, APNIC or AFRINIC, followed by the lines of the organisation object
|
|
||||||
// whose ID the aut-num object gives on its "org:" line. The answer's other
|
|
||||||
// objects are left out. RIPE and APNIC start their answer with the
|
|
||||||
// as-block object of the block the AS is in, whose dates are not the AS's.
|
|
||||||
//
|
|
||||||
// These answers are in RPSL. A blank line ends an object. An object's first
|
|
||||||
// line has the object's class as its key and the object's ID as its value,
|
|
||||||
// as in "organisation: ORG-APNI1-AP".
|
|
||||||
func asObjectLines(lines []string) []string {
|
|
||||||
var autNum []string
|
|
||||||
|
|
||||||
// The lines of each organisation object, by its ID.
|
|
||||||
organisations := make(map[string][]string)
|
|
||||||
|
|
||||||
// The class and ID of the object the line is in, "" between objects.
|
|
||||||
var class, id string
|
|
||||||
|
|
||||||
// The ID of the organisation object the aut-num object refers to.
|
|
||||||
var orgID string
|
|
||||||
|
|
||||||
for _, line := range lines {
|
|
||||||
if strings.TrimSpace(line) == "" {
|
|
||||||
class = ""
|
|
||||||
|
|
||||||
continue
|
|
||||||
}
|
|
||||||
|
|
||||||
key, value, ok := splitField(line)
|
|
||||||
if !ok {
|
|
||||||
continue
|
|
||||||
}
|
|
||||||
|
|
||||||
if class == "" {
|
|
||||||
class, id = key, value
|
|
||||||
}
|
|
||||||
|
|
||||||
switch class {
|
|
||||||
case autNumClass:
|
|
||||||
autNum = append(autNum, line)
|
|
||||||
|
|
||||||
if key == orgKey {
|
|
||||||
orgID = value
|
|
||||||
}
|
|
||||||
case organisationClass:
|
|
||||||
organisations[id] = append(organisations[id], line)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
return append(autNum, organisations[orgID]...)
|
|
||||||
}
|
|
||||||
|
|
||||||
// setResponseField stores the value of one "key: value" line of a WHOIS
|
// setResponseField stores the value of one "key: value" line of a WHOIS
|
||||||
// response in info, keeping the first value seen for each field.
|
// response in info, keeping the first value seen for each field.
|
||||||
func (c *Client) setResponseField(info *ASNInfo, key, value string) {
|
func (c *Client) setResponseField(info *ASNInfo, key, value string) {
|
||||||
|
|||||||
+28
-663
@@ -2,21 +2,15 @@ package whois_test
|
|||||||
|
|
||||||
import (
|
import (
|
||||||
"bufio"
|
"bufio"
|
||||||
"bytes"
|
|
||||||
"context"
|
"context"
|
||||||
"encoding/json"
|
|
||||||
"errors"
|
"errors"
|
||||||
"io"
|
"io"
|
||||||
"log/slog"
|
|
||||||
"maps"
|
|
||||||
"net"
|
"net"
|
||||||
"slices"
|
|
||||||
"strings"
|
"strings"
|
||||||
"sync"
|
"sync"
|
||||||
"syscall"
|
"syscall"
|
||||||
"testing"
|
"testing"
|
||||||
"time"
|
"time"
|
||||||
"unicode/utf8"
|
|
||||||
|
|
||||||
"git.eeqj.de/sneak/routewatch/internal/whois"
|
"git.eeqj.de/sneak/routewatch/internal/whois"
|
||||||
)
|
)
|
||||||
@@ -27,20 +21,8 @@ const (
|
|||||||
ripeServer = "whois.ripe.net:43"
|
ripeServer = "whois.ripe.net:43"
|
||||||
)
|
)
|
||||||
|
|
||||||
// testASN is the AS the tests look up, and asQuery the query for it.
|
// testASN is the AS the tests look up.
|
||||||
const (
|
const testASN = 3333
|
||||||
testASN = 3333
|
|
||||||
asQuery = "AS3333"
|
|
||||||
)
|
|
||||||
|
|
||||||
// The fake servers' answers are keyed by a server's address, a space and a
|
|
||||||
// query. These are the keys of the queries LookupASN sends for testASN:
|
|
||||||
// IANA's, then RIPE's, which asks RIPE to leave out person and role
|
|
||||||
// objects.
|
|
||||||
const (
|
|
||||||
ianaQuery = ianaServer + " " + asQuery
|
|
||||||
ripeQuery = ripeServer + " --no-personal " + asQuery
|
|
||||||
)
|
|
||||||
|
|
||||||
// ianaReferral is IANA's answer for an AS in a block it gave to RIPE.
|
// ianaReferral is IANA's answer for an AS in a block it gave to RIPE.
|
||||||
const ianaReferral = `% IANA WHOIS server
|
const ianaReferral = `% IANA WHOIS server
|
||||||
@@ -54,163 +36,27 @@ changed: 2002-09-01
|
|||||||
source: IANA
|
source: IANA
|
||||||
`
|
`
|
||||||
|
|
||||||
// lacnicReferral is IANA's answer for an AS in a block it gave to LACNIC,
|
// ripeRecord is RIPE's answer for the AS.
|
||||||
// shortened to the referral.
|
|
||||||
const lacnicReferral = "refer: whois.lacnic.net\n"
|
|
||||||
|
|
||||||
// ripeRecord is RIPE's answer for the AS when asked to leave out person and
|
|
||||||
// role objects. It starts with the as-block object of the block the AS is
|
|
||||||
// in. Then come the abuse address as a comment, the AS's object and the
|
|
||||||
// organisation object it refers to.
|
|
||||||
const ripeRecord = `% This is the RIPE Database query service.
|
const ripeRecord = `% This is the RIPE Database query service.
|
||||||
% The objects are in RPSL format.
|
|
||||||
|
|
||||||
% Note: --no-personal means ALL personal data has been filtered from this output.
|
|
||||||
|
|
||||||
% Information related to 'AS3209 - AS3353'
|
|
||||||
|
|
||||||
as-block: AS3209 - AS3353
|
|
||||||
descr: RIPE NCC ASN block
|
|
||||||
remarks: These AS Numbers are assigned to network operators
|
|
||||||
remarks: in the RIPE NCC service region.
|
|
||||||
mnt-by: RIPE-NCC-HM-MNT
|
|
||||||
created: 2018-11-22T15:27:19Z
|
|
||||||
last-modified: 2018-11-22T15:27:19Z
|
|
||||||
source: RIPE
|
|
||||||
|
|
||||||
% Information related to 'AS3333'
|
|
||||||
|
|
||||||
% Abuse contact for 'AS3333' is 'abuse@ripe.net'
|
|
||||||
|
|
||||||
aut-num: AS3333
|
aut-num: AS3333
|
||||||
as-name: RIPE-NCC-AS
|
as-name: RIPE-NCC-AS
|
||||||
org: ORG-RIEN1-RIPE
|
org: ORG-RIEN1-RIPE
|
||||||
admin-c: BRD-RIPE
|
|
||||||
tech-c: OPS4-RIPE
|
|
||||||
abuse-c: OPS4-RIPE
|
|
||||||
mnt-by: RIPE-NCC-MNT
|
|
||||||
created: 2002-09-17T13:08:32Z
|
created: 2002-09-17T13:08:32Z
|
||||||
last-modified: 2024-01-09T08:55:41Z
|
last-modified: 2024-01-09T08:55:41Z
|
||||||
source: RIPE
|
source: RIPE
|
||||||
|
|
||||||
organisation: ORG-RIEN1-RIPE
|
|
||||||
org-name: Reseaux IP Europeens Network Coordination Centre (RIPE NCC)
|
|
||||||
country: NL
|
|
||||||
org-type: RIR
|
|
||||||
address: P.O. Box 10096
|
|
||||||
address: 1001EB Amsterdam
|
|
||||||
abuse-c: OPS4-RIPE
|
|
||||||
created: 2012-03-09T13:08:29Z
|
|
||||||
last-modified: 2024-03-26T10:55:38Z
|
|
||||||
source: RIPE
|
|
||||||
`
|
`
|
||||||
|
|
||||||
// The error lines with which RIPE refuses a query: access denied to the
|
|
||||||
// host, and the host's limit on returned objects reached.
|
|
||||||
const (
|
|
||||||
ripeAccessDenied = "%ERROR:201: access denied for 192.0.2.1"
|
|
||||||
ripeLimitReached = "%ERROR:202: access control limit reached"
|
|
||||||
)
|
|
||||||
|
|
||||||
// ripeRefusal is RIPE's answer to a host it no longer serves.
|
// ripeRefusal is RIPE's answer to a host it no longer serves.
|
||||||
const ripeRefusal = `% This is the RIPE Database query service.
|
const ripeRefusal = `% This is the RIPE Database query service.
|
||||||
% The objects are in RPSL format.
|
% The objects are in RPSL format.
|
||||||
%
|
%
|
||||||
` + ripeAccessDenied + `
|
%ERROR:201: access denied for 192.0.2.1
|
||||||
%
|
%
|
||||||
% Sorry, access from your host has been permanently
|
% Sorry, access from your host has been permanently
|
||||||
% denied because of a repeated excessive querying.
|
% denied because of a repeated excessive querying.
|
||||||
`
|
`
|
||||||
|
|
||||||
// apnicAfrinicRecord is the answer of APNIC or AFRINIC for the AS when asked
|
|
||||||
// to leave out person and role objects. Like RIPE's, it holds the abuse
|
|
||||||
// address as a comment, the AS's object and the organisation object.
|
|
||||||
const apnicAfrinicRecord = `% Abuse contact for 'AS3333' is 'abuse@example.net'
|
|
||||||
|
|
||||||
aut-num: AS3333
|
|
||||||
as-name: EXAMPLE-AS
|
|
||||||
org: ORG-EXAMPLE1
|
|
||||||
admin-c: EX1-TEST
|
|
||||||
last-modified: 2026-08-26T00:40:00Z
|
|
||||||
|
|
||||||
organisation: ORG-EXAMPLE1
|
|
||||||
org-name: Example Networks
|
|
||||||
country: ZA
|
|
||||||
address: 1 Example Road
|
|
||||||
`
|
|
||||||
|
|
||||||
// apnicServer is APNIC's WHOIS server, and apnicASN an AS that APNIC holds.
|
|
||||||
const (
|
|
||||||
apnicServer = "whois.apnic.net:43"
|
|
||||||
apnicASN = 4608
|
|
||||||
)
|
|
||||||
|
|
||||||
// The keys of the queries LookupASN sends for apnicASN: IANA's, then
|
|
||||||
// APNIC's, which asks APNIC to leave out person and role objects.
|
|
||||||
const (
|
|
||||||
apnicIANAQuery = ianaServer + " AS4608"
|
|
||||||
apnicQuery = apnicServer + " --no-personal AS4608"
|
|
||||||
)
|
|
||||||
|
|
||||||
// apnicRecord is APNIC's answer for apnicASN when asked to leave out person
|
|
||||||
// and role objects, shortened. It starts with the as-block object of the
|
|
||||||
// block the AS is in. Then come the abuse address as a comment, the AS's
|
|
||||||
// object, and the irt and organisation objects it refers to.
|
|
||||||
const apnicRecord = `% [whois.apnic.net]
|
|
||||||
% Whois data copyright terms http://www.apnic.net/db/dbcopyright.html
|
|
||||||
|
|
||||||
% Note: --no-personal means ALL personal data has been filtered from this output.
|
|
||||||
|
|
||||||
% Information related to 'AS4608 - AS4865'
|
|
||||||
|
|
||||||
as-block: AS4608 - AS4865
|
|
||||||
descr: APNIC ASN block
|
|
||||||
remarks: These AS numbers are further assigned by APNIC
|
|
||||||
remarks: to APNIC members and end-users in the APNIC region
|
|
||||||
mnt-by: APNIC-HM
|
|
||||||
mnt-lower: APNIC-HM
|
|
||||||
last-modified: 2019-05-20T03:52:26Z
|
|
||||||
source: APNIC
|
|
||||||
|
|
||||||
% Information related to 'AS4608'
|
|
||||||
|
|
||||||
% Abuse contact for 'AS4608' is 'abuse@apnic.net'
|
|
||||||
|
|
||||||
aut-num: AS4608
|
|
||||||
as-name: APNIC-SERVICES
|
|
||||||
descr: Asia Pacific Network Information Centre
|
|
||||||
country: AU
|
|
||||||
org: ORG-APNI1-AP
|
|
||||||
admin-c: AIC1-AP
|
|
||||||
abuse-c: AA1589-AP
|
|
||||||
mnt-by: APNIC-HM
|
|
||||||
mnt-irt: IRT-APNIC-IS-AP
|
|
||||||
last-modified: 2026-08-26T00:40:00Z
|
|
||||||
source: APNIC
|
|
||||||
|
|
||||||
irt: IRT-APNIC-IS-AP
|
|
||||||
address: South Brisbane, Australia
|
|
||||||
e-mail: helpdesk@apnic.net
|
|
||||||
abuse-mailbox: abuse@apnic.net
|
|
||||||
mnt-by: MAINT-APNIC-IS-AP
|
|
||||||
last-modified: 2025-11-18T00:27:08Z
|
|
||||||
source: APNIC
|
|
||||||
|
|
||||||
organisation: ORG-APNI1-AP
|
|
||||||
org-name: Asia Pacific Network Information Centre
|
|
||||||
org-type: LIR
|
|
||||||
country: AU
|
|
||||||
address: PO Box 3646
|
|
||||||
address: South Brisbane, QLD 4101,
|
|
||||||
address: Australia
|
|
||||||
e-mail: noc@apnic.net
|
|
||||||
mnt-by: APNIC-HM
|
|
||||||
last-modified: 2026-08-24T12:58:43Z
|
|
||||||
source: APNIC
|
|
||||||
|
|
||||||
% This query was served by the APNIC Whois Service version 1.88.48 (WHOIS-UK2)
|
|
||||||
`
|
|
||||||
|
|
||||||
// ianaPrivateUse is IANA's answer for a private-use AS: its block and no
|
// ianaPrivateUse is IANA's answer for a private-use AS: its block and no
|
||||||
// referral.
|
// referral.
|
||||||
const ianaPrivateUse = `% IANA WHOIS server
|
const ianaPrivateUse = `% IANA WHOIS server
|
||||||
@@ -234,29 +80,20 @@ const fillerLine = "remarks: and more\n"
|
|||||||
const endlessAnswerSize = 4 << 20
|
const endlessAnswerSize = 4 << 20
|
||||||
|
|
||||||
// fakeServers returns a DialContext for a whois.Client that connects to a
|
// fakeServers returns a DialContext for a whois.Client that connects to a
|
||||||
// fake server at each address answers names. Each key of answers is a
|
// fake server for each address in answers, which reads the query and sends
|
||||||
// server's address, a space and a query, and its value is that server's
|
// answers[address]. Connecting to any other address is refused.
|
||||||
// answer to that query. A server reads one query and sends its answer, or
|
|
||||||
// nothing for a query it has no answer for. Connecting to any other address
|
|
||||||
// is refused.
|
|
||||||
func fakeServers(
|
func fakeServers(
|
||||||
t *testing.T, answers map[string]string,
|
t *testing.T, answers map[string]string,
|
||||||
) func(context.Context, string, string) (net.Conn, error) {
|
) func(context.Context, string, string) (net.Conn, error) {
|
||||||
t.Helper()
|
t.Helper()
|
||||||
|
|
||||||
addresses := make(map[string]bool)
|
|
||||||
|
|
||||||
for key := range answers {
|
|
||||||
address, _, _ := strings.Cut(key, " ")
|
|
||||||
addresses[address] = true
|
|
||||||
}
|
|
||||||
|
|
||||||
var servers sync.WaitGroup
|
var servers sync.WaitGroup
|
||||||
|
|
||||||
t.Cleanup(servers.Wait)
|
t.Cleanup(servers.Wait)
|
||||||
|
|
||||||
return func(_ context.Context, _, address string) (net.Conn, error) {
|
return func(_ context.Context, _, address string) (net.Conn, error) {
|
||||||
if !addresses[address] {
|
answer, ok := answers[address]
|
||||||
|
if !ok {
|
||||||
return nil, syscall.ECONNREFUSED
|
return nil, syscall.ECONNREFUSED
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -265,8 +102,7 @@ func fakeServers(
|
|||||||
servers.Go(func() {
|
servers.Go(func() {
|
||||||
defer func() { _ = server.Close() }()
|
defer func() { _ = server.Close() }()
|
||||||
|
|
||||||
query, _ := bufio.NewReader(server).ReadString('\n')
|
_, _ = bufio.NewReader(server).ReadString('\n')
|
||||||
answer := answers[address+" "+strings.TrimSpace(query)]
|
|
||||||
_, _ = io.WriteString(server, answer)
|
_, _ = io.WriteString(server, answer)
|
||||||
})
|
})
|
||||||
|
|
||||||
@@ -280,48 +116,31 @@ func lookUp(
|
|||||||
) (*whois.ASNInfo, error) {
|
) (*whois.ASNInfo, error) {
|
||||||
t.Helper()
|
t.Helper()
|
||||||
|
|
||||||
client := whois.NewClient(slog.New(slog.DiscardHandler))
|
client := whois.NewClient()
|
||||||
client.DialContext = fakeServers(t, answers)
|
client.DialContext = fakeServers(t, answers)
|
||||||
|
|
||||||
return client.LookupASN(t.Context(), testASN)
|
return client.LookupASN(t.Context(), testASN)
|
||||||
}
|
}
|
||||||
|
|
||||||
// TestLookupASNReadsTheRIRAnswer looks up an AS IANA refers to RIPE. RIPE
|
// TestLookupASNReadsTheRIRAnswer looks up an AS IANA refers to RIPE. The
|
||||||
// answers only a query that leaves out person and role objects, and its
|
// fields come from RIPE's answer, the last-modified date included: IANA's
|
||||||
// answer to that still fills the AS's fields: the abuse address from the
|
// "changed:" line dates its block, not the AS.
|
||||||
// comment line, and the organisation's name and country from its object.
|
|
||||||
// The dates are the AS's. IANA's "changed:" line dates the AS's block, and
|
|
||||||
// so do the dates of the as-block object at the start of RIPE's answer.
|
|
||||||
func TestLookupASNReadsTheRIRAnswer(t *testing.T) {
|
func TestLookupASNReadsTheRIRAnswer(t *testing.T) {
|
||||||
t.Parallel()
|
t.Parallel()
|
||||||
|
|
||||||
info, err := lookUp(t, map[string]string{
|
info, err := lookUp(t, map[string]string{
|
||||||
ianaQuery: ianaReferral,
|
ianaServer: ianaReferral,
|
||||||
ripeQuery: ripeRecord,
|
ripeServer: ripeRecord,
|
||||||
})
|
})
|
||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatalf("LookupASN: %v", err)
|
t.Fatalf("LookupASN: %v", err)
|
||||||
}
|
}
|
||||||
|
|
||||||
got := []string{
|
if info.ASName != "RIPE-NCC-AS" || info.OrgID != "ORG-RIEN1-RIPE" ||
|
||||||
info.ASName, info.OrgID, info.OrgName, info.CountryCode,
|
info.RIR != whois.RIRRIPE {
|
||||||
info.AbuseEmail, info.RIR,
|
t.Errorf("LookupASN = AS name %q, org %q, RIR %q; "+
|
||||||
}
|
"want RIPE-NCC-AS, ORG-RIEN1-RIPE, RIPE",
|
||||||
want := []string{
|
info.ASName, info.OrgID, info.RIR)
|
||||||
"RIPE-NCC-AS", "ORG-RIEN1-RIPE",
|
|
||||||
"Reseaux IP Europeens Network Coordination Centre (RIPE NCC)", "NL",
|
|
||||||
"abuse@ripe.net", whois.RIRRIPE,
|
|
||||||
}
|
|
||||||
|
|
||||||
if !slices.Equal(got, want) {
|
|
||||||
t.Errorf("LookupASN = AS name, org ID, org name, country, abuse "+
|
|
||||||
"address, RIR %q; want %q", got, want)
|
|
||||||
}
|
|
||||||
|
|
||||||
wantRegDate := time.Date(2002, 9, 17, 13, 8, 32, 0, time.UTC)
|
|
||||||
if info.RegDate == nil || !info.RegDate.Equal(wantRegDate) {
|
|
||||||
t.Errorf("LookupASN registered = %v, want %v",
|
|
||||||
info.RegDate, wantRegDate)
|
|
||||||
}
|
}
|
||||||
|
|
||||||
wantLastMod := time.Date(2024, 1, 9, 8, 55, 41, 0, time.UTC)
|
wantLastMod := time.Date(2024, 1, 9, 8, 55, 41, 0, time.UTC)
|
||||||
@@ -331,460 +150,6 @@ func TestLookupASNReadsTheRIRAnswer(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// TestLookupASNReadsOnlyTheASAndItsOrganisation looks up an AS at APNIC,
|
|
||||||
// whose answer starts with the as-block object of the block the AS is in.
|
|
||||||
// The last-modified date is the aut-num object's, not the block's. The
|
|
||||||
// address is the organisation object's alone, without the irt object's.
|
|
||||||
func TestLookupASNReadsOnlyTheASAndItsOrganisation(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
client := whois.NewClient(slog.New(slog.DiscardHandler))
|
|
||||||
client.DialContext = fakeServers(t, map[string]string{
|
|
||||||
apnicIANAQuery: "refer: whois.apnic.net\n",
|
|
||||||
apnicQuery: apnicRecord,
|
|
||||||
})
|
|
||||||
|
|
||||||
info, err := client.LookupASN(t.Context(), apnicASN)
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("LookupASN: %v", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
wantLastMod := time.Date(2026, 8, 26, 0, 40, 0, 0, time.UTC)
|
|
||||||
if info.LastMod == nil || !info.LastMod.Equal(wantLastMod) {
|
|
||||||
t.Errorf("LookupASN last modified = %v, want %v",
|
|
||||||
info.LastMod, wantLastMod)
|
|
||||||
}
|
|
||||||
|
|
||||||
got := []string{
|
|
||||||
info.ASName, info.OrgID, info.OrgName, info.CountryCode,
|
|
||||||
info.Address, info.AbuseEmail,
|
|
||||||
}
|
|
||||||
want := []string{
|
|
||||||
"APNIC-SERVICES", "ORG-APNI1-AP",
|
|
||||||
"Asia Pacific Network Information Centre", "AU",
|
|
||||||
"PO Box 3646\nSouth Brisbane, QLD 4101,\nAustralia", "abuse@apnic.net",
|
|
||||||
}
|
|
||||||
|
|
||||||
if !slices.Equal(got, want) {
|
|
||||||
t.Errorf("LookupASN = AS name, org ID, org name, country, address, "+
|
|
||||||
"abuse address %q; want %q", got, want)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestLookupASNAsksAPNICAndAFRINICToLeaveOutPersonalData looks up an AS at
|
|
||||||
// APNIC and at AFRINIC. Each answers only a query that leaves out person
|
|
||||||
// and role objects, as RIPE does. That one answer still holds the
|
|
||||||
// organisation object, so the organisation's name and country are filled.
|
|
||||||
func TestLookupASNAsksAPNICAndAFRINICToLeaveOutPersonalData(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
for _, server := range []string{"whois.apnic.net", "whois.afrinic.net"} {
|
|
||||||
t.Run(server, func(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
address := server + ":43"
|
|
||||||
|
|
||||||
info, err := lookUp(t, map[string]string{
|
|
||||||
ianaQuery: "refer: " + server + "\n",
|
|
||||||
address + " --no-personal " + asQuery: apnicAfrinicRecord,
|
|
||||||
})
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("LookupASN: %v", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
got := []string{
|
|
||||||
info.ASName, info.OrgName, info.CountryCode, info.AbuseEmail,
|
|
||||||
}
|
|
||||||
want := []string{
|
|
||||||
"EXAMPLE-AS", "Example Networks", "ZA", "abuse@example.net",
|
|
||||||
}
|
|
||||||
|
|
||||||
if !slices.Equal(got, want) {
|
|
||||||
t.Errorf("LookupASN = AS name, org name, country, abuse "+
|
|
||||||
"address %q; want %q", got, want)
|
|
||||||
}
|
|
||||||
})
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// The name of an organisation with accented letters, in Latin-1, one byte for
|
|
||||||
// each letter, and in UTF-8.
|
|
||||||
const (
|
|
||||||
latin1OrgName = "Fundaci\xf3n Ra\xedces de Espa\xf1a"
|
|
||||||
utf8OrgName = "Fundación Raíces de España"
|
|
||||||
)
|
|
||||||
|
|
||||||
// TestLookupASNStoresAnswersInUTF8 looks up an AS at RIPE, which answers in
|
|
||||||
// Latin-1, and at LACNIC, which answers in UTF-8. Both answers give the
|
|
||||||
// organisation's name with its accented letters in UTF-8, in the parsed field
|
|
||||||
// and in the stored answer. Before the fix the Latin-1 answer was kept as
|
|
||||||
// received, which is not valid UTF-8.
|
|
||||||
func TestLookupASNStoresAnswersInUTF8(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
for name, answers := range map[string]map[string]string{
|
|
||||||
"RIPE in Latin-1": {
|
|
||||||
ianaQuery: ianaReferral,
|
|
||||||
ripeQuery: "aut-num: AS3333\n" +
|
|
||||||
"org-name: " + latin1OrgName + "\n",
|
|
||||||
},
|
|
||||||
"LACNIC in UTF-8": {
|
|
||||||
ianaQuery: lacnicReferral,
|
|
||||||
"whois.lacnic.net:43 " + asQuery: "aut-num: AS3333\n" +
|
|
||||||
"owner: " + utf8OrgName + "\n",
|
|
||||||
},
|
|
||||||
} {
|
|
||||||
t.Run(name, func(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
info, err := lookUp(t, answers)
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("LookupASN: %v", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
if info.OrgName != utf8OrgName {
|
|
||||||
t.Errorf("LookupASN org name = %q, want %q",
|
|
||||||
info.OrgName, utf8OrgName)
|
|
||||||
}
|
|
||||||
|
|
||||||
if !utf8.ValidString(info.RawResponse) ||
|
|
||||||
!strings.Contains(info.RawResponse, utf8OrgName) {
|
|
||||||
t.Errorf("LookupASN answer = %q, want UTF-8 holding %q",
|
|
||||||
info.RawResponse, utf8OrgName)
|
|
||||||
}
|
|
||||||
})
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// lacnicServer is LACNIC's WHOIS server and nicbrServer NIC.br's, which
|
|
||||||
// registers Brazil's ASes under LACNIC. brazilASN is one of those ASes.
|
|
||||||
const (
|
|
||||||
lacnicServer = "whois.lacnic.net:43"
|
|
||||||
nicbrServer = "whois.registro.br:43"
|
|
||||||
brazilASN = 52751
|
|
||||||
)
|
|
||||||
|
|
||||||
// The keys of the queries LookupASN sends for brazilASN: IANA's, LACNIC's,
|
|
||||||
// then NIC.br's.
|
|
||||||
const (
|
|
||||||
brazilIANAQuery = ianaServer + " AS52751"
|
|
||||||
brazilLACNICQuery = lacnicServer + " AS52751"
|
|
||||||
brazilNICbrQuery = nicbrServer + " AS52751"
|
|
||||||
)
|
|
||||||
|
|
||||||
// The name of brazilASN's organisation in three forms: as LACNIC sends it,
|
|
||||||
// with each accented letter replaced by U+FFFD; as NIC.br sends it, in
|
|
||||||
// Latin-1; and in UTF-8.
|
|
||||||
const (
|
|
||||||
lacnicBrazilOrgName = "Brasrede Telecomunica\U0000FFFD\U0000FFFDes LTDA"
|
|
||||||
nicbrBrazilOrgName = "Brasrede Telecomunica\xe7\xf5es LTDA"
|
|
||||||
brazilOrgName = "Brasrede Telecomunicações LTDA"
|
|
||||||
)
|
|
||||||
|
|
||||||
// nicbrCopyright is the copyright notice at the top of NIC.br's answers.
|
|
||||||
const nicbrCopyright = "% Copyright (c) Nic.br - " +
|
|
||||||
"Use of this data is governed by the Use and\n" +
|
|
||||||
"% Privacy Policy at https://registro.br/upp . Distribution,\n" +
|
|
||||||
"% commercialization, reproduction, and use for advertising or similar\n" +
|
|
||||||
"% purposes are expressly prohibited.\n"
|
|
||||||
|
|
||||||
// lacnicBrazilRecord is LACNIC's answer for brazilASN, recorded on
|
|
||||||
// 2026-10-26. It is NIC.br's answer, which LACNIC passes on after a line
|
|
||||||
// of its own and a space. It is shortened. The client's address is a
|
|
||||||
// documentation address. The contact's name, the contact's object and the
|
|
||||||
// footer are left out.
|
|
||||||
const lacnicBrazilRecord = "% IP Client: 192.0.2.1\n" +
|
|
||||||
" " + nicbrCopyright + `% 2026-10-26T23:12:22-03:00 - 192.0.2.1
|
|
||||||
|
|
||||||
aut-num: AS52751
|
|
||||||
owner: ` + lacnicBrazilOrgName + `
|
|
||||||
owner-c: AVL217
|
|
||||||
routing-c: AVL217
|
|
||||||
abuse-c: AVL217
|
|
||||||
created: 20120829
|
|
||||||
changed: 20130306
|
|
||||||
inetnum: 177.38.152.0/21
|
|
||||||
inetnum: 2804:b34::/32
|
|
||||||
inetnum: 191.243.4.0/22
|
|
||||||
inetnum: 131.72.248.0/22
|
|
||||||
inetnum: 138.94.228.0/22
|
|
||||||
inetnum: 170.233.8.0/22
|
|
||||||
inetnum: 170.246.4.0/22
|
|
||||||
`
|
|
||||||
|
|
||||||
// nicbrRecord is NIC.br's answer for brazilASN, in Latin-1, recorded on
|
|
||||||
// 2026-10-26 and shortened as lacnicBrazilRecord is.
|
|
||||||
const nicbrRecord = nicbrCopyright + `% 2026-10-26T23:14:41-03:00 - 192.0.2.1
|
|
||||||
|
|
||||||
aut-num: AS52751
|
|
||||||
owner: ` + nicbrBrazilOrgName + `
|
|
||||||
owner-c: AVL217
|
|
||||||
routing-c: AVL217
|
|
||||||
abuse-c: AVL217
|
|
||||||
created: 20120829
|
|
||||||
changed: 20130306
|
|
||||||
inetnum: 177.38.152.0/21
|
|
||||||
inetnum: 2804:b34::/32
|
|
||||||
inetnum: 191.243.4.0/22
|
|
||||||
inetnum: 131.72.248.0/22
|
|
||||||
inetnum: 138.94.228.0/22
|
|
||||||
inetnum: 170.233.8.0/22
|
|
||||||
inetnum: 170.246.4.0/22
|
|
||||||
`
|
|
||||||
|
|
||||||
// nicbrOverLimit stands for NIC.br's answer for brazilASN to a client over
|
|
||||||
// its query limit, and is not recorded. Such an answer leaves information
|
|
||||||
// out, here the owner, and says so in its first line.
|
|
||||||
const nicbrOverLimit = `% Query rate limit exceeded. Reduced information.
|
|
||||||
|
|
||||||
aut-num: AS52751
|
|
||||||
created: 20120829
|
|
||||||
changed: 20130306
|
|
||||||
`
|
|
||||||
|
|
||||||
// newNICbrClient returns a client whose fake servers send answers and
|
|
||||||
// whose logger writes JSON lines to logs. It adds one to *nicbrAsks each
|
|
||||||
// time it connects to NIC.br's server.
|
|
||||||
func newNICbrClient(
|
|
||||||
t *testing.T, answers map[string]string, logs io.Writer, nicbrAsks *int,
|
|
||||||
) *whois.Client {
|
|
||||||
t.Helper()
|
|
||||||
|
|
||||||
dial := fakeServers(t, answers)
|
|
||||||
|
|
||||||
client := whois.NewClient(slog.New(slog.NewJSONHandler(logs, nil)))
|
|
||||||
client.DialContext = func(
|
|
||||||
ctx context.Context, network, address string,
|
|
||||||
) (net.Conn, error) {
|
|
||||||
if address == nicbrServer {
|
|
||||||
*nicbrAsks++
|
|
||||||
}
|
|
||||||
|
|
||||||
return dial(ctx, network, address)
|
|
||||||
}
|
|
||||||
|
|
||||||
return client
|
|
||||||
}
|
|
||||||
|
|
||||||
// lookUpBrazilianAS looks brazilASN up with fake servers sending answers,
|
|
||||||
// and reports whether the client connected to NIC.br's server.
|
|
||||||
func lookUpBrazilianAS(
|
|
||||||
t *testing.T, answers map[string]string,
|
|
||||||
) (*whois.ASNInfo, bool, error) {
|
|
||||||
t.Helper()
|
|
||||||
|
|
||||||
nicbrAsks := 0
|
|
||||||
client := newNICbrClient(t, answers, io.Discard, &nicbrAsks)
|
|
||||||
|
|
||||||
info, err := client.LookupASN(t.Context(), brazilASN)
|
|
||||||
|
|
||||||
return info, nicbrAsks > 0, err
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestLookupASNTakesABrazilianASFromNICbr looks up an AS that NIC.br
|
|
||||||
// registers. LACNIC's answer is NIC.br's with each accented letter replaced
|
|
||||||
// by U+FFFD, so the client also asks NIC.br's server, which answers in
|
|
||||||
// Latin-1. The organisation's name has its accented letters, in the parsed
|
|
||||||
// field and in the stored answer.
|
|
||||||
func TestLookupASNTakesABrazilianASFromNICbr(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
info, _, err := lookUpBrazilianAS(t, map[string]string{
|
|
||||||
brazilIANAQuery: lacnicReferral,
|
|
||||||
brazilLACNICQuery: lacnicBrazilRecord,
|
|
||||||
brazilNICbrQuery: nicbrRecord,
|
|
||||||
})
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("LookupASN: %v", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
if info.OrgName != brazilOrgName || info.RIR != whois.RIRLACNIC {
|
|
||||||
t.Errorf("LookupASN org name, RIR = %q, %q; want %q, %q",
|
|
||||||
info.OrgName, info.RIR, brazilOrgName, whois.RIRLACNIC)
|
|
||||||
}
|
|
||||||
|
|
||||||
if !strings.Contains(info.RawResponse, brazilOrgName) {
|
|
||||||
t.Errorf("LookupASN answer = %q, want one holding %q",
|
|
||||||
info.RawResponse, brazilOrgName)
|
|
||||||
}
|
|
||||||
|
|
||||||
wantRegDate := time.Date(2012, 8, 29, 0, 0, 0, 0, time.UTC)
|
|
||||||
if info.RegDate == nil || !info.RegDate.Equal(wantRegDate) {
|
|
||||||
t.Errorf("LookupASN registered = %v, want %v",
|
|
||||||
info.RegDate, wantRegDate)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestLookupASNAsksNICbrOnlyForItsASes looks up an AS at LACNIC whose
|
|
||||||
// answer does not carry NIC.br's copyright notice, as for an AS that LACNIC
|
|
||||||
// registers itself. The client does not connect to NIC.br's server, which
|
|
||||||
// would answer with a record, and keeps LACNIC's fields.
|
|
||||||
func TestLookupASNAsksNICbrOnlyForItsASes(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
info, askedNICbr, err := lookUpBrazilianAS(t, map[string]string{
|
|
||||||
brazilIANAQuery: lacnicReferral,
|
|
||||||
brazilLACNICQuery: "aut-num: AS52751\n" +
|
|
||||||
"owner: " + utf8OrgName + "\n",
|
|
||||||
brazilNICbrQuery: nicbrRecord,
|
|
||||||
})
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("LookupASN: %v", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
if askedNICbr {
|
|
||||||
t.Error("LookupASN asked NIC.br's server about an AS whose " +
|
|
||||||
"answer from LACNIC is not NIC.br's")
|
|
||||||
}
|
|
||||||
|
|
||||||
if info.OrgName != utf8OrgName {
|
|
||||||
t.Errorf("LookupASN org name = %q, want LACNIC's %q",
|
|
||||||
info.OrgName, utf8OrgName)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestLookupASNKeepsLACNICsAnswerWhenNICbrFails looks up an AS that NIC.br
|
|
||||||
// registers when NIC.br's server refuses the connection, sends nothing, or
|
|
||||||
// sends the reduced answer for a client over its query limit. The client
|
|
||||||
// asks NIC.br, and the lookup still succeeds with LACNIC's fields. The
|
|
||||||
// client logs one warning, which names the AS and the reason.
|
|
||||||
func TestLookupASNKeepsLACNICsAnswerWhenNICbrFails(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
for name, test := range map[string]struct {
|
|
||||||
nicbrAnswer map[string]string
|
|
||||||
reason string
|
|
||||||
}{
|
|
||||||
"connection refused": {
|
|
||||||
nicbrAnswer: map[string]string{},
|
|
||||||
reason: syscall.ECONNREFUSED.Error(),
|
|
||||||
},
|
|
||||||
"no answer": {
|
|
||||||
nicbrAnswer: map[string]string{brazilNICbrQuery: ""},
|
|
||||||
reason: "no WHOIS fields",
|
|
||||||
},
|
|
||||||
"over the query limit": {
|
|
||||||
nicbrAnswer: map[string]string{brazilNICbrQuery: nicbrOverLimit},
|
|
||||||
reason: "over its query limit",
|
|
||||||
},
|
|
||||||
} {
|
|
||||||
t.Run(name, func(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
answers := map[string]string{
|
|
||||||
brazilIANAQuery: lacnicReferral,
|
|
||||||
brazilLACNICQuery: lacnicBrazilRecord,
|
|
||||||
}
|
|
||||||
maps.Copy(answers, test.nicbrAnswer)
|
|
||||||
|
|
||||||
var logs bytes.Buffer
|
|
||||||
|
|
||||||
nicbrAsks := 0
|
|
||||||
client := newNICbrClient(t, answers, &logs, &nicbrAsks)
|
|
||||||
|
|
||||||
info, err := client.LookupASN(t.Context(), brazilASN)
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("LookupASN: %v", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
if nicbrAsks == 0 {
|
|
||||||
t.Error("LookupASN did not ask NIC.br's server")
|
|
||||||
}
|
|
||||||
|
|
||||||
if info.OrgName != lacnicBrazilOrgName {
|
|
||||||
t.Errorf("LookupASN org name = %q, want LACNIC's %q",
|
|
||||||
info.OrgName, lacnicBrazilOrgName)
|
|
||||||
}
|
|
||||||
|
|
||||||
var logged struct {
|
|
||||||
Level string `json:"level"`
|
|
||||||
ASN int `json:"asn"`
|
|
||||||
}
|
|
||||||
|
|
||||||
err = json.Unmarshal(logs.Bytes(), &logged)
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("want one log line, got %q: %v", logs.String(), err)
|
|
||||||
}
|
|
||||||
|
|
||||||
if logged.Level != slog.LevelWarn.String() ||
|
|
||||||
logged.ASN != brazilASN ||
|
|
||||||
!strings.Contains(logs.String(), test.reason) {
|
|
||||||
t.Errorf("logged %q, want a warning for AS%d holding %q",
|
|
||||||
logs.String(), brazilASN, test.reason)
|
|
||||||
}
|
|
||||||
})
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestLookupASNSkipsNICbrAfterItsLimit looks up an AS that NIC.br
|
|
||||||
// registers three times with one client, while NIC.br answers that the
|
|
||||||
// client is over its query limit. The first lookup asks NIC.br. The
|
|
||||||
// second, a second before NICbrSkipTime has passed, does not. The third,
|
|
||||||
// once NICbrSkipTime has passed, asks again. Each keeps LACNIC's fields.
|
|
||||||
func TestLookupASNSkipsNICbrAfterItsLimit(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
nicbrAsks := 0
|
|
||||||
client := newNICbrClient(t, map[string]string{
|
|
||||||
brazilIANAQuery: lacnicReferral,
|
|
||||||
brazilLACNICQuery: lacnicBrazilRecord,
|
|
||||||
brazilNICbrQuery: nicbrOverLimit,
|
|
||||||
}, io.Discard, &nicbrAsks)
|
|
||||||
|
|
||||||
start := time.Date(2026, 10, 27, 0, 0, 0, 0, time.UTC)
|
|
||||||
now := start
|
|
||||||
client.Now = func() time.Time { return now }
|
|
||||||
|
|
||||||
for _, step := range []struct {
|
|
||||||
after time.Duration
|
|
||||||
wantAsks int
|
|
||||||
}{
|
|
||||||
{after: 0, wantAsks: 1},
|
|
||||||
{after: whois.NICbrSkipTime - time.Second, wantAsks: 1},
|
|
||||||
{after: whois.NICbrSkipTime, wantAsks: 2},
|
|
||||||
} {
|
|
||||||
now = start.Add(step.after)
|
|
||||||
|
|
||||||
info, err := client.LookupASN(t.Context(), brazilASN)
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("LookupASN after %v: %v", step.after, err)
|
|
||||||
}
|
|
||||||
|
|
||||||
if nicbrAsks != step.wantAsks {
|
|
||||||
t.Errorf("after %v NIC.br's server was asked %d times, want %d",
|
|
||||||
step.after, nicbrAsks, step.wantAsks)
|
|
||||||
}
|
|
||||||
|
|
||||||
if info.OrgName != lacnicBrazilOrgName {
|
|
||||||
t.Errorf("LookupASN after %v org name = %q, want LACNIC's %q",
|
|
||||||
step.after, info.OrgName, lacnicBrazilOrgName)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestLookupASNReportsARefusal looks up an AS IANA refers to RIPE when RIPE
|
|
||||||
// refuses the query, with each of the error lines RIPE refuses with. The
|
|
||||||
// error holds RIPE's line, so the log says why the lookup failed.
|
|
||||||
func TestLookupASNReportsARefusal(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
for _, line := range []string{ripeAccessDenied, ripeLimitReached} {
|
|
||||||
t.Run(line, func(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
_, err := lookUp(t, map[string]string{
|
|
||||||
ianaQuery: ianaReferral,
|
|
||||||
ripeQuery: "% This is the RIPE Database query service.\n" +
|
|
||||||
line + "\n",
|
|
||||||
})
|
|
||||||
if err == nil || !strings.Contains(err.Error(), line) {
|
|
||||||
t.Errorf("LookupASN error = %v, want one holding %q",
|
|
||||||
err, line)
|
|
||||||
}
|
|
||||||
})
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestLookupASNFailsWhenTheRIRGivesNoAnswer looks up an AS IANA refers to
|
// TestLookupASNFailsWhenTheRIRGivesNoAnswer looks up an AS IANA refers to
|
||||||
// RIPE when RIPE's server refuses the connection, and when it answers only
|
// RIPE when RIPE's server refuses the connection, and when it answers only
|
||||||
// with an error. Neither is a lookup that found the AS.
|
// with an error. Neither is a lookup that found the AS.
|
||||||
@@ -792,8 +157,8 @@ func TestLookupASNFailsWhenTheRIRGivesNoAnswer(t *testing.T) {
|
|||||||
t.Parallel()
|
t.Parallel()
|
||||||
|
|
||||||
for name, answers := range map[string]map[string]string{
|
for name, answers := range map[string]map[string]string{
|
||||||
"connection refused": {ianaQuery: ianaReferral},
|
"connection refused": {ianaServer: ianaReferral},
|
||||||
"only an error": {ianaQuery: ianaReferral, ripeQuery: ripeRefusal},
|
"only an error": {ianaServer: ianaReferral, ripeServer: ripeRefusal},
|
||||||
} {
|
} {
|
||||||
t.Run(name, func(t *testing.T) {
|
t.Run(name, func(t *testing.T) {
|
||||||
t.Parallel()
|
t.Parallel()
|
||||||
@@ -812,7 +177,7 @@ func TestLookupASNFailsWhenTheRIRGivesNoAnswer(t *testing.T) {
|
|||||||
func TestLookupASNReportsNoReferral(t *testing.T) {
|
func TestLookupASNReportsNoReferral(t *testing.T) {
|
||||||
t.Parallel()
|
t.Parallel()
|
||||||
|
|
||||||
_, err := lookUp(t, map[string]string{ianaQuery: ianaPrivateUse})
|
_, err := lookUp(t, map[string]string{ianaServer: ianaPrivateUse})
|
||||||
if !errors.Is(err, whois.ErrNoReferral) {
|
if !errors.Is(err, whois.ErrNoReferral) {
|
||||||
t.Errorf("LookupASN error = %v, want ErrNoReferral", err)
|
t.Errorf("LookupASN error = %v, want ErrNoReferral", err)
|
||||||
}
|
}
|
||||||
@@ -824,7 +189,7 @@ func TestLookupASNReportsNoReferral(t *testing.T) {
|
|||||||
func TestLookupASNFailsWhenIANAAnswersNoBlock(t *testing.T) {
|
func TestLookupASNFailsWhenIANAAnswersNoBlock(t *testing.T) {
|
||||||
t.Parallel()
|
t.Parallel()
|
||||||
|
|
||||||
_, err := lookUp(t, map[string]string{ianaQuery: ianaNoObject})
|
_, err := lookUp(t, map[string]string{ianaServer: ianaNoObject})
|
||||||
if err == nil || errors.Is(err, whois.ErrNoReferral) {
|
if err == nil || errors.Is(err, whois.ErrNoReferral) {
|
||||||
t.Errorf("LookupASN error = %v, want one that is not ErrNoReferral",
|
t.Errorf("LookupASN error = %v, want one that is not ErrNoReferral",
|
||||||
err)
|
err)
|
||||||
@@ -841,8 +206,8 @@ func TestLookupASNFailsWhenTheRIRAnswerDoesNotEnd(t *testing.T) {
|
|||||||
strings.Repeat(fillerLine, endlessAnswerSize/len(fillerLine))
|
strings.Repeat(fillerLine, endlessAnswerSize/len(fillerLine))
|
||||||
|
|
||||||
_, err := lookUp(t, map[string]string{
|
_, err := lookUp(t, map[string]string{
|
||||||
ianaQuery: ianaReferral,
|
ianaServer: ianaReferral,
|
||||||
ripeQuery: endless,
|
ripeServer: endless,
|
||||||
})
|
})
|
||||||
if err == nil || errors.Is(err, whois.ErrNoReferral) {
|
if err == nil || errors.Is(err, whois.ErrNoReferral) {
|
||||||
t.Errorf("LookupASN error = %v, want one that is not ErrNoReferral",
|
t.Errorf("LookupASN error = %v, want one that is not ErrNoReferral",
|
||||||
@@ -868,9 +233,9 @@ func TestLookupASNFailsOnAReferralToAnotherServer(t *testing.T) {
|
|||||||
t.Parallel()
|
t.Parallel()
|
||||||
|
|
||||||
_, err := lookUp(t, map[string]string{
|
_, err := lookUp(t, map[string]string{
|
||||||
ianaQuery: "refer: " + refer + "\n\n" +
|
ianaServer: "refer: " + refer + "\n\n" +
|
||||||
"as-block: 3154-3353\n",
|
"as-block: 3154-3353\n",
|
||||||
address + " " + asQuery: ripeRecord,
|
address: ripeRecord,
|
||||||
})
|
})
|
||||||
if err == nil || errors.Is(err, whois.ErrNoReferral) {
|
if err == nil || errors.Is(err, whois.ErrNoReferral) {
|
||||||
t.Errorf("LookupASN error = %v, want one that is not "+
|
t.Errorf("LookupASN error = %v, want one that is not "+
|
||||||
|
|||||||
Binary file not shown.
+23
-49
@@ -2,13 +2,10 @@ package asinfo
|
|||||||
|
|
||||||
import (
|
import (
|
||||||
"bytes"
|
"bytes"
|
||||||
"cmp"
|
|
||||||
"compress/gzip"
|
"compress/gzip"
|
||||||
_ "embed"
|
_ "embed"
|
||||||
"encoding/json"
|
"encoding/json"
|
||||||
"io"
|
"io"
|
||||||
"slices"
|
|
||||||
"strings"
|
|
||||||
"sync"
|
"sync"
|
||||||
)
|
)
|
||||||
|
|
||||||
@@ -26,17 +23,6 @@ type Info struct {
|
|||||||
type Registry struct {
|
type Registry struct {
|
||||||
mu sync.RWMutex
|
mu sync.RWMutex
|
||||||
byASN map[int]*Info
|
byASN map[int]*Info
|
||||||
// searchEntries holds every entry in AS number order, for Search.
|
|
||||||
searchEntries []searchEntry
|
|
||||||
}
|
|
||||||
|
|
||||||
// searchEntry is an entry with its handle and description in lower case. They
|
|
||||||
// are put in lower case once, when the data loads, so that Search only
|
|
||||||
// compares.
|
|
||||||
type searchEntry struct {
|
|
||||||
info *Info
|
|
||||||
lowerHandle string
|
|
||||||
lowerDescription string
|
|
||||||
}
|
}
|
||||||
|
|
||||||
var (
|
var (
|
||||||
@@ -73,21 +59,9 @@ func initRegistry() {
|
|||||||
panic("failed to unmarshal embedded AS data: " + err.Error())
|
panic("failed to unmarshal embedded AS data: " + err.Error())
|
||||||
}
|
}
|
||||||
|
|
||||||
slices.SortFunc(asInfos, func(a, b Info) int {
|
|
||||||
return cmp.Compare(a.ASN, b.ASN)
|
|
||||||
})
|
|
||||||
|
|
||||||
defaultRegistry.searchEntries = make([]searchEntry, 0, len(asInfos))
|
|
||||||
|
|
||||||
for i := range asInfos {
|
for i := range asInfos {
|
||||||
info := &asInfos[i]
|
info := &asInfos[i]
|
||||||
defaultRegistry.byASN[info.ASN] = info
|
defaultRegistry.byASN[info.ASN] = info
|
||||||
defaultRegistry.searchEntries = append(defaultRegistry.searchEntries,
|
|
||||||
searchEntry{
|
|
||||||
info: info,
|
|
||||||
lowerHandle: strings.ToLower(info.Handle),
|
|
||||||
lowerDescription: strings.ToLower(info.Description),
|
|
||||||
})
|
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -155,36 +129,36 @@ func All() []Info {
|
|||||||
return result
|
return result
|
||||||
}
|
}
|
||||||
|
|
||||||
// Search finds the AS entries whose handle or description contains query,
|
// Search searches for AS entries where the handle or description contains the query
|
||||||
// ignoring case. It returns the first limit of them in AS number order, and the
|
// This is a simple case-sensitive substring search
|
||||||
// number of entries that match. An empty query matches nothing.
|
func Search(query string) []Info {
|
||||||
func Search(query string, limit int) ([]Info, int) {
|
|
||||||
once.Do(initRegistry)
|
once.Do(initRegistry)
|
||||||
|
|
||||||
if query == "" {
|
|
||||||
return nil, 0
|
|
||||||
}
|
|
||||||
|
|
||||||
query = strings.ToLower(query)
|
|
||||||
|
|
||||||
defaultRegistry.mu.RLock()
|
defaultRegistry.mu.RLock()
|
||||||
defer defaultRegistry.mu.RUnlock()
|
defer defaultRegistry.mu.RUnlock()
|
||||||
|
|
||||||
var (
|
var results []Info
|
||||||
results []Info
|
|
||||||
total int
|
|
||||||
)
|
|
||||||
|
|
||||||
for _, entry := range defaultRegistry.searchEntries {
|
for _, info := range defaultRegistry.byASN {
|
||||||
if strings.Contains(entry.lowerHandle, query) ||
|
if contains(info.Handle, query) || contains(info.Description, query) {
|
||||||
strings.Contains(entry.lowerDescription, query) {
|
results = append(results, *info)
|
||||||
if total < limit {
|
|
||||||
results = append(results, *entry.info)
|
|
||||||
}
|
|
||||||
|
|
||||||
total++
|
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
return results, total
|
return results
|
||||||
|
}
|
||||||
|
|
||||||
|
// contains is a simple substring search
|
||||||
|
func contains(s, substr string) bool {
|
||||||
|
return len(substr) > 0 && len(s) >= len(substr) && containsImpl(s, substr)
|
||||||
|
}
|
||||||
|
|
||||||
|
func containsImpl(s, substr string) bool {
|
||||||
|
for i := 0; i <= len(s)-len(substr); i++ {
|
||||||
|
if s[i:i+len(substr)] == substr {
|
||||||
|
return true
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
return false
|
||||||
}
|
}
|
||||||
|
|||||||
+20
-112
@@ -1,8 +1,6 @@
|
|||||||
package asinfo_test
|
package asinfo_test
|
||||||
|
|
||||||
import (
|
import (
|
||||||
"cmp"
|
|
||||||
"slices"
|
|
||||||
"strings"
|
"strings"
|
||||||
"testing"
|
"testing"
|
||||||
|
|
||||||
@@ -22,7 +20,7 @@ func TestGet(t *testing.T) {
|
|||||||
name: "AS1 Level 3",
|
name: "AS1 Level 3",
|
||||||
asn: 1,
|
asn: 1,
|
||||||
wantOK: true,
|
wantOK: true,
|
||||||
wantDesc: "Level 3 Parent LLC",
|
wantDesc: "Level 3 Parent, LLC",
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
name: "AS15169 Google",
|
name: "AS15169 Google",
|
||||||
@@ -36,10 +34,10 @@ func TestGet(t *testing.T) {
|
|||||||
wantOK: false,
|
wantOK: false,
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
name: "AS1251 with accented letters",
|
name: "AS0 IANA Reserved",
|
||||||
asn: 1251,
|
asn: 0,
|
||||||
wantOK: true,
|
wantOK: true,
|
||||||
wantDesc: "FUNDAÇÃO DE AMPARO À PESQUISA DO ESTADO SÃO PAULO",
|
wantDesc: "Internet Assigned Numbers Authority",
|
||||||
},
|
},
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -69,7 +67,7 @@ func TestGetDescription(t *testing.T) {
|
|||||||
asn int
|
asn int
|
||||||
want string
|
want string
|
||||||
}{
|
}{
|
||||||
{1, "Level 3 Parent LLC"},
|
{1, "Level 3 Parent, LLC"},
|
||||||
{999999999, ""},
|
{999999999, ""},
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -92,7 +90,8 @@ func TestGetHandle(t *testing.T) {
|
|||||||
asn int
|
asn int
|
||||||
want string
|
want string
|
||||||
}{
|
}{
|
||||||
{1, "LVLT-1"},
|
{1, "LVLT"},
|
||||||
|
{0, "IANA-RSVD"},
|
||||||
{999999999, ""},
|
{999999999, ""},
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -142,9 +141,9 @@ func TestSearch(t *testing.T) {
|
|||||||
wantMin: 10,
|
wantMin: 10,
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
name: "Search for São Paulo",
|
name: "Empty search",
|
||||||
query: "São Paulo",
|
query: "",
|
||||||
wantMin: 1,
|
wantMin: 0,
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
name: "Non-existent org",
|
name: "Non-existent org",
|
||||||
@@ -157,115 +156,26 @@ func TestSearch(t *testing.T) {
|
|||||||
t.Run(tt.name, func(t *testing.T) {
|
t.Run(tt.name, func(t *testing.T) {
|
||||||
t.Parallel()
|
t.Parallel()
|
||||||
|
|
||||||
results, _ := asinfo.Search(tt.query, asinfo.Total())
|
results := asinfo.Search(tt.query)
|
||||||
if len(results) < tt.wantMin {
|
if len(results) < tt.wantMin {
|
||||||
t.Errorf("Search(%q) returned %d results, want at least %d",
|
t.Errorf("Search(%q) returned %d results, want at least %d",
|
||||||
tt.query, len(results), tt.wantMin)
|
tt.query, len(results), tt.wantMin)
|
||||||
}
|
}
|
||||||
|
|
||||||
// Verify all results contain the query, ignoring case
|
// Verify all results contain the query
|
||||||
query := strings.ToLower(tt.query)
|
if tt.query != "" {
|
||||||
for _, r := range results {
|
for _, r := range results {
|
||||||
if !strings.Contains(strings.ToLower(r.Handle), query) &&
|
if !strings.Contains(r.Handle, tt.query) &&
|
||||||
!strings.Contains(strings.ToLower(r.Description), query) {
|
!strings.Contains(r.Description, tt.query) {
|
||||||
t.Errorf("Search result ASN %d doesn't contain query %q",
|
t.Errorf("Search result ASN %d doesn't contain query %q",
|
||||||
r.ASN, tt.query)
|
r.ASN, tt.query)
|
||||||
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
})
|
})
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// TestSearchEmptyQueryMatchesNothing expects an empty query to match no AS,
|
|
||||||
// although every handle contains the empty string.
|
|
||||||
func TestSearchEmptyQueryMatchesNothing(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
results, total := asinfo.Search("", asinfo.Total())
|
|
||||||
if len(results) != 0 || total != 0 {
|
|
||||||
t.Errorf(`Search("") returned %d results and a total of %d, want none`,
|
|
||||||
len(results), total)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestSearchIgnoresCase searches for Cloudflare's handle in lower case and its
|
|
||||||
// description in upper case, and expects both searches to find AS13335.
|
|
||||||
func TestSearchIgnoresCase(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
const cloudflareASN = 13335 // handle CLOUDFLARENET, "Cloudflare Inc."
|
|
||||||
|
|
||||||
isCloudflare := func(info asinfo.Info) bool {
|
|
||||||
return info.ASN == cloudflareASN
|
|
||||||
}
|
|
||||||
|
|
||||||
for _, query := range []string{"cloudflarenet", "CLOUDFLARE INC."} {
|
|
||||||
results, _ := asinfo.Search(query, asinfo.Total())
|
|
||||||
if !slices.ContainsFunc(results, isCloudflare) {
|
|
||||||
t.Errorf("Search(%q) did not find AS%d", query, cloudflareASN)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestSearchOrdersByASN expects the matches of a search with many of them to
|
|
||||||
// come in AS number order.
|
|
||||||
func TestSearchOrdersByASN(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
const query = "University"
|
|
||||||
|
|
||||||
results, _ := asinfo.Search(query, asinfo.Total())
|
|
||||||
|
|
||||||
for i := 1; i < len(results); i++ {
|
|
||||||
if results[i-1].ASN >= results[i].ASN {
|
|
||||||
t.Fatalf("Search(%q): AS%d comes after AS%d",
|
|
||||||
query, results[i].ASN, results[i-1].ASN)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestSearchStopsAtLimit searches for a word that matches more entries than
|
|
||||||
// the limit. It expects exactly limit entries, the first matches in AS number
|
|
||||||
// order, and the number of all matches, counted here from All.
|
|
||||||
func TestSearchStopsAtLimit(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
const (
|
|
||||||
query = "university"
|
|
||||||
limit = 10
|
|
||||||
)
|
|
||||||
|
|
||||||
var matches []asinfo.Info
|
|
||||||
|
|
||||||
for _, info := range asinfo.All() {
|
|
||||||
if strings.Contains(strings.ToLower(info.Handle), query) ||
|
|
||||||
strings.Contains(strings.ToLower(info.Description), query) {
|
|
||||||
matches = append(matches, info)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
if len(matches) <= limit {
|
|
||||||
t.Fatalf("%q matches %d entries; the test needs more than %d",
|
|
||||||
query, len(matches), limit)
|
|
||||||
}
|
|
||||||
|
|
||||||
slices.SortFunc(matches, func(a, b asinfo.Info) int {
|
|
||||||
return cmp.Compare(a.ASN, b.ASN)
|
|
||||||
})
|
|
||||||
|
|
||||||
results, total := asinfo.Search(query, limit)
|
|
||||||
|
|
||||||
if total != len(matches) {
|
|
||||||
t.Errorf("Search(%q, %d): total %d, want %d",
|
|
||||||
query, limit, total, len(matches))
|
|
||||||
}
|
|
||||||
|
|
||||||
if !slices.Equal(results, matches[:limit]) {
|
|
||||||
t.Errorf("Search(%q, %d) = %v, want %v",
|
|
||||||
query, limit, results, matches[:limit])
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestDataIntegrity(t *testing.T) {
|
func TestDataIntegrity(t *testing.T) {
|
||||||
t.Parallel()
|
t.Parallel()
|
||||||
|
|
||||||
@@ -305,13 +215,11 @@ func BenchmarkGet(b *testing.B) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
func BenchmarkSearch(b *testing.B) {
|
func BenchmarkSearch(b *testing.B) {
|
||||||
const limit = 50
|
|
||||||
|
|
||||||
queries := []string{"Google", "Amazon", "Microsoft", "University"}
|
queries := []string{"Google", "Amazon", "Microsoft", "University"}
|
||||||
|
|
||||||
b.ResetTimer()
|
b.ResetTimer()
|
||||||
|
|
||||||
for i := range b.N {
|
for i := range b.N {
|
||||||
asinfo.Search(queries[i%len(queries)], limit)
|
asinfo.Search(queries[i%len(queries)])
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
+3
-5
@@ -2,7 +2,7 @@
|
|||||||
Package asinfo provides information about Autonomous Systems (AS).
|
Package asinfo provides information about Autonomous Systems (AS).
|
||||||
|
|
||||||
The package embeds a comprehensive database of AS numbers, handles, and descriptions
|
The package embeds a comprehensive database of AS numbers, handles, and descriptions
|
||||||
sourced from https://github.com/ipverse/as-metadata. The data is embedded at compile
|
sourced from https://github.com/ipverse/asn-info. The data is embedded at compile
|
||||||
time and provides fast, offline lookups.
|
time and provides fast, offline lookups.
|
||||||
|
|
||||||
Basic usage:
|
Basic usage:
|
||||||
@@ -16,10 +16,8 @@ Basic usage:
|
|||||||
// Get just the description
|
// Get just the description
|
||||||
desc := asinfo.GetDescription(15169) // "Google LLC"
|
desc := asinfo.GetDescription(15169) // "Google LLC"
|
||||||
|
|
||||||
// Search the handles and descriptions, ignoring case, for the first 10
|
// Search for AS entries
|
||||||
// matches in AS number order and the number of all matches
|
results := asinfo.Search("University")
|
||||||
results, total := asinfo.Search("university", 10)
|
|
||||||
fmt.Printf("%d matches\n", total)
|
|
||||||
for _, as := range results {
|
for _, as := range results {
|
||||||
fmt.Printf("AS%d: %s\n", as.ASN, as.Description)
|
fmt.Printf("AS%d: %s\n", as.ASN, as.Description)
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -17,20 +17,13 @@ func ExampleGet() {
|
|||||||
func ExampleGetDescription() {
|
func ExampleGetDescription() {
|
||||||
desc := asinfo.GetDescription(13335)
|
desc := asinfo.GetDescription(13335)
|
||||||
fmt.Println(desc)
|
fmt.Println(desc)
|
||||||
// Output: Cloudflare Inc.
|
// Output: Cloudflare, Inc.
|
||||||
}
|
}
|
||||||
|
|
||||||
func ExampleSearch() {
|
func ExampleSearch() {
|
||||||
const limit = 10
|
results := asinfo.Search("MIT-GATEWAY")
|
||||||
|
|
||||||
results, total := asinfo.Search("mit-gateway", limit)
|
|
||||||
|
|
||||||
fmt.Printf("%d of %d matches\n", len(results), total)
|
|
||||||
|
|
||||||
for _, info := range results {
|
for _, info := range results {
|
||||||
fmt.Printf("AS%d: %s - %s\n", info.ASN, info.Handle, info.Description)
|
fmt.Printf("AS%d: %s - %s\n", info.ASN, info.Handle, info.Description)
|
||||||
}
|
}
|
||||||
// Output:
|
// Output: AS3: MIT-GATEWAYS - Massachusetts Institute of Technology
|
||||||
// 1 of 1 matches
|
|
||||||
// AS3: MIT-GATEWAYS - Massachusetts Institute of Technology
|
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -1,34 +0,0 @@
|
|||||||
#!/bin/sh
|
|
||||||
# script/asupdate: rebuild the AS list built into RouteWatch from
|
|
||||||
# as.csv with cmd/asinfo-gen, which fails when it finds no AS. The new
|
|
||||||
# list replaces the old one only when the generator succeeds. Otherwise
|
|
||||||
# the old list stays as it was and the script exits non-zero.
|
|
||||||
set -eu
|
|
||||||
|
|
||||||
ROOT="$(cd "$(dirname "$0")/.." && pwd -P)"
|
|
||||||
|
|
||||||
AS_DATA="pkg/asinfo/asdata.json.gz"
|
|
||||||
# Next to AS_DATA, so the final mv is a rename on one file system.
|
|
||||||
AS_DATA_TMP="$AS_DATA.tmp"
|
|
||||||
|
|
||||||
main() {
|
|
||||||
cd "$ROOT"
|
|
||||||
|
|
||||||
json="$(mktemp)"
|
|
||||||
trap 'rm -f "$json" "$AS_DATA_TMP"' EXIT
|
|
||||||
# A signal exits through the EXIT trap too.
|
|
||||||
trap 'exit 1' HUP INT TERM
|
|
||||||
|
|
||||||
echo "Updating AS info data..."
|
|
||||||
if ! go run ./cmd/asinfo-gen >"$json"; then
|
|
||||||
echo "asupdate: the generator failed; $AS_DATA is unchanged" >&2
|
|
||||||
exit 1
|
|
||||||
fi
|
|
||||||
|
|
||||||
# -n leaves the time out of the file, so an unchanged list gives the
|
|
||||||
# same bytes.
|
|
||||||
gzip -n -c <"$json" >"$AS_DATA_TMP"
|
|
||||||
mv "$AS_DATA_TMP" "$AS_DATA"
|
|
||||||
}
|
|
||||||
|
|
||||||
main "$@"
|
|
||||||
Reference in New Issue
Block a user