Compare commits
3
Commits
3bf20f4125
..
main
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
1c137c75a7 | ||
|
|
4c0ef132e7 | ||
|
|
2152395f36 |
No files matched your search
+2
-3
@@ -93,12 +93,11 @@
|
|||||||
**/*.dylib
|
**/*.dylib
|
||||||
|
|
||||||
# Local databases. The image carries a source archive of the whole build
|
# Local databases. The image carries a source archive of the whole build
|
||||||
# context, so these would otherwise ship inside it. The daemon's databases
|
# context, so these would otherwise ship inside it. The daemon's database
|
||||||
# are db.sqlite and cache.db, with their -wal and -shm files while it runs.
|
# is db.sqlite, with its -wal and -shm files while it runs.
|
||||||
**/*.db
|
**/*.db
|
||||||
**/*.db-journal
|
**/*.db-journal
|
||||||
**/*.db-wal
|
**/*.db-wal
|
||||||
**/*.db-shm
|
|
||||||
**/*.sqlite
|
**/*.sqlite
|
||||||
**/*.sqlite-wal
|
**/*.sqlite-wal
|
||||||
**/*.sqlite-shm
|
**/*.sqlite-shm
|
||||||
|
|||||||
+1
-3
@@ -68,12 +68,10 @@ go.work
|
|||||||
go.work.sum
|
go.work.sum
|
||||||
|
|
||||||
# Local databases, temporary files and debug output. The daemon's
|
# Local databases, temporary files and debug output. The daemon's
|
||||||
# databases are db.sqlite and cache.db, with their -wal and -shm files
|
# database is db.sqlite, with its -wal and -shm files while it runs.
|
||||||
# while it runs.
|
|
||||||
*.db
|
*.db
|
||||||
*.db-journal
|
*.db-journal
|
||||||
*.db-wal
|
*.db-wal
|
||||||
*.db-shm
|
|
||||||
*.sqlite
|
*.sqlite
|
||||||
*.sqlite-wal
|
*.sqlite-wal
|
||||||
*.sqlite-shm
|
*.sqlite-shm
|
||||||
|
|||||||
@@ -1,20 +0,0 @@
|
|||||||
---
|
|
||||||
title: API Policies
|
|
||||||
last_modified: 2026-10-27
|
|
||||||
---
|
|
||||||
|
|
||||||
A value in an API response is never a string with a label or unit in it. A value
|
|
||||||
of a native type is sent as that type, and its unit goes in the field name, for
|
|
||||||
example `_ms`.
|
|
||||||
|
|
||||||
Wrong:
|
|
||||||
|
|
||||||
```json
|
|
||||||
{ "execution_time": "0 ms" }
|
|
||||||
```
|
|
||||||
|
|
||||||
Right:
|
|
||||||
|
|
||||||
```json
|
|
||||||
{ "execution_time_ms": 0 }
|
|
||||||
```
|
|
||||||
+1
-1
@@ -105,7 +105,7 @@ RUN apt-get update && apt-get install -y --no-install-recommends \
|
|||||||
ARG CACHEBUST
|
ARG CACHEBUST
|
||||||
|
|
||||||
# Create non-root user
|
# Create non-root user
|
||||||
RUN useradd -u 1000 -m routewatch
|
RUN useradd -r -u 1000 -m routewatch
|
||||||
|
|
||||||
RUN mkdir -p /var/lib/berlin.sneak.app.routewatch && chown routewatch:routewatch /var/lib/berlin.sneak.app.routewatch
|
RUN mkdir -p /var/lib/berlin.sneak.app.routewatch && chown routewatch:routewatch /var/lib/berlin.sneak.app.routewatch
|
||||||
|
|
||||||
|
|||||||
+22
-92
@@ -1,6 +1,6 @@
|
|||||||
---
|
---
|
||||||
title: Repository Policies
|
title: Repository Policies
|
||||||
last_modified: 2026-10-27
|
last_modified: 2026-10-18
|
||||||
---
|
---
|
||||||
|
|
||||||
This document covers repository structure, tooling, and workflow standards. Code
|
This document covers repository structure, tooling, and workflow standards. Code
|
||||||
@@ -12,7 +12,6 @@ style conventions are in separate documents:
|
|||||||
- [JavaScript](https://git.eeqj.de/sneak/prompts/raw/branch/main/prompts/CODE_STYLEGUIDE_JS.md)
|
- [JavaScript](https://git.eeqj.de/sneak/prompts/raw/branch/main/prompts/CODE_STYLEGUIDE_JS.md)
|
||||||
- [Python](https://git.eeqj.de/sneak/prompts/raw/branch/main/prompts/CODE_STYLEGUIDE_PYTHON.md)
|
- [Python](https://git.eeqj.de/sneak/prompts/raw/branch/main/prompts/CODE_STYLEGUIDE_PYTHON.md)
|
||||||
- [Go HTTP Server Conventions](https://git.eeqj.de/sneak/prompts/raw/branch/main/prompts/GO_HTTP_SERVER_CONVENTIONS.md)
|
- [Go HTTP Server Conventions](https://git.eeqj.de/sneak/prompts/raw/branch/main/prompts/GO_HTTP_SERVER_CONVENTIONS.md)
|
||||||
- [API Policies](https://git.eeqj.de/sneak/prompts/raw/branch/main/prompts/API_POLICIES.md)
|
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
@@ -175,32 +174,27 @@ style conventions are in separate documents:
|
|||||||
`script/cibuild`. `script/lint`, `script/test`, `script/cibuild` and
|
`script/cibuild`. `script/lint`, `script/test`, `script/cibuild` and
|
||||||
`script/docker` are the four, and there is no fifth — `script/check` runs the
|
`script/docker` are the four, and there is no fifth — `script/check` runs the
|
||||||
two gate phases and `script/fmt-check`, and builds no image of its own. In
|
two gate phases and `script/fmt-check`, and builds no image of its own. In
|
||||||
every stage of the `Dockerfile`, every step that lints, tests or compiles
|
every stage of the `Dockerfile`, `ARG CACHEBUST` comes right after the steps
|
||||||
comes after `ARG CACHEBUST`. `ARG CACHEBUST` comes right after these steps:
|
that install system packages and download Go's modules, or right after `FROM`
|
||||||
- the steps that install system packages
|
in a stage with neither, and every step that lints, tests or compiles comes
|
||||||
- the steps that download Go's modules
|
after it. A step that installs anything else as well, such as a
|
||||||
- the step that runs `script/bootstrap` to install a Python `test` phase's
|
`script/bootstrap` that also installs node packages, comes after it too. A new
|
||||||
packages from `Pipfile.lock`
|
value reruns every `RUN` step after that line; `COPY` steps are compared by
|
||||||
|
content and need nothing. Docker keeps the layers above the line until the
|
||||||
A stage with none of these steps declares `ARG CACHEBUST` right after
|
base image, their own lines or the files they copy change. A stage without the
|
||||||
`FROM`. A step that installs anything else as well, such as a
|
`ARG` line is served from cache on an unchanged tree, so a repository
|
||||||
`script/bootstrap` that also installs node packages, comes after it too. A
|
re-vendors the scripts and adds the `ARG` lines in one commit. The scripts do
|
||||||
new value reruns every `RUN` step after that line; `COPY` steps are compared
|
not pass `--no-cache`: it reruns every step too, but it also builds the layers
|
||||||
by content and need nothing. Docker keeps the layers above the line until
|
above the line again and empties every cache mount, so Go's build cache and
|
||||||
the base image, their own lines or the files they copy change. A stage
|
golangci-lint's, which the canonical Go `Dockerfile` below mounts so that they
|
||||||
without the `ARG` line is served from cache on an unchanged tree, so a
|
survive between check builds, would start empty every time. A bare
|
||||||
repository re-vendors the scripts and adds the `ARG` lines in one commit.
|
`docker build .` is not evidence that anything ran: a sub-second build
|
||||||
The scripts do not pass `--no-cache`: it reruns every step too, but it also
|
reporting success is a cache hit, not a result. Never invalidate by pruning —
|
||||||
builds the layers above the line again and empties every cache mount, so
|
`docker builder prune` and friends destroy a build cache shared with every
|
||||||
Go's build cache and golangci-lint's, which the canonical Go `Dockerfile`
|
other build on the host. When a check is added or changed, prove it works by
|
||||||
below mounts so that they survive between check builds, would start empty
|
planting a defect it must catch and watching the run fail on it, then revert
|
||||||
every time. A bare `docker build .` is not evidence that anything ran: a
|
the defect. A green run alone shows neither that the check ran nor that it
|
||||||
sub-second build reporting success is a cache hit, not a result. Never
|
covers what it should.
|
||||||
invalidate by pruning — `docker builder prune` and friends destroy a build
|
|
||||||
cache shared with every other build on the host. When a check is added or
|
|
||||||
changed, prove it works by planting a defect it must catch and watching the
|
|
||||||
run fail on it, then revert the defect. A green run alone shows neither that
|
|
||||||
the check ran nor that it covers what it should.
|
|
||||||
|
|
||||||
- **The gate phases are separate stages, and the build stage depends on both.**
|
- **The gate phases are separate stages, and the build stage depends on both.**
|
||||||
The lint phase is based on the `golangci/golangci-lint` image (pinned by
|
The lint phase is based on the `golangci/golangci-lint` image (pinned by
|
||||||
@@ -315,9 +309,6 @@ style conventions are in separate documents:
|
|||||||
a package's findings only when the package's files and those of every
|
a package's findings only when the package's files and those of every
|
||||||
package it imports, `go.mod`, the linters and their settings in
|
package it imports, `go.mod`, the linters and their settings in
|
||||||
`.golangci.yml`, and the golangci-lint version are all unchanged.
|
`.golangci.yml`, and the golangci-lint version are all unchanged.
|
||||||
- In a Python `test` phase, the layer of the step that runs
|
|
||||||
`script/bootstrap` to install the packages from `Pipfile.lock`, as
|
|
||||||
described for that phase below.
|
|
||||||
- If the project uses `//go:embed` directives that reference build artifacts
|
- If the project uses `//go:embed` directives that reference build artifacts
|
||||||
(e.g. a web frontend compiled in a separate stage), the lint phase must
|
(e.g. a web frontend compiled in a separate stage), the lint phase must
|
||||||
create placeholder files so the embed directives resolve. Example:
|
create placeholder files so the embed directives resolve. Example:
|
||||||
@@ -372,67 +363,6 @@ style conventions are in separate documents:
|
|||||||
`--build-arg VERSION=...`, as `script/docker` and `script/cibuild` already
|
`--build-arg VERSION=...`, as `script/docker` and `script/cibuild` already
|
||||||
do.
|
do.
|
||||||
|
|
||||||
- **A Python `test` phase that installs its packages from `Pipfile.lock` does it
|
|
||||||
above `ARG CACHEBUST`, so check builds keep them.** The steps are
|
|
||||||
`COPY script/ script/`, `COPY Pipfile Pipfile.lock ./` and
|
|
||||||
`RUN script/bootstrap`, with `ARG CACHEBUST` right after them. Docker keeps
|
|
||||||
what that `RUN` installed until the base image, `script/`, `Pipfile`,
|
|
||||||
`Pipfile.lock` or a line up to and including the `RUN` changes. The tests run
|
|
||||||
after `ARG CACHEBUST`, on every check build. In every other stage, a step that
|
|
||||||
installs Python packages comes after `ARG CACHEBUST`.
|
|
||||||
- When `script/bootstrap` runs in that step, it installs nothing beyond
|
|
||||||
system packages, pipenv and its dependencies, and the packages from
|
|
||||||
`Pipfile.lock`. It installs no Python there: the tests use the base
|
|
||||||
image's Python, which the image's hash pins. If it installs anything else
|
|
||||||
there, the step goes after `ARG CACHEBUST`.
|
|
||||||
- Every external reference must be pinned by hash, so `script/bootstrap`
|
|
||||||
installs pipenv and its dependencies by hash. It does so with
|
|
||||||
`pip install --require-hashes -r` and a requirements file in `script/`
|
|
||||||
that lists each of them with `==` and its `--hash`. pip refuses to install
|
|
||||||
a dependency the file does not list.
|
|
||||||
- It then runs `pipenv install --deploy`, with `--dev` when the tests need
|
|
||||||
the development packages. `--deploy` installs only what `Pipfile.lock`
|
|
||||||
names, checks every package against the lockfile's hashes, and fails when
|
|
||||||
the lockfile is out of date with `Pipfile`.
|
|
||||||
- A kept layer holds nothing a new install would refuse: pip checked pipenv
|
|
||||||
and its dependencies, and pipenv checked the lockfile's packages, against
|
|
||||||
their hashes. pipenv creates its virtual environment with the `virtualenv`
|
|
||||||
package. The packages that `virtualenv` adds come from wheels inside that
|
|
||||||
package, so the hash pip checked for it covers them. Two things in the
|
|
||||||
layer are not fixed by a hash. A package with no ready-built wheel for the
|
|
||||||
image is built from source. pip downloads the build tools without a hash
|
|
||||||
check, so a kept layer's build of such a package may differ from a new
|
|
||||||
one. And system packages keep the versions the package manager gave when
|
|
||||||
the layer was built; a new install may get newer ones.
|
|
||||||
- As with Go's modules, the packages are kept in a layer, not a cache mount.
|
|
||||||
A cache mount is shared by every build on the host, and nothing checks its
|
|
||||||
files again.
|
|
||||||
|
|
||||||
In this example, `script/bootstrap` finds `make` and `git` already
|
|
||||||
installed, installs pipenv and its dependencies by hash, and runs
|
|
||||||
`pipenv install --dev --deploy`:
|
|
||||||
|
|
||||||
```dockerfile
|
|
||||||
# Test phase. The 90-second timeout covers both pytest runs.
|
|
||||||
# python:3.x-slim, YYYY-MM-DD
|
|
||||||
FROM python@sha256:... AS test
|
|
||||||
# make and git are base tools that script/bootstrap installs when they
|
|
||||||
# are missing. Installed here, they stay in their own layer when
|
|
||||||
# script/ changes.
|
|
||||||
RUN apt-get update \
|
|
||||||
&& apt-get install -y --no-install-recommends make git \
|
|
||||||
&& rm -rf /var/lib/apt/lists/*
|
|
||||||
WORKDIR /app
|
|
||||||
COPY script/ script/
|
|
||||||
COPY Pipfile Pipfile.lock ./
|
|
||||||
RUN script/bootstrap
|
|
||||||
ARG CACHEBUST
|
|
||||||
COPY . .
|
|
||||||
RUN timeout 90 sh -c 'pipenv run pytest || { \
|
|
||||||
echo "--- Rerunning with -v for details ---"; \
|
|
||||||
pipenv run pytest -v; exit 1; }'
|
|
||||||
```
|
|
||||||
|
|
||||||
- Every repo should have a Gitea Actions workflow (`.gitea/workflows/`) that
|
- Every repo should have a Gitea Actions workflow (`.gitea/workflows/`) that
|
||||||
runs `script/cibuild` on push, and checks out the repo as its only other step,
|
runs `script/cibuild` on push, and checks out the repo as its only other step,
|
||||||
with `persist-credentials: false`: `script/cibuild` needs no token, and
|
with `persist-credentials: false`: `script/cibuild` needs no token, and
|
||||||
|
|||||||
@@ -10,286 +10,96 @@
|
|||||||
|
|
||||||
# Status
|
# Status
|
||||||
|
|
||||||
RouteWatch is pre-1.0, with no git tags. `main` last took `next` on 2026-10-27,
|
pre-1.0. No git tags. `main` last took `next` on 2026-10-06, through
|
||||||
through https://git.eeqj.de/sneak/routewatch/pulls/166.
|
https://git.eeqj.de/sneak/routewatch/pulls/6. The linter and the tests are
|
||||||
https://git.eeqj.de/sneak/routewatch/pulls/202 carries the commits since then
|
phases of the Docker build, and the Gitea workflow runs `script/cibuild` (the
|
||||||
into `main`. It is labelled `merge-ready` and waits for sneak.
|
format check of the Go code and the markdown, both phases, then the image) on
|
||||||
|
every push. The image sets memory ceilings for a 5 GiB container (README
|
||||||
The linter and the tests are phases of the Docker build. The Gitea workflow runs
|
"Memory") and the README says how to run it under upaas (README "Running under
|
||||||
`script/cibuild` on every push. That script runs the two phases and checks the
|
upaas"). A 25-hour run of `a9b6860` on the live feed peaked at about 0.6 GiB
|
||||||
formatting of the Go code and the markdown. Then it builds the image.
|
(https://git.eeqj.de/sneak/routewatch/issues/3). A run of `825afa2` on the live
|
||||||
|
feed found the prefix handler dropping about a quarter of its updates because
|
||||||
The image sets memory ceilings for a 5 GiB container (README "Memory"). README
|
its route writes were slower than the feed
|
||||||
"Running under upaas" says how to run it.
|
(https://git.eeqj.de/sneak/routewatch/issues/96); the cheaper writes since then
|
||||||
|
are measured by a benchmark, not yet on the live feed.
|
||||||
Five runs of `next` on the live feed have ended since 2026-10-27, from three
|
|
||||||
hours to five days long. The runs found the handlers dropping updates for five
|
|
||||||
causes, and the WHOIS fetcher asking a registry that had refused it. Each is
|
|
||||||
fixed on `next`, and its Completed Steps entry below names its issue. A 21-hour
|
|
||||||
run of `742e759`, which has every fix but one, dropped no update except after
|
|
||||||
the feed stalled. That cause, https://git.eeqj.de/sneak/routewatch/issues/229,
|
|
||||||
is fixed on `next`.
|
|
||||||
|
|
||||||
A run of `e58c663` on the live feed began at 09:19 UTC on 2026-11-10
|
|
||||||
(https://git.eeqj.de/sneak/routewatch/issues/232). It is to confirm that fix.
|
|
||||||
Next Step lists what its outcome reports. The seven IP lookup additions of
|
|
||||||
https://git.eeqj.de/sneak/routewatch/issues/203 landed after it began, so it
|
|
||||||
does not exercise them.
|
|
||||||
|
|
||||||
The order of the project's remaining work is kept on
|
|
||||||
https://git.eeqj.de/sneak/routewatch/issues/86.
|
|
||||||
|
|
||||||
# Next Step
|
# Next Step
|
||||||
|
|
||||||
Post the outcome of the run of `e58c663` on
|
Run `next` on the live feed, as README "Running under upaas" describes, until
|
||||||
https://git.eeqj.de/sneak/routewatch/issues/232. The run lasts at least 24
|
the routing table stops growing. Post its size and memory figures on
|
||||||
hours. If the feed has not stalled or reconnected by then, the run goes on until
|
https://git.eeqj.de/sneak/routewatch/issues/122. The same run makes the final
|
||||||
it does, up to 72 hours in all. The outcome says:
|
check https://git.eeqj.de/sneak/routewatch/issues/96 names: whether the prefix
|
||||||
|
handler still drops updates after the first hour. If it does, a new issue
|
||||||
- whether the feed stalled or reconnected
|
follows with the numbers. Then an audit of the code at the commit that run
|
||||||
- whether any handler dropped an update, and the most messages the prefix
|
tested.
|
||||||
handler's queue held at once
|
|
||||||
- the most messages the prefix handler held back at once
|
|
||||||
- the size of the `origin_changes` table on disk
|
|
||||||
- the process's peak memory against 5 GiB
|
|
||||||
|
|
||||||
File an issue for each problem the run finds.
|
|
||||||
|
|
||||||
# Completed Steps
|
# Completed Steps
|
||||||
|
|
||||||
- 2026-11-10: `README.md` and this file are read whole against the rules of
|
- 2026-10-27: the per-peer route tables are gone, with the code that wrote,
|
||||||
https://git.eeqj.de/sneak/routewatch/issues/157, and every statement except
|
|
||||||
measured figures is checked against the code on `next`. Four README sentences
|
|
||||||
failed the prose rules, on origin changes, PTR lookups and geolocation, and
|
|
||||||
five disagreed with the code, on units, geolocation, a failed database write
|
|
||||||
and `script/test`. Each is rewritten. Status and Next Step describe the
|
|
||||||
live-feed run in progress (closes
|
|
||||||
https://git.eeqj.de/sneak/routewatch/issues/251).
|
|
||||||
- 2026-11-10: the image creates the `routewatch` user as a regular user with UID
|
|
||||||
1000, so the build no longer warns about the system UID range (closes
|
|
||||||
https://git.eeqj.de/sneak/routewatch/issues/248).
|
|
||||||
- 2026-11-10: each result of the IP lookup has `geo`, where the address is. It
|
|
||||||
comes from geojs, with DB-IP's City Lite database as the fallback. The geojs
|
|
||||||
answers stay in memory and in `cache.db` for 7 days (closes
|
|
||||||
https://git.eeqj.de/sneak/routewatch/issues/239).
|
|
||||||
- 2026-11-10: the IP lookup checks the `icloud_private_relay` list of Apple's
|
|
||||||
iCloud Private Relay networks, downloaded once a day. Each result has
|
|
||||||
`flags.is_relay` (closes https://git.eeqj.de/sneak/routewatch/issues/237).
|
|
||||||
- 2026-11-10: the IP lookup checks two ProtonVPN lists from a public mirror of
|
|
||||||
Proton's server list. `protonvpn` holds the /24 networks of ProtonVPN's
|
|
||||||
servers, and `protonvpn_entry` holds the servers' IPv4 and IPv6 addresses
|
|
||||||
(closes https://git.eeqj.de/sneak/routewatch/issues/236).
|
|
||||||
- 2026-11-10: the IP lookup finds PTR names over DNS-over-HTTPS and caches them
|
|
||||||
in memory and in `cache.db` for the record's TTL. A TTL is the time for which
|
|
||||||
a DNS record may be kept (closes
|
|
||||||
https://git.eeqj.de/sneak/routewatch/issues/238).
|
|
||||||
- 2026-11-10: the tests no longer fail when a busy host stalls for several
|
|
||||||
seconds. Most waits for something that should happen are now 15 seconds. Tests
|
|
||||||
that depend on a short idle timeout or a request limit run on a fake clock
|
|
||||||
(closes https://git.eeqj.de/sneak/routewatch/issues/240)
|
|
||||||
- 2026-11-10: the IP lookup checks the public VPN server lists of Mullvad,
|
|
||||||
NordVPN, Private Internet Access and IVPN, and X4BNet's community list. Each
|
|
||||||
result has `flags.is_vpn` and `vpn_providers` (closes
|
|
||||||
https://git.eeqj.de/sneak/routewatch/issues/235).
|
|
||||||
- 2026-11-10: `tor_exits` comes from Onionoo, so it holds IPv6 exits and
|
|
||||||
`is_tor` is true for them (closes
|
|
||||||
https://git.eeqj.de/sneak/routewatch/issues/234).
|
|
||||||
- 2026-11-10: each result of the IP lookup has a `flags` object with `is_tor`
|
|
||||||
and `is_bogon`. The public IP lists are named `ip_lists` in the answer,
|
|
||||||
`internal/iplist` in the code and `iplists` in the state directory. A list
|
|
||||||
file may be up to 200 MiB (closes
|
|
||||||
https://git.eeqj.de/sneak/routewatch/issues/233).
|
|
||||||
- 2026-11-05: the prefix page lists the prefix's origin changes, newest first.
|
|
||||||
The routing table records each change, and the prefix handler writes them to a
|
|
||||||
new `origin_changes` table. The database maintainer prunes each prefix to its
|
|
||||||
newest 500 rows or its rows from the last 72 hours, whichever keeps more. The
|
|
||||||
page shows at most the newest 5,000. Each row shows:
|
|
||||||
- the time
|
|
||||||
- announced or withdrawn
|
|
||||||
- the origin AS
|
|
||||||
- the peer count
|
|
||||||
|
|
||||||
A row's peer count is how many peers carry the prefix just after the change,
|
|
||||||
with any origin AS (closes #161).
|
|
||||||
|
|
||||||
- 2026-11-03: the streamer drops an update for a handler only when that
|
|
||||||
handler's queue is full. It used to drop a random share of the updates once
|
|
||||||
the queue was half full. The prefix handler's queue now holds 100,000
|
|
||||||
messages, about 100 MiB. It used to hold 20,000. While the routing table's
|
|
||||||
`Overview` walks every prefix, the prefix handler applies updates more slowly.
|
|
||||||
In a run on the live feed, the feed paused during such a walk and then sent
|
|
||||||
its backlog. The streamer then dropped about 10,000 of the prefix handler's
|
|
||||||
updates (closes #229)
|
|
||||||
- 2026-11-02: `README.md` is rewritten from scratch, sentence by sentence,
|
|
||||||
against the rules of https://git.eeqj.de/sneak/routewatch/issues/157. Every
|
|
||||||
statement was checked against the code on `next` (closes #157)
|
|
||||||
- 2026-11-02: `TODO.md` is rewritten from scratch, sentence by sentence, against
|
|
||||||
the rules of https://git.eeqj.de/sneak/routewatch/issues/157. Every Completed
|
|
||||||
Steps entry keeps its date, its facts and its issue. Status and Next Step
|
|
||||||
describe the repo as the tracker shows it (part of #157)
|
|
||||||
- 2026-11-02: when the sessions of several peers go down, the prefix handler
|
|
||||||
takes the peers out of the routing table together. The table's `PeersDown`
|
|
||||||
takes several peers out of every prefix in one walk. One walk runs at a time.
|
|
||||||
The next one takes every peer whose session went down since the previous one
|
|
||||||
began. Each peer used to have a walk of its own. A peer's messages stayed held
|
|
||||||
back while the walks queued ahead of its own ran. The handler holds back at
|
|
||||||
most 150,000 messages, about 150 MiB. It used to hold back at most 20,000. The
|
|
||||||
status page shows how many messages the handler holds back (closes #225)
|
|
||||||
- 2026-11-02: working out the home page's cards no longer stops the prefix
|
|
||||||
handler. The walk over the routing table holds the table's lock for 10,000
|
|
||||||
prefixes at a time, then lets the lock go for as long again, as `PeerDown`
|
|
||||||
does. It used to take the lock back at once. The prefix handler then applied
|
|
||||||
about one update per 10,000 prefixes walked. During a walk, its queue could
|
|
||||||
fill, and the streamer then dropped updates for it (closes #226)
|
|
||||||
- 2026-11-02: A skipped WHOIS lookup now makes the fetcher wait longer before
|
|
||||||
the next one, as a failed lookup does. A lookup is skipped when its RIR is
|
|
||||||
paused. An RIR is one of the five regional Internet registries. The longest
|
|
||||||
wait is one minute. The status page does not count a skip as a failure. A skip
|
|
||||||
used to leave the wait as it was. Each lookup asks IANA which RIR holds the AS
|
|
||||||
before it finds that RIR paused. So while most stale ASes belonged to a paused
|
|
||||||
RIR, the fetcher could ask IANA every second (closes #222)
|
|
||||||
- 2026-11-02: A peer's session going down no longer stops the prefix handler.
|
|
||||||
The handler runs the routing table's `PeerDown` on a goroutine of its own, one
|
|
||||||
peer at a time. Meanwhile it goes on taking messages. `PeerDown` holds the
|
|
||||||
table's lock for 10,000 prefixes at a time, then lets the lock go for as long
|
|
||||||
again. The handler holds back the down peer's later messages until that peer's
|
|
||||||
`PeerDown` returns. Then the handler applies them in order, one at a time, and
|
|
||||||
takes new messages in between. It holds back at most 20,000 messages. At that
|
|
||||||
number, `HandleMessage` waits until one of them is applied. When `Stop` runs
|
|
||||||
past its deadline, it logs how many held-back messages were not applied
|
|
||||||
(closes #216)
|
|
||||||
- 2026-11-02: RouteWatch pauses an RIR's or NIC.br's WHOIS server for an hour
|
|
||||||
after the server refuses a query. NIC.br is Brazil's registry. A refused or
|
|
||||||
reset connection is a refusal. So is an answer with an `%ERROR:201:` or
|
|
||||||
`%ERROR:202:` line. Lookups at the other servers go on. An AS whose RIR is
|
|
||||||
paused is looked up after the pause. While NIC.br is paused, a Brazilian AS
|
|
||||||
gets the details from LACNIC's answer. IANA is never paused, because a pause
|
|
||||||
of IANA would stop every lookup. A refusal by IANA is an ordinary failure.
|
|
||||||
NIC.br is also paused for an hour after it says RouteWatch has passed its
|
|
||||||
query limit. Before, the fetcher asked a server again within seconds of a
|
|
||||||
refusal (closes #218)
|
|
||||||
- 2026-11-02: The AS handler never waits for a database write. Its batch holds
|
|
||||||
one entry per AS, with the earliest and the latest time of its updates. So the
|
|
||||||
batch no longer grows with the feed's rate. It used to hold one entry for each
|
|
||||||
AS of each path. At 300,000 entries it waited for the write in progress. In a
|
|
||||||
burst of the feed its queue then filled, and the streamer dropped messages for
|
|
||||||
it (closes #217)
|
|
||||||
- 2026-10-27: The tests of `internal/routewatch` no longer fail on a loaded
|
|
||||||
host. Five of them gave `HandleMessage` two seconds to apply a full prefix
|
|
||||||
batch. Under the race detector that can take longer. A test now waits up to 15
|
|
||||||
seconds for each step it expects. The tests of the handlers' limits now give
|
|
||||||
the handlers small limits. So the package no longer runs past its 90-second
|
|
||||||
limit (closes #213, #215)
|
|
||||||
- 2026-10-27: The server works out the home page's cards every minute until it
|
|
||||||
stops. It used to work them out only once, at startup. The end of the context
|
|
||||||
given to `Start` stopped the refreshes (closes #211)
|
|
||||||
- 2026-10-27: README "Rationale" says why RouteWatch exists. RouteWatch answers
|
|
||||||
our other services' questions about IP addresses, from its own routing table
|
|
||||||
and the WHOIS details it keeps. The section still says what RIS Live does not
|
|
||||||
give (closes #204)
|
|
||||||
- 2026-10-27: Every JSON answer carries `@meta`. A panic on a JSON route answers
|
|
||||||
500 with the JSON error body. `/status.json` and the health check answer 405
|
|
||||||
with the JSON error body to any method they do not take. The 500 and the 405
|
|
||||||
used to have an empty body. The home page's IP lookup shows the whole JSON
|
|
||||||
answer, `@meta` included. It used to show only the answer's `data`, or only
|
|
||||||
the message of an error answer. A test checks `@meta` on every JSON route, in
|
|
||||||
its successes and its errors (closes #201)
|
|
||||||
- 2026-10-27: Every value with a unit in a JSON answer is a number. The unit is
|
|
||||||
at the end of the field name. `@meta.execution_time` is now
|
|
||||||
`execution_time_ms`. In the stats, `uptime` is now `uptime_seconds`,
|
|
||||||
`connection_duration` is now `connection_duration_seconds`, and `memory_usage`
|
|
||||||
is now `memory_usage_bytes`. `whois_stats.current_interval` is now
|
|
||||||
`whois_stats.current_interval_seconds`. They used to be strings such as "12
|
|
||||||
ms", "1h18m34s" or "1.1 GB". `connection_duration_seconds` is null while the
|
|
||||||
feed is not connected. `API_POLICIES.md` is vendored from `sneak/prompts`, and
|
|
||||||
`REPO_POLICIES.md` is re-vendored (closes #199)
|
|
||||||
- 2026-10-27: Every result of the IP lookup has the same fields. A field without
|
|
||||||
a value is null. It used to be left out. An address with no route in the
|
|
||||||
routing table has every route and AS field set to null. `ptr` is an empty list
|
|
||||||
when there are no PTR names. The new field `num_origins` gives the number of
|
|
||||||
origin ASes of the netblock, and `origin_asns` gives which they are (closes
|
|
||||||
#200)
|
|
||||||
- 2026-10-27: With `TRUSTED_PROXIES` unset, the server trusts reverse proxies at
|
|
||||||
the private addresses of RFC 1918. `/ip/`, the home page's `curl` card and the
|
|
||||||
request log then give the client's address from `X-Forwarded-For`. They used
|
|
||||||
to give the proxy's address. A list in `TRUSTED_PROXIES` replaces the default.
|
|
||||||
`TRUSTED_PROXIES` set but empty trusts no proxy (closes #198)
|
|
||||||
- 2026-10-27: The home page has cards with figures of the routing table. One
|
|
||||||
card shows the newest and the oldest kept route. One shows the newest and the
|
|
||||||
oldest updated AS. One shows the AS with the most prefixes. One shows the ASes
|
|
||||||
with a single prefix. One shows fifteen well-known networks. Two cards count
|
|
||||||
the bogon announcements and the prefixes with more than one origin AS. They
|
|
||||||
link to the new pages `/bogons` and `/multiple-origins`, which list them. A
|
|
||||||
new package, `internal/bogon`, tells which prefixes and AS numbers are
|
|
||||||
reserved. The server works the figures out once a minute (closes #154)
|
|
||||||
- 2026-10-27: The per-peer route tables are gone, with the code that wrote,
|
|
||||||
counted and read them. The prefix handler applies each update to the routing
|
counted and read them. The prefix handler applies each update to the routing
|
||||||
table in memory. It writes the table's changes every second. It also stores
|
table in memory and writes the table's changes every second. It also stores
|
||||||
each new prefix with its first-seen time. The image's memory limits fit a full
|
each new prefix with its first-seen time. The image's memory limits are set
|
||||||
routing table of about 1 GB. `GOMEMLIMIT` is 2.5 GiB. The writing connection's
|
for a full routing table of about 1 GB: `GOMEMLIMIT` is 2.5 GiB, the writing
|
||||||
page cache is 512 MiB. SQLite's soft and hard heap limits are 768 MiB and 1
|
connection's page cache 512 MiB, and SQLite's heap limits 768 MiB and 1 GiB
|
||||||
GiB (closes #122)
|
(closes #122)
|
||||||
- 2026-10-27: The pages, the API and the stats read every route from the routing
|
- 2026-10-27: the pages, the API and the stats read every route from the routing
|
||||||
table in memory. For a prefix with several origin ASes, the IP lookup gives
|
table in memory. For a prefix with several origin ASes, the IP lookup gives
|
||||||
the one most peers carry. Its `last_seen` is the newest kept route's time. The
|
the one most peers carry. Its `last_seen` is the newest kept route's time. The
|
||||||
prefix page and API list every peer that carries the prefix. Only the kept
|
prefix page and API list every peer that carries the prefix, with an AS path,
|
||||||
routes show an AS path, a next hop and a time. The stats count the origins
|
next hop and time only for the kept routes. The stats count the origins whose
|
||||||
whose kept routes were all withdrawn, in `origins_without_kept_route` (part of
|
kept routes were all withdrawn, in `origins_without_kept_route` (part of #122)
|
||||||
|
- 2026-10-27: the prefix, AS and peer handlers make a timed write on every tick
|
||||||
|
of their timers, every 1, 2 and 2 seconds. They used to skip every other tick.
|
||||||
|
A full batch's write restarts the timer, so the next timed write comes one
|
||||||
|
interval after that write began (closes #192)
|
||||||
|
- 2026-10-27: once the stop deadline has passed, the AS, peer and peering
|
||||||
|
handlers start no write either. A full batch or a tick of the timer that
|
||||||
|
waited while a write ran past the deadline could still start one. The flush
|
||||||
|
goroutines of the prefix, AS and peer handlers now run the same code (closes
|
||||||
|
#189)
|
||||||
|
- 2026-10-27: once the stop deadline has passed, the prefix handler starts no
|
||||||
|
flush. A batch that filled while a write ran past the deadline could still be
|
||||||
|
written after it (closes #188)
|
||||||
|
- 2026-10-27: the prefix handler applies each update to the routing table in
|
||||||
|
memory, and writes the table's changes to the database every second. A write
|
||||||
|
takes at most 20,000 changed pairs, and the rest follow in the next writes. A
|
||||||
|
failed write is written again later. Startup reads the table before the
|
||||||
|
streamer connects. The pages still read the per-peer route tables (part of
|
||||||
#122)
|
#122)
|
||||||
- 2026-10-27: The prefix, AS and peer handlers make a timed write on every tick
|
- 2026-10-27: the database keeps a copy of the whole routing table in a new
|
||||||
of their timers. The prefix handler's timer ticks every second. The other two
|
|
||||||
tick every two seconds. They used to skip every other tick. A full batch's
|
|
||||||
write restarts the timer. So the next timed write comes one interval after
|
|
||||||
that write began (closes #192)
|
|
||||||
- 2026-10-27: Once the stop deadline has passed, the AS, peer and peering
|
|
||||||
handlers start no write either. Before, a full batch or a timer tick could
|
|
||||||
wait while a write ran past the deadline. Either could then start one more
|
|
||||||
write. The flush goroutines of the prefix, AS and peer handlers now run the
|
|
||||||
same code (closes #189)
|
|
||||||
- 2026-10-27: Once the stop deadline has passed, the prefix handler starts no
|
|
||||||
flush. Before, a batch that filled during a late write could still be written
|
|
||||||
after the deadline (closes #188)
|
|
||||||
- 2026-10-27: The prefix handler applies each update to the routing table in
|
|
||||||
memory. It writes the table's changes to the database every second. A write
|
|
||||||
takes at most 20,000 changed pairs of prefix and origin AS. The rest follow in
|
|
||||||
the next writes. A failed write is written again later. Startup reads the
|
|
||||||
table before the streamer connects. The pages still read the per-peer route
|
|
||||||
tables (part of #122)
|
|
||||||
- 2026-10-27: The database keeps a copy of the whole routing table in a new
|
|
||||||
`routes` table, one row per prefix and origin AS. The peers' numbers are in a
|
`routes` table, one row per prefix and origin AS. The peers' numbers are in a
|
||||||
new `route_peers` table. One function writes the pairs that changed in memory.
|
new `route_peers` table. One function writes the pairs that changed in memory,
|
||||||
Another reads the whole table back into memory at startup. Nothing uses them
|
and another reads the whole table back into memory at startup. Nothing uses
|
||||||
yet (part of #122)
|
them yet (part of #122)
|
||||||
- 2026-10-27: A new package, `internal/routetable`, holds the whole routing
|
- 2026-10-27: a new package, `internal/routetable`, holds the whole routing
|
||||||
table in memory. For each prefix and origin AS it keeps the peers that carry
|
table in memory. For each prefix and origin AS it keeps the peers that carry
|
||||||
it, one bit per peer. It also keeps the two newest of their routes. Nothing
|
it, one bit per peer, and the two newest of their routes. Nothing uses it yet
|
||||||
uses it yet (part of #122)
|
(part of #122)
|
||||||
- 2026-10-27: Each result of the IP lookup lists the public blocklists the
|
- 2026-10-27: each result of the IP lookup lists the public blocklists the
|
||||||
address is on, in `blocklists`. The lists are DShield, Feodo Tracker, two
|
address is on, in `blocklists`. The lists are DShield, Feodo Tracker, two
|
||||||
FireHOL lists, two Spamhaus DROP lists and the Tor exit list. RouteWatch
|
FireHOL lists, Spamhaus DROP and the Tor exit list. RouteWatch downloads each
|
||||||
downloads each list on its own schedule and answers from copies in memory. The
|
list on its own schedule and answers from copies in memory. The copies are
|
||||||
copies are also kept in the state directory's `blocklists` directory. IP
|
also kept in the state directory's `blocklists` directory. IP lookup answers
|
||||||
lookup answers carry `Cache-Control`. An address with no route now has a
|
carry `Cache-Control`. An address with no live route now has a result too,
|
||||||
result too, without route fields. So the IP lookup no longer answers 404
|
without route fields, so the IP lookup no longer answers 404 (closes #158)
|
||||||
(closes #158)
|
- 2026-10-27: the home page has an IP information API card. It shows the `curl`
|
||||||
- 2026-10-27: The home page has an IP information API card. It shows the `curl`
|
command that looks up the visitor's own address at `/ip/{addr}`, and that
|
||||||
command that looks up the visitor's own address at `/ip/{addr}`. It also shows
|
command's live answer (closes #156)
|
||||||
that command's live answer (closes #156)
|
- 2026-10-27: on the AS page, a click on the heading of the IPv4 or the IPv6
|
||||||
- 2026-10-27: On the AS page, a click on the heading of the IPv4 or the IPv6
|
|
||||||
prefix list folds or unfolds the list. Both lists start open (closes #155)
|
prefix list folds or unfolds the list. Both lists start open (closes #155)
|
||||||
- 2026-10-27: The AS page lists its IPv4 and its IPv6 prefixes by mask length,
|
- 2026-10-27: the AS page lists its IPv4 and its IPv6 prefixes by mask length,
|
||||||
shortest first. Prefixes of one mask length are listed by address. They used
|
shortest first, and prefixes of one mask length by address. They used to be
|
||||||
to be sorted by address first, then by mask length (closes #153)
|
listed by address (closes #153)
|
||||||
- 2026-10-27: The connection status on `/` and `/status` gets a spinner. It
|
- 2026-10-27: on `/` and `/status`, a spinner beside the status turns one frame
|
||||||
turns one frame each time the stats show more messages than the time before.
|
each time the stats show more messages than the time before. A page that has
|
||||||
After three refreshes with no new messages, the page shows "stale" in red
|
not seen more messages for three refreshes shows "stale" and the time since
|
||||||
instead. Next to it is the time since the last fresh answer (closes #152)
|
then instead, in red (closes #152)
|
||||||
- 2026-10-27: The WHOIS details of a Brazilian AS keep their accented letters.
|
- 2026-10-27: the WHOIS details of a Brazilian AS keep their accented letters.
|
||||||
LACNIC's answer for such an AS is NIC.br's answer with each accented letter
|
LACNIC's answer for such an AS comes from NIC.br, Brazil's registry, with each
|
||||||
replaced. NIC.br is Brazil's registry. The details now come from NIC.br's own
|
accented letter replaced. The details now come from NIC.br's own server,
|
||||||
server, `whois.registro.br`. When that query fails, the details from LACNIC's
|
`whois.registro.br`. When that query fails, the details from LACNIC's answer
|
||||||
answer are kept and a warning is logged. RouteWatch stops asking NIC.br for an
|
are kept and a warning is logged. After NIC.br answers that RouteWatch is over
|
||||||
hour after NIC.br says RouteWatch is over its query limit (closes #174)
|
its query limit, RouteWatch stops asking NIC.br for an hour (closes #174)
|
||||||
- 2026-10-27: `make asupdate` stops with an error when the download of `as.csv`
|
- 2026-10-27: `make asupdate` stops with an error when the download of `as.csv`
|
||||||
breaks off. The generator used to retry the failed read forever. A record that
|
breaks off. The generator used to retry the failed read forever. A record that
|
||||||
does not parse is still logged and skipped (closes #177)
|
does not parse is still logged and skipped (closes #177)
|
||||||
@@ -298,192 +108,182 @@ File an issue for each problem the run finds.
|
|||||||
one AS. A failed download used to replace the list with an empty file. The
|
one AS. A failed download used to replace the list with an empty file. The
|
||||||
generator reads `as.csv` from `ipverse/as-metadata`, the source's new name
|
generator reads `as.csv` from `ipverse/as-metadata`, the source's new name
|
||||||
(closes #173)
|
(closes #173)
|
||||||
- 2026-10-27: The WHOIS details of an AS at RIPE, APNIC or AFRINIC come only
|
- 2026-10-27: the WHOIS details of an AS at RIPE, APNIC or AFRINIC come only
|
||||||
from its `aut-num` object and the organisation object it names. RIPE and APNIC
|
from its `aut-num` object and the organisation object it names. RIPE and APNIC
|
||||||
start their answer with the `as-block` object of the AS's block. Its dates
|
start their answer with the `as-block` object of the AS's block. Its dates
|
||||||
were stored as the AS's (closes #168)
|
were stored as the AS's (closes #168)
|
||||||
- 2026-10-27: AS names keep their accented letters. The AS list built into
|
- 2026-10-27: AS names keep their accented letters. The AS list built into
|
||||||
RouteWatch is rebuilt from today's `as.csv`, which added a country code
|
RouteWatch is rebuilt from today's `as.csv`, which added a country code
|
||||||
column. That file no longer lists AS 0 or the documentation AS numbers. A
|
column. That file no longer lists AS 0 or the documentation AS numbers. A
|
||||||
WHOIS answer that is not valid UTF-8 is decoded as Latin-1 before it is parsed
|
WHOIS answer that is not valid UTF-8 is decoded as Latin-1, which RIPE and
|
||||||
and stored. RIPE and APNIC send Latin-1 (closes #165)
|
APNIC send, before it is parsed and stored (closes #165)
|
||||||
- 2026-10-27: The Age column on the AS and prefix pages reads in words, such as
|
- 2026-10-27: the Age column on the AS and prefix pages reads in words, such as
|
||||||
"5 minutes ago" or "2 weeks ago". It used to print Go durations such as
|
"5 minutes ago" or "2 weeks ago", instead of Go durations such as "360h0m0s
|
||||||
"360h0m0s ago". An age under a minute reads "now" or in seconds. An age over a
|
ago". An age under a minute reads in seconds, and one over a month in months
|
||||||
month reads in months or years instead of a date (closes #163)
|
or years instead of a date (closes #163)
|
||||||
- 2026-10-26: A write to a stored AS, peering or peer keeps the earlier of the
|
- 2026-10-26: a write to a stored AS, peering or peer keeps the earlier of the
|
||||||
two first-seen times. It keeps the later of the two last-seen times. An old
|
two first-seen times and the later of the two last-seen times. An old update
|
||||||
update from a collector catching up on a backlog no longer moves last seen
|
from a collector catching up on a backlog no longer moves last seen back
|
||||||
back (closes #162)
|
(closes #162)
|
||||||
- 2026-10-26: The status page shows "Route Churn/min". That is the number of
|
- 2026-10-26: the status page shows "Route Churn/min", the number of prefix
|
||||||
prefix announcements and withdrawals from all peers in the last full clock
|
announcements and withdrawals, from all peers, in the last full clock minute.
|
||||||
minute. `/api/v1/stats` reports it as `route_churn_per_min` in `stream`. It
|
`/api/v1/stats` reports it as `route_churn_per_min` in `stream`, which
|
||||||
replaces the moving average `route_churn_per_sec`. The messages and updates
|
replaces the moving average `route_churn_per_sec`. The messages and updates
|
||||||
per second on `/` and `/status` are whole numbers (closes #160)
|
per second on `/` and `/status` are whole numbers (closes #160)
|
||||||
- 2026-10-26: The WHOIS fetcher backs off more slowly. A failure makes its wait
|
- 2026-10-26: the WHOIS fetcher backs off more slowly. A failure makes its wait
|
||||||
1.5 times as long, up to one minute instead of five. Its queries to RIPE,
|
1.5 times as long, up to one minute instead of five. Its queries to RIPE,
|
||||||
APNIC and AFRINIC leave out person and role objects. RIPE counts those objects
|
APNIC and AFRINIC leave out person and role objects, which RIPE counts against
|
||||||
against a daily limit per address. A refused query is logged with the RIR's
|
a daily limit per address. A refused query is logged with the RIR's error line
|
||||||
error line (closes #159)
|
(closes #159)
|
||||||
- 2026-10-26: The home page's AS name search works. It lists the ASes whose
|
- 2026-10-26: the home page's AS name search lists the ASes whose handle or
|
||||||
handle or description holds the text, as links to their pages. The new
|
description holds the text, as links to their pages, from the new
|
||||||
`/api/v1/search/as` answers it. The search ignores case. When the database has
|
`/api/v1/search/as`; the search ignores case. When the database has no record
|
||||||
no record of an AS, its page takes the handle and description from the AS list
|
of an AS, its page takes the handle and description from the AS list built
|
||||||
built into RouteWatch. It still lists the AS's prefixes (closes #151)
|
into RouteWatch and still lists the AS's prefixes (closes #151)
|
||||||
- 2026-10-19: README "HTTP Endpoints" lists every route with what it answers.
|
- 2026-10-19: README "HTTP Endpoints" lists every route with what it answers,
|
||||||
README "Database Schema" matches `schema.sql`. The rest of the README matches
|
"Database Schema" matches `schema.sql`, and the rest of the README matches the
|
||||||
the code. This file is brought up to date (closes #72)
|
code; this file brought up to date (closes #72)
|
||||||
- 2026-10-19: Route writes are cheaper. The live route and prefix tables lose
|
- 2026-10-19: route writes are cheaper: the live route and prefix tables lose
|
||||||
their random UUID keys. The prefix tables lose their duplicate index. A route
|
their random UUID keys and the prefix tables their duplicate index, a route
|
||||||
update sets only the columns that change. Every write goes through one
|
update sets only the columns that change, and every write goes through one
|
||||||
connection with a page cache of 1 GiB (README "Memory"). A benchmark writes
|
connection with a page cache of 1 GiB (README "Memory"). A benchmark writes
|
||||||
the prefix handler's largest batch faster than the feed sends it (closes #96)
|
the prefix handler's largest batch faster than the feed sends it (closes #96)
|
||||||
- 2026-10-19: The checkpoint of the WAL (SQLite's write-ahead log), the
|
- 2026-10-19: the WAL checkpoint, the incremental vacuum and ANALYZE take the
|
||||||
incremental vacuum and `ANALYZE` take the database lock that every batch write
|
database lock every batch write holds, so a vacuum no longer fails with
|
||||||
holds. So a vacuum no longer fails with "database is locked". An `ANALYZE` no
|
"database is locked" and an ANALYZE no longer makes a batch write lose its
|
||||||
longer makes a batch write lose its updates. `ANALYZE` runs as
|
updates; ANALYZE runs as `PRAGMA optimize` with an analysis limit (closes #94)
|
||||||
`PRAGMA optimize` with an analysis limit (closes #94)
|
- 2026-10-19: shutdown stops the streamer first, and each handler writes
|
||||||
- 2026-10-19: Shutdown stops the streamer first. Each handler then writes
|
everything left on its queue before it stops, so the updates waiting there are
|
||||||
everything left on its queue before it stops. So the updates waiting there are
|
|
||||||
no longer lost (closes #69)
|
no longer lost (closes #69)
|
||||||
- 2026-10-19: A path that ends in an AS set of several members has origin AS 0,
|
- 2026-10-19: a path that ends in an AS set of several members has origin AS 0,
|
||||||
which means no known origin. It used to have the set's last member as its
|
no known origin, instead of the set's last member, and the set adds no
|
||||||
origin. The set adds no peerings. The pages show such a route as having no
|
peerings; the pages show such a route as having no known origin (closes #75)
|
||||||
known origin (closes #75)
|
|
||||||
- 2026-10-19: `bgp_peer_count` drops to 0 when the last peer goes down (closes
|
- 2026-10-19: `bgp_peer_count` drops to 0 when the last peer goes down (closes
|
||||||
#92)
|
#92)
|
||||||
- 2026-10-19: `bgp_peer_count` counts every peer the feed has heard from. It
|
- 2026-10-19: `bgp_peer_count` counts every peer the feed has heard from, not
|
||||||
used to count only the peers whose session opened since startup. It drops a
|
only those whose session opened since startup, and drops a peer whose session
|
||||||
peer whose session goes down (closes #74)
|
goes down (closes #74)
|
||||||
- 2026-10-19: The shared files are re-vendored from `sneak/prompts` at
|
- 2026-10-19: the shared files are re-vendored from `sneak/prompts` at
|
||||||
`0653579`. The linter is `golangci-lint` v2.14.0 with its configuration. Lint
|
`0653579`: golangci-lint v2.14.0 and its configuration, lint and test as
|
||||||
and test are `Dockerfile` phases that keep Go's and `golangci-lint`'s caches.
|
`Dockerfile` phases that keep Go's and golangci-lint's caches, markdown
|
||||||
`make fmt` also formats the markdown with `prettier`. The agent guidance moves
|
formatted with prettier, agent guidance in `AGENTS.md`; the code is fixed for
|
||||||
to `AGENTS.md`. The code is fixed for what the new linter configuration finds
|
what the new linter configuration finds (closes #52)
|
||||||
(closes #52)
|
- 2026-10-19: the streamer reads feed lines up to 1 MiB, so a line over 64 KiB
|
||||||
- 2026-10-19: The streamer reads feed lines up to 1 MiB. A line over 64 KiB used
|
no longer ends the connection (closes #76)
|
||||||
to end the connection (closes #76)
|
- 2026-10-19: a peer's live routes are removed when its BGP session goes down;
|
||||||
- 2026-10-19: A peer's live routes are removed when its BGP session goes down.
|
`STATE` messages are never dropped for backpressure, and one that finds its
|
||||||
`STATE` messages are never dropped when a queue is full. One that finds its
|
|
||||||
queue full waits for room ahead of the handler's other messages (closes #56)
|
queue full waits for room ahead of the handler's other messages (closes #56)
|
||||||
- 2026-10-19: The prefix-length pages pick their sample with a bounded number of
|
- 2026-10-19: the prefix-length pages pick their sample with a bounded number of
|
||||||
index lookups. They used to read every route of that length. They list each
|
index lookups instead of reading every route of that length, and list each
|
||||||
prefix once (closes #66)
|
prefix once (closes #66)
|
||||||
- 2026-10-19: A request past its time limit answers 504 instead of 408.
|
- 2026-10-19: a request past its time limit answers 504 instead of 408, and
|
||||||
`/api/v1/stats` answers 504 instead of 500 when its read runs out of time
|
`/api/v1/stats` answers 504 instead of 500 when its read runs out of time
|
||||||
(closes #70)
|
(closes #70)
|
||||||
- 2026-10-19: The AS, peer and prefix handlers write each batch outside the lock
|
- 2026-10-19: the AS, peer and prefix handlers write each batch outside the lock
|
||||||
their message handling needs. So messages keep coming off their queues during
|
their message handling needs, so messages keep coming off their queues during
|
||||||
a write. Meanwhile the next batch grows to at most ten batch sizes (closes
|
a write; meanwhile the next batch grows to at most ten batch sizes (closes
|
||||||
#65)
|
#65)
|
||||||
- 2026-10-19: A live route is identified by prefix and peer. So an announcement
|
- 2026-10-19: a live route is identified by prefix and peer, so an announcement
|
||||||
with a new origin AS replaces the peer's route. A withdrawal removes it
|
with a new origin AS replaces the peer's route and a withdrawal removes it
|
||||||
whatever its origin AS (closes #55)
|
whatever its origin AS (closes #55)
|
||||||
- 2026-10-14: The streamer reconnects when RIS Live ends the stream or sends
|
- 2026-10-14: the streamer reconnects when RIS Live ends the stream or sends
|
||||||
nothing for 60 seconds. The health check reports the feed disconnected
|
nothing for 60 seconds, and the health check reports the feed disconnected
|
||||||
meanwhile (closes #53)
|
meanwhile (closes #53)
|
||||||
- 2026-10-14: The prefix page takes the names of the origin ASes from the prefix
|
- 2026-10-14: the prefix page takes the names of the origin ASes from the prefix
|
||||||
query. It used to read every route of each origin AS (closes #67)
|
query instead of reading every route of each origin AS (closes #67)
|
||||||
- 2026-10-14: The daemon exits with status 1 when its web server cannot listen
|
- 2026-10-14: the daemon exits with status 1 when its web server cannot listen
|
||||||
on its port. It used to run on without a web server (closes #73)
|
on its port, instead of running on without it (closes #73)
|
||||||
- 2026-10-14: The prefix pages parse the prefix before looking it up. One that
|
- 2026-10-14: the prefix pages parse the prefix before looking it up: one that
|
||||||
does not parse answers 400. Any valid way of writing a prefix finds its routes
|
does not parse answers 400, and any valid way of writing a prefix finds its
|
||||||
(closes #71)
|
routes (closes #71)
|
||||||
- 2026-10-14: The IP lookup ignores routes shorter than /8 for IPv4 and /16 for
|
- 2026-10-14: the IP lookup ignores routes shorter than /8 for IPv4 and /16 for
|
||||||
IPv6. So a default route is no longer named as an address's owner (closes #57)
|
IPv6, so a default route is no longer named as an address's owner (closes #57)
|
||||||
- 2026-10-14: A prefix batch keeps each peer's last update for a prefix. It used
|
- 2026-10-14: a prefix batch keeps each peer's last update for a prefix, instead
|
||||||
to keep one update per prefix across all peers (closes #54)
|
of one update per prefix across all peers (closes #54)
|
||||||
- 2026-10-14: `reconnect_count` in `/api/v1/stats` counts every reconnect
|
- 2026-10-14: `reconnect_count` in `/api/v1/stats` counts every reconnect
|
||||||
(closes #68)
|
(closes #68)
|
||||||
- 2026-10-03: `/api/v1/stats` serves the prefix distribution from memory. The
|
- 2026-10-03: `/api/v1/stats` serves the prefix distribution from memory, seeded
|
||||||
distribution is seeded at startup and adjusted on every live-route write. So a
|
at startup and adjusted on every live-route write, so a request no longer
|
||||||
request no longer reads every live route (closes #30)
|
reads every live route (closes #30)
|
||||||
- 2026-10-03: A new database is created with `auto_vacuum` set to incremental.
|
- 2026-10-03: a new database is created with `auto_vacuum` set to incremental,
|
||||||
The setting goes in the connection string, so it is set before the file is
|
through the connection string so it is set before the file is first written,
|
||||||
first written. Each periodic incremental vacuum now returns up to 1000 free
|
and each periodic incremental vacuum now returns up to 1000 free pages; the
|
||||||
pages. The late `PRAGMA auto_vacuum` in `Initialize`, which SQLite ignored, is
|
late `PRAGMA auto_vacuum` in `Initialize`, which SQLite ignored, is gone
|
||||||
gone (closes #43)
|
(closes #43)
|
||||||
- 2026-10-03: Looking up an IP address no longer reads every IPv6 route. Both
|
- 2026-10-03: looking up an IP address no longer reads every IPv6 route: both
|
||||||
families find the most specific live route with at most 33 or 129 lookups on
|
families find the most specific live route with at most 33 or 129 lookups on
|
||||||
the prefix index. The IPv4 range columns are gone. Prefixes from the feed are
|
the prefix index, and the IPv4 range columns are gone. Prefixes from the feed
|
||||||
stored in one text form. So an IPv6 withdrawal, which the feed sends
|
are stored in one text form, so an IPv6 withdrawal, which the feed sends
|
||||||
uncompressed, now removes its route (closes #48)
|
uncompressed, now removes its route (closes #48)
|
||||||
- 2026-10-02: A plain `docker build .` stamps the commit's tag or short commit
|
- 2026-10-02: a plain `docker build .` stamps the commit's tag or short commit
|
||||||
into the page footer. It used to stamp `unknown`. The value comes from
|
(`git describe --tags --always`) into the page footer instead of `unknown`:
|
||||||
`git describe --tags --always`. `.dockerignore` sends `.git` without
|
`.dockerignore` sends `.git` without `.git/config`, a `VERSION` build argument
|
||||||
`.git/config`. A `VERSION` build argument takes precedence. `make build`
|
takes precedence, and `make build` stamps the same value (closes #46)
|
||||||
stamps the same value (closes #46)
|
- 2026-09-29: the entrypoint creates the data directory if it is missing and
|
||||||
- 2026-09-29: The entrypoint creates the data directory if it is missing. It
|
stops the start if a step fails; README "Running under upaas" no longer asks
|
||||||
stops the start if a step fails. README "Running under upaas" no longer asks
|
|
||||||
for the host directory to be created first (closes #42)
|
for the host directory to be created first (closes #42)
|
||||||
- 2026-09-29: `.dockerignore` keeps `.git`, local build output, local databases
|
- 2026-09-29: `.dockerignore` keeps `.git`, local build output, local databases
|
||||||
and `.env` out of the Docker build context. So they stay out of the source
|
and `.env` out of the Docker build context, and so out of the source archive
|
||||||
archive in the image (closes #39)
|
in the image (closes #39)
|
||||||
- 2026-09-29: The README's first line names the MIT licence and the author. The
|
- 2026-09-29: README first line names the MIT license and the author; the
|
||||||
License section says MIT and links `LICENSE`. An Author section is added. This
|
License section now says MIT and links `LICENSE`, and an Author section was
|
||||||
file is brought up to date (closes #38)
|
added; this file brought up to date (closes #38)
|
||||||
- 2026-09-29: `LICENSE` is added, with the MIT licence (closes #1)
|
- 2026-09-29: MIT `LICENSE` (closes #1)
|
||||||
- 2026-09-28: Stopping the daemon while the feed is flowing no longer panics
|
- 2026-09-28: stopping the daemon while the feed is flowing no longer panics
|
||||||
with "send on closed channel". The read loop checks for a stop just before
|
with "send on closed channel": the read loop checks for a stop just before
|
||||||
handing a message to the handler queues. A second `Stop` no longer closes the
|
handing a message to the handler queues, and a second `Stop` no longer closes
|
||||||
queues again (closes #34)
|
the queues again (closes #34)
|
||||||
- 2026-09-28: `docker stop` no longer kills the daemon 2 seconds after the stop
|
- 2026-09-28: `docker stop` no longer kills the daemon 2 seconds after the stop
|
||||||
signal. The entrypoint switches to the `routewatch` user with `setpriv`
|
signal: the entrypoint switches to the `routewatch` user with `setpriv`
|
||||||
instead of `runuser`. So the daemon receives the signal itself. It gets the
|
instead of `runuser`, so the daemon receives the signal itself and gets the
|
||||||
whole wait `docker stop` allows, up to its own 60-second limit (closes #33)
|
whole wait `docker stop` allows, up to its own 60-second limit (closes #33)
|
||||||
- 2026-09-28: RouteWatch is ready to run under upaas. A set but invalid `PORT`,
|
- 2026-09-28: ready to run under upaas: a set but invalid `PORT`,
|
||||||
`XDG_DATA_HOME` or `MALLOC_ARENA_MAX` stops the start. The health check
|
`XDG_DATA_HOME` or `MALLOC_ARENA_MAX` stops the start, the health check
|
||||||
follows `PORT`. README gains a "Running under upaas" section (closes #31)
|
follows `PORT`, README "Running under upaas" section (closes #31)
|
||||||
- 2026-09-22: The database statistics are kept in memory. The counts are seeded
|
- 2026-09-22: realtime in-memory database statistics: counts seeded at startup
|
||||||
at startup and adjusted on every write. The oldest and newest route times come
|
and adjusted on every write, oldest/newest route timestamps via index-end
|
||||||
from a lookup at each end of an index. `/api/v1/stats` no longer scans the
|
lookups; `/api/v1/stats` no longer scans the tables (closes #27)
|
||||||
tables (closes #27)
|
- 2026-09-21: batch writes take the write lock when their transaction begins
|
||||||
- 2026-09-21: Batch writes take the write lock when their transaction begins
|
(`_txlock=immediate`), so they wait out a WAL checkpoint instead of failing
|
||||||
(`_txlock=immediate`). So they wait out a WAL checkpoint instead of failing
|
|
||||||
with "database is locked" (closes #25)
|
with "database is locked" (closes #25)
|
||||||
- 2026-09-21: `MALLOC_ARENA_MAX=2` in the image caps the number of glibc malloc
|
- 2026-09-21: `MALLOC_ARENA_MAX=2` in the image caps glibc malloc arenas, so
|
||||||
arenas. So memory outside the Go runtime no longer grows with the core count
|
memory outside the Go runtime no longer grows with the core count (closes #23)
|
||||||
(closes #23)
|
- 2026-09-21: `GOMEMLIMIT=1536MiB` in the image; README Memory section with the
|
||||||
- 2026-09-21: The image sets `GOMEMLIMIT=1536MiB`. README "Memory" gives the
|
|
||||||
memory budget and the 5 GiB container limit (closes #13)
|
memory budget and the 5 GiB container limit (closes #13)
|
||||||
- 2026-09-21: The four handler queues hold at most 20,000 messages each, down
|
- 2026-09-21: the four handler queues hold at most 20,000 messages each, down
|
||||||
from 100,000 (closes #11)
|
from 100,000 (closes #11)
|
||||||
- 2026-09-21: Two goroutine leaks are fixed. The stats handlers leaked after a
|
- 2026-09-21: two goroutine leaks fixed: the stats handlers after a timeout and
|
||||||
timeout. The streamer's tickers leaked on every reconnect (closes #12)
|
the streamer's tickers on every reconnect (closes #12)
|
||||||
- 2026-09-21: Parsed RIS messages no longer keep the unused `Community` and
|
- 2026-09-21: parsed RIS messages no longer keep the unused `Community` and
|
||||||
`Raw` fields (closes #9)
|
`Raw` fields (closes #9)
|
||||||
- 2026-09-21: `.editorconfig` is added. A Gitea workflow runs `script/cibuild`
|
- 2026-09-21: `.editorconfig`, and a Gitea workflow that runs `script/cibuild`
|
||||||
on every push (closes #14)
|
on every push (closes #14)
|
||||||
- 2026-09-21: The peering handler's AS-path map holds at most 500,000 paths.
|
- 2026-09-21: the peering handler's AS-path map holds at most 500,000 paths and
|
||||||
Every 30 seconds it is swapped for an empty one instead of copied (closes #10)
|
is swapped for an empty one every 30 seconds instead of copied (closes #10)
|
||||||
- 2026-09-21: SQLite's memory is bounded across the whole connection pool. Each
|
- 2026-09-21: SQLite memory bounded across the whole connection pool: a 64 MiB
|
||||||
connection has a 64 MiB page cache. The heap limits are 1 GiB soft and 1.5 GiB
|
page cache on each connection, 1 GiB soft and 1.5 GiB hard heap limits (closes
|
||||||
hard (closes #8)
|
#8)
|
||||||
- 2026-09-21: The Docker build runs the format check, the linter and the tests.
|
- 2026-09-21: the Docker build runs the format check, the linter and the tests,
|
||||||
Linting runs in a separate stage on a `golangci-lint` image pinned by digest
|
linting in a separate stage on a golangci-lint image pinned by digest (closes
|
||||||
(closes #5)
|
#5)
|
||||||
- 2026-09-21: `make test` skips the live feed test, with `-short`. So
|
- 2026-09-21: `make test` skips the live-network feed test (`-short`), so
|
||||||
`make check` no longer depends on the network (closes #2)
|
`make check` no longer depends on the network (closes #2)
|
||||||
- 2026-07-07: The scripts-to-rule-them-all layout is adopted. `script/` holds
|
- 2026-07-07 Adopted scripts-to-rule-them-all: `script/` entrypoints, Makefile
|
||||||
the entrypoints. The `Makefile` targets `check`, `test`, `lint`, `fmt` and
|
shims, README Entrypoints section
|
||||||
`fmt-check` call them. So do `bootstrap`, `setup`, `docker` and `hooks`.
|
- 2026-02-22: repo policy compliance: required policy files, .gitignore update,
|
||||||
README gains an "Entrypoints" section
|
Makefile fmt-check/check/docker/hooks targets, gofmt pass
|
||||||
- 2026-02-22: The repo is brought in line with the repo policies. The required
|
(repo-policies-compliance, unmerged)
|
||||||
policy files are added. `.gitignore` is updated. The `Makefile` gains
|
- 2026-01-01: WAL checkpointing: periodic (5s), on startup with logging,
|
||||||
`fmt-check`, `check`, `docker` and `hooks` targets. The Go code is formatted
|
TRUNCATE mode; fixed slow queries
|
||||||
with `gofmt`. The work was on the branch `repo-policies-compliance`, unmerged
|
- 2025-12-31: status page: navbar, home page with search, oldest/newest route
|
||||||
when this entry was written
|
timestamps, NULL handling in WHOIS stats; container runs as routewatch user
|
||||||
- 2026-01-01: The WAL is checkpointed every 5 seconds, and at startup with a log
|
with proper state dir
|
||||||
line. Checkpoints run in `TRUNCATE` mode. This fixed slow queries
|
- 2025-12-30: structured HTTP request logging, increased timeouts, CIDR prefix
|
||||||
- 2025-12-31: The pages get a navigation bar and a home page with a search. The
|
URL routing fixes, status page metrics and footer
|
||||||
status page shows the oldest and newest route times. The WHOIS stats query
|
- 2025-12-29: Dockerfile multi-stage build with source archive; SQLite
|
||||||
handles `NULL`. The container runs as the `routewatch` user with the right
|
incremental vacuum for non-blocking space reclamation
|
||||||
state directory
|
- 2025-07-27: initial RouteWatch BGP stream monitor
|
||||||
- 2025-12-30: HTTP requests are logged as structured log lines. The HTTP
|
|
||||||
timeouts are longer. Prefix URLs with a slash in them route correctly. The
|
|
||||||
status page gets more metrics and a footer
|
|
||||||
- 2025-12-29: The `Dockerfile` builds in stages and puts a source archive in the
|
|
||||||
image. SQLite reclaims space with incremental vacuum, which does not block
|
|
||||||
- 2025-07-27: The first commit: RouteWatch, a BGP stream monitor
|
|
||||||
|
|
||||||
# Future Steps
|
# Future Steps
|
||||||
|
|
||||||
|
None.
|
||||||
@@ -12,7 +12,7 @@ import (
|
|||||||
|
|
||||||
// parseTimeLimit is how long a test waits for parseCSV to return before it
|
// parseTimeLimit is how long a test waits for parseCSV to return before it
|
||||||
// fails.
|
// fails.
|
||||||
const parseTimeLimit = 15 * time.Second
|
const parseTimeLimit = 2 * time.Second
|
||||||
|
|
||||||
// TestParseCSVReadsTodaysLayout parses an as.csv with the country code column
|
// TestParseCSVReadsTodaysLayout parses an as.csv with the country code column
|
||||||
// that the source added after the description. It expects each AS with its
|
// that the source added after the description. It expects each AS with its
|
||||||
|
|||||||
@@ -8,15 +8,11 @@ require (
|
|||||||
github.com/go-chi/chi/v5 v5.2.2
|
github.com/go-chi/chi/v5 v5.2.2
|
||||||
github.com/google/btree v1.1.3
|
github.com/google/btree v1.1.3
|
||||||
github.com/google/uuid v1.6.0
|
github.com/google/uuid v1.6.0
|
||||||
github.com/hashicorp/golang-lru/v2 v2.0.7
|
|
||||||
github.com/mattn/go-sqlite3 v1.14.29
|
github.com/mattn/go-sqlite3 v1.14.29
|
||||||
github.com/maxmind/mmdbwriter v1.2.0
|
|
||||||
github.com/miekg/dns v1.1.73
|
|
||||||
github.com/oschwald/maxminddb-golang/v2 v2.9.0
|
|
||||||
github.com/rcrowley/go-metrics v0.0.0-20250401214520-65e299d6c5c9
|
github.com/rcrowley/go-metrics v0.0.0-20250401214520-65e299d6c5c9
|
||||||
go.uber.org/fx v1.24.0
|
go.uber.org/fx v1.24.0
|
||||||
go4.org/netipx v0.0.0-20260823151212-3075585bcbeb
|
go4.org/netipx v0.0.0-20260823151212-3075585bcbeb
|
||||||
golang.org/x/term v0.45.0
|
golang.org/x/term v0.33.0
|
||||||
golang.org/x/text v0.43.0
|
golang.org/x/text v0.43.0
|
||||||
)
|
)
|
||||||
|
|
||||||
@@ -24,6 +20,5 @@ require (
|
|||||||
go.uber.org/dig v1.19.0 // indirect
|
go.uber.org/dig v1.19.0 // indirect
|
||||||
go.uber.org/multierr v1.10.0 // indirect
|
go.uber.org/multierr v1.10.0 // indirect
|
||||||
go.uber.org/zap v1.26.0 // indirect
|
go.uber.org/zap v1.26.0 // indirect
|
||||||
golang.org/x/net v0.57.0 // indirect
|
golang.org/x/sys v0.34.0 // indirect
|
||||||
golang.org/x/sys v0.49.0 // indirect
|
|
||||||
)
|
)
|
||||||
@@ -1,5 +1,7 @@
|
|||||||
github.com/bits-and-blooms/bitset v1.26.0 h1:WPvcPZbP1yuKofmjs2SYIFO1EjzIGhCP9ducElLsh5Q=
|
github.com/bits-and-blooms/bitset v1.26.0 h1:WPvcPZbP1yuKofmjs2SYIFO1EjzIGhCP9ducElLsh5Q=
|
||||||
github.com/bits-and-blooms/bitset v1.26.0/go.mod h1:7hO7Gc7Pp1vODcmWvKMRA9BNmbv6a/7QIWpPxHddWR8=
|
github.com/bits-and-blooms/bitset v1.26.0/go.mod h1:7hO7Gc7Pp1vODcmWvKMRA9BNmbv6a/7QIWpPxHddWR8=
|
||||||
|
github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c=
|
||||||
|
github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
|
||||||
github.com/dustin/go-humanize v1.0.1 h1:GzkhY7T5VNhEkwH0PVJgjz+fX1rhBrR7pRT3mDkpeCY=
|
github.com/dustin/go-humanize v1.0.1 h1:GzkhY7T5VNhEkwH0PVJgjz+fX1rhBrR7pRT3mDkpeCY=
|
||||||
github.com/dustin/go-humanize v1.0.1/go.mod h1:Mu1zIs6XwVuF/gI1OepvI0qD18qycQx+mFykh5fBlto=
|
github.com/dustin/go-humanize v1.0.1/go.mod h1:Mu1zIs6XwVuF/gI1OepvI0qD18qycQx+mFykh5fBlto=
|
||||||
github.com/go-chi/chi/v5 v5.2.2 h1:CMwsvRVTbXVytCk1Wd72Zy1LAsAh9GxMmSNWLHCG618=
|
github.com/go-chi/chi/v5 v5.2.2 h1:CMwsvRVTbXVytCk1Wd72Zy1LAsAh9GxMmSNWLHCG618=
|
||||||
@@ -8,20 +10,14 @@ github.com/google/btree v1.1.3 h1:CVpQJjYgC4VbzxeGVHfvZrv1ctoYCAI8vbl07Fcxlyg=
|
|||||||
github.com/google/btree v1.1.3/go.mod h1:qOPhT0dTNdNzV6Z/lhRX0YXUafgPLFUh+gZMl761Gm4=
|
github.com/google/btree v1.1.3/go.mod h1:qOPhT0dTNdNzV6Z/lhRX0YXUafgPLFUh+gZMl761Gm4=
|
||||||
github.com/google/uuid v1.6.0 h1:NIvaJDMOsjHA8n1jAhLSgzrAzy1Hgr+hNrb57e+94F0=
|
github.com/google/uuid v1.6.0 h1:NIvaJDMOsjHA8n1jAhLSgzrAzy1Hgr+hNrb57e+94F0=
|
||||||
github.com/google/uuid v1.6.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo=
|
github.com/google/uuid v1.6.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo=
|
||||||
github.com/hashicorp/golang-lru/v2 v2.0.7 h1:a+bsQ5rvGLjzHuww6tVxozPZFVghXaHOwFs4luLUK2k=
|
|
||||||
github.com/hashicorp/golang-lru/v2 v2.0.7/go.mod h1:QeFd9opnmA6QUJc5vARoKUSoFhyfM2/ZepoAG6RGpeM=
|
|
||||||
github.com/mattn/go-sqlite3 v1.14.29 h1:1O6nRLJKvsi1H2Sj0Hzdfojwt8GiGKm+LOfLaBFaouQ=
|
github.com/mattn/go-sqlite3 v1.14.29 h1:1O6nRLJKvsi1H2Sj0Hzdfojwt8GiGKm+LOfLaBFaouQ=
|
||||||
github.com/mattn/go-sqlite3 v1.14.29/go.mod h1:Uh1q+B4BYcTPb+yiD3kU8Ct7aC0hY9fxUwlHK0RXw+Y=
|
github.com/mattn/go-sqlite3 v1.14.29/go.mod h1:Uh1q+B4BYcTPb+yiD3kU8Ct7aC0hY9fxUwlHK0RXw+Y=
|
||||||
github.com/maxmind/mmdbwriter v1.2.0 h1:hyvDopImmgvle3aR8AaddxXnT0iQH2KWJX3vNfkwzYM=
|
github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM=
|
||||||
github.com/maxmind/mmdbwriter v1.2.0/go.mod h1:EQmKHhk2y9DRVvyNxwCLKC5FrkXZLx4snc5OlLY5XLE=
|
github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
|
||||||
github.com/miekg/dns v1.1.73 h1:uhT8nJxmTrPJYClxVxTCX+CVn6qnzSiybRk72Z6DgrE=
|
|
||||||
github.com/miekg/dns v1.1.73/go.mod h1:RW2Obtfd5NZHvOFe3zYG0W8koWOQtAzyHaLo8vASBuQ=
|
|
||||||
github.com/oschwald/maxminddb-golang/v2 v2.9.0 h1:YpIl3WoA9oIuNvqNaO/aD1N/7Ci2wOqoKH5GWo53H5Y=
|
|
||||||
github.com/oschwald/maxminddb-golang/v2 v2.9.0/go.mod h1:3TxexMBcR0ss3SbyYvnqfsDzDjN4o1KCg/nTsaMAivU=
|
|
||||||
github.com/rcrowley/go-metrics v0.0.0-20250401214520-65e299d6c5c9 h1:bsUq1dX0N8AOIL7EB/X911+m4EHsnWEHeJ0c+3TTBrg=
|
github.com/rcrowley/go-metrics v0.0.0-20250401214520-65e299d6c5c9 h1:bsUq1dX0N8AOIL7EB/X911+m4EHsnWEHeJ0c+3TTBrg=
|
||||||
github.com/rcrowley/go-metrics v0.0.0-20250401214520-65e299d6c5c9/go.mod h1:bCqnVzQkZxMG4s8nGwiZ5l3QUCyqpo9Y+/ZMZ9VjZe4=
|
github.com/rcrowley/go-metrics v0.0.0-20250401214520-65e299d6c5c9/go.mod h1:bCqnVzQkZxMG4s8nGwiZ5l3QUCyqpo9Y+/ZMZ9VjZe4=
|
||||||
github.com/stretchr/testify v1.12.1 h1:EuwCh5fleGS7H32xRwO3wRGT7DxrDhLAT6FF8MpWDWE=
|
github.com/stretchr/testify v1.8.1 h1:w7B6lhMri9wdJUVmEZPGGhZzrYTPvgJArz7wNPgYKsk=
|
||||||
github.com/stretchr/testify v1.12.1/go.mod h1:MDEgiDPPsNp5cuIrHPPCyornHKgEVbtFUmoNlxoYthg=
|
github.com/stretchr/testify v1.8.1/go.mod h1:w2LPCIKwWwSfY2zedu0+kehJoqGctiVI29o6fzry7u4=
|
||||||
go.uber.org/dig v1.19.0 h1:BACLhebsYdpQ7IROQ1AGPjrXcP5dF80U3gKoFzbaq/4=
|
go.uber.org/dig v1.19.0 h1:BACLhebsYdpQ7IROQ1AGPjrXcP5dF80U3gKoFzbaq/4=
|
||||||
go.uber.org/dig v1.19.0/go.mod h1:Us0rSJiThwCv2GteUN0Q7OKvU7n5J4dxZ9JKUXozFdE=
|
go.uber.org/dig v1.19.0/go.mod h1:Us0rSJiThwCv2GteUN0Q7OKvU7n5J4dxZ9JKUXozFdE=
|
||||||
go.uber.org/fx v1.24.0 h1:wE8mruvpg2kiiL1Vqd0CC+tr0/24XIB10Iwp2lLWzkg=
|
go.uber.org/fx v1.24.0 h1:wE8mruvpg2kiiL1Vqd0CC+tr0/24XIB10Iwp2lLWzkg=
|
||||||
@@ -32,18 +28,12 @@ go.uber.org/multierr v1.10.0 h1:S0h4aNzvfcFsC3dRF1jLoaov7oRaKqRGC/pUEJ2yvPQ=
|
|||||||
go.uber.org/multierr v1.10.0/go.mod h1:20+QtiLqy0Nd6FdQB9TLXag12DsQkrbs3htMFfDN80Y=
|
go.uber.org/multierr v1.10.0/go.mod h1:20+QtiLqy0Nd6FdQB9TLXag12DsQkrbs3htMFfDN80Y=
|
||||||
go.uber.org/zap v1.26.0 h1:sI7k6L95XOKS281NhVKOFCUNIvv9e0w4BF8N3u+tCRo=
|
go.uber.org/zap v1.26.0 h1:sI7k6L95XOKS281NhVKOFCUNIvv9e0w4BF8N3u+tCRo=
|
||||||
go.uber.org/zap v1.26.0/go.mod h1:dtElttAiwGvoJ/vj4IwHBS/gXsEu/pZ50mUIRWuG0so=
|
go.uber.org/zap v1.26.0/go.mod h1:dtElttAiwGvoJ/vj4IwHBS/gXsEu/pZ50mUIRWuG0so=
|
||||||
go.yaml.in/yaml/v3 v3.0.5 h1:N6y/pJk8buWs9NY5ERU2HSMfm+IuD/OtfdAnq6kESPw=
|
|
||||||
go.yaml.in/yaml/v3 v3.0.5/go.mod h1:HVTZu1O7/Vkt2N+BFy8Zza+lnLsABggaTM2ZpNIGuKg=
|
|
||||||
go4.org/netipx v0.0.0-20260823151212-3075585bcbeb h1:XBM4hvfwGAttkkiTIFfeigdfcL1xIfdKXqFdgiHGtDs=
|
go4.org/netipx v0.0.0-20260823151212-3075585bcbeb h1:XBM4hvfwGAttkkiTIFfeigdfcL1xIfdKXqFdgiHGtDs=
|
||||||
go4.org/netipx v0.0.0-20260823151212-3075585bcbeb/go.mod h1:PLyyIXexvUFg3Owu6p/WfdlivPbZJsZdgWZlrGope/Y=
|
go4.org/netipx v0.0.0-20260823151212-3075585bcbeb/go.mod h1:PLyyIXexvUFg3Owu6p/WfdlivPbZJsZdgWZlrGope/Y=
|
||||||
golang.org/x/net v0.57.0 h1:K5+3DljvIuDG9/Jv9rvyMywYNFCQ9RSUY6OOTTkT+tE=
|
golang.org/x/sys v0.34.0 h1:H5Y5sJ2L2JRdyv7ROF1he/lPdvFsd0mJHFw2ThKHxLA=
|
||||||
golang.org/x/net v0.57.0/go.mod h1:KpXc8iv+r3XplLAG/f7Jsf9RPszJzdR0f58q9vGOuEU=
|
golang.org/x/sys v0.34.0/go.mod h1:BJP2sWEmIv4KK5OTEluFJCKSidICx8ciO85XgH3Ak8k=
|
||||||
golang.org/x/sync v0.24.0 h1:iqiDrGoDgyuvAKrhLQthUEzU/R378N78gXbBjxed7oA=
|
golang.org/x/term v0.33.0 h1:NuFncQrRcaRvVmgRkvM3j/F00gWIAlcmlB8ACEKmGIg=
|
||||||
golang.org/x/sync v0.24.0/go.mod h1:sUUOizhqBxiL6pEWpqNLUiaJn1ShEbZ6BBqskPbjZm0=
|
golang.org/x/term v0.33.0/go.mod h1:s18+ql9tYWp1IfpV9DmCtQDDSRBUjKaw9M1eAv5UeF0=
|
||||||
golang.org/x/sys v0.49.0 h1:XbzkgYJHdqh/8m2Uu0W/dQv8nktxx4BFHp1M0gROTXA=
|
|
||||||
golang.org/x/sys v0.49.0/go.mod h1:hNLxWAXmnKAxqDtdwIYC4bM9oQPEecfsnNMuSxOs3og=
|
|
||||||
golang.org/x/term v0.45.0 h1:NwWyBmoJCbfTHpxrWoZ9C6/VxOf7ic219I8xZZFdrf0=
|
|
||||||
golang.org/x/term v0.45.0/go.mod h1:9aqxs0blBcrm/n0L9QW0aRVD+ktan8ssZromtqJC43w=
|
|
||||||
golang.org/x/text v0.43.0 h1:1QivrlhwAsnMOcqgOdl9my7s9OyZl8O5/UGUTJVXoWc=
|
golang.org/x/text v0.43.0 h1:1QivrlhwAsnMOcqgOdl9my7s9OyZl8O5/UGUTJVXoWc=
|
||||||
golang.org/x/text v0.43.0/go.mod h1:hbSIYA/amXcRXTFSZqh8tWHEoqRvvbKPJ2uz/KU+a+s=
|
golang.org/x/text v0.43.0/go.mod h1:hbSIYA/amXcRXTFSZqh8tWHEoqRvvbKPJ2uz/KU+a+s=
|
||||||
gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA=
|
gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA=
|
||||||
|
|||||||
@@ -1,7 +1,7 @@
|
|||||||
// Package iplist downloads public IP lists on a schedule, keeps the last good
|
// Package blocklist downloads public IP blocklists on a schedule, keeps the
|
||||||
// copy of each on disk and in memory, and answers which lists contain an
|
// last good copy of each on disk and in memory, and answers which lists
|
||||||
// address.
|
// contain an address.
|
||||||
package iplist
|
package blocklist
|
||||||
|
|
||||||
import (
|
import (
|
||||||
"context"
|
"context"
|
||||||
@@ -29,41 +29,9 @@ const (
|
|||||||
// start with the network's first and last address. Lines that start
|
// start with the network's first and last address. Lines that start
|
||||||
// with # are comments.
|
// with # are comments.
|
||||||
FormatDShield
|
FormatDShield
|
||||||
|
|
||||||
// FormatOnionooJSON is one Onionoo details document. Its relays have
|
|
||||||
// addresses in two fields: exit_addresses, which has IPv4 only, and
|
|
||||||
// or_addresses, which has IPv4 and IPv6, each with a port.
|
|
||||||
FormatOnionooJSON
|
|
||||||
|
|
||||||
// FormatMullvadJSON is Mullvad's JSON list of relays. A relay's addresses
|
|
||||||
// are in ipv4_addr_in, ipv6_addr_in and ipv4_v2ray.
|
|
||||||
FormatMullvadJSON
|
|
||||||
|
|
||||||
// FormatNordVPNJSON is NordVPN's JSON list of servers. A server's
|
|
||||||
// addresses are in station and ipv6_station.
|
|
||||||
FormatNordVPNJSON
|
|
||||||
|
|
||||||
// FormatPIAJSON is Private Internet Access's JSON server list, then a
|
|
||||||
// blank line and a signature. Each region's servers has lists of servers
|
|
||||||
// by protocol, and each server has its address in ip.
|
|
||||||
FormatPIAJSON
|
|
||||||
|
|
||||||
// FormatIVPNJSON is IVPN's JSON server list. The hosts of its wireguard
|
|
||||||
// and openvpn gateways have their addresses in host and v2ray.
|
|
||||||
FormatIVPNJSON
|
|
||||||
|
|
||||||
// FormatProtonVPN has one address or network per line, with comment lines
|
|
||||||
// that start with #. Unlike FormatLines, a file with no addresses or
|
|
||||||
// networks fails.
|
|
||||||
FormatProtonVPN
|
|
||||||
|
|
||||||
// FormatICloudPrivateRelayCSV is Apple's CSV file of iCloud Private Relay
|
|
||||||
// networks. Each line starts with a network, followed by its country,
|
|
||||||
// region and city.
|
|
||||||
FormatICloudPrivateRelayCSV
|
|
||||||
)
|
)
|
||||||
|
|
||||||
// List is one public IP list.
|
// List is one public blocklist.
|
||||||
type List struct {
|
type List struct {
|
||||||
// Name names the list in lookup answers, and names its copy on disk.
|
// Name names the list in lookup answers, and names its copy on disk.
|
||||||
Name string
|
Name string
|
||||||
@@ -80,9 +48,6 @@ type List struct {
|
|||||||
|
|
||||||
// Format is how the list's file is written.
|
// Format is how the list's file is written.
|
||||||
Format Format
|
Format Format
|
||||||
|
|
||||||
// VPN says that the list holds servers of VPN services.
|
|
||||||
VPN bool
|
|
||||||
}
|
}
|
||||||
|
|
||||||
// Listing says that a list contains an address.
|
// Listing says that a list contains an address.
|
||||||
@@ -93,10 +58,6 @@ type Listing struct {
|
|||||||
// Updated is when routewatch last downloaded the list, or last heard from
|
// Updated is when routewatch last downloaded the list, or last heard from
|
||||||
// its publisher that its copy was still current.
|
// its publisher that its copy was still current.
|
||||||
Updated time.Time `json:"updated"`
|
Updated time.Time `json:"updated"`
|
||||||
|
|
||||||
// VPN says that the list holds servers of VPN services. A lookup answer
|
|
||||||
// reports it through is_vpn and vpn_providers.
|
|
||||||
VPN bool `json:"-"`
|
|
||||||
}
|
}
|
||||||
|
|
||||||
// listCopy is a list's last good copy in memory.
|
// listCopy is a list's last good copy in memory.
|
||||||
@@ -119,9 +80,9 @@ type Params struct {
|
|||||||
Logger *slog.Logger
|
Logger *slog.Logger
|
||||||
}
|
}
|
||||||
|
|
||||||
// IPLists keeps a copy of each list and answers which lists contain an
|
// Blocklists keeps a copy of each list and answers which lists contain an
|
||||||
// address. Start loads the copies from disk and starts downloading the lists.
|
// address. Start loads the copies from disk and starts downloading the lists.
|
||||||
type IPLists struct {
|
type Blocklists struct {
|
||||||
dir string
|
dir string
|
||||||
lists []List
|
lists []List
|
||||||
logger *slog.Logger
|
logger *slog.Logger
|
||||||
@@ -136,12 +97,12 @@ type IPLists struct {
|
|||||||
done chan struct{}
|
done chan struct{}
|
||||||
}
|
}
|
||||||
|
|
||||||
// New returns an IPLists with no copies. Start loads them.
|
// New returns a Blocklists with no copies. Start loads them.
|
||||||
func New(params Params) *IPLists {
|
func New(params Params) *Blocklists {
|
||||||
return &IPLists{
|
return &Blocklists{
|
||||||
dir: params.Dir,
|
dir: params.Dir,
|
||||||
lists: params.Lists,
|
lists: params.Lists,
|
||||||
logger: params.Logger.With("component", "iplists"),
|
logger: params.Logger.With("component", "blocklists"),
|
||||||
copies: make(map[string]listCopy),
|
copies: make(map[string]listCopy),
|
||||||
done: make(chan struct{}),
|
done: make(chan struct{}),
|
||||||
}
|
}
|
||||||
@@ -150,7 +111,7 @@ func New(params Params) *IPLists {
|
|||||||
// Lookup returns a Listing for each list whose copy contains addr, in the
|
// Lookup returns a Listing for each list whose copy contains addr, in the
|
||||||
// order of the lists given to New. It makes no outside request. When no list
|
// order of the lists given to New. It makes no outside request. When no list
|
||||||
// contains addr, it returns an empty list, not nil.
|
// contains addr, it returns an empty list, not nil.
|
||||||
func (b *IPLists) Lookup(addr netip.Addr) []Listing {
|
func (b *Blocklists) Lookup(addr netip.Addr) []Listing {
|
||||||
b.mu.RLock()
|
b.mu.RLock()
|
||||||
defer b.mu.RUnlock()
|
defer b.mu.RUnlock()
|
||||||
|
|
||||||
@@ -163,7 +124,6 @@ func (b *IPLists) Lookup(addr netip.Addr) []Listing {
|
|||||||
Name: list.Name,
|
Name: list.Name,
|
||||||
Description: list.Description,
|
Description: list.Description,
|
||||||
Updated: c.updated,
|
Updated: c.updated,
|
||||||
VPN: list.VPN,
|
|
||||||
})
|
})
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -173,7 +133,7 @@ func (b *IPLists) Lookup(addr netip.Addr) []Listing {
|
|||||||
|
|
||||||
// ShortestInterval returns the shortest Interval of the lists. That is the
|
// ShortestInterval returns the shortest Interval of the lists. That is the
|
||||||
// soonest a copy can change, and so how long a lookup answer stays current.
|
// soonest a copy can change, and so how long a lookup answer stays current.
|
||||||
func (b *IPLists) ShortestInterval() time.Duration {
|
func (b *Blocklists) ShortestInterval() time.Duration {
|
||||||
var shortest time.Duration
|
var shortest time.Duration
|
||||||
|
|
||||||
for i, list := range b.lists {
|
for i, list := range b.lists {
|
||||||
@@ -187,7 +147,7 @@ func (b *IPLists) ShortestInterval() time.Duration {
|
|||||||
|
|
||||||
// updated returns when the named list's copy was last downloaded or found
|
// updated returns when the named list's copy was last downloaded or found
|
||||||
// current, or the zero time when the list has no copy.
|
// current, or the zero time when the list has no copy.
|
||||||
func (b *IPLists) updated(name string) time.Time {
|
func (b *Blocklists) updated(name string) time.Time {
|
||||||
b.mu.RLock()
|
b.mu.RLock()
|
||||||
defer b.mu.RUnlock()
|
defer b.mu.RUnlock()
|
||||||
|
|
||||||
@@ -196,7 +156,7 @@ func (b *IPLists) updated(name string) time.Time {
|
|||||||
|
|
||||||
// copyValidators returns the validators of the named list's copy. They are
|
// copyValidators returns the validators of the named list's copy. They are
|
||||||
// empty when the list has no copy.
|
// empty when the list has no copy.
|
||||||
func (b *IPLists) copyValidators(name string) validators {
|
func (b *Blocklists) copyValidators(name string) validators {
|
||||||
b.mu.RLock()
|
b.mu.RLock()
|
||||||
defer b.mu.RUnlock()
|
defer b.mu.RUnlock()
|
||||||
|
|
||||||
@@ -204,7 +164,7 @@ func (b *IPLists) copyValidators(name string) validators {
|
|||||||
}
|
}
|
||||||
|
|
||||||
// setCopy makes c the named list's copy.
|
// setCopy makes c the named list's copy.
|
||||||
func (b *IPLists) setCopy(name string, c listCopy) {
|
func (b *Blocklists) setCopy(name string, c listCopy) {
|
||||||
b.mu.Lock()
|
b.mu.Lock()
|
||||||
defer b.mu.Unlock()
|
defer b.mu.Unlock()
|
||||||
|
|
||||||
@@ -212,7 +172,7 @@ func (b *IPLists) setCopy(name string, c listCopy) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
// setUpdated records that the named list's copy was found current at t.
|
// setUpdated records that the named list's copy was found current at t.
|
||||||
func (b *IPLists) setUpdated(name string, t time.Time) {
|
func (b *Blocklists) setUpdated(name string, t time.Time) {
|
||||||
b.mu.Lock()
|
b.mu.Lock()
|
||||||
defer b.mu.Unlock()
|
defer b.mu.Unlock()
|
||||||
|
|
||||||
@@ -0,0 +1,494 @@
|
|||||||
|
package blocklist
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"errors"
|
||||||
|
"io"
|
||||||
|
"io/fs"
|
||||||
|
"log/slog"
|
||||||
|
"net/http"
|
||||||
|
"net/http/httptest"
|
||||||
|
"net/netip"
|
||||||
|
"os"
|
||||||
|
"path/filepath"
|
||||||
|
"strings"
|
||||||
|
"sync"
|
||||||
|
"testing"
|
||||||
|
"time"
|
||||||
|
)
|
||||||
|
|
||||||
|
// The test list's name and description, the address its good version lists,
|
||||||
|
// and the address a newer version lists instead.
|
||||||
|
const (
|
||||||
|
testListName = "test_list"
|
||||||
|
testListDescription = "On the test list"
|
||||||
|
listedAddr = "192.0.2.1"
|
||||||
|
goodListBody = "# A test list\n" + listedAddr + "\n"
|
||||||
|
newListedAddr = "192.0.2.2"
|
||||||
|
newListBody = "# A test list\n" + newListedAddr + "\n"
|
||||||
|
)
|
||||||
|
|
||||||
|
// testETag is the ETag a publisher sends with a version of the test list.
|
||||||
|
const testETag = `"v1"`
|
||||||
|
|
||||||
|
// maxWait is how long a test waits for the download goroutine.
|
||||||
|
const maxWait = 10 * time.Second
|
||||||
|
|
||||||
|
// publisher stands in for a list's publisher. It answers every request with
|
||||||
|
// its status and body, and keeps the headers of each request. A 200 answer
|
||||||
|
// carries lastModified and etag, when set, as Last-Modified and ETag. As
|
||||||
|
// from a real publisher, a request whose If-None-Match or If-Modified-Since
|
||||||
|
// matches them is answered "not modified".
|
||||||
|
type publisher struct {
|
||||||
|
mu sync.Mutex
|
||||||
|
status int
|
||||||
|
body string
|
||||||
|
lastModified time.Time
|
||||||
|
etag string
|
||||||
|
headers []http.Header
|
||||||
|
}
|
||||||
|
|
||||||
|
// newPublisher starts a publisher that answers 200 with body, and returns it
|
||||||
|
// and its URL.
|
||||||
|
func newPublisher(t *testing.T, body string) (*publisher, string) {
|
||||||
|
t.Helper()
|
||||||
|
|
||||||
|
p := &publisher{status: http.StatusOK, body: body}
|
||||||
|
srv := httptest.NewServer(p)
|
||||||
|
t.Cleanup(srv.Close)
|
||||||
|
|
||||||
|
return p, srv.URL
|
||||||
|
}
|
||||||
|
|
||||||
|
func (p *publisher) ServeHTTP(w http.ResponseWriter, r *http.Request) {
|
||||||
|
p.mu.Lock()
|
||||||
|
defer p.mu.Unlock()
|
||||||
|
|
||||||
|
p.headers = append(p.headers, r.Header.Clone())
|
||||||
|
|
||||||
|
if p.status != http.StatusOK {
|
||||||
|
w.WriteHeader(p.status)
|
||||||
|
_, _ = io.WriteString(w, p.body)
|
||||||
|
|
||||||
|
return
|
||||||
|
}
|
||||||
|
|
||||||
|
if p.etag != "" {
|
||||||
|
w.Header().Set(etagHeader, p.etag)
|
||||||
|
}
|
||||||
|
|
||||||
|
// ServeContent sends Last-Modified unless lastModified is zero, and
|
||||||
|
// answers "not modified" when the request's conditions match.
|
||||||
|
http.ServeContent(w, r, testListName, p.lastModified,
|
||||||
|
strings.NewReader(p.body))
|
||||||
|
}
|
||||||
|
|
||||||
|
// answer makes the publisher answer later requests with status and body.
|
||||||
|
func (p *publisher) answer(status int, body string) {
|
||||||
|
p.mu.Lock()
|
||||||
|
defer p.mu.Unlock()
|
||||||
|
|
||||||
|
p.status, p.body = status, body
|
||||||
|
}
|
||||||
|
|
||||||
|
// publish makes the publisher answer later requests with a version of the
|
||||||
|
// list: body, with lastModified and etag. A zero lastModified or an empty
|
||||||
|
// etag is not sent.
|
||||||
|
func (p *publisher) publish(body string, lastModified time.Time, etag string) {
|
||||||
|
p.mu.Lock()
|
||||||
|
defer p.mu.Unlock()
|
||||||
|
|
||||||
|
p.status, p.body = http.StatusOK, body
|
||||||
|
p.lastModified, p.etag = lastModified, etag
|
||||||
|
}
|
||||||
|
|
||||||
|
// requests returns the headers of the requests the publisher has had.
|
||||||
|
func (p *publisher) requests() []http.Header {
|
||||||
|
p.mu.Lock()
|
||||||
|
defer p.mu.Unlock()
|
||||||
|
|
||||||
|
return append([]http.Header(nil), p.headers...)
|
||||||
|
}
|
||||||
|
|
||||||
|
// testList returns a list served at url, written in format.
|
||||||
|
func testList(url string, format Format) List {
|
||||||
|
return List{
|
||||||
|
Name: testListName,
|
||||||
|
Description: testListDescription,
|
||||||
|
URL: url,
|
||||||
|
Interval: time.Hour,
|
||||||
|
Format: format,
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func newTestBlocklists(dir string, list List) *Blocklists {
|
||||||
|
return New(Params{
|
||||||
|
Dir: dir,
|
||||||
|
Lists: []List{list},
|
||||||
|
Logger: slog.New(slog.DiscardHandler),
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
// lookup looks addr up in b.
|
||||||
|
func lookup(t *testing.T, b *Blocklists, addr string) []Listing {
|
||||||
|
t.Helper()
|
||||||
|
|
||||||
|
return b.Lookup(netip.MustParseAddr(addr))
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestLookupFindsListedAddresses downloads a list in each format and looks
|
||||||
|
// up addresses on it, inside its networks, and off it, in IPv4 and IPv6.
|
||||||
|
// A listed address has one listing that names the list; any other has an
|
||||||
|
// empty list of listings.
|
||||||
|
func TestLookupFindsListedAddresses(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
tests := []struct {
|
||||||
|
name string
|
||||||
|
format Format
|
||||||
|
body string
|
||||||
|
listed []string
|
||||||
|
unlisted []string
|
||||||
|
}{
|
||||||
|
{
|
||||||
|
name: "single addresses",
|
||||||
|
format: FormatLines,
|
||||||
|
body: "# comment\n192.0.2.1\n\n2001:db8::1\n",
|
||||||
|
listed: []string{"192.0.2.1", "::ffff:192.0.2.1", "2001:db8::1"},
|
||||||
|
unlisted: []string{"192.0.2.2", "2001:db8::2"},
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "networks",
|
||||||
|
format: FormatLines,
|
||||||
|
body: "# comment\n198.51.100.0/24\n2001:db8:1::/48\n",
|
||||||
|
listed: []string{"198.51.100.0", "198.51.100.255", "2001:db8:1::5"},
|
||||||
|
unlisted: []string{"198.51.101.0", "2001:db8:2::1"},
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "Spamhaus networks",
|
||||||
|
format: FormatSpamhausJSON,
|
||||||
|
body: `{"cidr":"203.0.113.0/25","sblid":"SBL1","rir":"arin"}` + "\n" +
|
||||||
|
`{"cidr":"2001:db8:3::/48","sblid":"SBL2","rir":"ripencc"}` + "\n" +
|
||||||
|
`{"type":"metadata","timestamp":1793074442,"records":2}` + "\n",
|
||||||
|
listed: []string{"203.0.113.1", "2001:db8:3::1"},
|
||||||
|
unlisted: []string{"203.0.113.200", "2001:db8:4::1"},
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "DShield networks",
|
||||||
|
format: FormatDShield,
|
||||||
|
body: "# DShield.org Recommended Block List\n" +
|
||||||
|
"192.0.2.0\t192.0.2.255\t24\t327\tEXAMPLE\tUS\tNone\n",
|
||||||
|
listed: []string{"192.0.2.0", "192.0.2.77"},
|
||||||
|
unlisted: []string{"192.0.3.1", "2001:db8::1"},
|
||||||
|
},
|
||||||
|
}
|
||||||
|
|
||||||
|
for _, tt := range tests {
|
||||||
|
t.Run(tt.name, func(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
_, url := newPublisher(t, tt.body)
|
||||||
|
b := newTestBlocklists(t.TempDir(), testList(url, tt.format))
|
||||||
|
|
||||||
|
err := b.download(t.Context(), b.lists[0])
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("download: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
for _, addr := range tt.listed {
|
||||||
|
listings := lookup(t, b, addr)
|
||||||
|
if len(listings) != 1 || listings[0].Name != testListName ||
|
||||||
|
listings[0].Description != testListDescription ||
|
||||||
|
listings[0].Updated.IsZero() {
|
||||||
|
t.Errorf("%s: listings %+v, want one for %s", addr,
|
||||||
|
listings, testListName)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
for _, addr := range tt.unlisted {
|
||||||
|
listings := lookup(t, b, addr)
|
||||||
|
if listings == nil || len(listings) != 0 {
|
||||||
|
t.Errorf("%s: listings %#v, want an empty list", addr, listings)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestFailedDownloadKeepsThePreviousCopy downloads a good version of a list,
|
||||||
|
// then one that fails. The good copy stays in memory and on disk, and no
|
||||||
|
// part of the failed download is left beside it.
|
||||||
|
func TestFailedDownloadKeepsThePreviousCopy(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
tests := []struct {
|
||||||
|
name string
|
||||||
|
status int
|
||||||
|
body string
|
||||||
|
}{
|
||||||
|
{"server error", http.StatusInternalServerError, "unavailable"},
|
||||||
|
{"not a list", http.StatusOK, "<!DOCTYPE html>\n<html></html>\n"},
|
||||||
|
{
|
||||||
|
"over the size limit", http.StatusOK,
|
||||||
|
strings.Repeat(listedAddr+"\n", maxDownloadSize/len(listedAddr)),
|
||||||
|
},
|
||||||
|
}
|
||||||
|
|
||||||
|
for _, tt := range tests {
|
||||||
|
t.Run(tt.name, func(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
dir := t.TempDir()
|
||||||
|
p, url := newPublisher(t, goodListBody)
|
||||||
|
b := newTestBlocklists(dir, testList(url, FormatLines))
|
||||||
|
|
||||||
|
err := b.download(t.Context(), b.lists[0])
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("first download: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
before := lookup(t, b, listedAddr)
|
||||||
|
|
||||||
|
p.answer(tt.status, tt.body)
|
||||||
|
|
||||||
|
err = b.download(t.Context(), b.lists[0])
|
||||||
|
if err == nil {
|
||||||
|
t.Fatal("second download returned no error")
|
||||||
|
}
|
||||||
|
|
||||||
|
after := lookup(t, b, listedAddr)
|
||||||
|
if len(after) != 1 || len(before) != 1 ||
|
||||||
|
!after[0].Updated.Equal(before[0].Updated) {
|
||||||
|
t.Errorf("listings %+v after the failure, want %+v", after, before)
|
||||||
|
}
|
||||||
|
|
||||||
|
copies := os.DirFS(dir)
|
||||||
|
|
||||||
|
onDisk, err := fs.ReadFile(copies, testListName)
|
||||||
|
if err != nil || string(onDisk) != goodListBody {
|
||||||
|
t.Errorf("copy on disk %q (%v), want %q", onDisk, err, goodListBody)
|
||||||
|
}
|
||||||
|
|
||||||
|
_, err = fs.Stat(copies, testListName+downloadSuffix)
|
||||||
|
if !errors.Is(err, fs.ErrNotExist) {
|
||||||
|
t.Errorf("the failed download's file is still there (%v)", err)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestRestartLoadsCopiesFromDisk downloads a list, then starts a new
|
||||||
|
// Blocklists on the same directory, as a restart does. The new one answers
|
||||||
|
// from the copy on disk at once, and does not download the list again while
|
||||||
|
// the copy is younger than the list's interval.
|
||||||
|
func TestRestartLoadsCopiesFromDisk(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
dir := t.TempDir()
|
||||||
|
p, url := newPublisher(t, goodListBody)
|
||||||
|
list := testList(url, FormatLines)
|
||||||
|
|
||||||
|
first := newTestBlocklists(dir, list)
|
||||||
|
|
||||||
|
err := first.download(t.Context(), list)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("download: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
downloaded := lookup(t, first, listedAddr)[0].Updated
|
||||||
|
|
||||||
|
second := newTestBlocklists(dir, list)
|
||||||
|
second.Start(t.Context())
|
||||||
|
second.Stop(t.Context())
|
||||||
|
|
||||||
|
listings := lookup(t, second, listedAddr)
|
||||||
|
if len(listings) != 1 ||
|
||||||
|
listings[0].Updated.Sub(downloaded).Abs() > time.Second {
|
||||||
|
t.Errorf("after the restart, listings %+v, want one updated at %s",
|
||||||
|
listings, downloaded)
|
||||||
|
}
|
||||||
|
|
||||||
|
if got := len(p.requests()); got != 1 {
|
||||||
|
t.Errorf("the publisher had %d requests, want 1", got)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestStartRefreshesAnOldCopy starts on a copy older than its list's
|
||||||
|
// interval. The list is downloaded at once, with routewatch's User-Agent and
|
||||||
|
// the Last-Modified and ETag the publisher sent with the copy, kept on disk.
|
||||||
|
// The publisher answers that the copy is still current, so the copy is kept
|
||||||
|
// and marked as updated now.
|
||||||
|
func TestStartRefreshesAnOldCopy(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
dir := t.TempDir()
|
||||||
|
p, url := newPublisher(t, goodListBody)
|
||||||
|
list := testList(url, FormatLines)
|
||||||
|
|
||||||
|
lastModified := time.Now().Add(-3 * list.Interval).UTC().Truncate(time.Second)
|
||||||
|
p.publish(goodListBody, lastModified, testETag)
|
||||||
|
|
||||||
|
err := newTestBlocklists(dir, list).download(t.Context(), list)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("download: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
old := time.Now().Add(-2 * list.Interval).UTC().Truncate(time.Second)
|
||||||
|
|
||||||
|
err = os.Chtimes(filepath.Join(dir, testListName), old, old)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("Chtimes: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
b := newTestBlocklists(dir, list)
|
||||||
|
b.Start(t.Context())
|
||||||
|
|
||||||
|
deadline := time.Now().Add(maxWait)
|
||||||
|
for !lookup(t, b, listedAddr)[0].Updated.After(old) {
|
||||||
|
if time.Now().After(deadline) {
|
||||||
|
t.Fatalf("the copy was not marked updated within %s", maxWait)
|
||||||
|
}
|
||||||
|
|
||||||
|
time.Sleep(time.Millisecond)
|
||||||
|
}
|
||||||
|
|
||||||
|
b.Stop(t.Context())
|
||||||
|
|
||||||
|
requests := p.requests()
|
||||||
|
last := requests[len(requests)-1]
|
||||||
|
|
||||||
|
want := lastModified.Format(http.TimeFormat)
|
||||||
|
if got := last.Get(ifModifiedSinceHeader); got != want {
|
||||||
|
t.Errorf("If-Modified-Since %q, want %q", got, want)
|
||||||
|
}
|
||||||
|
|
||||||
|
if got := last.Get(ifNoneMatchHeader); got != testETag {
|
||||||
|
t.Errorf("If-None-Match %q, want %q", got, testETag)
|
||||||
|
}
|
||||||
|
|
||||||
|
if got := last.Get(userAgentHeader); got != userAgent {
|
||||||
|
t.Errorf("User-Agent %q, want %q", got, userAgent)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestNewVersionOlderThanTheLastDownloadReplacesTheCopy downloads a list.
|
||||||
|
// The publisher then puts out a new version whose Last-Modified is earlier
|
||||||
|
// than that download, as when a cache in front of the publisher passes the
|
||||||
|
// version on late. The next download sends the Last-Modified of the copy,
|
||||||
|
// not the time of the download, so it gets the new version.
|
||||||
|
func TestNewVersionOlderThanTheLastDownloadReplacesTheCopy(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
p, url := newPublisher(t, goodListBody)
|
||||||
|
list := testList(url, FormatLines)
|
||||||
|
b := newTestBlocklists(t.TempDir(), list)
|
||||||
|
|
||||||
|
firstVersion := time.Now().Add(-2 * list.Interval).UTC().Truncate(time.Second)
|
||||||
|
p.publish(goodListBody, firstVersion, "")
|
||||||
|
|
||||||
|
err := b.download(t.Context(), list)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("first download: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
p.publish(newListBody, firstVersion.Add(list.Interval), "")
|
||||||
|
|
||||||
|
err = b.download(t.Context(), list)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("second download: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
if listings := lookup(t, b, newListedAddr); len(listings) != 1 {
|
||||||
|
t.Errorf("%s: listings %+v, want one from the new version",
|
||||||
|
newListedAddr, listings)
|
||||||
|
}
|
||||||
|
|
||||||
|
want := firstVersion.Format(http.TimeFormat)
|
||||||
|
if got := p.requests()[1].Get(ifModifiedSinceHeader); got != want {
|
||||||
|
t.Errorf("If-Modified-Since %q, want %q", got, want)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestDownloadWithoutLastModifiedOrETagSendsNoCondition downloads a list
|
||||||
|
// twice from a publisher that sends neither Last-Modified nor ETag. Neither
|
||||||
|
// request carries If-Modified-Since or If-None-Match, so the second download
|
||||||
|
// gets the whole list again.
|
||||||
|
func TestDownloadWithoutLastModifiedOrETagSendsNoCondition(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
p, url := newPublisher(t, goodListBody)
|
||||||
|
list := testList(url, FormatLines)
|
||||||
|
b := newTestBlocklists(t.TempDir(), list)
|
||||||
|
|
||||||
|
err := b.download(t.Context(), list)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("first download: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
err = b.download(t.Context(), list)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("second download: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
for i, request := range p.requests() {
|
||||||
|
if request.Get(ifModifiedSinceHeader) != "" ||
|
||||||
|
request.Get(ifNoneMatchHeader) != "" {
|
||||||
|
t.Errorf("request %d has a condition: %v", i, request)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestStopCancelsADownloadInProgress stops while the publisher has not
|
||||||
|
// answered. Stop cancels the download and returns once the download
|
||||||
|
// goroutine has stopped.
|
||||||
|
func TestStopCancelsADownloadInProgress(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
arrived := make(chan struct{})
|
||||||
|
|
||||||
|
var once sync.Once
|
||||||
|
|
||||||
|
srv := httptest.NewServer(http.HandlerFunc(
|
||||||
|
func(_ http.ResponseWriter, r *http.Request) {
|
||||||
|
once.Do(func() { close(arrived) })
|
||||||
|
<-r.Context().Done()
|
||||||
|
}))
|
||||||
|
t.Cleanup(srv.Close)
|
||||||
|
|
||||||
|
b := newTestBlocklists(t.TempDir(), testList(srv.URL, FormatLines))
|
||||||
|
b.Start(t.Context())
|
||||||
|
|
||||||
|
select {
|
||||||
|
case <-arrived:
|
||||||
|
case <-time.After(maxWait):
|
||||||
|
t.Fatalf("no download began within %s", maxWait)
|
||||||
|
}
|
||||||
|
|
||||||
|
ctx, cancel := context.WithTimeout(t.Context(), maxWait)
|
||||||
|
defer cancel()
|
||||||
|
|
||||||
|
b.Stop(ctx)
|
||||||
|
|
||||||
|
select {
|
||||||
|
case <-b.done:
|
||||||
|
default:
|
||||||
|
t.Error("Stop returned before the download goroutine stopped")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestPublicListsAreComplete checks that every public list has what a
|
||||||
|
// download and a lookup answer need, and a unique name.
|
||||||
|
func TestPublicListsAreComplete(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
names := make(map[string]bool)
|
||||||
|
|
||||||
|
for _, list := range Lists() {
|
||||||
|
if list.Name == "" || list.Description == "" || list.URL == "" ||
|
||||||
|
list.Interval <= 0 || names[list.Name] {
|
||||||
|
t.Errorf("list %+v is incomplete or has a name used before", list)
|
||||||
|
}
|
||||||
|
|
||||||
|
names[list.Name] = true
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -1,4 +1,4 @@
|
|||||||
package iplist
|
package blocklist
|
||||||
|
|
||||||
import (
|
import (
|
||||||
"context"
|
"context"
|
||||||
@@ -21,8 +21,8 @@ const (
|
|||||||
downloadTimeout = time.Minute
|
downloadTimeout = time.Minute
|
||||||
|
|
||||||
// maxDownloadSize is the size of the largest list file accepted, in
|
// maxDownloadSize is the size of the largest list file accepted, in
|
||||||
// bytes. The largest list, iCloud Private Relay, is about 12 MiB.
|
// bytes. The largest list, FireHOL abusers, is about 2.5 MiB.
|
||||||
maxDownloadSize = 200 << 20
|
maxDownloadSize = 16 << 20
|
||||||
)
|
)
|
||||||
|
|
||||||
// The request headers a download sets.
|
// The request headers a download sets.
|
||||||
@@ -78,10 +78,10 @@ var (
|
|||||||
// downloads each list once its copy is older than its Interval, and a list
|
// downloads each list once its copy is older than its Interval, and a list
|
||||||
// with no copy at once. The goroutine runs until Stop: the end of ctx does
|
// with no copy at once. The goroutine runs until Stop: the end of ctx does
|
||||||
// not stop it.
|
// not stop it.
|
||||||
func (b *IPLists) Start(ctx context.Context) {
|
func (b *Blocklists) Start(ctx context.Context) {
|
||||||
err := os.MkdirAll(b.dir, dirPermissions)
|
err := os.MkdirAll(b.dir, dirPermissions)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
b.logger.Error("Failed to create the IP lists directory",
|
b.logger.Error("Failed to create the blocklists directory",
|
||||||
"dir", b.dir, "error", err)
|
"dir", b.dir, "error", err)
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -90,7 +90,7 @@ func (b *IPLists) Start(ctx context.Context) {
|
|||||||
for i, list := range b.lists {
|
for i, list := range b.lists {
|
||||||
loadErr := b.loadCopy(list)
|
loadErr := b.loadCopy(list)
|
||||||
if loadErr != nil && !errors.Is(loadErr, fs.ErrNotExist) {
|
if loadErr != nil && !errors.Is(loadErr, fs.ErrNotExist) {
|
||||||
b.logger.Error("Failed to load an IP list copy",
|
b.logger.Error("Failed to load a blocklist copy",
|
||||||
"list", list.Name, "error", loadErr)
|
"list", list.Name, "error", loadErr)
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -104,20 +104,20 @@ func (b *IPLists) Start(ctx context.Context) {
|
|||||||
b.run(ctx, nextDownload)
|
b.run(ctx, nextDownload)
|
||||||
}()
|
}()
|
||||||
|
|
||||||
b.logger.Info("IP list downloads started", "dir", b.dir)
|
b.logger.Info("Blocklist downloads started", "dir", b.dir)
|
||||||
}
|
}
|
||||||
|
|
||||||
// Stop stops the downloads, cancelling one in progress, and waits until the
|
// Stop stops the downloads, cancelling one in progress, and waits until the
|
||||||
// goroutine has returned or ctx ends. The copies in memory stay, so Lookup
|
// goroutine has returned or ctx ends. The copies in memory stay, so Lookup
|
||||||
// still answers.
|
// still answers.
|
||||||
func (b *IPLists) Stop(ctx context.Context) {
|
func (b *Blocklists) Stop(ctx context.Context) {
|
||||||
b.stop()
|
b.stop()
|
||||||
|
|
||||||
select {
|
select {
|
||||||
case <-b.done:
|
case <-b.done:
|
||||||
b.logger.Info("IP list downloads stopped")
|
b.logger.Info("Blocklist downloads stopped")
|
||||||
case <-ctx.Done():
|
case <-ctx.Done():
|
||||||
b.logger.Warn("Stop deadline passed before the IP list downloads " +
|
b.logger.Warn("Stop deadline passed before the blocklist downloads " +
|
||||||
"stopped")
|
"stopped")
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -125,7 +125,7 @@ func (b *IPLists) Stop(ctx context.Context) {
|
|||||||
// run downloads each list when nextDownload says, until ctx ends. After each
|
// run downloads each list when nextDownload says, until ctx ends. After each
|
||||||
// try, whether it worked or not, the list's next download is an Interval
|
// try, whether it worked or not, the list's next download is an Interval
|
||||||
// later.
|
// later.
|
||||||
func (b *IPLists) run(ctx context.Context, nextDownload []time.Time) {
|
func (b *Blocklists) run(ctx context.Context, nextDownload []time.Time) {
|
||||||
defer close(b.done)
|
defer close(b.done)
|
||||||
|
|
||||||
if len(nextDownload) == 0 {
|
if len(nextDownload) == 0 {
|
||||||
@@ -155,7 +155,7 @@ func (b *IPLists) run(ctx context.Context, nextDownload []time.Time) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
if err != nil {
|
if err != nil {
|
||||||
b.logger.Error("IP list download failed; keeping the "+
|
b.logger.Error("Blocklist download failed; keeping the "+
|
||||||
"previous copy", "list", list.Name, "error", err)
|
"previous copy", "list", list.Name, "error", err)
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -169,7 +169,7 @@ func (b *IPLists) run(ctx context.Context, nextDownload []time.Time) {
|
|||||||
// on disk and in memory. An answer that the copy is still current marks the
|
// on disk and in memory. An answer that the copy is still current marks the
|
||||||
// copy as updated now. Anything else leaves the copy as it is and returns an
|
// copy as updated now. Anything else leaves the copy as it is and returns an
|
||||||
// error.
|
// error.
|
||||||
func (b *IPLists) download(ctx context.Context, list List) error {
|
func (b *Blocklists) download(ctx context.Context, list List) error {
|
||||||
ctx, cancel := context.WithTimeout(ctx, downloadTimeout)
|
ctx, cancel := context.WithTimeout(ctx, downloadTimeout)
|
||||||
defer cancel()
|
defer cancel()
|
||||||
|
|
||||||
@@ -212,7 +212,7 @@ func (b *IPLists) download(ctx context.Context, list List) error {
|
|||||||
// replaceCopy writes a new version of list, read from body, beside the list's
|
// replaceCopy writes a new version of list, read from body, beside the list's
|
||||||
// copy. Only when it parses does it replace the copy, on disk and in memory.
|
// copy. Only when it parses does it replace the copy, on disk and in memory.
|
||||||
// sent are the validators the publisher sent with the new version.
|
// sent are the validators the publisher sent with the new version.
|
||||||
func (b *IPLists) replaceCopy(list List, body io.Reader, sent validators) error {
|
func (b *Blocklists) replaceCopy(list List, body io.Reader, sent validators) error {
|
||||||
root, err := os.OpenRoot(b.dir)
|
root, err := os.OpenRoot(b.dir)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return err
|
return err
|
||||||
@@ -244,13 +244,13 @@ func (b *IPLists) replaceCopy(list List, body io.Reader, sent validators) error
|
|||||||
b.setCopy(list.Name, listCopy{
|
b.setCopy(list.Name, listCopy{
|
||||||
addresses: addresses, updated: time.Now().UTC(), validators: sent,
|
addresses: addresses, updated: time.Now().UTC(), validators: sent,
|
||||||
})
|
})
|
||||||
b.logger.Info("Downloaded a new version of an IP list", "list", list.Name)
|
b.logger.Info("Downloaded a new version of a blocklist", "list", list.Name)
|
||||||
|
|
||||||
// Without this file, the first download after a restart asks for the
|
// Without this file, the first download after a restart asks for the
|
||||||
// whole list.
|
// whole list.
|
||||||
err = writeValidators(root, list.Name, sent)
|
err = writeValidators(root, list.Name, sent)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
b.logger.Error("Failed to keep the validators of an IP list copy",
|
b.logger.Error("Failed to keep the validators of a blocklist copy",
|
||||||
"list", list.Name, "error", err)
|
"list", list.Name, "error", err)
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -288,7 +288,7 @@ func writeAndParse(
|
|||||||
|
|
||||||
// markCurrent records that the publisher says the list's copy is still
|
// markCurrent records that the publisher says the list's copy is still
|
||||||
// current: the copy, on disk and in memory, is marked as updated now.
|
// current: the copy, on disk and in memory, is marked as updated now.
|
||||||
func (b *IPLists) markCurrent(list List) error {
|
func (b *Blocklists) markCurrent(list List) error {
|
||||||
root, err := os.OpenRoot(b.dir)
|
root, err := os.OpenRoot(b.dir)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return err
|
return err
|
||||||
@@ -304,7 +304,7 @@ func (b *IPLists) markCurrent(list List) error {
|
|||||||
}
|
}
|
||||||
|
|
||||||
b.setUpdated(list.Name, now)
|
b.setUpdated(list.Name, now)
|
||||||
b.logger.Debug("IP list copy is still current", "list", list.Name)
|
b.logger.Debug("Blocklist copy is still current", "list", list.Name)
|
||||||
|
|
||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
@@ -312,7 +312,7 @@ func (b *IPLists) markCurrent(list List) error {
|
|||||||
// loadCopy makes the list's copy on disk, with the validators kept beside it,
|
// loadCopy makes the list's copy on disk, with the validators kept beside it,
|
||||||
// its copy in memory. The file's modification time is when the copy was
|
// its copy in memory. The file's modification time is when the copy was
|
||||||
// updated.
|
// updated.
|
||||||
func (b *IPLists) loadCopy(list List) error {
|
func (b *Blocklists) loadCopy(list List) error {
|
||||||
root, err := os.OpenRoot(b.dir)
|
root, err := os.OpenRoot(b.dir)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return err
|
return err
|
||||||
@@ -0,0 +1,93 @@
|
|||||||
|
package blocklist
|
||||||
|
|
||||||
|
import "time"
|
||||||
|
|
||||||
|
// How often each list is downloaded. No list is downloaded more often than
|
||||||
|
// its publisher updates it or allows.
|
||||||
|
const (
|
||||||
|
// DShield asks that its feeds be downloaded at most once an hour:
|
||||||
|
// https://isc.sans.edu/feeds_doc.html
|
||||||
|
dshieldInterval = time.Hour
|
||||||
|
|
||||||
|
// abuse.ch makes the list every 5 minutes and recommends a download at
|
||||||
|
// least every 15: https://feodotracker.abuse.ch/blocklist/
|
||||||
|
feodoTrackerInterval = 15 * time.Minute
|
||||||
|
|
||||||
|
// FireHOL's page for each list gives how often the list changes on
|
||||||
|
// average: about every 5 hours for abusers and every 3 hours for level
|
||||||
|
// 3. FireHOL sets no download limit.
|
||||||
|
// https://iplists.firehol.org/?ipset=firehol_abusers_30d
|
||||||
|
// https://iplists.firehol.org/?ipset=firehol_level3
|
||||||
|
fireHOLAbusersInterval = 5 * time.Hour
|
||||||
|
fireHOLLevel3Interval = 3 * time.Hour
|
||||||
|
|
||||||
|
// Spamhaus asks that DROP be downloaded at most once a day:
|
||||||
|
// https://www.spamhaus.org/blocklists/do-not-route-or-peer/
|
||||||
|
spamhausDROPInterval = 24 * time.Hour
|
||||||
|
|
||||||
|
// Tor publishes a new exit list every hour and sets no download limit:
|
||||||
|
// https://collector.torproject.org/recent/exit-lists/
|
||||||
|
torExitsInterval = time.Hour
|
||||||
|
)
|
||||||
|
|
||||||
|
// Lists returns the public blocklists routewatch downloads. Each can be
|
||||||
|
// downloaded whole without an account or a key. Spamhaus merged its EDROP
|
||||||
|
// list into DROP, so DROP covers both.
|
||||||
|
func Lists() []List {
|
||||||
|
return []List{
|
||||||
|
{
|
||||||
|
Name: "dshield",
|
||||||
|
Description: "In one of the 20 networks that DShield saw scan " +
|
||||||
|
"the most targets in the last three days",
|
||||||
|
URL: "https://feeds.dshield.org/block.txt",
|
||||||
|
Interval: dshieldInterval,
|
||||||
|
Format: FormatDShield,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
Name: "feodo_tracker",
|
||||||
|
Description: "A botnet command-and-control server",
|
||||||
|
URL: "https://feodotracker.abuse.ch/downloads/ipblocklist.txt",
|
||||||
|
Interval: feodoTrackerInterval,
|
||||||
|
Format: FormatLines,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
Name: "firehol_abusers_30d",
|
||||||
|
Description: "Reported for comment, form or forum spam in the " +
|
||||||
|
"last 30 days",
|
||||||
|
URL: "https://iplists.firehol.org/files/firehol_abusers_30d.netset",
|
||||||
|
Interval: fireHOLAbusersInterval,
|
||||||
|
Format: FormatLines,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
Name: "firehol_level3",
|
||||||
|
Description: "Reported for attacks, spyware or viruses in the " +
|
||||||
|
"last 30 days",
|
||||||
|
URL: "https://iplists.firehol.org/files/firehol_level3.netset",
|
||||||
|
Interval: fireHOLLevel3Interval,
|
||||||
|
Format: FormatLines,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
Name: "spamhaus_drop",
|
||||||
|
Description: "In an IPv4 network that Spamhaus says spammers " +
|
||||||
|
"or cyber criminals have hijacked or lease",
|
||||||
|
URL: "https://www.spamhaus.org/drop/drop_v4.json",
|
||||||
|
Interval: spamhausDROPInterval,
|
||||||
|
Format: FormatSpamhausJSON,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
Name: "spamhaus_dropv6",
|
||||||
|
Description: "In an IPv6 network that Spamhaus says spammers " +
|
||||||
|
"or cyber criminals have hijacked or lease",
|
||||||
|
URL: "https://www.spamhaus.org/drop/drop_v6.json",
|
||||||
|
Interval: spamhausDROPInterval,
|
||||||
|
Format: FormatSpamhausJSON,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
Name: "tor_exits",
|
||||||
|
Description: "A Tor exit node, where traffic from the Tor network leaves",
|
||||||
|
URL: "https://check.torproject.org/torbulkexitlist",
|
||||||
|
Interval: torExitsInterval,
|
||||||
|
Format: FormatLines,
|
||||||
|
},
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,158 @@
|
|||||||
|
package blocklist
|
||||||
|
|
||||||
|
import (
|
||||||
|
"bufio"
|
||||||
|
"encoding/json"
|
||||||
|
"errors"
|
||||||
|
"fmt"
|
||||||
|
"io"
|
||||||
|
"net/netip"
|
||||||
|
"strings"
|
||||||
|
|
||||||
|
"go4.org/netipx"
|
||||||
|
)
|
||||||
|
|
||||||
|
const (
|
||||||
|
// commentPrefix starts a comment line in a list file.
|
||||||
|
commentPrefix = "#"
|
||||||
|
|
||||||
|
// networkSeparator separates the address and the mask length of a
|
||||||
|
// network, as in 192.0.2.0/24.
|
||||||
|
networkSeparator = "/"
|
||||||
|
|
||||||
|
// dshieldFieldSeparator separates the fields of a DShield line.
|
||||||
|
dshieldFieldSeparator = "\t"
|
||||||
|
|
||||||
|
// spamhausMetadataType is the type of the line that ends a Spamhaus
|
||||||
|
// file. That line holds the file's metadata and no network.
|
||||||
|
spamhausMetadataType = "metadata"
|
||||||
|
)
|
||||||
|
|
||||||
|
var (
|
||||||
|
errUnknownFormat = errors.New("unknown list format")
|
||||||
|
errNoNetwork = errors.New("no network on the line")
|
||||||
|
errBadRange = errors.New("first and last address make no range")
|
||||||
|
)
|
||||||
|
|
||||||
|
// parse reads a list file written in format and returns the addresses it
|
||||||
|
// holds. A line it cannot read fails the whole file, so a file that is not
|
||||||
|
// the list, such as an error page, never replaces a good copy.
|
||||||
|
func parse(r io.Reader, format Format) (*netipx.IPSet, error) {
|
||||||
|
var builder netipx.IPSetBuilder
|
||||||
|
|
||||||
|
scanner := bufio.NewScanner(r)
|
||||||
|
|
||||||
|
for lineNumber := 1; scanner.Scan(); lineNumber++ {
|
||||||
|
line := strings.TrimSpace(scanner.Text())
|
||||||
|
if line == "" || strings.HasPrefix(line, commentPrefix) {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
|
||||||
|
err := addLine(&builder, line, format)
|
||||||
|
if err != nil {
|
||||||
|
return nil, fmt.Errorf("line %d: %w", lineNumber, err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
err := scanner.Err()
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
|
||||||
|
return builder.IPSet()
|
||||||
|
}
|
||||||
|
|
||||||
|
// addLine adds the address or network on one line of a list file to builder.
|
||||||
|
func addLine(builder *netipx.IPSetBuilder, line string, format Format) error {
|
||||||
|
switch format {
|
||||||
|
case FormatLines:
|
||||||
|
return addAddressOrNetwork(builder, line)
|
||||||
|
case FormatSpamhausJSON:
|
||||||
|
return addSpamhausNetwork(builder, line)
|
||||||
|
case FormatDShield:
|
||||||
|
return addDShieldNetwork(builder, line)
|
||||||
|
default:
|
||||||
|
return errUnknownFormat
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// addAddressOrNetwork adds an address, such as 192.0.2.1, or a network, such
|
||||||
|
// as 192.0.2.0/24, to builder.
|
||||||
|
func addAddressOrNetwork(builder *netipx.IPSetBuilder, text string) error {
|
||||||
|
if strings.Contains(text, networkSeparator) {
|
||||||
|
network, err := netip.ParsePrefix(text)
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
|
||||||
|
builder.AddPrefix(network)
|
||||||
|
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
addr, err := netip.ParseAddr(text)
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
|
||||||
|
builder.Add(addr)
|
||||||
|
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// addSpamhausNetwork adds the network on a line of a Spamhaus file to
|
||||||
|
// builder. The metadata line adds nothing.
|
||||||
|
func addSpamhausNetwork(builder *netipx.IPSetBuilder, line string) error {
|
||||||
|
var entry struct {
|
||||||
|
CIDR string `json:"cidr"`
|
||||||
|
Type string `json:"type"`
|
||||||
|
}
|
||||||
|
|
||||||
|
err := json.Unmarshal([]byte(line), &entry)
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
|
||||||
|
if entry.Type == spamhausMetadataType {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
network, err := netip.ParsePrefix(entry.CIDR)
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
|
||||||
|
builder.AddPrefix(network)
|
||||||
|
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// addDShieldNetwork adds the network on a line of a DShield file to builder.
|
||||||
|
// The line's first two fields are the network's first and last address.
|
||||||
|
func addDShieldNetwork(builder *netipx.IPSetBuilder, line string) error {
|
||||||
|
firstField, rest, found := strings.Cut(line, dshieldFieldSeparator)
|
||||||
|
if !found {
|
||||||
|
return errNoNetwork
|
||||||
|
}
|
||||||
|
|
||||||
|
lastField, _, _ := strings.Cut(rest, dshieldFieldSeparator)
|
||||||
|
|
||||||
|
first, err := netip.ParseAddr(firstField)
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
|
||||||
|
last, err := netip.ParseAddr(lastField)
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
|
||||||
|
network := netipx.IPRangeFrom(first, last)
|
||||||
|
if !network.IsValid() {
|
||||||
|
return errBadRange
|
||||||
|
}
|
||||||
|
|
||||||
|
builder.AddRange(network)
|
||||||
|
|
||||||
|
return nil
|
||||||
|
}
|
||||||
@@ -1,211 +0,0 @@
|
|||||||
// Package bogon tells whether a prefix, an address or an AS number is reserved:
|
|
||||||
// set aside for a special purpose, so that no network should announce it on the
|
|
||||||
// internet. A route is a bogon announcement when its prefix is reserved or its
|
|
||||||
// origin AS is.
|
|
||||||
//
|
|
||||||
// Most reserved prefixes come from the IANA special-purpose address registries
|
|
||||||
// for IPv4 and IPv6. The two multicast blocks, 224.0.0.0/4 and ff00::/8, come
|
|
||||||
// from the IANA IPv4 and IPv6 address space registries. The reserved AS
|
|
||||||
// numbers come from the IANA special-purpose AS numbers registry:
|
|
||||||
// - the AS numbers for private use
|
|
||||||
// - the AS numbers for documentation
|
|
||||||
// - AS 23456, which stands in for a 32-bit AS number
|
|
||||||
// - the last AS number of the 16-bit and of the 32-bit range
|
|
||||||
//
|
|
||||||
// Address space and AS numbers that the IANA has not handed out yet are not
|
|
||||||
// counted.
|
|
||||||
package bogon
|
|
||||||
|
|
||||||
import (
|
|
||||||
"net/netip"
|
|
||||||
"strconv"
|
|
||||||
)
|
|
||||||
|
|
||||||
// Prefix returns why prefix is reserved, and true, when it lies inside a
|
|
||||||
// reserved block. The most specific registry block that holds prefix decides,
|
|
||||||
// because some blocks that may be announced lie inside reserved ones. It
|
|
||||||
// returns "" and false when prefix is not reserved.
|
|
||||||
func Prefix(prefix netip.Prefix) (string, bool) {
|
|
||||||
var holder *block
|
|
||||||
|
|
||||||
for i, b := range registryBlocks {
|
|
||||||
inside := b.prefix.Bits() <= prefix.Bits() && b.prefix.Contains(prefix.Addr())
|
|
||||||
if inside && (holder == nil || b.prefix.Bits() > holder.prefix.Bits()) {
|
|
||||||
holder = ®istryBlocks[i]
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
if holder == nil || !holder.reserved {
|
|
||||||
return "", false
|
|
||||||
}
|
|
||||||
|
|
||||||
return holder.name + " (" + holder.prefix.String() + ")", true
|
|
||||||
}
|
|
||||||
|
|
||||||
// Addr returns why addr is reserved, and true, when it lies inside a reserved
|
|
||||||
// block, as Prefix decides. An IPv4 address written in IPv6 form counts as the
|
|
||||||
// IPv4 address it holds. It returns "" and false when addr is not reserved.
|
|
||||||
func Addr(addr netip.Addr) (string, bool) {
|
|
||||||
addr = addr.Unmap()
|
|
||||||
|
|
||||||
return Prefix(netip.PrefixFrom(addr, addr.BitLen()))
|
|
||||||
}
|
|
||||||
|
|
||||||
// ASN returns why asn is reserved, and true, when it is. It returns "" and
|
|
||||||
// false when asn is not reserved.
|
|
||||||
func ASN(asn int) (string, bool) {
|
|
||||||
for _, r := range reservedASRanges {
|
|
||||||
if asn < r.first || asn > r.last {
|
|
||||||
continue
|
|
||||||
}
|
|
||||||
|
|
||||||
if r.first == r.last {
|
|
||||||
return r.name + " (AS" + strconv.Itoa(asn) + ")", true
|
|
||||||
}
|
|
||||||
|
|
||||||
return r.name + " (AS" + strconv.Itoa(r.first) + "-AS" +
|
|
||||||
strconv.Itoa(r.last) + ")", true
|
|
||||||
}
|
|
||||||
|
|
||||||
return "", false
|
|
||||||
}
|
|
||||||
|
|
||||||
// The names that the registries give to more than one row.
|
|
||||||
const (
|
|
||||||
privateUse = "Private-Use"
|
|
||||||
documentation = "Documentation"
|
|
||||||
benchmarking = "Benchmarking"
|
|
||||||
ietfProtocolAssignments = "IETF Protocol Assignments"
|
|
||||||
portControlAnycast = "Port Control Protocol Anycast"
|
|
||||||
natTraversalAnycast = "Traversal Using Relays around NAT Anycast"
|
|
||||||
nat64Discovery = "NAT64/DNS64 Discovery"
|
|
||||||
amt = "AMT"
|
|
||||||
as112DirectDelegation = "Direct Delegation AS112 Service"
|
|
||||||
ipv4IPv6Translation = "IPv4-IPv6 Translat."
|
|
||||||
multicast = "Multicast"
|
|
||||||
)
|
|
||||||
|
|
||||||
// block is one row of an IANA address registry.
|
|
||||||
type block struct {
|
|
||||||
prefix netip.Prefix
|
|
||||||
name string
|
|
||||||
// reserved says that no network should announce the row's addresses on
|
|
||||||
// the internet.
|
|
||||||
reserved bool
|
|
||||||
}
|
|
||||||
|
|
||||||
// reservedBlock returns a reserved row: a multicast block, or a special-purpose
|
|
||||||
// row whose Globally Reachable column is False.
|
|
||||||
func reservedBlock(prefix, name string) block {
|
|
||||||
return block{prefix: netip.MustParsePrefix(prefix), name: name, reserved: true}
|
|
||||||
}
|
|
||||||
|
|
||||||
// reachableBlock returns a special-purpose row whose Globally Reachable column
|
|
||||||
// is True or N/A. Such a row matters only inside a reserved block, whose
|
|
||||||
// addresses it lets be announced.
|
|
||||||
func reachableBlock(prefix, name string) block {
|
|
||||||
return block{prefix: netip.MustParsePrefix(prefix), name: name}
|
|
||||||
}
|
|
||||||
|
|
||||||
// registryBlocks are the rows of the IANA special-purpose address registries,
|
|
||||||
// without the deprecated ones:
|
|
||||||
// - https://www.iana.org/assignments/iana-ipv4-special-registry/
|
|
||||||
// - https://www.iana.org/assignments/iana-ipv6-special-registry/
|
|
||||||
//
|
|
||||||
// They also hold the multicast blocks, which the IANA address space registries
|
|
||||||
// list as Multicast:
|
|
||||||
// - https://www.iana.org/assignments/ipv4-address-space/
|
|
||||||
// - https://www.iana.org/assignments/ipv6-address-space/
|
|
||||||
//
|
|
||||||
//nolint:gochecknoglobals // a fixed list, parsed once
|
|
||||||
var registryBlocks = []block{
|
|
||||||
reservedBlock("0.0.0.0/8", "This network"),
|
|
||||||
reservedBlock("0.0.0.0/32", "This host on this network"),
|
|
||||||
reservedBlock("10.0.0.0/8", privateUse),
|
|
||||||
reservedBlock("100.64.0.0/10", "Shared Address Space"),
|
|
||||||
reservedBlock("127.0.0.0/8", "Loopback"),
|
|
||||||
reservedBlock("169.254.0.0/16", "Link Local"),
|
|
||||||
reservedBlock("172.16.0.0/12", privateUse),
|
|
||||||
reservedBlock("192.0.0.0/24", ietfProtocolAssignments),
|
|
||||||
reservedBlock("192.0.0.0/29", "IPv4 Service Continuity Prefix"),
|
|
||||||
reservedBlock("192.0.0.8/32", "IPv4 dummy address"),
|
|
||||||
reachableBlock("192.0.0.9/32", portControlAnycast),
|
|
||||||
reachableBlock("192.0.0.10/32", natTraversalAnycast),
|
|
||||||
reservedBlock("192.0.0.170/32", nat64Discovery),
|
|
||||||
reservedBlock("192.0.0.171/32", nat64Discovery),
|
|
||||||
reservedBlock("192.0.2.0/24", "Documentation (TEST-NET-1)"),
|
|
||||||
reachableBlock("192.31.196.0/24", "AS112-v4"),
|
|
||||||
reachableBlock("192.52.193.0/24", amt),
|
|
||||||
reservedBlock("192.88.99.2/32", "6a44-relay anycast address"),
|
|
||||||
reservedBlock("192.168.0.0/16", privateUse),
|
|
||||||
reachableBlock("192.175.48.0/24", as112DirectDelegation),
|
|
||||||
reservedBlock("198.18.0.0/15", benchmarking),
|
|
||||||
reservedBlock("198.51.100.0/24", "Documentation (TEST-NET-2)"),
|
|
||||||
reservedBlock("203.0.113.0/24", "Documentation (TEST-NET-3)"),
|
|
||||||
reservedBlock("224.0.0.0/4", multicast),
|
|
||||||
reservedBlock("240.0.0.0/4", "Reserved"),
|
|
||||||
reservedBlock("255.255.255.255/32", "Limited Broadcast"),
|
|
||||||
|
|
||||||
reservedBlock("::1/128", "Loopback Address"),
|
|
||||||
reservedBlock("::/128", "Unspecified Address"),
|
|
||||||
reservedBlock("::ffff:0:0/96", "IPv4-mapped Address"),
|
|
||||||
reachableBlock("64:ff9b::/96", ipv4IPv6Translation),
|
|
||||||
reservedBlock("64:ff9b:1::/48", ipv4IPv6Translation),
|
|
||||||
reservedBlock("100::/64", "Discard-Only Address Block"),
|
|
||||||
reservedBlock("100:0:0:1::/64", "Dummy IPv6 Prefix"),
|
|
||||||
reservedBlock("2001::/23", ietfProtocolAssignments),
|
|
||||||
reachableBlock("2001::/32", "TEREDO"),
|
|
||||||
reachableBlock("2001:1::1/128", portControlAnycast),
|
|
||||||
reachableBlock("2001:1::2/128", natTraversalAnycast),
|
|
||||||
reachableBlock("2001:1::3/128", "DNS-SD Service Registration Protocol Anycast"),
|
|
||||||
reservedBlock("2001:2::/48", benchmarking),
|
|
||||||
reachableBlock("2001:3::/32", amt),
|
|
||||||
reachableBlock("2001:4:112::/48", "AS112-v6"),
|
|
||||||
reachableBlock("2001:20::/28", "ORCHIDv2"),
|
|
||||||
reachableBlock("2001:30::/28", "Drone Remote ID Protocol Entity Tags (DETs) Prefix"),
|
|
||||||
reservedBlock("2001:db8::/32", documentation),
|
|
||||||
reachableBlock("2002::/16", "6to4"),
|
|
||||||
reachableBlock("2620:4f:8000::/48", as112DirectDelegation),
|
|
||||||
reservedBlock("3fff::/20", documentation),
|
|
||||||
reservedBlock("5f00::/16", "Segment Routing (SRv6) SIDs"),
|
|
||||||
reservedBlock("fc00::/7", "Unique-Local"),
|
|
||||||
reservedBlock("fe80::/10", "Link-Local Unicast"),
|
|
||||||
reservedBlock("ff00::/8", multicast),
|
|
||||||
}
|
|
||||||
|
|
||||||
// asRange is a range of reserved AS numbers, first to last.
|
|
||||||
type asRange struct {
|
|
||||||
first, last int
|
|
||||||
name string
|
|
||||||
}
|
|
||||||
|
|
||||||
// The reserved AS numbers of the IANA special-purpose AS numbers registry.
|
|
||||||
const (
|
|
||||||
asTrans = 23456
|
|
||||||
firstDocumentationAS16 = 64496
|
|
||||||
lastDocumentationAS16 = 64511
|
|
||||||
firstPrivateUseAS16 = 64512
|
|
||||||
lastPrivateUseAS16 = 65534
|
|
||||||
lastAS16 = 65535
|
|
||||||
firstDocumentationAS32 = 65536
|
|
||||||
lastDocumentationAS32 = 65551
|
|
||||||
firstPrivateUseAS32 = 4200000000
|
|
||||||
lastPrivateUseAS32 = 4294967294
|
|
||||||
lastAS32 = 4294967295
|
|
||||||
)
|
|
||||||
|
|
||||||
// reservedASRanges are the reserved rows of the IANA special-purpose AS numbers
|
|
||||||
// registry, https://www.iana.org/assignments/iana-as-numbers-special-registry/.
|
|
||||||
// AS 0 is left out: RouteWatch uses it for a route with no known origin. AS 112
|
|
||||||
// is left out because the AS112 project announces routes with it.
|
|
||||||
//
|
|
||||||
//nolint:gochecknoglobals // a fixed list
|
|
||||||
var reservedASRanges = []asRange{
|
|
||||||
{first: asTrans, last: asTrans, name: "AS_TRANS"},
|
|
||||||
{first: firstDocumentationAS16, last: lastDocumentationAS16, name: documentation},
|
|
||||||
{first: firstPrivateUseAS16, last: lastPrivateUseAS16, name: privateUse},
|
|
||||||
{first: lastAS16, last: lastAS16, name: "Last 16-bit AS number"},
|
|
||||||
{first: firstDocumentationAS32, last: lastDocumentationAS32, name: documentation},
|
|
||||||
{first: firstPrivateUseAS32, last: lastPrivateUseAS32, name: privateUse},
|
|
||||||
{first: lastAS32, last: lastAS32, name: "Last 32-bit AS number"},
|
|
||||||
}
|
|
||||||
@@ -1,131 +0,0 @@
|
|||||||
package bogon_test
|
|
||||||
|
|
||||||
import (
|
|
||||||
"net/netip"
|
|
||||||
"strconv"
|
|
||||||
"testing"
|
|
||||||
|
|
||||||
"git.eeqj.de/sneak/routewatch/internal/bogon"
|
|
||||||
)
|
|
||||||
|
|
||||||
// privateUseTen is why a prefix or an address inside 10.0.0.0/8 is reserved.
|
|
||||||
const privateUseTen = "Private-Use (10.0.0.0/8)"
|
|
||||||
|
|
||||||
func TestPrefix(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
tests := []struct {
|
|
||||||
prefix string
|
|
||||||
// want is why the prefix is reserved, or "" when it is not.
|
|
||||||
want string
|
|
||||||
}{
|
|
||||||
{"10.1.2.0/24", privateUseTen},
|
|
||||||
{"10.0.0.0/8", privateUseTen},
|
|
||||||
{"192.0.2.0/24", "Documentation (TEST-NET-1) (192.0.2.0/24)"},
|
|
||||||
{"2001:db8:1::/48", "Documentation (2001:db8::/32)"},
|
|
||||||
{"fd00::/8", "Unique-Local (fc00::/7)"},
|
|
||||||
// Multicast space, from the IANA address space registries.
|
|
||||||
{"224.0.0.0/4", "Multicast (224.0.0.0/4)"},
|
|
||||||
{"239.255.0.0/16", "Multicast (224.0.0.0/4)"},
|
|
||||||
{"ff00::/8", "Multicast (ff00::/8)"},
|
|
||||||
{"ff02::/16", "Multicast (ff00::/8)"},
|
|
||||||
// The most specific block names the reason.
|
|
||||||
{"0.0.0.0/32", "This host on this network (0.0.0.0/32)"},
|
|
||||||
{"2001:2::/48", "Benchmarking (2001:2::/48)"},
|
|
||||||
// Blocks that may be announced, inside reserved blocks.
|
|
||||||
{"192.0.0.9/32", ""},
|
|
||||||
{"2001:4:112::/48", ""},
|
|
||||||
// Blocks that may be announced, inside no reserved block.
|
|
||||||
{"192.175.48.0/24", ""},
|
|
||||||
{"2002::/16", ""},
|
|
||||||
// A prefix that holds a reserved block is not inside it.
|
|
||||||
{"192.0.0.0/16", ""},
|
|
||||||
{"0.0.0.0/0", ""},
|
|
||||||
// Prefixes anyone may announce.
|
|
||||||
{"8.8.8.0/24", ""},
|
|
||||||
{"2001:4860::/32", ""},
|
|
||||||
}
|
|
||||||
|
|
||||||
for _, tc := range tests {
|
|
||||||
t.Run(tc.prefix, func(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
reason, reserved := bogon.Prefix(netip.MustParsePrefix(tc.prefix))
|
|
||||||
if reason != tc.want || reserved != (tc.want != "") {
|
|
||||||
t.Errorf("Prefix(%s) = %q, %v; want %q", tc.prefix, reason,
|
|
||||||
reserved, tc.want)
|
|
||||||
}
|
|
||||||
})
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestAddr(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
tests := []struct {
|
|
||||||
addr string
|
|
||||||
// want is why the address is reserved, or "" when it is not.
|
|
||||||
want string
|
|
||||||
}{
|
|
||||||
{"10.0.0.1", privateUseTen},
|
|
||||||
{"fd00::1", "Unique-Local (fc00::/7)"},
|
|
||||||
// An IPv4 address written in IPv6 form counts as the IPv4 address.
|
|
||||||
{"::ffff:10.0.0.1", privateUseTen},
|
|
||||||
{"::ffff:8.8.8.8", ""},
|
|
||||||
// An address that may be announced, inside a reserved block.
|
|
||||||
{"192.0.0.9", ""},
|
|
||||||
// Addresses anyone may announce.
|
|
||||||
{"8.8.8.8", ""},
|
|
||||||
{"2001:4860:4860::8888", ""},
|
|
||||||
}
|
|
||||||
|
|
||||||
for _, tc := range tests {
|
|
||||||
t.Run(tc.addr, func(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
reason, reserved := bogon.Addr(netip.MustParseAddr(tc.addr))
|
|
||||||
if reason != tc.want || reserved != (tc.want != "") {
|
|
||||||
t.Errorf("Addr(%s) = %q, %v; want %q", tc.addr, reason,
|
|
||||||
reserved, tc.want)
|
|
||||||
}
|
|
||||||
})
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestASN(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
tests := []struct {
|
|
||||||
asn int
|
|
||||||
// want is why the AS number is reserved, or "" when it is not.
|
|
||||||
want string
|
|
||||||
}{
|
|
||||||
{23456, "AS_TRANS (AS23456)"},
|
|
||||||
{64496, "Documentation (AS64496-AS64511)"},
|
|
||||||
{64512, "Private-Use (AS64512-AS65534)"},
|
|
||||||
{65534, "Private-Use (AS64512-AS65534)"},
|
|
||||||
{65535, "Last 16-bit AS number (AS65535)"},
|
|
||||||
{65551, "Documentation (AS65536-AS65551)"},
|
|
||||||
{4200000000, "Private-Use (AS4200000000-AS4294967294)"},
|
|
||||||
{4294967295, "Last 32-bit AS number (AS4294967295)"},
|
|
||||||
// AS 0 means no known origin, and the AS112 project uses AS 112.
|
|
||||||
{0, ""},
|
|
||||||
{112, ""},
|
|
||||||
// Not handed out yet, which is not counted.
|
|
||||||
{65552, ""},
|
|
||||||
{15169, ""},
|
|
||||||
{64495, ""},
|
|
||||||
}
|
|
||||||
|
|
||||||
for _, tc := range tests {
|
|
||||||
t.Run(strconv.Itoa(tc.asn), func(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
reason, reserved := bogon.ASN(tc.asn)
|
|
||||||
if reason != tc.want || reserved != (tc.want != "") {
|
|
||||||
t.Errorf("ASN(%d) = %q, %v; want %q", tc.asn, reason, reserved,
|
|
||||||
tc.want)
|
|
||||||
}
|
|
||||||
})
|
|
||||||
}
|
|
||||||
}
|
|
||||||
@@ -1,51 +0,0 @@
|
|||||||
// Package cachedb opens cache.db, the SQLite file in the state directory that
|
|
||||||
// holds RouteWatch's caches of answers from outside services. It is separate
|
|
||||||
// from the routes database so that a cache write never queues behind a write
|
|
||||||
// to that database.
|
|
||||||
package cachedb
|
|
||||||
|
|
||||||
import (
|
|
||||||
"context"
|
|
||||||
"database/sql"
|
|
||||||
_ "embed" // for schema.sql
|
|
||||||
"fmt"
|
|
||||||
"path/filepath"
|
|
||||||
|
|
||||||
_ "github.com/mattn/go-sqlite3" // CGO SQLite driver
|
|
||||||
)
|
|
||||||
|
|
||||||
const (
|
|
||||||
// fileName is the database's file in the state directory.
|
|
||||||
fileName = "cache.db"
|
|
||||||
|
|
||||||
// busyTimeoutMs is how long a write waits for another write to finish.
|
|
||||||
busyTimeoutMs = 5000
|
|
||||||
)
|
|
||||||
|
|
||||||
//go:embed schema.sql
|
|
||||||
var schema string
|
|
||||||
|
|
||||||
// Open opens cache.db in stateDir, creating the file and its tables when they
|
|
||||||
// do not exist. The caller closes the database.
|
|
||||||
func Open(ctx context.Context, stateDir string) (*sql.DB, error) {
|
|
||||||
dsn := fmt.Sprintf(
|
|
||||||
"file:%s?_journal_mode=WAL&_synchronous=NORMAL&_busy_timeout=%d"+
|
|
||||||
"&_txlock=immediate",
|
|
||||||
filepath.Join(stateDir, fileName),
|
|
||||||
busyTimeoutMs,
|
|
||||||
)
|
|
||||||
|
|
||||||
db, err := sql.Open("sqlite3", dsn)
|
|
||||||
if err != nil {
|
|
||||||
return nil, fmt.Errorf("opening the cache database: %w", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
_, err = db.ExecContext(ctx, schema)
|
|
||||||
if err != nil {
|
|
||||||
_ = db.Close()
|
|
||||||
|
|
||||||
return nil, fmt.Errorf("creating the cache database's tables: %w", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
return db, nil
|
|
||||||
}
|
|
||||||
@@ -1,64 +0,0 @@
|
|||||||
package cachedb
|
|
||||||
|
|
||||||
import (
|
|
||||||
"os"
|
|
||||||
"path/filepath"
|
|
||||||
"testing"
|
|
||||||
)
|
|
||||||
|
|
||||||
// TestOpenCreatesTheFileInWALMode opens a new database, and then opens it
|
|
||||||
// again, as a restart does.
|
|
||||||
func TestOpenCreatesTheFileInWALMode(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
dir := t.TempDir()
|
|
||||||
|
|
||||||
for range 2 {
|
|
||||||
db, err := Open(t.Context(), dir)
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("Open: %v", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
var mode string
|
|
||||||
|
|
||||||
err = db.QueryRowContext(t.Context(), "PRAGMA journal_mode").Scan(&mode)
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("reading the journal mode: %v", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
if mode != "wal" {
|
|
||||||
t.Errorf("journal mode %q, want wal", mode)
|
|
||||||
}
|
|
||||||
|
|
||||||
var rows int
|
|
||||||
|
|
||||||
err = db.QueryRowContext(t.Context(),
|
|
||||||
"SELECT COUNT(*) FROM hostnames").Scan(&rows)
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("reading the hostnames table: %v", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
err = db.Close()
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("Close: %v", err)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
_, err := os.Stat(filepath.Join(dir, fileName))
|
|
||||||
if err != nil {
|
|
||||||
t.Errorf("the file is not in the state directory: %v", err)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestOpenFailsWhenTheDirectoryDoesNotExist checks that a state directory that
|
|
||||||
// is not there is reported, not ignored.
|
|
||||||
func TestOpenFailsWhenTheDirectoryDoesNotExist(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
db, err := Open(t.Context(), filepath.Join(t.TempDir(), "missing"))
|
|
||||||
if err == nil {
|
|
||||||
_ = db.Close()
|
|
||||||
|
|
||||||
t.Fatal("Open succeeded in a directory that does not exist")
|
|
||||||
}
|
|
||||||
}
|
|
||||||
@@ -1,24 +0,0 @@
|
|||||||
-- hostnames holds the PTR names of addresses. names has the names separated by
|
|
||||||
-- spaces, and is empty for an address with no PTR names. expires_at is in
|
|
||||||
-- seconds since 1970.
|
|
||||||
CREATE TABLE IF NOT EXISTS hostnames (
|
|
||||||
address TEXT PRIMARY KEY,
|
|
||||||
names TEXT NOT NULL,
|
|
||||||
expires_at INTEGER NOT NULL
|
|
||||||
);
|
|
||||||
|
|
||||||
-- geolocations holds the answers of geojs. A part geojs does not know is NULL.
|
|
||||||
-- fetched_at is in seconds since 1970.
|
|
||||||
CREATE TABLE IF NOT EXISTS geolocations (
|
|
||||||
address TEXT PRIMARY KEY,
|
|
||||||
country_code TEXT,
|
|
||||||
country TEXT,
|
|
||||||
continent_code TEXT,
|
|
||||||
region TEXT,
|
|
||||||
city TEXT,
|
|
||||||
latitude REAL,
|
|
||||||
longitude REAL,
|
|
||||||
accuracy_radius_km INTEGER,
|
|
||||||
time_zone TEXT,
|
|
||||||
fetched_at INTEGER NOT NULL
|
|
||||||
);
|
|
||||||
@@ -30,14 +30,6 @@ const (
|
|||||||
maxPort = 65535
|
maxPort = 65535
|
||||||
)
|
)
|
||||||
|
|
||||||
// The private address ranges of RFC 1918. The HTTP server trusts a reverse
|
|
||||||
// proxy in any of them when TRUSTED_PROXIES is not set.
|
|
||||||
const (
|
|
||||||
privateRange10 = "10.0.0.0/8"
|
|
||||||
privateRange172 = "172.16.0.0/12"
|
|
||||||
privateRange192 = "192.168.0.0/16"
|
|
||||||
)
|
|
||||||
|
|
||||||
var (
|
var (
|
||||||
errRelativeXDGDataHome = errors.New("XDG_DATA_HOME must be an absolute path")
|
errRelativeXDGDataHome = errors.New("XDG_DATA_HOME must be an absolute path")
|
||||||
errUnsupportedOS = errors.New("unsupported operating system")
|
errUnsupportedOS = errors.New("unsupported operating system")
|
||||||
@@ -167,21 +159,11 @@ func getPort() (int, error) {
|
|||||||
return int(port), nil
|
return int(port), nil
|
||||||
}
|
}
|
||||||
|
|
||||||
// getTrustedProxies returns the reverse proxies the HTTP server trusts. When
|
// getTrustedProxies returns the addresses and CIDR ranges listed in
|
||||||
// TRUSTED_PROXIES is not set, they are the private ranges of RFC 1918. When it
|
// TRUSTED_PROXIES, such as "10.0.0.0/8,192.0.2.1", or none when it is not set.
|
||||||
// is set, they are the addresses and CIDR ranges it lists, such as
|
// An address is returned as a range holding only that address.
|
||||||
// "10.0.0.0/8,192.0.2.1", and none when it is empty. An address is returned as
|
|
||||||
// a range holding only that address.
|
|
||||||
func getTrustedProxies() ([]netip.Prefix, error) {
|
func getTrustedProxies() ([]netip.Prefix, error) {
|
||||||
value, set := os.LookupEnv("TRUSTED_PROXIES")
|
value := os.Getenv("TRUSTED_PROXIES")
|
||||||
if !set {
|
|
||||||
return []netip.Prefix{
|
|
||||||
netip.MustParsePrefix(privateRange10),
|
|
||||||
netip.MustParsePrefix(privateRange172),
|
|
||||||
netip.MustParsePrefix(privateRange192),
|
|
||||||
}, nil
|
|
||||||
}
|
|
||||||
|
|
||||||
if value == "" {
|
if value == "" {
|
||||||
return nil, nil
|
return nil, nil
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -2,7 +2,6 @@ package config
|
|||||||
|
|
||||||
import (
|
import (
|
||||||
"net/netip"
|
"net/netip"
|
||||||
"os"
|
|
||||||
"runtime"
|
"runtime"
|
||||||
"slices"
|
"slices"
|
||||||
"testing"
|
"testing"
|
||||||
@@ -47,47 +46,25 @@ func TestNewRefusesInvalidPort(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// TestNewTrustsPrivateRangesByDefault checks that with TRUSTED_PROXIES unset
|
func TestNewReadsTrustedProxies(t *testing.T) {
|
||||||
// the trusted proxies are the private ranges of RFC 1918. Before the fix no
|
|
||||||
// proxy was trusted, so a reverse proxy at a private address was taken for the
|
|
||||||
// client (https://git.eeqj.de/sneak/routewatch/issues/198).
|
|
||||||
func TestNewTrustsPrivateRangesByDefault(t *testing.T) {
|
|
||||||
t.Setenv("PORT", "")
|
t.Setenv("PORT", "")
|
||||||
t.Setenv("XDG_DATA_HOME", "")
|
t.Setenv("XDG_DATA_HOME", "")
|
||||||
|
|
||||||
// t.Setenv puts the variable back when the test ends.
|
|
||||||
t.Setenv("TRUSTED_PROXIES", "")
|
t.Setenv("TRUSTED_PROXIES", "")
|
||||||
|
|
||||||
err := os.Unsetenv("TRUSTED_PROXIES")
|
|
||||||
if err != nil {
|
|
||||||
t.Fatal(err)
|
|
||||||
}
|
|
||||||
|
|
||||||
cfg, err := New()
|
cfg, err := New()
|
||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatalf("New() without TRUSTED_PROXIES: %v", err)
|
t.Fatalf("New() without TRUSTED_PROXIES: %v", err)
|
||||||
}
|
}
|
||||||
|
|
||||||
want := []netip.Prefix{
|
if len(cfg.TrustedProxies) != 0 {
|
||||||
netip.MustParsePrefix("10.0.0.0/8"),
|
t.Errorf("New() without TRUSTED_PROXIES: TrustedProxies = %v, want none",
|
||||||
netip.MustParsePrefix("172.16.0.0/12"),
|
cfg.TrustedProxies)
|
||||||
netip.MustParsePrefix("192.168.0.0/16"),
|
|
||||||
}
|
}
|
||||||
if !slices.Equal(cfg.TrustedProxies, want) {
|
|
||||||
t.Errorf("New() without TRUSTED_PROXIES: TrustedProxies = %v, want %v",
|
|
||||||
cfg.TrustedProxies, want)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestNewReadsTrustedProxies checks that a list in TRUSTED_PROXIES replaces
|
|
||||||
// the default, and that TRUSTED_PROXIES set but empty trusts no proxy.
|
|
||||||
func TestNewReadsTrustedProxies(t *testing.T) {
|
|
||||||
t.Setenv("PORT", "")
|
|
||||||
t.Setenv("XDG_DATA_HOME", "")
|
|
||||||
|
|
||||||
t.Setenv("TRUSTED_PROXIES", "10.0.0.0/8, 192.0.2.1,2001:db8::/32")
|
t.Setenv("TRUSTED_PROXIES", "10.0.0.0/8, 192.0.2.1,2001:db8::/32")
|
||||||
|
|
||||||
cfg, err := New()
|
cfg, err = New()
|
||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatalf("New() with TRUSTED_PROXIES: %v", err)
|
t.Fatalf("New() with TRUSTED_PROXIES: %v", err)
|
||||||
}
|
}
|
||||||
@@ -100,18 +77,6 @@ func TestNewReadsTrustedProxies(t *testing.T) {
|
|||||||
if !slices.Equal(cfg.TrustedProxies, want) {
|
if !slices.Equal(cfg.TrustedProxies, want) {
|
||||||
t.Errorf("TrustedProxies = %v, want %v", cfg.TrustedProxies, want)
|
t.Errorf("TrustedProxies = %v, want %v", cfg.TrustedProxies, want)
|
||||||
}
|
}
|
||||||
|
|
||||||
t.Setenv("TRUSTED_PROXIES", "")
|
|
||||||
|
|
||||||
cfg, err = New()
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("New() with TRUSTED_PROXIES set but empty: %v", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
if len(cfg.TrustedProxies) != 0 {
|
|
||||||
t.Errorf("New() with TRUSTED_PROXIES set but empty: TrustedProxies = %v, "+
|
|
||||||
"want none", cfg.TrustedProxies)
|
|
||||||
}
|
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestNewRefusesInvalidTrustedProxies(t *testing.T) {
|
func TestNewRefusesInvalidTrustedProxies(t *testing.T) {
|
||||||
|
|||||||
@@ -972,34 +972,6 @@ func (d *Database) GetAS(ctx context.Context, asn int) (*ASN, bool, error) {
|
|||||||
return &asnInfo, true, nil
|
return &asnInfo, true, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
// The queries of GetLastSeenASes. The index on last_seen answers each without
|
|
||||||
// reading the whole table.
|
|
||||||
const (
|
|
||||||
newestLastSeenASQuery = `
|
|
||||||
SELECT asn, handle, description, first_seen, last_seen
|
|
||||||
FROM asns ORDER BY last_seen DESC LIMIT 1`
|
|
||||||
oldestLastSeenASQuery = `
|
|
||||||
SELECT asn, handle, description, first_seen, last_seen
|
|
||||||
FROM asns ORDER BY last_seen ASC LIMIT 1`
|
|
||||||
)
|
|
||||||
|
|
||||||
// GetLastSeenASes returns two ASes of the asns table, with their handles,
|
|
||||||
// descriptions and seen times: the one last seen most recently, and the one
|
|
||||||
// last seen longest ago. Both are nil when the table holds no AS.
|
|
||||||
func (d *Database) GetLastSeenASes(ctx context.Context) (*ASN, *ASN, error) {
|
|
||||||
newest, found, err := d.readOneAS(ctx, newestLastSeenASQuery)
|
|
||||||
if err != nil || !found {
|
|
||||||
return nil, nil, err
|
|
||||||
}
|
|
||||||
|
|
||||||
oldest, _, err := d.readOneAS(ctx, oldestLastSeenASQuery)
|
|
||||||
if err != nil {
|
|
||||||
return nil, nil, err
|
|
||||||
}
|
|
||||||
|
|
||||||
return newest, oldest, nil
|
|
||||||
}
|
|
||||||
|
|
||||||
// asWHOISQuery reads the handle, description and WHOIS details of one AS, and
|
// asWHOISQuery reads the handle, description and WHOIS details of one AS, and
|
||||||
// when the WHOIS details were fetched.
|
// when the WHOIS details were fetched.
|
||||||
const asWHOISQuery = `
|
const asWHOISQuery = `
|
||||||
@@ -1407,29 +1379,6 @@ func (d *Database) Ping(ctx context.Context) error {
|
|||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
|
|
||||||
// readOneAS returns the AS that query reads, and whether it read one.
|
|
||||||
func (d *Database) readOneAS(ctx context.Context, query string) (*ASN, bool, error) {
|
|
||||||
var (
|
|
||||||
asnInfo ASN
|
|
||||||
handle, description sql.NullString
|
|
||||||
)
|
|
||||||
|
|
||||||
err := d.db.QueryRowContext(ctx, query).Scan(&asnInfo.ASN, &handle,
|
|
||||||
&description, &asnInfo.FirstSeen, &asnInfo.LastSeen)
|
|
||||||
if errors.Is(err, sql.ErrNoRows) {
|
|
||||||
return nil, false, nil
|
|
||||||
}
|
|
||||||
|
|
||||||
if err != nil {
|
|
||||||
return nil, false, fmt.Errorf("failed to query AS by last seen: %w", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
asnInfo.Handle = handle.String
|
|
||||||
asnInfo.Description = description.String
|
|
||||||
|
|
||||||
return &asnInfo, true, nil
|
|
||||||
}
|
|
||||||
|
|
||||||
// truncateWAL runs the TRUNCATE checkpoint of Checkpoint on a WAL file of
|
// truncateWAL runs the TRUNCATE checkpoint of Checkpoint on a WAL file of
|
||||||
// walBytes, with the writing connection's busy timeout lowered to
|
// walBytes, with the writing connection's busy timeout lowered to
|
||||||
// truncateBusyTimeoutMs for it, and logs how long it took. The caller holds the
|
// truncateBusyTimeoutMs for it, and logs how long it took. The caller holds the
|
||||||
|
|||||||
@@ -72,34 +72,17 @@ const (
|
|||||||
// lockTestPrefix is the prefix the batch writes in the checkpoint tests
|
// lockTestPrefix is the prefix the batch writes in the checkpoint tests
|
||||||
// store.
|
// store.
|
||||||
lockTestPrefix = "192.0.2.0/24"
|
lockTestPrefix = "192.0.2.0/24"
|
||||||
// longBusyTimeoutMs is the busy timeout setLongBusyTimeout gives the
|
|
||||||
// writing connection. A checkpoint that waits for an open read takes this
|
|
||||||
// long.
|
|
||||||
longBusyTimeoutMs = 30000
|
|
||||||
// readerCheckpointLimit is how long a checkpoint, and a batch write
|
// readerCheckpointLimit is how long a checkpoint, and a batch write
|
||||||
// waiting for it, may take while a reader is open: well under
|
// waiting for it, may take while a reader is open: well under the busy
|
||||||
// longBusyTimeoutMs, with room for a stall on a busy host.
|
// timeout, which a checkpoint that waits for the reader takes.
|
||||||
readerCheckpointLimit = 15 * time.Second
|
readerCheckpointLimit = time.Second
|
||||||
// truncateCheckpointLimit is how long a checkpoint of a WAL file over
|
// truncateCheckpointLimit is how long a checkpoint of a WAL file over
|
||||||
// sqliteWALSizeLimitBytes may take to give up while a reader is open: its
|
// sqliteWALSizeLimitBytes may take to give up while a reader is open: its
|
||||||
// wait for the reader plus readerCheckpointLimit.
|
// wait for the reader plus a second, well under the busy timeout.
|
||||||
truncateCheckpointLimit = truncateBusyTimeoutMs*time.Millisecond +
|
truncateCheckpointLimit = truncateBusyTimeoutMs*time.Millisecond +
|
||||||
readerCheckpointLimit
|
time.Second
|
||||||
)
|
)
|
||||||
|
|
||||||
// setLongBusyTimeout gives the writing connection of db longBusyTimeoutMs, so
|
|
||||||
// that a checkpoint that waits for an open read takes far longer than a stall
|
|
||||||
// on a busy host.
|
|
||||||
func setLongBusyTimeout(ctx context.Context, t *testing.T, db *Database) {
|
|
||||||
t.Helper()
|
|
||||||
|
|
||||||
_, err := db.writer.ExecContext(ctx,
|
|
||||||
fmt.Sprintf("PRAGMA busy_timeout=%d", longBusyTimeoutMs))
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("failed to set the busy timeout: %v", err)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestConnectionPoolPragmas holds several pooled connections open at once and
|
// TestConnectionPoolPragmas holds several pooled connections open at once and
|
||||||
// checks each one carries the per-connection settings from the DSN, plus the
|
// checks each one carries the per-connection settings from the DSN, plus the
|
||||||
// process-wide heap limits, and sees the new file with auto_vacuum
|
// process-wide heap limits, and sees the new file with auto_vacuum
|
||||||
@@ -290,7 +273,7 @@ func TestVacuumReturnsFreePages(t *testing.T) {
|
|||||||
writeTableChanges(t, db, table)
|
writeTableChanges(t, db, table)
|
||||||
|
|
||||||
// The peer going down removes every prefix, and so every row.
|
// The peer going down removes every prefix, and so every row.
|
||||||
table.PeersDown([]string{peerA})
|
table.PeerDown(peerA)
|
||||||
writeTableChanges(t, db, table)
|
writeTableChanges(t, db, table)
|
||||||
|
|
||||||
var before int
|
var before int
|
||||||
@@ -596,8 +579,6 @@ func TestCheckpointDoesNotWaitForReader(t *testing.T) {
|
|||||||
ctx := context.Background()
|
ctx := context.Background()
|
||||||
ts := time.Now().UTC()
|
ts := time.Now().UTC()
|
||||||
|
|
||||||
setLongBusyTimeout(ctx, t, db)
|
|
||||||
|
|
||||||
err = db.UpdatePrefixesBatch(map[string]time.Time{"198.51.100.0/24": ts})
|
err = db.UpdatePrefixesBatch(map[string]time.Time{"198.51.100.0/24": ts})
|
||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatalf("failed to store the prefix the reader reads: %v", err)
|
t.Fatalf("failed to store the prefix the reader reads: %v", err)
|
||||||
@@ -709,7 +690,6 @@ func TestCheckpointEmptiesWALOnceNoReadIsOpen(t *testing.T) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
fillWALPastLimit(ctx, t, db)
|
fillWALPastLimit(ctx, t, db)
|
||||||
setLongBusyTimeout(ctx, t, db)
|
|
||||||
|
|
||||||
started := time.Now()
|
started := time.Now()
|
||||||
|
|
||||||
@@ -1023,62 +1003,6 @@ func TestGetASReadsTheRecordOrTheBuiltInList(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// TestGetLastSeenASes stores three ASes, each last seen at another time, and
|
|
||||||
// expects the one last seen most recently and the one last seen longest ago.
|
|
||||||
// An empty asns table gives neither.
|
|
||||||
func TestGetLastSeenASes(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
db, err := New(&config.Config{StateDir: t.TempDir()}, logger.New())
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("failed to create database: %v", err)
|
|
||||||
}
|
|
||||||
defer func() { _ = db.Close() }()
|
|
||||||
|
|
||||||
newest, oldest, err := db.GetLastSeenASes(t.Context())
|
|
||||||
if err != nil || newest != nil || oldest != nil {
|
|
||||||
t.Fatalf("GetLastSeenASes() on an empty table = %+v, %+v, %v; want "+
|
|
||||||
"neither", newest, oldest, err)
|
|
||||||
}
|
|
||||||
|
|
||||||
const (
|
|
||||||
newestASN = 13335
|
|
||||||
middleASN = 15169
|
|
||||||
oldestASN = 3
|
|
||||||
)
|
|
||||||
|
|
||||||
start := time.Date(2026, 1, 2, 3, 4, 5, 0, time.UTC)
|
|
||||||
|
|
||||||
err = db.GetOrCreateASNBatch(map[int]SeenTimes{
|
|
||||||
middleASN: {First: start, Last: start.Add(time.Hour)},
|
|
||||||
newestASN: {First: start, Last: start.Add(2 * time.Hour)},
|
|
||||||
oldestASN: {First: start, Last: start.Add(time.Minute)},
|
|
||||||
})
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("GetOrCreateASNBatch: %v", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
newest, oldest, err = db.GetLastSeenASes(t.Context())
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("GetLastSeenASes: %v", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
listed, _ := asinfo.Get(newestASN)
|
|
||||||
|
|
||||||
if newest == nil || newest.ASN != newestASN ||
|
|
||||||
!newest.LastSeen.Equal(start.Add(2*time.Hour)) ||
|
|
||||||
newest.Description != listed.Description {
|
|
||||||
t.Errorf("newest = %+v, want AS%d (%s) last seen %v", newest, newestASN,
|
|
||||||
listed.Description, start.Add(2*time.Hour))
|
|
||||||
}
|
|
||||||
|
|
||||||
if oldest == nil || oldest.ASN != oldestASN ||
|
|
||||||
!oldest.LastSeen.Equal(start.Add(time.Minute)) {
|
|
||||||
t.Errorf("oldest = %+v, want AS%d last seen %v", oldest, oldestASN,
|
|
||||||
start.Add(time.Minute))
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestGetASWHOISReadsTheStoredDetails stores an AS with WHOIS details and asks
|
// TestGetASWHOISReadsTheStoredDetails stores an AS with WHOIS details and asks
|
||||||
// GetASWHOIS for it and for an AS the asns table does not hold. The first gets
|
// GetASWHOIS for it and for an AS the asns table does not hold. The first gets
|
||||||
// its stored handle, description and details, and the time they were fetched.
|
// its stored handle, description and details, and the time they were fetched.
|
||||||
|
|||||||
@@ -1,7 +1,6 @@
|
|||||||
// Package database provides SQLite storage for BGP routing data: ASNs,
|
// Package database provides SQLite storage for BGP routing data: ASNs,
|
||||||
// prefixes, announcements, peerings, BGP peers and origin changes. Its routes
|
// prefixes, announcements, peerings and BGP peers. Its routes table holds a
|
||||||
// table holds a copy of the whole routing table that internal/routetable
|
// copy of the whole routing table that internal/routetable keeps in memory.
|
||||||
// keeps in memory.
|
|
||||||
package database
|
package database
|
||||||
|
|
||||||
import (
|
import (
|
||||||
@@ -56,14 +55,6 @@ type Store interface {
|
|||||||
WriteRouteChanges(changes []routetable.Change, newPeers map[string]uint) error
|
WriteRouteChanges(changes []routetable.Change, newPeers map[string]uint) error
|
||||||
ReadRouteTable(ctx context.Context) (*routetable.Table, error)
|
ReadRouteTable(ctx context.Context) (*routetable.Table, error)
|
||||||
|
|
||||||
// Origin change operations: the prefix and origin AS pairs' first and
|
|
||||||
// last peers, as the routing table records them.
|
|
||||||
WriteOriginChanges(changes []routetable.OriginChange) error
|
|
||||||
GetOriginChanges(
|
|
||||||
ctx context.Context, prefix netip.Prefix, limit int,
|
|
||||||
) ([]routetable.OriginChange, error)
|
|
||||||
PruneOriginChanges(ctx context.Context) error
|
|
||||||
|
|
||||||
// ASN WHOIS operations
|
// ASN WHOIS operations
|
||||||
GetNextStaleASN(ctx context.Context, staleThreshold time.Duration) (int, error)
|
GetNextStaleASN(ctx context.Context, staleThreshold time.Duration) (int, error)
|
||||||
UpdateASNWHOIS(ctx context.Context, update *ASNWHOISUpdate) error
|
UpdateASNWHOIS(ctx context.Context, update *ASNWHOISUpdate) error
|
||||||
@@ -71,7 +62,6 @@ type Store interface {
|
|||||||
|
|
||||||
// AS and prefix detail operations
|
// AS and prefix detail operations
|
||||||
GetAS(ctx context.Context, asn int) (*ASN, bool, error)
|
GetAS(ctx context.Context, asn int) (*ASN, bool, error)
|
||||||
GetLastSeenASes(ctx context.Context) (*ASN, *ASN, error)
|
|
||||||
GetASWHOIS(ctx context.Context, asn int) (*ASN, error)
|
GetASWHOIS(ctx context.Context, asn int) (*ASN, error)
|
||||||
GetPrefixFirstSeen(ctx context.Context, prefix netip.Prefix) (time.Time, error)
|
GetPrefixFirstSeen(ctx context.Context, prefix netip.Prefix) (time.Time, error)
|
||||||
GetASPeers(asn int) ([]ASPeer, error)
|
GetASPeers(asn int) ([]ASPeer, error)
|
||||||
|
|||||||
@@ -1,172 +0,0 @@
|
|||||||
package database
|
|
||||||
|
|
||||||
import (
|
|
||||||
"context"
|
|
||||||
"fmt"
|
|
||||||
"net/netip"
|
|
||||||
"time"
|
|
||||||
|
|
||||||
"git.eeqj.de/sneak/routewatch/internal/routetable"
|
|
||||||
)
|
|
||||||
|
|
||||||
// minOriginChangesPerPrefix is how many of a prefix's newest origin changes
|
|
||||||
// the prune always keeps, whatever their age.
|
|
||||||
const minOriginChangesPerPrefix = 500
|
|
||||||
|
|
||||||
// originChangesRetention is how long the prune keeps an origin change: rows
|
|
||||||
// older than this are deleted, unless they are among the prefix's newest
|
|
||||||
// minOriginChangesPerPrefix.
|
|
||||||
const originChangesRetention = 72 * time.Hour
|
|
||||||
|
|
||||||
// The statements of the origin_changes table (schema.sql).
|
|
||||||
const (
|
|
||||||
insertOriginChangeSQL = `INSERT INTO origin_changes
|
|
||||||
(prefix, origin_asn, announced, peer_count, time)
|
|
||||||
VALUES (?, ?, ?, ?, ?)`
|
|
||||||
originChangesSQL = `SELECT prefix, origin_asn, announced, peer_count, time
|
|
||||||
FROM origin_changes WHERE prefix = ? ORDER BY id DESC LIMIT ?`
|
|
||||||
prunePrefixesSQL = `SELECT prefix FROM origin_changes
|
|
||||||
GROUP BY prefix HAVING COUNT(*) > ?`
|
|
||||||
pruneOriginChangesSQL = `DELETE FROM origin_changes
|
|
||||||
WHERE prefix = ? AND time < ? AND id < (
|
|
||||||
SELECT id FROM origin_changes WHERE prefix = ?
|
|
||||||
ORDER BY id DESC LIMIT 1 OFFSET ?)`
|
|
||||||
)
|
|
||||||
|
|
||||||
// WriteOriginChanges writes, in one transaction, the origin changes the
|
|
||||||
// routing table handed over. When it fails, it writes nothing; the caller then
|
|
||||||
// writes them again.
|
|
||||||
func (d *Database) WriteOriginChanges(changes []routetable.OriginChange) error {
|
|
||||||
if len(changes) == 0 {
|
|
||||||
return nil
|
|
||||||
}
|
|
||||||
|
|
||||||
d.lock("WriteOriginChanges")
|
|
||||||
defer d.unlock()
|
|
||||||
|
|
||||||
tx, err := d.beginTx()
|
|
||||||
if err != nil {
|
|
||||||
return fmt.Errorf("failed to begin transaction: %w", err)
|
|
||||||
}
|
|
||||||
defer d.rollback(tx)
|
|
||||||
|
|
||||||
ctx := context.Background()
|
|
||||||
|
|
||||||
stmt, err := tx.PrepareContext(ctx, insertOriginChangeSQL)
|
|
||||||
if err != nil {
|
|
||||||
return fmt.Errorf("failed to prepare origin change insert: %w", err)
|
|
||||||
}
|
|
||||||
defer func() { _ = stmt.Close() }()
|
|
||||||
|
|
||||||
for _, change := range changes {
|
|
||||||
_, err := stmt.ExecContext(ctx, change.Prefix.String(),
|
|
||||||
change.OriginASN, change.Announced, change.PeerCount,
|
|
||||||
change.Time.Unix())
|
|
||||||
if err != nil {
|
|
||||||
return fmt.Errorf("failed to write origin change %s: %w",
|
|
||||||
change.Prefix, err)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
err = tx.Commit()
|
|
||||||
if err != nil {
|
|
||||||
return fmt.Errorf("failed to commit transaction: %w", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
return nil
|
|
||||||
}
|
|
||||||
|
|
||||||
// GetOriginChanges returns at most limit of prefix's origin changes, newest
|
|
||||||
// first.
|
|
||||||
func (d *Database) GetOriginChanges(
|
|
||||||
ctx context.Context, prefix netip.Prefix, limit int,
|
|
||||||
) ([]routetable.OriginChange, error) {
|
|
||||||
rows, err := d.db.QueryContext(ctx, originChangesSQL,
|
|
||||||
prefix.Masked().String(), limit)
|
|
||||||
if err != nil {
|
|
||||||
return nil, fmt.Errorf("failed to read origin changes: %w", err)
|
|
||||||
}
|
|
||||||
defer func() { _ = rows.Close() }()
|
|
||||||
|
|
||||||
var changes []routetable.OriginChange
|
|
||||||
|
|
||||||
for rows.Next() {
|
|
||||||
var (
|
|
||||||
change routetable.OriginChange
|
|
||||||
prefixText string
|
|
||||||
timeSec int64
|
|
||||||
)
|
|
||||||
|
|
||||||
err := rows.Scan(&prefixText, &change.OriginASN, &change.Announced,
|
|
||||||
&change.PeerCount, &timeSec)
|
|
||||||
if err != nil {
|
|
||||||
return nil, fmt.Errorf("failed to read an origin change: %w", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
change.Prefix, err = netip.ParsePrefix(prefixText)
|
|
||||||
if err != nil {
|
|
||||||
return nil, fmt.Errorf("failed to read origin change %s: %w",
|
|
||||||
prefixText, err)
|
|
||||||
}
|
|
||||||
|
|
||||||
change.Time = time.Unix(timeSec, 0).UTC()
|
|
||||||
changes = append(changes, change)
|
|
||||||
}
|
|
||||||
|
|
||||||
err = rows.Err()
|
|
||||||
if err != nil {
|
|
||||||
return nil, fmt.Errorf("failed to read origin changes: %w", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
return changes, nil
|
|
||||||
}
|
|
||||||
|
|
||||||
// PruneOriginChanges deletes, for each prefix with more than
|
|
||||||
// minOriginChangesPerPrefix rows, the rows older than originChangesRetention
|
|
||||||
// that are not among its newest minOriginChangesPerPrefix. It finds those
|
|
||||||
// prefixes on a reading connection and deletes one prefix per write, so no
|
|
||||||
// write holds the lock for long.
|
|
||||||
func (d *Database) PruneOriginChanges(ctx context.Context) error {
|
|
||||||
rows, err := d.db.QueryContext(ctx, prunePrefixesSQL,
|
|
||||||
minOriginChangesPerPrefix)
|
|
||||||
if err != nil {
|
|
||||||
return fmt.Errorf("failed to find prefixes to prune: %w", err)
|
|
||||||
}
|
|
||||||
defer func() { _ = rows.Close() }()
|
|
||||||
|
|
||||||
var prefixes []string
|
|
||||||
|
|
||||||
for rows.Next() {
|
|
||||||
var prefix string
|
|
||||||
|
|
||||||
err := rows.Scan(&prefix)
|
|
||||||
if err != nil {
|
|
||||||
return fmt.Errorf("failed to read a prefix to prune: %w", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
prefixes = append(prefixes, prefix)
|
|
||||||
}
|
|
||||||
|
|
||||||
err = rows.Err()
|
|
||||||
if err != nil {
|
|
||||||
return fmt.Errorf("failed to find prefixes to prune: %w", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
cutoff := time.Now().Add(-originChangesRetention).Unix()
|
|
||||||
|
|
||||||
for _, prefix := range prefixes {
|
|
||||||
d.lock("PruneOriginChanges")
|
|
||||||
|
|
||||||
_, err := d.writer.ExecContext(ctx, pruneOriginChangesSQL,
|
|
||||||
prefix, cutoff, prefix, minOriginChangesPerPrefix-1)
|
|
||||||
|
|
||||||
d.unlock()
|
|
||||||
|
|
||||||
if err != nil {
|
|
||||||
return fmt.Errorf("failed to prune origin changes of %s: %w",
|
|
||||||
prefix, err)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
return nil
|
|
||||||
}
|
|
||||||
@@ -1,217 +0,0 @@
|
|||||||
package database
|
|
||||||
|
|
||||||
import (
|
|
||||||
"net/netip"
|
|
||||||
"reflect"
|
|
||||||
"testing"
|
|
||||||
"time"
|
|
||||||
|
|
||||||
"git.eeqj.de/sneak/routewatch/internal/routetable"
|
|
||||||
)
|
|
||||||
|
|
||||||
// TestOriginChangesWriteThenRead writes three origin changes and reads them
|
|
||||||
// back newest first, honouring the limit.
|
|
||||||
func TestOriginChangesWriteThenRead(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
db := newRoutesTestDatabase(t)
|
|
||||||
prefix := netip.MustParsePrefix(tablePrefixV4)
|
|
||||||
|
|
||||||
changes := []routetable.OriginChange{
|
|
||||||
{Prefix: prefix, OriginASN: asA, Announced: true, PeerCount: 1,
|
|
||||||
Time: tableTime(10)},
|
|
||||||
{Prefix: prefix, OriginASN: asA, Announced: false, PeerCount: 0,
|
|
||||||
Time: tableTime(20)},
|
|
||||||
{Prefix: prefix, OriginASN: asB, Announced: true, PeerCount: 1,
|
|
||||||
Time: tableTime(30)},
|
|
||||||
}
|
|
||||||
|
|
||||||
err := db.WriteOriginChanges(changes)
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("WriteOriginChanges: %v", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
got, err := db.GetOriginChanges(t.Context(), prefix, 2)
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("GetOriginChanges: %v", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
want := []routetable.OriginChange{
|
|
||||||
{Prefix: prefix, OriginASN: asB, Announced: true, PeerCount: 1,
|
|
||||||
Time: tableTime(30)},
|
|
||||||
{Prefix: prefix, OriginASN: asA, Announced: false, PeerCount: 0,
|
|
||||||
Time: tableTime(20)},
|
|
||||||
}
|
|
||||||
|
|
||||||
if !reflect.DeepEqual(got, want) {
|
|
||||||
t.Errorf("GetOriginChanges = %+v, want %+v", got, want)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestOriginChangesPrune checks that the prune keeps, for a prefix with more
|
|
||||||
// than 500 rows, its newest 500 rows and every row newer than 72 hours, and
|
|
||||||
// deletes the rest. It checks which rows survive by their times, not only how
|
|
||||||
// many.
|
|
||||||
func TestOriginChangesPrune(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
db := newRoutesTestDatabase(t)
|
|
||||||
oldPrefix := netip.MustParsePrefix(tablePrefixV4)
|
|
||||||
straddling := netip.MustParsePrefix(tablePrefixShortV4)
|
|
||||||
busy := netip.MustParsePrefix(tablePrefixBusyV4)
|
|
||||||
other := netip.MustParsePrefix("10.20.0.0/16")
|
|
||||||
|
|
||||||
// Whole seconds, so the stored times round-trip exactly.
|
|
||||||
now := time.Now().Truncate(time.Second)
|
|
||||||
|
|
||||||
// 600 rows for oldPrefix, all older than 72 hours, oldest first. The
|
|
||||||
// prune keeps the newest 500 and deletes the oldest 100.
|
|
||||||
changes := rows(oldPrefix, now.Add(-100*time.Hour), 600)
|
|
||||||
// 600 rows for straddling: 400 older than 72 hours, then 200 newer. The
|
|
||||||
// prune keeps all 200 newer and the newest 300 of the 400 older.
|
|
||||||
changes = append(changes, rows(straddling, now.Add(-100*time.Hour), 400)...)
|
|
||||||
changes = append(changes, rows(straddling, now.Add(-71*time.Hour), 200)...)
|
|
||||||
// 100 rows older than 72 hours and minOriginChangesPerPrefix+100 newer
|
|
||||||
// rows for busy. The prune keeps all the newer rows and none of the older,
|
|
||||||
// because the newer rows alone are more than minOriginChangesPerPrefix.
|
|
||||||
changes = append(changes, rows(busy, now.Add(-100*time.Hour), 100)...)
|
|
||||||
changes = append(changes, rows(busy, now.Add(-71*time.Hour),
|
|
||||||
minOriginChangesPerPrefix+100)...)
|
|
||||||
// One row for other, which the prune leaves alone.
|
|
||||||
changes = append(changes, rows(other, now.Add(-100*time.Hour), 1)...)
|
|
||||||
|
|
||||||
err := db.WriteOriginChanges(changes)
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("WriteOriginChanges: %v", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
err = db.PruneOriginChanges(t.Context())
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("PruneOriginChanges: %v", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
// The newest row survives, and the oldest survivor is the 100th oldest,
|
|
||||||
// so a prune that kept the oldest 500 instead would fail.
|
|
||||||
checkSurvivors(t, db, oldPrefix, 500,
|
|
||||||
now.Add(-100*time.Hour).Add(599*time.Minute),
|
|
||||||
now.Add(-100*time.Hour).Add(100*time.Minute))
|
|
||||||
|
|
||||||
// The newest of the newer-than-72-hours rows survives, and the oldest
|
|
||||||
// survivor is the 100th oldest of the older-than-72-hours rows.
|
|
||||||
checkSurvivors(t, db, straddling, 500,
|
|
||||||
now.Add(-71*time.Hour).Add(199*time.Minute),
|
|
||||||
now.Add(-100*time.Hour).Add(100*time.Minute))
|
|
||||||
|
|
||||||
// Every newer-than-72-hours row survives, so the oldest survivor is the
|
|
||||||
// first of them and the newest is minOriginChangesPerPrefix+99 minutes
|
|
||||||
// after it.
|
|
||||||
checkSurvivors(t, db, busy, minOriginChangesPerPrefix+100,
|
|
||||||
now.Add(-71*time.Hour).
|
|
||||||
Add(time.Duration(minOriginChangesPerPrefix+99)*time.Minute),
|
|
||||||
now.Add(-71*time.Hour))
|
|
||||||
|
|
||||||
gotOther, err := db.GetOriginChanges(t.Context(), other, 10)
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("GetOriginChanges: %v", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
if len(gotOther) != 1 {
|
|
||||||
t.Errorf("GetOriginChanges(other) = %d rows, want 1", len(gotOther))
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// rows returns n origin changes for prefix, at times from plus one minute per
|
|
||||||
// step, oldest first.
|
|
||||||
func rows(prefix netip.Prefix, from time.Time, n int) []routetable.OriginChange {
|
|
||||||
changes := make([]routetable.OriginChange, 0, n)
|
|
||||||
|
|
||||||
for i := range n {
|
|
||||||
changes = append(changes, routetable.OriginChange{
|
|
||||||
Prefix: prefix, OriginASN: asA, Announced: true, PeerCount: 1,
|
|
||||||
Time: from.Add(time.Duration(i) * time.Minute),
|
|
||||||
})
|
|
||||||
}
|
|
||||||
|
|
||||||
return changes
|
|
||||||
}
|
|
||||||
|
|
||||||
// survivorTimes returns the times of prefix's rows after the prune, newest
|
|
||||||
// first.
|
|
||||||
func survivorTimes(
|
|
||||||
t *testing.T, db *Database, prefix netip.Prefix, limit int,
|
|
||||||
) []time.Time {
|
|
||||||
t.Helper()
|
|
||||||
|
|
||||||
rows, err := db.GetOriginChanges(t.Context(), prefix, limit)
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("GetOriginChanges: %v", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
times := make([]time.Time, len(rows))
|
|
||||||
for i, row := range rows {
|
|
||||||
times[i] = row.Time
|
|
||||||
}
|
|
||||||
|
|
||||||
return times
|
|
||||||
}
|
|
||||||
|
|
||||||
// checkSurvivors checks that prefix kept exactly wantCount rows, with the
|
|
||||||
// newest survivor at wantNewest and the oldest at wantOldest.
|
|
||||||
func checkSurvivors(
|
|
||||||
t *testing.T, db *Database, prefix netip.Prefix,
|
|
||||||
wantCount int, wantNewest, wantOldest time.Time,
|
|
||||||
) {
|
|
||||||
t.Helper()
|
|
||||||
|
|
||||||
times := survivorTimes(t, db, prefix, wantCount+1)
|
|
||||||
if len(times) != wantCount {
|
|
||||||
t.Errorf("GetOriginChanges(%s) = %d rows, want %d",
|
|
||||||
prefix, len(times), wantCount)
|
|
||||||
|
|
||||||
return
|
|
||||||
}
|
|
||||||
|
|
||||||
if !times[0].Equal(wantNewest) {
|
|
||||||
t.Errorf("the newest surviving row's time = %v, want %v",
|
|
||||||
times[0], wantNewest)
|
|
||||||
}
|
|
||||||
|
|
||||||
if !times[wantCount-1].Equal(wantOldest) {
|
|
||||||
t.Errorf("the oldest surviving row's time = %v, want %v",
|
|
||||||
times[wantCount-1], wantOldest)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestOriginChangesSurviveRouteDeletion checks that deleting a routes row
|
|
||||||
// leaves the prefix's origin_changes rows.
|
|
||||||
func TestOriginChangesSurviveRouteDeletion(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
db := newRoutesTestDatabase(t)
|
|
||||||
prefix := netip.MustParsePrefix(tablePrefixV4)
|
|
||||||
|
|
||||||
err := db.WriteOriginChanges([]routetable.OriginChange{{
|
|
||||||
Prefix: prefix, OriginASN: asA, Announced: true, PeerCount: 1,
|
|
||||||
Time: tableTime(10),
|
|
||||||
}})
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("WriteOriginChanges: %v", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
// Write a route, then delete it.
|
|
||||||
table := routetable.New(routetable.Params{})
|
|
||||||
announceTo(table, tablePrefixV4, asA, peerA, 10)
|
|
||||||
writeTableChanges(t, db, table)
|
|
||||||
|
|
||||||
table.Withdraw(prefix, peerA)
|
|
||||||
writeTableChanges(t, db, table)
|
|
||||||
|
|
||||||
got, err := db.GetOriginChanges(t.Context(), prefix, 10)
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("GetOriginChanges: %v", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
if len(got) != 1 {
|
|
||||||
t.Errorf("GetOriginChanges = %d rows, want 1", len(got))
|
|
||||||
}
|
|
||||||
}
|
|
||||||
@@ -34,7 +34,6 @@ const (
|
|||||||
|
|
||||||
tablePrefixV4 = "10.1.2.0/24"
|
tablePrefixV4 = "10.1.2.0/24"
|
||||||
tablePrefixShortV4 = "10.9.0.0/16"
|
tablePrefixShortV4 = "10.9.0.0/16"
|
||||||
tablePrefixBusyV4 = "10.10.0.0/16"
|
|
||||||
tablePrefixV6 = "2001:db8:1::/48"
|
tablePrefixV6 = "2001:db8:1::/48"
|
||||||
)
|
)
|
||||||
|
|
||||||
@@ -108,7 +107,7 @@ func TestReadRouteTableGivesBackWhatWasWritten(t *testing.T) {
|
|||||||
|
|
||||||
// A peer moves to another origin, one goes down and a new one comes.
|
// A peer moves to another origin, one goes down and a new one comes.
|
||||||
announceTo(source, tablePrefixV4, asB, peerA, 60)
|
announceTo(source, tablePrefixV4, asB, peerA, 60)
|
||||||
source.PeersDown([]string{peerC})
|
source.PeerDown(peerC)
|
||||||
announceTo(source, tablePrefixV6, asB, peerE, 70)
|
announceTo(source, tablePrefixV6, asB, peerE, 70)
|
||||||
|
|
||||||
writeTableChanges(t, db, source)
|
writeTableChanges(t, db, source)
|
||||||
@@ -134,7 +133,7 @@ func TestReadRouteTableGivesATableThatTakesLaterUpdates(t *testing.T) {
|
|||||||
// another one goes down, and a new peer comes.
|
// another one goes down, and a new peer comes.
|
||||||
for _, table := range []*routetable.Table{source, read} {
|
for _, table := range []*routetable.Table{source, read} {
|
||||||
table.Withdraw(netip.MustParsePrefix(tablePrefixV4), peerA)
|
table.Withdraw(netip.MustParsePrefix(tablePrefixV4), peerA)
|
||||||
table.PeersDown([]string{peerB})
|
table.PeerDown(peerB)
|
||||||
announceTo(table, tablePrefixShortV4, asB, peerD, 50)
|
announceTo(table, tablePrefixShortV4, asB, peerD, 50)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
@@ -83,9 +83,6 @@ CREATE INDEX IF NOT EXISTS idx_peerings_lookup ON peerings(as_a, as_b);
|
|||||||
-- Indexes for asns table
|
-- Indexes for asns table
|
||||||
CREATE INDEX IF NOT EXISTS idx_asns_asn ON asns(asn);
|
CREATE INDEX IF NOT EXISTS idx_asns_asn ON asns(asn);
|
||||||
CREATE INDEX IF NOT EXISTS idx_asns_whois_updated_at ON asns(whois_updated_at);
|
CREATE INDEX IF NOT EXISTS idx_asns_whois_updated_at ON asns(whois_updated_at);
|
||||||
-- Finds the AS last seen most recently and the one last seen longest ago, for
|
|
||||||
-- the home page.
|
|
||||||
CREATE INDEX IF NOT EXISTS idx_asns_last_seen ON asns(last_seen);
|
|
||||||
|
|
||||||
-- Indexes for bgp_peers table
|
-- Indexes for bgp_peers table
|
||||||
CREATE INDEX IF NOT EXISTS idx_bgp_peers_asn ON bgp_peers(peer_asn);
|
CREATE INDEX IF NOT EXISTS idx_bgp_peers_asn ON bgp_peers(peer_asn);
|
||||||
@@ -119,20 +116,4 @@ CREATE TABLE IF NOT EXISTS routes (
|
|||||||
CREATE TABLE IF NOT EXISTS route_peers (
|
CREATE TABLE IF NOT EXISTS route_peers (
|
||||||
number INTEGER PRIMARY KEY,
|
number INTEGER PRIMARY KEY,
|
||||||
address TEXT NOT NULL UNIQUE
|
address TEXT NOT NULL UNIQUE
|
||||||
);
|
);
|
||||||
|
|
||||||
-- The origin changes of the routing table: each time a prefix and origin AS
|
|
||||||
-- gained its first peer or lost its last. The prefix handler writes them, and
|
|
||||||
-- the database maintainer prunes each prefix to its newest 500 rows or its
|
|
||||||
-- rows from the last 72 hours, whichever keeps more. time is whole seconds
|
|
||||||
-- since 1970, UTC.
|
|
||||||
CREATE TABLE IF NOT EXISTS origin_changes (
|
|
||||||
id INTEGER PRIMARY KEY,
|
|
||||||
prefix TEXT NOT NULL,
|
|
||||||
origin_asn INTEGER NOT NULL,
|
|
||||||
announced BOOLEAN NOT NULL,
|
|
||||||
peer_count INTEGER NOT NULL,
|
|
||||||
time INTEGER NOT NULL
|
|
||||||
);
|
|
||||||
|
|
||||||
CREATE INDEX IF NOT EXISTS idx_origin_changes_prefix ON origin_changes(prefix);
|
|
||||||
@@ -1,253 +0,0 @@
|
|||||||
package geo
|
|
||||||
|
|
||||||
import (
|
|
||||||
"context"
|
|
||||||
"database/sql"
|
|
||||||
"errors"
|
|
||||||
"fmt"
|
|
||||||
"maps"
|
|
||||||
"time"
|
|
||||||
)
|
|
||||||
|
|
||||||
const (
|
|
||||||
// entryLifetime is how long a geojs answer is kept after its fetch.
|
|
||||||
entryLifetime = 7 * 24 * time.Hour
|
|
||||||
|
|
||||||
// flushInterval is how often new answers are written to cache.db.
|
|
||||||
flushInterval = 5 * time.Second
|
|
||||||
|
|
||||||
// sweepInterval is how often expired answers are deleted.
|
|
||||||
sweepInterval = 10 * time.Minute
|
|
||||||
|
|
||||||
// rowColumns are the columns of the geolocations table in the order
|
|
||||||
// scanRow reads them.
|
|
||||||
rowColumns = "address, country_code, country, continent_code, region, " +
|
|
||||||
"city, latitude, longitude, accuracy_radius_km, time_zone, fetched_at"
|
|
||||||
)
|
|
||||||
|
|
||||||
// rowScanner is the Scan method of *sql.Row and *sql.Rows.
|
|
||||||
type rowScanner interface {
|
|
||||||
Scan(dest ...any) error
|
|
||||||
}
|
|
||||||
|
|
||||||
// cached returns the current answer held for address, from memory, then from
|
|
||||||
// the answers waiting to be written, then from cache.db. It returns nil when
|
|
||||||
// there is none.
|
|
||||||
func (l *Locator) cached(ctx context.Context, address string) *Location {
|
|
||||||
found, ok := l.entries.Get(address)
|
|
||||||
if ok {
|
|
||||||
if !l.expired(found) {
|
|
||||||
return found
|
|
||||||
}
|
|
||||||
|
|
||||||
l.entries.Remove(address)
|
|
||||||
}
|
|
||||||
|
|
||||||
l.pendingMu.Lock()
|
|
||||||
waiting := l.pending[address]
|
|
||||||
l.pendingMu.Unlock()
|
|
||||||
|
|
||||||
if waiting != nil && !l.expired(waiting) {
|
|
||||||
l.entries.Add(address, waiting)
|
|
||||||
|
|
||||||
return waiting
|
|
||||||
}
|
|
||||||
|
|
||||||
stored, err := l.readRow(ctx, address)
|
|
||||||
if err != nil {
|
|
||||||
l.logger.Warn("Failed to read a geolocation cache row",
|
|
||||||
"address", address, "error", err)
|
|
||||||
|
|
||||||
return nil
|
|
||||||
}
|
|
||||||
|
|
||||||
if stored != nil {
|
|
||||||
l.entries.Add(address, stored)
|
|
||||||
}
|
|
||||||
|
|
||||||
return stored
|
|
||||||
}
|
|
||||||
|
|
||||||
// expired says whether the answer is at the end of its lifetime.
|
|
||||||
func (l *Locator) expired(answer *Location) bool {
|
|
||||||
return !l.now().Before(answer.Updated.Add(entryLifetime))
|
|
||||||
}
|
|
||||||
|
|
||||||
// cutoff returns the fetch time at or before which an answer has expired.
|
|
||||||
func (l *Locator) cutoff() int64 {
|
|
||||||
return l.now().Add(-entryLifetime).Unix()
|
|
||||||
}
|
|
||||||
|
|
||||||
// store keeps an answer in memory and queues it for cache.db. While cache.db
|
|
||||||
// cannot be written, the queue stops growing at maxEntries answers.
|
|
||||||
func (l *Locator) store(address string, answer *Location) {
|
|
||||||
l.entries.Add(address, answer)
|
|
||||||
|
|
||||||
l.pendingMu.Lock()
|
|
||||||
defer l.pendingMu.Unlock()
|
|
||||||
|
|
||||||
if len(l.pending) < l.maxEntries {
|
|
||||||
l.pending[address] = answer
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// keepCache writes new answers and deletes expired ones until ctx ends.
|
|
||||||
func (l *Locator) keepCache(ctx context.Context) {
|
|
||||||
flushes := time.NewTicker(flushInterval)
|
|
||||||
defer flushes.Stop()
|
|
||||||
|
|
||||||
sweeps := time.NewTicker(sweepInterval)
|
|
||||||
defer sweeps.Stop()
|
|
||||||
|
|
||||||
for {
|
|
||||||
select {
|
|
||||||
case <-ctx.Done():
|
|
||||||
l.flush(context.WithoutCancel(ctx))
|
|
||||||
|
|
||||||
return
|
|
||||||
case <-flushes.C:
|
|
||||||
l.flush(ctx)
|
|
||||||
case <-sweeps.C:
|
|
||||||
l.sweep(ctx)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// load reads the answers that have not expired into memory, at most maxEntries
|
|
||||||
// of them, the newest last so that they are the last to be dropped.
|
|
||||||
func (l *Locator) load(ctx context.Context) error {
|
|
||||||
rows, err := l.db.QueryContext(ctx,
|
|
||||||
"SELECT "+rowColumns+" FROM (SELECT "+rowColumns+" FROM geolocations "+
|
|
||||||
"WHERE fetched_at > ? ORDER BY fetched_at DESC LIMIT ?) "+
|
|
||||||
"ORDER BY fetched_at",
|
|
||||||
l.cutoff(), l.maxEntries)
|
|
||||||
if err != nil {
|
|
||||||
return fmt.Errorf("reading the geolocation cache: %w", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
defer func() { _ = rows.Close() }()
|
|
||||||
|
|
||||||
for rows.Next() {
|
|
||||||
address, answer, err := scanRow(rows)
|
|
||||||
if err != nil {
|
|
||||||
return fmt.Errorf("reading a geolocation cache row: %w", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
l.entries.Add(address, answer)
|
|
||||||
}
|
|
||||||
|
|
||||||
err = rows.Err()
|
|
||||||
if err != nil {
|
|
||||||
return fmt.Errorf("reading the geolocation cache: %w", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
return nil
|
|
||||||
}
|
|
||||||
|
|
||||||
// readRow returns the current answer for address in cache.db, or nil when it
|
|
||||||
// has none.
|
|
||||||
func (l *Locator) readRow(
|
|
||||||
ctx context.Context, address string,
|
|
||||||
) (*Location, error) {
|
|
||||||
row := l.db.QueryRowContext(ctx,
|
|
||||||
"SELECT "+rowColumns+" FROM geolocations "+
|
|
||||||
"WHERE address = ? AND fetched_at > ?",
|
|
||||||
address, l.cutoff())
|
|
||||||
|
|
||||||
_, answer, err := scanRow(row)
|
|
||||||
if errors.Is(err, sql.ErrNoRows) {
|
|
||||||
return nil, nil //nolint:nilnil // no row is not a failure
|
|
||||||
}
|
|
||||||
|
|
||||||
return answer, err
|
|
||||||
}
|
|
||||||
|
|
||||||
// scanRow reads one row of the geolocations table.
|
|
||||||
func scanRow(row rowScanner) (string, *Location, error) {
|
|
||||||
var (
|
|
||||||
address string
|
|
||||||
fetched int64
|
|
||||||
answer Location
|
|
||||||
)
|
|
||||||
|
|
||||||
err := row.Scan(&address, &answer.CountryCode, &answer.Country,
|
|
||||||
&answer.ContinentCode, &answer.Region, &answer.City, &answer.Latitude,
|
|
||||||
&answer.Longitude, &answer.AccuracyRadiusKm, &answer.TimeZone, &fetched)
|
|
||||||
if err != nil {
|
|
||||||
return "", nil, err
|
|
||||||
}
|
|
||||||
|
|
||||||
answer.Source = sourceGeoJS
|
|
||||||
answer.Updated = time.Unix(fetched, 0).UTC()
|
|
||||||
|
|
||||||
return address, &answer, nil
|
|
||||||
}
|
|
||||||
|
|
||||||
// flush writes the pending answers to cache.db. Those it cannot write stay
|
|
||||||
// pending, unless a newer answer for the address has arrived meanwhile.
|
|
||||||
func (l *Locator) flush(ctx context.Context) {
|
|
||||||
l.pendingMu.Lock()
|
|
||||||
batch := l.pending
|
|
||||||
l.pending = make(map[string]*Location)
|
|
||||||
l.pendingMu.Unlock()
|
|
||||||
|
|
||||||
if len(batch) == 0 {
|
|
||||||
return
|
|
||||||
}
|
|
||||||
|
|
||||||
err := l.writeRows(ctx, batch)
|
|
||||||
if err == nil {
|
|
||||||
return
|
|
||||||
}
|
|
||||||
|
|
||||||
l.logger.Error("Failed to write the geolocation cache", "error", err)
|
|
||||||
|
|
||||||
l.pendingMu.Lock()
|
|
||||||
defer l.pendingMu.Unlock()
|
|
||||||
|
|
||||||
maps.Insert(batch, maps.All(l.pending))
|
|
||||||
l.pending = batch
|
|
||||||
}
|
|
||||||
|
|
||||||
// writeRows writes batch to cache.db in one transaction.
|
|
||||||
func (l *Locator) writeRows(
|
|
||||||
ctx context.Context, batch map[string]*Location,
|
|
||||||
) error {
|
|
||||||
tx, err := l.db.BeginTx(ctx, nil)
|
|
||||||
if err != nil {
|
|
||||||
return err
|
|
||||||
}
|
|
||||||
|
|
||||||
defer func() { _ = tx.Rollback() }()
|
|
||||||
|
|
||||||
for address, answer := range batch {
|
|
||||||
_, err = tx.ExecContext(ctx,
|
|
||||||
"INSERT OR REPLACE INTO geolocations ("+rowColumns+") "+
|
|
||||||
"VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)",
|
|
||||||
address, answer.CountryCode, answer.Country, answer.ContinentCode,
|
|
||||||
answer.Region, answer.City, answer.Latitude, answer.Longitude,
|
|
||||||
answer.AccuracyRadiusKm, answer.TimeZone, answer.Updated.Unix())
|
|
||||||
if err != nil {
|
|
||||||
return err
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
return tx.Commit()
|
|
||||||
}
|
|
||||||
|
|
||||||
// sweep deletes the expired answers from memory and cache.db.
|
|
||||||
func (l *Locator) sweep(ctx context.Context) {
|
|
||||||
for _, address := range l.entries.Keys() {
|
|
||||||
answer, ok := l.entries.Peek(address)
|
|
||||||
if ok && l.expired(answer) {
|
|
||||||
l.entries.Remove(address)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
_, err := l.db.ExecContext(ctx,
|
|
||||||
"DELETE FROM geolocations WHERE fetched_at <= ?", l.cutoff())
|
|
||||||
if err != nil {
|
|
||||||
l.logger.Error("Failed to delete expired geolocation cache rows",
|
|
||||||
"error", err)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
@@ -1,349 +0,0 @@
|
|||||||
package geo
|
|
||||||
|
|
||||||
import (
|
|
||||||
"compress/gzip"
|
|
||||||
"context"
|
|
||||||
"errors"
|
|
||||||
"fmt"
|
|
||||||
"io"
|
|
||||||
"io/fs"
|
|
||||||
"net/http"
|
|
||||||
"net/netip"
|
|
||||||
"os"
|
|
||||||
"path/filepath"
|
|
||||||
"time"
|
|
||||||
|
|
||||||
"github.com/oschwald/maxminddb-golang/v2"
|
|
||||||
)
|
|
||||||
|
|
||||||
const (
|
|
||||||
// DatabaseDownloadFormat is the URL of DB-IP's City Lite database, with %s
|
|
||||||
// for the year and month.
|
|
||||||
DatabaseDownloadFormat = "https://download.db-ip.com/free/" +
|
|
||||||
"dbip-city-lite-%s.mmdb.gz"
|
|
||||||
|
|
||||||
// databaseMonthFormat is the year and month in the URL.
|
|
||||||
databaseMonthFormat = "2006-01"
|
|
||||||
|
|
||||||
// databaseFileName is the copy of the database in the state directory.
|
|
||||||
databaseFileName = "dbip-city-lite.mmdb"
|
|
||||||
|
|
||||||
// databaseDownloadSuffix ends the name of the file a download is written
|
|
||||||
// to, until it has opened.
|
|
||||||
databaseDownloadSuffix = ".download"
|
|
||||||
|
|
||||||
// maxDatabaseSize is the largest database accepted once unpacked, in
|
|
||||||
// bytes.
|
|
||||||
maxDatabaseSize = 200 << 20
|
|
||||||
|
|
||||||
// databaseDownloadTimeout limits one download.
|
|
||||||
databaseDownloadTimeout = 10 * time.Minute
|
|
||||||
|
|
||||||
// databaseCheckInterval is how often RouteWatch checks whether a new
|
|
||||||
// month's database is due. A new month's file appears some time on the
|
|
||||||
// first of the month, so a failed download is tried again this much later.
|
|
||||||
databaseCheckInterval = time.Hour
|
|
||||||
)
|
|
||||||
|
|
||||||
var (
|
|
||||||
errDatabaseStatus = errors.New("unexpected HTTP status from DB-IP")
|
|
||||||
errDatabaseTooLong = errors.New("database is over the size limit")
|
|
||||||
)
|
|
||||||
|
|
||||||
// dbipName is the part of a database record that holds an English name.
|
|
||||||
type dbipName struct {
|
|
||||||
Names struct {
|
|
||||||
English string `maxminddb:"en"`
|
|
||||||
} `maxminddb:"names"`
|
|
||||||
}
|
|
||||||
|
|
||||||
// dbipRecord is the part of a database record RouteWatch reads.
|
|
||||||
type dbipRecord struct {
|
|
||||||
City dbipName `maxminddb:"city"`
|
|
||||||
Continent struct {
|
|
||||||
Code string `maxminddb:"code"`
|
|
||||||
} `maxminddb:"continent"`
|
|
||||||
Country struct {
|
|
||||||
dbipName
|
|
||||||
|
|
||||||
ISOCode string `maxminddb:"iso_code"`
|
|
||||||
} `maxminddb:"country"`
|
|
||||||
Location struct {
|
|
||||||
Latitude *float64 `maxminddb:"latitude"`
|
|
||||||
Longitude *float64 `maxminddb:"longitude"`
|
|
||||||
} `maxminddb:"location"`
|
|
||||||
Subdivisions []dbipName `maxminddb:"subdivisions"`
|
|
||||||
}
|
|
||||||
|
|
||||||
// location returns the record as a Location fetched at the time given. The
|
|
||||||
// first subdivision is the region. The database has no accuracy radius and no
|
|
||||||
// time zone.
|
|
||||||
func (r *dbipRecord) location(fetched time.Time) *Location {
|
|
||||||
found := &Location{
|
|
||||||
CountryCode: nilIfEmpty(r.Country.ISOCode),
|
|
||||||
Country: nilIfEmpty(r.Country.Names.English),
|
|
||||||
ContinentCode: nilIfEmpty(r.Continent.Code),
|
|
||||||
City: nilIfEmpty(r.City.Names.English),
|
|
||||||
Latitude: r.Location.Latitude,
|
|
||||||
Longitude: r.Location.Longitude,
|
|
||||||
Source: sourceDBIP,
|
|
||||||
Updated: fetched,
|
|
||||||
}
|
|
||||||
|
|
||||||
if len(r.Subdivisions) > 0 {
|
|
||||||
found.Region = nilIfEmpty(r.Subdivisions[0].Names.English)
|
|
||||||
}
|
|
||||||
|
|
||||||
return found
|
|
||||||
}
|
|
||||||
|
|
||||||
// nilIfEmpty returns a pointer to text, or nil when text is empty.
|
|
||||||
func nilIfEmpty(text string) *string {
|
|
||||||
if text == "" {
|
|
||||||
return nil
|
|
||||||
}
|
|
||||||
|
|
||||||
return &text
|
|
||||||
}
|
|
||||||
|
|
||||||
// fromDatabase returns where the copy of the database places addr. It returns
|
|
||||||
// nil when there is no copy yet or the copy has no record for addr. It also
|
|
||||||
// returns nil, and logs the error, when reading the copy fails.
|
|
||||||
func (l *Locator) fromDatabase(addr netip.Addr) *Location {
|
|
||||||
l.databaseMu.RLock()
|
|
||||||
defer l.databaseMu.RUnlock()
|
|
||||||
|
|
||||||
if l.database == nil {
|
|
||||||
return nil
|
|
||||||
}
|
|
||||||
|
|
||||||
result := l.database.Lookup(addr)
|
|
||||||
|
|
||||||
err := result.Err()
|
|
||||||
if err != nil {
|
|
||||||
l.logger.Error("Failed to look up an address in the database",
|
|
||||||
"address", addr, "error", err)
|
|
||||||
|
|
||||||
return nil
|
|
||||||
}
|
|
||||||
|
|
||||||
if !result.Found() {
|
|
||||||
return nil
|
|
||||||
}
|
|
||||||
|
|
||||||
var record dbipRecord
|
|
||||||
|
|
||||||
err = result.Decode(&record)
|
|
||||||
if err != nil {
|
|
||||||
l.logger.Error("Failed to read a database record",
|
|
||||||
"address", addr, "error", err)
|
|
||||||
|
|
||||||
return nil
|
|
||||||
}
|
|
||||||
|
|
||||||
return record.location(l.databaseFetched)
|
|
||||||
}
|
|
||||||
|
|
||||||
// keepDatabaseCurrent downloads the database when it is due, at once and then
|
|
||||||
// every databaseCheckInterval, until ctx ends.
|
|
||||||
func (l *Locator) keepDatabaseCurrent(ctx context.Context) {
|
|
||||||
l.downloadDatabaseIfDue(ctx)
|
|
||||||
|
|
||||||
checks := time.NewTicker(databaseCheckInterval)
|
|
||||||
defer checks.Stop()
|
|
||||||
|
|
||||||
for {
|
|
||||||
select {
|
|
||||||
case <-ctx.Done():
|
|
||||||
return
|
|
||||||
case <-checks.C:
|
|
||||||
l.downloadDatabaseIfDue(ctx)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// downloadDatabaseIfDue downloads this month's database, unless the copy in
|
|
||||||
// use was downloaded this month. A failure leaves the copy as it is.
|
|
||||||
func (l *Locator) downloadDatabaseIfDue(ctx context.Context) {
|
|
||||||
now := l.now().UTC()
|
|
||||||
|
|
||||||
if !l.databaseDue(now) {
|
|
||||||
return
|
|
||||||
}
|
|
||||||
|
|
||||||
err := l.downloadDatabase(ctx, now)
|
|
||||||
if err != nil && ctx.Err() == nil {
|
|
||||||
l.logger.Error("Database download failed; keeping the previous copy",
|
|
||||||
"error", err)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// databaseDue says whether there is no open copy or it is from an earlier
|
|
||||||
// month than now.
|
|
||||||
func (l *Locator) databaseDue(now time.Time) bool {
|
|
||||||
l.databaseMu.RLock()
|
|
||||||
defer l.databaseMu.RUnlock()
|
|
||||||
|
|
||||||
return l.database == nil ||
|
|
||||||
l.databaseFetched.Format(databaseMonthFormat) !=
|
|
||||||
now.Format(databaseMonthFormat)
|
|
||||||
}
|
|
||||||
|
|
||||||
// downloadDatabase fetches the database of now's month. The file replaces the
|
|
||||||
// copy, on disk and in use, only once it has opened.
|
|
||||||
func (l *Locator) downloadDatabase(ctx context.Context, now time.Time) error {
|
|
||||||
ctx, cancel := context.WithTimeout(ctx, databaseDownloadTimeout)
|
|
||||||
defer cancel()
|
|
||||||
|
|
||||||
url := fmt.Sprintf(l.databaseURLFormat, now.Format(databaseMonthFormat))
|
|
||||||
|
|
||||||
request, err := http.NewRequestWithContext(ctx, http.MethodGet, url, nil)
|
|
||||||
if err != nil {
|
|
||||||
return err
|
|
||||||
}
|
|
||||||
|
|
||||||
request.Header.Set(userAgentHeader, userAgent)
|
|
||||||
|
|
||||||
response, err := l.client.Do(request)
|
|
||||||
if err != nil {
|
|
||||||
return err
|
|
||||||
}
|
|
||||||
|
|
||||||
defer func() { _ = response.Body.Close() }()
|
|
||||||
|
|
||||||
if response.StatusCode != http.StatusOK {
|
|
||||||
return fmt.Errorf("%w: %s", errDatabaseStatus, response.Status)
|
|
||||||
}
|
|
||||||
|
|
||||||
return l.replaceDatabase(response.Body, now)
|
|
||||||
}
|
|
||||||
|
|
||||||
// replaceDatabase unpacks the gzip stream body beside the copy, then makes the
|
|
||||||
// result the copy once it has opened. Whatever fails, the file it wrote is
|
|
||||||
// removed and the copy stays.
|
|
||||||
func (l *Locator) replaceDatabase(body io.Reader, now time.Time) error {
|
|
||||||
root, err := os.OpenRoot(l.dir)
|
|
||||||
if err != nil {
|
|
||||||
return err
|
|
||||||
}
|
|
||||||
|
|
||||||
defer func() { _ = root.Close() }()
|
|
||||||
|
|
||||||
downloadName := databaseFileName + databaseDownloadSuffix
|
|
||||||
|
|
||||||
err = unpackDatabase(root, downloadName, body, l.maxDatabaseBytes)
|
|
||||||
if err != nil {
|
|
||||||
_ = root.Remove(downloadName)
|
|
||||||
|
|
||||||
return err
|
|
||||||
}
|
|
||||||
|
|
||||||
opened, err := maxminddb.Open(filepath.Join(l.dir, downloadName))
|
|
||||||
if err != nil {
|
|
||||||
_ = root.Remove(downloadName)
|
|
||||||
|
|
||||||
return fmt.Errorf("opening the downloaded database: %w", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
err = root.Rename(downloadName, databaseFileName)
|
|
||||||
if err != nil {
|
|
||||||
_ = opened.Close()
|
|
||||||
_ = root.Remove(downloadName)
|
|
||||||
|
|
||||||
return err
|
|
||||||
}
|
|
||||||
|
|
||||||
l.useDatabase(opened, now.Truncate(time.Second))
|
|
||||||
l.logger.Info("Downloaded a new copy of the database")
|
|
||||||
|
|
||||||
return nil
|
|
||||||
}
|
|
||||||
|
|
||||||
// unpackDatabase writes the gzip stream body to the file name in root, up to
|
|
||||||
// limit bytes once unpacked.
|
|
||||||
func unpackDatabase(
|
|
||||||
root *os.Root, name string, body io.Reader, limit int64,
|
|
||||||
) error {
|
|
||||||
unpacked, err := gzip.NewReader(body)
|
|
||||||
if err != nil {
|
|
||||||
return fmt.Errorf("unpacking the database: %w", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
f, err := root.Create(name)
|
|
||||||
if err != nil {
|
|
||||||
return err
|
|
||||||
}
|
|
||||||
|
|
||||||
defer func() { _ = f.Close() }()
|
|
||||||
|
|
||||||
written, err := io.Copy(f, io.LimitReader(unpacked, limit+1))
|
|
||||||
if err != nil {
|
|
||||||
return fmt.Errorf("unpacking the database: %w", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
if written > limit {
|
|
||||||
return errDatabaseTooLong
|
|
||||||
}
|
|
||||||
|
|
||||||
return nil
|
|
||||||
}
|
|
||||||
|
|
||||||
// openDatabaseCopy opens the copy left by an earlier run, if there is one. A
|
|
||||||
// copy that does not open is treated as no copy, so it is downloaded again.
|
|
||||||
func (l *Locator) openDatabaseCopy() {
|
|
||||||
path := filepath.Join(l.dir, databaseFileName)
|
|
||||||
|
|
||||||
info, err := os.Stat(path)
|
|
||||||
if errors.Is(err, fs.ErrNotExist) {
|
|
||||||
return
|
|
||||||
}
|
|
||||||
|
|
||||||
if err != nil {
|
|
||||||
l.logger.Error("Failed to read the database copy", "error", err)
|
|
||||||
|
|
||||||
return
|
|
||||||
}
|
|
||||||
|
|
||||||
opened, err := maxminddb.Open(path)
|
|
||||||
if err != nil {
|
|
||||||
l.logger.Error("Failed to open the database copy", "error", err)
|
|
||||||
|
|
||||||
return
|
|
||||||
}
|
|
||||||
|
|
||||||
l.useDatabase(opened, info.ModTime().UTC().Truncate(time.Second))
|
|
||||||
}
|
|
||||||
|
|
||||||
// useDatabase makes opened the copy in use and closes the one it replaces.
|
|
||||||
func (l *Locator) useDatabase(opened *maxminddb.Reader, fetched time.Time) {
|
|
||||||
l.databaseMu.Lock()
|
|
||||||
defer l.databaseMu.Unlock()
|
|
||||||
|
|
||||||
if l.database != nil {
|
|
||||||
err := l.database.Close()
|
|
||||||
if err != nil {
|
|
||||||
l.logger.Error("Failed to close the old database copy", "error", err)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
l.database = opened
|
|
||||||
l.databaseFetched = fetched
|
|
||||||
}
|
|
||||||
|
|
||||||
// closeDatabase closes the copy in use, after which no source can answer from
|
|
||||||
// it.
|
|
||||||
func (l *Locator) closeDatabase() {
|
|
||||||
l.databaseMu.Lock()
|
|
||||||
defer l.databaseMu.Unlock()
|
|
||||||
|
|
||||||
if l.database == nil {
|
|
||||||
return
|
|
||||||
}
|
|
||||||
|
|
||||||
err := l.database.Close()
|
|
||||||
if err != nil {
|
|
||||||
l.logger.Error("Failed to close the database copy", "error", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
l.database = nil
|
|
||||||
}
|
|
||||||
@@ -1,151 +0,0 @@
|
|||||||
package geo
|
|
||||||
|
|
||||||
import (
|
|
||||||
"context"
|
|
||||||
"encoding/json"
|
|
||||||
"errors"
|
|
||||||
"fmt"
|
|
||||||
"io"
|
|
||||||
"math"
|
|
||||||
"net/http"
|
|
||||||
"net/netip"
|
|
||||||
"strconv"
|
|
||||||
"strings"
|
|
||||||
"time"
|
|
||||||
)
|
|
||||||
|
|
||||||
const (
|
|
||||||
// GeoJSEndpoint is the URL of the geojs geo endpoint.
|
|
||||||
GeoJSEndpoint = "https://get.geojs.io/v1/ip/geo.json"
|
|
||||||
|
|
||||||
// geojsTimeLimit limits one request to geojs.
|
|
||||||
geojsTimeLimit = time.Second
|
|
||||||
|
|
||||||
// geojsAddressSeparator separates the addresses of one request.
|
|
||||||
geojsAddressSeparator = ","
|
|
||||||
|
|
||||||
// geojsUnknown is what geojs sends for a value it does not know.
|
|
||||||
geojsUnknown = "nil"
|
|
||||||
|
|
||||||
// maxGeoJSAnswerBytes is far more than an answer for the most addresses
|
|
||||||
// an IP lookup can ask about.
|
|
||||||
maxGeoJSAnswerBytes = 1 << 20
|
|
||||||
)
|
|
||||||
|
|
||||||
var errGeoJSStatus = errors.New("unexpected HTTP status from geojs")
|
|
||||||
|
|
||||||
// geojsAnswer is one address's entry in an answer from geojs.
|
|
||||||
//
|
|
||||||
//nolint:tagliatelle // the keys geojs sends
|
|
||||||
type geojsAnswer struct {
|
|
||||||
IP string `json:"ip"`
|
|
||||||
CountryCode string `json:"country_code"`
|
|
||||||
Country string `json:"country"`
|
|
||||||
ContinentCode string `json:"continent_code"`
|
|
||||||
Region string `json:"region"`
|
|
||||||
City string `json:"city"`
|
|
||||||
Latitude string `json:"latitude"`
|
|
||||||
Longitude string `json:"longitude"`
|
|
||||||
Accuracy *int `json:"accuracy"`
|
|
||||||
TimeZone string `json:"timezone"`
|
|
||||||
}
|
|
||||||
|
|
||||||
// askGeoJS asks geojs about addresses in one request, and returns its answers
|
|
||||||
// by address. An address geojs left out, or gave no country, city or
|
|
||||||
// coordinates for, has no entry.
|
|
||||||
func (l *Locator) askGeoJS(
|
|
||||||
ctx context.Context, addresses []string,
|
|
||||||
) (map[string]*Location, error) {
|
|
||||||
ctx, cancel := context.WithTimeout(ctx, l.timeLimit)
|
|
||||||
defer cancel()
|
|
||||||
|
|
||||||
// A comma and a colon are legal in a query, and the escaped forms are
|
|
||||||
// not what geojs documents.
|
|
||||||
url := l.geojsURL + "?ip=" + strings.Join(addresses, geojsAddressSeparator)
|
|
||||||
|
|
||||||
request, err := http.NewRequestWithContext(ctx, http.MethodGet, url, nil)
|
|
||||||
if err != nil {
|
|
||||||
return nil, err
|
|
||||||
}
|
|
||||||
|
|
||||||
request.Header.Set(userAgentHeader, userAgent)
|
|
||||||
|
|
||||||
response, err := l.client.Do(request)
|
|
||||||
if err != nil {
|
|
||||||
return nil, err
|
|
||||||
}
|
|
||||||
|
|
||||||
defer func() { _ = response.Body.Close() }()
|
|
||||||
|
|
||||||
if response.StatusCode != http.StatusOK {
|
|
||||||
return nil, fmt.Errorf("%w: %d", errGeoJSStatus, response.StatusCode)
|
|
||||||
}
|
|
||||||
|
|
||||||
var parsed []geojsAnswer
|
|
||||||
|
|
||||||
err = json.NewDecoder(io.LimitReader(response.Body, maxGeoJSAnswerBytes)).
|
|
||||||
Decode(&parsed)
|
|
||||||
if err != nil {
|
|
||||||
return nil, fmt.Errorf("reading the geojs answer: %w", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
fetched := l.now().UTC().Truncate(time.Second)
|
|
||||||
answers := make(map[string]*Location, len(parsed))
|
|
||||||
|
|
||||||
for _, answer := range parsed {
|
|
||||||
addr, err := netip.ParseAddr(answer.IP)
|
|
||||||
if err != nil {
|
|
||||||
continue
|
|
||||||
}
|
|
||||||
|
|
||||||
location := answer.location(fetched)
|
|
||||||
if hasLocation(location) {
|
|
||||||
answers[plainAddress(addr)] = location
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
return answers, nil
|
|
||||||
}
|
|
||||||
|
|
||||||
// hasLocation says whether found has a country, a city or a coordinate.
|
|
||||||
func hasLocation(found *Location) bool {
|
|
||||||
return found.CountryCode != nil || found.Country != nil ||
|
|
||||||
found.City != nil || found.Latitude != nil || found.Longitude != nil
|
|
||||||
}
|
|
||||||
|
|
||||||
// location returns the answer as a Location fetched at the time given.
|
|
||||||
func (a geojsAnswer) location(fetched time.Time) *Location {
|
|
||||||
return &Location{
|
|
||||||
CountryCode: knownText(a.CountryCode),
|
|
||||||
Country: knownText(a.Country),
|
|
||||||
ContinentCode: knownText(a.ContinentCode),
|
|
||||||
Region: knownText(a.Region),
|
|
||||||
City: knownText(a.City),
|
|
||||||
Latitude: knownNumber(a.Latitude),
|
|
||||||
Longitude: knownNumber(a.Longitude),
|
|
||||||
AccuracyRadiusKm: a.Accuracy,
|
|
||||||
TimeZone: knownText(a.TimeZone),
|
|
||||||
Source: sourceGeoJS,
|
|
||||||
Updated: fetched,
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// knownText returns a pointer to text, or nil when geojs does not know it.
|
|
||||||
func knownText(text string) *string {
|
|
||||||
if text == "" || text == geojsUnknown {
|
|
||||||
return nil
|
|
||||||
}
|
|
||||||
|
|
||||||
return &text
|
|
||||||
}
|
|
||||||
|
|
||||||
// knownNumber returns text as a number, or nil when geojs does not know it. NaN
|
|
||||||
// and infinity parse, but JSON cannot carry them.
|
|
||||||
func knownNumber(text string) *float64 {
|
|
||||||
number, err := strconv.ParseFloat(text, 64)
|
|
||||||
if err != nil || math.IsNaN(number) || math.IsInf(number, 0) {
|
|
||||||
return nil
|
|
||||||
}
|
|
||||||
|
|
||||||
return &number
|
|
||||||
}
|
|
||||||
@@ -1,232 +0,0 @@
|
|||||||
// Package geo finds where IP addresses are. It asks geojs.io, and keeps those
|
|
||||||
// answers in memory and in cache.db. When geojs.io gives no location for an
|
|
||||||
// address, it reads a copy of DB-IP's City Lite database, which it downloads
|
|
||||||
// once a month.
|
|
||||||
package geo
|
|
||||||
|
|
||||||
import (
|
|
||||||
"context"
|
|
||||||
"database/sql"
|
|
||||||
"fmt"
|
|
||||||
"log/slog"
|
|
||||||
"net/http"
|
|
||||||
"net/netip"
|
|
||||||
"sync"
|
|
||||||
"time"
|
|
||||||
|
|
||||||
lru "github.com/hashicorp/golang-lru/v2"
|
|
||||||
"github.com/oschwald/maxminddb-golang/v2"
|
|
||||||
)
|
|
||||||
|
|
||||||
// The values of Location.Source.
|
|
||||||
const (
|
|
||||||
sourceGeoJS = "geojs"
|
|
||||||
sourceDBIP = "dbip"
|
|
||||||
)
|
|
||||||
|
|
||||||
const (
|
|
||||||
// maxEntries limits how many geojs answers are held in memory.
|
|
||||||
maxEntries = 250_000
|
|
||||||
|
|
||||||
// userAgentHeader and userAgent name routewatch to geojs and DB-IP.
|
|
||||||
userAgentHeader = "User-Agent"
|
|
||||||
userAgent = "routewatch (+https://git.eeqj.de/sneak/routewatch)"
|
|
||||||
)
|
|
||||||
|
|
||||||
// Params is what New needs.
|
|
||||||
type Params struct {
|
|
||||||
// DB is the open cache.db.
|
|
||||||
DB *sql.DB
|
|
||||||
|
|
||||||
// Dir is the directory that holds the copy of the database.
|
|
||||||
Dir string
|
|
||||||
|
|
||||||
// GeoJSURL is the URL of the geojs geo endpoint, GeoJSEndpoint in use.
|
|
||||||
GeoJSURL string
|
|
||||||
|
|
||||||
// DatabaseURLFormat is the URL of the database with %s for the year and
|
|
||||||
// month, DatabaseDownloadFormat in use.
|
|
||||||
DatabaseURLFormat string
|
|
||||||
|
|
||||||
Logger *slog.Logger
|
|
||||||
}
|
|
||||||
|
|
||||||
// Location is where an address is. A part the source does not know is nil.
|
|
||||||
//
|
|
||||||
//nolint:tagliatelle // snake_case keys, as the Go style guide asks
|
|
||||||
type Location struct {
|
|
||||||
CountryCode *string `json:"country_code"`
|
|
||||||
Country *string `json:"country"`
|
|
||||||
ContinentCode *string `json:"continent_code"`
|
|
||||||
Region *string `json:"region"`
|
|
||||||
City *string `json:"city"`
|
|
||||||
Latitude *float64 `json:"latitude"`
|
|
||||||
Longitude *float64 `json:"longitude"`
|
|
||||||
AccuracyRadiusKm *int `json:"accuracy_radius_km"`
|
|
||||||
TimeZone *string `json:"time_zone"`
|
|
||||||
|
|
||||||
// Source is geojs or dbip.
|
|
||||||
Source string `json:"source"`
|
|
||||||
|
|
||||||
// Updated is when the value was fetched: the geojs answer, or the copy of
|
|
||||||
// the database.
|
|
||||||
Updated time.Time `json:"updated"`
|
|
||||||
}
|
|
||||||
|
|
||||||
// Locator answers where addresses are. Start loads its cache and starts its
|
|
||||||
// goroutines.
|
|
||||||
type Locator struct {
|
|
||||||
db *sql.DB
|
|
||||||
dir string
|
|
||||||
geojsURL string
|
|
||||||
databaseURLFormat string
|
|
||||||
timeLimit time.Duration
|
|
||||||
maxEntries int
|
|
||||||
maxDatabaseBytes int64
|
|
||||||
logger *slog.Logger
|
|
||||||
client *http.Client
|
|
||||||
now func() time.Time
|
|
||||||
|
|
||||||
// entries holds geojs answers by address, and drops the least recently
|
|
||||||
// used first.
|
|
||||||
entries *lru.Cache[string, *Location]
|
|
||||||
|
|
||||||
// pending holds the answers that cache.db does not have yet.
|
|
||||||
pendingMu sync.Mutex
|
|
||||||
pending map[string]*Location
|
|
||||||
|
|
||||||
// database is the open copy, nil until one has opened, and databaseFetched
|
|
||||||
// is when that copy was downloaded. A lookup holds the read lock, so that
|
|
||||||
// a new copy never closes the old one under it.
|
|
||||||
databaseMu sync.RWMutex
|
|
||||||
database *maxminddb.Reader
|
|
||||||
databaseFetched time.Time
|
|
||||||
|
|
||||||
// stop ends the goroutines Start runs, and done is closed when they have
|
|
||||||
// returned.
|
|
||||||
stop context.CancelFunc
|
|
||||||
done chan struct{}
|
|
||||||
}
|
|
||||||
|
|
||||||
// New returns a Locator with an empty cache. Start loads it.
|
|
||||||
func New(params Params) (*Locator, error) {
|
|
||||||
entries, err := lru.New[string, *Location](maxEntries)
|
|
||||||
if err != nil {
|
|
||||||
return nil, fmt.Errorf("creating the geolocation cache: %w", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
return &Locator{
|
|
||||||
db: params.DB,
|
|
||||||
dir: params.Dir,
|
|
||||||
geojsURL: params.GeoJSURL,
|
|
||||||
databaseURLFormat: params.DatabaseURLFormat,
|
|
||||||
timeLimit: geojsTimeLimit,
|
|
||||||
maxEntries: maxEntries,
|
|
||||||
maxDatabaseBytes: maxDatabaseSize,
|
|
||||||
logger: params.Logger.With("component", "geo"),
|
|
||||||
client: &http.Client{},
|
|
||||||
now: time.Now,
|
|
||||||
entries: entries,
|
|
||||||
pending: make(map[string]*Location),
|
|
||||||
done: make(chan struct{}),
|
|
||||||
}, nil
|
|
||||||
}
|
|
||||||
|
|
||||||
// Lookup returns where each of addrs is, in the same order. The entry for an
|
|
||||||
// address neither source answers for is nil. Addresses with no current cached
|
|
||||||
// answer go to geojs in one request. The returned Locations are shared with
|
|
||||||
// the cache and must not be changed.
|
|
||||||
func (l *Locator) Lookup(ctx context.Context, addrs []netip.Addr) []*Location {
|
|
||||||
locations := make([]*Location, len(addrs))
|
|
||||||
|
|
||||||
var unanswered []string
|
|
||||||
|
|
||||||
for i, addr := range addrs {
|
|
||||||
address := plainAddress(addr)
|
|
||||||
|
|
||||||
locations[i] = l.cached(ctx, address)
|
|
||||||
if locations[i] == nil {
|
|
||||||
unanswered = append(unanswered, address)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
if len(unanswered) == 0 {
|
|
||||||
return locations
|
|
||||||
}
|
|
||||||
|
|
||||||
answers, err := l.askGeoJS(ctx, unanswered)
|
|
||||||
if err != nil {
|
|
||||||
l.logger.Warn("geojs lookup failed; using the database", "error", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
for i, addr := range addrs {
|
|
||||||
if locations[i] != nil {
|
|
||||||
continue
|
|
||||||
}
|
|
||||||
|
|
||||||
address := plainAddress(addr)
|
|
||||||
|
|
||||||
answer, ok := answers[address]
|
|
||||||
if ok {
|
|
||||||
l.store(address, answer)
|
|
||||||
locations[i] = answer
|
|
||||||
|
|
||||||
continue
|
|
||||||
}
|
|
||||||
|
|
||||||
locations[i] = l.fromDatabase(addr)
|
|
||||||
}
|
|
||||||
|
|
||||||
return locations
|
|
||||||
}
|
|
||||||
|
|
||||||
// plainAddress returns addr as the requests to geojs and the cache name it: an
|
|
||||||
// IPv4 address mapped into IPv6 as plain IPv4, and with no zone. A client
|
|
||||||
// chooses the zone, so it must not reach the request.
|
|
||||||
func plainAddress(addr netip.Addr) string {
|
|
||||||
return addr.Unmap().WithZone("").String()
|
|
||||||
}
|
|
||||||
|
|
||||||
// Start loads the cached answers that have not expired and opens the copy of
|
|
||||||
// the database, if there is one. It then starts the goroutines that write new
|
|
||||||
// answers, delete expired ones and download the database. They run until Stop:
|
|
||||||
// the end of ctx does not stop them.
|
|
||||||
func (l *Locator) Start(ctx context.Context) error {
|
|
||||||
err := l.load(ctx)
|
|
||||||
if err != nil {
|
|
||||||
return err
|
|
||||||
}
|
|
||||||
|
|
||||||
l.logger.Info("Geolocation cache loaded", "entries", l.entries.Len())
|
|
||||||
|
|
||||||
l.openDatabaseCopy()
|
|
||||||
|
|
||||||
ctx, l.stop = context.WithCancel(context.WithoutCancel(ctx))
|
|
||||||
|
|
||||||
go func() {
|
|
||||||
defer close(l.done)
|
|
||||||
defer l.closeDatabase()
|
|
||||||
|
|
||||||
var workers sync.WaitGroup
|
|
||||||
|
|
||||||
workers.Go(func() { l.keepCache(ctx) })
|
|
||||||
workers.Go(func() { l.keepDatabaseCurrent(ctx) })
|
|
||||||
workers.Wait()
|
|
||||||
}()
|
|
||||||
|
|
||||||
return nil
|
|
||||||
}
|
|
||||||
|
|
||||||
// Stop cancels a download in progress, writes the answers cache.db does not
|
|
||||||
// have yet, and waits until the goroutines have returned or ctx ends.
|
|
||||||
func (l *Locator) Stop(ctx context.Context) {
|
|
||||||
l.stop()
|
|
||||||
|
|
||||||
select {
|
|
||||||
case <-l.done:
|
|
||||||
l.logger.Info("Geolocation stopped")
|
|
||||||
case <-ctx.Done():
|
|
||||||
l.logger.Warn("Stop deadline passed before the geolocation cache " +
|
|
||||||
"was written")
|
|
||||||
}
|
|
||||||
}
|
|
||||||
File diff suppressed because it is too large.
Load diff
@@ -1,992 +0,0 @@
|
|||||||
package iplist
|
|
||||||
|
|
||||||
import (
|
|
||||||
"context"
|
|
||||||
"errors"
|
|
||||||
"io"
|
|
||||||
"io/fs"
|
|
||||||
"log/slog"
|
|
||||||
"net/http"
|
|
||||||
"net/http/httptest"
|
|
||||||
"net/netip"
|
|
||||||
"os"
|
|
||||||
"path/filepath"
|
|
||||||
"slices"
|
|
||||||
"strings"
|
|
||||||
"sync"
|
|
||||||
"testing"
|
|
||||||
"time"
|
|
||||||
)
|
|
||||||
|
|
||||||
// The test list's name and description, the address its good version lists,
|
|
||||||
// and the address a newer version lists instead.
|
|
||||||
const (
|
|
||||||
testListName = "test_list"
|
|
||||||
testListDescription = "On the test list"
|
|
||||||
listedAddr = "192.0.2.1"
|
|
||||||
goodListBody = "# A test list\n" + listedAddr + "\n"
|
|
||||||
newListedAddr = "192.0.2.2"
|
|
||||||
newListBody = "# A test list\n" + newListedAddr + "\n"
|
|
||||||
)
|
|
||||||
|
|
||||||
// A small Onionoo details document: a relay with an exit address and IPv4 and
|
|
||||||
// IPv6 relay addresses, one without exit addresses, and one with a list of
|
|
||||||
// ports.
|
|
||||||
const (
|
|
||||||
onionooExitAddr = "192.0.2.10"
|
|
||||||
onionooIPv6ExitAddr = "2001:db8::1"
|
|
||||||
onionooBody = `{"version":"8.0","relays":[
|
|
||||||
{"exit_addresses":["` + onionooExitAddr + `"],
|
|
||||||
"or_addresses":["192.0.2.11:443","[` + onionooIPv6ExitAddr + `]:9001"]},
|
|
||||||
{"or_addresses":["192.0.2.20:443"]},
|
|
||||||
{"or_addresses":["192.0.2.30:80,443"]}
|
|
||||||
],"bridges":[]}`
|
|
||||||
)
|
|
||||||
|
|
||||||
// Small documents in the VPN services' formats. PIA's and IVPN's have no
|
|
||||||
// public IPv6 addresses, and IVPN's holds addresses that are not its servers,
|
|
||||||
// which must not be read.
|
|
||||||
const (
|
|
||||||
mullvadBody = `[
|
|
||||||
{"hostname":"a-wg-001","ipv4_addr_in":"192.0.2.40","ipv6_addr_in":"2001:db8::40"},
|
|
||||||
{"hostname":"a-br-001","ipv4_addr_in":"192.0.2.41","ipv6_addr_in":null,
|
|
||||||
"ipv4_v2ray":"192.0.2.42"}
|
|
||||||
]`
|
|
||||||
nordVPNBody = `[
|
|
||||||
{"hostname":"a1.example.com","station":"192.0.2.50","ipv6_station":""},
|
|
||||||
{"hostname":"a2.example.com","station":"192.0.2.51","ipv6_station":"2001:db8::51"}
|
|
||||||
]`
|
|
||||||
piaBody = `{"groups":{"wg":[{"name":"wireguard","ports":[1337]}]},"regions":[
|
|
||||||
{"id":"a","servers":{
|
|
||||||
"wg":[{"ip":"192.0.2.60","cn":"a1"}],
|
|
||||||
"ikev2":[{"ip":"192.0.2.61","cn":"a1"}]}},
|
|
||||||
{"id":"b","servers":{"meta":[{"ip":"192.0.2.62","cn":"b1","van":true}]}}
|
|
||||||
]}` + "\n\nc2lnbmF0dXJl\nbGluZSB0d28=\n"
|
|
||||||
ivpnBody = `{"wireguard":[{"gateway":"a.wg.example.com","hosts":[
|
|
||||||
{"hostname":"a1","host":"192.0.2.70","v2ray":"192.0.2.71",
|
|
||||||
"ipv6":{"local_ip":"2001:db8::70/96"}}]}],
|
|
||||||
"openvpn":[{"gateway":"a.gw.example.com","hosts":[
|
|
||||||
{"hostname":"a1","host":"192.0.2.72"}]}],
|
|
||||||
"config":{"api":{"ips":["192.0.2.73"]}}}`
|
|
||||||
)
|
|
||||||
|
|
||||||
// protonVPNBody is a small file in the format of the ProtonVPN-IPs list.
|
|
||||||
const protonVPNBody = "#\n# protonvpn_ip_ranges.txt\n" +
|
|
||||||
"# https://github.com/tn3w/ProtonVPN-IPs\n#\n" +
|
|
||||||
"# The /24 block of every advertised IPv4.\n#\n" +
|
|
||||||
"192.0.2.0/24\n198.51.100.0/24\n"
|
|
||||||
|
|
||||||
// protonVPNEntryBody is a small file in the format of the ProtonVPN-IPs list
|
|
||||||
// of server addresses.
|
|
||||||
const protonVPNEntryBody = "#\n# protonvpn_entry_ips.txt\n" +
|
|
||||||
"# https://github.com/tn3w/ProtonVPN-IPs\n#\n" +
|
|
||||||
"# Entry IPs clients connect to, resolved from ProtonVPN hostnames.\n#\n" +
|
|
||||||
"192.0.2.80\n2001:db8::80\n"
|
|
||||||
|
|
||||||
// iCloudPrivateRelayAddr is an address on iCloudPrivateRelayBody.
|
|
||||||
const iCloudPrivateRelayAddr = "2001:db8:1::5"
|
|
||||||
|
|
||||||
// iCloudPrivateRelayBody is a small file in the format of Apple's iCloud
|
|
||||||
// Private Relay file.
|
|
||||||
const iCloudPrivateRelayBody = "192.0.2.0/31,GB,GB-EN,London,\n" +
|
|
||||||
"198.51.100.32/27,US,US-CA,Los Angeles,\n" +
|
|
||||||
"2001:db8:1::/64,US,US-AK,Anchorage,\n"
|
|
||||||
|
|
||||||
// emptyFileCase names the case of a wrong document that is an empty file.
|
|
||||||
const emptyFileCase = "an empty file"
|
|
||||||
|
|
||||||
// errorPage is a file that is no list, as a publisher's error page is.
|
|
||||||
const errorPage = "<!DOCTYPE html>\n<html></html>\n"
|
|
||||||
|
|
||||||
// testETag is the ETag a publisher sends with a version of the test list.
|
|
||||||
const testETag = `"v1"`
|
|
||||||
|
|
||||||
// maxWait is how long a test waits for the download goroutine.
|
|
||||||
const maxWait = 10 * time.Second
|
|
||||||
|
|
||||||
// publisher stands in for a list's publisher. It answers every request with
|
|
||||||
// its status and body, and keeps the headers of each request. A 200 answer
|
|
||||||
// carries lastModified and etag, when set, as Last-Modified and ETag. As
|
|
||||||
// from a real publisher, a request whose If-None-Match or If-Modified-Since
|
|
||||||
// matches them is answered "not modified".
|
|
||||||
type publisher struct {
|
|
||||||
mu sync.Mutex
|
|
||||||
status int
|
|
||||||
body string
|
|
||||||
lastModified time.Time
|
|
||||||
etag string
|
|
||||||
headers []http.Header
|
|
||||||
}
|
|
||||||
|
|
||||||
// newPublisher starts a publisher that answers 200 with body, and returns it
|
|
||||||
// and its URL.
|
|
||||||
func newPublisher(t *testing.T, body string) (*publisher, string) {
|
|
||||||
t.Helper()
|
|
||||||
|
|
||||||
p := &publisher{status: http.StatusOK, body: body}
|
|
||||||
srv := httptest.NewServer(p)
|
|
||||||
t.Cleanup(srv.Close)
|
|
||||||
|
|
||||||
return p, srv.URL
|
|
||||||
}
|
|
||||||
|
|
||||||
func (p *publisher) ServeHTTP(w http.ResponseWriter, r *http.Request) {
|
|
||||||
p.mu.Lock()
|
|
||||||
defer p.mu.Unlock()
|
|
||||||
|
|
||||||
p.headers = append(p.headers, r.Header.Clone())
|
|
||||||
|
|
||||||
if p.status != http.StatusOK {
|
|
||||||
w.WriteHeader(p.status)
|
|
||||||
_, _ = io.WriteString(w, p.body)
|
|
||||||
|
|
||||||
return
|
|
||||||
}
|
|
||||||
|
|
||||||
if p.etag != "" {
|
|
||||||
w.Header().Set(etagHeader, p.etag)
|
|
||||||
}
|
|
||||||
|
|
||||||
// ServeContent sends Last-Modified unless lastModified is zero, and
|
|
||||||
// answers "not modified" when the request's conditions match.
|
|
||||||
http.ServeContent(w, r, testListName, p.lastModified,
|
|
||||||
strings.NewReader(p.body))
|
|
||||||
}
|
|
||||||
|
|
||||||
// answer makes the publisher answer later requests with status and body.
|
|
||||||
func (p *publisher) answer(status int, body string) {
|
|
||||||
p.mu.Lock()
|
|
||||||
defer p.mu.Unlock()
|
|
||||||
|
|
||||||
p.status, p.body = status, body
|
|
||||||
}
|
|
||||||
|
|
||||||
// publish makes the publisher answer later requests with a version of the
|
|
||||||
// list: body, with lastModified and etag. A zero lastModified or an empty
|
|
||||||
// etag is not sent.
|
|
||||||
func (p *publisher) publish(body string, lastModified time.Time, etag string) {
|
|
||||||
p.mu.Lock()
|
|
||||||
defer p.mu.Unlock()
|
|
||||||
|
|
||||||
p.status, p.body = http.StatusOK, body
|
|
||||||
p.lastModified, p.etag = lastModified, etag
|
|
||||||
}
|
|
||||||
|
|
||||||
// requests returns the headers of the requests the publisher has had.
|
|
||||||
func (p *publisher) requests() []http.Header {
|
|
||||||
p.mu.Lock()
|
|
||||||
defer p.mu.Unlock()
|
|
||||||
|
|
||||||
return append([]http.Header(nil), p.headers...)
|
|
||||||
}
|
|
||||||
|
|
||||||
// testList returns a list served at url, written in format.
|
|
||||||
func testList(url string, format Format) List {
|
|
||||||
return List{
|
|
||||||
Name: testListName,
|
|
||||||
Description: testListDescription,
|
|
||||||
URL: url,
|
|
||||||
Interval: time.Hour,
|
|
||||||
Format: format,
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
func newTestIPLists(dir string, list List) *IPLists {
|
|
||||||
return New(Params{
|
|
||||||
Dir: dir,
|
|
||||||
Lists: []List{list},
|
|
||||||
Logger: slog.New(slog.DiscardHandler),
|
|
||||||
})
|
|
||||||
}
|
|
||||||
|
|
||||||
// lookup looks addr up in b.
|
|
||||||
func lookup(t *testing.T, b *IPLists, addr string) []Listing {
|
|
||||||
t.Helper()
|
|
||||||
|
|
||||||
return b.Lookup(netip.MustParseAddr(addr))
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestLookupFindsListedAddresses downloads a list in each format and looks
|
|
||||||
// up addresses on it, inside its networks, and off it, in IPv4 and IPv6.
|
|
||||||
// A listed address has one listing that names the list; any other has an
|
|
||||||
// empty list of listings.
|
|
||||||
func TestLookupFindsListedAddresses(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
runLookupCases(t, []lookupCase{
|
|
||||||
{
|
|
||||||
name: "single addresses",
|
|
||||||
format: FormatLines,
|
|
||||||
body: "# comment\n192.0.2.1\n\n2001:db8::1\n",
|
|
||||||
listed: []string{"192.0.2.1", "::ffff:192.0.2.1", "2001:db8::1"},
|
|
||||||
unlisted: []string{"192.0.2.2", "2001:db8::2"},
|
|
||||||
},
|
|
||||||
{
|
|
||||||
name: "networks",
|
|
||||||
format: FormatLines,
|
|
||||||
body: "# comment\n198.51.100.0/24\n2001:db8:1::/48\n",
|
|
||||||
listed: []string{"198.51.100.0", "198.51.100.255", "2001:db8:1::5"},
|
|
||||||
unlisted: []string{"198.51.101.0", "2001:db8:2::1"},
|
|
||||||
},
|
|
||||||
{
|
|
||||||
name: "Spamhaus networks",
|
|
||||||
format: FormatSpamhausJSON,
|
|
||||||
body: `{"cidr":"203.0.113.0/25","sblid":"SBL1","rir":"arin"}` + "\n" +
|
|
||||||
`{"cidr":"2001:db8:3::/48","sblid":"SBL2","rir":"ripencc"}` + "\n" +
|
|
||||||
`{"type":"metadata","timestamp":1793074442,"records":2}` + "\n",
|
|
||||||
listed: []string{"203.0.113.1", "2001:db8:3::1"},
|
|
||||||
unlisted: []string{"203.0.113.200", "2001:db8:4::1"},
|
|
||||||
},
|
|
||||||
{
|
|
||||||
name: "DShield networks",
|
|
||||||
format: FormatDShield,
|
|
||||||
body: "# DShield.org Recommended Block List\n" +
|
|
||||||
"192.0.2.0\t192.0.2.255\t24\t327\tEXAMPLE\tUS\tNone\n",
|
|
||||||
listed: []string{"192.0.2.0", "192.0.2.77"},
|
|
||||||
unlisted: []string{"192.0.3.1", "2001:db8::1"},
|
|
||||||
},
|
|
||||||
{
|
|
||||||
name: "Onionoo relays",
|
|
||||||
format: FormatOnionooJSON,
|
|
||||||
body: onionooBody,
|
|
||||||
listed: []string{
|
|
||||||
onionooExitAddr, "192.0.2.11", "192.0.2.20", "192.0.2.30",
|
|
||||||
onionooIPv6ExitAddr,
|
|
||||||
},
|
|
||||||
unlisted: []string{"192.0.2.12", "2001:db8::2"},
|
|
||||||
},
|
|
||||||
{
|
|
||||||
name: "iCloud Private Relay networks",
|
|
||||||
format: FormatICloudPrivateRelayCSV,
|
|
||||||
body: iCloudPrivateRelayBody,
|
|
||||||
listed: []string{
|
|
||||||
"192.0.2.1", "198.51.100.63", iCloudPrivateRelayAddr,
|
|
||||||
},
|
|
||||||
unlisted: []string{"192.0.2.2", "198.51.100.64", "2001:db8:2::1"},
|
|
||||||
},
|
|
||||||
})
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestLookupFindsVPNServers downloads a list in each VPN service's format and
|
|
||||||
// looks up addresses on it and off it.
|
|
||||||
func TestLookupFindsVPNServers(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
runLookupCases(t, []lookupCase{
|
|
||||||
{
|
|
||||||
name: "Mullvad relays",
|
|
||||||
format: FormatMullvadJSON,
|
|
||||||
body: mullvadBody,
|
|
||||||
listed: []string{"192.0.2.40", "192.0.2.41", "192.0.2.42", "2001:db8::40"},
|
|
||||||
unlisted: []string{"192.0.2.43", "2001:db8::41"},
|
|
||||||
},
|
|
||||||
{
|
|
||||||
name: "NordVPN servers",
|
|
||||||
format: FormatNordVPNJSON,
|
|
||||||
body: nordVPNBody,
|
|
||||||
listed: []string{"192.0.2.50", "192.0.2.51", "2001:db8::51"},
|
|
||||||
unlisted: []string{"192.0.2.52", "2001:db8::50"},
|
|
||||||
},
|
|
||||||
{
|
|
||||||
name: "PIA servers, with the signature after the JSON",
|
|
||||||
format: FormatPIAJSON,
|
|
||||||
body: piaBody,
|
|
||||||
listed: []string{"192.0.2.60", "192.0.2.61", "192.0.2.62"},
|
|
||||||
unlisted: []string{"192.0.2.63", "2001:db8::60"},
|
|
||||||
},
|
|
||||||
{
|
|
||||||
name: "IVPN hosts",
|
|
||||||
format: FormatIVPNJSON,
|
|
||||||
body: ivpnBody,
|
|
||||||
listed: []string{"192.0.2.70", "192.0.2.71", "192.0.2.72"},
|
|
||||||
unlisted: []string{"192.0.2.73", "2001:db8::70"},
|
|
||||||
},
|
|
||||||
{
|
|
||||||
name: "ProtonVPN networks",
|
|
||||||
format: FormatProtonVPN,
|
|
||||||
body: protonVPNBody,
|
|
||||||
listed: []string{"192.0.2.0", "192.0.2.255", "198.51.100.7"},
|
|
||||||
unlisted: []string{"192.0.3.1", "2001:db8::90"},
|
|
||||||
},
|
|
||||||
{
|
|
||||||
name: "ProtonVPN server addresses",
|
|
||||||
format: FormatProtonVPN,
|
|
||||||
body: protonVPNEntryBody,
|
|
||||||
listed: []string{"192.0.2.80", "2001:db8::80"},
|
|
||||||
unlisted: []string{"192.0.2.81", "2001:db8::81"},
|
|
||||||
},
|
|
||||||
})
|
|
||||||
}
|
|
||||||
|
|
||||||
// lookupCase is a list file, the addresses it holds and some it does not.
|
|
||||||
type lookupCase struct {
|
|
||||||
name string
|
|
||||||
format Format
|
|
||||||
body string
|
|
||||||
listed []string
|
|
||||||
unlisted []string
|
|
||||||
}
|
|
||||||
|
|
||||||
// runLookupCases downloads each case's list file and checks its addresses.
|
|
||||||
func runLookupCases(t *testing.T, cases []lookupCase) {
|
|
||||||
t.Helper()
|
|
||||||
|
|
||||||
for _, tt := range cases {
|
|
||||||
t.Run(tt.name, func(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
_, url := newPublisher(t, tt.body)
|
|
||||||
b := newTestIPLists(t.TempDir(), testList(url, tt.format))
|
|
||||||
|
|
||||||
err := b.download(t.Context(), b.lists[0])
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("download: %v", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
checkLookups(t, b, tt.listed, tt.unlisted)
|
|
||||||
})
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// checkLookups checks that each of listed has one listing that names the test
|
|
||||||
// list, and each of unlisted has an empty list of listings.
|
|
||||||
func checkLookups(t *testing.T, b *IPLists, listed, unlisted []string) {
|
|
||||||
t.Helper()
|
|
||||||
|
|
||||||
for _, addr := range listed {
|
|
||||||
listings := lookup(t, b, addr)
|
|
||||||
if len(listings) != 1 || listings[0].Name != testListName ||
|
|
||||||
listings[0].Description != testListDescription ||
|
|
||||||
listings[0].Updated.IsZero() {
|
|
||||||
t.Errorf("%s: listings %+v, want one for %s", addr,
|
|
||||||
listings, testListName)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
for _, addr := range unlisted {
|
|
||||||
listings := lookup(t, b, addr)
|
|
||||||
if listings == nil || len(listings) != 0 {
|
|
||||||
t.Errorf("%s: listings %#v, want an empty list", addr, listings)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestFailedDownloadKeepsThePreviousCopy downloads a good version of a list,
|
|
||||||
// then one that fails. The good copy stays in memory and on disk, and no
|
|
||||||
// part of the failed download is left beside it.
|
|
||||||
func TestFailedDownloadKeepsThePreviousCopy(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
tests := []struct {
|
|
||||||
name string
|
|
||||||
status int
|
|
||||||
body string
|
|
||||||
}{
|
|
||||||
{"server error", http.StatusInternalServerError, "unavailable"},
|
|
||||||
{"not a list", http.StatusOK, errorPage},
|
|
||||||
}
|
|
||||||
|
|
||||||
for _, tt := range tests {
|
|
||||||
t.Run(tt.name, func(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
dir := t.TempDir()
|
|
||||||
p, url := newPublisher(t, goodListBody)
|
|
||||||
b := newTestIPLists(dir, testList(url, FormatLines))
|
|
||||||
|
|
||||||
err := b.download(t.Context(), b.lists[0])
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("first download: %v", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
before := lookup(t, b, listedAddr)
|
|
||||||
|
|
||||||
p.answer(tt.status, tt.body)
|
|
||||||
|
|
||||||
err = b.download(t.Context(), b.lists[0])
|
|
||||||
if err == nil {
|
|
||||||
t.Fatal("second download returned no error")
|
|
||||||
}
|
|
||||||
|
|
||||||
after := lookup(t, b, listedAddr)
|
|
||||||
if len(after) != 1 || len(before) != 1 ||
|
|
||||||
!after[0].Updated.Equal(before[0].Updated) {
|
|
||||||
t.Errorf("listings %+v after the failure, want %+v", after, before)
|
|
||||||
}
|
|
||||||
|
|
||||||
checkGoodCopyOnDisk(t, dir)
|
|
||||||
})
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestFileThatIsNotAnOnionooDocumentKeepsThePreviousCopy downloads a good
|
|
||||||
// Onionoo document, then files that are not one. Each fails the parse, and
|
|
||||||
// the good copy stays.
|
|
||||||
func TestFileThatIsNotAnOnionooDocumentKeepsThePreviousCopy(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
tests := []struct {
|
|
||||||
name string
|
|
||||||
body string
|
|
||||||
}{
|
|
||||||
{"not JSON", errorPage},
|
|
||||||
{"other JSON", `{"status":"ok","data":{"relays_count":1}}`},
|
|
||||||
{"JSON array", `[{"or_addresses":["192.0.2.1:443"]}]`},
|
|
||||||
{"no relays", `{"relays":[],"bridges":[]}`},
|
|
||||||
{"relays without addresses", `{"relays":[{"nickname":"example"}]}`},
|
|
||||||
{"relay address without a port", `{"relays":[{"or_addresses":["192.0.2.1"]}]}`},
|
|
||||||
{"exit address that is not one", `{"relays":[{"exit_addresses":["example"]}]}`},
|
|
||||||
{"text after the document", `{"relays":[{"or_addresses":["192.0.2.1:1"]}]} x`},
|
|
||||||
}
|
|
||||||
|
|
||||||
for _, tt := range tests {
|
|
||||||
t.Run(tt.name, func(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
dir := t.TempDir()
|
|
||||||
p, url := newPublisher(t, onionooBody)
|
|
||||||
b := newTestIPLists(dir, testList(url, FormatOnionooJSON))
|
|
||||||
|
|
||||||
err := b.download(t.Context(), b.lists[0])
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("first download: %v", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
p.answer(http.StatusOK, tt.body)
|
|
||||||
|
|
||||||
err = b.download(t.Context(), b.lists[0])
|
|
||||||
if err == nil {
|
|
||||||
t.Fatal("second download returned no error")
|
|
||||||
}
|
|
||||||
|
|
||||||
if len(lookup(t, b, onionooExitAddr)) != 1 {
|
|
||||||
t.Errorf("%s is no longer listed", onionooExitAddr)
|
|
||||||
}
|
|
||||||
|
|
||||||
onDisk, err := fs.ReadFile(os.DirFS(dir), testListName)
|
|
||||||
if err != nil || string(onDisk) != onionooBody {
|
|
||||||
t.Errorf("copy on disk %q (%v), want the good version", onDisk, err)
|
|
||||||
}
|
|
||||||
})
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestWrongVPNDocumentFailsToParse parses documents that are not a VPN
|
|
||||||
// service's list. Each fails, so none can replace a good copy.
|
|
||||||
func TestWrongVPNDocumentFailsToParse(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
formats := []struct {
|
|
||||||
name string
|
|
||||||
format Format
|
|
||||||
good string
|
|
||||||
empty string
|
|
||||||
badAddress string
|
|
||||||
}{
|
|
||||||
{
|
|
||||||
"Mullvad", FormatMullvadJSON, mullvadBody,
|
|
||||||
`[{"hostname":"a"}]`, `[{"ipv4_addr_in":"example"}]`,
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"NordVPN", FormatNordVPNJSON, nordVPNBody,
|
|
||||||
`[{"hostname":"a","station":"","ipv6_station":""}]`,
|
|
||||||
`[{"station":"example"}]`,
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"PIA", FormatPIAJSON, piaBody,
|
|
||||||
`{"regions":[{"id":"a","servers":{}}]}`,
|
|
||||||
`{"regions":[{"servers":{"wg":[{"ip":"example"}]}}]}`,
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"IVPN", FormatIVPNJSON, ivpnBody,
|
|
||||||
`{"wireguard":[{"hosts":[]}],"openvpn":[]}`,
|
|
||||||
`{"wireguard":[{"hosts":[{"host":"example"}]}]}`,
|
|
||||||
},
|
|
||||||
}
|
|
||||||
|
|
||||||
for _, f := range formats {
|
|
||||||
bodies := map[string]string{
|
|
||||||
"not JSON": errorPage,
|
|
||||||
"other JSON": `{"status":"ok","data":{"count":1}}`,
|
|
||||||
"no addresses": f.empty,
|
|
||||||
"bad address": f.badAddress,
|
|
||||||
"cut off": f.good[:len(f.good)/2],
|
|
||||||
emptyFileCase: "",
|
|
||||||
}
|
|
||||||
|
|
||||||
for name, body := range bodies {
|
|
||||||
t.Run(f.name+" "+name, func(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
addresses, err := parse(strings.NewReader(body), f.format)
|
|
||||||
if err == nil {
|
|
||||||
t.Errorf("parse returned %v and no error", addresses.Ranges())
|
|
||||||
}
|
|
||||||
})
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestWrongProtonVPNDocumentFailsToParse parses documents that are not a
|
|
||||||
// ProtonVPN-IPs list. Each fails, so none can replace a good copy.
|
|
||||||
func TestWrongProtonVPNDocumentFailsToParse(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
bodies := map[string]string{
|
|
||||||
"error page": errorPage,
|
|
||||||
"JSON": `{"status":"ok"}`,
|
|
||||||
"comments only": "#\n# protonvpn_ip_ranges.txt\n#\n",
|
|
||||||
"bad network": "192.0.2.0/24\nexample\n",
|
|
||||||
emptyFileCase: "",
|
|
||||||
}
|
|
||||||
|
|
||||||
for name, body := range bodies {
|
|
||||||
t.Run(name, func(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
addresses, err := parse(strings.NewReader(body), FormatProtonVPN)
|
|
||||||
if err == nil {
|
|
||||||
t.Errorf("parse returned %v and no error", addresses.Ranges())
|
|
||||||
}
|
|
||||||
})
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestWrongICloudPrivateRelayDocumentFailsToParse parses documents that are
|
|
||||||
// not Apple's iCloud Private Relay file. Each fails, so none can replace a
|
|
||||||
// good copy.
|
|
||||||
func TestWrongICloudPrivateRelayDocumentFailsToParse(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
bodies := map[string]string{
|
|
||||||
"error page": errorPage,
|
|
||||||
"JSON": `{"status":"ok","count":1}`,
|
|
||||||
"no other field": "192.0.2.0/24\n",
|
|
||||||
"bad network": "192.0.2.0/24,GB,GB-EN,London,\nexample,GB,GB-EN,London,\n",
|
|
||||||
"an address": "192.0.2.1,GB,GB-EN,London,\n",
|
|
||||||
emptyFileCase: "",
|
|
||||||
}
|
|
||||||
|
|
||||||
for name, body := range bodies {
|
|
||||||
t.Run(name, func(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
addresses, err := parse(strings.NewReader(body),
|
|
||||||
FormatICloudPrivateRelayCSV)
|
|
||||||
if err == nil {
|
|
||||||
t.Errorf("parse returned %v and no error", addresses.Ranges())
|
|
||||||
}
|
|
||||||
})
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestLookupMarksVPNLists looks up an address on a VPN list and on another
|
|
||||||
// list. Only the VPN list's listing says it is one.
|
|
||||||
func TestLookupMarksVPNLists(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
_, url := newPublisher(t, goodListBody)
|
|
||||||
vpnList := testList(url, FormatLines)
|
|
||||||
vpnList.Name = "test_vpn_list"
|
|
||||||
vpnList.VPN = true
|
|
||||||
otherList := testList(url, FormatLines)
|
|
||||||
|
|
||||||
b := New(Params{
|
|
||||||
Dir: t.TempDir(),
|
|
||||||
Lists: []List{vpnList, otherList},
|
|
||||||
Logger: slog.New(slog.DiscardHandler),
|
|
||||||
})
|
|
||||||
|
|
||||||
for _, list := range b.lists {
|
|
||||||
err := b.download(t.Context(), list)
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("download of %s: %v", list.Name, err)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
listings := lookup(t, b, listedAddr)
|
|
||||||
if len(listings) != 2 || !listings[0].VPN || listings[1].VPN {
|
|
||||||
t.Errorf("listings %+v, want the first of two marked as a VPN list",
|
|
||||||
listings)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestPublicListsMarkTheVPNLists checks which public lists are VPN lists,
|
|
||||||
// since their names are what vpn_providers reports.
|
|
||||||
func TestPublicListsMarkTheVPNLists(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
var got []string
|
|
||||||
|
|
||||||
for _, list := range Lists() {
|
|
||||||
if list.VPN {
|
|
||||||
got = append(got, list.Name)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
want := []string{
|
|
||||||
"ivpn", "mullvad", "nordvpn", "pia", "protonvpn", "protonvpn_entry",
|
|
||||||
"x4bnet", "x4bnet_v6",
|
|
||||||
}
|
|
||||||
if !slices.Equal(got, want) {
|
|
||||||
t.Errorf("VPN lists %v, want %v", got, want)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// checkGoodCopyOnDisk checks that dir holds the good version of the test list
|
|
||||||
// and no part of a failed download.
|
|
||||||
func checkGoodCopyOnDisk(t *testing.T, dir string) {
|
|
||||||
t.Helper()
|
|
||||||
|
|
||||||
copies := os.DirFS(dir)
|
|
||||||
|
|
||||||
onDisk, err := fs.ReadFile(copies, testListName)
|
|
||||||
if err != nil || string(onDisk) != goodListBody {
|
|
||||||
t.Errorf("copy on disk %q (%v), want %q", onDisk, err, goodListBody)
|
|
||||||
}
|
|
||||||
|
|
||||||
_, err = fs.Stat(copies, testListName+downloadSuffix)
|
|
||||||
if !errors.Is(err, fs.ErrNotExist) {
|
|
||||||
t.Errorf("the failed download's file is still there (%v)", err)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// commentLines is an endless io.Reader of comment lines. It lets a test send
|
|
||||||
// a large list file without holding the file in memory.
|
|
||||||
type commentLines struct {
|
|
||||||
offset int
|
|
||||||
}
|
|
||||||
|
|
||||||
func (c *commentLines) Read(p []byte) (int, error) {
|
|
||||||
const line = "# a comment line that pads a large list file\n"
|
|
||||||
|
|
||||||
for filled := 0; filled < len(p); {
|
|
||||||
copied := copy(p[filled:], line[c.offset:])
|
|
||||||
filled += copied
|
|
||||||
c.offset = (c.offset + copied) % len(line)
|
|
||||||
}
|
|
||||||
|
|
||||||
return len(p), nil
|
|
||||||
}
|
|
||||||
|
|
||||||
// largeListFile returns a reader of a list file of size bytes: comment lines,
|
|
||||||
// then newListedAddr on the last line.
|
|
||||||
func largeListFile(size int64) io.Reader {
|
|
||||||
lastLine := "\n" + newListedAddr + "\n"
|
|
||||||
|
|
||||||
return io.MultiReader(
|
|
||||||
io.LimitReader(&commentLines{}, size-int64(len(lastLine))),
|
|
||||||
strings.NewReader(lastLine),
|
|
||||||
)
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestLargeFileUnderTheSizeLimitIsAccepted replaces a list with a 199 MiB
|
|
||||||
// file. The address on its last line is listed, so the whole file was read.
|
|
||||||
func TestLargeFileUnderTheSizeLimitIsAccepted(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
const size = 199 << 20
|
|
||||||
|
|
||||||
b := newTestIPLists(t.TempDir(), testList("", FormatLines))
|
|
||||||
|
|
||||||
err := b.replaceCopy(b.lists[0], largeListFile(size), validators{})
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("replaceCopy: %v", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
if listings := lookup(t, b, newListedAddr); len(listings) != 1 {
|
|
||||||
t.Errorf("listings %+v, want one", listings)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestFileOverTheSizeLimitKeepsThePreviousCopy downloads a good version of a
|
|
||||||
// list, then replaces it with a file one byte over the limit. The attempt
|
|
||||||
// fails, and the good copy stays in memory and on disk.
|
|
||||||
func TestFileOverTheSizeLimitKeepsThePreviousCopy(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
dir := t.TempDir()
|
|
||||||
_, url := newPublisher(t, goodListBody)
|
|
||||||
b := newTestIPLists(dir, testList(url, FormatLines))
|
|
||||||
|
|
||||||
err := b.download(t.Context(), b.lists[0])
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("download: %v", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
err = b.replaceCopy(b.lists[0], largeListFile(maxDownloadSize+1), validators{})
|
|
||||||
if !errors.Is(err, errTooLarge) {
|
|
||||||
t.Fatalf("replaceCopy: %v, want %v", err, errTooLarge)
|
|
||||||
}
|
|
||||||
|
|
||||||
if listings := lookup(t, b, listedAddr); len(listings) != 1 {
|
|
||||||
t.Errorf("listings of %s %+v, want one", listedAddr, listings)
|
|
||||||
}
|
|
||||||
|
|
||||||
checkGoodCopyOnDisk(t, dir)
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestRestartLoadsCopiesFromDisk downloads a list, then starts a new
|
|
||||||
// IPLists on the same directory, as a restart does. The new one answers
|
|
||||||
// from the copy on disk at once, and does not download the list again while
|
|
||||||
// the copy is younger than the list's interval.
|
|
||||||
func TestRestartLoadsCopiesFromDisk(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
dir := t.TempDir()
|
|
||||||
p, url := newPublisher(t, goodListBody)
|
|
||||||
list := testList(url, FormatLines)
|
|
||||||
|
|
||||||
first := newTestIPLists(dir, list)
|
|
||||||
|
|
||||||
err := first.download(t.Context(), list)
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("download: %v", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
downloaded := lookup(t, first, listedAddr)[0].Updated
|
|
||||||
|
|
||||||
second := newTestIPLists(dir, list)
|
|
||||||
second.Start(t.Context())
|
|
||||||
second.Stop(t.Context())
|
|
||||||
|
|
||||||
// The download takes its time after writing the file, and the restart
|
|
||||||
// reads the file's time. A busy host can put seconds between the two.
|
|
||||||
const maxUpdatedDifference = 15 * time.Second
|
|
||||||
|
|
||||||
listings := lookup(t, second, listedAddr)
|
|
||||||
if len(listings) != 1 ||
|
|
||||||
listings[0].Updated.Sub(downloaded).Abs() > maxUpdatedDifference {
|
|
||||||
t.Errorf("after the restart, listings %+v, want one updated at %s",
|
|
||||||
listings, downloaded)
|
|
||||||
}
|
|
||||||
|
|
||||||
if got := len(p.requests()); got != 1 {
|
|
||||||
t.Errorf("the publisher had %d requests, want 1", got)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestStartRefreshesAnOldCopy starts on a copy older than its list's
|
|
||||||
// interval. The list is downloaded at once, with routewatch's User-Agent and
|
|
||||||
// the Last-Modified and ETag the publisher sent with the copy, kept on disk.
|
|
||||||
// The publisher answers that the copy is still current, so the copy is kept
|
|
||||||
// and marked as updated now.
|
|
||||||
func TestStartRefreshesAnOldCopy(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
dir := t.TempDir()
|
|
||||||
p, url := newPublisher(t, goodListBody)
|
|
||||||
list := testList(url, FormatLines)
|
|
||||||
|
|
||||||
lastModified := time.Now().Add(-3 * list.Interval).UTC().Truncate(time.Second)
|
|
||||||
p.publish(goodListBody, lastModified, testETag)
|
|
||||||
|
|
||||||
err := newTestIPLists(dir, list).download(t.Context(), list)
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("download: %v", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
old := time.Now().Add(-2 * list.Interval).UTC().Truncate(time.Second)
|
|
||||||
|
|
||||||
err = os.Chtimes(filepath.Join(dir, testListName), old, old)
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("Chtimes: %v", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
b := newTestIPLists(dir, list)
|
|
||||||
b.Start(t.Context())
|
|
||||||
|
|
||||||
deadline := time.Now().Add(maxWait)
|
|
||||||
for !lookup(t, b, listedAddr)[0].Updated.After(old) {
|
|
||||||
if time.Now().After(deadline) {
|
|
||||||
t.Fatalf("the copy was not marked updated within %s", maxWait)
|
|
||||||
}
|
|
||||||
|
|
||||||
time.Sleep(time.Millisecond)
|
|
||||||
}
|
|
||||||
|
|
||||||
b.Stop(t.Context())
|
|
||||||
|
|
||||||
requests := p.requests()
|
|
||||||
last := requests[len(requests)-1]
|
|
||||||
|
|
||||||
want := lastModified.Format(http.TimeFormat)
|
|
||||||
if got := last.Get(ifModifiedSinceHeader); got != want {
|
|
||||||
t.Errorf("If-Modified-Since %q, want %q", got, want)
|
|
||||||
}
|
|
||||||
|
|
||||||
if got := last.Get(ifNoneMatchHeader); got != testETag {
|
|
||||||
t.Errorf("If-None-Match %q, want %q", got, testETag)
|
|
||||||
}
|
|
||||||
|
|
||||||
if got := last.Get(userAgentHeader); got != userAgent {
|
|
||||||
t.Errorf("User-Agent %q, want %q", got, userAgent)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestNewVersionOlderThanTheLastDownloadReplacesTheCopy downloads a list.
|
|
||||||
// The publisher then puts out a new version whose Last-Modified is earlier
|
|
||||||
// than that download, as when a cache in front of the publisher passes the
|
|
||||||
// version on late. The next download sends the Last-Modified of the copy,
|
|
||||||
// not the time of the download, so it gets the new version.
|
|
||||||
func TestNewVersionOlderThanTheLastDownloadReplacesTheCopy(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
p, url := newPublisher(t, goodListBody)
|
|
||||||
list := testList(url, FormatLines)
|
|
||||||
b := newTestIPLists(t.TempDir(), list)
|
|
||||||
|
|
||||||
firstVersion := time.Now().Add(-2 * list.Interval).UTC().Truncate(time.Second)
|
|
||||||
p.publish(goodListBody, firstVersion, "")
|
|
||||||
|
|
||||||
err := b.download(t.Context(), list)
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("first download: %v", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
p.publish(newListBody, firstVersion.Add(list.Interval), "")
|
|
||||||
|
|
||||||
err = b.download(t.Context(), list)
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("second download: %v", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
if listings := lookup(t, b, newListedAddr); len(listings) != 1 {
|
|
||||||
t.Errorf("%s: listings %+v, want one from the new version",
|
|
||||||
newListedAddr, listings)
|
|
||||||
}
|
|
||||||
|
|
||||||
want := firstVersion.Format(http.TimeFormat)
|
|
||||||
if got := p.requests()[1].Get(ifModifiedSinceHeader); got != want {
|
|
||||||
t.Errorf("If-Modified-Since %q, want %q", got, want)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestDownloadWithoutLastModifiedOrETagSendsNoCondition downloads a list
|
|
||||||
// twice from a publisher that sends neither Last-Modified nor ETag. Neither
|
|
||||||
// request carries If-Modified-Since or If-None-Match, so the second download
|
|
||||||
// gets the whole list again.
|
|
||||||
func TestDownloadWithoutLastModifiedOrETagSendsNoCondition(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
p, url := newPublisher(t, goodListBody)
|
|
||||||
list := testList(url, FormatLines)
|
|
||||||
b := newTestIPLists(t.TempDir(), list)
|
|
||||||
|
|
||||||
err := b.download(t.Context(), list)
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("first download: %v", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
err = b.download(t.Context(), list)
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("second download: %v", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
for i, request := range p.requests() {
|
|
||||||
if request.Get(ifModifiedSinceHeader) != "" ||
|
|
||||||
request.Get(ifNoneMatchHeader) != "" {
|
|
||||||
t.Errorf("request %d has a condition: %v", i, request)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestStopCancelsADownloadInProgress stops while the publisher has not
|
|
||||||
// answered. Stop cancels the download and returns once the download
|
|
||||||
// goroutine has stopped.
|
|
||||||
func TestStopCancelsADownloadInProgress(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
arrived := make(chan struct{})
|
|
||||||
|
|
||||||
var once sync.Once
|
|
||||||
|
|
||||||
srv := httptest.NewServer(http.HandlerFunc(
|
|
||||||
func(_ http.ResponseWriter, r *http.Request) {
|
|
||||||
once.Do(func() { close(arrived) })
|
|
||||||
<-r.Context().Done()
|
|
||||||
}))
|
|
||||||
t.Cleanup(srv.Close)
|
|
||||||
|
|
||||||
b := newTestIPLists(t.TempDir(), testList(srv.URL, FormatLines))
|
|
||||||
b.Start(t.Context())
|
|
||||||
|
|
||||||
select {
|
|
||||||
case <-arrived:
|
|
||||||
case <-time.After(maxWait):
|
|
||||||
t.Fatalf("no download began within %s", maxWait)
|
|
||||||
}
|
|
||||||
|
|
||||||
ctx, cancel := context.WithTimeout(t.Context(), maxWait)
|
|
||||||
defer cancel()
|
|
||||||
|
|
||||||
b.Stop(ctx)
|
|
||||||
|
|
||||||
select {
|
|
||||||
case <-b.done:
|
|
||||||
default:
|
|
||||||
t.Error("Stop returned before the download goroutine stopped")
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestTorExitsListHoldsIPv6Exits downloads an Onionoo document as the public
|
|
||||||
// Tor exit list, which is what an IPv6 address's is_tor answer comes from.
|
|
||||||
func TestTorExitsListHoldsIPv6Exits(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
checkPublicListHolds(t, TorExitsList, onionooBody, onionooIPv6ExitAddr)
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestICloudPrivateRelayListHoldsItsNetworks downloads a file in Apple's
|
|
||||||
// format as the public iCloud Private Relay list, which is what an address's
|
|
||||||
// is_relay answer comes from.
|
|
||||||
func TestICloudPrivateRelayListHoldsItsNetworks(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
checkPublicListHolds(t, ICloudPrivateRelayList, iCloudPrivateRelayBody,
|
|
||||||
iCloudPrivateRelayAddr)
|
|
||||||
}
|
|
||||||
|
|
||||||
// checkPublicListHolds downloads body as the public list called name and
|
|
||||||
// checks that addr has one listing, which names that list.
|
|
||||||
func checkPublicListHolds(t *testing.T, name, body, addr string) {
|
|
||||||
t.Helper()
|
|
||||||
|
|
||||||
index := slices.IndexFunc(Lists(), func(list List) bool {
|
|
||||||
return list.Name == name
|
|
||||||
})
|
|
||||||
if index < 0 {
|
|
||||||
t.Fatalf("no list named %s", name)
|
|
||||||
}
|
|
||||||
|
|
||||||
list := Lists()[index]
|
|
||||||
_, list.URL = newPublisher(t, body)
|
|
||||||
b := newTestIPLists(t.TempDir(), list)
|
|
||||||
|
|
||||||
err := b.download(t.Context(), list)
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("download: %v", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
listings := lookup(t, b, addr)
|
|
||||||
if len(listings) != 1 || listings[0].Name != name {
|
|
||||||
t.Errorf("listings %+v, want one for %s", listings, name)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestPublicListsAreComplete checks that every public list has what a
|
|
||||||
// download and a lookup answer need, and a unique name.
|
|
||||||
func TestPublicListsAreComplete(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
names := make(map[string]bool)
|
|
||||||
|
|
||||||
for _, list := range Lists() {
|
|
||||||
if list.Name == "" || list.Description == "" || list.URL == "" ||
|
|
||||||
list.Interval <= 0 || names[list.Name] {
|
|
||||||
t.Errorf("list %+v is incomplete or has a name used before", list)
|
|
||||||
}
|
|
||||||
|
|
||||||
names[list.Name] = true
|
|
||||||
}
|
|
||||||
}
|
|
||||||
@@ -1,209 +0,0 @@
|
|||||||
package iplist
|
|
||||||
|
|
||||||
import (
|
|
||||||
"slices"
|
|
||||||
"time"
|
|
||||||
)
|
|
||||||
|
|
||||||
// TorExitsList is the name of the list of Tor exit nodes.
|
|
||||||
const TorExitsList = "tor_exits"
|
|
||||||
|
|
||||||
// ICloudPrivateRelayList is the name of the list of iCloud Private Relay's
|
|
||||||
// networks.
|
|
||||||
const ICloudPrivateRelayList = "icloud_private_relay"
|
|
||||||
|
|
||||||
// protonVPNIPsURL is where the ProtonVPN-IPs mirror publishes its files.
|
|
||||||
const protonVPNIPsURL = "https://raw.githubusercontent.com/tn3w/ProtonVPN-IPs/master/"
|
|
||||||
|
|
||||||
// How often each list is downloaded. No list is downloaded more often than
|
|
||||||
// its publisher updates it or allows.
|
|
||||||
const (
|
|
||||||
// DShield asks that its feeds be downloaded at most once an hour:
|
|
||||||
// https://isc.sans.edu/feeds_doc.html
|
|
||||||
dshieldInterval = time.Hour
|
|
||||||
|
|
||||||
// abuse.ch makes the list every 5 minutes and recommends a download at
|
|
||||||
// least every 15: https://feodotracker.abuse.ch/blocklist/
|
|
||||||
feodoTrackerInterval = 15 * time.Minute
|
|
||||||
|
|
||||||
// FireHOL's page for each list gives how often the list changes on
|
|
||||||
// average: about every 5 hours for abusers and every 3 hours for level
|
|
||||||
// 3. FireHOL sets no download limit.
|
|
||||||
// https://iplists.firehol.org/?ipset=firehol_abusers_30d
|
|
||||||
// https://iplists.firehol.org/?ipset=firehol_level3
|
|
||||||
fireHOLAbusersInterval = 5 * time.Hour
|
|
||||||
fireHOLLevel3Interval = 3 * time.Hour
|
|
||||||
|
|
||||||
// Apple's iCloud Private Relay file is 12 MB, so once a day.
|
|
||||||
iCloudPrivateRelayInterval = 24 * time.Hour
|
|
||||||
|
|
||||||
// Spamhaus asks that DROP be downloaded at most once a day:
|
|
||||||
// https://www.spamhaus.org/blocklists/do-not-route-or-peer/
|
|
||||||
spamhausDROPInterval = 24 * time.Hour
|
|
||||||
|
|
||||||
// Onionoo's relay data follows Tor's hourly network consensus, and
|
|
||||||
// Onionoo sets no download limit:
|
|
||||||
// https://metrics.torproject.org/onionoo.html
|
|
||||||
torExitsInterval = time.Hour
|
|
||||||
|
|
||||||
// A VPN service's servers change slowly, so once a day is enough.
|
|
||||||
vpnListInterval = 24 * time.Hour
|
|
||||||
)
|
|
||||||
|
|
||||||
// Lists returns the public IP lists routewatch downloads. Each can be
|
|
||||||
// downloaded whole without an account or a key. Spamhaus merged its EDROP
|
|
||||||
// list into DROP, so DROP covers both.
|
|
||||||
func Lists() []List {
|
|
||||||
return slices.Concat(nonVPNLists(), vpnLists())
|
|
||||||
}
|
|
||||||
|
|
||||||
// nonVPNLists returns the public lists that are not lists of VPN servers.
|
|
||||||
func nonVPNLists() []List {
|
|
||||||
return []List{
|
|
||||||
{
|
|
||||||
Name: "dshield",
|
|
||||||
Description: "In one of the 20 networks that DShield saw scan " +
|
|
||||||
"the most targets in the last three days",
|
|
||||||
URL: "https://feeds.dshield.org/block.txt",
|
|
||||||
Interval: dshieldInterval,
|
|
||||||
Format: FormatDShield,
|
|
||||||
},
|
|
||||||
{
|
|
||||||
Name: "feodo_tracker",
|
|
||||||
Description: "A botnet command-and-control server",
|
|
||||||
URL: "https://feodotracker.abuse.ch/downloads/ipblocklist.txt",
|
|
||||||
Interval: feodoTrackerInterval,
|
|
||||||
Format: FormatLines,
|
|
||||||
},
|
|
||||||
{
|
|
||||||
Name: "firehol_abusers_30d",
|
|
||||||
Description: "Reported for comment, form or forum spam in the " +
|
|
||||||
"last 30 days",
|
|
||||||
URL: "https://iplists.firehol.org/files/firehol_abusers_30d.netset",
|
|
||||||
Interval: fireHOLAbusersInterval,
|
|
||||||
Format: FormatLines,
|
|
||||||
},
|
|
||||||
{
|
|
||||||
Name: "firehol_level3",
|
|
||||||
Description: "Reported for attacks, spyware or viruses in the " +
|
|
||||||
"last 30 days",
|
|
||||||
URL: "https://iplists.firehol.org/files/firehol_level3.netset",
|
|
||||||
Interval: fireHOLLevel3Interval,
|
|
||||||
Format: FormatLines,
|
|
||||||
},
|
|
||||||
{
|
|
||||||
Name: ICloudPrivateRelayList,
|
|
||||||
Description: "In a network where iCloud Private Relay traffic leaves",
|
|
||||||
URL: "https://mask-api.icloud.com/egress-ip-ranges.csv",
|
|
||||||
Interval: iCloudPrivateRelayInterval,
|
|
||||||
Format: FormatICloudPrivateRelayCSV,
|
|
||||||
},
|
|
||||||
{
|
|
||||||
Name: "spamhaus_drop",
|
|
||||||
Description: "In an IPv4 network that Spamhaus says spammers " +
|
|
||||||
"or cyber criminals have hijacked or lease",
|
|
||||||
URL: "https://www.spamhaus.org/drop/drop_v4.json",
|
|
||||||
Interval: spamhausDROPInterval,
|
|
||||||
Format: FormatSpamhausJSON,
|
|
||||||
},
|
|
||||||
{
|
|
||||||
Name: "spamhaus_dropv6",
|
|
||||||
Description: "In an IPv6 network that Spamhaus says spammers " +
|
|
||||||
"or cyber criminals have hijacked or lease",
|
|
||||||
URL: "https://www.spamhaus.org/drop/drop_v6.json",
|
|
||||||
Interval: spamhausDROPInterval,
|
|
||||||
Format: FormatSpamhausJSON,
|
|
||||||
},
|
|
||||||
{
|
|
||||||
Name: TorExitsList,
|
|
||||||
Description: "A Tor exit node, where traffic from the Tor network leaves",
|
|
||||||
URL: "https://onionoo.torproject.org/details?" +
|
|
||||||
"flag=Exit&running=true&fields=exit_addresses,or_addresses",
|
|
||||||
Interval: torExitsInterval,
|
|
||||||
Format: FormatOnionooJSON,
|
|
||||||
},
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// vpnLists returns the public lists of VPN servers. Their names are what
|
|
||||||
// vpn_providers reports.
|
|
||||||
func vpnLists() []List {
|
|
||||||
return []List{
|
|
||||||
{
|
|
||||||
Name: "ivpn",
|
|
||||||
Description: "A server of the IVPN service",
|
|
||||||
URL: "https://api.ivpn.net/v5/servers.json",
|
|
||||||
Interval: vpnListInterval,
|
|
||||||
Format: FormatIVPNJSON,
|
|
||||||
VPN: true,
|
|
||||||
},
|
|
||||||
{
|
|
||||||
Name: "mullvad",
|
|
||||||
Description: "A server of the Mullvad VPN service",
|
|
||||||
URL: "https://api.mullvad.net/www/relays/all/",
|
|
||||||
Interval: vpnListInterval,
|
|
||||||
Format: FormatMullvadJSON,
|
|
||||||
VPN: true,
|
|
||||||
},
|
|
||||||
{
|
|
||||||
Name: "nordvpn",
|
|
||||||
Description: "A server of the NordVPN service",
|
|
||||||
URL: "https://api.nordvpn.com/v1/servers?limit=0" +
|
|
||||||
"&fields[servers.station]&fields[servers.ipv6_station]" +
|
|
||||||
"&fields[servers.hostname]",
|
|
||||||
Interval: vpnListInterval,
|
|
||||||
Format: FormatNordVPNJSON,
|
|
||||||
VPN: true,
|
|
||||||
},
|
|
||||||
{
|
|
||||||
Name: "pia",
|
|
||||||
Description: "A server of the Private Internet Access VPN service",
|
|
||||||
URL: "https://serverlist.piaservers.net/vpninfo/servers/v6",
|
|
||||||
Interval: vpnListInterval,
|
|
||||||
Format: FormatPIAJSON,
|
|
||||||
VPN: true,
|
|
||||||
},
|
|
||||||
{
|
|
||||||
// Proton's own server list needs a logged-in session, so this is
|
|
||||||
// the public ProtonVPN-IPs mirror of it. Proton publishes only
|
|
||||||
// the addresses clients connect to. The mirror's README says
|
|
||||||
// traffic leaves from others in the same /24, so this file of
|
|
||||||
// /24 networks is the one it says to use for detecting exits.
|
|
||||||
Name: "protonvpn",
|
|
||||||
Description: "In a /24 network that holds a server of the " +
|
|
||||||
"ProtonVPN service",
|
|
||||||
URL: protonVPNIPsURL + "protonvpn_ip_ranges.txt",
|
|
||||||
Interval: vpnListInterval,
|
|
||||||
Format: FormatProtonVPN,
|
|
||||||
VPN: true,
|
|
||||||
},
|
|
||||||
{
|
|
||||||
Name: "protonvpn_entry",
|
|
||||||
Description: "A server of the ProtonVPN service",
|
|
||||||
URL: protonVPNIPsURL + "protonvpn_entry_ips.txt",
|
|
||||||
Interval: vpnListInterval,
|
|
||||||
Format: FormatProtonVPN,
|
|
||||||
VPN: true,
|
|
||||||
},
|
|
||||||
{
|
|
||||||
Name: "x4bnet",
|
|
||||||
Description: "In an IPv4 network of a VPN service, according to " +
|
|
||||||
"X4BNet's community list",
|
|
||||||
URL: "https://raw.githubusercontent.com/X4BNet/lists_vpn/main/" +
|
|
||||||
"output/vpn/ipv4.txt",
|
|
||||||
Interval: vpnListInterval,
|
|
||||||
Format: FormatLines,
|
|
||||||
VPN: true,
|
|
||||||
},
|
|
||||||
{
|
|
||||||
Name: "x4bnet_v6",
|
|
||||||
Description: "In an IPv6 network of a VPN service, according to " +
|
|
||||||
"X4BNet's community list",
|
|
||||||
URL: "https://raw.githubusercontent.com/X4BNet/lists_vpn/main/" +
|
|
||||||
"output/vpn/ipv6.txt",
|
|
||||||
Interval: vpnListInterval,
|
|
||||||
Format: FormatLines,
|
|
||||||
VPN: true,
|
|
||||||
},
|
|
||||||
}
|
|
||||||
}
|
|
||||||
@@ -1,435 +0,0 @@
|
|||||||
package iplist
|
|
||||||
|
|
||||||
import (
|
|
||||||
"bufio"
|
|
||||||
"encoding/json"
|
|
||||||
"errors"
|
|
||||||
"fmt"
|
|
||||||
"io"
|
|
||||||
"net"
|
|
||||||
"net/netip"
|
|
||||||
"slices"
|
|
||||||
"strings"
|
|
||||||
|
|
||||||
"go4.org/netipx"
|
|
||||||
)
|
|
||||||
|
|
||||||
const (
|
|
||||||
// commentPrefix starts a comment line in a list file.
|
|
||||||
commentPrefix = "#"
|
|
||||||
|
|
||||||
// networkSeparator separates the address and the mask length of a
|
|
||||||
// network, as in 192.0.2.0/24.
|
|
||||||
networkSeparator = "/"
|
|
||||||
|
|
||||||
// dshieldFieldSeparator separates the fields of a DShield line.
|
|
||||||
dshieldFieldSeparator = "\t"
|
|
||||||
|
|
||||||
// iCloudPrivateRelayFieldSeparator separates the fields of a line of
|
|
||||||
// Apple's iCloud Private Relay file.
|
|
||||||
iCloudPrivateRelayFieldSeparator = ","
|
|
||||||
|
|
||||||
// spamhausMetadataType is the type of the line that ends a Spamhaus
|
|
||||||
// file. That line holds the file's metadata and no network.
|
|
||||||
spamhausMetadataType = "metadata"
|
|
||||||
)
|
|
||||||
|
|
||||||
var (
|
|
||||||
errUnknownFormat = errors.New("unknown list format")
|
|
||||||
errNoNetwork = errors.New("no network on the line")
|
|
||||||
errBadRange = errors.New("first and last address make no range")
|
|
||||||
errNoAddresses = errors.New("no addresses in the document")
|
|
||||||
)
|
|
||||||
|
|
||||||
// parse reads a list file written in format and returns the addresses it
|
|
||||||
// holds. Anything it cannot read fails the whole file, so a file that is not
|
|
||||||
// the list, such as an error page, never replaces a good copy.
|
|
||||||
func parse(r io.Reader, format Format) (*netipx.IPSet, error) {
|
|
||||||
switch format {
|
|
||||||
case FormatLines:
|
|
||||||
return parseLines(r, addAddressOrNetwork)
|
|
||||||
case FormatSpamhausJSON:
|
|
||||||
return parseLines(r, addSpamhausNetwork)
|
|
||||||
case FormatDShield:
|
|
||||||
return parseLines(r, addDShieldNetwork)
|
|
||||||
case FormatOnionooJSON:
|
|
||||||
return parseOnionoo(r)
|
|
||||||
case FormatMullvadJSON:
|
|
||||||
return parseMullvad(r)
|
|
||||||
case FormatNordVPNJSON:
|
|
||||||
return parseNordVPN(r)
|
|
||||||
case FormatPIAJSON:
|
|
||||||
return parsePIA(r)
|
|
||||||
case FormatIVPNJSON:
|
|
||||||
return parseIVPN(r)
|
|
||||||
case FormatProtonVPN:
|
|
||||||
return parseLinesNotEmpty(r, addAddressOrNetwork)
|
|
||||||
case FormatICloudPrivateRelayCSV:
|
|
||||||
return parseLinesNotEmpty(r, addICloudPrivateRelayNetwork)
|
|
||||||
default:
|
|
||||||
return nil, errUnknownFormat
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// parseLines reads a list file line by line, skipping blank lines and
|
|
||||||
// comments, and calls addLine with each other line.
|
|
||||||
func parseLines(
|
|
||||||
r io.Reader, addLine func(*netipx.IPSetBuilder, string) error,
|
|
||||||
) (*netipx.IPSet, error) {
|
|
||||||
var builder netipx.IPSetBuilder
|
|
||||||
|
|
||||||
scanner := bufio.NewScanner(r)
|
|
||||||
|
|
||||||
for lineNumber := 1; scanner.Scan(); lineNumber++ {
|
|
||||||
line := strings.TrimSpace(scanner.Text())
|
|
||||||
if line == "" || strings.HasPrefix(line, commentPrefix) {
|
|
||||||
continue
|
|
||||||
}
|
|
||||||
|
|
||||||
err := addLine(&builder, line)
|
|
||||||
if err != nil {
|
|
||||||
return nil, fmt.Errorf("line %d: %w", lineNumber, err)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
err := scanner.Err()
|
|
||||||
if err != nil {
|
|
||||||
return nil, err
|
|
||||||
}
|
|
||||||
|
|
||||||
return builder.IPSet()
|
|
||||||
}
|
|
||||||
|
|
||||||
// addAddressOrNetwork adds an address, such as 192.0.2.1, or a network, such
|
|
||||||
// as 192.0.2.0/24, to builder.
|
|
||||||
func addAddressOrNetwork(builder *netipx.IPSetBuilder, text string) error {
|
|
||||||
if strings.Contains(text, networkSeparator) {
|
|
||||||
network, err := netip.ParsePrefix(text)
|
|
||||||
if err != nil {
|
|
||||||
return err
|
|
||||||
}
|
|
||||||
|
|
||||||
builder.AddPrefix(network)
|
|
||||||
|
|
||||||
return nil
|
|
||||||
}
|
|
||||||
|
|
||||||
addr, err := netip.ParseAddr(text)
|
|
||||||
if err != nil {
|
|
||||||
return err
|
|
||||||
}
|
|
||||||
|
|
||||||
builder.Add(addr)
|
|
||||||
|
|
||||||
return nil
|
|
||||||
}
|
|
||||||
|
|
||||||
// addSpamhausNetwork adds the network on a line of a Spamhaus file to
|
|
||||||
// builder. The metadata line adds nothing.
|
|
||||||
func addSpamhausNetwork(builder *netipx.IPSetBuilder, line string) error {
|
|
||||||
var entry struct {
|
|
||||||
CIDR string `json:"cidr"`
|
|
||||||
Type string `json:"type"`
|
|
||||||
}
|
|
||||||
|
|
||||||
err := json.Unmarshal([]byte(line), &entry)
|
|
||||||
if err != nil {
|
|
||||||
return err
|
|
||||||
}
|
|
||||||
|
|
||||||
if entry.Type == spamhausMetadataType {
|
|
||||||
return nil
|
|
||||||
}
|
|
||||||
|
|
||||||
network, err := netip.ParsePrefix(entry.CIDR)
|
|
||||||
if err != nil {
|
|
||||||
return err
|
|
||||||
}
|
|
||||||
|
|
||||||
builder.AddPrefix(network)
|
|
||||||
|
|
||||||
return nil
|
|
||||||
}
|
|
||||||
|
|
||||||
// addDShieldNetwork adds the network on a line of a DShield file to builder.
|
|
||||||
// The line's first two fields are the network's first and last address.
|
|
||||||
func addDShieldNetwork(builder *netipx.IPSetBuilder, line string) error {
|
|
||||||
firstField, rest, found := strings.Cut(line, dshieldFieldSeparator)
|
|
||||||
if !found {
|
|
||||||
return errNoNetwork
|
|
||||||
}
|
|
||||||
|
|
||||||
lastField, _, _ := strings.Cut(rest, dshieldFieldSeparator)
|
|
||||||
|
|
||||||
first, err := netip.ParseAddr(firstField)
|
|
||||||
if err != nil {
|
|
||||||
return err
|
|
||||||
}
|
|
||||||
|
|
||||||
last, err := netip.ParseAddr(lastField)
|
|
||||||
if err != nil {
|
|
||||||
return err
|
|
||||||
}
|
|
||||||
|
|
||||||
network := netipx.IPRangeFrom(first, last)
|
|
||||||
if !network.IsValid() {
|
|
||||||
return errBadRange
|
|
||||||
}
|
|
||||||
|
|
||||||
builder.AddRange(network)
|
|
||||||
|
|
||||||
return nil
|
|
||||||
}
|
|
||||||
|
|
||||||
// parseOnionoo returns every address in the exit_addresses and or_addresses
|
|
||||||
// fields of the relays in an Onionoo details document. A document with no
|
|
||||||
// addresses fails, because the Tor network always has relays.
|
|
||||||
func parseOnionoo(r io.Reader) (*netipx.IPSet, error) {
|
|
||||||
//nolint:tagliatelle // Onionoo's own field names
|
|
||||||
var details struct {
|
|
||||||
Relays []struct {
|
|
||||||
ExitAddresses []string `json:"exit_addresses"`
|
|
||||||
ORAddresses []string `json:"or_addresses"`
|
|
||||||
} `json:"relays"`
|
|
||||||
}
|
|
||||||
|
|
||||||
err := readJSON(r, &details)
|
|
||||||
if err != nil {
|
|
||||||
return nil, err
|
|
||||||
}
|
|
||||||
|
|
||||||
var builder netipx.IPSetBuilder
|
|
||||||
|
|
||||||
for _, relay := range details.Relays {
|
|
||||||
for _, text := range relay.ExitAddresses {
|
|
||||||
err = addAddress(&builder, text)
|
|
||||||
if err != nil {
|
|
||||||
return nil, err
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
for _, text := range relay.ORAddresses {
|
|
||||||
err = addAddressWithPort(&builder, text)
|
|
||||||
if err != nil {
|
|
||||||
return nil, err
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
addresses, err := builder.IPSet()
|
|
||||||
if err != nil {
|
|
||||||
return nil, err
|
|
||||||
}
|
|
||||||
|
|
||||||
if len(addresses.Ranges()) == 0 {
|
|
||||||
return nil, errNoAddresses
|
|
||||||
}
|
|
||||||
|
|
||||||
return addresses, nil
|
|
||||||
}
|
|
||||||
|
|
||||||
// parseMullvad returns the addresses of the relays in a Mullvad document. A
|
|
||||||
// relay with no IPv6 address has null for it, and only bridges have
|
|
||||||
// ipv4_v2ray.
|
|
||||||
func parseMullvad(r io.Reader) (*netipx.IPSet, error) {
|
|
||||||
//nolint:tagliatelle // Mullvad's own field names
|
|
||||||
var relays []struct {
|
|
||||||
IPv4 string `json:"ipv4_addr_in"`
|
|
||||||
IPv6 string `json:"ipv6_addr_in"`
|
|
||||||
IPv4V2Ray string `json:"ipv4_v2ray"`
|
|
||||||
}
|
|
||||||
|
|
||||||
err := readJSON(r, &relays)
|
|
||||||
if err != nil {
|
|
||||||
return nil, err
|
|
||||||
}
|
|
||||||
|
|
||||||
var texts []string
|
|
||||||
|
|
||||||
for _, relay := range relays {
|
|
||||||
texts = append(texts, relay.IPv4, relay.IPv6, relay.IPv4V2Ray)
|
|
||||||
}
|
|
||||||
|
|
||||||
return addressSet(texts)
|
|
||||||
}
|
|
||||||
|
|
||||||
// parseNordVPN returns the addresses of the servers in a NordVPN document. A
|
|
||||||
// server with no IPv6 address has an empty ipv6_station.
|
|
||||||
func parseNordVPN(r io.Reader) (*netipx.IPSet, error) {
|
|
||||||
//nolint:tagliatelle // NordVPN's own field names
|
|
||||||
var servers []struct {
|
|
||||||
Station string `json:"station"`
|
|
||||||
IPv6Station string `json:"ipv6_station"`
|
|
||||||
}
|
|
||||||
|
|
||||||
err := readJSON(r, &servers)
|
|
||||||
if err != nil {
|
|
||||||
return nil, err
|
|
||||||
}
|
|
||||||
|
|
||||||
var texts []string
|
|
||||||
|
|
||||||
for _, server := range servers {
|
|
||||||
texts = append(texts, server.Station, server.IPv6Station)
|
|
||||||
}
|
|
||||||
|
|
||||||
return addressSet(texts)
|
|
||||||
}
|
|
||||||
|
|
||||||
// parsePIA returns the addresses of the servers in a Private Internet Access
|
|
||||||
// document. json.Decoder stops at the end of the JSON, so the signature after
|
|
||||||
// it is not read.
|
|
||||||
func parsePIA(r io.Reader) (*netipx.IPSet, error) {
|
|
||||||
var list struct {
|
|
||||||
Regions []struct {
|
|
||||||
Servers map[string][]struct {
|
|
||||||
IP string `json:"ip"`
|
|
||||||
} `json:"servers"`
|
|
||||||
} `json:"regions"`
|
|
||||||
}
|
|
||||||
|
|
||||||
err := json.NewDecoder(r).Decode(&list)
|
|
||||||
if err != nil {
|
|
||||||
return nil, err
|
|
||||||
}
|
|
||||||
|
|
||||||
var texts []string
|
|
||||||
|
|
||||||
for _, region := range list.Regions {
|
|
||||||
for _, servers := range region.Servers {
|
|
||||||
for _, server := range servers {
|
|
||||||
texts = append(texts, server.IP)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
return addressSet(texts)
|
|
||||||
}
|
|
||||||
|
|
||||||
// parseIVPN returns the addresses of the hosts in an IVPN document. A host's
|
|
||||||
// ipv6 object holds only an address inside the VPN, so it is not read.
|
|
||||||
func parseIVPN(r io.Reader) (*netipx.IPSet, error) {
|
|
||||||
type gateway struct {
|
|
||||||
Hosts []struct {
|
|
||||||
Host string `json:"host"`
|
|
||||||
V2Ray string `json:"v2ray"`
|
|
||||||
} `json:"hosts"`
|
|
||||||
}
|
|
||||||
|
|
||||||
var list struct {
|
|
||||||
WireGuard []gateway `json:"wireguard"`
|
|
||||||
OpenVPN []gateway `json:"openvpn"`
|
|
||||||
}
|
|
||||||
|
|
||||||
err := readJSON(r, &list)
|
|
||||||
if err != nil {
|
|
||||||
return nil, err
|
|
||||||
}
|
|
||||||
|
|
||||||
var texts []string
|
|
||||||
|
|
||||||
for _, gw := range slices.Concat(list.WireGuard, list.OpenVPN) {
|
|
||||||
for _, host := range gw.Hosts {
|
|
||||||
texts = append(texts, host.Host, host.V2Ray)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
return addressSet(texts)
|
|
||||||
}
|
|
||||||
|
|
||||||
// parseLinesNotEmpty is parseLines for a list that always holds addresses, so
|
|
||||||
// a file with none fails.
|
|
||||||
func parseLinesNotEmpty(
|
|
||||||
r io.Reader, addLine func(*netipx.IPSetBuilder, string) error,
|
|
||||||
) (*netipx.IPSet, error) {
|
|
||||||
addresses, err := parseLines(r, addLine)
|
|
||||||
if err != nil {
|
|
||||||
return nil, err
|
|
||||||
}
|
|
||||||
|
|
||||||
if len(addresses.Ranges()) == 0 {
|
|
||||||
return nil, errNoAddresses
|
|
||||||
}
|
|
||||||
|
|
||||||
return addresses, nil
|
|
||||||
}
|
|
||||||
|
|
||||||
// addICloudPrivateRelayNetwork adds the network at the start of a line of
|
|
||||||
// Apple's iCloud Private Relay file to builder.
|
|
||||||
func addICloudPrivateRelayNetwork(builder *netipx.IPSetBuilder, line string) error {
|
|
||||||
field, _, found := strings.Cut(line, iCloudPrivateRelayFieldSeparator)
|
|
||||||
if !found {
|
|
||||||
return errNoNetwork
|
|
||||||
}
|
|
||||||
|
|
||||||
network, err := netip.ParsePrefix(field)
|
|
||||||
if err != nil {
|
|
||||||
return err
|
|
||||||
}
|
|
||||||
|
|
||||||
builder.AddPrefix(network)
|
|
||||||
|
|
||||||
return nil
|
|
||||||
}
|
|
||||||
|
|
||||||
// readJSON reads r to its end and decodes it, as one JSON document, into v.
|
|
||||||
func readJSON(r io.Reader, v any) error {
|
|
||||||
data, err := io.ReadAll(r)
|
|
||||||
if err != nil {
|
|
||||||
return err
|
|
||||||
}
|
|
||||||
|
|
||||||
return json.Unmarshal(data, v)
|
|
||||||
}
|
|
||||||
|
|
||||||
// addressSet returns the set of the addresses in texts, skipping empty ones.
|
|
||||||
// A list with no addresses fails, because every VPN service has servers.
|
|
||||||
func addressSet(texts []string) (*netipx.IPSet, error) {
|
|
||||||
var builder netipx.IPSetBuilder
|
|
||||||
|
|
||||||
for _, text := range texts {
|
|
||||||
if text == "" {
|
|
||||||
continue
|
|
||||||
}
|
|
||||||
|
|
||||||
err := addAddress(&builder, text)
|
|
||||||
if err != nil {
|
|
||||||
return nil, err
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
addresses, err := builder.IPSet()
|
|
||||||
if err != nil {
|
|
||||||
return nil, err
|
|
||||||
}
|
|
||||||
|
|
||||||
if len(addresses.Ranges()) == 0 {
|
|
||||||
return nil, errNoAddresses
|
|
||||||
}
|
|
||||||
|
|
||||||
return addresses, nil
|
|
||||||
}
|
|
||||||
|
|
||||||
// addAddress adds an address, such as 192.0.2.1, to builder.
|
|
||||||
func addAddress(builder *netipx.IPSetBuilder, text string) error {
|
|
||||||
addr, err := netip.ParseAddr(text)
|
|
||||||
if err != nil {
|
|
||||||
return err
|
|
||||||
}
|
|
||||||
|
|
||||||
builder.Add(addr)
|
|
||||||
|
|
||||||
return nil
|
|
||||||
}
|
|
||||||
|
|
||||||
// addAddressWithPort adds the address of text, such as 192.0.2.1:443 or
|
|
||||||
// [2001:db8::1]:443, to builder. Onionoo allows a list of ports after the
|
|
||||||
// address, so the part after the address is not read.
|
|
||||||
func addAddressWithPort(builder *netipx.IPSetBuilder, text string) error {
|
|
||||||
host, _, err := net.SplitHostPort(text)
|
|
||||||
if err != nil {
|
|
||||||
return err
|
|
||||||
}
|
|
||||||
|
|
||||||
return addAddress(builder, host)
|
|
||||||
}
|
|
||||||
@@ -1,324 +0,0 @@
|
|||||||
// Package ptr looks up the PTR names of addresses over DNS-over-HTTPS. It
|
|
||||||
// keeps answers in memory for as long as DNS allows, and in cache.db so that
|
|
||||||
// the answers survive a restart.
|
|
||||||
package ptr
|
|
||||||
|
|
||||||
import (
|
|
||||||
"context"
|
|
||||||
"database/sql"
|
|
||||||
"fmt"
|
|
||||||
"log/slog"
|
|
||||||
"maps"
|
|
||||||
"net/http"
|
|
||||||
"net/netip"
|
|
||||||
"slices"
|
|
||||||
"strings"
|
|
||||||
"sync"
|
|
||||||
"time"
|
|
||||||
|
|
||||||
"git.eeqj.de/sneak/routewatch/internal/bogon"
|
|
||||||
"github.com/miekg/dns"
|
|
||||||
)
|
|
||||||
|
|
||||||
const (
|
|
||||||
// flushInterval is how often new answers are written to cache.db.
|
|
||||||
flushInterval = 5 * time.Second
|
|
||||||
|
|
||||||
// sweepInterval is how often expired answers are deleted.
|
|
||||||
sweepInterval = 10 * time.Minute
|
|
||||||
|
|
||||||
// nameSeparator separates the names in a row of the hostnames table.
|
|
||||||
nameSeparator = " "
|
|
||||||
|
|
||||||
// maxAddresses limits how many addresses the cache holds, so that a public
|
|
||||||
// lookup cannot grow memory and cache.db without bound.
|
|
||||||
maxAddresses = 250_000
|
|
||||||
)
|
|
||||||
|
|
||||||
// Params is what New needs.
|
|
||||||
type Params struct {
|
|
||||||
// DB is the open cache.db.
|
|
||||||
DB *sql.DB
|
|
||||||
|
|
||||||
// Resolvers are the URLs of the DNS-over-HTTPS resolvers to ask, in order.
|
|
||||||
Resolvers []string
|
|
||||||
|
|
||||||
Logger *slog.Logger
|
|
||||||
}
|
|
||||||
|
|
||||||
// entry is one address's cached answer.
|
|
||||||
type entry struct {
|
|
||||||
// names are the address's PTR names, nil when it has none.
|
|
||||||
names []string
|
|
||||||
expires time.Time
|
|
||||||
}
|
|
||||||
|
|
||||||
// Cache answers PTR lookups from memory, and asks the resolvers for an address
|
|
||||||
// it holds no current answer for. Start loads it from cache.db.
|
|
||||||
type Cache struct {
|
|
||||||
db *sql.DB
|
|
||||||
resolvers []string
|
|
||||||
timeLimit time.Duration
|
|
||||||
maxEntries int
|
|
||||||
logger *slog.Logger
|
|
||||||
client *http.Client
|
|
||||||
now func() time.Time
|
|
||||||
|
|
||||||
// entries holds the kept answers by address, at most maxEntries of them.
|
|
||||||
// pending holds those that cache.db does not have yet.
|
|
||||||
mu sync.RWMutex
|
|
||||||
entries map[string]entry
|
|
||||||
pending map[string]entry
|
|
||||||
|
|
||||||
// stop ends the goroutine Start runs, and done is closed when it returns.
|
|
||||||
stop context.CancelFunc
|
|
||||||
done chan struct{}
|
|
||||||
}
|
|
||||||
|
|
||||||
// New returns an empty Cache. Start loads it.
|
|
||||||
func New(params Params) *Cache {
|
|
||||||
return &Cache{
|
|
||||||
db: params.DB,
|
|
||||||
resolvers: params.Resolvers,
|
|
||||||
timeLimit: LookupTimeout,
|
|
||||||
maxEntries: maxAddresses,
|
|
||||||
logger: params.Logger.With("component", "ptr"),
|
|
||||||
client: &http.Client{},
|
|
||||||
now: time.Now,
|
|
||||||
entries: make(map[string]entry),
|
|
||||||
pending: make(map[string]entry),
|
|
||||||
done: make(chan struct{}),
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// LookupAddr returns the PTR names of addr, fully qualified, or none when it
|
|
||||||
// has none. A reserved address has none, and is not sent to the resolvers. A
|
|
||||||
// lookup takes at most LookupTimeout. A cached answer makes no outside request.
|
|
||||||
// It returns an error when no resolver answered, and nothing is cached then.
|
|
||||||
func (c *Cache) LookupAddr(ctx context.Context, addr string) ([]string, error) {
|
|
||||||
parsed, err := netip.ParseAddr(addr)
|
|
||||||
if err != nil {
|
|
||||||
return nil, fmt.Errorf("parsing the address: %w", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
_, reserved := bogon.Addr(parsed)
|
|
||||||
if reserved {
|
|
||||||
return []string{}, nil
|
|
||||||
}
|
|
||||||
|
|
||||||
address := parsed.Unmap().String()
|
|
||||||
|
|
||||||
cached, ok := c.current(address)
|
|
||||||
if ok {
|
|
||||||
return slices.Clone(cached.names), nil
|
|
||||||
}
|
|
||||||
|
|
||||||
reverseName, err := dns.ReverseAddr(address)
|
|
||||||
if err != nil {
|
|
||||||
return nil, fmt.Errorf("building the reverse name: %w", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
result, err := c.lookup(ctx, reverseName)
|
|
||||||
if err != nil {
|
|
||||||
return nil, err
|
|
||||||
}
|
|
||||||
|
|
||||||
if result.ttl > 0 {
|
|
||||||
c.store(address, entry{names: result.names, expires: c.now().Add(result.ttl)})
|
|
||||||
}
|
|
||||||
|
|
||||||
return slices.Clone(result.names), nil
|
|
||||||
}
|
|
||||||
|
|
||||||
// Start loads the answers that have not expired from cache.db. It then starts
|
|
||||||
// the goroutine that writes new answers there and deletes expired ones. The
|
|
||||||
// goroutine runs until Stop: the end of ctx does not stop it.
|
|
||||||
func (c *Cache) Start(ctx context.Context) error {
|
|
||||||
err := c.load(ctx)
|
|
||||||
if err != nil {
|
|
||||||
return err
|
|
||||||
}
|
|
||||||
|
|
||||||
c.logger.Info("PTR cache loaded", "entries", len(c.entries))
|
|
||||||
|
|
||||||
ctx, c.stop = context.WithCancel(context.WithoutCancel(ctx))
|
|
||||||
|
|
||||||
go c.run(ctx)
|
|
||||||
|
|
||||||
return nil
|
|
||||||
}
|
|
||||||
|
|
||||||
// Stop ends the goroutine, which first writes the answers cache.db does not
|
|
||||||
// have yet, and waits until it has returned or ctx ends.
|
|
||||||
func (c *Cache) Stop(ctx context.Context) {
|
|
||||||
c.stop()
|
|
||||||
|
|
||||||
select {
|
|
||||||
case <-c.done:
|
|
||||||
c.logger.Info("PTR cache stopped")
|
|
||||||
case <-ctx.Done():
|
|
||||||
c.logger.Warn("Stop deadline passed before the PTR cache was written")
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// current returns the answer held for address, unless it has expired.
|
|
||||||
func (c *Cache) current(address string) (entry, bool) {
|
|
||||||
c.mu.RLock()
|
|
||||||
defer c.mu.RUnlock()
|
|
||||||
|
|
||||||
found, ok := c.entries[address]
|
|
||||||
if !ok || !c.now().Before(found.expires) {
|
|
||||||
return entry{}, false
|
|
||||||
}
|
|
||||||
|
|
||||||
return found, true
|
|
||||||
}
|
|
||||||
|
|
||||||
// store keeps an answer in memory and queues it for cache.db. Once entries
|
|
||||||
// holds maxEntries addresses it keeps no new address. It still replaces the
|
|
||||||
// answer of an address already held.
|
|
||||||
func (c *Cache) store(address string, answer entry) {
|
|
||||||
c.mu.Lock()
|
|
||||||
defer c.mu.Unlock()
|
|
||||||
|
|
||||||
_, held := c.entries[address]
|
|
||||||
if !held && len(c.entries) >= c.maxEntries {
|
|
||||||
return
|
|
||||||
}
|
|
||||||
|
|
||||||
c.entries[address] = answer
|
|
||||||
c.pending[address] = answer
|
|
||||||
}
|
|
||||||
|
|
||||||
// run writes new answers and deletes expired ones until ctx ends.
|
|
||||||
func (c *Cache) run(ctx context.Context) {
|
|
||||||
defer close(c.done)
|
|
||||||
|
|
||||||
flushes := time.NewTicker(flushInterval)
|
|
||||||
defer flushes.Stop()
|
|
||||||
|
|
||||||
sweeps := time.NewTicker(sweepInterval)
|
|
||||||
defer sweeps.Stop()
|
|
||||||
|
|
||||||
for {
|
|
||||||
select {
|
|
||||||
case <-ctx.Done():
|
|
||||||
c.flush(context.WithoutCancel(ctx))
|
|
||||||
|
|
||||||
return
|
|
||||||
case <-flushes.C:
|
|
||||||
c.flush(ctx)
|
|
||||||
case <-sweeps.C:
|
|
||||||
c.sweep(ctx)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// load reads the rows that have not expired into memory.
|
|
||||||
func (c *Cache) load(ctx context.Context) error {
|
|
||||||
rows, err := c.db.QueryContext(ctx,
|
|
||||||
"SELECT address, names, expires_at FROM hostnames WHERE expires_at > ?",
|
|
||||||
c.now().Unix())
|
|
||||||
if err != nil {
|
|
||||||
return fmt.Errorf("reading the PTR cache: %w", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
defer func() { _ = rows.Close() }()
|
|
||||||
|
|
||||||
for rows.Next() {
|
|
||||||
var (
|
|
||||||
address, names string
|
|
||||||
expires int64
|
|
||||||
)
|
|
||||||
|
|
||||||
err = rows.Scan(&address, &names, &expires)
|
|
||||||
if err != nil {
|
|
||||||
return fmt.Errorf("reading a PTR cache row: %w", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
c.entries[address] = entry{
|
|
||||||
names: strings.Fields(names),
|
|
||||||
expires: time.Unix(expires, 0).UTC(),
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
err = rows.Err()
|
|
||||||
if err != nil {
|
|
||||||
return fmt.Errorf("reading the PTR cache: %w", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
return nil
|
|
||||||
}
|
|
||||||
|
|
||||||
// flush writes the pending answers to cache.db. Those it cannot write stay
|
|
||||||
// pending, unless a newer answer for the address has arrived meanwhile.
|
|
||||||
func (c *Cache) flush(ctx context.Context) {
|
|
||||||
c.mu.Lock()
|
|
||||||
batch := c.pending
|
|
||||||
c.pending = make(map[string]entry)
|
|
||||||
c.mu.Unlock()
|
|
||||||
|
|
||||||
if len(batch) == 0 {
|
|
||||||
return
|
|
||||||
}
|
|
||||||
|
|
||||||
err := c.writeRows(ctx, batch)
|
|
||||||
if err == nil {
|
|
||||||
return
|
|
||||||
}
|
|
||||||
|
|
||||||
c.logger.Error("Failed to write the PTR cache", "error", err)
|
|
||||||
|
|
||||||
c.mu.Lock()
|
|
||||||
defer c.mu.Unlock()
|
|
||||||
|
|
||||||
for address, answer := range batch {
|
|
||||||
_, newer := c.pending[address]
|
|
||||||
if !newer {
|
|
||||||
c.pending[address] = answer
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// writeRows writes batch to cache.db in one transaction.
|
|
||||||
func (c *Cache) writeRows(ctx context.Context, batch map[string]entry) error {
|
|
||||||
tx, err := c.db.BeginTx(ctx, nil)
|
|
||||||
if err != nil {
|
|
||||||
return err
|
|
||||||
}
|
|
||||||
|
|
||||||
defer func() { _ = tx.Rollback() }()
|
|
||||||
|
|
||||||
for address, answer := range batch {
|
|
||||||
_, err = tx.ExecContext(ctx,
|
|
||||||
"INSERT OR REPLACE INTO hostnames (address, names, expires_at) "+
|
|
||||||
"VALUES (?, ?, ?)",
|
|
||||||
address,
|
|
||||||
strings.Join(answer.names, nameSeparator),
|
|
||||||
answer.expires.Unix())
|
|
||||||
if err != nil {
|
|
||||||
return err
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
return tx.Commit()
|
|
||||||
}
|
|
||||||
|
|
||||||
// sweep deletes the expired answers from memory and cache.db.
|
|
||||||
func (c *Cache) sweep(ctx context.Context) {
|
|
||||||
now := c.now()
|
|
||||||
expired := func(_ string, answer entry) bool {
|
|
||||||
return !now.Before(answer.expires)
|
|
||||||
}
|
|
||||||
|
|
||||||
c.mu.Lock()
|
|
||||||
maps.DeleteFunc(c.entries, expired)
|
|
||||||
maps.DeleteFunc(c.pending, expired)
|
|
||||||
c.mu.Unlock()
|
|
||||||
|
|
||||||
_, err := c.db.ExecContext(ctx,
|
|
||||||
"DELETE FROM hostnames WHERE expires_at <= ?", now.Unix())
|
|
||||||
if err != nil {
|
|
||||||
c.logger.Error("Failed to delete expired PTR cache rows", "error", err)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
@@ -1,951 +0,0 @@
|
|||||||
package ptr
|
|
||||||
|
|
||||||
import (
|
|
||||||
"context"
|
|
||||||
"database/sql"
|
|
||||||
"errors"
|
|
||||||
"io"
|
|
||||||
"log/slog"
|
|
||||||
"net/http"
|
|
||||||
"net/http/httptest"
|
|
||||||
"slices"
|
|
||||||
"strconv"
|
|
||||||
"sync"
|
|
||||||
"testing"
|
|
||||||
"testing/synctest"
|
|
||||||
"time"
|
|
||||||
|
|
||||||
"git.eeqj.de/sneak/routewatch/internal/cachedb"
|
|
||||||
"github.com/miekg/dns"
|
|
||||||
)
|
|
||||||
|
|
||||||
// These addresses are public, because a reserved address is not sent to the
|
|
||||||
// resolvers. That rules out the documentation ranges.
|
|
||||||
const (
|
|
||||||
testAddress = "8.8.4.4"
|
|
||||||
testReverse = "4.4.8.8.in-addr.arpa."
|
|
||||||
mappedAddress = "::ffff:" + testAddress
|
|
||||||
otherAddress = "8.8.8.8"
|
|
||||||
testV6Address = "2606:4700:4700::1111"
|
|
||||||
testV6Reverse = "1.1.1.1.0.0.0.0.0.0.0.0.0.0.0.0." +
|
|
||||||
"0.0.0.0.0.0.7.4.0.0.7.4.6.0.6.2.ip6.arpa."
|
|
||||||
)
|
|
||||||
|
|
||||||
const (
|
|
||||||
testName = "host.example.com."
|
|
||||||
otherName = "other.example.com."
|
|
||||||
|
|
||||||
// testTimeLimit replaces LookupTimeout in the tests that use a real
|
|
||||||
// server. A stall on a busy host cannot use it up, and no test waits for
|
|
||||||
// it.
|
|
||||||
testTimeLimit = 2 * time.Minute
|
|
||||||
)
|
|
||||||
|
|
||||||
// classlessOwner is the name that holds the PTR records of a CNAME chain.
|
|
||||||
const classlessOwner = "4.0-25.4.8.8.in-addr.arpa."
|
|
||||||
|
|
||||||
// fixedTime returns the time a test's clock starts at.
|
|
||||||
func fixedTime() time.Time {
|
|
||||||
return time.Date(2026, time.January, 2, 3, 4, 5, 0, time.UTC)
|
|
||||||
}
|
|
||||||
|
|
||||||
// fakeResolver is a DNS-over-HTTPS server. It answers each query with reply's
|
|
||||||
// result, and counts its requests.
|
|
||||||
type fakeResolver struct {
|
|
||||||
server *httptest.Server
|
|
||||||
|
|
||||||
mu sync.Mutex
|
|
||||||
requests int
|
|
||||||
reply func(http.ResponseWriter, *http.Request, *dns.Msg)
|
|
||||||
}
|
|
||||||
|
|
||||||
// newFakeResolver starts a fake resolver that answers with reply.
|
|
||||||
func newFakeResolver(
|
|
||||||
t *testing.T,
|
|
||||||
reply func(http.ResponseWriter, *http.Request, *dns.Msg),
|
|
||||||
) *fakeResolver {
|
|
||||||
t.Helper()
|
|
||||||
|
|
||||||
resolver := &fakeResolver{reply: reply}
|
|
||||||
resolver.server = httptest.NewServer(http.HandlerFunc(resolver.serve))
|
|
||||||
t.Cleanup(resolver.server.Close)
|
|
||||||
|
|
||||||
return resolver
|
|
||||||
}
|
|
||||||
|
|
||||||
// serve checks that the request is RFC 8484 wire format and passes the
|
|
||||||
// question on to the current reply.
|
|
||||||
func (f *fakeResolver) serve(w http.ResponseWriter, r *http.Request) {
|
|
||||||
f.mu.Lock()
|
|
||||||
f.requests++
|
|
||||||
reply := f.reply
|
|
||||||
f.mu.Unlock()
|
|
||||||
|
|
||||||
body, err := io.ReadAll(r.Body)
|
|
||||||
if err != nil {
|
|
||||||
http.Error(w, err.Error(), http.StatusBadRequest)
|
|
||||||
|
|
||||||
return
|
|
||||||
}
|
|
||||||
|
|
||||||
var query dns.Msg
|
|
||||||
|
|
||||||
err = query.Unpack(body)
|
|
||||||
if err != nil || r.Header.Get("Content-Type") != dnsMessageType ||
|
|
||||||
r.Header.Get("Accept") != dnsMessageType || r.Method != http.MethodPost {
|
|
||||||
http.Error(w, "not an RFC 8484 request", http.StatusBadRequest)
|
|
||||||
|
|
||||||
return
|
|
||||||
}
|
|
||||||
|
|
||||||
reply(w, r, &query)
|
|
||||||
}
|
|
||||||
|
|
||||||
func (f *fakeResolver) setReply(
|
|
||||||
reply func(http.ResponseWriter, *http.Request, *dns.Msg),
|
|
||||||
) {
|
|
||||||
f.mu.Lock()
|
|
||||||
defer f.mu.Unlock()
|
|
||||||
|
|
||||||
f.reply = reply
|
|
||||||
}
|
|
||||||
|
|
||||||
func (f *fakeResolver) requestCount() int {
|
|
||||||
f.mu.Lock()
|
|
||||||
defer f.mu.Unlock()
|
|
||||||
|
|
||||||
return f.requests
|
|
||||||
}
|
|
||||||
|
|
||||||
// writeReply packs reply and writes it as the answer to query.
|
|
||||||
func writeReply(w http.ResponseWriter, query, reply *dns.Msg) {
|
|
||||||
reply.SetReply(query)
|
|
||||||
|
|
||||||
wire, err := reply.Pack()
|
|
||||||
if err != nil {
|
|
||||||
http.Error(w, err.Error(), http.StatusInternalServerError)
|
|
||||||
|
|
||||||
return
|
|
||||||
}
|
|
||||||
|
|
||||||
w.Header().Set("Content-Type", dnsMessageType)
|
|
||||||
_, _ = w.Write(wire)
|
|
||||||
}
|
|
||||||
|
|
||||||
// ptrReply answers with one PTR record for each name, each with the TTL ttl.
|
|
||||||
func ptrReply(
|
|
||||||
ttl uint32, names ...string,
|
|
||||||
) func(http.ResponseWriter, *http.Request, *dns.Msg) {
|
|
||||||
return func(w http.ResponseWriter, _ *http.Request, query *dns.Msg) {
|
|
||||||
reply := new(dns.Msg)
|
|
||||||
|
|
||||||
for _, name := range names {
|
|
||||||
reply.Answer = append(reply.Answer, &dns.PTR{
|
|
||||||
Hdr: dns.RR_Header{
|
|
||||||
Name: query.Question[0].Name, Rrtype: dns.TypePTR,
|
|
||||||
Class: dns.ClassINET, Ttl: ttl,
|
|
||||||
},
|
|
||||||
Ptr: name,
|
|
||||||
})
|
|
||||||
}
|
|
||||||
|
|
||||||
writeReply(w, query, reply)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// rcodeReply answers with the response code and nothing else.
|
|
||||||
func rcodeReply(rcode int) func(http.ResponseWriter, *http.Request, *dns.Msg) {
|
|
||||||
return func(w http.ResponseWriter, _ *http.Request, query *dns.Msg) {
|
|
||||||
reply := new(dns.Msg)
|
|
||||||
reply.SetRcode(query, rcode)
|
|
||||||
|
|
||||||
wire, err := reply.Pack()
|
|
||||||
if err != nil {
|
|
||||||
http.Error(w, err.Error(), http.StatusInternalServerError)
|
|
||||||
|
|
||||||
return
|
|
||||||
}
|
|
||||||
|
|
||||||
_, _ = w.Write(wire)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// soaReply answers with the response code and a SOA record with these TTLs.
|
|
||||||
func soaReply(
|
|
||||||
rcode int, soaTTL, minimum uint32,
|
|
||||||
) func(http.ResponseWriter, *http.Request, *dns.Msg) {
|
|
||||||
return func(w http.ResponseWriter, _ *http.Request, query *dns.Msg) {
|
|
||||||
reply := new(dns.Msg)
|
|
||||||
reply.SetRcode(query, rcode)
|
|
||||||
reply.Ns = []dns.RR{&dns.SOA{
|
|
||||||
Hdr: dns.RR_Header{
|
|
||||||
Name: "arpa.", Rrtype: dns.TypeSOA, Class: dns.ClassINET, Ttl: soaTTL,
|
|
||||||
},
|
|
||||||
Ns: "ns.example.com.", Mbox: "hostmaster.example.com.", Minttl: minimum,
|
|
||||||
}}
|
|
||||||
|
|
||||||
wire, err := reply.Pack()
|
|
||||||
if err != nil {
|
|
||||||
http.Error(w, err.Error(), http.StatusInternalServerError)
|
|
||||||
|
|
||||||
return
|
|
||||||
}
|
|
||||||
|
|
||||||
_, _ = w.Write(wire)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// hangingReply never answers: it waits until the client gives up.
|
|
||||||
func hangingReply(_ http.ResponseWriter, r *http.Request, _ *dns.Msg) {
|
|
||||||
<-r.Context().Done()
|
|
||||||
}
|
|
||||||
|
|
||||||
// statusReply answers with an HTTP status.
|
|
||||||
func statusReply(status int) func(http.ResponseWriter, *http.Request, *dns.Msg) {
|
|
||||||
return func(w http.ResponseWriter, _ *http.Request, _ *dns.Msg) {
|
|
||||||
w.WriteHeader(status)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// openTestDB opens a cache.db in dir.
|
|
||||||
func openTestDB(t *testing.T, dir string) *sql.DB {
|
|
||||||
t.Helper()
|
|
||||||
|
|
||||||
db, err := cachedb.Open(t.Context(), dir)
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("opening the cache database: %v", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
t.Cleanup(func() { _ = db.Close() })
|
|
||||||
|
|
||||||
return db
|
|
||||||
}
|
|
||||||
|
|
||||||
// testClock is a clock a test moves by hand.
|
|
||||||
type testClock struct {
|
|
||||||
mu sync.Mutex
|
|
||||||
now time.Time
|
|
||||||
}
|
|
||||||
|
|
||||||
func (c *testClock) time() time.Time {
|
|
||||||
c.mu.Lock()
|
|
||||||
defer c.mu.Unlock()
|
|
||||||
|
|
||||||
return c.now
|
|
||||||
}
|
|
||||||
|
|
||||||
func (c *testClock) advance(d time.Duration) {
|
|
||||||
c.mu.Lock()
|
|
||||||
defer c.mu.Unlock()
|
|
||||||
|
|
||||||
c.now = c.now.Add(d)
|
|
||||||
}
|
|
||||||
|
|
||||||
// newTestCache returns a Cache that asks the resolvers in order, reads the
|
|
||||||
// time from a clock the test holds, has testTimeLimit, and is not started.
|
|
||||||
func newTestCache(
|
|
||||||
t *testing.T, db *sql.DB, resolvers ...*fakeResolver,
|
|
||||||
) (*Cache, *testClock) {
|
|
||||||
t.Helper()
|
|
||||||
|
|
||||||
urls := make([]string, 0, len(resolvers))
|
|
||||||
for _, resolver := range resolvers {
|
|
||||||
urls = append(urls, resolver.server.URL)
|
|
||||||
}
|
|
||||||
|
|
||||||
cache := New(Params{DB: db, Resolvers: urls, Logger: slog.Default()})
|
|
||||||
clock := &testClock{now: fixedTime()}
|
|
||||||
cache.now = clock.time
|
|
||||||
cache.timeLimit = testTimeLimit
|
|
||||||
|
|
||||||
return cache, clock
|
|
||||||
}
|
|
||||||
|
|
||||||
// handlerTransport hands each request to the fake resolver for its host, with
|
|
||||||
// no network between them. A test in a synctest bubble needs this: the bubble's
|
|
||||||
// clock stands still while one of its goroutines waits on a real connection. As
|
|
||||||
// over a network, a request whose time is up is not sent.
|
|
||||||
type handlerTransport map[string]*fakeResolver
|
|
||||||
|
|
||||||
func (h handlerTransport) RoundTrip(r *http.Request) (*http.Response, error) {
|
|
||||||
recorder := httptest.NewRecorder()
|
|
||||||
|
|
||||||
if r.Context().Err() == nil {
|
|
||||||
h[r.URL.Host].serve(recorder, r)
|
|
||||||
}
|
|
||||||
|
|
||||||
err := r.Context().Err()
|
|
||||||
if err != nil {
|
|
||||||
return nil, err
|
|
||||||
}
|
|
||||||
|
|
||||||
return recorder.Result(), nil
|
|
||||||
}
|
|
||||||
|
|
||||||
// newBubbleCache returns a Cache with the real time limit, for a test in a
|
|
||||||
// synctest bubble. It has one fake resolver for each reply, in order, reached
|
|
||||||
// through a handlerTransport.
|
|
||||||
func newBubbleCache(
|
|
||||||
replies ...func(http.ResponseWriter, *http.Request, *dns.Msg),
|
|
||||||
) (*Cache, []*fakeResolver) {
|
|
||||||
transport := make(handlerTransport, len(replies))
|
|
||||||
resolvers := make([]*fakeResolver, 0, len(replies))
|
|
||||||
urls := make([]string, 0, len(replies))
|
|
||||||
|
|
||||||
for i, reply := range replies {
|
|
||||||
host := "resolver" + strconv.Itoa(i) + ".example.com"
|
|
||||||
resolver := &fakeResolver{reply: reply}
|
|
||||||
|
|
||||||
transport[host] = resolver
|
|
||||||
resolvers = append(resolvers, resolver)
|
|
||||||
urls = append(urls, "https://"+host+"/dns-query")
|
|
||||||
}
|
|
||||||
|
|
||||||
cache := New(Params{Resolvers: urls, Logger: slog.Default()})
|
|
||||||
cache.client = &http.Client{Transport: transport}
|
|
||||||
|
|
||||||
return cache, resolvers
|
|
||||||
}
|
|
||||||
|
|
||||||
// lookupOK looks up address and fails the test on an error.
|
|
||||||
func lookupOK(t *testing.T, cache *Cache, address string) []string {
|
|
||||||
t.Helper()
|
|
||||||
|
|
||||||
names, err := cache.LookupAddr(t.Context(), address)
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("LookupAddr(%s): %v", address, err)
|
|
||||||
}
|
|
||||||
|
|
||||||
return names
|
|
||||||
}
|
|
||||||
|
|
||||||
func countRows(t *testing.T, db *sql.DB) int {
|
|
||||||
t.Helper()
|
|
||||||
|
|
||||||
var rows int
|
|
||||||
|
|
||||||
err := db.QueryRowContext(t.Context(),
|
|
||||||
"SELECT COUNT(*) FROM hostnames").Scan(&rows)
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("counting rows: %v", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
return rows
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestLookupAsksForTheReverseName(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
tests := []struct {
|
|
||||||
address string
|
|
||||||
want string
|
|
||||||
}{
|
|
||||||
{testAddress, testReverse},
|
|
||||||
{mappedAddress, testReverse},
|
|
||||||
{testV6Address, testV6Reverse},
|
|
||||||
}
|
|
||||||
|
|
||||||
for _, tt := range tests {
|
|
||||||
var asked string
|
|
||||||
|
|
||||||
resolver := newFakeResolver(t,
|
|
||||||
func(w http.ResponseWriter, r *http.Request, query *dns.Msg) {
|
|
||||||
asked = query.Question[0].Name
|
|
||||||
if query.Question[0].Qtype != dns.TypePTR {
|
|
||||||
t.Errorf("query type %d, want PTR", query.Question[0].Qtype)
|
|
||||||
}
|
|
||||||
|
|
||||||
ptrReply(300, testName)(w, r, query)
|
|
||||||
})
|
|
||||||
cache, _ := newTestCache(t, openTestDB(t, t.TempDir()), resolver)
|
|
||||||
|
|
||||||
names := lookupOK(t, cache, tt.address)
|
|
||||||
if !slices.Equal(names, []string{testName}) {
|
|
||||||
t.Errorf("%s: names %v, want %s", tt.address, names, testName)
|
|
||||||
}
|
|
||||||
|
|
||||||
if asked != tt.want {
|
|
||||||
t.Errorf("%s: asked for %s, want %s", tt.address, asked, tt.want)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestOnlyPublicAddressesAreSentToTheResolvers gives every resolver an answer
|
|
||||||
// to send, which a reserved address must not get.
|
|
||||||
func TestOnlyPublicAddressesAreSentToTheResolvers(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
tests := []struct {
|
|
||||||
name string
|
|
||||||
address string
|
|
||||||
want []string
|
|
||||||
requests int
|
|
||||||
}{
|
|
||||||
{"private IPv4", "10.0.0.1", []string{}, 0},
|
|
||||||
{"documentation IPv4", "192.0.2.1", []string{}, 0},
|
|
||||||
{"private IPv4 in IPv6 form", "::ffff:10.0.0.1", []string{}, 0},
|
|
||||||
{"unique-local IPv6", "fd00::1", []string{}, 0},
|
|
||||||
{"documentation IPv6", "2001:db8::1", []string{}, 0},
|
|
||||||
{"public IPv4", testAddress, []string{testName}, 1},
|
|
||||||
{"public IPv4 in IPv6 form", mappedAddress, []string{testName}, 1},
|
|
||||||
{"public IPv6", testV6Address, []string{testName}, 1},
|
|
||||||
}
|
|
||||||
|
|
||||||
for _, tt := range tests {
|
|
||||||
resolver := newFakeResolver(t, ptrReply(300, testName))
|
|
||||||
cache, _ := newTestCache(t, openTestDB(t, t.TempDir()), resolver)
|
|
||||||
|
|
||||||
names := lookupOK(t, cache, tt.address)
|
|
||||||
if names == nil || !slices.Equal(names, tt.want) {
|
|
||||||
t.Errorf("%s: names %#v, want %#v", tt.name, names, tt.want)
|
|
||||||
}
|
|
||||||
|
|
||||||
if got := resolver.requestCount(); got != tt.requests {
|
|
||||||
t.Errorf("%s: %d requests, want %d", tt.name, got, tt.requests)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestResolversAreTriedInOrderUntilOneAnswers has the first two resolvers fail,
|
|
||||||
// in different ways, and the third answer. The fourth is not asked.
|
|
||||||
func TestResolversAreTriedInOrderUntilOneAnswers(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
var (
|
|
||||||
mu sync.Mutex
|
|
||||||
order []int
|
|
||||||
)
|
|
||||||
|
|
||||||
asked := func(
|
|
||||||
index int, reply func(http.ResponseWriter, *http.Request, *dns.Msg),
|
|
||||||
) func(http.ResponseWriter, *http.Request, *dns.Msg) {
|
|
||||||
return func(w http.ResponseWriter, r *http.Request, query *dns.Msg) {
|
|
||||||
mu.Lock()
|
|
||||||
|
|
||||||
order = append(order, index)
|
|
||||||
|
|
||||||
mu.Unlock()
|
|
||||||
|
|
||||||
reply(w, r, query)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
resolvers := []*fakeResolver{
|
|
||||||
newFakeResolver(t, asked(0, statusReply(http.StatusServiceUnavailable))),
|
|
||||||
newFakeResolver(t, asked(1, rcodeReply(dns.RcodeServerFailure))),
|
|
||||||
newFakeResolver(t, asked(2, ptrReply(300, testName))),
|
|
||||||
newFakeResolver(t, asked(3, ptrReply(300, otherName))),
|
|
||||||
}
|
|
||||||
cache, _ := newTestCache(t, openTestDB(t, t.TempDir()), resolvers...)
|
|
||||||
|
|
||||||
names := lookupOK(t, cache, testAddress)
|
|
||||||
if !slices.Equal(names, []string{testName}) {
|
|
||||||
t.Errorf("names %v, want %s", names, testName)
|
|
||||||
}
|
|
||||||
|
|
||||||
if !slices.Equal(order, []int{0, 1, 2}) {
|
|
||||||
t.Errorf("resolvers asked %v, want [0 1 2]", order)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestResolverThatNeverAnswersLeavesTimeForTheNext would fail if the first
|
|
||||||
// resolver could take the whole time limit. The first gets half of it, the
|
|
||||||
// share of one resolver in two. The test runs in a synctest bubble, whose clock
|
|
||||||
// is fake, so a stall on the host uses none of the limit.
|
|
||||||
func TestResolverThatNeverAnswersLeavesTimeForTheNext(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
synctest.Test(t, func(t *testing.T) {
|
|
||||||
cache, _ := newBubbleCache(hangingReply, ptrReply(300, testName))
|
|
||||||
|
|
||||||
start := time.Now()
|
|
||||||
names := lookupOK(t, cache, testAddress)
|
|
||||||
elapsed := time.Since(start)
|
|
||||||
|
|
||||||
if !slices.Equal(names, []string{testName}) {
|
|
||||||
t.Errorf("names %v, want %s", names, testName)
|
|
||||||
}
|
|
||||||
|
|
||||||
if elapsed != time.Second {
|
|
||||||
t.Errorf("took %s, want 1s", elapsed)
|
|
||||||
}
|
|
||||||
})
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestLookupStopsAtTheTimeLimit has no resolver answer. The lookup fails when
|
|
||||||
// 2 seconds have passed, having asked every resolver. The test runs in a
|
|
||||||
// synctest bubble, whose clock is fake, so a stall on the host uses none of the
|
|
||||||
// limit.
|
|
||||||
func TestLookupStopsAtTheTimeLimit(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
synctest.Test(t, func(t *testing.T) {
|
|
||||||
cache, resolvers := newBubbleCache(
|
|
||||||
hangingReply, hangingReply, hangingReply, hangingReply)
|
|
||||||
|
|
||||||
start := time.Now()
|
|
||||||
_, err := cache.LookupAddr(t.Context(), testAddress)
|
|
||||||
elapsed := time.Since(start)
|
|
||||||
|
|
||||||
if !errors.Is(err, context.DeadlineExceeded) {
|
|
||||||
t.Errorf("error %v, want %v", err, context.DeadlineExceeded)
|
|
||||||
}
|
|
||||||
|
|
||||||
if elapsed != 2*time.Second {
|
|
||||||
t.Errorf("took %s, want 2s", elapsed)
|
|
||||||
}
|
|
||||||
|
|
||||||
for i, resolver := range resolvers {
|
|
||||||
if resolver.requestCount() != 1 {
|
|
||||||
t.Errorf("resolver %d asked %d times, want once",
|
|
||||||
i, resolver.requestCount())
|
|
||||||
}
|
|
||||||
}
|
|
||||||
})
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestAnswerIsServedFromMemoryUntilItExpires makes a second lookup within the
|
|
||||||
// TTL, which asks nobody, and a third after it, which asks again.
|
|
||||||
func TestAnswerIsServedFromMemoryUntilItExpires(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
resolver := newFakeResolver(t, ptrReply(300, testName))
|
|
||||||
cache, clock := newTestCache(t, openTestDB(t, t.TempDir()), resolver)
|
|
||||||
|
|
||||||
lookupOK(t, cache, testAddress)
|
|
||||||
|
|
||||||
names := lookupOK(t, cache, testAddress)
|
|
||||||
if !slices.Equal(names, []string{testName}) {
|
|
||||||
t.Errorf("cached names %v, want %s", names, testName)
|
|
||||||
}
|
|
||||||
|
|
||||||
if got := resolver.requestCount(); got != 1 {
|
|
||||||
t.Errorf("%d requests after the second lookup, want 1", got)
|
|
||||||
}
|
|
||||||
|
|
||||||
clock.advance(300 * time.Second)
|
|
||||||
lookupOK(t, cache, testAddress)
|
|
||||||
|
|
||||||
if got := resolver.requestCount(); got != 2 {
|
|
||||||
t.Errorf("%d requests after the TTL passed, want 2", got)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestAnAddressHasOneCacheEntryHoweverItIsWritten looks up an IPv4 address in
|
|
||||||
// its IPv4-mapped IPv6 form, then in its usual form.
|
|
||||||
func TestAnAddressHasOneCacheEntryHoweverItIsWritten(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
resolver := newFakeResolver(t, ptrReply(300, testName))
|
|
||||||
cache, _ := newTestCache(t, openTestDB(t, t.TempDir()), resolver)
|
|
||||||
|
|
||||||
lookupOK(t, cache, mappedAddress)
|
|
||||||
lookupOK(t, cache, testAddress)
|
|
||||||
|
|
||||||
if got := resolver.requestCount(); got != 1 {
|
|
||||||
t.Errorf("%d requests, want 1", got)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestCallerCannotChangeACachedAnswer changes the list each lookup returned, as
|
|
||||||
// the server does when it removes the trailing dots. The second lookup is the
|
|
||||||
// first answered from the cache. The third shows that the cached list did not
|
|
||||||
// change with the list the second returned.
|
|
||||||
func TestCallerCannotChangeACachedAnswer(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
resolver := newFakeResolver(t, ptrReply(300, testName))
|
|
||||||
cache, _ := newTestCache(t, openTestDB(t, t.TempDir()), resolver)
|
|
||||||
|
|
||||||
for range 3 {
|
|
||||||
names := lookupOK(t, cache, testAddress)
|
|
||||||
if !slices.Equal(names, []string{testName}) {
|
|
||||||
t.Errorf("names %v, want %s", names, testName)
|
|
||||||
}
|
|
||||||
|
|
||||||
names[0] = "changed"
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestTTLIsClamped(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
tests := []struct {
|
|
||||||
name string
|
|
||||||
ttl uint32
|
|
||||||
want time.Duration
|
|
||||||
}{
|
|
||||||
{"below the minimum", 1, time.Minute},
|
|
||||||
{"zero", 0, time.Minute},
|
|
||||||
{"at the minimum", 60, time.Minute},
|
|
||||||
{"in range", 300, 300 * time.Second},
|
|
||||||
{"at the maximum", 86400, 24 * time.Hour},
|
|
||||||
{"above the maximum", 604800, 24 * time.Hour},
|
|
||||||
}
|
|
||||||
|
|
||||||
for _, tt := range tests {
|
|
||||||
resolver := newFakeResolver(t, ptrReply(tt.ttl, testName))
|
|
||||||
cache, _ := newTestCache(t, openTestDB(t, t.TempDir()), resolver)
|
|
||||||
|
|
||||||
lookupOK(t, cache, testAddress)
|
|
||||||
|
|
||||||
got := cache.entries[testAddress].expires.Sub(fixedTime())
|
|
||||||
if got != tt.want {
|
|
||||||
t.Errorf("%s: kept for %s, want %s", tt.name, got, tt.want)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestSmallestTTLOfTheChainCounts answers with a CNAME record and two PTR
|
|
||||||
// records, whose TTLs all differ.
|
|
||||||
func TestSmallestTTLOfTheChainCounts(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
reply := func(w http.ResponseWriter, _ *http.Request, query *dns.Msg) {
|
|
||||||
owner := query.Question[0].Name
|
|
||||||
answer := new(dns.Msg)
|
|
||||||
answer.Answer = []dns.RR{
|
|
||||||
&dns.CNAME{
|
|
||||||
Hdr: dns.RR_Header{
|
|
||||||
Name: owner, Rrtype: dns.TypeCNAME, Class: dns.ClassINET, Ttl: 200,
|
|
||||||
},
|
|
||||||
Target: classlessOwner,
|
|
||||||
},
|
|
||||||
&dns.PTR{
|
|
||||||
Hdr: dns.RR_Header{
|
|
||||||
Name: classlessOwner, Rrtype: dns.TypePTR,
|
|
||||||
Class: dns.ClassINET, Ttl: 900,
|
|
||||||
},
|
|
||||||
Ptr: testName,
|
|
||||||
},
|
|
||||||
&dns.PTR{
|
|
||||||
Hdr: dns.RR_Header{
|
|
||||||
Name: classlessOwner, Rrtype: dns.TypePTR,
|
|
||||||
Class: dns.ClassINET, Ttl: 600,
|
|
||||||
},
|
|
||||||
Ptr: otherName,
|
|
||||||
},
|
|
||||||
}
|
|
||||||
|
|
||||||
writeReply(w, query, answer)
|
|
||||||
}
|
|
||||||
|
|
||||||
cache, _ := newTestCache(t, openTestDB(t, t.TempDir()), newFakeResolver(t, reply))
|
|
||||||
|
|
||||||
names := lookupOK(t, cache, testAddress)
|
|
||||||
if !slices.Equal(names, []string{testName, otherName}) {
|
|
||||||
t.Errorf("names %v, want both PTR names", names)
|
|
||||||
}
|
|
||||||
|
|
||||||
got := cache.entries[testAddress].expires.Sub(fixedTime())
|
|
||||||
if got != 200*time.Second {
|
|
||||||
t.Errorf("kept for %s, want 200s", got)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestNegativeTTL(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
tests := []struct {
|
|
||||||
name string
|
|
||||||
rcode int
|
|
||||||
soaTTL uint32
|
|
||||||
minimum uint32
|
|
||||||
want time.Duration
|
|
||||||
}{
|
|
||||||
{"name does not exist", dns.RcodeNameError, 3600, 300, 300 * time.Second},
|
|
||||||
{"no PTR record", dns.RcodeSuccess, 3600, 300, 300 * time.Second},
|
|
||||||
{"limited to an hour", dns.RcodeNameError, 86400, 7200, time.Hour},
|
|
||||||
{"SOA's own TTL is smaller", dns.RcodeNameError, 100, 300, 100 * time.Second},
|
|
||||||
}
|
|
||||||
|
|
||||||
for _, tt := range tests {
|
|
||||||
resolver := newFakeResolver(t, soaReply(tt.rcode, tt.soaTTL, tt.minimum))
|
|
||||||
cache, _ := newTestCache(t, openTestDB(t, t.TempDir()), resolver)
|
|
||||||
|
|
||||||
names := lookupOK(t, cache, testAddress)
|
|
||||||
if len(names) != 0 {
|
|
||||||
t.Errorf("%s: names %v, want none", tt.name, names)
|
|
||||||
}
|
|
||||||
|
|
||||||
got := cache.entries[testAddress].expires.Sub(fixedTime())
|
|
||||||
if got != tt.want {
|
|
||||||
t.Errorf("%s: kept for %s, want %s", tt.name, got, tt.want)
|
|
||||||
}
|
|
||||||
|
|
||||||
lookupOK(t, cache, testAddress)
|
|
||||||
|
|
||||||
if count := resolver.requestCount(); count != 1 {
|
|
||||||
t.Errorf("%s: %d requests, want 1", tt.name, count)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestAnswerWithoutNamesOrSOAIsNotCached has nothing to say how long the
|
|
||||||
// answer may be kept.
|
|
||||||
func TestAnswerWithoutNamesOrSOAIsNotCached(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
resolver := newFakeResolver(t, rcodeReply(dns.RcodeNameError))
|
|
||||||
cache, _ := newTestCache(t, openTestDB(t, t.TempDir()), resolver)
|
|
||||||
|
|
||||||
lookupOK(t, cache, testAddress)
|
|
||||||
lookupOK(t, cache, testAddress)
|
|
||||||
|
|
||||||
if got := resolver.requestCount(); got != 2 {
|
|
||||||
t.Errorf("%d requests, want 2", got)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestFailedQueryIsNotCached has the only resolver fail, and then recover.
|
|
||||||
func TestFailedQueryIsNotCached(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
resolver := newFakeResolver(t, rcodeReply(dns.RcodeServerFailure))
|
|
||||||
cache, _ := newTestCache(t, openTestDB(t, t.TempDir()), resolver)
|
|
||||||
|
|
||||||
_, err := cache.LookupAddr(t.Context(), testAddress)
|
|
||||||
if !errors.Is(err, errBadResponse) {
|
|
||||||
t.Fatalf("error %v, want %v", err, errBadResponse)
|
|
||||||
}
|
|
||||||
|
|
||||||
if len(cache.entries)+len(cache.pending) != 0 {
|
|
||||||
t.Error("a failed query left an entry in the cache")
|
|
||||||
}
|
|
||||||
|
|
||||||
resolver.setReply(ptrReply(300, testName))
|
|
||||||
|
|
||||||
names := lookupOK(t, cache, testAddress)
|
|
||||||
if !slices.Equal(names, []string{testName}) {
|
|
||||||
t.Errorf("names %v, want %s", names, testName)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestNamesThatAreNotHostNamesAreDropped answers with a name that holds markup
|
|
||||||
// beside a good one, as a PTR record can.
|
|
||||||
func TestNamesThatAreNotHostNamesAreDropped(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
resolver := newFakeResolver(t,
|
|
||||||
ptrReply(300, "bad<script>.example.com.", testName, "."))
|
|
||||||
cache, _ := newTestCache(t, openTestDB(t, t.TempDir()), resolver)
|
|
||||||
|
|
||||||
names := lookupOK(t, cache, testAddress)
|
|
||||||
if !slices.Equal(names, []string{testName}) {
|
|
||||||
t.Errorf("names %v, want only %s", names, testName)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestCacheSurvivesARestart stops a cache that holds a positive and a negative
|
|
||||||
// answer, and starts a new one on the reopened file. It makes no request.
|
|
||||||
func TestCacheSurvivesARestart(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
dir := t.TempDir()
|
|
||||||
db := openTestDB(t, dir)
|
|
||||||
|
|
||||||
resolver := newFakeResolver(t,
|
|
||||||
func(w http.ResponseWriter, r *http.Request, query *dns.Msg) {
|
|
||||||
if query.Question[0].Name == testReverse {
|
|
||||||
ptrReply(300, testName, otherName)(w, r, query)
|
|
||||||
|
|
||||||
return
|
|
||||||
}
|
|
||||||
|
|
||||||
soaReply(dns.RcodeNameError, 3600, 300)(w, r, query)
|
|
||||||
})
|
|
||||||
|
|
||||||
cache, _ := newTestCache(t, db, resolver)
|
|
||||||
|
|
||||||
err := cache.Start(t.Context())
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("Start: %v", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
lookupOK(t, cache, testAddress)
|
|
||||||
lookupOK(t, cache, otherAddress)
|
|
||||||
cache.Stop(t.Context())
|
|
||||||
|
|
||||||
err = db.Close()
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("closing the database: %v", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
restarted, _ := newTestCache(t, openTestDB(t, dir), resolver)
|
|
||||||
|
|
||||||
err = restarted.Start(t.Context())
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("Start after the restart: %v", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
defer restarted.Stop(t.Context())
|
|
||||||
|
|
||||||
requestsBefore := resolver.requestCount()
|
|
||||||
|
|
||||||
names := lookupOK(t, restarted, testAddress)
|
|
||||||
if !slices.Equal(names, []string{testName, otherName}) {
|
|
||||||
t.Errorf("names after restart %v", names)
|
|
||||||
}
|
|
||||||
|
|
||||||
if names := lookupOK(t, restarted, otherAddress); len(names) != 0 {
|
|
||||||
t.Errorf("names of the address with none after restart: %v", names)
|
|
||||||
}
|
|
||||||
|
|
||||||
if got := resolver.requestCount(); got != requestsBefore {
|
|
||||||
t.Errorf("%d requests after the restart, want none",
|
|
||||||
got-requestsBefore)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestSweepDeletesExpiredRows also checks that a restart does not load a row
|
|
||||||
// that has expired since it was written.
|
|
||||||
func TestSweepDeletesExpiredRows(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
db := openTestDB(t, t.TempDir())
|
|
||||||
resolver := newFakeResolver(t,
|
|
||||||
func(w http.ResponseWriter, r *http.Request, query *dns.Msg) {
|
|
||||||
ttl := uint32(300)
|
|
||||||
if query.Question[0].Name == testReverse {
|
|
||||||
ttl = 60
|
|
||||||
}
|
|
||||||
|
|
||||||
ptrReply(ttl, testName)(w, r, query)
|
|
||||||
})
|
|
||||||
cache, clock := newTestCache(t, db, resolver)
|
|
||||||
|
|
||||||
lookupOK(t, cache, testAddress)
|
|
||||||
lookupOK(t, cache, otherAddress)
|
|
||||||
cache.flush(t.Context())
|
|
||||||
|
|
||||||
if got := countRows(t, db); got != 2 {
|
|
||||||
t.Fatalf("%d rows before the sweep, want 2", got)
|
|
||||||
}
|
|
||||||
|
|
||||||
clock.advance(90 * time.Second)
|
|
||||||
|
|
||||||
loaded, loadedClock := newTestCache(t, db, resolver)
|
|
||||||
loadedClock.advance(90 * time.Second)
|
|
||||||
|
|
||||||
err := loaded.load(t.Context())
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("load: %v", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
if len(loaded.entries) != 1 {
|
|
||||||
t.Errorf("loaded %d entries, want the 1 not expired", len(loaded.entries))
|
|
||||||
}
|
|
||||||
|
|
||||||
cache.sweep(t.Context())
|
|
||||||
|
|
||||||
if got := countRows(t, db); got != 1 {
|
|
||||||
t.Errorf("%d rows after the sweep, want 1", got)
|
|
||||||
}
|
|
||||||
|
|
||||||
if len(cache.entries) != 1 {
|
|
||||||
t.Errorf("%d entries in memory after the sweep, want 1", len(cache.entries))
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestFlushKeepsAnswersItCouldNotWrite makes one write fail by hiding the
|
|
||||||
// table. The next write, once the table is back, stores the answer.
|
|
||||||
func TestFlushKeepsAnswersItCouldNotWrite(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
db := openTestDB(t, t.TempDir())
|
|
||||||
resolver := newFakeResolver(t, ptrReply(300, testName))
|
|
||||||
cache, _ := newTestCache(t, db, resolver)
|
|
||||||
|
|
||||||
lookupOK(t, cache, testAddress)
|
|
||||||
|
|
||||||
_, err := db.ExecContext(t.Context(), "ALTER TABLE hostnames RENAME TO hidden")
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("renaming the table: %v", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
cache.flush(t.Context())
|
|
||||||
|
|
||||||
_, err = db.ExecContext(t.Context(), "ALTER TABLE hidden RENAME TO hostnames")
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("renaming the table back: %v", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
if got := countRows(t, db); got != 0 {
|
|
||||||
t.Fatalf("%d rows after the failed write, want 0", got)
|
|
||||||
}
|
|
||||||
|
|
||||||
cache.flush(t.Context())
|
|
||||||
|
|
||||||
if got := countRows(t, db); got != 1 {
|
|
||||||
t.Errorf("%d rows after the second write, want 1", got)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestNewAddressIsNotKeptAtTheLimit fills a cache that holds one address. A
|
|
||||||
// lookup of a second address still returns its answer, and the cache keeps
|
|
||||||
// nothing for it until the sweep has deleted the first address's expired
|
|
||||||
// answer.
|
|
||||||
func TestNewAddressIsNotKeptAtTheLimit(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
db := openTestDB(t, t.TempDir())
|
|
||||||
resolver := newFakeResolver(t, ptrReply(300, testName))
|
|
||||||
cache, clock := newTestCache(t, db, resolver)
|
|
||||||
cache.maxEntries = 1
|
|
||||||
|
|
||||||
lookupOK(t, cache, testAddress)
|
|
||||||
|
|
||||||
names := lookupOK(t, cache, otherAddress)
|
|
||||||
if !slices.Equal(names, []string{testName}) {
|
|
||||||
t.Errorf("names %v at the limit, want %s", names, testName)
|
|
||||||
}
|
|
||||||
|
|
||||||
_, inEntries := cache.entries[otherAddress]
|
|
||||||
_, inPending := cache.pending[otherAddress]
|
|
||||||
|
|
||||||
if inEntries || inPending {
|
|
||||||
t.Errorf("a new address was kept at the limit: in entries %t, in pending %t",
|
|
||||||
inEntries, inPending)
|
|
||||||
}
|
|
||||||
|
|
||||||
cache.flush(t.Context())
|
|
||||||
|
|
||||||
if got := countRows(t, db); got != 1 {
|
|
||||||
t.Errorf("%d rows at the limit, want 1", got)
|
|
||||||
}
|
|
||||||
|
|
||||||
clock.advance(300 * time.Second)
|
|
||||||
cache.sweep(t.Context())
|
|
||||||
lookupOK(t, cache, otherAddress)
|
|
||||||
|
|
||||||
_, inEntries = cache.entries[otherAddress]
|
|
||||||
if !inEntries {
|
|
||||||
t.Error("a new address was not kept after the sweep made room")
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestHeldAddressIsReplacedAtTheLimit lets the answer of the only address a
|
|
||||||
// cache holds expire. The new answer replaces it in memory and is queued for
|
|
||||||
// cache.db.
|
|
||||||
func TestHeldAddressIsReplacedAtTheLimit(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
resolver := newFakeResolver(t, ptrReply(300, testName))
|
|
||||||
cache, clock := newTestCache(t, openTestDB(t, t.TempDir()), resolver)
|
|
||||||
cache.maxEntries = 1
|
|
||||||
|
|
||||||
lookupOK(t, cache, testAddress)
|
|
||||||
clock.advance(300 * time.Second)
|
|
||||||
resolver.setReply(ptrReply(300, otherName))
|
|
||||||
lookupOK(t, cache, testAddress)
|
|
||||||
|
|
||||||
want := []string{otherName}
|
|
||||||
|
|
||||||
if got := cache.entries[testAddress].names; !slices.Equal(got, want) {
|
|
||||||
t.Errorf("entries holds %v, want %v", got, want)
|
|
||||||
}
|
|
||||||
|
|
||||||
if got := cache.pending[testAddress].names; !slices.Equal(got, want) {
|
|
||||||
t.Errorf("pending holds %v, want %v", got, want)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
@@ -1,200 +0,0 @@
|
|||||||
package ptr
|
|
||||||
|
|
||||||
import (
|
|
||||||
"bytes"
|
|
||||||
"context"
|
|
||||||
"errors"
|
|
||||||
"fmt"
|
|
||||||
"io"
|
|
||||||
"math"
|
|
||||||
"net/http"
|
|
||||||
"strings"
|
|
||||||
"time"
|
|
||||||
|
|
||||||
"github.com/miekg/dns"
|
|
||||||
)
|
|
||||||
|
|
||||||
const (
|
|
||||||
// LookupTimeout limits one address's lookup, every resolver included.
|
|
||||||
LookupTimeout = 2 * time.Second
|
|
||||||
|
|
||||||
// minTTL and maxTTL limit how long PTR names are kept.
|
|
||||||
minTTL = time.Minute
|
|
||||||
maxTTL = 24 * time.Hour
|
|
||||||
|
|
||||||
// maxNegativeTTL limits how long an address with no PTR names is
|
|
||||||
// remembered as having none.
|
|
||||||
maxNegativeTTL = time.Hour
|
|
||||||
|
|
||||||
// dnsMessageType is the content type of RFC 8484 requests and answers.
|
|
||||||
dnsMessageType = "application/dns-message"
|
|
||||||
|
|
||||||
// maxMessageBytes is the largest possible DNS message.
|
|
||||||
maxMessageBytes = 65535
|
|
||||||
)
|
|
||||||
|
|
||||||
var (
|
|
||||||
errBadStatus = errors.New("unexpected HTTP status")
|
|
||||||
errBadResponse = errors.New("resolver reported an error")
|
|
||||||
)
|
|
||||||
|
|
||||||
// Resolvers returns the URLs of the public DNS-over-HTTPS resolvers, in the
|
|
||||||
// order they are tried.
|
|
||||||
func Resolvers() []string {
|
|
||||||
return []string{
|
|
||||||
"https://dns.google/dns-query",
|
|
||||||
"https://cloudflare-dns.com/dns-query",
|
|
||||||
"https://unfiltered.adguard-dns.com/dns-query",
|
|
||||||
"https://dns.quad9.net/dns-query",
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// answer is a resolver's answer for one address.
|
|
||||||
type answer struct {
|
|
||||||
// names are the address's PTR names, nil when it has none.
|
|
||||||
names []string
|
|
||||||
|
|
||||||
// ttl is how long the answer may be kept. Zero means it is not kept.
|
|
||||||
ttl time.Duration
|
|
||||||
}
|
|
||||||
|
|
||||||
// lookup asks the resolvers in turn for the PTR names of the reverse name
|
|
||||||
// until one answers. It fails when none has answered within the time limit.
|
|
||||||
func (c *Cache) lookup(ctx context.Context, reverseName string) (answer, error) {
|
|
||||||
ctx, cancel := context.WithTimeout(ctx, c.timeLimit)
|
|
||||||
defer cancel()
|
|
||||||
|
|
||||||
query := new(dns.Msg).SetQuestion(reverseName, dns.TypePTR)
|
|
||||||
query.Id = 0 // RFC 8484 asks for 0, so that equal queries can be cached.
|
|
||||||
|
|
||||||
wire, err := query.Pack()
|
|
||||||
if err != nil {
|
|
||||||
return answer{}, fmt.Errorf("packing the query: %w", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
var failures []error
|
|
||||||
|
|
||||||
for i, resolver := range c.resolvers {
|
|
||||||
// Each resolver gets an equal share of the time left, so that one
|
|
||||||
// that never answers leaves time for the next.
|
|
||||||
deadline, _ := ctx.Deadline()
|
|
||||||
share := time.Until(deadline) / time.Duration(len(c.resolvers)-i)
|
|
||||||
|
|
||||||
result, err := c.ask(ctx, resolver, wire, share)
|
|
||||||
if err == nil {
|
|
||||||
return result, nil
|
|
||||||
}
|
|
||||||
|
|
||||||
failures = append(failures, fmt.Errorf("%s: %w", resolver, err))
|
|
||||||
}
|
|
||||||
|
|
||||||
return answer{}, errors.Join(failures...)
|
|
||||||
}
|
|
||||||
|
|
||||||
// ask sends the query to one resolver and waits at most timeout for its answer.
|
|
||||||
func (c *Cache) ask(
|
|
||||||
ctx context.Context, resolver string, query []byte, timeout time.Duration,
|
|
||||||
) (answer, error) {
|
|
||||||
ctx, cancel := context.WithTimeout(ctx, timeout)
|
|
||||||
defer cancel()
|
|
||||||
|
|
||||||
request, err := http.NewRequestWithContext(
|
|
||||||
ctx, http.MethodPost, resolver, bytes.NewReader(query))
|
|
||||||
if err != nil {
|
|
||||||
return answer{}, err
|
|
||||||
}
|
|
||||||
|
|
||||||
request.Header.Set("Content-Type", dnsMessageType)
|
|
||||||
request.Header.Set("Accept", dnsMessageType)
|
|
||||||
|
|
||||||
response, err := c.client.Do(request)
|
|
||||||
if err != nil {
|
|
||||||
return answer{}, err
|
|
||||||
}
|
|
||||||
|
|
||||||
defer func() { _ = response.Body.Close() }()
|
|
||||||
|
|
||||||
if response.StatusCode != http.StatusOK {
|
|
||||||
return answer{}, fmt.Errorf("%w: %d", errBadStatus, response.StatusCode)
|
|
||||||
}
|
|
||||||
|
|
||||||
body, err := io.ReadAll(io.LimitReader(response.Body, maxMessageBytes))
|
|
||||||
if err != nil {
|
|
||||||
return answer{}, err
|
|
||||||
}
|
|
||||||
|
|
||||||
var reply dns.Msg
|
|
||||||
|
|
||||||
err = reply.Unpack(body)
|
|
||||||
if err != nil {
|
|
||||||
return answer{}, err
|
|
||||||
}
|
|
||||||
|
|
||||||
return answerFrom(&reply)
|
|
||||||
}
|
|
||||||
|
|
||||||
// answerFrom reads a resolver's reply. A name that does not exist and a name
|
|
||||||
// with no PTR records are answers. Every other response code is a failure.
|
|
||||||
func answerFrom(reply *dns.Msg) (answer, error) {
|
|
||||||
if reply.Rcode != dns.RcodeSuccess && reply.Rcode != dns.RcodeNameError {
|
|
||||||
return answer{}, fmt.Errorf("%w: %s",
|
|
||||||
errBadResponse, dns.RcodeToString[reply.Rcode])
|
|
||||||
}
|
|
||||||
|
|
||||||
var names []string
|
|
||||||
|
|
||||||
seconds := uint32(math.MaxUint32)
|
|
||||||
|
|
||||||
for _, record := range reply.Answer {
|
|
||||||
// The chain to a PTR record can hold CNAME records, and the answer
|
|
||||||
// is good only as long as every record in it.
|
|
||||||
seconds = min(seconds, record.Header().Ttl)
|
|
||||||
|
|
||||||
ptr, ok := record.(*dns.PTR)
|
|
||||||
if ok && isHostName(ptr.Ptr) {
|
|
||||||
names = append(names, ptr.Ptr)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
if len(names) == 0 {
|
|
||||||
return answer{ttl: negativeTTL(reply.Ns)}, nil
|
|
||||||
}
|
|
||||||
|
|
||||||
ttl := time.Duration(seconds) * time.Second
|
|
||||||
|
|
||||||
return answer{names: names, ttl: min(max(ttl, minTTL), maxTTL)}, nil
|
|
||||||
}
|
|
||||||
|
|
||||||
// negativeTTL returns how long a reply with no PTR names may be kept: the
|
|
||||||
// smaller of the SOA record's own TTL and its minimum field (RFC 2308), at most
|
|
||||||
// maxNegativeTTL. Without a SOA record it is zero.
|
|
||||||
func negativeTTL(authority []dns.RR) time.Duration {
|
|
||||||
for _, record := range authority {
|
|
||||||
soa, ok := record.(*dns.SOA)
|
|
||||||
if ok {
|
|
||||||
ttl := time.Duration(min(soa.Hdr.Ttl, soa.Minttl)) * time.Second
|
|
||||||
|
|
||||||
return min(ttl, maxNegativeTTL)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
return 0
|
|
||||||
}
|
|
||||||
|
|
||||||
// isHostName reports whether name is a host name. A PTR record can hold any
|
|
||||||
// bytes, and the names go into web pages, so anything else is dropped, as
|
|
||||||
// net.Resolver.LookupAddr drops it. The root name is not a host name.
|
|
||||||
func isHostName(name string) bool {
|
|
||||||
_, ok := dns.IsDomainName(name)
|
|
||||||
|
|
||||||
return ok && name != "." && !strings.ContainsFunc(name, notInHostName)
|
|
||||||
}
|
|
||||||
|
|
||||||
// notInHostName reports whether r cannot be part of a host name.
|
|
||||||
func notInHostName(r rune) bool {
|
|
||||||
return !strings.ContainsRune(hostNameCharacters, r)
|
|
||||||
}
|
|
||||||
|
|
||||||
// hostNameCharacters are the characters of a host name, in presentation format.
|
|
||||||
const hostNameCharacters = "abcdefghijklmnopqrstuvwxyz" +
|
|
||||||
"ABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789-_."
|
|
||||||
@@ -1,103 +0,0 @@
|
|||||||
package routetable
|
|
||||||
|
|
||||||
import (
|
|
||||||
"net/netip"
|
|
||||||
"slices"
|
|
||||||
"time"
|
|
||||||
)
|
|
||||||
|
|
||||||
// MaxWaitingOriginChanges is the most origin changes the table keeps waiting
|
|
||||||
// for the prefix handler to write. The oldest change past it is dropped and
|
|
||||||
// counted.
|
|
||||||
const MaxWaitingOriginChanges = 100000
|
|
||||||
|
|
||||||
// OriginChange is one change of a prefix and origin AS pair: the origin AS
|
|
||||||
// gained its first peer, or lost its last.
|
|
||||||
type OriginChange struct {
|
|
||||||
Prefix netip.Prefix
|
|
||||||
OriginASN int
|
|
||||||
// Announced says the origin AS gained its first peer. When false, it
|
|
||||||
// lost its last peer.
|
|
||||||
Announced bool
|
|
||||||
// PeerCount is how many peers carry the prefix just after the change,
|
|
||||||
// counting every origin AS.
|
|
||||||
PeerCount int
|
|
||||||
// Time is when RouteWatch applied the change.
|
|
||||||
Time time.Time
|
|
||||||
}
|
|
||||||
|
|
||||||
// recordOriginChange records that the pair of prefix and asn gained its first
|
|
||||||
// peer, when announced, or lost its last, when not. The caller holds the lock.
|
|
||||||
// peerCount is how many peers carry the prefix just after the change.
|
|
||||||
func (t *Table) recordOriginChange(
|
|
||||||
prefix netip.Prefix, asn int, announced bool, peerCount int,
|
|
||||||
) {
|
|
||||||
change := OriginChange{
|
|
||||||
Prefix: prefix,
|
|
||||||
OriginASN: asn,
|
|
||||||
Announced: announced,
|
|
||||||
PeerCount: peerCount,
|
|
||||||
Time: t.now(),
|
|
||||||
}
|
|
||||||
|
|
||||||
t.originChanges = append(t.originChanges, change)
|
|
||||||
t.trimOriginChanges()
|
|
||||||
}
|
|
||||||
|
|
||||||
// TakeOriginChanges returns at most limit of the origin changes that wait to
|
|
||||||
// be written, oldest first, and how many were dropped since the last call. It
|
|
||||||
// then forgets what it returned.
|
|
||||||
func (t *Table) TakeOriginChanges(limit int) ([]OriginChange, int) {
|
|
||||||
t.mu.Lock()
|
|
||||||
defer t.mu.Unlock()
|
|
||||||
|
|
||||||
n := min(limit, len(t.originChanges))
|
|
||||||
changes := slices.Clone(t.originChanges[:n])
|
|
||||||
t.originChanges = t.originChanges[n:]
|
|
||||||
|
|
||||||
dropped := t.originChangesDropped
|
|
||||||
t.originChangesDropped = 0
|
|
||||||
|
|
||||||
return changes, dropped
|
|
||||||
}
|
|
||||||
|
|
||||||
// PutBackOriginChanges records again what one TakeOriginChanges call returned,
|
|
||||||
// for a caller that could not write it. The next TakeOriginChanges returns
|
|
||||||
// those changes first, before the ones that arrived since. The oldest changes
|
|
||||||
// past the cap are dropped and counted.
|
|
||||||
func (t *Table) PutBackOriginChanges(changes []OriginChange) {
|
|
||||||
t.mu.Lock()
|
|
||||||
defer t.mu.Unlock()
|
|
||||||
|
|
||||||
t.originChanges = append(changes, t.originChanges...)
|
|
||||||
t.trimOriginChanges()
|
|
||||||
}
|
|
||||||
|
|
||||||
// trimOriginChanges drops the oldest origin changes past the cap and counts
|
|
||||||
// them. The caller holds the lock.
|
|
||||||
func (t *Table) trimOriginChanges() {
|
|
||||||
if extra := len(t.originChanges) - MaxWaitingOriginChanges; extra > 0 {
|
|
||||||
t.originChangesDropped += extra
|
|
||||||
t.originChanges = t.originChanges[extra:]
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// OriginChangeCount returns how many origin changes wait for TakeOriginChanges.
|
|
||||||
func (t *Table) OriginChangeCount() int {
|
|
||||||
t.mu.RLock()
|
|
||||||
defer t.mu.RUnlock()
|
|
||||||
|
|
||||||
return len(t.originChanges)
|
|
||||||
}
|
|
||||||
|
|
||||||
// prefixPeerCount returns how many peers carry prefix, counting every origin
|
|
||||||
// AS. The caller holds the lock.
|
|
||||||
func (t *Table) prefixPeerCount(prefix netip.Prefix) int {
|
|
||||||
count := 0
|
|
||||||
|
|
||||||
for _, o := range t.origins(prefix) {
|
|
||||||
count += int(o.peers.Count())
|
|
||||||
}
|
|
||||||
|
|
||||||
return count
|
|
||||||
}
|
|
||||||
@@ -1,302 +0,0 @@
|
|||||||
package routetable_test
|
|
||||||
|
|
||||||
import (
|
|
||||||
"math"
|
|
||||||
"net/netip"
|
|
||||||
"reflect"
|
|
||||||
"testing"
|
|
||||||
"time"
|
|
||||||
|
|
||||||
"github.com/bits-and-blooms/bitset"
|
|
||||||
|
|
||||||
"git.eeqj.de/sneak/routewatch/internal/routetable"
|
|
||||||
)
|
|
||||||
|
|
||||||
// originChange returns the origin change the table records for prefix and asn
|
|
||||||
// at the clock's time, announced or withdrawn, with peerCount peers.
|
|
||||||
func originChange(
|
|
||||||
prefix netip.Prefix, asn int, announced bool, peerCount int, at time.Time,
|
|
||||||
) routetable.OriginChange {
|
|
||||||
return routetable.OriginChange{
|
|
||||||
Prefix: prefix,
|
|
||||||
OriginASN: asn,
|
|
||||||
Announced: announced,
|
|
||||||
PeerCount: peerCount,
|
|
||||||
Time: at,
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestOriginChangesRecordAnnouncedAndWithdrawn checks that a prefix and origin
|
|
||||||
// AS pair records an announced row when it gains its first peer, nothing for a
|
|
||||||
// second peer, and a withdrawn row when it loses its last peer.
|
|
||||||
func TestOriginChangesRecordAnnouncedAndWithdrawn(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
now := at(0)
|
|
||||||
table := routetable.New(routetable.Params{
|
|
||||||
Now: func() time.Time { return now },
|
|
||||||
})
|
|
||||||
prefix := netip.MustParsePrefix(prefixV4)
|
|
||||||
|
|
||||||
table.Announce(prefix, asA, routeOf(peerA, asA, 10))
|
|
||||||
table.Announce(prefix, asA, routeOf(peerB, asA, 20))
|
|
||||||
|
|
||||||
checkOriginChanges(t, table, []routetable.OriginChange{
|
|
||||||
originChange(prefix, asA, true, 1, now),
|
|
||||||
})
|
|
||||||
|
|
||||||
table.Withdraw(prefix, peerA)
|
|
||||||
table.Withdraw(prefix, peerB)
|
|
||||||
|
|
||||||
checkOriginChanges(t, table, []routetable.OriginChange{
|
|
||||||
originChange(prefix, asA, false, 0, now),
|
|
||||||
})
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestOriginChangesRecordAPeerMovingOrigin checks that a peer moving from one
|
|
||||||
// origin AS to another records a withdrawn row for the old one and an
|
|
||||||
// announced row for the new one.
|
|
||||||
func TestOriginChangesRecordAPeerMovingOrigin(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
now := at(0)
|
|
||||||
table := routetable.New(routetable.Params{
|
|
||||||
Now: func() time.Time { return now },
|
|
||||||
})
|
|
||||||
prefix := netip.MustParsePrefix(prefixV4)
|
|
||||||
|
|
||||||
table.Announce(prefix, asA, routeOf(peerA, asA, 10))
|
|
||||||
table.Announce(prefix, asB, routeOf(peerA, asB, 20))
|
|
||||||
|
|
||||||
checkOriginChanges(t, table, []routetable.OriginChange{
|
|
||||||
originChange(prefix, asA, true, 1, now),
|
|
||||||
originChange(prefix, asA, false, 0, now),
|
|
||||||
originChange(prefix, asB, true, 1, now),
|
|
||||||
})
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestOriginChangesRecordPeersDown checks that PeersDown records a withdrawn
|
|
||||||
// row for each prefix and origin AS pair it empties.
|
|
||||||
func TestOriginChangesRecordPeersDown(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
now := at(0)
|
|
||||||
table := routetable.New(routetable.Params{
|
|
||||||
Now: func() time.Time { return now },
|
|
||||||
})
|
|
||||||
v4 := netip.MustParsePrefix(prefixV4)
|
|
||||||
v6 := netip.MustParsePrefix(prefixV6)
|
|
||||||
|
|
||||||
table.Announce(v4, asA, routeOf(peerA, asA, 10))
|
|
||||||
table.Announce(v6, asA, routeOf(peerA, asA, 10))
|
|
||||||
table.Announce(v4, asB, routeOf(peerB, asB, 20))
|
|
||||||
|
|
||||||
table.PeersDown([]string{peerA})
|
|
||||||
|
|
||||||
// peerA carried v4 and v6 with asA; both pairs are emptied. peerB still
|
|
||||||
// carries v4 with asB, so that pair is not.
|
|
||||||
checkOriginChanges(t, table, []routetable.OriginChange{
|
|
||||||
originChange(v4, asA, true, 1, now),
|
|
||||||
originChange(v6, asA, true, 1, now),
|
|
||||||
originChange(v4, asB, true, 2, now),
|
|
||||||
originChange(v4, asA, false, 1, now),
|
|
||||||
originChange(v6, asA, false, 0, now),
|
|
||||||
})
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestOriginChangesLoadRecordsNothing checks that Load records no origin
|
|
||||||
// change.
|
|
||||||
func TestOriginChangesLoadRecordsNothing(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
table := routetable.New(routetable.Params{
|
|
||||||
PeerNumbers: map[string]uint{peerA: 0},
|
|
||||||
})
|
|
||||||
prefix := netip.MustParsePrefix(prefixV4)
|
|
||||||
|
|
||||||
peers := new(bitset.BitSet)
|
|
||||||
peers.Set(0)
|
|
||||||
|
|
||||||
table.Load(routetable.Change{
|
|
||||||
Prefix: prefix,
|
|
||||||
OriginASN: asA,
|
|
||||||
Peers: peers,
|
|
||||||
Routes: []routetable.Route{routeOf(peerA, asA, 10)},
|
|
||||||
})
|
|
||||||
|
|
||||||
changes, dropped := table.TakeOriginChanges(math.MaxInt)
|
|
||||||
if dropped != 0 || len(changes) != 0 {
|
|
||||||
t.Errorf("TakeOriginChanges = %+v, %d; want none", changes, dropped)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestOriginChangesTakeReturnsOldestFirst checks that TakeOriginChanges
|
|
||||||
// returns the changes oldest first and at most max.
|
|
||||||
func TestOriginChangesTakeReturnsOldestFirst(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
table := routetable.New(routetable.Params{})
|
|
||||||
prefix := netip.MustParsePrefix(prefixV4)
|
|
||||||
|
|
||||||
table.Announce(prefix, asA, routeOf(peerA, asA, 10))
|
|
||||||
table.Announce(prefix, asB, routeOf(peerB, asB, 20))
|
|
||||||
table.Announce(prefix, asC, routeOf(peerC, asC, 30))
|
|
||||||
|
|
||||||
first, dropped := table.TakeOriginChanges(2)
|
|
||||||
if dropped != 0 {
|
|
||||||
t.Errorf("dropped = %d, want 0", dropped)
|
|
||||||
}
|
|
||||||
|
|
||||||
if len(first) != 2 || first[0].OriginASN != asA || first[1].OriginASN != asB {
|
|
||||||
t.Errorf("first TakeOriginChanges = %+v, want the first two", first)
|
|
||||||
}
|
|
||||||
|
|
||||||
second, _ := table.TakeOriginChanges(math.MaxInt)
|
|
||||||
if len(second) != 1 || second[0].OriginASN != asC {
|
|
||||||
t.Errorf("second TakeOriginChanges = %+v, want the third", second)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestOriginChangesPutBackPutsRowsInFront checks that PutBackOriginChanges
|
|
||||||
// puts rows back in front, so the next TakeOriginChanges returns them first.
|
|
||||||
func TestOriginChangesPutBackPutsRowsInFront(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
table := routetable.New(routetable.Params{})
|
|
||||||
prefix := netip.MustParsePrefix(prefixV4)
|
|
||||||
|
|
||||||
table.Announce(prefix, asA, routeOf(peerA, asA, 10))
|
|
||||||
table.Announce(prefix, asB, routeOf(peerB, asB, 20))
|
|
||||||
|
|
||||||
taken, _ := table.TakeOriginChanges(math.MaxInt)
|
|
||||||
table.Announce(prefix, asC, routeOf(peerC, asC, 30))
|
|
||||||
table.PutBackOriginChanges(taken)
|
|
||||||
|
|
||||||
changes, _ := table.TakeOriginChanges(math.MaxInt)
|
|
||||||
if len(changes) != 3 || changes[0].OriginASN != asA ||
|
|
||||||
changes[1].OriginASN != asB || changes[2].OriginASN != asC {
|
|
||||||
t.Errorf("TakeOriginChanges = %+v, want the put-back rows first", changes)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestOriginChangesDropPastTheLimit checks that a change past 100,000 waiting
|
|
||||||
// rows is dropped and counted.
|
|
||||||
func TestOriginChangesDropPastTheLimit(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
table := routetable.New(routetable.Params{})
|
|
||||||
addr := netip.MustParseAddr("10.0.0.0")
|
|
||||||
|
|
||||||
for range routetable.MaxWaitingOriginChanges + 1 {
|
|
||||||
prefix := netip.PrefixFrom(addr, 32)
|
|
||||||
table.Announce(prefix, asA, routeOf(peerA, asA, 10))
|
|
||||||
|
|
||||||
addr = addr.Next()
|
|
||||||
}
|
|
||||||
|
|
||||||
changes, dropped := table.TakeOriginChanges(routetable.MaxWaitingOriginChanges)
|
|
||||||
if dropped != 1 {
|
|
||||||
t.Errorf("dropped = %d, want 1", dropped)
|
|
||||||
}
|
|
||||||
|
|
||||||
if len(changes) != routetable.MaxWaitingOriginChanges {
|
|
||||||
t.Errorf("len(changes) = %d, want %d", len(changes),
|
|
||||||
routetable.MaxWaitingOriginChanges)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestOriginChangesTakeResetsTheDroppedCount checks that a TakeOriginChanges
|
|
||||||
// that reports dropped rows leaves the next one reporting none, so each drop
|
|
||||||
// is logged once, not on every later write.
|
|
||||||
func TestOriginChangesTakeResetsTheDroppedCount(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
table := routetable.New(routetable.Params{})
|
|
||||||
addr := netip.MustParseAddr("10.0.0.0")
|
|
||||||
|
|
||||||
for range routetable.MaxWaitingOriginChanges + 1 {
|
|
||||||
table.Announce(netip.PrefixFrom(addr, 32), asA, routeOf(peerA, asA, 10))
|
|
||||||
addr = addr.Next()
|
|
||||||
}
|
|
||||||
|
|
||||||
_, dropped := table.TakeOriginChanges(routetable.MaxWaitingOriginChanges)
|
|
||||||
if dropped != 1 {
|
|
||||||
t.Fatalf("dropped = %d, want 1", dropped)
|
|
||||||
}
|
|
||||||
|
|
||||||
_, dropped = table.TakeOriginChanges(0)
|
|
||||||
if dropped != 0 {
|
|
||||||
t.Errorf("the next take's dropped = %d, want 0", dropped)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestOriginChangesPutBackTrimsToTheLimit checks that putting back what a
|
|
||||||
// failed write took, after new changes filled the table again, drops the
|
|
||||||
// oldest changes past the cap and counts them, so the cap still holds. The
|
|
||||||
// clock advances one second per change, so a change's time says its age.
|
|
||||||
func TestOriginChangesPutBackTrimsToTheLimit(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
const (
|
|
||||||
taken = 20000
|
|
||||||
added = 20000
|
|
||||||
)
|
|
||||||
|
|
||||||
now := time.Unix(0, 0)
|
|
||||||
table := routetable.New(routetable.Params{
|
|
||||||
Now: func() time.Time {
|
|
||||||
now = now.Add(time.Second)
|
|
||||||
|
|
||||||
return now
|
|
||||||
},
|
|
||||||
})
|
|
||||||
addr := netip.MustParseAddr("10.0.0.0")
|
|
||||||
|
|
||||||
announce := func(n int) {
|
|
||||||
for range n {
|
|
||||||
table.Announce(netip.PrefixFrom(addr, 32), asA, routeOf(peerA, asA, 10))
|
|
||||||
addr = addr.Next()
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
announce(routetable.MaxWaitingOriginChanges)
|
|
||||||
|
|
||||||
takenChanges, _ := table.TakeOriginChanges(taken)
|
|
||||||
|
|
||||||
announce(added)
|
|
||||||
|
|
||||||
table.PutBackOriginChanges(takenChanges)
|
|
||||||
|
|
||||||
changes, dropped := table.TakeOriginChanges(math.MaxInt)
|
|
||||||
if dropped != added {
|
|
||||||
t.Errorf("dropped = %d, want %d", dropped, added)
|
|
||||||
}
|
|
||||||
|
|
||||||
if len(changes) != routetable.MaxWaitingOriginChanges {
|
|
||||||
t.Errorf("len(changes) = %d, want %d", len(changes),
|
|
||||||
routetable.MaxWaitingOriginChanges)
|
|
||||||
}
|
|
||||||
|
|
||||||
// The put-back rows are the oldest, so they are the ones dropped. The
|
|
||||||
// first remaining change is the one that followed them.
|
|
||||||
if want := time.Unix(int64(added+1), 0); !changes[0].Time.Equal(want) {
|
|
||||||
t.Errorf("first change's time = %v, want %v", changes[0].Time, want)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// checkOriginChanges checks that TakeOriginChanges, taking every change,
|
|
||||||
// returns want.
|
|
||||||
func checkOriginChanges(
|
|
||||||
t *testing.T, table *routetable.Table, want []routetable.OriginChange,
|
|
||||||
) {
|
|
||||||
t.Helper()
|
|
||||||
|
|
||||||
changes, dropped := table.TakeOriginChanges(math.MaxInt)
|
|
||||||
if dropped != 0 {
|
|
||||||
t.Errorf("dropped = %d, want 0", dropped)
|
|
||||||
}
|
|
||||||
|
|
||||||
if !reflect.DeepEqual(changes, want) {
|
|
||||||
t.Errorf("TakeOriginChanges = %+v, want %+v", changes, want)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
@@ -1,264 +0,0 @@
|
|||||||
package routetable
|
|
||||||
|
|
||||||
import (
|
|
||||||
"cmp"
|
|
||||||
"net/netip"
|
|
||||||
"slices"
|
|
||||||
"time"
|
|
||||||
|
|
||||||
"git.eeqj.de/sneak/routewatch/internal/bogon"
|
|
||||||
"git.eeqj.de/sneak/routewatch/internal/ristypes"
|
|
||||||
)
|
|
||||||
|
|
||||||
// Overview holds figures about the whole table. Working them out takes a walk
|
|
||||||
// over every prefix, so a caller asks for them now and then and keeps them.
|
|
||||||
type Overview struct {
|
|
||||||
// NewestRoute and OldestRoute are the newest and the oldest kept routes,
|
|
||||||
// or nil when the table keeps none. Of several with the same time, it is
|
|
||||||
// any one of them.
|
|
||||||
NewestRoute *PrefixRoute
|
|
||||||
OldestRoute *PrefixRoute
|
|
||||||
|
|
||||||
// MostPrefixesASN is the AS that originates the most prefixes, and
|
|
||||||
// MostPrefixes is how many it originates. Of several with as many, it is
|
|
||||||
// the lowest AS number. AS 0, no known origin, counts neither here nor
|
|
||||||
// in the single-prefix figures.
|
|
||||||
MostPrefixesASN int
|
|
||||||
MostPrefixes int
|
|
||||||
|
|
||||||
// SinglePrefixASes counts the ASes that originate one prefix.
|
|
||||||
SinglePrefixASes int
|
|
||||||
// NewestSinglePrefix is the newest kept route of those ASes, or nil when
|
|
||||||
// they keep none.
|
|
||||||
NewestSinglePrefix *PrefixRoute
|
|
||||||
|
|
||||||
// Bogons are the bogon announcements, in prefix order: at most the
|
|
||||||
// number Overview is asked for. BogonCount counts them all.
|
|
||||||
Bogons []Bogon
|
|
||||||
BogonCount int
|
|
||||||
|
|
||||||
// MultipleOrigins are the prefixes that peers carry with more than one
|
|
||||||
// origin AS, in prefix order: at most the number Overview is asked for.
|
|
||||||
// MultipleOriginCount counts them all.
|
|
||||||
MultipleOrigins []MultipleOriginPrefix
|
|
||||||
MultipleOriginCount int
|
|
||||||
}
|
|
||||||
|
|
||||||
// PrefixRoute is the time of a kept route, with its prefix and origin AS.
|
|
||||||
type PrefixRoute struct {
|
|
||||||
Prefix netip.Prefix
|
|
||||||
OriginASN int
|
|
||||||
Time time.Time
|
|
||||||
}
|
|
||||||
|
|
||||||
// Bogon is a bogon announcement: a prefix and origin AS pair whose prefix or
|
|
||||||
// origin AS is reserved, as package bogon tells.
|
|
||||||
type Bogon struct {
|
|
||||||
Prefix netip.Prefix
|
|
||||||
OriginASN int
|
|
||||||
// Peers is how many peers carry the prefix with this origin AS.
|
|
||||||
Peers int
|
|
||||||
// Reasons say why the prefix, the origin AS or both are reserved.
|
|
||||||
Reasons []string
|
|
||||||
}
|
|
||||||
|
|
||||||
// MultipleOriginPrefix is a prefix that peers carry with more than one origin
|
|
||||||
// AS.
|
|
||||||
type MultipleOriginPrefix struct {
|
|
||||||
Prefix netip.Prefix
|
|
||||||
// Origins are its origin ASes, in AS number order.
|
|
||||||
Origins []OriginPeers
|
|
||||||
}
|
|
||||||
|
|
||||||
// OriginPeers is one origin AS of a prefix, and how many peers carry the prefix
|
|
||||||
// with it.
|
|
||||||
type OriginPeers struct {
|
|
||||||
ASN int
|
|
||||||
Peers int
|
|
||||||
}
|
|
||||||
|
|
||||||
// Overview works out the table's overview. It lists at most maxListed bogon
|
|
||||||
// announcements and at most maxListed prefixes with more than one origin AS:
|
|
||||||
// the first in prefix order. It visits every prefix in the table, in batches
|
|
||||||
// of walkBatch. After each batch, it lets the table's lock go for as long as
|
|
||||||
// it held the lock for the batch.
|
|
||||||
func (t *Table) Overview(maxListed int) Overview {
|
|
||||||
var overview Overview
|
|
||||||
|
|
||||||
t.walkPrefixes(func(prefix netip.Prefix, origins []*origin) {
|
|
||||||
overview.noteKeptRoutes(prefix, origins)
|
|
||||||
overview.noteBogons(prefix, origins)
|
|
||||||
overview.noteMultipleOrigins(prefix, origins)
|
|
||||||
})
|
|
||||||
|
|
||||||
t.countPrefixesPerOrigin(&overview)
|
|
||||||
|
|
||||||
slices.SortFunc(overview.Bogons, func(a, b Bogon) int {
|
|
||||||
return cmp.Or(a.Prefix.Compare(b.Prefix),
|
|
||||||
cmp.Compare(a.OriginASN, b.OriginASN))
|
|
||||||
})
|
|
||||||
overview.BogonCount = len(overview.Bogons)
|
|
||||||
overview.Bogons = overview.Bogons[:min(maxListed, overview.BogonCount)]
|
|
||||||
|
|
||||||
slices.SortFunc(overview.MultipleOrigins, func(a, b MultipleOriginPrefix) int {
|
|
||||||
return a.Prefix.Compare(b.Prefix)
|
|
||||||
})
|
|
||||||
overview.MultipleOriginCount = len(overview.MultipleOrigins)
|
|
||||||
overview.MultipleOrigins =
|
|
||||||
overview.MultipleOrigins[:min(maxListed, overview.MultipleOriginCount)]
|
|
||||||
|
|
||||||
return overview
|
|
||||||
}
|
|
||||||
|
|
||||||
// walkPrefixes calls visit with every prefix of the table and its origins. It
|
|
||||||
// holds the read lock for walkBatch prefixes at a time. After each batch, it
|
|
||||||
// lets the table's lock go for as long as it held the lock for the batch.
|
|
||||||
// During a walk over many batches, updates can then take the lock about half
|
|
||||||
// of the time. The walk may or may not visit a prefix that is added or removed
|
|
||||||
// while the lock is let go. Go allows this in a range over a map that changes.
|
|
||||||
// visit must not keep origins once it returns.
|
|
||||||
func (t *Table) walkPrefixes(visit func(prefix netip.Prefix, origins []*origin)) {
|
|
||||||
t.mu.RLock()
|
|
||||||
defer t.mu.RUnlock()
|
|
||||||
|
|
||||||
visited := 0
|
|
||||||
batchStart := time.Now()
|
|
||||||
|
|
||||||
for _, f := range []*family{&t.ipv4, &t.ipv6} {
|
|
||||||
for _, prefixes := range f.byLength {
|
|
||||||
for prefix, origins := range prefixes {
|
|
||||||
visit(prefix, origins)
|
|
||||||
|
|
||||||
visited++
|
|
||||||
if visited%walkBatch == 0 {
|
|
||||||
held := time.Since(batchStart)
|
|
||||||
|
|
||||||
t.mu.RUnlock()
|
|
||||||
time.Sleep(held)
|
|
||||||
t.mu.RLock()
|
|
||||||
|
|
||||||
batchStart = time.Now()
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// countPrefixesPerOrigin works out the overview's figures of how many prefixes
|
|
||||||
// each AS originates. It holds the read lock once, for one look at each origin
|
|
||||||
// AS, of which there are far fewer than prefixes.
|
|
||||||
func (t *Table) countPrefixesPerOrigin(overview *Overview) {
|
|
||||||
t.mu.RLock()
|
|
||||||
defer t.mu.RUnlock()
|
|
||||||
|
|
||||||
for asn, prefixes := range t.byOrigin {
|
|
||||||
if asn == ristypes.NoOrigin {
|
|
||||||
continue
|
|
||||||
}
|
|
||||||
|
|
||||||
count := len(prefixes)
|
|
||||||
if count > overview.MostPrefixes ||
|
|
||||||
(count == overview.MostPrefixes && asn < overview.MostPrefixesASN) {
|
|
||||||
overview.MostPrefixesASN = asn
|
|
||||||
overview.MostPrefixes = count
|
|
||||||
}
|
|
||||||
|
|
||||||
if count == 1 {
|
|
||||||
overview.SinglePrefixASes++
|
|
||||||
overview.noteSinglePrefix(asn, prefixes)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// noteSinglePrefix makes the newest kept route of asn, whose one prefix is in
|
|
||||||
// prefixes, the overview's newest single-prefix route when it is newer.
|
|
||||||
func (overview *Overview) noteSinglePrefix(
|
|
||||||
asn int, prefixes map[netip.Prefix]*origin,
|
|
||||||
) {
|
|
||||||
for prefix, o := range prefixes {
|
|
||||||
newest := o.newestTime()
|
|
||||||
if newest == nil {
|
|
||||||
continue
|
|
||||||
}
|
|
||||||
|
|
||||||
if overview.NewestSinglePrefix == nil ||
|
|
||||||
newest.After(overview.NewestSinglePrefix.Time) {
|
|
||||||
overview.NewestSinglePrefix = &PrefixRoute{
|
|
||||||
Prefix: prefix, OriginASN: asn, Time: *newest,
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// noteKeptRoutes makes the newest and the oldest kept routes of prefix's
|
|
||||||
// origins the overview's newest and oldest routes when they are newer or older.
|
|
||||||
func (overview *Overview) noteKeptRoutes(prefix netip.Prefix, origins []*origin) {
|
|
||||||
for _, o := range origins {
|
|
||||||
if len(o.routes) == 0 {
|
|
||||||
continue
|
|
||||||
}
|
|
||||||
|
|
||||||
newest := o.routes[0].time
|
|
||||||
if overview.NewestRoute == nil || newest.After(overview.NewestRoute.Time) {
|
|
||||||
overview.NewestRoute = &PrefixRoute{
|
|
||||||
Prefix: prefix, OriginASN: o.asn, Time: newest,
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
oldest := o.routes[len(o.routes)-1].time
|
|
||||||
if overview.OldestRoute == nil || oldest.Before(overview.OldestRoute.Time) {
|
|
||||||
overview.OldestRoute = &PrefixRoute{
|
|
||||||
Prefix: prefix, OriginASN: o.asn, Time: oldest,
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// noteBogons adds each origin of prefix to the overview's bogon announcements
|
|
||||||
// when the prefix or the origin AS is reserved.
|
|
||||||
func (overview *Overview) noteBogons(prefix netip.Prefix, origins []*origin) {
|
|
||||||
prefixReason, prefixReserved := bogon.Prefix(prefix)
|
|
||||||
|
|
||||||
for _, o := range origins {
|
|
||||||
asnReason, asnReserved := bogon.ASN(o.asn)
|
|
||||||
if !prefixReserved && !asnReserved {
|
|
||||||
continue
|
|
||||||
}
|
|
||||||
|
|
||||||
announcement := Bogon{
|
|
||||||
Prefix: prefix, OriginASN: o.asn, Peers: int(o.peers.Count()),
|
|
||||||
}
|
|
||||||
|
|
||||||
if prefixReserved {
|
|
||||||
announcement.Reasons = append(announcement.Reasons, prefixReason)
|
|
||||||
}
|
|
||||||
|
|
||||||
if asnReserved {
|
|
||||||
announcement.Reasons = append(announcement.Reasons, asnReason)
|
|
||||||
}
|
|
||||||
|
|
||||||
overview.Bogons = append(overview.Bogons, announcement)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// noteMultipleOrigins adds prefix to the overview's prefixes with more than one
|
|
||||||
// origin AS when it has more than one.
|
|
||||||
func (overview *Overview) noteMultipleOrigins(
|
|
||||||
prefix netip.Prefix, origins []*origin,
|
|
||||||
) {
|
|
||||||
if len(origins) <= 1 {
|
|
||||||
return
|
|
||||||
}
|
|
||||||
|
|
||||||
entry := MultipleOriginPrefix{
|
|
||||||
Prefix: prefix,
|
|
||||||
Origins: make([]OriginPeers, 0, len(origins)),
|
|
||||||
}
|
|
||||||
|
|
||||||
for _, o := range origins {
|
|
||||||
entry.Origins = append(entry.Origins,
|
|
||||||
OriginPeers{ASN: o.asn, Peers: int(o.peers.Count())})
|
|
||||||
}
|
|
||||||
|
|
||||||
overview.MultipleOrigins = append(overview.MultipleOrigins, entry)
|
|
||||||
}
|
|
||||||
@@ -1,191 +0,0 @@
|
|||||||
package routetable
|
|
||||||
|
|
||||||
import (
|
|
||||||
"net/netip"
|
|
||||||
"runtime"
|
|
||||||
"testing"
|
|
||||||
"time"
|
|
||||||
)
|
|
||||||
|
|
||||||
// TestWalkPrefixesVisitsEveryPrefix walks a table with more prefixes than two
|
|
||||||
// of the walk's batches, while another goroutine keeps adding and removing
|
|
||||||
// other prefixes of the same mask length. The walk lets the lock go between
|
|
||||||
// batches, so those updates get in while it runs. Every prefix that was in
|
|
||||||
// the table all along must be visited once.
|
|
||||||
func TestWalkPrefixesVisitsEveryPrefix(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
const (
|
|
||||||
asn = 1000
|
|
||||||
peer = "192.0.2.1"
|
|
||||||
// prefixCount is more prefixes than two of the walk's batches.
|
|
||||||
prefixCount = 2*walkBatch + 1
|
|
||||||
)
|
|
||||||
|
|
||||||
routeTime := time.Date(2026, 1, 2, 3, 4, 5, 0, time.UTC)
|
|
||||||
table := New(Params{})
|
|
||||||
prefixes := make([]netip.Prefix, 0, prefixCount)
|
|
||||||
addr := netip.MustParseAddr("1.0.0.0")
|
|
||||||
|
|
||||||
for range prefixCount {
|
|
||||||
prefix := netip.PrefixFrom(addr, addr.BitLen())
|
|
||||||
table.Announce(prefix, asn, Route{Peer: peer, Time: routeTime})
|
|
||||||
prefixes = append(prefixes, prefix)
|
|
||||||
addr = addr.Next()
|
|
||||||
}
|
|
||||||
|
|
||||||
stop := make(chan struct{})
|
|
||||||
updated := make(chan struct{})
|
|
||||||
|
|
||||||
go func() {
|
|
||||||
defer close(updated)
|
|
||||||
|
|
||||||
other := netip.MustParseAddr("2.0.0.0")
|
|
||||||
|
|
||||||
for {
|
|
||||||
select {
|
|
||||||
case <-stop:
|
|
||||||
return
|
|
||||||
default:
|
|
||||||
}
|
|
||||||
|
|
||||||
prefix := netip.PrefixFrom(other, other.BitLen())
|
|
||||||
table.Announce(prefix, asn, Route{Peer: peer, Time: routeTime})
|
|
||||||
table.Withdraw(prefix, peer)
|
|
||||||
|
|
||||||
other = other.Next()
|
|
||||||
}
|
|
||||||
}()
|
|
||||||
|
|
||||||
visits := make(map[netip.Prefix]int)
|
|
||||||
|
|
||||||
table.walkPrefixes(func(prefix netip.Prefix, _ []*origin) {
|
|
||||||
visits[prefix]++
|
|
||||||
})
|
|
||||||
|
|
||||||
close(stop)
|
|
||||||
<-updated
|
|
||||||
|
|
||||||
for _, prefix := range prefixes {
|
|
||||||
if visits[prefix] != 1 {
|
|
||||||
t.Fatalf("the walk visited %s %d times, want once", prefix,
|
|
||||||
visits[prefix])
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestWalkPrefixesLetsUpdatesIn starts an update during the walk's first
|
|
||||||
// batch. The update must have been made by the time the walk visits the first
|
|
||||||
// prefix of its second batch, because the walk lets the lock go in between.
|
|
||||||
func TestWalkPrefixesLetsUpdatesIn(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
const (
|
|
||||||
asn = 1000
|
|
||||||
peer = "192.0.2.1"
|
|
||||||
)
|
|
||||||
|
|
||||||
routeTime := time.Date(2026, 1, 2, 3, 4, 5, 0, time.UTC)
|
|
||||||
table := New(Params{})
|
|
||||||
addr := netip.MustParseAddr("1.0.0.0")
|
|
||||||
|
|
||||||
for range walkBatch + 1 {
|
|
||||||
prefix := netip.PrefixFrom(addr, addr.BitLen())
|
|
||||||
table.Announce(prefix, asn, Route{Peer: peer, Time: routeTime})
|
|
||||||
|
|
||||||
addr = addr.Next()
|
|
||||||
}
|
|
||||||
|
|
||||||
updatePrefix := netip.MustParsePrefix("2.0.0.0/8")
|
|
||||||
updateDone := make(chan struct{})
|
|
||||||
visited := 0
|
|
||||||
madeBeforeSecondBatch := false
|
|
||||||
|
|
||||||
table.walkPrefixes(func(netip.Prefix, []*origin) {
|
|
||||||
visited++
|
|
||||||
|
|
||||||
switch visited {
|
|
||||||
case 1:
|
|
||||||
go func() {
|
|
||||||
defer close(updateDone)
|
|
||||||
|
|
||||||
table.Announce(updatePrefix, asn, Route{Peer: peer, Time: routeTime})
|
|
||||||
}()
|
|
||||||
|
|
||||||
// Wait until the update waits for the lock. TryRLock fails
|
|
||||||
// once a writer waits.
|
|
||||||
for table.mu.TryRLock() {
|
|
||||||
table.mu.RUnlock()
|
|
||||||
runtime.Gosched()
|
|
||||||
}
|
|
||||||
case walkBatch + 1:
|
|
||||||
// The walk holds the read lock, so the table can be read.
|
|
||||||
madeBeforeSecondBatch = table.origins(updatePrefix) != nil
|
|
||||||
}
|
|
||||||
})
|
|
||||||
|
|
||||||
<-updateDone
|
|
||||||
|
|
||||||
if !madeBeforeSecondBatch {
|
|
||||||
t.Fatal("an update started during the walk's first batch was not made " +
|
|
||||||
"before its second batch")
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestOverviewLetsUpdatesIn applies another peer's 1,000 announcements while
|
|
||||||
// Overview walks a table with more prefixes than five of the walk's batches.
|
|
||||||
// After each batch, the walk lets the lock go for as long as it held the lock.
|
|
||||||
// So all 1,000 announcements must be applied before Overview returns. A walk
|
|
||||||
// that takes the lock back at once lets in about one update per batch.
|
|
||||||
func TestOverviewLetsUpdatesIn(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
const (
|
|
||||||
asn = 1000
|
|
||||||
peer = "192.0.2.1"
|
|
||||||
otherPeer = "192.0.2.2"
|
|
||||||
// prefixCount is more prefixes than five of the walk's batches.
|
|
||||||
prefixCount = 5*walkBatch + 1
|
|
||||||
announcements = 1000
|
|
||||||
)
|
|
||||||
|
|
||||||
routeTime := time.Date(2026, 1, 2, 3, 4, 5, 0, time.UTC)
|
|
||||||
table := New(Params{})
|
|
||||||
prefixes := make([]netip.Prefix, 0, prefixCount)
|
|
||||||
addr := netip.MustParseAddr("1.0.0.0")
|
|
||||||
|
|
||||||
for range prefixCount {
|
|
||||||
prefix := netip.PrefixFrom(addr, addr.BitLen())
|
|
||||||
table.Announce(prefix, asn, Route{Peer: peer, Time: routeTime})
|
|
||||||
prefixes = append(prefixes, prefix)
|
|
||||||
addr = addr.Next()
|
|
||||||
}
|
|
||||||
|
|
||||||
overviewDone := make(chan struct{})
|
|
||||||
|
|
||||||
go func() {
|
|
||||||
defer close(overviewDone)
|
|
||||||
|
|
||||||
table.Overview(0)
|
|
||||||
}()
|
|
||||||
|
|
||||||
// Wait until Overview holds the read lock. TryLock fails while a reader
|
|
||||||
// holds it.
|
|
||||||
for table.mu.TryLock() {
|
|
||||||
table.mu.Unlock()
|
|
||||||
runtime.Gosched()
|
|
||||||
}
|
|
||||||
|
|
||||||
for _, prefix := range prefixes[:announcements] {
|
|
||||||
table.Announce(prefix, asn, Route{Peer: otherPeer, Time: routeTime})
|
|
||||||
}
|
|
||||||
|
|
||||||
select {
|
|
||||||
case <-overviewDone:
|
|
||||||
t.Fatal("Overview returned before another peer's 1,000 announcements " +
|
|
||||||
"were applied")
|
|
||||||
default:
|
|
||||||
}
|
|
||||||
|
|
||||||
<-overviewDone
|
|
||||||
}
|
|
||||||
@@ -1,231 +0,0 @@
|
|||||||
package routetable_test
|
|
||||||
|
|
||||||
import (
|
|
||||||
"net/netip"
|
|
||||||
"reflect"
|
|
||||||
"testing"
|
|
||||||
|
|
||||||
"git.eeqj.de/sneak/routewatch/internal/routetable"
|
|
||||||
)
|
|
||||||
|
|
||||||
// Origin ASes that are not reserved, unlike asA, asB and asC, which are
|
|
||||||
// documentation AS numbers and so make bogon announcements.
|
|
||||||
const (
|
|
||||||
asOne = 1000
|
|
||||||
asTwo = 2000
|
|
||||||
asThree = 3000
|
|
||||||
asFour = 4000
|
|
||||||
asFive = 5000
|
|
||||||
|
|
||||||
// noKnownOrigin is the origin AS of a route whose path ends in an AS
|
|
||||||
// set.
|
|
||||||
noKnownOrigin = 0
|
|
||||||
// privateUseAS and asTrans are reserved AS numbers.
|
|
||||||
privateUseAS = 64512
|
|
||||||
asTrans = 23456
|
|
||||||
|
|
||||||
// maxListed is how many bogon announcements and prefixes with more than
|
|
||||||
// one origin AS the tests ask Overview for.
|
|
||||||
maxListed = 10
|
|
||||||
)
|
|
||||||
|
|
||||||
// prefixRoute returns the PrefixRoute of prefix with origin AS asn, second
|
|
||||||
// seconds after the start.
|
|
||||||
func prefixRoute(prefix string, asn, second int) *routetable.PrefixRoute {
|
|
||||||
return &routetable.PrefixRoute{
|
|
||||||
Prefix: netip.MustParsePrefix(prefix), OriginASN: asn, Time: at(second),
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestOverviewOfAnEmptyTable(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
got := routetable.New(routetable.Params{}).Overview(maxListed)
|
|
||||||
if !reflect.DeepEqual(got, routetable.Overview{}) {
|
|
||||||
t.Errorf("Overview() = %+v, want no figures", got)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestOverviewNewestAndOldestRoutes checks that the newest and the oldest kept
|
|
||||||
// routes are found among every origin's kept routes, and that withdrawn routes
|
|
||||||
// no longer count.
|
|
||||||
func TestOverviewNewestAndOldestRoutes(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
table := routetable.New(routetable.Params{})
|
|
||||||
announceAll(table, "1.0.0.0/24", asOne, 20, peerA, peerB)
|
|
||||||
// The oldest is the older of this origin's two kept routes.
|
|
||||||
announceAll(table, "2.0.0.0/24", asTwo, 10, peerA, peerB)
|
|
||||||
announceAll(table, "2001:4860::/32", asThree, 40, peerC)
|
|
||||||
// Withdrawn routes, newer and older than every other.
|
|
||||||
announceAll(table, "3.0.0.0/24", asOne, 50, peerA)
|
|
||||||
announceAll(table, "4.0.0.0/24", asOne, 5, peerB)
|
|
||||||
table.Withdraw(netip.MustParsePrefix("3.0.0.0/24"), peerA)
|
|
||||||
table.Withdraw(netip.MustParsePrefix("4.0.0.0/24"), peerB)
|
|
||||||
|
|
||||||
got := table.Overview(maxListed)
|
|
||||||
|
|
||||||
want := prefixRoute("2001:4860::/32", asThree, 40)
|
|
||||||
if !reflect.DeepEqual(got.NewestRoute, want) {
|
|
||||||
t.Errorf("NewestRoute = %+v, want %+v", got.NewestRoute, want)
|
|
||||||
}
|
|
||||||
|
|
||||||
want = prefixRoute("2.0.0.0/24", asTwo, 10)
|
|
||||||
if !reflect.DeepEqual(got.OldestRoute, want) {
|
|
||||||
t.Errorf("OldestRoute = %+v, want %+v", got.OldestRoute, want)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestOverviewPrefixesPerOrigin checks the AS that originates the most
|
|
||||||
// prefixes and the ASes that originate one.
|
|
||||||
func TestOverviewPrefixesPerOrigin(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
table := routetable.New(routetable.Params{})
|
|
||||||
// asOne and asTwo originate two prefixes each; the lower AS number counts.
|
|
||||||
announceAll(table, "1.0.0.0/24", asTwo, 10, peerA)
|
|
||||||
announceAll(table, "1.0.1.0/24", asTwo, 10, peerA)
|
|
||||||
announceAll(table, "2.0.0.0/24", asOne, 10, peerA)
|
|
||||||
announceAll(table, "2.0.1.0/24", asOne, 10, peerB)
|
|
||||||
// Three ASes originate one prefix. asThree's route is the newest, and
|
|
||||||
// asFive's origin keeps no route: peers B and C withdrew theirs.
|
|
||||||
announceAll(table, "3.0.0.0/24", asThree, 30, peerA)
|
|
||||||
announceAll(table, "4.0.0.0/24", asFour, 20, peerA)
|
|
||||||
announceAll(table, "5.0.0.0/24", asFive, 40, peerA, peerB, peerC)
|
|
||||||
table.Withdraw(netip.MustParsePrefix("5.0.0.0/24"), peerB)
|
|
||||||
table.Withdraw(netip.MustParsePrefix("5.0.0.0/24"), peerC)
|
|
||||||
|
|
||||||
got := table.Overview(maxListed)
|
|
||||||
|
|
||||||
if got.MostPrefixesASN != asOne || got.MostPrefixes != 2 {
|
|
||||||
t.Errorf("most prefixes: AS%d with %d, want AS%d with 2",
|
|
||||||
got.MostPrefixesASN, got.MostPrefixes, asOne)
|
|
||||||
}
|
|
||||||
|
|
||||||
if got.SinglePrefixASes != 3 {
|
|
||||||
t.Errorf("SinglePrefixASes = %d, want 3", got.SinglePrefixASes)
|
|
||||||
}
|
|
||||||
|
|
||||||
want := prefixRoute("3.0.0.0/24", asThree, 30)
|
|
||||||
if !reflect.DeepEqual(got.NewestSinglePrefix, want) {
|
|
||||||
t.Errorf("NewestSinglePrefix = %+v, want %+v", got.NewestSinglePrefix, want)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestOverviewLeavesOutNoKnownOrigin checks that AS 0, which stands for no
|
|
||||||
// known origin, counts in neither the most nor the single-prefix figures. It
|
|
||||||
// originates one prefix, with a route newer than asOne's, and asOne would lose
|
|
||||||
// the tie for the most prefixes to it.
|
|
||||||
func TestOverviewLeavesOutNoKnownOrigin(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
table := routetable.New(routetable.Params{})
|
|
||||||
announceAll(table, "1.0.0.0/24", asOne, 10, peerA)
|
|
||||||
announceAll(table, "2.0.0.0/24", noKnownOrigin, 20, peerA)
|
|
||||||
|
|
||||||
got := table.Overview(maxListed)
|
|
||||||
|
|
||||||
if got.MostPrefixesASN != asOne || got.SinglePrefixASes != 1 {
|
|
||||||
t.Errorf("most prefixes AS%d, %d single-prefix ASes; want AS%d, 1",
|
|
||||||
got.MostPrefixesASN, got.SinglePrefixASes, asOne)
|
|
||||||
}
|
|
||||||
|
|
||||||
want := prefixRoute("1.0.0.0/24", asOne, 10)
|
|
||||||
if !reflect.DeepEqual(got.NewestSinglePrefix, want) {
|
|
||||||
t.Errorf("NewestSinglePrefix = %+v, want %+v", got.NewestSinglePrefix, want)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestOverviewBogons checks which routes are bogon announcements and why, in
|
|
||||||
// prefix order, and that a smaller maxListed lists fewer but counts them all.
|
|
||||||
func TestOverviewBogons(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
table := routetable.New(routetable.Params{})
|
|
||||||
// A reserved prefix, carried by two peers.
|
|
||||||
announceAll(table, "192.168.0.0/16", asOne, 10, peerA, peerB)
|
|
||||||
// A reserved origin AS.
|
|
||||||
announceAll(table, "1.0.0.0/24", privateUseAS, 10, peerA)
|
|
||||||
// Both.
|
|
||||||
announceAll(table, "10.0.0.0/8", asTrans, 10, peerA)
|
|
||||||
// Neither: a normal route, and a block that may be announced although it
|
|
||||||
// lies inside a reserved one.
|
|
||||||
announceAll(table, "1.0.1.0/24", asOne, 10, peerA)
|
|
||||||
announceAll(table, "2001:4:112::/48", asTwo, 10, peerA)
|
|
||||||
|
|
||||||
want := []routetable.Bogon{
|
|
||||||
{
|
|
||||||
Prefix: netip.MustParsePrefix("1.0.0.0/24"), OriginASN: privateUseAS,
|
|
||||||
Peers: 1, Reasons: []string{"Private-Use (AS64512-AS65534)"},
|
|
||||||
},
|
|
||||||
{
|
|
||||||
Prefix: netip.MustParsePrefix("10.0.0.0/8"), OriginASN: asTrans,
|
|
||||||
Peers: 1,
|
|
||||||
Reasons: []string{
|
|
||||||
"Private-Use (10.0.0.0/8)", "AS_TRANS (AS23456)",
|
|
||||||
},
|
|
||||||
},
|
|
||||||
{
|
|
||||||
Prefix: netip.MustParsePrefix("192.168.0.0/16"), OriginASN: asOne,
|
|
||||||
Peers: 2, Reasons: []string{"Private-Use (192.168.0.0/16)"},
|
|
||||||
},
|
|
||||||
}
|
|
||||||
|
|
||||||
got := table.Overview(maxListed)
|
|
||||||
if !reflect.DeepEqual(got.Bogons, want) || got.BogonCount != len(want) {
|
|
||||||
t.Errorf("Overview(%d): %d bogons %+v, want %d %+v", maxListed,
|
|
||||||
got.BogonCount, got.Bogons, len(want), want)
|
|
||||||
}
|
|
||||||
|
|
||||||
const fewer = 2
|
|
||||||
|
|
||||||
got = table.Overview(fewer)
|
|
||||||
if !reflect.DeepEqual(got.Bogons, want[:fewer]) || got.BogonCount != len(want) {
|
|
||||||
t.Errorf("Overview(%d): %d bogons %+v, want %d %+v", fewer,
|
|
||||||
got.BogonCount, got.Bogons, len(want), want[:fewer])
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestOverviewMultipleOrigins checks the prefixes with more than one origin
|
|
||||||
// AS, in prefix order, and that a smaller maxListed lists fewer but counts them
|
|
||||||
// all.
|
|
||||||
func TestOverviewMultipleOrigins(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
table := routetable.New(routetable.Params{})
|
|
||||||
announceAll(table, "2001:4860::/32", asThree, 10, peerA)
|
|
||||||
announceAll(table, "2001:4860::/32", asTwo, 10, peerB)
|
|
||||||
announceAll(table, "1.0.0.0/24", asOne, 10, peerA, peerB)
|
|
||||||
announceAll(table, "1.0.0.0/24", asTwo, 30, peerC)
|
|
||||||
// One origin only.
|
|
||||||
announceAll(table, "2.0.0.0/24", asOne, 10, peerA, peerB)
|
|
||||||
|
|
||||||
want := []routetable.MultipleOriginPrefix{
|
|
||||||
{
|
|
||||||
Prefix: netip.MustParsePrefix("1.0.0.0/24"),
|
|
||||||
Origins: []routetable.OriginPeers{
|
|
||||||
{ASN: asOne, Peers: 2}, {ASN: asTwo, Peers: 1},
|
|
||||||
},
|
|
||||||
},
|
|
||||||
{
|
|
||||||
Prefix: netip.MustParsePrefix("2001:4860::/32"),
|
|
||||||
Origins: []routetable.OriginPeers{
|
|
||||||
{ASN: asTwo, Peers: 1}, {ASN: asThree, Peers: 1},
|
|
||||||
},
|
|
||||||
},
|
|
||||||
}
|
|
||||||
|
|
||||||
got := table.Overview(maxListed)
|
|
||||||
if !reflect.DeepEqual(got.MultipleOrigins, want) ||
|
|
||||||
got.MultipleOriginCount != len(want) {
|
|
||||||
t.Errorf("Overview(%d): %d prefixes %+v, want %d %+v", maxListed,
|
|
||||||
got.MultipleOriginCount, got.MultipleOrigins, len(want), want)
|
|
||||||
}
|
|
||||||
|
|
||||||
got = table.Overview(1)
|
|
||||||
if !reflect.DeepEqual(got.MultipleOrigins, want[:1]) ||
|
|
||||||
got.MultipleOriginCount != len(want) {
|
|
||||||
t.Errorf("Overview(1): %d prefixes %+v, want %d %+v",
|
|
||||||
got.MultipleOriginCount, got.MultipleOrigins, len(want), want[:1])
|
|
||||||
}
|
|
||||||
}
|
|
||||||
@@ -14,9 +14,6 @@ type PrefixSummary struct {
|
|||||||
// OriginASN is the origin AS that the most peers carry the prefix with.
|
// OriginASN is the origin AS that the most peers carry the prefix with.
|
||||||
// When several have as many peers, it is the lowest AS number.
|
// When several have as many peers, it is the lowest AS number.
|
||||||
OriginASN int
|
OriginASN int
|
||||||
// OriginASNs are all the origin ASes that peers carry the prefix with, in
|
|
||||||
// AS number order.
|
|
||||||
OriginASNs []int
|
|
||||||
// Peers is how many peers carry the prefix.
|
// Peers is how many peers carry the prefix.
|
||||||
Peers int
|
Peers int
|
||||||
// Newest is the time of the newest kept route of the prefix, or nil when
|
// Newest is the time of the newest kept route of the prefix, or nil when
|
||||||
@@ -214,12 +211,10 @@ func (t *Table) addresses(peers *bitset.BitSet) []string {
|
|||||||
// summarize returns the summary of prefix, whose origins are in AS number
|
// summarize returns the summary of prefix, whose origins are in AS number
|
||||||
// order.
|
// order.
|
||||||
func summarize(prefix netip.Prefix, origins []*origin) PrefixSummary {
|
func summarize(prefix netip.Prefix, origins []*origin) PrefixSummary {
|
||||||
summary := PrefixSummary{Prefix: prefix, OriginASNs: make([]int, 0, len(origins))}
|
summary := PrefixSummary{Prefix: prefix}
|
||||||
mostPeers := 0
|
mostPeers := 0
|
||||||
|
|
||||||
for _, o := range origins {
|
for _, o := range origins {
|
||||||
summary.OriginASNs = append(summary.OriginASNs, o.asn)
|
|
||||||
|
|
||||||
peers := int(o.peers.Count())
|
peers := int(o.peers.Count())
|
||||||
summary.Peers += peers
|
summary.Peers += peers
|
||||||
|
|
||||||
|
|||||||
@@ -46,22 +46,21 @@ func TestLookup(t *testing.T) {
|
|||||||
tests := []struct {
|
tests := []struct {
|
||||||
addr string
|
addr string
|
||||||
// want is the summary's prefix, or empty when none is found.
|
// want is the summary's prefix, or empty when none is found.
|
||||||
want string
|
want string
|
||||||
originASN int
|
originASN int
|
||||||
originASNs []int
|
peers int
|
||||||
peers int
|
newest int // in seconds after the start; 0 for none
|
||||||
newest int // in seconds after the start; 0 for none
|
|
||||||
}{
|
}{
|
||||||
{"10.1.2.3", prefixV4, asA, []int{asA, asB}, 2, 20},
|
{"10.1.2.3", prefixV4, asA, 2, 20},
|
||||||
{"::ffff:10.1.2.3", prefixV4, asA, []int{asA, asB}, 2, 20},
|
{"::ffff:10.1.2.3", prefixV4, asA, 2, 20},
|
||||||
{"10.1.3.4", "10.1.0.0/16", asB, []int{asA, asB}, 3, 30},
|
{"10.1.3.4", "10.1.0.0/16", asB, 3, 30},
|
||||||
{"10.2.0.1", "10.0.0.0/8", asA, []int{asA}, 1, 10},
|
{"10.2.0.1", "10.0.0.0/8", asA, 1, 10},
|
||||||
{"10.9.0.1", "10.9.0.0/16", asA, []int{asA}, 1, 0},
|
{"10.9.0.1", "10.9.0.0/16", asA, 1, 0},
|
||||||
{"192.0.2.1", "", 0, nil, 0, 0},
|
{"192.0.2.1", "", 0, 0, 0},
|
||||||
{"2001:db8:1::5", prefixV6, asB, []int{asB}, 1, 20},
|
{"2001:db8:1::5", prefixV6, asB, 1, 20},
|
||||||
{"2001:db8:2::1", "2001:db8::/32", asA, []int{asA}, 1, 10},
|
{"2001:db8:2::1", "2001:db8::/32", asA, 1, 10},
|
||||||
{"3000::1", "3000::/16", asA, []int{asA}, 1, 10},
|
{"3000::1", "3000::/16", asA, 1, 10},
|
||||||
{"2001:db9::1", "", 0, nil, 0, 0},
|
{"2001:db9::1", "", 0, 0, 0},
|
||||||
}
|
}
|
||||||
|
|
||||||
for _, tc := range tests {
|
for _, tc := range tests {
|
||||||
@@ -78,10 +77,9 @@ func TestLookup(t *testing.T) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
want := routetable.PrefixSummary{
|
want := routetable.PrefixSummary{
|
||||||
Prefix: netip.MustParsePrefix(tc.want),
|
Prefix: netip.MustParsePrefix(tc.want),
|
||||||
OriginASN: tc.originASN,
|
OriginASN: tc.originASN,
|
||||||
OriginASNs: tc.originASNs,
|
Peers: tc.peers,
|
||||||
Peers: tc.peers,
|
|
||||||
}
|
}
|
||||||
if tc.newest != 0 {
|
if tc.newest != 0 {
|
||||||
want.Newest = timeAt(tc.newest)
|
want.Newest = timeAt(tc.newest)
|
||||||
@@ -168,8 +166,7 @@ func TestSamplePrefixes(t *testing.T) {
|
|||||||
|
|
||||||
for _, got := range sample {
|
for _, got := range sample {
|
||||||
want := routetable.PrefixSummary{
|
want := routetable.PrefixSummary{
|
||||||
Prefix: got.Prefix, OriginASN: asA, OriginASNs: []int{asA},
|
Prefix: got.Prefix, OriginASN: asA, Peers: 1, Newest: timeAt(10),
|
||||||
Peers: 1, Newest: timeAt(10),
|
|
||||||
}
|
}
|
||||||
if seen[got.Prefix] || got.Prefix.Bits() != tc.maskLength ||
|
if seen[got.Prefix] || got.Prefix.Bits() != tc.maskLength ||
|
||||||
!reflect.DeepEqual(got, want) {
|
!reflect.DeepEqual(got, want) {
|
||||||
@@ -227,7 +224,7 @@ func TestCounts(t *testing.T) {
|
|||||||
})
|
})
|
||||||
|
|
||||||
// peerC was the last peer of the IPv6 prefix. It keeps its number.
|
// peerC was the last peer of the IPv6 prefix. It keeps its number.
|
||||||
table.PeersDown([]string{peerC})
|
table.PeerDown(peerC)
|
||||||
checkCounts(t, table, routetable.Counts{
|
checkCounts(t, table, routetable.Counts{
|
||||||
Pairs: 2,
|
Pairs: 2,
|
||||||
Peers: 3,
|
Peers: 3,
|
||||||
|
|||||||
@@ -2,9 +2,7 @@
|
|||||||
// and origin AS, it keeps the peers whose current route to the prefix has that
|
// and origin AS, it keeps the peers whose current route to the prefix has that
|
||||||
// origin AS. It also keeps the two newest of those routes in full. It applies
|
// origin AS. It also keeps the two newest of those routes in full. It applies
|
||||||
// announcements, withdrawals and peers' sessions going down. It answers the
|
// announcements, withdrawals and peers' sessions going down. It answers the
|
||||||
// route reads that the pages and the API need. It also records each prefix and
|
// route reads that the pages and the API need.
|
||||||
// origin AS pair's origin changes: when it gains its first peer and when it
|
|
||||||
// loses its last.
|
|
||||||
package routetable
|
package routetable
|
||||||
|
|
||||||
import (
|
import (
|
||||||
@@ -29,11 +27,6 @@ const (
|
|||||||
// kept routes in order: each of its nodes holds up to 63 times.
|
// kept routes in order: each of its nodes holds up to 63 times.
|
||||||
keptTimesDegree = 32
|
keptTimesDegree = 32
|
||||||
|
|
||||||
// walkBatch is how many prefixes a walk over the whole table visits in
|
|
||||||
// one hold of the lock. Overview and PeersDown each walk the whole table.
|
|
||||||
// Callers waiting for the lock get it between batches.
|
|
||||||
walkBatch = 10000
|
|
||||||
|
|
||||||
ipv4Bits = 32
|
ipv4Bits = 32
|
||||||
ipv6Bits = 128
|
ipv6Bits = 128
|
||||||
|
|
||||||
@@ -93,9 +86,6 @@ type Params struct {
|
|||||||
// peer not in it gets the next number above all of them the first time it
|
// peer not in it gets the next number above all of them the first time it
|
||||||
// announces a route.
|
// announces a route.
|
||||||
PeerNumbers map[string]uint
|
PeerNumbers map[string]uint
|
||||||
// Now returns the time RouteWatch applies a change at. It is time.Now
|
|
||||||
// unless a test gives one.
|
|
||||||
Now func() time.Time
|
|
||||||
}
|
}
|
||||||
|
|
||||||
// Table is the routing table. Its methods are safe for concurrent use.
|
// Table is the routing table. Its methods are safe for concurrent use.
|
||||||
@@ -131,24 +121,11 @@ type Table struct {
|
|||||||
// originsWithoutKeptRoute counts the origins that peers carry but that
|
// originsWithoutKeptRoute counts the origins that peers carry but that
|
||||||
// keep no route.
|
// keep no route.
|
||||||
originsWithoutKeptRoute int
|
originsWithoutKeptRoute int
|
||||||
|
|
||||||
// now returns the time RouteWatch applies a change at.
|
|
||||||
now func() time.Time
|
|
||||||
// originChanges holds the origin changes that wait to be written, oldest
|
|
||||||
// first. originChangesDropped counts those dropped since the last
|
|
||||||
// TakeOriginChanges.
|
|
||||||
originChanges []OriginChange
|
|
||||||
originChangesDropped int
|
|
||||||
}
|
}
|
||||||
|
|
||||||
// New returns an empty table that numbers new peers on from
|
// New returns an empty table that numbers new peers on from
|
||||||
// params.PeerNumbers.
|
// params.PeerNumbers.
|
||||||
func New(params Params) *Table {
|
func New(params Params) *Table {
|
||||||
now := params.Now
|
|
||||||
if now == nil {
|
|
||||||
now = time.Now
|
|
||||||
}
|
|
||||||
|
|
||||||
t := &Table{
|
t := &Table{
|
||||||
ipv4: newFamily(ipv4Bits, minLookupMaskLengthV4),
|
ipv4: newFamily(ipv4Bits, minLookupMaskLengthV4),
|
||||||
ipv6: newFamily(ipv6Bits, minLookupMaskLengthV6),
|
ipv6: newFamily(ipv6Bits, minLookupMaskLengthV6),
|
||||||
@@ -157,7 +134,6 @@ func New(params Params) *Table {
|
|||||||
peerAddresses: make(map[uint]string, len(params.PeerNumbers)),
|
peerAddresses: make(map[uint]string, len(params.PeerNumbers)),
|
||||||
newPeers: make(map[string]uint),
|
newPeers: make(map[string]uint),
|
||||||
changed: make(map[pair]struct{}),
|
changed: make(map[pair]struct{}),
|
||||||
now: now,
|
|
||||||
keptTimes: btree.NewG(keptTimesDegree, func(a, b keptTime) bool {
|
keptTimes: btree.NewG(keptTimesDegree, func(a, b keptTime) bool {
|
||||||
return a.time.Before(b.time)
|
return a.time.Before(b.time)
|
||||||
}),
|
}),
|
||||||
@@ -224,60 +200,24 @@ func (t *Table) Withdraw(prefix netip.Prefix, peer string) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// PeersDown applies the sessions of peers going down: each of the peers leaves
|
// PeerDown applies a peer's session going down: the peer leaves every peer
|
||||||
// every peer set, and every route of each is dropped. When any of them carries
|
// set, and every route of it is dropped. When the peer carries any prefix, it
|
||||||
// a prefix, PeersDown visits every prefix in the table once, in batches of
|
// visits every prefix in the table.
|
||||||
// walkBatch. After each batch, it lets the table's lock go for as long as it
|
func (t *Table) PeerDown(peer string) {
|
||||||
// held the lock for the batch. During a walk over many batches, reads and
|
|
||||||
// updates can then take the lock about half of the time. An announcement by
|
|
||||||
// one of the peers applied during the walk may be dropped too. So the caller
|
|
||||||
// applies those peers' later updates only once PeersDown has returned.
|
|
||||||
func (t *Table) PeersDown(peers []string) {
|
|
||||||
t.mu.Lock()
|
t.mu.Lock()
|
||||||
defer t.mu.Unlock()
|
defer t.mu.Unlock()
|
||||||
|
|
||||||
// numbers holds the number of each of the peers that carries a prefix.
|
number, known := t.peerNumbers[peer]
|
||||||
numbers := make([]uint, 0, len(peers))
|
if !known || t.peerPrefixes[number] == 0 {
|
||||||
|
|
||||||
for _, peer := range peers {
|
|
||||||
number, known := t.peerNumbers[peer]
|
|
||||||
if known && t.peerPrefixes[number] > 0 {
|
|
||||||
numbers = append(numbers, number)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
if len(numbers) == 0 {
|
|
||||||
return
|
return
|
||||||
}
|
}
|
||||||
|
|
||||||
visited := 0
|
|
||||||
batchStart := time.Now()
|
|
||||||
|
|
||||||
for _, f := range []*family{&t.ipv4, &t.ipv6} {
|
for _, f := range []*family{&t.ipv4, &t.ipv6} {
|
||||||
for _, prefixes := range f.byLength {
|
for _, prefixes := range f.byLength {
|
||||||
for prefix, origins := range prefixes {
|
for prefix, origins := range prefixes {
|
||||||
for _, number := range numbers {
|
o := originWithPeer(origins, number)
|
||||||
o := originWithPeer(origins, number)
|
if o != nil {
|
||||||
if o != nil {
|
t.leave(prefix, o, number)
|
||||||
t.leave(prefix, o, number)
|
|
||||||
|
|
||||||
// leave may have removed an origin of the
|
|
||||||
// prefix, or the prefix itself.
|
|
||||||
origins = prefixes[prefix]
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// Go allows the map to change before the next step of
|
|
||||||
// the range over it.
|
|
||||||
visited++
|
|
||||||
if visited%walkBatch == 0 {
|
|
||||||
held := time.Since(batchStart)
|
|
||||||
|
|
||||||
t.mu.Unlock()
|
|
||||||
time.Sleep(held)
|
|
||||||
t.mu.Lock()
|
|
||||||
|
|
||||||
batchStart = time.Now()
|
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -287,9 +227,8 @@ func (t *Table) PeersDown(peers []string) {
|
|||||||
// Load puts one prefix and origin AS pair that the database holds into the
|
// Load puts one prefix and origin AS pair that the database holds into the
|
||||||
// table, as startup does with each row it reads. The change's peers join the
|
// table, as startup does with each row it reads. The change's peers join the
|
||||||
// pair's peer set, and its routes are kept as an announcement's would be.
|
// pair's peer set, and its routes are kept as an announcement's would be.
|
||||||
// Unlike an update, Load records no change and no origin change, so
|
// Unlike an update, Load records no change, so TakeChanges does not hand the
|
||||||
// TakeChanges does not hand the pair back to be written again. Loading a pair
|
// pair back to be written again. Loading a pair twice changes nothing.
|
||||||
// twice changes nothing.
|
|
||||||
//
|
//
|
||||||
// The caller makes sure that every peer change names, in its peer set or in
|
// The caller makes sure that every peer change names, in its peer set or in
|
||||||
// its routes, has a number in Params.PeerNumbers.
|
// its routes, has a number in Params.PeerNumbers.
|
||||||
@@ -480,7 +419,6 @@ func (t *Table) removeOrigin(prefix netip.Prefix, o *origin) {
|
|||||||
// r in full when it is one of the two newest routes of o. The peer's earlier
|
// r in full when it is one of the two newest routes of o. The peer's earlier
|
||||||
// route, if o keeps one, is replaced: it is no longer the peer's current route.
|
// route, if o keeps one, is replaced: it is no longer the peer's current route.
|
||||||
// The pair is recorded as changed only when its peer set or kept routes change.
|
// The pair is recorded as changed only when its peer set or kept routes change.
|
||||||
// When r is the origin's first peer, an announced origin change is recorded.
|
|
||||||
func (t *Table) join(prefix netip.Prefix, o *origin, r route) {
|
func (t *Table) join(prefix netip.Prefix, o *origin, r route) {
|
||||||
if slices.ContainsFunc(o.routes, r.equal) {
|
if slices.ContainsFunc(o.routes, r.equal) {
|
||||||
// The peer announced its kept route again.
|
// The peer announced its kept route again.
|
||||||
@@ -490,8 +428,6 @@ func (t *Table) join(prefix netip.Prefix, o *origin, r route) {
|
|||||||
wasWithoutKeptRoute := o.withoutKeptRoute()
|
wasWithoutKeptRoute := o.withoutKeptRoute()
|
||||||
|
|
||||||
joined := !o.peers.Test(r.peer)
|
joined := !o.peers.Test(r.peer)
|
||||||
|
|
||||||
firstPeer := joined && o.peers.None()
|
|
||||||
if joined {
|
if joined {
|
||||||
o.peers.Set(r.peer)
|
o.peers.Set(r.peer)
|
||||||
|
|
||||||
@@ -505,10 +441,6 @@ func (t *Table) join(prefix netip.Prefix, o *origin, r route) {
|
|||||||
if joined || dropped || kept {
|
if joined || dropped || kept {
|
||||||
t.recordChange(prefix, o, wasWithoutKeptRoute)
|
t.recordChange(prefix, o, wasWithoutKeptRoute)
|
||||||
}
|
}
|
||||||
|
|
||||||
if firstPeer {
|
|
||||||
t.recordOriginChange(prefix, o.asn, true, t.prefixPeerCount(prefix))
|
|
||||||
}
|
|
||||||
}
|
}
|
||||||
|
|
||||||
// leave takes peer out of the peer set of o, an origin of prefix, and drops
|
// leave takes peer out of the peer set of o, an origin of prefix, and drops
|
||||||
@@ -567,8 +499,7 @@ func (t *Table) dropKeptRoute(o *origin, peer uint) bool {
|
|||||||
// recordChange records that the pair of prefix and the AS of o changed, and
|
// recordChange records that the pair of prefix and the AS of o changed, and
|
||||||
// brings the count of origins without a kept route up to date.
|
// brings the count of origins without a kept route up to date.
|
||||||
// wasWithoutKeptRoute says whether o was such an origin before the change. An
|
// wasWithoutKeptRoute says whether o was such an origin before the change. An
|
||||||
// origin that no peer carries any more is removed, and a withdrawn origin
|
// origin that no peer carries any more is removed.
|
||||||
// change is recorded.
|
|
||||||
func (t *Table) recordChange(
|
func (t *Table) recordChange(
|
||||||
prefix netip.Prefix, o *origin, wasWithoutKeptRoute bool,
|
prefix netip.Prefix, o *origin, wasWithoutKeptRoute bool,
|
||||||
) {
|
) {
|
||||||
@@ -577,7 +508,6 @@ func (t *Table) recordChange(
|
|||||||
t.countOriginWithoutKeptRoute(o, wasWithoutKeptRoute)
|
t.countOriginWithoutKeptRoute(o, wasWithoutKeptRoute)
|
||||||
|
|
||||||
if o.peers.None() {
|
if o.peers.None() {
|
||||||
t.recordOriginChange(prefix, o.asn, false, t.prefixPeerCount(prefix))
|
|
||||||
t.removeOrigin(prefix, o)
|
t.removeOrigin(prefix, o)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -1,69 +0,0 @@
|
|||||||
package routetable
|
|
||||||
|
|
||||||
import (
|
|
||||||
"net/netip"
|
|
||||||
"runtime"
|
|
||||||
"testing"
|
|
||||||
"time"
|
|
||||||
)
|
|
||||||
|
|
||||||
// TestPeersDownLetsReadsIn reads the table's counts again and again while a
|
|
||||||
// peer that carries more than two batches of prefixes goes down. PeersDown lets
|
|
||||||
// the lock go after every walkBatch prefixes, so a read must get in while the
|
|
||||||
// peer still carries some of its prefixes but no longer all.
|
|
||||||
func TestPeersDownLetsReadsIn(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
const (
|
|
||||||
asn = 1000
|
|
||||||
peer = "192.0.2.1"
|
|
||||||
// prefixCount is more prefixes than two of walkBatch.
|
|
||||||
prefixCount = 2*walkBatch + 1
|
|
||||||
)
|
|
||||||
|
|
||||||
routeTime := time.Date(2026, 1, 2, 3, 4, 5, 0, time.UTC)
|
|
||||||
table := New(Params{})
|
|
||||||
addr := netip.MustParseAddr("1.0.0.0")
|
|
||||||
|
|
||||||
for range prefixCount {
|
|
||||||
prefix := netip.PrefixFrom(addr, addr.BitLen())
|
|
||||||
table.Announce(prefix, asn, Route{Peer: peer, Time: routeTime})
|
|
||||||
|
|
||||||
addr = addr.Next()
|
|
||||||
}
|
|
||||||
|
|
||||||
// Hold a read lock until PeersDown waits for the lock, so that the first
|
|
||||||
// read below starts once PeersDown holds it. TryRLock fails once a writer
|
|
||||||
// waits.
|
|
||||||
table.mu.RLock()
|
|
||||||
|
|
||||||
peerDownDone := make(chan struct{})
|
|
||||||
|
|
||||||
go func() {
|
|
||||||
defer close(peerDownDone)
|
|
||||||
|
|
||||||
table.PeersDown([]string{peer})
|
|
||||||
}()
|
|
||||||
|
|
||||||
for table.mu.TryRLock() {
|
|
||||||
table.mu.RUnlock()
|
|
||||||
runtime.Gosched()
|
|
||||||
}
|
|
||||||
|
|
||||||
table.mu.RUnlock()
|
|
||||||
|
|
||||||
for {
|
|
||||||
routes := table.Counts().IPv4Routes
|
|
||||||
if routes > 0 && routes < prefixCount {
|
|
||||||
break
|
|
||||||
}
|
|
||||||
|
|
||||||
select {
|
|
||||||
case <-peerDownDone:
|
|
||||||
t.Fatal("no read got in while PeersDown ran")
|
|
||||||
default:
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
<-peerDownDone
|
|
||||||
}
|
|
||||||
@@ -69,10 +69,10 @@ func withdraw(peer string) step {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// peersDown is the step of the sessions of peers going down.
|
// peerDown is the step of peer's session going down.
|
||||||
func peersDown(peers ...string) step {
|
func peerDown(peer string) step {
|
||||||
return func(table *routetable.Table, _ netip.Prefix) {
|
return func(table *routetable.Table, _ netip.Prefix) {
|
||||||
table.PeersDown(peers)
|
table.PeerDown(peer)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -272,28 +272,21 @@ func TestWithdrawalsAndPeerDowns(t *testing.T) {
|
|||||||
{
|
{
|
||||||
name: "a peer going down leaves the prefix",
|
name: "a peer going down leaves the prefix",
|
||||||
steps: []step{announce(peerA, asA, 10), announce(peerB, asA, 20),
|
steps: []step{announce(peerA, asA, 10), announce(peerB, asA, 20),
|
||||||
peersDown(peerB)},
|
peerDown(peerB)},
|
||||||
want: []routetable.Origin{{ASN: asA, Peers: []string{peerA},
|
want: []routetable.Origin{{ASN: asA, Peers: []string{peerA},
|
||||||
Routes: []routetable.Route{routeOf(peerA, asA, 10)}}},
|
Routes: []routetable.Route{routeOf(peerA, asA, 10)}}},
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
name: "a peer going down after it withdrew changes nothing",
|
name: "a peer going down after it withdrew changes nothing",
|
||||||
steps: []step{announce(peerA, asA, 10), announce(peerB, asA, 20),
|
steps: []step{announce(peerA, asA, 10), announce(peerB, asA, 20),
|
||||||
withdraw(peerB), peersDown(peerB)},
|
withdraw(peerB), peerDown(peerB)},
|
||||||
want: []routetable.Origin{{ASN: asA, Peers: []string{peerA},
|
want: []routetable.Origin{{ASN: asA, Peers: []string{peerA},
|
||||||
Routes: []routetable.Route{routeOf(peerA, asA, 10)}}},
|
Routes: []routetable.Route{routeOf(peerA, asA, 10)}}},
|
||||||
},
|
},
|
||||||
{
|
|
||||||
name: "peers going down together each leave the prefix",
|
|
||||||
steps: []step{announce(peerA, asA, 10), announce(peerB, asB, 20),
|
|
||||||
announce(peerC, asB, 30), peersDown(peerA, peerB)},
|
|
||||||
want: []routetable.Origin{{ASN: asB, Peers: []string{peerC},
|
|
||||||
Routes: []routetable.Route{routeOf(peerC, asB, 30)}}},
|
|
||||||
},
|
|
||||||
})
|
})
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestPeersDownLeavesEveryPrefix(t *testing.T) {
|
func TestPeerDownLeavesEveryPrefix(t *testing.T) {
|
||||||
t.Parallel()
|
t.Parallel()
|
||||||
|
|
||||||
table := routetable.New(routetable.Params{})
|
table := routetable.New(routetable.Params{})
|
||||||
@@ -310,7 +303,7 @@ func TestPeersDownLeavesEveryPrefix(t *testing.T) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
table.Announce(prefixes[0], asB, routeOf(peerB, asB, 20))
|
table.Announce(prefixes[0], asB, routeOf(peerB, asB, 20))
|
||||||
table.PeersDown([]string{peerA})
|
table.PeerDown(peerA)
|
||||||
|
|
||||||
want := []routetable.Origin{{ASN: asB, Peers: []string{peerB},
|
want := []routetable.Origin{{ASN: asB, Peers: []string{peerB},
|
||||||
Routes: []routetable.Route{routeOf(peerB, asB, 20)}}}
|
Routes: []routetable.Route{routeOf(peerB, asB, 20)}}}
|
||||||
@@ -340,32 +333,6 @@ func TestPeersDownLeavesEveryPrefix(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// TestPeersDownTakesOutPeersOfDifferentPrefixes takes peerA and peerB out in
|
|
||||||
// one PeersDown. peerA carries only prefixV4, and peerB only prefixV6. So
|
|
||||||
// peerA's leaving removes prefixV4 before PeersDown looks for peerB there. Both
|
|
||||||
// prefixes must then be gone.
|
|
||||||
func TestPeersDownTakesOutPeersOfDifferentPrefixes(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
table := routetable.New(routetable.Params{})
|
|
||||||
prefixes := []netip.Prefix{
|
|
||||||
netip.MustParsePrefix(prefixV4),
|
|
||||||
netip.MustParsePrefix(prefixV6),
|
|
||||||
}
|
|
||||||
|
|
||||||
table.Announce(prefixes[0], asA, routeOf(peerA, asA, 10))
|
|
||||||
table.Announce(prefixes[1], asB, routeOf(peerB, asB, 20))
|
|
||||||
table.PeersDown([]string{peerA, peerB})
|
|
||||||
|
|
||||||
for _, prefix := range prefixes {
|
|
||||||
got := table.Origins(prefix)
|
|
||||||
if got != nil {
|
|
||||||
t.Errorf("Origins(%s) = %+v after its only peer went down, want nil",
|
|
||||||
prefix, got)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestCountsOriginsWithoutKeptRoute(t *testing.T) {
|
func TestCountsOriginsWithoutKeptRoute(t *testing.T) {
|
||||||
t.Parallel()
|
t.Parallel()
|
||||||
|
|
||||||
@@ -391,7 +358,7 @@ func TestCountsOriginsWithoutKeptRoute(t *testing.T) {
|
|||||||
{announce(peerC, asA, 30), 0},
|
{announce(peerC, asA, 30), 0},
|
||||||
{withdraw(peerB), 0},
|
{withdraw(peerB), 0},
|
||||||
{withdraw(peerC), 1},
|
{withdraw(peerC), 1},
|
||||||
{peersDown(peerA), 0},
|
{peerDown(peerA), 0},
|
||||||
}
|
}
|
||||||
|
|
||||||
for i, s := range steps {
|
for i, s := range steps {
|
||||||
@@ -497,7 +464,7 @@ func TestTakeChangesHoldsOnlyPairsThatChanged(t *testing.T) {
|
|||||||
checkChanges(t, table, []change{}, map[string]uint{})
|
checkChanges(t, table, []change{}, map[string]uint{})
|
||||||
|
|
||||||
// peerA going down changes v4, the one prefix it carries.
|
// peerA going down changes v4, the one prefix it carries.
|
||||||
table.PeersDown([]string{peerA})
|
table.PeerDown(peerA)
|
||||||
checkChanges(t, table, []change{
|
checkChanges(t, table, []change{
|
||||||
{Prefix: prefixV4, OriginASN: asA, Peers: []uint{1, 2},
|
{Prefix: prefixV4, OriginASN: asA, Peers: []uint{1, 2},
|
||||||
Routes: []routetable.Route{
|
Routes: []routetable.Route{
|
||||||
@@ -506,7 +473,7 @@ func TestTakeChangesHoldsOnlyPairsThatChanged(t *testing.T) {
|
|||||||
}, map[string]uint{})
|
}, map[string]uint{})
|
||||||
|
|
||||||
// peerA, which carries nothing now, goes down again.
|
// peerA, which carries nothing now, goes down again.
|
||||||
table.PeersDown([]string{peerA})
|
table.PeerDown(peerA)
|
||||||
checkChanges(t, table, []change{}, map[string]uint{})
|
checkChanges(t, table, []change{}, map[string]uint{})
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -650,7 +617,7 @@ func TestConcurrentUse(t *testing.T) {
|
|||||||
func(round int) { table.Announce(prefix, asA, routeOf(peerA, asA, round)) },
|
func(round int) { table.Announce(prefix, asA, routeOf(peerA, asA, round)) },
|
||||||
func(round int) { table.Announce(prefix, asB, routeOf(peerB, asB, round)) },
|
func(round int) { table.Announce(prefix, asB, routeOf(peerB, asB, round)) },
|
||||||
func(int) { table.Withdraw(prefix, peerA) },
|
func(int) { table.Withdraw(prefix, peerA) },
|
||||||
func(int) { table.PeersDown([]string{peerB}) },
|
func(int) { table.PeerDown(peerB) },
|
||||||
func(int) { table.PutBackChanges(table.TakeChanges(math.MaxInt)) },
|
func(int) { table.PutBackChanges(table.TakeChanges(math.MaxInt)) },
|
||||||
func(int) { table.ChangeCount() },
|
func(int) { table.ChangeCount() },
|
||||||
func(int) { table.Lookup(prefix.Addr()) },
|
func(int) { table.Lookup(prefix.Addr()) },
|
||||||
@@ -658,7 +625,6 @@ func TestConcurrentUse(t *testing.T) {
|
|||||||
func(int) { table.OriginatedPrefixes(asA) },
|
func(int) { table.OriginatedPrefixes(asA) },
|
||||||
func(int) { table.SamplePrefixes(ipVersion, prefix.Bits(), sampleSize) },
|
func(int) { table.SamplePrefixes(ipVersion, prefix.Bits(), sampleSize) },
|
||||||
func(int) { table.Counts() },
|
func(int) { table.Counts() },
|
||||||
func(int) { table.Overview(sampleSize) },
|
|
||||||
}
|
}
|
||||||
|
|
||||||
var wg sync.WaitGroup
|
var wg sync.WaitGroup
|
||||||
|
|||||||
+17
-116
@@ -9,21 +9,17 @@ package routewatch
|
|||||||
|
|
||||||
import (
|
import (
|
||||||
"context"
|
"context"
|
||||||
"database/sql"
|
|
||||||
"errors"
|
"errors"
|
||||||
"fmt"
|
"fmt"
|
||||||
"path/filepath"
|
"path/filepath"
|
||||||
"sync"
|
"sync"
|
||||||
"time"
|
"time"
|
||||||
|
|
||||||
"git.eeqj.de/sneak/routewatch/internal/cachedb"
|
"git.eeqj.de/sneak/routewatch/internal/blocklist"
|
||||||
"git.eeqj.de/sneak/routewatch/internal/config"
|
"git.eeqj.de/sneak/routewatch/internal/config"
|
||||||
"git.eeqj.de/sneak/routewatch/internal/database"
|
"git.eeqj.de/sneak/routewatch/internal/database"
|
||||||
"git.eeqj.de/sneak/routewatch/internal/geo"
|
|
||||||
"git.eeqj.de/sneak/routewatch/internal/iplist"
|
|
||||||
"git.eeqj.de/sneak/routewatch/internal/logger"
|
"git.eeqj.de/sneak/routewatch/internal/logger"
|
||||||
"git.eeqj.de/sneak/routewatch/internal/metrics"
|
"git.eeqj.de/sneak/routewatch/internal/metrics"
|
||||||
"git.eeqj.de/sneak/routewatch/internal/ptr"
|
|
||||||
"git.eeqj.de/sneak/routewatch/internal/routetable"
|
"git.eeqj.de/sneak/routewatch/internal/routetable"
|
||||||
"git.eeqj.de/sneak/routewatch/internal/server"
|
"git.eeqj.de/sneak/routewatch/internal/server"
|
||||||
"git.eeqj.de/sneak/routewatch/internal/streamer"
|
"git.eeqj.de/sneak/routewatch/internal/streamer"
|
||||||
@@ -65,15 +61,12 @@ type RouteWatch struct {
|
|||||||
peeringHandler *PeeringHandler
|
peeringHandler *PeeringHandler
|
||||||
asnFetcher *ASNFetcher
|
asnFetcher *ASNFetcher
|
||||||
dbMaintainer *DBMaintainer
|
dbMaintainer *DBMaintainer
|
||||||
ipLists *iplist.IPLists
|
blocklists *blocklist.Blocklists
|
||||||
cacheDB *sql.DB
|
|
||||||
ptrCache *ptr.Cache
|
|
||||||
geoLocator *geo.Locator
|
|
||||||
}
|
}
|
||||||
|
|
||||||
// ipListsDirName is the directory in the state directory that holds the IP
|
// blocklistsDirName is the directory in the state directory that holds the
|
||||||
// list copies.
|
// blocklist copies.
|
||||||
const ipListsDirName = "iplists"
|
const blocklistsDirName = "blocklists"
|
||||||
|
|
||||||
// New creates a new RouteWatch instance
|
// New creates a new RouteWatch instance
|
||||||
func New(deps Dependencies) *RouteWatch {
|
func New(deps Dependencies) *RouteWatch {
|
||||||
@@ -145,27 +138,17 @@ func (rw *RouteWatch) Run(ctx context.Context) error {
|
|||||||
rw.asnFetcher.Start(ctx)
|
rw.asnFetcher.Start(ctx)
|
||||||
rw.server.SetASNFetcher(rw.asnFetcher)
|
rw.server.SetASNFetcher(rw.asnFetcher)
|
||||||
|
|
||||||
// Load the IP list copies from disk and start downloading the lists.
|
// Load the blocklist copies from disk and start downloading the lists.
|
||||||
// The HTTP handlers read the server's IP lists without a lock, so they
|
// The HTTP handlers read the server's blocklists without a lock, so they
|
||||||
// are set before the handlers run.
|
// are set before the handlers run.
|
||||||
ipLists := iplist.New(iplist.Params{
|
blocklists := blocklist.New(blocklist.Params{
|
||||||
Dir: filepath.Join(rw.config.StateDir, ipListsDirName),
|
Dir: filepath.Join(rw.config.StateDir, blocklistsDirName),
|
||||||
Lists: iplist.Lists(),
|
Lists: blocklist.Lists(),
|
||||||
Logger: rw.logger.Logger,
|
Logger: rw.logger.Logger,
|
||||||
})
|
})
|
||||||
ipLists.Start(ctx)
|
blocklists.Start(ctx)
|
||||||
rw.ipLists = ipLists
|
rw.blocklists = blocklists
|
||||||
rw.server.SetIPLists(ipLists)
|
rw.server.SetBlocklists(blocklists)
|
||||||
|
|
||||||
err = rw.startPTRCache(ctx)
|
|
||||||
if err != nil {
|
|
||||||
return err
|
|
||||||
}
|
|
||||||
|
|
||||||
err = rw.startGeoLocator(ctx)
|
|
||||||
if err != nil {
|
|
||||||
return err
|
|
||||||
}
|
|
||||||
|
|
||||||
// Start HTTP server
|
// Start HTTP server
|
||||||
err = rw.server.Start(ctx)
|
err = rw.server.Start(ctx)
|
||||||
@@ -185,9 +168,8 @@ func (rw *RouteWatch) Run(ctx context.Context) error {
|
|||||||
|
|
||||||
// Shutdown stops all services in order: the streamer, the handlers, which
|
// Shutdown stops all services in order: the streamer, the handlers, which
|
||||||
// write what they hold, the WHOIS fetcher, the database maintainer, the
|
// write what they hold, the WHOIS fetcher, the database maintainer, the
|
||||||
// IP list downloads, the HTTP server, the geolocation cache and the PTR cache,
|
// blocklist downloads and the HTTP server, and last the database, which it
|
||||||
// and last the database, which it checkpoints and closes. Each stage waits at
|
// checkpoints and closes. Each stage waits at most until ctx ends, so
|
||||||
// most until ctx ends, so
|
|
||||||
// Shutdown returns soon after: a handler logs what it could not write by
|
// Shutdown returns soon after: a handler logs what it could not write by
|
||||||
// then, and the HTTP server closes the connections of requests still
|
// then, and the HTTP server closes the connections of requests still
|
||||||
// running.
|
// running.
|
||||||
@@ -237,8 +219,8 @@ func (rw *RouteWatch) Shutdown(ctx context.Context) {
|
|||||||
rw.dbMaintainer.Stop(ctx)
|
rw.dbMaintainer.Stop(ctx)
|
||||||
}
|
}
|
||||||
|
|
||||||
if rw.ipLists != nil {
|
if rw.blocklists != nil {
|
||||||
rw.ipLists.Stop(ctx)
|
rw.blocklists.Stop(ctx)
|
||||||
}
|
}
|
||||||
|
|
||||||
err := rw.server.Stop(ctx)
|
err := rw.server.Stop(ctx)
|
||||||
@@ -247,12 +229,6 @@ func (rw *RouteWatch) Shutdown(ctx context.Context) {
|
|||||||
"error", err)
|
"error", err)
|
||||||
}
|
}
|
||||||
|
|
||||||
if rw.geoLocator != nil {
|
|
||||||
rw.geoLocator.Stop(ctx)
|
|
||||||
}
|
|
||||||
|
|
||||||
rw.stopPTRCache(ctx)
|
|
||||||
|
|
||||||
rw.closeDatabase(ctx)
|
rw.closeDatabase(ctx)
|
||||||
|
|
||||||
// Log final metrics
|
// Log final metrics
|
||||||
@@ -266,81 +242,6 @@ func (rw *RouteWatch) Shutdown(ctx context.Context) {
|
|||||||
)
|
)
|
||||||
}
|
}
|
||||||
|
|
||||||
// startPTRCache opens cache.db, loads the PTR cache from it and gives the
|
|
||||||
// server the cache. The HTTP handlers read it without a lock, so it is set
|
|
||||||
// before they run.
|
|
||||||
func (rw *RouteWatch) startPTRCache(ctx context.Context) error {
|
|
||||||
cacheDB, err := cachedb.Open(ctx, rw.config.StateDir)
|
|
||||||
if err != nil {
|
|
||||||
return err
|
|
||||||
}
|
|
||||||
|
|
||||||
ptrCache := ptr.New(ptr.Params{
|
|
||||||
DB: cacheDB,
|
|
||||||
Resolvers: ptr.Resolvers(),
|
|
||||||
Logger: rw.logger.Logger,
|
|
||||||
})
|
|
||||||
|
|
||||||
err = ptrCache.Start(ctx)
|
|
||||||
if err != nil {
|
|
||||||
_ = cacheDB.Close()
|
|
||||||
|
|
||||||
return err
|
|
||||||
}
|
|
||||||
|
|
||||||
rw.cacheDB = cacheDB
|
|
||||||
rw.ptrCache = ptrCache
|
|
||||||
rw.server.SetPTRResolver(ptrCache)
|
|
||||||
|
|
||||||
return nil
|
|
||||||
}
|
|
||||||
|
|
||||||
// startGeoLocator loads the geolocation cache from cache.db, which
|
|
||||||
// startPTRCache has opened, and gives the server the locator. The HTTP handlers
|
|
||||||
// read it without a lock, so it is set before they run.
|
|
||||||
func (rw *RouteWatch) startGeoLocator(ctx context.Context) error {
|
|
||||||
locator, err := geo.New(geo.Params{
|
|
||||||
DB: rw.cacheDB,
|
|
||||||
Dir: rw.config.StateDir,
|
|
||||||
GeoJSURL: geo.GeoJSEndpoint,
|
|
||||||
DatabaseURLFormat: geo.DatabaseDownloadFormat,
|
|
||||||
Logger: rw.logger.Logger,
|
|
||||||
})
|
|
||||||
if err != nil {
|
|
||||||
return err
|
|
||||||
}
|
|
||||||
|
|
||||||
err = locator.Start(ctx)
|
|
||||||
if err != nil {
|
|
||||||
return err
|
|
||||||
}
|
|
||||||
|
|
||||||
rw.geoLocator = locator
|
|
||||||
rw.server.SetGeoLocator(locator)
|
|
||||||
|
|
||||||
return nil
|
|
||||||
}
|
|
||||||
|
|
||||||
// stopPTRCache stops the PTR cache, which writes the answers cache.db does not
|
|
||||||
// have yet, and closes cache.db, unless ctx has ended and a write may still be
|
|
||||||
// running.
|
|
||||||
func (rw *RouteWatch) stopPTRCache(ctx context.Context) {
|
|
||||||
if rw.ptrCache == nil {
|
|
||||||
return
|
|
||||||
}
|
|
||||||
|
|
||||||
rw.ptrCache.Stop(ctx)
|
|
||||||
|
|
||||||
if ctx.Err() != nil {
|
|
||||||
return
|
|
||||||
}
|
|
||||||
|
|
||||||
err := rw.cacheDB.Close()
|
|
||||||
if err != nil {
|
|
||||||
rw.logger.Error("Failed to close the cache database", "error", err)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// closeDatabase checkpoints and closes the database, unless ctx has ended: a
|
// closeDatabase checkpoints and closes the database, unless ctx has ended: a
|
||||||
// write may then still be running, and both would wait for it. SQLite rolls
|
// write may then still be running, and both would wait for it. SQLite rolls
|
||||||
// back a write cut off by the exit when the database next opens.
|
// back a write cut off by the exit when the database next opens.
|
||||||
|
|||||||
@@ -8,8 +8,6 @@ import (
|
|||||||
"net/http"
|
"net/http"
|
||||||
"net/http/httptest"
|
"net/http/httptest"
|
||||||
"net/netip"
|
"net/netip"
|
||||||
"os"
|
|
||||||
"path/filepath"
|
|
||||||
"slices"
|
"slices"
|
||||||
"strconv"
|
"strconv"
|
||||||
"strings"
|
"strings"
|
||||||
@@ -225,35 +223,6 @@ func (m *mockStore) GetAS(_ context.Context, asn int) (*database.ASN, bool, erro
|
|||||||
return &database.ASN{ASN: asn}, false, nil
|
return &database.ASN{ASN: asn}, false, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
// GetLastSeenASes returns copies of the stored ASes last seen most recently
|
|
||||||
// and longest ago, or neither when none is stored.
|
|
||||||
func (m *mockStore) GetLastSeenASes(
|
|
||||||
_ context.Context,
|
|
||||||
) (*database.ASN, *database.ASN, error) {
|
|
||||||
m.mu.Lock()
|
|
||||||
defer m.mu.Unlock()
|
|
||||||
|
|
||||||
var newest, oldest *database.ASN
|
|
||||||
|
|
||||||
for _, asn := range m.ASNs {
|
|
||||||
if newest == nil || asn.LastSeen.After(newest.LastSeen) {
|
|
||||||
newest = asn
|
|
||||||
}
|
|
||||||
|
|
||||||
if oldest == nil || asn.LastSeen.Before(oldest.LastSeen) {
|
|
||||||
oldest = asn
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
if newest == nil {
|
|
||||||
return nil, nil, nil
|
|
||||||
}
|
|
||||||
|
|
||||||
newestCopy, oldestCopy := *newest, *oldest
|
|
||||||
|
|
||||||
return &newestCopy, &oldestCopy, nil
|
|
||||||
}
|
|
||||||
|
|
||||||
// GetASWHOIS returns the stored AS, or only its number when none is stored.
|
// GetASWHOIS returns the stored AS, or only its number when none is stored.
|
||||||
func (m *mockStore) GetASWHOIS(ctx context.Context, asn int) (*database.ASN, error) {
|
func (m *mockStore) GetASWHOIS(ctx context.Context, asn int) (*database.ASN, error) {
|
||||||
asnInfo, _, err := m.GetAS(ctx, asn)
|
asnInfo, _, err := m.GetAS(ctx, asn)
|
||||||
@@ -332,23 +301,6 @@ func (m *mockStore) ReadRouteTable(_ context.Context) (*routetable.Table, error)
|
|||||||
return routetable.New(routetable.Params{}), nil
|
return routetable.New(routetable.Params{}), nil
|
||||||
}
|
}
|
||||||
|
|
||||||
// WriteOriginChanges records nothing: the mock keeps no origin changes.
|
|
||||||
func (m *mockStore) WriteOriginChanges(_ []routetable.OriginChange) error {
|
|
||||||
return nil
|
|
||||||
}
|
|
||||||
|
|
||||||
// GetOriginChanges returns no origin changes.
|
|
||||||
func (m *mockStore) GetOriginChanges(
|
|
||||||
_ context.Context, _ netip.Prefix, _ int,
|
|
||||||
) ([]routetable.OriginChange, error) {
|
|
||||||
return nil, nil
|
|
||||||
}
|
|
||||||
|
|
||||||
// PruneOriginChanges does nothing.
|
|
||||||
func (m *mockStore) PruneOriginChanges(_ context.Context) error {
|
|
||||||
return nil
|
|
||||||
}
|
|
||||||
|
|
||||||
// GetOrCreateASNBatch mock implementation
|
// GetOrCreateASNBatch mock implementation
|
||||||
func (m *mockStore) GetOrCreateASNBatch(asns map[int]database.SeenTimes) error {
|
func (m *mockStore) GetOrCreateASNBatch(asns map[int]database.SeenTimes) error {
|
||||||
m.mu.Lock()
|
m.mu.Lock()
|
||||||
@@ -420,7 +372,7 @@ func TestRouteWatchLiveFeed(t *testing.T) {
|
|||||||
// Create streamer
|
// Create streamer
|
||||||
s := streamer.New(logger, metricsTracker)
|
s := streamer.New(logger, metricsTracker)
|
||||||
|
|
||||||
// The state directory holds the IP list copies Run downloads.
|
// The state directory holds the blocklist copies Run downloads.
|
||||||
cfg := &config.Config{
|
cfg := &config.Config{
|
||||||
StateDir: t.TempDir(),
|
StateDir: t.TempDir(),
|
||||||
MaxRuntime: 5 * time.Second,
|
MaxRuntime: 5 * time.Second,
|
||||||
@@ -528,6 +480,8 @@ func announcementFeed(prefixes []string) string {
|
|||||||
func TestShutdownWritesTheUpdatesQueuedInTheStreamer(t *testing.T) {
|
func TestShutdownWritesTheUpdatesQueuedInTheStreamer(t *testing.T) {
|
||||||
t.Parallel()
|
t.Parallel()
|
||||||
|
|
||||||
|
const maxWait = 10 * time.Second
|
||||||
|
|
||||||
prefixes := []string{"192.0.2.0/24", "198.51.100.0/24", "203.0.113.0/24"}
|
prefixes := []string{"192.0.2.0/24", "198.51.100.0/24", "203.0.113.0/24"}
|
||||||
feed := announcementFeed(prefixes)
|
feed := announcementFeed(prefixes)
|
||||||
|
|
||||||
@@ -573,9 +527,9 @@ func TestShutdownWritesTheUpdatesQueuedInTheStreamer(t *testing.T) {
|
|||||||
for range cap(linesSeen) {
|
for range cap(linesSeen) {
|
||||||
select {
|
select {
|
||||||
case <-linesSeen:
|
case <-linesSeen:
|
||||||
case <-time.After(stepTimeout):
|
case <-time.After(maxWait):
|
||||||
t.Fatalf("the streamer did not read %d lines within %s",
|
t.Fatalf("the streamer did not read %d lines within %s",
|
||||||
cap(linesSeen), stepTimeout)
|
cap(linesSeen), maxWait)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -588,8 +542,8 @@ func TestShutdownWritesTheUpdatesQueuedInTheStreamer(t *testing.T) {
|
|||||||
|
|
||||||
select {
|
select {
|
||||||
case <-shutDown:
|
case <-shutDown:
|
||||||
case <-time.After(stepTimeout):
|
case <-time.After(maxWait):
|
||||||
t.Fatalf("Shutdown did not return within %s", stepTimeout)
|
t.Fatalf("Shutdown did not return within %s", maxWait)
|
||||||
}
|
}
|
||||||
|
|
||||||
store.mu.Lock()
|
store.mu.Lock()
|
||||||
@@ -626,16 +580,21 @@ func freePort(t *testing.T) int {
|
|||||||
return addr.Port
|
return addr.Port
|
||||||
}
|
}
|
||||||
|
|
||||||
// TestShutdownDoesNotWaitForABlockedWrite starts the HTTP server and has the
|
// TestShutdownReturnsByItsDeadlineWhileAWriteIsBlocked starts the HTTP server
|
||||||
// AS handler start a write that the store holds until the test ends, past
|
// and has the AS handler start a write that the store holds until the test
|
||||||
// Shutdown's deadline. Shutdown must return without waiting for the write, with
|
// ends, past Shutdown's deadline. Shutdown must return within its deadline
|
||||||
// the HTTP server stopped, and must neither checkpoint nor close the database:
|
// plus a small margin, with the HTTP server stopped, and must neither
|
||||||
// the real one would wait for the write.
|
// checkpoint nor close the database: the real one would wait for the write.
|
||||||
// Before the fix Shutdown waited for the write.
|
// Before the fix Shutdown waited for the write.
|
||||||
func TestShutdownDoesNotWaitForABlockedWrite(t *testing.T) {
|
func TestShutdownReturnsByItsDeadlineWhileAWriteIsBlocked(t *testing.T) {
|
||||||
t.Parallel()
|
t.Parallel()
|
||||||
|
|
||||||
const deadline = 100 * time.Millisecond
|
const (
|
||||||
|
deadline = 100 * time.Millisecond
|
||||||
|
// margin is how long Shutdown may take after its deadline, to stop
|
||||||
|
// the HTTP server.
|
||||||
|
margin = 2 * time.Second
|
||||||
|
)
|
||||||
|
|
||||||
blocking := newBlockingStore()
|
blocking := newBlockingStore()
|
||||||
defer blocking.releaseWrites()
|
defer blocking.releaseWrites()
|
||||||
@@ -646,15 +605,12 @@ func TestShutdownDoesNotWaitForABlockedWrite(t *testing.T) {
|
|||||||
cfg := &config.Config{Port: port}
|
cfg := &config.Config{Port: port}
|
||||||
s := streamer.New(log, metrics.New())
|
s := streamer.New(log, metrics.New())
|
||||||
|
|
||||||
// A started server reads its routing table in the background, so it
|
|
||||||
// needs one.
|
|
||||||
rw := New(Dependencies{
|
rw := New(Dependencies{
|
||||||
DB: store,
|
DB: store,
|
||||||
Streamer: s,
|
Streamer: s,
|
||||||
Server: server.New(store, routetable.New(routetable.Params{}), s, log,
|
Server: server.New(store, nil, s, log, cfg),
|
||||||
cfg),
|
Logger: log,
|
||||||
Logger: log,
|
Config: cfg,
|
||||||
Config: cfg,
|
|
||||||
})
|
})
|
||||||
|
|
||||||
err := rw.server.Start(t.Context())
|
err := rw.server.Start(t.Context())
|
||||||
@@ -663,8 +619,9 @@ func TestShutdownDoesNotWaitForABlockedWrite(t *testing.T) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
rw.dbHandler = NewASHandler(store, log)
|
rw.dbHandler = NewASHandler(store, log)
|
||||||
|
fill := ristypes.ASPath{Sequence: slices.Repeat([]int{testASNA}, asnBatchSize)}
|
||||||
startBlockedWrite(t, blocking, rw.dbHandler,
|
startBlockedWrite(t, blocking, rw.dbHandler,
|
||||||
[]*ristypes.RISMessage{{Path: pathOfASes(asnBatchSize)}})
|
[]*ristypes.RISMessage{{Path: fill}})
|
||||||
|
|
||||||
ctx, cancel := context.WithTimeout(t.Context(), deadline)
|
ctx, cancel := context.WithTimeout(t.Context(), deadline)
|
||||||
defer cancel()
|
defer cancel()
|
||||||
@@ -678,9 +635,9 @@ func TestShutdownDoesNotWaitForABlockedWrite(t *testing.T) {
|
|||||||
|
|
||||||
select {
|
select {
|
||||||
case <-shutDown:
|
case <-shutDown:
|
||||||
case <-time.After(deadline + stepTimeout):
|
case <-time.After(deadline + margin):
|
||||||
t.Fatalf("Shutdown did not return within %s of its deadline while "+
|
t.Fatalf("Shutdown did not return within %s of its deadline while "+
|
||||||
"a write was blocked", stepTimeout)
|
"a write was blocked", margin)
|
||||||
}
|
}
|
||||||
|
|
||||||
var dialer net.Dialer
|
var dialer net.Dialer
|
||||||
@@ -755,6 +712,8 @@ func (s *closeOrderStore) record(call string) {
|
|||||||
func TestShutdownCheckpointsAndClosesTheDatabaseLast(t *testing.T) {
|
func TestShutdownCheckpointsAndClosesTheDatabaseLast(t *testing.T) {
|
||||||
t.Parallel()
|
t.Parallel()
|
||||||
|
|
||||||
|
const deadline = 10 * time.Second
|
||||||
|
|
||||||
store := &closeOrderStore{Store: newMockStore()}
|
store := &closeOrderStore{Store: newMockStore()}
|
||||||
log := logger.New()
|
log := logger.New()
|
||||||
cfg := &config.Config{}
|
cfg := &config.Config{}
|
||||||
@@ -772,7 +731,7 @@ func TestShutdownCheckpointsAndClosesTheDatabaseLast(t *testing.T) {
|
|||||||
Path: ristypes.ASPath{Sequence: []int{testASNA}},
|
Path: ristypes.ASPath{Sequence: []int{testASNA}},
|
||||||
})
|
})
|
||||||
|
|
||||||
ctx, cancel := context.WithTimeout(t.Context(), stepTimeout)
|
ctx, cancel := context.WithTimeout(t.Context(), deadline)
|
||||||
defer cancel()
|
defer cancel()
|
||||||
|
|
||||||
rw.Shutdown(ctx)
|
rw.Shutdown(ctx)
|
||||||
@@ -785,74 +744,3 @@ func TestShutdownCheckpointsAndClosesTheDatabaseLast(t *testing.T) {
|
|||||||
t.Errorf("Shutdown made the calls %q, want %q", store.calls, want)
|
t.Errorf("Shutdown made the calls %q, want %q", store.calls, want)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// TestPTRCacheAnswersTheIPLookup starts the PTR cache and the HTTP server as
|
|
||||||
// Run does, and looks up a reserved address, which asks no resolver. Without
|
|
||||||
// the PTR cache the server has no PTR resolver, and every IP lookup answers
|
|
||||||
// 500.
|
|
||||||
func TestPTRCacheAnswersTheIPLookup(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
store := newMockStore()
|
|
||||||
log := logger.New()
|
|
||||||
port := freePort(t)
|
|
||||||
cfg := &config.Config{Port: port, StateDir: t.TempDir()}
|
|
||||||
s := streamer.New(log, metrics.New())
|
|
||||||
|
|
||||||
// A started server reads its routing table in the background, so it
|
|
||||||
// needs one.
|
|
||||||
rw := New(Dependencies{
|
|
||||||
DB: store,
|
|
||||||
Streamer: s,
|
|
||||||
Server: server.New(store, routetable.New(routetable.Params{}), s, log,
|
|
||||||
cfg),
|
|
||||||
Logger: log,
|
|
||||||
Config: cfg,
|
|
||||||
})
|
|
||||||
|
|
||||||
err := rw.startPTRCache(t.Context())
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("failed to start the PTR cache: %v", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
defer rw.stopPTRCache(t.Context())
|
|
||||||
|
|
||||||
err = rw.server.Start(t.Context())
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("failed to start the HTTP server: %v", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
defer func() { _ = rw.server.Stop(t.Context()) }()
|
|
||||||
|
|
||||||
req, err := http.NewRequestWithContext(t.Context(), http.MethodGet,
|
|
||||||
"http://"+net.JoinHostPort("127.0.0.1", strconv.Itoa(port))+
|
|
||||||
"/api/v1/ip/10.0.0.1", nil)
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("failed to build the request: %v", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
resp, err := http.DefaultClient.Do(req)
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("the IP lookup failed: %v", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
defer func() { _ = resp.Body.Close() }()
|
|
||||||
|
|
||||||
body, err := io.ReadAll(resp.Body)
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("failed to read the answer: %v", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
if resp.StatusCode != http.StatusOK {
|
|
||||||
t.Fatalf("status %d, want %d; body %s", resp.StatusCode, http.StatusOK, body)
|
|
||||||
}
|
|
||||||
|
|
||||||
if !strings.Contains(string(body), `"ptr":[]`) {
|
|
||||||
t.Errorf("answer %s, want an empty ptr", body)
|
|
||||||
}
|
|
||||||
|
|
||||||
_, err = os.Stat(filepath.Join(cfg.StateDir, "cache.db"))
|
|
||||||
if err != nil {
|
|
||||||
t.Errorf("cache.db is not in the state directory: %v", err)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
@@ -18,34 +18,39 @@ const (
|
|||||||
// (asnBatchTimeout), so a queue smaller than asnBatchSize is fine.
|
// (asnBatchTimeout), so a queue smaller than asnBatchSize is fine.
|
||||||
asHandlerQueueSize = 20000
|
asHandlerQueueSize = 20000
|
||||||
|
|
||||||
// asnBatchSize is the number of ASes in a full batch, which wakes the
|
// asnBatchSize is the number of ASN operations to batch together
|
||||||
// flush goroutine to write it.
|
|
||||||
asnBatchSize = 30000
|
asnBatchSize = 30000
|
||||||
|
|
||||||
|
// asnBatchMaxSize is the most ASN operations the batch holds while the
|
||||||
|
// flush goroutine writes the previous one: ten batches, about 10 MiB
|
||||||
|
// (README "Memory"). At this size HandleMessage waits until the flush
|
||||||
|
// goroutine takes the batch.
|
||||||
|
asnBatchMaxSize = 300000
|
||||||
|
|
||||||
// asnBatchTimeout is the maximum time to wait before flushing a batch
|
// asnBatchTimeout is the maximum time to wait before flushing a batch
|
||||||
// DO NOT reduce this timeout - larger batches are more efficient
|
// DO NOT reduce this timeout - larger batches are more efficient
|
||||||
asnBatchTimeout = 2 * time.Second
|
asnBatchTimeout = 2 * time.Second
|
||||||
)
|
)
|
||||||
|
|
||||||
// ASHandler records every AS in the AS paths of BGP UPDATE messages. Its batch
|
// ASHandler processes Autonomous System Number (ASN) information extracted from
|
||||||
// holds the earliest and the latest time of each AS's updates. The flush
|
// BGP UPDATE messages. It uses batched database operations to efficiently store
|
||||||
// goroutine writes the batch to the database when it holds asnBatchSize ASes,
|
// ASN data, collecting operations into batches that are flushed either when the
|
||||||
// and every asnBatchTimeout.
|
// batch reaches a size threshold or after a timeout period.
|
||||||
type ASHandler struct {
|
type ASHandler struct {
|
||||||
db database.Store
|
db database.Store
|
||||||
logger *logger.Logger
|
logger *logger.Logger
|
||||||
|
|
||||||
// Batching. mu guards batch and writing only. The flush goroutine takes
|
// Batching. mu guards batch and writing only: the flush goroutine takes
|
||||||
// the batch under mu and writes it after releasing mu. So HandleMessage
|
// the batch under mu and writes it after releasing mu, so HandleMessage
|
||||||
// never waits for a database write.
|
// waits for a database write only while batch is at asnBatchMaxSize.
|
||||||
mu sync.Mutex
|
mu sync.Mutex
|
||||||
// batch holds the earliest and the latest time of the updates for each AS
|
batch []asnOp
|
||||||
// since the flush goroutine last took it. It holds one entry per AS,
|
// writing is how many operations the flush goroutine took from batch
|
||||||
// however many updates arrive.
|
// and has not finished writing.
|
||||||
batch map[int]database.SeenTimes
|
writing int
|
||||||
// writing is how many ASes the flush goroutine took from batch and has
|
// batchTaken is signalled when the flush goroutine takes batch; its L is
|
||||||
// not finished writing.
|
// &mu.
|
||||||
writing int
|
batchTaken sync.Cond
|
||||||
batchFullCh chan struct{} // wakes the flush goroutine when batch is full
|
batchFullCh chan struct{} // wakes the flush goroutine when batch is full
|
||||||
// stopCh hands Stop's context to the flush goroutine. It holds one, so
|
// stopCh hands Stop's context to the flush goroutine. It holds one, so
|
||||||
// Stop does not wait for a write in progress to hand it over.
|
// Stop does not wait for a write in progress to hand it over.
|
||||||
@@ -53,10 +58,16 @@ type ASHandler struct {
|
|||||||
done chan struct{} // closed when the flush goroutine returns
|
done chan struct{} // closed when the flush goroutine returns
|
||||||
}
|
}
|
||||||
|
|
||||||
|
type asnOp struct {
|
||||||
|
number int
|
||||||
|
timestamp time.Time
|
||||||
|
}
|
||||||
|
|
||||||
// NewASHandler creates and returns a new ASHandler instance. It initializes
|
// NewASHandler creates and returns a new ASHandler instance. It initializes
|
||||||
// the batching system and starts a background goroutine that periodically
|
// the batching system and starts a background goroutine that periodically
|
||||||
// writes the collected ASes to the database. The caller must call Stop when
|
// flushes accumulated ASN operations to the database. The caller must call
|
||||||
// finished, to write the pending ASes and end the background goroutine.
|
// Stop when finished to ensure all pending operations are flushed and the
|
||||||
|
// background goroutine is terminated.
|
||||||
func NewASHandler(db database.Store, logger *logger.Logger) *ASHandler {
|
func NewASHandler(db database.Store, logger *logger.Logger) *ASHandler {
|
||||||
return newASHandler(db, logger, asnBatchTimeout)
|
return newASHandler(db, logger, asnBatchTimeout)
|
||||||
}
|
}
|
||||||
@@ -70,11 +81,12 @@ func newASHandler(
|
|||||||
h := &ASHandler{
|
h := &ASHandler{
|
||||||
db: db,
|
db: db,
|
||||||
logger: logger,
|
logger: logger,
|
||||||
batch: make(map[int]database.SeenTimes),
|
batch: make([]asnOp, 0, asnBatchSize),
|
||||||
batchFullCh: make(chan struct{}, 1),
|
batchFullCh: make(chan struct{}, 1),
|
||||||
stopCh: make(chan context.Context, 1),
|
stopCh: make(chan context.Context, 1),
|
||||||
done: make(chan struct{}),
|
done: make(chan struct{}),
|
||||||
}
|
}
|
||||||
|
h.batchTaken.L = &h.mu
|
||||||
|
|
||||||
// Start the flush timer goroutine
|
// Start the flush timer goroutine
|
||||||
go flushLoop(batchTimeout, h.batchFullCh, h.stopCh, h.done, h.flushBatch)
|
go flushLoop(batchTimeout, h.batchFullCh, h.stopCh, h.done, h.flushBatch)
|
||||||
@@ -98,25 +110,48 @@ func (h *ASHandler) QueueCapacity() int {
|
|||||||
return asHandlerQueueSize
|
return asHandlerQueueSize
|
||||||
}
|
}
|
||||||
|
|
||||||
// HandleMessage adds the time of msg to the batch entry of each AS in its AS
|
// HandleMessage processes a RIS Live BGP message by extracting all ASNs from
|
||||||
// path, the origin AS and the members of an AS set included. When the batch
|
// the AS path and queuing them for batch insertion into the database. The
|
||||||
// holds asnBatchSize ASes, HandleMessage wakes the flush goroutine to write it.
|
// origin ASN and all transit ASNs are recorded with their associated
|
||||||
|
// timestamps. When the batch reaches the configured size threshold, the flush
|
||||||
|
// goroutine is woken to write it.
|
||||||
func (h *ASHandler) HandleMessage(msg *ristypes.RISMessage) {
|
func (h *ASHandler) HandleMessage(msg *ristypes.RISMessage) {
|
||||||
|
// Use the pre-parsed timestamp
|
||||||
|
timestamp := msg.ParsedTimestamp
|
||||||
|
|
||||||
|
originASN := msg.Path.Origin()
|
||||||
|
|
||||||
h.mu.Lock()
|
h.mu.Lock()
|
||||||
defer h.mu.Unlock()
|
defer h.mu.Unlock()
|
||||||
|
|
||||||
// The origin AS is the last AS of the sequence, so the loop adds it too.
|
// While the batch is at its maximum size, wait until the flush goroutine
|
||||||
|
// takes it. Meanwhile the handler takes nothing off its queue, which
|
||||||
|
// fills, and the streamer drops messages for it.
|
||||||
|
for len(h.batch) >= asnBatchMaxSize {
|
||||||
|
h.batchTaken.Wait()
|
||||||
|
}
|
||||||
|
|
||||||
|
// Queue origin ASN operation
|
||||||
|
if originASN > 0 {
|
||||||
|
h.batch = append(h.batch, asnOp{
|
||||||
|
number: originASN,
|
||||||
|
timestamp: timestamp,
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
// Also track all ASNs in the path, the members of an AS set included
|
||||||
for _, asn := range slices.Concat(msg.Path.Sequence, msg.Path.Set) {
|
for _, asn := range slices.Concat(msg.Path.Sequence, msg.Path.Set) {
|
||||||
if asn > 0 {
|
if asn > 0 {
|
||||||
seen := h.batch[asn]
|
h.batch = append(h.batch, asnOp{
|
||||||
seen.Add(msg.ParsedTimestamp)
|
number: asn,
|
||||||
h.batch[asn] = seen
|
timestamp: timestamp,
|
||||||
|
})
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// A full batch wakes the flush goroutine. If it is writing, the wake-up
|
// A full batch wakes the flush goroutine. If it is writing, the wake-up
|
||||||
// waits in batchFullCh and the batch keeps taking updates until the write
|
// waits in batchFullCh and the batch keeps growing until the write ends
|
||||||
// ends.
|
// or the batch reaches asnBatchMaxSize.
|
||||||
if len(h.batch) >= asnBatchSize {
|
if len(h.batch) >= asnBatchSize {
|
||||||
select {
|
select {
|
||||||
case h.batchFullCh <- struct{}{}:
|
case h.batchFullCh <- struct{}{}:
|
||||||
@@ -125,11 +160,10 @@ func (h *ASHandler) HandleMessage(msg *ristypes.RISMessage) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// Stop stops the flush goroutine and waits for it until ctx ends. The flush
|
// Stop stops the flush goroutine, which writes the operations the handler holds
|
||||||
// goroutine writes the batch before it returns, unless ctx has ended. A write
|
// before it returns, and waits for it until ctx ends. Once ctx has ended the
|
||||||
// still running when the process exits is lost. Stop logs how many entries are
|
// flush goroutine skips that last write, and a write still running is lost
|
||||||
// not written, in the batch being written and in the next batch. An AS in both
|
// when the process exits, so Stop logs how many operations are not written.
|
||||||
// batches counts twice.
|
|
||||||
func (h *ASHandler) Stop(ctx context.Context) {
|
func (h *ASHandler) Stop(ctx context.Context) {
|
||||||
h.stopCh <- ctx
|
h.stopCh <- ctx
|
||||||
|
|
||||||
@@ -144,14 +178,14 @@ func (h *ASHandler) Stop(ctx context.Context) {
|
|||||||
|
|
||||||
if unwritten > 0 {
|
if unwritten > 0 {
|
||||||
h.logger.Error("Stop deadline passed before the AS handler wrote "+
|
h.logger.Error("Stop deadline passed before the AS handler wrote "+
|
||||||
"its batches", "batch_entries", unwritten)
|
"its updates", "updates", unwritten)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// takeBatch returns the pending ASes and leaves an empty batch in their place
|
// takeBatch returns the pending operations and leaves an empty batch in their
|
||||||
// for HandleMessage to fill. They count in writing until flushBatch has written
|
// place for HandleMessage to fill. They count in writing until flushBatch has
|
||||||
// them.
|
// written them.
|
||||||
func (h *ASHandler) takeBatch() map[int]database.SeenTimes {
|
func (h *ASHandler) takeBatch() []asnOp {
|
||||||
h.mu.Lock()
|
h.mu.Lock()
|
||||||
defer h.mu.Unlock()
|
defer h.mu.Unlock()
|
||||||
|
|
||||||
@@ -161,7 +195,8 @@ func (h *ASHandler) takeBatch() map[int]database.SeenTimes {
|
|||||||
|
|
||||||
batch := h.batch
|
batch := h.batch
|
||||||
h.writing = len(batch)
|
h.writing = len(batch)
|
||||||
h.batch = make(map[int]database.SeenTimes)
|
h.batch = make([]asnOp, 0, asnBatchSize)
|
||||||
|
h.batchTaken.Broadcast()
|
||||||
|
|
||||||
return batch
|
return batch
|
||||||
}
|
}
|
||||||
@@ -174,11 +209,21 @@ func (h *ASHandler) flushBatch() {
|
|||||||
return
|
return
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// The earliest and the latest time the batch saw each AS. The batch holds
|
||||||
|
// several updates for an AS, from many peers, so a new AS is stored first
|
||||||
|
// seen at the earliest of them and last seen at the latest.
|
||||||
|
asns := make(map[int]database.SeenTimes)
|
||||||
|
for _, op := range batch {
|
||||||
|
seen := asns[op.number]
|
||||||
|
seen.Add(op.timestamp)
|
||||||
|
asns[op.number] = seen
|
||||||
|
}
|
||||||
|
|
||||||
// Process all ASNs in a single batch transaction
|
// Process all ASNs in a single batch transaction
|
||||||
err := h.db.GetOrCreateASNBatch(batch)
|
err := h.db.GetOrCreateASNBatch(asns)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
h.logger.Error("Failed to process ASN batch",
|
h.logger.Error("Failed to process ASN batch",
|
||||||
"error", err, "count", len(batch))
|
"error", err, "count", len(asns))
|
||||||
}
|
}
|
||||||
|
|
||||||
h.mu.Lock()
|
h.mu.Lock()
|
||||||
|
|||||||
@@ -5,43 +5,28 @@ import (
|
|||||||
"slices"
|
"slices"
|
||||||
"testing"
|
"testing"
|
||||||
|
|
||||||
"git.eeqj.de/sneak/routewatch/internal/database"
|
|
||||||
"git.eeqj.de/sneak/routewatch/internal/logger"
|
"git.eeqj.de/sneak/routewatch/internal/logger"
|
||||||
"git.eeqj.de/sneak/routewatch/internal/ristypes"
|
|
||||||
)
|
)
|
||||||
|
|
||||||
// pathOfASes returns an AS path of n different ASes: AS 1 to AS n. With n up to
|
|
||||||
// asnBatchSize, none of them is testASNA to testASND.
|
|
||||||
func pathOfASes(n int) ristypes.ASPath {
|
|
||||||
sequence := make([]int, n)
|
|
||||||
for i := range sequence {
|
|
||||||
sequence[i] = i + 1
|
|
||||||
}
|
|
||||||
|
|
||||||
return ristypes.ASPath{Sequence: sequence}
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestASHandlerStoresTheEarliestAndLatestTimesOfANewAS handles two updates
|
// TestASHandlerStoresTheEarliestAndLatestTimesOfANewAS handles two updates
|
||||||
// whose path carries an AS the database does not hold yet, in one batch into a
|
// whose path carries an AS the database does not hold yet, the second 5
|
||||||
// real database. The second is 5 seconds after the first, and the handler gets
|
// seconds after the first, in one batch into a real database. The AS is stored
|
||||||
// it first. The AS is stored first seen at the time of the first update and
|
// first seen at the time of the first update and last seen at the time of the
|
||||||
// last seen at the time of the second.
|
// second.
|
||||||
func TestASHandlerStoresTheEarliestAndLatestTimesOfANewAS(t *testing.T) {
|
func TestASHandlerStoresTheEarliestAndLatestTimesOfANewAS(t *testing.T) {
|
||||||
t.Parallel()
|
t.Parallel()
|
||||||
|
|
||||||
db := newTestDatabase(t)
|
db := newTestDatabase(t)
|
||||||
|
|
||||||
// Built without NewASHandler's flush goroutine, so the test flushes itself.
|
// Built without NewASHandler's flush goroutine, so the test flushes itself.
|
||||||
h := &ASHandler{
|
h := &ASHandler{db: db, logger: logger.New()}
|
||||||
db: db, logger: logger.New(), batch: make(map[int]database.SeenTimes),
|
|
||||||
}
|
|
||||||
|
|
||||||
first := peerAnnouncement(testPeerIP, testV4Prefix)
|
first := peerAnnouncement(testPeerIP, testV4Prefix)
|
||||||
second := peerAnnouncement(testPeerIP, testV4Prefix)
|
second := peerAnnouncement(testPeerIP, testV4Prefix)
|
||||||
second.ParsedTimestamp = first.ParsedTimestamp.Add(secondUpdateAfter)
|
second.ParsedTimestamp = first.ParsedTimestamp.Add(secondUpdateAfter)
|
||||||
|
|
||||||
h.HandleMessage(second)
|
|
||||||
h.HandleMessage(first)
|
h.HandleMessage(first)
|
||||||
|
h.HandleMessage(second)
|
||||||
h.flushBatch()
|
h.flushBatch()
|
||||||
|
|
||||||
asn, _, err := db.GetAS(t.Context(), testASNA)
|
asn, _, err := db.GetAS(t.Context(), testASNA)
|
||||||
@@ -69,9 +54,7 @@ func TestASHandlerRecordsNothingFromAPathWithASZero(t *testing.T) {
|
|||||||
store := newMockStore()
|
store := newMockStore()
|
||||||
|
|
||||||
// Built without NewASHandler's flush goroutine, so the test flushes itself.
|
// Built without NewASHandler's flush goroutine, so the test flushes itself.
|
||||||
h := &ASHandler{
|
h := &ASHandler{db: store, logger: logger.New()}
|
||||||
db: store, logger: logger.New(), batch: make(map[int]database.SeenTimes),
|
|
||||||
}
|
|
||||||
|
|
||||||
msg := peerAnnouncement(testPeerIP, testV4Prefix)
|
msg := peerAnnouncement(testPeerIP, testV4Prefix)
|
||||||
msg.Path = decodePath(t, testASNA, 0, testASNB)
|
msg.Path = decodePath(t, testASNA, 0, testASNB)
|
||||||
|
|||||||
@@ -187,33 +187,25 @@ func (f *ASNFetcher) recordSuccess() {
|
|||||||
f.statsMu.Unlock()
|
f.statsMu.Unlock()
|
||||||
}
|
}
|
||||||
|
|
||||||
// lengthenInterval makes the interval 1.5 times as long, up to maxInterval.
|
// recordFailure increases the interval on failed fetch using exponential backoff.
|
||||||
// A failed lookup and a skipped one both lengthen it.
|
func (f *ASNFetcher) recordFailure() {
|
||||||
func (f *ASNFetcher) lengthenInterval() {
|
|
||||||
f.intervalMu.Lock()
|
f.intervalMu.Lock()
|
||||||
defer f.intervalMu.Unlock()
|
f.consecutiveFails++
|
||||||
|
|
||||||
|
// Exponential backoff: multiply by 1.5, capped at max
|
||||||
newInterval := min(
|
newInterval := min(
|
||||||
f.currentInterval*backoffNumerator/backoffDenominator, maxInterval)
|
f.currentInterval*backoffNumerator/backoffDenominator, maxInterval)
|
||||||
|
|
||||||
if newInterval != f.currentInterval {
|
if newInterval != f.currentInterval {
|
||||||
f.logger.Debug("Increased fetch interval",
|
f.logger.Debug("Increased fetch interval due to failure",
|
||||||
"old_interval", f.currentInterval,
|
"old_interval", f.currentInterval,
|
||||||
"new_interval", newInterval,
|
"new_interval", newInterval,
|
||||||
"consecutive_failures", f.consecutiveFails,
|
"consecutive_failures", f.consecutiveFails,
|
||||||
)
|
)
|
||||||
f.currentInterval = newInterval
|
f.currentInterval = newInterval
|
||||||
}
|
}
|
||||||
}
|
|
||||||
|
|
||||||
// recordFailure counts a failed fetch and lengthens the interval.
|
|
||||||
func (f *ASNFetcher) recordFailure() {
|
|
||||||
f.intervalMu.Lock()
|
|
||||||
f.consecutiveFails++
|
|
||||||
f.intervalMu.Unlock()
|
f.intervalMu.Unlock()
|
||||||
|
|
||||||
f.lengthenInterval()
|
|
||||||
|
|
||||||
// Record error time for stats, dropping those older than statsWindow
|
// Record error time for stats, dropping those older than statsWindow
|
||||||
f.statsMu.Lock()
|
f.statsMu.Lock()
|
||||||
now := f.now()
|
now := f.now()
|
||||||
@@ -295,16 +287,6 @@ func (f *ASNFetcher) fetchNextStale(ctx context.Context) {
|
|||||||
// so the AS stays stale. An AS IANA refers to no RIR is stored with no
|
// so the AS stays stale. An AS IANA refers to no RIR is stored with no
|
||||||
// WHOIS data, so it is not looked up again until it is stale; that counts
|
// WHOIS data, so it is not looked up again until it is stale; that counts
|
||||||
// as neither a success nor a failure.
|
// as neither a success nor a failure.
|
||||||
//
|
|
||||||
// The WHOIS client pauses an RIR's or NIC.br's server for whois.PauseTime
|
|
||||||
// after the server refuses a query. Lookups at the other servers go on. IANA
|
|
||||||
// is never paused, so a refusal by IANA is an ordinary failure. A lookup
|
|
||||||
// whose RIR is paused fails with whois.ErrPaused and is skipped. A skipped
|
|
||||||
// lookup leaves the AS stale and is not counted as a failure. Like a
|
|
||||||
// failure, a skipped lookup lengthens the interval. Each lookup asks IANA
|
|
||||||
// before it finds its RIR paused, so with a one-second interval the fetcher
|
|
||||||
// would ask IANA every second. The lookups that QueueImmediate asks for also
|
|
||||||
// run through fetchAndUpdate.
|
|
||||||
func (f *ASNFetcher) fetchAndUpdate(ctx context.Context, asn int) bool {
|
func (f *ASNFetcher) fetchAndUpdate(ctx context.Context, asn int) bool {
|
||||||
ctx, cancel := context.WithTimeout(ctx, time.Minute)
|
ctx, cancel := context.WithTimeout(ctx, time.Minute)
|
||||||
defer cancel()
|
defer cancel()
|
||||||
@@ -325,14 +307,6 @@ func (f *ASNFetcher) fetchAndUpdate(ctx context.Context, asn int) bool {
|
|||||||
return false
|
return false
|
||||||
}
|
}
|
||||||
|
|
||||||
if errors.Is(err, whois.ErrPaused) {
|
|
||||||
f.logger.Info("Skipped WHOIS lookup, the RIR's server is paused",
|
|
||||||
"asn", asn, "error", err)
|
|
||||||
f.lengthenInterval()
|
|
||||||
|
|
||||||
return false
|
|
||||||
}
|
|
||||||
|
|
||||||
if err != nil {
|
if err != nil {
|
||||||
f.logger.Error("WHOIS lookup failed", "asn", asn, "error", err)
|
f.logger.Error("WHOIS lookup failed", "asn", asn, "error", err)
|
||||||
f.recordFailure()
|
f.recordFailure()
|
||||||
|
|||||||
@@ -13,7 +13,6 @@ import (
|
|||||||
|
|
||||||
"git.eeqj.de/sneak/routewatch/internal/database"
|
"git.eeqj.de/sneak/routewatch/internal/database"
|
||||||
"git.eeqj.de/sneak/routewatch/internal/logger"
|
"git.eeqj.de/sneak/routewatch/internal/logger"
|
||||||
"git.eeqj.de/sneak/routewatch/internal/whois"
|
|
||||||
)
|
)
|
||||||
|
|
||||||
// The WHOIS servers a lookup asks: IANA first, then the RIR IANA refers it
|
// The WHOIS servers a lookup asks: IANA first, then the RIR IANA refers it
|
||||||
@@ -132,151 +131,6 @@ func TestFetchAndUpdateKeepsTheRowWhenTheLookupFails(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// TestFetchAndUpdateWaitsOutAPause looks up an AS at RIPE while RIPE refuses
|
|
||||||
// the query. The test sets the WHOIS client's clock. The next lookup comes a
|
|
||||||
// second before whois.PauseTime has passed. It sends RIPE nothing and leaves
|
|
||||||
// the AS never looked up. It counts as neither a success nor a failure. A
|
|
||||||
// lookup at whois.PauseTime asks RIPE again.
|
|
||||||
func TestFetchAndUpdateWaitsOutAPause(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
f, db := newFetcherWithWHOISAnswers(t, map[string]string{
|
|
||||||
ianaWHOISServer: ianaRefersToRIPE,
|
|
||||||
ripeWHOISServer: ripeAccessDenied,
|
|
||||||
})
|
|
||||||
|
|
||||||
ripeAsks := 0
|
|
||||||
dial := f.whoisClient.DialContext
|
|
||||||
f.whoisClient.DialContext = func(
|
|
||||||
ctx context.Context, network, address string,
|
|
||||||
) (net.Conn, error) {
|
|
||||||
if address == ripeWHOISServer {
|
|
||||||
ripeAsks++
|
|
||||||
}
|
|
||||||
|
|
||||||
return dial(ctx, network, address)
|
|
||||||
}
|
|
||||||
|
|
||||||
start := time.Date(2026, 11, 2, 0, 0, 0, 0, time.UTC)
|
|
||||||
now := start
|
|
||||||
f.whoisClient.Now = func() time.Time { return now }
|
|
||||||
|
|
||||||
f.fetchAndUpdate(t.Context(), testASNA)
|
|
||||||
|
|
||||||
now = start.Add(whois.PauseTime - time.Second)
|
|
||||||
|
|
||||||
f.fetchAndUpdate(t.Context(), testASNA)
|
|
||||||
|
|
||||||
if ripeAsks != 1 {
|
|
||||||
t.Errorf("RIPE was asked %d times during its pause, want 1", ripeAsks)
|
|
||||||
}
|
|
||||||
|
|
||||||
stats, err := db.GetWHOISStats(t.Context(), whoisStaleThreshold)
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("GetWHOISStats: %v", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
if stats.NeverFetched != 1 {
|
|
||||||
t.Errorf("never looked up = %d, want 1", stats.NeverFetched)
|
|
||||||
}
|
|
||||||
|
|
||||||
got := f.GetStats()
|
|
||||||
if got.ConsecutiveFails != 1 || got.ErrorsLastHour != 1 ||
|
|
||||||
got.SuccessesLastHour != 0 {
|
|
||||||
t.Errorf("%d failures in a row, %d errors, %d successes; want 1, 1, 0",
|
|
||||||
got.ConsecutiveFails, got.ErrorsLastHour, got.SuccessesLastHour)
|
|
||||||
}
|
|
||||||
|
|
||||||
now = start.Add(whois.PauseTime)
|
|
||||||
|
|
||||||
f.fetchAndUpdate(t.Context(), testASNA)
|
|
||||||
|
|
||||||
if ripeAsks != 2 {
|
|
||||||
t.Errorf("RIPE was asked %d times after its pause, want 2", ripeAsks)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestFetchAndUpdateBacksOffDuringAPause looks up an AS at RIPE while RIPE
|
|
||||||
// refuses the query. The refusal pauses RIPE for an hour, so every later
|
|
||||||
// lookup in the test is skipped. The test sets the WHOIS client's clock and
|
|
||||||
// looks the AS up again each time the interval has passed. Like a failure,
|
|
||||||
// each skip must make the interval 1.5 times as long, up to one minute. The
|
|
||||||
// skips must not count as failures.
|
|
||||||
func TestFetchAndUpdateBacksOffDuringAPause(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
f, _ := newFetcherWithWHOISAnswers(t, map[string]string{
|
|
||||||
ianaWHOISServer: ianaRefersToRIPE,
|
|
||||||
ripeWHOISServer: ripeAccessDenied,
|
|
||||||
})
|
|
||||||
|
|
||||||
now := time.Date(2026, 11, 2, 0, 0, 0, 0, time.UTC)
|
|
||||||
f.whoisClient.Now = func() time.Time { return now }
|
|
||||||
|
|
||||||
f.fetchAndUpdate(t.Context(), testASNA)
|
|
||||||
|
|
||||||
for i, want := range []time.Duration{
|
|
||||||
33750 * time.Millisecond,
|
|
||||||
50625 * time.Millisecond,
|
|
||||||
time.Minute,
|
|
||||||
time.Minute,
|
|
||||||
} {
|
|
||||||
now = now.Add(f.getInterval())
|
|
||||||
|
|
||||||
f.fetchAndUpdate(t.Context(), testASNA)
|
|
||||||
|
|
||||||
if got := f.getInterval(); got != want {
|
|
||||||
t.Errorf("interval after %d skipped lookups = %v, want %v",
|
|
||||||
i+1, got, want)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
got := f.GetStats()
|
|
||||||
if got.ConsecutiveFails != 1 || got.ErrorsLastHour != 1 {
|
|
||||||
t.Errorf("%d failures in a row and %d errors, want 1 and 1",
|
|
||||||
got.ConsecutiveFails, got.ErrorsLastHour)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestFetchAndUpdateKeepsAskingIANA looks up an AS while IANA's server
|
|
||||||
// refuses the connection. The test sets the WHOIS client's clock. The next
|
|
||||||
// lookup comes a second before whois.PauseTime has passed. IANA is never
|
|
||||||
// paused, so that lookup asks IANA again. Each refusal counts as a failure.
|
|
||||||
func TestFetchAndUpdateKeepsAskingIANA(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
// With no answers, every server refuses the connection.
|
|
||||||
f, _ := newFetcherWithWHOISAnswers(t, map[string]string{})
|
|
||||||
|
|
||||||
ianaAsks := 0
|
|
||||||
dial := f.whoisClient.DialContext
|
|
||||||
f.whoisClient.DialContext = func(
|
|
||||||
ctx context.Context, network, address string,
|
|
||||||
) (net.Conn, error) {
|
|
||||||
if address == ianaWHOISServer {
|
|
||||||
ianaAsks++
|
|
||||||
}
|
|
||||||
|
|
||||||
return dial(ctx, network, address)
|
|
||||||
}
|
|
||||||
|
|
||||||
start := time.Date(2026, 11, 2, 0, 0, 0, 0, time.UTC)
|
|
||||||
now := start
|
|
||||||
f.whoisClient.Now = func() time.Time { return now }
|
|
||||||
|
|
||||||
f.fetchAndUpdate(t.Context(), testASNA)
|
|
||||||
|
|
||||||
now = start.Add(whois.PauseTime - time.Second)
|
|
||||||
|
|
||||||
f.fetchAndUpdate(t.Context(), testASNA)
|
|
||||||
|
|
||||||
got := f.GetStats()
|
|
||||||
if ianaAsks != 2 || got.ConsecutiveFails != 2 {
|
|
||||||
t.Errorf("IANA was asked %d times, with %d failures in a row; "+
|
|
||||||
"want 2 and 2", ianaAsks, got.ConsecutiveFails)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestRecordStepsTheInterval has the fetcher fail from its base interval of
|
// TestRecordStepsTheInterval has the fetcher fail from its base interval of
|
||||||
// 15 seconds, then succeed. Each failure makes the interval 1.5 times as
|
// 15 seconds, then succeed. Each failure makes the interval 1.5 times as
|
||||||
// long, up to one minute, and each success halves it, down to one second.
|
// long, up to one minute, and each success halves it, down to one second.
|
||||||
@@ -370,7 +224,7 @@ func TestASNFetcherStopReturnsByItsDeadline(t *testing.T) {
|
|||||||
|
|
||||||
select {
|
select {
|
||||||
case <-lookupStarted:
|
case <-lookupStarted:
|
||||||
case <-time.After(stepTimeout):
|
case <-time.After(handleMessageTimeout):
|
||||||
t.Fatal("the fetcher did not start the lookup")
|
t.Fatal("the fetcher did not start the lookup")
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -386,22 +240,22 @@ func TestASNFetcherStopReturnsByItsDeadline(t *testing.T) {
|
|||||||
|
|
||||||
select {
|
select {
|
||||||
case <-f.done:
|
case <-f.done:
|
||||||
case <-time.After(stepTimeout):
|
case <-time.After(handleMessageTimeout):
|
||||||
t.Fatal("the fetcher goroutine did not return after the lookup ended")
|
t.Fatal("the fetcher goroutine did not return after the lookup ended")
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// TestRecordKeepsOnlyTheStatsWindow records a success and a failure every
|
// TestRecordKeepsOnlyTheStatsWindow records a success and a failure every
|
||||||
// second, by a clock the test sets, for three times statsWindow, and never
|
// second, by a clock the test sets, far longer than statsWindow, and never
|
||||||
// asks for the stats. The fetcher must keep only the times within the last
|
// asks for the stats. The fetcher must keep only the times within the last
|
||||||
// statsWindow.
|
// statsWindow.
|
||||||
func TestRecordKeepsOnlyTheStatsWindow(t *testing.T) {
|
func TestRecordKeepsOnlyTheStatsWindow(t *testing.T) {
|
||||||
t.Parallel()
|
t.Parallel()
|
||||||
|
|
||||||
const (
|
const (
|
||||||
|
records = 100_000
|
||||||
step = time.Second
|
step = time.Second
|
||||||
want = int(statsWindow / step)
|
want = int(statsWindow / step)
|
||||||
records = 3 * want
|
|
||||||
)
|
)
|
||||||
|
|
||||||
clock := time.Now()
|
clock := time.Now()
|
||||||
|
|||||||
@@ -5,7 +5,6 @@ import (
|
|||||||
"context"
|
"context"
|
||||||
"encoding/json"
|
"encoding/json"
|
||||||
"fmt"
|
"fmt"
|
||||||
"io"
|
|
||||||
"log/slog"
|
"log/slog"
|
||||||
"slices"
|
"slices"
|
||||||
"sync"
|
"sync"
|
||||||
@@ -20,15 +19,9 @@ import (
|
|||||||
)
|
)
|
||||||
|
|
||||||
const (
|
const (
|
||||||
// stepTimeout is how long a test waits for a step it expects, such as
|
// handleMessageTimeout is how long a test waits for HandleMessage to
|
||||||
// HandleMessage returning, a write starting or Shutdown returning, before
|
// return, or for a write to start, before it fails.
|
||||||
// it fails. Only a step that never comes should reach it. On a loaded
|
handleMessageTimeout = 2 * time.Second
|
||||||
// host, under the race detector, HandleMessage takes seconds to apply a
|
|
||||||
// full prefix batch. The helpers below bound every wait with it, Stop
|
|
||||||
// included. A test that hangs in one of them waits out at most two of
|
|
||||||
// these bounds, its step's and Stop's, and then fails with its own
|
|
||||||
// message. That leaves time before go test's 90-second timeout.
|
|
||||||
stepTimeout = 15 * time.Second
|
|
||||||
|
|
||||||
// stillWaitingAfter is how long a test watches a HandleMessage that must
|
// stillWaitingAfter is how long a test watches a HandleMessage that must
|
||||||
// wait. One that does not wait returns well within it.
|
// wait. One that does not wait returns well within it.
|
||||||
@@ -175,18 +168,15 @@ func (s *blockingStore) releaseWrites() {
|
|||||||
//
|
//
|
||||||
//nolint:tagliatelle // field names are the keys the handlers log
|
//nolint:tagliatelle // field names are the keys the handlers log
|
||||||
type stopLog struct {
|
type stopLog struct {
|
||||||
Msg string `json:"msg"`
|
Msg string `json:"msg"`
|
||||||
Updates int `json:"updates"`
|
Updates int `json:"updates"`
|
||||||
TableChanges int `json:"table_changes"`
|
TableChanges int `json:"table_changes"`
|
||||||
HeldMessages int `json:"held_messages"`
|
Paths int `json:"paths"`
|
||||||
OriginChanges int `json:"origin_changes"`
|
|
||||||
Paths int `json:"paths"`
|
|
||||||
BatchEntries int `json:"batch_entries"`
|
|
||||||
}
|
}
|
||||||
|
|
||||||
// newErrorLogger returns a logger that writes only errors, as JSON lines, to
|
// newErrorLogger returns a logger that writes only errors, as JSON lines, to
|
||||||
// logs.
|
// logs.
|
||||||
func newErrorLogger(logs io.Writer) *logger.Logger {
|
func newErrorLogger(logs *bytes.Buffer) *logger.Logger {
|
||||||
return &logger.Logger{Logger: slog.New(slog.NewJSONHandler(logs,
|
return &logger.Logger{Logger: slog.New(slog.NewJSONHandler(logs,
|
||||||
&slog.HandlerOptions{Level: slog.LevelError}))}
|
&slog.HandlerOptions{Level: slog.LevelError}))}
|
||||||
}
|
}
|
||||||
@@ -208,7 +198,7 @@ func lastLogged(t *testing.T, logs *bytes.Buffer) stopLog {
|
|||||||
return logged
|
return logged
|
||||||
}
|
}
|
||||||
|
|
||||||
// returnsInTime reports whether f returns within stepTimeout.
|
// returnsInTime reports whether f returns within handleMessageTimeout.
|
||||||
func returnsInTime(f func()) bool {
|
func returnsInTime(f func()) bool {
|
||||||
done := make(chan struct{})
|
done := make(chan struct{})
|
||||||
|
|
||||||
@@ -220,22 +210,11 @@ func returnsInTime(f func()) bool {
|
|||||||
select {
|
select {
|
||||||
case <-done:
|
case <-done:
|
||||||
return true
|
return true
|
||||||
case <-time.After(stepTimeout):
|
case <-time.After(handleMessageTimeout):
|
||||||
return false
|
return false
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// stopInTime stops h, with no deadline, and fails the test if Stop does not
|
|
||||||
// return within stepTimeout. It only marks the test failed, so it can be
|
|
||||||
// deferred.
|
|
||||||
func stopInTime(t *testing.T, h interface{ Stop(ctx context.Context) }) {
|
|
||||||
t.Helper()
|
|
||||||
|
|
||||||
if !returnsInTime(func() { h.Stop(t.Context()) }) {
|
|
||||||
t.Errorf("Stop did not return within %s", stepTimeout)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// startBlockedWrite hands h the messages in fill and waits until h starts to
|
// startBlockedWrite hands h the messages in fill and waits until h starts to
|
||||||
// write them to store. The prefix, AS and peer handlers start when fill fills
|
// write them to store. The prefix, AS and peer handlers start when fill fills
|
||||||
// their batch, and the peering handler on its timer. store holds the write
|
// their batch, and the peering handler on its timer. store holds the write
|
||||||
@@ -261,7 +240,7 @@ func startBlockedWrite(
|
|||||||
|
|
||||||
select {
|
select {
|
||||||
case <-store.writeStarted:
|
case <-store.writeStarted:
|
||||||
case <-time.After(stepTimeout):
|
case <-time.After(handleMessageTimeout):
|
||||||
t.Fatal("the handler did not start to write the messages")
|
t.Fatal("the handler did not start to write the messages")
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -282,7 +261,7 @@ func handleWhileAWriteIsBlocked(
|
|||||||
) {
|
) {
|
||||||
t.Helper()
|
t.Helper()
|
||||||
|
|
||||||
defer stopInTime(t, h)
|
defer h.Stop(t.Context())
|
||||||
defer store.releaseWrites()
|
defer store.releaseWrites()
|
||||||
|
|
||||||
startBlockedWrite(t, store, h, fill)
|
startBlockedWrite(t, store, h, fill)
|
||||||
@@ -296,9 +275,7 @@ func handleWhileAWriteIsBlocked(
|
|||||||
// while store holds the write, hands h the messages in toMax, which bring its
|
// while store holds the write, hands h the messages in toMax, which bring its
|
||||||
// next batch to its maximum size, and the message extra. HandleMessage for
|
// next batch to its maximum size, and the message extra. HandleMessage for
|
||||||
// extra must wait until the test lets the write finish, and return after. Then
|
// extra must wait until the test lets the write finish, and return after. Then
|
||||||
// it stops h, which writes what it holds. The tests lower h's maximum batch
|
// it stops h, which writes what it holds.
|
||||||
// size to one batch, so toMax is one batch: the real maximum of ten batches is
|
|
||||||
// slow to reach under the race detector.
|
|
||||||
func handleAtMaxBatchSize(
|
func handleAtMaxBatchSize(
|
||||||
t *testing.T,
|
t *testing.T,
|
||||||
store *blockingStore,
|
store *blockingStore,
|
||||||
@@ -311,18 +288,13 @@ func handleAtMaxBatchSize(
|
|||||||
) {
|
) {
|
||||||
t.Helper()
|
t.Helper()
|
||||||
|
|
||||||
defer stopInTime(t, h)
|
defer h.Stop(t.Context())
|
||||||
defer store.releaseWrites()
|
defer store.releaseWrites()
|
||||||
|
|
||||||
startBlockedWrite(t, store, h, fill)
|
startBlockedWrite(t, store, h, fill)
|
||||||
|
|
||||||
if !returnsInTime(func() {
|
for _, msg := range toMax {
|
||||||
for _, msg := range toMax {
|
h.HandleMessage(msg)
|
||||||
h.HandleMessage(msg)
|
|
||||||
}
|
|
||||||
}) {
|
|
||||||
t.Fatal("HandleMessage did not return while the batch grew to its " +
|
|
||||||
"maximum size")
|
|
||||||
}
|
}
|
||||||
|
|
||||||
returned := make(chan struct{})
|
returned := make(chan struct{})
|
||||||
@@ -343,7 +315,7 @@ func handleAtMaxBatchSize(
|
|||||||
|
|
||||||
select {
|
select {
|
||||||
case <-returned:
|
case <-returned:
|
||||||
case <-time.After(stepTimeout):
|
case <-time.After(handleMessageTimeout):
|
||||||
t.Fatal("HandleMessage did not return after the database write finished")
|
t.Fatal("HandleMessage did not return after the database write finished")
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -373,12 +345,8 @@ func stopWhileAWriteIsBlocked(
|
|||||||
|
|
||||||
startBlockedWrite(t, store, h, fill)
|
startBlockedWrite(t, store, h, fill)
|
||||||
|
|
||||||
if !returnsInTime(func() {
|
for _, msg := range extra {
|
||||||
for _, msg := range extra {
|
h.HandleMessage(msg)
|
||||||
h.HandleMessage(msg)
|
|
||||||
}
|
|
||||||
}) {
|
|
||||||
t.Fatal("HandleMessage did not return while a database write was blocked")
|
|
||||||
}
|
}
|
||||||
|
|
||||||
ctx, cancel := context.WithTimeout(t.Context(), stopDeadline)
|
ctx, cancel := context.WithTimeout(t.Context(), stopDeadline)
|
||||||
@@ -393,7 +361,7 @@ func stopWhileAWriteIsBlocked(
|
|||||||
|
|
||||||
select {
|
select {
|
||||||
case <-done:
|
case <-done:
|
||||||
case <-time.After(stepTimeout):
|
case <-time.After(handleMessageTimeout):
|
||||||
t.Fatal("the flush goroutine did not return after the write finished")
|
t.Fatal("the flush goroutine did not return after the write finished")
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -412,8 +380,7 @@ func TestPrefixHandlerSkipsItsLastWriteAfterTheStopDeadline(t *testing.T) {
|
|||||||
|
|
||||||
store := newBlockingStore()
|
store := newBlockingStore()
|
||||||
h := newPrefixHandler(store, newErrorLogger(&logs),
|
h := newPrefixHandler(store, newErrorLogger(&logs),
|
||||||
routetable.New(routetable.Params{}), longerThanAnyTest,
|
routetable.New(routetable.Params{}), longerThanAnyTest)
|
||||||
maxTableChangesPerWrite)
|
|
||||||
fill := slices.Repeat([]string{testV4Prefix}, prefixBatchSize)
|
fill := slices.Repeat([]string{testV4Prefix}, prefixBatchSize)
|
||||||
stopWhileAWriteIsBlocked(t, store, h, h.done,
|
stopWhileAWriteIsBlocked(t, store, h, h.done,
|
||||||
[]*ristypes.RISMessage{prefixAnnouncement(fill)},
|
[]*ristypes.RISMessage{prefixAnnouncement(fill)},
|
||||||
@@ -473,42 +440,6 @@ func (s tableWriteBlockingStore) WriteRouteChanges(
|
|||||||
return s.mockStore.WriteRouteChanges(changes, newPeers)
|
return s.mockStore.WriteRouteChanges(changes, newPeers)
|
||||||
}
|
}
|
||||||
|
|
||||||
// stopPrefixHandlerWithBlockedWrite stops the prefix handler, whose store
|
|
||||||
// holds the write that fill starts, after handing it the message extra, and
|
|
||||||
// returns what Stop logged. blocking is the store the write waits on; store is
|
|
||||||
// a blockingStore wrapped for the routing table's changes or the origin
|
|
||||||
// changes.
|
|
||||||
func stopPrefixHandlerWithBlockedWrite(
|
|
||||||
t *testing.T,
|
|
||||||
blocking *blockingStore,
|
|
||||||
store database.Store,
|
|
||||||
logs *bytes.Buffer,
|
|
||||||
fill, extra *ristypes.RISMessage,
|
|
||||||
) stopLog {
|
|
||||||
t.Helper()
|
|
||||||
|
|
||||||
h := NewPrefixHandler(store, newErrorLogger(logs),
|
|
||||||
routetable.New(routetable.Params{}))
|
|
||||||
stopWhileAWriteIsBlocked(t, blocking, h, h.done,
|
|
||||||
[]*ristypes.RISMessage{fill}, extra)
|
|
||||||
|
|
||||||
return lastLogged(t, logs)
|
|
||||||
}
|
|
||||||
|
|
||||||
// originChangesBlockingStore is a blockingStore whose writes of the origin
|
|
||||||
// changes also wait until the test calls releaseWrites.
|
|
||||||
type originChangesBlockingStore struct {
|
|
||||||
*blockingStore
|
|
||||||
}
|
|
||||||
|
|
||||||
func (s originChangesBlockingStore) WriteOriginChanges(
|
|
||||||
changes []routetable.OriginChange,
|
|
||||||
) error {
|
|
||||||
s.waitForRelease()
|
|
||||||
|
|
||||||
return s.mockStore.WriteOriginChanges(changes)
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestPrefixHandlerStopCountsTheTableChangesOfAWriteInProgress holds the write
|
// TestPrefixHandlerStopCountsTheTableChangesOfAWriteInProgress holds the write
|
||||||
// of the routing table's changes that a full batch starts, then hands the
|
// of the routing table's changes that a full batch starts, then hands the
|
||||||
// prefix handler one more prefix and stops it with a deadline that passes
|
// prefix handler one more prefix and stops it with a deadline that passes
|
||||||
@@ -522,81 +453,33 @@ func TestPrefixHandlerStopCountsTheTableChangesOfAWriteInProgress(t *testing.T)
|
|||||||
var logs bytes.Buffer
|
var logs bytes.Buffer
|
||||||
|
|
||||||
blocking := newBlockingStore()
|
blocking := newBlockingStore()
|
||||||
|
h := NewPrefixHandler(tableWriteBlockingStore{blockingStore: blocking},
|
||||||
|
newErrorLogger(&logs), routetable.New(routetable.Params{}))
|
||||||
fill := slices.Repeat([]string{testV4Prefix}, prefixBatchSize)
|
fill := slices.Repeat([]string{testV4Prefix}, prefixBatchSize)
|
||||||
logged := stopPrefixHandlerWithBlockedWrite(t, blocking,
|
stopWhileAWriteIsBlocked(t, blocking, h, h.done,
|
||||||
tableWriteBlockingStore{blockingStore: blocking}, &logs,
|
[]*ristypes.RISMessage{prefixAnnouncement(fill)},
|
||||||
prefixAnnouncement(fill), prefixAnnouncement([]string{extraPrefix}))
|
prefixAnnouncement([]string{extraPrefix}))
|
||||||
|
|
||||||
if logged.TableChanges != 2 {
|
if logged := lastLogged(t, &logs); logged.TableChanges != 2 {
|
||||||
t.Errorf("Stop logged %d table changes not written, want 2",
|
t.Errorf("Stop logged %d table changes not written, want 2",
|
||||||
logged.TableChanges)
|
logged.TableChanges)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// TestPrefixHandlerStopCountsTheOriginChangesOfAWriteInProgress holds the
|
|
||||||
// write of the origin changes that a full batch starts, then hands the prefix
|
|
||||||
// handler one more prefix and stops it with a deadline that passes meanwhile.
|
|
||||||
// Stop must log as not written both the change the held write took and the one
|
|
||||||
// since.
|
|
||||||
func TestPrefixHandlerStopCountsTheOriginChangesOfAWriteInProgress(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
const extraPrefix = "198.51.100.0/24"
|
|
||||||
|
|
||||||
var logs bytes.Buffer
|
|
||||||
|
|
||||||
blocking := newBlockingStore()
|
|
||||||
fill := slices.Repeat([]string{testV4Prefix}, prefixBatchSize)
|
|
||||||
logged := stopPrefixHandlerWithBlockedWrite(t, blocking,
|
|
||||||
originChangesBlockingStore{blockingStore: blocking}, &logs,
|
|
||||||
prefixAnnouncement(fill), prefixAnnouncement([]string{extraPrefix}))
|
|
||||||
|
|
||||||
if logged.OriginChanges != 2 {
|
|
||||||
t.Errorf("Stop logged %d origin changes not written, want 2",
|
|
||||||
logged.OriginChanges)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestPrefixHandlerStopCountsTheWaitingOriginChanges stops the prefix handler
|
|
||||||
// with a deadline that has already passed, so the flush goroutine skips its
|
|
||||||
// last write. The origin changes still waiting in the routing table are then
|
|
||||||
// not written, and Stop must log them.
|
|
||||||
func TestPrefixHandlerStopCountsTheWaitingOriginChanges(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
var logs bytes.Buffer
|
|
||||||
|
|
||||||
store := newMockStore()
|
|
||||||
h := newPrefixHandler(store, newErrorLogger(&logs),
|
|
||||||
routetable.New(routetable.Params{}), longerThanAnyTest,
|
|
||||||
maxTableChangesPerWrite)
|
|
||||||
|
|
||||||
h.HandleMessage(peerAnnouncement(testPeerIP, testV4Prefix, testV6Prefix))
|
|
||||||
|
|
||||||
ctx, cancel := context.WithCancel(t.Context())
|
|
||||||
cancel()
|
|
||||||
h.Stop(ctx)
|
|
||||||
|
|
||||||
if logged := lastLogged(t, &logs); logged.OriginChanges != 2 {
|
|
||||||
t.Errorf("Stop logged %d origin changes not written, want 2",
|
|
||||||
logged.OriginChanges)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestASHandlerSkipsItsLastWriteAfterTheStopDeadline also checks what Stop
|
|
||||||
// logs: the asnBatchSize entries of the held write and the entry for testASNB,
|
|
||||||
// handled since, are not written.
|
|
||||||
func TestASHandlerSkipsItsLastWriteAfterTheStopDeadline(t *testing.T) {
|
func TestASHandlerSkipsItsLastWriteAfterTheStopDeadline(t *testing.T) {
|
||||||
t.Parallel()
|
t.Parallel()
|
||||||
|
|
||||||
const wantEntries = asnBatchSize + 1
|
// Each message gives an update for its origin AS and one for each AS
|
||||||
|
// in its path: fill asnBatchSize+1, extra 2.
|
||||||
|
const wantUpdates = asnBatchSize + 1 + 2
|
||||||
|
|
||||||
var logs bytes.Buffer
|
var logs bytes.Buffer
|
||||||
|
|
||||||
store := newBlockingStore()
|
store := newBlockingStore()
|
||||||
h := newASHandler(store, newErrorLogger(&logs), longerThanAnyTest)
|
h := newASHandler(store, newErrorLogger(&logs), longerThanAnyTest)
|
||||||
|
fill := ristypes.ASPath{Sequence: slices.Repeat([]int{testASNA}, asnBatchSize)}
|
||||||
stopWhileAWriteIsBlocked(t, store, h, h.done,
|
stopWhileAWriteIsBlocked(t, store, h, h.done,
|
||||||
[]*ristypes.RISMessage{{Path: pathOfASes(asnBatchSize)}},
|
[]*ristypes.RISMessage{{Path: fill}},
|
||||||
&ristypes.RISMessage{Path: ristypes.ASPath{Sequence: []int{testASNB}}})
|
&ristypes.RISMessage{Path: ristypes.ASPath{Sequence: []int{testASNB}}})
|
||||||
|
|
||||||
store.mu.Lock()
|
store.mu.Lock()
|
||||||
@@ -606,27 +489,26 @@ func TestASHandlerSkipsItsLastWriteAfterTheStopDeadline(t *testing.T) {
|
|||||||
t.Errorf("AS%d was written after the stop deadline", testASNB)
|
t.Errorf("AS%d was written after the stop deadline", testASNB)
|
||||||
}
|
}
|
||||||
|
|
||||||
if logged := lastLogged(t, &logs); logged.BatchEntries != wantEntries {
|
if logged := lastLogged(t, &logs); logged.Updates != wantUpdates {
|
||||||
t.Errorf("Stop logged %d batch entries not written, want %d",
|
t.Errorf("Stop logged %d updates not written, want %d",
|
||||||
logged.BatchEntries, wantEntries)
|
logged.Updates, wantUpdates)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// TestASHandlerSkipsAFullBatchAfterTheStopDeadline hands the AS handler a full
|
// TestASHandlerSkipsAFullBatchAfterTheStopDeadline hands the AS handler a full
|
||||||
// batch while a write is held, so the batch's wake-up of the flush goroutine
|
// batch while a write is held, so the batch's wake-up of the flush goroutine
|
||||||
// waits. The full batch holds the fill's ASes again, and testASNB. The stop
|
// waits. The stop deadline passes meanwhile. When the write ends, the flush
|
||||||
// deadline passes meanwhile. When the write ends, the flush goroutine must stop
|
// goroutine must stop instead of writing the batch.
|
||||||
// instead of writing the batch.
|
|
||||||
func TestASHandlerSkipsAFullBatchAfterTheStopDeadline(t *testing.T) {
|
func TestASHandlerSkipsAFullBatchAfterTheStopDeadline(t *testing.T) {
|
||||||
t.Parallel()
|
t.Parallel()
|
||||||
|
|
||||||
store := newBlockingStore()
|
store := newBlockingStore()
|
||||||
h := newASHandler(store, logger.New(), longerThanAnyTest)
|
h := newASHandler(store, logger.New(), longerThanAnyTest)
|
||||||
fill := &ristypes.RISMessage{Path: pathOfASes(asnBatchSize)}
|
fill := ristypes.ASPath{Sequence: slices.Repeat([]int{testASNA}, asnBatchSize)}
|
||||||
|
full := ristypes.ASPath{Sequence: slices.Repeat([]int{testASNB}, asnBatchSize)}
|
||||||
stopWhileAWriteIsBlocked(t, store, h, h.done,
|
stopWhileAWriteIsBlocked(t, store, h, h.done,
|
||||||
[]*ristypes.RISMessage{fill},
|
[]*ristypes.RISMessage{{Path: fill}},
|
||||||
fill,
|
&ristypes.RISMessage{Path: full})
|
||||||
&ristypes.RISMessage{Path: ristypes.ASPath{Sequence: []int{testASNB}}})
|
|
||||||
|
|
||||||
store.mu.Lock()
|
store.mu.Lock()
|
||||||
defer store.mu.Unlock()
|
defer store.mu.Unlock()
|
||||||
@@ -715,7 +597,7 @@ func TestPeeringHandlerStopCountsTheWriteInProgress(t *testing.T) {
|
|||||||
|
|
||||||
select {
|
select {
|
||||||
case <-h.done:
|
case <-h.done:
|
||||||
case <-time.After(stepTimeout):
|
case <-time.After(handleMessageTimeout):
|
||||||
t.Fatal("the processing goroutine did not return after the write " +
|
t.Fatal("the processing goroutine did not return after the write " +
|
||||||
"finished")
|
"finished")
|
||||||
}
|
}
|
||||||
@@ -736,7 +618,7 @@ func TestPeeringHandlerStartsNoWriteAfterTheStopDeadline(t *testing.T) {
|
|||||||
const processInterval = 10 * time.Millisecond
|
const processInterval = 10 * time.Millisecond
|
||||||
|
|
||||||
store := newBlockingStore()
|
store := newBlockingStore()
|
||||||
h := newPeeringHandler(store, logger.New(), processInterval, maxTrackedPaths)
|
h := newPeeringHandler(store, logger.New(), processInterval)
|
||||||
stopWhileAWriteIsBlocked(t, store, h, h.done,
|
stopWhileAWriteIsBlocked(t, store, h, h.done,
|
||||||
[]*ristypes.RISMessage{
|
[]*ristypes.RISMessage{
|
||||||
{Path: ristypes.ASPath{Sequence: []int{testASNA, testASNB}}},
|
{Path: ristypes.ASPath{Sequence: []int{testASNA, testASNB}}},
|
||||||
@@ -792,8 +674,9 @@ func TestASHandlerHandlesMessagesWhileAWriteIsBlocked(t *testing.T) {
|
|||||||
|
|
||||||
store := newBlockingStore()
|
store := newBlockingStore()
|
||||||
h := newASHandler(store, logger.New(), longerThanAnyTest)
|
h := newASHandler(store, logger.New(), longerThanAnyTest)
|
||||||
|
fill := ristypes.ASPath{Sequence: slices.Repeat([]int{testASNA}, asnBatchSize)}
|
||||||
handleWhileAWriteIsBlocked(t, store, h,
|
handleWhileAWriteIsBlocked(t, store, h,
|
||||||
[]*ristypes.RISMessage{{Path: pathOfASes(asnBatchSize)}},
|
[]*ristypes.RISMessage{{Path: fill}},
|
||||||
&ristypes.RISMessage{Path: ristypes.ASPath{Sequence: []int{testASNB}}})
|
&ristypes.RISMessage{Path: ristypes.ASPath{Sequence: []int{testASNB}}})
|
||||||
|
|
||||||
store.mu.Lock()
|
store.mu.Lock()
|
||||||
@@ -831,11 +714,9 @@ func TestPrefixHandlerWaitsAtTheMaximumBatchSize(t *testing.T) {
|
|||||||
const extraPrefix = "198.51.100.0/24"
|
const extraPrefix = "198.51.100.0/24"
|
||||||
|
|
||||||
store := newBlockingStore()
|
store := newBlockingStore()
|
||||||
h := newPrefixHandler(store, logger.New(), routetable.New(routetable.Params{}),
|
h := NewPrefixHandler(store, logger.New(), routetable.New(routetable.Params{}))
|
||||||
longerThanAnyTest, maxTableChangesPerWrite)
|
|
||||||
h.batchMaxSize = prefixBatchSize
|
|
||||||
fill := slices.Repeat([]string{testV4Prefix}, prefixBatchSize)
|
fill := slices.Repeat([]string{testV4Prefix}, prefixBatchSize)
|
||||||
toMax := slices.Repeat([]string{testV4Prefix}, prefixBatchSize)
|
toMax := slices.Repeat([]string{testV4Prefix}, prefixBatchMaxSize)
|
||||||
handleAtMaxBatchSize(t, store, h,
|
handleAtMaxBatchSize(t, store, h,
|
||||||
[]*ristypes.RISMessage{prefixAnnouncement(fill)},
|
[]*ristypes.RISMessage{prefixAnnouncement(fill)},
|
||||||
[]*ristypes.RISMessage{prefixAnnouncement(toMax)},
|
[]*ristypes.RISMessage{prefixAnnouncement(toMax)},
|
||||||
@@ -850,48 +731,31 @@ func TestPrefixHandlerWaitsAtTheMaximumBatchSize(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// TestASHandlerNeverWaitsForAWrite starts a write that the store holds. Then it
|
func TestASHandlerWaitsAtTheMaximumBatchSize(t *testing.T) {
|
||||||
// hands the AS handler twenty messages whose paths each hold one AS
|
|
||||||
// asnBatchSize times. HandleMessage must return each time, and the batch must
|
|
||||||
// hold one entry, for that AS.
|
|
||||||
func TestASHandlerNeverWaitsForAWrite(t *testing.T) {
|
|
||||||
t.Parallel()
|
t.Parallel()
|
||||||
|
|
||||||
const messages = 20
|
|
||||||
|
|
||||||
store := newBlockingStore()
|
store := newBlockingStore()
|
||||||
h := newASHandler(store, logger.New(), longerThanAnyTest)
|
h := newASHandler(store, logger.New(), longerThanAnyTest)
|
||||||
|
fill := ristypes.ASPath{Sequence: slices.Repeat([]int{testASNA}, asnBatchSize)}
|
||||||
|
toMax := ristypes.ASPath{Sequence: slices.Repeat([]int{testASNA}, asnBatchMaxSize)}
|
||||||
|
handleAtMaxBatchSize(t, store, h,
|
||||||
|
[]*ristypes.RISMessage{{Path: fill}},
|
||||||
|
[]*ristypes.RISMessage{{Path: toMax}},
|
||||||
|
&ristypes.RISMessage{Path: ristypes.ASPath{Sequence: []int{testASNB}}})
|
||||||
|
|
||||||
defer stopInTime(t, h)
|
store.mu.Lock()
|
||||||
defer store.releaseWrites()
|
defer store.mu.Unlock()
|
||||||
|
|
||||||
startBlockedWrite(t, store, h,
|
if !store.asns[testASNB] {
|
||||||
[]*ristypes.RISMessage{{Path: pathOfASes(asnBatchSize)}})
|
t.Errorf("Stop did not write AS%d, handled after the batch reached its "+
|
||||||
|
"maximum size", testASNB)
|
||||||
oneAS := &ristypes.RISMessage{
|
|
||||||
Path: ristypes.ASPath{Sequence: slices.Repeat([]int{testASNB}, asnBatchSize)},
|
|
||||||
}
|
|
||||||
|
|
||||||
if !returnsInTime(func() {
|
|
||||||
for range messages {
|
|
||||||
h.HandleMessage(oneAS)
|
|
||||||
}
|
|
||||||
}) {
|
|
||||||
t.Fatal("HandleMessage did not return while a database write was blocked")
|
|
||||||
}
|
|
||||||
|
|
||||||
h.mu.Lock()
|
|
||||||
defer h.mu.Unlock()
|
|
||||||
|
|
||||||
if len(h.batch) != 1 {
|
|
||||||
t.Errorf("the batch holds %d entries for one AS, want 1", len(h.batch))
|
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// TestPeerHandlerWaitsAtTheMaximumBatchSize must not have its flush timer fire.
|
// TestPeerHandlerWaitsAtTheMaximumBatchSize must not have its flush timer fire.
|
||||||
// A timer flush during the fill would take only part of it. The batch would
|
// A timer flush during the fill would take only part of it. The batch would
|
||||||
// then reach its maximum size before the test's last message, and the test
|
// then reach its maximum size before the test's last message, and the test
|
||||||
// would fail.
|
// would hang.
|
||||||
func TestPeerHandlerWaitsAtTheMaximumBatchSize(t *testing.T) {
|
func TestPeerHandlerWaitsAtTheMaximumBatchSize(t *testing.T) {
|
||||||
t.Parallel()
|
t.Parallel()
|
||||||
|
|
||||||
@@ -899,11 +763,10 @@ func TestPeerHandlerWaitsAtTheMaximumBatchSize(t *testing.T) {
|
|||||||
|
|
||||||
store := newBlockingStore()
|
store := newBlockingStore()
|
||||||
h := newPeerHandler(store, logger.New(), longerThanAnyTest)
|
h := newPeerHandler(store, logger.New(), longerThanAnyTest)
|
||||||
h.batchMaxSize = peerBatchSize
|
|
||||||
message := []*ristypes.RISMessage{{Peer: testPeerIP}}
|
message := []*ristypes.RISMessage{{Peer: testPeerIP}}
|
||||||
handleAtMaxBatchSize(t, store, h,
|
handleAtMaxBatchSize(t, store, h,
|
||||||
slices.Repeat(message, peerBatchSize),
|
slices.Repeat(message, peerBatchSize),
|
||||||
slices.Repeat(message, peerBatchSize),
|
slices.Repeat(message, peerBatchMaxSize),
|
||||||
&ristypes.RISMessage{Peer: otherPeerIP})
|
&ristypes.RISMessage{Peer: otherPeerIP})
|
||||||
|
|
||||||
store.mu.Lock()
|
store.mu.Lock()
|
||||||
@@ -933,9 +796,8 @@ func TestHandlersWakeTheFlushGoroutineWhenTheBatchIsFull(t *testing.T) {
|
|||||||
batchSize: prefixBatchSize,
|
batchSize: prefixBatchSize,
|
||||||
handle: func(n int) chan struct{} {
|
handle: func(n int) chan struct{} {
|
||||||
h := &PrefixHandler{
|
h := &PrefixHandler{
|
||||||
batchMaxSize: prefixBatchMaxSize,
|
batchFullCh: make(chan struct{}, 1),
|
||||||
batchFullCh: make(chan struct{}, 1),
|
table: routetable.New(routetable.Params{}),
|
||||||
table: routetable.New(routetable.Params{}),
|
|
||||||
}
|
}
|
||||||
h.HandleMessage(prefixAnnouncement(
|
h.HandleMessage(prefixAnnouncement(
|
||||||
slices.Repeat([]string{testV4Prefix}, n)))
|
slices.Repeat([]string{testV4Prefix}, n)))
|
||||||
@@ -947,12 +809,11 @@ func TestHandlersWakeTheFlushGoroutineWhenTheBatchIsFull(t *testing.T) {
|
|||||||
name: "AS",
|
name: "AS",
|
||||||
batchSize: asnBatchSize,
|
batchSize: asnBatchSize,
|
||||||
handle: func(n int) chan struct{} {
|
handle: func(n int) chan struct{} {
|
||||||
h := &ASHandler{
|
h := &ASHandler{batchFullCh: make(chan struct{}, 1)}
|
||||||
batch: make(map[int]database.SeenTimes),
|
// The origin AS and each AS in the path are one update each.
|
||||||
batchFullCh: make(chan struct{}, 1),
|
h.HandleMessage(&ristypes.RISMessage{
|
||||||
}
|
Path: ristypes.ASPath{Sequence: slices.Repeat([]int{testASNA}, n-1)},
|
||||||
// Each of the path's n ASes is one entry of the batch.
|
})
|
||||||
h.HandleMessage(&ristypes.RISMessage{Path: pathOfASes(n)})
|
|
||||||
|
|
||||||
return h.batchFullCh
|
return h.batchFullCh
|
||||||
},
|
},
|
||||||
@@ -961,10 +822,7 @@ func TestHandlersWakeTheFlushGoroutineWhenTheBatchIsFull(t *testing.T) {
|
|||||||
name: "peer",
|
name: "peer",
|
||||||
batchSize: peerBatchSize,
|
batchSize: peerBatchSize,
|
||||||
handle: func(n int) chan struct{} {
|
handle: func(n int) chan struct{} {
|
||||||
h := &PeerHandler{
|
h := &PeerHandler{batchFullCh: make(chan struct{}, 1)}
|
||||||
batchMaxSize: peerBatchMaxSize,
|
|
||||||
batchFullCh: make(chan struct{}, 1),
|
|
||||||
}
|
|
||||||
for range n {
|
for range n {
|
||||||
h.HandleMessage(&ristypes.RISMessage{Peer: testPeerIP})
|
h.HandleMessage(&ristypes.RISMessage{Peer: testPeerIP})
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -25,10 +25,6 @@ const (
|
|||||||
// analyzeInterval is how often to run ANALYZE.
|
// analyzeInterval is how often to run ANALYZE.
|
||||||
analyzeInterval = 1 * time.Hour
|
analyzeInterval = 1 * time.Hour
|
||||||
|
|
||||||
// originChangesPruneInterval is how often to prune the origin_changes
|
|
||||||
// table.
|
|
||||||
originChangesPruneInterval = 1 * time.Hour
|
|
||||||
|
|
||||||
// checkpointTimeout is the max time for WAL checkpoint.
|
// checkpointTimeout is the max time for WAL checkpoint.
|
||||||
checkpointTimeout = 10 * time.Second
|
checkpointTimeout = 10 * time.Second
|
||||||
|
|
||||||
@@ -37,9 +33,6 @@ const (
|
|||||||
|
|
||||||
// analyzeTimeout is the max time for ANALYZE.
|
// analyzeTimeout is the max time for ANALYZE.
|
||||||
analyzeTimeout = 5 * time.Minute
|
analyzeTimeout = 5 * time.Minute
|
||||||
|
|
||||||
// originChangesPruneTimeout is the max time for the origin changes prune.
|
|
||||||
originChangesPruneTimeout = 5 * time.Minute
|
|
||||||
)
|
)
|
||||||
|
|
||||||
// checkpointIncompleteMessage is logged, at debug level like a completed
|
// checkpointIncompleteMessage is logged, at debug level like a completed
|
||||||
@@ -87,7 +80,6 @@ func (m *DBMaintainer) Start(ctx context.Context) {
|
|||||||
"checkpoint_interval", checkpointInterval,
|
"checkpoint_interval", checkpointInterval,
|
||||||
"vacuum_interval", vacuumInterval,
|
"vacuum_interval", vacuumInterval,
|
||||||
"analyze_interval", analyzeInterval,
|
"analyze_interval", analyzeInterval,
|
||||||
"origin_changes_prune_interval", originChangesPruneInterval,
|
|
||||||
)
|
)
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -119,9 +111,6 @@ func (m *DBMaintainer) run(ctx context.Context) {
|
|||||||
analyzeTimer := time.NewTimer(analyzeInterval)
|
analyzeTimer := time.NewTimer(analyzeInterval)
|
||||||
defer analyzeTimer.Stop()
|
defer analyzeTimer.Stop()
|
||||||
|
|
||||||
pruneTimer := time.NewTimer(originChangesPruneInterval)
|
|
||||||
defer pruneTimer.Stop()
|
|
||||||
|
|
||||||
for {
|
for {
|
||||||
select {
|
select {
|
||||||
case <-m.stopCh:
|
case <-m.stopCh:
|
||||||
@@ -138,10 +127,6 @@ func (m *DBMaintainer) run(ctx context.Context) {
|
|||||||
case <-analyzeTimer.C:
|
case <-analyzeTimer.C:
|
||||||
m.runAnalyze(ctx)
|
m.runAnalyze(ctx)
|
||||||
analyzeTimer.Reset(analyzeInterval)
|
analyzeTimer.Reset(analyzeInterval)
|
||||||
|
|
||||||
case <-pruneTimer.C:
|
|
||||||
m.runPruneOriginChanges(ctx)
|
|
||||||
pruneTimer.Reset(originChangesPruneInterval)
|
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -190,18 +175,6 @@ func (m *DBMaintainer) runAnalyze(ctx context.Context) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// runPruneOriginChanges prunes the origin_changes table.
|
|
||||||
func (m *DBMaintainer) runPruneOriginChanges(ctx context.Context) {
|
|
||||||
duration, err := runWithTimeout(ctx, originChangesPruneTimeout,
|
|
||||||
m.db.PruneOriginChanges)
|
|
||||||
if err != nil {
|
|
||||||
m.logger.Error("Origin changes prune failed", "error", err,
|
|
||||||
"duration", duration)
|
|
||||||
} else {
|
|
||||||
m.logger.Debug("Origin changes pruned", "duration", duration)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// runWithTimeout runs operation with a context that ends after timeout and
|
// runWithTimeout runs operation with a context that ends after timeout and
|
||||||
// returns how long it took and its error.
|
// returns how long it took and its error.
|
||||||
func runWithTimeout(
|
func runWithTimeout(
|
||||||
|
|||||||
@@ -97,7 +97,7 @@ func TestDBMaintainerStopReturnsByItsDeadline(t *testing.T) {
|
|||||||
|
|
||||||
select {
|
select {
|
||||||
case <-store.started:
|
case <-store.started:
|
||||||
case <-time.After(stepTimeout):
|
case <-time.After(handleMessageTimeout):
|
||||||
t.Fatal("the maintainer did not start the checkpoint")
|
t.Fatal("the maintainer did not start the checkpoint")
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -113,7 +113,7 @@ func TestDBMaintainerStopReturnsByItsDeadline(t *testing.T) {
|
|||||||
|
|
||||||
select {
|
select {
|
||||||
case <-m.done:
|
case <-m.done:
|
||||||
case <-time.After(stepTimeout):
|
case <-time.After(handleMessageTimeout):
|
||||||
t.Fatal("the maintenance goroutine did not return after the " +
|
t.Fatal("the maintenance goroutine did not return after the " +
|
||||||
"checkpoint ended")
|
"checkpoint ended")
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -43,14 +43,10 @@ type PeerHandler struct {
|
|||||||
db database.Store
|
db database.Store
|
||||||
logger *logger.Logger
|
logger *logger.Logger
|
||||||
|
|
||||||
// batchMaxSize is peerBatchMaxSize. Only HandleMessage reads it, so a test
|
|
||||||
// may lower it before it hands the handler a message.
|
|
||||||
batchMaxSize int
|
|
||||||
|
|
||||||
// Batching. mu guards peerBatch and writing only: the flush goroutine
|
// Batching. mu guards peerBatch and writing only: the flush goroutine
|
||||||
// takes the batch under mu and writes it after releasing mu, so
|
// takes the batch under mu and writes it after releasing mu, so
|
||||||
// HandleMessage waits for a database write only while peerBatch is at
|
// HandleMessage waits for a database write only while peerBatch is at
|
||||||
// batchMaxSize.
|
// peerBatchMaxSize.
|
||||||
mu sync.Mutex
|
mu sync.Mutex
|
||||||
peerBatch []peerUpdate
|
peerBatch []peerUpdate
|
||||||
// writing is how many updates the flush goroutine took from peerBatch
|
// writing is how many updates the flush goroutine took from peerBatch
|
||||||
@@ -88,13 +84,12 @@ func newPeerHandler(
|
|||||||
db database.Store, logger *logger.Logger, batchTimeout time.Duration,
|
db database.Store, logger *logger.Logger, batchTimeout time.Duration,
|
||||||
) *PeerHandler {
|
) *PeerHandler {
|
||||||
h := &PeerHandler{
|
h := &PeerHandler{
|
||||||
db: db,
|
db: db,
|
||||||
logger: logger,
|
logger: logger,
|
||||||
batchMaxSize: peerBatchMaxSize,
|
peerBatch: make([]peerUpdate, 0, peerBatchSize),
|
||||||
peerBatch: make([]peerUpdate, 0, peerBatchSize),
|
batchFullCh: make(chan struct{}, 1),
|
||||||
batchFullCh: make(chan struct{}, 1),
|
stopCh: make(chan context.Context, 1),
|
||||||
stopCh: make(chan context.Context, 1),
|
done: make(chan struct{}),
|
||||||
done: make(chan struct{}),
|
|
||||||
}
|
}
|
||||||
h.batchTaken.L = &h.mu
|
h.batchTaken.L = &h.mu
|
||||||
|
|
||||||
@@ -140,7 +135,7 @@ func (h *PeerHandler) HandleMessage(msg *ristypes.RISMessage) {
|
|||||||
// While the batch is at its maximum size, wait until the flush goroutine
|
// While the batch is at its maximum size, wait until the flush goroutine
|
||||||
// takes it. Meanwhile the handler takes nothing off its queue, which
|
// takes it. Meanwhile the handler takes nothing off its queue, which
|
||||||
// fills, and the streamer drops messages for it.
|
// fills, and the streamer drops messages for it.
|
||||||
for len(h.peerBatch) >= h.batchMaxSize {
|
for len(h.peerBatch) >= peerBatchMaxSize {
|
||||||
h.batchTaken.Wait()
|
h.batchTaken.Wait()
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -154,7 +149,7 @@ func (h *PeerHandler) HandleMessage(msg *ristypes.RISMessage) {
|
|||||||
|
|
||||||
// A full batch wakes the flush goroutine. If it is writing, the wake-up
|
// A full batch wakes the flush goroutine. If it is writing, the wake-up
|
||||||
// waits in batchFullCh and the batch keeps growing until the write ends
|
// waits in batchFullCh and the batch keeps growing until the write ends
|
||||||
// or the batch reaches batchMaxSize.
|
// or the batch reaches peerBatchMaxSize.
|
||||||
if len(h.peerBatch) >= peerBatchSize {
|
if len(h.peerBatch) >= peerBatchSize {
|
||||||
select {
|
select {
|
||||||
case h.batchFullCh <- struct{}{}:
|
case h.batchFullCh <- struct{}{}:
|
||||||
|
|||||||
@@ -35,7 +35,7 @@ func TestPeerHandlerWritesTheEarliestAndLatestTimesOfANewPeer(t *testing.T) {
|
|||||||
|
|
||||||
// Built without NewPeerHandler's flush goroutine, so the test flushes
|
// Built without NewPeerHandler's flush goroutine, so the test flushes
|
||||||
// itself.
|
// itself.
|
||||||
h := &PeerHandler{db: store, logger: logger.New(), batchMaxSize: peerBatchMaxSize}
|
h := &PeerHandler{db: store, logger: logger.New()}
|
||||||
|
|
||||||
first := peerAnnouncement(testPeerIP, testV4Prefix)
|
first := peerAnnouncement(testPeerIP, testV4Prefix)
|
||||||
second := peerAnnouncement(testPeerIP, testV4Prefix)
|
second := peerAnnouncement(testPeerIP, testV4Prefix)
|
||||||
|
|||||||
@@ -42,10 +42,6 @@ type PeeringHandler struct {
|
|||||||
db database.Store
|
db database.Store
|
||||||
logger *logger.Logger
|
logger *logger.Logger
|
||||||
|
|
||||||
// maxPaths is the most paths asPaths holds: maxTrackedPaths, except in
|
|
||||||
// tests that give a smaller one.
|
|
||||||
maxPaths int
|
|
||||||
|
|
||||||
// In-memory AS path tracking: each path, JSON-encoded, with the earliest
|
// In-memory AS path tracking: each path, JSON-encoded, with the earliest
|
||||||
// and the latest time an update carried it.
|
// and the latest time an update carried it.
|
||||||
mu sync.Mutex
|
mu sync.Mutex
|
||||||
@@ -66,24 +62,21 @@ type PeeringHandler struct {
|
|||||||
// periodically processes accumulated AS paths into peering records. The
|
// periodically processes accumulated AS paths into peering records. The
|
||||||
// handler begins processing immediately upon creation.
|
// handler begins processing immediately upon creation.
|
||||||
func NewPeeringHandler(db database.Store, logger *logger.Logger) *PeeringHandler {
|
func NewPeeringHandler(db database.Store, logger *logger.Logger) *PeeringHandler {
|
||||||
return newPeeringHandler(db, logger, peeringProcessInterval, maxTrackedPaths)
|
return newPeeringHandler(db, logger, peeringProcessInterval)
|
||||||
}
|
}
|
||||||
|
|
||||||
// newPeeringHandler is NewPeeringHandler with the processing timer's interval,
|
// newPeeringHandler is NewPeeringHandler with the processing timer's interval,
|
||||||
// processInterval, in place of peeringProcessInterval, and the most paths it
|
// processInterval, in place of peeringProcessInterval. A test gives a short
|
||||||
// holds, maxPaths, in place of maxTrackedPaths. A test gives a short interval to
|
// one to have the timer start a write.
|
||||||
// have the timer start a write, and a small maxPaths to fill the map quickly.
|
|
||||||
func newPeeringHandler(
|
func newPeeringHandler(
|
||||||
db database.Store, logger *logger.Logger, processInterval time.Duration,
|
db database.Store, logger *logger.Logger, processInterval time.Duration,
|
||||||
maxPaths int,
|
|
||||||
) *PeeringHandler {
|
) *PeeringHandler {
|
||||||
h := &PeeringHandler{
|
h := &PeeringHandler{
|
||||||
db: db,
|
db: db,
|
||||||
logger: logger,
|
logger: logger,
|
||||||
asPaths: make(map[string]database.SeenTimes),
|
asPaths: make(map[string]database.SeenTimes),
|
||||||
maxPaths: maxPaths,
|
stopCh: make(chan context.Context, 1),
|
||||||
stopCh: make(chan context.Context, 1),
|
done: make(chan struct{}),
|
||||||
done: make(chan struct{}),
|
|
||||||
}
|
}
|
||||||
|
|
||||||
// Start the periodic processing goroutine
|
// Start the periodic processing goroutine
|
||||||
@@ -138,7 +131,7 @@ func (h *PeeringHandler) HandleMessage(msg *ristypes.RISMessage) {
|
|||||||
defer h.mu.Unlock()
|
defer h.mu.Unlock()
|
||||||
|
|
||||||
seen, exists := h.asPaths[key]
|
seen, exists := h.asPaths[key]
|
||||||
if !exists && len(h.asPaths) >= h.maxPaths {
|
if !exists && len(h.asPaths) >= maxTrackedPaths {
|
||||||
// Map is full; drop this new path and count it.
|
// Map is full; drop this new path and count it.
|
||||||
h.droppedPaths++
|
h.droppedPaths++
|
||||||
|
|
||||||
|
|||||||
@@ -5,6 +5,7 @@ import (
|
|||||||
"fmt"
|
"fmt"
|
||||||
"maps"
|
"maps"
|
||||||
"slices"
|
"slices"
|
||||||
|
"strconv"
|
||||||
"sync"
|
"sync"
|
||||||
"testing"
|
"testing"
|
||||||
"time"
|
"time"
|
||||||
@@ -19,10 +20,6 @@ const (
|
|||||||
testASNB = 64501
|
testASNB = 64501
|
||||||
testASNC = 64502
|
testASNC = 64502
|
||||||
testASND = 64503
|
testASND = 64503
|
||||||
|
|
||||||
// testMaxTrackedPaths is a limit of the path map that a test fills
|
|
||||||
// quickly, in place of maxTrackedPaths.
|
|
||||||
testMaxTrackedPaths = 10
|
|
||||||
)
|
)
|
||||||
|
|
||||||
// recordingStore wraps mockStore to count every RecordPeering call, so a
|
// recordingStore wraps mockStore to count every RecordPeering call, so a
|
||||||
@@ -55,10 +52,9 @@ func (r *recordingStore) callCount() int {
|
|||||||
// processing goroutine, so tests drive processing explicitly.
|
// processing goroutine, so tests drive processing explicitly.
|
||||||
func newTestHandler(db database.Store) *PeeringHandler {
|
func newTestHandler(db database.Store) *PeeringHandler {
|
||||||
return &PeeringHandler{
|
return &PeeringHandler{
|
||||||
db: db,
|
db: db,
|
||||||
logger: logger.New(),
|
logger: logger.New(),
|
||||||
asPaths: make(map[string]database.SeenTimes),
|
asPaths: make(map[string]database.SeenTimes),
|
||||||
maxPaths: maxTrackedPaths,
|
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -248,28 +244,25 @@ func TestPeeringHandlerStopWritesTheRemainingPaths(t *testing.T) {
|
|||||||
|
|
||||||
// TestPeeringHandlerCapDropsAndCounts verifies that a full map drops new paths
|
// TestPeeringHandlerCapDropsAndCounts verifies that a full map drops new paths
|
||||||
// and counts them, while a path already tracked is refreshed rather than
|
// and counts them, while a path already tracked is refreshed rather than
|
||||||
// dropped. The handler's limit is small, and its timer does not fire.
|
// dropped.
|
||||||
func TestPeeringHandlerCapDropsAndCounts(t *testing.T) {
|
func TestPeeringHandlerCapDropsAndCounts(t *testing.T) {
|
||||||
t.Parallel()
|
t.Parallel()
|
||||||
|
|
||||||
store := &recordingStore{mockStore: newMockStore()}
|
store := &recordingStore{mockStore: newMockStore()}
|
||||||
h := newPeeringHandler(store, logger.New(), longerThanAnyTest,
|
h := newTestHandler(store)
|
||||||
testMaxTrackedPaths)
|
|
||||||
|
|
||||||
defer stopInTime(t, h)
|
|
||||||
|
|
||||||
now := time.Now().UTC()
|
now := time.Now().UTC()
|
||||||
|
|
||||||
// Fill the map to exactly its limit, including one real path key so the
|
// Fill the map to exactly maxTrackedPaths, including one real path key so
|
||||||
// "already tracked" branch can be exercised. The other paths are of one
|
// the "already tracked" branch can be exercised. The filler keys are never
|
||||||
// AS, so they carry no peering.
|
// processed in this test, so their contents do not matter.
|
||||||
existing := pathKey(t, testASNA, testASNB)
|
existing := pathKey(t, testASNA, testASNB)
|
||||||
|
|
||||||
h.mu.Lock()
|
h.mu.Lock()
|
||||||
h.asPaths[existing] = database.SeenTimes{First: now, Last: now}
|
h.asPaths[existing] = database.SeenTimes{First: now, Last: now}
|
||||||
|
|
||||||
for asn := 1; len(h.asPaths) < testMaxTrackedPaths; asn++ {
|
for i := 0; len(h.asPaths) < maxTrackedPaths; i++ {
|
||||||
h.asPaths[pathKey(t, asn)] = database.SeenTimes{First: now, Last: now}
|
h.asPaths[strconv.Itoa(i)] = database.SeenTimes{First: now, Last: now}
|
||||||
}
|
}
|
||||||
h.mu.Unlock()
|
h.mu.Unlock()
|
||||||
|
|
||||||
@@ -277,9 +270,8 @@ func TestPeeringHandlerCapDropsAndCounts(t *testing.T) {
|
|||||||
handle(h, now.Add(time.Second), testASNA, testASNC)
|
handle(h, now.Add(time.Second), testASNA, testASNC)
|
||||||
|
|
||||||
tracked, dropped := snapshot(h)
|
tracked, dropped := snapshot(h)
|
||||||
if tracked != testMaxTrackedPaths {
|
if tracked != maxTrackedPaths {
|
||||||
t.Fatalf("want map size %d after drop, got %d", testMaxTrackedPaths,
|
t.Fatalf("want map size %d after drop, got %d", maxTrackedPaths, tracked)
|
||||||
tracked)
|
|
||||||
}
|
}
|
||||||
|
|
||||||
if dropped != 1 {
|
if dropped != 1 {
|
||||||
@@ -291,9 +283,8 @@ func TestPeeringHandlerCapDropsAndCounts(t *testing.T) {
|
|||||||
handle(h, refreshed, testASNA, testASNB)
|
handle(h, refreshed, testASNA, testASNB)
|
||||||
|
|
||||||
tracked, dropped = snapshot(h)
|
tracked, dropped = snapshot(h)
|
||||||
if tracked != testMaxTrackedPaths {
|
if tracked != maxTrackedPaths {
|
||||||
t.Fatalf("want map size %d after refresh, got %d", testMaxTrackedPaths,
|
t.Fatalf("want map size %d after refresh, got %d", maxTrackedPaths, tracked)
|
||||||
tracked)
|
|
||||||
}
|
}
|
||||||
|
|
||||||
if dropped != 1 {
|
if dropped != 1 {
|
||||||
@@ -312,21 +303,17 @@ func TestPeeringHandlerCapDropsAndCounts(t *testing.T) {
|
|||||||
|
|
||||||
// TestPeeringHandlerRunResetsTheDroppedCount fills the map, has the handler
|
// TestPeeringHandlerRunResetsTheDroppedCount fills the map, has the handler
|
||||||
// drop one path, and runs the processing. The count of dropped paths is the
|
// drop one path, and runs the processing. The count of dropped paths is the
|
||||||
// run's own, so it is 0 again after the run. The handler's limit is small, and
|
// run's own, so it is 0 again after the run.
|
||||||
// its timer does not fire.
|
|
||||||
func TestPeeringHandlerRunResetsTheDroppedCount(t *testing.T) {
|
func TestPeeringHandlerRunResetsTheDroppedCount(t *testing.T) {
|
||||||
t.Parallel()
|
t.Parallel()
|
||||||
|
|
||||||
h := newPeeringHandler(newMockStore(), logger.New(), longerThanAnyTest,
|
h := newTestHandler(newMockStore())
|
||||||
testMaxTrackedPaths)
|
|
||||||
|
|
||||||
defer stopInTime(t, h)
|
|
||||||
|
|
||||||
now := time.Now().UTC()
|
now := time.Now().UTC()
|
||||||
|
|
||||||
// Paths of one AS carry no peering, so the run records nothing.
|
// Paths of one AS carry no peering, so the run records nothing.
|
||||||
h.mu.Lock()
|
h.mu.Lock()
|
||||||
for asn := 1; len(h.asPaths) < testMaxTrackedPaths; asn++ {
|
for asn := 1; len(h.asPaths) < maxTrackedPaths; asn++ {
|
||||||
h.asPaths[pathKey(t, asn)] = database.SeenTimes{First: now, Last: now}
|
h.asPaths[pathKey(t, asn)] = database.SeenTimes{First: now, Last: now}
|
||||||
}
|
}
|
||||||
h.mu.Unlock()
|
h.mu.Unlock()
|
||||||
|
|||||||
@@ -15,15 +15,12 @@ import (
|
|||||||
)
|
)
|
||||||
|
|
||||||
const (
|
const (
|
||||||
// prefixHandlerQueueSize is the most messages the handler's queue holds.
|
// prefixHandlerQueueSize is the queue capacity for prefix tracking
|
||||||
// While the routing table's Overview or PeersDown walks every prefix, it
|
// operations, about 4 seconds of feed at peak. The streamer drops updates
|
||||||
// holds the table's lock about half of the time. The handler then applies
|
// rather than blocks when a queue is full (a STATE message is held until
|
||||||
// updates more slowly. When the feed pauses for some seconds and then
|
// there is room), so this bounds memory. Batches still flush on a timer
|
||||||
// catches up, its backlog can arrive faster than that, and the queue
|
// (prefixBatchTimeout).
|
||||||
// holds the backlog. The streamer drops an update that finds the queue
|
prefixHandlerQueueSize = 20000
|
||||||
// full. A message takes about 1 KiB, so a full queue takes about 100 MiB
|
|
||||||
// (README "Memory").
|
|
||||||
prefixHandlerQueueSize = 100000
|
|
||||||
|
|
||||||
// prefixBatchSize is the number of prefix updates to batch together
|
// prefixBatchSize is the number of prefix updates to batch together
|
||||||
prefixBatchSize = 25000
|
prefixBatchSize = 25000
|
||||||
@@ -47,25 +44,6 @@ const (
|
|||||||
// most one pair. So the million pairs that a full-table peer's session
|
// most one pair. So the million pairs that a full-table peer's session
|
||||||
// going down changes are written in about a minute.
|
// going down changes are written in about a minute.
|
||||||
maxTableChangesPerWrite = 20000
|
maxTableChangesPerWrite = 20000
|
||||||
|
|
||||||
// maxOriginChangesPerWrite is the most origin changes of the routing
|
|
||||||
// table that one flush writes; the rest wait for the next flushes.
|
|
||||||
maxOriginChangesPerWrite = 20000
|
|
||||||
|
|
||||||
// maxHeldBackMessages is the most messages the handler holds back from
|
|
||||||
// peers whose session went down. A peer's messages are held back from
|
|
||||||
// the moment its session goes down until peerDown has applied them. If
|
|
||||||
// peerDown is running at that moment, the peer waits for the rest of
|
|
||||||
// peerDown's current PeersDown call. The peer also waits while peerDown
|
|
||||||
// applies the messages held back from that call's peers, those that
|
|
||||||
// arrive meanwhile included. The next PeersDown call takes the peer out.
|
|
||||||
// Once that call returns, peerDown applies the messages held back from
|
|
||||||
// that call's peers, one peer after another, in the order the call lists
|
|
||||||
// them. A held-back message that says the peer's session went down again
|
|
||||||
// makes the peer's later messages wait for the next call. A message takes
|
|
||||||
// about 1 KiB, so 150,000 take about 150 MiB (README "Memory"). At this
|
|
||||||
// number HandleMessage waits until one of them is applied.
|
|
||||||
maxHeldBackMessages = 150000
|
|
||||||
)
|
)
|
||||||
|
|
||||||
// PrefixHandler keeps the routing table in memory current from the feed. It
|
// PrefixHandler keeps the routing table in memory current from the feed. It
|
||||||
@@ -74,39 +52,18 @@ const (
|
|||||||
// does an announcement whose AS path is empty or holds AS 0 anywhere. A peer
|
// does an announcement whose AS path is empty or holds AS 0 anywhere. A peer
|
||||||
// whose BGP session goes down leaves every prefix.
|
// whose BGP session goes down leaves every prefix.
|
||||||
//
|
//
|
||||||
// The table's PeersDown applies sessions going down. It visits every prefix,
|
// Each flush writes the table's changes to the database. It also stores each
|
||||||
// which takes seconds on a full table. So the handler runs PeersDown on a
|
// prefix the updates name with the time it was first seen.
|
||||||
// separate goroutine and goes on taking messages meanwhile. One PeersDown runs
|
|
||||||
// at a time. The next one takes every peer whose session went down since the
|
|
||||||
// previous one began. PeersDown lets the table's lock go about half of the
|
|
||||||
// time, so the handler applies other peers' messages while PeersDown runs. The
|
|
||||||
// handler holds back a down peer's later messages until peerDown applies them,
|
|
||||||
// as the doc of maxHeldBackMessages describes. It applies them in order, one at
|
|
||||||
// a time, and takes new messages in between. It holds back at most
|
|
||||||
// maxHeldBackMessages messages.
|
|
||||||
//
|
|
||||||
// Each flush writes the table's changes and origin changes to the database.
|
|
||||||
// It also stores each prefix the updates name with the time it was first seen.
|
|
||||||
type PrefixHandler struct {
|
type PrefixHandler struct {
|
||||||
db database.Store
|
db database.Store
|
||||||
logger *logger.Logger
|
logger *logger.Logger
|
||||||
metrics *metrics.Tracker
|
metrics *metrics.Tracker
|
||||||
table *routetable.Table
|
table *routetable.Table
|
||||||
|
|
||||||
// batchMaxSize is prefixBatchMaxSize. Only HandleMessage reads it, so a
|
// Batching. mu guards batch, writing and writingTableChanges only: the
|
||||||
// test may lower it before it hands the handler a message.
|
// flush goroutine takes them under mu and writes them after releasing
|
||||||
batchMaxSize int
|
// mu, so HandleMessage waits for a database write only while batch is at
|
||||||
// tableChangesPerWrite is maxTableChangesPerWrite, except in tests that
|
// prefixBatchMaxSize.
|
||||||
// give a smaller one.
|
|
||||||
tableChangesPerWrite int
|
|
||||||
// originChangesPerWrite is maxOriginChangesPerWrite, except in tests that
|
|
||||||
// give a smaller one.
|
|
||||||
originChangesPerWrite int
|
|
||||||
|
|
||||||
// Batching. mu guards batch, writing, writingTableChanges and
|
|
||||||
// writingOriginChanges: the flush goroutine takes them under mu and
|
|
||||||
// writes them after releasing mu, so HandleMessage waits for a database
|
|
||||||
// write only while batch is at batchMaxSize.
|
|
||||||
mu sync.Mutex
|
mu sync.Mutex
|
||||||
batch []seenPrefix
|
batch []seenPrefix
|
||||||
// writing is how many updates the flush goroutine took and has not
|
// writing is how many updates the flush goroutine took and has not
|
||||||
@@ -115,9 +72,6 @@ type PrefixHandler struct {
|
|||||||
// writingTableChanges is how many changes the flush goroutine took from
|
// writingTableChanges is how many changes the flush goroutine took from
|
||||||
// the routing table and has not finished writing.
|
// the routing table and has not finished writing.
|
||||||
writingTableChanges int
|
writingTableChanges int
|
||||||
// writingOriginChanges is how many origin changes the flush goroutine
|
|
||||||
// took from the routing table and has not finished writing.
|
|
||||||
writingOriginChanges int
|
|
||||||
// batchTaken is signalled when the flush goroutine takes batch; its L is
|
// batchTaken is signalled when the flush goroutine takes batch; its L is
|
||||||
// &mu.
|
// &mu.
|
||||||
batchTaken sync.Cond
|
batchTaken sync.Cond
|
||||||
@@ -126,26 +80,6 @@ type PrefixHandler struct {
|
|||||||
// Stop does not wait for a write in progress to hand it over.
|
// Stop does not wait for a write in progress to hand it over.
|
||||||
stopCh chan context.Context
|
stopCh chan context.Context
|
||||||
done chan struct{} // closed when the flush goroutine returns
|
done chan struct{} // closed when the flush goroutine returns
|
||||||
|
|
||||||
// Peers whose session went down. heldBack, under mu, holds each such
|
|
||||||
// peer's later messages, in order, until the peer has left every prefix
|
|
||||||
// and they are applied. A peer with no such messages yet has an empty
|
|
||||||
// entry. heldCount, under mu, is how many messages heldBack holds.
|
|
||||||
// heldTaken is signalled each time a held-back message is taken out to
|
|
||||||
// be applied; its L is &mu.
|
|
||||||
heldBack map[string][]*ristypes.RISMessage
|
|
||||||
heldCount int
|
|
||||||
heldTaken sync.Cond
|
|
||||||
// downPeers, under mu, lists the peers whose session went down and that
|
|
||||||
// the next PeersDown takes. peerDownRunning, under mu, says whether
|
|
||||||
// peerDown is running. peerDowns counts the peerDown goroutines that have
|
|
||||||
// not returned.
|
|
||||||
downPeers []string
|
|
||||||
peerDownRunning bool
|
|
||||||
peerDowns sync.WaitGroup
|
|
||||||
// tablePeersDown is the table's PeersDown. Only peerDown calls it, so a
|
|
||||||
// test may replace it before it hands the handler a message.
|
|
||||||
tablePeersDown func(peers []string)
|
|
||||||
}
|
}
|
||||||
|
|
||||||
// seenPrefix is what the batch keeps of an update: the prefix it names and its
|
// seenPrefix is what the batch keeps of an update: the prefix it names and its
|
||||||
@@ -185,36 +119,26 @@ func (u prefixUpdate) announcesRoute() bool {
|
|||||||
func NewPrefixHandler(
|
func NewPrefixHandler(
|
||||||
db database.Store, logger *logger.Logger, table *routetable.Table,
|
db database.Store, logger *logger.Logger, table *routetable.Table,
|
||||||
) *PrefixHandler {
|
) *PrefixHandler {
|
||||||
return newPrefixHandler(db, logger, table, prefixBatchTimeout,
|
return newPrefixHandler(db, logger, table, prefixBatchTimeout)
|
||||||
maxTableChangesPerWrite)
|
|
||||||
}
|
}
|
||||||
|
|
||||||
// newPrefixHandler is NewPrefixHandler with the flush timer's interval,
|
// newPrefixHandler is NewPrefixHandler with the flush timer's interval,
|
||||||
// batchTimeout, in place of prefixBatchTimeout, and tableChangesPerWrite in
|
// batchTimeout, in place of prefixBatchTimeout. A test that the timer must not
|
||||||
// place of maxTableChangesPerWrite. A test that the timer must not disturb
|
// disturb gives an interval longer than the test runs.
|
||||||
// gives an interval longer than the test runs. A test of writes split over
|
|
||||||
// several flushes gives a small tableChangesPerWrite, so that it exceeds it
|
|
||||||
// with few changes.
|
|
||||||
func newPrefixHandler(
|
func newPrefixHandler(
|
||||||
db database.Store, logger *logger.Logger, table *routetable.Table,
|
db database.Store, logger *logger.Logger, table *routetable.Table,
|
||||||
batchTimeout time.Duration, tableChangesPerWrite int,
|
batchTimeout time.Duration,
|
||||||
) *PrefixHandler {
|
) *PrefixHandler {
|
||||||
h := &PrefixHandler{
|
h := &PrefixHandler{
|
||||||
db: db,
|
db: db,
|
||||||
logger: logger,
|
logger: logger,
|
||||||
table: table,
|
table: table,
|
||||||
batch: make([]seenPrefix, 0, prefixBatchSize),
|
batch: make([]seenPrefix, 0, prefixBatchSize),
|
||||||
batchMaxSize: prefixBatchMaxSize,
|
batchFullCh: make(chan struct{}, 1),
|
||||||
tableChangesPerWrite: tableChangesPerWrite,
|
stopCh: make(chan context.Context, 1),
|
||||||
originChangesPerWrite: maxOriginChangesPerWrite,
|
done: make(chan struct{}),
|
||||||
batchFullCh: make(chan struct{}, 1),
|
|
||||||
stopCh: make(chan context.Context, 1),
|
|
||||||
done: make(chan struct{}),
|
|
||||||
heldBack: make(map[string][]*ristypes.RISMessage),
|
|
||||||
tablePeersDown: table.PeersDown,
|
|
||||||
}
|
}
|
||||||
h.batchTaken.L = &h.mu
|
h.batchTaken.L = &h.mu
|
||||||
h.heldTaken.L = &h.mu
|
|
||||||
|
|
||||||
// Start the flush timer goroutine
|
// Start the flush timer goroutine
|
||||||
go flushLoop(batchTimeout, h.batchFullCh, h.stopCh, h.done, h.flushBatch)
|
go flushLoop(batchTimeout, h.batchFullCh, h.stopCh, h.done, h.flushBatch)
|
||||||
@@ -242,102 +166,11 @@ func (h *PrefixHandler) QueueCapacity() int {
|
|||||||
return prefixHandlerQueueSize
|
return prefixHandlerQueueSize
|
||||||
}
|
}
|
||||||
|
|
||||||
// HandleMessage processes a message to track prefix information. A message
|
// HandleMessage processes a message to track prefix information
|
||||||
// from a peer whose session went down is held back until peerDown applies it.
|
|
||||||
func (h *PrefixHandler) HandleMessage(msg *ristypes.RISMessage) {
|
func (h *PrefixHandler) HandleMessage(msg *ristypes.RISMessage) {
|
||||||
h.mu.Lock()
|
|
||||||
defer h.mu.Unlock()
|
|
||||||
|
|
||||||
// While maxHeldBackMessages messages are held back, wait until one of them
|
|
||||||
// is taken out to be applied. Meanwhile the handler takes nothing off its
|
|
||||||
// queue, as at batchMaxSize.
|
|
||||||
for h.heldCount >= maxHeldBackMessages {
|
|
||||||
h.heldTaken.Wait()
|
|
||||||
}
|
|
||||||
|
|
||||||
held, goingDown := h.heldBack[msg.Peer]
|
|
||||||
if goingDown {
|
|
||||||
h.heldBack[msg.Peer] = append(held, msg)
|
|
||||||
h.heldCount++
|
|
||||||
|
|
||||||
return
|
|
||||||
}
|
|
||||||
|
|
||||||
h.applyMessage(msg)
|
|
||||||
}
|
|
||||||
|
|
||||||
// HeldBackCount returns how many messages the handler holds back from peers
|
|
||||||
// whose session went down. The status page shows it.
|
|
||||||
func (h *PrefixHandler) HeldBackCount() int {
|
|
||||||
h.mu.Lock()
|
|
||||||
defer h.mu.Unlock()
|
|
||||||
|
|
||||||
return h.heldCount
|
|
||||||
}
|
|
||||||
|
|
||||||
// Stop waits for every PeersDown the handler started and for the messages held
|
|
||||||
// back meanwhile to be applied. It gives up this wait once ctx ends. Then Stop
|
|
||||||
// stops the flush goroutine and waits for it until ctx ends. The flush
|
|
||||||
// goroutine makes one last flush before it returns. That flush writes at most
|
|
||||||
// tableChangesPerWrite of the routing table's changes, originChangesPerWrite of
|
|
||||||
// its origin changes, and the prefixes of the updates the handler holds. Once
|
|
||||||
// ctx has ended the flush goroutine skips that last flush, and a write still
|
|
||||||
// running is lost when the process exits. A failed write of the table's
|
|
||||||
// changes is not retried either. Stop logs how many updates, table changes and
|
|
||||||
// origin changes are not written. It also logs how many held-back messages are
|
|
||||||
// not applied. The caller calls Stop once its last HandleMessage has returned.
|
|
||||||
func (h *PrefixHandler) Stop(ctx context.Context) {
|
|
||||||
peerDownsDone := make(chan struct{})
|
|
||||||
|
|
||||||
go func() {
|
|
||||||
h.peerDowns.Wait()
|
|
||||||
close(peerDownsDone)
|
|
||||||
}()
|
|
||||||
|
|
||||||
select {
|
|
||||||
case <-peerDownsDone:
|
|
||||||
case <-ctx.Done():
|
|
||||||
}
|
|
||||||
|
|
||||||
h.stopCh <- ctx
|
|
||||||
|
|
||||||
select {
|
|
||||||
case <-h.done:
|
|
||||||
case <-ctx.Done():
|
|
||||||
}
|
|
||||||
|
|
||||||
h.mu.Lock()
|
|
||||||
unwritten := h.writing + len(h.batch)
|
|
||||||
tableChanges := h.writingTableChanges + h.table.ChangeCount()
|
|
||||||
heldMessages := h.heldCount
|
|
||||||
originChanges := h.writingOriginChanges + h.table.OriginChangeCount()
|
|
||||||
h.mu.Unlock()
|
|
||||||
|
|
||||||
if unwritten == 0 && tableChanges == 0 && heldMessages == 0 &&
|
|
||||||
originChanges == 0 {
|
|
||||||
return
|
|
||||||
}
|
|
||||||
|
|
||||||
if ctx.Err() != nil {
|
|
||||||
h.logger.Error("Stop deadline passed before the prefix handler wrote "+
|
|
||||||
"its updates", "updates", unwritten, "table_changes", tableChanges,
|
|
||||||
"held_messages", heldMessages, "origin_changes", originChanges)
|
|
||||||
|
|
||||||
return
|
|
||||||
}
|
|
||||||
|
|
||||||
h.logger.Error("The prefix handler stopped before it wrote everything",
|
|
||||||
"updates", unwritten, "table_changes", tableChanges,
|
|
||||||
"held_messages", heldMessages, "origin_changes", originChanges)
|
|
||||||
}
|
|
||||||
|
|
||||||
// applyMessage applies msg. The caller holds mu. A peer's session going down
|
|
||||||
// has the peer leave every prefix. An update is applied to the routing table,
|
|
||||||
// and its prefixes join the batch.
|
|
||||||
func (h *PrefixHandler) applyMessage(msg *ristypes.RISMessage) {
|
|
||||||
if msg.Type == ristypes.MessageTypeState {
|
if msg.Type == ristypes.MessageTypeState {
|
||||||
if msg.State == ristypes.StateDown {
|
if msg.State == ristypes.StateDown {
|
||||||
h.startPeerDown(msg.Peer)
|
h.table.PeerDown(msg.Peer)
|
||||||
}
|
}
|
||||||
|
|
||||||
return
|
return
|
||||||
@@ -352,10 +185,13 @@ func (h *PrefixHandler) applyMessage(msg *ristypes.RISMessage) {
|
|||||||
// the path after the rest of it.
|
// the path after the rest of it.
|
||||||
path := slices.Concat(msg.Path.Sequence, msg.Path.Set)
|
path := slices.Concat(msg.Path.Sequence, msg.Path.Set)
|
||||||
|
|
||||||
|
h.mu.Lock()
|
||||||
|
defer h.mu.Unlock()
|
||||||
|
|
||||||
// While the batch is at its maximum size, wait until the flush goroutine
|
// While the batch is at its maximum size, wait until the flush goroutine
|
||||||
// takes it. Meanwhile the handler takes nothing off its queue, which
|
// takes it. Meanwhile the handler takes nothing off its queue, which
|
||||||
// fills, and the streamer drops messages for it.
|
// fills, and the streamer drops messages for it.
|
||||||
for len(h.batch) >= h.batchMaxSize {
|
for len(h.batch) >= prefixBatchMaxSize {
|
||||||
h.batchTaken.Wait()
|
h.batchTaken.Wait()
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -403,7 +239,7 @@ func (h *PrefixHandler) applyMessage(msg *ristypes.RISMessage) {
|
|||||||
|
|
||||||
// A full batch wakes the flush goroutine. If it is writing, the wake-up
|
// A full batch wakes the flush goroutine. If it is writing, the wake-up
|
||||||
// waits in batchFullCh and the batch keeps growing until the write ends
|
// waits in batchFullCh and the batch keeps growing until the write ends
|
||||||
// or the batch reaches batchMaxSize.
|
// or the batch reaches prefixBatchMaxSize.
|
||||||
if len(h.batch) >= prefixBatchSize {
|
if len(h.batch) >= prefixBatchSize {
|
||||||
select {
|
select {
|
||||||
case h.batchFullCh <- struct{}{}:
|
case h.batchFullCh <- struct{}{}:
|
||||||
@@ -412,88 +248,39 @@ func (h *PrefixHandler) applyMessage(msg *ristypes.RISMessage) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// startPeerDown adds peer to downPeers. It starts peerDown on a new goroutine
|
// Stop stops the flush goroutine, which makes one last flush before it returns,
|
||||||
// unless peerDown is running. The caller holds mu. The peer's messages are
|
// and waits for it until ctx ends. That flush writes at most
|
||||||
// held back until peerDown has applied them. Messages held back for an earlier
|
// maxTableChangesPerWrite of the routing table's changes, and the prefixes of
|
||||||
// session of the peer going down, and not yet applied, stay held back.
|
// the updates the handler holds. Once ctx has ended the flush goroutine skips
|
||||||
func (h *PrefixHandler) startPeerDown(peer string) {
|
// that last flush, and a write still running is lost when the process exits. A
|
||||||
_, goingDown := h.heldBack[peer]
|
// failed write of the table's changes is not retried either. Stop logs how many
|
||||||
if !goingDown {
|
// updates and how many of the table's changes are not written.
|
||||||
h.heldBack[peer] = []*ristypes.RISMessage{}
|
func (h *PrefixHandler) Stop(ctx context.Context) {
|
||||||
|
h.stopCh <- ctx
|
||||||
|
|
||||||
|
select {
|
||||||
|
case <-h.done:
|
||||||
|
case <-ctx.Done():
|
||||||
}
|
}
|
||||||
|
|
||||||
h.downPeers = append(h.downPeers, peer)
|
h.mu.Lock()
|
||||||
|
unwritten := h.writing + len(h.batch)
|
||||||
|
tableChanges := h.writingTableChanges + h.table.ChangeCount()
|
||||||
|
h.mu.Unlock()
|
||||||
|
|
||||||
if h.peerDownRunning {
|
if unwritten == 0 && tableChanges == 0 {
|
||||||
return
|
return
|
||||||
}
|
}
|
||||||
|
|
||||||
h.peerDownRunning = true
|
if ctx.Err() != nil {
|
||||||
h.peerDowns.Add(1)
|
h.logger.Error("Stop deadline passed before the prefix handler wrote "+
|
||||||
|
"its updates", "updates", unwritten, "table_changes", tableChanges)
|
||||||
|
|
||||||
go h.peerDown()
|
return
|
||||||
}
|
|
||||||
|
|
||||||
// peerDown takes every peer in downPeers and runs the table's PeersDown for
|
|
||||||
// them. Then it applies the messages held back from each of those peers. It
|
|
||||||
// repeats this until downPeers is empty.
|
|
||||||
func (h *PrefixHandler) peerDown() {
|
|
||||||
defer h.peerDowns.Done()
|
|
||||||
|
|
||||||
for {
|
|
||||||
h.mu.Lock()
|
|
||||||
|
|
||||||
peers := h.downPeers
|
|
||||||
h.downPeers = nil
|
|
||||||
|
|
||||||
if len(peers) == 0 {
|
|
||||||
h.peerDownRunning = false
|
|
||||||
h.mu.Unlock()
|
|
||||||
|
|
||||||
return
|
|
||||||
}
|
|
||||||
|
|
||||||
h.mu.Unlock()
|
|
||||||
|
|
||||||
h.tablePeersDown(peers)
|
|
||||||
|
|
||||||
for _, peer := range peers {
|
|
||||||
h.applyHeldBack(peer)
|
|
||||||
}
|
|
||||||
}
|
}
|
||||||
}
|
|
||||||
|
|
||||||
// applyHeldBack applies the messages held back from peer, in order. It takes
|
h.logger.Error("The prefix handler stopped before it wrote everything",
|
||||||
// mu for one message at a time, so HandleMessage takes new messages in
|
"updates", unwritten, "table_changes", tableChanges)
|
||||||
// between. The peer's new messages join the end of the held-back ones. A
|
|
||||||
// held-back message that says the peer's session went down again adds the
|
|
||||||
// peer to downPeers. The rest stay held back until the next PeersDown has
|
|
||||||
// returned.
|
|
||||||
func (h *PrefixHandler) applyHeldBack(peer string) {
|
|
||||||
for {
|
|
||||||
h.mu.Lock()
|
|
||||||
|
|
||||||
held := h.heldBack[peer]
|
|
||||||
if len(held) == 0 {
|
|
||||||
delete(h.heldBack, peer)
|
|
||||||
h.mu.Unlock()
|
|
||||||
|
|
||||||
return
|
|
||||||
}
|
|
||||||
|
|
||||||
msg := held[0]
|
|
||||||
held[0] = nil // so that the message is freed once applied
|
|
||||||
h.heldBack[peer] = held[1:]
|
|
||||||
h.heldCount--
|
|
||||||
h.heldTaken.Broadcast()
|
|
||||||
h.applyMessage(msg)
|
|
||||||
h.mu.Unlock()
|
|
||||||
|
|
||||||
if msg.Type == ristypes.MessageTypeState &&
|
|
||||||
msg.State == ristypes.StateDown {
|
|
||||||
return
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
}
|
||||||
|
|
||||||
// takeBatch returns the pending updates and leaves an empty batch in their
|
// takeBatch returns the pending updates and leaves an empty batch in their
|
||||||
@@ -515,12 +302,11 @@ func (h *PrefixHandler) takeBatch() []seenPrefix {
|
|||||||
return batch
|
return batch
|
||||||
}
|
}
|
||||||
|
|
||||||
// flushBatch writes the routing table's changes and origin changes, then
|
// flushBatch writes the routing table's changes, then takes the pending batch
|
||||||
// takes the pending batch and stores the prefixes it names, without holding
|
// and stores the prefixes it names, without holding the mutex during the
|
||||||
// the mutex during the writes.
|
// writes.
|
||||||
func (h *PrefixHandler) flushBatch() {
|
func (h *PrefixHandler) flushBatch() {
|
||||||
h.writeTableChanges()
|
h.writeTableChanges()
|
||||||
h.writeOriginChanges()
|
|
||||||
|
|
||||||
batch := h.takeBatch()
|
batch := h.takeBatch()
|
||||||
if len(batch) == 0 {
|
if len(batch) == 0 {
|
||||||
@@ -592,13 +378,14 @@ func (h *PrefixHandler) applyToTable(update prefixUpdate) {
|
|||||||
})
|
})
|
||||||
}
|
}
|
||||||
|
|
||||||
// writeTableChanges writes at most tableChangesPerWrite changes of the routing
|
// writeTableChanges writes at most maxTableChangesPerWrite changes of the
|
||||||
// table in memory, and every peer the table numbered since the last write.
|
// routing table in memory, and every peer the table numbered since the last
|
||||||
|
// write.
|
||||||
// When the write fails, they go back to the table, so that a later write writes
|
// When the write fails, they go back to the table, so that a later write writes
|
||||||
// them again, each pair with its content then.
|
// them again, each pair with its content then.
|
||||||
func (h *PrefixHandler) writeTableChanges() {
|
func (h *PrefixHandler) writeTableChanges() {
|
||||||
h.mu.Lock()
|
h.mu.Lock()
|
||||||
changes, newPeers := h.table.TakeChanges(h.tableChangesPerWrite)
|
changes, newPeers := h.table.TakeChanges(maxTableChangesPerWrite)
|
||||||
h.writingTableChanges = len(changes)
|
h.writingTableChanges = len(changes)
|
||||||
h.mu.Unlock()
|
h.mu.Unlock()
|
||||||
|
|
||||||
@@ -622,39 +409,6 @@ func (h *PrefixHandler) writeTableChanges() {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// writeOriginChanges writes at most originChangesPerWrite of the routing
|
|
||||||
// table's origin changes. When the write fails, they go back to the table, so
|
|
||||||
// that a later write writes them again. A dropped change is logged as a
|
|
||||||
// warning.
|
|
||||||
func (h *PrefixHandler) writeOriginChanges() {
|
|
||||||
h.mu.Lock()
|
|
||||||
changes, dropped := h.table.TakeOriginChanges(h.originChangesPerWrite)
|
|
||||||
h.writingOriginChanges = len(changes)
|
|
||||||
h.mu.Unlock()
|
|
||||||
|
|
||||||
if dropped > 0 {
|
|
||||||
h.logger.Warn("Origin changes dropped", "dropped", dropped)
|
|
||||||
}
|
|
||||||
|
|
||||||
if len(changes) == 0 {
|
|
||||||
return
|
|
||||||
}
|
|
||||||
|
|
||||||
err := h.db.WriteOriginChanges(changes)
|
|
||||||
|
|
||||||
h.mu.Lock()
|
|
||||||
defer h.mu.Unlock()
|
|
||||||
|
|
||||||
h.writingOriginChanges = 0
|
|
||||||
|
|
||||||
if err != nil {
|
|
||||||
h.logger.Error("Failed to write the origin changes; the next flush "+
|
|
||||||
"retries", "changes", len(changes), "error", err)
|
|
||||||
|
|
||||||
h.table.PutBackOriginChanges(changes)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// canonicalPrefix returns prefix in the text form net/netip prints, the form
|
// canonicalPrefix returns prefix in the text form net/netip prints, the form
|
||||||
// the prefix tables store and the IP lookup reads them in. The feed sends IPv6
|
// the prefix tables store and the IP lookup reads them in. The feed sends IPv6
|
||||||
// announcements compressed ("2001:db8::/32") but IPv6 withdrawals uncompressed
|
// announcements compressed ("2001:db8::/32") but IPv6 withdrawals uncompressed
|
||||||
|
|||||||
@@ -237,120 +237,22 @@ func TestPrefixHandlerStopLogsNothingWhenItWroteEverything(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// TestPrefixHandlerStopLogsTheMessagesHeldBack stops a prefix handler with a
|
// longerThanAnyTest is a flush timer interval that no test lasts. A test whose
|
||||||
// deadline that has passed, while a peer's PeersDown is held up and one message
|
// prefix, AS or peer handler must flush only when its batch is full and on Stop
|
||||||
// of the peer is held back. Stop must log that message as not applied.
|
// gives it to newPrefixHandler, newASHandler or newPeerHandler.
|
||||||
func TestPrefixHandlerStopLogsTheMessagesHeldBack(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
var logs bytes.Buffer
|
|
||||||
|
|
||||||
h := newPrefixHandler(newMockStore(), newErrorLogger(&logs),
|
|
||||||
routetable.New(routetable.Params{}), longerThanAnyTest,
|
|
||||||
maxTableChangesPerWrite)
|
|
||||||
|
|
||||||
release := holdUpPeersDown(h)
|
|
||||||
defer close(release)
|
|
||||||
|
|
||||||
h.HandleMessage(peerAnnouncement(testPeerIP, testV4Prefix))
|
|
||||||
h.HandleMessage(sessionDown())
|
|
||||||
h.HandleMessage(peerAnnouncement(testPeerIP, testV6Prefix))
|
|
||||||
|
|
||||||
ctx, cancel := context.WithCancel(t.Context())
|
|
||||||
cancel()
|
|
||||||
h.Stop(ctx)
|
|
||||||
|
|
||||||
logged := lastLogged(t, &logs)
|
|
||||||
if logged.HeldMessages != 1 {
|
|
||||||
t.Errorf("Stop logged %d held messages not applied, want 1",
|
|
||||||
logged.HeldMessages)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestPrefixHandlerStopWaitsForTheMessagesHeldBack stops a prefix handler
|
|
||||||
// while a peer's PeersDown is held up and an announcement of the peer is held
|
|
||||||
// back. Stop must wait. Once the PeersDown is let go and Stop returns, the
|
|
||||||
// announcement must be in the database.
|
|
||||||
func TestPrefixHandlerStopWaitsForTheMessagesHeldBack(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
db := newTestDatabase(t)
|
|
||||||
h := newPrefixHandler(db, logger.New(), routetable.New(routetable.Params{}),
|
|
||||||
longerThanAnyTest, maxTableChangesPerWrite)
|
|
||||||
|
|
||||||
release := holdUpPeersDown(h)
|
|
||||||
h.HandleMessage(sessionDown())
|
|
||||||
h.HandleMessage(peerAnnouncement(testPeerIP, testV4Prefix))
|
|
||||||
|
|
||||||
stopped := make(chan struct{})
|
|
||||||
|
|
||||||
go func() {
|
|
||||||
h.Stop(t.Context())
|
|
||||||
close(stopped)
|
|
||||||
}()
|
|
||||||
|
|
||||||
select {
|
|
||||||
case <-stopped:
|
|
||||||
t.Error("Stop returned while a PeersDown was held up")
|
|
||||||
case <-time.After(stillWaitingAfter):
|
|
||||||
}
|
|
||||||
|
|
||||||
close(release)
|
|
||||||
|
|
||||||
select {
|
|
||||||
case <-stopped:
|
|
||||||
case <-time.After(stepTimeout):
|
|
||||||
t.Fatalf("Stop did not return within %s of the PeersDown's release",
|
|
||||||
stepTimeout)
|
|
||||||
}
|
|
||||||
|
|
||||||
read, err := db.ReadRouteTable(t.Context())
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("ReadRouteTable: %v", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
checkTableOrigins(t, read, map[string]map[int][]string{
|
|
||||||
testV4Prefix: {testASNB: {testPeerIP}},
|
|
||||||
})
|
|
||||||
}
|
|
||||||
|
|
||||||
// longerThanAnyTest is a timer interval that no test lasts. A test gives it to
|
|
||||||
// newPrefixHandler, newASHandler, newPeerHandler or newPeeringHandler when the
|
|
||||||
// handler must write only when its batch is full, when the test says, or on
|
|
||||||
// Stop.
|
|
||||||
const longerThanAnyTest = time.Hour
|
const longerThanAnyTest = time.Hour
|
||||||
|
|
||||||
// prefixHandlerWithoutTimer returns a prefix handler that writes to store and
|
// prefixHandlerWithoutTimer returns a prefix handler that writes to store and
|
||||||
// applies the updates to an empty routing table. It is built without
|
// applies the updates to an empty routing table. It is built without
|
||||||
// NewPrefixHandler's flush goroutine, so the test flushes itself.
|
// NewPrefixHandler's flush goroutine, so the test flushes itself.
|
||||||
func prefixHandlerWithoutTimer(store database.Store) *PrefixHandler {
|
func prefixHandlerWithoutTimer(store database.Store) *PrefixHandler {
|
||||||
table := routetable.New(routetable.Params{})
|
|
||||||
|
|
||||||
return &PrefixHandler{
|
return &PrefixHandler{
|
||||||
db: store,
|
db: store,
|
||||||
logger: logger.New(),
|
logger: logger.New(),
|
||||||
table: table,
|
table: routetable.New(routetable.Params{}),
|
||||||
batchMaxSize: prefixBatchMaxSize,
|
|
||||||
tableChangesPerWrite: maxTableChangesPerWrite,
|
|
||||||
originChangesPerWrite: maxOriginChangesPerWrite,
|
|
||||||
heldBack: make(map[string][]*ristypes.RISMessage),
|
|
||||||
tablePeersDown: table.PeersDown,
|
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// holdUpPeersDown makes each PeersDown of h wait until the returned channel is
|
|
||||||
// closed, as another peer's long PeersDown would hold it up. The test calls it
|
|
||||||
// before it hands h a message.
|
|
||||||
func holdUpPeersDown(h *PrefixHandler) chan struct{} {
|
|
||||||
release := make(chan struct{})
|
|
||||||
h.tablePeersDown = func(peers []string) {
|
|
||||||
<-release
|
|
||||||
h.table.PeersDown(peers)
|
|
||||||
}
|
|
||||||
|
|
||||||
return release
|
|
||||||
}
|
|
||||||
|
|
||||||
// newTestDatabase opens a database in a new temporary directory. It is closed
|
// newTestDatabase opens a database in a new temporary directory. It is closed
|
||||||
// when the test ends.
|
// when the test ends.
|
||||||
func newTestDatabase(t *testing.T) *database.Database {
|
func newTestDatabase(t *testing.T) *database.Database {
|
||||||
|
|||||||
@@ -1,181 +0,0 @@
|
|||||||
package routewatch
|
|
||||||
|
|
||||||
import (
|
|
||||||
"bytes"
|
|
||||||
"encoding/json"
|
|
||||||
"errors"
|
|
||||||
"log/slog"
|
|
||||||
"net/netip"
|
|
||||||
"slices"
|
|
||||||
"testing"
|
|
||||||
|
|
||||||
"git.eeqj.de/sneak/routewatch/internal/database"
|
|
||||||
"git.eeqj.de/sneak/routewatch/internal/logger"
|
|
||||||
"git.eeqj.de/sneak/routewatch/internal/routetable"
|
|
||||||
)
|
|
||||||
|
|
||||||
// TestPrefixHandlerWritesOriginChanges hands a running prefix handler an
|
|
||||||
// announcement and flushes. The flush must write the origin change to the
|
|
||||||
// database.
|
|
||||||
func TestPrefixHandlerWritesOriginChanges(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
db := newTestDatabase(t)
|
|
||||||
h := prefixHandlerWithoutTimer(db)
|
|
||||||
|
|
||||||
h.HandleMessage(peerAnnouncement(testPeerIP, testV4Prefix))
|
|
||||||
h.flushBatch()
|
|
||||||
|
|
||||||
got, err := db.GetOriginChanges(t.Context(),
|
|
||||||
netip.MustParsePrefix(testV4Prefix), 10)
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("GetOriginChanges: %v", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
if len(got) != 1 || !got[0].Announced || got[0].OriginASN != testASNB {
|
|
||||||
t.Errorf("GetOriginChanges = %+v, want one announced row for AS%d",
|
|
||||||
got, testASNB)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// errTestOriginChangeWrite is the error a test store's failing write of the
|
|
||||||
// origin changes returns.
|
|
||||||
var errTestOriginChangeWrite = errors.New(
|
|
||||||
"test: the write of the origin changes failed")
|
|
||||||
|
|
||||||
// originChangesFailsStore is a real database whose first write of the origin
|
|
||||||
// changes fails, writing nothing. writes counts the writes asked for.
|
|
||||||
type originChangesFailsStore struct {
|
|
||||||
*database.Database
|
|
||||||
|
|
||||||
writes int
|
|
||||||
}
|
|
||||||
|
|
||||||
func (s *originChangesFailsStore) WriteOriginChanges(
|
|
||||||
changes []routetable.OriginChange,
|
|
||||||
) error {
|
|
||||||
s.writes++
|
|
||||||
if s.writes == 1 {
|
|
||||||
return errTestOriginChangeWrite
|
|
||||||
}
|
|
||||||
|
|
||||||
return s.Database.WriteOriginChanges(changes)
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestPrefixHandlerWritesAFailedOriginChangeWriteAgain runs two flushes into a
|
|
||||||
// database whose first write of the origin changes fails. The second write must
|
|
||||||
// hold the change of the first.
|
|
||||||
func TestPrefixHandlerWritesAFailedOriginChangeWriteAgain(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
db := newTestDatabase(t)
|
|
||||||
store := &originChangesFailsStore{Database: db}
|
|
||||||
h := prefixHandlerWithoutTimer(store)
|
|
||||||
|
|
||||||
h.HandleMessage(peerAnnouncement(testPeerIP, testV4Prefix))
|
|
||||||
h.flushBatch()
|
|
||||||
h.flushBatch()
|
|
||||||
|
|
||||||
got, err := db.GetOriginChanges(t.Context(),
|
|
||||||
netip.MustParsePrefix(testV4Prefix), 10)
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("GetOriginChanges: %v", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
if len(got) != 1 || !got[0].Announced || got[0].OriginASN != testASNB {
|
|
||||||
t.Errorf("GetOriginChanges = %+v, want one announced row for AS%d",
|
|
||||||
got, testASNB)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// originChangesWriteSizesStore is a real database that records how many
|
|
||||||
// changes each write of the origin changes holds.
|
|
||||||
type originChangesWriteSizesStore struct {
|
|
||||||
*database.Database
|
|
||||||
|
|
||||||
counts []int
|
|
||||||
}
|
|
||||||
|
|
||||||
func (s *originChangesWriteSizesStore) WriteOriginChanges(
|
|
||||||
changes []routetable.OriginChange,
|
|
||||||
) error {
|
|
||||||
s.counts = append(s.counts, len(changes))
|
|
||||||
|
|
||||||
return s.Database.WriteOriginChanges(changes)
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestPrefixHandlerWritesManyOriginChangesOverSeveralFlushes has more origin
|
|
||||||
// changes than one write takes, then flushes three times. The first write must
|
|
||||||
// hold one write's worth, the second the rest, and the third nothing. The
|
|
||||||
// handler's limit is small.
|
|
||||||
func TestPrefixHandlerWritesManyOriginChangesOverSeveralFlushes(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
const perWrite = 10
|
|
||||||
|
|
||||||
db := newTestDatabase(t)
|
|
||||||
store := &originChangesWriteSizesStore{Database: db}
|
|
||||||
h := prefixHandlerWithoutTimer(store)
|
|
||||||
h.originChangesPerWrite = perWrite
|
|
||||||
|
|
||||||
announceManyPrefixes(h.table, perWrite*3/2)
|
|
||||||
|
|
||||||
h.flushBatch()
|
|
||||||
h.flushBatch()
|
|
||||||
h.flushBatch()
|
|
||||||
|
|
||||||
want := []int{perWrite, perWrite*3/2 - perWrite}
|
|
||||||
if !slices.Equal(store.counts, want) {
|
|
||||||
t.Errorf("WriteOriginChanges held %v changes, want %v", store.counts, want)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestPrefixHandlerLogsTheDroppedOriginChanges fills the routing table with
|
|
||||||
// one origin change more than it keeps, then flushes. The warning the flush
|
|
||||||
// logs must carry the dropped count.
|
|
||||||
func TestPrefixHandlerLogsTheDroppedOriginChanges(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
var logs bytes.Buffer
|
|
||||||
|
|
||||||
db := newTestDatabase(t)
|
|
||||||
h := prefixHandlerWithoutTimer(db)
|
|
||||||
h.logger = &logger.Logger{
|
|
||||||
Logger: slog.New(slog.NewJSONHandler(&logs,
|
|
||||||
&slog.HandlerOptions{Level: slog.LevelWarn})),
|
|
||||||
}
|
|
||||||
|
|
||||||
announceManyPrefixes(h.table, routetable.MaxWaitingOriginChanges+1)
|
|
||||||
|
|
||||||
h.flushBatch()
|
|
||||||
|
|
||||||
var warning struct {
|
|
||||||
Msg string `json:"msg"`
|
|
||||||
Dropped *int `json:"dropped"`
|
|
||||||
}
|
|
||||||
|
|
||||||
found := false
|
|
||||||
|
|
||||||
for line := range bytes.SplitSeq(bytes.TrimSpace(logs.Bytes()), []byte("\n")) {
|
|
||||||
warning.Dropped = nil
|
|
||||||
|
|
||||||
err := json.Unmarshal(line, &warning)
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("want a JSON log line, got %q: %v", line, err)
|
|
||||||
}
|
|
||||||
|
|
||||||
if warning.Msg != "Origin changes dropped" {
|
|
||||||
continue
|
|
||||||
}
|
|
||||||
|
|
||||||
found = true
|
|
||||||
|
|
||||||
if warning.Dropped == nil || *warning.Dropped != 1 {
|
|
||||||
t.Errorf("the warning's dropped = %v, want 1", warning.Dropped)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
if !found {
|
|
||||||
t.Errorf("no warning logged the dropped origin changes")
|
|
||||||
}
|
|
||||||
}
|
|
||||||
@@ -7,10 +7,8 @@ import (
|
|||||||
"log/slog"
|
"log/slog"
|
||||||
"net/netip"
|
"net/netip"
|
||||||
"reflect"
|
"reflect"
|
||||||
"runtime"
|
|
||||||
"slices"
|
"slices"
|
||||||
"strings"
|
"strings"
|
||||||
"sync"
|
|
||||||
"testing"
|
"testing"
|
||||||
"time"
|
"time"
|
||||||
|
|
||||||
@@ -32,13 +30,12 @@ const (
|
|||||||
|
|
||||||
// firstManyPrefix is the first prefix announceManyPrefixes announces.
|
// firstManyPrefix is the first prefix announceManyPrefixes announces.
|
||||||
firstManyPrefix = "10.0.0.0/32"
|
firstManyPrefix = "10.0.0.0/32"
|
||||||
|
|
||||||
// testMaxTableChangesPerWrite is a limit of one write of the routing
|
|
||||||
// table's changes that a test exceeds quickly, in place of
|
|
||||||
// maxTableChangesPerWrite.
|
|
||||||
testMaxTableChangesPerWrite = 10
|
|
||||||
)
|
)
|
||||||
|
|
||||||
|
// tableWriteTimeout is how long a test waits for the flush timer to write the
|
||||||
|
// routing table's changes: a few ticks of it.
|
||||||
|
const tableWriteTimeout = 5 * prefixBatchTimeout
|
||||||
|
|
||||||
// tableCase is one test of the routing table in memory: the messages handed to
|
// tableCase is one test of the routing table in memory: the messages handed to
|
||||||
// the prefix handler, and for each prefix the peers that carry it with each
|
// the prefix handler, and for each prefix the peers that carry it with each
|
||||||
// origin AS after them.
|
// origin AS after them.
|
||||||
@@ -103,359 +100,6 @@ func TestPrefixHandlerAppliesUpdatesToTheRoutingTable(t *testing.T) {
|
|||||||
})
|
})
|
||||||
}
|
}
|
||||||
|
|
||||||
// TestPrefixHandlerTakesMessagesWhileAPeerGoesDown holds up the PeersDown of a
|
|
||||||
// peer whose session went down, as another peer's long PeersDown would. The
|
|
||||||
// handler must go on taking messages. It applies another peer's announcement
|
|
||||||
// at once. It holds back these messages of the down peer, in this order:
|
|
||||||
// - an announcement;
|
|
||||||
// - its session going down again;
|
|
||||||
// - a second announcement.
|
|
||||||
//
|
|
||||||
// HeldBackCount must count them. Once the PeersDown calls have returned, the
|
|
||||||
// down peer must carry only the prefix of the second announcement.
|
|
||||||
func TestPrefixHandlerTakesMessagesWhileAPeerGoesDown(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
const heldMessages = 3
|
|
||||||
|
|
||||||
h := prefixHandlerWithoutTimer(newMockStore())
|
|
||||||
release := holdUpPeersDown(h)
|
|
||||||
h.HandleMessage(peerAnnouncement(testPeerIP, testV4Prefix))
|
|
||||||
|
|
||||||
handled := returnsInTime(func() {
|
|
||||||
for _, msg := range []*ristypes.RISMessage{
|
|
||||||
sessionDown(),
|
|
||||||
peerAnnouncement(secondPeerIP, otherV4Prefix),
|
|
||||||
peerAnnouncement(testPeerIP, testV6Prefix),
|
|
||||||
sessionDown(),
|
|
||||||
peerAnnouncement(testPeerIP, setV4Prefix),
|
|
||||||
} {
|
|
||||||
h.HandleMessage(msg)
|
|
||||||
}
|
|
||||||
})
|
|
||||||
if !handled {
|
|
||||||
t.Fatalf("the handler did not take the messages after the session "+
|
|
||||||
"went down within %s", stepTimeout)
|
|
||||||
}
|
|
||||||
|
|
||||||
checkTableOrigins(t, h.table, map[string]map[int][]string{
|
|
||||||
testV4Prefix: {testASNB: {testPeerIP}},
|
|
||||||
otherV4Prefix: {testASNB: {secondPeerIP}},
|
|
||||||
testV6Prefix: {},
|
|
||||||
setV4Prefix: {},
|
|
||||||
})
|
|
||||||
|
|
||||||
if h.HeldBackCount() != heldMessages {
|
|
||||||
t.Errorf("HeldBackCount() = %d, want %d", h.HeldBackCount(),
|
|
||||||
heldMessages)
|
|
||||||
}
|
|
||||||
|
|
||||||
close(release)
|
|
||||||
|
|
||||||
if !returnsInTime(h.peerDowns.Wait) {
|
|
||||||
t.Fatalf("the PeersDown calls did not return within %s", stepTimeout)
|
|
||||||
}
|
|
||||||
|
|
||||||
checkTableOrigins(t, h.table, map[string]map[int][]string{
|
|
||||||
testV4Prefix: {},
|
|
||||||
otherV4Prefix: {testASNB: {secondPeerIP}},
|
|
||||||
testV6Prefix: {},
|
|
||||||
setV4Prefix: {testASNB: {testPeerIP}},
|
|
||||||
})
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestPrefixHandlerTakesPeersDownTogether has four peers announce testV4Prefix.
|
|
||||||
// It holds up the PeersDown of the first peer, whose session goes down.
|
|
||||||
// Meanwhile the sessions of the three others go down, and each of them
|
|
||||||
// announces otherV4Prefix. Once the first PeersDown is let go, the next one
|
|
||||||
// must take the three others together. No peer may then carry testV4Prefix,
|
|
||||||
// and the three others must carry otherV4Prefix.
|
|
||||||
func TestPrefixHandlerTakesPeersDownTogether(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
others := []string{secondPeerIP, thirdPeerIP, fourthPeerIP}
|
|
||||||
h := prefixHandlerWithoutTimer(newMockStore())
|
|
||||||
|
|
||||||
// calls receives the peers of each PeersDown, with room for one call
|
|
||||||
// for each peer.
|
|
||||||
calls := make(chan []string, len(others)+1)
|
|
||||||
release := make(chan struct{})
|
|
||||||
h.tablePeersDown = func(peers []string) {
|
|
||||||
calls <- slices.Clone(peers)
|
|
||||||
|
|
||||||
<-release
|
|
||||||
h.table.PeersDown(peers)
|
|
||||||
}
|
|
||||||
|
|
||||||
for _, peer := range append([]string{testPeerIP}, others...) {
|
|
||||||
h.HandleMessage(peerAnnouncement(peer, testV4Prefix))
|
|
||||||
}
|
|
||||||
|
|
||||||
h.HandleMessage(sessionDown())
|
|
||||||
|
|
||||||
select {
|
|
||||||
case <-calls:
|
|
||||||
case <-time.After(stepTimeout):
|
|
||||||
t.Fatalf("PeersDown did not begin within %s", stepTimeout)
|
|
||||||
}
|
|
||||||
|
|
||||||
for _, peer := range others {
|
|
||||||
h.HandleMessage(sessionDownOf(peer))
|
|
||||||
h.HandleMessage(peerAnnouncement(peer, otherV4Prefix))
|
|
||||||
}
|
|
||||||
|
|
||||||
close(release)
|
|
||||||
|
|
||||||
if !returnsInTime(h.peerDowns.Wait) {
|
|
||||||
t.Fatalf("the PeersDown calls did not return within %s", stepTimeout)
|
|
||||||
}
|
|
||||||
|
|
||||||
close(calls)
|
|
||||||
|
|
||||||
later := make([][]string, 0, len(calls))
|
|
||||||
for peers := range calls {
|
|
||||||
later = append(later, peers)
|
|
||||||
}
|
|
||||||
|
|
||||||
if !reflect.DeepEqual(later, [][]string{others}) {
|
|
||||||
t.Errorf("the PeersDown calls after the first took %v, want %v "+
|
|
||||||
"together", later, others)
|
|
||||||
}
|
|
||||||
|
|
||||||
checkTableOrigins(t, h.table, map[string]map[int][]string{
|
|
||||||
testV4Prefix: {},
|
|
||||||
otherV4Prefix: {testASNB: others},
|
|
||||||
})
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestPrefixHandlerAppliesMessagesWhilePeersDownRuns gives a peer 50,000
|
|
||||||
// prefixes and hands the prefix handler that peer's session going down. The
|
|
||||||
// table's PeersDown then visits the prefixes in five batches. Once PeersDown
|
|
||||||
// has begun, the handler is handed 1,000 announcements of another peer. The
|
|
||||||
// handler must apply them all before PeersDown ends, while the down peer still
|
|
||||||
// carries some of its prefixes.
|
|
||||||
func TestPrefixHandlerAppliesMessagesWhilePeersDownRuns(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
const (
|
|
||||||
// downPeerPrefixes is five times the 10,000 prefixes PeersDown visits
|
|
||||||
// in one hold of the table's lock.
|
|
||||||
downPeerPrefixes = 50000
|
|
||||||
// otherPeerMessages is far more messages than the handler applies
|
|
||||||
// while PeersDown runs, if PeersDown takes the lock back at once after
|
|
||||||
// each batch. The handler then applies only a few for each batch.
|
|
||||||
otherPeerMessages = 1000
|
|
||||||
// firstOtherAddr is the address of the other peer's first prefix.
|
|
||||||
firstOtherAddr = "2001:db8:2::"
|
|
||||||
)
|
|
||||||
|
|
||||||
h := prefixHandlerWithoutTimer(newMockStore())
|
|
||||||
announceManyPrefixes(h.table, downPeerPrefixes)
|
|
||||||
|
|
||||||
h.HandleMessage(sessionDown())
|
|
||||||
|
|
||||||
// Wait until PeersDown has begun: the down peer has left some of its
|
|
||||||
// prefixes. The down peer's prefixes are IPv4 and the other peer's are
|
|
||||||
// IPv6, so the counts tell them apart.
|
|
||||||
for h.table.Counts().IPv4Routes == downPeerPrefixes {
|
|
||||||
runtime.Gosched()
|
|
||||||
}
|
|
||||||
|
|
||||||
// Each announcement names the /128 of the next address.
|
|
||||||
addr := netip.MustParseAddr(firstOtherAddr)
|
|
||||||
for range otherPeerMessages {
|
|
||||||
h.HandleMessage(peerAnnouncement(secondPeerIP,
|
|
||||||
netip.PrefixFrom(addr, addr.BitLen()).String()))
|
|
||||||
|
|
||||||
addr = addr.Next()
|
|
||||||
}
|
|
||||||
|
|
||||||
counts := h.table.Counts()
|
|
||||||
if counts.IPv6Routes != otherPeerMessages {
|
|
||||||
t.Errorf("the other peer carries %d prefixes, want %d",
|
|
||||||
counts.IPv6Routes, otherPeerMessages)
|
|
||||||
}
|
|
||||||
|
|
||||||
if counts.IPv4Routes == 0 {
|
|
||||||
t.Errorf("PeersDown ended before the handler applied the other peer's "+
|
|
||||||
"%d announcements", otherPeerMessages)
|
|
||||||
}
|
|
||||||
|
|
||||||
if !returnsInTime(h.peerDowns.Wait) {
|
|
||||||
t.Fatalf("the PeersDown did not return within %s", stepTimeout)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestPrefixHandlerTakesMessagesWhileAReturningPeerSendsItsTable gives a peer
|
|
||||||
// 50,000 prefixes and hands the prefix handler that peer's session going down.
|
|
||||||
// Once PeersDown has begun, the peer comes back and sends more announcements
|
|
||||||
// than the handler's queue holds, as a peer that sends its whole table again
|
|
||||||
// does. The handler must hold them back and go on taking messages. Another
|
|
||||||
// peer's announcement, handed next, must be applied before PeersDown ends.
|
|
||||||
func TestPrefixHandlerTakesMessagesWhileAReturningPeerSendsItsTable(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
const (
|
|
||||||
// downPeerPrefixes is five times the 10,000 prefixes PeersDown visits
|
|
||||||
// in one hold of the table's lock.
|
|
||||||
downPeerPrefixes = 50000
|
|
||||||
returningMessages = prefixHandlerQueueSize + 1
|
|
||||||
)
|
|
||||||
|
|
||||||
table := routetable.New(routetable.Params{})
|
|
||||||
announceManyPrefixes(table, downPeerPrefixes)
|
|
||||||
|
|
||||||
h := newPrefixHandler(newMockStore(), logger.New(), table,
|
|
||||||
longerThanAnyTest, maxTableChangesPerWrite)
|
|
||||||
defer stopInTime(t, h)
|
|
||||||
|
|
||||||
h.HandleMessage(sessionDown())
|
|
||||||
|
|
||||||
// Wait until PeersDown has begun: the down peer has left some of its
|
|
||||||
// prefixes, which are all IPv4.
|
|
||||||
for h.table.Counts().IPv4Routes == downPeerPrefixes {
|
|
||||||
runtime.Gosched()
|
|
||||||
}
|
|
||||||
|
|
||||||
returning := peerAnnouncement(testPeerIP, testV6Prefix)
|
|
||||||
|
|
||||||
if !returnsInTime(func() {
|
|
||||||
for range returningMessages {
|
|
||||||
h.HandleMessage(returning)
|
|
||||||
}
|
|
||||||
|
|
||||||
h.HandleMessage(peerAnnouncement(secondPeerIP, testV6Prefix))
|
|
||||||
}) {
|
|
||||||
t.Fatalf("the handler did not take the messages within %s", stepTimeout)
|
|
||||||
}
|
|
||||||
|
|
||||||
if h.table.Counts().IPv4Routes == 0 {
|
|
||||||
t.Error("PeersDown ended before the handler took the other peer's " +
|
|
||||||
"announcement")
|
|
||||||
}
|
|
||||||
|
|
||||||
checkTableOrigins(t, h.table, map[string]map[int][]string{
|
|
||||||
testV6Prefix: {testASNB: {secondPeerIP}},
|
|
||||||
})
|
|
||||||
|
|
||||||
if !returnsInTime(h.peerDowns.Wait) {
|
|
||||||
t.Fatalf("the PeersDown did not return within %s", stepTimeout)
|
|
||||||
}
|
|
||||||
|
|
||||||
checkTableOrigins(t, h.table, map[string]map[int][]string{
|
|
||||||
testV6Prefix: {testASNB: {testPeerIP, secondPeerIP}},
|
|
||||||
})
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestPrefixHandlerTakesMessagesWhileItAppliesHeldBackOnes holds back many
|
|
||||||
// messages of a peer whose session went down. Each names a prefix that does
|
|
||||||
// not parse, so applying it logs an error. Each log write takes a
|
|
||||||
// millisecond. While the handler applies the held-back messages, it must take
|
|
||||||
// another peer's announcement.
|
|
||||||
func TestPrefixHandlerTakesMessagesWhileItAppliesHeldBackOnes(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
const (
|
|
||||||
// heldMessages take the handler at least a second to apply.
|
|
||||||
heldMessages = 1000
|
|
||||||
badPrefix = "not a prefix"
|
|
||||||
)
|
|
||||||
|
|
||||||
logs := &slowWriter{firstWrite: make(chan struct{})}
|
|
||||||
h := newPrefixHandler(newMockStore(), newErrorLogger(logs),
|
|
||||||
routetable.New(routetable.Params{}), longerThanAnyTest,
|
|
||||||
maxTableChangesPerWrite)
|
|
||||||
|
|
||||||
defer stopInTime(t, h)
|
|
||||||
|
|
||||||
release := holdUpPeersDown(h)
|
|
||||||
h.HandleMessage(sessionDown())
|
|
||||||
|
|
||||||
for range heldMessages {
|
|
||||||
h.HandleMessage(peerAnnouncement(testPeerIP, badPrefix))
|
|
||||||
}
|
|
||||||
|
|
||||||
close(release)
|
|
||||||
|
|
||||||
select {
|
|
||||||
case <-logs.firstWrite:
|
|
||||||
case <-time.After(stepTimeout):
|
|
||||||
t.Fatal("the handler did not start to apply the held-back messages")
|
|
||||||
}
|
|
||||||
|
|
||||||
if !returnsInTime(func() {
|
|
||||||
h.HandleMessage(peerAnnouncement(secondPeerIP, otherV4Prefix))
|
|
||||||
}) {
|
|
||||||
t.Fatalf("HandleMessage did not return within %s", stepTimeout)
|
|
||||||
}
|
|
||||||
|
|
||||||
h.mu.Lock()
|
|
||||||
left := len(h.heldBack[testPeerIP])
|
|
||||||
h.mu.Unlock()
|
|
||||||
|
|
||||||
if left == 0 {
|
|
||||||
t.Error("the handler took another peer's announcement only once it " +
|
|
||||||
"had applied every held-back message")
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestPrefixHandlerWaitsAtTheMaximumHeldBack holds up the PeersDown of a peer
|
|
||||||
// whose session went down. It hands the handler maxHeldBackMessages messages
|
|
||||||
// of that peer. HandleMessage must then wait, even for another peer's
|
|
||||||
// message, until the PeersDown returns and a held-back message is applied.
|
|
||||||
func TestPrefixHandlerWaitsAtTheMaximumHeldBack(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
h := newPrefixHandler(newMockStore(), logger.New(),
|
|
||||||
routetable.New(routetable.Params{}), longerThanAnyTest,
|
|
||||||
maxTableChangesPerWrite)
|
|
||||||
defer stopInTime(t, h)
|
|
||||||
|
|
||||||
release := holdUpPeersDown(h)
|
|
||||||
h.HandleMessage(sessionDown())
|
|
||||||
|
|
||||||
held := peerAnnouncement(testPeerIP, testV6Prefix)
|
|
||||||
for range maxHeldBackMessages {
|
|
||||||
h.HandleMessage(held)
|
|
||||||
}
|
|
||||||
|
|
||||||
returned := make(chan struct{})
|
|
||||||
|
|
||||||
go func() {
|
|
||||||
h.HandleMessage(peerAnnouncement(secondPeerIP, otherV4Prefix))
|
|
||||||
close(returned)
|
|
||||||
}()
|
|
||||||
|
|
||||||
select {
|
|
||||||
case <-returned:
|
|
||||||
t.Error("HandleMessage returned while maxHeldBackMessages messages " +
|
|
||||||
"were held back")
|
|
||||||
case <-time.After(stillWaitingAfter):
|
|
||||||
}
|
|
||||||
|
|
||||||
close(release)
|
|
||||||
|
|
||||||
select {
|
|
||||||
case <-returned:
|
|
||||||
case <-time.After(stepTimeout):
|
|
||||||
t.Fatal("HandleMessage did not return after the PeersDown returned")
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// slowWriter takes a millisecond over each write and drops what it is given.
|
|
||||||
// It closes firstWrite at the first write.
|
|
||||||
type slowWriter struct {
|
|
||||||
firstWrite chan struct{}
|
|
||||||
once sync.Once
|
|
||||||
}
|
|
||||||
|
|
||||||
func (w *slowWriter) Write(p []byte) (int, error) {
|
|
||||||
w.once.Do(func() { close(w.firstWrite) })
|
|
||||||
time.Sleep(time.Millisecond)
|
|
||||||
|
|
||||||
return len(p), nil
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestPrefixHandlerAppliesAPathWithoutAnOriginToTheRoutingTable hands the
|
// TestPrefixHandlerAppliesAPathWithoutAnOriginToTheRoutingTable hands the
|
||||||
// prefix handler announcements whose AS path gives no origin. One whose path
|
// prefix handler announcements whose AS path gives no origin. One whose path
|
||||||
// is empty or holds AS 0 removes the peer's route, as a withdrawal does; one
|
// is empty or holds AS 0 removes the peer's route, as a withdrawal does; one
|
||||||
@@ -554,9 +198,8 @@ func TestPrefixHandlerKeepsThePathOfAPathEndingInASet(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// checkTableCases hands the messages of each case to a new prefix handler. It
|
// checkTableCases hands the messages of each case to a new prefix handler and
|
||||||
// checks the handler's routing table once the handler's PeersDown calls have
|
// checks its routing table right after, with no flush.
|
||||||
// returned, with no flush.
|
|
||||||
func checkTableCases(t *testing.T, tests []tableCase) {
|
func checkTableCases(t *testing.T, tests []tableCase) {
|
||||||
t.Helper()
|
t.Helper()
|
||||||
|
|
||||||
@@ -569,28 +212,17 @@ func checkTableCases(t *testing.T, tests []tableCase) {
|
|||||||
h.HandleMessage(msg)
|
h.HandleMessage(msg)
|
||||||
}
|
}
|
||||||
|
|
||||||
h.peerDowns.Wait()
|
for prefix, want := range tt.want {
|
||||||
checkTableOrigins(t, h.table, tt.want)
|
got := tableOrigins(h.table, prefix)
|
||||||
|
if !reflect.DeepEqual(got, want) {
|
||||||
|
t.Errorf("peers of %s by origin AS = %v, want %v",
|
||||||
|
prefix, got, want)
|
||||||
|
}
|
||||||
|
}
|
||||||
})
|
})
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// checkTableOrigins checks, for each prefix in want, the peers that carry it
|
|
||||||
// in table with each origin AS.
|
|
||||||
func checkTableOrigins(
|
|
||||||
t *testing.T, table *routetable.Table, want map[string]map[int][]string,
|
|
||||||
) {
|
|
||||||
t.Helper()
|
|
||||||
|
|
||||||
for prefix, wantOrigins := range want {
|
|
||||||
got := tableOrigins(table, prefix)
|
|
||||||
if !reflect.DeepEqual(got, wantOrigins) {
|
|
||||||
t.Errorf("peers of %s by origin AS = %v, want %v",
|
|
||||||
prefix, got, wantOrigins)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// tableWritesStore is a real database that hands each write of the routing
|
// tableWritesStore is a real database that hands each write of the routing
|
||||||
// table's changes to written once it is made.
|
// table's changes to written once it is made.
|
||||||
type tableWritesStore struct {
|
type tableWritesStore struct {
|
||||||
@@ -644,9 +276,9 @@ func TestPrefixHandlerWritesTheTableChangesEachSecond(t *testing.T) {
|
|||||||
t.Fatalf("wrote %+v, want the pair of %s and AS%d with "+
|
t.Fatalf("wrote %+v, want the pair of %s and AS%d with "+
|
||||||
"removed %t", changes, prefix, testASNB, step.wantRemoved)
|
"removed %t", changes, prefix, testASNB, step.wantRemoved)
|
||||||
}
|
}
|
||||||
case <-time.After(stepTimeout):
|
case <-time.After(tableWriteTimeout):
|
||||||
t.Fatalf("the routing table's change was not written within %s",
|
t.Fatalf("the routing table's change was not written within %s",
|
||||||
stepTimeout)
|
tableWriteTimeout)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -743,32 +375,25 @@ func (s *tableWriteSizesStore) WriteRouteChanges(
|
|||||||
// takes, then flushes three times. The first write must hold as many pairs as
|
// takes, then flushes three times. The first write must hold as many pairs as
|
||||||
// one write takes, and the peer. The second must hold the other pairs, and the
|
// one write takes, and the peer. The second must hold the other pairs, and the
|
||||||
// third flush must write nothing. The database must then hold the table in
|
// third flush must write nothing. The database must then hold the table in
|
||||||
// memory. The handler's limit is small, and its timer does not fire.
|
// memory.
|
||||||
func TestPrefixHandlerWritesManyTableChangesOverSeveralFlushes(t *testing.T) {
|
func TestPrefixHandlerWritesManyTableChangesOverSeveralFlushes(t *testing.T) {
|
||||||
t.Parallel()
|
t.Parallel()
|
||||||
|
|
||||||
const (
|
const (
|
||||||
pairs = testMaxTableChangesPerWrite * 3 / 2
|
pairs = maxTableChangesPerWrite * 3 / 2
|
||||||
flushes = 3
|
flushes = 3
|
||||||
)
|
)
|
||||||
|
|
||||||
db := newTestDatabase(t)
|
db := newTestDatabase(t)
|
||||||
store := &tableWriteSizesStore{Database: db}
|
store := &tableWriteSizesStore{Database: db}
|
||||||
table := routetable.New(routetable.Params{})
|
h := prefixHandlerWithoutTimer(store)
|
||||||
announceManyPrefixes(table, pairs)
|
announceManyPrefixes(h.table, pairs)
|
||||||
|
|
||||||
h := newPrefixHandler(store, logger.New(), table, longerThanAnyTest,
|
|
||||||
testMaxTableChangesPerWrite)
|
|
||||||
|
|
||||||
defer stopInTime(t, h)
|
|
||||||
|
|
||||||
for range flushes {
|
for range flushes {
|
||||||
h.flushBatch()
|
h.flushBatch()
|
||||||
}
|
}
|
||||||
|
|
||||||
wantChanges := []int{
|
wantChanges := []int{maxTableChangesPerWrite, pairs - maxTableChangesPerWrite}
|
||||||
testMaxTableChangesPerWrite, pairs - testMaxTableChangesPerWrite,
|
|
||||||
}
|
|
||||||
wantNewPeers := []int{1, 0}
|
wantNewPeers := []int{1, 0}
|
||||||
|
|
||||||
if !slices.Equal(store.changes, wantChanges) ||
|
if !slices.Equal(store.changes, wantChanges) ||
|
||||||
@@ -790,11 +415,11 @@ func TestPrefixHandlerWritesManyTableChangesOverSeveralFlushes(t *testing.T) {
|
|||||||
// takes. Its last flush writes one write's worth, so Stop must log the pairs
|
// takes. Its last flush writes one write's worth, so Stop must log the pairs
|
||||||
// that no write took, and must not say that the stop deadline passed. The
|
// that no write took, and must not say that the stop deadline passed. The
|
||||||
// flush timer must not fire, or its writes could leave no pairs for Stop to
|
// flush timer must not fire, or its writes could leave no pairs for Stop to
|
||||||
// log. The handler's limit of one write is small.
|
// log.
|
||||||
func TestPrefixHandlerStopLogsTheTableChangesLeftUnwritten(t *testing.T) {
|
func TestPrefixHandlerStopLogsTheTableChangesLeftUnwritten(t *testing.T) {
|
||||||
t.Parallel()
|
t.Parallel()
|
||||||
|
|
||||||
const pairs = testMaxTableChangesPerWrite + 1
|
const pairs = maxTableChangesPerWrite + 1
|
||||||
|
|
||||||
var logs bytes.Buffer
|
var logs bytes.Buffer
|
||||||
|
|
||||||
@@ -802,8 +427,7 @@ func TestPrefixHandlerStopLogsTheTableChangesLeftUnwritten(t *testing.T) {
|
|||||||
table := routetable.New(routetable.Params{})
|
table := routetable.New(routetable.Params{})
|
||||||
announceManyPrefixes(table, pairs)
|
announceManyPrefixes(table, pairs)
|
||||||
|
|
||||||
h := newPrefixHandler(store, newErrorLogger(&logs), table, longerThanAnyTest,
|
h := newPrefixHandler(store, newErrorLogger(&logs), table, longerThanAnyTest)
|
||||||
testMaxTableChangesPerWrite)
|
|
||||||
h.Stop(t.Context())
|
h.Stop(t.Context())
|
||||||
|
|
||||||
written := 0
|
written := 0
|
||||||
@@ -865,7 +489,6 @@ func TestARestartReadsTheRoutingTableBack(t *testing.T) {
|
|||||||
h.HandleMessage(msg)
|
h.HandleMessage(msg)
|
||||||
}
|
}
|
||||||
|
|
||||||
h.peerDowns.Wait()
|
|
||||||
h.flushBatch()
|
h.flushBatch()
|
||||||
|
|
||||||
var logs bytes.Buffer
|
var logs bytes.Buffer
|
||||||
@@ -904,7 +527,6 @@ func TestARestartReadsTheRoutingTableBack(t *testing.T) {
|
|||||||
restarted.HandleMessage(msg)
|
restarted.HandleMessage(msg)
|
||||||
}
|
}
|
||||||
|
|
||||||
h.peerDowns.Wait()
|
|
||||||
restarted.Stop(t.Context())
|
restarted.Stop(t.Context())
|
||||||
|
|
||||||
read, err := db.ReadRouteTable(t.Context())
|
read, err := db.ReadRouteTable(t.Context())
|
||||||
|
|||||||
+125
-335
@@ -15,16 +15,15 @@ import (
|
|||||||
"strings"
|
"strings"
|
||||||
"time"
|
"time"
|
||||||
|
|
||||||
"git.eeqj.de/sneak/routewatch/internal/bogon"
|
"git.eeqj.de/sneak/routewatch/internal/blocklist"
|
||||||
"git.eeqj.de/sneak/routewatch/internal/database"
|
"git.eeqj.de/sneak/routewatch/internal/database"
|
||||||
"git.eeqj.de/sneak/routewatch/internal/geo"
|
|
||||||
"git.eeqj.de/sneak/routewatch/internal/iplist"
|
|
||||||
"git.eeqj.de/sneak/routewatch/internal/metrics"
|
"git.eeqj.de/sneak/routewatch/internal/metrics"
|
||||||
"git.eeqj.de/sneak/routewatch/internal/ristypes"
|
"git.eeqj.de/sneak/routewatch/internal/ristypes"
|
||||||
"git.eeqj.de/sneak/routewatch/internal/routetable"
|
"git.eeqj.de/sneak/routewatch/internal/routetable"
|
||||||
"git.eeqj.de/sneak/routewatch/internal/streamer"
|
"git.eeqj.de/sneak/routewatch/internal/streamer"
|
||||||
"git.eeqj.de/sneak/routewatch/internal/templates"
|
"git.eeqj.de/sneak/routewatch/internal/templates"
|
||||||
asinfo "git.eeqj.de/sneak/routewatch/pkg/asinfo"
|
asinfo "git.eeqj.de/sneak/routewatch/pkg/asinfo"
|
||||||
|
"github.com/dustin/go-humanize"
|
||||||
"github.com/go-chi/chi/v5"
|
"github.com/go-chi/chi/v5"
|
||||||
)
|
)
|
||||||
|
|
||||||
@@ -40,9 +39,12 @@ const (
|
|||||||
const (
|
const (
|
||||||
// maxLookupTargets is the most addresses and host names an IP lookup may
|
// maxLookupTargets is the most addresses and host names an IP lookup may
|
||||||
// list, and the most addresses it may have once the names are resolved:
|
// list, and the most addresses it may have once the names are resolved:
|
||||||
// each address costs two database reads and a PTR lookup. Ten PTR lookups
|
// each address costs two database reads and a PTR lookup.
|
||||||
// that run to ptr.LookupTimeout fit in the request's 30-second limit.
|
|
||||||
maxLookupTargets = 10
|
maxLookupTargets = 10
|
||||||
|
|
||||||
|
// ptrLookupTimeout is each PTR lookup's own time limit, so that ten
|
||||||
|
// addresses whose PTR lookups never answer fit in the 30-second limit.
|
||||||
|
ptrLookupTimeout = 2 * time.Second
|
||||||
)
|
)
|
||||||
|
|
||||||
// The Cache-Control header of an IP lookup answer, such as
|
// The Cache-Control header of an IP lookup answer, such as
|
||||||
@@ -87,10 +89,6 @@ const (
|
|||||||
ipVersionV6 = 6
|
ipVersionV6 = 6
|
||||||
)
|
)
|
||||||
|
|
||||||
// maxOriginChangesOnPage is the most origin changes the prefix page shows. A
|
|
||||||
// prefix with more shows the newest ones and says so.
|
|
||||||
const maxOriginChangesOnPage = 5000
|
|
||||||
|
|
||||||
// HealthCheckResponse represents the health check response.
|
// HealthCheckResponse represents the health check response.
|
||||||
type HealthCheckResponse struct {
|
type HealthCheckResponse struct {
|
||||||
Status string `json:"status"`
|
Status string `json:"status"`
|
||||||
@@ -162,7 +160,6 @@ func (s *Server) handleIndex() http.HandlerFunc {
|
|||||||
err := tmpl.Execute(w, indexPageData{
|
err := tmpl.Execute(w, indexPageData{
|
||||||
CSPNonce: cspNonce(r.Context()),
|
CSPNonce: cspNonce(r.Context()),
|
||||||
ClientIP: extractClientIP(r),
|
ClientIP: extractClientIP(r),
|
||||||
Cards: s.cards.Load(),
|
|
||||||
})
|
})
|
||||||
if err != nil {
|
if err != nil {
|
||||||
s.logger.Error("Failed to render index template", "error", err)
|
s.logger.Error("Failed to render index template", "error", err)
|
||||||
@@ -184,9 +181,6 @@ type indexPageData struct {
|
|||||||
// ClientIP is the visitor's address, which the IP information API card
|
// ClientIP is the visitor's address, which the IP information API card
|
||||||
// looks up: the address /ip/ would look up for the same request.
|
// looks up: the address /ip/ would look up for the same request.
|
||||||
ClientIP string
|
ClientIP string
|
||||||
// Cards is the copy of the cards that show figures of the routing table,
|
|
||||||
// or nil before the server has first worked them out.
|
|
||||||
Cards *homeCards
|
|
||||||
}
|
}
|
||||||
|
|
||||||
// writeJSONError writes a standardized JSON error response with the given
|
// writeJSONError writes a standardized JSON error response with the given
|
||||||
@@ -228,23 +222,18 @@ type WHOISStatsInfo struct {
|
|||||||
NeverFetched int `json:"never_fetched"`
|
NeverFetched int `json:"never_fetched"`
|
||||||
SuccessesLastHour int `json:"successes_last_hour"`
|
SuccessesLastHour int `json:"successes_last_hour"`
|
||||||
ErrorsLastHour int `json:"errors_last_hour"`
|
ErrorsLastHour int `json:"errors_last_hour"`
|
||||||
|
CurrentInterval string `json:"current_interval"`
|
||||||
ConsecutiveFails int `json:"consecutive_fails"`
|
ConsecutiveFails int `json:"consecutive_fails"`
|
||||||
FreshPercent float64 `json:"fresh_percent"`
|
FreshPercent float64 `json:"fresh_percent"`
|
||||||
|
|
||||||
// CurrentIntervalSeconds is the wait between two fetches. It can have a
|
|
||||||
// fractional part, such as 22.5.
|
|
||||||
CurrentIntervalSeconds float64 `json:"current_interval_seconds"`
|
|
||||||
}
|
}
|
||||||
|
|
||||||
// Stats is the /status.json answer: uptime, message counts, database stats,
|
// Stats is the /status.json answer: uptime, message counts, database stats,
|
||||||
// and the routing table's figures. UptimeSeconds is the time since the stream
|
// and the routing table's figures. OldestRoute and NewestRoute are the times
|
||||||
// last connected, or 0 if it has never connected. MemoryUsageBytes is the Go
|
// of the oldest and the newest kept routes.
|
||||||
// heap in use. OldestRoute and NewestRoute are the times of the oldest and the
|
|
||||||
// newest kept routes.
|
|
||||||
//
|
//
|
||||||
//nolint:tagliatelle // field names are the existing JSON API
|
//nolint:tagliatelle // field names are the existing JSON API
|
||||||
type Stats struct {
|
type Stats struct {
|
||||||
UptimeSeconds int64 `json:"uptime_seconds"`
|
Uptime string `json:"uptime"`
|
||||||
TotalMessages uint64 `json:"total_messages"`
|
TotalMessages uint64 `json:"total_messages"`
|
||||||
TotalBytes uint64 `json:"total_bytes"`
|
TotalBytes uint64 `json:"total_bytes"`
|
||||||
TotalWireBytes uint64 `json:"total_wire_bytes"`
|
TotalWireBytes uint64 `json:"total_wire_bytes"`
|
||||||
@@ -254,7 +243,7 @@ type Stats struct {
|
|||||||
Connected bool `json:"connected"`
|
Connected bool `json:"connected"`
|
||||||
GoVersion string `json:"go_version"`
|
GoVersion string `json:"go_version"`
|
||||||
Goroutines int `json:"goroutines"`
|
Goroutines int `json:"goroutines"`
|
||||||
MemoryUsageBytes uint64 `json:"memory_usage_bytes"`
|
MemoryUsage string `json:"memory_usage"`
|
||||||
ASNs int `json:"asns"`
|
ASNs int `json:"asns"`
|
||||||
Prefixes int `json:"prefixes"`
|
Prefixes int `json:"prefixes"`
|
||||||
IPv4Prefixes int `json:"ipv4_prefixes"`
|
IPv4Prefixes int `json:"ipv4_prefixes"`
|
||||||
@@ -369,7 +358,7 @@ func (s *Server) statusStats(
|
|||||||
runtime.ReadMemStats(&memStats)
|
runtime.ReadMemStats(&memStats)
|
||||||
|
|
||||||
return Stats{
|
return Stats{
|
||||||
UptimeSeconds: uptimeSeconds(streamMetrics.ConnectedSince),
|
Uptime: streamUptime(streamMetrics.ConnectedSince),
|
||||||
TotalMessages: streamMetrics.TotalMessages,
|
TotalMessages: streamMetrics.TotalMessages,
|
||||||
TotalBytes: streamMetrics.TotalBytes,
|
TotalBytes: streamMetrics.TotalBytes,
|
||||||
TotalWireBytes: streamMetrics.TotalWireBytes,
|
TotalWireBytes: streamMetrics.TotalWireBytes,
|
||||||
@@ -379,7 +368,7 @@ func (s *Server) statusStats(
|
|||||||
Connected: streamMetrics.Connected,
|
Connected: streamMetrics.Connected,
|
||||||
GoVersion: runtime.Version(),
|
GoVersion: runtime.Version(),
|
||||||
Goroutines: runtime.NumGoroutine(),
|
Goroutines: runtime.NumGoroutine(),
|
||||||
MemoryUsageBytes: memStats.Alloc,
|
MemoryUsage: humanize.Bytes(memStats.Alloc),
|
||||||
ASNs: dbStats.ASNs,
|
ASNs: dbStats.ASNs,
|
||||||
Prefixes: counts.IPv4Prefixes + counts.IPv6Prefixes,
|
Prefixes: counts.IPv4Prefixes + counts.IPv6Prefixes,
|
||||||
IPv4Prefixes: counts.IPv4Prefixes,
|
IPv4Prefixes: counts.IPv4Prefixes,
|
||||||
@@ -446,31 +435,30 @@ func (s *Server) getWHOISStats(ctx context.Context) *WHOISStatsInfo {
|
|||||||
NeverFetched: dbStats.NeverFetched,
|
NeverFetched: dbStats.NeverFetched,
|
||||||
SuccessesLastHour: fetcherStats.SuccessesLastHour,
|
SuccessesLastHour: fetcherStats.SuccessesLastHour,
|
||||||
ErrorsLastHour: fetcherStats.ErrorsLastHour,
|
ErrorsLastHour: fetcherStats.ErrorsLastHour,
|
||||||
|
CurrentInterval: fetcherStats.CurrentInterval.String(),
|
||||||
ConsecutiveFails: fetcherStats.ConsecutiveFails,
|
ConsecutiveFails: fetcherStats.ConsecutiveFails,
|
||||||
FreshPercent: freshPercent,
|
FreshPercent: freshPercent,
|
||||||
|
|
||||||
CurrentIntervalSeconds: fetcherStats.CurrentInterval.Seconds(),
|
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// uptimeSeconds is the number of whole seconds since the stream last
|
// streamUptime is how long the stream has been connected, to the second, or
|
||||||
// connected, or 0 if it has never connected.
|
// "0s" if it has never connected.
|
||||||
func uptimeSeconds(connectedSince time.Time) int64 {
|
func streamUptime(connectedSince time.Time) string {
|
||||||
if connectedSince.IsZero() {
|
if connectedSince.IsZero() {
|
||||||
return 0
|
return "0s"
|
||||||
}
|
}
|
||||||
|
|
||||||
return int64(time.Since(connectedSince) / time.Second)
|
return time.Since(connectedSince).Truncate(time.Second).String()
|
||||||
}
|
}
|
||||||
|
|
||||||
// connectionDurationSeconds is the number of whole seconds the stream has
|
// connectionDuration is how long the stream has been connected, to the
|
||||||
// been connected, or nil while it is not connected.
|
// second, or "disconnected".
|
||||||
func connectionDurationSeconds(connected bool, connectedSince time.Time) *int64 {
|
func connectionDuration(connected bool, connectedSince time.Time) string {
|
||||||
if !connected || connectedSince.IsZero() {
|
if !connected || connectedSince.IsZero() {
|
||||||
return nil
|
return "disconnected"
|
||||||
}
|
}
|
||||||
|
|
||||||
return new(uptimeSeconds(connectedSince))
|
return time.Since(connectedSince).Truncate(time.Second).String()
|
||||||
}
|
}
|
||||||
|
|
||||||
// whoisStaleThreshold matches the fetcher's threshold for consistency.
|
// whoisStaleThreshold matches the fetcher's threshold for consistency.
|
||||||
@@ -489,7 +477,6 @@ type HandlerStatsInfo struct {
|
|||||||
QueueHighWaterMark int `json:"queue_high_water_mark"`
|
QueueHighWaterMark int `json:"queue_high_water_mark"`
|
||||||
ProcessedCount uint64 `json:"processed_count"`
|
ProcessedCount uint64 `json:"processed_count"`
|
||||||
DroppedCount uint64 `json:"dropped_count"`
|
DroppedCount uint64 `json:"dropped_count"`
|
||||||
HeldBackCount int `json:"held_back_count"`
|
|
||||||
AvgProcessTimeMs float64 `json:"avg_process_time_ms"`
|
AvgProcessTimeMs float64 `json:"avg_process_time_ms"`
|
||||||
MinProcessTimeMs float64 `json:"min_process_time_ms"`
|
MinProcessTimeMs float64 `json:"min_process_time_ms"`
|
||||||
MaxProcessTimeMs float64 `json:"max_process_time_ms"`
|
MaxProcessTimeMs float64 `json:"max_process_time_ms"`
|
||||||
@@ -521,7 +508,7 @@ type StreamStats struct {
|
|||||||
//
|
//
|
||||||
//nolint:tagliatelle // field names are the existing JSON API
|
//nolint:tagliatelle // field names are the existing JSON API
|
||||||
type StatsResponse struct {
|
type StatsResponse struct {
|
||||||
UptimeSeconds int64 `json:"uptime_seconds"`
|
Uptime string `json:"uptime"`
|
||||||
TotalMessages uint64 `json:"total_messages"`
|
TotalMessages uint64 `json:"total_messages"`
|
||||||
TotalBytes uint64 `json:"total_bytes"`
|
TotalBytes uint64 `json:"total_bytes"`
|
||||||
TotalWireBytes uint64 `json:"total_wire_bytes"`
|
TotalWireBytes uint64 `json:"total_wire_bytes"`
|
||||||
@@ -529,10 +516,11 @@ type StatsResponse struct {
|
|||||||
MbitsPerSec float64 `json:"mbits_per_sec"`
|
MbitsPerSec float64 `json:"mbits_per_sec"`
|
||||||
WireMbitsPerSec float64 `json:"wire_mbits_per_sec"`
|
WireMbitsPerSec float64 `json:"wire_mbits_per_sec"`
|
||||||
Connected bool `json:"connected"`
|
Connected bool `json:"connected"`
|
||||||
|
ConnectionDuration string `json:"connection_duration"`
|
||||||
ReconnectCount uint64 `json:"reconnect_count"`
|
ReconnectCount uint64 `json:"reconnect_count"`
|
||||||
GoVersion string `json:"go_version"`
|
GoVersion string `json:"go_version"`
|
||||||
Goroutines int `json:"goroutines"`
|
Goroutines int `json:"goroutines"`
|
||||||
MemoryUsageBytes uint64 `json:"memory_usage_bytes"`
|
MemoryUsage string `json:"memory_usage"`
|
||||||
GC GCStats `json:"gc"`
|
GC GCStats `json:"gc"`
|
||||||
Stream StreamStats `json:"stream"`
|
Stream StreamStats `json:"stream"`
|
||||||
ASNs int `json:"asns"`
|
ASNs int `json:"asns"`
|
||||||
@@ -554,9 +542,6 @@ type StatsResponse struct {
|
|||||||
IPv6PrefixDistribution []PrefixDistribution `json:"ipv6_prefix_distribution"`
|
IPv6PrefixDistribution []PrefixDistribution `json:"ipv6_prefix_distribution"`
|
||||||
WHOISStats *WHOISStatsInfo `json:"whois_stats,omitempty"`
|
WHOISStats *WHOISStatsInfo `json:"whois_stats,omitempty"`
|
||||||
|
|
||||||
// ConnectionDurationSeconds is null while the stream is not connected.
|
|
||||||
ConnectionDurationSeconds *int64 `json:"connection_duration_seconds"`
|
|
||||||
|
|
||||||
OriginsWithoutKeptRoute int `json:"origins_without_kept_route"`
|
OriginsWithoutKeptRoute int `json:"origins_without_kept_route"`
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -574,7 +559,6 @@ func handlerStatsInfo(handlerStats []streamer.HandlerStats) []HandlerStatsInfo {
|
|||||||
QueueHighWaterMark: hs.QueueHighWaterMark,
|
QueueHighWaterMark: hs.QueueHighWaterMark,
|
||||||
ProcessedCount: hs.ProcessedCount,
|
ProcessedCount: hs.ProcessedCount,
|
||||||
DroppedCount: hs.DroppedCount,
|
DroppedCount: hs.DroppedCount,
|
||||||
HeldBackCount: hs.HeldBackCount,
|
|
||||||
AvgProcessTimeMs: float64(hs.AvgProcessTime.Microseconds()) /
|
AvgProcessTimeMs: float64(hs.AvgProcessTime.Microseconds()) /
|
||||||
microsecondsPerMillisecond,
|
microsecondsPerMillisecond,
|
||||||
MinProcessTimeMs: float64(hs.MinProcessTime.Microseconds()) /
|
MinProcessTimeMs: float64(hs.MinProcessTime.Microseconds()) /
|
||||||
@@ -696,18 +680,20 @@ func (s *Server) statsResponse(
|
|||||||
runtime.ReadMemStats(&memStats)
|
runtime.ReadMemStats(&memStats)
|
||||||
|
|
||||||
return StatsResponse{
|
return StatsResponse{
|
||||||
UptimeSeconds: uptimeSeconds(streamMetrics.ConnectedSince),
|
Uptime: streamUptime(streamMetrics.ConnectedSince),
|
||||||
TotalMessages: streamMetrics.TotalMessages,
|
TotalMessages: streamMetrics.TotalMessages,
|
||||||
TotalBytes: streamMetrics.TotalBytes,
|
TotalBytes: streamMetrics.TotalBytes,
|
||||||
TotalWireBytes: streamMetrics.TotalWireBytes,
|
TotalWireBytes: streamMetrics.TotalWireBytes,
|
||||||
MessagesPerSec: streamMetrics.MessagesPerSec,
|
MessagesPerSec: streamMetrics.MessagesPerSec,
|
||||||
MbitsPerSec: streamMetrics.BitsPerSec / bitsPerMegabit,
|
MbitsPerSec: streamMetrics.BitsPerSec / bitsPerMegabit,
|
||||||
WireMbitsPerSec: streamMetrics.WireBitsPerSec / bitsPerMegabit,
|
WireMbitsPerSec: streamMetrics.WireBitsPerSec / bitsPerMegabit,
|
||||||
Connected: streamMetrics.Connected,
|
Connected: streamMetrics.Connected,
|
||||||
|
ConnectionDuration: connectionDuration(
|
||||||
|
streamMetrics.Connected, streamMetrics.ConnectedSince),
|
||||||
ReconnectCount: streamMetrics.ReconnectCount,
|
ReconnectCount: streamMetrics.ReconnectCount,
|
||||||
GoVersion: runtime.Version(),
|
GoVersion: runtime.Version(),
|
||||||
Goroutines: runtime.NumGoroutine(),
|
Goroutines: runtime.NumGoroutine(),
|
||||||
MemoryUsageBytes: memStats.Alloc,
|
MemoryUsage: humanize.Bytes(memStats.Alloc),
|
||||||
GC: gcStats(&memStats),
|
GC: gcStats(&memStats),
|
||||||
Stream: s.streamStats(),
|
Stream: s.streamStats(),
|
||||||
ASNs: dbStats.ASNs,
|
ASNs: dbStats.ASNs,
|
||||||
@@ -729,9 +715,6 @@ func (s *Server) statsResponse(
|
|||||||
IPv6PrefixDistribution: prefixDistribution(counts.IPv6PrefixesByLength),
|
IPv6PrefixDistribution: prefixDistribution(counts.IPv6PrefixesByLength),
|
||||||
WHOISStats: whoisStats,
|
WHOISStats: whoisStats,
|
||||||
|
|
||||||
ConnectionDurationSeconds: connectionDurationSeconds(
|
|
||||||
streamMetrics.Connected, streamMetrics.ConnectedSince),
|
|
||||||
|
|
||||||
OriginsWithoutKeptRoute: counts.OriginsWithoutKeptRoute,
|
OriginsWithoutKeptRoute: counts.OriginsWithoutKeptRoute,
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -764,48 +747,19 @@ type IPLookupResponse struct {
|
|||||||
Errors []string `json:"errors,omitempty"`
|
Errors []string `json:"errors,omitempty"`
|
||||||
}
|
}
|
||||||
|
|
||||||
// IPLookupResult is the answer for one address: its PTR names, its flags, its
|
// IPLookupResult is the answer for one address: its PTR names, its live route
|
||||||
// live route and that route's AS, the IP lists that contain it, and where it
|
// and that route's AS, and the blocklists that list it. IPInfo is nil when
|
||||||
// is. Every result has every field. The fields of IPInfo are null when the
|
// the address has no live route, and the answer then has no route or AS
|
||||||
// address has no live route.
|
// fields. IP hides the IPInfo field of the same name, so that an address with
|
||||||
//
|
// no route has it too.
|
||||||
//nolint:tagliatelle // snake_case keys, as the Go style guide asks
|
|
||||||
type IPLookupResult struct {
|
type IPLookupResult struct {
|
||||||
IPInfo
|
*IPInfo
|
||||||
|
|
||||||
IP string `json:"ip"`
|
IP string `json:"ip"`
|
||||||
// PTR is empty, never nil, when the address has no PTR names.
|
PTR []string `json:"ptr,omitempty"`
|
||||||
PTR []string `json:"ptr"`
|
|
||||||
|
|
||||||
Flags IPLookupFlags `json:"flags"`
|
// Blocklists has one entry for each blocklist the address is on.
|
||||||
|
Blocklists []blocklist.Listing `json:"blocklists"`
|
||||||
// VPNProviders names the VPN lists the address is on. It is empty, never
|
|
||||||
// nil, when there are none.
|
|
||||||
VPNProviders []string `json:"vpn_providers"`
|
|
||||||
|
|
||||||
// IPLists has one entry for each IP list the address is on.
|
|
||||||
IPLists []iplist.Listing `json:"ip_lists"`
|
|
||||||
|
|
||||||
// Geo is where the address is. It is nil when no source answers.
|
|
||||||
Geo *geo.Location `json:"geo"`
|
|
||||||
}
|
|
||||||
|
|
||||||
// IPLookupFlags are the yes-or-no facts about an address. Every result has
|
|
||||||
// them, whether or not the address has a live route.
|
|
||||||
//
|
|
||||||
//nolint:tagliatelle // snake_case keys, as the Go style guide asks
|
|
||||||
type IPLookupFlags struct {
|
|
||||||
// IsTor says that the address is on the Tor exit list.
|
|
||||||
IsTor bool `json:"is_tor"`
|
|
||||||
|
|
||||||
// IsVPN says that the address is on a list of VPN servers.
|
|
||||||
IsVPN bool `json:"is_vpn"`
|
|
||||||
|
|
||||||
// IsRelay says that the address is on the iCloud Private Relay list.
|
|
||||||
IsRelay bool `json:"is_relay"`
|
|
||||||
|
|
||||||
// IsBogon says that the address is reserved, as package bogon tells.
|
|
||||||
IsBogon bool `json:"is_bogon"`
|
|
||||||
}
|
}
|
||||||
|
|
||||||
// handleIPInfo returns a handler that provides comprehensive IP information.
|
// handleIPInfo returns a handler that provides comprehensive IP information.
|
||||||
@@ -846,8 +800,6 @@ func (s *Server) handleIPInfo() http.HandlerFunc {
|
|||||||
return
|
return
|
||||||
}
|
}
|
||||||
|
|
||||||
locations := s.geoLocations(ctx, ipsToLookup)
|
|
||||||
|
|
||||||
// Track ASNs that need WHOIS refresh
|
// Track ASNs that need WHOIS refresh
|
||||||
refreshASNs := make(map[int]bool)
|
refreshASNs := make(map[int]bool)
|
||||||
|
|
||||||
@@ -862,8 +814,8 @@ func (s *Server) handleIPInfo() http.HandlerFunc {
|
|||||||
|
|
||||||
switch {
|
switch {
|
||||||
case errors.Is(err, ErrNoRoute):
|
case errors.Is(err, ErrNoRoute):
|
||||||
// The address still gets a result, with every field of
|
// The address still gets a result, without route fields.
|
||||||
// ipInfo null.
|
ipInfo = nil
|
||||||
case errors.Is(err, ErrInvalidIP):
|
case errors.Is(err, ErrInvalidIP):
|
||||||
response.Errors = append(response.Errors, ip+": "+err.Error())
|
response.Errors = append(response.Errors, ip+": "+err.Error())
|
||||||
|
|
||||||
@@ -883,12 +835,15 @@ func (s *Server) handleIPInfo() http.HandlerFunc {
|
|||||||
return
|
return
|
||||||
}
|
}
|
||||||
|
|
||||||
result := s.ipLookupResult(ip, ptrs, ipInfo)
|
response.Results = append(response.Results, IPLookupResult{
|
||||||
result.Geo = locations[ip]
|
IP: ip,
|
||||||
response.Results = append(response.Results, result)
|
PTR: ptrs,
|
||||||
|
IPInfo: ipInfo,
|
||||||
|
Blocklists: s.blocklistListings(ip),
|
||||||
|
})
|
||||||
|
|
||||||
if ipInfo.NeedsWHOISRefresh {
|
if ipInfo != nil && ipInfo.NeedsWHOISRefresh {
|
||||||
refreshASNs[*ipInfo.ASN] = true
|
refreshASNs[ipInfo.ASN] = true
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -899,72 +854,60 @@ func (s *Server) handleIPInfo() http.HandlerFunc {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// ipInfo returns what the IP lookup knows about the route of ip. The routing
|
// ipInfo returns the IP lookup's answer for ip. The routing table gives the
|
||||||
// table gives the most specific prefix that holds the address, the number of
|
// most specific prefix that holds the address, the number of peers that carry
|
||||||
// peers that carry it, its origin ASes and the one that most peers carry, and
|
// it, the origin AS that most of them carry, and the time of its newest kept
|
||||||
// the time of its newest kept route. The database gives the prefix's
|
// route. The database gives the prefix's first-seen time and the AS's WHOIS
|
||||||
// first-seen time and the AS's WHOIS details. It returns ErrNoRoute when no
|
// details. It returns ErrNoRoute when no prefix holds the address, and
|
||||||
// prefix holds the address, and ErrInvalidIP when ip does not parse. With an
|
// ErrInvalidIP when ip does not parse.
|
||||||
// error, every field of the IPInfo it returns is nil.
|
func (s *Server) ipInfo(ctx context.Context, ip string) (*IPInfo, error) {
|
||||||
func (s *Server) ipInfo(ctx context.Context, ip string) (IPInfo, error) {
|
|
||||||
addr, err := netip.ParseAddr(ip)
|
addr, err := netip.ParseAddr(ip)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return IPInfo{}, fmt.Errorf("%w: %s", ErrInvalidIP, ip)
|
return nil, fmt.Errorf("%w: %s", ErrInvalidIP, ip)
|
||||||
}
|
}
|
||||||
|
|
||||||
summary, found := s.table.Lookup(addr)
|
summary, found := s.table.Lookup(addr)
|
||||||
if !found {
|
if !found {
|
||||||
return IPInfo{}, fmt.Errorf("%w for IP %s", ErrNoRoute, ip)
|
return nil, fmt.Errorf("%w for IP %s", ErrNoRoute, ip)
|
||||||
}
|
}
|
||||||
|
|
||||||
firstSeen, err := s.db.GetPrefixFirstSeen(ctx, summary.Prefix)
|
firstSeen, err := s.db.GetPrefixFirstSeen(ctx, summary.Prefix)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return IPInfo{}, err
|
return nil, err
|
||||||
}
|
}
|
||||||
|
|
||||||
asInfo, err := s.db.GetASWHOIS(ctx, summary.OriginASN)
|
asInfo, err := s.db.GetASWHOIS(ctx, summary.OriginASN)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return IPInfo{}, err
|
return nil, err
|
||||||
}
|
}
|
||||||
|
|
||||||
info := IPInfo{
|
info := &IPInfo{
|
||||||
Netblock: new(summary.Prefix.String()),
|
IP: ip,
|
||||||
MaskLength: new(summary.Prefix.Bits()),
|
Netblock: summary.Prefix.String(),
|
||||||
IPVersion: new(ipVersionOf(summary.Prefix)),
|
MaskLength: summary.Prefix.Bits(),
|
||||||
NumPeers: new(summary.Peers),
|
IPVersion: ipVersionOf(summary.Prefix),
|
||||||
ASN: new(summary.OriginASN),
|
NumPeers: summary.Peers,
|
||||||
NumOrigins: new(len(summary.OriginASNs)),
|
ASN: summary.OriginASN,
|
||||||
OriginASNs: summary.OriginASNs,
|
ASName: asInfo.ASName,
|
||||||
ASName: nilIfEmpty(asInfo.ASName),
|
Handle: asInfo.Handle,
|
||||||
Handle: nilIfEmpty(asInfo.Handle),
|
Description: asInfo.Description,
|
||||||
Description: nilIfEmpty(asInfo.Description),
|
OrgName: asInfo.OrgName,
|
||||||
OrgName: nilIfEmpty(asInfo.OrgName),
|
OrgID: asInfo.OrgID,
|
||||||
OrgID: nilIfEmpty(asInfo.OrgID),
|
Address: asInfo.Address,
|
||||||
Address: nilIfEmpty(asInfo.Address),
|
CountryCode: asInfo.CountryCode,
|
||||||
CountryCode: nilIfEmpty(asInfo.CountryCode),
|
AbuseEmail: asInfo.AbuseEmail,
|
||||||
AbuseEmail: nilIfEmpty(asInfo.AbuseEmail),
|
RIR: asInfo.RIR,
|
||||||
RIR: nilIfEmpty(asInfo.RIR),
|
FirstSeen: firstSeen,
|
||||||
LastSeen: summary.Newest,
|
|
||||||
NeedsWHOISRefresh: needsWHOISRefresh(asInfo),
|
NeedsWHOISRefresh: needsWHOISRefresh(asInfo),
|
||||||
}
|
}
|
||||||
|
|
||||||
if !firstSeen.IsZero() {
|
if summary.Newest != nil {
|
||||||
info.FirstSeen = &firstSeen
|
info.LastSeen = *summary.Newest
|
||||||
}
|
}
|
||||||
|
|
||||||
return info, nil
|
return info, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
// nilIfEmpty returns a pointer to s, or nil when s is empty, so that JSON
|
|
||||||
// gives a detail the database does not hold as null.
|
|
||||||
func nilIfEmpty(s string) *string {
|
|
||||||
if s == "" {
|
|
||||||
return nil
|
|
||||||
}
|
|
||||||
|
|
||||||
return &s
|
|
||||||
}
|
|
||||||
|
|
||||||
// needsWHOISRefresh says whether the WHOIS details of asInfo were never
|
// needsWHOISRefresh says whether the WHOIS details of asInfo were never
|
||||||
// fetched or are older than whoisStaleThreshold. AS 0, no known origin, has
|
// fetched or are older than whoisStaleThreshold. AS 0, no known origin, has
|
||||||
// none to fetch.
|
// none to fetch.
|
||||||
@@ -986,120 +929,23 @@ func ipVersionOf(prefix netip.Prefix) int {
|
|||||||
return ipVersionV6
|
return ipVersionV6
|
||||||
}
|
}
|
||||||
|
|
||||||
// ipLookupResult returns the result for ip, adding its flags and IP lists to
|
// blocklistListings returns the blocklists that list ip. It returns an empty
|
||||||
// its PTR names and route.
|
// list, never nil, so that an answer's blocklists is [] rather than null.
|
||||||
func (s *Server) ipLookupResult(ip string, ptrs []string, info IPInfo) IPLookupResult {
|
func (s *Server) blocklistListings(ip string) []blocklist.Listing {
|
||||||
listings := s.ipListListings(ip)
|
|
||||||
|
|
||||||
return IPLookupResult{
|
|
||||||
IP: ip,
|
|
||||||
PTR: ptrs,
|
|
||||||
IPInfo: info,
|
|
||||||
Flags: ipLookupFlags(ip, listings),
|
|
||||||
VPNProviders: vpnProviders(listings),
|
|
||||||
IPLists: listings,
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// vpnProviders returns the names of the VPN lists among listings. It returns
|
|
||||||
// an empty list, never nil, so that an answer's vpn_providers is [] rather
|
|
||||||
// than null.
|
|
||||||
func vpnProviders(listings []iplist.Listing) []string {
|
|
||||||
names := []string{}
|
|
||||||
|
|
||||||
for _, listing := range listings {
|
|
||||||
if listing.VPN {
|
|
||||||
names = append(names, listing.Name)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
return names
|
|
||||||
}
|
|
||||||
|
|
||||||
// ipListListings returns the IP lists that contain ip. It returns an empty
|
|
||||||
// list, never nil, so that an answer's ip_lists is [] rather than null.
|
|
||||||
func (s *Server) ipListListings(ip string) []iplist.Listing {
|
|
||||||
addr, err := netip.ParseAddr(ip)
|
addr, err := netip.ParseAddr(ip)
|
||||||
if s.ipLists == nil || err != nil {
|
if s.blocklists == nil || err != nil {
|
||||||
return []iplist.Listing{}
|
return []blocklist.Listing{}
|
||||||
}
|
}
|
||||||
|
|
||||||
return s.ipLists.Lookup(addr)
|
return s.blocklists.Lookup(addr)
|
||||||
}
|
|
||||||
|
|
||||||
// ipLookupFlags returns the flags of ip, whose IP list entries are listings.
|
|
||||||
func ipLookupFlags(ip string, listings []iplist.Listing) IPLookupFlags {
|
|
||||||
return IPLookupFlags{
|
|
||||||
IsTor: slices.ContainsFunc(listings, func(l iplist.Listing) bool {
|
|
||||||
return l.Name == iplist.TorExitsList
|
|
||||||
}),
|
|
||||||
IsVPN: slices.ContainsFunc(listings, func(l iplist.Listing) bool {
|
|
||||||
return l.VPN
|
|
||||||
}),
|
|
||||||
IsRelay: slices.ContainsFunc(listings, func(l iplist.Listing) bool {
|
|
||||||
return l.Name == iplist.ICloudPrivateRelayList
|
|
||||||
}),
|
|
||||||
IsBogon: isBogon(ip),
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// geoLocations returns where ips are, by address. An address that does not
|
|
||||||
// parse has no entry. Nor has a reserved one, which is sent to no outside
|
|
||||||
// source and has no location.
|
|
||||||
func (s *Server) geoLocations(
|
|
||||||
ctx context.Context, ips []string,
|
|
||||||
) map[string]*geo.Location {
|
|
||||||
if s.geoLocator == nil {
|
|
||||||
return nil
|
|
||||||
}
|
|
||||||
|
|
||||||
var (
|
|
||||||
kept []string
|
|
||||||
addrs []netip.Addr
|
|
||||||
)
|
|
||||||
|
|
||||||
for _, ip := range ips {
|
|
||||||
addr, err := netip.ParseAddr(ip)
|
|
||||||
if err != nil || isBogon(ip) {
|
|
||||||
continue
|
|
||||||
}
|
|
||||||
|
|
||||||
kept = append(kept, ip)
|
|
||||||
addrs = append(addrs, addr)
|
|
||||||
}
|
|
||||||
|
|
||||||
if len(addrs) == 0 {
|
|
||||||
return nil
|
|
||||||
}
|
|
||||||
|
|
||||||
answers := s.geoLocator.Lookup(ctx, addrs)
|
|
||||||
locations := make(map[string]*geo.Location, len(kept))
|
|
||||||
|
|
||||||
for i, ip := range kept {
|
|
||||||
locations[ip] = answers[i]
|
|
||||||
}
|
|
||||||
|
|
||||||
return locations
|
|
||||||
}
|
|
||||||
|
|
||||||
// isBogon says whether the address ip is reserved, as package bogon tells.
|
|
||||||
func isBogon(ip string) bool {
|
|
||||||
addr, err := netip.ParseAddr(ip)
|
|
||||||
if err != nil {
|
|
||||||
return false
|
|
||||||
}
|
|
||||||
|
|
||||||
_, reserved := bogon.Addr(addr)
|
|
||||||
|
|
||||||
return reserved
|
|
||||||
}
|
}
|
||||||
|
|
||||||
// setIPLookupCacheControl lets caches keep an IP lookup answer for the
|
// setIPLookupCacheControl lets caches keep an IP lookup answer for the
|
||||||
// shortest IP list interval, the soonest an IP list copy can change. The
|
// shortest blocklist interval, the soonest a blocklist copy can change. The
|
||||||
// answer about the caller's own address is private, so that a shared cache
|
// answer about the caller's own address is private, so that a shared cache
|
||||||
// does not give it to another caller. Without IP lists it sets nothing.
|
// does not give it to another caller. Without blocklists it sets nothing.
|
||||||
func (s *Server) setIPLookupCacheControl(w http.ResponseWriter, ownAddress bool) {
|
func (s *Server) setIPLookupCacheControl(w http.ResponseWriter, ownAddress bool) {
|
||||||
if s.ipLists == nil {
|
if s.blocklists == nil {
|
||||||
return
|
return
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -1108,7 +954,7 @@ func (s *Server) setIPLookupCacheControl(w http.ResponseWriter, ownAddress bool)
|
|||||||
visibility = cacheControlPrivate
|
visibility = cacheControlPrivate
|
||||||
}
|
}
|
||||||
|
|
||||||
maxAgeSeconds := int(s.ipLists.ShortestInterval().Seconds())
|
maxAgeSeconds := int(s.blocklists.ShortestInterval().Seconds())
|
||||||
w.Header().Set(cacheControlHeader,
|
w.Header().Set(cacheControlHeader,
|
||||||
visibility+cacheControlMaxAge+strconv.Itoa(maxAgeSeconds))
|
visibility+cacheControlMaxAge+strconv.Itoa(maxAgeSeconds))
|
||||||
}
|
}
|
||||||
@@ -1267,19 +1113,15 @@ func publicAddresses(addrs []string) []string {
|
|||||||
return public
|
return public
|
||||||
}
|
}
|
||||||
|
|
||||||
// lookupPTRNames returns the PTR records of ip without their trailing dots. It
|
// lookupPTRNames returns the PTR records of ip without their trailing dots, or
|
||||||
// returns an empty list, never nil, when there are none and when the lookup
|
// nil when the lookup fails or runs past ptrLookupTimeout.
|
||||||
// fails, so that an answer's ptr is [] rather than null.
|
|
||||||
func (s *Server) lookupPTRNames(ctx context.Context, ip string) []string {
|
func (s *Server) lookupPTRNames(ctx context.Context, ip string) []string {
|
||||||
ptrs, err := s.ptrResolver.LookupAddr(ctx, ip)
|
ctx, cancel := context.WithTimeout(ctx, ptrLookupTimeout)
|
||||||
|
defer cancel()
|
||||||
|
|
||||||
|
ptrs, err := s.resolver.LookupAddr(ctx, ip)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
s.logger.Warn("PTR lookup failed", "ip", ip, "error", err)
|
return nil
|
||||||
|
|
||||||
return []string{}
|
|
||||||
}
|
|
||||||
|
|
||||||
if ptrs == nil {
|
|
||||||
return []string{}
|
|
||||||
}
|
}
|
||||||
|
|
||||||
// Remove trailing dots from PTR records
|
// Remove trailing dots from PTR records
|
||||||
@@ -1665,23 +1507,26 @@ func (s *Server) handlePrefixDetail() http.HandlerFunc {
|
|||||||
|
|
||||||
routes := prefixRoutes(parsed, origins)
|
routes := prefixRoutes(parsed, origins)
|
||||||
|
|
||||||
changes, moreChanges, err := s.prefixOriginChanges(r.Context(), parsed)
|
|
||||||
if deadlinePassed(r.Context()) {
|
|
||||||
http.Error(w, requestTimeoutMessage, http.StatusGatewayTimeout)
|
|
||||||
|
|
||||||
return
|
|
||||||
}
|
|
||||||
|
|
||||||
if err != nil {
|
|
||||||
s.logger.Error("Failed to get the origin changes", "error", err)
|
|
||||||
http.Error(w, internalErrorMessage, http.StatusInternalServerError)
|
|
||||||
|
|
||||||
return
|
|
||||||
}
|
|
||||||
|
|
||||||
// Prepare template data
|
// Prepare template data
|
||||||
data := newPrefixDetailData(parsed, routes, originInfos, changes,
|
data := struct {
|
||||||
moreChanges, cspNonce(r.Context()))
|
Prefix string
|
||||||
|
MaskLength int
|
||||||
|
IPVersion int
|
||||||
|
Routes []EnhancedRoute
|
||||||
|
Origins []*ASNInfo
|
||||||
|
PeerCount int
|
||||||
|
OriginCount int
|
||||||
|
CSPNonce string
|
||||||
|
}{
|
||||||
|
Prefix: parsed.String(),
|
||||||
|
MaskLength: parsed.Bits(),
|
||||||
|
IPVersion: ipVersionOf(parsed),
|
||||||
|
Routes: routesWithASPathHandles(routes),
|
||||||
|
Origins: originInfos,
|
||||||
|
PeerCount: len(routes),
|
||||||
|
OriginCount: len(originInfos),
|
||||||
|
CSPNonce: cspNonce(r.Context()),
|
||||||
|
}
|
||||||
|
|
||||||
// Check if context is still valid before writing response
|
// Check if context is still valid before writing response
|
||||||
select {
|
select {
|
||||||
@@ -1734,61 +1579,6 @@ func (s *Server) originASNs(
|
|||||||
return infos, nil
|
return infos, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
// prefixOriginChanges returns prefix's origin changes for the page, newest
|
|
||||||
// first, at most maxOriginChangesOnPage, and whether there are more.
|
|
||||||
func (s *Server) prefixOriginChanges(
|
|
||||||
ctx context.Context, prefix netip.Prefix,
|
|
||||||
) ([]routetable.OriginChange, bool, error) {
|
|
||||||
// One more is read to know whether there are more.
|
|
||||||
changes, err := s.db.GetOriginChanges(ctx, prefix, maxOriginChangesOnPage+1)
|
|
||||||
if err != nil {
|
|
||||||
return nil, false, err
|
|
||||||
}
|
|
||||||
|
|
||||||
more := len(changes) > maxOriginChangesOnPage
|
|
||||||
if more {
|
|
||||||
changes = changes[:maxOriginChangesOnPage]
|
|
||||||
}
|
|
||||||
|
|
||||||
return changes, more, nil
|
|
||||||
}
|
|
||||||
|
|
||||||
// prefixDetailData is what the prefix detail page is rendered with.
|
|
||||||
type prefixDetailData struct {
|
|
||||||
Prefix string
|
|
||||||
MaskLength int
|
|
||||||
IPVersion int
|
|
||||||
Routes []EnhancedRoute
|
|
||||||
Origins []*ASNInfo
|
|
||||||
PeerCount int
|
|
||||||
OriginCount int
|
|
||||||
OriginChanges []routetable.OriginChange
|
|
||||||
MoreOriginChanges bool
|
|
||||||
// MaxOriginChanges is the most origin changes the page shows.
|
|
||||||
MaxOriginChanges int
|
|
||||||
CSPNonce string
|
|
||||||
}
|
|
||||||
|
|
||||||
// newPrefixDetailData builds the prefix detail page's data.
|
|
||||||
func newPrefixDetailData(
|
|
||||||
parsed netip.Prefix, routes []LiveRoute, originInfos []*ASNInfo,
|
|
||||||
changes []routetable.OriginChange, moreChanges bool, nonce string,
|
|
||||||
) prefixDetailData {
|
|
||||||
return prefixDetailData{
|
|
||||||
Prefix: parsed.String(),
|
|
||||||
MaskLength: parsed.Bits(),
|
|
||||||
IPVersion: ipVersionOf(parsed),
|
|
||||||
Routes: routesWithASPathHandles(routes),
|
|
||||||
Origins: originInfos,
|
|
||||||
PeerCount: len(routes),
|
|
||||||
OriginCount: len(originInfos),
|
|
||||||
OriginChanges: changes,
|
|
||||||
MoreOriginChanges: moreChanges,
|
|
||||||
MaxOriginChanges: maxOriginChangesOnPage,
|
|
||||||
CSPNonce: nonce,
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// prefixRoutes returns one route for each peer that carries prefix, by origin
|
// prefixRoutes returns one route for each peer that carries prefix, by origin
|
||||||
// AS in the order of origins. Each origin's kept routes come first, newest
|
// AS in the order of origins. Each origin's kept routes come first, newest
|
||||||
// first, with their AS path, next hop and time. Its other peers follow without
|
// first, with their AS path, next hop and time. Its other peers follow without
|
||||||
|
|||||||
File diff suppressed because it is too large.
Load diff
@@ -4,8 +4,6 @@ import (
|
|||||||
"encoding/json"
|
"encoding/json"
|
||||||
"net/http"
|
"net/http"
|
||||||
"net/http/httptest"
|
"net/http/httptest"
|
||||||
"net/netip"
|
|
||||||
"reflect"
|
|
||||||
"slices"
|
"slices"
|
||||||
"strconv"
|
"strconv"
|
||||||
"strings"
|
"strings"
|
||||||
@@ -77,15 +75,14 @@ func get(t *testing.T, s *Server, path string) *httptest.ResponseRecorder {
|
|||||||
|
|
||||||
// TestIPLookupGivesTheOriginMostPeersCarry looks up an address in a prefix
|
// TestIPLookupGivesTheOriginMostPeersCarry looks up an address in a prefix
|
||||||
// with two origin ASes, through each IP lookup route. The answer names the
|
// with two origin ASes, through each IP lookup route. The answer names the
|
||||||
// origin AS that more peers carry, counts every peer, lists both origin ASes,
|
// origin AS that more peers carry, counts every peer, and gives the time of
|
||||||
// and gives the time of the prefix's newest kept route. The prefix's
|
// the prefix's newest kept route. The prefix's first-seen time and the AS's
|
||||||
// first-seen time and the AS's WHOIS details come from the database. A WHOIS
|
// WHOIS details come from the database.
|
||||||
// detail the database does not hold is null.
|
|
||||||
func TestIPLookupGivesTheOriginMostPeersCarry(t *testing.T) {
|
func TestIPLookupGivesTheOriginMostPeersCarry(t *testing.T) {
|
||||||
t.Parallel()
|
t.Parallel()
|
||||||
|
|
||||||
s := newServerWithTable(t, newTwoOriginTable())
|
s := newServerWithTable(t, newTwoOriginTable())
|
||||||
useFakeResolver(s, &fakeResolver{})
|
s.resolver = &fakeResolver{}
|
||||||
|
|
||||||
firstSeen := testRouteTime(-1)
|
firstSeen := testRouteTime(-1)
|
||||||
|
|
||||||
@@ -108,26 +105,6 @@ func TestIPLookupGivesTheOriginMostPeersCarry(t *testing.T) {
|
|||||||
t.Fatalf("UpdateASNWHOIS: %v", err)
|
t.Fatalf("UpdateASNWHOIS: %v", err)
|
||||||
}
|
}
|
||||||
|
|
||||||
// The details the database does not hold, such as org_name, are null.
|
|
||||||
want := IPInfo{
|
|
||||||
Netblock: new(testPrefixV4),
|
|
||||||
MaskLength: new(netip.MustParsePrefix(testPrefixV4).Bits()),
|
|
||||||
IPVersion: new(ipVersionV4),
|
|
||||||
NumPeers: new(4),
|
|
||||||
ASN: new(mostPeersASN),
|
|
||||||
NumOrigins: new(2),
|
|
||||||
OriginASNs: []int{mostPeersASN, fewerPeersASN},
|
|
||||||
ASName: &whois.ASName,
|
|
||||||
CountryCode: &whois.CountryCode,
|
|
||||||
FirstSeen: &firstSeen,
|
|
||||||
LastSeen: new(testRouteTime(5)),
|
|
||||||
}
|
|
||||||
|
|
||||||
wantJSON, err := json.Marshal(want)
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("encoding %+v: %v", want, err)
|
|
||||||
}
|
|
||||||
|
|
||||||
// ownAddressPath looks up the request's own address, which httptest
|
// ownAddressPath looks up the request's own address, which httptest
|
||||||
// gives as addrInTestPrefixV4.
|
// gives as addrInTestPrefixV4.
|
||||||
for _, path := range []string{
|
for _, path := range []string{
|
||||||
@@ -138,15 +115,22 @@ func TestIPLookupGivesTheOriginMostPeersCarry(t *testing.T) {
|
|||||||
Data IPLookupResponse `json:"data"`
|
Data IPLookupResponse `json:"data"`
|
||||||
}
|
}
|
||||||
|
|
||||||
answer := get(t, s, path).Body.Bytes()
|
err := json.Unmarshal(get(t, s, path).Body.Bytes(), &body)
|
||||||
|
if err != nil || len(body.Data.Results) != 1 ||
|
||||||
err := json.Unmarshal(answer, &body)
|
body.Data.Results[0].IPInfo == nil {
|
||||||
if err != nil || len(body.Data.Results) != 1 {
|
t.Fatalf("%s: answer %+v (%v), want one result with a route",
|
||||||
t.Fatalf("%s: answer %s (%v), want one result", path, answer, err)
|
path, body.Data, err)
|
||||||
}
|
}
|
||||||
|
|
||||||
if !reflect.DeepEqual(body.Data.Results[0].IPInfo, want) {
|
got := body.Data.Results[0].IPInfo
|
||||||
t.Errorf("%s: answer %s, want the fields %s", path, answer, wantJSON)
|
if got.Netblock != testPrefixV4 || got.ASN != mostPeersASN ||
|
||||||
|
got.NumPeers != 4 || !got.LastSeen.Equal(testRouteTime(5)) ||
|
||||||
|
!got.FirstSeen.Equal(firstSeen) || got.ASName != whois.ASName ||
|
||||||
|
got.CountryCode != whois.CountryCode {
|
||||||
|
t.Errorf("%s: route %+v, want %s from AS%d, 4 peers, last seen %v, "+
|
||||||
|
"first seen %v, %s in %s", path, *got, testPrefixV4,
|
||||||
|
mostPeersASN, testRouteTime(5), firstSeen, whois.ASName,
|
||||||
|
whois.CountryCode)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -1,206 +0,0 @@
|
|||||||
package server
|
|
||||||
|
|
||||||
import (
|
|
||||||
"context"
|
|
||||||
"html/template"
|
|
||||||
"net/http"
|
|
||||||
"slices"
|
|
||||||
"time"
|
|
||||||
|
|
||||||
"git.eeqj.de/sneak/routewatch/internal/database"
|
|
||||||
"git.eeqj.de/sneak/routewatch/internal/routetable"
|
|
||||||
"git.eeqj.de/sneak/routewatch/internal/templates"
|
|
||||||
asinfo "git.eeqj.de/sneak/routewatch/pkg/asinfo"
|
|
||||||
)
|
|
||||||
|
|
||||||
const (
|
|
||||||
// homeCardsRefreshInterval is how often the server works out the home
|
|
||||||
// page's cards again. Some of their figures take a walk over the whole
|
|
||||||
// routing table, so no request works them out.
|
|
||||||
homeCardsRefreshInterval = time.Minute
|
|
||||||
|
|
||||||
// homeCardsDatabaseTimeout is how long the database read behind the
|
|
||||||
// cards of the newest and the oldest AS may take.
|
|
||||||
homeCardsDatabaseTimeout = 4 * time.Second
|
|
||||||
|
|
||||||
// maxListedEntries is how many entries the bogon and multiple-origin
|
|
||||||
// pages list at most.
|
|
||||||
maxListedEntries = 1000
|
|
||||||
)
|
|
||||||
|
|
||||||
// homeCards is what the home page's cards show. The bogon and multiple-origin
|
|
||||||
// pages list the entries their cards count. The server works it out every
|
|
||||||
// homeCardsRefreshInterval and serves this copy in between.
|
|
||||||
type homeCards struct {
|
|
||||||
routetable.Overview
|
|
||||||
|
|
||||||
// NewestAS and OldestAS are the ASes the asns table last saw most
|
|
||||||
// recently and longest ago. They are nil when the database did not
|
|
||||||
// answer or holds no AS.
|
|
||||||
NewestAS *database.ASN
|
|
||||||
OldestAS *database.ASN
|
|
||||||
|
|
||||||
// WellKnownNetworks are the networks of wellKnownASNs, newest update
|
|
||||||
// first.
|
|
||||||
WellKnownNetworks []wellKnownNetwork
|
|
||||||
}
|
|
||||||
|
|
||||||
// wellKnownNetwork is one network on the home page's card of well-known
|
|
||||||
// networks.
|
|
||||||
type wellKnownNetwork struct {
|
|
||||||
ASN int
|
|
||||||
// Name is its description in the AS list built into RouteWatch.
|
|
||||||
Name string
|
|
||||||
// Prefixes is how many prefixes it originates.
|
|
||||||
Prefixes int
|
|
||||||
// Newest is the time of its newest kept route, or nil when it keeps none.
|
|
||||||
Newest *time.Time
|
|
||||||
}
|
|
||||||
|
|
||||||
// listPageData is what the bogon and multiple-origin pages are rendered with.
|
|
||||||
type listPageData struct {
|
|
||||||
// CSPNonce is the nonce the page's style block carries.
|
|
||||||
CSPNonce string
|
|
||||||
// Cards is the copy of the home page's cards, or nil before the server
|
|
||||||
// has first worked them out.
|
|
||||||
Cards *homeCards
|
|
||||||
}
|
|
||||||
|
|
||||||
// The AS numbers of the networks the home page lists as well known.
|
|
||||||
const (
|
|
||||||
appleASN = 714
|
|
||||||
googleASN = 15169
|
|
||||||
amazonASN = 16509
|
|
||||||
microsoftASN = 8075
|
|
||||||
metaASN = 32934
|
|
||||||
cloudflareASN = 13335
|
|
||||||
akamaiASN = 20940
|
|
||||||
netflixASN = 2906
|
|
||||||
dodNetworkInfoCtrASN = 721
|
|
||||||
cogentASN = 174
|
|
||||||
hurricaneElectricASN = 6939
|
|
||||||
level3ASN = 3356
|
|
||||||
arelionASN = 1299
|
|
||||||
nttASN = 2914
|
|
||||||
mitASN = 3
|
|
||||||
)
|
|
||||||
|
|
||||||
// wellKnownASNs returns the AS numbers of the networks the home page lists as
|
|
||||||
// well known: large companies, large transit networks, the US Department of
|
|
||||||
// Defense's Network Information Center and a university.
|
|
||||||
func wellKnownASNs() []int {
|
|
||||||
return []int{
|
|
||||||
appleASN, googleASN, amazonASN, microsoftASN, metaASN, cloudflareASN,
|
|
||||||
akamaiASN, netflixASN, dodNetworkInfoCtrASN, cogentASN,
|
|
||||||
hurricaneElectricASN, level3ASN, arelionASN, nttASN, mitASN,
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// keepHomeCardsFresh works out the home page's cards now, and again every
|
|
||||||
// interval until ctx ends.
|
|
||||||
func (s *Server) keepHomeCardsFresh(ctx context.Context, interval time.Duration) {
|
|
||||||
ticker := time.NewTicker(interval)
|
|
||||||
defer ticker.Stop()
|
|
||||||
|
|
||||||
for {
|
|
||||||
s.refreshHomeCards(ctx)
|
|
||||||
|
|
||||||
select {
|
|
||||||
case <-ctx.Done():
|
|
||||||
return
|
|
||||||
case <-ticker.C:
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// refreshHomeCards works out the home page's cards and keeps them for the
|
|
||||||
// pages to serve. When the database does not answer, the failure is logged
|
|
||||||
// and the cards of the newest and the oldest AS stay empty.
|
|
||||||
func (s *Server) refreshHomeCards(ctx context.Context) {
|
|
||||||
cards := &homeCards{
|
|
||||||
Overview: s.table.Overview(maxListedEntries),
|
|
||||||
WellKnownNetworks: s.wellKnownNetworks(),
|
|
||||||
}
|
|
||||||
|
|
||||||
ctx, cancel := context.WithTimeout(ctx, homeCardsDatabaseTimeout)
|
|
||||||
defer cancel()
|
|
||||||
|
|
||||||
newest, oldest, err := s.db.GetLastSeenASes(ctx)
|
|
||||||
if err != nil {
|
|
||||||
s.logger.Warn("Failed to read the newest and the oldest AS", "error", err)
|
|
||||||
} else {
|
|
||||||
cards.NewestAS, cards.OldestAS = newest, oldest
|
|
||||||
}
|
|
||||||
|
|
||||||
s.cards.Store(cards)
|
|
||||||
}
|
|
||||||
|
|
||||||
// wellKnownNetworks returns the networks of wellKnownASNs, each with how many
|
|
||||||
// prefixes it originates and the time of its newest kept route. They come
|
|
||||||
// newest first, and those that keep no route last.
|
|
||||||
func (s *Server) wellKnownNetworks() []wellKnownNetwork {
|
|
||||||
asns := wellKnownASNs()
|
|
||||||
networks := make([]wellKnownNetwork, 0, len(asns))
|
|
||||||
|
|
||||||
for _, asn := range asns {
|
|
||||||
originated := s.table.OriginatedPrefixes(asn)
|
|
||||||
network := wellKnownNetwork{ASN: asn, Prefixes: len(originated)}
|
|
||||||
|
|
||||||
if info, ok := asinfo.Get(asn); ok {
|
|
||||||
network.Name = info.Description
|
|
||||||
}
|
|
||||||
|
|
||||||
for _, p := range originated {
|
|
||||||
if p.Newest != nil &&
|
|
||||||
(network.Newest == nil || p.Newest.After(*network.Newest)) {
|
|
||||||
network.Newest = p.Newest
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
networks = append(networks, network)
|
|
||||||
}
|
|
||||||
|
|
||||||
slices.SortStableFunc(networks, func(a, b wellKnownNetwork) int {
|
|
||||||
switch {
|
|
||||||
case a.Newest == nil && b.Newest == nil:
|
|
||||||
return 0
|
|
||||||
case a.Newest == nil:
|
|
||||||
return 1
|
|
||||||
case b.Newest == nil:
|
|
||||||
return -1
|
|
||||||
}
|
|
||||||
|
|
||||||
return b.Newest.Compare(*a.Newest)
|
|
||||||
})
|
|
||||||
|
|
||||||
return networks
|
|
||||||
}
|
|
||||||
|
|
||||||
// handleBogons returns a handler that serves the page listing the bogon
|
|
||||||
// announcements.
|
|
||||||
func (s *Server) handleBogons() http.HandlerFunc {
|
|
||||||
return s.listPageHandler(templates.BogonsTemplate())
|
|
||||||
}
|
|
||||||
|
|
||||||
// handleMultipleOrigins returns a handler that serves the page listing the
|
|
||||||
// prefixes announced by more than one origin AS.
|
|
||||||
func (s *Server) handleMultipleOrigins() http.HandlerFunc {
|
|
||||||
return s.listPageHandler(templates.MultipleOriginsTemplate())
|
|
||||||
}
|
|
||||||
|
|
||||||
// listPageHandler returns a handler that renders tmpl, a page that lists
|
|
||||||
// entries the home page's cards count, from the copy of the cards.
|
|
||||||
func (s *Server) listPageHandler(tmpl *template.Template) http.HandlerFunc {
|
|
||||||
return func(w http.ResponseWriter, r *http.Request) {
|
|
||||||
w.Header().Set("Content-Type", "text/html; charset=utf-8")
|
|
||||||
|
|
||||||
err := tmpl.Execute(w, listPageData{
|
|
||||||
CSPNonce: cspNonce(r.Context()),
|
|
||||||
Cards: s.cards.Load(),
|
|
||||||
})
|
|
||||||
if err != nil {
|
|
||||||
s.logger.Error("Failed to render a list page", "error", err)
|
|
||||||
http.Error(w, "Internal Server Error", http.StatusInternalServerError)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
@@ -1,423 +0,0 @@
|
|||||||
package server
|
|
||||||
|
|
||||||
import (
|
|
||||||
"context"
|
|
||||||
"net/netip"
|
|
||||||
"strconv"
|
|
||||||
"strings"
|
|
||||||
"testing"
|
|
||||||
"testing/synctest"
|
|
||||||
"time"
|
|
||||||
|
|
||||||
"git.eeqj.de/sneak/routewatch/internal/database"
|
|
||||||
"git.eeqj.de/sneak/routewatch/internal/routetable"
|
|
||||||
)
|
|
||||||
|
|
||||||
// The ASes of the routing table that newCardsTable fills. Each card test
|
|
||||||
// finds one of them on its card.
|
|
||||||
const (
|
|
||||||
twoPrefixASN = 13335 // originates two prefixes, the most
|
|
||||||
newestRouteASN = 15169 // the newest route
|
|
||||||
oldestRouteASN = 3 // the oldest route
|
|
||||||
secondOrigin = 16509 // a second origin AS of twoPrefixASN's prefix
|
|
||||||
privateUseASN = 64512 // reserved, so its route is a bogon announcement
|
|
||||||
)
|
|
||||||
|
|
||||||
// Paths of the pages that list what two of the home page's cards count.
|
|
||||||
const (
|
|
||||||
bogonsPath = "/bogons"
|
|
||||||
multipleOriginsPath = "/multiple-origins"
|
|
||||||
)
|
|
||||||
|
|
||||||
// cardsNotReadyText is what the pages say before the cards are first worked
|
|
||||||
// out.
|
|
||||||
const cardsNotReadyText = "being worked out"
|
|
||||||
|
|
||||||
// newCardsTable returns a routing table in which:
|
|
||||||
// - twoPrefixASN originates 1.0.0.0/24 and 1.0.4.0/22;
|
|
||||||
// - secondOrigin originates 1.0.0.0/24 too, which so has two origin ASes;
|
|
||||||
// - newestRouteASN's 8.8.8.0/24 has the newest route;
|
|
||||||
// - oldestRouteASN's 18.0.0.0/8 has the oldest;
|
|
||||||
// - privateUseASN originates 10.0.0.0/8, carried by two peers: a bogon
|
|
||||||
// announcement for both its prefix and its AS.
|
|
||||||
//
|
|
||||||
// The last four ASes originate one prefix each.
|
|
||||||
func newCardsTable() *routetable.Table {
|
|
||||||
table := emptyTable()
|
|
||||||
announce(table, "1.0.0.0/24", twoPrefixASN, testPeer(1), testRouteTime(10))
|
|
||||||
announce(table, "1.0.4.0/22", twoPrefixASN, testPeer(1), testRouteTime(5))
|
|
||||||
announce(table, "1.0.0.0/24", secondOrigin, testPeer(3), testRouteTime(12))
|
|
||||||
announce(table, "8.8.8.0/24", newestRouteASN, testPeer(2), testRouteTime(20))
|
|
||||||
announce(table, "18.0.0.0/8", oldestRouteASN, testPeer(2), testRouteTime(1))
|
|
||||||
announce(table, "10.0.0.0/8", privateUseASN, testPeer(1), testRouteTime(15))
|
|
||||||
announce(table, "10.0.0.0/8", privateUseASN, testPeer(2), testRouteTime(15))
|
|
||||||
|
|
||||||
return table
|
|
||||||
}
|
|
||||||
|
|
||||||
// homeCard returns the HTML of the home page's card with heading, up to the
|
|
||||||
// end of the card.
|
|
||||||
func homeCard(t *testing.T, page, heading string) string {
|
|
||||||
t.Helper()
|
|
||||||
|
|
||||||
_, card, found := strings.Cut(page, "<h2>"+heading+"</h2>")
|
|
||||||
if !found {
|
|
||||||
t.Fatalf("the home page has no card %q", heading)
|
|
||||||
}
|
|
||||||
|
|
||||||
card, _, _ = strings.Cut(card, "</div>")
|
|
||||||
|
|
||||||
return card
|
|
||||||
}
|
|
||||||
|
|
||||||
// asLink returns the HTML attribute of a link to the AS page of asn.
|
|
||||||
func asLink(asn int) string {
|
|
||||||
return `href="/as/` + strconv.Itoa(asn) + `"`
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestHomePageCardsShowTheTableFigures fills a routing table and the asns
|
|
||||||
// table, works out the cards, and checks that each card shows its figure. The
|
|
||||||
// cards are worked out in a synctest bubble, whose clock is fake, so a stall on
|
|
||||||
// the host does not use up the 4-second limit on their database read.
|
|
||||||
func TestHomePageCardsShowTheTableFigures(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
const (
|
|
||||||
newestSeenASN = 6939
|
|
||||||
oldestSeenASN = 174
|
|
||||||
)
|
|
||||||
|
|
||||||
s := newServerWithTable(t, newCardsTable())
|
|
||||||
|
|
||||||
err := s.db.GetOrCreateASNBatch(map[int]database.SeenTimes{
|
|
||||||
newestSeenASN: {First: testRouteTime(0), Last: testRouteTime(30)},
|
|
||||||
oldestSeenASN: {First: testRouteTime(-30), Last: testRouteTime(-30)},
|
|
||||||
})
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("GetOrCreateASNBatch: %v", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
synctest.Test(t, func(t *testing.T) {
|
|
||||||
s.refreshHomeCards(t.Context())
|
|
||||||
})
|
|
||||||
|
|
||||||
page := get(t, s, "/").Body.String()
|
|
||||||
|
|
||||||
tests := []struct {
|
|
||||||
heading string
|
|
||||||
want []string
|
|
||||||
}{
|
|
||||||
{"Newest Updated Route", []string{
|
|
||||||
"8.8.8.0/24", asLink(newestRouteASN),
|
|
||||||
testRouteTime(20).Format(routeTimeFormat),
|
|
||||||
}},
|
|
||||||
{"Oldest Updated Route", []string{
|
|
||||||
"18.0.0.0/8", asLink(oldestRouteASN),
|
|
||||||
testRouteTime(1).Format(routeTimeFormat),
|
|
||||||
}},
|
|
||||||
{"Newest Updated AS", []string{
|
|
||||||
asLink(newestSeenASN), testRouteTime(30).Format(routeTimeFormat),
|
|
||||||
}},
|
|
||||||
{"Oldest Updated AS", []string{
|
|
||||||
asLink(oldestSeenASN), testRouteTime(-30).Format(routeTimeFormat),
|
|
||||||
}},
|
|
||||||
{"AS with the Most Prefixes", []string{
|
|
||||||
asLink(twoPrefixASN), "Prefixes it originates: 2",
|
|
||||||
}},
|
|
||||||
{"ASes with the Fewest Prefixes", []string{
|
|
||||||
"ASes that originate a single prefix: 4", asLink(newestRouteASN),
|
|
||||||
"8.8.8.0/24",
|
|
||||||
}},
|
|
||||||
{"Bogon Announcements", []string{`href="/bogons">1<`}},
|
|
||||||
{"Prefixes with Several Origins", []string{`href="/multiple-origins">1<`}},
|
|
||||||
}
|
|
||||||
|
|
||||||
for _, tc := range tests {
|
|
||||||
card := homeCard(t, page, tc.heading)
|
|
||||||
for _, want := range tc.want {
|
|
||||||
if !strings.Contains(card, want) {
|
|
||||||
t.Errorf("card %q does not hold %s: %s", tc.heading, want, card)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// The well-known networks come newest update first, and those without
|
|
||||||
// a route after them.
|
|
||||||
networks := homeCard(t, page, "Well-Known Networks")
|
|
||||||
order := []int{newestRouteASN, secondOrigin, twoPrefixASN, oldestRouteASN, 714}
|
|
||||||
last := -1
|
|
||||||
|
|
||||||
for _, asn := range order {
|
|
||||||
i := strings.Index(networks, asLink(asn))
|
|
||||||
if i <= last {
|
|
||||||
t.Fatalf("well-known networks: AS%d is not after the ASes before "+
|
|
||||||
"it in %v: %s", asn, order, networks)
|
|
||||||
}
|
|
||||||
|
|
||||||
last = i
|
|
||||||
}
|
|
||||||
|
|
||||||
if !strings.Contains(networks, "<td>2</td>") {
|
|
||||||
t.Errorf("well-known networks: no network with 2 prefixes: %s", networks)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestListPagesListTheirEntries checks the rows of the bogon and multiple-origin
|
|
||||||
// pages for the table of newCardsTable.
|
|
||||||
func TestListPagesListTheirEntries(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
s := newServerWithTable(t, newCardsTable())
|
|
||||||
s.refreshHomeCards(t.Context())
|
|
||||||
|
|
||||||
pages := map[string][]string{
|
|
||||||
bogonsPath: {
|
|
||||||
"Bogon announcements: 1.", "10.0.0.0/8", asLink(privateUseASN),
|
|
||||||
"<td>2</td>",
|
|
||||||
"Private-Use (10.0.0.0/8); Private-Use (AS64512-AS65534)",
|
|
||||||
},
|
|
||||||
multipleOriginsPath: {
|
|
||||||
"Prefixes announced by more than one origin AS: 1.", "1.0.0.0/24",
|
|
||||||
asLink(twoPrefixASN), asLink(secondOrigin), ">1 peer<",
|
|
||||||
},
|
|
||||||
}
|
|
||||||
|
|
||||||
for path, wants := range pages {
|
|
||||||
page := get(t, s, path).Body.String()
|
|
||||||
for _, want := range wants {
|
|
||||||
if !strings.Contains(page, want) {
|
|
||||||
t.Errorf("%s does not hold %s", path, want)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestListPagesListAtMostMaxListedEntries fills a table with more prefixes than
|
|
||||||
// the list pages list, each announced by two origin ASes in reserved address
|
|
||||||
// space. Each page lists maxListedEntries of them and gives how many there are
|
|
||||||
// in all.
|
|
||||||
func TestListPagesListAtMostMaxListedEntries(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
const prefixCount = maxListedEntries + 1
|
|
||||||
|
|
||||||
table := emptyTable()
|
|
||||||
addr := netip.MustParseAddr("10.0.0.0")
|
|
||||||
|
|
||||||
for range prefixCount {
|
|
||||||
prefix := netip.PrefixFrom(addr, addr.BitLen()).String()
|
|
||||||
announce(table, prefix, twoPrefixASN, testPeer(1), testRouteTime(0))
|
|
||||||
announce(table, prefix, secondOrigin, testPeer(2), testRouteTime(0))
|
|
||||||
|
|
||||||
addr = addr.Next()
|
|
||||||
}
|
|
||||||
|
|
||||||
s := newServerWithTable(t, table)
|
|
||||||
s.refreshHomeCards(t.Context())
|
|
||||||
|
|
||||||
listed := "The first " + strconv.Itoa(maxListedEntries) + " are listed"
|
|
||||||
tests := []struct {
|
|
||||||
path string
|
|
||||||
total string
|
|
||||||
}{
|
|
||||||
{bogonsPath, "Bogon announcements: " + strconv.Itoa(2*prefixCount) + "."},
|
|
||||||
{multipleOriginsPath, "Prefixes announced by more than one origin AS: " +
|
|
||||||
strconv.Itoa(prefixCount) + "."},
|
|
||||||
}
|
|
||||||
|
|
||||||
for _, tc := range tests {
|
|
||||||
page := get(t, s, tc.path).Body.String()
|
|
||||||
|
|
||||||
rows := strings.Count(page, `class="prefix-link"`)
|
|
||||||
if rows != maxListedEntries {
|
|
||||||
t.Errorf("%s lists %d entries, want %d", tc.path, rows,
|
|
||||||
maxListedEntries)
|
|
||||||
}
|
|
||||||
|
|
||||||
if !strings.Contains(page, tc.total) || !strings.Contains(page, listed) {
|
|
||||||
t.Errorf("%s does not say %q and %q", tc.path, tc.total, listed)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestPagesServeTheCopyOfTheCards checks that the pages show the cards as they
|
|
||||||
// were last worked out, not as the routing table is now, and that they say so
|
|
||||||
// before the cards are first worked out.
|
|
||||||
func TestPagesServeTheCopyOfTheCards(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
table := emptyTable()
|
|
||||||
announce(table, "1.0.0.0/24", twoPrefixASN, testPeer(1), testRouteTime(0))
|
|
||||||
|
|
||||||
s := newServerWithTable(t, table)
|
|
||||||
|
|
||||||
for _, path := range []string{"/", bogonsPath, multipleOriginsPath} {
|
|
||||||
if !strings.Contains(get(t, s, path).Body.String(), cardsNotReadyText) {
|
|
||||||
t.Errorf("%s does not say the cards are %s", path, cardsNotReadyText)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
s.refreshHomeCards(t.Context())
|
|
||||||
announce(table, "8.8.8.0/24", newestRouteASN, testPeer(2), testRouteTime(1))
|
|
||||||
|
|
||||||
card := homeCard(t, get(t, s, "/").Body.String(), "Newest Updated Route")
|
|
||||||
if !strings.Contains(card, "1.0.0.0/24") {
|
|
||||||
t.Errorf("before the cards are worked out again, the newest route "+
|
|
||||||
"is not the one they were worked out with: %s", card)
|
|
||||||
}
|
|
||||||
|
|
||||||
s.refreshHomeCards(t.Context())
|
|
||||||
|
|
||||||
card = homeCard(t, get(t, s, "/").Body.String(), "Newest Updated Route")
|
|
||||||
if !strings.Contains(card, "8.8.8.0/24") {
|
|
||||||
t.Errorf("after the cards are worked out again, the newest route is "+
|
|
||||||
"not the table's: %s", card)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestKeepHomeCardsFreshWorksThemOutEveryInterval runs keepHomeCardsFresh with
|
|
||||||
// a short interval. It works the cards out at once, again after the table
|
|
||||||
// changes, and stops when its context ends.
|
|
||||||
func TestKeepHomeCardsFreshWorksThemOutEveryInterval(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
const (
|
|
||||||
interval = 10 * time.Millisecond
|
|
||||||
maxWait = 15 * time.Second
|
|
||||||
)
|
|
||||||
|
|
||||||
table := emptyTable()
|
|
||||||
announce(table, "1.0.0.0/24", twoPrefixASN, testPeer(1), testRouteTime(0))
|
|
||||||
|
|
||||||
s := newServerWithTable(t, table)
|
|
||||||
|
|
||||||
ctx, cancel := context.WithCancel(t.Context())
|
|
||||||
stopped := make(chan struct{})
|
|
||||||
|
|
||||||
go func() {
|
|
||||||
s.keepHomeCardsFresh(ctx, interval)
|
|
||||||
close(stopped)
|
|
||||||
}()
|
|
||||||
|
|
||||||
newestRouteASNIs := func(asn int) bool {
|
|
||||||
cards := s.cards.Load()
|
|
||||||
|
|
||||||
return cards != nil && cards.NewestRoute != nil &&
|
|
||||||
cards.NewestRoute.OriginASN == asn
|
|
||||||
}
|
|
||||||
|
|
||||||
waitUntil(t, maxWait, func() bool { return newestRouteASNIs(twoPrefixASN) })
|
|
||||||
|
|
||||||
announce(table, "8.8.8.0/24", newestRouteASN, testPeer(2), testRouteTime(1))
|
|
||||||
waitUntil(t, maxWait, func() bool { return newestRouteASNIs(newestRouteASN) })
|
|
||||||
|
|
||||||
cancel()
|
|
||||||
|
|
||||||
select {
|
|
||||||
case <-stopped:
|
|
||||||
case <-time.After(maxWait):
|
|
||||||
t.Fatal("keepHomeCardsFresh did not stop when its context ended")
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestStartedServerWorksOutTheCards starts a server and waits until its home
|
|
||||||
// page shows the cards.
|
|
||||||
func TestStartedServerWorksOutTheCards(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
const maxWait = 15 * time.Second
|
|
||||||
|
|
||||||
s := newServerWithTable(t, newCardsTable())
|
|
||||||
|
|
||||||
err := s.Start(t.Context())
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("Start: %v", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
defer func() { _ = s.Stop(context.Background()) }()
|
|
||||||
|
|
||||||
waitUntil(t, maxWait, func() bool {
|
|
||||||
return !strings.Contains(get(t, s, "/").Body.String(), cardsNotReadyText)
|
|
||||||
})
|
|
||||||
|
|
||||||
card := homeCard(t, get(t, s, "/").Body.String(), "Newest Updated Route")
|
|
||||||
if !strings.Contains(card, "8.8.8.0/24") {
|
|
||||||
t.Errorf("the newest route is not the table's: %s", card)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestStartedServerWorksOutTheCardsUntilStop starts a server with a context
|
|
||||||
// that ends as soon as Start returns. It runs in a synctest bubble, whose clock
|
|
||||||
// is fake: time.Sleep there moves the clock on and returns at once. The server
|
|
||||||
// must work the cards out again a minute later, and no more after Stop. Each
|
|
||||||
// time it works them out, it keeps a new copy. Before the fix the end of
|
|
||||||
// Start's context stopped the goroutine that works them out.
|
|
||||||
func TestStartedServerWorksOutTheCardsUntilStop(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
// The server is made outside the bubble. The streamer's metrics start a
|
|
||||||
// goroutine that never returns, and a bubble waits for each of its
|
|
||||||
// goroutines to return.
|
|
||||||
s := newServerWithTable(t, newCardsTable())
|
|
||||||
|
|
||||||
synctest.Test(t, func(t *testing.T) {
|
|
||||||
ctx, cancel := context.WithCancel(t.Context())
|
|
||||||
|
|
||||||
err := s.Start(ctx)
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("Start: %v", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
cancel()
|
|
||||||
|
|
||||||
// The bubble's clock moves only while each of its goroutines waits
|
|
||||||
// on the bubble, and the goroutine that serves HTTP waits on the
|
|
||||||
// network. The test sends no request, so it closes the HTTP server.
|
|
||||||
_ = s.srv.Close()
|
|
||||||
|
|
||||||
synctest.Wait()
|
|
||||||
|
|
||||||
started := s.cards.Load()
|
|
||||||
if started == nil {
|
|
||||||
t.Fatal("the server did not work out the cards when it started")
|
|
||||||
}
|
|
||||||
|
|
||||||
time.Sleep(homeCardsRefreshInterval)
|
|
||||||
synctest.Wait()
|
|
||||||
|
|
||||||
if s.cards.Load() == started {
|
|
||||||
t.Error("the server did not work out the cards again a minute " +
|
|
||||||
"after its start context ended")
|
|
||||||
}
|
|
||||||
|
|
||||||
err = s.Stop(t.Context())
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("Stop: %v", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
stopped := s.cards.Load()
|
|
||||||
|
|
||||||
time.Sleep(homeCardsRefreshInterval)
|
|
||||||
synctest.Wait()
|
|
||||||
|
|
||||||
if s.cards.Load() != stopped {
|
|
||||||
t.Error("the server worked out the cards again after Stop")
|
|
||||||
}
|
|
||||||
})
|
|
||||||
}
|
|
||||||
|
|
||||||
// waitUntil fails the test unless done reports true within maxWait.
|
|
||||||
func waitUntil(t *testing.T, maxWait time.Duration, done func() bool) {
|
|
||||||
t.Helper()
|
|
||||||
|
|
||||||
const pollInterval = 5 * time.Millisecond
|
|
||||||
|
|
||||||
deadline := time.Now().Add(maxWait)
|
|
||||||
for !done() {
|
|
||||||
if time.Now().After(deadline) {
|
|
||||||
t.Fatalf("the cards did not change within %s", maxWait)
|
|
||||||
}
|
|
||||||
|
|
||||||
time.Sleep(pollInterval)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
@@ -16,7 +16,6 @@ import (
|
|||||||
"time"
|
"time"
|
||||||
|
|
||||||
"github.com/go-chi/chi/v5"
|
"github.com/go-chi/chi/v5"
|
||||||
"github.com/go-chi/chi/v5/middleware"
|
|
||||||
)
|
)
|
||||||
|
|
||||||
// responseWriter wraps http.ResponseWriter to capture the response
|
// responseWriter wraps http.ResponseWriter to capture the response
|
||||||
@@ -54,15 +53,6 @@ func (rw *responseWriter) Header() http.Header {
|
|||||||
return rw.ResponseWriter.Header()
|
return rw.ResponseWriter.Header()
|
||||||
}
|
}
|
||||||
|
|
||||||
// The @meta object JSONResponseMiddleware adds to a JSON answer: its key, and
|
|
||||||
// the keys of its fields.
|
|
||||||
const (
|
|
||||||
jsonKeyMeta = "@meta"
|
|
||||||
metaKeyTimeZone = "time_zone"
|
|
||||||
metaKeyAPIVersion = "api_version"
|
|
||||||
metaKeyExecutionTimeMs = "execution_time_ms"
|
|
||||||
)
|
|
||||||
|
|
||||||
// JSONResponseMiddleware is an HTTP middleware that wraps all JSON responses
|
// JSONResponseMiddleware is an HTTP middleware that wraps all JSON responses
|
||||||
// with a @meta field containing execution metadata. The metadata includes the
|
// with a @meta field containing execution metadata. The metadata includes the
|
||||||
// time zone (always UTC), API version, and request execution time in milliseconds.
|
// time zone (always UTC), API version, and request execution time in milliseconds.
|
||||||
@@ -116,10 +106,10 @@ func JSONResponseMiddleware(next http.Handler) http.Handler {
|
|||||||
}
|
}
|
||||||
|
|
||||||
// Add @meta field
|
// Add @meta field
|
||||||
originalResponse[jsonKeyMeta] = map[string]any{
|
originalResponse["@meta"] = map[string]any{
|
||||||
metaKeyTimeZone: "UTC",
|
"time_zone": "UTC",
|
||||||
metaKeyAPIVersion: 1,
|
"api_version": 1,
|
||||||
metaKeyExecutionTimeMs: execTime.Milliseconds(),
|
"execution_time": fmt.Sprintf("%d ms", execTime.Milliseconds()),
|
||||||
}
|
}
|
||||||
|
|
||||||
// Write the enhanced response
|
// Write the enhanced response
|
||||||
@@ -155,8 +145,7 @@ func deadlinePassed(ctx context.Context) bool {
|
|||||||
// has passed, it answers 504 with the error message "Request timeout". A 4xx
|
// has passed, it answers 504 with the error message "Request timeout". A 4xx
|
||||||
// keeps its status, with the status's standard text as the message. For other
|
// keeps its status, with the status's standard text as the message. For other
|
||||||
// errors, it returns "Internal server error", with status 500 if the original
|
// errors, it returns "Internal server error", with status 500 if the original
|
||||||
// status was 200. A panic in the handler answers 500 with "Internal server
|
// status was 200.
|
||||||
// error" too.
|
|
||||||
//
|
//
|
||||||
// Non-JSON responses are passed through unchanged.
|
// Non-JSON responses are passed through unchanged.
|
||||||
func JSONValidationMiddleware(next http.Handler) http.Handler {
|
func JSONValidationMiddleware(next http.Handler) http.Handler {
|
||||||
@@ -168,9 +157,8 @@ func JSONValidationMiddleware(next http.Handler) http.Handler {
|
|||||||
statusCode: http.StatusOK,
|
statusCode: http.StatusOK,
|
||||||
}
|
}
|
||||||
|
|
||||||
// Serve the request. Recoverer logs a panic in the handler and turns it
|
// Serve the request
|
||||||
// into a 500 without a body, which gets the JSON error body below.
|
next.ServeHTTP(rw, r)
|
||||||
middleware.Recoverer(next).ServeHTTP(rw, r)
|
|
||||||
|
|
||||||
// Check if it's meant to be a JSON response
|
// Check if it's meant to be a JSON response
|
||||||
contentType := rw.Header().Get("Content-Type")
|
contentType := rw.Header().Get("Content-Type")
|
||||||
|
|||||||
@@ -1,41 +1,25 @@
|
|||||||
package server
|
package server
|
||||||
|
|
||||||
import (
|
import (
|
||||||
"bytes"
|
|
||||||
"context"
|
|
||||||
"encoding/json"
|
"encoding/json"
|
||||||
"io"
|
|
||||||
"log/slog"
|
|
||||||
"net"
|
|
||||||
"net/http"
|
"net/http"
|
||||||
"net/http/httptest"
|
"net/http/httptest"
|
||||||
"net/netip"
|
"net/netip"
|
||||||
"os"
|
|
||||||
"regexp"
|
|
||||||
"strings"
|
"strings"
|
||||||
"testing"
|
"testing"
|
||||||
"testing/synctest"
|
|
||||||
"time"
|
"time"
|
||||||
|
|
||||||
"git.eeqj.de/sneak/routewatch/internal/config"
|
|
||||||
"git.eeqj.de/sneak/routewatch/internal/database"
|
|
||||||
"git.eeqj.de/sneak/routewatch/internal/logger"
|
|
||||||
"git.eeqj.de/sneak/routewatch/internal/metrics"
|
|
||||||
"git.eeqj.de/sneak/routewatch/internal/streamer"
|
|
||||||
)
|
)
|
||||||
|
|
||||||
// The addresses the forwarding tests use: the client a proxy names in
|
// The addresses the forwarding tests use: the client a proxy names in
|
||||||
// X-Forwarded-For, the range of trusted proxies, a peer inside it, and a peer
|
// X-Forwarded-For, the range of trusted proxies, a peer inside it, and a peer
|
||||||
// outside it, which lies in testPrefixV4 so that an IP lookup of it finds a
|
// outside it, which lies in testPrefixV4 so that an IP lookup of it finds a
|
||||||
// route. reverseProxyPeer is a reverse proxy at a private address, as Docker
|
// route.
|
||||||
// gives one.
|
|
||||||
const (
|
const (
|
||||||
forwardedClient = "203.0.113.9"
|
forwardedClient = "203.0.113.9"
|
||||||
trustedProxyRange = "10.0.0.0/8"
|
trustedProxyRange = "10.0.0.0/8"
|
||||||
trustedPeer = "10.0.0.2:1234"
|
trustedPeer = "10.0.0.2:1234"
|
||||||
untrustedPeerAddr = "192.0.2.1"
|
untrustedPeerAddr = "192.0.2.1"
|
||||||
untrustedPeer = untrustedPeerAddr + ":1234"
|
untrustedPeer = untrustedPeerAddr + ":1234"
|
||||||
reverseProxyPeer = "172.19.0.14:1234"
|
|
||||||
)
|
)
|
||||||
|
|
||||||
// TestResponsesCarrySecurityHeaders checks that a page, a JSON route and a
|
// TestResponsesCarrySecurityHeaders checks that a page, a JSON route and a
|
||||||
@@ -103,225 +87,6 @@ func TestJSONValidationKeeps4xxWithoutBody(t *testing.T) {
|
|||||||
checkJSONErrorMessage(t, name, rec, http.StatusText(http.StatusBadRequest))
|
checkJSONErrorMessage(t, name, rec, http.StatusText(http.StatusBadRequest))
|
||||||
}
|
}
|
||||||
|
|
||||||
// TestJSONAnswersPutNoUnitInAString sends a request to each JSON route and
|
|
||||||
// checks each answer. Its @meta.execution_time_ms is a number, and none of its
|
|
||||||
// strings is a number with a unit, such as "12 ms", "1.1 GB" or "1h18m34s"
|
|
||||||
// (https://git.eeqj.de/sneak/routewatch/issues/199). The health check answers
|
|
||||||
// 503, since the stream is not connected, and the last path matches no route.
|
|
||||||
func TestJSONAnswersPutNoUnitInAString(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
numberWithUnit := regexp.MustCompile(
|
|
||||||
`^([0-9]+(\.[0-9]+)? ?(ns|µs|ms|s|m|h|B|kB|MB|GB|TB))+$`)
|
|
||||||
|
|
||||||
s := newServerWithTestRoutes(t)
|
|
||||||
useFakeResolver(s, &fakeResolver{})
|
|
||||||
s.SetASNFetcher(fixedIntervalASNFetcher{interval: time.Second})
|
|
||||||
|
|
||||||
paths := []string{
|
|
||||||
statusJSONPath,
|
|
||||||
statsPath,
|
|
||||||
healthCheckPath,
|
|
||||||
"/api/v1/ip/" + untrustedPeerAddr,
|
|
||||||
"/ip/" + untrustedPeerAddr,
|
|
||||||
asJSONPath,
|
|
||||||
"/api/v1/prefix/" + testPrefixV4,
|
|
||||||
"/api/v1/search/as?q=google",
|
|
||||||
"/api/v1/no-such-route",
|
|
||||||
}
|
|
||||||
|
|
||||||
for _, path := range paths {
|
|
||||||
rec := httptest.NewRecorder()
|
|
||||||
s.router.ServeHTTP(rec, httptest.NewRequestWithContext(
|
|
||||||
t.Context(), http.MethodGet, path, nil))
|
|
||||||
|
|
||||||
var answer map[string]any
|
|
||||||
|
|
||||||
err := json.Unmarshal(rec.Body.Bytes(), &answer)
|
|
||||||
if err != nil {
|
|
||||||
t.Errorf("%s: decoding the answer: %v; answer %s", path, err, rec.Body)
|
|
||||||
|
|
||||||
continue
|
|
||||||
}
|
|
||||||
|
|
||||||
meta, _ := answer["@meta"].(map[string]any)
|
|
||||||
if _, ok := meta["execution_time_ms"].(float64); !ok {
|
|
||||||
t.Errorf("%s: @meta.execution_time_ms is not a number: %s",
|
|
||||||
path, rec.Body)
|
|
||||||
}
|
|
||||||
|
|
||||||
for _, value := range stringsIn(answer) {
|
|
||||||
if numberWithUnit.MatchString(value) {
|
|
||||||
t.Errorf("%s: %q is a number with a unit in a string", path, value)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestJSONAnswersCarryMeta sends requests to every JSON route through the
|
|
||||||
// server's router, and checks that each answer carries @meta with its three
|
|
||||||
// fields. The answers include successes, and errors that the handlers, the
|
|
||||||
// router and the middleware write. Before the fix a panic in a handler, and a
|
|
||||||
// method /status.json or the health check does not take, answered with an
|
|
||||||
// empty body (https://git.eeqj.de/sneak/routewatch/issues/201).
|
|
||||||
//
|
|
||||||
// The requests run in a synctest bubble, whose clock is fake, so a stall on
|
|
||||||
// the host does not use up the 2-second limit of the health check or the
|
|
||||||
// 4-second limit of the stats handlers.
|
|
||||||
func TestJSONAnswersCarryMeta(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
// JSON routes whose handlers write nothing or panic, so that the JSON
|
|
||||||
// check on the route writes the error answer itself.
|
|
||||||
const (
|
|
||||||
writesNothingPath = "/writes-nothing.json"
|
|
||||||
panicsPath = "/panics.json"
|
|
||||||
)
|
|
||||||
|
|
||||||
s := newServerWithTestRoutes(t)
|
|
||||||
useFakeResolver(s, &fakeResolver{})
|
|
||||||
s.streamer.GetMetricsTracker().SetConnected(true)
|
|
||||||
s.router.Get(writesNothingPath, JSONValidationMiddleware(http.HandlerFunc(
|
|
||||||
func(http.ResponseWriter, *http.Request) {})).ServeHTTP)
|
|
||||||
s.router.Get(panicsPath, JSONValidationMiddleware(http.HandlerFunc(
|
|
||||||
func(http.ResponseWriter, *http.Request) { panic(errDiskIO) })).ServeHTTP)
|
|
||||||
|
|
||||||
failing := New(failingDB{}, newTestTable(),
|
|
||||||
streamer.New(logger.New(), metrics.New()), logger.New(), &config.Config{})
|
|
||||||
useFakeResolver(failing, &fakeResolver{})
|
|
||||||
|
|
||||||
tests := []struct {
|
|
||||||
method string
|
|
||||||
path string
|
|
||||||
wantCode int
|
|
||||||
}{
|
|
||||||
// Answers that succeed
|
|
||||||
{http.MethodGet, statusJSONPath, http.StatusOK},
|
|
||||||
{http.MethodGet, healthCheckPath, http.StatusOK},
|
|
||||||
{http.MethodGet, "/ip", http.StatusOK},
|
|
||||||
{http.MethodGet, ownAddressPath, http.StatusOK},
|
|
||||||
{http.MethodGet, "/ip/" + addrInTestPrefixV4, http.StatusOK},
|
|
||||||
{http.MethodGet, statsPath, http.StatusOK},
|
|
||||||
{http.MethodGet, "/api/v1/ip/" + addrInTestPrefixV4, http.StatusOK},
|
|
||||||
{http.MethodGet, asJSONPath, http.StatusOK},
|
|
||||||
{http.MethodGet, "/api/v1/prefix/" + testPrefixV4, http.StatusOK},
|
|
||||||
{http.MethodGet, "/api/v1/search/as?q=google", http.StatusOK},
|
|
||||||
// Errors the handlers write
|
|
||||||
{http.MethodGet, "/ip/no-such-host.example", http.StatusBadRequest},
|
|
||||||
{http.MethodGet, "/api/v1/ip/no-such-host.example", http.StatusBadRequest},
|
|
||||||
{http.MethodGet, "/api/v1/as/foo", http.StatusBadRequest},
|
|
||||||
{http.MethodGet, "/api/v1/as/0", http.StatusNotFound},
|
|
||||||
{http.MethodGet, "/api/v1/prefix/192.0.2.0/33", http.StatusBadRequest},
|
|
||||||
{http.MethodGet, "/api/v1/prefix/198.51.100.0/24", http.StatusNotFound},
|
|
||||||
{http.MethodGet, "/api/v1/search/as", http.StatusBadRequest},
|
|
||||||
// Errors the router and the middleware write
|
|
||||||
{http.MethodGet, "/api/v1/no-such-route", http.StatusNotFound},
|
|
||||||
{http.MethodGet, "/ip/192.0.2.1/nothing", http.StatusNotFound},
|
|
||||||
{http.MethodPost, statsPath, http.StatusMethodNotAllowed},
|
|
||||||
{http.MethodPost, ownAddressPath, http.StatusMethodNotAllowed},
|
|
||||||
{http.MethodPost, statusJSONPath, http.StatusMethodNotAllowed},
|
|
||||||
{http.MethodPut, healthCheckPath, http.StatusMethodNotAllowed},
|
|
||||||
{http.MethodGet, writesNothingPath, http.StatusInternalServerError},
|
|
||||||
{http.MethodGet, panicsPath, http.StatusInternalServerError},
|
|
||||||
}
|
|
||||||
|
|
||||||
synctest.Test(t, func(t *testing.T) {
|
|
||||||
for _, tt := range tests {
|
|
||||||
checkAnswerCarriesMeta(t, s, httptest.NewRequestWithContext(
|
|
||||||
t.Context(), tt.method, tt.path, nil), tt.wantCode)
|
|
||||||
}
|
|
||||||
|
|
||||||
// A request whose time limit has already passed answers 504. The
|
|
||||||
// handlers write the answers of the first five routes, and the JSON
|
|
||||||
// check on the route writes the last.
|
|
||||||
pastTimeLimit, cancel := context.WithDeadline(t.Context(), time.Now())
|
|
||||||
defer cancel()
|
|
||||||
|
|
||||||
for _, path := range []string{
|
|
||||||
statusJSONPath, statsPath, "/ip/" + addrInTestPrefixV4,
|
|
||||||
"/api/v1/ip/" + addrInTestPrefixV4, asJSONPath, writesNothingPath,
|
|
||||||
} {
|
|
||||||
checkAnswerCarriesMeta(t, s, httptest.NewRequestWithContext(
|
|
||||||
pastTimeLimit, http.MethodGet, path, nil), http.StatusGatewayTimeout)
|
|
||||||
}
|
|
||||||
|
|
||||||
// With a store whose reads fail, the health check answers 503, and
|
|
||||||
// the handlers of the other routes 500.
|
|
||||||
checkAnswerCarriesMeta(t, failing, httptest.NewRequestWithContext(
|
|
||||||
t.Context(), http.MethodGet, healthCheckPath, nil),
|
|
||||||
http.StatusServiceUnavailable)
|
|
||||||
|
|
||||||
for _, path := range []string{
|
|
||||||
statusJSONPath, statsPath, asJSONPath,
|
|
||||||
"/api/v1/ip/" + addrInTestPrefixV4, "/ip/" + addrInTestPrefixV4,
|
|
||||||
} {
|
|
||||||
checkAnswerCarriesMeta(t, failing, httptest.NewRequestWithContext(
|
|
||||||
t.Context(), http.MethodGet, path, nil), http.StatusInternalServerError)
|
|
||||||
}
|
|
||||||
})
|
|
||||||
}
|
|
||||||
|
|
||||||
// checkAnswerCarriesMeta serves req through the server's router, and checks
|
|
||||||
// that it answers wantCode with @meta and its three fields.
|
|
||||||
func checkAnswerCarriesMeta(
|
|
||||||
t *testing.T, s *Server, req *http.Request, wantCode int,
|
|
||||||
) {
|
|
||||||
t.Helper()
|
|
||||||
|
|
||||||
// The values README "JSON API" gives the two fields that never change.
|
|
||||||
const (
|
|
||||||
wantTimeZone = "UTC"
|
|
||||||
wantAPIVersion = 1
|
|
||||||
)
|
|
||||||
|
|
||||||
name := req.Method + " " + req.URL.String()
|
|
||||||
|
|
||||||
rec := httptest.NewRecorder()
|
|
||||||
s.router.ServeHTTP(rec, req)
|
|
||||||
|
|
||||||
if rec.Code != wantCode {
|
|
||||||
t.Errorf("%s: status %d, want %d; body %s", name, rec.Code, wantCode, rec.Body)
|
|
||||||
}
|
|
||||||
|
|
||||||
var answer map[string]any
|
|
||||||
|
|
||||||
err := json.Unmarshal(rec.Body.Bytes(), &answer)
|
|
||||||
if err != nil {
|
|
||||||
t.Errorf("%s: decoding the answer %q: %v", name, rec.Body, err)
|
|
||||||
|
|
||||||
return
|
|
||||||
}
|
|
||||||
|
|
||||||
meta, _ := answer[jsonKeyMeta].(map[string]any)
|
|
||||||
_, timeIsNumber := meta[metaKeyExecutionTimeMs].(float64)
|
|
||||||
|
|
||||||
if meta[metaKeyTimeZone] != wantTimeZone ||
|
|
||||||
meta[metaKeyAPIVersion] != float64(wantAPIVersion) || !timeIsNumber {
|
|
||||||
t.Errorf("%s: @meta %v, want time_zone %q, api_version %d and a "+
|
|
||||||
"number in execution_time_ms", name, meta, wantTimeZone, wantAPIVersion)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// stringsIn returns every string in a decoded JSON value, at any depth.
|
|
||||||
func stringsIn(value any) []string {
|
|
||||||
var found []string
|
|
||||||
|
|
||||||
switch v := value.(type) {
|
|
||||||
case string:
|
|
||||||
found = append(found, v)
|
|
||||||
case []any:
|
|
||||||
for _, item := range v {
|
|
||||||
found = append(found, stringsIn(item)...)
|
|
||||||
}
|
|
||||||
case map[string]any:
|
|
||||||
for _, item := range v {
|
|
||||||
found = append(found, stringsIn(item)...)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
return found
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestPagesCarryTheCSPNonce checks that every script and style block of each
|
// TestPagesCarryTheCSPNonce checks that every script and style block of each
|
||||||
// page carries the nonce its Content-Security-Policy header names, so that the
|
// page carries the nonce its Content-Security-Policy header names, so that the
|
||||||
// browser runs and applies them, that each answer has a nonce of its own, and
|
// browser runs and applies them, that each answer has a nonce of its own, and
|
||||||
@@ -418,141 +183,6 @@ func TestForwardedForFromUntrustedPeerIsIgnored(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// TestDefaultTrustedProxiesGiveTheForwardedClient builds the server from the
|
|
||||||
// configuration with TRUSTED_PROXIES unset. A request from a reverse proxy at
|
|
||||||
// a private address must be looked up by /ip/, shown on the home page's card
|
|
||||||
// and logged as the client its X-Forwarded-For names. A direct request from a
|
|
||||||
// public address with a forged X-Forwarded-For keeps its own address. Before
|
|
||||||
// the fix no proxy was trusted unless TRUSTED_PROXIES named it, so /ip/ gave
|
|
||||||
// the proxy's address (https://git.eeqj.de/sneak/routewatch/issues/198).
|
|
||||||
func TestDefaultTrustedProxiesGiveTheForwardedClient(t *testing.T) {
|
|
||||||
t.Setenv("PORT", "")
|
|
||||||
t.Setenv("XDG_DATA_HOME", "")
|
|
||||||
|
|
||||||
// t.Setenv puts the variable back when the test ends.
|
|
||||||
t.Setenv("TRUSTED_PROXIES", "")
|
|
||||||
|
|
||||||
err := os.Unsetenv("TRUSTED_PROXIES")
|
|
||||||
if err != nil {
|
|
||||||
t.Fatal(err)
|
|
||||||
}
|
|
||||||
|
|
||||||
cfg, err := config.New()
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("config.New: %v", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
var requestLog bytes.Buffer
|
|
||||||
|
|
||||||
s := newServerWithConfig(t, cfg, &requestLog)
|
|
||||||
|
|
||||||
tests := []struct {
|
|
||||||
name string
|
|
||||||
peer string
|
|
||||||
want string
|
|
||||||
}{
|
|
||||||
{"through a reverse proxy", reverseProxyPeer, forwardedClient},
|
|
||||||
{"direct from a public address", untrustedPeer, untrustedPeerAddr},
|
|
||||||
}
|
|
||||||
|
|
||||||
for _, tt := range tests {
|
|
||||||
requestLog.Reset()
|
|
||||||
|
|
||||||
get := func(path string) *httptest.ResponseRecorder {
|
|
||||||
req := httptest.NewRequestWithContext(t.Context(), http.MethodGet, path, nil)
|
|
||||||
req.RemoteAddr = tt.peer
|
|
||||||
req.Header.Set("X-Forwarded-For", forwardedClient)
|
|
||||||
|
|
||||||
rec := httptest.NewRecorder()
|
|
||||||
s.router.ServeHTTP(rec, req)
|
|
||||||
|
|
||||||
return rec
|
|
||||||
}
|
|
||||||
|
|
||||||
var answer struct {
|
|
||||||
Data IPLookupResponse `json:"data"`
|
|
||||||
}
|
|
||||||
|
|
||||||
rec := get("/ip/")
|
|
||||||
|
|
||||||
err := json.Unmarshal(rec.Body.Bytes(), &answer)
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("%s: decoding the answer %q: %v", tt.name, rec.Body, err)
|
|
||||||
}
|
|
||||||
|
|
||||||
if answer.Data.Query != tt.want {
|
|
||||||
t.Errorf("%s: /ip/ looked up %q, want %q", tt.name, answer.Data.Query, tt.want)
|
|
||||||
}
|
|
||||||
|
|
||||||
page := get("/").Body.String()
|
|
||||||
if !strings.Contains(page, `data-client-ip="`+tt.want+`"`) {
|
|
||||||
t.Errorf("%s: the home page's card does not hold %s", tt.name, tt.want)
|
|
||||||
}
|
|
||||||
|
|
||||||
logged := loggedAddresses(requestLog.String())
|
|
||||||
if len(logged) == 0 {
|
|
||||||
t.Errorf("%s: the request log names no address", tt.name)
|
|
||||||
}
|
|
||||||
|
|
||||||
for _, addr := range logged {
|
|
||||||
if addr != tt.want {
|
|
||||||
t.Errorf("%s: the request log gives %s, want %s", tt.name, addr, tt.want)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// newServerWithConfig returns a server built with cfg, on a new database, that
|
|
||||||
// reads the routes of newTestTable and logs its requests as JSON to
|
|
||||||
// requestLog.
|
|
||||||
func newServerWithConfig(
|
|
||||||
t *testing.T, cfg *config.Config, requestLog io.Writer,
|
|
||||||
) *Server {
|
|
||||||
t.Helper()
|
|
||||||
|
|
||||||
cfg.StateDir = t.TempDir()
|
|
||||||
|
|
||||||
db, err := database.New(cfg, logger.New())
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("database.New: %v", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
t.Cleanup(func() { _ = db.Close() })
|
|
||||||
|
|
||||||
s := New(db, newTestTable(), streamer.New(logger.New(), metrics.New()),
|
|
||||||
&logger.Logger{Logger: slog.New(slog.NewJSONHandler(requestLog, nil))}, cfg)
|
|
||||||
useFakeResolver(s, &fakeResolver{})
|
|
||||||
|
|
||||||
return s
|
|
||||||
}
|
|
||||||
|
|
||||||
// loggedAddresses returns the address each line of the JSON request log
|
|
||||||
// names, without the port a direct request is logged with.
|
|
||||||
func loggedAddresses(requestLog string) []string {
|
|
||||||
var addrs []string
|
|
||||||
|
|
||||||
for line := range strings.Lines(requestLog) {
|
|
||||||
//nolint:tagliatelle // the field name is the key the request log uses
|
|
||||||
var entry struct {
|
|
||||||
RemoteAddr string `json:"remote_addr"`
|
|
||||||
}
|
|
||||||
|
|
||||||
err := json.Unmarshal([]byte(line), &entry)
|
|
||||||
if err != nil || entry.RemoteAddr == "" {
|
|
||||||
continue
|
|
||||||
}
|
|
||||||
|
|
||||||
addr, _, err := net.SplitHostPort(entry.RemoteAddr)
|
|
||||||
if err != nil {
|
|
||||||
addr = entry.RemoteAddr
|
|
||||||
}
|
|
||||||
|
|
||||||
addrs = append(addrs, addr)
|
|
||||||
}
|
|
||||||
|
|
||||||
return addrs
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestRealIPMiddlewareTrustsOnlyConfiguredProxies checks which address
|
// TestRealIPMiddlewareTrustsOnlyConfiguredProxies checks which address
|
||||||
// RealIPMiddleware gives a request: the TCP peer, unless the peer is a trusted
|
// RealIPMiddleware gives a request: the TCP peer, unless the peer is a trusted
|
||||||
// proxy; then the last X-Forwarded-For entry that is not a trusted proxy,
|
// proxy; then the last X-Forwarded-For entry that is not a trusted proxy,
|
||||||
|
|||||||
+21
-28
@@ -38,37 +38,30 @@ type PrefixDistribution struct {
|
|||||||
Count int `json:"count"`
|
Count int `json:"count"`
|
||||||
}
|
}
|
||||||
|
|
||||||
// IPInfo is what the IP lookup knows about the route of an address and that
|
// IPInfo represents comprehensive IP information for the /ip endpoint
|
||||||
// route's origin AS. A field is nil, null in JSON, when its value is not known.
|
|
||||||
// Every field is nil for an address with no route.
|
|
||||||
//
|
//
|
||||||
//nolint:tagliatelle // field names are the existing JSON API
|
//nolint:tagliatelle // field names are the existing JSON API
|
||||||
type IPInfo struct {
|
type IPInfo struct {
|
||||||
// Netblock is the most specific prefix that holds the address.
|
IP string `json:"ip"`
|
||||||
Netblock *string `json:"netblock"`
|
Netblock string `json:"netblock"`
|
||||||
MaskLength *int `json:"mask_length"`
|
MaskLength int `json:"mask_length"`
|
||||||
IPVersion *int `json:"ip_version"`
|
IPVersion int `json:"ip_version"`
|
||||||
NumPeers *int `json:"num_peers"`
|
NumPeers int `json:"num_peers"`
|
||||||
// ASN is the origin AS that the most peers carry the netblock with.
|
// AS information
|
||||||
ASN *int `json:"asn"`
|
ASN int `json:"asn"`
|
||||||
// NumOrigins and OriginASNs are how many origin ASes peers carry the
|
ASName string `json:"as_name,omitempty"`
|
||||||
// netblock with, and which, in AS number order.
|
Handle string `json:"handle,omitempty"`
|
||||||
NumOrigins *int `json:"num_origins"`
|
Description string `json:"description,omitempty"`
|
||||||
OriginASNs []int `json:"origin_asns"`
|
OrgName string `json:"org_name,omitempty"`
|
||||||
// The details the database holds about the AS in ASN.
|
OrgID string `json:"org_id,omitempty"`
|
||||||
ASName *string `json:"as_name"`
|
Address string `json:"address,omitempty"`
|
||||||
Handle *string `json:"handle"`
|
CountryCode string `json:"country_code,omitempty"`
|
||||||
Description *string `json:"description"`
|
AbuseEmail string `json:"abuse_email,omitempty"`
|
||||||
OrgName *string `json:"org_name"`
|
RIR string `json:"rir,omitempty"`
|
||||||
OrgID *string `json:"org_id"`
|
// Timestamps. LastSeen is zero, and left out of JSON, when the prefix
|
||||||
Address *string `json:"address"`
|
// keeps no route.
|
||||||
CountryCode *string `json:"country_code"`
|
FirstSeen time.Time `json:"first_seen"`
|
||||||
AbuseEmail *string `json:"abuse_email"`
|
LastSeen time.Time `json:"last_seen,omitzero"`
|
||||||
RIR *string `json:"rir"`
|
|
||||||
// FirstSeen is when RouteWatch first saw the netblock. LastSeen is the
|
|
||||||
// time of the netblock's newest kept route.
|
|
||||||
FirstSeen *time.Time `json:"first_seen"`
|
|
||||||
LastSeen *time.Time `json:"last_seen"`
|
|
||||||
// Indicates if WHOIS data needs refresh (not serialized)
|
// Indicates if WHOIS data needs refresh (not serialized)
|
||||||
NeedsWHOISRefresh bool `json:"-"`
|
NeedsWHOISRefresh bool `json:"-"`
|
||||||
}
|
}
|
||||||
@@ -1,230 +0,0 @@
|
|||||||
package server
|
|
||||||
|
|
||||||
import (
|
|
||||||
"fmt"
|
|
||||||
"net/http"
|
|
||||||
"net/http/httptest"
|
|
||||||
"net/netip"
|
|
||||||
"strings"
|
|
||||||
"testing"
|
|
||||||
|
|
||||||
"git.eeqj.de/sneak/routewatch/internal/config"
|
|
||||||
"git.eeqj.de/sneak/routewatch/internal/database"
|
|
||||||
"git.eeqj.de/sneak/routewatch/internal/logger"
|
|
||||||
"git.eeqj.de/sneak/routewatch/internal/metrics"
|
|
||||||
"git.eeqj.de/sneak/routewatch/internal/ristypes"
|
|
||||||
"git.eeqj.de/sneak/routewatch/internal/routetable"
|
|
||||||
"git.eeqj.de/sneak/routewatch/internal/streamer"
|
|
||||||
)
|
|
||||||
|
|
||||||
// testUncarriedASN is an AS the test's routing table does not carry for the prefix.
|
|
||||||
const testUncarriedASN = 64501
|
|
||||||
|
|
||||||
// newPrefixDetailServer returns a server whose routing table holds testPrefixV4
|
|
||||||
// with origin AS 64500, and whose database holds the given origin changes.
|
|
||||||
func newPrefixDetailServer(
|
|
||||||
t *testing.T, changes []routetable.OriginChange,
|
|
||||||
) *Server {
|
|
||||||
t.Helper()
|
|
||||||
|
|
||||||
db, err := database.New(&config.Config{StateDir: t.TempDir()}, logger.New())
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("database.New: %v", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
t.Cleanup(func() { _ = db.Close() })
|
|
||||||
|
|
||||||
table := emptyTable()
|
|
||||||
announce(table, testPrefixV4, 64500, testPeer(1), testRouteTime(0))
|
|
||||||
|
|
||||||
err = db.WriteOriginChanges(changes)
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("WriteOriginChanges: %v", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
return New(db, table, streamer.New(logger.New(), metrics.New()),
|
|
||||||
logger.New(), &config.Config{})
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestPrefixDetailShowsOriginChanges checks that the prefix page shows the
|
|
||||||
// origin changes newest first, with the announced and withdrawn rows and the
|
|
||||||
// AS link. The rows use an AS the routing table does not carry for the
|
|
||||||
// prefix, so the link comes from the origin changes table.
|
|
||||||
func TestPrefixDetailShowsOriginChanges(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
prefix := netip.MustParsePrefix(testPrefixV4)
|
|
||||||
|
|
||||||
s := newPrefixDetailServer(t, []routetable.OriginChange{
|
|
||||||
{Prefix: prefix, OriginASN: testUncarriedASN, Announced: true,
|
|
||||||
PeerCount: 1, Time: testRouteTime(1)},
|
|
||||||
{Prefix: prefix, OriginASN: testUncarriedASN, Announced: false,
|
|
||||||
PeerCount: 0, Time: testRouteTime(2)},
|
|
||||||
})
|
|
||||||
|
|
||||||
rec := httptest.NewRecorder()
|
|
||||||
s.router.ServeHTTP(rec, httptest.NewRequestWithContext(
|
|
||||||
t.Context(), http.MethodGet, "/prefix/"+testPrefixV4, nil))
|
|
||||||
|
|
||||||
if rec.Code != http.StatusOK {
|
|
||||||
t.Fatalf("status %d, want %d; body %s", rec.Code, http.StatusOK, rec.Body)
|
|
||||||
}
|
|
||||||
|
|
||||||
page := rec.Body.String()
|
|
||||||
|
|
||||||
withdrawn := strings.Index(page, "Withdrawn")
|
|
||||||
announced := strings.Index(page, "Announced")
|
|
||||||
|
|
||||||
if withdrawn < 0 || announced < 0 {
|
|
||||||
t.Fatalf("page does not show the announced and withdrawn rows")
|
|
||||||
}
|
|
||||||
|
|
||||||
// The withdrawn row is newer, so it comes first.
|
|
||||||
if withdrawn > announced {
|
|
||||||
t.Errorf("the withdrawn row is not shown before the announced row")
|
|
||||||
}
|
|
||||||
|
|
||||||
if !strings.Contains(page, `href="/as/64501"`) {
|
|
||||||
t.Errorf("page does not link to AS64501")
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestPrefixDetailShowsOriginZeroAsNoKnownOrigin checks that the origin
|
|
||||||
// changes table shows AS 0 as no known origin, without a link to AS0.
|
|
||||||
func TestPrefixDetailShowsOriginZeroAsNoKnownOrigin(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
prefix := netip.MustParsePrefix(testPrefixV4)
|
|
||||||
|
|
||||||
s := newPrefixDetailServer(t, []routetable.OriginChange{
|
|
||||||
{Prefix: prefix, OriginASN: ristypes.NoOrigin, Announced: true,
|
|
||||||
PeerCount: 1, Time: testRouteTime(1)},
|
|
||||||
})
|
|
||||||
|
|
||||||
rec := httptest.NewRecorder()
|
|
||||||
s.router.ServeHTTP(rec, httptest.NewRequestWithContext(
|
|
||||||
t.Context(), http.MethodGet, "/prefix/"+testPrefixV4, nil))
|
|
||||||
|
|
||||||
if rec.Code != http.StatusOK {
|
|
||||||
t.Fatalf("status %d, want %d; body %s", rec.Code, http.StatusOK, rec.Body)
|
|
||||||
}
|
|
||||||
|
|
||||||
page := rec.Body.String()
|
|
||||||
|
|
||||||
if !strings.Contains(page, "No known origin") {
|
|
||||||
t.Errorf("page does not say the origin change has no known origin")
|
|
||||||
}
|
|
||||||
|
|
||||||
if strings.Contains(page, `href="/as/0"`) {
|
|
||||||
t.Errorf("page links to AS0")
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestPrefixDetailSaysWhenThereAreMoreChangesThanItShows checks that the
|
|
||||||
// prefix page says so when a prefix has more origin changes than the page's
|
|
||||||
// limit.
|
|
||||||
func TestPrefixDetailSaysWhenThereAreMoreChangesThanItShows(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
prefix := netip.MustParsePrefix(testPrefixV4)
|
|
||||||
|
|
||||||
changes := make([]routetable.OriginChange, 0, maxOriginChangesOnPage+1)
|
|
||||||
for i := range maxOriginChangesOnPage + 1 {
|
|
||||||
changes = append(changes, routetable.OriginChange{
|
|
||||||
Prefix: prefix, OriginASN: 64500, Announced: true, PeerCount: 1,
|
|
||||||
Time: testRouteTime(i),
|
|
||||||
})
|
|
||||||
}
|
|
||||||
|
|
||||||
s := newPrefixDetailServer(t, changes)
|
|
||||||
|
|
||||||
rec := httptest.NewRecorder()
|
|
||||||
s.router.ServeHTTP(rec, httptest.NewRequestWithContext(
|
|
||||||
t.Context(), http.MethodGet, "/prefix/"+testPrefixV4, nil))
|
|
||||||
|
|
||||||
if rec.Code != http.StatusOK {
|
|
||||||
t.Fatalf("status %d, want %d; body %s", rec.Code, http.StatusOK, rec.Body)
|
|
||||||
}
|
|
||||||
|
|
||||||
page := rec.Body.String()
|
|
||||||
|
|
||||||
note := fmt.Sprintf("more than %d changes", maxOriginChangesOnPage)
|
|
||||||
if !strings.Contains(page, note) {
|
|
||||||
t.Errorf("page does not say there are more than %d changes",
|
|
||||||
maxOriginChangesOnPage)
|
|
||||||
}
|
|
||||||
|
|
||||||
if got := strings.Count(page, "<td>Announced</td>"); got != maxOriginChangesOnPage {
|
|
||||||
t.Errorf("page shows %d change rows, want %d", got,
|
|
||||||
maxOriginChangesOnPage)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestPrefixDetailShowsNoNoteAtTheLimit checks that a prefix with exactly the
|
|
||||||
// page's limit of origin changes shows all of them and no note.
|
|
||||||
func TestPrefixDetailShowsNoNoteAtTheLimit(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
prefix := netip.MustParsePrefix(testPrefixV4)
|
|
||||||
|
|
||||||
changes := make([]routetable.OriginChange, 0, maxOriginChangesOnPage)
|
|
||||||
for i := range maxOriginChangesOnPage {
|
|
||||||
changes = append(changes, routetable.OriginChange{
|
|
||||||
Prefix: prefix, OriginASN: 64500, Announced: true, PeerCount: 1,
|
|
||||||
Time: testRouteTime(i),
|
|
||||||
})
|
|
||||||
}
|
|
||||||
|
|
||||||
s := newPrefixDetailServer(t, changes)
|
|
||||||
|
|
||||||
rec := httptest.NewRecorder()
|
|
||||||
s.router.ServeHTTP(rec, httptest.NewRequestWithContext(
|
|
||||||
t.Context(), http.MethodGet, "/prefix/"+testPrefixV4, nil))
|
|
||||||
|
|
||||||
if rec.Code != http.StatusOK {
|
|
||||||
t.Fatalf("status %d, want %d; body %s", rec.Code, http.StatusOK, rec.Body)
|
|
||||||
}
|
|
||||||
|
|
||||||
page := rec.Body.String()
|
|
||||||
|
|
||||||
if strings.Contains(page, "more than") {
|
|
||||||
t.Errorf("page says there are more changes, but a prefix has %d",
|
|
||||||
maxOriginChangesOnPage)
|
|
||||||
}
|
|
||||||
|
|
||||||
if got := strings.Count(page, "<td>Announced</td>"); got != maxOriginChangesOnPage {
|
|
||||||
t.Errorf("page shows %d change rows, want %d", got,
|
|
||||||
maxOriginChangesOnPage)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestPrefixDetailShowsOriginChangeTimeAndPeerCount checks that a row shows
|
|
||||||
// its time and its peer count.
|
|
||||||
func TestPrefixDetailShowsOriginChangeTimeAndPeerCount(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
prefix := netip.MustParsePrefix(testPrefixV4)
|
|
||||||
|
|
||||||
s := newPrefixDetailServer(t, []routetable.OriginChange{
|
|
||||||
{Prefix: prefix, OriginASN: 64500, Announced: true, PeerCount: 7,
|
|
||||||
Time: testRouteTime(5)},
|
|
||||||
})
|
|
||||||
|
|
||||||
rec := httptest.NewRecorder()
|
|
||||||
s.router.ServeHTTP(rec, httptest.NewRequestWithContext(
|
|
||||||
t.Context(), http.MethodGet, "/prefix/"+testPrefixV4, nil))
|
|
||||||
|
|
||||||
if rec.Code != http.StatusOK {
|
|
||||||
t.Fatalf("status %d, want %d; body %s", rec.Code, http.StatusOK, rec.Body)
|
|
||||||
}
|
|
||||||
|
|
||||||
page := rec.Body.String()
|
|
||||||
|
|
||||||
if !strings.Contains(page, "2026-01-02 03:09:05") {
|
|
||||||
t.Errorf("page does not show the origin change's time")
|
|
||||||
}
|
|
||||||
|
|
||||||
if !strings.Contains(page, "<td>7</td>") {
|
|
||||||
t.Errorf("page does not show the origin change's peer count")
|
|
||||||
}
|
|
||||||
}
|
|
||||||
@@ -8,8 +8,8 @@ import (
|
|||||||
"github.com/go-chi/chi/v5/middleware"
|
"github.com/go-chi/chi/v5/middleware"
|
||||||
)
|
)
|
||||||
|
|
||||||
// The error messages the JSON routes answer with when no route matches the
|
// The error messages the JSON routes under /api/v1 and /ip answer with when no
|
||||||
// path, and when the route does not take the method.
|
// route matches the path, and when none takes the method.
|
||||||
const (
|
const (
|
||||||
notFoundMessage = "Not found"
|
notFoundMessage = "Not found"
|
||||||
methodNotAllowedMessage = "Method not allowed"
|
methodNotAllowedMessage = "Method not allowed"
|
||||||
@@ -19,13 +19,6 @@ const (
|
|||||||
jsonRoutesAllow = "GET, HEAD"
|
jsonRoutesAllow = "GET, HEAD"
|
||||||
)
|
)
|
||||||
|
|
||||||
// The paths of the status figures and the health check, the two JSON routes
|
|
||||||
// outside /api/v1 and /ip.
|
|
||||||
const (
|
|
||||||
statusJSONPath = "/status.json"
|
|
||||||
healthCheckPath = "/.well-known/healthcheck.json"
|
|
||||||
)
|
|
||||||
|
|
||||||
// setupRoutes configures the HTTP routes
|
// setupRoutes configures the HTTP routes
|
||||||
func (s *Server) setupRoutes() {
|
func (s *Server) setupRoutes() {
|
||||||
r := chi.NewRouter()
|
r := chi.NewRouter()
|
||||||
@@ -39,8 +32,6 @@ func (s *Server) setupRoutes() {
|
|||||||
r.Use(middleware.RequestID)
|
r.Use(middleware.RequestID)
|
||||||
r.Use(RealIPMiddleware(s.trustedProxies))
|
r.Use(RealIPMiddleware(s.trustedProxies))
|
||||||
r.Use(RequestLoggerMiddleware(s.logger.Logger)) // Structured request logging
|
r.Use(RequestLoggerMiddleware(s.logger.Logger)) // Structured request logging
|
||||||
// Recoverer answers a panic on a page. JSONValidationMiddleware recovers a
|
|
||||||
// panic on a JSON route itself, so that the answer is JSON with @meta.
|
|
||||||
r.Use(middleware.Recoverer)
|
r.Use(middleware.Recoverer)
|
||||||
r.Use(JSONResponseMiddleware)
|
r.Use(JSONResponseMiddleware)
|
||||||
// Timeout must come after JSONResponseMiddleware, which sends its answer
|
// Timeout must come after JSONResponseMiddleware, which sends its answer
|
||||||
@@ -52,6 +43,11 @@ func (s *Server) setupRoutes() {
|
|||||||
// Routes
|
// Routes
|
||||||
r.Get("/", s.handleIndex())
|
r.Get("/", s.handleIndex())
|
||||||
r.Get("/status", s.handleStatusHTML())
|
r.Get("/status", s.handleStatusHTML())
|
||||||
|
r.Get("/status.json", JSONValidationMiddleware(s.handleStatusJSON()).ServeHTTP)
|
||||||
|
r.Get(
|
||||||
|
"/.well-known/healthcheck.json",
|
||||||
|
JSONValidationMiddleware(s.handleHealthCheck()).ServeHTTP,
|
||||||
|
)
|
||||||
|
|
||||||
// AS and prefix detail pages
|
// AS and prefix detail pages
|
||||||
r.Get("/as/{asn}", s.handleASDetail())
|
r.Get("/as/{asn}", s.handleASDetail())
|
||||||
@@ -59,10 +55,6 @@ func (s *Server) setupRoutes() {
|
|||||||
r.Get("/prefixlength/{length}", s.handlePrefixLength())
|
r.Get("/prefixlength/{length}", s.handlePrefixLength())
|
||||||
r.Get("/prefixlength6/{length}", s.handlePrefixLength6())
|
r.Get("/prefixlength6/{length}", s.handlePrefixLength6())
|
||||||
|
|
||||||
// The pages that list what two of the home page's cards count
|
|
||||||
r.Get("/bogons", s.handleBogons())
|
|
||||||
r.Get("/multiple-origins", s.handleMultipleOrigins())
|
|
||||||
|
|
||||||
// The JSON routes answer a path or method they do not take with the JSON
|
// The JSON routes answer a path or method they do not take with the JSON
|
||||||
// error body, as they answer everything else.
|
// error body, as they answer everything else.
|
||||||
jsonNotFound := func(w http.ResponseWriter, _ *http.Request) {
|
jsonNotFound := func(w http.ResponseWriter, _ *http.Request) {
|
||||||
@@ -74,21 +66,6 @@ func (s *Server) setupRoutes() {
|
|||||||
s.writeJSONError(w, http.StatusMethodNotAllowed, methodNotAllowedMessage)
|
s.writeJSONError(w, http.StatusMethodNotAllowed, methodNotAllowedMessage)
|
||||||
}
|
}
|
||||||
|
|
||||||
// The status figures and the health check. chi sets one 405 answer for a
|
|
||||||
// whole router. These two routes are on the pages' router, whose 405 stays
|
|
||||||
// chi's own. So each method they do not take is routed to the JSON 405
|
|
||||||
// here.
|
|
||||||
r.Get(statusJSONPath, JSONValidationMiddleware(s.handleStatusJSON()).ServeHTTP)
|
|
||||||
r.Get(healthCheckPath, JSONValidationMiddleware(s.handleHealthCheck()).ServeHTTP)
|
|
||||||
|
|
||||||
for _, method := range []string{
|
|
||||||
http.MethodPost, http.MethodPut, http.MethodPatch, http.MethodDelete,
|
|
||||||
http.MethodOptions, http.MethodTrace, http.MethodConnect,
|
|
||||||
} {
|
|
||||||
r.MethodFunc(method, statusJSONPath, jsonMethodNotAllowed)
|
|
||||||
r.MethodFunc(method, healthCheckPath, jsonMethodNotAllowed)
|
|
||||||
}
|
|
||||||
|
|
||||||
// IP info JSON endpoints (replaces old /ip redirect)
|
// IP info JSON endpoints (replaces old /ip redirect)
|
||||||
r.Route("/ip", func(r chi.Router) {
|
r.Route("/ip", func(r chi.Router) {
|
||||||
r.Use(JSONValidationMiddleware)
|
r.Use(JSONValidationMiddleware)
|
||||||
|
|||||||
@@ -5,7 +5,6 @@ import (
|
|||||||
"net/http"
|
"net/http"
|
||||||
"net/http/httptest"
|
"net/http/httptest"
|
||||||
"testing"
|
"testing"
|
||||||
"testing/synctest"
|
|
||||||
)
|
)
|
||||||
|
|
||||||
// TestRoutesAnswerHEAD sends HEAD for a page and JSON routes through the
|
// TestRoutesAnswerHEAD sends HEAD for a page and JSON routes through the
|
||||||
@@ -13,23 +12,15 @@ import (
|
|||||||
// lookup of the caller's own address, are in the list because chi's GetHead
|
// lookup of the caller's own address, are in the list because chi's GetHead
|
||||||
// does not reach them. Before the fix every route answered HEAD with 405
|
// does not reach them. Before the fix every route answered HEAD with 405
|
||||||
// (https://git.eeqj.de/sneak/routewatch/issues/111).
|
// (https://git.eeqj.de/sneak/routewatch/issues/111).
|
||||||
//
|
|
||||||
// The requests run in a synctest bubble, whose clock is fake, so a stall on
|
|
||||||
// the host does not use up the stats handlers' 4-second limit. The resolver
|
|
||||||
// is fake because no time limit would cut off a real DNS lookup in a bubble.
|
|
||||||
func TestRoutesAnswerHEAD(t *testing.T) {
|
func TestRoutesAnswerHEAD(t *testing.T) {
|
||||||
t.Parallel()
|
t.Parallel()
|
||||||
|
|
||||||
s := newServerWithTestRoutes(t)
|
s := newServerWithTestRoutes(t)
|
||||||
s.resolver = &fakeResolver{}
|
|
||||||
|
|
||||||
for _, path := range []string{"/", statusJSONPath, statsPath, "/ip", "/ip/"} {
|
for _, path := range []string{"/", statsPath, "/ip", "/ip/"} {
|
||||||
rec := httptest.NewRecorder()
|
rec := httptest.NewRecorder()
|
||||||
|
s.router.ServeHTTP(rec, httptest.NewRequestWithContext(
|
||||||
synctest.Test(t, func(t *testing.T) {
|
t.Context(), http.MethodHead, path, nil))
|
||||||
s.router.ServeHTTP(rec, httptest.NewRequestWithContext(
|
|
||||||
t.Context(), http.MethodHead, path, nil))
|
|
||||||
})
|
|
||||||
|
|
||||||
if rec.Code != http.StatusOK {
|
if rec.Code != http.StatusOK {
|
||||||
t.Errorf("HEAD %s: status %d, want %d; body %s",
|
t.Errorf("HEAD %s: status %d, want %d; body %s",
|
||||||
@@ -40,13 +31,10 @@ func TestRoutesAnswerHEAD(t *testing.T) {
|
|||||||
|
|
||||||
// TestJSONRoutesAnswerUnmatchedRequestsWithJSONErrors sends a method the JSON
|
// TestJSONRoutesAnswerUnmatchedRequestsWithJSONErrors sends a method the JSON
|
||||||
// routes do not take, and a path none of them matches, to each group of JSON
|
// routes do not take, and a path none of them matches, to each group of JSON
|
||||||
// routes through the server's router. It also sends a method they do not take
|
// routes through the server's router. Each must answer with the JSON error
|
||||||
// to /status.json and the health check. Each must answer with the JSON error
|
|
||||||
// body, and a 405 must name the methods the routes take in its Allow header.
|
// body, and a 405 must name the methods the routes take in its Allow header.
|
||||||
// Before the fix a 405 said "Internal server error" and a 404 was chi's plain
|
// Before the fix a 405 said "Internal server error" and a 404 was chi's plain
|
||||||
// text answer (https://git.eeqj.de/sneak/routewatch/issues/111).
|
// text answer (https://git.eeqj.de/sneak/routewatch/issues/111).
|
||||||
// /status.json and the health check gave chi's empty 405
|
|
||||||
// (https://git.eeqj.de/sneak/routewatch/issues/201).
|
|
||||||
func TestJSONRoutesAnswerUnmatchedRequestsWithJSONErrors(t *testing.T) {
|
func TestJSONRoutesAnswerUnmatchedRequestsWithJSONErrors(t *testing.T) {
|
||||||
t.Parallel()
|
t.Parallel()
|
||||||
|
|
||||||
@@ -60,14 +48,6 @@ func TestJSONRoutesAnswerUnmatchedRequestsWithJSONErrors(t *testing.T) {
|
|||||||
}{
|
}{
|
||||||
{http.MethodPost, statsPath, http.StatusMethodNotAllowed, methodNotAllowedMessage},
|
{http.MethodPost, statsPath, http.StatusMethodNotAllowed, methodNotAllowedMessage},
|
||||||
{http.MethodPost, "/ip/", http.StatusMethodNotAllowed, methodNotAllowedMessage},
|
{http.MethodPost, "/ip/", http.StatusMethodNotAllowed, methodNotAllowedMessage},
|
||||||
{
|
|
||||||
http.MethodDelete, statusJSONPath,
|
|
||||||
http.StatusMethodNotAllowed, methodNotAllowedMessage,
|
|
||||||
},
|
|
||||||
{
|
|
||||||
http.MethodOptions, healthCheckPath,
|
|
||||||
http.StatusMethodNotAllowed, methodNotAllowedMessage,
|
|
||||||
},
|
|
||||||
{http.MethodGet, "/api/v1/nothing", http.StatusNotFound, notFoundMessage},
|
{http.MethodGet, "/api/v1/nothing", http.StatusNotFound, notFoundMessage},
|
||||||
{http.MethodGet, "/ip/192.0.2.1/nothing", http.StatusNotFound, notFoundMessage},
|
{http.MethodGet, "/ip/192.0.2.1/nothing", http.StatusNotFound, notFoundMessage},
|
||||||
}
|
}
|
||||||
|
|||||||
+19
-84
@@ -1,14 +1,12 @@
|
|||||||
// Package server serves RouteWatch's web pages and JSON API:
|
// Package server serves RouteWatch's web pages and JSON API:
|
||||||
// - the home page, the status figures and the health check
|
// - the home page, the status figures and the health check
|
||||||
// - the pages that list the bogon announcements and the prefixes announced
|
|
||||||
// by more than one origin AS
|
|
||||||
// - the AS, prefix and prefix length pages, and the JSON answers behind them
|
// - the AS, prefix and prefix length pages, and the JSON answers behind them
|
||||||
// - the IP lookups
|
// - the IP lookups
|
||||||
// - the AS name search
|
// - the AS name search
|
||||||
//
|
//
|
||||||
// Every route it shows comes from the routing table in memory. AS names,
|
// Every route it shows comes from the routing table in memory. AS names,
|
||||||
// WHOIS details, the times a prefix and an AS were first seen, and a prefix's
|
// WHOIS details and the times a prefix and an AS were first seen come from
|
||||||
// origin changes come from the database.
|
// the database.
|
||||||
package server
|
package server
|
||||||
|
|
||||||
import (
|
import (
|
||||||
@@ -19,13 +17,11 @@ import (
|
|||||||
"net/http"
|
"net/http"
|
||||||
"net/netip"
|
"net/netip"
|
||||||
"strconv"
|
"strconv"
|
||||||
"sync/atomic"
|
|
||||||
"time"
|
"time"
|
||||||
|
|
||||||
|
"git.eeqj.de/sneak/routewatch/internal/blocklist"
|
||||||
"git.eeqj.de/sneak/routewatch/internal/config"
|
"git.eeqj.de/sneak/routewatch/internal/config"
|
||||||
"git.eeqj.de/sneak/routewatch/internal/database"
|
"git.eeqj.de/sneak/routewatch/internal/database"
|
||||||
"git.eeqj.de/sneak/routewatch/internal/geo"
|
|
||||||
"git.eeqj.de/sneak/routewatch/internal/iplist"
|
|
||||||
"git.eeqj.de/sneak/routewatch/internal/logger"
|
"git.eeqj.de/sneak/routewatch/internal/logger"
|
||||||
"git.eeqj.de/sneak/routewatch/internal/routetable"
|
"git.eeqj.de/sneak/routewatch/internal/routetable"
|
||||||
"git.eeqj.de/sneak/routewatch/internal/streamer"
|
"git.eeqj.de/sneak/routewatch/internal/streamer"
|
||||||
@@ -46,30 +42,20 @@ type ASNFetcher interface {
|
|||||||
GetStats() ASNFetcherStats
|
GetStats() ASNFetcherStats
|
||||||
}
|
}
|
||||||
|
|
||||||
// IPLists tells which public IP lists contain an address, from copies it keeps
|
// Blocklists tells which public blocklists list an address, from copies it
|
||||||
// in memory, as *iplist.IPLists does.
|
// keeps in memory, as *blocklist.Blocklists does.
|
||||||
type IPLists interface {
|
type Blocklists interface {
|
||||||
Lookup(addr netip.Addr) []iplist.Listing
|
Lookup(addr netip.Addr) []blocklist.Listing
|
||||||
ShortestInterval() time.Duration
|
ShortestInterval() time.Duration
|
||||||
}
|
}
|
||||||
|
|
||||||
// dnsResolver looks up the addresses of host names in DNS, as *net.Resolver
|
// dnsResolver looks up the addresses of host names and the names of addresses
|
||||||
// does.
|
// in DNS, as *net.Resolver does.
|
||||||
type dnsResolver interface {
|
type dnsResolver interface {
|
||||||
LookupHost(ctx context.Context, host string) ([]string, error)
|
LookupHost(ctx context.Context, host string) ([]string, error)
|
||||||
}
|
|
||||||
|
|
||||||
// PTRResolver looks up the PTR names of addresses, as *ptr.Cache does.
|
|
||||||
type PTRResolver interface {
|
|
||||||
LookupAddr(ctx context.Context, addr string) ([]string, error)
|
LookupAddr(ctx context.Context, addr string) ([]string, error)
|
||||||
}
|
}
|
||||||
|
|
||||||
// GeoLocator finds where addresses are, as *geo.Locator does. It returns one
|
|
||||||
// entry for each address, nil when no source answers.
|
|
||||||
type GeoLocator interface {
|
|
||||||
Lookup(ctx context.Context, addrs []netip.Addr) []*geo.Location
|
|
||||||
}
|
|
||||||
|
|
||||||
// Server provides HTTP endpoints for status monitoring
|
// Server provides HTTP endpoints for status monitoring
|
||||||
type Server struct {
|
type Server struct {
|
||||||
router *chi.Mux
|
router *chi.Mux
|
||||||
@@ -80,32 +66,15 @@ type Server struct {
|
|||||||
port int
|
port int
|
||||||
srv *http.Server
|
srv *http.Server
|
||||||
asnFetcher ASNFetcher
|
asnFetcher ASNFetcher
|
||||||
ipLists IPLists
|
blocklists Blocklists
|
||||||
|
|
||||||
// trustedProxies are the reverse proxies whose X-Forwarded-For header
|
// trustedProxies are the reverse proxies whose X-Forwarded-For header
|
||||||
// RealIPMiddleware believes
|
// RealIPMiddleware believes
|
||||||
trustedProxies []netip.Prefix
|
trustedProxies []netip.Prefix
|
||||||
|
|
||||||
// resolver looks up the IP lookup's host names: net.DefaultResolver, or a
|
// resolver makes the IP lookup's DNS lookups: net.DefaultResolver, or a
|
||||||
// fake in tests
|
// fake in tests
|
||||||
resolver dnsResolver
|
resolver dnsResolver
|
||||||
|
|
||||||
// ptrResolver looks up the IP lookup's PTR names. SetPTRResolver sets it.
|
|
||||||
ptrResolver PTRResolver
|
|
||||||
|
|
||||||
// geoLocator finds where the IP lookup's addresses are. SetGeoLocator
|
|
||||||
// sets it.
|
|
||||||
geoLocator GeoLocator
|
|
||||||
|
|
||||||
// cards is the copy of the home page's cards that the pages serve, nil
|
|
||||||
// until the server first works them out
|
|
||||||
cards atomic.Pointer[homeCards]
|
|
||||||
|
|
||||||
// stopCards ends the goroutine that works out the cards, and
|
|
||||||
// cardsStopped is closed once it has returned. Both are nil until Start
|
|
||||||
// starts the goroutine.
|
|
||||||
stopCards context.CancelFunc
|
|
||||||
cardsStopped chan struct{}
|
|
||||||
}
|
}
|
||||||
|
|
||||||
// New creates a new HTTP server that reads routes from table, the routing
|
// New creates a new HTTP server that reads routes from table, the routing
|
||||||
@@ -134,9 +103,7 @@ func New(
|
|||||||
|
|
||||||
// Start listens on the configured port and serves HTTP in the background.
|
// Start listens on the configured port and serves HTTP in the background.
|
||||||
// It returns an error if it cannot listen, for example because the port is
|
// It returns an error if it cannot listen, for example because the port is
|
||||||
// already in use. ctx bounds opening the port, not serving. Once listening,
|
// already in use. ctx bounds only opening the port, not serving.
|
||||||
// the server works out the home page's cards every minute until Stop: the end
|
|
||||||
// of ctx does not stop it.
|
|
||||||
func (s *Server) Start(ctx context.Context) error {
|
func (s *Server) Start(ctx context.Context) error {
|
||||||
const (
|
const (
|
||||||
network = "tcp"
|
network = "tcp"
|
||||||
@@ -173,22 +140,12 @@ func (s *Server) Start(ctx context.Context) error {
|
|||||||
}
|
}
|
||||||
}()
|
}()
|
||||||
|
|
||||||
ctx, s.stopCards = context.WithCancel(context.WithoutCancel(ctx))
|
|
||||||
s.cardsStopped = make(chan struct{})
|
|
||||||
|
|
||||||
go func() {
|
|
||||||
defer close(s.cardsStopped)
|
|
||||||
|
|
||||||
s.keepHomeCardsFresh(ctx, homeCardsRefreshInterval)
|
|
||||||
}()
|
|
||||||
|
|
||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
|
|
||||||
// Stop stops the goroutine that works out the home page's cards, and waits
|
// Stop stops the HTTP server. It waits, until ctx ends, for the requests in
|
||||||
// until it has returned or ctx ends. It then stops the HTTP server. It waits,
|
// progress to finish, and then closes the connections of those still running
|
||||||
// until ctx ends, for the requests in progress to finish, and then closes the
|
// and returns ctx's error.
|
||||||
// connections of those still running and returns ctx's error.
|
|
||||||
func (s *Server) Stop(ctx context.Context) error {
|
func (s *Server) Stop(ctx context.Context) error {
|
||||||
if s.srv == nil {
|
if s.srv == nil {
|
||||||
return nil
|
return nil
|
||||||
@@ -196,16 +153,6 @@ func (s *Server) Stop(ctx context.Context) error {
|
|||||||
|
|
||||||
s.logger.Info("Stopping HTTP server")
|
s.logger.Info("Stopping HTTP server")
|
||||||
|
|
||||||
// Start returns before it starts the goroutine when it cannot listen.
|
|
||||||
if s.stopCards != nil {
|
|
||||||
s.stopCards()
|
|
||||||
|
|
||||||
select {
|
|
||||||
case <-s.cardsStopped:
|
|
||||||
case <-ctx.Done():
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
err := s.srv.Shutdown(ctx)
|
err := s.srv.Shutdown(ctx)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return errors.Join(err, s.srv.Close())
|
return errors.Join(err, s.srv.Close())
|
||||||
@@ -214,26 +161,14 @@ func (s *Server) Stop(ctx context.Context) error {
|
|||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
|
|
||||||
// SetPTRResolver sets what looks up the IP lookup's PTR names. Call it before
|
|
||||||
// Start: the handlers read it without a lock.
|
|
||||||
func (s *Server) SetPTRResolver(resolver PTRResolver) {
|
|
||||||
s.ptrResolver = resolver
|
|
||||||
}
|
|
||||||
|
|
||||||
// SetGeoLocator sets what finds where the IP lookup's addresses are. Call it
|
|
||||||
// before Start: the handlers read it without a lock.
|
|
||||||
func (s *Server) SetGeoLocator(locator GeoLocator) {
|
|
||||||
s.geoLocator = locator
|
|
||||||
}
|
|
||||||
|
|
||||||
// SetASNFetcher sets the ASN WHOIS fetcher for on-demand lookups. Call it
|
// SetASNFetcher sets the ASN WHOIS fetcher for on-demand lookups. Call it
|
||||||
// before Start: the handlers read the fetcher without a lock.
|
// before Start: the handlers read the fetcher without a lock.
|
||||||
func (s *Server) SetASNFetcher(fetcher ASNFetcher) {
|
func (s *Server) SetASNFetcher(fetcher ASNFetcher) {
|
||||||
s.asnFetcher = fetcher
|
s.asnFetcher = fetcher
|
||||||
}
|
}
|
||||||
|
|
||||||
// SetIPLists sets the IP lists the IP lookups report. Call it before Start:
|
// SetBlocklists sets the blocklists the IP lookups report. Call it before
|
||||||
// the handlers read the IP lists without a lock.
|
// Start: the handlers read the blocklists without a lock.
|
||||||
func (s *Server) SetIPLists(ipLists IPLists) {
|
func (s *Server) SetBlocklists(blocklists Blocklists) {
|
||||||
s.ipLists = ipLists
|
s.blocklists = blocklists
|
||||||
}
|
}
|
||||||
@@ -22,10 +22,10 @@ func TestStopClosesTheConnectionOfARequestStillRunning(t *testing.T) {
|
|||||||
stopDeadline = 100 * time.Millisecond
|
stopDeadline = 100 * time.Millisecond
|
||||||
// waitTimeout is how long the test waits for the request to reach
|
// waitTimeout is how long the test waits for the request to reach
|
||||||
// the handler, and then for the client to see its connection closed.
|
// the handler, and then for the client to see its connection closed.
|
||||||
waitTimeout = 15 * time.Second
|
waitTimeout = 2 * time.Second
|
||||||
// readHeaderTimeout bounds the test server's read of the request
|
// readHeaderTimeout bounds the test server's read of the request
|
||||||
// header, as every http.Server must.
|
// header, as every http.Server must.
|
||||||
readHeaderTimeout = waitTimeout
|
readHeaderTimeout = time.Second
|
||||||
)
|
)
|
||||||
|
|
||||||
release := make(chan struct{})
|
release := make(chan struct{})
|
||||||
|
|||||||
@@ -12,6 +12,7 @@ import (
|
|||||||
"fmt"
|
"fmt"
|
||||||
"io"
|
"io"
|
||||||
"math"
|
"math"
|
||||||
|
"math/rand"
|
||||||
"net/http"
|
"net/http"
|
||||||
"sync"
|
"sync"
|
||||||
"sync/atomic"
|
"sync/atomic"
|
||||||
@@ -57,6 +58,10 @@ const (
|
|||||||
// Well above an extended BGP message (RFC 8654) in hex plus its prefixes
|
// Well above an extended BGP message (RFC 8654) in hex plus its prefixes
|
||||||
maxLineSize = bytesPerMB
|
maxLineSize = bytesPerMB
|
||||||
maxConcurrentHandlers = 800 // Maximum number of concurrent message handlers
|
maxConcurrentHandlers = 800 // Maximum number of concurrent message handlers
|
||||||
|
|
||||||
|
// Backpressure constants
|
||||||
|
backpressureThreshold = 0.5 // Start dropping at 50% queue utilization
|
||||||
|
backpressureSlope = 2.0 // Slope for linear drop probability increase
|
||||||
)
|
)
|
||||||
|
|
||||||
// messageTypeNotification is the type of a BGP NOTIFICATION message, which ends
|
// messageTypeNotification is the type of a BGP NOTIFICATION message, which ends
|
||||||
@@ -88,8 +93,9 @@ type MessageHandler interface {
|
|||||||
|
|
||||||
// QueueCapacity returns the desired queue capacity for this handler.
|
// QueueCapacity returns the desired queue capacity for this handler.
|
||||||
// Handlers that process quickly can have larger queues to buffer bursts.
|
// Handlers that process quickly can have larger queues to buffer bursts.
|
||||||
// While the queue is full, the streamer drops the handler's messages,
|
// When the queue fills up, messages will be dropped according to the
|
||||||
// except STATE messages, which are held until there is room.
|
// backpressure algorithm, except STATE messages, which are held until
|
||||||
|
// there is room.
|
||||||
QueueCapacity() int
|
QueueCapacity() int
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -120,8 +126,8 @@ type handlerInfo struct {
|
|||||||
|
|
||||||
// Streamer manages a connection to the RIPE RIS Live streaming API for receiving
|
// Streamer manages a connection to the RIPE RIS Live streaming API for receiving
|
||||||
// real-time BGP UPDATE messages. It handles automatic reconnection with exponential
|
// real-time BGP UPDATE messages. It handles automatic reconnection with exponential
|
||||||
// backoff and dispatches messages to registered handlers via per-handler queues.
|
// backoff, dispatches messages to registered handlers via per-handler queues, and
|
||||||
// Its read loop never waits for room on a queue (see queueMessage).
|
// implements backpressure to prevent queue overflow during high traffic periods.
|
||||||
type Streamer struct {
|
type Streamer struct {
|
||||||
logger *logger.Logger
|
logger *logger.Logger
|
||||||
client *http.Client
|
client *http.Client
|
||||||
@@ -138,6 +144,7 @@ type Streamer struct {
|
|||||||
workers sync.WaitGroup // the handler workers; Stop waits for them
|
workers sync.WaitGroup // the handler workers; Stop waits for them
|
||||||
metrics *metrics.Tracker
|
metrics *metrics.Tracker
|
||||||
totalDropped atomic.Uint64 // Total dropped messages across all handlers
|
totalDropped atomic.Uint64 // Total dropped messages across all handlers
|
||||||
|
random *rand.Rand // Random number generator for backpressure drops
|
||||||
bgpPeers map[string]bool // Track active BGP peers by peer IP
|
bgpPeers map[string]bool // Track active BGP peers by peer IP
|
||||||
bgpPeersMu sync.RWMutex // Protects bgpPeers map
|
bgpPeersMu sync.RWMutex // Protects bgpPeers map
|
||||||
}
|
}
|
||||||
@@ -161,6 +168,8 @@ func New(logger *logger.Logger, metrics *metrics.Tracker) *Streamer {
|
|||||||
},
|
},
|
||||||
handlers: make([]*handlerInfo, 0),
|
handlers: make([]*handlerInfo, 0),
|
||||||
metrics: metrics,
|
metrics: metrics,
|
||||||
|
//nolint:gosec // Non-cryptographic randomness is fine for backpressure
|
||||||
|
random: rand.New(rand.NewSource(time.Now().UnixNano())),
|
||||||
bgpPeers: make(map[string]bool),
|
bgpPeers: make(map[string]bool),
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -342,10 +351,6 @@ type HandlerStats struct {
|
|||||||
AvgProcessTime time.Duration
|
AvgProcessTime time.Duration
|
||||||
MinProcessTime time.Duration
|
MinProcessTime time.Duration
|
||||||
MaxProcessTime time.Duration
|
MaxProcessTime time.Duration
|
||||||
// HeldBackCount is how many messages the handler holds back to apply
|
|
||||||
// later: what its HeldBackCount method returns, or 0 for a handler
|
|
||||||
// without one.
|
|
||||||
HeldBackCount int
|
|
||||||
}
|
}
|
||||||
|
|
||||||
// GetHandlerStats returns a snapshot of performance statistics for all registered
|
// GetHandlerStats returns a snapshot of performance statistics for all registered
|
||||||
@@ -387,11 +392,6 @@ func (s *Streamer) GetHandlerStats() []HandlerStats {
|
|||||||
|
|
||||||
info.metrics.mu.Unlock()
|
info.metrics.mu.Unlock()
|
||||||
|
|
||||||
holder, holdsBack := info.handler.(interface{ HeldBackCount() int })
|
|
||||||
if holdsBack {
|
|
||||||
hs.HeldBackCount = holder.HeldBackCount()
|
|
||||||
}
|
|
||||||
|
|
||||||
stats = append(stats, hs)
|
stats = append(stats, hs)
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -399,7 +399,7 @@ func (s *Streamer) GetHandlerStats() []HandlerStats {
|
|||||||
}
|
}
|
||||||
|
|
||||||
// GetDroppedMessages returns the total number of messages dropped across all
|
// GetDroppedMessages returns the total number of messages dropped across all
|
||||||
// handlers because their queue was full. This counter is monotonically
|
// handlers due to queue overflow or backpressure. This counter is monotonically
|
||||||
// increasing and is safe to call concurrently.
|
// increasing and is safe to call concurrently.
|
||||||
func (s *Streamer) GetDroppedMessages() uint64 {
|
func (s *Streamer) GetDroppedMessages() uint64 {
|
||||||
return s.totalDropped.Load()
|
return s.totalDropped.Load()
|
||||||
@@ -882,13 +882,13 @@ func (s *Streamer) dispatchMessage(
|
|||||||
// msg instead. It never waits for room, so a full queue does not hold up the
|
// msg instead. It never waits for room, so a full queue does not hold up the
|
||||||
// read loop.
|
// read loop.
|
||||||
//
|
//
|
||||||
// A message that finds the queue full is dropped, except a STATE message. A
|
// A message is dropped while the queue is filling up (shouldDropForBackpressure)
|
||||||
// STATE message whose state is down is the only notice that all of a peer's
|
// or full, except a STATE message: one whose state is down is the only notice
|
||||||
// routes are gone. A STATE message that finds the queue full is held in
|
// that all of a peer's routes are gone. A STATE message skips the random drop,
|
||||||
// info.heldState instead. Held messages go into the queue, oldest first, when a
|
// and one that finds the queue full is held in info.heldState. Held messages
|
||||||
// later message for the handler finds room, or when Stop is called. While any
|
// go into the queue, oldest first, before anything else for the handler, as
|
||||||
// message is held, the handler's other messages are dropped, so none of them
|
// soon as there is room, or when Stop is called; while any is still held, the
|
||||||
// overtakes a held STATE message.
|
// handler's other messages are dropped, so none of them overtakes a held down.
|
||||||
//
|
//
|
||||||
// The held list stays small: a STATE message is a few hundred bytes and the
|
// The held list stays small: a STATE message is a few hundred bytes and the
|
||||||
// feed sends about ten a second, so a queue full for a minute holds about 600.
|
// feed sends about ten a second, so a queue full for a minute holds about 600.
|
||||||
@@ -907,7 +907,7 @@ func (s *Streamer) queueMessage(
|
|||||||
return true
|
return true
|
||||||
}
|
}
|
||||||
|
|
||||||
if len(info.heldState) > 0 {
|
if len(info.heldState) > 0 || s.shouldDropForBackpressure(info) {
|
||||||
return false
|
return false
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -923,3 +923,26 @@ func trySend(queue chan *ristypes.RISMessage, msg *ristypes.RISMessage) bool {
|
|||||||
return false
|
return false
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// shouldDropForBackpressure determines if a message should be dropped based on
|
||||||
|
// queue utilization
|
||||||
|
func (s *Streamer) shouldDropForBackpressure(info *handlerInfo) bool {
|
||||||
|
// Calculate queue utilization
|
||||||
|
queueLen := len(info.queue)
|
||||||
|
queueCap := cap(info.queue)
|
||||||
|
utilization := float64(queueLen) / float64(queueCap)
|
||||||
|
|
||||||
|
// No drops below threshold
|
||||||
|
if utilization < backpressureThreshold {
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
|
||||||
|
// Calculate drop probability (0.0 at threshold, 1.0 at 100% full)
|
||||||
|
dropProbability := (utilization - backpressureThreshold) * backpressureSlope
|
||||||
|
if dropProbability > 1.0 {
|
||||||
|
dropProbability = 1.0
|
||||||
|
}
|
||||||
|
|
||||||
|
// Random drop based on probability
|
||||||
|
return s.random.Float64() < dropProbability
|
||||||
|
}
|
||||||
@@ -5,7 +5,6 @@ import (
|
|||||||
"context"
|
"context"
|
||||||
"encoding/json"
|
"encoding/json"
|
||||||
"errors"
|
"errors"
|
||||||
"fmt"
|
|
||||||
"io"
|
"io"
|
||||||
"log/slog"
|
"log/slog"
|
||||||
"net/http"
|
"net/http"
|
||||||
@@ -14,7 +13,6 @@ import (
|
|||||||
"strings"
|
"strings"
|
||||||
"sync/atomic"
|
"sync/atomic"
|
||||||
"testing"
|
"testing"
|
||||||
"testing/synctest"
|
|
||||||
"time"
|
"time"
|
||||||
|
|
||||||
"git.eeqj.de/sneak/routewatch/internal/logger"
|
"git.eeqj.de/sneak/routewatch/internal/logger"
|
||||||
@@ -260,65 +258,6 @@ func TestHeldStateMessagesKeepTheirPlaceInTheQueue(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// burstQueueCapacity is the size of a burstHandler's queue.
|
|
||||||
const burstQueueCapacity = 1000
|
|
||||||
|
|
||||||
// burstHandler wants UPDATE messages and has a queue of burstQueueCapacity.
|
|
||||||
// Tests read what it is handed from its queue, so HandleMessage does nothing.
|
|
||||||
type burstHandler struct{}
|
|
||||||
|
|
||||||
func (burstHandler) WantsMessage(messageType string) bool {
|
|
||||||
return messageType == ristypes.MessageTypeUpdate
|
|
||||||
}
|
|
||||||
|
|
||||||
func (burstHandler) QueueCapacity() int { return burstQueueCapacity }
|
|
||||||
|
|
||||||
func (burstHandler) HandleMessage(*ristypes.RISMessage) {}
|
|
||||||
|
|
||||||
// TestUpdatesDroppedOnlyWhenTheQueueIsFull has the server send one UPDATE more
|
|
||||||
// than the handler's queue holds, each with its own timestamp. Nothing takes
|
|
||||||
// messages off the queue meanwhile. The streamer must queue the updates in
|
|
||||||
// feed order until the queue is full, and drop only the last one.
|
|
||||||
func TestUpdatesDroppedOnlyWhenTheQueueIsFull(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
const (
|
|
||||||
firstTimestamp = 1700000000
|
|
||||||
lineFormat = `{"type":"ris_message","data":{"type":"UPDATE",` +
|
|
||||||
`"peer":"192.0.2.1","peer_asn":"64496","timestamp":%d}}` + "\n"
|
|
||||||
)
|
|
||||||
|
|
||||||
var lines strings.Builder
|
|
||||||
|
|
||||||
for i := range burstQueueCapacity + 1 {
|
|
||||||
fmt.Fprintf(&lines, lineFormat, firstTimestamp+i)
|
|
||||||
}
|
|
||||||
|
|
||||||
s := New(logger.New(), metrics.New())
|
|
||||||
s.RegisterHandler(burstHandler{})
|
|
||||||
|
|
||||||
streamLines(t, s, lines.String())
|
|
||||||
|
|
||||||
dropped := s.GetHandlerStats()[0].DroppedCount
|
|
||||||
if dropped != 1 {
|
|
||||||
t.Errorf("%d updates dropped, want only the one that found the queue "+
|
|
||||||
"full", dropped)
|
|
||||||
}
|
|
||||||
|
|
||||||
queue := s.handlers[0].queue
|
|
||||||
if len(queue) != burstQueueCapacity {
|
|
||||||
t.Fatalf("%d updates queued, want %d", len(queue), burstQueueCapacity)
|
|
||||||
}
|
|
||||||
|
|
||||||
for i := range burstQueueCapacity {
|
|
||||||
msg := <-queue
|
|
||||||
if msg.Timestamp != float64(firstTimestamp+i) {
|
|
||||||
t.Fatalf("update %d on the queue has timestamp %.0f, want %d",
|
|
||||||
i, msg.Timestamp, firstTimestamp+i)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// takeAll takes the messages off queue, as a handler's worker would, and sends
|
// takeAll takes the messages off queue, as a handler's worker would, and sends
|
||||||
// them on the returned channel once queue is closed.
|
// them on the returned channel once queue is closed.
|
||||||
func takeAll(queue chan *ristypes.RISMessage) chan []*ristypes.RISMessage {
|
func takeAll(queue chan *ristypes.RISMessage) chan []*ristypes.RISMessage {
|
||||||
@@ -399,7 +338,7 @@ func TestStopQueuesTheHeldStateMessages(t *testing.T) {
|
|||||||
const (
|
const (
|
||||||
held = 2
|
held = 2
|
||||||
lines = held + 1
|
lines = held + 1
|
||||||
maxWait = 15 * time.Second
|
maxWait = 5 * time.Second
|
||||||
)
|
)
|
||||||
|
|
||||||
srv := httptest.NewServer(http.HandlerFunc(
|
srv := httptest.NewServer(http.HandlerFunc(
|
||||||
@@ -479,15 +418,15 @@ func TestStopQueuesTheHeldStateMessages(t *testing.T) {
|
|||||||
// TestStopReturnsByItsDeadline stops a streamer whose handler has a full queue
|
// TestStopReturnsByItsDeadline stops a streamer whose handler has a full queue
|
||||||
// and a STATE message held for it, and a worker that stays busy until the test
|
// and a STATE message held for it, and a worker that stays busy until the test
|
||||||
// ends, as one does whose handler waits for a database write. Stop must return
|
// ends, as one does whose handler waits for a database write. Stop must return
|
||||||
// within its deadline plus a margin instead of waiting for room on the queue
|
// within its deadline plus a small margin instead of waiting for room on the
|
||||||
// and for the worker, and log as not handed over the held message and every
|
// queue and for the worker, and log as not handed over the held message and
|
||||||
// message on the queue.
|
// every message on the queue.
|
||||||
func TestStopReturnsByItsDeadline(t *testing.T) {
|
func TestStopReturnsByItsDeadline(t *testing.T) {
|
||||||
t.Parallel()
|
t.Parallel()
|
||||||
|
|
||||||
const (
|
const (
|
||||||
deadline = 100 * time.Millisecond
|
deadline = 100 * time.Millisecond
|
||||||
margin = 15 * time.Second
|
margin = 2 * time.Second
|
||||||
)
|
)
|
||||||
|
|
||||||
var logs bytes.Buffer
|
var logs bytes.Buffer
|
||||||
@@ -578,7 +517,7 @@ func TestStopBeforeMessageReachesQueues(t *testing.T) {
|
|||||||
func TestReconnectWhenStreamEnds(t *testing.T) {
|
func TestReconnectWhenStreamEnds(t *testing.T) {
|
||||||
t.Parallel()
|
t.Parallel()
|
||||||
|
|
||||||
endResponse := pipeTransport{write: func(context.Context, io.Writer) {}}
|
endResponse := func(http.ResponseWriter, *http.Request) {}
|
||||||
checkReconnect(t, endResponse, streamIdleTimeout)
|
checkReconnect(t, endResponse, streamIdleTimeout)
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -590,8 +529,9 @@ func TestReconnectWhenStreamGoesIdle(t *testing.T) {
|
|||||||
|
|
||||||
const idleTimeout = 50 * time.Millisecond
|
const idleTimeout = 50 * time.Millisecond
|
||||||
|
|
||||||
holdResponse := pipeTransport{
|
holdResponse := func(w http.ResponseWriter, r *http.Request) {
|
||||||
write: func(ctx context.Context, _ io.Writer) { <-ctx.Done() },
|
flush(w)
|
||||||
|
<-r.Context().Done()
|
||||||
}
|
}
|
||||||
checkReconnect(t, holdResponse, idleTimeout)
|
checkReconnect(t, holdResponse, idleTimeout)
|
||||||
}
|
}
|
||||||
@@ -604,51 +544,16 @@ func TestReconnectWhenServerNeverAnswers(t *testing.T) {
|
|||||||
|
|
||||||
const idleTimeout = 50 * time.Millisecond
|
const idleTimeout = 50 * time.Millisecond
|
||||||
|
|
||||||
neverAnswer := roundTripFunc(
|
neverAnswer := func(_ http.ResponseWriter, r *http.Request) {
|
||||||
func(req *http.Request) (*http.Response, error) {
|
<-r.Context().Done()
|
||||||
<-req.Context().Done()
|
}
|
||||||
|
|
||||||
return nil, req.Context().Err()
|
|
||||||
},
|
|
||||||
)
|
|
||||||
checkReconnect(t, neverAnswer, idleTimeout)
|
checkReconnect(t, neverAnswer, idleTimeout)
|
||||||
}
|
}
|
||||||
|
|
||||||
// pipeTransport answers every request with a response whose body write fills.
|
// TestStreamNotDroppedWhileLinesArrive has the server send lines well within
|
||||||
// The body fails when the request's context ends, as a connection's does.
|
// the idle timeout for several times the timeout, then end the response. Each
|
||||||
// write gets that context, so that it can end when the request does.
|
// line must start the idle timeout over, so the stream lasts until the server
|
||||||
type pipeTransport struct {
|
// ends it instead of being dropped as idle.
|
||||||
write func(context.Context, io.Writer)
|
|
||||||
}
|
|
||||||
|
|
||||||
func (p pipeTransport) RoundTrip(req *http.Request) (*http.Response, error) {
|
|
||||||
body, w := io.Pipe()
|
|
||||||
|
|
||||||
context.AfterFunc(req.Context(), func() {
|
|
||||||
_ = w.CloseWithError(req.Context().Err())
|
|
||||||
})
|
|
||||||
|
|
||||||
go func(ctx context.Context) {
|
|
||||||
p.write(ctx, w)
|
|
||||||
|
|
||||||
_ = w.Close()
|
|
||||||
}(req.Context())
|
|
||||||
|
|
||||||
return &http.Response{StatusCode: http.StatusOK, Body: body}, nil
|
|
||||||
}
|
|
||||||
|
|
||||||
// roundTripFunc makes a function an http.RoundTripper.
|
|
||||||
type roundTripFunc func(*http.Request) (*http.Response, error)
|
|
||||||
|
|
||||||
func (f roundTripFunc) RoundTrip(req *http.Request) (*http.Response, error) {
|
|
||||||
return f(req)
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestStreamNotDroppedWhileLinesArrive has a response send lines well within
|
|
||||||
// the idle timeout for several times the timeout, then end. Each line must
|
|
||||||
// start the idle timeout over, so the stream lasts until the response ends
|
|
||||||
// instead of being dropped as idle. It runs in a synctest bubble, whose clock
|
|
||||||
// is fake, so a stall on the host does not use up the timeout.
|
|
||||||
func TestStreamNotDroppedWhileLinesArrive(t *testing.T) {
|
func TestStreamNotDroppedWhileLinesArrive(t *testing.T) {
|
||||||
t.Parallel()
|
t.Parallel()
|
||||||
|
|
||||||
@@ -658,35 +563,26 @@ func TestStreamNotDroppedWhileLinesArrive(t *testing.T) {
|
|||||||
sendFor = 5 * idleTimeout
|
sendFor = 5 * idleTimeout
|
||||||
)
|
)
|
||||||
|
|
||||||
// The streamer is made outside the bubble. Its metrics start a goroutine
|
srv := httptest.NewServer(http.HandlerFunc(
|
||||||
// that never returns, and a bubble waits for each of its goroutines to
|
func(w http.ResponseWriter, _ *http.Request) {
|
||||||
// return.
|
|
||||||
s := New(logger.New(), metrics.New())
|
|
||||||
s.idleTimeout = idleTimeout
|
|
||||||
s.client = &http.Client{Transport: pipeTransport{
|
|
||||||
write: func(ctx context.Context, w io.Writer) {
|
|
||||||
deadline := time.Now().Add(sendFor)
|
deadline := time.Now().Add(sendFor)
|
||||||
for time.Now().Before(deadline) {
|
for time.Now().Before(deadline) {
|
||||||
_, _ = io.WriteString(w, updateLine)
|
_, _ = io.WriteString(w, updateLine)
|
||||||
|
|
||||||
// The wait ends with the stream. A bubble's clock stops when
|
flush(w)
|
||||||
// the test function returns, and a goroutine then still
|
time.Sleep(lineInterval)
|
||||||
// asleep panics the test binary.
|
|
||||||
select {
|
|
||||||
case <-ctx.Done():
|
|
||||||
return
|
|
||||||
case <-time.After(lineInterval):
|
|
||||||
}
|
|
||||||
}
|
}
|
||||||
},
|
}))
|
||||||
}}
|
defer srv.Close()
|
||||||
|
|
||||||
synctest.Test(t, func(t *testing.T) {
|
s := New(logger.New(), metrics.New())
|
||||||
err := s.stream(context.Background())
|
s.url = srv.URL
|
||||||
if !errors.Is(err, errStreamEnded) {
|
s.idleTimeout = idleTimeout
|
||||||
t.Fatalf("stream returned %v, want %v", err, errStreamEnded)
|
|
||||||
}
|
err := s.stream(context.Background())
|
||||||
})
|
if !errors.Is(err, errStreamEnded) {
|
||||||
|
t.Fatalf("stream returned %v, want %v", err, errStreamEnded)
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// TestStreamReadsLongLine has the server send one UPDATE line of about
|
// TestStreamReadsLongLine has the server send one UPDATE line of about
|
||||||
@@ -750,7 +646,7 @@ func TestHandlerStatsReadWhileMessagesAreDropped(t *testing.T) {
|
|||||||
const (
|
const (
|
||||||
lines = 5
|
lines = 5
|
||||||
wantDropped = lines - 2
|
wantDropped = lines - 2
|
||||||
maxWait = 15 * time.Second
|
maxWait = 5 * time.Second
|
||||||
)
|
)
|
||||||
|
|
||||||
srv := httptest.NewServer(http.HandlerFunc(
|
srv := httptest.NewServer(http.HandlerFunc(
|
||||||
@@ -864,70 +760,61 @@ func streamLines(t *testing.T, s *Streamer, lines string) {
|
|||||||
// line and then stays open. The streamer must come back for a second
|
// line and then stays open. The streamer must come back for a second
|
||||||
// response, must not be marked connected when it does, and must handle the
|
// response, must not be marked connected when it does, and must handle the
|
||||||
// UPDATE in it.
|
// UPDATE in it.
|
||||||
//
|
|
||||||
// It runs in a synctest bubble, whose clock is fake, so a stall on the host
|
|
||||||
// does not use up the idle timeout. The fake server gets the requests in the
|
|
||||||
// order the streamer makes them, because no real connection carries them.
|
|
||||||
func checkReconnect(
|
func checkReconnect(
|
||||||
t *testing.T, firstResponse http.RoundTripper, idleTimeout time.Duration,
|
t *testing.T, firstResponse http.HandlerFunc, idleTimeout time.Duration,
|
||||||
) {
|
) {
|
||||||
t.Helper()
|
t.Helper()
|
||||||
|
|
||||||
const maxWait = 15 * time.Second
|
const maxWait = 5 * time.Second
|
||||||
|
|
||||||
// The tracker is made outside the bubble. It starts a goroutine that never
|
|
||||||
// returns, and a bubble waits for each of its goroutines to return.
|
|
||||||
tracker := metrics.New()
|
tracker := metrics.New()
|
||||||
|
connectedAtReconnect := make(chan bool, 1)
|
||||||
|
|
||||||
synctest.Test(t, func(t *testing.T) {
|
var requests atomic.Int32
|
||||||
connectedAtReconnect := make(chan bool, 1)
|
|
||||||
laterResponse := pipeTransport{
|
|
||||||
write: func(ctx context.Context, w io.Writer) {
|
|
||||||
_, _ = io.WriteString(w, updateLine)
|
|
||||||
|
|
||||||
<-ctx.Done()
|
srv := httptest.NewServer(http.HandlerFunc(
|
||||||
},
|
func(w http.ResponseWriter, r *http.Request) {
|
||||||
}
|
if requests.Add(1) == 1 {
|
||||||
|
firstResponse(w, r)
|
||||||
|
|
||||||
var requests atomic.Int32
|
return
|
||||||
|
}
|
||||||
|
|
||||||
s := New(logger.New(), tracker)
|
select {
|
||||||
s.client = &http.Client{Transport: roundTripFunc(
|
case connectedAtReconnect <- tracker.IsConnected():
|
||||||
func(req *http.Request) (*http.Response, error) {
|
default:
|
||||||
if requests.Add(1) == 1 {
|
}
|
||||||
return firstResponse.RoundTrip(req)
|
|
||||||
}
|
|
||||||
|
|
||||||
select {
|
_, _ = io.WriteString(w, updateLine)
|
||||||
case connectedAtReconnect <- tracker.IsConnected():
|
|
||||||
default:
|
|
||||||
}
|
|
||||||
|
|
||||||
return laterResponse.RoundTrip(req)
|
flush(w)
|
||||||
},
|
<-r.Context().Done()
|
||||||
)}
|
}))
|
||||||
s.idleTimeout = idleTimeout
|
defer srv.Close()
|
||||||
s.minBackoff = time.Millisecond
|
|
||||||
handler := updateHandler{handled: make(chan struct{}, 1)}
|
|
||||||
s.RegisterHandler(handler)
|
|
||||||
|
|
||||||
err := s.Start(t.Context())
|
s := New(logger.New(), tracker)
|
||||||
if err != nil {
|
s.url = srv.URL
|
||||||
t.Fatalf("Start returned error: %v", err)
|
s.idleTimeout = idleTimeout
|
||||||
}
|
s.minBackoff = time.Millisecond
|
||||||
defer s.Stop(t.Context())
|
handler := updateHandler{handled: make(chan struct{}, 1)}
|
||||||
|
s.RegisterHandler(handler)
|
||||||
|
|
||||||
select {
|
err := s.Start(t.Context())
|
||||||
case <-handler.handled:
|
if err != nil {
|
||||||
case <-time.After(maxWait):
|
t.Fatalf("Start returned error: %v", err)
|
||||||
t.Fatalf("no message from a second response handled within %s", maxWait)
|
}
|
||||||
}
|
defer s.Stop(t.Context())
|
||||||
|
|
||||||
if <-connectedAtReconnect {
|
select {
|
||||||
t.Error("streamer was still marked connected when it came back " +
|
case <-handler.handled:
|
||||||
"for a second response")
|
case <-time.After(maxWait):
|
||||||
}
|
t.Fatalf("no message from a second response handled within %s", maxWait)
|
||||||
})
|
}
|
||||||
|
|
||||||
|
if <-connectedAtReconnect {
|
||||||
|
t.Error("streamer was still marked connected when it came back " +
|
||||||
|
"for a second response")
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// settledGoroutineCount lets transient goroutines finish, then reports the
|
// settledGoroutineCount lets transient goroutines finish, then reports the
|
||||||
@@ -949,13 +836,9 @@ func settledGoroutineCount() int {
|
|||||||
return prev
|
return prev
|
||||||
}
|
}
|
||||||
|
|
||||||
// waitForGoroutines waits up to 15 seconds for the goroutine count to drop to
|
// waitForGoroutines waits until the goroutine count drops to target or below.
|
||||||
// target or below.
|
|
||||||
func waitForGoroutines(target int) bool {
|
func waitForGoroutines(target int) bool {
|
||||||
const maxWait = 15 * time.Second
|
for range 100 {
|
||||||
|
|
||||||
deadline := time.Now().Add(maxWait)
|
|
||||||
for time.Now().Before(deadline) {
|
|
||||||
if runtime.NumGoroutine() <= target {
|
if runtime.NumGoroutine() <= target {
|
||||||
return true
|
return true
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -1,170 +0,0 @@
|
|||||||
<!DOCTYPE html>
|
|
||||||
<html lang="en">
|
|
||||||
<head>
|
|
||||||
<meta charset="UTF-8">
|
|
||||||
<meta name="viewport" content="width=device-width, initial-scale=1.0">
|
|
||||||
<title>Bogon Announcements - RouteWatch</title>
|
|
||||||
<style nonce="{{.CSPNonce}}">
|
|
||||||
* {
|
|
||||||
box-sizing: border-box;
|
|
||||||
}
|
|
||||||
body {
|
|
||||||
font-family: -apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, sans-serif;
|
|
||||||
margin: 0;
|
|
||||||
padding: 0;
|
|
||||||
background: #f5f5f5;
|
|
||||||
}
|
|
||||||
|
|
||||||
/* Navbar styles */
|
|
||||||
.navbar {
|
|
||||||
background: #2c3e50;
|
|
||||||
padding: 0 20px;
|
|
||||||
display: flex;
|
|
||||||
align-items: center;
|
|
||||||
justify-content: space-between;
|
|
||||||
height: 56px;
|
|
||||||
box-shadow: 0 2px 4px rgba(0,0,0,0.1);
|
|
||||||
}
|
|
||||||
.navbar-brand {
|
|
||||||
display: flex;
|
|
||||||
align-items: center;
|
|
||||||
gap: 8px;
|
|
||||||
font-size: 18px;
|
|
||||||
font-weight: 600;
|
|
||||||
}
|
|
||||||
.navbar-brand a {
|
|
||||||
color: white;
|
|
||||||
text-decoration: none;
|
|
||||||
}
|
|
||||||
.navbar-brand a:hover {
|
|
||||||
color: #ecf0f1;
|
|
||||||
}
|
|
||||||
.navbar-brand .by {
|
|
||||||
font-weight: normal;
|
|
||||||
color: #95a5a6;
|
|
||||||
font-size: 14px;
|
|
||||||
}
|
|
||||||
.navbar-brand .author {
|
|
||||||
color: #3498db;
|
|
||||||
font-weight: normal;
|
|
||||||
}
|
|
||||||
.navbar-brand .author:hover {
|
|
||||||
text-decoration: underline;
|
|
||||||
}
|
|
||||||
.navbar-links {
|
|
||||||
display: flex;
|
|
||||||
gap: 20px;
|
|
||||||
}
|
|
||||||
.navbar-links a {
|
|
||||||
color: #ecf0f1;
|
|
||||||
text-decoration: none;
|
|
||||||
font-size: 14px;
|
|
||||||
padding: 8px 12px;
|
|
||||||
border-radius: 4px;
|
|
||||||
transition: background-color 0.2s;
|
|
||||||
}
|
|
||||||
.navbar-links a:hover {
|
|
||||||
background: rgba(255,255,255,0.1);
|
|
||||||
}
|
|
||||||
|
|
||||||
/* Main content */
|
|
||||||
.main-content {
|
|
||||||
max-width: 1200px;
|
|
||||||
margin: 0 auto;
|
|
||||||
padding: 20px;
|
|
||||||
}
|
|
||||||
h1 {
|
|
||||||
color: #333;
|
|
||||||
margin-bottom: 10px;
|
|
||||||
}
|
|
||||||
.subtitle {
|
|
||||||
color: #666;
|
|
||||||
margin-bottom: 30px;
|
|
||||||
}
|
|
||||||
table {
|
|
||||||
width: 100%;
|
|
||||||
border-collapse: collapse;
|
|
||||||
background: white;
|
|
||||||
box-shadow: 0 2px 4px rgba(0,0,0,0.1);
|
|
||||||
border-radius: 8px;
|
|
||||||
overflow: hidden;
|
|
||||||
}
|
|
||||||
th {
|
|
||||||
background: #f8f9fa;
|
|
||||||
padding: 12px;
|
|
||||||
text-align: left;
|
|
||||||
font-weight: 600;
|
|
||||||
color: #333;
|
|
||||||
border-bottom: 2px solid #dee2e6;
|
|
||||||
}
|
|
||||||
td {
|
|
||||||
padding: 12px;
|
|
||||||
border-bottom: 1px solid #eee;
|
|
||||||
}
|
|
||||||
tr:last-child td {
|
|
||||||
border-bottom: none;
|
|
||||||
}
|
|
||||||
tr:hover {
|
|
||||||
background: #f8f9fa;
|
|
||||||
}
|
|
||||||
a {
|
|
||||||
color: #0066cc;
|
|
||||||
text-decoration: none;
|
|
||||||
}
|
|
||||||
a:hover {
|
|
||||||
text-decoration: underline;
|
|
||||||
}
|
|
||||||
.prefix-link {
|
|
||||||
font-family: 'SF Mono', Monaco, 'Cascadia Mono', 'Roboto Mono', Consolas, 'Courier New', monospace;
|
|
||||||
}
|
|
||||||
</style>
|
|
||||||
</head>
|
|
||||||
<body>
|
|
||||||
<nav class="navbar">
|
|
||||||
<div class="navbar-brand">
|
|
||||||
<a href="/">routewatch</a>
|
|
||||||
<span class="by">by</span>
|
|
||||||
<a href="https://sneak.berlin" class="author">@sneak</a>
|
|
||||||
</div>
|
|
||||||
<div class="navbar-links">
|
|
||||||
<a href="/">Home</a>
|
|
||||||
<a href="/status">Status</a>
|
|
||||||
</div>
|
|
||||||
</nav>
|
|
||||||
|
|
||||||
<main class="main-content">
|
|
||||||
<h1>Bogon Announcements</h1>
|
|
||||||
{{ with .Cards }}
|
|
||||||
<p class="subtitle">
|
|
||||||
Bogon announcements: {{ .BogonCount }}.
|
|
||||||
A bogon announcement is a route that announces a reserved prefix, or that comes from a reserved AS number.
|
|
||||||
{{ if lt (len .Bogons) .BogonCount }}The first {{ len .Bogons }} are listed, in prefix order.{{ end }}
|
|
||||||
The list is worked out once a minute.
|
|
||||||
</p>
|
|
||||||
|
|
||||||
<table>
|
|
||||||
<thead>
|
|
||||||
<tr>
|
|
||||||
<th>Prefix</th>
|
|
||||||
<th>Origin AS</th>
|
|
||||||
<th>Peers</th>
|
|
||||||
<th>Why it is a bogon</th>
|
|
||||||
</tr>
|
|
||||||
</thead>
|
|
||||||
<tbody>
|
|
||||||
{{ range .Bogons }}
|
|
||||||
<tr>
|
|
||||||
<td><a href="{{ prefixURL .Prefix.String }}" class="prefix-link">{{ .Prefix }}</a></td>
|
|
||||||
<td>{{ if .OriginASN }}<a href="/as/{{ .OriginASN }}">AS{{ .OriginASN }}</a>{{ else }}No known origin{{ end }}</td>
|
|
||||||
<td>{{ .Peers }}</td>
|
|
||||||
<td>{{ range $i, $reason := .Reasons }}{{ if $i }}; {{ end }}{{ $reason }}{{ end }}</td>
|
|
||||||
</tr>
|
|
||||||
{{ end }}
|
|
||||||
</tbody>
|
|
||||||
</table>
|
|
||||||
{{ else }}
|
|
||||||
<p class="subtitle">The list is being worked out. Reload the page in a minute.</p>
|
|
||||||
{{ end }}
|
|
||||||
</main>
|
|
||||||
</body>
|
|
||||||
</html>
|
|
||||||
+11
-156
@@ -276,52 +276,6 @@
|
|||||||
margin-top: 12px;
|
margin-top: 12px;
|
||||||
}
|
}
|
||||||
|
|
||||||
/* Cards with figures of the routing table, worked out once a minute */
|
|
||||||
#table-cards p,
|
|
||||||
#well-known p {
|
|
||||||
margin: 0 0 8px 0;
|
|
||||||
font-size: 14px;
|
|
||||||
}
|
|
||||||
#table-cards a,
|
|
||||||
#well-known a {
|
|
||||||
color: #3498db;
|
|
||||||
text-decoration: none;
|
|
||||||
}
|
|
||||||
#table-cards a:hover,
|
|
||||||
#well-known a:hover {
|
|
||||||
text-decoration: underline;
|
|
||||||
}
|
|
||||||
#table-cards .card-figure {
|
|
||||||
font-size: 32px;
|
|
||||||
font-weight: bold;
|
|
||||||
}
|
|
||||||
#table-cards .card-detail,
|
|
||||||
#table-cards .card-time {
|
|
||||||
color: #7f8c8d;
|
|
||||||
font-size: 12px;
|
|
||||||
}
|
|
||||||
#table-cards .prefix {
|
|
||||||
font-family: 'SF Mono', Monaco, 'Cascadia Mono', 'Roboto Mono', Consolas, 'Courier New', monospace;
|
|
||||||
}
|
|
||||||
#well-known {
|
|
||||||
margin-bottom: 40px;
|
|
||||||
}
|
|
||||||
#well-known table {
|
|
||||||
width: 100%;
|
|
||||||
border-collapse: collapse;
|
|
||||||
font-size: 14px;
|
|
||||||
}
|
|
||||||
#well-known th,
|
|
||||||
#well-known td {
|
|
||||||
text-align: left;
|
|
||||||
padding: 8px 4px;
|
|
||||||
border-bottom: 1px solid #eee;
|
|
||||||
}
|
|
||||||
#well-known th {
|
|
||||||
color: #7f8c8d;
|
|
||||||
font-weight: 600;
|
|
||||||
}
|
|
||||||
|
|
||||||
/* Footer */
|
/* Footer */
|
||||||
.footer {
|
.footer {
|
||||||
margin-top: 40px;
|
margin-top: 40px;
|
||||||
@@ -427,109 +381,6 @@
|
|||||||
</div>
|
</div>
|
||||||
</div>
|
</div>
|
||||||
|
|
||||||
{{ with .Cards }}
|
|
||||||
<div class="search-section" id="table-cards">
|
|
||||||
<div class="search-card">
|
|
||||||
<h2>Newest Updated Route</h2>
|
|
||||||
{{ with .NewestRoute }}
|
|
||||||
<p><a href="{{ prefixURL .Prefix.String }}" class="prefix">{{ .Prefix }}</a> from {{ if .OriginASN }}<a href="/as/{{ .OriginASN }}">AS{{ .OriginASN }}</a>{{ else }}no known origin{{ end }}</p>
|
|
||||||
<p class="card-time">{{ .Time.UTC.Format "2006-01-02 15:04:05" }} UTC, {{ timeSince .Time }}</p>
|
|
||||||
{{ else }}
|
|
||||||
<p>The routing table keeps no route yet.</p>
|
|
||||||
{{ end }}
|
|
||||||
</div>
|
|
||||||
|
|
||||||
<div class="search-card">
|
|
||||||
<h2>Oldest Updated Route</h2>
|
|
||||||
{{ with .OldestRoute }}
|
|
||||||
<p><a href="{{ prefixURL .Prefix.String }}" class="prefix">{{ .Prefix }}</a> from {{ if .OriginASN }}<a href="/as/{{ .OriginASN }}">AS{{ .OriginASN }}</a>{{ else }}no known origin{{ end }}</p>
|
|
||||||
<p class="card-time">{{ .Time.UTC.Format "2006-01-02 15:04:05" }} UTC, {{ timeSince .Time }}</p>
|
|
||||||
{{ else }}
|
|
||||||
<p>The routing table keeps no route yet.</p>
|
|
||||||
{{ end }}
|
|
||||||
</div>
|
|
||||||
|
|
||||||
<div class="search-card">
|
|
||||||
<h2>Newest Updated AS</h2>
|
|
||||||
{{ with .NewestAS }}
|
|
||||||
<p><a href="/as/{{ .ASN }}">AS{{ .ASN }}</a> {{ .Handle }}</p>
|
|
||||||
<p class="card-detail">{{ .Description }}</p>
|
|
||||||
<p class="card-time">Last seen {{ .LastSeen.UTC.Format "2006-01-02 15:04:05" }} UTC, {{ timeSince .LastSeen }}</p>
|
|
||||||
{{ else }}
|
|
||||||
<p>No AS has been seen yet.</p>
|
|
||||||
{{ end }}
|
|
||||||
</div>
|
|
||||||
|
|
||||||
<div class="search-card">
|
|
||||||
<h2>Oldest Updated AS</h2>
|
|
||||||
{{ with .OldestAS }}
|
|
||||||
<p><a href="/as/{{ .ASN }}">AS{{ .ASN }}</a> {{ .Handle }}</p>
|
|
||||||
<p class="card-detail">{{ .Description }}</p>
|
|
||||||
<p class="card-time">Last seen {{ .LastSeen.UTC.Format "2006-01-02 15:04:05" }} UTC, {{ timeSince .LastSeen }}</p>
|
|
||||||
{{ else }}
|
|
||||||
<p>No AS has been seen yet.</p>
|
|
||||||
{{ end }}
|
|
||||||
</div>
|
|
||||||
|
|
||||||
<div class="search-card">
|
|
||||||
<h2>AS with the Most Prefixes</h2>
|
|
||||||
{{ if .MostPrefixes }}
|
|
||||||
<p><a href="/as/{{ .MostPrefixesASN }}">AS{{ .MostPrefixesASN }}</a></p>
|
|
||||||
<p>Prefixes it originates: {{ .MostPrefixes }}</p>
|
|
||||||
{{ else }}
|
|
||||||
<p>No AS originates a prefix yet.</p>
|
|
||||||
{{ end }}
|
|
||||||
</div>
|
|
||||||
|
|
||||||
<div class="search-card">
|
|
||||||
<h2>ASes with the Fewest Prefixes</h2>
|
|
||||||
<p>ASes that originate a single prefix: {{ .SinglePrefixASes }}</p>
|
|
||||||
{{ with .NewestSinglePrefix }}
|
|
||||||
<p>The most recently updated of them: <a href="/as/{{ .OriginASN }}">AS{{ .OriginASN }}</a>, with <a href="{{ prefixURL .Prefix.String }}" class="prefix">{{ .Prefix }}</a></p>
|
|
||||||
<p class="card-time">{{ .Time.UTC.Format "2006-01-02 15:04:05" }} UTC, {{ timeSince .Time }}</p>
|
|
||||||
{{ end }}
|
|
||||||
</div>
|
|
||||||
|
|
||||||
<div class="search-card">
|
|
||||||
<h2>Bogon Announcements</h2>
|
|
||||||
<p class="card-figure"><a href="/bogons">{{ .BogonCount }}</a></p>
|
|
||||||
<p>Routes that announce a reserved prefix, or that come from a reserved AS number</p>
|
|
||||||
</div>
|
|
||||||
|
|
||||||
<div class="search-card">
|
|
||||||
<h2>Prefixes with Several Origins</h2>
|
|
||||||
<p class="card-figure"><a href="/multiple-origins">{{ .MultipleOriginCount }}</a></p>
|
|
||||||
<p>Prefixes announced by more than one origin AS</p>
|
|
||||||
</div>
|
|
||||||
</div>
|
|
||||||
|
|
||||||
<div class="search-card" id="well-known">
|
|
||||||
<h2>Well-Known Networks</h2>
|
|
||||||
<table>
|
|
||||||
<thead>
|
|
||||||
<tr>
|
|
||||||
<th>Network</th>
|
|
||||||
<th>Prefixes</th>
|
|
||||||
<th>Last Update</th>
|
|
||||||
</tr>
|
|
||||||
</thead>
|
|
||||||
<tbody>
|
|
||||||
{{ range .WellKnownNetworks }}
|
|
||||||
<tr>
|
|
||||||
<td><a href="/as/{{ .ASN }}">AS{{ .ASN }}</a> {{ .Name }}</td>
|
|
||||||
<td>{{ .Prefixes }}</td>
|
|
||||||
<td>{{ with .Newest }}{{ timeSince . }}{{ else }}no route{{ end }}</td>
|
|
||||||
</tr>
|
|
||||||
{{ end }}
|
|
||||||
</tbody>
|
|
||||||
</table>
|
|
||||||
</div>
|
|
||||||
{{ else }}
|
|
||||||
<div class="search-card" id="well-known">
|
|
||||||
<p>The figures of the routing table are being worked out. Reload the page in a minute.</p>
|
|
||||||
</div>
|
|
||||||
{{ end }}
|
|
||||||
|
|
||||||
<div class="search-card" id="ip-api">
|
<div class="search-card" id="ip-api">
|
||||||
<h2>IP Information API</h2>
|
<h2>IP Information API</h2>
|
||||||
<p>The API answers what RouteWatch knows about an IP address, such as its AS number and country. <code>/ip/</code> with no address answers for the caller's own address.</p>
|
<p>The API answers what RouteWatch knows about an IP address, such as its AS number and country. <code>/ip/</code> with no address answers for the caller's own address.</p>
|
||||||
@@ -544,8 +395,6 @@
|
|||||||
<span>{{appLicense}}</span>
|
<span>{{appLicense}}</span>
|
||||||
<span class="separator">|</span>
|
<span class="separator">|</span>
|
||||||
<span><a href="{{appGitCommitURL}}">{{appGitRevision}}</a></span>
|
<span><a href="{{appGitCommitURL}}">{{appGitRevision}}</a></span>
|
||||||
<span class="separator">|</span>
|
|
||||||
<span>IP Geolocation by <a href="https://db-ip.com">DB-IP</a></span>
|
|
||||||
</footer>
|
</footer>
|
||||||
|
|
||||||
<script nonce="{{.CSPNonce}}">
|
<script nonce="{{.CSPNonce}}">
|
||||||
@@ -716,8 +565,7 @@
|
|||||||
});
|
});
|
||||||
});
|
});
|
||||||
|
|
||||||
// IP lookup: show the whole JSON answer, its @meta and an error answer
|
// IP lookup
|
||||||
// included, as the IP information API card does.
|
|
||||||
document.getElementById('ip-form').addEventListener('submit', function(e) {
|
document.getElementById('ip-form').addEventListener('submit', function(e) {
|
||||||
e.preventDefault();
|
e.preventDefault();
|
||||||
const ip = document.getElementById('ip-input').value.trim();
|
const ip = document.getElementById('ip-input').value.trim();
|
||||||
@@ -733,11 +581,18 @@
|
|||||||
|
|
||||||
fetch('/ip/' + encodeURIComponent(ip))
|
fetch('/ip/' + encodeURIComponent(ip))
|
||||||
.then(response => response.json())
|
.then(response => response.json())
|
||||||
.then(answer => {
|
.then(response => {
|
||||||
contentEl.className = '';
|
contentEl.classList.remove('loading');
|
||||||
contentEl.textContent = JSON.stringify(answer, null, 2);
|
if (response.status === 'error') {
|
||||||
|
contentEl.className = 'error';
|
||||||
|
contentEl.textContent = 'Error: ' + response.error.msg;
|
||||||
|
} else {
|
||||||
|
contentEl.className = '';
|
||||||
|
contentEl.textContent = JSON.stringify(response.data, null, 2);
|
||||||
|
}
|
||||||
})
|
})
|
||||||
.catch(error => {
|
.catch(error => {
|
||||||
|
contentEl.classList.remove('loading');
|
||||||
contentEl.className = 'error';
|
contentEl.className = 'error';
|
||||||
contentEl.textContent = 'Error: ' + error.message;
|
contentEl.textContent = 'Error: ' + error.message;
|
||||||
});
|
});
|
||||||
|
|||||||
@@ -1,182 +0,0 @@
|
|||||||
<!DOCTYPE html>
|
|
||||||
<html lang="en">
|
|
||||||
<head>
|
|
||||||
<meta charset="UTF-8">
|
|
||||||
<meta name="viewport" content="width=device-width, initial-scale=1.0">
|
|
||||||
<title>Prefixes with Several Origins - RouteWatch</title>
|
|
||||||
<style nonce="{{.CSPNonce}}">
|
|
||||||
* {
|
|
||||||
box-sizing: border-box;
|
|
||||||
}
|
|
||||||
body {
|
|
||||||
font-family: -apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, sans-serif;
|
|
||||||
margin: 0;
|
|
||||||
padding: 0;
|
|
||||||
background: #f5f5f5;
|
|
||||||
}
|
|
||||||
|
|
||||||
/* Navbar styles */
|
|
||||||
.navbar {
|
|
||||||
background: #2c3e50;
|
|
||||||
padding: 0 20px;
|
|
||||||
display: flex;
|
|
||||||
align-items: center;
|
|
||||||
justify-content: space-between;
|
|
||||||
height: 56px;
|
|
||||||
box-shadow: 0 2px 4px rgba(0,0,0,0.1);
|
|
||||||
}
|
|
||||||
.navbar-brand {
|
|
||||||
display: flex;
|
|
||||||
align-items: center;
|
|
||||||
gap: 8px;
|
|
||||||
font-size: 18px;
|
|
||||||
font-weight: 600;
|
|
||||||
}
|
|
||||||
.navbar-brand a {
|
|
||||||
color: white;
|
|
||||||
text-decoration: none;
|
|
||||||
}
|
|
||||||
.navbar-brand a:hover {
|
|
||||||
color: #ecf0f1;
|
|
||||||
}
|
|
||||||
.navbar-brand .by {
|
|
||||||
font-weight: normal;
|
|
||||||
color: #95a5a6;
|
|
||||||
font-size: 14px;
|
|
||||||
}
|
|
||||||
.navbar-brand .author {
|
|
||||||
color: #3498db;
|
|
||||||
font-weight: normal;
|
|
||||||
}
|
|
||||||
.navbar-brand .author:hover {
|
|
||||||
text-decoration: underline;
|
|
||||||
}
|
|
||||||
.navbar-links {
|
|
||||||
display: flex;
|
|
||||||
gap: 20px;
|
|
||||||
}
|
|
||||||
.navbar-links a {
|
|
||||||
color: #ecf0f1;
|
|
||||||
text-decoration: none;
|
|
||||||
font-size: 14px;
|
|
||||||
padding: 8px 12px;
|
|
||||||
border-radius: 4px;
|
|
||||||
transition: background-color 0.2s;
|
|
||||||
}
|
|
||||||
.navbar-links a:hover {
|
|
||||||
background: rgba(255,255,255,0.1);
|
|
||||||
}
|
|
||||||
|
|
||||||
/* Main content */
|
|
||||||
.main-content {
|
|
||||||
max-width: 1200px;
|
|
||||||
margin: 0 auto;
|
|
||||||
padding: 20px;
|
|
||||||
}
|
|
||||||
h1 {
|
|
||||||
color: #333;
|
|
||||||
margin-bottom: 10px;
|
|
||||||
}
|
|
||||||
.subtitle {
|
|
||||||
color: #666;
|
|
||||||
margin-bottom: 30px;
|
|
||||||
}
|
|
||||||
table {
|
|
||||||
width: 100%;
|
|
||||||
border-collapse: collapse;
|
|
||||||
background: white;
|
|
||||||
box-shadow: 0 2px 4px rgba(0,0,0,0.1);
|
|
||||||
border-radius: 8px;
|
|
||||||
overflow: hidden;
|
|
||||||
}
|
|
||||||
th {
|
|
||||||
background: #f8f9fa;
|
|
||||||
padding: 12px;
|
|
||||||
text-align: left;
|
|
||||||
font-weight: 600;
|
|
||||||
color: #333;
|
|
||||||
border-bottom: 2px solid #dee2e6;
|
|
||||||
}
|
|
||||||
td {
|
|
||||||
padding: 12px;
|
|
||||||
border-bottom: 1px solid #eee;
|
|
||||||
vertical-align: top;
|
|
||||||
}
|
|
||||||
tr:last-child td {
|
|
||||||
border-bottom: none;
|
|
||||||
}
|
|
||||||
tr:hover {
|
|
||||||
background: #f8f9fa;
|
|
||||||
}
|
|
||||||
a {
|
|
||||||
color: #0066cc;
|
|
||||||
text-decoration: none;
|
|
||||||
}
|
|
||||||
a:hover {
|
|
||||||
text-decoration: underline;
|
|
||||||
}
|
|
||||||
.prefix-link {
|
|
||||||
font-family: 'SF Mono', Monaco, 'Cascadia Mono', 'Roboto Mono', Consolas, 'Courier New', monospace;
|
|
||||||
}
|
|
||||||
.origins {
|
|
||||||
list-style: none;
|
|
||||||
margin: 0;
|
|
||||||
padding: 0;
|
|
||||||
}
|
|
||||||
.peers {
|
|
||||||
color: #666;
|
|
||||||
font-size: 0.9em;
|
|
||||||
}
|
|
||||||
</style>
|
|
||||||
</head>
|
|
||||||
<body>
|
|
||||||
<nav class="navbar">
|
|
||||||
<div class="navbar-brand">
|
|
||||||
<a href="/">routewatch</a>
|
|
||||||
<span class="by">by</span>
|
|
||||||
<a href="https://sneak.berlin" class="author">@sneak</a>
|
|
||||||
</div>
|
|
||||||
<div class="navbar-links">
|
|
||||||
<a href="/">Home</a>
|
|
||||||
<a href="/status">Status</a>
|
|
||||||
</div>
|
|
||||||
</nav>
|
|
||||||
|
|
||||||
<main class="main-content">
|
|
||||||
<h1>Prefixes with Several Origin ASes</h1>
|
|
||||||
{{ with .Cards }}
|
|
||||||
<p class="subtitle">
|
|
||||||
Prefixes announced by more than one origin AS: {{ .MultipleOriginCount }}.
|
|
||||||
Such a prefix can be a hijack.
|
|
||||||
{{ if lt (len .MultipleOrigins) .MultipleOriginCount }}The first {{ len .MultipleOrigins }} are listed, in prefix order.{{ end }}
|
|
||||||
The list is worked out once a minute.
|
|
||||||
</p>
|
|
||||||
|
|
||||||
<table>
|
|
||||||
<thead>
|
|
||||||
<tr>
|
|
||||||
<th>Prefix</th>
|
|
||||||
<th>Origin ASes</th>
|
|
||||||
</tr>
|
|
||||||
</thead>
|
|
||||||
<tbody>
|
|
||||||
{{ range .MultipleOrigins }}
|
|
||||||
<tr>
|
|
||||||
<td><a href="{{ prefixURL .Prefix.String }}" class="prefix-link">{{ .Prefix }}</a></td>
|
|
||||||
<td>
|
|
||||||
<ul class="origins">
|
|
||||||
{{ range .Origins }}
|
|
||||||
<li>{{ if .ASN }}<a href="/as/{{ .ASN }}">AS{{ .ASN }}</a>{{ else }}No known origin{{ end }} <span class="peers">{{ .Peers }} {{ if eq .Peers 1 }}peer{{ else }}peers{{ end }}</span></li>
|
|
||||||
{{ end }}
|
|
||||||
</ul>
|
|
||||||
</td>
|
|
||||||
</tr>
|
|
||||||
{{ end }}
|
|
||||||
</tbody>
|
|
||||||
</table>
|
|
||||||
{{ else }}
|
|
||||||
<p class="subtitle">The list is being worked out. Reload the page in a minute.</p>
|
|
||||||
{{ end }}
|
|
||||||
</main>
|
|
||||||
</body>
|
|
||||||
</html>
|
|
||||||
@@ -299,41 +299,6 @@
|
|||||||
</div>
|
</div>
|
||||||
{{end}}
|
{{end}}
|
||||||
|
|
||||||
{{if .OriginChanges}}
|
|
||||||
<div class="origins-section">
|
|
||||||
<h3>Origin Changes</h3>
|
|
||||||
{{if .MoreOriginChanges}}
|
|
||||||
<p>There are more than {{.MaxOriginChanges}} changes. Showing the newest {{.MaxOriginChanges}}.</p>
|
|
||||||
{{end}}
|
|
||||||
<table class="route-table">
|
|
||||||
<thead>
|
|
||||||
<tr>
|
|
||||||
<th>Time (UTC)</th>
|
|
||||||
<th>Change</th>
|
|
||||||
<th>Origin AS</th>
|
|
||||||
<th>Peers</th>
|
|
||||||
</tr>
|
|
||||||
</thead>
|
|
||||||
<tbody>
|
|
||||||
{{range .OriginChanges}}
|
|
||||||
<tr>
|
|
||||||
<td>{{.Time.Format "2006-01-02 15:04:05"}}</td>
|
|
||||||
<td>{{if .Announced}}Announced{{else}}Withdrawn{{end}}</td>
|
|
||||||
<td>
|
|
||||||
{{if .OriginASN}}
|
|
||||||
<a href="/as/{{.OriginASN}}" class="as-link">AS{{.OriginASN}}</a>
|
|
||||||
{{else}}
|
|
||||||
No known origin
|
|
||||||
{{end}}
|
|
||||||
</td>
|
|
||||||
<td>{{.PeerCount}}</td>
|
|
||||||
</tr>
|
|
||||||
{{end}}
|
|
||||||
</tbody>
|
|
||||||
</table>
|
|
||||||
</div>
|
|
||||||
{{end}}
|
|
||||||
|
|
||||||
{{if .Routes}}
|
{{if .Routes}}
|
||||||
<div class="routes-section">
|
<div class="routes-section">
|
||||||
<div class="routes-header">
|
<div class="routes-header">
|
||||||
|
|||||||
@@ -401,8 +401,6 @@
|
|||||||
const REFRESH_PERIOD_MS = 2000;
|
const REFRESH_PERIOD_MS = 2000;
|
||||||
const STALE_AFTER_REFRESHES = 3;
|
const STALE_AFTER_REFRESHES = 3;
|
||||||
const MS_PER_SECOND = 1000;
|
const MS_PER_SECOND = 1000;
|
||||||
const SECONDS_PER_MINUTE = 60;
|
|
||||||
const SECONDS_PER_HOUR = 3600;
|
|
||||||
const SPINNER_FRAMES = ['⠋', '⠙', '⠹', '⠸', '⠼', '⠴', '⠦', '⠧', '⠇', '⠏'];
|
const SPINNER_FRAMES = ['⠋', '⠙', '⠹', '⠸', '⠼', '⠴', '⠦', '⠧', '⠇', '⠏'];
|
||||||
const SPINNER_COLOURS = ['#e74c3c', '#e67e22', '#f1c40f', '#27ae60', '#3498db', '#9b59b6'];
|
const SPINNER_COLOURS = ['#e74c3c', '#e67e22', '#f1c40f', '#27ae60', '#3498db', '#9b59b6'];
|
||||||
|
|
||||||
@@ -445,18 +443,7 @@
|
|||||||
function formatNumber(num) {
|
function formatNumber(num) {
|
||||||
return num.toLocaleString();
|
return num.toLocaleString();
|
||||||
}
|
}
|
||||||
|
|
||||||
// formatDuration writes a number of seconds in hours, minutes and
|
|
||||||
// seconds, such as "1h 18m 34s" or "18.2s", with at most one decimal.
|
|
||||||
function formatDuration(totalSeconds) {
|
|
||||||
const hours = Math.floor(totalSeconds / SECONDS_PER_HOUR);
|
|
||||||
const minutes = Math.floor(totalSeconds % SECONDS_PER_HOUR / SECONDS_PER_MINUTE);
|
|
||||||
const seconds = parseFloat((totalSeconds % SECONDS_PER_MINUTE).toFixed(1));
|
|
||||||
if (hours > 0) return hours + 'h ' + minutes + 'm ' + seconds + 's';
|
|
||||||
if (minutes > 0) return minutes + 'm ' + seconds + 's';
|
|
||||||
return seconds + 's';
|
|
||||||
}
|
|
||||||
|
|
||||||
function formatProcessingTime(ms) {
|
function formatProcessingTime(ms) {
|
||||||
if (ms < 0.001) {
|
if (ms < 0.001) {
|
||||||
return (ms * 1000).toFixed(0) + ' µs';
|
return (ms * 1000).toFixed(0) + ' µs';
|
||||||
@@ -541,10 +528,6 @@
|
|||||||
<span class="metric-label">Dropped</span>
|
<span class="metric-label">Dropped</span>
|
||||||
<span class="metric-value ${handler.dropped_count > 0 ? 'disconnected' : ''}">${formatNumber(handler.dropped_count)}</span>
|
<span class="metric-value ${handler.dropped_count > 0 ? 'disconnected' : ''}">${formatNumber(handler.dropped_count)}</span>
|
||||||
</div>
|
</div>
|
||||||
<div class="metric">
|
|
||||||
<span class="metric-label">Held Back</span>
|
|
||||||
<span class="metric-value">${formatNumber(handler.held_back_count)}</span>
|
|
||||||
</div>
|
|
||||||
<div class="metric">
|
<div class="metric">
|
||||||
<span class="metric-label">Avg Time</span>
|
<span class="metric-label">Avg Time</span>
|
||||||
<span class="metric-value">${formatProcessingTime(handler.avg_process_time_ms)}</span>
|
<span class="metric-value">${formatProcessingTime(handler.avg_process_time_ms)}</span>
|
||||||
@@ -636,13 +619,11 @@
|
|||||||
noteMessageCount(data.total_messages);
|
noteMessageCount(data.total_messages);
|
||||||
|
|
||||||
// Update all metrics
|
// Update all metrics
|
||||||
document.getElementById('uptime').textContent = formatDuration(data.uptime_seconds);
|
document.getElementById('uptime').textContent = data.uptime;
|
||||||
document.getElementById('go_version').textContent = data.go_version;
|
document.getElementById('go_version').textContent = data.go_version;
|
||||||
document.getElementById('goroutines').textContent = formatNumber(data.goroutines);
|
document.getElementById('goroutines').textContent = formatNumber(data.goroutines);
|
||||||
document.getElementById('memory_usage').textContent = formatBytes(data.memory_usage_bytes);
|
document.getElementById('memory_usage').textContent = data.memory_usage;
|
||||||
// connection_duration_seconds is null while the stream is not connected.
|
document.getElementById('connection_duration').textContent = data.connection_duration;
|
||||||
document.getElementById('connection_duration').textContent =
|
|
||||||
data.connection_duration_seconds === null ? 'disconnected' : formatDuration(data.connection_duration_seconds);
|
|
||||||
document.getElementById('reconnect_count').textContent = formatNumber(data.reconnect_count);
|
document.getElementById('reconnect_count').textContent = formatNumber(data.reconnect_count);
|
||||||
document.getElementById('total_messages').textContent = formatNumber(data.total_messages);
|
document.getElementById('total_messages').textContent = formatNumber(data.total_messages);
|
||||||
document.getElementById('messages_per_sec').textContent = formatNumber(Math.round(data.messages_per_sec));
|
document.getElementById('messages_per_sec').textContent = formatNumber(Math.round(data.messages_per_sec));
|
||||||
@@ -691,7 +672,7 @@
|
|||||||
const errorsEl = document.getElementById('whois_errors');
|
const errorsEl = document.getElementById('whois_errors');
|
||||||
errorsEl.textContent = formatNumber(data.whois_stats.errors_last_hour);
|
errorsEl.textContent = formatNumber(data.whois_stats.errors_last_hour);
|
||||||
errorsEl.className = 'metric-value' + (data.whois_stats.errors_last_hour > 0 ? ' disconnected' : '');
|
errorsEl.className = 'metric-value' + (data.whois_stats.errors_last_hour > 0 ? ' disconnected' : '');
|
||||||
document.getElementById('whois_interval').textContent = formatDuration(data.whois_stats.current_interval_seconds);
|
document.getElementById('whois_interval').textContent = data.whois_stats.current_interval;
|
||||||
}
|
}
|
||||||
|
|
||||||
// Update handler stats
|
// Update handler stats
|
||||||
|
|||||||
@@ -27,12 +27,6 @@ var prefixDetailHTML string
|
|||||||
//go:embed prefix_length.html
|
//go:embed prefix_length.html
|
||||||
var prefixLengthHTML string
|
var prefixLengthHTML string
|
||||||
|
|
||||||
//go:embed bogons.html
|
|
||||||
var bogonsHTML string
|
|
||||||
|
|
||||||
//go:embed multiple_origins.html
|
|
||||||
var multipleOriginsHTML string
|
|
||||||
|
|
||||||
// Templates contains all parsed templates
|
// Templates contains all parsed templates
|
||||||
type Templates struct {
|
type Templates struct {
|
||||||
// Index is the template for the home page
|
// Index is the template for the home page
|
||||||
@@ -45,11 +39,6 @@ type Templates struct {
|
|||||||
PrefixDetail *template.Template
|
PrefixDetail *template.Template
|
||||||
// PrefixLength is the template for displaying prefixes by length
|
// PrefixLength is the template for displaying prefixes by length
|
||||||
PrefixLength *template.Template
|
PrefixLength *template.Template
|
||||||
// Bogons is the template of the page that lists the bogon announcements
|
|
||||||
Bogons *template.Template
|
|
||||||
// MultipleOrigins is the template of the page that lists the prefixes
|
|
||||||
// announced by more than one origin AS
|
|
||||||
MultipleOrigins *template.Template
|
|
||||||
}
|
}
|
||||||
|
|
||||||
var (
|
var (
|
||||||
@@ -127,18 +116,6 @@ func initTemplates() {
|
|||||||
if err != nil {
|
if err != nil {
|
||||||
panic("failed to parse prefix length template: " + err.Error())
|
panic("failed to parse prefix length template: " + err.Error())
|
||||||
}
|
}
|
||||||
|
|
||||||
defaultTemplates.Bogons, err = template.New("bogons").
|
|
||||||
Funcs(funcs).Parse(bogonsHTML)
|
|
||||||
if err != nil {
|
|
||||||
panic("failed to parse bogons template: " + err.Error())
|
|
||||||
}
|
|
||||||
|
|
||||||
defaultTemplates.MultipleOrigins, err = template.New("multipleOrigins").
|
|
||||||
Funcs(funcs).Parse(multipleOriginsHTML)
|
|
||||||
if err != nil {
|
|
||||||
panic("failed to parse multiple origins template: " + err.Error())
|
|
||||||
}
|
|
||||||
}
|
}
|
||||||
|
|
||||||
// Get returns the singleton Templates instance
|
// Get returns the singleton Templates instance
|
||||||
@@ -172,14 +149,3 @@ func PrefixDetailTemplate() *template.Template {
|
|||||||
func PrefixLengthTemplate() *template.Template {
|
func PrefixLengthTemplate() *template.Template {
|
||||||
return Get().PrefixLength
|
return Get().PrefixLength
|
||||||
}
|
}
|
||||||
|
|
||||||
// BogonsTemplate returns the parsed template of the bogon announcements page
|
|
||||||
func BogonsTemplate() *template.Template {
|
|
||||||
return Get().Bogons
|
|
||||||
}
|
|
||||||
|
|
||||||
// MultipleOriginsTemplate returns the parsed template of the page of prefixes
|
|
||||||
// announced by more than one origin AS
|
|
||||||
func MultipleOriginsTemplate() *template.Template {
|
|
||||||
return Get().MultipleOrigins
|
|
||||||
}
|
|
||||||
@@ -12,17 +12,13 @@ import (
|
|||||||
)
|
)
|
||||||
|
|
||||||
// renderPage renders a page with the data every page needs, and the visitor's
|
// renderPage renders a page with the data every page needs, and the visitor's
|
||||||
// address and the routing table's cards, which the home page also needs. The
|
// address, which the home page also needs.
|
||||||
// cards are left out, as before the server first works them out.
|
|
||||||
func renderPage(t *testing.T, tmpl *template.Template) string {
|
func renderPage(t *testing.T, tmpl *template.Template) string {
|
||||||
t.Helper()
|
t.Helper()
|
||||||
|
|
||||||
var page bytes.Buffer
|
var page bytes.Buffer
|
||||||
|
|
||||||
err := tmpl.Execute(&page, struct {
|
err := tmpl.Execute(&page, struct{ CSPNonce, ClientIP string }{})
|
||||||
CSPNonce, ClientIP string
|
|
||||||
Cards any
|
|
||||||
}{})
|
|
||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatalf("rendering the page: %v", err)
|
t.Fatalf("rendering the page: %v", err)
|
||||||
}
|
}
|
||||||
@@ -67,23 +63,6 @@ func TestFooterLicense(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// TestHomePageCreditsDBIP expects the home page's footer to carry the credit
|
|
||||||
// to DB-IP.
|
|
||||||
func TestHomePageCreditsDBIP(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
const attribution = `IP Geolocation by <a href="https://db-ip.com">DB-IP</a>`
|
|
||||||
|
|
||||||
_, footer, found := strings.Cut(renderPage(t, templates.IndexTemplate()), "<footer")
|
|
||||||
if !found {
|
|
||||||
t.Fatal("the page has no footer")
|
|
||||||
}
|
|
||||||
|
|
||||||
if !strings.Contains(footer, attribution) {
|
|
||||||
t.Errorf("footer does not credit DB-IP: %q", footer)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestFreshnessIndicator renders the home and status pages and expects each to
|
// TestFreshnessIndicator renders the home and status pages and expects each to
|
||||||
// have, right after the status element, the element where the page's script
|
// have, right after the status element, the element where the page's script
|
||||||
// shows its spinner or "stale".
|
// shows its spinner or "stale".
|
||||||
|
|||||||
+36
-82
@@ -12,7 +12,6 @@ import (
|
|||||||
"regexp"
|
"regexp"
|
||||||
"strings"
|
"strings"
|
||||||
"sync"
|
"sync"
|
||||||
"syscall"
|
|
||||||
"time"
|
"time"
|
||||||
"unicode/utf8"
|
"unicode/utf8"
|
||||||
|
|
||||||
@@ -71,12 +70,12 @@ const nicbrHeader = "% Copyright (c) Nic.br"
|
|||||||
// queries to stop the data being taken in bulk.
|
// queries to stop the data being taken in bulk.
|
||||||
const nicbrLimitReached = "Query rate limit exceeded"
|
const nicbrLimitReached = "Query rate limit exceeded"
|
||||||
|
|
||||||
// PauseTime is how long the client sends an RIR's or NIC.br's server no
|
// NICbrSkipTime is how long the client stops asking NIC.br after NIC.br
|
||||||
// query after that server refuses a query. queryRegistry says what a
|
// answers that the client is over its query limit. NIC.br publishes
|
||||||
// refusal is. RIPE refuses an address for good when the address keeps
|
// neither the limit nor how long it lasts. An hour outlasts a limit
|
||||||
// passing RIPE's limit (see noPersonalFlag). An hour outlasts a limit
|
// counted per minute or per hour. The ASes looked up meanwhile get
|
||||||
// counted per minute or per hour.
|
// NIC.br's fields when they are next refreshed.
|
||||||
const PauseTime = time.Hour
|
const NICbrSkipTime = time.Hour
|
||||||
|
|
||||||
// RIR identifiers.
|
// RIR identifiers.
|
||||||
const (
|
const (
|
||||||
@@ -126,10 +125,6 @@ var (
|
|||||||
errAnswerTooLarge = errors.New("WHOIS answer reached the size cap")
|
errAnswerTooLarge = errors.New("WHOIS answer reached the size cap")
|
||||||
)
|
)
|
||||||
|
|
||||||
// ErrPaused is returned when a lookup needs a server that refused a query
|
|
||||||
// less than PauseTime ago. The client sends that server nothing meanwhile.
|
|
||||||
var ErrPaused = errors.New("WHOIS server paused after refusing a query")
|
|
||||||
|
|
||||||
// ASNInfo contains parsed WHOIS information for an ASN.
|
// ASNInfo contains parsed WHOIS information for an ASN.
|
||||||
type ASNInfo struct {
|
type ASNInfo struct {
|
||||||
ASN int
|
ASN int
|
||||||
@@ -161,12 +156,12 @@ type Client struct {
|
|||||||
|
|
||||||
logger *slog.Logger
|
logger *slog.Logger
|
||||||
|
|
||||||
// mu guards pausedUntil.
|
// mu guards nicbrSkipUntil.
|
||||||
mu sync.Mutex
|
mu sync.Mutex
|
||||||
|
|
||||||
// pausedUntil maps a server, given as host:port, to the time the client
|
// nicbrSkipUntil is when the client next asks NIC.br, after NIC.br
|
||||||
// may ask it again.
|
// answered that the client is over its query limit.
|
||||||
pausedUntil map[string]time.Time
|
nicbrSkipUntil time.Time
|
||||||
}
|
}
|
||||||
|
|
||||||
// NewClient creates a new WHOIS client that logs to logger.
|
// NewClient creates a new WHOIS client that logs to logger.
|
||||||
@@ -177,29 +172,23 @@ func NewClient(logger *slog.Logger) *Client {
|
|||||||
DialContext: dialer.DialContext,
|
DialContext: dialer.DialContext,
|
||||||
Now: time.Now,
|
Now: time.Now,
|
||||||
logger: logger,
|
logger: logger,
|
||||||
pausedUntil: make(map[string]time.Time),
|
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// LookupASN asks IANA which RIR holds the given ASN, asks that RIR, and
|
// LookupASN asks IANA which RIR holds the given ASN, asks that RIR, and
|
||||||
// returns what the RIR's answer says. When LACNIC's answer is NIC.br's, it
|
// returns what the RIR's answer says. When LACNIC's answer is NIC.br's, it
|
||||||
// returns what NIC.br's own server says instead, unless that query fails or
|
// returns what NIC.br's own server says instead, unless that query fails or
|
||||||
// NIC.br is paused, as askNICbr says.
|
// is skipped, as askNICbr says.
|
||||||
//
|
//
|
||||||
// It returns ErrNoReferral when IANA refers the AS to no RIR. Its error
|
// It returns ErrNoReferral when IANA refers the AS to no RIR, and another
|
||||||
// wraps ErrPaused when the RIR is paused. It returns another error in each
|
// error when the query to IANA or to the RIR fails, when the RIR refuses
|
||||||
// of these cases:
|
// it, when IANA's answer has neither a referral nor the AS's block, when
|
||||||
//
|
// IANA refers the AS to a server none of the five RIRs run, or when the
|
||||||
// - the query to IANA or to the RIR fails
|
// answer carries none of ASNInfo's fields.
|
||||||
// - the RIR refuses the query
|
|
||||||
// - IANA's answer has neither a referral nor the AS's block
|
|
||||||
// - IANA refers the AS to a server none of the five RIRs run
|
|
||||||
// - the answer carries none of ASNInfo's fields
|
|
||||||
func (c *Client) LookupASN(ctx context.Context, asn int) (*ASNInfo, error) {
|
func (c *Client) LookupASN(ctx context.Context, asn int) (*ASNInfo, error) {
|
||||||
// Query IANA first to find the authoritative RIR
|
// Query IANA first to find the authoritative RIR
|
||||||
query := fmt.Sprintf("AS%d", asn)
|
query := fmt.Sprintf("AS%d", asn)
|
||||||
|
|
||||||
// IANA is never paused, because a pause of IANA would stop every lookup.
|
|
||||||
ianaResp, err := c.query(ctx, whoisServerIANA, query)
|
ianaResp, err := c.query(ctx, whoisServerIANA, query)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return nil, fmt.Errorf("IANA query failed: %w", err)
|
return nil, fmt.Errorf("IANA query failed: %w", err)
|
||||||
@@ -222,7 +211,7 @@ func (c *Client) LookupASN(ctx context.Context, asn int) (*ASNInfo, error) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
// Query the authoritative RIR
|
// Query the authoritative RIR
|
||||||
rirResp, err := c.queryRegistry(ctx, whoisServer, rirQuery(rir, query))
|
rirResp, err := c.queryRIR(ctx, whoisServer, rirQuery(rir, query))
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return nil, fmt.Errorf("RIR query failed: %w", err)
|
return nil, fmt.Errorf("RIR query failed: %w", err)
|
||||||
}
|
}
|
||||||
@@ -252,19 +241,22 @@ func (c *Client) LookupASN(ctx context.Context, asn int) (*ASNInfo, error) {
|
|||||||
// askNICbr returns lacnicInfo, what LACNIC's answer says, and logs a
|
// askNICbr returns lacnicInfo, what LACNIC's answer says, and logs a
|
||||||
// warning when the query fails, when NIC.br sends the reduced answer for a
|
// warning when the query fails, when NIC.br sends the reduced answer for a
|
||||||
// client over its limit, or when NIC.br's answer carries none of ASNInfo's
|
// client over its limit, or when NIC.br's answer carries none of ASNInfo's
|
||||||
// fields. The reduced answer pauses NIC.br, as a refusal does. While NIC.br
|
// fields. After the reduced answer, it returns lacnicInfo without asking
|
||||||
// is paused, askNICbr returns lacnicInfo without a warning. The ASes looked
|
// NIC.br until NICbrSkipTime has passed.
|
||||||
// up meanwhile get NIC.br's fields when they are next refreshed.
|
|
||||||
func (c *Client) askNICbr(
|
func (c *Client) askNICbr(
|
||||||
ctx context.Context, asQuery string, lacnicInfo *ASNInfo,
|
ctx context.Context, asQuery string, lacnicInfo *ASNInfo,
|
||||||
) *ASNInfo {
|
) *ASNInfo {
|
||||||
asn := lacnicInfo.ASN
|
asn := lacnicInfo.ASN
|
||||||
|
|
||||||
answer, err := c.queryRegistry(ctx, whoisServerNICbr, asQuery)
|
c.mu.Lock()
|
||||||
if errors.Is(err, ErrPaused) {
|
skipping := c.Now().Before(c.nicbrSkipUntil)
|
||||||
|
c.mu.Unlock()
|
||||||
|
|
||||||
|
if skipping {
|
||||||
return lacnicInfo
|
return lacnicInfo
|
||||||
}
|
}
|
||||||
|
|
||||||
|
answer, err := c.query(ctx, whoisServerNICbr, asQuery)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
c.logger.Warn("NIC.br query failed, keeping LACNIC's fields",
|
c.logger.Warn("NIC.br query failed, keeping LACNIC's fields",
|
||||||
"asn", asn, "error", err)
|
"asn", asn, "error", err)
|
||||||
@@ -273,11 +265,13 @@ func (c *Client) askNICbr(
|
|||||||
}
|
}
|
||||||
|
|
||||||
if strings.Contains(answer, nicbrLimitReached) {
|
if strings.Contains(answer, nicbrLimitReached) {
|
||||||
c.pause(whoisServerNICbr)
|
c.mu.Lock()
|
||||||
|
c.nicbrSkipUntil = c.Now().Add(NICbrSkipTime)
|
||||||
|
c.mu.Unlock()
|
||||||
|
|
||||||
c.logger.Warn("NIC.br answered that the client is over its query "+
|
c.logger.Warn("NIC.br answered that the client is over its query "+
|
||||||
"limit, keeping LACNIC's fields and pausing NIC.br",
|
"limit, keeping LACNIC's fields and skipping NIC.br",
|
||||||
"asn", asn, "pause_for", PauseTime)
|
"asn", asn, "skip_for", NICbrSkipTime)
|
||||||
|
|
||||||
return lacnicInfo
|
return lacnicInfo
|
||||||
}
|
}
|
||||||
@@ -315,65 +309,25 @@ func rirQuery(rir, asQuery string) string {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// queryRegistry sends query to the server of an RIR or of NIC.br, and
|
// queryRIR sends query to an RIR's server and returns the answer. An answer
|
||||||
// returns the answer. The client pauses the server for PauseTime when the
|
// holding an error line with which the server refuses the query is an
|
||||||
// server refuses the query. Each of these is a refusal:
|
// error, and the error holds that line.
|
||||||
//
|
func (c *Client) queryRIR(
|
||||||
// - a refused connection
|
|
||||||
// - a reset connection
|
|
||||||
// - an %ERROR:201: or %ERROR:202: line in the answer
|
|
||||||
//
|
|
||||||
// A refusal is returned as an error. While the server is paused,
|
|
||||||
// queryRegistry sends it nothing and returns an error that wraps ErrPaused.
|
|
||||||
func (c *Client) queryRegistry(
|
|
||||||
ctx context.Context, server, query string,
|
ctx context.Context, server, query string,
|
||||||
) (string, error) {
|
) (string, error) {
|
||||||
c.mu.Lock()
|
|
||||||
pausedUntil := c.pausedUntil[server]
|
|
||||||
c.mu.Unlock()
|
|
||||||
|
|
||||||
if c.Now().Before(pausedUntil) {
|
|
||||||
return "", fmt.Errorf("%w: %s until %s", ErrPaused, server,
|
|
||||||
pausedUntil.UTC().Format(time.RFC3339))
|
|
||||||
}
|
|
||||||
|
|
||||||
answer, err := c.query(ctx, server, query)
|
answer, err := c.query(ctx, server, query)
|
||||||
if err == nil {
|
|
||||||
err = checkRefusal(answer)
|
|
||||||
}
|
|
||||||
|
|
||||||
if errors.Is(err, errRefused) || errors.Is(err, syscall.ECONNREFUSED) ||
|
|
||||||
errors.Is(err, syscall.ECONNRESET) {
|
|
||||||
c.pause(server)
|
|
||||||
}
|
|
||||||
|
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return "", err
|
return "", err
|
||||||
}
|
}
|
||||||
|
|
||||||
return answer, nil
|
|
||||||
}
|
|
||||||
|
|
||||||
// checkRefusal returns an error holding the first line of answer that
|
|
||||||
// starts with accessDeniedError or accessLimitError. It returns nil when
|
|
||||||
// answer has no such line.
|
|
||||||
func checkRefusal(answer string) error {
|
|
||||||
for line := range strings.SplitSeq(answer, "\n") {
|
for line := range strings.SplitSeq(answer, "\n") {
|
||||||
if strings.HasPrefix(line, accessDeniedError) ||
|
if strings.HasPrefix(line, accessDeniedError) ||
|
||||||
strings.HasPrefix(line, accessLimitError) {
|
strings.HasPrefix(line, accessLimitError) {
|
||||||
return fmt.Errorf("%w: %s", errRefused, line)
|
return "", fmt.Errorf("%w: %s", errRefused, line)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
return nil
|
return answer, nil
|
||||||
}
|
|
||||||
|
|
||||||
// pause stops queryRegistry from querying server until PauseTime has passed.
|
|
||||||
func (c *Client) pause(server string) {
|
|
||||||
c.mu.Lock()
|
|
||||||
defer c.mu.Unlock()
|
|
||||||
|
|
||||||
c.pausedUntil[server] = c.Now().Add(PauseTime)
|
|
||||||
}
|
}
|
||||||
|
|
||||||
// query performs a raw WHOIS query to the specified server and returns the
|
// query performs a raw WHOIS query to the specified server and returns the
|
||||||
|
|||||||
@@ -718,10 +718,9 @@ func TestLookupASNKeepsLACNICsAnswerWhenNICbrFails(t *testing.T) {
|
|||||||
|
|
||||||
// TestLookupASNSkipsNICbrAfterItsLimit looks up an AS that NIC.br
|
// TestLookupASNSkipsNICbrAfterItsLimit looks up an AS that NIC.br
|
||||||
// registers three times with one client, while NIC.br answers that the
|
// registers three times with one client, while NIC.br answers that the
|
||||||
// client is over its query limit. The test sets the client's clock. The
|
// client is over its query limit. The first lookup asks NIC.br. The
|
||||||
// first lookup asks NIC.br. The next comes a second before PauseTime has
|
// second, a second before NICbrSkipTime has passed, does not. The third,
|
||||||
// passed, and does not ask NIC.br. The last comes at PauseTime and asks
|
// once NICbrSkipTime has passed, asks again. Each keeps LACNIC's fields.
|
||||||
// NIC.br again. Each keeps LACNIC's fields.
|
|
||||||
func TestLookupASNSkipsNICbrAfterItsLimit(t *testing.T) {
|
func TestLookupASNSkipsNICbrAfterItsLimit(t *testing.T) {
|
||||||
t.Parallel()
|
t.Parallel()
|
||||||
|
|
||||||
@@ -741,8 +740,8 @@ func TestLookupASNSkipsNICbrAfterItsLimit(t *testing.T) {
|
|||||||
wantAsks int
|
wantAsks int
|
||||||
}{
|
}{
|
||||||
{after: 0, wantAsks: 1},
|
{after: 0, wantAsks: 1},
|
||||||
{after: whois.PauseTime - time.Second, wantAsks: 1},
|
{after: whois.NICbrSkipTime - time.Second, wantAsks: 1},
|
||||||
{after: whois.PauseTime, wantAsks: 2},
|
{after: whois.NICbrSkipTime, wantAsks: 2},
|
||||||
} {
|
} {
|
||||||
now = start.Add(step.after)
|
now = start.Add(step.after)
|
||||||
|
|
||||||
@@ -786,142 +785,6 @@ func TestLookupASNReportsARefusal(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// resetServer starts a TCP server on the loopback address and returns the
|
|
||||||
// server's address. The server reads one query on each connection, then
|
|
||||||
// resets the connection.
|
|
||||||
func resetServer(t *testing.T) string {
|
|
||||||
t.Helper()
|
|
||||||
|
|
||||||
var listenConfig net.ListenConfig
|
|
||||||
|
|
||||||
listener, err := listenConfig.Listen(t.Context(), "tcp", "127.0.0.1:0")
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("listen: %v", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
t.Cleanup(func() { _ = listener.Close() })
|
|
||||||
|
|
||||||
go func() {
|
|
||||||
for {
|
|
||||||
conn, err := listener.Accept()
|
|
||||||
if err != nil {
|
|
||||||
return
|
|
||||||
}
|
|
||||||
|
|
||||||
_, _ = bufio.NewReader(conn).ReadString('\n')
|
|
||||||
|
|
||||||
// Closing a connection whose linger time is zero resets it.
|
|
||||||
if tcpConn, ok := conn.(*net.TCPConn); ok {
|
|
||||||
_ = tcpConn.SetLinger(0)
|
|
||||||
}
|
|
||||||
|
|
||||||
_ = conn.Close()
|
|
||||||
}
|
|
||||||
}()
|
|
||||||
|
|
||||||
return listener.Addr().String()
|
|
||||||
}
|
|
||||||
|
|
||||||
// newRIPEClient returns a client whose fake servers send answers. The
|
|
||||||
// client adds one to *ripeAsks each time it connects to RIPE's server. When
|
|
||||||
// reset is true, RIPE's server resets the connection instead of answering.
|
|
||||||
func newRIPEClient(
|
|
||||||
t *testing.T, answers map[string]string, reset bool, ripeAsks *int,
|
|
||||||
) *whois.Client {
|
|
||||||
t.Helper()
|
|
||||||
|
|
||||||
dial := fakeServers(t, answers)
|
|
||||||
resetAddress := resetServer(t)
|
|
||||||
|
|
||||||
client := whois.NewClient(slog.New(slog.DiscardHandler))
|
|
||||||
client.DialContext = func(
|
|
||||||
ctx context.Context, network, address string,
|
|
||||||
) (net.Conn, error) {
|
|
||||||
if address == ripeServer {
|
|
||||||
*ripeAsks++
|
|
||||||
}
|
|
||||||
|
|
||||||
if address == ripeServer && reset {
|
|
||||||
var dialer net.Dialer
|
|
||||||
|
|
||||||
return dialer.DialContext(ctx, network, resetAddress)
|
|
||||||
}
|
|
||||||
|
|
||||||
return dial(ctx, network, address)
|
|
||||||
}
|
|
||||||
|
|
||||||
return client
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestLookupASNPausesAServerThatRefuses looks up an AS at RIPE while RIPE
|
|
||||||
// refuses the query, in each of the ways a server refuses. The test sets
|
|
||||||
// the client's clock. A lookup a second before PauseTime has passed fails
|
|
||||||
// with ErrPaused and asks RIPE nothing. A lookup of an AS at APNIC succeeds
|
|
||||||
// during the pause. A lookup at PauseTime asks RIPE again.
|
|
||||||
func TestLookupASNPausesAServerThatRefuses(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
for name, test := range map[string]struct {
|
|
||||||
// ripeAnswer is RIPE's answer. With none, RIPE's server refuses
|
|
||||||
// the connection, or resets it when reset is true.
|
|
||||||
ripeAnswer string
|
|
||||||
reset bool
|
|
||||||
}{
|
|
||||||
"access denied": {ripeAnswer: ripeRefusal},
|
|
||||||
"limit reached": {ripeAnswer: ripeLimitReached + "\n"},
|
|
||||||
"refused connection": {},
|
|
||||||
"reset connection": {reset: true},
|
|
||||||
} {
|
|
||||||
t.Run(name, func(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
answers := map[string]string{
|
|
||||||
ianaQuery: ianaReferral,
|
|
||||||
apnicIANAQuery: "refer: whois.apnic.net\n",
|
|
||||||
apnicQuery: apnicRecord,
|
|
||||||
}
|
|
||||||
if test.ripeAnswer != "" {
|
|
||||||
answers[ripeQuery] = test.ripeAnswer
|
|
||||||
}
|
|
||||||
|
|
||||||
ripeAsks := 0
|
|
||||||
client := newRIPEClient(t, answers, test.reset, &ripeAsks)
|
|
||||||
|
|
||||||
start := time.Date(2026, 11, 2, 0, 0, 0, 0, time.UTC)
|
|
||||||
now := start
|
|
||||||
client.Now = func() time.Time { return now }
|
|
||||||
|
|
||||||
_, err := client.LookupASN(t.Context(), testASN)
|
|
||||||
if err == nil || errors.Is(err, whois.ErrPaused) {
|
|
||||||
t.Fatalf("LookupASN while RIPE refuses: error %v, want a "+
|
|
||||||
"refusal", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
now = start.Add(whois.PauseTime - time.Second)
|
|
||||||
|
|
||||||
_, err = client.LookupASN(t.Context(), testASN)
|
|
||||||
if !errors.Is(err, whois.ErrPaused) || ripeAsks != 1 {
|
|
||||||
t.Errorf("LookupASN during the pause: error %v after %d "+
|
|
||||||
"queries to RIPE, want ErrPaused after 1", err, ripeAsks)
|
|
||||||
}
|
|
||||||
|
|
||||||
_, err = client.LookupASN(t.Context(), apnicASN)
|
|
||||||
if err != nil {
|
|
||||||
t.Errorf("LookupASN of an AS at APNIC during RIPE's "+
|
|
||||||
"pause: %v", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
now = start.Add(whois.PauseTime)
|
|
||||||
|
|
||||||
_, _ = client.LookupASN(t.Context(), testASN)
|
|
||||||
if ripeAsks != 2 {
|
|
||||||
t.Errorf("RIPE was asked %d times after the pause, want 2",
|
|
||||||
ripeAsks)
|
|
||||||
}
|
|
||||||
})
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestLookupASNFailsWhenTheRIRGivesNoAnswer looks up an AS IANA refers to
|
// TestLookupASNFailsWhenTheRIRGivesNoAnswer looks up an AS IANA refers to
|
||||||
// RIPE when RIPE's server refuses the connection, and when it answers only
|
// RIPE when RIPE's server refuses the connection, and when it answers only
|
||||||
// with an error. Neither is a lookup that found the AS.
|
// with an error. Neither is a lookup that found the AS.
|
||||||
|
|||||||
Reference in new issue
Block a user