Compare commits
1
Commits
next
..
10515ce46b
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
10515ce46b |
@@ -1,20 +0,0 @@
|
||||
---
|
||||
title: API Policies
|
||||
last_modified: 2026-10-27
|
||||
---
|
||||
|
||||
A value in an API response is never a string with a label or unit in it. A value
|
||||
of a native type is sent as that type, and its unit goes in the field name, for
|
||||
example `_ms`.
|
||||
|
||||
Wrong:
|
||||
|
||||
```json
|
||||
{ "execution_time": "0 ms" }
|
||||
```
|
||||
|
||||
Right:
|
||||
|
||||
```json
|
||||
{ "execution_time_ms": 0 }
|
||||
```
|
||||
@@ -9,38 +9,19 @@ table in real time. It works like this:
|
||||
|
||||
## Rationale
|
||||
|
||||
RouteWatch exists to serve our other services as an internal API for information
|
||||
about IP addresses.
|
||||
RIPE RIS Live streams the BGP updates that RIPE's route collectors receive, as
|
||||
they happen, from the moment a client connects. It reports changes, not the
|
||||
routing table they add up to, so it cannot say which AS an IP address is routed
|
||||
to now or which routes a prefix has. RouteWatch applies the feed's updates to a
|
||||
routing table that it keeps in memory, with a copy in its own SQLite database.
|
||||
It answers IP-to-AS lookups and prefix pages from that table.
|
||||
|
||||
RouteWatch follows BGP routing table updates from RIPE RIS Live. RIS Live
|
||||
streams the updates that RIPE's route collectors receive, as they happen, from
|
||||
the moment a client connects. It reports changes, not the routing table they add
|
||||
up to. RouteWatch applies those changes to a routing table that it keeps in
|
||||
memory, with a copy in its own SQLite database. From that table, RouteWatch sees
|
||||
directly:
|
||||
|
||||
- which AS each IP address is routed to
|
||||
- which announced prefix holds each IP address, and how big that prefix is
|
||||
|
||||
RouteWatch adds details of each AS from WHOIS, the public directory of the
|
||||
internet registries. The details include the organisation that holds the AS.
|
||||
RouteWatch keeps them in its database and looks them up again once they are 30
|
||||
days old. An IP lookup answers from those stored details and never waits for
|
||||
WHOIS.
|
||||
|
||||
So our own services can ask RouteWatch about an IP address, instead of each
|
||||
working it out for itself.
|
||||
|
||||
RIS Live sends no routing table when a client connects. So the routing table
|
||||
holds only the routes announced since RouteWatch first started. An IP lookup
|
||||
finds no route for an address until a prefix that holds it is announced. The
|
||||
routing table also misses these updates:
|
||||
|
||||
- those sent while RouteWatch is stopped
|
||||
- those that RouteWatch drops when it falls behind
|
||||
|
||||
A lookup or a prefix page can therefore show a route that has since been
|
||||
withdrawn.
|
||||
The feed sends no table when a client connects. So the routing table holds only
|
||||
the routes announced since RouteWatch first started. An IP-to-AS lookup finds
|
||||
nothing for an address whose routes have not been announced since then. The
|
||||
routing table also misses updates RouteWatch drops when it falls behind, and
|
||||
updates sent while it is stopped. So a lookup or a prefix page can show a route
|
||||
the feed has withdrawn.
|
||||
|
||||
## Features
|
||||
|
||||
@@ -132,13 +113,7 @@ one of its peers sends an update.
|
||||
figures. The route figures come from the routing table. `oldest_route` and
|
||||
`newest_route` are the times of the oldest and the newest kept routes.
|
||||
`origins_without_kept_route` counts, across all prefixes, the origin ASes that
|
||||
have no kept route. Other figures include these:
|
||||
- `uptime_seconds`: the number of seconds since the RIS Live feed last
|
||||
connected, or 0 when it has never connected
|
||||
- `memory_usage_bytes`: the size of the Go heap in use, in bytes
|
||||
- `whois_stats.current_interval_seconds`: the WHOIS fetcher's wait between
|
||||
two fetches, in seconds, which can include a fraction of a second
|
||||
|
||||
have no kept route.
|
||||
- `GET /.well-known/healthcheck.json` - Health check (JSON): 200 when the
|
||||
database answers and the RIS Live feed is connected, otherwise 503;
|
||||
`data.status` is `ok` or `error` and `data.checks` says which check failed
|
||||
@@ -161,10 +136,9 @@ one of its peers sends an update.
|
||||
### API v1
|
||||
|
||||
- `GET /api/v1/stats` - The `/status.json` figures plus connection, garbage
|
||||
collector, stream and handler queue statistics. `connection_duration_seconds`
|
||||
is the number of seconds the RIS Live feed has been connected, or null while
|
||||
it is not connected. `stream.route_churn_per_min` is the number of prefix
|
||||
announcements and withdrawals, from all peers, in the last full clock minute.
|
||||
collector, stream and handler queue statistics. `stream.route_churn_per_min`
|
||||
is the number of prefix announcements and withdrawals, from all peers, in the
|
||||
last full clock minute.
|
||||
- `GET /api/v1/ip/{ip}` - Looks up each address. `{ip}` is an IP address, a host
|
||||
name, or a comma-separated list of at most 10 of these. `results` has one
|
||||
entry for each address. Every entry has the same fields:
|
||||
@@ -233,16 +207,10 @@ one of its peers sends an update.
|
||||
missing, empty or only spaces answers 400.
|
||||
|
||||
A JSON answer is `{"status": "ok", "data": ...}`, or for an error
|
||||
`{"status": "error", "error": {"msg": ..., "code": ...}}`. The health check is
|
||||
the exception: its 503 answer is also `{"status": "ok", "data": ...}`, with
|
||||
`error` in `data.status`. Every JSON answer also carries an `@meta` object:
|
||||
|
||||
- `time_zone`: always `UTC`
|
||||
- `api_version`: `1`
|
||||
- `execution_time_ms`: the number of milliseconds the request took
|
||||
|
||||
Every value with a unit is a number whose field name ends in the unit, such as
|
||||
`_ms`, `_seconds` or `_bytes`.
|
||||
`{"status": "error", "error": {"msg": ..., "code": ...}}`, and carries an
|
||||
`@meta` object with the time zone, the API version and the time the request
|
||||
took. The health check is the exception: its 503 answer is also
|
||||
`{"status": "ok", "data": ...}`, with `error` in `data.status`.
|
||||
|
||||
A request that runs past its time limit answers 504: the limit is 30 seconds,
|
||||
and 4 seconds for the database reads behind `/status.json` and `/api/v1/stats`.
|
||||
@@ -252,17 +220,11 @@ originates no prefix and has no record in the database or in the AS list built
|
||||
into RouteWatch. AS 0 always answers 404, because it means no known origin. The
|
||||
AS name search never lists it. A prefix with no live route answers 404 too. The
|
||||
HTML pages give these errors as plain text. Under `/api/v1` and `/ip/`, a path
|
||||
that matches no route answers 404 with the message `Not found`. A request with
|
||||
`POST`, `PUT`, `PATCH`, `DELETE`, `OPTIONS`, `TRACE` or `CONNECT` answers 405
|
||||
with `Method not allowed` on these routes:
|
||||
|
||||
- the routes under `/api/v1` and `/ip/`
|
||||
- `/status.json`
|
||||
- `/.well-known/healthcheck.json`
|
||||
|
||||
A 500 answer says only `Internal server error`. The error for a host name that
|
||||
does not resolve says only `could not resolve host name`. The detail goes to the
|
||||
log.
|
||||
no route matches answers 404 with the message `Not found`, and `POST`, `PUT`,
|
||||
`PATCH`, `DELETE`, `OPTIONS`, `TRACE` or `CONNECT` answers 405 with
|
||||
`Method not allowed`. A 500 answer says only `Internal server error`, and a host
|
||||
name that does not resolve only `could not resolve host name`; the detail goes
|
||||
to the log.
|
||||
|
||||
Every answer carries `Strict-Transport-Security`, `Content-Security-Policy`,
|
||||
`X-Frame-Options`, `X-Content-Type-Options`, `Referrer-Policy` and
|
||||
|
||||
+22
-92
@@ -1,6 +1,6 @@
|
||||
---
|
||||
title: Repository Policies
|
||||
last_modified: 2026-10-27
|
||||
last_modified: 2026-10-18
|
||||
---
|
||||
|
||||
This document covers repository structure, tooling, and workflow standards. Code
|
||||
@@ -12,7 +12,6 @@ style conventions are in separate documents:
|
||||
- [JavaScript](https://git.eeqj.de/sneak/prompts/raw/branch/main/prompts/CODE_STYLEGUIDE_JS.md)
|
||||
- [Python](https://git.eeqj.de/sneak/prompts/raw/branch/main/prompts/CODE_STYLEGUIDE_PYTHON.md)
|
||||
- [Go HTTP Server Conventions](https://git.eeqj.de/sneak/prompts/raw/branch/main/prompts/GO_HTTP_SERVER_CONVENTIONS.md)
|
||||
- [API Policies](https://git.eeqj.de/sneak/prompts/raw/branch/main/prompts/API_POLICIES.md)
|
||||
|
||||
---
|
||||
|
||||
@@ -175,32 +174,27 @@ style conventions are in separate documents:
|
||||
`script/cibuild`. `script/lint`, `script/test`, `script/cibuild` and
|
||||
`script/docker` are the four, and there is no fifth — `script/check` runs the
|
||||
two gate phases and `script/fmt-check`, and builds no image of its own. In
|
||||
every stage of the `Dockerfile`, every step that lints, tests or compiles
|
||||
comes after `ARG CACHEBUST`. `ARG CACHEBUST` comes right after these steps:
|
||||
- the steps that install system packages
|
||||
- the steps that download Go's modules
|
||||
- the step that runs `script/bootstrap` to install a Python `test` phase's
|
||||
packages from `Pipfile.lock`
|
||||
|
||||
A stage with none of these steps declares `ARG CACHEBUST` right after
|
||||
`FROM`. A step that installs anything else as well, such as a
|
||||
`script/bootstrap` that also installs node packages, comes after it too. A
|
||||
new value reruns every `RUN` step after that line; `COPY` steps are compared
|
||||
by content and need nothing. Docker keeps the layers above the line until
|
||||
the base image, their own lines or the files they copy change. A stage
|
||||
without the `ARG` line is served from cache on an unchanged tree, so a
|
||||
repository re-vendors the scripts and adds the `ARG` lines in one commit.
|
||||
The scripts do not pass `--no-cache`: it reruns every step too, but it also
|
||||
builds the layers above the line again and empties every cache mount, so
|
||||
Go's build cache and golangci-lint's, which the canonical Go `Dockerfile`
|
||||
below mounts so that they survive between check builds, would start empty
|
||||
every time. A bare `docker build .` is not evidence that anything ran: a
|
||||
sub-second build reporting success is a cache hit, not a result. Never
|
||||
invalidate by pruning — `docker builder prune` and friends destroy a build
|
||||
cache shared with every other build on the host. When a check is added or
|
||||
changed, prove it works by planting a defect it must catch and watching the
|
||||
run fail on it, then revert the defect. A green run alone shows neither that
|
||||
the check ran nor that it covers what it should.
|
||||
every stage of the `Dockerfile`, `ARG CACHEBUST` comes right after the steps
|
||||
that install system packages and download Go's modules, or right after `FROM`
|
||||
in a stage with neither, and every step that lints, tests or compiles comes
|
||||
after it. A step that installs anything else as well, such as a
|
||||
`script/bootstrap` that also installs node packages, comes after it too. A new
|
||||
value reruns every `RUN` step after that line; `COPY` steps are compared by
|
||||
content and need nothing. Docker keeps the layers above the line until the
|
||||
base image, their own lines or the files they copy change. A stage without the
|
||||
`ARG` line is served from cache on an unchanged tree, so a repository
|
||||
re-vendors the scripts and adds the `ARG` lines in one commit. The scripts do
|
||||
not pass `--no-cache`: it reruns every step too, but it also builds the layers
|
||||
above the line again and empties every cache mount, so Go's build cache and
|
||||
golangci-lint's, which the canonical Go `Dockerfile` below mounts so that they
|
||||
survive between check builds, would start empty every time. A bare
|
||||
`docker build .` is not evidence that anything ran: a sub-second build
|
||||
reporting success is a cache hit, not a result. Never invalidate by pruning —
|
||||
`docker builder prune` and friends destroy a build cache shared with every
|
||||
other build on the host. When a check is added or changed, prove it works by
|
||||
planting a defect it must catch and watching the run fail on it, then revert
|
||||
the defect. A green run alone shows neither that the check ran nor that it
|
||||
covers what it should.
|
||||
|
||||
- **The gate phases are separate stages, and the build stage depends on both.**
|
||||
The lint phase is based on the `golangci/golangci-lint` image (pinned by
|
||||
@@ -315,9 +309,6 @@ style conventions are in separate documents:
|
||||
a package's findings only when the package's files and those of every
|
||||
package it imports, `go.mod`, the linters and their settings in
|
||||
`.golangci.yml`, and the golangci-lint version are all unchanged.
|
||||
- In a Python `test` phase, the layer of the step that runs
|
||||
`script/bootstrap` to install the packages from `Pipfile.lock`, as
|
||||
described for that phase below.
|
||||
- If the project uses `//go:embed` directives that reference build artifacts
|
||||
(e.g. a web frontend compiled in a separate stage), the lint phase must
|
||||
create placeholder files so the embed directives resolve. Example:
|
||||
@@ -372,67 +363,6 @@ style conventions are in separate documents:
|
||||
`--build-arg VERSION=...`, as `script/docker` and `script/cibuild` already
|
||||
do.
|
||||
|
||||
- **A Python `test` phase that installs its packages from `Pipfile.lock` does it
|
||||
above `ARG CACHEBUST`, so check builds keep them.** The steps are
|
||||
`COPY script/ script/`, `COPY Pipfile Pipfile.lock ./` and
|
||||
`RUN script/bootstrap`, with `ARG CACHEBUST` right after them. Docker keeps
|
||||
what that `RUN` installed until the base image, `script/`, `Pipfile`,
|
||||
`Pipfile.lock` or a line up to and including the `RUN` changes. The tests run
|
||||
after `ARG CACHEBUST`, on every check build. In every other stage, a step that
|
||||
installs Python packages comes after `ARG CACHEBUST`.
|
||||
- When `script/bootstrap` runs in that step, it installs nothing beyond
|
||||
system packages, pipenv and its dependencies, and the packages from
|
||||
`Pipfile.lock`. It installs no Python there: the tests use the base
|
||||
image's Python, which the image's hash pins. If it installs anything else
|
||||
there, the step goes after `ARG CACHEBUST`.
|
||||
- Every external reference must be pinned by hash, so `script/bootstrap`
|
||||
installs pipenv and its dependencies by hash. It does so with
|
||||
`pip install --require-hashes -r` and a requirements file in `script/`
|
||||
that lists each of them with `==` and its `--hash`. pip refuses to install
|
||||
a dependency the file does not list.
|
||||
- It then runs `pipenv install --deploy`, with `--dev` when the tests need
|
||||
the development packages. `--deploy` installs only what `Pipfile.lock`
|
||||
names, checks every package against the lockfile's hashes, and fails when
|
||||
the lockfile is out of date with `Pipfile`.
|
||||
- A kept layer holds nothing a new install would refuse: pip checked pipenv
|
||||
and its dependencies, and pipenv checked the lockfile's packages, against
|
||||
their hashes. pipenv creates its virtual environment with the `virtualenv`
|
||||
package. The packages that `virtualenv` adds come from wheels inside that
|
||||
package, so the hash pip checked for it covers them. Two things in the
|
||||
layer are not fixed by a hash. A package with no ready-built wheel for the
|
||||
image is built from source. pip downloads the build tools without a hash
|
||||
check, so a kept layer's build of such a package may differ from a new
|
||||
one. And system packages keep the versions the package manager gave when
|
||||
the layer was built; a new install may get newer ones.
|
||||
- As with Go's modules, the packages are kept in a layer, not a cache mount.
|
||||
A cache mount is shared by every build on the host, and nothing checks its
|
||||
files again.
|
||||
|
||||
In this example, `script/bootstrap` finds `make` and `git` already
|
||||
installed, installs pipenv and its dependencies by hash, and runs
|
||||
`pipenv install --dev --deploy`:
|
||||
|
||||
```dockerfile
|
||||
# Test phase. The 90-second timeout covers both pytest runs.
|
||||
# python:3.x-slim, YYYY-MM-DD
|
||||
FROM python@sha256:... AS test
|
||||
# make and git are base tools that script/bootstrap installs when they
|
||||
# are missing. Installed here, they stay in their own layer when
|
||||
# script/ changes.
|
||||
RUN apt-get update \
|
||||
&& apt-get install -y --no-install-recommends make git \
|
||||
&& rm -rf /var/lib/apt/lists/*
|
||||
WORKDIR /app
|
||||
COPY script/ script/
|
||||
COPY Pipfile Pipfile.lock ./
|
||||
RUN script/bootstrap
|
||||
ARG CACHEBUST
|
||||
COPY . .
|
||||
RUN timeout 90 sh -c 'pipenv run pytest || { \
|
||||
echo "--- Rerunning with -v for details ---"; \
|
||||
pipenv run pytest -v; exit 1; }'
|
||||
```
|
||||
|
||||
- Every repo should have a Gitea Actions workflow (`.gitea/workflows/`) that
|
||||
runs `script/cibuild` on push, and checks out the repo as its only other step,
|
||||
with `persist-credentials: false`: `script/cibuild` needs no token, and
|
||||
|
||||
@@ -35,35 +35,6 @@ tested.
|
||||
|
||||
# Completed Steps
|
||||
|
||||
- 2026-10-27: the tests of `internal/routewatch` no longer fail on a loaded
|
||||
host. Five of them gave `HandleMessage` two seconds to apply a full prefix
|
||||
batch, which under the race detector can take longer. A test now waits up to
|
||||
15 seconds for each step it expects. The tests of the handlers' limits give
|
||||
the handlers small ones, so the package no longer runs past its 90-second
|
||||
limit (closes #213, #215)
|
||||
- 2026-10-27: the server works out the home page's cards every minute until it
|
||||
stops. It used to work them out only once, at startup: the end of the context
|
||||
that `Start` is given stopped the refreshes (closes #211)
|
||||
- 2026-10-27: README "Rationale" says why RouteWatch exists: to answer our other
|
||||
services' questions about IP addresses, from its own routing table and the
|
||||
WHOIS details it keeps. It still says what RIS Live does not give (closes
|
||||
#204)
|
||||
- 2026-10-27: every JSON answer carries `@meta`. A panic on a JSON route answers
|
||||
500 with the JSON error body. `/status.json` and the health check answer 405
|
||||
with the JSON error body to any method they do not take. The 500 and the 405
|
||||
used to have an empty body. The home page's IP lookup shows the whole JSON
|
||||
answer, `@meta` included. It used to show only the answer's `data`, or only
|
||||
the message of an error answer. A test checks `@meta` on every JSON route, in
|
||||
its successes and its errors (closes #201)
|
||||
- 2026-10-27: every value in a JSON answer that has a unit is a number, with the
|
||||
unit at the end of its field name. `@meta.execution_time` is now
|
||||
`execution_time_ms`. The stats' `uptime`, `connection_duration` and
|
||||
`memory_usage` are now `uptime_seconds`, `connection_duration_seconds` and
|
||||
`memory_usage_bytes`. `whois_stats.current_interval` is now
|
||||
`current_interval_seconds`. They used to be strings such as "12 ms",
|
||||
"1h18m34s" or "1.1 GB". `connection_duration_seconds` is null while the feed
|
||||
is not connected. `API_POLICIES.md` is vendored from `sneak/prompts`, and
|
||||
`REPO_POLICIES.md` re-vendored (closes #199)
|
||||
- 2026-10-27: every result of the IP lookup has the same fields. A field without
|
||||
a value is null. It used to be left out. An address with no live route has
|
||||
every route and AS field set to null, and `ptr` is an empty list when there
|
||||
|
||||
@@ -509,6 +509,8 @@ func announcementFeed(prefixes []string) string {
|
||||
func TestShutdownWritesTheUpdatesQueuedInTheStreamer(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
const maxWait = 10 * time.Second
|
||||
|
||||
prefixes := []string{"192.0.2.0/24", "198.51.100.0/24", "203.0.113.0/24"}
|
||||
feed := announcementFeed(prefixes)
|
||||
|
||||
@@ -554,9 +556,9 @@ func TestShutdownWritesTheUpdatesQueuedInTheStreamer(t *testing.T) {
|
||||
for range cap(linesSeen) {
|
||||
select {
|
||||
case <-linesSeen:
|
||||
case <-time.After(stepTimeout):
|
||||
case <-time.After(maxWait):
|
||||
t.Fatalf("the streamer did not read %d lines within %s",
|
||||
cap(linesSeen), stepTimeout)
|
||||
cap(linesSeen), maxWait)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -569,8 +571,8 @@ func TestShutdownWritesTheUpdatesQueuedInTheStreamer(t *testing.T) {
|
||||
|
||||
select {
|
||||
case <-shutDown:
|
||||
case <-time.After(stepTimeout):
|
||||
t.Fatalf("Shutdown did not return within %s", stepTimeout)
|
||||
case <-time.After(maxWait):
|
||||
t.Fatalf("Shutdown did not return within %s", maxWait)
|
||||
}
|
||||
|
||||
store.mu.Lock()
|
||||
@@ -607,16 +609,21 @@ func freePort(t *testing.T) int {
|
||||
return addr.Port
|
||||
}
|
||||
|
||||
// TestShutdownDoesNotWaitForABlockedWrite starts the HTTP server and has the
|
||||
// AS handler start a write that the store holds until the test ends, past
|
||||
// Shutdown's deadline. Shutdown must return without waiting for the write, with
|
||||
// the HTTP server stopped, and must neither checkpoint nor close the database:
|
||||
// the real one would wait for the write.
|
||||
// TestShutdownReturnsByItsDeadlineWhileAWriteIsBlocked starts the HTTP server
|
||||
// and has the AS handler start a write that the store holds until the test
|
||||
// ends, past Shutdown's deadline. Shutdown must return within its deadline
|
||||
// plus a small margin, with the HTTP server stopped, and must neither
|
||||
// checkpoint nor close the database: the real one would wait for the write.
|
||||
// Before the fix Shutdown waited for the write.
|
||||
func TestShutdownDoesNotWaitForABlockedWrite(t *testing.T) {
|
||||
func TestShutdownReturnsByItsDeadlineWhileAWriteIsBlocked(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
const deadline = 100 * time.Millisecond
|
||||
const (
|
||||
deadline = 100 * time.Millisecond
|
||||
// margin is how long Shutdown may take after its deadline, to stop
|
||||
// the HTTP server.
|
||||
margin = 2 * time.Second
|
||||
)
|
||||
|
||||
blocking := newBlockingStore()
|
||||
defer blocking.releaseWrites()
|
||||
@@ -660,9 +667,9 @@ func TestShutdownDoesNotWaitForABlockedWrite(t *testing.T) {
|
||||
|
||||
select {
|
||||
case <-shutDown:
|
||||
case <-time.After(deadline + stepTimeout):
|
||||
case <-time.After(deadline + margin):
|
||||
t.Fatalf("Shutdown did not return within %s of its deadline while "+
|
||||
"a write was blocked", stepTimeout)
|
||||
"a write was blocked", margin)
|
||||
}
|
||||
|
||||
var dialer net.Dialer
|
||||
@@ -737,6 +744,8 @@ func (s *closeOrderStore) record(call string) {
|
||||
func TestShutdownCheckpointsAndClosesTheDatabaseLast(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
const deadline = 10 * time.Second
|
||||
|
||||
store := &closeOrderStore{Store: newMockStore()}
|
||||
log := logger.New()
|
||||
cfg := &config.Config{}
|
||||
@@ -754,7 +763,7 @@ func TestShutdownCheckpointsAndClosesTheDatabaseLast(t *testing.T) {
|
||||
Path: ristypes.ASPath{Sequence: []int{testASNA}},
|
||||
})
|
||||
|
||||
ctx, cancel := context.WithTimeout(t.Context(), stepTimeout)
|
||||
ctx, cancel := context.WithTimeout(t.Context(), deadline)
|
||||
defer cancel()
|
||||
|
||||
rw.Shutdown(ctx)
|
||||
|
||||
@@ -40,13 +40,9 @@ type ASHandler struct {
|
||||
db database.Store
|
||||
logger *logger.Logger
|
||||
|
||||
// batchMaxSize is asnBatchMaxSize. Only HandleMessage reads it, so a test
|
||||
// may lower it before it hands the handler a message.
|
||||
batchMaxSize int
|
||||
|
||||
// Batching. mu guards batch and writing only: the flush goroutine takes
|
||||
// the batch under mu and writes it after releasing mu, so HandleMessage
|
||||
// waits for a database write only while batch is at batchMaxSize.
|
||||
// waits for a database write only while batch is at asnBatchMaxSize.
|
||||
mu sync.Mutex
|
||||
batch []asnOp
|
||||
// writing is how many operations the flush goroutine took from batch
|
||||
@@ -83,13 +79,12 @@ func newASHandler(
|
||||
db database.Store, logger *logger.Logger, batchTimeout time.Duration,
|
||||
) *ASHandler {
|
||||
h := &ASHandler{
|
||||
db: db,
|
||||
logger: logger,
|
||||
batchMaxSize: asnBatchMaxSize,
|
||||
batch: make([]asnOp, 0, asnBatchSize),
|
||||
batchFullCh: make(chan struct{}, 1),
|
||||
stopCh: make(chan context.Context, 1),
|
||||
done: make(chan struct{}),
|
||||
db: db,
|
||||
logger: logger,
|
||||
batch: make([]asnOp, 0, asnBatchSize),
|
||||
batchFullCh: make(chan struct{}, 1),
|
||||
stopCh: make(chan context.Context, 1),
|
||||
done: make(chan struct{}),
|
||||
}
|
||||
h.batchTaken.L = &h.mu
|
||||
|
||||
@@ -132,7 +127,7 @@ func (h *ASHandler) HandleMessage(msg *ristypes.RISMessage) {
|
||||
// While the batch is at its maximum size, wait until the flush goroutine
|
||||
// takes it. Meanwhile the handler takes nothing off its queue, which
|
||||
// fills, and the streamer drops messages for it.
|
||||
for len(h.batch) >= h.batchMaxSize {
|
||||
for len(h.batch) >= asnBatchMaxSize {
|
||||
h.batchTaken.Wait()
|
||||
}
|
||||
|
||||
@@ -156,7 +151,7 @@ func (h *ASHandler) HandleMessage(msg *ristypes.RISMessage) {
|
||||
|
||||
// A full batch wakes the flush goroutine. If it is writing, the wake-up
|
||||
// waits in batchFullCh and the batch keeps growing until the write ends
|
||||
// or the batch reaches batchMaxSize.
|
||||
// or the batch reaches asnBatchMaxSize.
|
||||
if len(h.batch) >= asnBatchSize {
|
||||
select {
|
||||
case h.batchFullCh <- struct{}{}:
|
||||
|
||||
@@ -19,7 +19,7 @@ func TestASHandlerStoresTheEarliestAndLatestTimesOfANewAS(t *testing.T) {
|
||||
db := newTestDatabase(t)
|
||||
|
||||
// Built without NewASHandler's flush goroutine, so the test flushes itself.
|
||||
h := &ASHandler{db: db, logger: logger.New(), batchMaxSize: asnBatchMaxSize}
|
||||
h := &ASHandler{db: db, logger: logger.New()}
|
||||
|
||||
first := peerAnnouncement(testPeerIP, testV4Prefix)
|
||||
second := peerAnnouncement(testPeerIP, testV4Prefix)
|
||||
@@ -54,7 +54,7 @@ func TestASHandlerRecordsNothingFromAPathWithASZero(t *testing.T) {
|
||||
store := newMockStore()
|
||||
|
||||
// Built without NewASHandler's flush goroutine, so the test flushes itself.
|
||||
h := &ASHandler{db: store, logger: logger.New(), batchMaxSize: asnBatchMaxSize}
|
||||
h := &ASHandler{db: store, logger: logger.New()}
|
||||
|
||||
msg := peerAnnouncement(testPeerIP, testV4Prefix)
|
||||
msg.Path = decodePath(t, testASNA, 0, testASNB)
|
||||
|
||||
@@ -224,7 +224,7 @@ func TestASNFetcherStopReturnsByItsDeadline(t *testing.T) {
|
||||
|
||||
select {
|
||||
case <-lookupStarted:
|
||||
case <-time.After(stepTimeout):
|
||||
case <-time.After(handleMessageTimeout):
|
||||
t.Fatal("the fetcher did not start the lookup")
|
||||
}
|
||||
|
||||
@@ -240,22 +240,22 @@ func TestASNFetcherStopReturnsByItsDeadline(t *testing.T) {
|
||||
|
||||
select {
|
||||
case <-f.done:
|
||||
case <-time.After(stepTimeout):
|
||||
case <-time.After(handleMessageTimeout):
|
||||
t.Fatal("the fetcher goroutine did not return after the lookup ended")
|
||||
}
|
||||
}
|
||||
|
||||
// TestRecordKeepsOnlyTheStatsWindow records a success and a failure every
|
||||
// second, by a clock the test sets, for three times statsWindow, and never
|
||||
// second, by a clock the test sets, far longer than statsWindow, and never
|
||||
// asks for the stats. The fetcher must keep only the times within the last
|
||||
// statsWindow.
|
||||
func TestRecordKeepsOnlyTheStatsWindow(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
const (
|
||||
records = 100_000
|
||||
step = time.Second
|
||||
want = int(statsWindow / step)
|
||||
records = 3 * want
|
||||
)
|
||||
|
||||
clock := time.Now()
|
||||
|
||||
@@ -19,15 +19,9 @@ import (
|
||||
)
|
||||
|
||||
const (
|
||||
// stepTimeout is how long a test waits for a step it expects, such as
|
||||
// HandleMessage returning, a write starting or Shutdown returning, before
|
||||
// it fails. Only a step that never comes should reach it. On a loaded
|
||||
// host, under the race detector, HandleMessage takes seconds to apply a
|
||||
// full prefix batch. The helpers below bound every wait with it, Stop
|
||||
// included. A test that hangs in one of them waits out at most two of
|
||||
// these bounds, its step's and Stop's, and then fails with its own
|
||||
// message. That leaves time before go test's 90-second timeout.
|
||||
stepTimeout = 15 * time.Second
|
||||
// handleMessageTimeout is how long a test waits for HandleMessage to
|
||||
// return, or for a write to start, before it fails.
|
||||
handleMessageTimeout = 2 * time.Second
|
||||
|
||||
// stillWaitingAfter is how long a test watches a HandleMessage that must
|
||||
// wait. One that does not wait returns well within it.
|
||||
@@ -204,7 +198,7 @@ func lastLogged(t *testing.T, logs *bytes.Buffer) stopLog {
|
||||
return logged
|
||||
}
|
||||
|
||||
// returnsInTime reports whether f returns within stepTimeout.
|
||||
// returnsInTime reports whether f returns within handleMessageTimeout.
|
||||
func returnsInTime(f func()) bool {
|
||||
done := make(chan struct{})
|
||||
|
||||
@@ -216,22 +210,11 @@ func returnsInTime(f func()) bool {
|
||||
select {
|
||||
case <-done:
|
||||
return true
|
||||
case <-time.After(stepTimeout):
|
||||
case <-time.After(handleMessageTimeout):
|
||||
return false
|
||||
}
|
||||
}
|
||||
|
||||
// stopInTime stops h, with no deadline, and fails the test if Stop does not
|
||||
// return within stepTimeout. It only marks the test failed, so it can be
|
||||
// deferred.
|
||||
func stopInTime(t *testing.T, h interface{ Stop(ctx context.Context) }) {
|
||||
t.Helper()
|
||||
|
||||
if !returnsInTime(func() { h.Stop(t.Context()) }) {
|
||||
t.Errorf("Stop did not return within %s", stepTimeout)
|
||||
}
|
||||
}
|
||||
|
||||
// startBlockedWrite hands h the messages in fill and waits until h starts to
|
||||
// write them to store. The prefix, AS and peer handlers start when fill fills
|
||||
// their batch, and the peering handler on its timer. store holds the write
|
||||
@@ -257,7 +240,7 @@ func startBlockedWrite(
|
||||
|
||||
select {
|
||||
case <-store.writeStarted:
|
||||
case <-time.After(stepTimeout):
|
||||
case <-time.After(handleMessageTimeout):
|
||||
t.Fatal("the handler did not start to write the messages")
|
||||
}
|
||||
}
|
||||
@@ -278,7 +261,7 @@ func handleWhileAWriteIsBlocked(
|
||||
) {
|
||||
t.Helper()
|
||||
|
||||
defer stopInTime(t, h)
|
||||
defer h.Stop(t.Context())
|
||||
defer store.releaseWrites()
|
||||
|
||||
startBlockedWrite(t, store, h, fill)
|
||||
@@ -292,9 +275,7 @@ func handleWhileAWriteIsBlocked(
|
||||
// while store holds the write, hands h the messages in toMax, which bring its
|
||||
// next batch to its maximum size, and the message extra. HandleMessage for
|
||||
// extra must wait until the test lets the write finish, and return after. Then
|
||||
// it stops h, which writes what it holds. The tests lower h's maximum batch
|
||||
// size to one batch, so toMax is one batch: the real maximum of ten batches is
|
||||
// slow to reach under the race detector.
|
||||
// it stops h, which writes what it holds.
|
||||
func handleAtMaxBatchSize(
|
||||
t *testing.T,
|
||||
store *blockingStore,
|
||||
@@ -307,18 +288,13 @@ func handleAtMaxBatchSize(
|
||||
) {
|
||||
t.Helper()
|
||||
|
||||
defer stopInTime(t, h)
|
||||
defer h.Stop(t.Context())
|
||||
defer store.releaseWrites()
|
||||
|
||||
startBlockedWrite(t, store, h, fill)
|
||||
|
||||
if !returnsInTime(func() {
|
||||
for _, msg := range toMax {
|
||||
h.HandleMessage(msg)
|
||||
}
|
||||
}) {
|
||||
t.Fatal("HandleMessage did not return while the batch grew to its " +
|
||||
"maximum size")
|
||||
for _, msg := range toMax {
|
||||
h.HandleMessage(msg)
|
||||
}
|
||||
|
||||
returned := make(chan struct{})
|
||||
@@ -339,7 +315,7 @@ func handleAtMaxBatchSize(
|
||||
|
||||
select {
|
||||
case <-returned:
|
||||
case <-time.After(stepTimeout):
|
||||
case <-time.After(handleMessageTimeout):
|
||||
t.Fatal("HandleMessage did not return after the database write finished")
|
||||
}
|
||||
}
|
||||
@@ -369,12 +345,8 @@ func stopWhileAWriteIsBlocked(
|
||||
|
||||
startBlockedWrite(t, store, h, fill)
|
||||
|
||||
if !returnsInTime(func() {
|
||||
for _, msg := range extra {
|
||||
h.HandleMessage(msg)
|
||||
}
|
||||
}) {
|
||||
t.Fatal("HandleMessage did not return while a database write was blocked")
|
||||
for _, msg := range extra {
|
||||
h.HandleMessage(msg)
|
||||
}
|
||||
|
||||
ctx, cancel := context.WithTimeout(t.Context(), stopDeadline)
|
||||
@@ -389,7 +361,7 @@ func stopWhileAWriteIsBlocked(
|
||||
|
||||
select {
|
||||
case <-done:
|
||||
case <-time.After(stepTimeout):
|
||||
case <-time.After(handleMessageTimeout):
|
||||
t.Fatal("the flush goroutine did not return after the write finished")
|
||||
}
|
||||
}
|
||||
@@ -408,8 +380,7 @@ func TestPrefixHandlerSkipsItsLastWriteAfterTheStopDeadline(t *testing.T) {
|
||||
|
||||
store := newBlockingStore()
|
||||
h := newPrefixHandler(store, newErrorLogger(&logs),
|
||||
routetable.New(routetable.Params{}), longerThanAnyTest,
|
||||
maxTableChangesPerWrite)
|
||||
routetable.New(routetable.Params{}), longerThanAnyTest)
|
||||
fill := slices.Repeat([]string{testV4Prefix}, prefixBatchSize)
|
||||
stopWhileAWriteIsBlocked(t, store, h, h.done,
|
||||
[]*ristypes.RISMessage{prefixAnnouncement(fill)},
|
||||
@@ -626,7 +597,7 @@ func TestPeeringHandlerStopCountsTheWriteInProgress(t *testing.T) {
|
||||
|
||||
select {
|
||||
case <-h.done:
|
||||
case <-time.After(stepTimeout):
|
||||
case <-time.After(handleMessageTimeout):
|
||||
t.Fatal("the processing goroutine did not return after the write " +
|
||||
"finished")
|
||||
}
|
||||
@@ -647,7 +618,7 @@ func TestPeeringHandlerStartsNoWriteAfterTheStopDeadline(t *testing.T) {
|
||||
const processInterval = 10 * time.Millisecond
|
||||
|
||||
store := newBlockingStore()
|
||||
h := newPeeringHandler(store, logger.New(), processInterval, maxTrackedPaths)
|
||||
h := newPeeringHandler(store, logger.New(), processInterval)
|
||||
stopWhileAWriteIsBlocked(t, store, h, h.done,
|
||||
[]*ristypes.RISMessage{
|
||||
{Path: ristypes.ASPath{Sequence: []int{testASNA, testASNB}}},
|
||||
@@ -743,11 +714,9 @@ func TestPrefixHandlerWaitsAtTheMaximumBatchSize(t *testing.T) {
|
||||
const extraPrefix = "198.51.100.0/24"
|
||||
|
||||
store := newBlockingStore()
|
||||
h := newPrefixHandler(store, logger.New(), routetable.New(routetable.Params{}),
|
||||
longerThanAnyTest, maxTableChangesPerWrite)
|
||||
h.batchMaxSize = prefixBatchSize
|
||||
h := NewPrefixHandler(store, logger.New(), routetable.New(routetable.Params{}))
|
||||
fill := slices.Repeat([]string{testV4Prefix}, prefixBatchSize)
|
||||
toMax := slices.Repeat([]string{testV4Prefix}, prefixBatchSize)
|
||||
toMax := slices.Repeat([]string{testV4Prefix}, prefixBatchMaxSize)
|
||||
handleAtMaxBatchSize(t, store, h,
|
||||
[]*ristypes.RISMessage{prefixAnnouncement(fill)},
|
||||
[]*ristypes.RISMessage{prefixAnnouncement(toMax)},
|
||||
@@ -767,9 +736,8 @@ func TestASHandlerWaitsAtTheMaximumBatchSize(t *testing.T) {
|
||||
|
||||
store := newBlockingStore()
|
||||
h := newASHandler(store, logger.New(), longerThanAnyTest)
|
||||
h.batchMaxSize = asnBatchSize
|
||||
fill := ristypes.ASPath{Sequence: slices.Repeat([]int{testASNA}, asnBatchSize)}
|
||||
toMax := ristypes.ASPath{Sequence: slices.Repeat([]int{testASNA}, asnBatchSize)}
|
||||
toMax := ristypes.ASPath{Sequence: slices.Repeat([]int{testASNA}, asnBatchMaxSize)}
|
||||
handleAtMaxBatchSize(t, store, h,
|
||||
[]*ristypes.RISMessage{{Path: fill}},
|
||||
[]*ristypes.RISMessage{{Path: toMax}},
|
||||
@@ -787,7 +755,7 @@ func TestASHandlerWaitsAtTheMaximumBatchSize(t *testing.T) {
|
||||
// TestPeerHandlerWaitsAtTheMaximumBatchSize must not have its flush timer fire.
|
||||
// A timer flush during the fill would take only part of it. The batch would
|
||||
// then reach its maximum size before the test's last message, and the test
|
||||
// would fail.
|
||||
// would hang.
|
||||
func TestPeerHandlerWaitsAtTheMaximumBatchSize(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
@@ -795,11 +763,10 @@ func TestPeerHandlerWaitsAtTheMaximumBatchSize(t *testing.T) {
|
||||
|
||||
store := newBlockingStore()
|
||||
h := newPeerHandler(store, logger.New(), longerThanAnyTest)
|
||||
h.batchMaxSize = peerBatchSize
|
||||
message := []*ristypes.RISMessage{{Peer: testPeerIP}}
|
||||
handleAtMaxBatchSize(t, store, h,
|
||||
slices.Repeat(message, peerBatchSize),
|
||||
slices.Repeat(message, peerBatchSize),
|
||||
slices.Repeat(message, peerBatchMaxSize),
|
||||
&ristypes.RISMessage{Peer: otherPeerIP})
|
||||
|
||||
store.mu.Lock()
|
||||
@@ -829,9 +796,8 @@ func TestHandlersWakeTheFlushGoroutineWhenTheBatchIsFull(t *testing.T) {
|
||||
batchSize: prefixBatchSize,
|
||||
handle: func(n int) chan struct{} {
|
||||
h := &PrefixHandler{
|
||||
batchMaxSize: prefixBatchMaxSize,
|
||||
batchFullCh: make(chan struct{}, 1),
|
||||
table: routetable.New(routetable.Params{}),
|
||||
batchFullCh: make(chan struct{}, 1),
|
||||
table: routetable.New(routetable.Params{}),
|
||||
}
|
||||
h.HandleMessage(prefixAnnouncement(
|
||||
slices.Repeat([]string{testV4Prefix}, n)))
|
||||
@@ -843,10 +809,7 @@ func TestHandlersWakeTheFlushGoroutineWhenTheBatchIsFull(t *testing.T) {
|
||||
name: "AS",
|
||||
batchSize: asnBatchSize,
|
||||
handle: func(n int) chan struct{} {
|
||||
h := &ASHandler{
|
||||
batchMaxSize: asnBatchMaxSize,
|
||||
batchFullCh: make(chan struct{}, 1),
|
||||
}
|
||||
h := &ASHandler{batchFullCh: make(chan struct{}, 1)}
|
||||
// The origin AS and each AS in the path are one update each.
|
||||
h.HandleMessage(&ristypes.RISMessage{
|
||||
Path: ristypes.ASPath{Sequence: slices.Repeat([]int{testASNA}, n-1)},
|
||||
@@ -859,10 +822,7 @@ func TestHandlersWakeTheFlushGoroutineWhenTheBatchIsFull(t *testing.T) {
|
||||
name: "peer",
|
||||
batchSize: peerBatchSize,
|
||||
handle: func(n int) chan struct{} {
|
||||
h := &PeerHandler{
|
||||
batchMaxSize: peerBatchMaxSize,
|
||||
batchFullCh: make(chan struct{}, 1),
|
||||
}
|
||||
h := &PeerHandler{batchFullCh: make(chan struct{}, 1)}
|
||||
for range n {
|
||||
h.HandleMessage(&ristypes.RISMessage{Peer: testPeerIP})
|
||||
}
|
||||
|
||||
@@ -97,7 +97,7 @@ func TestDBMaintainerStopReturnsByItsDeadline(t *testing.T) {
|
||||
|
||||
select {
|
||||
case <-store.started:
|
||||
case <-time.After(stepTimeout):
|
||||
case <-time.After(handleMessageTimeout):
|
||||
t.Fatal("the maintainer did not start the checkpoint")
|
||||
}
|
||||
|
||||
@@ -113,7 +113,7 @@ func TestDBMaintainerStopReturnsByItsDeadline(t *testing.T) {
|
||||
|
||||
select {
|
||||
case <-m.done:
|
||||
case <-time.After(stepTimeout):
|
||||
case <-time.After(handleMessageTimeout):
|
||||
t.Fatal("the maintenance goroutine did not return after the " +
|
||||
"checkpoint ended")
|
||||
}
|
||||
|
||||
@@ -43,14 +43,10 @@ type PeerHandler struct {
|
||||
db database.Store
|
||||
logger *logger.Logger
|
||||
|
||||
// batchMaxSize is peerBatchMaxSize. Only HandleMessage reads it, so a test
|
||||
// may lower it before it hands the handler a message.
|
||||
batchMaxSize int
|
||||
|
||||
// Batching. mu guards peerBatch and writing only: the flush goroutine
|
||||
// takes the batch under mu and writes it after releasing mu, so
|
||||
// HandleMessage waits for a database write only while peerBatch is at
|
||||
// batchMaxSize.
|
||||
// peerBatchMaxSize.
|
||||
mu sync.Mutex
|
||||
peerBatch []peerUpdate
|
||||
// writing is how many updates the flush goroutine took from peerBatch
|
||||
@@ -88,13 +84,12 @@ func newPeerHandler(
|
||||
db database.Store, logger *logger.Logger, batchTimeout time.Duration,
|
||||
) *PeerHandler {
|
||||
h := &PeerHandler{
|
||||
db: db,
|
||||
logger: logger,
|
||||
batchMaxSize: peerBatchMaxSize,
|
||||
peerBatch: make([]peerUpdate, 0, peerBatchSize),
|
||||
batchFullCh: make(chan struct{}, 1),
|
||||
stopCh: make(chan context.Context, 1),
|
||||
done: make(chan struct{}),
|
||||
db: db,
|
||||
logger: logger,
|
||||
peerBatch: make([]peerUpdate, 0, peerBatchSize),
|
||||
batchFullCh: make(chan struct{}, 1),
|
||||
stopCh: make(chan context.Context, 1),
|
||||
done: make(chan struct{}),
|
||||
}
|
||||
h.batchTaken.L = &h.mu
|
||||
|
||||
@@ -140,7 +135,7 @@ func (h *PeerHandler) HandleMessage(msg *ristypes.RISMessage) {
|
||||
// While the batch is at its maximum size, wait until the flush goroutine
|
||||
// takes it. Meanwhile the handler takes nothing off its queue, which
|
||||
// fills, and the streamer drops messages for it.
|
||||
for len(h.peerBatch) >= h.batchMaxSize {
|
||||
for len(h.peerBatch) >= peerBatchMaxSize {
|
||||
h.batchTaken.Wait()
|
||||
}
|
||||
|
||||
@@ -154,7 +149,7 @@ func (h *PeerHandler) HandleMessage(msg *ristypes.RISMessage) {
|
||||
|
||||
// A full batch wakes the flush goroutine. If it is writing, the wake-up
|
||||
// waits in batchFullCh and the batch keeps growing until the write ends
|
||||
// or the batch reaches batchMaxSize.
|
||||
// or the batch reaches peerBatchMaxSize.
|
||||
if len(h.peerBatch) >= peerBatchSize {
|
||||
select {
|
||||
case h.batchFullCh <- struct{}{}:
|
||||
|
||||
@@ -35,7 +35,7 @@ func TestPeerHandlerWritesTheEarliestAndLatestTimesOfANewPeer(t *testing.T) {
|
||||
|
||||
// Built without NewPeerHandler's flush goroutine, so the test flushes
|
||||
// itself.
|
||||
h := &PeerHandler{db: store, logger: logger.New(), batchMaxSize: peerBatchMaxSize}
|
||||
h := &PeerHandler{db: store, logger: logger.New()}
|
||||
|
||||
first := peerAnnouncement(testPeerIP, testV4Prefix)
|
||||
second := peerAnnouncement(testPeerIP, testV4Prefix)
|
||||
|
||||
@@ -42,10 +42,6 @@ type PeeringHandler struct {
|
||||
db database.Store
|
||||
logger *logger.Logger
|
||||
|
||||
// maxPaths is the most paths asPaths holds: maxTrackedPaths, except in
|
||||
// tests that give a smaller one.
|
||||
maxPaths int
|
||||
|
||||
// In-memory AS path tracking: each path, JSON-encoded, with the earliest
|
||||
// and the latest time an update carried it.
|
||||
mu sync.Mutex
|
||||
@@ -66,24 +62,21 @@ type PeeringHandler struct {
|
||||
// periodically processes accumulated AS paths into peering records. The
|
||||
// handler begins processing immediately upon creation.
|
||||
func NewPeeringHandler(db database.Store, logger *logger.Logger) *PeeringHandler {
|
||||
return newPeeringHandler(db, logger, peeringProcessInterval, maxTrackedPaths)
|
||||
return newPeeringHandler(db, logger, peeringProcessInterval)
|
||||
}
|
||||
|
||||
// newPeeringHandler is NewPeeringHandler with the processing timer's interval,
|
||||
// processInterval, in place of peeringProcessInterval, and the most paths it
|
||||
// holds, maxPaths, in place of maxTrackedPaths. A test gives a short interval to
|
||||
// have the timer start a write, and a small maxPaths to fill the map quickly.
|
||||
// processInterval, in place of peeringProcessInterval. A test gives a short
|
||||
// one to have the timer start a write.
|
||||
func newPeeringHandler(
|
||||
db database.Store, logger *logger.Logger, processInterval time.Duration,
|
||||
maxPaths int,
|
||||
) *PeeringHandler {
|
||||
h := &PeeringHandler{
|
||||
db: db,
|
||||
logger: logger,
|
||||
asPaths: make(map[string]database.SeenTimes),
|
||||
maxPaths: maxPaths,
|
||||
stopCh: make(chan context.Context, 1),
|
||||
done: make(chan struct{}),
|
||||
db: db,
|
||||
logger: logger,
|
||||
asPaths: make(map[string]database.SeenTimes),
|
||||
stopCh: make(chan context.Context, 1),
|
||||
done: make(chan struct{}),
|
||||
}
|
||||
|
||||
// Start the periodic processing goroutine
|
||||
@@ -138,7 +131,7 @@ func (h *PeeringHandler) HandleMessage(msg *ristypes.RISMessage) {
|
||||
defer h.mu.Unlock()
|
||||
|
||||
seen, exists := h.asPaths[key]
|
||||
if !exists && len(h.asPaths) >= h.maxPaths {
|
||||
if !exists && len(h.asPaths) >= maxTrackedPaths {
|
||||
// Map is full; drop this new path and count it.
|
||||
h.droppedPaths++
|
||||
|
||||
|
||||
@@ -5,6 +5,7 @@ import (
|
||||
"fmt"
|
||||
"maps"
|
||||
"slices"
|
||||
"strconv"
|
||||
"sync"
|
||||
"testing"
|
||||
"time"
|
||||
@@ -19,10 +20,6 @@ const (
|
||||
testASNB = 64501
|
||||
testASNC = 64502
|
||||
testASND = 64503
|
||||
|
||||
// testMaxTrackedPaths is a limit of the path map that a test fills
|
||||
// quickly, in place of maxTrackedPaths.
|
||||
testMaxTrackedPaths = 10
|
||||
)
|
||||
|
||||
// recordingStore wraps mockStore to count every RecordPeering call, so a
|
||||
@@ -55,10 +52,9 @@ func (r *recordingStore) callCount() int {
|
||||
// processing goroutine, so tests drive processing explicitly.
|
||||
func newTestHandler(db database.Store) *PeeringHandler {
|
||||
return &PeeringHandler{
|
||||
db: db,
|
||||
logger: logger.New(),
|
||||
asPaths: make(map[string]database.SeenTimes),
|
||||
maxPaths: maxTrackedPaths,
|
||||
db: db,
|
||||
logger: logger.New(),
|
||||
asPaths: make(map[string]database.SeenTimes),
|
||||
}
|
||||
}
|
||||
|
||||
@@ -248,28 +244,25 @@ func TestPeeringHandlerStopWritesTheRemainingPaths(t *testing.T) {
|
||||
|
||||
// TestPeeringHandlerCapDropsAndCounts verifies that a full map drops new paths
|
||||
// and counts them, while a path already tracked is refreshed rather than
|
||||
// dropped. The handler's limit is small, and its timer does not fire.
|
||||
// dropped.
|
||||
func TestPeeringHandlerCapDropsAndCounts(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
store := &recordingStore{mockStore: newMockStore()}
|
||||
h := newPeeringHandler(store, logger.New(), longerThanAnyTest,
|
||||
testMaxTrackedPaths)
|
||||
|
||||
defer stopInTime(t, h)
|
||||
h := newTestHandler(store)
|
||||
|
||||
now := time.Now().UTC()
|
||||
|
||||
// Fill the map to exactly its limit, including one real path key so the
|
||||
// "already tracked" branch can be exercised. The other paths are of one
|
||||
// AS, so they carry no peering.
|
||||
// Fill the map to exactly maxTrackedPaths, including one real path key so
|
||||
// the "already tracked" branch can be exercised. The filler keys are never
|
||||
// processed in this test, so their contents do not matter.
|
||||
existing := pathKey(t, testASNA, testASNB)
|
||||
|
||||
h.mu.Lock()
|
||||
h.asPaths[existing] = database.SeenTimes{First: now, Last: now}
|
||||
|
||||
for asn := 1; len(h.asPaths) < testMaxTrackedPaths; asn++ {
|
||||
h.asPaths[pathKey(t, asn)] = database.SeenTimes{First: now, Last: now}
|
||||
for i := 0; len(h.asPaths) < maxTrackedPaths; i++ {
|
||||
h.asPaths[strconv.Itoa(i)] = database.SeenTimes{First: now, Last: now}
|
||||
}
|
||||
h.mu.Unlock()
|
||||
|
||||
@@ -277,9 +270,8 @@ func TestPeeringHandlerCapDropsAndCounts(t *testing.T) {
|
||||
handle(h, now.Add(time.Second), testASNA, testASNC)
|
||||
|
||||
tracked, dropped := snapshot(h)
|
||||
if tracked != testMaxTrackedPaths {
|
||||
t.Fatalf("want map size %d after drop, got %d", testMaxTrackedPaths,
|
||||
tracked)
|
||||
if tracked != maxTrackedPaths {
|
||||
t.Fatalf("want map size %d after drop, got %d", maxTrackedPaths, tracked)
|
||||
}
|
||||
|
||||
if dropped != 1 {
|
||||
@@ -291,9 +283,8 @@ func TestPeeringHandlerCapDropsAndCounts(t *testing.T) {
|
||||
handle(h, refreshed, testASNA, testASNB)
|
||||
|
||||
tracked, dropped = snapshot(h)
|
||||
if tracked != testMaxTrackedPaths {
|
||||
t.Fatalf("want map size %d after refresh, got %d", testMaxTrackedPaths,
|
||||
tracked)
|
||||
if tracked != maxTrackedPaths {
|
||||
t.Fatalf("want map size %d after refresh, got %d", maxTrackedPaths, tracked)
|
||||
}
|
||||
|
||||
if dropped != 1 {
|
||||
@@ -312,21 +303,17 @@ func TestPeeringHandlerCapDropsAndCounts(t *testing.T) {
|
||||
|
||||
// TestPeeringHandlerRunResetsTheDroppedCount fills the map, has the handler
|
||||
// drop one path, and runs the processing. The count of dropped paths is the
|
||||
// run's own, so it is 0 again after the run. The handler's limit is small, and
|
||||
// its timer does not fire.
|
||||
// run's own, so it is 0 again after the run.
|
||||
func TestPeeringHandlerRunResetsTheDroppedCount(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
h := newPeeringHandler(newMockStore(), logger.New(), longerThanAnyTest,
|
||||
testMaxTrackedPaths)
|
||||
|
||||
defer stopInTime(t, h)
|
||||
h := newTestHandler(newMockStore())
|
||||
|
||||
now := time.Now().UTC()
|
||||
|
||||
// Paths of one AS carry no peering, so the run records nothing.
|
||||
h.mu.Lock()
|
||||
for asn := 1; len(h.asPaths) < testMaxTrackedPaths; asn++ {
|
||||
for asn := 1; len(h.asPaths) < maxTrackedPaths; asn++ {
|
||||
h.asPaths[pathKey(t, asn)] = database.SeenTimes{First: now, Last: now}
|
||||
}
|
||||
h.mu.Unlock()
|
||||
|
||||
@@ -60,17 +60,10 @@ type PrefixHandler struct {
|
||||
metrics *metrics.Tracker
|
||||
table *routetable.Table
|
||||
|
||||
// batchMaxSize is prefixBatchMaxSize. Only HandleMessage reads it, so a
|
||||
// test may lower it before it hands the handler a message.
|
||||
batchMaxSize int
|
||||
// tableChangesPerWrite is maxTableChangesPerWrite, except in tests that
|
||||
// give a smaller one.
|
||||
tableChangesPerWrite int
|
||||
|
||||
// Batching. mu guards batch, writing and writingTableChanges only: the
|
||||
// flush goroutine takes them under mu and writes them after releasing
|
||||
// mu, so HandleMessage waits for a database write only while batch is at
|
||||
// batchMaxSize.
|
||||
// prefixBatchMaxSize.
|
||||
mu sync.Mutex
|
||||
batch []seenPrefix
|
||||
// writing is how many updates the flush goroutine took and has not
|
||||
@@ -126,30 +119,24 @@ func (u prefixUpdate) announcesRoute() bool {
|
||||
func NewPrefixHandler(
|
||||
db database.Store, logger *logger.Logger, table *routetable.Table,
|
||||
) *PrefixHandler {
|
||||
return newPrefixHandler(db, logger, table, prefixBatchTimeout,
|
||||
maxTableChangesPerWrite)
|
||||
return newPrefixHandler(db, logger, table, prefixBatchTimeout)
|
||||
}
|
||||
|
||||
// newPrefixHandler is NewPrefixHandler with the flush timer's interval,
|
||||
// batchTimeout, in place of prefixBatchTimeout, and tableChangesPerWrite in
|
||||
// place of maxTableChangesPerWrite. A test that the timer must not disturb
|
||||
// gives an interval longer than the test runs. A test of writes split over
|
||||
// several flushes gives a small tableChangesPerWrite, so that it exceeds it
|
||||
// with few changes.
|
||||
// batchTimeout, in place of prefixBatchTimeout. A test that the timer must not
|
||||
// disturb gives an interval longer than the test runs.
|
||||
func newPrefixHandler(
|
||||
db database.Store, logger *logger.Logger, table *routetable.Table,
|
||||
batchTimeout time.Duration, tableChangesPerWrite int,
|
||||
batchTimeout time.Duration,
|
||||
) *PrefixHandler {
|
||||
h := &PrefixHandler{
|
||||
db: db,
|
||||
logger: logger,
|
||||
table: table,
|
||||
batch: make([]seenPrefix, 0, prefixBatchSize),
|
||||
batchMaxSize: prefixBatchMaxSize,
|
||||
tableChangesPerWrite: tableChangesPerWrite,
|
||||
batchFullCh: make(chan struct{}, 1),
|
||||
stopCh: make(chan context.Context, 1),
|
||||
done: make(chan struct{}),
|
||||
db: db,
|
||||
logger: logger,
|
||||
table: table,
|
||||
batch: make([]seenPrefix, 0, prefixBatchSize),
|
||||
batchFullCh: make(chan struct{}, 1),
|
||||
stopCh: make(chan context.Context, 1),
|
||||
done: make(chan struct{}),
|
||||
}
|
||||
h.batchTaken.L = &h.mu
|
||||
|
||||
@@ -204,7 +191,7 @@ func (h *PrefixHandler) HandleMessage(msg *ristypes.RISMessage) {
|
||||
// While the batch is at its maximum size, wait until the flush goroutine
|
||||
// takes it. Meanwhile the handler takes nothing off its queue, which
|
||||
// fills, and the streamer drops messages for it.
|
||||
for len(h.batch) >= h.batchMaxSize {
|
||||
for len(h.batch) >= prefixBatchMaxSize {
|
||||
h.batchTaken.Wait()
|
||||
}
|
||||
|
||||
@@ -252,7 +239,7 @@ func (h *PrefixHandler) HandleMessage(msg *ristypes.RISMessage) {
|
||||
|
||||
// A full batch wakes the flush goroutine. If it is writing, the wake-up
|
||||
// waits in batchFullCh and the batch keeps growing until the write ends
|
||||
// or the batch reaches batchMaxSize.
|
||||
// or the batch reaches prefixBatchMaxSize.
|
||||
if len(h.batch) >= prefixBatchSize {
|
||||
select {
|
||||
case h.batchFullCh <- struct{}{}:
|
||||
@@ -263,7 +250,7 @@ func (h *PrefixHandler) HandleMessage(msg *ristypes.RISMessage) {
|
||||
|
||||
// Stop stops the flush goroutine, which makes one last flush before it returns,
|
||||
// and waits for it until ctx ends. That flush writes at most
|
||||
// tableChangesPerWrite of the routing table's changes, and the prefixes of
|
||||
// maxTableChangesPerWrite of the routing table's changes, and the prefixes of
|
||||
// the updates the handler holds. Once ctx has ended the flush goroutine skips
|
||||
// that last flush, and a write still running is lost when the process exits. A
|
||||
// failed write of the table's changes is not retried either. Stop logs how many
|
||||
@@ -391,13 +378,14 @@ func (h *PrefixHandler) applyToTable(update prefixUpdate) {
|
||||
})
|
||||
}
|
||||
|
||||
// writeTableChanges writes at most tableChangesPerWrite changes of the routing
|
||||
// table in memory, and every peer the table numbered since the last write.
|
||||
// writeTableChanges writes at most maxTableChangesPerWrite changes of the
|
||||
// routing table in memory, and every peer the table numbered since the last
|
||||
// write.
|
||||
// When the write fails, they go back to the table, so that a later write writes
|
||||
// them again, each pair with its content then.
|
||||
func (h *PrefixHandler) writeTableChanges() {
|
||||
h.mu.Lock()
|
||||
changes, newPeers := h.table.TakeChanges(h.tableChangesPerWrite)
|
||||
changes, newPeers := h.table.TakeChanges(maxTableChangesPerWrite)
|
||||
h.writingTableChanges = len(changes)
|
||||
h.mu.Unlock()
|
||||
|
||||
|
||||
@@ -237,10 +237,9 @@ func TestPrefixHandlerStopLogsNothingWhenItWroteEverything(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
// longerThanAnyTest is a timer interval that no test lasts. A test gives it to
|
||||
// newPrefixHandler, newASHandler, newPeerHandler or newPeeringHandler when the
|
||||
// handler must write only when its batch is full, when the test says, or on
|
||||
// Stop.
|
||||
// longerThanAnyTest is a flush timer interval that no test lasts. A test whose
|
||||
// prefix, AS or peer handler must flush only when its batch is full and on Stop
|
||||
// gives it to newPrefixHandler, newASHandler or newPeerHandler.
|
||||
const longerThanAnyTest = time.Hour
|
||||
|
||||
// prefixHandlerWithoutTimer returns a prefix handler that writes to store and
|
||||
@@ -248,11 +247,9 @@ const longerThanAnyTest = time.Hour
|
||||
// NewPrefixHandler's flush goroutine, so the test flushes itself.
|
||||
func prefixHandlerWithoutTimer(store database.Store) *PrefixHandler {
|
||||
return &PrefixHandler{
|
||||
db: store,
|
||||
logger: logger.New(),
|
||||
table: routetable.New(routetable.Params{}),
|
||||
batchMaxSize: prefixBatchMaxSize,
|
||||
tableChangesPerWrite: maxTableChangesPerWrite,
|
||||
db: store,
|
||||
logger: logger.New(),
|
||||
table: routetable.New(routetable.Params{}),
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@@ -30,13 +30,12 @@ const (
|
||||
|
||||
// firstManyPrefix is the first prefix announceManyPrefixes announces.
|
||||
firstManyPrefix = "10.0.0.0/32"
|
||||
|
||||
// testMaxTableChangesPerWrite is a limit of one write of the routing
|
||||
// table's changes that a test exceeds quickly, in place of
|
||||
// maxTableChangesPerWrite.
|
||||
testMaxTableChangesPerWrite = 10
|
||||
)
|
||||
|
||||
// tableWriteTimeout is how long a test waits for the flush timer to write the
|
||||
// routing table's changes: a few ticks of it.
|
||||
const tableWriteTimeout = 5 * prefixBatchTimeout
|
||||
|
||||
// tableCase is one test of the routing table in memory: the messages handed to
|
||||
// the prefix handler, and for each prefix the peers that carry it with each
|
||||
// origin AS after them.
|
||||
@@ -277,9 +276,9 @@ func TestPrefixHandlerWritesTheTableChangesEachSecond(t *testing.T) {
|
||||
t.Fatalf("wrote %+v, want the pair of %s and AS%d with "+
|
||||
"removed %t", changes, prefix, testASNB, step.wantRemoved)
|
||||
}
|
||||
case <-time.After(stepTimeout):
|
||||
case <-time.After(tableWriteTimeout):
|
||||
t.Fatalf("the routing table's change was not written within %s",
|
||||
stepTimeout)
|
||||
tableWriteTimeout)
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -376,32 +375,25 @@ func (s *tableWriteSizesStore) WriteRouteChanges(
|
||||
// takes, then flushes three times. The first write must hold as many pairs as
|
||||
// one write takes, and the peer. The second must hold the other pairs, and the
|
||||
// third flush must write nothing. The database must then hold the table in
|
||||
// memory. The handler's limit is small, and its timer does not fire.
|
||||
// memory.
|
||||
func TestPrefixHandlerWritesManyTableChangesOverSeveralFlushes(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
const (
|
||||
pairs = testMaxTableChangesPerWrite * 3 / 2
|
||||
pairs = maxTableChangesPerWrite * 3 / 2
|
||||
flushes = 3
|
||||
)
|
||||
|
||||
db := newTestDatabase(t)
|
||||
store := &tableWriteSizesStore{Database: db}
|
||||
table := routetable.New(routetable.Params{})
|
||||
announceManyPrefixes(table, pairs)
|
||||
|
||||
h := newPrefixHandler(store, logger.New(), table, longerThanAnyTest,
|
||||
testMaxTableChangesPerWrite)
|
||||
|
||||
defer stopInTime(t, h)
|
||||
h := prefixHandlerWithoutTimer(store)
|
||||
announceManyPrefixes(h.table, pairs)
|
||||
|
||||
for range flushes {
|
||||
h.flushBatch()
|
||||
}
|
||||
|
||||
wantChanges := []int{
|
||||
testMaxTableChangesPerWrite, pairs - testMaxTableChangesPerWrite,
|
||||
}
|
||||
wantChanges := []int{maxTableChangesPerWrite, pairs - maxTableChangesPerWrite}
|
||||
wantNewPeers := []int{1, 0}
|
||||
|
||||
if !slices.Equal(store.changes, wantChanges) ||
|
||||
@@ -423,11 +415,11 @@ func TestPrefixHandlerWritesManyTableChangesOverSeveralFlushes(t *testing.T) {
|
||||
// takes. Its last flush writes one write's worth, so Stop must log the pairs
|
||||
// that no write took, and must not say that the stop deadline passed. The
|
||||
// flush timer must not fire, or its writes could leave no pairs for Stop to
|
||||
// log. The handler's limit of one write is small.
|
||||
// log.
|
||||
func TestPrefixHandlerStopLogsTheTableChangesLeftUnwritten(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
const pairs = testMaxTableChangesPerWrite + 1
|
||||
const pairs = maxTableChangesPerWrite + 1
|
||||
|
||||
var logs bytes.Buffer
|
||||
|
||||
@@ -435,8 +427,7 @@ func TestPrefixHandlerStopLogsTheTableChangesLeftUnwritten(t *testing.T) {
|
||||
table := routetable.New(routetable.Params{})
|
||||
announceManyPrefixes(table, pairs)
|
||||
|
||||
h := newPrefixHandler(store, newErrorLogger(&logs), table, longerThanAnyTest,
|
||||
testMaxTableChangesPerWrite)
|
||||
h := newPrefixHandler(store, newErrorLogger(&logs), table, longerThanAnyTest)
|
||||
h.Stop(t.Context())
|
||||
|
||||
written := 0
|
||||
|
||||
+33
-41
@@ -23,6 +23,7 @@ import (
|
||||
"git.eeqj.de/sneak/routewatch/internal/streamer"
|
||||
"git.eeqj.de/sneak/routewatch/internal/templates"
|
||||
asinfo "git.eeqj.de/sneak/routewatch/pkg/asinfo"
|
||||
"github.com/dustin/go-humanize"
|
||||
"github.com/go-chi/chi/v5"
|
||||
)
|
||||
|
||||
@@ -225,23 +226,18 @@ type WHOISStatsInfo struct {
|
||||
NeverFetched int `json:"never_fetched"`
|
||||
SuccessesLastHour int `json:"successes_last_hour"`
|
||||
ErrorsLastHour int `json:"errors_last_hour"`
|
||||
CurrentInterval string `json:"current_interval"`
|
||||
ConsecutiveFails int `json:"consecutive_fails"`
|
||||
FreshPercent float64 `json:"fresh_percent"`
|
||||
|
||||
// CurrentIntervalSeconds is the wait between two fetches. It can have a
|
||||
// fractional part, such as 22.5.
|
||||
CurrentIntervalSeconds float64 `json:"current_interval_seconds"`
|
||||
}
|
||||
|
||||
// Stats is the /status.json answer: uptime, message counts, database stats,
|
||||
// and the routing table's figures. UptimeSeconds is the time since the stream
|
||||
// last connected, or 0 if it has never connected. MemoryUsageBytes is the Go
|
||||
// heap in use. OldestRoute and NewestRoute are the times of the oldest and the
|
||||
// newest kept routes.
|
||||
// and the routing table's figures. OldestRoute and NewestRoute are the times
|
||||
// of the oldest and the newest kept routes.
|
||||
//
|
||||
//nolint:tagliatelle // field names are the existing JSON API
|
||||
type Stats struct {
|
||||
UptimeSeconds int64 `json:"uptime_seconds"`
|
||||
Uptime string `json:"uptime"`
|
||||
TotalMessages uint64 `json:"total_messages"`
|
||||
TotalBytes uint64 `json:"total_bytes"`
|
||||
TotalWireBytes uint64 `json:"total_wire_bytes"`
|
||||
@@ -251,7 +247,7 @@ type Stats struct {
|
||||
Connected bool `json:"connected"`
|
||||
GoVersion string `json:"go_version"`
|
||||
Goroutines int `json:"goroutines"`
|
||||
MemoryUsageBytes uint64 `json:"memory_usage_bytes"`
|
||||
MemoryUsage string `json:"memory_usage"`
|
||||
ASNs int `json:"asns"`
|
||||
Prefixes int `json:"prefixes"`
|
||||
IPv4Prefixes int `json:"ipv4_prefixes"`
|
||||
@@ -366,7 +362,7 @@ func (s *Server) statusStats(
|
||||
runtime.ReadMemStats(&memStats)
|
||||
|
||||
return Stats{
|
||||
UptimeSeconds: uptimeSeconds(streamMetrics.ConnectedSince),
|
||||
Uptime: streamUptime(streamMetrics.ConnectedSince),
|
||||
TotalMessages: streamMetrics.TotalMessages,
|
||||
TotalBytes: streamMetrics.TotalBytes,
|
||||
TotalWireBytes: streamMetrics.TotalWireBytes,
|
||||
@@ -376,7 +372,7 @@ func (s *Server) statusStats(
|
||||
Connected: streamMetrics.Connected,
|
||||
GoVersion: runtime.Version(),
|
||||
Goroutines: runtime.NumGoroutine(),
|
||||
MemoryUsageBytes: memStats.Alloc,
|
||||
MemoryUsage: humanize.Bytes(memStats.Alloc),
|
||||
ASNs: dbStats.ASNs,
|
||||
Prefixes: counts.IPv4Prefixes + counts.IPv6Prefixes,
|
||||
IPv4Prefixes: counts.IPv4Prefixes,
|
||||
@@ -443,31 +439,30 @@ func (s *Server) getWHOISStats(ctx context.Context) *WHOISStatsInfo {
|
||||
NeverFetched: dbStats.NeverFetched,
|
||||
SuccessesLastHour: fetcherStats.SuccessesLastHour,
|
||||
ErrorsLastHour: fetcherStats.ErrorsLastHour,
|
||||
CurrentInterval: fetcherStats.CurrentInterval.String(),
|
||||
ConsecutiveFails: fetcherStats.ConsecutiveFails,
|
||||
FreshPercent: freshPercent,
|
||||
|
||||
CurrentIntervalSeconds: fetcherStats.CurrentInterval.Seconds(),
|
||||
}
|
||||
}
|
||||
|
||||
// uptimeSeconds is the number of whole seconds since the stream last
|
||||
// connected, or 0 if it has never connected.
|
||||
func uptimeSeconds(connectedSince time.Time) int64 {
|
||||
// streamUptime is how long the stream has been connected, to the second, or
|
||||
// "0s" if it has never connected.
|
||||
func streamUptime(connectedSince time.Time) string {
|
||||
if connectedSince.IsZero() {
|
||||
return 0
|
||||
return "0s"
|
||||
}
|
||||
|
||||
return int64(time.Since(connectedSince) / time.Second)
|
||||
return time.Since(connectedSince).Truncate(time.Second).String()
|
||||
}
|
||||
|
||||
// connectionDurationSeconds is the number of whole seconds the stream has
|
||||
// been connected, or nil while it is not connected.
|
||||
func connectionDurationSeconds(connected bool, connectedSince time.Time) *int64 {
|
||||
// connectionDuration is how long the stream has been connected, to the
|
||||
// second, or "disconnected".
|
||||
func connectionDuration(connected bool, connectedSince time.Time) string {
|
||||
if !connected || connectedSince.IsZero() {
|
||||
return nil
|
||||
return "disconnected"
|
||||
}
|
||||
|
||||
return new(uptimeSeconds(connectedSince))
|
||||
return time.Since(connectedSince).Truncate(time.Second).String()
|
||||
}
|
||||
|
||||
// whoisStaleThreshold matches the fetcher's threshold for consistency.
|
||||
@@ -517,7 +512,7 @@ type StreamStats struct {
|
||||
//
|
||||
//nolint:tagliatelle // field names are the existing JSON API
|
||||
type StatsResponse struct {
|
||||
UptimeSeconds int64 `json:"uptime_seconds"`
|
||||
Uptime string `json:"uptime"`
|
||||
TotalMessages uint64 `json:"total_messages"`
|
||||
TotalBytes uint64 `json:"total_bytes"`
|
||||
TotalWireBytes uint64 `json:"total_wire_bytes"`
|
||||
@@ -525,10 +520,11 @@ type StatsResponse struct {
|
||||
MbitsPerSec float64 `json:"mbits_per_sec"`
|
||||
WireMbitsPerSec float64 `json:"wire_mbits_per_sec"`
|
||||
Connected bool `json:"connected"`
|
||||
ConnectionDuration string `json:"connection_duration"`
|
||||
ReconnectCount uint64 `json:"reconnect_count"`
|
||||
GoVersion string `json:"go_version"`
|
||||
Goroutines int `json:"goroutines"`
|
||||
MemoryUsageBytes uint64 `json:"memory_usage_bytes"`
|
||||
MemoryUsage string `json:"memory_usage"`
|
||||
GC GCStats `json:"gc"`
|
||||
Stream StreamStats `json:"stream"`
|
||||
ASNs int `json:"asns"`
|
||||
@@ -550,9 +546,6 @@ type StatsResponse struct {
|
||||
IPv6PrefixDistribution []PrefixDistribution `json:"ipv6_prefix_distribution"`
|
||||
WHOISStats *WHOISStatsInfo `json:"whois_stats,omitempty"`
|
||||
|
||||
// ConnectionDurationSeconds is null while the stream is not connected.
|
||||
ConnectionDurationSeconds *int64 `json:"connection_duration_seconds"`
|
||||
|
||||
OriginsWithoutKeptRoute int `json:"origins_without_kept_route"`
|
||||
}
|
||||
|
||||
@@ -691,18 +684,20 @@ func (s *Server) statsResponse(
|
||||
runtime.ReadMemStats(&memStats)
|
||||
|
||||
return StatsResponse{
|
||||
UptimeSeconds: uptimeSeconds(streamMetrics.ConnectedSince),
|
||||
TotalMessages: streamMetrics.TotalMessages,
|
||||
TotalBytes: streamMetrics.TotalBytes,
|
||||
TotalWireBytes: streamMetrics.TotalWireBytes,
|
||||
MessagesPerSec: streamMetrics.MessagesPerSec,
|
||||
MbitsPerSec: streamMetrics.BitsPerSec / bitsPerMegabit,
|
||||
WireMbitsPerSec: streamMetrics.WireBitsPerSec / bitsPerMegabit,
|
||||
Connected: streamMetrics.Connected,
|
||||
Uptime: streamUptime(streamMetrics.ConnectedSince),
|
||||
TotalMessages: streamMetrics.TotalMessages,
|
||||
TotalBytes: streamMetrics.TotalBytes,
|
||||
TotalWireBytes: streamMetrics.TotalWireBytes,
|
||||
MessagesPerSec: streamMetrics.MessagesPerSec,
|
||||
MbitsPerSec: streamMetrics.BitsPerSec / bitsPerMegabit,
|
||||
WireMbitsPerSec: streamMetrics.WireBitsPerSec / bitsPerMegabit,
|
||||
Connected: streamMetrics.Connected,
|
||||
ConnectionDuration: connectionDuration(
|
||||
streamMetrics.Connected, streamMetrics.ConnectedSince),
|
||||
ReconnectCount: streamMetrics.ReconnectCount,
|
||||
GoVersion: runtime.Version(),
|
||||
Goroutines: runtime.NumGoroutine(),
|
||||
MemoryUsageBytes: memStats.Alloc,
|
||||
MemoryUsage: humanize.Bytes(memStats.Alloc),
|
||||
GC: gcStats(&memStats),
|
||||
Stream: s.streamStats(),
|
||||
ASNs: dbStats.ASNs,
|
||||
@@ -724,9 +719,6 @@ func (s *Server) statsResponse(
|
||||
IPv6PrefixDistribution: prefixDistribution(counts.IPv6PrefixesByLength),
|
||||
WHOISStats: whoisStats,
|
||||
|
||||
ConnectionDurationSeconds: connectionDurationSeconds(
|
||||
streamMetrics.Connected, streamMetrics.ConnectedSince),
|
||||
|
||||
OriginsWithoutKeptRoute: counts.OriginsWithoutKeptRoute,
|
||||
}
|
||||
}
|
||||
|
||||
@@ -73,8 +73,8 @@ func emptyTable() *routetable.Table {
|
||||
// Paths several tests request.
|
||||
const (
|
||||
asPagePath = "/as/64500"
|
||||
asJSONPath = "/api/v1/as/64500"
|
||||
prefixLengthPagePath = "/prefixlength/24"
|
||||
statusJSONPath = "/status.json"
|
||||
statsPath = "/api/v1/stats"
|
||||
)
|
||||
|
||||
@@ -295,167 +295,6 @@ func TestStatsAnswerCarriesChurnPerMinute(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
// fixedIntervalASNFetcher is a WHOIS fetcher that does nothing and reports
|
||||
// interval as its wait between fetches.
|
||||
type fixedIntervalASNFetcher struct {
|
||||
idleASNFetcher
|
||||
|
||||
interval time.Duration
|
||||
}
|
||||
|
||||
func (f fixedIntervalASNFetcher) GetStats() ASNFetcherStats {
|
||||
return ASNFetcherStats{CurrentInterval: f.interval}
|
||||
}
|
||||
|
||||
// unitStats holds the figures of a stats answer that carry a unit. Each is a
|
||||
// pointer, so that a figure missing from the answer is nil.
|
||||
//
|
||||
//nolint:tagliatelle // field names are the existing JSON API
|
||||
type unitStats struct {
|
||||
UptimeSeconds *int64 `json:"uptime_seconds"`
|
||||
MemoryUsageBytes *uint64 `json:"memory_usage_bytes"`
|
||||
WHOISStats struct {
|
||||
CurrentIntervalSeconds *float64 `json:"current_interval_seconds"`
|
||||
} `json:"whois_stats"`
|
||||
GC struct {
|
||||
HeapAllocBytes uint64 `json:"heap_alloc_bytes"`
|
||||
} `json:"gc"`
|
||||
}
|
||||
|
||||
// TestStatsAnswersGiveUnitsInFieldNames checks that the figures of both stats
|
||||
// answers that carry a unit are numbers, with the unit at the end of their
|
||||
// names. They used to be strings with the unit in them, such as "1h18m34s" or
|
||||
// "1.1 GB" (https://git.eeqj.de/sneak/routewatch/issues/199). The stream has
|
||||
// never connected, so the uptime is 0. The memory use in /api/v1/stats is the
|
||||
// same number of bytes as its gc.heap_alloc_bytes.
|
||||
func TestStatsAnswersGiveUnitsInFieldNames(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
const fetchInterval = 18200 * time.Millisecond
|
||||
|
||||
s := newServerWithTestRoutes(t)
|
||||
s.SetASNFetcher(fixedIntervalASNFetcher{interval: fetchInterval})
|
||||
|
||||
for _, path := range []string{statusJSONPath, statsPath} {
|
||||
answer := get(t, s, path).Body.Bytes()
|
||||
|
||||
var body struct {
|
||||
Data unitStats `json:"data"`
|
||||
}
|
||||
|
||||
err := json.Unmarshal(answer, &body)
|
||||
if err != nil {
|
||||
t.Errorf("%s: figures are not numbers: %v; answer %s", path, err, answer)
|
||||
|
||||
continue
|
||||
}
|
||||
|
||||
got := body.Data
|
||||
if got.UptimeSeconds == nil || *got.UptimeSeconds != 0 {
|
||||
t.Errorf("%s: uptime_seconds is not 0: %s", path, answer)
|
||||
}
|
||||
|
||||
if got.MemoryUsageBytes == nil || *got.MemoryUsageBytes == 0 {
|
||||
t.Errorf("%s: memory_usage_bytes is not a number of bytes: %s",
|
||||
path, answer)
|
||||
}
|
||||
|
||||
interval := got.WHOISStats.CurrentIntervalSeconds
|
||||
if interval == nil || *interval != fetchInterval.Seconds() {
|
||||
t.Errorf("%s: whois_stats.current_interval_seconds is not %v: %s",
|
||||
path, fetchInterval.Seconds(), answer)
|
||||
}
|
||||
|
||||
if path == statsPath && got.MemoryUsageBytes != nil &&
|
||||
*got.MemoryUsageBytes != got.GC.HeapAllocBytes {
|
||||
t.Errorf("%s: memory_usage_bytes %d, want gc.heap_alloc_bytes %d",
|
||||
path, *got.MemoryUsageBytes, got.GC.HeapAllocBytes)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// TestStatsAnswerGivesConnectionDurationInSeconds checks that the
|
||||
// connection_duration_seconds of /api/v1/stats is null while the stream is not
|
||||
// connected, and a whole number of seconds once it is. It used to be the
|
||||
// string "disconnected", or a Go duration such as "1h18m34s"
|
||||
// (https://git.eeqj.de/sneak/routewatch/issues/199).
|
||||
func TestStatsAnswerGivesConnectionDurationInSeconds(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
s := newServerWithTestRoutes(t)
|
||||
|
||||
connectionDuration := func() json.RawMessage {
|
||||
answer := get(t, s, statsPath).Body.Bytes()
|
||||
|
||||
var body struct {
|
||||
Data map[string]json.RawMessage `json:"data"`
|
||||
}
|
||||
|
||||
err := json.Unmarshal(answer, &body)
|
||||
if err != nil {
|
||||
t.Fatalf("decoding the answer: %v; answer %s", err, answer)
|
||||
}
|
||||
|
||||
duration, ok := body.Data["connection_duration_seconds"]
|
||||
if !ok {
|
||||
t.Fatalf("no connection_duration_seconds: %s", answer)
|
||||
}
|
||||
|
||||
return duration
|
||||
}
|
||||
|
||||
disconnected := connectionDuration()
|
||||
if string(disconnected) != "null" {
|
||||
t.Errorf("disconnected: connection_duration_seconds = %s, want null",
|
||||
disconnected)
|
||||
}
|
||||
|
||||
s.streamer.GetMetricsTracker().SetConnected(true)
|
||||
|
||||
var seconds int64
|
||||
|
||||
connected := connectionDuration()
|
||||
|
||||
err := json.Unmarshal(connected, &seconds)
|
||||
if err != nil || seconds != 0 {
|
||||
t.Errorf("just connected: connection_duration_seconds = %s, want 0",
|
||||
connected)
|
||||
}
|
||||
}
|
||||
|
||||
// TestStreamDurationsCountSeconds checks that the uptime and the connection
|
||||
// duration of a stream that connected 90 seconds ago are both 90. A stream
|
||||
// that has never connected has an uptime of 0, and one that is not connected
|
||||
// has no connection duration.
|
||||
func TestStreamDurationsCountSeconds(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
const connectedFor = 90 * time.Second
|
||||
|
||||
connectedSince := time.Now().Add(-connectedFor)
|
||||
want := int64(connectedFor / time.Second)
|
||||
|
||||
if got := uptimeSeconds(connectedSince); got != want {
|
||||
t.Errorf("uptimeSeconds = %d, want %d", got, want)
|
||||
}
|
||||
|
||||
if got := uptimeSeconds(time.Time{}); got != 0 {
|
||||
t.Errorf("uptimeSeconds before any connection = %d, want 0", got)
|
||||
}
|
||||
|
||||
switch got := connectionDurationSeconds(true, connectedSince); {
|
||||
case got == nil:
|
||||
t.Errorf("connectionDurationSeconds = nil, want %d", want)
|
||||
case *got != want:
|
||||
t.Errorf("connectionDurationSeconds = %d, want %d", *got, want)
|
||||
}
|
||||
|
||||
if got := connectionDurationSeconds(false, connectedSince); got != nil {
|
||||
t.Errorf("connectionDurationSeconds while not connected = %d, want nil",
|
||||
*got)
|
||||
}
|
||||
}
|
||||
|
||||
// newTestTable returns a routing table that holds a route for testPrefixV4,
|
||||
// with origin AS 64500, and one for testPrefixV6, with origin AS 64501.
|
||||
func newTestTable() *routetable.Table {
|
||||
@@ -927,7 +766,7 @@ func TestRoutesAnswer504WhenDatabaseReadIsCutOff(t *testing.T) {
|
||||
{path: "/ip/" + addrInTestPrefixV4, wantJSONBody: true},
|
||||
{path: statsPath, wantJSONBody: true},
|
||||
{path: "/api/v1/ip/" + addrInTestPrefixV4, wantJSONBody: true},
|
||||
{path: asJSONPath, wantJSONBody: true},
|
||||
{path: "/api/v1/as/64500", wantJSONBody: true},
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
@@ -1108,10 +947,10 @@ func TestJSONRoutesDoNotSendErrorDetails(t *testing.T) {
|
||||
path string
|
||||
wantCode int
|
||||
}{
|
||||
{healthCheckPath, http.StatusServiceUnavailable},
|
||||
{"/.well-known/healthcheck.json", http.StatusServiceUnavailable},
|
||||
{statusJSONPath, http.StatusInternalServerError},
|
||||
{statsPath, http.StatusInternalServerError},
|
||||
{asJSONPath, http.StatusInternalServerError},
|
||||
{"/api/v1/as/64500", http.StatusInternalServerError},
|
||||
{"/api/v1/ip/" + addrInTestPrefixV4, http.StatusInternalServerError},
|
||||
{"/api/v1/prefix/foo/24", http.StatusBadRequest},
|
||||
}
|
||||
|
||||
@@ -6,7 +6,6 @@ import (
|
||||
"strconv"
|
||||
"strings"
|
||||
"testing"
|
||||
"testing/synctest"
|
||||
"time"
|
||||
|
||||
"git.eeqj.de/sneak/routewatch/internal/database"
|
||||
@@ -342,66 +341,6 @@ func TestStartedServerWorksOutTheCards(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
// TestStartedServerWorksOutTheCardsUntilStop starts a server with a context
|
||||
// that ends as soon as Start returns. It runs in a synctest bubble, whose clock
|
||||
// is fake: time.Sleep there moves the clock on and returns at once. The server
|
||||
// must work the cards out again a minute later, and no more after Stop. Each
|
||||
// time it works them out, it keeps a new copy. Before the fix the end of
|
||||
// Start's context stopped the goroutine that works them out.
|
||||
func TestStartedServerWorksOutTheCardsUntilStop(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
// The server is made outside the bubble. The streamer's metrics start a
|
||||
// goroutine that never returns, and a bubble waits for each of its
|
||||
// goroutines to return.
|
||||
s := newServerWithTable(t, newCardsTable())
|
||||
|
||||
synctest.Test(t, func(t *testing.T) {
|
||||
ctx, cancel := context.WithCancel(t.Context())
|
||||
|
||||
err := s.Start(ctx)
|
||||
if err != nil {
|
||||
t.Fatalf("Start: %v", err)
|
||||
}
|
||||
|
||||
cancel()
|
||||
|
||||
// The bubble's clock moves only while each of its goroutines waits
|
||||
// on the bubble, and the goroutine that serves HTTP waits on the
|
||||
// network. The test sends no request, so it closes the HTTP server.
|
||||
_ = s.srv.Close()
|
||||
|
||||
synctest.Wait()
|
||||
|
||||
started := s.cards.Load()
|
||||
if started == nil {
|
||||
t.Fatal("the server did not work out the cards when it started")
|
||||
}
|
||||
|
||||
time.Sleep(homeCardsRefreshInterval)
|
||||
synctest.Wait()
|
||||
|
||||
if s.cards.Load() == started {
|
||||
t.Error("the server did not work out the cards again a minute " +
|
||||
"after its start context ended")
|
||||
}
|
||||
|
||||
err = s.Stop(t.Context())
|
||||
if err != nil {
|
||||
t.Fatalf("Stop: %v", err)
|
||||
}
|
||||
|
||||
stopped := s.cards.Load()
|
||||
|
||||
time.Sleep(homeCardsRefreshInterval)
|
||||
synctest.Wait()
|
||||
|
||||
if s.cards.Load() != stopped {
|
||||
t.Error("the server worked out the cards again after Stop")
|
||||
}
|
||||
})
|
||||
}
|
||||
|
||||
// waitUntil fails the test unless done reports true within maxWait.
|
||||
func waitUntil(t *testing.T, maxWait time.Duration, done func() bool) {
|
||||
t.Helper()
|
||||
|
||||
@@ -16,7 +16,6 @@ import (
|
||||
"time"
|
||||
|
||||
"github.com/go-chi/chi/v5"
|
||||
"github.com/go-chi/chi/v5/middleware"
|
||||
)
|
||||
|
||||
// responseWriter wraps http.ResponseWriter to capture the response
|
||||
@@ -54,15 +53,6 @@ func (rw *responseWriter) Header() http.Header {
|
||||
return rw.ResponseWriter.Header()
|
||||
}
|
||||
|
||||
// The @meta object JSONResponseMiddleware adds to a JSON answer: its key, and
|
||||
// the keys of its fields.
|
||||
const (
|
||||
jsonKeyMeta = "@meta"
|
||||
metaKeyTimeZone = "time_zone"
|
||||
metaKeyAPIVersion = "api_version"
|
||||
metaKeyExecutionTimeMs = "execution_time_ms"
|
||||
)
|
||||
|
||||
// JSONResponseMiddleware is an HTTP middleware that wraps all JSON responses
|
||||
// with a @meta field containing execution metadata. The metadata includes the
|
||||
// time zone (always UTC), API version, and request execution time in milliseconds.
|
||||
@@ -116,10 +106,10 @@ func JSONResponseMiddleware(next http.Handler) http.Handler {
|
||||
}
|
||||
|
||||
// Add @meta field
|
||||
originalResponse[jsonKeyMeta] = map[string]any{
|
||||
metaKeyTimeZone: "UTC",
|
||||
metaKeyAPIVersion: 1,
|
||||
metaKeyExecutionTimeMs: execTime.Milliseconds(),
|
||||
originalResponse["@meta"] = map[string]any{
|
||||
"time_zone": "UTC",
|
||||
"api_version": 1,
|
||||
"execution_time": fmt.Sprintf("%d ms", execTime.Milliseconds()),
|
||||
}
|
||||
|
||||
// Write the enhanced response
|
||||
@@ -155,8 +145,7 @@ func deadlinePassed(ctx context.Context) bool {
|
||||
// has passed, it answers 504 with the error message "Request timeout". A 4xx
|
||||
// keeps its status, with the status's standard text as the message. For other
|
||||
// errors, it returns "Internal server error", with status 500 if the original
|
||||
// status was 200. A panic in the handler answers 500 with "Internal server
|
||||
// error" too.
|
||||
// status was 200.
|
||||
//
|
||||
// Non-JSON responses are passed through unchanged.
|
||||
func JSONValidationMiddleware(next http.Handler) http.Handler {
|
||||
@@ -168,9 +157,8 @@ func JSONValidationMiddleware(next http.Handler) http.Handler {
|
||||
statusCode: http.StatusOK,
|
||||
}
|
||||
|
||||
// Serve the request. Recoverer logs a panic in the handler and turns it
|
||||
// into a 500 without a body, which gets the JSON error body below.
|
||||
middleware.Recoverer(next).ServeHTTP(rw, r)
|
||||
// Serve the request
|
||||
next.ServeHTTP(rw, r)
|
||||
|
||||
// Check if it's meant to be a JSON response
|
||||
contentType := rw.Header().Get("Content-Type")
|
||||
|
||||
@@ -2,7 +2,6 @@ package server
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"encoding/json"
|
||||
"io"
|
||||
"log/slog"
|
||||
@@ -11,7 +10,6 @@ import (
|
||||
"net/http/httptest"
|
||||
"net/netip"
|
||||
"os"
|
||||
"regexp"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
@@ -102,219 +100,6 @@ func TestJSONValidationKeeps4xxWithoutBody(t *testing.T) {
|
||||
checkJSONErrorMessage(t, name, rec, http.StatusText(http.StatusBadRequest))
|
||||
}
|
||||
|
||||
// TestJSONAnswersPutNoUnitInAString sends a request to each JSON route and
|
||||
// checks each answer. Its @meta.execution_time_ms is a number, and none of its
|
||||
// strings is a number with a unit, such as "12 ms", "1.1 GB" or "1h18m34s"
|
||||
// (https://git.eeqj.de/sneak/routewatch/issues/199). The health check answers
|
||||
// 503, since the stream is not connected, and the last path matches no route.
|
||||
func TestJSONAnswersPutNoUnitInAString(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
numberWithUnit := regexp.MustCompile(
|
||||
`^([0-9]+(\.[0-9]+)? ?(ns|µs|ms|s|m|h|B|kB|MB|GB|TB))+$`)
|
||||
|
||||
s := newServerWithTestRoutes(t)
|
||||
s.resolver = &fakeResolver{}
|
||||
s.SetASNFetcher(fixedIntervalASNFetcher{interval: time.Second})
|
||||
|
||||
paths := []string{
|
||||
statusJSONPath,
|
||||
statsPath,
|
||||
healthCheckPath,
|
||||
"/api/v1/ip/" + untrustedPeerAddr,
|
||||
"/ip/" + untrustedPeerAddr,
|
||||
asJSONPath,
|
||||
"/api/v1/prefix/" + testPrefixV4,
|
||||
"/api/v1/search/as?q=google",
|
||||
"/api/v1/no-such-route",
|
||||
}
|
||||
|
||||
for _, path := range paths {
|
||||
rec := httptest.NewRecorder()
|
||||
s.router.ServeHTTP(rec, httptest.NewRequestWithContext(
|
||||
t.Context(), http.MethodGet, path, nil))
|
||||
|
||||
var answer map[string]any
|
||||
|
||||
err := json.Unmarshal(rec.Body.Bytes(), &answer)
|
||||
if err != nil {
|
||||
t.Errorf("%s: decoding the answer: %v; answer %s", path, err, rec.Body)
|
||||
|
||||
continue
|
||||
}
|
||||
|
||||
meta, _ := answer["@meta"].(map[string]any)
|
||||
if _, ok := meta["execution_time_ms"].(float64); !ok {
|
||||
t.Errorf("%s: @meta.execution_time_ms is not a number: %s",
|
||||
path, rec.Body)
|
||||
}
|
||||
|
||||
for _, value := range stringsIn(answer) {
|
||||
if numberWithUnit.MatchString(value) {
|
||||
t.Errorf("%s: %q is a number with a unit in a string", path, value)
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// TestJSONAnswersCarryMeta sends requests to every JSON route through the
|
||||
// server's router, and checks that each answer carries @meta with its three
|
||||
// fields. The answers include successes, and errors that the handlers, the
|
||||
// router and the middleware write. Before the fix a panic in a handler, and a
|
||||
// method /status.json or the health check does not take, answered with an
|
||||
// empty body (https://git.eeqj.de/sneak/routewatch/issues/201).
|
||||
func TestJSONAnswersCarryMeta(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
// JSON routes whose handlers write nothing or panic, so that the JSON
|
||||
// check on the route writes the error answer itself.
|
||||
const (
|
||||
writesNothingPath = "/writes-nothing.json"
|
||||
panicsPath = "/panics.json"
|
||||
)
|
||||
|
||||
s := newServerWithTestRoutes(t)
|
||||
s.resolver = &fakeResolver{}
|
||||
s.streamer.GetMetricsTracker().SetConnected(true)
|
||||
s.router.Get(writesNothingPath, JSONValidationMiddleware(http.HandlerFunc(
|
||||
func(http.ResponseWriter, *http.Request) {})).ServeHTTP)
|
||||
s.router.Get(panicsPath, JSONValidationMiddleware(http.HandlerFunc(
|
||||
func(http.ResponseWriter, *http.Request) { panic(errDiskIO) })).ServeHTTP)
|
||||
|
||||
tests := []struct {
|
||||
method string
|
||||
path string
|
||||
wantCode int
|
||||
}{
|
||||
// Answers that succeed
|
||||
{http.MethodGet, statusJSONPath, http.StatusOK},
|
||||
{http.MethodGet, healthCheckPath, http.StatusOK},
|
||||
{http.MethodGet, "/ip", http.StatusOK},
|
||||
{http.MethodGet, ownAddressPath, http.StatusOK},
|
||||
{http.MethodGet, "/ip/" + addrInTestPrefixV4, http.StatusOK},
|
||||
{http.MethodGet, statsPath, http.StatusOK},
|
||||
{http.MethodGet, "/api/v1/ip/" + addrInTestPrefixV4, http.StatusOK},
|
||||
{http.MethodGet, asJSONPath, http.StatusOK},
|
||||
{http.MethodGet, "/api/v1/prefix/" + testPrefixV4, http.StatusOK},
|
||||
{http.MethodGet, "/api/v1/search/as?q=google", http.StatusOK},
|
||||
// Errors the handlers write
|
||||
{http.MethodGet, "/ip/no-such-host.example", http.StatusBadRequest},
|
||||
{http.MethodGet, "/api/v1/ip/no-such-host.example", http.StatusBadRequest},
|
||||
{http.MethodGet, "/api/v1/as/foo", http.StatusBadRequest},
|
||||
{http.MethodGet, "/api/v1/as/0", http.StatusNotFound},
|
||||
{http.MethodGet, "/api/v1/prefix/192.0.2.0/33", http.StatusBadRequest},
|
||||
{http.MethodGet, "/api/v1/prefix/198.51.100.0/24", http.StatusNotFound},
|
||||
{http.MethodGet, "/api/v1/search/as", http.StatusBadRequest},
|
||||
// Errors the router and the middleware write
|
||||
{http.MethodGet, "/api/v1/no-such-route", http.StatusNotFound},
|
||||
{http.MethodGet, "/ip/192.0.2.1/nothing", http.StatusNotFound},
|
||||
{http.MethodPost, statsPath, http.StatusMethodNotAllowed},
|
||||
{http.MethodPost, ownAddressPath, http.StatusMethodNotAllowed},
|
||||
{http.MethodPost, statusJSONPath, http.StatusMethodNotAllowed},
|
||||
{http.MethodPut, healthCheckPath, http.StatusMethodNotAllowed},
|
||||
{http.MethodGet, writesNothingPath, http.StatusInternalServerError},
|
||||
{http.MethodGet, panicsPath, http.StatusInternalServerError},
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
checkAnswerCarriesMeta(t, s, httptest.NewRequestWithContext(
|
||||
t.Context(), tt.method, tt.path, nil), tt.wantCode)
|
||||
}
|
||||
|
||||
// A request whose time limit has already passed answers 504. The handlers
|
||||
// write the answers of the first five routes, and the JSON check on the
|
||||
// route writes the last.
|
||||
pastTimeLimit, cancel := context.WithDeadline(t.Context(), time.Now())
|
||||
defer cancel()
|
||||
|
||||
for _, path := range []string{
|
||||
statusJSONPath, statsPath, "/ip/" + addrInTestPrefixV4,
|
||||
"/api/v1/ip/" + addrInTestPrefixV4, asJSONPath, writesNothingPath,
|
||||
} {
|
||||
checkAnswerCarriesMeta(t, s, httptest.NewRequestWithContext(
|
||||
pastTimeLimit, http.MethodGet, path, nil), http.StatusGatewayTimeout)
|
||||
}
|
||||
|
||||
// With a store whose reads fail, the health check answers 503, and the
|
||||
// handlers of the other routes 500.
|
||||
failing := New(failingDB{}, newTestTable(),
|
||||
streamer.New(logger.New(), metrics.New()), logger.New(), &config.Config{})
|
||||
failing.resolver = &fakeResolver{}
|
||||
|
||||
checkAnswerCarriesMeta(t, failing, httptest.NewRequestWithContext(
|
||||
t.Context(), http.MethodGet, healthCheckPath, nil),
|
||||
http.StatusServiceUnavailable)
|
||||
|
||||
for _, path := range []string{
|
||||
statusJSONPath, statsPath, asJSONPath,
|
||||
"/api/v1/ip/" + addrInTestPrefixV4, "/ip/" + addrInTestPrefixV4,
|
||||
} {
|
||||
checkAnswerCarriesMeta(t, failing, httptest.NewRequestWithContext(
|
||||
t.Context(), http.MethodGet, path, nil), http.StatusInternalServerError)
|
||||
}
|
||||
}
|
||||
|
||||
// checkAnswerCarriesMeta serves req through the server's router, and checks
|
||||
// that it answers wantCode with @meta and its three fields.
|
||||
func checkAnswerCarriesMeta(
|
||||
t *testing.T, s *Server, req *http.Request, wantCode int,
|
||||
) {
|
||||
t.Helper()
|
||||
|
||||
// The values README "API v1" gives the two fields that never change.
|
||||
const (
|
||||
wantTimeZone = "UTC"
|
||||
wantAPIVersion = 1
|
||||
)
|
||||
|
||||
name := req.Method + " " + req.URL.String()
|
||||
|
||||
rec := httptest.NewRecorder()
|
||||
s.router.ServeHTTP(rec, req)
|
||||
|
||||
if rec.Code != wantCode {
|
||||
t.Errorf("%s: status %d, want %d; body %s", name, rec.Code, wantCode, rec.Body)
|
||||
}
|
||||
|
||||
var answer map[string]any
|
||||
|
||||
err := json.Unmarshal(rec.Body.Bytes(), &answer)
|
||||
if err != nil {
|
||||
t.Errorf("%s: decoding the answer %q: %v", name, rec.Body, err)
|
||||
|
||||
return
|
||||
}
|
||||
|
||||
meta, _ := answer[jsonKeyMeta].(map[string]any)
|
||||
_, timeIsNumber := meta[metaKeyExecutionTimeMs].(float64)
|
||||
|
||||
if meta[metaKeyTimeZone] != wantTimeZone ||
|
||||
meta[metaKeyAPIVersion] != float64(wantAPIVersion) || !timeIsNumber {
|
||||
t.Errorf("%s: @meta %v, want time_zone %q, api_version %d and a "+
|
||||
"number in execution_time_ms", name, meta, wantTimeZone, wantAPIVersion)
|
||||
}
|
||||
}
|
||||
|
||||
// stringsIn returns every string in a decoded JSON value, at any depth.
|
||||
func stringsIn(value any) []string {
|
||||
var found []string
|
||||
|
||||
switch v := value.(type) {
|
||||
case string:
|
||||
found = append(found, v)
|
||||
case []any:
|
||||
for _, item := range v {
|
||||
found = append(found, stringsIn(item)...)
|
||||
}
|
||||
case map[string]any:
|
||||
for _, item := range v {
|
||||
found = append(found, stringsIn(item)...)
|
||||
}
|
||||
}
|
||||
|
||||
return found
|
||||
}
|
||||
|
||||
// TestPagesCarryTheCSPNonce checks that every script and style block of each
|
||||
// page carries the nonce its Content-Security-Policy header names, so that the
|
||||
// browser runs and applies them, that each answer has a nonce of its own, and
|
||||
|
||||
@@ -8,8 +8,8 @@ import (
|
||||
"github.com/go-chi/chi/v5/middleware"
|
||||
)
|
||||
|
||||
// The error messages the JSON routes answer with when no route matches the
|
||||
// path, and when the route does not take the method.
|
||||
// The error messages the JSON routes under /api/v1 and /ip answer with when no
|
||||
// route matches the path, and when none takes the method.
|
||||
const (
|
||||
notFoundMessage = "Not found"
|
||||
methodNotAllowedMessage = "Method not allowed"
|
||||
@@ -19,13 +19,6 @@ const (
|
||||
jsonRoutesAllow = "GET, HEAD"
|
||||
)
|
||||
|
||||
// The paths of the status figures and the health check, the two JSON routes
|
||||
// outside /api/v1 and /ip.
|
||||
const (
|
||||
statusJSONPath = "/status.json"
|
||||
healthCheckPath = "/.well-known/healthcheck.json"
|
||||
)
|
||||
|
||||
// setupRoutes configures the HTTP routes
|
||||
func (s *Server) setupRoutes() {
|
||||
r := chi.NewRouter()
|
||||
@@ -39,8 +32,6 @@ func (s *Server) setupRoutes() {
|
||||
r.Use(middleware.RequestID)
|
||||
r.Use(RealIPMiddleware(s.trustedProxies))
|
||||
r.Use(RequestLoggerMiddleware(s.logger.Logger)) // Structured request logging
|
||||
// Recoverer answers a panic on a page. JSONValidationMiddleware recovers a
|
||||
// panic on a JSON route itself, so that the answer is JSON with @meta.
|
||||
r.Use(middleware.Recoverer)
|
||||
r.Use(JSONResponseMiddleware)
|
||||
// Timeout must come after JSONResponseMiddleware, which sends its answer
|
||||
@@ -52,6 +43,11 @@ func (s *Server) setupRoutes() {
|
||||
// Routes
|
||||
r.Get("/", s.handleIndex())
|
||||
r.Get("/status", s.handleStatusHTML())
|
||||
r.Get("/status.json", JSONValidationMiddleware(s.handleStatusJSON()).ServeHTTP)
|
||||
r.Get(
|
||||
"/.well-known/healthcheck.json",
|
||||
JSONValidationMiddleware(s.handleHealthCheck()).ServeHTTP,
|
||||
)
|
||||
|
||||
// AS and prefix detail pages
|
||||
r.Get("/as/{asn}", s.handleASDetail())
|
||||
@@ -74,21 +70,6 @@ func (s *Server) setupRoutes() {
|
||||
s.writeJSONError(w, http.StatusMethodNotAllowed, methodNotAllowedMessage)
|
||||
}
|
||||
|
||||
// The status figures and the health check. chi sets one 405 answer for a
|
||||
// whole router. These two routes are on the pages' router, whose 405 stays
|
||||
// chi's own. So each method they do not take is routed to the JSON 405
|
||||
// here.
|
||||
r.Get(statusJSONPath, JSONValidationMiddleware(s.handleStatusJSON()).ServeHTTP)
|
||||
r.Get(healthCheckPath, JSONValidationMiddleware(s.handleHealthCheck()).ServeHTTP)
|
||||
|
||||
for _, method := range []string{
|
||||
http.MethodPost, http.MethodPut, http.MethodPatch, http.MethodDelete,
|
||||
http.MethodOptions, http.MethodTrace, http.MethodConnect,
|
||||
} {
|
||||
r.MethodFunc(method, statusJSONPath, jsonMethodNotAllowed)
|
||||
r.MethodFunc(method, healthCheckPath, jsonMethodNotAllowed)
|
||||
}
|
||||
|
||||
// IP info JSON endpoints (replaces old /ip redirect)
|
||||
r.Route("/ip", func(r chi.Router) {
|
||||
r.Use(JSONValidationMiddleware)
|
||||
|
||||
@@ -17,7 +17,7 @@ func TestRoutesAnswerHEAD(t *testing.T) {
|
||||
|
||||
s := newServerWithTestRoutes(t)
|
||||
|
||||
for _, path := range []string{"/", statusJSONPath, statsPath, "/ip", "/ip/"} {
|
||||
for _, path := range []string{"/", statsPath, "/ip", "/ip/"} {
|
||||
rec := httptest.NewRecorder()
|
||||
s.router.ServeHTTP(rec, httptest.NewRequestWithContext(
|
||||
t.Context(), http.MethodHead, path, nil))
|
||||
@@ -31,13 +31,10 @@ func TestRoutesAnswerHEAD(t *testing.T) {
|
||||
|
||||
// TestJSONRoutesAnswerUnmatchedRequestsWithJSONErrors sends a method the JSON
|
||||
// routes do not take, and a path none of them matches, to each group of JSON
|
||||
// routes through the server's router. It also sends a method they do not take
|
||||
// to /status.json and the health check. Each must answer with the JSON error
|
||||
// routes through the server's router. Each must answer with the JSON error
|
||||
// body, and a 405 must name the methods the routes take in its Allow header.
|
||||
// Before the fix a 405 said "Internal server error" and a 404 was chi's plain
|
||||
// text answer (https://git.eeqj.de/sneak/routewatch/issues/111).
|
||||
// /status.json and the health check gave chi's empty 405
|
||||
// (https://git.eeqj.de/sneak/routewatch/issues/201).
|
||||
func TestJSONRoutesAnswerUnmatchedRequestsWithJSONErrors(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
@@ -51,14 +48,6 @@ func TestJSONRoutesAnswerUnmatchedRequestsWithJSONErrors(t *testing.T) {
|
||||
}{
|
||||
{http.MethodPost, statsPath, http.StatusMethodNotAllowed, methodNotAllowedMessage},
|
||||
{http.MethodPost, "/ip/", http.StatusMethodNotAllowed, methodNotAllowedMessage},
|
||||
{
|
||||
http.MethodDelete, statusJSONPath,
|
||||
http.StatusMethodNotAllowed, methodNotAllowedMessage,
|
||||
},
|
||||
{
|
||||
http.MethodOptions, healthCheckPath,
|
||||
http.StatusMethodNotAllowed, methodNotAllowedMessage,
|
||||
},
|
||||
{http.MethodGet, "/api/v1/nothing", http.StatusNotFound, notFoundMessage},
|
||||
{http.MethodGet, "/ip/192.0.2.1/nothing", http.StatusNotFound, notFoundMessage},
|
||||
}
|
||||
|
||||
@@ -82,12 +82,6 @@ type Server struct {
|
||||
// cards is the copy of the home page's cards that the pages serve, nil
|
||||
// until the server first works them out
|
||||
cards atomic.Pointer[homeCards]
|
||||
|
||||
// stopCards ends the goroutine that works out the cards, and
|
||||
// cardsStopped is closed once it has returned. Both are nil until Start
|
||||
// starts the goroutine.
|
||||
stopCards context.CancelFunc
|
||||
cardsStopped chan struct{}
|
||||
}
|
||||
|
||||
// New creates a new HTTP server that reads routes from table, the routing
|
||||
@@ -117,8 +111,7 @@ func New(
|
||||
// Start listens on the configured port and serves HTTP in the background.
|
||||
// It returns an error if it cannot listen, for example because the port is
|
||||
// already in use. ctx bounds opening the port, not serving. Once listening,
|
||||
// the server works out the home page's cards every minute until Stop: the end
|
||||
// of ctx does not stop it.
|
||||
// the server works out the home page's cards every minute until ctx ends.
|
||||
func (s *Server) Start(ctx context.Context) error {
|
||||
const (
|
||||
network = "tcp"
|
||||
@@ -155,22 +148,16 @@ func (s *Server) Start(ctx context.Context) error {
|
||||
}
|
||||
}()
|
||||
|
||||
ctx, s.stopCards = context.WithCancel(context.WithoutCancel(ctx))
|
||||
s.cardsStopped = make(chan struct{})
|
||||
|
||||
go func() {
|
||||
defer close(s.cardsStopped)
|
||||
|
||||
s.keepHomeCardsFresh(ctx, homeCardsRefreshInterval)
|
||||
}()
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
// Stop stops the goroutine that works out the home page's cards, and waits
|
||||
// until it has returned or ctx ends. It then stops the HTTP server. It waits,
|
||||
// until ctx ends, for the requests in progress to finish, and then closes the
|
||||
// connections of those still running and returns ctx's error.
|
||||
// Stop stops the HTTP server. It waits, until ctx ends, for the requests in
|
||||
// progress to finish, and then closes the connections of those still running
|
||||
// and returns ctx's error.
|
||||
func (s *Server) Stop(ctx context.Context) error {
|
||||
if s.srv == nil {
|
||||
return nil
|
||||
@@ -178,16 +165,6 @@ func (s *Server) Stop(ctx context.Context) error {
|
||||
|
||||
s.logger.Info("Stopping HTTP server")
|
||||
|
||||
// Start returns before it starts the goroutine when it cannot listen.
|
||||
if s.stopCards != nil {
|
||||
s.stopCards()
|
||||
|
||||
select {
|
||||
case <-s.cardsStopped:
|
||||
case <-ctx.Done():
|
||||
}
|
||||
}
|
||||
|
||||
err := s.srv.Shutdown(ctx)
|
||||
if err != nil {
|
||||
return errors.Join(err, s.srv.Close())
|
||||
|
||||
@@ -714,8 +714,7 @@
|
||||
});
|
||||
});
|
||||
|
||||
// IP lookup: show the whole JSON answer, its @meta and an error answer
|
||||
// included, as the IP information API card does.
|
||||
// IP lookup
|
||||
document.getElementById('ip-form').addEventListener('submit', function(e) {
|
||||
e.preventDefault();
|
||||
const ip = document.getElementById('ip-input').value.trim();
|
||||
@@ -731,11 +730,18 @@
|
||||
|
||||
fetch('/ip/' + encodeURIComponent(ip))
|
||||
.then(response => response.json())
|
||||
.then(answer => {
|
||||
contentEl.className = '';
|
||||
contentEl.textContent = JSON.stringify(answer, null, 2);
|
||||
.then(response => {
|
||||
contentEl.classList.remove('loading');
|
||||
if (response.status === 'error') {
|
||||
contentEl.className = 'error';
|
||||
contentEl.textContent = 'Error: ' + response.error.msg;
|
||||
} else {
|
||||
contentEl.className = '';
|
||||
contentEl.textContent = JSON.stringify(response.data, null, 2);
|
||||
}
|
||||
})
|
||||
.catch(error => {
|
||||
contentEl.classList.remove('loading');
|
||||
contentEl.className = 'error';
|
||||
contentEl.textContent = 'Error: ' + error.message;
|
||||
});
|
||||
|
||||
@@ -401,8 +401,6 @@
|
||||
const REFRESH_PERIOD_MS = 2000;
|
||||
const STALE_AFTER_REFRESHES = 3;
|
||||
const MS_PER_SECOND = 1000;
|
||||
const SECONDS_PER_MINUTE = 60;
|
||||
const SECONDS_PER_HOUR = 3600;
|
||||
const SPINNER_FRAMES = ['⠋', '⠙', '⠹', '⠸', '⠼', '⠴', '⠦', '⠧', '⠇', '⠏'];
|
||||
const SPINNER_COLOURS = ['#e74c3c', '#e67e22', '#f1c40f', '#27ae60', '#3498db', '#9b59b6'];
|
||||
|
||||
@@ -445,18 +443,7 @@
|
||||
function formatNumber(num) {
|
||||
return num.toLocaleString();
|
||||
}
|
||||
|
||||
// formatDuration writes a number of seconds in hours, minutes and
|
||||
// seconds, such as "1h 18m 34s" or "18.2s", with at most one decimal.
|
||||
function formatDuration(totalSeconds) {
|
||||
const hours = Math.floor(totalSeconds / SECONDS_PER_HOUR);
|
||||
const minutes = Math.floor(totalSeconds % SECONDS_PER_HOUR / SECONDS_PER_MINUTE);
|
||||
const seconds = parseFloat((totalSeconds % SECONDS_PER_MINUTE).toFixed(1));
|
||||
if (hours > 0) return hours + 'h ' + minutes + 'm ' + seconds + 's';
|
||||
if (minutes > 0) return minutes + 'm ' + seconds + 's';
|
||||
return seconds + 's';
|
||||
}
|
||||
|
||||
|
||||
function formatProcessingTime(ms) {
|
||||
if (ms < 0.001) {
|
||||
return (ms * 1000).toFixed(0) + ' µs';
|
||||
@@ -632,13 +619,11 @@
|
||||
noteMessageCount(data.total_messages);
|
||||
|
||||
// Update all metrics
|
||||
document.getElementById('uptime').textContent = formatDuration(data.uptime_seconds);
|
||||
document.getElementById('uptime').textContent = data.uptime;
|
||||
document.getElementById('go_version').textContent = data.go_version;
|
||||
document.getElementById('goroutines').textContent = formatNumber(data.goroutines);
|
||||
document.getElementById('memory_usage').textContent = formatBytes(data.memory_usage_bytes);
|
||||
// connection_duration_seconds is null while the stream is not connected.
|
||||
document.getElementById('connection_duration').textContent =
|
||||
data.connection_duration_seconds === null ? 'disconnected' : formatDuration(data.connection_duration_seconds);
|
||||
document.getElementById('memory_usage').textContent = data.memory_usage;
|
||||
document.getElementById('connection_duration').textContent = data.connection_duration;
|
||||
document.getElementById('reconnect_count').textContent = formatNumber(data.reconnect_count);
|
||||
document.getElementById('total_messages').textContent = formatNumber(data.total_messages);
|
||||
document.getElementById('messages_per_sec').textContent = formatNumber(Math.round(data.messages_per_sec));
|
||||
@@ -687,7 +672,7 @@
|
||||
const errorsEl = document.getElementById('whois_errors');
|
||||
errorsEl.textContent = formatNumber(data.whois_stats.errors_last_hour);
|
||||
errorsEl.className = 'metric-value' + (data.whois_stats.errors_last_hour > 0 ? ' disconnected' : '');
|
||||
document.getElementById('whois_interval').textContent = formatDuration(data.whois_stats.current_interval_seconds);
|
||||
document.getElementById('whois_interval').textContent = data.whois_stats.current_interval;
|
||||
}
|
||||
|
||||
// Update handler stats
|
||||
|
||||
Reference in New Issue
Block a user