Author SHA1 Message Date
sneak bc7c5e18a5 Answer 504, not 408, when a request outlasts its time limit (closes #70)
check / check (push) Waiting to run
The hand-written time limit middleware answered 408 and wrote that
answer from its own goroutine while the handler could still be writing.
It is replaced by chi's middleware.Timeout, placed after
JSONResponseMiddleware so its 504 is not lost behind a 200.

When a handler's database read or other lookup fails, it asks
deadlinePassed whether the request's time limit passed and, if so,
answers 504 instead of its 500 or 404. JSONValidationMiddleware answers
504 with the JSON error body when the limit has passed and the handler
wrote nothing. status.json and /api/v1/stats answer 504 when their
4-second database read runs out.

Model: opus-5-5
2026-10-14 15:37:24 +00:00
clawbot 616f13071f Parse the prefix on the prefix pages before looking it up (closes #71)
check / check (push) Waiting to run
`/prefix/{prefix}/{len}` and `/api/v1/prefix/{prefix}/{len}` passed the URL text to the lookup as typed: a prefix that does not parse answered 500, and a valid prefix written in another form (`2001:DB8:0:0::/32`, `192.0.2.1/24`) answered 404. Both now parse the prefix with `net/netip`, mask it and look up the form the tables store; one that does not parse answers 400. The answer shows the stored form of the prefix. The test drives both pages through the router against a real database.

Model: opus-5-5
2026-10-14 17:24:02 +02:00
clawbot 521c632087 Ignore routes shorter than /8 or /16 in the IP lookup (closes #57)
check / check (push) Waiting to run
The IP lookup tried every mask length down to /0, so a default route or a very short route such as `2000::/3` was reported as the owner of any address it covered until a more specific route had been seen. The lookup now stops at /8 for IPv4 and /16 for IPv6 (named constants); an address covered only by shorter routes gets "no route found". The tables still store short routes. The test checks each limit and the length one bit shorter.

Model: opus-5-5
2026-10-14 17:23:51 +02:00
clawbot 77a708d819 Apply each peer's last update for a prefix in a batch (closes #54)
check / check (push) Waiting to run
The prefix handler kept one update per prefix in each batch across all peers, chosen by a timestamp cut to whole seconds, so most peers' announcements and withdrawals were lost and the first update in a second won. A batch now keeps the last update per prefix and peer in feed order. A message's withdrawals are queued before its announcements, so a prefix both withdrawn and announced in one message keeps its route (RFC 4271 section 4.3). Tests run each case through the handler into a real database and check the in-memory counts.

Model: opus-5-5
2026-10-14 17:09:13 +02:00
clawbot a43935308e Count every reconnect in reconnect_count (closes #68)
check / check (push) Waiting to run
`reconnect_count` in `/api/v1/stats` stayed 0 however often the stream reconnected: `SetConnected` counted a reconnect only while the stream was still marked connected, and the streamer always marks it disconnected first. It now counts one reconnect for every connection after the first, recognised by the zero `connectedSince` before the first connection. A test connects and disconnects twice and expects 2.

Model: opus-5-5
2026-10-14 15:59:15 +02:00
59 changed files with 3334 additions and 4134 deletions
+10 -75
View File
@@ -1,83 +1,15 @@
# .dockerignore does NOT use .gitignore semantics. Docker matches with # Docker does not read .gitignore, and a pattern here matches from the root of
# moby/patternmatcher: filepath.Match plus `**`, so `*` does not cross # the build context only: a pattern meant for every directory needs `**/`.
# `/` and an unprefixed pattern is anchored at the context root. Every
# depth-independent pattern therefore needs `**/`, or `config/.env` and
# `certs/server.key` still ship while this file reads as solved. Only
# genuinely root-anchored entries go unprefixed. Never transplant these
# into .gitignore, where `**/` is wrong.
#
# Matching is case-sensitive, so secrets use character ranges rather
# than an ALL-CAPS twin, which would still miss `Server.Key`.
#
# Extend with this repo's own host-built artifacts, written anchored:
# `/myapp`, never `**/myapp`, which also matches `cmd/myapp/` and
# deletes the package directory from the context.
# .git is sent without its config. Without a VERSION build argument the # .git is sent without its config. Without a VERSION build argument the
# stage that compiles runs `git describe --tags --always` on .git, which # stage that compiles runs `git describe --tags --always` on .git, which
# does not need .git/config; that file can hold a credential, such as a # does not need .git/config; that file can hold a credential, such as a
# password in a remote URL or the token the CI checkout step stores there. # password in a remote URL or the token the CI checkout step stores there.
# Each submodule keeps a config with the same exposure in its git directory .git/config
# under .git/modules/, nested again for a submodule's own submodules, or in
# its own .git directory when it keeps one.
# KNOWN GAP: a submodule whose name has a `config` segment (`config`,
# `deploy/config`, `config/lib`) loses its whole git directory, because
# `**/.git/modules/**/config` also matches that segment's directory
# under .git/modules/. Go's version stamping then fails the build;
# nothing leaks. Name such a submodule without that segment:
# `git submodule add --name`.
**/.git/config
**/.git/modules/**/config
# Agent scratch: one full checkout of the repo per in-flight agent. # Local build and debug output: `make build`, `make run`, `make asupdate`, test
# Anchored because it occurs once where agents run at the repo root. # binaries, coverage profiles and source archives.
# KNOWN GAP: a repo running agents in subdirectories still ships
# `services/api/.claude/` and must add its own anchored entry.
.claude
# Environment files. `*.env` covers bare `.env` and the `prod.env`
# convention. Re-include a committed template with a negation if the
# build needs one: `!docs/example.env`.
**/*.[eE][nN][vV]
**/.[eE][nN][vV].*
**/.[eE][nN][vV][rR][cC]
# Private keys and the bundles carrying them. Public certificates
# (*.crt, *.cer) are deliberately absent: they are legitimate inputs.
**/*.[pP][eE][mM]
**/*.[kK][eE][yY]
**/*.[pP]12
**/*.[pP][fF][xX]
**/[iI][dD]_[rR][sS][aA]
**/[iI][dD]_[dD][sS][aA]
**/[iI][dD]_[eE][cC][dD][sS][aA]
**/[iI][dD]_[eE][cC][dD][sS][aA]_[sS][kK]
**/[iI][dD]_[eE][dD]25519
**/[iI][dD]_[eE][dD]25519_[sS][kK]
# Dependencies: restored inside the image, never copied in.
**/node_modules
# OS metadata.
**/.DS_Store
**/Thumbs.db
# Editor state: never a build input, and it churns COPY.
**/*.swp
**/*.swo
**/*~
**/*.bak
**/.idea
**/.vscode
**/*.sublime-*
# This repository's host-built artifacts: `make build` writes bin/, `go
# build` in the root the binaries, `make run` log.txt, `make asupdate` its
# .tmp file; test binaries, coverage profiles and source archives.
/bin /bin
/routewatch
/asinfo-gen
/streamdumper
/log.txt /log.txt
/out /out
/pkg/asinfo/asdata.json /pkg/asinfo/asdata.json
@@ -86,12 +18,15 @@
**/*.out **/*.out
**/*.tmp **/*.tmp
# Local databases. The image carries a source archive of the whole build # Local databases and secrets. The image carries a source archive of the whole
# context, so these would otherwise ship inside it. # build context, so these would otherwise ship inside it.
**/*.db **/*.db
**/*.db-journal **/*.db-journal
**/*.db-wal **/*.db-wal
**/.env
# A local Go workspace points at directories outside the build context. # A local Go workspace points at directories outside the build context.
/go.work /go.work
/go.work.sum /go.work.sum
**/.DS_Store
-3
View File
@@ -10,6 +10,3 @@ insert_final_newline = true
[Makefile] [Makefile]
indent_style = tab indent_style = tab
[*.go]
indent_style = tab
+1 -1
View File
@@ -4,6 +4,6 @@ jobs:
check: check:
runs-on: ubuntu-latest runs-on: ubuntu-latest
steps: steps:
# actions/checkout v4.2.2, 2026-02-22 # actions/checkout v4.2.2, 2026-02-28
- uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 - uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683
- run: script/cibuild - run: script/cibuild
+27 -63
View File
@@ -1,77 +1,41 @@
# OS # Binaries for programs and plugins
.DS_Store
Thumbs.db
# Editors
*.swp
*.swo
*~
*.bak
.idea/
.vscode/
*.sublime-*
# Agent scratch (worktrees of this repo, created and destroyed by
# in-flight tooling). Unanchored: .gitignore patterns already match at
# every depth, so no prefix is wanted here. This is not a .dockerignore
# entry and must not be given a `**/` prefix on the way into one.
.claude/
# Node
node_modules/
# Secrets. Unanchored like every entry above, so each matches at every
# depth. Matching is case-sensitive on Linux, so names use character
# ranges rather than a lowercase form that misses `Server.Key`.
# Environment files. `*.env` covers bare `.env` and the `prod.env`
# convention. Only the templates `example.env` and `sample.env` are
# re-included below. A repository that commits any other template adds
# its own negation after these lines, for example `!.env.example`.
*.[eE][nN][vV]
.[eE][nN][vV].*
.[eE][nN][vV][rR][cC]
!example.env
!sample.env
# Private keys and the bundles carrying them.
*.[pP][eE][mM]
*.[kK][eE][yY]
*.[pP]12
*.[pP][fF][xX]
[iI][dD]_[rR][sS][aA]
[iI][dD]_[dD][sS][aA]
[iI][dD]_[eE][cC][dD][sS][aA]
[iI][dD]_[eE][cC][dD][sS][aA]_[sS][kK]
[iI][dD]_[eE][dD]25519
[iI][dD]_[eE][dD]25519_[sS][kK]
# This repository's own entries, kept after the canonical content above.
# Go: logs, test binaries, coverage profiles, and the binaries `make build`
# and `go build` in the repository root write.
*.log
*.out
*.test
*.exe *.exe
*.exe~ *.exe~
*.dll *.dll
*.so *.so
*.dylib *.dylib
/bin/ /bin/
/routewatch
/asinfo-gen .DS_Store
/streamdumper log.txt
# Test binary, built with `go test -c`
*.test
# Output of the go coverage tool, specifically when used with LiteIDE
*.out
# Dependency directories (remove the comment below to include it)
# vendor/
# Go workspace file
go.work go.work
go.work.sum go.work.sum
# Local databases, temporary files and debug output. # env file
.env
# Database files
*.db *.db
*.db-journal *.db-journal
*.db-wal *.db-wal
*.tmp
log.txt
out
# Raw AS data (only the gzipped version is committed). # Temporary files
*.tmp
# Raw AS data (we only commit the gzipped version)
pkg/asinfo/asdata.json pkg/asinfo/asdata.json
# Debug output files
out
log.txt
+81 -88
View File
@@ -1,99 +1,92 @@
version: "2" version: "2"
# Config schema uses the golangci-lint v2 layout (settings live under
# linters.settings, not top-level linters-settings) so that the
# thresholds below are actually applied by golangci-lint >= v2.
run: run:
timeout: 5m go: "1.24"
modules-download-mode: readonly tests: false
linters: linters:
default: all
enable: enable:
# Successor to the deprecated gomodguard. Named explicitly, rather than # Additional linters requested
# left to `default: all`, because it carries the module policy below. - testifylint # Checks usage of github.com/stretchr/testify
- gomodguard_v2 - usetesting # usetesting is an analyzer that detects using os.Setenv instead of t.Setenv since Go 1.17
disable: # - tagliatelle # Disabled: we need snake_case for external API compatibility
# Genuinely incompatible with project patterns - nlreturn # nlreturn checks for a new line before return and branch statements
- exhaustruct # Requires all struct fields - nilnil # Checks that there is no simultaneous return of nil error and an invalid value
- exhaustruct_v5 # Requires all struct fields (successor to exhaustruct) - nestif # Reports deeply nested if statements
- godot # Requires comments to end with periods - mnd # An analyzer to detect magic numbers
- wrapcheck # Too verbose for internal packages - lll # Reports long lines
- varnamelen # Short names like db, id are idiomatic Go - intrange # intrange is a linter to find places where for loops could make use of an integer range
# Deprecated: the warning is attached to the old name, so it is - gochecknoglobals # Check that no global variables exist
# silenced by disabling that name, not by enabling the successor.
- wsl # Deprecated, replaced by wsl_v5 # Default/existing linters that are commonly useful
- gomodguard # Deprecated, replaced by gomodguard_v2 - govet
settings: - errcheck
lll: - staticcheck
line-length: 88 - unused
funlen: - ineffassign
lines: 80 - misspell
statements: 50 - revive
cyclop: - gosec
max-complexity: 15 - unconvert
dupl: - unparam
threshold: 100
depguard: linters-settings:
# Test-support code must not be compiled into the shipped binary. A lll:
# test-support package exists to hand a test privileges the program line-length: 120
# itself must never have, so a file that is not a test must not import
# one. Test files, and the files inside a package whose directory name nestif:
# ends in `test`, are where that code belongs, and are exempt. min-complexity: 4
#
# The deny list below is the one part of this file a repository is nlreturn:
# expected to extend, and the only part it may. depguard matches an block-size: 2
# import path against a list of prefixes, so it cannot be told "any path
# whose last segment ends in test"; a repository's own test-support revive:
# packages have to be named here one at a time, by full import path, rules:
# under a module path that differs from repository to repository. Add - name: var-naming
# them; change nothing else. arguments:
- []
- []
- "upperCaseConst=true"
tagliatelle:
case:
rules: rules:
test-support: json: snake
list-mode: lax yaml: snake
files: xml: snake
- "$all" bson: snake
- "!$test"
- "!**/*test/**" testifylint:
deny: enable-all: true
- pkg: net/http/httptest
desc: >- usetesting: {}
Test-support code belongs in test files and in packages whose
directory name ends in test, not in the shipped binary.
# Only decisions already recorded in the Go package defaults are
# listed here. Every entry matches the module path exactly.
gomodguard_v2:
blocked:
- module: github.com/rs/zerolog
recommendations:
- log/slog
reason: "Structured logging is stdlib log/slog."
# One entry per pre-fork module path, because the later releases
# are separate paths. A prefix match would be shorter but would
# also reach github.com/go-redis/redismock, the test double for
# the successor these entries recommend.
- module: github.com/go-redis/redis
recommendations:
- github.com/redis/go-redis/v9
reason: "Pre-fork module; use the maintained go-redis v9."
- module: github.com/go-redis/redis/v7
recommendations:
- github.com/redis/go-redis/v9
reason: "Pre-fork module; use the maintained go-redis v9."
- module: github.com/go-redis/redis/v8
recommendations:
- github.com/redis/go-redis/v9
reason: "Pre-fork module; use the maintained go-redis v9."
- module: github.com/sergi/go-diff
recommendations:
- github.com/aymanbagabas/go-udiff
reason: "No unified diff output; use go-udiff."
- module: github.com/hexops/gotextdiff
recommendations:
- github.com/aymanbagabas/go-udiff
reason: "Unmaintained fork; use go-udiff."
issues: issues:
max-issues-per-linter: 0 max-issues-per-linter: 0
max-same-issues: 0 max-same-issues: 0
exclude-rules:
# Exclude unused parameter warnings for cobra command signatures
- text: "parameter '(args|cmd)' seems to be unused"
linters:
- revive
# Allow ALL_CAPS constant names
- text: "don't use ALL_CAPS in Go names"
linters:
- revive
# Allow snake_case JSON tags for external API compatibility
- path: "internal/types/ris.go"
linters:
- tagliatelle
# Allow snake_case JSON tags for database models
- path: "internal/database/models.go"
linters:
- tagliatelle
# Allow generic package name for types that define data structures
- path: "internal/types/"
text: "avoid meaningless package names"
linters:
- revive
View File
+18 -29
View File
@@ -1,37 +1,22 @@
# Lint phase, built alone by script/lint. The linter is invoked directly # Lint stage — fast feedback on formatting and lint issues.
# rather than through `make lint`, which is itself a docker build and # The golangci-lint image bundles Go, gcc and make, so it can run go vet on
# would recurse into a daemon that does not exist in a build step. The # the CGO sqlite package and golangci-lint without extra installs.
# image is Debian-based and has Go and gcc, so it can check the cgo # golangci/golangci-lint:v2.7.2 (Go 1.25.5), 2026-09-21
# SQLite package without extra installs. FROM golangci/golangci-lint@sha256:5d6d5c70a61f1356adfd9dd6316ce286799fefc9d743421356ff1b00842368ba AS lint
# golangci/golangci-lint:v2.14.0 (Go 1.27.0), 2026-10-14
FROM golangci/golangci-lint@sha256:ad862ba6b3798cbe0fd9fd7408d498fd74fbd2623a92406b2fd3898faf0bf98f AS lint
WORKDIR /src WORKDIR /src
COPY go.mod go.sum ./ COPY go.mod go.sum ./
RUN go mod download RUN go mod download
COPY . .
RUN golangci-lint run --config .golangci.yml ./...
# Test phase, built alone by script/test. -race needs cgo and so a C
# compiler, which the Debian Go image ships. -short skips the live RIPE RIS
# feed test, so the suite needs no network. The rerun with -v only shows
# details: the build fails however it ends, because the first run failed.
# golang:1.24-bookworm, 2026-09-21
FROM golang@sha256:1a6d4452c65dea36aac2e2d606b01b4a029ec90cc1ae53890540ce6173ea77ac AS test
WORKDIR /src
COPY go.mod go.sum ./
RUN go mod download
COPY . . COPY . .
RUN go test -short -timeout 90s -race -cover ./... || \
{ echo "--- Rerunning with -v for details ---"; \
go test -short -timeout 90s -race -v ./...; exit 1; }
# Build stage. Nothing is wanted from the lint and test phases; the copies RUN make fmt-check
# are what make BuildKit build them first, so this stage cannot run unless RUN make lint
# lint and test passed.
# Build stage
# golang:1.24-bookworm, 2026-09-21 # golang:1.24-bookworm, 2026-09-21
FROM golang@sha256:1a6d4452c65dea36aac2e2d606b01b4a029ec90cc1ae53890540ce6173ea77ac AS builder FROM golang@sha256:1a6d4452c65dea36aac2e2d606b01b4a029ec90cc1ae53890540ce6173ea77ac AS builder
COPY --from=lint /src/go.sum /dev/null
COPY --from=test /src/go.sum /dev/null
# Install build dependencies (zstd for archive, gcc for CGO/sqlite3) # Install build dependencies (zstd for archive, gcc for CGO/sqlite3)
RUN apt-get update && apt-get install -y --no-install-recommends \ RUN apt-get update && apt-get install -y --no-install-recommends \
@@ -40,17 +25,21 @@ RUN apt-get update && apt-get install -y --no-install-recommends \
libc6-dev \ libc6-dev \
&& rm -rf /var/lib/apt/lists/* && rm -rf /var/lib/apt/lists/*
# A tar-stream context keeps the sender's file owners, which git refuses.
RUN git config --system --add safe.directory /src
WORKDIR /src WORKDIR /src
# Force BuildKit to run the lint stage before compiling or testing.
COPY --from=lint /src/go.sum /dev/null
# Copy everything # Copy everything
COPY . . COPY . .
# Vendor dependencies (must be after copying source) # Vendor dependencies (must be after copying source)
RUN go mod download && go mod vendor RUN go mod download && go mod vendor
# Run the test suite in the build stage: -race needs cgo and the C compiler
# installed above. The suite is offline (the live-feed test is opt-in).
RUN make test
# Build the binary with CGO enabled (required for sqlite3). The version the # Build the binary with CGO enabled (required for sqlite3). The version the
# page footer shows is the VERSION build argument when one is given, otherwise # page footer shows is the VERSION build argument when one is given, otherwise
# `git describe --tags --always` of the .git in the build context (git comes # `git describe --tags --always` of the .git in the build context (git comes
+11 -16
View File
@@ -144,8 +144,9 @@ routewatch/
- **Reconnection**: Exponential backoff (5s-320s) with reset after 30s of stable connection - **Reconnection**: Exponential backoff (5s-320s) with reset after 30s of stable connection
- **IP Lookup**: the most specific live route is found by looking up the - **IP Lookup**: the most specific live route is found by looking up the
address's prefix at each mask length, longest first, in the prefix index address's prefix at each mask length, longest first, in the prefix index
(at most 33 lookups for IPv4, 129 for IPv6); prefixes are stored in the (at most 25 lookups for IPv4, 113 for IPv6); prefixes are stored in the
text form Go's `net/netip` prints text form Go's `net/netip` prints. Routes shorter than /8 for IPv4 or /16
for IPv6, such as a default route, are not used.
### Database Schema ### Database Schema
@@ -273,30 +274,24 @@ standard: normalized scripts in `script/` are the entrypoints for the
development workflow, and the Makefile targets are thin shims that call development workflow, and the Makefile targets are thin shims that call
them. We provide: them. We provide:
Tests and the linter run only in Docker, as the `test` and `lint` phases of the - `script/bootstrap` — install all development dependencies (go,
`Dockerfile`; the image cannot be built unless both pass. Every `docker build` golangci-lint, Go module download)
in `script/` passes `--no-cache`, so each run executes every check.
- `script/bootstrap` — install all development dependencies (git, make, go, Go
module download); warns when docker is missing
- `script/setup` — make a fresh clone ready for development: runs - `script/setup` — make a fresh clone ready for development: runs
`script/bootstrap`, then `script/install-precommit` `script/bootstrap`, then `script/install-precommit`
- `script/projectname` — print the project name (used for the Docker - `script/projectname` — print the project name (used for the Docker
image tag) image tag)
- `script/test` — build the `test` phase alone, which runs - `script/test` — run the test suite
`go test -short -timeout 90s -race -cover ./...` (verbose rerun on failure). (`go test -short -timeout 30s -race -cover ./...`, verbose rerun on
The `-short` flag skips the live-network integration test so the run is failure). The `-short` flag skips the live-network integration test so the
deterministic and offline. default run is deterministic and offline.
- `script/lint` — build the `lint` phase alone, which runs golangci-lint - `script/lint` — run `go vet ./...` and `golangci-lint run`
v2.14.0 with `.golangci.yml`
- `script/fmt` — format all code (writes) - `script/fmt` — format all code (writes)
- `script/fmt-check` — check formatting (read-only) - `script/fmt-check` — check formatting (read-only)
- `script/check` — run `script/test`, `script/lint`, and - `script/check` — run `script/test`, `script/lint`, and
`script/fmt-check` `script/fmt-check`
- `script/docker` — build the Docker image tagged via - `script/docker` — build the Docker image tagged via
`script/projectname` `script/projectname`
- `script/cibuild` — CI entrypoint: runs `script/bootstrap` and - `script/cibuild` — CI entrypoint: `docker build .`
`script/check`, then builds the image
- `script/precommit` — pre-commit gate: `go mod tidy` + `go fmt` (must - `script/precommit` — pre-commit gate: `go mod tidy` + `go fmt` (must
not change files), then `script/check` not change files), then `script/check`
- `script/install-precommit` — install the git pre-commit hook that - `script/install-precommit` — install the git pre-commit hook that
+84 -355
View File
@@ -1,6 +1,6 @@
--- ---
title: Repository Policies title: Repository Policies
last_modified: 2026-10-04 last_modified: 2026-07-06
--- ---
This document covers repository structure, tooling, and workflow standards. Code This document covers repository structure, tooling, and workflow standards. Code
@@ -60,28 +60,17 @@ style conventions are in separate documents:
prerequisite since nvm requires bash. yarn is then pinned via prerequisite since nvm requires bash. yarn is then pinned via
`corepack prepare yarn@<version> --activate`. Never install "latest" or "lts"; `corepack prepare yarn@<version> --activate`. Never install "latest" or "lts";
always exact versions. `script/cibuild` runs the CI build: it changes to the always exact versions. `script/cibuild` runs the CI build: it changes to the
repo root, runs `script/bootstrap`, runs `script/check`, and builds the image repo root and runs `docker build .`; the Gitea workflow calls it. Four further
with the version; the Gitea workflow calls it. **`script/cibuild` runs scripts are our own extensions to the standard: `script/check` runs
`script/bootstrap` first**, because the workflow checks out the repo and runs `script/test`, `script/lint`, and `script/fmt-check`; `script/precommit` is
nothing else, while `script/fmt-check` runs the formatter on the host: on a what the git pre-commit hook runs, and it calls `script/check`;
pristine checkout with nothing installed the run dies there, after the `script/install-precommit` installs the git pre-commit hook (the `make hooks`
containerised gates have passed. **The bootstrap alone is not enough**: target shims to it); and `script/projectname` (literally that filename) simply
`script/bootstrap` installs node and yarn under nvm and leaves neither on the outputs the project's name. Scripts that need the name call
`PATH` of the shell that called it, so a bare `yarn` still exits 127. The host `script/projectname` — e.g. `script/docker` assembles its image tag from it —
entrypoints that need yarn — `script/fmt` and `script/fmt-check` — therefore so those scripts stay byte-identical across all repos. Repo-type-specific
source nvm for the pinned node version before invoking it, exactly as pre-commit extras (e.g. `go mod tidy` verification in Go repos) belong in
`script/bootstrap`'s own install step does. A runner carrying nothing but `script/precommit`, not in the hook itself. Model scripts are at
docker and git then gets through `script/check`. Four further scripts are our
own extensions to the standard: `script/check` runs `script/test`,
`script/lint` and `script/fmt-check`; `script/precommit` is what the git
pre-commit hook runs, and it calls `script/check`; `script/install-precommit`
installs the git pre-commit hook (the `make hooks` target shims to it); and
`script/projectname` (literally that filename) simply outputs the project's
name. Scripts that need the name call `script/projectname` — e.g.
`script/docker` assembles its image tag from it — so those scripts stay
byte-identical across all repos. Repo-type-specific pre-commit extras (e.g.
`go mod tidy` verification in Go repos) belong in `script/precommit`, not in
the hook itself. Model scripts are at
`https://git.eeqj.de/sneak/prompts/raw/branch/main/script/<name>`. The README `https://git.eeqj.de/sneak/prompts/raw/branch/main/script/<name>`. The README
must document the provided scripts in an **Entrypoints** section (see the must document the provided scripts in an **Entrypoints** section (see the
README requirements below). README requirements below).
@@ -100,198 +89,87 @@ style conventions are in separate documents:
contributor should be able to understand the entire development workflow by contributor should be able to understand the entire development workflow by
reading the Makefile. reading the Makefile.
- Every repo should have a `Dockerfile`, and it carries the repo's gates: a - Every repo should have a `Dockerfile`. All Dockerfiles must run `make check`
`lint` phase and a `test` phase, with the final stage depending on both so the as a build step so the build fails if the branch is not green. For non-server
image cannot be built unless they pass. For non-server repos the final stage repos, the Dockerfile should bring up a development environment and run
brings up a development environment; for server repos it is the runtime image. `make check`. For server repos, `make check` should run as an early build
The gate phases and the build stage start from their pinned base images and stage before the final image is assembled. Dockerfiles install development
install what those images lack either inline, as the canonical Go `Dockerfile` prerequisites by running `script/bootstrap` rather than duplicating installs
below does for `git`, or by running `script/bootstrap`, as the `prompts` inline; COPY `script/` and the dependency manifests (`package.json` +
repo's own `Dockerfile` does for its yarn packages. The development `yarn.lock`, `go.mod` + `go.sum`, etc.) before running it so the bootstrap
environment stage installs development prerequisites by running layer stays cached until dependencies change.
`script/bootstrap` rather than duplicating its installs inline. A stage that
runs `script/bootstrap` COPYs `script/` and the dependency manifests
(`package.json` + `yarn.lock`, `go.mod` + `go.sum`, etc.) before running it.
- **Linting and testing run in Docker, as phases of the `Dockerfile`.** There is - **Dockerfiles must use a separate lint stage for fail-fast feedback.** Go
no separate lint file. `script/lint` and `script/test` each build one phase repos use a multistage build where linting runs in an independent stage based
and nothing else: on the `golangci/golangci-lint` image (pinned by hash). This stage runs
`make fmt-check` and `make lint` before the full build begins. The build stage
then declares an explicit dependency on the lint stage via
`COPY --from=lint /src/go.sum /dev/null`, which forces BuildKit to complete
linting before proceeding to compilation and tests. This ensures lint failures
surface in seconds rather than minutes, without blocking on dependency
download or compilation in the build stage.
```sh The standard pattern for a Go repo Dockerfile is:
docker build --no-cache --target lint -t "$(script/projectname)-lint" .
docker build --no-cache --target test -t "$(script/projectname)-test" .
```
**A stage that is not the last one in the file is built only when the final
stage's chain depends on it, or when `--target` names it.** That is why the
two gates are always invoked by name here, and why the final stage carries a
`COPY --from=` of a harmless file from each of them: without that edge a
plain `docker build .` builds the last stage alone and exits 0 having linted
and tested nothing.
**Every `docker build` in `script/` is tagged**, here and in
`script/cibuild` and `script/docker`. An untagged build leaves a dangling
image behind on every invocation, on every developer host and every CI
runner; a tagged one replaces the previous image.
Inside a phase the tool is invoked directly — `golangci-lint`, `go test`,
`eslint`, `prettier` — never through `make lint` or `script/test`, which are
themselves a `docker build` and would recurse into a daemon that does not
exist in a build step. Formatting is the exception and stays on the host:
`script/fmt` writes the working tree, and `script/fmt-check` is its
read-only twin.
**No lint verdict may come from a host invocation of the linter.** On a
shared host golangci-lint reads a result cache keyed on file content rather
than location, so a second checkout of the same content is served the first
one's findings, and a host-global lock in `$TMPDIR` makes concurrent runs
exit non-zero with `parallel golangci-lint is running` — a status a caller
cannot tell from real findings. Both have produced wrong verdicts in this
org, in both directions. A container has its own cache, its own `TMPDIR` and
a digest-pinned binary, so neither is reachable.
- **Any build that runs checks is built with `--no-cache`.** Docker invalidates
a `COPY` layer only when the copied content changes, so on an unchanged tree
the check `RUN` is served from cache, nothing executes, and the build still
exits 0. Every `docker build` in `script/` therefore passes `--no-cache`:
`script/lint`, `script/test`, `script/cibuild` and `script/docker` are the
four, and there is no fifth — `script/check` runs the two gate phases and
`script/fmt-check`, and builds no image of its own. A bare `docker build .` is
not evidence that anything ran: a sub-second build reporting success is a
cache hit, not a result. Never invalidate by pruning — `docker builder prune`
and friends destroy a build cache shared with every other build on the host.
When a check is added or changed, prove it works by planting a defect it must
catch and watching the run fail on it, then revert the defect. A green run
alone shows neither that the check ran nor that it covers what it should.
- **The gate phases are separate stages, and the build stage depends on both.**
The lint phase is based on the `golangci/golangci-lint` image (pinned by
hash), so lint failures surface in seconds rather than after a full compile,
and the test phase is based on the Debian Go image. The canonical Go repo
`Dockerfile`:
```dockerfile ```dockerfile
# Lint phase # Lint stage — fast feedback on formatting and lint issues
# golangci/golangci-lint:v2.x.x, YYYY-MM-DD # golangci/golangci-lint:v2.x.x, YYYY-MM-DD
FROM golangci/golangci-lint@sha256:... AS lint FROM golangci/golangci-lint@sha256:... AS lint
WORKDIR /src WORKDIR /src
COPY go.mod go.sum ./ COPY go.mod go.sum ./
RUN go mod download RUN go mod download
COPY . . COPY . .
RUN golangci-lint run --config .golangci.yml ./... RUN make fmt-check
RUN make lint
# Test phase. -race needs cgo and so a C compiler, which the Debian Go # Build stage
# image ships and the alpine one does not.
# golang:1.x, YYYY-MM-DD
FROM golang@sha256:... AS test
WORKDIR /src
COPY go.mod go.sum ./
RUN go mod download
COPY . .
RUN go test -timeout 90s -race -cover ./... || \
{ echo "--- Rerunning with -v for details ---"; \
go test -timeout 90s -race -v ./...; exit 1; }
# Build stage. Nothing is wanted from either phase above; the copies
# are what make BuildKit build them first, so this stage cannot run
# unless lint and test passed.
# golang:1.x-alpine, YYYY-MM-DD # golang:1.x-alpine, YYYY-MM-DD
FROM golang@sha256:... AS builder FROM golang@sha256:... AS builder
COPY --from=lint /src/go.sum /dev/null
COPY --from=test /src/go.sum /dev/null
RUN apk add --no-cache git
# A tar-stream context keeps the sender's file owners, which git refuses.
RUN git config --system --add safe.directory /src
WORKDIR /src WORKDIR /src
# Force BuildKit to run the lint stage before proceeding
COPY --from=lint /src/go.sum /dev/null
COPY go.mod go.sum ./ COPY go.mod go.sum ./
RUN go mod download RUN go mod download
COPY . . COPY . .
RUN make test
# The VERSION build arg when one is given, otherwise ARG VERSION=dev
# `git describe --tags --always` on the .git in the build context. With RUN CGO_ENABLED=0 go build -trimpath \
# .git present, a version that is still empty, dev or unknown fails the -ldflags="-s -w -X main.Version=${VERSION}" \
# build: git is missing or could not read the checkout. -o /app ./cmd/app/
ARG VERSION
RUN VERSION="${VERSION:-$(git describe --tags --always)}"; \
if [ -e .git ]; then \
case "$VERSION" in ""|dev|unknown) \
echo "version is '$VERSION' although .git is present" >&2; \
exit 1 ;; \
esac; \
fi; \
CGO_ENABLED=0 go build -trimpath \
-ldflags="-s -w -X main.Version=${VERSION}" \
-o /app ./cmd/app/
# Runtime stage, and the last one # Runtime stage
FROM alpine@sha256:... FROM alpine@sha256:...
COPY --from=builder /app /usr/local/bin/app COPY --from=builder /app /usr/local/bin/app
ENTRYPOINT ["app"] ENTRYPOINT ["app"]
``` ```
Key points: Key points:
- The lint phase uses the `golangci/golangci-lint` image directly (it has - The lint stage uses the `golangci/golangci-lint` image directly (it
both Go and the linter), so nothing needs installing. includes both Go and the linter), so there is no need to install the
- `COPY --from=<phase> /src/go.sum /dev/null` is a no-op copy whose only linter separately.
purpose is the ordering edge. BuildKit runs stages in parallel by default, - `COPY --from=lint /src/go.sum /dev/null` is a no-op file copy that creates
and a stage nothing depends on is not built at all, so without these two a stage dependency. BuildKit runs stages in parallel by default; without
lines a red gate would not fail the build. this line, the build stage would not wait for lint to finish and a lint
- Keep the runtime stage last, and if you add a stage after it, give it the failure might not fail the overall build.
same two copies. A plain `docker build .` builds the last stage's chain
and nothing else.
- If the project uses `//go:embed` directives that reference build artifacts - If the project uses `//go:embed` directives that reference build artifacts
(e.g. a web frontend compiled in a separate stage), the lint phase must (e.g. a web frontend compiled in a separate stage), the lint stage must
create placeholder files so the embed directives resolve. Example: create placeholder files so the embed directives resolve. Example:
`RUN mkdir -p web/dist && touch web/dist/index.html web/dist/style.css`. `RUN mkdir -p web/dist && touch web/dist/index.html web/dist/style.css`.
- If the project requires CGO or system libraries for linting, install them The lint stage should not depend on the actual build output — it exists to
in the lint phase. The `golangci/golangci-lint` image is Debian-based and fail fast.
has no `apk`, so install with `apt-get` under the Debian package name - If the project requires CGO or system libraries for linting (e.g.
(`libvips-dev`, where alpine says `vips-dev`), and delete the package `vips-dev`), install them in the lint stage with `apk add`.
lists in the same `RUN`, so the layer does not keep them: - The build stage runs `make test` after compilation setup. Tests run in the
build stage, not the lint stage, because they may require compiled
```dockerfile artifacts or heavier dependencies.
RUN apt-get update \
&& apt-get install -y --no-install-recommends libvips-dev \
&& rm -rf /var/lib/apt/lists/*
```
- `.dockerignore` lets `.git` into the build context. It keeps out every git
`config` at any depth (`**/.git/config`, `**/.git/modules/**/config`): the
repository's own, each submodule's under `.git/modules/`, and that of a
submodule keeping its own `.git` directory. `git describe` does not need
them, and each can hold a credential: a password in a remote URL, or the
token the CI checkout step stores there. A submodule whose name has a
`config` segment (`config`, `deploy/config`, `config/lib`) loses its whole
git directory to `**/.git/modules/**/config`, and Go's version stamping
then fails the build: give it a name without that segment
(`git submodule add --name`). The stage that compiles has `git` (the
Debian Go image has it; an alpine one needs `apk add --no-cache git`) and
takes the version from the `VERSION` build argument when one is given,
otherwise from `git describe --tags --always`. That gives the tag on a
tagged commit; on a later commit, the tag, the number of commits since it
and the short commit (`v1.2.3-4-gabc1234`); and the short commit when no
tag is reachable. The stage that compiles also marks its working directory
safe for git (`git config --system --add safe.directory /src`): a context
sent as a tar stream keeps the sender's file owners, and git refuses a
checkout owned by another user, so the version would come out empty.
`ARG VERSION` has no default, and the build fails if the context carries
`.git` and the version still comes out empty, `dev` or `unknown`. A plain
`docker build .` with no build arguments must succeed; a Dockerfile that
refuses an empty build argument drops that refusal and keeps the argument.
- Every repo should have a Gitea Actions workflow (`.gitea/workflows/`) that - Every repo should have a Gitea Actions workflow (`.gitea/workflows/`) that
runs `script/cibuild` on push, and checks out the repo as its only other step. runs `script/cibuild` (which runs `docker build .`) on push. Since the
That script bootstraps, runs the gate phases, and then builds the image, so a Dockerfile already runs `make check`, a successful build implies all checks
successful run means every check passed; a bare `docker build .` does not pass.
carry the same guarantee, because its gate phases may come from the cache. The
image build is uncached and so runs the gate phases a second time. That is the
price of the rule above, and it is worth paying: the image that ships is built
from a run of its own gates rather than from a cache entry. A separate
workflow limited to `main` by a `branches` list under `on: push` cannot be
checked by review: to try a change to it, add the feature branch to that list
and push, then remove the branch from the list again before merging. Keep any
job in it that publishes behind `if: github.ref_name == 'main'`, so the run
from the feature branch publishes nothing.
- Use platform-standard formatters: `black` for Python, `prettier` for - Use platform-standard formatters: `black` for Python, `prettier` for
JS/CSS/Markdown/HTML, `go fmt` for Go. Always use default configuration with JS/CSS/Markdown/HTML, `go fmt` for Go. Always use default configuration with
@@ -311,21 +189,14 @@ style conventions are in separate documents:
module under test to verify it compiles/parses. There is no excuse for module under test to verify it compiles/parses. There is no excuse for
`make test` to be a no-op. `make test` to be a no-op.
- `make test` must complete in under 60 seconds. That is the hard cap, and a - `make test` must complete in under 20 seconds. Add a 30-second timeout in the
suite that exceeds it fails. Under 20 seconds is the target. A suite between Makefile.
20 and 60 seconds is still green, but the overage must be filed as an
improvement bug against that repo. Add a 90-second timeout to the test
invocation (`go test -timeout 90s`). The backstop deliberately sits above the
hard cap so that it catches a genuinely hung test rather than a merely slow
one.
- **The test command should use the conditional verbose rerun pattern.** Run - **`make test` should use the conditional verbose rerun pattern.** Run tests
tests without `-v` (verbose) first. If tests fail, automatically rerun with without `-v` (verbose) first. If tests fail, automatically rerun with `-v` to
`-v` to show full output. This keeps CI logs and `docker build` output clean show full output. This keeps CI logs and `docker build` output clean on
on success (just package/suite summaries) while providing full diagnostic success (just package/suite summaries) while providing full diagnostic detail
detail on failure (every test case, every assertion). The command lives in the on failure (every test case, every assertion). The general shell pattern:
`test` phase of the `Dockerfile`, since `script/test` builds that phase; the
Makefile form below is the same pattern for any repo-local invocation:
```makefile ```makefile
test: test:
@@ -338,26 +209,11 @@ style conventions are in separate documents:
```makefile ```makefile
test: test:
@go test -count=1 -timeout 90s -race -cover ./... || \ @go test -timeout 30s -race -cover ./... || \
{ echo "--- Rerunning with -v for details ---"; \ { echo "--- Rerunning with -v for details ---"; \
go test -count=1 -timeout 90s -race -v ./...; exit 1; } go test -timeout 30s -race -v ./...; exit 1; }
``` ```
`-count=1` is required on both invocations: it defeats Go's test _result_
cache, so neither run can report a stored pass in place of running the
tests. It leaves the build cache alone, so it costs the runtime of the suite
and no recompilation.
That cache is Go's own, separate from Docker's layer cache. Go stores a
passing result in its cache directory (`GOCACHE`), and when the same tests
run again on unchanged code it prints that result, marked `(cached)`,
without running them. That matters on a developer's machine, where this
target runs and the directory lasts from one run to the next. The `test`
phase of the `Dockerfile` needs no `-count=1`: its base image holds no
result for this repo's tests and nothing before its `go test` step runs a
test, so there is nothing to replay. `--no-cache` (above) is what makes that
step run on an unchanged tree.
Python example: Python example:
```makefile ```makefile
@@ -383,84 +239,10 @@ style conventions are in separate documents:
must be in `.gitignore`. No exceptions. must be in `.gitignore`. No exceptions.
- `.gitignore` should be comprehensive from the start: OS files (`.DS_Store`), - `.gitignore` should be comprehensive from the start: OS files (`.DS_Store`),
editor files (`.swp`, `*~`), in-repo agent scratch directories (`.claude/`), editor files (`.swp`, `*~`), language build artifacts, and `node_modules/`.
language build artifacts, and `node_modules/`. Fetch the standard `.gitignore` Fetch the standard `.gitignore` from
from `https://git.eeqj.de/sneak/prompts/raw/branch/main/.gitignore` when `https://git.eeqj.de/sneak/prompts/raw/branch/main/.gitignore` when setting up
setting up a new repo. These patterns are written to `.gitignore`'s own a new repo.
semantics, in which an unanchored pattern already matches at every depth; they
are not a `.dockerignore` and must not be transplanted into one unmodified.
- **`.dockerignore` does not use `.gitignore` semantics, and copying patterns
across unmodified leaves secrets in the build context.** Docker matches with
`moby/patternmatcher`: `filepath.Match` semantics plus a `**` extension, so
`*` does not cross `/` and a pattern without a leading `**/` is anchored at
the build-context root. A `.dockerignore` listing `.env`, `*.pem` and `*.key`
therefore excludes only the copies at the repository root, while `config/.env`
and `certs/server.key` still reach the context and can land in an image layer
— which is more dangerous than a short file with no secret patterns at all,
because it reads as solved and stops anyone looking. Give every
depth-independent pattern the `**/` prefix and leave only genuinely
root-anchored entries unprefixed: `.claude`, and the repo's own host-built
binary, written `/myapp` and never `**/myapp`, which would also match
`cmd/myapp/` and delete the package directory from the context. Matching is
case-sensitive, and an ALL-CAPS twin per pattern still misses `Server.Key`, so
secret names use character ranges — `**/*.[kK][eE][yY]`, `**/*.[pP][eE][mM]`,
and likewise for `.envrc` and the extensionless SSH keys. Where such a pattern
also catches something the build needs, re-include it with a negation
(`!docs/example.env`); deleting the pattern reopens the exposure for every
other file it covers. Fetch the standard `.dockerignore` from
`https://git.eeqj.de/sneak/prompts/raw/branch/main/.dockerignore` and extend
it with the repo's own artifacts.
- **In-repo agent scratch belongs in both files, written to each file's own
semantics.** `.claude/` holds one worktree per in-flight agent — an entire
additional checkout of the repo — so under `COPY . .` the build context
inflates by a multiple of the repo and another session's unreviewed work can
be copied into an image layer. In `.gitignore` the entry is `.claude/`,
unanchored. In `.dockerignore` it is `.claude`, anchored and with **no** `**/`
prefix, because the prefixed form would also delete any nested directory of
that name from the build. Anchoring carries a known gap that the canonical
`.dockerignore` states in its own comment, since consuming repos receive the
file and not the tracker: the directory is created in the agent's working
directory, so a repo running agents in subdirectories still ships
`services/api/.claude/` and must add its own anchored entry there.
- **A plain `docker build .` of a clone stamps the version that
`git describe --tags --always` gives**, derived from the `.git` in the build
context as the canonical `Dockerfile` above shows. Without its failure check,
a missing `git` or an unreadable checkout would leave `-X main.Version=` empty
and the build would still exit 0. `script/docker` and `script/cibuild` pass
the version they compute on the host; it takes precedence. They do this
byte-identically across repos:
```sh
# Own line: a failing command substitution inside an argument does not
# trip `set -e`, so the inline form degrades to an empty constant.
version="$(git describe --tags --always --dirty 2>/dev/null || true)"
[ -n "$version" ] || version="unknown"
docker build --no-cache \
--build-arg VERSION="$version" \
-t "$(script/projectname)" .
```
`--always` makes an untagged repo yield an abbreviated commit hash rather
than failing, and the `[ -n "$version" ]` line is the single place the
fallback is applied — a live check that fires on a build from an export with
no `.git` and on a repository with no commits yet. Do not fold it into the
substitution as `|| echo unknown`, which makes the guard unreachable. The
Dockerfile's side is `ARG VERSION` in the stage that compiles, declared
there because `ARG` is stage-scoped; passing `VERSION` to a repo whose
Dockerfile declares no such `ARG` is ignored and costs nothing, which is why
the scripts stay byte-identical. One consequence for CI: the standard
checkout action clones shallow and fetches no tags, so a repo that embeds a
tag-derived version must set `fetch-depth: 0` on its checkout step.
- **Verify `.dockerignore` by enumerating the image, not by reading the
patterns.** Plant files at the root _and_ at least two directories deep, build
a probe image that does `COPY . .`, and list what actually landed
(`docker run --rm --entrypoint find IMAGE /app`). The `transferring context`
size is not a substitute: a nested secret is a few bytes, and BuildKit
transfers only the delta from the previous build.
- **No build artifacts in version control.** Code-derived data (compiled - **No build artifacts in version control.** Code-derived data (compiled
bundles, minified output, generated assets) must never be committed to the bundles, minified output, generated assets) must never be committed to the
@@ -476,56 +258,9 @@ style conventions are in separate documents:
- Make all changes on a feature branch. You can do whatever you want on a - Make all changes on a feature branch. You can do whatever you want on a
feature branch. feature branch.
- `.golangci.yml` is standardized. The vendored copy in a consuming repo must - `.golangci.yml` is standardized and must _NEVER_ be modified by an agent, only
_NEVER_ be modified by an agent: fetch it from manually by the user. Fetch from
`https://git.eeqj.de/sneak/prompts/raw/branch/main/.golangci.yml` and keep it `https://git.eeqj.de/sneak/prompts/raw/branch/main/.golangci.yml`.
byte-identical, so that no repo can quietly loosen its own linting. Linter
configuration changes are made to the canonical copy in the `prompts` repo and
reach consuming repos by re-vendoring; an agent may open a PR against
canonical, which only the user merges. One list is exempt from byte-identity,
because it cannot be written once for every repo: the `deny` list of the
`test-support` depguard rule, where a repo names its own test-support packages
by full import path. A repo adds entries there and changes nothing else, and a
re-vendor carries its entries forward. The canonical golangci-lint version is
v2.14.0 (released 2026-09-24), pinned as the digest of the lint phase's base
image
(`golangci/golangci-lint@sha256:ad862ba6b3798cbe0fd9fd7408d498fd74fbd2623a92406b2fd3898faf0bf98f`,
which reports `2.14.0 built with go1.27.0 from 114493f9`). A module's `go`
directive must not name a newer Go minor version than the one golangci-lint
was built with, or golangci-lint refuses to lint it: this release lints
`go 1.27.1` but not `go 1.28`. That digest is the only pin, since no repo
installs golangci-lint on the host. A repo sets the lint phase digest to the
one named here and re-vendors `.golangci.yml` in the same commit, whichever of
the two prompted the change: the canonical copy can name linters that an older
golangci-lint rejects, and a newer golangci-lint can add linters that
`default: all` switches on until the canonical copy disables them.
- **`script/bootstrap` installs a pinned tool by comparing versions, never by
testing presence.** An `if ! command -v <tool>; then install; fi` guard tests
`PATH` only, so on an already-provisioned machine the pin is inert and a
version bump is a silent no-op — while the Dockerfile, installing into a clean
image, gets the pinned version, so a local `make check` and `make docker` can
disagree about what the tool even is. The canonical form:
- compares the installed version against the pin over the **whole** version
token; a parser that stops at the first `-` reports `2.12.2` for a host
running `2.12.2-rc1` and skips the install;
- treats absent, non-zero, empty or unrecognised `--version` output as a
mismatch, so the failure direction is a redundant install and never a
skipped one;
- after installing, re-resolves the binary the way callers do — `hash -r`,
then through `PATH`, not through the directory the installer wrote to —
and fails naming the resolved path, since an install that a shadowing
binary hides succeeds while changing nothing any caller sees;
- is actually called, and prints the version on both success paths: a
function defined and never invoked has the same exit status and the same
empty output as one that worked.
Keep it POSIX sh: no arrays, no `[[`, no `grep -P`.
A Go tool a repo needs on the host is installed with `go install` pinned to
a commit hash (`go install <package>@<commit hash>`). It is never tracked as
a `go.mod` tool dependency or through a `tools.go` file, either of which
pulls the tool's own dependencies into the repo's `go.mod` and `go.sum`.
- When pinning images or packages by hash, add a comment above the reference - When pinning images or packages by hash, add a comment above the reference
with the version and date (YYYY-MM-DD). with the version and date (YYYY-MM-DD).
@@ -639,14 +374,12 @@ style conventions are in separate documents:
settings. settings.
- Avoid putting files in the repo root unless necessary. Root should contain - Avoid putting files in the repo root unless necessary. Root should contain
only project-level config files (`README.md`, `AGENTS.md`, `Makefile`, only project-level config files (`README.md`, `Makefile`, `Dockerfile`,
`Dockerfile`, `LICENSE`, `.gitignore`, `.editorconfig`, `REPO_POLICIES.md`, `LICENSE`, `.gitignore`, `.editorconfig`, `REPO_POLICIES.md`, and
and language-specific config). Everything else goes in a subdirectory. language-specific config). Everything else goes in a subdirectory. Canonical
Canonical subdirectory names: subdirectory names:
- `bin/` — executable scripts and tools - `bin/` — executable scripts and tools
- `cmd/` — Go command entrypoints; thin only: one `main.go` per binary whose - `cmd/` — Go command entrypoints
body is a single call into `internal/` or `pkg/`, no project logic in
`cmd/`
- `configs/` — configuration templates and examples - `configs/` — configuration templates and examples
- `deploy/` — deployment manifests (k8s, compose, terraform) - `deploy/` — deployment manifests (k8s, compose, terraform)
- `docs/` — documentation and markdown (README.md stays in root) - `docs/` — documentation and markdown (README.md stays in root)
@@ -673,7 +406,3 @@ style conventions are in separate documents:
- Go: `go.mod`, `go.sum`, `.golangci.yml` - Go: `go.mod`, `go.sum`, `.golangci.yml`
- JS: `package.json`, `yarn.lock`, `.prettierrc`, `.prettierignore` - JS: `package.json`, `yarn.lock`, `.prettierrc`, `.prettierignore`
- Python: `pyproject.toml` - Python: `pyproject.toml`
- Guidance for coding agents lives in one `AGENTS.md` at the repository root. It
is never committed under a file or directory named after one agent tool, such
as `CLAUDE.md` or `.claude/`, and never split into separate memory files.
+2 -7
View File
@@ -10,9 +10,8 @@
# Status # Status
pre-1.0. No git tags. The linter and the tests are phases of the Docker pre-1.0. No git tags. The Docker build runs the format check, the linter
build, and the Gitea workflow runs `script/cibuild` (the format check, both and the tests, and the Gitea workflow runs that build on every push. The
phases, then the image) on every push. The
image sets memory ceilings for a 5 GiB container (README "Memory") and the image sets memory ceilings for a 5 GiB container (README "Memory") and the
README says how to run it under upaas (README "Running under upaas"). A README says how to run it under upaas (README "Running under upaas"). A
35-hour run of `3898daa` on the live feed peaked at about 1 GiB, without a 35-hour run of `3898daa` on the live feed peaked at about 1 GiB, without a
@@ -28,10 +27,6 @@ real 5 GiB limit (https://git.eeqj.de/sneak/routewatch/issues/3).
# Completed Steps # Completed Steps
- 2026-10-14: the shared files are re-vendored from `sneak/prompts` at
`dd4027b`: golangci-lint v2.14.0 and its configuration, lint and test as
separate uncached Dockerfile phases, agent guidance in `AGENTS.md`; the code
is fixed for what the new linter configuration finds (closes #52)
- 2026-10-03: `/api/v1/stats` serves the prefix distribution from memory, - 2026-10-03: `/api/v1/stats` serves the prefix distribution from memory,
seeded at startup and adjusted on every live-route write, so a request no seeded at startup and adjusted on every live-route write, so a request no
longer reads every live route (closes #30) longer reads every live route (closes #30)
+7 -22
View File
@@ -2,10 +2,8 @@
package main package main
import ( import (
"context"
"encoding/csv" "encoding/csv"
"encoding/json" "encoding/json"
"errors"
"io" "io"
"log" "log"
"net/http" "net/http"
@@ -29,17 +27,13 @@ func main() {
// Fetch the CSV data // Fetch the CSV data
log.Println("Fetching AS CSV data...") log.Println("Fetching AS CSV data...")
resp, err := http.Get(asCsvURL)
req, err := http.NewRequestWithContext(
context.Background(), http.MethodGet, asCsvURL, nil)
if err != nil {
log.Fatalf("Failed to fetch CSV: %v", err)
}
resp, err := http.DefaultClient.Do(req)
if err != nil { if err != nil {
log.Fatalf("Failed to fetch CSV: %v", err) log.Fatalf("Failed to fetch CSV: %v", err)
} }
defer func() {
_ = resp.Body.Close()
}()
if resp.StatusCode != http.StatusOK { if resp.StatusCode != http.StatusOK {
log.Fatalf("Failed to fetch CSV: HTTP %d", resp.StatusCode) log.Fatalf("Failed to fetch CSV: HTTP %d", resp.StatusCode)
@@ -54,20 +48,17 @@ func main() {
log.Fatalf("Failed to read CSV header: %v", err) log.Fatalf("Failed to read CSV header: %v", err)
} }
if len(header) != 3 || header[0] != "asn" || header[1] != "handle" || if len(header) != 3 || header[0] != "asn" || header[1] != "handle" || header[2] != "description" {
header[2] != "description" {
log.Fatalf("Unexpected CSV header: %v", header) log.Fatalf("Unexpected CSV header: %v", header)
} }
// Read all records // Read all records
var asInfos []ASInfo var asInfos []ASInfo
for { for {
record, err := reader.Read() record, err := reader.Read()
if errors.Is(err, io.EOF) { if err == io.EOF {
break break
} }
if err != nil { if err != nil {
log.Printf("Error reading CSV record: %v", err) log.Printf("Error reading CSV record: %v", err)
@@ -95,18 +86,12 @@ func main() {
}) })
} }
// Closed here rather than deferred: log.Fatalf exits without running
// deferred calls.
_ = resp.Body.Close()
log.Printf("Parsed %d AS records", len(asInfos)) log.Printf("Parsed %d AS records", len(asInfos))
// Convert to JSON and output to stdout // Convert to JSON and output to stdout
encoder := json.NewEncoder(os.Stdout) encoder := json.NewEncoder(os.Stdout)
encoder.SetIndent("", "\t") encoder.SetIndent("", "\t")
if err := encoder.Encode(asInfos); err != nil {
err = encoder.Encode(asInfos)
if err != nil {
log.Fatalf("Failed to encode JSON: %v", err) log.Fatalf("Failed to encode JSON: %v", err)
} }
} }
+7 -10
View File
@@ -26,17 +26,9 @@ func main() {
// Register raw message handler that prints to stdout // Register raw message handler that prints to stdout
s.RegisterRawHandler(func(line string) { s.RegisterRawHandler(func(line string) {
_, _ = fmt.Fprintln(os.Stdout, line) fmt.Println(line)
}) })
// Start streaming before any defer: log.Fatal exits without running
// deferred calls.
err := s.Start(context.Background())
if err != nil {
log.Fatal("Failed to start streamer:", err)
}
defer s.Stop()
// Set up context with cancellation // Set up context with cancellation
ctx, cancel := context.WithCancel(context.Background()) ctx, cancel := context.WithCancel(context.Background())
defer cancel() defer cancel()
@@ -44,13 +36,18 @@ func main() {
// Handle signals // Handle signals
sigChan := make(chan os.Signal, 1) sigChan := make(chan os.Signal, 1)
signal.Notify(sigChan, os.Interrupt, syscall.SIGTERM) signal.Notify(sigChan, os.Interrupt, syscall.SIGTERM)
go func() { go func() {
<-sigChan <-sigChan
log.Println("Received shutdown signal") log.Println("Received shutdown signal")
cancel() cancel()
}() }()
// Start streaming
if err := s.Start(); err != nil {
log.Fatal("Failed to start streamer:", err)
}
defer s.Stop()
// Wait for context cancellation // Wait for context cancellation
<-ctx.Done() <-ctx.Done()
} }
+12 -27
View File
@@ -2,7 +2,6 @@
package config package config
import ( import (
"errors"
"fmt" "fmt"
"os" "os"
"path/filepath" "path/filepath"
@@ -28,12 +27,6 @@ const (
maxPort = 65535 maxPort = 65535
) )
var (
errRelativeXDGDataHome = errors.New("XDG_DATA_HOME must be an absolute path")
errUnsupportedOS = errors.New("unsupported operating system")
errInvalidPort = errors.New("PORT must be a whole number")
)
// Config holds configuration for the entire application // Config holds configuration for the entire application
type Config struct { type Config struct {
// StateDir is the directory for all application state (database, snapshots) // StateDir is the directory for all application state (database, snapshots)
@@ -45,14 +38,11 @@ type Config struct {
// MaxRuntime is the maximum runtime (0 = run forever) // MaxRuntime is the maximum runtime (0 = run forever)
MaxRuntime time.Duration MaxRuntime time.Duration
// EnableBatchedDatabaseWrites enables batched database operations for // EnableBatchedDatabaseWrites enables batched database operations for better performance
// better performance
EnableBatchedDatabaseWrites bool EnableBatchedDatabaseWrites bool
// RouteExpirationTimeout is how long a route can go without being // RouteExpirationTimeout is how long a route can go without being refreshed before expiring
// refreshed before expiring // Default is 2 hours which is conservative for BGP (typical BGP hold time is 90-180 seconds)
// Default is 2 hours which is conservative for BGP (typical BGP hold
// time is 90-180 seconds)
RouteExpirationTimeout time.Duration RouteExpirationTimeout time.Duration
} }
@@ -69,14 +59,11 @@ func New() (*Config, error) {
} }
return &Config{ return &Config{
StateDir: stateDir, StateDir: stateDir,
Port: port, Port: port,
// Run forever by default MaxRuntime: 0, // Run forever by default
MaxRuntime: 0, EnableBatchedDatabaseWrites: true, // Enable batching by default
// Enable batching by default RouteExpirationTimeout: defaultRouteExpirationMinutes * time.Minute, // For active route monitoring
EnableBatchedDatabaseWrites: true,
// For active route monitoring
RouteExpirationTimeout: defaultRouteExpirationMinutes * time.Minute,
}, nil }, nil
} }
@@ -102,7 +89,7 @@ func getStateDirectory() (string, error) {
// the database somewhere unexpected. // the database somewhere unexpected.
xdgData := os.Getenv("XDG_DATA_HOME") xdgData := os.Getenv("XDG_DATA_HOME")
if xdgData != "" && !filepath.IsAbs(xdgData) { if xdgData != "" && !filepath.IsAbs(xdgData) {
return "", fmt.Errorf("%w, got %q", errRelativeXDGDataHome, xdgData) return "", fmt.Errorf("XDG_DATA_HOME must be an absolute path, got %q", xdgData)
} }
// Unix-like: /var/lib/berlin.sneak.app.routewatch if root, else XDG_DATA_HOME // Unix-like: /var/lib/berlin.sneak.app.routewatch if root, else XDG_DATA_HOME
@@ -124,7 +111,7 @@ func getStateDirectory() (string, error) {
return filepath.Join(home, ".local", "share", AppIdentifier), nil return filepath.Join(home, ".local", "share", AppIdentifier), nil
default: default:
return "", fmt.Errorf("%w: %s", errUnsupportedOS, runtime.GOOS) return "", fmt.Errorf("unsupported operating system: %s", runtime.GOOS)
} }
} }
@@ -138,8 +125,7 @@ func getPort() (int, error) {
// ParseUint, unlike Atoi, refuses a sign: the health check URL cannot use "+9090" // ParseUint, unlike Atoi, refuses a sign: the health check URL cannot use "+9090"
port, err := strconv.ParseUint(value, 10, 0) port, err := strconv.ParseUint(value, 10, 0)
if err != nil || port < 1 || port > maxPort { if err != nil || port < 1 || port > maxPort {
return 0, fmt.Errorf("%w from 1 to %d, got %q", return 0, fmt.Errorf("PORT must be a whole number from 1 to %d, got %q", maxPort, value)
errInvalidPort, maxPort, value)
} }
return int(port), nil return int(port), nil
@@ -148,8 +134,7 @@ func getPort() (int, error) {
// EnsureDirectories creates all necessary directories if they don't exist // EnsureDirectories creates all necessary directories if they don't exist
func (c *Config) EnsureDirectories() error { func (c *Config) EnsureDirectories() error {
// Ensure state directory exists // Ensure state directory exists
err := os.MkdirAll(c.StateDir, dirPermissions) if err := os.MkdirAll(c.StateDir, dirPermissions); err != nil {
if err != nil {
return fmt.Errorf("failed to create state directory: %w", err) return fmt.Errorf("failed to create state directory: %w", err)
} }
@@ -36,8 +36,7 @@ func TestNewRefusesInvalidPort(t *testing.T) {
t.Setenv("PORT", value) t.Setenv("PORT", value)
t.Setenv("XDG_DATA_HOME", "") t.Setenv("XDG_DATA_HOME", "")
_, err := New() if _, err := New(); err == nil {
if err == nil {
t.Errorf("New() with PORT=%q returned no error", value) t.Errorf("New() with PORT=%q returned no error", value)
} }
}) })
@@ -52,16 +51,12 @@ func TestNewRefusesRelativeXDGDataHome(t *testing.T) {
t.Setenv("PORT", "") t.Setenv("PORT", "")
t.Setenv("XDG_DATA_HOME", "relative/path") t.Setenv("XDG_DATA_HOME", "relative/path")
if _, err := New(); err == nil {
_, err := New()
if err == nil {
t.Error("New() with a relative XDG_DATA_HOME returned no error") t.Error("New() with a relative XDG_DATA_HOME returned no error")
} }
t.Setenv("XDG_DATA_HOME", "/var/lib") t.Setenv("XDG_DATA_HOME", "/var/lib")
if _, err := New(); err != nil {
_, err = New()
if err != nil {
t.Errorf("New() with XDG_DATA_HOME=/var/lib: %v", err) t.Errorf("New() with XDG_DATA_HOME=/var/lib: %v", err)
} }
} }
+6 -21
View File
@@ -33,10 +33,8 @@ type liveCounts struct {
// seed sets every count to the value read from the tables at startup. It runs // seed sets every count to the value read from the tables at startup. It runs
// before any writer, so it needs no coordination with the adjust methods. // before any writer, so it needs no coordination with the adjust methods.
func (c *liveCounts) seed( func (c *liveCounts) seed(asns, prefixesV4, prefixesV6, peerings, peers, routesV4, routesV6 int,
asns, prefixesV4, prefixesV6, peerings, peers, routesV4, routesV6 int, distributionV4, distributionV6 []PrefixDistribution) {
distributionV4, distributionV6 []PrefixDistribution,
) {
c.mu.Lock() c.mu.Lock()
defer c.mu.Unlock() defer c.mu.Unlock()
@@ -47,11 +45,9 @@ func (c *liveCounts) seed(
c.peers = peers c.peers = peers
c.routesV4 = routesV4 c.routesV4 = routesV4
c.routesV6 = routesV6 c.routesV6 = routesV6
for _, entry := range distributionV4 { for _, entry := range distributionV4 {
addAtMaskLength(c.distributionV4[:], entry.MaskLength, entry.Count) addAtMaskLength(c.distributionV4[:], entry.MaskLength, entry.Count)
} }
for _, entry := range distributionV6 { for _, entry := range distributionV6 {
addAtMaskLength(c.distributionV6[:], entry.MaskLength, entry.Count) addAtMaskLength(c.distributionV6[:], entry.MaskLength, entry.Count)
} }
@@ -106,7 +102,6 @@ func (c *liveCounts) addToDistribution(maskLengthsV4, maskLengthsV6 []int, n int
for _, maskLength := range maskLengthsV4 { for _, maskLength := range maskLengthsV4 {
addAtMaskLength(c.distributionV4[:], maskLength, n) addAtMaskLength(c.distributionV4[:], maskLength, n)
} }
for _, maskLength := range maskLengthsV6 { for _, maskLength := range maskLengthsV6 {
addAtMaskLength(c.distributionV6[:], maskLength, n) addAtMaskLength(c.distributionV6[:], maskLength, n)
} }
@@ -143,11 +138,9 @@ func (c *liveCounts) fill(s *Stats) {
// ascending order, the way the distribution query returns them. // ascending order, the way the distribution query returns them.
func distributionList(counts []int) []PrefixDistribution { func distributionList(counts []int) []PrefixDistribution {
var list []PrefixDistribution var list []PrefixDistribution
for maskLength, count := range counts { for maskLength, count := range counts {
if count > 0 { if count > 0 {
list = append(list, list = append(list, PrefixDistribution{MaskLength: maskLength, Count: count})
PrefixDistribution{MaskLength: maskLength, Count: count})
} }
} }
@@ -159,8 +152,7 @@ func distributionList(counts []int) []PrefixDistribution {
func (d *Database) countRows(ctx context.Context, table string) (int, error) { func (d *Database) countRows(ctx context.Context, table string) (int, error) {
var n int var n int
// table is one of a fixed set of literals below, never external input. // table is one of a fixed set of literals below, never external input.
err := d.db.QueryRowContext(ctx, "SELECT COUNT(*) FROM "+table).Scan(&n) if err := d.db.QueryRowContext(ctx, "SELECT COUNT(*) FROM "+table).Scan(&n); err != nil {
if err != nil {
return 0, fmt.Errorf("failed to count %s: %w", table, err) return 0, fmt.Errorf("failed to count %s: %w", table, err)
} }
@@ -175,37 +167,30 @@ func (d *Database) seedCounts(ctx context.Context) error {
if err != nil { if err != nil {
return err return err
} }
prefixesV4, err := d.countRows(ctx, "prefixes_v4") prefixesV4, err := d.countRows(ctx, "prefixes_v4")
if err != nil { if err != nil {
return err return err
} }
prefixesV6, err := d.countRows(ctx, "prefixes_v6") prefixesV6, err := d.countRows(ctx, "prefixes_v6")
if err != nil { if err != nil {
return err return err
} }
peerings, err := d.countRows(ctx, "peerings") peerings, err := d.countRows(ctx, "peerings")
if err != nil { if err != nil {
return err return err
} }
peers, err := d.countRows(ctx, "bgp_peers") peers, err := d.countRows(ctx, "bgp_peers")
if err != nil { if err != nil {
return err return err
} }
routesV4, err := d.countRows(ctx, "live_routes_v4")
routesV4, err := d.countRows(ctx, tableLiveRoutesV4)
if err != nil { if err != nil {
return err return err
} }
routesV6, err := d.countRows(ctx, "live_routes_v6")
routesV6, err := d.countRows(ctx, tableLiveRoutesV6)
if err != nil { if err != nil {
return err return err
} }
distributionV4, distributionV6, err := d.GetPrefixDistributionContext(ctx) distributionV4, distributionV6, err := d.GetPrefixDistributionContext(ctx)
if err != nil { if err != nil {
return err return err
@@ -12,12 +12,6 @@ import (
"github.com/google/uuid" "github.com/google/uuid"
) )
// testV4PeerIP is the peer the IPv4 routes in these tests are learned from.
const testV4PeerIP = "192.0.2.1"
// testV6Prefix is the prefix of every IPv6 route in these tests.
const testV6Prefix = "2001:db8::/32"
// mkV4Route builds an IPv4 live route with its mask length taken from the // mkV4Route builds an IPv4 live route with its mask length taken from the
// prefix. // prefix.
func mkV4Route(t *testing.T, prefix string, asn int, ts time.Time) *LiveRoute { func mkV4Route(t *testing.T, prefix string, asn int, ts time.Time) *LiveRoute {
@@ -34,18 +28,18 @@ func mkV4Route(t *testing.T, prefix string, asn int, ts time.Time) *LiveRoute {
MaskLength: maskLength, MaskLength: maskLength,
IPVersion: ipVersionV4, IPVersion: ipVersionV4,
OriginASN: asn, OriginASN: asn,
PeerIP: testV4PeerIP, PeerIP: "192.0.2.1",
ASPath: []int{asn}, ASPath: []int{asn},
NextHop: "192.0.2.254", NextHop: "192.0.2.254",
LastUpdated: ts, LastUpdated: ts,
} }
} }
// mkV6Route builds an IPv6 live route for testV6Prefix. // mkV6Route builds an IPv6 live route.
func mkV6Route(asn int, ts time.Time) *LiveRoute { func mkV6Route(prefix string, asn int, ts time.Time) *LiveRoute {
return &LiveRoute{ return &LiveRoute{
ID: uuid.New(), ID: uuid.New(),
Prefix: testV6Prefix, Prefix: prefix,
MaskLength: 32, MaskLength: 32,
IPVersion: ipVersionV6, IPVersion: ipVersionV6,
OriginASN: asn, OriginASN: asn,
@@ -62,8 +56,6 @@ func mkV6Route(asn int, ts time.Time) *LiveRoute {
// deleted. These counts are what /api/v1/stats reports; before this change the // deleted. These counts are what /api/v1/stats reports; before this change the
// endpoint recomputed them with a COUNT(*) over each table on every request. // endpoint recomputed them with a COUNT(*) over each table on every request.
func TestLiveCountsTrackWritesInRealtime(t *testing.T) { func TestLiveCountsTrackWritesInRealtime(t *testing.T) {
t.Parallel()
cfg := &config.Config{StateDir: t.TempDir()} cfg := &config.Config{StateDir: t.TempDir()}
db, err := New(cfg, logger.New()) db, err := New(cfg, logger.New())
@@ -78,7 +70,6 @@ func TestLiveCountsTrackWritesInRealtime(t *testing.T) {
if err != nil { if err != nil {
t.Fatalf("GetStatsContext on empty database: %v", err) t.Fatalf("GetStatsContext on empty database: %v", err)
} }
if empty.ASNs != 0 || empty.Prefixes != 0 || empty.Peerings != 0 || if empty.ASNs != 0 || empty.Prefixes != 0 || empty.Peerings != 0 ||
empty.Peers != 0 || empty.LiveRoutes != 0 { empty.Peers != 0 || empty.LiveRoutes != 0 {
t.Fatalf("empty database counts nonzero: %+v", empty) t.Fatalf("empty database counts nonzero: %+v", empty)
@@ -86,16 +77,30 @@ func TestLiveCountsTrackWritesInRealtime(t *testing.T) {
ts := time.Date(2026, 1, 2, 3, 4, 5, 0, time.UTC) ts := time.Date(2026, 1, 2, 3, 4, 5, 0, time.UTC)
writeASNsPrefixesAndPeers(t, db, ts) if err := db.GetOrCreateASNBatch(map[int]time.Time{64500: ts, 64501: ts}); err != nil {
t.Fatalf("GetOrCreateASNBatch: %v", err)
}
if err := db.UpdatePrefixesBatch(map[string]time.Time{
"198.51.100.0/24": ts,
"2001:db8::/32": ts,
}); err != nil {
t.Fatalf("UpdatePrefixesBatch: %v", err)
}
if err := db.UpdatePeerBatch(map[string]PeerUpdate{
"192.0.2.1": {PeerIP: "192.0.2.1", PeerASN: 64500, MessageType: "UPDATE", Timestamp: ts},
}); err != nil {
t.Fatalf("UpdatePeerBatch: %v", err)
}
if err := db.RecordPeering(64500, 64501, ts); err != nil {
t.Fatalf("RecordPeering: %v", err)
}
routes := []*LiveRoute{ routes := []*LiveRoute{
mkV4Route(t, "198.51.100.0/24", 64500, ts), mkV4Route(t, "198.51.100.0/24", 64500, ts),
mkV4Route(t, "203.0.113.0/24", 64501, ts.Add(time.Minute)), mkV4Route(t, "203.0.113.0/24", 64501, ts.Add(time.Minute)),
mkV6Route(64502, ts.Add(2*time.Minute)), mkV6Route("2001:db8::/32", 64502, ts.Add(2*time.Minute)),
} }
if err := db.UpsertLiveRouteBatch(routes); err != nil {
err = db.UpsertLiveRouteBatch(routes)
if err != nil {
t.Fatalf("UpsertLiveRouteBatch: %v", err) t.Fatalf("UpsertLiveRouteBatch: %v", err)
} }
@@ -103,88 +108,44 @@ func TestLiveCountsTrackWritesInRealtime(t *testing.T) {
if err != nil { if err != nil {
t.Fatalf("GetStatsContext: %v", err) t.Fatalf("GetStatsContext: %v", err)
} }
assertCounts(t, "after inserts", stats, wantCounts{ assertCounts(t, "after inserts", stats, wantCounts{
asns: 2, prefixes: 2, peerings: 1, peers: 1, asns: 2, prefixes: 2, peerings: 1, peers: 1,
ipv4Routes: 2, ipv6Routes: 1, liveRoutes: 3, ipv4Routes: 2, ipv6Routes: 1, liveRoutes: 3,
}) })
// Re-announcing the same routes is an update, not an insert: counts hold. // Re-announcing the same routes is an update, not an insert: counts hold.
err = db.UpsertLiveRouteBatch(routes) if err := db.UpsertLiveRouteBatch(routes); err != nil {
if err != nil {
t.Fatalf("UpsertLiveRouteBatch (re-announce): %v", err) t.Fatalf("UpsertLiveRouteBatch (re-announce): %v", err)
} }
stats, err = db.GetStatsContext(ctx) stats, err = db.GetStatsContext(ctx)
if err != nil { if err != nil {
t.Fatalf("GetStatsContext: %v", err) t.Fatalf("GetStatsContext: %v", err)
} }
assertCounts(t, "after re-announce", stats, wantCounts{ assertCounts(t, "after re-announce", stats, wantCounts{
asns: 2, prefixes: 2, peerings: 1, peers: 1, asns: 2, prefixes: 2, peerings: 1, peers: 1,
ipv4Routes: 2, ipv6Routes: 1, liveRoutes: 3, ipv4Routes: 2, ipv6Routes: 1, liveRoutes: 3,
}) })
// A withdrawal removes one route. // A withdrawal removes one route.
err = db.DeleteLiveRouteBatch([]LiveRouteDeletion{{ if err := db.DeleteLiveRouteBatch([]LiveRouteDeletion{
Prefix: "203.0.113.0/24", OriginASN: 64501, {Prefix: "203.0.113.0/24", OriginASN: 64501, PeerIP: "192.0.2.1", IPVersion: ipVersionV4},
PeerIP: testV4PeerIP, IPVersion: ipVersionV4, }); err != nil {
}})
if err != nil {
t.Fatalf("DeleteLiveRouteBatch: %v", err) t.Fatalf("DeleteLiveRouteBatch: %v", err)
} }
stats, err = db.GetStatsContext(ctx) stats, err = db.GetStatsContext(ctx)
if err != nil { if err != nil {
t.Fatalf("GetStatsContext: %v", err) t.Fatalf("GetStatsContext: %v", err)
} }
assertCounts(t, "after delete", stats, wantCounts{ assertCounts(t, "after delete", stats, wantCounts{
asns: 2, prefixes: 2, peerings: 1, peers: 1, asns: 2, prefixes: 2, peerings: 1, peers: 1,
ipv4Routes: 1, ipv6Routes: 1, liveRoutes: 2, ipv4Routes: 1, ipv6Routes: 1, liveRoutes: 2,
}) })
} }
// writeASNsPrefixesAndPeers writes two ASNs, two prefixes, one peer and one
// peering: the rows besides live routes that the stats count.
func writeASNsPrefixesAndPeers(t *testing.T, db *Database, ts time.Time) {
t.Helper()
err := db.GetOrCreateASNBatch(map[int]time.Time{64500: ts, 64501: ts})
if err != nil {
t.Fatalf("GetOrCreateASNBatch: %v", err)
}
err = db.UpdatePrefixesBatch(map[string]time.Time{
"198.51.100.0/24": ts,
testV6Prefix: ts,
})
if err != nil {
t.Fatalf("UpdatePrefixesBatch: %v", err)
}
err = db.UpdatePeerBatch(map[string]PeerUpdate{
testV4PeerIP: {
PeerIP: testV4PeerIP, PeerASN: 64500,
MessageType: "UPDATE", Timestamp: ts,
},
})
if err != nil {
t.Fatalf("UpdatePeerBatch: %v", err)
}
err = db.RecordPeering(64500, 64501, ts)
if err != nil {
t.Fatalf("RecordPeering: %v", err)
}
}
// TestLiveCountsSeededFromDatabaseAtStartup writes rows, reopens the same // TestLiveCountsSeededFromDatabaseAtStartup writes rows, reopens the same
// database file, and checks the counts and the prefix distribution come back // database file, and checks the counts and the prefix distribution come back
// from the seed scan rather than starting at zero. // from the seed scan rather than starting at zero.
func TestLiveCountsSeededFromDatabaseAtStartup(t *testing.T) { func TestLiveCountsSeededFromDatabaseAtStartup(t *testing.T) {
t.Parallel()
cfg := &config.Config{StateDir: t.TempDir()} cfg := &config.Config{StateDir: t.TempDir()}
db, err := New(cfg, logger.New()) db, err := New(cfg, logger.New())
@@ -193,22 +154,16 @@ func TestLiveCountsSeededFromDatabaseAtStartup(t *testing.T) {
} }
ts := time.Date(2026, 1, 2, 3, 4, 5, 0, time.UTC) ts := time.Date(2026, 1, 2, 3, 4, 5, 0, time.UTC)
if err := db.GetOrCreateASNBatch(map[int]time.Time{64500: ts, 64501: ts, 64502: ts}); err != nil {
err = db.GetOrCreateASNBatch(map[int]time.Time{64500: ts, 64501: ts, 64502: ts})
if err != nil {
t.Fatalf("GetOrCreateASNBatch: %v", err) t.Fatalf("GetOrCreateASNBatch: %v", err)
} }
if err := db.UpsertLiveRouteBatch([]*LiveRoute{
err = db.UpsertLiveRouteBatch([]*LiveRoute{
mkV4Route(t, "198.51.100.0/24", 64500, ts), mkV4Route(t, "198.51.100.0/24", 64500, ts),
mkV6Route(64502, ts), mkV6Route("2001:db8::/32", 64502, ts),
}) }); err != nil {
if err != nil {
t.Fatalf("UpsertLiveRouteBatch: %v", err) t.Fatalf("UpsertLiveRouteBatch: %v", err)
} }
if err := db.Close(); err != nil {
err = db.Close()
if err != nil {
t.Fatalf("Close: %v", err) t.Fatalf("Close: %v", err)
} }
@@ -222,16 +177,13 @@ func TestLiveCountsSeededFromDatabaseAtStartup(t *testing.T) {
if err != nil { if err != nil {
t.Fatalf("GetStatsContext after reopen: %v", err) t.Fatalf("GetStatsContext after reopen: %v", err)
} }
if stats.ASNs != 3 { if stats.ASNs != 3 {
t.Errorf("seeded ASNs = %d, want 3", stats.ASNs) t.Errorf("seeded ASNs = %d, want 3", stats.ASNs)
} }
if stats.IPv4Routes != 1 || stats.IPv6Routes != 1 || stats.LiveRoutes != 2 { if stats.IPv4Routes != 1 || stats.IPv6Routes != 1 || stats.LiveRoutes != 2 {
t.Errorf("seeded routes = (v4 %d, v6 %d, total %d), want (1, 1, 2)", t.Errorf("seeded routes = (v4 %d, v6 %d, total %d), want (1, 1, 2)",
stats.IPv4Routes, stats.IPv6Routes, stats.LiveRoutes) stats.IPv4Routes, stats.IPv6Routes, stats.LiveRoutes)
} }
assertDistribution(t, "seeded IPv4 distribution", stats.IPv4PrefixDistribution, assertDistribution(t, "seeded IPv4 distribution", stats.IPv4PrefixDistribution,
[]PrefixDistribution{{MaskLength: 24, Count: 1}}) []PrefixDistribution{{MaskLength: 24, Count: 1}})
assertDistribution(t, "seeded IPv6 distribution", stats.IPv6PrefixDistribution, assertDistribution(t, "seeded IPv6 distribution", stats.IPv6PrefixDistribution,
@@ -244,91 +196,26 @@ func TestLiveCountsSeededFromDatabaseAtStartup(t *testing.T) {
// tables. The steps run once through the batch methods the prefix handler uses // tables. The steps run once through the batch methods the prefix handler uses
// and once through the single-route methods. // and once through the single-route methods.
func TestPrefixDistributionTracksWrites(t *testing.T) { func TestPrefixDistributionTracksWrites(t *testing.T) {
t.Parallel()
steps := prefixDistributionSteps(t)
for _, batch := range []bool{true, false} {
name := "single-route writes"
if batch {
name = "batch writes"
}
t.Run(name, func(t *testing.T) {
t.Parallel()
db, err := New(&config.Config{StateDir: t.TempDir()}, logger.New())
if err != nil {
t.Fatalf("failed to create database: %v", err)
}
defer func() { _ = db.Close() }()
ctx := context.Background()
for _, step := range steps {
err := announceRoutes(db, batch, step.announce)
if err != nil {
t.Fatalf("%s: announce: %v", step.name, err)
}
err = withdrawRoutes(db, batch, step.withdraw)
if err != nil {
t.Fatalf("%s: withdraw: %v", step.name, err)
}
stats, err := db.GetStatsContext(ctx)
if err != nil {
t.Fatalf("%s: GetStatsContext: %v", step.name, err)
}
queryV4, queryV6, err := db.GetPrefixDistributionContext(ctx)
if err != nil {
t.Fatalf("%s: GetPrefixDistributionContext: %v", step.name, err)
}
assertDistribution(t, step.name+": IPv4 distribution",
stats.IPv4PrefixDistribution, step.wantV4)
assertDistribution(t, step.name+": IPv6 distribution",
stats.IPv6PrefixDistribution, step.wantV6)
assertDistribution(t, step.name+": IPv4 distribution query",
queryV4, step.wantV4)
assertDistribution(t, step.name+": IPv6 distribution query",
queryV6, step.wantV6)
}
})
}
}
// prefixDistributionStep is one step of TestPrefixDistributionTracksWrites:
// the routes it announces, then the routes it withdraws, then the
// distributions both the stats and the distribution query must report.
type prefixDistributionStep struct {
name string
announce []*LiveRoute
withdraw []*LiveRoute
wantV4 []PrefixDistribution
wantV6 []PrefixDistribution
}
// prefixDistributionSteps returns the steps of
// TestPrefixDistributionTracksWrites, in the order they run.
func prefixDistributionSteps(t *testing.T) []prefixDistributionStep {
t.Helper()
ts := time.Date(2026, 1, 2, 3, 4, 5, 0, time.UTC) ts := time.Date(2026, 1, 2, 3, 4, 5, 0, time.UTC)
shared := mkV4Route(t, "198.51.100.0/24", 64500, ts) shared := mkV4Route(t, "198.51.100.0/24", 64500, ts)
sharedSecondPeer := mkV4Route(t, "198.51.100.0/24", 64500, ts) sharedSecondPeer := mkV4Route(t, "198.51.100.0/24", 64500, ts)
sharedSecondPeer.PeerIP = "192.0.2.2" sharedSecondPeer.PeerIP = "192.0.2.2"
other := mkV4Route(t, "203.0.113.0/24", 64501, ts) other := mkV4Route(t, "203.0.113.0/24", 64501, ts)
wide := mkV4Route(t, "172.16.0.0/16", 64502, ts) wide := mkV4Route(t, "172.16.0.0/16", 64502, ts)
v6 := mkV6Route(64503, ts) v6 := mkV6Route("2001:db8::/32", 64503, ts)
v6SecondPeer := mkV6Route(64503, ts) v6SecondPeer := mkV6Route("2001:db8::/32", 64503, ts)
v6SecondPeer.PeerIP = "2001:db8::2" v6SecondPeer.PeerIP = "2001:db8::2"
all := []PrefixDistribution{{MaskLength: 16, Count: 1}, {MaskLength: 24, Count: 2}} all := []PrefixDistribution{{MaskLength: 16, Count: 1}, {MaskLength: 24, Count: 2}}
v6Only := []PrefixDistribution{{MaskLength: 32, Count: 1}} v6Only := []PrefixDistribution{{MaskLength: 32, Count: 1}}
return []prefixDistributionStep{ steps := []struct {
name string
announce []*LiveRoute
withdraw []*LiveRoute
wantV4 []PrefixDistribution
wantV6 []PrefixDistribution
}{
{ {
name: "new routes", name: "new routes",
announce: []*LiveRoute{shared, other, wide, v6}, announce: []*LiveRoute{shared, other, wide, v6},
@@ -352,9 +239,7 @@ func prefixDistributionSteps(t *testing.T) []prefixDistributionStep {
{ {
name: "withdrawal of the last route for a prefix", name: "withdrawal of the last route for a prefix",
withdraw: []*LiveRoute{sharedSecondPeer, v6SecondPeer}, withdraw: []*LiveRoute{sharedSecondPeer, v6SecondPeer},
wantV4: []PrefixDistribution{ wantV4: []PrefixDistribution{{MaskLength: 16, Count: 1}, {MaskLength: 24, Count: 1}},
{MaskLength: 16, Count: 1}, {MaskLength: 24, Count: 1},
},
}, },
{ {
name: "withdrawal of every remaining route", name: "withdrawal of every remaining route",
@@ -385,6 +270,45 @@ func prefixDistributionSteps(t *testing.T) []prefixDistributionStep {
wantV6: v6Only, wantV6: v6Only,
}, },
} }
for _, batch := range []bool{true, false} {
name := "single-route writes"
if batch {
name = "batch writes"
}
t.Run(name, func(t *testing.T) {
db, err := New(&config.Config{StateDir: t.TempDir()}, logger.New())
if err != nil {
t.Fatalf("failed to create database: %v", err)
}
defer func() { _ = db.Close() }()
ctx := context.Background()
for _, step := range steps {
if err := announceRoutes(db, batch, step.announce); err != nil {
t.Fatalf("%s: announce: %v", step.name, err)
}
if err := withdrawRoutes(db, batch, step.withdraw); err != nil {
t.Fatalf("%s: withdraw: %v", step.name, err)
}
stats, err := db.GetStatsContext(ctx)
if err != nil {
t.Fatalf("%s: GetStatsContext: %v", step.name, err)
}
queryV4, queryV6, err := db.GetPrefixDistributionContext(ctx)
if err != nil {
t.Fatalf("%s: GetPrefixDistributionContext: %v", step.name, err)
}
assertDistribution(t, step.name+": IPv4 distribution", stats.IPv4PrefixDistribution, step.wantV4)
assertDistribution(t, step.name+": IPv6 distribution", stats.IPv6PrefixDistribution, step.wantV6)
assertDistribution(t, step.name+": IPv4 distribution query", queryV4, step.wantV4)
assertDistribution(t, step.name+": IPv6 distribution query", queryV6, step.wantV6)
}
})
}
} }
// announceRoutes writes routes in one UpsertLiveRouteBatch, or with one // announceRoutes writes routes in one UpsertLiveRouteBatch, or with one
@@ -393,10 +317,8 @@ func announceRoutes(db *Database, batch bool, routes []*LiveRoute) error {
if batch { if batch {
return db.UpsertLiveRouteBatch(routes) return db.UpsertLiveRouteBatch(routes)
} }
for _, route := range routes { for _, route := range routes {
err := db.UpsertLiveRoute(route) if err := db.UpsertLiveRoute(route); err != nil {
if err != nil {
return err return err
} }
} }
@@ -411,8 +333,7 @@ func announceRoutes(db *Database, batch bool, routes []*LiveRoute) error {
func withdrawRoutes(db *Database, batch bool, routes []*LiveRoute) error { func withdrawRoutes(db *Database, batch bool, routes []*LiveRoute) error {
if !batch { if !batch {
for _, route := range routes { for _, route := range routes {
err := db.DeleteLiveRoute(route.Prefix, 0, route.PeerIP) if err := db.DeleteLiveRoute(route.Prefix, 0, route.PeerIP); err != nil {
if err != nil {
return err return err
} }
} }
@@ -435,8 +356,6 @@ func withdrawRoutes(db *Database, batch bool, routes []*LiveRoute) error {
// MIN/MAX union query read its result into *time.Time, which the driver could // MIN/MAX union query read its result into *time.Time, which the driver could
// not parse, so it logged a warning every call and left both timestamps nil. // not parse, so it logged a warning every call and left both timestamps nil.
func TestStatsRouteTimestamps(t *testing.T) { func TestStatsRouteTimestamps(t *testing.T) {
t.Parallel()
cfg := &config.Config{StateDir: t.TempDir()} cfg := &config.Config{StateDir: t.TempDir()}
db, err := New(cfg, logger.New()) db, err := New(cfg, logger.New())
@@ -451,7 +370,6 @@ func TestStatsRouteTimestamps(t *testing.T) {
if err != nil { if err != nil {
t.Fatalf("GetStatsContext on empty database: %v", err) t.Fatalf("GetStatsContext on empty database: %v", err)
} }
if empty.OldestRoute != nil || empty.NewestRoute != nil { if empty.OldestRoute != nil || empty.NewestRoute != nil {
t.Fatalf("empty database timestamps = (%v, %v), want (nil, nil)", t.Fatalf("empty database timestamps = (%v, %v), want (nil, nil)",
empty.OldestRoute, empty.NewestRoute) empty.OldestRoute, empty.NewestRoute)
@@ -460,13 +378,11 @@ func TestStatsRouteTimestamps(t *testing.T) {
base := time.Date(2026, 1, 2, 3, 4, 5, 0, time.UTC) base := time.Date(2026, 1, 2, 3, 4, 5, 0, time.UTC)
oldest := base oldest := base
newest := base.Add(2 * time.Minute) newest := base.Add(2 * time.Minute)
if err := db.UpsertLiveRouteBatch([]*LiveRoute{
err = db.UpsertLiveRouteBatch([]*LiveRoute{
mkV4Route(t, "198.51.100.0/24", 64500, base.Add(time.Minute)), mkV4Route(t, "198.51.100.0/24", 64500, base.Add(time.Minute)),
mkV4Route(t, "203.0.113.0/24", 64501, oldest), mkV4Route(t, "203.0.113.0/24", 64501, oldest),
mkV6Route(64502, newest), mkV6Route("2001:db8::/32", 64502, newest),
}) }); err != nil {
if err != nil {
t.Fatalf("UpsertLiveRouteBatch: %v", err) t.Fatalf("UpsertLiveRouteBatch: %v", err)
} }
@@ -474,11 +390,9 @@ func TestStatsRouteTimestamps(t *testing.T) {
if err != nil { if err != nil {
t.Fatalf("GetStatsContext: %v", err) t.Fatalf("GetStatsContext: %v", err)
} }
if stats.OldestRoute == nil || !stats.OldestRoute.Equal(oldest) { if stats.OldestRoute == nil || !stats.OldestRoute.Equal(oldest) {
t.Errorf("OldestRoute = %v, want %v", stats.OldestRoute, oldest) t.Errorf("OldestRoute = %v, want %v", stats.OldestRoute, oldest)
} }
if stats.NewestRoute == nil || !stats.NewestRoute.Equal(newest) { if stats.NewestRoute == nil || !stats.NewestRoute.Equal(newest) {
t.Errorf("NewestRoute = %v, want %v", stats.NewestRoute, newest) t.Errorf("NewestRoute = %v, want %v", stats.NewestRoute, newest)
} }
@@ -487,8 +401,6 @@ func TestStatsRouteTimestamps(t *testing.T) {
// TestLiveCountsConcurrentReadWrite runs writers and stats readers at once so // TestLiveCountsConcurrentReadWrite runs writers and stats readers at once so
// the race detector proves the counters are safe under concurrent use. // the race detector proves the counters are safe under concurrent use.
func TestLiveCountsConcurrentReadWrite(t *testing.T) { func TestLiveCountsConcurrentReadWrite(t *testing.T) {
t.Parallel()
cfg := &config.Config{StateDir: t.TempDir()} cfg := &config.Config{StateDir: t.TempDir()}
db, err := New(cfg, logger.New()) db, err := New(cfg, logger.New())
@@ -500,21 +412,17 @@ func TestLiveCountsConcurrentReadWrite(t *testing.T) {
ts := time.Date(2026, 1, 2, 3, 4, 5, 0, time.UTC) ts := time.Date(2026, 1, 2, 3, 4, 5, 0, time.UTC)
const writers = 4 const writers = 4
var wg sync.WaitGroup var wg sync.WaitGroup
wg.Add(writers)
wg.Add(writers)
for w := range writers { for w := range writers {
go func(base int) { go func(base int) {
defer wg.Done() defer wg.Done()
for i := range 25 { for i := range 25 {
asn := 65000 + base*100 + i asn := 65000 + base*100 + i
route := mkV6Route(asn, ts) route := mkV6Route("2001:db8::/32", asn, ts)
route.PeerIP = "2001:db8::" + uuid.NewString() route.PeerIP = "2001:db8::" + uuid.NewString()
if err := db.UpsertLiveRoute(route); err != nil {
err := db.UpsertLiveRoute(route)
if err != nil {
t.Errorf("UpsertLiveRoute: %v", err) t.Errorf("UpsertLiveRoute: %v", err)
return return
@@ -525,19 +433,15 @@ func TestLiveCountsConcurrentReadWrite(t *testing.T) {
var readerWG sync.WaitGroup var readerWG sync.WaitGroup
readerWG.Add(1) readerWG.Add(1)
stop := make(chan struct{}) stop := make(chan struct{})
go func() { go func() {
defer readerWG.Done() defer readerWG.Done()
for { for {
select { select {
case <-stop: case <-stop:
return return
default: default:
_, err := db.GetStatsContext(context.Background()) if _, err := db.GetStatsContext(context.Background()); err != nil {
if err != nil {
t.Errorf("GetStatsContext: %v", err) t.Errorf("GetStatsContext: %v", err)
return return
@@ -554,7 +458,6 @@ func TestLiveCountsConcurrentReadWrite(t *testing.T) {
if err != nil { if err != nil {
t.Fatalf("GetStatsContext: %v", err) t.Fatalf("GetStatsContext: %v", err)
} }
if want := writers * 25; stats.IPv6Routes != want { if want := writers * 25; stats.IPv6Routes != want {
t.Errorf("IPv6Routes = %d, want %d", stats.IPv6Routes, want) t.Errorf("IPv6Routes = %d, want %d", stats.IPv6Routes, want)
} }
@@ -579,27 +482,21 @@ func assertCounts(t *testing.T, when string, got Stats, want wantCounts) {
if got.ASNs != want.asns { if got.ASNs != want.asns {
t.Errorf("%s: ASNs = %d, want %d", when, got.ASNs, want.asns) t.Errorf("%s: ASNs = %d, want %d", when, got.ASNs, want.asns)
} }
if got.Prefixes != want.prefixes { if got.Prefixes != want.prefixes {
t.Errorf("%s: Prefixes = %d, want %d", when, got.Prefixes, want.prefixes) t.Errorf("%s: Prefixes = %d, want %d", when, got.Prefixes, want.prefixes)
} }
if got.Peerings != want.peerings { if got.Peerings != want.peerings {
t.Errorf("%s: Peerings = %d, want %d", when, got.Peerings, want.peerings) t.Errorf("%s: Peerings = %d, want %d", when, got.Peerings, want.peerings)
} }
if got.Peers != want.peers { if got.Peers != want.peers {
t.Errorf("%s: Peers = %d, want %d", when, got.Peers, want.peers) t.Errorf("%s: Peers = %d, want %d", when, got.Peers, want.peers)
} }
if got.IPv4Routes != want.ipv4Routes { if got.IPv4Routes != want.ipv4Routes {
t.Errorf("%s: IPv4Routes = %d, want %d", when, got.IPv4Routes, want.ipv4Routes) t.Errorf("%s: IPv4Routes = %d, want %d", when, got.IPv4Routes, want.ipv4Routes)
} }
if got.IPv6Routes != want.ipv6Routes { if got.IPv6Routes != want.ipv6Routes {
t.Errorf("%s: IPv6Routes = %d, want %d", when, got.IPv6Routes, want.ipv6Routes) t.Errorf("%s: IPv6Routes = %d, want %d", when, got.IPv6Routes, want.ipv6Routes)
} }
if got.LiveRoutes != want.liveRoutes { if got.LiveRoutes != want.liveRoutes {
t.Errorf("%s: LiveRoutes = %d, want %d", when, got.LiveRoutes, want.liveRoutes) t.Errorf("%s: LiveRoutes = %d, want %d", when, got.LiveRoutes, want.liveRoutes)
} }
File diff suppressed because it is too large Load Diff
@@ -50,8 +50,6 @@ const (
// address, ErrNoRoute when none covers it, and the next less specific prefix // address, ErrNoRoute when none covers it, and the next less specific prefix
// once the only route of the most specific one is withdrawn. // once the only route of the most specific one is withdrawn.
func TestGetIPInfoFindsMostSpecificLiveRoute(t *testing.T) { func TestGetIPInfoFindsMostSpecificLiveRoute(t *testing.T) {
t.Parallel()
cfg := &config.Config{StateDir: t.TempDir()} cfg := &config.Config{StateDir: t.TempDir()}
db, err := New(cfg, logger.New()) db, err := New(cfg, logger.New())
@@ -69,7 +67,24 @@ func TestGetIPInfoFindsMostSpecificLiveRoute(t *testing.T) {
"2001:db8:1::/48": 64501, "2001:db8:1::/48": 64501,
"2001:db8:1:2::/64": 64502, "2001:db8:1:2::/64": 64502,
} }
writeOriginRoutes(t, db, origins) ts := time.Date(2026, 1, 2, 3, 4, 5, 0, time.UTC)
routes := make([]*LiveRoute, 0, len(origins))
for prefix, asn := range origins {
routes = append(routes, &LiveRoute{
ID: uuid.New(),
Prefix: prefix,
MaskLength: netip.MustParsePrefix(prefix).Bits(),
IPVersion: detectIPVersion(prefix),
OriginASN: asn,
PeerIP: testPeerIP,
ASPath: []int{asn},
NextHop: testPeerIP,
LastUpdated: ts,
})
}
if err := db.UpsertLiveRouteBatch(routes); err != nil {
t.Fatalf("UpsertLiveRouteBatch: %v", err)
}
// lookup checks that ip resolves to the live prefix want, or to ErrNoRoute // lookup checks that ip resolves to the live prefix want, or to ErrNoRoute
// when want is empty. // when want is empty.
@@ -84,13 +99,11 @@ func TestGetIPInfoFindsMostSpecificLiveRoute(t *testing.T) {
return return
} }
if err != nil { if err != nil {
t.Errorf("GetIPInfo(%s): %v", ip, err) t.Errorf("GetIPInfo(%s): %v", ip, err)
return return
} }
if info.Netblock != want || info.MaskLength != netip.MustParsePrefix(want).Bits() || if info.Netblock != want || info.MaskLength != netip.MustParsePrefix(want).Bits() ||
info.ASN != origins[want] { info.ASN != origins[want] {
t.Errorf("GetIPInfo(%s) = %s (mask %d) AS%d, want %s AS%d", t.Errorf("GetIPInfo(%s) = %s (mask %d) AS%d, want %s AS%d",
@@ -109,14 +122,8 @@ func TestGetIPInfoFindsMostSpecificLiveRoute(t *testing.T) {
lookup("2001:db9::1", "") lookup("2001:db9::1", "")
err = db.DeleteLiveRouteBatch([]LiveRouteDeletion{ err = db.DeleteLiveRouteBatch([]LiveRouteDeletion{
{ {Prefix: "10.1.2.0/24", OriginASN: 64502, PeerIP: testPeerIP, IPVersion: ipVersionV4},
Prefix: "10.1.2.0/24", OriginASN: 64502, {Prefix: "2001:db8:1:2::/64", OriginASN: 64502, PeerIP: testPeerIP, IPVersion: ipVersionV6},
PeerIP: testPeerIP, IPVersion: ipVersionV4,
},
{
Prefix: "2001:db8:1:2::/64", OriginASN: 64502,
PeerIP: testPeerIP, IPVersion: ipVersionV6,
},
}) })
if err != nil { if err != nil {
t.Fatalf("DeleteLiveRouteBatch: %v", err) t.Fatalf("DeleteLiveRouteBatch: %v", err)
@@ -126,16 +133,24 @@ func TestGetIPInfoFindsMostSpecificLiveRoute(t *testing.T) {
lookup("2001:db8:1:2::3", "2001:db8:1::/48") lookup("2001:db8:1:2::3", "2001:db8:1::/48")
} }
// writeOriginRoutes writes one live route from testPeerIP for each prefix in // TestGetIPInfoIgnoresShortRoutes checks that the IP lookup does not report a
// origins, originated by the AS it maps to. // default route, or a route shorter than /8 for IPv4 or /16 for IPv6, as the
func writeOriginRoutes(t *testing.T, db *Database, origins map[string]int) { // route for an address, and that a route of exactly /8 or /16, or longer, still
t.Helper() // is. The prefixes are written out so that a wrong limit fails the test.
func TestGetIPInfoIgnoresShortRoutes(t *testing.T) {
cfg := &config.Config{StateDir: t.TempDir()}
db, err := New(cfg, logger.New())
if err != nil {
t.Fatalf("failed to create database: %v", err)
}
defer func() { _ = db.Close() }()
ts := time.Date(2026, 1, 2, 3, 4, 5, 0, time.UTC) ts := time.Date(2026, 1, 2, 3, 4, 5, 0, time.UTC)
routes := make([]*LiveRoute, 0, len(origins)) announce := func(prefix string, asn int) {
t.Helper()
for prefix, asn := range origins { err := db.UpsertLiveRouteBatch([]*LiveRoute{{
routes = append(routes, &LiveRoute{
ID: uuid.New(), ID: uuid.New(),
Prefix: prefix, Prefix: prefix,
MaskLength: netip.MustParsePrefix(prefix).Bits(), MaskLength: netip.MustParsePrefix(prefix).Bits(),
@@ -145,13 +160,49 @@ func writeOriginRoutes(t *testing.T, db *Database, origins map[string]int) {
ASPath: []int{asn}, ASPath: []int{asn},
NextHop: testPeerIP, NextHop: testPeerIP,
LastUpdated: ts, LastUpdated: ts,
}) }})
if err != nil {
t.Fatalf("UpsertLiveRouteBatch(%s): %v", prefix, err)
}
} }
err := db.UpsertLiveRouteBatch(routes) // lookup checks that ip resolves to the live prefix netblock from asn.
if err != nil { lookup := func(ip, netblock string, asn int) {
t.Fatalf("UpsertLiveRouteBatch: %v", err) t.Helper()
info, err := db.GetIPInfo(ip)
if err != nil {
t.Errorf("GetIPInfo(%s): %v", ip, err)
return
}
if info.Netblock != netblock || info.ASN != asn {
t.Errorf("GetIPInfo(%s) = %s AS%d, want %s AS%d", ip, info.Netblock, info.ASN, netblock, asn)
}
} }
// Default and very short routes, and routes one bit shorter than each limit.
announce("2000::/3", 64500)
announce("0.0.0.0/0", 64500)
announce("2000::/15", 64500)
announce("192.0.0.0/7", 64500)
for _, ip := range []string{"2001:db8::1", "192.0.2.1"} {
if info, err := db.GetIPInfo(ip); !errors.Is(err, ErrNoRoute) {
t.Errorf("GetIPInfo(%s) with only short routes = %+v, %v; want ErrNoRoute", ip, info, err)
}
}
// Routes of exactly the minimum length are used.
announce("2001::/16", 64501)
announce("192.0.0.0/8", 64501)
lookup("2001:db8::1", "2001::/16", 64501)
lookup("192.0.2.1", "192.0.0.0/8", 64501)
announce("2001:db8::/32", 64502)
lookup("2001:db8::1", "2001:db8::/32", 64502)
} }
// TestConnectionPoolPragmas holds several pooled connections open at once and // TestConnectionPoolPragmas holds several pooled connections open at once and
@@ -159,8 +210,6 @@ func writeOriginRoutes(t *testing.T, db *Database, origins map[string]int) {
// process-wide hard heap limit, and sees the new file with auto_vacuum // process-wide hard heap limit, and sees the new file with auto_vacuum
// incremental. // incremental.
func TestConnectionPoolPragmas(t *testing.T) { func TestConnectionPoolPragmas(t *testing.T) {
t.Parallel()
cfg := &config.Config{StateDir: t.TempDir()} cfg := &config.Config{StateDir: t.TempDir()}
db, err := New(cfg, logger.New()) db, err := New(cfg, logger.New())
@@ -180,64 +229,57 @@ func TestConnectionPoolPragmas(t *testing.T) {
} }
}() }()
for i := range heldConnections { for i := 0; i < heldConnections; i++ {
c, err := db.db.Conn(ctx) c, err := db.db.Conn(ctx)
if err != nil { if err != nil {
t.Fatalf("failed to open connection %d: %v", i, err) t.Fatalf("failed to open connection %d: %v", i, err)
} }
conns = append(conns, c) conns = append(conns, c)
} }
for i, c := range conns { for i, c := range conns {
cacheSize := readPragma(ctx, t, c, i, "cache_size") var cacheSize int
if err := c.QueryRowContext(ctx, "PRAGMA cache_size").Scan(&cacheSize); err != nil {
t.Fatalf("conn %d: failed to read cache_size: %v", i, err)
}
if cacheSize != sqliteCacheSizeKiB { if cacheSize != sqliteCacheSizeKiB {
t.Errorf("conn %d: cache_size = %d, want %d", t.Errorf("conn %d: cache_size = %d, want %d", i, cacheSize, sqliteCacheSizeKiB)
i, cacheSize, sqliteCacheSizeKiB)
} }
busyTimeout := readPragma(ctx, t, c, i, "busy_timeout") var busyTimeout int
if err := c.QueryRowContext(ctx, "PRAGMA busy_timeout").Scan(&busyTimeout); err != nil {
t.Fatalf("conn %d: failed to read busy_timeout: %v", i, err)
}
if busyTimeout != sqliteBusyTimeoutMs { if busyTimeout != sqliteBusyTimeoutMs {
t.Errorf("conn %d: busy_timeout = %d, want %d", t.Errorf("conn %d: busy_timeout = %d, want %d", i, busyTimeout, sqliteBusyTimeoutMs)
i, busyTimeout, sqliteBusyTimeoutMs)
} }
tempStore := readPragma(ctx, t, c, i, "temp_store") var tempStore int
if err := c.QueryRowContext(ctx, "PRAGMA temp_store").Scan(&tempStore); err != nil {
t.Fatalf("conn %d: failed to read temp_store: %v", i, err)
}
if tempStore == tempStoreMemory { if tempStore == tempStoreMemory {
t.Errorf("conn %d: temp_store = %d, want anything but %d (MEMORY)", t.Errorf("conn %d: temp_store = %d, want anything but %d (MEMORY)", i, tempStore, tempStoreMemory)
i, tempStore, tempStoreMemory)
} }
hardHeapLimit := readPragma(ctx, t, c, i, "hard_heap_limit") var hardHeapLimit int64
if err := c.QueryRowContext(ctx, "PRAGMA hard_heap_limit").Scan(&hardHeapLimit); err != nil {
t.Fatalf("conn %d: failed to read hard_heap_limit: %v", i, err)
}
if hardHeapLimit != sqliteHardHeapLimitBytes { if hardHeapLimit != sqliteHardHeapLimitBytes {
t.Errorf("conn %d: hard_heap_limit = %d, want %d", t.Errorf("conn %d: hard_heap_limit = %d, want %d", i, hardHeapLimit, sqliteHardHeapLimitBytes)
i, hardHeapLimit, sqliteHardHeapLimitBytes)
} }
autoVacuum := readPragma(ctx, t, c, i, "auto_vacuum") var autoVacuum int
if err := c.QueryRowContext(ctx, "PRAGMA auto_vacuum").Scan(&autoVacuum); err != nil {
t.Fatalf("conn %d: failed to read auto_vacuum: %v", i, err)
}
if autoVacuum != autoVacuumIncremental { if autoVacuum != autoVacuumIncremental {
t.Errorf("conn %d: auto_vacuum = %d, want %d (incremental)", t.Errorf("conn %d: auto_vacuum = %d, want %d (incremental)", i, autoVacuum, autoVacuumIncremental)
i, autoVacuum, autoVacuumIncremental)
} }
} }
} }
// readPragma reads the integer value of PRAGMA name on connection i, c.
func readPragma(
ctx context.Context, t *testing.T, c *sql.Conn, i int, name string,
) int64 {
t.Helper()
var value int64
err := c.QueryRowContext(ctx, "PRAGMA "+name).Scan(&value)
if err != nil {
t.Fatalf("conn %d: failed to read %s: %v", i, name, err)
}
return value
}
// TestVacuumReturnsFreePages checks that after routes are deleted from a new // TestVacuumReturnsFreePages checks that after routes are deleted from a new
// database, one Vacuum call returns every page they used. The deletes leave // database, one Vacuum call returns every page they used. The deletes leave
// fewer free pages than the 1000 Vacuum frees per call, so none may remain. // fewer free pages than the 1000 Vacuum frees per call, so none may remain.
@@ -245,8 +287,6 @@ func readPragma(
// the free pages stayed in the file, and with the PRAGMA run by ExecContext // the free pages stayed in the file, and with the PRAGMA run by ExecContext
// Vacuum freed only one page per call. // Vacuum freed only one page per call.
func TestVacuumReturnsFreePages(t *testing.T) { func TestVacuumReturnsFreePages(t *testing.T) {
t.Parallel()
cfg := &config.Config{StateDir: t.TempDir()} cfg := &config.Config{StateDir: t.TempDir()}
db, err := New(cfg, logger.New()) db, err := New(cfg, logger.New())
@@ -257,12 +297,10 @@ func TestVacuumReturnsFreePages(t *testing.T) {
ctx := context.Background() ctx := context.Background()
ts := time.Now().UTC() ts := time.Now().UTC()
const asn = 64500 const asn = 64500
routes := make([]*LiveRoute, 0, vacuumTestRoutes) routes := make([]*LiveRoute, 0, vacuumTestRoutes)
deletions := make([]LiveRouteDeletion, 0, vacuumTestRoutes) deletions := make([]LiveRouteDeletion, 0, vacuumTestRoutes)
for i := range vacuumTestRoutes { for i := range vacuumTestRoutes {
route := mkV4Route(t, fmt.Sprintf("10.%d.%d.0/24", i/256, i%256), asn, ts) route := mkV4Route(t, fmt.Sprintf("10.%d.%d.0/24", i/256, i%256), asn, ts)
routes = append(routes, route) routes = append(routes, route)
@@ -274,39 +312,29 @@ func TestVacuumReturnsFreePages(t *testing.T) {
}) })
} }
err = db.UpsertLiveRouteBatch(routes) if err := db.UpsertLiveRouteBatch(routes); err != nil {
if err != nil {
t.Fatalf("failed to write routes: %v", err) t.Fatalf("failed to write routes: %v", err)
} }
if err := db.DeleteLiveRouteBatch(deletions); err != nil {
err = db.DeleteLiveRouteBatch(deletions)
if err != nil {
t.Fatalf("failed to delete routes: %v", err) t.Fatalf("failed to delete routes: %v", err)
} }
var before int var before int
if err := db.db.QueryRowContext(ctx, "PRAGMA freelist_count").Scan(&before); err != nil {
err = db.db.QueryRowContext(ctx, "PRAGMA freelist_count").Scan(&before)
if err != nil {
t.Fatalf("failed to read freelist_count: %v", err) t.Fatalf("failed to read freelist_count: %v", err)
} }
if before == 0 { if before == 0 {
t.Fatalf("no free pages after deleting %d routes", vacuumTestRoutes) t.Fatalf("no free pages after deleting %d routes", vacuumTestRoutes)
} }
err = db.Vacuum(ctx) if err := db.Vacuum(ctx); err != nil {
if err != nil {
t.Fatalf("Vacuum failed: %v", err) t.Fatalf("Vacuum failed: %v", err)
} }
var after int var after int
if err := db.db.QueryRowContext(ctx, "PRAGMA freelist_count").Scan(&after); err != nil {
err = db.db.QueryRowContext(ctx, "PRAGMA freelist_count").Scan(&after)
if err != nil {
t.Fatalf("failed to read freelist_count: %v", err) t.Fatalf("failed to read freelist_count: %v", err)
} }
if after != 0 { if after != 0 {
t.Errorf("free pages after Vacuum = %d of %d, want 0", after, before) t.Errorf("free pages after Vacuum = %d of %d, want 0", after, before)
} }
@@ -321,8 +349,6 @@ func TestVacuumReturnsFreePages(t *testing.T) {
// no batch is dropped. The checkpoint runs without the Database mutex, exactly // no batch is dropped. The checkpoint runs without the Database mutex, exactly
// as the background maintainer does in production. // as the background maintainer does in production.
func TestBatchWriteDuringCheckpoint(t *testing.T) { func TestBatchWriteDuringCheckpoint(t *testing.T) {
t.Parallel()
cfg := &config.Config{StateDir: t.TempDir()} cfg := &config.Config{StateDir: t.TempDir()}
db, err := New(cfg, logger.New()) db, err := New(cfg, logger.New())
@@ -335,10 +361,8 @@ func TestBatchWriteDuringCheckpoint(t *testing.T) {
var wg sync.WaitGroup var wg sync.WaitGroup
wg.Add(1) wg.Add(1)
go func() { go func() {
defer wg.Done() defer wg.Done()
for { for {
select { select {
case <-ctx.Done(): case <-ctx.Done():
@@ -350,14 +374,12 @@ func TestBatchWriteDuringCheckpoint(t *testing.T) {
}() }()
ts := time.Now().UTC() ts := time.Now().UTC()
for i := range contentionIterations { for i := 0; i < contentionIterations; i++ {
asns := map[int]time.Time{ asns := map[int]time.Time{
i % contendedASNCount: ts, i % contendedASNCount: ts,
(i % contendedASNCount) + asnSecondBand: ts, (i % contendedASNCount) + asnSecondBand: ts,
} }
if err := db.GetOrCreateASNBatch(asns); err != nil {
err := db.GetOrCreateASNBatch(asns)
if err != nil {
cancel() cancel()
wg.Wait() wg.Wait()
t.Fatalf("batch write failed under checkpoint contention: %v", err) t.Fatalf("batch write failed under checkpoint contention: %v", err)
+3 -9
View File
@@ -29,8 +29,6 @@ type Stats struct {
// Store defines the interface for database operations. It provides methods for // Store defines the interface for database operations. It provides methods for
// managing ASNs, prefixes, announcements, peerings, BGP peers, and live routes. // managing ASNs, prefixes, announcements, peerings, BGP peers, and live routes.
// Implementations must be safe for concurrent use. // Implementations must be safe for concurrent use.
//
//nolint:interfacebloat // the one seam handlers and tests depend on
type Store interface { type Store interface {
// ASN operations // ASN operations
GetOrCreateASN(number int, timestamp time.Time) (*ASN, error) GetOrCreateASN(number int, timestamp time.Time) (*ASN, error)
@@ -59,10 +57,8 @@ type Store interface {
UpsertLiveRouteBatch(routes []*LiveRoute) error UpsertLiveRouteBatch(routes []*LiveRoute) error
DeleteLiveRoute(prefix string, originASN int, peerIP string) error DeleteLiveRoute(prefix string, originASN int, peerIP string) error
DeleteLiveRouteBatch(deletions []LiveRouteDeletion) error DeleteLiveRouteBatch(deletions []LiveRouteDeletion) error
GetPrefixDistribution() ( GetPrefixDistribution() (ipv4 []PrefixDistribution, ipv6 []PrefixDistribution, err error)
ipv4 []PrefixDistribution, ipv6 []PrefixDistribution, err error) GetPrefixDistributionContext(ctx context.Context) (ipv4 []PrefixDistribution, ipv6 []PrefixDistribution, err error)
GetPrefixDistributionContext(ctx context.Context) (
ipv4 []PrefixDistribution, ipv6 []PrefixDistribution, err error)
GetLiveRouteCounts() (ipv4Count, ipv6Count int, err error) GetLiveRouteCounts() (ipv4Count, ipv6Count int, err error)
GetLiveRouteCountsContext(ctx context.Context) (ipv4Count, ipv6Count int, err error) GetLiveRouteCountsContext(ctx context.Context) (ipv4Count, ipv6Count int, err error)
@@ -83,9 +79,7 @@ type Store interface {
GetPrefixDetails(prefix string) ([]LiveRoute, error) GetPrefixDetails(prefix string) ([]LiveRoute, error)
GetPrefixDetailsContext(ctx context.Context, prefix string) ([]LiveRoute, error) GetPrefixDetailsContext(ctx context.Context, prefix string) ([]LiveRoute, error)
GetRandomPrefixesByLength(maskLength, ipVersion, limit int) ([]LiveRoute, error) GetRandomPrefixesByLength(maskLength, ipVersion, limit int) ([]LiveRoute, error)
GetRandomPrefixesByLengthContext( GetRandomPrefixesByLengthContext(ctx context.Context, maskLength, ipVersion, limit int) ([]LiveRoute, error)
ctx context.Context, maskLength, ipVersion, limit int,
) ([]LiveRoute, error)
// Lifecycle // Lifecycle
Close() error Close() error
+19 -26
View File
@@ -1,3 +1,4 @@
// Package database provides SQLite storage for BGP routing data.
package database package database
import ( import (
@@ -8,8 +9,6 @@ import (
// ASN represents an Autonomous System Number with its metadata including // ASN represents an Autonomous System Number with its metadata including
// handle, description, WHOIS data, and first/last seen timestamps. // handle, description, WHOIS data, and first/last seen timestamps.
//
//nolint:tagliatelle // field names are the existing JSON API
type ASN struct { type ASN struct {
ASN int `json:"asn"` ASN int `json:"asn"`
Handle string `json:"handle"` Handle string `json:"handle"`
@@ -37,39 +36,37 @@ type ASN struct {
// Prefix represents an IP prefix (CIDR block) with its IP version (4 or 6) // Prefix represents an IP prefix (CIDR block) with its IP version (4 or 6)
// and first/last seen timestamps. // and first/last seen timestamps.
type Prefix struct { type Prefix struct {
ID uuid.UUID ID uuid.UUID `json:"id"`
Prefix string Prefix string `json:"prefix"`
IPVersion int // 4 or 6 IPVersion int `json:"ip_version"` // 4 or 6
FirstSeen time.Time FirstSeen time.Time `json:"first_seen"`
LastSeen time.Time LastSeen time.Time `json:"last_seen"`
} }
// Announcement represents a BGP announcement or withdrawal event, // Announcement represents a BGP announcement or withdrawal event,
// containing the prefix, AS path, origin ASN, peer ASN, next hop, and timestamp. // containing the prefix, AS path, origin ASN, peer ASN, next hop, and timestamp.
type Announcement struct { type Announcement struct {
ID uuid.UUID ID uuid.UUID `json:"id"`
PrefixID uuid.UUID PrefixID uuid.UUID `json:"prefix_id"`
PeerASN int PeerASN int `json:"peer_asn"`
OriginASN int OriginASN int `json:"origin_asn"`
Path string // JSON-encoded AS path Path string `json:"path"` // JSON-encoded AS path
NextHop string NextHop string `json:"next_hop"`
Timestamp time.Time Timestamp time.Time `json:"timestamp"`
IsWithdrawal bool IsWithdrawal bool `json:"is_withdrawal"`
} }
// ASNPeering represents a peering relationship between two ASNs, // ASNPeering represents a peering relationship between two ASNs,
// stored with the lower ASN as ASA and the higher as ASB. // stored with the lower ASN as ASA and the higher as ASB.
type ASNPeering struct { type ASNPeering struct {
ID uuid.UUID ID uuid.UUID `json:"id"`
ASA int ASA int `json:"as_a"`
ASB int ASB int `json:"as_b"`
FirstSeen time.Time FirstSeen time.Time `json:"first_seen"`
LastSeen time.Time LastSeen time.Time `json:"last_seen"`
} }
// LiveRoute represents a route in the live routing table // LiveRoute represents a route in the live routing table
//
//nolint:tagliatelle // field names are the existing JSON API
type LiveRoute struct { type LiveRoute struct {
ID uuid.UUID `json:"id"` ID uuid.UUID `json:"id"`
Prefix string `json:"prefix"` Prefix string `json:"prefix"`
@@ -83,16 +80,12 @@ type LiveRoute struct {
} }
// PrefixDistribution represents the distribution of prefixes by mask length // PrefixDistribution represents the distribution of prefixes by mask length
//
//nolint:tagliatelle // field names are the existing JSON API
type PrefixDistribution struct { type PrefixDistribution struct {
MaskLength int `json:"mask_length"` MaskLength int `json:"mask_length"`
Count int `json:"count"` Count int `json:"count"`
} }
// IPInfo represents comprehensive IP information for the /ip endpoint // IPInfo represents comprehensive IP information for the /ip endpoint
//
//nolint:tagliatelle // field names are the existing JSON API
type IPInfo struct { type IPInfo struct {
IP string `json:"ip"` IP string `json:"ip"`
PTR []string `json:"ptr,omitempty"` PTR []string `json:"ptr,omitempty"`
+36 -17
View File
@@ -19,21 +19,29 @@ func logSlowQuery(logger *logger.Logger, query string, start time.Time) {
} }
// queryRow wraps QueryRow with slow query logging // queryRow wraps QueryRow with slow query logging
// // nolint:unused // kept for consistency with other query wrappers
//nolint:unused // kept for consistency with other query wrappers func (d *Database) queryRow(query string, args ...interface{}) *sql.Row {
func (d *Database) queryRow(query string, args ...any) *sql.Row {
start := time.Now() start := time.Now()
defer logSlowQuery(d.logger, query, start) defer logSlowQuery(d.logger, query, start)
return d.db.QueryRowContext(context.Background(), query, args...) return d.db.QueryRow(query, args...)
}
// query wraps Query with slow query logging
// nolint:unused // kept for future use to ensure all queries go through slow query logging
func (d *Database) query(query string, args ...interface{}) (*sql.Rows, error) {
start := time.Now()
defer logSlowQuery(d.logger, query, start)
return d.db.Query(query, args...)
} }
// exec wraps Exec with slow query logging // exec wraps Exec with slow query logging
func (d *Database) exec(query string, args ...any) error { func (d *Database) exec(query string, args ...interface{}) error {
start := time.Now() start := time.Now()
defer logSlowQuery(d.logger, query, start) defer logSlowQuery(d.logger, query, start)
_, err := d.db.ExecContext(context.Background(), query, args...) _, err := d.db.Exec(query, args...)
return err return err
} }
@@ -41,22 +49,35 @@ func (d *Database) exec(query string, args ...any) error {
// loggingTx wraps sql.Tx to log slow queries // loggingTx wraps sql.Tx to log slow queries
type loggingTx struct { type loggingTx struct {
*sql.Tx *sql.Tx
logger *logger.Logger logger *logger.Logger
} }
// QueryRow wraps sql.Tx.QueryRow to log slow queries // QueryRow wraps sql.Tx.QueryRow to log slow queries
func (tx *loggingTx) QueryRow(query string, args ...any) *sql.Row { func (tx *loggingTx) QueryRow(query string, args ...interface{}) *sql.Row {
start := time.Now() start := time.Now()
defer logSlowQuery(tx.logger, query, start) defer logSlowQuery(tx.logger, query, start)
return tx.Tx.QueryRowContext(context.Background(), query, args...) return tx.Tx.QueryRow(query, args...)
}
// Query wraps sql.Tx.Query to log slow queries
func (tx *loggingTx) Query(query string, args ...interface{}) (*sql.Rows, error) {
start := time.Now()
defer logSlowQuery(tx.logger, query, start)
return tx.Tx.Query(query, args...)
}
// QueryContext wraps sql.Tx.QueryContext to log slow queries
func (tx *loggingTx) QueryContext(ctx context.Context, query string, args ...interface{}) (*sql.Rows, error) {
start := time.Now()
defer logSlowQuery(tx.logger, query, start)
return tx.Tx.QueryContext(ctx, query, args...)
} }
// QueryRowContext wraps sql.Tx.QueryRowContext to log slow queries // QueryRowContext wraps sql.Tx.QueryRowContext to log slow queries
func (tx *loggingTx) QueryRowContext( func (tx *loggingTx) QueryRowContext(ctx context.Context, query string, args ...interface{}) *sql.Row {
ctx context.Context, query string, args ...any,
) *sql.Row {
start := time.Now() start := time.Now()
defer logSlowQuery(tx.logger, query, start) defer logSlowQuery(tx.logger, query, start)
@@ -64,17 +85,15 @@ func (tx *loggingTx) QueryRowContext(
} }
// Exec wraps sql.Tx.Exec to log slow queries // Exec wraps sql.Tx.Exec to log slow queries
func (tx *loggingTx) Exec(query string, args ...any) (sql.Result, error) { func (tx *loggingTx) Exec(query string, args ...interface{}) (sql.Result, error) {
start := time.Now() start := time.Now()
defer logSlowQuery(tx.logger, query, start) defer logSlowQuery(tx.logger, query, start)
return tx.Tx.ExecContext(context.Background(), query, args...) return tx.Tx.Exec(query, args...)
} }
// ExecContext wraps sql.Tx.ExecContext to log slow queries // ExecContext wraps sql.Tx.ExecContext to log slow queries
func (tx *loggingTx) ExecContext( func (tx *loggingTx) ExecContext(ctx context.Context, query string, args ...interface{}) (sql.Result, error) {
ctx context.Context, query string, args ...any,
) (sql.Result, error) {
start := time.Now() start := time.Now()
defer logSlowQuery(tx.logger, query, start) defer logSlowQuery(tx.logger, query, start)
-1
View File
@@ -27,7 +27,6 @@ func prefixMaskLength(prefix string) (int, error) {
if err != nil { if err != nil {
return 0, fmt.Errorf("invalid prefix %s: %w", prefix, err) return 0, fmt.Errorf("invalid prefix %s: %w", prefix, err)
} }
maskLength, _ := network.Mask.Size() maskLength, _ := network.Mask.Size()
return maskLength, nil return maskLength, nil
-1
View File
@@ -74,7 +74,6 @@ func getSourceAttrs() []slog.Attr {
// Get the function name // Get the function name
fn := runtime.FuncForPC(pc) fn := runtime.FuncForPC(pc)
var funcName string var funcName string
if fn != nil { if fn != nil {
funcName = filepath.Base(fn.Name()) funcName = filepath.Base(fn.Name())
+19 -19
View File
@@ -60,17 +60,22 @@ func New() *Tracker {
} }
} }
// SetConnected updates the connection status // SetConnected updates the connection status. Every connection after the
// first counts one reconnect.
func (t *Tracker) SetConnected(connected bool) { func (t *Tracker) SetConnected(connected bool) {
wasConnected := t.isConnected.Swap(connected) t.isConnected.Store(connected)
if connected { if !connected {
t.mu.Lock() return
t.connectedSince = time.Now() }
t.mu.Unlock()
// Increment reconnect count (but not for the initial connection) t.mu.Lock()
if wasConnected || t.reconnectCount.Load() > 0 { // connectedSince is zero until the first connection.
t.reconnectCount.Add(1) connectedBefore := !t.connectedSince.IsZero()
} t.connectedSince = time.Now()
t.mu.Unlock()
if connectedBefore {
t.reconnectCount.Add(1)
} }
} }
@@ -115,11 +120,9 @@ func (t *Tracker) GetStreamMetrics() StreamMetrics {
if msgCount >= 0 { if msgCount >= 0 {
totalMessages = uint64(msgCount) totalMessages = uint64(msgCount)
} }
if byteCount >= 0 { if byteCount >= 0 {
totalBytes = uint64(byteCount) totalBytes = uint64(byteCount)
} }
if wireByteCount >= 0 { if wireByteCount >= 0 {
totalWireBytes = uint64(wireByteCount) totalWireBytes = uint64(wireByteCount)
} }
@@ -163,7 +166,7 @@ func (t *Tracker) RecordWithdrawal() {
func (t *Tracker) SetBGPPeerCount(count int) { func (t *Tracker) SetBGPPeerCount(count int) {
// BGP peer count is always small (< 1000), so int32 is safe // BGP peer count is always small (< 1000), so int32 is safe
if count > 0 && count < 1<<31 { if count > 0 && count < 1<<31 {
t.bgpPeerCount.Store(int32(count)) t.bgpPeerCount.Store(int32(count)) //nolint:gosec // count is validated
} }
} }
@@ -211,8 +214,7 @@ type StreamMetrics struct {
TotalMessages uint64 TotalMessages uint64
// TotalBytes is the total number of decompressed bytes received since startup // TotalBytes is the total number of decompressed bytes received since startup
TotalBytes uint64 TotalBytes uint64
// TotalWireBytes is the total number of bytes received on the wire (before // TotalWireBytes is the total number of bytes received on the wire (before decompression)
// decompression)
TotalWireBytes uint64 TotalWireBytes uint64
// ConnectedSince is the time when the current connection was established // ConnectedSince is the time when the current connection was established
ConnectedSince time.Time ConnectedSince time.Time
@@ -220,11 +222,9 @@ type StreamMetrics struct {
Connected bool Connected bool
// MessagesPerSec is the rate of messages received per second (1-minute average) // MessagesPerSec is the rate of messages received per second (1-minute average)
MessagesPerSec float64 MessagesPerSec float64
// BitsPerSec is the rate of decompressed bits received per second (1-minute // BitsPerSec is the rate of decompressed bits received per second (1-minute average)
// average)
BitsPerSec float64 BitsPerSec float64
// WireBitsPerSec is the rate of bits received on the wire per second // WireBitsPerSec is the rate of bits received on the wire per second (1-minute average)
// (1-minute average)
WireBitsPerSec float64 WireBitsPerSec float64
// ReconnectCount is the number of reconnections since startup // ReconnectCount is the number of reconnections since startup
ReconnectCount uint64 ReconnectCount uint64
+21
View File
@@ -0,0 +1,21 @@
package metrics
import "testing"
// TestReconnectCount connects three times with a disconnect between each, the
// way the streamer does, and expects every connection after the first to count
// one reconnect.
func TestReconnectCount(t *testing.T) {
tracker := New()
tracker.SetConnected(true)
tracker.SetConnected(false)
tracker.SetConnected(true)
tracker.SetConnected(false)
tracker.SetConnected(true)
const wantReconnects = 2
if got := tracker.GetReconnectCount(); got != wantReconnects {
t.Fatalf("reconnect count = %d, want %d", got, wantReconnects)
}
}
+5 -15
View File
@@ -1,5 +1,4 @@
// Package ristypes defines the data structures for RIS Live BGP messages and // Package ristypes defines the data structures for RIS Live BGP messages and announcements.
// announcements.
package ristypes package ristypes
import ( import (
@@ -18,30 +17,25 @@ type ASPath []int
func (p *ASPath) UnmarshalJSON(data []byte) error { func (p *ASPath) UnmarshalJSON(data []byte) error {
// First try to unmarshal as a simple array of integers // First try to unmarshal as a simple array of integers
var simple []int var simple []int
if err := json.Unmarshal(data, &simple); err == nil {
err := json.Unmarshal(data, &simple)
if err == nil {
*p = ASPath(simple) *p = ASPath(simple)
return nil return nil
} }
// If that fails, unmarshal as array of interfaces and flatten // If that fails, unmarshal as array of interfaces and flatten
var raw []any var raw []interface{}
if err := json.Unmarshal(data, &raw); err != nil {
err = json.Unmarshal(data, &raw)
if err != nil {
return err return err
} }
// Flatten the array // Flatten the array
result := make([]int, 0) result := make([]int, 0)
for _, item := range raw { for _, item := range raw {
switch v := item.(type) { switch v := item.(type) {
case float64: case float64:
result = append(result, int(v)) result = append(result, int(v))
case []any: case []interface{}:
// Nested array - flatten it // Nested array - flatten it
for _, nested := range v { for _, nested := range v {
if num, ok := nested.(float64); ok { if num, ok := nested.(float64); ok {
@@ -68,8 +62,6 @@ type RISLiveMessage struct {
// It contains metadata about the BGP session (peer, ASN, host) along with // It contains metadata about the BGP session (peer, ASN, host) along with
// the actual BGP update data including AS path, communities, announcements, // the actual BGP update data including AS path, communities, announcements,
// and withdrawals. // and withdrawals.
//
//nolint:tagliatelle // field names are the RIS Live feed's
type RISMessage struct { type RISMessage struct {
Type string `json:"type"` Type string `json:"type"`
Timestamp float64 `json:"timestamp"` Timestamp float64 `json:"timestamp"`
@@ -98,8 +90,6 @@ type RISMessage struct {
// RISAnnouncement represents a BGP route announcement within a RIS message. // RISAnnouncement represents a BGP route announcement within a RIS message.
// It contains the next hop IP address and the list of prefixes being announced // It contains the next hop IP address and the list of prefixes being announced
// via that next hop. // via that next hop.
//
//nolint:tagliatelle // field names are the RIS Live feed's
type RISAnnouncement struct { type RISAnnouncement struct {
NextHop string `json:"next_hop"` NextHop string `json:"next_hop"`
Prefixes []string `json:"prefixes"` Prefixes []string `json:"prefixes"`
+7 -24
View File
@@ -1,12 +1,10 @@
package ristypes_test package ristypes
import ( import (
"bufio" "bufio"
"encoding/json" "encoding/json"
"os" "os"
"testing" "testing"
"git.eeqj.de/sneak/routewatch/internal/ristypes"
) )
// messageExamplesPath is the captured RIS Live feed used as decode fixtures, // messageExamplesPath is the captured RIS Live feed used as decode fixtures,
@@ -17,43 +15,32 @@ const messageExamplesPath = "../../docs/message-examples.json"
// streamer does and checks that the fields no handler reads (Community, Raw) // streamer does and checks that the fields no handler reads (Community, Raw)
// stay empty while the fields handlers use (Path, Announcements) still decode. // stay empty while the fields handlers use (Path, Announcements) still decode.
func TestDecodeDropsCommunityAndRaw(t *testing.T) { func TestDecodeDropsCommunityAndRaw(t *testing.T) {
t.Parallel()
f, err := os.Open(messageExamplesPath) f, err := os.Open(messageExamplesPath)
if err != nil { if err != nil {
t.Fatalf("open fixtures: %v", err) t.Fatalf("open fixtures: %v", err)
} }
defer f.Close()
defer func() {
_ = f.Close()
}()
var messages, withPath, withAnnouncements int var messages, withPath, withAnnouncements int
scanner := bufio.NewScanner(f) scanner := bufio.NewScanner(f)
scanner.Buffer(make([]byte, 0, 64*1024), 1024*1024) scanner.Buffer(make([]byte, 0, 64*1024), 1024*1024)
for scanner.Scan() { for scanner.Scan() {
line := scanner.Bytes() line := scanner.Bytes()
if len(line) == 0 { if len(line) == 0 {
continue continue
} }
var wrapper ristypes.RISLiveMessage var wrapper RISLiveMessage
if err := json.Unmarshal(line, &wrapper); err != nil {
err := json.Unmarshal(line, &wrapper)
if err != nil {
t.Fatalf("unmarshal message %d: %v", messages+1, err) t.Fatalf("unmarshal message %d: %v", messages+1, err)
} }
messages++ messages++
msg := wrapper.Data msg := wrapper.Data
if msg.Community != nil { if msg.Community != nil {
t.Errorf("message %d: Community decoded, want empty: %v", t.Errorf("message %d: Community decoded, want empty: %v", messages, msg.Community)
messages, msg.Community)
} }
if msg.Raw != "" { if msg.Raw != "" {
t.Errorf("message %d: Raw decoded, want empty", messages) t.Errorf("message %d: Raw decoded, want empty", messages)
} }
@@ -61,14 +48,11 @@ func TestDecodeDropsCommunityAndRaw(t *testing.T) {
if len(msg.Path) > 0 { if len(msg.Path) > 0 {
withPath++ withPath++
} }
if len(msg.Announcements) > 0 { if len(msg.Announcements) > 0 {
withAnnouncements++ withAnnouncements++
} }
} }
if err := scanner.Err(); err != nil {
err = scanner.Err()
if err != nil {
t.Fatalf("scan fixtures: %v", err) t.Fatalf("scan fixtures: %v", err)
} }
@@ -78,7 +62,6 @@ func TestDecodeDropsCommunityAndRaw(t *testing.T) {
// The fixtures include announcement messages; a used field must still decode, // The fixtures include announcement messages; a used field must still decode,
// otherwise an empty Community/Raw would prove nothing. // otherwise an empty Community/Raw would prove nothing.
if withPath == 0 || withAnnouncements == 0 { if withPath == 0 || withAnnouncements == 0 {
t.Fatalf("used fields did not decode: withPath=%d withAnnouncements=%d", t.Fatalf("used fields did not decode: withPath=%d withAnnouncements=%d", withPath, withAnnouncements)
withPath, withAnnouncements)
} }
} }
+16 -32
View File
@@ -1,11 +1,10 @@
// Package routewatch contains the primary RouteWatch type that represents a // Package routewatch contains the primary RouteWatch type that represents a running instance
// running instance of the application and contains pointers to its core // of the application and contains pointers to its core dependencies, and is responsible for initialization.
// dependencies, and is responsible for initialization.
package routewatch package routewatch
import ( import (
"context" "context"
"errors" "fmt"
"sync" "sync"
"time" "time"
@@ -19,10 +18,6 @@ import (
"go.uber.org/fx" "go.uber.org/fx"
) )
// errNonBatchedNotImplemented is returned by Run when batched database writes
// are disabled, since only the batched handlers exist.
var errNonBatchedNotImplemented = errors.New("non-batched handlers not implemented")
// Dependencies contains all dependencies for RouteWatch // Dependencies contains all dependencies for RouteWatch
type Dependencies struct { type Dependencies struct {
fx.In fx.In
@@ -73,10 +68,8 @@ func (rw *RouteWatch) Run(ctx context.Context) error {
// Apply runtime limit if specified // Apply runtime limit if specified
if rw.maxRuntime > 0 { if rw.maxRuntime > 0 {
var cancel context.CancelFunc var cancel context.CancelFunc
ctx, cancel = context.WithTimeout(ctx, rw.maxRuntime) ctx, cancel = context.WithTimeout(ctx, rw.maxRuntime)
defer cancel() defer cancel()
rw.logger.Info("Running with time limit", "max_runtime", rw.maxRuntime) rw.logger.Info("Running with time limit", "max_runtime", rw.maxRuntime)
} }
@@ -103,31 +96,29 @@ func (rw *RouteWatch) Run(ctx context.Context) error {
// Non-batched handlers not implemented yet // Non-batched handlers not implemented yet
rw.logger.Error("Non-batched handlers not implemented") rw.logger.Error("Non-batched handlers not implemented")
return errNonBatchedNotImplemented return fmt.Errorf("non-batched handlers not implemented")
} }
// No longer need routing table handler - PrefixHandler maintains live_routes table // No longer need routing table handler - PrefixHandler maintains live_routes table
// Start streaming // Start streaming
err := rw.streamer.Start(ctx) if err := rw.streamer.Start(); err != nil {
if err != nil {
return err return err
} }
// Start HTTP server // Start HTTP server
err = rw.server.Start() if err := rw.server.Start(); err != nil {
if err != nil {
return err return err
} }
// Start ASN WHOIS fetcher for background updates // Start ASN WHOIS fetcher for background updates
rw.asnFetcher = NewASNFetcher(rw.db, rw.logger.Logger) rw.asnFetcher = NewASNFetcher(rw.db, rw.logger.Logger)
rw.asnFetcher.Start(ctx) rw.asnFetcher.Start()
rw.server.SetASNFetcher(rw.asnFetcher) rw.server.SetASNFetcher(rw.asnFetcher)
// Start database maintenance goroutine // Start database maintenance goroutine
rw.dbMaintainer = NewDBMaintainer(rw.db, rw.logger.Logger) rw.dbMaintainer = NewDBMaintainer(rw.db, rw.logger.Logger)
rw.dbMaintainer.Start(ctx) rw.dbMaintainer.Start()
// Wait for context cancellation // Wait for context cancellation
<-ctx.Done() <-ctx.Done()
@@ -136,14 +127,13 @@ func (rw *RouteWatch) Run(ctx context.Context) error {
} }
// Shutdown performs graceful shutdown of all services // Shutdown performs graceful shutdown of all services
func (rw *RouteWatch) Shutdown(ctx context.Context) { func (rw *RouteWatch) Shutdown() {
rw.mu.Lock() rw.mu.Lock()
if rw.shutdown { if rw.shutdown {
rw.mu.Unlock() rw.mu.Unlock()
return return
} }
rw.shutdown = true rw.shutdown = true
rw.mu.Unlock() rw.mu.Unlock()
@@ -152,17 +142,14 @@ func (rw *RouteWatch) Shutdown(ctx context.Context) {
rw.logger.Info("Flushing database handler") rw.logger.Info("Flushing database handler")
rw.dbHandler.Stop() rw.dbHandler.Stop()
} }
if rw.peerHandler != nil { if rw.peerHandler != nil {
rw.logger.Info("Flushing peer handler") rw.logger.Info("Flushing peer handler")
rw.peerHandler.Stop() rw.peerHandler.Stop()
} }
if rw.prefixHandler != nil { if rw.prefixHandler != nil {
rw.logger.Info("Flushing prefix handler") rw.logger.Info("Flushing prefix handler")
rw.prefixHandler.Stop() rw.prefixHandler.Stop()
} }
if rw.peeringHandler != nil { if rw.peeringHandler != nil {
rw.logger.Info("Flushing peering handler") rw.logger.Info("Flushing peering handler")
rw.peeringHandler.Stop() rw.peeringHandler.Stop()
@@ -181,15 +168,11 @@ func (rw *RouteWatch) Shutdown(ctx context.Context) {
// Stop services // Stop services
rw.streamer.Stop() rw.streamer.Stop()
// Stop HTTP server with a timeout of its own, however much of ctx is left // Stop HTTP server with a timeout
const serverStopTimeout = 5 * time.Second const serverStopTimeout = 5 * time.Second
stopCtx, cancel := context.WithTimeout(context.Background(), serverStopTimeout)
stopCtx, cancel := context.WithTimeout(
context.WithoutCancel(ctx), serverStopTimeout)
defer cancel() defer cancel()
if err := rw.server.Stop(stopCtx); err != nil {
err := rw.server.Stop(stopCtx)
if err != nil {
rw.logger.Error("Failed to stop HTTP server gracefully", "error", err) rw.logger.Error("Failed to stop HTTP server gracefully", "error", err)
} }
@@ -202,11 +185,12 @@ func (rw *RouteWatch) Shutdown(ctx context.Context) {
"bits_per_sec", metrics.BitsPerSec, "bits_per_sec", metrics.BitsPerSec,
"duration", time.Since(metrics.ConnectedSince), "duration", time.Since(metrics.ConnectedSince),
) )
} }
// getModule provides all fx dependencies // getModule provides all fx dependencies
func getModule() []fx.Option { func getModule() fx.Option {
return []fx.Option{ return fx.Options(
fx.Provide( fx.Provide(
logger.New, logger.New,
config.New, config.New,
@@ -219,5 +203,5 @@ func getModule() []fx.Option {
server.New, server.New,
New, New,
), ),
} )
} }
@@ -50,9 +50,7 @@ func newMockStore() *mockStore {
} }
// GetOrCreateASN mock implementation // GetOrCreateASN mock implementation
func (m *mockStore) GetOrCreateASN( func (m *mockStore) GetOrCreateASN(number int, timestamp time.Time) (*database.ASN, error) {
number int, timestamp time.Time,
) (*database.ASN, error) {
m.mu.Lock() m.mu.Lock()
defer m.mu.Unlock() defer m.mu.Unlock()
@@ -101,14 +99,11 @@ func (m *mockStore) UpdatePrefixesBatch(prefixes map[string]time.Time) error {
} }
} }
} }
return nil return nil
} }
// GetOrCreatePrefix mock implementation // GetOrCreatePrefix mock implementation
func (m *mockStore) GetOrCreatePrefix( func (m *mockStore) GetOrCreatePrefix(prefix string, timestamp time.Time) (*database.Prefix, error) {
prefix string, timestamp time.Time,
) (*database.Prefix, error) {
m.mu.Lock() m.mu.Lock()
defer m.mu.Unlock() defer m.mu.Unlock()
@@ -173,7 +168,7 @@ func (m *mockStore) RecordPeering(asA, asB int, _ time.Time) error {
} }
// UpdatePeer mock implementation // UpdatePeer mock implementation
func (m *mockStore) UpdatePeer(_ string, _ int, _ string, _ time.Time) error { func (m *mockStore) UpdatePeer(peerIP string, peerASN int, messageType string, timestamp time.Time) error {
// Simple mock - just return nil // Simple mock - just return nil
return nil return nil
} }
@@ -199,45 +194,41 @@ func (m *mockStore) GetStats() (database.Stats, error) {
} }
// GetStatsContext returns statistics about the mock store with context support // GetStatsContext returns statistics about the mock store with context support
func (m *mockStore) GetStatsContext(_ context.Context) (database.Stats, error) { func (m *mockStore) GetStatsContext(ctx context.Context) (database.Stats, error) {
return m.GetStats() return m.GetStats()
} }
// UpsertLiveRoute mock implementation // UpsertLiveRoute mock implementation
func (m *mockStore) UpsertLiveRoute(_ *database.LiveRoute) error { func (m *mockStore) UpsertLiveRoute(route *database.LiveRoute) error {
// Simple mock - just return nil // Simple mock - just return nil
return nil return nil
} }
// DeleteLiveRoute mock implementation // DeleteLiveRoute mock implementation
func (m *mockStore) DeleteLiveRoute(_ string, _ int, _ string) error { func (m *mockStore) DeleteLiveRoute(prefix string, originASN int, peerIP string) error {
// Simple mock - just return nil // Simple mock - just return nil
return nil return nil
} }
// GetPrefixDistribution mock implementation // GetPrefixDistribution mock implementation
func (m *mockStore) GetPrefixDistribution() ( func (m *mockStore) GetPrefixDistribution() (ipv4 []database.PrefixDistribution, ipv6 []database.PrefixDistribution, err error) {
[]database.PrefixDistribution, []database.PrefixDistribution, error,
) {
// Return empty distributions for now // Return empty distributions for now
return nil, nil, nil return nil, nil, nil
} }
// GetPrefixDistributionContext mock implementation with context support // GetPrefixDistributionContext mock implementation with context support
func (m *mockStore) GetPrefixDistributionContext(_ context.Context) ( func (m *mockStore) GetPrefixDistributionContext(ctx context.Context) (ipv4 []database.PrefixDistribution, ipv6 []database.PrefixDistribution, err error) {
[]database.PrefixDistribution, []database.PrefixDistribution, error,
) {
return m.GetPrefixDistribution() return m.GetPrefixDistribution()
} }
// GetLiveRouteCounts mock implementation // GetLiveRouteCounts mock implementation
func (m *mockStore) GetLiveRouteCounts() (int, int, error) { func (m *mockStore) GetLiveRouteCounts() (ipv4Count, ipv6Count int, err error) {
// Return mock counts // Return mock counts
return m.RouteCount / 2, m.RouteCount / 2, nil return m.RouteCount / 2, m.RouteCount / 2, nil
} }
// GetLiveRouteCountsContext mock implementation with context support // GetLiveRouteCountsContext mock implementation with context support
func (m *mockStore) GetLiveRouteCountsContext(_ context.Context) (int, int, error) { func (m *mockStore) GetLiveRouteCountsContext(ctx context.Context) (ipv4Count, ipv6Count int, err error) {
return m.GetLiveRouteCounts() return m.GetLiveRouteCounts()
} }
@@ -256,49 +247,39 @@ func (m *mockStore) GetASDetails(asn int) (*database.ASN, []database.LiveRoute,
} }
// GetASDetailsContext mock implementation with context support // GetASDetailsContext mock implementation with context support
func (m *mockStore) GetASDetailsContext( func (m *mockStore) GetASDetailsContext(ctx context.Context, asn int) (*database.ASN, []database.LiveRoute, error) {
_ context.Context, asn int,
) (*database.ASN, []database.LiveRoute, error) {
return m.GetASDetails(asn) return m.GetASDetails(asn)
} }
// GetPrefixDetails mock implementation // GetPrefixDetails mock implementation
func (m *mockStore) GetPrefixDetails(_ string) ([]database.LiveRoute, error) { func (m *mockStore) GetPrefixDetails(prefix string) ([]database.LiveRoute, error) {
// Return empty routes for now // Return empty routes for now
return []database.LiveRoute{}, nil return []database.LiveRoute{}, nil
} }
// GetPrefixDetailsContext mock implementation with context support // GetPrefixDetailsContext mock implementation with context support
func (m *mockStore) GetPrefixDetailsContext( func (m *mockStore) GetPrefixDetailsContext(ctx context.Context, prefix string) ([]database.LiveRoute, error) {
_ context.Context, prefix string,
) ([]database.LiveRoute, error) {
return m.GetPrefixDetails(prefix) return m.GetPrefixDetails(prefix)
} }
func (m *mockStore) GetRandomPrefixesByLength( func (m *mockStore) GetRandomPrefixesByLength(maskLength, ipVersion, limit int) ([]database.LiveRoute, error) {
_, _, _ int,
) ([]database.LiveRoute, error) {
// Return empty routes for now // Return empty routes for now
return []database.LiveRoute{}, nil return []database.LiveRoute{}, nil
} }
// GetRandomPrefixesByLengthContext mock implementation with context support // GetRandomPrefixesByLengthContext mock implementation with context support
func (m *mockStore) GetRandomPrefixesByLengthContext( func (m *mockStore) GetRandomPrefixesByLengthContext(ctx context.Context, maskLength, ipVersion, limit int) ([]database.LiveRoute, error) {
_ context.Context, maskLength, ipVersion, limit int,
) ([]database.LiveRoute, error) {
return m.GetRandomPrefixesByLength(maskLength, ipVersion, limit) return m.GetRandomPrefixesByLength(maskLength, ipVersion, limit)
} }
// GetASPeers mock implementation // GetASPeers mock implementation
func (m *mockStore) GetASPeers(_ int) ([]database.ASPeer, error) { func (m *mockStore) GetASPeers(asn int) ([]database.ASPeer, error) {
// Return empty peers for now // Return empty peers for now
return []database.ASPeer{}, nil return []database.ASPeer{}, nil
} }
// GetASPeersContext mock implementation with context support // GetASPeersContext mock implementation with context support
func (m *mockStore) GetASPeersContext( func (m *mockStore) GetASPeersContext(ctx context.Context, asn int) ([]database.ASPeer, error) {
_ context.Context, asn int,
) ([]database.ASPeer, error) {
return m.GetASPeers(asn) return m.GetASPeers(asn)
} }
@@ -308,11 +289,8 @@ func (m *mockStore) GetIPInfo(ip string) (*database.IPInfo, error) {
} }
// GetIPInfoContext mock implementation with context support // GetIPInfoContext mock implementation with context support
func (m *mockStore) GetIPInfoContext( func (m *mockStore) GetIPInfoContext(ctx context.Context, ip string) (*database.IPInfo, error) {
_ context.Context, ip string,
) (*database.IPInfo, error) {
now := time.Now() now := time.Now()
return &database.IPInfo{ return &database.IPInfo{
IP: ip, IP: ip,
Netblock: "8.8.8.0/24", Netblock: "8.8.8.0/24",
@@ -329,21 +307,17 @@ func (m *mockStore) GetIPInfoContext(
} }
// GetNextStaleASN mock implementation // GetNextStaleASN mock implementation
func (m *mockStore) GetNextStaleASN(_ context.Context, _ time.Duration) (int, error) { func (m *mockStore) GetNextStaleASN(ctx context.Context, staleThreshold time.Duration) (int, error) {
return 0, database.ErrNoStaleASN return 0, database.ErrNoStaleASN
} }
// UpdateASNWHOIS mock implementation // UpdateASNWHOIS mock implementation
func (m *mockStore) UpdateASNWHOIS( func (m *mockStore) UpdateASNWHOIS(ctx context.Context, update *database.ASNWHOISUpdate) error {
_ context.Context, _ *database.ASNWHOISUpdate,
) error {
return nil return nil
} }
// GetWHOISStats mock implementation // GetWHOISStats mock implementation
func (m *mockStore) GetWHOISStats( func (m *mockStore) GetWHOISStats(ctx context.Context, staleThreshold time.Duration) (*database.WHOISStats, error) {
_ context.Context, _ time.Duration,
) (*database.WHOISStats, error) {
m.mu.Lock() m.mu.Lock()
defer m.mu.Unlock() defer m.mu.Unlock()
@@ -371,22 +345,19 @@ func (m *mockStore) UpsertLiveRouteBatch(routes []*database.LiveRoute) error {
LastSeen: route.LastUpdated, LastSeen: route.LastUpdated,
} }
m.PrefixCount++ m.PrefixCount++
if route.IPVersion == 4 { if route.IPVersion == 4 {
m.IPv4Prefixes++ m.IPv4Prefixes++
} else { } else {
m.IPv6Prefixes++ m.IPv6Prefixes++
} }
} }
m.RouteCount++ m.RouteCount++
} }
return nil return nil
} }
// DeleteLiveRouteBatch mock implementation // DeleteLiveRouteBatch mock implementation
func (m *mockStore) DeleteLiveRouteBatch(_ []database.LiveRouteDeletion) error { func (m *mockStore) DeleteLiveRouteBatch(deletions []database.LiveRouteDeletion) error {
// Simple mock - just return nil // Simple mock - just return nil
return nil return nil
} }
@@ -406,53 +377,43 @@ func (m *mockStore) GetOrCreateASNBatch(asns map[int]time.Time) error {
m.ASNCount++ m.ASNCount++
} }
} }
return nil return nil
} }
// UpdatePeerBatch mock implementation // UpdatePeerBatch mock implementation
func (m *mockStore) UpdatePeerBatch(_ map[string]database.PeerUpdate) error { func (m *mockStore) UpdatePeerBatch(peers map[string]database.PeerUpdate) error {
// Simple mock - just return nil // Simple mock - just return nil
return nil return nil
} }
// Vacuum mock implementation // Vacuum mock implementation
func (m *mockStore) Vacuum(_ context.Context) error { func (m *mockStore) Vacuum(ctx context.Context) error {
return nil return nil
} }
// Analyze mock implementation // Analyze mock implementation
func (m *mockStore) Analyze(_ context.Context) error { func (m *mockStore) Analyze(ctx context.Context) error {
return nil return nil
} }
// Checkpoint mock implementation // Checkpoint mock implementation
func (m *mockStore) Checkpoint(_ context.Context) error { func (m *mockStore) Checkpoint(ctx context.Context) error {
return nil return nil
} }
// Ping mock implementation // Ping mock implementation
func (m *mockStore) Ping(_ context.Context) error { func (m *mockStore) Ping(ctx context.Context) error {
return nil return nil
} }
func TestRouteWatchLiveFeed(t *testing.T) { func TestRouteWatchLiveFeed(t *testing.T) {
t.Parallel()
if testing.Short() { if testing.Short() {
t.Skip("skipping live RIPE RIS network feed test in short mode; " + t.Skip("skipping live RIPE RIS network feed test in short mode; run without -short to include it")
"run without -short to include it")
} }
// Create mock database // Create mock database
mockDB := newMockStore() mockDB := newMockStore()
defer mockDB.Close()
defer func() {
err := mockDB.Close()
if err != nil {
t.Errorf("failed to close mock database: %v", err)
}
}()
logger := logger.New() logger := logger.New()
@@ -505,19 +466,16 @@ func TestRouteWatchLiveFeed(t *testing.T) {
if stats.ASNs == 0 { if stats.ASNs == 0 {
t.Error("Expected to receive some ASNs from live feed") t.Error("Expected to receive some ASNs from live feed")
} }
t.Logf("Received %d unique ASNs in 5 seconds", stats.ASNs) t.Logf("Received %d unique ASNs in 5 seconds", stats.ASNs)
if stats.Prefixes == 0 { if stats.Prefixes == 0 {
t.Error("Expected to receive some prefixes from live feed") t.Error("Expected to receive some prefixes from live feed")
} }
t.Logf("Received %d unique prefixes (%d IPv4, %d IPv6) in 5 seconds", stats.Prefixes, stats.IPv4Prefixes, stats.IPv6Prefixes)
t.Logf("Received %d unique prefixes (%d IPv4, %d IPv6) in 5 seconds",
stats.Prefixes, stats.IPv4Prefixes, stats.IPv6Prefixes)
if stats.Peerings == 0 { if stats.Peerings == 0 {
t.Error("Expected to receive some peerings from live feed") t.Error("Expected to receive some peerings from live feed")
} }
t.Logf("Recorded %d AS peering relationships in 5 seconds", stats.Peerings) t.Logf("Recorded %d AS peering relationships in 5 seconds", stats.Peerings)
} }
+1 -1
View File
@@ -1,3 +1,3 @@
package routewatch_test package routewatch
// Tests for routewatch package are in app_integration_test.go // Tests for routewatch package are in app_integration_test.go
+14 -18
View File
@@ -71,7 +71,7 @@ func NewASHandler(db database.Store, logger *logger.Logger) *ASHandler {
// the AS path information needed to track autonomous systems. // the AS path information needed to track autonomous systems.
func (h *ASHandler) WantsMessage(messageType string) bool { func (h *ASHandler) WantsMessage(messageType string) bool {
// We only care about UPDATE messages for the database // We only care about UPDATE messages for the database
return messageType == updateMessageType return messageType == "UPDATE"
} }
// QueueCapacity returns the recommended message queue size for this handler. // QueueCapacity returns the recommended message queue size for this handler.
@@ -124,20 +124,9 @@ func (h *ASHandler) HandleMessage(msg *ristypes.RISMessage) {
} }
} }
// Stop gracefully shuts down the ASHandler by signaling the background flush
// goroutine to terminate and waiting for it to complete. Any pending ASN
// operations in the current batch are flushed to the database before Stop
// returns. This method should be called during application shutdown to ensure
// no data is lost.
func (h *ASHandler) Stop() {
close(h.stopCh)
h.wg.Wait()
}
// flushLoop runs in a goroutine and periodically flushes batches // flushLoop runs in a goroutine and periodically flushes batches
func (h *ASHandler) flushLoop() { func (h *ASHandler) flushLoop() {
defer h.wg.Done() defer h.wg.Done()
ticker := time.NewTicker(asnBatchTimeout) ticker := time.NewTicker(asnBatchTimeout)
defer ticker.Stop() defer ticker.Stop()
@@ -160,8 +149,7 @@ func (h *ASHandler) flushLoop() {
} }
} }
// flushBatchLocked flushes the ASN batch to the database (must be called with // flushBatchLocked flushes the ASN batch to the database (must be called with mutex held)
// mutex held)
func (h *ASHandler) flushBatchLocked() { func (h *ASHandler) flushBatchLocked() {
if len(h.batch) == 0 { if len(h.batch) == 0 {
return return
@@ -176,13 +164,21 @@ func (h *ASHandler) flushBatchLocked() {
} }
// Process all ASNs in a single batch transaction // Process all ASNs in a single batch transaction
err := h.db.GetOrCreateASNBatch(asnMap) if err := h.db.GetOrCreateASNBatch(asnMap); err != nil {
if err != nil { h.logger.Error("Failed to process ASN batch", "error", err, "count", len(asnMap))
h.logger.Error("Failed to process ASN batch",
"error", err, "count", len(asnMap))
} }
// Clear batch // Clear batch
h.batch = h.batch[:0] h.batch = h.batch[:0]
h.lastFlush = time.Now() h.lastFlush = time.Now()
} }
// Stop gracefully shuts down the ASHandler by signaling the background flush
// goroutine to terminate and waiting for it to complete. Any pending ASN
// operations in the current batch are flushed to the database before Stop
// returns. This method should be called during application shutdown to ensure
// no data is lost.
func (h *ASHandler) Stop() {
close(h.stopCh)
h.wg.Wait()
}
+24 -21
View File
@@ -1,8 +1,8 @@
// Package routewatch contains the ASN WHOIS fetcher for background updates.
package routewatch package routewatch
import ( import (
"context" "context"
"errors"
"log/slog" "log/slog"
"sync" "sync"
"time" "time"
@@ -73,13 +73,10 @@ func NewASNFetcher(db database.Store, logger *slog.Logger) *ASNFetcher {
} }
} }
// Start begins the background ASN fetcher goroutine. It runs until Stop: // Start begins the background ASN fetcher goroutine.
// the end of ctx does not stop it. func (f *ASNFetcher) Start() {
func (f *ASNFetcher) Start(ctx context.Context) {
f.wg.Add(1) f.wg.Add(1)
go f.run()
go f.run(context.WithoutCancel(ctx))
f.logger.Info("ASN fetcher started", f.logger.Info("ASN fetcher started",
"base_interval", baseInterval, "base_interval", baseInterval,
"min_interval", minInterval, "min_interval", minInterval,
@@ -154,7 +151,10 @@ func (f *ASNFetcher) recordSuccess() {
f.consecutiveFails = 0 f.consecutiveFails = 0
// Decrease interval by half, but not below minimum // Decrease interval by half, but not below minimum
newInterval := max(f.currentInterval/backoffMultiplier, minInterval) newInterval := f.currentInterval / backoffMultiplier
if newInterval < minInterval {
newInterval = minInterval
}
if newInterval != f.currentInterval { if newInterval != f.currentInterval {
f.logger.Debug("Decreased fetch interval", f.logger.Debug("Decreased fetch interval",
@@ -177,7 +177,10 @@ func (f *ASNFetcher) recordFailure() {
f.consecutiveFails++ f.consecutiveFails++
// Exponential backoff: multiply by 2, capped at max // Exponential backoff: multiply by 2, capped at max
newInterval := min(f.currentInterval*backoffMultiplier, maxInterval) newInterval := f.currentInterval * backoffMultiplier
if newInterval > maxInterval {
newInterval = maxInterval
}
if newInterval != f.currentInterval { if newInterval != f.currentInterval {
f.logger.Debug("Increased fetch interval due to failure", f.logger.Debug("Increased fetch interval due to failure",
@@ -196,7 +199,7 @@ func (f *ASNFetcher) recordFailure() {
} }
// run is the main background loop. // run is the main background loop.
func (f *ASNFetcher) run(ctx context.Context) { func (f *ASNFetcher) run() {
defer f.wg.Done() defer f.wg.Done()
timer := time.NewTimer(f.getInterval()) timer := time.NewTimer(f.getInterval())
@@ -209,13 +212,13 @@ func (f *ASNFetcher) run(ctx context.Context) {
case asn := <-f.immediateQueue: case asn := <-f.immediateQueue:
// Process immediate request (respects lock) // Process immediate request (respects lock)
f.tryFetch(ctx, asn) f.tryFetch(asn)
// Reset timer after immediate fetch // Reset timer after immediate fetch
timer.Reset(f.getInterval()) timer.Reset(f.getInterval())
case <-timer.C: case <-timer.C:
// Background fetch of stale/missing ASN // Background fetch of stale/missing ASN
f.fetchNextStale(ctx) f.fetchNextStale()
// Reset timer with potentially updated interval // Reset timer with potentially updated interval
timer.Reset(f.getInterval()) timer.Reset(f.getInterval())
} }
@@ -224,7 +227,7 @@ func (f *ASNFetcher) run(ctx context.Context) {
// tryFetch attempts to fetch and update an ASN, respecting the fetch lock. // tryFetch attempts to fetch and update an ASN, respecting the fetch lock.
// Returns true if fetch was successful. // Returns true if fetch was successful.
func (f *ASNFetcher) tryFetch(ctx context.Context, asn int) bool { func (f *ASNFetcher) tryFetch(asn int) bool {
// Try to acquire lock, skip if another fetch is running // Try to acquire lock, skip if another fetch is running
if !f.fetchMu.TryLock() { if !f.fetchMu.TryLock() {
f.logger.Debug("Skipping fetch, another fetch in progress", "asn", asn) f.logger.Debug("Skipping fetch, another fetch in progress", "asn", asn)
@@ -233,11 +236,11 @@ func (f *ASNFetcher) tryFetch(ctx context.Context, asn int) bool {
} }
defer f.fetchMu.Unlock() defer f.fetchMu.Unlock()
return f.fetchAndUpdate(ctx, asn) return f.fetchAndUpdate(asn)
} }
// fetchNextStale finds and fetches the next ASN needing WHOIS data. // fetchNextStale finds and fetches the next ASN needing WHOIS data.
func (f *ASNFetcher) fetchNextStale(ctx context.Context) { func (f *ASNFetcher) fetchNextStale() {
// Try to acquire lock, skip if another fetch is running // Try to acquire lock, skip if another fetch is running
if !f.fetchMu.TryLock() { if !f.fetchMu.TryLock() {
f.logger.Debug("Skipping stale fetch, another fetch in progress") f.logger.Debug("Skipping stale fetch, another fetch in progress")
@@ -246,12 +249,12 @@ func (f *ASNFetcher) fetchNextStale(ctx context.Context) {
} }
defer f.fetchMu.Unlock() defer f.fetchMu.Unlock()
lookupCtx, cancel := context.WithTimeout(ctx, time.Minute) ctx, cancel := context.WithTimeout(context.Background(), time.Minute)
defer cancel() defer cancel()
asn, err := f.db.GetNextStaleASN(lookupCtx, whoisStaleThreshold) asn, err := f.db.GetNextStaleASN(ctx, whoisStaleThreshold)
if err != nil { if err != nil {
if !errors.Is(err, database.ErrNoStaleASN) { if err != database.ErrNoStaleASN {
f.logger.Error("Failed to get stale ASN", "error", err) f.logger.Error("Failed to get stale ASN", "error", err)
f.recordFailure() f.recordFailure()
} }
@@ -260,13 +263,13 @@ func (f *ASNFetcher) fetchNextStale(ctx context.Context) {
return return
} }
f.fetchAndUpdate(ctx, asn) f.fetchAndUpdate(asn)
} }
// fetchAndUpdate performs a WHOIS lookup and updates the database. // fetchAndUpdate performs a WHOIS lookup and updates the database.
// Returns true if successful. // Returns true if successful.
func (f *ASNFetcher) fetchAndUpdate(ctx context.Context, asn int) bool { func (f *ASNFetcher) fetchAndUpdate(asn int) bool {
ctx, cancel := context.WithTimeout(ctx, time.Minute) ctx, cancel := context.WithTimeout(context.Background(), time.Minute)
defer cancel() defer cancel()
f.logger.Info("Fetching WHOIS data", "asn", asn) f.logger.Info("Fetching WHOIS data", "asn", asn)
+9 -17
View File
@@ -55,19 +55,14 @@ func logDebugStats(logger *logger.Logger) {
// CLIEntry is the main entry point for the routewatch command-line interface. // CLIEntry is the main entry point for the routewatch command-line interface.
// It initializes the application using the fx dependency injection framework, // It initializes the application using the fx dependency injection framework,
// sets up signal handling for graceful shutdown, and starts the RouteWatch // sets up signal handling for graceful shutdown, and starts the RouteWatch service.
// service. This function blocks until the application receives a shutdown // This function blocks until the application receives a shutdown signal or encounters
// signal or encounters a fatal error. // a fatal error.
func CLIEntry() { func CLIEntry() {
app := fx.New( app := fx.New(
fx.Options(getModule()...), getModule(),
fx.StopTimeout(shutdownTimeout), // Allow 60 seconds for graceful shutdown fx.StopTimeout(shutdownTimeout), // Allow 60 seconds for graceful shutdown
fx.Invoke(func( fx.Invoke(func(lc fx.Lifecycle, rw *RouteWatch, logger *logger.Logger, shutdowner fx.Shutdowner) {
lc fx.Lifecycle,
rw *RouteWatch,
logger *logger.Logger,
shutdowner fx.Shutdowner,
) {
lc.Append(fx.Hook{ lc.Append(fx.Hook{
OnStart: func(ctx context.Context) error { OnStart: func(ctx context.Context) error {
// Start debug stats logging // Start debug stats logging
@@ -80,25 +75,22 @@ func CLIEntry() {
go func() { go func() {
<-sigCh <-sigCh
logger.Info("Received shutdown signal") logger.Info("Received shutdown signal")
if err := shutdowner.Shutdown(); err != nil {
err := shutdowner.Shutdown()
if err != nil {
logger.Error("Failed to shutdown gracefully", "error", err) logger.Error("Failed to shutdown gracefully", "error", err)
} }
}() }()
go func() { go func() {
err := rw.Run(ctx) if err := rw.Run(ctx); err != nil {
if err != nil {
logger.Error("RouteWatch error", "error", err) logger.Error("RouteWatch error", "error", err)
} }
}() }()
return nil return nil
}, },
OnStop: func(ctx context.Context) error { OnStop: func(_ context.Context) error {
logger.Info("Shutting down RouteWatch") logger.Info("Shutting down RouteWatch")
rw.Shutdown(ctx) rw.Shutdown()
return nil return nil
}, },
+58 -60
View File
@@ -1,3 +1,4 @@
// Package routewatch contains the database maintainer for background maintenance tasks.
package routewatch package routewatch
import ( import (
@@ -63,13 +64,10 @@ func NewDBMaintainer(db database.Store, logger *slog.Logger) *DBMaintainer {
} }
} }
// Start begins the background maintenance goroutine. It runs until Stop: // Start begins the background maintenance goroutine.
// the end of ctx does not stop it. func (m *DBMaintainer) Start() {
func (m *DBMaintainer) Start(ctx context.Context) {
m.wg.Add(1) m.wg.Add(1)
go m.run()
go m.run(context.WithoutCancel(ctx))
m.logger.Info("Database maintainer started", m.logger.Info("Database maintainer started",
"checkpoint_interval", checkpointInterval, "checkpoint_interval", checkpointInterval,
"vacuum_interval", vacuumInterval, "vacuum_interval", vacuumInterval,
@@ -85,17 +83,15 @@ func (m *DBMaintainer) Stop() {
} }
// run is the main background loop. // run is the main background loop.
func (m *DBMaintainer) run(ctx context.Context) { func (m *DBMaintainer) run() {
defer m.wg.Done() defer m.wg.Done()
// Use different timers for each task // Use different timers for each task
checkpointTimer := time.NewTimer(checkpointInterval) checkpointTimer := time.NewTimer(checkpointInterval)
defer checkpointTimer.Stop()
vacuumTimer := time.NewTimer(vacuumInterval) vacuumTimer := time.NewTimer(vacuumInterval)
defer vacuumTimer.Stop()
analyzeTimer := time.NewTimer(analyzeInterval) analyzeTimer := time.NewTimer(analyzeInterval)
defer checkpointTimer.Stop()
defer vacuumTimer.Stop()
defer analyzeTimer.Stop() defer analyzeTimer.Stop()
for { for {
@@ -104,88 +100,90 @@ func (m *DBMaintainer) run(ctx context.Context) {
return return
case <-checkpointTimer.C: case <-checkpointTimer.C:
m.runCheckpoint(ctx) m.runCheckpoint()
checkpointTimer.Reset(checkpointInterval) checkpointTimer.Reset(checkpointInterval)
case <-vacuumTimer.C: case <-vacuumTimer.C:
m.runVacuum(ctx) m.runVacuum()
vacuumTimer.Reset(vacuumInterval) vacuumTimer.Reset(vacuumInterval)
case <-analyzeTimer.C: case <-analyzeTimer.C:
m.runAnalyze(ctx) m.runAnalyze()
analyzeTimer.Reset(analyzeInterval) analyzeTimer.Reset(analyzeInterval)
} }
} }
} }
// runCheckpoint performs a WAL checkpoint to keep the WAL file small. // runCheckpoint performs a WAL checkpoint to keep the WAL file small.
func (m *DBMaintainer) runCheckpoint(ctx context.Context) { func (m *DBMaintainer) runCheckpoint() {
duration, err := runWithTimeout(ctx, checkpointTimeout, m.db.Checkpoint) ctx, cancel := context.WithTimeout(context.Background(), checkpointTimeout)
m.recordResult(&m.lastCheckpoint, &m.lastCheckpointError, &m.checkpointCount, err) defer cancel()
startTime := time.Now()
err := m.db.Checkpoint(ctx)
m.statsMu.Lock()
m.lastCheckpoint = time.Now()
m.lastCheckpointError = err
if err == nil {
m.checkpointCount++
}
m.statsMu.Unlock()
if err != nil { if err != nil {
m.logger.Error("WAL checkpoint failed", "error", err, "duration", duration) m.logger.Error("WAL checkpoint failed", "error", err, "duration", time.Since(startTime))
} else { } else {
m.logger.Debug("WAL checkpoint completed", "duration", duration) m.logger.Debug("WAL checkpoint completed", "duration", time.Since(startTime))
} }
} }
// runVacuum performs an incremental vacuum operation on the database. // runVacuum performs an incremental vacuum operation on the database.
func (m *DBMaintainer) runVacuum(ctx context.Context) { func (m *DBMaintainer) runVacuum() {
m.logger.Debug("Running incremental vacuum") ctx, cancel := context.WithTimeout(context.Background(), vacuumTimeout)
defer cancel()
duration, err := runWithTimeout(ctx, vacuumTimeout, m.db.Vacuum) m.logger.Debug("Running incremental vacuum")
m.recordResult(&m.lastVacuum, &m.lastVacuumError, &m.vacuumCount, err) startTime := time.Now()
err := m.db.Vacuum(ctx)
m.statsMu.Lock()
m.lastVacuum = time.Now()
m.lastVacuumError = err
if err == nil {
m.vacuumCount++
}
m.statsMu.Unlock()
if err != nil { if err != nil {
m.logger.Error("Incremental vacuum failed", "error", err, "duration", duration) m.logger.Error("Incremental vacuum failed", "error", err, "duration", time.Since(startTime))
} else { } else {
m.logger.Debug("Incremental vacuum completed", "duration", duration) m.logger.Debug("Incremental vacuum completed", "duration", time.Since(startTime))
} }
} }
// runAnalyze performs an ANALYZE operation on the database. // runAnalyze performs an ANALYZE operation on the database.
func (m *DBMaintainer) runAnalyze(ctx context.Context) { func (m *DBMaintainer) runAnalyze() {
m.logger.Info("Starting database ANALYZE") ctx, cancel := context.WithTimeout(context.Background(), analyzeTimeout)
duration, err := runWithTimeout(ctx, analyzeTimeout, m.db.Analyze)
m.recordResult(&m.lastAnalyze, &m.lastAnalyzeError, &m.analyzeCount, err)
if err != nil {
m.logger.Error("ANALYZE failed", "error", err, "duration", duration)
} else {
m.logger.Info("ANALYZE completed", "duration", duration)
}
}
// runWithTimeout runs operation with a context that ends after timeout and
// returns how long it took and its error.
func runWithTimeout(
ctx context.Context,
timeout time.Duration,
operation func(context.Context) error,
) (time.Duration, error) {
ctx, cancel := context.WithTimeout(ctx, timeout)
defer cancel() defer cancel()
m.logger.Info("Starting database ANALYZE")
startTime := time.Now() startTime := time.Now()
err := operation(ctx)
return time.Since(startTime), err err := m.db.Analyze(ctx)
}
// recordResult stores when a maintenance task last ran and its error, and
// counts it when it succeeded.
func (m *DBMaintainer) recordResult(
lastRun *time.Time, lastErr *error, successCount *int, err error,
) {
m.statsMu.Lock() m.statsMu.Lock()
defer m.statsMu.Unlock() m.lastAnalyze = time.Now()
m.lastAnalyzeError = err
*lastRun = time.Now()
*lastErr = err
if err == nil { if err == nil {
*successCount++ m.analyzeCount++
}
m.statsMu.Unlock()
if err != nil {
m.logger.Error("ANALYZE failed", "error", err, "duration", time.Since(startTime))
} else {
m.logger.Info("ANALYZE completed", "duration", time.Since(startTime))
} }
} }
+14 -16
View File
@@ -1,16 +1,10 @@
package routewatch package routewatch
import ( import (
"slices"
"git.eeqj.de/sneak/routewatch/internal/logger" "git.eeqj.de/sneak/routewatch/internal/logger"
"git.eeqj.de/sneak/routewatch/internal/ristypes" "git.eeqj.de/sneak/routewatch/internal/ristypes"
) )
// updateMessageType is the RIS message type of BGP UPDATE messages, the only
// type the database handlers process.
const updateMessageType = "UPDATE"
// SimpleHandler is a basic implementation of streamer.MessageHandler that // SimpleHandler is a basic implementation of streamer.MessageHandler that
// filters messages by type and delegates processing to a callback function. // filters messages by type and delegates processing to a callback function.
// It provides a simple way to handle specific RIS message types without // It provides a simple way to handle specific RIS message types without
@@ -21,11 +15,10 @@ type SimpleHandler struct {
callback func(*ristypes.RISMessage) callback func(*ristypes.RISMessage)
} }
// NewSimpleHandler creates a new SimpleHandler that accepts specific message // NewSimpleHandler creates a new SimpleHandler that accepts specific message types.
// types. The messageTypes parameter specifies which RIS message types this // The messageTypes parameter specifies which RIS message types this handler will process.
// handler will process. If messageTypes is empty, the handler will accept all // If messageTypes is empty, the handler will accept all message types.
// message types. The callback function is invoked for each message that // The callback function is invoked for each message that passes the type filter.
// passes the type filter.
func NewSimpleHandler( func NewSimpleHandler(
logger *logger.Logger, logger *logger.Logger,
messageTypes []string, messageTypes []string,
@@ -38,17 +31,22 @@ func NewSimpleHandler(
} }
} }
// WantsMessage returns true if this handler wants to process messages of the // WantsMessage returns true if this handler wants to process messages of the given type.
// given type. It checks whether messageType is in the handler's configured // It checks whether messageType is in the handler's configured list of accepted types.
// list of accepted types. If no specific types were configured (empty // If no specific types were configured (empty messageTypes slice), it returns true for all types.
// messageTypes slice), it returns true for all types.
func (h *SimpleHandler) WantsMessage(messageType string) bool { func (h *SimpleHandler) WantsMessage(messageType string) bool {
// If no specific types are set, accept all messages // If no specific types are set, accept all messages
if len(h.messageTypes) == 0 { if len(h.messageTypes) == 0 {
return true return true
} }
return slices.Contains(h.messageTypes, messageType) for _, t := range h.messageTypes {
if t == messageType {
return true
}
}
return false
} }
// HandleMessage processes a RIS message by invoking the configured callback function. // HandleMessage processes a RIS message by invoking the configured callback function.
+13 -21
View File
@@ -1,8 +1,7 @@
package routewatch package routewatch
// peerhandler.go provides batched peer tracking functionality for BGP route // peerhandler.go provides batched peer tracking functionality for BGP route monitoring.
// monitoring. It tracks BGP peers from all incoming RIS messages and // It tracks BGP peers from all incoming RIS messages and maintains peer state in the database.
// maintains peer state in the database.
import ( import (
"strconv" "strconv"
@@ -93,10 +92,8 @@ func (h *PeerHandler) QueueCapacity() int {
func (h *PeerHandler) HandleMessage(msg *ristypes.RISMessage) { func (h *PeerHandler) HandleMessage(msg *ristypes.RISMessage) {
// Parse peer ASN from string // Parse peer ASN from string
peerASN := 0 peerASN := 0
if msg.PeerASN != "" { if msg.PeerASN != "" {
asn, err := strconv.Atoi(msg.PeerASN) if asn, err := strconv.Atoi(msg.PeerASN); err == nil {
if err == nil {
peerASN = asn peerASN = asn
} }
} }
@@ -118,16 +115,9 @@ func (h *PeerHandler) HandleMessage(msg *ristypes.RISMessage) {
} }
} }
// Stop gracefully stops the handler and flushes remaining batches
func (h *PeerHandler) Stop() {
close(h.stopCh)
h.wg.Wait()
}
// flushLoop runs in a goroutine and periodically flushes batches // flushLoop runs in a goroutine and periodically flushes batches
func (h *PeerHandler) flushLoop() { func (h *PeerHandler) flushLoop() {
defer h.wg.Done() defer h.wg.Done()
ticker := time.NewTicker(peerBatchTimeout) ticker := time.NewTicker(peerBatchTimeout)
defer ticker.Stop() defer ticker.Stop()
@@ -150,8 +140,7 @@ func (h *PeerHandler) flushLoop() {
} }
} }
// flushBatchLocked flushes the peer batch to the database (must be called // flushBatchLocked flushes the peer batch to the database (must be called with mutex held)
// with mutex held)
func (h *PeerHandler) flushBatchLocked() { func (h *PeerHandler) flushBatchLocked() {
if len(h.peerBatch) == 0 { if len(h.peerBatch) == 0 {
return return
@@ -160,8 +149,7 @@ func (h *PeerHandler) flushBatchLocked() {
// Deduplicate by peer IP, keeping the latest update for each peer // Deduplicate by peer IP, keeping the latest update for each peer
peerMap := make(map[string]peerUpdate) peerMap := make(map[string]peerUpdate)
for _, update := range h.peerBatch { for _, update := range h.peerBatch {
existing, ok := peerMap[update.peerIP] if existing, ok := peerMap[update.peerIP]; !ok || update.timestamp.After(existing.timestamp) {
if !ok || update.timestamp.After(existing.timestamp) {
peerMap[update.peerIP] = update peerMap[update.peerIP] = update
} }
} }
@@ -178,13 +166,17 @@ func (h *PeerHandler) flushBatchLocked() {
} }
// Process all peers in a single batch transaction // Process all peers in a single batch transaction
err := h.db.UpdatePeerBatch(dbPeerMap) if err := h.db.UpdatePeerBatch(dbPeerMap); err != nil {
if err != nil { h.logger.Error("Failed to process peer batch", "error", err, "count", len(dbPeerMap))
h.logger.Error("Failed to process peer batch",
"error", err, "count", len(dbPeerMap))
} }
// Clear batch // Clear batch
h.peerBatch = h.peerBatch[:0] h.peerBatch = h.peerBatch[:0]
h.lastFlush = time.Now() h.lastFlush = time.Now()
} }
// Stop gracefully stops the handler and flushes remaining batches
func (h *PeerHandler) Stop() {
close(h.stopCh)
h.wg.Wait()
}
+19 -24
View File
@@ -72,7 +72,7 @@ func NewPeeringHandler(db database.Store, logger *logger.Logger) *PeeringHandler
// the AS path information needed to extract peering relationships. // the AS path information needed to extract peering relationships.
func (h *PeeringHandler) WantsMessage(messageType string) bool { func (h *PeeringHandler) WantsMessage(messageType string) bool {
// We only care about UPDATE messages that have AS paths // We only care about UPDATE messages that have AS paths
return messageType == updateMessageType return messageType == "UPDATE"
} }
// QueueCapacity returns the buffer size for the handler's message queue. // QueueCapacity returns the buffer size for the handler's message queue.
@@ -101,7 +101,6 @@ func (h *PeeringHandler) HandleMessage(msg *ristypes.RISMessage) {
return return
} }
key := string(pathJSON) key := string(pathJSON)
h.mu.Lock() h.mu.Lock()
@@ -117,23 +116,6 @@ func (h *PeeringHandler) HandleMessage(msg *ristypes.RISMessage) {
h.mu.Unlock() h.mu.Unlock()
} }
// ProcessPeeringsNow triggers immediate processing of all accumulated AS
// paths into peering records. This bypasses the normal periodic processing
// schedule and is primarily intended for testing purposes.
func (h *PeeringHandler) ProcessPeeringsNow() {
h.processPeerings()
}
// Stop gracefully shuts down the handler by signaling the background
// goroutine to stop and performing a final synchronous processing of
// any remaining AS paths. This ensures no peering data is lost during
// shutdown.
func (h *PeeringHandler) Stop() {
close(h.stopCh)
// Process any remaining peerings synchronously
h.processPeerings()
}
// processLoop runs periodically to process AS paths into peerings // processLoop runs periodically to process AS paths into peerings
func (h *PeeringHandler) processLoop() { func (h *PeeringHandler) processLoop() {
ticker := time.NewTicker(peeringProcessInterval) ticker := time.NewTicker(peeringProcessInterval)
@@ -152,6 +134,13 @@ func (h *PeeringHandler) processLoop() {
} }
} }
// ProcessPeeringsNow triggers immediate processing of all accumulated AS
// paths into peering records. This bypasses the normal periodic processing
// schedule and is primarily intended for testing purposes.
func (h *PeeringHandler) ProcessPeeringsNow() {
h.processPeerings()
}
// processPeerings extracts peerings from AS paths and writes to database // processPeerings extracts peerings from AS paths and writes to database
func (h *PeeringHandler) processPeerings() { func (h *PeeringHandler) processPeerings() {
// Take the accumulated paths and replace the map with a fresh empty one // Take the accumulated paths and replace the map with a fresh empty one
@@ -171,14 +160,11 @@ func (h *PeeringHandler) processPeerings() {
type peeringKey struct { type peeringKey struct {
low, high int low, high int
} }
peerings := make(map[peeringKey]time.Time) peerings := make(map[peeringKey]time.Time)
for pathJSON, timestamp := range paths { for pathJSON, timestamp := range paths {
var path []int var path []int
if err := json.Unmarshal([]byte(pathJSON), &path); err != nil {
err := json.Unmarshal([]byte(pathJSON), &path)
if err != nil {
h.logger.Error("Failed to decode AS path", "error", err) h.logger.Error("Failed to decode AS path", "error", err)
continue continue
@@ -211,7 +197,6 @@ func (h *PeeringHandler) processPeerings() {
// Record peerings in database // Record peerings in database
start := time.Now() start := time.Now()
successCount := 0 successCount := 0
for key, ts := range peerings { for key, ts := range peerings {
err := h.db.RecordPeering(key.low, key.high, ts) err := h.db.RecordPeering(key.low, key.high, ts)
if err != nil { if err != nil {
@@ -233,3 +218,13 @@ func (h *PeeringHandler) processPeerings() {
"duration", time.Since(start), "duration", time.Since(start),
) )
} }
// Stop gracefully shuts down the handler by signaling the background
// goroutine to stop and performing a final synchronous processing of
// any remaining AS paths. This ensures no peering data is lost during
// shutdown.
func (h *PeeringHandler) Stop() {
close(h.stopCh)
// Process any remaining peerings synchronously
h.processPeerings()
}
@@ -53,8 +53,7 @@ func newTestHandler(db database.Store) *PeeringHandler {
} }
} }
// snapshot returns how many paths the handler tracks and how many it dropped. func snapshot(h *PeeringHandler) (tracked, dropped int) {
func snapshot(h *PeeringHandler) (int, int) {
h.mu.Lock() h.mu.Lock()
defer h.mu.Unlock() defer h.mu.Unlock()
@@ -83,8 +82,6 @@ func handle(h *PeeringHandler, ts time.Time, asns ...int) {
// empties the path map (the swap) and that a path seen again after a run is // empties the path map (the swap) and that a path seen again after a run is
// recorded in the next run too. // recorded in the next run too.
func TestPeeringHandlerProcessesEachRunAndEmpties(t *testing.T) { func TestPeeringHandlerProcessesEachRunAndEmpties(t *testing.T) {
t.Parallel()
store := &recordingStore{mockStore: newMockStore()} store := &recordingStore{mockStore: newMockStore()}
h := newTestHandler(store) h := newTestHandler(store)
@@ -97,7 +94,6 @@ func TestPeeringHandlerProcessesEachRunAndEmpties(t *testing.T) {
if tracked, _ := snapshot(h); tracked != 0 { if tracked, _ := snapshot(h); tracked != 0 {
t.Fatalf("map not empty after first run: %d paths remain", tracked) t.Fatalf("map not empty after first run: %d paths remain", tracked)
} }
if got := store.callCount(); got != 1 { if got := store.callCount(); got != 1 {
t.Fatalf("want 1 RecordPeering call after first run, got %d", got) t.Fatalf("want 1 RecordPeering call after first run, got %d", got)
} }
@@ -109,7 +105,6 @@ func TestPeeringHandlerProcessesEachRunAndEmpties(t *testing.T) {
if tracked, _ := snapshot(h); tracked != 0 { if tracked, _ := snapshot(h); tracked != 0 {
t.Fatalf("map not empty after second run: %d paths remain", tracked) t.Fatalf("map not empty after second run: %d paths remain", tracked)
} }
if got := store.callCount(); got != 2 { if got := store.callCount(); got != 2 {
t.Fatalf("want 2 RecordPeering calls after second run, got %d", got) t.Fatalf("want 2 RecordPeering calls after second run, got %d", got)
} }
@@ -119,8 +114,6 @@ func TestPeeringHandlerProcessesEachRunAndEmpties(t *testing.T) {
// and counts them, while a path already tracked is refreshed rather than // and counts them, while a path already tracked is refreshed rather than
// dropped. // dropped.
func TestPeeringHandlerCapDropsAndCounts(t *testing.T) { func TestPeeringHandlerCapDropsAndCounts(t *testing.T) {
t.Parallel()
store := &recordingStore{mockStore: newMockStore()} store := &recordingStore{mockStore: newMockStore()}
h := newTestHandler(store) h := newTestHandler(store)
@@ -133,7 +126,6 @@ func TestPeeringHandlerCapDropsAndCounts(t *testing.T) {
h.mu.Lock() h.mu.Lock()
h.asPaths[existing] = now h.asPaths[existing] = now
for i := 0; len(h.asPaths) < maxTrackedPaths; i++ { for i := 0; len(h.asPaths) < maxTrackedPaths; i++ {
h.asPaths[strconv.Itoa(i)] = now h.asPaths[strconv.Itoa(i)] = now
} }
@@ -146,7 +138,6 @@ func TestPeeringHandlerCapDropsAndCounts(t *testing.T) {
if tracked != maxTrackedPaths { if tracked != maxTrackedPaths {
t.Fatalf("want map size %d after drop, got %d", maxTrackedPaths, tracked) t.Fatalf("want map size %d after drop, got %d", maxTrackedPaths, tracked)
} }
if dropped != 1 { if dropped != 1 {
t.Fatalf("want dropped count 1, got %d", dropped) t.Fatalf("want dropped count 1, got %d", dropped)
} }
@@ -159,7 +150,6 @@ func TestPeeringHandlerCapDropsAndCounts(t *testing.T) {
if tracked != maxTrackedPaths { if tracked != maxTrackedPaths {
t.Fatalf("want map size %d after refresh, got %d", maxTrackedPaths, tracked) t.Fatalf("want map size %d after refresh, got %d", maxTrackedPaths, tracked)
} }
if dropped != 1 { if dropped != 1 {
t.Fatalf("want dropped count still 1 after refresh, got %d", dropped) t.Fatalf("want dropped count still 1 after refresh, got %d", dropped)
} }
@@ -167,9 +157,7 @@ func TestPeeringHandlerCapDropsAndCounts(t *testing.T) {
h.mu.Lock() h.mu.Lock()
gotTS := h.asPaths[existing] gotTS := h.asPaths[existing]
h.mu.Unlock() h.mu.Unlock()
if !gotTS.Equal(refreshed) { if !gotTS.Equal(refreshed) {
t.Fatalf("existing path timestamp not refreshed: want %v, got %v", t.Fatalf("existing path timestamp not refreshed: want %v, got %v", refreshed, gotTS)
refreshed, gotTS)
} }
} }
+200 -88
View File
@@ -33,8 +33,8 @@ const (
ipv6Version = 6 ipv6Version = 6
) )
// PrefixHandler tracks BGP prefixes and maintains a live routing table in the // PrefixHandler tracks BGP prefixes and maintains a live routing table in the database.
// database. Routes are added on announcement and deleted on withdrawal. // Routes are added on announcement and deleted on withdrawal.
type PrefixHandler struct { type PrefixHandler struct {
db database.Store db database.Store
logger *logger.Logger logger *logger.Logger
@@ -79,11 +79,10 @@ func (h *PrefixHandler) SetMetricsTracker(metrics *metrics.Tracker) {
h.metrics = metrics h.metrics = metrics
} }
// WantsMessage returns true if this handler wants to process messages of the // WantsMessage returns true if this handler wants to process messages of the given type
// given type
func (h *PrefixHandler) WantsMessage(messageType string) bool { func (h *PrefixHandler) WantsMessage(messageType string) bool {
// We only care about UPDATE messages for the routing table // We only care about UPDATE messages for the routing table
return messageType == updateMessageType return messageType == "UPDATE"
} }
// QueueCapacity returns the desired queue capacity for this handler // QueueCapacity returns the desired queue capacity for this handler
@@ -106,7 +105,24 @@ func (h *PrefixHandler) HandleMessage(msg *ristypes.RISMessage) {
h.mu.Lock() h.mu.Lock()
defer h.mu.Unlock() defer h.mu.Unlock()
// Process announcements // Withdrawals are queued before announcements, because the batch keeps
// each peer's last update for a prefix: a message that lists a prefix as
// both withdrawn and announced keeps the route (RFC 4271 section 4.3).
for _, prefix := range msg.Withdrawals {
h.batch = append(h.batch, prefixUpdate{
prefix: canonicalPrefix(prefix),
originASN: originASN, // Use the originASN from path if available
peer: msg.Peer,
messageType: "withdrawal",
timestamp: timestamp,
path: msg.Path,
})
// Record withdrawal in metrics
if h.metrics != nil {
h.metrics.RecordWithdrawal()
}
}
for _, announcement := range msg.Announcements { for _, announcement := range msg.Announcements {
for _, prefix := range announcement.Prefixes { for _, prefix := range announcement.Prefixes {
h.batch = append(h.batch, prefixUpdate{ h.batch = append(h.batch, prefixUpdate{
@@ -124,38 +140,15 @@ func (h *PrefixHandler) HandleMessage(msg *ristypes.RISMessage) {
} }
} }
// Process withdrawals
for _, prefix := range msg.Withdrawals {
h.batch = append(h.batch, prefixUpdate{
prefix: canonicalPrefix(prefix),
originASN: originASN, // Use the originASN from path if available
peer: msg.Peer,
messageType: "withdrawal",
timestamp: timestamp,
path: msg.Path,
})
// Record withdrawal in metrics
if h.metrics != nil {
h.metrics.RecordWithdrawal()
}
}
// Check if we need to flush // Check if we need to flush
if len(h.batch) >= prefixBatchSize { if len(h.batch) >= prefixBatchSize {
h.flushBatchLocked() h.flushBatchLocked()
} }
} }
// Stop gracefully stops the handler and flushes remaining batches
func (h *PrefixHandler) Stop() {
close(h.stopCh)
h.wg.Wait()
}
// flushLoop runs in a goroutine and periodically flushes batches // flushLoop runs in a goroutine and periodically flushes batches
func (h *PrefixHandler) flushLoop() { func (h *PrefixHandler) flushLoop() {
defer h.wg.Done() defer h.wg.Done()
ticker := time.NewTicker(prefixBatchTimeout) ticker := time.NewTicker(prefixBatchTimeout)
defer ticker.Stop() defer ticker.Stop()
@@ -178,8 +171,7 @@ func (h *PrefixHandler) flushLoop() {
} }
} }
// flushBatchLocked flushes the prefix batch to the database (must be called // flushBatchLocked flushes the prefix batch to the database (must be called with mutex held)
// with mutex held)
func (h *PrefixHandler) flushBatchLocked() { func (h *PrefixHandler) flushBatchLocked() {
if len(h.batch) == 0 { if len(h.batch) == 0 {
return return
@@ -188,32 +180,26 @@ func (h *PrefixHandler) flushBatchLocked() {
startTime := time.Now() startTime := time.Now()
batchSize := len(h.batch) batchSize := len(h.batch)
// Group updates by prefix to deduplicate // For each prefix and peer, keep only the last update the feed delivered.
// For each prefix, keep the latest update // Updates from different peers for the same prefix are all kept. Arrival
prefixMap := make(map[string]prefixUpdate) // order decides, not the timestamp: timestamps are whole seconds, so
// updates within one second carry the same one.
type prefixAndPeer struct{ prefix, peer string }
lastUpdates := make(map[prefixAndPeer]prefixUpdate)
for _, update := range h.batch { for _, update := range h.batch {
key := update.prefix lastUpdates[prefixAndPeer{update.prefix, update.peer}] = update
existing, ok := prefixMap[key]
if !ok || update.timestamp.After(existing.timestamp) {
prefixMap[key] = update
}
} }
// Collect routes to upsert and delete // Collect routes to upsert and delete
var ( var routesToUpsert []*database.LiveRoute
routesToUpsert []*database.LiveRoute var routesToDelete []database.LiveRouteDeletion
routesToDelete []database.LiveRouteDeletion
)
// Collect unique prefixes to update // Collect unique prefixes to update
prefixesToUpdate := make(map[string]time.Time) prefixesToUpdate := make(map[string]time.Time)
for _, update := range prefixMap { for _, update := range lastUpdates {
// Track prefix for both announcements and withdrawals // Track prefix for both announcements and withdrawals
seen, exists := prefixesToUpdate[update.prefix] if _, exists := prefixesToUpdate[update.prefix]; !exists || update.timestamp.After(prefixesToUpdate[update.prefix]) {
if !exists || update.timestamp.After(seen) {
prefixesToUpdate[update.prefix] = update.timestamp prefixesToUpdate[update.prefix] = update.timestamp
} }
@@ -227,8 +213,7 @@ func (h *PrefixHandler) flushBatchLocked() {
// Parse CIDR to get IP version // Parse CIDR to get IP version
_, ipVersion, err := parseCIDR(update.prefix) _, ipVersion, err := parseCIDR(update.prefix)
if err != nil { if err != nil {
h.logger.Error("Failed to parse CIDR for withdrawal", h.logger.Error("Failed to parse CIDR for withdrawal", "prefix", update.prefix, "error", err)
"prefix", update.prefix, "error", err)
continue continue
} }
@@ -243,45 +228,19 @@ func (h *PrefixHandler) flushBatchLocked() {
} }
} }
successCount := h.writeBatch(routesToUpsert, routesToDelete, prefixesToUpdate) // Process batch operations
elapsed := time.Since(startTime)
h.logger.Debug("Flushed prefix batch",
"batch_size", batchSize,
"unique_prefixes", len(prefixMap),
"success", successCount,
"duration_ms", elapsed.Milliseconds(),
)
// Clear batch
h.batch = h.batch[:0]
h.lastFlush = time.Now()
}
// writeBatch writes the collected route upserts, route deletions and prefix
// updates to the database and returns how many routes it wrote.
func (h *PrefixHandler) writeBatch(
routesToUpsert []*database.LiveRoute,
routesToDelete []database.LiveRouteDeletion,
prefixesToUpdate map[string]time.Time,
) int {
successCount := 0 successCount := 0
if len(routesToUpsert) > 0 { if len(routesToUpsert) > 0 {
err := h.db.UpsertLiveRouteBatch(routesToUpsert) if err := h.db.UpsertLiveRouteBatch(routesToUpsert); err != nil {
if err != nil { h.logger.Error("Failed to upsert route batch", "error", err, "count", len(routesToUpsert))
h.logger.Error("Failed to upsert route batch",
"error", err, "count", len(routesToUpsert))
} else { } else {
successCount += len(routesToUpsert) successCount += len(routesToUpsert)
} }
} }
if len(routesToDelete) > 0 { if len(routesToDelete) > 0 {
err := h.db.DeleteLiveRouteBatch(routesToDelete) if err := h.db.DeleteLiveRouteBatch(routesToDelete); err != nil {
if err != nil { h.logger.Error("Failed to delete route batch", "error", err, "count", len(routesToDelete))
h.logger.Error("Failed to delete route batch",
"error", err, "count", len(routesToDelete))
} else { } else {
successCount += len(routesToDelete) successCount += len(routesToDelete)
} }
@@ -289,14 +248,22 @@ func (h *PrefixHandler) writeBatch(
// Update prefix tables // Update prefix tables
if len(prefixesToUpdate) > 0 { if len(prefixesToUpdate) > 0 {
err := h.db.UpdatePrefixesBatch(prefixesToUpdate) if err := h.db.UpdatePrefixesBatch(prefixesToUpdate); err != nil {
if err != nil { h.logger.Error("Failed to update prefix batch", "error", err, "count", len(prefixesToUpdate))
h.logger.Error("Failed to update prefix batch",
"error", err, "count", len(prefixesToUpdate))
} }
} }
return successCount elapsed := time.Since(startTime)
h.logger.Debug("Flushed prefix batch",
"batch_size", batchSize,
"unique_prefixes", len(prefixesToUpdate),
"success", successCount,
"duration_ms", elapsed.Milliseconds(),
)
// Clear batch
h.batch = h.batch[:0]
h.lastFlush = time.Now()
} }
// canonicalPrefix returns prefix in the text form net/netip prints, the form // canonicalPrefix returns prefix in the text form net/netip prints, the form
@@ -314,8 +281,8 @@ func canonicalPrefix(prefix string) string {
return p.Masked().String() return p.Masked().String()
} }
// parseCIDR returns the mask length and the IP version of a prefix string // parseCIDR extracts the mask length and IP version from a prefix string
func parseCIDR(prefix string) (int, int, error) { func parseCIDR(prefix string) (maskLength int, ipVersion int, err error) {
_, ipNet, err := net.ParseCIDR(prefix) _, ipNet, err := net.ParseCIDR(prefix)
if err != nil { if err != nil {
return 0, 0, err return 0, 0, err
@@ -329,6 +296,50 @@ func parseCIDR(prefix string) (int, int, error) {
return ones, ipv4Version, nil return ones, ipv4Version, nil
} }
// processAnnouncement handles storing an announcement in the database
// nolint:unused // kept for potential future use
func (h *PrefixHandler) processAnnouncement(_ *database.Prefix, update prefixUpdate) {
// Parse CIDR to get mask length
maskLength, ipVersion, err := parseCIDR(update.prefix)
if err != nil {
h.logger.Error("Failed to parse CIDR",
"prefix", update.prefix,
"error", err,
)
return
}
// Track route update metrics
if h.metrics != nil {
if ipVersion == ipv4Version {
h.metrics.RecordIPv4Update()
} else {
h.metrics.RecordIPv6Update()
}
}
// Create live route record
liveRoute := &database.LiveRoute{
ID: uuid.New(),
Prefix: update.prefix,
MaskLength: maskLength,
IPVersion: ipVersion,
OriginASN: update.originASN,
PeerIP: update.peer,
ASPath: update.path,
NextHop: update.peer, // Using peer as next hop
LastUpdated: update.timestamp,
}
if err := h.db.UpsertLiveRoute(liveRoute); err != nil {
h.logger.Error("Failed to upsert live route",
"prefix", update.prefix,
"error", err,
)
}
}
// createLiveRoute creates a LiveRoute from a prefix update // createLiveRoute creates a LiveRoute from a prefix update
func (h *PrefixHandler) createLiveRoute(update prefixUpdate) *database.LiveRoute { func (h *PrefixHandler) createLiveRoute(update prefixUpdate) *database.LiveRoute {
// Parse CIDR to get mask length // Parse CIDR to get mask length
@@ -366,3 +377,104 @@ func (h *PrefixHandler) createLiveRoute(update prefixUpdate) *database.LiveRoute
return liveRoute return liveRoute
} }
// processAnnouncementDirect handles storing an announcement directly without prefix table
// nolint:unused // kept for potential future use
func (h *PrefixHandler) processAnnouncementDirect(update prefixUpdate) {
// Parse CIDR to get mask length
maskLength, ipVersion, err := parseCIDR(update.prefix)
if err != nil {
h.logger.Error("Failed to parse CIDR",
"prefix", update.prefix,
"error", err,
)
return
}
// Track route update metrics
if h.metrics != nil {
if ipVersion == ipv4Version {
h.metrics.RecordIPv4Update()
} else {
h.metrics.RecordIPv6Update()
}
}
// Create live route record
liveRoute := &database.LiveRoute{
ID: uuid.New(),
Prefix: update.prefix,
MaskLength: maskLength,
IPVersion: ipVersion,
OriginASN: update.originASN,
PeerIP: update.peer,
ASPath: update.path,
NextHop: update.peer, // Using peer as next hop
LastUpdated: update.timestamp,
}
if err := h.db.UpsertLiveRoute(liveRoute); err != nil {
h.logger.Error("Failed to upsert live route",
"prefix", update.prefix,
"error", err,
)
}
}
// processWithdrawalDirect handles removing a route directly without prefix table
// nolint:unused // kept for potential future use
func (h *PrefixHandler) processWithdrawalDirect(update prefixUpdate) {
// For withdrawals, we need to delete the route from live_routes
if update.originASN > 0 {
if err := h.db.DeleteLiveRoute(update.prefix, update.originASN, update.peer); err != nil {
h.logger.Error("Failed to delete live route",
"prefix", update.prefix,
"origin_asn", update.originASN,
"peer", update.peer,
"error", err,
)
}
} else {
// If no origin ASN, just delete all routes for this prefix from this peer
if err := h.db.DeleteLiveRoute(update.prefix, 0, update.peer); err != nil {
h.logger.Error("Failed to delete live route (no origin ASN)",
"prefix", update.prefix,
"peer", update.peer,
"error", err,
)
}
}
}
// processWithdrawal handles removing a route from the live routing table
// nolint:unused // kept for potential future use
func (h *PrefixHandler) processWithdrawal(_ *database.Prefix, update prefixUpdate) {
// For withdrawals, we need to delete the route from live_routes
// Since we have the origin ASN from the update, we can delete the specific route
if update.originASN > 0 {
if err := h.db.DeleteLiveRoute(update.prefix, update.originASN, update.peer); err != nil {
h.logger.Error("Failed to delete live route",
"prefix", update.prefix,
"origin_asn", update.originASN,
"peer", update.peer,
"error", err,
)
}
} else {
// If no origin ASN, just delete all routes for this prefix from this peer
if err := h.db.DeleteLiveRoute(update.prefix, 0, update.peer); err != nil {
h.logger.Error("Failed to delete live route (no origin ASN)",
"prefix", update.prefix,
"peer", update.peer,
"error", err,
)
}
}
}
// Stop gracefully stops the handler and flushes remaining batches
func (h *PrefixHandler) Stop() {
close(h.stopCh)
h.wg.Wait()
}
@@ -1,84 +0,0 @@
package routewatch
import (
"errors"
"testing"
"time"
"git.eeqj.de/sneak/routewatch/internal/config"
"git.eeqj.de/sneak/routewatch/internal/database"
"git.eeqj.de/sneak/routewatch/internal/logger"
"git.eeqj.de/sneak/routewatch/internal/ristypes"
)
const (
testPeerIP = "2001:db8:ffff::1"
testV4Prefix = "192.0.2.0/24"
)
// TestPrefixHandlerStoresPrefixesTheIPLookupFinds runs announcements and
// withdrawals through the prefix handler into a real database and looks the
// addresses up. The prefixes are written the way the feed sends them: IPv6
// announcements compressed, IPv6 withdrawals uncompressed. The withdrawal must
// remove the route the announcement stored, and the IP lookup must find the
// stored prefix.
func TestPrefixHandlerStoresPrefixesTheIPLookupFinds(t *testing.T) {
t.Parallel()
db, err := database.New(&config.Config{StateDir: t.TempDir()}, logger.New())
if err != nil {
t.Fatalf("failed to create database: %v", err)
}
defer func() { _ = db.Close() }()
// Built without NewPrefixHandler's flush timer, so the test flushes itself.
h := &PrefixHandler{db: db, logger: logger.New()}
flush := func() {
h.mu.Lock()
defer h.mu.Unlock()
h.flushBatchLocked()
}
ts := time.Date(2026, 1, 2, 3, 4, 5, 0, time.UTC)
h.HandleMessage(&ristypes.RISMessage{
Peer: testPeerIP,
Path: ristypes.ASPath{testASNA, testASNB},
ParsedTimestamp: ts,
Announcements: []ristypes.RISAnnouncement{{
NextHop: testPeerIP,
Prefixes: []string{"2001:db8:1::/48", testV4Prefix},
}},
})
flush()
for ip, want := range map[string]string{
"2001:db8:1::1": "2001:db8:1::/48",
"192.0.2.1": testV4Prefix,
} {
info, err := db.GetIPInfo(ip)
if err != nil {
t.Fatalf("GetIPInfo(%s) after announcement: %v", ip, err)
}
if info.Netblock != want || info.ASN != testASNB {
t.Errorf("GetIPInfo(%s) = %s AS%d, want %s AS%d",
ip, info.Netblock, info.ASN, want, testASNB)
}
}
h.HandleMessage(&ristypes.RISMessage{
Peer: testPeerIP,
ParsedTimestamp: ts.Add(time.Minute),
Withdrawals: []string{"2001:db8:1:0:0:0:0:0/48", testV4Prefix},
})
flush()
for _, ip := range []string{"2001:db8:1::1", "192.0.2.1"} {
info, err := db.GetIPInfo(ip)
if !errors.Is(err, database.ErrNoRoute) {
t.Errorf("GetIPInfo(%s) after withdrawal = %+v, %v; want ErrNoRoute",
ip, info, err)
}
}
}
+213
View File
@@ -0,0 +1,213 @@
package routewatch
import (
"context"
"errors"
"slices"
"testing"
"time"
"git.eeqj.de/sneak/routewatch/internal/config"
"git.eeqj.de/sneak/routewatch/internal/database"
"git.eeqj.de/sneak/routewatch/internal/logger"
"git.eeqj.de/sneak/routewatch/internal/ristypes"
)
const testPeerIP = "2001:db8:ffff::1"
// TestPrefixHandlerStoresPrefixesTheIPLookupFinds runs announcements and
// withdrawals through the prefix handler into a real database and looks the
// addresses up. The prefixes are written the way the feed sends them: IPv6
// announcements compressed, IPv6 withdrawals uncompressed. The withdrawal must
// remove the route the announcement stored, and the IP lookup must find the
// stored prefix.
func TestPrefixHandlerStoresPrefixesTheIPLookupFinds(t *testing.T) {
db, err := database.New(&config.Config{StateDir: t.TempDir()}, logger.New())
if err != nil {
t.Fatalf("failed to create database: %v", err)
}
defer func() { _ = db.Close() }()
// Built without NewPrefixHandler's flush timer, so the test flushes itself.
h := &PrefixHandler{db: db, logger: logger.New()}
flush := func() {
h.mu.Lock()
defer h.mu.Unlock()
h.flushBatchLocked()
}
ts := time.Date(2026, 1, 2, 3, 4, 5, 0, time.UTC)
h.HandleMessage(&ristypes.RISMessage{
Peer: testPeerIP,
Path: ristypes.ASPath{testASNA, testASNB},
ParsedTimestamp: ts,
Announcements: []ristypes.RISAnnouncement{{
NextHop: testPeerIP,
Prefixes: []string{"2001:db8:1::/48", "192.0.2.0/24"},
}},
})
flush()
for ip, want := range map[string]string{
"2001:db8:1::1": "2001:db8:1::/48",
"192.0.2.1": "192.0.2.0/24",
} {
info, err := db.GetIPInfo(ip)
if err != nil {
t.Fatalf("GetIPInfo(%s) after announcement: %v", ip, err)
}
if info.Netblock != want || info.ASN != testASNB {
t.Errorf("GetIPInfo(%s) = %s AS%d, want %s AS%d", ip, info.Netblock, info.ASN, want, testASNB)
}
}
h.HandleMessage(&ristypes.RISMessage{
Peer: testPeerIP,
ParsedTimestamp: ts.Add(time.Minute),
Withdrawals: []string{"2001:db8:1:0:0:0:0:0/48", "192.0.2.0/24"},
})
flush()
for _, ip := range []string{"2001:db8:1::1", "192.0.2.1"} {
if info, err := db.GetIPInfo(ip); !errors.Is(err, database.ErrNoRoute) {
t.Errorf("GetIPInfo(%s) after withdrawal = %+v, %v; want ErrNoRoute", ip, info, err)
}
}
}
// TestPrefixHandlerAppliesEachPeersLastUpdateInABatch runs updates for one
// prefix through the prefix handler into a real database. All updates carry the
// same timestamp, as updates within one second of the feed do. Within a batch,
// the last update each peer sent for the prefix decides whether that peer has a
// live route for it, and updates from different peers are all applied. The
// in-memory route count and prefix distribution must match the route table.
func TestPrefixHandlerAppliesEachPeersLastUpdateInABatch(t *testing.T) {
const (
prefix = "198.51.100.0/24"
prefixMaskLength = 24
otherPeerIP = "2001:db8:ffff::2"
)
ts := time.Date(2026, 1, 2, 3, 4, 5, 0, time.UTC)
announce := func(peer string) *ristypes.RISMessage {
return &ristypes.RISMessage{
Peer: peer,
Path: ristypes.ASPath{testASNA, testASNB},
ParsedTimestamp: ts,
Announcements: []ristypes.RISAnnouncement{{NextHop: peer, Prefixes: []string{prefix}}},
}
}
withdraw := func(peer string) *ristypes.RISMessage {
return &ristypes.RISMessage{Peer: peer, ParsedTimestamp: ts, Withdrawals: []string{prefix}}
}
announceAndWithdraw := func(peer string) *ristypes.RISMessage {
msg := announce(peer)
msg.Withdrawals = []string{prefix}
return msg
}
tests := []struct {
name string
// Each inner list is handled and then flushed as one batch.
batches [][]*ristypes.RISMessage
// The peers left with a live route for the prefix, by peer IP.
wantPeers []string
}{
{
name: "two peers announce the prefix",
batches: [][]*ristypes.RISMessage{{announce(testPeerIP), announce(otherPeerIP)}},
wantPeers: []string{testPeerIP, otherPeerIP},
},
{
name: "two peers withdraw the prefix",
batches: [][]*ristypes.RISMessage{
{announce(testPeerIP)},
{announce(otherPeerIP)},
{withdraw(testPeerIP), withdraw(otherPeerIP)},
},
},
{
name: "a peer announces and then withdraws the prefix",
batches: [][]*ristypes.RISMessage{{announce(testPeerIP), withdraw(testPeerIP)}},
},
{
name: "a peer withdraws and then re-announces the prefix",
batches: [][]*ristypes.RISMessage{
{announce(testPeerIP)},
{withdraw(testPeerIP), announce(testPeerIP)},
},
wantPeers: []string{testPeerIP},
},
{
// RFC 4271 section 4.3: the announcement counts, not the withdrawal.
name: "one message announces and withdraws the prefix",
batches: [][]*ristypes.RISMessage{
{announce(testPeerIP)},
{announceAndWithdraw(testPeerIP)},
},
wantPeers: []string{testPeerIP},
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
db, err := database.New(&config.Config{StateDir: t.TempDir()}, logger.New())
if err != nil {
t.Fatalf("failed to create database: %v", err)
}
defer func() { _ = db.Close() }()
// Built without NewPrefixHandler's flush timer, so the test flushes itself.
h := &PrefixHandler{db: db, logger: logger.New()}
for _, batch := range tt.batches {
for _, msg := range batch {
h.HandleMessage(msg)
}
h.mu.Lock()
h.flushBatchLocked()
h.mu.Unlock()
}
ctx := context.Background()
routes, err := db.GetPrefixDetailsContext(ctx, prefix)
if err != nil && !errors.Is(err, database.ErrNoRoute) {
t.Fatalf("GetPrefixDetailsContext: %v", err)
}
var gotPeers []string
for _, route := range routes {
gotPeers = append(gotPeers, route.PeerIP)
}
if !slices.Equal(gotPeers, tt.wantPeers) {
t.Errorf("peers with a live route for %s = %v, want %v", prefix, gotPeers, tt.wantPeers)
}
stats, err := db.GetStatsContext(ctx)
if err != nil {
t.Fatalf("GetStatsContext: %v", err)
}
tableRoutesV4, _, err := db.GetLiveRouteCountsContext(ctx)
if err != nil {
t.Fatalf("GetLiveRouteCountsContext: %v", err)
}
if stats.IPv4Routes != len(tt.wantPeers) || tableRoutesV4 != len(tt.wantPeers) {
t.Errorf("IPv4 live routes: %d in memory, %d in the table, want %d",
stats.IPv4Routes, tableRoutesV4, len(tt.wantPeers))
}
var wantDistribution []database.PrefixDistribution
if len(tt.wantPeers) > 0 {
wantDistribution = []database.PrefixDistribution{{MaskLength: prefixMaskLength, Count: 1}}
}
queryDistributionV4, _, err := db.GetPrefixDistributionContext(ctx)
if err != nil {
t.Fatalf("GetPrefixDistributionContext: %v", err)
}
if !slices.Equal(stats.IPv4PrefixDistribution, wantDistribution) ||
!slices.Equal(queryDistributionV4, wantDistribution) {
t.Errorf("IPv4 prefix distribution: %v in memory, %v from the table, want %v",
stats.IPv4PrefixDistribution, queryDistributionV4, wantDistribution)
}
})
}
}
File diff suppressed because it is too large Load Diff
-200
View File
@@ -1,200 +0,0 @@
package server
import (
"context"
"encoding/json"
"net/http"
"net/http/httptest"
"runtime"
"slices"
"testing"
"time"
"git.eeqj.de/sneak/routewatch/internal/config"
"git.eeqj.de/sneak/routewatch/internal/database"
"git.eeqj.de/sneak/routewatch/internal/logger"
"git.eeqj.de/sneak/routewatch/internal/metrics"
"git.eeqj.de/sneak/routewatch/internal/streamer"
"github.com/google/uuid"
)
// blockingStatsDB embeds database.Store (left nil) and overrides only
// GetStatsContext, which blocks until release is closed. The stats handlers
// call it in a goroutine; every other Store method is unused on the timeout
// path and would panic if called.
type blockingStatsDB struct {
database.Store
release chan struct{}
}
func (d blockingStatsDB) GetStatsContext(_ context.Context) (database.Stats, error) {
<-d.release
return database.Stats{}, nil
}
// TestStatsHandlersDoNotLeakOnTimeout drives each stats handler repeatedly with
// a request whose context times out before the database responds, then releases
// the blocked queries and asserts the goroutine count returns to its starting
// value. Before the fix the per-request goroutine sent on an unbuffered channel
// that nothing received once the timeout won, so it blocked forever and every
// poll leaked one goroutine.
//
//nolint:paralleltest // counts goroutines, so no other test may run alongside
func TestStatsHandlersDoNotLeakOnTimeout(t *testing.T) {
release := make(chan struct{})
db := blockingStatsDB{release: release}
s := New(db, streamer.New(logger.New(), metrics.New()), logger.New(), &config.Config{})
handlers := map[string]http.HandlerFunc{
"status.json": s.handleStatusJSON(),
"stats": s.handleStats(),
}
baseline := settledGoroutineCount()
const (
iterations = 20
requestTimeout = 50 * time.Millisecond
)
for _, handler := range handlers {
for range iterations {
ctx, cancel := context.WithTimeout(context.Background(), requestTimeout)
req := httptest.NewRequestWithContext(ctx, http.MethodGet, "/", nil)
handler(httptest.NewRecorder(), req)
cancel()
}
}
// Let the blocked queries finish; with buffered channels each goroutine's
// send now succeeds and the goroutine exits.
close(release)
if !waitForGoroutines(baseline) {
t.Fatalf("goroutines did not return to baseline %d, got %d",
baseline, runtime.NumGoroutine())
}
}
// TestStatsHandlersAnswerFromMemory checks that both stats handlers answer 200
// with the live route counts and the prefix distribution while the database is
// closed, so that any query would fail: the request path reads them from
// memory. The prefix distribution query it used to run read every live route
// and, on a large database, took the whole 4-second deadline, so
// /api/v1/stats answered 500 (https://git.eeqj.de/sneak/routewatch/issues/30).
// The oldest and newest route times still come from one-row lookups at the ends
// of an index; with the database closed they are left out of the answer.
func TestStatsHandlersAnswerFromMemory(t *testing.T) {
t.Parallel()
db, err := database.New(&config.Config{StateDir: t.TempDir()}, logger.New())
if err != nil {
t.Fatalf("database.New: %v", err)
}
ts := time.Date(2026, 1, 2, 3, 4, 5, 0, time.UTC)
err = db.UpsertLiveRouteBatch([]*database.LiveRoute{
{
ID: uuid.New(), Prefix: "198.51.100.0/24", MaskLength: 24, IPVersion: 4,
OriginASN: 64500, PeerIP: "192.0.2.1", ASPath: []int{64500},
NextHop: "192.0.2.1", LastUpdated: ts,
},
{
ID: uuid.New(), Prefix: "2001:db8::/32", MaskLength: 32, IPVersion: 6,
OriginASN: 64501, PeerIP: "2001:db8::1", ASPath: []int{64501},
NextHop: "2001:db8::1", LastUpdated: ts,
},
})
if err != nil {
t.Fatalf("UpsertLiveRouteBatch: %v", err)
}
err = db.Close()
if err != nil {
t.Fatalf("Close: %v", err)
}
s := New(db, streamer.New(logger.New(), metrics.New()), logger.New(), &config.Config{})
handlers := map[string]http.HandlerFunc{
"status.json": s.handleStatusJSON(),
"stats": s.handleStats(),
}
for name, handler := range handlers {
rec := httptest.NewRecorder()
handler(rec, httptest.NewRequestWithContext(t.Context(), http.MethodGet, "/", nil))
if rec.Code != http.StatusOK {
t.Errorf("%s: status %d, want %d; body %s",
name, rec.Code, http.StatusOK, rec.Body)
continue
}
//nolint:tagliatelle // field names are the existing JSON API
var body struct {
Data struct {
IPv4Routes int `json:"ipv4_routes"`
IPv6Routes int `json:"ipv6_routes"`
IPv4Distribution []database.PrefixDistribution `json:"ipv4_prefix_distribution"`
IPv6Distribution []database.PrefixDistribution `json:"ipv6_prefix_distribution"`
} `json:"data"`
}
err := json.Unmarshal(rec.Body.Bytes(), &body)
if err != nil {
t.Fatalf("%s: decoding the answer: %v", name, err)
}
if body.Data.IPv4Routes != 1 || body.Data.IPv6Routes != 1 {
t.Errorf("%s: routes = (v4 %d, v6 %d), want (1, 1)",
name, body.Data.IPv4Routes, body.Data.IPv6Routes)
}
wantV4 := []database.PrefixDistribution{{MaskLength: 24, Count: 1}}
if !slices.Equal(body.Data.IPv4Distribution, wantV4) {
t.Errorf("%s: IPv4 distribution = %v, want %v",
name, body.Data.IPv4Distribution, wantV4)
}
wantV6 := []database.PrefixDistribution{{MaskLength: 32, Count: 1}}
if !slices.Equal(body.Data.IPv6Distribution, wantV6) {
t.Errorf("%s: IPv6 distribution = %v, want %v",
name, body.Data.IPv6Distribution, wantV6)
}
}
}
// settledGoroutineCount lets transient goroutines finish, then reports the
// current count.
func settledGoroutineCount() int {
prev := runtime.NumGoroutine()
for range 20 {
time.Sleep(10 * time.Millisecond)
cur := runtime.NumGoroutine()
if cur == prev {
return cur
}
prev = cur
}
return prev
}
// waitForGoroutines waits until the goroutine count drops to target or below.
func waitForGoroutines(target int) bool {
for range 100 {
if runtime.NumGoroutine() <= target {
return true
}
time.Sleep(10 * time.Millisecond)
}
return false
}
+402
View File
@@ -0,0 +1,402 @@
package server
import (
"context"
"encoding/json"
"net/http"
"net/http/httptest"
"runtime"
"slices"
"testing"
"time"
"git.eeqj.de/sneak/routewatch/internal/config"
"git.eeqj.de/sneak/routewatch/internal/database"
"git.eeqj.de/sneak/routewatch/internal/logger"
"git.eeqj.de/sneak/routewatch/internal/metrics"
"git.eeqj.de/sneak/routewatch/internal/streamer"
"github.com/google/uuid"
)
// blockingStatsDB embeds database.Store (left nil) and overrides only
// GetStatsContext, which blocks until release is closed. The stats handlers
// call it in a goroutine; every other Store method is unused on the timeout
// path and would panic if called.
type blockingStatsDB struct {
database.Store
release chan struct{}
}
func (d blockingStatsDB) GetStatsContext(_ context.Context) (database.Stats, error) {
<-d.release
return database.Stats{}, nil
}
// TestStatsHandlersDoNotLeakOnTimeout drives each stats handler repeatedly with
// a request whose context times out before the database responds, then releases
// the blocked queries and asserts the goroutine count returns to its starting
// value. Before the fix the per-request goroutine sent on an unbuffered channel
// that nothing received once the timeout won, so it blocked forever and every
// poll leaked one goroutine.
func TestStatsHandlersDoNotLeakOnTimeout(t *testing.T) {
release := make(chan struct{})
db := blockingStatsDB{release: release}
s := New(db, streamer.New(logger.New(), metrics.New()), logger.New(), &config.Config{})
handlers := map[string]http.HandlerFunc{
"status.json": s.handleStatusJSON(),
"stats": s.handleStats(),
}
baseline := settledGoroutineCount()
const (
iterations = 20
requestTimeout = 50 * time.Millisecond
)
for _, handler := range handlers {
for range iterations {
ctx, cancel := context.WithTimeout(context.Background(), requestTimeout)
req := httptest.NewRequest(http.MethodGet, "/", nil).WithContext(ctx)
handler(httptest.NewRecorder(), req)
cancel()
}
}
// Let the blocked queries finish; with buffered channels each goroutine's
// send now succeeds and the goroutine exits.
close(release)
if !waitForGoroutines(baseline) {
t.Fatalf("goroutines did not return to baseline %d, got %d",
baseline, runtime.NumGoroutine())
}
}
// TestStatsHandlersAnswerFromMemory checks that both stats handlers answer 200
// with the live route counts and the prefix distribution while the database is
// closed, so that any query would fail: the request path reads them from
// memory. The prefix distribution query it used to run read every live route
// and, on a large database, took the whole 4-second deadline, so
// /api/v1/stats answered 500 (https://git.eeqj.de/sneak/routewatch/issues/30).
// The oldest and newest route times still come from one-row lookups at the ends
// of an index; with the database closed they are left out of the answer.
func TestStatsHandlersAnswerFromMemory(t *testing.T) {
db, err := database.New(&config.Config{StateDir: t.TempDir()}, logger.New())
if err != nil {
t.Fatalf("database.New: %v", err)
}
ts := time.Date(2026, 1, 2, 3, 4, 5, 0, time.UTC)
if err := db.UpsertLiveRouteBatch([]*database.LiveRoute{
{
ID: uuid.New(), Prefix: "198.51.100.0/24", MaskLength: 24, IPVersion: 4,
OriginASN: 64500, PeerIP: "192.0.2.1", ASPath: []int{64500}, NextHop: "192.0.2.1",
LastUpdated: ts,
},
{
ID: uuid.New(), Prefix: "2001:db8::/32", MaskLength: 32, IPVersion: 6,
OriginASN: 64501, PeerIP: "2001:db8::1", ASPath: []int{64501}, NextHop: "2001:db8::1",
LastUpdated: ts,
},
}); err != nil {
t.Fatalf("UpsertLiveRouteBatch: %v", err)
}
if err := db.Close(); err != nil {
t.Fatalf("Close: %v", err)
}
s := New(db, streamer.New(logger.New(), metrics.New()), logger.New(), &config.Config{})
handlers := map[string]http.HandlerFunc{
"status.json": s.handleStatusJSON(),
"stats": s.handleStats(),
}
for name, handler := range handlers {
rec := httptest.NewRecorder()
handler(rec, httptest.NewRequest(http.MethodGet, "/", nil))
if rec.Code != http.StatusOK {
t.Errorf("%s: status %d, want %d; body %s", name, rec.Code, http.StatusOK, rec.Body)
continue
}
var body struct {
Data struct {
IPv4Routes int `json:"ipv4_routes"`
IPv6Routes int `json:"ipv6_routes"`
IPv4PrefixDistribution []database.PrefixDistribution `json:"ipv4_prefix_distribution"`
IPv6PrefixDistribution []database.PrefixDistribution `json:"ipv6_prefix_distribution"`
} `json:"data"`
}
if err := json.Unmarshal(rec.Body.Bytes(), &body); err != nil {
t.Fatalf("%s: decoding the answer: %v", name, err)
}
if body.Data.IPv4Routes != 1 || body.Data.IPv6Routes != 1 {
t.Errorf("%s: routes = (v4 %d, v6 %d), want (1, 1)", name, body.Data.IPv4Routes, body.Data.IPv6Routes)
}
wantV4 := []database.PrefixDistribution{{MaskLength: 24, Count: 1}}
if !slices.Equal(body.Data.IPv4PrefixDistribution, wantV4) {
t.Errorf("%s: IPv4 distribution = %v, want %v", name, body.Data.IPv4PrefixDistribution, wantV4)
}
wantV6 := []database.PrefixDistribution{{MaskLength: 32, Count: 1}}
if !slices.Equal(body.Data.IPv6PrefixDistribution, wantV6) {
t.Errorf("%s: IPv6 distribution = %v, want %v", name, body.Data.IPv6PrefixDistribution, wantV6)
}
}
}
// TestPrefixDetailHandlersParsePrefix checks that both prefix pages find a
// stored route whatever valid text form the URL writes its prefix in, and
// answer 400 for a prefix that does not parse. The tables store the form
// net/netip prints. Before the fix the URL text was looked up as typed, so
// "2001:DB8:0:0::/32" was not found although "2001:db8::/32" was stored, and
// "foo/24" answered 500 (https://git.eeqj.de/sneak/routewatch/issues/71).
func TestPrefixDetailHandlersParsePrefix(t *testing.T) {
db, err := database.New(&config.Config{StateDir: t.TempDir()}, logger.New())
if err != nil {
t.Fatalf("database.New: %v", err)
}
t.Cleanup(func() { _ = db.Close() })
ts := time.Date(2026, 1, 2, 3, 4, 5, 0, time.UTC)
if err := db.UpsertLiveRouteBatch([]*database.LiveRoute{
{
ID: uuid.New(), Prefix: "192.0.2.0/24", MaskLength: 24, IPVersion: 4,
OriginASN: 64500, PeerIP: "198.51.100.1", ASPath: []int{64500}, NextHop: "198.51.100.1",
LastUpdated: ts,
},
{
ID: uuid.New(), Prefix: "2001:db8::/32", MaskLength: 32, IPVersion: 6,
OriginASN: 64501, PeerIP: "2001:db8::1", ASPath: []int{64501}, NextHop: "2001:db8::1",
LastUpdated: ts,
},
}); err != nil {
t.Fatalf("UpsertLiveRouteBatch: %v", err)
}
s := New(db, streamer.New(logger.New(), metrics.New()), logger.New(), &config.Config{})
tests := []struct {
path string
want int
}{
{"/api/v1/prefix/foo/24", http.StatusBadRequest},
{"/prefix/foo/24", http.StatusBadRequest},
{"/api/v1/prefix/2001:DB8:0:0::/32", http.StatusOK},
{"/prefix/2001:DB8:0:0::/32", http.StatusOK},
{"/api/v1/prefix/192.0.2.1/24", http.StatusOK},
{"/prefix/192.0.2.1/24", http.StatusOK},
}
for _, tt := range tests {
rec := httptest.NewRecorder()
s.router.ServeHTTP(rec, httptest.NewRequest(http.MethodGet, tt.path, nil))
if rec.Code != tt.want {
t.Errorf("%s: status %d, want %d; body %s", tt.path, rec.Code, tt.want, rec.Body)
}
}
}
// shortRequestDeadline is the deadline the tests below give a request. The
// server's 30-second limit and the stats handlers' 4-second limit inherit it,
// so the tests do not wait for either.
const shortRequestDeadline = 50 * time.Millisecond
// TestRequestPastTimeLimitAnswers504 sends a request through the server's
// router to a page and to a JSON route whose handlers wait for the time limit
// to pass and return without answering. The page answers 504; the JSON route
// answers 504 with the JSON error body. Before the fix both answered 408,
// written by the time limit middleware from its own goroutine while the
// handler could still be writing.
func TestRequestPastTimeLimitAnswers504(t *testing.T) {
s := New(nil, streamer.New(logger.New(), metrics.New()), logger.New(), &config.Config{})
waitForTimeLimit := func(_ http.ResponseWriter, r *http.Request) {
<-r.Context().Done()
}
// Registered the way setupRoutes registers pages and JSON routes.
s.router.Get("/slow-page", waitForTimeLimit)
s.router.Get("/slow.json", JSONValidationMiddleware(http.HandlerFunc(waitForTimeLimit)).ServeHTTP)
tests := []struct {
path string
wantJSONBody bool
}{
{path: "/slow-page"},
{path: "/slow.json", wantJSONBody: true},
}
for _, tt := range tests {
ctx, cancel := context.WithTimeout(context.Background(), shortRequestDeadline)
rec := httptest.NewRecorder()
s.router.ServeHTTP(rec, httptest.NewRequest(http.MethodGet, tt.path, nil).WithContext(ctx))
cancel()
if rec.Code != http.StatusGatewayTimeout {
t.Errorf("%s: status %d, want %d; body %s", tt.path, rec.Code, http.StatusGatewayTimeout, rec.Body)
}
if tt.wantJSONBody {
checkJSONErrorCode(t, tt.path, rec, http.StatusGatewayTimeout)
}
}
}
// TestStatsHandlersAnswer504WhenDatabaseReadRunsOut calls each stats handler
// with a store that does not answer before the handler's 4-second limit runs
// out, and checks that the handler itself answers 504 with the JSON error
// body. Before the fix status.json answered 408, and /api/v1/stats wrote
// nothing, which the JSON check on its route turned into 500.
func TestStatsHandlersAnswer504WhenDatabaseReadRunsOut(t *testing.T) {
release := make(chan struct{})
defer close(release)
db := blockingStatsDB{release: release}
s := New(db, streamer.New(logger.New(), metrics.New()), logger.New(), &config.Config{})
handlers := map[string]http.HandlerFunc{
"status.json": s.handleStatusJSON(),
"stats": s.handleStats(),
}
for name, handler := range handlers {
ctx, cancel := context.WithTimeout(context.Background(), shortRequestDeadline)
rec := httptest.NewRecorder()
handler(rec, httptest.NewRequest(http.MethodGet, "/", nil).WithContext(ctx))
cancel()
if rec.Code != http.StatusGatewayTimeout {
t.Errorf("%s: status %d, want %d; body %s", name, rec.Code, http.StatusGatewayTimeout, rec.Body)
}
checkJSONErrorCode(t, name, rec, http.StatusGatewayTimeout)
}
}
// slowDB embeds database.Store (left nil) and overrides the reads the routes
// in TestRoutesAnswer504WhenDatabaseReadIsCutOff make. Each read waits until
// its context ends and returns the context's error, as a slow database read
// cut off by a time limit does.
type slowDB struct {
database.Store
}
func (slowDB) GetStatsContext(ctx context.Context) (database.Stats, error) {
<-ctx.Done()
return database.Stats{}, ctx.Err()
}
func (slowDB) GetIPInfoContext(ctx context.Context, _ string) (*database.IPInfo, error) {
<-ctx.Done()
return nil, ctx.Err()
}
func (slowDB) GetASDetailsContext(ctx context.Context, _ int) (*database.ASN, []database.LiveRoute, error) {
<-ctx.Done()
return nil, nil, ctx.Err()
}
func (slowDB) GetPrefixDetailsContext(ctx context.Context, _ string) ([]database.LiveRoute, error) {
<-ctx.Done()
return nil, ctx.Err()
}
func (slowDB) GetRandomPrefixesByLengthContext(ctx context.Context, _, _, _ int) ([]database.LiveRoute, error) {
<-ctx.Done()
return nil, ctx.Err()
}
// TestRoutesAnswer504WhenDatabaseReadIsCutOff sends a request through the
// server's router for each route that reads the database, with a store whose
// reads fail when the request's time limit passes. Each route answers 504, the
// JSON routes with the JSON error body. Before the fix they answered 408 or,
// when the handler wrote first, its own error: 500, or 404 on the IP lookups.
// The health check is left out: it answers 503 when its own 2-second ping runs
// out.
func TestRoutesAnswer504WhenDatabaseReadIsCutOff(t *testing.T) {
s := New(slowDB{}, streamer.New(logger.New(), metrics.New()), logger.New(), &config.Config{})
tests := []struct {
path string
wantJSONBody bool
}{
{path: "/as/64500"},
{path: "/prefix/192.0.2.0/24"},
{path: "/prefixlength/24"},
{path: "/prefixlength6/48"},
{path: "/status.json", wantJSONBody: true},
{path: "/ip/192.0.2.1", wantJSONBody: true},
{path: "/api/v1/stats", wantJSONBody: true},
{path: "/api/v1/ip/192.0.2.1", wantJSONBody: true},
{path: "/api/v1/as/64500", wantJSONBody: true},
{path: "/api/v1/prefix/192.0.2.0/24", wantJSONBody: true},
}
for _, tt := range tests {
ctx, cancel := context.WithTimeout(context.Background(), shortRequestDeadline)
rec := httptest.NewRecorder()
s.router.ServeHTTP(rec, httptest.NewRequest(http.MethodGet, tt.path, nil).WithContext(ctx))
cancel()
if rec.Code != http.StatusGatewayTimeout {
t.Errorf("%s: status %d, want %d; body %s", tt.path, rec.Code, http.StatusGatewayTimeout, rec.Body)
}
if tt.wantJSONBody {
checkJSONErrorCode(t, tt.path, rec, http.StatusGatewayTimeout)
}
}
}
// checkJSONErrorCode checks that rec holds the JSON error body with the given
// code.
func checkJSONErrorCode(t *testing.T, name string, rec *httptest.ResponseRecorder, want int) {
t.Helper()
var body struct {
Status string `json:"status"`
Error struct {
Code int `json:"code"`
} `json:"error"`
}
if err := json.Unmarshal(rec.Body.Bytes(), &body); err != nil {
t.Errorf("%s: decoding the answer %q: %v", name, rec.Body, err)
return
}
if body.Status != "error" || body.Error.Code != want {
t.Errorf("%s: answer %s, want status \"error\" and error code %d", name, rec.Body, want)
}
}
// settledGoroutineCount lets transient goroutines finish, then reports the
// current count.
func settledGoroutineCount() int {
prev := runtime.NumGoroutine()
for range 20 {
time.Sleep(10 * time.Millisecond)
cur := runtime.NumGoroutine()
if cur == prev {
return cur
}
prev = cur
}
return prev
}
// waitForGoroutines waits until the goroutine count drops to target or below.
func waitForGoroutines(target int) bool {
for range 100 {
if runtime.NumGoroutine() <= target {
return true
}
time.Sleep(10 * time.Millisecond)
}
return false
}
+30 -159
View File
@@ -4,6 +4,7 @@ import (
"bytes" "bytes"
"context" "context"
"encoding/json" "encoding/json"
"errors"
"fmt" "fmt"
"log/slog" "log/slog"
"net/http" "net/http"
@@ -14,7 +15,6 @@ import (
// responseWriter wraps http.ResponseWriter to capture the response // responseWriter wraps http.ResponseWriter to capture the response
type responseWriter struct { type responseWriter struct {
http.ResponseWriter http.ResponseWriter
body *bytes.Buffer body *bytes.Buffer
statusCode int statusCode int
written bool written bool
@@ -87,10 +87,8 @@ func JSONResponseMiddleware(next http.Handler) http.Handler {
} }
// Parse the original response // Parse the original response
var originalResponse map[string]any var originalResponse map[string]interface{}
if err := json.Unmarshal(rw.body.Bytes(), &originalResponse); err != nil {
err := json.Unmarshal(rw.body.Bytes(), &originalResponse)
if err != nil {
// If we can't parse it, just write original // If we can't parse it, just write original
w.WriteHeader(rw.statusCode) w.WriteHeader(rw.statusCode)
_, _ = w.Write(rw.body.Bytes()) _, _ = w.Write(rw.body.Bytes())
@@ -99,7 +97,7 @@ func JSONResponseMiddleware(next http.Handler) http.Handler {
} }
// Add @meta field // Add @meta field
originalResponse["@meta"] = map[string]any{ originalResponse["@meta"] = map[string]interface{}{
"time_zone": "UTC", "time_zone": "UTC",
"api_version": 1, "api_version": 1,
"execution_time": fmt.Sprintf("%d ms", execTime.Milliseconds()), "execution_time": fmt.Sprintf("%d ms", execTime.Milliseconds()),
@@ -112,148 +110,24 @@ func JSONResponseMiddleware(next http.Handler) http.Handler {
}) })
} }
// timeoutWriter wraps ResponseWriter to prevent concurrent writes after timeout // requestTimeoutMessage is the error message sent with 504 Gateway Timeout.
type timeoutWriter struct { const requestTimeoutMessage = "Request timeout"
http.ResponseWriter
mu sync.Mutex // deadlinePassed reports whether ctx ended because its deadline passed. When
written bool // a handler's database read or other lookup fails, it uses this to tell a
header http.Header // cached header to prevent concurrent access // request cut off by its time limit, which answers 504, from other failures.
} func deadlinePassed(ctx context.Context) bool {
return errors.Is(ctx.Err(), context.DeadlineExceeded)
func (tw *timeoutWriter) Write(b []byte) (int, error) {
tw.mu.Lock()
defer tw.mu.Unlock()
if tw.written {
return 0, nil // Discard writes after timeout
}
return tw.ResponseWriter.Write(b)
}
func (tw *timeoutWriter) WriteHeader(statusCode int) {
tw.mu.Lock()
defer tw.mu.Unlock()
if tw.written {
return // Discard writes after timeout
}
tw.ResponseWriter.WriteHeader(statusCode)
}
func (tw *timeoutWriter) Header() http.Header {
tw.mu.Lock()
defer tw.mu.Unlock()
if tw.written {
// Return a copy to prevent modifications after timeout
if tw.header == nil {
tw.header = make(http.Header)
}
return tw.header
}
return tw.ResponseWriter.Header()
}
func (tw *timeoutWriter) markWritten() {
tw.mu.Lock()
defer tw.mu.Unlock()
tw.written = true
}
// TimeoutMiddleware creates an HTTP middleware that enforces a request timeout.
// If the handler does not complete within the specified duration, the middleware
// returns a JSON error response with HTTP status 408 (Request Timeout).
//
// The timeout parameter specifies the maximum duration allowed for request processing.
// The returned middleware handles panics from the wrapped handler by re-panicking
// after cleanup, and prevents concurrent writes to the response after timeout occurs.
func TimeoutMiddleware(
timeout time.Duration,
logger *slog.Logger,
) func(http.Handler) http.Handler {
return func(next http.Handler) http.Handler {
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
startTime := time.Now()
ctx, cancel := context.WithTimeout(r.Context(), timeout)
defer cancel()
tw := &timeoutWriter{
ResponseWriter: w,
header: make(http.Header),
}
done := make(chan struct{})
panicChan := make(chan any, 1)
go func() {
defer func() {
if p := recover(); p != nil {
panicChan <- p
}
}()
next.ServeHTTP(tw, r.WithContext(ctx))
close(done)
}()
select {
case p := <-panicChan:
panic(p)
case <-done:
return
case <-ctx.Done():
tw.markWritten() // Prevent the handler from writing after timeout
execTime := time.Since(startTime)
// Log the timeout as a warning
logger.Warn("Request timeout",
"method", r.Method,
"path", r.URL.Path,
"duration_ms", execTime.Milliseconds(),
"remote_addr", r.RemoteAddr,
)
// Write directly to the underlying writer since we've marked tw as written
// This is safe because markWritten() prevents the handler from writing
tw.mu.Lock()
w.Header().Set("Content-Type", "application/json")
w.WriteHeader(http.StatusRequestTimeout)
tw.mu.Unlock()
response := map[string]any{
jsonKeyStatus: jsonStatusError,
jsonKeyError: map[string]any{
jsonKeyMsg: "Request timeout",
jsonKeyCode: http.StatusRequestTimeout,
},
"@meta": map[string]any{
"time_zone": "UTC",
"api_version": 1,
"execution_time": fmt.Sprintf("%d ms", execTime.Milliseconds()),
},
}
_ = json.NewEncoder(w).Encode(response)
}
})
}
} }
// JSONValidationMiddleware is an HTTP middleware that validates JSON API responses. // JSONValidationMiddleware is an HTTP middleware that validates JSON API responses.
// It ensures that responses with Content-Type "application/json" contain valid JSON. // It ensures that responses with Content-Type "application/json" contain valid JSON.
// //
// If a response is not valid JSON or is empty when JSON is expected, the middleware // If a response is not valid JSON or is empty when JSON is expected, the middleware
// returns a properly formatted JSON error response. For timeout errors (status 408), // returns a properly formatted JSON error response. If the request's time limit
// the error message will be "Request timeout". For other errors, it returns // has passed, it answers 504 with the error message "Request timeout". For other
// "Internal server error" with status 500 if the original status was 200. // errors, it returns "Internal server error" with status 500 if the original
// status was 200.
// //
// Non-JSON responses are passed through unchanged. // Non-JSON responses are passed through unchanged.
func JSONValidationMiddleware(next http.Handler) http.Handler { func JSONValidationMiddleware(next http.Handler) http.Handler {
@@ -282,10 +156,8 @@ func JSONValidationMiddleware(next http.Handler) http.Handler {
// For JSON responses, validate the JSON // For JSON responses, validate the JSON
if rw.body.Len() > 0 { if rw.body.Len() > 0 {
var testParse any var testParse interface{}
if err := json.Unmarshal(rw.body.Bytes(), &testParse); err == nil {
err := json.Unmarshal(rw.body.Bytes(), &testParse)
if err == nil {
// Valid JSON, write it out // Valid JSON, write it out
w.Header().Set("Content-Type", "application/json") w.Header().Set("Content-Type", "application/json")
w.WriteHeader(rw.statusCode) w.WriteHeader(rw.statusCode)
@@ -304,19 +176,21 @@ func JSONValidationMiddleware(next http.Handler) http.Handler {
if statusCode == http.StatusOK { if statusCode == http.StatusOK {
statusCode = http.StatusInternalServerError statusCode = http.StatusInternalServerError
} }
if deadlinePassed(r.Context()) {
statusCode = http.StatusGatewayTimeout
}
w.WriteHeader(statusCode) w.WriteHeader(statusCode)
errorMsg := "Internal server error" errorMsg := "Internal server error"
if statusCode == http.StatusRequestTimeout { if statusCode == http.StatusGatewayTimeout {
errorMsg = "Request timeout" errorMsg = requestTimeoutMessage
} }
response := map[string]any{ response := map[string]interface{}{
jsonKeyStatus: jsonStatusError, "status": "error",
jsonKeyError: map[string]any{ "error": map[string]interface{}{
jsonKeyMsg: errorMsg, "msg": errorMsg,
jsonKeyCode: statusCode, "code": statusCode,
}, },
} }
_ = json.NewEncoder(w).Encode(response) _ = json.NewEncoder(w).Encode(response)
@@ -326,7 +200,6 @@ func JSONValidationMiddleware(next http.Handler) http.Handler {
// statusWriter wraps http.ResponseWriter to capture the status code // statusWriter wraps http.ResponseWriter to capture the status code
type statusWriter struct { type statusWriter struct {
http.ResponseWriter http.ResponseWriter
statusCode int statusCode int
written bool written bool
} }
@@ -336,7 +209,6 @@ func (sw *statusWriter) WriteHeader(statusCode int) {
sw.statusCode = statusCode sw.statusCode = statusCode
sw.written = true sw.written = true
} }
sw.ResponseWriter.WriteHeader(statusCode) sw.ResponseWriter.WriteHeader(statusCode)
} }
@@ -376,12 +248,11 @@ func RequestLoggerMiddleware(logger *slog.Logger) func(http.Handler) http.Handle
// Slow query threshold (1 second) // Slow query threshold (1 second)
const slowQueryThreshold = 1 * time.Second const slowQueryThreshold = 1 * time.Second
switch { if sw.statusCode >= http.StatusInternalServerError {
case sw.statusCode >= http.StatusInternalServerError:
logLevel = slog.LevelError logLevel = slog.LevelError
case sw.statusCode >= http.StatusBadRequest: } else if sw.statusCode >= http.StatusBadRequest {
logLevel = slog.LevelWarn logLevel = slog.LevelWarn
case duration >= slowQueryThreshold: } else if duration >= slowQueryThreshold {
logLevel = slog.LevelWarn logLevel = slog.LevelWarn
} }
+6 -8
View File
@@ -16,20 +16,18 @@ func (s *Server) setupRoutes() {
r.Use(middleware.RealIP) r.Use(middleware.RealIP)
r.Use(RequestLoggerMiddleware(s.logger.Logger)) // Structured request logging r.Use(RequestLoggerMiddleware(s.logger.Logger)) // Structured request logging
r.Use(middleware.Recoverer) r.Use(middleware.Recoverer)
const requestTimeout = 30 * time.Second // Increased from 8s for slow queries
r.Use(TimeoutMiddleware(requestTimeout, s.logger.Logger))
r.Use(JSONResponseMiddleware) r.Use(JSONResponseMiddleware)
// Timeout must come after JSONResponseMiddleware, which sends its answer
// when the handler returns and would send a 200 ahead of Timeout's 504 for
// a handler that wrote nothing.
const requestTimeout = 30 * time.Second // Increased from 8s for slow queries
r.Use(middleware.Timeout(requestTimeout))
// Routes // Routes
r.Get("/", s.handleIndex()) r.Get("/", s.handleIndex())
r.Get("/status", s.handleStatusHTML()) r.Get("/status", s.handleStatusHTML())
r.Get("/status.json", JSONValidationMiddleware(s.handleStatusJSON()).ServeHTTP) r.Get("/status.json", JSONValidationMiddleware(s.handleStatusJSON()).ServeHTTP)
r.Get( r.Get("/.well-known/healthcheck.json", JSONValidationMiddleware(s.handleHealthCheck()).ServeHTTP)
"/.well-known/healthcheck.json",
JSONValidationMiddleware(s.handleHealthCheck()).ServeHTTP,
)
// AS and prefix detail pages // AS and prefix detail pages
r.Get("/as/{asn}", s.handleASDetail()) r.Get("/as/{asn}", s.handleASDetail())
+2 -10
View File
@@ -3,7 +3,6 @@ package server
import ( import (
"context" "context"
"errors"
"net/http" "net/http"
"strconv" "strconv"
"time" "time"
@@ -41,12 +40,7 @@ type Server struct {
} }
// New creates a new HTTP server // New creates a new HTTP server
func New( func New(db database.Store, streamer *streamer.Streamer, logger *logger.Logger, cfg *config.Config) *Server {
db database.Store,
streamer *streamer.Streamer,
logger *logger.Logger,
cfg *config.Config,
) *Server {
s := &Server{ s := &Server{
db: db, db: db,
streamer: streamer, streamer: streamer,
@@ -82,9 +76,7 @@ func (s *Server) Start() error {
// Start in goroutine but log when actually listening // Start in goroutine but log when actually listening
go func() { go func() {
s.logger.Info("HTTP server listening", "addr", s.srv.Addr) s.logger.Info("HTTP server listening", "addr", s.srv.Addr)
if err := s.srv.ListenAndServe(); err != nil && err != http.ErrServerClosed {
err := s.srv.ListenAndServe()
if err != nil && !errors.Is(err, http.ErrServerClosed) {
s.logger.Error("HTTP server error", "error", err) s.logger.Error("HTTP server error", "error", err)
} }
}() }()
+202 -271
View File
@@ -1,6 +1,5 @@
// Package streamer implements an HTTP client that connects to the RIPE RIS Live // Package streamer implements an HTTP client that connects to the RIPE RIS Live streaming API,
// streaming API, parses BGP UPDATE messages from the JSON stream, and // parses BGP UPDATE messages from the JSON stream, and dispatches them to registered handlers.
// dispatches them to registered handlers.
package streamer package streamer
import ( import (
@@ -8,7 +7,6 @@ import (
"compress/gzip" "compress/gzip"
"context" "context"
"encoding/json" "encoding/json"
"errors"
"fmt" "fmt"
"io" "io"
"math" "math"
@@ -26,20 +24,20 @@ import (
// countingReader wraps an io.Reader and counts bytes read // countingReader wraps an io.Reader and counts bytes read
type countingReader struct { type countingReader struct {
reader io.Reader reader io.Reader
count atomic.Int64 count int64
} }
// Read implements io.Reader and counts bytes // Read implements io.Reader and counts bytes
func (c *countingReader) Read(p []byte) (int, error) { func (c *countingReader) Read(p []byte) (int, error) {
n, err := c.reader.Read(p) n, err := c.reader.Read(p)
c.count.Add(int64(n)) atomic.AddInt64(&c.count, int64(n))
return n, err return n, err
} }
// Count returns the total bytes read // Count returns the total bytes read
func (c *countingReader) Count() int64 { func (c *countingReader) Count() int64 {
return c.count.Load() return atomic.LoadInt64(&c.count)
} }
// Configuration constants for the RIS Live streamer. // Configuration constants for the RIS Live streamer.
@@ -60,17 +58,11 @@ const (
backpressureSlope = 2.0 // Slope for linear drop probability increase backpressureSlope = 2.0 // Slope for linear drop probability increase
) )
var (
errAlreadyRunning = errors.New("streamer already running")
errUnexpectedStatus = errors.New("unexpected status code")
)
// MessageHandler defines the interface for processing RIS messages. // MessageHandler defines the interface for processing RIS messages.
// Implementations must specify which message types they want to receive, // Implementations must specify which message types they want to receive,
// how to process messages, and their desired queue capacity. // how to process messages, and their desired queue capacity.
type MessageHandler interface { type MessageHandler interface {
// WantsMessage returns true if this handler wants to process messages of // WantsMessage returns true if this handler wants to process messages of the given type.
// the given type.
WantsMessage(messageType string) bool WantsMessage(messageType string) bool
// HandleMessage processes a RIS message. This method is called from a dedicated // HandleMessage processes a RIS message. This method is called from a dedicated
@@ -121,7 +113,7 @@ type Streamer struct {
cancel context.CancelFunc cancel context.CancelFunc
running bool running bool
metrics *metrics.Tracker metrics *metrics.Tracker
totalDropped atomic.Uint64 // Total dropped messages across all handlers totalDropped uint64 // Total dropped messages across all handlers
random *rand.Rand // Random number generator for backpressure drops random *rand.Rand // Random number generator for backpressure drops
bgpPeers map[string]bool // Track active BGP peers by peer IP bgpPeers map[string]bool // Track active BGP peers by peer IP
bgpPeersMu sync.RWMutex // Protects bgpPeers map bgpPeersMu sync.RWMutex // Protects bgpPeers map
@@ -129,8 +121,7 @@ type Streamer struct {
// New creates a new Streamer instance configured to connect to the RIS Live API. // New creates a new Streamer instance configured to connect to the RIS Live API.
// The logger is used for structured logging of connection events and errors. // The logger is used for structured logging of connection events and errors.
// The metrics tracker is used to record message counts, bytes received, and // The metrics tracker is used to record message counts, bytes received, and connection status.
// connection status.
func New(logger *logger.Logger, metrics *metrics.Tracker) *Streamer { func New(logger *logger.Logger, metrics *metrics.Tracker) *Streamer {
return &Streamer{ return &Streamer{
logger: logger, logger: logger,
@@ -162,8 +153,7 @@ func (s *Streamer) RegisterHandler(handler MessageHandler) {
handler: handler, handler: handler,
queue: make(chan *ristypes.RISMessage, handler.QueueCapacity()), queue: make(chan *ristypes.RISMessage, handler.QueueCapacity()),
metrics: handlerMetrics{ metrics: handlerMetrics{
// Initialize to max so first value sets the floor minTime: time.Duration(math.MaxInt64), // Initialize to max so first value sets the floor
minTime: time.Duration(math.MaxInt64),
}, },
} }
@@ -181,24 +171,22 @@ func (s *Streamer) RegisterHandler(handler MessageHandler) {
func (s *Streamer) RegisterRawHandler(handler RawMessageHandler) { func (s *Streamer) RegisterRawHandler(handler RawMessageHandler) {
s.mu.Lock() s.mu.Lock()
defer s.mu.Unlock() defer s.mu.Unlock()
s.rawHandler = handler s.rawHandler = handler
} }
// Start begins streaming BGP updates from the RIS Live API in a background goroutine. // Start begins streaming BGP updates from the RIS Live API in a background goroutine.
// It starts worker goroutines for each registered handler and manages automatic // It starts worker goroutines for each registered handler and manages automatic
// reconnection with exponential backoff on connection failures. // reconnection with exponential backoff on connection failures.
// Returns an error if the streamer is already running. It streams until Stop: // Returns an error if the streamer is already running.
// the end of ctx does not stop it. func (s *Streamer) Start() error {
func (s *Streamer) Start(ctx context.Context) error {
s.mu.Lock() s.mu.Lock()
defer s.mu.Unlock() defer s.mu.Unlock()
if s.running { if s.running {
return errAlreadyRunning return fmt.Errorf("streamer already running")
} }
ctx, cancel := context.WithCancel(context.WithoutCancel(ctx)) ctx, cancel := context.WithCancel(context.Background())
s.cancel = cancel s.cancel = cancel
s.running = true s.running = true
@@ -228,21 +216,44 @@ func (s *Streamer) Stop() {
return return
} }
s.cancel() s.cancel()
s.cancel = nil s.cancel = nil
// Close all handler queues to signal workers to stop // Close all handler queues to signal workers to stop
for _, info := range s.handlers { for _, info := range s.handlers {
close(info.queue) close(info.queue)
} }
s.running = false s.running = false
s.mu.Unlock() s.mu.Unlock()
s.metrics.SetConnected(false) s.metrics.SetConnected(false)
} }
// IsRunning reports whether the streamer is currently connected and processing // runHandlerWorker processes messages for a specific handler
// messages. This is safe to call concurrently from multiple goroutines. func (s *Streamer) runHandlerWorker(info *handlerInfo) {
for msg := range info.queue {
start := time.Now()
info.handler.HandleMessage(msg)
elapsed := time.Since(start)
// Update metrics
info.metrics.mu.Lock()
info.metrics.processedCount++
info.metrics.totalTime += elapsed
// Update min time
if elapsed < info.metrics.minTime {
info.metrics.minTime = elapsed
}
// Update max time
if elapsed > info.metrics.maxTime {
info.metrics.maxTime = elapsed
}
info.metrics.mu.Unlock()
}
}
// IsRunning reports whether the streamer is currently connected and processing messages.
// This is safe to call concurrently from multiple goroutines.
func (s *Streamer) IsRunning() bool { func (s *Streamer) IsRunning() bool {
s.mu.RLock() s.mu.RLock()
defer s.mu.RUnlock() defer s.mu.RUnlock()
@@ -303,15 +314,12 @@ func (s *Streamer) GetHandlerStats() []HandlerStats {
// Calculate average time // Calculate average time
if info.metrics.processedCount > 0 { if info.metrics.processedCount > 0 {
processedCount := info.metrics.processedCount processedCount := info.metrics.processedCount
const maxInt64 = 1<<63 - 1 const maxInt64 = 1<<63 - 1
if processedCount > maxInt64 { if processedCount > maxInt64 {
processedCount = maxInt64 processedCount = maxInt64
} }
//nolint:gosec // processedCount is explicitly bounded above
hs.AvgProcessTime = info.metrics.totalTime / hs.AvgProcessTime = info.metrics.totalTime / time.Duration(processedCount)
time.Duration(processedCount)
} }
info.metrics.mu.Unlock() info.metrics.mu.Unlock()
@@ -322,38 +330,11 @@ func (s *Streamer) GetHandlerStats() []HandlerStats {
return stats return stats
} }
// GetDroppedMessages returns the total number of messages dropped across all // GetDroppedMessages returns the total number of messages dropped across all handlers
// handlers due to queue overflow or backpressure. This counter is monotonically // due to queue overflow or backpressure. This counter is monotonically increasing
// increasing and is safe to call concurrently. // and is safe to call concurrently.
func (s *Streamer) GetDroppedMessages() uint64 { func (s *Streamer) GetDroppedMessages() uint64 {
return s.totalDropped.Load() return atomic.LoadUint64(&s.totalDropped)
}
// runHandlerWorker processes messages for a specific handler
func (s *Streamer) runHandlerWorker(info *handlerInfo) {
for msg := range info.queue {
start := time.Now()
info.handler.HandleMessage(msg)
elapsed := time.Since(start)
// Update metrics
info.metrics.mu.Lock()
info.metrics.processedCount++
info.metrics.totalTime += elapsed
// Update min time
if elapsed < info.metrics.minTime {
info.metrics.minTime = elapsed
}
// Update max time
if elapsed > info.metrics.maxTime {
info.metrics.maxTime = elapsed
}
info.metrics.mu.Unlock()
}
} }
// logMetrics logs the current streaming statistics // logMetrics logs the current streaming statistics
@@ -362,8 +343,7 @@ func (s *Streamer) logMetrics() {
uptime := time.Since(metrics.ConnectedSince) uptime := time.Since(metrics.ConnectedSince)
const bitsPerMegabit = 1000000 const bitsPerMegabit = 1000000
totalDropped := atomic.LoadUint64(&s.totalDropped)
totalDropped := s.totalDropped.Load()
s.logger.Info( s.logger.Info(
"Stream statistics", "Stream statistics",
@@ -389,19 +369,16 @@ func (s *Streamer) logMetrics() {
// Log per-handler statistics // Log per-handler statistics
s.mu.RLock() s.mu.RLock()
for i, info := range s.handlers { for i, info := range s.handlers {
info.metrics.mu.Lock() info.metrics.mu.Lock()
if info.metrics.processedCount > 0 { if info.metrics.processedCount > 0 {
// Safe conversion: processedCount is bounded by maxInt64 // Safe conversion: processedCount is bounded by maxInt64
processedCount := info.metrics.processedCount processedCount := info.metrics.processedCount
const maxInt64 = 1<<63 - 1 const maxInt64 = 1<<63 - 1
if processedCount > maxInt64 { if processedCount > maxInt64 {
processedCount = maxInt64 processedCount = maxInt64
} }
//nolint:gosec // processedCount is explicitly bounded above
avgTime := info.metrics.totalTime / time.Duration(processedCount) avgTime := info.metrics.totalTime / time.Duration(processedCount)
s.logger.Info( s.logger.Info(
"Handler statistics", "Handler statistics",
@@ -418,7 +395,6 @@ func (s *Streamer) logMetrics() {
} }
info.metrics.mu.Unlock() info.metrics.mu.Unlock()
} }
s.mu.RUnlock() s.mu.RUnlock()
} }
@@ -427,8 +403,7 @@ func (s *Streamer) updateMetrics(messageBytes int) {
s.metrics.RecordMessage(int64(messageBytes)) s.metrics.RecordMessage(int64(messageBytes))
} }
// streamWithReconnect handles streaming with automatic reconnection and // streamWithReconnect handles streaming with automatic reconnection and exponential backoff
// exponential backoff
func (s *Streamer) streamWithReconnect(ctx context.Context) { func (s *Streamer) streamWithReconnect(ctx context.Context) {
backoffDelay := minBackoffDelay backoffDelay := minBackoffDelay
consecutiveFailures := 0 consecutiveFailures := 0
@@ -469,7 +444,6 @@ func (s *Streamer) streamWithReconnect(ctx context.Context) {
if streamDuration > 30*time.Second { if streamDuration > 30*time.Second {
s.logger.Info("Resetting backoff delay due to successful connection", s.logger.Info("Resetting backoff delay due to successful connection",
"stream_duration", streamDuration) "stream_duration", streamDuration)
backoffDelay = minBackoffDelay backoffDelay = minBackoffDelay
consecutiveFailures = 0 consecutiveFailures = 0
} else { } else {
@@ -503,7 +477,7 @@ func (s *Streamer) stream(ctx context.Context) error {
connCtx, connCancel := context.WithCancel(ctx) connCtx, connCancel := context.WithCancel(ctx)
defer connCancel() defer connCancel()
req, err := http.NewRequestWithContext(ctx, http.MethodGet, s.url, nil) req, err := http.NewRequestWithContext(ctx, "GET", s.url, nil)
if err != nil { if err != nil {
return fmt.Errorf("failed to create request: %w", err) return fmt.Errorf("failed to create request: %w", err)
} }
@@ -516,14 +490,13 @@ func (s *Streamer) stream(ctx context.Context) error {
return fmt.Errorf("failed to connect to RIS Live: %w", err) return fmt.Errorf("failed to connect to RIS Live: %w", err)
} }
defer func() { defer func() {
err := resp.Body.Close() if err := resp.Body.Close(); err != nil {
if err != nil {
s.logger.Error("Failed to close response body", "error", err) s.logger.Error("Failed to close response body", "error", err)
} }
}() }()
if resp.StatusCode != http.StatusOK { if resp.StatusCode != http.StatusOK {
return fmt.Errorf("%w: %d", errUnexpectedStatus, resp.StatusCode) return fmt.Errorf("unexpected status code: %d", resp.StatusCode)
} }
// Wrap body with counting reader to track actual wire bytes // Wrap body with counting reader to track actual wire bytes
@@ -537,9 +510,7 @@ func (s *Streamer) stream(ctx context.Context) error {
return fmt.Errorf("failed to create gzip reader: %w", err) return fmt.Errorf("failed to create gzip reader: %w", err)
} }
defer func() { _ = gzReader.Close() }() defer func() { _ = gzReader.Close() }()
reader = gzReader reader = gzReader
s.logger.Info("Connected to RIS Live stream", "compression", "gzip") s.logger.Info("Connected to RIS Live stream", "compression", "gzip")
} else { } else {
s.logger.Info("Connected to RIS Live stream", "compression", "none") s.logger.Info("Connected to RIS Live stream", "compression", "none")
@@ -547,6 +518,9 @@ func (s *Streamer) stream(ctx context.Context) error {
s.metrics.SetConnected(true) s.metrics.SetConnected(true)
// Track wire bytes for metrics updates
var lastWireBytes int64
// Start metrics logging goroutine // Start metrics logging goroutine
metricsTicker := time.NewTicker(metricsLogInterval) metricsTicker := time.NewTicker(metricsLogInterval)
defer metricsTicker.Stop() defer metricsTicker.Stop()
@@ -563,41 +537,25 @@ func (s *Streamer) stream(ctx context.Context) error {
}() }()
// Wire byte update ticker - update metrics with actual wire bytes periodically // Wire byte update ticker - update metrics with actual wire bytes periodically
go s.recordWireBytes(connCtx, wireCounter)
return s.readStream(ctx, reader)
}
// recordWireBytes adds the bytes wireCounter has counted to the metrics once a
// second until ctx is cancelled.
func (s *Streamer) recordWireBytes(
ctx context.Context, wireCounter *countingReader,
) {
wireUpdateTicker := time.NewTicker(time.Second) wireUpdateTicker := time.NewTicker(time.Second)
defer wireUpdateTicker.Stop() defer wireUpdateTicker.Stop()
var lastWireBytes int64 go func() {
for {
for { select {
select { case <-wireUpdateTicker.C:
case <-wireUpdateTicker.C: currentBytes := wireCounter.Count()
currentBytes := wireCounter.Count() delta := currentBytes - lastWireBytes
if delta > 0 {
delta := currentBytes - lastWireBytes s.metrics.RecordWireBytes(delta)
if delta > 0 { lastWireBytes = currentBytes
s.metrics.RecordWireBytes(delta) }
case <-connCtx.Done():
lastWireBytes = currentBytes return
} }
case <-ctx.Done():
return
} }
} }()
}
// readStream hands each line of the stream to processLine until the stream
// ends, a line fails, or ctx is cancelled.
func (s *Streamer) readStream(ctx context.Context, reader io.Reader) error {
scanner := bufio.NewScanner(reader) scanner := bufio.NewScanner(reader)
for scanner.Scan() { for scanner.Scan() {
@@ -614,176 +572,149 @@ func (s *Streamer) readStream(ctx context.Context, reader io.Reader) error {
continue continue
} }
err := s.processLine(ctx, line) // Update metrics with decompressed message size
if err != nil { s.updateMetrics(len(line))
return err
// Call raw handler if registered
s.mu.RLock()
rawHandler := s.rawHandler
s.mu.RUnlock()
if rawHandler != nil {
// Call raw handler synchronously to preserve order
rawHandler(string(line))
} }
// Parse the message first
var wrapper ristypes.RISLiveMessage
if err := json.Unmarshal(line, &wrapper); err != nil {
// Log the error and return to trigger reconnection
s.logger.Error("Failed to parse JSON",
"error", err,
"line", string(line),
"line_length", len(line))
return fmt.Errorf("JSON parse error: %w", err)
}
// Check if it's a ris_message wrapper
if wrapper.Type != "ris_message" {
s.logger.Error("Unexpected wrapper type",
"type", wrapper.Type,
"line", string(line),
)
continue
}
// Get the actual message
msg := wrapper.Data
// Parse the timestamp
msg.ParsedTimestamp = time.Unix(int64(msg.Timestamp), 0).UTC()
// Process based on message type
switch msg.Type {
case "UPDATE":
// Process BGP UPDATE messages
// Will be dispatched to handlers
case "RIS_PEER_STATE":
// RIS peer state messages - silently ignore
continue
case "KEEPALIVE":
// BGP keepalive messages - silently process
continue
case "OPEN":
// BGP open messages - track peer as active
s.bgpPeersMu.Lock()
s.bgpPeers[msg.Peer] = true
peerCount := len(s.bgpPeers)
s.bgpPeersMu.Unlock()
s.metrics.SetBGPPeerCount(peerCount)
s.logger.Info("BGP session opened",
"peer", msg.Peer,
"peer_asn", msg.PeerASN,
"total_peers", peerCount,
)
continue
case "NOTIFICATION":
// BGP notification messages (session closed)
s.bgpPeersMu.Lock()
delete(s.bgpPeers, msg.Peer)
peerCount := len(s.bgpPeers)
s.bgpPeersMu.Unlock()
s.metrics.SetBGPPeerCount(peerCount)
s.logger.Warn("BGP notification",
"peer", msg.Peer,
"peer_asn", msg.PeerASN,
"total_peers", peerCount,
)
continue
case "STATE":
// Peer state changes - silently ignore
continue
default:
s.logger.Warn("Unknown message type, skipping",
"type", msg.Type,
)
continue
}
// Dispatch to interested handlers. Stop cancels ctx and closes the
// queues under the write lock, so if ctx is cancelled here, under the
// read lock, the queues are closed and must not be sent to.
s.mu.RLock()
if ctx.Err() != nil {
s.mu.RUnlock()
return ctx.Err()
}
for _, info := range s.handlers {
if !info.handler.WantsMessage(msg.Type) {
continue
}
// Check if we should drop due to backpressure
if s.shouldDropForBackpressure(info) {
atomic.AddUint64(&info.metrics.droppedCount, 1)
atomic.AddUint64(&s.totalDropped, 1)
continue
}
// Try to queue the message
select {
case info.queue <- &msg:
// Message queued successfully
// Update high water mark if needed
queueLen := len(info.queue)
info.metrics.mu.Lock()
if queueLen > info.metrics.queueHighWaterMark {
info.metrics.queueHighWaterMark = queueLen
}
info.metrics.mu.Unlock()
default:
// Queue is full, drop the message
atomic.AddUint64(&info.metrics.droppedCount, 1)
atomic.AddUint64(&s.totalDropped, 1)
}
}
s.mu.RUnlock()
} }
err := scanner.Err() if err := scanner.Err(); err != nil {
if err != nil {
return fmt.Errorf("scanner error: %w", err) return fmt.Errorf("scanner error: %w", err)
} }
return nil return nil
} }
// processLine passes one line of the stream to the raw handler, parses it, // shouldDropForBackpressure determines if a message should be dropped based on queue utilization
// and dispatches an UPDATE message to the handlers. An error ends the
// connection.
func (s *Streamer) processLine(ctx context.Context, line []byte) error {
// Update metrics with decompressed message size
s.updateMetrics(len(line))
// Call raw handler if registered
s.mu.RLock()
rawHandler := s.rawHandler
s.mu.RUnlock()
if rawHandler != nil {
// Call raw handler synchronously to preserve order
rawHandler(string(line))
}
// Parse the message first
var wrapper ristypes.RISLiveMessage
err := json.Unmarshal(line, &wrapper)
if err != nil {
// Log the error and return to trigger reconnection
s.logger.Error("Failed to parse JSON",
"error", err,
"line", string(line),
"line_length", len(line))
return fmt.Errorf("JSON parse error: %w", err)
}
// Check if it's a ris_message wrapper
if wrapper.Type != "ris_message" {
s.logger.Error("Unexpected wrapper type",
"type", wrapper.Type,
"line", string(line),
)
return nil
}
// Get the actual message
msg := wrapper.Data
// Parse the timestamp
msg.ParsedTimestamp = time.Unix(int64(msg.Timestamp), 0).UTC()
// Process based on message type
switch msg.Type {
case "UPDATE":
// Process BGP UPDATE messages
// Will be dispatched to handlers
case "RIS_PEER_STATE":
// RIS peer state messages - silently ignore
return nil
case "KEEPALIVE":
// BGP keepalive messages - silently process
return nil
case "OPEN":
// BGP open messages - track peer as active
s.bgpPeersMu.Lock()
s.bgpPeers[msg.Peer] = true
peerCount := len(s.bgpPeers)
s.bgpPeersMu.Unlock()
s.metrics.SetBGPPeerCount(peerCount)
s.logger.Info("BGP session opened",
"peer", msg.Peer,
"peer_asn", msg.PeerASN,
"total_peers", peerCount,
)
return nil
case "NOTIFICATION":
// BGP notification messages (session closed)
s.bgpPeersMu.Lock()
delete(s.bgpPeers, msg.Peer)
peerCount := len(s.bgpPeers)
s.bgpPeersMu.Unlock()
s.metrics.SetBGPPeerCount(peerCount)
s.logger.Warn("BGP notification",
"peer", msg.Peer,
"peer_asn", msg.PeerASN,
"total_peers", peerCount,
)
return nil
case "STATE":
// Peer state changes - silently ignore
return nil
default:
s.logger.Warn("Unknown message type, skipping",
"type", msg.Type,
)
return nil
}
return s.dispatchMessage(ctx, &msg)
}
// dispatchMessage queues msg for each handler that wants it, dropping it for
// a handler whose queue is full or under backpressure.
func (s *Streamer) dispatchMessage(
ctx context.Context, msg *ristypes.RISMessage,
) error {
// Stop cancels ctx and closes the queues under the write lock, so if ctx
// is cancelled here, under the read lock, the queues are closed and must
// not be sent to.
s.mu.RLock()
if ctx.Err() != nil {
s.mu.RUnlock()
return ctx.Err()
}
for _, info := range s.handlers {
if !info.handler.WantsMessage(msg.Type) {
continue
}
// Check if we should drop due to backpressure
if s.shouldDropForBackpressure(info) {
atomic.AddUint64(&info.metrics.droppedCount, 1)
s.totalDropped.Add(1)
continue
}
// Try to queue the message
select {
case info.queue <- msg:
// Message queued successfully
// Update high water mark if needed
queueLen := len(info.queue)
info.metrics.mu.Lock()
if queueLen > info.metrics.queueHighWaterMark {
info.metrics.queueHighWaterMark = queueLen
}
info.metrics.mu.Unlock()
default:
// Queue is full, drop the message
atomic.AddUint64(&info.metrics.droppedCount, 1)
s.totalDropped.Add(1)
}
}
s.mu.RUnlock()
return nil
}
// shouldDropForBackpressure determines if a message should be dropped based on
// queue utilization
func (s *Streamer) shouldDropForBackpressure(info *handlerInfo) bool { func (s *Streamer) shouldDropForBackpressure(info *handlerInfo) bool {
// Calculate queue utilization // Calculate queue utilization
queueLen := len(info.queue) queueLen := len(info.queue)
@@ -16,8 +16,6 @@ import (
) )
func TestNewStreamer(t *testing.T) { func TestNewStreamer(t *testing.T) {
t.Parallel()
logger := logger.New() logger := logger.New()
metricsTracker := metrics.New() metricsTracker := metrics.New()
s := New(logger, metricsTracker) s := New(logger, metricsTracker)
@@ -48,13 +46,10 @@ func TestNewStreamer(t *testing.T) {
// goroutine count returns to its starting value. Each connection starts two // goroutine count returns to its starting value. Each connection starts two
// ticker goroutines; before the fix they lived until the streamer's lifetime // ticker goroutines; before the fix they lived until the streamer's lifetime
// context was cancelled, so every reconnect leaked two. // context was cancelled, so every reconnect leaked two.
//
//nolint:paralleltest // counts goroutines, so no other test may run alongside
func TestStreamDoesNotLeakTickersAcrossReconnects(t *testing.T) { func TestStreamDoesNotLeakTickersAcrossReconnects(t *testing.T) {
// The handler returns immediately, so the response body is empty and each // The handler returns immediately, so the response body is empty and each
// stream call ends at once, standing in for a dropped connection. // stream call ends at once, standing in for a dropped connection.
srv := httptest.NewServer(http.HandlerFunc( srv := httptest.NewServer(http.HandlerFunc(func(_ http.ResponseWriter, _ *http.Request) {}))
func(_ http.ResponseWriter, _ *http.Request) {}))
defer srv.Close() defer srv.Close()
s := New(logger.New(), metrics.New()) s := New(logger.New(), metrics.New())
@@ -62,23 +57,19 @@ func TestStreamDoesNotLeakTickersAcrossReconnects(t *testing.T) {
// One warm-up connection so any persistent HTTP transport goroutine exists // One warm-up connection so any persistent HTTP transport goroutine exists
// before we take the baseline. // before we take the baseline.
err := s.stream(context.Background()) if err := s.stream(context.Background()); err != nil {
if err != nil {
t.Fatalf("warm-up stream returned error: %v", err) t.Fatalf("warm-up stream returned error: %v", err)
} }
s.client.CloseIdleConnections() s.client.CloseIdleConnections()
baseline := settledGoroutineCount() baseline := settledGoroutineCount()
const reconnects = 20 const reconnects = 20
for range reconnects { for range reconnects {
err := s.stream(context.Background()) if err := s.stream(context.Background()); err != nil {
if err != nil {
t.Fatalf("stream returned error: %v", err) t.Fatalf("stream returned error: %v", err)
} }
} }
s.client.CloseIdleConnections() s.client.CloseIdleConnections()
if !waitForGoroutines(baseline) { if !waitForGoroutines(baseline) {
@@ -90,13 +81,9 @@ func TestStreamDoesNotLeakTickersAcrossReconnects(t *testing.T) {
// updateHandler wants UPDATE messages and does nothing with them. // updateHandler wants UPDATE messages and does nothing with them.
type updateHandler struct{} type updateHandler struct{}
func (updateHandler) WantsMessage(messageType string) bool { func (updateHandler) WantsMessage(messageType string) bool { return messageType == "UPDATE" }
return messageType == "UPDATE" func (updateHandler) HandleMessage(*ristypes.RISMessage) {}
} func (updateHandler) QueueCapacity() int { return 10 }
func (updateHandler) HandleMessage(*ristypes.RISMessage) {}
func (updateHandler) QueueCapacity() int { return 10 }
// TestStopBeforeMessageReachesQueues stops the streamer after the read loop // TestStopBeforeMessageReachesQueues stops the streamer after the read loop
// has checked for cancellation but before it hands the message to the handler // has checked for cancellation but before it hands the message to the handler
@@ -104,15 +91,11 @@ func (updateHandler) QueueCapacity() int { return 10 }
// used to end in "send on closed channel". The raw handler runs in that gap on // used to end in "send on closed channel". The raw handler runs in that gap on
// the read loop itself, so calling Stop from it hits the gap every time. // the read loop itself, so calling Stop from it hits the gap every time.
func TestStopBeforeMessageReachesQueues(t *testing.T) { func TestStopBeforeMessageReachesQueues(t *testing.T) {
t.Parallel() const line = `{"type":"ris_message","data":{"type":"UPDATE","peer":"192.0.2.1",` +
`"peer_asn":"64496","timestamp":1700000000}}` + "\n"
const line = `{"type":"ris_message","data":{"type":"UPDATE",` + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
`"peer":"192.0.2.1","peer_asn":"64496","timestamp":1700000000}}` + "\n" _, _ = io.WriteString(w, line)
}))
srv := httptest.NewServer(http.HandlerFunc(
func(w http.ResponseWriter, _ *http.Request) {
_, _ = io.WriteString(w, line)
}))
defer srv.Close() defer srv.Close()
s := New(logger.New(), metrics.New()) s := New(logger.New(), metrics.New())
@@ -126,8 +109,7 @@ func TestStopBeforeMessageReachesQueues(t *testing.T) {
ctx, cancel := context.WithCancel(context.Background()) ctx, cancel := context.WithCancel(context.Background())
s.cancel = cancel s.cancel = cancel
err := s.stream(ctx) if err := s.stream(ctx); !errors.Is(err, context.Canceled) {
if !errors.Is(err, context.Canceled) {
t.Fatalf("stream returned %v, want %v", err, context.Canceled) t.Fatalf("stream returned %v, want %v", err, context.Canceled)
} }
@@ -139,15 +121,12 @@ func TestStopBeforeMessageReachesQueues(t *testing.T) {
// current count. // current count.
func settledGoroutineCount() int { func settledGoroutineCount() int {
prev := runtime.NumGoroutine() prev := runtime.NumGoroutine()
for range 20 { for range 20 {
time.Sleep(10 * time.Millisecond) time.Sleep(10 * time.Millisecond)
cur := runtime.NumGoroutine() cur := runtime.NumGoroutine()
if cur == prev { if cur == prev {
return cur return cur
} }
prev = cur prev = cur
} }
@@ -160,7 +139,6 @@ func waitForGoroutines(target int) bool {
if runtime.NumGoroutine() <= target { if runtime.NumGoroutine() <= target {
return true return true
} }
time.Sleep(10 * time.Millisecond) time.Sleep(10 * time.Millisecond)
} }
+4 -11
View File
@@ -60,7 +60,6 @@ func timeSince(t time.Time) string {
if duration < time.Minute { if duration < time.Minute {
return "just now" return "just now"
} }
if duration < time.Hour { if duration < time.Hour {
minutes := int(duration.Minutes()) minutes := int(duration.Minutes())
if minutes == 1 { if minutes == 1 {
@@ -69,7 +68,6 @@ func timeSince(t time.Time) string {
return duration.Truncate(time.Minute).String() + " ago" return duration.Truncate(time.Minute).String() + " ago"
} }
if duration < hoursPerDay*time.Hour { if duration < hoursPerDay*time.Hour {
hours := int(duration.Hours()) hours := int(duration.Hours())
if hours == 1 { if hours == 1 {
@@ -78,12 +76,10 @@ func timeSince(t time.Time) string {
return duration.Truncate(time.Hour).String() + " ago" return duration.Truncate(time.Hour).String() + " ago"
} }
days := int(duration.Hours() / hoursPerDay) days := int(duration.Hours() / hoursPerDay)
if days == 1 { if days == 1 {
return "1 day ago" return "1 day ago"
} }
if days < daysPerMonth { if days < daysPerMonth {
return duration.Truncate(hoursPerDay*time.Hour).String() + " ago" return duration.Truncate(hoursPerDay*time.Hour).String() + " ago"
} }
@@ -122,7 +118,7 @@ func initTemplates() {
"appLicense": func() string { return version.License }, "appLicense": func() string { return version.License },
"appRepoURL": func() string { return version.RepoURL }, "appRepoURL": func() string { return version.RepoURL },
"appGitRevision": func() string { return version.GitRevisionShort }, "appGitRevision": func() string { return version.GitRevisionShort },
"appGitCommitURL": version.CommitURL, "appGitCommitURL": func() string { return version.CommitURL() },
} }
// Parse index template // Parse index template
@@ -138,22 +134,19 @@ func initTemplates() {
} }
// Parse AS detail template // Parse AS detail template
defaultTemplates.ASDetail, err = template.New("asDetail"). defaultTemplates.ASDetail, err = template.New("asDetail").Funcs(funcs).Parse(asDetailHTML)
Funcs(funcs).Parse(asDetailHTML)
if err != nil { if err != nil {
panic("failed to parse AS detail template: " + err.Error()) panic("failed to parse AS detail template: " + err.Error())
} }
// Parse prefix detail template // Parse prefix detail template
defaultTemplates.PrefixDetail, err = template.New("prefixDetail"). defaultTemplates.PrefixDetail, err = template.New("prefixDetail").Funcs(funcs).Parse(prefixDetailHTML)
Funcs(funcs).Parse(prefixDetailHTML)
if err != nil { if err != nil {
panic("failed to parse prefix detail template: " + err.Error()) panic("failed to parse prefix detail template: " + err.Error())
} }
// Parse prefix length template // Parse prefix length template
defaultTemplates.PrefixLength, err = template.New("prefixLength"). defaultTemplates.PrefixLength, err = template.New("prefixLength").Funcs(funcs).Parse(prefixLengthHTML)
Funcs(funcs).Parse(prefixLengthHTML)
if err != nil { if err != nil {
panic("failed to parse prefix length template: " + err.Error()) panic("failed to parse prefix length template: " + err.Error())
} }
+1 -1
View File
@@ -3,7 +3,7 @@ package version
// Build-time variables set via ldflags // Build-time variables set via ldflags
// //
//nolint:gochecknoglobals // variables so that ldflags can set them at build time //nolint:gochecknoglobals // These must be variables to allow ldflags injection at build time
var ( var (
// GitRevision is the git commit hash // GitRevision is the git commit hash
GitRevision = "unknown" GitRevision = "unknown"
+78 -106
View File
@@ -98,8 +98,7 @@ func (c *Client) LookupASN(ctx context.Context, asn int) (*ASNInfo, error) {
if err != nil { if err != nil {
// Return partial data from IANA if RIR query fails // Return partial data from IANA if RIR query fails
info := c.parseResponse(asn, rir, ianaResp) info := c.parseResponse(asn, rir, ianaResp)
info.RawResponse = fmt.Sprintf("%s\n--- RIR query failed: %v ---\n", info.RawResponse = ianaResp + "\n--- RIR query failed: " + err.Error() + " ---\n"
ianaResp, err)
return info, nil return info, nil
} }
@@ -121,33 +120,28 @@ func (c *Client) query(ctx context.Context, server, query string) (string, error
defer func() { _ = conn.Close() }() defer func() { _ = conn.Close() }()
// Set deadlines // Set deadlines
err = conn.SetWriteDeadline(time.Now().Add(writeTimeout)) if err := conn.SetWriteDeadline(time.Now().Add(writeTimeout)); err != nil {
if err != nil {
return "", fmt.Errorf("set write deadline: %w", err) return "", fmt.Errorf("set write deadline: %w", err)
} }
// Send query // Send query
_, err = fmt.Fprintf(conn, "%s\r\n", query) if _, err := fmt.Fprintf(conn, "%s\r\n", query); err != nil {
if err != nil {
return "", fmt.Errorf("write query: %w", err) return "", fmt.Errorf("write query: %w", err)
} }
// Read response // Read response
err = conn.SetReadDeadline(time.Now().Add(readTimeout)) if err := conn.SetReadDeadline(time.Now().Add(readTimeout)); err != nil {
if err != nil {
return "", fmt.Errorf("set read deadline: %w", err) return "", fmt.Errorf("set read deadline: %w", err)
} }
var sb strings.Builder var sb strings.Builder
scanner := bufio.NewScanner(conn) scanner := bufio.NewScanner(conn)
for scanner.Scan() { for scanner.Scan() {
sb.WriteString(scanner.Text()) sb.WriteString(scanner.Text())
sb.WriteString("\n") sb.WriteString("\n")
} }
err = scanner.Err() if err := scanner.Err(); err != nil {
if err != nil {
return sb.String(), fmt.Errorf("read response: %w", err) return sb.String(), fmt.Errorf("read response: %w", err)
} }
@@ -155,10 +149,10 @@ func (c *Client) query(ctx context.Context, server, query string) (string, error
} }
// parseIANAReferral extracts the RIR and WHOIS server from an IANA response. // parseIANAReferral extracts the RIR and WHOIS server from an IANA response.
func (c *Client) parseIANAReferral(response string) (string, string) { func (c *Client) parseIANAReferral(response string) (rir, whoisServer string) {
var rir string lines := strings.Split(response, "\n")
for line := range strings.SplitSeq(response, "\n") { for _, line := range lines {
line = strings.TrimSpace(line) line = strings.TrimSpace(line)
// Look for "refer:" line // Look for "refer:" line
@@ -189,9 +183,18 @@ func (c *Client) parseIANAReferral(response string) (string, string) {
// Also check organisation line for RIR hints // Also check organisation line for RIR hints
if strings.HasPrefix(strings.ToLower(line), "organisation:") { if strings.HasPrefix(strings.ToLower(line), "organisation:") {
orgRIR := rirInOrganisationLine(strings.ToLower(line)) org := strings.ToLower(line)
if orgRIR != "" { switch {
rir = orgRIR case strings.Contains(org, "arin"):
rir = RIRARIN
case strings.Contains(org, "ripe"):
rir = RIRRIPE
case strings.Contains(org, "apnic"):
rir = RIRAPNIC
case strings.Contains(org, "lacnic"):
rir = RIRLACNIC
case strings.Contains(org, "afrinic"):
rir = RIRAFRNIC
} }
} }
} }
@@ -199,25 +202,6 @@ func (c *Client) parseIANAReferral(response string) (string, string) {
return rir, "" return rir, ""
} }
// rirInOrganisationLine returns the RIR a lowercased IANA "organisation:"
// line names, or "" when it names none.
func rirInOrganisationLine(org string) string {
switch {
case strings.Contains(org, "arin"):
return RIRARIN
case strings.Contains(org, "ripe"):
return RIRRIPE
case strings.Contains(org, "apnic"):
return RIRAPNIC
case strings.Contains(org, "lacnic"):
return RIRLACNIC
case strings.Contains(org, "afrinic"):
return RIRAFRNIC
default:
return ""
}
}
// parseResponse extracts ASN information from a WHOIS response. // parseResponse extracts ASN information from a WHOIS response.
func (c *Client) parseResponse(asn int, rir, response string) *ASNInfo { func (c *Client) parseResponse(asn int, rir, response string) *ASNInfo {
info := &ASNInfo{ info := &ASNInfo{
@@ -227,13 +211,11 @@ func (c *Client) parseResponse(asn int, rir, response string) *ASNInfo {
} }
lines := strings.Split(response, "\n") lines := strings.Split(response, "\n")
var addressLines []string var addressLines []string
for _, line := range lines { for _, line := range lines {
line = strings.TrimSpace(line) line = strings.TrimSpace(line)
if line == "" || strings.HasPrefix(line, "%") || if line == "" || strings.HasPrefix(line, "%") || strings.HasPrefix(line, "#") {
strings.HasPrefix(line, "#") {
continue continue
} }
@@ -250,14 +232,63 @@ func (c *Client) parseResponse(asn int, rir, response string) *ASNInfo {
continue continue
} }
switch key {
// AS Name (varies by RIR)
case "asname", "as-name":
if info.ASName == "" {
info.ASName = value
}
// Organization
case "orgname", "org-name", "owner":
if info.OrgName == "" {
info.OrgName = value
}
case "orgid", "org-id", "org":
if info.OrgID == "" {
info.OrgID = value
}
// Address (collect multiple lines) // Address (collect multiple lines)
if key == "address" { case "address":
addressLines = append(addressLines, value) addressLines = append(addressLines, value)
continue // Country
} case "country":
if info.CountryCode == "" && len(value) == 2 {
info.CountryCode = strings.ToUpper(value)
}
c.setResponseField(info, key, value) // Abuse contact
case "orgabuseemail", "abuse-mailbox":
if info.AbuseEmail == "" {
info.AbuseEmail = value
}
case "orgabusephone":
if info.AbusePhone == "" {
info.AbusePhone = value
}
// Tech contact
case "orgtechemail":
if info.TechEmail == "" {
info.TechEmail = value
}
case "orgtechphone":
if info.TechPhone == "" {
info.TechPhone = value
}
// Registration dates
case "regdate", "created":
if info.RegDate == nil {
info.RegDate = c.parseDate(value)
}
case "updated", "last-modified", "changed":
if info.LastMod == nil {
info.LastMod = c.parseDate(value)
}
}
} }
// Combine address lines // Combine address lines
@@ -273,65 +304,6 @@ func (c *Client) parseResponse(asn int, rir, response string) *ASNInfo {
return info return info
} }
// setResponseField stores the value of one "key: value" line of a WHOIS
// response in info, keeping the first value seen for each field.
func (c *Client) setResponseField(info *ASNInfo, key, value string) {
switch key {
// Country
case "country":
if info.CountryCode == "" && len(value) == 2 {
info.CountryCode = strings.ToUpper(value)
}
// Registration dates
case "regdate", "created":
if info.RegDate == nil {
info.RegDate = c.parseDate(value)
}
case "updated", "last-modified", "changed":
if info.LastMod == nil {
info.LastMod = c.parseDate(value)
}
default:
field := textFieldForKey(info, key)
if field != nil && *field == "" {
*field = value
}
}
}
// textFieldForKey returns the text field of info that a WHOIS response key
// fills, or nil when the key fills none.
func textFieldForKey(info *ASNInfo, key string) *string {
switch key {
// AS Name (varies by RIR)
case "asname", "as-name":
return &info.ASName
// Organization
case "orgname", "org-name", "owner":
return &info.OrgName
case "orgid", "org-id", "org":
return &info.OrgID
// Abuse contact
case "orgabuseemail", "abuse-mailbox":
return &info.AbuseEmail
case "orgabusephone":
return &info.AbusePhone
// Tech contact
case "orgtechemail":
return &info.TechEmail
case "orgtechphone":
return &info.TechPhone
default:
return nil
}
}
// parseDate attempts to parse various date formats used in WHOIS responses. // parseDate attempts to parse various date formats used in WHOIS responses.
func (c *Client) parseDate(value string) *time.Time { func (c *Client) parseDate(value string) *time.Time {
// Common formats // Common formats
@@ -347,15 +319,15 @@ func (c *Client) parseDate(value string) *time.Time {
value = strings.TrimSpace(value) value = strings.TrimSpace(value)
// Handle "YYYYMMDD" format from LACNIC // Handle "YYYYMMDD" format from LACNIC
if len(value) == lacnicDateFormatLen { if len(value) == lacnicDateFormatLen {
t, err := time.Parse("20060102", value) if _, err := time.Parse("20060102", value); err == nil {
if err == nil { t, _ := time.Parse("20060102", value)
return &t return &t
} }
} }
for _, format := range formats { for _, format := range formats {
t, err := time.Parse(format, value) if t, err := time.Parse(format, value); err == nil {
if err == nil {
return &t return &t
} }
} }
+2 -4
View File
@@ -1,3 +1,4 @@
// Package asinfo provides information about Autonomous Systems (AS)
package asinfo package asinfo
import ( import (
@@ -53,9 +54,7 @@ func initRegistry() {
} }
var asInfos []Info var asInfos []Info
if err := json.Unmarshal(jsonData, &asInfos); err != nil {
err = json.Unmarshal(jsonData, &asInfos)
if err != nil {
panic("failed to unmarshal embedded AS data: " + err.Error()) panic("failed to unmarshal embedded AS data: " + err.Error())
} }
@@ -138,7 +137,6 @@ func Search(query string) []Info {
defer defaultRegistry.mu.RUnlock() defer defaultRegistry.mu.RUnlock()
var results []Info var results []Info
for _, info := range defaultRegistry.byASN { for _, info := range defaultRegistry.byASN {
if contains(info.Handle, query) || contains(info.Description, query) { if contains(info.Handle, query) || contains(info.Description, query) {
results = append(results, *info) results = append(results, *info)
+16 -52
View File
@@ -1,15 +1,10 @@
package asinfo_test package asinfo
import ( import (
"strings"
"testing" "testing"
"git.eeqj.de/sneak/routewatch/pkg/asinfo"
) )
func TestGet(t *testing.T) { func TestGet(t *testing.T) {
t.Parallel()
tests := []struct { tests := []struct {
name string name string
asn int asn int
@@ -43,26 +38,20 @@ func TestGet(t *testing.T) {
for _, tt := range tests { for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) { t.Run(tt.name, func(t *testing.T) {
t.Parallel() info, ok := Get(tt.asn)
info, ok := asinfo.Get(tt.asn)
if ok != tt.wantOK { if ok != tt.wantOK {
t.Errorf("Get(%d) ok = %v, want %v", tt.asn, ok, tt.wantOK) t.Errorf("Get(%d) ok = %v, want %v", tt.asn, ok, tt.wantOK)
return return
} }
if ok && info.Description != tt.wantDesc { if ok && info.Description != tt.wantDesc {
t.Errorf("Get(%d) description = %q, want %q", t.Errorf("Get(%d) description = %q, want %q", tt.asn, info.Description, tt.wantDesc)
tt.asn, info.Description, tt.wantDesc)
} }
}) })
} }
} }
func TestGetDescription(t *testing.T) { func TestGetDescription(t *testing.T) {
t.Parallel()
tests := []struct { tests := []struct {
asn int asn int
want string want string
@@ -73,10 +62,7 @@ func TestGetDescription(t *testing.T) {
for _, tt := range tests { for _, tt := range tests {
t.Run("", func(t *testing.T) { t.Run("", func(t *testing.T) {
t.Parallel() if got := GetDescription(tt.asn); got != tt.want {
got := asinfo.GetDescription(tt.asn)
if got != tt.want {
t.Errorf("GetDescription(%d) = %q, want %q", tt.asn, got, tt.want) t.Errorf("GetDescription(%d) = %q, want %q", tt.asn, got, tt.want)
} }
}) })
@@ -84,8 +70,6 @@ func TestGetDescription(t *testing.T) {
} }
func TestGetHandle(t *testing.T) { func TestGetHandle(t *testing.T) {
t.Parallel()
tests := []struct { tests := []struct {
asn int asn int
want string want string
@@ -97,10 +81,7 @@ func TestGetHandle(t *testing.T) {
for _, tt := range tests { for _, tt := range tests {
t.Run("", func(t *testing.T) { t.Run("", func(t *testing.T) {
t.Parallel() if got := GetHandle(tt.asn); got != tt.want {
got := asinfo.GetHandle(tt.asn)
if got != tt.want {
t.Errorf("GetHandle(%d) = %q, want %q", tt.asn, got, tt.want) t.Errorf("GetHandle(%d) = %q, want %q", tt.asn, got, tt.want)
} }
}) })
@@ -108,23 +89,18 @@ func TestGetHandle(t *testing.T) {
} }
func TestTotal(t *testing.T) { func TestTotal(t *testing.T) {
t.Parallel() total := Total()
total := asinfo.Total()
if total < 100000 { if total < 100000 {
t.Errorf("Total() = %d, expected > 100000", total) t.Errorf("Total() = %d, expected > 100000", total)
} }
// Verify it's consistent // Verify it's consistent
total2 := asinfo.Total() if total2 := Total(); total2 != total {
if total2 != total {
t.Errorf("Total() returned different values: %d vs %d", total, total2) t.Errorf("Total() returned different values: %d vs %d", total, total2)
} }
} }
func TestSearch(t *testing.T) { func TestSearch(t *testing.T) {
t.Parallel()
tests := []struct { tests := []struct {
name string name string
query string query string
@@ -154,21 +130,16 @@ func TestSearch(t *testing.T) {
for _, tt := range tests { for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) { t.Run(tt.name, func(t *testing.T) {
t.Parallel() results := Search(tt.query)
results := asinfo.Search(tt.query)
if len(results) < tt.wantMin { if len(results) < tt.wantMin {
t.Errorf("Search(%q) returned %d results, want at least %d", t.Errorf("Search(%q) returned %d results, want at least %d", tt.query, len(results), tt.wantMin)
tt.query, len(results), tt.wantMin)
} }
// Verify all results contain the query // Verify all results contain the query
if tt.query != "" { if tt.query != "" {
for _, r := range results { for _, r := range results {
if !strings.Contains(r.Handle, tt.query) && if !contains(r.Handle, tt.query) && !contains(r.Description, tt.query) {
!strings.Contains(r.Description, tt.query) { t.Errorf("Search result ASN %d doesn't contain query %q", r.ASN, tt.query)
t.Errorf("Search result ASN %d doesn't contain query %q",
r.ASN, tt.query)
} }
} }
} }
@@ -177,17 +148,13 @@ func TestSearch(t *testing.T) {
} }
func TestDataIntegrity(t *testing.T) { func TestDataIntegrity(t *testing.T) {
t.Parallel()
// Verify no duplicate ASNs // Verify no duplicate ASNs
all := asinfo.All() all := All()
seen := make(map[int]bool) seen := make(map[int]bool)
for _, info := range all { for _, info := range all {
if seen[info.ASN] { if seen[info.ASN] {
t.Errorf("Duplicate ASN found: %d", info.ASN) t.Errorf("Duplicate ASN found: %d", info.ASN)
} }
seen[info.ASN] = true seen[info.ASN] = true
} }
@@ -196,7 +163,6 @@ func TestDataIntegrity(t *testing.T) {
if info.Handle == "" && info.ASN != 0 { if info.Handle == "" && info.ASN != 0 {
t.Errorf("ASN %d has empty handle", info.ASN) t.Errorf("ASN %d has empty handle", info.ASN)
} }
if info.Description == "" { if info.Description == "" {
t.Errorf("ASN %d has empty description", info.ASN) t.Errorf("ASN %d has empty description", info.ASN)
} }
@@ -208,9 +174,8 @@ func BenchmarkGet(b *testing.B) {
asns := []int{1, 15169, 13335, 32934, 8075, 16509} asns := []int{1, 15169, 13335, 32934, 8075, 16509}
b.ResetTimer() b.ResetTimer()
for i := 0; i < b.N; i++ {
for i := range b.N { Get(asns[i%len(asns)])
asinfo.Get(asns[i%len(asns)])
} }
} }
@@ -218,8 +183,7 @@ func BenchmarkSearch(b *testing.B) {
queries := []string{"Google", "Amazon", "Microsoft", "University"} queries := []string{"Google", "Amazon", "Microsoft", "University"}
b.ResetTimer() b.ResetTimer()
for i := 0; i < b.N; i++ {
for i := range b.N { Search(queries[i%len(queries)])
asinfo.Search(queries[i%len(queries)])
} }
} }
+6 -6
View File
@@ -3,8 +3,7 @@
# this repo. Idempotent: every install is guarded by a check so already # this repo. Idempotent: every install is guarded by a check so already
# installed tools are skipped. Base tooling comes from nix, apt, brew, # installed tools are skipped. Base tooling comes from nix, apt, brew,
# or apk (detected in that order); assumes NOTHING is present (not git, # or apk (detected in that order); assumes NOTHING is present (not git,
# make, or go). The linter and the tests run only in Docker, as phases of # make, or go).
# the Dockerfile, so nothing installs golangci-lint here.
set -eu set -eu
ROOT="$(cd "$(dirname "$0")/.." && pwd -P)" ROOT="$(cd "$(dirname "$0")/.." && pwd -P)"
@@ -59,10 +58,11 @@ main() {
# Go toolchain # Go toolchain
if missing go; then pkg_install go golang go go; fi if missing go; then pkg_install go golang go go; fi
# Warn, don't fail: make build works without it. # golangci-lint: packaged in nix, brew, and apk. There is no apt
if missing docker; then # package; on apt systems install it manually from a hash-verified
echo "bootstrap: WARNING: docker not found; install it to run" \ # GitHub release archive (never curl | sh).
"make test, make lint, make check and make docker." >&2 if missing golangci-lint; then
pkg_install golangci-lint golangci-lint golangci-lint golangci-lint
fi fi
go mod download go mod download
+2 -3
View File
@@ -1,8 +1,7 @@
#!/bin/sh #!/bin/sh
# script/check: run all checks (test, lint, fmt-check). Our own # script/check: run all checks (test, lint, fmt-check). Our own
# extension to scripts-to-rule-them-all. test and lint are Docker # extension to scripts-to-rule-them-all. Must not modify any files.
# phases; fmt-check is native, because a formatter writes the working # Generic: usually needs no adaptation.
# tree. Must not modify any files.
set -eu set -eu
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd -P)" SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd -P)"
+5 -19
View File
@@ -1,28 +1,14 @@
#!/bin/sh #!/bin/sh
# script/cibuild: run the CI build. It bootstraps first: a CI runner # script/cibuild: run the CI build. The Dockerfile runs script/check
# checks out and runs this and nothing else, and script/fmt-check runs # (via make check), so a successful build implies all checks pass.
# the formatter on the host, which a pristine checkout cannot do. # Generic: needs no adaptation. The Gitea workflow runs this on push.
# --no-cache for the same reason as script/docker: the gate phases the
# final stage depends on are RUN steps, and a cached one is a check that
# did not run.
set -eu set -eu
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd -P)" ROOT="$(cd "$(dirname "$0")/.." && pwd -P)"
ROOT="$(cd "$SCRIPT_DIR/.." && pwd -P)"
main() { main() {
cd "$ROOT" cd "$ROOT"
"$SCRIPT_DIR/bootstrap" docker build .
"$SCRIPT_DIR/check"
# Own line: a failing command substitution inside an argument does
# not trip `set -e`, so the inline form degrades silently to an
# empty constant. The VERSION build argument takes precedence over
# the version a build stage derives from the .git in the context.
version="$(git describe --tags --always --dirty 2>/dev/null || true)"
[ -n "$version" ] || version="unknown"
docker build --no-cache \
--build-arg VERSION="$version" \
-t "$("$SCRIPT_DIR/projectname")" .
} }
main "$@" main "$@"
+4 -14
View File
@@ -1,23 +1,13 @@
#!/bin/sh #!/bin/sh
# script/lint: run the linter. Linting is a phase of the Dockerfile and # script/lint: run the linters.
# this builds that phase alone; the linter is never installed or run on
# a developer host, where a shared result cache and a host-global lock
# make its answer untrustworthy.
#
# The phase is not the last stage in the file, so it is built only when
# --target names it. --no-cache because a cached lint layer is a lint
# that did not run. The tag makes each build replace the previous image
# instead of leaving a dangling one behind.
set -eu set -eu
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd -P)" ROOT="$(cd "$(dirname "$0")/.." && pwd -P)"
ROOT="$(cd "$SCRIPT_DIR/.." && pwd -P)"
main() { main() {
cd "$ROOT" cd "$ROOT"
docker build --no-cache \ go vet ./...
--target lint \ golangci-lint run
-t "$("$SCRIPT_DIR/projectname")-lint" .
} }
main "$@" main "$@"
+8 -10
View File
@@ -1,19 +1,17 @@
#!/bin/sh #!/bin/sh
# script/test: run the test suite. Testing is a phase of the Dockerfile # script/test: run the test suite. On failure, rerun verbosely so the
# and this builds that phase alone, on the same terms as script/lint: # failing tests are visible in the output.
# --target because a phase that is not the last stage is built only when
# named, --no-cache because a cached test layer is a test that did not
# run, and a tag so each build replaces the previous image.
set -eu set -eu
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd -P)" ROOT="$(cd "$(dirname "$0")/.." && pwd -P)"
ROOT="$(cd "$SCRIPT_DIR/.." && pwd -P)"
main() { main() {
cd "$ROOT" cd "$ROOT"
docker build --no-cache \ go test -short -timeout 30s -race -cover ./... || {
--target test \ echo "--- Rerunning with -v for details ---"
-t "$("$SCRIPT_DIR/projectname")-test" . go test -short -timeout 30s -race -v ./...
exit 1
}
} }
main "$@" main "$@"