sneak/routewatch
1
0
Fork 0
Code Issues 12 Pull Requests 5 Actions 38 Packages Projects Releases Wiki Activity

Compare commits

merge into: sneak/routewatch:issue-66-prefix-length-sample
Branches Tags
sneak/routewatch:main sneak/routewatch:issue-52-revendor-prompts sneak/routewatch:issue-66-prefix-length-sample sneak/routewatch:issue-55-route-per-prefix-and-peer sneak/routewatch:issue-65-batch-write-off-mutex sneak/routewatch:next sneak/routewatch:issue-70-timeout-504 sneak/routewatch:issue-27-cached-stats sneak/routewatch:prod
..
pull from: sneak/routewatch:issue-27-cached-stats
Branches Tags
sneak/routewatch:issue-52-revendor-prompts sneak/routewatch:issue-66-prefix-length-sample sneak/routewatch:issue-55-route-per-prefix-and-peer sneak/routewatch:issue-65-batch-write-off-mutex sneak/routewatch:next sneak/routewatch:issue-70-timeout-504 sneak/routewatch:main sneak/routewatch:issue-27-cached-stats sneak/routewatch:prod
1 Commits
issue-66-prefix-length-sample .. issue-27-cached-stats
Author SHA1 Message Date
sneak cd59cb8a8d Serve /api/v1/stats from a cache and index-scan the route timestamps (closes #27)
check / check (push) Failing after 0s
Details 
Once the database passed about 4.5 GiB every stats request ran a COUNT(*)
over each table plus a MIN/MAX union scan of both route tables, took the
full timeout and returned HTTP 500, so the status page went blank.

The server now keeps the last database statistics in memory and recomputes
them at most once every 30 seconds; requests serve the cached copy and a
stale copy triggers a single background refresh, so no request runs the
scans. The route-count split is folded into the cached stats, removing the
separate per-request live-route count query.

The oldest/newest route timestamps now read one row from each end of the
last_updated index instead of scanning both tables, and select the column
directly so the driver parses it into time.Time; the old aggregate returned
an untyped string that failed to scan and logged a warning every call.

Model: opus-4-8
 2026-09-21 23:32:11 +00:00
34 changed files with 1460 additions and 3189 deletions
Expand all files Collapse all files
.dockerignore
-32
View File
@@ -1,32 +0,0 @@
# Docker does not read .gitignore, and a pattern here matches from the root of
# the build context only: a pattern meant for every directory needs `**/`.
# .git is sent without its config. Without a VERSION build argument the
# stage that compiles runs `git describe --tags --always` on .git, which
# does not need .git/config; that file can hold a credential, such as a
# password in a remote URL or the token the CI checkout step stores there.
.git/config
# Local build and debug output: `make build`, `make run`, `make asupdate`, test
# binaries, coverage profiles and source archives.
/bin
/log.txt
/out
/pkg/asinfo/asdata.json
**/*.tar.zst
**/*.test
**/*.out
**/*.tmp
# Local databases and secrets. The image carries a source archive of the whole
# build context, so these would otherwise ship inside it.
**/*.db
**/*.db-journal
**/*.db-wal
**/.env
# A local Go workspace points at directories outside the build context.
/go.work
/go.work.sum
**/.DS_Store
Dockerfile
+6 -21
View File
@@ -40,23 +40,8 @@ RUN go mod download && go mod vendor
# installed above. The suite is offline (the live-feed test is opt-in). # installed above. The suite is offline (the live-feed test is opt-in).
RUN make test RUN make test
# Build the binary with CGO enabled (required for sqlite3). The version the # Build the binary with CGO enabled (required for sqlite3)
# page footer shows is the VERSION build argument when one is given, otherwise RUN CGO_ENABLED=1 GOOS=linux go build -o /routewatch ./cmd/routewatch
# `git describe --tags --always` of the .git in the build context (git comes
# with this image): the tag on a tagged commit, tag-N-gHASH after one, the
# short commit when no tag is reachable. A context that carries .git and still
# yields no version fails the build. The footer links to the full commit.
ARG VERSION
RUN version="${VERSION:-$(git describe --tags --always || echo unknown)}"; \
if [ -e .git ] && { [ -z "$version" ] || [ "$version" = dev ] || \
[ "$version" = unknown ]; }; then \
echo "no version could be derived although the build context carries .git" >&2; \
exit 1; \
fi; \
CGO_ENABLED=1 GOOS=linux go build -o /routewatch -ldflags "\
-X git.eeqj.de/sneak/routewatch/internal/version.GitRevision=$(git rev-parse --verify HEAD || echo unknown) \
-X git.eeqj.de/sneak/routewatch/internal/version.GitRevisionShort=$version" \
./cmd/routewatch
# Create source archive with vendored dependencies # Create source archive with vendored dependencies
RUN tar --zstd -cf /routewatch-source.tar.zst \ RUN tar --zstd -cf /routewatch-source.tar.zst \
@@ -94,8 +79,8 @@ RUN chown -R routewatch:routewatch /app
ENV XDG_DATA_HOME=/var/lib ENV XDG_DATA_HOME=/var/lib
# Cap the Go heap at 1.5 GiB so the runtime collects harder before the # Cap the Go heap at 1.5 GiB so the runtime collects harder before the
# container's memory limit is reached. setpriv in the entrypoint preserves this # container's memory limit is reached. runuser preserves this the way it does
# the way it does XDG_DATA_HOME above. # XDG_DATA_HOME above.
ENV GOMEMLIMIT=1536MiB ENV GOMEMLIMIT=1536MiB
# Cap glibc's malloc arenas. The SQLite C library allocates and frees millions # Cap glibc's malloc arenas. The SQLite C library allocates and frees millions
@@ -111,8 +96,8 @@ EXPOSE 8080
COPY ./entrypoint.sh /entrypoint.sh COPY ./entrypoint.sh /entrypoint.sh
# Health check using the health endpoint, on the port PORT names # Health check using the health endpoint
HEALTHCHECK --interval=30s --timeout=5s --start-period=10s --retries=3 \ HEALTHCHECK --interval=30s --timeout=5s --start-period=10s --retries=3 \
CMD curl -sf "http://localhost:${PORT:-8080}/.well-known/healthcheck.json" || exit 1 CMD curl -sf http://localhost:8080/.well-known/healthcheck.json || exit 1
ENTRYPOINT ["/bin/bash", "/entrypoint.sh" ] ENTRYPOINT ["/bin/bash", "/entrypoint.sh" ]
LICENSE
-21
View File
@@ -1,21 +0,0 @@
MIT License
Copyright (c) 2026 Jeffrey Paul <sneak@sneak.berlin>
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
Makefile
+1 -1
View File
@@ -2,7 +2,7 @@ export DEBUG = routewatch
# Git revision for version embedding # Git revision for version embedding
GIT_REVISION := $(shell git rev-parse HEAD 2>/dev/null || echo "unknown") GIT_REVISION := $(shell git rev-parse HEAD 2>/dev/null || echo "unknown")
GIT_REVISION_SHORT := $(shell git describe --tags --always 2>/dev/null || echo "unknown") GIT_REVISION_SHORT := $(shell git rev-parse --short HEAD 2>/dev/null || echo "unknown")
VERSION_PKG := git.eeqj.de/sneak/routewatch/internal/version VERSION_PKG := git.eeqj.de/sneak/routewatch/internal/version
LDFLAGS := -X $(VERSION_PKG).GitRevision=$(GIT_REVISION) -X $(VERSION_PKG).GitRevisionShort=$(GIT_REVISION_SHORT) LDFLAGS := -X $(VERSION_PKG).GitRevision=$(GIT_REVISION) -X $(VERSION_PKG).GitRevisionShort=$(GIT_REVISION_SHORT)
README.md
+8 -45
View File
@@ -1,9 +1,6 @@
# RouteWatch # RouteWatch
RouteWatch is an MIT-licensed Go daemon by @sneak that monitors the BGP routing RouteWatch is a real-time BGP routing table monitor that streams BGP UPDATE messages from the RIPE RIS Live service, maintains a live routing table in SQLite, and provides HTTP APIs for querying routing information.
table in real time: it streams BGP UPDATE messages from the RIPE RIS Live
service, maintains a live routing table in SQLite, and provides HTTP APIs for
querying routing information.
## Features ## Features
@@ -141,15 +138,8 @@ routewatch/
- **Batched Writes**: All database operations are batched for performance - **Batched Writes**: All database operations are batched for performance
- **Backpressure**: Probabilistic message dropping when queues exceed 50% capacity - **Backpressure**: Probabilistic message dropping when queues exceed 50% capacity
- **Graceful Shutdown**: 60-second timeout, flushes all pending batches - **Graceful Shutdown**: 60-second timeout, flushes all pending batches
- **Reconnection**: Any end of the stream other than shutdown, including RIS Live - **Reconnection**: Exponential backoff (5s-320s) with reset after 30s of stable connection
ending the response normally, marks the feed disconnected and reconnects with - **IPv4 Optimization**: IP ranges stored as uint32 for O(1) lookups
exponential backoff (5s-320s). The backoff resets after a stream that lasted
more than 30s. A stream with no line for 60s is dropped and reconnected.
- **IP Lookup**: the most specific live route is found by looking up the
address's prefix at each mask length, longest first, in the prefix index
(at most 25 lookups for IPv4, 113 for IPv6); prefixes are stored in the
text form Go's `net/netip` prints. Routes shorter than /8 for IPv4 or /16
for IPv6, such as a default route, are not used.
### Database Schema ### Database Schema
@@ -161,7 +151,7 @@ prefixes_v6(id, prefix, mask_length, first_seen, last_seen)
-- Live routing tables (one per IP version) -- Live routing tables (one per IP version)
live_routes_v4(id, prefix, mask_length, origin_asn, peer_ip, as_path, live_routes_v4(id, prefix, mask_length, origin_asn, peer_ip, as_path,
next_hop, last_updated) next_hop, last_updated, v4_ip_start, v4_ip_end)
live_routes_v6(id, prefix, mask_length, origin_asn, peer_ip, as_path, live_routes_v6(id, prefix, mask_length, origin_asn, peer_ip, as_path,
next_hop, last_updated) next_hop, last_updated)
@@ -176,21 +166,14 @@ Configuration is handled via environment variables and OS-specific paths:
| Variable | Default | Description | | Variable | Default | Description |
|----------|----------|-------------| |----------|----------|-------------|
| `PORT` | `8080` | HTTP server port, a whole number from 1 to 65535 | | `PORT` | `8080` | HTTP server port |
| `DEBUG` | (empty) | Set to `routewatch` for debug logging | | `DEBUG` | (empty) | Set to `routewatch` for debug logging |
| `XDG_DATA_HOME` | `/var/lib` (in the Docker image) | Base of the state directory; must be an absolute path |
| `GOMEMLIMIT` | `1536MiB` (in the Docker image) | Go soft memory limit; see Memory | | `GOMEMLIMIT` | `1536MiB` (in the Docker image) | Go soft memory limit; see Memory |
| `MALLOC_ARENA_MAX` | `2` (in the Docker image) | glibc malloc arena cap, a positive whole number; see Memory | | `MALLOC_ARENA_MAX` | `2` (in the Docker image) | glibc malloc arena cap; see Memory |
A variable that is set to an invalid value stops the start with an error and a
non-zero exit. An empty variable counts as unset.
State directory (database location): State directory (database location):
- macOS: `~/Library/Application Support/routewatch/` - macOS: `~/Library/Application Support/routewatch/`
- Linux: `/var/lib/berlin.sneak.app.routewatch/` when running as root, - Linux: `/var/lib/routewatch/` or `~/.local/share/routewatch/`
otherwise `$XDG_DATA_HOME/berlin.sneak.app.routewatch/` (with `XDG_DATA_HOME`
unset, `~/.local/share/berlin.sneak.app.routewatch/`). In the Docker image
this is `/var/lib/berlin.sneak.app.routewatch/`.
## Memory ## Memory
@@ -237,22 +220,6 @@ What happens at each limit:
With `DEBUG=routewatch` the daemon logs a `System stats` line every 60 seconds With `DEBUG=routewatch` the daemon logs a `System stats` line every 60 seconds
with the goroutine count and Go memory figures. with the goroutine count and Go memory figures.
## Running under upaas
What the [upaas](https://git.eeqj.de/sneak/upaas) app needs:
- Container port: `8080`.
- Volume: one, at container path `/var/lib/berlin.sneak.app.routewatch`.
- Environment: nothing is required. Leave `XDG_DATA_HOME`, `GOMEMLIMIT` and
`MALLOC_ARENA_MAX` at the image's values. `DEBUG=routewatch` is optional and
adds the `System stats` memory line to the log.
- Memory Limit: `5g`, the 5 GiB limit from Memory above. upaas sets no swap
limit, so on a host with swap Docker allows the same amount of swap again.
- Health check: the image's `HEALTHCHECK` requests
`/.well-known/healthcheck.json` on the container port. upaas reads the
container's health 60 seconds after a deploy and fails the deploy unless it
is `healthy`.
## Development ## Development
```bash ```bash
@@ -307,8 +274,4 @@ invoke `go test` directly without `-short`:
## License ## License
MIT. See [`LICENSE`](LICENSE). See LICENSE file.
## Author
[@sneak](https://sneak.berlin)
TODO.md
+12 -86
View File
@@ -10,95 +10,19 @@
# Status # Status
pre-1.0. No git tags. The Docker build runs the format check, the linter pre-1.0. No git tags. Runs in production-style Docker deployment, but
and the tests, and the Gitea workflow runs that build on every push. The the policy compliance branch (repo-policies-compliance, make check
image sets memory ceilings for a 5 GiB container (README "Memory") and the passing, clean tree) is unmerged to main and the CI workflow is missing.
README says how to run it under upaas (README "Running under upaas"). A
35-hour run of `3898daa` on the live feed peaked at about 1 GiB, without a
container memory limit.
# Next Step # Next Step
`next` waits for sneak to merge it to `main` through Merge repo-policies-compliance into main (3 commits: policy files and
https://git.eeqj.de/sneak/routewatch/pulls/6. After that, setting .gitignore, Makefile targets fmt-check/check/docker/hooks, gofmt pass),
routewatch up under upaas on fsn1app1 and deploying it are his then add .gitea/workflows/check.yml as a small follow-up commit so CI
(https://git.eeqj.de/sneak/routewatch/issues/31), and so is the run under a runs make check on main.
real 5 GiB limit (https://git.eeqj.de/sneak/routewatch/issues/3).
# Completed Steps # Completed Steps
- 2026-10-03: `/api/v1/stats` serves the prefix distribution from memory,
seeded at startup and adjusted on every live-route write, so a request no
longer reads every live route (closes #30)
- 2026-10-03: a new database is created with `auto_vacuum` set to
incremental, through the connection string so it is set before the file
is first written, and each periodic incremental vacuum now returns up to
1000 free pages; the late `PRAGMA auto_vacuum` in `Initialize`, which
SQLite ignored, is gone (closes #43)
- 2026-10-03: looking up an IP address no longer reads every IPv6 route:
both families find the most specific live route with at most 33 or 129
lookups on the prefix index, and the IPv4 range columns are gone. Prefixes
from the feed are stored in one text form, so an IPv6 withdrawal, which the
feed sends uncompressed, now removes its route (closes #48)
- 2026-10-02: a plain `docker build .` stamps the commit's tag or short
commit (`git describe --tags --always`) into the page footer instead of
`unknown`: `.dockerignore` sends `.git` without `.git/config`, a `VERSION`
build argument takes precedence, and `make build` stamps the same value
(closes #46)
- 2026-09-29: the entrypoint creates the data directory if it is missing and
stops the start if a step fails; README "Running under upaas" no longer
asks for the host directory to be created first (closes #42)
- 2026-09-29: `.dockerignore` keeps `.git`, local build output, local
databases and `.env` out of the Docker build context, and so out of the
source archive in the image (closes #39)
- 2026-09-29: README first line names the MIT license and the author;
the License section now says MIT and links `LICENSE`, and an Author
section was added; this file brought up to date (closes #38)
- 2026-09-29: MIT `LICENSE` (closes #1)
- 2026-09-28: stopping the daemon while the feed is flowing no longer
panics with "send on closed channel": the read loop checks for a stop
just before handing a message to the handler queues, and a second
`Stop` no longer closes the queues again (closes #34)
- 2026-09-28: `docker stop` no longer kills the daemon 2 seconds after the
stop signal: the entrypoint switches to the `routewatch` user with
`setpriv` instead of `runuser`, so the daemon receives the signal itself
and gets the whole wait `docker stop` allows, up to its own 60-second
limit (closes #33)
- 2026-09-28: ready to run under upaas: a set but invalid `PORT`,
`XDG_DATA_HOME` or `MALLOC_ARENA_MAX` stops the start, the health
check follows `PORT`, README "Running under upaas" section (closes
#31)
- 2026-09-22: realtime in-memory database statistics: counts seeded at
startup and adjusted on every write, oldest/newest route timestamps via
index-end lookups; `/api/v1/stats` no longer scans the tables (closes
#27)
- 2026-09-21: batch writes take the write lock when their transaction
begins (`_txlock=immediate`), so they wait out a WAL checkpoint instead
of failing with "database is locked" (closes #25)
- 2026-09-21: `MALLOC_ARENA_MAX=2` in the image caps glibc malloc arenas,
so memory outside the Go runtime no longer grows with the core count
(closes #23)
- 2026-09-21: `GOMEMLIMIT=1536MiB` in the image; README Memory section
with the memory budget and the 5 GiB container limit (closes #13)
- 2026-09-21: the four handler queues hold at most 20,000 messages each,
down from 100,000 (closes #11)
- 2026-09-21: two goroutine leaks fixed: the stats handlers after a
timeout and the streamer's tickers on every reconnect (closes #12)
- 2026-09-21: parsed RIS messages no longer keep the unused `Community`
and `Raw` fields (closes #9)
- 2026-09-21: `.editorconfig`, and a Gitea workflow that runs
`script/cibuild` on every push (closes #14)
- 2026-09-21: the peering handler's AS-path map holds at most 500,000
paths and is swapped for an empty one every 30 seconds instead of copied
(closes #10)
- 2026-09-21: SQLite memory bounded across the whole connection pool: a
64 MiB page cache on each connection, 1 GiB soft and 1.5 GiB hard heap
limits (closes #8)
- 2026-09-21: the Docker build runs the format check, the linter and the
tests, linting in a separate stage on a golangci-lint image pinned by
digest (closes #5)
- 2026-09-21: `make test` skips the live-network feed test (`-short`), so
`make check` no longer depends on the network (closes #2)
- 2026-07-07 Adopted scripts-to-rule-them-all: `script/` entrypoints, - 2026-07-07 Adopted scripts-to-rule-them-all: `script/` entrypoints,
Makefile shims, README Entrypoints section Makefile shims, README Entrypoints section
- 2026-02-22: repo policy compliance: required policy files, .gitignore - 2026-02-22: repo policy compliance: required policy files, .gitignore
@@ -117,6 +41,8 @@ real 5 GiB limit (https://git.eeqj.de/sneak/routewatch/issues/3).
# Future Steps # Future Steps
- Production memory under 5 GiB: whether to test under a real 5 GiB - Verify main is green after the merge: make check locally and the new
container limit on fsn1app1 is open for sneak CI workflow passing
(https://git.eeqj.de/sneak/routewatch/issues/3) - Review stale remote branches fix-min-time-calculation and
optimize-sqlite-settings: land or delete
- Clean up the tmp/ directory at the repo root: gitignore or remove
entrypoint.sh
+1 -14
View File
@@ -1,20 +1,7 @@
#!/bin/bash #!/bin/bash
set -euo pipefail
# glibc silently ignores a malformed MALLOC_ARENA_MAX, so refuse it here.
if [[ -n "${MALLOC_ARENA_MAX:-}" && ! "$MALLOC_ARENA_MAX" =~ ^[1-9][0-9]*$ ]]; then
echo "MALLOC_ARENA_MAX must be a positive whole number, got '$MALLOC_ARENA_MAX'" >&2
exit 1
fi
# Give the data directory to the routewatch user before the daemon starts,
# whether it is missing, an empty root-owned mount, or holds another uid's files.
mkdir -p /var/lib/berlin.sneak.app.routewatch
cd /var/lib/berlin.sneak.app.routewatch cd /var/lib/berlin.sneak.app.routewatch
chown -R routewatch:routewatch . chown -R routewatch:routewatch .
chmod 700 . chmod 700 .
# setpriv replaces itself with the daemon, so the daemon receives the stop exec runuser -u routewatch -- /app/routewatch
# signal directly. runuser would stay in between and kill the daemon 2 seconds
# after passing the signal on.
exec setpriv --reuid=routewatch --regid=routewatch --init-groups -- /app/routewatch
internal/config/config.go
+1 -40
View File
@@ -6,7 +6,6 @@ import (
"os" "os"
"path/filepath" "path/filepath"
"runtime" "runtime"
"strconv"
"time" "time"
) )
@@ -19,12 +18,6 @@ const (
// defaultRouteExpirationMinutes is the default route expiration timeout in minutes // defaultRouteExpirationMinutes is the default route expiration timeout in minutes
defaultRouteExpirationMinutes = 5 defaultRouteExpirationMinutes = 5
// defaultPort is the HTTP port used when PORT is not set
defaultPort = 8080
// maxPort is the highest TCP port number
maxPort = 65535
) )
// Config holds configuration for the entire application // Config holds configuration for the entire application
@@ -32,9 +25,6 @@ type Config struct {
// StateDir is the directory for all application state (database, snapshots) // StateDir is the directory for all application state (database, snapshots)
StateDir string StateDir string
// Port is the TCP port the HTTP server listens on
Port int
// MaxRuntime is the maximum runtime (0 = run forever) // MaxRuntime is the maximum runtime (0 = run forever)
MaxRuntime time.Duration MaxRuntime time.Duration
@@ -53,14 +43,8 @@ func New() (*Config, error) {
return nil, fmt.Errorf("failed to determine state directory: %w", err) return nil, fmt.Errorf("failed to determine state directory: %w", err)
} }
port, err := getPort()
if err != nil {
return nil, err
}
return &Config{ return &Config{
StateDir: stateDir, StateDir: stateDir,
Port: port,
MaxRuntime: 0, // Run forever by default MaxRuntime: 0, // Run forever by default
EnableBatchedDatabaseWrites: true, // Enable batching by default EnableBatchedDatabaseWrites: true, // Enable batching by default
RouteExpirationTimeout: defaultRouteExpirationMinutes * time.Minute, // For active route monitoring RouteExpirationTimeout: defaultRouteExpirationMinutes * time.Minute, // For active route monitoring
@@ -85,20 +69,13 @@ func getStateDirectory() (string, error) {
return filepath.Join(home, "Library", "Application Support", AppIdentifier), nil return filepath.Join(home, "Library", "Application Support", AppIdentifier), nil
case "linux", "freebsd", "openbsd", "netbsd": case "linux", "freebsd", "openbsd", "netbsd":
// The XDG spec requires an absolute path; a relative one would put
// the database somewhere unexpected.
xdgData := os.Getenv("XDG_DATA_HOME")
if xdgData != "" && !filepath.IsAbs(xdgData) {
return "", fmt.Errorf("XDG_DATA_HOME must be an absolute path, got %q", xdgData)
}
// Unix-like: /var/lib/berlin.sneak.app.routewatch if root, else XDG_DATA_HOME // Unix-like: /var/lib/berlin.sneak.app.routewatch if root, else XDG_DATA_HOME
if os.Geteuid() == 0 { if os.Geteuid() == 0 {
return filepath.Join("/var/lib", AppIdentifier), nil return filepath.Join("/var/lib", AppIdentifier), nil
} }
// Check XDG_DATA_HOME first // Check XDG_DATA_HOME first
if xdgData != "" { if xdgData := os.Getenv("XDG_DATA_HOME"); xdgData != "" {
return filepath.Join(xdgData, AppIdentifier), nil return filepath.Join(xdgData, AppIdentifier), nil
} }
@@ -115,22 +92,6 @@ func getStateDirectory() (string, error) {
} }
} }
// getPort returns the HTTP port from PORT, or defaultPort when PORT is not set
func getPort() (int, error) {
value := os.Getenv("PORT")
if value == "" {
return defaultPort, nil
}
// ParseUint, unlike Atoi, refuses a sign: the health check URL cannot use "+9090"
port, err := strconv.ParseUint(value, 10, 0)
if err != nil || port < 1 || port > maxPort {
return 0, fmt.Errorf("PORT must be a whole number from 1 to %d, got %q", maxPort, value)
}
return int(port), nil
}
// EnsureDirectories creates all necessary directories if they don't exist // EnsureDirectories creates all necessary directories if they don't exist
func (c *Config) EnsureDirectories() error { func (c *Config) EnsureDirectories() error {
// Ensure state directory exists // Ensure state directory exists
internal/config/config_test.go
-62
View File
@@ -1,62 +0,0 @@
package config
import (
"runtime"
"testing"
)
func TestNewReadsPort(t *testing.T) {
tests := map[string]int{
"": defaultPort,
"1": 1,
"9090": 9090,
"65535": 65535,
}
for value, want := range tests {
t.Run(value, func(t *testing.T) {
t.Setenv("PORT", value)
t.Setenv("XDG_DATA_HOME", "")
cfg, err := New()
if err != nil {
t.Fatalf("New() with PORT=%q: %v", value, err)
}
if cfg.Port != want {
t.Errorf("New() with PORT=%q: Port = %d, want %d", value, cfg.Port, want)
}
})
}
}
func TestNewRefusesInvalidPort(t *testing.T) {
for _, value := range []string{"0", "65536", "-1", "+9090", "http", "80.5"} {
t.Run(value, func(t *testing.T) {
t.Setenv("PORT", value)
t.Setenv("XDG_DATA_HOME", "")
if _, err := New(); err == nil {
t.Errorf("New() with PORT=%q returned no error", value)
}
})
}
}
func TestNewRefusesRelativeXDGDataHome(t *testing.T) {
if runtime.GOOS == "darwin" {
t.Skip("macOS does not read XDG_DATA_HOME")
}
t.Setenv("PORT", "")
t.Setenv("XDG_DATA_HOME", "relative/path")
if _, err := New(); err == nil {
t.Error("New() with a relative XDG_DATA_HOME returned no error")
}
t.Setenv("XDG_DATA_HOME", "/var/lib")
if _, err := New(); err != nil {
t.Errorf("New() with XDG_DATA_HOME=/var/lib: %v", err)
}
}
internal/database/counts.go
-203
View File
@@ -1,203 +0,0 @@
package database
import (
"context"
"fmt"
"sync"
)
// liveCounts holds the running row counts and the prefix distribution that the
// stats endpoints report. They are seeded once at startup from the tables and
// then adjusted on every write, so a stats read serves them from memory instead
// of running a query over the tables. The COUNT(*) scans (issue 27) and then
// the prefix distribution query (issue 30) each grew with the database until
// they took the whole request timeout and made /api/v1/stats return 500.
//
// A single mutex guards all fields so the stats reader takes a consistent
// snapshot at one instant and writers, which already run under the database
// write lock, adjust the counts after their transaction commits.
type liveCounts struct {
mu sync.RWMutex
asns int
prefixesV4 int
prefixesV6 int
peerings int
peers int
routesV4 int
routesV6 int
// The prefix distribution: for each mask length, the number of distinct
// prefixes that have at least one live route.
distributionV4 [ipv4Bits + 1]int
distributionV6 [ipv6Bits + 1]int
}
// seed sets every count to the value read from the tables at startup. It runs
// before any writer, so it needs no coordination with the adjust methods.
func (c *liveCounts) seed(asns, prefixesV4, prefixesV6, peerings, peers, routesV4, routesV6 int,
distributionV4, distributionV6 []PrefixDistribution) {
c.mu.Lock()
defer c.mu.Unlock()
c.asns = asns
c.prefixesV4 = prefixesV4
c.prefixesV6 = prefixesV6
c.peerings = peerings
c.peers = peers
c.routesV4 = routesV4
c.routesV6 = routesV6
for _, entry := range distributionV4 {
addAtMaskLength(c.distributionV4[:], entry.MaskLength, entry.Count)
}
for _, entry := range distributionV6 {
addAtMaskLength(c.distributionV6[:], entry.MaskLength, entry.Count)
}
}
// addASNs adds n to the ASN count.
func (c *liveCounts) addASNs(n int) {
c.mu.Lock()
c.asns += n
c.mu.Unlock()
}
// addPrefixes adds to the IPv4 and IPv6 prefix counts.
func (c *liveCounts) addPrefixes(v4, v6 int) {
c.mu.Lock()
c.prefixesV4 += v4
c.prefixesV6 += v6
c.mu.Unlock()
}
// addPeerings adds n to the peering count.
func (c *liveCounts) addPeerings(n int) {
c.mu.Lock()
c.peerings += n
c.mu.Unlock()
}
// addPeers adds n to the BGP peer count.
func (c *liveCounts) addPeers(n int) {
c.mu.Lock()
c.peers += n
c.mu.Unlock()
}
// addRoutes adds to the IPv4 and IPv6 live-route counts. Deletions pass
// negative values.
func (c *liveCounts) addRoutes(v4, v6 int) {
c.mu.Lock()
c.routesV4 += v4
c.routesV6 += v6
c.mu.Unlock()
}
// addToDistribution adds n to the IPv4 and IPv6 prefix distributions once for
// each listed mask length. A write lists the mask lengths of the prefixes it
// gave their first live route with n = 1, and of the prefixes it left with no
// live route with n = -1.
func (c *liveCounts) addToDistribution(maskLengthsV4, maskLengthsV6 []int, n int) {
c.mu.Lock()
defer c.mu.Unlock()
for _, maskLength := range maskLengthsV4 {
addAtMaskLength(c.distributionV4[:], maskLength, n)
}
for _, maskLength := range maskLengthsV6 {
addAtMaskLength(c.distributionV6[:], maskLength, n)
}
}
// addAtMaskLength adds n to counts[maskLength]. A mask length the array has no
// entry for is ignored, so a malformed route cannot crash the daemon.
func addAtMaskLength(counts []int, maskLength, n int) {
if maskLength >= 0 && maskLength < len(counts) {
counts[maskLength] += n
}
}
// fill copies the counts into a Stats, including the derived totals, under a
// single read lock so the reader sees one consistent snapshot.
func (c *liveCounts) fill(s *Stats) {
c.mu.RLock()
defer c.mu.RUnlock()
s.ASNs = c.asns
s.IPv4Prefixes = c.prefixesV4
s.IPv6Prefixes = c.prefixesV6
s.Prefixes = c.prefixesV4 + c.prefixesV6
s.Peerings = c.peerings
s.Peers = c.peers
s.IPv4Routes = c.routesV4
s.IPv6Routes = c.routesV6
s.LiveRoutes = c.routesV4 + c.routesV6
s.IPv4PrefixDistribution = distributionList(c.distributionV4[:])
s.IPv6PrefixDistribution = distributionList(c.distributionV6[:])
}
// distributionList lists the mask lengths that have at least one prefix, in
// ascending order, the way the distribution query returns them.
func distributionList(counts []int) []PrefixDistribution {
var list []PrefixDistribution
for maskLength, count := range counts {
if count > 0 {
list = append(list, PrefixDistribution{MaskLength: maskLength, Count: count})
}
}
return list
}
// countRows returns the number of rows in the named table. It is used only at
// startup to seed the in-memory counters, so a full COUNT(*) is acceptable.
func (d *Database) countRows(ctx context.Context, table string) (int, error) {
var n int
// table is one of a fixed set of literals below, never external input.
if err := d.db.QueryRowContext(ctx, "SELECT COUNT(*) FROM "+table).Scan(&n); err != nil {
return 0, fmt.Errorf("failed to count %s: %w", table, err)
}
return n, nil
}
// seedCounts reads the current row counts and prefix distribution from the
// tables into the in-memory counters. It runs once at startup, before the
// streamer begins writing.
func (d *Database) seedCounts(ctx context.Context) error {
asns, err := d.countRows(ctx, "asns")
if err != nil {
return err
}
prefixesV4, err := d.countRows(ctx, "prefixes_v4")
if err != nil {
return err
}
prefixesV6, err := d.countRows(ctx, "prefixes_v6")
if err != nil {
return err
}
peerings, err := d.countRows(ctx, "peerings")
if err != nil {
return err
}
peers, err := d.countRows(ctx, "bgp_peers")
if err != nil {
return err
}
routesV4, err := d.countRows(ctx, "live_routes_v4")
if err != nil {
return err
}
routesV6, err := d.countRows(ctx, "live_routes_v6")
if err != nil {
return err
}
distributionV4, distributionV6, err := d.GetPrefixDistributionContext(ctx)
if err != nil {
return err
}
d.counts.seed(asns, prefixesV4, prefixesV6, peerings, peers, routesV4, routesV6,
distributionV4, distributionV6)
return nil
}
internal/database/counts_test.go
-511
View File
@@ -1,511 +0,0 @@
package database
import (
"context"
"slices"
"sync"
"testing"
"time"
"git.eeqj.de/sneak/routewatch/internal/config"
"git.eeqj.de/sneak/routewatch/internal/logger"
"github.com/google/uuid"
)
// mkV4Route builds an IPv4 live route with its mask length taken from the
// prefix.
func mkV4Route(t *testing.T, prefix string, asn int, ts time.Time) *LiveRoute {
t.Helper()
maskLength, err := prefixMaskLength(prefix)
if err != nil {
t.Fatalf("prefixMaskLength(%s): %v", prefix, err)
}
return &LiveRoute{
ID: uuid.New(),
Prefix: prefix,
MaskLength: maskLength,
IPVersion: ipVersionV4,
OriginASN: asn,
PeerIP: "192.0.2.1",
ASPath: []int{asn},
NextHop: "192.0.2.254",
LastUpdated: ts,
}
}
// mkV6Route builds an IPv6 live route.
func mkV6Route(prefix string, asn int, ts time.Time) *LiveRoute {
return &LiveRoute{
ID: uuid.New(),
Prefix: prefix,
MaskLength: 32,
IPVersion: ipVersionV6,
OriginASN: asn,
PeerIP: "2001:db8::1",
ASPath: []int{asn},
NextHop: "2001:db8::ffff",
LastUpdated: ts,
}
}
// TestLiveCountsTrackWritesInRealtime checks that the stats counts start at
// zero, reflect each write the moment it commits (no recompute, no timer), do
// not move when a route is merely re-announced, and drop when a route is
// deleted. These counts are what /api/v1/stats reports; before this change the
// endpoint recomputed them with a COUNT(*) over each table on every request.
func TestLiveCountsTrackWritesInRealtime(t *testing.T) {
cfg := &config.Config{StateDir: t.TempDir()}
db, err := New(cfg, logger.New())
if err != nil {
t.Fatalf("failed to create database: %v", err)
}
defer func() { _ = db.Close() }()
ctx := context.Background()
empty, err := db.GetStatsContext(ctx)
if err != nil {
t.Fatalf("GetStatsContext on empty database: %v", err)
}
if empty.ASNs != 0 || empty.Prefixes != 0 || empty.Peerings != 0 ||
empty.Peers != 0 || empty.LiveRoutes != 0 {
t.Fatalf("empty database counts nonzero: %+v", empty)
}
ts := time.Date(2026, 1, 2, 3, 4, 5, 0, time.UTC)
if err := db.GetOrCreateASNBatch(map[int]time.Time{64500: ts, 64501: ts}); err != nil {
t.Fatalf("GetOrCreateASNBatch: %v", err)
}
if err := db.UpdatePrefixesBatch(map[string]time.Time{
"198.51.100.0/24": ts,
"2001:db8::/32": ts,
}); err != nil {
t.Fatalf("UpdatePrefixesBatch: %v", err)
}
if err := db.UpdatePeerBatch(map[string]PeerUpdate{
"192.0.2.1": {PeerIP: "192.0.2.1", PeerASN: 64500, MessageType: "UPDATE", Timestamp: ts},
}); err != nil {
t.Fatalf("UpdatePeerBatch: %v", err)
}
if err := db.RecordPeering(64500, 64501, ts); err != nil {
t.Fatalf("RecordPeering: %v", err)
}
routes := []*LiveRoute{
mkV4Route(t, "198.51.100.0/24", 64500, ts),
mkV4Route(t, "203.0.113.0/24", 64501, ts.Add(time.Minute)),
mkV6Route("2001:db8::/32", 64502, ts.Add(2*time.Minute)),
}
if err := db.UpsertLiveRouteBatch(routes); err != nil {
t.Fatalf("UpsertLiveRouteBatch: %v", err)
}
stats, err := db.GetStatsContext(ctx)
if err != nil {
t.Fatalf("GetStatsContext: %v", err)
}
assertCounts(t, "after inserts", stats, wantCounts{
asns: 2, prefixes: 2, peerings: 1, peers: 1,
ipv4Routes: 2, ipv6Routes: 1, liveRoutes: 3,
})
// Re-announcing the same routes is an update, not an insert: counts hold.
if err := db.UpsertLiveRouteBatch(routes); err != nil {
t.Fatalf("UpsertLiveRouteBatch (re-announce): %v", err)
}
stats, err = db.GetStatsContext(ctx)
if err != nil {
t.Fatalf("GetStatsContext: %v", err)
}
assertCounts(t, "after re-announce", stats, wantCounts{
asns: 2, prefixes: 2, peerings: 1, peers: 1,
ipv4Routes: 2, ipv6Routes: 1, liveRoutes: 3,
})
// A withdrawal removes one route.
if err := db.DeleteLiveRouteBatch([]LiveRouteDeletion{
{Prefix: "203.0.113.0/24", OriginASN: 64501, PeerIP: "192.0.2.1", IPVersion: ipVersionV4},
}); err != nil {
t.Fatalf("DeleteLiveRouteBatch: %v", err)
}
stats, err = db.GetStatsContext(ctx)
if err != nil {
t.Fatalf("GetStatsContext: %v", err)
}
assertCounts(t, "after delete", stats, wantCounts{
asns: 2, prefixes: 2, peerings: 1, peers: 1,
ipv4Routes: 1, ipv6Routes: 1, liveRoutes: 2,
})
}
// TestLiveCountsSeededFromDatabaseAtStartup writes rows, reopens the same
// database file, and checks the counts and the prefix distribution come back
// from the seed scan rather than starting at zero.
func TestLiveCountsSeededFromDatabaseAtStartup(t *testing.T) {
cfg := &config.Config{StateDir: t.TempDir()}
db, err := New(cfg, logger.New())
if err != nil {
t.Fatalf("failed to create database: %v", err)
}
ts := time.Date(2026, 1, 2, 3, 4, 5, 0, time.UTC)
if err := db.GetOrCreateASNBatch(map[int]time.Time{64500: ts, 64501: ts, 64502: ts}); err != nil {
t.Fatalf("GetOrCreateASNBatch: %v", err)
}
if err := db.UpsertLiveRouteBatch([]*LiveRoute{
mkV4Route(t, "198.51.100.0/24", 64500, ts),
mkV6Route("2001:db8::/32", 64502, ts),
}); err != nil {
t.Fatalf("UpsertLiveRouteBatch: %v", err)
}
if err := db.Close(); err != nil {
t.Fatalf("Close: %v", err)
}
reopened, err := New(cfg, logger.New())
if err != nil {
t.Fatalf("failed to reopen database: %v", err)
}
defer func() { _ = reopened.Close() }()
stats, err := reopened.GetStatsContext(context.Background())
if err != nil {
t.Fatalf("GetStatsContext after reopen: %v", err)
}
if stats.ASNs != 3 {
t.Errorf("seeded ASNs = %d, want 3", stats.ASNs)
}
if stats.IPv4Routes != 1 || stats.IPv6Routes != 1 || stats.LiveRoutes != 2 {
t.Errorf("seeded routes = (v4 %d, v6 %d, total %d), want (1, 1, 2)",
stats.IPv4Routes, stats.IPv6Routes, stats.LiveRoutes)
}
assertDistribution(t, "seeded IPv4 distribution", stats.IPv4PrefixDistribution,
[]PrefixDistribution{{MaskLength: 24, Count: 1}})
assertDistribution(t, "seeded IPv6 distribution", stats.IPv6PrefixDistribution,
[]PrefixDistribution{{MaskLength: 32, Count: 1}})
}
// TestPrefixDistributionTracksWrites checks that the prefix distribution the
// stats read reports stays exact across each kind of live-route write, and that
// after every step it equals what the distribution query reads from the route
// tables. The steps run once through the batch methods the prefix handler uses
// and once through the single-route methods.
func TestPrefixDistributionTracksWrites(t *testing.T) {
ts := time.Date(2026, 1, 2, 3, 4, 5, 0, time.UTC)
shared := mkV4Route(t, "198.51.100.0/24", 64500, ts)
sharedSecondPeer := mkV4Route(t, "198.51.100.0/24", 64500, ts)
sharedSecondPeer.PeerIP = "192.0.2.2"
other := mkV4Route(t, "203.0.113.0/24", 64501, ts)
wide := mkV4Route(t, "172.16.0.0/16", 64502, ts)
v6 := mkV6Route("2001:db8::/32", 64503, ts)
v6SecondPeer := mkV6Route("2001:db8::/32", 64503, ts)
v6SecondPeer.PeerIP = "2001:db8::2"
all := []PrefixDistribution{{MaskLength: 16, Count: 1}, {MaskLength: 24, Count: 2}}
v6Only := []PrefixDistribution{{MaskLength: 32, Count: 1}}
steps := []struct {
name string
announce []*LiveRoute
withdraw []*LiveRoute
wantV4 []PrefixDistribution
wantV6 []PrefixDistribution
}{
{
name: "new routes",
announce: []*LiveRoute{shared, other, wide, v6},
wantV4: all, wantV6: v6Only,
},
{
name: "re-announcement",
announce: []*LiveRoute{shared, other, wide, v6},
wantV4: all, wantV6: v6Only,
},
{
name: "second peer announces a prefix that has a live route",
announce: []*LiveRoute{sharedSecondPeer, v6SecondPeer},
wantV4: all, wantV6: v6Only,
},
{
name: "withdrawal of a route that is not the last for its prefix",
withdraw: []*LiveRoute{shared, v6},
wantV4: all, wantV6: v6Only,
},
{
name: "withdrawal of the last route for a prefix",
withdraw: []*LiveRoute{sharedSecondPeer, v6SecondPeer},
wantV4: []PrefixDistribution{{MaskLength: 16, Count: 1}, {MaskLength: 24, Count: 1}},
},
{
name: "withdrawal of every remaining route",
withdraw: []*LiveRoute{other, wide},
},
{
name: "two peers announce a new prefix together",
announce: []*LiveRoute{shared, sharedSecondPeer, v6, v6SecondPeer},
wantV4: []PrefixDistribution{{MaskLength: 24, Count: 1}},
wantV6: v6Only,
},
{
name: "both routes for a prefix withdrawn together",
withdraw: []*LiveRoute{shared, sharedSecondPeer, v6, v6SecondPeer},
},
// The feed often withdraws a route that is not live. That must not take
// the prefix out of the distribution. A count wrongly taken below zero is
// left out of the answer, so the next step shows it: its announcement
// at the same mask length would then not be counted.
{
name: "withdrawal of routes that are not live",
withdraw: []*LiveRoute{other, v6},
},
{
name: "announcement after a withdrawal of routes that are not live",
announce: []*LiveRoute{other, v6},
wantV4: []PrefixDistribution{{MaskLength: 24, Count: 1}},
wantV6: v6Only,
},
}
for _, batch := range []bool{true, false} {
name := "single-route writes"
if batch {
name = "batch writes"
}
t.Run(name, func(t *testing.T) {
db, err := New(&config.Config{StateDir: t.TempDir()}, logger.New())
if err != nil {
t.Fatalf("failed to create database: %v", err)
}
defer func() { _ = db.Close() }()
ctx := context.Background()
for _, step := range steps {
if err := announceRoutes(db, batch, step.announce); err != nil {
t.Fatalf("%s: announce: %v", step.name, err)
}
if err := withdrawRoutes(db, batch, step.withdraw); err != nil {
t.Fatalf("%s: withdraw: %v", step.name, err)
}
stats, err := db.GetStatsContext(ctx)
if err != nil {
t.Fatalf("%s: GetStatsContext: %v", step.name, err)
}
queryV4, queryV6, err := db.GetPrefixDistributionContext(ctx)
if err != nil {
t.Fatalf("%s: GetPrefixDistributionContext: %v", step.name, err)
}
assertDistribution(t, step.name+": IPv4 distribution", stats.IPv4PrefixDistribution, step.wantV4)
assertDistribution(t, step.name+": IPv6 distribution", stats.IPv6PrefixDistribution, step.wantV6)
assertDistribution(t, step.name+": IPv4 distribution query", queryV4, step.wantV4)
assertDistribution(t, step.name+": IPv6 distribution query", queryV6, step.wantV6)
}
})
}
}
// announceRoutes writes routes in one UpsertLiveRouteBatch, or with one
// UpsertLiveRoute each.
func announceRoutes(db *Database, batch bool, routes []*LiveRoute) error {
if batch {
return db.UpsertLiveRouteBatch(routes)
}
for _, route := range routes {
if err := db.UpsertLiveRoute(route); err != nil {
return err
}
}
return nil
}
// withdrawRoutes removes routes in one DeleteLiveRouteBatch, or with one
// DeleteLiveRoute each. It names each route by prefix and peer only, with no
// origin ASN, as a withdrawal from the feed does when its message carries no AS
// path.
func withdrawRoutes(db *Database, batch bool, routes []*LiveRoute) error {
if !batch {
for _, route := range routes {
if err := db.DeleteLiveRoute(route.Prefix, 0, route.PeerIP); err != nil {
return err
}
}
return nil
}
deletions := make([]LiveRouteDeletion, 0, len(routes))
for _, route := range routes {
deletions = append(deletions, LiveRouteDeletion{
Prefix: route.Prefix, PeerIP: route.PeerIP, IPVersion: route.IPVersion,
})
}
return db.DeleteLiveRouteBatch(deletions)
}
// TestStatsRouteTimestamps checks the oldest/newest route timestamps are read
// from the right rows across both tables and parse into time.Time. The old
// MIN/MAX union query read its result into *time.Time, which the driver could
// not parse, so it logged a warning every call and left both timestamps nil.
func TestStatsRouteTimestamps(t *testing.T) {
cfg := &config.Config{StateDir: t.TempDir()}
db, err := New(cfg, logger.New())
if err != nil {
t.Fatalf("failed to create database: %v", err)
}
defer func() { _ = db.Close() }()
ctx := context.Background()
empty, err := db.GetStatsContext(ctx)
if err != nil {
t.Fatalf("GetStatsContext on empty database: %v", err)
}
if empty.OldestRoute != nil || empty.NewestRoute != nil {
t.Fatalf("empty database timestamps = (%v, %v), want (nil, nil)",
empty.OldestRoute, empty.NewestRoute)
}
base := time.Date(2026, 1, 2, 3, 4, 5, 0, time.UTC)
oldest := base
newest := base.Add(2 * time.Minute)
if err := db.UpsertLiveRouteBatch([]*LiveRoute{
mkV4Route(t, "198.51.100.0/24", 64500, base.Add(time.Minute)),
mkV4Route(t, "203.0.113.0/24", 64501, oldest),
mkV6Route("2001:db8::/32", 64502, newest),
}); err != nil {
t.Fatalf("UpsertLiveRouteBatch: %v", err)
}
stats, err := db.GetStatsContext(ctx)
if err != nil {
t.Fatalf("GetStatsContext: %v", err)
}
if stats.OldestRoute == nil || !stats.OldestRoute.Equal(oldest) {
t.Errorf("OldestRoute = %v, want %v", stats.OldestRoute, oldest)
}
if stats.NewestRoute == nil || !stats.NewestRoute.Equal(newest) {
t.Errorf("NewestRoute = %v, want %v", stats.NewestRoute, newest)
}
}
// TestLiveCountsConcurrentReadWrite runs writers and stats readers at once so
// the race detector proves the counters are safe under concurrent use.
func TestLiveCountsConcurrentReadWrite(t *testing.T) {
cfg := &config.Config{StateDir: t.TempDir()}
db, err := New(cfg, logger.New())
if err != nil {
t.Fatalf("failed to create database: %v", err)
}
defer func() { _ = db.Close() }()
ts := time.Date(2026, 1, 2, 3, 4, 5, 0, time.UTC)
const writers = 4
var wg sync.WaitGroup
wg.Add(writers)
for w := range writers {
go func(base int) {
defer wg.Done()
for i := range 25 {
asn := 65000 + base*100 + i
route := mkV6Route("2001:db8::/32", asn, ts)
route.PeerIP = "2001:db8::" + uuid.NewString()
if err := db.UpsertLiveRoute(route); err != nil {
t.Errorf("UpsertLiveRoute: %v", err)
return
}
}
}(w)
}
var readerWG sync.WaitGroup
readerWG.Add(1)
stop := make(chan struct{})
go func() {
defer readerWG.Done()
for {
select {
case <-stop:
return
default:
if _, err := db.GetStatsContext(context.Background()); err != nil {
t.Errorf("GetStatsContext: %v", err)
return
}
}
}
}()
wg.Wait()
close(stop)
readerWG.Wait()
stats, err := db.GetStatsContext(context.Background())
if err != nil {
t.Fatalf("GetStatsContext: %v", err)
}
if want := writers * 25; stats.IPv6Routes != want {
t.Errorf("IPv6Routes = %d, want %d", stats.IPv6Routes, want)
}
// Every writer announced the same prefix, so it counts once.
assertDistribution(t, "IPv6 distribution", stats.IPv6PrefixDistribution,
[]PrefixDistribution{{MaskLength: 32, Count: 1}})
}
type wantCounts struct {
asns int
prefixes int
peerings int
peers int
ipv4Routes int
ipv6Routes int
liveRoutes int
}
func assertCounts(t *testing.T, when string, got Stats, want wantCounts) {
t.Helper()
if got.ASNs != want.asns {
t.Errorf("%s: ASNs = %d, want %d", when, got.ASNs, want.asns)
}
if got.Prefixes != want.prefixes {
t.Errorf("%s: Prefixes = %d, want %d", when, got.Prefixes, want.prefixes)
}
if got.Peerings != want.peerings {
t.Errorf("%s: Peerings = %d, want %d", when, got.Peerings, want.peerings)
}
if got.Peers != want.peers {
t.Errorf("%s: Peers = %d, want %d", when, got.Peers, want.peers)
}
if got.IPv4Routes != want.ipv4Routes {
t.Errorf("%s: IPv4Routes = %d, want %d", when, got.IPv4Routes, want.ipv4Routes)
}
if got.IPv6Routes != want.ipv6Routes {
t.Errorf("%s: IPv6Routes = %d, want %d", when, got.IPv6Routes, want.ipv6Routes)
}
if got.LiveRoutes != want.liveRoutes {
t.Errorf("%s: LiveRoutes = %d, want %d", when, got.LiveRoutes, want.liveRoutes)
}
}
func assertDistribution(t *testing.T, what string, got, want []PrefixDistribution) {
t.Helper()
if !slices.Equal(got, want) {
t.Errorf("%s = %v, want %v", what, got, want)
}
}
internal/database/database.go
+544 -656
View File
File diff suppressed because it is too large Load Diff
internal/database/database_test.go
+358 -510
View File
@@ -3,10 +3,7 @@ package database
import ( import (
"context" "context"
"database/sql" "database/sql"
"errors" "net"
"fmt"
"net/netip"
"slices"
"sync" "sync"
"testing" "testing"
"time" "time"
@@ -20,20 +17,10 @@ import (
// memory"; the DSN change must leave temp_store below this so they spill to disk. // memory"; the DSN change must leave temp_store below this so they spill to disk.
const tempStoreMemory = 2 const tempStoreMemory = 2
// autoVacuumIncremental is the PRAGMA auto_vacuum value meaning "incremental".
const autoVacuumIncremental = 2
// vacuumTestRoutes is how many routes the vacuum test writes and then deletes,
// enough to leave many free pages in the file.
const vacuumTestRoutes = 2000
// heldConnections is how many pooled connections the pragma test holds open at // heldConnections is how many pooled connections the pragma test holds open at
// once so each is a distinct SQLite connection that parsed the DSN. // once so each is a distinct SQLite connection that parsed the DSN.
const heldConnections = 5 const heldConnections = 5
// testPeerIP is the peer every route in the IP lookup test is learned from.
const testPeerIP = "192.0.2.254"
// Parameters for the checkpoint-contention regression test. // Parameters for the checkpoint-contention regression test.
const ( const (
// contentionIterations is how many batch writes race the checkpoint loop. // contentionIterations is how many batch writes race the checkpoint loop.
@@ -46,196 +33,286 @@ const (
asnSecondBand = 100 asnSecondBand = 100
) )
// Parameters for the prefix-length sample tests. func TestIPToUint32(t *testing.T) {
const ( tests := []struct {
// sampleLimit is how many prefixes the tests ask the sample for. name string
sampleLimit = 10 ip string
// sampledPrefixCount is how many prefixes the sampling test stores, more expected uint32
// than sampleLimit so the sample has to choose among them. }{
sampledPrefixCount = 30 {
// secondV4PeerIP and secondV6PeerIP are the second peer each prefix is name: "Simple IP",
// learned from, besides the one mkV4Route and mkV6Route set. ip: "192.168.1.1",
secondV4PeerIP = "192.0.2.2" expected: 3232235777, // 192<<24 + 168<<16 + 1<<8 + 1
secondV6PeerIP = "2001:db8::2" },
// largeSamplePrefixCount is how many prefixes the test of a mask length {
// with more than maxPrefixesToShuffle prefixes stores: twice that, so about name: "Minimum IP",
// half of them come after the first maxPrefixesToShuffle+1 in text order. ip: "0.0.0.0",
largeSamplePrefixCount = 2 * maxPrefixesToShuffle expected: 0,
// largeSampleLimit is how many prefixes that test asks for. Each random },
// lookup lands after the first maxPrefixesToShuffle+1 in text order about {
// half the time, so none of them doing so is about as likely as name: "Maximum IP",
// largeSampleLimit coin tosses all coming up heads. ip: "255.255.255.255",
largeSampleLimit = 30 expected: 4294967295,
// liveRoutesV4Table and liveRoutesV6Table are the live route tables the },
// tests pass to the sampling steps. {
liveRoutesV4Table = "live_routes_v4" name: "10.0.0.0",
liveRoutesV6Table = "live_routes_v6" ip: "10.0.0.0",
) expected: 167772160,
},
// TestGetIPInfoFindsMostSpecificLiveRoute stores nested live prefixes for both {
// families and checks that a lookup returns the most specific one covering the name: "172.16.0.0",
// address, ErrNoRoute when none covers it, and the next less specific prefix ip: "172.16.0.0",
// once the only route of the most specific one is withdrawn. expected: 2886729728,
func TestGetIPInfoFindsMostSpecificLiveRoute(t *testing.T) { },
cfg := &config.Config{StateDir: t.TempDir()} {
name: "8.8.8.8",
db, err := New(cfg, logger.New()) ip: "8.8.8.8",
if err != nil { expected: 134744072,
t.Fatalf("failed to create database: %v", err) },
{
name: "1.2.3.4",
ip: "1.2.3.4",
expected: 16909060,
},
} }
defer func() { _ = db.Close() }()
// Nested live prefixes, each originated by its own AS. for _, tt := range tests {
origins := map[string]int{ t.Run(tt.name, func(t *testing.T) {
"10.0.0.0/8": 64500, ip := net.ParseIP(tt.ip)
"10.1.0.0/16": 64501, if ip == nil {
"10.1.2.0/24": 64502, t.Fatalf("Failed to parse IP: %s", tt.ip)
"2001:db8::/32": 64500, }
"2001:db8:1::/48": 64501,
"2001:db8:1:2::/64": 64502, result := ipToUint32(ip)
if result != tt.expected {
t.Errorf("ipToUint32(%s) = %d, want %d", tt.ip, result, tt.expected)
}
// Test with IPv4-mapped IPv6 address
ip6 := net.ParseIP(tt.ip).To16()
if ip6 != nil {
result6 := ipToUint32(ip6)
if result6 != tt.expected {
t.Errorf("ipToUint32(%s as IPv6) = %d, want %d", tt.ip, result6, tt.expected)
}
} }
ts := time.Date(2026, 1, 2, 3, 4, 5, 0, time.UTC)
routes := make([]*LiveRoute, 0, len(origins))
for prefix, asn := range origins {
routes = append(routes, &LiveRoute{
ID: uuid.New(),
Prefix: prefix,
MaskLength: netip.MustParsePrefix(prefix).Bits(),
IPVersion: detectIPVersion(prefix),
OriginASN: asn,
PeerIP: testPeerIP,
ASPath: []int{asn},
NextHop: testPeerIP,
LastUpdated: ts,
}) })
} }
if err := db.UpsertLiveRouteBatch(routes); err != nil {
t.Fatalf("UpsertLiveRouteBatch: %v", err)
}
// lookup checks that ip resolves to the live prefix want, or to ErrNoRoute
// when want is empty.
lookup := func(ip, want string) {
t.Helper()
info, err := db.GetIPInfo(ip)
if want == "" {
if !errors.Is(err, ErrNoRoute) {
t.Errorf("GetIPInfo(%s) = %+v, %v; want ErrNoRoute", ip, info, err)
}
return
}
if err != nil {
t.Errorf("GetIPInfo(%s): %v", ip, err)
return
}
if info.Netblock != want || info.MaskLength != netip.MustParsePrefix(want).Bits() ||
info.ASN != origins[want] {
t.Errorf("GetIPInfo(%s) = %s (mask %d) AS%d, want %s AS%d",
ip, info.Netblock, info.MaskLength, info.ASN, want, origins[want])
}
}
lookup("10.1.2.3", "10.1.2.0/24")
lookup("::ffff:10.1.2.3", "10.1.2.0/24")
lookup("10.1.3.4", "10.1.0.0/16")
lookup("10.2.0.1", "10.0.0.0/8")
lookup("192.0.2.1", "")
lookup("2001:db8:1:2::3", "2001:db8:1:2::/64")
lookup("2001:db8:1:3::4", "2001:db8:1::/48")
lookup("2001:db8:2::1", "2001:db8::/32")
lookup("2001:db9::1", "")
err = db.DeleteLiveRouteBatch([]LiveRouteDeletion{
{Prefix: "10.1.2.0/24", OriginASN: 64502, PeerIP: testPeerIP, IPVersion: ipVersionV4},
{Prefix: "2001:db8:1:2::/64", OriginASN: 64502, PeerIP: testPeerIP, IPVersion: ipVersionV6},
})
if err != nil {
t.Fatalf("DeleteLiveRouteBatch: %v", err)
}
lookup("10.1.2.3", "10.1.0.0/16")
lookup("2001:db8:1:2::3", "2001:db8:1::/48")
} }
// TestGetIPInfoIgnoresShortRoutes checks that the IP lookup does not report a func TestCalculateIPv4Range(t *testing.T) {
// default route, or a route shorter than /8 for IPv4 or /16 for IPv6, as the tests := []struct {
// route for an address, and that a route of exactly /8 or /16, or longer, still name string
// is. The prefixes are written out so that a wrong limit fails the test. cidr string
func TestGetIPInfoIgnoresShortRoutes(t *testing.T) { wantStart uint32
cfg := &config.Config{StateDir: t.TempDir()} wantEnd uint32
wantErr bool
db, err := New(cfg, logger.New()) }{
if err != nil { {
t.Fatalf("failed to create database: %v", err) name: "Single IP /32",
} cidr: "192.168.1.1/32",
defer func() { _ = db.Close() }() wantStart: 3232235777,
wantEnd: 3232235777,
ts := time.Date(2026, 1, 2, 3, 4, 5, 0, time.UTC) },
announce := func(prefix string, asn int) { {
t.Helper() name: "Class C /24",
cidr: "192.168.1.0/24",
err := db.UpsertLiveRouteBatch([]*LiveRoute{{ wantStart: 3232235776, // 192.168.1.0
ID: uuid.New(), wantEnd: 3232236031, // 192.168.1.255
Prefix: prefix, },
MaskLength: netip.MustParsePrefix(prefix).Bits(), {
IPVersion: detectIPVersion(prefix), name: "Class B /16",
OriginASN: asn, cidr: "192.168.0.0/16",
PeerIP: testPeerIP, wantStart: 3232235520, // 192.168.0.0
ASPath: []int{asn}, wantEnd: 3232301055, // 192.168.255.255
NextHop: testPeerIP, },
LastUpdated: ts, {
}}) name: "Class A /8",
if err != nil { cidr: "10.0.0.0/8",
t.Fatalf("UpsertLiveRouteBatch(%s): %v", prefix, err) wantStart: 167772160, // 10.0.0.0
} wantEnd: 184549375, // 10.255.255.255
},
{
name: "Entire IPv4 space /0",
cidr: "0.0.0.0/0",
wantStart: 0,
wantEnd: 4294967295,
},
{
name: "Small subnet /30",
cidr: "192.168.1.0/30",
wantStart: 3232235776, // 192.168.1.0
wantEnd: 3232235779, // 192.168.1.3
},
{
name: "Medium subnet /20",
cidr: "172.16.0.0/20",
wantStart: 2886729728, // 172.16.0.0
wantEnd: 2886733823, // 172.16.15.255
},
{
name: "Private range 172.16/12",
cidr: "172.16.0.0/12",
wantStart: 2886729728, // 172.16.0.0
wantEnd: 2887778303, // 172.31.255.255
},
{
name: "Google DNS /29",
cidr: "8.8.8.8/29",
wantStart: 134744072, // 8.8.8.8 (network is actually 8.8.8.8 with /29)
wantEnd: 134744079, // 8.8.8.15
},
{
name: "Non-zero host bits",
cidr: "192.168.1.5/24",
wantStart: 3232235776, // 192.168.1.0 (network address)
wantEnd: 3232236031, // 192.168.1.255
},
{
name: "Invalid CIDR",
cidr: "192.168.1.1/33",
wantErr: true,
},
{
name: "Invalid IP",
cidr: "256.256.256.256/24",
wantErr: true,
},
{
name: "IPv6 CIDR",
cidr: "2001:db8::/32",
wantErr: true,
},
{
name: "Empty CIDR",
cidr: "",
wantErr: true,
},
{
name: "Missing mask",
cidr: "192.168.1.1",
wantErr: true,
},
} }
// lookup checks that ip resolves to the live prefix netblock from asn. for _, tt := range tests {
lookup := func(ip, netblock string, asn int) { t.Run(tt.name, func(t *testing.T) {
t.Helper() start, end, err := CalculateIPv4Range(tt.cidr)
info, err := db.GetIPInfo(ip)
if err != nil {
t.Errorf("GetIPInfo(%s): %v", ip, err)
if tt.wantErr {
if err == nil {
t.Errorf("CalculateIPv4Range(%s) expected error, got nil", tt.cidr)
}
return return
} }
if info.Netblock != netblock || info.ASN != asn {
t.Errorf("GetIPInfo(%s) = %s AS%d, want %s AS%d", ip, info.Netblock, info.ASN, netblock, asn) if err != nil {
} t.Errorf("CalculateIPv4Range(%s) unexpected error: %v", tt.cidr, err)
return
} }
// Default and very short routes, and routes one bit shorter than each limit. if start != tt.wantStart {
announce("2000::/3", 64500) t.Errorf("CalculateIPv4Range(%s) start = %d, want %d", tt.cidr, start, tt.wantStart)
announce("0.0.0.0/0", 64500)
announce("2000::/15", 64500)
announce("192.0.0.0/7", 64500)
for _, ip := range []string{"2001:db8::1", "192.0.2.1"} {
if info, err := db.GetIPInfo(ip); !errors.Is(err, ErrNoRoute) {
t.Errorf("GetIPInfo(%s) with only short routes = %+v, %v; want ErrNoRoute", ip, info, err)
}
} }
// Routes of exactly the minimum length are used. if end != tt.wantEnd {
announce("2001::/16", 64501) t.Errorf("CalculateIPv4Range(%s) end = %d, want %d", tt.cidr, end, tt.wantEnd)
announce("192.0.0.0/8", 64501) }
lookup("2001:db8::1", "2001::/16", 64501) // Verify that start <= end
lookup("192.0.2.1", "192.0.0.0/8", 64501) if start > end {
t.Errorf("CalculateIPv4Range(%s) start (%d) > end (%d)", tt.cidr, start, end)
}
announce("2001:db8::/32", 64502) // Verify the range size matches the CIDR mask
if !tt.wantErr && tt.cidr != "" {
_, ipNet, _ := net.ParseCIDR(tt.cidr)
if ipNet != nil {
ones, bits := ipNet.Mask.Size()
expectedSize := uint32(1) << uint(bits-ones)
actualSize := end - start + 1
if actualSize != expectedSize {
t.Errorf("CalculateIPv4Range(%s) range size = %d, want %d", tt.cidr, actualSize, expectedSize)
}
}
}
})
}
}
lookup("2001:db8::1", "2001:db8::/32", 64502) func TestIPv4RangeIntegration(t *testing.T) {
// Test that our functions work correctly together
tests := []struct {
name string
cidr string
testIPs []string
shouldContain []bool
}{
{
name: "192.168.1.0/24",
cidr: "192.168.1.0/24",
testIPs: []string{
"192.168.1.0",
"192.168.1.1",
"192.168.1.255",
"192.168.0.255",
"192.168.2.0",
},
shouldContain: []bool{true, true, true, false, false},
},
{
name: "10.0.0.0/8",
cidr: "10.0.0.0/8",
testIPs: []string{
"10.0.0.0",
"10.255.255.255",
"10.1.2.3",
"9.255.255.255",
"11.0.0.0",
},
shouldContain: []bool{true, true, true, false, false},
},
{
name: "172.16.0.0/12",
cidr: "172.16.0.0/12",
testIPs: []string{
"172.16.0.0",
"172.31.255.255",
"172.20.1.1",
"172.15.255.255",
"172.32.0.0",
},
shouldContain: []bool{true, true, true, false, false},
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
start, end, err := CalculateIPv4Range(tt.cidr)
if err != nil {
t.Fatalf("Failed to calculate range for %s: %v", tt.cidr, err)
}
for i, testIP := range tt.testIPs {
ip := net.ParseIP(testIP)
if ip == nil {
t.Fatalf("Failed to parse test IP: %s", testIP)
}
ipUint := ipToUint32(ip)
contained := ipUint >= start && ipUint <= end
if contained != tt.shouldContain[i] {
t.Errorf("IP %s in range %s: got %v, want %v", testIP, tt.cidr, contained, tt.shouldContain[i])
}
}
})
}
} }
// TestConnectionPoolPragmas holds several pooled connections open at once and // TestConnectionPoolPragmas holds several pooled connections open at once and
// checks each one carries the per-connection settings from the DSN, plus the // checks each one carries the per-connection settings from the DSN, plus the
// process-wide hard heap limit, and sees the new file with auto_vacuum // process-wide hard heap limit.
// incremental.
func TestConnectionPoolPragmas(t *testing.T) { func TestConnectionPoolPragmas(t *testing.T) {
cfg := &config.Config{StateDir: t.TempDir()} cfg := &config.Config{StateDir: t.TempDir()}
@@ -296,74 +373,6 @@ func TestConnectionPoolPragmas(t *testing.T) {
if hardHeapLimit != sqliteHardHeapLimitBytes { if hardHeapLimit != sqliteHardHeapLimitBytes {
t.Errorf("conn %d: hard_heap_limit = %d, want %d", i, hardHeapLimit, sqliteHardHeapLimitBytes) t.Errorf("conn %d: hard_heap_limit = %d, want %d", i, hardHeapLimit, sqliteHardHeapLimitBytes)
} }
var autoVacuum int
if err := c.QueryRowContext(ctx, "PRAGMA auto_vacuum").Scan(&autoVacuum); err != nil {
t.Fatalf("conn %d: failed to read auto_vacuum: %v", i, err)
}
if autoVacuum != autoVacuumIncremental {
t.Errorf("conn %d: auto_vacuum = %d, want %d (incremental)", i, autoVacuum, autoVacuumIncremental)
}
}
}
// TestVacuumReturnsFreePages checks that after routes are deleted from a new
// database, one Vacuum call returns every page they used. The deletes leave
// fewer free pages than the 1000 Vacuum frees per call, so none may remain.
// With auto_vacuum off (issue https://git.eeqj.de/sneak/routewatch/issues/43)
// the free pages stayed in the file, and with the PRAGMA run by ExecContext
// Vacuum freed only one page per call.
func TestVacuumReturnsFreePages(t *testing.T) {
cfg := &config.Config{StateDir: t.TempDir()}
db, err := New(cfg, logger.New())
if err != nil {
t.Fatalf("failed to create database: %v", err)
}
defer func() { _ = db.Close() }()
ctx := context.Background()
ts := time.Now().UTC()
const asn = 64500
routes := make([]*LiveRoute, 0, vacuumTestRoutes)
deletions := make([]LiveRouteDeletion, 0, vacuumTestRoutes)
for i := range vacuumTestRoutes {
route := mkV4Route(t, fmt.Sprintf("10.%d.%d.0/24", i/256, i%256), asn, ts)
routes = append(routes, route)
deletions = append(deletions, LiveRouteDeletion{
Prefix: route.Prefix,
OriginASN: asn,
PeerIP: route.PeerIP,
IPVersion: ipVersionV4,
})
}
if err := db.UpsertLiveRouteBatch(routes); err != nil {
t.Fatalf("failed to write routes: %v", err)
}
if err := db.DeleteLiveRouteBatch(deletions); err != nil {
t.Fatalf("failed to delete routes: %v", err)
}
var before int
if err := db.db.QueryRowContext(ctx, "PRAGMA freelist_count").Scan(&before); err != nil {
t.Fatalf("failed to read freelist_count: %v", err)
}
if before == 0 {
t.Fatalf("no free pages after deleting %d routes", vacuumTestRoutes)
}
if err := db.Vacuum(ctx); err != nil {
t.Fatalf("Vacuum failed: %v", err)
}
var after int
if err := db.db.QueryRowContext(ctx, "PRAGMA freelist_count").Scan(&after); err != nil {
t.Fatalf("failed to read freelist_count: %v", err)
}
if after != 0 {
t.Errorf("free pages after Vacuum = %d of %d, want 0", after, before)
} }
} }
@@ -417,10 +426,12 @@ func TestBatchWriteDuringCheckpoint(t *testing.T) {
wg.Wait() wg.Wait()
} }
// TestGetRandomPrefixesByLengthListsPrefixOnce stores one prefix of each // TestStatsRouteTimestampsAndCounts checks GetStatsContext reports the correct
// family learned from two peers in the same second and checks that the // route counts and the oldest/newest last_updated across both route tables. The
// prefix-length sample lists each of them once. // old union-scan query read the aggregate result into *time.Time, which the
func TestGetRandomPrefixesByLengthListsPrefixOnce(t *testing.T) { // driver could not parse, so it logged a warning every call and left both
// timestamps nil; this asserts they are populated from the right rows.
func TestStatsRouteTimestampsAndCounts(t *testing.T) {
cfg := &config.Config{StateDir: t.TempDir()} cfg := &config.Config{StateDir: t.TempDir()}
db, err := New(cfg, logger.New()) db, err := New(cfg, logger.New())
@@ -429,268 +440,105 @@ func TestGetRandomPrefixesByLengthListsPrefixOnce(t *testing.T) {
} }
defer func() { _ = db.Close() }() defer func() { _ = db.Close() }()
ts := time.Date(2026, 1, 2, 3, 4, 5, 0, time.UTC)
v4First := mkV4Route(t, "198.51.100.0/24", 64500, ts)
v4Second := mkV4Route(t, "198.51.100.0/24", 64500, ts)
v4Second.PeerIP = secondV4PeerIP
v6First := mkV6Route("2001:db8::/32", 64500, ts)
v6Second := mkV6Route("2001:db8::/32", 64500, ts)
v6Second.PeerIP = secondV6PeerIP
if err := db.UpsertLiveRouteBatch([]*LiveRoute{v4First, v4Second, v6First, v6Second}); err != nil {
t.Fatalf("UpsertLiveRouteBatch: %v", err)
}
for _, want := range []*LiveRoute{v4First, v6First} {
routes, err := db.GetRandomPrefixesByLength(want.MaskLength, want.IPVersion, sampleLimit)
if err != nil {
t.Fatalf("GetRandomPrefixesByLength(%d, %d): %v", want.MaskLength, want.IPVersion, err)
}
if len(routes) != 1 || routes[0].Prefix != want.Prefix {
t.Errorf("GetRandomPrefixesByLength(%d, %d) = %+v, want %s listed once",
want.MaskLength, want.IPVersion, routes, want.Prefix)
}
}
}
// TestGetRandomPrefixesByLengthListsNewestRoute stores a prefix learned from
// two peers at different times, the older route first, and checks that the
// prefix-length sample lists the newer route.
func TestGetRandomPrefixesByLengthListsNewestRoute(t *testing.T) {
cfg := &config.Config{StateDir: t.TempDir()}
db, err := New(cfg, logger.New())
if err != nil {
t.Fatalf("failed to create database: %v", err)
}
defer func() { _ = db.Close() }()
older := time.Date(2026, 1, 2, 3, 4, 5, 0, time.UTC)
olderRoute := mkV4Route(t, "198.51.100.0/24", 64500, older)
newerRoute := mkV4Route(t, "198.51.100.0/24", 64500, older.Add(time.Minute))
newerRoute.PeerIP = secondV4PeerIP
if err := db.UpsertLiveRouteBatch([]*LiveRoute{olderRoute, newerRoute}); err != nil {
t.Fatalf("UpsertLiveRouteBatch: %v", err)
}
routes, err := db.GetRandomPrefixesByLength(newerRoute.MaskLength, ipVersionV4, sampleLimit)
if err != nil {
t.Fatalf("GetRandomPrefixesByLength: %v", err)
}
if len(routes) != 1 || routes[0].PeerIP != newerRoute.PeerIP {
t.Errorf("GetRandomPrefixesByLength = %+v, want only the newer route, from %s",
routes, newerRoute.PeerIP)
}
}
// TestGetRandomPrefixesByLengthSamplesManyPrefixes stores more /24 prefixes
// than the sample asks for, each followed by a /16 so that routes of another
// mask length sit between the /24 rows, and each prefix learned from two peers
// in the same second. It checks both the sample the pages get, picked from the
// shuffled prefixes, and the random lookups used for a mask length with more
// than maxPrefixesToShuffle prefixes.
func TestGetRandomPrefixesByLengthSamplesManyPrefixes(t *testing.T) {
cfg := &config.Config{StateDir: t.TempDir()}
db, err := New(cfg, logger.New())
if err != nil {
t.Fatalf("failed to create database: %v", err)
}
defer func() { _ = db.Close() }()
ts := time.Date(2026, 1, 2, 3, 4, 5, 0, time.UTC)
var routes []*LiveRoute
var stored []string
for i := range sampledPrefixCount {
prefix := fmt.Sprintf("10.0.%d.0/24", i)
for _, p := range []string{prefix, fmt.Sprintf("172.%d.0.0/16", i)} {
first := mkV4Route(t, p, 64500, ts)
second := mkV4Route(t, p, 64500, ts)
second.PeerIP = secondV4PeerIP
routes = append(routes, first, second)
}
stored = append(stored, prefix)
}
if err := db.UpsertLiveRouteBatch(routes); err != nil {
t.Fatalf("UpsertLiveRouteBatch: %v", err)
}
slices.Sort(stored)
maskLength := routes[0].MaskLength
sample, err := db.GetRandomPrefixesByLength(maskLength, ipVersionV4, sampleLimit)
if err != nil {
t.Fatalf("GetRandomPrefixesByLength: %v", err)
}
var listed []string
for _, route := range sample {
listed = append(listed, route.Prefix)
}
checkPrefixSample(t, "GetRandomPrefixesByLength", listed, stored)
looked, err := db.samplePrefixesByLength(
context.Background(), liveRoutesV4Table, maskLength, sampleLimit, nil)
if err != nil {
t.Fatalf("samplePrefixesByLength: %v", err)
}
checkPrefixSample(t, "samplePrefixesByLength", looked, stored)
}
// checkPrefixSample checks that sample lists sampleLimit prefixes, each once
// and each one of stored, and that it is not just the first sampleLimit of
// stored, which is in text order.
func checkPrefixSample(t *testing.T, name string, sample, stored []string) {
t.Helper()
if len(sample) != sampleLimit {
t.Errorf("%s listed %d prefixes, want %d", name, len(sample), sampleLimit)
}
listed := make(map[string]bool)
for _, prefix := range sample {
if listed[prefix] || !slices.Contains(stored, prefix) {
t.Errorf("%s lists %s twice or it is not one of the /24s", name, prefix)
}
listed[prefix] = true
}
if slices.Equal(slices.Sorted(slices.Values(sample)), stored[:sampleLimit]) {
t.Errorf("%s lists just the first %d prefixes in text order", name, sampleLimit)
}
}
// TestGetRandomPrefixesByLengthSamplesPastFirstPrefixes stores more prefixes
// of one mask length than maxPrefixesToShuffle. It checks that
// firstPrefixesByLength reads just the first maxPrefixesToShuffle+1 in text
// order, and that the sample lists the prefixes asked for, each once, some of
// them after those first ones, which only the random lookups reach.
func TestGetRandomPrefixesByLengthSamplesPastFirstPrefixes(t *testing.T) {
cfg := &config.Config{StateDir: t.TempDir()}
db, err := New(cfg, logger.New())
if err != nil {
t.Fatalf("failed to create database: %v", err)
}
defer func() { _ = db.Close() }()
ts := time.Date(2026, 1, 2, 3, 4, 5, 0, time.UTC)
var routes []*LiveRoute
var stored []string
for i := range largeSamplePrefixCount {
route := mkV6Route(fmt.Sprintf("2001:%x::/32", i), 64500, ts)
routes = append(routes, route)
stored = append(stored, route.Prefix)
}
if err := db.UpsertLiveRouteBatch(routes); err != nil {
t.Fatalf("UpsertLiveRouteBatch: %v", err)
}
slices.Sort(stored)
firstPrefixes := stored[:maxPrefixesToShuffle+1]
maskLength := routes[0].MaskLength
read, err := db.firstPrefixesByLength(
context.Background(), liveRoutesV6Table, maskLength, maxPrefixesToShuffle+1)
if err != nil {
t.Fatalf("firstPrefixesByLength: %v", err)
}
if !slices.Equal(read, firstPrefixes) {
t.Errorf("firstPrefixesByLength read %d prefixes, want the first %d in text order",
len(read), len(firstPrefixes))
}
sample, err := db.GetRandomPrefixesByLength(maskLength, ipVersionV6, largeSampleLimit)
if err != nil {
t.Fatalf("GetRandomPrefixesByLength: %v", err)
}
if len(sample) != largeSampleLimit {
t.Errorf("GetRandomPrefixesByLength listed %d prefixes, want %d", len(sample), largeSampleLimit)
}
listed := make(map[string]bool)
pastFirstPrefixes := 0
for _, route := range sample {
if listed[route.Prefix] {
t.Errorf("GetRandomPrefixesByLength lists %s twice", route.Prefix)
}
listed[route.Prefix] = true
if !slices.Contains(firstPrefixes, route.Prefix) {
pastFirstPrefixes++
}
}
if pastFirstPrefixes == 0 {
t.Errorf("GetRandomPrefixesByLength lists only prefixes among the first %d in text order",
len(firstPrefixes))
}
}
// TestSamplePrefixesByLengthFillsUpFromShuffled stores a single prefix, so
// every random lookup lands on it, and checks that the sample still lists
// sampleLimit prefixes: the stored one, then the shuffled ones in order. The
// stored prefix is the second shuffled one, so the fill-up must skip it.
func TestSamplePrefixesByLengthFillsUpFromShuffled(t *testing.T) {
cfg := &config.Config{StateDir: t.TempDir()}
db, err := New(cfg, logger.New())
if err != nil {
t.Fatalf("failed to create database: %v", err)
}
defer func() { _ = db.Close() }()
var shuffled []string
for i := range sampledPrefixCount {
shuffled = append(shuffled, fmt.Sprintf("10.0.%d.0/24", i))
}
route := mkV4Route(t, shuffled[1], 64500, time.Date(2026, 1, 2, 3, 4, 5, 0, time.UTC))
if err := db.UpsertLiveRouteBatch([]*LiveRoute{route}); err != nil {
t.Fatalf("UpsertLiveRouteBatch: %v", err)
}
sample, err := db.samplePrefixesByLength(
context.Background(), liveRoutesV4Table, route.MaskLength, sampleLimit, shuffled)
if err != nil {
t.Fatalf("samplePrefixesByLength: %v", err)
}
want := append([]string{shuffled[1], shuffled[0]}, shuffled[2:sampleLimit]...)
if !slices.Equal(sample, want) {
t.Errorf("samplePrefixesByLength = %v, want %v", sample, want)
}
}
// TestGetRandomPrefixesByLengthSkipsWithdrawnRoutes withdraws a route after
// storing it, as can happen between the steps of GetRandomPrefixesByLength. It
// checks that the random lookups pick nothing for a mask length with no route
// left, and that only the prefixes that still have a route get their newest
// route listed.
func TestGetRandomPrefixesByLengthSkipsWithdrawnRoutes(t *testing.T) {
cfg := &config.Config{StateDir: t.TempDir()}
db, err := New(cfg, logger.New())
if err != nil {
t.Fatalf("failed to create database: %v", err)
}
defer func() { _ = db.Close() }()
ts := time.Date(2026, 1, 2, 3, 4, 5, 0, time.UTC)
kept := mkV4Route(t, "198.51.100.0/24", 64500, ts)
withdrawn := mkV4Route(t, "172.16.0.0/16", 64500, ts)
if err := db.UpsertLiveRouteBatch([]*LiveRoute{kept, withdrawn}); err != nil {
t.Fatalf("UpsertLiveRouteBatch: %v", err)
}
if err := db.DeleteLiveRoute(withdrawn.Prefix, withdrawn.OriginASN, withdrawn.PeerIP); err != nil {
t.Fatalf("DeleteLiveRoute: %v", err)
}
ctx := context.Background() ctx := context.Background()
sample, err := db.samplePrefixesByLength(
ctx, liveRoutesV4Table, withdrawn.MaskLength, sampleLimit, []string{withdrawn.Prefix}) // Empty database: no routes, so both timestamps are nil and no error.
empty, err := db.GetStatsContext(ctx)
if err != nil { if err != nil {
t.Fatalf("samplePrefixesByLength: %v", err) t.Fatalf("GetStatsContext on empty database: %v", err)
} }
if len(sample) != 0 { if empty.OldestRoute != nil || empty.NewestRoute != nil {
t.Errorf("samplePrefixesByLength = %v, want none for a mask length with no route left", sample) t.Fatalf("empty database timestamps = (%v, %v), want (nil, nil)",
empty.OldestRoute, empty.NewestRoute)
}
if empty.LiveRoutes != 0 {
t.Fatalf("empty database LiveRoutes = %d, want 0", empty.LiveRoutes)
} }
routes, err := db.newestRouteOfEachPrefix( base := time.Date(2026, 1, 2, 3, 4, 5, 0, time.UTC)
ctx, liveRoutesV4Table, ipVersionV4, []string{withdrawn.Prefix, kept.Prefix}) oldest := base
if err != nil { middle := base.Add(time.Minute)
t.Fatalf("newestRouteOfEachPrefix: %v", err) newest := base.Add(2 * time.Minute)
mkV4 := func(prefix string, asn int, ts time.Time) *LiveRoute {
start, end, rerr := CalculateIPv4Range(prefix)
if rerr != nil {
t.Fatalf("CalculateIPv4Range(%s): %v", prefix, rerr)
} }
if len(routes) != 1 || routes[0].Prefix != kept.Prefix {
t.Errorf("newestRouteOfEachPrefix = %+v, want only the route of %s", routes, kept.Prefix) return &LiveRoute{
ID: uuid.New(),
Prefix: prefix,
MaskLength: 24,
IPVersion: ipVersionV4,
OriginASN: asn,
PeerIP: "192.0.2.1",
ASPath: []int{asn},
NextHop: "192.0.2.254",
LastUpdated: ts,
V4IPStart: &start,
V4IPEnd: &end,
}
}
// Two IPv4 routes (one oldest, one middle) and one IPv6 route (newest).
routes := []*LiveRoute{
mkV4("198.51.100.0/24", 64500, middle),
mkV4("203.0.113.0/24", 64501, oldest),
{
ID: uuid.New(),
Prefix: "2001:db8::/32",
MaskLength: 32,
IPVersion: ipVersionV6,
OriginASN: 64502,
PeerIP: "2001:db8::1",
ASPath: []int{64502},
NextHop: "2001:db8::ffff",
LastUpdated: newest,
},
}
for _, route := range routes {
if err := db.UpsertLiveRoute(route); err != nil {
t.Fatalf("UpsertLiveRoute(%s): %v", route.Prefix, err)
}
}
stats, err := db.GetStatsContext(ctx)
if err != nil {
t.Fatalf("GetStatsContext: %v", err)
}
if stats.IPv4Routes != 2 {
t.Errorf("IPv4Routes = %d, want 2", stats.IPv4Routes)
}
if stats.IPv6Routes != 1 {
t.Errorf("IPv6Routes = %d, want 1", stats.IPv6Routes)
}
if stats.LiveRoutes != 3 {
t.Errorf("LiveRoutes = %d, want 3", stats.LiveRoutes)
}
if stats.OldestRoute == nil || !stats.OldestRoute.Equal(oldest) {
t.Errorf("OldestRoute = %v, want %v", stats.OldestRoute, oldest)
}
if stats.NewestRoute == nil || !stats.NewestRoute.Equal(newest) {
t.Errorf("NewestRoute = %v, want %v", stats.NewestRoute, newest)
}
}
func BenchmarkIPToUint32(b *testing.B) {
ip := net.ParseIP("192.168.1.1")
b.ResetTimer()
for i := 0; i < b.N; i++ {
_ = ipToUint32(ip)
}
}
func BenchmarkCalculateIPv4Range(b *testing.B) {
cidr := "192.168.0.0/16"
b.ResetTimer()
for i := 0; i < b.N; i++ {
_, _, _ = CalculateIPv4Range(cidr)
} }
} }
internal/database/interface.go
+4 -2
View File
@@ -63,6 +63,8 @@ type Store interface {
GetLiveRouteCountsContext(ctx context.Context) (ipv4Count, ipv6Count int, err error) GetLiveRouteCountsContext(ctx context.Context) (ipv4Count, ipv6Count int, err error)
// IP lookup operations // IP lookup operations
GetASInfoForIP(ip string) (*ASInfo, error)
GetASInfoForIPContext(ctx context.Context, ip string) (*ASInfo, error)
GetIPInfo(ip string) (*IPInfo, error) GetIPInfo(ip string) (*IPInfo, error)
GetIPInfoContext(ctx context.Context, ip string) (*IPInfo, error) GetIPInfoContext(ctx context.Context, ip string) (*IPInfo, error)
@@ -76,8 +78,8 @@ type Store interface {
GetASDetailsContext(ctx context.Context, asn int) (*ASN, []LiveRoute, error) GetASDetailsContext(ctx context.Context, asn int) (*ASN, []LiveRoute, error)
GetASPeers(asn int) ([]ASPeer, error) GetASPeers(asn int) ([]ASPeer, error)
GetASPeersContext(ctx context.Context, asn int) ([]ASPeer, error) GetASPeersContext(ctx context.Context, asn int) ([]ASPeer, error)
GetPrefixDetails(prefix string) ([]LiveRoute, map[int]ASN, error) GetPrefixDetails(prefix string) ([]LiveRoute, error)
GetPrefixDetailsContext(ctx context.Context, prefix string) ([]LiveRoute, map[int]ASN, error) GetPrefixDetailsContext(ctx context.Context, prefix string) ([]LiveRoute, error)
GetRandomPrefixesByLength(maskLength, ipVersion, limit int) ([]LiveRoute, error) GetRandomPrefixesByLength(maskLength, ipVersion, limit int) ([]LiveRoute, error)
GetRandomPrefixesByLengthContext(ctx context.Context, maskLength, ipVersion, limit int) ([]LiveRoute, error) GetRandomPrefixesByLengthContext(ctx context.Context, maskLength, ipVersion, limit int) ([]LiveRoute, error)
internal/database/models.go
+13
View File
@@ -77,6 +77,9 @@ type LiveRoute struct {
ASPath []int `json:"as_path"` ASPath []int `json:"as_path"`
NextHop string `json:"next_hop"` NextHop string `json:"next_hop"`
LastUpdated time.Time `json:"last_updated"` LastUpdated time.Time `json:"last_updated"`
// IPv4 range fields for fast lookups (nil for IPv6)
V4IPStart *uint32 `json:"v4_ip_start,omitempty"`
V4IPEnd *uint32 `json:"v4_ip_end,omitempty"`
} }
// PrefixDistribution represents the distribution of prefixes by mask length // PrefixDistribution represents the distribution of prefixes by mask length
@@ -85,6 +88,16 @@ type PrefixDistribution struct {
Count int `json:"count"` Count int `json:"count"`
} }
// ASInfo represents AS information for an IP lookup (legacy format)
type ASInfo struct {
ASN int `json:"asn"`
Handle string `json:"handle"`
Description string `json:"description"`
Prefix string `json:"prefix"`
LastUpdated time.Time `json:"last_updated"`
Age string `json:"age"`
}
// IPInfo represents comprehensive IP information for the /ip endpoint // IPInfo represents comprehensive IP information for the /ip endpoint
type IPInfo struct { type IPInfo struct {
IP string `json:"ip"` IP string `json:"ip"`
internal/database/schema.sql
+6
View File
@@ -107,6 +107,9 @@ CREATE TABLE IF NOT EXISTS live_routes_v4 (
as_path TEXT NOT NULL, -- JSON array as_path TEXT NOT NULL, -- JSON array
next_hop TEXT NOT NULL, next_hop TEXT NOT NULL,
last_updated DATETIME NOT NULL, last_updated DATETIME NOT NULL,
-- IPv4 range columns for fast lookups
ip_start INTEGER NOT NULL, -- Start of IPv4 range as 32-bit unsigned int
ip_end INTEGER NOT NULL, -- End of IPv4 range as 32-bit unsigned int
UNIQUE(prefix, origin_asn, peer_ip) UNIQUE(prefix, origin_asn, peer_ip)
); );
@@ -120,6 +123,7 @@ CREATE TABLE IF NOT EXISTS live_routes_v6 (
as_path TEXT NOT NULL, -- JSON array as_path TEXT NOT NULL, -- JSON array
next_hop TEXT NOT NULL, next_hop TEXT NOT NULL,
last_updated DATETIME NOT NULL, last_updated DATETIME NOT NULL,
-- Note: IPv6 doesn't use integer range columns
UNIQUE(prefix, origin_asn, peer_ip) UNIQUE(prefix, origin_asn, peer_ip)
); );
@@ -128,6 +132,8 @@ CREATE INDEX IF NOT EXISTS idx_live_routes_v4_prefix ON live_routes_v4(prefix);
CREATE INDEX IF NOT EXISTS idx_live_routes_v4_mask_length ON live_routes_v4(mask_length); CREATE INDEX IF NOT EXISTS idx_live_routes_v4_mask_length ON live_routes_v4(mask_length);
CREATE INDEX IF NOT EXISTS idx_live_routes_v4_origin_asn ON live_routes_v4(origin_asn); CREATE INDEX IF NOT EXISTS idx_live_routes_v4_origin_asn ON live_routes_v4(origin_asn);
CREATE INDEX IF NOT EXISTS idx_live_routes_v4_last_updated ON live_routes_v4(last_updated); CREATE INDEX IF NOT EXISTS idx_live_routes_v4_last_updated ON live_routes_v4(last_updated);
-- Indexes for IPv4 range queries
CREATE INDEX IF NOT EXISTS idx_live_routes_v4_ip_range ON live_routes_v4(ip_start, ip_end);
-- Index to optimize prefix distribution queries -- Index to optimize prefix distribution queries
CREATE INDEX IF NOT EXISTS idx_live_routes_v4_mask_prefix ON live_routes_v4(mask_length, prefix); CREATE INDEX IF NOT EXISTS idx_live_routes_v4_mask_prefix ON live_routes_v4(mask_length, prefix);
internal/database/utils.go
-13
View File
@@ -1,8 +1,6 @@
package database package database
import ( import (
"fmt"
"net"
"strings" "strings"
"github.com/google/uuid" "github.com/google/uuid"
@@ -20,14 +18,3 @@ func detectIPVersion(prefix string) int {
return ipVersionV4 return ipVersionV4
} }
// prefixMaskLength returns the mask length of a prefix such as 192.0.2.0/24.
func prefixMaskLength(prefix string) (int, error) {
_, network, err := net.ParseCIDR(prefix)
if err != nil {
return 0, fmt.Errorf("invalid prefix %s: %w", prefix, err)
}
maskLength, _ := network.Mask.Size()
return maskLength, nil
}
internal/metrics/metrics.go
+6 -11
View File
@@ -60,23 +60,18 @@ func New() *Tracker {
} }
} }
// SetConnected updates the connection status. Every connection after the // SetConnected updates the connection status
// first counts one reconnect.
func (t *Tracker) SetConnected(connected bool) { func (t *Tracker) SetConnected(connected bool) {
t.isConnected.Store(connected) wasConnected := t.isConnected.Swap(connected)
if !connected { if connected {
return
}
t.mu.Lock() t.mu.Lock()
// connectedSince is zero until the first connection.
connectedBefore := !t.connectedSince.IsZero()
t.connectedSince = time.Now() t.connectedSince = time.Now()
t.mu.Unlock() t.mu.Unlock()
// Increment reconnect count (but not for the initial connection)
if connectedBefore { if wasConnected || t.reconnectCount.Load() > 0 {
t.reconnectCount.Add(1) t.reconnectCount.Add(1)
} }
}
} }
// GetReconnectCount returns the number of reconnections since startup // GetReconnectCount returns the number of reconnections since startup
internal/metrics/metrics_test.go
-21
View File
@@ -1,21 +0,0 @@
package metrics
import "testing"
// TestReconnectCount connects three times with a disconnect between each, the
// way the streamer does, and expects every connection after the first to count
// one reconnect.
func TestReconnectCount(t *testing.T) {
tracker := New()
tracker.SetConnected(true)
tracker.SetConnected(false)
tracker.SetConnected(true)
tracker.SetConnected(false)
tracker.SetConnected(true)
const wantReconnects = 2
if got := tracker.GetReconnectCount(); got != wantReconnects {
t.Fatalf("reconnect count = %d, want %d", got, wantReconnects)
}
}
internal/routewatch/app_integration_test.go
+23 -6
View File
@@ -232,6 +232,25 @@ func (m *mockStore) GetLiveRouteCountsContext(ctx context.Context) (ipv4Count, i
return m.GetLiveRouteCounts() return m.GetLiveRouteCounts()
} }
// GetASInfoForIP mock implementation
func (m *mockStore) GetASInfoForIP(ip string) (*database.ASInfo, error) {
// Simple mock - return a test AS
now := time.Now()
return &database.ASInfo{
ASN: 15169,
Handle: "GOOGLE",
Description: "Google LLC",
Prefix: "8.8.8.0/24",
LastUpdated: now.Add(-5 * time.Minute),
Age: "5m0s",
}, nil
}
// GetASInfoForIPContext mock implementation with context support
func (m *mockStore) GetASInfoForIPContext(ctx context.Context, ip string) (*database.ASInfo, error) {
return m.GetASInfoForIP(ip)
}
// GetASDetails mock implementation // GetASDetails mock implementation
func (m *mockStore) GetASDetails(asn int) (*database.ASN, []database.LiveRoute, error) { func (m *mockStore) GetASDetails(asn int) (*database.ASN, []database.LiveRoute, error) {
m.mu.Lock() m.mu.Lock()
@@ -252,15 +271,13 @@ func (m *mockStore) GetASDetailsContext(ctx context.Context, asn int) (*database
} }
// GetPrefixDetails mock implementation // GetPrefixDetails mock implementation
func (m *mockStore) GetPrefixDetails(prefix string) ([]database.LiveRoute, map[int]database.ASN, error) { func (m *mockStore) GetPrefixDetails(prefix string) ([]database.LiveRoute, error) {
// Return empty routes for now // Return empty routes for now
return []database.LiveRoute{}, map[int]database.ASN{}, nil return []database.LiveRoute{}, nil
} }
// GetPrefixDetailsContext mock implementation with context support // GetPrefixDetailsContext mock implementation with context support
func (m *mockStore) GetPrefixDetailsContext( func (m *mockStore) GetPrefixDetailsContext(ctx context.Context, prefix string) ([]database.LiveRoute, error) {
ctx context.Context, prefix string,
) ([]database.LiveRoute, map[int]database.ASN, error) {
return m.GetPrefixDetails(prefix) return m.GetPrefixDetails(prefix)
} }
@@ -433,7 +450,7 @@ func TestRouteWatchLiveFeed(t *testing.T) {
} }
// Create server // Create server
srv := server.New(mockDB, s, logger, cfg) srv := server.New(mockDB, s, logger)
// Create RouteWatch with 5 second limit // Create RouteWatch with 5 second limit
deps := Dependencies{ deps := Dependencies{
internal/routewatch/cli.go
-5
View File
@@ -20,8 +20,6 @@ const (
debugInterval = 60 * time.Second debugInterval = 60 * time.Second
// bytesPerMB is bytes per megabyte // bytesPerMB is bytes per megabyte
bytesPerMB = 1024 * 1024 bytesPerMB = 1024 * 1024
// failureExitCode is the exit status when RouteWatch stops on an error
failureExitCode = 1
) )
// logDebugStats logs goroutine count and memory usage // logDebugStats logs goroutine count and memory usage
@@ -85,9 +83,6 @@ func CLIEntry() {
go func() { go func() {
if err := rw.Run(ctx); err != nil { if err := rw.Run(ctx); err != nil {
logger.Error("RouteWatch error", "error", err) logger.Error("RouteWatch error", "error", err)
if err := shutdowner.Shutdown(fx.ExitCode(failureExitCode)); err != nil {
logger.Error("Failed to shutdown gracefully", "error", err)
}
} }
}() }()
internal/routewatch/prefixhandler.go
+69 -44
View File
@@ -2,7 +2,6 @@ package routewatch
import ( import (
"net" "net"
"net/netip"
"strings" "strings"
"sync" "sync"
"time" "time"
@@ -105,28 +104,11 @@ func (h *PrefixHandler) HandleMessage(msg *ristypes.RISMessage) {
h.mu.Lock() h.mu.Lock()
defer h.mu.Unlock() defer h.mu.Unlock()
// Withdrawals are queued before announcements, because the batch keeps // Process announcements
// each peer's last update for a prefix: a message that lists a prefix as
// both withdrawn and announced keeps the route (RFC 4271 section 4.3).
for _, prefix := range msg.Withdrawals {
h.batch = append(h.batch, prefixUpdate{
prefix: canonicalPrefix(prefix),
originASN: originASN, // Use the originASN from path if available
peer: msg.Peer,
messageType: "withdrawal",
timestamp: timestamp,
path: msg.Path,
})
// Record withdrawal in metrics
if h.metrics != nil {
h.metrics.RecordWithdrawal()
}
}
for _, announcement := range msg.Announcements { for _, announcement := range msg.Announcements {
for _, prefix := range announcement.Prefixes { for _, prefix := range announcement.Prefixes {
h.batch = append(h.batch, prefixUpdate{ h.batch = append(h.batch, prefixUpdate{
prefix: canonicalPrefix(prefix), prefix: prefix,
originASN: originASN, originASN: originASN,
peer: msg.Peer, peer: msg.Peer,
messageType: "announcement", messageType: "announcement",
@@ -140,6 +122,22 @@ func (h *PrefixHandler) HandleMessage(msg *ristypes.RISMessage) {
} }
} }
// Process withdrawals
for _, prefix := range msg.Withdrawals {
h.batch = append(h.batch, prefixUpdate{
prefix: prefix,
originASN: originASN, // Use the originASN from path if available
peer: msg.Peer,
messageType: "withdrawal",
timestamp: timestamp,
path: msg.Path,
})
// Record withdrawal in metrics
if h.metrics != nil {
h.metrics.RecordWithdrawal()
}
}
// Check if we need to flush // Check if we need to flush
if len(h.batch) >= prefixBatchSize { if len(h.batch) >= prefixBatchSize {
h.flushBatchLocked() h.flushBatchLocked()
@@ -180,14 +178,14 @@ func (h *PrefixHandler) flushBatchLocked() {
startTime := time.Now() startTime := time.Now()
batchSize := len(h.batch) batchSize := len(h.batch)
// For each prefix and peer, keep only the last update the feed delivered. // Group updates by prefix to deduplicate
// Updates from different peers for the same prefix are all kept. Arrival // For each prefix, keep the latest update
// order decides, not the timestamp: timestamps are whole seconds, so prefixMap := make(map[string]prefixUpdate)
// updates within one second carry the same one.
type prefixAndPeer struct{ prefix, peer string }
lastUpdates := make(map[prefixAndPeer]prefixUpdate)
for _, update := range h.batch { for _, update := range h.batch {
lastUpdates[prefixAndPeer{update.prefix, update.peer}] = update key := update.prefix
if existing, ok := prefixMap[key]; !ok || update.timestamp.After(existing.timestamp) {
prefixMap[key] = update
}
} }
// Collect routes to upsert and delete // Collect routes to upsert and delete
@@ -197,7 +195,7 @@ func (h *PrefixHandler) flushBatchLocked() {
// Collect unique prefixes to update // Collect unique prefixes to update
prefixesToUpdate := make(map[string]time.Time) prefixesToUpdate := make(map[string]time.Time)
for _, update := range lastUpdates { for _, update := range prefixMap {
// Track prefix for both announcements and withdrawals // Track prefix for both announcements and withdrawals
if _, exists := prefixesToUpdate[update.prefix]; !exists || update.timestamp.After(prefixesToUpdate[update.prefix]) { if _, exists := prefixesToUpdate[update.prefix]; !exists || update.timestamp.After(prefixesToUpdate[update.prefix]) {
prefixesToUpdate[update.prefix] = update.timestamp prefixesToUpdate[update.prefix] = update.timestamp
@@ -256,7 +254,7 @@ func (h *PrefixHandler) flushBatchLocked() {
elapsed := time.Since(startTime) elapsed := time.Since(startTime)
h.logger.Debug("Flushed prefix batch", h.logger.Debug("Flushed prefix batch",
"batch_size", batchSize, "batch_size", batchSize,
"unique_prefixes", len(prefixesToUpdate), "unique_prefixes", len(prefixMap),
"success", successCount, "success", successCount,
"duration_ms", elapsed.Milliseconds(), "duration_ms", elapsed.Milliseconds(),
) )
@@ -266,21 +264,6 @@ func (h *PrefixHandler) flushBatchLocked() {
h.lastFlush = time.Now() h.lastFlush = time.Now()
} }
// canonicalPrefix returns prefix in the text form net/netip prints, the form
// the IP lookup builds when it looks a prefix up. The feed sends IPv6
// announcements compressed ("2001:db8::/32") but IPv6 withdrawals uncompressed
// ("2001:db8:0:0:0:0:0:0/32"); stored as received, a withdrawal would not match
// the route its announcement stored. A prefix that does not parse is returned
// unchanged, and the batch flush reports it.
func canonicalPrefix(prefix string) string {
p, err := netip.ParsePrefix(prefix)
if err != nil {
return prefix
}
return p.Masked().String()
}
// parseCIDR extracts the mask length and IP version from a prefix string // parseCIDR extracts the mask length and IP version from a prefix string
func parseCIDR(prefix string) (maskLength int, ipVersion int, err error) { func parseCIDR(prefix string) (maskLength int, ipVersion int, err error) {
_, ipNet, err := net.ParseCIDR(prefix) _, ipNet, err := net.ParseCIDR(prefix)
@@ -332,6 +315,20 @@ func (h *PrefixHandler) processAnnouncement(_ *database.Prefix, update prefixUpd
LastUpdated: update.timestamp, LastUpdated: update.timestamp,
} }
// For IPv4, calculate the IP range
if ipVersion == ipv4Version {
start, end, err := database.CalculateIPv4Range(update.prefix)
if err == nil {
liveRoute.V4IPStart = &start
liveRoute.V4IPEnd = &end
} else {
h.logger.Error("Failed to calculate IPv4 range",
"prefix", update.prefix,
"error", err,
)
}
}
if err := h.db.UpsertLiveRoute(liveRoute); err != nil { if err := h.db.UpsertLiveRoute(liveRoute); err != nil {
h.logger.Error("Failed to upsert live route", h.logger.Error("Failed to upsert live route",
"prefix", update.prefix, "prefix", update.prefix,
@@ -375,6 +372,20 @@ func (h *PrefixHandler) createLiveRoute(update prefixUpdate) *database.LiveRoute
LastUpdated: update.timestamp, LastUpdated: update.timestamp,
} }
// For IPv4, calculate the IP range
if ipVersion == ipv4Version {
start, end, err := database.CalculateIPv4Range(update.prefix)
if err == nil {
liveRoute.V4IPStart = &start
liveRoute.V4IPEnd = &end
} else {
h.logger.Error("Failed to calculate IPv4 range",
"prefix", update.prefix,
"error", err,
)
}
}
return liveRoute return liveRoute
} }
@@ -414,6 +425,20 @@ func (h *PrefixHandler) processAnnouncementDirect(update prefixUpdate) {
LastUpdated: update.timestamp, LastUpdated: update.timestamp,
} }
// For IPv4, calculate the IP range
if ipVersion == ipv4Version {
start, end, err := database.CalculateIPv4Range(update.prefix)
if err == nil {
liveRoute.V4IPStart = &start
liveRoute.V4IPEnd = &end
} else {
h.logger.Error("Failed to calculate IPv4 range",
"prefix", update.prefix,
"error", err,
)
}
}
if err := h.db.UpsertLiveRoute(liveRoute); err != nil { if err := h.db.UpsertLiveRoute(liveRoute); err != nil {
h.logger.Error("Failed to upsert live route", h.logger.Error("Failed to upsert live route",
"prefix", update.prefix, "prefix", update.prefix,
internal/routewatch/prefixhandler_test.go
-213
View File
@@ -1,213 +0,0 @@
package routewatch
import (
"context"
"errors"
"slices"
"testing"
"time"
"git.eeqj.de/sneak/routewatch/internal/config"
"git.eeqj.de/sneak/routewatch/internal/database"
"git.eeqj.de/sneak/routewatch/internal/logger"
"git.eeqj.de/sneak/routewatch/internal/ristypes"
)
const testPeerIP = "2001:db8:ffff::1"
// TestPrefixHandlerStoresPrefixesTheIPLookupFinds runs announcements and
// withdrawals through the prefix handler into a real database and looks the
// addresses up. The prefixes are written the way the feed sends them: IPv6
// announcements compressed, IPv6 withdrawals uncompressed. The withdrawal must
// remove the route the announcement stored, and the IP lookup must find the
// stored prefix.
func TestPrefixHandlerStoresPrefixesTheIPLookupFinds(t *testing.T) {
db, err := database.New(&config.Config{StateDir: t.TempDir()}, logger.New())
if err != nil {
t.Fatalf("failed to create database: %v", err)
}
defer func() { _ = db.Close() }()
// Built without NewPrefixHandler's flush timer, so the test flushes itself.
h := &PrefixHandler{db: db, logger: logger.New()}
flush := func() {
h.mu.Lock()
defer h.mu.Unlock()
h.flushBatchLocked()
}
ts := time.Date(2026, 1, 2, 3, 4, 5, 0, time.UTC)
h.HandleMessage(&ristypes.RISMessage{
Peer: testPeerIP,
Path: ristypes.ASPath{testASNA, testASNB},
ParsedTimestamp: ts,
Announcements: []ristypes.RISAnnouncement{{
NextHop: testPeerIP,
Prefixes: []string{"2001:db8:1::/48", "192.0.2.0/24"},
}},
})
flush()
for ip, want := range map[string]string{
"2001:db8:1::1": "2001:db8:1::/48",
"192.0.2.1": "192.0.2.0/24",
} {
info, err := db.GetIPInfo(ip)
if err != nil {
t.Fatalf("GetIPInfo(%s) after announcement: %v", ip, err)
}
if info.Netblock != want || info.ASN != testASNB {
t.Errorf("GetIPInfo(%s) = %s AS%d, want %s AS%d", ip, info.Netblock, info.ASN, want, testASNB)
}
}
h.HandleMessage(&ristypes.RISMessage{
Peer: testPeerIP,
ParsedTimestamp: ts.Add(time.Minute),
Withdrawals: []string{"2001:db8:1:0:0:0:0:0/48", "192.0.2.0/24"},
})
flush()
for _, ip := range []string{"2001:db8:1::1", "192.0.2.1"} {
if info, err := db.GetIPInfo(ip); !errors.Is(err, database.ErrNoRoute) {
t.Errorf("GetIPInfo(%s) after withdrawal = %+v, %v; want ErrNoRoute", ip, info, err)
}
}
}
// TestPrefixHandlerAppliesEachPeersLastUpdateInABatch runs updates for one
// prefix through the prefix handler into a real database. All updates carry the
// same timestamp, as updates within one second of the feed do. Within a batch,
// the last update each peer sent for the prefix decides whether that peer has a
// live route for it, and updates from different peers are all applied. The
// in-memory route count and prefix distribution must match the route table.
func TestPrefixHandlerAppliesEachPeersLastUpdateInABatch(t *testing.T) {
const (
prefix = "198.51.100.0/24"
prefixMaskLength = 24
otherPeerIP = "2001:db8:ffff::2"
)
ts := time.Date(2026, 1, 2, 3, 4, 5, 0, time.UTC)
announce := func(peer string) *ristypes.RISMessage {
return &ristypes.RISMessage{
Peer: peer,
Path: ristypes.ASPath{testASNA, testASNB},
ParsedTimestamp: ts,
Announcements: []ristypes.RISAnnouncement{{NextHop: peer, Prefixes: []string{prefix}}},
}
}
withdraw := func(peer string) *ristypes.RISMessage {
return &ristypes.RISMessage{Peer: peer, ParsedTimestamp: ts, Withdrawals: []string{prefix}}
}
announceAndWithdraw := func(peer string) *ristypes.RISMessage {
msg := announce(peer)
msg.Withdrawals = []string{prefix}
return msg
}
tests := []struct {
name string
// Each inner list is handled and then flushed as one batch.
batches [][]*ristypes.RISMessage
// The peers left with a live route for the prefix, by peer IP.
wantPeers []string
}{
{
name: "two peers announce the prefix",
batches: [][]*ristypes.RISMessage{{announce(testPeerIP), announce(otherPeerIP)}},
wantPeers: []string{testPeerIP, otherPeerIP},
},
{
name: "two peers withdraw the prefix",
batches: [][]*ristypes.RISMessage{
{announce(testPeerIP)},
{announce(otherPeerIP)},
{withdraw(testPeerIP), withdraw(otherPeerIP)},
},
},
{
name: "a peer announces and then withdraws the prefix",
batches: [][]*ristypes.RISMessage{{announce(testPeerIP), withdraw(testPeerIP)}},
},
{
name: "a peer withdraws and then re-announces the prefix",
batches: [][]*ristypes.RISMessage{
{announce(testPeerIP)},
{withdraw(testPeerIP), announce(testPeerIP)},
},
wantPeers: []string{testPeerIP},
},
{
// RFC 4271 section 4.3: the announcement counts, not the withdrawal.
name: "one message announces and withdraws the prefix",
batches: [][]*ristypes.RISMessage{
{announce(testPeerIP)},
{announceAndWithdraw(testPeerIP)},
},
wantPeers: []string{testPeerIP},
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
db, err := database.New(&config.Config{StateDir: t.TempDir()}, logger.New())
if err != nil {
t.Fatalf("failed to create database: %v", err)
}
defer func() { _ = db.Close() }()
// Built without NewPrefixHandler's flush timer, so the test flushes itself.
h := &PrefixHandler{db: db, logger: logger.New()}
for _, batch := range tt.batches {
for _, msg := range batch {
h.HandleMessage(msg)
}
h.mu.Lock()
h.flushBatchLocked()
h.mu.Unlock()
}
ctx := context.Background()
routes, _, err := db.GetPrefixDetailsContext(ctx, prefix)
if err != nil && !errors.Is(err, database.ErrNoRoute) {
t.Fatalf("GetPrefixDetailsContext: %v", err)
}
var gotPeers []string
for _, route := range routes {
gotPeers = append(gotPeers, route.PeerIP)
}
if !slices.Equal(gotPeers, tt.wantPeers) {
t.Errorf("peers with a live route for %s = %v, want %v", prefix, gotPeers, tt.wantPeers)
}
stats, err := db.GetStatsContext(ctx)
if err != nil {
t.Fatalf("GetStatsContext: %v", err)
}
tableRoutesV4, _, err := db.GetLiveRouteCountsContext(ctx)
if err != nil {
t.Fatalf("GetLiveRouteCountsContext: %v", err)
}
if stats.IPv4Routes != len(tt.wantPeers) || tableRoutesV4 != len(tt.wantPeers) {
t.Errorf("IPv4 live routes: %d in memory, %d in the table, want %d",
stats.IPv4Routes, tableRoutesV4, len(tt.wantPeers))
}
var wantDistribution []database.PrefixDistribution
if len(tt.wantPeers) > 0 {
wantDistribution = []database.PrefixDistribution{{MaskLength: prefixMaskLength, Count: 1}}
}
queryDistributionV4, _, err := db.GetPrefixDistributionContext(ctx)
if err != nil {
t.Fatalf("GetPrefixDistributionContext: %v", err)
}
if !slices.Equal(stats.IPv4PrefixDistribution, wantDistribution) ||
!slices.Equal(queryDistributionV4, wantDistribution) {
t.Errorf("IPv4 prefix distribution: %v in memory, %v from the table, want %v",
stats.IPv4PrefixDistribution, queryDistributionV4, wantDistribution)
}
})
}
}
internal/server/handlers.go
+22 -74
View File
@@ -7,7 +7,6 @@ import (
"errors" "errors"
"net" "net"
"net/http" "net/http"
"net/netip"
"runtime" "runtime"
"sort" "sort"
"strconv" "strconv"
@@ -180,37 +179,14 @@ func (s *Server) handleStatusJSON() http.HandlerFunc {
metrics := s.streamer.GetMetrics() metrics := s.streamer.GetMetrics()
// Get database stats with timeout. The channels are buffered so the // Serve database statistics from the cache, which runs the table scans at
// goroutine's send never blocks if the timeout wins and nothing here // most once per interval so this request does not.
// receives; otherwise it would block forever and leak. dbStats, err := s.stats.get()
statsChan := make(chan database.Stats, 1)
errChan := make(chan error, 1)
go func() {
dbStats, err := s.db.GetStatsContext(ctx)
if err != nil { if err != nil {
s.logger.Debug("Database stats query failed", "error", err)
errChan <- err
return
}
statsChan <- dbStats
}()
var dbStats database.Stats
select {
case <-ctx.Done():
s.logger.Error("Database stats timeout in status.json")
writeJSONError(w, http.StatusRequestTimeout, "Database timeout")
return
case err := <-errChan:
s.logger.Error("Failed to get database stats", "error", err) s.logger.Error("Failed to get database stats", "error", err)
writeJSONError(w, http.StatusInternalServerError, err.Error()) writeJSONError(w, http.StatusInternalServerError, err.Error())
return return
case dbStats = <-statsChan:
// Success
} }
uptime := time.Since(metrics.ConnectedSince).Truncate(time.Second).String() uptime := time.Since(metrics.ConnectedSince).Truncate(time.Second).String()
@@ -394,36 +370,14 @@ func (s *Server) handleStats() http.HandlerFunc {
metrics := s.streamer.GetMetrics() metrics := s.streamer.GetMetrics()
// Get database stats with timeout. The channels are buffered so the // Serve database statistics from the cache, which runs the table scans at
// goroutine's send never blocks if the timeout wins and nothing here // most once per interval so this request does not.
// receives; otherwise it would block forever and leak. dbStats, err := s.stats.get()
statsChan := make(chan database.Stats, 1)
errChan := make(chan error, 1)
go func() {
dbStats, err := s.db.GetStatsContext(ctx)
if err != nil { if err != nil {
s.logger.Debug("Database stats query failed", "error", err)
errChan <- err
return
}
statsChan <- dbStats
}()
var dbStats database.Stats
select {
case <-ctx.Done():
s.logger.Error("Database stats timeout")
// Don't write response here - timeout middleware already handles it
return
case err := <-errChan:
s.logger.Error("Failed to get database stats", "error", err) s.logger.Error("Failed to get database stats", "error", err)
writeJSONError(w, http.StatusInternalServerError, err.Error()) writeJSONError(w, http.StatusInternalServerError, err.Error())
return return
case dbStats = <-statsChan:
// Success
} }
uptime := time.Since(metrics.ConnectedSince).Truncate(time.Second).String() uptime := time.Since(metrics.ConnectedSince).Truncate(time.Second).String()
@@ -749,17 +703,10 @@ func (s *Server) handlePrefixDetailJSON() http.HandlerFunc {
return return
} }
// Look the prefix up in the text form net/netip prints, the form the // Combine prefix and length into CIDR notation
// tables store, so that any valid way of writing it finds the route. prefix := prefixParam + "/" + lenParam
parsed, err := netip.ParsePrefix(prefixParam + "/" + lenParam)
if err != nil {
writeJSONError(w, http.StatusBadRequest, err.Error())
return routes, err := s.db.GetPrefixDetailsContext(r.Context(), prefix)
}
prefix := parsed.Masked().String()
routes, _, err := s.db.GetPrefixDetailsContext(r.Context(), prefix)
if err != nil { if err != nil {
if errors.Is(err, database.ErrNoRoute) { if errors.Is(err, database.ErrNoRoute) {
writeJSONError(w, http.StatusNotFound, err.Error()) writeJSONError(w, http.StatusNotFound, err.Error())
@@ -920,17 +867,10 @@ func (s *Server) handlePrefixDetail() http.HandlerFunc {
return return
} }
// Look the prefix up in the text form net/netip prints, the form the // Combine prefix and length into CIDR notation
// tables store, so that any valid way of writing it finds the route. prefix := prefixParam + "/" + lenParam
parsed, err := netip.ParsePrefix(prefixParam + "/" + lenParam)
if err != nil {
http.Error(w, "Invalid prefix", http.StatusBadRequest)
return routes, err := s.db.GetPrefixDetailsContext(r.Context(), prefix)
}
prefix := parsed.Masked().String()
routes, originASNs, err := s.db.GetPrefixDetailsContext(r.Context(), prefix)
if err != nil { if err != nil {
if errors.Is(err, database.ErrNoRoute) { if errors.Is(err, database.ErrNoRoute) {
http.Error(w, "Prefix not found", http.StatusNotFound) http.Error(w, "Prefix not found", http.StatusNotFound)
@@ -952,10 +892,18 @@ func (s *Server) handlePrefixDetail() http.HandlerFunc {
originMap := make(map[int]*ASNInfo) originMap := make(map[int]*ASNInfo)
for _, route := range routes { for _, route := range routes {
if _, exists := originMap[route.OriginASN]; !exists { if _, exists := originMap[route.OriginASN]; !exists {
// Get AS info from database
asInfo, _, _ := s.db.GetASDetailsContext(r.Context(), route.OriginASN)
handle := ""
description := ""
if asInfo != nil {
handle = asInfo.Handle
description = asInfo.Description
}
originMap[route.OriginASN] = &ASNInfo{ originMap[route.OriginASN] = &ASNInfo{
ASN: route.OriginASN, ASN: route.OriginASN,
Handle: originASNs[route.OriginASN].Handle, Handle: handle,
Description: originASNs[route.OriginASN].Description, Description: description,
PeerCount: 0, PeerCount: 0,
} }
} }
internal/server/handlers_test.go
+27 -268
View File
@@ -2,298 +2,57 @@ package server
import ( import (
"context" "context"
"encoding/json"
"errors"
"html"
"net/http" "net/http"
"net/http/httptest" "net/http/httptest"
"runtime" "sync/atomic"
"slices"
"strings"
"testing" "testing"
"time"
"git.eeqj.de/sneak/routewatch/internal/config"
"git.eeqj.de/sneak/routewatch/internal/database" "git.eeqj.de/sneak/routewatch/internal/database"
"git.eeqj.de/sneak/routewatch/internal/logger" "git.eeqj.de/sneak/routewatch/internal/logger"
"git.eeqj.de/sneak/routewatch/internal/metrics" "git.eeqj.de/sneak/routewatch/internal/metrics"
"git.eeqj.de/sneak/routewatch/internal/streamer" "git.eeqj.de/sneak/routewatch/internal/streamer"
"git.eeqj.de/sneak/routewatch/pkg/asinfo"
"github.com/google/uuid"
) )
// blockingStatsDB embeds database.Store (left nil) and overrides only // countingStatsDB embeds database.Store (left nil) and overrides only
// GetStatsContext, which blocks until release is closed. The stats handlers // GetStatsContext, counting how many times it runs. The stats handlers read
// call it in a goroutine; every other Store method is unused on the timeout // their database statistics through the cache, which calls this; every other
// path and would panic if called. // Store method is unused on the stats path and would panic if called.
type blockingStatsDB struct { type countingStatsDB struct {
database.Store database.Store
release chan struct{} calls *atomic.Int64
} }
func (d blockingStatsDB) GetStatsContext(_ context.Context) (database.Stats, error) { func (d countingStatsDB) GetStatsContext(_ context.Context) (database.Stats, error) {
<-d.release d.calls.Add(1)
return database.Stats{}, nil return database.Stats{}, nil
} }
// TestStatsHandlersDoNotLeakOnTimeout drives each stats handler repeatedly with // TestStatsHandlersServeFromCache drives both stats handlers many times and
// a request whose context times out before the database responds, then releases // checks that they answer 200 while the database statistics are computed at most
// the blocked queries and asserts the goroutine count returns to its starting // once within the refresh interval. Before the fix each request ran the counts
// value. Before the fix the per-request goroutine sent on an unbuffered channel // and MIN/MAX scans itself, which took the full timeout and returned 500 once
// that nothing received once the timeout won, so it blocked forever and every // the database grew large.
// poll leaked one goroutine. func TestStatsHandlersServeFromCache(t *testing.T) {
func TestStatsHandlersDoNotLeakOnTimeout(t *testing.T) { var calls atomic.Int64
release := make(chan struct{}) db := countingStatsDB{calls: &calls}
db := blockingStatsDB{release: release} s := New(db, streamer.New(logger.New(), metrics.New()), logger.New())
s := New(db, streamer.New(logger.New(), metrics.New()), logger.New(), &config.Config{})
handlers := map[string]http.HandlerFunc{ handlers := []http.HandlerFunc{s.handleStatusJSON(), s.handleStats()}
"status.json": s.handleStatusJSON(),
"stats": s.handleStats(),
}
baseline := settledGoroutineCount() const iterations = 20
const (
iterations = 20
requestTimeout = 50 * time.Millisecond
)
for _, handler := range handlers { for _, handler := range handlers {
for range iterations { for range iterations {
ctx, cancel := context.WithTimeout(context.Background(), requestTimeout) req := httptest.NewRequest(http.MethodGet, "/", nil)
req := httptest.NewRequest(http.MethodGet, "/", nil).WithContext(ctx)
handler(httptest.NewRecorder(), req)
cancel()
}
}
// Let the blocked queries finish; with buffered channels each goroutine's
// send now succeeds and the goroutine exits.
close(release)
if !waitForGoroutines(baseline) {
t.Fatalf("goroutines did not return to baseline %d, got %d",
baseline, runtime.NumGoroutine())
}
}
// TestStatsHandlersAnswerFromMemory checks that both stats handlers answer 200
// with the live route counts and the prefix distribution while the database is
// closed, so that any query would fail: the request path reads them from
// memory. The prefix distribution query it used to run read every live route
// and, on a large database, took the whole 4-second deadline, so
// /api/v1/stats answered 500 (https://git.eeqj.de/sneak/routewatch/issues/30).
// The oldest and newest route times still come from one-row lookups at the ends
// of an index; with the database closed they are left out of the answer.
func TestStatsHandlersAnswerFromMemory(t *testing.T) {
db, err := database.New(&config.Config{StateDir: t.TempDir()}, logger.New())
if err != nil {
t.Fatalf("database.New: %v", err)
}
ts := time.Date(2026, 1, 2, 3, 4, 5, 0, time.UTC)
if err := db.UpsertLiveRouteBatch([]*database.LiveRoute{
{
ID: uuid.New(), Prefix: "198.51.100.0/24", MaskLength: 24, IPVersion: 4,
OriginASN: 64500, PeerIP: "192.0.2.1", ASPath: []int{64500}, NextHop: "192.0.2.1",
LastUpdated: ts,
},
{
ID: uuid.New(), Prefix: "2001:db8::/32", MaskLength: 32, IPVersion: 6,
OriginASN: 64501, PeerIP: "2001:db8::1", ASPath: []int{64501}, NextHop: "2001:db8::1",
LastUpdated: ts,
},
}); err != nil {
t.Fatalf("UpsertLiveRouteBatch: %v", err)
}
if err := db.Close(); err != nil {
t.Fatalf("Close: %v", err)
}
s := New(db, streamer.New(logger.New(), metrics.New()), logger.New(), &config.Config{})
handlers := map[string]http.HandlerFunc{
"status.json": s.handleStatusJSON(),
"stats": s.handleStats(),
}
for name, handler := range handlers {
rec := httptest.NewRecorder() rec := httptest.NewRecorder()
handler(rec, httptest.NewRequest(http.MethodGet, "/", nil)) handler(rec, req)
if rec.Code != http.StatusOK { if rec.Code != http.StatusOK {
t.Errorf("%s: status %d, want %d; body %s", name, rec.Code, http.StatusOK, rec.Body) t.Fatalf("handler returned %d, want %d", rec.Code, http.StatusOK)
}
continue }
} }
var body struct { if got := calls.Load(); got != 1 {
Data struct { t.Fatalf("GetStatsContext ran %d times, want 1 within the interval", got)
IPv4Routes int `json:"ipv4_routes"`
IPv6Routes int `json:"ipv6_routes"`
IPv4PrefixDistribution []database.PrefixDistribution `json:"ipv4_prefix_distribution"`
IPv6PrefixDistribution []database.PrefixDistribution `json:"ipv6_prefix_distribution"`
} `json:"data"`
}
if err := json.Unmarshal(rec.Body.Bytes(), &body); err != nil {
t.Fatalf("%s: decoding the answer: %v", name, err)
}
if body.Data.IPv4Routes != 1 || body.Data.IPv6Routes != 1 {
t.Errorf("%s: routes = (v4 %d, v6 %d), want (1, 1)", name, body.Data.IPv4Routes, body.Data.IPv6Routes)
}
wantV4 := []database.PrefixDistribution{{MaskLength: 24, Count: 1}}
if !slices.Equal(body.Data.IPv4PrefixDistribution, wantV4) {
t.Errorf("%s: IPv4 distribution = %v, want %v", name, body.Data.IPv4PrefixDistribution, wantV4)
}
wantV6 := []database.PrefixDistribution{{MaskLength: 32, Count: 1}}
if !slices.Equal(body.Data.IPv6PrefixDistribution, wantV6) {
t.Errorf("%s: IPv6 distribution = %v, want %v", name, body.Data.IPv6PrefixDistribution, wantV6)
}
} }
} }
// TestPrefixDetailHandlersParsePrefix checks that both prefix pages find a
// stored route whatever valid text form the URL writes its prefix in, and
// answer 400 for a prefix that does not parse. The tables store the form
// net/netip prints. Before the fix the URL text was looked up as typed, so
// "2001:DB8:0:0::/32" was not found although "2001:db8::/32" was stored, and
// "foo/24" answered 500 (https://git.eeqj.de/sneak/routewatch/issues/71).
func TestPrefixDetailHandlersParsePrefix(t *testing.T) {
db, err := database.New(&config.Config{StateDir: t.TempDir()}, logger.New())
if err != nil {
t.Fatalf("database.New: %v", err)
}
t.Cleanup(func() { _ = db.Close() })
ts := time.Date(2026, 1, 2, 3, 4, 5, 0, time.UTC)
if err := db.UpsertLiveRouteBatch([]*database.LiveRoute{
{
ID: uuid.New(), Prefix: "192.0.2.0/24", MaskLength: 24, IPVersion: 4,
OriginASN: 64500, PeerIP: "198.51.100.1", ASPath: []int{64500}, NextHop: "198.51.100.1",
LastUpdated: ts,
},
{
ID: uuid.New(), Prefix: "2001:db8::/32", MaskLength: 32, IPVersion: 6,
OriginASN: 64501, PeerIP: "2001:db8::1", ASPath: []int{64501}, NextHop: "2001:db8::1",
LastUpdated: ts,
},
}); err != nil {
t.Fatalf("UpsertLiveRouteBatch: %v", err)
}
s := New(db, streamer.New(logger.New(), metrics.New()), logger.New(), &config.Config{})
tests := []struct {
path string
want int
}{
{"/api/v1/prefix/foo/24", http.StatusBadRequest},
{"/prefix/foo/24", http.StatusBadRequest},
{"/api/v1/prefix/2001:DB8:0:0::/32", http.StatusOK},
{"/prefix/2001:DB8:0:0::/32", http.StatusOK},
{"/api/v1/prefix/192.0.2.1/24", http.StatusOK},
{"/prefix/192.0.2.1/24", http.StatusOK},
}
for _, tt := range tests {
rec := httptest.NewRecorder()
s.router.ServeHTTP(rec, httptest.NewRequest(http.MethodGet, tt.path, nil))
if rec.Code != tt.want {
t.Errorf("%s: status %d, want %d; body %s", tt.path, rec.Code, tt.want, rec.Body)
}
}
}
// failingASDetailsDB wraps a real store and fails every AS details lookup.
type failingASDetailsDB struct {
database.Store
}
func (failingASDetailsDB) GetASDetailsContext(_ context.Context, _ int) (*database.ASN, []database.LiveRoute, error) {
return nil, nil, errors.New("AS details lookup failed")
}
// TestPrefixDetailShowsOriginNames checks that the prefix page shows each
// origin AS's handle and description without looking up the AS's details,
// which also read every route the AS originates. Before the fix the page did
// that lookup once per origin, so with it failing the names were missing
// (https://git.eeqj.de/sneak/routewatch/issues/67). The AS path column shows
// handles too, so the test looks for the origin list's "(handle)" form.
func TestPrefixDetailShowsOriginNames(t *testing.T) {
db, err := database.New(&config.Config{StateDir: t.TempDir()}, logger.New())
if err != nil {
t.Fatalf("database.New: %v", err)
}
t.Cleanup(func() { _ = db.Close() })
ts := time.Date(2026, 1, 2, 3, 4, 5, 0, time.UTC)
origins := []int{15169, 13335}
for _, asn := range origins {
if _, err := db.GetOrCreateASN(asn, ts); err != nil {
t.Fatalf("GetOrCreateASN(%d): %v", asn, err)
}
}
if err := db.UpsertLiveRouteBatch([]*database.LiveRoute{
{
ID: uuid.New(), Prefix: "192.0.2.0/24", MaskLength: 24, IPVersion: 4,
OriginASN: origins[0], PeerIP: "198.51.100.1", ASPath: []int{origins[0]}, NextHop: "198.51.100.1",
LastUpdated: ts,
},
{
ID: uuid.New(), Prefix: "192.0.2.0/24", MaskLength: 24, IPVersion: 4,
OriginASN: origins[1], PeerIP: "198.51.100.2", ASPath: []int{origins[1]}, NextHop: "198.51.100.2",
LastUpdated: ts,
},
}); err != nil {
t.Fatalf("UpsertLiveRouteBatch: %v", err)
}
s := New(failingASDetailsDB{db}, streamer.New(logger.New(), metrics.New()), logger.New(), &config.Config{})
rec := httptest.NewRecorder()
s.router.ServeHTTP(rec, httptest.NewRequest(http.MethodGet, "/prefix/192.0.2.0/24", nil))
if rec.Code != http.StatusOK {
t.Fatalf("status %d, want %d; body %s", rec.Code, http.StatusOK, rec.Body)
}
page := rec.Body.String()
for _, asn := range origins {
info, ok := asinfo.Get(asn)
if !ok {
t.Fatalf("AS%d is not in the AS info data", asn)
}
if want := "(" + html.EscapeString(info.Handle) + ")"; !strings.Contains(page, want) {
t.Errorf("page does not show AS%d's handle %q", asn, want)
}
if want := html.EscapeString(info.Description); !strings.Contains(page, want) {
t.Errorf("page does not show AS%d's description %q", asn, want)
}
}
}
// settledGoroutineCount lets transient goroutines finish, then reports the
// current count.
func settledGoroutineCount() int {
prev := runtime.NumGoroutine()
for range 20 {
time.Sleep(10 * time.Millisecond)
cur := runtime.NumGoroutine()
if cur == prev {
return cur
}
prev = cur
}
return prev
}
// waitForGoroutines waits until the goroutine count drops to target or below.
func waitForGoroutines(target int) bool {
for range 100 {
if runtime.NumGoroutine() <= target {
return true
}
time.Sleep(10 * time.Millisecond)
}
return false
}
internal/server/server.go
+17 -21
View File
@@ -3,13 +3,10 @@ package server
import ( import (
"context" "context"
"fmt"
"net"
"net/http" "net/http"
"strconv" "os"
"time" "time"
"git.eeqj.de/sneak/routewatch/internal/config"
"git.eeqj.de/sneak/routewatch/internal/database" "git.eeqj.de/sneak/routewatch/internal/database"
"git.eeqj.de/sneak/routewatch/internal/logger" "git.eeqj.de/sneak/routewatch/internal/logger"
"git.eeqj.de/sneak/routewatch/internal/streamer" "git.eeqj.de/sneak/routewatch/internal/streamer"
@@ -36,31 +33,35 @@ type Server struct {
db database.Store db database.Store
streamer *streamer.Streamer streamer *streamer.Streamer
logger *logger.Logger logger *logger.Logger
port int
srv *http.Server srv *http.Server
asnFetcher ASNFetcher asnFetcher ASNFetcher
stats *statsCache
} }
// New creates a new HTTP server // New creates a new HTTP server
func New(db database.Store, streamer *streamer.Streamer, logger *logger.Logger, cfg *config.Config) *Server { func New(db database.Store, streamer *streamer.Streamer, logger *logger.Logger) *Server {
s := &Server{ s := &Server{
db: db, db: db,
streamer: streamer, streamer: streamer,
logger: logger, logger: logger,
port: cfg.Port,
} }
s.stats = newStatsCache(func(ctx context.Context) (database.Stats, error) {
return s.db.GetStatsContext(ctx)
})
s.setupRoutes() s.setupRoutes()
return s return s
} }
// Start listens on the configured port and serves HTTP in the background. // Start starts the HTTP server
// It returns an error if it cannot listen, for example because the port is
// already in use.
func (s *Server) Start() error { func (s *Server) Start() error {
port := os.Getenv("PORT")
if port == "" {
port = "8080"
}
const ( const (
network = "tcp"
readHeaderTimeout = 40 * time.Second readHeaderTimeout = 40 * time.Second
readTimeout = 60 * time.Second readTimeout = 60 * time.Second
writeTimeout = 60 * time.Second writeTimeout = 60 * time.Second
@@ -68,7 +69,7 @@ func (s *Server) Start() error {
) )
s.srv = &http.Server{ s.srv = &http.Server{
Addr: ":" + strconv.Itoa(s.port), Addr: ":" + port,
Handler: s.router, Handler: s.router,
ReadHeaderTimeout: readHeaderTimeout, ReadHeaderTimeout: readHeaderTimeout,
ReadTimeout: readTimeout, ReadTimeout: readTimeout,
@@ -76,17 +77,12 @@ func (s *Server) Start() error {
IdleTimeout: idleTimeout, IdleTimeout: idleTimeout,
} }
s.logger.Info("Starting HTTP server", "port", s.port, "addr", s.srv.Addr) s.logger.Info("Starting HTTP server", "port", port, "addr", s.srv.Addr)
listener, err := net.Listen(network, s.srv.Addr)
if err != nil {
return fmt.Errorf("starting HTTP server: %w", err)
}
s.logger.Info("HTTP server listening", "addr", listener.Addr().String())
// Start in goroutine but log when actually listening
go func() { go func() {
if err := s.srv.Serve(listener); err != nil && err != http.ErrServerClosed { s.logger.Info("HTTP server listening", "addr", s.srv.Addr)
if err := s.srv.ListenAndServe(); err != nil && err != http.ErrServerClosed {
s.logger.Error("HTTP server error", "error", err) s.logger.Error("HTTP server error", "error", err)
} }
}() }()
internal/server/server_test.go
-33
View File
@@ -1,33 +0,0 @@
package server
import (
"context"
"net"
"testing"
"git.eeqj.de/sneak/routewatch/internal/config"
"git.eeqj.de/sneak/routewatch/internal/logger"
"git.eeqj.de/sneak/routewatch/internal/metrics"
"git.eeqj.de/sneak/routewatch/internal/streamer"
)
// TestStartFailsWhenPortIsHeld holds a port with a listener of its own and
// checks that Start on that port returns an error. Before the fix Start
// listened in a goroutine, only logged the failure and returned nil, so the
// daemon kept running without its web server.
func TestStartFailsWhenPortIsHeld(t *testing.T) {
held, err := net.Listen("tcp", ":0")
if err != nil {
t.Fatalf("failed to open the test listener: %v", err)
}
defer func() { _ = held.Close() }()
port := held.Addr().(*net.TCPAddr).Port
log := logger.New()
s := New(nil, streamer.New(log, metrics.New()), log, &config.Config{Port: port})
defer func() { _ = s.Stop(context.Background()) }()
if err := s.Start(); err == nil {
t.Fatalf("Start returned nil while port %d was held by another listener", port)
}
}
internal/server/statscache.go
+112
View File
@@ -0,0 +1,112 @@
package server
import (
"context"
"sync"
"time"
"git.eeqj.de/sneak/routewatch/internal/database"
)
const (
// statsRefreshInterval is how often the cached database statistics are
// recomputed. The scans behind GetStatsContext grow with the tables, so a
// request serves the cached copy instead of running them.
statsRefreshInterval = 30 * time.Second
// statsComputeTimeout bounds a single statistics computation so a stuck scan
// cannot block the refresh forever.
statsComputeTimeout = 20 * time.Second
)
// statsFetch computes fresh statistics. It is the expensive database scan that
// the cache runs at most once per interval.
type statsFetch func(ctx context.Context) (database.Stats, error)
// statsCache serves the most recent database statistics and recomputes them at
// most once per interval. The first request computes synchronously so it has
// real data to return; afterwards requests serve the cached copy immediately
// and a stale copy triggers a single background refresh, so no request waits on
// the scans.
type statsCache struct {
fetch statsFetch
interval time.Duration
now func() time.Time
mu sync.Mutex
stats database.Stats
haveStats bool
fetchedAt time.Time
refreshing bool
}
// newStatsCache returns a cache that recomputes statistics with fetch no more
// than once per statsRefreshInterval.
func newStatsCache(fetch statsFetch) *statsCache {
return &statsCache{
fetch: fetch,
interval: statsRefreshInterval,
now: time.Now,
}
}
// get returns the cached statistics. On the first call it computes them
// synchronously and returns any error. Later calls return the cached copy, and
// when that copy is older than the interval they start one background refresh.
func (c *statsCache) get() (database.Stats, error) {
c.mu.Lock()
if !c.haveStats {
// Cold start: compute once under the lock so concurrent first callers
// wait for this single computation rather than each starting their own.
stats, err := c.compute()
if err != nil {
c.mu.Unlock()
return database.Stats{}, err
}
c.store(stats)
c.mu.Unlock()
return stats, nil
}
if c.now().Sub(c.fetchedAt) >= c.interval && !c.refreshing {
c.refreshing = true
go c.refresh()
}
stats := c.stats
c.mu.Unlock()
return stats, nil
}
// refresh recomputes the statistics in the background and replaces the cached
// copy. A failed computation leaves the previous copy in place.
func (c *statsCache) refresh() {
stats, err := c.compute()
c.mu.Lock()
defer c.mu.Unlock()
c.refreshing = false
if err == nil {
c.store(stats)
}
}
// compute runs the fetch with its own bounded context, independent of any
// request, so one request's cancellation cannot abort a shared refresh.
func (c *statsCache) compute() (database.Stats, error) {
ctx, cancel := context.WithTimeout(context.Background(), statsComputeTimeout)
defer cancel()
return c.fetch(ctx)
}
// store records a fresh result. The caller must hold the mutex.
func (c *statsCache) store(stats database.Stats) {
c.stats = stats
c.haveStats = true
c.fetchedAt = c.now()
}
internal/server/statscache_test.go
+173
View File
@@ -0,0 +1,173 @@
package server
import (
"context"
"errors"
"runtime"
"sync/atomic"
"testing"
"time"
"git.eeqj.de/sneak/routewatch/internal/database"
)
// testClock is a concurrency-safe clock the cache tests advance by hand, so the
// interval boundary is exercised without waiting real time.
type testClock struct {
ns atomic.Int64
}
func (c *testClock) now() time.Time { return time.Unix(0, c.ns.Load()) }
func (c *testClock) advance(d time.Duration) { c.ns.Add(int64(d)) }
// waitForCalls waits until calls reaches want, giving a background refresh time
// to finish.
func waitForCalls(calls *atomic.Int64, want int64) bool {
const attempts = 200
for range attempts {
if calls.Load() >= want {
return true
}
time.Sleep(5 * time.Millisecond)
}
return false
}
// TestStatsCacheComputesOncePerInterval is the core guarantee: many reads in a
// row run the expensive fetch at most once per interval, and crossing the
// interval boundary allows exactly one more computation.
func TestStatsCacheComputesOncePerInterval(t *testing.T) {
clk := &testClock{}
clk.ns.Store(int64(time.Hour)) // start at a non-zero instant
var calls atomic.Int64
c := newStatsCache(func(_ context.Context) (database.Stats, error) {
calls.Add(1)
return database.Stats{}, nil
})
c.now = clk.now
const reads = 50
for range reads {
if _, err := c.get(); err != nil {
t.Fatalf("get returned error: %v", err)
}
}
if got := calls.Load(); got != 1 {
t.Fatalf("fetch ran %d times within the interval, want 1", got)
}
// Cross the interval: the next read serves the stale copy and starts one
// background refresh.
clk.advance(c.interval)
if _, err := c.get(); err != nil {
t.Fatalf("get after interval returned error: %v", err)
}
if !waitForCalls(&calls, 2) {
t.Fatalf("background refresh did not run, fetch ran %d times", calls.Load())
}
for range reads {
if _, err := c.get(); err != nil {
t.Fatalf("get returned error: %v", err)
}
}
if got := calls.Load(); got != 2 {
t.Fatalf("fetch ran %d times across one interval boundary, want 2", got)
}
}
// TestStatsCacheColdStartReturnsError checks the first computation's error
// reaches the caller, since there is no cached copy to serve instead.
func TestStatsCacheColdStartReturnsError(t *testing.T) {
wantErr := errors.New("boom")
c := newStatsCache(func(_ context.Context) (database.Stats, error) {
return database.Stats{}, wantErr
})
if _, err := c.get(); !errors.Is(err, wantErr) {
t.Fatalf("get returned %v, want %v", err, wantErr)
}
}
// TestStatsCacheServesLastGoodCopyOnRefreshError checks that once a copy exists,
// a later failing refresh does not surface an error or drop the good data.
func TestStatsCacheServesLastGoodCopyOnRefreshError(t *testing.T) {
clk := &testClock{}
clk.ns.Store(int64(time.Hour))
const wantASNs = 7
var calls atomic.Int64
var failing atomic.Bool
c := newStatsCache(func(_ context.Context) (database.Stats, error) {
calls.Add(1)
if failing.Load() {
return database.Stats{}, errors.New("boom")
}
return database.Stats{ASNs: wantASNs}, nil
})
c.now = clk.now
got, err := c.get()
if err != nil || got.ASNs != wantASNs {
t.Fatalf("cold start returned (%+v, %v), want ASNs=%d, nil", got, err, wantASNs)
}
failing.Store(true)
clk.advance(c.interval)
got, err = c.get()
if err != nil {
t.Fatalf("get during failing refresh returned error: %v", err)
}
if got.ASNs != wantASNs {
t.Fatalf("get returned ASNs=%d, want the last good copy %d", got.ASNs, wantASNs)
}
if !waitForCalls(&calls, 2) {
t.Fatalf("refresh was not attempted, fetch ran %d times", calls.Load())
}
}
// TestStatsCacheBackgroundRefreshDoesNotLeak forces many stale refreshes and
// checks the goroutine count returns to its starting value.
func TestStatsCacheBackgroundRefreshDoesNotLeak(t *testing.T) {
clk := &testClock{}
clk.ns.Store(int64(time.Hour))
var calls atomic.Int64
c := newStatsCache(func(_ context.Context) (database.Stats, error) {
calls.Add(1)
return database.Stats{}, nil
})
c.now = clk.now
if _, err := c.get(); err != nil {
t.Fatalf("cold start returned error: %v", err)
}
baseline := runtime.NumGoroutine()
const rounds = 20
for i := range rounds {
clk.advance(c.interval)
if _, err := c.get(); err != nil {
t.Fatalf("get returned error: %v", err)
}
if !waitForCalls(&calls, int64(i+2)) {
t.Fatalf("refresh %d did not run", i)
}
}
const settleAttempts = 100
for range settleAttempts {
if runtime.NumGoroutine() <= baseline {
return
}
time.Sleep(10 * time.Millisecond)
}
t.Fatalf("goroutines did not settle to baseline %d, got %d", baseline, runtime.NumGoroutine())
}
internal/streamer/streamer.go
+16 -50
View File
@@ -7,7 +7,6 @@ import (
"compress/gzip" "compress/gzip"
"context" "context"
"encoding/json" "encoding/json"
"errors"
"fmt" "fmt"
"io" "io"
"math" "math"
@@ -49,7 +48,6 @@ const (
metricsUpdateRate = time.Second metricsUpdateRate = time.Second
minBackoffDelay = 5 * time.Second minBackoffDelay = 5 * time.Second
maxBackoffDelay = 320 * time.Second maxBackoffDelay = 320 * time.Second
streamIdleTimeout = 60 * time.Second // Reconnect when no line arrives for this long
metricsLogInterval = 10 * time.Second metricsLogInterval = 10 * time.Second
bytesPerKB = 1024 bytesPerKB = 1024
bytesPerMB = 1024 * 1024 bytesPerMB = 1024 * 1024
@@ -60,9 +58,6 @@ const (
backpressureSlope = 2.0 // Slope for linear drop probability increase backpressureSlope = 2.0 // Slope for linear drop probability increase
) )
// errStreamEnded is returned by stream when RIS Live ends the response.
var errStreamEnded = errors.New("RIS Live ended the stream")
// MessageHandler defines the interface for processing RIS messages. // MessageHandler defines the interface for processing RIS messages.
// Implementations must specify which message types they want to receive, // Implementations must specify which message types they want to receive,
// how to process messages, and their desired queue capacity. // how to process messages, and their desired queue capacity.
@@ -112,8 +107,6 @@ type Streamer struct {
logger *logger.Logger logger *logger.Logger
client *http.Client client *http.Client
url string url string
idleTimeout time.Duration // Drop the connection when no line arrives for this long
minBackoff time.Duration // First wait before reconnecting; doubles up to maxBackoffDelay
handlers []*handlerInfo handlers []*handlerInfo
rawHandler RawMessageHandler rawHandler RawMessageHandler
mu sync.RWMutex mu sync.RWMutex
@@ -133,8 +126,6 @@ func New(logger *logger.Logger, metrics *metrics.Tracker) *Streamer {
return &Streamer{ return &Streamer{
logger: logger, logger: logger,
url: risLiveURL, url: risLiveURL,
idleTimeout: streamIdleTimeout,
minBackoff: minBackoffDelay,
client: &http.Client{ client: &http.Client{
Timeout: 0, // No timeout for streaming Timeout: 0, // No timeout for streaming
Transport: &http.Transport{ Transport: &http.Transport{
@@ -219,14 +210,9 @@ func (s *Streamer) Start() error {
// the connection status in metrics. This method is safe to call multiple times. // the connection status in metrics. This method is safe to call multiple times.
func (s *Streamer) Stop() { func (s *Streamer) Stop() {
s.mu.Lock() s.mu.Lock()
if s.cancel == nil { if s.cancel != nil {
// Not started, or already stopped: closing the queues again would panic.
s.mu.Unlock()
return
}
s.cancel() s.cancel()
s.cancel = nil }
// Close all handler queues to signal workers to stop // Close all handler queues to signal workers to stop
for _, info := range s.handlers { for _, info := range s.handlers {
close(info.queue) close(info.queue)
@@ -414,7 +400,7 @@ func (s *Streamer) updateMetrics(messageBytes int) {
// streamWithReconnect handles streaming with automatic reconnection and exponential backoff // streamWithReconnect handles streaming with automatic reconnection and exponential backoff
func (s *Streamer) streamWithReconnect(ctx context.Context) { func (s *Streamer) streamWithReconnect(ctx context.Context) {
backoffDelay := s.minBackoff backoffDelay := minBackoffDelay
consecutiveFailures := 0 consecutiveFailures := 0
for { for {
@@ -431,9 +417,8 @@ func (s *Streamer) streamWithReconnect(ctx context.Context) {
err := s.stream(ctx) err := s.stream(ctx)
streamDuration := time.Since(startTime) streamDuration := time.Since(startTime)
// Only Stop ends the streamer. A stream that ends any other way, if err == nil {
// with or without an error, is a disconnect and is reconnected. // Clean exit (context cancelled)
if ctx.Err() != nil {
return return
} }
@@ -444,13 +429,17 @@ func (s *Streamer) streamWithReconnect(ctx context.Context) {
"stream_duration", streamDuration) "stream_duration", streamDuration)
s.metrics.SetConnected(false) s.metrics.SetConnected(false)
// A stream that lasted more than 30 seconds resets the backoff. That // Check if context is cancelled
// includes a stream dropped after streamIdleTimeout with no line, so if ctx.Err() != nil {
// a silent feed is retried after the first backoff delay every time. return
}
// If we streamed for more than 30 seconds, reset the backoff
// This indicates we had a successful connection that received data
if streamDuration > 30*time.Second { if streamDuration > 30*time.Second {
s.logger.Info("Resetting backoff delay due to successful connection", s.logger.Info("Resetting backoff delay due to successful connection",
"stream_duration", streamDuration) "stream_duration", streamDuration)
backoffDelay = s.minBackoff backoffDelay = minBackoffDelay
consecutiveFailures = 0 consecutiveFailures = 0
} else { } else {
// Increment consecutive failures // Increment consecutive failures
@@ -483,13 +472,7 @@ func (s *Streamer) stream(ctx context.Context) error {
connCtx, connCancel := context.WithCancel(ctx) connCtx, connCancel := context.WithCancel(ctx)
defer connCancel() defer connCancel()
// The idle timer drops the connection, by cancelling connCtx, when no req, err := http.NewRequestWithContext(ctx, "GET", s.url, nil)
// line arrives for s.idleTimeout. Each line read starts it over. It
// also covers the wait for the server to answer.
idleTimer := time.AfterFunc(s.idleTimeout, connCancel)
defer idleTimer.Stop()
req, err := http.NewRequestWithContext(connCtx, "GET", s.url, nil)
if err != nil { if err != nil {
return fmt.Errorf("failed to create request: %w", err) return fmt.Errorf("failed to create request: %w", err)
} }
@@ -571,8 +554,6 @@ func (s *Streamer) stream(ctx context.Context) error {
scanner := bufio.NewScanner(reader) scanner := bufio.NewScanner(reader)
for scanner.Scan() { for scanner.Scan() {
idleTimer.Reset(s.idleTimeout)
select { select {
case <-ctx.Done(): case <-ctx.Done():
s.logger.Info("Stream stopped by context") s.logger.Info("Stream stopped by context")
@@ -679,15 +660,8 @@ func (s *Streamer) stream(ctx context.Context) error {
continue continue
} }
// Dispatch to interested handlers. Stop cancels ctx and closes the // Dispatch to interested handlers
// queues under the write lock, so if ctx is cancelled here, under the
// read lock, the queues are closed and must not be sent to.
s.mu.RLock() s.mu.RLock()
if ctx.Err() != nil {
s.mu.RUnlock()
return ctx.Err()
}
for _, info := range s.handlers { for _, info := range s.handlers {
if !info.handler.WantsMessage(msg.Type) { if !info.handler.WantsMessage(msg.Type) {
continue continue
@@ -721,19 +695,11 @@ func (s *Streamer) stream(ctx context.Context) error {
s.mu.RUnlock() s.mu.RUnlock()
} }
if ctx.Err() != nil {
return ctx.Err()
}
if connCtx.Err() != nil {
return fmt.Errorf("no line received for %s", s.idleTimeout)
}
if err := scanner.Err(); err != nil { if err := scanner.Err(); err != nil {
return fmt.Errorf("scanner error: %w", err) return fmt.Errorf("scanner error: %w", err)
} }
return errStreamEnded return nil
} }
// shouldDropForBackpressure determines if a message should be dropped based on queue utilization // shouldDropForBackpressure determines if a message should be dropped based on queue utilization
internal/streamer/streamer_test.go
+4 -178
View File
@@ -2,18 +2,14 @@ package streamer
import ( import (
"context" "context"
"errors"
"io"
"net/http" "net/http"
"net/http/httptest" "net/http/httptest"
"runtime" "runtime"
"sync/atomic"
"testing" "testing"
"time" "time"
"git.eeqj.de/sneak/routewatch/internal/logger" "git.eeqj.de/sneak/routewatch/internal/logger"
"git.eeqj.de/sneak/routewatch/internal/metrics" "git.eeqj.de/sneak/routewatch/internal/metrics"
"git.eeqj.de/sneak/routewatch/internal/ristypes"
) )
func TestNewStreamer(t *testing.T) { func TestNewStreamer(t *testing.T) {
@@ -40,14 +36,6 @@ func TestNewStreamer(t *testing.T) {
if s.metrics != metricsTracker { if s.metrics != metricsTracker {
t.Error("metrics tracker not set correctly") t.Error("metrics tracker not set correctly")
} }
if s.idleTimeout != streamIdleTimeout {
t.Errorf("idleTimeout = %s, want %s", s.idleTimeout, streamIdleTimeout)
}
if s.minBackoff != minBackoffDelay {
t.Errorf("minBackoff = %s, want %s", s.minBackoff, minBackoffDelay)
}
} }
// TestStreamDoesNotLeakTickersAcrossReconnects drives many short-lived // TestStreamDoesNotLeakTickersAcrossReconnects drives many short-lived
@@ -66,8 +54,8 @@ func TestStreamDoesNotLeakTickersAcrossReconnects(t *testing.T) {
// One warm-up connection so any persistent HTTP transport goroutine exists // One warm-up connection so any persistent HTTP transport goroutine exists
// before we take the baseline. // before we take the baseline.
if err := s.stream(context.Background()); !errors.Is(err, errStreamEnded) { if err := s.stream(context.Background()); err != nil {
t.Fatalf("warm-up stream returned %v, want %v", err, errStreamEnded) t.Fatalf("warm-up stream returned error: %v", err)
} }
s.client.CloseIdleConnections() s.client.CloseIdleConnections()
@@ -75,8 +63,8 @@ func TestStreamDoesNotLeakTickersAcrossReconnects(t *testing.T) {
const reconnects = 20 const reconnects = 20
for range reconnects { for range reconnects {
if err := s.stream(context.Background()); !errors.Is(err, errStreamEnded) { if err := s.stream(context.Background()); err != nil {
t.Fatalf("stream returned %v, want %v", err, errStreamEnded) t.Fatalf("stream returned error: %v", err)
} }
} }
s.client.CloseIdleConnections() s.client.CloseIdleConnections()
@@ -87,168 +75,6 @@ func TestStreamDoesNotLeakTickersAcrossReconnects(t *testing.T) {
} }
} }
// updateLine is one UPDATE message as RIS Live sends it.
const updateLine = `{"type":"ris_message","data":{"type":"UPDATE","peer":"192.0.2.1",` +
`"peer_asn":"64496","timestamp":1700000000}}` + "\n"
// updateHandler wants UPDATE messages. For each one it handles it sends on
// handled if there is room, and otherwise does nothing; a nil handled is
// never sent on.
type updateHandler struct {
handled chan struct{}
}
func (updateHandler) WantsMessage(messageType string) bool { return messageType == "UPDATE" }
func (updateHandler) QueueCapacity() int { return 10 }
func (h updateHandler) HandleMessage(*ristypes.RISMessage) {
select {
case h.handled <- struct{}{}:
default:
}
}
// TestStopBeforeMessageReachesQueues stops the streamer after the read loop
// has checked for cancellation but before it hands the message to the handler
// queues. That is the gap a stop from another goroutine can land in, and it
// used to end in "send on closed channel". The raw handler runs in that gap on
// the read loop itself, so calling Stop from it hits the gap every time.
func TestStopBeforeMessageReachesQueues(t *testing.T) {
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
_, _ = io.WriteString(w, updateLine)
}))
defer srv.Close()
s := New(logger.New(), metrics.New())
s.url = srv.URL
s.RegisterHandler(updateHandler{})
s.RegisterRawHandler(func(string) { s.Stop() })
// Start would run the stream in the background, where the test cannot
// wait for it. Setting cancel as Start does lets Stop cancel the stream
// run here instead.
ctx, cancel := context.WithCancel(context.Background())
s.cancel = cancel
if err := s.stream(ctx); !errors.Is(err, context.Canceled) {
t.Fatalf("stream returned %v, want %v", err, context.Canceled)
}
// A second Stop must not close the queues again.
s.Stop()
}
// TestReconnectWhenStreamEnds has the server end the first response
// normally, as when RIS Live restarts. The streamer must treat that as a
// disconnect and connect again.
func TestReconnectWhenStreamEnds(t *testing.T) {
endResponse := func(http.ResponseWriter, *http.Request) {}
checkReconnect(t, endResponse, streamIdleTimeout)
}
// TestReconnectWhenStreamGoesIdle has the server hold the first response
// open without sending anything. The streamer must drop it once the idle
// timeout passes and connect again.
func TestReconnectWhenStreamGoesIdle(t *testing.T) {
const idleTimeout = 50 * time.Millisecond
holdResponse := func(w http.ResponseWriter, r *http.Request) {
w.(http.Flusher).Flush()
<-r.Context().Done()
}
checkReconnect(t, holdResponse, idleTimeout)
}
// TestReconnectWhenServerNeverAnswers has the server accept the first
// request and never answer it, not even with headers. The streamer must give
// up on it once the idle timeout passes and connect again.
func TestReconnectWhenServerNeverAnswers(t *testing.T) {
const idleTimeout = 50 * time.Millisecond
neverAnswer := func(_ http.ResponseWriter, r *http.Request) {
<-r.Context().Done()
}
checkReconnect(t, neverAnswer, idleTimeout)
}
// TestStreamNotDroppedWhileLinesArrive has the server send lines well within
// the idle timeout for several times the timeout, then end the response. Each
// line must start the idle timeout over, so the stream lasts until the server
// ends it instead of being dropped as idle.
func TestStreamNotDroppedWhileLinesArrive(t *testing.T) {
const (
idleTimeout = 100 * time.Millisecond
lineInterval = idleTimeout / 10
sendFor = 5 * idleTimeout
)
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
deadline := time.Now().Add(sendFor)
for time.Now().Before(deadline) {
_, _ = io.WriteString(w, updateLine)
w.(http.Flusher).Flush()
time.Sleep(lineInterval)
}
}))
defer srv.Close()
s := New(logger.New(), metrics.New())
s.url = srv.URL
s.idleTimeout = idleTimeout
if err := s.stream(context.Background()); !errors.Is(err, errStreamEnded) {
t.Fatalf("stream returned %v, want %v", err, errStreamEnded)
}
}
// checkReconnect runs a streamer against a fake RIS Live server. The first
// response is served by firstResponse; every later one carries one UPDATE
// line and then stays open. The streamer must come back for a second
// response, must not be marked connected when it does, and must handle the
// UPDATE in it.
func checkReconnect(t *testing.T, firstResponse http.HandlerFunc, idleTimeout time.Duration) {
t.Helper()
const maxWait = 5 * time.Second
tracker := metrics.New()
connectedAtReconnect := make(chan bool, 1)
var requests atomic.Int32
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if requests.Add(1) == 1 {
firstResponse(w, r)
return
}
select {
case connectedAtReconnect <- tracker.IsConnected():
default:
}
_, _ = io.WriteString(w, updateLine)
w.(http.Flusher).Flush()
<-r.Context().Done()
}))
defer srv.Close()
s := New(logger.New(), tracker)
s.url = srv.URL
s.idleTimeout = idleTimeout
s.minBackoff = time.Millisecond
handler := updateHandler{handled: make(chan struct{}, 1)}
s.RegisterHandler(handler)
if err := s.Start(); err != nil {
t.Fatalf("Start returned error: %v", err)
}
defer s.Stop()
select {
case <-handler.handled:
case <-time.After(maxWait):
t.Fatalf("no message from a second response handled within %s", maxWait)
}
if <-connectedAtReconnect {
t.Error("streamer was still marked connected when it came back for a second response")
}
}
// settledGoroutineCount lets transient goroutines finish, then reports the // settledGoroutineCount lets transient goroutines finish, then reports the
// current count. // current count.
func settledGoroutineCount() int { func settledGoroutineCount() int {
internal/templates/prefix_detail.html
-1
View File
@@ -283,7 +283,6 @@
<div class="origin-item"> <div class="origin-item">
<a href="/as/{{.ASN}}" class="as-link">AS{{.ASN}}</a> <a href="/as/{{.ASN}}" class="as-link">AS{{.ASN}}</a>
{{if .Handle}} ({{.Handle}}){{end}} {{if .Handle}} ({{.Handle}}){{end}}
{{if .Description}} {{.Description}}{{end}}
<span style="color: #7f8c8d; margin-left: 10px;">{{.PeerCount}} peer{{if ne .PeerCount 1}}s{{end}}</span> <span style="color: #7f8c8d; margin-left: 10px;">{{.PeerCount}} peer{{if ne .PeerCount 1}}s{{end}}</span>
</div> </div>
{{end}} {{end}}
internal/version/version.go
+1 -2
View File
@@ -7,8 +7,7 @@ package version
var ( var (
// GitRevision is the git commit hash // GitRevision is the git commit hash
GitRevision = "unknown" GitRevision = "unknown"
// GitRevisionShort is the version the page footer shows: the tag or // GitRevisionShort is the short git commit hash (7 chars)
// short commit hash from `git describe --tags --always`
GitRevisionShort = "unknown" GitRevisionShort = "unknown"
) )
script/docker
+2 -11
View File
@@ -1,8 +1,7 @@
#!/bin/sh #!/bin/sh
# script/docker: build the Docker image tagged with the project name. # script/docker: build the Docker image tagged with the project name.
# Identical in all repos; the tag comes from script/projectname. # Identical in all repos; the tag comes from script/projectname.
# --no-cache because the gate phases the final stage depends on are RUN # Generic: needs no adaptation.
# steps, and a cached one is a check that did not run.
set -eu set -eu
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd -P)" SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd -P)"
@@ -10,15 +9,7 @@ ROOT="$(cd "$SCRIPT_DIR/.." && pwd -P)"
main() { main() {
cd "$ROOT" cd "$ROOT"
# Own line: a failing command substitution inside an argument does docker build -t "$("$SCRIPT_DIR/projectname")" .
# not trip `set -e`, so the inline form degrades silently to an
# empty constant. The VERSION build argument takes precedence over
# the version a build stage derives from the .git in the context.
version="$(git describe --tags --always --dirty 2>/dev/null || true)"
[ -n "$version" ] || version="unknown"
docker build --no-cache \
--build-arg VERSION="$version" \
-t "$("$SCRIPT_DIR/projectname")" .
} }
main "$@" main "$@"